Data Engineer
About the company
Valleysoft is a regional IT services provider delivering enterprise technology, application development, process management, and IT support services to clients around the world. Working across the information technology and services sector, the company helps organizations build reliable, scalable digital solutions that support complex business and operational needs.
Role overview
This role is an opportunity to contribute to modern data platform initiatives that power analytics and AI use cases across the enterprise. As a Data Engineer – Data Platforms (Databricks), you will play a central role in shaping data infrastructure, supporting cross-functional teams, and helping ensure data systems are performant, secure, and built to scale.
Responsibilities
- Design, develop, and maintain scalable data pipelines for batch and real-time processing.
- Build and optimize enterprise data platforms using Databricks and Apache Spark.
- Develop ETL/ELT processes to ingest, transform, and integrate data from multiple sources.
- Implement Delta Lake architecture and maintain data lake/lakehouse solutions.
- Collaborate with data scientists, AI engineers, business analysts, and application teams to support analytics and AI initiatives.
- Optimize data storage, partitioning, and query performance.
- Develop reusable data engineering frameworks and automation solutions.
- Implement data quality, validation, lineage, and governance processes.
- Ensure data security, privacy, and compliance with enterprise standards.
- Monitor platform performance, troubleshoot production issues, automate deployment and infrastructure, contribute to architecture reviews, and maintain technical documentation and runbooks.
Requirements
- Bachelor's degree in computer science, information technology, engineering, or a related field.
- 6+ years of data engineering experience working with Databricks and Apache Spark.
- Strong experience with Python and SQL, along with hands-on involvement in ETL/ELT processes.
- Experience working with Delta Lake and cloud platforms.
- Fluent in English.
- Eligibility to work in Egypt.