Senior Data Engineer
You design, develop, deploy, and operate production ELT pipelines and data architectures on a Big Data Platform. You integrate diverse data sources, establish engineering and governance practices, ensure data quality, optimize performance, and collaborate with stakeholders across technical and business functions.
Responsibilities
- Engage stakeholders to understand requirements and develop solutions
- Design, develop, deploy, and operate production ELT pipelines
- Design scalable data architectures
- Integrate data from varied sources and formats
- Establish and advocate for performance, code quality, validation, governance, and discoverability practices
- Mentor, train, and share knowledge
- Ensure data accuracy, completeness, reliability, relevance, and timeliness
- Implement testing, monitoring, and validation protocols
- Identify and resolve pipeline and system performance bottlenecks
- Optimize queries and resource utilization with caching, indexing, partitioning, and Spark optimizations
Requirements
- Over 7 years of experience as a Data Engineer or in a similar role
- Expert-level Software Engineering and Data Engineering practices
- Proficiency in Python, PySpark, Airflow, Hadoop, Spark, Kafka, SQL, and Git
- Ability to communicate complex data concepts to diverse stakeholders
- Experience establishing data standards and fostering a data-centric culture
- Testing and validation experience with tools such as Pytest