Senior Software Engineer Data Engineering
You will design, build, and operate production ETL pipelines for the Caspian Data Platform using Databricks, Delta Live Tables, and Spark. You will define governed data models, lead technical designs, build observability, integrate AI tooling into data workflows, mentor engineers, and contribute to infrastructure, data contracts, access controls, and lineage.
Responsibilities
- Design, build, and operate production ETL pipelines on Databricks
- Apply medallion architecture patterns across business domains
- Define governed and reusable data models
- Drive technical design for cross-functional projects
- Write design documents covering architecture, tradeoffs, and rollout plans
- Build pipeline failure detection, alerting, retry logic, and operational controls
- Integrate agentic systems, automated code generation, and natural language interfaces into data workflows
- Mentor engineers
- Review code and technical designs
- Contribute to platform infrastructure through Terraform
- Establish data contracts, access controls, and lineage with Unity Catalog
Requirements
- 5–8 years of data engineering experience
- Experience building and maintaining production pipelines and data models
- Proficiency with Databricks, Delta Live Tables, Unity Catalog, Delta Lake, and PySpark or Spark SQL
- Strong SQL skills
- AWS experience building and operating data systems at scale
- Python proficiency for data engineering
- Experience with GitLab or similar CI/CD tooling
- Experience with Terraform
- Exposure to AI tooling in a data context
- Ability to define data models and pipeline designs from ambiguous requirements
- Familiarity with financial, payments, or blockchain data
- Experience with Fivetran or similar ingestion tools
Benefits
- Professional development budget
- Flexible in-office schedule
- Team offsites
- Team bonding activities
- Happy hours
- Competitive salary
- Bonuses
- Equity
- Physical healthcare
- Mental healthcare
- Retirement benefits
- Family-forming benefits
- Family support
- Employee giving match
- Mobile phone stipend
- R&R days
- Wellness reimbursement
- Vacation policy
- Parental leave
- Family planning benefits
- Catered lunches
- Stocked kitchens
- Premium snacks and beverages