Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.

Responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes
• Lead the implementation of objects within the data generator framework
• Oversee deployment and testing across local and development environments
• Drive improvement of the current, actively changing solution
• Conduct bug investigation and resolution
• Perform unit testing to ensure solution quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members throughout the implementation phase

Requirements

• 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes
• Proficiency in Python
• Familiarity with the data engineering domain and cloud-agnostic platform migrations
• Capability to dive deeper into new technologies in a short time during active implementation phases
• Ability to work independently with syncs with a team lead
• Strong communication skills and proactiveness
• Showcase of being a hands-on team player and technical leader
• Proficiency in English at a B2+ level

Nice to have

• Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Skills in Ansible and Argo CD
• Understanding of TDD

Benefits

We connect like-minded people

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Five trust days per year (sick leave without a medical certificate)
• Benefits package (sports activities, a variety of stores and services)

We connect like-minded people

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Four trust days per year for personal needs
• Discounts for fitness clubs
• Benefits package (hotels, restaurants, stores and services)

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers
• Discounts in local language schools, including online courses for the Kazakh language

We cover it all:

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Six trust days per year (sick leave without a medical certificate)
• Coverage of psychology sessions of your choice
• Benefits package (sports activities, a variety of stores and services)
• Housing support program: preferential mortgage access via Otbasy Bank partnership

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all:

• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Six trust days per year (sick leave without a medical certificate)
• Coverage of psychology sessions of your choice
• Discounts for fitness clubs and sports programs
• Benefits package (sports activities, a variety of stores and services)

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

• Discounts in local language schools, including offline courses for the Uzbek language

We cover it all:

• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Four trust days per year (sick leave without a medical certificate)
• Discounts for fitness clubs, dance schools and sports programs
• Benefits package (sports activities, a variety of stores and services)