Senior Data Engineer

Job Description

We are seeking a highly skilled Senior Data Engineer to join our data and analytics team. The ideal candidate will have extensive experience in working with diverse data sources, performing data profiling, and identifying matching records across systems to create master data records. This role requires strong analytical and technical skills, along with expertise in data quality, integration, transformation, and cloud-based data platforms.

Roles & Responsibilities

Data Source Analysis:

• Analyze multiple structured and unstructured data sources to understand schemas, data quality, and relationships.

• Assess and document the metadata, formats, and business rules of incoming data sets.

Data Profiling & Matching:

• Perform comprehensive data profiling to assess completeness, accuracy, uniqueness, and consistency of data.

• Develop logic to match and reconcile records across different systems using keys, rules, or fuzzy matching techniques.

Master Data Management (MDM):

• Design and implement processes for building and maintaining golden master records.

• Support creation and maintenance of master data across domains such as customer, product, asset, etc.

• Handle data deduplication and standardization.

ETL & Data Pipelines:

• Build scalable data pipelines using Azure Data Factory for ingesting, transforming, and cleansing data from multiple sources.

• Ensure efficient data movement across staging, processing, and analytical layers.

• Leverage Azure Synapse Analytics (optional) for advanced data processing and analytics

Collaboration & Governance:

• Work closely with business analysts, data stewards, and application owners to understand data definitions and ownership.

• Enforce data governance standards and best practices.

Performance & Optimization:

• Monitor data workflows and optimize performance for large-scale processing.

• Troubleshoot data issues and implement robust logging and monitoring.