Python__Data Engineering

Job Description

• Data Pipeline Development

  • Build and maintain ETL/ELT pipelines using Python
  • Ingest data from multiple sources (APIs, databases, files, streaming systems)
  • Optimize pipelines for performance and scalability
  • Clean, transform, and validate raw datasets

  • Handle structured and unstructured data

    Use frameworks like\:

    • Pandas
    • PySpark
    • Dask

    • Database & Data Warehousing

    • SQL (PostgreSQL, MySQL, SQL Server)
    • NoSQL (MongoDB, Cassandra)
    • Design schemas and optimize queries
    • Build data warehouses using\:
      • Snowflake
      • Redshift
      • BigQuery

      • Big Data Technologies

      • Apache Spark

        • Hadoop

        • Process large-scale datasets efficiently

        • Workflow Orchestration

        • Apache Airflow

        • Cloud Platforms

        • Work on cloud environments\:

        • AWS (S3, Glue, Lambda, EMR)

        • Azure (Data Factory, Synapse)

        • GCP (Dataflow, BigQuery)

        • Data Quality & Monitoring

        • Implement data validation checks

        • Monitor pipeline failures and fix bugs

        • Ensure data reliability and integrity