Design, build, and maintain production-grade data pipelines (streaming and batch)
Implement event-driven ingestion and near real‑time processing
Build transformations using distributed data processing frameworks
Write and maintain clean, testable Python code following software engineering best practices
Support CI/CD pipelines, automated deployments, and environment promotions
Monitor and troubleshoot pipeline failures, latency issues, and data quality problems
Collaborate closely with Data Scientists, Analysts, and platform engineers

2+ years of experience as a Data Engineer or Software Engineer working with data
Strong experience writing production Python code
Experience building and supporting pipelines on Azure or an equivalent cloud platform (AWS/GCP)
Hands‑on experience with distributed data processing, such as: Apache Spark, Databricks (or Spark runners such as EMR, Dataproc)
Experience working with event streaming or messaging platforms, such as: Azure Event Hubs, Apache Kafka / Confluent Kafka, Amazon Kinesis
Practical experience with CI/CD pipelines, version control, and automated deployments
Familiarity with data modeling, schema management, and data reliability concepts
Experience operating systems in an agile, collaborative engineering team

Engineer II, Data

Key skills