About this role

Responsibilities and duties:

Data Engineering

Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
Optimize Databricks jobs, compute, and data layouts for performance and cost.
Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
Implement data quality checks, monitoring, error handling, and recovery mechanisms.

Cloud & Platform Engineering

Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry, and Azure Monitor.
Experience with cloud AI services, including vector search and embedding pipelines.
Improve platform performance, scalability, reliability, security, and cost efficiency.
Implement secure identity and access management using Azure managed identities, Key Vault, and secrets managed via scopes.
Support development, staging, and production environments.

Automation & Operations

Improve platform observability through logging, monitoring, dashboards, and alerting.
Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
Support production incidents and root-cause analysis.
Collaborate with application, AI/ML, and data engineering teams.

Requirements:

5+ years of experience in data engineering, cloud engineering, or a related role.
Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
Strong skills in Python/PySpark and SQL.
Experience building and operating production data ETL/ELT pipelines.
Experience with relational and semi-structured/unstructured data and database technologies.
Experience with Spark, Delta Lake, and lakehouse architectures.
Working knowledge of Databricks Unity Catalog and data-access controls.
Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.

Nice to Have:

Experience with Terraform or ARM templates.
Experience with Azure DevOps or GitHub Actions.
Familiarity with Azure networking, including VNets and private endpoints.
Experience with Azure Data Factory, Databricks Workflows, or Airflow.
Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.

Data Engineer

Key skills