Senior Data Engineer - Azure Data Bricks

Senior Data Engineer – Azure / Databricks

Job Title

Senior Data Engineer – Azure / Databricks

Location: Dallas, TX / Remote
Duration: 12+ Months Contract
Experience: 6+ Years

Job Description:

We are seeking a Senior Data Engineer with strong experience in designing, developing, and optimizing scalable cloud-based data platforms and ETL/ELT pipelines.

Responsibilities:

• Design and develop scalable ETL/ELT data pipelines using Azure Data Factory, Databricks, PySpark and SQL.
• Develop data ingestion and transformation workflows for structured and semi-structured data.
• Build and optimize PySpark and Spark SQL jobs for large-scale distributed data processing.
• Develop data models, data marts, and analytical datasets for enterprise reporting and analytics.
• Work with Azure Data Lake Storage, Azure Synapse, Databricks and Snowflake.
• Implement batch and real-time data processing using Kafka or similar streaming technologies.
• Optimize SQL queries, Spark jobs, partitions, joins and transformations for performance.
• Implement data quality, validation, reconciliation and monitoring processes.
• Collaborate with data scientists, analysts, application developers and business stakeholders.
• Deploy data pipelines through CI/CD using Azure DevOps, Git and Terraform.
• Troubleshoot production pipeline failures and improve reliability and performance.

Required Skills:

• 6+ years of Data Engineering experience
• Strong SQL and Python/PySpark
• Azure Data Factory
• Azure Databricks
• Azure Data Lake / ADLS
• Apache Spark
• Data Warehousing
• ETL/ELT
• Azure Synapse or Snowflake
• Git / CI/CD
• Strong data modeling and performance tuning experience

Preferred:

• Kafka
• Terraform
• Kubernetes
• Airflow
• Power BI
• AWS/Google Cloud Platform exposure