Senior Data Engineer / Databricks Engineer

Oremda Infotech Inc is seeking a Senior Data Engineer / Databricks Engineer to develop and maintain scalable data pipelines and end-to-end AWS ETL/ELT workflows. The role focuses on Databricks Lakehouse architecture, performance optimization, data integration, quality and security, and DevOps practices for data applications.

Responsibilities:

• Develop and maintain scalable data pipelines using Apache Spark on Databricks
• Build end-to-end ETL/ELT pipelines on AWS using S3, Glue, Lambda, EMR, and Step Functions
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions
• Design and implement data models, schemas, and Lakehouse architecture in Databricks
• Optimize and tune Spark jobs for performance and cost efficiency
• Integrate data from multiple structured and unstructured data sources
• Monitor and manage data workflows, ensuring data quality, consistency, and security
• Follow CI/CD, Git version control, and DevOps best practices for data applications
• Write clean, reusable, and well-documented code using Python, PySpark, and Scala

Requirements:

• 10+ years of total experience in Data Engineering or Big Data Development
• 7+ years of hands-on experience with Databricks and Apache Spark
• Proficiency in AWS cloud services: S3, Glue, Lambda, EMR, Redshift, CloudWatch, and IAM
• Strong programming skills in Python, PySpark, and optionally Scala
• Solid understanding of Data Lakes, Lakehouse, and Delta Lake concepts
• Experience in SQL development and performance tuning
• Familiarity with Airflow, dbt, or similar orchestration tools
• Experience with CI/CD tools such as Jenkins, GitHub Actions, or AWS CodePipeline
• Knowledge of data security, governance, and compliance frameworks