Senior Data Engineer- Data Genie, Databricks

We are seeking a highly experienced Databricks Genie Engineer with data engineering experience and deep expertise in the Databricks Lakehouse platform. The ideal candidate will design, build, and optimize scalable data pipelines, implement AI-powered data interactions using Databricks Genie, and drive enterprise-grade data solutions across cloud environments.

You will play a key role in architecting modern data platforms, enabling self-service analytics through natural language querying (Genie), and ensuring high performance, security, and cost optimization across large-scale workloads.

Key Responsibilities

Databricks & Lakehouse Engineering

• Design and implement scalable data pipelines using Databricks (PySpark, SQL).
• Architect and manage Delta Lake tables (Bronze, Silver, Gold layers).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement Unity Catalog for data governance and access control.
• Manage cluster configuration, autoscaling, and job orchestration.

Databricks Genie Implementation

• Configure and optimize Databricks Genie for natural language-to-SQL analytics.
• Build semantic models and curated datasets to support Genie use cases.
• Improve AI-driven query accuracy and optimize prompt engineering.
• Collaborate with business users to enable self-service analytics.
• Troubleshoot and tune Genie-generated SQL for performance and correctness.

Data Architecture & ELT

• Design modern ELT architectures leveraging Delta Lake.
• Implement incremental processing (CDC, streaming with Structured Streaming).
• Develop data quality frameworks and validation pipelines.
• Handle schema evolution and medallion architecture best practices.

Cloud & Integration

• Work across AWS / Azure / Google Cloud Platform cloud platforms.
• Integrate with S3 / ADLS / GCS object storage.
• Implement CI/CD pipelines for Databricks deployments.
• Manage secrets, tokens, and secure connectivity.
• Integrate BI tools (Power BI, Tableau) and AI/ML workflows.

Required Qualifications

• 10+ years of experience in Data Engineering.
• 5+ years of hands-on experience with Databricks.
• Strong expertise in PySpark and Spark SQL.
• Deep understanding of Delta Lake and Lakehouse architecture.
• Experience implementing Databricks Genie or AI-driven analytics solutions.
• Strong experience with cloud platforms (AWS/Azure/Google Cloud Platform).
• Expertise in building scalable ELT pipelines.
• Solid understanding of data governance and security frameworks.
• Experience with CI/CD (Azure DevOps, GitHub Actions, Terraform, etc.).
• Strong troubleshooting and debugging skills.