Databricks Engineer AWS / PySpark (Only W2)

Senior Databricks Data Engineer AWS / PySpark

Job Description:

We are looking for a Senior Databricks Data Engineer with strong hands-on experience in designing, implementing, and optimizing enterprise-scale data platforms. The ideal candidate must have real-world Databricks implementation experience, not just Databricks training or certification.

Key Responsibilities:

• Design, develop, and implement scalable batch and real-time data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize enterprise ETL/ELT pipelines using Databricks and Delta Lake.
• Develop scalable Lakehouse architecture and modernize existing enterprise data platforms.
• Implement and manage Delta Lake solutions, including data ingestion, transformations, optimization, and performance tuning.
• Work with AWS data services and integrate Databricks with cloud-based data platforms.
• Develop and maintain Kafka-based streaming pipelines for real-time data processing.
• Perform Spark and Databricks performance optimization, including cluster configuration, partitioning, caching, and query optimization.
• Implement Unity Catalog for data governance, security, access control, and data discovery.
• Establish data quality, validation, monitoring, and governance processes.
• Implement CI/CD pipelines for data engineering and Databricks deployments.
• Collaborate with architects, data analysts, developers, and business stakeholders to deliver enterprise data solutions.

Required Skills:

• Strong hands-on experience with Databricks
• Extensive experience with PySpark, Python, and SQL
• Strong experience with Delta Lake
• Enterprise ETL/ELT pipeline development
• Experience with AWS cloud data services
• Strong knowledge of Apache Spark and Spark optimization
• Hands-on experience with Kafka / real-time streaming
• Experience with Databricks Unity Catalog
• Knowledge of data governance and data quality
• Experience implementing CI/CD for data platforms
• Strong understanding of Lakehouse architecture
• Experience with enterprise data platform modernization

Important Requirement:

• Candidates must have proven hands-on Databricks implementation experience in production environments.
• Do NOT submit candidates whose primary experience is Snowflake and who have only completed Databricks training/certification.
• Databricks should be a core part of the candidate's actual project experience, including development, implementation, optimization, and production support.

Preferred Experience:

• Experience migrating legacy ETL/data warehouse platforms to Databricks.
• Experience working with large-scale enterprise datasets.
• Experience implementing real-time and batch data architectures.
• Experience with cloud-native data engineering and modern Lakehouse platforms.