Lead Databricks Data Engineer

Perficient is a global AI-first consultancy that partners with innovative enterprises to deliver business results through AI. They are seeking a Lead Databricks Data Engineer to lead development teams and create scalable data solutions, collaborating with architects and infrastructure teams.

Responsibilities:

• Design and develop scalable data solutions
• Build high-performance ETL pipelines using Spark/PySpark
• Implement CI/CD pipelines and orchestrate Databricks workflows
• Monitor production jobs and troubleshoot issues
• Participate in Agile ceremonies

Requirements:

• Passionate developer with 8+ years of data engineering experience, with at least 3 years in a lead or senior data engineer role
• Minimum 5 years of hands-on Teradata and Databricks experience using (Delta Lake, Notebooks, Pipelines, cluster management)
• Experience in AWS services (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, CloudWatch)
• DevOps experience with Databricks CI/CD (Git, Jenkins, Artifactory)
• Experience with Unix/Linux shell scripting and administration
• Experience with SQL (Teradata SQL, Hive SQL, Spark SQL)
• Experience with Job scheduling tools (CA7 Enterprise Scheduler)
• Demonstrated ability to leverage AI tools to enhance productivity, streamline workflows, and support data-informed task execution
• A solid understanding of AI capabilities and limitations including ethical considerations is expected
• Flexible and adaptable attitude, disciplined to manage multiple responsibilities and adjust to varied environments
• Ability to produce high quality products within deadlines
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
• Certification in Databricks, Spark, AWS, Azure or other cloud platform
• Master's degree in Computer Science or related field
• Client-facing or consulting experience is highly preferred