Senior AI Data Pipeline Engineer

Role Overview

• Design, build, and optimize scalable batch and real-time data pipelines
• Manage and administer Databricks workspaces, clusters, jobs, and performance tuning
• Develop and maintain streaming architectures using Kafka
• Implement and manage Change Data Capture (CDC) pipelines using Debezium connectors
• Deploy, monitor, and manage containerized workloads using Kubernetes
• Implement CI/CD practices for data engineering workflows
• Ensure data quality, observability, governance, and security best practices
• Collaborate with data scientists, ML engineers, and software engineers to deliver production-grade data solutions
• Support and optimize AI/ML data pipelines and model deployment workflows
• Troubleshoot production issues and implement performance improvements

Requirements

• 3+ years of experience in data engineering or a related field
• Strong hands-on experience managing and optimizing Databricks
• Experience building and maintaining streaming pipelines with Kafka
• Experience implementing Change Data Capture (CDC) using Debezium connectors
• Practical experience deploying and operating services in Kubernetes
• Strong proficiency in Python and/or Scala
• Experience with SQL and distributed data processing frameworks (e.g., Spark)
• Familiarity with cloud platforms (AWS, Azure, or GCP)
• Experience with infrastructure-as-code tools (Terraform, etc.)
• Strong understanding of distributed systems concepts

Tech Stack

• AWS
• Azure
• Cloud
• Distributed Systems
• Google Cloud Platform
• Kafka
• Kubernetes
• Python
• Scala
• Spark
• SQL
• Terraform

Benefits

• Medical
• Dental
• Vision
• Life
• Disability
• Time off plans
• Retirement plans
• Paid Parental Leave
• Employee Stock Purchase Plan
• Tax savings plans for health, dependent care and commuter expenses