Senior Data Engineer

Role Overview

• Design and implement scalable data architectures to process high-volume IoT sensor data and telemetry streams, ensuring reliable data capture and processing for AI/ML workloads
• Build and maintain data pipelines for AI product lifecycle, including training data preparation, feature engineering, and inference data flows
• Develop and optimize RAG (Retrieval Augmented Generation) systems, including vector databases, embedding pipelines, and efficient retrieval mechanisms
• Lead the architecture and development of scalable data platforms on Databricks
• Drive the integration of GenAI capabilities into data workflows and applications
• Optimize data processing for performance, cost, and reliability at scale
• Create robust data integration solutions that combine industrial IoT data streams with enterprise data sources for AI model training and inference
• Implement DataOps practices to ensure continuous integration and delivery of data pipelines powering AI solutions
• Design and maintain automated testing frameworks for data quality, data drift detection, and AI model performance monitoring
• Create self-service data assets enabling data scientists and ML engineers to access and utilize data efficiently
• Design and maintain automated documentation for data lineage and AI model provenance
• Partner with ML engineers and data scientists to implement efficient data workflows for model training, fine-tuning, and deployment
• Mentor team members and provide technical leadership on complex data engineering challenges
• Establish data engineering best practices, including modular code design and reusable frameworks
• Drive projects to completion while working in an agile environment with evolving requirements in the rapidly changing AI landscape

Requirements

• Minimum 5 years of experience building production data pipelines in Databricks processing TB scale data
• Extensive experience implementing medallion architecture (Bronze/Silver/Gold) with Delta Lake, Delta Live Tables (DLT), and Lakeflow for batch and streaming pipelines from Event Hub or Kafka sources
• Strong hands-on proficiency with PySpark for distributed data processing and transformation
• Strong experience working with cloud platforms such as Azure, GCP and Databricks, especially in designing and implementing AI/ML-driven data workflows
• Proficient in CI/CD practices using Databricks Asset Bundles (DAB), Git workflows, GitHub Actions, and understanding of DataOps practices including data quality testing and observability
• Hands-on experience building RAG applications with vector databases, LLM integration, and agentic frameworks like LangChain, LangGraph
• Natural analytical mindset with demonstrated ability to explore data, debug complex distributed systems, and optimize pipeline performance at scale
• Experience building RAG and agentic architecture solutions and working with LLM-powered applications
• Expertise in real-time data processing frameworks (Apache Spark Streaming, Structured Streaming)
• Knowledge of MLOps practices and experience building data pipelines for AI model deployment
• Experience with time-series databases and IoT data modeling patterns
• Familiarity with containerization (Docker) and orchestration (Kubernetes) for AI workloads
• Strong background in data quality implementation for AI training data
• Experience working with distributed teams and cross-functional collaboration
• Knowledge of data security and governance practices for AI systems
• Experience working on analytics projects with Agile and Scrum Methodologies
• Due to compliance with U.S. export control laws and regulations, candidate must be a U.S. Person, which is defined as a U.S. citizen, a U.S. permanent resident, or have protected status in the U.S. under asylum or refugee status, or have the ability to obtain an export authorization.

Tech Stack

• Apache
• Azure
• Cloud
• Distributed Systems
• Docker
• Google Cloud Platform
• IoT
• Kafka
• Kubernetes
• PySpark
• Spark

Benefits

• employer-subsidized Medical, Dental, Vision, and Life Insurance
• Short-Term and Long-Term Disability
• 401(k) match
• Flexible Spending Accounts
• Health Savings Accounts
• EAP
• Educational Assistance
• Parental Leave
• Paid Time Off (for vacation, personal business, sick time, and parental leave)
• 12 Paid Holidays