LEAD DATABRICKS DATA ENGINEER

Virtual Connect Solutions is seeking a Lead Databricks Data Engineer to lead the data platform and MLOps workstream for a large U.S. enterprise client. The role owns a governed Databricks lakehouse, production data pipelines, feature and prediction data products, MLOps operations, architecture decisions, technical standards, and engineering mentorship.

Responsibilities:

• Lead the Databricks data platform workstream; set technical direction, sequence delivery, and own outcomes with client data science and IT stakeholders
• Design and build production medallion architecture (Bronze, Silver, and Gold) using Databricks, Apache Spark, Delta Lake, and Delta Live Tables or Lakeflow Declarative Pipelines
• Engineer reliable batch and, where needed, streaming pipelines that integrate operational, CRM, HRIS/payroll, web, finance, and other enterprise sources
• Implement Unity Catalog governance, lineage, RBAC, service principals, secrets management, environment isolation, and least-privilege controls
• Build reusable feature, training, and prediction tables with versioning and retained history so models are reproducible and retrainable
• Establish CI/CD and Databricks Asset Bundles for data and ML workloads, including automated testing, environment parameterization, approvals, and rollback
• Own orchestration through Databricks Workflows/Jobs, including dependencies, retries, recovery, observability, alerting, and pipeline SLAs
• Build the MLOps layer with MLflow experiment tracking, model registry/versioning, batch inference, model promotion, monitoring, and rollback
• Implement data-quality and observability controls that detect schema drift, null spikes, freshness failures, and out-of-range values before downstream impact
• Tune Spark jobs, clusters, SQL warehouses, storage layout, and Delta tables for performance, reliability, and cost efficiency
• Enable governed access from Python and R notebooks, BI tools such as Power BI, and downstream APIs
• Mentor engineers, conduct code and design reviews, produce architecture decisions and runbooks, and keep delivery unblocked

Requirements:

• 6+ years of data engineering or data platform experience, including technical leadership or end-to-end ownership of a production platform
• At least 3 years of recent, hands-on Databricks experience in production environments. Candidates whose primary experience is only Snowflake, Microsoft Fabric, BigQuery, or another platform will not meet this requirement
• Demonstrated production expertise with Apache Spark/PySpark, Delta Lake, Databricks Workflows/Jobs, Unity Catalog, and MLflow
• Strong Python and SQL skills, including complex transformations, performance tuning, testing, and maintainable production code
• Experience building scalable ETL/ELT pipelines with incremental processing, schema evolution, change data capture, backfills, and dimensional/data-product modeling
• Hands-on CI/CD for Databricks using Git and automated deployment patterns; experience with Databricks Asset Bundles, Terraform, or equivalent infrastructure-as-code
• Experience implementing data quality, lineage, observability, security, and environment promotion across development, test, and production
• Working knowledge of a major cloud platform; Azure experience is preferred, but AWS or GCP is acceptable when paired with strong Databricks depth
• Ability to lead client working sessions, document decisions clearly, explain tradeoffs to senior stakeholders, and remain accountable for technical outcomes
• Excellent written and verbal communication and the ability to operate effectively with distributed U.S. and offshore teams
• Ability and willingness to travel occasionally within the United States based on client and project needs
• Currently located in the United States and able to work remotely during U.S. business hours and travel occasionally
• Interested in direct W-2 employment with NRnP Technology. No C2C candidates or third-party submissions
• Databricks Data Engineer Professional, Databricks Machine Learning Professional, or comparable certification
• Experience with Azure Databricks, ADLS Gen2, Azure DevOps, Entra ID, Key Vault, Event Hubs, and Power BI
• Experience with Structured Streaming, Kafka/Event Hubs, Change Data Feed, Auto Loader, and low-latency data or inference patterns
• Machine-learning engineering experience spanning feature engineering, model deployment, batch scoring, monitoring, and drift detection
• Familiarity with dbt, Airflow, R-based data science workflows, semantic layers, and enterprise API integration
• Prior consulting experience delivering data platforms for large U.S. enterprise clients
• Experience using AI-assisted engineering tools responsibly to accelerate implementation, testing, documentation, and code review