Principal Engineer, Data

Ensemble Health Partners is a provider of technology-enabled revenue cycle management solutions for health systems, including hospitals and affiliated physician groups. The Principal Data Engineer will serve as the senior technical authority for data architecture, designing a scalable Azure and Databricks data platform, enabling AI and advanced analytics, and owning the data engineering technology roadmap. The role will also establish standards, optimize platform costs and reliability, resolve complex engineering issues, and mentor technical teams without formal management authority.

Responsibilities:

• Conduct cross-organizational assessments of existing data pipelines, platforms, and engineering practices to identify inefficiencies, technical debt, risk, and opportunities for improvement
• Architect scalable, secure, and cost-efficient data solutions that span multiple teams and business domains, built on Databricks and Azure
• Act as a technical escalation point for the organization’s hardest data engineering problems, diagnosing recurring or systemic reliability issues in tier-1 pipelines and datasets
• Define and evolve data engineering standards, patterns, and architectural guardrails used across engineering teams, balancing standardization with the flexibility teams need to move quickly
• Establish and evolve a data platform architecture that enables AI, machine learning, and advanced analytics workloads, including feature readiness, training data quality, lineage, and observability
• Partner with Data Science, Analytics, Product, and Cloud teams to ensure data platforms support model development, deployment, and monitoring without creating operational or governance risk
• Define and enforce AI-ready data standards, including data quality thresholds, metadata completeness, schema stability, and data timeliness required to support advanced analytics and AI workloads
• Stay ahead of the industry by continuously evaluating emerging tools, frameworks, and techniques in the data and AI space for genuine fit within Ensemble’s environment, rather than chasing trends for their own sake
• Run proofs of concept and technical spikes to validate new approaches before recommending broader adoption
• Author and maintain a data engineering technology roadmap aligned to business priorities, with clear reasoning on trade-offs, cost, and adoption sequencing
• Present roadmap recommendations and their business impact to both technical and executive stakeholders, translating deep technical trade-offs into clear strategic choices
• Treat enterprise datasets as data products, with clear ownership, quality guarantees, documentation, and usage metrics
• Establish platform standards and architectural guardrails to optimize compute efficiency, storage utilization, query performance, and pipeline reliability
• Partner with Finance, Cloud, and Engineering leaders to implement cost transparency (FinOps) across Databricks and Azure, including budgets, usage monitoring, and proactive optimization
• Drive platform decisions using clear metrics such as cost-to-serve, workload efficiency, platform SLAs, and utilization trends
• Ensure platform-level observability, including workload performance, cost drivers, usage patterns, and capacity forecasting
• Mentor senior and staff engineers on architecture and design thinking, elevating their capability without holding formal management responsibility for their performance
• Influence technical direction across teams through credibility, clear communication, and sound technical judgment rather than positional authority
• Partner with data engineering, analytics, data science, and platform/infrastructure teams to ensure architectural decisions stay aligned with business goals

Requirements:

• 10+ years of relevant job experience
• Deep experience designing and reviewing SQL‑based data models and transformations at enterprise scale
• 3+ years working with big data technologies including but not limited to Databricks, SPARK, Azure, with a willingness and ability to learn new ones
• Excellent understanding of engineering fundamentals: testing automation, code reviews, telemetry, iterative delivery and DevOps
• Experience with polyglot storage architectures including relational, columnar, key-value, graph or equivalent
• Demonstrated ability to communicate effectively to both technical and non-technical, globally distributed audiences
• Solid foundations in formal architecture, design patterns and best practices
• Experience designing data platforms that support AI/ML, advanced analytics, or intelligent automation workloads
• Deep experience with Azure cloud services and Databricks, including Spark‑based data engineering patterns
• Proven ability to translate business needs into scalable data solutions
• Experience owning or operating large‑scale cloud data platforms, including responsibility for cost optimization, performance tuning, and platform reliability
• Demonstrated ability to scale data engineering through enablement, standards, and platforms, not just direct delivery
• Demonstrated experience assessing data engineering practices across multiple teams or business units and translating findings into a concrete architectural or technology roadmap
• Track record of evaluating emerging tools and technologies for real fit within an organization, and driving adoption decisions through credibility and clear communication rather than formal authority
• Must be inquisitive and demonstrate openness to innovation including AI to explore better processes and ways to alleviate friction and improve patient and client experiences
• This is a remote position; however, candidates must be willing and able to travel to and work onsite at client, temporary, or corporate office locations as business needs require
• Strong understanding of Databricks pricing models, workload optimization, and Azure cloud cost drivers preferred