Data Engineer

CAQH is the trusted data connector at the core of healthcare, and they are seeking a Data Engineer to design, develop, and optimize advanced data pipelines and infrastructure. This role is critical for delivering scalable data solutions that support analytics and AI/ML initiatives across the organization.

Responsibilities:

• Design, build, and maintain complex ETL/ELT pipelines across Databricks, Azure SQL, and downstream gold-layer models supporting priority projects
• Lead development and evolution of enriched data models, including field-level enrichment logic, recency rules, and provider-level enrichment flags
• Own data logic, including reconciliation between source and target data sources and resolution of duplication and data discrepancies
• Implement and refine medallion architecture patterns (bronze → gold), ensuring data quality, traceability, and performance at scale
• Identify, document, and remediate systemic data quality issues, including null handling, soft deletes, authorization flags, and incorrect organizational mappings
• Define and operationalize rules for data in collaboration with product, governance, and engineering stakeholders
• Produce authoritative documentation (Confluence, mapping workbooks) to serve as a single source of truth for enrichment logic and data behavior
• Act as a primary technical counterpart for vendors providing detailed queries, validation logic, and corrective guidance on upstream data issues
• Partner closely with product owners, architects, and application teams to ensure data models align with product defined use cases
• Support UAT and release readiness by preparing data, validating counts, and resolving last‑mile data defects under tight timelines
• Provide hands-on technical leadership without formal direct reports, mentoring peers and guiding best practices in SQL, Databricks notebooks, and data modeling
• Influence architectural decisions related to Databricks compute, job scheduling, and environment promotion strategies

Requirements:

• 4–7 years of hands-on experience in a data engineering or analytics engineering role
• Demonstrated success leading data modernization or migration initiatives in cloud environments
• Bachelor's degree in Computer Science, Information Systems, Data Engineering, or a related field
• Advanced SQL expertise (complex joins, reconciliation, performance tuning)
• Deep hands-on experience with Databricks, Delta Lake, and Azure SQL
• Strong data modeling skills for analytical, operational, and API‑driven use cases
• Proven ability to debug and stabilize messy, evolving enterprise data domains
• Excellent written and verbal communication, especially for explaining complex data behavior to non‑technical stakeholders
• Experience using Git, DevOps tools, and CI/CD pipelines for data engineering workflows
• Azure Data Engineer Associate or related certification
• Coursework or certification in AI/ML