Pyspark Data Engineer

Job Title: PySpark Data Engineer – HL7 Integration Specialist
Job Summary
We are seeking a highly skilled PySpark Data Engineer with strong experience in HL7 healthcare data standards and large-scale data processing. The ideal candidate will be responsible for designing, developing, and optimizing healthcare data integration pipelines, transforming HL7 messages, and enabling efficient data exchange across healthcare systems.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using PySpark.
Process, transform, and validate healthcare data based on HL7 standards (any version such as HL7 v2.x, HL7 v3, or FHIR).
Build and optimize ETL/ELT workflows for healthcare interoperability solutions.
Collaborate with business stakeholders, architects, and healthcare domain teams to understand data integration requirements.
Ensure data quality, integrity, security, and compliance with healthcare regulations.
Troubleshoot and resolve data processing and integration issues.
Develop reusable frameworks and best practices for healthcare data ingestion and transformation.
Monitor pipeline performance and implement optimization strategies.
Create and maintain technical documentation and data mapping specifications.
Required Skills (Must Have)
Strong hands-on experience with PySpark and distributed data processing
Solid understanding of HL7 standards (any version) and healthcare data exchange concepts
Experience designing and implementing data transformation and integration solutions
Proficiency in SQL and data modeling concepts
Experience working with large-scale datasets and ETL processes
Strong problem-solving and analytical skills
Excellent communication and collaboration abilities
Preferred Skills (Good to Have)
Experience with SMILE CDR.
Knowledge of Google Cloud Platform (GCP) services
Experience with Informatica (PowerCenter, IICS, or related tools)
Familiarity with healthcare interoperability standards such as FHIR
Exposure to cloud-based data engineering architectures
Qualifications
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
Relevant experience in data engineering, healthcare interoperability, or healthcare data integration projects.
Experience
4+ years of experience in Data Engineering
2+ years of hands-on experience with PySpark and healthcare data integration using HL7