Key skills
PySparkDatabricksCI/CD pipelinesApache KafkaGitHubAnalytical skillsTroubleshooting skillsHealthcare data integrationREST APIsAgile/ScrumData EngineeringSparkApache SparkVersion ControlAgileScrumCI/CD
About this role
Luxoft is seeking a skilled and hands-on Data Engineer with proven experience in Databricks and real-time data streaming solutions. The role involves implementing scalable PySpark code, maintaining CI/CD pipelines, and collaborating with various teams to ensure system stability and performance.
Responsibilities:
- Implement resource composition logic provided by the Product team into scalable and efficient PySpark code
- Develop, optimize, and maintain PySpark applications within the Databricks environment
- Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment
- Write and maintain unit tests to ensure high code quality and reliability
- Perform data validation and quality checks to ensure compliance with defined business rules
- Support and manage production deployments, ensuring system stability and performance
- Troubleshoot and resolve issues across development, testing, and production environments
- Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation
- Maintain technical documentation for code, workflows, and deployment processes
- Pull from and push code to GitHub repositories, following version control and branching best practices
Requirements:
- 5+ years of experience in data engineering roles
- Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)
- Strong hands-on experience with PySpark and/or Apache Spark
- Experience developing in Databricks
- Experience working with CI/CD pipelines and DevOps best practices
- Proficiency with CI/CD, GitHub (pull requests, branching, code reviews)
- Experience writing unit tests and performing data validation
- Experience supporting production systems and deployments
- Strong analytical and troubleshooting skills
- Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards
- Experience with healthcare data integration projects
- Familiarity with REST APIs and FHIR server integrations
- Experience in Agile/Scrum development environments