Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

EPAM Systems is a global technology and consulting company. The company is seeking a Lead Data Software Engineer to migrate an Azure-based data analytics platform to a cloud-agnostic, on-premises-deployable platform, while implementing Spark pipelines, overseeing deployments and testing, resolving bugs, and mentoring team members.

Responsibilities:

• Implement streaming and batch Spark pipelines running on Kubernetes
• Lead the implementation of objects within the data generator framework
• Oversee deployment and testing across local and development environments
• Drive improvement of the current, actively changing solution
• Conduct bug investigation and resolution
• Perform unit testing to ensure solution quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members throughout the implementation phase

Requirements:

• 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes
• Proficiency in Python
• Familiarity with the data engineering domain and cloud-agnostic platform migrations
• Capability to dive deeper into new technologies in a short time during active implementation phases
• Ability to work independently with syncs with a team lead
• Strong communication skills and proactiveness
• Showcase of being a hands-on team player and technical leader
• Proficiency in English at a B2+ level
• Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Skills in Ansible and Argo CD
• Understanding of TDD