Architecting and managing platforms where Spark runs, such as Kubernetes clusters or cloud services
Packaging Spark workloads and integrating with orchestration systems like Apache Flyte
Deploying Infrastructure via Terraform/Ansible
Troubleshooting job failures and execution anomalies
Optimizing Spark configurations to reduce costs
Building automation and tools in languages like Python, Java, or Scala
Writing medium to complex SQL Queries
Implementing monitoring systems for platform stability
Working closely with data engineers and scientists
Engaging with open-source communities to discuss technical challenges
Creating and maintaining documentation for Kubernetes infrastructure
Providing training and support to team members

Bachelor's degree in computer science or a related field, or equivalent experience
Expertise in Apache Spark
Proficiency in Python, Pyspark, and Scala/Java
Proficient in Linux Scripting (Bash)
Proficient in writing SQL
Experience in CI/CD tools, Github
Experience in using observability tools like Prometheus, Grafana
Knowledge of networking concepts like TCP/IP, DNS, Load Balancer
Automation via Terraform/Ansible
Hands-on experience with cloud providers (AWS, Azure, GCP)
Familiarity with technologies like Delta Lake, Apache Iceberg, Apache Kafka, Hadoop
Strong analytical skills for debugging complex issues
Strong communication and collaboration abilities

Platform Software Engineer, Kubernetes

Key skills