Senior Data Engineer (Spark, Kafka & Google Cloud Platform)

Senior Data Engineer (Spark, Kafka & Google Cloud Platform)
Sunnyvale, CA (Onsite – 5 Days)

Contract
UST Global // Walmart

22857-1

Mandatory Areas
Must Have Skills – Data Engineer
Skill 1 – big data, spark, pyspark, apache airflow, etl process
Skill 2 – 4+ years in Google Cloud Platform

Job Summary

We are looking for a highly experienced Senior Data Engineer with strong expertise in Apache Spark, PySpark, Kafka, and Google Cloud Platform (Google Cloud Platform) to build scalable real-time and batch data processing solutions. The ideal candidate will have extensive experience designing and optimizing enterprise data pipelines for analytics and machine learning workloads.

Required Skills

• 10+ years of overall IT experience
• 4+ years of recent hands-on experience with Google Cloud Platform (Google Cloud Platform)
• Strong programming skills in Python and SQL
• Experience with Scala or Java
• Expertise in Apache Spark (Spark SQL, DataFrames, Spark Streaming)
• Hands-on experience with PySpark
• Strong knowledge of Apache Kafka or Google Pub/Sub
• Experience building ETL/ELT pipelines
• Experience with BigQuery, Data Lakes, and Data Warehouses
• Knowledge of Snowflake or Amazon Redshift
• Experience with Apache Airflow
• Good understanding of data quality, monitoring, and performance tuning
• Experience with Databricks, Docker, and Kubernetes is a plus

Responsibilities

• Design, develop, and maintain scalable batch and real-time ETL/ELT pipelines.
• Build streaming data pipelines using Spark and Kafka.
• Develop optimized data solutions for analytics and machine learning.
• Optimize Spark jobs, SQL queries, and overall data processing performance.
• Implement data quality checks, monitoring, and alerting mechanisms.
• Work with cloud-native data services on Google Cloud Platform.
• Collaborate with cross-functional teams to deliver scalable data engineering solutions.
• Participate in design reviews, code reviews, and production deployments.
• Troubleshoot and resolve production issues while ensuring high availability.

Preferred Qualifications

• Experience working in large enterprise environments.
• Strong understanding of Agile/Scrum methodologies.
• Excellent communication and problem-solving skills.
• Experience mentoring junior engineers is preferred.

Nice to Have

• Databricks
• Docker
• Kubernetes
• BigQuery
• Snowflake
• Redshift
• Google Pub/Sub