About this role

Job Summary:
We are seeking an experienced Senior Engineer, Analytics with a focus on Batching to join our team. As a Senior Engineer, you will play a key role in designing, building, and maintaining our batch processing infrastructure. You will work closely with our data scientists, product managers, and other engineers to develop and deploy scalable, efficient, and reliable data pipelines.

Responsibilities:

Design, develop, and maintain batch processing pipelines for large-scale data transformation and analysis using technologies like Apache Hadoop, Pig, Hive, Spark, or similar frameworks.
Experience with Airflow and/or Cloud Composer for workflow management and orchestration
Work with data scientists and product managers to develop and deploy data pipelines for data warehousing and data lake use cases
Collaborate with other engineers to integrate batch processing systems with our data streaming and real-time analytics infrastructure
Ensure data quality, data security, and data privacy across all batch processing systems
Develop and implement batch processing jobs using programming languages like Java or Python. (Experience with both is a plus)
Work with public cloud providers such as Google Cloud Platform (Google Cloud Platform), Amazon Web Services (AWS), or Microsoft Azure
Experience with Dataproc is a strong preference
Optimize batch pipelines for performance and resource efficiency.
Monitor and troubleshoot batch jobs to ensure successful execution and data quality.
Implement DevOps principles and practices to ensure efficient and reliable deployment of batch processing systems
Knowledge of containerization technologies such as Docker and Kubernetes

Requirements:

Bachelor's degree in Computer Science or a related field
5+ years of experience in software engineering, with a focus on batch processing and analytics
Strong programming skills in Java or Python
Experience with a public cloud provider, with a focus on Google Cloud Platform
Strong experience in batch processing systems such as Hadoop, Pig, Hive, and Spark
Experience with Dataproc is a plus
Knowledge of DevOps principles and practices
Understanding of containerization technologies such as Docker and Kubernetes
Excellent problem-solving skills, with the ability to work independently and collaboratively
Excellent communication skills, with the ability to explain complex technical concepts to non-technical stakeholders
Experience with machine learning and data science technologies is a plus
Certification in a public cloud provider or a relevant technology is a plus

Senior Batch Infrastructure Engineer

Key skills

About this role