GCP Data Engineer

Tredence Inc. is seeking a highly skilled GCP Data Engineer with expertise in Scala, Apache Spark, and Google Cloud Platform (GCP). The role involves designing, developing, and optimizing large-scale data processing solutions, building cloud-native data pipelines, and collaborating with data scientists and stakeholders.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Scala and Apache Spark
• Build and optimize batch and real-time data processing solutions on Google Cloud Platform (GCP)
• Develop data ingestion, transformation, and orchestration workflows using GCP services
• Work with large datasets and implement efficient ETL/ELT processes
• Integrate data from various sources including databases, APIs, and streaming platforms
• Monitor, troubleshoot, and improve data pipeline performance and reliability
• Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions
• Implement data quality, governance, and security best practices
• Participate in code reviews and contribute to engineering standards and best practices
• Support production deployments and resolve data pipeline issues

Requirements:

• Strong hands-on experience with Scala programming
• Extensive experience with Apache Spark (Spark SQL, Spark Streaming, DataFrames, Datasets)
• Strong understanding of distributed computing concepts
• Experience working with Google Cloud Platform (GCP) services such as: BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, Composer (Airflow), Cloud Functions, Cloud Run
• Experience in ETL/ELT development and data warehousing concepts
• Proficiency in SQL and database optimization
• Experience with CI/CD pipelines and DevOps practices
• Hands-on experience with Git and version control systems
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field
• 4+ years of experience in Data Engineering, Big Data, or Cloud Data Platforms
• Relevant GCP certifications are a plus: Professional Data Engineer, Associate Cloud Engineer
• Strong analytical and problem-solving skills
• Excellent communication and stakeholder management skills
• Ability to work independently and collaboratively in a fast-paced environment
• Focus on performance optimization and continuous improvement
• Experience with Kafka or other streaming technologies
• Knowledge of Python in addition to Scala
• Familiarity with containerization technologies such as Docker and Kubernetes (GKE)
• Knowledge of data modeling and cloud data architecture
• Experience with Terraform or Infrastructure as Code (IaC)
• Experience with machine learning data pipelines
• Exposure to multi-cloud environments (AWS/Azure)
• Experience working in Agile/Scrum teams