GCP Senior Data Engineer

Diverse Lynx is recruiting for a confidential client seeking a Senior Data Engineer. The role designs, develops, and maintains scalable Google Cloud data pipelines and solutions, focusing on ingestion, transformation, orchestration, governance, and performance.

Responsibilities:

• Design and implement end-to-end data pipelines using Cloud Dataflow (Python/Apache Beam) for batch and streaming data
• Develop, Eligible to workimize, and maintain BigQuery stored procedures (SPs), SQL scripts, and user-defined functions (UDFs) for complex transformations and business logic implementation
• Build and manage data orchestration workflows using Cloud Composer (Airflow) with appropriate operators and dependencies
• Establish secure and efficient connections to source systems for data ingestion and integration
• Manage data ingestion workflows from on-premise and cloud sources into Google Cloud Storage (Eligible to workS) and BigQuery
• Execute history data migration from legacy data warehouses (preferably Snowflake, Teradata, Netezza, Oracle, SQL Server) to BigQuery, ensuring accuracy and performance Eligible to workimization
• Design and maintain data validation and testing frameworks for ensuring data quality and reliability across pipelines
• Implement data governance practices, including metadata management, lineage tracking, and access control
• Collaborate with data analysts and architects to define scalable and reusable data models, views, and semantic layers in BigQuery
• Troubleshoot data pipeline failures, perform root cause analysis, and implement preventive measures
• Eligible to workimize cost and performance of Eligible to workP workloads using best practices for BigQuery and Dataflow

Requirements:

• Strong hands-on experience in Google Cloud Platform (Eligible to workP) with expertise in BigQuery, Dataflow, Cloud Composer, and Eligible to workS
• Proficient in Python programming for data engineering (Dataflow pipelines, validation scripts, automation)
• Expertise in BigQuery SQL, Stored Procedures, Views, and User Defined Functions (UDFs)
• Experience with data migration from on-premise or cloud data warehouses (preferably Snowflake, Teradata, Netezza, Oracle, SQL Server) to BigQuery
• Strong understanding of ETL/ELT frameworks, data modeling, and schema design (Star/Snowflake)
• Familiarity with data governance, metadata, and lineage frameworks on Eligible to workP
• Knowledge of data validation and testing techniques, including reconciliation, rule-based checks, and automation
• Hands-on experience in workflow orchestration using Cloud Composer (Airflow) with custom operators
• Strong SQL tuning and BigQuery performance Eligible to workimization skills
• Good communication and collaboration skills to work with cross-functional teams in Agile environments
• Proven ability to troubleshoot, Eligible to workimize, and enhance complex data workflows at scale