Key skills
AirflowBigQueryDockerETLGoogle Cloud PlatformKafkaKubernetesPySparkPythonSQLTerraformBashdbtGCPGoogle CloudGitCI/CD
About this role
Role Overview
- Develop batch and real-time data pipelines on our GCP platform
- Build APIs to share data with other teams and develop CI/CD pipelines for automated deployments
- Develop Terraform IAC for deploying core infrastructure and use Datastream to ingest CDC data
- Use Dataform and dbt to build medallion data architecture while optimizing BigQuery and ClickHouse models
- Build Airflow DAG pipelines for data imports/exports with monitoring and alerting for data integrity
- Uphold GDPR/CCPA compliance and maintain data governance standards across platforms
Requirements
- Strong proficiency in Bash, Git, SQL and Python (essential skills)
- Hands-on experience with Terraform IAC, Docker and Kubernetes
- Solid GCP expertise including BigQuery, Dataform, Airflow, and Datastream
- Real-time processing experience with Kafka, PySpark, or Flink
- Proven ETL/CI-CD development skills and API development capabilities
- Bachelor's in Computer Science, Data Science, Information Systems, or equivalent experience
Tech Stack
- Airflow
- BigQuery
- Docker
- ETL
- Google Cloud Platform
- Kafka
- Kubernetes
- PySpark
- Python
- SQL
- Terraform
Benefits
- Flexible remote options
- Professional development opportunities
- A culture that celebrates diverse ideas and breakthrough thinking