Key skills
AWSCloudDockerGrafanaKubernetesPrometheusPySparkPythonSparkSQLTerraformData EngineeringGitHub ActionsGitHub
About this role
Role Overview
- Design and maintain ingestion frameworks for high-volume data
- Develop transformation logic using SQL, Python, and Spark
- Collaborate with domain teams to annotate datasets
- Work within a modern lakehouse architecture
- Instrument your pipelines with quality checks
- Support deployment workflows via GitHub Actions and Terraform
- Help codify data engineering standards and create scaffolding
Requirements
- Strong working proficiency in Python (PySpark) and SQL
- Solid understanding of AWS services
- Working knowledge of optimizing pipelines for cost efficiency and resource utilization
- Genuine interest in platform/infrastructure work
- Conceptual understanding or hands-on experience with Docker and Kubernetes
- Experience with cloud migrations or multi-cloud environments
- Familiarity with IaC tooling (Terraform, CDK, etc.)
- Familiarity with tools like Grafana and Prometheus for monitoring and alerting
Tech Stack
- AWS
- Cloud
- Docker
- Grafana
- Kubernetes
- Prometheus
- PySpark
- Python
- Spark
- SQL
- Terraform
Benefits
- Health insurance
- Retirement plans