Grafana Labs is a remote-first, open-source powerhouse with over 20M users globally. They are seeking a Senior Data Engineer to maintain frameworks and systems for data management and to lead the development of machine learning pipelines for internal predictive models.
Responsibilities:
- Build and maintain production quality data pipelines between operational systems and BigQuery (ingress and egress)
- Implement data quality and freshness checks and monitoring processes to ensure data accuracy and consistency
- Maintain and contribute to our ingestion framework that leverages various purpose-built data load tool (dlt) connectors
- Create and maintain comprehensive documentation for data engineering processes, systems, and workflows
- Maintain observability and monitoring of our internal data pipelines
- Troubleshoot and resolve data pipeline issues to ensure downstream data availability
- Contribute to our dbt systems by making sure the source and staging layers align with our standards, are efficient, cost-effective, and highly available
- Participate in the investigation and implementation of event-driven data movement and transformation processes
- Participate in the investigation and implementation of analytic data storage/table formats (e.g. Apache Iceberg)
Requirements:
- Software development skills (some combination of Python, Java, Scala, Go)
- High proficiency in SQL
- Experience building and maintaining data ingestion pipelines using a workflow orchestration system (e.g. Prefect, Dagster, Airflow)
- Working knowledge of dbt or similar data transformation tools
- Highly motivated self-starter that is keen to make an impact and is unafraid of tackling large, complicated problems
- Excellent communication skills, able to explain technical topics to non-technical audiences, and maintain many of the essential cross-team and cross-functional relationships necessary for the team's success
- Experience working with Prefect, BigQuery, and GCP services
- Knowledge of event driven architectures
- Knowledge about observability
- Previous experience with Grafana visualization, or a desire to invest the time to learn