The UVA VEC is seeking advanced Grafana practitioners to contribute to the development of high-quality evaluation workflows for AI systems. The role involves designing and executing multi-step workflows across dashboards, alerting systems, and data sources, with a focus on clarity, accuracy, and reproducibility.
Responsibilities:
- Design multi-step Grafana workflows including dashboards, alerts, and data source configurations
- Execute workflows in a live Grafana environment to generate reference implementations
- Develop structured task prompts with clearly defined and verifiable outcomes
- Implement programmatic validation scripts to assess task completion accuracy
- Analyze system-generated outputs and identify points of failure in execution
- Refine task difficulty and structure based on observed performance patterns
- Document workflows and evaluation criteria with precision and clarity
Requirements:
- Minimum 2 years of hands-on Grafana experience in production environments
- Strong proficiency with PromQL and observability data modeling
- Experience configuring dashboards, alerting pipelines, and data sources (e.g., Prometheus, InfluxDB)
- Ability to translate technical workflows into structured, testable instructions
- Proficiency in Python for writing validation or grading scripts
- Strong analytical skills for identifying execution gaps and failure modes
- Ability to work independently in a remote, asynchronous environment
- Availability for consistent weekly contribution (10–15 hours minimum)
- Experience with Grafana API automation
- Background in Kubernetes or infrastructure monitoring
- Familiarity with AI evaluation, benchmarking, or testing systems