Staff Software Engineer - Grafana Databases, Managed Services | USA | Remote

Grafana Labs is a remote-first, open-source powerhouse with over 20 million users globally. They are seeking a Staff Software Engineer to operate and evolve shared database infrastructure for their Managed Services team, focusing on high-throughput, multi-cloud streaming clusters and related systems.

Responsibilities:

• Operating and evolving 100+ multi-cloud streaming clusters and related database infrastructure
• Diagnosing and eliminating cross-layer failure modes (e.g., object storage latency, noisy neighbors, control-plane bottlenecks, query performance regressions, etc.)
• Designing safe upgrade and rollout strategies at scale
• Improving observability, automation, and operational ergonomics
• Partnering closely with database and platform teams to ensure safe scaling, partitioning, consumer fan-out, and query performance
• Working directly with distributed systems behavior, Kubernetes scheduling dynamics, storage engines, compression trade-offs, etc
• Serving as a primary escalation point and on-call for relevant incidents
• Owning the relationship with all system vendors, including WarpStream Labs and others
• Help define and evolve the technical direction for operating WarpStream and adjacent shared database systems at scale
• Lead complex initiatives such as migrations, rollout improvements, and reliability investments
• Establish best practices around SLOs, scaling limits, failure isolation, and change safety
• Investigate and drive resolution of multi-layer incidents spanning storage, compute, networking, and control-plane dependencies
• Identify systemic risks across 100+ clusters and contribute architectural improvements that reduce recurring issues
• Improve systems toil and operational ergonomics with automation
• Partner with database and platform teams to align on strategy and long-term scalability
• Mentor and support engineers as the team matures

Requirements:

• 8+ years of engineering experience, including meaningful time in SRE, platform engineering, production engineering, infrastructure engineering, or distributed systems roles
• Experience with high-throughput streaming systems, analytical or storage backends, or large-scale database infrastructure. Examples of these include Kafka, Redpanda, WarpStream, Postgres, ClickHouse, Snowflake, or Cassandra
• Strong Kubernetes experience in AWS, GCP, or Azure, and familiarity with infrastructure-as-code tooling (Helm, Terraform, Jsonnet, etc.)
• Experience leading or driving complex technical efforts, even without formal management responsibilities
• Ability to influence technical direction and align teams around reliability improvements
• Strong understanding of distributed systems failure modes in multi-cloud environments
• Proficiency in at least one systems-oriented language (Go preferred, but not required)
• Working knowledge of Linux internals, networking, cloud storage, and performance/scaling behavior
• Experience participating in blameless incident response and writing high-quality post-incident reviews
• Clear communicator who can collaborate across teams and work autonomously
• Intellectually curious, transparent, action-oriented, and kind (this is important!)