Lead and refine the incident lifecycle: detection, triage, communication, mitigation, resolution, and post-incident review.
Define and maintain severity models, escalation paths, on-call expectations, and runbooks/playbooks—keeping them current and usable under pressure.
Facilitate blameless postmortems; turn findings into tracked remediations and shared learning that reduces repeat incidents.
Improve coordination during major incidents: roles, tooling, customer/stakeholder updates, and handoffs.
Partner with security, support, and product on incident communications and regulatory or contractual obligations where applicable.
Establish and maintain organization-wide standards for metrics, logs, and traces in Datadog—including naming conventions, cardinality, retention, and sampling—so teams can instrument consistently and confidently.
Define and drive adoption of SLOs, SLIs, and error budgets across engineering teams; meet teams where they are—bootstrapping SLI/SLO programs for teams starting from scratch and improving rigor for teams that already have them, with the long-term goal of teams owning their own observability.
Build and maintain reusable Datadog dashboard templates, monitor templates, and alerting patterns that teams can adopt and adapt—reducing the activation energy for doing observability well.
Champion golden signals and RED/USE-style alerting philosophies; align alerts with user-impacting symptoms, not just low-level infrastructure noise.
Partner with the Infrastructure team on observability stack decisions, multi-tenancy, cost controls, and data lifecycle.
Continuously reduce alert noise through threshold tuning, ownership assignment, and on-call load management.
Mentor engineers on operational excellence, safe deployment practices, and production readiness; help engineering teams grow their own reliability instincts.
Contribute to capacity planning, chaos/game-day exercises, and reliability reviews for critical changes.
Serve as a connective layer between the SRE and Infrastructure teams—aligning on tooling, standards, and shared goals.
Requirements
Experience: 5+ years in SRE, production engineering, or equivalent, including on-call responsibility for customer-facing systems.
Incidents: Proven experience running or significantly improving incident response (process, tooling, or both) in a distributed systems environment.
Observability: Deep, hands-on experience with Datadog—building dashboards, monitors, and instrumentation standards across multiple teams or services. Experience with metrics, logging, and tracing at scale.
SLI/SLO Programs: Demonstrated experience defining SLOs/SLIs and error budget policies in production; comfortable working with teams to codify the metrics their reliability posture is based on.
Systems: Strong understanding of Linux, networking, distributed systems failure modes, and cloud or hybrid infrastructure (Kubernetes, load balancers, databases, queues).
Automation: Proficiency in at least one of Go, Python, or similar for tooling and automation; comfort with IaC concepts (Terraform or equivalent).
Communication: Clear written and verbal communication; ability to facilitate discussions during high-pressure incidents and deliberate postmortems alike.
Collaboration: Track record of influencing without direct authority and driving adoption across engineering teams.
Tech Stack
Cloud
Distributed Systems
Kubernetes
Linux
Python
Terraform
Go
Benefits
Competitive salary
Hybrid work environment (3 days in office per week)
100% individual and dependent medical + dental + vision coverage
401(K) with a 4% company match
20 days PTO
Iru Wellness Week the first week in July
Equity for full-time employees
In-office lunch stipend provided
Up to 16 weeks of paid leave for new parents
Paid Family and Medical Leave
Modern Health mental health benefits for individuals and dependents