Site Reliability Engineer

Arsenal AI is hiring a Site Reliability Engineering team on behalf of an unnamed healthcare technology client. The team will build and operate production reliability practices, including monitoring, incident response, availability reporting, and resilience testing, while automating operational work across cloud and platform systems.

Responsibilities:

• Own the reliability of monitoring and alerting: make sure alerts fire, reach the right team, and are themselves monitored
• Define and maintain SLIs, SLOs, and availability targets, and report uptime and incident metrics (such as MTTD, MTTR, paging volume, and postmortem action-item closure)
• Build and run the incident response practice: severity definitions, the incident commander role, tabletop exercises, postmortems, and tracking action items to closure. Work with Security on decisions about external and regulatory reporting
• Improve alert quality as an ongoing responsibility: broaden error coverage, tune and de-duplicate noisy alerts, and make sure every paging alert has a runbook
• Build dashboards and detection for system health and saturation across databases, workflow engines, message queues, and logging infrastructure
• Plan, run, time, and document backup/restore, failover, and maintenance-window procedures to meet audited recovery objectives
• Improve deployment safety with post-deploy health checks, gates after failed deployments, and automated rollback
• Write code that reduces operational work, such as internal tools for secret rotation, platform upgrades, and recurring compliance tasks, in Go, TypeScript, and Pulumi
• Turn recurring operational obligations into scheduled, visible practices that keep running during a busy week
• Work with application teams, product operations, and Security to agree who owns alerting, triage, and escalation, including the security incident triage tier
• Write and maintain runbooks, incident response plans, and disaster recovery documentation
• Participate in code reviews, design sessions, all sprint ceremonies (standup, sprint review, retrospective, grooming), and an on-call rotation
• Senior and manager roles: provide technical leadership through design reviews, written trade-off analyses, and constructive peer reviews; serve as an escalation point during major incidents; and coach other engineers, including onboarding new people into on-call

Requirements:

• Site Reliability Engineering, production operations, DevOps, or Platform Engineering experience, including on-call rotations (AWS preferred): mid-level 2–5 years; senior 5+ years; engineering manager 5+ years, including time leading or mentoring engineers
• Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, or a related field (or equivalent practical experience)
• Strong software development skills, with hands-on experience in Go and/or TypeScript (the client's primary languages), enough to build and maintain operational tooling and automation
• Experience with scripting languages such as Bash and/or Python
• Hands-on experience with observability and alerting stacks such as Grafana/LGTM (Mimir, Loki, Tempo), Prometheus/PromQL, or Splunk, including designing alerts people can act on and reducing alert noise
• Experience with incident management, such as serving as or supporting an incident commander, applying a severity model, and running blameless postmortems whose action items are tracked to completion
• Experience defining SLIs and SLOs and reporting availability to internal and external stakeholders
• Hands-on experience with core AWS services (ECS/Fargate, RDS, EBS, CloudWatch, IAM, Lambda, SSM/Secrets Manager)
• Experience operating Kubernetes workloads, ideally AWS EKS, including stateful services (StatefulSets, persistent volumes)
• Experience with Infrastructure as Code, ideally Pulumi (Terraform, CDK, or CloudFormation also acceptable)
• Linux administration and troubleshooting experience
• Experience with version control, especially Git and GitHub
• Strong verbal and written communication skills
• U.S. citizenship or U.S. permanent residency
• Experience with ServiceNow ITSM/CSM (major incident and outage records, assignment group rules)
• Experience planning and running backup/restore, disaster recovery, and failover tests against defined RTO/RPO targets
• Experience with synthetic monitoring, uptime monitoring, and public status pages
• Experience with anomaly detection or statistical alerting, and capacity/saturation monitoring
• Experience with deployment safety practices: automated rollback, post-deploy verification, and deployment gates (Jenkins experience a plus)
• Experience building an on-call program: rotations, shadowing, onboarding, and runbook coverage
• Experience automating secret rotation and other recurring operational procedures with least-privilege access
• Experience working in a FedRAMP and/or HIPAA-compliant environment, including contingency planning and incident reporting requirements
• AWS certifications (SysOps Administrator, DevOps Engineer, Solutions Architect, etc.)
• Experience working in an Agile development environment