Principal Site Reliability Engineer (GCP)

KPG99 INC is seeking a Principal Site Reliability Engineer to define and drive reliability strategy for services and platforms on Google Cloud Platform (GCP). The role involves hands-on technical leadership, mentoring engineers, and establishing reliability standards while collaborating with various teams to enhance operational excellence.

Responsibilities:

• Lead the GCP reliability reference architecture: compute (GKE / GCE), networking, IAM, storage, and multi-project / multi-environment patterns that Development and SRE teams can reuse safely
• Define and drive SLIs, SLOs, and error budgets for critical platform and application services; make reliability trade-offs visible to engineering leadership
• Establish observability standards (metrics, logs, traces)—Grafana LGTM (Mimir, Loki, Tempo) and/or GCP-native monitoring, OpenTelemetry / OTLP collectors, actionable alerting, and dashboards that SRE and Dev teams trust
• Lead incident management maturity: severity models, runbooks, Incident Commander practice, blameless postmortems, and systemic follow-through that permanently reduces repeat failures
• Set platform standards via RFCs and reusable modules: secure-by-default IAM, network boundaries, secret handling, capacity and cost guardrails, and production readiness checklists for Dev teams
• Mentor senior and mid-level SREs and DevOps engineers; run design reviews and raise the reliability bar across Cloud Engineering
• Design, build, and operate production infrastructure on GCP using Terraform (preferred) and configuration management with Ansible
• Package and run workloads with Docker; operate container platforms on GKE (and related GCP compute) with clear capacity, autoscaling, and upgrade practices
• Build and improve GitLab CI/CD pipelines: build, test, security gates, deploy, and rollback paths used by SRE, DevOps, and Development teams
• Automate SRE management workflows: environment provisioning, drift detection/remediation, health checks, deployment gates, self-service platform tooling, and toil elimination
• Partner with Development teams on production readiness: instrumentation, dependency failure modes, load/performance expectations, and operability reviews before major releases
• Drive disaster recovery / resilience: backup/restore patterns, multi-zone (and multi-region where required) designs, game days / failure testing, and documented RTO/RPO targets
• Collaborate with security and platform partners on least-privilege IAM, network segmentation, image/supply-chain hygiene, and auditability of infrastructure changes
• Partner with SRE, DevOps, and Development leadership to shift from reactive firefighting to proactive reliability and safer continuous delivery
• Represent Cloud Engineering in architecture forums; communicate reliability risk, cost, and operational trade-offs clearly
• Enable other engineers: golden-path templates, paved-road CI/CD, and documentation that shortens time-to-production without sacrificing safety

Requirements:

• 10+ years in SRE, DevOps, platform, or infrastructure engineering—or 8+ years with deep technical leadership of large-scale production cloud platforms—including clear Principal / Staff-level impact (architecture others build on, mentorship of senior engineers)
• Expert-level production experience with Google Cloud Platform (GKE and/or GCE, IAM, VPC/networking, Cloud Storage, load balancing, and related core services)
• Strong Infrastructure as Code practice with Terraform and configuration management with Ansible (or equivalent at similar depth, with clear ability to adopt Ansible)
• Hands-on ownership of GitLab CI/CD (pipelines, runners, environments, protected branches) and Docker-based delivery, plus IaC standards with Terraform and Ansible
• Proven application of SRE principles: SLIs/SLOs, error budgets, incident response, postmortems, and measurable toil reduction
• Experience with modern observability at scale (Prometheus/Grafana stack, OpenTelemetry, or equivalent cloud-native monitoring)
• Strong software / automation skills in Python and/or Go (Bash as supporting), plus solid Linux systems and networking fundamentals
• Ability to write clear design docs / RFCs and influence SRE, DevOps, and Dev teams without formal authority
• Professional-level GCP certifications (e.g. Professional Cloud Architect, Professional Cloud DevOps Engineer) or equivalent demonstrated depth
• Large-scale GKE platform engineering (multi-cluster, multi-tenant patterns, upgrade/rollout strategy, cost optimization)
• GitOps / progressive delivery (Argo CD, Flux, canary/blue-green) layered on GitLab CI
• Event-driven platforms (Kafka, Pub/Sub) as shared reliability concerns for producers/consumers
• Service mesh (Istio / Envoy) or advanced traffic management on GKE
• Chaos engineering, capacity forecasting, and FinOps / GCP cost governance
• Telecom, cable, ISP, or other high-availability, high-scale service environments
• Familiarity with Vault or GCP Secret Manager patterns, Binary Authorization / supply-chain controls, and policy-as-code