Senior Site Reliability Engineer

Chalice AI provides AI-powered data and analytics solutions for real-time decisioning in ad buying. The Senior Site Reliability Engineer will define and operate the architectural backbone of the company’s AI platform, designing scalable event-driven, multi-tenant ML infrastructure while establishing reliability, governance, observability, and operational standards. The role also includes mentoring engineers and collaborating across Engineering, Data Science, Machine Learning, and Product.

Responsibilities:

• Recommend architectural direction
• Reduce systemic complexity
• Introduce durable patterns
• Identify architectural risk early
• Retire services when necessary
• Define reliability standards across teams
• Shape how our AI platform is delivered to customers
• Build and operate the architectural backbone of Chalice's AI platform
• Mentor and elevate other engineers in infrastructure best practices, operational rigor, and architectural thinking
• Help establish a culture of reliability, ownership, and continuous improvement across the organization
• Design and operate a scalable, event-driven, multi-tenant ML infrastructure platform that supports:
• Distributed ML training (Databricks, Ray, Flyte on EKS)
• Containerized product delivery to external customers
• Internal event-driven services across AWS
• Centralized state-store-driven orchestration
• Governance across adtech integrations and third-party APIs
• Build and support Chalice's event-driven / API first platform and participate the build-out of a scalable SRE function to support it
• Design and implement AWS event-driven systems using:
• Architect centralized state stores (DynamoDB, Redis, Postgres) that:
• React to signals
• Trigger downstream services
• Maintain system integrity
• Establish architectural standards for:
• Idempotency
• Replay safety
• Event schema governance
• Operational clarity and traceability
• Operate multi-cluster Kubernetes environments in production
• Understand and tune: API server scaling, etcd performance, RBAC architecture, admission controllers
• Implement: GitOps patterns, progressive delivery, cluster-level security policies, multi-tenant isolation
• Participate in Terraform module standards and create reusable infrastructure primitives
• Enforce GitHub guardrails (branch protections, CI gates)
• Evolve and standardize CI/CD pipelines to support automated infrastructure testing, policy validation, progressive deployment, and rollback mechanisms
• Standardize federated identity management, Azure SSO, API authentication patterns, key management, and resource isolation
• Model packaging standards (OCI images)
• Secret injection patterns
• Network isolation models
• Telemetry back to Chalice
• Upgrade and compatibility strategy
• Runtime configuration contracts
• Define SLIs, SLOs, and error budgets
• Separate ML reliability from infrastructure reliability
• Implement distributed tracing
• Design golden signals for event pipeline health, data freshness, model serving reliability, and control plane stability
• Define on-call structure, escalation paths, incident response standards, and postmortem processes
• SSO implementation and maintenance
• Authentication and provisioning
• Terraform-based deployments
• Cluster policies
• Unity Catalog governance

Requirements:

• 8-10 years of experience designing and operating production infrastructure at scale
• Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience
• Deep AWS architecture expertise across networking, SSO, compute, storage, and event-driven services
• Experience managing Kubernetes control planes in production environments
• Experience building or migrating event-driven systems at scale
• Experience in ML-heavy or data-heavy environments
• Has replaced or eliminated legacy infrastructure components and simplified system design
• Has owned real production failures and led postmortems that resulted in systemic improvements
• Strong architectural judgment and the ability to challenge assumptions constructively
• Has refactored/replaced legacy architectures in the past
• Has migrated systems safely
• Has designed something from zero
• Challenges leadership constructively
• Certifications (AWS, Databricks, Kubernetes, etc.) are a plus but not required