Senior MLOps Engineer

Role Overview

• Design and operate ML platforms that support end-to-end workflows: data ingestion, feature engineering, training, evaluation, deployment, and monitoring.
• Build and maintain CI/CD for ML (testing, packaging, versioning, reproducibility, automated rollbacks, approvals).
• Implement MLOps best practices: model registry, experiment tracking, lineage, governance, and reproducible training environments.
• Develop scalable training infrastructure (distributed training, GPU scheduling, cost controls, auto-scaling).
• Create and maintain feature pipelines / feature stores, ensuring consistency between training and inference (training-serving skew prevention).
• Establish model monitoring and observability: performance, drift, bias/fairness signals (where relevant), latency, throughput, and data quality.
• Build and own end-to-end LLM delivery pipelines: prompt/versioning, retrieval, orchestration, evaluation, deployment, monitoring, and iterative improvement.
• Create robust LLM evaluation harnesses (offline + online): golden datasets, automated regression testing, human-in-the-loop review workflows, and risk scoring.
• Build cost controls: token/cost budgeting, caching strategies, autoscaling, and performance tuning.
• Productionize ML Models on GCP using containers and orchestration (e.g., GKE, Cloud Run), and build CI/CD for ML/LLM systems with automated tests and safe rollouts.
• Implement observability: tracing, metrics, logs, dashboards, alerting for model/system health (latency, token usage, error rates, retrieval quality, hallucination indicators, drift where relevant).
• Design systems with security and privacy by default: IAM, least privilege, secrets management, audit logs, encryption, data retention, and PHI/PII handling.
• Implement governance: model/prompt lineage, dataset provenance, evaluation traceability, and approval workflows aligned with healthcare compliance expectations.
• Integrate guardrails: content filters, policy checks, prompt injection defenses, structured output validation, and fallback strategies.

Requirements

• 6+ years in software/platform engineering, including 4+ years operating ML systems in production (or equivalent depth).
• Strong experience in ML engineering: training pipelines, evaluation, deployment patterns, monitoring, and iteration loops.
• Strong engineering skills in Python, plus production-grade experience building APIs/services.
• Demonstrated hands-on experience with LLM systems in production and ML engineering: training pipelines, evaluation, deployment patterns, monitoring, and iteration loops.
• Strong experience with GCP services and cloud-native patterns.
• Experience with Vertex AI (pipelines, endpoints, feature store, model registry, evaluation) and/or managed vector search on GCP.
• Experience with containerization and orchestration (Docker, Kubernetes/GKE and/or Cloud Run).

Tech Stack

• Cloud
• Docker
• Google Cloud Platform
• Kubernetes
• Python

Benefits

• Competitive salary and benefits package.
• Flexible working arrangements (remote or hybrid options available).
• The opportunity to work on life-changing AI technology that directly impacts patient outcomes.
• Join a team that combines cutting-edge innovation with a mission to save lives and improve health equity.
• Continuous learning opportunities with access to the latest tools and advancements in AI and healthcare.