LLM DevOps/Inference Engineer

ShiftCode Analytics, Inc. is seeking a LLM DevOps/Inference Engineer to support healthcare-focused AI benchmark and evaluation suite development. The role involves building and maintaining AWS infrastructure, provisioning GPU clusters, and ensuring the reliability of the platform for various clinical prediction tasks.

Responsibilities:

• Build and maintain the AWS infrastructure for the evaluation platform as code, including networking, compute, orchestration, secrets, observability, and CI/CD
• Stand up the self-hosted inference track for the long tail of vertical healthcare models. This involves provisioning infrastructure for models serving on GPU compute with sensible batching, quantization where appropriate, autoscaling, and a standard onboarding path so adding new models takes hours, not weeks
• Build the provider abstraction layer alongside the AI engineers so that API-based models (OpenAI, Anthropic, Gemini, and the growing list beyond) and self-hosted models present a uniform interface to the harness. Rate limiting, retry and backoff, quota management, request/response logging, and cost attribution per run are your responsibility
• Make benchmark runs reproducible and cost-optimized with pinned model and container versions, captured configuration, spot and reserved capacity strategy, and idle GPU elimination
• Build the observability story with throughput, latency, token and GPU-hour cost, failure taxonomy, and per-model dashboards for monitoring
• Support the surge model that the platform must let a burst of AI engineers land, run experiments, and leave without breaking anything or leaving orphaned resources behind
• Contribute to Trusted Execution Environment (TEE) architecture. Evaluate AWS Nitro Enclaves and comparable confidential computing approaches for the bring-your-own-data / bring-your-own-model scenario, including attestation-gated key release and the practical constraints of running model inference inside an enclave

Requirements:

• AWS infrastructure at production scale: EKS or ECS, EC2 GPU instance families (G5/G6, P4d/P5) and their capacity realities, VPC design, IAM, KMS, Secrets Manager, ECR, CloudWatch, and Service Quotas
• Infrastructure as code: Terraform. No console-clicked production resources
• Model serving and inference optimization: Hands-on experience working with LLMs. Practical command of batching strategy, KV cache behavior, quantization tradeoffs, and multi-GPU sharding
• Container orchestration and GPU scheduling: ECS/EKS with GPU workloads, node autoscaling, and image build pipelines for CUDA-dependent stacks
• Reliability and cost engineering. SLOs, alerting, and a demonstrated track record of optimizing cloud spend without cutting capability
• AWS SageMaker endpoints and Bedrock
• Hands-on experience with Python to contribute directly to the harness and the provider adapter layer
• Confidential computing fundamentals: Enclaves, remote attestation, sealed key release, and the security boundaries of TEEs
• Healthcare compliance posture: HIPAA-eligible service selection, BAA scope, audit logging, and the access-control mechanisms for PHI data
• Security hardening, including image scanning and network egress control for a closed-loop environment