Senior Machine Learning Engineer

Air develops an AI-native Enterprise Readiness platform for government agencies and industrial suppliers. The Senior Machine Learning Engineer will build and operate infrastructure for the full language-model lifecycle, including LLMOps, fine-tuning, evaluation, deployment, serving, observability, and continuous improvement. The role partners with AI engineers to create scalable, reproducible, and reliable systems for production AI and agentic applications.

Responsibilities:

• Design and build LLMOps infrastructure supporting the development, evaluation, deployment, and continuous improvement of production language models
• Build scalable training and fine-tuning infrastructure for commercial and open-weight language models
• Develop pipelines supporting supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and other post-training techniques
• Build infrastructure for distributed training and GPU-accelerated ML workloads
• Develop data pipelines for training, fine-tuning, evaluation, and synthetic data generation
• Build systems for dataset versioning, lineage, quality validation, transformation, and reproducible experimentation
• Develop experiment management infrastructure that enables engineers to compare models, datasets, hyperparameters, prompts, and training techniques
• Build automated evaluation pipelines that determine whether new models or model versions are ready for production deployment
• Design model registries, artifact management, versioning, and promotion workflows across development and production environments
• Build and operate scalable model-serving and inference infrastructure for open-weight and fine-tuned models
• Develop abstractions that allow product and AI engineering teams to use multiple models and inference providers without tightly coupling applications to a single model or vendor
• Build observability for model training and inference, including metrics, tracing, logging, resource utilization, model quality, latency, throughput, and cost
• Optimize training and inference workloads for GPU utilization, throughput, latency, reliability, and infrastructure cost
• Build automated workflows for model deployment, rollback, canarying, and production validation
• Investigate model and infrastructure failures across data pipelines, training jobs, inference services, distributed systems, and production environments
• Evaluate emerging models, training techniques, inference frameworks, and ML infrastructure and determine where they can improve our production systems
• Partner closely with AI engineers building agentic systems to provide the model, evaluation, and training infrastructure required to continuously improve those systems

Requirements:

• U.S. Citizenship is required
• 5+ years of experience building production machine learning systems, ML infrastructure, distributed systems, or similar technical systems
• Deep experience designing, building, and operating production ML infrastructure or ML platforms
• Experience building infrastructure for training, fine-tuning, evaluating, deploying, and monitoring large language models or other large-scale deep learning models
• Experience with LLM fine-tuning and post-training workflows, including techniques such as supervised fine-tuning, LoRA/QLoRA or other parameter-efficient approaches, and preference optimization
• Strong understanding of the modern LLM lifecycle, including data preparation, training, evaluation, model artifacts, deployment, inference, monitoring, and iteration
• Experience building reproducible ML pipelines involving dataset versioning, experiment tracking, model versioning, and automated evaluation
• Experience building and operating production GPU infrastructure across AWS, GCP, Azure, or dedicated GPU providers, including training and/or inference workloads
• Strong understanding of distributed systems and the challenges involved in running computationally intensive ML workloads at scale
• Strong programming experience in Python and experience building production-quality software
• Deep experience with containers, Kubernetes, and cloud platforms such as AWS, GCP, or Azure
• Experience designing scalable APIs, services, asynchronous workloads, and data-processing pipelines
• Strong understanding of observability and operational reliability for production ML systems
• Comfortable debugging failures across training code, datasets, models, GPUs, distributed systems, and cloud infrastructure
• Able to move between ML experimentation and infrastructure engineering, understanding the needs of researchers and AI engineers while building systems that make those workflows scalable and reproducible
• Comfortable working in a rapidly evolving field where tooling, models, and best practices change quickly
• Current possession of a U.S. security clearance, or the ability to obtain one with our sponsorship
• Experience in or exposure to the nuances of a startup or other entrepreneurial environment
• Experience building secure code execution environments or sandboxes for AI agents
• Experience with multi-agent architectures, agent-to-agent communication, or distributed agent execution
• Experience with fine-tuning, post-training, reinforcement learning, or synthetic data generation
• Experience building AI observability, tracing, and debugging infrastructure
• Experience optimizing inference latency, throughput, GPU utilization, or model-serving costs
• Experience with AI security, adversarial testing, or securing agentic systems
• Experience working in government, defense, or other mission-critical environments