DigitalOceanRemote
Senior Engineer 2: Inference Data Plane
United States of America
Full Time
1 hour ago
$167,000 - $209,000 USD
H1B Sponsor Likely
Key skills
Distributed Systems ExpertiseAI/ML Domain KnowledgeInference FrameworksSoftware EngineeringCloud OperationsArchitecture ProficiencyOpen Source MindsetOperational ExcellenceMentorshipPythonGolangAIGenerative AILLMgRPCDigitalOcean
About this role
DigitalOcean is a cutting-edge technology company focused on simplifying cloud services and AI. They are seeking a Senior Engineer 2 to lead the AI Inference Data Plane team, responsible for designing and delivering high-scale data plane services for their 'Inference as a Service' offering.
Responsibilities:
- Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models
- Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards
- Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing
- Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs
- Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement
- Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health
Requirements:
- Strong experience with microservices, messaging systems, databases, and infrastructure as code
- Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or Modular
- Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve
- Understanding of GPU-level optimization and experience with interconnect technologies like NVlink, XGMI, or RoCE
- Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization)
- Expert-level proficiency in GoLang or Python and familiarity with gRPC
- Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean
- Experience integrating and building with open-source software