Principal Software Engineer - CoreAI Model Inference & Serving

Jobs via Dice is seeking a Principal Software Engineer to join their CoreAI team, which focuses on building the AI data-plane that powers LLM inferencing workloads for Microsoft and Azure customers. The role involves leading the design and implementation of core serving systems for large-scale AI services, driving innovation and ensuring high performance and reliability.

Responsibilities:

• Be a hands-on technical leader, designing, coding, and shipping core serving systems, smart routing, and request distribution for a broad portfolio of LLMs, including OpenAI, Mistral, Grok, DeepSeek, and others
• Build large-scale AI services and platform capabilities that power new products and customer experiences
• Drive cutting-edge innovation in AI systems alongside world-class engineers and cross-functional partners
• Lead through architecture, code reviews, mentorship, and technical excellence while staying close to implementation
• Improve reliability, scalability, observability, efficiency, and performance across mission-critical services

Requirements:

• Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, or Java OR equivalent experience
• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
• 4+ years of design and problem-solving experience, with understanding of system performance, scalability, and engineering best practices
• Understanding of distributed systems specifically in request serving at scale; (e.g. inferencing, L7 gateways, high-performance storage, distributed databases across global-scale infrastructure)
• Demonstrated experience in building high-quality, reliable systems at scale
• Experience using modern AI-assisted development tools and workflows to move faster, improve quality, and amplify engineering impact
• Customer-obsessed approach to problem solving, with empathy and a drive to deliver impactful solutions