Senior Software Engineer II - Ads Quality

Instacart is transforming the grocery industry by delivering essential grocery and household goods services while creating flexible earning opportunities for personal shoppers. The Senior Software Engineer II will lead the design and development of machine learning infrastructure for ads, optimizing training and serving systems, real-time feature processing, and foundation model capabilities. The role also involves setting technical direction, collaborating cross-functionally, mentoring engineers, and improving advertising performance and operational stability.

Responsibilities:

• Design, develop, and deploy machine learning infrastructure to tackle practical challenges in our complex marketplace while advancing the training and serving platform. Work with the company wide ML Foundation team to develop solutions to meet Ads unique needs
• Optimize training and serving funnel bottlenecks—profiling real systems and driving meaningful wins in throughput, cost, and latency, from GPU/accelerator utilization to tail latency at serving time
• Identify instabilities and shortcomings of our systems and operating procedures to deliver best in class availability
• Build real-time feature processing infrastructure—low-latency pipelines that compute, join, and serve fresh features to models on the critical path of every ad request
• Lead the infrastructure foundation for our real-time sequential foundation model, including online sequence feature stores, streaming update paths, efficient long-context serving, and training pipelines that keep pace with rapidly evolving data
• Collaborate closely with product managers, data scientists, and MLEs to deeply understand business needs and co-design systems where product, model architecture and infrastructure constraints are solved together
• Actively engage with diverse stakeholders to ensure that solutions are well-integrated and aligned with business goals
• Mentor engineers, set technical direction, and conduct thorough code reviews to ensure high code quality and maintainability. Lead by example in fostering a culture of continuous improvement and technical excellence

Requirements:

• Have a graduate degree (masters or PhD) in computer science, artificial intelligence, machine learning, operations research or equivalent self study and experience
• Have 3+ years of industry experience building ML infrastructure or using machine learning to solve real-world problems with large datasets
• Deep expertise building, scaling and managing ML infrastructure for training and/or serving in production, with a strong understanding of modern machine learning systems and MLOps
• Strong programming skills in software engineering (e.g., Python, Go), along with expertise in data manipulation using tools (SQL, Spark, and Pandas), and ML frameworks (Torch, Tensorflow, XGBoost)
• Strong analytical and problem-solving skills
• Excellent communication skills (both verbal and written) and can collaborate with a diverse group of stakeholders across all levels
• Eager to jump into domains, languages, and problem areas that might be new and unfamiliar
• Have 5+ years of industry experience tech leading a team to build ML infrastructure or using machine learning to solve real-world problems with large datasets
• Demonstrated experience solving complex model training and serving efficiency challenges, including optimizing training pipelines, reducing model training times, and improving serving architectures to enhance operational efficiency and scalability
• Hands-on experience with real-time / online feature processing and serving—streaming systems (e.g., Kafka, Flink), online feature stores, and low-latency retrieval on the request path
• Experience with sequential modeling, transformer architecture, and generative retrieval or building large scale online recommendation systems
• Experience serving large sequential or foundation models in low-latency, high-QPS production settings
• Experience in digital advertising platforms
• Familiarity with LLM integrations, prompt engineering, and productivity tooling