Senior Software Engineer, Data Governance & Foundations

Instacart is transforming the grocery industry by providing essential services for grocery delivery. The Senior Software Engineer in Data Governance & Foundations will define architecture roadmaps, own platform initiatives, and collaborate with various teams to optimize data infrastructure while ensuring governance and compliance.

Responsibilities:

• Define and drive multi-year architecture roadmaps for large-scale data ingestion and processing infrastructure, setting technical direction that balances reliability, scalability, and cost
• Own end-to-end platform initiatives — from build vs. buy decisions and migration design through production rollout and risk management — across Kafka-based streaming and Postgres-based systems
• Partner with vendors (Snowflake, Databricks, Confluent) on technical integration, contract evaluation, and TCO modeling to inform infrastructure investment decisions
• Collaborate with various teams to embed governance and compliance controls (SOX, CPRA, GDPR) directly into platform architecture and data lifecycle management
• Optimize infrastructure spend at scale: identify cost reduction opportunities across compute, storage, and pipeline efficiency; manage multi-million dollar infrastructure budgets
• Write compelling architecture documents, strategy memos, and proposals that drive alignment with engineering leadership and senior stakeholders across the organization
• Mentor engineers on the team, model strong engineering culture, and help grow a high-performing data infrastructure organization
• Collaborate with Data Science, ML Platform, Ads Infrastructure, Finance Engineering, and Product teams to ensure the platform meets evolving needs

Requirements:

• 5+ years of software engineering focused on data infrastructure or distributed systems at scale, in a high-growth, data-intensive environment
• Experience in modern data lakehouse architectures and open table formats — Apache Iceberg, Delta Lake, Hudi — with strong understanding of compute/storage trade-offs
• Hands-on experience with distributed query and compute systems (Trino, Spark, ClickHouse) including performance tuning and production reliability work
• Proven depth in event-driven infrastructure: Kafka for high-throughput data ingestion and Flink (or equivalent) for stream processing at scale
• Track record owning and executing major platform transitions, including migration design, phased rollout, and risk management under production constraints
• Experience building business cases for infrastructure investments: cost-benefit analysis, TCO modeling, and presenting recommendations to leadership
• Exceptional written technical communication — clear architecture docs, strategy memos, and cross-team proposals that drive decisions and alignment
• Strong ownership and comfort operating in ambiguity; ability to drive large, multi-team initiatives from concept to production with organizational influence
• Familiarity with data governance and compliance frameworks (SOX, CPRA, GDPR) and experience designing governance controls into platform architecture
• Experience with FinOps and data platform cost optimization, including managing large infrastructure budgets and negotiating enterprise vendor contracts
• Deep SQL expertise and strong proficiency in Python or Scala for systems-level work
• Experience with orchestration (Apache Airflow) and transformation pipelines (dbt) in large-scale production environments
• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience