Instacart is transforming the grocery industry by providing essential services for grocery delivery. The Senior Software Engineer in Data Governance & Foundations will define architecture roadmaps, own platform initiatives, and collaborate with various teams to optimize data infrastructure while ensuring governance and compliance.
Responsibilities:
- Define and drive multi-year architecture roadmaps for large-scale data ingestion and processing infrastructure, setting technical direction that balances reliability, scalability, and cost
- Own end-to-end platform initiatives — from build vs. buy decisions and migration design through production rollout and risk management — across Kafka-based streaming and Postgres-based systems
- Partner with vendors (Snowflake, Databricks, Confluent) on technical integration, contract evaluation, and TCO modeling to inform infrastructure investment decisions
- Collaborate with various teams to embed governance and compliance controls (SOX, CPRA, GDPR) directly into platform architecture and data lifecycle management
- Optimize infrastructure spend at scale: identify cost reduction opportunities across compute, storage, and pipeline efficiency; manage multi-million dollar infrastructure budgets
- Write compelling architecture documents, strategy memos, and proposals that drive alignment with engineering leadership and senior stakeholders across the organization
- Mentor engineers on the team, model strong engineering culture, and help grow a high-performing data infrastructure organization
- Collaborate with Data Science, ML Platform, Ads Infrastructure, Finance Engineering, and Product teams to ensure the platform meets evolving needs
Requirements:
- 5+ years of software engineering focused on data infrastructure or distributed systems at scale, in a high-growth, data-intensive environment
- Experience in modern data lakehouse architectures and open table formats — Apache Iceberg, Delta Lake, Hudi — with strong understanding of compute/storage trade-offs
- Hands-on experience with distributed query and compute systems (Trino, Spark, ClickHouse) including performance tuning and production reliability work
- Proven depth in event-driven infrastructure: Kafka for high-throughput data ingestion and Flink (or equivalent) for stream processing at scale
- Track record owning and executing major platform transitions, including migration design, phased rollout, and risk management under production constraints
- Experience building business cases for infrastructure investments: cost-benefit analysis, TCO modeling, and presenting recommendations to leadership
- Exceptional written technical communication — clear architecture docs, strategy memos, and cross-team proposals that drive decisions and alignment
- Strong ownership and comfort operating in ambiguity; ability to drive large, multi-team initiatives from concept to production with organizational influence
- Familiarity with data governance and compliance frameworks (SOX, CPRA, GDPR) and experience designing governance controls into platform architecture
- Experience with FinOps and data platform cost optimization, including managing large infrastructure budgets and negotiating enterprise vendor contracts
- Deep SQL expertise and strong proficiency in Python or Scala for systems-level work
- Experience with orchestration (Apache Airflow) and transformation pipelines (dbt) in large-scale production environments
- Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience