Key skills
AWSAzureCloudGoogle Cloud PlatformJavaPythonScalaSQLAIPineconeWeaviateData EngineeringBIPower BISnowflakedbtGCPGoogle CloudCI/CDCommunication
About this role
Role Overview
- Architect and own the AI context platform
- Design end-to-end platform architecture: ingestion → parsing/chunking → enrichment → embeddings → vector indexing → retrieval/serving
- Define scalable patterns for incremental refresh, backfills, re-embeddings, deduplication, and lineage across unstructured sources
- Set technical direction for retrieval quality (query strategies, hybrid search, metadata filtering, reranking) in partnership with AI engineers
- Evaluate and select infrastructure, tooling, and cloud services to support platform needs across AWS/Azure/GCP environments
- Design and deliver semantic and governed data products
- Architect and implement semantic layers (metrics/entities) that power BI and agent reasoning consistently across the platform
- Define data contracts and context contracts for AI inputs (schemas, metadata requirements, freshness, citation expectations)
- Establish standards for discoverability, documentation, and reusability across datasets and indexes
- Own the dbt or semantic layer tooling strategy and ensure consistent application across workstreams
- Own reliability and performance at the platform level: monitoring, alerting, SLAs/SLOs, runbooks, incident response, and postmortems
- Drive cost and latency optimization across Snowflake, lakehouse, and vector infrastructure
- Set engineering standards for CI/CD, testing, and evaluation (retrieval eval sets, regression tests, online telemetry)
- Implement security-by-design: RBAC/ABAC patterns, PII redaction, retention controls, audit logging, and safe access pathways for agent tools
- Partner with Security/Legal/Compliance to define and enforce guardrails for AI access to enterprise knowledge
- Own governance patterns for sensitive data handling across the platform
- Facilitate architectural decisions across teams and functions, building alignment without direct authority
- Set best practices and mentor engineers via design reviews, code reviews, and documentation
Requirements
- 8–12+ years in data engineering, data architecture, or platform roles with significant hands-on delivery
- Expert SQL and strong Python (or Scala/Java); deep production engineering habits
- Hands-on Snowflake expertise including advanced data modeling, pipeline design, performance tuning, and operating at scale in production
- Proven experience designing cloud data architectures on AWS, Azure, or GCP — including storage, compute, orchestration, and networking considerations
- Hands-on experience with vector search and embeddings (pgvector /Pinecone/ Weaviate /OpenSearch/Elastic) and retrieval patterns (semantic retrieval, hybrid search, reranking)
- Experience with dbt or comparable semantic layer tooling in a production environment
- Demonstrated ability to lead cross-functional technical initiatives and drive alignment across teams
- Strong written and verbal communication skills — able to present architecture decisions to both technical and non-technical audiences.
Tech Stack
- AWS
- Azure
- Cloud
- Google Cloud Platform
- Java
- Python
- Scala
- SQL
Benefits
- medical, dental and vision coverage
- annual incentive compensation program
- 401(k) plan with generous employer match
- employee stock purchase plan
- generous Paid Time Off policy
- paid parental leave
- adoption assistance
- wellness programs