Key skills
BigQueryGoogle Cloud PlatformPySparkPythonScalaSparkSQLUnityData EngineeringDatabricksGCPGoogle Cloud
About this role
Role Overview
- Design and oversee the migration/optimization of our data lakehouse architecture using Databricks.
- Serve as the escalation point for the platform, leveraging a proven record of handling production issues to ensure 24/7 reliability of mission-critical pipelines.
- Work directly with the SVP of Tech to translate product roadmaps into technical requirements.
- Drive the adoption of Databricks Asset Bundles (DABs) to standardize deployments across dev, staging, and production.
- Provide high-level guidance to engineering squads in India and the US, defining schemas and governance models for BigQuery and Databricks.
Requirements
- Expert-Level Databricks: Minimum 4+ years of hands-on experience specifically within the Databricks ecosystem (Delta Lake, Unity Catalog, Photon).
- The Stack: Deep proficiency in Spark (PySpark/Scala), Python, and SQL. Experience with GCP (BigQuery) is a major plus.
- Scale: Proven experience managing petabyte-scale datasets and high-concurrency pipelines (AdTech experience preferred).
- Independent Execution: Must be a self-starter capable of taking a high-level concept from the SVP and driving it to completion with minimal supervision.
- Experience: 10+ years in Data Engineering, with at least 3 years in a Principal or Architect capacity.
Tech Stack
- BigQuery
- Google Cloud Platform
- PySpark
- Python
- Scala
- Spark
- SQL
- Unity
Benefits
- participation in a retirement plan
- quarterly and annual incentive plan bonuses
- life and disability insurance
- health, dental and vision plans
- flexible spending accounts
- 15 paid vacation days
- 2 paid personal days
- 9 paid holidays
- 40 hours of paid sick leave
- parental leave
- employee stock purchase plan