Databricks Architect

Role Overview

• Own the Azure Databricks landing zone architecture: subscription and resource group strategy, workspace hierarchy, VNet injection, Private Link/endpoints, firewall rules, and cross region DR/BCP approaches.
• Define and implement Lakehouse patterns (bronze/silver/gold) using Delta Lake, including CDC ingestion with Auto Loader, streaming with Structured Streaming and Delta Live Tables (DLT), and table optimization (OPTIMIZE/Z ORDER/VACUUM policies).
• Establish and govern Unity Catalog: metastore design, external locations, schema/catalog standards, lineage integration, access models (RBAC/ABAC), and data protection (row/column level security, masking, tags).
• Create platform governance standards: SCIM provisioning, SSO/SAML, secret management (Key Vault backed scopes), cluster and pool policies, IP access lists, workspace and job ACLs, repo standards, and promotion workflows across Dev/Test/Prod.
• Build CI/CD and release engineering for Databricks (Repos, Git integration, ADO/GitHub Actions, Terraform/Bicep/ARM), including environment promotion, approvals, and automated quality gates.
• Implement observability and reliability practices: platform health dashboards, UC audit log analytics, job/runtime metrics, event logs, incident response runbooks, SLO/SLI reporting, and capacity planning.
• Drive performance engineering: benchmark Spark workloads, Photon adoption, caching strategies, file sizing and partitioning, concurrency controls, and SQL warehouse tuning.
• Lead FinOps for Databricks: budgets and tagging strategy, cluster policy guardrails (autoscale, instance families, spot usage, idle timeouts), serverless/SQL cost controls, and unit economics reporting.
• Partner with Security & Compliance to meet enterprise and regulatory requirements (encryption at rest/in transit, key management, PII handling, audits, retention policies, vulnerability management).
• Integrate the platform with Azure and enterprise services: ADLS Gen2, Event Hubs/Kafka, AAD, Purview, Key Vault, Log Analytics, Tableau, Salesforce, Genesys, and Delta Sharing.
• Enable MLOps and analytics at scale: MLflow model registry standards, feature engineering conventions, real time/batch inference patterns, Workflows orchestration, and blue/green or canary deployment strategies.
• Coach, guide, and review solution designs from data engineers and analysts; conduct architecture/design reviews and code reviews; publish reference architectures, standards, and reusable templates.
• Develop and maintain platform documentation, onboarding materials, and runbooks; lead knowledge transfer and internal training.
• Lead design and rollout of Databricks Mosaic AI (Foundation Model APIs, Model Serving, Vector Search, and Agent Framework) for enterprise GenAI use cases (RAG over governed data, copilots, and agents).
• Define prompt/response patterns, evaluation frameworks, guardrails (safety filters, PII redaction), offline/online tests, latency/error SLOs, and cost controls.
• Build LLMOps on the Lakehouse: instrument MLflow/Unity Catalog–backed model and prompt registries; manage feature and embedding pipelines; stand up serverless model endpoints with blue/green deploys; implement continuous evaluation (A/B), drift detection, usage telemetry, and human feedback loops; integrate with Purview/UC lineage and incident response.

Requirements

• B.A./B.S. in Computer Science, Software/Systems Engineering, Data Engineering, Information Systems, or related field is required.
• Master’s degree in a related field, such as Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, Data Science or Data Engineering is preferred.
• 6-8+ years designing or operating cloud data platforms at enterprise scale.
• 3-4+ years hands on with Azure Databricks (or Databricks on cloud) including workspace administration, Unity Catalog, and Delta Lake.
• 3+ years building secure data pipelines (batch/streaming) with Spark, SQL, Python/Scala, and Delta; experience with Auto Loader and DLT.
• Proven experience in Databricks workspace and Lakehouse architecture.
• 3+ years implementing CI/CD and infrastructure as code (Terraform strongly preferred) for data platforms.
• Demonstrated success migrating or modernizing platforms (e.g., Hadoop/Synapse/SQL to Databricks Lakehouse) and establishing Dev/Test/Prod standards.
• Experience implementing lineage, cataloging, and governance (e.g., Purview/Unity Catalog) and integrating with enterprise SSO/SCIM.
• Experience in the insurance domain, particularly Auto, Home, and Umbrella.
• Databricks Certified Data Architecture Professional is preferred.
• Familiarity with agile software delivery methodologies such as Scrum is preferred.
• Familiarity with platforms and enabling tools such as Azure Machine Learning, Azure Databricks, Microsoft Fabric, Synapse Analytics, Power BI, Snowflake, and APIs like Azure OpenAI, Azure Cognitive Services, and Azure ML Endpoints is preferred.
• Detail oriented with strong organizational, analytical, and communication skills.
• Service minded with experience deciphering ambiguous requests and taking ownership.
• Ability to create and maintain positive relationships with employees at all levels of the organization.
• Experience in the insurance industry and understanding of Auto, Home, and Umbrella insurance-related AI/ML applications.
• Strong knowledge of real-time data streaming frameworks like Apache Kafka, Azure Event Hubs, and Delta Live Tables for Databricks.
• Ability to work independently and manage tasks with minimal supervision.
• Experience with unsupervised learning techniques to identify patterns and anomalies in data.
• Strong knowledge of cloud-based security protocols, compliance standards, and governance frameworks is required.
• Proficiency with AI/ML frameworks and embedding AI solutions into data ecosystems. Familiarity with AI/ML tools like TensorFlow, PyTorch, , Azure ML is required.
• Proficiency in designing data models and architectures for both cloud and on-premises environments is required.
• Team Player: Is responsive, flexible, and able to succeed in a team-oriented, collaborative environment, building effective working relationships and partnerships with internal partners, customers, and vendors is required.
• Highly Analytical: Is passionate about working with disparate datasets bringing data together to answer business questions. Collaborates to create, manage, and translate data to meaningful insights. Can work with external vendors to integrate data for analysis; knows how to build efficient and scalable infrastructure and data models is required.
• Analytical and data-driven: Thinks analytically; a structured thinker who can put complex ideas into clear frameworks; uses data to conduct root cause analysis and develops high quality, consumable, and consistent metrics that drive strategic objectives and priorities is required.
• Problem Solver: Ability to analyze, diagnose and resolve complex unstructured problems quickly, efficiently, and collaboratively is required.
• Communicator: The ability to communicate clearly and informatively, verbally and in writing, with colleagues, customers, and the community in both technical and non-technical professional language is required.
• Job specific: Strong understanding of APIs, microservices architecture, and containerization (e.g., Kubernetes, Docker) is required.
• Job specific: Skilled in generative AI models like GPT, DALL·E, etc. is required.

Tech Stack

• Apache
• Azure
• Cloud
• Docker
• Hadoop
• Kafka
• Kubernetes
• Microservices
• Python
• PyTorch
• Scala
• Spark
• SQL
• Tableau
• Tensorflow
• Terraform
• Unity
• Vault

Benefits

• covered by employer-paid basic life and accidental death & dismemberment insurance policies as well as long
• and short-term disability benefit coverages.
• eligible to participate in PEMCO’s 401(k) plan, which includes a generous employer match (2 for 1 on the first 6% employee pre-tax and/or Roth deferral, up to federal maximums).
• Vacation accrues at a rate of 10 days for new hires, and increases based on a schedule to a maximum annual accrual of 25 days based on tenure.
• Granted four (4) personal days immediately upon hire.
• Paid holidays for the eight (8) holidays observed by PEMCO throughout the calendar year.
• Granted up to ten (10) days of sick leave immediately upon hire (pro-rated based on hire date and full-time/part-time status), which is approximately 28 hours more per year than the Washington state-required accrual.
• In addition, PEMCO provides paid time off for bereavement, jury duty, and employee volunteering in the community.
• Flexible Spending Accounts
• Education Assistance Program after one year of service
• Scholarship program for children of PEMCO employees after one year of service
• Employee Assistance Program
• Well-being program
• Discretionary taxable gifts and gift cards
• And other Perks & Benefits including discounts on computer software and hardware, cell phone plans, and rental cars