Data Engineer, Databricks (Mid-Level)

Full Tilt Data, LLC is a data, analytics, and IT consulting firm specializing in health-related services for the federal government. The company is seeking a mid-level Data Engineer to develop secure, scalable Databricks lakehouse capabilities, data pipelines, unified data models, access-control frameworks, and AI-enabled search solutions for a federal program.

Responsibilities:

• Design, build, and maintain data pipelines on Databricks using Python and PySpark, including schema normalization, metadata capture, data quality checks, and lineage tracking
• Develop and improve canonical/unified data model (UDM) structures, including star schema, fact, and dimension table design, to support repeatable onboarding of new agency datasets
• Build and integrate Retrieval-Augmented Generation (RAG) and vector search capabilities to support AI-assisted contract search and natural language querying
• Contribute to Databricks Asset Bundles (DAB) and CI/CD pipelines to improve deployment consistency, testing, and release velocity
• Develop Python-based integration layers connecting the OPA/Rego policy engine to Databricks and Spark, enabling dynamic enforcement of access controls and data masking at query time
• Support Delta Sharing configurations, text extraction/OCR workflows, and Databricks Apps as the team expands into these areas
• Package, document, and harden pipeline and data model patterns into reusable, well-documented reference implementations that other agencies and teams can adopt independently
• Participate in weekly standups and contribute to monthly status reporting on task progress and milestones

Requirements:

• 3–5+ years of hands-on data engineering experience, including at least 1–2 years working directly in Databricks
• Strong proficiency in Python and PySpark for building and troubleshooting data pipelines
• Working knowledge of dimensional data modeling concepts (star schema, fact/dimension tables) and willingness to grow this into a core strength
• Experience with SQL and working across structured and semi-structured data sources
• Familiarity with CI/CD concepts and version-controlled deployment workflows (Git-based)
• Comfortable working in a fast-paced, collaborative team environment and picking up new tools quickly
• Ability to obtain/maintain a Federal Public Trust clearance
• Direct, Hands-on Experience In One Or More Of The Following, And Be Comfortable Ramping Quickly Across The Rest
• Complex pipeline development and improvement on Databricks/PySpark
• Star schema and unified data model (UDM) design and refactoring
• RAG / vector search implementation for search and natural language querying use cases
• Fact and dimension table design for analytic workloads
• Databricks Asset Bundles (DAB) and CI/CD best practices
• Delta Sharing, text extraction/OCR, and Databricks Apps (secondary priority areas)
• Exposure to Databricks Asset Bundles (DAB) or similar infrastructure-as-code deployment tooling
• Exposure to vector search, embeddings, or RAG-style architectures — specifically vector database/embedding tooling (e.g., Databricks Vector Search, pgvector, FAISS, or Chroma) and embedding or generation model integration (e.g., Databricks Model Serving, Azure OpenAI, Bedrock)
• Familiarity with Open Policy Agent (OPA) / Rego or other policy-as-code frameworks
• Experience with a general-purpose backend language and a modern frontend framework for adjacent API or UI work
• Prior experience on a federal contract or in a regulated data environment
• Databricks certification(s) (Data Engineer Associate/Professional, or Generative AI Engineer Associate)
• Familiarity with Unity Catalog governance features (fine-grained access control, row/column-level security, data lineage)
• Familiarity with federal compliance frameworks (e.g., NIST 800-53, FISMA, ATO processes) or experience handling CUI/PII
• An active Public Trust (or higher) clearance or investigation already in process
• Experience with data quality/testing frameworks