Hi,
Hope you are doing great!!!
Please revert me resume and details ASAP.
Data Engineer/Architect with AI Agent Automation Engineer (Databricks)
Azure Databricks Environment*******
Austin, TX Need only Locals or very nearby places within 50Miles
Hybrid Role (3d onsite in a week)
Duration: 12+ Months
Must have skills: Databricks( 5-10 Years), Azure cloud, LLM, AI
Responsibilities
Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks
Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads
Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments
Create agents for metadata management, lineage tracking, and governance automation within Databricks
Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic
Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines
Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases
Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap
Establish best practices for agent development, testing, deployment, and observability within Azure Databricks
Document agent workflows, decision logic, and operational runbooks for team knowledge sharing
Evaluate emerging AI/agent frameworks and technologies; recommend adoption where applicable
Job Requirements
Education
Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred)
Experience
10+ years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure)
Proven experience developing automation solutions, agents, or AI-powered tools (e.g., using LLMs, prompt engineering, or orchestration frameworks like Databricks Agent Framework (or LangChain, AutoGen)
Strong proficiency in Python and SQL with focus on production-grade code quality
Demonstrated experience with Azure Databricks (notebooks, jobs, workflows, DLT) or equivalent modern data platforms
Hands-on experience with CI/CD/CT tools (GitHub Actions, Azure DevOps Pipelines) and automation frameworks
Experience building data quality frameworks, testing patterns, or QA automation
Prior work in regulated or risk-sensitive environments preferred
Skills & Competencies
Expert-level Python programming with strong software engineering practices (testing, documentation, version control)
Strong SQL and PySpark optimization skills
Hands-on experience with LLM APIs, prompt engineering, and AI agent frameworks (e.g., Databricks Agent Bricks, LangChain, OpenAI, Azure OpenAI)
Proficiency in Azure Databricks, including Delta Lake, workflows, and Databricks Asset Bundles
Experience designing and implementing automation workflows using orchestration tools or serverless platforms
Strong problem-solving and systems thinking skills; ability to decompose complex workflows into automatable components
Excellent communication and documentation skills; ability to articulate technical concepts and agent design rationale
Self-directed learner with curiosity about emerging AI/automation technologies
Ability to collaborate effectively with cross-functional teams (data engineers, architects, QA, operations)
