Databricks Engineer

Job Title: Databricks Engineer
Location: Wilmington, DE / Philadelphia, PA / Washington, DC (Local Candidates Only)
Duration: 12+ Months

Must have "Databricks Certified Data Engineer Professional certification'

Key Responsibilities:

Design, develop, and maintain scalable end-to-end data pipelines using Databricks, Delta Lake, PySpark, SQL, and AWS.

Build batch, real-time, and streaming data solutions using Kafka, AWS Kinesis, and related technologies.

Implement data ingestion, transformation, curation, and storage frameworks within a Databricks Lakehouse architecture.

Optimize Spark jobs, troubleshoot production issues, and ensure data quality, governance, lineage, and reliability.

Develop secure, scalable, and cost-efficient solutions leveraging AWS S3, Glue, Lambda, Kinesis, and Redshift.

Implement security controls including RBAC, encryption, and data masking.

Follow CI/CD, DevOps, and data engineering best practices using Git and automation tools.

Collaborate with BI, Analytics, Data Science, and business stakeholders to deliver curated datasets supporting reporting and AI/ML initiatives.

Lead architecture, design, and implementation of enterprise data solutions while driving Lakehouse best practices.

Mentor engineers, conduct code reviews, establish development standards, and oversee sprint planning and delivery.

Partner with cloud, security, and platform teams to ensure compliance, governance, and operational excellence.

Support analytics initiatives using Power BI and Tableau.

Required Qualifications;

Strong hands-on experience with Databricks.

Expertise in Python, PySpark, SQL, and distributed data processing.

Extensive experience with AWS services including S3, Glue, Lambda, Kinesis, and Redshift.

Strong experience building and supporting ETL/ELT pipelines.

Deep understanding of Delta Lake and Lakehouse Architecture.

Experience with both streaming and batch processing frameworks.

Knowledge of CI/CD, Git, performance tuning, and troubleshooting.

Experience with Power BI and Tableau.

Databricks Certified Data Engineer Professional certification is mandatory.

About IDEXCEL, INC
Idexcel is an IT services organization, with a mission to bring great people and great organizations together. Our diverse client base represents a wide range of industries, including technology, telecom, insurance, healthcare, manufacturing, banking & financial services, food & commodities trading and federal organizations. Our teams of experienced recruiters directly work with client companies seeking exceptional people to help with their business initiatives. Idexcel, Inc. is an Equal Opportunity Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, disability, military status, national origin or any other characteristic protected under federal, state, or applicable local law.