Lead Data Engineer – AWS, Databricks & PySpark

ScolerTec Inc is seeking a Lead Data Engineer to design, build, and lead enterprise-scale cloud data engineering solutions. The role focuses on scalable data platforms and pipelines, AWS and Databricks implementation, and technical direction and mentoring of data engineers.

Responsibilities:

• Design and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures
• Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads
• Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies
• Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers
• Integrate pipelines with AWS compute, storage, messaging, and analytics services
• Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns
• Provide day-to-day technical guidance and mentoring to data engineers
• Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring
• Support streaming and event-driven data processing
• Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment
• Optimize pipeline performance, scalability, reliability, and cloud cost
• Support production incident resolution and root-cause analysis
• Partner with architecture, governance, migration, QA, DevSecOps, and security teams
• Maintain technical documentation, runbooks, and audit-ready engineering artifacts

Requirements:

• 7+ years of specialized data-engineering experience
• Strong hands-on experience building enterprise-scale data pipelines
• Strong production experience with AWS data and compute services
• Strong hands-on Databricks experience
• Strong Apache Spark/PySpark and Delta Lake experience
• Strong Python and SQL skills
• Strong ETL/ELT and data-transformation experience
• Experience with batch and streaming/near-real-time ingestion
• Experience with data orchestration tools
• Experience with CI/CD and Infrastructure as Code
• Experience with data quality, governance, metadata, and classification
• Proven technical leadership, design review, code review, and mentoring experience
• Federal, public-sector, or regulated-environment experience preferred
• Databricks Certified Data Engineer Professional
• AWS Certified Data Engineer
• AWS Certified Solutions Architect – Professional
• AWS Certified Data Analytics – Specialty
• CDMP