Key skills
Data engineeringSQLPythonPySparkDatabricksBig dataDistributed systemsParquetDelta LakeHDInsightCloud-native data platformsAzure Data FactoryAzure SynapseAzure Storage AccountsAzure Streaming AnalyticsAzure Service FabricData lake architectureData governanceData quality frameworksData warehouseData lakeOperational data storeData pipeline developmentData pipeline automationInfrastructure as codeTerraformAzure DevOps PipelinesCI/CD workflowsGitAzure DevOps ReposVersion controlBranching strategiesAgile methodologyObservability best practicesContinuous learning mindsetData EngineeringData LakeAnalyticsAzureAzure DevOpsVersion ControlAgileScrumKanbanCI/CDLeadershipMentoring
About this role
Excella is a transformative technology firm that helps organizations unlock new possibilities. The Lead Data Engineer will drive the architecture, development, and optimization of data solutions while mentoring team members and providing strategic input on data platform decisions.
Responsibilities:
- Lead the design, development, and optimization of scalable data solutions
- Define data architecture strategies and ensure best practices are followed
- Oversee and guide the creation and automation of data pipelines and platforms
- Establish and enforce data quality and governance frameworks
- Collaborate with Architects, Product Owner, Data Scientists, and DevOps to align data solutions with business needs
- Research and evaluate emerging data technologies and methodologies
- Ensure seamless integration of data management solutions into client environments
- Develop risk mitigation strategies and implement data recovery plans
- Lead the development of data repositories, including data warehouses, data lakes, and operational data stores
- Mentor and support the development of junior and senior team members
Requirements:
- 6+ years of experience in data engineering and related technical domains
- Designed and built scalable, extensible data pipelines for production environments with strong expertise in SQL, Python, and PySpark
- 2+ years of hands-on experience with Databricks, including development, data processing, and pipeline optimization in a cloud-based environment
- Hands-on experience with big data and distributed systems, Parquet, Delta Lake, HDInsight
- Extensive work with cloud-native data platforms and services, such as Azure Data Factory, Synapse, Azure Storage Accounts, Streaming Analytics, Azure Fabric
- Deep understanding of data lake architecture, secure and performant data store design, and analytical platform patterns
- Proficient in orchestration tools and CI/CD workflows, infrastructure-as-code (e.g., Terraform, Azure DevOps Pipelines), and observability best practices
- Advanced Git and Azure DevOps Repos expertise and strong familiarity with version control, branching strategies, and collaborative development
- Leadership experience mentoring engineers and guiding cross-functional technical teams
- Excellent communicator, capable of translating complex technical ideas for both technical and non-technical audiences
- Agile practitioner with experience leading and delivering within Scrum/Kanban frameworks, and a continuous learning mindset
- Technically savvy, entrepreneurial spirit who thrives in environments that reward self-initiative and resourcefulness
- Advocating for adopting industry tools and practices at the right time
- Appreciate the importance of schema design and can evolve an analytics schema on top of unstructured data
- Leadership and mentoring of team members on project and within Excella
- Excited to try out new technologies and produce proofs-of-concept that balance technical advancement and user experience
- Empathetic working with stakeholders, listening to them, asking the right questions, and collaboratively developing the best solutions for their needs
- Champion for data privacy and integrity, and always act in the best interest of clients