Novia Infotech is seeking an experienced Azure Data Engineer to design, build, and support large-scale data integration and analytics solutions. This role will focus on building modern data pipelines using Azure technologies to deliver analytics-ready datasets for enterprise reporting and financial systems.
Responsibilities:
- Design and develop enterprise-grade data integration and application services using Azure-based technologies
- Build and maintain ADF pipelines integrated with Azure services, including Azure Function Apps for custom Python execution
- Develop and optimize ETL/ELT workflows in Databricks using PySpark, leveraging Delta Lake (delta-spark) for reliable data management
- Implement and manage Azure Data Lake Storage (ADLS Gen2) using medallion architecture (Bronze, Silver, Gold)
- Orchestrate ADF pipelines that trigger Databricks jobs for downstream data transformations and aggregations
- Manage Databricks Unity Catalog for metadata, data governance, and access control
- Create Gold-layer datasets for analytics and reporting using PySpark joins and aggregations
- Develop high-performing SQL queries, stored procedures, and transformations
- Support Power BI reporting and analytics needs
- Participate in Agile development ceremonies and collaborate with cross-functional teams
- Use Azure DevOps for source control, work tracking, and CI/CD pipeline execution
- Create unit test plans, perform unit testing, conduct code walkthroughs, and support peer reviews
- Provide production support, including issue analysis, troubleshooting, and remediation
- Support data integrations and reporting initiatives for SACT Systems (100%)
Requirements:
- 7+ years of experience in analysis, design, and software development
- 5+ years of experience designing and implementing big data solutions in distributed environments
- Strong hands-on experience with Python
- Azure Databricks (PySpark)
- Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2 (ADLS)
- Databricks Unity Catalog
- Power BI
- Expert-level knowledge of SQL, including complex queries, functions, and stored procedures
- Experience working in Agile / Scrum environments
- Proven experience using Azure DevOps for development, reporting, and deployments
- Strong understanding of application lifecycle management and production support
- Excellent written and verbal communication skills
- Strong work ethic, positive attitude, and a passion for continuous learning
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience)
- Microsoft Certified: Azure Data Engineer Associate
- Microsoft Certified: Power BI Data Analyst Associate
- Experience with .NET / C#, .NET Core, ASP.NET, and Web API development
- Advanced experience with Azure DevOps
- CI/CD pipelines using YAML
- Experience developing or supporting financial systems