Data Pipeline Development: Design, build, and optimize robust ETL/ELT pipelines using Azure Databricks, Spark, and SQL.
Data Processing & Transformation: Utilize Python, PySpark, and SQL to clean, transform, and aggregate complex data for analytics.
Azure Data Lake Management: Manage and optimize data storage and retrieval in Azure Data Lake Storage (ADLS) Gen2.
Delta Lake Implementation: Implement Delta Lake for ACID compliance, data versioning, and high-performance data lake operations.
Integration with Azure Services: Integrate Databricks with Azure Data Factory for orchestration, Azure Synapse Analytics for warehousing, and Azure Key Vault for security.
Performance Optimization: Monitor, troubleshoot, and optimize Databricks clusters and spark jobs to manage costs and performance.
Data Security & Governance: Implement Role-Based Access Control (RBAC), data encryption, and data lineage tracking.
Requirements Collaboration: Work with data scientists and analysts to support machine learning models and business intelligence (BI) reporting.

Databricks & Spark: Strong proficiency in PySpark, Spark SQL, and Databricks Jobs.
Azure Infrastructure: Familiarity with Azure Data Factory, ADLS, and Synapse Analytics.
Data Modelling: Experience in building data models (e.g., Star Schema, Snowflake).
Programming: Proficient in Python and SQL.
DevOps: Experience with Git and CI/CD tools.

Diversity Inclusion: At Exavalu, we are committed to building a diverse and inclusive workforce.
We nurture a culture that embraces all individuals and promotes diverse perspectives, where you can make an impact and grow your career.
Exavalu also promotes flexibility depending on the needs of employees, customers and the business. It might be part-time work, working outside normal 9-5 business hours or working remotely.
We also have a welcome back program to help people get back to mainstream after a long break due to health or family reasons.

Data Engineer

Key skills