Identify and assess source datasets available in the customer’s global data lake for ingestion into the platform.
Map and align new datasets with the existing local data lake structure to maintain a consistent data format and schema.
Implement data ingestion pipelines across the lakehouse architecture layers (Bronze, Silver, and Gold).
Integrate newly ingested data into the existing data model, including Dimension and Fact tables within the current star schema architecture.
Replicate and adapt existing KPI calculation logic by redirecting established processing pipelines to the newly ingested datasets.
Develop output datasets and data products to deliver newly calculated KPIs to the relevant Business Units (BUs) using existing delivery processes.
Develop data validation logic and data quality checks using PySpark within Databricks to ensure accuracy and reliability of ingested data.
Integrate, process, transform, and cleanse datasets originating from multiple legacy systems.

Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
1-3 years of relevant experience in data engineering, with a strong portfolio of designing and implementing data solutions.
Expertise in big data technologies (Apache Spark, DBT), cloud platforms (Azure, GCP), and data development in data lake/delta lake architectures.
Proficiency in programming language SQL, and infrastructure as code technologies (Terraform, Helm charts for Kubernetes).
Expert knowledge of Kubernetes, object storages (S3, Azure Data Lake Store)
Strong problem-solving skills, excellent communication abilities, and the capacity to thrive in a fast-paced environment.

Data Engineer

Key skills