Design, develop, and maintain scalable batch data pipelines using Scala, Databricks Spark, Databricks SQL and Airflow
Implement optimized ETL/ELT processes to ingest, cleanse, transform, and enrich large volumes of telecom network, usage, and operational data
Ensure pipeline reliability, observability, and performance tuning of Spark workloads
Build and manage data architectures leveraging AWS services (such as but not limited to) S3, Lambda, IAM, and CloudWatch
Implement infrastructure-as-code using Terraform
Ensure security best practices and compliance with telecom regulatory requirements (GDPR, Data sovereignty, retention)
Collaborate with cross-functional teams (Architecture, DevOps, Network Engineering, Business Intelligence)
Document system designs, data flows, and best practices

4+ years of experience as a Data Engineer or Big Data Developer
Strong proficiency in Scala and functional programming concepts
Advanced experience with Apache Spark (batch processing using Data Frame API and low-level Spark API's, performance tuning, cluster optimization)
Experience with optimized SQL-based data transformations for analytics and machine learning workloads
Hands-on experience with Databricks including notebooks, jobs, Delta Lake, Unity Catalog, and MLflow (nice to have)
Solid understanding of CI/CD practices with Git, Jenkins/Gitlab Actions
Strong AWS skills: S3, Lambda, IAM, CloudWatch, and related services
Knowledge of distributed systems, data governance, and security best practices
Experience with Airflow integration with AWS services for end-to-end orchestration across cloud data pipelines
Experience with IaC tools: Terraform or CloudFormation
Experience with Python is a Plus
Experience with DBT is a Plus
Experience with Snowflake is s Plus

Senior Data Engineer – Tieto Tech Consulting

Key skills