Key skills
AirflowApacheAWSCloudPythonScalaSparkData EngineeringApache AirflowVersion ControlCI/CDMentoring
About this role
Role Overview
- Responsible for critical data ingestion (ingestion, processing)
- Ensure data engineering standards are met, both in code and in ingested data
- Collaborate with the Data Architect to define design decisions (focusing on scalability, performance, cost, and security)
- Guide and develop the data engineering team (code reviews, mentoring, defining best practices)
- Work with the product team on prioritization and roadmap development
- Lead critical projects
Requirements
- Strong expertise in large-scale data processing
- Advanced knowledge of Spark, Python and/or Scala
- Experience with orchestrators, preferably Apache Airflow
- Advanced knowledge of Python
- Experience with cloud platforms (preferably AWS)
- Familiarity with software development and data engineering best practices, version control, and CI/CD
Tech Stack
- Airflow
- Apache
- AWS
- Cloud
- Python
- Scala
- Spark
Benefits
- Experian is also proud to be an equal opportunity and affirmative action employer. If you have a disability or need an accommodation, please let us know as soon as possible.