Key skills
ApacheAWSAzureCloudETLMS SQL ServerOpen SourcePythonSparkSQLSSISAIMLAnalyticsBIBusiness IntelligenceApache SparkSQL ServerCI/CDDecision Making
About this role
Role Overview
- Serve as the subject matter expert for data and systems.
- Develops and maintains scalable data pipelines and build new API integrations to support continuing increases in data volume and complexity.
- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increase data accessibility and foster data-driven decision making across the organization.
- Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
- Develop end-to-end Data pipelines encompassing the full lifecycle from data ingestion, data transformation and Data Marts in our Enterprise Data Warehouse.
- Collaborate closely with Data Scientists to accelerate development of ML and AI solutions ready for production.
- Setup CI/CD/CT pipelines
- Contributes to engineering wiki, and documents work.
- Performs data analysis required to troubleshoot data related issues and assists in the resolution of data issues.
- Works closely with cross-functional teams of frontend and backend engineers, product managers, and analysts to enhance data models and support advanced BI and analytics.
- Defines company data assets (data models), ETL jobs to populate data models.
- Designs data integrations and data quality frameworks.
- Designs and evaluates open source and vendor tools for data lineage.
- Works closely with all business units and engineering teams to develop strategy for long term data platform architecture.
- Mentor junior data engineers, lead code reviews, and promote best practices and skill development.
Requirements
- BSc or MSc degree in Computer Science or a related technical field
- 5+ years of Python development experience
- 5+ years of MS SQL experience
- 5+ years of experience with Warehouse Architecture, schema design and dimensional data modeling.
- Ability in managing and communicating data warehouse plans to internal clients
- Experience designing, building, and maintaining data processing systems on multiple platforms both Cloud (Azure, AWS, MS Fabric) and On-Premises (MS SQL Server, SSIS) is advantageous
- Good experience in Apache Spark.
- Experience debugging and reasoning about production issues is essential.
- Experience presenting demos and training of technical, non-technical and analytical resources.
Tech Stack
- Apache
- AWS
- Azure
- Cloud
- ETL
- MS SQL Server
- Open Source
- Python
- Spark
- SQL
- SSIS
Benefits
- Hybrid work flexibility
- Professional development opportunities