Key skills
AirflowAmazon RedshiftApacheAWSCloudETLInformaticaKafkaMS SQL ServerPythonSparkSQLSSISMachine LearningMLflowELTData WarehousingAnalyticsSnowflakeRedshiftDatabricksApache KafkaApache AirflowData MiningS3GlueSQL Server
About this role
Role Overview
- Own design and maintenance of all aspects of data solutions including modeling, developing, technical documentation, data diagrams and data dictionaries.
- Provide expertise in the development of standards, architectural governance, design patterns, and practices, evaluate best applicable solutions for different use cases
- Determines and develops architectural approaches and solutions, conducts business reviews, documents current systems, and develops recommendations
- Lead the data strategy and own the vision and roadmap of data products
- Work with stakeholders to ensure that data related business requirements for protecting sensitive data are clearly defined, communicated, and well understood and considered as part of operational prioritization and planning
- Develop, maintain, and optimize data infrastructure using Delta Lake, MLflow, and Databricks SQL to enhance data management, processing, and analytics.
- Utilize Snowflake’s features such as data sharing, zero-copy cloning, and automatic scaling to optimize data storage, accessibility, and performance.
- Ensure effective management of both semi-structured and structured data within Snowflake’s architecture.
- Implement and manage data storage solutions using Amazon S3, perform data warehousing with Amazon Redshift.
- Design and implement data integration workflows using AWS Glue to orchestrate and automate data movement and transformation.
- Design and implement scalable data pipelines using tools like Apache Kafka or Apache Airflow to facilitate real-time data processing and batch data workflows.
- Apply advanced analytics techniques, including predictive modeling and data mining, to uncover insights and drive data-driven decision-making.
Requirements
- Overall experience of 15+ years
- Data architecture, data platform, data warehouse related experience of 10+ years
- Hands on experience with snowflake
- 4+ years experience, Data bricks 4+ years
- Proficiency in features such as Delta Lake, MLflow, and Databricks SQL.
- Experience in managing Spark clusters and implementing machine learning workflows.
- Solid experience in emerging and traditional data stack components such as: batch and real time data ingestion, ETL, ELT, orchestration tools, on-prem and cloud DW, Python, structured, semi and unstructured databases
- Knowledge of features like Snowflake’s data sharing, zero-copy cloning, and automatic scaling.
- Experience in working with Snowflake’s architecture for semi-structured and structured data.
- Experience with services like Amazon S3, Amazon Redshift, and AWS Glue.
- Proficiency in tools such as Apache NiFi, Talend, Informatica, or Microsoft SQL Server Integration Services (SSIS).
- Experience in designing and implementing data pipelines using tools like Apache Kafka or Apache Airflow.
- Ability to perform data profiling, data quality assessments, and performance tuning.
- Experience in comparing and evaluating different data technologies based on criteria like performance, scalability, and cost.
- Skills in applying advanced analytics techniques, including predictive modeling and data mining.
- Expert with industry standard data practices, data strategies and data concepts
- Demonstrated experience in architecting/re-architecting complex data systems and data models.
- Demonstrated experience in overall system design, including database selection and solutioning.
Tech Stack
- Airflow
- Amazon Redshift
- Apache
- AWS
- Cloud
- ETL
- Informatica
- Kafka
- MS SQL Server
- Python
- Spark
- SQL
- SSIS
Benefits
- Flexible work arrangements