Kraken is a mission-focused company rooted in crypto values, seeking a skilled Data Engineer to join their Data Platform team. The role involves building and maintaining high-performance data pipelines, ensuring data reliability and scalability, and enabling access to clean and timely data across the organization.
Responsibilities:
- Build scalable and reliable data pipelines that collect, transform, load and curate data from internal systems
- Augment data platform with data pipelines from external systems
- Ensure high data quality for pipelines you build and make them auditable
- Drive data systems to be as near real-time as possible
- Support design and deployment of distributed data store that will be central source of truth across the organization
- Build data connections to company's internal IT systems
- Develop, customize, configure self service tools that help our data consumers to extract and analyze data from our massive internal data store
- Evaluate new technologies and build prototypes for continuous improvements in data engineering
Requirements:
- 5+ years of work experience in relevant field (Data Engineer, DWH Engineer, Software Engineer, etc)
- Experience with data-lake and data-warehousing technologies and relevant data modeling best practices (Presto, Athena, Glue, etc)
- Proficiency in at least one of the main programming languages used: Python and Scala. Additional programming languages expertise is a big plus!
- Experience building data pipelines/ETL in Airflow, and familiarity with software design principles
- Excellent SQL and data manipulation skills using common frameworks like Spark/PySpark, or similar
- Expertise in Apache Spark, or similar Big Data technologies, with a proven record of processing high volumes and velocity of datasets
- Experience with business requirements gathering for data sourcing
- Bonus - Kafka and other streaming technologies like Apache Flink