Develop, optimize, and improve data systems, including ETL pipelines, storage, and entity resolution
Build infrastructure and data automation pipelines to ingest, process, and load data from various sources
Automate and integrate new components into the data pipeline
Collaborate with stakeholders and data science teams to develop data products
Create unit and stress-test components to monitor technical performance
Develop data analysis tools to provide data insights and capture key metrics
Follow best practices for data governance, quality, cleansing, and other ETL-related activities

7+ years of development experience in data engineering within a production environment
Proven experience working with large datasets exceeding 100M+ records or multiple terabytes
2+ years of development experience in highly scalable, distributed systems and cluster architectures using AWS and utilizing EMR
5+ years of hands-on programming experience with Python
5+ years of professional experience working in big data ecosystems; Spark is required; PySpark is preferable
3+ years of experience with SQL, schema design, and dimensional data modeling
2+ years of professional experience working with dataflow orchestration tools, such as Airflow
2+ years of experience with non-relational databases (e.g., DynamoDB, Elasticsearch, etc.)
A bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field is required

Senior Data Engineer

Key skills