Key skills
ApacheETLPySparkSparkSQLELTData EngineeringApache Spark
About this role
Role Overview
- Leads the creation of comprehensive end-to-end detailed solutions for projects and platforms
- Provides technical expertise with a focus on efficiency, reliability, scalability, and security
- Consults on development of requirements and specifications
- Works with Technology partners to define metrics and performance goals for the application
- Ensures respective programs/policies/practices are well managed and meet business needs
Requirements
- Undergraduate degree, Post Graduate degree or Technical Certificate
- Strong academic background (e.g., computer science, engineering)
- 5-7 years relevant experience
- 5+ years of experience building scalable data engineering solutions, with deep expertise in Apache Spark and PySpark
- 5+ years of experience in Design and developed high-performance ETL/ELT pipelines using PySpark for batch and real-time workloads
- 5+ years of experience in Apache Spark (Batch & Structured Streaming)
- 5+ years of experience in PySpark (DataFrames, Spark SQL, UDFs)
- 5+ years of experience in Data Modeling & Dimensional Data Marts
- 5+ years of experience in Real-Time Data Ingestion Pipelines
Tech Stack
- Apache
- ETL
- PySpark
- Spark
- SQL
Benefits
- Health insurance
- 401(k) matching
- Paid time off (including Vacation PTO, Flex PTO, and Holiday PTO)
- Career development
- Banking benefits and discounts