Key skills
ApacheCloudETLNoSQLPySparkSparkSQLLLMOpenAIELTData EngineeringApache SparkGitHub
About this role
Role Overview
- Leads the creation of comprehensive end-to-end detailed solutions for projects and platforms
- Develops and maintains technical solutions, ensuring adherence to engineering and architectural design principles
- Provides technical expertise with a focus on efficiency, reliability, scalability, and security
- Acts as a subject matter expert in multiple areas and consults with clients/project teams
- Develops functional, technical, and user interface designs for applications/systems that meet requirements
- Identifies and implements improvement solutions to enhance customer experience
Requirements
- 5+ years of experience building scalable data engineering solutions
- 5+ years of experience in Design and developed high-performance ETL/ELT pipelines using PySpark for batch and real-time workloads
- 5+ years of experience in Apache Spark (Batch & Structured Streaming)
- 5+ years of experience in PySpark (DataFrames, Spark SQL, UDFs)
- 5+ years of experience in Data Modeling & Dimensional Data Marts
- 5+ years of experience in Real-Time Data Ingestion Pipelines
- NoSQL Databases
- Cloud Data Platforms
- GitHub Copilot–Assisted Development
- OpenAI / LLM Integration
- Data Quality, Performance & Scalability
Tech Stack
- Apache
- Cloud
- ETL
- NoSQL
- PySpark
- Spark
- SQL
Benefits
- Health insurance
- Retirement plans
- Paid time off (including Vacation PTO, Flex PTO, and Holiday PTO)
- Banking benefits and discounts
- Career development
- Reward and recognition