Data Engineer

Role Overview

• Build batch and streaming ingestion pipelines
• Design and structure data lakes and data warehouses
• Create ML-optimized datasets
• Implement embedding pipelines
• Build vector indexing for RAG
• Ensure data quality, governance, and security
• Optimize storage and processing costs
• Work with AI Engineers to structure feature stores

Requirements

• Recommended Technology Stack:
• Languages: Python, advanced SQL, Scala (optional)
• Orchestration & Transformation: Apache Airflow, dbt, Prefect
• Data Processing: Spark, Pandas, PySpark
• Storage: Data Lakes (S3, GCS, Azure Blob), Data Warehouses (BigQuery, Snowflake, Redshift), NoSQL databases, vector databases (Pinecone, Weaviate, FAISS)
• Streaming: Kafka, Pub/Sub
• Infrastructure: Docker, Kubernetes, Cloud (AWS, GCP or Azure)
• The following hard skills are desirable:
• Data modeling
• ETL / ELT
• Distributed processing
• Advanced SQL
• Scalable data architecture
• Experience with unstructured data (text, logs, PDFs)
• DataOps concepts
• Data versioning and data quality
• Required soft skills:
• Systems thinking
• Strong organization
• Attention to detail
• Scalability mindset
• Collaboration with ML teams
• Proactivity in preventing bottlenecks

Tech Stack

• Airflow
• Amazon Redshift
• Apache
• AWS
• Azure
• BigQuery
• Cloud
• Docker
• ETL
• Google Cloud Platform
• Kafka
• Kubernetes
• NoSQL
• Pandas
• PySpark
• Python
• Scala
• Spark
• SQL