Quisitive is one of Microsoft’s most recognized global partners, focusing on cloud transformation and data strategy. They are seeking a Senior Data Engineer to build and maintain data-driven solutions, optimizing pipelines and ensuring data quality for intelligent applications.
Responsibilities:
- Building and maintaining backend data ingestion and embedding pipelines
- Setting up environments, clone repositories, and running pipelines in JupyterHub
- Working on large-scale ETL processes, including converting Iceberg tables to Parquet and exporting data to S3 buckets
- Troubleshooting data quality issues and optimizing Spark jobs for efficiency
- Implementing retry mechanisms and debugging full-stack issues related to large file operations
- Working with GitLab for source control and Jira for project tracking
Requirements:
- Strong proficiency in Python for data processing and pipeline development
- Hands-on experience with Spark
- Experience with Oracle DB, PostgreSQL, and PGVector for embedding strategies
- Comfortable working with S3 buckets, Parquet, and CSV formats
- Power BI for dashboard development and reporting