Senior Data Engineer

Quisitive is one of Microsoft’s most recognized global partners, focusing on cloud transformation and data strategy. They are seeking a Senior Data Engineer to build and maintain data-driven solutions, optimizing pipelines and ensuring data quality for intelligent applications.

Responsibilities:

• Building and maintaining backend data ingestion and embedding pipelines
• Setting up environments, clone repositories, and running pipelines in JupyterHub
• Working on large-scale ETL processes, including converting Iceberg tables to Parquet and exporting data to S3 buckets
• Troubleshooting data quality issues and optimizing Spark jobs for efficiency
• Implementing retry mechanisms and debugging full-stack issues related to large file operations
• Working with GitLab for source control and Jira for project tracking

Requirements:

• Strong proficiency in Python for data processing and pipeline development
• Hands-on experience with Spark
• Experience with Oracle DB, PostgreSQL, and PGVector for embedding strategies
• Comfortable working with S3 buckets, Parquet, and CSV formats
• Power BI for dashboard development and reporting