Key skills

SQLPythonAzure Data PlatformApache SparkApache KafkaHadoop, MapReduce, and HiveData WarehousingLLM Pipelines and GenAI Data PreprocessingRetrieval-Augmented Generation (RAG) and Embedding ModelsVector Databases MilvusVector Databases PostgreSQLML Pipeline Deployment and CI/CD KubeflowML Pipeline Deployment and CI/CD MLflowML Pipeline Deployment and CI/CD AirflowML Pipeline Deployment and CI/CD SageMaker PipelinesPower BINoSQLAIMLGenAILLMRAGMLflowKubeflowMilvusETLAnalyticsBISparkKafkaAirflowHadoopAzureSageMakerVertex AIPostgresCI/CDCommunication

About this role

Sapphire Software Solutions Inc is recruiting a Senior Azure Data Engineer for an unnamed client. The role supports GenAI solutions for enterprise products by building and maintaining large-scale analytics and data pipelines, including ingestion, streaming, batch processing, ETL, and SQL performance tuning.

Responsibilities:

Responsible for the building, deployment, and maintenance of mission-critical analytics solutions that process terabytes of data quickly at big-data scales
Contributes design, code, configurations, manage data ingestion, real-time streaming, batch processing, ETL across multiple data storages
Responsible for performance tuning of complicated SQL queries and Data flows

Requirements:

At least 9+ years of experience required
The candidate must have excellent communication skills
Experience coding in SQL/Python(\•), with solid CS fundamentals including data structure and algorithm design
Hands-on implementation experience working with a combination of the following technologies: Hadoop, Map Reduce, Kafka, Hive, Spark(\•), SQL and NoSQL data warehouses (\•)
Experience in Azure cloud data platform(\•)
Must have: understanding of LLM Pipelines, including data preprocessing for GenAI Models
Experience working with vector databases (Milvus, Postgres, etc.) (\•)
Knowledge of embedding models and retrieval-augmented generation (RAG) architectures
Understanding of LLM pipelines, including data preprocessing for GenAI models (\•)
Experience deploying data pipelines for AI/ML workloads(\•), ensuring scalability and efficiency
Familiarity with model monitoring(\•), feature stores (Feast, Vertex AI Feature Store), and data versioning(\•)
Experience with CI/CD for ML pipelines(\•) (Kubeflow, MLflow, Airflow, SageMaker Pipelines)
Understanding of real-time streaming for ML model inference (Kafka, Spark Streaming) (\•)
Knowledge of Data Warehousing, design, implementation and optimization
Knowledge of Data Quality testing, automation and results visualization
Knowledge of BI reports and dashboards design and implementation (PowerBI)(\•)
Experience with supporting data scientists and complex statistical use cases