Lead AI Engineer with Spark, AWS Services

EPAM Systems is a global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. The Lead AI Engineer will build and maintain data pipelines for AI/GenAI applications supporting Risk-Based Quality Management for clinical trials, with a focus on RAG document ingestion, vector indexing, data APIs, and AWS data infrastructure.

Responsibilities:

• Architect and construct RAG document ingestion pipelines (chunking, embedding, vector indexing) to support clinical trial quality data
• Establish and oversee vector databases (AWS OpenSearch) to support RAG-driven AI workflows
• Create batch and streaming ETL/ELT pipelines from the ground up for unstructured clinical data (PDF, DOCX, clinical reports)
• Construct and expose data APIs that AI applications can consume
• Enhance chunking strategies, embedding generation, and retrieval performance within RAG architectures
• Oversee data quality, lineage, and governance across AI/ML data pipelines
• Set up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
• Partner with Data Scientists and Backend Developers as part of a unified pod team

Requirements:

• Minimum 7 years of practical, large-scale data engineering experience
• Strong background in RAG document ingestion pipelines (chunking, embedding, vector indexing)
• Skilled in using AWS OpenSearch as a vector database for RAG workflows
• High-level command of Python, along with SQL and Spark SQL
• Experience transforming unstructured data (PDF, DOCX) for use in RAG/LLM applications
• Working knowledge of AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
• Understanding of Docker-based containerization
• Ability to develop custom pipelines from the ground up, going beyond simple configuration of pre-built services
• Proficiency in English at a B2+ level
• Experience within the pharmaceutical or life sciences sector
• Exposure to Snowflake and Pinecone (as an alternative vector database)
• Understanding of SageMaker processing jobs
• Proficiency with CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure-as-code tools (CDK or Terraform)
• Familiarity with clinical data standards (CDISC, ADaM, SDTM)