Key skills
PythonSQLMachine LearningMLLLMAnalyticsLeadershipCommunicationCollaboration
About this role
Role Overview
- Build ML operations analytics tracking pipeline health, costs, latency, and quality metrics across all services
- Lead production LLM evaluation initiatives, designing custom metrics, quality monitoring and LLM observability in general
- Drive development efforts in collaboration with different ML teams based on analytics outputs
- Collaborate with our data annotation team on prompt engineering and evaluation dataset creation
- Help with curating the Filevine dataset
- Create actionable dashboards and insights for ML teams and leadership
- Help define SLAs, cost optimization and failure prevention strategies for our ML pipelines
Requirements
- 3+ years in data science, ML operations, or analytics engineering
- Experience with LLM evaluation frameworks and quality metrics design
- Strong Python, SQL, and data visualization skills
- Knowledge of prompt optimization techniques
- Deep understanding of latest trends in LLMs and their production deployments
- Excellent communication skills in English to translate technical metrics into business insights
- Collaborative mindset with ability to work across machine learning teams
Tech Stack
- Python
- SQL
Benefits
- Independent contractors responsible for their own taxes and compliance with applicable laws