Senior Data Analyst

Rightsline is seeking a Senior Data Analyst – LLMOps to make the behavior and quality of its AI-powered features measurable. The role owns end-to-end observability, evaluation suites, telemetry pipelines, dashboards, experimentation, cost monitoring, and quality investigations while partnering with engineering, product, and data teams.

Responsibilities:

• Instrument Rightsline’s LLM and agent workflows end to end with tracing and structured logging, using tools such as LangSmith, Langfuse, or equivalent
• Define and maintain the metrics that describe AI system health — latency, token and cost per request, error and fallback rates, tool-call success, and retrieval hit rates
• Build and maintain dashboards and alerting so quality and cost regressions surface in hours rather than in customer tickets
• Design and maintain offline evaluation suites — golden datasets, scoring rubrics, and LLM-as-judge grading — for prompts, retrieval pipelines, and agent flows
• Run online evaluations and A/B experiments on production traffic, and report results with enough rigor to support a ship / no-ship decision
• Track quality across model, prompt, and retrieval changes, and maintain the benchmark history that shows whether the product is actually improving
• Partner with engineers on human review and annotation workflows, including labeling guidelines and inter-rater agreement
• Build and maintain the pipelines that move trace, evaluation, and usage telemetry into the analytics stack using SQL and Python
• Analyze how customers actually use AI features — adoption, drop-off, and failure patterns — and translate findings into prioritized recommendations
• Produce recurring reporting on AI quality, cost, and usage for engineering, product, and leadership audiences
• Monitor token spend and model usage, and identify where prompt, model, or caching changes cut cost without hurting quality
• Investigate quality incidents — bad or unsafe outputs, hallucinations, tool failures, latency spikes — from trace to root cause
• Follow and help extend Rightsline’s development best practices and standards in the pipelines, queries, and evaluation code you ship
• Attend and contribute to team meetings, and present telemetry and evaluation findings in a way non-specialists can act on
• Raise the team’s LLMOps literacy — help engineers instrument their own features, write their own evals, and read the dashboards

Requirements:

• 5+ years in data analytics, analytics engineering, or data science, including hands-on work with LLM-based systems in production
• Strong SQL and Python (including pandas and notebook-based analysis), with the discipline to build repeatable pipelines rather than one-off queries
• Hands-on experience with LLM observability and evaluation tooling such as LangSmith, Langfuse, LangChain/LangGraph, or equivalent
• Practical understanding of LLM application architecture — prompting, retrieval-augmented generation, tool calling, and agent workflows — and how each of them fails
• Experience designing evaluation methodology: golden datasets, scoring rubrics, LLM-as-judge grading, human annotation, and A/B testing on production traffic
• Working knowledge of experiment design and statistics, enough to say confidently whether a difference is real
• Ability to work independently, meet deadlines, and be accountable for your work, while operating within established standards
• Excellent written and verbal communication skills, and a strong sense of ownership, urgency, and initiative — you can explain a quality regression to an engineer and to an executive in the same afternoon
• BS in Computer Science, Statistics, Data Science, or equivalent knowledge
• Experience with AWS services such as Lambda, SQS/SNS, API Gateway, Step Functions, and Amazon Bedrock
• Familiarity with dbt, Airflow or a similar orchestrator, Git, and BI tools such as Looker, Power BI, or Tableau
• Working knowledge of MS SQL, Redis or another NoSQL store, and vector stores used for retrieval
• Exposure to model providers and platforms (Anthropic, OpenAI, Amazon Bedrock), guardrail and PII-redaction tooling, and prompt versioning and release workflows
• Experience with coding agents or AI developer tools (Claude Code, GitHub Copilot, Cursor, or similar) in day-to-day analysis work