Key skills
AWSBigQueryCloudGrafanaJavaScriptKubernetesPostgresPrometheusPythonPyTorchRayRedisTerraformTypeScriptGoGolangAIMLLLMOpenAIAnthropicGeminiRAGLangChainAgenticLangGraphExpressFastAPISpringOpenTelemetry
About this role
Role Overview
- Help support our emerging product, Night Shift, an AI research assistant that amplifies the impact of investigators by automating the tedious, repetitive steps involved in working a case.
- Work closely with partners in Engineering (Backend, Frontend, and Design) in a fast-paced environment.
- Be one of the earliest technical contributors to our system architecture for agentic AI, and will own our AI evaluation framework.
- Achieve measurably faster, more accurate leads for every officer and every shift.
- Immerse yourself in the current system design and agent/tooling landscape. Understand the core customer use cases and data flows.
- Support the team by shipping a few quick wins (e.g., refining tool APIs, prompt engineering, fixing bugs).
- Stand up the foundational eval and observability scaffolding (datasets, metrics, KPIs, reporting).
- Propose a technical architecture and implementation plan for an agent evaluation framework.
- Deliver the MVP evaluation harness to produce initial metrics, enable debugging and perform regression testing.
- Take on a system feature that offers demonstrated improvement against your MVP evaluation suite.
- Productionize the evaluation and observability platform and make it the source of truth for quality and safety.
Requirements
- Familiarity with Agentic Systems: Hands-on experience with LLM agents including:
- LLM API use (e.g. LangChain/LangGraph, vLLM, OpenAI/Gemini/Anthropic APIs)
- Agent Design: tool use (e.g. via MCP), retrieval, memory, grounding/attribution for claims, and guardrails.
- Architectural patterns: planning and hand-off for multi-agent systems, context management.
- RAG: vector/hybrid search (e.g. pgvector, turbopuffer, rerankers, etc.)
- ML Platform expertise: 5+ years building and shipping ML systems to production; experience in the following areas:
- Backend Python and JS familiarity required; Typescript/Golang familiarity welcome
- Web services (e.g. Express/FastAPI, REST, SSE, JWTs)
- Cloud Infrastructure (e.g. AWS, Terraform, VPC, Networking)
- Backend databases/stores (e.g. Postgres, Redis)
- Observability (e.g. Prometheus, Grafana, OpenTelemetry, LangSmith/Langfuse)
- [Preferred] Durable execution (e.g. Temporal, Hatchet)
- [Preferred] OLAP (e.g. ClickHouse, Bigquery)
- [Preferred] ML Inference (e.g. PyTorch, TensorRT, NVIDIA Triton), ideally in multimodal domains (text/image/video)
- [Preferred] Compute orchestration (e.g. Kubernetes, Prefect, Ray)
- Experience with LLM Evaluations at scale: You’ve built offline/online eval harnesses and are familiar with the methodologies and metrics to measure:
- Search, retrieval, and recommendation performance
- Safety & robustness (security, compliance, red-teaming, regression testing)
- Cost, performance and latency trade-offs
- [Preferred] Agentic task success, trajectory quality, preference learning (e.g. SFT, DPO, RLHF, LLM-as-judge)
Tech Stack
- AWS
- BigQuery
- Cloud
- Grafana
- JavaScript
- Kubernetes
- Postgres
- Prometheus
- Python
- PyTorch
- Ray
- Redis
- Terraform
- TypeScript
- Go
Benefits
- Flexible PTO: We offer non-accrual PTO, plus 11 company holidays.
- Fully-paid health benefits plan for employees: including Medical, Dental, and Vision and an HSA match.
- Family Leave: All employees receive 12 weeks of 100% paid parental leave. Birthing parents are eligible for an additional 6-8 weeks of physical recovery time.
- Fertility & Family Benefits: We have partnered with Maven, a complete digital health benefit for starting and raising a family. Flock will provide a $50,000-lifetime maximum benefit related to eligible adoption, surrogacy, or fertility expenses.
- Spring Health: Spring Health offers a variety of mental health benefits, including therapy, coaching, medication management, and digital tools, all tailored to each individual's needs.
- Caregiver Support: We have partnered with Cariloop to provide our employees with caregiver support.
- Carta Tax Advisor: Employees receive 1:1 sessions with Equity Tax Advisors who can address individual grants, model tax scenarios, and answer general questions.
- ERGs: We offer four ERGs today
- Women of Flock, Flock Proud, LEOs and Melanin Motion.
- WFH Stipend: $150 per month to cover the costs of working from home.
- Productivity Stipend: $300 per year for Audible, Calm, Masterclass, Duolingo, etc.
- Home Office Stipend: A one-time $750 to help you create your dream office.