Staff AI Systems Engineer

Flock Safety is the leading safety technology platform, helping communities thrive by taking a proactive approach to crime prevention and security. The Staff AI Systems Engineer will support the development of Night Shift, an AI research assistant, by contributing to system architecture and owning the AI evaluation framework to enhance investigative processes.

Responsibilities:

• Immerse yourself in the current system design and agent/tooling landscape. Understand the core customer use cases and data flows
• Support the team by shipping a few quick wins (e.g., refining tool APIs, prompt engineering, fixing bugs)
• Stand up the foundational eval and observability scaffolding (datasets, metrics, KPIs, reporting)
• Propose a technical architecture and implementation plan for an agent evaluation framework
• Deliver the MVP evaluation harness to produce initial metrics, enable debugging and perform regression testing
• Take on a system feature that offers demonstrated improvement against your MVP evaluation suite
• Productionize the evaluation and observability platform and make it the source of truth for quality and safety. (e.g. Online/offline tracing, alerting, dashboards, evaluations and PR-gated regression suite)
• Own the roadmap for evolving the agent evaluation platform
• Lead deeper R&D threads (e.g., lightweight fine-tuned projection layers, specialized embeddings, multimodal understanding) that can improve system performance on core metrics

Requirements:

• Familiarity with Agentic Systems: Hands-on experience with LLM agents including: LLM API use (e.g. LangChain/LangGraph, vLLM, OpenAI/Gemini/Anthropic APIs)
• Agent Design: tool use (e.g. via MCP), retrieval, memory, grounding/attribution for claims, and guardrails
• Architectural patterns: planning and hand-off for multi-agent systems, context management
• RAG: vector/hybrid search (e.g. pgvector, turbopuffer, rerankers, etc.)
• ML Platform expertise: 5+ years building and shipping ML systems to production; experience in the following areas: Backend Python and JS familiarity required; Typescript/Golang familiarity welcome
• Web services (e.g. Express/FastAPI, REST, SSE, JWTs)
• Cloud Infrastructure (e.g. AWS, Terraform, VPC, Networking)
• Backend databases/stores (e.g. Postgres, Redis)
• Observability (e.g. Prometheus, Grafana, OpenTelemetry, LangSmith/Langfuse)
• Experience with LLM Evaluations at scale: You've built offline/online eval harnesses and are familiar with the methodologies and metrics to measure: Search, retrieval, and recommendation performance
• Safety & robustness (security, compliance, red-teaming, regression testing)
• Cost, performance and latency trade-offs
• Feeling uneasy that you haven't ticked every box? That's okay; we've felt that way too. Studies have shown women and minorities are less likely to apply unless they meet all qualifications. We encourage you to break the status quo and apply to roles that would make you excited to come to work every day
• Durable execution (e.g. Temporal, Hatchet)
• OLAP (e.g. ClickHouse, Bigquery)
• ML Inference (e.g. PyTorch, TensorRT, NVIDIA Triton), ideally in multimodal domains (text/image/video)
• Compute orchestration (e.g. Kubernetes, Prefect, Ray)
• Agentic task success, trajectory quality, preference learning (e.g. SFT, DPO, RLHF, LLM-as-judge)