Senior Staff Machine Learning Engineer

Zscaler is a cloud security company whose Zero Trust Exchange platform protects customers from cyberattacks and data loss. The Senior Staff Machine Learning Engineer will build AI-, machine learning-, and data engineering-driven solutions for high-fidelity risk identification, including threat detection, data analysis, agent development, and production deployment.

Responsibilities:

• Translate risk identification methods into agent logic, understanding the benefits and limitations of agents and ensuring quality across risk analysis, explanations, and recommendations
• Collaborate closely with threat-research to understand data, threats, and their approach to developing security heuristics
• Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
• Follow a data-driven quality approach to threat detection, including backtesting, balancing precision vs recall, tuning, and quality control
• Deploy and monitor your solutions in production within our CI/CD framework

Requirements:

• Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
• 8+ years of professional Python development with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
• Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, incident debugging in live distributed systems, and familiarity with cloud and infrastructure-as-code in an AWS environment
• Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects
• Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
• Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with risk analysis teams
• Data engineering experience building and operating pipelines over large-volume event data using SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto), schema evolution, backfills, and data quality validation