Staff Software Engineer - Agent Architecture

PayNearMe is a company focused on simplifying payment experiences through innovative technology. They are seeking a Staff Software Engineer to lead the architecture and implementation of agentic AI products that enhance customer interactions across various modalities, ensuring safety and compliance in a regulated environment.

Responsibilities:

• Own the architectural direction for agentic AI at PayNearMe in partnership with other engineering leaders
• Design, build, and ship production agents—including voice and chat agents for a wide range of payment-related activities—that integrate cleanly with our Ruby on Rails / MySQL platform and partner services (ElevenLabs, Twilio, and others)
• Make and defend the "what kind of intelligence goes where" decisions: when to lean on a partner's stack vs. orchestrate frontier LLMs directly, when RAG is the right answer vs. tool calls vs. fine-tuning, when a small/fast/cheap model is sufficient vs. when a frontier model is warranted, and where classical ML or deterministic logic is a better fit than an LLM at all
• Design and operate the agent lifecycle as a closed loop: testing, offline evals, online evals, observability, scoring, and a disciplined feedback path from what we observe in production back into the test suite and eval set
• Own the unit economics of agent interactions. Token budgets, prompt and semantic caching, model cascades (cheap model first, escalate on uncertainty), batch APIs, latency-vs-cost tradeoffs, and per-tenant cost attribution should be instrumented and reasoned about explicitly—at scale, the gap between a well-engineered conversation and a naive one is the business
• Build the guardrails that make agents safe in a payments context: scope enforcement, refusal behaviors, deterministic handoffs for anything money-changing, PCI-compliant handling of card data, PII protection, and clear human-in-the-loop or fallback paths when confidence is low
• Treat voice as its own modality, not a text agent with a microphone—design for latency budgets, barge-in and turn-taking, STT/TTS error modes, DTMF fallback, recording and consent, and the operational realities of telephony
• Partner with Security, Compliance, and Legal to ensure agent behavior meets PCI-DSS, state-level payments regulations, and our customers' own compliance obligations
• Raise the bar across the org for agent engineering: define shared patterns for prompts, tools, evals, telemetry, and incident response; serve as a reviewer and approver for architecture decision records (ADRs) and major designs in the agent domain
• Partner with the Engineering Managers, Product, and other Staff peers to shape the roadmap—develop deep expertise in both the technical system and the business need (what our customers and their consumers actually want from an agent), and translate that into durable platform capabilities

Requirements:

• 8+ years of software engineering experience, with Staff-level scope (cross-team influence, major initiatives, long-term technical direction)
• Demonstrated experience shipping agentic AI systems to production—not prototypes, not internal copilots, but agents that real users have relied on. You should be able to talk concretely about what broke, what you changed, and how you knew it got better
• Hands-on experience with at least one modern agent framework (LangGraph, or comparable). You understand the tradeoffs between graph-based orchestration, ReAct-style loops, and more deterministic state machines, and you have opinions about when each is appropriate
• Deep, lived experience with the full agent lifecycle: prompt and tool design, offline and online evaluation, scoring rubrics, observability and tracing, and the discipline of feeding production signals back into your eval set and test suite
• Strong system design fundamentals: reliability, consistency, data modeling, and pragmatic API/service boundaries. You can integrate an agent into an existing transactional system without compromising the integrity of that system
• Comfortable working in a Ruby on Rails / MySQL environment, or confident in your ability to ramp quickly. You don't need to be a Rails expert, but you need to be able to read the code, work with the team that owns it, and design integrations that fit how the platform actually behaves
• Clear communication and strong judgment in high-stakes, cross-functional environments—especially in conversations with Security, Compliance, and Legal where the right answer is rarely the fastest one
• Ability to move between high-level architecture and hands-on coding. This role builds
• Experience shipping voice agents specifically, and a working understanding of how voice differs from text/chat: latency, turn-taking, STT/TTS failure modes, prosody, barge-in, DTMF, telephony quirks, and recording/consent
• Hands-on experience building and shipping voice-enabled applications — including conversational AI, TTS/STT pipelines, or telephony integrations — using any platform or stack (e.g., ElevenLabs, Twilio, Vonage, Deepgram, Vapi, LiveKit, or similar)
• Experience designing agents that combine multiple styles of intelligence—partner-managed conversational stacks, frontier LLMs, smaller/faster models, RAG, classical ML, and deterministic logic—and choosing among them based on cost, latency, accuracy, and risk
• Payments/fintech experience, or other regulated/high-integrity domains (healthcare, lending, insurance). Direct experience with PCI-DSS, TCPA, or similar regimes is a strong plus
• Experience building and operating evaluation infrastructure—LLM-as-judge, rubric-based scoring, regression suites, A/B and shadow testing for agents in production
• Experience designing agent platforms with multi-tenant configuration—per-customer rules, knowledge, allowed actions, escalation paths, and isolation of evals and guardrails
• Experience adversarially testing LLM-based systems (prompt injection, jailbreaking, social-engineering of tool-using agents), particularly where the agent can take consequential actions
• Experience optimizing the unit economics of LLM-based products: caching strategies, model cascades, prompt compression, and per-tenant cost attribution
• Experience with observability for non-deterministic systems: tracing across LLM calls and tool invocations, capturing the right signals for debugging and eval mining, and turning production traces into test cases
• Experience integrating AI capabilities into a Ruby on Rails monolith, or evolving a monolith to support a new class of workload without destabilizing it