Software Development Engineer in Test

SOFICO Intelligence is hiring a Software Development Engineer in Test to build and own quality evaluation for its production agentic AI platform. The role develops evaluation harnesses and automated tests, verifies AI outputs and system safeguards, tests performance and security, and strengthens quality engineering across the platform.

Responsibilities:

• Design and build SOFICO Intelligence's evaluation harness for agentic AI: golden datasets, automated scoring, faithfulness and grounding checks, and regression detection across prompt, model, and agent changes
• Verify grounding and faithfulness, confirming every insight and every number traces back to real data, and build the automated checks that stop hallucinations from reaching production
• Test multi-agent orchestration end to end: agentic tool-calling loops, supervisor-specialist pipelines, autonomous background jobs, and the synthesis of findings into trustworthy reports
• Red-team the system for prompt injection, jailbreaks, and tool misuse, and rigorously prove multi-tenant data isolation so one organization can never reach another's data
• Validate that AI guard rails hold under pressure: iteration limits, agent-recursion depth, and token/cost ceilings behave correctly at the edges
• Pressure-test performance and scale: real-time streaming, high-concurrency conversations, latency, and cost under realistic and adversarial load
• Build automated tests for the Go microservices, data/SQL tools, and APIs that power the platform, and wire them into CI/CD for continuous evaluation
• Partner with AI and data engineers to define quality bars and acceptance criteria, and help evolve the supervisor review-and-retry loop that keeps the system honest
• Strengthen automated testing across the broader SOFICO Intelligence platform and, where it intersects, the wider SOFICO engineering environment, bringing modern test practices to established systems and tooling
• Help raise quality engineering standards consistently across the combined group

Requirements:

• You are comfortable with the ambiguity of testing systems that don't give the same answer twice
• You take pride in shipping software people can trust, and you want to see the products you work on succeed in the real world
• You do well in collaborative problem-solving environments, prefer coding and collaborating over sitting in meetings, and can prioritize your own work while working closely with a team
• You give and take constructive criticism well, and you bring a structured, analytical approach to building scalable evaluation systems
• Proven experience spanning both software development and quality assurance / quality engineering
• Strong coding skills, able to program confidently in at least one language (we work primarily in Go and Python)
• Hands-on experience with open-source automated testing tools and frameworks
• A solid understanding of RESTful APIs and the ability to perform HTTP requests from the command line or a tool like Postman
• Comfortable working from the command line and reasoning about distributed, concurrent systems
• Hands-on experience testing, evaluating, or building with LLM-based or agentic systems, not just interest. You have worked directly with model APIs, tool-calling, or multi-agent workflows and understand how they fail
• Demonstrated ability to evaluate non-deterministic AI output against a measurable quality bar. For example, building or running eval harnesses, scoring grounding and faithfulness, detecting quality regressions, or red-teaming for prompt injection and jailbreaks
• Working knowledge of LLM concepts that drive quality: prompting, context windows and token limits, hallucination and grounding, and the sources of non-determinism in model output
• A sense of pride and a strong desire to ensure trustworthy, bug-free software ships every day
• Fluent in English, as the role involves interaction with international teams
• Depth in AI evaluation methodology, such as designing golden datasets, LLM-as-judge scoring, supervisor/critic patterns, or automated regression detection for agentic pipelines
• Experience with Go, data warehouses / SQL analytics (e.g. Databricks), event-driven systems, or real-time streaming
• Familiarity with enterprise back-office or ERP environments, including Java back-ends and relational databases (Oracle, MSSQL, Postgres), is a plus for the cross-platform work
• Exposure to the automotive finance, leasing, fleet, or mobility domain