Define guardrails and policies: security, privacy, compliance, and hallucination prevention.
Create evaluation strategy: metrics, datasets, automated tests, and acceptance criteria.
Optimize quality and cost (latency, context, error rates, caching, model routing).
Partner with engineering for production readiness (logging, auditing, monitoring, versioning).
Requirements
Hands-on experience with LLMs and applied AI (e.g., agents, prompting, RAG).
Strong foundation in Python and experimentation engineering (reproducibility, testing, metrics).
Knowledge of retrieval techniques (vectors/embeddings, hybrid search, reranking).
Experience or familiarity with quality evaluation (offline/online) and error reduction.
Ability to communicate technical decisions clearly to both business and engineering stakeholders.
Desirable (nice-to-have):
Experience in the financial sector and/or with risk, fraud, credit, operations, or compliance processes.
Experience with fine-tuning, evaluation frameworks and LLM observability.
Experience working in regulated environments with auditability requirements.
Tech Stack
Python
Benefits
Company-subsidized health plan for the employee.
Option to include dependents in the health plan with payroll deduction.
Dental assistance (optional).
Option to include dependents in the dental plan with payroll deduction.
Meal voucher or food allowance.
Transportation voucher (optional).
Impact & Care – Personal guidance program offering confidential emotional support and counseling in psychological, legal, financial, social and pet-related areas at no cost for the employee and legal dependents.
Gympass
Wellhub (access to 700+ gyms across Brazil with plans starting at R$29.90, deducted via payroll).
Option to include dependents in Gympass
Wellhub (up to 3 dependents — paid via credit card).
Access to Udemy through our intranet.
Partnerships with major consumer brands (discounts/offers).
SESC agreement for employee and dependents.
Discounts with educational institutions (undergraduate and postgraduate) and language/certification schools.