TEKsystems is a leading provider of business and technology services, and they are seeking a Senior Content Engineer to join their team. In this role, you will be responsible for building, evaluating, and optimizing AI-driven content systems while ensuring high-quality outcomes through collaboration with program managers, engineers, and researchers.
Responsibilities:
- Partner with the Program Manager to design, iterate, and maintain end-to-end content evaluation guidelines
- Create structured artifacts such as specs, decision frameworks, mappings, and changelogs to document guideline evolution
- Ensure guidelines are actionable, internally consistent, and aligned with human evaluation standards
- Identify and resolve cases where guidelines are ambiguous, inconsistent, or fundamentally unlabelable
- Design, test, and refine system prompts to guide AI models toward consistent, high-quality, human-aligned outputs
- Apply structured experimentation to evaluate prompt effectiveness and document tradeoffs
- Collaborate cross-functionally to version, validate, and improve prompts over time
- Analyze model evaluation results using appropriate metrics, focusing on distributions, prevalence, slices, and uncertainty, not just averages
- Identify patterns, failure modes, and quality risks across datasets and model outputs
- Translate analysis into clear, actionable recommendations for model behavior and process improvements
- Support supervised fine-tuning efforts by preparing, annotating, and quality-controlling large-scale text datasets
- Define and maintain gold set lifecycles, annotation specs, and QC plans
- Provide editorial oversight to ensure labeling consistency, accuracy, and alignment with evaluation goals
- Define, document, and maintain known and emerging failure states for content evaluation and generation systems
- Proactively surface risks related to consistency, nuance, safety, and edge cases
- Collaborate with partners to address root causes and implement mitigation strategies
- Identify opportunities to improve content pipelines, workflows, and quality controls
- Think in systems: inputs, contracts, versioning, monitoring, and outputs
- Balance tradeoffs such as speed vs. quality and consistency vs. nuance when proposing improvements
Requirements:
- Experience in developing and maintaining content evaluation guidelines
- Ability to create structured artifacts such as specs, decision frameworks, mappings, and changelogs
- Experience in designing, testing, and refining system prompts for AI models
- Ability to apply structured experimentation to evaluate prompt effectiveness
- Experience in analyzing model evaluation results using appropriate metrics
- Ability to identify patterns, failure modes, and quality risks across datasets and model outputs
- Experience in preparing, annotating, and quality-controlling large-scale text datasets
- Ability to define and maintain gold set lifecycles, annotation specs, and QC plans
- Experience in defining, documenting, and maintaining known and emerging failure states for content evaluation and generation systems
- Ability to identify opportunities to improve content pipelines, workflows, and quality controls
- Experience in balancing tradeoffs such as speed vs. quality and consistency vs. nuance