AI Agent Evaluation Engineer

Posted 3 days ago

Apply Now

Resume Score

Check how well your resume matches this job before you apply.

Sign in to check score

About the role

  • AI Agent Evaluation Engineer defining quality standards and evaluation systems for Cresta’s conversational AI receptionist. Analyzing multilingual calls, calibrating LLM judges, and maintaining regression suites.

Responsibilities

  • Define and maintain evaluation rubrics covering adherence, outcomes, and conversation experience
  • Write annotation guidelines and failure taxonomies; run calibration sessions and measure inter-rater agreement
  • Classify and prioritize failure modes by severity and frequency
  • Review and annotate production conversations weekly, including multilingual evaluation
  • Trace failures to likely causes and route evidence to the appropriate owners
  • Build and operate evaluation pipelines combining automated checks, LLM-based judges, and human review
  • Calibrate automated judges against human labels
  • Design stratified sampling strategies
  • Build and maintain golden sets and regression suites
  • Maintain failure-mode coverage maps and close evaluation gaps
  • Design evaluations for high-stakes edge cases and upstream failures
  • Define quality and latency thresholds
  • Report quality trends and uncertainty to stakeholders and customers
  • Collaborate with forward-deployed engineers, backend engineers, product managers, and voice/language engineering teams
  • Document methodology for reproducible evaluation practices

Requirements

  • 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles
  • At least 1–2 years working hands-on with LLM-based systems in production
  • Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience
  • Built and operated an end-to-end evaluation system for a production LLM product
  • Deep understanding of evaluation methodology, including golden sets, judges, regression suites, and calibration
  • Understanding of LLM agents, including prompts, tool calls, context handling, and multi-turn state
  • Strong programming skills in Python and experience with LLM APIs
  • Ability to write rubrics and measure inter-rater agreement
  • Ability to design tests from open-ended quality questions
  • Willingness to listen to calls, label conversations, and build a failure taxonomy
  • Professional fluency in at least one language other than English
  • Strong written communication
  • Completion of a self-paced CoderPad technical assessment may be required after resume review

Benefits

  • Comprehensive medical, dental, and vision coverage with plans to fit you and your family
  • Flexible PTO to take the time you need, when you need it
  • Paid parental leave for all new parents welcoming a new child
  • Retirement savings plan to help you plan for the future
  • Remote work setup budget to help you create a productive home office
  • Monthly wellness and communication stipend to keep you connected and balanced
  • In-office meal program and commuter benefits provided for onsite employees
  • Equity as part of total compensation

Job type

Full Time

Experience level

Mid levelSenior

Salary

Not specified

Degree requirement

Bachelor's Degree

Tech skills

Python

Location requirements

RemoteCanada

Report this job

Found something wrong with the page? Please let us know by submitting a report below.