AI Agent Evaluation Engineer defining quality standards and evaluation systems for Cresta’s conversational AI receptionist. Analyzing multilingual calls, calibrating LLM judges, and maintaining regression suites.
Responsibilities
Define and maintain evaluation rubrics covering adherence, outcomes, and conversation experience
Write annotation guidelines and failure taxonomies; run calibration sessions and measure inter-rater agreement
Classify and prioritize failure modes by severity and frequency
Review and annotate production conversations weekly, including multilingual evaluation
Trace failures to likely causes and route evidence to the appropriate owners
Build and operate evaluation pipelines combining automated checks, LLM-based judges, and human review
Calibrate automated judges against human labels
Design stratified sampling strategies
Build and maintain golden sets and regression suites
Maintain failure-mode coverage maps and close evaluation gaps
Design evaluations for high-stakes edge cases and upstream failures
Define quality and latency thresholds
Report quality trends and uncertainty to stakeholders and customers
Collaborate with forward-deployed engineers, backend engineers, product managers, and voice/language engineering teams
Document methodology for reproducible evaluation practices
Requirements
3+ years of professional experience in software engineering, applied ML, or data-intensive research roles
At least 1–2 years working hands-on with LLM-based systems in production
Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience
Built and operated an end-to-end evaluation system for a production LLM product
Deep understanding of evaluation methodology, including golden sets, judges, regression suites, and calibration
Understanding of LLM agents, including prompts, tool calls, context handling, and multi-turn state
Strong programming skills in Python and experience with LLM APIs
Ability to write rubrics and measure inter-rater agreement
Ability to design tests from open-ended quality questions
Willingness to listen to calls, label conversations, and build a failure taxonomy
Professional fluency in at least one language other than English
Strong written communication
Completion of a self-paced CoderPad technical assessment may be required after resume review
Benefits
Comprehensive medical, dental, and vision coverage with plans to fit you and your family
Flexible PTO to take the time you need, when you need it
Paid parental leave for all new parents welcoming a new child
Retirement savings plan to help you plan for the future
Remote work setup budget to help you create a productive home office
Monthly wellness and communication stipend to keep you connected and balanced
In-office meal program and commuter benefits provided for onsite employees
P&C insurance professionals converting underwriting, reinsurance, pricing, reserving, and claims expertise into AI training datasets. Fully remote, flexible contractor project.
Practicing economists creating and evaluating training data for advanced AI systems. Focusing on forecasting, antitrust, trade policy, public policy, and economic impact analysis.
Head of AI building Momentum’s self - improving ecommerce platform. Designing AI systems that identify retail opportunities, recommend improvements, and learn from experiments.
Freelance KI - Management lecturer delivering practice - oriented digital courses for Learning Digital GmbH, a certified continuing - education provider. Supporting and individually mentoring students worldwide.
AI technical expert building enterprise AI agents and learning tools for Equinix’s Digital University. Teaching engineers practical AI use, prompt engineering, and responsible AI practices.
US - licensed physicians evaluate clinical EHR cases and reasoning to test Mindrift’s medical AI systems. Flexible, task - based independent contractor project with no patient contact.
Physician Reviewer evaluating EHR vignettes, reasoning, diagnoses, and treatments for Mindrift’s medical AI projects. Flexible task - based contract for US - licensed physicians.
Economists translating applied economic analysis into AI training tasks for Terac. Evaluating model outputs and creating prompts on forecasting, antitrust, trade, and policy.
AI Tutor refining Grok’s Maltese voice and language capabilities for SpaceXSI. Annotating multilingual audio and localizing English content into natural, culturally accurate Maltese.
Temporary Irish AI Tutor refining Grok’s Irish voice and language capabilities for SpaceXSI. Annotating audio, localizing text, and improving speech datasets for AI products.