Evaluating and optimizing the performance of audio models for customer support scenarios. Create role-play scenarios and evaluate service interactions across various domains.
Responsibilities
Create and execute role-play–based evaluation scenarios that simulate realistic customer service interactions across multiple domains
Contribute to the development of diverse and representative datasets used to assess conversational audio agents.
Evaluate model performance across a standardized set of qualitative and quantitative metrics.
Ensure evaluations reflect real customer expectations for clarity, efficiency, and natural conversational flow.
Requirements
Strong fluency in English
Strong verbal communication skills in a simulated customer support context
Spanish proficiency including fluency across all language skills: reading, listening, writing, and speaking.
Access to a high-quality microphone to ensure clean, reliable audio input during evaluations
Comfort working with structured prompts, evaluation rubrics, and technical guidelines
Device capable of running audio recording software and opening large technical documentation
Benefits
Company-sponsored benefits such as health insurance and PTO do not apply
GHL and AI automation manager building outbound, nurture, and conversational AI systems. Supporting Jumpfactor’s digital marketing campaigns for MSP and IT clients.
Research Director analyzing integration, API management, and AI - enablement markets for IDC, a global technology intelligence provider. Forecasting growth, advising clients, and presenting insights to technology leaders.
Director shaping Desjardins’ Personal Services data, analytics, and AI strategy. Leading transformation, governance, executive partnerships, and high - impact use cases for a cooperative financial group.
AI solutions lead rebuilding Instacart’s grocery - delivery Marketing processes with agentic workflows. Driving adoption, measurable impact, and cross - functional AI planning.
Director leading Desjardins’ AI, data and analytics strategy for Personal Services. Driving data transformation, executive decisions, advanced analytics and responsible governance.
Senior AI Solutions Analyst guiding Desjardins Group’s financial services sectors in responsible productivity AI adoption. Assessing use cases, value, risks and Microsoft AI solutions.
AI Safety Red Teamer stress - testing Mercor’s frontier AI models. Identifying vulnerabilities, jailbreaks, hallucinations, and policy failures through adversarial evaluation.
AVP leading AI transformation, including fraud detection, for Manulife, a global financial services provider. Driving secure, governed Classical, Generative, and Agentic AI solutions across Corporate Functions.
Generative AI Analyst validating Canadian French translations for Welo Data, a global AI data company. Reviewing and annotating multilingual AI - generated content for training and quality improvement.
Senior engineers evaluating AI coding - agent interactions for G2i. Assessing reasoning, explanations, outputs, and engineering judgment across Codex, Claude Code, and Cursor.