Evaluating and optimizing the performance of audio models for customer support scenarios. Create role-play scenarios and evaluate service interactions across various domains.
Responsibilities
Create and execute role-play–based evaluation scenarios that simulate realistic customer service interactions across multiple domains
Contribute to the development of diverse and representative datasets used to assess conversational audio agents.
Evaluate model performance across a standardized set of qualitative and quantitative metrics.
Ensure evaluations reflect real customer expectations for clarity, efficiency, and natural conversational flow.
Requirements
Strong fluency in English
Strong verbal communication skills in a simulated customer support context
Spanish proficiency including fluency across all language skills: reading, listening, writing, and speaking.
Access to a high-quality microphone to ensure clean, reliable audio input during evaluations
Comfort working with structured prompts, evaluation rubrics, and technical guidelines
Device capable of running audio recording software and opening large technical documentation
Benefits
Company-sponsored benefits such as health insurance and PTO do not apply
Senior Governance Advisor supporting analytics teams in responsible AI governance. Overseeing model risks and promoting a culture of ethical data use at Desjardins.
Senior Software Development Manager leading enterprise - scale AI/ML platform development at Autodesk. Guiding a team of software and ML engineers in building innovative solutions.
Manager of Technical Staff leading the Sovereign AI Modelling team at Cohere. Designing and implementing AI models to advance cutting - edge research and solutions.
Senior Marketing Coordinator launching and scaling AI education programs at an innovative startup. Driving applications and enrollment through hands - on marketing execution across key channels.
Lead Analyst for People Technology and AI driving automation in HR tech stack with Workday expertise. Transforming HR processes and ensuring system integrity through strategic partnerships and innovative solutions.
Manager in BDO’s Digital, Data & AI Strategy & Transformation practice. Leading engagements and driving strategies to enable digital transformation for Canadian organizations.
Junior AI Cybersecurity Specialist for FortiGuard IoC team using AI for threat detection. Designing ML models and developing AI solutions to combat cybersecurity threats.
AI Automation Analyst designing and scaling automation solutions for Instacart's Commercial organization. Leading technical strategy and mentoring teams in AI initiatives.
AI Evaluation & Annotation Specialist reviewing and annotating AI - generated responses while adhering to project guidelines and maintaining quality standards.
Forward Deployed Engineer working with internal teams to design and optimize workflows leveraging AI at Knak. Collaborating closely with DevOps to build internal tools and systems.