Technical lead building safe, evaluable LLM agents and AI infrastructure for OpenLoop’s telehealth platform. Setting architecture, observability, retrieval, and model operations direction.
LP
AI testing Engineer
Posted via LinkedIn Recruiter (not a company profile)
Posted 2 months ago
About the role
- We are seeking an AI Testing Engineer to validate and assure quality of AI/ML systems powering real-time contextual advertising on live video streams.
Responsibilities
- Validate AI-generated outputs against labelled datasets and expected outcomes. Perform content validation for native moments detection across video streams. Design and implement automated testing frameworks (Python-based). Build validation harness for JSON parsing and comparison, structured output validation using defined schemas. Conduct multi-modal validation (video, audio, transcript outputs). Execute system integration and end-to-end testing. Track accuracy metrics, defect logging, and root cause analysis. Automate regression and validation pipelines across datasets. Collaborate with Applied Scientists to validate LLM and GenAI outputs. Work within AWS environment using services such as S3, Lambda, CloudWatch.
Requirements
- Strong Python scripting for automation and testing. Experience in AI/ML model validation and QA workflows. Hands-on with JSON parsing, schema comparison, data validation pipelines. Experience in testing multi-modal AI systems (vision + NLP + audio), LLM outputs and generative AI pipelines. Good understanding of structured output validation and AI accuracy evaluation techniques. Familiarity with AWS cloud environment (S3, Lambda, logging/monitoring). Experience with media/video content analysis workflows preferred. Ability to handle mixed manual + automated testing approaches.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior software engineer evaluating AI coding agents such as Codex and Claude Code. Providing rigorous written and video feedback on engineering quality.
Senior software engineer evaluating AI coding agents for G2i. Assessing engineering judgment, explanations, and trustworthiness across Codex, Claude Code, and Cursor.
Senior engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering judgment, reasoning, and interaction quality.
Forward Deployed Engineer building full - stack AI solutions on AWS and Azure for Huron’s consulting clients. Partnering daily with business users to deliver tested features.
Staff AI Security Engineer securing EQ Bank’s enterprise AI, machine learning, and cloud platforms. Building guardrails, controls, automation, and detection capabilities for Canada’s Challenger Bank.
Staff Software Engineer owning OAuth, authorization, and agent delegation systems. Building governed identity infrastructure for Redpanda’s enterprise AI data platform.
Senior software engineer evaluating AI coding agents for G2i’s engineering team. Assessing reasoning, explanations, and engineering judgment in Codex, Claude Code, and Cursor interactions.
Senior software engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering quality.
Senior engineer evaluating Codex, Claude Code, and Cursor interactions for G2i. Providing rigorous written and video feedback on AI - generated engineering work.