Quality Assurance Engineer testing Speer Technologies’ innovative software products. Creating automated tests, tracking bugs, and improving development quality.
About the role
- QA Engineer building evaluation infrastructure for Nexxa’s autonomous AI agents in heavy industries. Testing reasoning, tool use, safety, and reliability across industrial workflows.
Responsibilities
- Own quality for Nexxa's AI agent systems used in industrial environments
- Design and build evaluation harnesses and regression suites for LLM-based agents
- Develop golden datasets and labeled test sets covering edge cases, ambiguous inputs, and adversarial prompts
- Define and track quality metrics including groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations
- Build automated evaluation pipelines for model, prompt, and tool-integration changes and integrate them into CI/CD
- Conduct structured red-teaming and adversarial testing, including prompt injection, jailbreaks, tool misuse, and unsafe actions
- Test the full agent action loop: planning, tool selection, tool execution, error recovery, and final output
- Investigate and triage failures involving models, prompts, tools/APIs, or orchestration logic
- Partner with ML and backend engineers to produce actionable, reproducible bug reports
- Establish quality bars and sign-off criteria for new agent capabilities
- Mentor engineers on testing strategies for probabilistic, LLM-driven systems
- Advocate for testability and observability in agent architecture
- Collaborate with ML engineers, backend engineers, and Forward Deployed Engineers
Requirements
- 5+ years in QA/SDET roles, with demonstrated ownership of test strategy for complex systems
- Hands-on experience testing LLM-based products, chatbots, or AI agents
- Practical experience with eval frameworks or tooling such as promptfoo, DeepEval, RAGAS, or LangSmith, or a track record of building your own
- Strong scripting/programming ability, Python preferred
- Understanding of LLM agents, including prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks
- Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift
- Familiarity with LLM-specific failure modes, including hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism
- Strong written communication skills
- Experience with human-in-the-loop evaluation workflows
- Background in ML/data science sufficient to read model evals and statistical significance
- Experience red-teaming or conducting adversarial/security testing on ML systems
- Familiarity with LLM observability/tracing tools such as LangSmith, Arize, Langfuse, or Weights & Biases
- Experience testing AI systems in industrial, IoT, or operational technology environments
- Prior experience setting up eval infrastructure from scratch at a startup or fast-moving team
Benefits
- Equity package
- Significant opportunities for career development and advancement
- Collaborative culture valuing innovation, discipline, and continuous improvement
- Competitive compensation
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior Test Analyst validating the Government of Alberta’s CLEO licensing solution. Supporting UAT, defect management, training, AI - assisted quality assurance, and go - live readiness.
LP
QA Professionals
LinkedIn Recruiter Post
Looking for #QA Professionals in Canada, Who is resided OR ready relocate to Vancouver, BC. If Interested - Please share your resume to [email protected]
QA Coordinator inspecting SUNPAN’s global furniture products for quality and compliance. Coordinating claims, corrective actions, supplier collaboration, and continuous improvement.
Quality Assurance Analyst providing independent assurance and risk - based testing across motor finance customer journeys. Assessing complaints, redress, accuracy, consistency, and fairness of outcomes.
Quality Assurance Team Lead leading independent testing of banking customer outcomes across the Motor Finance Commissions programme. Managing a small QA team and embedding the second - line framework.
Quality Assurance Analyst testing Computershare’s customer communication solutions. Designing test strategies, validating implementations, and supporting client UAT across global teams.
Systems Quality Engineer investigating software, systems, and hardware interaction issues for Kardium’s atrial fibrillation medical device. Driving root - cause analysis and product reliability improvements.
Manager leading R&D quality assurance for Apotex, a Canadian global health company developing affordable medicines. Overseeing GMP compliance, batch disposition, investigations, audits, and QA staff.
QA Analyst strengthening automated and manual testing for airline software solutions. Defining test strategies, validating requirements, and improving quality processes.