QA automation analyst testing PayFacto’s merchant onboarding web application and APIs. Automating Playwright tests and validating acquiring settlement and reconciliation flows.
About the role
- SDET building automated tests, evals, and performance frameworks for CrowdStrike’s AI-native cybersecurity platform. Improving reliability of distributed cloud services and agentic workflows.
Responsibilities
- Design, develop and maintain automated test suites and frameworks for distributed, cloud-based systems
- Maintain and extend agentic test generation and analysis tooling built into major testing frameworks
- Build AI-powered workflows and agent skill files to automate manual test toil, including test data setup, failure triage and release validation
- Maintain eval suites, golden datasets and scoring pipelines for AI agents, scoring correctness, tool use, trajectory, faithfulness, safety, latency and cost
- Enforce CI eval gates so changes to prompts, models, tools or orchestration are measured before release
- Feed production failures back into evaluation sets
- Detect and mitigate unreliable AI output, including hallucinated assertions, flaky or redundant tests, incorrect failure triage and judge bias
- Design, run and analyze performance, load and stress tests to find bottlenecks and validate scalability across distributed services
- Validate integration of platform systems, services, data and applications, as well as customer-facing features end-to-end
- Review product, system and agent designs for testability, evaluability, flaws and improvements
- Share knowledge and contribute to the team's quality and evaluation standards
Requirements
- Degree in Computer Science or equivalent experience (OOP, data structures, algorithms, distributed systems)
- 5+ years in software quality, including 3+ years of hands-on SDET or test automation development
- Strong programming skills in at least one of Java, Node.js or Python, with Python preferred for eval and agent work
- Proven experience designing, building and maintaining test frameworks integrated with CI/CD pipelines (e.g., Jenkins, GitLab CI/CD)
- Experience testing web services and APIs (REST, gRPC, JSON)
- Hands-on experience with performance and load testing using tools such as Locust or k6, including designing realistic workloads and interpreting results (latency percentiles, throughput, error rates)
- A skeptical, verification-first mindset toward AI output, with a clear sense of when to use deterministic assertions and when to use probabilistic evaluation
- Proven experience utilizing AI technologies to enhance decision-making, streamline workflows and processes, improve efficiency and drive business outcomes
- Expected to maintain legal entitlement to work in Canada
- Subject to background checks, including criminal record, credit, and/or reference checks
- Bonus: End-to-end eval-driven development experience, including release gating, trajectory and tool-use evals, and calibrating LLM-as-judge against human review
- Bonus: Prompt engineering and context or tool design for agents
- Bonus: Integrating performance tests into CI/CD and using observability tools (e.g., Grafana, Prometheus, CloudWatch)
- Bonus: Background in cybersecurity, threat intelligence, malware analysis or sandboxing/detonation environments
Benefits
- Market leader in compensation and equity awards
- Comprehensive physical and mental wellness programs
- Competitive vacation and holidays for recharge
- Paid parental and adoption leaves
- Professional development opportunities for all employees regardless of level or role
- Employee Networks, geographic neighborhood groups, and volunteer opportunities to build connections
- Vibrant office culture with world class amenities
- Variable/incentive compensation
- Equity
- Equal pay for equal work
Job title
Job type
Experience level
Salary
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Junior SDET testing March Networks’ cloud video surveillance and analytics solutions. Automating and executing end - to - end tests across cloud applications, hardware, APIs, and databases.
Senior SDET automating web, mobile, API, and integration testing for Aequilibrium’s Canadian credit union banking modernization projects. Validating secure, reliable member banking experiences.
Senior QA Test Engineer building automated tests and simulators for Litmus’s industrial AI platform. Covering industrial protocols, REST APIs, CI pipelines, upgrades, and production reliability.
Test engineering project lead developing aerospace test applications for Honeywell Aerospace. Managing multi - program schedules, risks, technical plans, and validation through production.
Software Development Engineer in Test defining quality gates and automating testing, deployment, and validation. Supporting Akamai's distributed cloud and edge platform for digital experiences.
Senior SDET improving automated testing, CI/CD quality gates, and agentic tooling. Supporting Onfleet’s logistics platform that powers millions of deliveries worldwide.
Honeywell Aerospace test engineering lead managing test applications across aerospace programs. Coordinating requirements, schedules, risks, design reviews, and production introduction.
Senior SDET building evaluation infrastructure and testing autonomous AI agents. Nexxa applies AI systems to heavy industries, manufacturing, infrastructure, logistics, and legacy environments.
Principal SDET owning VM and fleet test strategy for Akamai’s distributed cloud and edge platform. Validating hypervisor behavior, performance benchmarks, and deployment safety.