Principal SDET owning VM and fleet test strategy for Akamai’s distributed cloud and edge platform. Validating hypervisor behavior, performance benchmarks, and deployment safety.
About the role
- Senior SDET building evaluation infrastructure and testing autonomous AI agents. Nexxa applies AI systems to heavy industries, manufacturing, infrastructure, logistics, and legacy environments.
Responsibilities
- Own quality for Nexxa’s AI agent systems through automated and manual testing
- Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents
- Build automated evaluation and test pipelines integrated into CI/CD quality gates
- Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs
- Implement API, contract, integration, and end-to-end tests for services, tools, and orchestration layers
- Develop performance, load, reliability, failure-injection, and resilience tests
- Perform exploratory testing of agent behavior across ambiguous requests, multi-turn sessions, and unusual operational scenarios
- Review agent traces and outputs for reasoning quality, action appropriateness, and safety
- Execute manual regression and release verification in staging and customer-like environments
- Conduct manual red-teaming for prompt injection, jailbreaks, tool misuse, and unsafe actions
- Support UAT and pilot rollouts with Forward Deployed Engineers
- Write test plans, test charters, bug reports, and manual test documentation
- Create and manage golden datasets and labeled test sets with versioning and drift detection
- Design rubrics and conduct human-in-the-loop evaluations
- Define quality metrics such as groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations
- Build dashboards to track quality metrics over time
- Test full agent action loops, including planning, tool selection, execution, recovery, and final output
- Investigate and triage failures across models, prompts, tools/APIs, and orchestration logic
- Establish quality bars, release criteria, and sign-off processes
- Decide what to automate and continually shift repeatable manual work into automation
- Mentor engineers on testing strategies and test-driven practices for probabilistic LLM systems
- Advocate for testability, observability, and tracing in agent architecture
- Collaborate with ML engineers, backend engineers, and Forward Deployed Engineers
Requirements
- 5+ years as an SDET, QA engineer, or software engineer with a strong testing focus
- Demonstrated ownership of test strategy and automation for complex distributed systems
- Proven experience in both manual and automated testing, including exploratory testing, test planning, and high-quality bug reports
- Strong programming skills, with Python preferred
- Experience building test frameworks and infrastructure from the ground up
- Hands-on experience with CI/CD systems such as GitHub Actions, GitLab CI, or Jenkins
- Experience with containerized test environments, including Docker and Kubernetes
- Experience testing APIs and microservices, including contract, integration, and end-to-end testing
- Hands-on experience testing LLM-based products, chatbots, or AI agents
- Understanding of generative-system testing and limitations of deterministic assertions
- Practical experience with eval frameworks or equivalent self-built tooling
- Understanding of LLM agents, including prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks
- Experience designing test data and labeled datasets, including sourcing, sampling, and dataset drift management
- Familiarity with LLM-specific failure modes, including hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism
- Comfort defining correctness where no single right answer exists
- Strong written communication skills
- Preferred: experience with human-in-the-loop evaluation workflows
- Preferred: performance/load testing and chaos/fault-injection testing
- Preferred: ML/data science background sufficient to read model evaluations and statistical significance
- Preferred: ML systems red-teaming or adversarial/security testing
- Preferred: observability and tracing tools for LLM applications and general observability stacks
- Preferred: industrial, IoT, or operational technology testing experience
- Preferred: UAT, pilot, or customer-facing validation experience
- Preferred: startup or fast-moving-team experience setting up test and evaluation infrastructure from scratch
Benefits
- Comprehensive salary and equity package
- Significant opportunities for career development and advancement
- Innovative environment focused on transforming heavy industries through AI and automation
- Collaborative culture valuing innovation, discipline, and continuous improvement
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
QA Automation Analyst building scripts, dashboards, and validation workflows for Keywords Studios’ game projects. Improving test coverage, reliability, and QA efficiency.
SDET building automated testing and stress suites for a startup’s distributed AI infrastructure. Validating GPU kernels, AI frameworks, and performance - critical systems at scale.
Senior DevOps engineer building full - stack automation tools and CI/CD platforms for General Motors’ automotive software teams. Improving testing, quality monitoring, and engineering delivery workflows.
Quality Engineering Lead modernizing automated testing for POMS’ life sciences MES software. Building Playwright frameworks, leading QA teams, and improving release quality through metrics.
Senior SDET automating testing, deployment, and validation for Akamai's Kubernetes - based cloud platform. Building reliable test frameworks for distributed cloud and edge services.
Software Engineer in Test building Playwright, API, CI, and performance automation for Stord’s commerce and fulfillment platform. Developing AI - assisted testing workflows and release - quality infrastructure.
Senior test engineer automating deployment, testing, and validation for Akamai’s distributed cloud and edge platform. Building reliable frameworks for microservices and platform services.
Head of Production Test Engineering leading production testing for Flipper Zero, Flipper One, and BUSY Bar devices. Managing test engineering, factory deployment, and manufacturing quality improvements.
Senior SDET testing Akamai's distributed cloud APIs and identity management platform. Building Python automation, integration tests, and CI/CD quality infrastructure.