About the role

AI Evaluation Engineer building statistical evaluation, CI/CD testing, and observability infrastructure for Scopic’s production AI agents. Designing adversarial, multi-turn, and structured-output test systems.

Responsibilities

Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment
Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency
Implement fast, low-cost code-based checks, including JSON schema validation, regex matching, API status checks, and embedding similarity
Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops
Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to secure sandboxes and guardrails
Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools
Reduce LLM API costs through caching and mocking

Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems
Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs
Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution
Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads
Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging
Ability to independently design and own an evaluation architecture from scratch
Strong written and spoken English
At least 3 hours of daily overlap with US Central Time business hours
Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI
Deep understanding of embedding models and vector databases
Experience load testing non-deterministic endpoints