Software Developer – Agentic Evaluation

Posted 3 hours ago

Apply Now

Resume Score

Check how well your resume matches this job before you apply.

Sign in to check score

About the role

  • Software Developer on Fusion platform creating AI solutions in Autodesk's Product Development and Manufacturing Solutions. Collaborating with engineers and architects for intelligent agentic systems development.

Responsibilities

  • Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and end-to-end development workflow optimization using frameworks like LangGraph, AutoGen, or the Anthropic Agent SDK (Claude Code)
  • Design and implement agentic workflows that coordinate multiple AI agents to autonomously drive test automation across UI, API, integration, and system levels
  • Build evaluation frameworks and custom benchmarks for agentic systems, including comparisons of AI agents against commercial solvers
  • Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, context fidelity, and end-to-end task completion rates

Requirements

  • BS/MS in Computer Science, Machine Learning, or related applied AI field
  • Expertise in Python and ML frameworks (PyTorch, Transformers, scikit-learn)
  • Experience with Large Language Models applied to software understanding or test generation
  • Knowledge of AI evaluation methodologies and metrics for agentic task completion and test quality
  • Strong foundation in statistical analysis and experimental design
  • Experience with developer workflow and productivity measurement frameworks
  • Background in software engineering or QA with close collaboration with development teams (preferred)
  • Familiarity with test automation frameworks (e.g., Playwright, Selenium, Pytest, Appium) and CI/CD pipelines (preferred)
  • Experience designing benchmarks that compare AI agents against commercial or domain-specific solvers (preferred)
  • Hands-on experience with MCP (Model Context Protocol) within agentic pipelines (preferred)
  • Knowledge in vision-language models or multi-modal AI for UI and system-level understanding and evaluation (preferred)
  • Experience with Azure AI Foundry/ML or AWS cloud ML platforms (preferred)

Benefits

  • Annual cash bonuses
  • Commissions for sales roles
  • Stock grants
  • Comprehensive benefits package

Job type

Full Time

Experience level

Junior

Salary

CA$88,000 - CA$128,700 per year

Degree requirement

Bachelor's Degree

Tech skills

AWSAzureCloudPythonPyTorchScikit-LearnSelenium

Location requirements

OnsiteTorontoCanada

Report this job

Found something wrong with the page? Please let us know by submitting a report below.