About the role

Senior AI Engineer building autonomous agent features and evaluation infrastructure. Improving Monte Carlo's platform for observing, troubleshooting, and trusting production AI systems.

Responsibilities

Take open problems end-to-end from research and prototyping through production
Design and ship agent-powered features such as root-cause analysis, incident triage, and monitor generation
Integrate agent-powered features into the platform with engineering teams
Build eval infrastructure including golden datasets, regression suites, offline and online scoring
Own retrieval and context pipelines over customer metadata, lineage, and query history
Instrument agent behavior in production using traces, failure taxonomies, cost budgets, and latency budgets
Partner with data science on detection quality and experiment design
Partner with product management on agent capabilities and behavior
Set technical patterns, guardrails, and reusable internal tooling for LLM development

Experience building agents in production with real autonomy and internal loops
Experience running evals and monitoring agents after launch
Ownership of an eval framework with golden datasets, regression suites, offline and online scoring
Python as a daily language
Solid backend skills
ML or data science background
Understanding of model behavior
Ability to work from ambiguous problem statements and take solutions into production
Daily use of AI tools such as Claude or equivalent
Experience with backend and model-layer work; no frontend requirement
Ability to prioritize shipping effective solutions over unnecessary polish
Nice to have: statistics and hypothesis testing
Nice to have: building and maintaining MCP servers
Nice to have: experience with Snowflake, Databricks, dbt, or Airflow