Technical lead building safe, evaluable LLM agents and AI infrastructure for OpenLoop’s telehealth platform. Setting architecture, observability, retrieval, and model operations direction.
About the role
- Principal AI Engineer developing LLM-powered product capabilities for ABC Fitness. Collaborating cross-functionally to drive innovation and build cutting-edge solutions in AI technology.
Responsibilities
- Work cross-functionally with Product/Design to translate ambiguous problems into measurable outcomes and shippable iterations.
- Lead the estimation of work at the quarterly level.
- Provide alternative solutions and negotiate with Product Management.
- Build and ship LLM-powered product capabilities end-to-end: APIs, orchestration logic, integrations, and production rollouts
- Design and implement agentic workflows (tool use, planners/routers, multi-step reasoning, handoffs, memory/state management) using modern orchestration frameworks (e.g., Lang Chain/Lang Graph-style)
- Develop strong prompting + context engineering practices, including prompt versioning and experimentation (e.g., A/B testing approaches)
- Build evaluation harnesses for LLM apps: curated test sets, automated checks, offline/online metrics, human review loops, and regression gates for changes
- Collaborate with platform/infra teams to deploy on AWS, ensuring reliability, security, scalability, latency, and cost controls
- Be an active participant in the day-to-day agile activities of the team including sprint planning, daily standups, sprint reviews, and retrospectives
- Instrument systems for observability (quality, latency, token/cost, errors) and create feedback loops to improve model+system behavior over time
- Develop high quality solutions; leads the creation of new standards, patterns, and best practices
- Write architecture & design documentation for new products, systems, and patterns
- Review the work of teammates to ensure quality and adherence to system architecture and best practices; identify where new designs will require architectural changes
- Mentor junior developers to grow their skills
- Troubleshoot and resolve complex and highly escalated customer/QA found issues in a timely manner
- Experiment with new technologies that can enhance our tech stack. Provide new technology insights to remainder of development team.
- Lead the implementation of development process improvements that add efficiency for the entire development team
- Participate in build vs. buy analysis and complex decision making
- Conduct regular, self-guided study to stay current on new and existing technologies
- Participate in the hiring process and technical screens with an aim of attracting and hiring the very best engineers; work to expand the onboarding efficacy and vision
- May represent the team through presentations to other company departments and customers
Requirements
- Engineering or Comp Science degree or equivalent work experience
- 9+ years of professional experience in software engineering, applied ML, data science engineering, or AI product engineering
- Strong Python skills, experience building backend services (e.g. Flask/FastAPI) and integrating with REST APIs
- Some applied ML / data science experience (e.g., experiments, model evaluation, error analysis, feature work, or shipping ML-informed product changes), even if your recent focus has shifted toward software/AI engineering
- Practical understanding of LLMs/SLMs and how they behave in production (hallucinations, prompt sensitivity, latency/cost tradeoffs)
- Hands-on experience building LLM applications (RAG, tool use, agents, workflow orchestration), with the ability to demonstrate end-to-end ownership from prototype‚ production‚ measurement, iteration
- Experience with cloud infrastructure (e.g. AWS) and shipping production workloads (containers, deployments, monitoring)
- Strong engineering fundamentals: testing, debugging, performance thinking, and clean, maintainable design
- Excellent communication skills and ability to collaborate effectively in fast-paced, agile environments
- Comfortable learning new technologies and system
Benefits
- Time Off – with our open PTO policy, we offer flexibility in how and when you take PTO!
- Days of Disconnect – once a quarter, we take a collective breather and enjoy a day off together around the globe. #oneteam
- Parental & Pawternity Leave – we offer leave for when your family grows by two feet or four paws!
- Best Life Perk – we are committed to meeting you wherever you are in your fitness journey with a quarterly reimbursement
- Medical/Dental/Vision coverage
- EAP – we get you help when you need it. Period.
- Premium Calm App – enjoy tranquility with a Calm App subscription for you and up to 4 dependents over the age of 16
- And more! – so many benefits we couldn’t even fit them all here!
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior software engineer evaluating AI coding agents such as Codex and Claude Code. Providing rigorous written and video feedback on engineering quality.
Senior software engineer evaluating AI coding agents for G2i. Assessing engineering judgment, explanations, and trustworthiness across Codex, Claude Code, and Cursor.
Senior engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering judgment, reasoning, and interaction quality.
Forward Deployed Engineer building full - stack AI solutions on AWS and Azure for Huron’s consulting clients. Partnering daily with business users to deliver tested features.
Staff AI Security Engineer securing EQ Bank’s enterprise AI, machine learning, and cloud platforms. Building guardrails, controls, automation, and detection capabilities for Canada’s Challenger Bank.
Staff Software Engineer owning OAuth, authorization, and agent delegation systems. Building governed identity infrastructure for Redpanda’s enterprise AI data platform.
Senior software engineer evaluating AI coding agents for G2i’s engineering team. Assessing reasoning, explanations, and engineering judgment in Codex, Claude Code, and Cursor interactions.
Senior software engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering quality.
Senior engineer evaluating Codex, Claude Code, and Cursor interactions for G2i. Providing rigorous written and video feedback on AI - generated engineering work.