Technical lead building safe, evaluable LLM agents and AI infrastructure for OpenLoop’s telehealth platform. Setting architecture, observability, retrieval, and model operations direction.
About the role
- Senior Applied AI Engineer building agent-powered, production AI systems for Newfold Digital’s global web technology platforms. Developing RAG, LLM, and intelligent automation products for millions of customers.
Responsibilities
- Build production AI experiences across Network Solutions, including conversational agents, business and website creation, knowledge and FAQ agents, content experiences, domain discovery, and intelligent automation
- Translate loosely defined product problems into practical AI solutions
- Evaluate different approaches and independently turn the strongest solution into reliable, secure, observable, and cost-efficient production systems
- Design and build production AI services using Python, FastAPI, asynchronous workers, PostgreSQL, Redis, queues, model APIs, and external tools
- Build reliable tool-calling agents and multi-step workflows interacting with internal APIs, MCP tools, business systems, and knowledge sources
- Design event-driven systems with retries, dead-letter handling, idempotency, back pressure, and failure recovery
- Build and improve RAG systems covering ingestion, chunking, embeddings, hybrid retrieval, reranking, metadata filtering, context construction, and citations
- Experiment with models from OpenAI, Anthropic, Google, xAI, and open-weight ecosystems
- Create AI evaluation pipelines using curated datasets, regression tests, retrieval metrics, LLM-as-judge techniques, groundedness checks, and tool-execution evaluation
- Diagnose hallucinations, retrieval failures, incorrect tool usage, agent loops, latency issues, provider failures, and unexpected inference costs
- Use AI coding agents to accelerate design, implementation, testing, debugging, and refactoring
- Turn product goals into testable technical hypotheses and rapidly prototype alternatives
- Define quality, latency, reliability, safety, and cost targets and measure production performance
- Own implementation across APIs, workflows, data, queues, model integration, evaluations, observability, and production support
Requirements
- 5 or more years of professional software engineering experience building production backend, distributed, or cloud-based systems
- Hands-on experience building Applied AI, LLM, RAG, NLP, or agent-based applications, with meaningful production exposure
- Advanced Python skills including FastAPI, asynchronous programming, Pydantic, SQLAlchemy or SQLModel, and production API development
- Strong backend and distributed systems fundamentals including REST APIs, concurrency, background processing, caching, reliability, and production debugging
- Production experience with RabbitMQ, Kafka, Azure Service Bus, or equivalent queue and messaging architectures
- Hands-on experience integrating LLM APIs and building structured output, function calling, tool calling, or agent execution workflows
- Experience building at least one RAG or knowledge-grounded system with measurable quality and latency outcomes
- Strong PostgreSQL and data modeling skills, with experience using Redis, pgvector, vector databases, or hybrid search technologies
- Demonstrated use of AI coding agents such as Cursor, Claude Code, Codex, or equivalent tools as a core part of daily software engineering
- Comfort understanding how modern AI systems behave, fail, and scale in production
- Strong understanding of prompting, structured outputs, tool schemas, context management, model routing, retries, fallbacks, rate limits, and streaming responses
- Ability to design experiments and evaluations using measurable outcomes
- Understanding of retrieval quality, grounding, hallucination mitigation, prompt injection risks, tool authorization, and practical agent guardrails
- Experience with Docker, CI/CD, automated testing, observability, distributed tracing, OpenTelemetry, Langfuse, or similar tooling
- Experience with Semantic Kernel, Microsoft Agent Framework, LangGraph, PydanticAI, or equivalent agent frameworks
- Experience building or consuming Model Context Protocol tools and servers, or working with model gateways such as LiteLLM
- Experience with Azure AI Search, pgvector, Chroma, Kubernetes, Azure, OCI, or similar production AI platform technologies
- Experience building high-scale customer-facing AI products, open-weight model inference, AI safety controls, or automated agent evaluation systems
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior software engineer evaluating AI coding agents such as Codex and Claude Code. Providing rigorous written and video feedback on engineering quality.
Senior software engineer evaluating AI coding agents for G2i. Assessing engineering judgment, explanations, and trustworthiness across Codex, Claude Code, and Cursor.
Senior engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering judgment, reasoning, and interaction quality.
Forward Deployed Engineer building full - stack AI solutions on AWS and Azure for Huron’s consulting clients. Partnering daily with business users to deliver tested features.
Staff AI Security Engineer securing EQ Bank’s enterprise AI, machine learning, and cloud platforms. Building guardrails, controls, automation, and detection capabilities for Canada’s Challenger Bank.
Staff Software Engineer owning OAuth, authorization, and agent delegation systems. Building governed identity infrastructure for Redpanda’s enterprise AI data platform.
Senior software engineer evaluating AI coding agents for G2i’s engineering team. Assessing reasoning, explanations, and engineering judgment in Codex, Claude Code, and Cursor interactions.
Senior software engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering quality.
Senior engineer evaluating Codex, Claude Code, and Cursor interactions for G2i. Providing rigorous written and video feedback on AI - generated engineering work.