Technical lead building safe, evaluable LLM agents and AI infrastructure for OpenLoop’s telehealth platform. Setting architecture, observability, retrieval, and model operations direction.
About the role
- Senior AI Engineer developing and launching AI products for Valsoft Corporation. Collaborating on AI initiative across portfolio companies with hands-on software engineering tasks.
Responsibilities
- Drive the development and launch of new AI-powered products — from concept and prototype to market-ready release — with a focus on rapid iteration and measurable impact.
- Operate with a startup mindset: bias toward action, ownership across the full stack, and the ability to turn ambiguity into execution.
- Build scalable systems that not only attract new customers but also deepen adoption and upsell opportunities across our existing base.
- Build RESTful APIs, microservices, front and back-end components using a polyglot approach (Python, JavaScript frameworks)
- Develop context strategies (e.g., embeddings, retrieval augmentation, windowing) for reliability, security, and cost efficiency
- Apply evaluation frameworks (Promptfoo, OpenAI Evals, LangSmith) to systematically test, compare, and validate LLM outputs
- Build and tune AI models, assistants, agentic workflows, vector databases.
- Work closely with senior engineers and product owners to clarify requirements and deliver solutions that add measurable business value
- Work closely with other teams to design interfaces and get them implemented for consumption by AI tooling and products.
- Mentor engineers on prompt design, evaluation best practices, and context strategies
- Participate in sprint planning, code reviews, and backlog grooming.
- Integrate regression and adversarial testing into CI/CD pipelines to prevent prompt drift
- Define automated evaluation suites for LLM applications (truthfulness, consistency, compliance)
- Support efforts to reduce technical debt and improve maintainability.
- Experiment with and apply Generative AI and ML tools where they can accelerate development or improve user experience
- Create, optimize, and manage complex prompt strategies to maximize AI model performance
- Develop reusable prompt templates for consistent outputs across multiple use cases
- Design and optimize NLQ interfaces that let users query enterprise data (SQL/NoSQL, APIs, vector stores) in natural language, applying context engineering and evaluation frameworks to ensure accuracy, security, and business relevance.
- Troubleshoot and optimize performance, scalability, and data-handling issues
- Adapt to different technical environments across the portfolio.
Requirements
- 5–8 years of professional software engineering experience with Python and JavaScript frameworks
- Hands-on expertise with Generative AI and ML tools (Promptfoo, OpenAI Evals, LangSmith, LangChain, AWS Bedrock, Azure AI, etc.) is a must
- Familiarity with vector databases (e.g., PostgreSQL/pgvector, Pinecone, Weaviate, Milvus) and RAG (Retrieval-Augmented Generation) pipelines is preferred
- Deep and recent experience with Generative AI or machine learning development tools (Claude Code, Github Copilot, AWS Bedrock, OpenAI, LangChain, Azure AI, etc.) is preferred
- Proven ability to design and implement NLQ solutions enabling users to query structured/unstructured data sources conversationally, with focus on accuracy, performance, and security.
- Familiarity with Git-based workflows and CI/CD pipelines.
- Collaborative mindset with strong communication skills
- Comfortable working in a remote, distributed environment
- Willingness to learn, experiment, and take initiative.
Benefits
- Competitive compensation package
- Performance-based bonus potential
- Comprehensive benefits package.
- Hybrid or Fully remote work environment
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior software engineer evaluating AI coding agents such as Codex and Claude Code. Providing rigorous written and video feedback on engineering quality.
Senior software engineer evaluating AI coding agents for G2i. Assessing engineering judgment, explanations, and trustworthiness across Codex, Claude Code, and Cursor.
Senior engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering judgment, reasoning, and interaction quality.
Forward Deployed Engineer building full - stack AI solutions on AWS and Azure for Huron’s consulting clients. Partnering daily with business users to deliver tested features.
Staff AI Security Engineer securing EQ Bank’s enterprise AI, machine learning, and cloud platforms. Building guardrails, controls, automation, and detection capabilities for Canada’s Challenger Bank.
Staff Software Engineer owning OAuth, authorization, and agent delegation systems. Building governed identity infrastructure for Redpanda’s enterprise AI data platform.
Senior software engineer evaluating AI coding agents for G2i’s engineering team. Assessing reasoning, explanations, and engineering judgment in Codex, Claude Code, and Cursor interactions.
Senior software engineer evaluating AI coding agents such as Codex, Claude Code, and Cursor. Providing rigorous written and video feedback on engineering quality.
Senior engineer evaluating Codex, Claude Code, and Cursor interactions for G2i. Providing rigorous written and video feedback on AI - generated engineering work.