Senior Engineer accountable for aircraft/system safety assessments and certifying compliance with safety regulations. Working with an international team to ensure aircraft reliability and safety standards are met.
About the role
- Senior Software Engineer developing AI evaluation frameworks and systems at Lattice. Engaging in building robust AI infrastructure and ensuring performance and reliability of AI products.
Responsibilities
- Design and ship a robust, end-to-end AI evaluation framework, covering offline evals, production tracing, and human-in-the-loop feedback loops, connected across all of Lattice’s AI use cases.
- Define and instrument the metrics that actually matter: agent task completion, hallucination rates, response quality, user engagement, and downstream business outcomes.
- Build and maintain evaluation datasets, test harnesses, and automated scoring pipelines to catch regressions before they ship.
- Identify and surface the drivers of agent quality improvement, giving the team clear signals on where to invest.
- Architect and implement reusable agent infrastructure: multi-turn conversation workflows, recommendation services, LLM DAGs, and standardized agent topology patterns using LangGraph.
- Build and scale RAG pipelines and retrieval infrastructure, including vector store management and retrieval quality optimization.
- Make principled build vs. buy decisions across LLM providers, agent frameworks, and evaluation tooling, balancing capability, cost, latency, and vendor risk.
- Contribute to production AI systems with a strong focus on reliability, observability, and performance, not just prototypes.
- Own projects end-to-end: scope them, drive them to completion, and bring in the right people at the right time.
- Partner with engineering leads and managers to inform technical direction on agent quality and evaluation strategy you’ll be expected to hold intelligent, substantive conversations about methodology, not just implementation.
- Raise the AI engineering bar across the broader team through code review, documentation, and thoughtful technical debate.
Requirements
- 5+ years of professional software engineering experience with significant time spent on production AI/ML systems.
- Deep hands-on experience with LLM-based systems: prompt engineering, RAG pipelines, agent orchestration, evaluation metrics, and model fine-tuning.
- Proven ability to work with data and understand statistics, especially in experiments.
- Proven ability to build and operate agentic AI systems in production: multi-step workflows, multi-agent topologies, and the failure modes that come with them.
- Strong command of AI evaluation: you’ve built eval frameworks before, you know the difference between a good eval and a vanity metric, and you have opinions about it.
- Production-grade Python engineering: clean, maintainable, testable code.
- LangGraph or comparable agent orchestration frameworks. You’ve built real agent workflows with it, not just tutorials.
- LangSmith or comparable LLM observability tooling for tracing, evaluation, and debugging.
- Reads AI papers & blogs regularly and is a trusted source of AI trends.
- Vector databases (Pinecone or similar) and retrieval system design.
- AWS ecosystem or other cloud infrastructure (ex GCP). Comfortable with lambdas, queues, and cloud-native architecture.
- Familiarity with TypeScript is a plus.
Benefits
- Medical insurance
- Dental insurance
- Life, AD&D, and Disability Insurance
- Natural Disaster Support Program
- Wellness Apps
- Paid Parental Leave
- Paid Time off inclusive of holidays and sick time
- Working Remotely Stipend
- One time WFH Office Set-Up Stipend
- Retirement Plan
- Financial Planning
- Learning & Development Budget
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Staff Software Developer responsible for integration architecture in a leading Canadian InsurTech company. Collaborating on Salesforce and other system integrations for streamlined operations.
Transmission Line Lead Engineer overseeing planning and execution of transmission line projects for AtkinsRéalis. Guiding teams, ensuring technical excellence, and mentoring staff.
Senior Modeling & Simulation Engineer at Kepler Communications developing models and simulations for space systems. Collaborating with experts to enhance systems models and decision - making processes.
Senior Engineer building and owning systems for AI agents in healthcare automation. Driving technical direction and improving operational efficiencies for healthcare workflows.
Senior Fullstack Developer on Stay22's Neuro team designing and building robust backend systems. Focused on AI/ML infrastructure for applications that enhance monetization for digital creators.
Intern supporting AI - based software development for marine engineering research on ice - loads prediction. Collaborating with global teams at ABS HET&DRC in St. John's, NL.
Junior Software Developer contributing to full - stack development of web - based applications in a geospatial intelligence company. Collaborating in an agile team with exposure to innovative projects.
Software Developer involved in web application development for a geospatial intelligence company. Focusing on full - stack development and collaborating in a small, agile team environment.
Senior Tag Management/MarTech Engineer responsible for client - side tagging in a remote role at Northern. Collaborating with teams to ensure data governance in a regulated fintech environment.