About the role

AI safety analyst evaluating and governing Generative and Agentic AI systems. Supporting responsible healthcare AI deployment at Trillium Health Partners teaching hospitals.

Responsibilities

Lead business analysis for AI systems across the lifecycle with clinical, operational, and technical stakeholders
Define problems, intended use, and success measures
Map current and future-state workflows
Create use cases, user stories, business cases, and business/technical requirements
Design and execute pre-deployment evaluation and validation of Generative and Agentic AI systems
Assess accuracy, grounding, hallucination, bias and fairness, usability, workflow fit, and human-in-the-loop review points
Support and lead project planning and product management activities for AI solutions
Manage AI lifecycle activities including performance metrics, dashboards, drift and incident tracking, user feedback, and outcome measures
Analyze results and prepare reports and recommendations for leadership and AI governance bodies
Participate in project status meetings and project management activities
Prepare presentations, briefing materials, and education content for clinical, operational, executive, and community-oriented audiences
Document AI systems and design decisions through system fact sheets, risk and impact assessments, evaluation reports, and standard operating procedures
Contribute to AI safety policies and procedures supporting PHIPA, responsible AI, and applicable regulatory requirements
Report to the Director, AI Systems, Safety and Effectiveness

Master's degree in Health Informatics (MHI), Management of Innovation (MMI), or equivalent required
Strong business analysis skills, including requirements elicitation, workflow mapping, use case and user story development, and Agile methodology
Technical understanding of Generative AI, Agentic AI, large language models, retrieval-augmented generation (RAG), and prompt engineering
Analytical and quantitative skills to design evaluation plans, define metrics, analyze performance data, and interpret results
Broad project management skills, including planning, tracking deliverables, and managing risks and issues
Excellent oral and written communication skills, including translating complex technical findings into clear reports and presentations using tools such as MS PowerPoint
Critical thinking, attention to detail, and commitment to patient safety, equity, privacy, and human-centred design
Must be eligible to work in Canada
Must be fully vaccinated for COVID-19 and have other vaccinations required by the Public Hospitals Act
Preferred: certification in AI Product Management, Generative AI, and/or AI evaluation
Preferred: training in AI governance, Agile methodology, health data privacy (PHIPA), and quality improvement or patient safety methods
Preferred: proficiency with Excel, Power BI, SQL, or Python

Accommodation throughout the recruitment and selection process for applicants with disabilities
Equal opportunity and respectful, safe, and supportive workplace
Vaccination support/requirement as a condition of employment
Teaching and learning environment
Meaningful human involvement in hiring decisions