Senior actuarial advisor commercializing predictive modeling and advanced analytics for Desjardins insurance pricing. Deploying innovative capabilities across Personal and Commercial Insurance.
About the role
- Senior Data Scientist defining rigorous evaluation systems for AI models and agents. Improving quality and safe rollout at Alpaca, a global brokerage-infrastructure company.
Responsibilities
- Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents
- Build repeatable evaluation loops to track quality over time and catch regressions before release
- Partner with engineering and analytics engineering to operationalize evaluation harnesses
- Translate evaluation results into actionable recommendations for system improvements
- Establish evaluation guidelines, documentation, and review practices
- Mentor and align teams on evaluation best practices and measurable AI quality
- Partner with Product, Engineering, Analytics Engineering, and business stakeholders to define quality standards and drive iteration
- Own the quality bar independently of the teams that build and optimize the systems
Requirements
- Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation)
- Strong statistical and ML foundation, including sample sizing, confidence intervals, significance, handling non-determinism, and validating automated graders against human ground truth
- Proficiency in Python and SQL, with experience evaluating models in production environments
- Strong judgment in defining quality metrics and ground truth for ambiguous outputs
- Excellent communication and cross-functional collaboration skills
- Strong problem-solving ability in fast-paced, greenfield environments
- 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation
- Quantitative degree is a plus; equivalent industry experience is equally welcome
- Nice to have: hands-on LLM/agent evaluation in production, eval harnesses, LLM-as-judge calibration, and CI regression gates
- Nice to have: experience evaluating text-to-SQL, analytics agents, or systems where correctness is verifiable against data
- Nice to have: background in fintech, brokerage, or domains with business or risk consequences
- Nice to have: fluency with AI tools in research and engineering workflows
Benefits
- Competitive Salary & Stock Options
- Health Benefits
- New Hire Home-Office Setup: One-time USD $500
- Monthly Stipend: USD $150 per month via a Brex Card
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Lead Data Scientist building production ML for Life360’s family safety and tracking products. Driving revenue experimentation, personalization, advertising, and subscription growth.
Data Scientist developing network - based financial - crime detection for TD Bank. Building models, visualizations, and decision - support tools to prioritize high - risk customer connections.
Data Scientist developing machine - learning models for The Athletic’s sports media platform. Optimizing subscriptions, workflows, experiments, and personalized article recommendations.
Data Scientist at Desjardins, a financial services cooperative, developing advanced analytics, machine learning and AI models. Advising stakeholders and supporting data - driven business decisions.
Data Scientist building IFRS 9, behavioural, and predictive credit - risk models for Financeit’s Canadian point - of - sale financing platform. Automating analytics and translating borrower data into actionable risk insights.
Scientifique de données principal optimisant acquisition, fidélisation et monétisation chez Shakepay, plateforme canadienne de services financiers Bitcoin. Pilotage de l'expérimentation, de l'analyse statistique et des décisions de croissance.
Senior Data Scientist developing machine learning and LLM solutions for Qventus, transforming healthcare operations. Building production models, experiments, and data pipelines that improve patient outcomes.
Data Scientist developing statistical and machine learning algorithms to detect bots and fraud. Supporting HUMAN Security’s platform that protects enterprises and internet platforms from cybercrime.
Data Manager leading clinical - trial database, validation, reconciliation, and lock activities for ClinChoice, a global full - service CRO. Hybrid permanent role in Mississauga, Ontario.