Software Engineering & DevOps professional evaluating AI models for multilingual accuracy and quality. Contribute expert judgment to human-in-the-loop AI evaluation workflows used by leading enterprises.
Responsibilities
Evaluate AI outputs related to software engineering, DevOps, and infrastructure topics
Perform structured scoring, comparison, classification, and judgment tasks
Assess technical correctness, completeness, security implications, and best-practice alignment
Identify hallucinations, incorrect code, unsafe recommendations, or misleading system guidance
Apply domain-specific engineering and DevOps guidelines consistently across tasks
Requirements
Software engineers, site reliability engineers, DevOps engineers, or platform engineers
Experience with production systems, CI/CD pipelines, cloud infrastructure, or distributed systems
Strong attention to detail and comfort working with structured evaluation criteria
Native or professional fluency in one or more supported languages is required
English fluency is required for guidelines, feedback, and collaboration
Benefits
Contract-based, flexible participation
Project-based work with clear expectations and timelines
Opportunities for recurring work based on performance and demand
Compensation communicated upfront per project or task type
AI Security Control Developer/Site Reliability Engineer for RBC's enterprise AI ecosystem. Design, implement, and validate security controls to protect AI systems with 24/7 reliability.
DevOps Engineering Manager leading a team to improve SDLC at Vancity, Canada's largest Living Wage Employer. Collaborating across teams for reliable delivery of mission - critical systems.
Site Reliability Engineer managing scalable, self - healing systems at Yelp. Collaborating with global teams and ensuring platform reliability across thousands of users.
Principal Site Reliability Engineer responsible for AWS infrastructure and reliability engineering. Collaborating across teams to enhance platform performance and security practices.
Junior/Intermediate DevOps Engineer role in Toronto (Hybrid). Build CI/CD pipelines with GitHub Actions, deploy Java/Spring Boot apps on OpenShift, and collaborate with DevOps teams.
Platform DevOps managing the Enterprise Data and AI Platform across AWS and Kubernetes. Implementing Infrastructure as Code with Terraform and maintaining CI/CD pipelines for secure solutions.