Software Engineer, DevOps AI Rater/Evaluator

Posted last month

Apply Now

Resume Score

Check how well your resume matches this job before you apply.

Sign in to check score

About the role

  • Software Engineering & DevOps professional evaluating AI models for multilingual accuracy and quality. Contribute expert judgment to human-in-the-loop AI evaluation workflows used by leading enterprises.

Responsibilities

  • Evaluate AI outputs related to software engineering, DevOps, and infrastructure topics
  • Perform structured scoring, comparison, classification, and judgment tasks
  • Assess technical correctness, completeness, security implications, and best-practice alignment
  • Identify hallucinations, incorrect code, unsafe recommendations, or misleading system guidance
  • Apply domain-specific engineering and DevOps guidelines consistently across tasks

Requirements

  • Software engineers, site reliability engineers, DevOps engineers, or platform engineers
  • Experience with production systems, CI/CD pipelines, cloud infrastructure, or distributed systems
  • Strong attention to detail and comfort working with structured evaluation criteria
  • Native or professional fluency in one or more supported languages is required
  • English fluency is required for guidelines, feedback, and collaboration

Benefits

  • Contract-based, flexible participation
  • Project-based work with clear expectations and timelines
  • Opportunities for recurring work based on performance and demand
  • Compensation communicated upfront per project or task type

Job type

Contract

Experience level

Mid levelSenior

Salary

Not specified

Degree requirement

Bachelor's Degree

Tech skills

CloudDistributed Systems

Location requirements

RemoteWorldwide

Report this job

Found something wrong with the page? Please let us know by submitting a report below.