AI Evaluation & Annotation Specialist reviewing and annotating AI - generated responses while adhering to project guidelines and maintaining quality standards.
About the role
- AI Automation Analyst designing and scaling automation solutions for Instacart's Commercial organization. Leading technical strategy and mentoring teams in AI initiatives.
Responsibilities
- Lead technical strategy and architecture for AI automation across the Commercial organization, identifying high-ROI opportunities and translating business requirements into scalable solutions.
- Design and build production-grade AI tools that understand natural language, provide actionable insights, generate reports, and automate complex workflows such as campaign planning, ticket management, and sales enablement.
- Architect reliable, scalable integrations across internal systems and third-party SaaS (REST/GraphQL APIs, webhooks, OAuth, JWT), establishing best practices for rate limiting, retries, idempotency, and error handling.
- Lead development of multi-step AI agent architectures using frameworks like LangChain, LangGraph, or MCP, with appropriate human-in-the-loop controls and production-ready evaluation.
- Establish and enforce AI safety and reliability standards including guardrails, evaluation frameworks, fallback logic, RBAC, audit trails, and PII protections across all AI systems.
- Design and optimize data pipelines (SQL, Python, dbt) that power AI agents at scale, ensuring reliability, observability, and performance.
- Create frameworks and templates for rapid prototyping and deployment of new automations, enabling faster time-to-value for future projects.
- Drive adoption by building intuitive UIs and self-serve experiences (e.g., Slack apps, low-code internal tools) and creating documentation that makes AI tools accessible to non-technical users.
- Partner with Sales, Partnerships, Product, Engineering, and Data Science leadership to prioritize initiatives, align on roadmaps, and ensure solutions drive measurable business outcomes.
- Mentor and guide other engineers on AI/LLM best practices, prompt engineering, evaluation frameworks, and production deployment patterns.
- Champion DevOps excellence including CI/CD, automated testing, observability, alerting, and incident response.
- Enforce security and compliance best practices across all automations and data workflows.
- Measure and communicate impact through clear metrics tied to business outcomes (time saved, revenue enabled, efficiency gains).
Requirements
- 3–6 years experience in AI, automation, data engineering, or analytics engineering.
- Proven track record of designing and deploying AI/automation solutions that drove measurable business impact.
- Deep proficiency in SQL and Python; extensive experience building and operating production ETL/ELT pipelines.
- Strong experience designing and consuming APIs/webhooks and building reliable services (FastAPI, Flask, or Node.js) at scale.
- Production experience with dbt and modern cloud data warehouses (Snowflake, BigQuery, Databricks).
- Hands-on experience incorporating AI/LLMs into production workflows, including prompt engineering, retrieval strategies, evaluation, and deployment.
- Expert knowledge of Git, CI/CD, automated testing, and event-driven systems (queues, pub/sub, webhooks).
- Experience leading technical initiatives and mentoring other engineers.
- Excellent communication skills: able to translate complex technical concepts for business stakeholders and influence cross-functional teams.
- Experience with at least one data visualization tool (Tableau, Mode, Power BI, Looker, or similar).
- Hands-on experience with workflow automation and low-code development platforms (Zapier, n8n, Gumloop, Superblocks).
Benefits
- Competitive salary
- Flexible work arrangements
- New hire equity grant
- Annual refresh grants
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Forward Deployed Engineer working with internal teams to design and optimize workflows leveraging AI at Knak. Collaborating closely with DevOps to build internal tools and systems.
Drive AI outcomes at Granum to enhance operational efficiency across departments. Collaborate with teams to develop AI - driven solutions and improve workflows with measurable ROI.
AI Outbound Specialist operating AI - powered outbound systems and generating qualified pipeline for event marketing platform Hive. Collaborating with sales and leadership to innovate outreach strategies.
AI Enablement Intern supporting teams at CDW in exploring and applying AI solutions effectively. Hands - on role in a cross - functional internship for students eager to learn about AI.
AI & Automation Delivery Lead delivering AI and automation initiatives for health solutions company Medavie. Leading cross - functional teams to improve health outcomes for Canadians.
AI Intern role focusing on Agentic AI strategy and development at Acosta Group Canada. Collaborate with business and technology stakeholders on innovative AI solutions.
Senior Manager bridging business goals with AI technical execution. Driving AI initiatives and governance frameworks in a leading weather data company.
AI Automation Analyst managing Claude - based workflows for customer experience in a finance tech company. Drive automation initiatives and integrate insights across customer lifecycle.
Evaluate multilingual AI systems for consumer goods and retail. Contribute to improving AI model quality and safety across various languages.