Software Developer in Test II ensuring automated quality for Warner Bros. Discovery’s CNN Digital products. Building test suites and supporting reliable releases across web, mobile, OTT, and backend platforms.
About the role
- Software Engineer, AI at Calabrio designing AI-powered products and platforms. Focused on LLM applications, agentic AI systems, and applied machine learning.
Responsibilities
- Design AI systems
- Build end-to-end AI solutions using machine learning, deep learning, NLP, and generative AI technologies.
- Develop LLM-powered applications
- Create applications using foundation models, prompt engineering, retrieval-augmented generation, structured outputs, function/tool calling, and agent workflows.
- Build agentic AI solutions
- Design and implement AI agents that can plan, reason through multi-step tasks, interact with external tools and APIs, retrieve relevant context, and execute actions within controlled business processes.
- Develop multi-agent and orchestration workflows
- Create orchestrated AI systems where multiple agents or components collaborate to solve complex tasks, with clear control flow, observability, and fallback handling.
- Productionize models and AI agents
- Deploy, monitor, and maintain AI/ML models and agentic systems in production environments with strong reliability, performance, and safety standards.
- Build data and inference pipelines
- Develop pipelines for ingestion, preprocessing, vector search, model inference, agent memory/context management.
- Improve quality and evaluation
- Define offline and online evaluation frameworks for model quality, latency, safety, task completion, agent reliability, and business outcomes.
- Optimize performance and cost
- Improve model selection, prompt efficiency, agent orchestration, latency, throughput, caching, token usage, and serving efficiency.
- Ensure governance and safety
- Apply best practices for security, privacy, responsible AI, model risk controls, guardrails, agent permissions, compliance, and human-in-the-loop review where needed.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field required. Master’s degree preferred.
- 3+ years of end-to-end experience training, evaluating, testing, deploying, and monitoring machine learning models in production.
- Hands-on experience building applications with LLMs, prompt engineering, retrieval-augmented generation, structured outputs, and model evaluation.
- Experience with frameworks or platforms for LLM and agent orchestration, such as LangChain, LangGraph, Strands AI, or equivalent architectures.
- Experience designing or building AI agents that use planning, memory, tool calling, workflow orchestration, agent-to-agent and external system integration to complete multi-step tasks.
- Strong experience with Python and backend frameworks such as Flask or Django for building production APIs and AI services.
- Strong understanding of machine learning fundamentals and practical experience with NLP tasks such as text classification, NER, clustering, topic modeling, semantic search, and conversational AI.
- Experience with fine-tuning LLMs and transformer-based models such as BERT, RoBERTa, ALBERT, and a solid understanding of tokenizers, embeddings, pre-trained models, and adaptation techniques.
- Experience with SQL and NoSQL databases, vector databases or embedding stores, and data pipelines for AI applications.
- Experience with model serving, observability, evaluation, error analysis, prompt/version management, and monitoring of AI systems in production.
- Familiarity with Linux systems and standard software engineering practices including testing, CI/CD, APIs, and version control.
- Nice to have:
- Experience with AWS, Azure, or GCP
- Experience with Docker and Kubernetes
- Experience with ETL and Data Engineering projects
- Experience with PostgreSQL, Snowflake, or MongoDB
- Experience with Kubeflow, or Airflow
Benefits
- Global team recognized for their passion and innovation
- Innovative product culture and project exposure
- Training and development from industry-leading experts
- Cutting edge benefit programs that include: 401(k) with company matching; medical, dental, and vision insurance; disability and life insurance; flexible PTO; paid holidays and parental leave; tuition reimbursement and more
- We offer market competitive pay and benefits based upon the candidate’s skills, experience, and qualifications.
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Software Developer in Test II ensuring automated quality releases for Warner Bros. Discovery’s CNN Digital products. Building test suites across the software development lifecycle.
Full Stack Engineer building Yelp web features that drive app installs and user acquisition. Architecting full - stack solutions, running experiments, and mentoring engineers remotely across Canada.
Senior fullstack developer building .NET/C# and TypeScript legal - document software. Supporting SOVRA’s public procurement platform used by government agencies across North America.
Développeur full - stack senior pour la plateforme de documents juridiques de SOVRA. Conception d’API .NET, interfaces TypeScript et fonctionnalités assistées par l’IA.
Senior backend developer building Java and Spring Boot services powering Warner Bros. Discovery streaming subscriptions, commerce, and partner integrations. Operating scalable event - driven systems across AWS.
Guidewire tech lead driving Desjardins’ property and casualty insurance transformation. Leading scalable platform solutions, development practices and technical delivery.
Full - stack developer building Equisoft’s Kotlin - based insurance policy administration platform. Developing secure web features with React, Java, SQL, and microservices in an Agile team.
Senior Principal Architect shaping Rocket Software’s DataEdge data integration, cloud, and AI platform strategy. Driving cross - product architecture, shared services, and agentic systems.
Senior Full - Stack Developer building Laravel and Nuxt/Vue products for ScholarshipOwl’s AI - powered scholarship marketplace. Developing widgets, APIs, integrations, and LLM - driven education funding features.