Location
toronto, on, Canada
Posted
July 10, 2026
Job Description
Become a pivotal part of Rootly as an AI Quality Engineer, optimizing AI functionalities in a fast-paced environment. Leverage your quality assessment skills to ensure top-tier incident management solutions.
Rootly is seeking an AI Quality Engineer with over five years of experience to enhance our cutting-edge AI features. You will be instrumental in designing prompt-based tests, evaluating AI outputs, and collaborating with product and engineering teams. This role emphasizes critical analytical thinking and strong communication skills for both technical and non-technical stakeholders.
Key Responsibilities:
• Design prompt-based test scenarios for AI features
• Evaluate AI outputs for accuracy and consistency
• Build and maintain evaluation frameworks and libraries
• Identify and document AI failure modes and gaps
• Track and report quality metrics to stakeholders
Requirements:
• 5+ years in QA, AI evaluation, o...
Rootly is seeking an AI Quality Engineer with over five years of experience to enhance our cutting-edge AI features. You will be instrumental in designing prompt-based tests, evaluating AI outputs, and collaborating with product and engineering teams. This role emphasizes critical analytical thinking and strong communication skills for both technical and non-technical stakeholders.
Key Responsibilities:
• Design prompt-based test scenarios for AI features
• Evaluate AI outputs for accuracy and consistency
• Build and maintain evaluation frameworks and libraries
• Identify and document AI failure modes and gaps
• Track and report quality metrics to stakeholders
Requirements:
• 5+ years in QA, AI evaluation, o...