Location
Remote, Remote, Mexico
Posted
June 03, 2026
Job Description
The Role
We're hiring an AI Evaluation Engineer to own the quality bar for every LLM-powered feature we ship. You'll design, build, and scale the infrastructure that tells us — with evidence — whether a prompt change, model swap, or agent refactor made things better or worse.
This is a high-leverage role. Every customer-facing AI capability at FirstIgnite flows through your evals. You'll work directly with the Head of Engineering and partner closely with product, applied AI, and the full-stack team to establish evaluation as a first-class discipline across the company.
What You'll Do
- Build evaluation infrastructure: Design and maintain eval suites using Promptfoo, LLM-as-judge methodologies, and custom harnesses for features like our expert search system, natural language grants search, and AI SDR agents.
- Define what good means: Partner with product and domain experts to translate fuzzy customer outcomes (does this surface the right p...