Sr Quality Analyst with AI/ML and LLM testing

Full Time Computer Occupations

Role Summary

This role requires deep understanding of AI fundamentals, hands‑on experience with LLM testing, guardrail design, data sanitization,

and evaluation of agent trajectories. The tester will work closely with AI engineers, data scientists, and product teams to ensure safe, reliable, and high‑quality AI behavior.

5–8+ years in QA, with at least 2–3 years in AI/ML or LLM testing.

Strong understanding of AI fundamentals: embeddings, vector stores, RAG, LLMs, agentic frameworks.

Hands‑on experience testing:

LLM prompts and outputs

Agentic flows (e.g., LangChain, AutoGen, Semantic Kernel, AIP agents)

Model evaluation metrics

Experience writing guardrails, safety rules, and evaluation prompts.

Knowledge of data sanitization, PII detection, and data governance.

Familiarity with pipeline validation and MLOps workflows.

Ability to evaluate model performance usin...