AI Agent Evaluation Engineer
Cresta is hiring for AI Agent Evaluation Engineer in Canada (Remote); United States (Remote). Remote, Mid level. First seen November 13, 2026; open as of November 18, 2026. Jobs taken straight from company career pages — no aggregator in between.
Save to applyFree forever. No commitment, no card required.
- Company
- Cresta
- Where
- Canada (Remote); United States (Remote)
- Arrangement
- Remote
- Level
- Mid
- First seen
- November 13, 2026
- Listed via
- Greenhouse
- Open for
- 5 days, as of November 18, 2026
Applying to this one?
See how well you fit, before you apply
Save it to your board. Add your résumé once, and Destava shows where you are strong, what is missing and how to pitch yourself, then tailors your résumé to this posting.
- Found it
- Save it
- See your fit
Free forever. No commitment, no card required.
What the posting says
What You'll Do : Evaluation Design & Quality Definition Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition). Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely; run calibration sessions and measure inter-rater agreement. Classify and prioritize failure modes by severity and frequency so engineering and product work on what…
Read more and apply on Cresta's site
Questions about this job
When was this AI Agent Evaluation Engineer job at Cresta posted?
November 13, 2026
Destava first saw it on November 13, 2026, on Cresta's own careers page (Greenhouse).
Is Cresta still accepting applications for it?
Open as of November 18, 2026
It was open on Cresta's own site when Destava last read it, on November 18, 2026, 5 days after we first saw it. When the company takes it down, this page says the job has closed.
How do I apply for AI Agent Evaluation Engineer at Cresta?
On Cresta's own site
On Cresta's own site, at job-boards.greenhouse.io: open the posting. Destava does not take the application. Save the job to your Destava board to track it, see how well you fit and tailor your résumé to it.
Is this Cresta job posting legit?
Listed on Cresta's own careers page
Destava read it on Cresta's own careers page at job-boards.greenhouse.io, run on Greenhouse, not on a job board. First seen November 13, 2026. Apply on the company's own site, and never pay to apply for a job.
What other jobs is Cresta hiring for?
89 open postings
Cresta has 89 open postings on its own site: see all jobs at Cresta.
How many software quality assurance analysts and testers jobs are open?
10,064 open
10,064 open, 2,744 of them new this week, at 2,842 companies: see them all.
More jobs like this
Software Test Manager – Automated Productivity Solutions (APS)
Save to applySenior QA Engineer CUSS
Save to applyAI TEST
Save to applyStaff Quality Engineer
Save to apply
See all 10,064 Software Quality Assurance Analysts and Testers jobs
A search agent finds the jobs this list misses: the ones that mean the same without the same words. Have Destava watch for ai agent evaluation engineer jobs in Canada (Remote); United States (Remote), new matches every morning on your own account.
With Destava: Save to apply · Tailor my résumé · Write a cover letter
This page includes information from the O*NET 31.0 Database by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA). Used under the CC BY 4.0 license. O*NET® is a trademark of USDOL/ETA.