AI Data Evaluators & Video Annotators

We're looking for experienced AI evaluators to join us on a set of active projects. The work is multimodal — images, video, and audio — and accuracy matters more than speed.

What you'll be doing:

Head-to-head evals — comparing two AI-generated images or videos and rating them on instruction following, visual quality, motion/temporal quality, audio sync, and how "AI generated" they look. Also comparing live speech-to-speech responses on naturalness, utility, and delivery.
Detailed video captioning — writing timestamped, comprehensive captions covering scene composition, objects per character, every motion and reaction, camera movement, and all dialogue. The standard is strict: no omissions, no hallucinations. If it's visible, it's annotated. If it's annotated, it's visible.
Grounded annotation tasks — entity tagging, exact-product verification, visible-text transcription, AI-artifact review, camera movement analysis, and audio transcript correction.
Adversarial question writing — designing realistic, web-dependent research questions that stump frontier models, then documenting the "gold" path to the correct answer with reliable sources.

Who we're looking for:

You've done this kind of work before — Handshake AI, Micro1, Outlier, Scale, Mercor, Surge, DataAnnotation, or similar.
You can follow a rubric precisely instead of going with your gut. You know that the prettier image isn't automatically the one that followed instructions, and that "both are fine" isn't a valid tie.
Strong written English and real attention to detail.
Reliable turnaround and consistent availability.

To apply, please send:

A short portfolio or work samples showing similar eval/annotation work you've done, and which platforms you did it on.
Any onboarding assessments you've passed, plus your quality scores or ratings if you have them.

Applications without a portfolio or platform history won't be reviewed. If you've only done basic transcription or generic data entry, this likely isn't the right fit.