Populiva for AI
Human evaluation data you can prove.
Run preference, rating, rewrite and labeling tasks with verified, fairly paid US raters, and get a signed provenance report with every dataset.
Evaluation tasks, built in
Upload items as JSONL or CSV and choose a task type. Raters work through them one at a time in Populiva, in English or Spanish, with keyboard shortcuts and an accessible interface.
- Pairwise preference A or B, with optional ties and a rationale. We randomize which side each response appears on.
- Rubric rating Score each response on up to 10 criteria with a 5- or 7-point scale.
- Rewrite Raters edit a response to improve it, for supervised fine-tuning data.
- Classification and labeling One label or several, from your own list.
Explain why the sky looks blue to a ten-year-old.
Response 1
Sunlight has every color in it. Blue light bounces around the air the most, so blue reaches your eyes from all over the sky.
Response 2
Rayleigh scattering is inversely proportional to the fourth power of wavelength, which…
1 Response 1 2 Response 2 3 About the same
- Krippendorff's α
- 0.71
- gold accuracy
- 94%
- raters per item
- 3 / 3
Every item rated the way you planned
Choose how many raters see each item. Items with the fewest ratings go out first, no rater sees an item twice, and a gold item with a known answer is mixed into every session.
Turn a task into a qualification test: raters answer your gold items, and those who pass join a group you can limit later tasks to.
Evidence on every rating
Each rating comes with time on the item, pastes and tab switches (counts only), and the rater's verification level. Sessions are flagged for failed gold items, implausibly fast answers, or the same answer every time. Flags inform your review; they never reject anyone on their own.
Your results page shows coverage, Krippendorff's alpha for each dimension, and accuracy on gold items. Download everything as JSONL or CSV, ready for reward-model, DPO or fine-tuning pipelines.
Verified expert panels
- US clinicians NPI checked against the public NPPES registry: an active individual provider in a clinical specialty, with a matching name. At least $40 an hour.
- Licensed professionals Lawyers, accountants, engineers and other state-licensed professionals, checked by our staff on the licensing board's public lookup. At least $40 an hour.
- Software engineers Qualified by passing a graded task. At least $25 an hour.
- ID-verified Every expert study also requires a document and selfie ID check, and memberships are rechecked each year.
A signed report for every dataset
Issue a provenance report at any time: how raters were recruited and screened, verification levels, expert panels, coverage, agreement, gold accuracy, exclusions, and rater age and sex in aggregate, with small groups combined. It never includes your items, your dataset's name, or rater identities.
Each report is canonical JSON signed with our published Ed25519 key, so anyone can check it hasn't changed.
- participants
- 300
- payout-verified (L2+)
- 100%
- ID-verified (L3)
- 22%
- device + network checks passed
- 98.7%
- flagged after review
- 4 → refunded
- screening
- census-matched, US
- methodology
- 2026.5
An API and MCP server for your pipelines and agents
Create datasets, launch tasks, page through ratings and fetch agreement statistics over the API, and get signed webhooks when items and tasks complete or a rater is flagged.
AI agents can do the same through our MCP server. Each token has daily and per-call spending caps that only people in your lab can change, and launching needs an idempotency key, so a retried call never spends twice.
Fair to the people doing the work
Raters opt in to AI work separately and see a notice on every AI task. AI tasks pay at least $15 an hour. Sensitive content reaches only raters who opted into every category it contains, with a warning first, and any item can be skipped without penalty.
Your data stays yours
Items and ratings are encrypted, used only for your organization, and deleted 90 days after the task closes. Raters see items only inside your task. Participants appear only as pseudonymous IDs.
Human evaluation data you can prove.