Provenance report methodology

The panel

Participants are adults in the United States who joined Populiva and accepted the Participant Agreement. Studies are shown only to participants at verification level L2 or above.

L0 means the participant confirmed their email. L1 means our device and network checks passed: device fingerprint, duplicate-device detection, and VPN, proxy, and data-center network checks. L2 means they also connected a PayPal account that PayPal has verified, with one PayPal account per participant.

Recruitment

Eligible participants see a study on their study list and reserve a place first come, first served. Studies aren't emailed to the whole panel. Each participant can take each study once and at most 20 studies in a rolling week, which spreads studies across the panel.

For the first hour after launch, a study is shown only to trusted participants, but only when enough trusted participants are eligible to fill its open places. Otherwise everyone eligible sees it at once.

Screening and quotas

Screening uses the profile participants keep up to date, such as age, state, sex, education, and political party. A report lists the criteria a study used. Census-matched studies split their places across age, sex, and race or ethnicity in proportion to US adults, using largest-remainder allocation; the report shows how many quota places were filled.

Labs can also limit a study to, or exclude, their own saved participant groups, and require a minimum number of approved studies or a minimum approval rate (approved ÷ reviewed; participants with nothing reviewed yet pass the rate). Participants screened out inside a survey are paid a screen-out amount of at least $0.25 and $10 an hour for the screening time; they're counted separately and never as rejections.

Quality checks

Every submission is recorded with the participant's verification level and trust tier at the time, and with plain-language flags: finished in under a third of the median time, a device shared with another account, a hosting, cloud, or VPN network, no device check, a low trust score, or a new participant. Flags inform researchers' reviews; they never reject anyone on their own.

Trust tiers come from a score built from verification level, approved and rejected studies, attention and consistency checks, and device and network signals. Time on task is measured from reserving a place to submitting the completion code.

In-survey signals

When a researcher adds our snippet (q.js) to their survey pages, it counts time on each page and time visible, how often the participant left the tab, pasted, or copied, how fast they typed, pointer activity, and whether the browser reports that automation software controls it. It never records answers, keystrokes, or screenshots.

An optional hidden instruction is invisible to people and skipped by screen readers but read by automated agents; the snippet reports only whether its word appeared in an answer. Flags from these signals (automation, the hidden instruction, 3 or more pastes, typing over 1,200 characters a minute, or leaving the tab 10 or more times) inform review; they never reject anyone on their own.

Task datasets

For in-platform evaluation tasks, the report adds a task section: the task type, how many raters each item was planned for, how many items reached that number, and how many ratings and skips there were. Item content, dataset names, and rater IDs are never included.

Agreement is Krippendorff's alpha, nominal for choices and labels and interval for rating scales, computed over non-gold, non-skipped ratings from submissions that weren't rejected. Gold accuracy is the share of gold items answered correctly (ratings within one point count as correct). Rater age bands and sex are counted in aggregate; any group under 5 raters is combined so no one can be singled out.

Expert panels

Studies limited to an expert panel are shown only to ID-verified (L3) participants with a current panel membership, and pay at least the panel's floor: $40 an hour for clinicians and licensed professionals, $25 for software engineers. Memberships last one year.

Clinicians are checked against the public NPPES NPI registry: the NPI must belong to an active individual provider with a clinical primary taxonomy, and the name must match what the participant entered. Licensed professionals are checked by Populiva staff on their state board's public lookup. Software engineers qualify by passing a graded task. The report lists the panel, how it's verified, and members' specialties in aggregate, with groups under 5 combined.

Review, exclusions, and refunds

Researchers approve or reject each submission with a stated reason; submissions not reviewed within the study's review period are approved automatically. When we confirm that an account was fraudulent, its submissions are refunded to the lab, and later report versions count them.

Versions and signatures

Researchers can issue a report at any time after launch. Each issue is a new numbered version with its issue time; issued versions can't be changed or deleted.

The report is stored as canonical JSON (keys sorted, no extra whitespace). Its SHA-256 hash is recorded when it's issued, and the exact bytes are signed with Populiva's Ed25519 key, published at populiva.com/provenance/key. The PDF is generated from the JSON.