Data annotation: how to buy labeled data without wasting the budget

Data annotation is the work of marking up raw material so a model can learn from it: drawing boxes around objects in images, tagging intent in support tickets, marking which of two model answers is better. It is the least glamorous line in an AI budget and one of the few that decides whether the system works. Teams that skip the question end up debugging a model when the problem sits in the labels.

Most companies buying this are not building foundation models. They are fine-tuning something, or evaluating outputs, or running a workflow where a person checks what the machine produced. All three need the same thing, which is people doing careful work against a clear standard.

What the work covers

Labeling raw data is the part everyone pictures. Bounding boxes, polygons, segmentation masks, keypoints for images and video. Classification and entity tagging for text. Transcription and speaker labels for audio.

Evaluation is the faster-growing half. Someone has to judge whether a model’s answer is correct, safe, and useful, and those judgments become the training signal for the next version. This work needs domain knowledge more than volume: a nurse rating clinical summaries, a lawyer rating contract extractions.

Edge case handling sits between the two. Every deployed system produces inputs nobody planned for, and those cases need a human decision plus a rule for next time. Then monitoring, where people review a sample of live output to catch drift before customers do.

Why the labels decide the outcome

A model learns the standard in the labels, including the parts that are wrong. Inconsistent guidelines produce a model that is confidently inconsistent, and the failure appears in production rather than in testing, because the test set carries the same flaws as the training set.

The wider record on enterprise AI supports the point. MIT’s Project NANDA reported in its July 2025 GenAI Divide study that around 95 percent of enterprise generative AI pilots showed no measurable impact on profit and loss, with the authors pointing at integration and learning rather than model quality as the reason. Data preparation is where a lot of that gap lives, because a pilot built on hastily labeled examples looks fine in a demo and falls apart on real inputs.

The practical version: write the annotation guidelines before commissioning the work, run a small pilot batch, measure agreement between two annotators on the same items, and fix the guidelines until they agree. That loop costs a few days and saves months.

Where the humans go

Human in the loop describes any workflow where a person sits inside the automated path rather than beside it. The useful design question is not whether to include people, it is which decisions they see.

Confidence routing is the common pattern. The model handles what it is sure about, and anything below a threshold goes to a reviewer. The threshold is a business decision: too high and you pay for review you did not need, too low and errors reach customers.

Approval gates work differently, sending everything in a category to a person regardless of confidence, which suits decisions with legal or safety weight. Sampling sits behind both as quality control on the automated share.

Documenting who reviews what has become a compliance question as well as an operational one. The NIST AI Risk Management Framework remains the reference most US companies use for this, and it is voluntary, which means auditors and enterprise buyers treat adherence as evidence rather than as a checkbox. It is also being revised under the federal AI Action Plan, so anyone citing it in a policy document should reference the version they used.

Choosing a vendor

Three routes exist. Platform tools sell software and leave the workforce to you. Marketplaces sell access to a crowd. Managed services sell a trained team plus quality control, which is what most companies need and do not know to ask for.

Humans in the Loop is a reasonable example of the managed model: a Sofia-based social enterprise, operating since 2017, offering dataset collection, annotation, model verification, and live monitoring, with a two-layer review process and work delivered in whatever format a team already uses. It is a certified B Corp that employs annotators from conflict-affected and displaced communities and routes a share of profit into training programmes. Larger competitors will quote lower unit prices on high-volume, low-complexity work.

Labour conditions in this supply chain now carry procurement weight. Fairwork’s research on cloudwork platforms has reported average pay near $2.15 an hour on some of them, with roughly a quarter of working time unpaid, and in June 2026 the International Labour Conference adopted the Decent Work in the Platform Economy Convention, the first international labour standard covering this kind of work. Enterprise customers have started asking suppliers where their annotation happens and under what terms. Cheap labels sourced through an opaque chain are a diligence problem waiting to surface.

Any team weighing whether to run this internally should price it honestly against bringing in an outside AI partner, because the coordination cost of an in-house annotation team surprises people more than the wage bill does.

Data annotation buying checklist

• Write the guidelines first, with examples of correct and incorrect labels, before asking for quotes
• Run a paid pilot batch with two or three vendors on the same sample and compare output, not sales decks
• Measure inter-annotator agreement on that batch and treat a low score as a guidelines problem
• Ask who the annotators are, where they work, and how they are paid
• Confirm where your data is stored and processed, and whether any of it trains the vendor’s own models
• Require a documented quality process with a second review layer, not a single pass
• Check the vendor works in your format and tooling rather than exporting a conversion problem to you
• Agree how edge cases get escalated and how guideline changes get applied to work already done
• Price per unit and per hour, then compare both against the in-house alternative including management time
• Keep ownership of the labels and the guidelines in the contract

Data annotation FAQ

What is data annotation?

Data annotation is the process of adding labels to raw data so a machine learning model can learn from it or be measured against it. That includes drawing boxes around objects in images, tagging text, transcribing audio, and rating model outputs for accuracy or safety.

How much does data annotation cost?

Pricing is per unit for simple work and per hour for work needing judgment or domain expertise, and the spread between those two is wide. Complexity, quality requirements, and turnaround drive the number more than volume does. The cost worth comparing is not vendor against vendor but vendor against the internal team you would otherwise assemble and manage.

Can AI label its own training data?

Partly. Model-assisted labeling does a first pass that people correct, which cuts cost on straightforward items. Using model output as ground truth without review compounds the model’s existing errors, so the human review step stays.

What does human in the loop mean?

It means a person sits inside the automated workflow, reviewing, correcting, or approving what the system produces before it takes effect. In practice that is usually confidence-based routing, where low-certainty cases go to a reviewer, combined with sampling of the rest for quality control.