Data labeling is the process of tagging raw data — a photo, a sentence, an audio clip — with the information a machine learning model needs in order to learn from it: this is a stop sign, this review is negative, this word is a person’s name. Nearly every AI system trained with supervision, from spam filters to self-driving cars to chatbots, learned from data that a human labeled first. Despite decades of automation, that first pass still runs through human judgment more often than most people assume.
What counts as labeling
“Labeling” covers a wide range of tasks. In computer vision, it means drawing a box around a pedestrian in a street photo or tracing the outline of a tumor in a scan. In text, it means tagging a sentence as spam, marking the boundaries of a company name inside a paragraph, or rating how toxic a comment is. In audio, it means transcribing speech or marking where one speaker stops and another starts. For today’s chatbots, it increasingly means something more subjective: given two different AI-generated answers to the same question, a person picks the better one, or writes notes on what a good answer should look like.
This last category sits at the center of reinforcement learning from human feedback (RLHF), the technique companies use to steer a model’s tone and behavior after its initial training. Data labeling and RLHF are related but not the same thing: labeling is the broader, older practice of annotating data before a model is built at all, while RLHF is a specific, later step that shapes a model that already exists. Our explainer on how AI models are actually trained covers where each step fits in the larger pipeline.
Who does the labeling
For most of the 2010s, the cheapest way to get data labeled at scale was to break it into small tasks and pay a crowd of remote workers a few cents each. Amazon Mechanical Turk, launched in 2005 and named after an 18th-century chess-playing automaton that secretly hid a human operator, became the best-known version of this model. Businesses posted “Human Intelligence Tasks” — label this image, transcribe this clip, complete this survey — and anyone could pick them up and get paid per task, typically low hourly rates for simple work and more for specialized skills.
As AI models grew more capable and the labeling work grew more demanding, the market split. General crowd work moved to platforms like Appen and CloudFactory. Purpose-built data labeling companies such as Scale AI and Surge AI began managing large annotation workforces as a service, with quality-control layers on top. And a newer tier of platforms — Mercor and Prolific among them — started recruiting subject-matter experts: doctors, lawyers, and software engineers who label or grade AI outputs in their own field, since teaching a model to write correct legal or medical text takes a different kind of judgment than tagging cat photos.
Why AI still needs it
Modern AI labs use plenty of automation to speed labeling up: models can pre-label a dataset and let a human correct only the mistakes, which is far faster than labeling from scratch. But automation hasn’t removed the need for people. Ground-truth data — the examples a model is ultimately checked against — has to come from somewhere outside the model itself, or errors just compound. Edge cases, ambiguous language, and judgment calls about tone, safety, or factual nuance still need a human to decide what “correct” looks like before a model can be trained or evaluated against it.
That’s also why demand for labeling work hasn’t disappeared even as the tools around it have changed — it has shifted toward harder, more specialized tasks that pay better and require more expertise, while the simplest, lowest-paid crowd tasks are increasingly handled by AI-assisted tools instead of people typing labels one by one.
In the news
That shift is exactly what pushed Amazon to announce Mechanical Turk’s retirement after 21 years: the platform that once defined crowd-sourced data work is being overtaken by specialized labeling firms and expert networks built for the demands of today’s AI models.