Most AI projects do not fail because the model architecture is wrong. They fail because the training data was labeled badly. After 10+ years and 200+ projects at Silk Data, we see the same pattern. Teams pour budget into picking the most sophisticated algorithm. Meanwhile, the labeling step that decides what the model actually learns gets treated as a back-office task. This guide explains why that order is wrong, and what to do about it.
Key takeaways
| Point | What it means for your project |
|---|---|
| Labels define the model | The model only knows what your labels say. Wrong labels = a wrong lesson repeated at every inference. |
| Data prep is 50-65% of the work | In our project economics, modeling is 10-15%. The hidden cost is labeling and cleaning data. |
| Quality beats volume | Two annotators with debiased labels can match eight annotators with simple averaging. Fewer, better raters win. |
| Drift is the silent failure | Definitions shift over months. Without a versioned spec, your model trains on yesterday's policy. |
| Labeling is governance, not data entry | In finance and healthcare under GDPR or the EU AI Act, every label needs a traceable origin. |
What data labeling is and where it fits
Data labeling assigns tags or categories to raw data so a supervised model has a target to learn. Spam or not spam. Tumor or no tumor. Fraudulent or legitimate. Without labels, a supervised model has no reference point.
Labeling and annotation are not the same thing. Labeling assigns a category to a data point. Annotation adds structured metadata - bounding boxes on images, timestamps in audio, entity spans in text. Both feed supervised learning. But they cost different amounts and require different tools.
A few examples we have built or consulted on:
- Classifying online ad events for click prediction with strong class imbalance at 0.1-0.5% CTR.
- Tagging exam submissions to train the classifier inside Plagiarix, our anti-plagiarism product.
- Structuring resumes into candidate fields for AI resume screening.
- Clustering and tagging news articles for our automatic news analysis Chrome extension.
The pattern across these projects is the same. The choice of CatBoost vs. another gradient booster rarely decides the outcome. The label definitions do. This is also where most of the project's hidden risk lives: a wrong algorithm shows itself in a test report; a wrong label definition shows itself months later, after the model has been making confidently wrong calls in production.
How labeling quality shapes model accuracy
Labeling quality is measurable. It directly bounds how well your model can perform. The first metric to track is inter-rater reliability - how often two annotators give the same data point the same label. When they disagree, the model trains on contradictory signals. It learns nothing useful from those cases.
Work on aggregating noisy annotator labels, including this arXiv paper on annotator debiasing, points to a practical takeaway. A small pool of carefully debiased annotators can match the accuracy of much larger pools using naive majority voting. Spending on better guidelines and adjudication often beats spending on more raters.
How we run this in practice:
- Run a pilot on a representative sample - usually 200 to 1000 items.
- Compute agreement with Cohen's kappa or Cronbach's alpha, depending on the task.
- List the items where raters disagree. Read them. Rewrite the guideline.
- Re-run the pilot. Compare agreement.
- Apply debiasing (Random Effects Models, Dawid-Skene) only after guidelines stabilize.
The trade-off is direct. Debiasing recovers signal when you have a few annotators per item. It does not save you when guidelines themselves are wrong. We have seen teams chase elegant aggregation methods on top of a broken spec. The result is precisely consistent nonsense.
As Yuliya Marazenko, who leads our AI implementation work, puts it: the cheapest way to improve a model is usually to rewrite the labeling guideline, not to retrain. We have re-done that step on most projects where a client came to us frustrated by a stalled model.
A practical anchor on cost. Rewriting a labeling guideline typically takes one engineer-week plus two to four hours of an SME. Retraining a stalled model with the same bad labels typically takes one to two engineer-months and produces the same result. We have done this comparison enough times that it has become the first question we ask when a client says "the model isn't working."
Keeping labels consistent at scale
Drift is the silent failure mode in long-running labeling work. Guidelines evolve, annotators rotate, edge cases pile up. A fraud-detection dataset labeled six months ago may already use a different definition of "fraudulent" than your compliance team uses today. The model will keep learning the old policy until you re-label.
The practical fix is to treat the labeling specification like production code: versioned, reviewed, and tied to the dataset snapshot it produced. Pair it with two review layers - an automated checker for obvious rule violations, and a human reviewer for edge cases the spec did not anticipate.
What you gain:
- Less drift, because every annotator works against the same versioned text.
- Faster onboarding - new annotators read a spec instead of inheriting tribal knowledge.
- An audit trail, which matters under GDPR for personal data and under the EU AI Act for high-risk systems.
- Edge cases caught early, before they reach the training set.
What you pay for it:
- Upfront time writing the spec - usually 1 to 3 weeks for a non-trivial domain.
- A reviewer role that does not directly produce labels but blocks bad ones.
- Slower throughput at the start, faster and steadier later.
Break-even is usually around two to three months into the project, depending on dataset size. Past that point, teams without a versioned spec are paying the same cost in re-work and audit prep, just spread out and harder to see.
If you are choosing a vendor or building this in-house, the team running AI consulting and feasibility reviews should be the same team that owns the spec. Splitting strategy from labeling guidelines is how drift starts.
Data-centric vs. model-centric: where to spend
The shift from model-centric to data-centric AI is not marketing. It is what our project economics already reflect. On a typical build, metric definition takes about 10% of effort. Data preparation takes 50-65%. Modeling takes 10-15%. Deployment takes 10-15%. Monitoring is open-ended. The labels live inside the largest slice.
Both approaches have a place. The direct comparison:
| Approach | When it wins | Where it fails | Governance load |
|---|---|---|---|
| Model-centric | Benchmark tasks with clean public data; problems where SOTA architecture moves the metric a lot | Real business data with label noise, drift, or rare classes | Low |
| Data-centric | Most business problems: imbalanced classes, regulated domains, evolving definitions | When data simply is not enough - no amount of labeling fixes 0.1% CTR with 100k rows | High |
The data-centric view has limits. From our click prediction project with 0.1-0.5% CTR: when the positive class is that rare, you need tens of millions of records. Cleaner labels help, but resampling does not replace volume. Engineering judgment still matters.
The rule we use internally: if your problem has more than a few hundred relevant features per record and clean ground truth, model-centric work still pays. If your problem has fifty features, ambiguous labels, and a compliance officer in the room, data-centric work pays many times over. Most enterprise projects sit firmly in the second category.
Regulation pushes the same direction. The EU AI Act sets data governance requirements for high-risk AI systems placed on the EU market, including documentation of training data and labeling choices. GDPR applies wherever personal data sits inside the dataset. FERPA constrains student records in EdTech in the US. None of this asks for a fancier model. All of it asks for labels you can defend.
A working playbook for data labeling
The playbook is short. The discipline to follow it is the hard part.
- Define the metric before the labels. If you cannot say what "good" looks like for the business, no label scheme will save you. We spend ~10% of the project here, and it pays back across the rest.
- Get a subject matter expert on the project. We do not start labeling without one. A radiologist, an underwriter, a compliance officer - someone who can settle hard cases. The animal-farm dataset we worked on contained an entry of "several dozen tons" for one animal. Only a domain expert spots that fast.
- Run a small pilot. 200-1000 items, two to three raters, measure agreement. Read the disagreements out loud. Rewrite the spec.
- Version the spec. Tag each dataset snapshot with the spec version that produced it. When you retrain, you know what changed.
- Use debiasing where it fits. Random Effects Models, Dawid-Skene, MACE - useful when you have multiple raters per item. Useless if the spec is unclear.
- Build a review loop. Automated checks catch rule violations. Humans handle edge cases and feed them back into the spec.
- Monitor in production. Track label distribution and model confidence over time. Drift in either is your early warning.
- Keep traceability. Who labeled it, when, against which spec version. Mandatory under GDPR for personal data and under the EU AI Act for high-risk systems.
Polina Volodina, who advises clients on AI strategy and feasibility, often warns leaders not to skip steps 1 and 2. The pull to jump straight to modeling is strong. The cost of skipping the metric and the SME shows up later. A model ships, then quietly fails on the cases that matter most.
The uncomfortable truth about AI budgets
Across finance, healthcare, marketing, and publishing projects, one pattern is consistent. Organizations overspend on model selection and underspend on labeling. The assumption is that a better algorithm will compensate for noisy data. It will not.
Labeling is not a commodity to send to the lowest bidder. A radiologist who mislabels a scan is not making a typo. They are teaching the model a wrong lesson that will repeat at every inference. On a system seeing 10,000 scans a month, one consistent labeling error is 120,000 wrong predictions a year.
The counter-argument matters too. Not every project needs heavy labeling investment. Some problems are well-served by SQL and a rules engine. We tell clients this when it is true. If a join and a threshold solve the problem, no model is needed, and no labeling either. Our engineering principle is to start with the simplest method that could work. Yuri Svirid, our CEO, frames it bluntly in build-vs-buy conversations. If the business value is small or the rules are stable, ML adds cost without adding accuracy. Spend the labeling budget where the model actually changes a decision.
About Silk Data
Most of our 200+ projects ran into a labeling problem at some point. We learned to design for it from the start. On the EdTech work behind Plagiarix, careful label design for exam submissions cut review time by 90% for our client APT. On our 7-year publishing partnership with Reemers, we built a local LLM over their archives. Their content - and the labels we generated on top of it - never left their perimeter.
If your AI project is stalling, the bottleneck is usually not the model. Start with the labels. That is where we tend to start too.
