Features and Labels
Data does not arrive ready to learn from. See how two measurements that are each useless on their own combine into one that works.
Data does not arrive ready to learn from. Before any model runs, somebody decides what to measure and what counts as the right answer — and those two decisions matter more than almost anything the model does afterwards.
The two words
Nearly all of supervised machine learning is described by two words, and they are both simpler than they sound.
Something you measure about a thing. A loaf's baking time. A message's word count. A house's floor area. Models only ever see features.
The answer you want, for examples where you already know it. Was this loaf cooked? Was this message spam? What did this house sell for?
Learning means finding a pattern that gets from the features to the label reliably enough to be useful on examples you have not seen.
A feature is a choice
Here is the part beginners rarely get told. Features are not handed down. Somebody chose them, and a different choice would have produced a different model.
Take bread. You have two measurements for every loaf: how many minutes it baked, and how hot the oven was. Neither one is the answer on its own — a loaf can be in a long time at a low temperature, or briefly at a high one, and come out the same.
Inventing a better one
So invent a third measurement out of the two you have. Slide between them and watch both the direction you are measuring in and the error at the best possible cut.
140 loaves, plotted by how long they baked against how hot the oven was. Filled circles came out cooked through; hollow ones were still doughy. The strip underneath shows the same loaves measured by whatever blend you pick — and where the best single cut falls.
- cooked through (86)
- still doughy (54)
Judging by minutes alone. It gets you a long way — but the short-and-hot loaves land right on top of the long-and-cool ones, and no single cut can separate them.
Mistakes at the best cut
199 cooked called doughy + 10 doughy called cooked = 19 of 140 (14%)
The feature you are measuring
100% of time + 0% of temperature
That cut means roughly 25 minutes at 200 °C, or 25 at 235 °C — the invented number does describe something real.
Where labels come from
Time alone is not useless — it gets you most of the way, because longer usually does mean more cooked. Temperature alone is much worse. But a mix of the two roughly halves the error of the better one, and it is a number nobody handed you. You made it up, and it corresponds to something real: roughly thirty minutes at 200 °C.
That is feature engineering, and for most of the history of machine learning it was where the actual work happened. Modern systems can sometimes discover combinations like this themselves, which is a large part of why they are impressive — but somebody still chose what to measure in the first place.
Key takeaways
- A feature is something you measure. A label is the answer you already know. Models only see features.
- Features are chosen, not given — and you are allowed to invent new ones from the ones you have.
- A combination of two mediocre measurements can beat either of them alone.
- An invented feature should still mean something you can say out loud, like “thirty minutes at 200 degrees”.
- Labels come from human judgement. Wrong labels are the one problem better modelling cannot fix.