Olivier Drybooms & Jan Moons 14 Octobre 7 min read
Researchers built a model to tell wolves from huskies and gave it a flaw on purpose: every wolf photo they trained it on had snow behind it, and not one husky photo did. Then they askedpeople who knew machine learning whether they trusted it. More than a third did. Only when they were shown which parts of each photo the model was looking at did almost all of them see the problem: the model had never looked at an animal. It looked at the snow.
Nothing in its answers gave it away. So what would? AI models learn
the wrong patterns because that is what a model is for: it splits two groups of photos by whatever separates them most cleanly, and snow separated them cleanly. The researchers caught it by making the modelshow its working. The cheaper check comes earlier, in where the data came from.
It learned the background. Snow was the most consistent difference
between the two piles, so snow is the rule it found, and the animals were beside the point. None of that is a malfunction, which is the uncomfortable part.
That is the shape to look for. The thing you wanted to detect happened to travel with something in the background, and in the case of the wolf, the background was the easier signal to pick up.
Because from where the model sits there is no decision to notice. A
model detects what is statistically consistent and it never asks why a pattern exists, so a pattern holding across every example in the set is precisely what it is hunting for.
AI looks a great deal like understanding from the outside, and that is what makes this so hard to keep hold of. Image recognition feels like seeing, and it’s matching patterns of pixels against labels picked up from millions of earlier examples. Spotify doesn’t know your taste in music. It knows that people who play what you play also played this, which is a less flattering thing to be told and a more accurate one. Your spam filter doesn’t recognise bad intentions. It recognises subject lines,sender metadata and writing style that resemble spam it has seen before.
None of them reads meaning. Garbage in, garbage out is the older phrase for it, and it hasn’t been improved on.
At collection, nearly always, and long before anyone opens a modelling
tool. The snow didn’t arrive during training. It arrived with the photographs, in somebody’s choice of which pictures went into the pile. Training data comes from three places, and which one you’re holding tells you what to be suspicious of.- Manually created. Typed in by people. Every entry has already passed through somebody’s judgement about what was worth writing down and how to word it.
- Automatically collected. Captured in the background by systems, which is what makes it scale so well. It records what the system was built to record, and nothing else.
- Sensing the environment. Raw and rich, and shaped by where the sensor sits, what it points at and when it happened to be running.
