Cathy O’Neil
Weapons of Math Destruction
Penguin, 2017
Exam results decided by a statistic model! The First Minister of Scotland has been quick to apologise for the consequences of using a computer statistical model to decide the results for pupils not able to sit exams due to the pandemic. Will England have the same problem? Judicial Reviews are planned.
Cathy O’Neil is a mathematician who reveals the dark side of the world of ‘big data’. She investigates stories of how algorithms can ruin lives. She names such models – weapons of math destruction (WMD). The models built to decide exam results or many other issues, are based on choices made by fallible and biased human beings. Prejudice and mistakes become encoded in models that increasingly impact our lives. The book starts with a story about Sarah Wysocki, a high performing teacher in Washington. In 2011 Sarah received a low score on her impact evaluation – a new scoring system which purported to measure her effectiveness in teaching maths and language skills. Behind the scoring model was the aim of improving education by identifying and sacking poor performing teachers. Sarah was sacked along with 205 others.
A model is a simplification of complex reality and needs feedback loops that help it correct itself. Sarah was victim of a model with too few numbers. A key number was attainment results in tests of pupils in their previous school year. In Sarah’s case it turned out that that number had been manipulated and improved by the previous school to make them look good and help teachers avoid being given low scores. You cannot appeal against a WMD – it is a black box. Those who commission models don’t understand them.
O’Neil is in favour of models that are transparent and continuously updated. Her book uncovers lots of examples where that is not the case, from going to college, borrowing money, getting sentenced, finding or holding a job. Mathematical models now micromanage large parts of our economy. The system and not the individual matters – as Nicola Sturgeon has found to her political cost. The poor are disadvantaged even further, and democracy is threatened by WMDs.
In this well written and researched book O’Neil concludes that WMDs are not morally neutral – they are ‘constructed not just from data but from the choices we make about which data to pay attention to – and which to leave out’. Big Data is not going away. We need to find a way to tame and disarm the negative impact of models in modern life. Perhaps the exam results will awake us all to the consequences of the “computer says no” reality.
Review by Canon Mike D Williams