Discrimination Why do government algorithms go wrong so often?

Discrimination Why do government algorithms go wrong so often?

Hairdressing school students practice on a doll.Image: ANP/ANP Why does the government so often go wrong with dubious algorithms? Public administration expert Bram Klievink points to a dangerous cocktail of techno-optimism and austerity. Merijn van Nuland March 2, 2024, 16:11 The cabinet apologised for discrimination in detecting scholarship fraud on Friday. Just like in the benefits affair, a discriminatory algorithm also played a bad role here, even though it has been warned about the risks for years. Professor Bram Klievink, who specializes in digitization, blames it on the zeitgeist. Last week, the committee of inquiry ruled harshly about the tax authorities and the UWV, now government agency Duo also appears to have gone wrong. What do both have in common? “They both go back to the zeitgeist of the previous decade. Around 2010, combating fraud with public money was one of the most important political themes, and at the same time, the task was largely transferred from the state to implementing agencies and municipalities, which also had to do it with less money. Technology was seen as a solution to combine fraud prevention and austerity.” We now know the painful results. Was it a matter of naivety? “On the side of the implementing organizations, sometimes yes. They did not take sufficient account of how the risk analyses used work out in practice. Duo, for example, found it suspicious if a student living away from home lived close to their family home. That is not surprising, but too little attention was paid to the unintended consequences. For example, that students with a migration background in particular often stay closer to their parents, and were therefore more likely to be seen as potential fraudsters.” And on the side of policy makers? “They were much less naive. Indeed, it was a conscious choice to combine fraud prevention with austerity measures and to present ICT as a solution. That remains a source of misery. You can now see that the questionable algorithms are being shut down, but municipalities and implementing agencies still have the same legal tasks in the field of combating fraud. As a result, the State has faced them with an impossible task.” Duo's model had so many snags that it's on hold for now. Should all government fraud algorithms be deactivated? “Some of those algorithms do. I am specifically thinking of the algorithms whose exact effects are not well understood beforehand. They need to be investigated first. In previous cases, algorithms were only disabled when the innocent were the victims, which is far too late. When in doubt, don't start.” You call the period around 2010 a techno-optimistic time. How would you define the current zeitgeist? Klievink thinks for a moment. “Normative-pessimistic. In recent years, the challenges of algorithms and artificial intelligence have become increasingly central. Previously, efficiency and the fight against misuse of public funds prevailed. Now we see more and more clearly that they can conflict with other social norms and values, such as the right to equal treatment and privacy. The naivete is gone. The question now is: how do you find the balance?” The easiest option is probably a total ban on these types of algorithms. “A government without risk models is unworkable, whether you like it or not. The tax authorities receive millions of returns in a month. If someone is shoving millions of euros, an automated system must be able to add a warning flag so that extra control can take place. You just have to be able to substantiate and justify it properly.” Read also:Cabinet says sorry for discriminatory fraud hunting Scholarships The detection of scholarship fraudsters led to indirect discrimination based on educational level and migration background, research shows. The cabinet is now apologising. Harsh report on the benefits affair: “Any of us can still be crushed by the government”. The benefits affair was caused by a toxic interplay between the cabinet, parliament, implementing organizations, justice and journalism. But all the patterns that lead to crushing tens of thousands of lives have led, are still there. Listen also: How imaging made The Hague blind to people and law https://www.trouw.nl/podcasts/hoe-beeldvorming-den-haag-blind-maakte-voor-mens-en-recht~b280eb41/