Serbia, a laboratory for algorithmic discrimination
The use of algorithms to optimise the allocation of social benefits is commonplace in Europe. In Serbia, data collected from the population is used to target certain groups, including the Roma. But voices are emerging in civil society to denounce this system.
When the Social Card law came into force in Serbia in March 2021, many expressed concerns about its possible impact on people in need. Since then, over 22,000 people have stopped receiving social benefits. The concerns were clearly well founded.
The law provides for the introduction of a register for the collection and processing of personal data of those who receive some form of financial social assistance, in order to assess their overall economic situation and possibly redistribute the resources allocated to this item of the state budget. To determine socio-economic status, the law prescribes the collection of an extensive list of personal data, taken from various sources and centralised in a single register which, through an automated evaluation process, establishes whether and at what level one is eligible to continue receiving benefits.
This system has aroused two concerns. The first is related to the already problematic situation of social assistance in Serbia, which needs a reform that includes more people by giving them higher subsidies, while the law seems aimed at redistributing existing resources and excluding some of the current beneficiaries, rather than including new ones.
The second concerns the violation of the rules on the collection and processing of personal data as well as those that should protect citizens from decisions made against them by automated systems, which risk reproducing and accentuating discrimination that already affects vulnerable subjects, such as for example the Roma community.
In November 2022, a network of associations that goes by the name of ESCR-Net, and which includes Amnesty International, A 11 – Initiative for Economic and Social Rights and the European Roma Rights Center, has presented to the Serbian Constitutional Court a legal opinion in which it lists the critical issues of the law.
A non-inclusive system
According to data collected by A 11 , in February 2022 (before the Social Card went into operation) there were 211,266 people receiving social assistance, while by August the number had dropped to 189,036 – a drop by 10.5 per cent.
The system has only come into effect in some cities, explained A11 coordinator Danilo Ćurčić, because many social work centres do not have the technology and expertise to connect to the central register. There is therefore also an issue related to the cost of implementing and maintaining this system, which makes it unsustainable, according to Ćurčić.
In 11 of the 15 cities from which A11 obtained data, there has been a drop in the number of beneficiaries, in some cases with percentages far above the national figure (Ada -28.7%, Irig -28.8%). In other cases, instead, the number has increased, sometimes significantly (Leskovac +17.1%). "Since there is no transparency on the functioning of the system, it is not clear why some social work centres have increased the number of beneficiaries", Curcic explains. "In any case, the increase cannot be due to the Social Card, since it does not give centres the opportunity to identify those who are still not receiving assistance and possibly include them in the coverage".
Targeting the Roma
As Radio Slobodna Evropa explains, social assistance for single individuals currently amounts to 10,385 dinars per month (about 88 Euros), a figure which is also lower than the threshold of 12,695 dinars set by the government itself as a minimum for getting out of poverty in 2020. Furthermore, again according to government data, in the same year 6.9% of the Serbian population fell into conditions of absolute poverty: just over 475,000 people, i.e. two and a half times the number of people who receive social assistance today.
One of the elements that can lead to exclusion from social assistance is reselling materials such as plastic, cardboard, and glass for separate collection, practiced by many Roma people, who are also the poorest segment of the population. "When conferring the material, people must show their tax code in order to be paid – explains Francesca Feruglio, coordinator of ESCR-Net – in this way these earnings, however limited, are recorded and the data ends up in the central register of the Social Card, in several cases resulting in the suspension of the contribution. This is totally unfair considering the overall condition of these people".
Automated discrimination systems
One of the criticisms levelled at the law lies in the fact that it produces automated decisions that have a strong impact on people’s lives, without foreseeing the intervention of a human being in the process. In essence, once the Social Card algorithm processes the assessment of the individual case, it sends an alert to the local office managing the person’s file. The official does have a window of time to verify the validity of the decision, but such window is deemed insufficient, especially considering that offices are widely and severely understaffed. In fact, an automated process is put in place.
The Ministry of Labour, in its impact assessment of the law, claims that the citizen is protected because there is always the possibility for an official of the competent office or the Ministry to review the assessment and change its outcome, at the citizen’s request. The reality, however, Curcic explains, is that "you only have 15 days to appeal against the decision, and almost nobody does. In many cases because they do not know they can do it, or because being illiterate they do not have the means to act. So they just let the six months pass to reapply for the social allowance. But even if this is successful, that is six months of benefits lost, and there is no guarantee that the application will be accepted or that there will not be a new revocation".
Biljana Đorđević, MP of the Ne davimo Beograd movement, asked for transparency from the Ministry about the functioning of the algorithm and the number of people whose benefits were withdrawn due to the introduction of the Social Card. "They answered that there is no algorithm", Đorđević explains. "According to them, the system merely collects data, then it is the social worker who makes the assessment. The testimonies collected show that this is not the case".
And indeed, if there is no algorithm, on what basis would alerts be sent from the ministry to the social work centres? It is quite obvious that there is data processing, but not on what basis it takes place. "We also asked to have access to the code that actually performs the data collection", Đorđević continues, "but we were denied due to ‘copyright issues’ and ‘national security’".
As regards the drop in beneficiaries, the ministry did not provide precise data, but explained away the reduction with the fact that the country has been experiencing progressive development over the years and consequently an improvement in people’s living conditions, therefore decreasing the need for assistance. While it is true that figures on absolute poverty in Serbia have been slowly improving in recent years (but still getting worse than in 2008 and 2009), the need remains to broaden the pool of beneficiaries, while "it is quite clear that the law serves to exclude", says Đorđević.
"I am struck by the fact that the impact assessment made by the ministry is so poor", comments Gianclaudio Malgieri, associate professor of Law at the University of Leiden. "There is now a certain consensus on the fact that when we talk about human intervention in decisions made through automated systems, this must be significant. Nothing like this can be glimpsed here. Furthermore, when faced with systems that make decisions within a second, the procedure for appealing should be simplified. In this case, however, the standard procedures apply, which essentially lead to ordinary jurisdiction, with all that this entails in terms of time and costs.
In the text of the law on the Social Card, then, it is said that this system may also have effects on vulnerable social groups and subjects. At this point, however, a list of additional protections should follow: there is none of this. Serbia” – Malgieri continues – “has also adopted a law for personal data protection modelled after the European GDPR Regulation (GDPR) and has also signed and ratified the Council of Europe Convention 108+. The country is not lagging behind on the subject of personal data. The law on the Social Card acts in violation of the rules that Serbia itself established".
Unlimited data collection
According to the analysis carried out by A 11, the Social Card Act collects as many as 135 categories of data on the person. To do this, it draws from various sources including the Central Register of Citizens, the Ministry of Internal Affairs, the Tax Administration, and the Pension and Disability Insurance Fund. Some of the requested data are particularly problematic, such as ethnicity (Art. 7(1)(6)). Others, like whether a person owns a weapon, are hardly relevant to the assessment of socio-economic status. "Citizens are asked about ethnicity", Danilo Ćurčić comments, "but not whether their house is connected to the electricity grid or the water system".
The fear is that the law, beyond the purposes it states, will end up becoming a system of profiling and mass surveillance of beneficiaries. "Also in this case – continues Malgieri – there is the risk of a violation of the Serbian law on the protection of personal data. The Social Card implies the collection of a quantity of data that is not justified by the purpose of the law itself. Among other things, Article 4 mentions "the prevention of poverty and the elimination of the consequences of social exclusion” among the purposes of collecting personal data. But the purposes of the processing, also in accordance with the Serbian data protection law, must be specific and precise, they cannot be so generic".
|Civio: going to court to obtain decision-making software source code
In the midst of the economic crisis, in 2009, the Spanish government passed a law subsidizing the electricity bills of about five million poor households. In 2018, the government had to introduce new, tighter regulations. Over half a million Spaniards have then seen their application rejected. The culprit, for some of them at least, is called BOSCO. This software, which is run by the Spanish Ministry for Green Energy Transition, reviews applications for the social bonus and decides on their fate.
The Madrid-based non-profit newsroom Civio (an EDJNet member) found out that the BOSCO failed to properly review each claim and asked the ministry for the source code of the software. After a series of appeals and counter-appeals to the Council of Transparency and Good Governance, Spain’s freedom of information authority (which first was opposed to Civio’s request to release the source code and now supports it), and The Supreme Court, which ruled against opening the code in 2021, the case has now landed before the Audiencia Nacional, Spain’s supreme court.