Europe’s proposed AI regulation falls short on protecting rights

While the proposal’s risk-based approach means it contains a number of measures focused on how high-risk AI systems can still be used, critics argue that the thresholds placed on their use are currently too low to prevent the worst abuses.

Although it includes provisions for the creation of an EU-wide database of high-risk systems – which will be publicly viewable and based on “conformity assessments” that seek to assess the system’s compliance with the legal criteria – multiple experts argue this is the “bare minimum” that should be done to increase transparency around, and therefore trust in, artificial intelligence technology.  

Sarah Chander, a senior policy advisor at European Digital Rights (EDRi), says while the database can assist journalists, activists and civil society figures in obtaining more information about AI systems than is currently available, it will not necessarily increase accountability.

“That database is the high-risk applications of AI on the market, not necessarily those that are in use,” she says. “For example, if a police service is using a predictive policing system that technically is categorised as high risk under the regulatory proposal as it exists now, we wouldn’t know if Amsterdam police were using it, we would just know that it’s on the market for them to potentially buy.”

Giving the example of Article 10 in the proposal, which dictates that AI systems need to be trained on high-quality datasets, Chander says the requirement is too focused on how AI operates at a technical level to be useful in fixing what is, fundamentally, a social problem.

“Who defines what high quality is? The police force, for example, using police operational data, that will be high-quality datasets to them because they have trust in the system, the political construction of those datasets [and] in the institutional processes that led to those datasets – the whole proposal overlooks the highly political nature of what it means to develop AI,” she says.

“A few technical tweaks won’t make police use of data less discriminatory, because the issue is much broader than the AI system or the dataset – it’s about institutional policing [in that case].”

On this point, Geese agrees that the need for high-quality datasets is not an adequate safeguard, as it again leaves the door open to too much interpretation by those developing and deploying the AI systems. This is exacerbated, she says, by the lack of measures included to combat bias in the datasets.

“It says the data has to be representative, but representative of what? Police will say, ‘This is representative of crime’, and there’s also no provision that says, ‘You not only need to identify the bias, but you also have to propose corrective measures’,” she says.

“There is no obligation to remove the original bias [from the system]. I talked to Vestager’s cabinet about it and they said, ‘We stopped the feedback loops from worsening it, but the bias in the data is there and it needs to be representative,’ but nobody can answer the question ‘representative of what?’,” says Geese.

Chander also points out that in most of the high-risk use cases, the proposal allows the developers of the systems to conduct the conformity assessments themselves, meaning they are in charge of determining the extent to which their systems align with the regulation’s rules.

“They don’t really categorise these uses as high risk, otherwise you would have some sort of external verification or checks on these processes – that’s a huge red flag, and as a system check it won’t overcome many of the potential harms,” she says.

Leufer adds that while the proposal does establish “notified bodies” to check the validity of conformity assessments if a complaint about an AI system arises, the measure risks creating a “privatised compliance industry” if commercial firms are relied on over data protection authorities and other similar entities.

“Ideally, [notified bodies] should be focused on protecting human rights, whereas if it’s Deloitte, that’s a paid service, they’re focused on compliance, they’re focused on getting through the process,” he says. “The incentives, I think, are quite off, and the notified body doesn’t seem to be involved in the majority of cases. Even if they were, it doesn’t seem like a sufficient measure to catch the worst harms.”

He says while the processes around databases and conformity assessments are “an improvement on a really bad current state of affairs… it’s just a basic level of transparency [that] doesn’t actually solve any issues”.

Referring to a “slow process of privatisation”, Chander adds that the proposal also sets in motion a governance model whereby the user of an AI system must follow the “instructions of use” provided by the developer.

“This could be viewed in a very neutral way to say, ‘Well, the developer of the system knows how it works so that makes sense’,” she says. “But in a more political way, this means that actually what we’re doing is tying in a relationship between a service provider – a private company – and a public institution… [and] embedding the reliance of the public sector on the private sector.”

Technology and ethics researcher Stephanie Hare says the EU’s current thinking around conformity assessments and databases only provides a “veneer of transparency” as there is an inherent tension within the proposal between creating transparency and protecting the “proprietary information” and interests of private companies.

“The increased transparency obligations will also not disproportionately affect the right to protection of intellectual property since they will be limited only to the minimum necessary information for individuals to exercise their right to an effective remedy,” says the proposal.

“Any disclosure of information will be carried out in compliance with relevant legislation in the field, including Directive 2016/943 on the protection of undisclosed know-how and business information (trade secrets) against their unlawful acquisition, use and disclosure.”