On 21 April 2021, the European Commission published draft regulations (“AI Regulations“) governing the use of artificial intelligence (AI). The European Parliament and the member states have not yet adopted these proposed AI Regulations.
Key takeaways
The proposed AI Regulations:
- Apply to providers offering or implementing AI systems in the European Union (EU), all users located within the EU and all providers or users located outside of the EU if the output produced by the system is utilized within the EU.
- Follow a risk-based approach with differentiation between prohibited, high-risk, limited and minimal-risk uses of AI systems, with applicable regulatory intervention increasing along with the increase in the potential of algorithmic systems to cause harm.
- Rely on enforcement by the member states.
- Provide that most key employment decision-making processes involving AI will be classed as high-risk and therefore subject to the strongest regulation.
In more detail
The European Commission’s proposed AI Regulations are the first attempt the world has seen at creating a uniform legal framework governing the use, development and marketing of AI. They will likely have a resounding impact on all businesses that use AI for years to come.
Scope
The AI Regulations will apply to the following:
- providers (both public and private actors) offering or putting into service AI systems within the EU irrespective of whether the providers are situated inside the EU
- AI users located within the EU
- AI providers and users located outside the EU provided that the output produced by the system is utilized within the EU
Timing
The AI Regulations will become effective 20 days after publication in the Official Journal. They will then be need to be implemented within 24 months, with some provisions going into effect sooner. The long implementation period increases the risk that some provisions will become irrelevant or moot because of technological developments.
Risk-based approach
In its 2020 white paper, the European Commission proposed splitting the AI ecosystem into two general categories: high risk or low risk. The European Commission’s new graded system is more nuanced and likely to ensure a more targeted approach, since the level of compliance requirements matches the risk level of a specific use case.
The new AI Regulations follow a risk-based approach and differentiate between the following: (i) prohibited AI systems whose use is considered unacceptable and that contravene union values (e.g., by violating fundamental rights); (ii) uses of AI that create a high risk; (iii) those which create a limited risk (e.g., where there is a risk of manipulation, for instance via the use of chatbots)); and (iv) uses of AI that create minimal risk.
Under the requirements of the new AI Regulations, the greater the potential of algorithmic systems to cause harm, the more far-reaching the intervention. Limited risk uses of AI face minimal transparency requirements and minimal risk uses can be developed and used without additional legal obligations. However, makers of “limited” or “minimal” risk AI systems will be encouraged to adopt non-legally binding codes of conduct. The “high risk” uses will be subject to specific regulatory requirements before and after launching into the market (e.g., ensuring the quality of data sets used to train AI systems, applying a level of human oversight, creating records to enable compliance checks and providing relevant information to users). Some obligations may also apply to distributors, importers, users or any other third parties, thus affecting the entire AI supply chain.
Enforcement
Member states will be responsible for enforcing these regulations. Penalties for noncompliance are up to 6% of global annual turnover or EUR 30 million, whichever is greater.
Criticism of exemptions for law enforcement
Overly broad exemptions for law enforcement’s use of remote biometric surveillance have been the target of criticism. There are also concerns that the AI Regulations do not go far enough to address the risk of possible discrimination by AI systems.
Although the AI Regulations detail prohibited AI practices, some find there are too many problematic exceptions and caveats. For example, the AI Regulations create an exception for narrowly defined law enforcement purposes such as searching for a missing child or a wanted individual or preventing a terror attack. In response, some EU lawmakers and digital rights groups want the carve-out removed due to fears authorities may use it to justify the widespread future use of the technology, which can be intrusive and inaccurate.
Support
The AI Regulations include measures supporting innovation such as setting up regulatory sandboxes. These facilitate the development, testing and validation of innovative AI systems for a limited time before their placement on the market, under the direct supervision and guidance of the competent authorities to ensure compliance.
Database
According to the AI Regulations, the European Commission will be responsible for setting up and maintaining a database for high-risk AI practices (Article 60). The database will contain data on all stand-alone AI systems considered high-risk. To ensure transparency, all information processed in the database will be accessible to the public.
It remains to be seen whether the European Commission will extend this database to low-risk practices to increase transparency and enhance the possibility of supervision for practices that are not high-risk initially but may become so at a later stage.
Employment-specific observations
1. High-risk
AI practices that involve employment, worker management and access to self-employment are considered high-risk. These high-risk systems specifically include the following AI systems:
- AI systems intended for the recruitment or selection of natural persons, including advertising vacancies, screening or filtering applications, and evaluating candidates in the course of interviews or tests
- AI systems intended to make promotion and termination decisions, engage in task allocation, and monitor and assess individuals’ performance and behavior
2. Biases
According to the AI Regulations, the training, validation and testing of data sets must be subject to appropriate data governance and management practices, including in relation to possible biases. Providers of continuously learning high-risk AI systems must ensure that possibly biased outputs are equipped with proper mitigation measures if they will be used as input in future operations (“feedback loops”).
However, the AI Regulations are unclear on how AI systems will be tested for possible biases, specifically whether the benchmark will be equality in opportunity or equality in outcomes. Companies should consider how these systems might affect individuals with disabilities and individuals at the intersection of multiple social groups.
3. Processing of special categories of personal data to mitigate biases is permissible
The AI Regulations carve out an exception allowing AI providers to process special categories of personal data if it is strictly necessary to ensure bias monitoring, detection and correction. However, AI providers processing this personal data are still subject to appropriate safeguards for the fundamental rights and freedoms of natural persons (e.g., technical limitations on the reuse and use of state-of-the-art security and privacy-preserving measures such as pseudonymization or encryption). It remains to be seen whether individuals will sufficiently trust these systems to provide them with their sensitive personal data.
4. Human oversight
High-risk AI systems must be capable of human oversight. Individuals tasked with oversight must:
- Have a complete understanding of the capacities and limitations of the high-risk AI system and possess the capability to duly monitor its operation (e.g., anomalies, malfunctions and unexpected performance).
- Remain vigilant against the tendency to rely automatically or over-rely on the output produced by a high-risk AI system (“automation bias”).
- Be able to correctly interpret the output from a high-risk AI system, while factoring in that system’s characteristics, interpretation tools and available methods.
- Decide to disregard, override or reverse the output of the high-risk AI system if necessary.
- Be capable of intervening or interrupting the operation of the high-risk AI system through a “stop” button or a similar procedure.
As indicated in our recent Trust Continuum report, this will require substantial involvement from the human decision-maker (in practice, often an individual from HR), which proves to be challenging for most companies.
Beyond the EU
We have already noted the potential for extra-territorial effect of the AI Regulations. But many AI systems will not be caught if they have no EU nexus. The EU is, as it often is, at the vanguard of governmental intervention into protection of human rights – it is the first to lay down a clear marker of expectations around use of AI. But the issue is under review in countries across the world. In the UK, the Information Commissioner has published Guidance on AI and data protection. In the US, a number of states have introduced draft legislation governing use of AI. None have quite the grand plan feel of the EU’s AI Regulations, but there will certainly be more to follow.
