Fundamentals of IT Law: AI Act and risk assessment tools

UNIVERSITÀ POLITECNICA DELLE MARCHE - Facoltà di Economia "Giorgio Fuà"

Fundamentals of
IT LAW
Prof. Roberto Ruoppo
www.univpm.it2

EU AI Act: Hierarchical Structure

EU AI Act: its hierarchical structure

The regulation lays down:

• 1) Prohibitions of certain AI practices (e.g. those with
  subliminal effects, real time biometric identification);
• 2) Specific requirements for high-risk AI systems and
  obligations for operators of such systems;
• 3) Requirements on a voluntary basis for not high-risk AI
  systems3

Classification of AI Systems

Why has it provided this classification among AI systems (1
prohibited, 2 high-risk and 3 not high-risk)?
Because AI systems are able to provide benefits and risks as
well:
Benefits from an economic and social perspective;
Risks for users'rights4

AI Risks

AI risks

• AI systems can deploy subliminal techniques beyond a
  person's consciousness with the objective or the effect of
  materially distorting the behaviour of a person;
• AI systems can exploit any of the vulnerabilities of a person
  due to its age, disability or a specific social or economic
  situation with the effect of causing a significant harm
• They are not transparent: this is the so called «black box
  effect»
• Risks of «cognitive bias>> can arise, due to false or uncorrect
  inputs used by the developer to learn the systemR SITA PO

High-Risk AI Systems

High risk AI systems:

1. Systems useful to grant the enjoyment of social services, in
   particular for people in conditions of vulnerability;
2. Systems in order to evaluate the credit scoring, conditioning the
   access to bank services or to facilities
3. Risk assessment tools in preventing crimesR SITA PO

Requirements for Lawful Employment of High-Risk AI Systems

High risk AI systems: which requirements addressed to providers
for their lawful employment?

1. A risk management system shall be established (Arts. 9 and 11):
   It means that the provider must draft a set of documents through
   which analyse and explaining the outcome and risks through the
   entire lifecycle of a high-risk system, ensuring regular review and
   updating
   This duty is instrumental in order to minimise risks deriving from
   the employment of these systems7

Transparency and Information Provision for High-Risk AI Systems

High risk AI systems: which requirements addressed to providers
for their lawful employment?
2. Transparency and provision of information to deployers (Art. 13)
High-risk AI systems shall be designed and developed in such a way
as to ensure that their operation is sufficiently transparent to
enable deployers to interpret a system's output and use it
appropriately
These systems shall be accompanied by instructions for use that
shall contain the explanation of their characteristics, such as
capabilities, their intended purposes, level of accuracy, any known
or foreseeable circumstance which may lead to risksR SITA PO

Human Oversight in High-Risk AI Systems

High risk AI systems: which requirements addressed to providers
for their lawful employment?
3. Human oversight, having the aim to prevent or minimise the
risks to health, safety or fundamental rights that may emerge when
a high-risk AI system is used (Art. 14):
High-risk AI systems shall be designed and developed in such a way
that they can be effectively overseen by natural persons during the
period in which they are in use
These systems shall be provided in such a way that natural persons
to whom human oversight is assigned are enabled to understand
the relevant capacities and limitations of the system and be able to
duly monitor its operation in view of detecting and addressing
anomalies or unexpected performanceR SITA PO

Accuracy, Robustness, and Cybersecurity for High-Risk AI Systems

High risk AI systems: which requirements addressed to providers
for their lawful employment?
4. They shall be designed and developed in such a way that they
achieve an appropriate level of accuracy, robustness and
cybersecurity (Art. 15)
High-risk AI systems shall be as resilient as possible regarding
errors, faults or inconsistencies that may occur within the system or
the environment in which the system operates
They shall be resilient against attempts by unauthorised third
parties to alter their use, outputs or performance by exploiting
system vulnerabilities10

Requirements for Deployers of High-Risk AI Systems

High risk AI systems: which requirements addressed to deployers?
They are obliged to follow some duties since risks deriving from
high-risk AI systems can be determined by their operative
employment by the deployers
Deployers are better placed in order to determine and understand
which are the risks linked with the concrete use of high-risk AI
systems, in particular when they could not be foreseeable by the
developers
Pursuant to Art. 26 human oversight shall be assigned to natural
persons who have the necessary competence, training and
authority to monitor their operation and to inform the provider
and the relevant market surveillance authority in case of risk11

Information and Explicability for Deployers

High risk AI systems: which requirements addressed to deployers?
First of all they are required to inform users about the fact that
they are interacting with an AI system;
They must provide informations concerning their purposes and the
type of decisions that can be adopted;
Thay must ensure the so called «explicability>>: they must be able
to explain how the final decision has been adopted by the system
(Art. 86);
Deployers of an AI systems that generates or manipulates image,
audio or video content constituting a deep fake shall disclose that
the content has been artificially generated or manipulated12

Ethics Guidelines for Trustworthy AI

High risk AI systems: requirements for their lawfulness

The same requirements that have been introduced by the AI Act
were previously drafted by the Ethics Guidelines for Trustworthy AI
by the High-Level Expert Group on Artificial Intelligence (AI HLEG),
an independent body set up by the European Commission
The guidelines listed seven key requirements for AI systems:

• Human agency and oversight;
• Technical robustness and safety;
• Privacy and Data governance
• Transparency;
• Diversity, non-discrimination and fairness;
• Societal and environmental well-being;
• Accountability13

AI HLEG Ethics Guidelines and EU AI Act

AI HLEG Ethics Guidelines for Trustworthy AI
They have been subsequently reproduced by the EU AI Act for high-
risk Al systems in order to safeguard users' fundamental rights
Through this legal framework and the relationship with AI Act, it
can be understood the role of soft law legal rules, having the
purpose to condition and influence the following binding legal
rules14

AI Systems in Judicial Proceedings

High risk AI systems requirements in light of specific issues

Employment of AI systems in judicial proceedings, when they are
used in order to adopt the final decision (both in civil or criminal
proceedings)
There are many rights that can be affected by this algorithms:

• due process clause (adversarial proceeding, contradictory);
• right to access to judge;
• right to know the reasoning of the judgment15

AI Systems as Risk Assessment Tools

High risk AI systems requirements in light of specific issues

AI systems as «risk assessment tools», when algorithms are used
as a predictive instrument, in order to predict the risk of recidivism
Risk assessment tools use socio-economic status, family
background, neighborhood crime, employment status, and other
factors to reach a supposed prediction of an individual's criminal
risk, either on a scale from 'low' to 'high' or with specific
percentages16

Beneficial Outcomes of AI Risk Assessment Tools

AI specific systems: risk assessment tools
There are some useful and beneficial outcome that do not
determine negative effects for users: this is the case of predictive
instruments that can be used by lawyers and legal firms to plan
their strategies
In France a particular LegalTech startup has been developed,
Predictice, a software useful to provide predictions about legal
disputes' outcome (through huge amounts of data concerning
lawsuits)
It is able to provide informations useful to plan and optimize their
legal actions17

Harmful Outcomes and Guarantees for Risk Assessment Tools

AI specific systems: risk assessment tools
An harmful outcome for fundamental human rights can be realized
by those tools that can be used by judicial authorities in order to
predict the defendant involvement in future crimes
It is particularly dangerous when the risk of recidivism can be
evaluated as one of the elements for the sanction determination
It should only be used if some guarantees have been respected
The European Commission for the Efficiency of Justice (CEPEJ)
drafted a document, European Ethical Charter on the use of
Artificial Intelligence in judicial systems and their environment18

Risk Assessment Tools in US Sentencing

AI specific systems: risk assessment tools
In US many States allow making use of these tools in order to adopt
the final decision, at the sentencing stage: these instruments will
be used to determine the precise amount of the penalty
They have been developed by both public and private bodies
These tools usually make use of dynamic and static risk assessment
factors:

• dynamic factors, changing over time: age, job, use of toxic
  substances;
• static factors, do not change over time: gender, first charge age,
  social context where the defendant was born, previous convictions