The State of California says that it uses algorithms to predict whether those in prison will return to a life of crime. Research shows they’ve also used predictive technology to deny 600,000 people unemployment benefits. Despite these applications, state administrators say that no agency uses high-risk forms of automated decision-making technology.
This is from a report from the California Department of Technology which surveyed about 200 state entities. The agencies are required by a 2023 law to report annually about the use of high-risk automated systems that can make decisions impacting the lives of Californians.
If an agency is found to use a high-risk system, it must report the kind of personal data the systems uses to make decisions about people, and the actions they’d take to reduce the likelihood of that usage resulting in discrimination, bias, or unfair outcome.
The California Department of Technology doesn’t know which algorithms state agencies use and only reported what agencies told them. When asked if the employment or corrections department algorithms qualify, the Department said the agencies are responsible to interpret the law.
Given this weak enforcement of the 2023 law, the California Senate recently approved Senate Bill 420 which aims to provide a regulatory structure to make certain that AI systems respect human rights, promote fairness, transparency, accountability, and safeguard Californian’s well-being.
SB 420 is authored by Senator Steve Padilla (D-San Diego). The bill would add Chapter 24.6, “Automated Decision Systems,” to Division 8 of the Business and Professions Code.
“California has a proud heritage as the home of technological vision and innovation while maintaining ethical, safe standards and the development of AI must be no exception,” Senator Padilla remarked. “Tech companies would have us believe any regulation at all would be disastrous. The truth is it would be disastrous to allow tech titans to operate AI without oversight, accountability, or restraint.”
What is a High-Risk Automated Decision System”?
The bill says that “high-risk automated decision system” means an automated decision system that’s used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, any of the following:
- An education enrollment or opportunity;
- An employment or employment opportunity;
- Essential utilities;
- Temporary, short-term, or long-term housing;
- Health care services;
- Lending services;
- A legal right or service; or
- An essential government service.
But the definition of “high-risk automated decision system” doesn’t include an automated decision system that only performs narrow procedural tasks, enhances human activities, detects patterns without influencing decisions, or assists in preparatory tasks for assessment. The bill defines “narrow procedural task” as “a limited, procedural task that has a minimal impact on outcomes.”
What Would Be Required of AI Users?
The proposed legislation would regulate the development and deployment (by both public and private entities) of “high-risk” automated decision systems (ADS) by requiring the following:
- An impact assessment to evaluate their purpose;
- The use of the data;
- The potential for bias; and
- The actions taken to address those risks.
The bill would also require that individuals subject to ADS be told when the tool is being used to make decisions about them, details about the ADS, and, where technically possible, the opportunity to appeal such decisions for review by a person.
SB 420 won’t apply to (i) an entity with 50 or fewer employees; or (ii) a high-risk automated decision system that’s been approved, certified, or cleared by a federal agency that complies with another law that is substantially the same or more stringent than this legislation.
The bill is modeled on the EU AI Act and Colorado’s recently passed AI law and attempts to strike a balance between providing guardrails and not restricting innovation—which is vital to California’s economy.
The Senate Bill 420 passed by a vote of 26-9 and now moves to the Assembly.
Takeaway
Eanet, PC will monitor the activity of SB 420 because of its potential effect on recruitment practices and compliance requirements in California.
Contact us for more information and with any questions.