TUESDAY, JANUARY 31, 2023

10:00 A.M.  EST

 + + + + +

PRESENT:

CHARLOTTE A.  BURROWS Chair

JOCELYN SAMUELS Vice Chair

KEITH E.  SONDERLING Commissioner

ANDREA R.  LUCAS Commissioner

This transcript was produced from audio provided by the Equal Employment Opportunity Commission.

WITNESSES:

SURESH VENKATASUBRAMANIAN, Deputy Director of Data Science Initiative and Professor of Computer Science, Brown University

PAULINE KIM, Daniel Noyes Kirby Professor of Law, Washington University School of Law

JORDAN CRENSHAW, Vice President, U.S. Chamber of Commerce                   

ReNIKA MOORE, Director of Racial Justice Program, American Civil Liberties Union (ACLU)

MANISH RAGHAVAN, Assistant Professor, Massachusetts Institute of Technology

NANCY TIPPINS, Principal, The Nancy T.  Tippins Group LLC

GARY FRIEDMAN, Senior Partner, Employment Litigation Practice Group, Weil, Gotshal & Manges LLP

ADAM KLEIN, Managing Partner, Outten & Golden

MATTHEW SCHERER, Senior Policy Counsel for Workers' Rights and Technology, Center for

Democracy and Technology

HEATHER TINSLEY-FIX, Senior Advisor, Financial Resilience, AARP

IFEOMA AJUNWA, Associate Professor of Law, University of North Carolina School of Law

ALEX ENGLER, Fellow, Brookings Institution and Adjunct Professor, Georgetown University

                      

 

TABLE OF CONTENTS

Introduction

Chair Burrows

Opening Statements

Chair Burrows

Vice Chair Samuels

Commissioner Sonderling

Commissioner Lucas

Panel 1

Introduction of Witnesses by Chair Burrows

Suresh Venkatasubramanian

Pauline Kim

Jordan Crenshaw

ReNika Moore

Commissioner Questions

Panel 2

Introduction of Witnesses by Chair Burrows

Manish Raghavan

Nancy Tippins

Gary Friedman

Adam Klein

Commissioner Questions

Panel 3

Introduction of Witnesses by Chair Burrows

Matthew Scherer

Heather Tinsley-Fix

Ifeoma Ajunwa

Alex Engler

Commissioner Questions

Conclusion/Adjourn

Chair Burrows

P-R-O-C-E-E-D-I-N-G-S

                               (10:00 a.m.)

CHAIR BURROWS:  Good morning and welcome to today's public hearing of the US Equal Employment Opportunity Commission, which will focus on navigating employment discrimination in automated systems, including systems using artificial intelligence, or AI.  

The Commission has been examining these issues since at least 2016 when we first held our first big hearing on big data and employment decision-making.   Then in 2001 -- I'm sorry, 2021, we launched an AI Algorithmic Fairness Initiative, in which the EEOC has engaged stakeholders through the listening sessions to identify key issues, build our internal capacity through training, and issue technical assistance. Today's meeting continues that work.  The hearing will now come to order.   This hearing is being held in accordance with the requirements of the Sunshine Act, and thus is open to the public.   Real-time captioning is available.   Please visit our website, www.eeoc.gov for details on this service. 

And I'd like to extend a very warm welcome and sincere thanks to each of the witnesses for your thoughtful written testimony and for joining this important discussion today. Before we begin, I will briefly explain the procedures for today's hearing.   The hearing is being recorded, and a verbatim transcript will be made of today's proceedings.   The recording and the transcript, as well as the biographies and written testimonies of our esteemed witnesses, will be posted on the EEOC's website following the hearing.  As the presiding officer, I am responsible for regulating the course of this hearing.

We'll begin with opening statements from each member of the Commission, followed by witness testimony and questions from the Commission.  Although this hearing is open to the public, remarks and questions will not be taken from the audience.  We're honored to have a dozen truly expert witnesses with us today, which means that we'll need to be mindful of the time throughout the proceedings.  And I have the truly unenviable task of keeping us all on track.  Each member of the Commission will have three minutes for opening statements.  Now to begin.  One of this nation's greatest strengths is our commitment to the principles of fairness, justice, and equality.  As a society, we must ensure that new technologies are used in ways that reflect the basic values that throughout our history have helped make America better, stronger, and fairer. 

As the title of today's hearing suggests, rapid adoption of AI and other automated systems has truly opened a new frontier in the effort to protect civil rights.  Increasingly, automated systems are used in all aspects of employment, from recruiting, interviewing, and hiring, to evaluations, promotions, among many others. 

By some estimates, as many as 83 percent of employers and up to 99 percent of Fortune 500 companies now use some form of automated tool to screen or rank candidates for hire.  In recent surveys of its members of the Society for Human Resource Management found that nearly one in four medium sized employers uses automation or AI in their hiring process.

So today's hearing has two goals.  First, to raise awareness about the promise, but also the risks of AI employment.  Everyone should understand and contribute to a public debate over these technologies.  AI and other algorithmic decision making tools offer potential for great advances, but they also may perpetuate or create discriminatory barriers, including in employment.  The stakes are simply too high to leave this topic just to the experts.

The second goal is to ensure that the EEOC continues to do its part from educating employers, vendors, unions, and workers to bring enforcement actions when necessary to address violations.  Simply put, there's no exception under the civil rights laws for high-tech discrimination.  With those goals in mind, expanding public awareness and ensuring compliance with our civil rights laws, I look forward to today's testimony. 

I turn now to Vice Chair Samuels for her opening statement.

MR. WONG:  And if we may, Chair Burrows -- Mark here.  If we could have the Vice Chair quickly do a test as we are experiencing some echo.  And I just want to avoid that moving forward.

VICE CHAIR SAMUELS:  Can you hear me?

MR. WONG:  We can.  And the echo is resolved.  You may begin with your remarks now, Vice Chair.

VICE CHAIR SAMUELS:  Thank you so much.  So thank you so much Chair Burrows for that introduction.  Thank you to all our witnesses for being with us today for this important discussion.  Thank you to the members of the public who are joining us, and whose input on these issues will be so important going forward. We as a society have come so far in the area of artificial intelligence in just the last decade.  The availability of massive amounts of data plus advancements in computing power needed to process all those data have propelled the development of new AI systems.  As Chair Burrows noted, former EEOC Chair Jenny Yang and former Commissioner Victoria Lipnic were prescient when they held a Commission hearing back in 2016 on big data in the workplace.  And our current Chair Burrows has continued this important focus through the Commission's artificial intelligence and algorithmic fairness initiative, of which today's hearing is a part.

I'd also like to thank in particular Commissioner Sonderling for his really thoughtful contributions in this area.  The EEOC supports innovation and we're excited about the benefits that AI can provide.  These benefits can revolutionize the way that we work, potentially improving efficiency, precision, -- excuse me -- and even known discrimination in employment decisions.  But we must ensure that those benefits are delivered in compliance with our EEO laws and don't perpetuate historical discrimination, or even inadvertently exclude or discriminate against applicants or employees on prohibited bases. 

The concepts and legal standards are familiar based on long-standing legal principles that have governed non-discrimination obligations for decades.  For example, disparate impact theory makes clear that where a selection criterion disproportionately excludes a protected group, the employer must show that the criterion is job related and consistent with business necessity.  The uniform guidelines on employee selection procedures provide substantial insight into how to make that showing.  And much of that information remains relevant and useful today.  But we have a lot to learn about how AI is being and can be used in the workplace, how our legal standards apply, and how we can prevent discrimination on this (audio interference) frontier.

For example, how can employers ensure that the data used to train algorithms is unbiased and representative? Independent of the input data, how can we track whether algorithms are implemented in a non-discriminatory way? How can vendors and employers work together to integrate non-discrimination principles into the design and use of AI systems?  How do we evaluate AI systems that rely on such a vast quantity of data, such a deep web of inferences, that even their designers cannot explain how they work?  Are there different considerations that should guide us when AI is used in different workplace contexts, whether recruitment, screening, or employee monitoring?

I'm so excited to hear from our witnesses today and to work in partnership with my EEOC colleagues and our stakeholders as we pursue a goal that I know we all share.  Ensuring that we can enjoy the benefits of new technology while protecting the fundamental civil rights that are enshrined in our laws.  Today is just a down payment on this continuing discussion, and I look forward to working together on these critical issues.  Thank you.

CHAIR BURROWS:  Thank you.  And we'll go now to Commissioner Sonderling.

COMMISSIONER SONDERLING:  Thank you.  Over the past few years, as I began to study the issues surrounding technology in the workplace, I've gotten to know many of the witnesses testifying today.  I'm grateful you're all here.  While some are inclined to focus on the challenges of AI, it is equally important to highlight that AI can mitigate the risk of unlawful discrimination and create accessible workplaces that were previously unimaginable, all goals of the EEOC. 

Today we are overwhelmingly going to hear about the potential of AI to discriminate, to codify and scale individual bias.  We will hear bold face assumptions about how vendors are designing and selling these programs and how employers are allegedly implementing them to the detriment of their workers.  However, determining how the EEOC should regulate in this area, it is essential to hear from a diverse and wide range of stakeholders, including everyone from workers, civil rights groups, employers, and critically, developers and vendors, and even those investing in the creation of this technology.

I have personally spoken with all of these groups, each with extremely different views on how to address this budding issue.  No matter my own personal opinions, as a Commissioner, I am duty bound to listen carefully to anyone and everyone who's willing to discuss.  Unfortunately, that is not going to be the case today.  Our first formal hearing on artificial intelligence alone is curiously missing representation from those who are actually innovating, designing, building, and selling these products. 

How are we supposed to know what products are being developed, how they're supposed to be used versus how they're actually being used without hearing from a single entrepreneur in the trenches making them? Further, out of the 12 witnesses invited to testify today, two are on behalf of employer organizations.  Not a single witness today has ever designed, built, marketed, sold, bought, or had the burden of implementing workplace AI programs, nor are they the employers ultimately responsible to the EEOC for the unlawful use of AI employment tools. 

The vendors that I have met with are taking their anti-discrimination efforts seriously.  They believe in their algorithms and in their potential to promote equal opportunity.  They're willing to discuss their method's designs and provide us with information on what is working, what is not, and where they need the EEOC's help to ensure that bad actors do not discredit the entire industry.

This hearing would greatly benefit from their lessons learned, innovation, and best practices.  The EEOC should welcome this interest as we are in a critical time where the software can still be developed and designed in a lawful, ethical, and trustworthy fashion with the EEOC's expertise before it is completely pervasive in businesses of all size. 

Today the Commission and the public are being deprived of this benefit.  Instead, we'll hear numerous theories, assumptions, and accusations on how these vendor's tools are being used to discriminate with no rebuttal.  This is a disservice to the public.  It undermines whatever guidance or regulations are subsequently issued from this hearing.  Therefore, unlike the last time the EEOC issued guidance on AI, it is critical that any further AI action be subject to a robust public comment period, whereby all are entitled to participate by law.  Thank you.

CHAIR BURROWS:  Thank you Commissioner Sonderling.  We'll go now to Commissioner Lucas.

COMMISSIONER LUCAS:  Thank you, Chair Burrows.  Good morning and thank you to all the witnesses for taking time to prepare your testimony and share your expertise with us today.  Each of you brings a unique perspective on this important topic and I look forward to an engaging discussion. 

As we've already heard today, and will hear again from many witnesses, artificial intelligence is a part of many of our daily lives.  And a large portion of Americans use and interact with some form of artificial intelligence every day.  The employment context is no exception.  Companies large and small are incorporating AI into recruitment, hiring, training, assessments, compensation, and terminations.  Proponents of AI, a promise that that'll provide new benefits and efficiencies in these processes. 

However, everyone involved in the discussion around AI, both AI proponents, those urging caution, and others recognize the potential for intentional and unintentional discrimination to occur from an increased dependence and use of AI in the employment decision making process.

So how do we balance the benefits and efficiencies that AI can provide with potential pitfalls? What standards should be adopted to allow for continued use and growth of AI while protecting against the potential for discriminatory impacts and outcomes?  How do we provide -- approach the process of providing compliance assistance and guidance so employers can actively work to prevent discrimination without us overreaching or overregulating?

These questions are all important, and how we answer them will impact employers and workers nationwide.  As we consider these and others questions, I want to emphasize that existing civil rights laws such as Title VII already provide robust protections against discrimination in the workplace, provided that we use them to their full, including taking a fresh look at underused provisions.  These laws have been successfully applied to new and emerging technologies in the past, and I'm hopeful that they can apply to AI -- in the AI context in a manner that ensures that emerging technology is used in a way that is consistent with equal opportunity at work.

Navigating the promise and peril of AI in our workplaces will require a thoughtful and collaborative approach, involving all stakeholders.  So, today, as we engage in this deeply important discussion, I look forward to hearing each of the witness's perspectives on this matter.  However, I echo Commissioner Sonderling's concerns that an important stakeholder perspective is missing for this conversation. It seems unquestionable the vendors and other categories of third party entities as well as the employers who are using AI themselves are a critical part of the use of AI in employment lifecycle. 

And in particular, some -- as if, some argue, vendors and other developers of AI play a crucial role in contributing to potential problems with the use of AI.  And it seems to me that these entities must be directly engaged in order to reach any meaningful productive solutions.  Therefore, I think it is unfortunate that we do not have witnesses here today that can directly represent this important category of stakeholders.  Thank you.

CHAIR BURROWS:  Thank you.  It's now my pleasure to introduce the speakers on our first panel in the order that they will be speaking today.  Professor Suresh Venkatasubramanian.  Welcome.  He's a professor in computer science at Brown University and the director of the Center for Technological Responsibility.  He also previously served as the Assistant Director for Science and Justice at the White House. 

Professor Pauline Kim, welcome, is the Daniel Noyes Kirby Professor of Law at Washington University School of Law in St. Louis.  She's an expert on the law governing the workplace and the employment relationship.  Professor Kim's current research focuses on the use of big data and artificial intelligence in the workplace. 

Next we have Jordan Crenshaw.  Welcome to you as well.  So he serves as vice president and leads the day-to-day operations at the US Chamber of Commerce's Technology Engagement Center.  Mr. Crenshaw also manages the Chamber's Privacy Working Group, which is comprised of nearly 300 companies and trade associations.

Welcome as well to ReNika Moore, who is the director of the ACLU's Racial Justice Program.  She is leading a dedicated team that uses litigation, advocacy, grassroots mobilization, and public education to dismantle barriers to equality for persons of color.  Before joining the ACLU, Ms. Moore served as Labor Bureau Chief of the New York Office of the Attorney General. 

Welcome and thank you again to our witnesses.  And as a reminder, you each have five minutes for your remarks.  Our information technology team will keep track of the time with a timer.  You should be able to see that on your screens.  And we will begin with Professor Venkatasubramanian.  So you have the floor.

DR. VENKATASUBRAMANIAN:  Thank you, Chair Burrows.  And thank you Commissioners of the EEOC for the opportunity to provide witness to the Commission today.  My name is Suresh Venkatasubramanian.  I'm a professor at Brown University and director of the Center for Tech Responsibility.  I'm a computer scientist who for the last decade has studied the ways in which automated systems, and especially those that use artificial intelligence, may produce discriminatory outcomes in employment, performance evaluation, and in many other domains. 

And most recently I served as the Assistant Director for Science and Justice in the White House Office of Science and Technology Policy in the Biden-Harris Administration, and co-authored the Blueprint for an AI Bill of Rights, a document that lays out five key protections for those meaningfully impacted by the use of automation and provides detailed technical companion for how these protections can be realized.

 Today I'd like to emphasize some of the key points from my written testimony in these oral remarks.  We are living in an age of fast moving technological innovation that expands our idea of what's even possible.  As a computer scientist, this gives me great joy to be part of this revolution.  We're also living in an age of unchecked tech deployment that has disrupted systems and harms people in ways that we're only beginning to reckon with.  There are algorithms for screening candidates that have discriminatory outcomes.  There are tools that claim they can see inside our mind into our personality, but that are based off of an unsound science. 

There are unaccountable and opaque systems that lack the accountability we expect when life altering decisions are made about us.  Over the past decade, I've helped create and have been part of a research community comprised of technologists, social scientists, lawyers, and academics from many disciplines.  And what our research has shown is that data driven tech like AI and machine learning when deployed unthinkingly and without proper guardrails in place will inevitably cause harms.  So what should we do about it?  There are two problems that we need to solve here.

Firstly, we need to know how to install guardrails around technology.  And thankfully, years of innovative research and collective action by policymakers, practitioners, and researchers has given us a strong set of guardrails to work with.  The blueprint for an AI Bill of Rights.  The AI Risk Management Framework developed by NIST, which compliments and is aligned when appropriate.  And the EEOC's very own guidance on how automated hiring systems might comply with the Americans with Disabilities Act and so much more.

These guidelines have many elements in common.  They say that claims that a piece of hiring tech is safe and effective should be verified.  They say that claims that tech mitigates disparate impact should be verified.  They said that this verification should continue after deployment on an ongoing basis.  Because after all, the key advantage in machine learning is that it learns and adapts, and so should its verification.  They say that stakeholder input, especially from those impacted by technology, is crucial to build trust and ensure that we get the benefits that technology promises us.

Now, there's a second problem that we solve.  We need to create positive momentum towards developing and installing such guardrails.  Many argue that guardrails limit innovation and the potential for world-changing technological development.  I wish those who make those arguments could talk to the people I speak with.  Developers inside some of the biggest tech firms out there, students entering the field of technology, advocates who see the promise of technology to help their communities.

What is common amongst all these folks is fearlessness and imagination and a strong belief that tech is in fact malleable enough to be shaped in whatever direction we want to take it in.  They have much more faith in technology and it's immense potential than in fact those arguing against those guardrails.  They see that protections help shape and direct innovation in ways that we can all benefit, while also building trust.

And they recognize that investment is needed to innovate, but that this investment will pay off richly because the benefits will be widely distributed rather than only to a few, and will in fact keep the United States at the forefront of tech innovation.  We can help channel this fearlessness and creativity with the guidance that the EEOC can provide.  We don't need to be scared.  Scared of experimenting or scared of changing the way tech development has gone on thus far.  But we need your help to do this.  Thank you for your time.

CHAIR BURROWS:  Thank you.  And we'll go now to Professor Kim.

MS. KIM:  Chair Burrows, Commissioners, thank you for the opportunity to speak to you today.  I'm the Daniel Noyes Kirby Professor of Law at Washington University School of Law in St. Louis.  Before I was a law professor, I was a civil rights lawyer in San Francisco. 

One of the very first cases I worked on as a new lawyer, and this was decades ago, was a suit against a temporary employment agency.  This firm received requests for temporary help and matched them with workers with the required skills.  In doing so, this agency willingly fulfilled customers' discriminatory requests.  If, for example, a company needed a temporary receptionist and wanted a white worker, the temporary agency would only send white workers. 

Now, today, that kind of old fashioned discrimination can occur through AI tools that automatically screen job candidates or match them with open positions.  The employer doesn't have to explicitly state a discriminatory preference.  The software might simply learn those preferences by observing its past hiring decisions.

And even employers who have no discriminatory intent could inadvertently rely on an AI tool that is systematically biased.  So these automated systems truly do represent a new frontier in civil rights.  They use novel, technologically complex processes, but they can produce the same troubling effects as past discriminatory practices.  So Title VII clearly prohibits the blatantly discriminatory acts of the temp agency years ago, and it undoubtedly applies to new forms of discrimination that are emerging today. 

However, the doctrine that has developed with human decision makers in mind may need to be clarified to address the risks that are posed by automated systems.  Simply prohibiting a model from using race, sex, or other protected characteristics as a feature will not prevent it from discriminating because an AI tool can exclude workers by relying on proxy variables.

On the other hand, a blanket prohibition on considering protected characteristics when building a model is counterproductive.  In order to conduct audits, to identify discriminatory effects, or to diagnose why they are occurring, AI designers and employers will need information about those characteristics. 

So, given the unique risks of automated systems, I think there are several points on which the EEOC could offer some guides to clarify the law right now.  So, first, the Agency could make clear that AI tools that cause discriminatory effects cannot be defended solely on the basis of statistical correlations.  Instead, the employer should have to demonstrate the substantive validity of its selection tools.             In other words, that they are actually measuring job related skills or attributes, not simply relying on correlations, and that they're not relying on any kind of arbitrary or implicitly biased features. 

Second, the EEOC could offer guidance on the duty of employers to explore less discriminatory alternatives.  In many situations, there are multiple solutions for a given optimization problem.  Employers should explore the available options and select one with the least discriminatory effects.

Third, the EEOC should make clear that taking steps to correct or prevent a model from having a disparate impact is not a form of disparate treatment.  The most effective strategies for de-biasing automated systems require paying attention to race and other protected characteristics when building the model. 

These strategies do not make decisions about individual workers turn on a protected characteristic, and therefore they do not constitute disparate treatment.  By making this clear, the Agency can encourage voluntary employer efforts to rigorously examine their practices and to avoid discriminatory effects. 

And finally, the EEOC could offer guidance about the legal responsibilities of labor market intermediaries like online advertising and job matching platforms.  Employers should also be educated about how these platforms work and the risks that even when they are neutrally targeting their ads to try to reach a diverse audience, they might be delivered in a biased way because of the algorithm in the platform, and that would reduce the diversity of the applicant pool.

And I think increasing understanding about that among employers could help with their recruitment process.  So automated decision systems are not inevitably discriminatory.  A well-designed and implemented system can reduce the influence of human bias.  And that's an important opportunity and important development.  But these tools are also not inherently objective or neutral either.  And the EEOC can play a critical role in ensuring that they are not misused, and that equal opportunity is open to all in the era of big data and AI.  So thank you again for the opportunity to speak.

CHAIR BURROWS:  Thank you, Professor Kim.  And now, Mr. Crenshaw, we go to you.

MR. CRENSHAW:  Thank you.  Good morning, Chair Burrows, members of the EEOC.  My name is Jordan Crenshaw and I'm the Vice President of the US Chamber of Commerce's Technology Engagement Center.  It's my pleasure to talk to you today about how we can work to build trustworthy artificial intelligence.

 AI is changing the world as we know it.  By 2030, AI will have a $16 trillion impact on the global economy.  But from a practical level, what does this mean?  AI is helping medical researchers develop future cures and tailor treatments to new variants of viruses.  And it's bolstering our cyber defenses against an evolving digital threat landscape.  But also from an employment context, it's helping tackle supply chain issues where we have a lack of available truckers, filling gaps like patient monitoring in places where there is unmet need for skilled nurses, assisting small businesses to find qualified candidates for open positions, particularly from non-traditional applicant pools.

And it's helping employers avoid their own potential bias in the hiring process.  AI is already here, it's not going away, and we cannot sit on the sidelines.  Businesses at all levels must come to rely upon this technology.  A recent report that we release highlighted the importance of using technology by small businesses during the pandemic. 

And a few of the takeaways from this report are 93 percent of small businesses are using at least one technology platform.  When small businesses use technology, they contribute $17.7 trillion to the US economy.  86 percent of small businesses say technology helped their business survive COVID-19.  And 25 percent of small businesses actively plan to use AI in the near future.  However, for AI to be fully embraced by society, Americans must have trust in it. 

And while AI has many benefits, as I previously mentioned, we must also be cognizant of the fact that AI also brings a set of unique challenges that should be addressed so that concerns over its risks do not dampen innovation, and also help the assure -- ensure the United States lead globally in trustworthy AI.  We recently polled the American public about their views on artificial intelligence and if they generally help or hurt society.

The findings were unanimous in that most Americans learn about -- more Americans learn about AI the more comfortable they become with its potential role in society.  In fact, the same polling showed that 85 percent of Americans believe the US should lead in AI, and nearly that same number voiced that America is best positioned to develop ethical standards for it.  We agree.  It's why the Chamber last year established its Commission on AI Competitiveness, Inclusion, and Innovation led by former representatives John Delaney and Mike Ferguson. 

This Commission has been tasked with developing policy recommendations in three core areas, trustworthiness, workforce preparation, and international competitiveness.  Our Commission held field hearings in Austin, Silicon Valley, Cleveland, London, and even here in Washington DC.  We've learned a lot, and we plan to release our final recommendations in March of this year. 

In the meantime, while we wait for those recommendations, we offer the following observations and recommendations about what it will take to maintain trustworthy AI leadership.  The federal government has a significant role to play in conducting fundamental research and trustworthy AI.

The Chamber was pleased to see passage of the Chips and Science Act in hopes to see the necessary appropriations made for its implementation for trustworthy AI R&D.  We also encourage investment in STEM education.  We need a trained, skilled, and diverse workforce that can bring multiple voices to coding and developing systems.  Artificial intelligence is only as good as the data that it uses.  And that is why it is key to both government and the private sector team up to ensure there is quality data for more accurate and trustworthy AI.  Government should prioritize improving access to their own data and models in ways that respect individual privacy. 

And as Congress looks to address issues like consumer privacy, policymakers must be careful in whether or not they place too many restrictions on private sector collection and use of data that could actually inhibit the ability of developers to improve and deploy AI systems in a more equitable manner.  We discourage one size fits all government mandates, such as the use of third party auditors until standardization and best practices have been developed.

As technology like AI can be used as a force for good in employment, we would caution against agencies viewing the mere use of sophisticated tech tools like AI as suspicious.  And finally, we must embrace open source frameworks like the NIST AI Risk Management Framework, which is a consensus-driven, cross-sector, and voluntary framework to leverage best practices.  These recommendations are only the beginning, and we look forward to sharing our own AI Commission's results with you soon.  So thank you for your time to address how the business community can partner with you to maintain trustworthy AI leadership, especially in employment.  Thank you.

CHAIR BURROWS:  Thank you, Mr. Crenshaw, and we look forward to those recommendations.  We'll go now to Ms. Moore.

MS. MOORE:  Thank you.  So I want to thank you Chair Burrows and the Commission for the invitation to join such an esteemed panel of speakers today. 

The ACLU has engaged in advocacy and research to advance equity in the use of technology.  Several years ago, we, with Outten & Golden and the Communication Workers of America, filed the charge with the Commission challenging Facebook's discriminatory ad targeting of housing and employment ads.  And we've also worked with CDT and other civil rights organizations to develop civil rights standards for 21st century employment selection procedures, which you'll hear more about from CDT's Matt Scherer later today. 

And so we've already heard from the Commissioners and from other witnesses about the benefits and the potential that new AI and machine learning tools can make to the process of finding a job and make it more open and inclusive for people who've historically faced discrimination.  But those benefits are by no means inherent to the tools.

And it's important that we have necessary oversight and accountability to ensure that all workers have access to jobs.  And one of the major reasons that we see that AI does not function properly is that it powers tools that are trained and operated using large troves of data to other systems that may have structural discrimination. 

And what I'll talk about, two examples where we see marginalized groups being disadvantaged in the data.  First, we see people of color are overrepresented in undesirable and unreliable data that often foreclose job opportunities, so what -- I'm thinking of records that come from criminal proceedings, from addiction proceedings, and from credit history.  With criminal legal records, we know that Black, Latino, and Native American people are disproportionately represented in criminal databases due to a variety of factors, including racial profiling of people of color and harsher outcomes in the criminal legal system.  With eviction records, we similarly see that Black women are more likely to be targeted for eviction by landlords than other similarly situated groups of renters.

And we know the data sets containing criminal legal records and eviction records are also notoriously poor quality.  They contain incorrect or complete names, old or out of date entries, and non-uniform terms that describe charges or dispositions and other information that's necessary to understand outcomes.  And with credit history, because of a history of redlining and the targeting of predatory credit products like payday loans and other barriers to credit, we see that people of color have disproportionately high rates of negative credit data. 

And like the data from criminal and eviction proceedings, credit data is rife with data quality problems, including errors, and misleading or incomplete information.  And so many AI driven selection tools heavily, if not exclusively, rely on data from these sources to conduct background checks.  And as a result, we see the Black, Latino, Native American and other people of color may be more likely to be disadvantaged or lose out on job opportunities because the data tell an incomplete or inaccurate story of the kind of candidate or employee they are.

A second example we see is under representation of people with protected characteristics in the data that's often used for AI tools like ad targeting or resume screening.  And several factors can contribute to under representation.  One, we see groups of color on the wrong side of the digital divide.  We know that Black, Native American, and Latino households and people with disabilities are less likely to have reliable high-speed internet access.  And without that kind of access, people are less likely to engage online with many of the systems that produce data, including high quality data, that are used to train and develop tools. 

We also know that trans people and other LGBTQ people are more likely to use names and pronouns that do not match their government identification, which can obscure their information in data.  And this under representation of marginalized groups can create barriers to opportunity, including learning about jobs, completing preliminary screening exercises, or requesting accommodations or assistance.

And so I and other advocates speaking today offer several recommendations to the Commission to strengthen compliance, and educate employers, and to empower workers.  And I'll emphasize, too, that particularly aligned with the EEOCs recently released strategic enforcement plan, we recommend that the Commission continues to pursue opportunities to increase enforcement, including strategically selected targets to ensure that we have accountability in the use of these systems. 

And we commend the EEOC on the initial guidance on ADA compliance last year, and we recommend that the Commission issue guidance for Title VII compliance as well as the A -- ADEA identifying the most problematic tools and practices.  These technologies are complex and they're operating at a scale that's really unprecedented.  And so I want to end by thanking the Commission for leading this effort to guarantee the rights of workers online and offline.  Thank you.

CHAIR BURROWS:  Thank you, Ms. Moore, and to all of our panelists.  And so we will go now to questions from the Commission.  Each Commissioner will have 10 minutes for questions of this panel and I'll begin with Vice Chair Samuels on the floor.

VICE CHAIR SAMUELS:  Thank you so much, Chair Burrows, and thank you so much to our witnesses for such insightful testimony.  I have many more questions than I'll have time to ask today, so I really do hope that we can continue this dialogue down the road.  But my first question I think is for Professor Kim. 

You talked about the difference between correlation and substantive validity.  I wonder if you could say a little bit more about that, and about your views about what you guess the uniform guidelines on employee selection procedures requires.  You guess prefers to correlation, but also makes clear that employers have to abide by generally accepted validation standards and talks about being job related and consistent with business necessity.  Could you talk about what that's showing would require?

MS. KIM:  Sure.  So the uniform guidelines talk about several different ways in which validity can be established, and I think one of those particular, the criterion related basis for validity.  There's some ambiguity around that and some have suggested that if -- that if a tool can show a statistically significant, a robust correlation, between a job outcome without really understanding why the features that are fed into the system are producing that robust statistical correlation, that would nevertheless be sufficient to validate the tool.

My concern is that particularly because of the big data that is being used right now, where some of these systems -- some systems use, you know, very carefully curated data sets, and others, some less responsible vendors can go out and collect a lot of data and just sort of feed it into the system and see what kind of patterns emerge.  And the concern there is if they're -- they really don't know what the different features are that are being relied on.  How they might relate to job performance or job skills, they might be turning on -- they might show a strong correlation, but it might actually be that the correlation is a proxy for a protected characteristic or is turning on some kind of characteristic that if we were to examine it more closely, would say, well, may maybe that's actually a proxy for some kind of biased outcome.

And this goes again to this question of explainability where there's a lot of discussion around AI explainability.  So if, for example, it turns out that positive job performance is correlated with something like the prediction that somebody is likely to stay on the job for a long period of time, if turnover is a consideration, right, or that they're not likely to take breaks in employment, we might want to know that that's what's driving the outcome.  Because it might be that that's actually just turning out to be a proxy for something like women of childbearing age or individuals with disabilities who are more likely to take a break.  So I think it's important to clarify that a mere statistical correlation is not enough to justify a tool.

VICE CHAIR SAMUELS:  Thank you.  That's very helpful.  And I take it your view is that you, yes, requires some showing of job relationship and causation beyond correlation.

MS. KIM:  Yeah.

VICE CHAIR SAMUELS:  Thanks very much.  I have a question for Professor Venkatasubramanian.  You previously worked with an AI vendor and I'm interested about your experience there.  Can you talk a little bit about what kinds of audits or evaluations vendors are doing to try to evaluate both the disparate impact of their potential -- disparate impact of the tools that they're developing and the validity or job relatedness of those criteria?

DR. VENKATASUBRAMANIAN:  Yeah thank you for that question.  And I will say that I, you know, while I'll try to answer the question in the general and not try to be too specific, but yes, I did work with one of the vendors or -- in a past period.  In general, I think, at least in my experience, vendors look to guidelines like those provided by the EEOC to decide what to test and test only those things.  If -- they're -- if they're -- if they're asked to sort of test for a specific form of disparate impact with regard to a specific characteristic, like say, for example, race, that is exactly what they will do and they will do nothing else.

And as Professor Kim mentioned, the broader -- a broader understanding of the use of these technologies in these contexts requires not just a narrow focus on specific tests, but a broader understanding of where these causation might be coming from and what could be possible proxies.

And so I think a lot of the testing is done in good faith with whatever available technologies there, but is done in a way that is sufficient to answer potential legal inquiries.  And more testing is not done because of the risk of the potential for future exposure if someone asks what the result of the testing was.  So it's very conservative, very geared towards satisfying the guidelines.  And that's what I've seen in my experience, both working directly as well as just seeing how this works. 

And I should also say that there are a lot of very good -- I mentioned -- like I mentioned, there are a lot of very good technologists working in these settings, really trying to push the envelope of what they can do regarding understanding how this AI system work and how -- what can be done to mitigate problems.  And often they are -- they need to be careful in what they test to make sure they don't incur liability later on.  And so that becomes a tension between the developers who want to do more testing and more evaluation and those often above them in the management chain who ask them not to.

VICE CHAIR SAMUELS:  That a helpful cautionary note.  Thank you.  With my remaining time, I'm hoping that I can ask two questions.  The next one is for Ms. Moore.  I think you talked about under representation of certain groups in the data that algorithms rely on.  And I think that you are likely familiar with the announcement that Meta made recently about its variance reduction system where it is trying to identify variables that might be proxies for protected characteristics. 

One of the things that Meta noted was that the absence of data on certain things is hampering efforts to evaluate impact.  And so my question for you is, should we be collecting more data about demographic characteristics, or is that something that is too risky for us to do, even if it's in the interest of ensuring the fairness and validity of particular selection tools?

MS. MOORE:  I think it is critical to collect this data.  We have seen that across different sectors, including in the criminal justice sector as well.  And we rely on the collection of that data to be able to understand what is happening in terms of race.  And we know that there are proxies for race that even if you're not explicitly collecting, for example, for race, that proxies like zip code, proxies like college education, there are other ways in which often we have seen AI systems learn and be able to accurately identify race even when that specific characteristic is removed from the system.

So I think we do need that visibility in order to monitor and we need the additional transparency, and we have recommended that that be a -- that that be a feature of audits and that is an area there where the EEOC can provide additional guidance.  And we would like to see the ability for independent audits in this space in order to understand and learn how these systems are working.

As Professor Kim noted, it's often unclear how exactly the algorithm is learning and why it is coming up with the -- with the outcomes and recommendations that it is.  And that's -- that can be based on correlations that have nothing to do with job relatedness.

VICE CHAIR SAMUELS:  Thank you.  That's helpful to hear.  Let me circle back around to Professor Kim, if I might.

MS. KIM:  Sure.

VICE CHAIR SAMUELS:  So you've noted that you think it is both imperative and lawful for designers and users of AI tools to take protected characteristics into account.  Can you say a little bit more about why you think that's important and what the lawful approach is to doing so are?

MS. KIM:  Yeah, so I think the big -- because of the possibility of proxy variables, because of the possibility that past patterns of disadvantage are inadvertently brought into these stems, it's really important to be able to audit them and to assess them on an ongoing basis. 

And I think there's a pretty broad scale agreement about that, even if there's some disagreement in the details of what that would look like.  But in order to do that, in order to see if there are disparate impacts that are occurring, or to figure out where the problem is coming from, is it a problem with underrepresented data? Is it a problem with poor quality data? You know, is it a problem with some kind of bias feature? We need to have information.  The designers of these products need to have information and they need to have certain degrees of freedom to be able to explore that.

And I think the concern is if the message gets out, if you take race or other protected characteristics into account ever, anywhere in the model building process, you're somehow going to run afoul of the law, that's going to tie the hands of the builders of these systems to actually try to understand what's going on and how to remove that bias. 

So I think it -- it's a fine line to walk, but I think it is important that there's a recognition that using this information to de-bias these systems is not a form of discrimination and that employers and builders of these systems should have some degrees of freedom to do that in a way that is producing less discriminatory effects.

VICE CHAIR SAMUELS:  Thank you so much to all of you.  Again, I have many more questions than I have time for, but I really appreciate all your input.

CHAIR BURROWS:  Thank you.  We'll go now to Commissioner Sonderling.

COMMISSIONER SONDERLING:  Thank you.  First, I'd like to go to Mr. Crenshaw, the Chamber.  Mr. Crenshaw, at the Chamber's C_TEC you -- you've really been looking broadly about regulation of artificial intelligence, not just here in the United States, but worldwide.  I've had a seat at that table.  So, you know, today we're talking about the use of AI employment, but in your work at the Chamber, you've been looking at every agency under every law, whether independent agencies or their actual cabinet agencies. 

So can you just kind of give us an overview of the general approach that you were seeing, the different factions of government with different jurisdictions taking on regulating AI? And then the second part of that, you know, from your opinion and the private sector's opinion, you know, which one seems to be the best approach?

MR. CRENSHAW:  Yeah.  Thank you Commissioner.  I think that's a great question.  I think what we have seen in the business community is a variety of approaches that have been taken.  So currently the National Institute for Standards and Technology over at the Department of Commerce released its voluntary risk management framework designed to mitigate risk.  This was congressionally mandated from legislation put forward by Representatives Lucas and Johnson.  It was bipartisan in nature, and I think one of the things about that approach was that it was open, it was transparent, it brought together multiple stakeholders to develop voluntary best practices and standards to mitigate risk for AI.  And I think in particular, that approach is helpful for small businesses, for example, who generally cannot weather or compete with regulations like their larger counterparts can.

And this enables greater flexibility for those small businesses to mitigate risk.  And I think one of the things about this approach was that it was -- it's iterative.  Every step of the way, the public was involved, had notice about what was coming next, and what the language would look like, potentially.  And also would it, in fact, take stakeholder input across the board.  

There's also a process right now that's congressionally mandated, that's the National AI Advisory Committee, and that is also congressionally mandated.  That brings together an entire committee of industry and academic experts in a way that's very similar to our own AI Commission at the Chamber.  And they're looking at developing policy recommendations as well.  They're also doing field hearings, they're also seeking robust public record in their own recommendations.

And I think one of the things that's important as independent agencies look at potential guidance, or regulations, or frameworks that they develop a robust record.  You know, we actually sent a letter to independent agencies in November of 2021 really emphasizing the importance of agencies taking an independent look at the data in front of them and developing a robust record so that they're in compliance with the APA as well as enabling them to still continue to receive Chevron deference in courts.  And it's important that agencies do develop those robust records. 

So those are -- those are examples of processes in place that have been put together to really develop those frameworks.  Other things we've seen though, obviously we've seen the introduction of the AI Bill of Rights out of the White House.  I would say that one of our concerns there is we did not see quite as much public input as we would've liked to have seen into that approach before it went out.

So I think that's one consideration that we have, and that was also not a congressionally mandated product.  Also, we've seen legislation going forward, the American Data Privacy and Protection Act, that does make it illegal to use algorithms to discriminate against protected classes.  I think most in industry would be, if not all, are supportive of that, full stop, but I do think that there are concerns there about potential requirements around impact assessments and what type of risk is required in that instance for impact assessments to be triggered.  And finally, we've seen legislation in cities like DC that could potentially require small businesses to have to conduct impact assessments and be subject to private rights of action that have failed so far.

And employment specific regulation, too, in the City of New York.  So we have seen efforts in the states, we've seen ever efforts on the federal side, but you know, in terms of the approaches that we find to be the best, when you bring different interest and stakeholder groups together, whether it be academia, whether or not it be civil society, or industry, it's best to have an open process, one with a robust record and one that's iterative so everyone is on notice about the different processes and steps going forward with the development of a framework.

COMMISSIONER SONDERLING:  Thank you for that very thorough comprehensive overview.  What I got out of it is obviously transparency and  building trust through having a diverse set of stakeholders involved, which seems other federal government agencies are doing, as well as what you're doing in the private sector.  You know, the testimonies you've been doing across the country also have not been limited to certain groups.  You have Civil Rights groups, you have employers, you have vendors there, so I think that obviously builds long-term guidance that everyone has a part in opposed to unilateral issuing guidance.

So, thank you.  I'm going to move on in the time I have remaining.  I want to -- I'm just going to go with Suresh.  If I may, question for you.  Thank you for your service recently in the federal government.  I know you were back in the private sector now, so you can maybe talk a little more freely that the Bill of Rights we just heard about, I know you were a co-author of that and spent a lot of time in government working on that with the five principles, safe and effective systems, algorithmic discrimination, protections, data privacy, notice and explanation, alternative options.  I'm just going fast for the people who don't know of it. 

So now, and -- and those are -- you know, it's a framework for everyone involved in the private sector and government to use when taking AI into account, the designing it, using it, et cetera.

So now that you're out of the government, what, you know, would've been, if -- it was just more than five, if there was a five through ten or five through seven, what essentially would you like to be in there in the second version of it, what didn't make it in there, and what could be in there that can really help us here at the EEOC, what we're doing that didn't make the first cut?

DR. VENKATASUBRAMANIAN:  Thank you for that question Commissioner.  Actually, I have to say that I think that we spent a lot of -- you know, a lot of time, a lot of consultation figuring out what would be protections that made sense that weren't too many, that weren't too excessive, but that captured the essence of what the concerns that people have been talking about in regard to the AI systems. 

And I have to say these five cover all of it and, you know, we were -- you know, when you look at the regulations, and the guidelines, and the frameworks all across the world, the OECD guidelines, the guidelines and the general partnership and AI and others, they're very much in line with the guidelines we had here, those five principles.  So I think -- at least I, personally, am quite happy with where those five principles landed.

Your other question was, you know, what things didn't make it in, what things I think -- I think it would've been really great, and I think we can -- we're continuing to work on this, is to really spell out the expectations and technical companion.  So I'm a computer scientist and I, you know, I'm a card-carrying computer scientist and I like to say it.  I believe that we have the technology and we have the innovative tools like, things like explainability, things like audits, to be able to make sure that we can actually deploy AI in a way that's great for everyone, but with guardrails. 

And I think the technical companion does -- goes a long way towards spelling of what those might look like, but we can do a lot more.  And I think specific sector guidance, especially in the context of hiring, like that EEOC do, could really push things a long way forward to making these real and making them concrete and tailored for the context where it matters here in hiring in a way that's, again, balances the importance of using AI when it's effective with the guardrails that we want to put in place.

COMMISSIONER SONDERLING:  I appreciate that and as card-carrying lawyers on the Commission here, we certainly need your input on, on how to do that.  In my remaining minute and a half, I do want to get to Professor Kim.  You know, as we've discussed your paper in 2017, Data-Driven Discrimination at Work, you know, I've -- I've read carefully and I -- you know, we talk about -- in that paper, you talk about classification bias and how classification bias suddenly, you know, under Title VII, which is a term that I think you kind of coined or raised awareness of in this paper on how AI can really bring this sort of newer or less used theory of discrimination under Title VII.  In the remaining time, can you just tell us your point on that, you brought up in the paper?

MS. KIM:  Yeah, sure.  So classification bias was something that I talked about in that 2017 paper because I wanted to point out that kind of the point that a number of us had made, right, that the new forms of discrimination, they're concerning.  They look a lot like traditional disparate impact, but they raise some different questions because of the way in which these tools operate. 

And so I think, you know, we can think about them as disparate impact, but I thought it was also important to sort of have another way of conceptualizing it so that we wouldn't necessarily be completely trapped in the existing doctrine around disparate impact if it didn't fit this -- these new technological tools.  And so that was part of the idea to kind of go back to the text of the statute and really think about in 703 A.2, what was it that was intended to be forbidden and could that go a little bit beyond our, you know, sort of a two formalistic understanding of different impact that we might have today.

COMMISSIONER SONDERLING:  And I think it's a, you know, fascinating theory and something that, you know, we should consider how, when applying these new technologies to the older laws, you know, there could be different looks at, you know, a statute you've seen many, many times.  So thank you all very much for testifying and for answering the questions.

CHAIR BURROWS:  Thank you.  And we'll go now to Commissioner Lucas.

COMMISSIONER LUCAS:  Thank you.  I want to follow up with you, Professor Kim, and continue on some of this.  I am very interested, in particular about exploring the scope of textual liability under Title VII.  We talked about classification bias and the classification provisions.  Something I don't think I saw as much in your article in 2017, but perhaps was there, and I apologize if I missed it, was that classification bias, classification prohibitions are also in the employment agency prongs of Title VII, which there are several of them. 

And in particular, I'm just very interested in the scope of textual liability for non-employer entities, particularly employment agencies.  So as I'm sure you know, but for the benefit of the -- of the group discussion, Title VII has its separate liability provision for employment agencies.

And the Title VII defines the term employment agencies to mean an entity that regularly undertakes or with or without compensation to procure employees from an employer or to procure for employees opportunities to work for an employer.  And the Commission has both a 1990 and a 1991 guidance on employment agencies where the operative consideration is whether or not the principle function or major activity is to procure employees or employment opportunities. 

So in light of those background considerations, I was interested in having you explain more why you think that it's uncertain whether an existing law reaches labor market intermediaries, like online advertising and job matching platforms, and unclear whether or not they would be considered employment agencies.

MS. KIM:  So I think that job matching platforms would more easily fall under the definition of an employment agency because those platforms, their primary function is to match job candidates and employers who are looking for -- looking to hire somebody. 

So in terms of the definition, those platforms really are in the business of procuring employees or procuring employment opportunities for job candidates.  That seems to me pretty -- pretty comfortably to fit within the definition of an employment agency.  I think the online advertising platforms is a little bit more ambiguous when you have, for example, Facebook or Google and they are sending out employment ads, but they're also obviously a platform for all kinds of advertising.

And their role is a little bit different from that of the job matching platforms.  I think that's a closer question and I don't know that I have a definitive answer other than I think it's a little bit more on the edge, it's a little bit harder to answer definitively because the -- their role in the labor market is a little bit more diffuse than in the case of the job matching platforms.  I hope that answers your question.

COMMISSIONER LUCAS:  Yeah, that's helpful.  That is sort of a continuum in terms of what you think is clear versus less clear in terms of these third party entities.

MS. KIM:  Of course, I guess I should also mention that it's not the EEOCs bailiwick, but the Section 230 limitation on liability, which the Supreme Court is considering now is the huge elephant in the room in terms of whether you could hold, even if these platforms were considered to be employment agencies, could they be held liable for discriminatory matching? And that 230 piece is an important unsettled question right now.

COMMISSIONER LUCAS:  Well certainly if Facebook, or some other online advertiser, or other entity in that bucket was an employer, we could hold them liable for employment, just employer focused liability provisions?

MS. KIM:  Yes.  Yes.

COMMISSIONER LUCAS:  So it seems like 230 wouldn't necessarily automatically be an impediment there when we're talking about a different parallel prong of Title VII.

MS. KIM:  Yes, I think that's right.

COMMISSIONER LUCAS:  Depending on the facts, could an AI vendor theoretically meet the definition of an employment agency under Title VII?

MS. KIM:  I think that would be possible as well, yes, because they're essentially -- they're building the software that is then being used to procure the -- you know, the worker for the employer or to procure the employment opportunity, so I think that there's a possibility.  Again, it might require a little bit of inquiry into the kind of the technical details of exactly what the vendor is building, and how it's being marketed, and how it's being used by employers. 

But I would certainly think some of them would fall within the definition of an employment agency.  I -- but I also think there's a lot of pressure on vendors because of the employer's potential liability.  They should be asking the vendor to ensure that these tools are not discriminatory.  So I think that there's legal pressure on the vendors from that direction as well.

COMMISSIONER LUCAS:  What about third party interference theory? Obviously that can be addressed in a -- in a variety of ways. It's still a possible (phonetic) theory, can be a little bit more purposeless, but there's arguably textualist considerations there, too, about holding third parties liable for interfering with employment opportunities, given Title VII refers to discriminating against an individual and not simply an employee.  Is that an avenue for potential liability for some of these third parties?

MS. KIM:  I think it -- that is the potential avenue for liability.  I think the third party interference cases, the ones that I'm familiar with are much, much more intentional in nature.  And so they would certainly reach any kind of use of these tools, you know, building of the tools.  For intentional discrimination, I think it may be harder if it's an unintentional effect of, for example, you know, sort of if there's a -- there's a pattern that wasn't immediately apparent or detected until later on.

Although, you know, if there's active auditing going on, it becomes apparent that one of these tools is having a discriminatory effect.  At that point, it might be possible to say, well, if you know it and you continue to use the tool with an awareness of these problems, at that point maybe the third party interference theory could come into play.

COMMISSIONER LUCAS:  Yeah, in my counseling of clients, I sometimes found that they were hesitant to do a disparate impact analysis because then they would find themselves with some knowledge and then struggling about whether or not that actually put them in a better or worse situation and not doing it at all. 

And that brings me to some of your points about the idea that using demographics to audit doesn't automatically result in disparate treatment.  From my perspective, that doesn't seem to be a novel concept in that, again, when you're dealing with like a rift, for example, a large scale reduction in force, after you had conducted a first line decision, you would use demographics to audit, essentially. 

But what seemed to be important there to me was that you had a very bright line between the decision maker and the auditor, and it seems like your proposals possibly would be fusing that.  And that concerns me a little bit, right? That if you build demographics into the algorithm, then you both have the decision maker, the algorithm, self-auditing, but how do you remove the knowledge of the -- of the demographic?

So I don't know, this may be a question more for someone who's technical, maybe you know it, but having that firewall would seem to be an important way there, a way to deal with that.

MS. KIM:  So I think there is -- there is a difficult gray area.  I think there are a lot of uses of race in the model building process that don't cross that line, even though it's the designers of the algorithm that might have access to this information.  So, I mean, a lot of the sort of best practices in building these tools involve things like looking at the data, right? And the quality of the data and the representativeness of the data. 

And one of the ways in which that inquiry should occur is by looking at is it -- is it demographically balanced, right? Are the errors -- we want to make sure the errors aren't concentrated in a particular group.  In designing an algorithm, we should be thinking about what the target variable is.  Have we chosen a target that is neutral or does it in itself have some bias built into it in terms of how we're defining what would make a good employee?

And all those kinds of inquiries require thinking about things like race and gender, right? Is this -- is this target variable going to be biased against women? Is it going to be biased against people who live in certain neighborhoods or as ReNika Moore was talking about, right? Who might have an eviction record without paying attention to the problems with the quality of that data. 

So I think there are a lot of things when building the model, it's important to have an awareness of these characteristics and to make sure they're not causing bias.  There are definitely -- there's definitely a gray area if these characteristics are being used in some way in developing the predictions.  And I think that there's some close questions there, which I will defer because I'm out of time to some of the more technical people you'll be talking to.

COMMISSIONER LUCAS:  Thank you.  It's very helpful.  I appreciate it.

CHAIR BURROWS:  Thank you.  So I appreciate that and I will begin my 10 minutes.  I wanted to start with an issue and going back to those two critical goals, to make sure we, as the Commission, do as much as we can in this area to ensure that the civil rights are complied with and to assist vendors and employers, but also to make sure that there's a public debate and dialog about these things. 

So I'm going to start with what we can do and one of the issues that's came up a little bit in the conversation is that, you know, we have the uniform guidelines for employment and selection procedures, and they sort of represent a consistent federal framework for analyzing whether employment selection procedures could violate anti-discrimination laws.  And they include, among other things, what's known as the four-fifths rule of thumb.

That basically sort of gives a general outline for figuring out whether or not there's a disparate impact or an adverse impact on a particular demographic group.  And it is very clear in those guidelines that that's really a general rule of thumb, not an absolute legal standard. 

And yet in the conversations around how to make sure that there's not discrimination with these tools, it seems that both with the employer vendor community and in general, there's a conversation as though that is the be all and end all. 

So it does seem that that's an area where the Commission probably needs to help employers, vendors, the public understand that the uniform guidelines are a bit more nuanced than what seems to have gotten traction.  I wonder if, and I will address this to all of you, starting with Professor Kim, if you could just comment on that, and do you have suggestions clarifying those nuances?

MS. KIM:  I'm sorry, the audio went out and I just lost the last --

MR. WONG:  Please stand by for a moment.

CHAIR BURROWS:  I apologize.  Yes, let me just say that I was asking about the uniform guidelines and whether or not each of you could, in my view, they're very clear.

MR. WONG:  Please stand by.

CHAIR BURROWS:  Yes.  Okay.

MR. WONG:  Please proceed Chair Burrows.

CHAIR BURROWS:  Thank you.  With respect to the four-fifths rule of thumb in the uniform guidelines, the question that I had is whether or not there are ways in which the Commission might help make it clear that that's just a rule of thumb because that seems to be getting lost and it's almost as though this has become the hallmark of what needs to be done to prevent discrimination in this area.  It's certainly relevant, but not the only relevant factor.  If you could comment on that professor, if you were able to hear me.  If not, I'll repeat.

MS. KIM:  Yes, I did catch at that time.  I'm sorry to have to have you repeat it.  Yeah, that's something that I have observed as well is that I think that there has been a tendency to kind of fasten on the four-fifths rule as a rule.  And of course, it was not ever intended to be a strict rule of liability.  It was intended to be a rule of thumb guiding enforcement actions by federal agencies. 

And the courts in fact don't rely on the four-fifths rule as a way of determining whether or not there's liability.  I do think that's an area in which it would be helpful to have more education about what's required in terms of legally, the test is much more nuanced than that.

And that in past cases, courts and the Commission have looked at things like the pool, the applicant flow data, and so on in ways that are much more nuanced.  I think, and perhaps others are better situated to speak to this, but I think from the computer science perspective, they don't want to read all those cases.  They want to take, have the quick takeaway to know what it is that they need to do.  But I think these increasing conversations between technical people and legal people, I think will deepen understanding on both sides of how that liability should be measured.  And that if it's going to be a much more subtle test than simply a cutoff like the four-fifths rule.

CHAIR BURROWS:  Thank you.  So that's a perfect segue for me to go to Professor Venkatasubramanian.  And if you could just comment, I mean, are there ways that in designing these technologies, the automatic technologies that are being used in employment where people could sort of test for adverse impact without simply relying on the four-fifths rule?

DR. VENKATASUBRAMANIAN:  So funny anecdote, my very first research work on this topic nearly 10 years ago was on using the four-fifths rule.  So it's kind of funny that I got into this space through the four-fifths rule itself. 

To your question, and I think the broader point about measures, I very much agree that the, you know, computer scientists and being one of them, you know, if you have a measure, people will fixate on one particular measure as a way to decide, okay, what we need to do something, this is something we will do it. 

And I think to your -- to your point, there are many ways, in fact, in the research community and beyond for how to evaluate the disparate outcomes arising from any kind of automated tool that don't just rely on the four-fifths rule, that rely on a number of different measures.

So one concrete suggestion I might have is that, you know, as you sort of think about guidance and educational material, you sort of give people sort of a pallet -- a pallet of options to think about.  And also sort of try to explain to folks how these different measures, like the four-fifths rule and others capture different aspects of concern around disparate outcomes for individuals. 

And I think there's a lot of now understanding and the technical side on what these different measures represent and what they don't, right? What their limits are and what their powers are.  And so thinking of this as a -- as a battery of tests that together draw a picture of what the algorithm is doing versus having one measure, I think is an effective way to communicate this to developers and vendors building these tools.  And that's what I would recommend.

CHAIR BURROWS:  Thank you.  So I will go now to Mr. Crenshaw on this, Ms. Moore, if you could each just tell me, are there things that the Commission could do to sort of make clear what the uniform guidelines actually say with respect to the four-fifths rule?

MR. CRENSHAW:  I think this, and Chair Burrows, I think this brings up a very excellent point about the need to educate the community, the business community in particular about, you know, the guidance that's out there.  At the same time, it really strikes home the importance of having open dialog that's iterative and much like some of the processes we've seen over at NIS to ensure that everyone gets a -- view addressed and the process.  As we go forward too, and looking at potential policy recommendations, you know, we believe any framework really needs to have a risk based approach as we go forward.  And those approaches also really do need to consider the benefits that technology provides.

And for example, you know, we need to look at whether or not the technology is opening up new applicant pools that may not necessarily get access to employment opportunities or, you know, we're looking at employee evaluations, for example.  AI actually can be used to take out some of the bias against an employee. 

For example, if an employee is going through a promotional period and you look at reviews and evaluations, AI would actually could self-select out some of the negative reviews, which consumers always tend to more so go to a company and complain.  And so this could actually level the playing field there as well.  So, you know, as we go forward, open transparent processes, risk-based approach, but also looking at the benefits as well.  But excellent question.

CHAIR BURROWS:  Thank you.  And Ms. Moore, I will let you have the last word.  If you could talk about how we could educate around this issue of the four-fifths rule, but also you talked so eloquently about the effects and I really appreciate your testimony with respect to persons of color and impact, the real risk for employment opportunities there.  And so to the extent that you also wanted to speak to educating those communities and opening up this public conversation, I would invite you to address that as well.

MS. MOORE:  Sure.  And I think those actually are related in terms of the understanding of what's happening.  The information deficit has always been there for hiring in that often the candidates don't necessarily know what's happening, but I think with the use of these new technologies that gap has really widened. 

And so one of the roles that the EEOC can play is helping to bring some transparency, and as Suresh commented on, I know in our own organization we have data scientists and analysts, and they also ask us about the application of the four-fifths rule.  And I think there is something very attractive about that hard number that Suresh talked about.

And so one of the things that we can talk about is in the palette that Suresh talked about, what are the baselines that we're understanding in terms of the applicant pool, what are some ways to understand and evaluate the baseline for evaluating disparate impact.  And then I think what we want to be thinking about as well is recognizing and understanding where the data comes from. 

And I think Suresh talked about the technical background we've talked about as model cards, but really understanding where the training data came from; when it was implemented, like this version of the technology, and having some understanding so that other independent auditors can also look to see the value, I'm sorry, the quality of the data that is being used.  And so I think we've got to make more transparent the process for developing the technology, the process for deployment.

And that can then be used both by advocates like the ACLU and other organizations to analyze, but also by independent academics to understand the value.  And then that will also provide value for workers to understand how these processes work.  And I think for notice to workers as well of when these technologies are being used, what rights they have to request accommodations.  I think the ADA compliance was -- extremely helpful, but also understanding what the value is for workers in terms of when information about them is being used under various federal statutes. 

So for example, compliance under the Fair Credit Reporting Act and knowing when in a background check adverse credit information is being used and the EEOC can work with other agencies to make that information more available in terms of their rights of notice and remedy to correct faulty information because so many problems exist with the data that's coming in from these other systems.

CHAIR BURROWS:  Well thank you so much.  Unfortunately, that concludes our first panel.  You all have been incredibly helpful both in your written testimony, and your comments today.  I actually could speak to you all day, but I'm not going to do that to you.  We have a, you know, a number of other future conversations, I'm sure.  So thank you very much, and thank you as well to the Commissioners.  We're going to take a brief recess and we'll resume at 11:25 a.m.  I'll try to get us back on track if that's okay.  Thanks everyone.

(Whereupon, the above-entitled matter went off the record at 11:17 a.m. and resumed at 11:25 a.m.)

CHAIR BURROWS: -- panels of our guests who will be with us today for your participation.  So I would like to introduce each of you in the order of speaking today. 

So starting with Professor Manish Raghavan, who is the Drew Houston Career Development Professor at the MIT Sloan School of Management and Department of Electrical Engineering and Computer Science.  Before taking that position, he was the post-doctoral fellow at the Harvard Center for Research on Computation and Society.  His research centers on the societal impacts of algorithmic decision-making.

We will also hear from Nancy Tippins, welcome, who's a principal of the Nancy T.  Tippins Group LLC, where she brings 30 years of experience to the company.  Her firm creates strategies related to workforce planning, sourcing and recruiting, job analysis, employee selection, succession planning, executive assessment, and employee and leadership development.

We also have Mr. Gary Friedman, a senior partner in Weil, Gotshal & Manges, a nationally recognized employment litigation practice group.  He has served as the chair of the practice and has more than 35 years of experience in employment law. 

Next, Adam Klein, who is the managing partner of Outten & Golden LLP, and founded the firm's class action practice area.  Mr. Klein currently serves as lead or co-lead plaintiff's counsel in numerous major class action lawsuits, including those involving discrimination claims in the financial services industry, the high-tech industry, and credit and criminal history records for employment decisions. 

So welcome to each of you, and again to our witnesses for being here today.  As a reminder, each of you has five minutes for your opening remarks and our IT team will be keeping track of time with the timer that you should be able to see on your screens this morning.  So let's begin with Professor Raghavan.  Thank you so much and you have the floor.

DR. RAGHAVAN:  Great thank you.  Thank you Chair Burrows Vice Chair Samuels, and members of the Commission for the opportunity to participate today.  So my name is Manish Raghavan, I'm an assistant professor at the MIT School of Management and Department of Electrical Engineering and Computer Science. 

As Suresh put it earlier, I'm a card-carrying computer scientist.  I researched the impacts of algorithmic tools on society and in particular the use of machine learning in the employment context.  I've extensively studied the development of these tools and had multiple in-depth conversations with the data scientists who build them.

And so my testimony today will build a little bit on our conversation in the previous panel and go into a little bit of the technical aspects of how the four-fifths rule of thumb has been applied to algorithmic systems in practice.  Now automated systems are increasingly used in these employment context as we've heard, and these modern AI systems need to be trained on data.  They're built to replicate the patterns in those data.

And this is the primary avenue through which past discrimination can be carried on into the present.  Now, without active intervention from developers, automated employment tools will inevitably lead to disparities between legally protected groups.  My testimony today will focus on how developers of these predictive models and practice attempt to comply with the law. 

Now, when I say predictive model, or sometimes simply model, what I mean is a piece of software that takes his input data about an applicant, for example, a resume and outputs a score intended to measure the quality of that applicants.  Now, developers typically create models based on historical data.  So, for example, given a stack of resumes, each annotated with its quality, somehow.  A developer can then build a model that essentially extrapolates these quality labels to new resumes.  Developers of these models often test them to see if they will result in significantly different selection rates between different protected groups, and they primarily use the four-fifths rule of thumb and practice to do this.

Now, as discussed earlier today, this is not necessarily what the law requires, but in my observation, multiple firms have ended up converging on these practices.  Now, importantly, developers run these tests before the model is actually deployed.  And I want to get into the mechanics a little bit of how this is actually done.  So if you're a developer, what you have to do is you collect a dataset on which you are going to measure those selection rates, and you have to hope that this dataset is somehow representative.  That is, that it resembles the actual population who will be evaluated with your model. 

Now, using this data set, a developer can attempt to determine whether the model in question will satisfy the four-fifths rule of thumb.  Often, they use other statistical tests that are more robust than the four-fifths rule, but we'll stick with that for now.

Now, if the model fails such a test, the developer can modify or rebuild it to reduce the selection rate disparities.  And this is where a comment made by Professor Kim comes in that these practices require that a vendor consider protected characteristics while building the model, even though the model itself is agnostic to those characteristics. 

Now one way of thinking about this is that a vendor has to use information about protected characteristics to identify proxy variables that they can then remove from the models.  Under these practices, a firm can try to guarantee that a tool that it releases will not exhibit selection rate disparities. 

In practice, this guarantee is hard to make.  Selection rate disparities depend not only on the model but on the data on which it's evaluated.  So a model that appears to have no selection rate disparities on past data may still produce selection rate disparities when deployed simply because a firm cannot guarantee that the past data will be representative of future applicants.

There are several limitations to this approach.  I'll spell out one of them concretely here, but my testimony, my written testimony contains more details.  One particular limitation of this approach is that it fails to really consider the validity of a tool.  In particular, how this validity might differ across protected groups.  This is often known as differential validity.  And as Professor Venkatasubramanian mentioned earlier, firms tend to test for exactly what they think the law requires and nothing else.  And this is for fear of the exposure that it might bring if they do. 

Now, differential validity has been recognized as a problem throughout many machine learning systems and it has been identified by the American Psychological Association as a particular thing to test for when validating personnel selection procedures.  Yet because it doesn't explicitly appear in the uniform guidelines and firms do not perceive that the law requires them to test for it, they have often let it go onto the radar.

The final thing I will mention is one particular strength of the four-fifths rule of thumb, which is that it can create some benefits by pressuring firms to search for less discriminatory alternatives.  And what I mean by this is equally accurate models with smaller selection rate disparities, and there's been a lot of recent empirical work showing that models with very similar accuracy can actually vary dramatically in their selection rates for different subgroups. 

And so the four-fifths rule of thumb can encourage firms to try to reduce adverse impact without actually reducing the performance of their models.  Thank you for your time and I'm looking forward to any questions you might have.

CHAIR BURROWS:  Thank you very much.  We'll go now to Ms. Tippins.  You have the floor.

DR. TIPPINS:  Good morning, everyone.  The Society for Industrial and Organizational Psychology has set professional standards for employment assessments that are based on scientific research and best practices.  These are called the Principles for the Validation and Use of Personnel Selection Procedures.  SIOP has also developed several documents that clarify how the principles apply to AI-based assessments. 

Much of the principles is aligned with the uniform guidelines and they apply to AI-based assessments.  However, today I'd like to highlight five areas in which the principal's professional standards go beyond those in the guidelines.

First, the principles emphasize the importance of some form of job analysis, not only to justify appropriate measures of work behaviors or job performance as required by the guidelines, but also to determine what knowledge, skills, and abilities, what we call KSAOs, should be measured.  In addition, because a correlation between a predictor and a criterion alone is not sufficient to indicate job relevance.  A job analysis facilitates our understanding of how that predictor relates to the requirements of the job and the criterion measure. 

Second, the guidelines require validation of selection procedures in most situations where there is adverse impact.  From a professional perspective, validation is also necessary to demonstrate the accuracy of a selection procedure regardless of whether adverse impact exists.  Validation evidence is also necessary for employers to evaluate alternative selection procedures.

Third, for psychologist fairness is an assessment is a multi-dimensional concept of many aspects.  One is equal outcomes, which refers to equal pass rates or equal means scores for the cost groups.  That definition of fairness has been rejected by testing professionals, but when found, we believe it should stimulate further investigation into the source of those differences.  Equitable treatment refers to equitable testing conditions, including access to practice materials, performance feedback, opportunities for retesting, and opportunities for reasonable accommodations. 

The principles recommend that employers audit their selection systems to ensure equal treatment for all applicants.  Inform applicants of the ideal conditions for taking an assessment, provide alternatives to applicants who lack proper testing conditions or equipment, and also provide reasonable accommodations.

Equal access to constructs refers to the opportunity for all test takers to show their level of ability on the job; relevant KSAOs being measured without being unduly advantaged or disadvantaged by job-irrelevant personal characteristics such as race, ethnicity, gender, or disability.  Content and format of the assessment mechanism should not limit an individual from demonstrating relevant skills unless they are job related. 

There are two kinds of bias that are very important in employee testing.  The measurement bias refers to systematic errors and assessment scores or criterion measures that are not related to the KSAOs being measured.  Measurement bias may be evaluated by sensitivity review conducted by subject-matter experts who examine items and instructions and determine if predictor is differently understood by demographic, cultural, or linguistic groups.

When hundreds of variables are being used in an algorithm demonstrating freedom from measurement bias, it may be very difficult evaluating each item is not feasible.  Predictive bias refers to systematic errors that result in subgroup differences in the predictor-criterion relationship.  The method for evaluating bias when complex algorithms are used have not been thoroughly researched or tested in court decisions. 

Fourth, documentation of the development and validation of assessments should be considered for computation -- should be sufficient for computational reproducibility.  And encompass all of the information listed in the guidelines as well as details that are specific to AI-based assessments.  Fifth, the guidelines are clear on the requirements for documenting adverse impact of the selection process.

The guidelines describe the four-fifths rule as an appropriate measure of adverse impact, but it may not be sufficient.  Because the principles represent professional standards for employment tests, they do not discuss adverse impact except to admonish IO psychologists to comply with applicable regulations.  In practice, most IO psychologists recognize the complexity of evaluating adverse impact and assess it in a variety of ways. 

In conclusion, AI-based assessments hold the promise of being effective tools for predicting future behavior and systematic unbiased ways.  However, those assessments need to meet legal guidelines and professional standards.  Thank you.

CHAIR BURROWS:  Thank you.  We'll go now to Mr. Friedman.  Welcome.  If you could begin again, we're having trouble with your sound.  If you could try one more time.

MR. FRIEDMAN:  Okay.  Is that better? Thank you.  Chair Burrows and Commissioners, thank you very much for inviting me to testify today before the Commission on this important emerging issue in the field of employment law.  I am a management side employment lawyer who represents businesses in a wide array of sectors across the US and hope to bring the perspective of employers who are using and contemplating using AI in my testimony.  My written testimony discusses these issues in greater detail, but I'm here this morning to just highlight orally several topics.

As an important threshold matter, I can state with confidence that our clients are focused like a laser on using AI responsibly.  And in addition to deploying those tools for speed, efficiency, quality, and performance are equally focused on mitigating the effects of unconscious bias and stereotyping in human decision-making.

So to that extent, I think the Commission, employers and employees are all rowing in the same direction.  Over the past five years, I have seen a markedly increased focus among employers on racial justice and gender equality in the workplace; resulting in growing efforts to use AI tools and other types of automation to enhance diversity, equity, and inclusion initiatives. 

Companies have found, and studies have shown, that the use of AI can reduce unconscious bias in making employment decisions.  A Yale study showed that when evaluating candidates for police chief, human evaluators justifies choosing men without college degrees over women with college degrees because street-smart purportedly was the most important criterion. 

However, when the names and the applications were reversed, evaluators chose men with college degrees over women without college degrees, claiming that degrees were the more important criteria.  If the criteria had been set in advance, unconscious biases against women could have been mitigated because evaluators would not have been able to justify their decisions post hoc.

Another study, AI can reduce reliance on workplace decision-making that is heavily influenced by who you know or the nature of your personal or professional relationship with the individual, is that those types of considerations tend to skew in favor of those who are not in legally protected classifications. 

Illustrative of this is a study of the Fisher College of Business, which analyzed the use of machine learning in selecting board members by comparing human selected boards with predictions of machine learning algorithms.  The study found that in comparison to algorithms-selected directors, management-selected directors were more likely to be male, had larger networks and had many past and current directorships. 

By contrast, the machine algorithm found that directors who were not friends of management had smaller networks and had different backgrounds from those of management were more likely to be effective directors, including by monitoring management more rigorously and offering potentially more useful opinions about policy, suggesting that directors from diverse backgrounds would be more effective.

In terms of the path forward, regardless of the industry, there are some key guideposts that can help companies use AI responsibly and help mitigate the risk of violating the anti-discrimination laws.  Transparency, companies should be upfront about the use of AI as required by some of the state and city laws that have regulated in this space.

Applicants and employees should know when they're being evaluated by a machine algorithm as opposed to a human reviewer.  Auditing, whether it is self-auditing or third-party auditing.  It is important that companies are proactive in mitigating potential biases of AI.  To date, there is a lack of consensus of which metrics and data auditors should use to audit AI technology, how it should become standard practice for auditing companies to disclose the assumptions used for determining relevant protected characteristic used in bias audits.  Vendor vetting, critical, as members of the Commission have stated, The AI tool made me do it, is not a defense to a discrimination claim.

There is guidance out there from a number of organizations such as the Data and Trust Alliance that tell employers that questions they need to ask include what measures are taken to detect and mitigate bias, what approaches are used to remediate bias and what measures have been taken to correct any potential errors.  So I thank you very much for giving me the time this morning and I look forward to answering your questions.

CHAIR BURROWS:  Thank you.  And now we'll go to Adam Klein.  Welcome.

MR. KLEIN:  Good morning.  Thank you Chair Burrows, and thank you members of the Commission for inviting me to speak today.  Very interested in the subject and look forward to an interesting and helpful discussion. 

Let me start my comments by just observing that my role at Outten & Golden is to represent individual employees and applicants who are exposed to or subject to hiring selection using automated AI systems.  And in that context, there are several serious concerns about the use of these types of technologies.  And let me just outline what those are. 

One, and I think critically there's a complete lack of transparency or, if you like, opaqueness in terms of how these systems are deployed by employers for recruiting, sourcing and hiring selection.

And to focus on recruiting and sourcing in particular, a topic that's not been addressed substantively yet today we have identified particular platforms on social media, particularly Facebook and others, that use systems that differentiate potential applicants or recruiting tools that use amenable characteristics as a means to identify and filter potential applicants through a sourcing recruiting channel.  It's commonly used, endemic to the workplace and a serious concern. 

Another major problem that we've identified, and something again that the EEOC should really focus on is a point that Dr. Tippins made, which is that conceptually the starting point of any hiring selection system must include a job analysis and an assessment of the essential functions or competencies of ASIOs of the job.  That is a fundamental step in any hiring selection procedure that is consistent with the uniform guidelines, SI principles and Title VII itself.

And yet there's a conceptual break from that requirement using AI hiring selection procedures.  Essentially, AI systems do not rely on the job requirements for competencies through a job analysis and competency model, and instead looks at correlations based on information that may or may not be relevant to requirements of the job or performance of the job.  That's a fundamental break from the types of selection procedures and systems used by all employers since the start of the Civil Rights Act of 1964's enactment and the EEOC role in this area.

Another core point I want to make about this topic, two things.  One, there's an assumption that there's a practical value in selecting applicants using AI systems.  Meaning that employers using AI technology identify and hire people, applicants who are capable or perhaps better at the target -- performing the target job relative to others.

And that's an unproven assertion.  I've not heard today, and I don't think the Commission has heard any real evidence to support that idea that AI is capable of predicting the performance of applicants in the target jobs.  That is something that's of serious concern. 

Another concern relating to that general point is that the AI systems may instead screen out characteristics or focus on, or favor characteristics that instead we focus on or relate to cost savings.  Like whether a potential employee would be available and not out sick or take leave for other medical reasons as a way to cost save or to save the company money. 

And obviously those kinds of characteristics while understandable from a profit and loss or a revenue perspective or profitability perspective, likely screen out based on gender and age characteristics.  Meaning that there are correlations between the availability of particular applicant or employee and gender and age characteristics.

And I would say to you that age discrimination in particular in hiring is of significant concern and I would say is under enforced and underappreciated in our workforce.  A topic that needs to be focused on, and I have significant concern that the use of AI technologies in particular will simply perpetuate continuing age bias in hiring selection. 

I would also urge EEOC to consider a Cross-Agency coalition working with the Department of Justice and Department of Labor, FTC and the Office of Science and Technology policy and to construct a network or a consortium of subject matter experts who would understand how these technologies work in practice and are able to advise the Commission.  Thank you and I appreciate any additional questions.

DR. TIPPINS:  Thank you very much.  And we will begin now with questions starting with the Vice Chair.  You have the floor.

VICE CHAIR SAMUELS:  Well, thank you so much Chair Burrows, and thank you so much to our witnesses for this invaluable testimony.  As I said in our first panel, I have many more questions than I can ask in 10 minutes, and I do hope that this will be a continuing dialogue. 

Let me start with Mr. Friedman and just reference a study that you mentioned in your testimony about the ways in which AI can potentially be used to diversify boards of directors.  That's an interesting concept and I wonder if you can speak a little bit more about it and also tell me if companies that are using AI for their employees, because they believe it enhances efficiency and maximizes accuracy, helps to promote diversity or to eliminate implicit bias, are also using those tools in selecting members of their boards of directors.

MR. FRIEDMAN:  So, and that's an excellent question, Vice Chair, that particular study was trying to determine who would be the most effective at their role as a board member.  And what it was looking to accomplish was to determine whether -- what -- the population they were taking a look at in particular involved those who had really been supported by management and were frankly being, you know, reelected on a regular basis.  And it was trying to predict whether -- and the criteria for who constitutes an effective director.  Minds can vary in that regard, but among the factors that they looked at is who is going to hold management accountable, who is going to hold them in check, who's going to come up with diverse perspectives, et cetera.

And what the study concluded is that if you look at those who are sort of perpetually elected, those who serve perpetually on boards, it tends to skew towards a particular population, in this case overwhelmingly male, those who have large networks and those who have done this many times before, as opposed to those who may not have had the robust supportive management, I don't necessarily call them outsiders per se, but those who really didn't have the large networks and connectivities. 

And so what was interesting about the study is that it really showed that those who were maybe part of the, quote, inner circle, were not necessarily those who were going to be the most effective in their roles.  And therefore the algorithm was designed to determine, you know, who is going to be most effective in keeping a company on task and coming up with creative solutions to complex problems.  And what it concluded is that if you look at those characteristics, you are more likely to have a diverse board membership in terms of --

VICE CHAIR SAMUELS:  Okay.  Companies that use AI for their employment decisions also used it for board of director selections?

MR. FRIEDMAN:  I think that that is in its infancy, frankly.  I'm not seeing that being done in the same way that it's been done for screening and hiring and assessments, but I think it's in its gestational stages.

VICE CHAIR SAMUELS:  Thank you that's very helpful.  Let me turn to Dr. Tippins.  Thank you so much for your suggestions about ways in which we can expand upon to ensure validity and fairness of tests and AI criteria.  I want to ask you about off the shelf AI products where employers purchase tools from vendors that are not customized for their particular jobs. 

Could you talk a little bit about the ways in which employers need to be sure that a test that has potentially been validated for one job needs to be assessed in the context of a perhaps similar but different job?

DR. TIPPINS:  That's a great question.  From an -- the perspective of industrial and organizational psychology, we would argue that each test used has to be validated with a particular interpretation you're going to make from that test score.  So if a test has been developed and validated on job A, you cannot assume that it will also be effective for job B in making that interpretation that the test is going to predict performance in job B. 

So we would argue that there needs to be a job analysis to understand what the job requires, look at the relationship between that test and the job requirements, and then do a validity study so that there is some evidence to support the predictor criterion relationship and the interpretation you're going to make from that test score.

VICE CHAIR SAMUELS:  That's helpful and I suspect that vendors are not in fact doing that with these off the shelf products.  I don't know if you know.

DR. TIPPINS:  Some do, some don't.

VICE CHAIR SAMUELS:  Fair answer.  Thanks.  Professor Raghavan, you wrote an article a little while ago that focused on issues of model multiplicity and the fact that there are different AI tools that not only can have differing levels of disparate impact if they in fact have disparate impact, but that also may be more or less accurate in predicting success on the job or whatever it is that the employer is trying to assess for. 

Can you talk about the extent to which assessment of alternatives is part of what vendors are looking at, whether that's something that should be more standardized and the extent to which -- sorry, my organization is going to shut down my computer at noon.  Yikes.  The extent to which that ought to be part of the validation analysis that employers and vendors are engaging in.

DR. RAGHAVAN:  Yeah, absolutely.  So when models are trained, what you typically do is you say, Here's the data, find me the model that maximizes accuracy on this data.  And the assumption baked into that is that if there are many models that achieve roughly the same accuracy, you don't care which one you get. 

But there actually might be significant differences between those models.  There might be model A over here that has 90 percent accuracy and very large selection rate disparities and there might be model B over here that has 89.5 percent accuracy and much smaller selection rate disparities.  And so what you have to do is to start to think about what are the trade-offs that you're willing to make as a developer of a model.

If you have these multiple objectives, you might care about accuracy, you might also care about minimizing selection rate disparities.  If you care about both of those things, then you need to specify that you care about both of those things when building and developing a model. 

And so what these new techniques that we're building in the computer science community allow you to do is to say, in this search for a model, find me something that gives me high accuracy, but also tries to reduce selection rate disparities.  And so this, in some sense, automates the search for less discriminatory alternatives because you can say, among these many possible models you can get, can you find me one that reduces selection rate disparities? And we're developing the techniques and tools to do that now. 

Empirically, we've seen evidence to say this is actually impossible in practice.  This is both, you know, reported in the research, also in my conversations with many of the developers of these tools.  This -- what they say is there's actually a lot of heterogeneity among different models.  That there's empirically a lot of room for maneuver -- for maneuvering from the perspective of reducing selection rate disparities.

Now, should this become -- can this become standard practice? I believe so.  It requires some additional work on the part of a developer, there's some cost to it to actually conduct this search to develop an algorithm in a way that actually is sensitive to both of these objectives as opposed to only focusing on accuracy.  I believe that that's worth it and this could be something that the EEOC encourages.

VICE CHAIR SAMUELS:  Thanks, that's very helpful.  And let me use my remaining one minute and two seconds to ask Mr. Klein, you're a lawyer.  As you know, the law provides that a selection procedure -- that plaintiffs have to identify the component of a selection procedure that is responsible for any disparate impact unless those components are not capable of separation for analysis.  How do you think that standard works with AI, particularly when there are thousands of pieces of data that go into predicting any algorithmic formula?

MR. KLEIN:  It's a very challenging problem with software.  Essentially what happens is that if there are several steps in a hiring selection process that leads to a particular candidate being selected for employment and AI is part of that chain of events, then it would be very difficult to identify the adverse impact or the impact of the AI system on the hiring selection process overall. 

But if I could just back up and just point out that I do think there's just a fundamental lack of agreement from the IO community and the computer scientists in terms of what the fundamentals are of how these systems work and practice, and the idea that AI is essentially identifying competencies or the job requirements and then searching for those characteristics.  That's not how these systems work.  And I think that's a very important and fundamental problem that needs to be addressed.

VICE CHAIR SAMUELS:  Okay.

COURT REPORTER:  Apologies, Vice Chair, you're on mute.

VICE CHAIR SAMUELS:  Sorry, I'm out of time, but thank you all for these helpful responses.  I will now turn it back to Chair Rose.

CHAIR BURROWS:  Thanks so much.  So now Commissioner Sonderling, you have the floor.

COMMISSIONER SONDERLING:  I want to start with Ms. Tippins.  Are you -- are you there? So as you know, I've been spending a lot of time with IOs and have really been trying to make sure that the -- both -- everyone involved is aware of your very important practice and all the work you're doing in this area.  And I've often said at the end of the day when, you know, who is going to do those -- this testing? Who can actually assist employers, lawyers, auditors with the actual standards and putting pen to paper.

And I believe that largely is going to fall on IOs.  So from your perspective, can you talk about a little bit what you and SIOP are doing to sort of raise awareness of IOs' place in this big picture here? Again, an area where now that employment assessments and sort of testing on your side are now being digitized or, you know, going to be scaled more, so there's certainly going to be a greater need.  So can you talk about some of the work and guidelines that you've put out through SIOP?

DR. TIPPINS:  Sure.  Just to give a little background, industrial and organizational psychologists are trained to study people in the context of work.  Many of us are very deeply trained in selection and assessment testing.  We have created this document, The Principles For the Validation and Use of the Personnel Selection Procedures, which is aligned with the standards for educational and psychological testing that is produced by the American Psychological Association, the AERA and the NCME. 

In -- because of this change that Commissioner Sonderling is -- has noted with respect to the digitalization of testing and the introduction of AI based assessments, we have taken an active interest in how do our testing standards apply to these somewhat unique forms of testing.

So we have created a document that takes the principles and says, this is what's relevant, this is what's not relevant, and given some guidance on how you might meet the requirements that the principles puts out.  We have also created basic guidelines to say to employers, Here are the things you need to do.  You need to have evidence of your test validity, you need to have evidence of your test reliability.  You need to document this information.  It's not enough for me to say as a professional, oh, I think this test looks good.  I need to have the evidence to support it.

And then I think a third area where we are really active is we are trying to inform the HR community of the -- of the rules and the guidelines and the things they need to do in order to make sure that an AI-based assessment or any other test meets the uniform guidelines but also meets the professional requirements. 

We want to make sure that people are using a test that actually does what it -- they think it's going to do.  Typically, it's predicting performance or some other job related criteria.  And so we want to make sure that -- that they are aware that they need evidence to support the use of that test.

COMMISSIONER SONDERLING:  Well, that -- that's really helpful and it really seems very much geared towards the people who are, you know, not only designing these products, but the employers also, with the ultimate liability of using these products.  And, you know -- and from your perspective and with your specialty, you see where it fits in a larger piece of the puzzle of having that really important knowledge that is now bring back at the table in a way because of these new assessments. 

So certainly, you know, I recommend everyone look at the great work that you all are doing there because it really lays it out, as you heard, specifically related to how to -- how to actually do this.  You hear so much talking about, oh, perform an audit.  Do this, be -- you know, have standards.

And -- but you know, from the IO field, which is obviously different than legal, it actually is giving you some qualifications based upon our longstanding law.  So thank you very much for your work. 

I want to now switch to Mr. Friedman.  And thank you for, you know, your extensive testimony, you've written on this topic for the American Bar Association before we've had discussions.  So you're in a unique position that, you know, you're likely going to have to not only assist employers with creating the lawful use of it on the front end, but if things go wrong, you're going to be also the ones showing up in court to sort of defending these cases.

And you know how to defend Title VII cases, but now with this extra layer of technology potentially making decisions.  So let me just ask you a very broad question as essentially the only representative for the employer community who has the purchasing power for this.  You know, what do employers really want here? And if the Commission is going to actually say something, how can we actually say it in a way that employers can implement, understand, and not violate the law, which I do not believe anyone intentionally wants to do?

MR. FRIEDMAN:  Yeah, now that's a -- that's an excellent question and I think you can draw from existing statutes, both the ADA and also the Equal Pay Act.  So for example, one of the things that my clients have been doing for decades, and actually now we're doing it a lot more frequently, are pay equity audits. 

Now, when you do those pay equity audits, you are basically opening up the hood and looking at the motor, the carburetor and everything else to assess frankly the human capital factors that are necessary in determining whether there's a disparity.  I think employers are far more advanced and skilled in self-analysis here than perhaps is out there in the common understanding because they've been doing it for a while. 

They've also been doing it for a while in the ADA context.  Obviously, you know, going back to the early '90s when you have to determine what are the essential functions of the job, that has been honed over a long period of time.  So this is not a novelty act for our clients.  And the answer to your question, I just think --

(Simultaneous speaking.)

COMMISSIONER SONDERLING:  -- saying overtime audits in the wage and hour context or minimum wage or classification, I mean, employers are used to and comfortable with doing these audits and how do we now bring that same level of comfort where employers are going to hire outside counsel to do it themselves? To do, you know, pay equity audit, which is popular now, or just traditional audits on classification?

How do we get that mindset now for employers to start doing it right now as they're thinking about putting this into effect or as they use it?

MR. FRIEDMAN:  Yeah, and so what -- what I -- what I think in addition to obviously guidance from the EEOC, I think -- I mentioned in my paper, one, a statute as it relates to equal pay as a good faith sort of self-haven, if you will.  If you engage in self-critical analysis of pay issues, you can take that concept and broaden it.  And you want to incentivize our clients, our corporate clients to actually self critically analyze what's going on in terms of the job requirements, in terms of the requirements for selection. 

And the way to do that is to walk that fine balance between, look, requiring transparency.  Everyone that I've spoken to among my clients raises their hand and says, there should be no mystery around what we're doing, particularly at the hiring stage.

And so that disclosure is not a problem, but what they want to be able to do is they want to be able to self-audit and self critically analyze in a way that allows them to do it without a huge risk in the process.  I haven't spoken to anybody who is against the concept of an audit, even an outside audit, as long as they are competent to do it. 

The New York law of course, requires you to publish that on your website and that's still a work in progress as to exactly what's going to be required.  But that's an example of something that will disincentivize our clients to use this technology effectively.  So I think what they're looking for from the EEOC is really guidance, room to maneuver, room to self critically analyze without punishment.

COMMISSIONER SONDERLING:  Thank you.  And in my remaining minute, Mr. Klein, I want to turn to you.  We've had this discussion before and I think it's a really critical point from your perspective, in that the vast majority of employees who are subject to this technology have no idea that they're being subject to an automated tool. 

And I know there's some new disclosure requirements, but just, you know, whether it's a determination about pay, about your location of work or whether or not you get hired, whether you get fired.  A lot of times they may say, well, a human manager didn't -- they didn't like me and it wasn't potentially based on a discriminatory algorithm that, you know, affected me.

And from an enforcement perspective, as a civil law enforcement agency, obviously our mission is to enforce these laws and that's what you do on the plaintiff's side as well.  So you know, what can we do and what are you doing to raise awareness to employees that they may be subject to this technology and that they may have rights under our longstanding law -- laws to file a charge even though they may not know that a computer made that decision?

MR. KLEIN:  Sorry, I have two quick answers to that.  Thank you for the question.  One is, as you've noted, Commissioner, the EEOC itself can issue Commissioner's charges in furtherance of an investigation of potential violations of Title VII and the other statutes, the ADA in particular as well. 

And you're right, look, you're -- the Agency itself, my law firm, the civil rights committee, more generally, we're not going to receive complaints that -- from people who say to us that, I was the victim of algorithmic discrimination, or the victim of discrimination based on the application of the computer algorithm.  It's simply not going to happen.  They have no capacity or ability to understand that they were subject to or exposed to these systems or that they were denied employment or had their rights violated under the various civil right statute.  It's just simply not observable.  So it's a serious problem.

COMMISSIONER SONDERLING:  Thank you.

CHAIR BURROWS:  Thank you.  So we'll go now to Commissioner Lucas.

COMMISSIONER LUCAS:  Thank you.  I want to continue with Mr. Klein along that same line.  So, you know, in terms of disclosure requirements, you've just been saying that many employees won't be able to articulate that it's algorithmic discrimination, they won't really understand it.

Even if you have some sort of disclosure requirement, what kind of disclosure could possibly be intelligible to your average employee or layperson? Or even someone who's relatively sophisticated but not a computer scientist that would be meaningful because obviously any disclosure requirement's going to have some burden on it.  And then you -- you're hopeful that you're -- you're not just creating a paper requirement, essentially.

MR. KLEIN:  Right.  No, it's a serious concern.  Well, I'll say this, I -- you know, the civil rights community at large, so I would include organizations, I would include organized labor.  CWA, for example, has been very active in this area.  AARP, the ACLU, there are a number of different civil rights organizations that have substantial resources and the capacity to understand what people are discussing in terms of their interactions with AI technology, particularly in hiring context.

And so it could be that there are charges that need to be filed with the EEOC, Commission needs to investigate.  I would urge EEOC to coordinate with various other government agencies.  The FTC for example, the Department of Justice has been very active.  Christian Clark (phonetic) leadership, in particular, on this issue with Facebook.  There are lots of tools available to the federal government and it may require a lot more diligence and focus in order to identify and expose problems with these systems in terms of impact on civil rights.

COMMISSIONER LUCAS:  You also mentioned that you're very concerned about age discrimination in this space and in just in general that age discrimination is under enforced.  As a general matter, I'm very concerned about age discrimination and its under-enforcement as well.  It's an interest in mine.  Could you tell me a little bit more about why you think age discrimination in particular is something that is a risk here?

MR. KLEIN:  So one of the issues here is what's driving employers to use automated or AI systems.  And I would assert to you that part of it may very well be cost savings or a profit motive, which is perfectly understandable and reasonable.  But in some sectors of our workforce, it could be call centers, for example, availability of employee -- employees drives profitability.  It drives revenue. 

And so it would make logical sense for employers to filter out or not hire people who may need sick leave or are not as available.  And some of the stereotypes around age and gender may force employers into, you know -- or rather, you know, create opportunities rather for mischief and violations of our statutes because of those --

COMMISSIONER LUCAS:  That -- that's an interesting point.  You know, in terms of the benefits of AI for diversity purposes.  I do find though that sometimes the highest risk for age discrimination in RIFs is when you have some forms of diversity; racial, gender diversity being prioritized over age diversity, and that you could end up having conflicts between different protective characteristics.  Do you view that as a risk from AI in terms of perhaps its use for separation decisions or RIFs or other discriminate -- other sort of employment life cycle decisions?

MR. KLEIN:  I think that that is a serious concern and in fact, you know, you see a lot of adverse impact based on age when RIFs do occur.  That is a serious issue.  I will say, you know, just go back to what Dr. Tippins identified, which is, look, there needs to be a fundamental conceptual framework to understand what the sectional criteria are, either for hiring or for selection for RIF based on a job analysis; based on valid, reliable, fair, decisional processes that are linked to accurate and reliable data. 

And I would urge you or suggest to you that we've not heard a real connection between those concepts and use of these AI technologies, either in terms of hiring or selection for reductions in force.  And so that sort of fundamental disconnect is a serious concern and something that the EEOC really needs to focus on.

COMMISSIONER LUCAS:  Have you seen any complaints alleging algorithm discrimination or use of AI in a -- in a RIF or termination decision?

MR. KLEIN:  I've not personally seen employers use an AI system, at least not that's visible to my clients.  It may be that it's used but not something that the employer wants to talk about or reveal to its employees.  So I don't know, to be honest with you, it's an interesting --

COMMISSIONER LUCAS:  Yeah.  I'm just thinking about the fact that, you know, the OWBPA notices require, you know, disclosure of a variety of positions, and the UC has taken the position in its -- in its, I think, very helpful guidance, that selection criteria need to be part of that notice.  So should AI be used in that, there would need to be some kind of complex disclosure there.  That's already a requirement without us doing anything else.  On the flip side of that, I'd like to hear from Mr. Friedman about whether or not he's seen his clients using AI in any termination or separation or large scale separation decisions.

MR. FRIEDMAN:  Really not much, and -- and I think that there's probably some discomfort at this point in time in going down that road.  I think that my clients have focused on it primarily at the screening/hiring stage, and also in terms of certain performance assessments in certain industries, like transportation for example, where, for example, safety can be measured through artificial intelligence. 

But not seen it -- your point is an excellent one, and I will say this also, and that this is an area in which our clients have been doing this for a long time in terms of reductions in force.

And the analysis that goes with it that correlates with age is, as I'm sure you know, cost is always a factor in reductions in force, and I'm sure you know all the case law with respect to correlation of cost and age.  And I can tell you from my experience, I'm involved in these large scale reductions.  I'm -- I do have a seat at the table, and there is an intense focus on age, as well as race and gender.

COMMISSIONER LUCAS:  That sort of answers one of the questions I was going to ask for you to respond to Mr. Klein's concerns about age and cost.  So from your perspective, you're seeing your clients be very mindful of potential age impacts and try to mitigate them, is that right?

MR. FRIEDMAN:  Yes.  I've definitely seen it.  And there is really, you know, in a -- in a reduction in force, although cost is obviously a huge driver in that process, what I've not seen is sort of an across-the-board, well so this population for the most part, 50 and over, are typically making, you know, 15 percent more than the population that's under 50 or under 40, and so therefore we really need to focus on this group.  I think that the focus is very much on the job.  It's very focused on the department.  And there is an acute awareness of an adverse impact on older workers.  You know?

We can engage in a discussion about the sense of how productive some of these older workers are.  And my clients are often sort of agonizing when there is a position that needs to be eliminated with someone who's been there for 15, 20 years and has been incredibly productive.  So I have seen a high level of consciousness on the issue of age, over -- particularly over the last 10 years.

COMMISSIONER LUCAS:  In terms of other uses of AI, have you seen anyone using AI or algorithms for voluntary diversity initiatives -- putting a thumb on the scale, for example, to incorporate demographics so that you were having a larger set in terms of your candidate pool, or possibly having a preference in the AI to select more, to create some sort of balance?

MR. FRIEDMAN:  Yeah.  Very good question.  And the answer is I'm beginning to see it.  I'm beginning to see it in a variety of ways.  One of them is casting a wider geographic net, you know, particularly in a more remote-oriented world where, a lot of hybrid arrangements.  We all know that there can be bias associated with race.  If, for example, you're hiring only geographically proximate to your place of business, I have seen a focus on widening the net.

I've also seen a focus on widening the types of, for example, colleges and universities from which businesses seek to draw.  And a real focus on, you know, rather than drawing assumptions or conclusions based on a certain swath of colleges and universities, it's expanding that base which is pulling in a more diverse population.

COMMISSIONER LUCAS:  Thank you.  Very helpful.

CHAIR BURROWS:  Thank you.  So I will go next to Professor Raghavan.  Just back to first principles with respect to how we, shall we say, build, you know, under the hood, if you will.  And one of the things that I thought was really interesting about your testimony was this concept of differential validity, and I'd like you to just expound on that a little bit, that -- you know? We've heard a lot today about the importance of what training data is used, and how good the data is with respect to various demographic populations.  And talk to me a bit, if you could explain how you might get differential validity, and what significance that may have to this conversation.

DR. RAGHAVAN:  Sure.  So the first thing to recognize is that different populations are distributed differently.  People go to schools in different places.  People have different activities that they do.  They have different work histories, and so on.  And what that means is that a predictor that works well for one demographic group may not necessarily work well for a different demographic group. 

The other thing to keep in mind is that with machine learning, with AI, with all these automated tools, the general rule is that the more data you have, the more accurate your predictions will be.

Now if you combine these two things together, what you get is the following.  If you have more data from historical applicants from one demographic group relative to another, then you should expect that your predictor will perform more accurately on that demographic group than the other.  Now one place that this has manifested quite publicly is in facial recognition systems.  There was a lot of public examples about how facial analysis and facial recognition systems did better on male and lighter skinned faces, and this was primarily because a lot of their training data, the vast majority of their training data, was male and lighter-skinned faces.

Now you might be worried the same thing might happen in the employment context, where if a lot of your applicants in the past have been from particular demographic groups, then that is where your predictor will perform the best, at the expense of other demographic groups.  And that's how something like differential validity can arise, where you're able to make very accurate judgements, or at least more accurate judgements, about people who look like the people that you've seen in the past, and less accurate judgements about people who you haven't seen frequently in the past.

CHAIR BURROWS:  Thank you.  That's helpful.  So even if you have something that may be valid in selecting candidates and tell you who's a good candidate from a particular demographic, if you -- your training data hasn't really been in sort of diverse if you will, it may not work as well for other demographic groups. 

Since you mentioned the question of, you know, facial recognition, it reminded me that we've started to hear about the use of that actually in employment, other kinds of tests.  We talked a lot about hiring and recruitment today, but are there other kinds of, you know, areas in which these selection devices, or evaluation devices, are being used, other than the just, you know, what we sort of commonly think of as hiring and recruitment?

DR. RAGHAVAN:  I think I would take this hiring task and actually break it up into multiple stages, because I don't think it's one discrete decision, right? There's, you advertise your job.  Some people interview or some people apply, and then you select some of those resumes, and then you interview certain people, and then you make offers to people.  Each of those is a different stage of the pipeline and requires different tools. 

So for instance, we heard a little bit today about the variance reduction system that Meta is implementing in their advertising system, right? That is going to be a very different type of application and a different solution to what you might see for resume screening, or for perhaps making predictions about salary that you should offer to someone.

These are all completely different applications, I think have different considerations.  Now beyond the recruitment context, I haven't seen quite as many that get used throughout the employment life cycle.  There are some in employee evaluation, and then some for termination, reduction in force. 

But I think at least the ones that garner the most attention, and in my estimation have the most use, are in recruitment.  I would say that they're primarily things like resume screening.  And then also large platforms like LinkedIn, CareerBuilder, Monster, who are using machine learning and algorithms on the backend to determine -- for instance, you run a LinkedIn search, who shows up in your search results?

That is a machine learning system, or an AI system, working back there, which follows a pretty different format from the ones that we've been discussing, because the goal of that system is to say, take a search query, software engineer, and return a ranked list of results.  And this has different implications for who ends up near the top of the results, who shows up on Page 2 and never really gets seen as much, so the analysis that you might do there is a bit different. 

So I would say even within recruitment, there's a wide variety of applications that we should be thinking about that don't follow this -- the simple format of somebody applies and you make a yes/no decision on them.

CHAIR BURROWS:  Thank you.  That's helpful.  Going back to this question of differential validity, I take it from your written testimony that, that's something that, you know, as you're designing such a system, you could check for.  Can you talk to me about that as well?

DR. RAGHAVAN:  Right.  So often when you -- when you're building and designing a system, you'll evaluate it.  And one of the things that you'll commonly see in an evaluation is validity, right, aggregate across the entire population.  What is less common is to see that validity disaggregated by different parts of the population or for different demographic groups.  And I think that disaggregated evaluation is actually quite important.  It's definitely feasible.

The challenge is that you do need to have demographic data on the people in your training set.  Now this often is the case in employment context, but may not always be the case.  Now when you have that demographic information, then you can say, I can not only report the validity of this model, or of this assessment, I can also report the validity on men, or the validity over women, and I can compare those numbers and say, Are they similar to each other, or are they dramatically different?

And if they are dramatically different, that is an indication that your assessment isn't going to perform the way that you want it to in practice, that it will perform better for certain subgroups of the population than others.  And it is an indication that the primary thing that you should do to fix that is there's likely a problem with the data, that it's imbalanced towards one group versus another.  Now there are other problems that can cause differential validity, but that in practice is the most common source.

CHAIR BURROWS:  Thank you.  That's very helpful.  I wanted to go now to Mr. Friedman.  And one of the things that I've been thinking about a lot has been in the ADA context, which obviously you know, as you assess discrimination under the ADA, it works differently than under Title VII with respect to whether or not a candidate might be screened out of a process. 

And one of the things that I wanted to ask about, you mentioned transparency as one of the concepts that has been coming up in your practice in this area, and I'm curious as to whether or not you think there are things that we as a Commission could do to encourage employers to really focus on transparency with respect to persons with disabilities, or particularly if they need some sort of accommodation. 

And in particular, if you have an automated -- a purely automated process, it occurs to me that you will find it difficult to raise your hand and ask for that interactive process that could lead to getting a reasonable accommodation if you need one for disability.  So I was wondering if you could just speak to that generally.

MR. FRIEDMAN:  Sure.  So maybe I can answer that, and it's an excellent question, by giving you an illustration from one of my clients.  So they use -- in hiring for a particular set of compliance positions, they will use an online tool.  It's a 45-minute time test that is designed to ascertain who's a good researcher and who's a good writer. 

And I would say that one of the benefits of this is that they're less interested in focusing on, you know, PhDs in English, they really are interested -- and, you know, the top schools you went to -- they're really interested in focusing in on how effective you are.  Because the algorithm scores the exam, during the 45 minutes, at the -- at the end, based on your ability to research and write.

To your question, Chair Burrows, let's say someone has a generalized anxiety disorder, or, you know, a mental disability, or ADHD, or something like that, you might say, well they're not going to perform as well on a timed test.  What are they supposed to do under these circumstances?

And in answer to your question as to how the EEOC can be effective in this space, in that kind of context, I know traditionally you're not supposed to raise any issues with respect to your disability, pre-offer, and in this instance, right?

If you really wanted this position, you were a great researcher and a great writer, but you were anxious about being able to accomplish it in 45 minutes, I think what would be useful is the EEOC should create some guidance for employers to be able to accommodate someone who is going to be just as effective, once they hit the job, in researching and writing in that compliance position, without actually being adversely impacted by virtue of the fact that maybe that particular time test did not inure to their benefit for a sort of maximum result.  So I think guidance in that space is going to be very important for the EEOC.

CHAIR BURROWS:  Okay.  So I think I'm out of time.  I was, you know, just wanted to commend all of you for your time here, and the excellent, truly excellent written testimony.  I was hoping to get a chance to ask questions of Dr. Tippins and Mr. Klein, so I apologize for that.  But I'm going to try and stick to the schedule that we promised to you and to my colleagues here at the Commission, and so I will say thank you and wrap up this round. 

We are at time for that, and that concludes our second panel.  So we will be breaking now for lunch and I would say let's come back at 1:30 so that we can stay on time.  Thanks so much.

(Whereupon, the above-entitled matter went off the record at 12:27 p.m. and resumed at 1:30 p.m.)

CHAIR BURROWS:  Welcome back to our hearing on automated systems and AI in employment.  Welcome also to our members of the public, and the fellow Commissioners, and of course those who are speaking today and who have sent us such excellent testimony to share your insights.  So with that, let me introduce the speakers on our final panel in the order that they will be speaking today. 

First, Matt Scherer is senior policy counsel for workers' rights and technology policy at the Center for Democracy and Technology.  His work focuses on the use of AI in hiring and other employment decisions, workplace privacy and surveillance, and helping workers to use data and technology to empower themselves.

Next, Heather Tinsley-Fix is a thought leader and influencer in age diversity, working to advance the value of older workers and the business case for building age-diverse and age-inclusive organizations.  As a senior advisor of employer engagement at a AARP, she leads the AARP Employer Pledge Program, a nationwide group of employers that --

MR. WONG:  Please stand by.

CHAIR BURROWS:  -- Ajunwa, tenured law professor (audio interference) School of Law, and an adjunct professor of business, where she is a Rethink Lab fellow (audio interference) director of the artificial intelligence decision-making (audio interference) Berkman Klein Center at Harvard University, since (audio interference) was a research interest are at the intersection with a particular focus --

MR. WONG:  If our attendees would please stand by while we resolve the technical issue.  Chair Burrows, could we please begin again, and could we please begin again with the bio from Heather Tinsley-Fix?

CHAIR BURROWS:  So I'm sorry, I'm not hearing you.  From which witness?

MR. WONG:  Apologies, Chair Burrows.  We are going to have to resolve the connection issue with the computer before we can begin, as your audio is too distorted.  So I think we have support coming for you now.

CHAIR BURROWS:  Thank you.

MR. WONG:  I'm just going to put you on mute now, Chair Burrows, while the issue is resolved.  If our witnesses could please stand by.  Thank you to our attendees.  If you could please stand by with us, while we resolve a technical issue.

VICE CHAIR SAMUELS:  I'm Vice Chair Jocelyn Samuels, and apologies for our technical difficulties.  Wouldn't you know that in the hearing about artificial intelligence, we would end up having to deal with technical issues.  But I'm just going to start from scratch to introduce our panelists so that we can get right into their testimony.  I think that Chair Burrows introduced Mr. Scherer, who I am just now scrolling to, where I can -- apologize.  Excuse me one second.  Apologies.

CHAIR BURROWS:  Hi.  Is everyone able to hear me?

VICE CHAIR SAMUELS:  Oh, are you back?

CHAIR BURROWS:  I am.  And I so apologize.  It is such a pleasure to have such patient and thoughtful colleagues, so thank you, Vice Chair Samuels.  And I think what I will do is return to introduce Professor Ajunwa -- I think I was just starting there, if that is correct.  Bear with me.  I did not know I had lost you all.  So very happy to have you here.

And once again, Professor Ajunwa is a tenured law professor at the University of North Carolina School of Law and an adjunct professor at the Kenan-Flagler School of Business where she is a Rethink Lab fellow.  She's also the founding director of the Artificial Intelligence Decision-Making Research Program at UNC Law and a faculty associate at the.  Since 2017.  Professor Ajunwa's research interests are at the intersection of law and technology, with a particular focus on the ethical governance of workplace technology, so perfect for today's discussion.

Also, Alex Engler is a professor at the Brookings Institution where he studies the implications of artificial intelligence on governance, with a focus on social policy.  So he recently returned from Berlin and Brussels, where he worked at -- on the European Union's AI Act as a Fulbright Human Innovation scholar.  So welcome to all of you, and you will each have five minutes for your opening statements.  Thank you for your patience, once again, and we'll begin with Mr. Scherer.  You have the floor.

MR. SCHERER:  Chair Burrows Vice Chair Samuels and Commissioners good afternoon.  My name is Matt Scherer, and I am senior policy counsel for workers' rights and technology at the Center for Democracy and Technology.  CDT is a nonprofit, nonpartisan organization that advocates for stronger civil rights protections in the digital age. 

CDT's workers' rights project examines, among other workplace technologies, automated employment decision tools or AEDTs.  While these tools can improve the efficiency of the recruitment and hiring process, they can also interfere with workers access to employment and limit their advancement opportunities.

CDT has worked with a broad coalition of civil rights and public interest organizations, over the past several years, to develop principles and standards regarding these technologies that center and advance the interests of workers, particularly those from historically marginalized and disadvantaged groups. 

As the Commission is aware, more and more employers are using new hiring technologies to make critical employment decisions.  But history shows that while technology has the potential to make work and workplaces safer, fairer, and more accessible, not all new technologies live up to their hype, and in certain cases they've even caused great harm.

The stakes are especially high with AEDTs, which could impact the careers and livelihoods of countless workers.  Policymakers and employers alike should scrutinize AEDTs carefully, due to the unique risks of discrimination they pose.

Today's automated tools rarely, if ever, make an effort to directly measure a worker's actual ability to perform the essential duties and tasks that will be expected of whoever the employer hires for a given position.  Instead, some claim to assess workers based on personality, or others, subjective characteristics untethered from actual job duties. 

Others use correlation-driven machine learning methods that can lead the tool to focus on irrelevant and potentially discriminatory characteristics.  Such tools pose a risk of discrimination against already disadvantaged groups of workers who often are underrepresented in the data used to train AEDTs, and whose relevant skills and abilities may not be as obvious to an automated system.

The current legal landscape needs clarification and refinement to address these concerns.  The uniform guidelines do not adequately reflect the many changes in law and social science that have occurred in the five decades since they were drafted.  Perhaps even more disturbingly, some vendors and allied special interest groups have actively sought policy changes that would weaken or undercut existing protections, or confuse employers and workers about what current law requires. 

It was against this backdrop that CDT partnered with many of the nation's leading civil rights organizations, including the ACLU, which is also represented at today's hearing, to create the civil rights standards for 21st century employment selection procedures.

The standards were published last month with the endorsements of 13 civil and digital rights groups.  The standards provisions would detect and prevent discrimination by requiring that all selection tools be clearly linked to essential job functions; mandating pre-deployment and ongoing audits to ensure tools are non-discriminatory and job related throughout their life cycle; ensuring that companies select the least discriminatory assessment method available; prohibiting certain tools that propose a particularly high risk of discrimination, such as facial analysis and personality testing. 

Adopting the standards would also improve transparency and accountability by creating multiple layers of disclosure requirements, including concise candidate disclosures, detailed audit summaries, and comprehensive record keeping by ensuring candidates can communicate concerns; mandating clear procedures for disabled candidates to access accommodation; and giving candidates a right to human review, in the case of automated tools.

While the rise of automated decision tools was the impetus for this work, the standards apply to all formalized selection procedures.  They provide a roadmap to managing the risks associated with modern selection tools, while centering the rights and dignity of workers, particularly those most at risk of technological discrimination.  They also provide a concrete alternative to proposals that would set very weak notice audit and fairness standards for automated tools, and my written testimony describes the standards in the other topics I have discussed today in greater detail.

As the Commission completes its strategic enforcement plan over the coming weeks and months, we urge it to use its platform and authority to ensure that workers, not machines, remain at the center of the future labor market.  The rights of workers, particularly vulnerable and marginalized workers, must not be trampled or glossed over for the sake of convenience or efficiency.  Thank you.

CHAIR BURROWS:  Thank you.  We'll go now to Ms. Tinsley-Fix.

MS. TINSLEY-FIX:  Thank you Chair Burrows and distinguished Commissioners.  On behalf of our 38 million members and all older Americans, thank you for the opportunity to speak to you today.  AARP believes that any type of discrimination in the workplace is unacceptable.  And too often when discussing discrimination, age is not included, although ageism continues to be a widespread problem. 

In terms of age bias and discrimination, the potential pitfalls associated with the use of AI in hiring and workforce management platforms are, at the root, the same for older candidates as they are for other protected classes -- namely, the quality or relevance of available data and the normative judgements baked into the process of what good looks like.

However, the way those pitfalls affect older workers can look a little different, or come from unexpected places.  So here are some examples, in terms of the type and amount of data collected.  To the extent that algorithms scrape and use data from social media posts and activity from professional digital profiles or internet browsing history to power their predictive rankings, older adults may be left out of the consideration set, due to either a lack of those types of data in their digital footprint, or the fact that fewer older job candidates are considered when building ideal candidate profiles. 

Furthermore, any data point collected that explicitly reveals or serves as a proxy for age, such as date of birth, years of experience, or date of graduation, can be noticed by the algorithm as part of a pattern denoting undesirable candidates, and signal the algorithm to lower their ranking or screen them out entirely.

Number two, cultural norms.  There are a host of unconscious assumptions baked into our culture that associate age with slowing cognitive decline, and inability to learn new things and resistance to change.  These norms inform the way job descriptions are worded, target variables are defined, interviews are conducted, and assessments are designed and scored.  For example, if reaction time is a variable on which candidates are scored, older workers may be at a disadvantage.  Research shows that older brains exhibit slower processing speeds, but greater contextual knowledge. 

However, if skills assessments or, for example, the analysis of interview footage are optimized towards the performance of younger brains by the data scientists working on them, older workers could receive arbitrarily lower scores.

Additionally, older workers could be excluded at the start of the hiring process because they never see the job ads to begin with.  In 2017, ProPublica revealed that Facebook was allowing organizations to age target their employment ads.  And this can also include the way job descriptions are worded.  So phrases like recent college grad, and digital native are explicitly ageist, but even subtle references such as fast-paced, high energy, and super fun have been shown to deter older workers from applying. 

And finally, there's the feedback loop of decisions taken by recruiters or hiring managers.  To the extent that algorithms learn from the preferences and decisions made against older candidates during the recruiting process, they will spot the patterns in the data that indicate an older candidate, and subsequently promote those candidates less frequently and less far through the automated process.

So for example, if an older candidate makes it past the resume screening process, but then gets confused by, or interacts poorly with, a chatbot, that data could teach the algorithm that candidates with similar profiles should be ranked lower.  This also applies to performance data. 

So again, to the extent that performance evaluations or other employment-related decisions -- such as who is selected for training, who gets assigned to innovative projects or high performing teams -- the extent to which those decisions are influenced by the ageism of human actors, and then that data is fed into ranking algorithms as proof points, older workers could be disadvantaged.  So what can employers and the EEOC do to mitigate the risks of this kind of discrimination?

In terms of practical guidance, there are many steps employers can take to specifically address the risk of unintended age discrimination and bias.  First, stop asking for age-related data in applications, such as dates of birth or graduation, unless there is a proven business reason to do so. 

Second, pay close attention to the words used in job descriptions, and don't cap the years of experience required.  So replacing the phrase two to five years' experience with at least two years' experience signals that candidates of all ages are welcome to apply.  Don't age target employment ads on platforms that allow such targeting.  Just don't do it -- even if that includes filters that approximate age, such as job seniority or years of experience.

Look for vendors who work with certified IO psychologists -- in particular, as has been noted before by other panelists, any use of non-employment-related data should be vigorously scrutinized for its potential to rely on correlation rather than causation.  Request regular and independent audits of algorithm performance. 

Include age as an element of diversity, equity, and inclusion initiatives.  Driving awareness of the value of age diversity at work will help shift a culture of unconscious ageism.  And finally, empower recruiters to challenge hiring managers who exhibit conscious or unconscious preferences for candidates based on age.  There is a strong business case for the inclusion of older workers as part of an age-diverse workforce.

Very quickly, on the legislative front, AARP supports both federal and state level initiatives to ban age-related questions during the application process.  Connecticut and Delaware have enacted such bans.  And at the federal level, AARP supports the Protect Older Job Applicants Act, as well as the bipartisan Protecting Older Workers Against Discrimination Act.  Again, thank you for providing us the opportunity to testify today, and I look forward to answering any questions.

CHAIR BURROWS:  Thank you.  Professor Ajunwa?

DR. AJUNWA:  Greetings to all.  Chair Burrows and Commissioners thank you for inviting me to testify at this important public meeting on employment discrimination in AI and automated systems.  I am a law professor at University of North Carolina School of Law, where I am also the founding director of the AI Decision-Making Research Program.  I have previously testified before this Commission in 2016 and in 2020, I also testified before the Congressional Education and Labor Committee on the issue of workers' rights in the digital age. 

First, I make note of the business trend towards the use of automated hiring programs.  The top 20 private employers in the Fortune 500 list all make use of automated hiring systems.  This list is comprised of mostly retail companies with large numbers of workers in low wage or entry level work.

It is true that many businesses turn to automated hiring in an attempt to diversify your workplace.  Yet there is evidence that such algorithmic decision-making processes may stymie the progress of anti-discrimination laws.  Automated hiring programs are initially advertised as a way to clone your best worker, a slogan that in effect replicates bias. 

This is precisely because automated hiring programs take the nebulous concept of cultural fit for a given corporation organization and concretize it into select algorithmic variables that are stand in for protected categories.

The creation of this proxy variables can have racial, gender, and age discrimination effects in contra intervention of anti-discrimination laws for which this Commission has regulatory power, such as Title VII of the Civil Rights Act and the Age Discrimination and Employment Act.  I'll illustrate this problem with an example shared by an employment and labor lawyer who had been hired to audit an automated hiring system. 

As reported by the lawyer when the automated system was presented with training data and then queried as to which two variables were found to be the most relevant, the system reported back that those variables are the name Jared, and also whether the applicant had played high school lacrosse.  As confirmed by Social Security records, the name Jared is highly correlated to individuals who are both white and male.  Furthermore, lacrosse is a sport that isn't found in all high schools.

Rather, it is an expensive sport that is found in well-funded high schools, located in affluent neighborhoods that are more likely to be predominantly white given the history of racial segregation in the United States.  Thus, in this insidious manner, proxy variables as part of automated hiring systems can enact unlawful racial and gender employment discrimination. 

Finally, I point to the use of automated video interviewing with facial analysis and emotion detection as an inherently flawed and discriminatory automated hiring practice.  In 2018, 60 percent of organizations were using video interviews, but the use of automated video interviews sharply decrease -- increased to 86 percent in 2020 due to the COVID-19 pandemic.  This automated practice is akin to disproved pseudoscience of phonology and thus should be banned. 

Automated video interviews that are scored by speech algorithms provide opportunity for accent discrimination and the ones that claim to detect emotion from facial analysis, further enable racial and gender discrimination given cultural and gender differences in how emotions are expressed.

I offer four proposals that EEOC should consider as part of its enforcement of employment anti-discrimination laws.  First, the EEOC should consider the addition of a third cause of action for Title VII of the Civil Rights Act.  A third cause of action, which I call discrimination per se, would shift the burden of proof from applicant to employer so long as an applicant is able to point to a feature or requirement of an automated hiring system that seems egregiously discriminatory for a particular protected class.  The employer would then bear the burden of showing what the statistical results of the audits of the automated hiring system that identified automated hiring feature does not in fact have a disparate impact on a protected class. 

Second, the EEOC should mandate employer audits of any automated hiring systems in use.  I argue that the EEOC should require external audits as internal audits are not enough. The EEOC could choose to take on these -- external audits, or it could certify third-party vendors that would provide those audits. 

Third, the EEOC perhaps incorporation with FTC should develop its own automated governance tools in the form of AI or automated systems that could then provide audit services to corporations deploying automated hiring systems.

Finally, the EEOC should release an advisory notice that the unite -- the Uniform Guidelines on employee selection procedures govern the use of a -- of variables used in auto algorithmic hiring, and that the design of automated hiring systems should be to retain full records of both completed and also failed application attempts.  All variables selected for automated hiring systems should then meet criterion content and construct validity for the specified job position. This will lessen opportunities for proxy variables to be deployed from lawful employment discrimination against protected categories. 

With that, I once again expressed my gratitude to Chair Burrows her fellow Commissioners for the opportunity to present these remarks and proposals in support of the EEOC mission of equal Employment Opportunity for all.  Thank you so much.

CHAIR BURROWS:  Thank you.  And now we go to Mr. Engler.  You have the word.

MR. ENGLER:  Good afternoon.  My name is Alex Engler.  I'm a fellow at the Brookings Institution, an associate fellow at the Center for European Policy Studies and an adjunct professor at Georgetown University.  I primarily study the interaction between algorithms and social policy, and this research is informed by a decade of experiences as a data scientist. I thank the Commission for the opportunity to offer testimony today, and I note with apologies that I have used the privilege of going last to make some small changes to my submitted text so I can emphasize what you haven't already heard today.

First, I want to commend the EEOC on last year's technical assistance detailing how AI hiring tools can be discriminatory against people with disabilities and how their developers might comply with the American with Disabilities Act.  I applaud this work and will continue to hold it up as an example to other federal agencies, especially for how it considers the entire socio-technical process of hiring and not just the algorithms alone.

This work from EEOC is especially encouraging because the story of AI hiring is not unique.  Almost all critical decisions in the employment context are experiencing algorithmization.  This includes AI applications for targeted job ads, recruitment, background checks, task allocation, evaluation of employee performance, wage setting, promotion, termination, and others.  While AI hiring receives the most attention, these other AI employment systems are used by thousands of businesses and affect millions of Americans.  The prevailing evidence suggests it's a bit uncertain, but it suggests that for medium and small businesses -- or medium and large businesses, algorithmic systems will soon contribute significantly to or perform outright the majority of all employment decisions in the categories mentioned above.  That most employment decisions will be assisted by or made by an AI system, is a sea change in the employer employee relationship, and in turn requires a significant adaptation by the EEOC.

Continuing the work of the AI and Algorithmic Fairness Initiative, the EEOC should systemically consider these AI applications, develop tailored guidance for each under all of the AI's -- EEOC's legal authorities, including Title VII of the Civil Rights Act, the Americans with Disabilities Act, the Age Discrimination Employment Act, the Equal Pay Act and others, and also further build necessary capacity for information gathering and enforcement that I'll talk about in a minute. 

This is an enormous undertaking.  It will take time and dedicated staffing and resources, and I also expect that over time it will affect the structure and core functions of the EEOC.  While a great challenge that is the appropriate response to the new algorithmic paradigm in employment.

The European Union has recognized the gravity of this challenge.  The EU's AI Act is focused on the use of algorithms in socioeconomic decisions, including access to public benefits, financial services, as well as employment.  Most relevant, the EU AIF will categorize almost everything I've mentioned so far as a high risk AI application.  That's job ads, recruitment, background checks, task allocation, employee performance, promotion, all those categories will be high risk. 

As a result, covered AI developers will have to meet standards for the input data, for the accuracy and robustness of those models, for transparency and explainability to users for some degree of human oversight, as well as writing technical documentation.  All of these requirements, this is potentially of interest to the EEOC, all of these requirements necessitate the creation of standards, which may help inform some of the difficult questions that the EEOC may also face relevant to the regulation of specific AI employment applications.

The EU AI Act will also require that developers register their AI systems and perform and preserve the data reflecting its use and performance, which EU regulators are then empowered to review.  This effectively enables AI audits, algorithmic audits to ensure conformity with the AI Act. 

The registration will also lead to a public database of all of these systems.  Also potentially an informative resource for the EEOC.  Much like the EU, in addition to new regulatory guidance and technical assistance, the EEOC should actively consider developing new AI oversight capacity, especially while further developing the draft strategic enforcement plan.

An important takeaway from my Brookings research is that a -- the transition to an AI employment systems may represent a possibility for a more just labor market, but those better outcomes are absolutely not guaranteed.  While an effective and independent auditing market might help self-regulate AI systems, that's not going to emerge on its own.  By acting as a regulatory backstop and creating real stakes for violating anti-discrimination laws, the EEOC has the potential to change these incentives for the better.

A few things you can do in terms of building that capacity.  The EEOC can continue its efforts to hire data scientists, especially those who specialize in regulatory compliance or algorithmic auditors.  Aside from these and -- view of enforcement actions, the EEOC can look to acquire and evaluate AI employment systems in order to improve its own understanding, as well as public knowledge of AI employment systems.  This might be informed by the EUAI Act database or the National Institute for Standards and Technologies Facial Recognition Testing Program, which evaluates facial recognition software and publishes results.

Both enforcement actions and more public information can influence the key players as we've talked about a lot today, the vendors of employment systems.  These vendors have enormous amount of leverage over the market and influencing their behavior is key to leading to a more equitable and transparent outcomes in the future of an algorithmically driven labor market.

In the short and medium term, the development of new EEOC capacity for algorithmic oversight is a significant hurdle, but this will be just as critical as the development of new policy guidance and technical assistance in order to ensure the just application of AI and employment.  I, again, thank the Commission for its invitation today and I welcome your questions.

CHAIR BURROWS:  Thank you.  So we will go now to questions from each of the Commissioners, beginning with Vice Chair Samuels, and each of us will have 10 minutes.  Thank you.

VICE CHAIR SAMUELS:  Thank you so much, Chair Burrows, and thank you to all of our witnesses.  As has been true of every panel today, you all have shared such valuable insights and so much expertise in this big new civil rights frontier.  I really do hope that this can be part of a continuing dialogue where we continue to learn from all of the valuable work that you're doing. 

But let me start with Mr. Scherer.  I found your civil rights standards very thoughtful and interesting, and among them you call for expansion of recordkeeping requirements so that individuals, stakeholders, vendors, employers, can learn from the work that developers of AI have done. 

Can you say a little bit more about what documentation you think entities are required to keep now and what you would expand beyond what may be the current requirements?

MR. SCHERER:  Thank you, Vice Chair.  Right now, the Uniform Guidelines require, and the related EEOC regulations regarding EEO 1 forms, et cetera, require employers to keep a fair amount of documentation relating to the outcomes of their selection processes, the outcomes of their hiring decisions. 

They typically do not require a great level of technical record keeping when it comes to how it was that they selected the selection procedures that they did, how those selection procedures operate, the manner in which candidates are given notice require -- regarding what selection tools are being used and how they operate, what provisions for accommodation were given to candidates and how it was explained to them, or whether it even was explained to them, how they could access accommodations.  A lot of that information has not been in place, in part because the Uniform Guidelines were developed before the ADA came into effect.

So the Uniform Guidelines, which is the main source of information that employers have to maintain regarding the selection procedures that they use, don't take into account a great deal of the rights that workers now have under federal law. 

So really from the ground we tried to kind of come up from a ground up perspective when we were developing the standards and thinking of what is all the information that a regulator or a more sophisticated stakeholder such as workers' rights advocates, if there's litigation, the attorneys involved, what information might they need to look at in order to determine whether the selection process as a whole, as well as the individual selection tools were chosen in a manner that were consistent with federal law and not just certain federal laws, but all federal anti-discrimination laws.

So it would include things like what sorts of notices did you give candidates? What were the exact content of them? Did you receive information or requests from candidates for accommodation? Did you provide accommodation? And if you did not, what -- how did you communicate that decision to the applicant?

For targeted job advertisements, it would -- you'd have to keep records regarding who was shown the ads, or not necessarily the specific individuals, but you would have to keep records indicating how those ads were -- how it was decided that those ads would be shown to prospective candidates.  All of this is basically meant to provide a basis for regulators and investigators to come along later and be able to determine whether or not compliance was achieved.  This isn't to say --

VICE CHAIR SAMUELS:  That's very helpful.

MR. SCHERER:  Yeah, this isn't to say that existing record keeping obligations, you know, are not a great start and that the EEOC lacks enforcement authority in order to collect a lot of this information already, but it is not clearly spelled out in existing regulations.

VICE CHAIR SAMUELS:  Thanks.  That is very helpful.  Let me follow up with a quick question on a different topic.  The Civil Rights Standards also call for the availability of human alternatives if individuals want to request a human review of what has happened through an online technological process.  How could those kinds of human review systems be created in ways that wouldn't undermine the efficiency that I think employers are seeking from use of AI tools?

MR. SCHERER:  Sadly, I don't think it can.  I think -- I think that you have to have some trade off of efficiency in order to ensure that each candidate receives the individualized assessment of their ability to perform the job, which I would argue that Title VII, the ADA and other federal anti-discrimination laws require. 

You could conceivably design a process where the human review is done in a manner that allows essentially kind of an expeditious review where you provide not the entire suite of application material so that it's not a complete start from scratch review of it, but rather there is some sort of recommendation that is output by the system and the factors that the system took into account when making that recommendation are provided to the human review.

So it's not necessarily the case that -- it was kind of fudged exactly what human review entails, but the idea would not necessarily be okay you would have some candidates that would essentially be reviewed by a human recruiter without the AI, and then others would be reviewed by an AI without a human recruiter.

The idea is you'd have some sort of check and backstop to make sure that the output that the machine was creating actually makes sense and properly accounts for the candidate's individualized characteristics.  And briefly, a major reason that's important is that for -- particularly for underrepresented groups, the ways in which their skills and competencies show up may not be obvious to an automated system. 

So there needs to be some sort of opportunity to explain, well, no, it says that I don't -- the machine says that I don't have any experience in X, Y, or Z, but I do.  Look at this.  And if it's a completely automated process and there's no opportunity for a candidate to make that explanation, then that is where you can run into potential discrimination issues because that is often keyed to what people are more represented in the training data.

VICE CHAIR SAMUELS:  Thank you so much.  That's -- that is really helpful.  Let me turn to Mr. Engler and say, you've written a lot about open-source software as a way for AI designers to be able to evaluate and assess the validity and impact of their AI tools.  Can you say a little more about what open source software would enable designers to do and whether that can automate -- this is kind of a meta question, automate the audits that a lot of our witnesses have called for today.

MR. ENGLER:  Sure, thanks for that question.  This is an under-explored angle.  A lot of the data science community, which is like the underlying community of people who actually build these things, are incredibly dependent on open source software to make these tools.  The vast majority of tools we're talking about are built in open source software and then made proprietary and sold, right?

So the availability of tools that can be used for more fair AI systems affects the average data scientist's ability to do their job.  There are a bunch of these systems out there already.  IBM has made systems called AI Fairness 360.  Microsoft has one called Fair Learn.  There's one from the University of Chicago and others, individual researchers and academics like Chris Mollner (phonetic) have made these.  And so across all of these, they do increase sort of the average capacity of a data science to make fair tools.

They don't necessarily solve all of your problems.  They don't automatically solve the incentives.  That still takes time and energy for an average data scientist to make a really fair system, right? That's more time you're spending on that and less time delivering on a new feature for a client. 

So it doesn't solve the incentives problem, but better open source software can make it easier, incrementally easier to make safer and fair systems.  You know, and I was encouraged that the National Science Foundation, for instance, had funded some of this research and it would be great to see more.  I hope that answers your question.

VICE CHAIR SAMUELS:  It does.  Thank you so much, and unfortunately I only have 20 -- 17 seconds left, but I do hope that Professor Ajunwa and Ms. Tinsley-Fix will have an opportunity to continue this conversation.  Thank you all for your really valuable input.

CHAIR BURROWS:  Thank you.  We'll go now to Commissioner Sonderling.

COMMISSIONER SONDERLING:  Thank you.  I'll stick with Mr. Engler.  Mr. -- okay.  There you are.  All right.  Well, thank you for testifying. 

You know, for you who has been looking at this issue for a long time and sort of poking through your writings and advocacy poking federal agencies to start taking a look at this, especially the EEOC, this must be a very welcome day for you and pinching yourself to be testifying about all the noise you've been making for a long time on this.  So I do want to thank you personally for pushing this issue forward, both here at the Commission and other Agency for many, many years.

But I want to talk about your recent time in Europe, the A -- the EU AI Act is obviously talked about globally, because it's really the first comprehensive AI piece of legislation.  Obviously there's significant issues there about who's going to enforce it and how that's going to work. 

But from your perspective, having really dug into it, I think it's worth discussing a little more about -- it's kind of a two part question.  Number one, what do you think we can learn from the implementation there, where they're specifically identifying employment as a higher risk? And then on the flip side, you know, what can they learn from us?

Realizing that a lot of these higher risk programs require robust disclosure and auditing and sort of the same issue where we're having with New York's proposal, when you get down to, well, how do you do an audit? And people are going to still look, whether it's in Europe or in New York City or wherever, to what the EEOC says on this issue.  So this is the kind of dual part question of what can the EU learn from us and what can we learn from them as we're all sort of tackling the same issue?

MR. ENGLER:  Sure.  Great question.  Thanks again for that.  And thanks for the kind words about my work.  I really appreciate that.  Though frankly, your job is much harder.  Okay, so the EU AI Act does some things that I think many of us who are concerned about algorithmic protections want to see. 

The good news is that it's comprehensive.  It's going to require a reaction to all of these different areas of employment, and that has just sort of good coverage, right? So you're going to -- we're going to get standards and law on all those things.  And I do think the process of actually writing down some technical standards across all those different applications of AI and employment will be very challenging for the -- for the EU, but also very informative for the rest of the world.  So they're going to take a crack at this, it's going to be challenging, but something meaningful will likely result.

The AI high-risk database I also mentioned will be literally every single piece of software that does this will have to be made publicly available in a database.  So just the sort of density in this market will also be available.  Now, there are downsides of the European approach.  For instance, it's very reliant on a single set of requirements that will have to be evaluated in all of these different applications.  AI hiring and AI wage setting and AI task management are actually pretty different, right? Even within AI hiring, you might have an analysis of a cover letter versus a predictive model towards a specific employee performance.

Again, very different.  So are they really going to be able to write consistent standards across the board for even within those sectors? It's hard to know. 

One thing that's nice about the US approach that the EEOC has already demonstrated an ability to do, is really get very, very deep into, with specific area of potential AI harms, like it did with the guidance for AI hiring under the Americans with Disabilities Act. 

You can consider both the socio-technical side, that is how humans and algorithms and the interaction between the software and the humans unfolds and then create standards and guidance around that.  Despite the fact that the EU is going to have more coverage, and I think there is a lot to learn there, I do think the US, in some ways will have better specific tailored as it gets through all these applications, even if that takes a long time.  I hope that trade off makes sense.

Just to answer your last question, algorithmic audits are another important part of this.  That you saying it's going to build the capacity to do this, that is implicit in the law that they can check to inform compliance with these standards.  There's also going to be a lot of lessons learned there, right? And I've written some about this for Brookings, and I will submit that longer report on what an algorithmic audited hiring is.  But hopefully that is enough to give you a sense.  Thank you for the question.

COMMISSIONER SONDERLING:  Thank you very much.  I want to go to Professor Ajunwa.  I -- guess I -- we must have crossed paths on Friday at UNC.  I'm sorry we didn't get a chance to catch up, but we will in front of everyone in the public.  So, you know, in your -- in your written testimony, in your remarks, I want to hone in on the EEOC should create its own audit tools. 

And this is something I'm really interested in because I've heard from a lot of vendors that I've spoken to that they can go to the -- you know, if you're making an organic food at Whole Foods or something, you can get a sticker from the USDA saying, it's organic.  We've all seen those on there.  And they've been asking, you know, why can't the EEOC or OCCP at the Department of Labor essentially stamp our AI products and say, We're EEOC certified?

I mean, certainly that would help with sales, but in a way, I think there is some analogy there, and I think you're alluding to this, if we create our own audit tools and we have some sort of certification programs, that means the vendors come and work with us.  That means they're agreeing to a standard. 

Although everyone has to agree to the standard of Title VII, there's no opting in or opting out of federal employment laws.  But at least going, you know, above and beyond and saying, you know, we're at least doing some testing, transparency, explainability, reporting, whatever that looks like.

You know, I certainly think there would be a lot of interest in that from what I'm hearing with vendors.  And I think that's a really good thing because it just enhances the compliance of this program.  It gives the EEOC and the federal government, I don't want to say partnership, but it gives them, you know, at the table with the people who are creating this potentially life-changing software.  But that being said, you bring it up and I'm really curious from your perspective of what that looks like and how we would do something like that.

DR. AJUNWA:  Yes.  So first, thank you so much Commissioner Sonderling for that question.  As you mentioned, employment is such an indelible part of our lives.  It's such a life-changing position to either have a job or not have a job, right? So the fact that we have this automated hiring systems or a gatekeeping employment opportunity is a matter of concern.  It's a matter of legal concern. 

And I do believe that auditing those systems is imperative.  That's the name of my paper, The Auditing Imperative for Automated Hiring Systems.  And, you know, as you mentioned, I believe that the EEOC should certainly be in the business of certifying any third-party vendors that would offer audits, because that would be a way of ensuring that those audits are actually meaningful audits.

You know, we've heard in the past of companies conducting internal audits or even external audits, and then the results are questionable or don't necessarily event at like a very rigorous audit.  So I believe that if the EEOC actually gets involved, it can make sure that any audits carried out with its certification actually will meet important standards, you know, already in place in terms of case law, in terms of Title VII for ensuring equal employment opportunity.

COMMISSIONER SONDERLING:  And certainly from the market, if you don't have that standard, if you don't have that sticker or working with the EEOC, you know, employers, although they're free to probably wouldn't buy it.

DR. AJUNWA:  Right.  Right.

COMMISSIONER SONDERLING:  And you know, investors probably wouldn't invest in -- you know, with all the amount of money going to artificial intelligence wouldn't invest in a program that doesn't meet those standards of certification.  So it is a wonderful idea and I thank you for raising it and I'd definitely like to talk more about that. 

But in my limited time, I'm going to go to Mr. Scherer.  I want to talk to you briefly about a point, and you probably can anticipate where I'm going to go about your -- the model legislation you put out, which we've discussed at length, but I do want to raise awareness of it. 

I think it kind of piggybacks into a testimony earlier about a safe harbor provision or something for employers to be able to test and try.  In your proposed legislation, there's a small business exception.  Can you explain that just a little bit for smaller employers and how that can be applicable to some of the concerns? You know, if your standards did become, let's say, law or regulation from some of the management side attorneys you heard for -- after about trying to implement it and having some leeway there.

MR. SCHERER:  Of course, first brief correction.  Standards, not legislation.

COMMISSIONER SONDERLING:  Standards.

MR. SCHERER:  It used -- it could be used as the basis for legislation.  But we certainly hope that the --

COMMISSIONER SONDERLING:  Maybe just giving you some free advertising.

MR. SCHERER:  Yes, yes.  But we certainly hope that the Commission will consider it in the absence, consider some of the principles in the absence of legislation.  But to answer your question about small employers, there are a few provisions that where -- to back up briefly, really what we tried to do with the Civil Rights Standards was to make it so that between vendors and employers, whichever party was best positioned to bear the cost of an audit or of compliance would do so. 

So in the case of small vendors and large employers, they may well keep -- make the decision that the employer bears the full cost of it and vice versa.  If it's a large vendor that sells, its wears to many, many small employers.

But there are a few things such as an audit of the existing employment practices of an employer that really it doesn't make sense to apply them to small employers for a few reasons.  One is the cost burden that it would impose, but another for the audit of the existing practices is that if you're going to do a statistical audit of some kind, you may need a larger number of employees or candidates than a small employer will have in their database, as it were, in order to conduct an effective statistical audit.

So for smaller employers, certain of the requirements were either -- in that case it was completely relieved.  In another instance where it was -- usually there's joint and several liability that's recommended when a selection tool results in discrimination, and in the case of small employers, the -- any discrimination that results from a selection tool that is sold by a vendor to a small employer, so the vendor can the vendor is solely responsible essentially for the cost of that discrimination.  The small employer doesn't bear the expense in that instance.

And the hope with these sorts of, is that every party involved has a strong stake in ensuring that everybody's doing what they're supposed to.  So, the vendor has a strong incentive to make sure that the small employer uses the tool exactly in the manner in which it is intended to be used, that they don't make modifications to it, that may change the characteristics of it in a way that may cause discrimination.  And that way, again, the party that is kind of in the best position to bring resources to bear to prevent discrimination from occurring can do so.

PARTICIPANT:  Thank you all very much.

CHAIR BURROWS:  Thank you.  We'll go now to Commissioner Lucas.

COMMISSIONER LUCAS:  Thank you.  And thank you to all the witnesses this entire day, it's been very interesting and we appreciate your -- you all coming here virtually to speak with us about this important topic.  I'd like to turn back to Professor Ajunwa.  I have a few questions about some of your additional recommendations.  Commissioner Sonderling talked to you about your recommendation about the EEOC developing its own audit program. 

I think it's your third recommendation, apologies if it's not that -- the right order, but it's the one that in which you recommend that the EEOC release an advisory notice that you ask governs the use of variables and algorithmic hiring.  Can you tell me a little bit more about that recommendation?

DR. AJUNWA:  Yes.  So, thank you so much for that question.  As it stands, you know, a lot of what is happening with automated hiring systems is very black box.  Meaning that employers can really just use whatever training data they feel is relevant and have the algorithms, really machine learning algorithms, sort out what variables can then be used for the target variable, which is finding the best worker. 

Unfortunately, in doing that, as I mentioned, this can result actually in the creation of proxy variables that are stand-ins for protected categories such as race, gender, et cetera, which then actually results in employment discrimination.

So for my proposal, which is based on looking at the uniform guidelines on employee selection, those guidelines advise that job selection criteria should focus on criterion validity, content validity, and construct validity.  So, what that means is that when choosing variables for selecting individuals for jobs, we actually need to pay attention to ensure that those variables actually are related to the job at hand. 

What does that mean? It means that evidence of the validity of a test or other selection procedure by criterion related validity study should consist of empirical data demonstrating that the selection procedure is predictive, right? Or significantly correlated, not just slightly correlated, significantly correlated with important elements of job performance. 

Also, evidence of the validity of a test or other selection procedure by content validity study should consist of data showing that the content of the selection procedure is representative of the important aspects of performance on the job.  And then finally, evidence of the validity of a test of construct validity should consist of data showing that the procedure actually measures the degree to which candidates have identifiable characteristics which have been determined to be important in successful performance in the job.

So what is all this saying, you know, for the lay people out there? It's saying that the variables actually have to be highly predictive of job performance or highly correlated to successful job performance.  They can't just be, we found that most of our, you know, top performance have this trait and therefore we are going with those variables.

Because if you do that, if you just use traits that are highly correlated to your top performance, what happens is that you're basically replicating the workforce you already have.  And if you think about workforces where women have been typically excluded, people of color have been typically excluded, using such training data is basically replicating those racial and gender imbalances. 

And if you allow me, Commissioner, I can share a story, you know, that illustrates this.  So, you know, the audit study that showed the variable Jared and high school lacrosse, right? I mentioned this in my testimony, those were highly correlated in terms of the top performers of that position but are they highly correlated to the actual job? That's what the uniform guidelines on employee selection actually requires.

COMMISSIONER LUCAS:  If I -- if I can just interject.  So, I guess my question to some degree is whether or not we just need to do a better job of informing people about what is already in UGESP.  I -- just drawing your attention to our 1979 Q&A, Section One, Question Five and Six.  And I think this is a large scale knowledge gap for things more than just AI, but also things like using college degree requirements, regardless of whether or not you have a complex algorithm screening out or simply it's just in your job ad, you know, by its face these guidelines state that, you know, the question is do the guidelines only apply to written tests?

And it points out, no, they apply to all selection procedures used to make employment decisions, including interviews, review of experience, education, work samples, physical requirements, evaluations of performance.

And then what practices are covered by the guidelines goes into even more detail, that's Question Six.  So I mean, really pretty much any employee selection procedure, which I think by its face would cover algorithms, is already covered by UGESP.  So, you're thinking perhaps just more education and outreach to advise --

DR. AJUNWA:  Exactly.

COMMISSIONER LUCAS:  -- stakeholders of the -- of that coverage?

DR. AJUNWA:  Right.  Right.  I certainly think there is a need for education and outreach, and I think the trend towards AI can make some employers forget, right, that this AI is still a selection procedure that still needs to follow EEOC guidelines, and I think a reminder is very necessary.

COMMISSIONER LUCAS:  And this is a question for both you and Ms. Tinsley-Fix, both of you talked about dropdown menus with respect to age discrimination.  I've heard this discussed in a variety of different contexts but, you know, I'm very troubled by the processes that's still happening.  So, I guess are -- you know, it looks like a lot of that arose -- the scrutiny there came out of the 2017 investigation, Illinois.  Are either of you aware of this being a prevalent practice or is this more of sort of an ongoing recommendation for people to not fall into this obvious pitfall? Because if it's still happening, I'm very interested in hearing about it.

MS. TINSLEY-FIX:  Well, I'll -- I mean, I'll jump in real quick and say that we don't have a good sense of the prevalence with which this kind of selection data point is collected.  We do know, however, that it's pretty prevalent to ask some kind of age related question. 

Whether or not it's the dropdown that's limited to a certain number of years or it's just asking for the date of your, you know, graduation from high school or university.  So, that is a lot more prominent than the dropdown, and I don't know how much to what degree, you know, vendors are still using that dropdown that's limited.

Or another thing that often happens and we sometimes will counsel order job applicants to do is just put in 9999 if you're given a -- you know, a date field that you have to fill out so that you're sort of masking your age.  That -- you know, that's a practice that we will sometimes tell older jobseekers but ultimately that's a hack, that's a workaround, it's not solving for the problem, which is, you know, do you need to know that this candidate is older than 18? Fine, ask them if they're older than 18, you don't need to know if they're 35, 55, 75, right? So in terms of its prevalence, I'm sorry I can't answer that, but my -- you know, Ms. Ajunwa might know more.

DR. AJUNWA:  Right.  You know, I -- as Ms. Tinsley-Fix mentioned, it is difficult to actually note the prevalence of many of these issues because there are no records, right, being kept on them in any sort of centralized way.  I would actually also note that the dropdown menu problem is illustrative of a bigger problem of design, right, on user interface.  So, a lot of these automated hiring programs aren't necessarily designed with EEO -- EEOC principles in mind, they're not necessarily designed with anti-discrimination principles in mind.

So, I do think education of people who are doing the software development or the creation of this automated hiring programs is also essential just to think about, you know, how the user interface could encourage advertently or inadvertently things like age discrimination and the ways that certain interfaces actually can enable proxy variables.

Because a lot of times they don't ask the age directly, they ask something like college graduation or date of your first job, things like that.  So I think, you know, really paying attention to design mandates I think is also a really important area to focus on.

MS. TINSLEY-FIX:  Yeah, I would definitely -- I think that's a really good point, which is sometimes it's just a matter of -- of educating the designers, right? Like these, you know, the folks in age inclusive design principles or ADA compliant design principles.

COMMISSIONER LUCAS:  Thank you.  If only we had some users or developers here today, unfortunate in my mind.  I think I'm out of time.  Thank you.

CHAIR BURROWS:  Thank you.  So, I wanted to start off by first of all talking a little bit about this question of third party auditing.  It's come up quite a bit today and I wanted to follow up on some of it.  So, I think I would start first of all with Professor Ajunwa and Mr. Scherer, just to drill down a little bit on that, because third party auditing obviously carries some risks depending on the incentive structure, let's say, between the relationship with the auditor and what's most likely their clients.

And so, just thinking through that, there may be an incentive for the auditor to be more lenient, you've sort of raised some of those.  But what role in terms of that, there's also this question of not wanting to stifle innovation, right, so the question of the EEOCs role potentially in auditing. 

Can you talk to both of those a little bit in terms of the risks with third party auditors and how we might do that more effectively, and also the risks if the government does it, that that might actually end up sort of unduly channeling innovation in a particular way?

DR. AJUNWA:  Right.  So, you know, if you don't mind Matt, I'll jump in.  So I think, you know, with third party audits, you know, there are pros and cons as you know, right? There -- there's the idea that, you know, third party auditors could be co-opted right by the corporations that they serve and, you know, return audits that are not necessarily meaningful or transparent. 

And I think, you know, that is an important risk to consider, but I also think that the market will quickly sort of disallow such gaming, right? Because if say a third party auditor is not rigorous or transparent and then later on obviously there arises an issue where an applicant then observes a problem with the automated hiring system, that will be a signal for other audit corporations not to use that auditor.

So, I think actually in terms of the market survival of the third party auditors that will, I think incentivize them -- incentivize them actually to have rigorous audits, meaningful audits.  That being said, I think also having a safe harbor as other, you know, panelists have mentioned that, as I mentioned in my paper, for corporations who do take it upon themselves to invite third party audits is essential, right?

So, that's sort of, you know, the -- for the kind of like rigorous audits that can actually expose problems, you do want safe harbor, you do want to grant some sort of grace period for corporations to then fix the issues that arise.  As for stifling innovation, I don't see this as actually a problem, I actually see this as encouraging innovation.

I think when you allow third party audits, you can actually more easily find issues with the design processes of automated hiring systems.  You can find features that are perhaps not as efficient as was, you know, conceived and therefore this can actually spur innovation, it can actually in fact allow third party auditors to be spearheading innovative, you know, developments of automated hiring systems that would actually better conform to EEOC guidelines.  So, I don't think those worries are -- I mean they're founded, but I don't think they're necessarily an impediment for the EEOC to take action in allowing third party audits.

CHAIR BURROWS:  Thank you.  So, Mr. Scherer?

MR. SCHERER:  So, I definitely agree with Professor Ajunwa's take on both the need for third party auditing, if for no other reason than it is not realistic to expect a government agency to take on the full burden of auditing all the companies that need auditing.  And if you have some sort of system for certifying independent auditors, then you can essentially ensure that somebody other than the internal people who have a stake in seeing the selection tool succeed and be used, take a look at it and check it for compliance with federal law. 

Now, there are potential issues with third party auditing, the most famous example is probably Arthur Anderson and Enron.  You know, you can have a third party auditor that is more interested in maintaining its relationship with the business and being brought back as the auditor year after year than they are with the external impressions that they leave about the quality of their audits.  And that can lead to ignoring issues that are even glaringly obvious.  That has unfortunately happened repeatedly with auditors in the securities world and in other sectors before.

It's not an easy thing to guard against, I think that there would have to be strict rules requiring the independence of the auditors, both from a financial perspective and from banning them from trying to cross-sell other services and things like that, that might create a greater incentive for them to try and develop a relationship with the business that is based on something other than ensuring that their tools are going to be audited correctly.

And by -- probably by establishing some sort of system where the enforcement agency, in this case, the EEOC that certifies them, would provide information to the public so that they can see, okay, when these auditors audit a tool, how often is it that there is a finding of discrimination regarding a tool that was made by that company? And then information can kind of reduce the friction and ensure that bad auditors do not repeatedly get hired. 

There's no perfect solution to it, I think that as with the efficiency trade-off that I mentioned earlier, there is an innovation trade-off when you have auditors, but that cost benefit analysis is something that I think needs to be done.  And in the case of anti-discrimination laws, the cost to society of having tools out there that are discriminating against workers is worth some sort of trade off in terms of slowing down innovation to the degree necessary to make sure that the innovation is actually beneficial.

MR. WONG:  Pardon me, Chair Burrows, you're on mute.

CHAIR BURROWS:  Thank you.  It's interesting that you mentioned some other contexts, and so I take from that potentially that you think maybe looking at how audits have worked in other areas such as the financial industry might be of some help to us if we were to sort of think this through more deeply? Okay.

And I wanted to go to Mr. Engler, I have really appreciated having you here and -- and your work as along with everyone's else's and have watched very closely some of the discussion in the European Union around AI and those developments. 

And also the -- some of the examples as we've had our own, but one that comes to mind of some of the risks, and I think it's been informative in the policy discussions there have been around what happened with the Dutch Cabinet a few years back and having -- it was a, apparently in the Child Welfare System, an algorithm that was supposed to ferret out fraud, but actually just really used things that were indicia of, you know, particular socioeconomic status and other demographic elements such that, you know, something to the tune of 10,000 families were separated from their -- you know, told to pay back public benefits that they did not actually owe.

And so, that I think has been illustrative to them and to us of what high, high stakes can be at issue.  Obviously not an employment situation, but clearly a reminder of why it's so important to get this right. 

One of the things I thought was interesting in your testimony was the reference to public filing, if you will, or making public the different AI systems that are used there, and that that could be something that's of use to the Commission and other federal regulators here in the United States. 

Could you expand on that a little bit in terms of how you could see that being of help to us and -- you know, tell us a little bit more about how that's intended to work?

MR. ENGLER:  Sure.  I'll try to be brief.  So in two ways, I think what we're going to see from the European Union is really valuable in the U.S.  First, purely how many of these AI employment systems there are, how many companies are using them, I mean just pure coverage, right? I think that we -- that is going to be the single best source of knowledge for the total impact of these systems broadly that exists. 

Even if it's pretty light on the specific details of each, it'll give a much better sense of coverage.  There will also likely be a non-trivial amount of overlap between the companies that operate there and the companies that operate here.

Just off the top of my head, there's lots of international business software like Microsoft Viva that could qualify in certain ways, and that'll both be used in the EU and multinational businesses as it is in the U.S. 

Secondly, the process of writing standards for these requires a lot of technical knowledge, meaning in order to create standards around their function, which I'm not endorsing these standards, they could be significantly industry written or not written by a broad set of stakeholders, it's not clear.  But in order to write them, you have to really get into the weeds of these applications.

And so, simply the pure amount of exploration they're going to do, I think will be really valuable.  And those standards should be largely public or you may actually have to pay a small fee, but pretty valuable in sort of looking deeply at the system.

So, I'm optimistic that the public knowledge from that will be substantial.  With your -- with apologies.  I also just want to mention that I've actually written a paper on AI auditing in hiring, including considering the incentives of third party auditors, including considering analogies in other areas like financial services and financial auditing, and I will happily submit that for the record.  But it gets at some of the -- of your last question as well.  Thank you.

CHAIR BURROWS:  Thank you.  And I am delighted to see that I still have an additional minute, so that's very exciting, I thought I was almost out of time, so I appreciate that.  Wanted to just ask each of you, if you have, with respect to some of the conversations we've been having, additional thoughts on the kinds of expertise that the EEOC should be hiring. 

We've talked very high level about that, but in terms of our -- we're looking at sort of our data strategy in the future and obviously in this particular area specifically.  But talk to me if you have suggestions, we've been working on our training, et cetera, in more detail. 

So, I would offer that to any of you.  I would start probably with Mr. Scherer, but anyone who wants to jump in for the sake of time, go ahead.

MR. SCHERER:  Just briefly, I think that you need a combination of technical expertise from people who are familiar with the machine learning and statistical methods that are now becoming predominant in a lot of the development of these tools. 

You certainly still need industrial and organizational psychologists, preferably those that have specialized in some way in the use of selection procedures, modern selection procedures that rely on correlation and statistical methods.  And beyond that, I think that just looking for people who have -- who have a proper understanding of that, it's not just automated tools, but all -- a lot of different types of selection procedures that are being used today are -- have the potential for discrimination.  And automated tools and the rise of them should be a basis for taking a closer look at the ways in which companies are auditing -- or excuse me, are deploying these tools in general.

CHAIR BURROWS:  Okay.  Thank you.  And I just realized I am actually out of time, so I -- but what I will do is plan to circle back with each of you.  I am so grateful and also congratulations on the Civil Rights standards, that was, you know, a huge lift.  So, thanks to each and every one of you.  I know that it took an enormous amount of time to prepare for today and we are enormously grateful.

So, that concludes today's presentation of testimony and questions.  And we truly are facing a new civil rights frontier as this conversation so aptly demonstrates.  So, as we at the Commission continue to advance equal employment opportunity, it's really clear that we've got to be focusing, in particular at employer's use of automated tools in our ongoing work, and doing it in ways that really will support those who work in this area. 

And first and foremost, ensure that these tools are used consistently with our Civil Rights Laws because everyone needs to really understand how much is at stake here, how clearly these decisions and this new approach to employment is going to, and is already affecting Americans' everyday lives. 

So, our AI and Algorithmic Fairness Initiative here at the EEOC will continue moving forward in gathering additional information, educating stakeholders, educating ourselves, and of course combating algorithm discrimination where we find it.

So in closing, I'd like to recognize the many EEOC staff who worked tirelessly to prepare for today's hearing.  We owe them a special debt of gratitude to our -- and particularly to our Office of Information Technology team whose extraordinary work and dedication has really made this hearing possible.

I'd like also to thank the Office of Executive Secretariat, the Office of Communications and Legislative Affairs, our EEOC Office of Legal Counsel, my dedicated colleagues in the Chair's Office who worked on this hearing, thank you very much, so grateful to you.  Our AA -- AI working group, and of course the many others at this Agency.  And especially to acknowledge the contributions and the excellent work and expertise of Vice Chair Samuels, each of the Commissioners and their respective staffs for their support and contributions and ideas.

So finally, thanks to the public and to our EEOC staff across the country that joined us virtually for this hearing.  We're going to be holding the Commission Hearing Record open for 15 days and invite any of you who would like to from the virtual audience as well as members of the public and our panelists to submit any additional written comments on the subject of today's hearing.  You can learn more about how to submit comments on our website, www.eeoc.gov.  And with that we are adjourned.  Thank you.

(Whereupon, the above-entitled matter went off the record at 2:37 p.m.)