Notable research on the use of machine learning in government, policy-making, and regulation. Please email regulation@law.upenn.edu if you would like to submit additional sources.
General Interest
U.S. Executive Office of the President, Preparing for the Future of Artificial Intelligence (2016).
U.S. Executive Office of the President, The Administration’s Report on the Future of Artificial Intelligence (2016).
U.S. Executive Office of the President, Big Data: Seizing Opportunities, Preserving Values (2014).
U.S. Government Accountability Office, Data Mining: Federal Efforts Cover a Wide Range of Uses, (May 2004).
U.S. Joint ad hoc Committee of the Committee on Adjudication and Committee on Administration and Management, Electronic Case Management in Federal Administrative Adjudication, (June 2018).
U.K. House of Commons, Science and Technology Committee, Robotics and Artificial Intelligence (2016).
A.I. Now Institute, A.I. Now 2017 Report (2017).
L. Anastasopoulos & Andrew Whitford, Machine Learning for Public Administration Research with Application to Organizational Reputation, (2018).
Oren Bar-Gill, Algorithimic Price Discrimination When Demand is a Function of Both Preferences and (Mis)Perceptions, U. Chicago L. Rev. (forthcoming 2018).
Solon Barocas, Ethics and Policy in Data Science, course syllabus (2017).
Solon Barocas et al. Governing Algorithms: A Provocation Piece, (presented at the Governing Algorithms Conference, 2013).
Stuart Minor Benjamin, Algorithms and Speech, 161 U. Pa. L. Rev. 1445 (2013).
Sarah Bird, Solon Barocas, Kate Crawford, et al., Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI (Oct. 2016).
Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014).
danah boyd & Kate Crawford, Six Provocations for Big Data (presented at A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society, 2011).
John Brockman (editor), What to Think About Machines That Think (2015).
Erik Brynjolfsson & Andrew McAffee, The Second Machine Age (2014).
Mark Buchanan, Our Robot Overlords Might be Delayed, Bloomberg (2018).
Ryan Calo, Robotics and the Lessons of Cyberlaw, 103 Calif. L. Rev. 513 (2015).
Centre for Public Impact, How to Make AI Work in Government and for People, (2018).
Centre for Public Impact, Destination Unknown: Exploring the Impact of Artificial Intelligence on Government,(2017).
Cary Coglianese, Optimizing Government for an Optimizing Economy, University of Pennsylvania Institute for Law and Economics (June 2016).
Cary Coglianese & David Lehr, Regulating by Robot: Administrative Decision-Making in the Machine-Learning Era, 105 Geo. L.J. 1147 (2017).
Mariano-Florentino Cuéllar, The Surprising Use of Automation by Regulatory Agencies, The Regulatory Review (2016).
Deloitte Center for Government Insights, Using Advanced Analytics to Drive Regulatory Reform, (2017).
Kevin C. Desouza, Delivering Artificial Intelligence in Government: Challenges and Opportunities (2018).
Megan Rose Dickey, Algorithmic Accountability, Techcrunch.com (April 30, 2017).
Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World (2015).
The Return of the Machinery Question, The Economist (June 25, 2016).
Lilian Edwards & Michael Veale, Enslaving the Algorithm: From a ‘Right to an Explanation’ to a ‘Right to Better Decisions’? (Oct. 31, 2017).
Lilian Edwards & Michael Veale, Slave to the Algorithm? Why a ‘Right to an Explanation’ is Probably Not the Remedy You are Looking For, Duke L. & Tech. J. (forthcoming 2017).
Ed Felten, Accountable Algorithms, Freedom to Tinker (Sep. 12, 2012).
Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future (2015).
Dirk Helbing et al., Will Democracy Survive Big Data and Artificial Intelligence?, Scientific American (Feb. 25, 2017).
Dan Hurley, Can an Algorithm Tell When Kids Are in Danger?, N.Y. Times (Jan. 2, 2018).
Henry Kissinger, How the Enlightenment Ends, The Atlantic (June 2018).
IBM Center for the Business of Government, The Future Has Begun: Using Artificial Intelligence to Transform Government, (2017).
Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633 (2017).
John Kamensky, Artificial Intelligence and the Future of Government, Government Executive (2018).
Jon Kleinberg et al., Human Decisions and Machine Predictions (National Bureau of Economic Research Working Paper No. 23180, 2017).
Will Knight, The Dark Secret at the Heart of AI, MIT Technology Review (April 11, 2017).
Rory Van Loo, Rise of the Digital Regulator, 66 Duke L. J. 1267 (2017).
John Markoff, Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots (2015).
Viktor Mayer-Schönberger & Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work, and Think (2013).
Cathy O’Neil, How Can We Stop Algorithms Telling Lies?, The Guardian (July 16, 2017).
Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016).
Joshua New & Daniel Castro, How Policymakers Can Foster Algorithmic Accountability, Information Technology & Innovation Foundation (2018).
New York University Information Law Institute, Algorithms and Explanations (April 27-28, 2017) (panel slides).
Nextgov, Disrupting Government, 2018.
Partnership for Public Service, The Future Has Begun: Using Artificial Intelligence to Transform Government (2018).
Vijay Pande, Artificial Intelligence’s ‘Black Box’ is Nothing to Fear, N.Y. Times (Jan. 25, 2018).
W. N. Price II, Regulating Black-Box Medicine, 116 Mich. L. Rev. 421 (2017).
Andrew Selbst & Solon Barocas, Regulating Inscrutable Systems, (2018).
Cassidy R. Sugimoto, Hamid R. Ekbia & Michael Mattioli (editors), Big Data is Not a Monolith (2016).
Michael Veale, Logics and Practices of Transparency and Opacity in Real-World Applications of Public Sector Machine Learning, Remarks at the 2017 Workshop on Fairness, Accountability, and Transparency in Machine Learning (Aug. 14, 2017).
University of Michigan Center on Finance, Law, and Policy, Big Data in Finance (Oct. 27-28, 2016) (panel videos and notes).
Upturn and Omidyar Network, Public Scrutiny of Automated Decisions: Early Lessons and Emerging Methods,(2018).
Li Zhou, Is Your Software Racist?, Politico (Feb. 8, 2018).
Challenges
Solon Barocas & Andrew Selbst, Big Data’s Disparate Impact, 104 Cal. L. Rev 671 (2016).
Nanette Byrnes, Why We Should Expect Algorithms to Be Biased, MIT Tech. Rev. (June 24, 2016).
Cynthia Dwork, et al., Fairness Through Awareness (Nov. 29, 2011).
Sorelle Friedler et al., Certifying and Removing Disparate Impact (presented at Fairness, Accountability, and Transparency in Machine Learning, 2014).
Aziz Huq, Racial Equity in Algorithmic Criminal Justice, Duke L. Journal (2019).
Toshihiro Kamishima et al., Fairness-Aware Classifier with Prejudice Remover Regularizer, in Machine Learning and Knowledge Discovery in Databases (Peter Flach, Tijl De Bie & Nello Cristianini eds., 2012).
Will Knight, Biased Algorithms are Everywhere, and No One Seems to Care, MIT Technology Review (July 12, 2017).
Hannah Laqueur & Ryan Copus, Synthetic Crowdsourcing: A Machine-Learning Approach to Problems of Inconsistency and Bias in Adjudication (Oct. 21, 2016).
Adam Liptak, Sent to Prison by a Software Program’s Secret Algorithms, N.Y. Times (May 1, 2017).
Claire Cain Miller, Algorithms and Bias: Q. and A. With Cynthia Dwork, N.Y. Times (Aug. 10, 2015).
Katherine Noyes, Will Big Data Help End Discrimination—or Make It Worse?, Fortune (Jan. 15, 2015).
Michael Schrage, Big Data’s Dangerous New Era of Discrimination, Harv. Bus. Rev. (Jan. 29, 2014).
Sonja Starr, Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, 66 Stan. L. Rev. 804 (2014).
Latanya Sweeney, Discrimination in Online Ad Delivery, 56 Comms. of the Ass’n for Computer Machinery 44 (May 2013).
Laura Sydell, Can Computers Be Racist? The Human-Like Bias of Algorithms, NPR (March 14, 2016).
Kim Zetter, Researchers Sue the Government Over Computer Hacking Law, Wired (July 29, 2016).
Privacy
Steven M. Bellovin et al., When Enough is Enough: Location Tracking, Mosaic Theory, and Machine Learning, 8 N.Y.U. J. L. & Liberty 555 (2014).
Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms, 55 B.C. L. Rev. 93 (2014).
Roger Allan Ford & W. Nicholson Price II, Privacy and Accountability in Black-Box Medicine, 22 Mich. Telecomm. & Tech. L. Rev. (forthcoming 2016).
Neil Richards & Jonathan King, Three Paradoxes of Big Data, 66 Stan. L. Rev. Online 41 (2013).
Omer Tene & Jules Polonestky, Privacy in the Age of Big Data: A Time for Big Decisions, 64 Stan. L. Rev. Online 63 (2012).
Applications
Criminal Justice
Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016).
Julia Angwin & Jeff Larson, Bias in Criminal Risk Scores is Mathematically Inevitable, Researchers Say, ProPublica (Dec. 30, 2016).
Richard Berk, Susan B. Sorenson & Geoffrey Barnes, Forecasting Domestic Violence: A Machine Learning Approach to Help Inform Arraignment Decisions, 13. J. Empirical L. Studies 94 (2016).
Richard Berk, Lawrence Sherman, Geoffrey Barnes, Ellen Kurtz & Lindsay Ahlman, Forecasting Murder Within a Population of Probationers and Parolees: A High Stakes Application of Statistical Learning, 172 J. Royal Stat. Soc’y Series A Stat. in Soc’y 191 (2009).
Richard Berk & Jordan Hyatt, Machine Learning Forecasts of Risk to Inform Sentencing Decisions, 27 Federal Sentencing Reporter 222 (2015).
Richard Berk & Justin Bleich, Statistical Procedures for Forecasting Criminal Behavior, 12 Criminology & Public Policy 513 (2013).
Tim Brennan & William Oliver, The Emergence of Machine Learning Techniques in Criminology: Implications of Complexity in Our Data and in Research Questions, 12 Criminology & Pub. Pol’y 551 (2013).
Tim Brennan et al., Evaluating the Predictive Validity of the COMPAS Risk and Needs Assessment System, 36 Crim. Just. & Behav. 21 (2009).
Maurice Chammah, Policing the Future, The Verge (Feb. 3, 2016).
Alexandra Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments (Oct. 2016).
Andrew Guthrie Ferguson, Big Data and Predictive Reasonable Suspicion, 163 U. Pa. L. Rev. 327 (2015).
Maria Konnikova, The Future of Fraud-Busting, The Atlantic (March 2016).
Brian Kriegler & Richard Berk, Small Area Estimation of the Homeless in Los Angeles: An Application of Cost-Sensitive Stochastic Gradient Boosting, 4 Annals Applied Stat. 1234 (2010).
Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871 (2016).
Andrew D. Selbst, Disparate Impact in Big Data Policing, Yale Information Society Project (Oct. 3, 2016).
Tom Simonite, How to Upgrade Judges with Machine Learning, MIT Technology Review (March 6, 2017).
Matt Stroud, Chicago’s New Police Computer Predicts Crimes, But is it Racist?, The Verge (Feb. 19, 2014).
Rebecca Wexler, When a Computer Program Keeps You in Jail, N.Y. Times (June 13, 2017).
Environmental Protection
U.S. Environmental Protection Agency, Toxicology Testing in the 21st Century (Tox21).
Richard S. Judson et al., Estimating Toxicity-Related Biological Pathway Altering Doses for High-Throughput Chemical Risk Assessment, 24 Chemical Res. in Toxicology 451, 457-60 (2014).
Robert Kavlock et al., Update on EPA’s ToxCast Program: Providing High Throughput Decision Support Tools for Chemical Risk Management, 25 Chemical Res. in Toxicology 1287 (2012).
Cleridy Lennert-Cody & Richard Berk, Statistical Learning Procedures for Monitoring Regulatory Compliance: An Application to Fisheries Data, 170 J. Royal Stat. Soc. 671 (2007).
Huanxiang Liu, Xiaojun Yao, and Paola Gramatica, The Applications of Machine Learning Algorithms in the Modeling of Estrogen-Like Chemicals, 12 Combinatorial Chem. & High Throughput Screening 490 (2009).
Matthew Martin et al., Economic Benefits of Using Adaptive Predictive Models of Reproductive Toxicity in the Context of a Tiered Testing Program, 58 Systems Biology in Reproductive Med. 3 (2012).
Financial Regulation and Tax
Taxpayer Advocacy Service, IRS Policy Implementation Through Systems Programming Lacks Transparency And Precludes Adequate Review, in 2010 Annual Report to Congress 71 (2010).
U.S. Federal Deposit Insurance Corporation, Business Technology Strategic Plan 2013-2017 (2013).
The Government Office for Science, London, Foresight: The Future of Computer Trading in Financial Markets (2012).
Scott W. Bauguess, The Hope and Limitations of Machine Learning in Market Risk Assessment (Mar. 6, 2015).
Christopher Condon, Quest for Robo-Yellen Advances as Computers Gain on Rate Setters, Bloomberg (May 24. 2016).
David DeBarr et al., Relational Mining for Compliance Risk (presented at the Internal Revenue Service Research Conference, 2004).
Steve Donoho, Early Detection of Insider Trading in Option Markets (presented at the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004).
Mikella Hurley & Julius Adebayo, Credit Scoring in the Era of Big Data, 18 Yale J.L. & Tech. 148 (2016).
Amir Khandani et al., Consumer Credit-risk Models via Machine-learning Algorithms, 34 J. Banking & Fin. 2767 (2010).
Andrei A. Kirilenko & Andrew W. Lo, Moore’s Law vs. Murphy’s Law: Algorithmic Trading and Its Discontents, 27 J. Econ. Perspectives 51 (2013).
Jane Martin & Rick Stephenson, Risk-Based Collection Model Development and Testing (presented at the Internal Revenue Service Research Conference, 2005).
Shawn Mankad, George Michailidis & Andrei A. Kirilenko, Discovering the Ecosystem of an Electronic Financial Market with a Dynamic Machine-Learning Method, 2 Algorithmic Finance 151 (2013).
Scott D. O’Malia, Opening Statement at 12th Meeting of the Technology Advisory Committee (June 3, 2014).
Johan Perols, Financial Statement Fraud Detection: An Analysis of Statistical and Machine Learning Algorithms, 30 Auditing: J. Practice & Theory 19 (2011).
Gregory Scopino, Preparing Financial Regulation for the Second Machine Age: The Need for Oversight of Digital Intermediaries in the Futures Markets, 2015 Col. Bus. L. Rev. 439 (2015).
Laws and Courts
Sean Braswell, All Rise for Chief Justice Robot!, OZY (June 7, 2015).
Danielle Keats Citron, Technological Due Process, 85 Wash. U. L. Rev. 1249 (2008).
Danielle Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89 Wash. U. L. Rev. (2014).
Mariano-Florentino Cuéllar, Cyberdelegation and the Administrative State, in Administrative Law From the Inside Out: Essays on Themes in the Work of Jerry Mashaw (Nicholas R. Carillo ed., forthcoming).
Of Prediction and Policy, The Economist (Aug. 20, 2016).
If Computers Wrote Laws: Decisions Handed Down by Data, The Economist (May 16, 2016).
Stephen Goldsmith & William D. Eggers, Governing by Network: The New Shape of the Public Sector (2004).
Wallis M. Hampton, Predictive Coding: It’s Here to Stay, Practical L. (May 5, 2014).
Smart Cities
Jacob Abernethy, Alex Chojnacki, Arya Farahi, Eric Schwartz, & Jared Webb, ActiveRemediation: The Search for Lead Pipes in Flint, Michigan, Association for Computing Machinery (2018).
Ash Center Mayors Challenge Research Team, Chicago’s SmartData Platform: Pioneering Open Source Municipal Analytics, Data-Smart City Solutions (2014).
Robert Brauneis & Ellen Goodman, Algorithmic Transparency for the Smart City, 20 Yale J. of Law & Tech. 103 (2018).
Gabe Cherry, Google, U-M to Build Digital Tools for Flint Water Crisis, University of Michigan News (May 3, 2016).
Bechara Choucair, Jay Bhatt & Raed Mansour, How Cities Are Using Analytics to Improve Public Health, Harv. Bus. Rev. (Sept. 15, 2014).
Daniel Cusick, Artificial Intelligence Used to Interpret ‘Climate Signals,’ ClimateWire (May 8, 2018)
Will Davies, Robot Amelia – A Glimpse of the Future for Local Government, The Guardian (July 4, 2016).
Brian Heaton, New York City Fights Fire with Data, Gov’t Tech (May 15, 2015).
Edward Glaeser et al., Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy(National Bureau of Economic Research Working Paper No. 22124, 2016).
Stephen Goldsmith & Susan Crawford, The Responsive City: Engaging Communities Through Data-Smart Governance (2014).
Will Knight, Can Machine Learning Help Lift China’s Smog?, MIT Tech. Rev. (Mar. 28, 2016).
Ian Lovett, To Fight Gridlock, Los Angeles Synchronizes Every Red Light, N.Y. Times (Apr. 1, 2013).
Alexis Madrigal, How a Feel-Good AI Story Went Wrong in Flint, The Atlantic (January 3, 2019).
David Morris, How Swarming Traffic Lights Could Save Drivers Billions of Dollars, Fortune (July 13, 2015).
Mohana Ravindranath, In Chicago, Food Inspectors Are Guided by Big Data, Wash. Post (Sept. 28, 2014).
Nick Rojas, Chicago and Big Data, TechCrunch (Oct. 22, 2014).
Methods
Martín Abadi, et al., Deep Learning with Differential Privacy (July 2016).
Julius Adebayo & Lalana Kagal, Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models (Nov. 2016).
Emrah Akyol, Cedric Langbort & Tamer Basar, Price of Transparency in Strategic Machine Learning (Oct. 2016).
Aws Albarghouthi, Loris D’Antoni, Samuel Drews & Aditya Nori, Fairness as a Program Property (Oct. 2016).
Ethem Alpaydin, Introduction to Machine Learning (4th ed. 2020).
David Barber, Bayesian Reasoning and Machine Learning (2012).
Richard Berk, Statistical Learning from a Regression Perspective (2008).
Richard Berk, Criminal Justice Forecasts of Risk (2012).
Christopher Bishop, Pattern Recognition and Machine Learning (2006).
Leo Breiman, Statistical Modeling: The Two Cultures (With Comments and a Rejoinder by the Author), 16 Stat. Sci. 199 (2001).
L. Elisa Celis, Amit Deshpande, Tarun Kathuria & Nisheeth K. Vishnoi, How to be Fair and Diverse (Oct. 2016).
Bertrand Clarke, Ernest Fokoué & Hao Helen Zhang, Principles and Theory for Data Mining and Machine Learning (2009).
Miguel Ferreira, Muhammad Bilal Zafar & Kirshna P. Gummadi, The Case for Temporal Transparency: Detecting Policy Change Events in Black-Box Decision Making Systems (Oct. 2016).
Sorelle Friedler et al., On the (Im)possibility of Fairness (Sept. 2016).
Ian Goodfellow, et al., Deep Learning (2016).
Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks (2008).
Trevor Hastie, et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2009).
Moritz Hardt, Eric Price & Nathan Srebo, Equality of Opportunity in Supervised Learning (2016).
Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern & Aaron Roth, Fair Learning in Markovian Environments (Nov. 2016).
I. Jordan & T. M. Mitchell, Machine Learning: Trends, Perspectives, and Prospects, 349 Science 255 (2015).
Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel & Aaron Roth, Rawlsian Fairness for Machine Learning (Nov. 2016).
Matthew Joseph, Michael Kearns, Jamie Morgenstern & Aaron Roth, Fairness in Learning: Classic and Contextual Bandits (May 2016).
Jon Kleinberg, Sendhil Mullainathan & Manish Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores (Nov. 2016).
Pat Langley, The Changing Science of Machine Learning, 82 Machine Learning 275 (2011).
Kristian Lum & James Johndrow, A statistical framework for fair predictive algorithms (Oct. 2016).
David MacKay, Information Theory, Inference, and Learning Algorithms (2003).
William Rand, Machine Learning Meets Agent-Based Modeling: When not to Go to a Bar (Nw. Univ. working paper, 2006).
David Scharfenberg, Computers Can Solve Your Problems. You May Not Like the Answer., The Boston Globe (September 21, 2018).
Richard S. Sutton & Andrew G. Barto, Reinforcement Learning: An Introduction (2015).
Ke Yang & Jula Stoyanovich, Measuring Fairness in Ranked Outputs (Oct. 2016).
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, et al., Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Measurement (Oct. 2016).