Facial Recognition Technology
Facial recognition technology—which can verify or identify an individual from a facial image—has rapidly improved in performance and now can surpass human performance in some cases. The Department of Commerce convened stakeholders to review privacy issues related to commercial use of this technology, which GAO was also asked to examine. GAO analyzed laws, regulations, and documents; interviewed federal agencies; and interviewed officials and reviewed privacy policies and proposals of companies, trade groups, and privacy groups. Companies were selected because they were among the largest in industries identified as potential major users of the technology, and privacy groups were selected because they had written on this issue. GAO suggested in GAO-13-663 that Congress consider strengthening the consumer privacy framework to reflect changes in technology and the marketplace.
Facial recognition technology can be used in numerous consumer and business applications, but the extent of its current use in commercial settings is not fully known. The technology is commonly used in software that manages personal photographs and in social networking applications to identify friends. In addition, several companies use the technology to provide secure access to computers, phones, and gaming systems in lieu of a password. Although facial recognition technology can have applications for customer service and marketing, at present, use in the United States of the technology for such purposes appears to be largely for detecting characteristics (such as age or gender) to tailor digital advertising, rather than identifying unique individuals. Some security systems serving retailers, banks, and casinos incorporate facial recognition technology, but the extent of such use at present is not fully known.
Privacy advocacy organizations, government agencies, and others have cited several privacy concerns related to the commercial use of facial recognition technology. They say that if its use became widespread, it could give businesses or individuals the ability to identify almost anyone in public without their knowledge or consent and to track people’s locations, movements, and companions. They have also raised concerns that information collected or associated with facial recognition technology could be used, shared, or sold in ways that consumers do not understand, anticipate, or consent to.
Several government, industry, and privacy organizations have proposed or are developing voluntary privacy guidelines for commercial use of facial recognition technology. Suggested best practices vary, but most call for disclosing the technology’s use and obtaining consent before using it to identify someone from anonymous images. The privacy policies of companies GAO reviewed varied in whether and how they addressed facial recognition technology.
No federal privacy law expressly regulates commercial uses of facial recognition technology, and laws do not fully address key privacy issues stakeholders have raised, such as the circumstances under which the technology may be used to identify individuals or track their whereabouts and companions. Laws governing the collection, use, and storage of personal information may potentially apply to the commercial use of facial recognition in specific contexts, such as information collected by health care entities and financial institutions. In addition, the Federal Trade Commission Act has been interpreted to require companies to abide by their stated privacy policies. Stakeholder views vary on the efficacy of voluntary and self-regulatory approaches versus legislation and regulation to protect privacy.
Facial recognition technology is currently being used in a number of U.S. commercial applications for functions including safety and security, secure access, and marketing and customer service. Some retailers, casinos, financial institutions, and apartment buildings use facial recognition technology for safety and security purposes. According to the National Retail Federation, some retailers in the United States are testing systems that use facial recognition technology with closed-circuit television for theft prevention. According to one vendor of such a system with whom we spoke, security cameras in a retail location compare images of individuals who walk into a store against a database of images of known shoplifters, members of organized retail crime syndicates, or other persons of interest. If a match is found, security personnel or management are alerted and provided whatever information is known about the individual.
Some casinos in the United States similarly use facial recognition systems to help them identify known or suspected gambling cheaters, members of organized crime networks, or other known persons of concern. Facial recognition technology has also been incorporated into the security systems of some financial institutions to identify robbery suspects or accomplices. According to a vendor of this technology, these systems deter crime and help identify suspects much faster than traditional means, which require staff to spend hours reviewing video recordings. Facial recognition systems have also been used in large apartment buildings to help identify perpetrators of crimes or other known persons of concern who seek to enter the property, according to one software vendor with whom we met.
Industry trade organizations have said they envision retailers and others using facial recognition technology to target marketing and advertising more effectively and improve customer service. The Direct Marketing Association has stated that facial recognition technology has the potential to help businesses provide more customized and improved products and services, conduct market research and product development, provide more tailored and relevant messaging and advertising, and offer a more secure shopping experience.
Facial recognition technology is already used in digital signs—usually televisions or kiosks displaying advertisements in stores—with cameras that recognize characteristics of the viewer, such as gender or age range, and target advertisements accordingly. This allows retailers and advertisers to show relevant products and deals in real time, possibly leading to more sales, according to the Digital Signage Federation. In the future, such signs may be used to identify customers by name and target advertising to them based on past purchases or other personal information available about them, according to FTC staff. Facial recognition systems can also be designed to alert staff when known customers enter the store, according to a software vendor with whom we spoke. Representatives of the National Retail Federation said they could envision retailers using facial recognition systems to track customer movements around the store to provide the customer with a better shopping experience.
The FBI Perspective
In a statement before the House Committee on Oversight and Government Reform in March 2017 it was revealed that the FBI does use FR technology for law enforcement purposes including (1) the FBI’s Next Generation Identification (NGI) System located at the FBI’s Criminal Justice Information Services (CJIS) Division, and (2) the Facial Analysis, Comparison, and Evaluation (FACE) Services Unit also located at the FBI CJIS Division.
NGI maintains a mugshot repository that is known as the Interstate Photo System (IPS). All mugshots are associated with tenprint fingerprints and a criminal history record. The NGI-IPS allows automated FR searches by authorized local, state, tribal, and federal law enforcement agencies. The law enforcement agency submits a probe photo that is obtained pursuant to an authorized law enforcement investigation, to be searched against the mugshot repository. The NGI-IPS returns a gallery of candidate photos of 2-50 individuals (default is 20). The law enforcement agencies then must manually review the candidate photos and perform further investigation to determine if any of the candidate photos are the same person as the probe photo.
The NGI-IPS technology is only used as an investigative lead, and not as a means of positive identification. The NGI-IPS Policy Implementation Guide has been made available to authorized law enforcement users who receive candidate photos from the Next Generation Identification-Interstate Photo System. The policy advises that the photos are not being provided as positive identification and cannot serve as the sole basis for law enforcement action. In addition, the FBI has promulgated policies and procedures that place legal, policy, and security requirements on the law enforcement users of the NGI-IPS, including a prohibition against submitting probe photos that were obtained in violation of the First or Fourth Amendments. It is important to note that the FBI does not retain the probe photos; the probes are searched and deleted. Therefore, the NGI-IPS remains a repository solely of mugshots that are submitted voluntarily with fingerprints pursuant to arrest.
The FACE Services Unit provides investigative lead support to the FBI field offices, operational divisions, and legal attachés by comparing the face images of persons associated with open assessments2 and active investigations3 against face images available in state and federal FR systems. In limited instances, the FACE Services Unit provides FR support for closed FBI cases (e.g., missing and wanted persons) and may offer recognition support to federal partners. The FACE Services Unit only accepts probe photos that have been collected pursuant to applicable legal authorities as part of an authorized FBI investigation. Upon receipt of the photo(s), the FACE Services Unit searches them using FR software against databases authorized for use by the FBI, which results in a photo gallery of potential candidates. The FACE Services Unit performs manual comparisons of candidate photos against the probe photo(s) to determine a candidate’s value as an investigative lead. This service does NOT provide positive identification, but rather, an investigative lead and analysis to support that lead.
In performing the search(es), the FACE Services Unit operates under the authority of the United States Code (U.S.C) Sections 533 and 534; Title 28, Code of Federal Regulations Section 0.85; Title 42, U.S.C. Section 3771; and Title 18, U.S.C. Chapter 123. The FACE Services Unit performs FR searches of FBI databases (e.g., FBI’s NGI), other federal databases (e.g., Department of State’s Visa Photo File, Department of Defense’s Automated Biometric Identification System, Department of State’s Passport Photo File), and state photo repositories (e.g., select state Departments of Motor Vehicles). Memoranda of understanding and agreements have been established with all partners.
Privacy Impact Assessments (PIAs) for the FACE Services Unit and the NGI-IPS have been prepared by the FBI, approved by the DOJ, and posted at https://www.fbi.gov/services/records-management/foipa/privacy-impact-assessments. These PIAs provide to the public an accurate and complete explanation of how specific FBI components are using face recognition technology in support of the FBI’s mission to defend against terrorism and enforce criminal laws, while protecting civil liberties. The PIAs also reflect many of the privacy and civil liberties choices made during the implementation of these programs.
In addition to understanding how these FBI programs operate, it is also important to have an understanding of how automated FR works. The following is a brief description of automated FR: The automated FR software uses pattern matching approaches developed within the field of computer vision. Such approaches do not rely upon intrinsic models of what a face is, how it should appear, or what it may represent. In other words, the potential matching is not based on biological or anatomical models of what a face—or the features which make up a face—look like. Instead, the algorithm performance is entirely dependent upon the patterns which the algorithm developer finds to be most useful for matching. The patterns used in automated FR algorithms do not correlate to obvious anatomical features such as the eyes, nose or mouth in a one-to-one manner, although they are affected by these features. Put another way, the algorithms see faces in a way that differs from how humans see faces.
The FBI conducted a trade study of FR products, leveraging the NGI Integrator Lockheed Martin, which led to the determination of MorphoTrust as the best cost solution in fall 2010. The FBI has tested and verified that the NGI FR solution returns the correct candidate a minimum of 85 percent of the time within the top 50 candidates. In 2017 eleven states had connectivity with the NGI-IPS.
The TSA Perspective
Beginning in October 2018, TSA started testing facial recognition technology at the Hartsfield–Jackson Atlanta International Airport (ATL) in Terminal F as an alternative means to verify a passenger’s identity when they begin screening at a TSA security checkpoint. TSA has conducted its pilot in conjunction with U.S. Customs and Border Protection (CBP) to test biometric technology for purposes of identity verification. The technology used will be the Traveler Verification System maintained by CBP. TSA had recently completed testing this system in the Tom Bradley International Terminal at Los Angeles International Airport.
Facial Recognition Technology at Los Angeles International Airport is used when passengers present their boarding passes, a TSA officer asked if they would like to have their picture taken instead of providing physical identification documents. CBP’s system would attempt to compare that picture to photos in government databases, such as photos obtained from passports or visa applications, to verify the passenger’s identity. TSA did not store the photograph.
If CBP’s system confirmed a match to a photograph in a government database, the passenger’s name and date of birth was sent from the database to a tablet used by the TSA officer. The TSA officer then used the information displayed on the tablet to verify the identity of the passenger and direct the passenger to proceed with physical screening. If CBP’s system did not match the passenger’s picture, the database indicated that no picture match could be made. Where the system did not produce a match, the TSA officer used standard document checking procedures.
On August 22, 2018, a 26-year-old man traveling from Sao Paulo, Brazil presented a French passport to the CBP officer conducting primary inspections. The officer utilized CBP’s new facial comparison biometric technology which confirmed the man was not a match to the passport he presented. The CBP officer referred the traveler to secondary for a comprehensive examination. In secondary, CBP officers noted the traveler’s behavior changed and he became visibly nervous. A search revealed the man’s authentic Republic of Congo identification card concealed in his shoe.
Using another person’s identity document is a serious violation of U.S. immigration laws that could result in criminal prosecution. CBP is withholding the man’s name until the investigation is completed.
While providing a level of convenience for the traveler, facial recognition helps CBP swiftly identify impostors. CBP’s primary mission is to protect the United States from potential threats and the facial comparison biometrics system is one part of CBPs’ strategy to deploy the best technology available to protect the American people.
IAD is one of 14 early adopter airports to launch the use of facial recognition technology to expedite the entry inspection process of arriving international passengers and began the enhanced entry process on August 20, 2018. Metropolitan Washington Airports Authority partnered with CBP at Washington-Dulles International Airport to deploy biometric entry and exit technology using facial comparison to provide additional security and to improve efficiency for international travelers. The new, simplified arrival process enables increased security, faster throughput, and better efficiency.
CBP is assessing the use of biometric technology as part of a future end-to-end process, from check-in to departure, in which travelers use biometrics instead of their boarding pass or ID throughout the security and boarding processes. Testing was recently expanded to include one checkpoint at the Transportation Security Administration (TSA) in at John F. Kennedy International Airport.
Past and Future of Biometrics
Beginning in July 1999, the CJIS Division operated and maintained the IAFIS, the world’s largest person-centric database. The IAFIS provided automated tenprint and latent fingerprint searches, electronic image storage, electronic exchanges of fingerprints and responses, as well as text-based searches based on descriptive information. Because of growing threats, new identification capabilities were necessary. Advancements in technology allowed further development of biometric identification services. The CJIS Division, with guidance from the user community, developed the NGI System to meet the evolving business needs of its IAFIS customers.
Building on the foundation of the IAFIS, the NGI brought the FBI’s biometric identification services and criminal history information to the next level. The NGI system improved the efficiency and accuracy of biometric services to address evolving local, state, tribal, federal, national, and international criminal justice requirements. New capabilities include a national Rap Back service; the Interstate Photo System; fingerprint verification services; more complete and accurate identity records; and enhancements to the biometric identification repository. Below are descriptions of some of those capabilities.
The FBI deployed the first increment of the NGI System in February 2011, when the AFIT replaced the legacy Automated Fingerprint Identification System (AFIS) segment of the IAFIS. The AFIT enhanced fingerprint and latent processing services, increased the accuracy and daily fingerprint processing capacity, and improved system availability. The CJIS Division implemented a new fingerprint-matching algorithm that improved matching accuracy from 92 percent to more than 99.6 percent. In addition, contributors experienced faster response times, fewer transaction rejects, and increased frequency of identification and file maintenance notifications triggered by consolidations.
In August 2011, the RISC, a rapid search service accessible to law enforcement officers nationwide, became available through the use of a mobile fingerprint device. The NGI rapid search, with response times of less than 10 seconds, offers additional officer safety and situational awareness by providing on-scene access to a national repository of wants and warrants including the Immigration Violator File (IVF) of the National Crime Information Center (NCIC), convicted sex offenders, and known or appropriately suspected terrorists. The NGI RISC rapid search service is available only to authorized criminal justice personnel for criminal justice purposes in compliance with federal and state laws.
The NGI System’s latent functionality uses a Friction Ridge Investigative File composed of all retained events for an individual as opposed to one composite image set per identity. These multiple events in the repository result in three times the previous latent search accuracy and allow for additional event image retrieval to support difficult casework.
Prior to the NGI System, latent images searched against the criminal repository. Now, latent users can search latent images against the criminal, civil, and Unsolved Latent File (ULF) repositories. Moreover, incoming criminal and civil submissions (tenprint, palm print, RISC, and supplemental fingerprints) are cascaded against the ULF, generating new investigative leads in unsolved and/or cold cases. The CJIS Division recommends latent fingerprint images submitted prior to 2013 be resubmitted to the NGI system if no identification was made during the initial search.
In May 2013, the FBI established the National Palm Print System (NPPS). This system contains palm prints that are searchable to law enforcement nationwide. The NGI System also allows direct enrollment and deletion of palm prints and supplemental fingerprints similar to the existing direct fingerprint enrollment capability. These types of search and enrollment enhancements provide powerful new crime-solving capabilities to local, state, tribal, and federal law enforcement agencies across the country.
The Rap Back service allows authorized agencies to receive notification of activity on individuals who hold positions of trust (e.g. school teachers, daycare workers) or who are under criminal justice supervision or investigation, thus eliminating the need for repeated background checks on a person from the same applicant agency. Prior to the deployment of Rap Back, the national criminal history background check system provided a one-time snapshot view of an individual’s criminal history status. With Rap Back, authorized agencies can receive on-going status notifications of any criminal history reported to the FBI after the initial processing and retention of criminal or civil transactions. By using fingerprint identification to identify persons arrested and prosecuted for crimes, Rap Back provides a nationwide notice to both criminal justice and noncriminal justice authorities regarding subsequent actions.
The Interstate Photo System, or IPS, is the FBI's repository of all photos received with tenprint transactions, by qualifying submission or bulk submission, when verified with an existing tenprint record. The IPS permits broader acceptance and use of photos by allowing: more photo sets per FBI record for criminal subjects; bulk submission of photos maintained at state repositories; and submission of photos other than facial (i.e., scars, marks, tattoo, symbols).
A feature of the NGI IPS is the facial recognition search, another way biometrics can be used as an investigative tool. The IPS offers an automated search and response system targeted toward state and local law enforcement. Authorized law enforcement may submit a probe photo for a search against over 30 million criminal mug shot photos and receive a list of ranked candidates as potential investigative leads.
To further increase the tools available to the LE community, the CJIS Division has made a commitment to establish enhanced services to assist in meeting the challenges that face the criminal justice community in the identification of cold case/unknown deceased investigations. Using the advanced search algorithms within NGI, and the ability to cascade NGI searches against the criminal and civil files, as well as event based searches, this tool will strengthen criminal investigations and increase the use of enhanced state-of-the-art biometric technologies.
The FBI deployed an Iris Pilot (IP) in September 2013 to evaluate technology, address key challenges, and develop a system capable of performing iris image recognition services. The IP is building a criminal iris repository and will eventually deploy initial matching functionality. The IP continues to provide CJIS the opportunity to assess and/or develop privacy and policy documentation, best practices for iris image capture, iris camera specification requirements, specifications for iris image compression, and a review of new and existing iris image quality metrics. Furthermore, the IP provides the CJIS Division with the opportunity to create a national iris repository that will increase the usability of iris biometrics.
Biometrics are unique physical characteristics, such as fingerprints, that can be used for automated recognition. At the Department of Homeland Security, biometrics are used to detect and prevent illegal entry into the U.S., grant and administer proper immigration benefits, vetting and credentialing, facilitating legitimate travel and trade, enforcing federal laws, and enabling verification for visa applications to the U.S.
DHS provides biometric identification services to protect the nation through its Office of Biometric Identity Management (OBIM), which supplies the technology for matching, storing, and sharing biometric data. OBIM is the lead designated provider of biometric identity services for DHS, and maintains the largest biometric repository in the U.S. government.
This system, called the Automated Biometric Identification System or IDENT, is operated and maintained by OBIM. IDENT currently holds more than 200 million unique identities and processes more than 300,000 biometric transactions per day.
Through biometric interoperability with the Department of Defense (DoD) and the Department of Justice (DoJ), DHS shares critical biometric information using advanced data filtering and privacy controls to support the homeland security, defense, and justice missions.