By Cliff Montgomery – Nov. 13th, 2013
Since 2007, the Transportation Security Administration (TSA) has deployed a program which profiles airlinepassengers “without validating its scientific basis” or performance, a federal watchdog has concluded.
The TSA began its Screening of Passengers by Observation Techniques (SPOT) program “in fiscal year2007 – and has since spent about $900 million – to identify persons who may pose a risk to aviation securitythrough the observation of behavioral indicators,” even though the program has no clear logical or scientificbasis, according to the U.S. Government Accountability Office (GAO).
The TSA disagrees with the GAO’s findings.
Below, The American Spark quotes the findings discussed in the GAO study’s ‘Highlights’ page:
Why GAO Did This Study
“TSA [the Transportation Security Administration] began deploying the SPOT [Screening of Passengers byObservation Techniques] program in fiscal year 2007—and has since spent about $900 million—to identifypersons who may pose a risk to aviation security through the observation of behavioral indicators. In May2010, GAO concluded, among other things, that TSA deployed SPOT without validating its scientific basisand SPOT lacked performance measures.
“GAO was asked to update its assessment. This report addresses the extent to which: (1) available evidence supports the use of behavioral indicators to identify aviation security threats, and (2) TSA has the data necessary to assess the SPOT program’s effectiveness.
“GAO analyzed fiscal year 2011 and 2012 SPOT program data. GAO visited four SPOT airports, chosen onthe basis of size, among other things, and interviewed TSA officials and a non-probability sample of 25randomly selected BDOs. These results are not generalizable, but provided insights.
What GAO Found
“Available evidence does not support whether behavioral indicators, which are used in the TransportationSecurity Administration’s (TSA) Screening of Passengers by Observation Techniques (SPOT) program, canbe used to identify persons who may pose a risk to aviation security.
“GAO reviewed four meta-analyses (reviews that analyze other studies and synthesize their findings) thatincluded over 400 studies from the past 60 years and found that the human ability to accurately identifydeceptive behavior based on behavioral indicators is the same as or slightly better than chance.
“Further, the Department of land Security’s (DHS) April 2011 study conducted to validate SPOT’sbehavioral indicators did not demonstrate their effectiveness because of study limitations, including the useof unreliable data. Twenty-one of the 25 Behavior Detection Officers (BDO) GAO interviewed at four airportssaid that some behavioral indicators are subjective.
“TSA officials agree, and said they are working to better define them. GAO analyzed data from fiscal years2011 and 2012 on the rates at which BDOs referred passengers for additional screening based on behavioralindicators and found that BDOs’ referral rates varied significantly across airports, raising questions about theuse of behavioral indicators by BDOs. To help ensure consistency, TSA officials said they deployed teamsnationally to verify compliance with SPOT procedures in August 2013.
“However, these teams are not designed to help ensure BDOs consistently interpret SPOT indicators.
“TSA has limited information to evaluate SPOT’s effectiveness, but plans to collect additional performancedata. The April 2011 study found that SPOT was more likely to correctly identify outcomes representing ahigh-risk passenger—such as possession of a fraudulent document—than through a random selectionprocess.
“However, the study results are inconclusive because of limitations in the design and data collection andcannot be used to demonstrate the effectiveness of SPOT.
“For example, TSA collected the study data unevenly. In December 2009, TSA began collecting data from 24airports, added 1 airport after 3 months, and an additional 18 airports more than 7 months later when itdetermined that the airports were not collecting enough data to reach the study’s required sample size. Sinceaviation activity and passenger demographics are not constant throughout the year, this uneven datacollection may have conflated the effect of random versus SPOT selection methods.
“Further, BDOs knew if passengers they screened were selected using the random selection protocol orSPOT procedures, a fact that may have introduced bias into the study. TSA completed a performance metricsplan in November 2012 that details the performance measures required for TSA to determine whether itsbehavior detection activities are effective, as GAO recommended in May 2010.
“However, the plan notes that it will be 3 years before TSA can begin to report on the effectiveness of itsbehavior detection activities. Until TSA can provide scientifically validated evidence demonstrating thatbehavioral indicators can be used to identify passengers who may pose a threat to aviation security, theagency risks funding activities that have not been determined to be effective.
“This is a public version of a sensitive report that GAO issued in November 2013. Information that TSAdeemed sensitive has been redacted.
What GAO Recommends
“Congress should consider the absence of scientifically validated evidence for using behavioral indicators toidentify threats to aviation security when assessing the potential benefits and cost in making future fundingdecisions for aviation security.
“GAO included this matter because DHS did not concur with GAO’s recommendation that TSA limit futurefunding for these activities until it can provide such evidence, in part because DHS disagreed with GAO’sanalysis of indicators.
“GAO continues to believe the report findings and recommendation are valid.”
