How Not to Red Team

During the 2015 summer travel season, airline passengers were stunned by a finding that was never supposed to be made public, but which leaked to ABC News. Auditors from the Department of Homeland Security (DHS) Office of Inspector General (OIG) had successfully smuggled weapons and fake explosives past Transportation Security Administration (TSA) checkpoints sixty-seven times out of seventy attempts at multiple domestic airports earlier that year. The DHS Inspector General John Roth later told a Congress that the auditors did not have “any specialized background or training,” meaning they were not especially proficient or skilled red teamers.

Roth also warned in prepared congressional testimony that after 115 audits conducted over the previous eleven years and “despite spending billions on aviation security technology, our testing of certain systems has revealed no resulting improvement.” Roth added: “Our audits have repeatedly found that human error—often a simple failure to follow protocol—poses significant vulnerabilities.” This revelation of inadequate security at domestic airports led to a series of reforms and retraining within TSA, which explained the longer lines that passengers faced in early 2016.

The results of these leaked covert tests relate to a new Government Accountability Office (GAO) report that reviewed the results of TSA’s own testing of its Transportation Security Officers (TSO). There were two remarkable GAO findings about these covert tests, which the TSA calls its Aviation Screening Assessment Program (ASAP). Both findings violate two of my red team best practices: vulnerability probes should be independent and unannounced, and conducted in a manner that resembles how motivated adversaries would attempt to breach a system; and institutions must be willing to hear the bad news from the red team, and develop a work plan to mitigate the vulnerabilities that only the red teamers can uncover.

First, TSA had determined that the TSOs performed much better at finding prohibited items when covert tests were conducted by TSA field officers at local airports, when compared to those by outside contractors up to October 2015. The GAO report states:

“According to TSA officials, TSOs at these forty airports performed more poorly in the ASAP tests conducted by the contractor personnel as compared to the prior ASAP testing done by the local TSA personnel—indicating that these prior-year pass rates were likely showing a higher level of performance than was actually the case….

“According to TSA officials, initial results from the contractor’s work seem to confirm their prior concerns (before the contractor testing was conducted) that problems exist with successfully maintaining the covert nature of tests at airports…With respect to the difficulty in maintaining the covert nature of the tests, TSA officials at seven of ten of the airports we contacted indicated challenges with obtaining anonymous role players to ensure that the ASAP tests remain covert. For example, TSA officials at one airport we visited reported having to rely on the availability of state and local government employees and U.S. Customs and Border Protection personnel to perform as role players. Another smaller airport we visited reported challenges finding role players among local TSA personnel that the TSOs working the screening lanes would not recognize. As a result, they tend to use new hires, National Guard, Federal Aviation Administration, and Federal Bureau of Investigation personnel.”

In other words, the TSOs could identify the testers by their appearance or knew them personally. The testers’ appearances or behaviors while standing in the screening lines could have also given away their occupation as government employees and not civilian travelers. This can happen even unconsciously when covert testers have a professional affinity with defenders, and want them to succeed at stopping their smuggling of prohibited items. Finally, the testers may have known airport workers or TSOs individually, and tipped-them off ahead of time. This happens far too frequently in private sector and government security testing. (TSA officials responded to the GAO finding by stating they would give local security directors more authority to use testers that were not government employees.)

The second finding from the GAO report was even more disturbing. The entire point of testing the security of a defensive system is to identify vulnerabilities in order to provide the defenders with prioritized, corrective measures that improve security. Unfortunately, the GAO investigators found that:

“TSA headquarters does not require FSDs [Federal Security Directors] to implement recommendations from the six-month cycle reports nor does it track whether the recommendations have been implemented, or conversely, reasons for not implementing them. TSA officials stated that the various recommendations cited in the cycle reports are strictly for the consideration of FSDs in the field and implementation is not mandatory.”

TSA’s senior leaders did not monitor whether local security officials were implementing the corrective measures, which were revealed to be necessary by the covert tests. If there are no mandatory requirements to reduce vulnerabilities uncovered by covert tests, then such testing is pointless. A red team vulnerability probe that is conducted and ignored can be far worse than one that is never conducted at all. These two red team “worst practices” are not unique to TSA. However, given terrorists’ longstanding interests of attacking commercial aircraft, as well as the sobering DHS findings from 2015, this new GAO report should be a cause of concern for policymakers and airline passengers alike.