TSA Modernization: Use of Sound Program Management and Oversight Practices Is Needed to Avoid Repeating Past Problems. GAO-18-46: Oct 17, 2017
TSA conducts security threat assessment screening and credentialing activities for millions of workers and travelers in the maritime, surface, and aviation transportation industries that are seeking access to transportation systems. In 2008, TSA initiated the TIM program to enhance the sophistication of its security threat assessments and to improve the capacity of its supporting systems. However, the program experienced significant cost and schedule overruns, and performance issues, and was suspended in January 2015 while TSA established a new strategy. The program was rebaselined in September 2016 and is estimated to cost approximately $1.27 billion and be fully operational by 2021 (about $639 million more and 6 years later than originally planned).
GAO was asked to review the TIM program's new strategy. This report determined, among other things, the extent to which (1) TSA implemented selected key practices for transitioning to Agile software development for the program; and (2) TSA and DHS are effectively overseeing the program's cost, schedule, and performance. GAO compared program documentation to key practices identified by the Software Engineering Institute and the Office of Management and Budget, as being critical to transitioning to Agile and for overseeing and governing programs.
The Transportation Security Administration's (TSA) new strategy for the Technology Infrastructure Modernization (TIM) program includes using Agile software development, but the program only fully implemented two of six leading practices necessary to ensure successful Agile adoption. Specifically, the Department of Homeland Security (DHS) and TSA leadership fully committed to adopt Agile and TSA provided Agile training. Nonetheless, the program had not defined key roles and responsibilities, prioritized system requirements, or implemented automated capabilities that are essential to ensuring effective adoption of Agile. Until TSA adheres to all leading practices for Agile implementation, the program will be putting at risk its ability to deliver a quality system that strengthens and enhances the sophistication of TSA's security threat assessments and credentialing programs.
TSA and DHS fully implemented one of the key practices for overseeing the TIM program, by establishing a process for ensuring corrective actions are identified and tracked. However, TSA and DHS did not fully implement the remaining three key practices, which impede the effectiveness of their oversight. Specifically,
TSA and DHS documented selected policies and procedures for governance and oversight of the TIM program, but they did not develop or finalize other key oversight and governance documents. For example, TSA officials developed a risk management plan tailored for Agile; however, they did not update the TIM system life-cycle plan to reflect the Agile governance framework they were using.
The TIM program management office conducted frequent performance reviews, but did not establish thresholds or targets for oversight bodies to use to ensure that the program was meeting acceptable levels of performance. In addition, department-level oversight bodies have focused on reviewing selected program life-cycle metrics for the TIM program; however, they did not measure the program against the rebaselined cost, or important Agile release-level metrics.
TIM's reported performance data were not always complete and accurate. For example, program officials reported that they were testing every line of code, even though they were unable to confirm that they were actually doing so, thus calling into question the accuracy of the data reported.
These gaps in oversight and governance of the TIM program were due to, among other things, TSA officials not updating key program management documentation and DHS leadership not obtaining consensus on needed oversight and governance changes related to Agile programs. Given that TIM is a historically troubled program and is at least 6 months behind its rebaselined schedule, it is especially concerning that TSA and DHS have not fully implemented oversight and governance practices for this program. Until TSA and DHS fully implement these practices to ensure the TIM program meets its cost, schedule, and performance targets, the program is at risk of repeating past mistakes and not delivering the capabilities that were initiated 9 years ago to protect the nation's transportation infrastructure.
GAO is made 14 recommendations, including that DHS should prioritize requirements and obtain leadership consensus on oversight and governance changes. DHS concurred with all 14 recommendations.
Recommendation: The TSA Administrator should ensure that the TIM program management office establishes and implements specific time frames for determining key strategic implementation details, including how the program will transition from the current state to the final TIM state. (Recommendation 1)
Recommendation: The TSA Administrator should ensure that the TIM program management office establishes a schedule that provides planned completion dates based on realistic estimates of how long it will take to deliver capabilities. (Recommendation 2)
Recommendation: The TSA Administrator should ensure that the TIM program management office establishes new time frames for implementing the actions identified in the organizational change management strategy and effectively executes against these time frames. (Recommendation 3)
Recommendation: The TSA Administrator should ensure that the TIM program management office defines and documents the roles and responsibilities among product owners, the solution team, and any other relevant stakeholders for prioritizing and approving Agile software development work. (Recommendation 4)
Recommendation: The TSA Administrator should ensure that the TIM program management office establishes specific prioritization levels for current and future features and user stories. (Recommendation 5)
Recommendation: The TSA Administrator should ensure that the TIM program management office implements automated Agile management testing and deployment tools, as soon as possible. (Recommendation 6)
Recommendation: The TSA Administrator should ensure that the TIM program management office updates the Systems Engineering Life Cycle Tailoring Plan to reflect the current governance framework and milestone review processes. (Recommendation 7)
Recommendation: The TSA Administrator should ensure that the TIM program management office establishes thresholds or targets for acceptable performance-levels. (Recommendation 8)
Recommendation: The TSA Administrator should ensure that the TIM program management office begins collecting and reporting on Agile-related cost metrics. (Recommendation 9)
Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that program velocity is measured and reported consistently. (Recommendation 10)
Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that unit test coverage for software releases is measured and reported accurately. (Recommendation 11)
Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that appropriate DHS leadership reaches consensus on needed oversight and governance changes related to the frequency of reviewing Agile programs, and then documents and implements associated changes. (Recommendation 12)
Comments: In October 2017, DHS provided its completed guidance which included recommended practices for collecting and reporting on agile performance metrics, as well as a set of core agile performance metrics that programs should report to the Department. As a result, DHS has better assurance that agile development programs will report informative performance metrics to oversight entities so that they can ensure the programs are effectively delivering their intended capabilities and outcomes.
Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics. (Recommendation 13)
Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that DHS-level oversight bodies review key Agile performance and cost metrics for the TIM program and use them to inform management oversight decisions. (Recommendation 14)