Previous Chapter: 2 Overview and High-Level Recommendations
Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.

3
Evaluation of the WSLAT Feasibility Study

This chapter presents the results of the committee’s review of the WSLAT feasibility study1 that was specifically requested in the Statement of Work. The WSLAT feasibility study responds to the directive dated July 9, 2002, from the Director, Operation Test and Evaluation (DOT&E) to incorporate whole system testing with live biological warfare agents, both before and after the full-rate production decision for JBPDS.2 The committee’s consideration of the overall need for WSLAT, and a recommended alternative approach, are presented elsewhere in this report. This section is directed at the feasibility study of February 2004 and the requirement on which it is based, and is independent of the committee’s other recommendations.

The WSLAT feasibility study proposes a single design approach to satisfy the requirements of the DOT&E directive. It concludes with a plan and construction schedule to meet the technical and operational deadlines, with the clear proviso that certain decisions and funding should be made in a timely manner. The committee notes that the time for decision and funding is now several months overdue. As a result, the risks of not establishing WSLAT successfully and not

1  

Feasibility Study for the Whole System Live Agent Testing Program, Battelle Memorial Institute, 505 King Avenue, Columbus, OH 43201. WDTC Document No. WDTC-FS-03-125, West Desert Test Center, U.S. Army Dugway Proving Ground, UT 84022-5000. February 2004. This document is referred to as the WSLAT feasibility study in this report.

2  

Memo, July 9, 2002, Office of the Secretary of Defense, Operational Test and Evaluation, Deputy Director, SUBJECT: Joint Biological Point Detection System (JBPDS) Test and Evaluation (T&E) Strategy.

Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.

obtaining data and results to support the JBPDS decision schedule are already very high and becoming higher as the schedule compresses.

The committee is not optimistic that the proposed WSLAT approach will succeed. The design has major unresolved technical questions. The schedule is highly compressed and already behind. The committee also concluded that risk identification, analysis, and proposed management were particularly weak. The committee’s observations on the feasibility study will be presented in three primary groups. The first addresses the requirements analysis, the second addresses the scientific and engineering approach, and the third group of comments addresses regulatory and policy concerns.

In addition to these comments specifically about the WSLAT feasibility study, the committee has noted in the report that a credible testing program requires more than testing facilities. It also needs supporting methodology, materials (e.g., agents and simulants), and validated procedures. To be useful, the WSLAT should be operated according to an overall, integrated test and evaluation methodology. The live agents it will use must be validated and correlated with a suite of optimized and dependable simulants. Neither the methodology nor the suite of simulants is adequate, and WSLAT alone will not suffice.

SCIENTIFIC AND ENGINEERING CONCERNS ON THE PROPOSED WSLAT

The scientific and engineering concerns arise from the apparent lack of engineering design and analysis, even at a late stage of the schedule. The schedule itself includes many elements of high risk, and is already behind. The study has insufficient engineering detail, so the comments are more general in nature than they would have been had detailed design information been available for the committee. If a detailed engineering design for WSLAT is prepared, then the independent advisory panel recommended in this report could serve as a review body for the design.

IMPACT OF POLICIES AND REGULATIONS

In its evaluation of the regulatory aspects of WSLAT, the committee has no comment regarding DOD regulations and procedures, nor on DPG’s obligations to the State of Utah. The committee’s comments address only implications of compliance with the National Environmental Policy Act requirements.

The committee concludes that to be complete, the WSLAT feasibility study should address environmental requirements, and the potential for controversy and subsequent delay, in a rigorous fashion.

Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.

SUMMARY COMMENTS AND RECOMMENDATION

The committee is concerned that the WSLAT feasibility study is not mature and that the methodology, technology, time, and management attention available are not adequate to provide a reasonable expectation of success. The study writers themselves refer frequently to data that are not available (but essential), and the need to develop additional technology and methodology. The project is very high risk, and is already behind schedule. The difficulty the committee has experienced in obtaining information and data critical to its own study, cited in the Interim Report, is not a good indicator that the funds, attention, and management needed to execute this program will be made available.

Recommendation 3-1: An analysis of alternative WSLAT design approaches that would include comparisons of the various key performance requirements in each design approach should be undertaken. The analysis should include a thorough discussion and analysis of risk, mitigation methods, and development of a sound risk mitigation plan.

The committee considers these to be essential features before realistic funding and schedule decisions can be made.

Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.
Page 16
Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.
Page 17
Suggested Citation: "3 Evaluation of the WSLAT Feasibility Study." National Research Council. 2005. Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary. Washington, DC: The National Academies Press. doi: 10.17226/11287.
Page 18
Next Chapter: 4 Recommended Evaluation Strategies
Subscribe to Email from the National Academies
Keep up with all of the activities, publications, and events by subscribing to free updates by email.