Hurricane Maria struck the Arecibo Telescope during the period in which the National Science Foundation (NSF) would have been considering proposals in response to its solicitation for the Management and Operations of the Arecibo Telescope, due on May 4, 2017 (NSF-538), and the award to the University of Central Florida (UCF). Following the award of the Arecibo Telescope operating contract to UCF, the full management contract started on April 1, 2018.1 The transition period of Arecibo Observatory (AO) to UCF lasted from February 2018 through June 2018, supported by an award for $913,935.2 SRI International was the managing organization until the end of March 2018, and the organization received an extension to support close out until May 31, 2018, more than 6 months after Hurricane Maria. There were several meetings among NSF, SRI, and UCF during the facility’s turnover. The AO site director, as well as the majority of the AO staff, remained throughout this transition. The $20 million grant to fund UCF’s operations and maintenance (O&M) award was not received until April 2018,3 7 months after Maria had struck the Arecibo Telescope. The UCF proposal separated the Arecibo Telescope’s science and maintenance functions, with the responsibility “for facilities infrastructure, engineering, operations, maintenance, information technology and support services, as well as logistics and security services” to be assumed by Yang Enterprises, Inc.4 While the announcement stated that “Yang Enterprises, Inc. will introduce new technologies and cutting-edge tools to support Arecibo’s requirements and modernize operations,”5 the committee did not find evidence of significant change in the Arecibo Telescope’s inspection methods and maintenance practices after UCF assumed control of operations. During the transition period, UCF management planned to implement a computer maintenance management system. Most maintenance records, however, were kept onsite as hard copies.
UCF assumed the Arecibo Telescope’s operation in the face of a budget that had been declining over the previous 5 years, and NSF announced a decision in 2017 (a few weeks after Maria) that “operations at the observatory
___________________
1 National Science Foundation (NSF), 2023, “Operations and Maintenance of the Arecibo Observatory,” NSF Grant Award #1822073, https://www.nsf.gov/awardsearch/showAward?AWD_ID=1822073&HistoricalAwards=false.
2 NSF, 2018, “Statement on Award of Cooperative Agreement for Management and Operations of Arecibo Observatory,” February 22, https://www.nsf.gov/news/news_summ.jsp?cntn_id=244536&org=NSF&from=news.
3 Z. Kotala, 2019, “UCF Marks 1st Year at Arecibo Observatory in Puerto Rico,” UCF TODAY, University of Central Florida, April 5, https://www.ucf.edu/news/ucf-marks-1st-year-arecibo-observatory-puerto-rico.
4 N.T. Tillman, 2018, “Arecibo Observatory Announces Identity of ‘Mysterious Benefactors,’” Space.com, February 27, https://www.space.com/39827-arecibo-observatory-new-management.html.
5 Ibid.
[continue] with reduced agency funding.”6 However, when partner funding failed to materialize to make up the difference, NSF maintained the O&M budget. Eight months after Hurricane Maria, NSF awarded UCF $2 million to be focused on time-critical repairs.7 In 2019, “NSF submitted waiver to OMB to permit $11.3M of $14.3M to be spent over 60 months.”8 Following the hurricane, repairs began in December; however, they could not access the platform because of the catwalk damage.
Both through solicitations and oversight, NSF requires the contractor to take ownership of the maintenance and operation of the facility.9 However, maintenance, inspection, and repairs are terms often misunderstood and thus not correctly applied when justifying funding for the operation of aging and dated facilities. Based on the committee’s professional experience, this lack of appreciation for the importance of maintenance, inspection, and repair is especially true for large-scale assets that have become technologically dated for their original intended purposes and are no longer state of the art. Financial administrators may regard maintenance and inspection as optional expenses that can be deferred while regarding repairs as expenses, and even then, only as long as the damaged items needing repair negatively affect the main operation of the facility and thus cannot be postponed. Decisions to reduce operations and maintenance funding in favor of other activities assume some risk to the health and performance of the structure. The challenge, then, is that during times of managerial change, the new personnel may not be fully informed of these assumed risks. Therefore, if a low-probability event does occur, it may happen to staff unaware of the risk.
Recommendation: The National Science Foundation and organizations that use similar site management contracts to manage their portfolios should consider funding for the inspection, monitoring, maintenance, and repair of aging facilities and infrastructure as important as they are critical to the structure’s performance and longevity.
For any complex, complicated, and unique facility operated by NSF, such as the Arecibo Telescope, inspection, monitoring, maintenance, and repairs must occur at two levels. First and most obvious, a Level One maintenance, inspection, and repair program should apply to the operational components of the facility, such as the antennas, receivers, and mechanical and electronic equipment. The malfunctioning of any one of these components is readily noticed and addressed by the operators of the facility in charge of conducting the research activities. The facility’s equipment items are often supplied by a manufacturer who typically provides maintenance, inspection, and repair manuals for their equipment. Furthermore, these manuals also provide guidance for addressing probable malfunctioning and troubleshooting problems and verifying correct operation. These manufacturers have the most knowledge of all the design assumptions, risks, and unknowns regarding the performance of their equipment. Other aspects of this Level One maintenance, inspection, and repair of a facility include typical tasks such as painting, corrosion staining, disconnected cables, etc., which do not require an understanding of the complexity of the facility, just experience with typical daily maintenance of any facility.
Level Two maintenance, inspection, and repair of the facility is often problematic unless there is a specific and informed delegation of this responsibility. NSF facilities with long life expectancies, like the Arecibo Telescope, often incorporate or upgrade with state-of-the-art technologies and materials that, at the time of design and construction, were justified as a calculated risk to obtain other benefits. By the very nature of advanced design, long-term performance has not been demonstrated under the design conditions. Experience is being gained through the application of many continuously evolving technologies, and thus, their performance must be verified with time.
___________________
6 NSF, 2017, “Arecibo: Statement on NSF Record of Decision,” November 16, https://www.nsf.gov/news/news_summ.jsp?cntn_id=243729.
7 NSF, 2018, Award 1838728, “Hurricane Maria Relief Funding for Restoration of Scientific Capabilities at the Arecibo Observatory,” May, https://www.nsf.gov/awardsearch/showAward?AWD_ID=1838728.
8 A. VanderLey, 2022, “Arecibo Observatory: Failure Event Sequence,” National Science Foundation presentation to the committee, January 25, slide 13.
9 See the following NSF Program Solicitations: NSF 10-562, “Management and Operations of the National Astronomy and Ionosphere Center (NAIC),” posted April 29, 2010, https://new.nsf.gov/funding/opportunities/nrao-management-operation-national-radio-astronomy-observatory/503352/nsf10-562/solicitation, and NSF 17-538, “Management and Operations of the Arecibo Observatory,” posted January 25, 2017, https://new.nsf.gov/funding/opportunities/management-operations-arecibo-observatory/505401/nsf17-538/solicitation.
The Design Engineer or Engineer of Record (EOR) for the original design or significant modifications/upgrades is most knowledgeable concerning the risks, technologies, assumptions, redundancies, weaknesses considered in the design, and the potential threats to overall structural integrity. The EOR generally has knowledge of proof testing, field validation, and quality control in design and construction, what critical structural components need to be monitored, locations to inspect, and the limits of adequate performance of the critical components within the context of their design scope. Unfortunately, the original EOR is often not involved in significant changes or upgrades, particularly when there is a significant time lag involved. In these cases, the EOR for the changes must assume this role for all parts of the structure that are impacted by modifications, even if they were not modified.
The most recent EOR should provide a written narrative for the facility records describing the final conceptual design and its expected performance, as well as identify the critical components affecting the structure’s performance and how these should be monitored. This step is doubly important for a facility whose planned operation is by competitively bidding contractors who will have had no role in the facility design. Furthermore, the EOR should be retained to develop an inspection and monitoring manual for the ultimate facility operator, clearly outlining the procedures to monitor the performance of the most critical components affecting the facility’s structural integrity. To the extent possible, the prescribed monitoring should not require engineering knowledge or judgment from the inspectors and instead prescribe physical monitoring of non-judgmental parameters, such as counted wire breaks or measured cable pullouts. The inspection manual should also clearly identify the acceptable limits for the facility’s operation and inspection actions to be taken when the operation, or degradation such as a cable pullout, exceeds the acceptable limits, including safety measures to ensure the safety of the personnel, employees, scientists using the facility, and the visiting public. Online structural monitoring and health management, condition monitoring, condition-based maintenance, and reliability-centered maintenance are typical of today’s monitoring techniques.
The inspection, monitoring, maintenance, and repair program of the structural portion of the facility developed by the EOR should include clear guidelines for reporting and evaluating the information in the reports and records from both the Level One and the Level Two programs. Of utmost importance is the fact that the inspection, monitoring, maintenance, and repair program should include guidelines for reporting and assessing the criticality of any structural findings, especially those considered unusual or unexpected. The Arecibo Telescope lacked the proper inspection, monitoring, maintenance, and repair manuals addressing critical structural considerations that only the EORs in the original design and the upgrade knew. This deficiency is most significant when considering that the expected service life for the facility was 10 years.10 The Arecibo Telescope underwent several changes, which significantly increased the applied loads. At the time of the collapse, it had been in service for 57 years. NASA concluded, “The design did not explicitly consider the time-dependent effects of creep and cyclic loading on design capability, nor did it set service life inspection intervals with pass/fail inspection criteria. It also did not specify an end-of-life capability requirement associated with service life degradation.”11 All of these deficiencies resulted in a facility that had exhibited unheeded signs of distress for several years.
The collapse on December 1, 2020, resulted from the lack of recognition of the significance of the cable pullout by the experts contracted to assess the Arecibo Telescope and to develop a response plan after Hurricane Maria. There was inadequate funding for the repairs or replacements needed because of the cable pullout. The operating entity and its managers failed to consider all the findings from the facility inspections to ensure the safety of its employees, scientists, staff, and the public visiting the facility.
The following findings and recommendations are relevant to any large structural facility owned by NSF with contract operators.
Finding: A Level Two inspection, maintenance, monitoring, and repair manual was not developed by the EOR or NSF, nor was such a policy implemented during the management of the Arecibo Telescope by its last two operators.
___________________
10 Interview of William Gordon by Andrew Butrica, Niels Bohr Library & Archives, American Institute of Physics, College Park, MD, November 28, 1994, http://www.aip.org/history-programs/niels-bohr-library/oral-histories/22789.
11 NESC Report, p. 120.
Conclusion: A good practice would be to have separate accounting and funding, independent of facility operations, for inspection, maintenance, monitoring, and repair of facilities. Otherwise, inspection, maintenance, monitoring, and repair may suffer in financial competition with the science operations that can gain the facility further funding.
Recommendation: Facility owners should enforce compliance with contract requirements through independent auditing of inspection, monitoring, maintenance, and repair records.
The Arecibo Telescope’s condition monitoring, while spotty throughout its history, demonstrably degraded after the 1997 upgrade and further degraded after 2011. The first comprehensive Ammann & Whitney inspection of the telescope after the completion of the Gregorian upgrade in 1997 was in 2003. The following comprehensive inspection was not until late 2010. While there were continuous inspections, apart from broken wire counts, these were generally only visual inspections, and photographic documentation is irregular. In the obsolete system, there was little codification or organization of the data with which to observe any trends. There appear to have been no specified procedures as to who should be contacted if there were any serious or discrepant findings. After supervisors were informed, there was no process to involve the chief engineer for additional analysis. There was also no central repository of the maintenance and inspection data at the facility, as shown when UCF began management of the facility, and UCF contemplated visiting Cornell University to review the archives.12 Even though the Arecibo Telescope was aging, inspection methodologies did not change and seem to have become less frequent, as indicated by the reduced available inspection data discussed previously. By the time extensometers were considered, it was too late. Cable and socket maintenance consisted of applying sacrificial zinc paint; epoxy paint was used everywhere else.
___________________
12 R. Lugo and F. Cordova, 2022, “Perspectives on Grant Award and Operations of Arecibo Observatory Cooperative Agreement by the University of Central Florida,” University of Central Florida presentation to the committee, February 17, meeting transcript minute 53:13.