NetCDF4 Reformatting Toolkit: BUFR and GRIB2 Tailoring

NetCDF4 Reformatting Toolkit: BUFR and GRIB2 Tailoring

NetCDF4 Reformatting Toolkit (N4RT): BUFR and GRIB2 Tailoring Test Readiness Review (TRR) for Phase 1 SDR Products October 25, 2011 Prepared By: Tom King1, Yi Song1, 1 Kexin Zhang, Larisa Koval1, and Walter Wolf2, 1 Riverside, 2STAR 1 Purpose of TRR/CTR Review the status of all the open issues/risks Review Project Requirements Describe the Software Architecture Describe the tests for the software units and show the test results Establish the contents of the Delivered Algorithm Package Identify any new issues/risks 2 Review Agenda Section

Time Presenter Introduction 9:00-9:10 Walter Wolf CDR Review Report/Risks and Actions 9:10-9:25 Thomas King Requirements 9:25-9:40 Thomas King Quality Assurance 9:40-9:50 Thomas King

Software Architecture 9:50-10:10 Thomas King Unit Tests 10:10-10:50 Yi Song Delivered Algorithm Package 10:50-11:00 Thomas King Risks and Actions Summary 11:00-11:15 Thomas King Summary and Conclusions 11:15-11:20

Walter Wolf 3 Review Outline Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 4 Section 1

Introduction Presented by Walter Wolf NOAA/NESDIS/STAR 5 Introduction Project Background IJPS NPP/JPSS NDE Project Objectives Integrated Product Team Project Plan Entry and Exit Criteria 6 JTA and IJPS JTA Joint Transition Activities JTA is a replacement of the Initial Joint Polar-Orbiting Operational Satellite System (IJPS) agreement and is designed to cover only the NPP mission. IJPS started with NOAA-N and covers the MetOp series. JTA and IJPS are cooperative efforts between NOAA and EUMETSAT to provide and improve the operational meteorological and

environmental forecasting and global climate monitoring services worldwide. The JPSS J1 and J3 data availability will be covered by the Joint Polar-Orbiting Operational Satellite System (JPS) agreement. 7 NPP/JPSS NPP and JPSS, a joint NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the JPSS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (13:30 pm). Meteorological/Climatological Instrument packages on NPP/JPSS:

CrIS, ATMS, VIIRS, OMPS, CERES NPP is the first of 3 missions with a launch date of October 28, 2011. 8 Project Background NDE Disseminate NPP/JPSS Data Records to customers. Generate and disseminate tailored NPP/JPSS Data Records (versions of NPP/JPSS Data Records in previously agreed alternative formats and views). Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPP/JPSS Data Records). Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving.

Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA. Provide software for NPP/JPSS Data Record format translation and other data manipulations. 9 N4RT Project Objectives To build a software package that will tailor JPSS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDEs overall tailoring efforts. The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products. Flexible Extendable The software must be able run within the NDE system architecture and operate within the NDE functional guidelines. Output product formats and content must meet the needs of NOAA customers. 10

N4RT Project Objectives Phase 1 Products Phase 1 SDR CrIS Radiances (BUFR) ATMS Radiances (BUFR) VIIRS Radiances (BUFR) Phase 1 EDR Sea Surface Temperature (BUFR) Aerosol Optical Thickness (BUFR) Nadir Profile and Total Column Ozone (BUFR) Phase 2 EDR Polar Winds (BUFR) Green Vegetation Fraction (GRIB2) Phase 3 EDR OMPS Limb Profiles (BUFR) AVHRR, GOES, VIIRS Cloud Products (GRIB2) 11 Integrated Product Team IPT Lead: Walter Wolf (STAR)

IPT Backup Lead: AK Sharma (OSPO) NESDIS team: STAR: Walter Wolf, Jaime Daniels, Yi Song, Thomas King, Kexin Zhang, Larisa Koval NDE: Jim Silva, Geof Goodrum, Richard Sikorski, Kevin Berberich OSPO: Dave Benner, AK Sharma, Ricky Irving OSD: Tom Schott User team Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber (NWS/NCEP/ EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard and Jim Yoe (JCSDA), Simon Elliott (EUMETSAT), Tony McNally (ECMWF), Fiona Hilton (UK-Met) Others: International NWP users, NWP FOs, Climate Users Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

12 Project Stakeholders NOAA National Weather Service Weather Forecast Offices National Center for Environmental Prediction Department of Defense NRL FNMOC AFWA Global NWP EUMETSAT ECMWF UK Met Meteo France CMC JMA

BOM 13 Plan of Operations Phase 1 Year 1 Design and Development Evaluate the requirements; work with NDE Discuss with the current developers of similar translators to determine what is required in their output files Design the NetCDF4 reformatting toolkit; distribute to OSPO and NDE for review Conduct PDR Create generic NetCDF4 readers and writers Develop BUFR tables and GRIB formats with the product teams for Phase 1 products Work with NDE to determine the interface between the SDR and EDR NPP products and the reformatter Conduct CDR 14 Plan of Operations Phase 1 Year 2 Transition to Pre-Operations of Phase 1 Products

Set up infrastructure to implement the readers and writers for the data formats; work with NDE to determine the interface to the data handling system; make available to OSPO for review Implement BUFR tables and GRIB formats for the Phase 1 products on the NDE hardware; work with NDE and OSPO to evaluate the implementation Conduct Test Readiness Review for Phase 1 products Transition Phase 1 product reformatters to pre-operational system on the NDE hardware Test system within the NDE environment Prepare Documentation Conduct Code Review for Phase 1 products Year 3 Transition to Operations of Phase 1 Products Evaluate with NDE and OSPO the implementation of the Reformatter within the NDE data handling system Conduct Algorithm Readiness Reviews (separate reviews for Phase 1 SDR and EDR products) Transition pre-operational Phase 1 product reformatting system to operations 15 Plan of Operations Phase 2 Year 1 Design and Development Evaluate the requirements; work with NDE

Discuss with the current developers of similar translators to determine what is required in their output files Develop BUFR tables and GRIB formats with the product teams for Phase 2 products Conduct CDR Implement BUFR tables and GRIB formats for the Phase 2 products on the NDE hardware; work with NDE and OSPO to evaluate the implementation 16 Plan of Operations Phase 2 Year 2 Transition to Pre-Operations of Phase 2 Products Conduct Test Readiness Review for Phase 2 products Transition Phase 2 product reformatters to pre-operational system on the NDE hardware Test system within the NDE environment Prepare Documentation Conduct Code Review for Phase 2 products Year 2 Transition to Operations of Phase 2 Products Evaluate with NDE and OSPO the implementation of the Reformatter within the NDE data handling system Conduct Algorithm Readiness Review for Phase 2 products Transition pre-operational Phase 2 product reformatting system to operations

17 Plan of Operations Phase 3 Year 1 Design and Development Evaluate the requirements; work with OSPO Discuss with the current developers of similar translators to determine what is required in their output files Develop BUFR tables and GRIB formats with the product teams for Phase 3 products Year 2 Transition to Pre-Operations of Phase 3 Products Implement BUFR tables and GRIB formats for the Phase 3 products on the OPSO hardware; work with OSPO to evaluate the implementation Conduct Test Readiness Review for Phase 3 products Transition Phase 3 product reformatters to pre-operational system on the OSPO hardware Test system within the OPSO environment Prepare Documentation Conduct Code Review for Phase 3 products Evaluate with OSPO the implementation of the Reformatter within the OSPO data handling system Conduct Algorithm Readiness Review for Phase 3 products 18

Transition pre-operational Phase 3 product reformatting system to operations Project Timeline (1) PDR 04/28/2009 CDR 09/29/2009 19 Project Timeline (2) TRR 10/25/2011 ARR 11/28/2011 DAP 1 Delivery 11/30/2011 20 Project Timeline (3) 21

Project Timeline (4) TRR 01/23/2012 SCR 03/5/2012 ARR 05/30/2012 DAP 2 Delivery 06/20/2012 22 Project Timeline (5) 23 Project Timeline (6) TRR 07/18/2013 SCR 08/26/2013 ARR 03/10/2014

DAP 3 Delivery 03/31/2014 24 Project Plan Schedule Schedule (Milestones) Project begins - 7/1/08 PDR - 4/14/09 CDR - 9/14/09 Phase 1 SDR TRR/CTR - 10/25/11 Phase 1 SDR ARR - 11/15/11

Phase 1 SDR DAP Delivery - 11/30/11 Phase 1 & 2 EDR TRR/CTR - 1/23/12 Phase 1 & 2 EDR SCR - 3/5/12 Phase 1 & 2 EDR ARR - 5/30/12 Phase 1 & 2 EDR DAP Delivery - 6/4/12 Phase 3 TRR/CTR - 7/18/13 Phase 3 SCR - 8/26/13 Phase 3 ARR - 3/10/14 Phase 3 EDR DAP Delivery - 3/31/14 25 N4RT TRR Entry Criteria CDR Report (Review Item Disposition) http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/NetCDF4_Reformatting _Toolkit_CDR_Review_Item_Disposition_20110818.xlsx PDR Risks and Actions CDR Risks and Actions CDR Actions and Comments

Updated CDR Presentation http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_CDR_20090914. ppt Updated Requirements Allocation Document http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_RAD_v1.2.docx Review of N4RT http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_TRR_Phase1_S DR_Products.pptx Requirement Allocation Quality Assurance Software Architecture Unit Tests and Results Delivered Algorithm Package 26

N4RT TRR Exit Criteria Test Readiness Review Report The TRR Report (TRRR), a standard artifact of the STAR Enterprise Process Lifecycle (EPL), will be compiled after the TRR The report will contain: Review Item Disposition containing all risks, actions and comments Updated TRR presentation 27 Review Objectives Review the CDR Review Report (CDRR) Focus on actions Review the Requirements Allocation Review the software system architecture External interfaces and data flows Dependency Tree Document Review the software tests and results of each software unit Identify risks and actions 28 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 29 Section 2 CDR Review Report and Actions Presented by Thomas King Riverside 30

CDR Reports and Actions Open PDR Risks and Actions at CDR. CDR Risks and Actions. Later in the review, new risks, actions, and comments originating from the CDR, and the period since, will be presented. 31 CDR Review Report A CDR Report (CDRR) is produced following a projects Critical Design Review (CDR). It is a required project artifact of the STAR Enterprise Product Lifecycle (EPL). Specifically, it is a designated artifact for the Test Readiness Review (TRR), which is a standard Technical Review in the STAR EPL process. The intended target audiences are program management, the product development team, and the TRR review team. A CDR Review Report is available for review in the project repository at: http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/ NetCDF4_Reformatting_Toolkit_CDR_Review_Item_Disposition_20110818.xlsx 32 Open PDR Risks and Actions Since CDR

33 PDR Risk Risk 1: JPSS product formats and content are still changing, especially for VIIRS Assessment: Low Impact: May need to revise software several times during development to adjust to new formats, names, and types. Likelihood: High Mitigation: Work through the Data Format Working Group to obtain information on format and algorithm updates. Monitor the latest copies of the Common Data Format Control Books (CDFCB) for any updates. Maintain contact with customers to inform them of any upstream product changes. Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision. Status: Closed. We recommend closing this as CDFCB formats are now frozen and the reformatter is able to read all necessary data from the P72 datasets to which NDE subscribes. 34 PDR Risk Risk 2: The roles and responsibilities regarding who shall generate the set of required SPSRB documents for NDE has not yet been decided. Assessment: Low Impact: Difficult to budget time needed for the team to generate

documentation. Likelihood: Moderate Mitigation: This issue, and that of document content, is being worked by Maurice McHugh, STAR, NDE, OSD, and OSPO personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions. Status: Closed. We recommend closing this. The updated SPSRB documentation has defined roles and responsibilities for writing of these documents on the STAR side. At TRR new project-level risk was opened to address NDE needing to modify their contract to complete this delivered documentation in time for transition to OSPO. 35 PDR Risk Risk 3: There are small variations in the types of platforms and the versions of the compilers Assessment: Low Impact: May obtain different results using different compilers. Likelihood: Low Mitigation: Reformatting Toolkit developers will work with NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The NDE Build Content Reviews and delivery of DAP prototypes should help to resolve these issues early on in development. Status: Open 36

PDR Risk Risk 4: Data format translation may involve some unit conversion and possible reduction of precision (significant digits) Assessment: Low Impact: Any modification of the data from its original form may not be apparent to the user. Likelihood: Low Mitigation: Document data manipulations in the NDE Delivered Algorithm Package (in place of ATBD). This will be done in the System Maintenance Manual (SMM). Status: Open 37 PDR Risk Risk 6: Risk on NDVI and snow mask. Have not yet demonstrated ability to encode GRIB2 files. Assessment: Moderate Impact: Failure to meet the requirement to have demonstrated GRIB2 tailoring capability. Likelihood: Low Mitigation: NDVI and snow mask will not be produced. However, GVF will need to be in GRIB2 format, but thats something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR scheduled for 1/23/2012. Status: Open. 38

CDR Risks and Actions 39 CDR Risk Risk 14: STAR and NDE CM need to collaborate to discuss formats, defining fields, the process for delivering DAPS with an official naming convention. Assessment: Low Impact: Confusion and delays caused by not having common naming conventions and not having standard DAP delivery process defined. Likelihood: Low Mitigation: STAR and NDE CM need to collaborate to discuss formats, defining fields, the process for delivering DAPS with an official naming convention. Status: Closed. We recommend closing as all this is covered in the NDE document entitled Algorithm Delivery Standards, Integration, and Test Version 1.3. 40 CDR Risk Risk 15: Need to identify an independent code review Assessment: Low Impact: Reduction in the quality of the delivered software. Likelihood: Low Mitigation: Document plan for an independent code review. Understand NESDIS QA process is being defined.

Status: Closed. We recommend closing this as this project schedule has TRR/CTRs for each phase and has added software reviews (code walk-throughs) for all future phases. OSPO and NDE are invited to participate in these reviews and OSPO will be expected to participate in the software reviews. The question is how much effort can groups outside STAR afford to invest given current budgets. If the reviewers agree this can be sufficient, this risk can be closed. 41 CDR Risk Risk 16: EMC are concerned that the conversion of HDF5 to netCDF4 may add significant time to the delivery of the products. Assessment: Low Impact: Increase in product latency and failure to meet customer needs. Likelihood: Low Mitigation: Need to conduct tests to verify that the conversion times meet latency requirements. Status: Closed. We recommend closing this given that the tests shown here demonstrate that the conversion times were sufficient to maintain latency. 42 CDR Risk Risk 17: Encourage NWS (EMC, AWIPS) personnel to take part in future reviews, major meetings, as well as working-level meetings. JCSDA and EMC were lightly represented in the CDR.

Assessment: Low Impact: Changing user needs may not be regularly communicated to the developers. Likelihood: Low Mitigation: May need to establish a working group that facilitates this communication. Status: Closed. We recommend closing this as regular monthly EMC/NDE/STAR meetings are held (Kevin Berberich). In addition, EMC & JCSDA representatives are always invited to this projects reviews. Tom Schotts VIIRS TIM helped resolve many remaining VIIRS SDR BUFR issues for Phase 1. 43 CDR Risk Risk 18: ESPC may not have the BUFR expertise to maintain the toolkit. Assessment: Low Impact: OSPO may not be able to maintain the toolkit in the future. Likelihood: Low Mitigation: OSPO is getting FY12 funding to support the transition and maintenance from NDE on the OSPO side. Status: Open. At TRR it was decided that this is a project-level risk that should remain open. The N4RT project plan anticipates STAR will provide upgrades and maintenance through 2014. 44 CDR Risk

Risk 19: Should OMPS NP and TC be in the same BUFR file? Assessment: Low Impact: Flexibility of FOV for OMPS NP pose a problem for converter if combined. Likelihood: Low Mitigation: These will be in separate files. Status: Closed. We recommend closing this. 45 Summary There are 11 risks: 5 from PDR 6 from CDR We recommend closing: 2 of the PDR risks. 5 of the CDR risks. 4 risks will remain open. 46 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 47 Section 3 Requirements Presented by Thomas King Riverside 48 Requirements Overview SPSRB Requirements were presented to the developers in a document entitled: Level 1 Requirements for a NetCDF4 Reformatting Tool (Version 1.5).

Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, the customers (EMC/JCSDA and EUMETSAT) and the heritage product teams. Using all of this information, a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project. All current Phase 1 and Phase 2 requirements are listed, but well focus mainly on those for Phase 1 SDR. Text highlighted Yellow indicate basic requirements Orange indicates new, modified, or removed requirements since CDR. 49 Phase I User-to-STAR Mapping

Prioritized Product EMC User Contacts STAR/OSPO Product and Heritage Contacts Heritage Product ATMS Radiances Dennis Keyser, John Derber Dennis Keyser, John Derber, Sid Boukabara AMSU, MHS, HSB CrIS Radiances Jim Jung, Dennis Keyser Jack Woollen, Simon Elliott, Tom King

IASI, AIRS VIIRS Radiances Dennis Keyser, John Derber Dennis Keyser, John Derber AVHRR GAC Nadir Profile and Total Column Ozone (OMPS) Dennis Keyser Larry Flynn, Donna McNamara SBUV, GOME Aerosol Optical Thickness Jeff McQeen Paul Haggerty

MODIS Aerosols Sea Surface Temperature Bert Katz, William Gemmill Shasha Ignatov, John Sapper, Robert Grumbine AVHRR derived SST (ACSPO) White = Phase 1 SDR BUFR Gray = Phase 1 EDR BUFR 50 Requirement 1.0 Requirement 1.0: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2). 51 Requirement 1.0

Requirement 1.1: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list. Requirement 1.2: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. Requirement 1.2.1: The Toolkit shall compile and run on the NDE IBM AIX P5, P6, and P7 series hardware. Requirement 1.2.2: The Toolkit shall be able interact with the NDE Data Handling System (DHS). Requirement 1.2.2.1: The Toolkit shall be able to read a Production Control File (PCF). Requirement 1.2.2.2: The Toolkit shall handle and return errors according to NDE/STAR standard codes.

Requirement 1.2.2.3: The Toolkit shall be able to write a (Product Status File) PSF. 52 Requirement 1.0 Requirement 1.3: The toolkit shall consist of modular components that can be tested independently and data should be stored in allocatable structures. Requirement 1.3.1: The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines. Requirement 1.4: STAR shall include one update to the reformatting toolkit within its initial project plan. Requirement 1.5: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products.

Requirement 1.6: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit. 53 Requirement 1.0 Requirement 1.7: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries. Requirement 1.7.1: Updates shall be made when there are updates to the versions of the NetCDF4 library being used by NDE. Requirement 1.8: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development. Requirement 1.9: The output from the toolkit shall be compared with the input to verify that the conversion was performed correctly.

Requirement 1.10: The translation toolkit shall convert from the new format back into NetCDF4. Requirement 1.11: The reformatting software shall log each transactions control information, 54 including: the calling application, the type of transaction requested, the start and end times, Requirement 1.0 Requirement 1.11.1: The Reformatting Toolkit software shall generate run logs (contain human-readable messaging and time stamps) and return NDE/STAR standard (agreed upon) error codes to the DHS. Requirement 1.12: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination). Requirement 1.13: The toolkit parameters (e.g., how to use the service) shall be well documented.

Requirement 1.13.1: Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP) whose name and contents are defined in the NDE document entitled Algorithm Delivery Standards, Integration, and Test V1.3. Requirement 1.13.2: The DAP shall contain the following two SPSRB documents: the SMM (System Maintenance Manual) and the EUM (External Users Manual) as well as the Test Readiness Document. 55 a Requirement 14.1: The messages provided by the toolkit in the event of failure to perform requested service shall be comprehensible by untrained operators. Requirement 1.0 Requirement 1.14.1: Reformatting Toolkit shall use the standard set of error return codes developed by NDE for code running within the DHS. Requirement 1.15: The messages provided by the toolkit in the event of failure to perform a

requested service shall include diagnostic details needed for troubleshooting. Requirement 1.15.1: All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP. Requirement 1.16: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. Requirement 1.17: Toolkit code shall adhere to the SPSRB coding standards. Requirement 1.18: Performance shall be measured on a product level. Requirement 1.19: The Toolkit shall output BUFR and GRIB2 files whose names adhere to the NDE file naming convention in Algorithm Delivery Standards, Integration, and Test V1.3. 56 Requirement 2.0

Requirement 2.0: STAR shall provide monthly project status reports to OSPO and OSD. 57 Requirement 3.0 Requirement 3.0: Earned Value Management shall be performed on the project. 58 Requirement 4.0 Requirement 4.0: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration. 59 Requirement 5.0 Requirement 5.0: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness. 60 Requirement 6.0 Requirement 6.0: The Reformatting Toolkit shall

tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC. 61 Requirement 6.0 Requirement 6.1: The Reformatting Toolkit developers shall work with EMC and the rest of the NWP community to create a BUFR table for the NUCAPS thinned and full resolution radiances based on AIRS and IASI. Requirement 6.2: The table shall use delayed replication for storing the radiances. Requirement 6.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) Requirement 6.4: The BUFR format shall allow for the storage of negative radiances.

Requirement 6.5: The file shall contain the following data fields (see table next slide): 62 Product Requirements: CrIS Radiances Variable Name Variable Name Variable Name Variable Name Satellite ID Latitude Height of Land Surface Start Channel (per band) ID of Originating Center Satellite Instrument

Longitude Satellite Height End Channel (per band) Satellite Zenith Angle Land Fraction Wavenumber Satellite Classification Satellite Azimuth Land/Sea Qualifier Calibration Quality Flags Year Solar Zenith Cloud Cover Field of View Quality Flags

Month Solar Azimuth Height of Cloud Top Geolocation Quality Day Radiance Type Flags NUCAPS Quality Hour Ascending/Descending flag Scan Line Number Scan-Level Quality Flags Channel Radiance Minute Field of Regard

Type of Band Second Field of View Location of Platform Orbit Number Starting Wavenumber (per band) Ending Wavenumber (per band) In dir. of North Pole, distance from the Earth's center In direction of 0 deg E, distance from Earth's center In direction of 90 deg E, distance from Earth's center 63 Requirement 6.0 Requirement 6.6: The reformatting Toolkit shall use the thinned CrIS radiances (399 channels) files from NUCAPS as an input for generating the

CrIS radiance BUFR files for EMC. Requirement 6.7: The reformatting Toolkit shall use the full spatial and spectral resolution CrIS radiances (1305 channels and all FOVs on all FORs) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EUMETSAT. 64 Requirement 7.0 Requirement 7.0: The Reformatting Toolkit shall tailor the JPSS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC. 65 Requirement 7.0 Requirement 7.1: The ATMS BUFR file shall contain, from all channels, the antenna and brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data. Requirement .72: The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS. Requirement 7.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)

Requirement 7.4: The file shall contain the following data fields (see table next slide): 66 Product Requirements: ATMS Radiances Variable Name Variable Name Variable Name Satellite ID FOV Number Solar Azimuth ID of Originating Center Granule-Level Quality Flags ATMS Channel Number ID of Originating Sub-Center Scan-Level Quality Flags

ATMS Central Frequencies Satellite Instrument Geolocation Quality ATMS Channel Bandwidth Satellite Classification Channel-Level Quality Flags Antenna Polarization Year Orbit Number Antenna Temperatures Month Latitude Brightness Temperatures Day

Longitude NeDT cold target Hour Satellite Height NeDT warm target Minute Satellite Zenith Angle Satellite Antenna Correction Version Number Second Satellite Azimuth Scan Line Number Solar Zenith 67 Requirement 7.0

Requirement 7.5: The Reformatting Toolkit shall use the JPSS ATMS TDR and SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files. 68 Requirement 8.0 Requirement 8.0: The Reformatting Toolkit shall tailor JPSS OMPS Ozone products from NetCDF4 into BUFR for EMC. 69 Requirement 8.0 Requirement 8.1: The product shall contain OMPS Nadir Profile and Total Column (multiple triplet algorithm, not v8, but output is equivalent to v8 according to Larry Flynn) in separate files. Requirement 8.2: The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV. Requirement 8.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) Requirement 8.4: This files content is currently under development. 70 Product Requirements: OMPS Nadir Profile

Variable Name Variable Name Variable Name Satellite ID Satellite Zenith Angle ID of Originating Center Satellite Azimuth Significance and Volumetric Mixing Ratio SO2 Index ID of Originating Sub-Center Solar Zenith Volcano Contamination Index Satellite Instrument Solar Azimuth

SBUV Total Ozone Quality Year Satellite Height SBUV Profile Ozone Quality Month Orbit Number Geolocation Quality Day Asc/Desc Flag Ozone Quality Flag Hour Vertical Significance Minute Pressure

Second Number of Retrieved Layers Latitude Total Ozone Longitude Ozone p (Dobson Units) 71 Requirement 8.0 Requirement 8.5: The Reformatting Toolkit shall use the JPSS OMPS Total Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files. 72 Requirement 9.0 Requirement 9.0: The Reformatting Toolkit shall tailor JPSS VIIRS SST products from NetCDF4 into BUFR for EMC. 73 Requirement 9.0

Requirement 9.1: Product shall contain Skin SST, Bulk SST, Quality Flags, and geolocation data. Requirement 9.2: Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS SST product. Requirement 9.3: The VIIRS SST BUFR table shall be derived from that currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans). Requirement 9.4: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) Requirement 9.5: The file shall contain the following data fields (see table next slide): 74 Product

Requirements: VIIRS SST Variable Name Variable Name Variable Name Satellite ID Latitude Pixel type ID of Originating Center Longitude Asc/Desc flag ID of Originating Sub-Center Satellite Zenith Angle Geolocation Quality Satellite Instrument

Satellite Azimuth VIIRS Geolocation Quality Satellite Classification Solar Zenith Retrieval Data Quality Year Solar Azimuth Adjacency Cloud Mask Month Satellite Height SST Pixel-Level Quality flag Day Scan Line Number SST

Hour FOV number SST bulk Minute Orbit Number Second Day/Night flag 75 Requirement 9.0 Requirement 9.6: The Reformatting Toolkit shall use the JPSS VIIRS SST EDR files and associated geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files. 76 Requirement 10.0 Requirement 10.0: The Reformatting Toolkit shall tailor JPSS VIIRS radiances, brightness temperatures, and reflectances from NetCDF4 into BUFR for EMC.

77 Requirement 10.0 Requirement 10.1: Each BUFR file will contain the VIIRS data for a single band (Imagery band, Moderate band, or Day/Night band resolution). Requirement 10.2: Coverage will be global. Requirement 10.3: Each of the 3 bands will use the same VIIRS BUFR table. Requirement 10.4: Removed: The product shall contain the land and cloud mask if it doesnt take too long for the IDPS to generate those EDRs. Requirement 10.5: Reformatting Toolkit developers shall work with EMC and the rest of the NWP user community to create a VIIRS BUFR table derived from that used earlier for the GAC AVHRR. 78 Requirement 10.0 Requirement 10.6: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) Requirement 10.7: The file shall contain the following data fields (see table next slide): 79 Product Requirements: VIIRS Radiances Variable Name

Variable Name Variable Name Satellite ID Latitude VIIRS Geolocation Quality ID of Originating Center Longitude Radiance Quality ID of Originating Sub-Center Satellite Zenith Angle Type of Band Satellite Instrument Satellite Azimuth Cloud Mask

Satellite Classification Solar Zenith Channel Number Year Solar Azimuth Channel Wavelength Month Satellite Height Channel Radiance Day Orbit number Channel Reflectance Hour Scan line number

Surface Type Minute FOV number Second Geolocation Quality 80 Requirement 10.0 Requirement 10.8: The BUFR shall contain the following Moderate and Imagery resolution channels as requested by EMC: Channel M12 (3.70 microns) Channel M13 (4.05 microns) Channel M15 (10.763 microns) Channel M16 (12.013 microns) Channel I5 (11.450 microns) 81 Requirement 10.0 Requirement 10.9: The toolkit shall not write data for obviously cloudy VIIRS FOVs. This will reduce the total data volumes. Obviously cloudy FOVs are those with

brightness temperatures in one of the longwave SST channels (M15 or M16) less than 270K. To preserve sea ice FOVs, this test shall only be applied for latitudes equatorward of 50 degrees. (Requirement removed see below) Note: An important concern is that more CPU time required to encode the BUFR than what NDE can supply. A recommended solution was to reduce the number of VIIRS channels to only a required subset, reject obviously cloudy data, and reject land-only granules. Very few granules, if any, will be entirely land-only. This would also require comparing the lat/lons of all points to a land mask which means more processing and we'd still have to read all the data anyway. Andrew Collard in an email on 6/8/2011 says "We don't NEED data thinning, we can ACCEPT data thinning." Therefore, we have decided to supply all the granules, but reduce the channels and employ the obviously-cloudy check. Removed, Bob Grumbine wants to keep frozen inland lakes, to do this we decided to not thin. This is documented in an email from Andrew on 11/9/2011. CCd were Bob Grumbine, John Derber, 82 Tom Schott, and Walter Wolf. Requirement 11.0 Requirement: The Reformatting Toolkit shall tailor JPSS Aerosol Optical Thickness (AOT) from NetCDF4 into BUFR for EMC. 83 Requirement 11.0 Requirement 11.1: The product shall contain the AOT, wavelength of AOT, and Aerosol Size. Requirement 11.2: Reformatting Toolkit developers shall work with EMC to develop the AOT BUFR table based on what has already been done for

MODIS. Requirement 11.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) Requirement 11.4: The file shall contain the following data fields (see table next slide): 84 Product Requirements: VIIRS Aerosol Optical Thickness Variable Name Variable Name Variable Name Satellite ID Latitude Asc/Desc Flag ID of Originating Center Longitude Geolocation Quality

ID of Originating Sub-Center Satellite Zenith Angle VIIRS Geolocation Quality Satellite Instrument Satellite Azimuth Retrieval Quality Satellite Classification Solar Zenith Aerosol Type (land) Year Solar Azimuth AOT Quality Flag Month Satellite Height

Day Orbit Number Aerosol Wavelength Angstrom Exponent Channel Wavelength Hour Scan Line Number Optical Depth Minute FOV Number Second Remotely Sensed Surface Type 85 Requirement 11.0 Requirement 11.5: The Reformatting Toolkit shall use the JPSS Aerosol Optical Thickness EDR files and associated Geolocation files tailored into netCDF4 as an input for generating the Aerosol Optical Thickness EDR

BUFR files. 86 Requirement 12.0 Requirement 12.0: The Reformatting Toolkit shall tailor NDE-generated Polar Winds product from NetCDF4 to BUFR format for EMC. Additional requirements for this product will be forthcoming. 87 Requirement 13.0 Requirement 13.0: The Reformatting Toolkit shall tailor NDE-generated Green Vegetation Fraction products from NetCDF4 to GRIB2 format for EMC. Additional requirements for this product will be forthcoming. 88 Interface Requirements Requirement 14.0: The Reformatting toolkit shall comply with OSPO coding standards identified in the OSPO security checklist. 89

Requirements Summary The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products. All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD. As development continues, the detailed product requirements shall continue to be refined. 90 Review Outline Introduction CDR review report/actions Requirements Quality Assurance

Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 91 Section 4 Quality Assurance Presented by Thomas King Riverside 92 Quality Assurance Background STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations. The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards.

This development group has experience with IASI and NUCAPS. They are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully. The Reformatting Toolkit algorithm development is following the same process but tailored to the scale of the project. 93 Quality Assurance Phase 1 SDR Implemented Tailored STAR EPL Reviews The Reformatting Toolkit Preliminary Design Review (April 2009) The Reformatting Toolkit Critical Design Review (September 2009)

To finalize requirements and to verify that the chosen design is able to meet those requirements. An Reformatting Toolkit Test Readiness Review/Code Test Review (Oct 2011) Present the initial draft of the requirements and discuss a proposed design. An Reformatting Toolkit Requirements Allocation Document (RAD) has been made available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project. Present the unit test plans and test results to demonstrate that the toolkit is ready to be run in the Test Environment. The Reformatting Toolkit Algorithm Readiness Review (Nov 2011) Demonstrate that the code has been system tested at STAR, software has met coding standards, and that all documentation is complete and ready for the official Phase 1 SDR DAP delivery. 94 Configuration Management (CM) CM Tool (IBM Rational ClearCase, Version 7.0 )

Has been purchased and implemented in the Collaborative Environment. CM personnel have been identified. CM training: Administrator training completed. Developers will be trained by the CM administrator. We are using a modified version of the CM plan developed for GOES-R framework being developed at STAR. The Reformatting Toolkit software has been checked into 95 ClearCase. Coding Standards We follow the SPSRB-approved coding standards developed by the Common Standards Working Group (composed of STAR, OSPO, NDE personnel). Coding standards guidelines and quick references are available. Provide a common list of abbreviations. Adhere to the standards throughout the development life cycle. Have checklists available for developers to keep track of the delivery status of the code. 96 Quality Assurance Software

All code development is being conducted on a platform that is nearly identical to the NDE integration and production target platforms using the same compilers and operating system. Only the latest and official releases of the NCEP BUFRLIB, GRIB2, and NetCDF4/HDF5 libraries are used in the software and these match to what is available on NDEs SADIE platform. STAR code checking tools are used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards. Unit tests are conducted on each software unit to demonstrate that products can be correctly produced. Test methods and results are documented and presented to stakeholders. 97 Quality Assurance Software Two prototype versions of the Toolkit have be tested in the NDE hardware. The final DAP will be delivered at the end of November 2011.

Customers have access to test BUFR/GRIB2 data products to verify required content is present and to test ingest. From STAR simulated CrIS/ATMS datasets From JPSS P72 datasets Software Code Reviews will be conducted on future phases of this project. Comply with SPSRB coding standards Comply with OSPO security checklist 98 Quality Assurance Products Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR and GRIB2 tables and that, whenever possible, they are WMO approved (as opposed to local). Chosen descriptors must work with NCEP BUFRLIB. Reformatting Toolkit developers will work with EMC, heritage product developers, and JPSS operational algorithm teams to ensure consistency with heritage products with respect to format and content.

Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB2) available to users before the operational products are made available. This will allow for preliminary product content validation. CrIS and ATMS are currently being simulated at STAR. For the other products such as VIIRS, we use NDE P72 data. 99 Current Status of BUFR Table Development File Table Status Test Data Available CrIS SDR BUFR Table Finished and approved by WMO Yes (July 2009) ATMS SDR BUFR Table

Finished and approved by WMO Yes (October 2009) VIIRS SDR BUFR Table Finished and approved by WMO Yes (June 2010) OMPS EDR BUFR Table OMPS Nadir Profile table has been finished and is going through the WMO approval process. No OMPS Total Column table is being developed. AOT EDR BUFR Table Finished and going through the WMO approval process. No

SST EDR BUFR Table Finished and going through the WMO approval process. No 100 Phase 1 Sample Product Availability First CrIS and ATMS BUFR sample data were made available to NCEP - 2/20/2009 CrIS ~300-channel and ATMS BUFR made continuously available on STAR server 10/9/2009 First VIIRS sample BUFR files were made available to NCEP - 1/8/2011 First Prototype N4RT DAP - 5/25/2011

CrIS 1305-channel were made available - 5/31/2011 VIIRS BUFR updated to only 4 Moderate and 1 Imagery band - 7/28/11 CrIS BUFR updated to final 399-channel set - 8/3/11 Second Prototype N4RT DAP (in support of NCO testing) - 9/2/2011 101 Quality Assurance Archive and Maintenance Archive Plan Reformatting toolkit will be integrated into NDE system. Our understanding is that NDE will archive the N4RT DAP with the other NDE DAPs. Product archive requirements are already addressed within product development projects. JPSS program office works with CLASS to archive xDRs NOAA Unique Product (NUP) projects work with CLASS as required There are no plans to archive N4RT output products since these can be

regenerated from the SDR, TDR, and EDR. Long Term Maintenance Plan The reformatting toolkit will be maintained by the OSPO staff STAR system developers will be available 102 Quality Assurance Documentation The following documents will be created and included as part of the official DAP delivery SMM (System Maintenance Manual) EUM (External Users Manual) No ATBD TRD (Test Readiness Document) These additional STAR EPL documents will not be delivered, but will be available to reviewers on the project website:

RAD (Requirements Allocation Document) CDD (Critical Design Document) PDD (Preliminary Design Document) Toolkit RID (Review Item Disposition) No SA (Submission Agreement) with archive 103 Quality Assurance Summary Quality assurance plan will consist of: Project reviews at which stakeholders are encouraged to participate. Ongoing interaction with customers, heritage product developers, operations, NDE, and the SPSRB. Adhering to STAR/NDE software standards and use of standard libraries only. Software unit tests shall be presented in the TRR/CTR. Documentation of the Reformatting Toolkit code operation, production rules, and software tests will be in the DAP. Documentation of requirements will be in the Reformatting Toolkit RAD. Early release of BUFR and GRIB2 products, tables, and additional resources will allow for customer validation of products. 104 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software Architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 105 Section 5 Software Architecture Presented by Thomas King Riverside

106 Reformatting Toolkit Architecture Hardware Environment Data Files Development and Unit testing Test Product Distribution System Testing and Integration Production Input Files

Static/Ancillary Files Output Files Run Files Software Description External Interfaces System Level Unit Level 107 Reformatting Toolkit Development Hardware Reformatting Toolkit Development Hardware - Unit tests were conducted on the orbit002b development machine at STAR. This machine is maintained by STAR IT. IBM P6 (AIX 6.1) 9.6 TB disk space 14 CPU (4208 MHz) 46366 MB Memory IBM XL Version 10.1 C/C++ IBM XL Version 12.1 Fortran 77/90

Configuration Management of Reformatting Toolkit code is being conducted in the STAR Collaborative Environment on 108 STAR Linux hardware running IBM Clear Case. Reformatting Toolkit Test Product Distribution STAR Data Server - Reformatting Toolkit test products are available on a distribution server at STAR (ftp2.orbit.nesdis.noaa.gov). This server hosts: Linux 3.2 TB disk space

Access via anonymous ftp Products available on a near real time basis CrIS SDR BUFR From simulation continuously available 399 and 1305 channels at full spatial resolution ATMS SDR/TDR BUFR From simulation continuously available at full spatial resolution VIIRS SDR BUFR 1 day from the P72 data set for 4 M-band and 1 I-band channels The BUFR tables of OMPS Nadir Profile, VIIRS EDR SST, and VIIRS EDR AOT are currently going through the WMO approval process. When sample data are ready for these products, they will also be made available through the ftp2 server. 109 Reformatting Toolkit Integration and Production Hardware NDE SADIE The Reformatting Toolkit system testing and integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract.

IBM P6 (AIX 6.1) 52 TB disk space 8 CPUs (4204 MHz) 63232 MB Memory IBM XL Version 11.1 C/C++ IBM XL Version 13.1 Fortran 77/90 110 Reformatting Toolkit Integration and Production Hardware NDE Test/Production Hardware - After successful system tests, NDE will check the code into their CM and then promote it to the Test/Production machine. This machine will be located at NSOF. As per email with Peter MacHarrie these will be the specs at launch: IBM P6 (AIX 6.1) 96 AIX CPUs 48 Intel CPUs

No compilers installed 111 Reformatting Toolkit Data The tables on the following slides show all the Reformatting Toolkit input and output. The tables contain all files for Phase 1 and Phase 2 tailoring. Phase 1 SDR related files are in light blue shaded cells. Phase 1 EDR and Phase 2 EDR related files are in gray shaded cells. Phase 3 information is not yet shown as the requirements are still being developed for these products. Ancillary/Static files such as the BUFR tables are listed. All Phase 1 SDR output will be in BUFR. 112 Reformatting Toolkit Input Data Files File Format Source Description

Purpose CrIS thinned SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 399 channel radiances for all FOVs. CrIS BUFR CrIS all-channel SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 1305 channel radiances for all FOVs. CrIS BUFR ATMS TDR NetCDF4

JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5. ATMS BUFR ATMS SDR NetCDF4 JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5. ATMS BUFR ATMS SDR/TDR Geo NetCDF4 JPSS JPSS ATMS Geolocation file that is associated with the ATMS SDR and TDR. This is the same file. ATMS BUFR

VIIRS M-Band 12 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 12. VIIRS BUFR VIIRS M-Band 13 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 13. VIIRS BUFR VIIRS M-Band 15 SDR NetCDF4 JPSS

JPSS VIIRS Moderate resolution band SDR for channel 15. VIIRS BUFR VIIRS M-Band 16 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 16. VIIRS BUFR VIIRS I-Band 5 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 5. VIIRS BUFR

113 Reformatting Toolkit Input Data Files File Format Source Description Purpose VIIRS M-Band SDR Geo NetCDF4 JPSS VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR VIIRS I-Band SDR Geo NetCDF4

JPSS VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR OMPS Nadir IP NetCDF4 JPSS JPSS OMPS Nadir Profile ozone Intermediate Product file tailored into NetCDF4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD. OMPS Nadir Profile BUFR OMPS Total Column EDR NetCDF4 JPSS JPSS OMPS Total Column ozone EDR file tailored

into NetCDF4. OMPS TC BUFR OMPS Total Column SDR Geo NetCDF4 JPSS JPSS OMPS Total Column ozone SDR Geo file tailored into NetCDF4. This same Geo file is the used with the OMPS Total Column EDR. OMPS TC BUFR 114 Reformatting Toolkit Input Data Files File Format Source

Description Purpose Aerosol Optical Thickness EDR NetCDF4 JPSS JPSS AOT EDR file tailored into NetCDF4. AOT BUFR Aerosol Optical Thickness EDR Geo NetCDF4 JPSS JPSS AOT EDR Geo file tailored into NetCDF4. AOT BUFR Sea Surface Temperature EDR NetCDF4

JPSS JPSS SST EDR file tailored into NetCDF4. SST BUFR Sea Surface Temperature EDR Geo NetCDF4 JPSS JPSS Moderate Resolution Terrain-Corrected Geolocation file tailored into NetCDF4. This is to be used with the JPSS SST EDR. SST BUFR Polar Winds EDR NetCDF4 Polar Winds Product Processing The Polar winds EDR produced by Jeff Key and Jamie Daniels product system running within NDE

BUFR GVF EDR NetCDF4 GVF Product Processing The Green Vegetation Fraction EDR produced by Ivan Csiszars product system running within NDE. GRIB2 115 Reformatting Toolkit Ancillary/Static Data Files File Format Source Description

Purpose CrIS SDR BUFR Table ASCII N4RT CrIS radiances (399 and 1305 channels, all FOVs) BUFR Template ATMS SDR BUFR Table ASCII N4RT ATMS radiances full resolution BUFR Template VIIRS SDR BUFR Table ASCII N4RT

VIIRS radiances full resolution BUFR Template OMPS EDR BUFR Table ASCII N4RT OMPS Nadir Profile and Total Column Ozone BUFR Template AOT EDR BUFR Table ASCII N4RT Aerosol Optical Thickness and Particle Size BUFR Template SST EDR BUFR Table ASCII

N4RT Sea Surface Temperatures, Cloud Mask BUFR Template PW BUFR Table ASCII N4RT Polar Wind products EDR BUFR Template GVF GRIB2 Table ASCII N4RT Green Vegetation Fraction EDR GRIB2 Template 116

Reformatting Toolkit Output Data Files File Format Description Users CrIS SDR BUFR CrIS thinned radiances (399 channels, all FOVs) JCSDA, NCEP CrIS SDR BUFR CrIS thinned radiances (1305 channels, all FOVs) EUMETSAT, UKMet, ECMWF ATMS SDR

BUFR ATMS radiances full resolution JCSDA, NCEP VIIRS M-band SDR BUFR VIIRS radiances moderate band JCSDA, NCEP VIIRS I-band SDR BUFR VIIRS radiances image band JCSDA, NCEP OMPS Nadir EDR BUFR OMPS Nadir Profile Ozone

JCSDA, NCEP OMPS TC EDR BUFR OMPS Total Column Ozone JCSDA, NCEP AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP PW EDR

BUFR Polar Winds JCSDA, NCEP GVF EDR GRIB2 Green Vegetation Fraction JCSDA, NCEP 117 Reformatting Toolkit Resource and Log Files File Format Source Description

Purpose PCF ASCII NUCAPS The PCF file containing all the required input parameters for the N4RT driver script. Driver input N4RT Run Log ASCII N4RT The N4RT driver script log file containing all the high-level (e.g. file management) standard error and standard output from a given run. Diagnostic Executable Log ASCII

N4RT The executables log file for a given run. The contains the low-level (e.g. file IO, systemlevel) error messages. Diagnostic N4RT Main Resource ASCII N4RT The internal control file required by the N4RT executable main program. Executable input PSF ASCII N4RT The Process Status File containing only the successfully generated output files.

Driver output list 118 Reformatting Toolkit Software: External Interfaces The Reformatting Toolkit system will be run via the execution of a single driver script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager. Execution of the script will be event-driven (i.e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE). An invocation of Reformatting Toolkit by the NDE DHS will be used to perform a single file-to-file (e.g. granule-to-granule) translation (as opposed to reformatting many files as once).

The driver script is designed to run in a single working directory. All Reformatting Toolkit output will be produced in this same working directory. The driver script will require an input PCF file containing parameters needed for file conversions. 119 Reformatting Toolkit Software: External Interfaces The driver script will: Parse the PCF content Run h5augjpss if necessary Generate the intermediate resource file for the conversion executable Run the compiled conversion code Direct program standard output and error to log files Generate the return code for the DHS

Generate a PSF If there are errors for a given run, NDE will direct the contents of the run in a forensics repository. NDE will transfer the output files to the NDE distribution server. 120 Reformatting Toolkit External Interfaces to NDE Reformatting Toolkit External Interfaces NDE DHS Boundary Systems Configuration s Product Generation Specifications Process Req.

NDE Product Generation Manager Reformatting Toolkit Driver Script Invocation Return Code Rule Sets Output Files & PSF Working Directory Output DAP Specifications Forensics Repository PSF (N4RT output) Working Directory

PCF (N4RT input) BUFR & GRIB2 Output Files Input Files (NetCDF4) Data Areas Configurations Info N4RT System NDE Production Manager Input Files & PCF Input Files (NetCDF4) SAN NDE DDS 121 Reformatting Toolkit Software: System Level The Reformatting Toolkit driver script is a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code.

There are no hard coded paths in the script. All needed information regarding locations of files will come through the PCF. All Perl library function calls and system calls have their return values checked and errors trapped so the exits are graceful and informative. All standard out and standard error is directed to a single log file that can be read by NDE for obtaining any error or warning messages. The driver script translates the low-level program errors into a high-level numerical value expected by the PGM (0 = success, 1=failure). The driver will run the h5augjpss conversion utility if the input data type requires conversion. The driver script generates an internal control file for the main Reformatting Toolkit program. 122 Reformatting Toolkit Software: System Level The PCF content: Job coverage start date and time string (yyyymmddhhmmsss) Job coverage end data and time string (yyyymmddhhmmsss)

Input NetCDF4 file names Direction of conversion mnemonic (eg. NC2BUFR) Input and output product types mnemonic (e.g. NUCAPS) Input BUFR or GRIB2 table needed for conversion Path to the executables (main_npr and h5augjpss) The PSF content (test mode): Output BUFR or GRIB2 file name 123 Reformatting Toolkit Software: System Level The h5augjpss conversion utility Command line tool run within the driver script upstream of the N4RT conversion program. The JPSS HDF5 files are not compliant with the netCDF4 data model so they cannot be read by the netCDF4 library API functions. h5augjpss is a tool that modifies a JPSS HDF5 file by adding associated data or

metadata or by hiding HDF5 elements in order to make the file accessible to netCDF based applications and tools. This is required for the converting ATMS and VIIRS HDF5 files, but not NUCAPS because those files are already processed into netCDF4. The HDF Group was funded by JPSS to develop the tool. Yi Song was one of the early testers of this tool working with the HDF Group to increase is functionality. The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation. 124 Reformatting Toolkit System Level Data Flow Reformatting Toolkit System Level Data Flow PCF Execution from PGM Return Value to PGM N4RT Driver Script h5augjpss NC Template

BUFR/GRIB2 NetCDF4 Resource BF or GB table N4RT Main Converter NetCDF4 Nominal Mode Test Mode Working directory N4RT log BUFR file GRIB2 file Working directory

Working directory PSF Working directory 125 Reformatting Toolkit Software: Unit Level The Reformatting Toolkit consists of a single main Fortran 90 program. It contains all the predefined data structures for the various BUFR data types. APIs for BUFR are only Fortran 77. Much of the code being reused is already in Fortran 90. Many users tend to request readers in Fortran 90. In the Reformatting Toolkit main program, all data structures are declared and a series of cases statements directs the program flow based on the type of input files and the direction of conversion.

The Reformatting Toolkit subroutines at the next level down manage: Allocation, initialization, deallocation of data structures. They are designed for overseeing specific product definitions and conversions. 126 Reformatting Toolkit Software: Unit Level The lowest-level Reformatting Toolkit subroutines: Perform the actual reading and writing of the specific file types. Directly call the library APIs for BUFR, GRIB2, and NetCDF4. General Reformatting Toolkit compiled code characteristics: The status of all functions are checked to allow for graceful and informative exits.

No paths are hard coded in the compiled code. There are no system calls from within the compiled code. All code is compiled as 64 bit to utilize the IBM architecture. All code is mostly statically linked (i.e. only system libraries like libc.a and the 64-bit UNIX kernel will be dynamically linked). The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation. 127 Reformatting Toolkit Unit Level Data Flow Reformatting Toolkit Unit-Level Data Flow N4RT resource N4RT log file N4RT Main Prod N - NC2BF Prod N - BF2NC Prod N - GB2NC

Allocate Allocate Allocate Allocate Initialize Initialize Initialize Initialize Prod N - Read NC Prod N - Read NC Prod N - Read BF Prod N - Read GB Prod N - Write BF

Prod N - Write GB Prod N Write NC Prod N - Write NC Deallocate Nominal Mode Test Mode Prod N - NC2GB Deallocate BUFR Deallocate GRIB2 NetCDF4 Deallocate

NetCDF4 128 Reformatting Toolkit Software: Libraries NCEP BUFR Library (available on the NCEP website) NCEP GRIB2 Encoder/Decoder Library (available on the NCEP website) C and Fortran 90 code Compiled as 64 bit Source code supplied within the N4RT DAP NetCDF4 Library version 4.1.3 (available from Unidata website)

Fortran 77 and a little C code Compiled as 64 bit Source code supplied within the N4RT DAP C code Compiled as 64 bit Requires HDF5 1.8.6 and zlib-1.2.5 (available on HDF Group and zlib websites) Libraries supplied by NDE h5augjpss 1.0.0 HDF5 conversion tool (available from the HDF Group) C code Supported (Funded) by the JPSS Program Office Compiled as 64-bit Source code supplied within the N4RT DAP 129

Toolkit Error Handling Error checking at all levels. Checking the status of all return values from executable program language statements (READ, WRITE, ALLOCATE, DEALLOCATE), script-level functions (Perl), and system calls (from ksh). Range checking of values and checks on the sizes of passed arrays are made. All errors or noteworthy conditions are trapped. Script-level log file (NPR.pl.log) will contain the string Error or Warning. These strings are identified in the N4RT System Maintenance Manual as strings that NDE can search for production monitoring. If found, the program-level log files can be checked manually for greater detail on the nature of the error.

Script-level messaging is designed to comply with section 5.1 Log File Specification of the NDE document Standards for Algorithm Delivery and Integration Using Delivered Algorithm Packages V1.3. Program-level log files are the output logs from the executable code (main_npr.log). This will contain messages labeled as either FATAL, WARNING, or NOTICE are directed to log files. Script-level and program-level messages are human-readable explaining the error, location (in program flow or on which file), consequence, and possible cause. Logic surrounding the error trapping takes corrective action or exits gracefully. The return code of the driver script will indicate whether an error has occurred in the 130 script or down in the executable code. Software Architecture Summary Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR. All input and output files have been identified. The Toolkit software architecture and program flow has been described.

The code will use only the official releases or libraries and code will be compatible with those libraries supplied by NDE on SADIE. Error handling will be present and messaging will be available to NDE production monitoring. 131 Review Outline Introduction CDR review report/actions Requirements Quality Assurance Software architecture

Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 132 Section 6 Unit Tests Presented by Yi Song Riverside 133 NC2BF Test Environment The development and tests are performed on orbit002b.orbit2.nesdis.noaa.gov IBM P6. This machine is located at STAR and is maintained by STAR IT. 14 CPUs 4.208 GHz 46.336 GB memory 9.6 TB disk space AIX 6.1

IBM XL Fortran for AIX, V12.1 IBM XL C/C++ for AIX, V10.1 Perl 5.8.8 The NetCDF, HDF5 and NCEP BUFR libraries are the latest versions: hdf5-1.8.7 netcdf-4.1.3 xlf version 11.1 or greater (Fortran77/90/95) NCEP BUFRLIB Version 10.0.0 134 NC2BF Test Data Sets Except CrIS, all the data sets come from the NDE P72 data. ATMS data set ATMS SDR Geolocation ATMS Science SDR ATMS TDR VIIRS Moderate Bands data set VIIRS Moderate Bands SDR Geolocation VIIRS Moderate Resolution SDR Band 12, Band 13, Band 15 and Band 16 VIIRS Image Bands data set VIIRS Image Bands SDR Geolocation VIIRS Imagery band 5 SDR 135 NC2BF Test Data Sets

The CrIS dataset comes from NESDIS/STAR CrIS simulation observation system that emulates the instrumental and orbital characteristics of the CrIS instruments on the NPP platform to produce: 2.9 million CrIS spectra/day The output of the simulation system is in the CrIS HDF5 JPSS format. One program in NUCAPS converts CrIS SDR HDF5 file into two NetCDF4 files All CrIS channels One subset with 399 channels. Both Geolocation and Science data are included in one CrIS SDR NetCDF4 file 136 NC2BF System Needs BUFR table of each product. Product type: NUCAP, or ATMS, or VIIRS Conversion type: NC2BUFR, or BUFR2NC. Channel type: All channels, or subset of channels. Input data: Geolocation data and Science data files. 137 System Description Function and Purpose It reads a PCF, writes a resource file and a PSF, and returns error handling information to the calling NDE Data Handling System.

It converts HDF5 files into NetCDF4 files with utility h5augjpss. It reads the resource file into the NC2BF processing chain. Decides the converting action: NC2BUFR, or BUFR2NC. Applies the BUFR table to the products, and converts them between NetCDF4 and BUFR format. 138 System Test Items Software Unit Purpose File Input File Output NPRpl This is a driver script that: (1) Reads in the PCF and parses its contents. (2) Performs local file Management (converts input data file from HDF5 format to NetCDF4 file with utility

h5augjpss), generates internal resources file, handles the system calls, and drives main program main_npr. (3) Writes out log files and the PSF file, and returns error status to the NDE DHS. PCF PSF Driver script log file Executable program log file Resource input file main_npr It converts products between NetCDF4 and BUFR formats. For NC2BUFR, it reads BUFR table and NetCDF4 files listed in the resource file, and writes out BUFR file. For BUFR2NC, it reads BUFR table and BUFR file, and writes out NetCDF4 files. For NC2BUFR: BUFR

table, Geo NetCDF4 file, SDR NetCDF4 For BUFR2NC: BUFR table and BUFR file. For NC2BUFR: BUFR file. For BUFR2NC: NetCDF4 Geo and SDR files 139 Test Data Description Simulated near real-time NPP 72 hours data (09/06/2010 09/08/2010) Random choose one granule file for ATMS, VIIRS M-band, and VIIRS I-band SDR data. According to the time mark of the chosen granule, choose the matched geolocation data. One VIIRS granule length: 85.7856 seconds. One ATMS length: 32 seconds. One CrIS granule length: 32 seconds. 140 BUFR Table Description All the BUFR tables are discussed among the user communities including NCEP, ECMWF, EUMETSAT and UK Met Office, and get WMO approved.

The BUFR tables have the required variables and descriptors. The delayed replication factor is applied for the channel data, which adds the flexibility to choose the channel subset. VIIRS M-band (16 channels), I-band (5 channels) and DNB Band (1 channel) share one BUFR table. 141 Test Configuration Each item in the test configuration has been placed in the project baseline under configuration control The Perl script can be run as it is. The Fortran 90 and C code must be compiled. The entire NC2BUFR system is compiled with a single Makefile. It can be run using gmake at AIX machine /usr/bin/gmake. After gmake has run, the user must have main_npr copied to /disk2/pub/czhang/CrISOPS/Common_Bin, or update the NPR.pl.PCF file to use the correct OPS_BIN directory. 142 Test Sequence/Results Test Step 1: Changed into the test directory on orbit002b: /disk2/pub/ysong/test_code/ATMS Test Step 2: Verified that the PCF contents (NPR.pl.PCF) were created by following the NDE Algorithm Delivery Standards (Appendix A NDE DHS

Product Generation Algorithm Interface) by dumping the contents of the file: job_coverage_start=201009072208471 job_coverage_end=201009072209185 PROD_TYPE=ATMS CONVERSION=NC2BUFR NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/ GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5 NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR/ SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5 NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-TDR/ TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5 BUFR_TABLE=/disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_Table OPS_BIN=/disk2/pub/ysong/code/main 143 Test Sequence/Results Test Step 3: NPR.pl can check the presence of NPR.pl.PCF: if( ! -e "NPR.pl.PCF" ) { print "Error in ${PROGRAM}.pl: NPR.pl.PCF does not exist. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; } Test Step 4: NPR.pl can read NPR.pl.PCF:

sysopen(PCF, "NPR.pl.PCF",O_RDONLY) or $rc = 1; if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: failed to open NPR.pl.PCF. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; } @Line_List = ; close (PCF); 144 Test Sequence/Results Test Step 5: Ran the driver script as: perl -w /disk2/pub/ysong/test_code/ NPR.pl 2>&1 Test Step 6: Verified that the PSF contents (NPR.pl.PSF) were created indeed in accordance with the NDE Algorithm Delivery Standards (Appendix A NDE DHS Product Generation Algorithm Interface) by dumping the contents of the file: orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls NPR.pl.PSF NPR.pl.PSF orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>cat NPR.pl.PSF ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221457390.bufr orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS> The file contains the output file name only when the program completes

successfully. Otherwise, it is empty. 145 Test Sequence/Results Test Step 7: Each of the 5 different BUFR files can be produced successfully: ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4. An ls lrt in the test directory can verify that the output files were generated by this run of the script. Here is ATMS output for a run at 11:49 on Sept. 22, 2011: orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS> orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls -lrt total 0 -rw-r--r-- 1 ysong orbit 597 Sep 22 11:38 NPR.pl.PCF -rw-r--r-- 1 ysong orbit 366 Sep 22 11:49 npr.filenames -rwxr-xr-x 1 ysong orbit 157600 Sep 22 11:49 TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc -rwxr-xr-x 1 ysong orbit 171152 Sep 22 11:49 SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.nc -rwxr-xr-x 1 ysong orbit 157784 Sep 22 11:49 GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc -rw-r--r-- 1 ysong orbit 1975 Sep 22 11:49 NPR.pl.log -rw-r--r-- 1 ysong orbit

70 Sep 22 11:49 NPR.pl.PSF -rw-r--r-- 1 ysong orbit 127072 Sep 22 11:49 ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr 146 Test Sequence/Results Here is CrIS 1305 Channels output for a run at 12:47 on Sept. 22, 2011: orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>ls -lrt total 0 -rw-r--r-- 1 ysong orbit 5871947 Aug 25 15:57 NUCAPS_ALL_20110825_1227240_1227538.nc -rw-r--r-- 1 ysong orbit 245 Aug 26 10:36 NPR.pl.PCF -rw-r--r-- 1 ysong orbit 178 Sep 22 12:47 npr.filenames -rw-r--r-- 1 ysong orbit 2881664 Sep 22 12:47 NUCAPSC1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr -rw-r--r-- 1 ysong orbit 727 Sep 22 12:47 NPR.pl.log -rw-r--r-- 1 ysong orbit 78 Sep 22 12:47 NPR.pl.PSF This is CrIS 399 Channels output for a run at 14:54 on Oct. 21, 2011: orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>ls -lrt

total 0 -rw-r--r-- 1 ysong orbit 1898843 Aug 25 15:57 NUCAPS_C0300_ALLFOVS_20110825_1241540_1242237.nc -rw-r--r-- 1 ysong orbit 255 Aug 26 10:38 NPR.pl.PCF -rw-r--r-- 1 ysong orbit 188 Oct 21 14:54 npr.filenames -rw-r--r-- 1 ysong orbit 27900 Oct 21 14:54 fort.30 -rw-r--r-- 1 ysong orbit 803288 Oct 21 14:54 NUCAPSC0399_v1r0_npp_s201108251241540_e201108251242237_c201110211854110.bufr -rw-r--r-- 1 ysong orbit 790 Oct 21 14:54 NPR.pl.log 147 -rw-r--r-- 1 ysong orbit 78 Oct 21 14:54 NPR.pl.PSF Test Sequence/Results Here is VIIRS M4 output for a run at 12:53 on Sept. 22, 2011: orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>ls -lrt total 0 -rw-r--r-- 1 ysong orbit 904 Aug 22 15:39 NPR.pl.PCF -rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.nc -rwxr-xr-x 1 ysong orbit

81187816 Sep 22 12:51 GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.nc -rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.nc -rwxr-xr-x 1 ysong orbit 22190144 Sep 22 12:51 SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.nc -rw-r--r-- 1 ysong orbit 537 Sep 22 12:51 npr.filenames -rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.nc -rw-r--r-- 1 ysong orbit 33499512 Sep 22 12:53 VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr -rw-r--r-- 1 ysong orbit 2661 Sep 22 12:53 NPR.pl.log -rw-r--r-- 1 ysong orbit 73 Sep 22 12:53 NPR.pl.PSF 148 Test Sequence/Results VIIRS I5 output for a run at 12:58 on Sept. 22, 2011 is: orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>ls -lrt total 0 -rw-r--r-- 1 ysong orbit

489 Aug 24 16:35 NPR.pl.PCF -rwxr-xr-x 1 ysong orbit 324489816 Sep 22 12:50 GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.nc -rw-r--r-- 1 ysong orbit 300 Sep 22 12:50 npr.filenames -rwxr-xr-x 1 ysong orbit 49230656 Sep 22 12:50 SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.nc -rw-r--r-- 1 ysong orbit 118547320 Sep 22 12:58 VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr -rw-r--r-- 1 ysong orbit 2032 Sep 22 12:58 NPR.pl.log -rw-r--r-- 1 ysong orbit 73 Sep 22 12:58 NPR.pl.PSF 149 Test Sequence/Results The existence of the NetCDF4 files: GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.nc TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc demonstrates that h5augjpss was able to convert HDF5 file to the readable NetCDF4 files. The existence of the output file: ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr

demonstrates that the script NPR.pl was able to: Read the PCF Create a resource file for the executable (npr.filenames) Run the executable main program main_npr. 150 Test Sequence/Results Test Step 8: Dump HDF5 file and its NetCDF4 file. The same values within precision show that the utility h5augjpss is able to convert HDF5 to NetCDF4. Part of Latitude data in /net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/GATMO_npp_d20100907_t2208471_e2209185_b00027_c201 10406231119993808_noaa_ops.h5 DATASET "Latitude" { DATATYPE H5T_IEEE_F32BE DATASPACE SIMPLE { ( 12, 96 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } DATA { (0,0): -73.1603, -73.2373, -73.2889, -73.3205, -73.3363, (0,5): -73.3392, -73.3319, -73.3162, -73.2936, -73.2653, -73.232, (0,11): -73.195, -73.1546, -73.1115, -73.066, -73.0186, -72.9694, (0,17): -72.9189, -72.8671, -72.8142, -72.7604, -72.7058, (0,22): -72.6505, -72.5946, -72.5381, -72.4809, -72.4234, Part of Latitude data in GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc

Latitude = -73.16026, -73.23733, -73.28891, -73.32051, -73.33625, -73.33925, -73.33195, -73.31623, -73.2936, -73.26526, -73.23201, -73.19497, -73.15462, -73.1115, -73.06602, -73.01857, -72.96943, -72.91886, -72.86705, -72.81418, -72.76039, -72.7058, -72.65051, -72.5946, -72.53812, -72.48086, -72.42341, -72.36552, -72.30724, -72.24854, -72.18947, -72.13002, -72.07019, -72.00996, -71.94936, -71.88834, 151 Test Sequence/Results Test Step 9: Dumped the Contents of the internal resource file (npr.filenames). This shows that NPR.pl was able to read and parse the PCF file contents to the resource file, and use internal readable NetCDF4 files for input to main program main_npr. ATMS NC2BUFR GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.nc TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr /disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_Table 152 Test Sequence/Results

Test Step 10: Dumped the contents of the log file (NPR.pl.log) to show that everything ran correctly. There were no errors in the script itself, any of the system calls, the main program, its subroutines and BUFR library calls. This log file contained the starting and ending times indicating everything completed in 1 seconds of clock time. The CPU for the main program is 0.4 second for one ATMS granule data. Starting at Thu Sep 22 11:49:10 EDT 2011 main_npr is now starting. Starting ATMS_netcdf_to_bufr Output_FileName=ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr Starting ATMS_write_bufr +++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++ BUFRLIB: MAXOUT - THE RECORD LENGTH OF ALL BUFR MESSAGES CREATED FROM THIS POIN T ON IS BEING CHANGED FROM 10000 TO 20000 +++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++ Closing BUFR file. Finishing ATMS_write_bufr Finishing ATMS_netcdf_to_bufr main_npr is now finished. CPU time for the whole program is: 0.400000000000000022 Ending at Thu Sep 22 11:49:11 EDT 2011 153 Test Sequence/Results Test Step 11: The BUFR tables have the required variables and descriptors. Some special requirements, such as negative radiance, are kept. Example of CrIS BUFR table.

.------------------------------------------------------------------------------. | -----------USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- | |------------------------------------------------------------------------------| | MNEMONIC | NUMBER | DESCRIPTION | |----------|--------|----------------------------------------------------------| | | | | | NC021202 | A10060 | MSG TYPE 021-202 CrIS radiance data | | | | | | YYMMDD | 301011 | Date -- year, month, day | | HHMM | 301012 | Time -- hour, minute | | LTLONH | 301021 | High accuracy latitude/longitude position | | LOCPLAT | 304030 | Location of platform (satellite) sequence |

| BCFQFSQ | 350200 | NPP CrIS band calibration & f-o-v quality flag sequence | | CRCHN | 350201 | NPP CrIS channel data | | | | | | SAID | 001007 | Satellite identifier | | OGCE | 001033 | Identification of originating/generating center | | SIID | 002019 | Satellite instruments | | SCLF | 002020 | Satellite classification | | RDTF | 002165 | Radiance type flags | | YEAR | 004001 | Year | | MNTH | 004002 | Month

| | DAYS | 004003 | Day | | HOUR | 004004 | Hour | | MINU | 004005 | Minute | | SECO | 004006 | Second | | CLATH | 005001 | Latitude (high accuracy) | | BEARAZ | 005021 | Bearing or azimuth | | SOLAZI | 005022 | Solar azimuth | | ORBN | 005040 | Orbit number | | SLNM | 005041 | Scan line number |

| CHNM | 005042 | Channel number | | FOVN | 005043 | Field of view number | | FORN | 005045 | Field of regard number | 154 Example of BUFR table (CrIS) Continued | CLONH | 006001 | Longitude (high accuracy) | | WVNM | 006029 | Wave number | | HMSL | 007002 | Height or altitude | | SAZA | 007024 | Satellite zenith angle | | SOZA | 007025 | Solar zenith angle

| | LSQL | 008012 | Land/Sea qualifier | | STKO | 008075 | Ascending/Descending orbit qualifier | | TOBD | 008076 | Type of band | | HOLS | 010001 | Height of land surface | | PDNP | 010031 | In dir. of North Pole, distance from the Earth's center | | SRAD | 014044 | Channel radiance | | TOCC | 020010 | Cloud cover (total) | | HOCT | 020014 | Height of top of cloud | | ALFR | 021166 | Land fraction | | STCH

| 025140 | Start channel | | ENCH | 025141 | End channel | | PD00 | 027031 | In direction of 0 deg E, distance from Earth's center | | PD90 | 028031 | In direction of 90 deg E, distance from Earth's center | | QMRKH | 033003 | Quality information | | NSQF | 033075 | Scan-level quality flags | | NCQF | 033076 | Calibration quality flags | | NFQF | 033077 | Field of view quality flags | | NGQI | 033078 | Geolocation quality | | |

| | |------------------------------------------------------------------------------| | MNEMONIC | SEQUENCE | |----------|-------------------------------------------------------------------| | NC021202 | SAID OGCE SIID SCLF YYMMDD HHMM | | NC021202 | 207003 SECO 207000 LOCPLAT LTLONH SAZA BEARAZ | | NC021202 | SOZA SOLAZI STKO 201133 SLNM 201000 FORN |

| NC021202 | FOVN ORBN HOLS 201129 HMSL 201000 | | NC021202 | 202127 201125 ALFR 201000 202000 LSQL TOCC | | NC021202 | HOCT RDTF NSQF "BCFQFSQ"3 | | NC021202 | TOBD NGQI QMRKH (CRCHN) | | | |

| YYMMDD | YEAR MNTH DAYS | | HHMM | HOUR MINU | 155 Example of BUFR table (CrIS) Continued | LTLONH | CLATH CLONH | | LOCPLAT | PD00 PD90 PDNP | | BCFQFSQ | TOBD WVNM WVNM STCH ENCH NCQF

NFQF | | CRCHN | 201133 CHNM 201000 SRAD | | | | |------------------------------------------------------------------------------| | MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------| |----------|------|-------------|-----|--------------------------|-------------| | | | | | |-------------| | SAID | 0 | 0 | 10 | Code table |-------------| | OGCE

| 0 | 0 | 8 | Code table |-------------| | SIID | 0 | 0 | 11 | Code table |-------------| | SCLF | 0 | 0 | 9 | Code table |-------------| | RDTF | 0 | 0 | 15 | Flag table |-------------| | YEAR | 0 | 0 | 12 | Year |-------------| | MNTH |

0 | 0 | 4 | Month |-------------| | DAYS | 0 | 0 | 6 | Day |-------------| | HOUR | 0 | 0 | 5 | Hour |-------------| | MINU | 0 | 0 | 6 | Minute |-------------| | SECO | 0 | 0 | 6 | Second |-------------|

| CLATH | 5 | -9000000 | 25 | Degree |-------------| | BEARAZ | 2 | 0 | 16 | Degree true |-------------| | SOLAZI | 2 | 0 | 16 | Degree true |-------------| | ORBN | 0 | 0 | 24 | Numeric |-------------| | SLNM | 0 | 0 | 8 | Numeric |-------------| | CHNM |

0 | 0 | 6 | Numeric |-------------| | FOVN | 0 | 0 | 8 | Numeric |-------------| | FORN | 0 | 0 | 8 | Numeric |-------------| | CLONH | 5 | -18000000 | 26 | Degree |-------------| | WVNM | 1 | 0 | 22 | m**-1 |-------------| | HMSL |

-1 | -40 | 16 | m |-------------| | SAZA | 2 | -9000 | 15 | Degree |-------------| | SOZA | 2 | -9000 | 15 | Degree |-------------| | LSQL | 0 | 0 | 2 | Code table |-------------| | STKO | 0 | 0 | 2 | Code table |-------------| | TOBD | 0 |

0 | 6 | Code table |-------------| | HOLS | 0 | -400 | 15 | m |-------------| | PDNP | 2 | -1073741824 | 31 | m |-------------| | SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------| 156 Example of BUFR table (CrIS) Continued | TOCC | 0 | 0 | 7 | % |-------------|

| HOCT | -1 | -40 | 11 | m |-------------| | ALFR | 3 | 0 | 10 | Numeric |-------------| | STCH | 0 | 0 | 14 | Numeric |-------------| | ENCH | 0 | 0 | 14 | Numeric |-------------| | PD00 | 2 | -1073741824 | 31 | m |-------------| | PD90 | 2 | -1073741824 | 31 | m |-------------|

| QMRKH | 0 | 0 | 3 | Code table |-------------| | NSQF | 0 | 0 | 13 | Flag table |-------------| | NCQF | 0 | 0 | 9 | Flag table |-------------| | NFQF | 0 | 0 | 19 | Flag table |-------------| | NGQI | 0 | 0 | 4 | Code table |-------------|

| | | | | |-------------| `------------------------------------------------------------------------------' The reference value of -100000 of Channel radiance (SRAD 0-14-044) allows to keep negative radiance in the CrIS BUFR file. | SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------| 157 Test Sequence/Results Test Step 12: In BUFR table, the delayed replication factor is applied for the channel data. It adds the flexibility to apply the BUFR table to the data

subset, such as delayed replication for channel data in CrIS BUFR table Entry number Meaning 1 04 000 Delayed replication of 4 descriptors Extended delayed descriptor replication factor 0 31 002 2 01 133 0 05 042 2 01 000 Increase bit width Channel number Cancel increase bit width 0 14 044 Channel radiance 158

Test Sequence/Results Test Step 13: The delayed replication is implemented in the program code. Part of code in subroutine nucape_write_bufr.f90 for CrIS. ! ! Open a BUFR message ! CALL OPENMB(LUBFR,SUBSET,IDATE) CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)') CALL UFBSEQ(LUBFR,ARR,ARRDIM,1,IRET,SUBSET) CALL WRITCP(LUBFR) The argument NUM_CRIS in the NCEP BUFR library call CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)') tells the system there are how many replication of CrIS channel data. 159 Test Sequence/Results Test Step 14: The output BUFR file name follow the NDE file name convention defined in the NDE document called "Algorithm Delivery Standards, Integration, and Test_V1.3.docx. ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr Data Product Short Name is ATMS The version of this product, which is 1.0 (v1r0). The platform from which this file was created is the NPP satellite

(npp). Start date and time of the data is September 07, 2010 at 22:08:47.1 UTC (s201009072208471). End date and time of the data is September 07, 2010 at 22:09:18.5 UTC (e201009072209185). This output product file was created on September 22, 2011 at 15:49:10.0 UTC (c201109221549100). The file is a BUFR file (.bufr extension). 160 Test Sequence/Results Test Step 15: The created BUFR file can be read back and are comparable with the original NetCDF4 file. Part of channel radiances that are converted back from its BUFR file ( CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.4617, 52.7766, 69.9809, 62.8292, 54.4361, 47.8841, 44.4845, 45.2233, 47.1264, 41.4511, 43.0444, 45.3617, 43.0024, 42.4175, 39.3311, 41.1938, 40.666, 39.6869, 39.0525, 40.0077, 40.4335, 39.883, 41.3726, 42.5681, Part of channel radiances in the original NetCDF4 file. CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.46172, 52.7766, 69.98087, 62.82917, 54.43607, 47.88414, 44.48447, 45.22329, 47.1264, 41.45107, 43.04439, 45.3617, 43.00241, 42.41754, 39.3311, 41.19378, 40.666, 39.68685, 39.05251, 40.00766, 40.43351, The comparable values in the two files demonstrate that the system can convert from NetCDF4 to BUFR and then back to NetCDF4

161 Test Sequence/Results Test Step 16: The created BUFR file can also be read with the ECMWF BUFR library. This indicates that the products are readable among global user communities. Part of values read from the BUFR file with ECMWF BUFR library. 57 EXTENDED DELAYED DESCRIPTOR REPL 0.39900000000000E+003 NUMERIC 58 CHANNEL NUMBER 0.27000000000000E+002 NUMERIC 59 CHANNEL RADIANCE 0.45461700000000E-001 (W/M**2)*(1/SR)*CM 60 CHANNEL NUMBER 0.28000000000000E+002 NUMERIC 61 CHANNEL RADIANCE 0.52776600000000E-001 (W/M**2)*(1/SR)*CM 62 CHANNEL NUMBER 0.31000000000000E+002 NUMERIC 63 CHANNEL RADIANCE 0.69980900000000E-001 (W/M**2)*(1/SR)*CM 64 CHANNEL NUMBER 0.32000000000000E+002 NUMERIC 65 CHANNEL RADIANCE 0.62829200000000E-001 (W/M**2)*(1/SR)*CM 66 CHANNEL NUMBER 0.33000000000000E+002 NUMERIC 67 CHANNEL RADIANCE 0.54436100000000E-001 (W/M**2)*(1/SR)*CM

68 CHANNEL NUMBER 0.37000000000000E+002 NUMERIC 69 CHANNEL RADIANCE 0.47884100000000E-001 (W/M**2)*(1/SR)*CM 70 CHANNEL NUMBER 0.49000000000000E+002 NUMERIC 71 CHANNEL RADIANCE 0.44484500000000E-001 (W/M**2)*(1/SR)*CM 72 CHANNEL NUMBER 0.51000000000000E+002 NUMERIC 73 CHANNEL RADIANCE 0.45223300000000E-001 (W/M**2)*(1/SR)*CM 74 CHANNEL NUMBER 0.53000000000000E+002 NUMERIC 75 CHANNEL RADIANCE 0.47126400000000E-001 (W/M**2)*(1/SR)*CM 76 CHANNEL NUMBER 0.59000000000000E+002 NUMERIC 77 CHANNEL RADIANCE 0.41451100000000E-001 (W/M**2)*(1/SR)*CM 78 CHANNEL NUMBER 0.61000000000000E+002 NUMERIC 79 CHANNEL RADIANCE 0.43044400000000E-001 (W/M**2)*(1/SR)*CM 162 Test Sequence/Results

Test Step 17: Error checks execute in script. In the script file (NPR.pl), all system calls are check and exit gracefully with an error message when a problem occurs. Part of script program NPR.pl. $rc=system("cp $input_file $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: the copy data returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; } $rc=system("$h5augjpss -o1 -o4 $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: h5augjpss returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; } 163 Test Sequence/Results Test Step 18: Error checks process in the main NC2BUFR system. In the main program (main_npr.f90) and its subroutines, all system calls are check and exit gracefully with an error message when a problem occurs. Part of program main_npr.f90

PRINT*,'main_npr is now starting.' call cpu_time(t1) Unit_Number_Resource_File = Get_Lun(); OPEN (UNIT=Unit_Number_Resource_File, File = "npr.filenames", & Status="OLD", ACTION = "READ", IOSTAT=Open_Status) IF(Open_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to open npr.filenames', & FATAL) ENDIF ! ! Read the input resource file. ! READ(Unit_Number_Resource_File,FMT='(a)',IOSTAT=Read_Status)Product_Type IF(Read_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to read npr.filenames', & FATAL) ENDIF 164 Test Sequence/Results Test Step 19: Run times are very short for converting the HDF5 ATMS and VIIRS files using the h5augjpss tool. It is less than 1 seconds for ATMS, 4 seconds for VIIRS 4 M-bands, and 7 seconds for VIIRS 1 Iband. Part of NPR.pl.log for ATMS start_h5augjpss=1316795761, end_h5augjpss=1316795761, h5augjpss_time=0

start_h5augjpss=1316803047, end_h5augjpss=1316803051, h5augjpss_time=4 Part of NPR.pl.log for VIIRS 4 M-bands start_h5augjpss=1316803089, end_h5augjpss=1316803096, h5augjpss_time=7 Part of NPR.pl.log for VIIRS 5 I-bands 165 Test Sequence/Results Test Step 20: CPU times for generating BUFR file. It is short for CrIS and ATMS, but it takes long to convert VIIRS SDR files to BUFR files. It took 0.4 seconds CPU time to process one granule ATMS. CPU time for the whole program is: 0.400000000000000022 CPU time for the whole program is: 6.23999999999999932 It took 6.24 seconds CPU time to process one granule CrIS all channels. CPU time for the whole program is: 3.61999999999999966 It took 3.62 seconds CPU time to process one granule CrIS 399 channels. CPU time for the whole program is: 146.960000000000008 CPU It took 146.96 seconds CPUis:time to process one granule VIIRS 4 M-bands. time for the whole

program 437.560000000000002 166 Test Sequence/Results Test Step 21: BUFR file sizes of the created BUFR files. Size of one granule all channel CrIS BUFR file is 2.882 mb. The size of whole day BUFR files is: 2.882*(86400/32)=7781.4 mb=7.781 gb Size of one granule 399 channels CrIS BUFR file is 0.803 mb. The size of whole day CrIS 399 channels BUFR files is: 0.803*(86400/32)=2168.1 mb = 2.168gb. Size of one granule ATMS BUFR file is 0.127 mb. The size of whole day BUFR files is: 0.127*(86400/32)=342.9 mb=0.343 gb. Size of one granule VIIRS 4 M-bands BUFR file is 33.5 mb. The size of whole day BUFR files is: 33.5*(86400/85.7856)=33739.9 mb=33.7 gb Size of one granule VIIRS 1 I-bands BUFR file is 118.5 mb. The size of whole 167 day BUFR files is: 118.5*(86400/85.7856)=119348.7 mb=119.3 gb Test Risks

Risk #1: The required input data are NetCDF4, but the SDR data are in HDF5. The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDEs solution wouldve involved bundling all the SDRs into one file and changing variable names so the converters readers wouldve needed to change. Risk Assessment: Low Impact: This unavailable converter may cause this system development delay and affect the test. Likelihood: Low Mitigation: The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution. To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available. Status: Close. 168 Test Risks Risk #2: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 Ibands. It is too slow. Risk Assessment: Low Impact: The long run times may delay the products. Likelihood: Low Mitigation: After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs).

VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded. Status: Close. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change. 169 Summary & Conclusions NC2BUFR system has been test step by step, and it has passes all the tests. The main program and script check the status of all system calls and exit gracefully with an error message when a problem occurs. This system can convert from NetCDF4 to BUFR and then back to NetCDF4. The BUFR tables have the required variables and descriptors. ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4 BUFR files can be produced successfully. Output BUFR files follow the NDE file name convention. 170 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 171 Section 7 Delivered Algorithm Package Presented by Thomas King Riverside 172 NDE DAP Requirements

1. Delivery Memo 2. README text file 3. Science algorithm source code, including make files, and scripts. 4. Input test data and resulting output data. 5. Static coefficient files and/or look-up table files if applicable 6. Description of the processing steps in the form of a data flow diagram consistent with SA software architecture. 7. Description of all input data files and any intermediate data files produced during SA execution. 8. Description of output data files contents, requirements (latency and quality), storage requirements, and format. 9. Description of product quality monitoring information (exception handling, exit codes and quality flags along with their meanings). 10. Detailed algorithm test plan (description, procedures, product quality requirements, and results). 11. Estimates of computer and software storage resources required for execution. Include software storage requirements for SA code, static data files, and computer resources such as wall clock time, CPU time, memory, page faults, and disk space.

12. Algorithm Theoretical Basis Document (ATBD). This may be a specific pointer (or hyperlink) to the document. 173 Toolkit DAP Requirements Delivery memo will contain: Point(s) of contact (POC) for questions specific to the algorithm (include name, telephone, and e-mail address). At a minimum, include algorithm developer and PAL. Date of Current DAP Release Purpose of the delivery (e.g. an initial release, modification) Brief description of package (e.g. DAP Version X represents the initial delivery of the, see Appendix B for a more detailed example) List of DAP contents Description of problem(s) resolved, if any, and method of resolution. Description of significant changes from previous version, if any. List of known remaining updates/issues. 174 Toolkit DAP Requirements The README text file in the DAP must contain: DAP version number. Date of current DAP release Target configuration for SA source code execution. List of all pointers (or hyperlinks) to reference documents (DAP contents 6-12 above).

Detailed description of how to compile the source code in the NDE SADIE environment. List of expected compiler warnings. Other pertinent information as judged by the algorithm developer(s) (compiler settings, etc.). Description of PCF and PSF file. Production Rule Definitions 175 N4RT DAP Contents The DAP will be delivered to SADIE and will consist of a gzipd tar file that will extract 4 main directories into whatever is the current directory of the DAP tar file. The directories are: Source All Fortran source code for N4RT (4 MB) BUFRLIB source (3.6 MB) h5augjpss source (96 MB) OPS All scripts (0.012 MB) BUFR tables (0.032 MB) Compiled (executable) (22 MB) DATA ATMS (1 MB) CrIS_C0399 (2.9 MB) CrIS_C1305 (8.6 MB) VIIRS_I5 (512 MB)

VIIRS_M4 (193 MB) DOCS All SPSRB and NDE documentation (17 MB) Total size of DAP will be approximately 860 MB 176 Toolkit Source Contents Code Language Number of Files Number of Lines N4RT Fortran 90 325 78551

N4RT Perl 1 424 BUFRLIB 4.0 Fortran 77/C 218 27472 h5augjpss source 1.0.0 C 7 1983 177 Toolkit OPS Contents

Content Files Description Scripts NPR.pl The driver script invoked by the NDE DHS. BUFR Tables ATMS_BUFR_Table CrIS_BUFR_Table VIIRS_BUFR_Table The ASCII BUFR tables input to the Fortran executable Compiled Code main_npr h5augjpss The executables 178

Toolkit DATA Contents (Input Files) Files Description NUCAPS_C0300_ALLFOVS_20110825_1227240_1227538.nc NUCAPS 399-channel input file NUCAPS_ALL_20110825_1227240_1227538.nc NUCAPS 1305-channel input file TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5 SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5 GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5 ATMS input HDF5 input files GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.h5 SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.h5 VIIRS I5 HDF5 input files GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.h5

SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.h5 SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.h5 SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.h5 SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.h5 VIIRS M4 HDF5 input files 179 Toolkit DATA Contents (Output Files) Files Description ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr ATMS output BUFR file C0399_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 399-channel BUFR file C1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 1305-channel BUFR file

VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr VIIRS I5 BUFR file VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr VIIRS M4 BUFR file 180 Toolkit DOCS Contents Code Document Type Description System Maintenance Manual SPSRB Describes everything in the N4RT system: all input/output/intermediate data, LUTs, and overall software architecture (processing steps). Also describes production rules, error handling, quality monitoring, computer resource requirements. External Users Manual

SPSRB Describes formats of all output products and everything required to use them. Test Readiness Document STAR EPL The slide package showing the test plan and the results of all the units tests. Also describes all input/output test data. README NDE README file for NDE DAP 181 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 182 Section 8 Risks and Actions Summary Presented by Thomas King Riverside 183 Risk and Actions The status of risks from the PDR and CDR Report were reported earlier in this review Only the remaining open risks will be summarized here.

The status of any new risks since the CDR will be reported here as well. 184 Risk Summary Risk 3: There are small variations in the types of platforms and the versions of the compilers Mitigation: Reformatting Toolkit developers will work with NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The delivery of prototype DAPs should help to resolve these issues early on in development. Risk 4: Data format translation involves some unit conversion and possible reduction of precision (significant digits) Mitigation: Document data manipulations in the SMM which will be included in the NDE Delivered Algorithm Package. This will server as a replacement for an ATBD. 185 Risk Summary Risk 6: Risk on NDVI and snow mask. Have not yet demonstrated ability to encode GRIB2 files. Mitigation: NDVI and snow mask will not be produced. However, GVF will need to be in GRIB2 format, but thats something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR

scheduled for 1/23/2012. Risk 18: ESPC may not have the BUFR expertise to maintain the toolkit. Mitigation: OSPO is getting FY12 funding to support the transition and maintenance from NDE on the OSPO side. 186 New Risks Risk 20: The required input data are NetCDF4, but the SDR data are in HDF5. The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDEs solution wouldve involved bundling all the SDRs into one file and changing variable names so the converters readers wouldve needed to change. Risk Assessment: Low Impact: This unavailable converter may cause this system development delay and affect the test. Likelihood: Low Mitigation:

The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution. To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available. Status: Closed. 187 New Risks Risk 21: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 Ibands. It is too slow. Risk Assessment: Low Impact: The long run times may delay the products. Likelihood: Low Mitigation: After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs). VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded. Status: Closed. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change. 188 New Risks Risk 22: NDE needs to modify their contract to support completion of the SPSRB documents delivered by the NDE product teams. This is needed for a timely and smooth transition to OSPO. This is a projectlevel risk because there is concern over whether this can be done

before NDE system is declared operational. Assessment: Low Impact: OSPO may not have sufficient documentation of the delivered NDE system. Likelihood: Moderate Mitigation: Geof Goodrum is modifying the NDE contract to include this documentation. Status: Open. 189 Risks and Actions Summary There are 11 risks that carry over from PDR/CDR: We recommend closing 8 of these risks: 4 risks will remain open. There are 3 new risks between CDR and TRR 2 of these can be closed. 190 Review Outline

Introduction CDR review report/actions Requirements Quality Assurance Software architecture Unit Tests Delivered Algorithm Package Risks and Actions Summary Summary and Conclusions 191 Section 9 Summary and Conclusions Presented by Walter Wolf NOAA/NESDIS/STAR 192 Review Objectives Have Been Addressed The Project Mission and Schedule has been reviewed.

The CDR Report has been reviewed. The Project Requirements have been reviewed. Plans for quality assurance have been reviewed. Software architecture, hardware, data, and interfaces have been reviewed. Test plans and results have been documented and presented. Risks and Actions have been reviewed. 193 Current Status Reformatter toolkit SPSRB-required documentation development is currently in progress. All coding is complete for the Phase 1 SDR toolkit DAP. Unless any new issues arise, this is the version that will run operationally. The prototype DAP that NDE currently has should be able to run as soon as data are available after launch. N4RT developers will be available to provide any assistance and troubleshooting with the Toolkit for issues that may arise when data begin to flow. 194

Next Steps After this review, the RID (Review Item Disposition) displaying the projects risks and actions will be updated. Any necessary edits to the Test Readiness Document will also be made. All modified project documents will be reposted to the project website and the updated links will be emailed to the reviewers. The Reformatter toolkit Algorithm Readiness Review will be held on 11/28/2011. The final review prior to delivery. It will be a working review to verify that the DAP is ready for implementation in the NDE environment and well address and remaining risks. The Reformatter toolkit development team will deliver a final DAP to NDE on 11/30/2011. It will contain:

Final version of the code Sample data Final version of SMM, EUM, and TRR documentation. Note: there are sections in the SPSRB documents that will be left blank as they are assigned to the integrators (NDE). 195 Open Discussion The review is now open for free discussion 196

Recently Viewed Presentations