CAMS NDACC server news

On this page we will list any major user-visible changes to the CAMS NDACC validation server. The Known Issues page lists all the problems we are currently aware of.

All feedback, questions, and comments are welcome at .

Wednesday, 28 September 2016 (v1.2.1)
  • New control model: gjjh
    Support has been added for the new (as of July 2016) 'gjjh' control model.
  • Higher model resolution
    As of July 2016, the o-suite and control models are being provided at a higher resolution.
  • Old control model deprecated
    The pre-July 2016 control model 'geuh' has been made VIP-visible only.
Wednesday, 23 March 2016 (v1.2)
  • Rebranding
    Applied CAMS template to the website. Active reports are now labelled as o-suite and control (currently, these are 0001 and geuh, respectively).
Tuesday, 23 Feb 2016 (v1.1.2)
  • geuh reports
    Support has been added for the new (as of September 2015) MACC_esuite control model geuh.
Tuesday, 04 Nov 2015 (v1.1.1)
  • BIRA deployment
    The server has now officially been moved to a physical host within the BIRA infrastructure (rather than in the S&T test environment)
  • 0001 reports
    Support has been added for the new (as of September 2015) MACC_osuite model 0001.
Thursday, 06 Nov 2014 (v1.1)
  • The ready-to-use NORS Validation Server has been operational since the beginning of 2014. In the period from then until the end of the NORS project (November 2014), a number of remaining issues have been addressed, and even some new functionality added.
  • The following new features have been added:
    g4e2 reports
    Support has been added for the new (as of September 2014) MACC_osuite model g4e2.
    H2O reports
    Support has been added for H2O as a Physical Parameter, along with a number of H2O intercomparisons. The resulting reports are visible to VIP users only.
    FTIR-NO2 reports
    Support has been added for FTIR-NO2 intercomparisons. The resulting reports are visible to VIP users only.
  • The following issues have been addressed:
    Certain reports still missing
    There are some issues with the metadata ingestion back end that are currently preventing all reports from being generated. This is actively being pursued.
    [5.4] Multiple experiments in one plot; No reports/plots involving multiple models or instruments
    Cross-reports for multiple models are now generated and shown by the server.
    [5.6] Some errors in the data plotting (e.g. negative O3 model outputs)
    NORS data providers have fixed various problems in their data products, and remaining bugs in the Validatoin server toolchain have been addressed.
    [7.4] Reports not yet being regenerated if updated NORS data arrives
    An algorithm for determining when reports needed to be regenerated was decided upon and implemented.
    [7.9] Integrate service at Belgian Institute for Space Aeronomy
    Integration finished at the end of the project.
Thursday, 13 Feb 2014 (v1.0.1)
  • The validation server web site is now also available through its official URL: nors-server.aeronomie.be
  • The following issues have now been closed (or are in the test phase):
    [UVVIS.DOAS] Support for the splitting of UVVIS.DOAS into two different instruments (UVVIS.DOAS.OFFAXIS and UVVIS.DOAS.ZENITH) has been added.
    Based on the new GEOMS templates from the UVVIS working group. Not all reports are yet being generated.
    [GUI] intercomparison selection options are now sorted alphabetically
    Used to be an arbitrary ordering
Friday, 20 Dec 2013 (v1.0)
  • The ready-to-use NORS validation server goes live and is now accessible to the wider Internet without requiring a password.
  • The following issues have now been closed:
    [2.1] There is no description of the MACC versions
    Added a Models, Instruments, and Parameters explanation page.
    [2.2] Better naming of the experiments
    Added a Models, Instruments, and Parameters explanation page.
    [4.1] There is no general introduction about the scope of the NVS website
    Added a new Overview/portal home page, as well as a mission statement page (based on text in the NORS white paper)
    [4.2] Confusing punctuation symbols in the password
    The password has been removed from the web site for UALL users.
    [4.6] The 'fnyp' experiment model (M2; IFS-MOZ AN e-suite) reports should be shown by default
    UALL users will only see the 'fnyp' experiment model
    [4.7] Some empty frames show up in some instances (e.g. O3.column.partial for MACC and NORS and their differences
    The toolchain has been updated and this should no longer be a problem.
    [5.1] Better identification of the data sets
    Addressed by the Models, Instruments, and Parameters explanation page.
    [5.3] Consistent scales for the plot axes
    Has been addressed in the reporting engine. Axis labeling is now a lot more consistent throughout the report, though some autoscaling still occurs here and there.
    [7.5] Internal timezone issue in storing of metadata may lead to time series in intervals being a couple of hours off at the edges
    Bug found and squashed.
    [7.7] Uncertainty propagation
    Has been added to the generated reports. Propagation for column integration will be included, but propagation when performing vertical regridding will not be included (requires availability of accurate covariance matrices, which is currently not mandated by GEOMS)). MACC data does not have uncertainty information.
    [7.8] Filtering of NDACC data based on version number (to use the latest data)
    The metadata ingestion backend has been updated accordingly.
Wednesday, 17 Dec 2013
  • The following issues have now been closed:
    [1.1] Some sites and/or instruments are currently missing; No LIDAR reports are being created; No UVVIS reports are being created
    The toolchain has been expanded to cover the missing algorithms. There are some reports still not being generated by the NVS, but this is now because of issues with the data or the metadata itself, not with the NVS. The data issues have been raised with the responsible PIs, and as soon as the products are updated, the reports will start appearing in the NVS. LIDAR reports are now being generated. UVVIS reports are now being generated.
    [1.2] Extending the files available to all GEOMS-described NDACC data
    All GEOMS-described data from the NDACC archive is now automatically included in the NORS report-generation process.
    [1.3] Do not limit the files available to the official NORS stations
    See answer for [1.2]
    [2.3] No information on the smoothing and regridding processes
    For [2.3] it is not really known what is requested. Some level of traceability is already provided at the end of each report by the list of parameters. A 'resample vertically' parameter will indicate vertical regridding actions and a 'vertical smoothing' parameter will indicate any smoothing actions. Absence of these parameters means that no smoothing/regridding was performed.
    [3.1] How does the NVS deal with submitted incomplete files? (e.g. files with columns only, but no profile)
    Dealing with incomplete files is not within the scope of the NVS. Such files will be discarded as invalid if the missing data was mandatory. For optional data we can only support one single intercomparison approach; More generally per combination of GEOMS template and species there can only be one approach to the intercomparison.
    [4.3] No shortcut to specific graphs inside the reports
    Outside NORS scope - will not be addressed.
    [4.4'] No location (latitude/longitude) and/or clickable world map of the stations
    Outside NORS scope - will not be addressed.
    [4.5] Is the NVS properly configured to provide specific information according to the different categories of users?
    Yes. VIP user functionality is not shown to UALL users.
    [5.2] Consistent size for the symbols (see very small symbols in the appended MWR reports)
    This is an intentional scaling effect: the size of the individually plotted data points in the scatter plots decreases as the total number of measurements increases. Will not be fixed.
    [5.5] Same colour conventions for all plots (e.g. NORS in green, MACC in red)
    Has been addressed: NORS measurements in scatter and profile plots are black, MACC models are orange/red/blue according to that community's convention, plots involving expressions over both NORS and MACC data (e.g. difference plots) are green.
    [6.2] Monthly results are available in anti-chronological order
    Will not be changed - the chosen order (latest reports first) will remain, as the expectation is that users will be primarily interested in the newest reports.
    [6.3] How to deal with the increasing number of plots in the reports due to the increasing number of stations?
    At present the number of stations is not a factor in the length of the reports (although it may become a factor in the complexity of individual figures, e.g. the category scatter plots). Addressing this issue is outside of the scope of NORS.
    [7.1] Generating the 'last three months' reports takes too long to be done on a daily basis
    This performance issue can be solved by an architectural change of the backend to support incremental collocation. This is outside the scope of the NORS project.
    [7.6] No user-upload of custom NORS or MACC data streams
    Descoped. Will not be addressed.
Wednesday, 16 Oct 2013
Reports
  • Support for resampling to a common grid, improving the profile plots when data from multiple stations is used (no more zig-zag profiles in the '[ALL]' reports).
  • Support for resampling to a 1DF grid, resulting in the addition of three new partial column mosaic plots (MACC, NORS, (MACC-NORS)/NORS).
  • New colour scheme for total column/profile plots to better match current practice in the MACC validation community. NORS measurements are now black, the three MACC models fnyp, fkya, and fsd7 are red, orange, and blue, respectively.
Web Server
  • Added a 'Special Functions > Create custom report' menu option for creating a custom report (user interface demo only — this functionality does not work yet.
  • Regenerating all the existing reports to use the new plots and colours. This is a process that will take a few days to complete.
  • The usual collection of smaller tweaks and bug fixes, especially in the report navigation mode.
  • Support download of full set of internal data files through the 'zip' links.
Monday, 30 Sep 2013
Friday, 27 Sep 2013
  • Improvements and fixes to the navigation and look & feel.
Friday, 20 Sep 2013
  • Many improvements and fixes have been made in the NORS toolchain backend, leading to better quality plots.
Wednesday, 11 Sep 2013
  • Based on input from the recent NORS teleconf, the NDACC stream now also includes non-RD data. As long as there is GEOMS-compatible metadata available, it will be included in the UALL report generation.
Wednesday, 21 Aug 2013
  • The test server goes live and is now accessible to the NORS team members.

Original announcement

Currently available for testing is mainly functionality for the 'UALL' type user:

  • Browsing and navigating through the generated reports
  • Viewing individual reports
  • Downloading individual reports as ZIP archives

You can also request a specific user account from us, and gain access to the 'UINT' and 'UVIP' functionality, which at present consists of:

  • Login module
  • Browsing and navigating though the MACC and NORS metadata catalogue
  • Downloading individual MACC/NORS source data files

Currently in-development functionality missing from the system:

  • user-driven creation of new reports
  • user-upload of custom NORS or MACC data streams
  • download of report-specific data source files
  • certain plots (Taylor plots and AVK plots specifically)
  • UVVIS reports (pending some issues with the algorithm)
  • LIDAR reports (no data yet)

Some other considerations:

  • Reports are generated for as much NORS RD data as we have available from Feb 2013 onwards. Anything you upload to the RD directory on NDACC should be automatically included.
  • We aim to have at least all the monthly reports as well as the daily 3-months-windows report available. At the moment, however, a number of these reports are still missing because of issues with the intercomparison tools we are working to resolve.
  • We are very actively developing the server, and expect to make frequent updates to the site's functionality as well as to be regenerating the reports themselves as we improve the toolchain. There is a "Server Status" menu item available on the web site that will bring you to a page where we will list all the major changes to the server.
The CAMS NDACC Validation Server is copyright © 2013-16 by s[&]t, The Netherlands.