[Woc-notify] [CM Request for Change - AWS 0011657]: WIFS HELPDESK Bug Submission - GRIB2 data not be purged enough?

Tue Jul 31 18:48:36 UTC 2012

The following issue has been UPDATED. 
====================================================================== 
https://mantis.awids/view.php?id=11657 
====================================================================== 
Reported By:                Rob Sallee
Assigned To:                Dan Vietor
====================================================================== 
Project:                    CM Request for Change - AWS
Issue ID:                   11657
Category:                   20.c) Numerical Products and Services (ASB)
Reproducibility:            have not tried
Severity:                   minor
Priority:                   normal
Status:                     assigned
TIN Required?         (If "YES" please upload):  
Deadline:                   08-01-12 
Disposition:                 
Configuration Change Authority: Lead - AWS 
Disposition Date:            
Level of Change:            Level 0 - Minor 
Change Notification:         
====================================================================== 
Date Submitted:             06-29-2012 20:14 UTC
Last Modified:              07-31-2012 18:48 UTC
====================================================================== 
Summary:                    WIFS HELPDESK Bug Submission - GRIB2 data not be
purged enough?
Description: 
Bug Submission

You received a new bug report from:

    Name: Neil Thompson
    Email Address: nthompson at atiaero.com
    Date: 18:00 06-29-12
    URL: https://aviationweather.gov/wifs/data/GRIB2/EGRR/T+36/
    Description: THE PROBLEM AS ENCOUNTERED ON WIFS The backup GRIB2 files
from EGRR are not being purged from WIFS as they are on EGRRâ€™s SADIS
FTP server. Because the EGRR GRIB2 file names are not unique, this is
causing unnecessary downloads of the EGRR GRIB2 files from WIFS to
determine what they contain. The load on the network could be reduced if
yesterdayâ€™s backup GRIB2 EGRR files were purged from WIFS just as
they are purged on SADIS. For example, the 18:00 model run GRIB2 files
from EGRR yesterday could be purged from WIFS at 18:00 today. QUESTION Is
there any chance that purging of old EGRR GRIB2 files can be implemented?
Soon? Sometime? Never? ADDITIONAL INFORMATION The following snapshot was
taken at 19:30 UTC 2012/06/29. It shows that the EGRR SADIS FTP server has
removed the 18:00 UTC model run forecast from yesterday (28 JUNE) in
preparation for the loading of todayâ€™s (29 JUNE) 18:00 UTC model
run. However, the KWBC WIFS SERVER snapshot still continues to show
yesterdayâ€™s 18:00 model run data files. When an automated
application is trying to download the GRIB2 files for the 18:00 model run
today, no confusion exists when loading from EGRR SADIS since the old
files have been removed a few hours ago. However, the KWBC WIFS server
keeps yesterdayâ€™s files until they are replaced with todayâ€™s
files. An application will unavoidably download yesterdayâ€™s 18:00
model run files while looking for todayâ€™s files to appear. Because
the filenames used by EGRR do not identify a fileâ€™s content, the
only way an application can confirm that todayâ€™s file has been
downloaded and not yesterdayâ€™s file is to actually download the
file and use wgrib2 or some other application to unpack the file. This is
a tremendous waste of network bandwidth considering the number of files,
their size (greater than 1 MB) and the number of users involved. It is
possible to look at the fileâ€™s date/time stamp without downloading
the file to determine if the file to be downloaded is todayâ€™s file
or yesterdayâ€™s file. However, on days when EGRR or KWBC are running
late or have problems generating the data an application should not really
use the date/time stamp to identify a fileâ€™s content. In an ideal
world, yesterdayâ€™s EGRR files should be purged from WIFS some time
before todayâ€™s files are made available similar to what is being
done on SADIS. Index of
ftp://sadisftp.metoffice.gov.uk/GRIB2/COMPRESSED/EGRR/T+36/ Name Size Last
Modified T+36_0600 1504 KB 2012-06-29 09:45:00 T+36_1200 1501 KB 2012-06-29
15:50:00 Index of /wifs/data/GRIB2/EGRR/T+36 Name Last modified Size
T+36_0000 29-Jun-2012 04:20 1.5M T+36_0600 29-Jun-2012 10:20 1.5M
T+36_1200 29-Jun-2012 16:20 1.5M T+36_1800 28-Jun-2012 22:20 1.5M
    Priority: med

====================================================================== 

---------------------------------------------------------------------- 
 (0030358) Dan Vietor (developer) - 07-05-12 15:27
 https://mantis.awids/view.php?id=11657#c30358 
---------------------------------------------------------------------- 
SADIS FTP only keeps the last 3 runs online.  We can scour our local
archive back to the last three.  This is an easy fix. 

---------------------------------------------------------------------- 
 (0030359) Dan Vietor (developer) - 07-05-12 15:29
 https://mantis.awids/view.php?id=11657#c30359 
---------------------------------------------------------------------- 
<b>Summary:</b>

Reset scour for EGRR GRIB2 data to 0.8 days

<b>Files:</b>

None

<b>Procedure:</b>

Update scour.conf file.

<b>Technical Impact:</b>

None.

<b>User Impact:</b>

None.

<b>Test Results:</b>

Tested on bubble and bcaws.