Improvments to THEMIS GMAG RMD file download and processing

[DCarSunDr]

The recent data gap with THEMIS GBO data revealed some areas where the download and processing scripts could be improved.

The primary change is that we need to switch from using wget to rsync, because the new Calgary server does not allow wget.

• Determine how to get the GBO RMD files retrieved using rsync while maintaining mirror structure
• Implement changes into gbo_uc_rmd_mirror.ksh
• Test functionality with gbo_uc_rmd_mirror_batch.ksh
• Commit changes to svn
• send to production

In the case of the download script, the actual wget calls complete relatively quickly; however, the script seems to spend a long time completing database updates, calling gbo_uc_rmd_mirror.php and updating a single file per query. We need to:

• Determine if this database is still being used by any other scripts/if it's still needed.
  • If we still want to keep this database, we need to pass the queries to a single .sql file, and then have that sql file passed to single a mysql database call, minimizing repeated connections.
    • If we prefer, we can have a separate script/cronjob actually process the .sql file in a type of "mysql workdir", minimizing the number of scripts which require password access and enabling the database logging to function even if the original script encounters a fatal error.
  • If not, we can remove the database and database logging calls.

In the case of the processing script, completing the backlog took several days; this is fine on its own, but since it works by checking for files containing directory information in a single location, it interferes with routine processing for not just GBO sites, but all networks which we retrieve data via RMD files. Even if the processing script was run on a different machine, the directory files and the directory that the script checks would need to be different as well (to prevent the routine RMD processing script from trying to process the same files).

Additionally, processing a single month's worth of data seems to take a long time, but the exact reason for why this is is currently unknown. We need to:

• Analyze and improve the performance of RMD processing for individual directories/files.
• Reconfigure the RMD scripts to allow routine processing while a reprocessing job is going on.
• Enable the processing script to be run in parallel and allow script options of start and end dates to aid future reprocessing jobs.

[DCarSunDr]

"asi_mirror_all.ksh" under "/disks/socware/thmsoc_dp_current/src/ksh/asi_new/" is a great example of using rsync, specifically from calgary, but I'm trying to finesse the "gbo_uc_rmd_mirror.ksh" script so that the files get downloaded via rsync, but also so that the output is readable by "thm_rmd2all.ksh".

I think the first part is straightforward, but the latter part takes some more work. Even though THEMIS GBO seems to be the only group that still uses thm_rmd2all.ksh, I think we should leave the thm_rmd2all.ksh script alone if it can be helped.

Which means the output format of "gbo_uc_rmd_mirror.ksh" must remain unchanged.

In "asi_mirror_all.ksh", the rsync output gets directed to a log file which exclusively records the rsync output. The rsync output is formatted using '%t $mirror_start_dir/%f [%l/%l]', which records the current date time, full mirrored filepath of retrieved file, and the length of the file in bytes.

Currently, "gbo_uc_rmd_mirror.ksh" creates a wget logfile and sends the output of a grep to a temporary logfile:

grep saved $WGET_LOGFILE | grep -v 'index\|robot' | awk '{print $6 " " $8}' > /tmp/wget_gbo_uc_rmd$$

The elements $6 and $8 should be the file path and size, respectively. I think if we ignore the date time part of the output format (or adjust the loop that reads the rsync log), we could probably use the output format used by "asi_mirror_all.ksh". Perhaps using '$mirror_start_dir/%f %l]'

[DCarSunDr]

Edits for gbo_uc_rmd_mirror.ksh should be done--going to do last verification and testing before sending to production.

[DCarSunDr]

Running thm_rmd2all.ksh for a day's worth of data, it again, I see that it only takes about a second to actually run--I think the database updates were what really bogged this script down. I noticed an error in how it searched for files in the gmag_mirror workdir directory. Now, it should only list the directory files in the order they were created.

@jameswilburlewis, I noticed that thm_rmd2all.ksh uses several directories which are outside of themisdata:

#ksh log - this is not deleted
KSHLOG=/home/thmsoc/rmd_processing/KSHLOG_${thisYear}_${thisMonth}
#IDL log - this is used inside IDL - more brief and concise - this is not deleted
idl_log=/home/thmsoc/rmd_processing/IDLLOG_${thisYear}_${thisMonth}
#IDL log - general idl output - this is a temporary file
TMPIDLLOG=/mydisks/home/thmsoc/rmd_processing/TMPIDLLOG_${thisYear}_${thisMonth}_${thisDay}_$$
#Temp directory
temp_dir=/home/thmsoc/rmd_processing/
#Calibration file dir
rmd_cal_dir=/home/thmsoc/software_dev/c_prog/UCLA_Cal/
#Directory of RMD files
rmd_mirror=/home/thmsoc/rmd_processing/rmd_mirror
#Directory for temporary RMD files
rmd_copy_dir=/home/thmsoc/rmd_processing/process_rmd2ff

I can understand creating a temporary directory to handle processing, but perhaps the logs should get saved/moved to a "rmd" directory in the gmag process logs? It also only makes a log for each month, so any daily processing just gets written to that month's log; however, they still accumulate--for gbo_uc_rmd_mirror.ksh, I've set it to remove the logs unless there's an error. That way, it should only have a saved log if something went wrong. Maybe we should do the same for thm_rmd2all.ksh, and other scripts which tend to write a lot of logs?

Otherwise, I am pretty happy with tests of the rsync version of gbo_uc_rmd_mirror.ksh and the tweaked thm_rmd2all.ksh script, so I've committed their changes to the svn and sent them to production. The thm_rmd2all.ksh isn't parallelized and can't be reprocessed independently, but I think it's performance without writing the logs is suitable enough (especially given that GBO seems to be the only group which still uses this script).

[DCarSunDr]

I'm going to move the log files to the themisdata processing logs gmag directory, so that all of the processing logs get kept in one place.

[DCarSunDr]

I've updated the directories involved in logging, and I've moved the old RMD logs to the new logging directory. I'm running a test of the GBO rmd processing pipeline, and so far so good! It indeed appears to take only about a minute to process a given date for a given station, which isn't too bad.

I'll check back in once the testing is done, but it seems the new temporary directory in mydisks/home/thmsoc is also working as intended. I don't think it'll run into this issue, but I do want to make sure it doesn't run up to memory limits.

[DCarSunDr]

I stopped the processing script this morning--it still had not finished processing by this morning, and the temporary IDL log it created had ballooned to a massive size: 714770848849 (bytes?)! Something is going on; I paused the cron job on talos for the processing, and I'm going to take a closer look to figure out what's causing the log file to explode like this.

To start, I think we should figure out what parts of the idl script are getting written to the log, and if there are any conditions where it could loop endlessly, or where something can become recursive. Another aspect worth investigating is if there are any ways to clear the log (or if there currently are, are they getting triggered).