Skip to content

Scripts - HRRR - Check that all requested variables are in the grib file #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brentwilder wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Scripts - HRRR - Check that all requested variables are in the grib file #35

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7aa5d4d
Checks requested variables are in the grib file
brentwilder May 29, 2026
8e012dc
Merge branch 'main' into check-vars
brentwilder May 29, 2026
5e8e859
Centralize HRRR validation and index creation
brentwilder May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 49 additions & 9 deletions scripts/HRRR/download_hrrr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,11 @@ check_file_existence(){
# Pass EXIT_ON_SUCCESS to exit the job when found.
# return 3 on failure and signal alt pathway
if [[ -s "${FILE_NAME}" ]]; then
# If data is missing vars we remove it from disk
if ! check_file_is_valid_and_index "${FILE_NAME}"; then
rm -f "${FILE_NAME}"
return 3
fi
Comment thread
jomey marked this conversation as resolved.
Comment thread
jomey marked this conversation as resolved.
if [[ "${1}" == "${EXIT_ON_SUCCESS}" ]]; then
printf " exists \n"
exit 0
Expand All @@ -117,6 +122,39 @@ check_file_existence(){
}
export -f check_file_existence


check_file_is_valid_and_index() {
# Checks file has all required HRRR Variables
# If this check passes, we write the index file
local file=$1
local metadata
local var
local file_idx_data

# Check to make sure index exists, if not create it
if [[ -f "${file}.idx" ]]; then
metadata=$(cut -d: -f4,5 "${file}.idx")
else
file_idx_data=$(wgrib2 "${file}" -s 2>/dev/null)
metadata=$(echo "${file_idx_data}" | cut -d: -f4,5)
fi

# Look through to ensure hrrr vars are present
IFS='|' read -ra vars <<< "${HRRR_VARS}"
for var in "${vars[@]}"; do
if ! echo "${metadata}" | grep -E -q "^${var}(:| )?"; then
return 1
fi
done

# Write index if it does not exist yet
if [[ -n "${file_idx_data}" ]]; then
echo "${file_idx_data}" > "${file}.idx"
fi
Copy link
Copy Markdown
Member

@jomey jomey May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about doing this on line #136 and then using that as the basis for "grepping". If we have a miss, we will delete it again on line #146

I know that creating this index is quick and this way we would still only do it once

return 0
}
export -f check_file_is_valid_and_index

get_grib_range(){
INDEX_FILE=$(curl -s "${ARCHIVE_URL}.idx")
RANGE_GREP="grep -A 1 -B 1 "
Expand Down Expand Up @@ -204,9 +242,10 @@ download_hrrr() {

rm "$TMP_FILE"

# If download produced zero size file, retry with alternate archives
if [[ ! -s "$FILE_NAME" ]]; then
>&2 printf " File is zero size, checking alternate archives...\n"
# If download produced zero size file (or missing vars), retry with alternate archives
check_file_existence
if [[ $? -eq 3 ]]; then
>&2 printf " File is zero size or has missing variables, checking alternate archives...\n"

# Loop through archives until file is no longer zero size
# or all archives have been checked.
Expand All @@ -220,22 +259,23 @@ download_hrrr() {
wgrib2 - -v0 ${GRIB_THREADS} -match "${HRRR_VARS}" -grib "$FILE_NAME" >&1
find . -type f -name "${FILE_NAME}.missing" -size 0 -delete
rm "$TMP_FILE"
# break loop once file successfully downloaded and nonzero
[[ -s "$FILE_NAME" ]] && break
# Break loop once file successfully downloaded and has valid data
check_file_existence
if [[ $? -eq 0 ]]; then
break
fi
# Otherwise keep going
>&2 printf " Still zero size from %s\n" "$ALT_ARCHIVE"
fi
done
fi

if [[ -s "$FILE_NAME" ]]; then
# Create index file
wgrib2 -s "${FILE_NAME}" > "${FILE_NAME}.idx"
if [[ -f "${FILE_NAME}.idx" ]]; then
printf " created \n"
fi

# If the previous hour forecast successfully downloaded, replace missing file with it
if [[ -n "$MISSING_FILE" ]] && [[ -s "$FILE_NAME" ]]; then
if [[ -n "$MISSING_FILE" ]] && [[ -f "${FILE_NAME}.idx" ]]; then
"$COPY_SCRIPT" "$FILE_NAME" "$MISSING_FILE"
fi
}
Expand Down