Skip to content

[Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds - Track 106 Exclusively #152

Open
Open
[Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds - Track 106 Exclusively#152
Labels
bugSomething isn't working
@cmarshak

Description

@cmarshak
cmarshak
opened on Sep 1, 2023

Describe the bug

This accounts for 207 errors of the 94k jobs submitted. The error log looks like this:

Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 233, in gunw_slc
    nc_path = update_gunw_with_solid_earth_tide(nc_path, "secondary", loc_data['secondary_orbits'])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 474, in update_gunw_with_solid_earth_tide
    solid_earth_tide_ds = compute_solid_earth_tide_from_gunw(gunw_path=gunw_path,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 204, in compute_solid_earth_tide_from_gunw
    tide_e, tide_n, tide_u = compute_enu_solid_earth_tide(orbit_xmls=orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 372, in compute_enu_solid_earth_tide
    azimuth_time_arr = get_azimuth_time_array(orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 270, in get_azimuth_time_array
    datetime_isce, rng = orb.geo2rdr([lat, lon, hgt])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/Orbit/Orbit.py", line 1055, in geo2rdr
    raise Exception('Interpolation time out of bounds')
Exception: Interpolation time out of bounds

Here are 10 input dicts for the errors

{'job_id': '9b8cfae5-4f17-47b8-96a7-4c2ed43844cf',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8386.931]},
 {'job_id': 'ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8647.712]},
 {'job_id': '79b9485e-9d86-4a1b-a634-d00afb93a9ef',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/79b9485e-9d86-4a1b-a634-d00afb93a9ef/79b9485e-9d86-4a1b-a634-d00afb93a9ef.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8434.798]},
 {'job_id': '172d77d1-c524-43c9-8e73-5b53ccc0f0d9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20201004T225943_20201004T230010_034653_04092E_B85D',
    'S1A_IW_SLC__1SDV_20201004T230008_20201004T230035_034653_04092E_3208'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/172d77d1-c524-43c9-8e73-5b53ccc0f0d9/172d77d1-c524-43c9-8e73-5b53ccc0f0d9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8500.854]},
 {'job_id': '6602b23b-d1f1-427a-bd04-bf299ae041c9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200922T225943_20200922T230010_034478_04030D_928D',
    'S1A_IW_SLC__1SDV_20200922T230007_20200922T230034_034478_04030D_DD2D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/6602b23b-d1f1-427a-bd04-bf299ae041c9/6602b23b-d1f1-427a-bd04-bf299ae041c9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8770.476]},
 {'job_id': '33692e6b-8835-4a26-990a-0ccccec5502e',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180124T225946_20180124T230013_020303_022ABA_6976',
    'S1A_IW_SLC__1SDV_20180124T230010_20180124T230048_020303_022ABA_CBE6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20171219T225947_20171219T230014_019778_021A33_184F',
    'S1A_IW_SLC__1SDV_20171219T230012_20171219T230049_019778_021A33_6C12'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/33692e6b-8835-4a26-990a-0ccccec5502e/33692e6b-8835-4a26-990a-0ccccec5502e.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8454.583]},
 {'job_id': '2645f3f9-f796-41ec-afab-45c7cb8c2f6f',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180301T225945_20180301T230012_020828_023B72_4B9D',
    'S1A_IW_SLC__1SDV_20180301T230010_20180301T230047_020828_023B72_C96D'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/2645f3f9-f796-41ec-afab-45c7cb8c2f6f/2645f3f9-f796-41ec-afab-45c7cb8c2f6f.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8444.457]},
 {'job_id': 'a5358300-2c10-44c2-bf8c-b5750774baa1',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200314T225933_20200314T230000_031678_03A6F1_1983',
    'S1A_IW_SLC__1SDV_20200314T225958_20200314T230025_031678_03A6F1_931E'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/a5358300-2c10-44c2-bf8c-b5750774baa1/a5358300-2c10-44c2-bf8c-b5750774baa1.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [7856.89]},
 {'job_id': 'd5cacfdd-fb3e-4da8-979b-c168d891f743',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20190224T225952_20190224T230019_026078_02E88A_1102',
    'S1A_IW_SLC__1SDV_20190224T230016_20190224T230054_026078_02E88A_A34D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180205T225945_20180205T230012_020478_02304E_B53C',
    'S1A_IW_SLC__1SDV_20180205T230010_20180205T230047_020478_02304E_3008'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/d5cacfdd-fb3e-4da8-979b-c168d891f743/d5cacfdd-fb3e-4da8-979b-c168d891f743.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8640.922]},
 {'job_id': '9b91341b-8d28-4ee2-9cb6-d24097545928',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20181027T225955_20181027T230022_024328_02A9DB_AD58',
    'S1A_IW_SLC__1SDV_20181027T230020_20181027T230057_024328_02A9DB_85D6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b91341b-8d28-4ee2-9cb6-d24097545928/9b91341b-8d28-4ee2-9cb6-d24097545928.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8542.152]}

These are concentrated exclusively in 106.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions