Attempts to open a dataset of an OPenDAP dataset with a query throws an error

[ChrisJohnNOAA]

Versions impacted by the bug

5.9.1

What went wrong?

Exception when opening an OPenDAP dataset with query parameters: NetcdfDataset ncd = NetcdfDatasets.openDataset(fullUrl)

I've tried with both an http and dods url:
http://apdrc.soest.hawaii.edu/dods/public_data/SODA/soda_pop2.2.4?temp[1571:1571][0:18][197:197][370:370]
dods://apdrc.soest.hawaii.edu/dods/public_data/SODA/soda_pop2.2.4?temp[1571:1571][0:18][197:197][370:370]

(let me know if this is a different issue) I also tried this url:
https://ferret.pmel.noaa.gov/pmel/erddap/tabledap/ChukchiSea_454a_037a_fcf4?prof,id,cast,cruise,time,longitude,lon360,latitude&time%3E=2012-09-04&time%3C=2012-09-07&distinct()
It did open the dataset, but it failed to apply the constraints.

Relevant stack trace

java.lang.IllegalStateException: Cant modify
 at ucar.nc2.Attribute.setName(Attribute.java:655)
 at ucar.nc2.dods.DODSNetcdfFile.reGroup(DODSNetcdfFile.java:574)
 at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:264)
 at ucar.nc2.dods.DODSNetcdfFileProvider.open(DODSNetcdfFileProvider.java:24)
 at ucar.nc2.dataset.NetcdfDatasets.openProtocolOrFile(NetcdfDatasets.java:431)
 at ucar.nc2.dataset.NetcdfDatasets.openDataset(NetcdfDatasets.java:152)
 at ucar.nc2.dataset.NetcdfDatasets.openDataset(NetcdfDatasets.java:135)
 at ucar.nc2.dataset.NetcdfDatasets.openDataset(NetcdfDatasets.java:118)
 at ucar.nc2.dataset.NetcdfDatasets.openDataset(NetcdfDatasets.java:104)
 at gov.noaa.pfel.erddap.dap.DapServiceHelper$NetcdfDapStrategy.getGridData(DapServiceHelper.java:1065)
 at gov.noaa.pfel.erddap.dap.DapServiceHelper.getGridData(DapServiceHelper.java:254)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[lesserwhirls]

Greetings @ChrisJohnNOAA! NetCDF-Java does not support interacting with OPeNDAP endpoints with constraints prescribed (the documentation isn't super clear on that, I'll get that updated). Generally only the metadata and coordinate variable data are read on the initial opening of the dataset. Data from variables will be requested as the variables are sliced or a full read is called, so reading data is mostly a lazy operation. Are you able to open datasets with those URLs if you drop constraint expressions? (http://apdrc.soest.hawaii.edu/dods/public_data/SODA/soda_pop2.2.4, https://ferret.pmel.noaa.gov/pmel/erddap/tabledap/ChukchiSea_454a_037a_fcf4)

[ChrisJohnNOAA]

Yes, I can open them without the constraints.

It would allow for much more efficient interactions with the datasets (and much simpler code) if the constraints were applied on the OPenDAP server. Maybe this should be a feature request?

[lesserwhirls]

(and much simpler code)

Do you have some ideas on what the overall workflow would look like vs the current?

We do allow for users to download the raw DAP response and load that into netCDF-Java. For example, if you grab:

http://apdrc.soest.hawaii.edu/dods/public_data/SODA/soda_pop2.2.4.dods?temp[1571:1571][0:18][197:197][370:370]

you can load the resulting file into netCDF-Java. It might be possible to recognize the DAP request as having constraints applied, then grab the raw .das and .dods response and load those from memory using a similar code path. Using netCDF-Java at a level higher than NetcdfFile may or may not work, depending on if all of the data required by any metadata conventions were included in the .dods response, and that could certainly cause some confusion. @ethanrd do you have any thoughts or other perspectives on this?

I almost wonder if it would be better to add a new API to the opendap client that would take a dap2 URL with constraints and return a Map<String, Array> since that's a lower-level way of interacting with a DAP server.

[DennisHeimbigner]

I take it back. I notice that the TDS/opendap code has several file that appear to support constraints.
It may be that the code was disabled for some reason, or that the code did not work. But
there is some code for processing constraints.

FYI. The lack of constraint support is historical. When I took over the code some years ago,I accepted that status quo of no constraint support. Adding constraint support is doable, but the constraint expression language for OPeNDAP (aka DAP2) is a bit if a mess compared to, say, DAP4.

[lesserwhirls]

I found an interesting discussion from 2012 that suggests it may have only been partially implemented (and that sometimes only some parts of the constraint applied).

[ethanrd]

This has definitely come up before. I seem to recall that there were difficulties supporting a dataset open with a constrained DAP URL and then having to support further constraints if the Variable.read() requested slices of the data. Does that sound familiar @DennisHeimbigner?

I also think it would add complexity to the library for a capability (reading slices from a variable) that is already supported by providing the data slice information to the Variable.read() requests. So, I think it comes down to whether the added complexity to the library is worth it for an alternate way of specifying the slicing of a variable when read.

[ChrisJohnNOAA]

(and much simpler code)

Do you have some ideas on what the overall workflow would look like vs the current?

Sorry, I should have been clearer. I meant the users of netcdf-java would be able to have much simpler code if a file/dataset could be opened with the query.

For the query that I have ranges for already, this error was easy to workaround. For the url with a value based constraint:
https://ferret.pmel.noaa.gov/pmel/erddap/tabledap/ChukchiSea_454a_037a_fcf4?prof,id,cast,cruise,time,longitude,lon360,latitude&time%3E=2012-09-04&time%3C=2012-09-07&distinct()
It's going to be much more complicated to support (and likely require multiple requests to the dap server to handle). This in particular is what's going to be complicated code, that I think could be simpler and more efficient if the constraints were applied on the dap server side.

We do allow for users to download the raw DAP response and load that into netCDF-Java. For example, if you grab:

http://apdrc.soest.hawaii.edu/dods/public_data/SODA/soda_pop2.2.4.dods?temp[1571:1571][0:18][197:197][370:370]

you can load the resulting file into netCDF-Java. It might be possible to recognize the DAP request as having constraints applied, then grab the raw .das and .dods response and load those from memory using a similar code path. Using netCDF-Java at a level higher than NetcdfFile may or may not work, depending on if all of the data required by any metadata conventions were included in the .dods response, and that could certainly cause some confusion. @ethanrd do you have any thoughts or other perspectives on this?

I have not tried loading the raw .das and .dods response in. I'll look into that.

I almost wonder if it would be better to add a new API to the opendap client that would take a dap2 URL with constraints and return a Map<String, Array> since that's a lower-level way of interacting with a DAP server.

Just for some more context from my side. One of the primary reasons for me trying to migrate the OPeNDAP interactions in my codebase to netcdf-java is to have DAP2 and DAP4 support.

[DennisHeimbigner]

One of the complications of constraints has to do with the fact that there are two potential constraint expressions,
The first constraint is in the URL and I assume this is the one being discussed above.
The second constraint is in the netcdf-c API as specified by the netcdf.h file.
The latter constraint is, in the most general case, used in this function:

int nc_get_vars(int ncid, int varid,  const size_t *startp. const size_t *countp, const ptrdiff_t *stridep, void *ip);
````
So, when reading a variable thru nc_get_vars(), we must by some means apply the nc_get_vars() constraint
to the data sent by the server using the URL constraint.
* The hardest way to do it is to try to combine the two constraints into a single constraint and send that to the server as part of the URL.
* A somewhat simpler (but not simple) way is to get the data  from the server and constrained by the URL constraint. Then we compute new metadata to describe that constrained data. Then we use the nc_get_vars() constraint to extract the final data. This pushes off to the user the problem of combining the constraints since the user has to figure out what nc_get_vars() arguments to use to get the final desired data.

If either of the constraints are missing then the problem is simpler. Unfortunately, we (the maintainers of netcdf-java)  must assume the worst case of two constraints.

Bottom line: this is all doable (and I did it for DAP4), but painful.