Use cases with response function directly in ObsCore

[Bonnarel]

Use case A.1.3 is introduced this way

Identify all event lists and their associated IRFs of the region of Cygnus loop
(3 ◦ diameter) taken with SWGO. Only data of the event type ‘very-good’ are
selected, in order to limit the amount of downloaded data.

1 ) if we want to really find response functions associated to a given event list the spatial constraint

(i) s_region position intersects with the source modeled by a circle of 1.5
deg around (312.775, 30.683)

is not sufficient. Datasets related to the same observation should share the same obs_id.

2 ) But anyway my main concern as explained in my recent email to the list is to expose the IRF in ObsCore in the same way we do for event-lists, images of the sky, spectra of the sky. This overloads the ObsCore table with material we generally don't include in other spectral domains. And many of the characterisation parameters do not describe the response dataset itself but are borrowed from the event list. And I don't think the fact that response functions in high energy are generally used in inverse calibration methods instead of direct calibration methods makes a significant difference between all the response functions according to their discovery/access mode

3 ) of course one solution to avoid that is to use the standard datalink process as in use case A.1.6

4 ) but I understand that a drawback of this process is that there are two steps before accessing the response functions instead of one.

5 ) To avoid this issue we can install the DataLink response for all the datasets in a TAP service (single table) in the same TAP server than the ObsTAP one. Let's call this schema/table datalink.response. The semantics would be #calibration for response functions (or #irf if you prefer) and the response function class will be given in content_qualifier (as in A.1.6)
6 ) in that case the ADQL query proposed for this use case :

SELECT * FROM ivoa.obscore
NATURAL JOIN ivoa.obscore_hea
WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6)
AND (dataproduct_type = ’event-list’ OR dataproduct_type = ’aeff’
OR dataproduct_type = ’edisp’ OR dataproduct_type = ’psf’
OR dataproduct_type = ’bkgrate’)
AND (obs_collection = ’SWGO-DR1’)
AND (event_type = ’very-good’)

will be modified this way :

SELECT * FROM ivoa.obscore
NATURAL JOIN ivoa.obscore_hea
JOIN datalink.response ON ivoa.obscore.obs_publisher_did = datalink.response.ID
WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6)
AND (dataproduct_type = ’event-list’ )
AND (obs_collection = ’SWGO-DR1’)
AND (event_type = ’very-good’)
AND ( semantics = '#this'
OR (semantics = '#calibration'
AND (content_qualifier = ’aeff’
OR content_qualifier = ’edisp’
OR content_qualifier = ’psf’
OR content_qualifier = ’bkgrate’)
)
)

[Bonnarel]

Use case A.1.9 : the primary constraint is on obs_id and obs_collection. So the query should be modified to eiher discover a bundle or an event-list related to its response functions by a join.

[Bonnarel]

Use case A.1.8 : here I have a question . You really have different psf for the same observation ? In that case the use case seems to say the psf is different in various spatial locations inside the observation spatial bounds. If this is correct. How is this organized in practice ? As a cube ? a set of psf images ? something else ?

And is the event list obtained by this observation partitioned in several pieces ?

[bkhelifi]

Use case A.1.3:
In the case of WCD, observations stand for days, maybe month. So the data are "cut" into pieces and an unique identifier is set for the obs_id.
Then, because of the large volume of the dataset, and because of the analysis use cases, people are kind to select only one event_type. So, it is anticipated that the SWGO data will be cut into time and event_type to fullful the analysis use cases. And the data producer will set an "random" unique identifier is set for the obs_id.

Of course, an other possibility is to use a DataLink to separate the event_type.

The data producers will have at this end 2 technical ways of delivering their data

[bkhelifi]

IRFs and Use case A.1.9
We must define as data products each individual IRFs. We have NO choice. The associated files can be then delivered via ObsCore or DataLink.
It will be up to the data producers.

And they will be a third technical solution, that a tar file bundles the event list and the 4 IRFs.

Depending of the experiment, IRFs are associated to 1 observation, or 1 event_type, or a set of observations. For the case where the IRFs are associated to a set of observations (X-rays, WCD, neutrinos), it is completly stupid to copy the IRFs for each observation and to make a bundle. This is why we MUST be able to find and access the IRFs independently to the observation. They will be defined for a time range and a spatial region, so they can be query via ObsCore.

Do not forget that the HE domain covers different experimental techniques, such that multiple technical solutions are needed. It is a bit complicated, so do not hesitate to ask again.

[bkhelifi]

Use case A.1.8
In general, indeed, the PSF is different in various spatial locations inside the observation spatial bounds. I let @iannevans answer to the data organisation for Chandra.
For IACT, WCD and neutrino, the PSF IRFs has 2 or 3 axes (True energy, distance from the field-of-view center OR [Xfov, Yfov]) and the data is not a float but an array as a function of distance from the position. So, in pratice, this is an hypercube of 3 or 4 axes

[Bonnarel]

And is the event list obtained by this observation partitioned in several pieces ?

Sorry I think the event_type subsection answers my question about partionning

[Bonnarel]

Use case A.1.8 In general, indeed, the PSF is different in various spatial locations inside the observation spatial bounds. For IACT, WCD and neutrino, the PSF IRFs has 2 or 3 axes (True energy, distance from the field-of-view center OR [Xfov, Yfov]) and the data is not a float but an array as a function of distance from the position. So, in pratice, this is an hypercube of 3 or 4 axes

Thanks. In that case the spatial select statement in the ADQL query cannot alone extract the relevant PSF. This requires a processing of the hypercube to reduce it to the relevant subpart.

On the other side if you split the observation result in several event-list with different event_types you may be able to attach a specific psf to the event-list for each event_type

[iannevans]

Use case A.1.8 : here I have a question . You really have different psf for the same observation ? In that case the use case seems to say the psf is different in various spatial locations inside the observation spatial bounds. If this is correct. How is this organized in practice ? As a cube ? a set of psf images ? something else ?

And is the event list obtained by this observation partitioned in several pieces ?

The telescope+instrument PSF varies significantly with position in the field, energy, and (to a very much lesser extent) with the time history of the observation (the latter due to the spacecraft dither). For the Chandra Source Catalog for each observation we have PSF images for each energy band at the location of each X-ray detection, so it has defined spatial/spectral/temporal world coordinates. That is why use case 1.8 works the way it does. A single hypercube that recorded the PSF as a function of position and energy with sufficiently high fidelity to be useful is impractical as it would exceed 1TB in size (rough estimate).