Dataset series support

[rokf]

Hello,

I was wondering if there were perhaps already any discussions about including dataset series support in the dataspace protocol, in addition to datasets I mean.

I.e. where the catalogue would support an additional array called datasetSeries which would contain entries of the DatasetSeries entity from DCAT. This was added in DCAT v3. Each series would also have a policy through hasPolicy, just as regular datasets.

One could then negotiate a contract for DatasetSeries instances in addition to Dataset instances. The target in the offer might then point to either a DatasetSeries or a Dataset - https://eclipse-dataspace-protocol-base.github.io/DataspaceProtocol/2025-1-err1/#negotiations-request-post.

This would be used in cases where individual datasets aren't offered to consumers - the consumer would have to negotiate a contract for a batch of them (dataset series).

Something along those lines 🤔 Any thoughts?

Thanks!

[jimmarino]

Thank you for reaching out.

In my opinion, this adds significant additional complexity and ambiguity (e.g., how do policies at the series level merge/override/replace set policies?) than it provides in capabilities.

It also has the potential to encourage users to model data sharing in ways that go against best practices. In particular, data sets should be coarse-grained so that contract negotiations and transfer processes are opened infrequently. The separation of the control plane and data plane then enables efficient data exchange by reusing the transfer process for repeated operations, without reproducing heavyweight contract negotiations and transfer processes.

In other words, handle batches, data linking and other related aspects in the data plane API, not the catalog.

[rokf]

Alright, I think that I know what we're going to do in our connector implementation. Internally our catalogue structure utilizes dataset series and works like this 👇 For the purpose of the dataspace protocol API we'll do some flattening where our internal datasets will disappear from the dataspace protocol implementation, instead our DatasetSeries instances will be presented as Datasets.

The only thing left to reevaluate then (internally I mean) is if we want to keep the option to have datasets be part of multiple different dataset series instances, because that complicates things a bit. We don't really want to allow a participant to get access to the same dataset (distribution) through multiple different agreements, which means that we'd have to combine ODRL policies and things like that during evaluation as you've already mentioned. Currently we prevent this kind of duplication by preventing this from happening during the negotiation phase - conflicting contracts (transactions) can't be instantiated.

When flattened the same catalogue would look like this from the perspective of our dataspace procotol implementation:

So in our implementation right now if you'd already have access to the dataset (series) on the left you couldn't create a contract for the dataset on the right, since they have overlapping distributions (datasets).