Use a single dataset per data provider / frontend app

[Laurin-W]

Issue
I'm about to refactor the data provider to use ldo in general.

This comes with improvements such as

• typed objects
• link-traversal (auto-dereferencing required resources)
• the possibility to keep track of changes per object using transactions
• translating changes to diffs or SPARQL queries
• stores all data in RDF.js datasets, which store data as quads.

What ldo doesn't implement (yet) but which we can augment:

• Schema Validation
• Caching
• Pagination

My question
Can we store all data in one RDF dataset or should we create one dataset per resource?

In practice a resource URI translates to the graph URI. So storing everything in one RDF dataset should be fine in the vast majority of cases.

Approach to using a single dataset
The dataprovider needs to be refactored so that it queries resources accordingly, if they are not cached already or stale. This requires a slightly different caching approach.
Also, caching and filtering will need to be addressed.

In either case, we need to add/refactor pagination and schema validation.

Possible Issues
Where do you see issues with using one dataset for all frontend data?

• Also, we might run into issues when the graph URI does not translate to the resource URI: JsonLd objects can store information about graphs that do not belong to the same resource. E.g. I fetch https://foo.bar which returns a jsonld resource which contains a graph with id https://graph.id - should data from that graph as defined here be stored in the general dataset? What happends if I fetch https://graph.id and the data is contradictory?
• Refactoring filtering / queries could be challenging. We might have to come up with a strategy to decide on whether we need to fetch ldp resources or use a sparql query (if sparql is available).

Benefits
Currently, there is a lot of redundancy in the data fetched so it would probably bring performance gains and reduce server load.

I think that the single-dataset approach brings us closer to a unified framework which supports both the solid and the NextGraph sparql-based world.

Also, I suppose that it reduces complexity because you don't have to think about the dataset where data is stored in (almost) every case.

What next?
I would like to try using one dataset only with the use(typed)Collection implementation first and see how it goes.

---

@srosset81 do you have an opinion here?

[srosset81]

Can we store all data in one RDF dataset or should we create one dataset per resource?

In practice a resource URI translates to the graph URI. So storing everything in one RDF dataset should be fine in the vast majority of cases.

I've the feeling this is a implementation detail so I don't really have an opinion. In the future, all documents will be in their own named graph. If we can have named graphs in the local datasets, then that will probably work fine. On the backend, some resources may be in one dataset (storage) and another one in a different storage, but the URI will never collide so it should be fine.

The dataprovider needs to be refactored so that it queries resources accordingly, if they are not cached already or stale. This requires a slightly different caching approach. Also, caching and filtering will need to be addressed.

In either case, we need to add/refactor pagination and schema validation.

Hmm I think we should be careful here. The goal is that React-Admin data provider returns LDO instead of simple JS objects. LDO may offer features such as pagination, caching, but it may enter into conflict with React-Admin's way of doing things. React-Admin already handles pagination, it already caches resources internally, etc. LDO is not needed here. I know that Niko wants to use LDO has a "reactive store", but this will not work with React-Admin. Eventually we can have two libraries for accessing data, but I don't want to give up with React-Admin's way of doing things.

If the need for having a single dataset arise from the need of doing complex operations like these with LDO, then I would say we don't need a single dataset.

• Also, we might run into issues when the graph URI does not translate to the resource URI: JsonLd objects can store information about graphs that do not belong to the same resource. E.g. I fetch https://foo.bar/ which returns a jsonld resource which contains a graph with id https://graph.id/ - should data from that graph as defined here be stored in the general dataset? What happends if I fetch https://graph.id/ and the data is contradictory?

If we use named graphs, there shouldn't be any problem ?

• Refactoring filtering / queries could be challenging. We might have to come up with a strategy to decide on whether we need to fetch ldp resources or use a sparql query (if sparql is available).

React-Admin already has options to choose between fetching a LDP container or doing a SPARQL query. This is a perfect example of something where we don't need LDO.

I would like to try using one dataset only with the use(typed)Collection implementation first and see how it goes.

The useCollection hook only use the data provider fetch function, so in fact it doesn't really use any mechanisms from React-Admin. So my above remarks don't apply: this hook could be used completely independently from React-Admin, which is probably a good idea.