Issue 256 - move camsl to geo_point table

[jo-asplin-met-no]

Fixes Issue 256.

[lukas-phaf]

I did a high level review, focusing mostly on the protobuf and postgres migrations. I have added some comments around these, and will try to get to the golang code later.

A general question is how we test this, as there is no ingestion and/or API support to see if inserting and querying actually works (unless I am missing something).

Ideally, we would have the ingestion and API in the same PR, so that we can add some camsl's to the test dataset, and confirm that everything works by testing that the data shows up in the API, preferable using the camsl range query.

[lukas-phaf]

The original table definition was this:

   id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
   point GEOGRAPHY(Point, 4326) NOT NULL UNIQUE
);

CREATE INDEX geo_point_idx ON geo_point USING GIST(point);

This gave the following constraint and indieces:

After forcing the migration to your branch, I have the following constraint and indices:

This corresponds to the indices that you would get with the following table definition (without any migrations):

CREATE TABLE geo_point(
   id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
   point GEOGRAPHY(Point, 4326),
   camsl INTEGER,
   CONSTRAINT geo_point_point_camsl_key UNIQUE NULLS NOT DISTINCT (point, camsl)
);

CREATE INDEX geo_point_idx ON geo_point USING GIST(point);

So I guess the question is, do we want to have a GIST index that includes camsl? This depends on the query, which I haven't looked at.

[jo-asplin-met-no]

Good question. Need to look into it again.

[lukas-phaf]

Note to also fix just load. Gives concurrency errors...

[fjugipe]

When you continue working with this can you add it to LocMetadata, so we can use it with/locations and also GetExtentsResponse should return the vertical_extent together with temporal & spatial

[jo-asplin-met-no]

I got stuck on just ingest-load failing due to a concurrency error. I think @lukas-phaf needs to take over this particular problem since he's the one who implemented the ingest optimization in putobservations.go. For reasons that are beyond me, the seemingly harmless addition of camsl next to point in the geo_point table now causes the complex INSERT statement in the function getGeoPointIDs to consistently fail in all 3 attempts, thus resulting in the error.

[lukas-phaf]

I got stuck on just ingest-load failing due to a concurrency error. I think @lukas-phaf needs to take over this particular problem since he's the one who implemented the ingest optimization in putobservations.go. For reasons that are beyond me, the seemingly harmless addition of camsl next to point in the geo_point table now causes the complex INSERT statement in the function getGeoPointIDs to consistently fail in all 3 attempts, thus resulting in the error.

Fair enough. The problem is not obvious to me by just reading the statement. I will try to look at this tomorrow.

[lukas-phaf]

I got stuck on just ingest-load failing due to a concurrency error. I think @lukas-phaf needs to take over this particular problem since he's the one who implemented the ingest optimization in putobservations.go. For reasons that are beyond me, the seemingly harmless addition of camsl next to point in the geo_point table now causes the complex INSERT statement in the function getGeoPointIDs to consistently fail in all 3 attempts, thus resulting in the error.

I committed a fix, which make the just load integration work for me.
The issue was that the JOIN at the end of the query does not deal correctly with NULL's (in camsl). I replaced the implicit join condition with an explicit one that deals properly with NULL's.

I did not yet update the file with example queries.

[github-actions]

Title	Coverage	Tests	Skipped	Failures	Errors	Time
API Unit Tests		47	0 💤	0 ❌	0 🔥	1.581s ⏱️
Ingest Unit Tests		16	0 💤	0 ❌	0 🔥	2.533s ⏱️