Executor by koerberm · Pull Request #419 · geo-engine/geoengine

koerberm · 2021-12-10T19:30:12Z

This is a first draft of the executor integration. I implemented a noop executor that is used when pro-feature is inactive.
There is a sample application in the "plots" endpoint.

# Conflicts: # services/Cargo.toml

koerberm · 2021-12-10T19:31:02Z

Executor that decouples processing from web requests

coveralls · 2021-12-10T19:54:21Z

Pull Request Test Coverage Report for Build 1914911059

953 of 1147 (83.09%) changed or added relevant lines in 24 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 85.327%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
datatypes/src/primitives/multi_polygon.rs	32	33	96.97%
services/src/handlers/wfs.rs	12	13	92.31%
services/src/pro/util/tests.rs	0	1	0.0%
operators/src/pro/executor/operators.rs	68	70	97.14%
operators/src/util/raster_stream_to_png.rs	8	10	80.0%
services/src/pro/contexts/postgres.rs	1	4	25.0%
services/src/pro/server.rs	0	3	0.0%
services/src/handlers/wms.rs	10	14	71.43%
services/src/pro/contexts/in_memory.rs	1	5	20.0%
operators/src/pro/executor/error.rs	0	8	0.0%

Totals
Change from base Build 1893423386:	-0.03%
Covered Lines:	30826
Relevant Lines:	36127

💛 - Coveralls

operators/src/executor/mod.rs

operators/src/pro/executor/mod.rs

services/Cargo.toml

services/src/contexts/mod.rs

services/src/error.rs

services/src/handlers/plots.rs

# Conflicts: # services/src/error.rs

Co-authored-by: Christian Beilschmidt <fyaa@ntacity.de>

# Conflicts: # operators/src/error.rs # services/src/pro/contexts/in_memory.rs

This reverts commit fe648d3.

Restricted Hash and Eq implementation to concrete Key type

# Conflicts: # operators/src/error.rs

… state, i.e., only plots use the executor);

# Conflicts: # services/src/error.rs

…tain each other;

…r FeatureCollectionRows to allow spatio-temporal filtering;

…atic result stream. This commit contains implementations for VectorQueryProcessor and RasterQueryProcessor.

…reams;

…ng executors;

# Conflicts: # services/src/error.rs

ChristianBeilschmidt

It seems to work 😄

2x NDVI Raster

2022-02-17T08:29:49.881696Z DEBUG geoengine_operators::pro::executor: Starting new computation for request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 33.75, y: 0.0 }, lower_right_coordinate: Coordinate2D { x: 45.0, y: -11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:29:49.881885Z DEBUG geoengine_operators::pro::executor: Joining running computation for request. New: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 33.75, y: 0.0 }, lower_right_coordinate: Coordinate2D { x: 45.0, y: -11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }, Running: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 33.75, y: 0.0 }, lower_right_coordinate: Coordinate2D { x: 45.0, y: -11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:29:49.884395Z DEBUG geoengine_operators::pro::executor: Computation finished. Notifying consumer streams. Request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 33.75, y: 0.0 }, lower_right_coordinate: Coordinate2D { x: 45.0, y: -11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }

Raster-Vector-Join of NDVI and Natural Earth Points, then 2x Histogram:

2022-02-17T08:34:16.807763Z DEBUG geoengine_operators::pro::executor: Starting new computation for request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 45.0, y: 22.5 }, lower_right_coordinate: Coordinate2D { x: 56.25, y: 11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:34:16.809175Z DEBUG geoengine_operators::pro::executor: Stream progressed too far or results do not cover requested result. Starting new computation for request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 33.75, y: 11.25 }, lower_right_coordinate: Coordinate2D { x: 45.0, y: 0.0 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:34:16.809332Z DEBUG geoengine_operators::pro::executor: Stream progressed too far or results do not cover requested result. Starting new computation for request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 45.0, y: 45.0 }, lower_right_coordinate: Coordinate2D { x: 56.25, y: 33.75 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:34:16.809545Z DEBUG geoengine_operators::pro::executor: Stream progressed too far or results do not cover requested result. Starting new computation for request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 0.0, y: 56.25 }, lower_right_coordinate: Coordinate2D { x: 11.25, y: 45.0 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }    
2022-02-17T08:34:16.810209Z DEBUG geoengine_operators::pro::executor: Computation finished. Notifying consumer streams. Request: RasterTaskDescription { id: WorkflowId(8df9b0e6-e4b4-586e-90a3-6cf0f08c4e62), spatial_bounds: SpatialPartition2D { upper_left_coordinate: Coordinate2D { x: 45.0, y: 22.5 }, lower_right_coordinate: Coordinate2D { x: 56.25, y: 11.25 } }, temporal_bounds: TimeInterval [1396353600000, 1396353600000), _p: PhantomData }

Maybe we need some experiments when it joins and when this does not work. Maybe chunk sizes for vector data are too small 🤷 .

datatypes/src/primitives/spatial_partition.rs

services/src/pro/contexts/mod.rs

operators/src/pro/executor/mod.rs

ChristianBeilschmidt · 2022-02-17T07:59:12Z

services/src/pro/executor/mod.rs

+    fn slice_result(&self, result: &Self::ResultType) -> Option<Self::ResultType> {
+        Some(match result {
+            Ok(wpo) => Ok(wpo.clone()),
+            Err(_e) => Err(crate::error::Error::NotYetImplemented),


What would be necessary to do in case of an error?

The problem here is that our Error type is not cloneable. So I cannot return the original error received from the computation.
Either we find a way to make our error cloneable (which is impossible due to wrapped errors I think), or we introduce a new error type that contains e.g, a meaningful message .

Yeah, I guess this is a huge topic to discuss.

# Conflicts: # services/src/util/config.rs

koerberm added 5 commits November 23, 2021 16:36

Proof of concept

50623b5

Merge branch 'master' into executor

f1cda0f

# Conflicts: # services/Cargo.toml

Adapted to new executor api

ad65b85

Merge branch 'master' into executor

e03b9fd

First draft of executor integration

b622945

ChristianBeilschmidt requested changes Jan 10, 2022

View reviewed changes

koerberm and others added 15 commits January 10, 2022 13:40

Merge branch 'master' into executor

f85fedc

# Conflicts: # services/src/error.rs

removed clap dependency;

ec09bc3

Clarified error message when returning a stream proxied by the executor;

1ebc788

Made executor API available in PRO only

30a6c58

Removed executor/task manager from non-pro version;

7cd7fd0

Fixed pro import;

d5d159b

Introduced ReceiverStream<T> to the replay channel;

7efd9fe

Added test for ReceiverStream<T>

e887daa

Update operators/src/pro/executor/mod.rs

7894ba5

Co-authored-by: Christian Beilschmidt <fyaa@ntacity.de>

reverted commit suggestion;

714d3d5

Made unreachable error explicit;

11819ad

Completed test with dropped consumer;

6be3632

Clippy & Typo;

eb1e3e9

Added docs

b3cce36

Made replay channel queue size configurable;

f9b207e

ChristianBeilschmidt marked this pull request as ready for review January 19, 2022 13:41

ChristianBeilschmidt previously approved these changes Jan 19, 2022

View reviewed changes

koerberm added 4 commits January 26, 2022 09:01

Merge remote-tracking branch 'origin/master' into executor

c23bad5

# Conflicts: # operators/src/error.rs # services/src/pro/contexts/in_memory.rs

tokio time feature now in regular dependencies;

937b37d

Merge branch 'spatial-resolution-fix' into executor

1bc31f9

Draft for adding the QueryRectangle to the executor key;

fe648d3

koerberm dismissed ChristianBeilschmidt’s stale review via fe648d3 January 26, 2022 12:11

koerberm added 3 commits January 28, 2022 13:03

Revert "Draft for adding the QueryRectangle to the executor key;"

3c89f70

This reverts commit fe648d3.

Second draft for adding the QueryRectangle to the Executor Key;

377a1ad

Restricted Hash and Eq implementation to concrete Key type

Merge branch 'master' into executor

0d4a097

# Conflicts: # operators/src/error.rs

ChristianBeilschmidt previously approved these changes Jan 31, 2022

View reviewed changes

koerberm added 16 commits February 9, 2022 11:48

Executor: First draft for advanced result sharing (recovered previous…

d45ebaa

… state, i.e., only plots use the executor);

Merge branch 'master' into executor

f34062b

# Conflicts: # services/src/error.rs

Added ExecutorDescriptions for Vector data;

a6dc6c6

Fixed SpatialPartition2D::contains: Ensuring that equal instances con…

23e1c4d

…tain each other;

Implementer Intersects<BoundingBox2D> for all GeometryRef types;

82d40b2

Implemented Intersects<BoundingBox2D> and Intersects<TimeInterval> fo…

851cea0

…r FeatureCollectionRows to allow spatio-temporal filtering;

Added Debug trait to ExecutorTaskDescription, refined logging and docs;

d24134f

New trait 'OneshotQueryProcessor' to turn a QueryProcessor into a 'st…

f57d65d

…atic result stream. This commit contains implementations for VectorQueryProcessor and RasterQueryProcessor.

Added new png-rendering which directly consumes a stream of tiles;

115a267

Implemented ExecutorTaskDescriptions for Plots, Vector- and Raster st…

016b0d5

…reams;

Added executors for all data types to the TaskManager;

b4a6c0d

Implemented Pro-Handlers for wms and wfs that utilize the correspondi…

865840e

…ng executors;

Fixed intersects implementations of primitives and added tests;

0ecee9a

Added tests for OneshotQueryProcessor implementations;

9534040

Added tests for executor task descriptions

4a40f0b

clippy;

c28f6f3

koerberm dismissed ChristianBeilschmidt’s stale review via c28f6f3 February 14, 2022 16:27

koerberm added 2 commits February 14, 2022 17:30

Merge branch 'master' into executor

04d1607

# Conflicts: # services/src/error.rs

Snafu

6efff61

ChristianBeilschmidt requested changes Feb 17, 2022

View reviewed changes

koerberm added 4 commits February 28, 2022 09:14

Renamed can_join -> is_contained_in in ExecutorTaskDescription

448c9c2

Introduced constant for the Executor task queue size;

693a17d

Clarified that a submitted future always returns a result;

6bd7fad

Merge branch 'master' into executor

4b83dea

# Conflicts: # services/src/util/config.rs

ChristianBeilschmidt approved these changes Mar 1, 2022

View reviewed changes

koerberm mentioned this pull request Mar 24, 2022

Executor scheduler #514

Open

Conversation

koerberm commented Dec 10, 2021

Uh oh!

koerberm commented Dec 10, 2021

Uh oh!

coveralls commented Dec 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 1914911059

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristianBeilschmidt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristianBeilschmidt Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

koerberm Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

ChristianBeilschmidt Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coveralls commented Dec 10, 2021 •

edited

Loading