Features/#800 clustering with focus by KathiEsterl · Pull Request #840 · openego/eTraGo

KathiEsterl · 2026-02-25T09:33:28Z

Closes #800

This PR introduces a new functionality enabling spatial clustering with a regional focus.

The focus region can currently be defined either via a shapefile, or by selecting the corresponding Kreis.
A default parameterization is implemented to control the distribution of buses outside the focus region. The clustering weights are adjusted based on the shortest path distance of each bus to the boundary of the focus region. Therefore, the distance it used within a sigmoid decay function with a characteristic length scale of 50 km.

While the current implementation is fully functional and usable, several aspects require further refinement. These improvements will be tracked in a dedicated follow-up issue.

In addition to the clustering extensions, the spatial clustering arguments were partially restructured to improve usability and clarity. Some related aspects are documented in #811.

…ustering

…ort-check

…weighting based on electrical distance of buses from buses in focus region

…culation of distance to border of focus region, set up default parameters, add pot. h2 clustering

…th busmap import

etrago/cluster/spatial.py

ClaraBuettner · 2026-02-25T12:02:35Z

Do you have anything in particular in mind that I should try out?

KathiEsterl · 2026-02-25T12:21:48Z

Maybe you could try it once with a shape-file and once with a list of Kreise and check the clustered network afterwards? I also did that, of course, but that is mainly what I think could be beneficial.

etrago/appl.py

ClaraBuettner · 2026-02-26T09:07:47Z

I ran into some problems when I wanted to test the focus region:

Do I have to add in the focus region list sth. like "Schleswig-Flensburg" or "DEF01"? I guess it is the first option, but it is not completely clear yet. It would be also nice if it fails if names are added there that don't exist.
When I try to run it for "Schleswig-Flensburg" I run into this error:

  File ~/etrago/eTraGo/etrago/appl.py:624 in run_etrago
    etrago.spatial_clustering()

  File ~/etrago/eTraGo/etrago/cluster/electrical.py:1074 in run_spatial_clustering
    weight = focus_weighting(

  File ~/etrago/eTraGo/etrago/cluster/spatial.py:923 in focus_weighting
    border_buses = buses_gdf.loc[

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1073 in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1301 in _getitem_axis
    return self._getitem_iterable(key, axis=axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1239 in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1432 in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexes/base.py:6070 in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexes/base.py:6133 in _raise_if_missing
    raise KeyError(f"{not_found} not in index")

KeyError: "['9395', '14304', '20264', '14772'] not in index"

ClaraBuettner · 2026-02-26T12:00:26Z

When I try to use the "kmeans" method, the gas_clsutering fails:

  File ~/etrago/eTraGo/etrago/appl.py:625 in run_etrago
    etrago.spatial_clustering_gas()

  File ~/etrago/eTraGo/etrago/cluster/gas.py:1194 in run_spatial_clustering_gas
    self.network, busmap = gas_postprocessing(self, busmap, medoid_idx)

  File ~/etrago/eTraGo/etrago/cluster/gas.py:448 in gas_postprocessing
    medoid_idx[busmap.values.astype(int)].values,

TypeError: 'NoneType' object is not subscriptable

KathiEsterl · 2026-03-10T10:08:22Z

I ran into some problems when I wanted to test the focus region:

* Do I have to add in the focus region list sth. like "Schleswig-Flensburg" or "DEF01"? I guess it is the first option, but it is not completely clear yet.  It would be also nice if it fails if names are added there that don't exist.

* When I try to run it for "Schleswig-Flensburg" I run into this error:

  File ~/etrago/eTraGo/etrago/appl.py:624 in run_etrago
    etrago.spatial_clustering()

  File ~/etrago/eTraGo/etrago/cluster/electrical.py:1074 in run_spatial_clustering
    weight = focus_weighting(

  File ~/etrago/eTraGo/etrago/cluster/spatial.py:923 in focus_weighting
    border_buses = buses_gdf.loc[

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1073 in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1301 in _getitem_axis
    return self._getitem_iterable(key, axis=axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1239 in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexing.py:1432 in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexes/base.py:6070 in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)

  File ~/etrago/venv-py310/lib/python3.10/site-packages/pandas/core/indexes/base.py:6133 in _raise_if_missing
    raise KeyError(f"{not_found} not in index")

KeyError: "['9395', '14304', '20264', '14772'] not in index"

I added a warning for non-valid names.

I also fixed the problem with 'Schleswig-Flensburg' which was related to cross-border lines. Should be working now for this case and also other Kreise near borders.

KathiEsterl · 2026-03-10T10:08:54Z

When I try to use the "kmeans" method, the gas_clsutering fails:

  File ~/etrago/eTraGo/etrago/appl.py:625 in run_etrago
    etrago.spatial_clustering_gas()

  File ~/etrago/eTraGo/etrago/cluster/gas.py:1194 in run_spatial_clustering_gas
    self.network, busmap = gas_postprocessing(self, busmap, medoid_idx)

  File ~/etrago/eTraGo/etrago/cluster/gas.py:448 in gas_postprocessing
    medoid_idx[busmap.values.astype(int)].values,

TypeError: 'NoneType' object is not subscriptable

Solved this issue.

…sioning

…nding and adapt parameter names wihtin clustering process

…non-valid names or regions

ClaraBuettner · 2026-03-24T14:33:23Z

With these settings:

    "network_clustering": {
        "method": {
            "focus_region": ["Schleswig-Flensburg", "Flensburg"],  # None, shape-file or list with string for Kreise
            "per_country": True,  # if True, buses are restricted to one cluster per country
            "algorithm": "kmeans",  # choose clustering method: kmeans or kmedoids-dijkstra
            "remove_stubs": False,  # remove stubs before kmeans clustering
            "use_reduced_coordinates": False,  # if True, do not average cluster coordinates (in remove stubs)
            "line_length_factor": 1,  # Factor to multiply distance between new buses for new line lengths
            "random_state": 42,  # random state for replicability of clustering results
            "n_init": 10,  # affects clustering algorithm, only change when neccesary
            "max_iter": 100,  # affects clustering algorithm, only change when neccesary
            "tol": 1e-6,  # affects clustering algorithm, only change when neccesary
            "cpu_cores": 4,  # number of cores used during clustering, "max" for all cores available.
        },
        "electricity_grid": {
            "active": True,  # choose if clustering is activated
            "cluster_within_focus": False,  # False for very low clustering within focus region
            "n_clusters": 100,  # total number of resulting AC nodes
            "k_elec_busmap": False,  # False or path/to/busmap.csv
        },
        "gas_grids": {
            "active": True,  # choose if clustering is activated
            "cluster_within_focus": False,  #  False for very low clustering within focus region
            "n_clusters_ch4": 15,  # total number of resulting CH4 nodes
            "n_clusters_h2": 15,  # total number of resulting H2 nodes
            "k_ch4_busmap": False,  # False or path/to/ch4_busmap.csv
        },
    },

I unfortunatly run into this error:

  File ~/etrago/eTraGo/etrago/appl.py:651
    etrago = run_etrago(args, json_path=None)

  File ~/etrago/eTraGo/etrago/appl.py:621 in run_etrago
    etrago.spatial_clustering()

  File ~/etrago/eTraGo/etrago/cluster/electrical.py:1076 in run_spatial_clustering
    weight = focus_weighting(

  File ~/etrago/eTraGo/etrago/cluster/spatial.py:996 in focus_weighting
    container = p.starmap(shortest_path, gen(paths, chunksize, graph))

  File /usr/lib/python3.10/multiprocessing/pool.py:375 in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()

  File /usr/lib/python3.10/multiprocessing/pool.py:774 in get
    raise self._value

NodeNotFound: Node 13665 not found in graph

KathiEsterl · 2026-03-24T19:08:45Z

I tested with the same settings and a few minor changes, but I still can't reproduce this error. Maybe we can have a look together tomorrow in the office.

KathiEsterl · 2026-03-25T08:27:39Z

We found this error and fixed it.

I will start one more bigger calculation with all the new updates and fixes for a final check today.

… functionality

…than before

KathiEsterl added 19 commits July 10, 2025 15:18

Adjusts args and its calls with regard to the settings for spatial cl…

b9d7eaa

…ustering

Apply correct black version

e2caa5f

Fix errors from flake8 and isort checks

c7eca8e

Apply black again after manual changes to fulfill requirements for is…

3543aa9

…ort-check

Implement clustering with focus region as Zooming-Approach including …

31b1aa2

…weighting based on electrical distance of buses from buses in focus region

Apply isort and black

08d4c46

Merge branch 'dev' into features/#800-clustering-with-focus

aac1b10

Minor changes to default settings

71d3dd9

Merge dev

2236389

Merge dev

430aab2

Merge branch 'dev' into features/#800-clustering-with-focus

078c07d

Merge dev

e0e1fce

Apply black

f062193

Fix aggr strategy

6392643

Set up spatial zooming clustering in first verson to merge: adapt cal…

f423803

…culation of distance to border of focus region, set up default parameters, add pot. h2 clustering

Apply black

583a13e

Remove option to load weighting from clustering as it is redundant wi…

b59a598

…th busmap import

Merge dev

a81ae43

Apply black

fac97ad

KathiEsterl requested a review from ClaraBuettner February 25, 2026 09:33

Fix flake8 issue

30bb3a1

ClaraBuettner reviewed Feb 25, 2026

View reviewed changes

etrago/cluster/spatial.py Show resolved Hide resolved

ClaraBuettner reviewed Feb 25, 2026

View reviewed changes

etrago/cluster/spatial.py Show resolved Hide resolved

KathiEsterl added 2 commits February 25, 2026 13:44

Add docstring to focus_weighting function

a431db1

Fix flake8 issues

5e62d25

ClaraBuettner reviewed Feb 25, 2026

View reviewed changes

etrago/appl.py Show resolved Hide resolved

Explicitly set datatype to avoid problems in pandas<2.1

cb51356

KathiEsterl and others added 7 commits March 10, 2026 11:59

Add information on needed format for focus region, remove scn_decommi…

0c5bd04

…sioning

Adapt json-file to new args

0a70255

Add short desription of clustering with focus to documentation

762fcee

Adapt args within spatial clustering for better overview and understa…

c36f4b1

…nding and adapt parameter names wihtin clustering process

Fix error with regions close to borders and add warning when calling …

1a106e4

…non-valid names or regions

Apply black

ce1f8c6

Merge branch 'dev' into features/#800-clustering-with-focus

9b055c1

KathiEsterl and others added 5 commits March 25, 2026 09:41

Fix problem with focus region, kmeans and foreign DC links

22b2adb

Set back to default settings

7168776

Merge dev

1796944

Add Li-ion as default for negative load shedding within load shedding…

b11903d

… functionality

Adapt description for per_country-parameter which is named diffently …

33a73db

…than before

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features/#800 clustering with focus#840

Features/#800 clustering with focus#840
KathiEsterl wants to merge 35 commits intodevfrom
features/#800-clustering-with-focus

KathiEsterl commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

ClaraBuettner commented Feb 25, 2026

Uh oh!

KathiEsterl commented Feb 25, 2026

Uh oh!

Uh oh!

ClaraBuettner commented Feb 26, 2026

Uh oh!

ClaraBuettner commented Feb 26, 2026

Uh oh!

KathiEsterl commented Mar 10, 2026

Uh oh!

KathiEsterl commented Mar 10, 2026

Uh oh!

ClaraBuettner commented Mar 24, 2026

Uh oh!

KathiEsterl commented Mar 24, 2026

Uh oh!

KathiEsterl commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

KathiEsterl commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

ClaraBuettner commented Feb 25, 2026

Uh oh!

KathiEsterl commented Feb 25, 2026

Uh oh!

Uh oh!

ClaraBuettner commented Feb 26, 2026

Uh oh!

ClaraBuettner commented Feb 26, 2026

Uh oh!

KathiEsterl commented Mar 10, 2026

Uh oh!

KathiEsterl commented Mar 10, 2026

Uh oh!

ClaraBuettner commented Mar 24, 2026

Uh oh!

KathiEsterl commented Mar 24, 2026

Uh oh!

KathiEsterl commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants