Prune map analysis search to save time and memory

[robvanderveer]

Map analysis takes days to calculate and gigabytes to store, but we're only interested in the top matches from standard to standard. It's no use to find 100 weakly linked topics if you have a couple of direct links and a number of strong links.
The idea therefore is to prune the search we do for map analysis, by stopping search early if we know that we're looking

1. design a strategy for pruning. First stab: don't search for average and weak links if we have direct and strong links. Don't search for weak links if we have direct, strong or average links'.
2. update the search algorithm. This will require some smartness, depending on how it is implemented.

By the way, I notice we don't seem to have weak links anymore: should we change our categorization and make 3-6 average and 7+ weak?

First step: find expertise in Neo4J. Rob

[northdpole]

By the way, I notice we don't seem to have weak links anymore: should we change our categorization and make 3-6 average and 7+ weak?

The query is the following:

OPTIONAL MATCH (BaseStandard:NeoStandard {name: $name1}) OPTIONAL MATCH (CompareStandard:NeoStandard {name: $name2}) OPTIONAL MATCH p = allShortestPaths((BaseStandard)-[*..20]-(CompareStandard)) WITH p WHERE length(p) > 1 AND ALL(n in NODES(p) WHERE (n:NeoCRE or n = BaseStandard or n = CompareStandard) AND NOT n.name in $denylist) RETURN p
followed by

OPTIONAL MATCH (BaseStandard:NeoStandard {name: $name1}) OPTIONAL MATCH (CompareStandard:NeoStandard {name: $name2}) OPTIONAL MATCH p = allShortestPaths((BaseStandard)-[:(LINKED_TO|CONTAINS)*..20]-(CompareStandard)) WITH p WHERE length(p) > 1 AND ALL(n in NODES(p) WHERE (n:NeoCRE or n = BaseStandard or n = CompareStandard) AND NOT n.name in $denylist) RETURN p

and then we filter and assign weights in code
we can start by optimizing these pretty brute-forcy queries

[sd2604]

Hey if this issue is still available could you pls assign this to me...I'd really want to work on this issue

[northdpole]

Hey, sure, this is about optimizin gap analysis. What do you think you will do to achieve this?

[ashutosh-engineer]

@robvanderveer @northdpole
I’d like to suggest a different direction for map analysis that focuses more on confidence than on exhaustiveness. Right now, the system does a very broad traversal and only later decides which paths are useful, which makes the process slow and memory-heavy while producing a lot of low-value results. In practice, we usually care about a small number of strong, meaningful connections, and once those are found, continuing to explore weaker paths doesn’t add much value. Instead of relying purely on shortest paths, the traversal could take relationship semantics into account and apply the existing penalty logic during the search itself. That way, as paths are expanded, those that quickly become “too weak” can be dropped early rather than being fully explored and filtered later in code. This also makes it reasonable to limit the maximum depth, since very deep paths tend to be hard to interpret and rarely useful in a standards context. With confidence handled inside the traversal, link categorization becomes simpler as well: strong links naturally surface first, weaker but still acceptable ones are bounded, and very low-confidence paths never appear at all. Overall, this shifts the system toward returning fewer but more meaningful results, while significantly reducing traversal cost and improving scalability without changing the core data model.

[robvanderveer]

This sounds good, but the proof of the pudding is in the eating. If this can be a feature toggle first, we can experiment and compare. I do believe that weak connections can be helpful to have when no stronger connection is known.

[ashutosh-engineer]

@robvanderveer

Right now, the traversal is fully exhaustive: all paths are computed via allShortestPaths() (with hardcoded depth limits), and link strength is only evaluated afterwards in application code. There’s no early-stop mechanism in the traversal itself, and weak links are already treated as a fallback (stored separately and only shown when requested).

This makes a feature-toggled approach quite natural: keep the current exhaustive behaviour as-is, and introduce an alternative traversal mode that searches progressively and stops early once enough strong paths are found, while still falling back to weaker links when no strong connections exist.

If you’re okay with this direction, I’d be happy to take this forward, I can take this forward at code level and prove it out by implementing the toggle and comparing runtime, memory usage, and result quality against the current approach — without changing the data model or default behaviour.