spike: Tackle the problem of generated ids

[jcpitre]

In #1 (comment) we realized that some providers would generate new ids for each version of datasets.
When we think about it, it makes sense that ids are only meaningful within the dataset itself. We are not guaranteed that they will make sense in the context of another dataset, even of the same feed.

We looked at mdb-2014. and saw that the shape_ids in shapes.txt a regenerated for each dataset. e.g. from mdb-2014-202603110034:

shape_id,shape_pt_lat,shape_pt_lon,shape_pt_sequence,shape_dist_traveled
0000abe0-5266-475b-808a-5cf929120a80,50.118257595,-5.540823891,229,

But for mdb-2014-202603090029 the "same" line is:

shape_id,shape_pt_lat,shape_pt_lon,shape_pt_sequence,shape_dist_traveled
3d88c650-6c5b-4f14-996b-e5b01b4eec33,50.118257595,-5.540823891,229,

We need to establish how prevalent this in the feeds we host.
And find a way to evaluate the diff considering these generated ids.

Here is copilot take on it:

A rough breakdown by field

Field	Stability	Reason
shape_id	🔴 Very unstable	Almost universally regenerated on export by scheduling tools
trip_id	🟠 Often unstable	HASTUS, Trapeze, and other tools generate these internally
service_id	🟠 Often unstable	Date-based generation is common (e.g. 20240115_WD)
block_id	🟠 Often unstable	Operational scheduling artifact
route_id	🟡 Moderately stable	Often matches public route numbers, but not always
fare_id	🟡 Moderately stable	Fare structures change slowly
stop_id	🟢 Usually stable	Stops are physical — agencies maintain these
agency_id	🟢 Very stable	Rarely changes

So realistically stop_id and agency_id are the only keys you can reliably trust. Everything else is potentially a surrogate.

This means for a large fraction of feeds, the diff engine's output for shapes.txt, trips.txt, and calendar.txt is likely dominated by key churn rather than real changes — which
significantly undermines the value of the tool for those files without content-based matching.

This is arguably the most important design problem for gtfs-diff-engine to solve

[davidgamez]

timebox: 2d

[emmambd]

Have @tzujenchanmbd and @skalexch been consulted on this one? I'd be curious about their takes on the Copilot generated table above.

[skalexch]

@emmambd I remember having some worries about this specific issue, mainly:
With no consistent primary key values to follow, the tool will not be useful for those feeds that change ids frequently. It will show a mix of removed and added rows that might essentially be just modified values. From my experience, route_ids can be unstable, and they are the worst to handle if they change. We might want to limit the display of changing shape_ids. We could mention the id that changed if the changes in coordinates are minimum.

For routes, we might explore avenues where the route id changes but the route long name is the same. But that's a v2 of the Diff tracker.

[emmambd]

As discussed earlier with @skalexch, I think the goal will be to provide pilot test users with a set of heuristics that work for them, even if it might not address other edge cases or cases outside their needs. For this reason, I think testing out relying on route_long_name for example, makes sense, rather than producing a diff report that they can't use easily to monitor route changes.

[ianktc]

For something like shapes, perhaps the best way to detect diffs is actually to actually compare the geojson of the generated shapes... but its most likely too costly... Is it something to put behind a membership?

[emmambd]

@ianktc I don't know if it's high value enough, but interesting consideration!

[jcpitre]

What we are doing:

• Establish how prevalent generated ids are in the feeds we have
• Add a way to determine if a given dataset has generated ids so we can just not process it. See Add a heuristic that tells if the standard diff is possible #5

One thing is sure, we will not be able to stay generic in the processing, meaning that all files and all fields will not be treated equally.
For example, for shapes, the files that is most likely to have generated ids according to AI, we can do what @ianktc is suggesting. We treat each shape as a polyline and use geometric proximity algorithms to determine the correspondence between a shape in the base dataset and a shape in the new dataset.
I know that this is not a priority. But we already tried the algorithm with success.