Overlay v2 sharding

[graydon]

This is a sketch of eager txset propagation using sharding and reed-solomon coding:

• nomination triggers a message to overlay to push the txset eagerly
• the txset gets sharded into a set of up to 255 original+recovery shards
• the shards are flooded in parallel to all direct peers, each of which then forwards whatever it receives to all of its peers (a sort of "two hops" all-to-all parallel broadcast)
• everyone tries to reconstruct the txset from the shards they receive, adding it to the txset cache when successful
• when anyone asks for a txset, if there's already one in the txset cache from this path, it's just returned to core immediately

note: there's no new XDR here -- I went back to "a custom wire format" for the shard messages, because I was feeling lazy, the shard-blobs have minimal structure, and also this really doesn't correspond to anything that ought to make it outside the overlay. if we want to do this "for real" we should probably make up some XDR type.

I haven't done any real testing or benchmarking besides the stuff copilot threw in here, but there are at least some metrics. I'll try to debug / test / start doing measurements tomorrow.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/​reed-solomon-simd@​3.1.0

View full report