perf: fold/map/filter optimization

[federico-stacks]

Description

This PR revives the work done in this #6955.

In short here what it does:

• fold, map, and filter were wrapping every sequence element in a SymbolicExpression only to immediately unwrap it again inside eval. This change eliminates that round-trip with a lazy SequenceIter iterator and a new apply_evaluated function that accepts Values directly.
• Result: ~7–8% faster fold and map, ~2–7% faster filter

Here the original description and benchmarking:

special_fold, special_map, and special_filter previously converted each sequence element into a SymbolicExpression wrapper via atom_values(), then fed those wrappers to apply(), which called eval() on each one to then unwrap them BACK to a Value. Per iteration this meant:

• One SymbolicExpression::atom_value(v) allocation (wraps the Value)
• One eval() call that immediately returns the wrapped value (no-op work)
• For fold/map: an intermediate Vec<SymbolicExpression> holding all N wrappers before the first step begins. For a fold over a 2000-element list, that is 2000 unnecessary heap allocations and 2000 no-op eval round-trips per call.

Added SequenceIter, a lazy owned iterator over sequence elements. Changed the predicate in retain_values to remove the internal SymbolicExpression::atom_value() and added apply_evaluated(..) to take pre-evaluated Value's directly, skipping the eval pass entirely.

Added clarity/benches/sequence_higher_order.rs with end-to-end regression benchmarks for fold, map, and filter at list sizes 100 / 500 / 2000.

Baseline: feat/variable-lookup-by-ref. (To rerun: cargo bench --bench sequence_higher_order -p clarity)

fold: sum a list of integers

list size	before	after	change
100	85.8 µs	78.7 µs	−7.1%
500	403 µs	366 µs	−7.8%
2000	1.564 ms	1.447 ms	−5.9%

map: increment every element

list size	before	after	change
100	75.7 µs	69.9 µs	−8.0%
500	347.9 µs	322.7 µs	−6.6%
2000	1.362 ms	1.253 ms	−7.3%

filter: keep even elements

list size	before	after	change
100	94.8 µs	89.4 µs	−7.3%
500	494.4 µs	476.6 µs	−2.9%
2000	3.308 ms	3.236 ms	−2.2%

fold and map see a consistent ~7–8% improvement across all sizes because they use the new IntoIterator path end-to-end. filter sees a smaller gain (~2–7%) because retain_values still iterates by index internally (see other retain_values PR for further improvements); the saving comes only from removing the SymbolicExpression wrap and eval round-trip per predicate call.

The improvement is uniform across list sizes (per-element overhead is eliminated).

Applicable issues

• fixes #

Additional info (benefits, drawbacks, caveats)

Checklist

• Test coverage for new or modified code paths
• For new Clarity features or consensus changes, add property tests (see docs/property-testing.md)
• Changelog is updated
• Required documentation changes (e.g., rpc/openapi.yaml for RPC endpoints, event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo

[coveralls]

Coverage Report for CI Build 24402514996

Coverage decreased (-0.06%) to 85.655%

Details

• Coverage decreased (-0.06%) from the base build.
• Patch coverage: 33 uncovered changes across 3 files (153 of 186 lines covered, 82.26%).
• 1945 coverage regressions across 65 files.

Uncovered Changes

File	Changed	Covered	%
clarity/src/vm/mod.rs	118	101	85.59%
clarity-types/src/types/mod.rs	24	16	66.67%
clarity/src/vm/functions/sequences.rs	44	36	81.82%

Coverage Regressions

1945 previously-covered lines in 65 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
stackslib/src/net/inv/epoch2x.rs	207	80.24%
stackslib/src/net/chat.rs	200	92.93%
stacks-node/src/nakamoto_node/miner.rs	134	87.08%
stackslib/src/net/api/postblock_proposal.rs	108	82.02%
stackslib/src/clarity_vm/database/marf.rs	99	60.67%
clarity/src/vm/contexts.rs	75	92.36%
clarity/src/vm/functions/database.rs	75	92.36%
clarity-types/src/types/mod.rs	71	90.34%
stackslib/src/net/codec.rs	69	91.63%
stackslib/src/clarity_vm/database/ephemeral.rs	61	56.82%

---

Coverage Stats

Relevant Lines:	217964
Covered Lines:	186698
Line Coverage:	85.66%
Coverage Strength:	17091394.25 hits per line

---

💛 - Coveralls

[jcnelson]

LGTM, as long as we leave this comment unaddressed (since it can be handled in a follow-up PR #7068): https://github.com/stacks-network/stacks-core/pull/7022/changes#r2980910603

[benjamin-stacks]

LGTM

[cylewitruk-stacks]

LGTM