Add safetensors BF16 indexing pipeline and reasoning scaffold analysis

[AdaWorldAPI]

Summary

This PR adds comprehensive infrastructure for analyzing reasoning distillation across Qwen3.5 models by implementing a safetensors BF16 indexing pipeline and multi-model causal diff analysis framework.

Key Changes

• Projection classification fix: Extended classify_projection() to recognize both "embed" and "embd" naming conventions for embedding layers, improving tensor matching accuracy across different model variants.

• Safetensors BF16 indexing pipeline: Added complete test infrastructure for streaming and indexing 5 Qwen3.5 models (27B base, 27B v1, 27B v2, 9B base, 9B distilled) from Hugging Face, with HTTP range-based chunked downloads and BF16 compression.

• Sharded causal diff analysis: Implemented causal_diff_sharded() to perform cross-model weight space analysis across multiple safetensors shards, aggregating statistics and causal edges while handling missing or incomplete shard files gracefully.

• Reasoning scaffold detection: Added multi-stage analysis pipeline including:

  • Scale-invariant scaffold identification (blocks present in both 27B and 9B models)
  • Capacity-dependent reasoning detection (27B-only blocks)
  • Convergence tracking across distillation iterations (v1→v2)
  • NARS-based truth revision across all 4 diffs with integrated frequency/confidence metrics

• Comprehensive test suite: Added 5 per-model indexing tests and 1 full pipeline test (test_qwen35_claude_reasoning_diff) that:

  • Performs 4 sequential diffs (27B base→v1, base→v2, v1→v2, 9B base→distilled)
  • Generates markdown report with model matrix, diff summaries, scaffold analysis, and top 20 shifted attention heads
  • Writes results to knowledge base for downstream analysis

Notable Implementation Details

• HTTP range reader with 256MB chunks for efficient streaming of multi-GB safetensors files
• Automatic content-length detection via curl HEAD requests
• Per-shard progress reporting with tensor counts, shift percentages, and compression ratios
• Aggregated statistics across shards with weighted averaging for per-projection metrics
• NARS integration for multi-evidence truth revision across heterogeneous diffs
• All tests marked #[ignore] with data size annotations (55GB for 27B, 18GB for 9B)

https://claude.ai/code/session_01M3at4EuHVvQ8S95mSnKgtK