Skip to content

Support compressed virtual chunk locations (needs fzstd dictionary support) #3

@Shane98c

Description

@Shane98c

icechunk (earth-mover/icechunk#1776) introduced dictionary-compressed virtual chunk locations. The manifest FlatBuffer schema has location_dictionary, compression_algorithm, and compressed_location fields that we don't read yet.

Current state

What's needed

  • Wait for fzstd#18 to land and release
  • Update manifest-parser.ts to read compressedLocation from ChunkRef
  • Update manifest-parser.ts to read locationDictionary/compressionAlgorithm from Manifest
  • Decompress locations at parse time (matching Rust impl), priority: compressed_location → location → chunk_id → inline
  • Add tests (will need synthetic fixtures or wait for upstream data)

References

  • Upstream schema: manifest.fbsChunkRef.compressed_location, Manifest.location_dictionary, Manifest.compression_algorithm

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions