feat(to): re-add to parquet subcommand powered by Polars#3674
Merged
jqnatividad merged 9 commits intomasterfrom Apr 3, 2026
Merged
feat(to): re-add to parquet subcommand powered by Polars#3674jqnatividad merged 9 commits intomasterfrom
to parquet subcommand powered by Polars#3674jqnatividad merged 9 commits intomasterfrom
Conversation
The `to parquet` subcommand was removed in 16.0.0 due to a heavy dependency on an old arrow crate. This re-adds it using Polars directly (requires the `polars` feature), supporting compression codecs (zstd/gzip/snappy/lz4raw/uncompressed) and all existing input modes (files, directories, infile-lists, stdin). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 35 |
TIP This summary will be updated as you push new changes. Give us feedback
Contributor
There was a problem hiding this comment.
Pull request overview
Re-introduces the qsv to parquet subcommand (previously removed) by implementing Parquet writing via Polars, including compression options and support for the existing “many inputs” modes (files/dirs/infile-lists/stdin).
Changes:
- Add
to parquetsubcommand plumbing, CLI help text, and Polars-based Parquet writer implementation (with compression flags). - Update top-level command listing to mention Parquet under
to. - Add integration tests for basic parquet generation, multiple inputs, compression flag, custom
--tablename, and directory inputs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
src/cmd/to.rs |
Adds parquet subcommand docs/args and implements CSV→Parquet conversion using Polars. |
src/main.rs |
Updates the to command description in the command list to include Parquet. |
tests/test_to.rs |
Adds #[cfg(feature = "polars")] tests exercising the new to parquet behavior. |
Fix XOR bug (768^2→768*768), add Windows-safe --table validation, fix empty-stdin error message, implement --all-strings support, add output collision detection, clarify usage placeholder and help text, conditionally show Parquet in main help, and improve test assertions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Revert <parquet_dir> back to <parquet> — docopt couldn't map the underscore placeholder to the struct field. Fix scan_parquet call to use PlRefPath as required by Polars API. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Validate gzip compression level (1-9) before casting to u8, use IncorrectUsage error for invalid --compression values, and enhance snappy compression test to verify parquet contents. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add range validation (1-22) for zstd compression level to match the existing gzip validation pattern, producing a friendly error message instead of an opaque library error. Add tests for: invalid compression codec, out-of-range gzip level, and out-of-range zstd level. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…e/qsv into to-parquet-polars-powered
[skip ci]
…p description - Change "table/sheet/filename name" to "table/sheet/file name" in --table option docs - Restore missing Parquet hyperlink in sqlp description in README for consistency Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
to parquetsubcommand was removed in 16.0.0 due to a heavy dependency on an old arrow crate. This re-adds it using Polars directly (requires thepolarsfeature), supporting compression codecs (zstd/gzip/snappy/lz4raw/uncompressed) and all existing input modes (files, directories, infile-lists, stdin).