Skip to content

MINIFICPP-2792 Fix S3 upload above 2GB on Windows#2167

Open
lordgamez wants to merge 3 commits intoapache:mainfrom
lordgamez:MINIFICPP-2792
Open

MINIFICPP-2792 Fix S3 upload above 2GB on Windows#2167
lordgamez wants to merge 3 commits intoapache:mainfrom
lordgamez:MINIFICPP-2792

Conversation

@lordgamez
Copy link
Copy Markdown
Contributor

  • Set content length to avoid narrowing size to int on Windows
  • Add MinifiToAwsInputStream to wrap MiNiFi InputStream and avoid stream copy

https://issues.apache.org/jira/browse/MINIFICPP-2792


Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically main)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file?
  • If applicable, have you updated the NOTICE file?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

@lordgamez lordgamez force-pushed the MINIFICPP-2792 branch 2 times, most recently from 1808a2e to 091f599 Compare May 4, 2026 10:08
@lordgamez lordgamez added priority Review these first bug-fix labels May 4, 2026
@lordgamez lordgamez marked this pull request as ready for review May 4, 2026 15:00
@martinzink martinzink requested a review from Copilot May 5, 2026 08:41
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the AWS S3 upload path to avoid copying FlowFile content into an intermediate Aws::StringStream, and to explicitly set request content length so large uploads work correctly on Windows. In the MiNiFi AWS extension, that means PutS3Object now streams directly from MiNiFi io::InputStream into the AWS SDK request bodies.

Changes:

  • Replace the old copy-based readFlowFileStream helper with a new MinifiToAwsInputStream adapter around io::InputStream.
  • Explicitly set ContentLength on both single-part and multipart S3 upload requests.
  • Update S3 test mocks/helpers to model body consumption more like the AWS SDK for multipart uploads.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
extensions/aws/tests/MockS3RequestSender.h Adjusts test mock behavior so multipart request bodies are consumed/read more like the real SDK.
extensions/aws/s3/S3Wrapper.h Removes the old helper declaration for copying FlowFile data into an AWS string stream.
extensions/aws/s3/S3Wrapper.cpp Switches S3 uploads to the new stream adapter and sets request content lengths explicitly.
extensions/aws/s3/MinifiToAwsInputStream.h Adds a new streambuf/iostream adapter that exposes MiNiFi io::InputStream as an AWS-compatible input stream.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread extensions/aws/s3/MinifiToAwsInputStream.h Outdated
Comment thread extensions/aws/s3/S3Wrapper.cpp
lordgamez added 3 commits May 7, 2026 11:39
- Set content length to avoid narrowing size to int on Windows
- Add MinifiToAwsInputStream to wrap MiNiFi InputStream and avoid stream
  copy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix priority Review these first

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants