[filesystems] Update S3 multipart upload for Hadoop 3.4+ (AWS SDK v2) #7187
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tub
changed the title
This was referenced Feb 2, 2026
Closed
tub
force-pushed
the
hadoop-3.4-upgrade
branch
2 times, most recently
from
ac02b55 to
a6de06a
Compare
Hadoop 3.4.x switched hadoop-aws from AWS SDK v1 to v2 (HADOOP-18073). This updates the S3 multipart upload implementation to use the new SDK types: - CompletedPart instead of PartETag - CompleteMultipartUploadResponse instead of CompleteMultipartUploadResult - UploadPartRequest/UploadPartResponse for part uploads Also updates NOTICE files for the new Hadoop 3.4.2 dependencies. Co-Authored-By: Claude <noreply@anthropic.com>
tub
force-pushed
the
hadoop-3.4-upgrade
branch
from
a6de06a to
641b724
Compare
Hadoop 3.4+ uses the pjfanning fork of jersey-json (com.github.pjfanning:jersey-json) instead of the original (com.sun.jersey:jersey-json). This fork contains properties files with GPL license references that fail Apache license checks. Add exclusions for the pjfanning jersey-json fork to: - paimon-hadoop-shaded (fixes OSS, OBS, COSN modules) - paimon-gs-impl (uses direct hadoop-common dependency) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
|
@JingsongLi I'm prepping a PR based on this one that will allow users to have multiple processes writing to the same table without having to use an external catalog/metastore for S3 and Azure. I'd really appreciate it if you could help me find a reviewer 🙇 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Hadoop 3.4.x switched hadoop-aws from AWS SDK v1 to v2 (HADOOP-18073).
This updates the S3 multipart upload implementation to use the new SDK types:
This is a step towards implementing conditional writes for AWS S3, Azure and others. I'm preparing a PR for this as a follow-up once this is merged.
Also updates NOTICE files for the new Hadoop 3.4.2 dependencies.
Linked issue: #6563
Note
Although many of the changes were generated using Claude code, they've been manually verified locally and against an actual S3 bucket.
Tests
Existing S3 and Azure tests cover the changes.
API and Format
No
Documentation
No