Fix multipart direct upload buffering chunks in memory #52
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use aiofiles' AsyncBufferedReader to stream file data instead of loading each chunk into memory. Add DEFAULT_CHUNK_SIZE and typing Union/AsyncBufferedReader imports. _chunked_upload now reads and uploads slices directly from the file, passing chunk_size to _upload_chunk. _upload_chunk accepts either BytesIO or AsyncBufferedReader and uses chunk_size to set Content-length for streamed reads. upload_bytes now supports AsyncBufferedReader and enforces chunk_size limits while updating the hash function. These changes reduce memory usage and ensure correct chunk sizing for multipart uploads.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request refactors the chunked file upload process to improve efficiency and flexibility, especially when handling large files. The main changes involve switching from loading entire chunks into memory to streaming file data directly from disk, supporting both in-memory and asynchronous file readers, and making chunk size handling more robust.