OUT-3584: stream Dropbox→Assembly uploads to prevent OOM crash#97
OUT-3584: stream Dropbox→Assembly uploads to prevent OOM crash#97SandipBajracharya merged 2 commits intomainfrom
Conversation
The sync-dropbox-file-to-assembly task was OOMing on video files because
`uploadFileInAssembly` called `filesDownload` solely to read file size —
the Dropbox SDK buffers the entire file into a Buffer on that call, then
the bytes were discarded. Actual upload already streamed via a separate
path, so memory was wasted.
- Remove the buffering `filesDownload` call.
- `DropboxClient._downloadFile` now returns `{ body, contentLength }`,
sourcing Content-Length from the download response headers so it
always matches the streamed bytes (avoids S3 PUT mismatch on file
types where listing-time size diverges from download size).
- Add explicit access-token refresh in `_downloadFile`. The removed
`filesDownload` SDK call was implicitly refreshing the token; without
it the manual fetch ran with an unpopulated Bearer and Dropbox 400'd.
- Guard resync path against non-file entries (folder/deleted) that
could reach the upload path after a Dropbox rename/delete-and-recreate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mirrors the _downloadFile hardening from the previous commit. Previously _uploadFile relied on an implicit token refresh from a preceding SDK call (filesGetMetadata, filesMoveV2, filesCreateFolderV2) in the Assembly→Dropbox path. That contract was fragile: any future caller reaching _uploadFile without a prior SDK call would 400 on an unpopulated Bearer — the same regression we just shipped in _downloadFile. Make the refresh explicit so the contract lives with the method, not the call site. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
priosshrsth
left a comment
priosshrsth
left a comment
There was a problem hiding this comment.
@SandipBajracharya PR looks good. And this is a good optimization as well. Excellent job. I have added one question. But the pr has been approved.
| throw new Error( | ||
| `DropboxClient#downloadFile. Missing Content-Length header for file: ${filePath}`, | ||
| ) |
There was a problem hiding this comment.
@SandipBajracharya Do we actually expect this to be called or is it there just for type safety? And if this can be a legit cause, is it critical enough to not sync the file? Could you provide more context on what happens when we actually get this error?
Will it be marked as failed? retried again??
There was a problem hiding this comment.
When we get this error, it is retried 3 times by syncDropboxFilesToAssembly task before it completely fails. On failure we get reported via Sentry and we've also set up slack alerts. The failure of single task does not impact the overall sync. We have a trigger.dev task: resyncFailedFilesInAssembly that tries to resync those failed records.
SandipBajracharya
changed the title
2 participants
Summary
sync-dropbox-file-to-assemblyTrigger.dev task was OOMing on video files. Root cause:uploadFileInAssemblycalledfilesDownloadpurely to read file size, and the Dropbox SDK buffers the entire file into aBufferon that call. For multi-GB videos, this blew past the machine's memory limit.small-2x— no bump needed.Content-Lengthfor the Assembly PUT is sourced from the Dropbox download response headers, guaranteeing it matches the streamed bytes (avoids S3 Content-Length mismatch on file types where listing-timesizediverges from download size).DropboxClient._downloadFileand_uploadFile. The previous code relied on an implicit refresh as a side effect of preceding SDK calls (filesDownload,filesMoveV2, etc.). RemovingfilesDownloadexposed the regression on the download path (Dropbox 400 on unpopulated Bearer). Hardening_uploadFilewith the same pattern eliminates the latent risk on the Assembly→Dropbox path.['.tag'] === 'file'guard in the resync helper so folder/deleted entries never reach the upload path after a Dropbox rename/delete-and-recreate.Test plan
sync-dropbox-file-to-assemblytask completes without OOM and the file lands in Assembly with correct bytes._uploadFile's explicit refresh doesn't regress the existing path.small-2x's 1 GB limit.Testing Criteria
https://www.loom.com/share/ac57be85f2784e02ac7d5ead583ae965
Screenshots
🤖 Generated with Claude Code