Add fallback for S3 error code 416

[matthdsm]

Gemini generated fix for #6888, most changes are automated formatting though.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	91d4779
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69bbd6e428afe4000842f66c

[pditommaso]

Thanks for submitting this. Please remove code reformatting that introduces noise in the commit history

[matthdsm]

Cleaned up the commit!

[bentsherman]

@jorgee can you look into this? I wonder if there is a simpler solution than adding a fallback

[jorgee]

@matthdsm, I am trying to replicate the error of the issue with a pipeline that downloads an s3 directory with an empty files. But it is not failing. So, there is no general problem with empty files. I would like to know more about your case, could you provide more details (full log and config) or a MRE?

I was trying to see if there is an issue in S3 CRT with empty files and I have only seen an issue when using minio minio/minio#19648. Is it your case?

Range downloads are used in multipart, so another option is that the file was not really empty when the download was started, and there has been a concurrent modification during the download. The message with 'size=0' in the logs is for the directory, not the file. Is it possible that another process is modifying the file 's3://test-data/genomics/homo_sapiens/illumina/bcl/CopyComplete.txt' during the download?

[matthdsm]

Hi @jorgee ,

The test-data bucket is on an on-prem S3 compatible storage farm from Dell, so the issue might be the same from MinIO.
I can confirm the file was indeed empty and unused by anything else at the time of testing.

[jorgee]

@matthdsm thank you for the feedback.

Could you check if copying to another s3 bucket/folder in your storage is also getting errors with empty files or is it just downloads? you can do a test like the following:

nextflow cp 's3://test-data/genomics/homo_sapiens/illumina/bcl' 's3://test-data/genomics/homo_sapiens/illumina/bcl-copy

If it is just when downloading, it is passing the size to the download method, it could just skip the download call to the API when size is 0 and create an empty destination file.

[matthdsm]

Just checked the following cmds:
Upload seems to work just fine

nextflow -C s3_ugent.config fs cp empty.txt s3://test-data/genomics/homo_sapiens/illumina/bcl/  

Copy S3 -> S3 just hangs indefinitely

nextflow -C s3_ugent.config fs cp s3://test-data/genomics/homo_sapiens/illumina/bcl/empty.txt s3://test-data/genomics/homo_sapiens/illumina/bcl/empty2.txt
``

[jorgee]

@matthdsm, Could you check if you have defined any of these env variables in your environment?

AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED

I only was able to reproduce the error with an old Minio version using these variables that disables checksum validation

I have also created an alternative in PR #6944, that creates an empty file instead of the fallback. Could you check if it also fixes your issue?

[matthdsm]

Hi @jorgee,

Yes, the checksums are disabled, since those are not compatible with the DELL platform

[bentsherman]

Closing in favor of #6944

@matthdsm let us know if the other PR solves the issue for you 😄

[matthdsm]

Works for me! Thank you all 😄