False cache hits leading to corrupt downloads

[jeroen]

A user of the r bindings reported weird behavior when downloading a file https://openml.org/data/v1/download/16797613/CIFAR_10.arff

After some testing I think I see the bug: you are using a server-side cache which seems to not understand the Vary: Accept-Encoding header that you use to key the cache.

As a result, the cache server is sending cached responses that are compressed with gzip or brotli to clients that do not support it, or at least did not request this in the Accept-Encoding request header, and hence are not decompressing it.

I confirmed the problem goes away when you add some random ?nocache=1234 parameter to the URL to force bypassing the cache.

[PGijsbers]

Thanks for the report, Jeroen! We'll investigate.

[Siddhartha-singh01]

Hey! @jeroen @PGijsbers I looked into this and I think I found where the problem is.

The proxy route that serves file downloads (app-next/src/app/api/proxy-file/route.ts)
doesn't pass the client's Accept-Encoding header to the upstream server, so the upstream can return gzip-compressed data even when the client never asked for it. On top of that, the responses don't include Vary: Accept-Encoding, which means caches have no way of knowing they should store different versions for different clients. And the Content-Encoding header from upstream isn't forwarded either, so the client doesn't even know the data is compressed.

I'm going to fix this by forwarding Accept-Encoding to upstream, adding Vary: Accept-Encoding on the proxy responses, and passing through Content-Encoding when it's present. Will open a PR soon.