Why do I get OutOfDeviceMemory for 2Gb safetensors model on 4Gb GPU with --offload-to-cpu enabled?

[alex-ie]

Edit 2:

Now when re-running koboldcpp I do (via script): 1) end w-browser (ff) 2) end kobold 3) clear ~/.cache, ~/.mozilla, /tmp.
And sometimes for same image parameters (10 steps, 512x512) I see image generated (once even w/out "offload to CPU"), sometimes OutOfDeviceMemory. I'm puzzled. Maybe it's some system glitches, I see also "Firefox: unable to connect to ibus" and keyboard do not work in Firefox and sometimes in other apps (file manager), I've restarted Window Manager (Cinnamon) - keyboard started to work fine then soon - again not everywhere.

Edit:

Seems the issue below is at least partly caused by data from previous koboldcpp run(s). After I've deleted files from /tmp and user's cash I didn't see the issue (I probably have to do that after each program exit - how to exit properly? I use Ctlr-C or pkill).

--

I'm new to AI generation, koboldcpp works on my laptop in CPU mode more or less, I'm learning. Now I try to use my NVIDIA 4Gb GPU via vulkan option. For TTS in GPU option with 200Mb model I saw no difference in CPU loads in vulkan vs. CPU option. Does in use GPU...?

Now trying image gen, initially I've noted almost no CPU load with vulkan, but then noted that in several seconds in terminal koboldcpp exits with 'OutOfDeviceMemory'. Why? I run in terminal:

koboldcpp-linux-x64 --usevulkan 1 --sdoffloadcpu --launch --showgui --sdmodel /path/Deliberate_v2.safetensors, then in GUI check correctness on "Image Gen" tab, e.g. offload cpu checkmark then launch, simple prompt for generation, see image placeholder, but in in terminal error after several seconds.

https://huggingface.co/leejet/Z-Image-Turbo-GGUF gives an example of 3.1Gb model run on 4Gb:

sd.exe --diffusion-model z_image_turbo-Q3_K.gguf --vae ae.safetensors --llm Qwen3-4B-Instruct-2507-Q4_K_M.gguf -p "...photograph..." --cfg-scale 1.0 -v --offload-to-cpu --diffusion-fa -H 1024 -W 512

And with models from above code ( z_image_turbo etc., though I use z_image_turbo-Q4 3.7Gb) I'm now getting "DeviceLostError".

[LostRuins]

Hi, please add --sdquant to your launch args. Then it will work

[wbruna]

One possible reason for the failure is that some Vulkan devices have fairly low limits for single allocations (sometimes as low as 2G), even if there is plenty of total memory available. See if --sdconvdirect vaeonly (or --sdconvdirect full) helps; it lower memory usage, especially during the VAE phase.

[LostRuins]

If it's oom on vae phase, try add --sdvaeauto

[alex-ie]

One possible reason for the failure is that some Vulkan devices have fairly low limits for single allocations (sometimes as low as 2G), even if there is plenty of total memory available. See if --sdconvdirect vaeonly (or --sdconvdirect full) helps; it lower memory usage, especially during the VAE phase.

@wbruna , thanks. I've checked vulkan output and indeed single allocations is 2Gb. Is it software issue (version of Mesa and might be fixed in future releases)? I guess if I succeed in installing CUDA it does not have such limit or does it? BTW I've tried to install CUDA today and failed again. Could another AI "engine" (not koboldcpp or future versions of koboldcpp) allocate memory is smaller chunks for same task as now koboldcpp try to allocate in one go? TIA

[LostRuins]

Try using the embedded vae with --sdvaeauto

[wbruna]

Is it software issue (version of Mesa and might be fixed in future releases)?

It could be. On my card, radv has a 4G limit, while for amdvlk is 2G. I don't know about CUDA, but I believe ROCm on my card also has a limit of around 8G (16G card).

Could another AI "engine" (not koboldcpp or future versions of koboldcpp) allocate memory is smaller chunks for same task as now koboldcpp try to allocate in one go? TIA

Other approaches could have different requirements, yes. That's the case of that --sdconvdirect vaeonly option I've mentioned: it uses an alternative approach that consumes much less memory (it's not always a benefit, though: in some backends it can be much slower). Lowering the VAE tiling limit with --sdtiledvae would be another option; or --sdvaeauto to use a less-demanding VAE, as LostRuins mentioned.