Skip to content

Add Vulkan rendering backend#7233

Open
laanwj wants to merge 41 commits intoscp-fs2open:masterfrom
laanwj:vulkan-pr
Open

Add Vulkan rendering backend#7233
laanwj wants to merge 41 commits intoscp-fs2open:masterfrom
laanwj:vulkan-pr

Conversation

@laanwj
Copy link
Contributor

@laanwj laanwj commented Feb 16, 2026

Implement a Vulkan 1.1 renderer that replaces the previous stub with a fully functional backend, mostly matching the OpenGL backend's rendering capabilities. The game should be playable with minimal divergence from OpenGL rendering.

This is, most likely, too big to go in all at once, but just filing it here for reference because it's reached a testable state.

Core rendering infrastructure. The code lives under code/graphics/vulkan:

  • VulkanMemory: Custom allocator with sub-allocation from device-local and host-visible memory pools
  • VulkanBuffer: Per-frame bump allocator for streaming uniform/vertex/index data (persistently mapped, double-buffered, auto-growing)
  • VulkanTexture: Full texture management including 2D, 2D-array, 3D, and cubemap types with automatic mipmap generation and sampler caching
  • VulkanPipeline: Lazy pipeline creation from hashed render state, with persistent VkPipelineCache
  • VulkanShader: SPIR-V shader loading (main, deferred, effects, post-processing, shadows, decals, fog, MSAA resolve, etc.)
  • VulkanDescriptorManager: 3-set descriptor layout (Global/Material/PerDraw) with per-frame pool allocation, auto-grow, and batched updates
  • VulkanDeletionQueue: Deferred resource destruction synchronized to frame-in-flight fences

Design choices:

  • Two frames in flight with fence-based synchronization
  • Asynchronous texture upload, no waitIdle or other CPU-on-GPU blocking in hot path
  • Single command buffer per frame; render passes begun/ended as needed for the multi-pass deferred pipeline
  • Per-frame descriptor pools
  • All descriptor bindings pre-initialized with fallback resources (zero UBO + 1x1 white texture) so partial updates never leave undefined state
  • Streaming data (such as immediates) uses a bump allocator (one large VkBuffer per frame)
  • Pipeline cache persisted to disk for fast startup on subsequent runs

Some notable Vulkan vs OpenGL differences are:

  • Because shaders are pre-compiled to SPIR-V, shader variants are less feasible in Vulkan. Preprocessing directives have been converted to run-time uniform based branching.
  • Depth range is [0,1] not [-1,1]: shadow projection matrices adjusted, shaders that linearize depth need isinf/zero guards at depth boundaries where OpenGL gives finite values
  • Vulkan render target is "upside down", y-flip for render target is handled through negative viewport height, as is common
  • gl_ClipDistance is always evaluated: must write 1.0 when clipping is disabled (OpenGL allows leaving it uninitialized)
  • Texture addressing for AABITMAP/INTERFACE/CUBEMAP forced to clamp (OpenGL's sampler state happens to do this implicitly)
  • Render pass architecture requires explicit transitions between G-buffer, shadow, decal, light accumulation, fog, and post-processing passes (OpenGL just switches FBO bindings)
  • No geometry shaders. They're possible with Vulkan, but less common. Currently they're not used.

Preparation patches to common game code (these commits need to go in first):

  • Extract sphere and cylinder mesh generation into shared graphics utility: Needed in both GL and Vulkan
  • Route ImGui calls through gr_screen function pointers: Makes it possible for the Vulkan backend to provide its own ImGui implementation
  • Free bitmaps before destroying graphics backend: Fix shutdown order issue
  • Use float shader input instead of SCREEN_POS in gr_flash_internal: Compatibilty with Vulkan shaders
  • Remove now-unused SCREEN_POS vertex format: Cleanup after previous commit
  • Add dds_block_size and dds_compressed_mip_size utilities: Factor out utility code to be used in Vulkan backend
  • Add CAPABILITY_QUERIES_REUSABLE for GPU queries: Vulkan needs different lifecycle for GPU queries
  • Fix gr_flip debug output ordering: Prevent immediate buffer from being overwritten
  • Fix gr_end_2d_matrix viewport for render-to-texture: Fix RTT for Vulkan
  • Fix undefined gl_ClipDistance and use uint for std140 bool: Shader compatibility with Vulkan
  • Fix shader build MAIN_DEPENDENCY and add conditional GLSL/struct generation: Build system change for OpenGL/Vulkan shader split
  • Add missing memcpy_if_trivial_else_error for void *, const void*

What's possibly left to be done:

  • Unify OpenGL and Vulkan shaders where possible: the only shader shared with OpenGL (defined in the buid system's SHADERS_GL_SHARED) is still the default material. Although the Vulkan backend does some things differently, it would definitely be possible to share more code. But i didn't want to accidentally break OpenGL in some way.

  • Integrate VMA (Vulkan Memory Allocator). Some of the memory handling could be simplified by importing this dependency.

  • OpenXR anything. This is currently not implemented at all.

Build steps:

cmake -B build -DCMAKE_BUILD_TYPE=Debug -DFSO_BUILD_WITH_VULKAN=ON -DFSO_BUILD_WITH_OPENXR=OFF
cmake --build build

To run (with maximum debugging and Vulkan layer validation):

export VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation
export VK_LOADER_DEBUG=all

build/bin/fs2_open_25_1_0_x64_AVX2-DEBUG -vulkan -gr_debug -stdout_log -profile_frame_time

Full disclosure: i used Claude Opus 4.6 while developing this. However, the overall direction and design is my own, and i've paid careful attention to the code.

@BMagnu
Copy link
Member

BMagnu commented Feb 17, 2026

Thanks for the PR!
I'll be looking at it and playing around with it soon.
Please be aware, this being as big as it is, that it might be a while until we get through it.

@BMagnu
Copy link
Member

BMagnu commented Feb 17, 2026

Okay, played around with it a little.
Got it to run, though with a slew of visual artifacts and some crashes on some mods.
Still, a great first step to see it running in vulkan, at quite impressive performance numbers.
I'd love to discuss some of the design decisions in more detail. Are you on the discord, or somewhere else sensible for extended discussion?

@The-E
Copy link
Member

The-E commented Feb 19, 2026

I played around with it a little bit as well, and using nSight I could at least get as far as seeing that rendering a background with a skybox causes some amount of corruption to get into the main framebuffer - I haven't yet been able to see where it's coming from (as rendering a skybox should be one of the simpler things, just some basic geo and a couple textures, no lighting), but, well, there it is.

@laanwj
Copy link
Contributor Author

laanwj commented Feb 21, 2026

Are you on the discord, or somewhere else sensible for extended discussion?

i'm on the hard-light discord server i'm "mara" there. i'm not very active on discord, but happy to discuss.

Got it to run, though with a slew of visual artifacts and some crashes on some mods.

To be honest i've only been replaying the retail campaign with it. So code paths not exercised there will be less (or even not) tested. Please let me know which mods this happens with!

I played around with it a little bit as well, and using nSight I could at least get as far as seeing that rendering a background with a skybox causes some amount of corruption to get into the main framebuffer - I haven't yet been able to see where it's coming from (as rendering a skybox should be one of the simpler things, just some basic geo and a couple textures, no lighting), but, well, there it is.

Thanks for trying. i've installed NVidia Coresight but couldn't get it to report any issues in the level i tried (and i'd been using Vulkan's validation layer as well as RenderDoc during development and it should be clean). Can you send me the level file this happens with? And the messages that i should watch for?

@The-E
Copy link
Member

The-E commented Feb 21, 2026

Thanks for trying. i've installed NVidia Coresight but couldn't get it to report any issues in the level i tried (and i'd been using Vulkan's validation layer as well as RenderDoc during development and it should be clean). Can you send me the level file this happens with? And the messages that i should watch for?

My testing was done using the latest mediaVPs mod, running both the first mission and using the lab environment.
image

RenderDoc capture available here: https://drive.google.com/file/d/1ficdGUP-e8xfmUjZWAzWZmwpa9aAtFmt/view?usp=drive_link

@BMagnu
Copy link
Member

BMagnu commented Feb 21, 2026

I was similarly running the MediaVPs' first mission (where I got the artifacts), and I was testing the "Icarus" Cutscene from Blue Planet (which crashes on trying to render the opening movie, skippable with -nomovies)

@Shivansps
Copy link
Contributor

Im not in any position to ask, but instead of getting the vulkan lib and headers currently installed in the host system, maybe its better to use a glad2 loader for vulkan in the same way as it is for OpenGL?

@SamuelCho
Copy link
Contributor

Wow, nice work. Pretty straightforward design, nothing surprising. I have a local WIP DX12 implementation I've been working on and off on in my spare time for general practice and I see a lot of similar decisions you've made here in your VK implementation.

I kind of wonder if we need to double buffer the immediate buffer so that we leave alone the one that's in-flight. But maybe it doesn't matter if the fence in the command buffer submission and flip takes care of everything. Or is it the buffer manager that keeps track of the frame num?

Surprised that my batching code made it out intact. Also surprised that my render primitives immediate code also made it out intact. Sorry if it caused any headaches.

@Shivansps
Copy link
Contributor

Shivansps commented Feb 28, 2026

Ill put this in here for reference in case anyone is interested.

i did tried to see if i can change it to use the glad2 loader instead, as i expected since it is using vulkan.hpp, it is using the Vulkan C++ bindings, glad 2 loader has the C bindings, in the exact same way as with the version OpenGL. So its not a huge amount of work to change it, but it is still considerable work to change all bindings. (like 2-3 days). It is some work just to get it compile again not knkwing it is going to still work after that.

I also got the current PR version to compile for android by just adding the missing .hpp vulkan headers to the Android NDK, not elegant as im adding stuff to the toolchain but, it will do for now. Buuuuut it does not compile for 32 bits (x86/arm32), but it does for x86_64/arm64, not sure if this also the case for regular builds

On my phone with a Mali-G57
fs2_open.log
Crashes during shader compilation
0000000001911d04 /vendor/lib64/egl/libGLES_mali.so (cmpbe_v2_compile_multiple_shaders+2372) (BuildId: 747cc1a89e3838ab)
02-28 11:41:37.638 7140 7140 F DEBUG : Cause: null pointer dereference

On my Retroid G2 Handheld with a Qualcomm G2 and an Adreno 22 GPU, it fails to init vulkan because it lacks a transfer queue. I guess it is VK_QUEUE_TRANSFER_BIT? So its not completely 1.1 it uses an optional extension/feature.
fs2_open.log.txt

@laanwj
Copy link
Contributor Author

laanwj commented Mar 1, 2026

@The-E

My testing was done using the latest mediaVPs mod, running both the first mission and using the lab environment.

Thanks. The renderdoc capture should be helpful for reproduction.
(had to send a request to access it)

@SamuelCho

I kind of wonder if we need to double buffer the immediate buffer so that we leave alone the one that's in-flight. But maybe it doesn't matter if the fence in the command buffer submission and flip takes care of everything. Or is it the buffer manager that keeps track of the frame num?

It does. This is handled purely in the Vulkan layer. The buffer manager does a double buffering of all dynamic and streaming buffers in FrameBumpAllocator m_frameAllocs[MAX_FRAMES_IN_FLIGHT]. So it should have the same behavior as the GL backend with regard to orphaned buffers.
(It's also enforced that dynamic and streaming buffer content isn't reused between frames, by throwing a failure in that case)

Surprised that my batching code made it out intact. Also surprised that my render primitives immediate code also made it out intact. Sorry if it caused any headaches.

Hahah it wasn't too bad!

Im not in any position to ask, but instead of getting the vulkan lib and headers currently installed in the host system, maybe its better to use a glad2 loader for vulkan in the same way as it is for OpenGL?

Will look into it. It seems it would be way easier to vendor vulkan.hpp instead of switching to using C bindings, so i'll go for that first.

@laanwj
Copy link
Contributor Author

laanwj commented Mar 1, 2026

Okay. i've bundled the Vulkan and Vulkan-CPP headers in lib/vulkan-headers and updated the build system for this. Function loading was already happening dynamically through SDL, except for ImGui, which now does so too. i did not need to use glad2.

With this, it should be possible to build it on (or for) platforms without the Vulkan library and headers installed.

@laanwj
Copy link
Contributor Author

laanwj commented Mar 1, 2026

Trying to get it to pass the CI now. Will squash all these changes into the main (or otherwise original) commit when done.

@laanwj
Copy link
Contributor Author

laanwj commented Mar 1, 2026

i'm not happy where clang-tidy is taking some of these. It first wants to make these functions static (because it could), and now it want to refer to them by fully qualified class name instead of instance:

-	auto* texSlot = texManager->getTextureSlot(handle);
+	auto* texSlot = graphics::vulkan::VulkanTextureManager::getTextureSlot(handle);
-	drawManager->stencilClear();
+	graphics::vulkan::VulkanDrawManager::stencilClear();

Which is strictly correct but it's also less readable, and asymmetric with the rest of the API. Will see if (void)this works.

Edit: it did. Will look into rendering issues next.

@laanwj laanwj force-pushed the vulkan-pr branch 2 times, most recently from e5c9a34 to 61f890e Compare March 1, 2026 22:36
@GamingCity
Copy link

GamingCity commented Mar 2, 2026

Hi, Shivansps here, im on a diferent account, i think i know why it says there is no transfer queue on the adreno driver.

I think this if here is wrong
https://github.com/laanwj/fs2open.github.com/blob/61f890e2966bbce9650d94eee9249ce13cae864b/code/graphics/vulkan/VulkanRenderer.cpp#L104

if (!values.transferQueueIndex.initialized && queue.queueFlags & vk::QueueFlagBits::eTransfer) {
//False if no eTransfer (optional)
} else if (queue.queueFlags & vk::QueueFlagBits::eTransfer && !(queue.queueFlags & vk::QueueFlagBits::eGraphics)) {
//False if no eTransfer (optional)
}

Acording to the documentation
https://registry.khronos.org/VulkanSC/specs/1.0-extensions/man/html/VkQueueFlagBits.html

"All commands that are allowed on a queue that supports transfer operations are also allowed on a queue that supports either graphics or compute operations. Thus, if the capabilities of a queue family include VK_QUEUE_GRAPHICS_BIT or VK_QUEUE_COMPUTE_BIT, then reporting the VK_QUEUE_TRANSFER_BIT capability separately for that queue family is optional."

eGraphics (and eCompute) all include a transfer queue but may not report it.
So i think " & vk::QueueFlagBits::eTransfer" should be removed from the first if and assume it is. (and maybe make sure it is not eCompute? im not sure about that)

@laanwj
Copy link
Contributor Author

laanwj commented Mar 2, 2026

So i think " & vk::QueueFlagBits::eTransfer" should be removed from the first if and assume it is. (and maybe make sure it is not eCompute? im not sure about that)

Good catch. Yes, the logic there is wrong. "It worked on NVidia" 😊 Will fix.

Edit: Mind that the transfer queue is currently unused, as this makes the upload code simpler, due to there being no cross-queue synchronization requirement. In the current design there wouldn't be a benefit to using it, just overhead, as there's (AFAIK) no way to exploit parallelism here. So we could even decide to completely remove checking for it.

@laanwj
Copy link
Contributor Author

laanwj commented Mar 2, 2026

i've pushed a few rendering corruption fixes. Some wrong assumptions about renderpass state, and Vulkan vs GL differences. The cubemap corruption and random framebuffer noise should be solved now.

@Shivansps
Copy link
Contributor

Shivansps commented Mar 3, 2026

Just reporting back here, the change to the transfer queue selection did work. Now the Adreno GPU works and can get into the game.
The Mali GPU still crashes while compiling the default material shader, but no matter, ill guess that will be something to look at after the PR is merged.

@The-E
Copy link
Member

The-E commented Mar 3, 2026

Alright, your latest changes definitely fixed the framebuffer corruption, but I have more:
image
Not exactly sure what's going on here, but it seems there's something going weird when post processing is enabled: without post processing, the frame renders normally
One thing to examine would be wireframe rendering: I think there might be some options not being set correctly here.
image
Note that shutting off post processing in the lab also turns off imgui rendering.

Transparency is not rendered correctly
image

Particle and glowpoint blending modes are not set correctly:
image
Note the black halo around the glowpoint attached to the chin fin (or whatever that thing is called....)

textures appear to be downsampled in the lab:
image

I would also recommend running through the Blue Planet: War in Heaven intro - it shows a couple instances of textures rendered as pure white for some reason

@GamingCity
Copy link

Today i saw two things:
again, ill remember you android is not a working platform yet and ill work on the android PR after this and SDL3 is merged, so, i just mention things to keep track of it to see if can be fixed or it creates problems on other platforms. That said i dont know why 32bit CI does not complains about this and is only a problem on the ndk toolchain.

  1. I discovered why i was unable to compile 32 bits builds with the android-ndk, there is a mix of C and C++ types here:
    VulkanMemory.h
    struct VulkanAllocation {
    VkDeviceMemory memory = VK_NULL_HANDLE;
    VkDeviceSize offset = 0;
    VkDeviceSize size = 0;
    void* mappedPtr = nullptr; // Non-null if memory is mapped
    uint32_t memoryTypeIndex = 0;
    bool dedicated = false; // True if this is a dedicated allocation
    };

Changing to C++ types fixes 32 bit compilation

struct VulkanAllocation {
vk::DeviceMemory memory = VK_NULL_HANDLE;
vk::DeviceSize offset = 0;
vk::DeviceSize size = 0;
void* mappedPtr = nullptr; // Non-null if memory is mapped
uint32_t memoryTypeIndex = 0;
bool dedicated = false; // True if this is a dedicated allocation
};

Why this compiles its not going to work or it is going to have additional issues as VulkanPipeline.cpp has shifts to go out of range for 32 bit types.
shift warnings.txt

@BMagnu
Copy link
Member

BMagnu commented Mar 3, 2026

While not an immediate priority, I'd love to question the following design goal:
"Because shaders are pre-compiled to SPIR-V, shader variants are less feasible in Vulkan. Preprocessing directives have been converted to run-time uniform based branching."

Long / Medium term, I would like for FSO to ship with shadertool or something to allow it to compile to SPIR-V itself. This gets rid of a lot of issues here. First, we'd be able to keep text-based shaders that can be dual-use for OpenGL and Vulkan. Any incompatibilities can just be put in preprocessor blocks like main-f's prereplace, allowing full dual-use of all shaders. Furthermore, it'd allow table-able postprocessing and shader changes. While currently a full shader replace is necessary for custom shaders, I eventually want this to be properly modular, so being able to modify parts of shaders is a goal, and that for sure requires compilation on-the-fly.

Shipping with shadertool and then compiling on load (ideally after game-settings.tbl, especially since the recent Z-Compress changes) all available shaderfiles to SPRIV shouldn't be that hard either.

@laanwj
Copy link
Contributor Author

laanwj commented Mar 4, 2026

@Shivansps
Good!
i could in principle test Android + Adreno on my Ayn Thor. i don't have any device with a Mali GPU. But one thing at a time. i've never really done android development so it'll be some things to figure out.

@GamingCity
Ah yes, you're right. It's better to be consistent about using the C++ vulkan types instead of the C ones. Will switch it over. Though i'm very surprised that it makes a difference in practice.

@The-E
Thanks for the reports. At least the rendering issues are getting more subtle.

@BMagnu
Yes. i think it would be fine to make FSO depend on a GLSL-to-SPIR-V compiler library, and then do the compilation at runtime instead of compile-time. i can look into it.
i was just trying to be careful here to not introduce any big dependencies.
In principle, modular shaders can also be done with simpler SPIR-V level linking. But that'd be incompatible with the goal of unifying with the OpenGL backend.

@Shivansps
Copy link
Contributor

Please, I dont want to make you waste time on android testing, its not even a working platform yet. Ill post if i can find out something.

If you want to see i have a Fso_Android_Wrapper](https://github.com/Shivansps/Fso_Android_Wrapper) as the android test app, Fso-Android-Prebuilts were i have the script and instructions to build the fso dependencies and fso itself, and i have a "android-build-vulkan" branch on my fork where i added this pr to my previous android work,

I did found one problem with android on VulkanRenderer
createInfo.preTransform = deviceValues.surfaceCapabilities.currentTransform;

It seems that if you leave at that and use
SDL_SetHint(SDL_HINT_ORIENTATIONS, "LandscapeLeft LandscapeRight");
that im using to force landscape mode on android, it will (if i understood right) get a surface that is already rotated and then rotate it again, making it rander in portrait mode.

I changed it to this that did worked.

auto supported = deviceValues.surfaceCapabilities.supportedTransforms;
if (supported & vk::SurfaceTransformFlagBitsKHR::eIdentity) {
createInfo.preTransform = vk::SurfaceTransformFlagBitsKHR::eIdentity;
} else {
createInfo.preTransform = deviceValues.surfaceCapabilities.currentTransform;
}

I dont know if thats the right fix, it does not seems to do anything in windows. Keep in mind i used an AI to point me to this and the potential fix as i did not know if anything in vulkan could cause this, it told me to check where the preTransfor and surface capabilities are set for the transform and that i should use the eidentity flag.

https://docs.vulkan.org/refpages/latest/refpages/source/VkSurfaceTransformFlagBitsKHR.html

@BMagnu
Copy link
Member

BMagnu commented Mar 5, 2026

Fair enough re: compiling shaders and large dependencies, but I think it is worth here.
Even just having a unified backend to maintain (where the shader compiler likely needs little to no continued maintainance) is worth it alone IMO, but with tableable variants, it is for sure.

Implement a Vulkan 1.1 renderer that replaces the previous stub with a
fully functional backend, mostly matching the OpenGL backend's rendering
capabilities.

Core rendering infrastructure:

- `VulkanMemory`: Custom allocator with sub-allocation from device-local and
  host-visible memory pools
- `VulkanBuffer`: Per-frame bump allocator for streaming uniform/vertex/index
  data (persistently mapped, double-buffered, auto-growing)
- `VulkanTexture`: Full texture management including 2D, 2D-array, 3D, and
  cubemap types with automatic mipmap generation and sampler caching
- `VulkanPipeline`: Lazy pipeline creation from hashed render state, with
  persistent VkPipelineCache
- `VulkanShader`: GLSL shader loading. Shader code and metadata are
  shared with OpenGL, with differences guarded by preprocessor
  conditions
- `VulkanDescriptorManager`: 3-set descriptor layout (Global/Material/PerDraw)
  with per-frame pool allocation, auto-grow, and batched updates
- `VulkanDeletionQueue`: Deferred resource destruction synchronized to
  frame-in-flight fences

Design choices:

- Two frames in flight with fence-based synchronization
- Asynchronous texture upload, no `waitIdle` in hot path
- Single command buffer per frame; render passes begun/ended as needed
  for the multi-pass deferred pipeline
- Per-frame descriptor pools
- All descriptor bindings pre-initialized with fallback resources (zero
  UBO + 1x1 white texture) so partial updates never leave undefined state
- Streaming data uses a bump allocator (one large VkBuffer per frame)
- Pipeline cache persisted to disk for fast startup on subsequent runs
- Use VMA (Vulkan Memory Allocator) for buffer management

Some notable Vulkan vs OpenGL differences are:

- Depth range is [0,1] not [-1,1]: shadow projection matrices adjusted,
  shaders that linearize depth need isinf/zero guards at depth boundaries
  where OpenGL gives finite values
- In Vulkan, all shader outputs must be initialized. Leaving them
  uninitialized can result in random corruptions, while
  OpenGL allows leaving them in some cases
- Swap chain is B8G8R8A8: screenshot/save_screen paths swizzle to RGBA
- Vulkan render target is "upside down", y-flip for render target is
  handled through negative viewport height, as is common
- Texture addressing for AABITMAP/INTERFACE/CUBEMAP forced to clamp
  (OpenGL's sampler state happens to do this implicitly)
- Render pass architecture requires explicit transitions between G-buffer,
  shadow, decal, light accumulation, fog, and post-processing passes
  (OpenGL just switches FBO bindings)
laanwj added 17 commits March 11, 2026 09:58
Include shadows.sdr unconditionally in main-v.sdr and main-f.sdr
(was guarded by #ifndef VULKAN / #ifdef OPENGL). Add shadowUV[4]
and shadowPos varyings to Vulkan's VertexOutput. Add shadow_map
sampler to Vulkan's fragment declarations. Remove #ifndef VULKAN
guard around forward shadow getShadowValue() call. Unify shadow
depth write to use VARIANCE_SHADOW_SCALE_INV for both backends.
Write the real shadow map texture to Global Set 0 Binding 2 during
model draw calls (was always fallback). Enables forward-pass shadows
for the Vulkan backend.
Cloak effect: declare sFramebuffer (scene color copy) at Set 1
Binding 5 in the Vulkan model fragment shader. The texture was
already bound by VulkanDraw.

Lightshaft cockpit mask: declare cockpit sampler at texture array
element 1 in Vulkan lightshaft shader. Currently samples a white
fallback (no cockpit depth isolation yet), matching existing Vulkan
behavior but unifying the shader code.
Cleaner, and avoids accidentally leaving holes
This is implemented differently in different places. And was missing in
others.
Replace the bare texture view getters with ready-made structures.
…eBuffer overloads

Just build the structures as needed.
Add semantic constants for framebuffer formats.
@laanwj
Copy link
Contributor Author

laanwj commented Mar 11, 2026

Pushed some cleanups/refactors.
Also i've implemented texture cache flush in vulkan_bm_page_in_start, similar to OpenGL. GPU memory use should now grow less over time, when playing multiple missions.

laanwj added 6 commits March 11, 2026 23:19
Switch to a fully structured approach that prepares the write with
fallback buffers/textures in advance, using templates.
…add ArrayView

Also use ArrayView to pass texture array into DescriptorWriter.
We ended up duplicating this information.
@GamingCity
Copy link

GamingCity commented Mar 12, 2026

I have a hint of what could be causing the crash during default material shader compile on Mali. There is not really that much there that could cause a null pointer dereference.
I think it is the "gl_ClipDistance", this needs "shaderClipDistance" that is mandatory on desktop hardware, but optional on mobile hardware. Im not sure if this is being check on VkPhysicalDeviceFeatures.

And i know the GLES driver for this mali GPU do not support clipdistance, i had to code in an alternative in the gles shaders, so it might just be it does not have it either for Vulkan.

And just so happens, the Andreno 22 GPU does have gl_ClipDistance support in GLES and it does not crashes on shader compile. At least not at the start.

ill attach what i did for the gles shaders, i reeplaced the gl_clipdistance, but maybe something like this could be implemented based if the clipdistance support is there or not?
default-material.vert.txt
default-material.vert.spv.glsl.txt

UPDATE:
After testing by commenting all instances of gl_clipdistance in shaders, it now works on Mali. So yes, it is a case of a optional feature.

@notimaginative
Copy link
Contributor

Still have more testing to do but, particularly with the more recent changes, but I wanted to point out a few things I'd found earlier in better detail...

I'm not sure why the ci builds don't have this issue, but I had to add

#if defined(_WIN32)
#include <direct.h>
#endif

to gr_vulkan.cpp in order for it to compile on Windows due to the _mkdir() call.

Like with #7278 the imgui Init() and Shutdown() calls could really use some safe checks. The vulkan code handles that much better than the old FSO code does and is far less likely to have a problem here, but just for consistency sake: an Assertion() before init to check that an imgui context exists should be added to the top of VulkanRenderer::initImGui(), and a context&backendrenderer check added to VulkanRenderer::shutdownImGui() (or just confirm that initImGui() was successful).

If a shaderc library isn't available then it will pop up an error message and exit. This is expected, however it triggers an assert in the process. In VulkanMemory::shutdown(), the vmaDestroyAllocator(m_allocator) call:

Assertion failed: (false && "Unfreed dedicated allocations found!"), function ~VmaDedicatedAllocationList, file vk_mem_alloc.h, line 6512.

m_allocationCount is 1 when this happens.

If I run it in an address sanitizer then the mprintf() calls in the shutdown() functions of VulkanBufferManager, VulkanMemoryManager, VulkanTextureManager will trigger a memory error (heap-use-after-free, from filter_vector().push_back( new_filter )). This condition only appears to be set in motion once VulkanBufferManager has been initialized. If the init process fails at any time before m_bufferManager is initialized then it doesn't happen. It's obviously very strange, but may point to some memory bug in one of those three locations (Buffer/Memory/Texture Manager).

@notimaginative
Copy link
Contributor

notimaginative commented Mar 14, 2026

Here are some findings after more testing. I tested on macOS (M1 Max), a Linux VM (Parallels on Mac), and bare metal Linux with an Intel iGPU. I was not able to test on Windows as Vulkan is not supported in the VM there.

  1. Memory usage is substantially improved after recent updates. Still a bit higher than OpenGL on Mac, but definitely within the margin of error for the compressed texture handling difference. It could probably still be checked using a proper Windows machine to confirm usage differences between Vulkan and OpenGL, but I'll tentatively say that you've resolved that issue.

  2. Notable performance issues were noticed across all test systems. However, I was able to reproduce those same issues in OpenGL as well so it's probably safe to say that they're due to general FSO issues. Overall FPS, even in those performance dips, remained higher with Vulkan than OpenGL. I didn't have a test where the performance drops really affected playability when using Vulkan, whereas they did when using OpenGL.

  3. As noted in an earlier comment, wireframe rendering does not appear to work. Not in the lab and not in-game with the hud target view.

  4. Adding to that, lines are noticeably thicker using Vulkan compared to OpenGL. This is easily verified by loading the first briefing screen of the first non-training mission of retail (sm1-01). Jump nodes didn't appear to have this same issue. Line size also affects how stars look, the difference being more noticeable running retail without MVPS.

  5. Models are rendered slightly transparent. This only happens when post-processing is enabled and everything appears to render fine when it's disabled. It's easiest to notice in the lab, but is also readily seen when large ships are blowing up in a mission. It's possible this is contributing to strange performance degradation in certain missions (like asteroid missions) so I'll be skipping further performance tests until that's fixed. EDIT: I should note that I saw this with MVPS assets, but not the old retail models. This also may be related to lighting but that's just a wild guess based on how the transparency looks.

  6. There is some rendering glitch which appears to be related to shadows. I had shadow quality at medium (didn't test other quality settings), loaded "The Sicilian Defense" (sm2-05) in the sim room, entered the mission, then exited. Returning to the tech room the models don't render correctly. I didn't notice this behavior with other missions that I tested, but my tests only included 6 different missions.

  7. There are frequent rendering issues and what appears to be corrupted textures in the "massive battle2" mission. That mission is part of the multi-mission-pack.vp, even though it's a single-player test mission. I don't believe that the mission pack is available through Knossos but you should find a link to it on HLP's Multiplayer Getting Started Guide.

@GamingCity
Copy link

GamingCity commented Mar 17, 2026

Ok i did tried to adjust it myseft, using a a non hardware solution for clipDistance. I ended up with invisible ships on software path...

I dont know much about shaders, so ill attach what i had done so far, it may help to solve the problem.
sw_clipdistance.zip

On the two cpps check for the use of "hwClipDistance" thats were all changes are, it currently on VulkanRenderer.cpp it detects if the gpu support hardware clipDistance and saves it into a gloval variable, this is not the cleanest solution, but it was getting the job done for now, then on VulkanShaderCompiler.cpp it will pick up the global "hwClipDistance" and define "USE_SW_CLIP_DISTANCE" in the shaders headers.

Then on the shaders check for the use "USE_SW_CLIP_DISTANCE" every change is behind a #ifdef... since this is the same thing i did for gles i belive the problem may be on the selected layout=x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discussion This issue has (or wants) a discussion feature A totally new sort of functionality graphics A feature or issue related to graphics (2d and 3d)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants