Unmanaged Callbacks marshal bool return values incorrectly

[NoelFB]

Really nice bindings!

The unmanaged callbacks that use bool return values are marshalled incorrectly, as C# assumes they are 4-byte values. In some cases it may be fine with this, but in others it will crash.

[DllImport(BindgenInternal.DllImportPath, EntryPoint = "b2World_SetPreSolveCallback")]
public static extern void WorldSetPreSolveCallback(WorldId worldId, delegate* unmanaged<ShapeId, ShapeId, Manifold*, void*, bool> fcn, void* context);

This can be easily fixed by changing the return value to a byte instead of a bool, but it does mean you have to manually return a byte when implementing the callbacks. There may be a way to use the [MarshalAs] attribute here, but I'm not sure how to with these unmanaged function pointers.

[DllImport(BindgenInternal.DllImportPath, EntryPoint = "b2World_SetPreSolveCallback")]
public static extern void WorldSetPreSolveCallback(WorldId worldId, delegate* unmanaged<ShapeId, ShapeId, Manifold*, void*, byte> fcn, void* context);

[BeanCheeseBurrito]

Box2D.NET/Box2D.NET.Bindings/B2.g.cs

Line 6 in 1b27965

[assembly: DisableRuntimeMarshalling]

The assembly is annotated with the DisableRuntimeMarshalling attribute. When an assembly has this attribute, booleans are assumed to be 1 byte.
https://learn.microsoft.com/en-us/dotnet/standard/native-interop/disabled-marshalling

[NoelFB]

Interesting! I consistently get an InvalidProgramException when using the bool return types on the unmanaged function pointers. If I change it to a byte, it behaves as expected. I'll have to investigate why this is happening.

[BeanCheeseBurrito]

Yep, you're right. I'm guessing the exception might be happening because the function pointer is pointing to a function that doesn't belong to the Box2D.NET.Bindings assembly. Will try to see if there is some way to fix this while still allowing booleans to be used.

For now, if you don't mind building from source, you can change this line to false in the binding generator. All occurrences of booleans in native code will be mapped to System.Byte.

Box2D.NET/Box2D.NET.Bindgen/Program.cs

Line 18 in 1b27965

GenerateDisableRuntimeMarshallingAttribute = true,

[RichardPreziosi]

Also affects WorldOverlapShape and WorldOverlapAABB (b2OverlapResultFcn callbacks).

Workaround until fixed: keep [UnmanagedCallersOnly] callback returning byte, then cast at the call site

(delegate* unmanaged<ShapeId, void*, bool>)(delegate* unmanaged<ShapeId, void*, byte>)&MyCallback