Add runtime detection for macFUSE 4.x/5.x rename protocol

[zah]

Summary

This patch adds support for macFUSE's extended rename format and the RENAME_SWAP/RENAME_EXCL capabilities, enabling atomic rename operations on macOS.

Problem

macFUSE supports extended rename operations (renamex_np syscall with RENAME_SWAP and RENAME_EXCL flags) through an extended fuse_rename_in structure. However, there's an ABI incompatibility between macFUSE versions:

• macFUSE 4.x: Had a bug where it sent the extended 16-byte format unconditionally, regardless of whether RENAME_SWAP/RENAME_EXCL capabilities were negotiated during FUSE_INIT.

• macFUSE 5.x: Fixed this behavior - only sends the extended format when capabilities ARE granted; otherwise sends the standard 8-byte format.

This caused fuser to break on macFUSE 5.x when parsing rename requests, as the library wasn't handling both format variations.

References

• GitHub Issue: osxfuse/osxfuse#839
• macFUSE kernel header: fuse_kernel.h

Solution

This patch implements a two-pronged approach:

1. Capability Negotiation (lib.rs)

Request FUSE_RENAME_SWAP and FUSE_RENAME_EXCL capabilities during FUSE_INIT:

#[cfg(target_os = "macos")]
const INIT_FLAGS: u64 = FUSE_ASYNC_READ
    | FUSE_CASE_INSENSITIVE
    | FUSE_VOL_RENAME
    | FUSE_XTIMES
    | FUSE_RENAME_SWAP
    | FUSE_RENAME_EXCL;

This tells macFUSE 5.x that we want extended rename support, so it will grant the capabilities and send the extended format.

2. Runtime Format Detection (request.rs)

Parse FUSE_RENAME requests with runtime detection of the format:

Short format (8 bytes):
  newdir: u64
  <filename bytes follow immediately>

Extended format (16 bytes):
  newdir: u64
  flags: u32
  padding: u32
  <filename bytes follow>

The detection heuristic is based on the observation that filenames cannot start with null bytes or ASCII control characters (< 32). By inspecting the first byte after newdir:

• If < 32: Extended format (flags field contains a small number or zero)
• If >= 32: Short format (filename starts with a printable character)

This handles:

1. macFUSE 5.x with capabilities granted (extended format)
2. macFUSE 5.x with capabilities refused (short format)
3. macFUSE 4.x bug behavior (extended format unconditionally)

Files Changed

src/lib.rs

• Added FUSE_RENAME_SWAP and FUSE_RENAME_EXCL to macOS INIT_FLAGS
• Added documentation explaining the macFUSE version differences

src/ll/fuse_abi.rs

• Added FUSE_RENAME_SWAP (bit 25) and FUSE_RENAME_EXCL (bit 26) capability constants for macOS
• Updated fuse_rename_in documentation to explain the variable format

src/ll/request.rs

• Added flags field to the Rename struct
• Implemented runtime format detection in FUSE_RENAME parsing
• Added comprehensive documentation of the detection algorithm

src/ll/argument.rs

• Added skip_bytes() method to ArgumentIterator for skipping optional fields
• Added peek_byte() method for non-consuming byte inspection

src/request.rs

• Updated dispatch to pass x.flags() to Filesystem::rename() instead of hardcoded 0

[cacay]

Filenames can definitely contain ASCII control sequences. Finder won't allow it, but the terminal and system calls will. The only actual restriction is that filenames cannot contain NUL or /.

[cacay]

Do we need this detection mechanism? macFUSE 4.x always uses extended. macFUSE 5.x will always use extended since we request the capabilities. I guess there is a theoretical chance the kernel refuses the capabilities, but I don't know if that's possible in practice.

[cacay]

@zah I'm looking to support the renamex_np callback on macOS. From what I can tell, fuser doesn't pipe these flags to Filesystem::rename. Any change you'll revisit this PR?

There is now a Linux-only RenameFlags struct. Maybe that could be extended to macOS given this PR?