syscall.c fails to compile on Linux 5.10 due to SYS_openat2 not being defined in asm/unistd_64.h

[darkshram]

Description:

When compiling rsync 3.4.3 on a Linux 5.10.x system with kernel-headers-5.10.256 (LTS), the build fails with the following error:

syscall.c: In function 'secure_relative_open_linux':
syscall.c:1723:19: error: 'SYS_openat2' undeclared (first use in this function); did you mean 'SYS_openat'?
   dirfd = syscall(SYS_openat2, AT_FDCWD, basedir, &bhow, sizeof bhow);
                   ^~~~~~~~~~~
                   SYS_openat

Environment:

• Kernel: 5.10.256 (running)
• kernel-headers: 5.10.256 (installed)
• Architecture: x86_64 (also affects i386 builds)
• glibc: 2.30

Root cause analysis:

The secure_relative_open_linux() function introduced in PR #887 assumes that if <linux/openat2.h> is included and the running kernel is >=5.6, then the macro SYS_openat2 will be defined. This assumption is false.

Although the running kernel (5.10.256) supports the openat2 syscall, the constant SYS_openat2 is not defined in the standard user-space headers provided by kernel-headers-5.10.256. Specifically, it is missing from /usr/include/asm/unistd_64.h (and equivalent for i386).

The issue occurs because:

1. The code includes <linux/openat2.h> (defines RESOLVE_BENEATH, struct open_how, etc.)
2. However, SYS_openat2 (the syscall number) is defined in <asm/unistd.h> which is not consistently exported or may be outdated in distribution kernel-header packages.

Temporary workaround used:

Adding -DSYS_openat2=437 to CFLAGS resolves the compilation. The syscall number 437 is correct for both x86_64 and i386 architectures.

Expected behavior:

The build should succeed on Linux 5.10+ systems without requiring manual definition of SYS_openat2. The code should either:

1. Include the necessary headers in the correct order to ensure SYS_openat2 is defined, or
2. Use the syscall number directly (e.g., __NR_openat2 from an appropriate kernel header), or
3. Provide a fallback definition when SYS_openat2 is not defined by the system headers.

Additional notes:

1. This issue affects both 64-bit and 32-bit builds.
2. The problem is not specific to an outdated kernel; the running kernel (5.10.256 LTS) fully supports openat2.

Proposed fix suggestion:

Modify syscall.c to conditionally define SYS_openat2 if not already provided by system headers, for example:

#ifdef __linux__
#include <sys/syscall.h>
#include <linux/openat2.h>
#ifndef SYS_openat2
#define SYS_openat2 __NR_openat2
#endif
#endif

Alternatively, use runtime detection of the syscall number via syscall(__NR_openat2, ...) if __NR_openat2 is defined, or add a configure-time check.

Versions affected:

• rsync 3.4.3 (with PR use openat2 RESOLVE_BENEATH for secure_relative_open when available #887)
• Any backport of the openat2 change

Thank you for maintaining rsync!