The C compiler is asked to add a `cs` prefix to the `call` and `jmp` instructions that go through the indirect thunks for the `r8`-`r15` registers, which makes them 6 bytes long instead of 5. That gives `patch_retpoline()` enough room to inline them, e.g. as `lfence; call *%r11` for `spectre_v2=retpoline,lfence`. However, Rust code does not get the prefix, e.g. for a `CONFIG_MITIGATION_RETPOLINE=y` build: $ objdump -dr rust/kernel.o ... e2b: e8 00 00 00 00 call e30 <...> e2c: R_X86_64_PLT32 __x86_indirect_thunk_r11-0x4 Moreover, since commit 4a1e02b15ac1 ("x86,retpoline: Optimize patch_retpoline()"), `emit_indirect()` assumes there is enough room for the inlined sequence -- e.g. with `spectre_v2=retpoline,lfence`, the kernel warns and dies while patching a site inside `core`: WARNING: arch/x86/kernel/alternative.c:736 at apply_retpolines+0x2ee/0x400, CPU#0: swapper/0/0 ... Call Trace: ? _RNvMs6_NtNtCshS2CUL9MEDn_4core3fmt8buildersNtB5_9DebugList21finish_non_exhaustive+0x5b/0xd0 ... alternative_instructions+0x1d/0x60 arch_cpu_finalize_init+0x5a/0xb0 start_kernel+0x1f4/0x280 Similarly, paranoid FineIBT rewrites each kCFI call site into a fixed 20-byte sequence, i.e. the 14 bytes of the kCFI check plus a 6-byte call. For a 5-byte call, that overwrites the first byte of the next instruction, corrupting Rust code: SMP alternatives: Using paranoid FineIBT CFI ... Oops: double fault: 0000 [#1] ... RIP: 0010:_RNvNtCsdVypMZsLRAh_4core3fmt5write+0x1e9/0x210 Code: ... b8 68 81 16 9f 41 3b 43 f5 2e 4d 8d 5b f0 75 fd 41 ff e3 c0 eb 02 b0 01 ... ... Kernel panic - not syncing: Fatal exception in interrupt Non-paranoid FineIBT is not affected, since it only rewrites the kCFI check and leaves the call itself to `apply_retpolines()`. We added support for `-Zindirect-branch-cs-prefix` [1][2] in Rust 1.91.0 [3]. Thus pass it for `RETPOLINE` builds when possible, which makes all sites indeed get the prefix: e3c: 2e e8 00 00 00 00 cs call e42 <...> e3e: R_X86_64_PLT32 __x86_indirect_thunk_r11-0x4 The kernel above boots fine with `spectre_v2=retpoline,lfence`, inlining the Rust sites as well. For older Rust compilers, set the LLVM module flag instead, which has the same code-generation effect. This means that every Rust version that the kernel supports gets the prefix, that the kernel boots fine with `spectre_v2=retpoline,lfence`, and that paranoid FineIBT no longer corrupts. In addition, add it to our target modifier workaround for `rustdoc`. Cc: stable@vger.kernel.org # 6.18.y+ Link: https://github.com/rust-lang/rust/issues/116852 [1] Link: https://github.com/rust-lang/compiler-team/issues/899 [2] Link: https://github.com/rust-lang/rust/pull/140740 [3] Reported-by: Kent Overstreet Assisted-by: LLM Signed-off-by: Miguel Ojeda --- arch/x86/Makefile | 1 + rust/Makefile | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 8af6b80cffdd..01c848855f99 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -21,6 +21,7 @@ RETPOLINE_CFLAGS := -mretpoline-external-thunk RETPOLINE_VDSO_CFLAGS := -mretpoline endif RETPOLINE_CFLAGS += $(call cc-option,-mindirect-branch-cs-prefix) +RETPOLINE_RUSTFLAGS += $(if $(call rustc-min-version,109100),-Zindirect-branch-cs-prefix,-Zllvm_module_flag=indirect_branch_cs_prefix:u32:1:override) ifdef CONFIG_MITIGATION_RETHUNK RETHUNK_CFLAGS := -mfunction-return=thunk-extern diff --git a/rust/Makefile b/rust/Makefile index da1a7409d984..35868a47e18b 100644 --- a/rust/Makefile +++ b/rust/Makefile @@ -167,7 +167,7 @@ pin_init-flags := \ rustdoc_modifiers_workaround := $(if $(call rustc-min-version,108800),-Cunsafe-allow-abi-mismatch=fixed-x18) # Similarly, for doctests (https://github.com/rust-lang/rust/issues/146465). -doctests_modifiers_workaround := $(rustdoc_modifiers_workaround)$(if $(call rustc-min-version,109100),$(comma)sanitizer) +doctests_modifiers_workaround := $(rustdoc_modifiers_workaround)$(if $(call rustc-min-version,109100),$(comma)sanitizer$(comma)indirect-branch-cs-prefix) quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $< cmd_rustdoc = \ base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907 -- 2.56.0