From: Gary Guo This config detects if Rust and Clang have matching LLVM major version. All IR or bitcode operations (e.g. LTO) rely on LLVM major version to be matching, otherwise it may generate errors, or worse, miscompile silently due to change of IR semantics. It's usually suggested to use the exact same LLVM version, but this can be difficult to guarantee. Rust's suggestion [1] is also major-version only, so I think this check is sufficient for the kernel. Link: https://doc.rust-lang.org/rustc/linker-plugin-lto.html [1] Reviewed-by: Andreas Hindborg Signed-off-by: Gary Guo Signed-off-by: Matthew Maurer Signed-off-by: Alice Ryhl --- init/Kconfig | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/init/Kconfig b/init/Kconfig index e95d43457851862afc8313389777e4dd9348c178..0e900d3d8be7874a33e0f44754a8d038e68d7e65 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -82,6 +82,21 @@ config RUSTC_LLVM_VERSION int default $(rustc-llvm-version) +config RUSTC_LLVM_MAJOR_VERSION + int + default $(shell,expr $(rustc-llvm-version) / 10000) + +config RUSTC_CLANG_LLVM_COMPATIBLE + bool + default y if CC_IS_CLANG && RUSTC_LLVM_MAJOR_VERSION = $(shell,expr $(cc-version) / 10000) + help + This indicates whether Rust and Clang use LLVM of the same major + version. + + Operations involving handling LLVM IR or bitcode (e.g. cross-language + LTO) requires the same LLVM major version to work properly. For best + compatibility it is recommended that the exact same LLVM is used. + config ARCH_HAS_CC_CAN_LINK bool -- 2.53.0.rc1.225.gd81095ad13-goog From: Gary Guo Because of LLVM inling checks, it's generally not possible to inline a C helper into Rust code, even with LTO: * LLVM doesn't want to inline functions compiled with `-fno-delete-null-pointer-checks` with code compiled without. The C CGUs all have this enabled and Rust CGUs don't. Inlining is okay since this is one of the hardening features that does not change the ABI, and we shouldn't have null pointer dereferences in these helpers. * LLVM doesn't want to inline functions with different list of builtins. C side has `-fno-builtin-wcslen`; `wcslen` is not a Rust builtin, so they should be compatible, but LLVM does not perform inlining due to attributes mismatch. * clang and Rust doesn't have the exact target string. Clang generates `+cmov,+cx8,+fxsr` but Rust doesn't enable them (in fact, Rust will complain if `-Ctarget-feature=+cmov,+cx8,+fxsr` is used). x86-64 always enable these features, so they are in fact the same target string, but LLVM doesn't understand this and so inlining is inhibited. This can be bypassed with `--ignore-tti-inline-compatible`, but this is a hidden option. To fix this, we can add __always_inline on every helper, which skips these LLVM inlining checks. For this purpose, introduce a new __rust_helper macro that needs to be added to every helper. Most helpers already have __rust_helper specified, but there are a few missing. The only consequence of this is that those specific helpers do not get inlined. Signed-off-by: Gary Guo Signed-off-by: Alice Ryhl --- rust/helpers/helpers.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c index a3c42e51f00a0990bea81ebce6e99bb397ce7533..e05c6e7e4abb7e6a4c4a3a417e35022dac1d9c58 100644 --- a/rust/helpers/helpers.c +++ b/rust/helpers/helpers.c @@ -7,7 +7,36 @@ * Sorted alphabetically. */ +#include + +#ifdef __BINDGEN__ +// Omit `inline` for bindgen as it ignores inline functions. #define __rust_helper +#else +// The helper functions are all inline functions. +// +// We use `__always_inline` here to bypass LLVM inlining checks, in case the +// helpers are inlined directly into Rust CGUs. +// +// The LLVM inlining checks are false positives: +// * LLVM doesn't want to inline functions compiled with +// `-fno-delete-null-pointer-checks` with code compiled without. +// The C CGUs all have this enabled and Rust CGUs don't. Inlining is okay +// since this is one of the hardening features that does not change the ABI, +// and we shouldn't have null pointer dereferences in these helpers. +// * LLVM doesn't want to inline functions with different list of builtins. C +// side has `-fno-builtin-wcslen`; `wcslen` is not a Rust builtin, so they +// should be compatible, but LLVM does not perform inlining due to attributes +// mismatch. +// * clang and Rust doesn't have the exact target string. Clang generates +// `+cmov,+cx8,+fxsr` but Rust doesn't enable them (in fact, Rust will +// complain if `-Ctarget-feature=+cmov,+cx8,+fxsr` is used). x86-64 always +// enable these features, so they are in fact the same target string, but +// LLVM doesn't understand this and so inlining is inhibited. This can be +// bypassed with `--ignore-tti-inline-compatible`, but this is a hidden +// option. +#define __rust_helper __always_inline +#endif #include "atomic.c" #include "atomic_ext.c" -- 2.53.0.rc1.225.gd81095ad13-goog From: Gary Guo A new experimental Kconfig option, `RUST_INLINE_HELPERS` is added to allow C helpers (which were created to allow Rust to call into inline/macro C functions without having to re-implement the logic in Rust) to be inlined into Rust crates without performing global LTO. If the option is enabled, the following is performed: * For helpers, instead of compiling them to an object file to be linked into vmlinux, they're compiled to LLVM IR bitcode. Two versions are generated: one for built-in code (`helpers.bc`) and one for modules (`helpers_module.bc`, with -DMODULE defined). This ensures that C macros/inlines that behave differently for modules (e.g. static calls) function correctly when inlined. * When a Rust crate or object is compiled, instead of generating an object file, LLVM bitcode is generated. * llvm-link is invoked with --internalize to combine the helper bitcode with the crate bitcode. This step is similar to LTO, but this is much faster since it only needs to inline the helpers. * clang is invoked to turn the combined bitcode into a final object file. * Since clang may produce LLVM bitcode when LTO is enabled, and objtool requires ELF input, $(cmd_ld_single) is invoked to ensure the object is converted to ELF before objtool runs. The --internalize flag tells llvm-link to treat all symbols in helpers.bc using `internal` linkage [1]. This matches the behavior of `clang` on `static inline` functions, and avoids exporting the symbol from the object file. To ensure that RUST_INLINE_HELPERS is not incompatible with BTF, we pass the -g0 flag when building helpers. See commit 5daa0c35a1f0 ("rust: Disallow BTF generation with Rust + LTO") for details. We have an intended triple mismatch of `aarch64-unknown-none` vs `aarch64-unknown-linux-gnu`, so we pass --suppress-warnings to llvm-link to suppress it. We add $(cmd_ld_single) before $(cmd_objtool). Otherwise objtool fails to parse the resulting files as Elf when CONFIG_LTO is enabled. I considered adding some sort of check that KBUILD_MODNAME is not present in helpers_module.bc, but this is actually not so easy to carry out because .bc files store strings in a weird binary format, so you cannot just grep it for a string to check whether it ended up using KBUILD_MODNAME anywhere. Link: https://github.com/llvm/llvm-project/pull/170397 [1] Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Co-developed-by: Matthew Maurer Signed-off-by: Matthew Maurer Signed-off-by: Gary Guo Co-developed-by: Alice Ryhl Signed-off-by: Alice Ryhl --- Makefile | 3 ++- lib/Kconfig.debug | 15 +++++++++++++++ rust/Makefile | 29 +++++++++++++++++++++++++---- rust/exports.c | 5 ++++- scripts/Makefile.build | 7 ++++++- 5 files changed, 52 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index c97f06ee0dda1c922aa23bd5249052591d528eb6..155db03cf489d931eefdb4bd7e3b93e5aa3ea2d6 100644 --- a/Makefile +++ b/Makefile @@ -519,6 +519,7 @@ OBJCOPY = $(LLVM_PREFIX)llvm-objcopy$(LLVM_SUFFIX) OBJDUMP = $(LLVM_PREFIX)llvm-objdump$(LLVM_SUFFIX) READELF = $(LLVM_PREFIX)llvm-readelf$(LLVM_SUFFIX) STRIP = $(LLVM_PREFIX)llvm-strip$(LLVM_SUFFIX) +LLVM_LINK = $(LLVM_PREFIX)llvm-link$(LLVM_SUFFIX) else CC = $(CROSS_COMPILE)gcc LD = $(CROSS_COMPILE)ld @@ -627,7 +628,7 @@ export RUSTC_BOOTSTRAP := 1 export CLIPPY_CONF_DIR := $(srctree) export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE LD CC HOSTPKG_CONFIG -export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN +export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN LLVM_LINK export HOSTRUSTC KBUILD_HOSTRUSTFLAGS export CPP AR NM STRIP OBJCOPY OBJDUMP READELF PAHOLE RESOLVE_BTFIDS LEX YACC AWK INSTALLKERNEL export PERL PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 7570a694d54e6170ae6738634218b37e38a4b76a..c44c08b49f3a3c2e06fca42adeb9e04304e7a869 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -3551,6 +3551,21 @@ config RUST_KERNEL_DOCTESTS If unsure, say N. +config RUST_INLINE_HELPERS + bool "Inline C helpers into Rust code" + depends on RUST && RUSTC_CLANG_LLVM_COMPATIBLE + depends on EXPERT + help + Inlines C helpers into Rust code using Link Time Optimization. + + If this option is enabled, C helper functions declared in + rust/helpers/ are inlined into Rust code, which is helpful for + performance of Rust code. This requires a matching LLVM version for + Clang and rustc. + + If you are sure that you're using Clang and rustc with matching LLVM + versions, say Y. Otherwise say N. + endmenu # "Rust" endmenu # Kernel hacking diff --git a/rust/Makefile b/rust/Makefile index 63464bd2c1e9734cd6e659f7ee3db58bf995d6dd..a6839f2d39feefcfea497384ae202a7c6b475942 100644 --- a/rust/Makefile +++ b/rust/Makefile @@ -6,15 +6,19 @@ rustdoc_output := $(objtree)/Documentation/output/rust/rustdoc obj-$(CONFIG_RUST) += core.o compiler_builtins.o ffi.o always-$(CONFIG_RUST) += exports_core_generated.h +ifdef CONFIG_RUST_INLINE_HELPERS +always-$(CONFIG_RUST) += helpers/helpers.bc helpers/helpers_module.bc +else +obj-$(CONFIG_RUST) += helpers/helpers.o +always-$(CONFIG_RUST) += exports_helpers_generated.h +endif # Missing prototypes are expected in the helpers since these are exported # for Rust only, thus there is no header nor prototypes. -obj-$(CONFIG_RUST) += helpers/helpers.o CFLAGS_REMOVE_helpers/helpers.o = -Wmissing-prototypes -Wmissing-declarations always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs obj-$(CONFIG_RUST) += bindings.o pin_init.o kernel.o -always-$(CONFIG_RUST) += exports_helpers_generated.h \ - exports_bindings_generated.h exports_kernel_generated.h +always-$(CONFIG_RUST) += exports_bindings_generated.h exports_kernel_generated.h always-$(CONFIG_RUST) += uapi/uapi_generated.rs obj-$(CONFIG_RUST) += uapi.o @@ -480,6 +484,14 @@ $(obj)/bindings/bindings_helpers_generated.rs: private bindgen_target_extra = ; $(obj)/bindings/bindings_helpers_generated.rs: $(src)/helpers/helpers.c FORCE $(call if_changed_dep,bindgen) +quiet_cmd_rust_helper = HELPER $@ + cmd_rust_helper = \ + $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) \ + -c -g0 $< $(if $(filter %_module.bc,$@),-DMODULE) -emit-llvm -o $@ + +$(obj)/helpers/helpers.bc $(obj)/helpers/helpers_module.bc: $(src)/helpers/helpers.c FORCE + +$(call if_changed_dep,rust_helper) + rust_exports = $(NM) -p --defined-only $(1) | awk '$$2~/(T|R|D|B)/ && $$3!~/__(pfx|cfi|odr_asan)/ { printf $(2),$$3 }' quiet_cmd_exports = EXPORTS $@ @@ -561,12 +573,16 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L OBJTREE=$(abspath $(objtree)) \ $(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \ $(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) \ - --emit=dep-info=$(depfile) --emit=obj=$@ \ + --emit=dep-info=$(depfile) --emit=$(if $(link_helper),llvm-bc=$(patsubst %.o,%.bc,$@),obj=$@) \ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \ --crate-type rlib -L$(objtree)/$(obj) \ --crate-name $(patsubst %.o,%,$(notdir $@)) $< \ --sysroot=/dev/null \ + $(if $(link_helper),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \ + $(obj)/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \ + $(CC) $(CLANG_FLAGS) $(KBUILD_CFLAGS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@) \ $(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@) \ + $(cmd_ld_single) \ $(cmd_objtool) rust-analyzer: @@ -696,4 +712,9 @@ $(obj)/kernel.o: $(obj)/kernel/generated_arch_warn_asm.rs $(obj)/kernel/generate endif endif +ifdef CONFIG_RUST_INLINE_HELPERS +$(obj)/kernel.o: private link_helper = 1 +$(obj)/kernel.o: $(obj)/helpers/helpers.bc +endif + endif # CONFIG_RUST diff --git a/rust/exports.c b/rust/exports.c index 587f0e776aba52854080f15aa91094b55996c072..1b52460b0f4eeef6df9081abb9b7e054a28c3c21 100644 --- a/rust/exports.c +++ b/rust/exports.c @@ -16,10 +16,13 @@ #define EXPORT_SYMBOL_RUST_GPL(sym) extern int sym; EXPORT_SYMBOL_GPL(sym) #include "exports_core_generated.h" -#include "exports_helpers_generated.h" #include "exports_bindings_generated.h" #include "exports_kernel_generated.h" +#ifndef CONFIG_RUST_INLINE_HELPERS +#include "exports_helpers_generated.h" +#endif + // For modules using `rust/build_error.rs`. #ifdef CONFIG_RUST_BUILD_ASSERT_ALLOW EXPORT_SYMBOL_RUST_GPL(rust_build_error); diff --git a/scripts/Makefile.build b/scripts/Makefile.build index 0c838c467c764e14a51ad132444544373e90a84c..7176d997ecab7c3ffd1e00400a8a79a8309485e1 100644 --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -343,7 +343,12 @@ rust_common_cmd = \ # would not match each other. quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@ - cmd_rustc_o_rs = $(rust_common_cmd) --emit=obj=$@ $< $(cmd_objtool) + cmd_rustc_o_rs = $(rust_common_cmd) --emit=$(if $(CONFIG_RUST_INLINE_HELPERS),llvm-bc=$(patsubst %.o,%.bc,$@),obj=$@) $< \ + $(if $(CONFIG_RUST_INLINE_HELPERS),;$(LLVM_LINK) --internalize --suppress-warnings $(patsubst %.o,%.bc,$@) \ + $(objtree)/rust/helpers/helpers$(if $(part-of-module),_module).bc -o $(patsubst %.o,%.m.bc,$@); \ + $(CC) $(CLANG_FLAGS) $(KBUILD_CFLAGS) -Wno-override-module -c $(patsubst %.o,%.m.bc,$@) -o $@) \ + $(cmd_ld_single) \ + $(cmd_objtool) define rule_rustc_o_rs $(call cmd_and_fixdep,rustc_o_rs) -- 2.53.0.rc1.225.gd81095ad13-goog