When get_hwpoison_page() returns a negative value, distinguish reserved pages from other failure cases by reporting MF_MSG_KERNEL instead of MF_MSG_GET_HWPOISON. Reserved pages belong to the kernel and should be classified accordingly for proper handling. Sample PG_reserved before the get_hwpoison_page() call. In the MF_COUNT_INCREASED path get_any_page() can drop the caller's reference before returning -EIO, after which the underlying page may have been freed and reallocated with page->flags reset; reading PageReserved(p) at that point would observe stale or unrelated state. The pre-call snapshot reflects what the page actually was at the time of the failure event. Acked-by: Miaohe Lin Reviewed-by: Lance Yang Signed-off-by: Breno Leitao --- mm/memory-failure.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 866c4428ac7ef..f112fb27a8ff6 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2348,6 +2348,7 @@ int memory_failure(unsigned long pfn, int flags) unsigned long page_flags; bool retry = true; int hugetlb = 0; + bool is_reserved; if (!sysctl_memory_failure_recovery) panic("Memory failure on page %lx", pfn); @@ -2411,6 +2412,18 @@ int memory_failure(unsigned long pfn, int flags) * In fact it's dangerous to directly bump up page count from 0, * that may make page_ref_freeze()/page_ref_unfreeze() mismatch. */ + /* + * Pages with PG_reserved set are not currently managed by the + * page allocator (memblock-reserved memory, driver reservations, + * etc.), so classify them as kernel-owned for reporting. + * + * Sample the flag before get_hwpoison_page(): in the + * MF_COUNT_INCREASED path, get_any_page() can drop the caller's + * reference before returning -EIO, after which page->flags may + * have been reset by the allocator. + */ + is_reserved = PageReserved(p); + res = get_hwpoison_page(p, flags); if (!res) { if (is_free_buddy_page(p)) { @@ -2432,7 +2445,11 @@ int memory_failure(unsigned long pfn, int flags) } goto unlock_mutex; } else if (res < 0) { - res = action_result(pfn, MF_MSG_GET_HWPOISON, MF_IGNORED); + if (is_reserved) + res = action_result(pfn, MF_MSG_KERNEL, MF_IGNORED); + else + res = action_result(pfn, MF_MSG_GET_HWPOISON, + MF_IGNORED); goto unlock_mutex; } -- 2.53.0-Meta When get_any_page() fails to grab a page reference, the *reason* it failed is known at the call site but is not surfaced to callers: the HWPoisonHandlable() rejection path (a stable kernel page hwpoison cannot handle — slab, vmalloc, page tables, kernel stacks, ...) and the page_count() / put_page race paths (a transient page-allocator lifecycle race) all collapse to a single negative errno by the time memory_failure() sees them. memory_failure() can only observe the conflated result and reports both as MF_MSG_GET_HWPOISON. Surface the diagnosis explicitly. Add an mf_get_page_status enum, plumbed out through get_any_page() and get_hwpoison_page() (NULL is accepted by callers that do not care — unpoison_memory() and soft_offline_page() pass NULL). get_any_page() sets the status at the moment it gives up: MF_GET_PAGE_UNHANDLABLE — HWPoisonHandlable() rejected the page after retries. MF_GET_PAGE_RACE — exhausted retries on a refcount / lifecycle race with the allocator. memory_failure() then promotes the unhandlable case to MF_MSG_KERNEL alongside the existing PageReserved branch, and leaves the transient-race case as MF_MSG_GET_HWPOISON. This forms the foundation a later patch will rely on to decide whether an unrecoverable failure should panic. Drop the "reserved" qualifier from action_page_types[MF_MSG_KERNEL] and the matching tracepoint string in MF_PAGE_TYPE: the enum value now covers both PageReserved pages and unhandlable kernel pages (slab, vmalloc, page tables, kernel stacks, ...), so "kernel page" is the accurate label for both populations. Suggested-by: Lance Yang Signed-off-by: Breno Leitao --- include/trace/events/memory-failure.h | 2 +- mm/memory-failure.c | 46 +++++++++++++++++++++++++++++------ 2 files changed, 39 insertions(+), 9 deletions(-) diff --git a/include/trace/events/memory-failure.h b/include/trace/events/memory-failure.h index aa57cc8f896be..8a860e6fcb4e9 100644 --- a/include/trace/events/memory-failure.h +++ b/include/trace/events/memory-failure.h @@ -24,7 +24,7 @@ EMe ( MF_RECOVERED, "Recovered" ) #define MF_PAGE_TYPE \ - EM ( MF_MSG_KERNEL, "reserved kernel page" ) \ + EM ( MF_MSG_KERNEL, "kernel page" ) \ EM ( MF_MSG_KERNEL_HIGH_ORDER, "high-order kernel page" ) \ EM ( MF_MSG_HUGE, "huge page" ) \ EM ( MF_MSG_FREE_HUGE, "free huge page" ) \ diff --git a/mm/memory-failure.c b/mm/memory-failure.c index f112fb27a8ff6..4210173060aac 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -878,7 +878,7 @@ static const char *action_name[] = { }; static const char * const action_page_types[] = { - [MF_MSG_KERNEL] = "reserved kernel page", + [MF_MSG_KERNEL] = "kernel page", [MF_MSG_KERNEL_HIGH_ORDER] = "high-order kernel page", [MF_MSG_HUGE] = "huge page", [MF_MSG_FREE_HUGE] = "free huge page", @@ -1389,11 +1389,29 @@ static int __get_hwpoison_page(struct page *page, unsigned long flags) #define GET_PAGE_MAX_RETRY_NUM 3 -static int get_any_page(struct page *p, unsigned long flags) +enum mf_get_page_status { + MF_GET_PAGE_OK = 0, + MF_GET_PAGE_RACE, + MF_GET_PAGE_UNHANDLABLE, +}; + +static void set_mf_get_page_status(enum mf_get_page_status *gp_status, + enum mf_get_page_status value) +{ + if (!gp_status) + return; + + *gp_status = value; +} + +static int get_any_page(struct page *p, unsigned long flags, + enum mf_get_page_status *gp_status) { int ret = 0, pass = 0; bool count_increased = false; + set_mf_get_page_status(gp_status, MF_GET_PAGE_OK); + if (flags & MF_COUNT_INCREASED) count_increased = true; @@ -1406,11 +1424,13 @@ static int get_any_page(struct page *p, unsigned long flags) if (pass++ < GET_PAGE_MAX_RETRY_NUM) goto try_again; ret = -EBUSY; + set_mf_get_page_status(gp_status, MF_GET_PAGE_RACE); } else if (!PageHuge(p) && !is_free_buddy_page(p)) { /* We raced with put_page, retry. */ if (pass++ < GET_PAGE_MAX_RETRY_NUM) goto try_again; ret = -EIO; + set_mf_get_page_status(gp_status, MF_GET_PAGE_RACE); } goto out; } else if (ret == -EBUSY) { @@ -1423,6 +1443,7 @@ static int get_any_page(struct page *p, unsigned long flags) goto try_again; } ret = -EIO; + set_mf_get_page_status(gp_status, MF_GET_PAGE_UNHANDLABLE); goto out; } } @@ -1442,6 +1463,7 @@ static int get_any_page(struct page *p, unsigned long flags) } put_page(p); ret = -EIO; + set_mf_get_page_status(gp_status, MF_GET_PAGE_UNHANDLABLE); } out: if (ret == -EIO) @@ -1480,6 +1502,7 @@ static int __get_unpoison_page(struct page *page) * get_hwpoison_page() - Get refcount for memory error handling * @p: Raw error page (hit by memory error) * @flags: Flags controlling behavior of error handling + * @gp_status: Optional output for the reason get_any_page() failed * * get_hwpoison_page() takes a page refcount of an error page to handle memory * error on it, after checking that the error page is in a well-defined state @@ -1503,7 +1526,8 @@ static int __get_unpoison_page(struct page *page) * operations like allocation and free, * -EHWPOISON when the page is hwpoisoned and taken off from buddy. */ -static int get_hwpoison_page(struct page *p, unsigned long flags) +static int get_hwpoison_page(struct page *p, unsigned long flags, + enum mf_get_page_status *gp_status) { int ret; @@ -1511,7 +1535,7 @@ static int get_hwpoison_page(struct page *p, unsigned long flags) if (flags & MF_UNPOISON) ret = __get_unpoison_page(p); else - ret = get_any_page(p, flags); + ret = get_any_page(p, flags, gp_status); zone_pcp_enable(page_zone(p)); return ret; @@ -2349,6 +2373,7 @@ int memory_failure(unsigned long pfn, int flags) bool retry = true; int hugetlb = 0; bool is_reserved; + enum mf_get_page_status gp_status = MF_GET_PAGE_OK; if (!sysctl_memory_failure_recovery) panic("Memory failure on page %lx", pfn); @@ -2424,7 +2449,7 @@ int memory_failure(unsigned long pfn, int flags) */ is_reserved = PageReserved(p); - res = get_hwpoison_page(p, flags); + res = get_hwpoison_page(p, flags, &gp_status); if (!res) { if (is_free_buddy_page(p)) { if (take_page_off_buddy(p)) { @@ -2445,7 +2470,12 @@ int memory_failure(unsigned long pfn, int flags) } goto unlock_mutex; } else if (res < 0) { - if (is_reserved) + /* + * Promote a stable unhandlable kernel page diagnosed by + * get_hwpoison_page() to MF_MSG_KERNEL alongside reserved + * pages; transient lifecycle races stay as MF_MSG_GET_HWPOISON. + */ + if (is_reserved || gp_status == MF_GET_PAGE_UNHANDLABLE) res = action_result(pfn, MF_MSG_KERNEL, MF_IGNORED); else res = action_result(pfn, MF_MSG_GET_HWPOISON, @@ -2750,7 +2780,7 @@ int unpoison_memory(unsigned long pfn) goto unlock_mutex; } - ghp = get_hwpoison_page(p, MF_UNPOISON); + ghp = get_hwpoison_page(p, MF_UNPOISON, NULL); if (!ghp) { if (folio_test_hugetlb(folio)) { huge = true; @@ -2957,7 +2987,7 @@ int soft_offline_page(unsigned long pfn, int flags) retry: get_online_mems(); - ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE); + ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE, NULL); put_online_mems(); if (hwpoison_filter(page)) { -- 2.53.0-Meta Add a sysctl panic_on_unrecoverable_memory_failure that triggers a kernel panic when memory_failure() encounters pages that cannot be recovered. This provides a clean crash with useful debug information rather than allowing silent data corruption or a delayed crash at an unrelated code path. Panic eligibility is intentionally narrow: only MF_MSG_KERNEL with result == MF_IGNORED panics. That covers reserved pages (PageReserved) and the kernel page types that the prior patch promotes from MF_MSG_GET_HWPOISON via MF_GET_PAGE_UNHANDLABLE — slab, vmalloc, page tables, kernel stacks, and similar non-LRU/non-buddy kernel-owned pages. All other action types are excluded: - MF_MSG_GET_HWPOISON and MF_MSG_KERNEL_HIGH_ORDER can be reached by transient refcount races with the page allocator (an in-flight buddy allocation has refcount 0 and is no longer on the buddy free list, briefly), and panicking on them would risk killing the box for what is actually a recoverable userspace page. - MF_MSG_UNKNOWN means identify_page_state() could not classify the page; that is precisely the wrong basis for a panic decision. Signed-off-by: Breno Leitao --- mm/memory-failure.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 4210173060aac..e4a9ceacaf36b 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -74,6 +74,8 @@ static int sysctl_memory_failure_recovery __read_mostly = 1; static int sysctl_enable_soft_offline __read_mostly = 1; +static int sysctl_panic_on_unrecoverable_mf __read_mostly; + atomic_long_t num_poisoned_pages __read_mostly = ATOMIC_LONG_INIT(0); static bool hw_memory_failure __read_mostly = false; @@ -155,6 +157,15 @@ static const struct ctl_table memory_failure_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE, + }, + { + .procname = "panic_on_unrecoverable_memory_failure", + .data = &sysctl_panic_on_unrecoverable_mf, + .maxlen = sizeof(sysctl_panic_on_unrecoverable_mf), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, } }; @@ -1281,6 +1292,18 @@ static void update_per_node_mf_stats(unsigned long pfn, ++mf_stats->total; } +static bool panic_on_unrecoverable_mf(enum mf_action_page_type type, + enum mf_result result) +{ + if (!sysctl_panic_on_unrecoverable_mf || result != MF_IGNORED) + return false; + + if (type == MF_MSG_KERNEL) + return true; + + return false; +} + /* * "Dirty/Clean" indication is not 100% accurate due to the possibility of * setting PG_dirty outside page lock. See also comment above set_page_dirty(). @@ -1298,6 +1321,9 @@ static int action_result(unsigned long pfn, enum mf_action_page_type type, pr_err("%#lx: recovery action for %s: %s\n", pfn, action_page_types[type], action_name[result]); + if (panic_on_unrecoverable_mf(type, result)) + panic("Memory failure: %#lx: unrecoverable page", pfn); + return (result == MF_RECOVERED || result == MF_DELAYED) ? 0 : -EBUSY; } -- 2.53.0-Meta Add documentation for the new vm.panic_on_unrecoverable_memory_failure sysctl, describing which failures trigger a panic (kernel-owned pages the handler cannot recover) and which are intentionally left out (transient allocator races and unclassified pages). Signed-off-by: Breno Leitao --- Documentation/admin-guide/sysctl/vm.rst | 70 +++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 97e12359775c9..802c51ba8c43b 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -67,6 +67,7 @@ Currently, these files are in /proc/sys/vm: - page-cluster - page_lock_unfairness - panic_on_oom +- panic_on_unrecoverable_memory_failure - percpu_pagelist_high_fraction - stat_interval - stat_refresh @@ -925,6 +926,75 @@ panic_on_oom=2+kdump gives you very strong tool to investigate why oom happens. You can get snapshot. +panic_on_unrecoverable_memory_failure +====================================== + +When a hardware memory error (e.g. multi-bit ECC) hits a kernel page +that cannot be recovered by the memory failure handler, the default +behaviour is to ignore the error and continue operation. This is +dangerous because the corrupted data remains accessible to the kernel, +risking silent data corruption or a delayed crash when the poisoned +memory is next accessed. + +When enabled, this sysctl triggers a panic on kernel-owned pages that +the memory failure handler cannot recover: reserved pages +(``PageReserved``) and stable kernel pages that hwpoison cannot handle +(slab, vmalloc, page tables, kernel stacks, and similar non-LRU, +non-buddy pages). + +Other failure paths are intentionally left out because they can be +reached by transient races with the page allocator (an in-flight +buddy allocation has refcount 0 and is no longer on the buddy free +list, briefly), and panicking on them would risk killing the box for +a page that was actually destined for userspace where the standard +SIGBUS recovery path applies. Pages whose state could not be +classified at all are also not covered, since an unknown state is +not a sound basis for a panic decision. + +For many environments it is preferable to panic immediately with a clean +crash dump that captures the original error context, rather than to +continue and face a random crash later whose cause is difficult to +diagnose. + +Use cases +--------- + +This option is most useful in environments where unattributed crashes +are expensive to debug or where data integrity must take precedence +over availability: + +* Large fleets, where multi-bit ECC errors on kernel pages are observed + regularly and post-mortem analysis of an unrelated downstream crash + (often seconds to minutes after the original error) consumes + significant engineering effort. + +* Systems configured with kdump, where panicking at the moment of the + hardware error produces a vmcore that still contains the faulting + address, the affected page state, and the originating MCE/GHES + record — context that is typically lost by the time a delayed crash + occurs. + +* High-availability clusters that rely on fast, deterministic node + failure for failover, and prefer an immediate panic over silent data + corruption propagating to replicas or persistent storage. + +* Kernel and platform developers reproducing hwpoison issues with + tools such as ``mce-inject`` or error-injection debugfs interfaces, + where panicking on the unrecoverable path makes regressions + immediately visible instead of surfacing as later, unrelated + failures. + += ===================================================================== +0 Try to continue operation (default). +1 Panic immediately. If the ``panic`` sysctl is also non-zero then the + machine will be rebooted. += ===================================================================== + +Example:: + + echo 1 > /proc/sys/vm/panic_on_unrecoverable_memory_failure + + percpu_pagelist_high_fraction ============================= -- 2.53.0-Meta