The shmem allocation counters cannot identify which tmpfs file is affected by allocation fallback. They also do not describe order filtering caused by VMA constraints or existing page-cache entries. Trace failed calls to shmem_alloc_and_add_folio() and successful calls that allocate below the highest order permitted by the supplied mask. Record the original index and the highest permitted, highest suitable and allocated orders to distinguish filtering from allocator fallback. A suitable order of -1 identifies an empty filtered mask, where no folio allocation is attempted; such records use a separate filter stage and keep the stage error at zero while preserving the helper return value. Omit successes at the requested order, including ordinary base-page allocations. Identify failures by stage: allocation, memcg charging, cache insertion or filesystem accounting. Preserve both the stage's errno and the returned errno. In particular, a charge failure remains visible even when a page-cache conflict makes the function return -EEXIST instead. Stage and errno distinguish this case from a direct insertion race. Document that each event describes one call to the helper, which may try multiple candidate orders. The caller may subsequently retry at order 0 or restart a lookup, so an error in this event is not necessarily a failed user operation. A successful order-0 retry is omitted as a success at that call's requested order. Signed-off-by: Xuewen Wang --- Documentation/trace/events-shmem.rst | 45 +++++++++++++++++++++++ include/trace/events/shmem.h | 55 ++++++++++++++++++++++++++++ mm/shmem.c | 34 ++++++++++++++++- 3 files changed, 133 insertions(+), 1 deletion(-) diff --git a/Documentation/trace/events-shmem.rst b/Documentation/trace/events-shmem.rst index 45da82761360..4a0e50bdbdf3 100644 --- a/Documentation/trace/events-shmem.rst +++ b/Documentation/trace/events-shmem.rst @@ -71,3 +71,48 @@ Filter or aggregate by both ``dev`` and ``ino``; inode numbers alone are not unique across mounts. Trace filters use the numeric values shown by the event's ``format`` file. Check ``per_cpu/cpu*/stats`` for dropped events before interpreting counts. + +Allocation attempts +------------------- + +``shmem_alloc_folio_finish`` records failed calls to +``shmem_alloc_and_add_folio()`` and successful calls that allocate below +their requested order. Successes at the requested order are omitted, +including ordinary order-0 allocations. + +This is an attempt result, not the final result of a read, write or fault. +After a higher-order failure the caller may retry at order 0, or restart +its lookup on ``-EEXIST``. A successful order-0 retry is not recorded. +Do not count all nonzero returns as failed user operations. + +``requested_order`` is the highest order allowed by this call's order mask, +or 0 for a base-page request. ``suitable_order`` is the highest remaining +order after VMA and page-cache filtering, or -1 if none remains. +``actual_order`` is the allocated folio's order, or -1 if allocation did +not produce a folio. Orders refer to base pages, not bytes. + +``stage`` identifies where the attempt ended: + +* ``filter``: filtering removed every candidate order, so no folio + allocation was attempted. ``suitable_order=-1`` identifies this case. +* ``alloc``: a suitable order existed, but every allocation failed. +* ``charge``: memcg charging failed. +* ``cache``: page-cache insertion failed. +* ``account``: filesystem block accounting failed after its retry. +* ``done``: allocation succeeded at a smaller order. + +``error`` is the error at that stage; ``ret`` is the error returned to the +caller. ``stage=filter`` has ``error=0`` because no allocator call occurred, +while ``ret=-ENOMEM`` preserves the helper's return value. Both are zero for +``done``. For a charge failure followed by a page-cache conflict, ``error`` +retains the charge error (such as ``-ENOMEM``), while ``ret=-EEXIST`` +requests a lookup retry. A direct insertion race instead has +``stage=cache error=-EEXIST ret=-EEXIST``. Accounting errors retain +``-ENOSPC`` or ``-EDQUOT`` as appropriate. + +Enable ``events/shmem/shmem_alloc_folio_finish/enable`` in the trace +instance to collect these records. On successful fallbacks, +``suitable_order < requested_order`` shows filtering, and +``actual_order < suitable_order`` shows allocator fallback. Both may +occur in one attempt. The existing shmem allocation and fallback +counters remain useful for aggregate statistics. diff --git a/include/trace/events/shmem.h b/include/trace/events/shmem.h index 7a0050a1686f..72b20d6b3caf 100644 --- a/include/trace/events/shmem.h +++ b/include/trace/events/shmem.h @@ -80,6 +80,61 @@ TRACE_EVENT(shmem_swapin_folio, __entry->fallback_error, __entry->error) ); +#define SHMEM_ALLOC_ALLOC 0 +#define SHMEM_ALLOC_CHARGE 1 +#define SHMEM_ALLOC_CACHE 2 +#define SHMEM_ALLOC_ACCOUNT 3 +#define SHMEM_ALLOC_DONE 4 +#define SHMEM_ALLOC_FILTER 5 + +TRACE_EVENT(shmem_alloc_folio_finish, + + TP_PROTO(struct inode *inode, pgoff_t index, + int requested_order, int suitable_order, int actual_order, + unsigned int stage, int error, int ret), + + TP_ARGS(inode, index, requested_order, suitable_order, actual_order, + stage, error, ret), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(unsigned long, ino) + __field(unsigned long, index) + __field(int, requested_order) + __field(int, suitable_order) + __field(int, actual_order) + __field(unsigned int, stage) + __field(int, error) + __field(int, ret) + ), + + TP_fast_assign( + __entry->dev = inode->i_sb->s_dev; + __entry->ino = inode->i_ino; + __entry->index = index; + __entry->requested_order = requested_order; + __entry->suitable_order = suitable_order; + __entry->actual_order = actual_order; + __entry->stage = stage; + __entry->error = error; + __entry->ret = ret; + ), + + TP_printk("dev=%d:%d ino=%lu index=%lu requested_order=%d suitable_order=%d actual_order=%d stage=%s error=%d ret=%d", + MAJOR(__entry->dev), MINOR(__entry->dev), + __entry->ino, __entry->index, + __entry->requested_order, __entry->suitable_order, + __entry->actual_order, + __print_symbolic(__entry->stage, + { SHMEM_ALLOC_ALLOC, "alloc" }, + { SHMEM_ALLOC_CHARGE, "charge" }, + { SHMEM_ALLOC_CACHE, "cache" }, + { SHMEM_ALLOC_ACCOUNT, "account" }, + { SHMEM_ALLOC_DONE, "done" }, + { SHMEM_ALLOC_FILTER, "filter" }), + __entry->error, __entry->ret) +); + #endif /* _TRACE_SHMEM_H */ /* This part must be outside protection */ diff --git a/mm/shmem.c b/mm/shmem.c index 3b910aec5bcf..68321d61dfdb 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2155,17 +2155,23 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, unsigned long suitable_orders = 0; struct folio *folio = NULL; pgoff_t aligned_index; + pgoff_t orig_index = index; long pages; int error, order; + int requested_order, suitable_order; + int charge_error = 0; + unsigned int stage; if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) orders = 0; + requested_order = orders ? highest_order(orders) : 0; if (orders > 0) { suitable_orders = shmem_suitable_orders(inode, vmf, mapping, index, orders); order = highest_order(suitable_orders); + suitable_order = order; while (suitable_orders) { pages = 1UL << order; aligned_index = round_down(index, pages); @@ -2181,19 +2187,35 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, order = next_order(&suitable_orders, order); } } else { + suitable_order = 0; pages = 1; folio = shmem_alloc_folio(gfp, 0, info, index); } - if (!folio) + if (!folio) { + if (suitable_order < 0) + trace_shmem_alloc_folio_finish(inode, orig_index, + requested_order, + suitable_order, -1, + SHMEM_ALLOC_FILTER, 0, + -ENOMEM); + else + trace_shmem_alloc_folio_finish(inode, orig_index, + requested_order, + suitable_order, -1, + SHMEM_ALLOC_ALLOC, -ENOMEM, + -ENOMEM); return ERR_PTR(-ENOMEM); + } allocated: __folio_set_locked(folio); __folio_set_swapbacked(folio); gfp &= GFP_RECLAIM_MASK; + stage = SHMEM_ALLOC_CHARGE; error = mem_cgroup_charge(folio, fault_mm, gfp); if (error) { + charge_error = error; if (xa_find(&mapping->i_pages, &index, index + pages - 1, XA_PRESENT)) { error = -EEXIST; @@ -2208,10 +2230,12 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, goto unlock; } + stage = SHMEM_ALLOC_CACHE; error = shmem_add_to_page_cache(folio, mapping, index, NULL, gfp); if (error) goto unlock; + stage = SHMEM_ALLOC_ACCOUNT; error = shmem_inode_acct_blocks(inode, pages); if (error) { struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); @@ -2242,9 +2266,17 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, shmem_recalc_inode(inode, pages, 0); folio_add_lru(folio); + if (folio_order(folio) < requested_order) + trace_shmem_alloc_folio_finish(inode, orig_index, + requested_order, suitable_order, + folio_order(folio), + SHMEM_ALLOC_DONE, 0, 0); return folio; unlock: + trace_shmem_alloc_folio_finish(inode, orig_index, requested_order, + suitable_order, folio_order(folio), stage, + charge_error ?: error, error); folio_unlock(folio); folio_put(folio); return ERR_PTR(error); -- 2.25.1