Performance when clearing with string instructions (x86-64-stosq and similar) can vary significantly based on the chunk-size used. $ perf bench mem memset -k 4KB -s 4GB -f x86-64-stosq # Running 'mem/memset' benchmark: # function 'x86-64-stosq' (movsq-based memset() in arch/x86/lib/memset_64.S) # Copying 4GB bytes ... 13.748208 GB/sec $ perf bench mem memset -k 2MB -s 4GB -f x86-64-stosq # Running 'mem/memset' benchmark: # function 'x86-64-stosq' (movsq-based memset() in # arch/x86/lib/memset_64.S) # Copying 4GB bytes ... 15.067900 GB/sec $ perf bench mem memset -k 1GB -s 4GB -f x86-64-stosq # Running 'mem/memset' benchmark: # function 'x86-64-stosq' (movsq-based memset() in arch/x86/lib/memset_64.S) # Copying 4GB bytes ... 38.104311 GB/sec (Both on AMD Milan.) With a change in chunk-size from 4KB to 1GB, we see the performance go from 13.7 GB/sec to 38.1 GB/sec. For the chunk-size of 2MB the change isn't quite as drastic but it is worth adding a clear_page() variant that can handle contiguous page-extents. Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- Notes: - get rid of ARCH_PAGE_CONTIG_NR. Subsequent patches add this constant (which was relevant for cooperative preemption models) to common code. arch/x86/include/asm/page_64.h | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 6157bf46590e..043e57c4ad12 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -44,8 +44,9 @@ void __clear_pages_unrolled(void *page); KCFI_REFERENCE(__clear_pages_unrolled); /** - * clear_page() - clear a page using a kernel virtual address. - * @addr: address of kernel page + * clear_pages() - clear a page range using a kernel virtual address. + * @addr: start address of kernel page range + * @npages: number of pages * * Switch between three implementations of page clearing based on CPU * capabilities: @@ -66,11 +67,11 @@ KCFI_REFERENCE(__clear_pages_unrolled); * * Does absolutely no exception handling. */ -static inline void clear_page(void *addr) +static inline void clear_pages(void *addr, unsigned int npages) { - u64 len = PAGE_SIZE; + u64 len = npages * PAGE_SIZE; /* - * Clean up KMSAN metadata for the page being cleared. The assembly call + * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @addr, so we perform unpoisoning before it. */ kmsan_unpoison_memory(addr, len); @@ -81,6 +82,12 @@ static inline void clear_page(void *addr) : "a" (0) : "cc", "memory"); } +#define clear_pages clear_pages + +static inline void clear_page(void *addr) +{ + clear_pages(addr, 1); +} void copy_page(void *to, void *from); KCFI_REFERENCE(copy_page); -- 2.31.1