From: Jonathan Flynn The parts of a direct-mode WRITE that are not O_DIRECT (the misaligned start and end of a split WRITE, or the whole of a WRITE that is not split) are issued IOCB_DONTCACHE when the file system sets FOP_DONTCACHE. That suits the workloads a direct mode is chosen for, but it is a policy, not a requirement: a workload that reads back or rewrites what it just wrote is better served by those pages staying cached. Add /sys/kernel/debug/nfsd/direct_misaligned_dontcache (default Y). Set to N, those parts are issued as normal buffered I/O. The O_DIRECT middle keeps IOCB_DONTCACHE either way. With N, a WRITE that is not split, including one whose payload is misaligned in memory, is cached in full. The knob sits beside direct_misaligned_min_bytes, which decides whether a misaligned WRITE is split at all; this knob decides how the parts that are not O_DIRECT are cached. Signed-off-by: Jonathan Flynn [snitzer: switched from a module parameter to a debugfs knob, moved next to direct_misaligned_min_bytes, documented in nfsd-io-modes.rst] Assisted-by: LLM Signed-off-by: Mike Snitzer --- .../filesystems/nfs/nfsd-io-modes.rst | 19 +++++++++++++------ fs/nfsd/debugfs.c | 13 +++++++++++++ fs/nfsd/nfsd.h | 1 + fs/nfsd/vfs.c | 12 ++++++++---- 4 files changed, 35 insertions(+), 10 deletions(-) diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst index eab35a9bbdb93..2679d85152dfc 100644 --- a/Documentation/filesystems/nfs/nfsd-io-modes.rst +++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst @@ -131,15 +131,17 @@ Misaligned WRITE: offset, so the O_DIRECT segment never shares a page with a buffered one; a start or end that is already DIO-aligned gets no buffered segment at all. If the filesystem supports FOP_DONTCACHE, the - misaligned segments are marked DONTCACHE. - So is the O_DIRECT segment, where the flag has no effect unless the - filesystem services it with buffered IO instead, as XFS does when it - cannot invalidate page cache that overlaps the segment. The - iomap_dio_invalidate_fail trace event reports such a fallback. + O_DIRECT segment is marked DONTCACHE, and so are the misaligned + segments unless direct_misaligned_dontcache is N (below). The flag + has no effect on the O_DIRECT segment unless the filesystem services + it with buffered IO instead, as XFS does when it cannot invalidate + page cache that overlaps the segment. The iomap_dio_invalidate_fail + trace event reports such a fallback. If NFSD does not split a misaligned WRITE, it issues the whole WRITE as a single DONTCACHE buffered IO (normal buffered IO if the - filesystem lacks FOP_DONTCACHE). NFSD does not split a WRITE when: + filesystem lacks FOP_DONTCACHE or direct_misaligned_dontcache is N). + NFSD does not split a WRITE when: - the filesystem advertises no DIO alignment; - the WRITE is smaller than the larger of the offset and memory @@ -150,6 +152,11 @@ Misaligned WRITE: - the WRITE payload is not aligned in memory to the block device's dma_alignment, so the middle cannot be O_DIRECT either. + Writing N to /sys/kernel/debug/nfsd/direct_misaligned_dontcache + (default Y) issues the start and end segments, and a WRITE that is + not split, as normal buffered IO instead of DONTCACHE, which suits a + workload that reads back or rewrites what it just wrote. + Tracing: The nfsd_read_direct trace event shows how NFSD expands any misaligned READ to the next DIO-aligned block (on either end of the diff --git a/fs/nfsd/debugfs.c b/fs/nfsd/debugfs.c index 02b0369f17648..80f53a4ada7bf 100644 --- a/fs/nfsd/debugfs.c +++ b/fs/nfsd/debugfs.c @@ -152,6 +152,19 @@ void nfsd_debugfs_init(void) */ debugfs_create_u32("direct_misaligned_min_bytes", 0644, nfsd_top_dir, &nfsd_direct_misaligned_min_bytes); + + /* + * /sys/kernel/debug/nfsd/direct_misaligned_dontcache + * + * Y: the start and end segments of a split direct-mode WRITE, and + * a WRITE that is not split, use DONTCACHE buffered IO when the + * file system supports FOP_DONTCACHE + * N: those segments use normal buffered IO + * + * The default value of this setting is Y. + */ + debugfs_create_bool("direct_misaligned_dontcache", 0644, nfsd_top_dir, + &nfsd_direct_misaligned_dontcache); #ifdef CONFIG_NFSD_V4 debugfs_create_bool("delegated_timestamps", 0644, nfsd_top_dir, &nfsd_delegts_enabled); diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index db81620bd5168..b520b51692b09 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -143,6 +143,7 @@ enum { extern u64 nfsd_io_cache_read __read_mostly; extern u64 nfsd_io_cache_write __read_mostly; extern u32 nfsd_direct_misaligned_min_bytes __read_mostly; +extern bool nfsd_direct_misaligned_dontcache __read_mostly; bool nfsd_v4client(struct svc_rqst *rqstp); diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index a419eee401d9d..b596bd61f8cb7 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -55,6 +55,7 @@ bool nfsd_disable_splice_read __read_mostly; u64 nfsd_io_cache_read __read_mostly = NFSD_IO_BUFFERED; u64 nfsd_io_cache_write __read_mostly = NFSD_IO_BUFFERED; u32 nfsd_direct_misaligned_min_bytes __read_mostly = 8192; +bool nfsd_direct_misaligned_dontcache __read_mostly = true; /** * nfserrno - Map Linux errnos to NFS errnos @@ -1325,11 +1326,13 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec, size_t prefix, middle, suffix; loff_t offset = iocb->ki_pos; unsigned int dontcache_flags = 0; + unsigned int buffered_flags; unsigned int nsegs = 0; u32 cut; if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE) dontcache_flags = IOCB_DONTCACHE; + buffered_flags = nfsd_direct_misaligned_dontcache ? dontcache_flags : 0; /* * Check if direct I/O is feasible for this write request. @@ -1368,7 +1371,7 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec, if (prefix) { nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total, 0, prefix, iocb); - segments[nsegs++].flags |= dontcache_flags; + segments[nsegs++].flags |= buffered_flags; } nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, @@ -1392,7 +1395,7 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec, if (suffix) { nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total, prefix + middle, suffix, iocb); - segments[nsegs++].flags |= dontcache_flags; + segments[nsegs++].flags |= buffered_flags; } return nsegs; @@ -1400,11 +1403,12 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec, no_dio: /* * Issue the whole WRITE as a single buffered segment, uncached when - * the file system supports FOP_DONTCACHE. + * the file system supports FOP_DONTCACHE and + * direct_misaligned_dontcache is set. */ nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0, total, iocb); - segments[0].flags |= dontcache_flags; + segments[0].flags |= buffered_flags; return 1; } -- 2.52.0