| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/07/28 19:02 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": [
"EXT4_FS",
"XFS_FS",
"BTRFS_FS",
"F2FS_FS",
"FS_DAX"
],
"FocusSymbols": [
"iomap_iter",
"iomap_iter_continue",
"__iomap_dio_read_simple",
"iomap_dio_rw",
"iomap_file_buffered_write",
"iomap_read_folio",
"iomap_readahead",
"iomap_fiemap",
"iomap_seek_hole",
"iomap_seek_data",
"dax_iomap_rw",
"dax_iomap_fault"
],
"Reasoning": "The patch refactors the iomap subsystem to use a single `iomap_iter_next_fn` callback instead of a `struct iomap_ops` with `iomap_begin` and `iomap_end` functions. This changes the control flow and calling convention for all iomap operations across many filesystems (ext4, xfs, btrfs, f2fs, etc.). It also introduces `__iomap_dio_read_simple` and `iomap_iter_continue`. These are core functional changes to the I/O path that should be fuzzed to ensure no regressions or bugs were introduced in the iteration logic.",
"WorthFuzzing": true
} |
| 1/1 | 2026/07/28 19:02 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 5bd95dfd710ae16d6150e042e0a6fd5c6d20fdac\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Jul 28 19:02:00 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/filesystems/iomap/design.rst b/Documentation/filesystems/iomap/design.rst\nindex 0f7672676c0ba..bed8a5beac819 100644\n--- a/Documentation/filesystems/iomap/design.rst\n+++ b/Documentation/filesystems/iomap/design.rst\n@@ -75,7 +75,10 @@ At a high level, an iomap operation `looks like this\n \n 1. For each byte in the operation range...\n \n- 1. Obtain a space mapping via ``-\u003eiomap_begin``\n+ 1. Obtain the next space mapping via the ``iomap_iter_next_fn`` callback.\n+ From the second iteration onwards this same callback first finishes\n+ the previous mapping (committing or unreserving space as needed)\n+ and then produces the next one.\n \n 2. For each sub-unit of work...\n \n@@ -86,7 +89,13 @@ At a high level, an iomap operation `looks like this\n \n 3. Increment operation cursor\n \n- 4. Release the mapping via ``-\u003eiomap_end``, if necessary\n+iomap repeats this until the range is fully consumed. The ``iomap_iter_next_fn``\n+callback returns ``1`` while there is more of the range left to process,\n+``0`` once it is fully consumed, or a negative errno on error.\n+Filesystems rarely implement this callback by hand. The ``iomap_iter_next``\n+helper implements the finish-then-produce sequence in terms of two smaller\n+callbacks, ``begin`` and ``end``. See `The Mapping Callback`_ below for more\n+info.\n \n Each iomap operation will be covered in more detail below.\n This library was covered previously by an `LWN article\n@@ -189,7 +198,7 @@ The fields are as follows:\n * **IOMAP_DELALLOC**: A promise to allocate space at a later time\n (\"delayed allocation\").\n If the filesystem returns IOMAP_F_NEW here and the write fails, the\n- ``-\u003eiomap_end`` function must delete the reservation.\n+ ``end`` function must delete the reservation.\n The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.\n \n * **IOMAP_MAPPED**: The file range maps to specific space on the\n@@ -208,12 +217,11 @@ The fields are as follows:\n \n * **IOMAP_INLINE**: The file range maps to the memory buffer\n specified by ``inline_data``.\n- For write operation, the ``-\u003eiomap_end`` function presumably\n- handles persisting the data.\n+ For write operation, the ``end`` function must persist the data.\n The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.\n \n * ``flags`` describe the status of the space mapping.\n- These flags should be set by the filesystem in ``-\u003eiomap_begin``:\n+ These flags should be set by the filesystem in ``begin``:\n \n * **IOMAP_F_NEW**: The space under the mapping is newly allocated.\n Areas that will not be written to must be zeroed.\n@@ -262,15 +270,15 @@ The fields are as follows:\n update.\n \n These flags can be set by iomap itself during file operations.\n- The filesystem should supply an ``-\u003eiomap_end`` function if it needs\n+ The filesystem should supply an ``end`` function if it needs\n to observe these flags:\n \n * **IOMAP_F_SIZE_CHANGED**: The file size has changed as a result of\n using this mapping.\n \n * **IOMAP_F_STALE**: The mapping was found to be stale.\n- iomap will call ``-\u003eiomap_end`` on this mapping and then\n- ``-\u003eiomap_begin`` to obtain a new mapping.\n+ iomap will call ``end`` on this mapping and then ``begin`` to obtain a\n+ new mapping.\n \n Currently, these flags are only set by pagecache operations.\n \n@@ -289,41 +297,103 @@ The fields are as follows:\n \n * ``private`` is a pointer to `filesystem-private information\n \u003chttps://lore.kernel.org/all/20180619164137.13720-7-hch@lst.de/\u003e`_.\n- This value will be passed unchanged to ``-\u003eiomap_end``.\n+ This value will be passed unchanged to ``end``.\n \n * ``validity_cookie`` is a magic freshness value set by the filesystem\n that should be used to detect stale mappings.\n For pagecache operations this is critical for correct operation\n because page faults can occur, which implies that filesystem locks\n- should not be held between ``-\u003eiomap_begin`` and ``-\u003eiomap_end``.\n+ should not be held between ``begin`` and ``end``.\n Filesystems with completely static mappings need not set this value.\n Only pagecache operations revalidate mappings; see the section about\n ``iomap_valid`` for details.\n \n-``struct iomap_ops``\n+The Mapping Callback\n --------------------\n \n-Every iomap function requires the filesystem to pass an operations\n-structure to obtain a mapping and (optionally) to release the mapping:\n+Every iomap operation takes an ``iomap_iter_next_fn`` callback from the\n+filesystem. iomap calls it once per iteration of the file range:\n \n .. code-block:: c\n \n- struct iomap_ops {\n- int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,\n- unsigned flags, struct iomap *iomap,\n- struct iomap *srcmap);\n+ typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,\n+ struct iomap *iomap, struct iomap *srcmap);\n \n- int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,\n- ssize_t written, unsigned flags,\n- struct iomap *iomap);\n- };\n+``iomap_iter_next_fn``\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+This must finish the previous mapping, if any, and then produce the next\n+mapping for the current iteration position described by ``iter``.\n+The mapping is returned through ``iomap`` (and through ``srcmap`` for\n+operations that read from one mapping while writing to another; see\n+``begin`` below).\n+\n+The callback returns ``1`` to continue iterating, ``0`` once the file\n+range has been fully consumed, or a negative errno on error.\n+\n+Most filesystems are not expected to implement all of these behaviors\n+in this callback themselves. They should instead call ``iomap_iter_next`` as\n+described below.\n+\n+``iomap_iter_next``\n+~~~~~~~~~~~~~~~~~~~\n+\n+Filesystems rarely need a hand-written ``iomap_iter_next_fn`` callback. The\n+``iomap_iter_next`` helper implements the finish-then-produce sequence in\n+terms of two smaller callbacks, ``begin`` and ``end``, so most\n+``iomap_iter_next_fn`` implementations are simply:\n+\n+.. code-block:: c\n+\n+ static int my_iomap_next(const struct iomap_iter *iter,\n+ struct iomap *iomap, struct iomap *srcmap)\n+ {\n+ return iomap_iter_next(iter, iomap, srcmap,\n+ my_iomap_begin, my_iomap_end);\n+ }\n+\n+This boilerplate is normally generated with the ``DEFINE_IOMAP_ITER_NEXT()``\n+and ``DEFINE_IOMAP_ITER_NEXT_END()`` macros rather than written by hand. Use\n+``DEFINE_IOMAP_ITER_NEXT()`` for the common case with no ``end`` callback, and\n+``DEFINE_IOMAP_ITER_NEXT_END()`` when an ``end`` callback is needed:\n+\n+.. code-block:: c\n+\n+ /* no end() callback */\n+ static DEFINE_IOMAP_ITER_NEXT(my_iomap_next, my_iomap_begin);\n+\n+ /* with an end() callback */\n+ static DEFINE_IOMAP_ITER_NEXT_END(my_iomap_next, my_iomap_begin,\n+ my_iomap_end);\n+\n+The macro generates a function with external linkage. Prefix the invocation\n+with ``static`` for a file-local callback, or leave it non-``static`` and add\n+a matching declaration to a header when the callback is referenced from\n+another file.\n+\n+``end`` may be ``NULL`` when the filesystem has nothing to finish.\n+The two callbacks have these prototypes:\n+\n+.. code-block:: c\n+\n+ typedef int (*iomap_begin_fn)(struct inode *inode, loff_t pos,\n+ loff_t length, unsigned flags,\n+ struct iomap *iomap, struct iomap *srcmap);\n+\n+ typedef int (*iomap_end_fn)(struct inode *inode, loff_t pos,\n+ loff_t length, ssize_t written,\n+ unsigned flags, struct iomap *iomap);\n+\n+``iomap_iter_next`` is an inline helper, so when it is called with fixed\n+``begin`` and ``end`` functions the compiler can inline both into the\n+filesystem's ``iomap_iter_next_fn`` callback, keeping indirect calls out of the\n+iteration hot path. The two callbacks are described next.\n \n-``-\u003eiomap_begin``\n-~~~~~~~~~~~~~~~~~\n+``begin``\n+~~~~~~~~~\n \n-iomap operations call ``-\u003eiomap_begin`` to obtain one file mapping for\n-the range of bytes specified by ``pos`` and ``length`` for the file\n-``inode``.\n+The ``begin`` callback obtains one file mapping for the range of bytes\n+specified by ``pos`` and ``length`` for the file ``inode``.\n This mapping should be returned through the ``iomap`` pointer.\n The mapping must cover at least the first byte of the supplied file\n range, but it does not need to cover the entire requested range.\n@@ -377,18 +447,18 @@ information via ``srcmap``.\n Only pagecache and fsdax operations support reading from one mapping and\n writing to another.\n \n-``-\u003eiomap_end``\n-~~~~~~~~~~~~~~~\n+``end``\n+~~~~~~~\n \n-After the operation completes, the ``-\u003eiomap_end`` function, if present,\n-is called to signal that iomap is finished with a mapping.\n+After the operation completes, the ``end`` callback, if present, is called to\n+signal that iomap is finished with a mapping.\n Typically, implementations will use this function to tear down any\n-context that were set up in ``-\u003eiomap_begin``.\n+context that was set up in ``begin``.\n For example, a write might wish to commit the reservations for the bytes\n that were operated upon and unreserve any space that was not operated\n upon.\n ``written`` might be zero if no bytes were touched.\n-``flags`` will contain the same value passed to ``-\u003eiomap_begin``.\n+``flags`` will contain the same value passed to ``begin``.\n iomap ops for reads are not likely to need to supply this function.\n \n Both functions should return a negative errno code on error, or zero on\n@@ -421,8 +491,8 @@ iomap is concerned:\n accessing the folio until writeback is underway.\n \n * The **lower** level primitive is taken by the filesystem in the\n- ``-\u003eiomap_begin`` and ``-\u003eiomap_end`` functions to coordinate\n- access to the file space mapping information.\n+ ``begin`` and ``end`` functions to coordinate access to the file space\n+ mapping information.\n The fields of the iomap object should be filled out while holding\n this primitive.\n The upper level synchronization primitive, if any, remains held\ndiff --git a/Documentation/filesystems/iomap/operations.rst b/Documentation/filesystems/iomap/operations.rst\nindex da982ca7e4137..a5784dee08509 100644\n--- a/Documentation/filesystems/iomap/operations.rst\n+++ b/Documentation/filesystems/iomap/operations.rst\n@@ -17,6 +17,12 @@ Supported File Operations\n Below are a discussion of the high level file operations that iomap\n implements.\n \n+Each operation takes an ``iomap_iter_next_fn`` callback that supplies the file\n+mappings, as described in the iomap design document. The per-operation\n+``flags`` documented below are passed to that callback; references to\n+``begin`` and ``end`` name the two steps a typical callback is built from\n+via ``iomap_iter_next``.\n+\n Buffered I/O\n ============\n \n@@ -90,10 +96,10 @@ iomap calls these functions:\n \u003chttps://lore.kernel.org/all/20180619164137.13720-6-hch@lst.de/\u003e`_\n that was set up by ``-\u003eget_folio``.\n \n- - ``iomap_valid``: The filesystem may not hold locks between\n- ``-\u003eiomap_begin`` and ``-\u003eiomap_end`` because pagecache operations\n- can take folio locks, fault on userspace pages, initiate writeback\n- for memory reclamation, or engage in other time-consuming actions.\n+ - ``iomap_valid``: The filesystem may not hold locks between ``begin`` and\n+ ``end`` because pagecache operations can take folio locks, fault on\n+ userspace pages, initiate writeback for memory reclamation, or engage in\n+ other time-consuming actions.\n If a file's space mapping data are mutable, it is possible that the\n mapping for a particular pagecache folio can `change in the time it\n takes\n@@ -114,12 +120,12 @@ iomap calls these functions:\n If the mapping is not valid, the mapping will be sampled again.\n \n To support making the validity decision, the filesystem's\n- ``-\u003eiomap_begin`` function may set ``struct iomap::validity_cookie``\n+ ``begin`` function may set ``struct iomap::validity_cookie``\n at the same time that it populates the other iomap fields.\n A simple validation cookie implementation is a sequence counter.\n If the filesystem bumps the sequence counter every time it modifies\n the inode's extent map, it can be placed in the ``struct\n- iomap::validity_cookie`` during ``-\u003eiomap_begin``.\n+ iomap::validity_cookie`` during ``begin``.\n If the value in the cookie is found to be different to the value\n the filesystem holds when the mapping is passed back to\n ``-\u003eiomap_valid``, then the iomap should considered stale and the\n@@ -199,7 +205,7 @@ Buffered Readahead and Reads\n The ``iomap_readahead`` function initiates readahead to the pagecache.\n The ``iomap_read_folio`` function reads one folio's worth of data into\n the pagecache.\n-The ``flags`` argument to ``-\u003eiomap_begin`` will be set to zero.\n+The ``flags`` argument to ``begin`` will be set to zero.\n The pagecache takes whatever locks it needs before calling the\n filesystem.\n \n@@ -231,7 +237,7 @@ Buffered Writes\n The ``iomap_file_buffered_write`` function writes an ``iocb`` to the\n pagecache.\n ``IOMAP_WRITE`` or ``IOMAP_WRITE`` | ``IOMAP_NOWAIT`` will be passed as\n-the ``flags`` argument to ``-\u003eiomap_begin``.\n+the ``flags`` argument to ``begin``.\n Callers commonly take ``i_rwsem`` in either shared or exclusive mode\n before calling this function.\n \n@@ -241,7 +247,7 @@ mmap Write Faults\n The ``iomap_page_mkwrite`` function handles a write fault to a folio in\n the pagecache.\n ``IOMAP_WRITE | IOMAP_FAULT`` will be passed as the ``flags`` argument\n-to ``-\u003eiomap_begin``.\n+to ``begin``.\n Callers commonly take the mmap ``invalidate_lock`` in shared or\n exclusive mode before calling this function.\n \n@@ -256,7 +262,7 @@ such `reservations\n \u003chttps://lore.kernel.org/linux-xfs/20220817093627.GZ3600936@dread.disaster.area/\u003e`_\n because writeback will not consume the reservation.\n The ``iomap_write_delalloc_release`` can be called from a\n-``-\u003eiomap_end`` function to find all the clean areas of the folios\n+``end`` function to find all the clean areas of the folios\n caching a fresh (``IOMAP_F_NEW``) delalloc mapping.\n It takes the ``invalidate_lock``.\n \n@@ -274,7 +280,7 @@ Filesystems can call ``iomap_zero_range`` to perform zeroing of the\n pagecache for non-truncation file operations that are not aligned to\n the fsblock size.\n ``IOMAP_ZERO`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``begin``.\n Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive\n mode before calling this function.\n \n@@ -285,7 +291,7 @@ Filesystems can call ``iomap_file_unshare`` to force a file sharing\n storage with another file to preemptively copy the shared data to newly\n allocate storage.\n ``IOMAP_WRITE | IOMAP_UNSHARE`` will be passed as the ``flags`` argument\n-to ``-\u003eiomap_begin``.\n+to ``begin``.\n Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive\n mode before calling this function.\n \n@@ -298,7 +304,7 @@ operation.\n ``truncate_setsize`` or ``truncate_pagecache`` will take care of\n everything after the EOF block.\n ``IOMAP_ZERO`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``begin``.\n Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive\n mode before calling this function.\n \n@@ -341,8 +347,8 @@ The fields are as follows:\n though it will `reuse mappings\n \u003chttps://lore.kernel.org/all/20231207072710.176093-15-hch@lst.de/\u003e`_\n for runs of contiguous dirty fsblocks within a folio.\n- Do not return ``IOMAP_INLINE`` mappings here; the ``-\u003eiomap_end``\n- function must deal with persisting written data.\n+ Do not return ``IOMAP_INLINE`` mappings here; the ``end`` function must\n+ deal with persisting written data.\n Do not return ``IOMAP_DELALLOC`` mappings here; iomap currently\n requires mapping to allocated space.\n Filesystems can skip a potentially expensive mapping lookup if the\n@@ -428,7 +434,7 @@ writes for files.\n .. code-block:: c\n \n ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n- const struct iomap_ops *ops,\n+ iomap_iter_next_fn iomap_next,\n const struct iomap_dio_ops *dops,\n unsigned int dio_flags, void *private,\n size_t done_before);\n@@ -511,7 +517,7 @@ Return Values\n * ``-ENOTBLK``: Fall back to buffered I/O.\n iomap itself will return this value if it cannot invalidate the page\n cache before issuing the I/O to storage.\n- The ``-\u003eiomap_begin`` or ``-\u003eiomap_end`` functions may also return\n+ The ``begin`` or ``end`` functions may also return\n this value.\n \n * ``-EIOCBQUEUED``: The asynchronous direct I/O request has been\n@@ -526,7 +532,7 @@ A direct I/O read initiates a read I/O from the storage device to the\n caller's buffer.\n Dirty parts of the pagecache are flushed to storage before initiating\n the read io.\n-The ``flags`` value for ``-\u003eiomap_begin`` will be ``IOMAP_DIRECT`` with\n+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT`` with\n any combination of the following enhancements:\n \n * ``IOMAP_NOWAIT``, as defined previously.\n@@ -542,7 +548,7 @@ caller's buffer.\n Dirty parts of the pagecache are flushed to storage before initiating\n the write io.\n The pagecache is invalidated both before and after the write io.\n-The ``flags`` value for ``-\u003eiomap_begin`` will be ``IOMAP_DIRECT |\n+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT |\n IOMAP_WRITE`` with any combination of the following enhancements:\n \n * ``IOMAP_NOWAIT``, as defined previously.\n@@ -644,7 +650,7 @@ fsdax Reads\n \n A fsdax read performs a memcpy from storage device to the caller's\n buffer.\n-The ``flags`` value for ``-\u003eiomap_begin`` will be ``IOMAP_DAX`` with any\n+The ``flags`` value for ``begin`` will be ``IOMAP_DAX`` with any\n combination of the following enhancements:\n \n * ``IOMAP_NOWAIT``, as defined previously.\n@@ -657,7 +663,7 @@ fsdax Writes\n \n A fsdax write initiates a memcpy to the storage device from the caller's\n buffer.\n-The ``flags`` value for ``-\u003eiomap_begin`` will be ``IOMAP_DAX |\n+The ``flags`` value for ``begin`` will be ``IOMAP_DAX |\n IOMAP_WRITE`` with any combination of the following enhancements:\n \n * ``IOMAP_NOWAIT``, as defined previously.\n@@ -680,9 +686,9 @@ fsdax mmap Faults\n The ``dax_iomap_fault`` function handles read and write faults to fsdax\n storage.\n For a read fault, ``IOMAP_DAX | IOMAP_FAULT`` will be passed as the\n-``flags`` argument to ``-\u003eiomap_begin``.\n+``flags`` argument to ``begin``.\n For a write fault, ``IOMAP_DAX | IOMAP_FAULT | IOMAP_WRITE`` will be\n-passed as the ``flags`` argument to ``-\u003eiomap_begin``.\n+passed as the ``flags`` argument to ``begin``.\n \n Callers commonly hold the same locks as they do to call their iomap\n pagecache counterparts.\n@@ -692,8 +698,8 @@ fsdax Truncation, fallocate, and Unsharing\n \n For fsdax files, the following functions are provided to replace their\n iomap pagecache I/O counterparts.\n-The ``flags`` argument to ``-\u003eiomap_begin`` are the same as the\n-pagecache counterparts, with ``IOMAP_DAX`` added.\n+The ``flags`` argument to ``begin`` are the same as the pagecache counterparts,\n+with ``IOMAP_DAX`` added.\n \n * ``dax_file_unshare``\n * ``dax_zero_range``\n@@ -719,8 +725,7 @@ SEEK_DATA\n \n The ``iomap_seek_data`` function implements the SEEK_DATA \"whence\" value\n for llseek.\n-``IOMAP_REPORT`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.\n \n For unwritten mappings, the pagecache will be searched.\n Regions of the pagecache with a folio mapped and uptodate fsblocks\n@@ -734,8 +739,7 @@ SEEK_HOLE\n \n The ``iomap_seek_hole`` function implements the SEEK_HOLE \"whence\" value\n for llseek.\n-``IOMAP_REPORT`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.\n \n For unwritten mappings, the pagecache will be searched.\n Regions of the pagecache with no folio mapped, or a !uptodate fsblock\n@@ -750,8 +754,7 @@ Swap File Activation\n The ``iomap_swapfile_activate`` function finds all the base-page aligned\n regions in a file and sets them up as swap space.\n The file will be ``fsync()``'d before activation.\n-``IOMAP_REPORT`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.\n All mappings must be mapped or unwritten; cannot be dirty or shared, and\n cannot span multiple block devices.\n Callers must hold ``i_rwsem`` in exclusive mode; this is already\n@@ -767,8 +770,7 @@ FS_IOC_FIEMAP\n \n The ``iomap_fiemap`` function exports file extent mappings to userspace\n in the format specified by the ``FS_IOC_FIEMAP`` ioctl.\n-``IOMAP_REPORT`` will be passed as the ``flags`` argument to\n-``-\u003eiomap_begin``.\n+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.\n Callers commonly hold ``i_rwsem`` in shared mode before calling this\n function.\n \ndiff --git a/Documentation/filesystems/iomap/porting.rst b/Documentation/filesystems/iomap/porting.rst\nindex 3d49a32c0fff9..b995f426808a9 100644\n--- a/Documentation/filesystems/iomap/porting.rst\n+++ b/Documentation/filesystems/iomap/porting.rst\n@@ -50,8 +50,14 @@ Build the kernel, run fstests with the ``-g all`` option across a wide\n variety of your filesystem's supported configurations to build a\n baseline of which tests pass and which ones fail.\n \n-The recommended approach is first to implement ``-\u003eiomap_begin`` (and\n-``-\u003eiomap_end`` if necessary) to allow iomap to obtain a read-only\n+Every iomap operation is driven by an ``iomap_iter_next_fn`` callback.\n+Filesystems normally do not write one by hand: implement ``begin``\n+(and ``end`` if necessary) and generate the callback with the\n+``DEFINE_IOMAP_ITER_NEXT()`` macro, or ``DEFINE_IOMAP_ITER_NEXT_END()`` if an\n+``end`` callback is needed.\n+\n+The recommended approach is first to implement ``begin`` (and\n+``end`` if necessary) to allow iomap to obtain a read-only\n mapping of a file range.\n In most cases, this is a relatively trivial conversion of the existing\n ``get_block()`` function for read-only mappings.\n@@ -62,7 +68,7 @@ If FIEMAP is returning the correct information, it's a good sign that\n other read-only mapping operations will do the right thing.\n \n Next, modify the filesystem's ``get_block(create = false)``\n-implementation to use the new ``-\u003eiomap_begin`` implementation to map\n+implementation to use the new ``begin`` implementation to map\n file space for selected read operations.\n Hide behind a debugging knob the ability to switch on the iomap mapping\n functions for selected call paths.\n@@ -82,14 +88,14 @@ I/O path because of bufferheads.\n The buffered read I/O paths doesn't need to be converted yet, though the\n direct I/O read path should be converted in this phase.\n \n-At this point, you should look over your ``-\u003eiomap_begin`` function.\n+At this point, you should look over your ``begin`` function.\n If it switches between large blocks of code based on dispatching of the\n ``flags`` argument, you should consider breaking it up into\n per-operation iomap ops with smaller, more cohesive functions.\n XFS is a good example of this.\n \n The next thing to do is implement ``get_blocks(create == true)``\n-functionality in the ``-\u003eiomap_begin``/``-\u003eiomap_end`` methods.\n+functionality in the ``begin``/``end`` methods.\n It is strongly recommended to create separate mapping functions and\n iomap ops for write operations.\n Then convert the direct I/O write path to iomap, and start running fsx\ndiff --git a/block/fops.c b/block/fops.c\nindex 15783a6180dec..53fde467f6ee8 100644\n--- a/block/fops.c\n+++ b/block/fops.c\n@@ -453,9 +453,7 @@ static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-static const struct iomap_ops blkdev_iomap_ops = {\n-\t.iomap_begin\t\t= blkdev_iomap_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(blkdev_iomap_next, blkdev_iomap_begin);\n \n #ifdef CONFIG_BUFFER_HEAD\n static int blkdev_get_block(struct inode *inode, sector_t iblock,\n@@ -510,13 +508,13 @@ const struct address_space_operations def_blk_aops = {\n #else /* CONFIG_BUFFER_HEAD */\n static int blkdev_read_folio(struct file *file, struct folio *folio)\n {\n-\tiomap_bio_read_folio(folio, \u0026blkdev_iomap_ops);\n+\tiomap_bio_read_folio(folio, blkdev_iomap_next);\n \treturn 0;\n }\n \n static void blkdev_readahead(struct readahead_control *rac)\n {\n-\tiomap_bio_readahead(rac, \u0026blkdev_iomap_ops);\n+\tiomap_bio_readahead(rac, blkdev_iomap_next);\n }\n \n static ssize_t blkdev_writeback_range(struct iomap_writepage_ctx *wpc,\n@@ -707,7 +705,7 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter *from)\n \n static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *from)\n {\n-\treturn iomap_file_buffered_write(iocb, from, \u0026blkdev_iomap_ops, NULL,\n+\treturn iomap_file_buffered_write(iocb, from, blkdev_iomap_next, NULL,\n \t\t\tNULL);\n }\n \ndiff --git a/fs/btrfs/direct-io.c b/fs/btrfs/direct-io.c\nindex 460326d34143c..b9ff7f9e59f03 100644\n--- a/fs/btrfs/direct-io.c\n+++ b/fs/btrfs/direct-io.c\n@@ -798,10 +798,8 @@ static void btrfs_dio_submit_io(const struct iomap_iter *iter, struct bio *bio,\n \tbtrfs_submit_bbio(bbio, 0);\n }\n \n-static const struct iomap_ops btrfs_dio_iomap_ops = {\n-\t.iomap_begin = btrfs_dio_iomap_begin,\n-\t.iomap_end = btrfs_dio_iomap_end,\n-};\n+static DEFINE_IOMAP_ITER_NEXT_END(btrfs_dio_iomap_next, btrfs_dio_iomap_begin,\n+\t\t\t\t btrfs_dio_iomap_end);\n \n static const struct iomap_dio_ops btrfs_dio_ops = {\n \t.submit_io\t\t= btrfs_dio_submit_io,\n@@ -813,7 +811,7 @@ static ssize_t btrfs_dio_read(struct kiocb *iocb, struct iov_iter *iter,\n {\n \tstruct btrfs_dio_data data = { 0 };\n \n-\treturn iomap_dio_rw(iocb, iter, \u0026btrfs_dio_iomap_ops, \u0026btrfs_dio_ops,\n+\treturn iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, \u0026btrfs_dio_ops,\n \t\t\t IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, \u0026data, done_before);\n }\n \n@@ -822,7 +820,7 @@ static struct iomap_dio *btrfs_dio_write(struct kiocb *iocb, struct iov_iter *it\n {\n \tstruct btrfs_dio_data data = { 0 };\n \n-\treturn __iomap_dio_rw(iocb, iter, \u0026btrfs_dio_iomap_ops, \u0026btrfs_dio_ops,\n+\treturn __iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, \u0026btrfs_dio_ops,\n \t\t\t IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, \u0026data, done_before);\n }\n \ndiff --git a/fs/dax.c b/fs/dax.c\nindex 6d175cd47a99b..19e5bbe31b69f 100644\n--- a/fs/dax.c\n+++ b/fs/dax.c\n@@ -1492,7 +1492,7 @@ static int dax_unshare_iter(struct iomap_iter *iter)\n }\n \n int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_iter iter = {\n \t\t.inode\t\t= inode,\n@@ -1506,7 +1506,7 @@ int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n \t\treturn 0;\n \n \titer.len = min(len, size - pos);\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = dax_unshare_iter(\u0026iter);\n \treturn ret;\n }\n@@ -1584,7 +1584,7 @@ static int dax_zero_iter(struct iomap_iter *iter, bool *did_zero)\n }\n \n int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_iter iter = {\n \t\t.inode\t\t= inode,\n@@ -1594,14 +1594,14 @@ int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,\n \t};\n \tint ret;\n \n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = dax_zero_iter(\u0026iter, did_zero);\n \treturn ret;\n }\n EXPORT_SYMBOL_GPL(dax_zero_range);\n \n int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tunsigned int blocksize = i_blocksize(inode);\n \tunsigned int off = pos \u0026 (blocksize - 1);\n@@ -1609,7 +1609,7 @@ int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n \t/* Block boundary? Nothing to do */\n \tif (!off)\n \t\treturn 0;\n-\treturn dax_zero_range(inode, pos, blocksize - off, did_zero, ops);\n+\treturn dax_zero_range(inode, pos, blocksize - off, did_zero, next);\n }\n EXPORT_SYMBOL_GPL(dax_truncate_page);\n \n@@ -1734,7 +1734,8 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)\n * dax_iomap_rw - Perform I/O to a DAX file\n * @iocb:\tThe control block for this I/O\n * @iter:\tThe addresses to do I/O from or to\n- * @ops:\tiomap ops passed from the file system\n+ * @next:\tCallback passed from the file system for advancing to next\n+ *\t\titeration\n *\n * This function performs read and write operations to directly mapped\n * persistent memory. The callers needs to take care of read/write exclusion\n@@ -1742,7 +1743,7 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)\n */\n ssize_t\n dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_iter iomi = {\n \t\t.inode\t\t= iocb-\u003eki_filp-\u003ef_mapping-\u003ehost,\n@@ -1769,7 +1770,7 @@ dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,\n \tif (iocb-\u003eki_flags \u0026 IOCB_NOWAIT)\n \t\tiomi.flags |= IOMAP_NOWAIT;\n \n-\twhile ((ret = iomap_iter(\u0026iomi, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iomi, next)) \u003e 0)\n \t\tiomi.status = dax_iomap_iter(\u0026iomi, iter);\n \n \tdone = iomi.pos - iocb-\u003eki_pos;\n@@ -1897,7 +1898,7 @@ static vm_fault_t dax_fault_iter(struct vm_fault *vmf,\n }\n \n static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,\n-\t\t\t int *iomap_errp, const struct iomap_ops *ops)\n+\t\t\t int *iomap_errp, iomap_iter_next_fn next)\n {\n \tstruct address_space *mapping = vmf-\u003evma-\u003evm_file-\u003ef_mapping;\n \tXA_STATE(xas, \u0026mapping-\u003ei_pages, vmf-\u003epgoff);\n@@ -1942,7 +1943,7 @@ static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,\n \t\tgoto unlock_entry;\n \t}\n \n-\twhile ((error = iomap_iter(\u0026iter, ops)) \u003e 0) {\n+\twhile ((error = iomap_iter(\u0026iter, next)) \u003e 0) {\n \t\tif (WARN_ON_ONCE(iomap_length(\u0026iter) \u003c PAGE_SIZE)) {\n \t\t\titer.status = -EIO;\t/* fs corruption? */\n \t\t\tcontinue;\n@@ -2007,7 +2008,7 @@ static bool dax_fault_check_fallback(struct vm_fault *vmf, struct xa_state *xas,\n }\n \n static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n-\t\t\t const struct iomap_ops *ops)\n+\t\t\t iomap_iter_next_fn next)\n {\n \tstruct address_space *mapping = vmf-\u003evma-\u003evm_file-\u003ef_mapping;\n \tXA_STATE_ORDER(xas, \u0026mapping-\u003ei_pages, vmf-\u003epgoff, PMD_ORDER);\n@@ -2064,7 +2065,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n \t}\n \n \titer.pos = (loff_t)xas.xa_index \u003c\u003c PAGE_SHIFT;\n-\twhile (iomap_iter(\u0026iter, ops) \u003e 0) {\n+\twhile (iomap_iter(\u0026iter, next) \u003e 0) {\n \t\tif (iomap_length(\u0026iter) \u003c PMD_SIZE)\n \t\t\tcontinue; /* actually breaks out of the loop */\n \n@@ -2086,7 +2087,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n }\n #else\n static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n-\t\t\t const struct iomap_ops *ops)\n+\t\t\t iomap_iter_next_fn next)\n {\n \treturn VM_FAULT_FALLBACK;\n }\n@@ -2098,7 +2099,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n * @order: Order of the page to fault in\n * @pfnp: PFN to insert for synchronous faults if fsync is required\n * @iomap_errp: Storage for detailed error code in case of error\n- * @ops: Iomap ops passed from the file system\n+ * @next: The iomap iteration function for the filesystem\n *\n * When a page fault occurs, filesystems may call this helper in\n * their fault handler for DAX files. dax_iomap_fault() assumes the caller\n@@ -2107,12 +2108,12 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,\n */\n vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,\n \t\t\tunsigned long *pfnp, int *iomap_errp,\n-\t\t\tconst struct iomap_ops *ops)\n+\t\t\tiomap_iter_next_fn next)\n {\n \tif (order == 0)\n-\t\treturn dax_iomap_pte_fault(vmf, pfnp, iomap_errp, ops);\n+\t\treturn dax_iomap_pte_fault(vmf, pfnp, iomap_errp, next);\n \telse if (order == PMD_ORDER)\n-\t\treturn dax_iomap_pmd_fault(vmf, pfnp, ops);\n+\t\treturn dax_iomap_pmd_fault(vmf, pfnp, next);\n \telse\n \t\treturn VM_FAULT_FALLBACK;\n }\n@@ -2240,7 +2241,7 @@ static int dax_range_compare_iter(struct iomap_iter *it_src,\n \n int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,\n \t\tstruct inode *dst, loff_t dstoff, loff_t len, bool *same,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_iter src_iter = {\n \t\t.inode\t\t= src,\n@@ -2256,8 +2257,8 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,\n \t};\n \tint ret, status;\n \n-\twhile ((ret = iomap_iter(\u0026src_iter, ops)) \u003e 0 \u0026\u0026\n-\t (ret = iomap_iter(\u0026dst_iter, ops)) \u003e 0) {\n+\twhile ((ret = iomap_iter(\u0026src_iter, next)) \u003e 0 \u0026\u0026\n+\t (ret = iomap_iter(\u0026dst_iter, next)) \u003e 0) {\n \t\tstatus = dax_range_compare_iter(\u0026src_iter, \u0026dst_iter,\n \t\t\t\tmin(src_iter.len, dst_iter.len), same);\n \t\tif (status \u003c 0)\n@@ -2270,9 +2271,10 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,\n int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\t\t struct file *file_out, loff_t pos_out,\n \t\t\t loff_t *len, unsigned int remap_flags,\n-\t\t\t const struct iomap_ops *ops)\n+\t\t\t iomap_iter_next_fn next)\n {\n \treturn __generic_remap_file_range_prep(file_in, pos_in, file_out,\n-\t\t\t\t\t pos_out, len, remap_flags, ops);\n+\t\t\t\t\t pos_out, len, remap_flags,\n+\t\t\t\t\t next);\n }\n EXPORT_SYMBOL_GPL(dax_remap_file_range_prep);\ndiff --git a/fs/erofs/data.c b/fs/erofs/data.c\nindex 9aa48c8d67d12..d965ed015214f 100644\n--- a/fs/erofs/data.c\n+++ b/fs/erofs/data.c\n@@ -380,10 +380,8 @@ static int erofs_iomap_end(struct inode *inode, loff_t pos, loff_t length,\n \treturn written;\n }\n \n-static const struct iomap_ops erofs_iomap_ops = {\n-\t.iomap_begin = erofs_iomap_begin,\n-\t.iomap_end = erofs_iomap_end,\n-};\n+static DEFINE_IOMAP_ITER_NEXT_END(erofs_iomap_next, erofs_iomap_begin,\n+\t\t\t\t erofs_iomap_end);\n \n int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \t\t u64 start, u64 len)\n@@ -392,9 +390,9 @@ int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \t\tif (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))\n \t\t\treturn -EOPNOTSUPP;\n \t\treturn iomap_fiemap(inode, fieinfo, start, len,\n-\t\t\t\t \u0026z_erofs_iomap_report_ops);\n+\t\t\t\t z_erofs_iomap_next_report);\n \t}\n-\treturn iomap_fiemap(inode, fieinfo, start, len, \u0026erofs_iomap_ops);\n+\treturn iomap_fiemap(inode, fieinfo, start, len, erofs_iomap_next);\n }\n \n /*\n@@ -413,7 +411,7 @@ static int erofs_read_folio(struct file *file, struct folio *folio)\n \t};\n \n \ttrace_erofs_read_folio(iter_ctx.realinode, folio, true);\n-\tiomap_read_folio(\u0026erofs_iomap_ops, \u0026read_ctx, \u0026iter_ctx);\n+\tiomap_read_folio(erofs_iomap_next, \u0026read_ctx, \u0026iter_ctx);\n \tif (need_iput)\n \t\tiput(iter_ctx.realinode);\n \treturn 0;\n@@ -432,14 +430,14 @@ static void erofs_readahead(struct readahead_control *rac)\n \n \ttrace_erofs_readahead(iter_ctx.realinode, readahead_index(rac),\n \t\t\t readahead_count(rac), true);\n-\tiomap_readahead(\u0026erofs_iomap_ops, \u0026read_ctx, \u0026iter_ctx);\n+\tiomap_readahead(erofs_iomap_next, \u0026read_ctx, \u0026iter_ctx);\n \tif (need_iput)\n \t\tiput(iter_ctx.realinode);\n }\n \n static sector_t erofs_bmap(struct address_space *mapping, sector_t block)\n {\n-\treturn iomap_bmap(mapping, block, \u0026erofs_iomap_ops);\n+\treturn iomap_bmap(mapping, block, erofs_iomap_next);\n }\n \n static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n@@ -451,14 +449,14 @@ static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\treturn 0;\n \n \tif (IS_ENABLED(CONFIG_FS_DAX) \u0026\u0026 IS_DAX(inode))\n-\t\treturn dax_iomap_rw(iocb, to, \u0026erofs_iomap_ops);\n+\t\treturn dax_iomap_rw(iocb, to, erofs_iomap_next);\n \n \tif ((iocb-\u003eki_flags \u0026 IOCB_DIRECT) \u0026\u0026 inode-\u003ei_sb-\u003es_bdev) {\n \t\tstruct erofs_iomap_iter_ctx iter_ctx = {\n \t\t\t.realinode = inode,\n \t\t};\n \n-\t\treturn iomap_dio_rw(iocb, to, \u0026erofs_iomap_ops,\n+\t\treturn iomap_dio_rw(iocb, to, erofs_iomap_next,\n \t\t\t\t NULL, 0, \u0026iter_ctx, 0);\n \t}\n \treturn filemap_read(iocb, to, 0);\n@@ -478,7 +476,7 @@ const struct address_space_operations erofs_aops = {\n static vm_fault_t erofs_dax_huge_fault(struct vm_fault *vmf,\n \t\tunsigned int order)\n {\n-\treturn dax_iomap_fault(vmf, order, NULL, NULL, \u0026erofs_iomap_ops);\n+\treturn dax_iomap_fault(vmf, order, NULL, NULL, erofs_iomap_next);\n }\n \n static vm_fault_t erofs_dax_fault(struct vm_fault *vmf)\n@@ -510,18 +508,18 @@ static int erofs_file_mmap_prepare(struct vm_area_desc *desc)\n static loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence)\n {\n \tstruct inode *inode = file-\u003ef_mapping-\u003ehost;\n-\tconst struct iomap_ops *ops = \u0026erofs_iomap_ops;\n+\tiomap_iter_next_fn next = erofs_iomap_next;\n \n \tif (erofs_inode_is_data_compressed(EROFS_I(inode)-\u003edatalayout)) {\n \t\tif (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))\n \t\t\treturn generic_file_llseek(file, offset, whence);\n-\t\tops = \u0026z_erofs_iomap_report_ops;\n+\t\tnext = z_erofs_iomap_next_report;\n \t}\n \n \tif (whence == SEEK_HOLE)\n-\t\toffset = iomap_seek_hole(inode, offset, ops);\n+\t\toffset = iomap_seek_hole(inode, offset, next);\n \telse if (whence == SEEK_DATA)\n-\t\toffset = iomap_seek_data(inode, offset, ops);\n+\t\toffset = iomap_seek_data(inode, offset, next);\n \telse\n \t\treturn generic_file_llseek(file, offset, whence);\n \ndiff --git a/fs/erofs/internal.h b/fs/erofs/internal.h\nindex 580f8d9f14e7f..72ccd6f335b82 100644\n--- a/fs/erofs/internal.h\n+++ b/fs/erofs/internal.h\n@@ -397,7 +397,8 @@ extern const struct file_operations erofs_file_fops;\n extern const struct file_operations erofs_dir_fops;\n extern const struct file_operations erofs_ishare_fops;\n \n-extern const struct iomap_ops z_erofs_iomap_report_ops;\n+int z_erofs_iomap_next_report(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n \n void *erofs_read_metadata(struct super_block *sb, struct erofs_buf *buf,\n \t\t\t erofs_off_t *offset, int *lengthp);\ndiff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c\nindex bab5216135524..98968a9d942ee 100644\n--- a/fs/erofs/zmap.c\n+++ b/fs/erofs/zmap.c\n@@ -821,6 +821,4 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset,\n \treturn 0;\n }\n \n-const struct iomap_ops z_erofs_iomap_report_ops = {\n-\t.iomap_begin = z_erofs_iomap_begin_report,\n-};\n+DEFINE_IOMAP_ITER_NEXT(z_erofs_iomap_next_report, z_erofs_iomap_begin_report);\ndiff --git a/fs/exfat/file.c b/fs/exfat/file.c\nindex 5fc13378d35f7..c05849d305ae4 100644\n--- a/fs/exfat/file.c\n+++ b/fs/exfat/file.c\n@@ -668,7 +668,7 @@ static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size)\n \n \t\tret = iomap_zero_range(inode, old_valid_size,\n \t\t\t\tnew_valid_size - old_valid_size, NULL,\n-\t\t\t\t\u0026exfat_write_iomap_ops, NULL, NULL);\n+\t\t\t\texfat_write_iomap_next, NULL, NULL);\n \t\tif (ret) {\n \t\t\ttruncate_setsize(inode, old_valid_size);\n \t\t\texfat_truncate(inode);\n@@ -687,7 +687,7 @@ static ssize_t exfat_fallback_buffered_write(struct kiocb *iocb,\n \n \tiocb-\u003eki_flags \u0026= ~IOCB_DIRECT;\n \n-\twritten = iomap_file_buffered_write(iocb, from, \u0026exfat_write_iomap_ops,\n+\twritten = iomap_file_buffered_write(iocb, from, exfat_write_iomap_next,\n \t\t\tNULL, NULL);\n \tif (written \u003c 0)\n \t\treturn written;\n@@ -709,7 +709,7 @@ static ssize_t exfat_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)\n {\n \tssize_t ret;\n \n-\tret = iomap_dio_rw(iocb, from, \u0026exfat_write_iomap_ops,\n+\tret = iomap_dio_rw(iocb, from, exfat_write_iomap_next,\n \t\t\t\u0026exfat_write_dio_ops, 0, NULL, 0);\n \tif (ret == -ENOTBLK)\n \t\tret = 0;\n@@ -773,7 +773,7 @@ static ssize_t exfat_file_write_iter(struct kiocb *iocb, struct iov_iter *iter)\n \t\tret = exfat_dio_write_iter(iocb, iter);\n \telse\n \t\tret = iomap_file_buffered_write(iocb, iter,\n-\t\t\t\t\u0026exfat_write_iomap_ops, NULL, NULL);\n+\t\t\t\texfat_write_iomap_next, NULL, NULL);\n \tif (ret \u003c 0)\n \t\tgoto unlock;\n \n@@ -809,7 +809,7 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)\n \n \tif (iocb-\u003eki_flags \u0026 IOCB_DIRECT) {\n \t\tfile_accessed(iocb-\u003eki_filp);\n-\t\tret = iomap_dio_rw(iocb, iter, \u0026exfat_iomap_ops, NULL, 0,\n+\t\tret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0,\n \t\t\t\tNULL, 0);\n \t} else {\n \t\tret = generic_file_read_iter(iocb, iter);\n@@ -850,7 +850,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)\n \t\t\t */\n \t\t\terr = iomap_zero_range(inode, ei-\u003ezeroed_size,\n \t\t\t\t\tmmap_valid_size - ei-\u003ezeroed_size, NULL,\n-\t\t\t\t\t\u0026exfat_iomap_ops, NULL, NULL);\n+\t\t\t\t\texfat_iomap_next, NULL, NULL);\n \t\t\tif (err \u003c 0) {\n \t\t\t\tinode_unlock(inode);\n \t\t\t\treturn vmf_fs_error(err);\n@@ -866,7 +866,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)\n \tfile_update_time(vmf-\u003evma-\u003evm_file);\n \n \tfilemap_invalidate_lock_shared(inode-\u003ei_mapping);\n-\tret = iomap_page_mkwrite(vmf, \u0026exfat_write_iomap_ops, NULL);\n+\tret = iomap_page_mkwrite(vmf, exfat_write_iomap_next, NULL);\n \tfilemap_invalidate_unlock_shared(inode-\u003ei_mapping);\n \tsb_end_pagefault(inode-\u003ei_sb);\n \tinode_unlock(inode);\n@@ -939,12 +939,12 @@ static loff_t exfat_file_llseek(struct file *file, loff_t offset, int whence)\n \tswitch (whence) {\n \tcase SEEK_HOLE:\n \t\tinode_lock_shared(inode);\n-\t\toffset = iomap_seek_hole(inode, offset, \u0026exfat_iomap_ops);\n+\t\toffset = iomap_seek_hole(inode, offset, exfat_iomap_next);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \tcase SEEK_DATA:\n \t\tinode_lock_shared(inode);\n-\t\toffset = iomap_seek_data(inode, offset, \u0026exfat_iomap_ops);\n+\t\toffset = iomap_seek_data(inode, offset, exfat_iomap_next);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \tdefault:\ndiff --git a/fs/exfat/inode.c b/fs/exfat/inode.c\nindex 89826aea5e1e3..a6b9aa2ad7926 100644\n--- a/fs/exfat/inode.c\n+++ b/fs/exfat/inode.c\n@@ -248,7 +248,7 @@ static int exfat_read_folio(struct file *file, struct folio *folio)\n \t\t.ops = \u0026exfat_iomap_bio_read_ops,\n \t};\n \n-\tiomap_read_folio(\u0026exfat_iomap_ops, \u0026ctx, NULL);\n+\tiomap_read_folio(exfat_iomap_next, \u0026ctx, NULL);\n \treturn 0;\n }\n \n@@ -269,7 +269,7 @@ static void exfat_readahead(struct readahead_control *rac)\n \t ei-\u003evalid_size \u003c pos + readahead_length(rac))\n \t\treturn;\n \n-\tiomap_readahead(\u0026exfat_iomap_ops, \u0026ctx, NULL);\n+\tiomap_readahead(exfat_iomap_next, \u0026ctx, NULL);\n }\n \n static int exfat_writepages(struct address_space *mapping,\n@@ -293,7 +293,7 @@ static sector_t exfat_aop_bmap(struct address_space *mapping, sector_t block)\n \n \t/* exfat_get_cluster() assumes the requested blocknr isn't truncated. */\n \tdown_read(\u0026EXFAT_I(mapping-\u003ehost)-\u003etruncate_lock);\n-\tblocknr = iomap_bmap(mapping, block, \u0026exfat_iomap_ops);\n+\tblocknr = iomap_bmap(mapping, block, exfat_iomap_next);\n \tup_read(\u0026EXFAT_I(mapping-\u003ehost)-\u003etruncate_lock);\n \treturn blocknr;\n }\ndiff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c\nindex 1aac38e63fe6f..631de19559d5c 100644\n--- a/fs/exfat/iomap.c\n+++ b/fs/exfat/iomap.c\n@@ -151,9 +151,7 @@ static int exfat_write_iomap_begin(struct inode *inode, loff_t offset, loff_t le\n \treturn __exfat_iomap_begin(inode, offset, length, flags, iomap, true);\n }\n \n-const struct iomap_ops exfat_iomap_ops = {\n-\t.iomap_begin = exfat_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(exfat_iomap_next, exfat_iomap_begin);\n \n /*\n * exfat_write_iomap_end - Update the state after write\n@@ -186,10 +184,8 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,\n \treturn written;\n }\n \n-const struct iomap_ops exfat_write_iomap_ops = {\n-\t.iomap_begin\t= exfat_write_iomap_begin,\n-\t.iomap_end\t= exfat_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(exfat_write_iomap_next, exfat_write_iomap_begin,\n+\t\texfat_write_iomap_end);\n \n /*\n * exfat_writeback_range - Map folio during writeback\n@@ -267,5 +263,5 @@ const struct iomap_read_ops exfat_iomap_bio_read_ops = {\n int exfat_iomap_swap_activate(struct swap_info_struct *sis,\n \t\t\t struct file *file, sector_t *span)\n {\n-\treturn iomap_swapfile_activate(sis, file, span, \u0026exfat_iomap_ops);\n+\treturn iomap_swapfile_activate(sis, file, span, exfat_iomap_next);\n }\ndiff --git a/fs/exfat/iomap.h b/fs/exfat/iomap.h\nindex fd8a913f77946..47d7b753735e8 100644\n--- a/fs/exfat/iomap.h\n+++ b/fs/exfat/iomap.h\n@@ -7,8 +7,10 @@\n #define _LINUX_EXFAT_IOMAP_H\n \n extern const struct iomap_dio_ops exfat_write_dio_ops;\n-extern const struct iomap_ops exfat_iomap_ops;\n-extern const struct iomap_ops exfat_write_iomap_ops;\n+int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int exfat_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n extern const struct iomap_writeback_ops exfat_writeback_ops;\n extern const struct iomap_read_ops exfat_iomap_bio_read_ops;\n \ndiff --git a/fs/ext2/ext2.h b/fs/ext2/ext2.h\nindex 79f7b395258c2..59ef8b898940f 100644\n--- a/fs/ext2/ext2.h\n+++ b/fs/ext2/ext2.h\n@@ -780,7 +780,8 @@ extern const struct file_operations ext2_file_operations;\n /* inode.c */\n extern void ext2_set_file_ops(struct inode *inode);\n extern const struct address_space_operations ext2_aops;\n-extern const struct iomap_ops ext2_iomap_ops;\n+int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n \n /* namei.c */\n extern const struct inode_operations ext2_dir_inode_operations;\ndiff --git a/fs/ext2/file.c b/fs/ext2/file.c\nindex 8dca9ec4cacd9..1fc00ad775174 100644\n--- a/fs/ext2/file.c\n+++ b/fs/ext2/file.c\n@@ -70,7 +70,7 @@ static ssize_t ext2_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \n \ttrace_ext2_dio_read_begin(iocb, to, 0);\n \tinode_lock_shared(inode);\n-\tret = iomap_dio_rw(iocb, to, \u0026ext2_iomap_ops, NULL, 0, NULL, 0);\n+\tret = iomap_dio_rw(iocb, to, ext2_iomap_next, NULL, 0, NULL, 0);\n \tinode_unlock_shared(inode);\n \ttrace_ext2_dio_read_end(iocb, to, ret);\n \n@@ -134,7 +134,7 @@ static ssize_t ext2_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t (!IS_ALIGNED(iocb-\u003eki_pos | iov_iter_alignment(from), blocksize)))\n \t\tflags |= IOMAP_DIO_FORCE_WAIT;\n \n-\tret = iomap_dio_rw(iocb, from, \u0026ext2_iomap_ops, \u0026ext2_dio_write_ops,\n+\tret = iomap_dio_rw(iocb, from, ext2_iomap_next, \u0026ext2_dio_write_ops,\n \t\t\t flags, NULL, 0);\n \n \t/* ENOTBLK is magic return value for fallback to buffered-io */\ndiff --git a/fs/ext2/inode.c b/fs/ext2/inode.c\nindex 29808629cce56..873ad1d90e0dd 100644\n--- a/fs/ext2/inode.c\n+++ b/fs/ext2/inode.c\n@@ -860,10 +860,7 @@ ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-const struct iomap_ops ext2_iomap_ops = {\n-\t.iomap_begin\t\t= ext2_iomap_begin,\n-\t.iomap_end\t\t= ext2_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(ext2_iomap_next, ext2_iomap_begin, ext2_iomap_end);\n \n int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \t\tu64 start, u64 len)\n@@ -882,7 +879,7 @@ int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \tif (i_size == 0)\n \t\ti_size = 1;\n \tlen = min_t(u64, len, i_size);\n-\tret = iomap_fiemap(inode, fieinfo, start, len, \u0026ext2_iomap_ops);\n+\tret = iomap_fiemap(inode, fieinfo, start, len, ext2_iomap_next);\n \tinode_unlock(inode);\n \n \treturn ret;\ndiff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h\nindex b37c136ea3ab3..f386a86e945cd 100644\n--- a/fs/ext4/ext4.h\n+++ b/fs/ext4/ext4.h\n@@ -4004,8 +4004,13 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)\n \t\tio_end-\u003eflag \u0026= ~EXT4_IO_END_UNWRITTEN;\n }\n \n-extern const struct iomap_ops ext4_iomap_ops;\n-extern const struct iomap_ops ext4_iomap_report_ops;\n+int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+\n+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n+\t\tunsigned flags, struct iomap *iomap, struct iomap *srcmap);\n \n static inline int ext4_buffer_uptodate(struct buffer_head *bh)\n {\ndiff --git a/fs/ext4/extents.c b/fs/ext4/extents.c\nindex 91c97af64b317..71bbf8e77261f 100644\n--- a/fs/ext4/extents.c\n+++ b/fs/ext4/extents.c\n@@ -5171,9 +5171,7 @@ static int ext4_iomap_xattr_begin(struct inode *inode, loff_t offset,\n \treturn error;\n }\n \n-static const struct iomap_ops ext4_iomap_xattr_ops = {\n-\t.iomap_begin\t\t= ext4_iomap_xattr_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_xattr_next, ext4_iomap_xattr_begin);\n \n static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)\n {\n@@ -5217,10 +5215,10 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \tif (fieinfo-\u003efi_flags \u0026 FIEMAP_FLAG_XATTR) {\n \t\tfieinfo-\u003efi_flags \u0026= ~FIEMAP_FLAG_XATTR;\n \t\terror = iomap_fiemap(inode, fieinfo, start, len,\n-\t\t\t\t \u0026ext4_iomap_xattr_ops);\n+\t\t\t\t ext4_iomap_xattr_next);\n \t} else {\n \t\terror = iomap_fiemap(inode, fieinfo, start, len,\n-\t\t\t\t \u0026ext4_iomap_report_ops);\n+\t\t\t\t ext4_iomap_next_report);\n \t}\n unlock:\n \tinode_unlock_shared(inode);\ndiff --git a/fs/ext4/file.c b/fs/ext4/file.c\nindex eb1a323962b10..da6fbf76ca51b 100644\n--- a/fs/ext4/file.c\n+++ b/fs/ext4/file.c\n@@ -91,7 +91,9 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\treturn generic_file_read_iter(iocb, to);\n \t}\n \n-\tret = iomap_dio_rw(iocb, to, \u0026ext4_iomap_ops, NULL, 0, NULL, 0);\n+\tret = iomap_dio_read_simple(iocb, to, ext4_iomap_begin);\n+\tif (ret == -ENOTBLK)\n+\t\tret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);\n \tinode_unlock_shared(inode);\n \n \tfile_accessed(iocb-\u003eki_filp);\n@@ -119,7 +121,7 @@ static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\t/* Fallback to buffered IO in case we cannot support DAX */\n \t\treturn generic_file_read_iter(iocb, to);\n \t}\n-\tret = dax_iomap_rw(iocb, to, \u0026ext4_iomap_ops);\n+\tret = dax_iomap_rw(iocb, to, ext4_iomap_next);\n \tinode_unlock_shared(inode);\n \n \tfile_accessed(iocb-\u003eki_filp);\n@@ -589,7 +591,7 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\t\tgoto out;\n \t}\n \n-\tret = iomap_dio_rw(iocb, from, \u0026ext4_iomap_ops, \u0026ext4_dio_write_ops,\n+\tret = iomap_dio_rw(iocb, from, ext4_iomap_next, \u0026ext4_dio_write_ops,\n \t\t\t dio_flags, NULL, 0);\n \tif (ret == -ENOTBLK)\n \t\tret = 0;\n@@ -688,7 +690,7 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\text4_journal_stop(handle);\n \t}\n \n-\tret = dax_iomap_rw(iocb, from, \u0026ext4_iomap_ops);\n+\tret = dax_iomap_rw(iocb, from, ext4_iomap_next);\n \n \tif (extend) {\n \t\tret = ext4_handle_inode_extension(inode, offset, ret, count);\n@@ -776,7 +778,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, unsigned int order)\n \t} else {\n \t\tfilemap_invalidate_lock_shared(mapping);\n \t}\n-\tresult = dax_iomap_fault(vmf, order, \u0026pfn, \u0026error, \u0026ext4_iomap_ops);\n+\tresult = dax_iomap_fault(vmf, order, \u0026pfn, \u0026error, ext4_iomap_next);\n \tif (write) {\n \t\text4_journal_stop(handle);\n \n@@ -955,13 +957,13 @@ loff_t ext4_llseek(struct file *file, loff_t offset, int whence)\n \tcase SEEK_HOLE:\n \t\tinode_lock_shared(inode);\n \t\toffset = iomap_seek_hole(inode, offset,\n-\t\t\t\t\t \u0026ext4_iomap_report_ops);\n+\t\t\t\t\t ext4_iomap_next_report);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \tcase SEEK_DATA:\n \t\tinode_lock_shared(inode);\n \t\toffset = iomap_seek_data(inode, offset,\n-\t\t\t\t\t \u0026ext4_iomap_report_ops);\n+\t\t\t\t\t ext4_iomap_next_report);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \t}\ndiff --git a/fs/ext4/inode.c b/fs/ext4/inode.c\nindex ce99807c5f5b2..6e5a6135381fb 100644\n--- a/fs/ext4/inode.c\n+++ b/fs/ext4/inode.c\n@@ -3391,7 +3391,7 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)\n \t\tfilemap_write_and_wait(mapping);\n \t}\n \n-\tret = iomap_bmap(mapping, block, \u0026ext4_iomap_ops);\n+\tret = iomap_bmap(mapping, block, ext4_iomap_next);\n \n out:\n \tinode_unlock_shared(inode);\n@@ -3771,7 +3771,7 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,\n }\n \n \n-static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \t\tunsigned flags, struct iomap *iomap, struct iomap *srcmap)\n {\n \tint ret;\n@@ -3850,9 +3850,7 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-const struct iomap_ops ext4_iomap_ops = {\n-\t.iomap_begin\t\t= ext4_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);\n \n static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,\n \t\t\t\t loff_t length, unsigned int flags,\n@@ -3905,9 +3903,7 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,\n \treturn 0;\n }\n \n-const struct iomap_ops ext4_iomap_report_ops = {\n-\t.iomap_begin = ext4_iomap_begin_report,\n-};\n+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);\n \n /*\n * For data=journal mode, folio should be marked dirty only when it was\n@@ -3944,7 +3940,7 @@ static int ext4_iomap_swap_activate(struct swap_info_struct *sis,\n \t\t\t\t struct file *file, sector_t *span)\n {\n \treturn iomap_swapfile_activate(sis, file, span,\n-\t\t\t\t \u0026ext4_iomap_report_ops);\n+\t\t\t\t ext4_iomap_next_report);\n }\n \n static const struct address_space_operations ext4_aops = {\n@@ -4191,7 +4187,7 @@ static int ext4_block_zero_range(struct inode *inode,\n \n \tif (IS_DAX(inode)) {\n \t\treturn dax_zero_range(inode, from, length, did_zero,\n-\t\t\t\t \u0026ext4_iomap_ops);\n+\t\t\t\t ext4_iomap_next);\n \t} else if (ext4_should_journal_data(inode)) {\n \t\treturn ext4_block_journalled_zero_range(inode, from, length,\n \t\t\t\t\t\t\tdid_zero);\ndiff --git a/fs/f2fs/data.c b/fs/f2fs/data.c\nindex a765fda715363..1cb993244b1c4 100644\n--- a/fs/f2fs/data.c\n+++ b/fs/f2fs/data.c\n@@ -4653,6 +4653,4 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-const struct iomap_ops f2fs_iomap_ops = {\n-\t.iomap_begin\t= f2fs_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(f2fs_iomap_next, f2fs_iomap_begin);\ndiff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h\nindex 8f3e632f315cc..946a91834aec2 100644\n--- a/fs/f2fs/f2fs.h\n+++ b/fs/f2fs/f2fs.h\n@@ -4216,7 +4216,8 @@ int f2fs_init_post_read_processing(void);\n void f2fs_destroy_post_read_processing(void);\n int f2fs_init_wq(struct f2fs_sb_info *sbi);\n void f2fs_destroy_wq(struct f2fs_sb_info *sbi);\n-extern const struct iomap_ops f2fs_iomap_ops;\n+int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\t struct iomap *srcmap);\n \n /*\n * gc.c\ndiff --git a/fs/f2fs/file.c b/fs/f2fs/file.c\nindex 4b52c56d71f07..74514b1172571 100644\n--- a/fs/f2fs/file.c\n+++ b/fs/f2fs/file.c\n@@ -4884,7 +4884,7 @@ static ssize_t f2fs_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t * F2FS_DIO_READ counter will be decremented correctly in all cases.\n \t */\n \tinc_page_count(sbi, F2FS_DIO_READ);\n-\tdio = __iomap_dio_rw(iocb, to, \u0026f2fs_iomap_ops,\n+\tdio = __iomap_dio_rw(iocb, to, f2fs_iomap_next,\n \t\t\t \u0026f2fs_iomap_dio_read_ops, 0, NULL, 0);\n \tif (IS_ERR_OR_NULL(dio)) {\n \t\tret = PTR_ERR_OR_ZERO(dio);\n@@ -5220,7 +5220,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,\n \tdio_flags = 0;\n \tif (pos + count \u003e inode-\u003ei_size)\n \t\tdio_flags |= IOMAP_DIO_FORCE_WAIT;\n-\tdio = __iomap_dio_rw(iocb, from, \u0026f2fs_iomap_ops,\n+\tdio = __iomap_dio_rw(iocb, from, f2fs_iomap_next,\n \t\t\t \u0026f2fs_iomap_dio_write_ops, dio_flags, iocb, 0);\n \tif (IS_ERR_OR_NULL(dio)) {\n \t\tret = PTR_ERR_OR_ZERO(dio);\ndiff --git a/fs/fuse/dax.c b/fs/fuse/dax.c\nindex 8b53625ac7abd..480cfd2d1fa5e 100644\n--- a/fs/fuse/dax.c\n+++ b/fs/fuse/dax.c\n@@ -653,10 +653,8 @@ static int fuse_iomap_end(struct inode *inode, loff_t pos, loff_t length,\n \treturn 0;\n }\n \n-static const struct iomap_ops fuse_iomap_ops = {\n-\t.iomap_begin = fuse_iomap_begin,\n-\t.iomap_end = fuse_iomap_end,\n-};\n+static DEFINE_IOMAP_ITER_NEXT_END(fuse_iomap_next, fuse_iomap_begin,\n+\t\t\t\t fuse_iomap_end);\n \n static void fuse_wait_dax_page(struct inode *inode)\n {\n@@ -685,7 +683,7 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\tinode_lock_shared(inode);\n \t}\n \n-\tret = dax_iomap_rw(iocb, to, \u0026fuse_iomap_ops);\n+\tret = dax_iomap_rw(iocb, to, fuse_iomap_next);\n \tinode_unlock_shared(inode);\n \n \t/* TODO file_accessed(iocb-\u003ef_filp) */\n@@ -740,7 +738,7 @@ ssize_t fuse_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \tif (file_extending_write(iocb, from))\n \t\tret = fuse_dax_direct_write(iocb, from);\n \telse\n-\t\tret = dax_iomap_rw(iocb, from, \u0026fuse_iomap_ops);\n+\t\tret = dax_iomap_rw(iocb, from, fuse_iomap_next);\n \n out:\n \tinode_unlock(inode);\n@@ -775,7 +773,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, unsigned int order,\n \t * to populate page cache or access memory we are trying to free.\n \t */\n \tfilemap_invalidate_lock_shared(inode-\u003ei_mapping);\n-\tret = dax_iomap_fault(vmf, order, \u0026pfn, \u0026error, \u0026fuse_iomap_ops);\n+\tret = dax_iomap_fault(vmf, order, \u0026pfn, \u0026error, fuse_iomap_next);\n \tif ((ret \u0026 VM_FAULT_ERROR) \u0026\u0026 error == -EAGAIN) {\n \t\terror = 0;\n \t\tretry = true;\ndiff --git a/fs/fuse/file.c b/fs/fuse/file.c\nindex ea4a15a7635a6..7a95ac36c4c5b 100644\n--- a/fs/fuse/file.c\n+++ b/fs/fuse/file.c\n@@ -890,9 +890,7 @@ static int fuse_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-static const struct iomap_ops fuse_iomap_ops = {\n-\t.iomap_begin\t= fuse_iomap_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(fuse_iomap_next, fuse_iomap_begin);\n \n struct fuse_fill_read_data {\n \tstruct file *file;\n@@ -1014,7 +1012,7 @@ static int fuse_read_folio(struct file *file, struct folio *folio)\n \t\treturn -EIO;\n \t}\n \n-\tiomap_read_folio(\u0026fuse_iomap_ops, \u0026ctx, NULL);\n+\tiomap_read_folio(fuse_iomap_next, \u0026ctx, NULL);\n \tfuse_invalidate_atime(inode);\n \treturn 0;\n }\n@@ -1115,7 +1113,7 @@ static void fuse_readahead(struct readahead_control *rac)\n \tif (fuse_is_bad(inode))\n \t\treturn;\n \n-\tiomap_readahead(\u0026fuse_iomap_ops, \u0026ctx, NULL);\n+\tiomap_readahead(fuse_iomap_next, \u0026ctx, NULL);\n }\n \n static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to)\n@@ -1531,7 +1529,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\t * and granular dirty tracking for large folios.\n \t\t */\n \t\twritten = iomap_file_buffered_write(iocb, from,\n-\t\t\t\t\t\t \u0026fuse_iomap_ops,\n+\t\t\t\t\t\t fuse_iomap_next,\n \t\t\t\t\t\t \u0026fuse_iomap_write_ops,\n \t\t\t\t\t\t file);\n \t} else {\ndiff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c\nindex df25d4faca41c..f15e516ebcb5c 100644\n--- a/fs/fuse/virtio_fs.c\n+++ b/fs/fuse/virtio_fs.c\n@@ -1024,8 +1024,7 @@ static void virtio_fs_cleanup_vqs(struct virtio_device *vdev)\n }\n \n /* Map a window offset to a page frame number. The window offset will have\n- * been produced by .iomap_begin(), which maps a file offset to a window\n- * offset.\n+ * been produced by .iomap_next(), which maps a file offset to a window offset.\n */\n static long virtio_fs_direct_access(struct dax_device *dax_dev, pgoff_t pgoff,\n \t\t\t\t long nr_pages, enum dax_access_mode mode,\ndiff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c\nindex 0a7b8076af3a5..66bc19c011cc0 100644\n--- a/fs/gfs2/aops.c\n+++ b/fs/gfs2/aops.c\n@@ -425,7 +425,7 @@ static int gfs2_read_folio(struct file *file, struct folio *folio)\n \n \tif (!gfs2_is_jdata(ip) ||\n \t (i_blocksize(inode) == PAGE_SIZE \u0026\u0026 !folio_buffers(folio))) {\n-\t\tiomap_bio_read_folio(folio, \u0026gfs2_iomap_ops);\n+\t\tiomap_bio_read_folio(folio, gfs2_iomap_next);\n \t} else if (gfs2_is_stuffed(ip)) {\n \t\terror = stuffed_read_folio(ip, folio);\n \t} else {\n@@ -500,7 +500,7 @@ static void gfs2_readahead(struct readahead_control *rac)\n \telse if (gfs2_is_jdata(ip))\n \t\tmpage_readahead(rac, gfs2_block_map);\n \telse\n-\t\tiomap_bio_readahead(rac, \u0026gfs2_iomap_ops);\n+\t\tiomap_bio_readahead(rac, gfs2_iomap_next);\n }\n \n /**\n@@ -571,7 +571,7 @@ static sector_t gfs2_bmap(struct address_space *mapping, sector_t lblock)\n \t\treturn 0;\n \n \tif (!gfs2_is_stuffed(ip))\n-\t\tdblock = iomap_bmap(mapping, lblock, \u0026gfs2_iomap_ops);\n+\t\tdblock = iomap_bmap(mapping, lblock, gfs2_iomap_next);\n \n \tgfs2_glock_dq_uninit(\u0026i_gh);\n \ndiff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c\nindex 51ac1fd44f78a..f835bd343b6e6 100644\n--- a/fs/gfs2/bmap.c\n+++ b/fs/gfs2/bmap.c\n@@ -1200,10 +1200,7 @@ static int gfs2_iomap_end(struct inode *inode, loff_t pos, loff_t length,\n \treturn 0;\n }\n \n-const struct iomap_ops gfs2_iomap_ops = {\n-\t.iomap_begin = gfs2_iomap_begin,\n-\t.iomap_end = gfs2_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(gfs2_iomap_next, gfs2_iomap_begin, gfs2_iomap_end);\n \n /**\n * gfs2_block_map - Map one or more blocks of an inode to a disk block\n@@ -1318,7 +1315,7 @@ static int gfs2_block_zero_range(struct inode *inode, loff_t from, loff_t length\n \tif (from \u003e= inode-\u003ei_size)\n \t\treturn 0;\n \tlength = min(length, inode-\u003ei_size - from);\n-\treturn iomap_zero_range(inode, from, length, NULL, \u0026gfs2_iomap_ops,\n+\treturn iomap_zero_range(inode, from, length, NULL, gfs2_iomap_next,\n \t\t\t\u0026gfs2_iomap_write_ops, NULL);\n }\n \ndiff --git a/fs/gfs2/bmap.h b/fs/gfs2/bmap.h\nindex e3d6efdfd8903..2c2b7ab39259c 100644\n--- a/fs/gfs2/bmap.h\n+++ b/fs/gfs2/bmap.h\n@@ -43,7 +43,8 @@ static inline void gfs2_write_calc_reserv(const struct gfs2_inode *ip,\n \t}\n }\n \n-extern const struct iomap_ops gfs2_iomap_ops;\n+int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n extern const struct iomap_write_ops gfs2_iomap_write_ops;\n extern const struct iomap_writeback_ops gfs2_writeback_ops;\n \ndiff --git a/fs/gfs2/file.c b/fs/gfs2/file.c\nindex b8c10de113ba7..ef5f521a46c05 100644\n--- a/fs/gfs2/file.c\n+++ b/fs/gfs2/file.c\n@@ -844,7 +844,7 @@ static ssize_t gfs2_file_direct_read(struct kiocb *iocb, struct iov_iter *to,\n \t\tgoto out_uninit;\n \tpagefault_disable();\n \tto-\u003enofault = true;\n-\tret = iomap_dio_rw(iocb, to, \u0026gfs2_iomap_ops, NULL,\n+\tret = iomap_dio_rw(iocb, to, gfs2_iomap_next, NULL,\n \t\t\t IOMAP_DIO_PARTIAL, NULL, read);\n \tto-\u003enofault = false;\n \tpagefault_enable();\n@@ -910,7 +910,7 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from,\n \t\tgoto out_unlock;\n \n \tfrom-\u003enofault = true;\n-\tret = iomap_dio_rw(iocb, from, \u0026gfs2_iomap_ops, NULL,\n+\tret = iomap_dio_rw(iocb, from, gfs2_iomap_next, NULL,\n \t\t\t IOMAP_DIO_PARTIAL, NULL, written);\n \tfrom-\u003enofault = false;\n \tif (ret \u003c= 0) {\n@@ -1062,7 +1062,7 @@ static ssize_t gfs2_file_buffered_write(struct kiocb *iocb,\n \t\tgoto out_unlock;\n \n \tpagefault_disable();\n-\tret = iomap_file_buffered_write(iocb, from, \u0026gfs2_iomap_ops,\n+\tret = iomap_file_buffered_write(iocb, from, gfs2_iomap_next,\n \t\t\t\u0026gfs2_iomap_write_ops, NULL);\n \tpagefault_enable();\n \tif (ret \u003e 0)\ndiff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c\nindex 8a77794bbd4af..737a3b6c52683 100644\n--- a/fs/gfs2/inode.c\n+++ b/fs/gfs2/inode.c\n@@ -2217,7 +2217,7 @@ static int gfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \t\tgoto out;\n \n \tpagefault_disable();\n-\tret = iomap_fiemap(inode, fieinfo, start, len, \u0026gfs2_iomap_ops);\n+\tret = iomap_fiemap(inode, fieinfo, start, len, gfs2_iomap_next);\n \tpagefault_enable();\n \n \tgfs2_glock_dq_uninit(\u0026gh);\n@@ -2242,7 +2242,7 @@ loff_t gfs2_seek_data(struct file *file, loff_t offset)\n \tinode_lock_shared(inode);\n \tret = gfs2_glock_nq_init(ip-\u003ei_gl, LM_ST_SHARED, 0, \u0026gh);\n \tif (!ret)\n-\t\tret = iomap_seek_data(inode, offset, \u0026gfs2_iomap_ops);\n+\t\tret = iomap_seek_data(inode, offset, gfs2_iomap_next);\n \tgfs2_glock_dq_uninit(\u0026gh);\n \tinode_unlock_shared(inode);\n \n@@ -2261,7 +2261,7 @@ loff_t gfs2_seek_hole(struct file *file, loff_t offset)\n \tinode_lock_shared(inode);\n \tret = gfs2_glock_nq_init(ip-\u003ei_gl, LM_ST_SHARED, 0, \u0026gh);\n \tif (!ret)\n-\t\tret = iomap_seek_hole(inode, offset, \u0026gfs2_iomap_ops);\n+\t\tret = iomap_seek_hole(inode, offset, gfs2_iomap_next);\n \tgfs2_glock_dq_uninit(\u0026gh);\n \tinode_unlock_shared(inode);\n \ndiff --git a/fs/hpfs/file.c b/fs/hpfs/file.c\nindex 29e876705369b..ef7eb601be09b 100644\n--- a/fs/hpfs/file.c\n+++ b/fs/hpfs/file.c\n@@ -156,9 +156,7 @@ static int hpfs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,\n \treturn 0;\n }\n \n-static const struct iomap_ops hpfs_iomap_ops = {\n-\t.iomap_begin\t\t= hpfs_iomap_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(hpfs_iomap_next, hpfs_iomap_begin);\n \n static int hpfs_read_folio(struct file *file, struct folio *folio)\n {\n@@ -236,7 +234,7 @@ static int hpfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \n \tinode_lock(inode);\n \tlen = min_t(u64, len, i_size_read(inode));\n-\tret = iomap_fiemap(inode, fieinfo, start, len, \u0026hpfs_iomap_ops);\n+\tret = iomap_fiemap(inode, fieinfo, start, len, hpfs_iomap_next);\n \tinode_unlock(inode);\n \n \treturn ret;\ndiff --git a/fs/internal.h b/fs/internal.h\nindex 355d93f922086..19601f8406dcb 100644\n--- a/fs/internal.h\n+++ b/fs/internal.h\n@@ -8,7 +8,6 @@\n struct super_block;\n struct file_system_type;\n struct iomap;\n-struct iomap_ops;\n struct linux_binprm;\n struct path;\n struct mount;\ndiff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c\nindex f6040199d1140..6f160612d0fd3 100644\n--- a/fs/iomap/buffered-io.c\n+++ b/fs/iomap/buffered-io.c\n@@ -624,8 +624,8 @@ static int iomap_read_folio_iter(struct iomap_iter *iter,\n \treturn 0;\n }\n \n-void iomap_read_folio(const struct iomap_ops *ops,\n-\t\tstruct iomap_read_folio_ctx *ctx, void *private)\n+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,\n+\t\tvoid *private)\n {\n \tstruct folio *folio = ctx-\u003ecur_folio;\n \tstruct iomap_iter iter = {\n@@ -648,7 +648,7 @@ void iomap_read_folio(const struct iomap_ops *ops,\n \t\tfsverity_readahead(ctx-\u003evi, folio-\u003eindex,\n \t\t\t\t folio_nr_pages(folio));\n \n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_read_folio_iter(\u0026iter, ctx,\n \t\t\t\t\u0026bytes_submitted);\n \n@@ -687,22 +687,22 @@ static int iomap_readahead_iter(struct iomap_iter *iter,\n \n /**\n * iomap_readahead - Attempt to read pages from a file.\n- * @ops: The operations vector for the filesystem.\n+ * @next: The iomap iteration function for the filesystem.\n * @ctx: The ctx used for issuing readahead.\n * @private: The filesystem-specific information for issuing iomap_iter.\n *\n * This function is for filesystems to call to implement their readahead\n * address_space operation.\n *\n- * Context: The @ops callbacks may submit I/O (eg to read the addresses of\n+ * Context: The @next callback may submit I/O (eg to read the addresses of\n * blocks from disc), and may wait for it. The caller may be trying to\n * access a different page, and so sleeping excessively should be avoided.\n * It may allocate memory, but should avoid costly allocations. This\n * function is called with memalloc_nofs set, so allocations will not cause\n * the filesystem to be reentered.\n */\n-void iomap_readahead(const struct iomap_ops *ops,\n-\t\tstruct iomap_read_folio_ctx *ctx, void *private)\n+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,\n+\t\tvoid *private)\n {\n \tstruct readahead_control *rac = ctx-\u003erac;\n \tstruct iomap_iter iter = {\n@@ -724,7 +724,7 @@ void iomap_readahead(const struct iomap_ops *ops,\n \t\tfsverity_readahead(ctx-\u003evi, readahead_index(rac),\n \t\t\t\treadahead_count(rac));\n \n-\twhile (iomap_iter(\u0026iter, ops) \u003e 0)\n+\twhile (iomap_iter(\u0026iter, next) \u003e 0)\n \t\titer.status = iomap_readahead_iter(\u0026iter, ctx,\n \t\t\t\t\t\u0026cur_bytes_submitted);\n \n@@ -1268,7 +1268,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,\n \n ssize_t\n iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private)\n {\n \tstruct iomap_iter iter = {\n@@ -1285,7 +1285,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,\n \tif (iocb-\u003eki_flags \u0026 IOCB_DONTCACHE)\n \t\titer.flags |= IOMAP_DONTCACHE;\n \n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_write_iter(\u0026iter, i, write_ops);\n \n \tif (unlikely(iter.pos == iocb-\u003eki_pos))\n@@ -1297,7 +1297,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,\n EXPORT_SYMBOL_GPL(iomap_file_buffered_write);\n \n int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,\n-\t\tconst void *buf, const struct iomap_ops *ops,\n+\t\tconst void *buf, iomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops)\n {\n \tint\t\t\tret;\n@@ -1314,7 +1314,7 @@ int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,\n \n \tiov_iter_kvec(\u0026iiter, WRITE, \u0026kvec, 1, length);\n \n-\tret = iomap_file_buffered_write(\u0026iocb, \u0026iiter, ops, write_ops, NULL);\n+\tret = iomap_file_buffered_write(\u0026iocb, \u0026iiter, next, write_ops, NULL);\n \tif (ret \u003c 0)\n \t\treturn ret;\n \treturn ret == length ? 0 : -EIO;\n@@ -1586,7 +1586,7 @@ static int iomap_unshare_iter(struct iomap_iter *iter,\n \n int\n iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops)\n {\n \tstruct iomap_iter iter = {\n@@ -1601,7 +1601,7 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n \t\treturn 0;\n \n \titer.len = min(len, size - pos);\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_unshare_iter(\u0026iter, write_ops);\n \treturn ret;\n }\n@@ -1710,7 +1710,7 @@ EXPORT_SYMBOL_GPL(iomap_fill_dirty_folios);\n \n int\n iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private)\n {\n \tstruct folio_batch fbatch;\n@@ -1735,7 +1735,7 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,\n \t */\n \trange_dirty = filemap_range_needs_writeback(mapping, iter.pos,\n \t\t\t\t\titer.pos + iter.len - 1);\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0) {\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0) {\n \t\tconst struct iomap *srcmap = iomap_iter_srcmap(\u0026iter);\n \n \t\tif (!(iter.iomap.flags \u0026 IOMAP_F_FOLIO_BATCH) \u0026\u0026\n@@ -1761,7 +1761,7 @@ EXPORT_SYMBOL_GPL(iomap_zero_range);\n \n int\n iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private)\n {\n \tunsigned int blocksize = i_blocksize(inode);\n@@ -1770,7 +1770,7 @@ iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n \t/* Block boundary? Nothing to do */\n \tif (!off)\n \t\treturn 0;\n-\treturn iomap_zero_range(inode, pos, blocksize - off, did_zero, ops,\n+\treturn iomap_zero_range(inode, pos, blocksize - off, did_zero, next,\n \t\t\twrite_ops, private);\n }\n EXPORT_SYMBOL_GPL(iomap_truncate_page);\n@@ -1795,7 +1795,7 @@ static int iomap_folio_mkwrite_iter(struct iomap_iter *iter,\n \treturn iomap_iter_advance(iter, length);\n }\n \n-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,\n+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,\n \t\tvoid *private)\n {\n \tstruct iomap_iter iter = {\n@@ -1812,7 +1812,7 @@ vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,\n \t\tgoto out_unlock;\n \titer.pos = folio_pos(folio);\n \titer.len = ret;\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_folio_mkwrite_iter(\u0026iter, folio);\n \n \tif (ret \u003c 0)\ndiff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c\nindex ca790239e5eb3..36dca8fc8fa5a 100644\n--- a/fs/iomap/direct-io.c\n+++ b/fs/iomap/direct-io.c\n@@ -682,7 +682,7 @@ static int iomap_dio_iter(struct iomap_iter *iter, struct iomap_dio *dio)\n */\n struct iomap_dio *\n __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops, const struct iomap_dio_ops *dops,\n+\t\tiomap_iter_next_fn next, const struct iomap_dio_ops *dops,\n \t\tunsigned int dio_flags, void *private, size_t done_before)\n {\n \tstruct inode *inode = file_inode(iocb-\u003eki_filp);\n@@ -806,7 +806,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n \tinode_dio_begin(inode);\n \n \tblk_start_plug(\u0026plug);\n-\twhile ((ret = iomap_iter(\u0026iomi, ops)) \u003e 0) {\n+\twhile ((ret = iomap_iter(\u0026iomi, next)) \u003e 0) {\n \t\tiomi.status = iomap_dio_iter(\u0026iomi, dio);\n \n \t\t/*\n@@ -894,6 +894,21 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n }\n EXPORT_SYMBOL_GPL(__iomap_dio_rw);\n \n+ssize_t\n+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n+\t\tiomap_iter_next_fn next, const struct iomap_dio_ops *dops,\n+\t\tunsigned int dio_flags, void *private, size_t done_before)\n+{\n+\tstruct iomap_dio *dio;\n+\n+\tdio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,\n+\t\t\t done_before);\n+\tif (IS_ERR_OR_NULL(dio))\n+\t\treturn PTR_ERR_OR_ZERO(dio);\n+\treturn iomap_dio_complete(dio);\n+}\n+EXPORT_SYMBOL_GPL(iomap_dio_rw);\n+\n struct iomap_dio_simple {\n \tstruct kiocb\t\t*iocb;\n \tsize_t\t\t\tsize;\n@@ -968,211 +983,93 @@ static void iomap_dio_simple_end_io(struct bio *bio)\n \tiocb-\u003eki_complete(iocb, iomap_dio_simple_complete(sr));\n }\n \n-static inline bool\n-iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,\n-\t\t\t const struct iomap_dio_ops *dops,\n-\t\t\t unsigned int dio_flags, size_t done_before)\n+ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,\n+\t\tstruct iomap_iter *iomi)\n {\n-\tstruct inode *inode = file_inode(iocb-\u003eki_filp);\n-\tsize_t count = iov_iter_count(iter);\n-\n-\tif (dops || done_before)\n-\t\treturn false;\n-\tif (iov_iter_rw(iter) != READ)\n-\t\treturn false;\n-\tif (!count)\n-\t\treturn false;\n-\t/*\n-\t * Simple dio is an optimization for small IO. Filter out large IO\n-\t * early as it's the most common case to fail for typical direct IO\n-\t * workloads.\n-\t */\n-\tif (count \u003e inode-\u003ei_sb-\u003es_blocksize)\n-\t\treturn false;\n-\tif (dio_flags \u0026 (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |\n-\t\t\t IOMAP_DIO_BOUNCE))\n-\t\treturn false;\n-\tif (iocb-\u003eki_pos + count \u003e i_size_read(inode))\n-\t\treturn false;\n-\tif (IS_ENCRYPTED(inode))\n-\t\treturn false;\n-\n-\treturn true;\n-}\n-\n-/*\n- * Fast path for small, block-aligned direct I/Os that map to a single\n- * contiguous on-disk extent.\n- *\n- * iomap_dio_simple_supported() enforces the cheap up-front constraints before\n- * entering this path.\n- *\n- * @dops must be NULL: a non-NULL @dops means the caller wants its\n- * -\u003eend_io / -\u003esubmit_io hooks invoked, and in particular wants its bios to be\n- * allocated from the filesystem-private @dops-\u003ebio_set (whose front_pad sizes a\n- * filesystem-private wrapper around the bio). The fast path instead allocates\n- * from the shared iomap_dio_simple_pool, whose front_pad matches struct\n- * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall\n- * back to __iomap_dio_rw() in that case.\n- *\n- * @done_before must be zero: a non-zero caller-accumulated residual cannot be\n- * carried through a single-bio inline completion.\n- *\n- * @iter must describe a non-empty READ no larger than the inode block size:\n- * writes, zero-length I/O, and larger requests need the generic iomap direct\n- * I/O path.\n- *\n- * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or\n- * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct\n- * I/O, or bouncing. The range must also stay within i_size and encrypted\n- * inodes must use the generic iomap direct I/O path.\n- *\n- * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it\n- * decides the request does not fit the fast path. In that case we proceed to\n- * the generic __iomap_dio_rw() slow path. Any other errno is a real result and\n- * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the\n- * caller.\n- */\n-static ssize_t\n-iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,\n-\t\t const struct iomap_ops *ops, void *private,\n-\t\t unsigned int dio_flags)\n-{\n-\tstruct inode *inode = file_inode(iocb-\u003eki_filp);\n-\tsize_t count = iov_iter_count(iter);\n-\tbool wait_for_completion = is_sync_kiocb(iocb);\n-\tstruct iomap_iter iomi = {\n-\t\t.inode\t\t= inode,\n-\t\t.pos\t\t= iocb-\u003eki_pos,\n-\t\t.len\t\t= count,\n-\t\t.flags\t\t= IOMAP_DIRECT,\n-\t\t.private\t= private,\n-\t};\n+\tgfp_t gfp = (iomi-\u003eflags \u0026 IOMAP_NOWAIT) ? GFP_NOWAIT : GFP_KERNEL;\n \tstruct iomap_dio_simple *sr;\n \tunsigned int alignment;\n \tstruct bio *bio;\n \tssize_t ret;\n \n-\tif (iocb-\u003eki_flags \u0026 IOCB_NOWAIT)\n-\t\tiomi.flags |= IOMAP_NOWAIT;\n-\n-\tret = kiocb_write_and_wait(iocb, count);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\tinode_dio_begin(inode);\n-\n-\tret = ops-\u003eiomap_begin(inode, iomi.pos, count, iomi.flags,\n-\t\t\t \u0026iomi.iomap, \u0026iomi.srcmap);\n-\tif (ret) {\n-\t\tinode_dio_end(inode);\n-\t\treturn ret;\n-\t}\n-\n-\tif (iomi.iomap.type != IOMAP_MAPPED ||\n-\t iomi.iomap.offset + iomi.iomap.length \u003c iomi.pos + count ||\n-\t (iomi.iomap.flags \u0026 IOMAP_F_INTEGRITY)) {\n+\tif (iomi-\u003eiomap.type != IOMAP_MAPPED ||\n+\t iomi-\u003eiomap.offset + iomi-\u003eiomap.length \u003c iomi-\u003epos + iomi-\u003elen ||\n+\t (iomi-\u003eiomap.flags \u0026 IOMAP_F_INTEGRITY)) {\n \t\tret = -ENOTBLK;\n-\t\tgoto out_iomap_end;\n+\t\tgoto out_dio_end;\n \t}\n \n-\talignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);\n-\tif ((iomi.pos | count) \u0026 (alignment - 1)) {\n+\talignment = iomap_dio_alignment(iomi-\u003einode, iomi-\u003eiomap.bdev, 0);\n+\tif ((iomi-\u003epos | iomi-\u003elen) \u0026 (alignment - 1)) {\n \t\tret = -EINVAL;\n-\t\tgoto out_iomap_end;\n+\t\tgoto out_dio_end;\n \t}\n \n-\tif (!wait_for_completion \u0026\u0026 unlikely(!inode-\u003ei_sb-\u003es_dio_done_wq)) {\n-\t\tret = sb_init_dio_done_wq(inode-\u003ei_sb);\n+\tif (unlikely(!iomi-\u003einode-\u003ei_sb-\u003es_dio_done_wq \u0026\u0026\n+\t\t\t!is_sync_kiocb(iocb))) {\n+\t\tret = sb_init_dio_done_wq(iomi-\u003einode-\u003ei_sb);\n \t\tif (ret \u003c 0)\n-\t\t\tgoto out_iomap_end;\n+\t\t\tgoto out_dio_end;\n \t}\n \n-\ttrace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);\n-\n-\tif (user_backed_iter(iter))\n-\t\tdio_flags |= IOMAP_DIO_USER_BACKED;\n+\ttrace_iomap_dio_rw_begin(iocb, iter, 0, 0);\n \n-\tbio = bio_alloc_bioset(iomi.iomap.bdev,\n+\tbio = bio_alloc_bioset(iomi-\u003eiomap.bdev,\n \t\t\t bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),\n-\t\t\t REQ_OP_READ, GFP_KERNEL, \u0026iomap_dio_simple_pool);\n+\t\t\t REQ_OP_READ, gfp, \u0026iomap_dio_simple_pool);\n+\tif (!bio) {\n+\t\tret = -EAGAIN;\n+\t\tgoto out_dio_end;\n+\t}\n \tsr = container_of(bio, struct iomap_dio_simple, bio);\n \tsr-\u003eiocb = iocb;\n-\tsr-\u003edio_flags = dio_flags;\n+\tsr-\u003edio_flags = 0;\n \n-\tbio-\u003ebi_iter.bi_sector = iomap_sector(\u0026iomi.iomap, iomi.pos);\n+\tbio-\u003ebi_iter.bi_sector = iomap_sector(\u0026iomi-\u003eiomap, iomi-\u003epos);\n \tbio-\u003ebi_ioprio = iocb-\u003eki_ioprio;\n \n \tret = bio_iov_iter_get_pages(bio, iter, alignment - 1);\n \tif (unlikely(ret))\n \t\tgoto out_bio_put;\n \n-\tif (bio-\u003ebi_iter.bi_size != count) {\n+\tif (bio-\u003ebi_iter.bi_size != iomi-\u003elen) {\n \t\tiov_iter_revert(iter, bio-\u003ebi_iter.bi_size);\n \t\tret = -ENOTBLK;\n \t\tgoto out_bio_release_pages;\n \t}\n \n \tsr-\u003esize = bio-\u003ebi_iter.bi_size;\n-\n-\tif (dio_flags \u0026 IOMAP_DIO_USER_BACKED)\n+\tif (user_backed_iter(iter)) {\n \t\tbio_set_pages_dirty(bio);\n+\t\tsr-\u003edio_flags |= IOMAP_DIO_USER_BACKED;\n+\t}\n \n \tif (iocb-\u003eki_flags \u0026 IOCB_NOWAIT)\n \t\tbio-\u003ebi_opf |= REQ_NOWAIT;\n-\tif ((iocb-\u003eki_flags \u0026 IOCB_HIPRI) \u0026\u0026 !wait_for_completion) {\n-\t\tbio-\u003ebi_opf |= REQ_POLLED;\n-\t\tWRITE_ONCE(iocb-\u003eprivate, bio);\n-\t}\n-\n-\tif (ops-\u003eiomap_end)\n-\t\tops-\u003eiomap_end(inode, iomi.pos, count, count, iomi.flags,\n-\t\t\t \u0026iomi.iomap);\n \n-\tif (!wait_for_completion) {\n-\t\tbio-\u003ebi_end_io = iomap_dio_simple_end_io;\n-\t\tsubmit_bio(bio);\n-\t\ttrace_iomap_dio_rw_queued(inode, iomi.pos, count);\n-\t\treturn -EIOCBQUEUED;\n+\tif (is_sync_kiocb(iocb)) {\n+\t\tsubmit_bio_wait(bio);\n+\t\treturn iomap_dio_simple_complete(sr);\n \t}\n \n-\tsubmit_bio_wait(bio);\n-\treturn iomap_dio_simple_complete(sr);\n+\tif ((iocb-\u003eki_flags \u0026 IOCB_HIPRI)) {\n+\t\tbio-\u003ebi_opf |= REQ_POLLED;\n+\t\tWRITE_ONCE(iocb-\u003eprivate, bio);\n+\t}\n+\tbio-\u003ebi_end_io = iomap_dio_simple_end_io;\n+\tsubmit_bio(bio);\n+\ttrace_iomap_dio_rw_queued(iomi-\u003einode, iocb-\u003eki_pos, iomi-\u003elen);\n+\treturn -EIOCBQUEUED;\n \n out_bio_release_pages:\n \tbio_release_pages(bio, false);\n out_bio_put:\n \tbio_put(bio);\n-out_iomap_end:\n-\tif (ops-\u003eiomap_end)\n-\t\tops-\u003eiomap_end(inode, iomi.pos, count, 0, iomi.flags,\n-\t\t\t \u0026iomi.iomap);\n-\tinode_dio_end(inode);\n+out_dio_end:\n+\tinode_dio_end(iomi-\u003einode);\n \treturn ret;\n }\n-\n-ssize_t\n-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops, const struct iomap_dio_ops *dops,\n-\t\tunsigned int dio_flags, void *private, size_t done_before)\n-{\n-\tstruct iomap_dio *dio;\n-\tssize_t ret;\n-\n-\tif (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,\n-\t\t\t\t done_before)) {\n-\t\tret = iomap_dio_simple(iocb, iter, ops, private, dio_flags);\n-\t\tif (ret != -ENOTBLK)\n-\t\t\treturn ret;\n-\t}\n-\n-\tdio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,\n-\t\t\t done_before);\n-\tif (IS_ERR_OR_NULL(dio))\n-\t\treturn PTR_ERR_OR_ZERO(dio);\n-\treturn iomap_dio_complete(dio);\n-}\n-EXPORT_SYMBOL_GPL(iomap_dio_rw);\n+EXPORT_SYMBOL_GPL(__iomap_dio_read_simple);\n \n static int __init iomap_dio_init(void)\n {\ndiff --git a/fs/iomap/fiemap.c b/fs/iomap/fiemap.c\nindex d11dadff82865..c79c13cc617bd 100644\n--- a/fs/iomap/fiemap.c\n+++ b/fs/iomap/fiemap.c\n@@ -56,7 +56,7 @@ static int iomap_fiemap_iter(struct iomap_iter *iter,\n }\n \n int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,\n-\t\tu64 start, u64 len, const struct iomap_ops *ops)\n+\t\tu64 start, u64 len, iomap_iter_next_fn next)\n {\n \tstruct iomap_iter iter = {\n \t\t.inode\t\t= inode,\n@@ -73,7 +73,7 @@ int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,\n \tif (ret)\n \t\treturn ret;\n \n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_fiemap_iter(\u0026iter, fi, \u0026prev);\n \n \tif (prev.type != IOMAP_HOLE) {\n@@ -92,7 +92,7 @@ EXPORT_SYMBOL_GPL(iomap_fiemap);\n /* legacy -\u003ebmap interface. 0 is the error return (!) */\n sector_t\n iomap_bmap(struct address_space *mapping, sector_t bno,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_iter iter = {\n \t\t.inode\t= mapping-\u003ehost,\n@@ -107,7 +107,7 @@ iomap_bmap(struct address_space *mapping, sector_t bno,\n \t\treturn 0;\n \n \tbno = 0;\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0) {\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0) {\n \t\tif (iter.iomap.type == IOMAP_MAPPED)\n \t\t\tbno = iomap_sector(\u0026iter.iomap, iter.pos) \u003e\u003e blkshift;\n \t\t/* leave iter.status unset to abort loop */\ndiff --git a/fs/iomap/iter.c b/fs/iomap/iter.c\nindex e4a29829591a7..89071b0a8b9c6 100644\n--- a/fs/iomap/iter.c\n+++ b/fs/iomap/iter.c\n@@ -6,15 +6,6 @@\n #include \u003clinux/iomap.h\u003e\n #include \"trace.h\"\n \n-static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)\n-{\n-\tif (iter-\u003eiomap.flags \u0026 IOMAP_F_FOLIO_BATCH) {\n-\t\tfolio_batch_release(iter-\u003efbatch);\n-\t\tfolio_batch_reinit(iter-\u003efbatch);\n-\t\titer-\u003eiomap.flags \u0026= ~IOMAP_F_FOLIO_BATCH;\n-\t}\n-}\n-\n /* Advance the current iterator position and decrement the remaining length */\n int iomap_iter_advance(struct iomap_iter *iter, u64 count)\n {\n@@ -40,51 +31,28 @@ static inline void iomap_iter_done(struct iomap_iter *iter)\n }\n \n /**\n- * iomap_iter - iterate over a ranges in a file\n- * @iter: iteration structue\n- * @ops: iomap ops provided by the file system\n+ * iomap_iter_continue - decide whether iteration should continue\n+ * @iter: iteration structure\n+ * @iomap: the mapping that was just processed\n+ * @srcmap: the source mapping that was just processed\n *\n- * Iterate over filesystem-provided space mappings for the provided file range.\n+ * Helper normally called via iomap_iter_next(). Called after the previous\n+ * mapping has been finished to determine whether there is more of the file\n+ * range left to process.\n *\n- * This function handles cleanup of resources acquired for iteration when the\n- * filesystem indicates there are no more space mappings, which means that this\n- * function must be called in a loop that continues as long it returns a\n- * positive value. If 0 or a negative value is returned, the caller must not\n- * return to the loop body. Within a loop body, there are two ways to break out\n- * of the loop body: leave @iter.status unchanged, or set it to a negative\n- * errno.\n+ * Returns 1 if there is more work to do, in which case @iomap and @srcmap are\n+ * cleared so the caller can produce the next mapping; zero if the range is\n+ * fully consumed; or a negative errno on error. Any folio batch attached to\n+ * the mapping is released before returning.\n */\n-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)\n+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap, int ret)\n {\n-\tbool stale = iter-\u003eiomap.flags \u0026 IOMAP_F_STALE;\n-\tssize_t advanced;\n-\tu64 olen;\n-\tint ret;\n-\n-\ttrace_iomap_iter(iter, ops, _RET_IP_);\n-\n-\tif (!iter-\u003eiomap.length)\n-\t\tgoto begin;\n-\n-\t/*\n-\t * Calculate how far the iter was advanced and the original length bytes\n-\t * for -\u003eiomap_end().\n-\t */\n-\tadvanced = iter-\u003epos - iter-\u003eiter_start_pos;\n-\tolen = iter-\u003elen + advanced;\n+\tconst bool stale = iomap-\u003eflags \u0026 IOMAP_F_STALE;\n+\tconst ssize_t advanced = iter-\u003epos - iter-\u003eiter_start_pos;\n \n-\tif (ops-\u003eiomap_end) {\n-\t\tret = ops-\u003eiomap_end(iter-\u003einode, iter-\u003eiter_start_pos,\n-\t\t\t\tiomap_length_trim(iter, iter-\u003eiter_start_pos,\n-\t\t\t\t\t\t olen),\n-\t\t\t\tadvanced, iter-\u003eflags, \u0026iter-\u003eiomap);\n-\t\tif (ret \u003c 0 \u0026\u0026 !advanced)\n-\t\t\treturn ret;\n-\t}\n-\n-\t/* detect old return semantics where this would advance */\n-\tif (WARN_ON_ONCE(iter-\u003estatus \u003e 0))\n-\t\titer-\u003estatus = -EIO;\n+\tif (ret \u003c 0 \u0026\u0026 !advanced)\n+\t\treturn ret;\n \n \t/*\n \t * Use iter-\u003elen to determine whether to continue onto the next mapping.\n@@ -92,25 +60,57 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)\n \t * advanced at all (i.e. no work was done for some reason) unless the\n \t * mapping has been marked stale and needs to be reprocessed.\n \t */\n-\tif (iter-\u003estatus \u003c 0)\n+\tif (WARN_ON_ONCE(iter-\u003estatus \u003e 0))\n+\t\t/* detect old return semantics where this would advance */\n+\t\tret = -EIO;\n+\telse if (iter-\u003estatus \u003c 0)\n \t\tret = iter-\u003estatus;\n \telse if (iter-\u003elen == 0 || (!advanced \u0026\u0026 !stale))\n \t\tret = 0;\n \telse\n \t\tret = 1;\n-\tiomap_iter_clean_fbatch(iter);\n-\titer-\u003estatus = 0;\n+\n+\tif (iomap-\u003eflags \u0026 IOMAP_F_FOLIO_BATCH) {\n+\t\tfolio_batch_release(iter-\u003efbatch);\n+\t\tfolio_batch_reinit(iter-\u003efbatch);\n+\t\tiomap-\u003eflags \u0026= ~IOMAP_F_FOLIO_BATCH;\n+\t}\n+\n \tif (ret \u003c= 0)\n \t\treturn ret;\n \n-\tmemset(\u0026iter-\u003eiomap, 0, sizeof(iter-\u003eiomap));\n-\tmemset(\u0026iter-\u003esrcmap, 0, sizeof(iter-\u003esrcmap));\n+\tmemset(iomap, 0, sizeof(*iomap));\n+\tmemset(srcmap, 0, sizeof(*srcmap));\n \n-begin:\n-\tret = ops-\u003eiomap_begin(iter-\u003einode, iter-\u003epos, iter-\u003elen, iter-\u003eflags,\n-\t\t\t \u0026iter-\u003eiomap, \u0026iter-\u003esrcmap);\n-\tif (ret \u003c 0)\n-\t\treturn ret;\n-\tiomap_iter_done(iter);\n-\treturn 1;\n+\treturn ret;\n+}\n+EXPORT_SYMBOL_GPL(iomap_iter_continue);\n+\n+/**\n+ * iomap_iter - iterate over ranges in a file\n+ * @iter: iteration structure\n+ * @next: the iomap iteration function for the filesystem\n+ *\n+ * Iterate over filesystem-provided space mappings for the provided file range.\n+ *\n+ * This function handles cleanup of resources acquired for iteration when the\n+ * filesystem indicates there are no more space mappings, which means that this\n+ * function must be called in a loop that continues as long it returns a\n+ * positive value. If 0 or a negative value is returned, the caller must not\n+ * return to the loop body. Within a loop body, there are two ways to break out\n+ * of the loop body: leave @iter.status unchanged, or set it to a negative\n+ * errno.\n+ */\n+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next)\n+{\n+\tint ret;\n+\n+\ttrace_iomap_iter(iter, next, _RET_IP_);\n+\n+\tret = next(iter, \u0026iter-\u003eiomap, \u0026iter-\u003esrcmap);\n+\titer-\u003estatus = 0;\n+\tif (ret \u003e 0)\n+\t\tiomap_iter_done(iter);\n+\n+\treturn ret;\n }\ndiff --git a/fs/iomap/seek.c b/fs/iomap/seek.c\nindex 6cbc587c93dab..6fd3173d25f51 100644\n--- a/fs/iomap/seek.c\n+++ b/fs/iomap/seek.c\n@@ -27,7 +27,7 @@ static int iomap_seek_hole_iter(struct iomap_iter *iter,\n }\n \n loff_t\n-iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)\n+iomap_seek_hole(struct inode *inode, loff_t pos, iomap_iter_next_fn next)\n {\n \tloff_t size = i_size_read(inode);\n \tstruct iomap_iter iter = {\n@@ -42,7 +42,7 @@ iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)\n \t\treturn -ENXIO;\n \n \titer.len = size - pos;\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_seek_hole_iter(\u0026iter, \u0026pos);\n \tif (ret \u003c 0)\n \t\treturn ret;\n@@ -73,7 +73,7 @@ static int iomap_seek_data_iter(struct iomap_iter *iter,\n }\n \n loff_t\n-iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)\n+iomap_seek_data(struct inode *inode, loff_t pos, iomap_iter_next_fn next)\n {\n \tloff_t size = i_size_read(inode);\n \tstruct iomap_iter iter = {\n@@ -88,7 +88,7 @@ iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)\n \t\treturn -ENXIO;\n \n \titer.len = size - pos;\n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_seek_data_iter(\u0026iter, \u0026pos);\n \tif (ret \u003c 0)\n \t\treturn ret;\ndiff --git a/fs/iomap/swapfile.c b/fs/iomap/swapfile.c\nindex 0db77c449467a..c59f67a7de097 100644\n--- a/fs/iomap/swapfile.c\n+++ b/fs/iomap/swapfile.c\n@@ -139,7 +139,7 @@ static int iomap_swapfile_iter(struct iomap_iter *iter,\n */\n int iomap_swapfile_activate(struct swap_info_struct *sis,\n \t\tstruct file *swap_file, sector_t *pagespan,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct inode *inode = swap_file-\u003ef_mapping-\u003ehost;\n \tstruct iomap_iter iter = {\n@@ -163,7 +163,7 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,\n \tif (ret)\n \t\treturn ret;\n \n-\twhile ((ret = iomap_iter(\u0026iter, ops)) \u003e 0)\n+\twhile ((ret = iomap_iter(\u0026iter, next)) \u003e 0)\n \t\titer.status = iomap_swapfile_iter(\u0026iter, \u0026iter.iomap, \u0026isi);\n \tif (ret \u003c 0)\n \t\treturn ret;\ndiff --git a/fs/iomap/trace.h b/fs/iomap/trace.h\nindex e1ea8392cf47d..d36a77b854aa5 100644\n--- a/fs/iomap/trace.h\n+++ b/fs/iomap/trace.h\n@@ -218,9 +218,9 @@ TRACE_EVENT(iomap_add_to_ioend,\n );\n \n TRACE_EVENT(iomap_iter,\n-\tTP_PROTO(struct iomap_iter *iter, const void *ops,\n+\tTP_PROTO(struct iomap_iter *iter, const void *next,\n \t\t unsigned long caller),\n-\tTP_ARGS(iter, ops, caller),\n+\tTP_ARGS(iter, next, caller),\n \tTP_STRUCT__entry(\n \t\t__field(dev_t, dev)\n \t\t__field(u64, ino)\n@@ -228,7 +228,7 @@ TRACE_EVENT(iomap_iter,\n \t\t__field(u64, length)\n \t\t__field(int, status)\n \t\t__field(unsigned int, flags)\n-\t\t__field(const void *, ops)\n+\t\t__field(const void *, next)\n \t\t__field(unsigned long, caller)\n \t),\n \tTP_fast_assign(\n@@ -238,10 +238,10 @@ TRACE_EVENT(iomap_iter,\n \t\t__entry-\u003elength = iomap_length(iter);\n \t\t__entry-\u003estatus = iter-\u003estatus;\n \t\t__entry-\u003eflags = iter-\u003eflags;\n-\t\t__entry-\u003eops = ops;\n+\t\t__entry-\u003enext = next;\n \t\t__entry-\u003ecaller = caller;\n \t),\n-\tTP_printk(\"dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) ops %ps caller %pS\",\n+\tTP_printk(\"dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) next %ps caller %pS\",\n \t\t MAJOR(__entry-\u003edev), MINOR(__entry-\u003edev),\n \t\t __entry-\u003eino,\n \t\t __entry-\u003epos,\n@@ -249,7 +249,7 @@ TRACE_EVENT(iomap_iter,\n \t\t __entry-\u003estatus,\n \t\t __print_flags(__entry-\u003eflags, \"|\", IOMAP_FLAGS_STRINGS),\n \t\t __entry-\u003eflags,\n-\t\t __entry-\u003eops,\n+\t\t __entry-\u003enext,\n \t\t (void *)__entry-\u003ecaller)\n );\n \ndiff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c\nindex 1fbf832ad1654..43ad597ed491e 100644\n--- a/fs/ntfs/aops.c\n+++ b/fs/ntfs/aops.c\n@@ -97,7 +97,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)\n \t\t\treturn ntfs_read_compressed_block(folio);\n \t}\n \n-\tiomap_read_folio(\u0026ntfs_read_iomap_ops, \u0026ctx, NULL);\n+\tiomap_read_folio(ntfs_read_iomap_next, \u0026ctx, NULL);\n \treturn 0;\n }\n \n@@ -238,7 +238,7 @@ static void ntfs_readahead(struct readahead_control *rac)\n \t */\n \tif (!NInoNonResident(ni) || NInoCompressed(ni))\n \t\treturn;\n-\tiomap_readahead(\u0026ntfs_read_iomap_ops, \u0026ctx, NULL);\n+\tiomap_readahead(ntfs_read_iomap_next, \u0026ctx, NULL);\n }\n \n static int ntfs_writepages(struct address_space *mapping,\n@@ -274,7 +274,7 @@ static int ntfs_swap_activate(struct swap_info_struct *sis,\n \t\tstruct file *swap_file, sector_t *span)\n {\n \treturn iomap_swapfile_activate(sis, swap_file, span,\n-\t\t\t\u0026ntfs_read_iomap_ops);\n+\t\t\tntfs_read_iomap_next);\n }\n \n const struct address_space_operations ntfs_aops = {\ndiff --git a/fs/ntfs/file.c b/fs/ntfs/file.c\nindex 6a7b638e523d9..a4f99128b46c5 100644\n--- a/fs/ntfs/file.c\n+++ b/fs/ntfs/file.c\n@@ -281,7 +281,7 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr)\n \t\t\t\tround_up(old_size, PAGE_SIZE) - old_size,\n \t\t\t\tattr-\u003eia_size - old_size);\n \t\terr = iomap_zero_range(vi, old_size, len,\n-\t\t\t\tNULL, \u0026ntfs_seek_iomap_ops,\n+\t\t\t\tNULL, ntfs_seek_iomap_next,\n \t\t\t\t\u0026ntfs_iomap_folio_ops, NULL);\n \t}\n \n@@ -417,12 +417,12 @@ static loff_t ntfs_file_llseek(struct file *file, loff_t offset, int whence)\n \tswitch (whence) {\n \tcase SEEK_HOLE:\n \t\tinode_lock_shared(inode);\n-\t\toffset = iomap_seek_hole(inode, offset, \u0026ntfs_seek_iomap_ops);\n+\t\toffset = iomap_seek_hole(inode, offset, ntfs_seek_iomap_next);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \tcase SEEK_DATA:\n \t\tinode_lock_shared(inode);\n-\t\toffset = iomap_seek_data(inode, offset, \u0026ntfs_seek_iomap_ops);\n+\t\toffset = iomap_seek_data(inode, offset, ntfs_seek_iomap_next);\n \t\tinode_unlock_shared(inode);\n \t\tbreak;\n \tdefault:\n@@ -458,7 +458,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\t}\n \n \t\tfile_accessed(iocb-\u003eki_filp);\n-\t\tret = iomap_dio_rw(iocb, to, \u0026ntfs_read_iomap_ops, NULL, 0,\n+\t\tret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,\n \t\t\t\tNULL, 0);\n \t} else {\n \t\tret = generic_file_read_iter(iocb, to);\n@@ -496,7 +496,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)\n {\n \tssize_t ret;\n \n-\tret = iomap_dio_rw(iocb, from, \u0026ntfs_dio_iomap_ops,\n+\tret = iomap_dio_rw(iocb, from, ntfs_dio_iomap_next,\n \t\t\t\u0026ntfs_write_dio_ops, 0, NULL, 0);\n \tif (ret == -ENOTBLK)\n \t\tret = 0;\n@@ -511,7 +511,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\toffset = iocb-\u003eki_pos;\n \t\tiocb-\u003eki_flags \u0026= ~IOCB_DIRECT;\n \t\twritten = iomap_file_buffered_write(iocb, from,\n-\t\t\t\t\u0026ntfs_write_iomap_ops, \u0026ntfs_iomap_folio_ops,\n+\t\t\t\tntfs_write_iomap_next, \u0026ntfs_iomap_folio_ops,\n \t\t\t\tNULL);\n \t\tif (written \u003c 0) {\n \t\t\tret = written;\n@@ -594,7 +594,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \tif (NInoNonResident(ni) \u0026\u0026 iocb-\u003eki_flags \u0026 IOCB_DIRECT)\n \t\tret = ntfs_dio_write_iter(iocb, from);\n \telse\n-\t\tret = iomap_file_buffered_write(iocb, from, \u0026ntfs_write_iomap_ops,\n+\t\tret = iomap_file_buffered_write(iocb, from, ntfs_write_iomap_next,\n \t\t\t\t\u0026ntfs_iomap_folio_ops, NULL);\n out:\n \tif (ret \u003c 0 \u0026\u0026 ret != -EIOCBQUEUED) {\n@@ -623,7 +623,7 @@ static vm_fault_t ntfs_filemap_page_mkwrite(struct vm_fault *vmf)\n \tsb_start_pagefault(inode-\u003ei_sb);\n \tfile_update_time(vmf-\u003evma-\u003evm_file);\n \n-\tret = iomap_page_mkwrite(vmf, \u0026ntfs_page_mkwrite_iomap_ops, NULL);\n+\tret = iomap_page_mkwrite(vmf, ntfs_page_mkwrite_iomap_next, NULL);\n \tsb_end_pagefault(inode-\u003ei_sb);\n \treturn ret;\n }\n@@ -670,7 +670,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc)\n static int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \t\tu64 start, u64 len)\n {\n-\treturn iomap_fiemap(inode, fieinfo, start, len, \u0026ntfs_read_iomap_ops);\n+\treturn iomap_fiemap(inode, fieinfo, start, len, ntfs_read_iomap_next);\n }\n \n static const char *ntfs_get_link(struct dentry *dentry, struct inode *inode,\n@@ -911,7 +911,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,\n \t\t\t\t ntfs_cluster_to_bytes(vol, start_vcn + 1),\n \t\t\t\t end_offset);\n \t\t\terr = iomap_zero_range(vi, offset, to - offset,\n-\t\t\t\t\t NULL, \u0026ntfs_seek_iomap_ops,\n+\t\t\t\t\t NULL, ntfs_seek_iomap_next,\n \t\t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\t\tif (err \u003c 0)\n \t\t\t\tgoto out;\n@@ -927,7 +927,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,\n \t\tfrom = ntfs_cluster_to_bytes(vol, end_vcn - 1);\n \t\tif (from \u003c ni-\u003einitialized_size) {\n \t\t\terr = iomap_zero_range(vi, from, end_offset - from,\n-\t\t\t\t\t NULL, \u0026ntfs_seek_iomap_ops,\n+\t\t\t\t\t NULL, ntfs_seek_iomap_next,\n \t\t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\t\tif (err \u003c 0)\n \t\t\t\tgoto out;\n@@ -1131,7 +1131,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le\n \t\t\t\t\t round_up(old_size, PAGE_SIZE) - old_size,\n \t\t\t\t\t offset - old_size);\n \t\t\terr = iomap_zero_range(vi, old_size, len, NULL,\n-\t\t\t\t\t \u0026ntfs_seek_iomap_ops,\n+\t\t\t\t\t ntfs_seek_iomap_next,\n \t\t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\t}\n \t\tNInoSetFileNameDirty(ni);\ndiff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c\nindex c2715521e5628..05132d92e87b2 100644\n--- a/fs/ntfs/inode.c\n+++ b/fs/ntfs/inode.c\n@@ -2415,7 +2415,7 @@ int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset,\n \tif (!NInoCompressed(ni) \u0026\u0026 old_init_size \u003c offset) {\n \t\terr = iomap_zero_range(vi, old_init_size,\n \t\t\t\t offset - old_init_size,\n-\t\t\t\t NULL, \u0026ntfs_seek_iomap_ops,\n+\t\t\t\t NULL, ntfs_seek_iomap_next,\n \t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\tif (err)\n \t\t\treturn err;\ndiff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c\nindex 52eecf5cb256a..271902e692003 100644\n--- a/fs/ntfs/iomap.c\n+++ b/fs/ntfs/iomap.c\n@@ -277,9 +277,7 @@ static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t leng\n \t\t\tsrcmap, true);\n }\n \n-const struct iomap_ops ntfs_read_iomap_ops = {\n-\t.iomap_begin = ntfs_read_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin);\n \n /*\n * Check that the cached iomap still matches the NTFS runlist before\n@@ -329,14 +327,10 @@ static int ntfs_zero_read_iomap_end(struct inode *inode, loff_t pos, loff_t leng\n \treturn written;\n }\n \n-static const struct iomap_ops ntfs_zero_read_iomap_ops = {\n-\t.iomap_begin = ntfs_seek_iomap_begin,\n-\t.iomap_end = ntfs_zero_read_iomap_end,\n-};\n+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_zero_read_iomap_next,\n+\t\tntfs_seek_iomap_begin, ntfs_zero_read_iomap_end);\n \n-const struct iomap_ops ntfs_seek_iomap_ops = {\n-\t.iomap_begin = ntfs_seek_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(ntfs_seek_iomap_next, ntfs_seek_iomap_begin);\n \n int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length)\n {\n@@ -355,7 +349,7 @@ static int ntfs_zero_range(struct inode *inode, loff_t offset, loff_t length)\n \treturn iomap_zero_range(inode,\n \t\t\t\toffset, length,\n \t\t\t\tNULL,\n-\t\t\t\t\u0026ntfs_zero_read_iomap_ops,\n+\t\t\t\tntfs_zero_read_iomap_next,\n \t\t\t\t\u0026ntfs_zero_iomap_folio_ops,\n \t\t\t\tNULL);\n }\n@@ -764,10 +758,8 @@ static int ntfs_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,\n \treturn written;\n }\n \n-const struct iomap_ops ntfs_write_iomap_ops = {\n-\t.iomap_begin\t\t= ntfs_write_iomap_begin,\n-\t.iomap_end\t\t= ntfs_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(ntfs_write_iomap_next, ntfs_write_iomap_begin,\n+\t\tntfs_write_iomap_end);\n \n static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,\n \t\t\t\t loff_t length, unsigned int flags,\n@@ -777,10 +769,8 @@ static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,\n \t\t\tNTFS_IOMAP_FLAGS_MKWRITE);\n }\n \n-const struct iomap_ops ntfs_page_mkwrite_iomap_ops = {\n-\t.iomap_begin\t\t= ntfs_page_mkwrite_iomap_begin,\n-\t.iomap_end\t\t= ntfs_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(ntfs_page_mkwrite_iomap_next,\n+\t\tntfs_page_mkwrite_iomap_begin, ntfs_write_iomap_end);\n \n static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,\n \t\t\t\t loff_t length, unsigned int flags,\n@@ -790,10 +780,8 @@ static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,\n \t\t\tNTFS_IOMAP_FLAGS_DIO);\n }\n \n-const struct iomap_ops ntfs_dio_iomap_ops = {\n-\t.iomap_begin\t\t= ntfs_dio_iomap_begin,\n-\t.iomap_end\t\t= ntfs_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(ntfs_dio_iomap_next, ntfs_dio_iomap_begin,\n+\t\tntfs_write_iomap_end);\n \n static ssize_t ntfs_writeback_range(struct iomap_writepage_ctx *wpc,\n \t\tstruct folio *folio, u64 offset, unsigned int len, u64 end_pos)\ndiff --git a/fs/ntfs/iomap.h b/fs/ntfs/iomap.h\nindex 3abc1d493e918..69443de1fefd2 100644\n--- a/fs/ntfs/iomap.h\n+++ b/fs/ntfs/iomap.h\n@@ -12,11 +12,16 @@\n #include \"volume.h\"\n #include \"inode.h\"\n \n-extern const struct iomap_ops ntfs_write_iomap_ops;\n-extern const struct iomap_ops ntfs_read_iomap_ops;\n-extern const struct iomap_ops ntfs_seek_iomap_ops;\n-extern const struct iomap_ops ntfs_page_mkwrite_iomap_ops;\n-extern const struct iomap_ops ntfs_dio_iomap_ops;\n+int ntfs_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int ntfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int ntfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n+int ntfs_dio_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n extern const struct iomap_writeback_ops ntfs_writeback_ops;\n extern const struct iomap_write_ops ntfs_iomap_folio_ops;\n extern int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length);\ndiff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c\nindex d601f088618c0..55844b42920a1 100644\n--- a/fs/ntfs3/file.c\n+++ b/fs/ntfs3/file.c\n@@ -315,7 +315,7 @@ static int ntfs_extend_initialized_size(struct file *file,\n \t}\n \n \terr = iomap_zero_range(inode, valid, new_valid - valid, NULL,\n-\t\t\t \u0026ntfs_iomap_ops, \u0026ntfs_iomap_folio_ops, NULL);\n+\t\t\t ntfs_iomap_next, \u0026ntfs_iomap_folio_ops, NULL);\n \tif (err) {\n \t\tni-\u003ei_valid = valid;\n \t\tntfs_inode_warn(inode,\n@@ -554,7 +554,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)\n \t\t/* Zero head of punch. */\n \t\tif (tmp \u003e from) {\n \t\t\terr = iomap_zero_range(inode, from, tmp - from, NULL,\n-\t\t\t\t\t \u0026ntfs_iomap_ops,\n+\t\t\t\t\t ntfs_iomap_next,\n \t\t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\t\tif (err)\n \t\t\t\tgoto out;\n@@ -572,7 +572,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)\n \t\t/* Zero tail of punch. */\n \t\tif (vbo \u003c end_a \u0026\u0026 end_a \u003c end) {\n \t\t\terr = iomap_zero_range(inode, end_a, end - end_a, NULL,\n-\t\t\t\t\t \u0026ntfs_iomap_ops,\n+\t\t\t\t\t ntfs_iomap_next,\n \t\t\t\t\t \u0026ntfs_iomap_folio_ops, NULL);\n \t\t\tif (err)\n \t\t\t\tgoto out;\n@@ -872,7 +872,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)\n \t\t\t\tgoto out;\n \t\t}\n \n-\t\terr = iomap_dio_rw(iocb, iter, \u0026ntfs_iomap_ops, NULL, dio_flags,\n+\t\terr = iomap_dio_rw(iocb, iter, ntfs_iomap_next, NULL, dio_flags,\n \t\t\t\t NULL, 0);\n \n \t\tif (err \u003c= 0)\n@@ -1286,7 +1286,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t !ntfs_should_use_dio(iocb, from)) {\n \t\tiocb-\u003eki_flags \u0026= ~IOCB_DIRECT;\n \n-\t\tret = iomap_file_buffered_write(iocb, from, \u0026ntfs_iomap_ops,\n+\t\tret = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,\n \t\t\t\t\t\t\u0026ntfs_iomap_folio_ops, NULL);\n \t\tinode_unlock(inode);\n \n@@ -1303,7 +1303,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\t\tgoto out;\n \t}\n \n-\tret = iomap_dio_rw(iocb, from, \u0026ntfs_iomap_ops, NULL,\n+\tret = iomap_dio_rw(iocb, from, ntfs_iomap_next, NULL,\n \t\t\t IOMAP_DIO_FORCE_WAIT, NULL, 0);\n \n \tif (ret == -ENOTBLK) {\n@@ -1316,7 +1316,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\tvbo = iocb-\u003eki_pos;\n \n \t\tiocb-\u003eki_flags \u0026= ~IOCB_DIRECT;\n-\t\terr = iomap_file_buffered_write(iocb, from, \u0026ntfs_iomap_ops,\n+\t\terr = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,\n \t\t\t\t\t\t\u0026ntfs_iomap_folio_ops, NULL);\n \t\tif (err \u003c 0) {\n \t\t\tret = err;\n@@ -1465,7 +1465,7 @@ int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n \n \tinode_lock_shared(inode);\n \n-\terr = iomap_fiemap(inode, fieinfo, start, len, \u0026ntfs_iomap_ops);\n+\terr = iomap_fiemap(inode, fieinfo, start, len, ntfs_iomap_next);\n \n \tinode_unlock_shared(inode);\n \treturn err;\ndiff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c\nindex c43101cc064d5..edd4fa499a66f 100644\n--- a/fs/ntfs3/inode.c\n+++ b/fs/ntfs3/inode.c\n@@ -576,7 +576,7 @@ static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)\n \t\tni_allocate_da_blocks(ni);\n \t}\n \n-\treturn iomap_bmap(mapping, block, \u0026ntfs_iomap_ops);\n+\treturn iomap_bmap(mapping, block, ntfs_iomap_next);\n }\n \n static void ntfs_iomap_read_end_io(struct bio *bio)\n@@ -649,7 +649,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)\n \t\treturn err;\n \t}\n \n-\tiomap_read_folio(\u0026ntfs_iomap_ops, \u0026ctx, NULL);\n+\tiomap_read_folio(ntfs_iomap_next, \u0026ctx, NULL);\n \treturn 0;\n }\n \n@@ -673,7 +673,7 @@ static void ntfs_readahead(struct readahead_control *rac)\n \t\treturn;\n \t}\n \n-\tiomap_readahead(\u0026ntfs_iomap_ops, \u0026ctx, NULL);\n+\tiomap_readahead(ntfs_iomap_next, \u0026ctx, NULL);\n }\n \n int ntfs_set_size(struct inode *inode, u64 new_size)\n@@ -2101,10 +2101,7 @@ const struct address_space_operations ntfs_aops_cmpr = {\n \t.invalidate_folio = iomap_invalidate_folio,\n };\n \n-const struct iomap_ops ntfs_iomap_ops = {\n-\t.iomap_begin\t= ntfs_iomap_begin,\n-\t.iomap_end\t= ntfs_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(ntfs_iomap_next, ntfs_iomap_begin, ntfs_iomap_end);\n \n const struct iomap_write_ops ntfs_iomap_folio_ops = {\n \t.put_folio = ntfs_iomap_put_folio,\ndiff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h\nindex d98d7e474476c..e00dae3ce7009 100644\n--- a/fs/ntfs3/ntfs_fs.h\n+++ b/fs/ntfs3/ntfs_fs.h\n@@ -785,7 +785,8 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,\n int ntfs_link_inode(struct inode *inode, struct dentry *dentry);\n int ntfs_unlink_inode(struct inode *dir, const struct dentry *dentry);\n void ntfs_evict_inode(struct inode *inode);\n-extern const struct iomap_ops ntfs_iomap_ops;\n+int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\t struct iomap *srcmap);\n extern const struct iomap_write_ops ntfs_iomap_folio_ops;\n extern const struct inode_operations ntfs_link_inode_operations;\n extern const struct address_space_operations ntfs_aops;\ndiff --git a/fs/remap_range.c b/fs/remap_range.c\nindex 26afbbbfb10c2..eab94d42654d4 100644\n--- a/fs/remap_range.c\n+++ b/fs/remap_range.c\n@@ -277,7 +277,7 @@ int\n __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\t\t\tstruct file *file_out, loff_t pos_out,\n \t\t\t\tloff_t *len, unsigned int remap_flags,\n-\t\t\t\tconst struct iomap_ops *dax_read_ops)\n+\t\t\t\tiomap_iter_next_fn dax_read_next)\n {\n \tstruct inode *inode_in = file_inode(file_in);\n \tstruct inode *inode_out = file_inode(file_out);\n@@ -340,10 +340,10 @@ __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\tif (!IS_DAX(inode_in))\n \t\t\tret = vfs_dedupe_file_range_compare(file_in, pos_in,\n \t\t\t\t\tfile_out, pos_out, *len, \u0026is_same);\n-\t\telse if (dax_read_ops)\n+\t\telse if (dax_read_next)\n \t\t\tret = dax_dedupe_file_range_compare(inode_in, pos_in,\n \t\t\t\t\tinode_out, pos_out, *len, \u0026is_same,\n-\t\t\t\t\tdax_read_ops);\n+\t\t\t\t\tdax_read_next);\n \t\telse\n \t\t\treturn -EINVAL;\n \t\tif (ret)\ndiff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c\nindex 2a0c54256e93f..91480cb6a4d85 100644\n--- a/fs/xfs/xfs_aops.c\n+++ b/fs/xfs/xfs_aops.c\n@@ -752,7 +752,7 @@ xfs_vm_bmap(\n \t */\n \tif (xfs_is_cow_inode(ip) || XFS_IS_REALTIME_INODE(ip))\n \t\treturn 0;\n-\treturn iomap_bmap(mapping, block, \u0026xfs_read_iomap_ops);\n+\treturn iomap_bmap(mapping, block, xfs_read_iomap_next);\n }\n \n static void\n@@ -793,7 +793,7 @@ xfs_vm_read_folio(\n \tstruct iomap_read_folio_ctx\tctx = { .cur_folio = folio };\n \n \tctx.ops = xfs_get_iomap_read_ops(folio-\u003emapping);\n-\tiomap_read_folio(\u0026xfs_read_iomap_ops, \u0026ctx, NULL);\n+\tiomap_read_folio(xfs_read_iomap_next, \u0026ctx, NULL);\n \treturn 0;\n }\n \n@@ -804,7 +804,7 @@ xfs_vm_readahead(\n \tstruct iomap_read_folio_ctx\tctx = { .rac = rac };\n \n \tctx.ops = xfs_get_iomap_read_ops(rac-\u003emapping),\n-\tiomap_readahead(\u0026xfs_read_iomap_ops, \u0026ctx, NULL);\n+\tiomap_readahead(xfs_read_iomap_next, \u0026ctx, NULL);\n }\n \n static int\n@@ -850,7 +850,7 @@ xfs_vm_swap_activate(\n \tsis-\u003ebdev = xfs_inode_buftarg(ip)-\u003ebt_bdev;\n \n \treturn iomap_swapfile_activate(sis, swap_file, span,\n-\t\t\t\u0026xfs_read_iomap_ops);\n+\t\t\txfs_read_iomap_next);\n }\n \n const struct address_space_operations xfs_address_space_operations = {\ndiff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c\nindex 845a97c9b0630..877f9024d333d 100644\n--- a/fs/xfs/xfs_file.c\n+++ b/fs/xfs/xfs_file.c\n@@ -251,8 +251,6 @@ xfs_file_dio_read(\n \tstruct iov_iter\t\t*to)\n {\n \tstruct xfs_inode\t*ip = XFS_I(file_inode(iocb-\u003eki_filp));\n-\tunsigned int\t\tdio_flags = 0;\n-\tconst struct iomap_dio_ops *dio_ops = NULL;\n \tssize_t\t\t\tret;\n \n \ttrace_xfs_file_direct_read(iocb, to);\n@@ -266,11 +264,15 @@ xfs_file_dio_read(\n \tif (ret)\n \t\treturn ret;\n \tif (mapping_stable_writes(iocb-\u003eki_filp-\u003ef_mapping)) {\n-\t\tdio_ops = \u0026xfs_dio_read_bounce_ops;\n-\t\tdio_flags |= IOMAP_DIO_BOUNCE;\n+\t\tret = iomap_dio_rw(iocb, to, xfs_read_iomap_next,\n+\t\t\t\t\u0026xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,\n+\t\t\t\tNULL, 0);\n+\t} else {\n+\t\tret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);\n+\t\tif (ret == -ENOTBLK)\n+\t\t\tret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, NULL,\n+\t\t\t\t\t0, NULL, 0);\n \t}\n-\tret = iomap_dio_rw(iocb, to, \u0026xfs_read_iomap_ops, dio_ops, dio_flags,\n-\t\t\tNULL, 0);\n \txfs_iunlock(ip, XFS_IOLOCK_SHARED);\n \n \treturn ret;\n@@ -292,7 +294,7 @@ xfs_file_dax_read(\n \tret = xfs_ilock_iocb(iocb, XFS_IOLOCK_SHARED);\n \tif (ret)\n \t\treturn ret;\n-\tret = dax_iomap_rw(iocb, to, \u0026xfs_read_iomap_ops);\n+\tret = dax_iomap_rw(iocb, to, xfs_read_iomap_next);\n \txfs_iunlock(ip, XFS_IOLOCK_SHARED);\n \n \tfile_accessed(iocb-\u003eki_filp);\n@@ -742,7 +744,7 @@ xfs_file_dio_write_aligned(\n \tstruct xfs_inode\t*ip,\n \tstruct kiocb\t\t*iocb,\n \tstruct iov_iter\t\t*from,\n-\tconst struct iomap_ops\t*ops,\n+\tiomap_iter_next_fn\tnext,\n \tconst struct iomap_dio_ops *dops,\n \tstruct xfs_zone_alloc_ctx *ac)\n {\n@@ -777,7 +779,7 @@ xfs_file_dio_write_aligned(\n \tif (mapping_stable_writes(iocb-\u003eki_filp-\u003ef_mapping))\n \t\tdio_flags |= IOMAP_DIO_BOUNCE;\n \ttrace_xfs_file_direct_write(iocb, from);\n-\tret = iomap_dio_rw(iocb, from, ops, dops, dio_flags, ac, 0);\n+\tret = iomap_dio_rw(iocb, from, next, dops, dio_flags, ac, 0);\n out_unlock:\n \txfs_iunlock(ip, iolock);\n \treturn ret;\n@@ -799,7 +801,7 @@ xfs_file_dio_write_zoned(\n \tif (ret \u003c 0)\n \t\treturn ret;\n \tret = xfs_file_dio_write_aligned(ip, iocb, from,\n-\t\t\t\u0026xfs_zoned_direct_write_iomap_ops,\n+\t\t\txfs_zoned_direct_write_iomap_next,\n \t\t\t\u0026xfs_dio_zoned_write_ops, \u0026ac);\n \txfs_zoned_space_unreserve(ip-\u003ei_mount, \u0026ac);\n \treturn ret;\n@@ -824,16 +826,16 @@ xfs_file_dio_write_atomic(\n \tunsigned int\t\tiolock = XFS_IOLOCK_SHARED;\n \tssize_t\t\t\tret, ocount = iov_iter_count(from);\n \tunsigned int\t\tdio_flags = 0;\n-\tconst struct iomap_ops\t*dops;\n+\tiomap_iter_next_fn\tnext;\n \n \t/*\n \t * HW offload should be faster, so try that first if it is already\n \t * known that the write length is not too large.\n \t */\n \tif (ocount \u003e xfs_inode_buftarg(ip)-\u003ebt_awu_max)\n-\t\tdops = \u0026xfs_atomic_write_cow_iomap_ops;\n+\t\tnext = xfs_atomic_write_cow_iomap_next;\n \telse\n-\t\tdops = \u0026xfs_direct_write_iomap_ops;\n+\t\tnext = xfs_direct_write_iomap_next;\n \n retry:\n \tret = xfs_ilock_iocb_for_write(iocb, \u0026iolock);\n@@ -853,18 +855,18 @@ xfs_file_dio_write_atomic(\n \ttrace_xfs_file_direct_write(iocb, from);\n \tif (mapping_stable_writes(iocb-\u003eki_filp-\u003ef_mapping))\n \t\tdio_flags |= IOMAP_DIO_BOUNCE;\n-\tret = iomap_dio_rw(iocb, from, dops, \u0026xfs_dio_write_ops, dio_flags,\n+\tret = iomap_dio_rw(iocb, from, next, \u0026xfs_dio_write_ops, dio_flags,\n \t\t\tNULL, 0);\n \n \t/*\n-\t * The retry mechanism is based on the -\u003eiomap_begin method returning\n+\t * The retry mechanism is based on the -\u003eiomap_next method returning\n \t * -ENOPROTOOPT, which would be when the REQ_ATOMIC-based write is not\n-\t * possible. The REQ_ATOMIC-based method typically not be possible if\n+\t * possible. The REQ_ATOMIC-based method is typically not possible if\n \t * the write spans multiple extents or the disk blocks are misaligned.\n \t */\n-\tif (ret == -ENOPROTOOPT \u0026\u0026 dops == \u0026xfs_direct_write_iomap_ops) {\n+\tif (ret == -ENOPROTOOPT \u0026\u0026 next == xfs_direct_write_iomap_next) {\n \t\txfs_iunlock(ip, iolock);\n-\t\tdops = \u0026xfs_atomic_write_cow_iomap_ops;\n+\t\tnext = xfs_atomic_write_cow_iomap_next;\n \t\tgoto retry;\n \t}\n \n@@ -947,7 +949,7 @@ xfs_file_dio_write_unaligned(\n \t\tflags |= IOMAP_DIO_BOUNCE;\n \n \ttrace_xfs_file_direct_write(iocb, from);\n-\tret = iomap_dio_rw(iocb, from, \u0026xfs_direct_write_iomap_ops,\n+\tret = iomap_dio_rw(iocb, from, xfs_direct_write_iomap_next,\n \t\t\t \u0026xfs_dio_write_ops, flags, NULL, 0);\n \n \t/*\n@@ -987,7 +989,7 @@ xfs_file_dio_write(\n \tif (iocb-\u003eki_flags \u0026 IOCB_ATOMIC)\n \t\treturn xfs_file_dio_write_atomic(ip, iocb, from);\n \treturn xfs_file_dio_write_aligned(ip, iocb, from,\n-\t\t\t\u0026xfs_direct_write_iomap_ops, \u0026xfs_dio_write_ops, NULL);\n+\t\t\txfs_direct_write_iomap_next, \u0026xfs_dio_write_ops, NULL);\n }\n \n static noinline ssize_t\n@@ -1011,7 +1013,7 @@ xfs_file_dax_write(\n \tpos = iocb-\u003eki_pos;\n \n \ttrace_xfs_file_dax_write(iocb, from);\n-\tret = dax_iomap_rw(iocb, from, \u0026xfs_dax_write_iomap_ops);\n+\tret = dax_iomap_rw(iocb, from, xfs_dax_write_iomap_next);\n \tif (ret \u003e 0 \u0026\u0026 iocb-\u003eki_pos \u003e i_size_read(inode)) {\n \t\ti_size_write(inode, iocb-\u003eki_pos);\n \t\terror = xfs_setfilesize(ip, pos, ret);\n@@ -1054,7 +1056,7 @@ xfs_file_buffered_write(\n \n \ttrace_xfs_file_buffered_write(iocb, from);\n \tret = iomap_file_buffered_write(iocb, from,\n-\t\t\t\u0026xfs_buffered_write_iomap_ops, \u0026xfs_iomap_write_ops,\n+\t\t\txfs_buffered_write_iomap_next, \u0026xfs_iomap_write_ops,\n \t\t\tNULL);\n \n \t/*\n@@ -1135,7 +1137,7 @@ xfs_file_buffered_write_zoned(\n retry:\n \ttrace_xfs_file_buffered_write(iocb, from);\n \tret = iomap_file_buffered_write(iocb, from,\n-\t\t\t\u0026xfs_buffered_write_iomap_ops, \u0026xfs_iomap_write_ops,\n+\t\t\txfs_buffered_write_iomap_next, \u0026xfs_iomap_write_ops,\n \t\t\t\u0026ac);\n \tif (ret == -ENOSPC \u0026\u0026 !cleared_space) {\n \t\t/*\n@@ -1856,10 +1858,10 @@ xfs_file_llseek(\n \tdefault:\n \t\treturn generic_file_llseek(file, offset, whence);\n \tcase SEEK_HOLE:\n-\t\toffset = iomap_seek_hole(inode, offset, \u0026xfs_seek_iomap_ops);\n+\t\toffset = iomap_seek_hole(inode, offset, xfs_seek_iomap_next);\n \t\tbreak;\n \tcase SEEK_DATA:\n-\t\toffset = iomap_seek_data(inode, offset, \u0026xfs_seek_iomap_ops);\n+\t\toffset = iomap_seek_data(inode, offset, xfs_seek_iomap_next);\n \t\tbreak;\n \t}\n \n@@ -1883,8 +1885,8 @@ xfs_dax_fault_locked(\n \t}\n \tret = dax_iomap_fault(vmf, order, \u0026pfn, NULL,\n \t\t\t(write_fault \u0026\u0026 !vmf-\u003ecow_page) ?\n-\t\t\t\t\u0026xfs_dax_write_iomap_ops :\n-\t\t\t\t\u0026xfs_read_iomap_ops);\n+\t\t\t\txfs_dax_write_iomap_next :\n+\t\t\t\txfs_read_iomap_next);\n \tif (ret \u0026 VM_FAULT_NEEDDSYNC)\n \t\tret = dax_finish_sync_fault(vmf, order, pfn);\n \treturn ret;\n@@ -1948,7 +1950,7 @@ __xfs_write_fault(\n \tif (IS_DAX(inode))\n \t\tret = xfs_dax_fault_locked(vmf, order, true);\n \telse\n-\t\tret = iomap_page_mkwrite(vmf, \u0026xfs_buffered_write_iomap_ops,\n+\t\tret = iomap_page_mkwrite(vmf, xfs_buffered_write_iomap_next,\n \t\t\t\tac);\n \txfs_iunlock(ip, lock_mode);\n \ndiff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c\nindex 225c3de88d03b..771580e01c0a5 100644\n--- a/fs/xfs/xfs_iomap.c\n+++ b/fs/xfs/xfs_iomap.c\n@@ -1037,9 +1037,8 @@ xfs_direct_write_iomap_begin(\n \treturn error;\n }\n \n-const struct iomap_ops xfs_direct_write_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_direct_write_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_direct_write_iomap_next,\n+\t\txfs_direct_write_iomap_begin);\n \n #ifdef CONFIG_XFS_RT\n /*\n@@ -1089,9 +1088,9 @@ xfs_zoned_direct_write_iomap_begin(\n \treturn 0;\n }\n \n-const struct iomap_ops xfs_zoned_direct_write_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_zoned_direct_write_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_zoned_direct_write_iomap_next,\n+\t\txfs_zoned_direct_write_iomap_begin);\n+\n #endif /* CONFIG_XFS_RT */\n \n #ifdef DEBUG\n@@ -1274,9 +1273,8 @@ xfs_atomic_write_cow_iomap_begin(\n \treturn error;\n }\n \n-const struct iomap_ops xfs_atomic_write_cow_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_atomic_write_cow_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_atomic_write_cow_iomap_next,\n+\t\txfs_atomic_write_cow_iomap_begin);\n \n static int\n xfs_dax_write_iomap_end(\n@@ -1298,10 +1296,8 @@ xfs_dax_write_iomap_end(\n \treturn xfs_reflink_end_cow(ip, pos, written);\n }\n \n-const struct iomap_ops xfs_dax_write_iomap_ops = {\n-\t.iomap_begin\t= xfs_direct_write_iomap_begin,\n-\t.iomap_end\t= xfs_dax_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(xfs_dax_write_iomap_next,\n+\t\txfs_direct_write_iomap_begin, xfs_dax_write_iomap_end);\n \n /*\n * Convert a hole to a delayed allocation.\n@@ -2168,12 +2164,10 @@ xfs_buffered_write_iomap_end(\n \treturn 0;\n }\n \n-const struct iomap_ops xfs_buffered_write_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_buffered_write_iomap_begin,\n-\t.iomap_end\t\t= xfs_buffered_write_iomap_end,\n-};\n+DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,\n+\t\txfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);\n \n-static int\n+int\n xfs_read_iomap_begin(\n \tstruct inode\t\t*inode,\n \tloff_t\t\t\toffset,\n@@ -2214,9 +2208,7 @@ xfs_read_iomap_begin(\n \t\t\t\t shared ? IOMAP_F_SHARED : 0, seq);\n }\n \n-const struct iomap_ops xfs_read_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_read_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_read_iomap_next, xfs_read_iomap_begin);\n \n static int\n xfs_seek_iomap_begin(\n@@ -2302,9 +2294,7 @@ xfs_seek_iomap_begin(\n \treturn error;\n }\n \n-const struct iomap_ops xfs_seek_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_seek_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_seek_iomap_next, xfs_seek_iomap_begin);\n \n static int\n xfs_xattr_iomap_begin(\n@@ -2349,9 +2339,7 @@ xfs_xattr_iomap_begin(\n \treturn xfs_bmbt_to_iomap(ip, iomap, \u0026imap, flags, IOMAP_F_XATTR, seq);\n }\n \n-const struct iomap_ops xfs_xattr_iomap_ops = {\n-\t.iomap_begin\t\t= xfs_xattr_iomap_begin,\n-};\n+DEFINE_IOMAP_ITER_NEXT(xfs_xattr_iomap_next, xfs_xattr_iomap_begin);\n \n int\n xfs_zero_range(\n@@ -2367,9 +2355,9 @@ xfs_zero_range(\n \n \tif (IS_DAX(inode))\n \t\treturn dax_zero_range(inode, pos, len, did_zero,\n-\t\t\t\t \u0026xfs_dax_write_iomap_ops);\n+\t\t\t\t xfs_dax_write_iomap_next);\n \treturn iomap_zero_range(inode, pos, len, did_zero,\n-\t\t\t\u0026xfs_buffered_write_iomap_ops, \u0026xfs_iomap_write_ops,\n+\t\t\txfs_buffered_write_iomap_next, \u0026xfs_iomap_write_ops,\n \t\t\tac);\n }\n \n@@ -2384,8 +2372,8 @@ xfs_truncate_page(\n \n \tif (IS_DAX(inode))\n \t\treturn dax_truncate_page(inode, pos, did_zero,\n-\t\t\t\t\t\u0026xfs_dax_write_iomap_ops);\n+\t\t\t\t\txfs_dax_write_iomap_next);\n \treturn iomap_truncate_page(inode, pos, did_zero,\n-\t\t\t\u0026xfs_buffered_write_iomap_ops, \u0026xfs_iomap_write_ops,\n+\t\t\txfs_buffered_write_iomap_next, \u0026xfs_iomap_write_ops,\n \t\t\tac);\n }\ndiff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h\nindex ebcce7d494468..d2d3c84d93f9e 100644\n--- a/fs/xfs/xfs_iomap.h\n+++ b/fs/xfs/xfs_iomap.h\n@@ -49,14 +49,26 @@ xfs_aligned_fsb_count(\n \treturn count_fsb;\n }\n \n-extern const struct iomap_ops xfs_buffered_write_iomap_ops;\n-extern const struct iomap_ops xfs_direct_write_iomap_ops;\n-extern const struct iomap_ops xfs_zoned_direct_write_iomap_ops;\n-extern const struct iomap_ops xfs_read_iomap_ops;\n-extern const struct iomap_ops xfs_seek_iomap_ops;\n-extern const struct iomap_ops xfs_xattr_iomap_ops;\n-extern const struct iomap_ops xfs_dax_write_iomap_ops;\n-extern const struct iomap_ops xfs_atomic_write_cow_iomap_ops;\n+int xfs_read_iomap_begin(struct inode *inode, loff_t offset,\n+\t\tloff_t length, unsigned flags, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+\n+int xfs_buffered_write_iomap_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n+int xfs_direct_write_iomap_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n+int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n+int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int xfs_xattr_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int xfs_dax_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n+int xfs_atomic_write_cow_iomap_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n extern const struct iomap_write_ops xfs_iomap_write_ops;\n \n #endif /* __XFS_IOMAP_H__*/\ndiff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c\nindex 6339f4956ecb3..5c3d9a365f935 100644\n--- a/fs/xfs/xfs_iops.c\n+++ b/fs/xfs/xfs_iops.c\n@@ -1239,10 +1239,10 @@ xfs_vn_fiemap(\n \tif (fieinfo-\u003efi_flags \u0026 FIEMAP_FLAG_XATTR) {\n \t\tfieinfo-\u003efi_flags \u0026= ~FIEMAP_FLAG_XATTR;\n \t\terror = iomap_fiemap(inode, fieinfo, start, length,\n-\t\t\t\t\u0026xfs_xattr_iomap_ops);\n+\t\t\t\txfs_xattr_iomap_next);\n \t} else {\n \t\terror = iomap_fiemap(inode, fieinfo, start, length,\n-\t\t\t\t\u0026xfs_read_iomap_ops);\n+\t\t\t\txfs_read_iomap_next);\n \t}\n \txfs_iunlock(XFS_I(inode), XFS_IOLOCK_SHARED);\n \ndiff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c\nindex a5c188b781382..2b9792626bab5 100644\n--- a/fs/xfs/xfs_reflink.c\n+++ b/fs/xfs/xfs_reflink.c\n@@ -1683,7 +1683,7 @@ xfs_reflink_remap_prep(\n \t\t\t\tpos_out, len, remap_flags);\n \telse\n \t\tret = dax_remap_file_range_prep(file_in, pos_in, file_out,\n-\t\t\t\tpos_out, len, remap_flags, \u0026xfs_read_iomap_ops);\n+\t\t\t\tpos_out, len, remap_flags, xfs_read_iomap_next);\n \tif (ret || *len == 0)\n \t\tgoto out_unlock;\n \n@@ -1878,10 +1878,10 @@ xfs_reflink_unshare(\n \n \tif (IS_DAX(inode))\n \t\terror = dax_file_unshare(inode, offset, len,\n-\t\t\t\t\u0026xfs_dax_write_iomap_ops);\n+\t\t\t\txfs_dax_write_iomap_next);\n \telse\n \t\terror = iomap_file_unshare(inode, offset, len,\n-\t\t\t\t\u0026xfs_buffered_write_iomap_ops,\n+\t\t\t\txfs_buffered_write_iomap_next,\n \t\t\t\t\u0026xfs_iomap_write_ops);\n \tif (error)\n \t\tgoto out;\ndiff --git a/fs/zonefs/file.c b/fs/zonefs/file.c\nindex 5ada33f70bb47..319d9502ba565 100644\n--- a/fs/zonefs/file.c\n+++ b/fs/zonefs/file.c\n@@ -57,9 +57,7 @@ static int zonefs_read_iomap_begin(struct inode *inode, loff_t offset,\n \treturn 0;\n }\n \n-static const struct iomap_ops zonefs_read_iomap_ops = {\n-\t.iomap_begin\t= zonefs_read_iomap_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(zonefs_read_iomap_next, zonefs_read_iomap_begin);\n \n static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,\n \t\t\t\t loff_t length, unsigned int flags,\n@@ -106,19 +104,18 @@ static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,\n \treturn 0;\n }\n \n-static const struct iomap_ops zonefs_write_iomap_ops = {\n-\t.iomap_begin\t= zonefs_write_iomap_begin,\n-};\n+static DEFINE_IOMAP_ITER_NEXT(zonefs_write_iomap_next,\n+\t\t\t zonefs_write_iomap_begin);\n \n static int zonefs_read_folio(struct file *unused, struct folio *folio)\n {\n-\tiomap_bio_read_folio(folio, \u0026zonefs_read_iomap_ops);\n+\tiomap_bio_read_folio(folio, zonefs_read_iomap_next);\n \treturn 0;\n }\n \n static void zonefs_readahead(struct readahead_control *rac)\n {\n-\tiomap_bio_readahead(rac, \u0026zonefs_read_iomap_ops);\n+\tiomap_bio_readahead(rac, zonefs_read_iomap_next);\n }\n \n /*\n@@ -179,7 +176,7 @@ static int zonefs_swap_activate(struct swap_info_struct *sis,\n \t}\n \n \treturn iomap_swapfile_activate(sis, swap_file, span,\n-\t\t\t\t \u0026zonefs_read_iomap_ops);\n+\t\t\t\t zonefs_read_iomap_next);\n }\n \n const struct address_space_operations zonefs_file_aops = {\n@@ -309,7 +306,7 @@ static vm_fault_t zonefs_filemap_page_mkwrite(struct vm_fault *vmf)\n \n \t/* Serialize against truncates */\n \tfilemap_invalidate_lock_shared(inode-\u003ei_mapping);\n-\tret = iomap_page_mkwrite(vmf, \u0026zonefs_write_iomap_ops, NULL);\n+\tret = iomap_page_mkwrite(vmf, zonefs_write_iomap_next, NULL);\n \tfilemap_invalidate_unlock_shared(inode-\u003ei_mapping);\n \n \tsb_end_pagefault(inode-\u003ei_sb);\n@@ -525,7 +522,7 @@ static ssize_t zonefs_file_dio_write(struct kiocb *iocb, struct iov_iter *from)\n \t * page invalidation. Overwrite that error code with EBUSY so that\n \t * the user can make sense of the error.\n \t */\n-\tret = iomap_dio_rw(iocb, from, \u0026zonefs_write_iomap_ops,\n+\tret = iomap_dio_rw(iocb, from, zonefs_write_iomap_next,\n \t\t\t \u0026zonefs_write_dio_ops, 0, NULL, 0);\n \tif (ret == -ENOTBLK)\n \t\tret = -EBUSY;\n@@ -575,7 +572,7 @@ static ssize_t zonefs_file_buffered_write(struct kiocb *iocb,\n \tif (ret \u003c= 0)\n \t\tgoto inode_unlock;\n \n-\tret = iomap_file_buffered_write(iocb, from, \u0026zonefs_write_iomap_ops,\n+\tret = iomap_file_buffered_write(iocb, from, zonefs_write_iomap_next,\n \t\t\tNULL, NULL);\n \tif (ret == -EIO)\n \t\tzonefs_io_error(inode, true);\n@@ -670,7 +667,7 @@ static ssize_t zonefs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\t\tgoto inode_unlock;\n \t\t}\n \t\tfile_accessed(iocb-\u003eki_filp);\n-\t\tret = iomap_dio_rw(iocb, to, \u0026zonefs_read_iomap_ops,\n+\t\tret = iomap_dio_rw(iocb, to, zonefs_read_iomap_next,\n \t\t\t\t \u0026zonefs_read_dio_ops, 0, NULL, 0);\n \t} else {\n \t\tret = generic_file_read_iter(iocb, to);\ndiff --git a/include/linux/dax.h b/include/linux/dax.h\nindex fe6c3ded1b50f..e0bc653ffe9ff 100644\n--- a/include/linux/dax.h\n+++ b/include/linux/dax.h\n@@ -3,6 +3,7 @@\n #define _LINUX_DAX_H\n \n #include \u003clinux/fs.h\u003e\n+#include \u003clinux/iomap.h\u003e\n #include \u003clinux/mm.h\u003e\n #include \u003clinux/radix-tree.h\u003e\n \n@@ -10,9 +11,6 @@ typedef unsigned long dax_entry_t;\n \n struct dax_device;\n struct gendisk;\n-struct iomap_ops;\n-struct iomap_iter;\n-struct iomap;\n \n enum dax_access_mode {\n \tDAX_ACCESS,\n@@ -213,11 +211,11 @@ static inline void dax_unlock_mapping_entry(struct address_space *mapping,\n #endif\n \n int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n \n static inline bool dax_page_is_idle(struct page *page)\n {\n@@ -266,10 +264,10 @@ int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, u64 len,\n void dax_flush(struct dax_device *dax_dev, void *addr, size_t size);\n \n ssize_t dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,\n \t\t\tunsigned long *pfnp, int *errp,\n-\t\t\tconst struct iomap_ops *ops);\n+\t\t\tiomap_iter_next_fn next);\n vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf,\n \t\tunsigned int order, unsigned long pfn);\n int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index);\n@@ -288,11 +286,11 @@ void dax_break_layout_final(struct inode *inode);\n int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,\n \t\t\t\t struct inode *dest, loff_t destoff,\n \t\t\t\t loff_t len, bool *is_same,\n-\t\t\t\t const struct iomap_ops *ops);\n+\t\t\t\t iomap_iter_next_fn next);\n int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\t\t struct file *file_out, loff_t pos_out,\n \t\t\t loff_t *len, unsigned int remap_flags,\n-\t\t\t const struct iomap_ops *ops);\n+\t\t\t iomap_iter_next_fn next);\n static inline bool dax_mapping(struct address_space *mapping)\n {\n \treturn mapping-\u003ehost \u0026\u0026 IS_DAX(mapping-\u003ehost);\ndiff --git a/include/linux/fs.h b/include/linux/fs.h\nindex d10897b3a1e35..2eb063438a3bb 100644\n--- a/include/linux/fs.h\n+++ b/include/linux/fs.h\n@@ -70,7 +70,8 @@ struct fsnotify_mark_connector;\n struct fs_context;\n struct fs_parameter_spec;\n struct file_kattr;\n-struct iomap_ops;\n+struct iomap_iter;\n+struct iomap;\n struct delegated_inode;\n \n extern void __init inode_init(void);\n@@ -2079,7 +2080,9 @@ int remap_verify_area(struct file *file, loff_t pos, loff_t len, bool write);\n int __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\t\t\t struct file *file_out, loff_t pos_out,\n \t\t\t\t loff_t *len, unsigned int remap_flags,\n-\t\t\t\t const struct iomap_ops *dax_read_ops);\n+\t\t\t\t int (*dax_read_next)(const struct iomap_iter *iter,\n+\t\t\t\t\t\t\t struct iomap *iomap,\n+\t\t\t\t\t\t\t struct iomap *srcmap));\n int generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,\n \t\t\t\t struct file *file_out, loff_t pos_out,\n \t\t\t\t loff_t *count, unsigned int remap_flags);\ndiff --git a/include/linux/iomap.h b/include/linux/iomap.h\nindex 21e73cb9c51e3..1875e21ce1646 100644\n--- a/include/linux/iomap.h\n+++ b/include/linux/iomap.h\n@@ -10,6 +10,7 @@\n #include \u003clinux/mm_types.h\u003e\n #include \u003clinux/blkdev.h\u003e\n #include \u003clinux/folio_batch.h\u003e\n+#include \u003clinux/pagemap.h\u003e\n \n struct address_space;\n struct fiemap_extent_info;\n@@ -194,7 +195,7 @@ struct iomap_write_ops {\n };\n \n /*\n- * Flags for iomap_begin / iomap_end. No flag implies a read.\n+ * Flags for iomap_next. No flag implies a read.\n */\n #define IOMAP_WRITE\t\t(1 \u003c\u003c 0) /* writing, must allocate blocks */\n #define IOMAP_ZERO\t\t(1 \u003c\u003c 1) /* zeroing operation, may skip holes */\n@@ -212,25 +213,31 @@ struct iomap_write_ops {\n #define IOMAP_ATOMIC\t\t(1 \u003c\u003c 9) /* torn-write protection */\n #define IOMAP_DONTCACHE\t\t(1 \u003c\u003c 10)\n \n-struct iomap_ops {\n-\t/*\n-\t * Return the existing mapping at pos, or reserve space starting at\n-\t * pos for up to length, as long as we can do it as a single mapping.\n-\t * The actual length is returned in iomap-\u003elength.\n-\t */\n-\tint (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,\n-\t\t\tunsigned flags, struct iomap *iomap,\n-\t\t\tstruct iomap *srcmap);\n+/*\n+ * Return the existing mapping at pos, or reserve space starting at pos for up\n+ * to length, as long as we can do it as a single mapping.\n+ * The actual length is returned in iomap-\u003elength.\n+ */\n+typedef int (*iomap_iter_begin_fn)(struct inode *inode, loff_t pos,\n+\t\tloff_t length, unsigned flags, struct iomap *iomap,\n+\t\tstruct iomap *srcmap);\n \n-\t/*\n-\t * Commit and/or unreserve space previous allocated using iomap_begin.\n-\t * Written indicates the length of the successful write operation which\n-\t * needs to be commited, while the rest needs to be unreserved.\n-\t * Written might be zero if no data was written.\n-\t */\n-\tint (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,\n-\t\t\tssize_t written, unsigned flags, struct iomap *iomap);\n-};\n+/*\n+ * Commit and/or unreserve space previously allocated by iomap_iter_begin_fn.\n+ * Written indicates the length of the successful write operation which needs\n+ * to be committed, while the rest needs to be unreserved.\n+ * Written might be zero if no data was written.\n+ */\n+typedef int (*iomap_iter_end_fn)(struct inode *inode, loff_t pos, loff_t length,\n+\t\tssize_t written, unsigned flags, struct iomap *iomap);\n+\n+/*\n+ * Produce the next mapping (finishing the previous one if needed).\n+ * Return 1 to continue iterating, 0 if the range is fully consumed, or a\n+ * negative error on failure.\n+ */\n+typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap);\n \n /**\n * struct iomap_iter - Iterate through a range of a file\n@@ -243,7 +250,7 @@ struct iomap_ops {\n *\tincremental iter advance.\n * @status: Status of the most recent iteration. Zero on success or a negative\n *\terrno on error.\n- * @flags: Zero or more of the iomap_begin flags above.\n+ * @flags: Zero or more of the iomap_iter_next flags above.\n * @iomap: Map describing the I/O iteration\n * @srcmap: Source map for COW operations\n */\n@@ -260,7 +267,7 @@ struct iomap_iter {\n \tvoid *private;\n };\n \n-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops);\n+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next);\n int iomap_iter_advance(struct iomap_iter *iter, u64 count);\n \n /**\n@@ -317,6 +324,71 @@ static inline const struct iomap *iomap_iter_srcmap(const struct iomap_iter *i)\n \treturn \u0026i-\u003eiomap;\n }\n \n+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,\n+\t\tstruct iomap *srcmap, int ret);\n+\n+/**\n+ * iomap_iter_next - finish the previous mapping and produce the next one\n+ * @iter: iteration structure\n+ * @iomap: mapping to finish and then repopulate\n+ * @srcmap: source mapping to finish and then repopulate\n+ * @begin: callback that produces a mapping for the current position\n+ * @end: optional callback that finishes the previous mapping, or NULL\n+ *\n+ * Inline helper that implements the common body of an iomap_iter_next_fn\n+ * callback: it finishes the previous mapping via @end (if present), decides\n+ * via iomap_iter_continue() whether to keep going, and obtains the next\n+ * mapping via @begin.\n+ *\n+ * This helper is marked __always_inline so that when a caller passes\n+ * compile-time-constant @begin and @end callbacks, the compiler can call them\n+ * directly, avoiding the indirect-call overhead.\n+ *\n+ * Returns 1 to continue iterating, 0 once the range is fully consumed, or a\n+ * negative errno on error.\n+ */\n+static __always_inline int iomap_iter_next(const struct iomap_iter *iter,\n+\t\tstruct iomap *iomap, struct iomap *srcmap,\n+\t\tiomap_iter_begin_fn begin, iomap_iter_end_fn end)\n+{\n+\tint ret = 0;\n+\n+\tif (iomap-\u003elength) {\n+\t\tif (end) {\n+\t\t\t/*\n+\t\t\t * Calculate how far the iter was advanced and the\n+\t\t\t * original length bytes for end().\n+\t\t\t */\n+\t\t\tssize_t advanced = iter-\u003epos - iter-\u003eiter_start_pos;\n+\t\t\tloff_t len;\n+\n+\t\t\tlen = iomap_length_trim(iter, iter-\u003eiter_start_pos,\n+\t\t\t\t\titer-\u003elen + advanced);\n+\n+\t\t\tret = end(iter-\u003einode, iter-\u003eiter_start_pos, len,\n+\t\t\t\t\tadvanced, iter-\u003eflags, iomap);\n+\t\t}\n+\t\tret = iomap_iter_continue(iter, iomap, srcmap, ret);\n+\t\tif (ret \u003c= 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\tret = begin(iter-\u003einode, iter-\u003epos, iter-\u003elen, iter-\u003eflags, iomap,\n+\t\t\tsrcmap);\n+\n+\treturn ret \u003c 0 ? ret : 1;\n+}\n+\n+#define DEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, end_fn)\t\t\\\n+int name(const struct iomap_iter *iter, struct iomap *iomap,\t\t\\\n+\t\tstruct iomap *srcmap)\t\t\t\t\t\\\n+{\t\t\t\t\t\t\t\t\t\\\n+\treturn iomap_iter_next(iter, iomap, srcmap, begin_fn, end_fn);\t\\\n+}\n+\n+#define DEFINE_IOMAP_ITER_NEXT(name, begin_fn)\t\t\t\t\\\n+\tDEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, NULL)\n+\n /*\n * Return the file offset for the first unchanged block after a short write.\n *\n@@ -351,15 +423,15 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter)\n }\n \n ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private);\n int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,\n-\t\tconst void *buf, const struct iomap_ops *ops,\n+\t\tconst void *buf, iomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops);\n-void iomap_read_folio(const struct iomap_ops *ops,\n-\t\tstruct iomap_read_folio_ctx *ctx, void *private);\n-void iomap_readahead(const struct iomap_ops *ops,\n-\t\tstruct iomap_read_folio_ctx *ctx, void *private);\n+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,\n+\t\tvoid *private);\n+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,\n+\t\tvoid *private);\n bool iomap_is_partially_uptodate(struct folio *, size_t from, size_t count);\n struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len);\n bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);\n@@ -367,17 +439,17 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);\n bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);\n void iomap_folio_mark_uptodate(struct folio *folio);\n int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops);\n unsigned int iomap_fill_dirty_folios(struct iomap_iter *iter, loff_t *start,\n \t\tloff_t end, unsigned int *iomap_flags);\n int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len,\n-\t\tbool *did_zero, const struct iomap_ops *ops,\n+\t\tbool *did_zero, iomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private);\n int iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,\n-\t\tconst struct iomap_ops *ops,\n+\t\tiomap_iter_next_fn next,\n \t\tconst struct iomap_write_ops *write_ops, void *private);\n-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,\n+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,\n \t\tvoid *private);\n typedef void (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length,\n \t\tstruct iomap *iomap);\n@@ -386,13 +458,13 @@ void iomap_write_delalloc_release(struct inode *inode, loff_t start_byte,\n \t\tiomap_punch_t punch);\n \n int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,\n-\t\tu64 start, u64 len, const struct iomap_ops *ops);\n+\t\tu64 start, u64 len, iomap_iter_next_fn next);\n loff_t iomap_seek_hole(struct inode *inode, loff_t offset,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n loff_t iomap_seek_data(struct inode *inode, loff_t offset,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n sector_t iomap_bmap(struct address_space *mapping, sector_t bno,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n \n /*\n * Flags for iomap_ioend-\u003eio_flags.\n@@ -599,23 +671,88 @@ struct iomap_dio_ops {\n #define IOMAP_DIO_BOUNCE\t\t(1 \u003c\u003c 4)\n \n ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops, const struct iomap_dio_ops *dops,\n+\t\tiomap_iter_next_fn next, const struct iomap_dio_ops *dops,\n \t\tunsigned int dio_flags, void *private, size_t done_before);\n struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,\n-\t\tconst struct iomap_ops *ops, const struct iomap_dio_ops *dops,\n+\t\tiomap_iter_next_fn next, const struct iomap_dio_ops *dops,\n \t\tunsigned int dio_flags, void *private, size_t done_before);\n ssize_t iomap_dio_complete(struct iomap_dio *dio);\n void iomap_dio_bio_end_io(struct bio *bio);\n \n+/*\n+ * Fast path for small, block-aligned direct I/Os that map to a single\n+ * contiguous on-disk extent.\n+ *\n+ * @iter must describe a non-empty READ no larger than the inode block size:\n+ * writes, zero-length I/O, and larger requests need the generic iomap direct\n+ * I/O path.\n+ *\n+ * Does not support iomap_dio_ops, dio_flags, done_before or private data.\n+ * The range must also stay within i_size and encrypted inodes must use the\n+ * generic iomap direct I/O path.\n+ *\n+ * -ENOTBLK indicates the generic path must be used by the caller instead.\n+ * Any other errno is a real result and is propagated as-is, in particular\n+ * -EAGAIN for IOCB_NOWAIT must reach the caller.\n+ *\n+ * The caller can only provide an iomap begin handler, and the iterator\n+ * is never advanced.\n+ */\n+ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,\n+\t\tstruct iomap_iter *iomi);\n+static __always_inline ssize_t iomap_dio_read_simple(struct kiocb *iocb,\n+\t\tstruct iov_iter *iter, iomap_iter_begin_fn begin)\n+{\n+\tstruct iomap_iter iomi = {\n+\t\t.inode\t\t= file_inode(iocb-\u003eki_filp),\n+\t\t.pos\t\t= iocb-\u003eki_pos,\n+\t\t.len\t\t= iov_iter_count(iter),\n+\t\t.flags\t\t= IOMAP_DIRECT,\n+\t};\n+\tssize_t ret;\n+\n+\tif (!iomi.len)\n+\t\treturn 0;\n+\n+\t/*\n+\t * Simple dio is an optimization for small IO. Filter out large IO\n+\t * early as it's the most common case to fail for typical direct IO\n+\t * workloads.\n+\t */\n+\tif (iomi.len \u003e iomi.inode-\u003ei_sb-\u003es_blocksize)\n+\t\treturn -ENOTBLK;\n+\tif (iocb-\u003eki_pos + iomi.len \u003e i_size_read(iomi.inode))\n+\t\treturn -ENOTBLK;\n+\tif (IS_ENCRYPTED(iomi.inode))\n+\t\treturn -ENOTBLK;\n+\n+\tret = kiocb_write_and_wait(iocb, iomi.len);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tif (iocb-\u003eki_flags \u0026 IOCB_NOWAIT)\n+\t\tiomi.flags |= IOMAP_NOWAIT;\n+\n+\tinode_dio_begin(iomi.inode);\n+\tret = begin(iomi.inode, iomi.pos, iomi.len, iomi.flags, \u0026iomi.iomap,\n+\t\t\t\u0026iomi.srcmap);\n+\tif (ret) {\n+\t\tinode_dio_end(iomi.inode);\n+\t\treturn ret;\n+\t}\n+\n+\treturn __iomap_dio_read_simple(iocb, iter, \u0026iomi);\n+}\n+\n #ifdef CONFIG_SWAP\n struct file;\n struct swap_info_struct;\n \n int iomap_swapfile_activate(struct swap_info_struct *sis,\n \t\tstruct file *swap_file, sector_t *pagespan,\n-\t\tconst struct iomap_ops *ops);\n+\t\tiomap_iter_next_fn next);\n #else\n-# define iomap_swapfile_activate(sis, swapfile, pagespan, ops)\t(-EIO)\n+# define iomap_swapfile_activate(sis, swapfile, pagespan, next) (-EIO)\n #endif /* CONFIG_SWAP */\n \n extern struct bio_set iomap_ioend_bioset;\n@@ -627,25 +764,25 @@ int iomap_bio_read_folio_range(const struct iomap_iter *iter,\n extern const struct iomap_read_ops iomap_bio_read_ops;\n \n static inline void iomap_bio_read_folio(struct folio *folio,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_read_folio_ctx ctx = {\n \t\t.ops\t\t= \u0026iomap_bio_read_ops,\n \t\t.cur_folio\t= folio,\n \t};\n \n-\tiomap_read_folio(ops, \u0026ctx, NULL);\n+\tiomap_read_folio(next, \u0026ctx, NULL);\n }\n \n static inline void iomap_bio_readahead(struct readahead_control *rac,\n-\t\tconst struct iomap_ops *ops)\n+\t\tiomap_iter_next_fn next)\n {\n \tstruct iomap_read_folio_ctx ctx = {\n \t\t.ops\t\t= \u0026iomap_bio_read_ops,\n \t\t.rac\t\t= rac,\n \t};\n \n-\tiomap_readahead(ops, \u0026ctx, NULL);\n+\tiomap_readahead(next, \u0026ctx, NULL);\n }\n #endif /* CONFIG_BLOCK */\n \n"
} |
| 2/1 | 2026/07/28 19:02 | agent | patch-evaluator |
0mModel:best-expensive Args: null Results: {
"EnableConfigs": [
"EXT4_FS",
"XFS_FS",
"BTRFS_FS",
"F2FS_FS",
"FS_DAX"
],
"FocusSymbols": [
"iomap_iter",
"iomap_iter_continue",
"__iomap_dio_read_simple",
"iomap_dio_rw",
"iomap_file_buffered_write",
"iomap_read_folio",
"iomap_readahead",
"iomap_fiemap",
"iomap_seek_hole",
"iomap_seek_data",
"dax_iomap_rw",
"dax_iomap_fault"
],
"Reasoning": "The patch refactors the iomap subsystem to use a single `iomap_iter_next_fn` callback instead of a `struct iomap_ops` with `iomap_begin` and `iomap_end` functions. This changes the control flow and calling convention for all iomap operations across many filesystems (ext4, xfs, btrfs, f2fs, etc.). It also introduces `__iomap_dio_read_simple` and `iomap_iter_continue`. These are core functional changes to the I/O path that should be fuzzed to ensure no regressions or bugs were introduced in the iteration logic.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 5bd95dfd710ae16d6150e042e0a6fd5c6d20fdac
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Jul 28 19:02:00 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/filesystems/iomap/design.rst b/Documentation/filesystems/iomap/design.rst
index 0f7672676c0ba..bed8a5beac819 100644
--- a/Documentation/filesystems/iomap/design.rst
+++ b/Documentation/filesystems/iomap/design.rst
@@ -75,7 +75,10 @@ At a high level, an iomap operation `looks like this
1. For each byte in the operation range...
- 1. Obtain a space mapping via ``->iomap_begin``
+ 1. Obtain the next space mapping via the ``iomap_iter_next_fn`` callback.
+ From the second iteration onwards this same callback first finishes
+ the previous mapping (committing or unreserving space as needed)
+ and then produces the next one.
2. For each sub-unit of work...
@@ -86,7 +89,13 @@ At a high level, an iomap operation `looks like this
3. Increment operation cursor
- 4. Release the mapping via ``->iomap_end``, if necessary
+iomap repeats this until the range is fully consumed. The ``iomap_iter_next_fn``
+callback returns ``1`` while there is more of the range left to process,
+``0`` once it is fully consumed, or a negative errno on error.
+Filesystems rarely implement this callback by hand. The ``iomap_iter_next``
+helper implements the finish-then-produce sequence in terms of two smaller
+callbacks, ``begin`` and ``end``. See `The Mapping Callback`_ below for more
+info.
Each iomap operation will be covered in more detail below.
This library was covered previously by an `LWN article
@@ -189,7 +198,7 @@ The fields are as follows:
* **IOMAP_DELALLOC**: A promise to allocate space at a later time
("delayed allocation").
If the filesystem returns IOMAP_F_NEW here and the write fails, the
- ``->iomap_end`` function must delete the reservation.
+ ``end`` function must delete the reservation.
The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.
* **IOMAP_MAPPED**: The file range maps to specific space on the
@@ -208,12 +217,11 @@ The fields are as follows:
* **IOMAP_INLINE**: The file range maps to the memory buffer
specified by ``inline_data``.
- For write operation, the ``->iomap_end`` function presumably
- handles persisting the data.
+ For write operation, the ``end`` function must persist the data.
The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.
* ``flags`` describe the status of the space mapping.
- These flags should be set by the filesystem in ``->iomap_begin``:
+ These flags should be set by the filesystem in ``begin``:
* **IOMAP_F_NEW**: The space under the mapping is newly allocated.
Areas that will not be written to must be zeroed.
@@ -262,15 +270,15 @@ The fields are as follows:
update.
These flags can be set by iomap itself during file operations.
- The filesystem should supply an ``->iomap_end`` function if it needs
+ The filesystem should supply an ``end`` function if it needs
to observe these flags:
* **IOMAP_F_SIZE_CHANGED**: The file size has changed as a result of
using this mapping.
* **IOMAP_F_STALE**: The mapping was found to be stale.
- iomap will call ``->iomap_end`` on this mapping and then
- ``->iomap_begin`` to obtain a new mapping.
+ iomap will call ``end`` on this mapping and then ``begin`` to obtain a
+ new mapping.
Currently, these flags are only set by pagecache operations.
@@ -289,41 +297,103 @@ The fields are as follows:
* ``private`` is a pointer to `filesystem-private information
<https://lore.kernel.org/all/20180619164137.13720-7-hch@lst.de/>`_.
- This value will be passed unchanged to ``->iomap_end``.
+ This value will be passed unchanged to ``end``.
* ``validity_cookie`` is a magic freshness value set by the filesystem
that should be used to detect stale mappings.
For pagecache operations this is critical for correct operation
because page faults can occur, which implies that filesystem locks
- should not be held between ``->iomap_begin`` and ``->iomap_end``.
+ should not be held between ``begin`` and ``end``.
Filesystems with completely static mappings need not set this value.
Only pagecache operations revalidate mappings; see the section about
``iomap_valid`` for details.
-``struct iomap_ops``
+The Mapping Callback
--------------------
-Every iomap function requires the filesystem to pass an operations
-structure to obtain a mapping and (optionally) to release the mapping:
+Every iomap operation takes an ``iomap_iter_next_fn`` callback from the
+filesystem. iomap calls it once per iteration of the file range:
.. code-block:: c
- struct iomap_ops {
- int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,
- unsigned flags, struct iomap *iomap,
- struct iomap *srcmap);
+ typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
- int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,
- ssize_t written, unsigned flags,
- struct iomap *iomap);
- };
+``iomap_iter_next_fn``
+~~~~~~~~~~~~~~~~~~~~~~
+
+This must finish the previous mapping, if any, and then produce the next
+mapping for the current iteration position described by ``iter``.
+The mapping is returned through ``iomap`` (and through ``srcmap`` for
+operations that read from one mapping while writing to another; see
+``begin`` below).
+
+The callback returns ``1`` to continue iterating, ``0`` once the file
+range has been fully consumed, or a negative errno on error.
+
+Most filesystems are not expected to implement all of these behaviors
+in this callback themselves. They should instead call ``iomap_iter_next`` as
+described below.
+
+``iomap_iter_next``
+~~~~~~~~~~~~~~~~~~~
+
+Filesystems rarely need a hand-written ``iomap_iter_next_fn`` callback. The
+``iomap_iter_next`` helper implements the finish-then-produce sequence in
+terms of two smaller callbacks, ``begin`` and ``end``, so most
+``iomap_iter_next_fn`` implementations are simply:
+
+.. code-block:: c
+
+ static int my_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap)
+ {
+ return iomap_iter_next(iter, iomap, srcmap,
+ my_iomap_begin, my_iomap_end);
+ }
+
+This boilerplate is normally generated with the ``DEFINE_IOMAP_ITER_NEXT()``
+and ``DEFINE_IOMAP_ITER_NEXT_END()`` macros rather than written by hand. Use
+``DEFINE_IOMAP_ITER_NEXT()`` for the common case with no ``end`` callback, and
+``DEFINE_IOMAP_ITER_NEXT_END()`` when an ``end`` callback is needed:
+
+.. code-block:: c
+
+ /* no end() callback */
+ static DEFINE_IOMAP_ITER_NEXT(my_iomap_next, my_iomap_begin);
+
+ /* with an end() callback */
+ static DEFINE_IOMAP_ITER_NEXT_END(my_iomap_next, my_iomap_begin,
+ my_iomap_end);
+
+The macro generates a function with external linkage. Prefix the invocation
+with ``static`` for a file-local callback, or leave it non-``static`` and add
+a matching declaration to a header when the callback is referenced from
+another file.
+
+``end`` may be ``NULL`` when the filesystem has nothing to finish.
+The two callbacks have these prototypes:
+
+.. code-block:: c
+
+ typedef int (*iomap_begin_fn)(struct inode *inode, loff_t pos,
+ loff_t length, unsigned flags,
+ struct iomap *iomap, struct iomap *srcmap);
+
+ typedef int (*iomap_end_fn)(struct inode *inode, loff_t pos,
+ loff_t length, ssize_t written,
+ unsigned flags, struct iomap *iomap);
+
+``iomap_iter_next`` is an inline helper, so when it is called with fixed
+``begin`` and ``end`` functions the compiler can inline both into the
+filesystem's ``iomap_iter_next_fn`` callback, keeping indirect calls out of the
+iteration hot path. The two callbacks are described next.
-``->iomap_begin``
-~~~~~~~~~~~~~~~~~
+``begin``
+~~~~~~~~~
-iomap operations call ``->iomap_begin`` to obtain one file mapping for
-the range of bytes specified by ``pos`` and ``length`` for the file
-``inode``.
+The ``begin`` callback obtains one file mapping for the range of bytes
+specified by ``pos`` and ``length`` for the file ``inode``.
This mapping should be returned through the ``iomap`` pointer.
The mapping must cover at least the first byte of the supplied file
range, but it does not need to cover the entire requested range.
@@ -377,18 +447,18 @@ information via ``srcmap``.
Only pagecache and fsdax operations support reading from one mapping and
writing to another.
-``->iomap_end``
-~~~~~~~~~~~~~~~
+``end``
+~~~~~~~
-After the operation completes, the ``->iomap_end`` function, if present,
-is called to signal that iomap is finished with a mapping.
+After the operation completes, the ``end`` callback, if present, is called to
+signal that iomap is finished with a mapping.
Typically, implementations will use this function to tear down any
-context that were set up in ``->iomap_begin``.
+context that was set up in ``begin``.
For example, a write might wish to commit the reservations for the bytes
that were operated upon and unreserve any space that was not operated
upon.
``written`` might be zero if no bytes were touched.
-``flags`` will contain the same value passed to ``->iomap_begin``.
+``flags`` will contain the same value passed to ``begin``.
iomap ops for reads are not likely to need to supply this function.
Both functions should return a negative errno code on error, or zero on
@@ -421,8 +491,8 @@ iomap is concerned:
accessing the folio until writeback is underway.
* The **lower** level primitive is taken by the filesystem in the
- ``->iomap_begin`` and ``->iomap_end`` functions to coordinate
- access to the file space mapping information.
+ ``begin`` and ``end`` functions to coordinate access to the file space
+ mapping information.
The fields of the iomap object should be filled out while holding
this primitive.
The upper level synchronization primitive, if any, remains held
diff --git a/Documentation/filesystems/iomap/operations.rst b/Documentation/filesystems/iomap/operations.rst
index da982ca7e4137..a5784dee08509 100644
--- a/Documentation/filesystems/iomap/operations.rst
+++ b/Documentation/filesystems/iomap/operations.rst
@@ -17,6 +17,12 @@ Supported File Operations
Below are a discussion of the high level file operations that iomap
implements.
+Each operation takes an ``iomap_iter_next_fn`` callback that supplies the file
+mappings, as described in the iomap design document. The per-operation
+``flags`` documented below are passed to that callback; references to
+``begin`` and ``end`` name the two steps a typical callback is built from
+via ``iomap_iter_next``.
+
Buffered I/O
============
@@ -90,10 +96,10 @@ iomap calls these functions:
<https://lore.kernel.org/all/20180619164137.13720-6-hch@lst.de/>`_
that was set up by ``->get_folio``.
- - ``iomap_valid``: The filesystem may not hold locks between
- ``->iomap_begin`` and ``->iomap_end`` because pagecache operations
- can take folio locks, fault on userspace pages, initiate writeback
- for memory reclamation, or engage in other time-consuming actions.
+ - ``iomap_valid``: The filesystem may not hold locks between ``begin`` and
+ ``end`` because pagecache operations can take folio locks, fault on
+ userspace pages, initiate writeback for memory reclamation, or engage in
+ other time-consuming actions.
If a file's space mapping data are mutable, it is possible that the
mapping for a particular pagecache folio can `change in the time it
takes
@@ -114,12 +120,12 @@ iomap calls these functions:
If the mapping is not valid, the mapping will be sampled again.
To support making the validity decision, the filesystem's
- ``->iomap_begin`` function may set ``struct iomap::validity_cookie``
+ ``begin`` function may set ``struct iomap::validity_cookie``
at the same time that it populates the other iomap fields.
A simple validation cookie implementation is a sequence counter.
If the filesystem bumps the sequence counter every time it modifies
the inode's extent map, it can be placed in the ``struct
- iomap::validity_cookie`` during ``->iomap_begin``.
+ iomap::validity_cookie`` during ``begin``.
If the value in the cookie is found to be different to the value
the filesystem holds when the mapping is passed back to
``->iomap_valid``, then the iomap should considered stale and the
@@ -199,7 +205,7 @@ Buffered Readahead and Reads
The ``iomap_readahead`` function initiates readahead to the pagecache.
The ``iomap_read_folio`` function reads one folio's worth of data into
the pagecache.
-The ``flags`` argument to ``->iomap_begin`` will be set to zero.
+The ``flags`` argument to ``begin`` will be set to zero.
The pagecache takes whatever locks it needs before calling the
filesystem.
@@ -231,7 +237,7 @@ Buffered Writes
The ``iomap_file_buffered_write`` function writes an ``iocb`` to the
pagecache.
``IOMAP_WRITE`` or ``IOMAP_WRITE`` | ``IOMAP_NOWAIT`` will be passed as
-the ``flags`` argument to ``->iomap_begin``.
+the ``flags`` argument to ``begin``.
Callers commonly take ``i_rwsem`` in either shared or exclusive mode
before calling this function.
@@ -241,7 +247,7 @@ mmap Write Faults
The ``iomap_page_mkwrite`` function handles a write fault to a folio in
the pagecache.
``IOMAP_WRITE | IOMAP_FAULT`` will be passed as the ``flags`` argument
-to ``->iomap_begin``.
+to ``begin``.
Callers commonly take the mmap ``invalidate_lock`` in shared or
exclusive mode before calling this function.
@@ -256,7 +262,7 @@ such `reservations
<https://lore.kernel.org/linux-xfs/20220817093627.GZ3600936@dread.disaster.area/>`_
because writeback will not consume the reservation.
The ``iomap_write_delalloc_release`` can be called from a
-``->iomap_end`` function to find all the clean areas of the folios
+``end`` function to find all the clean areas of the folios
caching a fresh (``IOMAP_F_NEW``) delalloc mapping.
It takes the ``invalidate_lock``.
@@ -274,7 +280,7 @@ Filesystems can call ``iomap_zero_range`` to perform zeroing of the
pagecache for non-truncation file operations that are not aligned to
the fsblock size.
``IOMAP_ZERO`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``begin``.
Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
mode before calling this function.
@@ -285,7 +291,7 @@ Filesystems can call ``iomap_file_unshare`` to force a file sharing
storage with another file to preemptively copy the shared data to newly
allocate storage.
``IOMAP_WRITE | IOMAP_UNSHARE`` will be passed as the ``flags`` argument
-to ``->iomap_begin``.
+to ``begin``.
Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
mode before calling this function.
@@ -298,7 +304,7 @@ operation.
``truncate_setsize`` or ``truncate_pagecache`` will take care of
everything after the EOF block.
``IOMAP_ZERO`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``begin``.
Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
mode before calling this function.
@@ -341,8 +347,8 @@ The fields are as follows:
though it will `reuse mappings
<https://lore.kernel.org/all/20231207072710.176093-15-hch@lst.de/>`_
for runs of contiguous dirty fsblocks within a folio.
- Do not return ``IOMAP_INLINE`` mappings here; the ``->iomap_end``
- function must deal with persisting written data.
+ Do not return ``IOMAP_INLINE`` mappings here; the ``end`` function must
+ deal with persisting written data.
Do not return ``IOMAP_DELALLOC`` mappings here; iomap currently
requires mapping to allocated space.
Filesystems can skip a potentially expensive mapping lookup if the
@@ -428,7 +434,7 @@ writes for files.
.. code-block:: c
ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn iomap_next,
const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private,
size_t done_before);
@@ -511,7 +517,7 @@ Return Values
* ``-ENOTBLK``: Fall back to buffered I/O.
iomap itself will return this value if it cannot invalidate the page
cache before issuing the I/O to storage.
- The ``->iomap_begin`` or ``->iomap_end`` functions may also return
+ The ``begin`` or ``end`` functions may also return
this value.
* ``-EIOCBQUEUED``: The asynchronous direct I/O request has been
@@ -526,7 +532,7 @@ A direct I/O read initiates a read I/O from the storage device to the
caller's buffer.
Dirty parts of the pagecache are flushed to storage before initiating
the read io.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DIRECT`` with
+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT`` with
any combination of the following enhancements:
* ``IOMAP_NOWAIT``, as defined previously.
@@ -542,7 +548,7 @@ caller's buffer.
Dirty parts of the pagecache are flushed to storage before initiating
the write io.
The pagecache is invalidated both before and after the write io.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DIRECT |
+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT |
IOMAP_WRITE`` with any combination of the following enhancements:
* ``IOMAP_NOWAIT``, as defined previously.
@@ -644,7 +650,7 @@ fsdax Reads
A fsdax read performs a memcpy from storage device to the caller's
buffer.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DAX`` with any
+The ``flags`` value for ``begin`` will be ``IOMAP_DAX`` with any
combination of the following enhancements:
* ``IOMAP_NOWAIT``, as defined previously.
@@ -657,7 +663,7 @@ fsdax Writes
A fsdax write initiates a memcpy to the storage device from the caller's
buffer.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DAX |
+The ``flags`` value for ``begin`` will be ``IOMAP_DAX |
IOMAP_WRITE`` with any combination of the following enhancements:
* ``IOMAP_NOWAIT``, as defined previously.
@@ -680,9 +686,9 @@ fsdax mmap Faults
The ``dax_iomap_fault`` function handles read and write faults to fsdax
storage.
For a read fault, ``IOMAP_DAX | IOMAP_FAULT`` will be passed as the
-``flags`` argument to ``->iomap_begin``.
+``flags`` argument to ``begin``.
For a write fault, ``IOMAP_DAX | IOMAP_FAULT | IOMAP_WRITE`` will be
-passed as the ``flags`` argument to ``->iomap_begin``.
+passed as the ``flags`` argument to ``begin``.
Callers commonly hold the same locks as they do to call their iomap
pagecache counterparts.
@@ -692,8 +698,8 @@ fsdax Truncation, fallocate, and Unsharing
For fsdax files, the following functions are provided to replace their
iomap pagecache I/O counterparts.
-The ``flags`` argument to ``->iomap_begin`` are the same as the
-pagecache counterparts, with ``IOMAP_DAX`` added.
+The ``flags`` argument to ``begin`` are the same as the pagecache counterparts,
+with ``IOMAP_DAX`` added.
* ``dax_file_unshare``
* ``dax_zero_range``
@@ -719,8 +725,7 @@ SEEK_DATA
The ``iomap_seek_data`` function implements the SEEK_DATA "whence" value
for llseek.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
For unwritten mappings, the pagecache will be searched.
Regions of the pagecache with a folio mapped and uptodate fsblocks
@@ -734,8 +739,7 @@ SEEK_HOLE
The ``iomap_seek_hole`` function implements the SEEK_HOLE "whence" value
for llseek.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
For unwritten mappings, the pagecache will be searched.
Regions of the pagecache with no folio mapped, or a !uptodate fsblock
@@ -750,8 +754,7 @@ Swap File Activation
The ``iomap_swapfile_activate`` function finds all the base-page aligned
regions in a file and sets them up as swap space.
The file will be ``fsync()``'d before activation.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
All mappings must be mapped or unwritten; cannot be dirty or shared, and
cannot span multiple block devices.
Callers must hold ``i_rwsem`` in exclusive mode; this is already
@@ -767,8 +770,7 @@ FS_IOC_FIEMAP
The ``iomap_fiemap`` function exports file extent mappings to userspace
in the format specified by the ``FS_IOC_FIEMAP`` ioctl.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
Callers commonly hold ``i_rwsem`` in shared mode before calling this
function.
diff --git a/Documentation/filesystems/iomap/porting.rst b/Documentation/filesystems/iomap/porting.rst
index 3d49a32c0fff9..b995f426808a9 100644
--- a/Documentation/filesystems/iomap/porting.rst
+++ b/Documentation/filesystems/iomap/porting.rst
@@ -50,8 +50,14 @@ Build the kernel, run fstests with the ``-g all`` option across a wide
variety of your filesystem's supported configurations to build a
baseline of which tests pass and which ones fail.
-The recommended approach is first to implement ``->iomap_begin`` (and
-``->iomap_end`` if necessary) to allow iomap to obtain a read-only
+Every iomap operation is driven by an ``iomap_iter_next_fn`` callback.
+Filesystems normally do not write one by hand: implement ``begin``
+(and ``end`` if necessary) and generate the callback with the
+``DEFINE_IOMAP_ITER_NEXT()`` macro, or ``DEFINE_IOMAP_ITER_NEXT_END()`` if an
+``end`` callback is needed.
+
+The recommended approach is first to implement ``begin`` (and
+``end`` if necessary) to allow iomap to obtain a read-only
mapping of a file range.
In most cases, this is a relatively trivial conversion of the existing
``get_block()`` function for read-only mappings.
@@ -62,7 +68,7 @@ If FIEMAP is returning the correct information, it's a good sign that
other read-only mapping operations will do the right thing.
Next, modify the filesystem's ``get_block(create = false)``
-implementation to use the new ``->iomap_begin`` implementation to map
+implementation to use the new ``begin`` implementation to map
file space for selected read operations.
Hide behind a debugging knob the ability to switch on the iomap mapping
functions for selected call paths.
@@ -82,14 +88,14 @@ I/O path because of bufferheads.
The buffered read I/O paths doesn't need to be converted yet, though the
direct I/O read path should be converted in this phase.
-At this point, you should look over your ``->iomap_begin`` function.
+At this point, you should look over your ``begin`` function.
If it switches between large blocks of code based on dispatching of the
``flags`` argument, you should consider breaking it up into
per-operation iomap ops with smaller, more cohesive functions.
XFS is a good example of this.
The next thing to do is implement ``get_blocks(create == true)``
-functionality in the ``->iomap_begin``/``->iomap_end`` methods.
+functionality in the ``begin``/``end`` methods.
It is strongly recommended to create separate mapping functions and
iomap ops for write operations.
Then convert the direct I/O write path to iomap, and start running fsx
diff --git a/block/fops.c b/block/fops.c
index 15783a6180dec..53fde467f6ee8 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -453,9 +453,7 @@ static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static const struct iomap_ops blkdev_iomap_ops = {
- .iomap_begin = blkdev_iomap_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(blkdev_iomap_next, blkdev_iomap_begin);
#ifdef CONFIG_BUFFER_HEAD
static int blkdev_get_block(struct inode *inode, sector_t iblock,
@@ -510,13 +508,13 @@ const struct address_space_operations def_blk_aops = {
#else /* CONFIG_BUFFER_HEAD */
static int blkdev_read_folio(struct file *file, struct folio *folio)
{
- iomap_bio_read_folio(folio, &blkdev_iomap_ops);
+ iomap_bio_read_folio(folio, blkdev_iomap_next);
return 0;
}
static void blkdev_readahead(struct readahead_control *rac)
{
- iomap_bio_readahead(rac, &blkdev_iomap_ops);
+ iomap_bio_readahead(rac, blkdev_iomap_next);
}
static ssize_t blkdev_writeback_range(struct iomap_writepage_ctx *wpc,
@@ -707,7 +705,7 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter *from)
static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *from)
{
- return iomap_file_buffered_write(iocb, from, &blkdev_iomap_ops, NULL,
+ return iomap_file_buffered_write(iocb, from, blkdev_iomap_next, NULL,
NULL);
}
diff --git a/fs/btrfs/direct-io.c b/fs/btrfs/direct-io.c
index 460326d34143c..b9ff7f9e59f03 100644
--- a/fs/btrfs/direct-io.c
+++ b/fs/btrfs/direct-io.c
@@ -798,10 +798,8 @@ static void btrfs_dio_submit_io(const struct iomap_iter *iter, struct bio *bio,
btrfs_submit_bbio(bbio, 0);
}
-static const struct iomap_ops btrfs_dio_iomap_ops = {
- .iomap_begin = btrfs_dio_iomap_begin,
- .iomap_end = btrfs_dio_iomap_end,
-};
+static DEFINE_IOMAP_ITER_NEXT_END(btrfs_dio_iomap_next, btrfs_dio_iomap_begin,
+ btrfs_dio_iomap_end);
static const struct iomap_dio_ops btrfs_dio_ops = {
.submit_io = btrfs_dio_submit_io,
@@ -813,7 +811,7 @@ static ssize_t btrfs_dio_read(struct kiocb *iocb, struct iov_iter *iter,
{
struct btrfs_dio_data data = { 0 };
- return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+ return iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
}
@@ -822,7 +820,7 @@ static struct iomap_dio *btrfs_dio_write(struct kiocb *iocb, struct iov_iter *it
{
struct btrfs_dio_data data = { 0 };
- return __iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+ return __iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
}
diff --git a/fs/dax.c b/fs/dax.c
index 6d175cd47a99b..19e5bbe31b69f 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -1492,7 +1492,7 @@ static int dax_unshare_iter(struct iomap_iter *iter)
}
int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -1506,7 +1506,7 @@ int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
return 0;
iter.len = min(len, size - pos);
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = dax_unshare_iter(&iter);
return ret;
}
@@ -1584,7 +1584,7 @@ static int dax_zero_iter(struct iomap_iter *iter, bool *did_zero)
}
int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -1594,14 +1594,14 @@ int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
};
int ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = dax_zero_iter(&iter, did_zero);
return ret;
}
EXPORT_SYMBOL_GPL(dax_zero_range);
int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
unsigned int blocksize = i_blocksize(inode);
unsigned int off = pos & (blocksize - 1);
@@ -1609,7 +1609,7 @@ int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
/* Block boundary? Nothing to do */
if (!off)
return 0;
- return dax_zero_range(inode, pos, blocksize - off, did_zero, ops);
+ return dax_zero_range(inode, pos, blocksize - off, did_zero, next);
}
EXPORT_SYMBOL_GPL(dax_truncate_page);
@@ -1734,7 +1734,8 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
* dax_iomap_rw - Perform I/O to a DAX file
* @iocb: The control block for this I/O
* @iter: The addresses to do I/O from or to
- * @ops: iomap ops passed from the file system
+ * @next: Callback passed from the file system for advancing to next
+ * iteration
*
* This function performs read and write operations to directly mapped
* persistent memory. The callers needs to take care of read/write exclusion
@@ -1742,7 +1743,7 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
*/
ssize_t
dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_iter iomi = {
.inode = iocb->ki_filp->f_mapping->host,
@@ -1769,7 +1770,7 @@ dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
if (iocb->ki_flags & IOCB_NOWAIT)
iomi.flags |= IOMAP_NOWAIT;
- while ((ret = iomap_iter(&iomi, ops)) > 0)
+ while ((ret = iomap_iter(&iomi, next)) > 0)
iomi.status = dax_iomap_iter(&iomi, iter);
done = iomi.pos - iocb->ki_pos;
@@ -1897,7 +1898,7 @@ static vm_fault_t dax_fault_iter(struct vm_fault *vmf,
}
static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
- int *iomap_errp, const struct iomap_ops *ops)
+ int *iomap_errp, iomap_iter_next_fn next)
{
struct address_space *mapping = vmf->vma->vm_file->f_mapping;
XA_STATE(xas, &mapping->i_pages, vmf->pgoff);
@@ -1942,7 +1943,7 @@ static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
goto unlock_entry;
}
- while ((error = iomap_iter(&iter, ops)) > 0) {
+ while ((error = iomap_iter(&iter, next)) > 0) {
if (WARN_ON_ONCE(iomap_length(&iter) < PAGE_SIZE)) {
iter.status = -EIO; /* fs corruption? */
continue;
@@ -2007,7 +2008,7 @@ static bool dax_fault_check_fallback(struct vm_fault *vmf, struct xa_state *xas,
}
static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct address_space *mapping = vmf->vma->vm_file->f_mapping;
XA_STATE_ORDER(xas, &mapping->i_pages, vmf->pgoff, PMD_ORDER);
@@ -2064,7 +2065,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
}
iter.pos = (loff_t)xas.xa_index << PAGE_SHIFT;
- while (iomap_iter(&iter, ops) > 0) {
+ while (iomap_iter(&iter, next) > 0) {
if (iomap_length(&iter) < PMD_SIZE)
continue; /* actually breaks out of the loop */
@@ -2086,7 +2087,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
}
#else
static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
return VM_FAULT_FALLBACK;
}
@@ -2098,7 +2099,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
* @order: Order of the page to fault in
* @pfnp: PFN to insert for synchronous faults if fsync is required
* @iomap_errp: Storage for detailed error code in case of error
- * @ops: Iomap ops passed from the file system
+ * @next: The iomap iteration function for the filesystem
*
* When a page fault occurs, filesystems may call this helper in
* their fault handler for DAX files. dax_iomap_fault() assumes the caller
@@ -2107,12 +2108,12 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
*/
vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
unsigned long *pfnp, int *iomap_errp,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
if (order == 0)
- return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, ops);
+ return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, next);
else if (order == PMD_ORDER)
- return dax_iomap_pmd_fault(vmf, pfnp, ops);
+ return dax_iomap_pmd_fault(vmf, pfnp, next);
else
return VM_FAULT_FALLBACK;
}
@@ -2240,7 +2241,7 @@ static int dax_range_compare_iter(struct iomap_iter *it_src,
int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
struct inode *dst, loff_t dstoff, loff_t len, bool *same,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_iter src_iter = {
.inode = src,
@@ -2256,8 +2257,8 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
};
int ret, status;
- while ((ret = iomap_iter(&src_iter, ops)) > 0 &&
- (ret = iomap_iter(&dst_iter, ops)) > 0) {
+ while ((ret = iomap_iter(&src_iter, next)) > 0 &&
+ (ret = iomap_iter(&dst_iter, next)) > 0) {
status = dax_range_compare_iter(&src_iter, &dst_iter,
min(src_iter.len, dst_iter.len), same);
if (status < 0)
@@ -2270,9 +2271,10 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
return __generic_remap_file_range_prep(file_in, pos_in, file_out,
- pos_out, len, remap_flags, ops);
+ pos_out, len, remap_flags,
+ next);
}
EXPORT_SYMBOL_GPL(dax_remap_file_range_prep);
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 9aa48c8d67d12..d965ed015214f 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -380,10 +380,8 @@ static int erofs_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return written;
}
-static const struct iomap_ops erofs_iomap_ops = {
- .iomap_begin = erofs_iomap_begin,
- .iomap_end = erofs_iomap_end,
-};
+static DEFINE_IOMAP_ITER_NEXT_END(erofs_iomap_next, erofs_iomap_begin,
+ erofs_iomap_end);
int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
@@ -392,9 +390,9 @@ int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
return -EOPNOTSUPP;
return iomap_fiemap(inode, fieinfo, start, len,
- &z_erofs_iomap_report_ops);
+ z_erofs_iomap_next_report);
}
- return iomap_fiemap(inode, fieinfo, start, len, &erofs_iomap_ops);
+ return iomap_fiemap(inode, fieinfo, start, len, erofs_iomap_next);
}
/*
@@ -413,7 +411,7 @@ static int erofs_read_folio(struct file *file, struct folio *folio)
};
trace_erofs_read_folio(iter_ctx.realinode, folio, true);
- iomap_read_folio(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+ iomap_read_folio(erofs_iomap_next, &read_ctx, &iter_ctx);
if (need_iput)
iput(iter_ctx.realinode);
return 0;
@@ -432,14 +430,14 @@ static void erofs_readahead(struct readahead_control *rac)
trace_erofs_readahead(iter_ctx.realinode, readahead_index(rac),
readahead_count(rac), true);
- iomap_readahead(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+ iomap_readahead(erofs_iomap_next, &read_ctx, &iter_ctx);
if (need_iput)
iput(iter_ctx.realinode);
}
static sector_t erofs_bmap(struct address_space *mapping, sector_t block)
{
- return iomap_bmap(mapping, block, &erofs_iomap_ops);
+ return iomap_bmap(mapping, block, erofs_iomap_next);
}
static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -451,14 +449,14 @@ static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
return 0;
if (IS_ENABLED(CONFIG_FS_DAX) && IS_DAX(inode))
- return dax_iomap_rw(iocb, to, &erofs_iomap_ops);
+ return dax_iomap_rw(iocb, to, erofs_iomap_next);
if ((iocb->ki_flags & IOCB_DIRECT) && inode->i_sb->s_bdev) {
struct erofs_iomap_iter_ctx iter_ctx = {
.realinode = inode,
};
- return iomap_dio_rw(iocb, to, &erofs_iomap_ops,
+ return iomap_dio_rw(iocb, to, erofs_iomap_next,
NULL, 0, &iter_ctx, 0);
}
return filemap_read(iocb, to, 0);
@@ -478,7 +476,7 @@ const struct address_space_operations erofs_aops = {
static vm_fault_t erofs_dax_huge_fault(struct vm_fault *vmf,
unsigned int order)
{
- return dax_iomap_fault(vmf, order, NULL, NULL, &erofs_iomap_ops);
+ return dax_iomap_fault(vmf, order, NULL, NULL, erofs_iomap_next);
}
static vm_fault_t erofs_dax_fault(struct vm_fault *vmf)
@@ -510,18 +508,18 @@ static int erofs_file_mmap_prepare(struct vm_area_desc *desc)
static loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence)
{
struct inode *inode = file->f_mapping->host;
- const struct iomap_ops *ops = &erofs_iomap_ops;
+ iomap_iter_next_fn next = erofs_iomap_next;
if (erofs_inode_is_data_compressed(EROFS_I(inode)->datalayout)) {
if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
return generic_file_llseek(file, offset, whence);
- ops = &z_erofs_iomap_report_ops;
+ next = z_erofs_iomap_next_report;
}
if (whence == SEEK_HOLE)
- offset = iomap_seek_hole(inode, offset, ops);
+ offset = iomap_seek_hole(inode, offset, next);
else if (whence == SEEK_DATA)
- offset = iomap_seek_data(inode, offset, ops);
+ offset = iomap_seek_data(inode, offset, next);
else
return generic_file_llseek(file, offset, whence);
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 580f8d9f14e7f..72ccd6f335b82 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -397,7 +397,8 @@ extern const struct file_operations erofs_file_fops;
extern const struct file_operations erofs_dir_fops;
extern const struct file_operations erofs_ishare_fops;
-extern const struct iomap_ops z_erofs_iomap_report_ops;
+int z_erofs_iomap_next_report(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
void *erofs_read_metadata(struct super_block *sb, struct erofs_buf *buf,
erofs_off_t *offset, int *lengthp);
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index bab5216135524..98968a9d942ee 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -821,6 +821,4 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset,
return 0;
}
-const struct iomap_ops z_erofs_iomap_report_ops = {
- .iomap_begin = z_erofs_iomap_begin_report,
-};
+DEFINE_IOMAP_ITER_NEXT(z_erofs_iomap_next_report, z_erofs_iomap_begin_report);
diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index 5fc13378d35f7..c05849d305ae4 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -668,7 +668,7 @@ static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size)
ret = iomap_zero_range(inode, old_valid_size,
new_valid_size - old_valid_size, NULL,
- &exfat_write_iomap_ops, NULL, NULL);
+ exfat_write_iomap_next, NULL, NULL);
if (ret) {
truncate_setsize(inode, old_valid_size);
exfat_truncate(inode);
@@ -687,7 +687,7 @@ static ssize_t exfat_fallback_buffered_write(struct kiocb *iocb,
iocb->ki_flags &= ~IOCB_DIRECT;
- written = iomap_file_buffered_write(iocb, from, &exfat_write_iomap_ops,
+ written = iomap_file_buffered_write(iocb, from, exfat_write_iomap_next,
NULL, NULL);
if (written < 0)
return written;
@@ -709,7 +709,7 @@ static ssize_t exfat_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
ssize_t ret;
- ret = iomap_dio_rw(iocb, from, &exfat_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, exfat_write_iomap_next,
&exfat_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -773,7 +773,7 @@ static ssize_t exfat_file_write_iter(struct kiocb *iocb, struct iov_iter *iter)
ret = exfat_dio_write_iter(iocb, iter);
else
ret = iomap_file_buffered_write(iocb, iter,
- &exfat_write_iomap_ops, NULL, NULL);
+ exfat_write_iomap_next, NULL, NULL);
if (ret < 0)
goto unlock;
@@ -809,7 +809,7 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
if (iocb->ki_flags & IOCB_DIRECT) {
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, iter, &exfat_iomap_ops, NULL, 0,
+ ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0,
NULL, 0);
} else {
ret = generic_file_read_iter(iocb, iter);
@@ -850,7 +850,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
*/
err = iomap_zero_range(inode, ei->zeroed_size,
mmap_valid_size - ei->zeroed_size, NULL,
- &exfat_iomap_ops, NULL, NULL);
+ exfat_iomap_next, NULL, NULL);
if (err < 0) {
inode_unlock(inode);
return vmf_fs_error(err);
@@ -866,7 +866,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
file_update_time(vmf->vma->vm_file);
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = iomap_page_mkwrite(vmf, &exfat_write_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, exfat_write_iomap_next, NULL);
filemap_invalidate_unlock_shared(inode->i_mapping);
sb_end_pagefault(inode->i_sb);
inode_unlock(inode);
@@ -939,12 +939,12 @@ static loff_t exfat_file_llseek(struct file *file, loff_t offset, int whence)
switch (whence) {
case SEEK_HOLE:
inode_lock_shared(inode);
- offset = iomap_seek_hole(inode, offset, &exfat_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, exfat_iomap_next);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
- offset = iomap_seek_data(inode, offset, &exfat_iomap_ops);
+ offset = iomap_seek_data(inode, offset, exfat_iomap_next);
inode_unlock_shared(inode);
break;
default:
diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index 89826aea5e1e3..a6b9aa2ad7926 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -248,7 +248,7 @@ static int exfat_read_folio(struct file *file, struct folio *folio)
.ops = &exfat_iomap_bio_read_ops,
};
- iomap_read_folio(&exfat_iomap_ops, &ctx, NULL);
+ iomap_read_folio(exfat_iomap_next, &ctx, NULL);
return 0;
}
@@ -269,7 +269,7 @@ static void exfat_readahead(struct readahead_control *rac)
ei->valid_size < pos + readahead_length(rac))
return;
- iomap_readahead(&exfat_iomap_ops, &ctx, NULL);
+ iomap_readahead(exfat_iomap_next, &ctx, NULL);
}
static int exfat_writepages(struct address_space *mapping,
@@ -293,7 +293,7 @@ static sector_t exfat_aop_bmap(struct address_space *mapping, sector_t block)
/* exfat_get_cluster() assumes the requested blocknr isn't truncated. */
down_read(&EXFAT_I(mapping->host)->truncate_lock);
- blocknr = iomap_bmap(mapping, block, &exfat_iomap_ops);
+ blocknr = iomap_bmap(mapping, block, exfat_iomap_next);
up_read(&EXFAT_I(mapping->host)->truncate_lock);
return blocknr;
}
diff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c
index 1aac38e63fe6f..631de19559d5c 100644
--- a/fs/exfat/iomap.c
+++ b/fs/exfat/iomap.c
@@ -151,9 +151,7 @@ static int exfat_write_iomap_begin(struct inode *inode, loff_t offset, loff_t le
return __exfat_iomap_begin(inode, offset, length, flags, iomap, true);
}
-const struct iomap_ops exfat_iomap_ops = {
- .iomap_begin = exfat_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(exfat_iomap_next, exfat_iomap_begin);
/*
* exfat_write_iomap_end - Update the state after write
@@ -186,10 +184,8 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return written;
}
-const struct iomap_ops exfat_write_iomap_ops = {
- .iomap_begin = exfat_write_iomap_begin,
- .iomap_end = exfat_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(exfat_write_iomap_next, exfat_write_iomap_begin,
+ exfat_write_iomap_end);
/*
* exfat_writeback_range - Map folio during writeback
@@ -267,5 +263,5 @@ const struct iomap_read_ops exfat_iomap_bio_read_ops = {
int exfat_iomap_swap_activate(struct swap_info_struct *sis,
struct file *file, sector_t *span)
{
- return iomap_swapfile_activate(sis, file, span, &exfat_iomap_ops);
+ return iomap_swapfile_activate(sis, file, span, exfat_iomap_next);
}
diff --git a/fs/exfat/iomap.h b/fs/exfat/iomap.h
index fd8a913f77946..47d7b753735e8 100644
--- a/fs/exfat/iomap.h
+++ b/fs/exfat/iomap.h
@@ -7,8 +7,10 @@
#define _LINUX_EXFAT_IOMAP_H
extern const struct iomap_dio_ops exfat_write_dio_ops;
-extern const struct iomap_ops exfat_iomap_ops;
-extern const struct iomap_ops exfat_write_iomap_ops;
+int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int exfat_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_writeback_ops exfat_writeback_ops;
extern const struct iomap_read_ops exfat_iomap_bio_read_ops;
diff --git a/fs/ext2/ext2.h b/fs/ext2/ext2.h
index 79f7b395258c2..59ef8b898940f 100644
--- a/fs/ext2/ext2.h
+++ b/fs/ext2/ext2.h
@@ -780,7 +780,8 @@ extern const struct file_operations ext2_file_operations;
/* inode.c */
extern void ext2_set_file_ops(struct inode *inode);
extern const struct address_space_operations ext2_aops;
-extern const struct iomap_ops ext2_iomap_ops;
+int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
/* namei.c */
extern const struct inode_operations ext2_dir_inode_operations;
diff --git a/fs/ext2/file.c b/fs/ext2/file.c
index 8dca9ec4cacd9..1fc00ad775174 100644
--- a/fs/ext2/file.c
+++ b/fs/ext2/file.c
@@ -70,7 +70,7 @@ static ssize_t ext2_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
trace_ext2_dio_read_begin(iocb, to, 0);
inode_lock_shared(inode);
- ret = iomap_dio_rw(iocb, to, &ext2_iomap_ops, NULL, 0, NULL, 0);
+ ret = iomap_dio_rw(iocb, to, ext2_iomap_next, NULL, 0, NULL, 0);
inode_unlock_shared(inode);
trace_ext2_dio_read_end(iocb, to, ret);
@@ -134,7 +134,7 @@ static ssize_t ext2_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
(!IS_ALIGNED(iocb->ki_pos | iov_iter_alignment(from), blocksize)))
flags |= IOMAP_DIO_FORCE_WAIT;
- ret = iomap_dio_rw(iocb, from, &ext2_iomap_ops, &ext2_dio_write_ops,
+ ret = iomap_dio_rw(iocb, from, ext2_iomap_next, &ext2_dio_write_ops,
flags, NULL, 0);
/* ENOTBLK is magic return value for fallback to buffered-io */
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 29808629cce56..873ad1d90e0dd 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -860,10 +860,7 @@ ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-const struct iomap_ops ext2_iomap_ops = {
- .iomap_begin = ext2_iomap_begin,
- .iomap_end = ext2_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ext2_iomap_next, ext2_iomap_begin, ext2_iomap_end);
int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
@@ -882,7 +879,7 @@ int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (i_size == 0)
i_size = 1;
len = min_t(u64, len, i_size);
- ret = iomap_fiemap(inode, fieinfo, start, len, &ext2_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, ext2_iomap_next);
inode_unlock(inode);
return ret;
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index b37c136ea3ab3..f386a86e945cd 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -4004,8 +4004,13 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
}
-extern const struct iomap_ops ext4_iomap_ops;
-extern const struct iomap_ops ext4_iomap_report_ops;
+int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+
+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+ unsigned flags, struct iomap *iomap, struct iomap *srcmap);
static inline int ext4_buffer_uptodate(struct buffer_head *bh)
{
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 91c97af64b317..71bbf8e77261f 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -5171,9 +5171,7 @@ static int ext4_iomap_xattr_begin(struct inode *inode, loff_t offset,
return error;
}
-static const struct iomap_ops ext4_iomap_xattr_ops = {
- .iomap_begin = ext4_iomap_xattr_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_xattr_next, ext4_iomap_xattr_begin);
static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)
{
@@ -5217,10 +5215,10 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
error = iomap_fiemap(inode, fieinfo, start, len,
- &ext4_iomap_xattr_ops);
+ ext4_iomap_xattr_next);
} else {
error = iomap_fiemap(inode, fieinfo, start, len,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
}
unlock:
inode_unlock_shared(inode);
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index eb1a323962b10..da6fbf76ca51b 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,9 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
return generic_file_read_iter(iocb, to);
}
- ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL, 0, NULL, 0);
+ ret = iomap_dio_read_simple(iocb, to, ext4_iomap_begin);
+ if (ret == -ENOTBLK)
+ ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
inode_unlock_shared(inode);
file_accessed(iocb->ki_filp);
@@ -119,7 +121,7 @@ static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
/* Fallback to buffered IO in case we cannot support DAX */
return generic_file_read_iter(iocb, to);
}
- ret = dax_iomap_rw(iocb, to, &ext4_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, ext4_iomap_next);
inode_unlock_shared(inode);
file_accessed(iocb->ki_filp);
@@ -589,7 +591,7 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
goto out;
}
- ret = iomap_dio_rw(iocb, from, &ext4_iomap_ops, &ext4_dio_write_ops,
+ ret = iomap_dio_rw(iocb, from, ext4_iomap_next, &ext4_dio_write_ops,
dio_flags, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -688,7 +690,7 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
ext4_journal_stop(handle);
}
- ret = dax_iomap_rw(iocb, from, &ext4_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, ext4_iomap_next);
if (extend) {
ret = ext4_handle_inode_extension(inode, offset, ret, count);
@@ -776,7 +778,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, unsigned int order)
} else {
filemap_invalidate_lock_shared(mapping);
}
- result = dax_iomap_fault(vmf, order, &pfn, &error, &ext4_iomap_ops);
+ result = dax_iomap_fault(vmf, order, &pfn, &error, ext4_iomap_next);
if (write) {
ext4_journal_stop(handle);
@@ -955,13 +957,13 @@ loff_t ext4_llseek(struct file *file, loff_t offset, int whence)
case SEEK_HOLE:
inode_lock_shared(inode);
offset = iomap_seek_hole(inode, offset,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
offset = iomap_seek_data(inode, offset,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
inode_unlock_shared(inode);
break;
}
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ce99807c5f5b2..6e5a6135381fb 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3391,7 +3391,7 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)
filemap_write_and_wait(mapping);
}
- ret = iomap_bmap(mapping, block, &ext4_iomap_ops);
+ ret = iomap_bmap(mapping, block, ext4_iomap_next);
out:
inode_unlock_shared(inode);
@@ -3771,7 +3771,7 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,
}
-static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned flags, struct iomap *iomap, struct iomap *srcmap)
{
int ret;
@@ -3850,9 +3850,7 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-const struct iomap_ops ext4_iomap_ops = {
- .iomap_begin = ext4_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);
static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
@@ -3905,9 +3903,7 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
return 0;
}
-const struct iomap_ops ext4_iomap_report_ops = {
- .iomap_begin = ext4_iomap_begin_report,
-};
+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);
/*
* For data=journal mode, folio should be marked dirty only when it was
@@ -3944,7 +3940,7 @@ static int ext4_iomap_swap_activate(struct swap_info_struct *sis,
struct file *file, sector_t *span)
{
return iomap_swapfile_activate(sis, file, span,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
}
static const struct address_space_operations ext4_aops = {
@@ -4191,7 +4187,7 @@ static int ext4_block_zero_range(struct inode *inode,
if (IS_DAX(inode)) {
return dax_zero_range(inode, from, length, did_zero,
- &ext4_iomap_ops);
+ ext4_iomap_next);
} else if (ext4_should_journal_data(inode)) {
return ext4_block_journalled_zero_range(inode, from, length,
did_zero);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index a765fda715363..1cb993244b1c4 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -4653,6 +4653,4 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-const struct iomap_ops f2fs_iomap_ops = {
- .iomap_begin = f2fs_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(f2fs_iomap_next, f2fs_iomap_begin);
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 8f3e632f315cc..946a91834aec2 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4216,7 +4216,8 @@ int f2fs_init_post_read_processing(void);
void f2fs_destroy_post_read_processing(void);
int f2fs_init_wq(struct f2fs_sb_info *sbi);
void f2fs_destroy_wq(struct f2fs_sb_info *sbi);
-extern const struct iomap_ops f2fs_iomap_ops;
+int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
/*
* gc.c
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 4b52c56d71f07..74514b1172571 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -4884,7 +4884,7 @@ static ssize_t f2fs_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
* F2FS_DIO_READ counter will be decremented correctly in all cases.
*/
inc_page_count(sbi, F2FS_DIO_READ);
- dio = __iomap_dio_rw(iocb, to, &f2fs_iomap_ops,
+ dio = __iomap_dio_rw(iocb, to, f2fs_iomap_next,
&f2fs_iomap_dio_read_ops, 0, NULL, 0);
if (IS_ERR_OR_NULL(dio)) {
ret = PTR_ERR_OR_ZERO(dio);
@@ -5220,7 +5220,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
dio_flags = 0;
if (pos + count > inode->i_size)
dio_flags |= IOMAP_DIO_FORCE_WAIT;
- dio = __iomap_dio_rw(iocb, from, &f2fs_iomap_ops,
+ dio = __iomap_dio_rw(iocb, from, f2fs_iomap_next,
&f2fs_iomap_dio_write_ops, dio_flags, iocb, 0);
if (IS_ERR_OR_NULL(dio)) {
ret = PTR_ERR_OR_ZERO(dio);
diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c
index 8b53625ac7abd..480cfd2d1fa5e 100644
--- a/fs/fuse/dax.c
+++ b/fs/fuse/dax.c
@@ -653,10 +653,8 @@ static int fuse_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return 0;
}
-static const struct iomap_ops fuse_iomap_ops = {
- .iomap_begin = fuse_iomap_begin,
- .iomap_end = fuse_iomap_end,
-};
+static DEFINE_IOMAP_ITER_NEXT_END(fuse_iomap_next, fuse_iomap_begin,
+ fuse_iomap_end);
static void fuse_wait_dax_page(struct inode *inode)
{
@@ -685,7 +683,7 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
inode_lock_shared(inode);
}
- ret = dax_iomap_rw(iocb, to, &fuse_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, fuse_iomap_next);
inode_unlock_shared(inode);
/* TODO file_accessed(iocb->f_filp) */
@@ -740,7 +738,7 @@ ssize_t fuse_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (file_extending_write(iocb, from))
ret = fuse_dax_direct_write(iocb, from);
else
- ret = dax_iomap_rw(iocb, from, &fuse_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, fuse_iomap_next);
out:
inode_unlock(inode);
@@ -775,7 +773,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, unsigned int order,
* to populate page cache or access memory we are trying to free.
*/
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = dax_iomap_fault(vmf, order, &pfn, &error, &fuse_iomap_ops);
+ ret = dax_iomap_fault(vmf, order, &pfn, &error, fuse_iomap_next);
if ((ret & VM_FAULT_ERROR) && error == -EAGAIN) {
error = 0;
retry = true;
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index ea4a15a7635a6..7a95ac36c4c5b 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -890,9 +890,7 @@ static int fuse_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static const struct iomap_ops fuse_iomap_ops = {
- .iomap_begin = fuse_iomap_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(fuse_iomap_next, fuse_iomap_begin);
struct fuse_fill_read_data {
struct file *file;
@@ -1014,7 +1012,7 @@ static int fuse_read_folio(struct file *file, struct folio *folio)
return -EIO;
}
- iomap_read_folio(&fuse_iomap_ops, &ctx, NULL);
+ iomap_read_folio(fuse_iomap_next, &ctx, NULL);
fuse_invalidate_atime(inode);
return 0;
}
@@ -1115,7 +1113,7 @@ static void fuse_readahead(struct readahead_control *rac)
if (fuse_is_bad(inode))
return;
- iomap_readahead(&fuse_iomap_ops, &ctx, NULL);
+ iomap_readahead(fuse_iomap_next, &ctx, NULL);
}
static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -1531,7 +1529,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from)
* and granular dirty tracking for large folios.
*/
written = iomap_file_buffered_write(iocb, from,
- &fuse_iomap_ops,
+ fuse_iomap_next,
&fuse_iomap_write_ops,
file);
} else {
diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c
index df25d4faca41c..f15e516ebcb5c 100644
--- a/fs/fuse/virtio_fs.c
+++ b/fs/fuse/virtio_fs.c
@@ -1024,8 +1024,7 @@ static void virtio_fs_cleanup_vqs(struct virtio_device *vdev)
}
/* Map a window offset to a page frame number. The window offset will have
- * been produced by .iomap_begin(), which maps a file offset to a window
- * offset.
+ * been produced by .iomap_next(), which maps a file offset to a window offset.
*/
static long virtio_fs_direct_access(struct dax_device *dax_dev, pgoff_t pgoff,
long nr_pages, enum dax_access_mode mode,
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 0a7b8076af3a5..66bc19c011cc0 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -425,7 +425,7 @@ static int gfs2_read_folio(struct file *file, struct folio *folio)
if (!gfs2_is_jdata(ip) ||
(i_blocksize(inode) == PAGE_SIZE && !folio_buffers(folio))) {
- iomap_bio_read_folio(folio, &gfs2_iomap_ops);
+ iomap_bio_read_folio(folio, gfs2_iomap_next);
} else if (gfs2_is_stuffed(ip)) {
error = stuffed_read_folio(ip, folio);
} else {
@@ -500,7 +500,7 @@ static void gfs2_readahead(struct readahead_control *rac)
else if (gfs2_is_jdata(ip))
mpage_readahead(rac, gfs2_block_map);
else
- iomap_bio_readahead(rac, &gfs2_iomap_ops);
+ iomap_bio_readahead(rac, gfs2_iomap_next);
}
/**
@@ -571,7 +571,7 @@ static sector_t gfs2_bmap(struct address_space *mapping, sector_t lblock)
return 0;
if (!gfs2_is_stuffed(ip))
- dblock = iomap_bmap(mapping, lblock, &gfs2_iomap_ops);
+ dblock = iomap_bmap(mapping, lblock, gfs2_iomap_next);
gfs2_glock_dq_uninit(&i_gh);
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 51ac1fd44f78a..f835bd343b6e6 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -1200,10 +1200,7 @@ static int gfs2_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return 0;
}
-const struct iomap_ops gfs2_iomap_ops = {
- .iomap_begin = gfs2_iomap_begin,
- .iomap_end = gfs2_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(gfs2_iomap_next, gfs2_iomap_begin, gfs2_iomap_end);
/**
* gfs2_block_map - Map one or more blocks of an inode to a disk block
@@ -1318,7 +1315,7 @@ static int gfs2_block_zero_range(struct inode *inode, loff_t from, loff_t length
if (from >= inode->i_size)
return 0;
length = min(length, inode->i_size - from);
- return iomap_zero_range(inode, from, length, NULL, &gfs2_iomap_ops,
+ return iomap_zero_range(inode, from, length, NULL, gfs2_iomap_next,
&gfs2_iomap_write_ops, NULL);
}
diff --git a/fs/gfs2/bmap.h b/fs/gfs2/bmap.h
index e3d6efdfd8903..2c2b7ab39259c 100644
--- a/fs/gfs2/bmap.h
+++ b/fs/gfs2/bmap.h
@@ -43,7 +43,8 @@ static inline void gfs2_write_calc_reserv(const struct gfs2_inode *ip,
}
}
-extern const struct iomap_ops gfs2_iomap_ops;
+int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_write_ops gfs2_iomap_write_ops;
extern const struct iomap_writeback_ops gfs2_writeback_ops;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index b8c10de113ba7..ef5f521a46c05 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -844,7 +844,7 @@ static ssize_t gfs2_file_direct_read(struct kiocb *iocb, struct iov_iter *to,
goto out_uninit;
pagefault_disable();
to->nofault = true;
- ret = iomap_dio_rw(iocb, to, &gfs2_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, to, gfs2_iomap_next, NULL,
IOMAP_DIO_PARTIAL, NULL, read);
to->nofault = false;
pagefault_enable();
@@ -910,7 +910,7 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from,
goto out_unlock;
from->nofault = true;
- ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, from, gfs2_iomap_next, NULL,
IOMAP_DIO_PARTIAL, NULL, written);
from->nofault = false;
if (ret <= 0) {
@@ -1062,7 +1062,7 @@ static ssize_t gfs2_file_buffered_write(struct kiocb *iocb,
goto out_unlock;
pagefault_disable();
- ret = iomap_file_buffered_write(iocb, from, &gfs2_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, gfs2_iomap_next,
&gfs2_iomap_write_ops, NULL);
pagefault_enable();
if (ret > 0)
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index 8a77794bbd4af..737a3b6c52683 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -2217,7 +2217,7 @@ static int gfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
goto out;
pagefault_disable();
- ret = iomap_fiemap(inode, fieinfo, start, len, &gfs2_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, gfs2_iomap_next);
pagefault_enable();
gfs2_glock_dq_uninit(&gh);
@@ -2242,7 +2242,7 @@ loff_t gfs2_seek_data(struct file *file, loff_t offset)
inode_lock_shared(inode);
ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
if (!ret)
- ret = iomap_seek_data(inode, offset, &gfs2_iomap_ops);
+ ret = iomap_seek_data(inode, offset, gfs2_iomap_next);
gfs2_glock_dq_uninit(&gh);
inode_unlock_shared(inode);
@@ -2261,7 +2261,7 @@ loff_t gfs2_seek_hole(struct file *file, loff_t offset)
inode_lock_shared(inode);
ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
if (!ret)
- ret = iomap_seek_hole(inode, offset, &gfs2_iomap_ops);
+ ret = iomap_seek_hole(inode, offset, gfs2_iomap_next);
gfs2_glock_dq_uninit(&gh);
inode_unlock_shared(inode);
diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
index 29e876705369b..ef7eb601be09b 100644
--- a/fs/hpfs/file.c
+++ b/fs/hpfs/file.c
@@ -156,9 +156,7 @@ static int hpfs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static const struct iomap_ops hpfs_iomap_ops = {
- .iomap_begin = hpfs_iomap_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(hpfs_iomap_next, hpfs_iomap_begin);
static int hpfs_read_folio(struct file *file, struct folio *folio)
{
@@ -236,7 +234,7 @@ static int hpfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
inode_lock(inode);
len = min_t(u64, len, i_size_read(inode));
- ret = iomap_fiemap(inode, fieinfo, start, len, &hpfs_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, hpfs_iomap_next);
inode_unlock(inode);
return ret;
diff --git a/fs/internal.h b/fs/internal.h
index 355d93f922086..19601f8406dcb 100644
--- a/fs/internal.h
+++ b/fs/internal.h
@@ -8,7 +8,6 @@
struct super_block;
struct file_system_type;
struct iomap;
-struct iomap_ops;
struct linux_binprm;
struct path;
struct mount;
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index f6040199d1140..6f160612d0fd3 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -624,8 +624,8 @@ static int iomap_read_folio_iter(struct iomap_iter *iter,
return 0;
}
-void iomap_read_folio(const struct iomap_ops *ops,
- struct iomap_read_folio_ctx *ctx, void *private)
+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+ void *private)
{
struct folio *folio = ctx->cur_folio;
struct iomap_iter iter = {
@@ -648,7 +648,7 @@ void iomap_read_folio(const struct iomap_ops *ops,
fsverity_readahead(ctx->vi, folio->index,
folio_nr_pages(folio));
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_read_folio_iter(&iter, ctx,
&bytes_submitted);
@@ -687,22 +687,22 @@ static int iomap_readahead_iter(struct iomap_iter *iter,
/**
* iomap_readahead - Attempt to read pages from a file.
- * @ops: The operations vector for the filesystem.
+ * @next: The iomap iteration function for the filesystem.
* @ctx: The ctx used for issuing readahead.
* @private: The filesystem-specific information for issuing iomap_iter.
*
* This function is for filesystems to call to implement their readahead
* address_space operation.
*
- * Context: The @ops callbacks may submit I/O (eg to read the addresses of
+ * Context: The @next callback may submit I/O (eg to read the addresses of
* blocks from disc), and may wait for it. The caller may be trying to
* access a different page, and so sleeping excessively should be avoided.
* It may allocate memory, but should avoid costly allocations. This
* function is called with memalloc_nofs set, so allocations will not cause
* the filesystem to be reentered.
*/
-void iomap_readahead(const struct iomap_ops *ops,
- struct iomap_read_folio_ctx *ctx, void *private)
+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+ void *private)
{
struct readahead_control *rac = ctx->rac;
struct iomap_iter iter = {
@@ -724,7 +724,7 @@ void iomap_readahead(const struct iomap_ops *ops,
fsverity_readahead(ctx->vi, readahead_index(rac),
readahead_count(rac));
- while (iomap_iter(&iter, ops) > 0)
+ while (iomap_iter(&iter, next) > 0)
iter.status = iomap_readahead_iter(&iter, ctx,
&cur_bytes_submitted);
@@ -1268,7 +1268,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
ssize_t
iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private)
{
struct iomap_iter iter = {
@@ -1285,7 +1285,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
if (iocb->ki_flags & IOCB_DONTCACHE)
iter.flags |= IOMAP_DONTCACHE;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_write_iter(&iter, i, write_ops);
if (unlikely(iter.pos == iocb->ki_pos))
@@ -1297,7 +1297,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
EXPORT_SYMBOL_GPL(iomap_file_buffered_write);
int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
- const void *buf, const struct iomap_ops *ops,
+ const void *buf, iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops)
{
int ret;
@@ -1314,7 +1314,7 @@ int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
iov_iter_kvec(&iiter, WRITE, &kvec, 1, length);
- ret = iomap_file_buffered_write(&iocb, &iiter, ops, write_ops, NULL);
+ ret = iomap_file_buffered_write(&iocb, &iiter, next, write_ops, NULL);
if (ret < 0)
return ret;
return ret == length ? 0 : -EIO;
@@ -1586,7 +1586,7 @@ static int iomap_unshare_iter(struct iomap_iter *iter,
int
iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops)
{
struct iomap_iter iter = {
@@ -1601,7 +1601,7 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
return 0;
iter.len = min(len, size - pos);
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_unshare_iter(&iter, write_ops);
return ret;
}
@@ -1710,7 +1710,7 @@ EXPORT_SYMBOL_GPL(iomap_fill_dirty_folios);
int
iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private)
{
struct folio_batch fbatch;
@@ -1735,7 +1735,7 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
*/
range_dirty = filemap_range_needs_writeback(mapping, iter.pos,
iter.pos + iter.len - 1);
- while ((ret = iomap_iter(&iter, ops)) > 0) {
+ while ((ret = iomap_iter(&iter, next)) > 0) {
const struct iomap *srcmap = iomap_iter_srcmap(&iter);
if (!(iter.iomap.flags & IOMAP_F_FOLIO_BATCH) &&
@@ -1761,7 +1761,7 @@ EXPORT_SYMBOL_GPL(iomap_zero_range);
int
iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private)
{
unsigned int blocksize = i_blocksize(inode);
@@ -1770,7 +1770,7 @@ iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
/* Block boundary? Nothing to do */
if (!off)
return 0;
- return iomap_zero_range(inode, pos, blocksize - off, did_zero, ops,
+ return iomap_zero_range(inode, pos, blocksize - off, did_zero, next,
write_ops, private);
}
EXPORT_SYMBOL_GPL(iomap_truncate_page);
@@ -1795,7 +1795,7 @@ static int iomap_folio_mkwrite_iter(struct iomap_iter *iter,
return iomap_iter_advance(iter, length);
}
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,
void *private)
{
struct iomap_iter iter = {
@@ -1812,7 +1812,7 @@ vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
goto out_unlock;
iter.pos = folio_pos(folio);
iter.len = ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_folio_mkwrite_iter(&iter, folio);
if (ret < 0)
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index ca790239e5eb3..36dca8fc8fa5a 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -682,7 +682,7 @@ static int iomap_dio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
*/
struct iomap_dio *
__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before)
{
struct inode *inode = file_inode(iocb->ki_filp);
@@ -806,7 +806,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
inode_dio_begin(inode);
blk_start_plug(&plug);
- while ((ret = iomap_iter(&iomi, ops)) > 0) {
+ while ((ret = iomap_iter(&iomi, next)) > 0) {
iomi.status = iomap_dio_iter(&iomi, dio);
/*
@@ -894,6 +894,21 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
}
EXPORT_SYMBOL_GPL(__iomap_dio_rw);
+ssize_t
+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
+ iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
+ unsigned int dio_flags, void *private, size_t done_before)
+{
+ struct iomap_dio *dio;
+
+ dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
+ done_before);
+ if (IS_ERR_OR_NULL(dio))
+ return PTR_ERR_OR_ZERO(dio);
+ return iomap_dio_complete(dio);
+}
+EXPORT_SYMBOL_GPL(iomap_dio_rw);
+
struct iomap_dio_simple {
struct kiocb *iocb;
size_t size;
@@ -968,211 +983,93 @@ static void iomap_dio_simple_end_io(struct bio *bio)
iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
}
-static inline bool
-iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_dio_ops *dops,
- unsigned int dio_flags, size_t done_before)
+ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,
+ struct iomap_iter *iomi)
{
- struct inode *inode = file_inode(iocb->ki_filp);
- size_t count = iov_iter_count(iter);
-
- if (dops || done_before)
- return false;
- if (iov_iter_rw(iter) != READ)
- return false;
- if (!count)
- return false;
- /*
- * Simple dio is an optimization for small IO. Filter out large IO
- * early as it's the most common case to fail for typical direct IO
- * workloads.
- */
- if (count > inode->i_sb->s_blocksize)
- return false;
- if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
- IOMAP_DIO_BOUNCE))
- return false;
- if (iocb->ki_pos + count > i_size_read(inode))
- return false;
- if (IS_ENCRYPTED(inode))
- return false;
-
- return true;
-}
-
-/*
- * Fast path for small, block-aligned direct I/Os that map to a single
- * contiguous on-disk extent.
- *
- * iomap_dio_simple_supported() enforces the cheap up-front constraints before
- * entering this path.
- *
- * @dops must be NULL: a non-NULL @dops means the caller wants its
- * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
- * allocated from the filesystem-private @dops->bio_set (whose front_pad sizes a
- * filesystem-private wrapper around the bio). The fast path instead allocates
- * from the shared iomap_dio_simple_pool, whose front_pad matches struct
- * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
- * back to __iomap_dio_rw() in that case.
- *
- * @done_before must be zero: a non-zero caller-accumulated residual cannot be
- * carried through a single-bio inline completion.
- *
- * @iter must describe a non-empty READ no larger than the inode block size:
- * writes, zero-length I/O, and larger requests need the generic iomap direct
- * I/O path.
- *
- * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
- * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
- * I/O, or bouncing. The range must also stay within i_size and encrypted
- * inodes must use the generic iomap direct I/O path.
- *
- * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
- * decides the request does not fit the fast path. In that case we proceed to
- * the generic __iomap_dio_rw() slow path. Any other errno is a real result and
- * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
- * caller.
- */
-static ssize_t
-iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, void *private,
- unsigned int dio_flags)
-{
- struct inode *inode = file_inode(iocb->ki_filp);
- size_t count = iov_iter_count(iter);
- bool wait_for_completion = is_sync_kiocb(iocb);
- struct iomap_iter iomi = {
- .inode = inode,
- .pos = iocb->ki_pos,
- .len = count,
- .flags = IOMAP_DIRECT,
- .private = private,
- };
+ gfp_t gfp = (iomi->flags & IOMAP_NOWAIT) ? GFP_NOWAIT : GFP_KERNEL;
struct iomap_dio_simple *sr;
unsigned int alignment;
struct bio *bio;
ssize_t ret;
- if (iocb->ki_flags & IOCB_NOWAIT)
- iomi.flags |= IOMAP_NOWAIT;
-
- ret = kiocb_write_and_wait(iocb, count);
- if (ret)
- return ret;
-
- inode_dio_begin(inode);
-
- ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
- &iomi.iomap, &iomi.srcmap);
- if (ret) {
- inode_dio_end(inode);
- return ret;
- }
-
- if (iomi.iomap.type != IOMAP_MAPPED ||
- iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
- (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
+ if (iomi->iomap.type != IOMAP_MAPPED ||
+ iomi->iomap.offset + iomi->iomap.length < iomi->pos + iomi->len ||
+ (iomi->iomap.flags & IOMAP_F_INTEGRITY)) {
ret = -ENOTBLK;
- goto out_iomap_end;
+ goto out_dio_end;
}
- alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
- if ((iomi.pos | count) & (alignment - 1)) {
+ alignment = iomap_dio_alignment(iomi->inode, iomi->iomap.bdev, 0);
+ if ((iomi->pos | iomi->len) & (alignment - 1)) {
ret = -EINVAL;
- goto out_iomap_end;
+ goto out_dio_end;
}
- if (!wait_for_completion && unlikely(!inode->i_sb->s_dio_done_wq)) {
- ret = sb_init_dio_done_wq(inode->i_sb);
+ if (unlikely(!iomi->inode->i_sb->s_dio_done_wq &&
+ !is_sync_kiocb(iocb))) {
+ ret = sb_init_dio_done_wq(iomi->inode->i_sb);
if (ret < 0)
- goto out_iomap_end;
+ goto out_dio_end;
}
- trace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);
-
- if (user_backed_iter(iter))
- dio_flags |= IOMAP_DIO_USER_BACKED;
+ trace_iomap_dio_rw_begin(iocb, iter, 0, 0);
- bio = bio_alloc_bioset(iomi.iomap.bdev,
+ bio = bio_alloc_bioset(iomi->iomap.bdev,
bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
- REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
+ REQ_OP_READ, gfp, &iomap_dio_simple_pool);
+ if (!bio) {
+ ret = -EAGAIN;
+ goto out_dio_end;
+ }
sr = container_of(bio, struct iomap_dio_simple, bio);
sr->iocb = iocb;
- sr->dio_flags = dio_flags;
+ sr->dio_flags = 0;
- bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
+ bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
bio->bi_ioprio = iocb->ki_ioprio;
ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
if (unlikely(ret))
goto out_bio_put;
- if (bio->bi_iter.bi_size != count) {
+ if (bio->bi_iter.bi_size != iomi->len) {
iov_iter_revert(iter, bio->bi_iter.bi_size);
ret = -ENOTBLK;
goto out_bio_release_pages;
}
sr->size = bio->bi_iter.bi_size;
-
- if (dio_flags & IOMAP_DIO_USER_BACKED)
+ if (user_backed_iter(iter)) {
bio_set_pages_dirty(bio);
+ sr->dio_flags |= IOMAP_DIO_USER_BACKED;
+ }
if (iocb->ki_flags & IOCB_NOWAIT)
bio->bi_opf |= REQ_NOWAIT;
- if ((iocb->ki_flags & IOCB_HIPRI) && !wait_for_completion) {
- bio->bi_opf |= REQ_POLLED;
- WRITE_ONCE(iocb->private, bio);
- }
-
- if (ops->iomap_end)
- ops->iomap_end(inode, iomi.pos, count, count, iomi.flags,
- &iomi.iomap);
- if (!wait_for_completion) {
- bio->bi_end_io = iomap_dio_simple_end_io;
- submit_bio(bio);
- trace_iomap_dio_rw_queued(inode, iomi.pos, count);
- return -EIOCBQUEUED;
+ if (is_sync_kiocb(iocb)) {
+ submit_bio_wait(bio);
+ return iomap_dio_simple_complete(sr);
}
- submit_bio_wait(bio);
- return iomap_dio_simple_complete(sr);
+ if ((iocb->ki_flags & IOCB_HIPRI)) {
+ bio->bi_opf |= REQ_POLLED;
+ WRITE_ONCE(iocb->private, bio);
+ }
+ bio->bi_end_io = iomap_dio_simple_end_io;
+ submit_bio(bio);
+ trace_iomap_dio_rw_queued(iomi->inode, iocb->ki_pos, iomi->len);
+ return -EIOCBQUEUED;
out_bio_release_pages:
bio_release_pages(bio, false);
out_bio_put:
bio_put(bio);
-out_iomap_end:
- if (ops->iomap_end)
- ops->iomap_end(inode, iomi.pos, count, 0, iomi.flags,
- &iomi.iomap);
- inode_dio_end(inode);
+out_dio_end:
+ inode_dio_end(iomi->inode);
return ret;
}
-
-ssize_t
-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
- unsigned int dio_flags, void *private, size_t done_before)
-{
- struct iomap_dio *dio;
- ssize_t ret;
-
- if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
- done_before)) {
- ret = iomap_dio_simple(iocb, iter, ops, private, dio_flags);
- if (ret != -ENOTBLK)
- return ret;
- }
-
- dio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,
- done_before);
- if (IS_ERR_OR_NULL(dio))
- return PTR_ERR_OR_ZERO(dio);
- return iomap_dio_complete(dio);
-}
-EXPORT_SYMBOL_GPL(iomap_dio_rw);
+EXPORT_SYMBOL_GPL(__iomap_dio_read_simple);
static int __init iomap_dio_init(void)
{
diff --git a/fs/iomap/fiemap.c b/fs/iomap/fiemap.c
index d11dadff82865..c79c13cc617bd 100644
--- a/fs/iomap/fiemap.c
+++ b/fs/iomap/fiemap.c
@@ -56,7 +56,7 @@ static int iomap_fiemap_iter(struct iomap_iter *iter,
}
int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
- u64 start, u64 len, const struct iomap_ops *ops)
+ u64 start, u64 len, iomap_iter_next_fn next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -73,7 +73,7 @@ int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
if (ret)
return ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_fiemap_iter(&iter, fi, &prev);
if (prev.type != IOMAP_HOLE) {
@@ -92,7 +92,7 @@ EXPORT_SYMBOL_GPL(iomap_fiemap);
/* legacy ->bmap interface. 0 is the error return (!) */
sector_t
iomap_bmap(struct address_space *mapping, sector_t bno,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_iter iter = {
.inode = mapping->host,
@@ -107,7 +107,7 @@ iomap_bmap(struct address_space *mapping, sector_t bno,
return 0;
bno = 0;
- while ((ret = iomap_iter(&iter, ops)) > 0) {
+ while ((ret = iomap_iter(&iter, next)) > 0) {
if (iter.iomap.type == IOMAP_MAPPED)
bno = iomap_sector(&iter.iomap, iter.pos) >> blkshift;
/* leave iter.status unset to abort loop */
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index e4a29829591a7..89071b0a8b9c6 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -6,15 +6,6 @@
#include <linux/iomap.h>
#include "trace.h"
-static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)
-{
- if (iter->iomap.flags & IOMAP_F_FOLIO_BATCH) {
- folio_batch_release(iter->fbatch);
- folio_batch_reinit(iter->fbatch);
- iter->iomap.flags &= ~IOMAP_F_FOLIO_BATCH;
- }
-}
-
/* Advance the current iterator position and decrement the remaining length */
int iomap_iter_advance(struct iomap_iter *iter, u64 count)
{
@@ -40,51 +31,28 @@ static inline void iomap_iter_done(struct iomap_iter *iter)
}
/**
- * iomap_iter - iterate over a ranges in a file
- * @iter: iteration structue
- * @ops: iomap ops provided by the file system
+ * iomap_iter_continue - decide whether iteration should continue
+ * @iter: iteration structure
+ * @iomap: the mapping that was just processed
+ * @srcmap: the source mapping that was just processed
*
- * Iterate over filesystem-provided space mappings for the provided file range.
+ * Helper normally called via iomap_iter_next(). Called after the previous
+ * mapping has been finished to determine whether there is more of the file
+ * range left to process.
*
- * This function handles cleanup of resources acquired for iteration when the
- * filesystem indicates there are no more space mappings, which means that this
- * function must be called in a loop that continues as long it returns a
- * positive value. If 0 or a negative value is returned, the caller must not
- * return to the loop body. Within a loop body, there are two ways to break out
- * of the loop body: leave @iter.status unchanged, or set it to a negative
- * errno.
+ * Returns 1 if there is more work to do, in which case @iomap and @srcmap are
+ * cleared so the caller can produce the next mapping; zero if the range is
+ * fully consumed; or a negative errno on error. Any folio batch attached to
+ * the mapping is released before returning.
*/
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap, int ret)
{
- bool stale = iter->iomap.flags & IOMAP_F_STALE;
- ssize_t advanced;
- u64 olen;
- int ret;
-
- trace_iomap_iter(iter, ops, _RET_IP_);
-
- if (!iter->iomap.length)
- goto begin;
-
- /*
- * Calculate how far the iter was advanced and the original length bytes
- * for ->iomap_end().
- */
- advanced = iter->pos - iter->iter_start_pos;
- olen = iter->len + advanced;
+ const bool stale = iomap->flags & IOMAP_F_STALE;
+ const ssize_t advanced = iter->pos - iter->iter_start_pos;
- if (ops->iomap_end) {
- ret = ops->iomap_end(iter->inode, iter->iter_start_pos,
- iomap_length_trim(iter, iter->iter_start_pos,
- olen),
- advanced, iter->flags, &iter->iomap);
- if (ret < 0 && !advanced)
- return ret;
- }
-
- /* detect old return semantics where this would advance */
- if (WARN_ON_ONCE(iter->status > 0))
- iter->status = -EIO;
+ if (ret < 0 && !advanced)
+ return ret;
/*
* Use iter->len to determine whether to continue onto the next mapping.
@@ -92,25 +60,57 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
* advanced at all (i.e. no work was done for some reason) unless the
* mapping has been marked stale and needs to be reprocessed.
*/
- if (iter->status < 0)
+ if (WARN_ON_ONCE(iter->status > 0))
+ /* detect old return semantics where this would advance */
+ ret = -EIO;
+ else if (iter->status < 0)
ret = iter->status;
else if (iter->len == 0 || (!advanced && !stale))
ret = 0;
else
ret = 1;
- iomap_iter_clean_fbatch(iter);
- iter->status = 0;
+
+ if (iomap->flags & IOMAP_F_FOLIO_BATCH) {
+ folio_batch_release(iter->fbatch);
+ folio_batch_reinit(iter->fbatch);
+ iomap->flags &= ~IOMAP_F_FOLIO_BATCH;
+ }
+
if (ret <= 0)
return ret;
- memset(&iter->iomap, 0, sizeof(iter->iomap));
- memset(&iter->srcmap, 0, sizeof(iter->srcmap));
+ memset(iomap, 0, sizeof(*iomap));
+ memset(srcmap, 0, sizeof(*srcmap));
-begin:
- ret = ops->iomap_begin(iter->inode, iter->pos, iter->len, iter->flags,
- &iter->iomap, &iter->srcmap);
- if (ret < 0)
- return ret;
- iomap_iter_done(iter);
- return 1;
+ return ret;
+}
+EXPORT_SYMBOL_GPL(iomap_iter_continue);
+
+/**
+ * iomap_iter - iterate over ranges in a file
+ * @iter: iteration structure
+ * @next: the iomap iteration function for the filesystem
+ *
+ * Iterate over filesystem-provided space mappings for the provided file range.
+ *
+ * This function handles cleanup of resources acquired for iteration when the
+ * filesystem indicates there are no more space mappings, which means that this
+ * function must be called in a loop that continues as long it returns a
+ * positive value. If 0 or a negative value is returned, the caller must not
+ * return to the loop body. Within a loop body, there are two ways to break out
+ * of the loop body: leave @iter.status unchanged, or set it to a negative
+ * errno.
+ */
+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next)
+{
+ int ret;
+
+ trace_iomap_iter(iter, next, _RET_IP_);
+
+ ret = next(iter, &iter->iomap, &iter->srcmap);
+ iter->status = 0;
+ if (ret > 0)
+ iomap_iter_done(iter);
+
+ return ret;
}
diff --git a/fs/iomap/seek.c b/fs/iomap/seek.c
index 6cbc587c93dab..6fd3173d25f51 100644
--- a/fs/iomap/seek.c
+++ b/fs/iomap/seek.c
@@ -27,7 +27,7 @@ static int iomap_seek_hole_iter(struct iomap_iter *iter,
}
loff_t
-iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_hole(struct inode *inode, loff_t pos, iomap_iter_next_fn next)
{
loff_t size = i_size_read(inode);
struct iomap_iter iter = {
@@ -42,7 +42,7 @@ iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
return -ENXIO;
iter.len = size - pos;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_seek_hole_iter(&iter, &pos);
if (ret < 0)
return ret;
@@ -73,7 +73,7 @@ static int iomap_seek_data_iter(struct iomap_iter *iter,
}
loff_t
-iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_data(struct inode *inode, loff_t pos, iomap_iter_next_fn next)
{
loff_t size = i_size_read(inode);
struct iomap_iter iter = {
@@ -88,7 +88,7 @@ iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
return -ENXIO;
iter.len = size - pos;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_seek_data_iter(&iter, &pos);
if (ret < 0)
return ret;
diff --git a/fs/iomap/swapfile.c b/fs/iomap/swapfile.c
index 0db77c449467a..c59f67a7de097 100644
--- a/fs/iomap/swapfile.c
+++ b/fs/iomap/swapfile.c
@@ -139,7 +139,7 @@ static int iomap_swapfile_iter(struct iomap_iter *iter,
*/
int iomap_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *pagespan,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct inode *inode = swap_file->f_mapping->host;
struct iomap_iter iter = {
@@ -163,7 +163,7 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,
if (ret)
return ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, next)) > 0)
iter.status = iomap_swapfile_iter(&iter, &iter.iomap, &isi);
if (ret < 0)
return ret;
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47d..d36a77b854aa5 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -218,9 +218,9 @@ TRACE_EVENT(iomap_add_to_ioend,
);
TRACE_EVENT(iomap_iter,
- TP_PROTO(struct iomap_iter *iter, const void *ops,
+ TP_PROTO(struct iomap_iter *iter, const void *next,
unsigned long caller),
- TP_ARGS(iter, ops, caller),
+ TP_ARGS(iter, next, caller),
TP_STRUCT__entry(
__field(dev_t, dev)
__field(u64, ino)
@@ -228,7 +228,7 @@ TRACE_EVENT(iomap_iter,
__field(u64, length)
__field(int, status)
__field(unsigned int, flags)
- __field(const void *, ops)
+ __field(const void *, next)
__field(unsigned long, caller)
),
TP_fast_assign(
@@ -238,10 +238,10 @@ TRACE_EVENT(iomap_iter,
__entry->length = iomap_length(iter);
__entry->status = iter->status;
__entry->flags = iter->flags;
- __entry->ops = ops;
+ __entry->next = next;
__entry->caller = caller;
),
- TP_printk("dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) ops %ps caller %pS",
+ TP_printk("dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) next %ps caller %pS",
MAJOR(__entry->dev), MINOR(__entry->dev),
__entry->ino,
__entry->pos,
@@ -249,7 +249,7 @@ TRACE_EVENT(iomap_iter,
__entry->status,
__print_flags(__entry->flags, "|", IOMAP_FLAGS_STRINGS),
__entry->flags,
- __entry->ops,
+ __entry->next,
(void *)__entry->caller)
);
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
index 1fbf832ad1654..43ad597ed491e 100644
--- a/fs/ntfs/aops.c
+++ b/fs/ntfs/aops.c
@@ -97,7 +97,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
return ntfs_read_compressed_block(folio);
}
- iomap_read_folio(&ntfs_read_iomap_ops, &ctx, NULL);
+ iomap_read_folio(ntfs_read_iomap_next, &ctx, NULL);
return 0;
}
@@ -238,7 +238,7 @@ static void ntfs_readahead(struct readahead_control *rac)
*/
if (!NInoNonResident(ni) || NInoCompressed(ni))
return;
- iomap_readahead(&ntfs_read_iomap_ops, &ctx, NULL);
+ iomap_readahead(ntfs_read_iomap_next, &ctx, NULL);
}
static int ntfs_writepages(struct address_space *mapping,
@@ -274,7 +274,7 @@ static int ntfs_swap_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *span)
{
return iomap_swapfile_activate(sis, swap_file, span,
- &ntfs_read_iomap_ops);
+ ntfs_read_iomap_next);
}
const struct address_space_operations ntfs_aops = {
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index 6a7b638e523d9..a4f99128b46c5 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -281,7 +281,7 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr)
round_up(old_size, PAGE_SIZE) - old_size,
attr->ia_size - old_size);
err = iomap_zero_range(vi, old_size, len,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
}
@@ -417,12 +417,12 @@ static loff_t ntfs_file_llseek(struct file *file, loff_t offset, int whence)
switch (whence) {
case SEEK_HOLE:
inode_lock_shared(inode);
- offset = iomap_seek_hole(inode, offset, &ntfs_seek_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, ntfs_seek_iomap_next);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
- offset = iomap_seek_data(inode, offset, &ntfs_seek_iomap_ops);
+ offset = iomap_seek_data(inode, offset, ntfs_seek_iomap_next);
inode_unlock_shared(inode);
break;
default:
@@ -458,7 +458,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
}
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, to, &ntfs_read_iomap_ops, NULL, 0,
+ ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
NULL, 0);
} else {
ret = generic_file_read_iter(iocb, to);
@@ -496,7 +496,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
ssize_t ret;
- ret = iomap_dio_rw(iocb, from, &ntfs_dio_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, ntfs_dio_iomap_next,
&ntfs_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -511,7 +511,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
offset = iocb->ki_pos;
iocb->ki_flags &= ~IOCB_DIRECT;
written = iomap_file_buffered_write(iocb, from,
- &ntfs_write_iomap_ops, &ntfs_iomap_folio_ops,
+ ntfs_write_iomap_next, &ntfs_iomap_folio_ops,
NULL);
if (written < 0) {
ret = written;
@@ -594,7 +594,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (NInoNonResident(ni) && iocb->ki_flags & IOCB_DIRECT)
ret = ntfs_dio_write_iter(iocb, from);
else
- ret = iomap_file_buffered_write(iocb, from, &ntfs_write_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, ntfs_write_iomap_next,
&ntfs_iomap_folio_ops, NULL);
out:
if (ret < 0 && ret != -EIOCBQUEUED) {
@@ -623,7 +623,7 @@ static vm_fault_t ntfs_filemap_page_mkwrite(struct vm_fault *vmf)
sb_start_pagefault(inode->i_sb);
file_update_time(vmf->vma->vm_file);
- ret = iomap_page_mkwrite(vmf, &ntfs_page_mkwrite_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, ntfs_page_mkwrite_iomap_next, NULL);
sb_end_pagefault(inode->i_sb);
return ret;
}
@@ -670,7 +670,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc)
static int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
{
- return iomap_fiemap(inode, fieinfo, start, len, &ntfs_read_iomap_ops);
+ return iomap_fiemap(inode, fieinfo, start, len, ntfs_read_iomap_next);
}
static const char *ntfs_get_link(struct dentry *dentry, struct inode *inode,
@@ -911,7 +911,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
ntfs_cluster_to_bytes(vol, start_vcn + 1),
end_offset);
err = iomap_zero_range(vi, offset, to - offset,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0)
goto out;
@@ -927,7 +927,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
from = ntfs_cluster_to_bytes(vol, end_vcn - 1);
if (from < ni->initialized_size) {
err = iomap_zero_range(vi, from, end_offset - from,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0)
goto out;
@@ -1131,7 +1131,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le
round_up(old_size, PAGE_SIZE) - old_size,
offset - old_size);
err = iomap_zero_range(vi, old_size, len, NULL,
- &ntfs_seek_iomap_ops,
+ ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
}
NInoSetFileNameDirty(ni);
diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c
index c2715521e5628..05132d92e87b2 100644
--- a/fs/ntfs/inode.c
+++ b/fs/ntfs/inode.c
@@ -2415,7 +2415,7 @@ int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset,
if (!NInoCompressed(ni) && old_init_size < offset) {
err = iomap_zero_range(vi, old_init_size,
offset - old_init_size,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
return err;
diff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c
index 52eecf5cb256a..271902e692003 100644
--- a/fs/ntfs/iomap.c
+++ b/fs/ntfs/iomap.c
@@ -277,9 +277,7 @@ static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t leng
srcmap, true);
}
-const struct iomap_ops ntfs_read_iomap_ops = {
- .iomap_begin = ntfs_read_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin);
/*
* Check that the cached iomap still matches the NTFS runlist before
@@ -329,14 +327,10 @@ static int ntfs_zero_read_iomap_end(struct inode *inode, loff_t pos, loff_t leng
return written;
}
-static const struct iomap_ops ntfs_zero_read_iomap_ops = {
- .iomap_begin = ntfs_seek_iomap_begin,
- .iomap_end = ntfs_zero_read_iomap_end,
-};
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_zero_read_iomap_next,
+ ntfs_seek_iomap_begin, ntfs_zero_read_iomap_end);
-const struct iomap_ops ntfs_seek_iomap_ops = {
- .iomap_begin = ntfs_seek_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(ntfs_seek_iomap_next, ntfs_seek_iomap_begin);
int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length)
{
@@ -355,7 +349,7 @@ static int ntfs_zero_range(struct inode *inode, loff_t offset, loff_t length)
return iomap_zero_range(inode,
offset, length,
NULL,
- &ntfs_zero_read_iomap_ops,
+ ntfs_zero_read_iomap_next,
&ntfs_zero_iomap_folio_ops,
NULL);
}
@@ -764,10 +758,8 @@ static int ntfs_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return written;
}
-const struct iomap_ops ntfs_write_iomap_ops = {
- .iomap_begin = ntfs_write_iomap_begin,
- .iomap_end = ntfs_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_write_iomap_next, ntfs_write_iomap_begin,
+ ntfs_write_iomap_end);
static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
@@ -777,10 +769,8 @@ static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
NTFS_IOMAP_FLAGS_MKWRITE);
}
-const struct iomap_ops ntfs_page_mkwrite_iomap_ops = {
- .iomap_begin = ntfs_page_mkwrite_iomap_begin,
- .iomap_end = ntfs_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_page_mkwrite_iomap_next,
+ ntfs_page_mkwrite_iomap_begin, ntfs_write_iomap_end);
static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
@@ -790,10 +780,8 @@ static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
NTFS_IOMAP_FLAGS_DIO);
}
-const struct iomap_ops ntfs_dio_iomap_ops = {
- .iomap_begin = ntfs_dio_iomap_begin,
- .iomap_end = ntfs_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_dio_iomap_next, ntfs_dio_iomap_begin,
+ ntfs_write_iomap_end);
static ssize_t ntfs_writeback_range(struct iomap_writepage_ctx *wpc,
struct folio *folio, u64 offset, unsigned int len, u64 end_pos)
diff --git a/fs/ntfs/iomap.h b/fs/ntfs/iomap.h
index 3abc1d493e918..69443de1fefd2 100644
--- a/fs/ntfs/iomap.h
+++ b/fs/ntfs/iomap.h
@@ -12,11 +12,16 @@
#include "volume.h"
#include "inode.h"
-extern const struct iomap_ops ntfs_write_iomap_ops;
-extern const struct iomap_ops ntfs_read_iomap_ops;
-extern const struct iomap_ops ntfs_seek_iomap_ops;
-extern const struct iomap_ops ntfs_page_mkwrite_iomap_ops;
-extern const struct iomap_ops ntfs_dio_iomap_ops;
+int ntfs_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int ntfs_dio_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_writeback_ops ntfs_writeback_ops;
extern const struct iomap_write_ops ntfs_iomap_folio_ops;
extern int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length);
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index d601f088618c0..55844b42920a1 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -315,7 +315,7 @@ static int ntfs_extend_initialized_size(struct file *file,
}
err = iomap_zero_range(inode, valid, new_valid - valid, NULL,
- &ntfs_iomap_ops, &ntfs_iomap_folio_ops, NULL);
+ ntfs_iomap_next, &ntfs_iomap_folio_ops, NULL);
if (err) {
ni->i_valid = valid;
ntfs_inode_warn(inode,
@@ -554,7 +554,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
/* Zero head of punch. */
if (tmp > from) {
err = iomap_zero_range(inode, from, tmp - from, NULL,
- &ntfs_iomap_ops,
+ ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
goto out;
@@ -572,7 +572,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
/* Zero tail of punch. */
if (vbo < end_a && end_a < end) {
err = iomap_zero_range(inode, end_a, end - end_a, NULL,
- &ntfs_iomap_ops,
+ ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
goto out;
@@ -872,7 +872,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
goto out;
}
- err = iomap_dio_rw(iocb, iter, &ntfs_iomap_ops, NULL, dio_flags,
+ err = iomap_dio_rw(iocb, iter, ntfs_iomap_next, NULL, dio_flags,
NULL, 0);
if (err <= 0)
@@ -1286,7 +1286,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
!ntfs_should_use_dio(iocb, from)) {
iocb->ki_flags &= ~IOCB_DIRECT;
- ret = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
inode_unlock(inode);
@@ -1303,7 +1303,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
goto out;
}
- ret = iomap_dio_rw(iocb, from, &ntfs_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, from, ntfs_iomap_next, NULL,
IOMAP_DIO_FORCE_WAIT, NULL, 0);
if (ret == -ENOTBLK) {
@@ -1316,7 +1316,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
vbo = iocb->ki_pos;
iocb->ki_flags &= ~IOCB_DIRECT;
- err = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+ err = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0) {
ret = err;
@@ -1465,7 +1465,7 @@ int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
inode_lock_shared(inode);
- err = iomap_fiemap(inode, fieinfo, start, len, &ntfs_iomap_ops);
+ err = iomap_fiemap(inode, fieinfo, start, len, ntfs_iomap_next);
inode_unlock_shared(inode);
return err;
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index c43101cc064d5..edd4fa499a66f 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -576,7 +576,7 @@ static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)
ni_allocate_da_blocks(ni);
}
- return iomap_bmap(mapping, block, &ntfs_iomap_ops);
+ return iomap_bmap(mapping, block, ntfs_iomap_next);
}
static void ntfs_iomap_read_end_io(struct bio *bio)
@@ -649,7 +649,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
return err;
}
- iomap_read_folio(&ntfs_iomap_ops, &ctx, NULL);
+ iomap_read_folio(ntfs_iomap_next, &ctx, NULL);
return 0;
}
@@ -673,7 +673,7 @@ static void ntfs_readahead(struct readahead_control *rac)
return;
}
- iomap_readahead(&ntfs_iomap_ops, &ctx, NULL);
+ iomap_readahead(ntfs_iomap_next, &ctx, NULL);
}
int ntfs_set_size(struct inode *inode, u64 new_size)
@@ -2101,10 +2101,7 @@ const struct address_space_operations ntfs_aops_cmpr = {
.invalidate_folio = iomap_invalidate_folio,
};
-const struct iomap_ops ntfs_iomap_ops = {
- .iomap_begin = ntfs_iomap_begin,
- .iomap_end = ntfs_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_iomap_next, ntfs_iomap_begin, ntfs_iomap_end);
const struct iomap_write_ops ntfs_iomap_folio_ops = {
.put_folio = ntfs_iomap_put_folio,
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index d98d7e474476c..e00dae3ce7009 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -785,7 +785,8 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
int ntfs_link_inode(struct inode *inode, struct dentry *dentry);
int ntfs_unlink_inode(struct inode *dir, const struct dentry *dentry);
void ntfs_evict_inode(struct inode *inode);
-extern const struct iomap_ops ntfs_iomap_ops;
+int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_write_ops ntfs_iomap_folio_ops;
extern const struct inode_operations ntfs_link_inode_operations;
extern const struct address_space_operations ntfs_aops;
diff --git a/fs/remap_range.c b/fs/remap_range.c
index 26afbbbfb10c2..eab94d42654d4 100644
--- a/fs/remap_range.c
+++ b/fs/remap_range.c
@@ -277,7 +277,7 @@ int
__generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *dax_read_ops)
+ iomap_iter_next_fn dax_read_next)
{
struct inode *inode_in = file_inode(file_in);
struct inode *inode_out = file_inode(file_out);
@@ -340,10 +340,10 @@ __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
if (!IS_DAX(inode_in))
ret = vfs_dedupe_file_range_compare(file_in, pos_in,
file_out, pos_out, *len, &is_same);
- else if (dax_read_ops)
+ else if (dax_read_next)
ret = dax_dedupe_file_range_compare(inode_in, pos_in,
inode_out, pos_out, *len, &is_same,
- dax_read_ops);
+ dax_read_next);
else
return -EINVAL;
if (ret)
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 2a0c54256e93f..91480cb6a4d85 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -752,7 +752,7 @@ xfs_vm_bmap(
*/
if (xfs_is_cow_inode(ip) || XFS_IS_REALTIME_INODE(ip))
return 0;
- return iomap_bmap(mapping, block, &xfs_read_iomap_ops);
+ return iomap_bmap(mapping, block, xfs_read_iomap_next);
}
static void
@@ -793,7 +793,7 @@ xfs_vm_read_folio(
struct iomap_read_folio_ctx ctx = { .cur_folio = folio };
ctx.ops = xfs_get_iomap_read_ops(folio->mapping);
- iomap_read_folio(&xfs_read_iomap_ops, &ctx, NULL);
+ iomap_read_folio(xfs_read_iomap_next, &ctx, NULL);
return 0;
}
@@ -804,7 +804,7 @@ xfs_vm_readahead(
struct iomap_read_folio_ctx ctx = { .rac = rac };
ctx.ops = xfs_get_iomap_read_ops(rac->mapping),
- iomap_readahead(&xfs_read_iomap_ops, &ctx, NULL);
+ iomap_readahead(xfs_read_iomap_next, &ctx, NULL);
}
static int
@@ -850,7 +850,7 @@ xfs_vm_swap_activate(
sis->bdev = xfs_inode_buftarg(ip)->bt_bdev;
return iomap_swapfile_activate(sis, swap_file, span,
- &xfs_read_iomap_ops);
+ xfs_read_iomap_next);
}
const struct address_space_operations xfs_address_space_operations = {
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 845a97c9b0630..877f9024d333d 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,8 +251,6 @@ xfs_file_dio_read(
struct iov_iter *to)
{
struct xfs_inode *ip = XFS_I(file_inode(iocb->ki_filp));
- unsigned int dio_flags = 0;
- const struct iomap_dio_ops *dio_ops = NULL;
ssize_t ret;
trace_xfs_file_direct_read(iocb, to);
@@ -266,11 +264,15 @@ xfs_file_dio_read(
if (ret)
return ret;
if (mapping_stable_writes(iocb->ki_filp->f_mapping)) {
- dio_ops = &xfs_dio_read_bounce_ops;
- dio_flags |= IOMAP_DIO_BOUNCE;
+ ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next,
+ &xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,
+ NULL, 0);
+ } else {
+ ret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);
+ if (ret == -ENOTBLK)
+ ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, NULL,
+ 0, NULL, 0);
}
- ret = iomap_dio_rw(iocb, to, &xfs_read_iomap_ops, dio_ops, dio_flags,
- NULL, 0);
xfs_iunlock(ip, XFS_IOLOCK_SHARED);
return ret;
@@ -292,7 +294,7 @@ xfs_file_dax_read(
ret = xfs_ilock_iocb(iocb, XFS_IOLOCK_SHARED);
if (ret)
return ret;
- ret = dax_iomap_rw(iocb, to, &xfs_read_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, xfs_read_iomap_next);
xfs_iunlock(ip, XFS_IOLOCK_SHARED);
file_accessed(iocb->ki_filp);
@@ -742,7 +744,7 @@ xfs_file_dio_write_aligned(
struct xfs_inode *ip,
struct kiocb *iocb,
struct iov_iter *from,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_dio_ops *dops,
struct xfs_zone_alloc_ctx *ac)
{
@@ -777,7 +779,7 @@ xfs_file_dio_write_aligned(
if (mapping_stable_writes(iocb->ki_filp->f_mapping))
dio_flags |= IOMAP_DIO_BOUNCE;
trace_xfs_file_direct_write(iocb, from);
- ret = iomap_dio_rw(iocb, from, ops, dops, dio_flags, ac, 0);
+ ret = iomap_dio_rw(iocb, from, next, dops, dio_flags, ac, 0);
out_unlock:
xfs_iunlock(ip, iolock);
return ret;
@@ -799,7 +801,7 @@ xfs_file_dio_write_zoned(
if (ret < 0)
return ret;
ret = xfs_file_dio_write_aligned(ip, iocb, from,
- &xfs_zoned_direct_write_iomap_ops,
+ xfs_zoned_direct_write_iomap_next,
&xfs_dio_zoned_write_ops, &ac);
xfs_zoned_space_unreserve(ip->i_mount, &ac);
return ret;
@@ -824,16 +826,16 @@ xfs_file_dio_write_atomic(
unsigned int iolock = XFS_IOLOCK_SHARED;
ssize_t ret, ocount = iov_iter_count(from);
unsigned int dio_flags = 0;
- const struct iomap_ops *dops;
+ iomap_iter_next_fn next;
/*
* HW offload should be faster, so try that first if it is already
* known that the write length is not too large.
*/
if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
- dops = &xfs_atomic_write_cow_iomap_ops;
+ next = xfs_atomic_write_cow_iomap_next;
else
- dops = &xfs_direct_write_iomap_ops;
+ next = xfs_direct_write_iomap_next;
retry:
ret = xfs_ilock_iocb_for_write(iocb, &iolock);
@@ -853,18 +855,18 @@ xfs_file_dio_write_atomic(
trace_xfs_file_direct_write(iocb, from);
if (mapping_stable_writes(iocb->ki_filp->f_mapping))
dio_flags |= IOMAP_DIO_BOUNCE;
- ret = iomap_dio_rw(iocb, from, dops, &xfs_dio_write_ops, dio_flags,
+ ret = iomap_dio_rw(iocb, from, next, &xfs_dio_write_ops, dio_flags,
NULL, 0);
/*
- * The retry mechanism is based on the ->iomap_begin method returning
+ * The retry mechanism is based on the ->iomap_next method returning
* -ENOPROTOOPT, which would be when the REQ_ATOMIC-based write is not
- * possible. The REQ_ATOMIC-based method typically not be possible if
+ * possible. The REQ_ATOMIC-based method is typically not possible if
* the write spans multiple extents or the disk blocks are misaligned.
*/
- if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
+ if (ret == -ENOPROTOOPT && next == xfs_direct_write_iomap_next) {
xfs_iunlock(ip, iolock);
- dops = &xfs_atomic_write_cow_iomap_ops;
+ next = xfs_atomic_write_cow_iomap_next;
goto retry;
}
@@ -947,7 +949,7 @@ xfs_file_dio_write_unaligned(
flags |= IOMAP_DIO_BOUNCE;
trace_xfs_file_direct_write(iocb, from);
- ret = iomap_dio_rw(iocb, from, &xfs_direct_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, xfs_direct_write_iomap_next,
&xfs_dio_write_ops, flags, NULL, 0);
/*
@@ -987,7 +989,7 @@ xfs_file_dio_write(
if (iocb->ki_flags & IOCB_ATOMIC)
return xfs_file_dio_write_atomic(ip, iocb, from);
return xfs_file_dio_write_aligned(ip, iocb, from,
- &xfs_direct_write_iomap_ops, &xfs_dio_write_ops, NULL);
+ xfs_direct_write_iomap_next, &xfs_dio_write_ops, NULL);
}
static noinline ssize_t
@@ -1011,7 +1013,7 @@ xfs_file_dax_write(
pos = iocb->ki_pos;
trace_xfs_file_dax_write(iocb, from);
- ret = dax_iomap_rw(iocb, from, &xfs_dax_write_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, xfs_dax_write_iomap_next);
if (ret > 0 && iocb->ki_pos > i_size_read(inode)) {
i_size_write(inode, iocb->ki_pos);
error = xfs_setfilesize(ip, pos, ret);
@@ -1054,7 +1056,7 @@ xfs_file_buffered_write(
trace_xfs_file_buffered_write(iocb, from);
ret = iomap_file_buffered_write(iocb, from,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
NULL);
/*
@@ -1135,7 +1137,7 @@ xfs_file_buffered_write_zoned(
retry:
trace_xfs_file_buffered_write(iocb, from);
ret = iomap_file_buffered_write(iocb, from,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
&ac);
if (ret == -ENOSPC && !cleared_space) {
/*
@@ -1856,10 +1858,10 @@ xfs_file_llseek(
default:
return generic_file_llseek(file, offset, whence);
case SEEK_HOLE:
- offset = iomap_seek_hole(inode, offset, &xfs_seek_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, xfs_seek_iomap_next);
break;
case SEEK_DATA:
- offset = iomap_seek_data(inode, offset, &xfs_seek_iomap_ops);
+ offset = iomap_seek_data(inode, offset, xfs_seek_iomap_next);
break;
}
@@ -1883,8 +1885,8 @@ xfs_dax_fault_locked(
}
ret = dax_iomap_fault(vmf, order, &pfn, NULL,
(write_fault && !vmf->cow_page) ?
- &xfs_dax_write_iomap_ops :
- &xfs_read_iomap_ops);
+ xfs_dax_write_iomap_next :
+ xfs_read_iomap_next);
if (ret & VM_FAULT_NEEDDSYNC)
ret = dax_finish_sync_fault(vmf, order, pfn);
return ret;
@@ -1948,7 +1950,7 @@ __xfs_write_fault(
if (IS_DAX(inode))
ret = xfs_dax_fault_locked(vmf, order, true);
else
- ret = iomap_page_mkwrite(vmf, &xfs_buffered_write_iomap_ops,
+ ret = iomap_page_mkwrite(vmf, xfs_buffered_write_iomap_next,
ac);
xfs_iunlock(ip, lock_mode);
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 225c3de88d03b..771580e01c0a5 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -1037,9 +1037,8 @@ xfs_direct_write_iomap_begin(
return error;
}
-const struct iomap_ops xfs_direct_write_iomap_ops = {
- .iomap_begin = xfs_direct_write_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_direct_write_iomap_next,
+ xfs_direct_write_iomap_begin);
#ifdef CONFIG_XFS_RT
/*
@@ -1089,9 +1088,9 @@ xfs_zoned_direct_write_iomap_begin(
return 0;
}
-const struct iomap_ops xfs_zoned_direct_write_iomap_ops = {
- .iomap_begin = xfs_zoned_direct_write_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_zoned_direct_write_iomap_next,
+ xfs_zoned_direct_write_iomap_begin);
+
#endif /* CONFIG_XFS_RT */
#ifdef DEBUG
@@ -1274,9 +1273,8 @@ xfs_atomic_write_cow_iomap_begin(
return error;
}
-const struct iomap_ops xfs_atomic_write_cow_iomap_ops = {
- .iomap_begin = xfs_atomic_write_cow_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_atomic_write_cow_iomap_next,
+ xfs_atomic_write_cow_iomap_begin);
static int
xfs_dax_write_iomap_end(
@@ -1298,10 +1296,8 @@ xfs_dax_write_iomap_end(
return xfs_reflink_end_cow(ip, pos, written);
}
-const struct iomap_ops xfs_dax_write_iomap_ops = {
- .iomap_begin = xfs_direct_write_iomap_begin,
- .iomap_end = xfs_dax_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(xfs_dax_write_iomap_next,
+ xfs_direct_write_iomap_begin, xfs_dax_write_iomap_end);
/*
* Convert a hole to a delayed allocation.
@@ -2168,12 +2164,10 @@ xfs_buffered_write_iomap_end(
return 0;
}
-const struct iomap_ops xfs_buffered_write_iomap_ops = {
- .iomap_begin = xfs_buffered_write_iomap_begin,
- .iomap_end = xfs_buffered_write_iomap_end,
-};
+DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
+ xfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);
-static int
+int
xfs_read_iomap_begin(
struct inode *inode,
loff_t offset,
@@ -2214,9 +2208,7 @@ xfs_read_iomap_begin(
shared ? IOMAP_F_SHARED : 0, seq);
}
-const struct iomap_ops xfs_read_iomap_ops = {
- .iomap_begin = xfs_read_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_read_iomap_next, xfs_read_iomap_begin);
static int
xfs_seek_iomap_begin(
@@ -2302,9 +2294,7 @@ xfs_seek_iomap_begin(
return error;
}
-const struct iomap_ops xfs_seek_iomap_ops = {
- .iomap_begin = xfs_seek_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_seek_iomap_next, xfs_seek_iomap_begin);
static int
xfs_xattr_iomap_begin(
@@ -2349,9 +2339,7 @@ xfs_xattr_iomap_begin(
return xfs_bmbt_to_iomap(ip, iomap, &imap, flags, IOMAP_F_XATTR, seq);
}
-const struct iomap_ops xfs_xattr_iomap_ops = {
- .iomap_begin = xfs_xattr_iomap_begin,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_xattr_iomap_next, xfs_xattr_iomap_begin);
int
xfs_zero_range(
@@ -2367,9 +2355,9 @@ xfs_zero_range(
if (IS_DAX(inode))
return dax_zero_range(inode, pos, len, did_zero,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
return iomap_zero_range(inode, pos, len, did_zero,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
ac);
}
@@ -2384,8 +2372,8 @@ xfs_truncate_page(
if (IS_DAX(inode))
return dax_truncate_page(inode, pos, did_zero,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
return iomap_truncate_page(inode, pos, did_zero,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
ac);
}
diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
index ebcce7d494468..d2d3c84d93f9e 100644
--- a/fs/xfs/xfs_iomap.h
+++ b/fs/xfs/xfs_iomap.h
@@ -49,14 +49,26 @@ xfs_aligned_fsb_count(
return count_fsb;
}
-extern const struct iomap_ops xfs_buffered_write_iomap_ops;
-extern const struct iomap_ops xfs_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_zoned_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_read_iomap_ops;
-extern const struct iomap_ops xfs_seek_iomap_ops;
-extern const struct iomap_ops xfs_xattr_iomap_ops;
-extern const struct iomap_ops xfs_dax_write_iomap_ops;
-extern const struct iomap_ops xfs_atomic_write_cow_iomap_ops;
+int xfs_read_iomap_begin(struct inode *inode, loff_t offset,
+ loff_t length, unsigned flags, struct iomap *iomap,
+ struct iomap *srcmap);
+
+int xfs_buffered_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_xattr_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_dax_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_atomic_write_cow_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
extern const struct iomap_write_ops xfs_iomap_write_ops;
#endif /* __XFS_IOMAP_H__*/
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 6339f4956ecb3..5c3d9a365f935 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1239,10 +1239,10 @@ xfs_vn_fiemap(
if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
error = iomap_fiemap(inode, fieinfo, start, length,
- &xfs_xattr_iomap_ops);
+ xfs_xattr_iomap_next);
} else {
error = iomap_fiemap(inode, fieinfo, start, length,
- &xfs_read_iomap_ops);
+ xfs_read_iomap_next);
}
xfs_iunlock(XFS_I(inode), XFS_IOLOCK_SHARED);
diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c
index a5c188b781382..2b9792626bab5 100644
--- a/fs/xfs/xfs_reflink.c
+++ b/fs/xfs/xfs_reflink.c
@@ -1683,7 +1683,7 @@ xfs_reflink_remap_prep(
pos_out, len, remap_flags);
else
ret = dax_remap_file_range_prep(file_in, pos_in, file_out,
- pos_out, len, remap_flags, &xfs_read_iomap_ops);
+ pos_out, len, remap_flags, xfs_read_iomap_next);
if (ret || *len == 0)
goto out_unlock;
@@ -1878,10 +1878,10 @@ xfs_reflink_unshare(
if (IS_DAX(inode))
error = dax_file_unshare(inode, offset, len,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
else
error = iomap_file_unshare(inode, offset, len,
- &xfs_buffered_write_iomap_ops,
+ xfs_buffered_write_iomap_next,
&xfs_iomap_write_ops);
if (error)
goto out;
diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index 5ada33f70bb47..319d9502ba565 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -57,9 +57,7 @@ static int zonefs_read_iomap_begin(struct inode *inode, loff_t offset,
return 0;
}
-static const struct iomap_ops zonefs_read_iomap_ops = {
- .iomap_begin = zonefs_read_iomap_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(zonefs_read_iomap_next, zonefs_read_iomap_begin);
static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
@@ -106,19 +104,18 @@ static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
return 0;
}
-static const struct iomap_ops zonefs_write_iomap_ops = {
- .iomap_begin = zonefs_write_iomap_begin,
-};
+static DEFINE_IOMAP_ITER_NEXT(zonefs_write_iomap_next,
+ zonefs_write_iomap_begin);
static int zonefs_read_folio(struct file *unused, struct folio *folio)
{
- iomap_bio_read_folio(folio, &zonefs_read_iomap_ops);
+ iomap_bio_read_folio(folio, zonefs_read_iomap_next);
return 0;
}
static void zonefs_readahead(struct readahead_control *rac)
{
- iomap_bio_readahead(rac, &zonefs_read_iomap_ops);
+ iomap_bio_readahead(rac, zonefs_read_iomap_next);
}
/*
@@ -179,7 +176,7 @@ static int zonefs_swap_activate(struct swap_info_struct *sis,
}
return iomap_swapfile_activate(sis, swap_file, span,
- &zonefs_read_iomap_ops);
+ zonefs_read_iomap_next);
}
const struct address_space_operations zonefs_file_aops = {
@@ -309,7 +306,7 @@ static vm_fault_t zonefs_filemap_page_mkwrite(struct vm_fault *vmf)
/* Serialize against truncates */
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = iomap_page_mkwrite(vmf, &zonefs_write_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, zonefs_write_iomap_next, NULL);
filemap_invalidate_unlock_shared(inode->i_mapping);
sb_end_pagefault(inode->i_sb);
@@ -525,7 +522,7 @@ static ssize_t zonefs_file_dio_write(struct kiocb *iocb, struct iov_iter *from)
* page invalidation. Overwrite that error code with EBUSY so that
* the user can make sense of the error.
*/
- ret = iomap_dio_rw(iocb, from, &zonefs_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, zonefs_write_iomap_next,
&zonefs_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = -EBUSY;
@@ -575,7 +572,7 @@ static ssize_t zonefs_file_buffered_write(struct kiocb *iocb,
if (ret <= 0)
goto inode_unlock;
- ret = iomap_file_buffered_write(iocb, from, &zonefs_write_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, zonefs_write_iomap_next,
NULL, NULL);
if (ret == -EIO)
zonefs_io_error(inode, true);
@@ -670,7 +667,7 @@ static ssize_t zonefs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
goto inode_unlock;
}
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, to, &zonefs_read_iomap_ops,
+ ret = iomap_dio_rw(iocb, to, zonefs_read_iomap_next,
&zonefs_read_dio_ops, 0, NULL, 0);
} else {
ret = generic_file_read_iter(iocb, to);
diff --git a/include/linux/dax.h b/include/linux/dax.h
index fe6c3ded1b50f..e0bc653ffe9ff 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -3,6 +3,7 @@
#define _LINUX_DAX_H
#include <linux/fs.h>
+#include <linux/iomap.h>
#include <linux/mm.h>
#include <linux/radix-tree.h>
@@ -10,9 +11,6 @@ typedef unsigned long dax_entry_t;
struct dax_device;
struct gendisk;
-struct iomap_ops;
-struct iomap_iter;
-struct iomap;
enum dax_access_mode {
DAX_ACCESS,
@@ -213,11 +211,11 @@ static inline void dax_unlock_mapping_entry(struct address_space *mapping,
#endif
int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
static inline bool dax_page_is_idle(struct page *page)
{
@@ -266,10 +264,10 @@ int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, u64 len,
void dax_flush(struct dax_device *dax_dev, void *addr, size_t size);
ssize_t dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
unsigned long *pfnp, int *errp,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf,
unsigned int order, unsigned long pfn);
int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index);
@@ -288,11 +286,11 @@ void dax_break_layout_final(struct inode *inode);
int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
struct inode *dest, loff_t destoff,
loff_t len, bool *is_same,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
static inline bool dax_mapping(struct address_space *mapping)
{
return mapping->host && IS_DAX(mapping->host);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index d10897b3a1e35..2eb063438a3bb 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -70,7 +70,8 @@ struct fsnotify_mark_connector;
struct fs_context;
struct fs_parameter_spec;
struct file_kattr;
-struct iomap_ops;
+struct iomap_iter;
+struct iomap;
struct delegated_inode;
extern void __init inode_init(void);
@@ -2079,7 +2080,9 @@ int remap_verify_area(struct file *file, loff_t pos, loff_t len, bool write);
int __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *dax_read_ops);
+ int (*dax_read_next)(const struct iomap_iter *iter,
+ struct iomap *iomap,
+ struct iomap *srcmap));
int generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *count, unsigned int remap_flags);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 21e73cb9c51e3..1875e21ce1646 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -10,6 +10,7 @@
#include <linux/mm_types.h>
#include <linux/blkdev.h>
#include <linux/folio_batch.h>
+#include <linux/pagemap.h>
struct address_space;
struct fiemap_extent_info;
@@ -194,7 +195,7 @@ struct iomap_write_ops {
};
/*
- * Flags for iomap_begin / iomap_end. No flag implies a read.
+ * Flags for iomap_next. No flag implies a read.
*/
#define IOMAP_WRITE (1 << 0) /* writing, must allocate blocks */
#define IOMAP_ZERO (1 << 1) /* zeroing operation, may skip holes */
@@ -212,25 +213,31 @@ struct iomap_write_ops {
#define IOMAP_ATOMIC (1 << 9) /* torn-write protection */
#define IOMAP_DONTCACHE (1 << 10)
-struct iomap_ops {
- /*
- * Return the existing mapping at pos, or reserve space starting at
- * pos for up to length, as long as we can do it as a single mapping.
- * The actual length is returned in iomap->length.
- */
- int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,
- unsigned flags, struct iomap *iomap,
- struct iomap *srcmap);
+/*
+ * Return the existing mapping at pos, or reserve space starting at pos for up
+ * to length, as long as we can do it as a single mapping.
+ * The actual length is returned in iomap->length.
+ */
+typedef int (*iomap_iter_begin_fn)(struct inode *inode, loff_t pos,
+ loff_t length, unsigned flags, struct iomap *iomap,
+ struct iomap *srcmap);
- /*
- * Commit and/or unreserve space previous allocated using iomap_begin.
- * Written indicates the length of the successful write operation which
- * needs to be commited, while the rest needs to be unreserved.
- * Written might be zero if no data was written.
- */
- int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,
- ssize_t written, unsigned flags, struct iomap *iomap);
-};
+/*
+ * Commit and/or unreserve space previously allocated by iomap_iter_begin_fn.
+ * Written indicates the length of the successful write operation which needs
+ * to be committed, while the rest needs to be unreserved.
+ * Written might be zero if no data was written.
+ */
+typedef int (*iomap_iter_end_fn)(struct inode *inode, loff_t pos, loff_t length,
+ ssize_t written, unsigned flags, struct iomap *iomap);
+
+/*
+ * Produce the next mapping (finishing the previous one if needed).
+ * Return 1 to continue iterating, 0 if the range is fully consumed, or a
+ * negative error on failure.
+ */
+typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
/**
* struct iomap_iter - Iterate through a range of a file
@@ -243,7 +250,7 @@ struct iomap_ops {
* incremental iter advance.
* @status: Status of the most recent iteration. Zero on success or a negative
* errno on error.
- * @flags: Zero or more of the iomap_begin flags above.
+ * @flags: Zero or more of the iomap_iter_next flags above.
* @iomap: Map describing the I/O iteration
* @srcmap: Source map for COW operations
*/
@@ -260,7 +267,7 @@ struct iomap_iter {
void *private;
};
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops);
+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next);
int iomap_iter_advance(struct iomap_iter *iter, u64 count);
/**
@@ -317,6 +324,71 @@ static inline const struct iomap *iomap_iter_srcmap(const struct iomap_iter *i)
return &i->iomap;
}
+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap, int ret);
+
+/**
+ * iomap_iter_next - finish the previous mapping and produce the next one
+ * @iter: iteration structure
+ * @iomap: mapping to finish and then repopulate
+ * @srcmap: source mapping to finish and then repopulate
+ * @begin: callback that produces a mapping for the current position
+ * @end: optional callback that finishes the previous mapping, or NULL
+ *
+ * Inline helper that implements the common body of an iomap_iter_next_fn
+ * callback: it finishes the previous mapping via @end (if present), decides
+ * via iomap_iter_continue() whether to keep going, and obtains the next
+ * mapping via @begin.
+ *
+ * This helper is marked __always_inline so that when a caller passes
+ * compile-time-constant @begin and @end callbacks, the compiler can call them
+ * directly, avoiding the indirect-call overhead.
+ *
+ * Returns 1 to continue iterating, 0 once the range is fully consumed, or a
+ * negative errno on error.
+ */
+static __always_inline int iomap_iter_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap,
+ iomap_iter_begin_fn begin, iomap_iter_end_fn end)
+{
+ int ret = 0;
+
+ if (iomap->length) {
+ if (end) {
+ /*
+ * Calculate how far the iter was advanced and the
+ * original length bytes for end().
+ */
+ ssize_t advanced = iter->pos - iter->iter_start_pos;
+ loff_t len;
+
+ len = iomap_length_trim(iter, iter->iter_start_pos,
+ iter->len + advanced);
+
+ ret = end(iter->inode, iter->iter_start_pos, len,
+ advanced, iter->flags, iomap);
+ }
+ ret = iomap_iter_continue(iter, iomap, srcmap, ret);
+ if (ret <= 0)
+ return ret;
+ }
+
+ ret = begin(iter->inode, iter->pos, iter->len, iter->flags, iomap,
+ srcmap);
+
+ return ret < 0 ? ret : 1;
+}
+
+#define DEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, end_fn) \
+int name(const struct iomap_iter *iter, struct iomap *iomap, \
+ struct iomap *srcmap) \
+{ \
+ return iomap_iter_next(iter, iomap, srcmap, begin_fn, end_fn); \
+}
+
+#define DEFINE_IOMAP_ITER_NEXT(name, begin_fn) \
+ DEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, NULL)
+
/*
* Return the file offset for the first unchanged block after a short write.
*
@@ -351,15 +423,15 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter)
}
ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private);
int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
- const void *buf, const struct iomap_ops *ops,
+ const void *buf, iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops);
-void iomap_read_folio(const struct iomap_ops *ops,
- struct iomap_read_folio_ctx *ctx, void *private);
-void iomap_readahead(const struct iomap_ops *ops,
- struct iomap_read_folio_ctx *ctx, void *private);
+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+ void *private);
+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+ void *private);
bool iomap_is_partially_uptodate(struct folio *, size_t from, size_t count);
struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len);
bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);
@@ -367,17 +439,17 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);
bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);
void iomap_folio_mark_uptodate(struct folio *folio);
int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops);
unsigned int iomap_fill_dirty_folios(struct iomap_iter *iter, loff_t *start,
loff_t end, unsigned int *iomap_flags);
int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len,
- bool *did_zero, const struct iomap_ops *ops,
+ bool *did_zero, iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private);
int iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_iter_next_fn next,
const struct iomap_write_ops *write_ops, void *private);
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,
void *private);
typedef void (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length,
struct iomap *iomap);
@@ -386,13 +458,13 @@ void iomap_write_delalloc_release(struct inode *inode, loff_t start_byte,
iomap_punch_t punch);
int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
- u64 start, u64 len, const struct iomap_ops *ops);
+ u64 start, u64 len, iomap_iter_next_fn next);
loff_t iomap_seek_hole(struct inode *inode, loff_t offset,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
loff_t iomap_seek_data(struct inode *inode, loff_t offset,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
/*
* Flags for iomap_ioend->io_flags.
@@ -599,23 +671,88 @@ struct iomap_dio_ops {
#define IOMAP_DIO_BOUNCE (1 << 4)
ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before);
struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before);
ssize_t iomap_dio_complete(struct iomap_dio *dio);
void iomap_dio_bio_end_io(struct bio *bio);
+/*
+ * Fast path for small, block-aligned direct I/Os that map to a single
+ * contiguous on-disk extent.
+ *
+ * @iter must describe a non-empty READ no larger than the inode block size:
+ * writes, zero-length I/O, and larger requests need the generic iomap direct
+ * I/O path.
+ *
+ * Does not support iomap_dio_ops, dio_flags, done_before or private data.
+ * The range must also stay within i_size and encrypted inodes must use the
+ * generic iomap direct I/O path.
+ *
+ * -ENOTBLK indicates the generic path must be used by the caller instead.
+ * Any other errno is a real result and is propagated as-is, in particular
+ * -EAGAIN for IOCB_NOWAIT must reach the caller.
+ *
+ * The caller can only provide an iomap begin handler, and the iterator
+ * is never advanced.
+ */
+ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,
+ struct iomap_iter *iomi);
+static __always_inline ssize_t iomap_dio_read_simple(struct kiocb *iocb,
+ struct iov_iter *iter, iomap_iter_begin_fn begin)
+{
+ struct iomap_iter iomi = {
+ .inode = file_inode(iocb->ki_filp),
+ .pos = iocb->ki_pos,
+ .len = iov_iter_count(iter),
+ .flags = IOMAP_DIRECT,
+ };
+ ssize_t ret;
+
+ if (!iomi.len)
+ return 0;
+
+ /*
+ * Simple dio is an optimization for small IO. Filter out large IO
+ * early as it's the most common case to fail for typical direct IO
+ * workloads.
+ */
+ if (iomi.len > iomi.inode->i_sb->s_blocksize)
+ return -ENOTBLK;
+ if (iocb->ki_pos + iomi.len > i_size_read(iomi.inode))
+ return -ENOTBLK;
+ if (IS_ENCRYPTED(iomi.inode))
+ return -ENOTBLK;
+
+ ret = kiocb_write_and_wait(iocb, iomi.len);
+ if (ret)
+ return ret;
+
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ iomi.flags |= IOMAP_NOWAIT;
+
+ inode_dio_begin(iomi.inode);
+ ret = begin(iomi.inode, iomi.pos, iomi.len, iomi.flags, &iomi.iomap,
+ &iomi.srcmap);
+ if (ret) {
+ inode_dio_end(iomi.inode);
+ return ret;
+ }
+
+ return __iomap_dio_read_simple(iocb, iter, &iomi);
+}
+
#ifdef CONFIG_SWAP
struct file;
struct swap_info_struct;
int iomap_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *pagespan,
- const struct iomap_ops *ops);
+ iomap_iter_next_fn next);
#else
-# define iomap_swapfile_activate(sis, swapfile, pagespan, ops) (-EIO)
+# define iomap_swapfile_activate(sis, swapfile, pagespan, next) (-EIO)
#endif /* CONFIG_SWAP */
extern struct bio_set iomap_ioend_bioset;
@@ -627,25 +764,25 @@ int iomap_bio_read_folio_range(const struct iomap_iter *iter,
extern const struct iomap_read_ops iomap_bio_read_ops;
static inline void iomap_bio_read_folio(struct folio *folio,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_read_folio_ctx ctx = {
.ops = &iomap_bio_read_ops,
.cur_folio = folio,
};
- iomap_read_folio(ops, &ctx, NULL);
+ iomap_read_folio(next, &ctx, NULL);
}
static inline void iomap_bio_readahead(struct readahead_control *rac,
- const struct iomap_ops *ops)
+ iomap_iter_next_fn next)
{
struct iomap_read_folio_ctx ctx = {
.ops = &iomap_bio_read_ops,
.rac = rac,
};
- iomap_readahead(ops, &ctx, NULL);
+ iomap_readahead(next, &ctx, NULL);
}
#endif /* CONFIG_BLOCK */
|
| 3/2 | 2026/07/28 19:02 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 61056 output: 334 thoughts: 2257 Thoughts: **My Assessment of the iomap Refactoring for Fuzzing** Okay, so here's the deal. I've got this patch in front of me that's shaking up the `iomap` subsystem in the kernel. This is *significant*, no two ways about it. It’s a core change, moving from a `struct iomap_ops` to a callback function, `iomap_iter_next_fn`, for handling those iomap iterations. Basically, we are changing the control flow. We used to have separate `iomap_begin` and `iomap_end` functions defined in the `struct iomap_ops`. Now, it’s all consolidated into this single `next` function, which is often generated by macros, `DEFINE_IOMAP_ITER_NEXT` and `DEFINE_IOMAP_ITER_NEXT_END`. This touches a *ton* of filesystems – btrfs, dax, erofs, exfat, ext2, ext4, f2fs, fuse, gfs2, hpfs, ntfs, ntfs3, xfs, zonefs – all of them! And it's not just filesystem internals; this hits core I/O paths like buffered I/O, direct I/O, DAX, fiemap, seek, and swapfile. We have got some interesting functional changes here. This is a classic case for fuzzing. It's a functional change, it’s testable, and it impacts a wide range of functionality. We need to hammer this thing and make sure we don't introduce regressions or bugs in the iteration logic. I need to nail down the specific entry points to target. Looking at the code, these functions stand out: * `iomap_iter` * `iomap_iter_continue` * `iomap_file_buffered_write` * `iomap_read_folio` * `iomap_readahead` * `iomap_dio_rw` * `__iomap_dio_read_simple` (This one looks new, and that's always interesting) * `iomap_fiemap` * `iomap_seek_hole` * `iomap_seek_data` * `iomap_bmap` * `iomap_swapfile_activate` * `dax_iomap_rw` * `dax_iomap_fault` `iomap_iter` itself is core as hell, but it's *precisely* what the patch changed. So it's absolutely crucial. I need to make sure we're including filesystems using the iomap subsystem, like those enabled by `CONFIG_FS_IOMAP`. We'll probably need `EXT4_FS`, `XFS_FS`, `BTRFS_FS`, `F2FS_FS`, and `FS_DAX`. Okay, let's take a look at the specific functions. I am most interested in `iomap_iter`, `iomap_iter_continue`, and `__iomap_dio_read_simple`. Also the ones that are called from the top level (like file write, dio, etc.). I am going to use these specific functions as the focal points for the fuzzing campaign. I am also going to include the appropriate config options to ensure that all these relevant filesystems will be active. I need to confirm that `iomap_iter` is indeed as hot as I think it is. Yes, it is the core loop. Since the changes impact all iomap operations, it should be the right place to focus the fuzzing. |
| 4/2 | 2026/07/28 19:02 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"EXT4_FS",
"XFS_FS",
"BTRFS_FS",
"F2FS_FS",
"FS_DAX"
],
"FocusSymbols": [
"iomap_iter",
"iomap_iter_continue",
"__iomap_dio_read_simple",
"iomap_dio_rw",
"iomap_file_buffered_write",
"iomap_read_folio",
"iomap_readahead",
"iomap_fiemap",
"iomap_seek_hole",
"iomap_seek_data",
"dax_iomap_rw",
"dax_iomap_fault"
],
"Reasoning": "The patch refactors the iomap subsystem to use a single `iomap_iter_next_fn` callback instead of a `struct iomap_ops` with `iomap_begin` and `iomap_end` functions. This changes the control flow and calling convention for all iomap operations across many filesystems (ext4, xfs, btrfs, f2fs, etc.). It also introduces `__iomap_dio_read_simple` and `iomap_iter_continue`. These are core functional changes to the I/O path that should be fuzzed to ensure no regressions or bugs were introduced in the iteration logic.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"EXT4_FS",
"XFS_FS",
"BTRFS_FS",
"F2FS_FS",
"FS_DAX"
],
"FocusSymbols": [
"iomap_iter",
"iomap_iter_continue",
"__iomap_dio_read_simple",
"iomap_dio_rw",
"iomap_file_buffered_write",
"iomap_read_folio",
"iomap_readahead",
"iomap_fiemap",
"iomap_seek_hole",
"iomap_seek_data",
"dax_iomap_rw",
"dax_iomap_fault"
],
"Reasoning": "The patch refactors the iomap subsystem to use a single `iomap_iter_next_fn` callback instead of a `struct iomap_ops` with `iomap_begin` and `iomap_end` functions. This changes the control flow and calling convention for all iomap operations across many filesystems (ext4, xfs, btrfs, f2fs, etc.). It also introduces `__iomap_dio_read_simple` and `iomap_iter_continue`. These are core functional changes to the I/O path that should be fuzzed to ensure no regressions or bugs were introduced in the iteration logic.",
"WorthFuzzing": true
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|