| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/07/23 14:19 | flow | patch-triage |
1mArgs:null Results: {
"EnableConfigs": [
"NETFS_SUPPORT",
"FSCACHE",
"CACHEFILES"
],
"FocusSymbols": [
"bvecq_alloc_chain",
"bvecq_expand_buffer",
"bvecq_shorten_buffer",
"bvecq_pos_advance",
"bvecq_zero",
"bvecq_slice",
"bvecq_extract",
"bvecq_load_from_ra",
"netfs_prepare_read_buffer",
"netfs_prepare_write_buffer",
"netfs_extract_iter"
],
"Reasoning": "The patch replaces the `folio_queue` data structure with `bvecq` in the netfs library and updates all dependent filesystems (9p, afs, ceph, nfs, smb) to use it. It also removes the on-demand read mode from cachefiles. This is a major functional change to the core I/O buffering logic in netfs and should be heavily fuzzed.",
"WorthFuzzing": true
} |
| 1/1 | 2026/07/23 14:19 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit eec393828081ac61b3256a58b75135a47309806f\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Thu Jul 23 14:19:18 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/core-api/folio_queue.rst b/Documentation/core-api/folio_queue.rst\ndeleted file mode 100644\nindex b7628896d2b695..00000000000000\n--- a/Documentation/core-api/folio_queue.rst\n+++ /dev/null\n@@ -1,209 +0,0 @@\n-.. SPDX-License-Identifier: GPL-2.0+\n-\n-===========\n-Folio Queue\n-===========\n-\n-:Author: David Howells \u003cdhowells@redhat.com\u003e\n-\n-.. Contents:\n-\n- * Overview\n- * Initialisation\n- * Adding and removing folios\n- * Querying information about a folio\n- * Querying information about a folio_queue\n- * Folio queue iteration\n- * Folio marks\n- * Lockless simultaneous production/consumption issues\n-\n-\n-Overview\n-========\n-\n-The folio_queue struct forms a single segment in a segmented list of folios\n-that can be used to form an I/O buffer. As such, the list can be iterated over\n-using the ITER_FOLIOQ iov_iter type.\n-\n-The publicly accessible members of the structure are::\n-\n-\tstruct folio_queue {\n-\t\tstruct folio_queue *next;\n-\t\tstruct folio_queue *prev;\n-\t\t...\n-\t};\n-\n-A pair of pointers are provided, ``next`` and ``prev``, that point to the\n-segments on either side of the segment being accessed. Whilst this is a\n-doubly-linked list, it is intentionally not a circular list; the outward\n-sibling pointers in terminal segments should be NULL.\n-\n-Each segment in the list also stores:\n-\n- * an ordered sequence of folio pointers,\n- * the size of each folio and\n- * three 1-bit marks per folio,\n-\n-but these should not be accessed directly as the underlying data structure may\n-change, but rather the access functions outlined below should be used.\n-\n-The facility can be made accessible by::\n-\n-\t#include \u003clinux/folio_queue.h\u003e\n-\n-and to use the iterator::\n-\n-\t#include \u003clinux/uio.h\u003e\n-\n-\n-Initialisation\n-==============\n-\n-A segment should be initialised by calling::\n-\n-\tvoid folioq_init(struct folio_queue *folioq);\n-\n-with a pointer to the segment to be initialised. Note that this will not\n-necessarily initialise all the folio pointers, so care must be taken to check\n-the number of folios added.\n-\n-\n-Adding and removing folios\n-==========================\n-\n-Folios can be set in the next unused slot in a segment struct by calling one\n-of::\n-\n-\tunsigned int folioq_append(struct folio_queue *folioq,\n-\t\t\t\t struct folio *folio);\n-\n-\tunsigned int folioq_append_mark(struct folio_queue *folioq,\n-\t\t\t\t\tstruct folio *folio);\n-\n-Both functions update the stored folio count, store the folio and note its\n-size. The second function also sets the first mark for the folio added. Both\n-functions return the number of the slot used. [!] Note that no attempt is made\n-to check that the capacity wasn't overrun and the list will not be extended\n-automatically.\n-\n-A folio can be excised by calling::\n-\n-\tvoid folioq_clear(struct folio_queue *folioq, unsigned int slot);\n-\n-This clears the slot in the array and also clears all the marks for that folio,\n-but doesn't change the folio count - so future accesses of that slot must check\n-if the slot is occupied.\n-\n-\n-Querying information about a folio\n-==================================\n-\n-Information about the folio in a particular slot may be queried by the\n-following function::\n-\n-\tstruct folio *folioq_folio(const struct folio_queue *folioq,\n-\t\t\t\t unsigned int slot);\n-\n-If a folio has not yet been set in that slot, this may yield an undefined\n-pointer. The size of the folio in a slot may be queried with either of::\n-\n-\tunsigned int folioq_folio_order(const struct folio_queue *folioq,\n-\t\t\t\t\tunsigned int slot);\n-\n-\tsize_t folioq_folio_size(const struct folio_queue *folioq,\n-\t\t\t\t unsigned int slot);\n-\n-The first function returns the size as an order and the second as a number of\n-bytes.\n-\n-\n-Querying information about a folio_queue\n-========================================\n-\n-Information may be retrieved about a particular segment with the following\n-functions::\n-\n-\tunsigned int folioq_nr_slots(const struct folio_queue *folioq);\n-\n-\tunsigned int folioq_count(struct folio_queue *folioq);\n-\n-\tbool folioq_full(struct folio_queue *folioq);\n-\n-The first function returns the maximum capacity of a segment. It must not be\n-assumed that this won't vary between segments. The second returns the number\n-of folios added to a segments and the third is a shorthand to indicate if the\n-segment has been filled to capacity.\n-\n-Not that the count and fullness are not affected by clearing folios from the\n-segment. These are more about indicating how many slots in the array have been\n-initialised, and it assumed that slots won't get reused, but rather the segment\n-will get discarded as the queue is consumed.\n-\n-\n-Folio marks\n-===========\n-\n-Folios within a queue can also have marks assigned to them. These marks can be\n-used to note information such as if a folio needs folio_put() calling upon it.\n-There are three marks available to be set for each folio.\n-\n-The marks can be set by::\n-\n-\tvoid folioq_mark(struct folio_queue *folioq, unsigned int slot);\n-\tvoid folioq_mark2(struct folio_queue *folioq, unsigned int slot);\n-\n-Cleared by::\n-\n-\tvoid folioq_unmark(struct folio_queue *folioq, unsigned int slot);\n-\tvoid folioq_unmark2(struct folio_queue *folioq, unsigned int slot);\n-\n-And the marks can be queried by::\n-\n-\tbool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot);\n-\tbool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot);\n-\n-The marks can be used for any purpose and are not interpreted by this API.\n-\n-\n-Folio queue iteration\n-=====================\n-\n-A list of segments may be iterated over using the I/O iterator facility using\n-an ``iov_iter`` iterator of ``ITER_FOLIOQ`` type. The iterator may be\n-initialised with::\n-\n-\tvoid iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,\n-\t\t\t\t const struct folio_queue *folioq,\n-\t\t\t\t unsigned int first_slot, unsigned int offset,\n-\t\t\t\t size_t count);\n-\n-This may be told to start at a particular segment, slot and offset within a\n-queue. The iov iterator functions will follow the next pointers when advancing\n-and prev pointers when reverting when needed.\n-\n-\n-Lockless simultaneous production/consumption issues\n-===================================================\n-\n-If properly managed, the list can be extended by the producer at the head end\n-and shortened by the consumer at the tail end simultaneously without the need\n-to take locks. The ITER_FOLIOQ iterator inserts appropriate barriers to aid\n-with this.\n-\n-Care must be taken when simultaneously producing and consuming a list. If the\n-last segment is reached and the folios it refers to are entirely consumed by\n-the IOV iterators, an iov_iter struct will be left pointing to the last segment\n-with a slot number equal to the capacity of that segment. The iterator will\n-try to continue on from this if there's another segment available when it is\n-used again, but care must be taken lest the segment got removed and freed by\n-the consumer before the iterator was advanced.\n-\n-It is recommended that the queue always contain at least one segment, even if\n-that segment has never been filled or is entirely spent. This prevents the\n-head and tail pointers from collapsing.\n-\n-\n-API Function Reference\n-======================\n-\n-.. kernel-doc:: include/linux/folio_queue.h\ndiff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst\nindex 13769d5c40bf2f..16c529a33ac488 100644\n--- a/Documentation/core-api/index.rst\n+++ b/Documentation/core-api/index.rst\n@@ -39,7 +39,6 @@ Library functionality that is used throughout the kernel.\n kref\n cleanup\n assoc_array\n- folio_queue\n xarray\n maple_tree\n idr\ndiff --git a/Documentation/filesystems/caching/cachefiles.rst b/Documentation/filesystems/caching/cachefiles.rst\nindex b3ccc782cb3b2b..adfb7d07902710 100644\n--- a/Documentation/filesystems/caching/cachefiles.rst\n+++ b/Documentation/filesystems/caching/cachefiles.rst\n@@ -28,8 +28,6 @@ Cache on Already Mounted Filesystem\n \n (*) Debugging.\n \n- (*) On-demand Read.\n-\n \n Overview\n ========\n@@ -483,180 +481,3 @@ the control file. For example::\n \techo $((1|4|8)) \u003e/sys/module/cachefiles/parameters/debug\n \n will turn on all function entry debugging.\n-\n-\n-On-demand Read\n-==============\n-\n-When working in its original mode, CacheFiles serves as a local cache for a\n-remote networking fs - while in on-demand read mode, CacheFiles can boost the\n-scenario where on-demand read semantics are needed, e.g. container image\n-distribution.\n-\n-The essential difference between these two modes is seen when a cache miss\n-occurs: In the original mode, the netfs will fetch the data from the remote\n-server and then write it to the cache file; in on-demand read mode, fetching\n-the data and writing it into the cache is delegated to a user daemon.\n-\n-``CONFIG_CACHEFILES_ONDEMAND`` should be enabled to support on-demand read mode.\n-\n-\n-Protocol Communication\n-----------------------\n-\n-The on-demand read mode uses a simple protocol for communication between kernel\n-and user daemon. The protocol can be modeled as::\n-\n-\tkernel --[request]--\u003e user daemon --[reply]--\u003e kernel\n-\n-CacheFiles will send requests to the user daemon when needed. The user daemon\n-should poll the devnode ('/dev/cachefiles') to check if there's a pending\n-request to be processed. A POLLIN event will be returned when there's a pending\n-request.\n-\n-The user daemon then reads the devnode to fetch a request to process. It should\n-be noted that each read only gets one request. When it has finished processing\n-the request, the user daemon should write the reply to the devnode.\n-\n-Each request starts with a message header of the form::\n-\n-\tstruct cachefiles_msg {\n-\t\t__u32 msg_id;\n-\t\t__u32 opcode;\n-\t\t__u32 len;\n-\t\t__u32 object_id;\n-\t\t__u8 data[];\n-\t};\n-\n-where:\n-\n-\t* ``msg_id`` is a unique ID identifying this request among all pending\n-\t requests.\n-\n-\t* ``opcode`` indicates the type of this request.\n-\n-\t* ``object_id`` is a unique ID identifying the cache file operated on.\n-\n-\t* ``data`` indicates the payload of this request.\n-\n-\t* ``len`` indicates the whole length of this request, including the\n-\t header and following type-specific payload.\n-\n-\n-Turning on On-demand Mode\n--------------------------\n-\n-An optional parameter becomes available to the \"bind\" command::\n-\n-\tbind [ondemand]\n-\n-When the \"bind\" command is given no argument, it defaults to the original mode.\n-When it is given the \"ondemand\" argument, i.e. \"bind ondemand\", on-demand read\n-mode will be enabled.\n-\n-\n-The OPEN Request\n-----------------\n-\n-When the netfs opens a cache file for the first time, a request with the\n-CACHEFILES_OP_OPEN opcode, a.k.a an OPEN request will be sent to the user\n-daemon. The payload format is of the form::\n-\n-\tstruct cachefiles_open {\n-\t\t__u32 volume_key_size;\n-\t\t__u32 cookie_key_size;\n-\t\t__u32 fd;\n-\t\t__u32 flags;\n-\t\t__u8 data[];\n-\t};\n-\n-where:\n-\n-\t* ``data`` contains the volume_key followed directly by the cookie_key.\n-\t The volume key is a NUL-terminated string; the cookie key is binary\n-\t data.\n-\n-\t* ``volume_key_size`` indicates the size of the volume key in bytes.\n-\n-\t* ``cookie_key_size`` indicates the size of the cookie key in bytes.\n-\n-\t* ``fd`` indicates an anonymous fd referring to the cache file, through\n-\t which the user daemon can perform write/llseek file operations on the\n-\t cache file.\n-\n-\n-The user daemon can use the given (volume_key, cookie_key) pair to distinguish\n-the requested cache file. With the given anonymous fd, the user daemon can\n-fetch the data and write it to the cache file in the background, even when\n-kernel has not triggered a cache miss yet.\n-\n-Be noted that each cache file has a unique object_id, while it may have multiple\n-anonymous fds. The user daemon may duplicate anonymous fds from the initial\n-anonymous fd indicated by the @fd field through dup(). Thus each object_id can\n-be mapped to multiple anonymous fds, while the usr daemon itself needs to\n-maintain the mapping.\n-\n-When implementing a user daemon, please be careful of RLIMIT_NOFILE,\n-``/proc/sys/fs/nr_open`` and ``/proc/sys/fs/file-max``. Typically these needn't\n-be huge since they're related to the number of open device blobs rather than\n-open files of each individual filesystem.\n-\n-The user daemon should reply the OPEN request by issuing a \"copen\" (complete\n-open) command on the devnode::\n-\n-\tcopen \u003cmsg_id\u003e,\u003ccache_size\u003e\n-\n-where:\n-\n-\t* ``msg_id`` must match the msg_id field of the OPEN request.\n-\n-\t* When \u003e= 0, ``cache_size`` indicates the size of the cache file;\n-\t when \u003c 0, ``cache_size`` indicates any error code encountered by the\n-\t user daemon.\n-\n-\n-The CLOSE Request\n------------------\n-\n-When a cookie withdrawn, a CLOSE request (opcode CACHEFILES_OP_CLOSE) will be\n-sent to the user daemon. This tells the user daemon to close all anonymous fds\n-associated with the given object_id. The CLOSE request has no extra payload,\n-and shouldn't be replied.\n-\n-\n-The READ Request\n-----------------\n-\n-When a cache miss is encountered in on-demand read mode, CacheFiles will send a\n-READ request (opcode CACHEFILES_OP_READ) to the user daemon. This tells the user\n-daemon to fetch the contents of the requested file range. The payload is of the\n-form::\n-\n-\tstruct cachefiles_read {\n-\t\t__u64 off;\n-\t\t__u64 len;\n-\t};\n-\n-where:\n-\n-\t* ``off`` indicates the starting offset of the requested file range.\n-\n-\t* ``len`` indicates the length of the requested file range.\n-\n-\n-When it receives a READ request, the user daemon should fetch the requested data\n-and write it to the cache file identified by object_id.\n-\n-When it has finished processing the READ request, the user daemon should reply\n-by using the CACHEFILES_IOC_READ_COMPLETE ioctl on one of the anonymous fds\n-associated with the object_id given in the READ request. The ioctl is of the\n-form::\n-\n-\tioctl(fd, CACHEFILES_IOC_READ_COMPLETE, msg_id);\n-\n-where:\n-\n-\t* ``fd`` is one of the anonymous fds associated with the object_id\n-\t given.\n-\n-\t* ``msg_id`` must match the msg_id field of the READ request.\ndiff --git a/Documentation/filesystems/netfs_library.rst b/Documentation/filesystems/netfs_library.rst\nindex ddd799df6ce363..18e3c3aae57cda 100644\n--- a/Documentation/filesystems/netfs_library.rst\n+++ b/Documentation/filesystems/netfs_library.rst\n@@ -449,7 +449,7 @@ be called from the writeback code to write the data to the cache, if there is\n one.\n \n The inode should be marked ``NETFS_ICTX_SINGLE_NO_UPLOAD`` if this API is to be\n-used. The writeback function requires the buffer to be of ITER_FOLIOQ type.\n+used.\n \n High-Level VM API\n ==================\ndiff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c\nindex 1ac0b3dcc0778e..fc5b9c677986e8 100644\n--- a/fs/9p/vfs_addr.c\n+++ b/fs/9p/vfs_addr.c\n@@ -48,31 +48,70 @@ static void v9fs_begin_writeback(struct netfs_io_request *wreq)\n \twreq-\u003eio_streams[0].avail = true;\n }\n \n+/*\n+ * Estimate how much data should be accumulated before we start issuing\n+ * write subrequests.\n+ */\n+static int v9fs_estimate_write(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_write_estimate *estimate)\n+{\n+\tstruct p9_fid *fid = wreq-\u003enetfs_priv;\n+\tunsigned long long limit = ULLONG_MAX - stream-\u003eissue_from;\n+\tunsigned long long max_len = fid-\u003eclnt-\u003emsize - P9_IOHDRSZ;\n+\n+\testimate-\u003eissue_at = stream-\u003eissue_from + umin(max_len, limit);\n+\treturn 0;\n+}\n+\n /*\n * Issue a subrequest to write to the server.\n */\n-static void v9fs_issue_write(struct netfs_io_subrequest *subreq)\n+static int v9fs_issue_write(struct netfs_io_subrequest *subreq)\n {\n+\tstruct iov_iter iter;\n \tstruct p9_fid *fid = subreq-\u003erreq-\u003enetfs_priv;\n-\tint err, len;\n+\tint err, len = 0;\n+\n+\tsubreq-\u003elen = umin(subreq-\u003elen, fid-\u003eclnt-\u003emsize - P9_IOHDRSZ);\n \n-\tlen = p9_client_write(fid, subreq-\u003estart, \u0026subreq-\u003eio_iter, \u0026err);\n+\terr = netfs_prepare_write_buffer(subreq, INT_MAX);\n+\tif (err \u003c 0)\n+\t\treturn err;\n+\t/* After this point, must fail by termination. */\n+\n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n+\tlen = p9_client_write(fid, subreq-\u003estart, \u0026iter, \u0026err);\n \tif (len \u003e 0)\n \t\t__set_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n+\n \tnetfs_write_subrequest_terminated(subreq, len ?: err);\n+\treturn 0;\n }\n \n /**\n * v9fs_issue_read - Issue a read from 9P\n * @subreq: The read to make\n */\n-static void v9fs_issue_read(struct netfs_io_subrequest *subreq)\n+static int v9fs_issue_read(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n+\tstruct iov_iter iter;\n \tstruct p9_fid *fid = rreq-\u003enetfs_priv;\n \tchar *target;\n \tunsigned long long pos = subreq-\u003estart + subreq-\u003etransferred;\n-\tint total = 0, err, len, n;\n+\tsize_t len;\n+\tint total = 0, err;\n+\n+\terr = netfs_prepare_read_buffer(subreq, INT_MAX);\n+\tif (err \u003c 0)\n+\t\treturn err;\n+\t/* After this point, must fail by termination. */\n+\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n \n \tif (S_ISLNK(rreq-\u003einode-\u003ei_mode)) {\n \t\t/* p9_client_readlink() must not be called for legacy protocols\n@@ -89,12 +128,13 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)\n \t\terr = p9_client_readlink(fid, \u0026target);\n \t\tif (err != 0)\n \t\t\tgoto fill_subreq;\n-\t\tlen = strlen(target);\n-\t\tn = copy_to_iter(target, len, \u0026subreq-\u003eio_iter);\n+\t\tlen = min(strlen(target), subreq-\u003elen);\n+\t\ttotal = copy_to_iter(target, len, \u0026iter);\n \t\tkfree(target);\n-\t\ttotal = n;\n \t} else {\n-\t\ttotal = p9_client_read(fid, pos, \u0026subreq-\u003eio_iter, \u0026err);\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n+\n+\t\ttotal = p9_client_read(fid, pos, \u0026iter, \u0026err);\n \t}\n \n fill_subreq:\n@@ -112,6 +152,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)\n \n \tsubreq-\u003eerror = err;\n \tnetfs_read_subreq_terminated(subreq);\n+\treturn 0;\n }\n \n /**\n@@ -124,7 +165,6 @@ static int v9fs_init_request(struct netfs_io_request *rreq, struct file *file)\n \tstruct p9_fid *fid;\n \tstruct dentry *dentry;\n \tbool writing = (rreq-\u003eorigin == NETFS_READ_FOR_WRITE ||\n-\t\t\trreq-\u003eorigin == NETFS_WRITETHROUGH ||\n \t\t\trreq-\u003eorigin == NETFS_UNBUFFERED_WRITE ||\n \t\t\trreq-\u003eorigin == NETFS_DIO_WRITE);\n \n@@ -186,6 +226,7 @@ const struct netfs_request_ops v9fs_req_ops = {\n \t.free_request\t\t= v9fs_free_request,\n \t.issue_read\t\t= v9fs_issue_read,\n \t.begin_writeback\t= v9fs_begin_writeback,\n+\t.estimate_write\t\t= v9fs_estimate_write,\n \t.issue_write\t\t= v9fs_issue_write,\n };\n \ndiff --git a/fs/afs/dir.c b/fs/afs/dir.c\nindex 6df56fe9163fd7..174114c3963955 100644\n--- a/fs/afs/dir.c\n+++ b/fs/afs/dir.c\n@@ -140,9 +140,9 @@ static void afs_dir_dump(struct afs_vnode *dvnode)\n \tpr_warn(\"DIR %llx:%llx is=%llx\\n\",\n \t\tdvnode-\u003efid.vid, dvnode-\u003efid.vnode, i_size);\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n-\titerate_folioq(\u0026iter, iov_iter_count(\u0026iter), NULL, NULL,\n-\t\t afs_dir_dump_step);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n+\titerate_bvecq(\u0026iter, iov_iter_count(\u0026iter), NULL, NULL,\n+\t\t afs_dir_dump_step);\n }\n \n /*\n@@ -203,9 +203,9 @@ static int afs_dir_check(struct afs_vnode *dvnode)\n \tif (unlikely(!i_size))\n \t\treturn 0;\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n-\tchecked = iterate_folioq(\u0026iter, iov_iter_count(\u0026iter), dvnode, NULL,\n-\t\t\t\t afs_dir_check_step);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n+\tchecked = iterate_bvecq(\u0026iter, iov_iter_count(\u0026iter), dvnode, NULL,\n+\t\t\t\tafs_dir_check_step);\n \tif (checked != i_size) {\n \t\tafs_dir_dump(dvnode);\n \t\treturn -EIO;\n@@ -250,15 +250,14 @@ static ssize_t afs_do_read_single(struct afs_vnode *dvnode, struct file *file)\n \tif (dvnode-\u003edirectory_size \u003c i_size) {\n \t\tsize_t cur_size = dvnode-\u003edirectory_size;\n \n-\t\tret = netfs_alloc_folioq_buffer(NULL,\n-\t\t\t\t\t\t\u0026dvnode-\u003edirectory, \u0026cur_size, i_size,\n-\t\t\t\t\t\tmapping_gfp_mask(dvnode-\u003enetfs.inode.i_mapping));\n+\t\tret = bvecq_expand_buffer(\u0026dvnode-\u003edirectory, \u0026cur_size,\n+\t\t\t\t\t round_up(i_size, PAGE_SIZE), GFP_KERNEL);\n \t\tdvnode-\u003edirectory_size = cur_size;\n \t\tif (ret \u003c 0)\n \t\t\treturn ret;\n \t}\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_DEST, dvnode-\u003edirectory, 0, 0, dvnode-\u003edirectory_size);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, dvnode-\u003edirectory, 0, 0, dvnode-\u003edirectory_size);\n \n \t/* AFS requires us to perform the read of a directory synchronously as\n \t * a single unit to avoid issues with the directory contents being\n@@ -294,8 +293,8 @@ static ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file)\n }\n \n /*\n- * Read the directory into a folio_queue buffer in one go, scrubbing the\n- * previous contents. We return -ESTALE if the caller needs to call us again.\n+ * Read the directory into the buffer in one go, scrubbing the previous\n+ * contents. We return -ESTALE if the caller needs to call us again.\n */\n ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)\n \t__acquires(\u0026dvnode-\u003evalidate_lock)\n@@ -483,7 +482,7 @@ static size_t afs_dir_iterate_step(void *iter_base, size_t progress, size_t len,\n }\n \n /*\n- * Iterate through the directory folios.\n+ * Iterate through the directory content.\n */\n static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_ctx)\n {\n@@ -498,11 +497,11 @@ static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_c\n \tif (i_size \u003c= 0 || dir_ctx-\u003epos \u003e= i_size)\n \t\treturn 0;\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0, i_size);\n \tiov_iter_advance(\u0026iter, round_down(dir_ctx-\u003epos, AFS_DIR_BLOCK_SIZE));\n \n-\titerate_folioq(\u0026iter, iov_iter_count(\u0026iter), dvnode, \u0026ctx,\n-\t\t afs_dir_iterate_step);\n+\titerate_bvecq(\u0026iter, iov_iter_count(\u0026iter), dvnode, \u0026ctx,\n+\t\t afs_dir_iterate_step);\n \n \tif (ctx.error == -ESTALE)\n \t\tafs_invalidate_dir(dvnode, afs_dir_invalid_iter_stale);\n@@ -2228,11 +2227,10 @@ static int afs_dir_writepages(struct address_space *mapping,\n \t}\n \n \tif (test_bit(AFS_VNODE_DIR_VALID, \u0026dvnode-\u003eflags)) {\n-\t\tiov_iter_folio_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0,\n-\t\t\t\t i_size_read(\u0026dvnode-\u003enetfs.inode));\n-\t\tret = netfs_writeback_single(mapping, wbc, \u0026iter);\n-\t\tif (ret == 1)\n-\t\t\tret = 0; /* Skipped write due to lock conflict. */\n+\t\tsize_t len = i_size_read(\u0026dvnode-\u003enetfs.inode);\n+\t\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, dvnode-\u003edirectory, 0, 0,\n+\t\t\t\t round_up(len, PAGE_SIZE));\n+\t\tret = netfs_writeback_single(mapping, wbc, \u0026iter, len);\n \t}\n \n \tup_read(\u0026dvnode-\u003evalidate_lock);\ndiff --git a/fs/afs/dir_edit.c b/fs/afs/dir_edit.c\nindex fd3aa9f97ce64c..5f00c41403e5a5 100644\n--- a/fs/afs/dir_edit.c\n+++ b/fs/afs/dir_edit.c\n@@ -10,7 +10,6 @@\n #include \u003clinux/namei.h\u003e\n #include \u003clinux/pagemap.h\u003e\n #include \u003clinux/iversion.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n #include \"internal.h\"\n #include \"xdr_fs.h\"\n \n@@ -110,9 +109,8 @@ static void afs_clear_contig_bits(union afs_xdr_dir_block *block,\n */\n static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, size_t block)\n {\n-\tstruct folio_queue *fq;\n \tstruct afs_vnode *dvnode = iter-\u003edvnode;\n-\tstruct folio *folio;\n+\tstruct bvecq *bq;\n \tsize_t blpos = block * AFS_DIR_BLOCK_SIZE;\n \tsize_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter-\u003efpos;\n \tint ret;\n@@ -120,41 +118,38 @@ static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, siz\n \tif (dvnode-\u003edirectory_size \u003c blend) {\n \t\tsize_t cur_size = dvnode-\u003edirectory_size;\n \n-\t\tret = netfs_alloc_folioq_buffer(\n-\t\t\tNULL, \u0026dvnode-\u003edirectory, \u0026cur_size, blend,\n-\t\t\tmapping_gfp_mask(dvnode-\u003enetfs.inode.i_mapping));\n+\t\tret = bvecq_expand_buffer(\u0026dvnode-\u003edirectory, \u0026cur_size, blend,\n+\t\t\t\t\t GFP_KERNEL);\n \t\tdvnode-\u003edirectory_size = cur_size;\n \t\tif (ret \u003c 0)\n \t\t\tgoto fail;\n \t}\n \n-\tfq = iter-\u003efq;\n-\tif (!fq)\n-\t\tfq = dvnode-\u003edirectory;\n+\tbq = iter-\u003ebq;\n+\tif (!bq)\n+\t\tbq = dvnode-\u003edirectory;\n \n-\t/* Search the folio queue for the folio containing the block... */\n-\tfor (; fq; fq = fq-\u003enext) {\n-\t\tfor (int s = iter-\u003efq_slot; s \u003c folioq_count(fq); s++) {\n-\t\t\tsize_t fsize = folioq_folio_size(fq, s);\n+\t/* Search the contents for the region containing the block... */\n+\tfor (; bq; bq = bq-\u003enext) {\n+\t\tfor (int s = iter-\u003ebq_slot; s \u003c bq-\u003enr_slots; s++) {\n+\t\t\tstruct bio_vec *bv = \u0026bq-\u003ebv[s];\n+\t\t\tsize_t bsize = bv-\u003ebv_len;\n \n-\t\t\tif (blend \u003c= fpos + fsize) {\n+\t\t\tif (blend \u003c= fpos + bsize) {\n \t\t\t\t/* ... and then return the mapped block. */\n-\t\t\t\tfolio = folioq_folio(fq, s);\n-\t\t\t\tif (WARN_ON_ONCE(folio_pos(folio) != fpos))\n-\t\t\t\t\tgoto fail;\n-\t\t\t\titer-\u003efq = fq;\n-\t\t\t\titer-\u003efq_slot = s;\n+\t\t\t\titer-\u003ebq = bq;\n+\t\t\t\titer-\u003ebq_slot = s;\n \t\t\t\titer-\u003efpos = fpos;\n-\t\t\t\treturn kmap_local_folio(folio, blpos - fpos);\n+\t\t\t\treturn bvec_kmap_partial(bv, blpos - fpos);\n \t\t\t}\n-\t\t\tfpos += fsize;\n+\t\t\tfpos += bsize;\n \t\t}\n-\t\titer-\u003efq_slot = 0;\n+\t\titer-\u003ebq_slot = 0;\n \t}\n \n fail:\n-\titer-\u003efq = NULL;\n-\titer-\u003efq_slot = 0;\n+\titer-\u003ebq = NULL;\n+\titer-\u003ebq_slot = 0;\n \tafs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);\n \treturn NULL;\n }\n@@ -415,7 +410,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \tif (!afs_dir_init_iter(\u0026iter, name))\n \t\treturn;\n \n-\tmeta = afs_dir_find_block(\u0026iter, 0);\n+\tmeta = afs_dir_get_block(\u0026iter, 0);\n \tif (!meta)\n \t\treturn;\n \n@@ -442,7 +437,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \t/* Check and clear the entry. */\n \tde = \u0026block-\u003edirents[slot];\n \tif (de-\u003eu.valid != 1)\n-\t\tgoto error_unmap;\n+\t\tgoto error;\n \n \ttrace_afs_edit_dir(vnode, why, afs_edit_dir_delete, b, slot,\n \t\t\t ntohl(de-\u003eu.vnode), ntohl(de-\u003eu.unique),\n@@ -458,7 +453,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \t/* Clear the constituent entries. */\n \tnext = de-\u003eu.hash_next;\n \tmemset(de, 0, sizeof(*de) * iter.nr_slots);\n-\tkunmap_local(block);\n \n \t/* Adjust the hash chain: if iter-\u003eprev_entry is 0, the hashtable head\n \t * index is previous; otherwise it's slot number of the previous entry.\n@@ -485,7 +479,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \t\tpde = \u0026pblock-\u003edirents[ps];\n \t\tprev_next = pde-\u003eu.hash_next;\n \t\tif (prev_next != htons(entry)) {\n-\t\t\tkunmap_local(pblock);\n \t\t\tpr_warn(\"%llx:%llx:%x: not prev in chain b=%x p=%x,%x e=%x %*s\",\n \t\t\t\tvnode-\u003efid.vid, vnode-\u003efid.vnode, vnode-\u003efid.unique,\n \t\t\t\titer.bucket, iter.prev_entry, prev_next, entry,\n@@ -493,7 +486,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \t\t\tgoto error;\n \t\t}\n \t\tpde-\u003eu.hash_next = next;\n-\t\tkunmap_local(pblock);\n \t}\n \n \tnetfs_single_mark_inode_dirty(\u0026vnode-\u003enetfs.inode);\n@@ -503,18 +495,16 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,\n \t_debug(\"Remove %s from %u[%u]\", name-\u003ename, b, slot);\n \n out_unmap:\n+\tafs_dir_end_iter(\u0026iter);\n \tkunmap_local(meta);\n \t_leave(\"\");\n \treturn;\n \n already_invalidated:\n-\tkunmap_local(block);\n \ttrace_afs_edit_dir(vnode, why, afs_edit_dir_delete_inval,\n \t\t\t 0, 0, 0, 0, name-\u003ename);\n \tgoto out_unmap;\n \n-error_unmap:\n-\tkunmap_local(block);\n error:\n \ttrace_afs_edit_dir(vnode, why, afs_edit_dir_delete_error,\n \t\t\t 0, 0, 0, 0, name-\u003ename);\ndiff --git a/fs/afs/dir_search.c b/fs/afs/dir_search.c\nindex 104411c0692f57..e1f91b3ac6e25e 100644\n--- a/fs/afs/dir_search.c\n+++ b/fs/afs/dir_search.c\n@@ -66,53 +66,47 @@ bool afs_dir_init_iter(struct afs_dir_iter *iter, const struct qstr *name)\n */\n union afs_xdr_dir_block *afs_dir_find_block(struct afs_dir_iter *iter, size_t block)\n {\n-\tstruct folio_queue *fq = iter-\u003efq;\n \tstruct afs_vnode *dvnode = iter-\u003edvnode;\n-\tstruct folio *folio;\n+\tstruct bvecq *bq = iter-\u003ebq;\n \tsize_t blpos = block * AFS_DIR_BLOCK_SIZE;\n \tsize_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter-\u003efpos;\n-\tint slot = iter-\u003efq_slot;\n+\tint slot = iter-\u003ebq_slot;\n \n \t_enter(\"%zx,%d\", block, slot);\n \n-\tif (iter-\u003eblock) {\n-\t\tkunmap_local(iter-\u003eblock);\n-\t\titer-\u003eblock = NULL;\n-\t}\n+\tafs_dir_end_iter(iter);\n \n \tif (dvnode-\u003edirectory_size \u003c blend)\n \t\tgoto fail;\n \n-\tif (!fq || blpos \u003c fpos) {\n-\t\tfq = dvnode-\u003edirectory;\n+\tif (!bq || blpos \u003c fpos) {\n+\t\tbq = dvnode-\u003edirectory;\n \t\tslot = 0;\n \t\tfpos = 0;\n \t}\n \n \t/* Search the folio queue for the folio containing the block... */\n-\tfor (; fq; fq = fq-\u003enext) {\n-\t\tfor (; slot \u003c folioq_count(fq); slot++) {\n-\t\t\tsize_t fsize = folioq_folio_size(fq, slot);\n+\tfor (; bq; bq = bq-\u003enext) {\n+\t\tfor (; slot \u003c bq-\u003enr_slots; slot++) {\n+\t\t\tstruct bio_vec *bv = \u0026bq-\u003ebv[slot];\n+\t\t\tsize_t bsize = bv-\u003ebv_len;\n \n-\t\t\tif (blend \u003c= fpos + fsize) {\n+\t\t\tif (blend \u003c= fpos + bsize) {\n \t\t\t\t/* ... and then return the mapped block. */\n-\t\t\t\tfolio = folioq_folio(fq, slot);\n-\t\t\t\tif (WARN_ON_ONCE(folio_pos(folio) != fpos))\n-\t\t\t\t\tgoto fail;\n-\t\t\t\titer-\u003efq = fq;\n-\t\t\t\titer-\u003efq_slot = slot;\n+\t\t\t\titer-\u003ebq = bq;\n+\t\t\t\titer-\u003ebq_slot = slot;\n \t\t\t\titer-\u003efpos = fpos;\n-\t\t\t\titer-\u003eblock = kmap_local_folio(folio, blpos - fpos);\n+\t\t\t\titer-\u003eblock = bvec_kmap_partial(bv, blpos - fpos);\n \t\t\t\treturn iter-\u003eblock;\n \t\t\t}\n-\t\t\tfpos += fsize;\n+\t\t\tfpos += bsize;\n \t\t}\n \t\tslot = 0;\n \t}\n \n fail:\n-\titer-\u003efq = NULL;\n-\titer-\u003efq_slot = 0;\n+\titer-\u003ebq = NULL;\n+\titer-\u003ebq_slot = 0;\n \tafs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);\n \treturn NULL;\n }\n@@ -173,12 +167,8 @@ int afs_dir_search_bucket(struct afs_dir_iter *iter, const struct qstr *name,\n \n \tret = -ENOENT;\n found:\n-\tif (iter-\u003eblock) {\n-\t\tkunmap_local(iter-\u003eblock);\n-\t\titer-\u003eblock = NULL;\n-\t}\n-\n bad:\n+\tafs_dir_end_iter(iter);\n \tif (ret == -ESTALE)\n \t\tafs_invalidate_dir(iter-\u003edvnode, afs_dir_invalid_iter_stale);\n \t_leave(\" = %d\", ret);\ndiff --git a/fs/afs/file.c b/fs/afs/file.c\nindex 0467742bfeee34..476b9f3a776ebf 100644\n--- a/fs/afs/file.c\n+++ b/fs/afs/file.c\n@@ -332,11 +332,12 @@ void afs_fetch_data_immediate_cancel(struct afs_call *call)\n /*\n * Fetch file data from the volume.\n */\n-static void afs_issue_read(struct netfs_io_subrequest *subreq)\n+static int afs_issue_read(struct netfs_io_subrequest *subreq)\n {\n \tstruct afs_operation *op;\n \tstruct afs_vnode *vnode = AFS_FS_I(subreq-\u003erreq-\u003einode);\n \tstruct key *key = subreq-\u003erreq-\u003enetfs_priv;\n+\tint ret;\n \n \t_enter(\"%s{%llx:%llu.%u},%x,,,\",\n \t vnode-\u003evolume-\u003ename,\n@@ -345,11 +346,15 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)\n \t vnode-\u003efid.unique,\n \t key_serial(key));\n \n+\tret = netfs_prepare_read_buffer(subreq, INT_MAX);\n+\tif (ret \u003c 0)\n+\t\treturn ret;\n+\t/* After this point, must fail by termination. */\n+\n \top = afs_alloc_operation(key, vnode-\u003evolume);\n \tif (IS_ERR(op)) {\n-\t\tsubreq-\u003eerror = PTR_ERR(op);\n-\t\tnetfs_read_subreq_terminated(subreq);\n-\t\treturn;\n+\t\tret = PTR_ERR(op);\n+\t\tgoto failed;\n \t}\n \n \tafs_op_set_vnode(op, 0, vnode);\n@@ -364,20 +369,24 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)\n \t\top-\u003eflags |= AFS_OPERATION_ASYNC;\n \n \t\tif (!afs_begin_vnode_operation(op)) {\n-\t\t\tsubreq-\u003eerror = afs_put_operation(op);\n-\t\t\tnetfs_read_subreq_terminated(subreq);\n-\t\t\treturn;\n+\t\t\tret = afs_put_operation(op);\n+\t\t\tgoto failed;\n \t\t}\n \n \t\tif (!afs_select_fileserver(op)) {\n-\t\t\tafs_end_read(op);\n-\t\t\treturn;\n+\t\t\tafs_end_read(op); /* Error recorded here. */\n+\t\t\treturn 0;\n \t\t}\n \n \t\tafs_issue_read_call(op);\n \t} else {\n \t\tafs_do_sync_operation(op);\n \t}\n+\treturn 0;\n+failed:\n+\tsubreq-\u003eerror = ret;\n+\tnetfs_read_subreq_terminated(subreq);\n+\treturn 0;\n }\n \n static int afs_init_request(struct netfs_io_request *rreq, struct file *file)\n@@ -400,7 +409,6 @@ static int afs_init_request(struct netfs_io_request *rreq, struct file *file)\n \t\t}\n \t\tbreak;\n \tcase NETFS_WRITEBACK:\n-\tcase NETFS_WRITETHROUGH:\n \tcase NETFS_UNBUFFERED_WRITE:\n \tcase NETFS_DIO_WRITE:\n \t\tif (S_ISREG(rreq-\u003einode-\u003ei_mode))\n@@ -448,7 +456,6 @@ void afs_set_i_size(struct afs_vnode *vnode, loff_t new_i_size)\n \t}\n \tspin_unlock(\u0026inode-\u003ei_lock);\n \twrite_sequnlock(\u0026vnode-\u003ecb_lock);\n-\tfscache_update_cookie(afs_vnode_cache(vnode), NULL, \u0026new_i_size);\n }\n \n static void afs_update_i_size(struct inode *inode, loff_t new_i_size)\n@@ -471,7 +478,7 @@ const struct netfs_request_ops afs_req_ops = {\n \t.update_i_size\t\t= afs_update_i_size,\n \t.invalidate_cache\t= afs_netfs_invalidate_cache,\n \t.begin_writeback\t= afs_begin_writeback,\n-\t.prepare_write\t\t= afs_prepare_write,\n+\t.estimate_write\t\t= afs_estimate_write,\n \t.issue_write\t\t= afs_issue_write,\n \t.retry_request\t\t= afs_retry_request,\n };\ndiff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c\nindex a2ffd60889f89d..c332b733d7a733 100644\n--- a/fs/afs/fsclient.c\n+++ b/fs/afs/fsclient.c\n@@ -339,7 +339,9 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)\n \t\tif (call-\u003eremaining == 0)\n \t\t\tgoto no_more_data;\n \n-\t\tcall-\u003eiter = \u0026subreq-\u003eio_iter;\n+\t\tiov_iter_bvec_queue(\u0026call-\u003edef_iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n \t\tcall-\u003eiov_len = umin(call-\u003eremaining, subreq-\u003elen - subreq-\u003etransferred);\n \t\tcall-\u003eunmarshall++;\n \t\tfallthrough;\n@@ -1085,7 +1087,7 @@ static void afs_fs_store_data64(struct afs_operation *op)\n \tif (!call)\n \t\treturn afs_op_nomem(op);\n \n-\tcall-\u003ewrite_iter = op-\u003estore.write_iter;\n+\tcall-\u003ewrite_iter = \u0026op-\u003estore.write_iter;\n \n \t/* marshall the parameters */\n \tbp = call-\u003erequest;\n@@ -1139,7 +1141,7 @@ void afs_fs_store_data(struct afs_operation *op)\n \tif (!call)\n \t\treturn afs_op_nomem(op);\n \n-\tcall-\u003ewrite_iter = op-\u003estore.write_iter;\n+\tcall-\u003ewrite_iter = \u0026op-\u003estore.write_iter;\n \n \t/* marshall the parameters */\n \tbp = call-\u003erequest;\ndiff --git a/fs/afs/inode.c b/fs/afs/inode.c\nindex 14f39a9bea6cfe..634fbf8eb21206 100644\n--- a/fs/afs/inode.c\n+++ b/fs/afs/inode.c\n@@ -683,7 +683,7 @@ void afs_evict_inode(struct inode *inode)\n \tflush_delayed_work(\u0026vnode-\u003elock_work);\n \tnetfs_wait_for_outstanding_io(inode);\n \ttruncate_inode_pages_final(\u0026inode-\u003ei_data);\n-\tnetfs_free_folioq_buffer(vnode-\u003edirectory);\n+\tbvecq_put(vnode-\u003edirectory);\n \tif (vnode-\u003esymlink)\n \t\tafs_evict_symlink(vnode);\n \ndiff --git a/fs/afs/internal.h b/fs/afs/internal.h\nindex 601f01e5c15fcc..0d4e61504ef755 100644\n--- a/fs/afs/internal.h\n+++ b/fs/afs/internal.h\n@@ -710,7 +710,7 @@ struct afs_vnode {\n #define AFS_VNODE_MODIFYING\t10\t\t/* Set if we're performing a modification op */\n #define AFS_VNODE_DIR_READ\t11\t\t/* Set if we've read a dir's contents */\n \n-\tstruct folio_queue\t*directory;\t/* Directory contents */\n+\tstruct bvecq\t\t*directory;\t/* Directory contents */\n \tstruct afs_symlink __rcu *symlink;\t/* Symlink content */\n \tstruct list_head\twb_keys;\t/* List of keys available for writeback */\n \tstruct list_head\tpending_locks;\t/* locks waiting to be granted */\n@@ -914,7 +914,7 @@ struct afs_operation {\n \t\t\tafs_lock_type_t type;\n \t\t} lock;\n \t\tstruct {\n-\t\t\tstruct iov_iter\t*write_iter;\n+\t\t\tstruct iov_iter\twrite_iter;\n \t\t\tloff_t\tpos;\n \t\t\tloff_t\tsize;\n \t\t\tloff_t\ti_size;\n@@ -991,9 +991,9 @@ static inline void afs_invalidate_cache(struct afs_vnode *vnode, unsigned int fl\n struct afs_dir_iter {\n \tstruct afs_vnode\t*dvnode;\n \tunion afs_xdr_dir_block *block;\n-\tstruct folio_queue\t*fq;\n+\tstruct bvecq\t\t*bq;\n \tunsigned int\t\tfpos;\n-\tint\t\t\tfq_slot;\n+\tint\t\t\tbq_slot;\n \tunsigned int\t\tloop_check;\n \tu8\t\t\tnr_slots;\n \tu8\t\t\tbucket;\n@@ -1133,6 +1133,14 @@ int afs_dir_search_bucket(struct afs_dir_iter *iter, const struct qstr *name,\n int afs_dir_search(struct afs_vnode *dvnode, const struct qstr *name,\n \t\t struct afs_fid *_fid, afs_dataversion_t *_dir_version);\n \n+static inline void afs_dir_end_iter(struct afs_dir_iter *iter)\n+{\n+\tif (iter-\u003eblock) {\n+\t\tkunmap_local(iter-\u003eblock);\n+\t\titer-\u003eblock = NULL;\n+\t}\n+}\n+\n /*\n * dir_silly.c\n */\n@@ -1697,8 +1705,10 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *);\n /*\n * write.c\n */\n-void afs_prepare_write(struct netfs_io_subrequest *subreq);\n-void afs_issue_write(struct netfs_io_subrequest *subreq);\n+int afs_estimate_write(struct netfs_io_request *wreq,\n+\t\t struct netfs_io_stream *stream,\n+\t\t struct netfs_write_estimate *estimate);\n+int afs_issue_write(struct netfs_io_subrequest *subreq);\n void afs_begin_writeback(struct netfs_io_request *wreq);\n void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stream);\n extern int afs_writepages(struct address_space *, struct writeback_control *);\ndiff --git a/fs/afs/symlink.c b/fs/afs/symlink.c\nindex 16b4823cb7b7e7..ddd3307adccf3f 100644\n--- a/fs/afs/symlink.c\n+++ b/fs/afs/symlink.c\n@@ -56,7 +56,6 @@ void afs_evict_symlink(struct afs_vnode *vnode)\n void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)\n {\n \tstruct afs_symlink *symlink = op-\u003ecreate.symlink;\n-\tsize_t dsize = 0;\n \tsize_t size = strlen(symlink-\u003econtent) + 1;\n \tchar *p;\n \n@@ -66,13 +65,15 @@ void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)\n \tif (!fscache_cookie_enabled(netfs_i_cookie(\u0026vnode-\u003enetfs)))\n \t\treturn;\n \n-\tif (netfs_alloc_folioq_buffer(NULL, \u0026vnode-\u003edirectory, \u0026dsize, size,\n-\t\t\t\t mapping_gfp_mask(vnode-\u003enetfs.inode.i_mapping)) \u003c 0)\n+\tvnode-\u003edirectory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL);\n+\tif (!vnode-\u003edirectory)\n \t\treturn;\n \n-\tvnode-\u003edirectory_size = dsize;\n-\tp = kmap_local_folio(folioq_folio(vnode-\u003edirectory, 0), 0);\n+\tvnode-\u003edirectory_size = size;\n+\tp = bvec_kmap_partial(\u0026vnode-\u003edirectory-\u003ebv[0], 0);\n \tmemcpy(p, symlink-\u003econtent, size);\n+\tif (size \u003c PAGE_SIZE)\n+\t\tmemset(p + size, 0, PAGE_SIZE - size);\n \tkunmap_local(p);\n \tnetfs_single_mark_inode_dirty(\u0026vnode-\u003enetfs.inode);\n }\n@@ -94,17 +95,12 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)\n \t}\n \n \tif (!vnode-\u003edirectory) {\n-\t\tsize_t cur_size = 0;\n-\n-\t\tret = netfs_alloc_folioq_buffer(NULL,\n-\t\t\t\t\t\t\u0026vnode-\u003edirectory, \u0026cur_size, PAGE_SIZE,\n-\t\t\t\t\t\tmapping_gfp_mask(vnode-\u003enetfs.inode.i_mapping));\n-\t\tvnode-\u003edirectory_size = PAGE_SIZE - 1;\n-\t\tif (ret \u003c 0)\n-\t\t\treturn ret;\n+\t\tvnode-\u003edirectory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL);\n+\t\tif (!vnode-\u003edirectory)\n+\t\t\treturn -ENOMEM;\n \t}\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_DEST, vnode-\u003edirectory, 0, 0, PAGE_SIZE);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, vnode-\u003edirectory, 0, 0, PAGE_SIZE);\n \n \t/* AFS requires us to perform the read of a symlink as a single unit to\n \t * avoid issues with the content being changed between reads.\n@@ -127,7 +123,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)\n \t\trefcount_set(\u0026symlink-\u003eref, 1);\n \t\tsymlink-\u003econtent[i_size] = 0;\n \n-\t\tconst char *s = kmap_local_folio(folioq_folio(vnode-\u003edirectory, 0), 0);\n+\t\tconst char *s = bvec_kmap_partial(\u0026vnode-\u003edirectory-\u003ebv[0], 0);\n \n \t\tmemcpy(symlink-\u003econtent, s, i_size);\n \t\tkunmap_local(s);\n@@ -136,7 +132,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)\n \t}\n \n \tif (!fscache_cookie_enabled(netfs_i_cookie(\u0026vnode-\u003enetfs))) {\n-\t\tnetfs_free_folioq_buffer(vnode-\u003edirectory);\n+\t\tbvecq_put(vnode-\u003edirectory);\n \t\tvnode-\u003edirectory = NULL;\n \t\tvnode-\u003edirectory_size = 0;\n \t}\n@@ -249,14 +245,15 @@ int afs_symlink_writepages(struct address_space *mapping,\n \n \tif (vnode-\u003edirectory \u0026\u0026\n \t atomic64_read(\u0026vnode-\u003ecb_expires_at) != AFS_NO_CB_PROMISE) {\n-\t\tiov_iter_folio_queue(\u0026iter, ITER_SOURCE, vnode-\u003edirectory, 0, 0,\n-\t\t\t\t i_size_read(\u0026vnode-\u003enetfs.inode));\n-\t\tret = netfs_writeback_single(mapping, wbc, \u0026iter);\n+\t\tsize_t len = i_size_read(\u0026vnode-\u003enetfs.inode);\n+\t\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, vnode-\u003edirectory, 0, 0,\n+\t\t\t\t round_up(len, PAGE_SIZE));\n+\t\tret = netfs_writeback_single(mapping, wbc, \u0026iter, len);\n \t}\n \n \tif (ret == 0) {\n \t\tnetfs_wb_begin(\u0026vnode-\u003enetfs, false);\n-\t\tnetfs_free_folioq_buffer(vnode-\u003edirectory);\n+\t\tbvecq_put(vnode-\u003edirectory);\n \t\tvnode-\u003edirectory = NULL;\n \t\tvnode-\u003edirectory_size = 0;\n \t\tnetfs_wb_end(\u0026vnode-\u003enetfs);\ndiff --git a/fs/afs/write.c b/fs/afs/write.c\nindex 7f34b939706a07..46f81d9e7d7ff1 100644\n--- a/fs/afs/write.c\n+++ b/fs/afs/write.c\n@@ -83,17 +83,20 @@ static const struct afs_operation_ops afs_store_data_operation = {\n };\n \n /*\n- * Prepare a subrequest to write to the server. This sets the max_len\n- * parameter.\n+ * Estimate the maximum size of a write we can send to the server.\n */\n-void afs_prepare_write(struct netfs_io_subrequest *subreq)\n+int afs_estimate_write(struct netfs_io_request *wreq,\n+\t\t struct netfs_io_stream *stream,\n+\t\t struct netfs_write_estimate *estimate)\n {\n-\tstruct netfs_io_stream *stream = \u0026subreq-\u003erreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tunsigned long long limit = ULLONG_MAX - stream-\u003eissue_from;\n+\tunsigned long long max_len = 256 * 1024 * 1024;\n \n \t//if (test_bit(NETFS_SREQ_RETRYING, \u0026subreq-\u003eflags))\n-\t//\tsubreq-\u003emax_len = 512 * 1024;\n-\t//else\n-\tstream-\u003esreq_max_len = 256 * 1024 * 1024;\n+\t//\tmax_len = 512 * 1024;\n+\n+\testimate-\u003eissue_at = stream-\u003eissue_from + umin(max_len, limit);\n+\treturn 0;\n }\n \n /*\n@@ -139,12 +142,15 @@ static void afs_issue_write_worker(struct work_struct *work)\n \top-\u003eflags\t\t|= AFS_OPERATION_UNINTR;\n \top-\u003eops\t\t\t= \u0026afs_store_data_operation;\n \n+\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n \tafs_begin_vnode_operation(op);\n \n-\top-\u003estore.write_iter\t= \u0026subreq-\u003eio_iter;\n \top-\u003estore.i_size\t= umax(pos + len, netfs_read_remote_i_size(\u0026vnode-\u003enetfs.inode));\n \top-\u003emtime\t\t= inode_get_mtime(\u0026vnode-\u003enetfs.inode);\n \n+\tiov_iter_bvec_queue(\u0026op-\u003estore.write_iter, ITER_SOURCE, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n \tafs_wait_for_operation(op);\n \tret = afs_put_operation(op);\n \tswitch (ret) {\n@@ -168,11 +174,21 @@ static void afs_issue_write_worker(struct work_struct *work)\n \tnetfs_write_subrequest_terminated(subreq, ret \u003c 0 ? ret : subreq-\u003elen);\n }\n \n-void afs_issue_write(struct netfs_io_subrequest *subreq)\n+int afs_issue_write(struct netfs_io_subrequest *subreq)\n {\n+\tint ret;\n+\n+\tif (subreq-\u003elen \u003e 256 * 1024 * 1024)\n+\t\tsubreq-\u003elen = 256 * 1024 * 1024;\n+\tret = netfs_prepare_write_buffer(subreq, INT_MAX);\n+\tif (ret \u003c 0)\n+\t\treturn ret;\n+\t/* After this point, must fail by termination. */\n+\n \tsubreq-\u003ework.func = afs_issue_write_worker;\n \tif (!queue_work(system_dfl_wq, \u0026subreq-\u003ework))\n \t\tWARN_ON_ONCE(1);\n+\treturn 0;\n }\n \n /*\n@@ -183,6 +199,8 @@ void afs_begin_writeback(struct netfs_io_request *wreq)\n {\n \tif (S_ISREG(wreq-\u003einode-\u003ei_mode))\n \t\tafs_get_writeback_key(wreq);\n+\n+\twreq-\u003eio_streams[0].avail = true;\n }\n \n /*\ndiff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c\nindex d941179730a989..52c588092050de 100644\n--- a/fs/afs/yfsclient.c\n+++ b/fs/afs/yfsclient.c\n@@ -385,7 +385,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)\n \t\tif (call-\u003eremaining == 0)\n \t\t\tgoto no_more_data;\n \n-\t\tcall-\u003eiter = \u0026subreq-\u003eio_iter;\n+\t\tiov_iter_bvec_queue(\u0026call-\u003edef_iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n \t\tcall-\u003eiov_len = min(call-\u003eremaining, subreq-\u003elen - subreq-\u003etransferred);\n \t\tcall-\u003eunmarshall++;\n \t\tfallthrough;\n@@ -1357,7 +1359,7 @@ void yfs_fs_store_data(struct afs_operation *op)\n \tif (!call)\n \t\treturn afs_op_nomem(op);\n \n-\tcall-\u003ewrite_iter = op-\u003estore.write_iter;\n+\tcall-\u003ewrite_iter = \u0026op-\u003estore.write_iter;\n \n \t/* marshall the parameters */\n \tbp = call-\u003erequest;\ndiff --git a/fs/cachefiles/Kconfig b/fs/cachefiles/Kconfig\nindex c5a070550ee334..afb25b6af5aa17 100644\n--- a/fs/cachefiles/Kconfig\n+++ b/fs/cachefiles/Kconfig\n@@ -26,15 +26,3 @@ config CACHEFILES_ERROR_INJECTION\n \thelp\n \t This permits error injection to be enabled in cachefiles whilst a\n \t cache is in service.\n-\n-config CACHEFILES_ONDEMAND\n-\tbool \"Support for on-demand read\"\n-\tdepends on CACHEFILES\n-\tdefault n\n-\thelp\n-\t This permits userspace to enable the cachefiles on-demand read mode.\n-\t In this mode, when a cache miss occurs, responsibility for fetching\n-\t the data lies with the cachefiles backend instead of with the netfs\n-\t and is delegated to userspace.\n-\n-\t If unsure, say N.\ndiff --git a/fs/cachefiles/Makefile b/fs/cachefiles/Makefile\nindex c37a7a9af10b60..16d811f1a2faed 100644\n--- a/fs/cachefiles/Makefile\n+++ b/fs/cachefiles/Makefile\n@@ -16,6 +16,5 @@ cachefiles-y := \\\n \txattr.o\n \n cachefiles-$(CONFIG_CACHEFILES_ERROR_INJECTION) += error_inject.o\n-cachefiles-$(CONFIG_CACHEFILES_ONDEMAND) += ondemand.o\n \n obj-$(CONFIG_CACHEFILES) := cachefiles.o\ndiff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c\nindex 4117b145ac9431..6e0248a62c1397 100644\n--- a/fs/cachefiles/daemon.c\n+++ b/fs/cachefiles/daemon.c\n@@ -76,10 +76,6 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = {\n \t{ \"inuse\",\tcachefiles_daemon_inuse\t\t},\n \t{ \"secctx\",\tcachefiles_daemon_secctx\t},\n \t{ \"tag\",\tcachefiles_daemon_tag\t\t},\n-#ifdef CONFIG_CACHEFILES_ONDEMAND\n-\t{ \"copen\",\tcachefiles_ondemand_copen\t},\n-\t{ \"restore\",\tcachefiles_ondemand_restore\t},\n-#endif\n \t{ \"\",\t\tNULL\t\t\t\t}\n };\n \n@@ -114,8 +110,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)\n \tINIT_LIST_HEAD(\u0026cache-\u003eobject_list);\n \tspin_lock_init(\u0026cache-\u003eobject_list_lock);\n \trefcount_set(\u0026cache-\u003eunbind_pincount, 1);\n-\txa_init_flags(\u0026cache-\u003ereqs, XA_FLAGS_ALLOC);\n-\txa_init_flags(\u0026cache-\u003eondemand_ids, XA_FLAGS_ALLOC1);\n \n \t/* set default caching limits\n \t * - limit at 1% free space and/or free files\n@@ -134,40 +128,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)\n \treturn 0;\n }\n \n-void cachefiles_flush_reqs(struct cachefiles_cache *cache)\n-{\n-\tstruct xarray *xa = \u0026cache-\u003ereqs;\n-\tstruct cachefiles_req *req;\n-\tunsigned long index;\n-\n-\t/*\n-\t * Make sure the following two operations won't be reordered.\n-\t * 1) set CACHEFILES_DEAD bit\n-\t * 2) flush requests in the xarray\n-\t * Otherwise the request may be enqueued after xarray has been\n-\t * flushed, leaving the orphan request never being completed.\n-\t *\n-\t * CPU 1\t\t\tCPU 2\n-\t * =====\t\t\t=====\n-\t * flush requests in the xarray\n-\t *\t\t\t\ttest CACHEFILES_DEAD bit\n-\t *\t\t\t\tenqueue the request\n-\t * set CACHEFILES_DEAD bit\n-\t */\n-\tsmp_mb();\n-\n-\txa_lock(xa);\n-\txa_for_each(xa, index, req) {\n-\t\treq-\u003eerror = -EIO;\n-\t\tcomplete(\u0026req-\u003edone);\n-\t\t__xa_erase(xa, index);\n-\t}\n-\txa_unlock(xa);\n-\n-\txa_destroy(\u0026cache-\u003ereqs);\n-\txa_destroy(\u0026cache-\u003eondemand_ids);\n-}\n-\n void cachefiles_put_unbind_pincount(struct cachefiles_cache *cache)\n {\n \tif (refcount_dec_and_test(\u0026cache-\u003eunbind_pincount)) {\n@@ -195,9 +155,6 @@ static int cachefiles_daemon_release(struct inode *inode, struct file *file)\n \n \tset_bit(CACHEFILES_DEAD, \u0026cache-\u003eflags);\n \n-\tif (cachefiles_in_ondemand_mode(cache))\n-\t\tcachefiles_flush_reqs(cache);\n-\n \t/* clean up the control file interface */\n \tcache-\u003ecachefilesd = NULL;\n \tfile-\u003eprivate_data = NULL;\n@@ -266,10 +223,7 @@ static ssize_t cachefiles_daemon_read(struct file *file, char __user *_buffer,\n \tif (!test_bit(CACHEFILES_READY, \u0026cache-\u003eflags))\n \t\treturn 0;\n \n-\tif (cachefiles_in_ondemand_mode(cache))\n-\t\treturn cachefiles_ondemand_daemon_read(cache, _buffer, buflen);\n-\telse\n-\t\treturn cachefiles_do_daemon_read(cache, _buffer, buflen);\n+\treturn cachefiles_do_daemon_read(cache, _buffer, buflen);\n }\n \n /*\n@@ -358,28 +312,13 @@ static __poll_t cachefiles_daemon_poll(struct file *file,\n \t\t\t\t\t struct poll_table_struct *poll)\n {\n \tstruct cachefiles_cache *cache = file-\u003eprivate_data;\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, 0);\n-\tstruct cachefiles_req *req;\n \t__poll_t mask;\n \n \tpoll_wait(file, \u0026cache-\u003edaemon_pollwq, poll);\n \tmask = 0;\n \n-\tif (cachefiles_in_ondemand_mode(cache)) {\n-\t\tif (!xa_empty(\u0026cache-\u003ereqs)) {\n-\t\t\txas_lock(\u0026xas);\n-\t\t\txas_for_each_marked(\u0026xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {\n-\t\t\t\tif (!cachefiles_ondemand_is_reopening_read(req)) {\n-\t\t\t\t\tmask |= EPOLLIN;\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\t\t\t}\n-\t\t\txas_unlock(\u0026xas);\n-\t\t}\n-\t} else {\n-\t\tif (test_bit(CACHEFILES_STATE_CHANGED, \u0026cache-\u003eflags))\n-\t\t\tmask |= EPOLLIN;\n-\t}\n+\tif (test_bit(CACHEFILES_STATE_CHANGED, \u0026cache-\u003eflags))\n+\t\tmask |= EPOLLIN;\n \n \tif (test_bit(CACHEFILES_CULLING, \u0026cache-\u003eflags))\n \t\tmask |= EPOLLOUT;\n@@ -779,14 +718,7 @@ static int cachefiles_daemon_bind(struct cachefiles_cache *cache, char *args)\n \t\treturn -EBUSY;\n \t}\n \n-\tif (IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND)) {\n-\t\tif (!strcmp(args, \"ondemand\")) {\n-\t\t\tset_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags);\n-\t\t} else if (*args) {\n-\t\t\tpr_err(\"Invalid argument to the 'bind' command\\n\");\n-\t\t\treturn -EINVAL;\n-\t\t}\n-\t} else if (*args) {\n+\tif (*args) {\n \t\tpr_err(\"'bind' command doesn't take an argument\\n\");\n \t\treturn -EINVAL;\n \t}\ndiff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c\nindex a08250d244ea25..638e00717119c8 100644\n--- a/fs/cachefiles/interface.c\n+++ b/fs/cachefiles/interface.c\n@@ -32,11 +32,6 @@ struct cachefiles_object *cachefiles_alloc_object(struct fscache_cookie *cookie)\n \tif (!object)\n \t\treturn NULL;\n \n-\tif (cachefiles_ondemand_init_obj_info(object, volume)) {\n-\t\tkmem_cache_free(cachefiles_object_jar, object);\n-\t\treturn NULL;\n-\t}\n-\n \trefcount_set(\u0026object-\u003eref, 1);\n \n \tspin_lock_init(\u0026object-\u003elock);\n@@ -94,7 +89,6 @@ void cachefiles_put_object(struct cachefiles_object *object,\n \t\tASSERTCMP(object-\u003efile, ==, NULL);\n \n \t\tkfree(object-\u003ed_name);\n-\t\tcachefiles_ondemand_deinit_obj_info(object);\n \t\tcache = object-\u003evolume-\u003ecache-\u003ecache;\n \t\tfscache_put_cookie(object-\u003ecookie, fscache_cookie_put_object);\n \t\tobject-\u003ecookie = NULL;\n@@ -105,73 +99,6 @@ void cachefiles_put_object(struct cachefiles_object *object,\n \t_leave(\"\");\n }\n \n-/*\n- * Adjust the size of a cache file if necessary to match the DIO size. We keep\n- * the EOF marker a multiple of DIO blocks so that we don't fall back to doing\n- * non-DIO for a partial block straddling the EOF, but we also have to be\n- * careful of someone expanding the file and accidentally accreting the\n- * padding.\n- */\n-static int cachefiles_adjust_size(struct cachefiles_object *object)\n-{\n-\tstruct iattr newattrs;\n-\tstruct file *file = object-\u003efile;\n-\tuint64_t ni_size;\n-\tloff_t oi_size;\n-\tint ret;\n-\n-\tni_size = object-\u003ecookie-\u003eobject_size;\n-\tni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);\n-\n-\t_enter(\"{OBJ%x},[%llu]\",\n-\t object-\u003edebug_id, (unsigned long long) ni_size);\n-\n-\tif (!file)\n-\t\treturn -ENOBUFS;\n-\n-\toi_size = i_size_read(file_inode(file));\n-\tif (oi_size == ni_size)\n-\t\treturn 0;\n-\n-\tinode_lock(file_inode(file));\n-\n-\t/* if there's an extension to a partial page at the end of the backing\n-\t * file, we need to discard the partial page so that we pick up new\n-\t * data after it */\n-\tif (oi_size \u0026 ~PAGE_MASK \u0026\u0026 ni_size \u003e oi_size) {\n-\t\t_debug(\"discard tail %llx\", oi_size);\n-\t\tnewattrs.ia_valid = ATTR_SIZE;\n-\t\tnewattrs.ia_size = oi_size \u0026 PAGE_MASK;\n-\t\tret = cachefiles_inject_remove_error();\n-\t\tif (ret == 0)\n-\t\t\tret = notify_change(\u0026nop_mnt_idmap, file-\u003ef_path.dentry,\n-\t\t\t\t\t \u0026newattrs, NULL);\n-\t\tif (ret \u003c 0)\n-\t\t\tgoto truncate_failed;\n-\t}\n-\n-\tnewattrs.ia_valid = ATTR_SIZE;\n-\tnewattrs.ia_size = ni_size;\n-\tret = cachefiles_inject_write_error();\n-\tif (ret == 0)\n-\t\tret = notify_change(\u0026nop_mnt_idmap, file-\u003ef_path.dentry,\n-\t\t\t\t \u0026newattrs, NULL);\n-\n-truncate_failed:\n-\tinode_unlock(file_inode(file));\n-\n-\tif (ret \u003c 0)\n-\t\ttrace_cachefiles_io_error(NULL, file_inode(file), ret,\n-\t\t\t\t\t cachefiles_trace_notify_change_error);\n-\tif (ret == -EIO) {\n-\t\tcachefiles_io_error_obj(object, \"Size set failed\");\n-\t\tret = -ENOBUFS;\n-\t}\n-\n-\t_leave(\" = %d\", ret);\n-\treturn ret;\n-}\n-\n /*\n * Attempt to look up the nominated node in this cache\n */\n@@ -204,7 +131,6 @@ static bool cachefiles_lookup_cookie(struct fscache_cookie *cookie)\n \tspin_lock(\u0026cache-\u003eobject_list_lock);\n \tlist_add(\u0026object-\u003ecache_link, \u0026cache-\u003eobject_list);\n \tspin_unlock(\u0026cache-\u003eobject_list_lock);\n-\tcachefiles_adjust_size(object);\n \n \tcachefiles_end_secure(cache, saved_cred);\n \t_leave(\" = t\");\n@@ -238,7 +164,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,\n \tloff_t i_size, dio_size;\n \tint ret;\n \n-\tdio_size = round_up(new_size, CACHEFILES_DIO_BLOCK_SIZE);\n+\tdio_size = round_up(new_size, cache-\u003ebsize);\n \ti_size = i_size_read(inode);\n \n \ttrace_cachefiles_trunc(object, inode, i_size, dio_size,\n@@ -270,6 +196,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,\n \t\t}\n \t}\n \n+\tobject-\u003eobject_size = new_size;\n \treturn true;\n }\n \n@@ -284,22 +211,31 @@ static void cachefiles_resize_cookie(struct netfs_cache_resources *cres,\n \tstruct fscache_cookie *cookie = object-\u003ecookie;\n \tconst struct cred *saved_cred;\n \tstruct file *file = cachefiles_cres_file(cres);\n-\tloff_t old_size = cookie-\u003eobject_size;\n+\tunsigned long long i_size = i_size_read(file_inode(file));\n \n-\t_enter(\"%llu-\u003e%llu\", old_size, new_size);\n+\t_enter(\"%llu-\u003e%llu\", object-\u003eobject_size, new_size);\n \n-\tif (new_size \u003c old_size) {\n+\t/* If the file is being shrunk, we need to downsize the backing file\n+\t * and clear the end of the final block.\n+\t */\n+\tif (new_size \u003c object-\u003eobject_size) {\n+\t\tif (new_size \u003e= i_size)\n+\t\t\tgoto out;\n \t\tcachefiles_begin_secure(cache, \u0026saved_cred);\n \t\tcachefiles_shorten_object(object, file, new_size);\n \t\tcachefiles_end_secure(cache, saved_cred);\n \t\tobject-\u003ecookie-\u003eobject_size = new_size;\n+\t\tif (new_size == 0)\n+\t\t\tobject-\u003econtent_info = CACHEFILES_CONTENT_NO_DATA;\n \t\treturn;\n \t}\n \n \t/* The file is being expanded. We don't need to do anything\n-\t * particularly. cookie-\u003einitial_size doesn't change and so the point\n-\t * at which we have to download before doesn't change.\n+\t * particularly. The tail of the last block should have been cleared\n+\t * both when it is written and when it is shrunk.\n \t */\n+out:\n+\tobject-\u003eobject_size = new_size;\n \tcookie-\u003eobject_size = new_size;\n }\n \n@@ -374,8 +310,6 @@ static void cachefiles_withdraw_cookie(struct fscache_cookie *cookie)\n \t\tspin_unlock(\u0026cache-\u003eobject_list_lock);\n \t}\n \n-\tcachefiles_ondemand_clean_object(object);\n-\n \tif (object-\u003efile) {\n \t\tcachefiles_begin_secure(cache, \u0026saved_cred);\n \t\tcachefiles_clean_up_object(object, cache);\ndiff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h\nindex b62cd3e9a18e4c..b31d7e93fe992b 100644\n--- a/fs/cachefiles/internal.h\n+++ b/fs/cachefiles/internal.h\n@@ -15,11 +15,8 @@\n #include \u003clinux/fscache-cache.h\u003e\n #include \u003clinux/cred.h\u003e\n #include \u003clinux/security.h\u003e\n-#include \u003clinux/xarray.h\u003e\n #include \u003clinux/cachefiles.h\u003e\n \n-#define CACHEFILES_DIO_BLOCK_SIZE 4096\n-\n struct cachefiles_cache;\n struct cachefiles_object;\n \n@@ -44,21 +41,6 @@ struct cachefiles_volume {\n \tstruct dentry\t\t\t*fanout[256];\t/* Fanout subdirs */\n };\n \n-enum cachefiles_object_state {\n-\tCACHEFILES_ONDEMAND_OBJSTATE_CLOSE, /* Anonymous fd closed by daemon or initial state */\n-\tCACHEFILES_ONDEMAND_OBJSTATE_OPEN, /* Anonymous fd associated with object is available */\n-\tCACHEFILES_ONDEMAND_OBJSTATE_REOPENING, /* Object that was closed and is being reopened. */\n-\tCACHEFILES_ONDEMAND_OBJSTATE_DROPPING, /* Object is being dropped. */\n-};\n-\n-struct cachefiles_ondemand_info {\n-\tstruct work_struct\t\tondemand_work;\n-\tint\t\t\t\tondemand_id;\n-\tenum cachefiles_object_state\tstate;\n-\tstruct cachefiles_object\t*object;\n-\tspinlock_t\t\t\tlock;\n-};\n-\n /*\n * Backing file state.\n */\n@@ -68,19 +50,19 @@ struct cachefiles_object {\n \tstruct list_head\t\tcache_link;\t/* Link in cache-\u003e*_list */\n \tstruct file\t\t\t*file;\t\t/* The file representing this object */\n \tchar\t\t\t\t*d_name;\t/* Backing file name */\n+\tunsigned long\t\t\tflags;\n+#define CACHEFILES_OBJECT_USING_TMPFILE\t0\t\t/* Have an unlinked tmpfile */\n+\tunsigned long long\t\tobject_size;\t/* Size of the object stored\n+\t\t\t\t\t\t\t * (independent of cookie-\u003eobject_size for\n+\t\t\t\t\t\t\t * coherency reasons)\n+\t\t\t\t\t\t\t */\n+\tatomic64_t\t\t\tread_limit;\t/* Point beyond which uncommitted writes */\n \tint\t\t\t\tdebug_id;\n \tspinlock_t\t\t\tlock;\n \trefcount_t\t\t\tref;\n-\tenum cachefiles_content\t\tcontent_info:8;\t/* Info about content presence */\n-\tunsigned long\t\t\tflags;\n-#define CACHEFILES_OBJECT_USING_TMPFILE\t0\t\t/* Have an unlinked tmpfile */\n-#ifdef CONFIG_CACHEFILES_ONDEMAND\n-\tstruct cachefiles_ondemand_info\t*ondemand;\n-#endif\n+\tenum cachefiles_content\t\tcontent_info;\t/* Info about content presence */\n };\n \n-#define CACHEFILES_ONDEMAND_ID_CLOSED\t-1\n-\n /*\n * Cache files cache definition\n */\n@@ -119,35 +101,13 @@ struct cachefiles_cache {\n #define CACHEFILES_DEAD\t\t\t1\t/* T if cache dead */\n #define CACHEFILES_CULLING\t\t2\t/* T if cull engaged */\n #define CACHEFILES_STATE_CHANGED\t3\t/* T if state changed (poll trigger) */\n-#define CACHEFILES_ONDEMAND_MODE\t4\t/* T if in on-demand read mode */\n \tchar\t\t\t\t*rootdirname;\t/* name of cache root directory */\n \tchar\t\t\t\t*tag;\t\t/* cache binding tag */\n \trefcount_t\t\t\tunbind_pincount;/* refcount to do daemon unbind */\n-\tstruct xarray\t\t\treqs;\t\t/* xarray of pending on-demand requests */\n-\tunsigned long\t\t\treq_id_next;\n-\tstruct xarray\t\t\tondemand_ids;\t/* xarray for ondemand_id allocation */\n-\tu32\t\t\t\tondemand_id_next;\n-\tu32\t\t\t\tmsg_id_next;\n \tu32\t\t\t\tsecid;\t\t/* LSM security id */\n \tbool\t\t\t\thave_secid;\t/* whether \"secid\" was set */\n };\n \n-static inline bool cachefiles_in_ondemand_mode(struct cachefiles_cache *cache)\n-{\n-\treturn IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND) \u0026\u0026\n-\t\ttest_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags);\n-}\n-\n-struct cachefiles_req {\n-\tstruct cachefiles_object *object;\n-\tstruct completion done;\n-\trefcount_t ref;\n-\tint error;\n-\tstruct cachefiles_msg msg;\n-};\n-\n-#define CACHEFILES_REQ_NEW\tXA_MARK_1\n-\n #include \u003ctrace/events/cachefiles.h\u003e\n \n static inline\n@@ -298,90 +258,6 @@ extern struct file *cachefiles_create_tmpfile(struct cachefiles_object *object);\n extern bool cachefiles_commit_tmpfile(struct cachefiles_cache *cache,\n \t\t\t\t struct cachefiles_object *object);\n \n-/*\n- * ondemand.c\n- */\n-#ifdef CONFIG_CACHEFILES_ONDEMAND\n-extern ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,\n-\t\t\t\t\tchar __user *_buffer, size_t buflen);\n-\n-extern int cachefiles_ondemand_copen(struct cachefiles_cache *cache,\n-\t\t\t\t char *args);\n-\n-extern int cachefiles_ondemand_restore(struct cachefiles_cache *cache,\n-\t\t\t\t\tchar *args);\n-\n-extern int cachefiles_ondemand_init_object(struct cachefiles_object *object);\n-extern void cachefiles_ondemand_clean_object(struct cachefiles_object *object);\n-\n-extern int cachefiles_ondemand_read(struct cachefiles_object *object,\n-\t\t\t\t loff_t pos, size_t len);\n-\n-extern int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,\n-\t\t\t\t\tstruct cachefiles_volume *volume);\n-extern void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj);\n-\n-#define CACHEFILES_OBJECT_STATE_FUNCS(_state, _STATE)\t\\\n-static inline bool\t\t\t\t\t\t\t\t\\\n-cachefiles_ondemand_object_is_##_state(const struct cachefiles_object *object) \\\n-{\t\t\t\t\t\t\t\t\t\t\t\t\\\n-\treturn object-\u003eondemand-\u003estate == CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \\\n-}\t\t\t\t\t\t\t\t\t\t\t\t\\\n-\t\t\t\t\t\t\t\t\t\t\t\t\\\n-static inline void\t\t\t\t\t\t\t\t\\\n-cachefiles_ondemand_set_object_##_state(struct cachefiles_object *object) \\\n-{\t\t\t\t\t\t\t\t\t\t\t\t\\\n-\tobject-\u003eondemand-\u003estate = CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \\\n-}\n-\n-CACHEFILES_OBJECT_STATE_FUNCS(open, OPEN);\n-CACHEFILES_OBJECT_STATE_FUNCS(close, CLOSE);\n-CACHEFILES_OBJECT_STATE_FUNCS(reopening, REOPENING);\n-CACHEFILES_OBJECT_STATE_FUNCS(dropping, DROPPING);\n-\n-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)\n-{\n-\treturn cachefiles_ondemand_object_is_reopening(req-\u003eobject) \u0026\u0026\n-\t\t\treq-\u003emsg.opcode == CACHEFILES_OP_READ;\n-}\n-\n-#else\n-static inline ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,\n-\t\t\t\t\tchar __user *_buffer, size_t buflen)\n-{\n-\treturn -EOPNOTSUPP;\n-}\n-\n-static inline int cachefiles_ondemand_init_object(struct cachefiles_object *object)\n-{\n-\treturn 0;\n-}\n-\n-static inline void cachefiles_ondemand_clean_object(struct cachefiles_object *object)\n-{\n-}\n-\n-static inline int cachefiles_ondemand_read(struct cachefiles_object *object,\n-\t\t\t\t\t loff_t pos, size_t len)\n-{\n-\treturn -EOPNOTSUPP;\n-}\n-\n-static inline int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,\n-\t\t\t\t\t\tstruct cachefiles_volume *volume)\n-{\n-\treturn 0;\n-}\n-static inline void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj)\n-{\n-}\n-\n-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)\n-{\n-\treturn false;\n-}\n-#endif\n-\n /*\n * security.c\n */\n@@ -430,8 +306,6 @@ do {\t\t\t\t\t\t\t\\\n \tpr_err(\"I/O Error: \" FMT\"\\n\", ##__VA_ARGS__);\t\\\n \tfscache_io_error((___cache)-\u003ecache);\t\t\\\n \tset_bit(CACHEFILES_DEAD, \u0026(___cache)-\u003eflags);\t\\\n-\tif (cachefiles_in_ondemand_mode(___cache))\t\\\n-\t\tcachefiles_flush_reqs(___cache);\t\\\n } while (0)\n \n #define cachefiles_io_error_obj(object, FMT, ...)\t\t\t\\\ndiff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c\nindex d879b80a0bedc9..f289a3bddd63bb 100644\n--- a/fs/cachefiles/io.c\n+++ b/fs/cachefiles/io.c\n@@ -26,12 +26,17 @@ struct cachefiles_kiocb {\n \t};\n \tstruct cachefiles_object *object;\n \tnetfs_io_terminated_t\tterm_func;\n-\tvoid\t\t\t*term_func_priv;\n+\tunion {\n+\t\tstruct netfs_io_subrequest *subreq;\n+\t\tvoid\t\t\t*term_func_priv;\n+\t};\n \tbool\t\t\twas_async;\n \tunsigned int\t\tinval_counter;\t/* Copy of cookie-\u003einval_counter */\n \tu64\t\t\tb_writing;\n };\n \n+#define IS_ERR_VALUE_LL(x) unlikely((x) \u003e= (unsigned long long)-MAX_ERRNO)\n+\n static inline void cachefiles_put_kiocb(struct cachefiles_kiocb *ki)\n {\n \tif (refcount_dec_and_test(\u0026ki-\u003eki_refcnt)) {\n@@ -193,61 +198,204 @@ static int cachefiles_read(struct netfs_cache_resources *cres,\n }\n \n /*\n- * Query the occupancy of the cache in a region, returning where the next chunk\n- * of data starts and how long it is.\n+ * Handle completion of a read from the cache issued by netfslib.\n+ */\n+static void cachefiles_issue_read_complete(struct kiocb *iocb, long ret)\n+{\n+\tstruct cachefiles_kiocb *ki = container_of(iocb, struct cachefiles_kiocb, iocb);\n+\tstruct netfs_io_subrequest *subreq = ki-\u003esubreq;\n+\tstruct inode *inode = file_inode(ki-\u003eiocb.ki_filp);\n+\n+\t_enter(\"%ld\", ret);\n+\n+\tif (ret \u003c 0) {\n+\t\tsubreq-\u003eerror = -ESTALE;\n+\t\ttrace_cachefiles_io_error(ki-\u003eobject, inode, ret,\n+\t\t\t\t\t cachefiles_trace_read_error);\n+\t}\n+\n+\tif (ret \u003e= 0) {\n+\t\tif (ki-\u003eobject-\u003ecookie-\u003einval_counter == ki-\u003einval_counter) {\n+\t\t\tsubreq-\u003eerror = 0;\n+\t\t\tif (ret \u003e 0) {\n+\t\t\t\tsubreq-\u003etransferred += ret;\n+\t\t\t\t__set_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tsubreq-\u003eerror = -ESTALE;\n+\t\t}\n+\t}\n+\n+\tnetfs_read_subreq_terminated(subreq);\n+\tcachefiles_put_kiocb(ki);\n+}\n+\n+/*\n+ * Issue a read operation to the cache.\n */\n-static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,\n-\t\t\t\t loff_t start, size_t len, size_t granularity,\n-\t\t\t\t loff_t *_data_start, size_t *_data_len)\n+static int cachefiles_issue_read(struct netfs_io_subrequest *subreq)\n {\n+\tstruct netfs_cache_resources *cres = \u0026subreq-\u003erreq-\u003ecache_resources;\n \tstruct cachefiles_object *object;\n+\tstruct cachefiles_kiocb *ki;\n+\tstruct iov_iter iter;\n \tstruct file *file;\n-\tloff_t off, off2;\n-\n-\t*_data_start = -1;\n-\t*_data_len = 0;\n+\tunsigned int old_nofs;\n+\tssize_t ret = -ENOBUFS;\n \n \tif (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))\n \t\treturn -ENOBUFS;\n \n+\tfscache_count_read();\n \tobject = cachefiles_cres_object(cres);\n \tfile = cachefiles_cres_file(cres);\n-\tgranularity = max_t(size_t, object-\u003evolume-\u003ecache-\u003ebsize, granularity);\n \n-\t_enter(\"%pD,%llu,%llx,%zx/%llx\",\n-\t file, file_inode(file)-\u003ei_ino, start, len,\n+\t_enter(\"%pD,%lli,%llx,%zx/%llx\",\n+\t file, file_inode(file)-\u003ei_ino, subreq-\u003estart, subreq-\u003elen,\n \t i_size_read(file_inode(file)));\n \n-\toff = cachefiles_inject_read_error();\n-\tif (off == 0)\n-\t\toff = vfs_llseek(file, start, SEEK_DATA);\n-\tif (off == -ENXIO)\n-\t\treturn -ENODATA; /* Beyond EOF */\n-\tif (off \u003c 0 \u0026\u0026 off \u003e= (loff_t)-MAX_ERRNO)\n-\t\treturn -ENOBUFS; /* Error. */\n-\tif (round_up(off, granularity) \u003e= start + len)\n-\t\treturn -ENODATA; /* No data in range */\n-\n-\toff2 = cachefiles_inject_read_error();\n-\tif (off2 == 0)\n-\t\toff2 = vfs_llseek(file, off, SEEK_HOLE);\n-\tif (off2 == -ENXIO)\n-\t\treturn -ENODATA; /* Beyond EOF */\n-\tif (off2 \u003c 0 \u0026\u0026 off2 \u003e= (loff_t)-MAX_ERRNO)\n-\t\treturn -ENOBUFS; /* Error. */\n-\n-\t/* Round away partial blocks */\n-\toff = round_up(off, granularity);\n-\toff2 = round_down(off2, granularity);\n-\tif (off2 \u003c= off)\n-\t\treturn -ENODATA;\n-\n-\t*_data_start = off;\n-\tif (off2 \u003e start + len)\n-\t\t*_data_len = len;\n-\telse\n-\t\t*_data_len = off2 - off;\n+\tif (subreq-\u003elen \u003e MAX_RW_COUNT)\n+\t\tsubreq-\u003elen = MAX_RW_COUNT;\n+\n+\tret = netfs_prepare_read_buffer(subreq, BIO_MAX_VECS);\n+\tif (ret \u003c 0)\n+\t\treturn ret;\n+\t/* After this point, must fail by termination. */\n+\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n+\tret = -ENOMEM;\n+\tki = kzalloc_obj(struct cachefiles_kiocb);\n+\tif (!ki)\n+\t\tgoto failed;\n+\n+\trefcount_set(\u0026ki-\u003eki_refcnt, 2);\n+\tki-\u003eiocb.ki_filp\t= file;\n+\tki-\u003eiocb.ki_pos\t\t= subreq-\u003estart;\n+\tki-\u003eiocb.ki_flags\t= IOCB_DIRECT;\n+\tki-\u003eiocb.ki_ioprio\t= get_current_ioprio();\n+\tki-\u003eiocb.ki_complete\t= cachefiles_issue_read_complete;\n+\tki-\u003eobject\t\t= object;\n+\tki-\u003einval_counter\t= cres-\u003einval_counter;\n+\tki-\u003esubreq\t\t= subreq;\n+\tki-\u003ewas_async\t\t= true;\n+\n+\tget_file(ki-\u003eiocb.ki_filp);\n+\tcachefiles_grab_object(object, cachefiles_obj_get_ioreq);\n+\n+\ttrace_cachefiles_read(object, file_inode(file), ki-\u003eiocb.ki_pos, subreq-\u003elen);\n+\told_nofs = memalloc_nofs_save();\n+\tret = cachefiles_inject_read_error();\n+\tif (ret == 0)\n+\t\tret = vfs_iocb_iter_read(file, \u0026ki-\u003eiocb, \u0026iter);\n+\tmemalloc_nofs_restore(old_nofs);\n+\n+\tswitch (ret) {\n+\tcase -EIOCBQUEUED:\n+\t\tbreak;\n+\n+\tcase -ERESTARTSYS:\n+\tcase -ERESTARTNOINTR:\n+\tcase -ERESTARTNOHAND:\n+\tcase -ERESTART_RESTARTBLOCK:\n+\t\t/* There's no easy way to restart the syscall since other AIO's\n+\t\t * may be already running. Just fail this IO with EINTR.\n+\t\t */\n+\t\tret = -EINTR;\n+\t\tfallthrough;\n+\tdefault:\n+\t\tki-\u003ewas_async = false;\n+\t\tcachefiles_issue_read_complete(\u0026ki-\u003eiocb, ret);\n+\t\tbreak;\n+\t}\n+\n+\tcachefiles_put_kiocb(ki);\n+\t_leave(\" = %zd\", ret);\n \treturn 0;\n+failed:\n+\tsubreq-\u003eerror = ret;\n+\tnetfs_read_subreq_terminated(subreq);\n+\treturn 0;\n+}\n+\n+/*\n+ * Query the occupancy of the cache in a region, returning the extent of the\n+ * next two chunks of cached data and the next hole. The occupancy map is\n+ * preloaded to show just one giant hole.\n+ */\n+static void cachefiles_query_occupancy(struct netfs_cache_resources *cres,\n+\t\t\t\t struct fscache_occupancy *occ)\n+{\n+\tstruct cachefiles_object *object;\n+\tstruct inode *inode;\n+\tstruct file *file;\n+\tunsigned long long read_limit;\n+\tloff_t ret;\n+\tint i;\n+\n+\tif (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))\n+\t\treturn;\n+\n+\tobject = cachefiles_cres_object(cres);\n+\tfile = cachefiles_cres_file(cres);\n+\tinode = file_inode(file);\n+\tocc-\u003egranularity = object-\u003evolume-\u003ecache-\u003ebsize;\n+\t/* Read read_limit before content_info. */\n+\tread_limit = atomic64_read_acquire(\u0026object-\u003eread_limit);\n+\n+\t_enter(\"%pD,%llu,%llx-%llx/%llx\",\n+\t file, inode-\u003ei_ino, occ-\u003equery_from, occ-\u003equery_to, read_limit);\n+\n+\tif (read_limit == 0)\n+\t\tgoto done;\n+\n+\tswitch (READ_ONCE(object-\u003econtent_info)) {\n+\tcase CACHEFILES_CONTENT_ALL:\n+\tcase CACHEFILES_CONTENT_SINGLE:\n+\t\tif (read_limit \u003e occ-\u003equery_from) {\n+\t\t\tocc-\u003ecached_from[0] = 0;\n+\t\t\tocc-\u003ecached_to[0] = read_limit;\n+\t\t\tocc-\u003ecached_type[0] = FSCACHE_EXTENT_DATA;\n+\t\t\tocc-\u003equery_from = ULLONG_MAX;\n+\t\t}\n+\t\tgoto done;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(occ-\u003ecached_from); i++) {\n+\t\tret = cachefiles_inject_read_error();\n+\t\tif (ret == 0)\n+\t\t\tret = vfs_llseek(file, occ-\u003equery_from, SEEK_DATA);\n+\t\tif (IS_ERR_VALUE_LL(ret)) {\n+\t\t\tif (ret != -ENXIO)\n+\t\t\t\tgoto done;\n+\t\t\tocc-\u003equery_from = ULLONG_MAX;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tocc-\u003ecached_type[i] = FSCACHE_EXTENT_DATA;\n+\t\tocc-\u003ecached_from[i] = ret;\n+\t\tocc-\u003equery_from = ret;\n+\n+\t\tret = cachefiles_inject_read_error();\n+\t\tif (ret == 0)\n+\t\t\tret = vfs_llseek(file, occ-\u003equery_from, SEEK_HOLE);\n+\t\tif (IS_ERR_VALUE_LL(ret)) {\n+\t\t\tif (ret != -ENXIO)\n+\t\t\t\tgoto done;\n+\t\t\tocc-\u003equery_from = ULLONG_MAX;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tocc-\u003ecached_to[i] = ret;\n+\t\tocc-\u003equery_from = ret;\n+\t\tif (occ-\u003equery_from \u003e= occ-\u003equery_to)\n+\t\t\tbreak;\n+\t}\n+\n+done:\n+\t_debug(\"query[0] %llx-%llx\", occ-\u003ecached_from[0], occ-\u003ecached_to[0]);\n+\t_debug(\"query[1] %llx-%llx\", occ-\u003ecached_from[1], occ-\u003ecached_to[1]);\n }\n \n /*\n@@ -390,7 +538,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,\n \tsize_t len = *_len;\n \tloff_t off, to;\n \tino_t ino = file ? file_inode(file)-\u003ei_ino : 0;\n-\tint rc;\n \n \t_enter(\"%zx @%llx/%llx\", len, start, i_size);\n \n@@ -403,8 +550,7 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,\n \tif (test_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, \u0026cookie-\u003eflags)) {\n \t\t__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);\n \t\twhy = cachefiles_trace_read_no_data;\n-\t\tif (!test_bit(NETFS_SREQ_ONDEMAND, _flags))\n-\t\t\tgoto out_no_object;\n+\t\tgoto out_no_object;\n \t}\n \n \t/* The object and the file may be being created in the background. */\n@@ -421,7 +567,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,\n \tobject = cachefiles_cres_object(cres);\n \tcache = object-\u003evolume-\u003ecache;\n \tcachefiles_begin_secure(cache, \u0026saved_cred);\n-retry:\n \toff = cachefiles_inject_read_error();\n \tif (off == 0)\n \t\toff = vfs_llseek(file, start, SEEK_DATA);\n@@ -474,14 +619,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,\n \n download_and_store:\n \t__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);\n-\tif (test_bit(NETFS_SREQ_ONDEMAND, _flags)) {\n-\t\trc = cachefiles_ondemand_read(object, start, len);\n-\t\tif (!rc) {\n-\t\t\t__clear_bit(NETFS_SREQ_ONDEMAND, _flags);\n-\t\t\tgoto retry;\n-\t\t}\n-\t\tret = NETFS_INVALID_READ;\n-\t}\n out:\n \tcachefiles_end_secure(cache, saved_cred);\n out_no_object:\n@@ -489,30 +626,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,\n \treturn ret;\n }\n \n-/*\n- * Prepare a read operation, shortening it to a cached/uncached\n- * boundary as appropriate.\n- */\n-static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *subreq,\n-\t\t\t\t\t\t unsigned long long i_size)\n-{\n-\treturn cachefiles_do_prepare_read(\u0026subreq-\u003erreq-\u003ecache_resources,\n-\t\t\t\t\t subreq-\u003estart, \u0026subreq-\u003elen, i_size,\n-\t\t\t\t\t \u0026subreq-\u003eflags, subreq-\u003erreq-\u003einode-\u003ei_ino);\n-}\n-\n-/*\n- * Prepare an on-demand read operation, shortening it to a cached/uncached\n- * boundary as appropriate.\n- */\n-static enum netfs_io_source\n-cachefiles_prepare_ondemand_read(struct netfs_cache_resources *cres,\n-\t\t\t\t loff_t start, size_t *_len, loff_t i_size,\n-\t\t\t\t unsigned long *_flags, ino_t ino)\n-{\n-\treturn cachefiles_do_prepare_read(cres, start, _len, i_size, _flags, ino);\n-}\n-\n /*\n * Prepare for a write to occur.\n */\n@@ -527,7 +640,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,\n \tint ret;\n \n \t/* Round to DIO size */\n-\tstart = round_down(*_start, PAGE_SIZE);\n+\tstart = round_down(*_start, cache-\u003ebsize);\n \tif (start != *_start || *_len \u003e upper_len) {\n \t\t/* Probably asked to cache a streaming write written into the\n \t\t * pagecache when the cookie was temporarily out of service to\n@@ -537,7 +650,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,\n \t\treturn -ENOBUFS;\n \t}\n \n-\t*_len = round_up(len, PAGE_SIZE);\n+\t*_len = round_up(len, cache-\u003ebsize);\n \n \t/* We need to work out whether there's sufficient disk space to perform\n \t * the write - but we can skip that check if we have space already\n@@ -563,7 +676,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,\n \t * space, we need to see if it's fully allocated. If it's not, we may\n \t * want to cull it.\n \t */\n-\tif (cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,\n+\tif (cachefiles_has_space(cache, 0, *_len / cache-\u003ebsize,\n \t\t\t\t cachefiles_has_space_check) == 0)\n \t\treturn 0; /* Enough space to simply overwrite the whole block */\n \n@@ -595,13 +708,13 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,\n \treturn ret;\n \n check_space:\n-\treturn cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,\n+\treturn cachefiles_has_space(cache, 0, *_len / cache-\u003ebsize,\n \t\t\t\t cachefiles_has_space_for_write);\n }\n \n-static int cachefiles_prepare_write(struct netfs_cache_resources *cres,\n-\t\t\t\t loff_t *_start, size_t *_len, size_t upper_len,\n-\t\t\t\t loff_t i_size, bool no_space_allocated_yet)\n+static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres,\n+\t\t\t\t\tloff_t *_start, size_t *_len, size_t upper_len,\n+\t\t\t\t\tloff_t i_size, bool no_space_allocated_yet)\n {\n \tstruct cachefiles_object *object = cachefiles_cres_object(cres);\n \tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n@@ -623,92 +736,238 @@ static int cachefiles_prepare_write(struct netfs_cache_resources *cres,\n \treturn ret;\n }\n \n-static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq)\n+static int cachefiles_estimate_write(struct netfs_io_request *wreq,\n+\t\t\t\t struct netfs_io_stream *stream,\n+\t\t\t\t struct netfs_write_estimate *estimate)\n {\n-\tstruct netfs_io_request *wreq = subreq-\u003erreq;\n-\tstruct netfs_cache_resources *cres = \u0026wreq-\u003ecache_resources;\n-\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[subreq-\u003estream_nr];\n-\n-\t_enter(\"W=%x[%x] %llx\", wreq-\u003edebug_id, subreq-\u003edebug_index, subreq-\u003estart);\n-\n-\tstream-\u003esreq_max_len = MAX_RW_COUNT;\n-\tstream-\u003esreq_max_segs = BIO_MAX_VECS;\n-\n-\tif (!cachefiles_cres_file(cres)) {\n-\t\tif (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))\n-\t\t\treturn netfs_prepare_write_failed(subreq);\n-\t\tif (!cachefiles_cres_file(cres))\n-\t\t\treturn netfs_prepare_write_failed(subreq);\n-\t}\n+\testimate-\u003eissue_at = stream-\u003eissue_from + MAX_RW_COUNT;\n+\testimate-\u003emax_segs = BIO_MAX_VECS;\n+\treturn 0;\n }\n \n-static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)\n+static int cachefiles_issue_write(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *wreq = subreq-\u003erreq;\n \tstruct netfs_cache_resources *cres = \u0026wreq-\u003ecache_resources;\n \tstruct cachefiles_object *object = cachefiles_cres_object(cres);\n \tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n-\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tstruct iov_iter iter;\n \tconst struct cred *saved_cred;\n-\tsize_t off, pre, post, len = subreq-\u003elen;\n \tloff_t start = subreq-\u003estart;\n-\tint ret;\n+\tsize_t len = subreq-\u003elen;\n+\tint ret = -EINVAL;\n \n \t_enter(\"W=%x[%x] %llx-%llx\",\n \t wreq-\u003edebug_id, subreq-\u003edebug_index, start, start + len - 1);\n \n-\t/* We need to start on the cache granularity boundary */\n-\toff = start \u0026 (CACHEFILES_DIO_BLOCK_SIZE - 1);\n-\tif (off) {\n-\t\tpre = CACHEFILES_DIO_BLOCK_SIZE - off;\n-\t\tif (pre \u003e= len) {\n-\t\t\tfscache_count_dio_misfit();\n-\t\t\tnetfs_write_subrequest_terminated(subreq, len);\n-\t\t\treturn;\n-\t\t}\n-\t\tsubreq-\u003etransferred += pre;\n-\t\tstart += pre;\n-\t\tlen -= pre;\n-\t\tiov_iter_advance(\u0026subreq-\u003eio_iter, pre);\n+\tif (!cachefiles_cres_file(cres)) {\n+\t\tif (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))\n+\t\t\treturn -EINVAL;\n+\t\tif (!cachefiles_cres_file(cres))\n+\t\t\treturn -EINVAL;\n \t}\n \n-\t/* We also need to end on the cache granularity boundary */\n-\tif (start + len == wreq-\u003ei_size) {\n-\t\tsize_t part = len % CACHEFILES_DIO_BLOCK_SIZE;\n-\t\tsize_t need = CACHEFILES_DIO_BLOCK_SIZE - part;\n+\tret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS);\n+\tif (ret \u003c 0)\n+\t\treturn ret;\n+\t/* After this point, must fail by termination. */\n \n-\t\tif (part \u0026\u0026 stream-\u003esubmit_extendable_to \u003e= need) {\n-\t\t\tlen += need;\n-\t\t\tsubreq-\u003elen += need;\n-\t\t\tsubreq-\u003eio_iter.count += need;\n-\t\t}\n-\t}\n+\t/* The buffer extraction func may round out start and end. */\n+\tstart = subreq-\u003estart;\n+\tlen = subreq-\u003elen;\n \n-\tpost = len \u0026 (CACHEFILES_DIO_BLOCK_SIZE - 1);\n-\tif (post) {\n-\t\tlen -= post;\n-\t\tif (len == 0) {\n-\t\t\tfscache_count_dio_misfit();\n-\t\t\tnetfs_write_subrequest_terminated(subreq, post);\n-\t\t\treturn;\n-\t\t}\n-\t\tiov_iter_truncate(\u0026subreq-\u003eio_iter, len);\n+\t/* We need to start and end on cache granularity boundaries. */\n+\tif (WARN_ON_ONCE(start \u0026 (cache-\u003ebsize - 1)) ||\n+\t WARN_ON_ONCE(len \u0026 (cache-\u003ebsize - 1))) {\n+\t\tfscache_count_dio_misfit();\n+\t\tret = -EIO;\n+\t\tgoto failed;\n \t}\n \n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, len);\n+\n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_cache_prepare);\n \tcachefiles_begin_secure(cache, \u0026saved_cred);\n \tret = __cachefiles_prepare_write(object, cachefiles_cres_file(cres),\n \t\t\t\t\t \u0026start, \u0026len, len, true);\n \tcachefiles_end_secure(cache, saved_cred);\n-\tif (ret \u003c 0) {\n-\t\tnetfs_write_subrequest_terminated(subreq, ret);\n-\t\treturn;\n-\t}\n+\tif (ret \u003c 0)\n+\t\tgoto failed;\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_cache_write);\n-\tcachefiles_write(\u0026subreq-\u003erreq-\u003ecache_resources,\n-\t\t\t subreq-\u003estart, \u0026subreq-\u003eio_iter,\n+\tcachefiles_write(\u0026subreq-\u003erreq-\u003ecache_resources, subreq-\u003estart, \u0026iter,\n \t\t\t netfs_write_subrequest_terminated, subreq);\n+\treturn 0;\n+failed:\n+\tnetfs_write_subrequest_terminated(subreq, ret);\n+\treturn 0;\n+}\n+\n+/*\n+ * Collect the result of buffered writeback to the cache. This includes\n+ * copying a read to the cache. Netfslib collates the results, which might\n+ * occur out of order, and delivers them to the cache so that it can update its\n+ * content record.\n+ *\n+ * block_type is one of:\n+ * - NETFS_CACHE_COLLECT_WRITE_DATA for a contiguous block of data\n+ * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped\n+ * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a hole due to a failed/cancelled write\n+ *\n+ * The writes we made are all rounded out at both sides to the nearest DIO\n+ * block boundary, so if the final block contains the EOF in the middle of it\n+ * (rather than at the end), padding will have been written to the file. The\n+ * backing file's filesize will have been updated if the write extended the\n+ * file; the filesize may still change due to outstanding subreqs.\n+ *\n+ * The metadata in the cache file xattr records the size of the object we have\n+ * stored, but the cache file EOF only goes up to where we've cached data to\n+ * and, furthermore, is rounded up to the nearest DIO block boundary.\n+ *\n+ * Concurrent updates should be protected against by the caller. Netfslib\n+ * holds NETFS_ICTX_WB_LOCK as a lock on writeback requests. DIO writes\n+ * invalidate the cookie and caching is kept disabled until all users have\n+ * unused the cookie.\n+ */\n+static void cachefiles_collect_write(struct netfs_io_request *wreq,\n+\t\t\t\t unsigned long long start, size_t len,\n+\t\t\t\t enum netfs_cache_collect block_type)\n+{\n+\tstruct netfs_cache_resources *cres = \u0026wreq-\u003ecache_resources;\n+\tstruct cachefiles_object *object = cachefiles_cres_object(cres);\n+\tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n+\tstruct file *file = cachefiles_cres_file(cres);\n+\tstruct inode *inode = file_inode(file);\n+\tunsigned long long read_limit;\n+\tunsigned long long old_size = cres-\u003ecache_i_size;\n+\tunsigned long long new_size = i_size_read(inode);\n+\tunsigned long long data_to = object-\u003eobject_size;\n+\tunsigned long long end = start + len;\n+\tint ret;\n+\n+\t_enter(\"%llx,%zx,%x\", start, len, cache-\u003ebsize);\n+\n+\tif (WARN_ON(old_size\t\u0026 (cache-\u003ebsize - 1)) ||\n+\t WARN_ON(new_size\t\u0026 (cache-\u003ebsize - 1)) ||\n+\t WARN_ON(start\t\u0026 (cache-\u003ebsize - 1)) ||\n+\t WARN_ON(len\t\t\u0026 (cache-\u003ebsize - 1))) {\n+\t\ttrace_cachefiles_io_error(object, inode, -EIO,\n+\t\t\t\t\t cachefiles_trace_alignment_error);\n+\t\tcachefiles_remove_object_xattr(cache, object, file-\u003ef_path.dentry);\n+\t\treturn;\n+\t}\n+\n+\t/* If this is recording a gap, due to discontiguous writes or lack of\n+\t * cache space, then a hole may have been introduced into the backing\n+\t * file. Treat it as a zero-length data block.\n+\t */\n+\tif (block_type == NETFS_CACHE_COLLECT_WRITE_GAP ||\n+\t block_type == NETFS_CACHE_COLLECT_WRITE_CANCEL) {\n+\t\tstart = end;\n+\t\tlen = 0;\n+\t}\n+\n+\t/* Zeroth case: Single monolithic files are handled specially.\n+\t */\n+\tif (wreq-\u003eorigin == NETFS_WRITEBACK_SINGLE) {\n+\t\tobject-\u003econtent_info = CACHEFILES_CONTENT_SINGLE;\n+\t\tgoto update_sizes;\n+\t}\n+\n+\t/* First case: The backing file was empty. */\n+\tif (old_size == 0) {\n+\t\tif (start == 0)\n+\t\t\tobject-\u003econtent_info = CACHEFILES_CONTENT_ALL;\n+\t\telse\n+\t\t\tobject-\u003econtent_info = CACHEFILES_CONTENT_BACKFS_MAP;\n+\t\tgoto update_sizes;\n+\t}\n+\n+\t/* Second case: The backing file is entirely within the old object size\n+\t * and thus there can be no partial tail block to deal with in the\n+\t * cache file.\n+\t */\n+\tif (old_size \u003c= data_to) {\n+\t\tif (start \u003e old_size)\n+\t\t\tgoto discontiguous;\n+\t\tgoto update_sizes;\n+\t}\n+\n+\t/* Third case: The write happened entirely within the bounds of the\n+\t * current cache file's size.\n+\t */\n+\tif (end \u003c= old_size)\n+\t\tgoto update_sizes;\n+\n+\t/* Fourth case: The write overwrote the partial tail block and extended\n+\t * the file. We only need to update the object size because netfslib\n+\t * rounds out/pads cache writes to whole disk blocks.\n+\t */\n+\tif (start \u003c old_size)\n+\t\tgoto update_sizes;\n+\n+\t/* Fifth case: The write started from the end of the whole tail block\n+\t * and extended the file. Just extend our notion of the filesize.\n+\t */\n+\tif (start == old_size \u0026\u0026 old_size == data_to)\n+\t\tgoto update_sizes;\n+\n+\t/* Sixth case: The write continued on from the partial tail block and\n+\t * extended the file. Need to clear the gap.\n+\t */\n+\tif (start == old_size \u0026\u0026 old_size \u003e data_to)\n+\t\tgoto clear_gap;\n+\n+discontiguous:\n+\t/* Seventh case: The write was beyond the EOF on the cache file, so now\n+\t * there's a hole in the file and we can no longer say in the metadata\n+\t * that we can assume we have it all. We may also need to clear the\n+\t * end of the partial tail block.\n+\t */\n+\t/* TODO: For the moment, we will have to use SEEK_HOLE/SEEK_DATA. */\n+\tif (object-\u003econtent_info != CACHEFILES_CONTENT_BACKFS_MAP) {\n+\t\tobject-\u003econtent_info = CACHEFILES_CONTENT_BACKFS_MAP;\n+\t\ttrace_cachefiles_coherency(object, inode-\u003ei_ino, data_to,\n+\t\t\t\t\t be64_to_cpup((__be64 *)object-\u003ecookie-\u003einline_aux),\n+\t\t\t\t\t CACHEFILES_CONTENT_BACKFS_MAP,\n+\t\t\t\t\t cachefiles_coherency_discontiguous);\n+\t}\n+\n+clear_gap:\n+\t/* We need to clear any partial padding that got jumped over. It\n+\t * *should* be all zeros, but shared-writable mmap exists...\n+\t */\n+\tif (old_size \u003e data_to) {\n+\t\ttrace_cachefiles_trunc(object, inode, data_to, old_size,\n+\t\t\t\t cachefiles_trunc_clear_padding);\n+\t\tret = cachefiles_inject_write_error();\n+\t\tif (ret == 0)\n+\t\t\tret = vfs_fallocate(file, FALLOC_FL_ZERO_RANGE,\n+\t\t\t\t\t data_to, old_size - data_to);\n+\t\tif (ret \u003c 0) {\n+\t\t\ttrace_cachefiles_io_error(object, inode, ret,\n+\t\t\t\t\t\t cachefiles_trace_fallocate_error);\n+\t\t\tcachefiles_io_error_obj(object, \"fallocate zero pad failed %d\", ret);\n+\t\t\tcachefiles_remove_object_xattr(cache, object, file-\u003ef_path.dentry);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+update_sizes:\n+\tread_limit = umax(old_size, end);\n+\tcres-\u003ecache_i_size = read_limit;\n+\n+\t/* We need to be careful setting the object_size: we may have written\n+\t * more to the cache than to the server (due to cache DIO rounding) and\n+\t * the i_size set on the netfs inode may include unwritten data that\n+\t * the server doesn't know about yet.\n+\t */\n+\tobject-\u003eobject_size = umin(read_limit, wreq-\u003ei_size);\n+\n+\t/* Raise the limit at which reads can access the file. */\n+\t/* Update read_limit after content_info */\n+\tatomic64_set_release(\u0026object-\u003eread_limit, read_limit);\n }\n \n /*\n@@ -727,12 +986,12 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {\n \t.end_operation\t\t= cachefiles_end_operation,\n \t.read\t\t\t= cachefiles_read,\n \t.write\t\t\t= cachefiles_write,\n+\t.issue_read\t\t= cachefiles_issue_read,\n \t.issue_write\t\t= cachefiles_issue_write,\n-\t.prepare_read\t\t= cachefiles_prepare_read,\n-\t.prepare_write\t\t= cachefiles_prepare_write,\n-\t.prepare_write_subreq\t= cachefiles_prepare_write_subreq,\n-\t.prepare_ondemand_read\t= cachefiles_prepare_ondemand_read,\n+\t.estimate_write\t\t= cachefiles_estimate_write,\n+\t.prepare_write_old\t= cachefiles_prepare_write_old,\n \t.query_occupancy\t= cachefiles_query_occupancy,\n+\t.collect_write\t\t= cachefiles_collect_write,\n };\n \n /*\n@@ -742,13 +1001,19 @@ bool cachefiles_begin_operation(struct netfs_cache_resources *cres,\n \t\t\t\tenum fscache_want_state want_state)\n {\n \tstruct cachefiles_object *object = cachefiles_cres_object(cres);\n+\tstruct file *file;\n \n \tif (!cachefiles_cres_file(cres)) {\n \t\tcres-\u003eops = \u0026cachefiles_netfs_cache_ops;\n+\t\tcres-\u003eobject_id = object-\u003edebug_id;\n \t\tif (object-\u003efile) {\n \t\t\tspin_lock(\u0026object-\u003elock);\n-\t\t\tif (!cres-\u003ecache_priv2 \u0026\u0026 object-\u003efile)\n-\t\t\t\tcres-\u003ecache_priv2 = get_file(object-\u003efile);\n+\t\t\tfile = object-\u003efile;\n+\t\t\tif (!cres-\u003ecache_priv2 \u0026\u0026 file) {\n+\t\t\t\tcres-\u003ecache_priv2 = get_file(file);\n+\t\t\t\tcres-\u003ecache_i_size = i_size_read(file_inode(file));\n+\t\t\t\tcres-\u003edio_size = object-\u003evolume-\u003ecache-\u003ebsize;\n+\t\t\t}\n \t\t\tspin_unlock(\u0026object-\u003elock);\n \t\t}\n \t}\ndiff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c\nindex 8a9f6be15828ba..a30df0f91e5e5d 100644\n--- a/fs/cachefiles/namei.c\n+++ b/fs/cachefiles/namei.c\n@@ -414,7 +414,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)\n \tstruct dentry *fan = volume-\u003efanout[(u8)object-\u003ecookie-\u003ekey_hash];\n \tstruct file *file;\n \tconst struct path parentpath = { .mnt = cache-\u003emnt, .dentry = fan };\n-\tuint64_t ni_size;\n \tlong ret;\n \n \n@@ -442,27 +441,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)\n \tif (!cachefiles_mark_inode_in_use(object, file_inode(file)))\n \t\tWARN_ON(1);\n \n-\tret = cachefiles_ondemand_init_object(object);\n-\tif (ret \u003c 0)\n-\t\tgoto err_unuse;\n-\n-\tni_size = object-\u003ecookie-\u003eobject_size;\n-\tni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);\n-\n-\tif (ni_size \u003e 0) {\n-\t\ttrace_cachefiles_trunc(object, file_inode(file), 0, ni_size,\n-\t\t\t\t cachefiles_trunc_expand_tmpfile);\n-\t\tret = cachefiles_inject_write_error();\n-\t\tif (ret == 0)\n-\t\t\tret = vfs_truncate(\u0026file-\u003ef_path, ni_size);\n-\t\tif (ret \u003c 0) {\n-\t\t\ttrace_cachefiles_vfs_error(\n-\t\t\t\tobject, file_inode(file), ret,\n-\t\t\t\tcachefiles_trace_trunc_error);\n-\t\t\tgoto err_unuse;\n-\t\t}\n-\t}\n-\n \tret = -EINVAL;\n \tif (unlikely(!file-\u003ef_op-\u003eread_iter) ||\n \t unlikely(!file-\u003ef_op-\u003ewrite_iter)) {\n@@ -471,6 +449,7 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)\n \t}\n out:\n \tcachefiles_end_secure(cache, saved_cred);\n+\tobject-\u003econtent_info = CACHEFILES_CONTENT_ALL;\n \treturn file;\n \n err_unuse:\n@@ -546,10 +525,6 @@ static bool cachefiles_open_file(struct cachefiles_object *object,\n \t}\n \t_debug(\"file -\u003e %pd positive\", dentry);\n \n-\tret = cachefiles_ondemand_init_object(object);\n-\tif (ret \u003c 0)\n-\t\tgoto error_fput;\n-\n \tret = cachefiles_check_auxdata(object, file);\n \tif (ret \u003c 0)\n \t\tgoto check_failed;\ndiff --git a/fs/cachefiles/ondemand.c b/fs/cachefiles/ondemand.c\ndeleted file mode 100644\nindex 0849eaf583cd27..00000000000000\n--- a/fs/cachefiles/ondemand.c\n+++ /dev/null\n@@ -1,761 +0,0 @@\n-// SPDX-License-Identifier: GPL-2.0-or-later\n-#include \u003clinux/anon_inodes.h\u003e\n-#include \u003clinux/uio.h\u003e\n-#include \"internal.h\"\n-\n-struct ondemand_anon_file {\n-\tstruct file *file;\n-\tint fd;\n-};\n-\n-static inline void cachefiles_req_put(struct cachefiles_req *req)\n-{\n-\tif (refcount_dec_and_test(\u0026req-\u003eref))\n-\t\tkfree(req);\n-}\n-\n-static int cachefiles_ondemand_fd_release(struct inode *inode,\n-\t\t\t\t\t struct file *file)\n-{\n-\tstruct cachefiles_object *object = file-\u003eprivate_data;\n-\tstruct cachefiles_cache *cache;\n-\tstruct cachefiles_ondemand_info *info;\n-\tint object_id;\n-\tstruct cachefiles_req *req;\n-\tXA_STATE(xas, NULL, 0);\n-\n-\tif (!object)\n-\t\treturn 0;\n-\n-\tinfo = object-\u003eondemand;\n-\tcache = object-\u003evolume-\u003ecache;\n-\txas.xa = \u0026cache-\u003ereqs;\n-\n-\txa_lock(\u0026cache-\u003ereqs);\n-\tspin_lock(\u0026info-\u003elock);\n-\tobject_id = info-\u003eondemand_id;\n-\tinfo-\u003eondemand_id = CACHEFILES_ONDEMAND_ID_CLOSED;\n-\tcachefiles_ondemand_set_object_close(object);\n-\tspin_unlock(\u0026info-\u003elock);\n-\n-\t/* Only flush CACHEFILES_REQ_NEW marked req to avoid race with daemon_read */\n-\txas_for_each_marked(\u0026xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {\n-\t\tif (req-\u003emsg.object_id == object_id \u0026\u0026\n-\t\t req-\u003emsg.opcode == CACHEFILES_OP_CLOSE) {\n-\t\t\tcomplete(\u0026req-\u003edone);\n-\t\t\txas_store(\u0026xas, NULL);\n-\t\t}\n-\t}\n-\txa_unlock(\u0026cache-\u003ereqs);\n-\n-\txa_erase(\u0026cache-\u003eondemand_ids, object_id);\n-\ttrace_cachefiles_ondemand_fd_release(object, object_id);\n-\tcachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);\n-\tcachefiles_put_unbind_pincount(cache);\n-\treturn 0;\n-}\n-\n-static ssize_t cachefiles_ondemand_fd_write_iter(struct kiocb *kiocb,\n-\t\t\t\t\t\t struct iov_iter *iter)\n-{\n-\tstruct cachefiles_object *object = kiocb-\u003eki_filp-\u003eprivate_data;\n-\tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n-\tstruct file *file;\n-\tsize_t len = iter-\u003ecount, aligned_len = len;\n-\tloff_t pos = kiocb-\u003eki_pos;\n-\tconst struct cred *saved_cred;\n-\tint ret;\n-\n-\tspin_lock(\u0026object-\u003elock);\n-\tfile = object-\u003efile;\n-\tif (!file) {\n-\t\tspin_unlock(\u0026object-\u003elock);\n-\t\treturn -ENOBUFS;\n-\t}\n-\tget_file(file);\n-\tspin_unlock(\u0026object-\u003elock);\n-\n-\tcachefiles_begin_secure(cache, \u0026saved_cred);\n-\tret = __cachefiles_prepare_write(object, file, \u0026pos, \u0026aligned_len, len, true);\n-\tcachefiles_end_secure(cache, saved_cred);\n-\tif (ret \u003c 0)\n-\t\tgoto out;\n-\n-\ttrace_cachefiles_ondemand_fd_write(object, file_inode(file), pos, len);\n-\tret = __cachefiles_write(object, file, pos, iter, NULL, NULL);\n-\tif (ret \u003e 0)\n-\t\tkiocb-\u003eki_pos += ret;\n-\n-out:\n-\tfput(file);\n-\treturn ret;\n-}\n-\n-static loff_t cachefiles_ondemand_fd_llseek(struct file *filp, loff_t pos,\n-\t\t\t\t\t int whence)\n-{\n-\tstruct cachefiles_object *object = filp-\u003eprivate_data;\n-\tstruct file *file;\n-\tloff_t ret;\n-\n-\tspin_lock(\u0026object-\u003elock);\n-\tfile = object-\u003efile;\n-\tif (!file) {\n-\t\tspin_unlock(\u0026object-\u003elock);\n-\t\treturn -ENOBUFS;\n-\t}\n-\tget_file(file);\n-\tspin_unlock(\u0026object-\u003elock);\n-\n-\tret = vfs_llseek(file, pos, whence);\n-\tfput(file);\n-\n-\treturn ret;\n-}\n-\n-static long cachefiles_ondemand_fd_ioctl(struct file *filp, unsigned int ioctl,\n-\t\t\t\t\t unsigned long id)\n-{\n-\tstruct cachefiles_object *object = filp-\u003eprivate_data;\n-\tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n-\tstruct cachefiles_req *req;\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, id);\n-\n-\tif (ioctl != CACHEFILES_IOC_READ_COMPLETE)\n-\t\treturn -EINVAL;\n-\n-\tif (!test_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags))\n-\t\treturn -EOPNOTSUPP;\n-\n-\txa_lock(\u0026cache-\u003ereqs);\n-\treq = xas_load(\u0026xas);\n-\tif (!req || req-\u003emsg.opcode != CACHEFILES_OP_READ ||\n-\t req-\u003eobject != object) {\n-\t\txa_unlock(\u0026cache-\u003ereqs);\n-\t\treturn -EINVAL;\n-\t}\n-\txas_store(\u0026xas, NULL);\n-\txa_unlock(\u0026cache-\u003ereqs);\n-\n-\ttrace_cachefiles_ondemand_cread(object, id);\n-\tcomplete(\u0026req-\u003edone);\n-\treturn 0;\n-}\n-\n-static const struct file_operations cachefiles_ondemand_fd_fops = {\n-\t.owner\t\t= THIS_MODULE,\n-\t.release\t= cachefiles_ondemand_fd_release,\n-\t.write_iter\t= cachefiles_ondemand_fd_write_iter,\n-\t.llseek\t\t= cachefiles_ondemand_fd_llseek,\n-\t.unlocked_ioctl\t= cachefiles_ondemand_fd_ioctl,\n-};\n-\n-/*\n- * OPEN request Completion (copen)\n- * - command: \"copen \u003cid\u003e,\u003ccache_size\u003e\"\n- * \u003ccache_size\u003e indicates the object size if \u003e=0, error code if negative\n- */\n-int cachefiles_ondemand_copen(struct cachefiles_cache *cache, char *args)\n-{\n-\tstruct cachefiles_req *req;\n-\tstruct fscache_cookie *cookie;\n-\tstruct cachefiles_ondemand_info *info;\n-\tchar *pid, *psize;\n-\tunsigned long id;\n-\tlong size;\n-\tint ret;\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, 0);\n-\n-\tif (!test_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags))\n-\t\treturn -EOPNOTSUPP;\n-\n-\tif (!*args) {\n-\t\tpr_err(\"Empty id specified\\n\");\n-\t\treturn -EINVAL;\n-\t}\n-\n-\tpid = args;\n-\tpsize = strchr(args, ',');\n-\tif (!psize) {\n-\t\tpr_err(\"Cache size is not specified\\n\");\n-\t\treturn -EINVAL;\n-\t}\n-\n-\t*psize = 0;\n-\tpsize++;\n-\n-\tret = kstrtoul(pid, 0, \u0026id);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\txa_lock(\u0026cache-\u003ereqs);\n-\txas.xa_index = id;\n-\treq = xas_load(\u0026xas);\n-\tif (!req || req-\u003emsg.opcode != CACHEFILES_OP_OPEN ||\n-\t !req-\u003eobject-\u003eondemand-\u003eondemand_id) {\n-\t\txa_unlock(\u0026cache-\u003ereqs);\n-\t\treturn -EINVAL;\n-\t}\n-\txas_store(\u0026xas, NULL);\n-\txa_unlock(\u0026cache-\u003ereqs);\n-\n-\tinfo = req-\u003eobject-\u003eondemand;\n-\t/* fail OPEN request if copen format is invalid */\n-\tret = kstrtol(psize, 0, \u0026size);\n-\tif (ret) {\n-\t\treq-\u003eerror = ret;\n-\t\tgoto out;\n-\t}\n-\n-\t/* fail OPEN request if daemon reports an error */\n-\tif (size \u003c 0) {\n-\t\tif (!IS_ERR_VALUE(size)) {\n-\t\t\treq-\u003eerror = -EINVAL;\n-\t\t\tret = -EINVAL;\n-\t\t} else {\n-\t\t\treq-\u003eerror = size;\n-\t\t\tret = 0;\n-\t\t}\n-\t\tgoto out;\n-\t}\n-\n-\tspin_lock(\u0026info-\u003elock);\n-\t/*\n-\t * The anonymous fd was closed before copen ? Fail the request.\n-\t *\n-\t * t1 | t2\n-\t * ---------------------------------------------------------\n-\t * cachefiles_ondemand_copen\n-\t * req = xa_erase(\u0026cache-\u003ereqs, id)\n-\t * // Anon fd is maliciously closed.\n-\t * cachefiles_ondemand_fd_release\n-\t * xa_lock(\u0026cache-\u003ereqs)\n-\t * cachefiles_ondemand_set_object_close(object)\n-\t * xa_unlock(\u0026cache-\u003ereqs)\n-\t * cachefiles_ondemand_set_object_open\n-\t * // No one will ever close it again.\n-\t * cachefiles_ondemand_daemon_read\n-\t * cachefiles_ondemand_select_req\n-\t *\n-\t * Get a read req but its fd is already closed. The daemon can't\n-\t * issue a cread ioctl with an closed fd, then hung.\n-\t */\n-\tif (info-\u003eondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED) {\n-\t\tspin_unlock(\u0026info-\u003elock);\n-\t\treq-\u003eerror = -EBADFD;\n-\t\tgoto out;\n-\t}\n-\tcookie = req-\u003eobject-\u003ecookie;\n-\tcookie-\u003eobject_size = size;\n-\tif (size)\n-\t\tclear_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, \u0026cookie-\u003eflags);\n-\telse\n-\t\tset_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, \u0026cookie-\u003eflags);\n-\ttrace_cachefiles_ondemand_copen(req-\u003eobject, id, size);\n-\n-\tcachefiles_ondemand_set_object_open(req-\u003eobject);\n-\tspin_unlock(\u0026info-\u003elock);\n-\twake_up_all(\u0026cache-\u003edaemon_pollwq);\n-\n-out:\n-\tspin_lock(\u0026info-\u003elock);\n-\t/* Need to set object close to avoid reopen status continuing */\n-\tif (info-\u003eondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED)\n-\t\tcachefiles_ondemand_set_object_close(req-\u003eobject);\n-\tspin_unlock(\u0026info-\u003elock);\n-\tcomplete(\u0026req-\u003edone);\n-\treturn ret;\n-}\n-\n-int cachefiles_ondemand_restore(struct cachefiles_cache *cache, char *args)\n-{\n-\tstruct cachefiles_req *req;\n-\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, 0);\n-\n-\tif (!test_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags))\n-\t\treturn -EOPNOTSUPP;\n-\n-\t/*\n-\t * Reset the requests to CACHEFILES_REQ_NEW state, so that the\n-\t * requests have been processed halfway before the crash of the\n-\t * user daemon could be reprocessed after the recovery.\n-\t */\n-\txas_lock(\u0026xas);\n-\txas_for_each(\u0026xas, req, ULONG_MAX)\n-\t\txas_set_mark(\u0026xas, CACHEFILES_REQ_NEW);\n-\txas_unlock(\u0026xas);\n-\n-\twake_up_all(\u0026cache-\u003edaemon_pollwq);\n-\treturn 0;\n-}\n-\n-static int cachefiles_ondemand_get_fd(struct cachefiles_req *req,\n-\t\t\t\t struct ondemand_anon_file *anon_file)\n-{\n-\tstruct cachefiles_object *object;\n-\tstruct cachefiles_cache *cache;\n-\tstruct cachefiles_open *load;\n-\tu32 object_id;\n-\tint ret;\n-\n-\tobject = cachefiles_grab_object(req-\u003eobject,\n-\t\t\tcachefiles_obj_get_ondemand_fd);\n-\tcache = object-\u003evolume-\u003ecache;\n-\n-\tret = xa_alloc_cyclic(\u0026cache-\u003eondemand_ids, \u0026object_id, NULL,\n-\t\t\t XA_LIMIT(1, INT_MAX),\n-\t\t\t \u0026cache-\u003eondemand_id_next, GFP_KERNEL);\n-\tif (ret \u003c 0)\n-\t\tgoto err;\n-\n-\tanon_file-\u003efd = get_unused_fd_flags(O_WRONLY);\n-\tif (anon_file-\u003efd \u003c 0) {\n-\t\tret = anon_file-\u003efd;\n-\t\tgoto err_free_id;\n-\t}\n-\n-\tanon_file-\u003efile = anon_inode_getfile_fmode(\"[cachefiles]\",\n-\t\t\t\t\u0026cachefiles_ondemand_fd_fops, object,\n-\t\t\t\tO_WRONLY, FMODE_PWRITE | FMODE_LSEEK);\n-\tif (IS_ERR(anon_file-\u003efile)) {\n-\t\tret = PTR_ERR(anon_file-\u003efile);\n-\t\tgoto err_put_fd;\n-\t}\n-\n-\tspin_lock(\u0026object-\u003eondemand-\u003elock);\n-\tif (object-\u003eondemand-\u003eondemand_id \u003e 0) {\n-\t\tspin_unlock(\u0026object-\u003eondemand-\u003elock);\n-\t\t/* Pair with check in cachefiles_ondemand_fd_release(). */\n-\t\tanon_file-\u003efile-\u003eprivate_data = NULL;\n-\t\tret = -EEXIST;\n-\t\tgoto err_put_file;\n-\t}\n-\n-\tload = (void *)req-\u003emsg.data;\n-\tload-\u003efd = anon_file-\u003efd;\n-\tobject-\u003eondemand-\u003eondemand_id = object_id;\n-\tspin_unlock(\u0026object-\u003eondemand-\u003elock);\n-\n-\tcachefiles_get_unbind_pincount(cache);\n-\ttrace_cachefiles_ondemand_open(object, \u0026req-\u003emsg, load);\n-\treturn 0;\n-\n-err_put_file:\n-\tfput(anon_file-\u003efile);\n-\tanon_file-\u003efile = NULL;\n-err_put_fd:\n-\tput_unused_fd(anon_file-\u003efd);\n-\tanon_file-\u003efd = ret;\n-err_free_id:\n-\txa_erase(\u0026cache-\u003eondemand_ids, object_id);\n-err:\n-\tspin_lock(\u0026object-\u003eondemand-\u003elock);\n-\t/* Avoid marking an opened object as closed. */\n-\tif (object-\u003eondemand-\u003eondemand_id \u003c= 0)\n-\t\tcachefiles_ondemand_set_object_close(object);\n-\tspin_unlock(\u0026object-\u003eondemand-\u003elock);\n-\tcachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);\n-\treturn ret;\n-}\n-\n-static void ondemand_object_worker(struct work_struct *work)\n-{\n-\tstruct cachefiles_ondemand_info *info =\n-\t\tcontainer_of(work, struct cachefiles_ondemand_info, ondemand_work);\n-\n-\tcachefiles_ondemand_init_object(info-\u003eobject);\n-}\n-\n-/*\n- * If there are any inflight or subsequent READ requests on the\n- * closed object, reopen it.\n- * Skip read requests whose related object is reopening.\n- */\n-static struct cachefiles_req *cachefiles_ondemand_select_req(struct xa_state *xas,\n-\t\t\t\t\t\t\t unsigned long xa_max)\n-{\n-\tstruct cachefiles_req *req;\n-\tstruct cachefiles_object *object;\n-\tstruct cachefiles_ondemand_info *info;\n-\n-\txas_for_each_marked(xas, req, xa_max, CACHEFILES_REQ_NEW) {\n-\t\tif (req-\u003emsg.opcode != CACHEFILES_OP_READ)\n-\t\t\treturn req;\n-\t\tobject = req-\u003eobject;\n-\t\tinfo = object-\u003eondemand;\n-\t\tif (cachefiles_ondemand_object_is_close(object)) {\n-\t\t\tcachefiles_ondemand_set_object_reopening(object);\n-\t\t\tqueue_work(fscache_wq, \u0026info-\u003eondemand_work);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (cachefiles_ondemand_object_is_reopening(object))\n-\t\t\tcontinue;\n-\t\treturn req;\n-\t}\n-\treturn NULL;\n-}\n-\n-static inline bool cachefiles_ondemand_finish_req(struct cachefiles_req *req,\n-\t\t\t\t\t\t struct xa_state *xas, int err)\n-{\n-\tif (unlikely(!xas || !req))\n-\t\treturn false;\n-\n-\tif (xa_cmpxchg(xas-\u003exa, xas-\u003exa_index, req, NULL, 0) != req)\n-\t\treturn false;\n-\n-\treq-\u003eerror = err;\n-\tcomplete(\u0026req-\u003edone);\n-\treturn true;\n-}\n-\n-ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,\n-\t\t\t\t\tchar __user *_buffer, size_t buflen)\n-{\n-\tstruct cachefiles_req *req;\n-\tstruct cachefiles_msg *msg;\n-\tsize_t n;\n-\tint ret = 0;\n-\tstruct ondemand_anon_file anon_file;\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, cache-\u003ereq_id_next);\n-\n-\txa_lock(\u0026cache-\u003ereqs);\n-\t/*\n-\t * Cyclically search for a request that has not ever been processed,\n-\t * to prevent requests from being processed repeatedly, and make\n-\t * request distribution fair.\n-\t */\n-\treq = cachefiles_ondemand_select_req(\u0026xas, ULONG_MAX);\n-\tif (!req \u0026\u0026 cache-\u003ereq_id_next \u003e 0) {\n-\t\txas_set(\u0026xas, 0);\n-\t\treq = cachefiles_ondemand_select_req(\u0026xas, cache-\u003ereq_id_next - 1);\n-\t}\n-\tif (!req) {\n-\t\txa_unlock(\u0026cache-\u003ereqs);\n-\t\treturn 0;\n-\t}\n-\n-\tmsg = \u0026req-\u003emsg;\n-\tn = msg-\u003elen;\n-\n-\tif (n \u003e buflen) {\n-\t\txa_unlock(\u0026cache-\u003ereqs);\n-\t\treturn -EMSGSIZE;\n-\t}\n-\n-\txas_clear_mark(\u0026xas, CACHEFILES_REQ_NEW);\n-\tcache-\u003ereq_id_next = xas.xa_index + 1;\n-\trefcount_inc(\u0026req-\u003eref);\n-\tcachefiles_grab_object(req-\u003eobject, cachefiles_obj_get_read_req);\n-\txa_unlock(\u0026cache-\u003ereqs);\n-\n-\tif (msg-\u003eopcode == CACHEFILES_OP_OPEN) {\n-\t\tret = cachefiles_ondemand_get_fd(req, \u0026anon_file);\n-\t\tif (ret)\n-\t\t\tgoto out;\n-\t}\n-\n-\tmsg-\u003emsg_id = xas.xa_index;\n-\tmsg-\u003eobject_id = req-\u003eobject-\u003eondemand-\u003eondemand_id;\n-\n-\tif (copy_to_user(_buffer, msg, n) != 0)\n-\t\tret = -EFAULT;\n-\n-\tif (msg-\u003eopcode == CACHEFILES_OP_OPEN) {\n-\t\tif (ret \u003c 0) {\n-\t\t\tfput(anon_file.file);\n-\t\t\tput_unused_fd(anon_file.fd);\n-\t\t\tgoto out;\n-\t\t}\n-\t\tfd_install(anon_file.fd, anon_file.file);\n-\t}\n-out:\n-\tcachefiles_put_object(req-\u003eobject, cachefiles_obj_put_read_req);\n-\t/* Remove error request and CLOSE request has no reply */\n-\tif (ret || msg-\u003eopcode == CACHEFILES_OP_CLOSE)\n-\t\tcachefiles_ondemand_finish_req(req, \u0026xas, ret);\n-\tcachefiles_req_put(req);\n-\treturn ret ? ret : n;\n-}\n-\n-typedef int (*init_req_fn)(struct cachefiles_req *req, void *private);\n-\n-static int cachefiles_ondemand_send_req(struct cachefiles_object *object,\n-\t\t\t\t\tenum cachefiles_opcode opcode,\n-\t\t\t\t\tsize_t data_len,\n-\t\t\t\t\tinit_req_fn init_req,\n-\t\t\t\t\tvoid *private)\n-{\n-\tstruct cachefiles_cache *cache = object-\u003evolume-\u003ecache;\n-\tstruct cachefiles_req *req = NULL;\n-\tXA_STATE(xas, \u0026cache-\u003ereqs, 0);\n-\tint ret;\n-\n-\tif (!test_bit(CACHEFILES_ONDEMAND_MODE, \u0026cache-\u003eflags))\n-\t\treturn 0;\n-\n-\tif (test_bit(CACHEFILES_DEAD, \u0026cache-\u003eflags)) {\n-\t\tret = -EIO;\n-\t\tgoto out;\n-\t}\n-\n-\treq = kzalloc(sizeof(*req) + data_len, GFP_KERNEL);\n-\tif (!req) {\n-\t\tret = -ENOMEM;\n-\t\tgoto out;\n-\t}\n-\n-\trefcount_set(\u0026req-\u003eref, 1);\n-\treq-\u003eobject = object;\n-\tinit_completion(\u0026req-\u003edone);\n-\treq-\u003emsg.opcode = opcode;\n-\treq-\u003emsg.len = sizeof(struct cachefiles_msg) + data_len;\n-\n-\tret = init_req(req, private);\n-\tif (ret)\n-\t\tgoto out;\n-\n-\tdo {\n-\t\t/*\n-\t\t * Stop enqueuing the request when daemon is dying. The\n-\t\t * following two operations need to be atomic as a whole.\n-\t\t * 1) check cache state, and\n-\t\t * 2) enqueue request if cache is alive.\n-\t\t * Otherwise the request may be enqueued after xarray has been\n-\t\t * flushed, leaving the orphan request never being completed.\n-\t\t *\n-\t\t * CPU 1\t\t\tCPU 2\n-\t\t * =====\t\t\t=====\n-\t\t *\t\t\t\ttest CACHEFILES_DEAD bit\n-\t\t * set CACHEFILES_DEAD bit\n-\t\t * flush requests in the xarray\n-\t\t *\t\t\t\tenqueue the request\n-\t\t */\n-\t\txas_lock(\u0026xas);\n-\n-\t\tif (test_bit(CACHEFILES_DEAD, \u0026cache-\u003eflags) ||\n-\t\t cachefiles_ondemand_object_is_dropping(object)) {\n-\t\t\txas_unlock(\u0026xas);\n-\t\t\tret = -EIO;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\t/* coupled with the barrier in cachefiles_flush_reqs() */\n-\t\tsmp_mb();\n-\n-\t\tif (opcode == CACHEFILES_OP_CLOSE \u0026\u0026\n-\t\t !cachefiles_ondemand_object_is_open(object)) {\n-\t\t\tWARN_ON_ONCE(object-\u003eondemand-\u003eondemand_id == 0);\n-\t\t\txas_unlock(\u0026xas);\n-\t\t\tret = -EIO;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\t/*\n-\t\t * Cyclically find a free xas to avoid msg_id reuse that would\n-\t\t * cause the daemon to successfully copen a stale msg_id.\n-\t\t */\n-\t\txas.xa_index = cache-\u003emsg_id_next;\n-\t\txas_find_marked(\u0026xas, UINT_MAX, XA_FREE_MARK);\n-\t\tif (xas.xa_node == XAS_RESTART) {\n-\t\t\txas.xa_index = 0;\n-\t\t\txas_find_marked(\u0026xas, cache-\u003emsg_id_next - 1, XA_FREE_MARK);\n-\t\t}\n-\t\tif (xas.xa_node == XAS_RESTART)\n-\t\t\txas_set_err(\u0026xas, -EBUSY);\n-\n-\t\txas_store(\u0026xas, req);\n-\t\tif (xas_valid(\u0026xas)) {\n-\t\t\tcache-\u003emsg_id_next = xas.xa_index + 1;\n-\t\t\txas_clear_mark(\u0026xas, XA_FREE_MARK);\n-\t\t\txas_set_mark(\u0026xas, CACHEFILES_REQ_NEW);\n-\t\t}\n-\t\txas_unlock(\u0026xas);\n-\t} while (xas_nomem(\u0026xas, GFP_KERNEL));\n-\n-\tret = xas_error(\u0026xas);\n-\tif (ret)\n-\t\tgoto out;\n-\n-\twake_up_all(\u0026cache-\u003edaemon_pollwq);\n-wait:\n-\tret = wait_for_completion_killable(\u0026req-\u003edone);\n-\tif (!ret) {\n-\t\tret = req-\u003eerror;\n-\t} else {\n-\t\tret = -EINTR;\n-\t\tif (!cachefiles_ondemand_finish_req(req, \u0026xas, ret)) {\n-\t\t\t/* Someone will complete it soon. */\n-\t\t\tcpu_relax();\n-\t\t\tgoto wait;\n-\t\t}\n-\t}\n-\tcachefiles_req_put(req);\n-\treturn ret;\n-out:\n-\t/* Reset the object to close state in error handling path.\n-\t * If error occurs after creating the anonymous fd,\n-\t * cachefiles_ondemand_fd_release() will set object to close.\n-\t */\n-\tif (opcode == CACHEFILES_OP_OPEN \u0026\u0026\n-\t !cachefiles_ondemand_object_is_dropping(object))\n-\t\tcachefiles_ondemand_set_object_close(object);\n-\tkfree(req);\n-\treturn ret;\n-}\n-\n-static int cachefiles_ondemand_init_open_req(struct cachefiles_req *req,\n-\t\t\t\t\t void *private)\n-{\n-\tstruct cachefiles_object *object = req-\u003eobject;\n-\tstruct fscache_cookie *cookie = object-\u003ecookie;\n-\tstruct fscache_volume *volume = object-\u003evolume-\u003evcookie;\n-\tstruct cachefiles_open *load = (void *)req-\u003emsg.data;\n-\tsize_t volume_key_size, cookie_key_size;\n-\tvoid *volume_key, *cookie_key;\n-\n-\t/*\n-\t * Volume key is a NUL-terminated string. key[0] stores strlen() of the\n-\t * string, followed by the content of the string (excluding '\\0').\n-\t */\n-\tvolume_key_size = volume-\u003ekey[0] + 1;\n-\tvolume_key = volume-\u003ekey + 1;\n-\n-\t/* Cookie key is binary data, which is netfs specific. */\n-\tcookie_key_size = cookie-\u003ekey_len;\n-\tcookie_key = fscache_get_key(cookie);\n-\n-\tif (!(object-\u003ecookie-\u003eadvice \u0026 FSCACHE_ADV_WANT_CACHE_SIZE)) {\n-\t\tpr_err(\"WANT_CACHE_SIZE is needed for on-demand mode\\n\");\n-\t\treturn -EINVAL;\n-\t}\n-\n-\tload-\u003evolume_key_size = volume_key_size;\n-\tload-\u003ecookie_key_size = cookie_key_size;\n-\tmemcpy(load-\u003edata, volume_key, volume_key_size);\n-\tmemcpy(load-\u003edata + volume_key_size, cookie_key, cookie_key_size);\n-\n-\treturn 0;\n-}\n-\n-static int cachefiles_ondemand_init_close_req(struct cachefiles_req *req,\n-\t\t\t\t\t void *private)\n-{\n-\tstruct cachefiles_object *object = req-\u003eobject;\n-\n-\tif (!cachefiles_ondemand_object_is_open(object))\n-\t\treturn -ENOENT;\n-\n-\ttrace_cachefiles_ondemand_close(object, \u0026req-\u003emsg);\n-\treturn 0;\n-}\n-\n-struct cachefiles_read_ctx {\n-\tloff_t off;\n-\tsize_t len;\n-};\n-\n-static int cachefiles_ondemand_init_read_req(struct cachefiles_req *req,\n-\t\t\t\t\t void *private)\n-{\n-\tstruct cachefiles_object *object = req-\u003eobject;\n-\tstruct cachefiles_read *load = (void *)req-\u003emsg.data;\n-\tstruct cachefiles_read_ctx *read_ctx = private;\n-\n-\tload-\u003eoff = read_ctx-\u003eoff;\n-\tload-\u003elen = read_ctx-\u003elen;\n-\ttrace_cachefiles_ondemand_read(object, \u0026req-\u003emsg, load);\n-\treturn 0;\n-}\n-\n-int cachefiles_ondemand_init_object(struct cachefiles_object *object)\n-{\n-\tstruct fscache_cookie *cookie = object-\u003ecookie;\n-\tstruct fscache_volume *volume = object-\u003evolume-\u003evcookie;\n-\tsize_t volume_key_size, cookie_key_size, data_len;\n-\n-\tif (!object-\u003eondemand)\n-\t\treturn 0;\n-\n-\t/*\n-\t * CacheFiles will firstly check the cache file under the root cache\n-\t * directory. If the coherency check failed, it will fallback to\n-\t * creating a new tmpfile as the cache file. Reuse the previously\n-\t * allocated object ID if any.\n-\t */\n-\tif (cachefiles_ondemand_object_is_open(object))\n-\t\treturn 0;\n-\n-\tvolume_key_size = volume-\u003ekey[0] + 1;\n-\tcookie_key_size = cookie-\u003ekey_len;\n-\tdata_len = sizeof(struct cachefiles_open) +\n-\t\t volume_key_size + cookie_key_size;\n-\n-\treturn cachefiles_ondemand_send_req(object, CACHEFILES_OP_OPEN,\n-\t\t\tdata_len, cachefiles_ondemand_init_open_req, NULL);\n-}\n-\n-void cachefiles_ondemand_clean_object(struct cachefiles_object *object)\n-{\n-\tunsigned long index;\n-\tstruct cachefiles_req *req;\n-\tstruct cachefiles_cache *cache;\n-\n-\tif (!object-\u003eondemand)\n-\t\treturn;\n-\n-\tcachefiles_ondemand_send_req(object, CACHEFILES_OP_CLOSE, 0,\n-\t\t\tcachefiles_ondemand_init_close_req, NULL);\n-\n-\tif (!object-\u003eondemand-\u003eondemand_id)\n-\t\treturn;\n-\n-\t/* Cancel all requests for the object that is being dropped. */\n-\tcache = object-\u003evolume-\u003ecache;\n-\txa_lock(\u0026cache-\u003ereqs);\n-\tcachefiles_ondemand_set_object_dropping(object);\n-\txa_for_each(\u0026cache-\u003ereqs, index, req) {\n-\t\tif (req-\u003eobject == object) {\n-\t\t\treq-\u003eerror = -EIO;\n-\t\t\tcomplete(\u0026req-\u003edone);\n-\t\t\t__xa_erase(\u0026cache-\u003ereqs, index);\n-\t\t}\n-\t}\n-\txa_unlock(\u0026cache-\u003ereqs);\n-\n-\t/* Wait for ondemand_object_worker() to finish to avoid UAF. */\n-\tcancel_work_sync(\u0026object-\u003eondemand-\u003eondemand_work);\n-}\n-\n-int cachefiles_ondemand_init_obj_info(struct cachefiles_object *object,\n-\t\t\t\tstruct cachefiles_volume *volume)\n-{\n-\tif (!cachefiles_in_ondemand_mode(volume-\u003ecache))\n-\t\treturn 0;\n-\n-\tobject-\u003eondemand = kzalloc_obj(struct cachefiles_ondemand_info);\n-\tif (!object-\u003eondemand)\n-\t\treturn -ENOMEM;\n-\n-\tobject-\u003eondemand-\u003eobject = object;\n-\tspin_lock_init(\u0026object-\u003eondemand-\u003elock);\n-\tINIT_WORK(\u0026object-\u003eondemand-\u003eondemand_work, ondemand_object_worker);\n-\treturn 0;\n-}\n-\n-void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *object)\n-{\n-\tkfree(object-\u003eondemand);\n-\tobject-\u003eondemand = NULL;\n-}\n-\n-int cachefiles_ondemand_read(struct cachefiles_object *object,\n-\t\t\t loff_t pos, size_t len)\n-{\n-\tstruct cachefiles_read_ctx read_ctx = {pos, len};\n-\n-\treturn cachefiles_ondemand_send_req(object, CACHEFILES_OP_READ,\n-\t\t\tsizeof(struct cachefiles_read),\n-\t\t\tcachefiles_ondemand_init_read_req, \u0026read_ctx);\n-}\ndiff --git a/fs/cachefiles/xattr.c b/fs/cachefiles/xattr.c\nindex f8ae78b3f7b6d3..25f2a906f9841c 100644\n--- a/fs/cachefiles/xattr.c\n+++ b/fs/cachefiles/xattr.c\n@@ -54,7 +54,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)\n \tif (!buf)\n \t\treturn -ENOMEM;\n \n-\tbuf-\u003eobject_size\t= cpu_to_be64(object-\u003ecookie-\u003eobject_size);\n+\tbuf-\u003eobject_size\t= cpu_to_be64(object-\u003eobject_size);\n \tbuf-\u003ezero_point\t\t= 0;\n \tbuf-\u003etype\t\t= CACHEFILES_COOKIE_TYPE_DATA;\n \tbuf-\u003econtent\t\t= object-\u003econtent_info;\n@@ -77,6 +77,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)\n \t\ttrace_cachefiles_vfs_error(object, file_inode(file), ret,\n \t\t\t\t\t cachefiles_trace_setxattr_error);\n \t\ttrace_cachefiles_coherency(object, file_inode(file)-\u003ei_ino,\n+\t\t\t\t\t object-\u003eobject_size,\n \t\t\t\t\t be64_to_cpup((__be64 *)buf-\u003edata),\n \t\t\t\t\t buf-\u003econtent,\n \t\t\t\t\t cachefiles_coherency_set_fail);\n@@ -86,6 +87,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)\n \t\t\t\t\"Failed to set xattr with error %d\", ret);\n \t} else {\n \t\ttrace_cachefiles_coherency(object, file_inode(file)-\u003ei_ino,\n+\t\t\t\t\t object-\u003eobject_size,\n \t\t\t\t\t be64_to_cpup((__be64 *)buf-\u003edata),\n \t\t\t\t\t buf-\u003econtent,\n \t\t\t\t\t cachefiles_coherency_set_ok);\n@@ -103,9 +105,11 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file\n {\n \tstruct cachefiles_xattr *buf;\n \tstruct dentry *dentry = file-\u003ef_path.dentry;\n+\tstruct inode *inode = file_inode(file);\n \tunsigned int len = object-\u003ecookie-\u003eaux_len, tlen;\n \tconst void *p = fscache_get_aux(object-\u003ecookie);\n \tenum cachefiles_coherency_trace why;\n+\tunsigned long long obj_size;\n \tssize_t xlen;\n \tint ret = -ESTALE;\n \n@@ -120,36 +124,41 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file\n \tif (xlen != tlen) {\n \t\tif (xlen \u003c 0) {\n \t\t\tret = xlen;\n-\t\t\ttrace_cachefiles_vfs_error(object, file_inode(file), xlen,\n+\t\t\ttrace_cachefiles_vfs_error(object, inode, xlen,\n \t\t\t\t\t\t cachefiles_trace_getxattr_error);\n \t\t}\n \t\tif (xlen == -EIO)\n \t\t\tcachefiles_io_error_obj(\n \t\t\t\tobject,\n \t\t\t\t\"Failed to read aux with error %zd\", xlen);\n-\t\twhy = cachefiles_coherency_check_xattr;\n+\t\ttrace_cachefiles_coherency(object, inode-\u003ei_ino, 0, 0, 0,\n+\t\t\t\t\t cachefiles_coherency_check_xattr);\n \t\tgoto out;\n \t}\n \n+\tobj_size = be64_to_cpu(buf-\u003eobject_size);\n \tif (buf-\u003etype != CACHEFILES_COOKIE_TYPE_DATA) {\n \t\twhy = cachefiles_coherency_check_type;\n \t} else if (memcmp(buf-\u003edata, p, len) != 0) {\n \t\twhy = cachefiles_coherency_check_aux;\n-\t} else if (be64_to_cpu(buf-\u003eobject_size) != object-\u003ecookie-\u003eobject_size) {\n+\t} else if (obj_size != object-\u003ecookie-\u003eobject_size) {\n \t\twhy = cachefiles_coherency_check_objsize;\n \t} else if (buf-\u003econtent == CACHEFILES_CONTENT_DIRTY) {\n \t\t// TODO: Begin conflict resolution\n \t\tpr_warn(\"Dirty object in cache\\n\");\n \t\twhy = cachefiles_coherency_check_dirty;\n \t} else {\n+\t\tobject-\u003econtent_info = buf-\u003econtent;\n+\t\tobject-\u003eobject_size = obj_size;\n+\t\tatomic64_set(\u0026object-\u003eread_limit, i_size_read(inode));\n \t\twhy = cachefiles_coherency_check_ok;\n \t\tret = 0;\n \t}\n \n-out:\n-\ttrace_cachefiles_coherency(object, file_inode(file)-\u003ei_ino,\n+\ttrace_cachefiles_coherency(object, inode-\u003ei_ino, obj_size,\n \t\t\t\t be64_to_cpup((__be64 *)buf-\u003edata),\n \t\t\t\t buf-\u003econtent, why);\n+out:\n \tkfree(buf);\n \treturn ret;\n }\n@@ -163,6 +172,9 @@ int cachefiles_remove_object_xattr(struct cachefiles_cache *cache,\n {\n \tint ret;\n \n+\ttrace_cachefiles_coherency(object, d_inode(dentry)-\u003ei_ino, 0, 0, 0,\n+\t\t\t\t cachefiles_coherency_remove);\n+\n \tret = cachefiles_inject_remove_error();\n \tif (ret == 0) {\n \t\tret = mnt_want_write(cache-\u003emnt);\ndiff --git a/fs/ceph/Kconfig b/fs/ceph/Kconfig\nindex 3d64a316ca31d3..aa6ccd7794d233 100644\n--- a/fs/ceph/Kconfig\n+++ b/fs/ceph/Kconfig\n@@ -4,6 +4,7 @@ config CEPH_FS\n \tdepends on INET\n \tselect CEPH_LIB\n \tselect NETFS_SUPPORT\n+\tselect NETFS_PGPRIV2\n \tselect FS_ENCRYPTION_ALGS if FS_ENCRYPTION\n \tdefault n\n \thelp\ndiff --git a/fs/ceph/addr.c b/fs/ceph/addr.c\nindex 61bd6d92ff257a..d3d1e32a8a4aee 100644\n--- a/fs/ceph/addr.c\n+++ b/fs/ceph/addr.c\n@@ -274,7 +274,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)\n \tceph_dec_osd_stopping_blocker(fsc-\u003emdsc);\n }\n \n-static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n+static int ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n \tstruct inode *inode = rreq-\u003einode;\n@@ -283,7 +283,8 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n \tstruct ceph_mds_request *req;\n \tstruct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode-\u003ei_sb);\n \tstruct ceph_inode_info *ci = ceph_inode(inode);\n-\tssize_t err = 0;\n+\tstruct iov_iter iter;\n+\tssize_t err;\n \tsize_t len;\n \tint mode;\n \n@@ -292,8 +293,20 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n \t\t__set_bit(NETFS_SREQ_CLEAR_TAIL, \u0026subreq-\u003eflags);\n \t__clear_bit(NETFS_SREQ_COPY_TO_CACHE, \u0026subreq-\u003eflags);\n \n-\tif (subreq-\u003estart \u003e= inode-\u003ei_size)\n+\terr = netfs_prepare_read_buffer(subreq, INT_MAX);\n+\tif (err \u003c 0)\n+\t\treturn err;\n+\t/* After this point, must fail by termination. */\n+\n+\tif (subreq-\u003estart \u003e= inode-\u003ei_size) {\n+\t\t__set_bit(NETFS_SREQ_HIT_EOF, \u0026subreq-\u003eflags);\n+\t\terr = 0;\n \t\tgoto out;\n+\t}\n+\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset,\n+\t\t\t subreq-\u003elen);\n \n \t/* We need to fetch the inline data. */\n \tmode = ceph_try_to_choose_auth_mds(inode, CEPH_STAT_CAP_INLINE_DATA);\n@@ -302,11 +315,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n \t\terr = PTR_ERR(req);\n \t\tgoto out;\n \t}\n+\n \treq-\u003er_ino1 = ci-\u003ei_vino;\n \treq-\u003er_args.getattr.mask = cpu_to_le32(CEPH_STAT_CAP_INLINE_DATA);\n \treq-\u003er_num_caps = 2;\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n+\n \terr = ceph_mdsc_do_request(mdsc, NULL, req);\n \tif (err \u003c 0)\n \t\tgoto out;\n@@ -316,11 +331,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n \tif (iinfo-\u003einline_version == CEPH_INLINE_NONE) {\n \t\t/* The data got uninlined */\n \t\tceph_mdsc_put_request(req);\n-\t\treturn false;\n+\t\t__set_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t\terr = -EAGAIN;\n+\t\tgoto out;\n \t}\n \n \tlen = min_t(size_t, iinfo-\u003einline_len - subreq-\u003estart, subreq-\u003elen);\n-\terr = copy_to_iter(iinfo-\u003einline_data + subreq-\u003estart, len, \u0026subreq-\u003eio_iter);\n+\terr = copy_to_iter(iinfo-\u003einline_data + subreq-\u003estart, len, \u0026iter);\n \tif (err == 0) {\n \t\terr = -EFAULT;\n \t} else {\n@@ -333,26 +350,10 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)\n \tsubreq-\u003eerror = err;\n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);\n \tnetfs_read_subreq_terminated(subreq);\n-\treturn true;\n-}\n-\n-static int ceph_netfs_prepare_read(struct netfs_io_subrequest *subreq)\n-{\n-\tstruct netfs_io_request *rreq = subreq-\u003erreq;\n-\tstruct inode *inode = rreq-\u003einode;\n-\tstruct ceph_inode_info *ci = ceph_inode(inode);\n-\tstruct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);\n-\tu64 objno, objoff;\n-\tu32 xlen;\n-\n-\t/* Truncate the extent at the end of the current block */\n-\tceph_calc_file_object_mapping(\u0026ci-\u003ei_layout, subreq-\u003estart, subreq-\u003elen,\n-\t\t\t\t \u0026objno, \u0026objoff, \u0026xlen);\n-\trreq-\u003eio_streams[0].sreq_max_len = umin(xlen, fsc-\u003emount_options-\u003ersize);\n \treturn 0;\n }\n \n-static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n+static int ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n \tstruct inode *inode = rreq-\u003einode;\n@@ -361,19 +362,18 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n \tstruct ceph_client *cl = fsc-\u003eclient;\n \tstruct ceph_osd_request *req = NULL;\n \tstruct ceph_vino vino = ceph_vino(inode);\n-\tint err;\n-\tu64 len;\n+\tstruct iov_iter iter;\n+\tu64 objno, objoff, len, off = subreq-\u003estart;\n+\tu32 maxlen;\n+\tint err = -EIO;\n \tbool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD);\n-\tu64 off = subreq-\u003estart;\n \tint extent_cnt;\n \n-\tif (ceph_inode_is_shutdown(inode)) {\n-\t\terr = -EIO;\n-\t\tgoto out;\n-\t}\n+\tif (ceph_inode_is_shutdown(inode))\n+\t\treturn -EIO;\n \n-\tif (ceph_has_inline_data(ci) \u0026\u0026 ceph_netfs_issue_op_inline(subreq))\n-\t\treturn;\n+\tif (ceph_has_inline_data(ci))\n+\t\treturn ceph_netfs_issue_op_inline(subreq);\n \n \t// TODO: This rounding here is slightly dodgy. It *should* work, for\n \t// now, as the cache only deals in blocks that are a multiple of\n@@ -383,26 +383,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n \tlen = subreq-\u003elen;\n \tceph_fscrypt_adjust_off_and_len(inode, \u0026off, \u0026len);\n \n+\t/* Truncate the extent at the end of the current block */\n+\tceph_calc_file_object_mapping(\u0026ci-\u003ei_layout, subreq-\u003estart, len,\n+\t\t\t\t \u0026objno, \u0026objoff, \u0026maxlen);\n+\tmaxlen = min(maxlen, fsc-\u003emount_options-\u003ersize);\n+\tlen = min(len, maxlen);\n+\tsubreq-\u003elen = len;\n+\n+\t/* Grab a slice of read buffer. This may shrink the subreq. */\n+\terr = netfs_prepare_read_buffer(subreq, INT_MAX);\n+\tif (err \u003c 0)\n+\t\treturn err;\n+\t/* After this point, must fail by termination. */\n+\n+\t/* Create a request. In theory, this may shrink the request again, but\n+\t * it shouldn't since we calculated the object size above and already\n+\t * shrank to that, but if it does, we'll just end up doing a short read\n+\t * and retrying to get the rest.\n+\t */\n+\tlen = subreq-\u003elen;\n \treq = ceph_osdc_new_request(\u0026fsc-\u003eclient-\u003eosdc, \u0026ci-\u003ei_layout, vino,\n \t\t\toff, \u0026len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ,\n \t\t\tCEPH_OSD_FLAG_READ, NULL, ci-\u003ei_truncate_seq,\n \t\t\tci-\u003ei_truncate_size, false);\n \tif (IS_ERR(req)) {\n \t\terr = PTR_ERR(req);\n-\t\treq = NULL;\n-\t\tgoto out;\n+\t\tgoto failed_noput;\n \t}\n \n \tif (sparse) {\n \t\textent_cnt = __ceph_sparse_read_ext_count(inode, len);\n \t\terr = ceph_alloc_sparse_ext_map(\u0026req-\u003er_ops[0], extent_cnt);\n \t\tif (err)\n-\t\t\tgoto out;\n+\t\t\tgoto failed;\n \t}\n \n \tdoutc(cl, \"%llx.%llx pos=%llu orig_len=%zu len=%llu\\n\",\n \t ceph_vinop(inode), subreq-\u003estart, subreq-\u003elen, len);\n \n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset,\n+\t\t\t subreq-\u003elen);\n+\n \t/*\n \t * FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for\n \t * encrypted inodes. We'd need infrastructure that handles an iov_iter\n@@ -421,13 +443,11 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n \t\t * ceph_msg_data_cursor_init() triggers BUG_ON() in the case\n \t\t * if msg-\u003esparse_read_total \u003e msg-\u003edata_length.\n \t\t */\n-\t\tsubreq-\u003eio_iter.count = len;\n-\n-\t\terr = iov_iter_get_pages_alloc2(\u0026subreq-\u003eio_iter, \u0026pages, len, \u0026page_off);\n+\t\terr = iov_iter_get_pages_alloc2(\u0026iter, \u0026pages, len, \u0026page_off);\n \t\tif (err \u003c 0) {\n \t\t\tdoutc(cl, \"%llx.%llx failed to allocate pages, %d\\n\",\n \t\t\t ceph_vinop(inode), err);\n-\t\t\tgoto out;\n+\t\t\tgoto eio;\n \t\t}\n \n \t\t/* should always give us a page-aligned read */\n@@ -438,12 +458,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n \t\tosd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false,\n \t\t\t\t\t\t false);\n \t} else {\n-\t\tosd_req_op_extent_osd_iter(req, 0, \u0026subreq-\u003eio_iter);\n-\t}\n-\tif (!ceph_inc_osd_stopping_blocker(fsc-\u003emdsc)) {\n-\t\terr = -EIO;\n-\t\tgoto out;\n+\t\tosd_req_op_extent_osd_iter(req, 0, \u0026iter);\n \t}\n+\tif (!ceph_inc_osd_stopping_blocker(fsc-\u003emdsc))\n+\t\tgoto eio;\n \treq-\u003er_callback = finish_netfs_read;\n \treq-\u003er_priv = subreq;\n \treq-\u003er_inode = inode;\n@@ -451,19 +469,22 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n \tceph_osdc_start_request(req-\u003er_osdc, req);\n-out:\n \tceph_osdc_put_request(req);\n-\tif (err) {\n-\t\tsubreq-\u003eerror = err;\n-\t\tnetfs_read_subreq_terminated(subreq);\n-\t}\n-\tdoutc(cl, \"%llx.%llx result %d\\n\", ceph_vinop(inode), err);\n+\tdoutc(cl, \"%llx.%llx result -EIOCBQUEUED\\n\", ceph_vinop(inode));\n+\treturn 0;\n+eio:\n+\terr = -EIO;\n+failed:\n+\tceph_osdc_put_request(req);\n+failed_noput:\n+\tsubreq-\u003eerror = err;\n+\tnetfs_read_subreq_terminated(subreq);\n+\treturn 0;\n }\n \n static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)\n {\n \tstruct inode *inode = rreq-\u003einode;\n-\tstruct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);\n \tstruct ceph_client *cl = ceph_inode_to_client(inode);\n \tint got = 0, want = CEPH_CAP_FILE_CACHE;\n \tstruct ceph_netfs_request_data *priv;\n@@ -515,7 +536,6 @@ static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)\n \n \tpriv-\u003ecaps = got;\n \trreq-\u003enetfs_priv = priv;\n-\trreq-\u003eio_streams[0].sreq_max_len = fsc-\u003emount_options-\u003ersize;\n \n out:\n \tif (ret \u003c 0) {\n@@ -543,7 +563,6 @@ static void ceph_netfs_free_request(struct netfs_io_request *rreq)\n const struct netfs_request_ops ceph_netfs_ops = {\n \t.init_request\t\t= ceph_init_request,\n \t.free_request\t\t= ceph_netfs_free_request,\n-\t.prepare_read\t\t= ceph_netfs_prepare_read,\n \t.issue_read\t\t= ceph_netfs_issue_read,\n \t.expand_readahead\t= ceph_netfs_expand_readahead,\n \t.check_write_begin\t= ceph_netfs_check_write_begin,\ndiff --git a/fs/netfs/Kconfig b/fs/netfs/Kconfig\nindex 7701c037c3283f..d0e7b0971fa3f0 100644\n--- a/fs/netfs/Kconfig\n+++ b/fs/netfs/Kconfig\n@@ -22,6 +22,9 @@ config NETFS_STATS\n \t between CPUs. On the other hand, the stats are very useful for\n \t debugging purposes. Saying 'Y' here is recommended.\n \n+config NETFS_PGPRIV2\n+\tbool\n+\n config NETFS_DEBUG\n \tbool \"Enable dynamic debugging netfslib and FS-Cache\"\n \tdepends on NETFS_SUPPORT\ndiff --git a/fs/netfs/Makefile b/fs/netfs/Makefile\nindex b43188d64bd87e..421dd0be413b3d 100644\n--- a/fs/netfs/Makefile\n+++ b/fs/netfs/Makefile\n@@ -3,6 +3,7 @@\n netfs-y := \\\n \tbuffered_read.o \\\n \tbuffered_write.o \\\n+\tbvecq.o \\\n \tdirect_read.o \\\n \tdirect_write.o \\\n \titerator.o \\\n@@ -11,14 +12,13 @@ netfs-y := \\\n \tmisc.o \\\n \tobjects.o \\\n \tread_collect.o \\\n-\tread_pgpriv2.o \\\n \tread_retry.o \\\n \tread_single.o \\\n-\trolling_buffer.o \\\n \twrite_collect.o \\\n \twrite_issue.o \\\n \twrite_retry.o\n \n+netfs-$(CONFIG_NETFS_PGPRIV2) += read_pgpriv2.o\n netfs-$(CONFIG_NETFS_STATS) += stats.o\n \n netfs-$(CONFIG_FSCACHE) += \\\ndiff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c\nindex 24a8a5418e3119..c1ff33dcf536ee 100644\n--- a/fs/netfs/buffered_read.c\n+++ b/fs/netfs/buffered_read.c\n@@ -54,6 +54,42 @@ static void netfs_rreq_expand(struct netfs_io_request *rreq,\n \t}\n }\n \n+/*\n+ * Drop the folio refs acquired from the readahead API.\n+ */\n+static void netfs_bulk_drop_ra_refs(struct netfs_io_request *rreq)\n+{\n+\tstruct folio_batch fbatch;\n+\tstruct folio *folio;\n+\tpgoff_t nr_pages = DIV_ROUND_UP(rreq-\u003elen, PAGE_SIZE);\n+\tpgoff_t first = rreq-\u003estart / PAGE_SIZE;\n+\tXA_STATE(xas, \u0026rreq-\u003emapping-\u003ei_pages, first);\n+\n+\tfolio_batch_init(\u0026fbatch);\n+\n+\trcu_read_lock();\n+\n+\txas_for_each(\u0026xas, folio, first + nr_pages - 1) {\n+\t\tif (xas_retry(\u0026xas, folio))\n+\t\t\tcontinue;\n+\n+\t\tif (!folio_batch_add(\u0026fbatch, folio))\n+\t\t\tfolio_batch_release(\u0026fbatch);\n+\t}\n+\n+\trcu_read_unlock();\n+\tfolio_batch_release(\u0026fbatch);\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_ra_put_ref);\n+\tclear_bit_unlock(NETFS_RREQ_NEED_PUT_RA_REFS, \u0026rreq-\u003eflags);\n+\twake_up(\u0026rreq-\u003ewaitq);\n+}\n+\n+static void netfs_maybe_bulk_drop_ra_refs(struct netfs_io_request *rreq)\n+{\n+\tif (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, \u0026rreq-\u003eflags))\n+\t\tnetfs_bulk_drop_ra_refs(rreq);\n+}\n+\n /*\n * Begin an operation, and fetch the stored zero point value from the cookie if\n * available.\n@@ -64,103 +100,98 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in\n }\n \n /*\n- * netfs_prepare_read_iterator - Prepare the subreq iterator for I/O\n- * @subreq: The subrequest to be set up\n- *\n- * Prepare the I/O iterator representing the read buffer on a subrequest for\n- * the filesystem to use for I/O (it can be passed directly to a socket). This\n- * is intended to be called from the -\u003eissue_read() method once the filesystem\n- * has trimmed the request to the size it wants.\n- *\n- * Returns the limited size if successful and -ENOMEM if insufficient memory\n- * available.\n- *\n- * [!] NOTE: This must be run in the same thread as -\u003eissue_read() was called\n- * in as we access the readahead_control struct.\n+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to\n+ * use as a bvec queue.\n */\n-static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,\n-\t\t\t\t\t struct readahead_control *ractl)\n+static int netfs_prepare_buffered_read_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n-\tsize_t rsize = subreq-\u003elen;\n-\n-\tif (subreq-\u003esource == NETFS_DOWNLOAD_FROM_SERVER)\n-\t\trsize = umin(rsize, rreq-\u003eio_streams[0].sreq_max_len);\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\tssize_t extracted;\n \n-\tif (ractl) {\n-\t\t/* If we don't have sufficient folios in the rolling buffer,\n-\t\t * extract a folioq's worth from the readahead region at a time\n-\t\t * into the buffer. Note that this acquires a ref on each page\n-\t\t * that we will need to release later - but we don't want to do\n-\t\t * that until after we've started the I/O.\n-\t\t */\n-\t\tstruct folio_batch put_batch;\n+\t_enter(\"R=%08x[%x] l=%zx s=%u\",\n+\t rreq-\u003edebug_id, subreq-\u003edebug_index, subreq-\u003elen, max_segs);\n \n-\t\tfolio_batch_init(\u0026put_batch);\n-\t\twhile (rreq-\u003esubmitted \u003c subreq-\u003estart + rsize) {\n-\t\t\tssize_t added;\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026subreq-\u003edispatch_pos);\n+\textracted = bvecq_slice(\u0026stream-\u003edispatch_cursor, subreq-\u003elen,\n+\t\t\t\tmax_segs, \u0026subreq-\u003enr_segs);\n \n-\t\t\tadded = rolling_buffer_load_from_ra(\u0026rreq-\u003ebuffer, ractl,\n-\t\t\t\t\t\t\t \u0026put_batch);\n-\t\t\tif (added \u003c 0)\n-\t\t\t\treturn added;\n-\t\t\trreq-\u003esubmitted += added;\n-\t\t}\n-\t\tfolio_batch_release(\u0026put_batch);\n+\tif (extracted \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = extracted;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n \t}\n+\tstream-\u003ebuffered -= extracted;\n+\tstream-\u003eissue_from = subreq-\u003estart + subreq-\u003elen;\n+\trreq-\u003esubmitted = stream-\u003eissue_from;\n \n-\tsubreq-\u003elen = rsize;\n-\tif (unlikely(rreq-\u003eio_streams[0].sreq_max_segs)) {\n-\t\tsize_t limit = netfs_limit_iter(\u0026rreq-\u003ebuffer.iter, 0, rsize,\n-\t\t\t\t\t\trreq-\u003eio_streams[0].sreq_max_segs);\n+\tif (!stream-\u003ebuffered)\n+\t\tnetfs_all_subreqs_queued(rreq);\n+\treturn 0;\n+}\n \n-\t\tif (limit \u003c rsize) {\n-\t\t\tsubreq-\u003elen = limit;\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n-\t\t}\n+/**\n+ * netfs_prepare_read_buffer - Get the buffer for a subrequest\n+ * @subreq: The subrequest to get the buffer for\n+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)\n+ *\n+ * Extract a slice of buffer from the stream and attach it to the subrequest as\n+ * a bio_vec queue. The maximum amount of data attached is set by\n+ * @subreq-\u003elen, but this may be shortened if @max_segs would be exceeded.\n+ *\n+ * [!] NOTE: This must be run in the same thread as -\u003eissue_read() was called\n+ * in as we access the readahead_control struct if there is one.\n+ */\n+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t unsigned int max_segs)\n+{\n+\tswitch (subreq-\u003erreq-\u003eorigin) {\n+\tcase NETFS_READAHEAD:\n+\tcase NETFS_READPAGE:\n+\tcase NETFS_READ_FOR_WRITE:\n+\t\tif (subreq-\u003eretry_count)\n+\t\t\treturn netfs_prepare_buffered_read_retry_buffer(subreq, max_segs);\n+\t\treturn netfs_prepare_buffered_read_buffer(subreq, max_segs);\n+\n+\tcase NETFS_UNBUFFERED_READ:\n+\tcase NETFS_DIO_READ:\n+\tcase NETFS_READ_GAPS:\n+\t\treturn netfs_prepare_unbuffered_read_buffer(subreq, max_segs);\n+\tcase NETFS_READ_SINGLE:\n+\t\treturn netfs_prepare_read_single_buffer(subreq, max_segs);\n+\tdefault:\n+\t\tWARN_ON_ONCE(1);\n+\t\treturn -EIO;\n \t}\n-\n-\tsubreq-\u003eio_iter\t= rreq-\u003ebuffer.iter;\n-\n-\tiov_iter_truncate(\u0026subreq-\u003eio_iter, subreq-\u003elen);\n-\trolling_buffer_advance(\u0026rreq-\u003ebuffer, subreq-\u003elen);\n-\treturn subreq-\u003elen;\n }\n+EXPORT_SYMBOL(netfs_prepare_read_buffer);\n \n-static enum netfs_io_source netfs_cache_prepare_read(struct netfs_io_request *rreq,\n-\t\t\t\t\t\t struct netfs_io_subrequest *subreq,\n-\t\t\t\t\t\t loff_t i_size)\n+void netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)\n {\n \tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n-\tenum netfs_io_source source;\n \n+\tocc-\u003egranularity = PAGE_SIZE;\n+\tif (occ-\u003equery_from \u003e= occ-\u003equery_to)\n+\t\treturn;\n \tif (!cres-\u003eops)\n-\t\treturn NETFS_DOWNLOAD_FROM_SERVER;\n-\tsource = cres-\u003eops-\u003eprepare_read(subreq, i_size);\n-\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n-\treturn source;\n-\n+\t\treturn;\n+\tocc-\u003equery_from = round_up(occ-\u003equery_from, occ-\u003egranularity);\n+\tcres-\u003eops-\u003equery_occupancy(cres, occ);\n }\n \n /*\n- * Issue a read against the cache.\n- * - Eats the caller's ref on subreq.\n+ * Allocate and prepare a read subrequest.\n */\n-static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq,\n-\t\t\t\t\t struct netfs_io_subrequest *subreq)\n-{\n-\tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n-\n-\tnetfs_stat(\u0026netfs_n_rh_read);\n-\tcres-\u003eops-\u003eread(cres, subreq-\u003estart, \u0026subreq-\u003eio_iter, NETFS_READ_HOLE_IGNORE,\n-\t\t\tnetfs_cache_read_terminated, subreq);\n-}\n-\n-void netfs_queue_read(struct netfs_io_request *rreq,\n-\t\t struct netfs_io_subrequest *subreq)\n+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq)\n {\n+\tstruct netfs_io_subrequest *subreq;\n \tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n \n+\tsubreq = netfs_alloc_subrequest(rreq);\n+\tif (!subreq)\n+\t\treturn subreq;\n+\n \t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n \n \t/* We add to the end of the list whilst the collector may be walking\n@@ -179,26 +210,53 @@ void netfs_queue_read(struct netfs_io_request *rreq,\n \t}\n \n \tspin_unlock(\u0026rreq-\u003elock);\n+\treturn subreq;\n }\n \n static void netfs_issue_read(struct netfs_io_request *rreq,\n \t\t\t struct netfs_io_subrequest *subreq)\n {\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\tint ret;\n+\n+\t_enter(\"R=%08x[%x]\", rreq-\u003edebug_id, subreq-\u003edebug_index);\n+\n \tswitch (subreq-\u003esource) {\n \tcase NETFS_DOWNLOAD_FROM_SERVER:\n-\t\trreq-\u003enetfs_ops-\u003eissue_read(subreq);\n-\t\tbreak;\n-\tcase NETFS_READ_FROM_CACHE:\n-\t\tnetfs_read_cache_to_pagecache(rreq, subreq);\n-\t\tbreak;\n+\t\tret = rreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\t\tif (ret \u003c 0)\n+\t\t\tgoto fail;\n+\t\treturn;\n+\tcase NETFS_READ_FROM_CACHE: {\n+\t\tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n+\n+\t\tnetfs_stat(\u0026netfs_n_rh_read);\n+\t\tret = cres-\u003eops-\u003eissue_read(subreq);\n+\t\tif (ret \u003c 0)\n+\t\t\tgoto fail;\n+\t\treturn;\n+\t}\n \tdefault:\n-\t\t__set_bit(NETFS_SREQ_CLEAR_TAIL, \u0026subreq-\u003eflags);\n-\t\tsubreq-\u003eerror = 0;\n-\t\tiov_iter_zero(subreq-\u003elen, \u0026subreq-\u003eio_iter);\n+\t\tWARN_ON_ONCE(1);\n+\t\tfallthrough;\n+\tcase NETFS_FILL_WITH_ZEROES:\n+\t\tstream-\u003eissue_from = subreq-\u003estart + subreq-\u003elen;\n+\t\tstream-\u003ebuffered -= subreq-\u003elen;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n+\t\tif (!stream-\u003ebuffered)\n+\t\t\tnetfs_all_subreqs_queued(rreq);\n+\t\tbvecq_zero(\u0026stream-\u003edispatch_cursor, subreq-\u003elen);\n \t\tsubreq-\u003etransferred = subreq-\u003elen;\n-\t\tnetfs_read_subreq_terminated(subreq);\n-\t\tbreak;\n+\t\tsubreq-\u003eerror = 0;\n+\t\treturn netfs_read_subreq_terminated(subreq);\n \t}\n+\n+fail:\n+\t/* Ownership of subreq was returned to us. */\n+\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\tstream-\u003ebuffered -= subreq-\u003elen;\n+\tsubreq-\u003eerror = ret;\n+\tnetfs_read_subreq_terminated(subreq);\n }\n \n /*\n@@ -206,116 +264,141 @@ static void netfs_issue_read(struct netfs_io_request *rreq,\n * slicing up the region to be read according to available cache blocks and\n * network rsize.\n */\n-static void netfs_read_to_pagecache(struct netfs_io_request *rreq,\n-\t\t\t\t struct readahead_control *ractl)\n+static void netfs_read_to_pagecache(struct netfs_io_request *rreq)\n {\n-\tunsigned long long start = rreq-\u003estart;\n-\tssize_t size = rreq-\u003elen;\n+\tstruct fscache_occupancy _occ = {\n+\t\t.query_from\t= rreq-\u003estart,\n+\t\t.query_to\t= rreq-\u003estart + rreq-\u003elen,\n+\t\t.cached_from[0]\t= 0,\n+\t\t.cached_to[0]\t= 0,\n+\t\t.cached_from[1]\t= ULLONG_MAX,\n+\t\t.cached_to[1]\t= ULLONG_MAX,\n+\t};\n+\tstruct fscache_occupancy *occ = \u0026_occ;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n \tint ret = 0;\n \n+\t_enter(\"R=%08x\", rreq-\u003edebug_id);\n+\n+\tbvecq_pos_set(\u0026stream-\u003edispatch_cursor, \u0026rreq-\u003eload_cursor);\n+\tbvecq_pos_set(\u0026rreq-\u003ecollect_cursor, \u0026rreq-\u003eload_cursor);\n+\n \tdo {\n \t\tstruct netfs_io_subrequest *subreq;\n-\t\tenum netfs_io_source source = NETFS_SOURCE_UNKNOWN;\n-\t\tssize_t slice;\n+\t\tunsigned long long hole_to, cache_to, stop;\n+\n+\t\t/* If we don't have any, find out the next couple of data\n+\t\t * extents from the cache, containing of following the\n+\t\t * specified start offset. Holes have to be fetched from the\n+\t\t * server; data regions from the cache.\n+\t\t */\n+\t\thole_to = occ-\u003ecached_from[0];\n+\t\tcache_to = occ-\u003ecached_to[0];\n+\t\tif (stream-\u003eissue_from \u003e= cache_to) {\n+\t\t\t/* Extent exhausted; shuffle down. */\n+\t\t\tint i;\n+\n+\t\t\tfor (i = 0; i \u003c ARRAY_SIZE(occ-\u003ecached_from) - 1; i++) {\n+\t\t\t\tocc-\u003ecached_from[i] = occ-\u003ecached_from[i + 1];\n+\t\t\t\tocc-\u003ecached_to[i] = occ-\u003ecached_to[i + 1];\n+\t\t\t\tocc-\u003ecached_type[i] = occ-\u003ecached_type[i + 1];\n+\t\t\t}\n+\t\t\tocc-\u003ecached_from[i] = ULLONG_MAX;\n+\t\t\tocc-\u003ecached_to[i] = ULLONG_MAX;\n+\n+\t\t\tif (occ-\u003ecached_from[0] != ULLONG_MAX)\n+\t\t\t\tcontinue;\n \n-\t\tsubreq = netfs_alloc_subrequest(rreq);\n+\t\t\t/* Get new extents */\n+\t\t\tnetfs_read_query_cache(rreq, occ);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tsubreq = netfs_alloc_read_subrequest(rreq);\n \t\tif (!subreq) {\n \t\t\tret = -ENOMEM;\n \t\t\tbreak;\n \t\t}\n \n-\t\tsubreq-\u003estart\t= start;\n-\t\tsubreq-\u003elen\t= size;\n-\n-\t\tnetfs_queue_read(rreq, subreq);\n-\n-\t\tsource = netfs_cache_prepare_read(rreq, subreq, rreq-\u003ei_size);\n-\t\tsubreq-\u003esource = source;\n-\t\tif (source == NETFS_DOWNLOAD_FROM_SERVER) {\n-\t\t\tunsigned long long zero_point = netfs_read_zero_point(rreq-\u003einode);\n-\t\t\tunsigned long long zp = umin(zero_point, rreq-\u003ei_size);\n-\t\t\tsize_t len = subreq-\u003elen;\n-\n-\t\t\tif (unlikely(rreq-\u003eorigin == NETFS_READ_SINGLE))\n-\t\t\t\tzp = rreq-\u003ei_size;\n-\t\t\tif (subreq-\u003estart \u003e= zp) {\n-\t\t\t\tsubreq-\u003esource = source = NETFS_FILL_WITH_ZEROES;\n-\t\t\t\tgoto fill_with_zeroes;\n-\t\t\t}\n-\n-\t\t\tif (len \u003e zp - subreq-\u003estart)\n-\t\t\t\tlen = zp - subreq-\u003estart;\n-\t\t\tif (len == 0) {\n-\t\t\t\tpr_err(\"ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx\",\n-\t\t\t\t rreq-\u003edebug_id, subreq-\u003edebug_index,\n-\t\t\t\t subreq-\u003elen, size,\n-\t\t\t\t subreq-\u003estart, zero_point, rreq-\u003ei_size);\n-\t\t\t\tnetfs_cancel_read(subreq, ret);\n-\t\t\t\tbreak;\n+\t\tsubreq-\u003estart = stream-\u003eissue_from;\n+\t\tstop = stream-\u003eissue_from + stream-\u003ebuffered;\n+\n+\t\tunsigned long long zero_point = netfs_read_zero_point(rreq-\u003einode);\n+\t\tunsigned long long zlimit = umin(zero_point, rreq-\u003ei_size);\n+\n+\t\t_debug(\"rsub %llx %llx-%llx\", subreq-\u003estart, hole_to, cache_to);\n+\n+\t\tif (stream-\u003eissue_from \u003e= hole_to \u0026\u0026 stream-\u003eissue_from \u003c cache_to) {\n+\t\t\t/* Overlap with a cached region, where the cache may\n+\t\t\t * record a block of zeroes.\n+\t\t\t */\n+\t\t\t_debug(\"cached s=%llx c=%llx l=%zx\",\n+\t\t\t stream-\u003eissue_from, cache_to, stream-\u003ebuffered);\n+\t\t\tsubreq-\u003elen = umin(cache_to - stream-\u003eissue_from, stream-\u003ebuffered);\n+\t\t\tif (occ-\u003ecached_type[0] == FSCACHE_EXTENT_ZERO) {\n+\t\t\t\tsubreq-\u003esource = NETFS_FILL_WITH_ZEROES;\n+\t\t\t\tnetfs_stat(\u0026netfs_n_rh_zero);\n+\t\t\t} else {\n+\t\t\t\tsubreq-\u003elen = round_up(subreq-\u003elen, occ-\u003egranularity);\n+\t\t\t\tsubreq-\u003esource = NETFS_READ_FROM_CACHE;\n \t\t\t}\n-\t\t\tsubreq-\u003elen = len;\n-\n-\t\t\tnetfs_stat(\u0026netfs_n_rh_download);\n-\t\t\tif (rreq-\u003enetfs_ops-\u003eprepare_read) {\n-\t\t\t\tret = rreq-\u003enetfs_ops-\u003eprepare_read(subreq);\n-\t\t\t\tif (ret \u003c 0) {\n-\t\t\t\t\tnetfs_cancel_read(subreq, ret);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\t\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n-\t\t\t}\n-\t\t\tgoto issue;\n-\t\t}\n-\n-\tfill_with_zeroes:\n-\t\tif (source == NETFS_FILL_WITH_ZEROES) {\n+\t\t} else if (subreq-\u003estart \u003e= zlimit \u0026\u0026\n+\t\t\t subreq-\u003estart \u003c stop) {\n+\t\t\t/* If this range lies beyond the zero-point, that part\n+\t\t\t * can just be cleared locally.\n+\t\t\t */\n+\t\t\t_debug(\"zero %llx-%llx\", subreq-\u003estart, stop);\n+\t\t\tsubreq-\u003elen = stream-\u003ebuffered;\n \t\t\tsubreq-\u003esource = NETFS_FILL_WITH_ZEROES;\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n+\t\t\tif (rreq-\u003ecache_resources.ops)\n+\t\t\t\t__set_bit(NETFS_SREQ_COPY_TO_CACHE, \u0026subreq-\u003eflags);\n \t\t\tnetfs_stat(\u0026netfs_n_rh_zero);\n-\t\t\tgoto issue;\n-\t\t}\n-\n-\t\tif (source == NETFS_READ_FROM_CACHE) {\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n-\t\t\tgoto issue;\n+\t\t} else {\n+\t\t\t/* Read a cache hole from the server. If any part of\n+\t\t\t * this range lies beyond the zero-point or the EOF,\n+\t\t\t * that part can just be cleared locally.\n+\t\t\t */\n+\t\t\tunsigned long long limit = min3(zlimit, stop, hole_to);\n+\n+\t\t\t_debug(\"limit %llx %llx\", rreq-\u003ei_size, zero_point);\n+\t\t\t_debug(\"download %llx-%llx\", subreq-\u003estart, stop);\n+\t\t\tsubreq-\u003elen = umin(limit - subreq-\u003estart, ULONG_MAX);\n+\t\t\tsubreq-\u003esource = NETFS_DOWNLOAD_FROM_SERVER;\n+\t\t\tif (rreq-\u003ecache_resources.ops)\n+\t\t\t\t__set_bit(NETFS_SREQ_COPY_TO_CACHE, \u0026subreq-\u003eflags);\n+\t\t\tnetfs_stat(\u0026netfs_n_rh_download);\n \t\t}\n \n-\t\tpr_err(\"Unexpected read source %u\\n\", source);\n-\t\tWARN_ON_ONCE(1);\n-\t\tnetfs_cancel_read(subreq, ret);\n-\t\tbreak;\n-\n-\tissue:\n-\t\tslice = netfs_prepare_read_iterator(subreq, ractl);\n-\t\tif (slice \u003c 0) {\n-\t\t\tret = slice;\n+\t\tif (subreq-\u003elen == 0) {\n+\t\t\tpr_err(\"ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx\",\n+\t\t\t rreq-\u003edebug_id, subreq-\u003edebug_index,\n+\t\t\t subreq-\u003elen, stream-\u003ebuffered,\n+\t\t\t subreq-\u003estart, zero_point, rreq-\u003ei_size);\n \t\t\tnetfs_cancel_read(subreq, ret);\n \t\t\tbreak;\n \t\t}\n-\t\tstart += slice;\n-\t\tsize -= slice;\n-\t\tif (size \u003c= 0) {\n-\t\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n-\t\t}\n \n \t\tnetfs_issue_read(rreq, subreq);\n+\t\tnetfs_maybe_bulk_drop_ra_refs(rreq);\n \n \t\tif (test_bit(NETFS_RREQ_PAUSE, \u0026rreq-\u003eflags))\n \t\t\tnetfs_wait_for_paused_read(rreq);\n \t\tif (test_bit(NETFS_RREQ_FAILED, \u0026rreq-\u003eflags))\n \t\t\tbreak;\n \t\tcond_resched();\n-\t} while (size \u003e 0);\n+\t} while (stream-\u003ebuffered \u003e 0);\n \n-\tif (unlikely(size \u003e 0)) {\n-\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n+\tif (unlikely(!netfs_are_all_subreqs_queued(rreq))) {\n+\t\tnetfs_all_subreqs_queued(rreq);\n \t\tnetfs_wake_collector(rreq);\n \t}\n \n \t/* Defer error return as we may need to wait for outstanding I/O. */\n-\tcmpxchg(\u0026rreq-\u003eerror, 0, ret);\n+\tif (ret \u003c 0)\n+\t\tcmpxchg(\u0026rreq-\u003eerror, 0, ret);\n+\n+\tbvecq_pos_unset(\u0026rreq-\u003eload_cursor);\n+\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n }\n \n /**\n@@ -336,16 +419,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq,\n void netfs_readahead(struct readahead_control *ractl)\n {\n \tstruct netfs_io_request *rreq;\n+\tstruct netfs_io_stream *stream;\n \tstruct netfs_inode *ictx = netfs_inode(ractl-\u003emapping-\u003ehost);\n \tunsigned long long start = readahead_pos(ractl);\n+\tssize_t added;\n \tsize_t size = readahead_length(ractl);\n \tint ret;\n \n+\t_enter(\"\");\n+\n \trreq = netfs_alloc_request(ractl-\u003emapping, ractl-\u003efile, start, size,\n \t\t\t\t NETFS_READAHEAD);\n \tif (IS_ERR(rreq))\n \t\treturn;\n \n+\tstream = \u0026rreq-\u003eio_streams[0];\n+\n \t__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, \u0026rreq-\u003eflags);\n \n \tret = netfs_begin_cache_read(rreq, ictx);\n@@ -358,11 +447,25 @@ void netfs_readahead(struct readahead_control *ractl)\n \n \tnetfs_rreq_expand(rreq, ractl);\n \n-\trreq-\u003esubmitted = rreq-\u003estart;\n-\tif (rolling_buffer_init(\u0026rreq-\u003ebuffer, rreq-\u003edebug_id, ITER_DEST) \u003c 0)\n+\t/* Load the folios to be read into a bvecq chain. Note that this\n+\t * acquires a ref on each folio that we will need to release later -\n+\t * but we don't want to do that until after we've started the I/O.\n+\t */\n+\tadded = bvecq_load_from_ra(\u0026rreq-\u003eload_cursor, ractl);\n+\tif (added \u003c 0) {\n+\t\tret = added;\n \t\tgoto cleanup_free;\n-\tnetfs_read_to_pagecache(rreq, ractl);\n+\t}\n+\t__set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, \u0026rreq-\u003eflags);\n+\n+\trreq-\u003esubmitted = rreq-\u003estart + added;\n+\trreq-\u003ecleaned_to = rreq-\u003estart;\n+\trreq-\u003efront_folio_order = get_order(rreq-\u003eload_cursor.bvecq-\u003ebv[0].bv_len);\n+\tstream-\u003eissue_from = rreq-\u003estart;\n+\tstream-\u003ebuffered = added;\n \n+\tnetfs_read_to_pagecache(rreq);\n+\tnetfs_maybe_bulk_drop_ra_refs(rreq);\n \treturn netfs_put_request(rreq, netfs_rreq_trace_put_return);\n \n cleanup_free:\n@@ -371,20 +474,23 @@ void netfs_readahead(struct readahead_control *ractl)\n EXPORT_SYMBOL(netfs_readahead);\n \n /*\n- * Create a rolling buffer with a single occupying folio.\n+ * Create a buffer queue with a single occupying folio.\n */\n-static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio,\n-\t\t\t\t\tunsigned int rollbuf_flags)\n+static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)\n {\n-\tssize_t added;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\tstruct bvecq *bq;\n+\tsize_t fsize = folio_size(folio);\n \n-\tif (rolling_buffer_init(\u0026rreq-\u003ebuffer, rreq-\u003edebug_id, ITER_DEST) \u003c 0)\n+\tif (bvecq_buffer_init(\u0026rreq-\u003eload_cursor, GFP_KERNEL) \u003c 0)\n \t\treturn -ENOMEM;\n \n-\tadded = rolling_buffer_append(\u0026rreq-\u003ebuffer, folio, rollbuf_flags);\n-\tif (added \u003c 0)\n-\t\treturn added;\n-\trreq-\u003esubmitted = rreq-\u003estart + added;\n+\tbq = rreq-\u003eload_cursor.bvecq;\n+\tbvec_set_folio(\u0026bq-\u003ebv[0], folio, fsize, 0);\n+\tbvecq_filled_to(bq, 1);\n+\trreq-\u003esubmitted = rreq-\u003estart + fsize;\n+\tstream-\u003eissue_from = rreq-\u003estart;\n+\tstream-\u003ebuffered = fsize;\n \treturn 0;\n }\n \n@@ -394,15 +500,16 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo\n static int netfs_read_gaps(struct file *file, struct folio *folio)\n {\n \tstruct netfs_io_request *rreq;\n+\tstruct netfs_io_stream *stream;\n \tstruct address_space *mapping = folio-\u003emapping;\n \tstruct netfs_group *group = netfs_folio_group(folio);\n \tstruct netfs_folio *finfo = netfs_folio_info(folio);\n \tstruct netfs_inode *ctx = netfs_inode(mapping-\u003ehost);\n-\tstruct folio *sink = NULL;\n-\tstruct bio_vec *bvec;\n+\tstruct bvecq *bq = NULL;\n+\tstruct page *sink = NULL;\n \tunsigned int from = finfo-\u003edirty_offset;\n \tunsigned int to = from + finfo-\u003edirty_len;\n-\tunsigned int off = 0, i = 0;\n+\tunsigned int off = 0, slot = 0;\n \tsize_t flen = folio_size(folio);\n \tsize_t nr_bvec = flen / PAGE_SIZE + 2;\n \tsize_t part;\n@@ -415,6 +522,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)\n \t\tret = PTR_ERR(rreq);\n \t\tgoto alloc_error;\n \t}\n+\tstream = \u0026rreq-\u003eio_streams[0];\n \n \tret = netfs_begin_cache_read(rreq, ctx);\n \tif (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)\n@@ -427,35 +535,52 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)\n \t * end get copied to, but the middle is discarded.\n \t */\n \tret = -ENOMEM;\n-\tbvec = kmalloc_objs(*bvec, nr_bvec);\n-\tif (!bvec)\n+\tbq = bvecq_alloc_chain(nr_bvec, GFP_KERNEL);\n+\tif (!bq)\n \t\tgoto discard;\n+\trreq-\u003eload_cursor.bvecq = bq;\n \n-\tsink = folio_alloc(GFP_KERNEL, 0);\n-\tif (!sink) {\n-\t\tkfree(bvec);\n+\tsink = alloc_page(GFP_KERNEL);\n+\tif (!sink)\n \t\tgoto discard;\n-\t}\n \n \ttrace_netfs_folio(folio, netfs_folio_trace_read_gaps);\n \n-\trreq-\u003edirect_bv = bvec;\n-\trreq-\u003edirect_bv_count = nr_bvec;\n+\tfor (struct bvecq *p = bq; p; p = p-\u003enext)\n+\t\tp-\u003emem_type = BVECQ_MEM_PAGECACHE;\n+\n \tif (from \u003e 0) {\n-\t\tbvec_set_folio(\u0026bvec[i++], folio, from, 0);\n+\t\tfolio_get(folio);\n+\t\tbvec_set_folio(\u0026bq-\u003ebv[slot++], folio, from, 0);\n \t\toff = from;\n \t}\n \twhile (off \u003c to) {\n-\t\tpart = min_t(size_t, to - off, PAGE_SIZE);\n-\t\tbvec_set_folio(\u0026bvec[i++], sink, part, 0);\n+\t\tif (slot \u003e= bq-\u003emax_slots) {\n+\t\t\tbvecq_filled_to(bq, slot);\n+\t\t\tbq = bq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t}\n+\t\tpart = min(to - off, PAGE_SIZE);\n+\t\tget_page(sink);\n+\t\tbvec_set_page(\u0026bq-\u003ebv[slot++], sink, part, 0);\n \t\toff += part;\n \t}\n-\tif (to \u003c flen)\n-\t\tbvec_set_folio(\u0026bvec[i++], folio, flen - to, to);\n-\tiov_iter_bvec(\u0026rreq-\u003ebuffer.iter, ITER_DEST, bvec, i, rreq-\u003elen);\n+\tif (to \u003c flen) {\n+\t\tif (slot \u003e= bq-\u003emax_slots) {\n+\t\t\tbvecq_filled_to(bq, slot);\n+\t\t\tbq = bq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t}\n+\t\tfolio_get(folio);\n+\t\tbvec_set_folio(\u0026bq-\u003ebv[slot++], folio, flen - to, to);\n+\t}\n+\tbvecq_filled_to(bq, slot);\n+\n \trreq-\u003esubmitted = rreq-\u003estart + flen;\n+\tstream-\u003eissue_from = rreq-\u003estart;\n+\tstream-\u003ebuffered = flen;\n \n-\tnetfs_read_to_pagecache(rreq, NULL);\n+\tnetfs_read_to_pagecache(rreq);\n \n \tret = netfs_wait_for_read(rreq);\n \tif (ret \u003e= 0) {\n@@ -469,13 +594,14 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)\n \t\tfolio_mark_uptodate(folio);\n \t}\n \n-\tif (sink)\n-\t\tfolio_put(sink);\n+\tput_page(sink);\n \tfolio_unlock(folio);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_return);\n \treturn ret \u003c 0 ? ret : 0;\n \n discard:\n+\tif (sink)\n+\t\tput_page(sink);\n \tnetfs_put_failed_request(rreq);\n alloc_error:\n \tfolio_unlock(folio);\n@@ -526,11 +652,12 @@ int netfs_read_folio(struct file *file, struct folio *folio)\n \ttrace_netfs_read(rreq, rreq-\u003estart, rreq-\u003elen, netfs_read_trace_readpage);\n \n \t/* Set up the output buffer */\n-\tret = netfs_create_singular_buffer(rreq, folio, 0);\n+\tret = netfs_create_singular_buffer(rreq, folio);\n \tif (ret \u003c 0)\n \t\tgoto discard;\n \n-\tnetfs_read_to_pagecache(rreq, NULL);\n+\tnetfs_read_to_pagecache(rreq);\n+\n \tret = netfs_wait_for_read(rreq);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_return);\n \treturn ret \u003c 0 ? ret : 0;\n@@ -683,11 +810,11 @@ int netfs_write_begin(struct netfs_inode *ctx,\n \ttrace_netfs_read(rreq, pos, len, netfs_read_trace_write_begin);\n \n \t/* Set up the output buffer */\n-\tret = netfs_create_singular_buffer(rreq, folio, 0);\n+\tret = netfs_create_singular_buffer(rreq, folio);\n \tif (ret \u003c 0)\n \t\tgoto error_put;\n \n-\tnetfs_read_to_pagecache(rreq, NULL);\n+\tnetfs_read_to_pagecache(rreq);\n \tret = netfs_wait_for_read(rreq);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_return);\n \tif (ret \u003c 0)\n@@ -748,11 +875,11 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,\n \ttrace_netfs_read(rreq, start, flen, netfs_read_trace_prefetch_for_write);\n \n \t/* Set up the output buffer */\n-\tret = netfs_create_singular_buffer(rreq, folio, NETFS_ROLLBUF_PAGECACHE_MARK);\n+\tret = netfs_create_singular_buffer(rreq, folio);\n \tif (ret \u003c 0)\n \t\tgoto error_put;\n \n-\tnetfs_read_to_pagecache(rreq, NULL);\n+\tnetfs_read_to_pagecache(rreq);\n \tret = netfs_wait_for_read(rreq);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_return);\n \treturn ret \u003c 0 ? ret : 0;\ndiff --git a/fs/netfs/buffered_write.c b/fs/netfs/buffered_write.c\nindex 2cdb68e6b16fff..8e84ec6d26c621 100644\n--- a/fs/netfs/buffered_write.c\n+++ b/fs/netfs/buffered_write.c\n@@ -54,9 +54,6 @@ void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,\n \ti_size = i_size_read(inode);\n \tif (end \u003e i_size) {\n \t\ti_size_write(inode, end);\n-#if IS_ENABLED(CONFIG_FSCACHE)\n-\t\tfscache_update_cookie(ctx-\u003ecache, NULL, \u0026end);\n-#endif\n \n \t\tgap = SECTOR_SIZE - (i_size \u0026 (SECTOR_SIZE - 1));\n \t\tif (copied \u003e gap) {\n@@ -91,44 +88,14 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,\n \tstruct inode *inode = file_inode(file);\n \tstruct address_space *mapping = inode-\u003ei_mapping;\n \tstruct netfs_inode *ctx = netfs_inode(inode);\n-\tstruct writeback_control wbc = {\n-\t\t.sync_mode\t= WB_SYNC_NONE,\n-\t\t.for_sync\t= true,\n-\t\t.nr_to_write\t= LONG_MAX,\n-\t\t.range_start\t= iocb-\u003eki_pos,\n-\t\t.range_end\t= iocb-\u003eki_pos + iter-\u003ecount,\n-\t};\n-\tstruct netfs_io_request *wreq = NULL;\n-\tstruct folio *folio = NULL, *writethrough = NULL;\n+\tstruct folio *folio = NULL;\n \tunsigned int bdp_flags = (iocb-\u003eki_flags \u0026 IOCB_NOWAIT) ? BDP_ASYNC : 0;\n-\tssize_t written = 0, ret, ret2;\n+\tssize_t written = 0, ret;\n \tloff_t pos = iocb-\u003eki_pos;\n \tsize_t max_chunk = mapping_max_folio_size(mapping);\n \tbool maybe_trouble = false;\n \n-\tif (unlikely(iocb-\u003eki_flags \u0026 (IOCB_DSYNC | IOCB_SYNC))\n-\t ) {\n-\t\twbc_attach_fdatawrite_inode(\u0026wbc, mapping-\u003ehost);\n-\n-\t\tret = filemap_write_and_wait_range(mapping, pos, pos + iter-\u003ecount);\n-\t\tif (ret \u003c 0) {\n-\t\t\twbc_detach_inode(\u0026wbc);\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\twreq = netfs_begin_writethrough(iocb, iter-\u003ecount);\n-\t\tif (IS_ERR(wreq)) {\n-\t\t\twbc_detach_inode(\u0026wbc);\n-\t\t\tret = PTR_ERR(wreq);\n-\t\t\twreq = NULL;\n-\t\t\tgoto out;\n-\t\t}\n-\t\tif (!is_sync_kiocb(iocb))\n-\t\t\twreq-\u003eiocb = iocb;\n-\t\tnetfs_stat(\u0026netfs_n_wh_writethrough);\n-\t} else {\n-\t\tnetfs_stat(\u0026netfs_n_wh_buffered_write);\n-\t}\n+\tnetfs_stat(\u0026netfs_n_wh_buffered_write);\n \n \tdo {\n \t\tenum netfs_folio_trace trace;\n@@ -390,15 +357,8 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,\n \t\tpos += copied;\n \t\twritten += copied;\n \n-\t\tif (likely(!wreq)) {\n-\t\t\tfolio_mark_dirty(folio);\n-\t\t\tfolio_unlock(folio);\n-\t\t} else {\n-\t\t\tnetfs_advance_writethrough(wreq, \u0026wbc, folio, copied,\n-\t\t\t\t\t\t offset + copied == flen,\n-\t\t\t\t\t\t \u0026writethrough);\n-\t\t\t/* Folio unlocked */\n-\t\t}\n+\t\tfolio_mark_dirty(folio);\n+\t\tfolio_unlock(folio);\n \tretry:\n \t\tfolio_put(folio);\n \t\tfolio = NULL;\n@@ -420,15 +380,6 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,\n \t\t\tctx-\u003eops-\u003epost_modify(inode);\n \t}\n \n-\tif (unlikely(wreq)) {\n-\t\tret2 = netfs_end_writethrough(wreq, \u0026wbc, writethrough);\n-\t\twbc_detach_inode(\u0026wbc);\n-\t\tif (ret2 == -EIOCBQUEUED)\n-\t\t\treturn ret2;\n-\t\tif (ret == 0 \u0026\u0026 ret2 \u003c 0)\n-\t\t\tret = ret2;\n-\t}\n-\n \tiocb-\u003eki_pos += written;\n \t_leave(\" = %zd [%zd]\", written, ret);\n \treturn written ? written : ret;\ndiff --git a/fs/netfs/bvecq.c b/fs/netfs/bvecq.c\nnew file mode 100644\nindex 00000000000000..830d5199bab7e1\n--- /dev/null\n+++ b/fs/netfs/bvecq.c\n@@ -0,0 +1,768 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+/* Buffering helpers for bvec queues\n+ *\n+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.\n+ * Written by David Howells (dhowells@redhat.com)\n+ */\n+\n+#include \u003clinux/bvecq.h\u003e\n+#include \"internal.h\"\n+\n+void bvecq_dump(const struct bvecq *bq)\n+{\n+\tint b = 0;\n+\n+\tfor (; bq; bq = bvecq_next(bq), b++) {\n+\t\tint skipz = 0;\n+\n+\t\tpr_notice(\"BQ[%u] %u/%u fp=%llx%s\\n\",\n+\t\t\t b, bq-\u003enr_slots, bq-\u003emax_slots, bq-\u003efpos,\n+\t\t\t bq-\u003ediscontig ? \" discontig\" : \"\");\n+\t\tfor (int s = 0; s \u003c bq-\u003enr_slots; s++) {\n+\t\t\tconst struct bio_vec *bv = \u0026bq-\u003ebv[s];\n+\n+\t\t\tif (!bv-\u003ebv_page \u0026\u0026 !bv-\u003ebv_len \u0026\u0026 skipz \u003c 2) {\n+\t\t\t\tskipz = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (skipz == 1)\n+\t\t\t\tpr_notice(\"BQ[%u:00-%02u] ...\\n\", b, s - 1);\n+\t\t\tskipz = 2;\n+\t\t\tpr_notice(\"BQ[%u:%02u] %10lx %04x %04x %u\\n\",\n+\t\t\t\t b, s,\n+\t\t\t\t bv-\u003ebv_page ? page_to_pfn(bv-\u003ebv_page) : 0,\n+\t\t\t\t bv-\u003ebv_offset, bv-\u003ebv_len,\n+\t\t\t\t bv-\u003ebv_page ? page_count(bv-\u003ebv_page) : 0);\n+\t\t}\n+\t}\n+}\n+EXPORT_SYMBOL(bvecq_dump);\n+\n+/**\n+ * bvecq_alloc_one - Allocate a single bvecq node with unpopulated slots\n+ * @nr_slots: Number of slots to allocate\n+ * @gfp: The allocation constraints.\n+ *\n+ * Allocate a single bvecq node and initialise the header. A number of inline\n+ * slots are also allocated, rounded up to fit after the header in a power-of-2\n+ * slab object of up to 512 bytes (up to 29 slots on a 64-bit cpu). The slot\n+ * array is not initialised.\n+ *\n+ * Return: The node pointer or NULL on allocation failure.\n+ */\n+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp)\n+{\n+\tstruct bvecq *bq;\n+\tconst size_t max_size = 512;\n+\tconst size_t max_slots = (max_size - sizeof(*bq)) / sizeof(bq-\u003e__bv[0]);\n+\tsize_t part = umin(nr_slots, max_slots);\n+\tsize_t size = roundup_pow_of_two(struct_size(bq, __bv, part));\n+\n+\tbq = kmalloc(size, gfp \u0026 ~GFP_ZONEMASK);\n+\tif (bq) {\n+\t\t*bq = (struct bvecq) {\n+\t\t\t.ref\t\t= REFCOUNT_INIT(1),\n+\t\t\t.bv\t\t= bq-\u003e__bv,\n+\t\t\t.inline_bv\t= true,\n+\t\t\t.max_slots\t= (size - sizeof(*bq)) / sizeof(bq-\u003e__bv[0]),\n+\t\t};\n+\t\tnetfs_stat(\u0026netfs_n_bvecq);\n+\t}\n+\treturn bq;\n+}\n+EXPORT_SYMBOL(bvecq_alloc_one);\n+\n+/**\n+ * bvecq_alloc_chain - Allocate an unpopulated bvecq chain\n+ * @nr_slots: Number of slots to allocate\n+ * @gfp: The allocation constraints.\n+ *\n+ * Allocate a chain of bvecq nodes providing at least the requested cumulative\n+ * number of slots.\n+ *\n+ * Return: The first node pointer or NULL on allocation failure.\n+ */\n+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp)\n+{\n+\tstruct bvecq *head = NULL, *tail = NULL;\n+\n+\t_enter(\"%zu\", nr_slots);\n+\n+\tfor (;;) {\n+\t\tstruct bvecq *bq;\n+\n+\t\tbq = bvecq_alloc_one(nr_slots, gfp);\n+\t\tif (!bq)\n+\t\t\tgoto oom;\n+\n+\t\tif (tail)\n+\t\t\tbvecq_append(tail, bq);\n+\t\telse\n+\t\t\thead = bq;\n+\t\ttail = bq;\n+\t\tif (tail-\u003emax_slots \u003e= nr_slots)\n+\t\t\tbreak;\n+\t\tnr_slots -= tail-\u003emax_slots;\n+\t}\n+\n+\treturn head;\n+oom:\n+\tbvecq_put(head);\n+\treturn NULL;\n+}\n+EXPORT_SYMBOL(bvecq_alloc_chain);\n+\n+/**\n+ * bvecq_alloc_buffer2 - Allocate a bvecq chain and populate with buffers\n+ * @size: Target size of the buffer (can be 0 for an empty buffer)\n+ * @pre_slots: Number of preamble slots to set aside\n+ * @gfp: The allocation constraints.\n+ *\n+ * Allocate a chain of bvecq nodes and populate the slots with sufficient pages\n+ * to provide at least the requested amount of space, leaving the first\n+ * @pre_slots slots unset. The pre-slots must all fit into the the first\n+ * bvecq.\n+ *\n+ * The pages allocated may be compound pages larger than PAGE_SIZE and thus\n+ * occupy fewer slots. The pages have their refcounts set to 1 and can be\n+ * passed to MSG_SPLICE_PAGES.\n+ *\n+ * Return: The first node pointer or NULL on allocation failure.\n+ */\n+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp)\n+{\n+\tstruct bvecq *head = NULL, *p = NULL;\n+\tsize_t nr_per_bq = BVECQ_STD_SLOTS;\n+\tsize_t count = pre_slots + DIV_ROUND_UP(size, PAGE_SIZE);\n+\n+\t_enter(\"%zx,%zx,%u\", size, count, pre_slots);\n+\n+\tif (WARN_ON_ONCE(pre_slots \u003e nr_per_bq))\n+\t\treturn NULL;\n+\n+\thead = bvecq_alloc_chain(count, gfp);\n+\tif (!head)\n+\t\treturn NULL;\n+\n+\tp = head;\n+\tdo {\n+\t\tstruct page **pages;\n+\t\tsize_t unused, want, got, slot;\n+\n+\t\tif (!count)\n+\t\t\tbreak;\n+\t\tif (WARN_ON_ONCE(!p))\n+\t\t\tgoto oom;\n+\n+\t\tif (p-\u003enr_slots == 0) {\n+\t\t\t/* Need to clear pre slots and pages[], so just clear all. */\n+\t\t\tmemset(p-\u003ebv, 0, p-\u003emax_slots * sizeof(p-\u003ebv[0]));\n+\t\t\tp-\u003emem_type = BVECQ_MEM_ALLOCED;\n+\t\t\tp-\u003enr_slots = pre_slots;\n+\t\t\tcount -= pre_slots;\n+\t\t\tpre_slots = 0;\n+\t\t}\n+\n+\t\tif (p-\u003enr_slots \u003e= p-\u003emax_slots) {\n+\t\t\tp = p-\u003enext;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tunused = p-\u003emax_slots - p-\u003enr_slots;\n+\n+\t\tpages = (struct page **)\u0026p-\u003ebv[p-\u003emax_slots];\n+\t\tpages -= unused;\n+\n+\t\twant = min(count, unused);\n+\t\tgot = alloc_pages_bulk(gfp, want, pages);\n+\t\tif (!got)\n+\t\t\tgoto oom;\n+\n+\t\tslot = p-\u003enr_slots;\n+\t\tfor (int i = 0; i \u003c got; i++) {\n+\t\t\tset_page_count(pages[i], 1);\n+\t\t\tbvec_set_page(\u0026p-\u003ebv[slot++], pages[i], PAGE_SIZE, 0);\n+\t\t}\n+\n+\t\tbvecq_filled_to(p, slot);\n+\t\tcount -= got;\n+\t} while (count \u003e 0);\n+\n+\treturn head;\n+oom:\n+\tbvecq_put(head);\n+\treturn NULL;\n+}\n+EXPORT_SYMBOL(bvecq_alloc_buffer2);\n+\n+/*\n+ * Free the page pointed to by a slot as necessary.\n+ */\n+static void bvecq_free_slot(struct bvecq *bq, unsigned int slot)\n+{\n+\tstruct page *page = bq-\u003ebv[slot].bv_page;\n+\n+\tif (!page)\n+\t\treturn;\n+\n+\tswitch (bq-\u003emem_type) {\n+\tcase BVECQ_MEM_EXTERNAL:\n+\t\tbreak;\n+\tcase BVECQ_MEM_PAGECACHE:\n+\t\tput_page(page);\n+\t\tbreak;\n+\tcase BVECQ_MEM_GUP:\n+\t\tunpin_user_page(page);\n+\t\tbreak;\n+\tcase BVECQ_MEM_ALLOCED:\n+\t\t__free_pages(page, compound_order(page));\n+\t\tbreak;\n+\tdefault:\n+\t\tWARN_ON_ONCE(1);\n+\t\tbreak;\n+\t}\n+}\n+\n+/**\n+ * bvecq_put - Put a ref on a bvec queue\n+ * @bq: The start of the folio queue to free\n+ *\n+ * Put the ref(s) on the nodes in a bvec queue, freeing up the node and the\n+ * page fragments it points to as the refcounts become zero.\n+ */\n+void bvecq_put(struct bvecq *bq)\n+{\n+\tstruct bvecq *next;\n+\n+\tfor (; bq; bq = next) {\n+\t\tif (!refcount_dec_and_test(\u0026bq-\u003eref))\n+\t\t\tbreak;\n+\t\tfor (int slot = 0; slot \u003c bq-\u003enr_slots; slot++)\n+\t\t\tbvecq_free_slot(bq, slot);\n+\t\tnext = bq-\u003enext;\n+\t\tnetfs_stat_d(\u0026netfs_n_bvecq);\n+\t\tkfree(bq);\n+\t}\n+}\n+EXPORT_SYMBOL(bvecq_put);\n+\n+/**\n+ * bvecq_expand_buffer - Allocate buffer space into a bvec queue\n+ * @_buffer: Pointer to the bvecq chain to expand (may point to a NULL; updated).\n+ * @_cur_size: Current size of the buffer (updated).\n+ * @size: Target size of the buffer.\n+ * @gfp: The allocation constraints.\n+ *\n+ * Append extra pages to a buffer to increase its capacity to the @size\n+ * specified. If the current tail has space, but is not of the\n+ * BVECQ_MEM_ALLOCED memory type, a separate bvecq will be allocated to hold\n+ * the new memory.\n+ */\n+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp)\n+{\n+\tstruct bvecq *tail = *_buffer;\n+\n+\tsize = round_up(size, PAGE_SIZE);\n+\tif (tail)\n+\t\twhile (tail-\u003enext)\n+\t\t\ttail = tail-\u003enext;\n+\n+\twhile (*_cur_size \u003c size) {\n+\t\tstruct page *page;\n+\t\tsize_t need = size - *_cur_size;\n+\t\tint order = 0;\n+\n+\t\tif (!tail || bvecq_is_full(tail) || tail-\u003emem_type != BVECQ_MEM_ALLOCED) {\n+\t\t\tstruct bvecq *p;\n+\n+\t\t\tp = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp);\n+\t\t\tif (!p)\n+\t\t\t\treturn -ENOMEM;\n+\t\t\tif (tail)\n+\t\t\t\tbvecq_append(tail, p);\n+\t\t\telse\n+\t\t\t\t*_buffer = p;\n+\t\t\ttail = p;\n+\t\t\tp-\u003emem_type = BVECQ_MEM_ALLOCED;\n+\t\t}\n+\n+\t\tif (need \u003e PAGE_SIZE)\n+\t\t\torder = umin(ilog2(need) - PAGE_SHIFT, MAX_PAGECACHE_ORDER);\n+\n+\t\tpage = alloc_pages(gfp | __GFP_COMP, order);\n+\t\tif (!page \u0026\u0026 order \u003e 0) {\n+\t\t\tpage = alloc_pages(gfp | __GFP_COMP, 0);\n+\t\t\torder = 0;\n+\t\t}\n+\t\tif (!page)\n+\t\t\treturn -ENOMEM;\n+\n+\t\tbvec_set_page(\u0026tail-\u003ebv[tail-\u003enr_slots], page, PAGE_SIZE \u003c\u003c order, 0);\n+\t\t*_cur_size += PAGE_SIZE \u003c\u003c order;\n+\t\tbvecq_filled_to(tail, tail-\u003enr_slots + 1);\n+\t}\n+\n+\treturn 0;\n+}\n+EXPORT_SYMBOL(bvecq_expand_buffer);\n+\n+/**\n+ * bvecq_shorten_buffer - Shorten a bvec queue buffer\n+ * @bq: The start of the buffer to shorten\n+ * @slot: The slot to start from\n+ * @size: The size to retain\n+ *\n+ * Shorten the content of a bvec queue down to the minimum number of slots,\n+ * starting at the specified slot, to retain the specified size.\n+ *\n+ * Return: 0 if successful; -EMSGSIZE if there is insufficient content.\n+ */\n+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size)\n+{\n+\tstruct bvecq *next;\n+\n+\t/* Skip through the segments we want to keep. */\n+\tfor (; bq; bq = bq-\u003enext) {\n+\t\tfor (; slot \u003c bq-\u003enr_slots; slot++) {\n+\t\t\tif (size \u003c bq-\u003ebv[slot].bv_len)\n+\t\t\t\tgoto found;\n+\t\t\tsize -= bq-\u003ebv[slot].bv_len;\n+\t\t}\n+\t\tslot = 0;\n+\t}\n+\tif (WARN_ON_ONCE(size \u003e 0))\n+\t\treturn -EMSGSIZE;\n+\treturn 0;\n+\n+found:\n+\t/* Shorten any partial entry and clean the rest of this bvecq. */\n+\tif (size \u003e 0) {\n+\t\tbq-\u003ebv[slot].bv_len = size;\n+\t\tslot++;\n+\t}\n+\tfor (int i = slot; i \u003c bq-\u003enr_slots; i++)\n+\t\tbvecq_free_slot(bq, i);\n+\tbq-\u003enr_slots = slot;\n+\n+\t/* Free the queue tail. */\n+\tnext = bq-\u003enext;\n+\tbq-\u003enext = NULL;\n+\tbvecq_put(next);\n+\treturn 0;\n+}\n+EXPORT_SYMBOL(bvecq_shorten_buffer);\n+\n+/**\n+ * bvecq_buffer_init - Initialise a buffer and set position\n+ * @pos: The position to point at the new buffer.\n+ * @gfp: The allocation constraints.\n+ *\n+ * Initialise a rolling buffer. We allocate an unpopulated bvecq node to so\n+ * that the pointers can be independently driven by the producer and the\n+ * consumer.\n+ *\n+ * Return 0 if successful; -ENOMEM on allocation failure.\n+ */\n+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp)\n+{\n+\tstruct bvecq *bq;\n+\n+\tbq = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp);\n+\tif (!bq)\n+\t\treturn -ENOMEM;\n+\n+\tpos-\u003ebvecq = bq; /* Comes with a ref. */\n+\tpos-\u003eslot = 0;\n+\tpos-\u003eoffset = 0;\n+\treturn 0;\n+}\n+\n+/**\n+ * bvecq_buffer_append - Append a new bvecq node to a buffer\n+ * @pos: The position of the last node.\n+ * @bq: The buffer to add.\n+ *\n+ * Add a new node on to the buffer chain at the specified position, either\n+ * because the previous one is full or because we have a discontiguity to\n+ * contend with, and update @pos to point to it.\n+ */\n+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq)\n+{\n+\tstruct bvecq *head = pos-\u003ebvecq;\n+\n+\tpos-\u003ebvecq = bvecq_get(bq);\n+\tpos-\u003eslot = 0;\n+\tpos-\u003eoffset = 0;\n+\n+\t/* [!] NOTE: After we set head-\u003enext, the consumer is at liberty to\n+\t * immediately delete the old head.\n+\t */\n+\tbvecq_append(head, bq);\n+\tbvecq_put(head);\n+}\n+\n+/**\n+ * bvecq_pos_advance - Advance a bvecq position\n+ * @pos: The position to advance.\n+ * @amount: The amount of bytes to advance by.\n+ *\n+ * Advance the specified bvecq position by @amount bytes. @pos is updated and\n+ * bvecq ref counts may have been manipulated. If the position hits the end of\n+ * the queue, then it is left pointing beyond the last slot of the last bvecq\n+ * so that it doesn't break the chain.\n+ */\n+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount)\n+{\n+\tstruct bvecq *bq = pos-\u003ebvecq, *next;\n+\tunsigned int slot = pos-\u003eslot;\n+\tsize_t offset = pos-\u003eoffset;\n+\n+\twhile (amount) {\n+\t\tsize_t part;\n+\n+\t\tif (!bvecq_acquire_slot(bq, slot)) {\n+\t\t\tnext = bvecq_next(bq);\n+\t\t\tif (!next) {\n+\t\t\t\tWARN_ON_ONCE(amount \u003e 0);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbq = next;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tpart = bq-\u003ebv[slot].bv_len - offset;\n+\n+\t\tif (part \u003e amount) {\n+\t\t\toffset += amount;\n+\t\t\tbreak;\n+\t\t}\n+\t\tamount -= part;\n+\t\toffset = 0;\n+\t\tslot++;\n+\t}\n+\n+\tpos-\u003eslot = slot;\n+\tpos-\u003eoffset = offset;\n+\tbvecq_pos_move(pos, bq);\n+}\n+\n+/*\n+ * Clear part of the memory pointed to by a bio_vec.\n+ */\n+static void bvec_zero(const struct bio_vec *bv, size_t offset, size_t len)\n+{\n+\tstruct page *page = bv-\u003ebv_page;\n+\n+\toffset += bv-\u003ebv_offset;\n+\n+\tpage += offset / PAGE_SIZE;\n+\toffset = offset % PAGE_SIZE;\n+\n+\twhile (len) {\n+\t\tsize_t part = umin(len, PAGE_SIZE - offset);\n+\t\tchar *p = kmap_local_page(page);\n+\n+\t\tmemset(p + offset, 0, part);\n+\t\tkunmap_local(p);\n+\n+\t\tlen -= part;\n+\t\toffset = 0;\n+\t\tpage++;\n+\t}\n+}\n+\n+/**\n+ * bvecq_zero - Clear memory starting at the bvecq position.\n+ * @pos: The position in the bvecq chain to start clearing.\n+ * @amount: The number of bytes to clear.\n+ *\n+ * Clear memory fragments pointed to by a bvec queue. @pos is updated and\n+ * bvecq ref counts may have been manipulated. If the position hits the end of\n+ * the queue, then it is left pointing beyond the last slot of the last bvecq\n+ * so that it doesn't break the chain.\n+ *\n+ * Return: The number of bytes cleared.\n+ */\n+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount)\n+{\n+\tstruct bvecq *bq = pos-\u003ebvecq, *next;\n+\tunsigned int slot = pos-\u003eslot;\n+\tssize_t cleared = 0;\n+\tsize_t offset = pos-\u003eoffset;\n+\n+\twhile (amount) {\n+\t\tconst struct bio_vec *bv;\n+\t\tsize_t part;\n+\n+\t\tif (!bvecq_acquire_slot(bq, slot)) {\n+\t\t\tnext = bvecq_next(bq);\n+\t\t\tif (!next) {\n+\t\t\t\tWARN_ON_ONCE(amount \u003e 0);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbq = next;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbv = \u0026bq-\u003ebv[slot];\n+\t\tif (offset \u003e= bv-\u003ebv_len) {\n+\t\t\tslot++;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tpart = min(bv-\u003ebv_len - offset, amount);\n+\t\tbvec_zero(bv, offset, part);\n+\t\tcleared += part;\n+\t\toffset += part;\n+\t\tamount -= part;\n+\t}\n+\n+\tpos-\u003eslot = slot;\n+\tpos-\u003eoffset = offset;\n+\tbvecq_pos_move(pos, bq);\n+\treturn cleared;\n+}\n+\n+/**\n+ * bvecq_slice - Find a slice of a bvecq queue\n+ * @pos: The position to start at.\n+ * @max_size: The maximum size of the slice (or ULONG_MAX).\n+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).\n+ * @_nr_slots: Where to put the number of slots (updated).\n+ *\n+ * Determine the size and number of slots that can be obtained the next slice\n+ * of bvec queue up to the maximum size and slot count specified. The slice is\n+ * also limited if a discontiguity is found.\n+ *\n+ * @pos is updated to the end of the slice. If the position hits the end of\n+ * the queue, then it is left pointing beyond the last slot of the last bvecq\n+ * so that it doesn't break the chain.\n+ *\n+ * Return: The number of bytes in the slice.\n+ */\n+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,\n+\t\t unsigned int max_slots, unsigned int *_nr_slots)\n+{\n+\tstruct bvecq *bq, *next;\n+\tunsigned int slot = pos-\u003eslot, nslots = 0;\n+\tsize_t size = 0, offset = pos-\u003eoffset;\n+\n+\tbq = pos-\u003ebvecq;\n+\tfor (;;) {\n+\t\tfor (; slot \u003c bvecq_nr_slots_acquire(bq); slot++) {\n+\t\t\tconst struct bio_vec *bvec = \u0026bq-\u003ebv[slot];\n+\n+\t\t\tif (offset \u003c bvec-\u003ebv_len \u0026\u0026 bvec-\u003ebv_page) {\n+\t\t\t\tsize_t part = umin(bvec-\u003ebv_len - offset, max_size);\n+\n+\t\t\t\tsize += part;\n+\t\t\t\toffset += part;\n+\t\t\t\tmax_size -= part;\n+\t\t\t\tnslots++;\n+\t\t\t\tif (!max_size || nslots \u003e= max_slots)\n+\t\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\toffset = 0;\n+\t\t}\n+\n+\t\t/* pos-\u003ebvecq isn't allowed to go NULL as the queue may get\n+\t\t * extended and we would lose our place.\n+\t\t */\n+\t\tnext = bvecq_next(bq);\n+\t\tif (!next)\n+\t\t\tbreak;\n+\t\tslot = 0;\n+\t\tbq = next;\n+\t\tif (bq-\u003ediscontig \u0026\u0026 size \u003e 0)\n+\t\t\tbreak;\n+\t}\n+\n+out:\n+\t*_nr_slots = nslots;\n+\tif (slot == bvecq_nr_slots_acquire(bq)) {\n+\t\tnext = bvecq_next(bq);\n+\t\tif (next) {\n+\t\t\tbq = next;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t}\n+\t}\n+\tbvecq_pos_move(pos, bq);\n+\tpos-\u003eslot = slot;\n+\tpos-\u003eoffset = offset;\n+\treturn size;\n+}\n+\n+/**\n+ * bvecq_extract - Extract a slice of a bvecq queue into a new bvecq queue\n+ * @pos: The position to start at.\n+ * @max_size: The maximum size of the slice (or ULONG_MAX).\n+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).\n+ * @to: Where to put the extraction bvecq chain head (updated).\n+ *\n+ * Allocate a new bvecq and extract into it memory fragments from a slice of\n+ * bvec queue, starting at @pos. The slice is also limited if a discontiguity\n+ * is found. No refs are taken on the page.\n+ *\n+ * @pos is updated to the end of the slice. If the position hits the end of\n+ * the queue, then it is left pointing beyond the last slot of the last bvecq\n+ * so that it doesn't break the chain.\n+ *\n+ * If successful, *@to is set to point to the head of the newly allocated chain\n+ * and the caller inherits a ref to it.\n+ *\n+ * Return: The number of bytes extracted; -ENOMEM on allocation failure or -EIO\n+ * if no slots were available to extract.\n+ */\n+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size,\n+\t\t unsigned int max_slots, struct bvecq **to)\n+{\n+\tstruct bvecq_pos tmp_pos;\n+\tstruct bvecq *src, *dst = NULL, *next;\n+\tunsigned int slot = pos-\u003eslot, dslot = 0, nslots;\n+\tssize_t extracted = 0;\n+\tsize_t offset = pos-\u003eoffset, amount;\n+\n+\t*to = NULL;\n+\tif (WARN_ON_ONCE(!max_slots))\n+\t\tmax_slots = INT_MAX;\n+\n+\tbvecq_pos_set(\u0026tmp_pos, pos);\n+\tamount = bvecq_slice(\u0026tmp_pos, max_size, max_slots, \u0026nslots);\n+\tbvecq_pos_unset(\u0026tmp_pos);\n+\tif (nslots == 0)\n+\t\treturn -EIO;\n+\n+\tdst = bvecq_alloc_chain(nslots, GFP_NOFS);\n+\tif (!dst)\n+\t\treturn -ENOMEM;\n+\t*to = dst;\n+\tmax_slots = nslots;\n+\tnslots = 0;\n+\n+\t/* Transcribe the slots */\n+\tsrc = pos-\u003ebvecq;\n+\tfor (;;) {\n+\t\tfor (; slot \u003c bvecq_nr_slots_acquire(src); slot++) {\n+\t\t\tconst struct bio_vec *sv = \u0026src-\u003ebv[slot];\n+\t\t\tstruct bio_vec *dv = \u0026dst-\u003ebv[dslot];\n+\n+\t\t\t_debug(\"EXTR BQ=%x[%x] off=%zx am=%zx p=%lx\",\n+\t\t\t src-\u003epriv, slot, offset, amount, page_to_pfn(sv-\u003ebv_page));\n+\n+\t\t\tif (offset \u003c sv-\u003ebv_len \u0026\u0026 sv-\u003ebv_page) {\n+\t\t\t\tsize_t part = umin(sv-\u003ebv_len - offset, amount);\n+\n+\t\t\t\tbvec_set_page(dv, sv-\u003ebv_page, part,\n+\t\t\t\t\t sv-\u003ebv_offset + offset);\n+\t\t\t\textracted += part;\n+\t\t\t\tamount -= part;\n+\t\t\t\toffset += part;\n+\t\t\t\ttrace_netfs_bv_slot(dst, dslot);\n+\t\t\t\tdslot++;\n+\t\t\t\tnslots++;\n+\t\t\t\tif (dslot \u003e= dst-\u003emax_slots) {\n+\t\t\t\t\tbvecq_filled_to(dst, dslot);\n+\t\t\t\t\tdst = dst-\u003enext;\n+\t\t\t\t\tdslot = 0;\n+\t\t\t\t}\n+\t\t\t\tif (nslots \u003e= max_slots)\n+\t\t\t\t\tgoto out;\n+\t\t\t\tif (amount == 0)\n+\t\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\toffset = 0;\n+\t\t}\n+\n+\t\t/* pos-\u003ebvecq isn't allowed to go NULL as the queue may get\n+\t\t * extended and we would lose our place.\n+\t\t */\n+\t\tnext = bvecq_next(src);\n+\t\tif (!next)\n+\t\t\tbreak;\n+\t\tslot = 0;\n+\t\tsrc = next;\n+\t\tif (src-\u003ediscontig \u0026\u0026 extracted \u003e 0)\n+\t\t\tbreak;\n+\t}\n+\n+out:\n+\tif (dst)\n+\t\tbvecq_filled_to(dst, dslot);\n+\tif (slot == bvecq_nr_slots_acquire(src)) {\n+\t\tnext = bvecq_next(src);\n+\t\tif (next) {\n+\t\t\tsrc = next;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t}\n+\t}\n+\tbvecq_pos_move(pos, src);\n+\tpos-\u003eslot = slot;\n+\tpos-\u003eoffset = offset;\n+\treturn extracted;\n+}\n+\n+/**\n+ * bvecq_load_from_ra - Allocate a bvecq chain and load from readahead\n+ * @pos: Blank position object to attach the new chain to.\n+ * @ractl: The readahead control context.\n+ *\n+ * Decant the set of folios to be read from the readahead context into a bvecq\n+ * chain. Each folio occupies one bio_vec element.\n+ *\n+ * Return: Amount of data loaded or -ENOMEM on allocation failure.\n+ */\n+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl)\n+{\n+\tXA_STATE(xas, \u0026ractl-\u003emapping-\u003ei_pages, ractl-\u003e_index);\n+\tstruct folio *folio;\n+\tstruct bvecq *bq;\n+\tunsigned int slot = 0;\n+\tsize_t loaded = 0;\n+\n+\tbq = bvecq_alloc_chain(ractl-\u003e_nr_folios, GFP_NOFS);\n+\tif (!bq)\n+\t\treturn -ENOMEM;\n+\n+\tpos-\u003ebvecq = bq;\n+\tpos-\u003eslot = 0;\n+\tpos-\u003eoffset = 0;\n+\n+\trcu_read_lock();\n+\n+\txas_for_each(\u0026xas, folio, ractl-\u003e_index + ractl-\u003e_nr_pages - 1) {\n+\t\tsize_t len;\n+\n+\t\tif (xas_retry(\u0026xas, folio))\n+\t\t\tcontinue;\n+\t\tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n+\n+\t\tlen = folio_size(folio);\n+\t\tbvec_set_folio(\u0026bq-\u003ebv[slot], folio, len, 0);\n+\t\tloaded += len;\n+\t\tslot++;\n+\t\ttrace_netfs_folio(folio, netfs_folio_trace_read);\n+\n+\t\tif (slot \u003e= bq-\u003emax_slots) {\n+\t\t\tbvecq_filled_to(bq, slot);\n+\t\t\tbq = bq-\u003enext;\n+\t\t\tif (!bq)\n+\t\t\t\tbreak;\n+\t\t\tslot = 0;\n+\t\t}\n+\t}\n+\n+\trcu_read_unlock();\n+\n+\tif (bq)\n+\t\tbvecq_filled_to(bq, slot);\n+\n+\tractl-\u003e_index += ractl-\u003e_nr_pages;\n+\tractl-\u003e_nr_pages = 0;\n+\treturn loaded;\n+}\ndiff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c\nindex 6a8fb0d55e040e..743a2921cdd9ff 100644\n--- a/fs/netfs/direct_read.c\n+++ b/fs/netfs/direct_read.c\n@@ -16,29 +16,30 @@\n #include \u003clinux/netfs.h\u003e\n #include \"internal.h\"\n \n-static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)\n+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n-\tsize_t rsize;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\tsize_t len;\n \n-\trsize = umin(subreq-\u003elen, rreq-\u003eio_streams[0].sreq_max_len);\n-\tsubreq-\u003elen = rsize;\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026stream-\u003edispatch_cursor);\n+\tlen = bvecq_slice(\u0026stream-\u003edispatch_cursor, subreq-\u003elen, max_segs,\n+\t\t\t \u0026subreq-\u003enr_segs);\n \n-\tif (unlikely(rreq-\u003eio_streams[0].sreq_max_segs)) {\n-\t\tsize_t limit = netfs_limit_iter(\u0026rreq-\u003ebuffer.iter, 0, rsize,\n-\t\t\t\t\t\trreq-\u003eio_streams[0].sreq_max_segs);\n-\n-\t\tif (limit \u003c rsize) {\n-\t\t\tsubreq-\u003elen = limit;\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n-\t\t}\n+\tif (len \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = len;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n \t}\n \n-\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n+\tstream-\u003ebuffered -= subreq-\u003elen;\n+\tstream-\u003eissue_from += subreq-\u003elen;\n+\trreq-\u003esubmitted = stream-\u003eissue_from;\n \n-\tsubreq-\u003eio_iter\t= rreq-\u003ebuffer.iter;\n-\tiov_iter_truncate(\u0026subreq-\u003eio_iter, subreq-\u003elen);\n-\tiov_iter_advance(\u0026rreq-\u003ebuffer.iter, subreq-\u003elen);\n+\tif (stream-\u003ebuffered == 0)\n+\t\tnetfs_all_subreqs_queued(rreq);\n+\treturn 0;\n }\n \n /*\n@@ -47,15 +48,16 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)\n */\n static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)\n {\n-\tunsigned long long start = rreq-\u003estart;\n-\tssize_t size = rreq-\u003elen;\n-\tint ret;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\n+\tbvecq_pos_set(\u0026rreq-\u003ecollect_cursor, \u0026rreq-\u003eload_cursor);\n+\tbvecq_pos_transfer(\u0026stream-\u003edispatch_cursor, \u0026rreq-\u003eload_cursor);\n \n \tdo {\n \t\tstruct netfs_io_subrequest *subreq;\n-\t\tssize_t slice;\n+\t\tint ret;\n \n-\t\tsubreq = netfs_alloc_subrequest(rreq);\n+\t\tsubreq = netfs_alloc_read_subrequest(rreq);\n \t\tif (!subreq) {\n \t\t\t/* Stash the error in the request if there's not\n \t\t\t * already an error set.\n@@ -65,44 +67,32 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)\n \t\t}\n \n \t\tsubreq-\u003esource\t= NETFS_DOWNLOAD_FROM_SERVER;\n-\t\tsubreq-\u003estart\t= start;\n-\t\tsubreq-\u003elen\t= size;\n-\n-\t\tnetfs_queue_read(rreq, subreq);\n+\t\tsubreq-\u003estart\t= stream-\u003eissue_from;\n+\t\tsubreq-\u003elen\t= stream-\u003ebuffered;\n \n \t\tnetfs_stat(\u0026netfs_n_rh_download);\n-\t\tif (rreq-\u003enetfs_ops-\u003eprepare_read) {\n-\t\t\tret = rreq-\u003enetfs_ops-\u003eprepare_read(subreq);\n-\t\t\tif (ret \u003c 0) {\n-\t\t\t\tnetfs_cancel_read(subreq, ret);\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n \n-\t\tnetfs_prepare_dio_read_iterator(subreq);\n-\t\tslice = subreq-\u003elen;\n-\t\tsize -= slice;\n-\t\tstart += slice;\n-\t\trreq-\u003esubmitted += slice;\n-\t\tif (size \u003c= 0) {\n-\t\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n+\t\tret = rreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\tstream-\u003ebuffered -= subreq-\u003elen;\n+\t\t\tsubreq-\u003eerror = ret;\n+\t\t\tnetfs_read_subreq_terminated(subreq);\n+\t\t\tbreak;\n \t\t}\n \n-\t\trreq-\u003enetfs_ops-\u003eissue_read(subreq);\n-\n \t\tif (test_bit(NETFS_RREQ_PAUSE, \u0026rreq-\u003eflags))\n \t\t\tnetfs_wait_for_paused_read(rreq);\n \t\tif (test_bit(NETFS_RREQ_FAILED, \u0026rreq-\u003eflags))\n \t\t\tbreak;\n \t\tcond_resched();\n-\t} while (size \u003e 0);\n+\t} while (stream-\u003ebuffered \u003e 0);\n \n-\tif (unlikely(size \u003e 0)) {\n-\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n+\tif (unlikely(stream-\u003ebuffered \u003e 0)) {\n+\t\tnetfs_all_subreqs_queued(rreq);\n \t\tnetfs_wake_collector(rreq);\n \t}\n+\n+\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n }\n \n /*\n@@ -153,6 +143,7 @@ static ssize_t netfs_unbuffered_read(struct netfs_io_request *rreq, bool sync)\n ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter)\n {\n \tstruct netfs_io_request *rreq;\n+\tstruct netfs_io_stream *stream;\n \tssize_t ret;\n \tsize_t orig_count = iov_iter_count(iter);\n \tbool sync = is_sync_kiocb(iocb);\n@@ -177,29 +168,25 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i\n \tnetfs_stat(\u0026netfs_n_rh_dio_read);\n \ttrace_netfs_read(rreq, rreq-\u003estart, rreq-\u003elen, netfs_read_trace_dio_read);\n \n+\tstream = \u0026rreq-\u003eio_streams[0];\n+\n \t/* If this is an async op, we have to keep track of the destination\n \t * buffer for ourselves as the caller's iterator will be trashed when\n \t * we return.\n \t *\n-\t * In such a case, extract an iterator to represent as much of the the\n-\t * output buffer as we can manage. Note that the extraction might not\n-\t * be able to allocate a sufficiently large bvec array and may shorten\n-\t * the request.\n+\t * Extract a buffer queue to represent as much of the output buffer as\n+\t * we can manage. The fragments are extracted into a bvecq which will\n+\t * have sufficient nodes allocated to hold all the data, though this\n+\t * may end up truncated if ENOMEM is encountered.\n \t */\n-\tif (user_backed_iter(iter)) {\n-\t\tret = netfs_extract_user_iter(iter, rreq-\u003elen, \u0026rreq-\u003ebuffer.iter, 0);\n-\t\tif (ret \u003c 0)\n-\t\t\tgoto error_put;\n-\t\trreq-\u003edirect_bv = (struct bio_vec *)rreq-\u003ebuffer.iter.bvec;\n-\t\trreq-\u003edirect_bv_count = ret;\n-\t\trreq-\u003edirect_bv_unpin = iov_iter_extract_will_pin(iter);\n-\t\trreq-\u003elen = iov_iter_count(\u0026rreq-\u003ebuffer.iter);\n-\t} else {\n-\t\trreq-\u003ebuffer.iter = *iter;\n-\t\trreq-\u003elen = orig_count;\n-\t\trreq-\u003edirect_bv_unpin = false;\n-\t\tiov_iter_advance(iter, orig_count);\n-\t}\n+\tret = netfs_extract_iter(iter, rreq-\u003elen, INT_MAX, iocb-\u003eki_pos,\n+\t\t\t\t \u0026rreq-\u003eload_cursor.bvecq, 0);\n+\tif (ret \u003c 0)\n+\t\tgoto error_put;\n+\n+\trreq-\u003elen = ret;\n+\tstream-\u003ebuffered = ret;\n+\tstream-\u003eissue_from = rreq-\u003estart;\n \n \t// TODO: Set up bounce buffer if needed\n \ndiff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c\nindex c16fbad286a177..e5140bdf2a2465 100644\n--- a/fs/netfs/direct_write.c\n+++ b/fs/netfs/direct_write.c\n@@ -9,6 +9,34 @@\n #include \u003clinux/uio.h\u003e\n #include \"internal.h\"\n \n+/*\n+ * Prepare the buffer for an unbuffered/DIO write.\n+ */\n+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n+{\n+\tstruct netfs_io_stream *stream = \u0026subreq-\u003erreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tsize_t len;\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026stream-\u003edispatch_cursor);\n+\tlen = bvecq_slice(\u0026stream-\u003edispatch_cursor, subreq-\u003elen, max_segs,\n+\t\t\t \u0026subreq-\u003enr_segs);\n+\n+\tif (len \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = len;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n+\t}\n+\n+\t// TODO: Wait here for completion of prev subreq\n+\n+\tstream-\u003eissue_from += subreq-\u003elen;\n+\tstream-\u003ebuffered -= subreq-\u003elen;\n+\tif (stream-\u003ebuffered == 0)\n+\t\tnetfs_all_subreqs_queued(subreq-\u003erreq);\n+\treturn 0;\n+}\n+\n /*\n * Perform the cleanup rituals after an unbuffered write is complete.\n */\n@@ -73,7 +101,11 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,\n \tspin_unlock(\u0026wreq-\u003elock);\n \n \twreq-\u003etransferred += subreq-\u003etransferred;\n-\tiov_iter_advance(\u0026wreq-\u003ebuffer.iter, subreq-\u003etransferred);\n+\tif (subreq-\u003etransferred \u003c subreq-\u003elen) {\n+\t\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n+\t\tbvecq_pos_transfer(\u0026stream-\u003edispatch_cursor, \u0026subreq-\u003edispatch_pos);\n+\t\tbvecq_pos_advance(\u0026stream-\u003edispatch_cursor, subreq-\u003etransferred);\n+\t}\n \n \tstream-\u003ecollected_to = subreq-\u003estart + subreq-\u003etransferred;\n \twreq-\u003ecollected_to = stream-\u003ecollected_to;\n@@ -81,6 +113,7 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,\n \n \ttrace_netfs_collect_stream(wreq, stream);\n \ttrace_netfs_collect_state(wreq, wreq-\u003ecollected_to, 0);\n+\t/* TODO: Progressively clean up wreq-\u003edirect_bq */\n }\n \n /*\n@@ -99,53 +132,44 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)\n \n \t_enter(\"%llx\", wreq-\u003elen);\n \n+\tstream-\u003eissue_from = wreq-\u003estart;\n+\tstream-\u003ebuffered = wreq-\u003elen;\n+\tbvecq_pos_set(\u0026stream-\u003edispatch_cursor, \u0026wreq-\u003eload_cursor);\n+\tbvecq_pos_set(\u0026wreq-\u003ecollect_cursor, \u0026stream-\u003edispatch_cursor);\n+\n \tif (wreq-\u003eorigin == NETFS_DIO_WRITE)\n \t\tinode_dio_begin(wreq-\u003einode);\n \n-\tstream-\u003ecollected_to = wreq-\u003estart;\n-\n \tfor (;;) {\n \t\tbool retry = false;\n \n \t\tif (!subreq) {\n-\t\t\tnetfs_prepare_write(wreq, stream, wreq-\u003estart + wreq-\u003etransferred);\n-\t\t\tsubreq = stream-\u003econstruct;\n-\t\t\tstream-\u003econstruct = NULL;\n+\t\t\tsubreq = netfs_alloc_write_subreq(wreq, stream);\n+\t\t\tif (!subreq)\n+\t\t\t\treturn -ENOMEM;\n \t\t}\n \n-\t\t/* Check if (re-)preparation failed. */\n-\t\tif (unlikely(test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags))) {\n-\t\t\tnetfs_write_subrequest_terminated(subreq, subreq-\u003eerror);\n-\t\t\twreq-\u003eerror = subreq-\u003eerror;\n-\t\t\tbreak;\n+\t\tret = stream-\u003eissue_write(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\t/* Ownership of subreq was returned to us. */\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n \t\t}\n \n-\t\tiov_iter_truncate(\u0026subreq-\u003eio_iter, wreq-\u003elen - wreq-\u003etransferred);\n-\t\tif (!iov_iter_count(\u0026subreq-\u003eio_iter))\n-\t\t\tbreak;\n-\n-\t\tsubreq-\u003elen = netfs_limit_iter(\u0026subreq-\u003eio_iter, 0,\n-\t\t\t\t\t stream-\u003esreq_max_len,\n-\t\t\t\t\t stream-\u003esreq_max_segs);\n-\t\tiov_iter_truncate(\u0026subreq-\u003eio_iter, subreq-\u003elen);\n-\t\tstream-\u003esubmit_extendable_to = subreq-\u003elen;\n-\n-\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n-\t\tstream-\u003eissue_write(subreq);\n-\n-\t\t/* Async, need to wait. */\n-\t\tnetfs_wait_for_in_progress_stream(wreq, stream);\n-\n-\t\tif (test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n+\t\tret = netfs_wait_for_in_progress_subreq(wreq, subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\tif (ret != -EAGAIN) {\n+\t\t\t\t/* Don't need to lock here as the collection is\n+\t\t\t\t * done in this thread.\n+\t\t\t\t */\n+\t\t\t\tlist_del_init(\u0026subreq-\u003erreq_link);\n+\t\t\t\tret = subreq-\u003eerror;\n+\t\t\t\tnetfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);\n+\t\t\t\tsubreq = NULL;\n+\t\t\t\tgoto failed;\n+\t\t\t}\n \t\t\tretry = true;\n-\t\t} else if (test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags)) {\n-\t\t\tret = subreq-\u003eerror;\n-\t\t\twreq-\u003eerror = ret;\n-\t\t\tnetfs_see_subrequest(subreq, netfs_sreq_trace_see_failed);\n-\t\t\tsubreq = NULL;\n-\t\t\tbreak;\n \t\t}\n-\t\tret = 0;\n \n \t\tif (!retry) {\n \t\t\tnetfs_unbuffered_write_collect(wreq, stream, subreq);\n@@ -160,15 +184,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)\n \t\t\tcontinue;\n \t\t}\n \n-\t\t/* We need to retry the last subrequest, so first reset the\n-\t\t * iterator, taking into account what, if anything, we managed\n-\t\t * to transfer.\n+\t\t/* We need to retry the last subrequest, so first wind back the\n+\t\t * buffer position.\n \t\t */\n \t\tsubreq-\u003eerror = -EAGAIN;\n \t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n+\n+\t\tbvecq_pos_unset(\u0026subreq-\u003econtent);\n+\t\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n+\t\tbvecq_pos_transfer(\u0026stream-\u003edispatch_cursor, \u0026subreq-\u003edispatch_pos);\n+\n+\t\tstream-\u003eissue_from -= subreq-\u003elen - subreq-\u003etransferred;\n+\t\tstream-\u003ebuffered += subreq-\u003elen - subreq-\u003etransferred;\n \t\tif (subreq-\u003etransferred \u003e 0) {\n-\t\t\tiov_iter_advance(\u0026wreq-\u003ebuffer.iter, subreq-\u003etransferred);\n-\t\t\twreq-\u003etransferred += subreq-\u003etransferred;\n+\t\t\twreq-\u003etransferred += subreq-\u003etransferred;\n+\t\t\tbvecq_pos_advance(\u0026stream-\u003edispatch_cursor, subreq-\u003etransferred);\n \t\t}\n \n \t\tif (stream-\u003esource == NETFS_UPLOAD_TO_SERVER \u0026\u0026\n@@ -177,24 +207,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)\n \n \t\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n \t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n-\t\t__clear_bit(NETFS_SREQ_BOUNDARY, \u0026subreq-\u003eflags);\n \t\t__clear_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n-\t\tsubreq-\u003eio_iter\t\t= wreq-\u003ebuffer.iter;\n-\t\tsubreq-\u003estart\t\t= wreq-\u003estart + wreq-\u003etransferred;\n-\t\tsubreq-\u003elen\t\t= wreq-\u003elen - wreq-\u003etransferred;\n+\t\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n+\t\tsubreq-\u003estart\t\t= stream-\u003eissue_from;\n+\t\tsubreq-\u003elen\t\t= stream-\u003ebuffered;\n \t\tsubreq-\u003etransferred\t= 0;\n \t\tsubreq-\u003eretry_count\t+= 1;\n-\t\tstream-\u003esreq_max_len\t= UINT_MAX;\n-\t\tstream-\u003esreq_max_segs\t= INT_MAX;\n \n \t\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n \n-\t\tif (stream-\u003eprepare_write)\n-\t\t\tstream-\u003eprepare_write(subreq);\n \t\t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n \t\tnetfs_stat(\u0026netfs_n_wh_retry_write_subreq);\n \t}\n \n+failed:\n+\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n \tnetfs_unbuffered_write_done(wreq);\n \t_leave(\" = %d\", ret);\n \treturn ret;\n@@ -213,12 +240,12 @@ static void netfs_unbuffered_write_async(struct work_struct *work)\n * encrypted file. This can also be used for direct I/O writes.\n */\n ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *iter,\n-\t\t\t\t\t\t struct netfs_group *netfs_group)\n+\t\t\t\t\t struct netfs_group *netfs_group)\n {\n \tstruct netfs_io_request *wreq;\n \tunsigned long long start = iocb-\u003eki_pos;\n \tunsigned long long end = start + iov_iter_count(iter);\n-\tssize_t ret, n;\n+\tssize_t ret;\n \tsize_t len = iov_iter_count(iter);\n \tbool async = !is_sync_kiocb(iocb);\n \n@@ -238,6 +265,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *\n \tif (IS_ERR(wreq))\n \t\treturn PTR_ERR(wreq);\n \n+\twreq-\u003elen = iov_iter_count(iter);\n \twreq-\u003eio_streams[0].avail = true;\n \ttrace_netfs_write(wreq, (iocb-\u003eki_flags \u0026 IOCB_DIRECT ?\n \t\t\t\t netfs_write_trace_dio_write :\n@@ -248,33 +276,21 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *\n \t\t * we have to save the source buffer as the iterator is only\n \t\t * good until we return. In such a case, extract an iterator\n \t\t * to represent as much of the the output buffer as we can\n-\t\t * manage. Note that the extraction might not be able to\n-\t\t * allocate a sufficiently large bvec array and may shorten the\n-\t\t * request.\n+\t\t * manage. Note that the extraction may shorten the request.\n \t\t */\n-\t\tif (user_backed_iter(iter)) {\n-\t\t\tn = netfs_extract_user_iter(iter, len, \u0026wreq-\u003ebuffer.iter, 0);\n-\t\t\tif (n \u003c 0) {\n-\t\t\t\tret = n;\n-\t\t\t\tgoto error_put;\n-\t\t\t}\n-\t\t\twreq-\u003edirect_bv = (struct bio_vec *)wreq-\u003ebuffer.iter.bvec;\n-\t\t\twreq-\u003edirect_bv_count = n;\n-\t\t\twreq-\u003edirect_bv_unpin = iov_iter_extract_will_pin(iter);\n-\t\t} else {\n-\t\t\t/* If this is a kernel-generated async DIO request,\n-\t\t\t * assume that any resources the iterator points to\n-\t\t\t * (eg. a bio_vec array) will persist till the end of\n-\t\t\t * the op.\n-\t\t\t */\n-\t\t\twreq-\u003ebuffer.iter = *iter;\n-\t\t}\n+\t\tssize_t n = netfs_extract_iter(iter, len, INT_MAX, iocb-\u003eki_pos,\n+\t\t\t\t\t \u0026wreq-\u003eload_cursor.bvecq, 0);\n \n-\t\twreq-\u003elen = iov_iter_count(\u0026wreq-\u003ebuffer.iter);\n+\t\tif (n \u003c 0) {\n+\t\t\tret = n;\n+\t\t\tgoto error_put;\n+\t\t}\n+\t\twreq-\u003elen = n;\n+\t\t_debug(\"dio-write %zx/%zx %u/%u\",\n+\t\t n, len, wreq-\u003eload_cursor.bvecq-\u003enr_slots,\n+\t\t wreq-\u003eload_cursor.bvecq-\u003emax_slots);\n \t}\n \n-\t__set_bit(NETFS_RREQ_USE_IO_ITER, \u0026wreq-\u003eflags);\n-\n \t/* Copy the data into the bounce buffer and encrypt it. */\n \t// TODO\n \ndiff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c\nindex 37f05b4d34699d..a04cd96cd09ac9 100644\n--- a/fs/netfs/fscache_io.c\n+++ b/fs/netfs/fscache_io.c\n@@ -79,7 +79,7 @@ static int fscache_begin_operation(struct netfs_cache_resources *cres,\n \tcres-\u003eops\t\t= NULL;\n \tcres-\u003ecache_priv\t= cookie;\n \tcres-\u003ecache_priv2\t= NULL;\n-\tcres-\u003edebug_id\t\t= cookie-\u003edebug_id;\n+\tcres-\u003ecookie_id\t\t= cookie-\u003edebug_id;\n \tcres-\u003einval_counter\t= cookie-\u003einval_counter;\n \n \tif (!fscache_begin_cookie_access(cookie, why)) {\n@@ -239,7 +239,7 @@ void __fscache_write_to_cache(struct fscache_cookie *cookie,\n \t\t\t\t fscache_access_io_write) \u003c 0)\n \t\tgoto abandon_free;\n \n-\tret = cres-\u003eops-\u003eprepare_write(cres, \u0026start, \u0026len, len, i_size, false);\n+\tret = cres-\u003eops-\u003eprepare_write_old(cres, \u0026start, \u0026len, len, i_size, false);\n \tif (ret \u003c 0)\n \t\tgoto abandon_end;\n \ndiff --git a/fs/netfs/internal.h b/fs/netfs/internal.h\nindex d889caa401dc24..79695b59e5cd57 100644\n--- a/fs/netfs/internal.h\n+++ b/fs/netfs/internal.h\n@@ -7,7 +7,6 @@\n \n #include \u003clinux/slab.h\u003e\n #include \u003clinux/seq_file.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n #include \u003clinux/netfs.h\u003e\n #include \u003clinux/fscache.h\u003e\n #include \u003clinux/fscache-cache.h\u003e\n@@ -23,8 +22,9 @@\n /*\n * buffered_read.c\n */\n-void netfs_queue_read(struct netfs_io_request *rreq,\n-\t\t struct netfs_io_subrequest *subreq);\n+void netfs_read_query_cache(struct netfs_io_request *rreq,\n+\t\t\t struct fscache_occupancy *occ);\n+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq);\n void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);\n int netfs_prefetch_for_write(struct file *file, struct folio *folio,\n \t\t\t size_t offset, size_t len);\n@@ -35,6 +35,18 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,\n void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,\n \t\t\t loff_t pos, size_t copied);\n \n+/*\n+ * direct_read.c\n+ */\n+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs);\n+\n+/*\n+ * direct_write.c\n+ */\n+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs);\n+\n /*\n * main.c\n */\n@@ -67,17 +79,19 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}\n /*\n * misc.c\n */\n-struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq,\n-\t\t\t\t\t enum netfs_folioq_trace trace);\n-void netfs_reset_iter(struct netfs_io_subrequest *subreq);\n+struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq,\n+\t\t\t\t enum netfs_bvecq_trace trace);\n void netfs_wake_collector(struct netfs_io_request *rreq);\n void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq);\n+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,\n+\t\t\t\t struct netfs_io_subrequest *subreq);\n void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,\n \t\t\t\t struct netfs_io_stream *stream);\n ssize_t netfs_wait_for_read(struct netfs_io_request *rreq);\n ssize_t netfs_wait_for_write(struct netfs_io_request *rreq);\n void netfs_wait_for_paused_read(struct netfs_io_request *rreq);\n void netfs_wait_for_paused_write(struct netfs_io_request *rreq);\n+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq);\n \n /*\n * objects.c\n@@ -116,16 +130,53 @@ void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);\n /*\n * read_pgpriv2.c\n */\n+#ifdef CONFIG_NETFS_PGPRIV2\n+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs);\n void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio);\n void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq);\n bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq);\n+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)\n+{\n+\treturn test_bit(NETFS_RREQ_USE_PGPRIV2, \u0026rreq-\u003eflags);\n+}\n+#else\n+static inline int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t\t unsigned int max_segs)\n+{\n+\treturn -EIO;\n+}\n+static inline void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio)\n+{\n+}\n+static inline void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)\n+{\n+}\n+static inline bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)\n+{\n+\treturn true;\n+}\n+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)\n+{\n+\treturn false;\n+}\n+#endif\n \n /*\n * read_retry.c\n */\n+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs);\n+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq);\n void netfs_retry_reads(struct netfs_io_request *rreq);\n void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq);\n \n+/*\n+ * read_single.c\n+ */\n+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs);\n+\n /*\n * stats.c\n */\n@@ -153,7 +204,6 @@ extern atomic_t netfs_n_rh_write_zskip;\n extern atomic_t netfs_n_rh_retry_read_req;\n extern atomic_t netfs_n_rh_retry_read_subreq;\n extern atomic_t netfs_n_wh_buffered_write;\n-extern atomic_t netfs_n_wh_writethrough;\n extern atomic_t netfs_n_wh_dio_write;\n extern atomic_t netfs_n_wh_writepages;\n extern atomic_t netfs_n_wh_copy_to_cache;\n@@ -168,7 +218,7 @@ extern atomic_t netfs_n_wh_retry_write_req;\n extern atomic_t netfs_n_wh_retry_write_subreq;\n extern atomic_t netfs_n_wb_lock_skip;\n extern atomic_t netfs_n_wb_lock_wait;\n-extern atomic_t netfs_n_folioq;\n+extern atomic_t netfs_n_bvecq;\n \n int netfs_stats_show(struct seq_file *m, void *v);\n \n@@ -197,31 +247,19 @@ void netfs_write_collection_worker(struct work_struct *work);\n /*\n * write_issue.c\n */\n+struct netfs_writethrough;\n struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,\n \t\t\t\t\t\tstruct file *file,\n \t\t\t\t\t\tloff_t start,\n \t\t\t\t\t\tenum netfs_io_origin origin);\n-void netfs_prepare_write(struct netfs_io_request *wreq,\n-\t\t\t struct netfs_io_stream *stream,\n-\t\t\t loff_t start);\n-void netfs_reissue_write(struct netfs_io_stream *stream,\n-\t\t\t struct netfs_io_subrequest *subreq,\n-\t\t\t struct iov_iter *source);\n-void netfs_issue_write(struct netfs_io_request *wreq,\n-\t\t struct netfs_io_stream *stream);\n-size_t netfs_advance_write(struct netfs_io_request *wreq,\n-\t\t\t struct netfs_io_stream *stream,\n-\t\t\t loff_t start, size_t len, bool to_eof);\n-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len);\n-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,\n-\t\t\t struct folio *folio, size_t copied, bool to_page_end,\n-\t\t\t struct folio **writethrough_cache);\n-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,\n-\t\t\t struct folio *writethrough_cache);\n+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,\n+\t\t\t\t\t\t struct netfs_io_stream *stream);\n \n /*\n * write_retry.c\n */\n+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs);\n void netfs_retry_writes(struct netfs_io_request *wreq);\n \n /*\n@@ -316,6 +354,25 @@ static inline bool netfs_check_subreq_in_progress(const struct netfs_io_subreque\n \treturn test_bit_acquire(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n }\n \n+/*\n+ * Indicate that we've generated and queued all the subrequests we're going to.\n+ */\n+static inline void netfs_all_subreqs_queued(struct netfs_io_request *rreq)\n+{\n+\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n+\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_all_queued);\n+}\n+\n+/*\n+ * Query if all subrequests are queued.\n+ */\n+static inline bool netfs_are_all_subreqs_queued(const struct netfs_io_request *rreq)\n+{\n+\t/* Read lists after ALL_QUEUED. */\n+\treturn test_bit_acquire(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n+}\n+\n /*\n * fscache-cache.c\n */\ndiff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c\nindex b375567e0520ed..7dea84c40e5bfc 100644\n--- a/fs/netfs/iterator.c\n+++ b/fs/netfs/iterator.c\n@@ -14,295 +14,143 @@\n #include \"internal.h\"\n \n /**\n- * netfs_extract_user_iter - Extract the pages from a user iterator into a bvec\n+ * netfs_extract_iter - Extract virtually contiguous pages from an iterator into a bvecq\n * @orig: The original iterator\n- * @orig_len: The amount of iterator to copy\n- * @new: The iterator to be set up\n+ * @max_len: Maximum number of bytes to extract\n+ * @max_pages: Maximum number of pages to extract\n+ * @fpos: Starting file position to label the bvecq with\n+ * @_bvecq_head: Where to cache the bvec queue\n * @extraction_flags: Flags to qualify the request\n *\n- * Extract the page fragments from the given amount of the source iterator and\n- * build up a second iterator that refers to all of those bits. This allows\n- * the original iterator to be disposed of.\n+ * Extract virtually contiguous page fragments from the source iterator up to\n+ * the given maxima and build bvec queue that refers to all of those bits.\n+ * This allows the original iterator to disposed of.\n *\n * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA be\n * allowed on the pages extracted.\n *\n- * On success, the number of elements in the bvec is returned, the original\n- * iterator will have been advanced by the amount extracted.\n+ * On success or partial success, the amount of data in the bvec is returned,\n+ * the original iterator will have been advanced by the amount extracted.\n *\n- * The iov_iter_extract_mode() function should be used to query how cleanup\n- * should be performed.\n+ * If an error occurs and no pages are extracted, an error will be returned and\n+ * any allocated bvecq will be freed. If there is no data to be extracted (or\n+ * @max_len or @max_pages are zero), a single empty bvecq will be returned.\n+ *\n+ * The bvecq segments are marked with indications on how to get clean up the\n+ * extracted fragments.\n */\n-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,\n-\t\t\t\tstruct iov_iter *new,\n-\t\t\t\tiov_iter_extraction_t extraction_flags)\n+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,\n+\t\t\t unsigned long long fpos, struct bvecq **_bvecq_head,\n+\t\t\t iov_iter_extraction_t extraction_flags)\n {\n-\tstruct bio_vec *bv = NULL;\n-\tstruct page **pages;\n-\tunsigned int cur_npages;\n-\tunsigned int max_pages;\n-\tunsigned int npages = 0;\n-\tunsigned int i;\n+\tstruct bvecq *bq_tail = NULL, *bq;\n \tssize_t ret = 0;\n-\tsize_t count = orig_len, offset, len;\n-\tsize_t bv_size, pg_size;\n-\n-\tif (WARN_ON_ONCE(!iter_is_ubuf(orig) \u0026\u0026 !iter_is_iovec(orig)))\n-\t\treturn -EIO;\n-\n-\tmax_pages = iov_iter_npages(orig, INT_MAX);\n-\tbv_size = array_size(max_pages, sizeof(*bv));\n-\tbv = kvmalloc(bv_size, GFP_KERNEL);\n-\tif (!bv)\n-\t\treturn -ENOMEM;\n-\n-\t/* Put the page list at the end of the bvec list storage. bvec\n-\t * elements are larger than page pointers, so as long as we work\n-\t * 0-\u003elast, we should be fine.\n-\t */\n-\tpg_size = array_size(max_pages, sizeof(*pages));\n-\tpages = (void *)bv + bv_size - pg_size;\n-\n-\twhile (count \u0026\u0026 npages \u003c max_pages) {\n-\t\tret = iov_iter_extract_pages(orig, \u0026pages, count,\n-\t\t\t\t\t max_pages - npages, extraction_flags,\n-\t\t\t\t\t \u0026offset);\n-\t\tif (unlikely(ret \u003c= 0)) {\n-\t\t\tret = ret ?: -EIO;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tif (WARN(ret \u003e count,\n-\t\t\t \"%s: extract_pages overrun %zd \u003e %zu bytes\\n\",\n-\t\t\t __func__, ret, count)) {\n-\t\t\tret = -EIO;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tcur_npages = DIV_ROUND_UP(offset + ret, PAGE_SIZE);\n-\t\tif (WARN(cur_npages \u003e max_pages - npages,\n-\t\t\t \"%s: extract_pages overrun %u \u003e %u pages\\n\",\n-\t\t\t __func__, npages + cur_npages, max_pages)) {\n-\t\t\tret = -EIO;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tcount -= ret;\n-\t\tret += offset;\n-\n-\t\tfor (i = 0; i \u003c cur_npages; i++) {\n-\t\t\tlen = ret \u003e PAGE_SIZE ? PAGE_SIZE : ret;\n-\t\t\tbvec_set_page(bv + npages + i, *pages++, len - offset, offset);\n-\t\t\tret -= len;\n-\t\t\toffset = 0;\n-\t\t}\n-\n-\t\tnpages += cur_npages;\n-\t}\n-\n-\t/* Note: Don't try to clean up after EIO. Either we got no pages, so\n-\t * nothing to clean up, or we got a buffer overrun, memory corruption\n-\t * and can't trust the stuff in the buffer (a WARN was emitted).\n-\t */\n-\n-\tif (ret \u003c 0 \u0026\u0026 (ret == -ENOMEM || npages == 0)) {\n-\t\tfor (i = 0; i \u003c npages; i++)\n-\t\t\tunpin_user_page(bv[i].bv_page);\n-\t\tkvfree(bv);\n-\t\treturn ret;\n-\t}\n-\n-\tiov_iter_bvec(new, orig-\u003edata_source, bv, npages, orig_len - count);\n-\treturn npages;\n-}\n-EXPORT_SYMBOL_GPL(netfs_extract_user_iter);\n-\n-/*\n- * Select the span of a bvec iterator we're going to use. Limit it by both maximum\n- * size and maximum number of segments. Returns the size of the span in bytes.\n- */\n-static size_t netfs_limit_bvec(const struct iov_iter *iter, size_t start_offset,\n-\t\t\t size_t max_size, size_t max_segs)\n-{\n-\tconst struct bio_vec *bvecs = iter-\u003ebvec;\n-\tunsigned int nbv = iter-\u003enr_segs, ix = 0, nsegs = 0;\n-\tsize_t len, span = 0, n = iter-\u003ecount;\n-\tsize_t skip = iter-\u003eiov_offset + start_offset;\n-\n-\tif (WARN_ON(!iov_iter_is_bvec(iter)) ||\n-\t WARN_ON(start_offset \u003e n) ||\n-\t n == 0)\n-\t\treturn 0;\n-\n-\twhile (n \u0026\u0026 ix \u003c nbv \u0026\u0026 skip) {\n-\t\tlen = bvecs[ix].bv_len;\n-\t\tif (skip \u003c len)\n-\t\t\tbreak;\n-\t\tskip -= len;\n-\t\tn -= len;\n-\t\tix++;\n-\t}\n+\tsize_t extracted = 0;\n \n-\twhile (n \u0026\u0026 ix \u003c nbv) {\n-\t\tlen = min3(n, bvecs[ix].bv_len - skip, max_size);\n-\t\tspan += len;\n-\t\tnsegs++;\n-\t\tix++;\n-\t\tif (span \u003e= max_size || nsegs \u003e= max_segs)\n-\t\t\tbreak;\n-\t\tskip = 0;\n-\t\tn -= len;\n-\t}\n+\t_enter(\"{%u,%zx},%zx\", orig-\u003eiter_type, orig-\u003ecount, max_len);\n \n-\treturn min(span, max_size);\n-}\n+\t*_bvecq_head = NULL;\n+\tif (max_len \u003e orig-\u003ecount)\n+\t\tmax_len = orig-\u003ecount;\n+\tif (WARN_ON_ONCE(!max_len || !max_pages))\n+\t\tgoto alloc_empty;\n \n-/*\n- * Select the span of a kvec iterator we're going to use. Limit it by both\n- * maximum size and maximum number of segments. Returns the size of the span\n- * in bytes.\n- */\n-static size_t netfs_limit_kvec(const struct iov_iter *iter, size_t start_offset,\n-\t\t\t size_t max_size, size_t max_segs)\n-{\n-\tconst struct kvec *kvecs = iter-\u003ekvec;\n-\tunsigned int nkv = iter-\u003enr_segs, ix = 0, nsegs = 0;\n-\tsize_t len, span = 0, n = iter-\u003ecount;\n-\tsize_t skip = iter-\u003eiov_offset + start_offset;\n+\tmax_pages = iov_iter_npages(orig, max_pages);\n+\tif (!max_pages)\n+\t\tgoto alloc_empty;\n \n-\tif (WARN_ON(!iov_iter_is_kvec(iter)) ||\n-\t WARN_ON(start_offset \u003e n) ||\n-\t n == 0)\n-\t\treturn 0;\n-\n-\twhile (n \u0026\u0026 ix \u003c nkv \u0026\u0026 skip) {\n-\t\tlen = kvecs[ix].iov_len;\n-\t\tif (skip \u003c len)\n-\t\t\tbreak;\n-\t\tskip -= len;\n-\t\tn -= len;\n-\t\tix++;\n-\t}\n-\n-\twhile (n \u0026\u0026 ix \u003c nkv) {\n-\t\tlen = min3(n, kvecs[ix].iov_len - skip, max_size);\n-\t\tspan += len;\n-\t\tnsegs++;\n-\t\tix++;\n-\t\tif (span \u003e= max_size || nsegs \u003e= max_segs)\n-\t\t\tbreak;\n-\t\tskip = 0;\n-\t\tn -= len;\n-\t}\n-\n-\treturn min(span, max_size);\n-}\n-\n-/*\n- * Select the span of an xarray iterator we're going to use. Limit it by both\n- * maximum size and maximum number of segments. It is assumed that segments\n- * can be larger than a page in size, provided they're physically contiguous.\n- * Returns the size of the span in bytes.\n- */\n-static size_t netfs_limit_xarray(const struct iov_iter *iter, size_t start_offset,\n-\t\t\t\t size_t max_size, size_t max_segs)\n-{\n-\tstruct folio *folio;\n-\tunsigned int nsegs = 0;\n-\tloff_t pos = iter-\u003exarray_start + iter-\u003eiov_offset;\n-\tpgoff_t index = pos / PAGE_SIZE;\n-\tsize_t span = 0, n = iter-\u003ecount;\n-\n-\tXA_STATE(xas, iter-\u003exarray, index);\n-\n-\tif (WARN_ON(!iov_iter_is_xarray(iter)) ||\n-\t WARN_ON(start_offset \u003e n) ||\n-\t n == 0)\n-\t\treturn 0;\n-\tmax_size = min(max_size, n - start_offset);\n-\n-\trcu_read_lock();\n-\txas_for_each(\u0026xas, folio, ULONG_MAX) {\n-\t\tsize_t offset, flen, len;\n-\t\tif (xas_retry(\u0026xas, folio))\n-\t\t\tcontinue;\n-\t\tif (WARN_ON(xa_is_value(folio)))\n-\t\t\tbreak;\n-\t\tif (WARN_ON(folio_test_hugetlb(folio)))\n-\t\t\tbreak;\n-\n-\t\tflen = folio_size(folio);\n-\t\toffset = offset_in_folio(folio, pos);\n-\t\tlen = min(max_size, flen - offset);\n-\t\tspan += len;\n-\t\tnsegs++;\n-\t\tif (span \u003e= max_size || nsegs \u003e= max_segs)\n-\t\t\tbreak;\n-\t}\n-\n-\trcu_read_unlock();\n-\treturn min(span, max_size);\n-}\n-\n-/*\n- * Select the span of a folio queue iterator we're going to use. Limit it by\n- * both maximum size and maximum number of segments. Returns the size of the\n- * span in bytes.\n- */\n-static size_t netfs_limit_folioq(const struct iov_iter *iter, size_t start_offset,\n-\t\t\t\t size_t max_size, size_t max_segs)\n-{\n-\tconst struct folio_queue *folioq = iter-\u003efolioq;\n-\tunsigned int nsegs = 0;\n-\tunsigned int slot = iter-\u003efolioq_slot;\n-\tsize_t span = 0, n = iter-\u003ecount;\n-\n-\tif (WARN_ON(!iov_iter_is_folioq(iter)) ||\n-\t WARN_ON(start_offset \u003e n) ||\n-\t n == 0)\n-\t\treturn 0;\n-\tmax_size = umin(max_size, n - start_offset);\n-\n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tslot = 0;\n-\t}\n-\n-\tstart_offset += iter-\u003eiov_offset;\n \tdo {\n-\t\tsize_t flen = folioq_folio_size(folioq, slot);\n-\n-\t\tif (start_offset \u003c flen) {\n-\t\t\tspan += flen - start_offset;\n-\t\t\tnsegs++;\n-\t\t\tstart_offset = 0;\n-\t\t} else {\n-\t\t\tstart_offset -= flen;\n-\t\t}\n-\t\tif (span \u003e= max_size || nsegs \u003e= max_segs)\n+\t\tbq = bvecq_alloc_one(max_pages, GFP_NOFS);\n+\t\tif (!bq) {\n+\t\t\tret = -ENOMEM;\n \t\t\tbreak;\n-\n-\t\tslot++;\n-\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\tfolioq = folioq-\u003enext;\n-\t\t\tslot = 0;\n \t\t}\n-\t} while (folioq);\n+\t\tif (user_backed_iter(orig))\n+\t\t\tbq-\u003emem_type = iov_iter_extract_will_pin(orig) ?\n+\t\t\t\tBVECQ_MEM_GUP : BVECQ_MEM_PAGECACHE;\n+\t\tbq-\u003efpos\t= fpos + extracted;\n+\n+\t\tif (bq_tail)\n+\t\t\tbvecq_append(bq_tail, bq);\n+\t\telse\n+\t\t\t*_bvecq_head = bq;\n+\t\tbq_tail = bq;\n+\n+\t\tif (max_len == 0)\n+\t\t\tbreak;\n \n-\treturn umin(span, max_size);\n-}\n+\t\tstruct bio_vec *bv = bq-\u003ebv;\n+\t\tunsigned int slot = 0;\n+\t\tdo {\n+\t\t\tstruct page **pages;\n+\t\t\tssize_t got;\n+\t\t\tsize_t offset;\n+\t\t\tsize_t space = bq-\u003emax_slots - slot;\n+\t\t\tsize_t bv_size = array_size(bq-\u003emax_slots, sizeof(*bv));\n+\t\t\tsize_t pg_size = array_size(space, sizeof(*pages));\n+\n+\t\t\t/* Put the page list at the end of the bvec list\n+\t\t\t * storage. bvec elements are larger than page\n+\t\t\t * pointers, so as long as we work 0-\u003elast, we should\n+\t\t\t * be fine.\n+\t\t\t */\n+\t\t\tpages = (void *)bv + bv_size - pg_size;\n+\n+\t\t\tgot = iov_iter_extract_pages(orig, \u0026pages, max_len,\n+\t\t\t\t\t\t min(space, max_pages),\n+\t\t\t\t\t\t extraction_flags, \u0026offset);\n+\t\t\tif (got \u003c 0) {\n+\t\t\t\tret = got;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tif (got == 0) {\n+\t\t\t\tpr_err(\"extract_pages gave nothing from %zx, %zx\\n\",\n+\t\t\t\t extracted, max_len);\n+\t\t\t\tret = -EIO;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tif (WARN(got \u003e max_len,\n+\t\t\t\t \"%s: extract_pages overrun %zx \u003e %zx bytes\\n\",\n+\t\t\t\t __func__, got, max_len)) {\n+\t\t\t\tret = -EIO;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\textracted += got;\n+\t\t\tmax_len -= got;\n+\n+\t\t\tdo {\n+\t\t\t\tsize_t len = umin(got, PAGE_SIZE - offset);\n+\n+\t\t\t\tBUG_ON(slot \u003e= bq-\u003emax_slots);\n+\n+\t\t\t\tbvec_set_page(\u0026bq-\u003ebv[slot], *pages++, len, offset);\n+\t\t\t\tslot++;\n+\t\t\t\tmax_pages--;\n+\t\t\t\tgot -= len;\n+\t\t\t\toffset = 0;\n+\t\t\t} while (got \u003e 0);\n+\n+\t\t\tbvecq_filled_to(bq, slot);\n+\t\t} while (max_len \u003e 0 \u0026\u0026 max_pages \u003e 0 \u0026\u0026 !bvecq_is_full(bq));\n+\n+\t} while (max_len \u003e 0 \u0026\u0026 max_pages \u003e 0);\n+\n+out:\n+\tif (extracted || ret == 0)\n+\t\treturn extracted;\n+\tbvecq_put(*_bvecq_head);\n+\t*_bvecq_head = NULL;\n+\treturn ret;\n+\n+alloc_empty:\n+\tbq = bvecq_alloc_one(1, GFP_NOFS);\n+\tif (!bq)\n+\t\treturn -ENOMEM;\n+\t*_bvecq_head = bq;\n+\treturn 0;\n \n-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,\n-\t\t\tsize_t max_size, size_t max_segs)\n-{\n-\tif (iov_iter_is_folioq(iter))\n-\t\treturn netfs_limit_folioq(iter, start_offset, max_size, max_segs);\n-\tif (iov_iter_is_bvec(iter))\n-\t\treturn netfs_limit_bvec(iter, start_offset, max_size, max_segs);\n-\tif (iov_iter_is_xarray(iter))\n-\t\treturn netfs_limit_xarray(iter, start_offset, max_size, max_segs);\n-\tif (iov_iter_is_kvec(iter))\n-\t\treturn netfs_limit_kvec(iter, start_offset, max_size, max_segs);\n-\tBUG();\n }\n-EXPORT_SYMBOL(netfs_limit_iter);\n+EXPORT_SYMBOL_GPL(netfs_extract_iter);\ndiff --git a/fs/netfs/main.c b/fs/netfs/main.c\nindex 73da6c9f5777cf..6a5a6b1b10a34b 100644\n--- a/fs/netfs/main.c\n+++ b/fs/netfs/main.c\n@@ -43,7 +43,6 @@ static const char *netfs_origins[nr__netfs_io_origin] = {\n \t[NETFS_DIO_READ]\t\t= \"DR\",\n \t[NETFS_WRITEBACK]\t\t= \"WB\",\n \t[NETFS_WRITEBACK_SINGLE]\t= \"W1\",\n-\t[NETFS_WRITETHROUGH]\t\t= \"WT\",\n \t[NETFS_UNBUFFERED_WRITE]\t= \"UW\",\n \t[NETFS_DIO_WRITE]\t\t= \"DW\",\n \t[NETFS_PGPRIV2_COPY_TO_CACHE]\t= \"2C\",\ndiff --git a/fs/netfs/misc.c b/fs/netfs/misc.c\nindex 5d554512ed23a2..0af45204fabc80 100644\n--- a/fs/netfs/misc.c\n+++ b/fs/netfs/misc.c\n@@ -8,120 +8,6 @@\n #include \u003clinux/swap.h\u003e\n #include \"internal.h\"\n \n-/**\n- * netfs_alloc_folioq_buffer - Allocate buffer space into a folio queue\n- * @mapping: Address space to set on the folio (or NULL).\n- * @_buffer: Pointer to the folio queue to add to (may point to a NULL; updated).\n- * @_cur_size: Current size of the buffer (updated).\n- * @size: Target size of the buffer.\n- * @gfp: The allocation constraints.\n- */\n-int netfs_alloc_folioq_buffer(struct address_space *mapping,\n-\t\t\t struct folio_queue **_buffer,\n-\t\t\t size_t *_cur_size, ssize_t size, gfp_t gfp)\n-{\n-\tstruct folio_queue *tail = *_buffer, *p;\n-\n-\tsize = round_up(size, PAGE_SIZE);\n-\tif (*_cur_size \u003e= size)\n-\t\treturn 0;\n-\n-\tif (tail)\n-\t\twhile (tail-\u003enext)\n-\t\t\ttail = tail-\u003enext;\n-\n-\tdo {\n-\t\tstruct folio *folio;\n-\t\tint order = 0, slot;\n-\n-\t\tif (!tail || folioq_full(tail)) {\n-\t\t\tp = netfs_folioq_alloc(0, GFP_NOFS, netfs_trace_folioq_alloc_buffer);\n-\t\t\tif (!p)\n-\t\t\t\treturn -ENOMEM;\n-\t\t\tif (tail) {\n-\t\t\t\ttail-\u003enext = p;\n-\t\t\t\tp-\u003eprev = tail;\n-\t\t\t} else {\n-\t\t\t\t*_buffer = p;\n-\t\t\t}\n-\t\t\ttail = p;\n-\t\t}\n-\n-\t\tif (size - *_cur_size \u003e PAGE_SIZE)\n-\t\t\torder = umin(ilog2(size - *_cur_size) - PAGE_SHIFT,\n-\t\t\t\t MAX_PAGECACHE_ORDER);\n-\n-\t\tfolio = folio_alloc(gfp, order);\n-\t\tif (!folio \u0026\u0026 order \u003e 0)\n-\t\t\tfolio = folio_alloc(gfp, 0);\n-\t\tif (!folio)\n-\t\t\treturn -ENOMEM;\n-\n-\t\tfolio-\u003emapping = mapping;\n-\t\tfolio-\u003eindex = *_cur_size / PAGE_SIZE;\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_alloc_buffer);\n-\t\tslot = folioq_append_mark(tail, folio);\n-\t\t*_cur_size += folioq_folio_size(tail, slot);\n-\t} while (*_cur_size \u003c size);\n-\n-\treturn 0;\n-}\n-EXPORT_SYMBOL(netfs_alloc_folioq_buffer);\n-\n-/**\n- * netfs_free_folioq_buffer - Free a folio queue.\n- * @fq: The start of the folio queue to free\n- *\n- * Free up a chain of folio_queues and, if marked, the marked folios they point\n- * to.\n- */\n-void netfs_free_folioq_buffer(struct folio_queue *fq)\n-{\n-\tstruct folio_queue *next;\n-\tstruct folio_batch fbatch;\n-\n-\tfolio_batch_init(\u0026fbatch);\n-\n-\tfor (; fq; fq = next) {\n-\t\tfor (int slot = 0; slot \u003c folioq_count(fq); slot++) {\n-\t\t\tstruct folio *folio = folioq_folio(fq, slot);\n-\n-\t\t\tif (!folio ||\n-\t\t\t !folioq_is_marked(fq, slot))\n-\t\t\t\tcontinue;\n-\n-\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_put);\n-\t\t\tif (folio_batch_add(\u0026fbatch, folio))\n-\t\t\t\tfolio_batch_release(\u0026fbatch);\n-\t\t}\n-\n-\t\tnetfs_stat_d(\u0026netfs_n_folioq);\n-\t\tnext = fq-\u003enext;\n-\t\tkfree(fq);\n-\t}\n-\n-\tfolio_batch_release(\u0026fbatch);\n-}\n-EXPORT_SYMBOL(netfs_free_folioq_buffer);\n-\n-/*\n- * Reset the subrequest iterator to refer just to the region remaining to be\n- * read. The iterator may or may not have been advanced by socket ops or\n- * extraction ops to an extent that may or may not match the amount actually\n- * read.\n- */\n-void netfs_reset_iter(struct netfs_io_subrequest *subreq)\n-{\n-\tstruct iov_iter *io_iter = \u0026subreq-\u003eio_iter;\n-\tsize_t remain = subreq-\u003elen - subreq-\u003etransferred;\n-\n-\tif (io_iter-\u003ecount \u003e remain)\n-\t\tiov_iter_advance(io_iter, io_iter-\u003ecount - remain);\n-\telse if (io_iter-\u003ecount \u003c remain)\n-\t\tiov_iter_revert(io_iter, remain - io_iter-\u003ecount);\n-\tiov_iter_truncate(\u0026subreq-\u003eio_iter, remain);\n-}\n-\n /**\n * netfs_dirty_folio - Mark folio dirty and pin a cache object for writeback\n * @mapping: The mapping the folio belongs to.\n@@ -364,6 +250,37 @@ void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq)\n \t\tnetfs_wake_collector(rreq);\n }\n \n+/*\n+ * Wait for a subrequest to come to completion.\n+ */\n+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,\n+\t\t\t\t struct netfs_io_subrequest *subreq)\n+{\n+\tif (netfs_check_subreq_in_progress(subreq)) {\n+\t\tDEFINE_WAIT(myself);\n+\n+\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_wait_quiesce);\n+\t\tfor (;;) {\n+\t\t\tprepare_to_wait(\u0026rreq-\u003ewaitq, \u0026myself, TASK_UNINTERRUPTIBLE);\n+\n+\t\t\tif (!netfs_check_subreq_in_progress(subreq))\n+\t\t\t\tbreak;\n+\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_wait_for);\n+\t\t\tschedule();\n+\t\t}\n+\n+\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_waited_quiesce);\n+\t\tfinish_wait(\u0026rreq-\u003ewaitq, \u0026myself);\n+\t}\n+\n+\tif (test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags))\n+\t\treturn -EAGAIN;\n+\tif (test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags))\n+\t\treturn subreq-\u003eerror;\n+\treturn 0;\n+}\n+\n /*\n * Wait for all outstanding I/O in a stream to quiesce.\n */\n@@ -424,7 +341,7 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq,\n \t\t\tneed_collect = true;\n \t\t\tbreak;\n \t\t}\n-\t\tif (subreq || !test_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags))\n+\t\tif (subreq || !netfs_are_all_subreqs_queued(rreq))\n \t\t\tdone = false;\n \t}\n \n@@ -494,7 +411,7 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,\n \t\tcase NETFS_UNBUFFERED_WRITE:\n \t\t\tbreak;\n \t\tdefault:\n-\t\t\tif (rreq-\u003esubmitted \u003c rreq-\u003elen) {\n+\t\t\tif (rreq-\u003etransferred \u003c rreq-\u003elen) {\n \t\t\t\ttrace_netfs_failure(rreq, NULL, ret, netfs_fail_short_read);\n \t\t\t\tret = -EIO;\n \t\t\t}\n@@ -563,3 +480,22 @@ void netfs_wait_for_paused_write(struct netfs_io_request *rreq)\n {\n \treturn netfs_wait_for_pause(rreq, netfs_write_collection);\n }\n+\n+/*\n+ * Wait for the readahead-acquired refs to be put.\n+ */\n+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq)\n+{\n+\tDEFINE_WAIT(myself);\n+\n+\tfor (;;) {\n+\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_wait_put_ra_refs);\n+\t\tprepare_to_wait(\u0026rreq-\u003ewaitq, \u0026myself, TASK_UNINTERRUPTIBLE);\n+\t\tif (!test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, \u0026rreq-\u003eflags))\n+\t\t\tbreak;\n+\t\tschedule();\n+\t}\n+\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_waited_put_ra_refs);\n+\tfinish_wait(\u0026rreq-\u003ewaitq, \u0026myself);\n+}\ndiff --git a/fs/netfs/objects.c b/fs/netfs/objects.c\nindex b8c4918d3dcdab..d4a95a462576f4 100644\n--- a/fs/netfs/objects.c\n+++ b/fs/netfs/objects.c\n@@ -46,8 +46,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,\n \trreq-\u003ei_size\t= i_size_read(inode);\n \trreq-\u003edebug_id\t= atomic_inc_return(\u0026debug_ids);\n \trreq-\u003ewsize\t= INT_MAX;\n-\trreq-\u003eio_streams[0].sreq_max_len = ULONG_MAX;\n-\trreq-\u003eio_streams[0].sreq_max_segs = 0;\n \tspin_lock_init(\u0026rreq-\u003elock);\n \tINIT_LIST_HEAD(\u0026rreq-\u003eio_streams[0].subrequests);\n \tINIT_LIST_HEAD(\u0026rreq-\u003eio_streams[1].subrequests);\n@@ -119,7 +117,6 @@ static void netfs_free_request_rcu(struct rcu_head *rcu)\n static void netfs_deinit_request(struct netfs_io_request *rreq)\n {\n \tstruct netfs_inode *ictx = netfs_inode(rreq-\u003einode);\n-\tunsigned int i;\n \n \ttrace_netfs_rreq(rreq, netfs_rreq_trace_free);\n \n@@ -134,16 +131,12 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)\n \t\trreq-\u003enetfs_ops-\u003efree_request(rreq);\n \tif (rreq-\u003ecache_resources.ops)\n \t\trreq-\u003ecache_resources.ops-\u003eend_operation(\u0026rreq-\u003ecache_resources);\n-\tif (rreq-\u003edirect_bv) {\n-\t\tfor (i = 0; i \u003c rreq-\u003edirect_bv_count; i++) {\n-\t\t\tif (rreq-\u003edirect_bv[i].bv_page) {\n-\t\t\t\tif (rreq-\u003edirect_bv_unpin)\n-\t\t\t\t\tunpin_user_page(rreq-\u003edirect_bv[i].bv_page);\n-\t\t\t}\n-\t\t}\n-\t\tkvfree(rreq-\u003edirect_bv);\n-\t}\n-\trolling_buffer_clear(\u0026rreq-\u003ebuffer);\n+\tbvecq_pos_unset(\u0026rreq-\u003eload_cursor);\n+\tbvecq_pos_unset(\u0026rreq-\u003ecollect_cursor);\n+\tbvecq_pos_unset(\u0026rreq-\u003eretry_cursor);\n+\tbvecq_put(rreq-\u003espare);\n+\tfor (int i = 0; i \u003c NR_IO_STREAMS; i++)\n+\t\tbvecq_pos_unset(\u0026rreq-\u003eio_streams[i].dispatch_cursor);\n \n \tif (atomic_dec_and_test(\u0026ictx-\u003eio_count))\n \t\twake_up_var(\u0026ictx-\u003eio_count);\n@@ -234,8 +227,11 @@ static void netfs_free_subrequest(struct netfs_io_subrequest *subreq)\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_free);\n+\tWARN_ON_ONCE(!list_empty(\u0026subreq-\u003erreq_link));\n \tif (rreq-\u003enetfs_ops-\u003efree_subrequest)\n \t\trreq-\u003enetfs_ops-\u003efree_subrequest(subreq);\n+\tbvecq_pos_unset(\u0026subreq-\u003edispatch_pos);\n+\tbvecq_pos_unset(\u0026subreq-\u003econtent);\n \tmempool_free(subreq, rreq-\u003enetfs_ops-\u003esubrequest_pool ?: \u0026netfs_subrequest_pool);\n \tnetfs_stat_d(\u0026netfs_n_rh_sreq);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_subreq);\ndiff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c\nindex 23660a59012464..5899ba022fabc2 100644\n--- a/fs/netfs/read_collect.c\n+++ b/fs/netfs/read_collect.c\n@@ -19,19 +19,25 @@\n #define MADE_PROGRESS\t0x04\t/* Made progress cleaning up a stream or the folio set */\n #define BUFFERED\t0x08\t/* The pagecache needs cleaning up */\n #define NEED_RETRY\t0x10\t/* A front op requests retrying */\n-#define COPY_TO_CACHE\t0x40\t/* Need to copy subrequest to cache */\n-#define ABANDON_SREQ\t0x80\t/* Need to abandon untransferred part of subrequest */\n+#define COPY_TO_CACHE\t0x20\t/* Need to copy subrequest to cache */\n+#define ABANDON_SREQ\t0x40\t/* Need to abandon untransferred part of subrequest */\n+#define ABANDON_RREQ\t0x80\t/* Need to abandon the rest of a request */\n \n /*\n * Clear the unread part of an I/O request.\n */\n static void netfs_clear_unread(struct netfs_io_subrequest *subreq)\n {\n-\tnetfs_reset_iter(subreq);\n-\tWARN_ON_ONCE(subreq-\u003elen - subreq-\u003etransferred != iov_iter_count(\u0026subreq-\u003eio_iter));\n-\tiov_iter_zero(iov_iter_count(\u0026subreq-\u003eio_iter), \u0026subreq-\u003eio_iter);\n+\tstruct iov_iter iter;\n+\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\tiov_iter_advance(\u0026iter, subreq-\u003etransferred);\n+\tiov_iter_zero(subreq-\u003elen, \u0026iter);\n+\n \tif (subreq-\u003estart + subreq-\u003etransferred \u003e= subreq-\u003erreq-\u003ei_size)\n \t\t__set_bit(NETFS_SREQ_HIT_EOF, \u0026subreq-\u003eflags);\n+\ttrace_netfs_rreq(subreq-\u003erreq, netfs_rreq_trace_zero_unread);\n }\n \n /*\n@@ -40,11 +46,11 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)\n * dirty and let writeback handle it.\n */\n static void netfs_unlock_read_folio(struct netfs_io_request *rreq,\n-\t\t\t\t struct folio_queue *folioq,\n+\t\t\t\t struct bvecq *bvecq,\n \t\t\t\t int slot)\n {\n \tstruct netfs_folio *finfo;\n-\tstruct folio *folio = folioq_folio(folioq, slot);\n+\tstruct folio *folio = page_folio(bvecq-\u003ebv[slot].bv_page);\n \n \tif (unlikely(folio_pos(folio) \u003c rreq-\u003eabandon_to)) {\n \t\ttrace_netfs_folio(folio, netfs_folio_trace_abandon);\n@@ -54,7 +60,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,\n \tflush_dcache_folio(folio);\n \tfolio_mark_uptodate(folio);\n \n-\tif (!test_bit(NETFS_RREQ_USE_PGPRIV2, \u0026rreq-\u003eflags)) {\n+\tif (!netfs_using_pgpriv2(rreq)) {\n \t\tfinfo = netfs_folio_info(folio);\n \t\tif (finfo) {\n \t\t\ttrace_netfs_folio(folio, netfs_folio_trace_filled_gaps);\n@@ -75,7 +81,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,\n \t\t\ttrace_netfs_folio(folio, netfs_folio_trace_read_done);\n \t\t}\n \n-\t\tfolioq_clear(folioq, slot);\n+\t\tbvecq-\u003ebv[slot].bv_page = NULL;\n \t} else {\n \t\t// TODO: Use of PG_private_2 is deprecated.\n \t\tif (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026rreq-\u003eflags))\n@@ -91,7 +97,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,\n \t\tfolio_unlock(folio);\n \t}\n \n-\tfolioq_clear(folioq, slot);\n+\tbvecq-\u003ebv[slot].bv_page = NULL;\n }\n \n /*\n@@ -100,23 +106,21 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,\n static void netfs_read_unlock_folios(struct netfs_io_request *rreq,\n \t\t\t\t unsigned int *notes)\n {\n-\tstruct folio_queue *folioq = rreq-\u003ebuffer.tail;\n+\tstruct bvecq *bvecq = rreq-\u003ecollect_cursor.bvecq;\n \tunsigned long long collected_to = rreq-\u003ecollected_to;\n-\tunsigned int slot = rreq-\u003ebuffer.first_tail_slot;\n+\tunsigned int slot = rreq-\u003ecollect_cursor.slot;\n \n \tif (rreq-\u003ecleaned_to \u003e= rreq-\u003ecollected_to)\n \t\treturn;\n \n \t// TODO: Begin decryption\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = rolling_buffer_delete_spent(\u0026rreq-\u003ebuffer);\n-\t\tif (!folioq) {\n-\t\t\trreq-\u003efront_folio_order = 0;\n-\t\t\treturn;\n-\t\t}\n-\t\tslot = 0;\n-\t}\n+\t/* We have to wait for readahead refs to have been released before we\n+\t * can unlock any folios as the ref-dropper walks i_pages and the only\n+\t * thing preventing these folios from being removed is the folio lock.\n+\t */\n+\tif (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, \u0026rreq-\u003eflags))\n+\t\tnetfs_wait_for_put_ra_refs(rreq);\n \n \tfor (;;) {\n \t\tstruct folio *folio;\n@@ -124,16 +128,30 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,\n \t\tunsigned int order;\n \t\tsize_t fsize;\n \n+\t\t/* Clean up the head bvecq segment. If we clear an entire\n+\t\t * segment, then we can get rid of it provided it's not also\n+\t\t * the tail segment being filled by the issuer.\n+\t\t */\n+\t\tif (!bvecq_acquire_slot(bvecq, slot)) {\n+\t\t\trreq-\u003ecollect_cursor.slot = slot;\n+\t\t\tif (!bvecq_delete_spent(\u0026rreq-\u003ecollect_cursor)) {\n+\t\t\t\trreq-\u003efront_folio_order = 0;\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t\tbvecq = rreq-\u003ecollect_cursor.bvecq;\n+\t\t\tslot = rreq-\u003ecollect_cursor.slot;\n+\t\t}\n+\n \t\tif (*notes \u0026 COPY_TO_CACHE)\n \t\t\tset_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026rreq-\u003eflags);\n \n-\t\tfolio = folioq_folio(folioq, slot);\n+\t\tfolio = page_folio(bvecq-\u003ebv[slot].bv_page);\n \t\tif (WARN_ONCE(!folio_test_locked(folio),\n \t\t\t \"R=%08x: folio %lx is not locked\\n\",\n \t\t\t rreq-\u003edebug_id, folio-\u003eindex))\n \t\t\ttrace_netfs_folio(folio, netfs_folio_trace_not_locked);\n \n-\t\torder = folioq_folio_order(folioq, slot);\n+\t\torder = folio_order(folio);\n \t\trreq-\u003efront_folio_order = order;\n \t\tfsize = PAGE_SIZE \u003c\u003c order;\n \t\tfpos = folio_pos(folio);\n@@ -145,33 +163,19 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,\n \t\tif (collected_to \u003c fend)\n \t\t\tbreak;\n \n-\t\tnetfs_unlock_read_folio(rreq, folioq, slot);\n+\t\tnetfs_unlock_read_folio(rreq, bvecq, slot);\n+\t\tslot++;\n \t\tWRITE_ONCE(rreq-\u003ecleaned_to, fpos + fsize);\n \t\t*notes |= MADE_PROGRESS;\n \n \t\tclear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026rreq-\u003eflags);\n \n-\t\t/* Clean up the head folioq. If we clear an entire folioq, then\n-\t\t * we can get rid of it provided it's not also the tail folioq\n-\t\t * being filled by the issuer.\n-\t\t */\n-\t\tfolioq_clear(folioq, slot);\n-\t\tslot++;\n-\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\tfolioq = rolling_buffer_delete_spent(\u0026rreq-\u003ebuffer);\n-\t\t\tif (!folioq)\n-\t\t\t\tgoto done;\n-\t\t\tslot = 0;\n-\t\t\ttrace_netfs_folioq(folioq, netfs_trace_folioq_read_progress);\n-\t\t}\n-\n \t\tif (fpos + fsize \u003e= collected_to)\n \t\t\tbreak;\n \t}\n \n-\trreq-\u003ebuffer.tail = folioq;\n-done:\n-\trreq-\u003ebuffer.first_tail_slot = slot;\n+\tbvecq_pos_move(\u0026rreq-\u003ecollect_cursor, bvecq);\n+\trreq-\u003ecollect_cursor.slot = slot;\n }\n \n /*\n@@ -199,6 +203,8 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)\n \t\tnotes = BUFFERED;\n \telse\n \t\tnotes = 0;\n+\tif (test_bit(NETFS_RREQ_ABANDON_REQ, \u0026rreq-\u003eflags))\n+\t\tnotes |= ABANDON_RREQ;\n \n \t/* Remove completed subrequests from the front of the stream and\n \t * advance the completion point. We stop when we hit something that's\n@@ -224,14 +230,43 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)\n \t\tif (netfs_check_subreq_in_progress(front))\n \t\t\tnotes |= HIT_PENDING;\n \t\tsmp_rmb(); /* Read counters after IN_PROGRESS flag. */\n+\n \t\ttransferred = READ_ONCE(front-\u003etransferred);\n+\t\tif (unlikely(transferred \u003e front-\u003elen)) {\n+\t\t\t/* Ugh... A subreq overran its allotted length. It\n+\t\t\t * may have corrupted the read buffer.\n+\t\t\t */\n+\t\t\tset_bit(NETFS_RREQ_FAILED, \u0026rreq-\u003eflags);\n+\t\t\tset_bit(NETFS_RREQ_ABANDON_REQ, \u0026rreq-\u003eflags);\n+\t\t\tnotes |= ABANDON_RREQ;\n+\t\t}\n+\n+\t\t/* Deal with an abandoned request. Wait for each subreq to\n+\t\t * complete before abandoning them.\n+\t\t */\n+\t\tif (unlikely(notes \u0026 ABANDON_RREQ)) {\n+\t\t\tif (notes \u0026 HIT_PENDING)\n+\t\t\t\tbreak;\n+\n+\t\t\t/* The subreq now belongs to us. */\n+\t\t\tstream-\u003eerror = -EIO;\n+\t\t\tstream-\u003efailed = true;\n+\t\t\trreq-\u003eabandon_to = front-\u003estart + front-\u003elen;\n+\t\t\trreq-\u003eerror = -EIO;\n+\t\t\ttransferred = 0;\n+\t\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);\n+\t\t\tnotes |= ABANDON_SREQ;\n+\t\t\tgoto sreq_complete;\n+\t\t}\n \n \t\t/* If we can now collect the next folio, do so. We don't want\n \t\t * to defer this as we have to decide whether we need to copy\n \t\t * to the cache or not, and that may differ between adjacent\n \t\t * subreqs.\n \t\t */\n-\t\tif (notes \u0026 BUFFERED) {\n+\t\tif (notes \u0026 ABANDON_SREQ) {\n+\t\t\t/* Don't collect anything. */\n+\t\t} else if (notes \u0026 BUFFERED) {\n \t\t\tsize_t fsize = PAGE_SIZE \u003c\u003c rreq-\u003efront_folio_order;\n \n \t\t\t/* Clear the tail of a short read. */\n@@ -257,8 +292,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)\n \t\t\t\ttransferred = front-\u003elen;\n \t\t\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);\n \t\t\t}\n-\t\t\tif (front-\u003estart + transferred \u003e= rreq-\u003ecleaned_to + fsize ||\n-\t\t\t test_bit(NETFS_SREQ_HIT_EOF, \u0026front-\u003eflags))\n+\t\t\tif (front-\u003estart + transferred \u003e= rreq-\u003ecleaned_to + fsize)\n \t\t\t\tnetfs_read_unlock_folios(rreq, \u0026notes);\n \t\t} else {\n \t\t\tstream-\u003ecollected_to = front-\u003estart + transferred;\n@@ -293,6 +327,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)\n \t\t\tnotes |= MADE_PROGRESS;\n \t\t}\n \n+sreq_complete:\n \t\t/* Remove if completely consumed. */\n \t\tstream-\u003esource = front-\u003esource;\n \t\tspin_lock(\u0026rreq-\u003elock);\n@@ -317,6 +352,8 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)\n \tif (!(notes \u0026 BUFFERED))\n \t\trreq-\u003ecleaned_to = rreq-\u003ecollected_to;\n \n+\tif (notes \u0026 ABANDON_RREQ)\n+\t\tgoto out;\n \tif (notes \u0026 NEED_RETRY)\n \t\tgoto need_retry;\n \tif (notes \u0026 MADE_PROGRESS) {\n@@ -348,12 +385,17 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)\n \n \tif (rreq-\u003eorigin == NETFS_UNBUFFERED_READ ||\n \t rreq-\u003eorigin == NETFS_DIO_READ) {\n-\t\tfor (i = 0; i \u003c rreq-\u003edirect_bv_count; i++) {\n-\t\t\tflush_dcache_page(rreq-\u003edirect_bv[i].bv_page);\n-\t\t\t// TODO: cifs marks pages in the destination buffer\n-\t\t\t// dirty under some circumstances after a read. Do we\n-\t\t\t// need to do that too?\n-\t\t\tset_page_dirty(rreq-\u003edirect_bv[i].bv_page);\n+\t\tfor (struct bvecq *bq = rreq-\u003ecollect_cursor.bvecq; bq; bq = bvecq_next(bq)) {\n+\t\t\tunsigned int nr_slots = bvecq_nr_slots_acquire(bq);\n+\t\t\t/* Read the slot count before the slots. */\n+\n+\t\t\tfor (i = 0; i \u003c nr_slots; i++) {\n+\t\t\t\tflush_dcache_page(bq-\u003ebv[i].bv_page);\n+\t\t\t\t// TODO: cifs marks pages in the destination buffer\n+\t\t\t\t// dirty under some circumstances after a read. Do we\n+\t\t\t\t// need to do that too?\n+\t\t\t\tset_page_dirty(bq-\u003ebv[i].bv_page);\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -372,31 +414,6 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)\n \t\tinode_dio_end(rreq-\u003einode);\n }\n \n-/*\n- * Do processing after reading a monolithic single object.\n- */\n-static void netfs_rreq_assess_single(struct netfs_io_request *rreq)\n-{\n-\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n-\n-\tif (!rreq-\u003eerror \u0026\u0026 stream-\u003esource == NETFS_DOWNLOAD_FROM_SERVER \u0026\u0026\n-\t fscache_resources_valid(\u0026rreq-\u003ecache_resources)) {\n-\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_dirty);\n-\t\tnetfs_single_mark_inode_dirty(rreq-\u003einode);\n-\t}\n-\n-\tif (rreq-\u003eiocb) {\n-\t\trreq-\u003eiocb-\u003eki_pos += rreq-\u003etransferred;\n-\t\tif (rreq-\u003eiocb-\u003eki_complete) {\n-\t\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_ki_complete);\n-\t\t\trreq-\u003eiocb-\u003eki_complete(\n-\t\t\t\trreq-\u003eiocb, rreq-\u003eerror ? rreq-\u003eerror : rreq-\u003etransferred);\n-\t\t}\n-\t}\n-\tif (rreq-\u003enetfs_ops-\u003edone)\n-\t\trreq-\u003enetfs_ops-\u003edone(rreq);\n-}\n-\n /*\n * Perform the collection of subrequests and folios.\n *\n@@ -412,9 +429,8 @@ bool netfs_read_collection(struct netfs_io_request *rreq)\n \t/* We're done when the app thread has finished posting subreqs and the\n \t * queue is empty.\n \t */\n-\tif (!test_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags))\n+\tif (!netfs_are_all_subreqs_queued(rreq))\n \t\treturn false;\n-\tsmp_rmb(); /* Read ALL_QUEUED before subreq lists. */\n \n \tif (!list_empty(\u0026stream-\u003esubrequests))\n \t\treturn false;\n@@ -432,7 +448,7 @@ bool netfs_read_collection(struct netfs_io_request *rreq)\n \t\tnetfs_rreq_assess_dio(rreq);\n \t\tbreak;\n \tcase NETFS_READ_SINGLE:\n-\t\tnetfs_rreq_assess_single(rreq);\n+\t\tWARN_ON_ONCE(1);\n \t\tbreak;\n \tdefault:\n \t\tbreak;\n@@ -444,7 +460,15 @@ bool netfs_read_collection(struct netfs_io_request *rreq)\n \n \ttrace_netfs_rreq(rreq, netfs_rreq_trace_done);\n \tnetfs_clear_subrequests(rreq);\n-\tnetfs_unlock_abandoned_read_pages(rreq);\n+\tswitch (rreq-\u003eorigin) {\n+\tcase NETFS_READAHEAD:\n+\tcase NETFS_READPAGE:\n+\tcase NETFS_READ_FOR_WRITE:\n+\t\tnetfs_unlock_abandoned_read_pages(rreq);\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n \tif (unlikely(rreq-\u003ecopy_to_cache))\n \t\tnetfs_pgpriv2_end_copy_to_cache(rreq);\n \treturn true;\n@@ -477,15 +501,15 @@ void netfs_read_collection_worker(struct work_struct *work)\n void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n-\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n-\tsize_t fsize = PAGE_SIZE \u003c\u003c rreq-\u003efront_folio_order;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tsize_t fsize = umax(PAGE_SIZE \u003c\u003c rreq-\u003efront_folio_order, 256 * 1024);\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_progress);\n \n \t/* If we are at the head of the queue, wake up the collector,\n \t * getting a ref to it if we were the ones to do so.\n \t */\n-\tif (subreq-\u003estart + subreq-\u003etransferred \u003e rreq-\u003ecleaned_to + fsize \u0026\u0026\n+\tif (subreq-\u003estart + subreq-\u003etransferred \u003e= stream-\u003ecollected_to + fsize \u0026\u0026\n \t (rreq-\u003eorigin == NETFS_READAHEAD ||\n \t rreq-\u003eorigin == NETFS_READPAGE ||\n \t rreq-\u003eorigin == NETFS_READ_FOR_WRITE) \u0026\u0026\n@@ -528,6 +552,21 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)\n \t\tbreak;\n \t}\n \n+\t/* If the subrequest read more than it was supposed to, abort\n+\t * the request with EIO as we may have clobbered other parts\n+\t * of the buffer that are already read.\n+\t */\n+\tif (subreq-\u003etransferred \u003e subreq-\u003elen) {\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_too_much);\n+\t\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n+\t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t\tsubreq-\u003eerror = -EIO;\n+\t\ttrace_netfs_failure(rreq, subreq, subreq-\u003eerror, netfs_fail_read);\n+\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_set_pause);\n+\t\tset_bit(NETFS_RREQ_PAUSE, \u0026rreq-\u003eflags);\n+\t\tgoto skip_error_checks;\n+\t}\n+\n \t/* Deal with retry requests, short reads and errors. If we retry\n \t * but don't make progress, we abandon the attempt.\n \t */\n@@ -541,6 +580,11 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)\n \t\t} else if (test_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags)) {\n \t\t\t__set_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n \t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_partial_read);\n+\t\t} else if (subreq-\u003esource == NETFS_READ_FROM_CACHE) {\n+\t\t\tnetfs_stat(\u0026netfs_n_rh_read_failed);\n+\t\t\t__set_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t\t\tsubreq-\u003eerror = -ENODATA;\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_short);\n \t\t} else {\n \t\t\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n \t\t\tsubreq-\u003eerror = -ENODATA;\n@@ -559,6 +603,8 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)\n \n \tif (unlikely(subreq-\u003eerror \u003c 0)) {\n \t\ttrace_netfs_failure(rreq, subreq, subreq-\u003eerror, netfs_fail_read);\n+\t\tif (subreq-\u003eerror == -ENOMEM)\n+\t\t\tset_bit(NETFS_RREQ_SAW_ENOMEM, \u0026rreq-\u003eflags);\n \t\tif (subreq-\u003esource == NETFS_READ_FROM_CACHE) {\n \t\t\tnetfs_stat(\u0026netfs_n_rh_read_failed);\n \t\t\t__set_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\ndiff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c\nindex a1489aa29f782a..491cedc4ed1be9 100644\n--- a/fs/netfs/read_pgpriv2.c\n+++ b/fs/netfs/read_pgpriv2.c\n@@ -13,15 +13,46 @@\n #include \u003clinux/task_io_accounting_ops.h\u003e\n #include \"internal.h\"\n \n+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs)\n+{\n+\tstruct netfs_io_request *creq = subreq-\u003erreq;\n+\tstruct netfs_io_stream *stream = \u0026creq-\u003eio_streams[1];\n+\tsize_t len;\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026stream-\u003edispatch_cursor);\n+\tlen = bvecq_slice(\u0026stream-\u003edispatch_cursor, subreq-\u003elen, max_segs,\n+\t\t\t \u0026subreq-\u003enr_segs);\n+\n+\tif (len \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = len;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n+\t}\n+\n+\t// TODO: Wait here for completion of prev subreq\n+\n+\tstream-\u003eissue_from += subreq-\u003elen;\n+\tstream-\u003ebuffered -= subreq-\u003elen;\n+\tif (stream-\u003ebuffered == 0)\n+\t\tnetfs_all_subreqs_queued(creq);\n+\treturn 0;\n+}\n+\n /*\n- * [DEPRECATED] Copy a folio to the cache with PG_private_2 set.\n+ * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. Note that the\n+ * folio won't necessarily be contiguous with the previous one as there might\n+ * be a mixture of folios read from the cache and downloaded from the server\n+ * (or just zeroed).\n */\n static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio)\n {\n \tstruct netfs_io_stream *cache = \u0026creq-\u003eio_streams[1];\n+\tstruct bvecq *queue;\n+\tunsigned int slot;\n+\tsize_t dio_size = PAGE_SIZE;\n \tsize_t fsize = folio_size(folio), flen = fsize;\n \tloff_t fpos = folio_pos(folio), i_size;\n-\tbool to_eof = false;\n \n \t_enter(\"\");\n \n@@ -41,54 +72,43 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio\n \tif (fpos + fsize \u003e creq-\u003ei_size)\n \t\tcreq-\u003ei_size = i_size;\n \n-\tif (flen \u003e i_size - fpos) {\n+\tif (flen \u003e i_size - fpos)\n \t\tflen = i_size - fpos;\n-\t\tto_eof = true;\n-\t} else if (flen == i_size - fpos) {\n-\t\tto_eof = true;\n-\t}\n+\n+\tflen = round_up(flen, dio_size);\n \n \t_debug(\"folio %zx %zx\", flen, fsize);\n \n \ttrace_netfs_folio(folio, netfs_folio_trace_store_copy);\n \n-\t/* Attach the folio to the rolling buffer. */\n-\tif (rolling_buffer_append(\u0026creq-\u003ebuffer, folio, 0) \u003c 0) {\n-\t\tclear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026creq-\u003eflags);\n-\t\treturn;\n+\t/* Institute a new bvec queue segment if the current one is full or if\n+\t * we encounter a discontiguity. The discontiguity break is important\n+\t * when it comes to bulk unlocking folios by file range.\n+\t */\n+\tqueue = creq-\u003eload_cursor.bvecq;\n+\tif (bvecq_is_full(queue) ||\n+\t (fpos != creq-\u003elast_end \u0026\u0026 creq-\u003elast_end \u003e 0 \u0026\u0026 queue-\u003enr_slots \u003e 0)) {\n+\t\tbvecq_buffer_append(\u0026creq-\u003eload_cursor, creq-\u003espare);\n+\t\tcreq-\u003espare = NULL;\n+\n+\t\tqueue = creq-\u003eload_cursor.bvecq;\n+\t\tqueue-\u003efpos = fpos;\n+\t\tif (fpos != creq-\u003elast_end)\n+\t\t\tqueue-\u003ediscontig = true;\n \t}\n \n-\tcache-\u003esubmit_extendable_to = fsize;\n-\tcache-\u003esubmit_off = 0;\n-\tcache-\u003esubmit_len = flen;\n-\n-\t/* Attach the folio to one or more subrequests. For a big folio, we\n-\t * could end up with thousands of subrequests if the wsize is small -\n-\t * but we might need to wait during the creation of subrequests for\n-\t * network resources (eg. SMB credits).\n-\t */\n-\tdo {\n-\t\tssize_t part;\n-\n-\t\tcreq-\u003ebuffer.iter.iov_offset = cache-\u003esubmit_off;\n-\n-\t\tatomic64_set(\u0026creq-\u003eissued_to, fpos + cache-\u003esubmit_off);\n-\t\tcache-\u003esubmit_extendable_to = fsize - cache-\u003esubmit_off;\n-\t\tpart = netfs_advance_write(creq, cache, fpos + cache-\u003esubmit_off,\n-\t\t\t\t\t cache-\u003esubmit_len, to_eof);\n-\t\tcache-\u003esubmit_off += part;\n-\t\tif (part \u003e cache-\u003esubmit_len)\n-\t\t\tcache-\u003esubmit_len = 0;\n-\t\telse\n-\t\t\tcache-\u003esubmit_len -= part;\n-\t} while (cache-\u003esubmit_len \u003e 0);\n-\n-\tcreq-\u003ebuffer.iter.iov_offset = 0;\n-\trolling_buffer_advance(\u0026creq-\u003ebuffer, fsize);\n-\tatomic64_set(\u0026creq-\u003eissued_to, fpos + fsize);\n-\n-\tif (flen \u003c fsize)\n-\t\tnetfs_issue_write(creq, cache);\n+\t/* Attach the folio to the rolling buffer. */\n+\tslot = queue-\u003enr_slots;\n+\tbvec_set_folio(\u0026queue-\u003ebv[slot], folio, fsize, 0);\n+\ttrace_netfs_bv_slot(queue, slot);\n+\tslot++;\n+\tbvecq_filled_to(queue, slot);\n+\tcreq-\u003eload_cursor.slot = slot;\n+\tcreq-\u003eload_cursor.offset = 0;\n+\tcreq-\u003elast_end = fpos + flen;\n+\ttrace_netfs_wback(creq, folio, 0);\n+\n+\tcache-\u003ebuffered += flen;\n }\n \n /*\n@@ -98,6 +118,7 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(\n \tstruct netfs_io_request *rreq, struct folio *folio)\n {\n \tstruct netfs_io_request *creq;\n+\tstruct netfs_io_stream *cache;\n \n \tif (!fscache_resources_valid(\u0026rreq-\u003ecache_resources))\n \t\tgoto cancel;\n@@ -107,9 +128,16 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(\n \tif (IS_ERR(creq))\n \t\tgoto cancel;\n \n-\tif (!creq-\u003eio_streams[1].avail)\n+\tcache = \u0026creq-\u003eio_streams[1];\n+\tif (!cache-\u003eavail)\n+\t\tgoto cancel_put;\n+\n+\tif (bvecq_buffer_init(\u0026creq-\u003eload_cursor, GFP_KERNEL) \u003c 0)\n \t\tgoto cancel_put;\n \n+\tbvecq_pos_set(\u0026cache-\u003edispatch_cursor, \u0026creq-\u003eload_cursor);\n+\tbvecq_pos_set(\u0026creq-\u003ecollect_cursor, \u0026creq-\u003eload_cursor);\n+\n \t__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, \u0026creq-\u003eflags);\n \ttrace_netfs_copy2cache(rreq, creq);\n \ttrace_netfs_write(creq, netfs_write_trace_copy_to_cache);\n@@ -138,30 +166,74 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo\n \tif (IS_ERR(creq))\n \t\treturn;\n \n+\tif (!creq-\u003espare) {\n+\t\tcreq-\u003espare = bvecq_alloc_one(BVECQ_STD_SLOTS, GFP_NOFS);\n+\t\tif (!creq-\u003espare) {\n+\t\t\tclear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026creq-\u003eflags);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n \ttrace_netfs_folio(folio, netfs_folio_trace_copy_to_cache);\n \tfolio_start_private_2(folio);\n \tnetfs_pgpriv2_copy_folio(creq, folio);\n }\n \n+/*\n+ * Issue all pending writes on the cache stream.\n+ */\n+static void netfs_pgpriv2_issue_stream(struct netfs_io_request *wreq,\n+\t\t\t\t struct netfs_io_stream *stream)\n+{\n+\tatomic64_set_release(\u0026stream-\u003eissued_to, wreq-\u003estart);\n+\n+\tdo {\n+\t\tstruct netfs_io_subrequest *subreq;\n+\t\tint ret;\n+\n+\t\tsubreq = netfs_alloc_write_subreq(wreq, stream);\n+\t\tif (!subreq)\n+\t\t\tbreak;\n+\n+\t\tret = stream-\u003eissue_write(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\t/* Ownership of subreq was returned to us. Punt the\n+\t\t\t * error to the collector and stop the issuance of new\n+\t\t\t * subreqs.\n+\t\t\t */\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\tstream-\u003ebuffered -= subreq-\u003elen;\n+\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n+\t\t\tbreak;\n+\t\t}\n+\t\t/* We no longer own subreq. */\n+\n+\t\tif (test_bit(NETFS_RREQ_SAW_ENOMEM, \u0026wreq-\u003eflags))\n+\t\t\tbreak;\n+\n+\t} while (stream-\u003ebuffered \u003e 0);\n+\n+\tnetfs_all_subreqs_queued(wreq);\n+}\n+\n /*\n * [DEPRECATED] End writing to the cache, flushing out any outstanding writes.\n */\n void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)\n {\n \tstruct netfs_io_request *creq = rreq-\u003ecopy_to_cache;\n+\tstruct netfs_io_stream *stream = \u0026creq-\u003eio_streams[1];\n \n \tif (IS_ERR_OR_NULL(creq))\n \t\treturn;\n \n-\tnetfs_issue_write(creq, \u0026creq-\u003eio_streams[1]);\n-\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026creq-\u003eflags);\n+\tnetfs_pgpriv2_issue_stream(creq, stream);\n \ttrace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache);\n \tif (list_empty_careful(\u0026creq-\u003eio_streams[1].subrequests))\n \t\tnetfs_wake_collector(creq);\n \n \tnetfs_put_request(creq, netfs_rreq_trace_put_return);\n-\tcreq-\u003ecopy_to_cache = NULL;\n+\trreq-\u003ecopy_to_cache = NULL;\n }\n \n /*\n@@ -170,22 +242,27 @@ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)\n */\n bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)\n {\n-\tstruct folio_queue *folioq = creq-\u003ebuffer.tail;\n+\tstruct bvecq *bq = creq-\u003ecollect_cursor.bvecq;\n \tunsigned long long collected_to = creq-\u003ecollected_to;\n-\tunsigned int slot = creq-\u003ebuffer.first_tail_slot;\n+\tunsigned int slot;\n \tbool made_progress = false;\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = rolling_buffer_delete_spent(\u0026creq-\u003ebuffer);\n-\t\tslot = 0;\n-\t}\n+\tslot = creq-\u003ecollect_cursor.slot;\n \n \tfor (;;) {\n \t\tstruct folio *folio;\n \t\tunsigned long long fpos, fend;\n \t\tsize_t fsize, flen;\n \n-\t\tfolio = folioq_folio(folioq, slot);\n+\t\tif (!bvecq_acquire_slot(bq, slot)) {\n+\t\t\tcreq-\u003ecollect_cursor.slot = slot;\n+\t\t\tif (!bvecq_delete_spent(\u0026creq-\u003ecollect_cursor))\n+\t\t\t\tgoto out;\n+\t\t\tbq = creq-\u003ecollect_cursor.bvecq;\n+\t\t\tslot = creq-\u003ecollect_cursor.slot;\n+\t\t}\n+\n+\t\tfolio = page_folio(bq-\u003ebv[slot].bv_page);\n \t\tif (WARN_ONCE(!folio_test_private_2(folio),\n \t\t\t \"R=%08x: folio %lx is not marked private_2\\n\",\n \t\t\t creq-\u003edebug_id, folio-\u003eindex))\n@@ -208,25 +285,18 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)\n \t\tcreq-\u003ecleaned_to = fpos + fsize;\n \t\tmade_progress = true;\n \n-\t\t/* Clean up the head folioq. If we clear an entire folioq, then\n-\t\t * we can get rid of it provided it's not also the tail folioq\n-\t\t * being filled by the issuer.\n+\t\t/* Clean up the head segment. If we clear an entire segment,\n+\t\t * then we can get rid of it provided it's not also the tail\n+\t\t * segment being filled by the issuer.\n \t\t */\n-\t\tfolioq_clear(folioq, slot);\n+\t\tbq-\u003ebv[slot].bv_page = NULL;\n \t\tslot++;\n-\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\tfolioq = rolling_buffer_delete_spent(\u0026creq-\u003ebuffer);\n-\t\t\tif (!folioq)\n-\t\t\t\tgoto done;\n-\t\t\tslot = 0;\n-\t\t}\n \n \t\tif (fpos + fsize \u003e= collected_to)\n \t\t\tbreak;\n \t}\n \n-\tcreq-\u003ebuffer.tail = folioq;\n-done:\n-\tcreq-\u003ebuffer.first_tail_slot = slot;\n+\tcreq-\u003ecollect_cursor.slot = slot;\n+out:\n \treturn made_progress;\n }\ndiff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c\nindex 2b42758e01ec94..2f96b6e7a1b606 100644\n--- a/fs/netfs/read_retry.c\n+++ b/fs/netfs/read_retry.c\n@@ -9,14 +9,55 @@\n #include \u003clinux/slab.h\u003e\n #include \"internal.h\"\n \n-static void netfs_reissue_read(struct netfs_io_request *rreq,\n-\t\t\t struct netfs_io_subrequest *subreq)\n+/*\n+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to\n+ * use as a bvec queue.\n+ */\n+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n {\n-\tsubreq-\u003eerror = 0;\n+\tstruct netfs_io_request *rreq = subreq-\u003erreq;\n+\tsize_t len;\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026rreq-\u003eretry_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026subreq-\u003edispatch_pos);\n+\tlen = bvecq_slice(\u0026rreq-\u003eretry_cursor, subreq-\u003elen, max_segs,\n+\t\t\t \u0026subreq-\u003enr_segs);\n+\tif (len \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = len;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n+\t}\n+\trreq-\u003eretry_buffered -= subreq-\u003elen;\n+\trreq-\u003eretry_start += subreq-\u003elen;\n+\treturn 0;\n+}\n+\n+/*\n+ * Reset the state of the subrequest and discard any buffering so that we can\n+ * retry (where this may include sending it to the server instead of the\n+ * cache).\n+ */\n+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq)\n+{\n+\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n+\n+\tif (subreq-\u003eretry_count \u003e 3) {\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_too_many_retries);\n+\t\treturn subreq-\u003eerror;\n+\t}\n+\n+\tsubreq-\u003eretry_count++;\n \t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n+\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t__clear_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n \t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n-\tnetfs_stat(\u0026netfs_n_rh_retry_read_subreq);\n-\tsubreq-\u003erreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\tbvecq_pos_unset(\u0026subreq-\u003econtent);\n+\tbvecq_pos_unset(\u0026subreq-\u003edispatch_pos);\n+\tsubreq-\u003eerror = 0;\n+\tsubreq-\u003etransferred = 0;\n+\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n+\tnetfs_stat(\u0026netfs_n_wh_retry_write_subreq);\n+\treturn 0;\n }\n \n /*\n@@ -28,6 +69,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \tstruct netfs_io_subrequest *subreq;\n \tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n \tstruct list_head *next;\n+\tint ret;\n \n \t_enter(\"R=%x\", rreq-\u003edebug_id);\n \n@@ -37,47 +79,19 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \tif (rreq-\u003enetfs_ops-\u003eretry_request)\n \t\trreq-\u003enetfs_ops-\u003eretry_request(rreq, NULL);\n \n-\t/* If there's no renegotiation to do, just resend each retryable subreq\n-\t * up to the first permanently failed one.\n-\t */\n-\tif (!rreq-\u003enetfs_ops-\u003eprepare_read \u0026\u0026\n-\t !rreq-\u003ecache_resources.ops) {\n-\t\tlist_for_each_entry(subreq, \u0026stream-\u003esubrequests, rreq_link) {\n-\t\t\tif (test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags))\n-\t\t\t\tbreak;\n-\t\t\tif (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n-\t\t\t\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n-\t\t\t\tsubreq-\u003eretry_count++;\n-\t\t\t\tnetfs_reset_iter(subreq);\n-\t\t\t\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n-\t\t\t\tnetfs_reissue_read(rreq, subreq);\n-\t\t\t}\n-\t\t}\n-\t\treturn;\n-\t}\n+\t/* Read pointer to subreq before reading subreq state. */\n+\tnext = smp_load_acquire(\u0026stream-\u003esubrequests.next);\n \n-\t/* Okay, we need to renegotiate all the download requests and flip any\n-\t * failed cache reads over to being download requests and negotiate\n-\t * those also. All fully successful subreqs have been removed from the\n-\t * list and any spare data from those has been donated.\n-\t *\n-\t * What we do is decant the list and rebuild it one subreq at a time so\n-\t * that we don't end up with donations jumping over a gap we're busy\n-\t * populating with smaller subrequests. In the event that the subreq\n-\t * we just launched finishes before we insert the next subreq, it'll\n-\t * fill in rreq-\u003eprev_donated instead.\n-\t *\n-\t * Note: Alternatively, we could split the tail subrequest right before\n-\t * we reissue it and fix up the donations under lock.\n+\t/* Renegotiate all the download requests and flip any failed cache\n+\t * reads over to being download requests and negotiate those also.\n \t */\n-\tnext = stream-\u003esubrequests.next;\n-\n \tdo {\n \t\tstruct netfs_io_subrequest *from, *to, *tmp;\n-\t\tstruct iov_iter source;\n-\t\tunsigned long long start, len;\n-\t\tsize_t part;\n-\t\tbool boundary = false, subreq_superfluous = false;\n+\t\tunsigned long long start;\n+\t\tsize_t len;\n+\t\tbool subreq_superfluous = false;\n+\n+\t\tbvecq_pos_unset(\u0026rreq-\u003eretry_cursor);\n \n \t\t/* Go through the subreqs and find the next span of contiguous\n \t\t * buffer that we then rejig (cifs, for example, needs the\n@@ -92,8 +106,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t rreq-\u003edebug_id, from-\u003edebug_index,\n \t\t from-\u003estart, from-\u003etransferred, from-\u003elen);\n \n-\t\tif (test_bit(NETFS_SREQ_FAILED, \u0026from-\u003eflags) ||\n-\t\t !test_bit(NETFS_SREQ_NEED_RETRY, \u0026from-\u003eflags)) {\n+\t\tif (!test_bit(NETFS_SREQ_NEED_RETRY, \u0026from-\u003eflags)) {\n \t\t\tsubreq = from;\n \t\t\tgoto abandon;\n \t\t}\n@@ -105,66 +118,55 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t\t\tbreak;\n \n \t\t\tsubreq = list_entry(next, struct netfs_io_subrequest, rreq_link);\n-\t\t\tif (subreq-\u003estart + subreq-\u003etransferred != start + len ||\n-\t\t\t test_bit(NETFS_SREQ_BOUNDARY, \u0026subreq-\u003eflags) ||\n+\t\t\tif (subreq-\u003estart != start + len ||\n+\t\t\t subreq-\u003etransferred \u003e 0 ||\n \t\t\t !test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags))\n \t\t\t\tbreak;\n \t\t\tto = subreq;\n \t\t\tlen += to-\u003elen;\n \t\t}\n \n-\t\t_debug(\" - range: %llx-%llx %llx\", start, start + len - 1, len);\n+\t\t_debug(\" - range: %llx-%llx %zx\", start, start + len - 1, len);\n \n \t\t/* Determine the set of buffers we're going to use. Each\n-\t\t * subreq gets a subset of a single overall contiguous buffer.\n+\t\t * subreq takes a subset of a single overall contiguous buffer.\n+\t\t */\n+\t\tbvecq_pos_transfer(\u0026rreq-\u003eretry_cursor, \u0026from-\u003edispatch_pos);\n+\t\tbvecq_pos_advance(\u0026rreq-\u003eretry_cursor, from-\u003etransferred);\n+\t\trreq-\u003eretry_start = start;\n+\t\trreq-\u003eretry_buffered = len;\n+\t\tfrom-\u003etransferred = 0;\n+\n+\t\t/* Work through the sublist. The chain of buffers we're going\n+\t\t * to fill is attached to dispatch_cursor and we need to read\n+\t\t * 'len' amount of data from 'start'.\n \t\t */\n-\t\tnetfs_reset_iter(from);\n-\t\tsource = from-\u003eio_iter;\n-\t\tsource.count = len;\n-\n-\t\t/* Work through the sublist. */\n \t\tsubreq = from;\n \t\tlist_for_each_entry_from(subreq, \u0026stream-\u003esubrequests, rreq_link) {\n-\t\t\tif (!len) {\n+\t\t\tif (rreq-\u003eretry_buffered == 0) {\n \t\t\t\tsubreq_superfluous = true;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tsubreq-\u003esource\t= NETFS_DOWNLOAD_FROM_SERVER;\n-\t\t\tsubreq-\u003estart\t= start - subreq-\u003etransferred;\n-\t\t\tsubreq-\u003elen\t= len + subreq-\u003etransferred;\n-\t\t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n-\t\t\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n-\t\t\tsubreq-\u003eretry_count++;\n-\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n+\t\t\tsubreq-\u003estart\t= rreq-\u003eretry_start;\n+\t\t\tsubreq-\u003elen\t= rreq-\u003eretry_buffered;\n \n-\t\t\t/* Renegotiate max_len (rsize) */\n-\t\t\tstream-\u003esreq_max_len = subreq-\u003elen;\n-\t\t\tif (rreq-\u003enetfs_ops-\u003eprepare_read \u0026\u0026\n-\t\t\t rreq-\u003enetfs_ops-\u003eprepare_read(subreq) \u003c 0) {\n-\t\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);\n+\t\t\tret = netfs_reset_for_read_retry(subreq);\n+\t\t\tif (ret \u003c 0) {\n \t\t\t\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n+\t\t\t\trreq-\u003eerror = ret;\n \t\t\t\tgoto abandon;\n \t\t\t}\n \n-\t\t\tpart = umin(len, stream-\u003esreq_max_len);\n-\t\t\tif (unlikely(stream-\u003esreq_max_segs))\n-\t\t\t\tpart = netfs_limit_iter(\u0026source, 0, part, stream-\u003esreq_max_segs);\n-\t\t\tsubreq-\u003elen = subreq-\u003etransferred + part;\n-\t\t\tsubreq-\u003eio_iter = source;\n-\t\t\tiov_iter_truncate(\u0026subreq-\u003eio_iter, part);\n-\t\t\tiov_iter_advance(\u0026source, part);\n-\t\t\tlen -= part;\n-\t\t\tstart += part;\n-\t\t\tif (!len) {\n-\t\t\t\tif (boundary)\n-\t\t\t\t\t__set_bit(NETFS_SREQ_BOUNDARY, \u0026subreq-\u003eflags);\n-\t\t\t} else {\n-\t\t\t\t__clear_bit(NETFS_SREQ_BOUNDARY, \u0026subreq-\u003eflags);\n+\t\t\tnetfs_stat(\u0026netfs_n_rh_download);\n+\t\t\tret = rreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\t\t\tif (ret \u003c 0) {\n+\t\t\t\tsubreq-\u003eerror = ret;\n+\t\t\t\tnetfs_read_subreq_terminated(subreq);\n+\t\t\t\tgoto abandon_after;\n \t\t\t}\n-\n-\t\t\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n-\t\t\tnetfs_reissue_read(rreq, subreq);\n+\t\t\tif (test_bit(NETFS_RREQ_SAW_ENOMEM, \u0026rreq-\u003eflags))\n+\t\t\t\tgoto abandon_after;\n \t\t\tif (subreq == to) {\n \t\t\t\tsubreq_superfluous = false;\n \t\t\t\tbreak;\n@@ -174,7 +176,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t/* If we managed to use fewer subreqs, we can discard the\n \t\t * excess; if we used the same number, then we're done.\n \t\t */\n-\t\tif (!len) {\n+\t\tif (rreq-\u003eretry_buffered == 0) {\n \t\t\tif (!subreq_superfluous)\n \t\t\t\tcontinue;\n \t\t\tlist_for_each_entry_safe_from(subreq, tmp,\n@@ -192,7 +194,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t}\n \n \t\t/* We ran out of subrequests, so we need to allocate some more\n-\t\t * and insert them after.\n+\t\t * and insert them after. They must start with being marked\n+\t\t * for retry to switch to the retry cursor.\n \t\t */\n \t\tdo {\n \t\t\tsubreq = netfs_alloc_subrequest(rreq);\n@@ -201,8 +204,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t\t\tgoto abandon_after;\n \t\t\t}\n \t\t\tsubreq-\u003esource\t\t= NETFS_DOWNLOAD_FROM_SERVER;\n-\t\t\tsubreq-\u003estart\t\t= start;\n-\t\t\tsubreq-\u003elen\t\t= len;\n+\t\t\tsubreq-\u003estart\t\t= rreq-\u003eretry_start;\n+\t\t\tsubreq-\u003elen\t\t= rreq-\u003eretry_buffered;\n \t\t\tsubreq-\u003estream_nr\t= stream-\u003estream_nr;\n \t\t\tsubreq-\u003eretry_count\t= 1;\n \n@@ -210,42 +213,32 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t\t\t\t refcount_read(\u0026subreq-\u003eref),\n \t\t\t\t\t netfs_sreq_trace_new);\n \n+\t\t\t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n+\n \t\t\tspin_lock(\u0026rreq-\u003elock);\n+\t\t\t/* Write IN_PROGRESS before pointer to new subreq */\n+\t\t\tsmp_wmb();\n \t\t\tlist_add(\u0026subreq-\u003erreq_link, \u0026to-\u003erreq_link);\n \t\t\tspin_unlock(\u0026rreq-\u003elock);\n \t\t\tto = subreq;\n \t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n \n-\t\t\tstream-\u003esreq_max_len\t= umin(len, rreq-\u003ersize);\n-\t\t\tstream-\u003esreq_max_segs\t= 0;\n-\t\t\tif (unlikely(stream-\u003esreq_max_segs))\n-\t\t\t\tpart = netfs_limit_iter(\u0026source, 0, part, stream-\u003esreq_max_segs);\n-\n \t\t\tnetfs_stat(\u0026netfs_n_rh_download);\n-\t\t\tif (rreq-\u003enetfs_ops-\u003eprepare_read(subreq) \u003c 0) {\n-\t\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);\n-\t\t\t\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n-\t\t\t\tgoto abandon;\n-\t\t\t}\n-\n-\t\t\tpart = umin(len, stream-\u003esreq_max_len);\n-\t\t\tsubreq-\u003elen = subreq-\u003etransferred + part;\n-\t\t\tsubreq-\u003eio_iter = source;\n-\t\t\tiov_iter_truncate(\u0026subreq-\u003eio_iter, part);\n-\t\t\tiov_iter_advance(\u0026source, part);\n-\n-\t\t\tlen -= part;\n-\t\t\tstart += part;\n-\t\t\tif (!len \u0026\u0026 boundary) {\n-\t\t\t\t__set_bit(NETFS_SREQ_BOUNDARY, \u0026to-\u003eflags);\n-\t\t\t\tboundary = false;\n+\t\t\tret = rreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\t\t\tif (ret \u003c 0) {\n+\t\t\t\tsubreq-\u003eerror = ret;\n+\t\t\t\tnetfs_read_subreq_terminated(subreq);\n+\t\t\t\tgoto abandon_after;\n \t\t\t}\n+\t\t\tif (test_bit(NETFS_RREQ_SAW_ENOMEM, \u0026rreq-\u003eflags))\n+\t\t\t\tgoto abandon_after;\n \n-\t\t\tnetfs_reissue_read(rreq, subreq);\n-\t\t} while (len);\n+\t\t} while (rreq-\u003eretry_buffered \u003e 0);\n \n \t} while (!list_is_head(next, \u0026stream-\u003esubrequests));\n \n+out:\n+\tbvecq_pos_unset(\u0026rreq-\u003eretry_cursor);\n \treturn;\n \n \t/* If we hit an error, fail all remaining incomplete subrequests */\n@@ -262,6 +255,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)\n \t\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n \t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n \t}\n+\tgoto out;\n }\n \n /*\n@@ -272,6 +266,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)\n \tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n \n \tnetfs_stat(\u0026netfs_n_rh_retry_read_req);\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_retry_begin);\n \n \t/* Wait for all outstanding I/O to quiesce before performing retries as\n \t * we may need to renegotiate the I/O sizes.\n@@ -282,6 +277,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)\n \n \ttrace_netfs_rreq(rreq, netfs_rreq_trace_resubmit);\n \tnetfs_retry_read_subrequests(rreq);\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_retry_end);\n }\n \n /*\n@@ -290,23 +286,25 @@ void netfs_retry_reads(struct netfs_io_request *rreq)\n */\n void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)\n {\n-\tstruct folio_queue *p;\n-\n-\tfor (p = rreq-\u003ebuffer.tail; p; p = p-\u003enext) {\n-\t\tfor (int slot = 0; slot \u003c folioq_count(p); slot++) {\n-\t\t\tstruct folio *folio = folioq_folio(p, slot);\n-\n-\t\t\tif (folio \u0026\u0026 !folioq_is_marked2(p, slot)) {\n-\t\t\t\tif (folio == rreq-\u003eno_unlock_folio \u0026\u0026\n-\t\t\t\t test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO,\n-\t\t\t\t\t \u0026rreq-\u003eflags)) {\n-\t\t\t\t\t_debug(\"no unlock\");\n-\t\t\t\t} else {\n-\t\t\t\t\ttrace_netfs_folio(folio,\n-\t\t\t\t\t\tnetfs_folio_trace_abandon);\n-\t\t\t\t\tfolio_unlock(folio);\n-\t\t\t\t}\n+\tstruct bvecq *p;\n+\n+\tfor (p = rreq-\u003ecollect_cursor.bvecq; p; p = bvecq_next(p)) {\n+\t\tunsigned int nr_slots = bvecq_nr_slots_acquire(p);\n+\n+\t\tfor (int slot = 0; slot \u003c nr_slots; slot++) {\n+\t\t\tif (!p-\u003ebv[slot].bv_page)\n+\t\t\t\tcontinue;\n+\n+\t\t\tstruct folio *folio = page_folio(p-\u003ebv[slot].bv_page);\n+\n+\t\t\tif (folio == rreq-\u003eno_unlock_folio \u0026\u0026\n+\t\t\t test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO, \u0026rreq-\u003eflags)) {\n+\t\t\t\t_debug(\"no unlock\");\n+\t\t\t\tcontinue;\n \t\t\t}\n+\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_abandon);\n+\t\t\tfolio_unlock(folio);\n+\t\t\tp-\u003ebv[slot].bv_page = NULL;\n \t\t}\n \t}\n }\ndiff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c\nindex 8833550d2eb608..447b2e1e050c28 100644\n--- a/fs/netfs/read_single.c\n+++ b/fs/netfs/read_single.c\n@@ -16,6 +16,22 @@\n #include \u003clinux/netfs.h\u003e\n #include \"internal.h\"\n \n+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs)\n+{\n+\tstruct netfs_io_request *rreq = subreq-\u003erreq;\n+\tstruct netfs_io_stream *stream = \u0026rreq-\u003eio_streams[0];\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026rreq-\u003eload_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026subreq-\u003edispatch_pos);\n+\n+\tstream-\u003ebuffered = 0;\n+\tstream-\u003eissue_from += subreq-\u003elen;\n+\trreq-\u003esubmitted = stream-\u003eissue_from;\n+\tnetfs_all_subreqs_queued(rreq);\n+\treturn 0;\n+}\n+\n /**\n * netfs_single_mark_inode_dirty - Mark a single, monolithic object inode dirty\n * @inode: The inode to mark\n@@ -58,87 +74,99 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n\n \treturn fscache_begin_read_operation(\u0026rreq-\u003ecache_resources, netfs_i_cookie(ctx));\n }\n \n-static void netfs_single_cache_prepare_read(struct netfs_io_request *rreq,\n-\t\t\t\t\t struct netfs_io_subrequest *subreq)\n-{\n-\tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n-\n-\tif (!cres-\u003eops) {\n-\t\tsubreq-\u003esource = NETFS_DOWNLOAD_FROM_SERVER;\n-\t\treturn;\n-\t}\n-\tsubreq-\u003esource = cres-\u003eops-\u003eprepare_read(subreq, rreq-\u003ei_size);\n-\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n-\n-}\n-\n-static void netfs_single_read_cache(struct netfs_io_request *rreq,\n-\t\t\t\t struct netfs_io_subrequest *subreq)\n-{\n-\tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n-\n-\t_enter(\"R=%08x[%x]\", rreq-\u003edebug_id, subreq-\u003edebug_index);\n-\tnetfs_stat(\u0026netfs_n_rh_read);\n-\tcres-\u003eops-\u003eread(cres, subreq-\u003estart, \u0026subreq-\u003eio_iter, NETFS_READ_HOLE_FAIL,\n-\t\t\tnetfs_cache_read_terminated, subreq);\n-}\n-\n /*\n * Perform a read to a buffer from the cache or the server. Only a single\n * subreq is permitted as the object must be fetched in a single transaction.\n */\n static int netfs_single_dispatch_read(struct netfs_io_request *rreq)\n {\n+\tstruct fscache_occupancy occ = {\n+\t\t.query_from\t= 0,\n+\t\t.query_to\t= rreq-\u003elen,\n+\t\t.cached_from[0]\t= ULLONG_MAX,\n+\t\t.cached_to[0]\t= ULLONG_MAX,\n+\t\t.cached_from[1]\t= ULLONG_MAX,\n+\t\t.cached_to[1]\t= ULLONG_MAX,\n+\t};\n \tstruct netfs_io_subrequest *subreq;\n-\tint ret = 0;\n+\tint ret;\n+\n+\tnetfs_read_query_cache(rreq, \u0026occ);\n \n-\tsubreq = netfs_alloc_subrequest(rreq);\n+\tsubreq = netfs_alloc_read_subrequest(rreq);\n \tif (!subreq)\n \t\treturn -ENOMEM;\n \n-\tsubreq-\u003esource\t= NETFS_SOURCE_UNKNOWN;\n \tsubreq-\u003estart\t= 0;\n \tsubreq-\u003elen\t= rreq-\u003elen;\n-\tsubreq-\u003eio_iter\t= rreq-\u003ebuffer.iter;\n \n-\tnetfs_queue_read(rreq, subreq);\n+\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n \n-\tnetfs_single_cache_prepare_read(rreq, subreq);\n-\tswitch (subreq-\u003esource) {\n-\tcase NETFS_DOWNLOAD_FROM_SERVER:\n+\t/* Try to use the cache if the cache content matches the size of the\n+\t * remote file.\n+\t */\n+\tif (occ.cached_from[0] == 0 \u0026\u0026\n+\t occ.cached_to[0] \u003e= rreq-\u003elen) {\n+\t\tstruct netfs_cache_resources *cres = \u0026rreq-\u003ecache_resources;\n+\n+\t\tsubreq-\u003esource = NETFS_READ_FROM_CACHE;\n+\t\tnetfs_stat(\u0026netfs_n_rh_read);\n+\t\tret = cres-\u003eops-\u003eissue_read(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\tsubreq-\u003eerror = ret;\n+\t\t\tnetfs_read_subreq_terminated(subreq);\n+\t\t}\n+\n+\t\tret = netfs_wait_for_in_progress_subreq(rreq, subreq);\n+\t\tif (ret == 0)\n+\t\t\tgoto success;\n+\t\tif (ret == -ENOMEM)\n+\t\t\tgoto cancel;\n+\n+\t\t/* Didn't manage to retrieve from the cache, so toss it to the\n+\t\t * server instead.\n+\t\t */\n+\t\tif (netfs_reset_for_read_retry(subreq) \u003c 0)\n+\t\t\tgoto cancel;\n+\t}\n+\n+\t__set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026rreq-\u003eflags);\n+\n+\t/* Try to send it to the cache. */\n+\tfor (;;) {\n+\t\tsubreq-\u003esource = NETFS_DOWNLOAD_FROM_SERVER;\n \t\tnetfs_stat(\u0026netfs_n_rh_download);\n-\t\tif (rreq-\u003enetfs_ops-\u003eprepare_read) {\n-\t\t\tret = rreq-\u003enetfs_ops-\u003eprepare_read(subreq);\n-\t\t\tif (ret \u003c 0)\n-\t\t\t\tgoto cancel;\n+\t\tret = rreq-\u003enetfs_ops-\u003eissue_read(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\tsubreq-\u003eerror = ret;\n+\t\t\tnetfs_read_subreq_terminated(subreq);\n \t\t}\n \n-\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n-\t\trreq-\u003enetfs_ops-\u003eissue_read(subreq);\n-\t\trreq-\u003esubmitted += subreq-\u003elen;\n-\t\tbreak;\n-\tcase NETFS_READ_FROM_CACHE:\n-\t\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\t\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n-\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n-\t\tnetfs_single_read_cache(rreq, subreq);\n-\t\trreq-\u003esubmitted += subreq-\u003elen;\n-\t\tret = 0;\n-\t\tbreak;\n-\tdefault:\n-\t\tpr_warn(\"Unexpected single-read source %u\\n\", subreq-\u003esource);\n-\t\tWARN_ON_ONCE(true);\n-\t\tret = -EIO;\n-\t\tgoto cancel;\n+\t\tret = netfs_wait_for_in_progress_subreq(rreq, subreq);\n+\t\tif (ret == 0)\n+\t\t\tgoto success;\n+\t\tif (ret == -ENOMEM)\n+\t\t\tgoto cancel;\n+\t\tif (ret != -EAGAIN)\n+\t\t\tgoto failed;\n+\t\tif (netfs_reset_for_read_retry(subreq) \u003c 0)\n+\t\t\tgoto cancel;\n \t}\n \n-\treturn ret;\n+success:\n+\trreq-\u003etransferred = subreq-\u003etransferred;\n+\tlist_del_init(\u0026subreq-\u003erreq_link);\n+\tnetfs_put_subrequest(subreq, netfs_sreq_trace_put_consumed);\n+\treturn 0;\n cancel:\n-\tnetfs_cancel_read(subreq, ret);\n-\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026rreq-\u003eflags);\n-\tnetfs_wake_collector(rreq);\n+\trreq-\u003eerror = ret;\n+\tlist_del_init(\u0026subreq-\u003erreq_link);\n+\tnetfs_put_subrequest(subreq, netfs_sreq_trace_put_cancel);\n+\treturn ret;\n+failed:\n+\trreq-\u003eerror = ret;\n+\tlist_del_init(\u0026subreq-\u003erreq_link);\n+\tnetfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);\n \treturn ret;\n }\n \n@@ -170,6 +198,14 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite\n \tif (IS_ERR(rreq))\n \t\treturn PTR_ERR(rreq);\n \n+\tret = netfs_extract_iter(iter, rreq-\u003elen, INT_MAX, 0, \u0026rreq-\u003eload_cursor.bvecq, 0);\n+\tif (ret \u003c 0)\n+\t\tgoto cleanup_free;\n+\tif (ret \u003c rreq-\u003elen) {\n+\t\tret = -EIO;\n+\t\tgoto cleanup_free;\n+\t}\n+\n \tret = netfs_single_begin_cache_read(rreq, ictx);\n \tif (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)\n \t\tgoto cleanup_free;\n@@ -177,10 +213,29 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite\n \tnetfs_stat(\u0026netfs_n_rh_read_single);\n \ttrace_netfs_read(rreq, 0, rreq-\u003elen, netfs_read_trace_read_single);\n \n-\trreq-\u003ebuffer.iter = *iter;\n-\tnetfs_single_dispatch_read(rreq);\n+\tret = netfs_single_dispatch_read(rreq);\n+\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_complete);\n+\tif (ret == 0) {\n+\t\ttask_io_account_read(rreq-\u003etransferred);\n+\n+\t\tif (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, \u0026rreq-\u003eflags) \u0026\u0026\n+\t\t fscache_resources_valid(\u0026rreq-\u003ecache_resources)) {\n+\t\t\ttrace_netfs_rreq(rreq, netfs_rreq_trace_dirty);\n+\t\t\tnetfs_single_mark_inode_dirty(rreq-\u003einode);\n+\t\t}\n+\t\tret = rreq-\u003etransferred;\n+\t}\n+\n+\tif (rreq-\u003enetfs_ops-\u003edone)\n+\t\trreq-\u003enetfs_ops-\u003edone(rreq);\n+\n+\tnetfs_wake_rreq_flag(rreq, NETFS_RREQ_IN_PROGRESS, netfs_rreq_trace_wake_ip);\n+\t/* As we cleared NETFS_RREQ_IN_PROGRESS, we acquired its ref. */\n+\tnetfs_put_request(rreq, netfs_rreq_trace_put_work_ip);\n+\n+\ttrace_netfs_rreq(rreq, netfs_rreq_trace_done);\n \n-\tret = netfs_wait_for_read(rreq);\n \tnetfs_put_request(rreq, netfs_rreq_trace_put_return);\n \treturn ret;\n \ndiff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c\ndeleted file mode 100644\nindex a17fbf9853a443..00000000000000\n--- a/fs/netfs/rolling_buffer.c\n+++ /dev/null\n@@ -1,222 +0,0 @@\n-// SPDX-License-Identifier: GPL-2.0-or-later\n-/* Rolling buffer helpers\n- *\n- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.\n- * Written by David Howells (dhowells@redhat.com)\n- */\n-\n-#include \u003clinux/bitops.h\u003e\n-#include \u003clinux/pagemap.h\u003e\n-#include \u003clinux/rolling_buffer.h\u003e\n-#include \u003clinux/slab.h\u003e\n-#include \"internal.h\"\n-\n-static atomic_t debug_ids;\n-\n-/**\n- * netfs_folioq_alloc - Allocate a folio_queue struct\n- * @rreq_id: Associated debugging ID for tracing purposes\n- * @gfp: Allocation constraints\n- * @trace: Trace tag to indicate the purpose of the allocation\n- *\n- * Allocate, initialise and account the folio_queue struct and log a trace line\n- * to mark the allocation.\n- */\n-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,\n-\t\t\t\t unsigned int /*enum netfs_folioq_trace*/ trace)\n-{\n-\tstruct folio_queue *fq;\n-\n-\tfq = kmalloc_obj(*fq, gfp);\n-\tif (fq) {\n-\t\tnetfs_stat(\u0026netfs_n_folioq);\n-\t\tfolioq_init(fq, rreq_id);\n-\t\tfq-\u003edebug_id = atomic_inc_return(\u0026debug_ids);\n-\t\ttrace_netfs_folioq(fq, trace);\n-\t}\n-\treturn fq;\n-}\n-EXPORT_SYMBOL(netfs_folioq_alloc);\n-\n-/**\n- * netfs_folioq_free - Free a folio_queue struct\n- * @folioq: The object to free\n- * @trace: Trace tag to indicate which free\n- *\n- * Free and unaccount the folio_queue struct.\n- */\n-void netfs_folioq_free(struct folio_queue *folioq,\n-\t\t unsigned int /*enum netfs_trace_folioq*/ trace)\n-{\n-\ttrace_netfs_folioq(folioq, trace);\n-\tnetfs_stat_d(\u0026netfs_n_folioq);\n-\tkfree(folioq);\n-}\n-EXPORT_SYMBOL(netfs_folioq_free);\n-\n-/*\n- * Initialise a rolling buffer. We allocate an empty folio queue struct to so\n- * that the pointers can be independently driven by the producer and the\n- * consumer.\n- */\n-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,\n-\t\t\tunsigned int direction)\n-{\n-\tstruct folio_queue *fq;\n-\n-\tfq = netfs_folioq_alloc(rreq_id, GFP_NOFS, netfs_trace_folioq_rollbuf_init);\n-\tif (!fq)\n-\t\treturn -ENOMEM;\n-\n-\troll-\u003ehead = fq;\n-\troll-\u003etail = fq;\n-\tiov_iter_folio_queue(\u0026roll-\u003eiter, direction, fq, 0, 0, 0);\n-\treturn 0;\n-}\n-\n-/*\n- * Add another folio_queue to a rolling buffer if there's no space left.\n- */\n-int rolling_buffer_make_space(struct rolling_buffer *roll)\n-{\n-\tstruct folio_queue *fq, *head = roll-\u003ehead;\n-\n-\tif (!folioq_full(head))\n-\t\treturn 0;\n-\n-\tfq = netfs_folioq_alloc(head-\u003erreq_id, GFP_NOFS, netfs_trace_folioq_make_space);\n-\tif (!fq)\n-\t\treturn -ENOMEM;\n-\tfq-\u003eprev = head;\n-\n-\troll-\u003ehead = fq;\n-\tif (folioq_full(head)) {\n-\t\t/* Make sure we don't leave the master iterator pointing to a\n-\t\t * block that might get immediately consumed.\n-\t\t */\n-\t\tif (roll-\u003eiter.folioq == head \u0026\u0026\n-\t\t roll-\u003eiter.folioq_slot == folioq_nr_slots(head)) {\n-\t\t\troll-\u003eiter.folioq = fq;\n-\t\t\troll-\u003eiter.folioq_slot = 0;\n-\t\t}\n-\t}\n-\n-\t/* Make sure the initialisation is stored before the next pointer.\n-\t *\n-\t * [!] NOTE: After we set head-\u003enext, the consumer is at liberty to\n-\t * immediately delete the old head.\n-\t */\n-\tsmp_store_release(\u0026head-\u003enext, fq);\n-\treturn 0;\n-}\n-\n-/*\n- * Decant the list of folios to read into a rolling buffer.\n- */\n-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,\n-\t\t\t\t struct readahead_control *ractl,\n-\t\t\t\t struct folio_batch *put_batch)\n-{\n-\tstruct folio_queue *fq;\n-\tstruct page **vec;\n-\tint nr, ix, to;\n-\tssize_t size = 0;\n-\n-\tif (rolling_buffer_make_space(roll) \u003c 0)\n-\t\treturn -ENOMEM;\n-\n-\tfq = roll-\u003ehead;\n-\tvec = (struct page **)fq-\u003evec.folios;\n-\tnr = __readahead_batch(ractl, vec + folio_batch_count(\u0026fq-\u003evec),\n-\t\t\t folio_batch_space(\u0026fq-\u003evec));\n-\tix = fq-\u003evec.nr;\n-\tto = ix + nr;\n-\tfq-\u003evec.nr = to;\n-\tfor (; ix \u003c to; ix++) {\n-\t\tstruct folio *folio = folioq_folio(fq, ix);\n-\t\tunsigned int order = folio_order(folio);\n-\n-\t\tfq-\u003eorders[ix] = order;\n-\t\tsize += PAGE_SIZE \u003c\u003c order;\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_read);\n-\t\tif (!folio_batch_add(put_batch, folio))\n-\t\t\tfolio_batch_release(put_batch);\n-\t}\n-\tWRITE_ONCE(roll-\u003eiter.count, roll-\u003eiter.count + size);\n-\n-\t/* Store the counter after setting the slot. */\n-\tsmp_store_release(\u0026roll-\u003enext_head_slot, to);\n-\treturn size;\n-}\n-\n-/*\n- * Append a folio to the rolling buffer.\n- */\n-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,\n-\t\t\t unsigned int flags)\n-{\n-\tssize_t size = folio_size(folio);\n-\tint slot;\n-\n-\tif (rolling_buffer_make_space(roll) \u003c 0)\n-\t\treturn -ENOMEM;\n-\n-\tslot = folioq_append(roll-\u003ehead, folio);\n-\tif (flags \u0026 ROLLBUF_MARK_1)\n-\t\tfolioq_mark(roll-\u003ehead, slot);\n-\tif (flags \u0026 ROLLBUF_MARK_2)\n-\t\tfolioq_mark2(roll-\u003ehead, slot);\n-\n-\tWRITE_ONCE(roll-\u003eiter.count, roll-\u003eiter.count + size);\n-\n-\t/* Store the counter after setting the slot. */\n-\tsmp_store_release(\u0026roll-\u003enext_head_slot, slot);\n-\treturn size;\n-}\n-\n-/*\n- * Delete a spent buffer from a rolling queue and return the next in line. We\n- * don't return the last buffer to keep the pointers independent, but return\n- * NULL instead.\n- */\n-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll)\n-{\n-\tstruct folio_queue *spent = roll-\u003etail, *next = READ_ONCE(spent-\u003enext);\n-\n-\tif (!next)\n-\t\treturn NULL;\n-\tnext-\u003eprev = NULL;\n-\tnetfs_folioq_free(spent, netfs_trace_folioq_delete);\n-\troll-\u003etail = next;\n-\treturn next;\n-}\n-\n-/*\n- * Clear out a rolling queue. Folios that have mark 1 set are put.\n- */\n-void rolling_buffer_clear(struct rolling_buffer *roll)\n-{\n-\tstruct folio_batch fbatch;\n-\tstruct folio_queue *p;\n-\n-\tfolio_batch_init(\u0026fbatch);\n-\n-\twhile ((p = roll-\u003etail)) {\n-\t\troll-\u003etail = p-\u003enext;\n-\t\tfor (int slot = 0; slot \u003c folioq_count(p); slot++) {\n-\t\t\tstruct folio *folio = folioq_folio(p, slot);\n-\n-\t\t\tif (!folio)\n-\t\t\t\tcontinue;\n-\t\t\tif (folioq_is_marked(p, slot)) {\n-\t\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_put);\n-\t\t\t\tif (!folio_batch_add(\u0026fbatch, folio))\n-\t\t\t\t\tfolio_batch_release(\u0026fbatch);\n-\t\t\t}\n-\t\t}\n-\n-\t\tnetfs_folioq_free(p, netfs_trace_folioq_clear);\n-\t}\n-\n-\tfolio_batch_release(\u0026fbatch);\n-}\ndiff --git a/fs/netfs/stats.c b/fs/netfs/stats.c\nindex ab6b916addc443..0ba6ce9295b257 100644\n--- a/fs/netfs/stats.c\n+++ b/fs/netfs/stats.c\n@@ -32,7 +32,6 @@ atomic_t netfs_n_rh_write_zskip;\n atomic_t netfs_n_rh_retry_read_req;\n atomic_t netfs_n_rh_retry_read_subreq;\n atomic_t netfs_n_wh_buffered_write;\n-atomic_t netfs_n_wh_writethrough;\n atomic_t netfs_n_wh_dio_write;\n atomic_t netfs_n_wh_writepages;\n atomic_t netfs_n_wh_copy_to_cache;\n@@ -47,7 +46,7 @@ atomic_t netfs_n_wh_retry_write_req;\n atomic_t netfs_n_wh_retry_write_subreq;\n atomic_t netfs_n_wb_lock_skip;\n atomic_t netfs_n_wb_lock_wait;\n-atomic_t netfs_n_folioq;\n+atomic_t netfs_n_bvecq;\n \n int netfs_stats_show(struct seq_file *m, void *v)\n {\n@@ -58,9 +57,8 @@ int netfs_stats_show(struct seq_file *m, void *v)\n \t\t atomic_read(\u0026netfs_n_rh_read_single),\n \t\t atomic_read(\u0026netfs_n_rh_write_begin),\n \t\t atomic_read(\u0026netfs_n_rh_write_zskip));\n-\tseq_printf(m, \"Writes : BW=%u WT=%u DW=%u WP=%u 2C=%u\\n\",\n+\tseq_printf(m, \"Writes : BW=%u DW=%u WP=%u 2C=%u\\n\",\n \t\t atomic_read(\u0026netfs_n_wh_buffered_write),\n-\t\t atomic_read(\u0026netfs_n_wh_writethrough),\n \t\t atomic_read(\u0026netfs_n_wh_dio_write),\n \t\t atomic_read(\u0026netfs_n_wh_writepages),\n \t\t atomic_read(\u0026netfs_n_wh_copy_to_cache));\n@@ -90,10 +88,10 @@ int netfs_stats_show(struct seq_file *m, void *v)\n \t\t atomic_read(\u0026netfs_n_rh_retry_read_subreq),\n \t\t atomic_read(\u0026netfs_n_wh_retry_write_req),\n \t\t atomic_read(\u0026netfs_n_wh_retry_write_subreq));\n-\tseq_printf(m, \"Objs : rr=%u sr=%u foq=%u wsc=%u\\n\",\n+\tseq_printf(m, \"Objs : rr=%u sr=%u bq=%u wsc=%u\\n\",\n \t\t atomic_read(\u0026netfs_n_rh_rreq),\n \t\t atomic_read(\u0026netfs_n_rh_sreq),\n-\t\t atomic_read(\u0026netfs_n_folioq),\n+\t\t atomic_read(\u0026netfs_n_bvecq),\n \t\t atomic_read(\u0026netfs_n_wh_wstream_conflict));\n \tseq_printf(m, \"WbLock : skip=%u wait=%u\\n\",\n \t\t atomic_read(\u0026netfs_n_wb_lock_skip),\ndiff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c\nindex 210eb8f3958d77..33302cdb08f0f0 100644\n--- a/fs/netfs/write_collect.c\n+++ b/fs/netfs/write_collect.c\n@@ -28,8 +28,8 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)\n \t rreq-\u003eorigin, rreq-\u003eerror);\n \tpr_err(\" st=%llx tsl=%zx/%llx/%llx\\n\",\n \t rreq-\u003estart, rreq-\u003etransferred, rreq-\u003esubmitted, rreq-\u003elen);\n-\tpr_err(\" cci=%llx/%llx/%llx\\n\",\n-\t rreq-\u003ecleaned_to, rreq-\u003ecollected_to, atomic64_read(\u0026rreq-\u003eissued_to));\n+\tpr_err(\" cci=%llx/%llx\\n\",\n+\t rreq-\u003ecleaned_to, rreq-\u003ecollected_to);\n \tpr_err(\" iw=%pSR\\n\", rreq-\u003enetfs_ops-\u003eissue_write);\n \tfor (int i = 0; i \u003c NR_IO_STREAMS; i++) {\n \t\tconst struct netfs_io_subrequest *sreq;\n@@ -38,8 +38,9 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)\n \t\tpr_err(\" str[%x] s=%x e=%d acnf=%u,%u,%u,%u\\n\",\n \t\t s-\u003estream_nr, s-\u003esource, s-\u003eerror,\n \t\t s-\u003eavail, s-\u003eactive, s-\u003eneed_retry, s-\u003efailed);\n-\t\tpr_err(\" str[%x] ct=%llx t=%zx\\n\",\n-\t\t s-\u003estream_nr, s-\u003ecollected_to, s-\u003etransferred);\n+\t\tpr_err(\" str[%x] it=%llx ct=%llx t=%zx\\n\",\n+\t\t s-\u003estream_nr, atomic64_read(\u0026s-\u003eissued_to),\n+\t\t s-\u003ecollected_to, s-\u003etransferred);\n \t\tlist_for_each_entry(sreq, \u0026s-\u003esubrequests, rreq_link) {\n \t\t\tpr_err(\" sreq[%x:%x] sc=%u s=%llx t=%zx/%zx r=%d f=%lx\\n\",\n \t\t\t sreq-\u003estream_nr, sreq-\u003edebug_index, sreq-\u003esource,\n@@ -56,7 +57,7 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)\n */\n int netfs_folio_written_back(struct folio *folio)\n {\n-\tenum netfs_folio_trace why = netfs_folio_trace_clear;\n+\tenum netfs_folio_trace why = netfs_folio_trace_endwb;\n \tstruct inode *inode = folio_inode(folio);\n \tstruct netfs_inode *ictx = netfs_inode(inode);\n \tstruct netfs_folio *finfo;\n@@ -79,13 +80,13 @@ int netfs_folio_written_back(struct folio *folio)\n \t\tgroup = finfo-\u003enetfs_group;\n \t\tgcount++;\n \t\tkfree(finfo);\n-\t\twhy = netfs_folio_trace_clear_s;\n+\t\twhy = netfs_folio_trace_endwb_s;\n \t\tgoto end_wb;\n \t}\n \n \tif ((group = netfs_folio_group(folio))) {\n \t\tif (group == NETFS_FOLIO_COPY_TO_CACHE) {\n-\t\t\twhy = netfs_folio_trace_clear_cc;\n+\t\t\twhy = netfs_folio_trace_endwb_cc;\n \t\t\tfolio_detach_private(folio);\n \t\t\tgoto end_wb;\n \t\t}\n@@ -98,7 +99,7 @@ int netfs_folio_written_back(struct folio *folio)\n \t\tif (!folio_test_dirty(folio)) {\n \t\t\tfolio_detach_private(folio);\n \t\t\tgcount++;\n-\t\t\twhy = netfs_folio_trace_clear_g;\n+\t\t\twhy = netfs_folio_trace_endwb_g;\n \t\t}\n \t}\n \n@@ -114,12 +115,12 @@ int netfs_folio_written_back(struct folio *folio)\n static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,\n \t\t\t\t\t unsigned int *notes)\n {\n-\tstruct folio_queue *folioq = wreq-\u003ebuffer.tail;\n+\tstruct bvecq *bvecq = wreq-\u003ecollect_cursor.bvecq;\n \tunsigned long long collected_to = wreq-\u003ecollected_to;\n-\tunsigned int slot = wreq-\u003ebuffer.first_tail_slot;\n+\tunsigned int slot = wreq-\u003ecollect_cursor.slot;\n \n-\tif (WARN_ON_ONCE(!folioq)) {\n-\t\tpr_err(\"[!] Writeback unlock found empty rolling buffer!\\n\");\n+\tif (WARN_ON_ONCE(!bvecq)) {\n+\t\tpr_err(\"[!] Writeback unlock found empty buffer!\\n\");\n \t\tnetfs_dump_request(wreq);\n \t\treturn;\n \t}\n@@ -130,20 +131,34 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,\n \t\treturn;\n \t}\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = rolling_buffer_delete_spent(\u0026wreq-\u003ebuffer);\n-\t\tif (!folioq)\n-\t\t\treturn;\n-\t\tslot = 0;\n-\t}\n-\n \tfor (;;) {\n \t\tstruct folio *folio;\n \t\tstruct netfs_folio *finfo;\n \t\tunsigned long long fpos, fend;\n \t\tsize_t fsize, flen;\n \n-\t\tfolio = folioq_folio(folioq, slot);\n+\t\t/* Try to clean up the head of the queue if it appears to be\n+\t\t * used up, but we need to be very careful - the cleanup can\n+\t\t * catch the dispatcher, which could lead to us having nothing\n+\t\t * left in the queue, causing the front and back pointers to\n+\t\t * end up on different tracks. To avoid this, we must always\n+\t\t * keep at least one segment in the queue.\n+\t\t */\n+\t\tif (!bvecq_acquire_slot(bvecq, slot)) {\n+\t\t\twreq-\u003ecollect_cursor.slot = slot;\n+\t\t\tif (!bvecq_delete_spent(\u0026wreq-\u003ecollect_cursor))\n+\t\t\t\treturn;\n+\t\t\tbvecq = wreq-\u003ecollect_cursor.bvecq;\n+\t\t\tslot = wreq-\u003ecollect_cursor.slot;\n+\t\t}\n+\n+\t\tif (!bvecq-\u003ebv[slot].bv_page) {\n+\t\t\tWARN_ONCE(1, \"R=%08x slot already cleared?\\n\", wreq-\u003edebug_id);\n+\t\t\tfsize = bvecq-\u003ebv[slot].bv_len;\n+\t\t\tgoto skip;\n+\t\t}\n+\n+\t\tfolio = page_folio(bvecq-\u003ebv[slot].bv_page);\n \t\tif (WARN_ONCE(!folio_test_writeback(folio),\n \t\t\t \"R=%08x: folio %lx is not under writeback\\n\",\n \t\t\t wreq-\u003edebug_id, folio-\u003eindex))\n@@ -166,26 +181,34 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,\n \t\twreq-\u003ecleaned_to = fpos + fsize;\n \t\t*notes |= MADE_PROGRESS;\n \n-\t\t/* Clean up the head folioq. If we clear an entire folioq, then\n-\t\t * we can get rid of it provided it's not also the tail folioq\n-\t\t * being filled by the issuer.\n-\t\t */\n-\t\tfolioq_clear(folioq, slot);\n+\t\tbvecq-\u003ebv[slot].bv_page = NULL;\n+\tskip:\n \t\tslot++;\n-\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\tfolioq = rolling_buffer_delete_spent(\u0026wreq-\u003ebuffer);\n-\t\t\tif (!folioq)\n-\t\t\t\tgoto done;\n-\t\t\tslot = 0;\n-\t\t}\n-\n \t\tif (fpos + fsize \u003e= collected_to)\n \t\t\tbreak;\n \t}\n \n-\twreq-\u003ebuffer.tail = folioq;\n-done:\n-\twreq-\u003ebuffer.first_tail_slot = slot;\n+\twreq-\u003ecollect_cursor.slot = slot;\n+}\n+\n+/*\n+ * Collect cache results.\n+ */\n+static void netfs_cache_collect(struct netfs_io_request *wreq,\n+\t\t\t\tstruct netfs_io_stream *stream,\n+\t\t\t\tenum netfs_cache_collect block_type)\n+{\n+\tstruct netfs_cache_resources *cres = \u0026wreq-\u003ecache_resources;\n+\n+\tif (stream-\u003esource != NETFS_WRITE_TO_CACHE ||\n+\t wreq-\u003ecache_coll_to \u003e= stream-\u003ecollected_to)\n+\t\treturn;\n+\n+\tif (cres-\u003eops \u0026\u0026 cres-\u003eops-\u003ecollect_write)\n+\t\tcres-\u003eops-\u003ecollect_write(wreq, wreq-\u003ecache_coll_to,\n+\t\t\t\t\t stream-\u003ecollected_to - wreq-\u003ecache_coll_to,\n+\t\t\t\t\t block_type);\n+\twreq-\u003ecache_coll_to = stream-\u003ecollected_to;\n }\n \n /*\n@@ -210,11 +233,9 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)\n \ttrace_netfs_rreq(wreq, netfs_rreq_trace_collect);\n \n reassess_streams:\n-\tissued_to = atomic64_read(\u0026wreq-\u003eissued_to);\n-\tsmp_rmb();\n+\tissued_to = ULLONG_MAX;\n \tcollected_to = ULLONG_MAX;\n \tif (wreq-\u003eorigin == NETFS_WRITEBACK ||\n-\t wreq-\u003eorigin == NETFS_WRITETHROUGH ||\n \t wreq-\u003eorigin == NETFS_PGPRIV2_COPY_TO_CACHE)\n \t\tnotes = NEED_UNLOCK;\n \telse\n@@ -226,23 +247,43 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)\n \t * to the tail whilst we're doing this.\n \t */\n \tfor (s = 0; s \u003c NR_IO_STREAMS; s++) {\n+\t\tunsigned long long s_issued_to;\n+\n \t\tstream = \u0026wreq-\u003eio_streams[s];\n-\t\t/* Read active flag before list pointers */\n+\t\t/* Read active flag before issued_to */\n \t\tif (!smp_load_acquire(\u0026stream-\u003eactive))\n \t\t\tcontinue;\n \n-\t\tfront = list_first_entry_or_null_acquire(\u0026stream-\u003esubrequests,\n-\t\t\t\t\t\t\t struct netfs_io_subrequest, rreq_link);\n-\t\t/* Read first subreq pointer before IN_PROGRESS flag. */\n+\t\tfor (;;) {\n+\t\t\tbool cancelled;\n+\n+\t\t\t/* Order reading the issued_to point before reading the\n+\t\t\t * queue it refers to.\n+\t\t\t */\n+\t\t\ts_issued_to = atomic64_read_acquire(\u0026stream-\u003eissued_to);\n+\t\t\tif (s_issued_to \u003c issued_to)\n+\t\t\t\tissued_to = s_issued_to;\n+\n+\t\t\tfront = list_first_entry_or_null_acquire(\u0026stream-\u003esubrequests,\n+\t\t\t\t\t\t\t\t struct netfs_io_subrequest,\n+\t\t\t\t\t\t\t\t rreq_link);\n+\t\t\t/* Read first subreq pointer before IN_PROGRESS flag. */\n+\t\t\tif (!front) {\n+\t\t\t\tif (stream-\u003esource == NETFS_UPLOAD_TO_SERVER \u0026\u0026\n+\t\t\t\t test_bit(NETFS_RREQ_PAUSE, \u0026wreq-\u003eflags))\n+\t\t\t\t\tnotes |= MADE_PROGRESS;\n+\t\t\t\tbreak;\n+\t\t\t}\n \n-\t\twhile (front) {\n \t\t\ttrace_netfs_collect_sreq(wreq, front);\n \t\t\t//_debug(\"sreq [%x] %llx %zx/%zx\",\n \t\t\t// front-\u003edebug_index, front-\u003estart, front-\u003etransferred, front-\u003elen);\n \n \t\t\tif (stream-\u003ecollected_to \u003c front-\u003estart) {\n \t\t\t\ttrace_netfs_collect_gap(wreq, stream, issued_to, 'F');\n+\t\t\t\tnetfs_cache_collect(wreq, stream, NETFS_CACHE_COLLECT_WRITE_DATA);\n \t\t\t\tstream-\u003ecollected_to = front-\u003estart;\n+\t\t\t\tnetfs_cache_collect(wreq, stream, NETFS_CACHE_COLLECT_WRITE_GAP);\n \t\t\t}\n \n \t\t\t/* Stall if the front is still undergoing I/O. */\n@@ -250,7 +291,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)\n \t\t\t\tnotes |= HIT_PENDING;\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\tsmp_rmb(); /* Read counters after I-P flag. */\n \n \t\t\tif (stream-\u003efailed) {\n \t\t\t\tstream-\u003ecollected_to = front-\u003estart + front-\u003elen;\n@@ -263,15 +303,44 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)\n \t\t\t\tstream-\u003etransferred_valid = true;\n \t\t\t\tnotes |= MADE_PROGRESS;\n \t\t\t}\n-\t\t\tif (test_bit(NETFS_SREQ_FAILED, \u0026front-\u003eflags)) {\n-\t\t\t\tstream-\u003efailed = true;\n-\t\t\t\tstream-\u003eerror = front-\u003eerror;\n-\t\t\t\tif (stream-\u003esource == NETFS_UPLOAD_TO_SERVER)\n-\t\t\t\t\tmapping_set_error(wreq-\u003emapping, front-\u003eerror);\n-\t\t\t\tnotes |= NEED_REASSESS | SAW_FAILURE;\n+\n+\t\t\t/* Handle failed or cancelled subreqs. Failure of\n+\t\t\t * cache writes are handled differently to upload\n+\t\t\t * failures. Cache writes aren't fatal, provided we're\n+\t\t\t * not doing disconnected operation, and so we can kind\n+\t\t\t * of treat them as if they had succeeded - except that\n+\t\t\t * we need to log any holes they cause.\n+\t\t\t */\n+\t\t\tswitch (stream-\u003esource) {\n+\t\t\tcase NETFS_UPLOAD_TO_SERVER:\n+\t\t\t\tif (test_bit(NETFS_SREQ_FAILED, \u0026front-\u003eflags)) {\n+\t\t\t\t\tif (!stream-\u003efailed) {\n+\t\t\t\t\t\tstream-\u003efailed = true;\n+\t\t\t\t\t\tstream-\u003eerror = front-\u003eerror;\n+\t\t\t\t\t\tmapping_set_error(wreq-\u003emapping, front-\u003eerror);\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t\tnotes |= NEED_REASSESS | SAW_FAILURE;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\n+\t\t\tcase NETFS_WRITE_TO_CACHE:\n+\t\t\t\tcancelled = test_bit(NETFS_SREQ_CANCELLED, \u0026front-\u003eflags);\n+\t\t\t\tif (cancelled != stream-\u003ecancelled) {\n+\t\t\t\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_cache_fail_collect);\n+\t\t\t\t\tnetfs_cache_collect(wreq, stream,\n+\t\t\t\t\t\t\t cancelled ?\n+\t\t\t\t\t\t\t NETFS_CACHE_COLLECT_WRITE_CANCEL :\n+\t\t\t\t\t\t\t NETFS_CACHE_COLLECT_WRITE_DATA);\n+\t\t\t\t\tstream-\u003ecancelled = !stream-\u003ecancelled;\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tWARN_ON(1);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\tif (front-\u003etransferred \u003c front-\u003elen) {\n+\t\t\tif (test_bit(NETFS_SREQ_NEED_RETRY, \u0026front-\u003eflags)) {\n \t\t\t\tstream-\u003eneed_retry = true;\n \t\t\t\tnotes |= NEED_RETRY | MADE_PROGRESS;\n \t\t\t\tbreak;\n@@ -360,6 +429,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)\n */\n bool netfs_write_collection(struct netfs_io_request *wreq)\n {\n+\tstruct netfs_io_stream *cstream = \u0026wreq-\u003eio_streams[1];\n \tstruct netfs_inode *ictx = netfs_inode(wreq-\u003einode);\n \tsize_t transferred;\n \tbool transferred_valid = false;\n@@ -372,9 +442,8 @@ bool netfs_write_collection(struct netfs_io_request *wreq)\n \t/* We're done when the app thread has finished posting subreqs and all\n \t * the queues in all the streams are empty.\n \t */\n-\tif (!test_bit(NETFS_RREQ_ALL_QUEUED, \u0026wreq-\u003eflags))\n+\tif (!netfs_are_all_subreqs_queued(wreq))\n \t\treturn false;\n-\tsmp_rmb(); /* Read ALL_QUEUED before lists. */\n \n \ttransferred = LONG_MAX;\n \tfor (s = 0; s \u003c NR_IO_STREAMS; s++) {\n@@ -395,13 +464,22 @@ bool netfs_write_collection(struct netfs_io_request *wreq)\n \t\twreq-\u003etransferred = transferred;\n \ttrace_netfs_rreq(wreq, netfs_rreq_trace_write_done);\n \n-\tif (wreq-\u003eio_streams[1].active \u0026\u0026\n-\t wreq-\u003eio_streams[1].failed \u0026\u0026\n-\t ictx-\u003eops-\u003einvalidate_cache) {\n-\t\t/* Cache write failure doesn't prevent writeback completion\n-\t\t * unless we're in disconnected mode.\n-\t\t */\n-\t\tictx-\u003eops-\u003einvalidate_cache(wreq);\n+\tif (cstream-\u003eactive) {\n+\t\tif (test_bit(NETFS_RREQ_CACHE_ERROR, \u0026wreq-\u003eflags)) {\n+\t\t\tif (ictx-\u003eops-\u003einvalidate_cache) {\n+\t\t\t\t/* Cache write failure doesn't prevent\n+\t\t\t\t * writeback completion unless we're in\n+\t\t\t\t * disconnected mode.\n+\t\t\t\t */\n+\t\t\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_inval_cache);\n+\t\t\t\tictx-\u003eops-\u003einvalidate_cache(wreq);\n+\t\t\t}\n+\t\t} else if (!cstream-\u003efailed) {\n+\t\t\tnetfs_cache_collect(wreq, cstream,\n+\t\t\t\t\t cstream-\u003ecancelled ?\n+\t\t\t\t\t NETFS_CACHE_COLLECT_WRITE_CANCEL :\n+\t\t\t\t\t NETFS_CACHE_COLLECT_WRITE_DATA);\n+\t\t}\n \t}\n \n \t_debug(\"finished\");\n@@ -411,7 +489,6 @@ bool netfs_write_collection(struct netfs_io_request *wreq)\n \tswitch (wreq-\u003eorigin) {\n \tcase NETFS_WRITEBACK:\n \tcase NETFS_WRITEBACK_SINGLE:\n-\tcase NETFS_WRITETHROUGH:\n \t\tnetfs_wb_end(ictx);\n \t\tbreak;\n \tdefault:\n@@ -486,24 +563,57 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error)\n \n \tif (IS_ERR_VALUE(transferred_or_error)) {\n \t\tsubreq-\u003eerror = transferred_or_error;\n-\t\t/* if need retry is set, error should not matter */\n-\t\tif (!test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n-\t\t\tset_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n-\t\t\ttrace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);\n-\t\t}\n+\t\tif (transferred_or_error == -ENOMEM)\n+\t\t\tset_bit(NETFS_RREQ_SAW_ENOMEM, \u0026wreq-\u003eflags);\n \n \t\tswitch (subreq-\u003esource) {\n \t\tcase NETFS_WRITE_TO_CACHE:\n+\t\t\t/* We don't mark a cache-write subreq as failed.\n+\t\t\t * Instead we tell the issuer to produce dummy subreqs\n+\t\t\t * instead and make a note if we need to invalidate the\n+\t\t\t * cache at the end. We also don't pause the loop that\n+\t\t\t * grabs pages and launches upload subreqs.\n+\t\t\t *\n+\t\t\t * Note that we need to distinguish between -ENOBUFS\n+\t\t\t * (no space available in the cache) and other errors.\n+\t\t\t * In the former case, we can keep the data we have,\n+\t\t\t * though we might have to change the way the on-disk\n+\t\t\t * data is tracked.\n+\t\t\t */\n \t\t\tnetfs_stat(\u0026netfs_n_wh_write_failed);\n+\t\t\tif (test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n+\t\t\t\t/* We don't retry failed cache writes. */\n+\t\t\t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t\t\t\tif (!subreq-\u003eerror)\n+\t\t\t\t\tsubreq-\u003eerror = -ENOBUFS;\n+\t\t\t}\n+\n+\t\t\ttrace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);\n+\t\t\t__set_bit(NETFS_SREQ_CANCELLED, \u0026subreq-\u003eflags);\n+\t\t\tset_bit(NETFS_RREQ_CACHE_STOP, \u0026wreq-\u003eflags);\n+\t\t\tif (transferred_or_error == -ENOBUFS)\n+\t\t\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_cache_no_space);\n+\t\t\telse if (!test_and_set_bit(NETFS_RREQ_CACHE_ERROR, \u0026wreq-\u003eflags))\n+\t\t\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_cache_failed);\n+\t\t\tsubreq-\u003etransferred = subreq-\u003elen;\n \t\t\tbreak;\n+\n \t\tcase NETFS_UPLOAD_TO_SERVER:\n+\t\t\t/* If need_retry is set, error should not matter */\n+\t\t\tif (!test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n+\t\t\t\tset_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n+\t\t\t\ttrace_netfs_failure(wreq, subreq, transferred_or_error,\n+\t\t\t\t\t\t netfs_fail_upload);\n+\t\t\t}\n+\n+\t\t\tset_bit(NETFS_RREQ_PAUSE, \u0026wreq-\u003eflags);\n+\t\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);\n \t\t\tnetfs_stat(\u0026netfs_n_wh_upload_failed);\n \t\t\tbreak;\n+\n \t\tdefault:\n \t\t\tbreak;\n \t\t}\n-\t\ttrace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);\n-\t\tset_bit(NETFS_RREQ_PAUSE, \u0026wreq-\u003eflags);\n \t} else {\n \t\tif (WARN(transferred_or_error \u003e subreq-\u003elen - subreq-\u003etransferred,\n \t\t\t \"Subreq excess write: R=%x[%x] %zd \u003e %zu - %zu\",\ndiff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c\nindex f2761c99795a6b..3fa2aae5eaadf5 100644\n--- a/fs/netfs/write_issue.c\n+++ b/fs/netfs/write_issue.c\n@@ -36,6 +36,33 @@\n #include \u003clinux/pagemap.h\u003e\n #include \"internal.h\"\n \n+#define NOTE_UPLOAD_AVAIL\t0x001\t/* Upload is available */\n+#define NOTE_CACHE_AVAIL\t0x002\t/* Local cache is available */\n+#define NOTE_CACHE_COPY\t\t0x004\t/* Copy folio to cache */\n+#define NOTE_UPLOAD\t\t0x008\t/* Upload folio to server */\n+#define NOTE_UPLOAD_STARTED\t0x010\t/* Upload started */\n+#define NOTE_STREAMW\t\t0x020\t/* Folio is from a streaming write */\n+#define NOTE_DISCONTIG_BEFORE\t0x040\t/* Folio discontiguous with the previous folio */\n+#define NOTE_DISCONTIG_AFTER\t0x080\t/* Folio discontiguous with the next folio */\n+#define NOTE_TO_EOF\t\t0x100\t/* Data in folio ends at EOF */\n+#define NOTE_FLUSH_ANYWAY\t0x200\t/* Flush data, even if not hit estimated limit */\n+\n+#define NOTES__KEEP_MASK (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL | NOTE_UPLOAD_STARTED)\n+\n+struct netfs_wb_params {\n+\tunsigned long long\tlast_end;\t/* End file pos of previous folio */\n+\tunsigned long long\tfolio_start;\t/* File pos of folio */\n+\tunsigned int\t\tfolio_len;\t/* Length of folio */\n+\tunsigned int\t\tdirty_offset;\t/* Offset of dirty region in folio */\n+\tunsigned int\t\tdirty_len;\t/* Length of dirty region in folio */\n+\tunsigned int\t\tnotes;\t\t/* Notes on applicability */\n+\tstruct bvecq_pos\tdispatch_cursor; /* Folio queue anchor for issue_at */\n+\tstruct netfs_write_estimate estimates[NR_IO_STREAMS];\n+};\n+\n+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs);\n+\n /*\n * Kill all dirty folios in the event of an unrecoverable error, starting with\n * a locked folio we've already obtained from writeback_iter().\n@@ -96,7 +123,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,\n \tstruct netfs_inode *ictx;\n \tbool is_cacheable = (origin == NETFS_WRITEBACK ||\n \t\t\t origin == NETFS_WRITEBACK_SINGLE ||\n-\t\t\t origin == NETFS_WRITETHROUGH ||\n \t\t\t origin == NETFS_PGPRIV2_COPY_TO_CACHE);\n \n \twreq = netfs_alloc_request(mapping, file, start, 0, origin);\n@@ -108,83 +134,55 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,\n \tictx = netfs_inode(wreq-\u003einode);\n \tif (is_cacheable)\n \t\tfscache_begin_write_operation(\u0026wreq-\u003ecache_resources, netfs_i_cookie(ictx));\n-\tif (rolling_buffer_init(\u0026wreq-\u003ebuffer, wreq-\u003edebug_id, ITER_SOURCE) \u003c 0)\n-\t\tgoto nomem;\n \n \twreq-\u003ecleaned_to = wreq-\u003estart;\n+\tif (wreq-\u003ecache_resources.dio_size \u003e 1)\n+\t\twreq-\u003ecache_coll_to = round_down(wreq-\u003estart, wreq-\u003ecache_resources.dio_size);\n \n \twreq-\u003eio_streams[0].stream_nr\t\t= 0;\n \twreq-\u003eio_streams[0].source\t\t= NETFS_UPLOAD_TO_SERVER;\n-\twreq-\u003eio_streams[0].prepare_write\t= ictx-\u003eops-\u003eprepare_write;\n+\twreq-\u003eio_streams[0].applicable\t\t= NOTE_UPLOAD;\n+\twreq-\u003eio_streams[0].estimate_write\t= ictx-\u003eops-\u003eestimate_write;\n \twreq-\u003eio_streams[0].issue_write\t\t= ictx-\u003eops-\u003eissue_write;\n \twreq-\u003eio_streams[0].collected_to\t= start;\n \twreq-\u003eio_streams[0].transferred\t\t= 0;\n \n \twreq-\u003eio_streams[1].stream_nr\t\t= 1;\n \twreq-\u003eio_streams[1].source\t\t= NETFS_WRITE_TO_CACHE;\n+\twreq-\u003eio_streams[1].applicable\t\t= NOTE_CACHE_COPY;\n \twreq-\u003eio_streams[1].collected_to\t= start;\n \twreq-\u003eio_streams[1].transferred\t\t= 0;\n \tif (fscache_resources_valid(\u0026wreq-\u003ecache_resources)) {\n \t\twreq-\u003eio_streams[1].avail\t= true;\n \t\twreq-\u003eio_streams[1].active\t= true;\n-\t\twreq-\u003eio_streams[1].prepare_write = wreq-\u003ecache_resources.ops-\u003eprepare_write_subreq;\n+\t\twreq-\u003eio_streams[1].estimate_write = wreq-\u003ecache_resources.ops-\u003eestimate_write;\n \t\twreq-\u003eio_streams[1].issue_write = wreq-\u003ecache_resources.ops-\u003eissue_write;\n \t}\n \n \treturn wreq;\n-nomem:\n-\tnetfs_put_failed_request(wreq);\n-\treturn ERR_PTR(-ENOMEM);\n-}\n-\n-/**\n- * netfs_prepare_write_failed - Note write preparation failed\n- * @subreq: The subrequest to mark\n- *\n- * Mark a subrequest to note that preparation for write failed.\n- */\n-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq)\n-{\n-\t__set_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n-\ttrace_netfs_sreq(subreq, netfs_sreq_trace_prep_failed);\n }\n-EXPORT_SYMBOL(netfs_prepare_write_failed);\n \n /*\n- * Prepare a write subrequest. We need to allocate a new subrequest\n- * if we don't have one.\n+ * Allocate and prepare a write subrequest.\n */\n-void netfs_prepare_write(struct netfs_io_request *wreq,\n-\t\t\t struct netfs_io_stream *stream,\n-\t\t\t loff_t start)\n+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,\n+\t\t\t\t\t\t struct netfs_io_stream *stream)\n {\n \tstruct netfs_io_subrequest *subreq;\n-\tstruct iov_iter *wreq_iter = \u0026wreq-\u003ebuffer.iter;\n-\n-\t/* Make sure we don't point the iterator at a used-up folio_queue\n-\t * struct being used as a placeholder to prevent the queue from\n-\t * collapsing. In such a case, extend the queue.\n-\t */\n-\tif (iov_iter_is_folioq(wreq_iter) \u0026\u0026\n-\t wreq_iter-\u003efolioq_slot \u003e= folioq_nr_slots(wreq_iter-\u003efolioq))\n-\t\trolling_buffer_make_space(\u0026wreq-\u003ebuffer);\n \n \tsubreq = netfs_alloc_subrequest(wreq);\n \tsubreq-\u003esource\t\t= stream-\u003esource;\n-\tsubreq-\u003estart\t\t= start;\n+\tsubreq-\u003estart\t\t= stream-\u003eissue_from;\n+\tsubreq-\u003elen\t\t= stream-\u003ebuffered;\n \tsubreq-\u003estream_nr\t= stream-\u003estream_nr;\n-\tsubreq-\u003eio_iter\t\t= *wreq_iter;\n \n \t_enter(\"R=%x[%x]\", wreq-\u003edebug_id, subreq-\u003edebug_index);\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_prepare);\n \n-\tstream-\u003esreq_max_len\t= UINT_MAX;\n-\tstream-\u003esreq_max_segs\t= INT_MAX;\n \tswitch (stream-\u003esource) {\n \tcase NETFS_UPLOAD_TO_SERVER:\n \t\tnetfs_stat(\u0026netfs_n_wh_upload);\n-\t\tstream-\u003esreq_max_len = wreq-\u003ewsize;\n \t\tbreak;\n \tcase NETFS_WRITE_TO_CACHE:\n \t\tnetfs_stat(\u0026netfs_n_wh_write);\n@@ -194,9 +192,6 @@ void netfs_prepare_write(struct netfs_io_request *wreq,\n \t\tbreak;\n \t}\n \n-\tif (stream-\u003eprepare_write)\n-\t\tstream-\u003eprepare_write(subreq);\n-\n \t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n \n \t/* We add to the end of the list whilst the collector may be walking\n@@ -206,140 +201,377 @@ void netfs_prepare_write(struct netfs_io_request *wreq,\n \tspin_lock(\u0026wreq-\u003elock);\n \t/* Write IN_PROGRESS before pointer to new subreq */\n \tlist_add_tail_release(\u0026subreq-\u003erreq_link, \u0026stream-\u003esubrequests);\n-\tif (list_is_first(\u0026subreq-\u003erreq_link, \u0026stream-\u003esubrequests)) {\n-\t\tif (!stream-\u003eactive) {\n-\t\t\tstream-\u003ecollected_to = subreq-\u003estart;\n-\t\t\t/* Write list pointers before active flag */\n-\t\t\tsmp_store_release(\u0026stream-\u003eactive, true);\n-\t\t}\n-\t}\n+\tif (list_is_first(\u0026subreq-\u003erreq_link, \u0026stream-\u003esubrequests) \u0026\u0026\n+\t stream-\u003ecollected_to == 0)\n+\t\tstream-\u003ecollected_to = subreq-\u003estart;\n \n \tspin_unlock(\u0026wreq-\u003elock);\n-\n-\tstream-\u003econstruct = subreq;\n+\treturn subreq;\n }\n \n /*\n- * Set the I/O iterator for the filesystem/cache to use and dispatch the I/O\n- * operation. The operation may be asynchronous and should call\n- * netfs_write_subrequest_terminated() when complete.\n+ * Prepare the buffer for a buffered write.\n */\n-static void netfs_do_issue_write(struct netfs_io_stream *stream,\n-\t\t\t\t struct netfs_io_subrequest *subreq)\n+static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n {\n \tstruct netfs_io_request *wreq = subreq-\u003erreq;\n+\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tssize_t len;\n \n-\t_enter(\"R=%x[%x],%zx\", wreq-\u003edebug_id, subreq-\u003edebug_index, subreq-\u003elen);\n+\t_enter(\"%zx,{,%u,%u},%u\",\n+\t subreq-\u003elen, stream-\u003edispatch_cursor.slot, stream-\u003edispatch_cursor.offset, max_segs);\n \n-\tif (test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags))\n-\t\treturn netfs_write_subrequest_terminated(subreq, subreq-\u003eerror);\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n \n-\ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n-\tstream-\u003eissue_write(subreq);\n+\t/* If we have a write to the cache, we need to round out the first and\n+\t * last entries (only those as the data will be on virtually contiguous\n+\t * folios) to cache DIO boundaries.\n+\t */\n+\tif (subreq-\u003esource == NETFS_WRITE_TO_CACHE) {\n+\t\tstruct bio_vec *bv;\n+\t\tstruct bvecq *bq;\n+\t\tsize_t dio_size = wreq-\u003ecache_resources.dio_size;\n+\t\tsize_t disp, dlen;\n+\n+\t\tlen = bvecq_extract(\u0026stream-\u003edispatch_cursor, subreq-\u003elen, max_segs,\n+\t\t\t\t \u0026subreq-\u003econtent.bvecq);\n+\t\tif (len \u003c 0)\n+\t\t\treturn -ENOMEM;\n+\n+\t\t_debug(\"extract %zx/%zx\", len, subreq-\u003elen);\n+\t\tsubreq-\u003elen = len;\n+\n+\t\t/* Round the first entry down. We should be able to get away\n+\t\t * with this as this path only happens for buffered reads and\n+\t\t * writes. As such, a bio_vec must always point to a whole\n+\t\t * folio (or part thereof) in the pagecache with writeback set,\n+\t\t * so presuming that dio_size \u003c folio size, we should be able\n+\t\t * to round out bv_offset and bv_len.\n+\t\t *\n+\t\t * Further, streaming-write pages don't get sent to the cache\n+\t\t * (and aren't normally generated if there is a cache), so we\n+\t\t * only see fully uptodate pages here.\n+\t\t */\n+\t\tbq = subreq-\u003econtent.bvecq;\n+\t\tbv = \u0026bq-\u003ebv[0];\n+\t\tdisp = bv-\u003ebv_offset \u0026 (dio_size - 1);\n+\t\tif (disp) {\n+\t\t\tbv-\u003ebv_offset -= disp;\n+\t\t\tbv-\u003ebv_len += disp;\n+\t\t\tbq-\u003efpos -= disp;\n+\t\t\tsubreq-\u003estart -= disp;\n+\t\t\tsubreq-\u003elen += disp;\n+\t\t}\n+\n+\t\t/* Round the end of the last entry up. */\n+\t\twhile (bq-\u003enext)\n+\t\t\tbq = bq-\u003enext;\n+\t\tbv = \u0026bq-\u003ebv[bq-\u003enr_slots - 1];\n+\t\tdlen = round_up(bv-\u003ebv_len, dio_size);\n+\t\tif (dlen \u003e bv-\u003ebv_len) {\n+\t\t\tsubreq-\u003elen += dlen - bv-\u003ebv_len;\n+\t\t\tbv-\u003ebv_len = dlen;\n+\t\t}\n+\t} else {\n+\t\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026stream-\u003edispatch_cursor);\n+\t\tlen = bvecq_slice(\u0026stream-\u003edispatch_cursor, subreq-\u003elen, max_segs,\n+\t\t\t\t \u0026subreq-\u003enr_segs);\n+\n+\t\tif (len \u003c subreq-\u003elen) {\n+\t\t\tsubreq-\u003elen = len;\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n+\t\t}\n+\t}\n+\n+\tstream-\u003eissue_from += len;\n+\tstream-\u003ebuffered -= len;\n+\tif (stream-\u003ebuffered == 0) {\n+\t\tstream-\u003ebuffering = false;\n+\t\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n+\t}\n+\t/* Order loading the queue before updating the issue_to point */\n+\tatomic64_set_release(\u0026stream-\u003eissued_to, stream-\u003eissue_from);\n+\treturn 0;\n }\n \n-void netfs_reissue_write(struct netfs_io_stream *stream,\n-\t\t\t struct netfs_io_subrequest *subreq,\n-\t\t\t struct iov_iter *source)\n+/**\n+ * netfs_prepare_write_buffer - Get the buffer for a subrequest\n+ * @subreq: The subrequest to get the buffer for\n+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)\n+ *\n+ * Extract a slice of buffer from the stream and attach it to the subrequest as\n+ * a bio_vec queue. The maximum amount of data attached is set by\n+ * @subreq-\u003elen, but this may be shortened if @max_segs would be exceeded.\n+ */\n+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t unsigned int max_segs)\n {\n-\tsize_t size = subreq-\u003elen - subreq-\u003etransferred;\n+\tstruct netfs_io_request *rreq = subreq-\u003erreq;\n \n-\t// TODO: Use encrypted buffer\n-\tsubreq-\u003eio_iter = *source;\n-\tiov_iter_advance(source, size);\n-\tiov_iter_truncate(\u0026subreq-\u003eio_iter, size);\n+\tswitch (rreq-\u003eorigin) {\n+\tcase NETFS_WRITEBACK:\n+\t\tif (test_bit(NETFS_RREQ_RETRYING, \u0026rreq-\u003eflags))\n+\t\t\treturn netfs_prepare_write_retry_buffer(subreq, max_segs);\n+\t\treturn netfs_prepare_buffered_write_buffer(subreq, max_segs);\n \n-\tsubreq-\u003eretry_count++;\n-\tsubreq-\u003eerror = 0;\n-\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n-\t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n-\tnetfs_stat(\u0026netfs_n_wh_retry_write_subreq);\n-\tnetfs_do_issue_write(stream, subreq);\n+\tcase NETFS_UNBUFFERED_WRITE:\n+\tcase NETFS_DIO_WRITE:\n+\t\treturn netfs_prepare_unbuffered_write_buffer(subreq, max_segs);\n+\n+\tcase NETFS_WRITEBACK_SINGLE:\n+\t\treturn netfs_prepare_write_single_buffer(subreq, max_segs);\n+\n+\tcase NETFS_PGPRIV2_COPY_TO_CACHE:\n+\t\treturn netfs_prepare_pgpriv2_write_buffer(subreq, max_segs);\n+\n+\tdefault:\n+\t\tWARN_ON_ONCE(1);\n+\t\treturn -EIO;\n+\t}\n }\n+EXPORT_SYMBOL(netfs_prepare_write_buffer);\n \n-void netfs_issue_write(struct netfs_io_request *wreq,\n-\t\t struct netfs_io_stream *stream)\n+/*\n+ * Issue writes for a stream.\n+ */\n+static int netfs_issue_writes(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_wb_params *params)\n {\n-\tstruct netfs_io_subrequest *subreq = stream-\u003econstruct;\n+\tstruct netfs_write_estimate *estimate = \u0026params-\u003eestimates[stream-\u003estream_nr];\n \n-\tif (!subreq)\n-\t\treturn;\n-\tstream-\u003econstruct = NULL;\n-\tsubreq-\u003eio_iter.count = subreq-\u003elen;\n-\tnetfs_do_issue_write(stream, subreq);\n+\tfor (;;) {\n+\t\tstruct netfs_io_subrequest *subreq;\n+\t\tint ret;\n+\n+\t\tif (test_bit(NETFS_RREQ_PAUSE, \u0026wreq-\u003eflags))\n+\t\t\tnetfs_wait_for_paused_write(wreq);\n+\n+\t\tsubreq = netfs_alloc_write_subreq(wreq, stream);\n+\t\tif (!subreq)\n+\t\t\treturn -ENOMEM;\n+\n+\t\tif (stream-\u003esource == NETFS_WRITE_TO_CACHE \u0026\u0026\n+\t\t unlikely(test_bit(NETFS_RREQ_CACHE_STOP, \u0026wreq-\u003eflags))) {\n+\t\t\tsize_t dio_size = wreq-\u003ecache_resources.dio_size;\n+\t\t\tsize_t len, disp;\n+\n+\t\t\tdisp = subreq-\u003estart \u0026 (dio_size - 1);\n+\t\t\tlen = round_up(subreq-\u003elen + disp, dio_size);\n+\n+\t\t\tsubreq-\u003estart -= disp;\n+\t\t\tsubreq-\u003elen = len;\n+\n+\t\t\tstream-\u003eissue_from = subreq-\u003estart + len;\n+\t\t\tstream-\u003ebuffered = 0;\n+\t\t\tstream-\u003ebuffering = false;\n+\t\t\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n+\t\t\testimate-\u003eissue_at = subreq-\u003estart + len + 16 * 1024 * 1024;\n+\t\t\testimate-\u003emax_segs = INT_MAX;\n+\t\t\t__set_bit(NETFS_SREQ_CANCELLED, \u0026subreq-\u003eflags);\n+\t\t\tnetfs_write_subrequest_terminated(subreq, len);\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tret = stream-\u003eissue_write(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\t/* Ownership of subreq was returned to us. */\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\tbvecq_pos_advance(\u0026stream-\u003edispatch_cursor, subreq-\u003elen);\n+\t\t\tstream-\u003eissue_from += subreq-\u003elen;\n+\t\t\tstream-\u003ebuffered -= subreq-\u003elen;\n+\t\t\tatomic64_set_release(\u0026stream-\u003eissued_to, stream-\u003eissue_from);\n+\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n+\t\t}\n+\t\t/* We no longer own subreq. */\n+\n+\t\tif (test_bit(NETFS_RREQ_SAW_ENOMEM, \u0026wreq-\u003eflags))\n+\t\t\treturn -ENOMEM;\n+\t\tif (stream-\u003ebuffered == 0) {\n+\t\t\tif (stream-\u003estream_nr == 0)\n+\t\t\t\tparams-\u003enotes \u0026= ~NOTE_UPLOAD_STARTED;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!(params-\u003enotes \u0026 NOTE_FLUSH_ANYWAY)) {\n+\t\t\testimate-\u003eissue_at = ULLONG_MAX;\n+\t\t\testimate-\u003emax_segs = INT_MAX;\n+\t\t\tstream-\u003eestimate_write(wreq, stream, estimate);\n+\t\t\tif (stream-\u003eissue_from + stream-\u003ebuffered \u003c estimate-\u003eissue_at \u0026\u0026\n+\t\t\t estimate-\u003emax_segs \u003e 0)\n+\t\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Issue pending writes on a stream.\n+ */\n+static int netfs_issue_stream(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_wb_params *params, int s)\n+{\n+\tstruct netfs_write_estimate *estimate = \u0026params-\u003eestimates[s];\n+\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[s];\n+\tunsigned long long dirty_start;\n+\tbool discontig_before = params-\u003enotes \u0026 NOTE_DISCONTIG_BEFORE;\n+\tint ret;\n+\n+\t_enter(\"%x\", params-\u003enotes);\n+\n+\t/* If the current folio doesn't contribute to this stream, see if we\n+\t * need to flush it.\n+\t */\n+\tif (!(params-\u003enotes \u0026 stream-\u003eapplicable)) {\n+\t\tif (!stream-\u003ebuffering) {\n+\t\t\tatomic64_set_release(\u0026stream-\u003eissued_to,\n+\t\t\t\t\t params-\u003efolio_start + params-\u003efolio_len);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tdiscontig_before = true;\n+\t}\n+\n+\t/* Issue writes if we meet a discontiguity before the current folio.\n+\t * Even if the filesystem can do sparse/vectored writes, we still\n+\t * generate a subreq per contiguous region rather than generating\n+\t * separate extent lists.\n+\t */\n+\tif (stream-\u003ebuffering \u0026\u0026 discontig_before) {\n+\t\tparams-\u003enotes |= NOTE_FLUSH_ANYWAY;\n+\t\tret = netfs_issue_writes(wreq, stream, params);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t\tstream-\u003ebuffering = false;\n+\t\tparams-\u003enotes \u0026= ~NOTE_FLUSH_ANYWAY;\n+\t}\n+\n+\tif (!(params-\u003enotes \u0026 stream-\u003eapplicable)) {\n+\t\tatomic64_set_release(\u0026stream-\u003eissued_to,\n+\t\t\t\t params-\u003efolio_start + params-\u003efolio_len);\n+\t\treturn 0;\n+\t}\n+\n+\t/* If we're not currently buffering on this stream, we need to get an\n+\t * estimate of when we need to issue a write. It might be within the\n+\t * starting folio.\n+\t */\n+\tdirty_start = params-\u003efolio_start + params-\u003edirty_offset;\n+\tif (!stream-\u003ebuffering) {\n+\t\tstream-\u003ebuffering = true;\n+\t\tstream-\u003eissue_from = dirty_start;\n+\t\tbvecq_pos_set(\u0026stream-\u003edispatch_cursor, \u0026params-\u003edispatch_cursor);\n+\t\testimate-\u003eissue_at = ULLONG_MAX;\n+\t\testimate-\u003emax_segs = INT_MAX;\n+\t\tstream-\u003eestimate_write(wreq, stream, estimate);\n+\t}\n+\n+\tstream-\u003ebuffered += params-\u003edirty_len;\n+\testimate-\u003emax_segs--;\n+\n+\t/* Poke the filesystem to issue writes when we hit the limit it set or\n+\t * if the data ends before the end of the page.\n+\t */\n+\tif (params-\u003enotes \u0026 NOTE_DISCONTIG_AFTER)\n+\t\tparams-\u003enotes |= NOTE_FLUSH_ANYWAY;\n+\t_debug(\"[%u] %llx + %zx \u003e= %llx, %u %x\",\n+\t s, stream-\u003eissue_from, stream-\u003ebuffered, estimate-\u003eissue_at,\n+\t estimate-\u003emax_segs, params-\u003enotes);\n+\tif (stream-\u003eissue_from + stream-\u003ebuffered \u003e= estimate-\u003eissue_at ||\n+\t estimate-\u003emax_segs \u003c= 0 ||\n+\t (params-\u003enotes \u0026 NOTE_FLUSH_ANYWAY)) {\n+\t\tret = netfs_issue_writes(wreq, stream, params);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n }\n \n /*\n- * Add data to the write subrequest, dispatching each as we fill it up or if it\n- * is discontiguous with the previous. We only fill one part at a time so that\n- * we can avoid overrunning the credits obtained (cifs) and try to parallelise\n- * content-crypto preparation with network writes.\n+ * See which streams need writes issuing and issue them.\n */\n-size_t netfs_advance_write(struct netfs_io_request *wreq,\n-\t\t\t struct netfs_io_stream *stream,\n-\t\t\t loff_t start, size_t len, bool to_eof)\n+static int netfs_issue_streams(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_wb_params *params)\n {\n-\tstruct netfs_io_subrequest *subreq = stream-\u003econstruct;\n-\tsize_t part;\n+\tint ret = 0, ret2;\n+\n+\t_enter(\"%x\", params-\u003enotes);\n \n-\tif (!stream-\u003eavail) {\n-\t\t_leave(\"no write\");\n-\t\treturn len;\n+\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n+\t\tret2 = netfs_issue_stream(wreq, params, s);\n+\t\tif (ret2 \u003c 0)\n+\t\t\tret = ret2;\n \t}\n+\treturn ret;\n+}\n \n-\t_enter(\"R=%x[%x]\", wreq-\u003edebug_id, subreq ? subreq-\u003edebug_index : 0);\n+/*\n+ * End the issuing of writes, let the collector know we're done.\n+ */\n+static void netfs_end_issue_write(struct netfs_io_request *wreq,\n+\t\t\t\t struct netfs_wb_params *params)\n+{\n+\tbool needs_poke = true;\n \n-\tif (subreq \u0026\u0026 start != subreq-\u003estart + subreq-\u003elen) {\n-\t\tnetfs_issue_write(wreq, stream);\n-\t\tsubreq = NULL;\n+\tparams-\u003enotes |= NOTE_FLUSH_ANYWAY;\n+\n+\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n+\t\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[s];\n+\t\tint ret;\n+\n+\t\tif (stream-\u003ebuffering) {\n+\t\t\tret = netfs_issue_writes(wreq, stream, params);\n+\t\t\tif (ret \u003c 0 \u0026\u0026 stream-\u003esource != NETFS_WRITE_TO_CACHE) {\n+\t\t\t\t/* Leave the error somewhere the completion\n+\t\t\t\t * path can pick it up if there isn't already\n+\t\t\t\t * another error logged.\n+\t\t\t\t */\n+\t\t\t\tcmpxchg(\u0026wreq-\u003eerror, 0, ret);\n+\t\t\t}\n+\t\t\tstream-\u003ebuffering = false;\n+\t\t}\n \t}\n \n-\tif (!stream-\u003econstruct)\n-\t\tnetfs_prepare_write(wreq, stream, start);\n-\tsubreq = stream-\u003econstruct;\n-\n-\tpart = umin(stream-\u003esreq_max_len - subreq-\u003elen, len);\n-\t_debug(\"part %zx/%zx %zx/%zx\", subreq-\u003elen, stream-\u003esreq_max_len, part, len);\n-\tsubreq-\u003elen += part;\n-\tsubreq-\u003enr_segs++;\n-\tstream-\u003esubmit_extendable_to -= part;\n-\n-\tif (subreq-\u003elen \u003e= stream-\u003esreq_max_len ||\n-\t subreq-\u003enr_segs \u003e= stream-\u003esreq_max_segs ||\n-\t to_eof) {\n-\t\tnetfs_issue_write(wreq, stream);\n-\t\tsubreq = NULL;\n+\tnetfs_all_subreqs_queued(wreq);\n+\n+\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n+\t\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[s];\n+\n+\t\tif (!stream-\u003eactive)\n+\t\t\tcontinue;\n+\t\tif (!list_empty(\u0026stream-\u003esubrequests))\n+\t\t\tneeds_poke = false;\n \t}\n \n-\treturn part;\n+\tif (needs_poke)\n+\t\tnetfs_wake_collector(wreq);\n }\n \n /*\n- * Write some of a pending folio data back to the server.\n+ * Queue a folio for writeback.\n */\n-static int netfs_write_folio(struct netfs_io_request *wreq,\n-\t\t\t struct writeback_control *wbc,\n-\t\t\t struct folio *folio)\n+static void netfs_queue_wb_folio(struct netfs_io_request *wreq,\n+\t\t\t\t struct writeback_control *wbc,\n+\t\t\t\t struct folio *folio,\n+\t\t\t\t struct netfs_wb_params *params)\n {\n-\tstruct netfs_io_stream *upload = \u0026wreq-\u003eio_streams[0];\n-\tstruct netfs_io_stream *cache = \u0026wreq-\u003eio_streams[1];\n-\tstruct netfs_io_stream *stream;\n \tstruct netfs_group *fgroup; /* TODO: Use this with ceph */\n \tstruct netfs_folio *finfo;\n-\tsize_t iter_off = 0;\n+\tstruct bvecq *queue = wreq-\u003eload_cursor.bvecq;\n+\tunsigned int slot;\n \tsize_t fsize = folio_size(folio), flen = fsize, foff = 0;\n \tloff_t fpos = folio_pos(folio), i_size;\n-\tbool to_eof = false, streamw = false;\n-\tbool debug = false;\n \n-\t_enter(\"\");\n+\t_enter(\"%x\", params-\u003enotes);\n \n-\tif (rolling_buffer_make_space(\u0026wreq-\u003ebuffer) \u003c 0)\n-\t\treturn -ENOMEM;\n-\n-\t/* netfs_perform_write() may shift i_size around the page or from out\n-\t * of the page to beyond it, but cannot move i_size into or through the\n-\t * page since we have it locked.\n+\t/* netfs_perform_write() may shift i_size around the folio or from out\n+\t * of the folio to beyond it, but cannot move i_size into or through\n+\t * the folio since we have it locked.\n+\t *\n+\t * Truncate could in theory move i_size into or before the folio, but\n+\t * it should take steps to prevent writeback from happening\n+\t * concurrently and should wait for any in-progress writebacks before\n+\t * proceeding.\n \t */\n \ti_size = i_size_read(wreq-\u003einode);\n \n@@ -351,7 +583,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,\n \t\twreq-\u003enr_group_rel += netfs_folio_written_back(folio);\n \t\tnetfs_put_group_many(wreq-\u003egroup, wreq-\u003enr_group_rel);\n \t\twreq-\u003enr_group_rel = 0;\n-\t\treturn 0;\n+\t\treturn;\n \t}\n \n \tif (fpos + fsize \u003e wreq-\u003ei_size)\n@@ -362,23 +594,32 @@ static int netfs_write_folio(struct netfs_io_request *wreq,\n \tif (finfo) {\n \t\tfoff = finfo-\u003edirty_offset;\n \t\tflen = foff + finfo-\u003edirty_len;\n-\t\tstreamw = true;\n+\t\tparams-\u003enotes |= NOTE_STREAMW;\n+\t\tif (foff \u003e 0)\n+\t\t\tparams-\u003enotes |= NOTE_DISCONTIG_BEFORE;\n+\t\tif (flen \u003c fsize)\n+\t\t\tparams-\u003enotes |= NOTE_DISCONTIG_AFTER;\n \t}\n \n-\tif (wreq-\u003eorigin == NETFS_WRITETHROUGH) {\n-\t\tto_eof = false;\n-\t\tif (flen \u003e i_size - fpos)\n-\t\t\tflen = i_size - fpos;\n-\t} else if (flen \u003e i_size - fpos) {\n+\tif (params-\u003elast_end \u0026\u0026 fpos != params-\u003elast_end)\n+\t\tparams-\u003enotes |= NOTE_DISCONTIG_BEFORE;\n+\tparams-\u003elast_end = fpos + fsize;\n+\n+\tif (flen \u003e i_size - fpos) {\n \t\tflen = i_size - fpos;\n-\t\tif (!streamw)\n+\t\tif (!(params-\u003enotes \u0026 NOTE_STREAMW))\n \t\t\tfolio_zero_segment(folio, flen, fsize);\n-\t\tto_eof = true;\n+\t\tparams-\u003enotes |= NOTE_TO_EOF;\n \t} else if (flen == i_size - fpos) {\n-\t\tto_eof = true;\n+\t\tparams-\u003enotes |= NOTE_TO_EOF;\n \t}\n \tflen -= foff;\n \n+\tparams-\u003efolio_start\t= fpos;\n+\tparams-\u003efolio_len\t= fsize;\n+\tparams-\u003edirty_offset\t= foff;\n+\tparams-\u003edirty_len\t= flen;\n+\n \t_debug(\"folio %zx %zx %zx\", foff, flen, fsize);\n \n \t/* Deal with discontinuities in the stream of dirty pages. These can\n@@ -398,146 +639,80 @@ static int netfs_write_folio(struct netfs_io_request *wreq,\n \t * write-back group.\n \t */\n \tif (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {\n-\t\tnetfs_issue_write(wreq, upload);\n-\t} else if (fgroup != wreq-\u003egroup) {\n-\t\t/* We can't write this page to the server yet. */\n-\t\tkdebug(\"wrong group\");\n-\t\tfolio_redirty_for_writepage(wbc, folio);\n-\t\tfolio_unlock(folio);\n-\t\tnetfs_issue_write(wreq, upload);\n-\t\tnetfs_issue_write(wreq, cache);\n-\t\treturn 0;\n-\t}\n-\n-\tif (foff \u003e 0)\n-\t\tnetfs_issue_write(wreq, upload);\n-\tif (streamw)\n-\t\tnetfs_issue_write(wreq, cache);\n-\n-\tfolio_start_writeback(folio);\n-\tfolio_unlock(folio);\n-\n-\tif (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {\n-\t\tif (!cache-\u003eavail) {\n+\t\tif (!(params-\u003enotes \u0026 NOTE_CACHE_AVAIL)) {\n \t\t\ttrace_netfs_folio(folio, netfs_folio_trace_cancel_copy);\n-\t\t\tnetfs_issue_write(wreq, upload);\n-\t\t\tnetfs_folio_written_back(folio);\n-\t\t\treturn 0;\n+\t\t\tgoto cancel_folio;\n \t\t}\n+\t\tparams-\u003enotes |= NOTE_CACHE_COPY;\n \t\ttrace_netfs_folio(folio, netfs_folio_trace_store_copy);\n-\t} else if (!upload-\u003eavail \u0026\u0026 !cache-\u003eavail) {\n+\t} else if (fgroup != wreq-\u003egroup) {\n+\t\t/* We can't write this page to the server yet. */\n+\t\tkdebug(\"wrong group\");\n+\t\tgoto skip_folio;\n+\t} else if (!(params-\u003enotes \u0026 (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL))) {\n \t\ttrace_netfs_folio(folio, netfs_folio_trace_cancel_store);\n-\t\tnetfs_folio_written_back(folio);\n-\t\treturn 0;\n-\t} else if (!upload-\u003econstruct) {\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_store);\n+\t\tgoto cancel_folio_discard;\n \t} else {\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_store_plus);\n-\t}\n-\n-\t/* Attach the folio to the rolling buffer. */\n-\trolling_buffer_append(\u0026wreq-\u003ebuffer, folio, 0);\n-\n-\t/* Move the submission point forward to allow for write-streaming data\n-\t * not starting at the front of the page. We don't do write-streaming\n-\t * with the cache as the cache requires DIO alignment.\n-\t *\n-\t * Also skip uploading for data that's been read and just needs copying\n-\t * to the cache.\n-\t */\n-\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n-\t\tstream = \u0026wreq-\u003eio_streams[s];\n-\t\tstream-\u003esubmit_off = foff;\n-\t\tstream-\u003esubmit_len = flen;\n-\t\tif (!stream-\u003eavail ||\n-\t\t (stream-\u003esource == NETFS_WRITE_TO_CACHE \u0026\u0026 streamw) ||\n-\t\t (stream-\u003esource == NETFS_UPLOAD_TO_SERVER \u0026\u0026\n-\t\t fgroup == NETFS_FOLIO_COPY_TO_CACHE)) {\n-\t\t\tstream-\u003esubmit_off = UINT_MAX;\n-\t\t\tstream-\u003esubmit_len = 0;\n+\t\tif (params-\u003enotes \u0026 NOTE_UPLOAD_STARTED) {\n+\t\t\tparams-\u003enotes |= NOTE_UPLOAD;\n+\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_store_plus);\n+\t\t} else {\n+\t\t\tparams-\u003enotes |= NOTE_UPLOAD | NOTE_UPLOAD_STARTED;\n+\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_store);\n \t\t}\n+\t\tif ((params-\u003enotes \u0026 NOTE_CACHE_AVAIL) \u0026\u0026\n+\t\t !(params-\u003enotes \u0026 NOTE_STREAMW))\n+\t\t\tparams-\u003enotes |= NOTE_CACHE_COPY;\n \t}\n \n-\t/* Attach the folio to one or more subrequests. For a big folio, we\n-\t * could end up with thousands of subrequests if the wsize is small -\n-\t * but we might need to wait during the creation of subrequests for\n-\t * network resources (eg. SMB credits).\n-\t */\n-\tfor (;;) {\n-\t\tssize_t part;\n-\t\tsize_t lowest_off = ULONG_MAX;\n-\t\tint choose_s = -1;\n-\n-\t\t/* Always add to the lowest-submitted stream first. */\n-\t\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n-\t\t\tstream = \u0026wreq-\u003eio_streams[s];\n-\t\t\tif (stream-\u003esubmit_len \u003e 0 \u0026\u0026\n-\t\t\t stream-\u003esubmit_off \u003c lowest_off) {\n-\t\t\t\tlowest_off = stream-\u003esubmit_off;\n-\t\t\t\tchoose_s = s;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (choose_s \u003c 0)\n-\t\t\tbreak;\n-\t\tstream = \u0026wreq-\u003eio_streams[choose_s];\n-\n-\t\t/* Advance the iterator(s). */\n-\t\tif (stream-\u003esubmit_off \u003e iter_off) {\n-\t\t\trolling_buffer_advance(\u0026wreq-\u003ebuffer, stream-\u003esubmit_off - iter_off);\n-\t\t\titer_off = stream-\u003esubmit_off;\n-\t\t}\n+\tfolio_start_writeback(folio);\n+\tfolio_unlock(folio);\n \n-\t\tatomic64_set(\u0026wreq-\u003eissued_to, fpos + stream-\u003esubmit_off);\n-\t\tstream-\u003esubmit_extendable_to = fsize - stream-\u003esubmit_off;\n-\t\tpart = netfs_advance_write(wreq, stream, fpos + stream-\u003esubmit_off,\n-\t\t\t\t\t stream-\u003esubmit_len, to_eof);\n-\t\tstream-\u003esubmit_off += part;\n-\t\tif (part \u003e stream-\u003esubmit_len)\n-\t\t\tstream-\u003esubmit_len = 0;\n-\t\telse\n-\t\t\tstream-\u003esubmit_len -= part;\n-\t\tif (part \u003e 0)\n-\t\t\tdebug = true;\n+\t/* Institute a new bvec queue segment if the current one is full or if\n+\t * we encounter a discontiguity. The discontiguity break is important\n+\t * when it comes to bulk unlocking folios by file range.\n+\t */\n+\tif (bvecq_is_full(queue) ||\n+\t ((params-\u003enotes \u0026 NOTE_DISCONTIG_BEFORE) \u0026\u0026 queue-\u003enr_slots \u003e 0)) {\n+\t\tbvecq_buffer_append(\u0026wreq-\u003eload_cursor, wreq-\u003espare);\n+\t\twreq-\u003espare = NULL;\n+\n+\t\tqueue = wreq-\u003eload_cursor.bvecq;\n+\t\tqueue-\u003efpos = fpos;\n+\t\tif (params-\u003enotes \u0026 NOTE_DISCONTIG_BEFORE)\n+\t\t\tqueue-\u003ediscontig = true;\n+\t\tbvecq_pos_move(\u0026params-\u003edispatch_cursor, queue);\n+\t\tparams-\u003edispatch_cursor.slot = 0;\n \t}\n \n-\tif (fsize \u003e iter_off)\n-\t\trolling_buffer_advance(\u0026wreq-\u003ebuffer, fsize - iter_off);\n-\tatomic64_set(\u0026wreq-\u003eissued_to, fpos + fsize);\n-\n-\tif (!debug)\n-\t\tkdebug(\"R=%x: No submit\", wreq-\u003edebug_id);\n-\n-\tif (foff + flen \u003c fsize)\n-\t\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++)\n-\t\t\tnetfs_issue_write(wreq, \u0026wreq-\u003eio_streams[s]);\n-\n-\t_leave(\" = 0\");\n-\treturn 0;\n-}\n-\n-/*\n- * End the issuing of writes, letting the collector know we're done.\n- */\n-static void netfs_end_issue_write(struct netfs_io_request *wreq)\n-{\n-\tbool needs_poke = true;\n-\n-\tsmp_wmb(); /* Write subreq lists before ALL_QUEUED. */\n-\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026wreq-\u003eflags);\n-\n-\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n-\t\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[s];\n+\t/* Attach the folio to the rolling buffer. */\n+\tslot = queue-\u003enr_slots;\n+\tbvec_set_folio(\u0026queue-\u003ebv[slot], folio, flen, foff);\n+\ttrace_netfs_bv_slot(queue, slot);\n+\tslot++;\n+\tbvecq_filled_to(queue, slot);\n+\twreq-\u003eload_cursor.slot = slot;\n+\twreq-\u003eload_cursor.offset = 0;\n+\ttrace_netfs_wback(wreq, folio, params-\u003enotes);\n \n-\t\tif (!stream-\u003eactive)\n-\t\t\tcontinue;\n-\t\tif (!list_empty(\u0026stream-\u003esubrequests))\n-\t\t\tneeds_poke = false;\n-\t\tnetfs_issue_write(wreq, stream);\n-\t}\n+out:\n+\t_leave(\" = %x\", params-\u003enotes);\n+\treturn;\n \n-\tif (needs_poke)\n-\t\tnetfs_wake_collector(wreq);\n+skip_folio:\n+\tfolio_redirty_for_writepage(wbc, folio);\n+\tfolio_unlock(folio);\n+\tparams-\u003enotes |= NOTE_DISCONTIG_BEFORE;\n+\tgoto out;\n+cancel_folio_discard:\n+\tnetfs_put_group(fgroup);\n+cancel_folio:\n+\tfolio_detach_private(folio);\n+\tkfree(finfo);\n+\tfolio_unlock(folio);\n+\tfolio_cancel_dirty(folio);\n+\tparams-\u003enotes |= NOTE_DISCONTIG_BEFORE;\n+\tgoto out;\n }\n \n /*\n@@ -548,6 +723,7 @@ int netfs_writepages(struct address_space *mapping,\n {\n \tstruct netfs_inode *ictx = netfs_inode(mapping-\u003ehost);\n \tstruct netfs_io_request *wreq = NULL;\n+\tstruct netfs_wb_params params = {};\n \tstruct folio *folio;\n \tint error = 0;\n \n@@ -565,36 +741,65 @@ int netfs_writepages(struct address_space *mapping,\n \t\tgoto couldnt_start;\n \t}\n \n+\tif (bvecq_buffer_init(\u0026wreq-\u003eload_cursor, GFP_NOFS) \u003c 0)\n+\t\tgoto nomem;\n+\tbvecq_pos_set(\u0026params.dispatch_cursor, \u0026wreq-\u003eload_cursor);\n+\tbvecq_pos_set(\u0026wreq-\u003ecollect_cursor, \u0026wreq-\u003eload_cursor);\n+\n \t__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, \u0026wreq-\u003eflags);\n \ttrace_netfs_write(wreq, netfs_write_trace_writeback);\n \tnetfs_stat(\u0026netfs_n_wh_writepages);\n \n-\tdo {\n-\t\t_debug(\"wbiter %lx %llx\", folio-\u003eindex, atomic64_read(\u0026wreq-\u003eissued_to));\n+\tif (wreq-\u003eio_streams[1].avail)\n+\t\tparams.notes |= NOTE_CACHE_AVAIL;\n \n-\t\t/* It appears we don't have to handle cyclic writeback wrapping. */\n-\t\tWARN_ON_ONCE(wreq \u0026\u0026 folio_pos(folio) \u003c atomic64_read(\u0026wreq-\u003eissued_to));\n+\tdo {\n+\t\t_debug(\"wbiter %lx\", folio-\u003eindex);\n+\n+\t\tif (!wreq-\u003espare) {\n+\t\t\twreq-\u003espare = bvecq_alloc_one(BVECQ_STD_SLOTS, GFP_NOFS);\n+\t\t\tif (!wreq-\u003espare) {\n+\t\t\t\tfolio_redirty_for_writepage(wbc, folio);\n+\t\t\t\tfolio_unlock(folio);\n+\t\t\t\terror = -ENOMEM;\n+\t\t\t\tgoto end_loop;\n+\t\t\t}\n+\t\t}\n \n \t\tif (netfs_folio_group(folio) != NETFS_FOLIO_COPY_TO_CACHE \u0026\u0026\n \t\t unlikely(!test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, \u0026wreq-\u003eflags))) {\n \t\t\tset_bit(NETFS_RREQ_UPLOAD_TO_SERVER, \u0026wreq-\u003eflags);\n \t\t\twreq-\u003enetfs_ops-\u003ebegin_writeback(wreq);\n+\t\t\tif (wreq-\u003eio_streams[0].avail) {\n+\t\t\t\tparams.notes |= NOTE_UPLOAD_AVAIL;\n+\t\t\t\t/* Order setting the active flag after other fields. */\n+\t\t\t\tsmp_store_release(\u0026wreq-\u003eio_streams[0].active, true);\n+\t\t\t}\n \t\t}\n \n-\t\terror = netfs_write_folio(wreq, wbc, folio);\n-\t\tif (error == -ENOMEM) {\n-\t\t\tfolio_redirty_for_writepage(wbc, folio);\n-\t\t\tfolio_unlock(folio);\n-\t\t}\n+\t\tparams.notes \u0026= NOTES__KEEP_MASK;\n+\t\tnetfs_queue_wb_folio(wreq, wbc, folio, \u0026params);\n+\t\terror = netfs_issue_streams(wreq, \u0026params);\n+\t\tif (!error)\n+\t\t\tbvecq_pos_step(\u0026params.dispatch_cursor);\n+end_loop:\n \t} while ((folio = writeback_iter(mapping, wbc, folio, \u0026error)));\n \n-\tnetfs_end_issue_write(wreq);\n+\tnetfs_end_issue_write(wreq, \u0026params);\n+\n+\tbvecq_pos_unset(\u0026wreq-\u003eload_cursor);\n+\tbvecq_pos_unset(\u0026params.dispatch_cursor);\n+\tfor (int i = 0; i \u003c NR_IO_STREAMS; i++)\n+\t\tbvecq_pos_unset(\u0026wreq-\u003eio_streams[i].dispatch_cursor);\n \tnetfs_wake_collector(wreq);\n \n \tnetfs_put_request(wreq, netfs_rreq_trace_put_return);\n \t_leave(\" = %d\", error);\n \treturn error;\n \n+nomem:\n+\terror = -ENOMEM;\n+\tnetfs_put_failed_request(wreq);\n couldnt_start:\n \tif (error == -ENOMEM) {\n \t\tfolio_redirty_for_writepage(wbc, folio);\n@@ -612,212 +817,36 @@ int netfs_writepages(struct address_space *mapping,\n EXPORT_SYMBOL(netfs_writepages);\n \n /*\n- * Begin a write operation for writing through the pagecache.\n- */\n-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len)\n-{\n-\tstruct netfs_io_request *wreq = NULL;\n-\tstruct netfs_inode *ictx = netfs_inode(file_inode(iocb-\u003eki_filp));\n-\n-\tnetfs_wb_begin(ictx, false);\n-\n-\twreq = netfs_create_write_req(iocb-\u003eki_filp-\u003ef_mapping, iocb-\u003eki_filp,\n-\t\t\t\t iocb-\u003eki_pos, NETFS_WRITETHROUGH);\n-\tif (IS_ERR(wreq)) {\n-\t\tnetfs_wb_end(ictx);\n-\t\treturn wreq;\n-\t}\n-\n-\twreq-\u003eio_streams[0].avail = true;\n-\t__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, \u0026wreq-\u003eflags);\n-\ttrace_netfs_write(wreq, netfs_write_trace_writethrough);\n-\treturn wreq;\n-}\n-\n-/*\n- * Advance the state of the write operation used when writing through the\n- * pagecache. Data has been copied into the pagecache that we need to append\n- * to the request. If we've added more than wsize then we need to create a new\n- * subrequest.\n+ * Prepare a buffer for a single monolithic write.\n */\n-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,\n-\t\t\t struct folio *folio, size_t copied, bool to_page_end,\n-\t\t\t struct folio **writethrough_cache)\n+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t\t unsigned int max_segs)\n {\n-\tint ret;\n-\n-\t_enter(\"R=%x ic=%zu ws=%u cp=%zu tp=%u\",\n-\t wreq-\u003edebug_id, wreq-\u003ebuffer.iter.count, wreq-\u003ewsize, copied, to_page_end);\n-\n-\t/* The folio is locked. */\n-\n-\tif (*writethrough_cache != folio) {\n-\t\tif (*writethrough_cache) {\n-\t\t\t/* Did the folio get moved? */\n-\t\t\tfolio_put(*writethrough_cache);\n-\t\t\t*writethrough_cache = NULL;\n-\t\t}\n-\t\t/* We can make multiple writes to the folio... */\n-\t\tif (wreq-\u003elen == 0)\n-\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_wthru);\n-\t\telse\n-\t\t\ttrace_netfs_folio(folio, netfs_folio_trace_wthru_plus);\n-\t\t*writethrough_cache = folio;\n-\t\tfolio_get(folio);\n-\t}\n-\n-\twreq-\u003elen += copied;\n-\n-\tif (!to_page_end) {\n-\t\tfolio_mark_dirty(folio);\n-\t\tfolio_unlock(folio);\n-\t\treturn 0;\n-\t}\n-\n-\tret = netfs_write_folio(wreq, wbc, folio);\n-\tfolio_put(*writethrough_cache);\n-\t*writethrough_cache = NULL;\n-\twreq-\u003esubmitted = wreq-\u003elen;\n-\treturn ret;\n-}\n-\n-/*\n- * End a write operation used when writing through the pagecache.\n- */\n-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,\n-\t\t\t struct folio *writethrough_cache)\n-{\n-\tssize_t ret;\n-\n-\t_enter(\"R=%x\", wreq-\u003edebug_id);\n-\n-\tif (writethrough_cache) {\n-\t\tfolio_lock(writethrough_cache);\n-\t\tnetfs_write_folio(wreq, wbc, writethrough_cache);\n-\t\tfolio_put(writethrough_cache);\n-\t\twreq-\u003esubmitted = wreq-\u003elen;\n-\t}\n-\n-\tnetfs_end_issue_write(wreq);\n-\n-\tif (wreq-\u003eiocb)\n-\t\tret = -EIOCBQUEUED;\n-\telse\n-\t\tret = netfs_wait_for_write(wreq);\n-\tnetfs_put_request(wreq, netfs_rreq_trace_put_return);\n-\treturn ret;\n-}\n-\n-/*\n- * Write some of a pending folio data back to the server and/or the cache.\n- */\n-static int netfs_write_folio_single(struct netfs_io_request *wreq,\n-\t\t\t\t struct folio *folio)\n-{\n-\tstruct netfs_io_stream *upload = \u0026wreq-\u003eio_streams[0];\n-\tstruct netfs_io_stream *cache = \u0026wreq-\u003eio_streams[1];\n-\tstruct netfs_io_stream *stream;\n-\tsize_t iter_off = 0;\n-\tsize_t fsize = folio_size(folio), flen;\n-\tloff_t fpos = folio_pos(folio);\n-\tbool to_eof = false;\n-\tbool no_debug = false;\n-\n-\t_enter(\"\");\n-\n-\tflen = folio_size(folio);\n-\tif (flen \u003e wreq-\u003ei_size - fpos) {\n-\t\tflen = wreq-\u003ei_size - fpos;\n-\t\tfolio_zero_segment(folio, flen, fsize);\n-\t\tto_eof = true;\n-\t} else if (flen == wreq-\u003ei_size - fpos) {\n-\t\tto_eof = true;\n-\t}\n-\n-\t_debug(\"folio %zx/%zx\", flen, fsize);\n-\n-\tif (!upload-\u003eavail \u0026\u0026 !cache-\u003eavail) {\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_cancel_store);\n-\t\treturn 0;\n-\t}\n-\n-\tif (!upload-\u003econstruct)\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_store);\n-\telse\n-\t\ttrace_netfs_folio(folio, netfs_folio_trace_store_plus);\n-\n-\t/* Attach the folio to the rolling buffer. */\n-\tfolio_get(folio);\n-\trolling_buffer_append(\u0026wreq-\u003ebuffer, folio, NETFS_ROLLBUF_PUT_MARK);\n-\n-\t/* Move the submission point forward to allow for write-streaming data\n-\t * not starting at the front of the page. We don't do write-streaming\n-\t * with the cache as the cache requires DIO alignment.\n-\t *\n-\t * Also skip uploading for data that's been read and just needs copying\n-\t * to the cache.\n-\t */\n-\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n-\t\tstream = \u0026wreq-\u003eio_streams[s];\n-\t\tstream-\u003esubmit_off = 0;\n-\t\tstream-\u003esubmit_len = flen;\n-\t\tif (!stream-\u003eavail) {\n-\t\t\tstream-\u003esubmit_off = UINT_MAX;\n-\t\t\tstream-\u003esubmit_len = 0;\n-\t\t}\n-\t}\n-\n-\t/* Attach the folio to one or more subrequests. For a big folio, we\n-\t * could end up with thousands of subrequests if the wsize is small -\n-\t * but we might need to wait during the creation of subrequests for\n-\t * network resources (eg. SMB credits).\n-\t */\n-\tfor (;;) {\n-\t\tssize_t part;\n-\t\tsize_t lowest_off = ULONG_MAX;\n-\t\tint choose_s = -1;\n-\n-\t\t/* Always add to the lowest-submitted stream first. */\n-\t\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n-\t\t\tstream = \u0026wreq-\u003eio_streams[s];\n-\t\t\tif (stream-\u003esubmit_len \u003e 0 \u0026\u0026\n-\t\t\t stream-\u003esubmit_off \u003c lowest_off) {\n-\t\t\t\tlowest_off = stream-\u003esubmit_off;\n-\t\t\t\tchoose_s = s;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (choose_s \u003c 0)\n-\t\t\tbreak;\n-\t\tstream = \u0026wreq-\u003eio_streams[choose_s];\n-\n-\t\t/* Advance the iterator(s). */\n-\t\tif (stream-\u003esubmit_off \u003e iter_off) {\n-\t\t\trolling_buffer_advance(\u0026wreq-\u003ebuffer, stream-\u003esubmit_off - iter_off);\n-\t\t\titer_off = stream-\u003esubmit_off;\n-\t\t}\n-\n-\t\tatomic64_set(\u0026wreq-\u003eissued_to, fpos + stream-\u003esubmit_off);\n-\t\tstream-\u003esubmit_extendable_to = fsize - stream-\u003esubmit_off;\n-\t\tpart = netfs_advance_write(wreq, stream, fpos + stream-\u003esubmit_off,\n-\t\t\t\t\t stream-\u003esubmit_len, to_eof);\n-\t\tstream-\u003esubmit_off += part;\n-\t\tif (part \u003e stream-\u003esubmit_len)\n-\t\t\tstream-\u003esubmit_len = 0;\n-\t\telse\n-\t\t\tstream-\u003esubmit_len -= part;\n-\t\tif (part \u003e 0)\n-\t\t\tno_debug = true;\n+\tstruct netfs_io_request *wreq = subreq-\u003erreq;\n+\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[subreq-\u003estream_nr];\n+\tstruct bio_vec *bv;\n+\tstruct bvecq *bq;\n+\tsize_t dio_size = wreq-\u003ecache_resources.dio_size;\n+\tsize_t dlen;\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026stream-\u003edispatch_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026subreq-\u003edispatch_pos);\n+\n+\t/* Round the end of the last entry up. */\n+\tbq = subreq-\u003econtent.bvecq;\n+\twhile (bq-\u003enext)\n+\t\tbq = bq-\u003enext;\n+\tbv = \u0026bq-\u003ebv[bq-\u003enr_slots - 1];\n+\tdlen = round_up(bv-\u003ebv_len, dio_size);\n+\tif (dlen \u003e bv-\u003ebv_len) {\n+\t\tsubreq-\u003elen += dlen - bv-\u003ebv_len;\n+\t\tbv-\u003ebv_len = dlen;\n \t}\n \n-\twreq-\u003ebuffer.iter.iov_offset = 0;\n-\tif (fsize \u003e iter_off)\n-\t\trolling_buffer_advance(\u0026wreq-\u003ebuffer, fsize - iter_off);\n-\tatomic64_set(\u0026wreq-\u003eissued_to, fpos + fsize);\n-\n-\tif (!no_debug)\n-\t\tkdebug(\"R=%x: No submit\", wreq-\u003edebug_id);\n-\t_leave(\" = 0\");\n+\tstream-\u003ebuffered = 0;\n+\tstream-\u003eissue_from = subreq-\u003elen;\n+\twreq-\u003esubmitted = subreq-\u003elen;\n+\tnetfs_all_subreqs_queued(wreq);\n \treturn 0;\n }\n \n@@ -825,26 +854,29 @@ static int netfs_write_folio_single(struct netfs_io_request *wreq,\n * netfs_writeback_single - Write back a monolithic payload\n * @mapping: The mapping to write from\n * @wbc: Hints from the VM\n- * @iter: Data to write, must be ITER_FOLIOQ.\n+ * @iter: Data to write\n+ * @len: Amount of data to write\n *\n * Write a monolithic, non-pagecache object back to the server and/or\n- * the cache.\n+ * the cache. There's a maximum of one subrequest per stream.\n *\n * Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE;\n * or a negative error code.\n+ * the cache. There's a maximum of one subrequest per stream.\n */\n int netfs_writeback_single(struct address_space *mapping,\n \t\t\t struct writeback_control *wbc,\n-\t\t\t struct iov_iter *iter)\n+\t\t\t struct iov_iter *iter,\n+\t\t\t size_t len)\n {\n \tstruct netfs_io_request *wreq;\n \tstruct netfs_inode *ictx = netfs_inode(mapping-\u003ehost);\n-\tstruct folio_queue *fq;\n-\tsize_t size = iov_iter_count(iter);\n \tint ret;\n \n-\tif (WARN_ON_ONCE(!iov_iter_is_folioq(iter)))\n-\t\treturn -EIO;\n+\t_enter(\"%zx,%zx\", iov_iter_count(iter), len);\n+\n+\tif (!len)\n+\t\treturn 0;\n \n \tif (!netfs_wb_begin(ictx, wbc-\u003esync_mode == WB_SYNC_NONE)) {\n \t\t/* The VFS will have undirtied the inode. */\n@@ -858,41 +890,72 @@ int netfs_writeback_single(struct address_space *mapping,\n \t\tgoto couldnt_start;\n \t}\n \n+\twreq-\u003elen = len;\n+\n+\tret = netfs_extract_iter(iter, len, INT_MAX, 0, \u0026wreq-\u003eload_cursor.bvecq, 0);\n+\tif (ret \u003c 0)\n+\t\tgoto cleanup_free;\n+\tif (ret \u003c len) {\n+\t\tret = -EIO;\n+\t\tgoto cleanup_free;\n+\t}\n+\n+\tbvecq_pos_set(\u0026wreq-\u003ecollect_cursor, \u0026wreq-\u003eload_cursor);\n+\n \t__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, \u0026wreq-\u003eflags);\n \ttrace_netfs_write(wreq, netfs_write_trace_writeback_single);\n \tnetfs_stat(\u0026netfs_n_wh_writepages);\n \n-\tif (__test_and_set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, \u0026wreq-\u003eflags))\n+\tif (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, \u0026wreq-\u003eflags))\n \t\twreq-\u003enetfs_ops-\u003ebegin_writeback(wreq);\n \n-\tfor (fq = (struct folio_queue *)iter-\u003efolioq; fq; fq = fq-\u003enext) {\n-\t\tfor (int slot = 0; slot \u003c folioq_count(fq); slot++) {\n-\t\t\tstruct folio *folio = folioq_folio(fq, slot);\n-\t\t\tsize_t part = umin(folioq_folio_size(fq, slot), size);\n+\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++) {\n+\t\tstruct netfs_io_subrequest *subreq;\n+\t\tstruct netfs_io_stream *stream = \u0026wreq-\u003eio_streams[s];\n \n-\t\t\t_debug(\"wbiter %lx %llx\", folio-\u003eindex, atomic64_read(\u0026wreq-\u003eissued_to));\n+\t\tif (!stream-\u003eavail)\n+\t\t\tcontinue;\n \n-\t\t\tret = netfs_write_folio_single(wreq, folio);\n-\t\t\tif (ret \u003c 0)\n-\t\t\t\tgoto stop;\n-\t\t\tsize -= part;\n-\t\t\tif (size \u003c= 0)\n-\t\t\t\tgoto stop;\n+\t\tstream-\u003eissue_from = 0;\n+\t\tstream-\u003ebuffered = len;\n+\n+\t\tsubreq = netfs_alloc_write_subreq(wreq, stream);\n+\t\tif (!subreq) {\n+\t\t\tret = -ENOMEM;\n+\t\t\tbreak;\n \t\t}\n+\n+\t\tbvecq_pos_set(\u0026stream-\u003edispatch_cursor, \u0026wreq-\u003eload_cursor);\n+\n+\t\tret = stream-\u003eissue_write(subreq);\n+\t\tif (ret \u003c 0) {\n+\t\t\t/* Ownership of subreq was returned to us. */\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\tstream-\u003ebuffered -= subreq-\u003elen;\n+\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n+\t\t\t/* Punt the error to the collector. */\n+\t\t}\n+\n+\t\tbvecq_pos_unset(\u0026stream-\u003edispatch_cursor);\n \t}\n \n-stop:\n-\tfor (int s = 0; s \u003c NR_IO_STREAMS; s++)\n-\t\tnetfs_issue_write(wreq, \u0026wreq-\u003eio_streams[s]);\n-\tsmp_wmb(); /* Write lists before ALL_QUEUED. */\n-\tset_bit(NETFS_RREQ_ALL_QUEUED, \u0026wreq-\u003eflags);\n+\twreq-\u003esubmitted = wreq-\u003elen;\n+\tif (unlikely(!netfs_are_all_subreqs_queued(wreq))) {\n+\t\tnetfs_all_subreqs_queued(wreq);\n+\t\tnetfs_wake_collector(wreq);\n+\t}\n \n-\tnetfs_wake_collector(wreq);\n+\t/* TODO: Might want to be async here if WB_SYNC_NONE, but then need to\n+\t * wait before modifying.\n+\t */\n+\tret = netfs_wait_for_write(wreq);\n \n \tnetfs_put_request(wreq, netfs_rreq_trace_put_return);\n \t_leave(\" = %d\", ret);\n \treturn ret;\n \n+cleanup_free:\n+\tnetfs_put_failed_request(wreq);\n couldnt_start:\n \tnetfs_wb_end(ictx);\n \t_leave(\" = %d\", ret);\ndiff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c\nindex 058bc7a166a59f..4baac57f577417 100644\n--- a/fs/netfs/write_retry.c\n+++ b/fs/netfs/write_retry.c\n@@ -12,52 +12,66 @@\n #include \"internal.h\"\n \n /*\n- * Perform retries on the streams that need it.\n+ * Prepare the write buffer for a retry. We can't necessarily reuse the write\n+ * buffer from the previous run of a subrequest because the filesystem is\n+ * permitted to modify it (add headers/trailers, encrypt it). Further, the\n+ * subrequest may now be a different size (e.g. cifs has to negotiate for\n+ * maximum transfer size). Also, we can't look at *stream as that may still\n+ * refer to the source material being broken up into original subrequests.\n+ */\n+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,\n+\t\t\t\t unsigned int max_segs)\n+{\n+\tstruct netfs_io_request *wreq = subreq-\u003erreq;\n+\tsize_t len;\n+\n+\tbvecq_pos_set(\u0026subreq-\u003edispatch_pos, \u0026wreq-\u003eretry_cursor);\n+\tbvecq_pos_set(\u0026subreq-\u003econtent, \u0026wreq-\u003eretry_cursor);\n+\tlen = bvecq_slice(\u0026wreq-\u003eretry_cursor, subreq-\u003elen, max_segs, \u0026subreq-\u003enr_segs);\n+\n+\tif (len \u003c subreq-\u003elen) {\n+\t\tsubreq-\u003elen = len;\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_limited);\n+\t}\n+\n+\twreq-\u003eretry_start += len;\n+\twreq-\u003eretry_buffered -= len;\n+\tif (wreq-\u003eretry_buffered == 0)\n+\t\tbvecq_pos_unset(\u0026wreq-\u003eretry_cursor);\n+\treturn 0;\n+}\n+\n+/*\n+ * Perform retries on the streams that need it. This only has to deal with\n+ * buffered writes; unbuffered write retry is handled in direct_write.c.\n */\n static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t\t\t struct netfs_io_stream *stream)\n {\n \tstruct list_head *next;\n+\tint ret;\n \n \t_enter(\"R=%x[%x:]\", wreq-\u003edebug_id, stream-\u003estream_nr);\n \n \tif (list_empty(\u0026stream-\u003esubrequests))\n \t\treturn;\n+\tif (WARN_ON_ONCE(stream-\u003esource != NETFS_UPLOAD_TO_SERVER))\n+\t\treturn; /* Shouldn't be retrying cache writes. */\n \n-\tif (stream-\u003esource == NETFS_UPLOAD_TO_SERVER \u0026\u0026\n-\t wreq-\u003enetfs_ops-\u003eretry_request)\n+\tif (wreq-\u003enetfs_ops-\u003eretry_request)\n \t\twreq-\u003enetfs_ops-\u003eretry_request(wreq, stream);\n \n \tif (unlikely(stream-\u003efailed))\n \t\treturn;\n \n-\t/* If there's no renegotiation to do, just resend each failed subreq. */\n-\tif (!stream-\u003eprepare_write) {\n-\t\tstruct netfs_io_subrequest *subreq;\n-\n-\t\tlist_for_each_entry(subreq, \u0026stream-\u003esubrequests, rreq_link) {\n-\t\t\tif (test_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags))\n-\t\t\t\tbreak;\n-\t\t\tif (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags)) {\n-\t\t\t\tstruct iov_iter source;\n-\n-\t\t\t\tnetfs_reset_iter(subreq);\n-\t\t\t\tsource = subreq-\u003eio_iter;\n-\t\t\t\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n-\t\t\t\tnetfs_reissue_write(stream, subreq, \u0026source);\n-\t\t\t}\n-\t\t}\n-\t\treturn;\n-\t}\n-\n-\tnext = stream-\u003esubrequests.next;\n+\t/* Read pointer to subreq before reading subreq state. */\n+\tnext = smp_load_acquire(\u0026stream-\u003esubrequests.next);\n \n \tdo {\n \t\tstruct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;\n-\t\tstruct iov_iter source;\n \t\tunsigned long long start, len;\n-\t\tsize_t part;\n-\t\tbool boundary = false;\n+\n+\t\tbvecq_pos_unset(\u0026wreq-\u003eretry_cursor);\n \n \t\t/* Go through the stream and find the next span of contiguous\n \t\t * data that we then rejig (cifs, for example, needs the wsize\n@@ -70,7 +84,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \n \t\tif (test_bit(NETFS_SREQ_FAILED, \u0026from-\u003eflags) ||\n \t\t !test_bit(NETFS_SREQ_NEED_RETRY, \u0026from-\u003eflags))\n-\t\t\treturn;\n+\t\t\tgoto out;\n \n \t\tfor (;;) {\n \t\t\t/* Read pointer to subreq before reading subreq state. */\n@@ -79,8 +93,8 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t\t\tbreak;\n \n \t\t\tsubreq = list_entry(next, struct netfs_io_subrequest, rreq_link);\n-\t\t\tif (subreq-\u003estart + subreq-\u003etransferred != start + len ||\n-\t\t\t test_bit(NETFS_SREQ_BOUNDARY, \u0026subreq-\u003eflags) ||\n+\t\t\tif (subreq-\u003estart != start + len ||\n+\t\t\t subreq-\u003etransferred \u003e 0 ||\n \t\t\t !test_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags))\n \t\t\t\tbreak;\n \t\t\tto = subreq;\n@@ -90,38 +104,48 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t/* Determine the set of buffers we're going to use. Each\n \t\t * subreq gets a subset of a single overall contiguous buffer.\n \t\t */\n-\t\tnetfs_reset_iter(from);\n-\t\tsource = from-\u003eio_iter;\n-\t\tsource.count = len;\n-\n-\t\t/* Work through the sublist. */\n+\t\tbvecq_pos_transfer(\u0026wreq-\u003eretry_cursor, \u0026from-\u003edispatch_pos);\n+\t\tbvecq_pos_advance(\u0026wreq-\u003eretry_cursor, from-\u003etransferred);\n+\t\twreq-\u003eretry_start = start;\n+\t\twreq-\u003eretry_buffered = len;\n+\n+\t\t/* Work through the sublist. The chain of buffers we're going\n+\t\t * to fill is attached to dispatch_cursor and we need to read\n+\t\t * 'len' amount of data from 'start'.\n+\t\t */\n \t\tsubreq = from;\n \t\tlist_for_each_entry_from(subreq, \u0026stream-\u003esubrequests, rreq_link) {\n-\t\t\tif (!len)\n+\t\t\tif (!wreq-\u003eretry_buffered)\n \t\t\t\tbreak;\n \n-\t\t\tsubreq-\u003estart\t= start;\n-\t\t\tsubreq-\u003elen\t= len;\n+\t\t\tbvecq_pos_unset(\u0026subreq-\u003econtent);\n+\t\t\tbvecq_pos_unset(\u0026subreq-\u003edispatch_pos);\n+\n+\t\t\t__clear_bit(NETFS_SREQ_MADE_PROGRESS, \u0026subreq-\u003eflags);\n \t\t\t__clear_bit(NETFS_SREQ_NEED_RETRY, \u0026subreq-\u003eflags);\n+\t\t\t__clear_bit(NETFS_SREQ_FAILED, \u0026subreq-\u003eflags);\n+\t\t\t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n+\t\t\tsubreq-\u003estart\t\t= wreq-\u003eretry_start;\n+\t\t\tsubreq-\u003elen\t\t= wreq-\u003eretry_buffered;\n+\t\t\tsubreq-\u003etransferred\t= 0;\n+\t\t\tsubreq-\u003eretry_count\t+= 1;\n+\t\t\tsubreq-\u003eerror\t\t= 0;\n+\n+\t\t\tnetfs_stat(\u0026netfs_n_wh_retry_write_subreq);\n \t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n-\n-\t\t\t/* Renegotiate max_len (wsize) */\n-\t\t\tstream-\u003esreq_max_len = len;\n-\t\t\tstream-\u003eprepare_write(subreq);\n-\n-\t\t\tpart = umin(len, stream-\u003esreq_max_len);\n-\t\t\tif (unlikely(stream-\u003esreq_max_segs))\n-\t\t\t\tpart = netfs_limit_iter(\u0026source, 0, part, stream-\u003esreq_max_segs);\n-\t\t\tsubreq-\u003elen = part;\n-\t\t\tsubreq-\u003etransferred = 0;\n-\t\t\tlen -= part;\n-\t\t\tstart += part;\n-\t\t\tif (len \u0026\u0026 subreq == to \u0026\u0026\n-\t\t\t __test_and_clear_bit(NETFS_SREQ_BOUNDARY, \u0026to-\u003eflags))\n-\t\t\t\tboundary = true;\n-\n \t\t\tnetfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);\n-\t\t\tnetfs_reissue_write(stream, subreq, \u0026source);\n+\t\t\tret = stream-\u003eissue_write(subreq);\n+\t\t\tif (ret \u003c 0) {\n+\t\t\t\t/* Ownership of subreq was returned to us.\n+\t\t\t\t * Expand the subreq to consume the entire\n+\t\t\t\t * remaining amount to be retried and fail it.\n+\t\t\t\t */\n+\t\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\t\twreq-\u003eretry_buffered -= subreq-\u003elen;\n+\t\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n \t\t\tif (subreq == to)\n \t\t\t\tbreak;\n \t\t}\n@@ -129,7 +153,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t/* If we managed to use fewer subreqs, we can discard the\n \t\t * excess; if we used the same number, then we're done.\n \t\t */\n-\t\tif (!len) {\n+\t\tif (!wreq-\u003eretry_buffered) {\n \t\t\tif (subreq == to)\n \t\t\t\tcontinue;\n \t\t\tlist_for_each_entry_safe_from(subreq, tmp,\n@@ -152,6 +176,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t\tsubreq = netfs_alloc_subrequest(wreq);\n \t\t\tsubreq-\u003esource\t\t= to-\u003esource;\n \t\t\tsubreq-\u003estart\t\t= start;\n+\t\t\tsubreq-\u003elen\t\t= wreq-\u003eretry_buffered;\n \t\t\tsubreq-\u003estream_nr\t= to-\u003estream_nr;\n \t\t\tsubreq-\u003eretry_count\t= 1;\n \n@@ -160,44 +185,40 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,\n \t\t\t\t\t netfs_sreq_trace_new);\n \t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_split);\n \n+\t\t\t__set_bit(NETFS_SREQ_IN_PROGRESS, \u0026subreq-\u003eflags);\n \t\t\tspin_lock(\u0026wreq-\u003elock);\n+\t\t\t/* Write IN_PROGRESS before pointer to new subreq */\n+\t\t\tsmp_wmb();\n \t\t\tlist_add(\u0026subreq-\u003erreq_link, \u0026to-\u003erreq_link);\n \t\t\tspin_unlock(\u0026wreq-\u003elock);\n \t\t\tto = subreq;\n-\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n \n-\t\t\tstream-\u003esreq_max_len\t= len;\n-\t\t\tstream-\u003esreq_max_segs\t= INT_MAX;\n \t\t\tswitch (stream-\u003esource) {\n \t\t\tcase NETFS_UPLOAD_TO_SERVER:\n \t\t\t\tnetfs_stat(\u0026netfs_n_wh_upload);\n-\t\t\t\tstream-\u003esreq_max_len = umin(len, wreq-\u003ewsize);\n-\t\t\t\tbreak;\n-\t\t\tcase NETFS_WRITE_TO_CACHE:\n-\t\t\t\tnetfs_stat(\u0026netfs_n_wh_write);\n \t\t\t\tbreak;\n \t\t\tdefault:\n \t\t\t\tWARN_ON_ONCE(1);\n \t\t\t}\n \n-\t\t\tstream-\u003eprepare_write(subreq);\n-\n-\t\t\tpart = umin(len, stream-\u003esreq_max_len);\n-\t\t\tsubreq-\u003elen = subreq-\u003etransferred + part;\n-\t\t\tlen -= part;\n-\t\t\tstart += part;\n-\t\t\tif (!len \u0026\u0026 boundary) {\n-\t\t\t\t__set_bit(NETFS_SREQ_BOUNDARY, \u0026to-\u003eflags);\n-\t\t\t\tboundary = false;\n-\t\t\t}\n-\n-\t\t\tnetfs_reissue_write(stream, subreq, \u0026source);\n-\t\t\tif (!len)\n+\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n+\t\t\tret = stream-\u003eissue_write(subreq);\n+\t\t\tif (ret \u003c 0) {\n+\t\t\t\t/* Ownership of subreq was returned to us.\n+\t\t\t\t * Expand the subreq to consume the entire\n+\t\t\t\t * remaining amount to be retried and fail it.\n+\t\t\t\t */\n+\t\t\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n+\t\t\t\twreq-\u003eretry_buffered -= subreq-\u003elen;\n+\t\t\t\tnetfs_write_subrequest_terminated(subreq, ret);\n \t\t\t\tbreak;\n-\n-\t\t} while (len);\n+\t\t\t}\n+\t\t} while (wreq-\u003eretry_buffered \u003e 0);\n \n \t} while (!list_is_head(next, \u0026stream-\u003esubrequests));\n+\n+out:\n+\tbvecq_pos_unset(\u0026wreq-\u003eretry_cursor);\n }\n \n /*\n@@ -211,6 +232,7 @@ void netfs_retry_writes(struct netfs_io_request *wreq)\n \tint s;\n \n \tnetfs_stat(\u0026netfs_n_wh_retry_write_req);\n+\ttrace_netfs_rreq(wreq, netfs_rreq_trace_retry_begin);\n \n \t/* Wait for all outstanding I/O to quiesce before performing retries as\n \t * we may need to renegotiate the I/O sizes.\n@@ -235,4 +257,6 @@ void netfs_retry_writes(struct netfs_io_request *wreq)\n \t\t\tnetfs_retry_write_stream(wreq, stream);\n \t\t}\n \t}\n+\n+\ttrace_netfs_rreq(wreq, netfs_rreq_trace_retry_end);\n }\ndiff --git a/fs/nfs/Kconfig b/fs/nfs/Kconfig\nindex 6bb30543eff00f..e7862f35b72c8c 100644\n--- a/fs/nfs/Kconfig\n+++ b/fs/nfs/Kconfig\n@@ -174,6 +174,7 @@ config NFS_FSCACHE\n \tbool \"Provide NFS client caching support\"\n \tdepends on NFS_FS\n \tselect NETFS_SUPPORT\n+\tselect NETFS_PGPRIV2\n \tselect FSCACHE\n \thelp\n \t Say Y here if you want NFS data to be cached locally on disc through\ndiff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c\nindex 9b7fdad4a92019..fb1441f88661a4 100644\n--- a/fs/nfs/fscache.c\n+++ b/fs/nfs/fscache.c\n@@ -23,6 +23,7 @@\n #include \"iostat.h\"\n #include \"fscache.h\"\n #include \"nfstrace.h\"\n+#include \u003ctrace/events/netfs.h\u003e\n \n #define NFS_MAX_KEY_LEN 1000\n \n@@ -273,8 +274,6 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fi\n \trreq-\u003edebug_id = atomic_inc_return(\u0026nfs_netfs_debug_id);\n \t/* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */\n \t__set_bit(NETFS_RREQ_USE_PGPRIV2, \u0026rreq-\u003eflags);\n-\trreq-\u003eio_streams[0].sreq_max_len = NFS_SB(rreq-\u003einode-\u003ei_sb)-\u003ersize;\n-\n \treturn 0;\n }\n \n@@ -296,8 +295,9 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sre\n \treturn netfs;\n }\n \n-static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)\n+static int nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)\n {\n+\tstruct netfs_io_request\t\t*rreq = sreq-\u003erreq;\n \tstruct nfs_netfs_io_data\t*netfs;\n \tstruct nfs_pageio_descriptor\tpgio;\n \tstruct inode *inode = sreq-\u003erreq-\u003einode;\n@@ -307,6 +307,16 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)\n \tpgoff_t start, last;\n \tint err;\n \n+\tif (sreq-\u003elen \u003e NFS_SB(rreq-\u003einode-\u003ei_sb)-\u003ersize)\n+\t\tsreq-\u003elen = NFS_SB(rreq-\u003einode-\u003ei_sb)-\u003ersize;\n+\n+\terr = netfs_prepare_read_buffer(sreq, INT_MAX);\n+\tif (err \u003c 0) {\n+\t\tsreq-\u003eerror = err;\n+\t\treturn err;\n+\t}\n+\t/* After this point, must fail by termination. */\n+\n \tstart = (sreq-\u003estart + sreq-\u003etransferred) \u003e\u003e PAGE_SHIFT;\n \tlast = ((sreq-\u003estart + sreq-\u003elen - sreq-\u003etransferred - 1) \u003e\u003e PAGE_SHIFT);\n \n@@ -316,12 +326,15 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)\n \tnetfs = nfs_netfs_alloc(sreq);\n \tif (!netfs) {\n \t\tsreq-\u003eerror = -ENOMEM;\n-\t\treturn netfs_read_subreq_terminated(sreq);\n+\t\tnetfs_read_subreq_terminated(sreq);\n+\t\treturn 0;\n \t}\n \n+\ttrace_netfs_sreq(sreq, netfs_sreq_trace_submit);\n+\n \tpgio.pg_netfs = netfs; /* used in completion */\n \n-\txa_for_each_range(\u0026sreq-\u003erreq-\u003emapping-\u003ei_pages, idx, page, start, last) {\n+\txa_for_each_range(\u0026rreq-\u003emapping-\u003ei_pages, idx, page, start, last) {\n \t\t/* nfs_read_add_folio() may schedule() due to pNFS layout and other RPCs */\n \t\terr = nfs_read_add_folio(\u0026pgio, ctx, page_folio(page));\n \t\tif (err \u003c 0) {\n@@ -332,6 +345,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)\n out:\n \tnfs_pageio_complete_read(\u0026pgio);\n \tnfs_netfs_put(netfs);\n+\treturn 0;\n }\n \n void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr)\ndiff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h\nindex 08e94633a9c1e2..e06cae4267dc39 100644\n--- a/fs/smb/client/cifsglob.h\n+++ b/fs/smb/client/cifsglob.h\n@@ -289,7 +289,7 @@ struct smb_rqst {\n \tstruct kvec\t*rq_iov;\t/* array of kvecs */\n \tunsigned int\trq_nvec;\t/* number of kvecs in array */\n \tstruct iov_iter\trq_iter;\t/* Data iterator */\n-\tstruct folio_queue *rq_buffer;\t/* Buffer for encryption */\n+\tstruct bvecq\t*rq_buffer;\t/* Buffer for encryption */\n };\n \n struct mid_q_entry;\ndiff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c\nindex 40162d5554eacc..fec824c1b3c596 100644\n--- a/fs/smb/client/cifssmb.c\n+++ b/fs/smb/client/cifssmb.c\n@@ -1467,8 +1467,7 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)\n \tstruct cifs_tcon *tcon = tlink_tcon(rdata-\u003ereq-\u003ecfile-\u003etlink);\n \tstruct inode *inode = \u0026ictx-\u003einode;\n \tstruct smb_rqst rqst = { .rq_iov = rdata-\u003eiov,\n-\t\t\t\t .rq_nvec = 1,\n-\t\t\t\t .rq_iter = rdata-\u003esubreq.io_iter };\n+\t\t\t\t .rq_nvec = 1};\n \tstruct cifs_credits credits = {\n \t\t.value = 1,\n \t\t.instance = 0,\n@@ -1482,6 +1481,11 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)\n \t\t __func__, mid-\u003emid, mid-\u003emid_state, rdata-\u003eresult,\n \t\t rdata-\u003esubreq.len);\n \n+\tif (rdata-\u003egot_bytes)\n+\t\tiov_iter_bvec_queue(\u0026rqst.rq_iter, ITER_DEST,\n+\t\t\t\t rdata-\u003esubreq.content.bvecq, rdata-\u003esubreq.content.slot,\n+\t\t\t\t rdata-\u003esubreq.content.offset, rdata-\u003esubreq.len);\n+\n \tswitch (mid-\u003emid_state) {\n \tcase MID_RESPONSE_RECEIVED:\n \t\t/* result already set, check signature */\n@@ -2003,7 +2007,10 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)\n \n \trqst.rq_iov = iov;\n \trqst.rq_nvec = 1;\n-\trqst.rq_iter = wdata-\u003esubreq.io_iter;\n+\n+\tiov_iter_bvec_queue(\u0026rqst.rq_iter, ITER_SOURCE,\n+\t\t\t wdata-\u003esubreq.content.bvecq, wdata-\u003esubreq.content.slot,\n+\t\t\t wdata-\u003esubreq.content.offset, wdata-\u003esubreq.len);\n \n \tcifs_dbg(FYI, \"async write at %llu %zu bytes\\n\",\n \t\t wdata-\u003esubreq.start, wdata-\u003esubreq.len);\ndiff --git a/fs/smb/client/file.c b/fs/smb/client/file.c\nindex 968740e7c9c3aa..c7dd10b6de327d 100644\n--- a/fs/smb/client/file.c\n+++ b/fs/smb/client/file.c\n@@ -43,18 +43,34 @@ static int cifs_reopen_file(struct cifsFileInfo *cfile, bool can_flush);\n * Prepare a subrequest to upload to the server. We need to allocate credits\n * so that we know the maximum amount of data that we can include in it.\n */\n-static void cifs_prepare_write(struct netfs_io_subrequest *subreq)\n+static int cifs_estimate_write(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_write_estimate *estimate)\n+{\n+\tstruct cifs_sb_info *cifs_sb = CIFS_SB(wreq-\u003einode-\u003ei_sb);\n+\n+\testimate-\u003eissue_at = stream-\u003eissue_from + cifs_sb-\u003ectx-\u003ewsize;\n+\treturn 0;\n+}\n+\n+/*\n+ * Issue a subrequest to upload to the server.\n+ */\n+static int cifs_issue_write(struct netfs_io_subrequest *subreq)\n {\n \tstruct cifs_io_subrequest *wdata =\n \t\tcontainer_of(subreq, struct cifs_io_subrequest, subreq);\n \tstruct cifs_io_request *req = wdata-\u003ereq;\n-\tstruct netfs_io_stream *stream = \u0026req-\u003erreq.io_streams[subreq-\u003estream_nr];\n \tstruct TCP_Server_Info *server;\n \tstruct cifsFileInfo *open_file = req-\u003ecfile;\n-\tstruct cifs_sb_info *cifs_sb = CIFS_SB(wdata-\u003erreq-\u003einode-\u003ei_sb);\n-\tsize_t wsize = req-\u003erreq.wsize;\n+\tstruct cifs_sb_info *cifs_sb = CIFS_SB(subreq-\u003erreq-\u003einode-\u003ei_sb);\n+\tunsigned int max_segs = INT_MAX;\n+\tsize_t len;\n \tint rc;\n \n+\tif (cifs_forced_shutdown(cifs_sb))\n+\t\treturn smb_EIO(smb_eio_trace_forced_shutdown);\n+\n \tif (!wdata-\u003ehave_xid) {\n \t\twdata-\u003exid = get_xid();\n \t\twdata-\u003ehave_xid = true;\n@@ -73,18 +89,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)\n \t\tif (rc \u003c 0) {\n \t\t\tif (rc == -EAGAIN)\n \t\t\t\tgoto retry;\n-\t\t\tsubreq-\u003eerror = rc;\n-\t\t\treturn netfs_prepare_write_failed(subreq);\n+\t\t\treturn rc;\n \t\t}\n \t}\n \n-\trc = server-\u003eops-\u003ewait_mtu_credits(server, wsize, \u0026stream-\u003esreq_max_len,\n-\t\t\t\t\t \u0026wdata-\u003ecredits);\n-\tif (rc \u003c 0) {\n-\t\tsubreq-\u003eerror = rc;\n-\t\treturn netfs_prepare_write_failed(subreq);\n-\t}\n+\tlen = umin(subreq-\u003elen, cifs_sb-\u003ectx-\u003ewsize);\n+\trc = server-\u003eops-\u003ewait_mtu_credits(server, len, \u0026len, \u0026wdata-\u003ecredits);\n+\tif (rc \u003c 0)\n+\t\treturn rc;\n \n+\tsubreq-\u003elen = len;\n \twdata-\u003ecredits.rreq_debug_id = subreq-\u003erreq-\u003edebug_id;\n \twdata-\u003ecredits.rreq_debug_index = subreq-\u003edebug_index;\n \twdata-\u003ecredits.in_flight_check = 1;\n@@ -100,46 +114,34 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)\n \t\tconst struct smbdirect_socket_parameters *sp =\n \t\t\tsmbd_get_parameters(server-\u003esmbd_conn);\n \n-\t\tstream-\u003esreq_max_segs = sp-\u003emax_frmr_depth;\n+\t\tmax_segs = sp-\u003emax_frmr_depth;\n \t}\n #endif\n-}\n \n-/*\n- * Issue a subrequest to upload to the server.\n- */\n-static void cifs_issue_write(struct netfs_io_subrequest *subreq)\n-{\n-\tstruct cifs_io_subrequest *wdata =\n-\t\tcontainer_of(subreq, struct cifs_io_subrequest, subreq);\n-\tstruct cifs_sb_info *sbi = CIFS_SB(subreq-\u003erreq-\u003einode-\u003ei_sb);\n-\tint rc;\n-\n-\tif (cifs_forced_shutdown(sbi)) {\n-\t\trc = smb_EIO(smb_eio_trace_forced_shutdown);\n-\t\tgoto fail;\n+\trc = netfs_prepare_write_buffer(subreq, max_segs);\n+\tif (rc \u003c 0) {\n+\t\tadd_credits_and_wake_if(wdata-\u003eserver, \u0026wdata-\u003ecredits, 0);\n+\t\treturn rc;\n \t}\n+\t/* After this point, must fail by termination. */\n \n-\trc = adjust_credits(wdata-\u003eserver, wdata, cifs_trace_rw_credits_issue_write_adjust);\n+\trc = adjust_credits(server, wdata, cifs_trace_rw_credits_issue_write_adjust);\n \tif (rc)\n-\t\tgoto fail;\n+\t\tgoto fail_with_credits;\n \n \trc = -EAGAIN;\n \tif (wdata-\u003ereq-\u003ecfile-\u003einvalidHandle)\n-\t\tgoto fail;\n+\t\tgoto fail_with_credits;\n \n \twdata-\u003eserver-\u003eops-\u003easync_writev(wdata);\n-out:\n-\treturn;\n+\treturn 0;\n \n-fail:\n+fail_with_credits:\n \tif (rc == -EAGAIN)\n \t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n-\telse\n-\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n \tadd_credits_and_wake_if(wdata-\u003eserver, \u0026wdata-\u003ecredits, 0);\n \tcifs_write_subrequest_terminated(wdata, rc);\n-\tgoto out;\n+\treturn 0;\n }\n \n static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)\n@@ -148,17 +150,25 @@ static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)\n }\n \n /*\n- * Negotiate the size of a read operation on behalf of the netfs library.\n+ * Issue a read operation on behalf of the netfs helper functions. We're asked\n+ * to make a read of a certain size at a point in the file. We are permitted\n+ * to only read a portion of that, but as long as we read something, the netfs\n+ * helper will call us again so that we can issue another read.\n */\n-static int cifs_prepare_read(struct netfs_io_subrequest *subreq)\n+static int cifs_issue_read(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *rreq = subreq-\u003erreq;\n \tstruct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);\n \tstruct cifs_io_request *req = container_of(subreq-\u003erreq, struct cifs_io_request, rreq);\n-\tstruct TCP_Server_Info *server;\n+\tstruct TCP_Server_Info *server = rdata-\u003eserver;\n \tstruct cifs_sb_info *cifs_sb = CIFS_SB(rreq-\u003einode-\u003ei_sb);\n-\tsize_t size;\n-\tint rc = 0;\n+\tunsigned int max_segs = INT_MAX;\n+\tsize_t len;\n+\tint rc;\n+\n+\tcifs_dbg(FYI, \"%s: op=%08x[%x] mapping=%p len=%zu/%zu\\n\",\n+\t\t __func__, rreq-\u003edebug_id, subreq-\u003edebug_index, rreq-\u003emapping,\n+\t\t subreq-\u003etransferred, subreq-\u003elen);\n \n \tif (!rdata-\u003ehave_xid) {\n \t\trdata-\u003exid = get_xid();\n@@ -172,17 +182,15 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)\n \t\tcifs_negotiate_rsize(server, cifs_sb-\u003ectx,\n \t\t\t\t tlink_tcon(req-\u003ecfile-\u003etlink));\n \n-\trc = server-\u003eops-\u003ewait_mtu_credits(server, cifs_sb-\u003ectx-\u003ersize,\n-\t\t\t\t\t \u0026size, \u0026rdata-\u003ecredits);\n+\tlen = umin(subreq-\u003elen, cifs_sb-\u003ectx-\u003ersize);\n+\trc = server-\u003eops-\u003ewait_mtu_credits(server, len, \u0026len, \u0026rdata-\u003ecredits);\n \tif (rc)\n \t\treturn rc;\n \n-\trreq-\u003eio_streams[0].sreq_max_len = size;\n-\n-\trdata-\u003ecredits.in_flight_check = 1;\n+\tsubreq-\u003elen = len;\n \trdata-\u003ecredits.rreq_debug_id = rreq-\u003edebug_id;\n \trdata-\u003ecredits.rreq_debug_index = subreq-\u003edebug_index;\n-\n+\trdata-\u003ecredits.in_flight_check = 1;\n \ttrace_smb3_rw_credits(rdata-\u003erreq-\u003edebug_id,\n \t\t\t rdata-\u003esubreq.debug_index,\n \t\t\t rdata-\u003ecredits.value,\n@@ -194,40 +202,27 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)\n \t\tconst struct smbdirect_socket_parameters *sp =\n \t\t\tsmbd_get_parameters(server-\u003esmbd_conn);\n \n-\t\trreq-\u003eio_streams[0].sreq_max_segs = sp-\u003emax_frmr_depth;\n+\t\tmax_segs = sp-\u003emax_frmr_depth;\n \t}\n #endif\n-\treturn 0;\n-}\n-\n-/*\n- * Issue a read operation on behalf of the netfs helper functions. We're asked\n- * to make a read of a certain size at a point in the file. We are permitted\n- * to only read a portion of that, but as long as we read something, the netfs\n- * helper will call us again so that we can issue another read.\n- */\n-static void cifs_issue_read(struct netfs_io_subrequest *subreq)\n-{\n-\tstruct netfs_io_request *rreq = subreq-\u003erreq;\n-\tstruct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);\n-\tstruct cifs_io_request *req = container_of(subreq-\u003erreq, struct cifs_io_request, rreq);\n-\tstruct TCP_Server_Info *server = rdata-\u003eserver;\n-\tint rc = 0;\n \n-\tcifs_dbg(FYI, \"%s: op=%08x[%x] mapping=%p len=%zu/%zu\\n\",\n-\t\t __func__, rreq-\u003edebug_id, subreq-\u003edebug_index, rreq-\u003emapping,\n-\t\t subreq-\u003etransferred, subreq-\u003elen);\n+\trc = netfs_prepare_read_buffer(subreq, max_segs);\n+\tif (rc \u003c 0) {\n+\t\tadd_credits_and_wake_if(rdata-\u003eserver, \u0026rdata-\u003ecredits, 0);\n+\t\treturn rc;\n+\t}\n+\t/* After this point, must fail by termination. */\n \n \trc = adjust_credits(server, rdata, cifs_trace_rw_credits_issue_read_adjust);\n \tif (rc)\n-\t\tgoto failed;\n+\t\tgoto fail_with_credits;\n \n \tif (req-\u003ecfile-\u003einvalidHandle) {\n \t\tdo {\n \t\t\trc = cifs_reopen_file(req-\u003ecfile, true);\n \t\t} while (rc == -EAGAIN);\n \t\tif (rc)\n-\t\t\tgoto failed;\n+\t\t\tgoto fail_with_credits;\n \t}\n \n \tif (subreq-\u003erreq-\u003eorigin != NETFS_UNBUFFERED_READ \u0026\u0026\n@@ -235,15 +230,22 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)\n \t\t__set_bit(NETFS_SREQ_CLEAR_TAIL, \u0026subreq-\u003eflags);\n \n \ttrace_netfs_sreq(subreq, netfs_sreq_trace_submit);\n+\n \trc = rdata-\u003eserver-\u003eops-\u003easync_readv(rdata);\n \tif (rc)\n \t\tgoto failed;\n-\treturn;\n+\treturn 0;\n \n+fail_with_credits:\n+\tif (rc == -EAGAIN)\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_retry);\n+\telse\n+\t\ttrace_netfs_sreq(subreq, netfs_sreq_trace_fail);\n failed:\n \tadd_credits_and_wake_if(rdata-\u003eserver, \u0026rdata-\u003ecredits, 0);\n \tsubreq-\u003eerror = rc;\n \tnetfs_read_subreq_terminated(subreq);\n+\treturn 0;\n }\n \n /*\n@@ -358,11 +360,10 @@ const struct netfs_request_ops cifs_req_ops = {\n \t.init_request\t\t= cifs_init_request,\n \t.free_request\t\t= cifs_free_request,\n \t.free_subrequest\t= cifs_free_subrequest,\n-\t.prepare_read\t\t= cifs_prepare_read,\n \t.issue_read\t\t= cifs_issue_read,\n \t.done\t\t\t= cifs_rreq_done,\n \t.begin_writeback\t= cifs_begin_writeback,\n-\t.prepare_write\t\t= cifs_prepare_write,\n+\t.estimate_write\t\t= cifs_estimate_write,\n \t.issue_write\t\t= cifs_issue_write,\n \t.invalidate_cache\t= cifs_netfs_invalidate_cache,\n };\ndiff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c\nindex cbd51a08e97e6e..28553b74333a50 100644\n--- a/fs/smb/client/smb2ops.c\n+++ b/fs/smb/client/smb2ops.c\n@@ -13,7 +13,6 @@\n #include \u003clinux/sort.h\u003e\n #include \u003ccrypto/aead.h\u003e\n #include \u003clinux/fiemap.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n #include \u003cuapi/linux/magic.h\u003e\n #include \"cifsfs.h\"\n #include \"cifsglob.h\"\n@@ -4683,19 +4682,18 @@ crypt_message(struct TCP_Server_Info *server, int num_rqst,\n }\n \n /*\n- * Copy data from an iterator to the folios in a folio queue buffer.\n+ * Copy data from an iterator to the pages in a bvec queue buffer.\n */\n-static bool cifs_copy_iter_to_folioq(struct iov_iter *iter, size_t size,\n-\t\t\t\t struct folio_queue *buffer)\n+static bool cifs_copy_iter_to_bvecq(struct iov_iter *iter, size_t size,\n+\t\t\t\t struct bvecq *buffer)\n {\n \tfor (; buffer; buffer = buffer-\u003enext) {\n-\t\tfor (int s = 0; s \u003c folioq_count(buffer); s++) {\n-\t\t\tstruct folio *folio = folioq_folio(buffer, s);\n-\t\t\tsize_t part = folioq_folio_size(buffer, s);\n+\t\tfor (int s = 0; s \u003c buffer-\u003enr_slots; s++) {\n+\t\t\tstruct bio_vec *bv = \u0026buffer-\u003ebv[s];\n+\t\t\tsize_t part = umin(bv-\u003ebv_len, size);\n \n-\t\t\tpart = umin(part, size);\n-\n-\t\t\tif (copy_folio_from_iter(folio, 0, part, iter) != part)\n+\t\t\tif (copy_page_from_iter(bv-\u003ebv_page, bv-\u003ebv_offset,\n+\t\t\t\t\t\tpart, iter) != part)\n \t\t\t\treturn false;\n \t\t\tsize -= part;\n \t\t}\n@@ -4707,7 +4705,7 @@ void\n smb3_free_compound_rqst(int num_rqst, struct smb_rqst *rqst)\n {\n \tfor (int i = 0; i \u003c num_rqst; i++)\n-\t\tnetfs_free_folioq_buffer(rqst[i].rq_buffer);\n+\t\tbvecq_put(rqst[i].rq_buffer);\n }\n \n /*\n@@ -4734,7 +4732,7 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,\n \tfor (int i = 1; i \u003c num_rqst; i++) {\n \t\tstruct smb_rqst *old = \u0026old_rq[i - 1];\n \t\tstruct smb_rqst *new = \u0026new_rq[i];\n-\t\tstruct folio_queue *buffer = NULL;\n+\t\tstruct bvecq *buffer = NULL;\n \t\tsize_t size = iov_iter_count(\u0026old-\u003erq_iter);\n \n \t\torig_len += smb_rqst_len(server, old);\n@@ -4742,17 +4740,16 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,\n \t\tnew-\u003erq_nvec = old-\u003erq_nvec;\n \n \t\tif (size \u003e 0) {\n-\t\t\tsize_t cur_size = 0;\n-\t\t\trc = netfs_alloc_folioq_buffer(NULL, \u0026buffer, \u0026cur_size,\n-\t\t\t\t\t\t size, GFP_NOFS);\n-\t\t\tif (rc \u003c 0)\n+\t\t\trc = -ENOMEM;\n+\t\t\tbuffer = bvecq_alloc_buffer(size, GFP_NOFS);\n+\t\t\tif (!buffer)\n \t\t\t\tgoto err_free;\n \n \t\t\tnew-\u003erq_buffer = buffer;\n-\t\t\tiov_iter_folio_queue(\u0026new-\u003erq_iter, ITER_SOURCE,\n-\t\t\t\t\t buffer, 0, 0, size);\n+\t\t\tiov_iter_bvec_queue(\u0026new-\u003erq_iter, ITER_SOURCE,\n+\t\t\t\t\t buffer, 0, 0, size);\n \n-\t\t\tif (!cifs_copy_iter_to_folioq(\u0026old-\u003erq_iter, size, buffer)) {\n+\t\t\tif (!cifs_copy_iter_to_bvecq(\u0026old-\u003erq_iter, size, buffer)) {\n \t\t\t\trc = smb_EIO1(smb_eio_trace_tx_copy_iter_to_buf, size);\n \t\t\t\tgoto err_free;\n \t\t\t}\n@@ -4842,22 +4839,20 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,\n }\n \n static int\n-cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,\n-\t\t\t size_t skip, struct iov_iter *iter)\n+cifs_copy_bvecq_to_iter(struct bvecq *bq, size_t data_size,\n+\t\t\tsize_t skip, struct iov_iter *iter)\n {\n-\tfor (; folioq; folioq = folioq-\u003enext) {\n-\t\tfor (int s = 0; s \u003c folioq_count(folioq); s++) {\n-\t\t\tstruct folio *folio;\n-\t\t\tsize_t fsize, n, len;\n+\tfor (; bq; bq = bq-\u003enext) {\n+\t\tfor (int s = 0; s \u003c bq-\u003enr_slots; s++) {\n+\t\t\tstruct bio_vec *bv = \u0026bq-\u003ebv[s];\n+\t\t\tsize_t n, len;\n \n \t\t\tif (data_size == 0)\n \t\t\t\treturn 0;\n \n-\t\t\tfolio = folioq_folio(folioq, s);\n-\t\t\tfsize = folio_size(folio);\n-\t\t\tlen = umin(fsize - skip, data_size);\n+\t\t\tlen = umin(bv-\u003ebv_len - skip, data_size);\n \n-\t\t\tn = copy_folio_to_iter(folio, skip, len, iter);\n+\t\t\tn = copy_page_to_iter(bv-\u003ebv_page, bv-\u003ebv_offset + skip, len, iter);\n \t\t\tif (n != len) {\n \t\t\t\tcifs_dbg(VFS, \"%s: something went wrong\\n\", __func__);\n \t\t\t\treturn smb_EIO2(smb_eio_trace_rx_copy_to_iter,\n@@ -4879,7 +4874,7 @@ cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,\n \n static int\n handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n-\t\t char *buf, unsigned int buf_len, struct folio_queue *buffer,\n+\t\t char *buf, unsigned int buf_len, struct bvecq *buffer,\n \t\t unsigned int buffer_len, bool is_offloaded)\n {\n \tunsigned int data_offset;\n@@ -4889,6 +4884,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n \tunsigned int cur_page_idx;\n \tunsigned int pad_len;\n \tstruct cifs_io_subrequest *rdata = mid-\u003ecallback_data;\n+\tstruct iov_iter iter;\n \tstruct smb2_hdr *shdr = (struct smb2_hdr *)buf;\n \tsize_t copied;\n \tbool use_rdma_mr = false;\n@@ -4961,6 +4957,10 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n \n \tpad_len = data_offset - server-\u003evals-\u003eread_rsp_size;\n \n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST,\n+\t\t\t rdata-\u003esubreq.content.bvecq, rdata-\u003esubreq.content.slot,\n+\t\t\t rdata-\u003esubreq.content.offset, rdata-\u003esubreq.len);\n+\n \tif (buf_len \u003c= data_offset) {\n \t\t/* read response payload is in pages */\n \t\tcur_page_idx = pad_len / PAGE_SIZE;\n@@ -4989,8 +4989,8 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n \t\t}\n \n \t\t/* Copy the data to the output I/O iterator. */\n-\t\trdata-\u003eresult = cifs_copy_folioq_to_iter(buffer, data_len,\n-\t\t\t\t\t\t\t cur_off, \u0026rdata-\u003esubreq.io_iter);\n+\t\trdata-\u003eresult = cifs_copy_bvecq_to_iter(buffer, data_len,\n+\t\t\t\t\t\t\tcur_off, \u0026iter);\n \t\tif (rdata-\u003eresult != 0) {\n \t\t\tif (is_offloaded)\n \t\t\t\tmid-\u003emid_state = MID_RESPONSE_MALFORMED;\n@@ -5004,7 +5004,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n \t\t buf_len \u003e= end_off) {\n \t\t/* read response payload is in buf */\n \t\tWARN_ONCE(buffer, \"read data can be either in buf or in buffer\");\n-\t\tcopied = copy_to_iter(buf + data_offset, data_len, \u0026rdata-\u003esubreq.io_iter);\n+\t\tcopied = copy_to_iter(buf + data_offset, data_len, \u0026iter);\n \t\tif (copied == 0)\n \t\t\treturn smb_EIO2(smb_eio_trace_rx_copy_to_iter, copied, data_len);\n \t\trdata-\u003egot_bytes = copied;\n@@ -5029,7 +5029,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,\n struct smb2_decrypt_work {\n \tstruct work_struct decrypt;\n \tstruct TCP_Server_Info *server;\n-\tstruct folio_queue *buffer;\n+\tstruct bvecq *buffer;\n \tchar *buf;\n \tunsigned int len;\n };\n@@ -5043,7 +5043,7 @@ static void smb2_decrypt_offload(struct work_struct *work)\n \tstruct mid_q_entry *mid;\n \tstruct iov_iter iter;\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_DEST, dw-\u003ebuffer, 0, 0, dw-\u003elen);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, dw-\u003ebuffer, 0, 0, dw-\u003elen);\n \trc = decrypt_raw_data(dw-\u003eserver, dw-\u003ebuf, dw-\u003eserver-\u003evals-\u003eread_rsp_size,\n \t\t\t \u0026iter, true);\n \tif (rc) {\n@@ -5092,7 +5092,7 @@ static void smb2_decrypt_offload(struct work_struct *work)\n \t}\n \n free_pages:\n-\tnetfs_free_folioq_buffer(dw-\u003ebuffer);\n+\tbvecq_put(dw-\u003ebuffer);\n \tcifs_small_buf_release(dw-\u003ebuf);\n \tkfree(dw);\n }\n@@ -5138,12 +5138,12 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,\n \tdw-\u003elen = len;\n \tlen = round_up(dw-\u003elen, PAGE_SIZE);\n \n-\tsize_t cur_size = 0;\n-\trc = netfs_alloc_folioq_buffer(NULL, \u0026dw-\u003ebuffer, \u0026cur_size, len, GFP_NOFS);\n-\tif (rc \u003c 0)\n+\trc = -ENOMEM;\n+\tdw-\u003ebuffer = bvecq_alloc_buffer(len, GFP_NOFS);\n+\tif (!dw-\u003ebuffer)\n \t\tgoto discard_data;\n \n-\tiov_iter_folio_queue(\u0026iter, ITER_DEST, dw-\u003ebuffer, 0, 0, len);\n+\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, dw-\u003ebuffer, 0, 0, len);\n \n \t/* Read the data into the buffer and clear excess bufferage. */\n \trc = cifs_read_iter_from_socket(server, \u0026iter, dw-\u003elen);\n@@ -5201,7 +5201,7 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,\n \t}\n \n free_pages:\n-\tnetfs_free_folioq_buffer(dw-\u003ebuffer);\n+\tbvecq_put(dw-\u003ebuffer);\n free_dw:\n \tkfree(dw);\n \treturn rc;\ndiff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c\nindex 4ce165e40657f3..085fdc7d8bec09 100644\n--- a/fs/smb/client/smb2pdu.c\n+++ b/fs/smb/client/smb2pdu.c\n@@ -4592,9 +4592,13 @@ smb2_new_read_req(void **buf, unsigned int *total_len,\n \t */\n \tif (rdata \u0026\u0026 smb3_use_rdma_offload(io_parms)) {\n \t\tstruct smbdirect_buffer_descriptor_v1 *v1;\n+\t\tstruct iov_iter iter;\n \t\tbool need_invalidate = server-\u003edialect == SMB30_PROT_ID;\n \n-\t\trdata-\u003emr = smbd_register_mr(server-\u003esmbd_conn, \u0026rdata-\u003esubreq.io_iter,\n+\t\tiov_iter_bvec_queue(\u0026iter, ITER_DEST,\n+\t\t\t\t rdata-\u003esubreq.content.bvecq, rdata-\u003esubreq.content.slot,\n+\t\t\t\t rdata-\u003esubreq.content.offset, rdata-\u003esubreq.len);\n+\t\trdata-\u003emr = smbd_register_mr(server-\u003esmbd_conn, \u0026iter,\n \t\t\t\t\t true, need_invalidate);\n \t\tif (!rdata-\u003emr)\n \t\t\treturn -EAGAIN;\n@@ -4658,9 +4662,10 @@ smb2_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)\n \tunsigned int rreq_debug_id = rdata-\u003erreq-\u003edebug_id;\n \tunsigned int subreq_debug_index = rdata-\u003esubreq.debug_index;\n \n-\tif (rdata-\u003egot_bytes) {\n-\t\trqst.rq_iter\t = rdata-\u003esubreq.io_iter;\n-\t}\n+\tif (rdata-\u003egot_bytes)\n+\t\tiov_iter_bvec_queue(\u0026rqst.rq_iter, ITER_DEST,\n+\t\t\t\t rdata-\u003esubreq.content.bvecq, rdata-\u003esubreq.content.slot,\n+\t\t\t\t rdata-\u003esubreq.content.offset, rdata-\u003esubreq.len);\n \n \tWARN_ONCE(rdata-\u003eserver != server,\n \t\t \"rdata server %p != mid server %p\",\n@@ -5148,7 +5153,9 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)\n \t\tgoto out;\n \n \trqst.rq_iov = iov;\n-\trqst.rq_iter = wdata-\u003esubreq.io_iter;\n+\tiov_iter_bvec_queue(\u0026rqst.rq_iter, ITER_SOURCE,\n+\t\t\t wdata-\u003esubreq.content.bvecq, wdata-\u003esubreq.content.slot,\n+\t\t\t wdata-\u003esubreq.content.offset, wdata-\u003esubreq.len);\n \n \trqst.rq_iov[0].iov_len = total_len - 1;\n \trqst.rq_iov[0].iov_base = (char *)req;\n@@ -5187,9 +5194,14 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)\n \t */\n \tif (smb3_use_rdma_offload(io_parms)) {\n \t\tstruct smbdirect_buffer_descriptor_v1 *v1;\n+\t\tstruct iov_iter iter;\n \t\tbool need_invalidate = server-\u003edialect == SMB30_PROT_ID;\n \n-\t\twdata-\u003emr = smbd_register_mr(server-\u003esmbd_conn, \u0026wdata-\u003esubreq.io_iter,\n+\t\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE,\n+\t\t\t\t wdata-\u003esubreq.content.bvecq, wdata-\u003esubreq.content.slot,\n+\t\t\t\t wdata-\u003esubreq.content.offset, wdata-\u003esubreq.len);\n+\n+\t\twdata-\u003emr = smbd_register_mr(server-\u003esmbd_conn, \u0026iter,\n \t\t\t\t\t false, need_invalidate);\n \t\tif (!wdata-\u003emr) {\n \t\t\trc = -EAGAIN;\n@@ -5226,8 +5238,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)\n \t\tsmb2_set_replay(server, \u0026rqst);\n \t}\n \n-\tcifs_dbg(FYI, \"async write at %llu %u bytes iter=%zx\\n\",\n-\t\t io_parms-\u003eoffset, io_parms-\u003elength, iov_iter_count(\u0026wdata-\u003esubreq.io_iter));\n+\tcifs_dbg(FYI, \"async write at %llu %u bytes len=%zx\\n\",\n+\t\t io_parms-\u003eoffset, io_parms-\u003elength, wdata-\u003esubreq.len);\n \n \tif (wdata-\u003ecredits.value \u003e 0) {\n \t\tshdr-\u003eCreditCharge = cpu_to_le16(DIV_ROUND_UP(wdata-\u003esubreq.len,\ndiff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c\nindex fdf4e50c27ceb4..be2f6b909c34bd 100644\n--- a/fs/smb/client/transport.c\n+++ b/fs/smb/client/transport.c\n@@ -1267,12 +1267,19 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid)\n \t}\n \n #ifdef CONFIG_CIFS_SMB_DIRECT\n-\tif (rdata-\u003emr)\n+\tif (rdata-\u003emr) {\n \t\tlength = data_len; /* An RDMA read is already done. */\n-\telse\n+\t} else {\n+#endif\n+\t\tstruct iov_iter iter;\n+\n+\t\tiov_iter_bvec_queue(\u0026iter, ITER_DEST, rdata-\u003esubreq.content.bvecq,\n+\t\t\t\t rdata-\u003esubreq.content.slot, rdata-\u003esubreq.content.offset,\n+\t\t\t\t data_len);\n+\t\tlength = cifs_read_iter_from_socket(server, \u0026iter, data_len);\n+#ifdef CONFIG_CIFS_SMB_DIRECT\n+\t}\n #endif\n-\t\tlength = cifs_read_iter_from_socket(server, \u0026rdata-\u003esubreq.io_iter,\n-\t\t\t\t\t\t data_len);\n \tif (length \u003e 0)\n \t\trdata-\u003egot_bytes += length;\n \tserver-\u003etotal_read += length;\ndiff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c\nindex 8adf580975344b..f217cf127cf9ba 100644\n--- a/fs/smb/smbdirect/connection.c\n+++ b/fs/smb/smbdirect/connection.c\n@@ -5,7 +5,7 @@\n */\n \n #include \"internal.h\"\n-#include \u003clinux/folio_queue.h\u003e\n+#include \u003clinux/bvecq.h\u003e\n \n struct smbdirect_map_sges {\n \tstruct ib_sge *sge;\n@@ -2006,6 +2006,69 @@ static ssize_t smbdirect_map_sges_from_bvec(struct iov_iter *iter,\n \treturn ret;\n }\n \n+/*\n+ * Extract memory fragments from a BVECQ-class iterator and add them to an RDMA\n+ * list. The fragments are not pinned.\n+ */\n+static ssize_t smbdirect_map_sges_from_bvecq(struct iov_iter *iter,\n+\t\t\t\t\t struct smbdirect_map_sges *state,\n+\t\t\t\t\t ssize_t maxsize)\n+{\n+\tconst struct bvecq *bq = iter-\u003ebvecq, *next;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n+\tssize_t extracted = 0;\n+\tsize_t offset = iter-\u003eiov_offset;\n+\n+\tmaxsize = umin(maxsize, iov_iter_count(iter));\n+\n+\tdo {\n+\t\tstruct bio_vec *bv;\n+\t\tsize_t bsize;\n+\n+\t\twhile (slot \u003e= bq-\u003enr_slots) {\n+\t\t\tnext = bvecq_next(bq);\n+\t\t\tif (!next) {\n+\t\t\t\tif (WARN_ON_ONCE(maxsize \u003e 0))\n+\t\t\t\t\treturn -EIO;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\tbq = next;\n+\t\t\tslot = 0;\n+\t\t}\n+\n+\t\tbv = \u0026bq-\u003ebv[slot];\n+\t\tbsize = bv-\u003ebv_len;\n+\n+\t\tif (offset \u003c bsize) {\n+\t\t\tsize_t part = umin(maxsize, bsize - offset);\n+\t\t\tbool ok;\n+\n+\t\t\tok = smbdirect_map_sges_single_page(state,\n+\t\t\t\t\t\t\t bv-\u003ebv_page,\n+\t\t\t\t\t\t\t bv-\u003ebv_offset + offset,\n+\t\t\t\t\t\t\t part);\n+\t\t\tif (!ok)\n+\t\t\t\treturn -EIO;\n+\n+\t\t\toffset += part;\n+\t\t\textracted += part;\n+\t\t\tmaxsize -= part;\n+\t\t}\n+\n+\t\tif (offset \u003e= bsize) {\n+\t\t\toffset = 0;\n+\t\t\tslot++;\n+\t\t}\n+\t} while (state-\u003enum_sge \u003c state-\u003emax_sge \u0026\u0026 maxsize \u003e 0);\n+\n+out:\n+\titer-\u003ebvecq = bq;\n+\titer-\u003ebvecq_slot = slot;\n+\titer-\u003eiov_offset = offset;\n+\titer-\u003ecount -= extracted;\n+\treturn extracted;\n+}\n+\n /*\n * Extract fragments from a KVEC-class iterator and add them to an ib_sge list.\n * This can deal with vmalloc'd buffers as well as kmalloc'd or static buffers.\n@@ -2067,70 +2130,6 @@ static ssize_t smbdirect_map_sges_from_kvec(struct iov_iter *iter,\n \treturn ret;\n }\n \n-/*\n- * Extract folio fragments from a FOLIOQ-class iterator and add them to an\n- * ib_sge list. The folios are not pinned.\n- */\n-static ssize_t smbdirect_map_sges_from_folioq(struct iov_iter *iter,\n-\t\t\t\t\t struct smbdirect_map_sges *state,\n-\t\t\t\t\t ssize_t maxsize)\n-{\n-\tconst struct folio_queue *folioq = iter-\u003efolioq;\n-\tunsigned int slot = iter-\u003efolioq_slot;\n-\tssize_t ret = 0;\n-\tsize_t offset = iter-\u003eiov_offset;\n-\n-\tif (WARN_ON_ONCE(!folioq))\n-\t\treturn -EIO;\n-\n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tif (WARN_ON_ONCE(!folioq))\n-\t\t\treturn -EIO;\n-\t\tslot = 0;\n-\t}\n-\n-\tdo {\n-\t\tstruct folio *folio = folioq_folio(folioq, slot);\n-\t\tsize_t fsize = folioq_folio_size(folioq, slot);\n-\n-\t\tif (offset \u003c fsize) {\n-\t\t\tsize_t part = umin(maxsize, fsize - offset);\n-\t\t\tbool ok;\n-\n-\t\t\tok = smbdirect_map_sges_single_page(state,\n-\t\t\t\t\t\t\t folio_page(folio, 0),\n-\t\t\t\t\t\t\t offset,\n-\t\t\t\t\t\t\t part);\n-\t\t\tif (!ok)\n-\t\t\t\treturn -EIO;\n-\n-\t\t\toffset += part;\n-\t\t\tret += part;\n-\t\t\tmaxsize -= part;\n-\t\t}\n-\n-\t\tif (offset \u003e= fsize) {\n-\t\t\toffset = 0;\n-\t\t\tslot++;\n-\t\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\t\tif (!folioq-\u003enext) {\n-\t\t\t\t\tWARN_ON_ONCE(ret \u003c iter-\u003ecount);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\t\t\t\tfolioq = folioq-\u003enext;\n-\t\t\t\tslot = 0;\n-\t\t\t}\n-\t\t}\n-\t} while (state-\u003enum_sge \u003c state-\u003emax_sge \u0026\u0026 maxsize \u003e 0);\n-\n-\titer-\u003efolioq = folioq;\n-\titer-\u003efolioq_slot = slot;\n-\titer-\u003eiov_offset = offset;\n-\titer-\u003ecount -= ret;\n-\treturn ret;\n-}\n-\n /*\n * Extract page fragments from up to the given amount of the source iterator\n * and build up an ib_sge list that refers to all of those bits. The ib_sge list\n@@ -2155,12 +2154,12 @@ static ssize_t smbdirect_map_sges_from_iter(struct iov_iter *iter, size_t len,\n \tcase ITER_BVEC:\n \t\tret = smbdirect_map_sges_from_bvec(iter, state, len);\n \t\tbreak;\n+\tcase ITER_BVECQ:\n+\t\tret = smbdirect_map_sges_from_bvecq(iter, state, len);\n+\t\tbreak;\n \tcase ITER_KVEC:\n \t\tret = smbdirect_map_sges_from_kvec(iter, state, len);\n \t\tbreak;\n-\tcase ITER_FOLIOQ:\n-\t\tret = smbdirect_map_sges_from_folioq(iter, state, len);\n-\t\tbreak;\n \tdefault:\n \t\tWARN_ONCE(1, \"iov_iter_type[%u]\\n\", iov_iter_type(iter));\n \t\treturn -EIO;\ndiff --git a/include/linux/bvec.h b/include/linux/bvec.h\nindex 92837e2743f19a..53cf36e73967f1 100644\n--- a/include/linux/bvec.h\n+++ b/include/linux/bvec.h\n@@ -343,4 +343,22 @@ static inline phys_addr_t bvec_phys(const struct bio_vec *bvec)\n \treturn page_to_phys(bvec-\u003ebv_page) + bvec-\u003ebv_offset;\n }\n \n+/**\n+ * bvec_kmap_partial - Map part of a bvec into the kernel virtual address space\n+ * @bvec: bvec to map\n+ * @offset: Offset into bvec\n+ *\n+ * Map the page containing the byte at @offset into the kernel virtual address\n+ * space. The caller is responsible for making sure this doesn't overrun.\n+ *\n+ * Call kunmap_local on the returned address to unmap.\n+ */\n+static inline void *bvec_kmap_partial(struct bio_vec *bvec, size_t offset)\n+{\n+\toffset += bvec-\u003ebv_offset;\n+\n+\treturn kmap_local_page(bvec-\u003ebv_page + (offset \u003e\u003e PAGE_SHIFT)) +\n+\t\t(offset \u0026 ~PAGE_MASK);\n+}\n+\n #endif /* __LINUX_BVEC_H */\ndiff --git a/include/linux/bvecq.h b/include/linux/bvecq.h\nnew file mode 100644\nindex 00000000000000..fa46be520649c2\n--- /dev/null\n+++ b/include/linux/bvecq.h\n@@ -0,0 +1,358 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Implementation of a segmented queue of bio_vec[].\n+ *\n+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.\n+ * Written by David Howells (dhowells@redhat.com)\n+ */\n+\n+#ifndef _LINUX_BVECQ_H\n+#define _LINUX_BVECQ_H\n+\n+#include \u003clinux/bvec.h\u003e\n+\n+/*\n+ * The type of memory retention used by the elements in bvecq-\u003ebv[] and how to\n+ * clean it up.\n+ */\n+enum bvecq_mem {\n+\tBVECQ_MEM_EXTERNAL,\t/* Externally retained memory - no freeing */\n+\tBVECQ_MEM_PAGECACHE,\t/* Ref'd pagecache pages - must put */\n+\tBVECQ_MEM_GUP,\t\t/* Pinned memory from get_user_pages() - unpin */\n+\tBVECQ_MEM_ALLOCED,\t/* Memory alloc'd by bvecq - can be freed/pooled */\n+} __mode(byte);\n+\n+/*\n+ * Segmented bio_vec queue.\n+ *\n+ * These can be linked together to form messages of indefinite length and\n+ * iterated over with an ITER_BVECQ iterator. The list is non-circular; next\n+ * and prev are NULL at the ends.\n+ *\n+ * The bv pointer points to the bio_vec array; this may be __bv if allocated\n+ * together. The caller is responsible for determining whether or not this is\n+ * the case as the array pointed to by bv may be follow on directly from the\n+ * bvecq by accident of allocation (ie. -\u003ebv == -\u003e__bv is *not* sufficient to\n+ * determine this).\n+ *\n+ * The file position and discontiguity flag allow non-contiguous data sets to\n+ * be chained together, but still teased apart without the need to convert the\n+ * info in the bio_vec back into a folio pointer.\n+ */\n+struct bvecq {\n+\tstruct bvecq\t*next;\t\t/* Next bvec in the list or NULL */\n+\tstruct bvecq\t*prev;\t\t/* Prev bvec in the list or NULL */\n+\tunsigned long long fpos;\t/* File position */\n+\trefcount_t\tref;\n+\tu32\t\tpriv;\t\t/* Private data */\n+\tu16\t\tnr_slots;\t/* Number of elements in bv[] used */\n+\tu16\t\tmax_slots;\t/* Number of elements allocated in bv[] */\n+\tenum bvecq_mem\tmem_type:3;\t/* What sort of memory and how to free it */\n+\tbool\t\tinline_bv:1;\t/* T if __bv[] is being used */\n+\tbool\t\tdiscontig:1;\t/* T if not contiguous with previous bvecq */\n+\tstruct bio_vec\t*bv;\t\t/* Pointer to array of page fragments */\n+\tstruct bio_vec\t__bv[];\t\t/* Default array (if -\u003einline_bv) */\n+};\n+\n+#if BITS_PER_LONG == 64\n+/* Number of slots in __bv[] for a bvecq in a 512-byte kmalloc block. */\n+#define BVECQ_STD_SLOTS\t\t29\t/* 2 words/slot; 32 slots; bvecq is 6 words (3 slots) */\n+#elif BITS_PER_LONG == 32\n+/* Number of slots in __bv[] for a bvecq in a 256-byte kmalloc block. */\n+#define BVECQ_STD_SLOTS\t\t18\t/* 3 words/slot; 21 slots; bvecq is 9 words (3 slots) */\n+#else\n+#error BVECQ_STD_SLOTS undetermined\n+#endif\n+\n+/*\n+ * Position in a bio_vec queue. The bvecq holds a ref on the queue segment it\n+ * points to.\n+ */\n+struct bvecq_pos {\n+\tstruct bvecq\t\t*bvecq;\t\t/* The first bvecq */\n+\tunsigned int\t\toffset;\t\t/* The offset within the starting slot */\n+\tu16\t\t\tslot;\t\t/* The starting slot */\n+};\n+\n+void bvecq_dump(const struct bvecq *bq);\n+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp);\n+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp);\n+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp);\n+void bvecq_put(struct bvecq *bq);\n+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp);\n+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size);\n+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp);\n+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq);\n+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount);\n+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount);\n+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,\n+\t\t unsigned int max_slots, unsigned int *_nr_slots);\n+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size,\n+\t\t unsigned int max_slots, struct bvecq **to);\n+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl);\n+\n+/**\n+ * bvecq_alloc_buffer - Allocate a bvecq chain and populate with buffers\n+ * @size: Target size of the buffer (can be 0 for an empty buffer)\n+ * @gfp: The allocation constraints.\n+ *\n+ * Wrapper around %bvecq_alloc_buffer2().\n+ */\n+static inline struct bvecq *bvecq_alloc_buffer(size_t size, gfp_t gfp)\n+{\n+\treturn bvecq_alloc_buffer2(size, 0, gfp);\n+}\n+\n+/**\n+ * bvecq_get - Get a ref on a bvecq\n+ * @bq: The bvecq to get a ref on\n+ */\n+static inline struct bvecq *bvecq_get(struct bvecq *bq)\n+{\n+\trefcount_inc(\u0026bq-\u003eref);\n+\treturn bq;\n+}\n+\n+/**\n+ * bvecq_is_full - Determine if a bvecq is full\n+ * @bvecq: The object to query\n+ *\n+ * Return: true if full; false if not.\n+ */\n+static inline bool bvecq_is_full(const struct bvecq *bvecq)\n+{\n+\treturn bvecq-\u003enr_slots \u003e= bvecq-\u003emax_slots;\n+}\n+\n+/**\n+ * bvecq_filled_to - Release filled slots with release barrier\n+ * @bvecq: The object modified\n+ * @to: The latest slot filled + 1\n+ */\n+static inline void bvecq_filled_to(struct bvecq *bvecq, unsigned int to)\n+{\n+\t/* Set the slot counter after filling the slot */\n+\tsmp_store_release(\u0026bvecq-\u003enr_slots, to);\n+}\n+\n+/**\n+ * bvecq_nr_slots_acquire - Get the number of filled slots with acquire barrier\n+ * @bvecq: The object to query\n+ *\n+ * Return: The number of filled slots\n+ */\n+static inline unsigned int bvecq_nr_slots_acquire(const struct bvecq *bvecq)\n+{\n+\t/* Read the slot counter before looking at the slot */\n+\treturn smp_load_acquire(\u0026bvecq-\u003enr_slots);\n+}\n+\n+/**\n+ * bvecq_acquire_slot - Determine if a slot is valid with acquire barrier\n+ * @bvecq: The object to query\n+ * @slot: The next slot\n+ *\n+ * Return: true if valid; false if might not be valid\n+ */\n+static inline bool bvecq_acquire_slot(const struct bvecq *bvecq, unsigned int slot)\n+{\n+\t/* Read the slot counter before looking at the slot */\n+\treturn slot \u003c bvecq_nr_slots_acquire(bvecq);\n+}\n+\n+/**\n+ * bvecq_append - Get the next bvecq with appropriate barrier\n+ * @to: The bvecq to append to\n+ * @add: The bvecq to append\n+ *\n+ * Attach a new bvecq to a chain using an appropriate barrier to protect the\n+ * write.\n+ *\n+ * [!] Note that this function transfers the caller's ref to the chain.\n+ */\n+static inline void bvecq_append(struct bvecq *to, struct bvecq *add)\n+{\n+\tadd-\u003eprev = to;\n+\n+\t/* Make sure the initialisation is stored before the next pointer. */\n+\tsmp_store_release(\u0026to-\u003enext, add);\n+}\n+\n+/**\n+ * bvecq_next - Get the next bvecq with appropriate barrier\n+ * @bq: The bvecq to start from\n+ *\n+ * Return the next bvecq in a chain, using an appropriate barrier to protect\n+ * the access.\n+ */\n+static inline struct bvecq *bvecq_next(const struct bvecq *bq)\n+{\n+\t/* Read the contents of the next node after the pointer to it. */\n+\treturn smp_load_acquire(\u0026bq-\u003enext);\n+}\n+\n+/**\n+ * bvecq_pos_set - Set one position to be the same as another\n+ * @pos: The position object to set\n+ * @at: The source position.\n+ *\n+ * Set @pos to have the same position as @at. This may take a ref on the\n+ * bvecq pointed to.\n+ */\n+static inline void bvecq_pos_set(struct bvecq_pos *pos, const struct bvecq_pos *at)\n+{\n+\t*pos = *at;\n+\tbvecq_get(pos-\u003ebvecq);\n+}\n+\n+/**\n+ * bvecq_pos_unset - Unset a position\n+ * @pos: The position object to unset\n+ *\n+ * Unset @pos. This does any needed ref cleanup.\n+ */\n+static inline void bvecq_pos_unset(struct bvecq_pos *pos)\n+{\n+\tbvecq_put(pos-\u003ebvecq);\n+\tpos-\u003ebvecq = NULL;\n+\tpos-\u003eslot = 0;\n+\tpos-\u003eoffset = 0;\n+}\n+\n+/**\n+ * bvecq_pos_transfer - Transfer one position to another, clearing the first\n+ * @pos: The position object to set\n+ * @from: The source position to clear.\n+ *\n+ * Set @pos to have the same position as @from and then clear @from. This may\n+ * transfer a ref on the bvecq pointed to.\n+ */\n+static inline void bvecq_pos_transfer(struct bvecq_pos *pos, struct bvecq_pos *from)\n+{\n+\t*pos = *from;\n+\tfrom-\u003ebvecq = NULL;\n+\tfrom-\u003eslot = 0;\n+\tfrom-\u003eoffset = 0;\n+}\n+\n+/**\n+ * bvecq_pos_move - Update a position to a new bvecq\n+ * @pos: The position object to update.\n+ * @to: The new bvecq to point at.\n+ *\n+ * Update @pos to point to @to if it doesn't already do so. This may\n+ * manipulate refs on the bvecqs pointed to.\n+ */\n+static inline void bvecq_pos_move(struct bvecq_pos *pos, struct bvecq *to)\n+{\n+\tstruct bvecq *old = pos-\u003ebvecq;\n+\n+\tif (old != to) {\n+\t\tpos-\u003ebvecq = bvecq_get(to);\n+\t\tbvecq_put(old);\n+\t}\n+}\n+\n+/**\n+ * bvecq_pos_nudge - Nudge a position onto the next segment if current used up\n+ * @pos: The position object to nudge.\n+ *\n+ * Update @pos to point to the next segment in the chain if we've used up the\n+ * current segment. This may manipulate refs on the bvecqs pointed to.\n+ *\n+ * Return: true if found a new segment, false if hit the end.\n+ */\n+static inline bool bvecq_pos_nudge(struct bvecq_pos *pos)\n+{\n+\tstruct bvecq *bq = pos-\u003ebvecq;\n+\n+\tfor (;;) {\n+\t\tif (!bvecq_acquire_slot(bq, pos-\u003eslot)) {\n+\t\t\tbq = bvecq_next(bq);\n+\t\t\tif (!bq)\n+\t\t\t\treturn false;\n+\t\t\tbvecq_pos_move(pos, bq);\n+\t\t\tpos-\u003eslot = 0;\n+\t\t\tpos-\u003eoffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (pos-\u003eoffset \u003e= bq-\u003ebv[pos-\u003eslot].bv_len) {\n+\t\t\tpos-\u003eslot++;\n+\t\t\tpos-\u003eoffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\treturn true;\n+\t}\n+}\n+\n+/**\n+ * bvecq_pos_step - Step a position to the next slot if possible\n+ * @pos: The position object to step.\n+ *\n+ * Update @pos to point to the next slot in the queue if not at the end. This\n+ * may manipulate refs on the bvecqs pointed to.\n+ *\n+ * Return: true if successful, false if was at the end.\n+ */\n+static inline bool bvecq_pos_step(struct bvecq_pos *pos)\n+{\n+\tstruct bvecq *bq = pos-\u003ebvecq, *next;\n+\n+\tpos-\u003eslot++;\n+\tpos-\u003eoffset = 0;\n+\tif (pos-\u003eslot \u003c bq-\u003enr_slots)\n+\t\treturn true;\n+\tnext = bvecq_next(bq);\n+\tif (!next)\n+\t\treturn false;\n+\tbvecq_pos_move(pos, next);\n+\tpos-\u003eslot = 0;\n+\treturn true;\n+}\n+\n+/**\n+ * bvecq_delete_spent - Delete the bvecq at the front if possible\n+ * @pos: The position object to update.\n+ *\n+ * Delete the used up bvecq at the front of the queue that @pos points to if it\n+ * is not the last node in the queue; if it is the last node in the queue, it\n+ * is kept so that the queue doesn't become detached from the other end. This\n+ * may manipulate refs on the bvecqs pointed to. It is also possible that the\n+ * producer will fill more slots in the current bvecq.\n+ *\n+ * Also, we have to be very careful: the consumer can catch the producer, which\n+ * could lead to us having nothing left in the queue, causing the front and\n+ * back pointers to end up on different tracks. To avoid this, we must always\n+ * keep at least one segment in the queue.\n+ *\n+ * The caller must reload from @pos after calling this.\n+ *\n+ * Return: true if there's more available; false if not.\n+ */\n+static inline bool bvecq_delete_spent(struct bvecq_pos *pos)\n+{\n+\tstruct bvecq *spent = pos-\u003ebvecq;\n+\tstruct bvecq *next;\n+\tunsigned int slot = pos-\u003eslot;\n+\n+again:\n+\t/* Read the contents of the queue node after the pointer to it. */\n+\tnext = bvecq_next(spent);\n+\tif (!next)\n+\t\treturn false; /* Nothing more to consume at the moment. */\n+\tif (slot \u003c bvecq_nr_slots_acquire(spent))\n+\t\treturn true; /* The producer added more. */\n+\tnext-\u003eprev = NULL;\n+\tspent-\u003enext = NULL;\n+\tbvecq_put(spent);\n+\tpos-\u003ebvecq = next; /* We take spent's ref. */\n+\tpos-\u003eslot = 0;\n+\tpos-\u003eoffset = 0;\n+\tif (!bvecq_acquire_slot(next, 0)) {\n+\t\tspent = next;\n+\t\tslot = 0;\n+\t\tgoto again;\n+\t}\n+\treturn true;\n+}\n+\n+#endif /* _LINUX_BVECQ_H */\ndiff --git a/include/linux/folio_queue.h b/include/linux/folio_queue.h\ndeleted file mode 100644\nindex f6d5f1f127c9f3..00000000000000\n--- a/include/linux/folio_queue.h\n+++ /dev/null\n@@ -1,282 +0,0 @@\n-/* SPDX-License-Identifier: GPL-2.0-or-later */\n-/* Queue of folios definitions\n- *\n- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.\n- * Written by David Howells (dhowells@redhat.com)\n- *\n- * See:\n- *\n- *\tDocumentation/core-api/folio_queue.rst\n- *\n- * for a description of the API.\n- */\n-\n-#ifndef _LINUX_FOLIO_QUEUE_H\n-#define _LINUX_FOLIO_QUEUE_H\n-\n-#include \u003clinux/folio_batch.h\u003e\n-#include \u003clinux/mm.h\u003e\n-\n-/*\n- * Segment in a queue of running buffers. Each segment can hold a number of\n- * folios and a portion of the queue can be referenced with the ITER_FOLIOQ\n- * iterator. The possibility exists of inserting non-folio elements into the\n- * queue (such as gaps).\n- *\n- * Explicit prev and next pointers are used instead of a list_head to make it\n- * easier to add segments to tail and remove them from the head without the\n- * need for a lock.\n- */\n-struct folio_queue {\n-\tstruct folio_batch\tvec;\t\t/* Folios in the queue segment */\n-\tu8\t\t\torders[FOLIO_BATCH_SIZE]; /* Order of each folio */\n-\tstruct folio_queue\t*next;\t\t/* Next queue segment or NULL */\n-\tstruct folio_queue\t*prev;\t\t/* Previous queue segment of NULL */\n-\tunsigned long\t\tmarks;\t\t/* 1-bit mark per folio */\n-\tunsigned long\t\tmarks2;\t\t/* Second 1-bit mark per folio */\n-#if FOLIO_BATCH_SIZE \u003e BITS_PER_LONG\n-#error marks is not big enough\n-#endif\n-\tunsigned int\t\trreq_id;\n-\tunsigned int\t\tdebug_id;\n-};\n-\n-/**\n- * folioq_init - Initialise a folio queue segment\n- * @folioq: The segment to initialise\n- * @rreq_id: The request identifier to use in tracelines.\n- *\n- * Initialise a folio queue segment and set an identifier to be used in traces.\n- *\n- * Note that the folio pointers are left uninitialised.\n- */\n-static inline void folioq_init(struct folio_queue *folioq, unsigned int rreq_id)\n-{\n-\tfolio_batch_init(\u0026folioq-\u003evec);\n-\tfolioq-\u003enext = NULL;\n-\tfolioq-\u003eprev = NULL;\n-\tfolioq-\u003emarks = 0;\n-\tfolioq-\u003emarks2 = 0;\n-\tfolioq-\u003erreq_id = rreq_id;\n-\tfolioq-\u003edebug_id = 0;\n-}\n-\n-/**\n- * folioq_nr_slots: Query the capacity of a folio queue segment\n- * @folioq: The segment to query\n- *\n- * Query the number of folios that a particular folio queue segment might hold.\n- * [!] NOTE: This must not be assumed to be the same for every segment!\n- */\n-static inline unsigned int folioq_nr_slots(const struct folio_queue *folioq)\n-{\n-\treturn FOLIO_BATCH_SIZE;\n-}\n-\n-/**\n- * folioq_count: Query the occupancy of a folio queue segment\n- * @folioq: The segment to query\n- *\n- * Query the number of folios that have been added to a folio queue segment.\n- * Note that this is not decreased as folios are removed from a segment.\n- */\n-static inline unsigned int folioq_count(struct folio_queue *folioq)\n-{\n-\treturn folio_batch_count(\u0026folioq-\u003evec);\n-}\n-\n-/**\n- * folioq_full: Query if a folio queue segment is full\n- * @folioq: The segment to query\n- *\n- * Query if a folio queue segment is fully occupied. Note that this does not\n- * change if folios are removed from a segment.\n- */\n-static inline bool folioq_full(struct folio_queue *folioq)\n-{\n-\t//return !folio_batch_space(\u0026folioq-\u003evec);\n-\treturn folioq_count(folioq) \u003e= folioq_nr_slots(folioq);\n-}\n-\n-/**\n- * folioq_is_marked: Check first folio mark in a folio queue segment\n- * @folioq: The segment to query\n- * @slot: The slot number of the folio to query\n- *\n- * Determine if the first mark is set for the folio in the specified slot in a\n- * folio queue segment.\n- */\n-static inline bool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot)\n-{\n-\treturn test_bit(slot, \u0026folioq-\u003emarks);\n-}\n-\n-/**\n- * folioq_mark: Set the first mark on a folio in a folio queue segment\n- * @folioq: The segment to modify\n- * @slot: The slot number of the folio to modify\n- *\n- * Set the first mark for the folio in the specified slot in a folio queue\n- * segment.\n- */\n-static inline void folioq_mark(struct folio_queue *folioq, unsigned int slot)\n-{\n-\tset_bit(slot, \u0026folioq-\u003emarks);\n-}\n-\n-/**\n- * folioq_unmark: Clear the first mark on a folio in a folio queue segment\n- * @folioq: The segment to modify\n- * @slot: The slot number of the folio to modify\n- *\n- * Clear the first mark for the folio in the specified slot in a folio queue\n- * segment.\n- */\n-static inline void folioq_unmark(struct folio_queue *folioq, unsigned int slot)\n-{\n-\tclear_bit(slot, \u0026folioq-\u003emarks);\n-}\n-\n-/**\n- * folioq_is_marked2: Check second folio mark in a folio queue segment\n- * @folioq: The segment to query\n- * @slot: The slot number of the folio to query\n- *\n- * Determine if the second mark is set for the folio in the specified slot in a\n- * folio queue segment.\n- */\n-static inline bool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot)\n-{\n-\treturn test_bit(slot, \u0026folioq-\u003emarks2);\n-}\n-\n-/**\n- * folioq_mark2: Set the second mark on a folio in a folio queue segment\n- * @folioq: The segment to modify\n- * @slot: The slot number of the folio to modify\n- *\n- * Set the second mark for the folio in the specified slot in a folio queue\n- * segment.\n- */\n-static inline void folioq_mark2(struct folio_queue *folioq, unsigned int slot)\n-{\n-\tset_bit(slot, \u0026folioq-\u003emarks2);\n-}\n-\n-/**\n- * folioq_unmark2: Clear the second mark on a folio in a folio queue segment\n- * @folioq: The segment to modify\n- * @slot: The slot number of the folio to modify\n- *\n- * Clear the second mark for the folio in the specified slot in a folio queue\n- * segment.\n- */\n-static inline void folioq_unmark2(struct folio_queue *folioq, unsigned int slot)\n-{\n-\tclear_bit(slot, \u0026folioq-\u003emarks2);\n-}\n-\n-/**\n- * folioq_append: Add a folio to a folio queue segment\n- * @folioq: The segment to add to\n- * @folio: The folio to add\n- *\n- * Add a folio to the tail of the sequence in a folio queue segment, increasing\n- * the occupancy count and returning the slot number for the folio just added.\n- * The folio size is extracted and stored in the queue and the marks are left\n- * unmodified.\n- *\n- * Note that it's left up to the caller to check that the segment capacity will\n- * not be exceeded and to extend the queue.\n- */\n-static inline unsigned int folioq_append(struct folio_queue *folioq, struct folio *folio)\n-{\n-\tunsigned int slot = folioq-\u003evec.nr++;\n-\n-\tfolioq-\u003evec.folios[slot] = folio;\n-\tfolioq-\u003eorders[slot] = folio_order(folio);\n-\treturn slot;\n-}\n-\n-/**\n- * folioq_append_mark: Add a folio to a folio queue segment\n- * @folioq: The segment to add to\n- * @folio: The folio to add\n- *\n- * Add a folio to the tail of the sequence in a folio queue segment, increasing\n- * the occupancy count and returning the slot number for the folio just added.\n- * The folio size is extracted and stored in the queue, the first mark is set\n- * and and the second and third marks are left unmodified.\n- *\n- * Note that it's left up to the caller to check that the segment capacity will\n- * not be exceeded and to extend the queue.\n- */\n-static inline unsigned int folioq_append_mark(struct folio_queue *folioq, struct folio *folio)\n-{\n-\tunsigned int slot = folioq-\u003evec.nr++;\n-\n-\tfolioq-\u003evec.folios[slot] = folio;\n-\tfolioq-\u003eorders[slot] = folio_order(folio);\n-\tfolioq_mark(folioq, slot);\n-\treturn slot;\n-}\n-\n-/**\n- * folioq_folio: Get a folio from a folio queue segment\n- * @folioq: The segment to access\n- * @slot: The folio slot to access\n- *\n- * Retrieve the folio in the specified slot from a folio queue segment. Note\n- * that no bounds check is made and if the slot hasn't been added into yet, the\n- * pointer will be undefined. If the slot has been cleared, NULL will be\n- * returned.\n- */\n-static inline struct folio *folioq_folio(const struct folio_queue *folioq, unsigned int slot)\n-{\n-\treturn folioq-\u003evec.folios[slot];\n-}\n-\n-/**\n- * folioq_folio_order: Get the order of a folio from a folio queue segment\n- * @folioq: The segment to access\n- * @slot: The folio slot to access\n- *\n- * Retrieve the order of the folio in the specified slot from a folio queue\n- * segment. Note that no bounds check is made and if the slot hasn't been\n- * added into yet, the order returned will be 0.\n- */\n-static inline unsigned int folioq_folio_order(const struct folio_queue *folioq, unsigned int slot)\n-{\n-\treturn folioq-\u003eorders[slot];\n-}\n-\n-/**\n- * folioq_folio_size: Get the size of a folio from a folio queue segment\n- * @folioq: The segment to access\n- * @slot: The folio slot to access\n- *\n- * Retrieve the size of the folio in the specified slot from a folio queue\n- * segment. Note that no bounds check is made and if the slot hasn't been\n- * added into yet, the size returned will be PAGE_SIZE.\n- */\n-static inline size_t folioq_folio_size(const struct folio_queue *folioq, unsigned int slot)\n-{\n-\treturn PAGE_SIZE \u003c\u003c folioq_folio_order(folioq, slot);\n-}\n-\n-/**\n- * folioq_clear: Clear a folio from a folio queue segment\n- * @folioq: The segment to clear\n- * @slot: The folio slot to clear\n- *\n- * Clear a folio from a sequence in a folio queue segment and clear its marks.\n- * The occupancy count is left unchanged.\n- */\n-static inline void folioq_clear(struct folio_queue *folioq, unsigned int slot)\n-{\n-\tfolioq-\u003evec.folios[slot] = NULL;\n-\tfolioq_unmark(folioq, slot);\n-\tfolioq_unmark2(folioq, slot);\n-}\n-\n-#endif /* _LINUX_FOLIO_QUEUE_H */\ndiff --git a/include/linux/fscache.h b/include/linux/fscache.h\nindex 58fdb9605425dc..850d20241075bf 100644\n--- a/include/linux/fscache.h\n+++ b/include/linux/fscache.h\n@@ -147,6 +147,23 @@ struct fscache_cookie {\n \t};\n };\n \n+enum fscache_extent_type {\n+\tFSCACHE_EXTENT_DATA,\n+\tFSCACHE_EXTENT_ZERO,\n+} __mode(byte);\n+\n+/*\n+ * Cache occupancy information.\n+ */\n+struct fscache_occupancy {\n+\tunsigned long long\tquery_from;\t/* Point to query from */\n+\tunsigned long long\tquery_to;\t/* Point to query to */\n+\tunsigned long long\tcached_from[2];\t/* Point at which cache extents start */\n+\tunsigned long long\tcached_to[2];\t/* Point at which cache extents end */\n+\tunsigned int\t\tgranularity;\t/* Granularity desired */\n+\tenum fscache_extent_type cached_type[2];\t/* Type of cache extent */\n+};\n+\n /*\n * slow-path functions for when there is actually caching available, and the\n * netfs does actually have a valid token\ndiff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h\nindex f9a17fbbd3980b..4e7d69b4ace239 100644\n--- a/include/linux/iov_iter.h\n+++ b/include/linux/iov_iter.h\n@@ -10,7 +10,7 @@\n \n #include \u003clinux/uio.h\u003e\n #include \u003clinux/bvec.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n+#include \u003clinux/bvecq.h\u003e\n \n typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len,\n \t\t\t void *priv, void *priv2);\n@@ -142,56 +142,65 @@ size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2,\n }\n \n /*\n- * Handle ITER_FOLIOQ.\n+ * Handle ITER_BVECQ.\n */\n static __always_inline\n-size_t iterate_folioq(struct iov_iter *iter, size_t len, void *priv, void *priv2,\n-\t\t iov_step_f step)\n+size_t iterate_bvecq(struct iov_iter *iter, size_t len, void *priv, void *priv2,\n+\t\t iov_step_f step)\n {\n-\tconst struct folio_queue *folioq = iter-\u003efolioq;\n-\tunsigned int slot = iter-\u003efolioq_slot;\n+\tconst struct bvecq *bq = iter-\u003ebvecq;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n \tsize_t progress = 0, skip = iter-\u003eiov_offset;\n \n-\tif (slot == folioq_nr_slots(folioq)) {\n-\t\t/* The iterator may have been extended. */\n-\t\tfolioq = folioq-\u003enext;\n-\t\tslot = 0;\n-\t}\n-\n \tdo {\n-\t\tstruct folio *folio = folioq_folio(folioq, slot);\n-\t\tsize_t part, remain = 0, consumed;\n-\t\tsize_t fsize;\n+\t\tconst struct bio_vec *bvec;\n+\t\tstruct page *page;\n+\t\tsize_t poff, plen;\n \t\tvoid *base;\n \n-\t\tif (!folio)\n-\t\t\tbreak;\n+\t\tif (slot \u003e= bq-\u003enr_slots) {\n+\t\t\tif (!bq-\u003enext)\n+\t\t\t\tbreak;\n+\t\t\tbq = bq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbvec = \u0026bq-\u003ebv[slot];\n+\t\t/*\n+\t\t * The caller must ensure that a slot with bv_len\u003e0 has a valid\n+\t\t * bv_page.\n+\t\t */\n+\t\tpage = bvec-\u003ebv_page + (bvec-\u003ebv_offset + skip) / PAGE_SIZE;\n+\t\tpoff = (bvec-\u003ebv_offset + skip) % PAGE_SIZE;\n+\t\tplen = umin(bvec-\u003ebv_len - skip, len);\n+\n+\t\twhile (plen \u003e 0) {\n+\t\t\tsize_t part, remain, consumed;\n \n-\t\tfsize = folioq_folio_size(folioq, slot);\n-\t\tif (skip \u003c fsize) {\n-\t\t\tbase = kmap_local_folio(folio, skip);\n-\t\t\tpart = umin(len, PAGE_SIZE - skip % PAGE_SIZE);\n+\t\t\tpart = umin(plen, PAGE_SIZE - poff);\n+\t\t\tbase = kmap_local_page(page) + poff;\n \t\t\tremain = step(base, progress, part, priv, priv2);\n \t\t\tkunmap_local(base);\n+\n \t\t\tconsumed = part - remain;\n-\t\t\tlen -= consumed;\n \t\t\tprogress += consumed;\n \t\t\tskip += consumed;\n+\t\t\tlen -= consumed;\n+\t\t\tif (!len || remain)\n+\t\t\t\tgoto stop;\n+\t\t\tpage++;\n+\t\t\tpoff = 0;\n+\t\t\tplen -= consumed;\n \t\t}\n-\t\tif (skip \u003e= fsize) {\n-\t\t\tskip = 0;\n-\t\t\tslot++;\n-\t\t\tif (slot == folioq_nr_slots(folioq) \u0026\u0026 folioq-\u003enext) {\n-\t\t\t\tfolioq = folioq-\u003enext;\n-\t\t\t\tslot = 0;\n-\t\t\t}\n-\t\t}\n-\t\tif (remain)\n-\t\t\tbreak;\n+\n+\t\tskip = 0;\n+\t\tslot++;\n \t} while (len);\n \n-\titer-\u003efolioq_slot = slot;\n-\titer-\u003efolioq = folioq;\n+stop:\n+\titer-\u003ebvecq_slot = slot;\n+\titer-\u003ebvecq = bq;\n \titer-\u003eiov_offset = skip;\n \titer-\u003ecount -= progress;\n \treturn progress;\n@@ -306,8 +315,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,\n \t\treturn iterate_bvec(iter, len, priv, priv2, step);\n \tif (iov_iter_is_kvec(iter))\n \t\treturn iterate_kvec(iter, len, priv, priv2, step);\n-\tif (iov_iter_is_folioq(iter))\n-\t\treturn iterate_folioq(iter, len, priv, priv2, step);\n+\tif (iov_iter_is_bvecq(iter))\n+\t\treturn iterate_bvecq(iter, len, priv, priv2, step);\n \tif (iov_iter_is_xarray(iter))\n \t\treturn iterate_xarray(iter, len, priv, priv2, step);\n \treturn iterate_discard(iter, len, priv, priv2, step);\n@@ -342,7 +351,7 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv,\n * buffer is presented in segments, which for kernel iteration are broken up by\n * physical pages and mapped, with the mapped address being presented.\n *\n- * [!] Note This will only handle BVEC, KVEC, FOLIOQ, XARRAY and DISCARD-type\n+ * [!] Note This will only handle BVEC, KVEC, BVECQ, XARRAY and DISCARD-type\n * iterators; it will not handle UBUF or IOVEC-type iterators.\n *\n * A step functions, @step, must be provided, one for handling mapped kernel\n@@ -370,8 +379,8 @@ size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv,\n \t\treturn iterate_bvec(iter, len, priv, priv2, step);\n \tif (iov_iter_is_kvec(iter))\n \t\treturn iterate_kvec(iter, len, priv, priv2, step);\n-\tif (iov_iter_is_folioq(iter))\n-\t\treturn iterate_folioq(iter, len, priv, priv2, step);\n+\tif (iov_iter_is_bvecq(iter))\n+\t\treturn iterate_bvecq(iter, len, priv, priv2, step);\n \tif (iov_iter_is_xarray(iter))\n \t\treturn iterate_xarray(iter, len, priv, priv2, step);\n \treturn iterate_discard(iter, len, priv, priv2, step);\ndiff --git a/include/linux/netfs.h b/include/linux/netfs.h\nindex 1bc120d61c5beb..2ed79836ecd6e6 100644\n--- a/include/linux/netfs.h\n+++ b/include/linux/netfs.h\n@@ -17,12 +17,15 @@\n #include \u003clinux/workqueue.h\u003e\n #include \u003clinux/fs.h\u003e\n #include \u003clinux/pagemap.h\u003e\n+#include \u003clinux/bvecq.h\u003e\n #include \u003clinux/uio.h\u003e\n-#include \u003clinux/rolling_buffer.h\u003e\n \n enum netfs_sreq_ref_trace;\n typedef struct mempool mempool_t;\n-struct folio_queue;\n+struct readahead_control;\n+struct netfs_io_request;\n+struct netfs_io_subrequest;\n+struct fscache_occupancy;\n \n /**\n * folio_start_private_2 - Start an fscache write on a folio. [DEPRECATED]\n@@ -63,7 +66,7 @@ struct netfs_inode {\n #endif\n \tstruct list_head\twb_queue;\t/* Queue of processes wanting to do writeback */\n \tloff_t\t\t\t_remote_i_size;\t/* Size of the remote file */\n-\tloff_t\t\t\t_zero_point;\t/* Size after which we assume there's no data\n+\tunsigned long long\t_zero_point;\t/* Size after which we assume there's no data\n \t\t\t\t\t\t * on the server */\n \tspinlock_t\t\tlock;\t\t/* Lock covering wb_queue */\n \tatomic_t\t\tio_count;\t/* Number of outstanding reqs */\n@@ -125,26 +128,39 @@ static inline struct netfs_group *netfs_folio_group(struct folio *folio)\n \treturn priv;\n }\n \n+/*\n+ * Estimate of maximum write subrequest for writeback. The filesystem is\n+ * responsible for filling this in when called from -\u003eestimate_write(), though\n+ * netfslib will preset infinite defaults.\n+ */\n+struct netfs_write_estimate {\n+\tunsigned long long\tissue_at;\t/* Point at which we must submit */\n+\tint\t\t\tmax_segs;\t/* Max number of segments in a single RPC */\n+};\n+\n /*\n * Stream of I/O subrequests going to a particular destination, such as the\n * server or the local cache. This is mainly intended for writing where we may\n * have to write to multiple destinations concurrently.\n */\n struct netfs_io_stream {\n-\t/* Submission tracking */\n-\tstruct netfs_io_subrequest *construct;\t/* Op being constructed */\n-\tsize_t\t\t\tsreq_max_len;\t/* Maximum size of a subrequest */\n-\tunsigned int\t\tsreq_max_segs;\t/* 0 or max number of segments in an iterator */\n-\tunsigned int\t\tsubmit_off;\t/* Folio offset we're submitting from */\n-\tunsigned int\t\tsubmit_len;\t/* Amount of data left to submit */\n-\tunsigned int\t\tsubmit_extendable_to; /* Amount I/O can be rounded up to */\n-\tvoid (*prepare_write)(struct netfs_io_subrequest *subreq);\n-\tvoid (*issue_write)(struct netfs_io_subrequest *subreq);\n+\t/* Submission tracking (main dispatch only; not retry) */\n+\tstruct bvecq_pos\tdispatch_cursor; /* Point from which buffers are dispatched */\n+\tunsigned long long\tissue_from;\t/* Current issue point */\n+\tsize_t\t\t\tbuffered;\t/* Amount in buffer */\n+\tu8\t\t\tapplicable;\t/* What sources are applicable (NOTE_* mask) */\n+\tbool\t\t\tbuffering;\t/* T if buffering on this stream */\n+\tint (*estimate_write)(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_write_estimate *estimate);\n+\tint (*issue_write)(struct netfs_io_subrequest *subreq);\n+\tatomic64_t\t\tissued_to;\t/* Point to which can be considered issued */\n+\n \t/* Collection tracking */\n \tstruct list_head\tsubrequests;\t/* Contributory I/O operations */\n \tunsigned long long\tcollected_to;\t/* Position we've collected results to */\n \tsize_t\t\t\ttransferred;\t/* The amount transferred from this stream */\n-\tunsigned short\t\terror;\t\t/* Aggregate error for the stream */\n+\tshort\t\t\terror;\t\t/* Aggregate error for the stream */\n \tenum netfs_io_source\tsource;\t\t/* Where to read from/write to */\n \tunsigned char\t\tstream_nr;\t/* Index of stream in parent table */\n \tbool\t\t\tavail;\t\t/* T if stream is available */\n@@ -152,6 +168,7 @@ struct netfs_io_stream {\n \tbool\t\t\tneed_retry;\t/* T if this stream needs retrying */\n \tbool\t\t\tfailed;\t\t/* T if this stream failed */\n \tbool\t\t\ttransferred_valid; /* T is -\u003etransferred is valid */\n+\tbool\t\t\tcancelled;\t/* T if stream is cancelled */\n };\n \n /*\n@@ -161,8 +178,11 @@ struct netfs_cache_resources {\n \tconst struct netfs_cache_ops\t*ops;\n \tvoid\t\t\t\t*cache_priv;\n \tvoid\t\t\t\t*cache_priv2;\n-\tunsigned int\t\t\tdebug_id;\t/* Cookie debug ID */\n+\tunsigned long long\t\tcache_i_size;\t/* Initial size of cache file */\n+\tunsigned int\t\t\tcookie_id;\t/* Cache cookie debug ID */\n+\tunsigned int\t\t\tobject_id;\t/* Cache object debug ID */\n \tunsigned int\t\t\tinval_counter;\t/* object-\u003einval_counter at begin_op */\n+\tunsigned int\t\t\tdio_size;\t/* DIO block size */\n };\n \n /*\n@@ -176,14 +196,15 @@ struct netfs_io_subrequest {\n \tstruct netfs_io_request *rreq;\t\t/* Supervising I/O request */\n \tstruct work_struct\twork;\n \tstruct list_head\trreq_link;\t/* Link in rreq-\u003esubrequests */\n-\tstruct iov_iter\t\tio_iter;\t/* Iterator for this subrequest */\n+\tstruct bvecq_pos\tdispatch_pos;\t/* Bookmark in the combined queue of the start */\n+\tstruct bvecq_pos\tcontent;\t/* The (copied) content of the subrequest */\n \tunsigned long long\tstart;\t\t/* Where to start the I/O */\n \tsize_t\t\t\tlen;\t\t/* Size of the I/O */\n \tsize_t\t\t\ttransferred;\t/* Amount of data transferred */\n+\tunsigned int\t\tnr_segs;\t/* Number of segments in content */\n \trefcount_t\t\tref;\n \tshort\t\t\terror;\t\t/* 0 or error that occurred */\n \tunsigned short\t\tdebug_index;\t/* Index in list (for debugging output) */\n-\tunsigned int\t\tnr_segs;\t/* Number of segs in io_iter */\n \tu8\t\t\tretry_count;\t/* The number of retries (0 on initial pass) */\n \tenum netfs_io_source\tsource;\t\t/* Where to read from/write to */\n \tunsigned char\t\tstream_nr;\t/* I/O stream this belongs to */\n@@ -191,12 +212,11 @@ struct netfs_io_subrequest {\n #define NETFS_SREQ_COPY_TO_CACHE\t0\t/* Set if should copy the data to the cache */\n #define NETFS_SREQ_CLEAR_TAIL\t\t1\t/* Set if the rest of the read should be cleared */\n #define NETFS_SREQ_MADE_PROGRESS\t4\t/* Set if we transferred at least some data */\n-#define NETFS_SREQ_ONDEMAND\t\t5\t/* Set if it's from on-demand read mode */\n-#define NETFS_SREQ_BOUNDARY\t\t6\t/* Set if ends on hard boundary (eg. ceph object) */\n #define NETFS_SREQ_HIT_EOF\t\t7\t/* Set if short due to EOF */\n #define NETFS_SREQ_IN_PROGRESS\t\t8\t/* Unlocked when the subrequest completes */\n #define NETFS_SREQ_NEED_RETRY\t\t9\t/* Set if the filesystem requests a retry */\n #define NETFS_SREQ_FAILED\t\t10\t/* Set if the subreq failed unretryably */\n+#define NETFS_SREQ_CANCELLED\t\t11\t/* Set if the subreq was cancelled by netfslib */\n };\n \n enum netfs_io_origin {\n@@ -209,7 +229,6 @@ enum netfs_io_origin {\n \tNETFS_DIO_READ,\t\t\t/* This is a direct I/O read */\n \tNETFS_WRITEBACK,\t\t/* This write was triggered by writepages */\n \tNETFS_WRITEBACK_SINGLE,\t\t/* This monolithic write was triggered by writepages */\n-\tNETFS_WRITETHROUGH,\t\t/* This write was made by netfs_perform_write() */\n \tNETFS_UNBUFFERED_WRITE,\t\t/* This is an unbuffered write */\n \tNETFS_DIO_WRITE,\t\t/* This is a direct I/O write */\n \tNETFS_PGPRIV2_COPY_TO_CACHE,\t/* [DEPRECATED] This is writing read data to the cache */\n@@ -237,25 +256,29 @@ struct netfs_io_request {\n \tstruct netfs_io_stream\tio_streams[2];\t/* Streams of parallel I/O operations */\n #define NR_IO_STREAMS 2 //wreq-\u003enr_io_streams\n \tstruct netfs_group\t*group;\t\t/* Writeback group being written back */\n-\tstruct rolling_buffer\tbuffer;\t\t/* Unencrypted buffer */\n-#define NETFS_ROLLBUF_PUT_MARK\t\tROLLBUF_MARK_1\n-#define NETFS_ROLLBUF_PAGECACHE_MARK\tROLLBUF_MARK_2\n+\tstruct bvecq\t\t*spare;\t\t/* Advance allocation of bvecq */\n+\tstruct bvecq_pos\tload_cursor;\t/* Point at which new folios are loaded in */\n+\tstruct bvecq_pos\tcollect_cursor;\t/* Clear-up point of I/O buffer */\n+\tstruct bvecq_pos\tretry_cursor;\t/* Point from which retries are dispatched */\n \twait_queue_head_t\twaitq;\t\t/* Processor waiter */\n \tvoid\t\t\t*netfs_priv;\t/* Private data for the netfs */\n \tvoid\t\t\t*netfs_priv2;\t/* Private data for the netfs */\n-\tstruct bio_vec\t\t*direct_bv;\t/* DIO buffer list (when handling iovec-iter) */\n \tunsigned long long\tsubmitted;\t/* Amount submitted for I/O so far */\n \tunsigned long long\tlen;\t\t/* Length of the request */\n \tsize_t\t\t\ttransferred;\t/* Amount to be indicated as transferred */\n \tlong\t\t\terror;\t\t/* 0 or error that occurred */\n \tunsigned long long\ti_size;\t\t/* Size of the file */\n \tunsigned long long\tstart;\t\t/* Start position */\n-\tatomic64_t\t\tissued_to;\t/* Write issuer folio cursor */\n \tunsigned long long\tcollected_to;\t/* Point we've collected to */\n+\tunsigned long long\tcache_coll_to;\t/* Point the cache has collected to */\n \tunsigned long long\tcleaned_to;\t/* Position we've cleaned folios to */\n \tunsigned long long\tabandon_to;\t/* Position to abandon folios to */\n+#ifdef CONFIG_NETFS_PGPRIV2\n+\tunsigned long long\tlast_end;\t/* End of last folio added */\n+#endif\n+\tunsigned long long\tretry_start;\t/* Position to retry from */\n+\tsize_t\t\t\tretry_buffered;\t/* Amount of data to retry */\n \tconst struct folio\t*no_unlock_folio; /* Don't unlock this folio after read */\n-\tunsigned int\t\tdirect_bv_count; /* Number of elements in direct_bv[] */\n \tunsigned int\t\tdebug_id;\n \tunsigned int\t\trsize;\t\t/* Maximum read size (0 for none) */\n \tunsigned int\t\twsize;\t\t/* Maximum write size (0 for none) */\n@@ -264,7 +287,6 @@ struct netfs_io_request {\n \tspinlock_t\t\tlock;\t\t/* Lock for queuing subreqs */\n \tunsigned char\t\tfront_folio_order; /* Order (size) of front folio */\n \tenum netfs_io_origin\torigin;\t\t/* Origin of the request */\n-\tbool\t\t\tdirect_bv_unpin; /* T if direct_bv[] must be unpinned */\n \trefcount_t\t\tref;\n \tunsigned long\t\tflags;\n #define NETFS_RREQ_IN_PROGRESS\t\t0\t/* Unlocked when the request completes (has ref) */\n@@ -273,13 +295,20 @@ struct netfs_io_request {\n #define NETFS_RREQ_FAILED\t\t3\t/* The request failed */\n #define NETFS_RREQ_RETRYING\t\t4\t/* Set if we're in the retry path */\n #define NETFS_RREQ_SHORT_TRANSFER\t5\t/* Set if we have a short transfer */\n-#define NETFS_RREQ_OFFLOAD_COLLECTION\t8\t/* Offload collection to workqueue */\n-#define NETFS_RREQ_NO_UNLOCK_FOLIO\t9\t/* Don't unlock no_unlock_folio on completion */\n-#define NETFS_RREQ_FOLIO_COPY_TO_CACHE\t10\t/* Copy current folio to cache from read */\n-#define NETFS_RREQ_UPLOAD_TO_SERVER\t11\t/* Need to write to the server */\n-#define NETFS_RREQ_USE_IO_ITER\t\t12\t/* Use -\u003eio_iter rather than -\u003ei_pages */\n+#define NETFS_RREQ_ABANDON_REQ\t\t6\t/* Set if the request is to be abandoned */\n+#define NETFS_RREQ_SAW_ENOMEM\t\t7\t/* Set if we encounted ENOMEM */\n+#define NETFS_RREQ_CACHE_STOP\t\t8\t/* Set to stop caching (ENOBUFS or error) */\n+#define NETFS_RREQ_CACHE_ERROR\t\t9\t/* Set if we got an error from the cache */\n+#define NETFS_RREQ_OFFLOAD_COLLECTION\t12\t/* Offload collection to workqueue */\n+#define NETFS_RREQ_NO_UNLOCK_FOLIO\t13\t/* Don't unlock no_unlock_folio on completion */\n+#define NETFS_RREQ_FOLIO_COPY_TO_CACHE\t14\t/* Copy current folio to cache from read */\n+#define NETFS_RREQ_UPLOAD_TO_SERVER\t15\t/* Need to write to the server */\n+#define NETFS_RREQ_USE_IO_ITER\t\t16\t/* Use -\u003eio_iter rather than -\u003ei_pages */\n+#define NETFS_RREQ_NEED_PUT_RA_REFS\t17\t/* Need to put the folio refs RA gave us */\n+#ifdef CONFIG_NETFS_PGPRIV2\n #define NETFS_RREQ_USE_PGPRIV2\t\t31\t/* [DEPRECATED] Use PG_private_2 to mark\n \t\t\t\t\t\t * write to cache on read */\n+#endif\n \tconst struct netfs_request_ops *netfs_ops;\n };\n \n@@ -295,8 +324,7 @@ struct netfs_request_ops {\n \n \t/* Read request handling */\n \tvoid (*expand_readahead)(struct netfs_io_request *rreq);\n-\tint (*prepare_read)(struct netfs_io_subrequest *subreq);\n-\tvoid (*issue_read)(struct netfs_io_subrequest *subreq);\n+\tint (*issue_read)(struct netfs_io_subrequest *subreq);\n \tbool (*is_still_valid)(struct netfs_io_request *rreq);\n \tint (*check_write_begin)(struct file *file, loff_t pos, unsigned len,\n \t\t\t\t struct folio **foliop, void **_fsdata);\n@@ -308,8 +336,10 @@ struct netfs_request_ops {\n \n \t/* Write request handling */\n \tvoid (*begin_writeback)(struct netfs_io_request *wreq);\n-\tvoid (*prepare_write)(struct netfs_io_subrequest *subreq);\n-\tvoid (*issue_write)(struct netfs_io_subrequest *subreq);\n+\tint (*estimate_write)(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_write_estimate *estimate);\n+\tint (*issue_write)(struct netfs_io_subrequest *subreq);\n \tvoid (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream);\n \tvoid (*invalidate_cache)(struct netfs_io_request *wreq);\n };\n@@ -322,6 +352,12 @@ enum netfs_read_from_hole {\n \tNETFS_READ_HOLE_FAIL,\n };\n \n+enum netfs_cache_collect {\n+\tNETFS_CACHE_COLLECT_WRITE_DATA,\n+\tNETFS_CACHE_COLLECT_WRITE_GAP,\n+\tNETFS_CACHE_COLLECT_WRITE_CANCEL,\n+};\n+\n /*\n * Table of operations for access to a cache.\n */\n@@ -344,8 +380,16 @@ struct netfs_cache_ops {\n \t\t netfs_io_terminated_t term_func,\n \t\t void *term_func_priv);\n \n+\t/* Estimate the amount of data that can be written in an op. */\n+\tint (*estimate_write)(struct netfs_io_request *wreq,\n+\t\t\t struct netfs_io_stream *stream,\n+\t\t\t struct netfs_write_estimate *estimate);\n+\n+\t/* Read data from the cache for a netfs subrequest. */\n+\tint (*issue_read)(struct netfs_io_subrequest *subreq);\n+\n \t/* Write data to the cache from a netfs subrequest. */\n-\tvoid (*issue_write)(struct netfs_io_subrequest *subreq);\n+\tint (*issue_write)(struct netfs_io_subrequest *subreq);\n \n \t/* Expand readahead request */\n \tvoid (*expand_readahead)(struct netfs_cache_resources *cres,\n@@ -353,40 +397,28 @@ struct netfs_cache_ops {\n \t\t\t\t unsigned long long *_len,\n \t\t\t\t unsigned long long i_size);\n \n-\t/* Prepare a read operation, shortening it to a cached/uncached\n-\t * boundary as appropriate.\n-\t */\n-\tenum netfs_io_source (*prepare_read)(struct netfs_io_subrequest *subreq,\n-\t\t\t\t\t unsigned long long i_size);\n-\n-\t/* Prepare a write subrequest, working out if we're allowed to do it\n-\t * and finding out the maximum amount of data to gather before\n-\t * attempting to submit. If we're not permitted to do it, the\n-\t * subrequest should be marked failed.\n-\t */\n-\tvoid (*prepare_write_subreq)(struct netfs_io_subrequest *subreq);\n-\n \t/* Prepare a write operation, working out what part of the write we can\n \t * actually do.\n \t */\n-\tint (*prepare_write)(struct netfs_cache_resources *cres,\n-\t\t\t loff_t *_start, size_t *_len, size_t upper_len,\n-\t\t\t loff_t i_size, bool no_space_allocated_yet);\n-\n-\t/* Prepare an on-demand read operation, shortening it to a cached/uncached\n-\t * boundary as appropriate.\n-\t */\n-\tenum netfs_io_source (*prepare_ondemand_read)(struct netfs_cache_resources *cres,\n-\t\t\t\t\t\t loff_t start, size_t *_len,\n-\t\t\t\t\t\t loff_t i_size,\n-\t\t\t\t\t\t unsigned long *_flags, ino_t ino);\n+\tint (*prepare_write_old)(struct netfs_cache_resources *cres,\n+\t\t\t\t loff_t *_start, size_t *_len, size_t upper_len,\n+\t\t\t\t loff_t i_size, bool no_space_allocated_yet);\n \n \t/* Query the occupancy of the cache in a region, returning where the\n \t * next chunk of data starts and how long it is.\n \t */\n-\tint (*query_occupancy)(struct netfs_cache_resources *cres,\n-\t\t\t loff_t start, size_t len, size_t granularity,\n-\t\t\t loff_t *_data_start, size_t *_data_len);\n+\tvoid (*query_occupancy)(struct netfs_cache_resources *cres,\n+\t\t\t\tstruct fscache_occupancy *occ);\n+\n+\t/* Collect the result of buffered writeback to the cache. This\n+\t * includes copying a read to the cache. block_type is one of:\n+\t * - NETFS_CACHE_COLLECT_WRITE_DATA for a block of data\n+\t * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped\n+\t * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a cancellation gap\n+\t */\n+\tvoid (*collect_write)(struct netfs_io_request *wreq,\n+\t\t\t unsigned long long start, size_t len,\n+\t\t\t enum netfs_cache_collect block_type);\n };\n \n /* High-level read API. */\n@@ -410,10 +442,9 @@ void netfs_single_mark_inode_dirty(struct inode *inode);\n ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_iter *iter);\n int netfs_writeback_single(struct address_space *mapping,\n \t\t\t struct writeback_control *wbc,\n-\t\t\t struct iov_iter *iter);\n+\t\t\t struct iov_iter *iter, size_t len);\n \n /* Address operations API */\n-struct readahead_control;\n void netfs_readahead(struct readahead_control *);\n int netfs_read_folio(struct file *, struct folio *);\n int netfs_write_begin(struct netfs_inode *, struct file *,\n@@ -437,12 +468,11 @@ void netfs_get_subrequest(struct netfs_io_subrequest *subreq,\n \t\t\t enum netfs_sreq_ref_trace what);\n void netfs_put_subrequest(struct netfs_io_subrequest *subreq,\n \t\t\t enum netfs_sreq_ref_trace what);\n-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,\n-\t\t\t\tstruct iov_iter *new,\n-\t\t\t\tiov_iter_extraction_t extraction_flags);\n-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,\n-\t\t\tsize_t max_size, size_t max_segs);\n-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);\n+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,\n+\t\t\t unsigned long long fpos, struct bvecq **_bvecq_head,\n+\t\t\t iov_iter_extraction_t extraction_flags);\n+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);\n+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);\n void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);\n \n int netfs_start_io_read(struct inode *inode);\n@@ -452,18 +482,6 @@ void netfs_end_io_write(struct inode *inode);\n int netfs_start_io_direct(struct inode *inode);\n void netfs_end_io_direct(struct inode *inode);\n \n-/* Miscellaneous APIs. */\n-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,\n-\t\t\t\t unsigned int trace /*enum netfs_folioq_trace*/);\n-void netfs_folioq_free(struct folio_queue *folioq,\n-\t\t unsigned int trace /*enum netfs_trace_folioq*/);\n-\n-/* Buffer wrangling helpers API. */\n-int netfs_alloc_folioq_buffer(struct address_space *mapping,\n-\t\t\t struct folio_queue **_buffer,\n-\t\t\t size_t *_cur_size, ssize_t size, gfp_t gfp);\n-void netfs_free_folioq_buffer(struct folio_queue *fq);\n-\n /* Writeback exclusion API. */\n bool netfs_wb_begin(struct netfs_inode *ictx, bool nowait);\n void netfs_wb_end(struct netfs_inode *ictx);\ndiff --git a/include/linux/pagemap.h b/include/linux/pagemap.h\nindex 2c3718d592d6dd..e1e51bace388c9 100644\n--- a/include/linux/pagemap.h\n+++ b/include/linux/pagemap.h\n@@ -1348,6 +1348,7 @@ struct readahead_control {\n \tstruct file_ra_state *ra;\n /* private: use the readahead_* accessors instead */\n \tpgoff_t _index;\n+\tunsigned int _nr_folios;\n \tunsigned int _nr_pages;\n \tunsigned int _batch_count;\n \tbool dropbehind;\n@@ -1527,6 +1528,15 @@ static inline size_t readahead_batch_length(const struct readahead_control *rac)\n \treturn rac-\u003e_batch_count * PAGE_SIZE;\n }\n \n+/**\n+ * readahead_folio_count - Get the number of folios in this readahead request.\n+ * @rac: The readahead request.\n+ */\n+static inline unsigned int readahead_folio_count(const struct readahead_control *rac)\n+{\n+\treturn rac-\u003e_nr_folios;\n+}\n+\n static inline unsigned long dir_pages(const struct inode *inode)\n {\n \treturn (unsigned long)(inode-\u003ei_size + PAGE_SIZE - 1) \u003e\u003e\ndiff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h\ndeleted file mode 100644\nindex ac15b1ffdd8315..00000000000000\n--- a/include/linux/rolling_buffer.h\n+++ /dev/null\n@@ -1,61 +0,0 @@\n-/* SPDX-License-Identifier: GPL-2.0-or-later */\n-/* Rolling buffer of folios\n- *\n- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.\n- * Written by David Howells (dhowells@redhat.com)\n- */\n-\n-#ifndef _ROLLING_BUFFER_H\n-#define _ROLLING_BUFFER_H\n-\n-#include \u003clinux/folio_queue.h\u003e\n-#include \u003clinux/uio.h\u003e\n-\n-/*\n- * Rolling buffer. Whilst the buffer is live and in use, folios and folio\n- * queue segments can be added to one end by one thread and removed from the\n- * other end by another thread. The buffer isn't allowed to be empty; it must\n- * always have at least one folio_queue in it so that neither side has to\n- * modify both queue pointers.\n- *\n- * The iterator in the buffer is extended as buffers are inserted. It can be\n- * snapshotted to use a segment of the buffer.\n- */\n-struct rolling_buffer {\n-\tstruct folio_queue\t*head;\t\t/* Producer's insertion point */\n-\tstruct folio_queue\t*tail;\t\t/* Consumer's removal point */\n-\tstruct iov_iter\t\titer;\t\t/* Iterator tracking what's left in the buffer */\n-\tu8\t\t\tnext_head_slot;\t/* Next slot in -\u003ehead */\n-\tu8\t\t\tfirst_tail_slot; /* First slot in -\u003etail */\n-};\n-\n-/*\n- * Snapshot of a rolling buffer.\n- */\n-struct rolling_buffer_snapshot {\n-\tstruct folio_queue\t*curr_folioq;\t/* Queue segment in which current folio resides */\n-\tunsigned char\t\tcurr_slot;\t/* Folio currently being read */\n-\tunsigned char\t\tcurr_order;\t/* Order of folio */\n-};\n-\n-/* Marks to store per-folio in the internal folio_queue structs. */\n-#define ROLLBUF_MARK_1\tBIT(0)\n-#define ROLLBUF_MARK_2\tBIT(1)\n-\n-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,\n-\t\t\tunsigned int direction);\n-int rolling_buffer_make_space(struct rolling_buffer *roll);\n-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,\n-\t\t\t\t struct readahead_control *ractl,\n-\t\t\t\t struct folio_batch *put_batch);\n-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,\n-\t\t\t unsigned int flags);\n-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);\n-void rolling_buffer_clear(struct rolling_buffer *roll);\n-\n-static inline void rolling_buffer_advance(struct rolling_buffer *roll, size_t amount)\n-{\n-\tiov_iter_advance(\u0026roll-\u003eiter, amount);\n-}\n-\n-#endif /* _ROLLING_BUFFER_H */\ndiff --git a/include/linux/uio.h b/include/linux/uio.h\nindex a9bc5b3067e320..e84a0c4f28c6c8 100644\n--- a/include/linux/uio.h\n+++ b/include/linux/uio.h\n@@ -11,7 +11,6 @@\n #include \u003cuapi/linux/uio.h\u003e\n \n struct page;\n-struct folio_queue;\n \n typedef unsigned int __bitwise iov_iter_extraction_t;\n \n@@ -26,7 +25,7 @@ enum iter_type {\n \tITER_IOVEC,\n \tITER_BVEC,\n \tITER_KVEC,\n-\tITER_FOLIOQ,\n+\tITER_BVECQ,\n \tITER_XARRAY,\n \tITER_DISCARD,\n };\n@@ -68,7 +67,7 @@ struct iov_iter {\n \t\t\t\tconst struct iovec *__iov;\n \t\t\t\tconst struct kvec *kvec;\n \t\t\t\tconst struct bio_vec *bvec;\n-\t\t\t\tconst struct folio_queue *folioq;\n+\t\t\t\tconst struct bvecq *bvecq;\n \t\t\t\tstruct xarray *xarray;\n \t\t\t\tvoid __user *ubuf;\n \t\t\t};\n@@ -77,7 +76,7 @@ struct iov_iter {\n \t};\n \tunion {\n \t\tunsigned long nr_segs;\n-\t\tu8 folioq_slot;\n+\t\tu16 bvecq_slot;\n \t\tloff_t xarray_start;\n \t};\n };\n@@ -145,9 +144,9 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i)\n \treturn iov_iter_type(i) == ITER_DISCARD;\n }\n \n-static inline bool iov_iter_is_folioq(const struct iov_iter *i)\n+static inline bool iov_iter_is_bvecq(const struct iov_iter *i)\n {\n-\treturn iov_iter_type(i) == ITER_FOLIOQ;\n+\treturn iov_iter_type(i) == ITER_BVECQ;\n }\n \n static inline bool iov_iter_is_xarray(const struct iov_iter *i)\n@@ -295,9 +294,9 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec\n void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec,\n \t\t\tunsigned long nr_segs, size_t count);\n void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count);\n-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,\n-\t\t\t const struct folio_queue *folioq,\n-\t\t\t unsigned int first_slot, unsigned int offset, size_t count);\n+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,\n+\t\t\t const struct bvecq *bvecq,\n+\t\t\t unsigned int first_slot, unsigned int offset, size_t count);\n void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray,\n \t\t loff_t start, size_t count);\n ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,\ndiff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h\nindex 6e3b1424eea4dc..b13322b376e923 100644\n--- a/include/trace/events/cachefiles.h\n+++ b/include/trace/events/cachefiles.h\n@@ -31,8 +31,6 @@ enum cachefiles_obj_ref_trace {\n \tcachefiles_obj_see_lookup_failed,\n \tcachefiles_obj_see_withdraw_cookie,\n \tcachefiles_obj_see_withdrawal,\n-\tcachefiles_obj_get_ondemand_fd,\n-\tcachefiles_obj_put_ondemand_fd,\n \tcachefiles_obj_get_read_req,\n \tcachefiles_obj_put_read_req,\n };\n@@ -56,6 +54,8 @@ enum cachefiles_coherency_trace {\n \tcachefiles_coherency_check_ok,\n \tcachefiles_coherency_check_type,\n \tcachefiles_coherency_check_xattr,\n+\tcachefiles_coherency_discontiguous,\n+\tcachefiles_coherency_remove,\n \tcachefiles_coherency_set_fail,\n \tcachefiles_coherency_set_ok,\n \tcachefiles_coherency_vol_check_cmp,\n@@ -67,6 +67,7 @@ enum cachefiles_coherency_trace {\n };\n \n enum cachefiles_trunc_trace {\n+\tcachefiles_trunc_clear_padding,\n \tcachefiles_trunc_dio_adjust,\n \tcachefiles_trunc_expand_tmpfile,\n \tcachefiles_trunc_shrink,\n@@ -84,6 +85,7 @@ enum cachefiles_prepare_read_trace {\n };\n \n enum cachefiles_error_trace {\n+\tcachefiles_trace_alignment_error,\n \tcachefiles_trace_fallocate_error,\n \tcachefiles_trace_getxattr_error,\n \tcachefiles_trace_link_error,\n@@ -130,8 +132,6 @@ enum cachefiles_error_trace {\n \tEM(cachefiles_obj_see_lookup_failed,\t\"SEE lookup_failed\")\t\\\n \tEM(cachefiles_obj_see_withdraw_cookie,\t\"SEE withdraw_cookie\")\t\\\n \tEM(cachefiles_obj_see_withdrawal,\t\"SEE withdrawal\")\t\\\n-\tEM(cachefiles_obj_get_ondemand_fd, \"GET ondemand_fd\")\t\\\n-\tEM(cachefiles_obj_put_ondemand_fd, \"PUT ondemand_fd\")\t\\\n \tEM(cachefiles_obj_get_read_req,\t\t\"GET read_req\")\t\t\\\n \tE_(cachefiles_obj_put_read_req,\t\t\"PUT read_req\")\n \n@@ -144,6 +144,8 @@ enum cachefiles_error_trace {\n \tEM(cachefiles_coherency_check_ok,\t\"OK \")\t\t\\\n \tEM(cachefiles_coherency_check_type,\t\"BAD type\")\t\t\\\n \tEM(cachefiles_coherency_check_xattr,\t\"BAD xatt\")\t\t\\\n+\tEM(cachefiles_coherency_discontiguous,\t\"--- gap \")\t\t\\\n+\tEM(cachefiles_coherency_remove,\t\t\"REMOVE \")\t\t\\\n \tEM(cachefiles_coherency_set_fail,\t\"SET fail\")\t\t\\\n \tEM(cachefiles_coherency_set_ok,\t\t\"SET ok \")\t\t\\\n \tEM(cachefiles_coherency_vol_check_cmp,\t\"VOL BAD cmp \")\t\t\\\n@@ -154,6 +156,7 @@ enum cachefiles_error_trace {\n \tE_(cachefiles_coherency_vol_set_ok,\t\"VOL SET ok \")\n \n #define cachefiles_trunc_traces\t\t\t\t\t\t\\\n+\tEM(cachefiles_trunc_clear_padding,\t\"CLRPAD\")\t\t\\\n \tEM(cachefiles_trunc_dio_adjust,\t\t\"DIOADJ\")\t\t\\\n \tEM(cachefiles_trunc_expand_tmpfile,\t\"EXPTMP\")\t\t\\\n \tE_(cachefiles_trunc_shrink,\t\t\"SHRINK\")\n@@ -169,6 +172,7 @@ enum cachefiles_error_trace {\n \tE_(cachefiles_trace_read_seek_nxio,\t\"seek-enxio\")\n \n #define cachefiles_error_traces\t\t\t\t\t\t\\\n+\tEM(cachefiles_trace_alignment_error,\t\"align\")\t\t\\\n \tEM(cachefiles_trace_fallocate_error,\t\"fallocate\")\t\t\\\n \tEM(cachefiles_trace_getxattr_error,\t\"getxattr\")\t\t\\\n \tEM(cachefiles_trace_link_error,\t\t\"link\")\t\t\t\\\n@@ -379,12 +383,12 @@ TRACE_EVENT(cachefiles_rename,\n \n TRACE_EVENT(cachefiles_coherency,\n \t TP_PROTO(struct cachefiles_object *obj,\n-\t\t ino_t ino,\n+\t\t ino_t ino, unsigned long long obj_size,\n \t\t u64 disk_aux,\n \t\t enum cachefiles_content content,\n \t\t enum cachefiles_coherency_trace why),\n \n-\t TP_ARGS(obj, ino, disk_aux, content, why),\n+\t TP_ARGS(obj, ino, obj_size, disk_aux, content, why),\n \n \t /* Note that obj may be NULL */\n \t TP_STRUCT__entry(\n@@ -392,6 +396,7 @@ TRACE_EVENT(cachefiles_coherency,\n \t\t __field(enum cachefiles_coherency_trace,\twhy)\n \t\t __field(enum cachefiles_content,\t\tcontent)\n \t\t __field(u64,\t\t\t\tino)\n+\t\t __field(u64,\t\t\t\tobj_size)\n \t\t __field(u64,\t\t\t\taux)\n \t\t __field(u64,\t\t\t\tdisk_aux)\n \t\t\t ),\n@@ -401,14 +406,16 @@ TRACE_EVENT(cachefiles_coherency,\n \t\t __entry-\u003ewhy\t= why;\n \t\t __entry-\u003econtent\t= content;\n \t\t __entry-\u003eino\t= ino;\n+\t\t __entry-\u003eobj_size\t= obj_size;\n \t\t __entry-\u003eaux\t= be64_to_cpup((__be64 *)obj-\u003ecookie-\u003einline_aux);\n \t\t __entry-\u003edisk_aux\t= disk_aux;\n \t\t\t ),\n \n-\t TP_printk(\"o=%08x %s B=%llx c=%u aux=%llx dsk=%llx\",\n+\t TP_printk(\"o=%08x %s B=%llx oz=%llx c=%u aux=%llx dsk=%llx\",\n \t\t __entry-\u003eobj,\n \t\t __print_symbolic(__entry-\u003ewhy, cachefiles_coherency_traces),\n \t\t __entry-\u003eino,\n+\t\t __entry-\u003eobj_size,\n \t\t __entry-\u003econtent,\n \t\t __entry-\u003eaux,\n \t\t __entry-\u003edisk_aux)\n@@ -687,180 +694,6 @@ TRACE_EVENT(cachefiles_io_error,\n \t\t __entry-\u003eerror)\n \t );\n \n-TRACE_EVENT(cachefiles_ondemand_open,\n-\t TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,\n-\t\t struct cachefiles_open *load),\n-\n-\t TP_ARGS(obj, msg, load),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tmsg_id)\n-\t\t __field(unsigned int,\tobject_id)\n-\t\t __field(unsigned int,\tfd)\n-\t\t __field(unsigned int,\tflags)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003emsg_id\t= msg-\u003emsg_id;\n-\t\t __entry-\u003eobject_id\t= msg-\u003eobject_id;\n-\t\t __entry-\u003efd\t\t= load-\u003efd;\n-\t\t __entry-\u003eflags\t= load-\u003eflags;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x mid=%x oid=%x fd=%d f=%x\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003emsg_id,\n-\t\t __entry-\u003eobject_id,\n-\t\t __entry-\u003efd,\n-\t\t __entry-\u003eflags)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_copen,\n-\t TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id,\n-\t\t long len),\n-\n-\t TP_ARGS(obj, msg_id, len),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tmsg_id)\n-\t\t __field(long,\t\tlen)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003emsg_id\t= msg_id;\n-\t\t __entry-\u003elen\t= len;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x mid=%x l=%lx\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003emsg_id,\n-\t\t __entry-\u003elen)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_close,\n-\t TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg),\n-\n-\t TP_ARGS(obj, msg),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tmsg_id)\n-\t\t __field(unsigned int,\tobject_id)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003emsg_id\t= msg-\u003emsg_id;\n-\t\t __entry-\u003eobject_id\t= msg-\u003eobject_id;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x mid=%x oid=%x\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003emsg_id,\n-\t\t __entry-\u003eobject_id)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_read,\n-\t TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,\n-\t\t struct cachefiles_read *load),\n-\n-\t TP_ARGS(obj, msg, load),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tmsg_id)\n-\t\t __field(unsigned int,\tobject_id)\n-\t\t __field(loff_t,\t\tstart)\n-\t\t __field(size_t,\t\tlen)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003emsg_id\t= msg-\u003emsg_id;\n-\t\t __entry-\u003eobject_id\t= msg-\u003eobject_id;\n-\t\t __entry-\u003estart\t= load-\u003eoff;\n-\t\t __entry-\u003elen\t= load-\u003elen;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x mid=%x oid=%x s=%llx l=%zx\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003emsg_id,\n-\t\t __entry-\u003eobject_id,\n-\t\t __entry-\u003estart,\n-\t\t __entry-\u003elen)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_cread,\n-\t TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id),\n-\n-\t TP_ARGS(obj, msg_id),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tmsg_id)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003emsg_id\t= msg_id;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x mid=%x\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003emsg_id)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_fd_write,\n-\t TP_PROTO(struct cachefiles_object *obj, struct inode *backer,\n-\t\t loff_t start, size_t len),\n-\n-\t TP_ARGS(obj, backer, start, len),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tbacker)\n-\t\t __field(loff_t,\t\tstart)\n-\t\t __field(size_t,\t\tlen)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003ebacker\t= backer-\u003ei_ino;\n-\t\t __entry-\u003estart\t= start;\n-\t\t __entry-\u003elen\t= len;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x iB=%x s=%llx l=%zx\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003ebacker,\n-\t\t __entry-\u003estart,\n-\t\t __entry-\u003elen)\n-\t );\n-\n-TRACE_EVENT(cachefiles_ondemand_fd_release,\n-\t TP_PROTO(struct cachefiles_object *obj, int object_id),\n-\n-\t TP_ARGS(obj, object_id),\n-\n-\t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\tobj)\n-\t\t __field(unsigned int,\tobject_id)\n-\t\t\t ),\n-\n-\t TP_fast_assign(\n-\t\t __entry-\u003eobj\t= obj ? obj-\u003edebug_id : 0;\n-\t\t __entry-\u003eobject_id\t= object_id;\n-\t\t\t ),\n-\n-\t TP_printk(\"o=%08x oid=%x\",\n-\t\t __entry-\u003eobj,\n-\t\t __entry-\u003eobject_id)\n-\t );\n-\n #endif /* _TRACE_CACHEFILES_H */\n \n /* This part must be outside protection */\ndiff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h\nindex 082cb03c613160..e6bbf2a09d92e8 100644\n--- a/include/trace/events/netfs.h\n+++ b/include/trace/events/netfs.h\n@@ -30,8 +30,7 @@\n \tEM(netfs_write_trace_dio_write,\t\t\"DIO-WRITE\")\t\\\n \tEM(netfs_write_trace_unbuffered_write,\t\"UNB-WRITE\")\t\\\n \tEM(netfs_write_trace_writeback,\t\t\"WRITEBACK\")\t\\\n-\tEM(netfs_write_trace_writeback_single,\t\"WB-SINGLE\") \\\n-\tE_(netfs_write_trace_writethrough,\t\"WRITETHRU\")\n+\tE_(netfs_write_trace_writeback_single,\t\"WB-SINGLE\")\n \n #define netfs_rreq_origins\t\t\t\t\t\\\n \tEM(NETFS_READAHEAD,\t\t\t\"RA\")\t\t\\\n@@ -43,13 +42,17 @@\n \tEM(NETFS_DIO_READ,\t\t\t\"DR\")\t\t\\\n \tEM(NETFS_WRITEBACK,\t\t\t\"WB\")\t\t\\\n \tEM(NETFS_WRITEBACK_SINGLE,\t\t\"W1\")\t\t\\\n-\tEM(NETFS_WRITETHROUGH,\t\t\t\"WT\")\t\t\\\n \tEM(NETFS_UNBUFFERED_WRITE,\t\t\"UW\")\t\t\\\n \tEM(NETFS_DIO_WRITE,\t\t\t\"DW\")\t\t\\\n \tE_(NETFS_PGPRIV2_COPY_TO_CACHE,\t\t\"2C\")\n \n #define netfs_rreq_traces\t\t\t\t\t\\\n+\tEM(netfs_rreq_trace_all_queued,\t\t\"ALL-Q \")\t\\\n \tEM(netfs_rreq_trace_assess,\t\t\"ASSESS \")\t\\\n+\tEM(netfs_rreq_trace_cache_cancelled,\t\"CA-CNCL\")\t\\\n+\tEM(netfs_rreq_trace_cache_failed,\t\"CA-FAIL\")\t\\\n+\tEM(netfs_rreq_trace_cache_fail_collect,\t\"CA-F-CO\")\t\\\n+\tEM(netfs_rreq_trace_cache_no_space,\t\"CA-NOSP\")\t\\\n \tEM(netfs_rreq_trace_collect,\t\t\"COLLECT\")\t\\\n \tEM(netfs_rreq_trace_complete,\t\t\"COMPLET\")\t\\\n \tEM(netfs_rreq_trace_copy,\t\t\"COPY \")\t\\\n@@ -58,10 +61,14 @@\n \tEM(netfs_rreq_trace_end_copy_to_cache,\t\"END-C2C\")\t\\\n \tEM(netfs_rreq_trace_free,\t\t\"FREE \")\t\\\n \tEM(netfs_rreq_trace_intr,\t\t\"INTR \")\t\\\n+\tEM(netfs_rreq_trace_inval_cache,\t\"INVL-CA\")\t\\\n \tEM(netfs_rreq_trace_ki_complete,\t\"KI-CMPL\")\t\\\n+\tEM(netfs_rreq_trace_ra_put_ref,\t\t\"RA-PUT \")\t\\\n \tEM(netfs_rreq_trace_recollect,\t\t\"RECLLCT\")\t\\\n \tEM(netfs_rreq_trace_redirty,\t\t\"REDIRTY\")\t\\\n \tEM(netfs_rreq_trace_resubmit,\t\t\"RESUBMT\")\t\\\n+\tEM(netfs_rreq_trace_retry_begin,\t\"RETRY-BEGIN\")\t\\\n+\tEM(netfs_rreq_trace_retry_end,\t\t\"RETRY-END\")\t\\\n \tEM(netfs_rreq_trace_set_abandon,\t\"S-ABNDN\")\t\\\n \tEM(netfs_rreq_trace_set_pause,\t\t\"PAUSE \")\t\\\n \tEM(netfs_rreq_trace_unlock,\t\t\"UNLOCK \")\t\\\n@@ -70,13 +77,16 @@\n \tEM(netfs_rreq_trace_unpause,\t\t\"UNPAUSE\")\t\\\n \tEM(netfs_rreq_trace_wait_ip,\t\t\"WAIT-IP\")\t\\\n \tEM(netfs_rreq_trace_wait_pause,\t\t\"--PAUSED--\")\t\\\n+\tEM(netfs_rreq_trace_wait_put_ra_refs,\t\"WAIT-P-RA\")\t\\\n \tEM(netfs_rreq_trace_wait_quiesce,\t\"WAIT-QUIESCE\")\t\\\n \tEM(netfs_rreq_trace_waited_ip,\t\t\"DONE-IP\")\t\\\n \tEM(netfs_rreq_trace_waited_pause,\t\"--UNPAUSED--\")\t\\\n+\tEM(netfs_rreq_trace_waited_put_ra_refs,\t\"DONE-P-RA\")\t\\\n \tEM(netfs_rreq_trace_waited_quiesce,\t\"DONE-QUIESCE\")\t\\\n \tEM(netfs_rreq_trace_wake_ip,\t\t\"WAKE-IP\")\t\\\n \tEM(netfs_rreq_trace_wake_queue,\t\t\"WAKE-Q \")\t\\\n-\tE_(netfs_rreq_trace_write_done,\t\t\"WR-DONE\")\n+\tEM(netfs_rreq_trace_write_done,\t\t\"WR-DONE\")\t\\\n+\tE_(netfs_rreq_trace_zero_unread,\t\"ZERO-UR\")\n \n #define netfs_sreq_sources\t\t\t\t\t\\\n \tEM(NETFS_SOURCE_UNKNOWN,\t\t\"----\")\t\t\\\n@@ -124,6 +134,8 @@\n \tEM(netfs_sreq_trace_submit,\t\t\"SUBMT\")\t\\\n \tEM(netfs_sreq_trace_superfluous,\t\"SPRFL\")\t\\\n \tEM(netfs_sreq_trace_terminated,\t\t\"TERM \")\t\\\n+\tEM(netfs_sreq_trace_too_much,\t\t\"!TOOM\")\t\\\n+\tEM(netfs_sreq_trace_too_many_retries,\t\"!RETR\")\t\\\n \tEM(netfs_sreq_trace_wait_for,\t\t\"_WAIT\")\t\\\n \tEM(netfs_sreq_trace_write,\t\t\"WRITE\")\t\\\n \tEM(netfs_sreq_trace_write_skip,\t\t\"SKIP \")\t\\\n@@ -131,12 +143,12 @@\n \n #define netfs_failures\t\t\t\t\t\t\t\\\n \tEM(netfs_fail_check_write_begin,\t\"check-write-begin\")\t\\\n-\tEM(netfs_fail_copy_to_cache,\t\t\"copy-to-cache\")\t\\\n \tEM(netfs_fail_dio_read_short,\t\t\"dio-read-short\")\t\\\n \tEM(netfs_fail_dio_read_zero,\t\t\"dio-read-zero\")\t\\\n \tEM(netfs_fail_read,\t\t\t\"read\")\t\t\t\\\n \tEM(netfs_fail_short_read,\t\t\"short-read\")\t\t\\\n \tEM(netfs_fail_prepare_write,\t\t\"prep-write\")\t\t\\\n+\tEM(netfs_fail_upload,\t\t\t\"upload\")\t\t\\\n \tE_(netfs_fail_write,\t\t\t\"write\")\n \n #define netfs_rreq_ref_traces\t\t\t\t\t\\\n@@ -191,12 +203,12 @@\n \tEM(netfs_folio_trace_alloc_buffer,\t\"alloc-buf\")\t\\\n \tEM(netfs_folio_trace_cancel_copy,\t\"cancel-copy\")\t\\\n \tEM(netfs_folio_trace_cancel_store,\t\"cancel-store\")\t\\\n-\tEM(netfs_folio_trace_clear,\t\t\"clear\")\t\\\n-\tEM(netfs_folio_trace_clear_cc,\t\t\"clear-cc\")\t\\\n-\tEM(netfs_folio_trace_clear_g,\t\t\"clear-g\")\t\\\n-\tEM(netfs_folio_trace_clear_s,\t\t\"clear-s\")\t\\\n \tEM(netfs_folio_trace_copy_to_cache,\t\"mark-copy\")\t\\\n \tEM(netfs_folio_trace_end_copy,\t\t\"end-copy\")\t\\\n+\tEM(netfs_folio_trace_endwb,\t\t\"endwb\")\t\\\n+\tEM(netfs_folio_trace_endwb_cc,\t\t\"endwb-cc\")\t\\\n+\tEM(netfs_folio_trace_endwb_g,\t\t\"endwb-g\")\t\\\n+\tEM(netfs_folio_trace_endwb_s,\t\t\"endwb-s\")\t\\\n \tEM(netfs_folio_trace_filled_gaps,\t\"filled-gaps\")\t\\\n \tEM(netfs_folio_trace_invalidate_all,\t\"inval-all\")\t\\\n \tEM(netfs_folio_trace_invalidate_front,\t\"inval-front\")\t\\\n@@ -219,8 +231,8 @@\n \tEM(netfs_folio_trace_store,\t\t\"store\")\t\\\n \tEM(netfs_folio_trace_store_copy,\t\"store-copy\")\t\\\n \tEM(netfs_folio_trace_store_plus,\t\"store+\")\t\\\n-\tEM(netfs_folio_trace_wthru,\t\t\"wthru\")\t\\\n-\tE_(netfs_folio_trace_wthru_plus,\t\"wthru+\")\n+\tEM(netfs_folio_trace_zero,\t\t\"zero\")\t\t\\\n+\tE_(netfs_folio_trace_zero_ra,\t\t\"zero-ra\")\n \n #define netfs_collect_contig_traces\t\t\t\t\\\n \tEM(netfs_contig_trace_collect,\t\t\"Collect\")\t\\\n@@ -233,13 +245,13 @@\n \tEM(netfs_trace_donate_to_next,\t\t\"to-next\")\t\\\n \tE_(netfs_trace_donate_to_deferred_next,\t\"defer-next\")\n \n-#define netfs_folioq_traces\t\t\t\t\t\\\n-\tEM(netfs_trace_folioq_alloc_buffer,\t\"alloc-buf\")\t\\\n-\tEM(netfs_trace_folioq_clear,\t\t\"clear\")\t\\\n-\tEM(netfs_trace_folioq_delete,\t\t\"delete\")\t\\\n-\tEM(netfs_trace_folioq_make_space,\t\"make-space\")\t\\\n-\tEM(netfs_trace_folioq_rollbuf_init,\t\"roll-init\")\t\\\n-\tE_(netfs_trace_folioq_read_progress,\t\"r-progress\")\n+#define netfs_bvecq_traces\t\t\t\t\t\\\n+\tEM(netfs_trace_bvecq_alloc_buffer,\t\"alloc-buf\")\t\\\n+\tEM(netfs_trace_bvecq_clear,\t\t\"clear\")\t\\\n+\tEM(netfs_trace_bvecq_delete,\t\t\"delete\")\t\\\n+\tEM(netfs_trace_bvecq_make_space,\t\"make-space\")\t\\\n+\tEM(netfs_trace_bvecq_rollbuf_init,\t\"roll-init\")\t\\\n+\tE_(netfs_trace_bvecq_read_progress,\t\"r-progress\")\n \n #ifndef __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY\n #define __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY\n@@ -259,7 +271,7 @@ enum netfs_sreq_ref_trace { netfs_sreq_ref_traces } __mode(byte);\n enum netfs_folio_trace { netfs_folio_traces } __mode(byte);\n enum netfs_collect_contig_trace { netfs_collect_contig_traces } __mode(byte);\n enum netfs_donate_trace { netfs_donate_traces } __mode(byte);\n-enum netfs_folioq_trace { netfs_folioq_traces } __mode(byte);\n+enum netfs_bvecq_trace { netfs_bvecq_traces } __mode(byte);\n \n #endif\n \n@@ -283,7 +295,7 @@ netfs_sreq_ref_traces;\n netfs_folio_traces;\n netfs_collect_contig_traces;\n netfs_donate_traces;\n-netfs_folioq_traces;\n+netfs_bvecq_traces;\n \n /*\n * Now redefine the EM() and E_() macros to map the enums to the strings that\n@@ -304,6 +316,7 @@ TRACE_EVENT(netfs_read,\n \t TP_STRUCT__entry(\n \t\t __field(unsigned int,\t\trreq)\n \t\t __field(unsigned int,\t\tcookie)\n+\t\t __field(unsigned int,\t\tobject)\n \t\t __field(loff_t,\t\t\ti_size)\n \t\t __field(loff_t,\t\t\tstart)\n \t\t __field(size_t,\t\t\tlen)\n@@ -313,7 +326,8 @@ TRACE_EVENT(netfs_read,\n \n \t TP_fast_assign(\n \t\t __entry-\u003erreq\t= rreq-\u003edebug_id;\n-\t\t __entry-\u003ecookie\t= rreq-\u003ecache_resources.debug_id;\n+\t\t __entry-\u003ecookie\t= rreq-\u003ecache_resources.cookie_id;\n+\t\t __entry-\u003eobject\t= rreq-\u003ecache_resources.object_id;\n \t\t __entry-\u003ei_size\t= rreq-\u003ei_size;\n \t\t __entry-\u003estart\t= start;\n \t\t __entry-\u003elen\t= len;\n@@ -321,10 +335,10 @@ TRACE_EVENT(netfs_read,\n \t\t __entry-\u003enetfs_inode = rreq-\u003einode-\u003ei_ino;\n \t\t\t ),\n \n-\t TP_printk(\"R=%08x %s c=%08x ni=%llx s=%llx l=%zx sz=%llx\",\n+\t TP_printk(\"R=%08x %s c=%08x o=%08x ni=%llx s=%llx l=%zx sz=%llx\",\n \t\t __entry-\u003erreq,\n \t\t __print_symbolic(__entry-\u003ewhat, netfs_read_traces),\n-\t\t __entry-\u003ecookie,\n+\t\t __entry-\u003ecookie, __entry-\u003eobject,\n \t\t __entry-\u003enetfs_inode,\n \t\t __entry-\u003estart, __entry-\u003elen, __entry-\u003ei_size)\n \t );\n@@ -385,10 +399,10 @@ TRACE_EVENT(netfs_sreq,\n \t\t __entry-\u003elen\t= sreq-\u003elen;\n \t\t __entry-\u003etransferred = sreq-\u003etransferred;\n \t\t __entry-\u003estart\t= sreq-\u003estart;\n-\t\t __entry-\u003eslot\t= sreq-\u003eio_iter.folioq_slot;\n+\t\t __entry-\u003eslot\t= sreq-\u003econtent.slot;\n \t\t\t ),\n \n-\t TP_printk(\"R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx s=%u e=%d\",\n+\t TP_printk(\"R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx bv=%u e=%d\",\n \t\t __entry-\u003erreq, __entry-\u003eindex,\n \t\t __print_symbolic(__entry-\u003esource, netfs_sreq_sources),\n \t\t __print_symbolic(__entry-\u003ewhat, netfs_sreq_traces),\n@@ -496,6 +510,7 @@ TRACE_EVENT(netfs_folio,\n \t TP_STRUCT__entry(\n \t\t __field(u64,\t\t\tino)\n \t\t __field(pgoff_t,\t\t\tindex)\n+\t\t __field(unsigned long,\t\tpfn)\n \t\t __field(unsigned int,\t\tnr)\n \t\t __field(enum netfs_folio_trace,\twhy)\n \t\t\t ),\n@@ -506,13 +521,40 @@ TRACE_EVENT(netfs_folio,\n \t\t __entry-\u003ewhy = why;\n \t\t __entry-\u003eindex = folio-\u003eindex;\n \t\t __entry-\u003enr = folio_nr_pages(folio);\n+\t\t __entry-\u003epfn = folio_pfn(folio);\n \t\t\t ),\n \n-\t TP_printk(\"i=%05llx ix=%05lx-%05lx %s\",\n+\t TP_printk(\"p=%lx i=%05llx ix=%05lx-%05lx %s\",\n+\t\t __entry-\u003epfn,\n \t\t __entry-\u003eino, __entry-\u003eindex, __entry-\u003eindex + __entry-\u003enr - 1,\n \t\t __print_symbolic(__entry-\u003ewhy, netfs_folio_traces))\n \t );\n \n+TRACE_EVENT(netfs_wback,\n+\t TP_PROTO(struct netfs_io_request *wreq, struct folio *folio, unsigned int notes),\n+\n+\t TP_ARGS(wreq, folio, notes),\n+\n+\t TP_STRUCT__entry(\n+\t\t __field(pgoff_t,\t\t\tindex)\n+\t\t __field(unsigned int,\t\twreq)\n+\t\t __field(unsigned int,\t\tnr)\n+\t\t __field(unsigned int,\t\tnotes)\n+\t\t\t ),\n+\n+\t TP_fast_assign(\n+\t\t __entry-\u003ewreq = wreq-\u003edebug_id;\n+\t\t __entry-\u003enotes = notes;\n+\t\t __entry-\u003eindex = folio-\u003eindex;\n+\t\t __entry-\u003enr = folio_nr_pages(folio);\n+\t\t\t ),\n+\n+\t TP_printk(\"R=%08x ix=%05lx-%05lx n=%02x\",\n+\t\t __entry-\u003ewreq,\n+\t\t __entry-\u003eindex, __entry-\u003eindex + __entry-\u003enr - 1,\n+\t\t __entry-\u003enotes)\n+\t );\n+\n TRACE_EVENT(netfs_write_iter,\n \t TP_PROTO(const struct kiocb *iocb, const struct iov_iter *from),\n \n@@ -545,6 +587,7 @@ TRACE_EVENT(netfs_write,\n \t TP_STRUCT__entry(\n \t\t __field(unsigned int,\t\twreq)\n \t\t __field(unsigned int,\t\tcookie)\n+\t\t __field(unsigned int,\t\tobject)\n \t\t __field(unsigned int,\t\tino)\n \t\t __field(enum netfs_write_trace,\twhat)\n \t\t __field(unsigned long long,\t\tstart)\n@@ -552,20 +595,19 @@ TRACE_EVENT(netfs_write,\n \t\t\t ),\n \n \t TP_fast_assign(\n-\t\t struct netfs_inode *__ctx = netfs_inode(wreq-\u003einode);\n-\t\t struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);\n \t\t __entry-\u003ewreq\t= wreq-\u003edebug_id;\n-\t\t __entry-\u003ecookie\t= __cookie ? __cookie-\u003edebug_id : 0;\n+\t\t __entry-\u003ecookie\t= wreq-\u003ecache_resources.cookie_id;\n+\t\t __entry-\u003eobject\t= wreq-\u003ecache_resources.object_id;\n \t\t __entry-\u003eino\t= wreq-\u003einode-\u003ei_ino;\n \t\t __entry-\u003ewhat\t= what;\n \t\t __entry-\u003estart\t= wreq-\u003estart;\n \t\t __entry-\u003elen\t= wreq-\u003elen;\n \t\t\t ),\n \n-\t TP_printk(\"R=%08x %s c=%08x i=%x by=%llx-%llx\",\n+\t TP_printk(\"R=%08x %s c=%08x o=%08x i=%x by=%llx-%llx\",\n \t\t __entry-\u003ewreq,\n \t\t __print_symbolic(__entry-\u003ewhat, netfs_write_traces),\n-\t\t __entry-\u003ecookie,\n+\t\t __entry-\u003ecookie, __entry-\u003eobject,\n \t\t __entry-\u003eino,\n \t\t __entry-\u003estart, __entry-\u003estart + __entry-\u003elen - 1)\n \t );\n@@ -580,22 +622,23 @@ TRACE_EVENT(netfs_copy2cache,\n \t\t __field(unsigned int,\t\trreq)\n \t\t __field(unsigned int,\t\tcreq)\n \t\t __field(unsigned int,\t\tcookie)\n+\t\t __field(unsigned int,\t\tobject)\n \t\t __field(unsigned int,\t\tino)\n \t\t\t ),\n \n \t TP_fast_assign(\n-\t\t struct netfs_inode *__ctx = netfs_inode(rreq-\u003einode);\n-\t\t struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);\n \t\t __entry-\u003erreq\t= rreq-\u003edebug_id;\n \t\t __entry-\u003ecreq\t= creq-\u003edebug_id;\n-\t\t __entry-\u003ecookie\t= __cookie ? __cookie-\u003edebug_id : 0;\n+\t\t __entry-\u003ecookie\t= rreq-\u003ecache_resources.cookie_id;\n+\t\t __entry-\u003eobject\t= rreq-\u003ecache_resources.object_id;\n \t\t __entry-\u003eino\t= rreq-\u003einode-\u003ei_ino;\n \t\t\t ),\n \n-\t TP_printk(\"R=%08x CR=%08x c=%08x i=%x \",\n+\t TP_printk(\"R=%08x CR=%08x c=%08x o=%08x i=%x \",\n \t\t __entry-\u003erreq,\n \t\t __entry-\u003ecreq,\n \t\t __entry-\u003ecookie,\n+\t\t __entry-\u003eobject,\n \t\t __entry-\u003eino)\n \t );\n \n@@ -755,7 +798,7 @@ TRACE_EVENT(netfs_collect_stream,\n \t\t __entry-\u003ewreq\t= wreq-\u003edebug_id;\n \t\t __entry-\u003estream\t= stream-\u003estream_nr;\n \t\t __entry-\u003ecollected_to = stream-\u003ecollected_to;\n-\t\t __entry-\u003eissued_to\t= atomic64_read(\u0026wreq-\u003eissued_to);\n+\t\t __entry-\u003eissued_to\t= atomic64_read(\u0026stream-\u003eissued_to);\n \t\t\t ),\n \n \t TP_printk(\"R=%08x[%x:] cto=%llx ito=%llx\",\n@@ -763,27 +806,49 @@ TRACE_EVENT(netfs_collect_stream,\n \t\t __entry-\u003ecollected_to, __entry-\u003eissued_to)\n \t );\n \n-TRACE_EVENT(netfs_folioq,\n-\t TP_PROTO(const struct folio_queue *fq,\n-\t\t enum netfs_folioq_trace trace),\n+TRACE_EVENT(netfs_bvecq,\n+\t TP_PROTO(const struct bvecq *bq,\n+\t\t enum netfs_bvecq_trace trace),\n \n-\t TP_ARGS(fq, trace),\n+\t TP_ARGS(bq, trace),\n \n \t TP_STRUCT__entry(\n-\t\t __field(unsigned int,\t\trreq)\n \t\t __field(unsigned int,\t\tid)\n-\t\t __field(enum netfs_folioq_trace,\ttrace)\n+\t\t __field(enum netfs_bvecq_trace,\ttrace)\n \t\t\t ),\n \n \t TP_fast_assign(\n-\t\t __entry-\u003erreq\t= fq ? fq-\u003erreq_id : 0;\n-\t\t __entry-\u003eid\t\t= fq ? fq-\u003edebug_id : 0;\n+\t\t __entry-\u003eid\t\t= bq ? bq-\u003epriv : 0;\n \t\t __entry-\u003etrace\t= trace;\n \t\t\t ),\n \n-\t TP_printk(\"R=%08x fq=%x %s\",\n-\t\t __entry-\u003erreq, __entry-\u003eid,\n-\t\t __print_symbolic(__entry-\u003etrace, netfs_folioq_traces))\n+\t TP_printk(\"bq=%x %s\",\n+\t\t __entry-\u003eid,\n+\t\t __print_symbolic(__entry-\u003etrace, netfs_bvecq_traces))\n+\t );\n+\n+TRACE_EVENT(netfs_bv_slot,\n+\t TP_PROTO(const struct bvecq *bq, int slot),\n+\n+\t TP_ARGS(bq, slot),\n+\n+\t TP_STRUCT__entry(\n+\t\t __field(unsigned long,\t\tpfn)\n+\t\t __field(unsigned int,\t\toffset)\n+\t\t __field(unsigned int,\t\tlen)\n+\t\t __field(unsigned int,\t\tslot)\n+\t\t\t ),\n+\n+\t TP_fast_assign(\n+\t\t __entry-\u003eslot = slot;\n+\t\t __entry-\u003epfn = page_to_pfn(bq-\u003ebv[slot].bv_page);\n+\t\t __entry-\u003eoffset = bq-\u003ebv[slot].bv_offset;\n+\t\t __entry-\u003elen = bq-\u003ebv[slot].bv_len;\n+\t\t\t ),\n+\n+\t TP_printk(\"bq[%x] p=%lx %x-%x\",\n+\t\t __entry-\u003eslot,\n+\t\t __entry-\u003epfn, __entry-\u003eoffset, __entry-\u003eoffset + __entry-\u003elen)\n \t );\n \n #undef EM\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex c4673a54c4baf2..0268314ef60ba2 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -6749,8 +6749,6 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {\n \t/* amdgpu */\n \t{ \"amdgpu_vm_bo_map\", 0x1 },\n \t{ \"amdgpu_vm_bo_unmap\", 0x1 },\n-\t/* netfs */\n-\t{ \"netfs_folioq\", 0x1 },\n \t/* xfs from xfs_defer_pending_class */\n \t{ \"xfs_defer_create_intent\", 0x1 },\n \t{ \"xfs_defer_cancel_list\", 0x1 },\ndiff --git a/lib/iov_iter.c b/lib/iov_iter.c\nindex c2484551a4e863..b1d253f6986294 100644\n--- a/lib/iov_iter.c\n+++ b/lib/iov_iter.c\n@@ -538,37 +538,38 @@ static void iov_iter_iovec_advance(struct iov_iter *i, size_t size)\n \ti-\u003e__iov = iov;\n }\n \n-static void iov_iter_folioq_advance(struct iov_iter *i, size_t size)\n+static void iov_iter_bvecq_advance(struct iov_iter *i, size_t by)\n {\n-\tconst struct folio_queue *folioq = i-\u003efolioq;\n-\tunsigned int slot = i-\u003efolioq_slot;\n+\tconst struct bvecq *bq = i-\u003ebvecq;\n+\tunsigned int slot = i-\u003ebvecq_slot;\n \n \tif (!i-\u003ecount)\n \t\treturn;\n-\ti-\u003ecount -= size;\n-\n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tslot = 0;\n-\t}\n+\ti-\u003ecount -= by;\n \n-\tsize += i-\u003eiov_offset; /* From beginning of current segment. */\n+\tby += i-\u003eiov_offset; /* From beginning of current segment. */\n \tdo {\n-\t\tsize_t fsize = folioq_folio_size(folioq, slot);\n+\t\tsize_t len;\n \n-\t\tif (likely(size \u003c fsize))\n-\t\t\tbreak;\n-\t\tsize -= fsize;\n-\t\tslot++;\n-\t\tif (slot \u003e= folioq_nr_slots(folioq) \u0026\u0026 folioq-\u003enext) {\n-\t\t\tfolioq = folioq-\u003enext;\n+\t\tif (slot \u003e= bq-\u003enr_slots) {\n+\t\t\tif (!bq-\u003enext)\n+\t\t\t\tbreak;\n+\t\t\tbq = bq-\u003enext;\n \t\t\tslot = 0;\n+\t\t\tcontinue;\n \t\t}\n-\t} while (size);\n \n-\ti-\u003eiov_offset = size;\n-\ti-\u003efolioq_slot = slot;\n-\ti-\u003efolioq = folioq;\n+\t\tlen = bq-\u003ebv[slot].bv_len;\n+\n+\t\tif (likely(by \u003c len))\n+\t\t\tbreak;\n+\t\tby -= len;\n+\t\tslot++;\n+\t} while (by);\n+\n+\ti-\u003eiov_offset = by;\n+\ti-\u003ebvecq_slot = slot;\n+\ti-\u003ebvecq = bq;\n }\n \n void iov_iter_advance(struct iov_iter *i, size_t size)\n@@ -583,38 +584,39 @@ void iov_iter_advance(struct iov_iter *i, size_t size)\n \t\tiov_iter_iovec_advance(i, size);\n \t} else if (iov_iter_is_bvec(i)) {\n \t\tiov_iter_bvec_advance(i, size);\n-\t} else if (iov_iter_is_folioq(i)) {\n-\t\tiov_iter_folioq_advance(i, size);\n+\t} else if (iov_iter_is_bvecq(i)) {\n+\t\tiov_iter_bvecq_advance(i, size);\n \t} else if (iov_iter_is_discard(i)) {\n \t\ti-\u003ecount -= size;\n \t}\n }\n EXPORT_SYMBOL(iov_iter_advance);\n \n-static void iov_iter_folioq_revert(struct iov_iter *i, size_t unroll)\n+static void iov_iter_bvecq_revert(struct iov_iter *i, size_t unroll)\n {\n-\tconst struct folio_queue *folioq = i-\u003efolioq;\n-\tunsigned int slot = i-\u003efolioq_slot;\n+\tconst struct bvecq *bq = i-\u003ebvecq;\n+\tunsigned int slot = i-\u003ebvecq_slot;\n \n \tfor (;;) {\n-\t\tsize_t fsize;\n+\t\tsize_t len;\n \n \t\tif (slot == 0) {\n-\t\t\tfolioq = folioq-\u003eprev;\n-\t\t\tslot = folioq_nr_slots(folioq);\n+\t\t\tbq = bq-\u003eprev;\n+\t\t\tslot = bq-\u003enr_slots;\n+\t\t\tcontinue;\n \t\t}\n \t\tslot--;\n \n-\t\tfsize = folioq_folio_size(folioq, slot);\n-\t\tif (unroll \u003c= fsize) {\n-\t\t\ti-\u003eiov_offset = fsize - unroll;\n+\t\tlen = bq-\u003ebv[slot].bv_len;\n+\t\tif (unroll \u003c= len) {\n+\t\t\ti-\u003eiov_offset = len - unroll;\n \t\t\tbreak;\n \t\t}\n-\t\tunroll -= fsize;\n+\t\tunroll -= len;\n \t}\n \n-\ti-\u003efolioq_slot = slot;\n-\ti-\u003efolioq = folioq;\n+\ti-\u003ebvecq_slot = slot;\n+\ti-\u003ebvecq = bq;\n }\n \n void iov_iter_revert(struct iov_iter *i, size_t unroll)\n@@ -648,9 +650,9 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)\n \t\t\t}\n \t\t\tunroll -= n;\n \t\t}\n-\t} else if (iov_iter_is_folioq(i)) {\n+\t} else if (iov_iter_is_bvecq(i)) {\n \t\ti-\u003eiov_offset = 0;\n-\t\tiov_iter_folioq_revert(i, unroll);\n+\t\tiov_iter_bvecq_revert(i, unroll);\n \t} else { /* same logics for iovec and kvec */\n \t\tconst struct iovec *iov = iter_iov(i);\n \t\twhile (1) {\n@@ -678,9 +680,28 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i)\n \t\tif (iov_iter_is_bvec(i))\n \t\t\treturn min(i-\u003ecount, i-\u003ebvec-\u003ebv_len - i-\u003eiov_offset);\n \t}\n-\tif (unlikely(iov_iter_is_folioq(i)))\n-\t\treturn !i-\u003ecount ? 0 :\n-\t\t\tumin(folioq_folio_size(i-\u003efolioq, i-\u003efolioq_slot), i-\u003ecount);\n+\tif (!i-\u003ecount)\n+\t\treturn 0;\n+\tif (unlikely(iov_iter_is_bvecq(i))) {\n+\t\tconst struct bvecq *bq = i-\u003ebvecq;\n+\t\tunsigned int slot = i-\u003ebvecq_slot;\n+\t\tsize_t offset = i-\u003eiov_offset;\n+\n+\t\tfor (;;) {\n+\t\t\twhile (slot \u003e= bq-\u003enr_slots) {\n+\t\t\t\tbq = bq-\u003enext;\n+\t\t\t\tif (!bq)\n+\t\t\t\t\treturn 0;\n+\t\t\t\tslot = 0;\n+\t\t\t\toffset = 0;\n+\t\t\t}\n+\t\t\tif (bq-\u003ebv[slot].bv_len \u003e offset)\n+\t\t\t\tbreak;\n+\t\t\tslot++;\n+\t\t\toffset = 0;\n+\t\t}\n+\t\treturn umin(i-\u003ecount, bq-\u003ebv[slot].bv_len - offset);\n+\t}\n \treturn i-\u003ecount;\n }\n EXPORT_SYMBOL(iov_iter_single_seg_count);\n@@ -718,34 +739,33 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction,\n EXPORT_SYMBOL(iov_iter_bvec);\n \n /**\n- * iov_iter_folio_queue - Initialise an I/O iterator to use the folios in a folio queue\n+ * iov_iter_bvec_queue - Initialise an I/O iterator to use a segmented bvec queue\n * @i: The iterator to initialise.\n * @direction: The direction of the transfer.\n- * @folioq: The starting point in the folio queue.\n- * @first_slot: The first slot in the folio queue to use\n- * @offset: The offset into the folio in the first slot to start at\n+ * @bvecq: The starting point in the bvec queue.\n+ * @first_slot: The first slot in the bvec queue to use\n+ * @offset: The offset into the bvec in the first slot to start at\n * @count: The size of the I/O buffer in bytes.\n *\n- * Set up an I/O iterator to either draw data out of the pages attached to an\n- * inode or to inject data into those pages. The pages *must* be prevented\n- * from evaporation, either by taking a ref on them or locking them by the\n- * caller.\n+ * Set up an I/O iterator to either draw data out of the buffers attached to an\n+ * inode or to inject data into those buffers. The pages *must* be prevented\n+ * from evaporation, either by the caller.\n */\n-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,\n-\t\t\t const struct folio_queue *folioq, unsigned int first_slot,\n-\t\t\t unsigned int offset, size_t count)\n+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,\n+\t\t\t const struct bvecq *bvecq, unsigned int first_slot,\n+\t\t\t unsigned int offset, size_t count)\n {\n-\tBUG_ON(direction \u0026 ~1);\n+\tWARN_ON(direction \u0026 ~(READ | WRITE));\n \t*i = (struct iov_iter) {\n-\t\t.iter_type = ITER_FOLIOQ,\n-\t\t.data_source = direction,\n-\t\t.folioq = folioq,\n-\t\t.folioq_slot = first_slot,\n-\t\t.count = count,\n-\t\t.iov_offset = offset,\n+\t\t.iter_type\t= ITER_BVECQ,\n+\t\t.data_source\t= direction,\n+\t\t.bvecq\t\t= bvecq,\n+\t\t.bvecq_slot\t= first_slot,\n+\t\t.count\t\t= count,\n+\t\t.iov_offset\t= offset,\n \t};\n }\n-EXPORT_SYMBOL(iov_iter_folio_queue);\n+EXPORT_SYMBOL(iov_iter_bvec_queue);\n \n /**\n * iov_iter_xarray - Initialise an I/O iterator to use the pages in an xarray\n@@ -839,6 +859,39 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)\n \treturn res;\n }\n \n+static unsigned long iov_iter_alignment_bvecq(const struct iov_iter *iter)\n+{\n+\tconst struct bvecq *bq;\n+\tunsigned long res = 0;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n+\tsize_t skip = iter-\u003eiov_offset;\n+\tsize_t size = iter-\u003ecount;\n+\n+\tif (!size)\n+\t\treturn res;\n+\n+\tfor (bq = iter-\u003ebvecq; bq; bq = bq-\u003enext) {\n+\t\tfor (; slot \u003c bq-\u003enr_slots; slot++) {\n+\t\t\tconst struct bio_vec *bvec = \u0026bq-\u003ebv[slot];\n+\t\t\tsize_t part = umin(bvec-\u003ebv_len - skip, size);\n+\n+\t\t\tif (part) {\n+\t\t\t\tres |= bvec-\u003ebv_offset + skip;\n+\t\t\t\tres |= part;\n+\t\t\t}\n+\n+\t\t\tsize -= part;\n+\t\t\tif (size == 0)\n+\t\t\t\treturn res;\n+\t\t\tskip = 0;\n+\t\t}\n+\n+\t\tslot = 0;\n+\t}\n+\n+\treturn res;\n+}\n+\n unsigned long iov_iter_alignment(const struct iov_iter *i)\n {\n \tif (likely(iter_is_ubuf(i))) {\n@@ -854,10 +907,10 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)\n \n \tif (iov_iter_is_bvec(i))\n \t\treturn iov_iter_alignment_bvec(i);\n+\tif (iov_iter_is_bvecq(i))\n+\t\treturn iov_iter_alignment_bvecq(i);\n \n-\t/* With both xarray and folioq types, we're dealing with whole folios. */\n-\tif (iov_iter_is_folioq(i))\n-\t\treturn i-\u003eiov_offset | i-\u003ecount;\n+\t/* With the xarray type, we're dealing with whole folios. */\n \tif (iov_iter_is_xarray(i))\n \t\treturn (i-\u003exarray_start + i-\u003eiov_offset) | i-\u003ecount;\n \n@@ -910,118 +963,34 @@ static int want_pages_array(struct page ***res, size_t size,\n \treturn count;\n }\n \n-static ssize_t iter_folioq_get_pages(struct iov_iter *iter,\n+/*\n+ * Wrap iov_iter_extract_pages() and then pin the non-slab pages we got back.\n+ * This only works for non-user iterator types as get_pages uses get_user_pages\n+ * not pin_user_pages.\n+ */\n+static ssize_t iter_get_kernel_pages(struct iov_iter *iter,\n \t\t\t\t struct page ***ppages, size_t maxsize,\n \t\t\t\t unsigned maxpages, size_t *_start_offset)\n {\n-\tconst struct folio_queue *folioq = iter-\u003efolioq;\n \tstruct page **pages;\n-\tunsigned int slot = iter-\u003efolioq_slot;\n-\tsize_t extracted = 0, count = iter-\u003ecount, iov_offset = iter-\u003eiov_offset;\n+\tssize_t ret, done;\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tslot = 0;\n-\t\tif (WARN_ON(iov_offset != 0))\n-\t\t\treturn -EIO;\n-\t}\n+\tret = iov_iter_extract_pages(iter, ppages, maxsize, maxpages,\n+\t\t\t\t 0, _start_offset);\n+\tif (ret \u003c= 0)\n+\t\treturn ret;\n \n-\tmaxpages = want_pages_array(ppages, maxsize, iov_offset \u0026 ~PAGE_MASK, maxpages);\n-\tif (!maxpages)\n-\t\treturn -ENOMEM;\n-\t*_start_offset = iov_offset \u0026 ~PAGE_MASK;\n \tpages = *ppages;\n+\tfor (done = ret + *_start_offset; done \u003e 0; done -= PAGE_SIZE) {\n+\t\tstruct folio *folio = page_folio(*pages);\n \n-\tfor (;;) {\n-\t\tstruct folio *folio = folioq_folio(folioq, slot);\n-\t\tsize_t offset = iov_offset, fsize = folioq_folio_size(folioq, slot);\n-\t\tsize_t part = PAGE_SIZE - offset % PAGE_SIZE;\n-\n-\t\tif (offset \u003c fsize) {\n-\t\t\tpart = umin(part, umin(maxsize - extracted, fsize - offset));\n-\t\t\tcount -= part;\n-\t\t\tiov_offset += part;\n-\t\t\textracted += part;\n-\n-\t\t\t*pages = folio_page(folio, offset / PAGE_SIZE);\n-\t\t\tget_page(*pages);\n-\t\t\tpages++;\n-\t\t\tmaxpages--;\n-\t\t}\n-\n-\t\tif (maxpages == 0 || extracted \u003e= maxsize)\n-\t\t\tbreak;\n-\n-\t\tif (iov_offset \u003e= fsize) {\n-\t\t\tiov_offset = 0;\n-\t\t\tslot++;\n-\t\t\tif (slot == folioq_nr_slots(folioq) \u0026\u0026 folioq-\u003enext) {\n-\t\t\t\tfolioq = folioq-\u003enext;\n-\t\t\t\tslot = 0;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\titer-\u003ecount = count;\n-\titer-\u003eiov_offset = iov_offset;\n-\titer-\u003efolioq = folioq;\n-\titer-\u003efolioq_slot = slot;\n-\treturn extracted;\n-}\n-\n-static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,\n-\t\t\t\t\t pgoff_t index, unsigned int nr_pages)\n-{\n-\tXA_STATE(xas, xa, index);\n-\tstruct folio *folio;\n-\tunsigned int ret = 0;\n-\n-\trcu_read_lock();\n-\tfor (folio = xas_load(\u0026xas); folio; folio = xas_next(\u0026xas)) {\n-\t\tif (xas_retry(\u0026xas, folio))\n-\t\t\tcontinue;\n-\n-\t\t/* Has the folio moved or been split? */\n-\t\tif (unlikely(folio != xas_reload(\u0026xas))) {\n-\t\t\txas_reset(\u0026xas);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tpages[ret] = folio_file_page(folio, xas.xa_index);\n-\t\tfolio_get(folio);\n-\t\tif (++ret == nr_pages)\n-\t\t\tbreak;\n+\t\tif (!folio_test_slab(folio))\n+\t\t\tfolio_get(folio);\n+\t\tpages++;\n \t}\n-\trcu_read_unlock();\n \treturn ret;\n }\n \n-static ssize_t iter_xarray_get_pages(struct iov_iter *i,\n-\t\t\t\t struct page ***pages, size_t maxsize,\n-\t\t\t\t unsigned maxpages, size_t *_start_offset)\n-{\n-\tunsigned nr, offset, count;\n-\tpgoff_t index;\n-\tloff_t pos;\n-\n-\tpos = i-\u003exarray_start + i-\u003eiov_offset;\n-\tindex = pos \u003e\u003e PAGE_SHIFT;\n-\toffset = pos \u0026 ~PAGE_MASK;\n-\t*_start_offset = offset;\n-\n-\tcount = want_pages_array(pages, maxsize, offset, maxpages);\n-\tif (!count)\n-\t\treturn -ENOMEM;\n-\tnr = iter_xarray_populate_pages(*pages, i-\u003exarray, index, count);\n-\tif (nr == 0)\n-\t\treturn 0;\n-\n-\tmaxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize);\n-\ti-\u003eiov_offset += maxsize;\n-\ti-\u003ecount -= maxsize;\n-\treturn maxsize;\n-}\n-\n /* must be done on non-empty ITER_UBUF or ITER_IOVEC one */\n static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)\n {\n@@ -1044,22 +1013,6 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)\n \tBUG(); // if it had been empty, we wouldn't get called\n }\n \n-/* must be done on non-empty ITER_BVEC one */\n-static struct page *first_bvec_segment(const struct iov_iter *i,\n-\t\t\t\t size_t *size, size_t *start)\n-{\n-\tstruct page *page;\n-\tsize_t skip = i-\u003eiov_offset, len;\n-\n-\tlen = i-\u003ebvec-\u003ebv_len - skip;\n-\tif (*size \u003e len)\n-\t\t*size = len;\n-\tskip += i-\u003ebvec-\u003ebv_offset;\n-\tpage = i-\u003ebvec-\u003ebv_page + skip / PAGE_SIZE;\n-\t*start = skip % PAGE_SIZE;\n-\treturn page;\n-}\n-\n static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,\n \t\t struct page ***pages, size_t maxsize,\n \t\t unsigned int maxpages, size_t *start)\n@@ -1095,36 +1048,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,\n \t\tiov_iter_advance(i, maxsize);\n \t\treturn maxsize;\n \t}\n-\tif (iov_iter_is_bvec(i)) {\n-\t\tstruct page **p;\n-\t\tstruct page *page;\n \n-\t\tpage = first_bvec_segment(i, \u0026maxsize, start);\n-\t\tn = want_pages_array(pages, maxsize, *start, maxpages);\n-\t\tif (!n)\n-\t\t\treturn -ENOMEM;\n-\t\tp = *pages;\n-\t\tfor (int k = 0; k \u003c n; k++) {\n-\t\t\tstruct folio *folio = page_folio(page + k);\n-\t\t\tp[k] = page + k;\n-\t\t\tif (!folio_test_slab(folio))\n-\t\t\t\tfolio_get(folio);\n-\t\t}\n-\t\tmaxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start);\n-\t\ti-\u003ecount -= maxsize;\n-\t\ti-\u003eiov_offset += maxsize;\n-\t\tif (i-\u003eiov_offset == i-\u003ebvec-\u003ebv_len) {\n-\t\t\ti-\u003eiov_offset = 0;\n-\t\t\ti-\u003ebvec++;\n-\t\t\ti-\u003enr_segs--;\n-\t\t}\n-\t\treturn maxsize;\n-\t}\n-\tif (iov_iter_is_folioq(i))\n-\t\treturn iter_folioq_get_pages(i, pages, maxsize, maxpages, start);\n-\tif (iov_iter_is_xarray(i))\n-\t\treturn iter_xarray_get_pages(i, pages, maxsize, maxpages, start);\n-\treturn -EFAULT;\n+\tif (iov_iter_is_kvec(i))\n+\t\treturn -EFAULT;\n+\treturn iter_get_kernel_pages(i, pages, maxsize, maxpages, start);\n }\n \n ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,\n@@ -1192,6 +1119,38 @@ static int bvec_npages(const struct iov_iter *i, int maxpages)\n \treturn npages;\n }\n \n+static size_t iov_npages_bvecq(const struct iov_iter *iter, size_t maxpages)\n+{\n+\tconst struct bvecq *bq;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n+\tsize_t npages = 0;\n+\tsize_t skip = iter-\u003eiov_offset;\n+\tsize_t size = iter-\u003ecount;\n+\n+\tfor (bq = iter-\u003ebvecq; bq; bq = bq-\u003enext) {\n+\t\tfor (; slot \u003c bq-\u003enr_slots; slot++) {\n+\t\t\tconst struct bio_vec *bvec = \u0026bq-\u003ebv[slot];\n+\t\t\tsize_t offs = (bvec-\u003ebv_offset + skip) % PAGE_SIZE;\n+\t\t\tsize_t part = umin(bvec-\u003ebv_len - skip, size);\n+\n+\t\t\tif (part) {\n+\t\t\t\tnpages += DIV_ROUND_UP(offs + part, PAGE_SIZE);\n+\t\t\t\tif (npages \u003e= maxpages)\n+\t\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tsize -= part;\n+\t\t\tif (!size)\n+\t\t\t\tgoto out;\n+\t\t\tskip = 0;\n+\t\t}\n+\n+\t\tslot = 0;\n+\t}\n+out:\n+\treturn umin(npages, maxpages);\n+}\n+\n int iov_iter_npages(const struct iov_iter *i, int maxpages)\n {\n \tif (unlikely(!i-\u003ecount))\n@@ -1206,11 +1165,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages)\n \t\treturn iov_npages(i, maxpages);\n \tif (iov_iter_is_bvec(i))\n \t\treturn bvec_npages(i, maxpages);\n-\tif (iov_iter_is_folioq(i)) {\n-\t\tunsigned offset = i-\u003eiov_offset % PAGE_SIZE;\n-\t\tint npages = DIV_ROUND_UP(offset + i-\u003ecount, PAGE_SIZE);\n-\t\treturn min(npages, maxpages);\n-\t}\n+\tif (iov_iter_is_bvecq(i))\n+\t\treturn iov_npages_bvecq(i, maxpages);\n \tif (iov_iter_is_xarray(i)) {\n \t\tunsigned offset = (i-\u003exarray_start + i-\u003eiov_offset) % PAGE_SIZE;\n \t\tint npages = DIV_ROUND_UP(offset + i-\u003ecount, PAGE_SIZE);\n@@ -1493,64 +1449,143 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)\n }\n \n /*\n- * Extract a list of contiguous pages from an ITER_FOLIOQ iterator. This does\n- * not get references on the pages, nor does it get a pin on them.\n+ * Count the number of virtually contiguous pages coming up next in an\n+ * ITER_BVECQ iterator, up to the specified maxima.\n */\n-static ssize_t iov_iter_extract_folioq_pages(struct iov_iter *i,\n-\t\t\t\t\t struct page ***pages, size_t maxsize,\n-\t\t\t\t\t unsigned int maxpages,\n-\t\t\t\t\t iov_iter_extraction_t extraction_flags,\n-\t\t\t\t\t size_t *offset0)\n+static unsigned int iter_count_bvecq_pages(const struct iov_iter *iter,\n+\t\t\t\t\t size_t maxsize,\n+\t\t\t\t\t unsigned int maxpages)\n+{\n+\tconst struct bvecq *bvecq = iter-\u003ebvecq;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n+\tssize_t remain = umin(maxsize, iter-\u003ecount);\n+\tsize_t count = 0, offset = iter-\u003eiov_offset;\n+\n+\tdo {\n+\t\tconst struct bio_vec *bv;\n+\t\tsize_t boff, blen;\n+\n+\t\tif (slot \u003e= bvecq-\u003enr_slots) {\n+\t\t\tif (!bvecq-\u003enext) {\n+\t\t\t\tWARN_ON_ONCE(remain \u003e 0);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbvecq = bvecq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbv = \u0026bvecq-\u003ebv[slot++];\n+\t\tboff = bv-\u003ebv_offset;\n+\t\tblen = bv-\u003ebv_len;\n+\n+\t\tif (unlikely(!bv-\u003ebv_page)) {\n+\t\t\tif (blen \u0026\u0026 count \u003e 0)\n+\t\t\t\tbreak;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!PAGE_ALIGNED(boff) \u0026\u0026 count \u003e 0)\n+\t\t\tbreak;\n+\n+\t\tboff += offset;\n+\t\tblen -= offset;\n+\t\toffset = 0;\n+\t\tif (!blen)\n+\t\t\tcontinue;\n+\n+\t\tblen = umin(blen, remain);\n+\t\tremain -= blen;\n+\t\tblen += offset_in_page(boff);\n+\t\tcount += DIV_ROUND_UP(blen, PAGE_SIZE);\n+\n+\t\tif (!PAGE_ALIGNED(blen))\n+\t\t\tbreak;\n+\t} while (remain \u003e 0 \u0026\u0026 count \u003c maxpages);\n+\n+\treturn umin(count, maxpages);\n+}\n+\n+/*\n+ * Extract a list of virtually contiguous pages from an ITER_BVECQ iterator.\n+ * This does not get references on the pages, nor does it get a pin on them.\n+ */\n+static ssize_t iov_iter_extract_bvecq_pages(struct iov_iter *iter,\n+\t\t\t\t\t struct page ***pages, size_t maxsize,\n+\t\t\t\t\t unsigned int maxpages,\n+\t\t\t\t\t iov_iter_extraction_t extraction_flags,\n+\t\t\t\t\t size_t *offset0)\n {\n-\tconst struct folio_queue *folioq = i-\u003efolioq;\n+\tconst struct bvecq *bvecq;\n \tstruct page **p;\n-\tunsigned int nr = 0;\n-\tsize_t extracted = 0, offset, slot = i-\u003efolioq_slot;\n+\tunsigned int slot, nr = 0;\n+\tsize_t extracted = 0, offset;\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tslot = 0;\n-\t\tif (WARN_ON(i-\u003eiov_offset != 0))\n-\t\t\treturn -EIO;\n-\t}\n+\t/* Count the next run of virtually contiguous pages. */\n+\tmaxpages = iter_count_bvecq_pages(iter, maxsize, maxpages);\n \n-\toffset = i-\u003eiov_offset \u0026 ~PAGE_MASK;\n-\t*offset0 = offset;\n+\tif (!*pages) {\n+\t\t*pages = kvmalloc_array(maxpages, sizeof(struct page *), GFP_KERNEL);\n+\t\tif (!*pages)\n+\t\t\treturn -ENOMEM;\n+\t}\n \n-\tmaxpages = want_pages_array(pages, maxsize, offset, maxpages);\n-\tif (!maxpages)\n-\t\treturn -ENOMEM;\n \tp = *pages;\n \n-\tfor (;;) {\n-\t\tstruct folio *folio = folioq_folio(folioq, slot);\n-\t\tsize_t offset = i-\u003eiov_offset, fsize = folioq_folio_size(folioq, slot);\n-\t\tsize_t part = PAGE_SIZE - offset % PAGE_SIZE;\n-\n-\t\tif (offset \u003c fsize) {\n-\t\t\tpart = umin(part, umin(maxsize - extracted, fsize - offset));\n-\t\t\ti-\u003ecount -= part;\n-\t\t\ti-\u003eiov_offset += part;\n-\t\t\textracted += part;\n+\t/* Now transcribe the page pointers. */\n+\textracted = 0;\n+\tbvecq = iter-\u003ebvecq;\n+\toffset = iter-\u003eiov_offset;\n+\tslot = iter-\u003ebvecq_slot;\n \n-\t\t\tp[nr++] = folio_page(folio, offset / PAGE_SIZE);\n+\tdo {\n+\t\tconst struct bio_vec *bv;\n+\t\tsize_t boff, blen;\n+\n+\t\tif (slot \u003e= bvecq-\u003enr_slots) {\n+\t\t\tif (!bvecq-\u003enext) {\n+\t\t\t\tWARN_ON_ONCE(extracted \u003c iter-\u003ecount);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbvecq = bvecq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n \t\t}\n \n-\t\tif (nr \u003e= maxpages || extracted \u003e= maxsize)\n-\t\t\tbreak;\n+\t\tbv = \u0026bvecq-\u003ebv[slot];\n+\t\tboff = bv-\u003ebv_offset;\n+\t\tblen = bv-\u003ebv_len;\n+\n+\t\tif (!bv-\u003ebv_page)\n+\t\t\tblen = 0;\n \n-\t\tif (i-\u003eiov_offset \u003e= fsize) {\n-\t\t\ti-\u003eiov_offset = 0;\n+\t\tif (offset \u003c blen) {\n+\t\t\tsize_t part = umin(maxsize - extracted, blen - offset);\n+\t\t\tsize_t poff = (boff + offset) % PAGE_SIZE;\n+\t\t\tsize_t pix = (boff + offset) / PAGE_SIZE;\n+\n+\t\t\tif (poff + part \u003e PAGE_SIZE)\n+\t\t\t\tpart = PAGE_SIZE - poff;\n+\n+\t\t\tif (!extracted)\n+\t\t\t\t*offset0 = poff;\n+\n+\t\t\tp[nr++] = bv-\u003ebv_page + pix;\n+\t\t\toffset += part;\n+\t\t\textracted += part;\n+\t\t}\n+\n+\t\tif (offset \u003e= blen) {\n+\t\t\toffset = 0;\n \t\t\tslot++;\n-\t\t\tif (slot == folioq_nr_slots(folioq) \u0026\u0026 folioq-\u003enext) {\n-\t\t\t\tfolioq = folioq-\u003enext;\n-\t\t\t\tslot = 0;\n-\t\t\t}\n \t\t}\n-\t}\n+\t} while (nr \u003c maxpages \u0026\u0026 extracted \u003c maxsize);\n \n-\ti-\u003efolioq = folioq;\n-\ti-\u003efolioq_slot = slot;\n+\titer-\u003ebvecq = bvecq;\n+\titer-\u003ebvecq_slot = slot;\n+\titer-\u003eiov_offset = offset;\n+\titer-\u003ecount -= extracted;\n \treturn extracted;\n }\n \n@@ -1816,8 +1851,8 @@ static ssize_t iov_iter_extract_user_pages(struct iov_iter *i,\n * added to the pages, but refs will not be taken.\n * iov_iter_extract_will_pin() will return true.\n *\n- * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_FOLIOQ or ITER_XARRAY, the\n- * pages are merely listed; no extra refs or pins are obtained.\n+ * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_XARRAY, the pages are\n+ * merely listed; no extra refs or pins are obtained.\n * iov_iter_extract_will_pin() will return 0.\n *\n * Note also:\n@@ -1852,10 +1887,10 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,\n \t\treturn iov_iter_extract_bvec_pages(i, pages, maxsize,\n \t\t\t\t\t\t maxpages, extraction_flags,\n \t\t\t\t\t\t offset0);\n-\tif (iov_iter_is_folioq(i))\n-\t\treturn iov_iter_extract_folioq_pages(i, pages, maxsize,\n-\t\t\t\t\t\t maxpages, extraction_flags,\n-\t\t\t\t\t\t offset0);\n+\tif (iov_iter_is_bvecq(i))\n+\t\treturn iov_iter_extract_bvecq_pages(i, pages, maxsize,\n+\t\t\t\t\t\t maxpages, extraction_flags,\n+\t\t\t\t\t\t offset0);\n \tif (iov_iter_is_xarray(i))\n \t\treturn iov_iter_extract_xarray_pages(i, pages, maxsize,\n \t\t\t\t\t\t maxpages, extraction_flags,\ndiff --git a/lib/scatterlist.c b/lib/scatterlist.c\nindex 6ea40d2e624755..b9a7298306d95d 100644\n--- a/lib/scatterlist.c\n+++ b/lib/scatterlist.c\n@@ -10,8 +10,8 @@\n #include \u003clinux/highmem.h\u003e\n #include \u003clinux/kmemleak.h\u003e\n #include \u003clinux/bvec.h\u003e\n+#include \u003clinux/bvecq.h\u003e\n #include \u003clinux/uio.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n \n /**\n * sg_nents - return total count of entries in scatterlist\n@@ -1268,61 +1268,59 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter,\n }\n \n /*\n- * Extract up to sg_max folios from an FOLIOQ-type iterator and add them to\n+ * Extract up to sg_max folios from an BVECQ-type iterator and add them to\n * the scatterlist. The pages are not pinned.\n */\n-static ssize_t extract_folioq_to_sg(struct iov_iter *iter,\n+static ssize_t extract_bvecq_to_sg(struct iov_iter *iter,\n \t\t\t\t ssize_t maxsize,\n \t\t\t\t struct sg_table *sgtable,\n \t\t\t\t unsigned int sg_max,\n \t\t\t\t iov_iter_extraction_t extraction_flags)\n {\n-\tconst struct folio_queue *folioq = iter-\u003efolioq;\n+\tconst struct bvecq *bvecq = iter-\u003ebvecq;\n \tstruct scatterlist *sg = sgtable-\u003esgl + sgtable-\u003enents;\n-\tunsigned int slot = iter-\u003efolioq_slot;\n+\tunsigned int slot = iter-\u003ebvecq_slot;\n \tssize_t ret = 0;\n \tsize_t offset = iter-\u003eiov_offset;\n \n-\tBUG_ON(!folioq);\n+\tmaxsize = umin(maxsize, iov_iter_count(iter));\n \n-\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\tfolioq = folioq-\u003enext;\n-\t\tif (WARN_ON_ONCE(!folioq))\n-\t\t\treturn 0;\n-\t\tslot = 0;\n-\t}\n-\n-\tdo {\n-\t\tstruct folio *folio = folioq_folio(folioq, slot);\n-\t\tsize_t fsize = folioq_folio_size(folioq, slot);\n+\twhile (sg_max \u003e 0 \u0026\u0026 ret \u003c maxsize) {\n+\t\tconst struct bio_vec *bv;\n+\t\tsize_t blen, part;\n \n-\t\tif (offset \u003c fsize) {\n-\t\t\tsize_t part = umin(maxsize - ret, fsize - offset);\n-\n-\t\t\tsg_set_page(sg, folio_page(folio, 0), part, offset);\n-\t\t\tsgtable-\u003enents++;\n-\t\t\tsg++;\n-\t\t\tsg_max--;\n-\t\t\toffset += part;\n-\t\t\tret += part;\n+\t\tif (slot \u003e= bvecq-\u003enr_slots) {\n+\t\t\tif (!bvecq-\u003enext) {\n+\t\t\t\tWARN_ON_ONCE(ret \u003c iter-\u003ecount);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbvecq = bvecq-\u003enext;\n+\t\t\tslot = 0;\n+\t\t\toffset = 0;\n+\t\t\tcontinue;\n \t\t}\n \n-\t\tif (offset \u003e= fsize) {\n+\t\tbv = \u0026bvecq-\u003ebv[slot];\n+\t\tblen = bv-\u003ebv_len;\n+\n+\t\tif (offset \u003e= blen) {\n \t\t\toffset = 0;\n \t\t\tslot++;\n-\t\t\tif (slot \u003e= folioq_nr_slots(folioq)) {\n-\t\t\t\tif (!folioq-\u003enext) {\n-\t\t\t\t\tWARN_ON_ONCE(ret \u003c iter-\u003ecount);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\t\t\t\tfolioq = folioq-\u003enext;\n-\t\t\t\tslot = 0;\n-\t\t\t}\n+\t\t\tcontinue;\n \t\t}\n-\t} while (sg_max \u003e 0 \u0026\u0026 ret \u003c maxsize);\n \n-\titer-\u003efolioq = folioq;\n-\titer-\u003efolioq_slot = slot;\n+\t\tpart = umin(maxsize - ret, blen - offset);\n+\n+\t\tsg_set_page(sg, bv-\u003ebv_page, part, bv-\u003ebv_offset + offset);\n+\t\tsgtable-\u003enents++;\n+\t\tsg++;\n+\t\tsg_max--;\n+\t\toffset += part;\n+\t\tret += part;\n+\t}\n+\n+\titer-\u003ebvecq = bvecq;\n+\titer-\u003ebvecq_slot = slot;\n \titer-\u003eiov_offset = offset;\n \titer-\u003ecount -= ret;\n \treturn ret;\n@@ -1391,8 +1389,8 @@ static ssize_t extract_xarray_to_sg(struct iov_iter *iter,\n * addition of @sg_max elements.\n *\n * The pages referred to by UBUF- and IOVEC-type iterators are extracted and\n- * pinned; BVEC-, KVEC-, FOLIOQ- and XARRAY-type are extracted but aren't\n- * pinned; DISCARD-type is not supported.\n+ * pinned; BVEC-, BVECQ-, KVEC-, XARRAY-type are extracted but aren't pinned;\n+ * DISCARD-type is not supported.\n *\n * No end mark is placed on the scatterlist; that's left to the caller.\n *\n@@ -1424,9 +1422,9 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize,\n \tcase ITER_KVEC:\n \t\treturn extract_kvec_to_sg(iter, maxsize, sgtable, sg_max,\n \t\t\t\t\t extraction_flags);\n-\tcase ITER_FOLIOQ:\n-\t\treturn extract_folioq_to_sg(iter, maxsize, sgtable, sg_max,\n-\t\t\t\t\t extraction_flags);\n+\tcase ITER_BVECQ:\n+\t\treturn extract_bvecq_to_sg(iter, maxsize, sgtable, sg_max,\n+\t\t\t\t\t extraction_flags);\n \tcase ITER_XARRAY:\n \t\treturn extract_xarray_to_sg(iter, maxsize, sgtable, sg_max,\n \t\t\t\t\t extraction_flags);\ndiff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c\nindex d9690ba1db887c..1c708523400684 100644\n--- a/lib/tests/kunit_iov_iter.c\n+++ b/lib/tests/kunit_iov_iter.c\n@@ -12,7 +12,7 @@\n #include \u003clinux/mm.h\u003e\n #include \u003clinux/uio.h\u003e\n #include \u003clinux/bvec.h\u003e\n-#include \u003clinux/folio_queue.h\u003e\n+#include \u003clinux/bvecq.h\u003e\n #include \u003clinux/scatterlist.h\u003e\n #include \u003clinux/minmax.h\u003e\n #include \u003clinux/mman.h\u003e\n@@ -382,58 +382,67 @@ static void __init iov_kunit_copy_from_bvec(struct kunit *test)\n \tKUNIT_SUCCEED(test);\n }\n \n-static void iov_kunit_destroy_folioq(void *data)\n+static void iov_kunit_destroy_bvecq(void *data)\n {\n-\tstruct folio_queue *folioq, *next;\n+\tstruct bvecq *bq, *next;\n \n-\tfor (folioq = data; folioq; folioq = next) {\n-\t\tnext = folioq-\u003enext;\n-\t\tkfree(folioq);\n+\tfor (bq = data; bq; bq = next) {\n+\t\tnext = bq-\u003enext;\n+\t\t/* The pages are freed by vmap with VM_MAP_PUT_PAGES. */\n+\t\tkfree(bq);\n \t}\n }\n \n-static void __init iov_kunit_load_folioq(struct kunit *test,\n+static struct bvecq *iov_kunit_alloc_bvecq(struct kunit *test, unsigned int max_slots)\n+{\n+\tstruct bvecq *bq;\n+\n+\tbq = kzalloc(struct_size(bq, __bv, max_slots), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, bq);\n+\tbq-\u003emax_slots = max_slots;\n+\tbq-\u003ebv = bq-\u003e__bv;\n+\tbq-\u003einline_bv = true;\n+\treturn bq;\n+}\n+\n+static struct bvecq *iov_kunit_create_bvecq(struct kunit *test, unsigned int max_slots)\n+{\n+\tstruct bvecq *bq;\n+\n+\tbq = iov_kunit_alloc_bvecq(test, max_slots);\n+\tkunit_add_action_or_reset(test, iov_kunit_destroy_bvecq, bq);\n+\treturn bq;\n+}\n+\n+static void __init iov_kunit_load_bvecq(struct kunit *test,\n \t\t\t\t\tstruct iov_iter *iter, int dir,\n-\t\t\t\t\tstruct folio_queue *folioq,\n+\t\t\t\t\tstruct bvecq *bq_head,\n \t\t\t\t\tstruct page **pages, size_t npages)\n {\n-\tstruct folio_queue *p = folioq;\n+\tstruct bvecq *bq = bq_head;\n \tsize_t size = 0;\n-\tint i;\n \n-\tfor (i = 0; i \u003c npages; i++) {\n-\t\tif (folioq_full(p)) {\n-\t\t\tp-\u003enext = kzalloc_obj(struct folio_queue);\n-\t\t\tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, p-\u003enext);\n-\t\t\tfolioq_init(p-\u003enext, 0);\n-\t\t\tp-\u003enext-\u003eprev = p;\n-\t\t\tp = p-\u003enext;\n+\tfor (int i = 0; i \u003c npages; i++) {\n+\t\tif (bq-\u003enr_slots \u003e= bq-\u003emax_slots) {\n+\t\t\tbq-\u003enext = iov_kunit_alloc_bvecq(test, 13);\n+\t\t\tbq-\u003enext-\u003eprev = bq;\n+\t\t\tbq = bq-\u003enext;\n \t\t}\n-\t\tfolioq_append(p, page_folio(pages[i]));\n+\t\tbvec_set_page(\u0026bq-\u003ebv[bq-\u003enr_slots], pages[i], PAGE_SIZE, 0);\n+\t\tbq-\u003enr_slots++;\n \t\tsize += PAGE_SIZE;\n \t}\n-\tiov_iter_folio_queue(iter, dir, folioq, 0, 0, size);\n-}\n-\n-static struct folio_queue *iov_kunit_create_folioq(struct kunit *test)\n-{\n-\tstruct folio_queue *folioq;\n-\n-\tfolioq = kzalloc_obj(struct folio_queue);\n-\tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, folioq);\n-\tkunit_add_action_or_reset(test, iov_kunit_destroy_folioq, folioq);\n-\tfolioq_init(folioq, 0);\n-\treturn folioq;\n+\tiov_iter_bvec_queue(iter, dir, bq_head, 0, 0, size);\n }\n \n /*\n- * Test copying to a ITER_FOLIOQ-type iterator.\n+ * Test copying to a ITER_BVECQ-type iterator.\n */\n-static void __init iov_kunit_copy_to_folioq(struct kunit *test)\n+static void __init iov_kunit_copy_to_bvecq(struct kunit *test)\n {\n \tconst struct kvec_test_range *pr;\n \tstruct iov_iter iter;\n-\tstruct folio_queue *folioq;\n+\tstruct bvecq *bq;\n \tstruct page **spages, **bpages;\n \tu8 *scratch, *buffer;\n \tsize_t bufsize, npages, size, copied;\n@@ -442,7 +451,7 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)\n \tbufsize = 0x100000;\n \tnpages = bufsize / PAGE_SIZE;\n \n-\tfolioq = iov_kunit_create_folioq(test);\n+\tbq = iov_kunit_create_bvecq(test, 13);\n \n \tscratch = iov_kunit_create_buffer(test, \u0026spages, npages);\n \tfor (i = 0; i \u003c bufsize; i++)\n@@ -451,20 +460,19 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)\n \tbuffer = iov_kunit_create_buffer(test, \u0026bpages, npages);\n \tmemset(buffer, 0, bufsize);\n \n-\tiov_kunit_load_folioq(test, \u0026iter, READ, folioq, bpages, npages);\n+\tiov_kunit_load_bvecq(test, \u0026iter, READ, bq, bpages, npages);\n \n \ti = 0;\n \tfor (pr = kvec_test_ranges; pr-\u003efrom \u003e= 0; pr++) {\n \t\tsize = pr-\u003eto - pr-\u003efrom;\n \t\tKUNIT_ASSERT_LE(test, pr-\u003eto, bufsize);\n \n-\t\tiov_iter_folio_queue(\u0026iter, READ, folioq, 0, 0, pr-\u003eto);\n+\t\tiov_iter_bvec_queue(\u0026iter, READ, bq, 0, 0, pr-\u003eto);\n \t\tiov_iter_advance(\u0026iter, pr-\u003efrom);\n \t\tcopied = copy_to_iter(scratch + i, size, \u0026iter);\n \n \t\tKUNIT_EXPECT_EQ(test, copied, size);\n \t\tKUNIT_EXPECT_EQ(test, iter.count, 0);\n-\t\tKUNIT_EXPECT_EQ(test, iter.iov_offset, pr-\u003eto % PAGE_SIZE);\n \t\ti += size;\n \t\tif (test-\u003estatus == KUNIT_FAILURE)\n \t\t\tgoto stop;\n@@ -489,13 +497,13 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)\n }\n \n /*\n- * Test copying from a ITER_FOLIOQ-type iterator.\n+ * Test copying from a ITER_BVECQ-type iterator.\n */\n-static void __init iov_kunit_copy_from_folioq(struct kunit *test)\n+static void __init iov_kunit_copy_from_bvecq(struct kunit *test)\n {\n \tconst struct kvec_test_range *pr;\n \tstruct iov_iter iter;\n-\tstruct folio_queue *folioq;\n+\tstruct bvecq *bq;\n \tstruct page **spages, **bpages;\n \tu8 *scratch, *buffer;\n \tsize_t bufsize, npages, size, copied;\n@@ -504,7 +512,7 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)\n \tbufsize = 0x100000;\n \tnpages = bufsize / PAGE_SIZE;\n \n-\tfolioq = iov_kunit_create_folioq(test);\n+\tbq = iov_kunit_create_bvecq(test, 13);\n \n \tbuffer = iov_kunit_create_buffer(test, \u0026bpages, npages);\n \tfor (i = 0; i \u003c bufsize; i++)\n@@ -513,20 +521,19 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)\n \tscratch = iov_kunit_create_buffer(test, \u0026spages, npages);\n \tmemset(scratch, 0, bufsize);\n \n-\tiov_kunit_load_folioq(test, \u0026iter, READ, folioq, bpages, npages);\n+\tiov_kunit_load_bvecq(test, \u0026iter, READ, bq, bpages, npages);\n \n \ti = 0;\n \tfor (pr = kvec_test_ranges; pr-\u003efrom \u003e= 0; pr++) {\n \t\tsize = pr-\u003eto - pr-\u003efrom;\n \t\tKUNIT_ASSERT_LE(test, pr-\u003eto, bufsize);\n \n-\t\tiov_iter_folio_queue(\u0026iter, WRITE, folioq, 0, 0, pr-\u003eto);\n+\t\tiov_iter_bvec_queue(\u0026iter, WRITE, bq, 0, 0, pr-\u003eto);\n \t\tiov_iter_advance(\u0026iter, pr-\u003efrom);\n \t\tcopied = copy_from_iter(scratch + i, size, \u0026iter);\n \n \t\tKUNIT_EXPECT_EQ(test, copied, size);\n \t\tKUNIT_EXPECT_EQ(test, iter.count, 0);\n-\t\tKUNIT_EXPECT_EQ(test, iter.iov_offset, pr-\u003eto % PAGE_SIZE);\n \t\ti += size;\n \t}\n \n@@ -868,13 +875,13 @@ static void __init iov_kunit_extract_pages_bvec(struct kunit *test)\n }\n \n /*\n- * Test the extraction of ITER_FOLIOQ-type iterators.\n+ * Test the extraction of ITER_BVECQ-type iterators.\n */\n-static void __init iov_kunit_extract_pages_folioq(struct kunit *test)\n+static void __init iov_kunit_extract_pages_bvecq(struct kunit *test)\n {\n \tconst struct kvec_test_range *pr;\n-\tstruct folio_queue *folioq;\n \tstruct iov_iter iter;\n+\tstruct bvecq *bq;\n \tstruct page **bpages, *pagelist[8], **pages = pagelist;\n \tssize_t len;\n \tsize_t bufsize, size = 0, npages;\n@@ -883,17 +890,17 @@ static void __init iov_kunit_extract_pages_folioq(struct kunit *test)\n \tbufsize = 0x100000;\n \tnpages = bufsize / PAGE_SIZE;\n \n-\tfolioq = iov_kunit_create_folioq(test);\n+\tbq = iov_kunit_create_bvecq(test, 13);\n \n \tiov_kunit_create_buffer(test, \u0026bpages, npages);\n-\tiov_kunit_load_folioq(test, \u0026iter, READ, folioq, bpages, npages);\n+\tiov_kunit_load_bvecq(test, \u0026iter, READ, bq, bpages, npages);\n \n \tfor (pr = kvec_test_ranges; pr-\u003efrom \u003e= 0; pr++) {\n \t\tfrom = pr-\u003efrom;\n \t\tsize = pr-\u003eto - from;\n \t\tKUNIT_ASSERT_LE(test, pr-\u003eto, bufsize);\n \n-\t\tiov_iter_folio_queue(\u0026iter, WRITE, folioq, 0, 0, pr-\u003eto);\n+\t\tiov_iter_bvec_queue(\u0026iter, WRITE, bq, 0, 0, pr-\u003eto);\n \t\tiov_iter_advance(\u0026iter, from);\n \n \t\tdo {\n@@ -1173,23 +1180,6 @@ static void __init iov_kunit_iter_to_sg_bvec(struct kunit *test)\n \tiov_kunit_iter_to_sg_check(test, \u0026iter, bufsize, \u0026data);\n }\n \n-static void __init iov_kunit_iter_to_sg_folioq(struct kunit *test)\n-{\n-\tstruct iov_kunit_iter_to_sg_data data;\n-\tstruct folio_queue *folioq;\n-\tstruct iov_iter iter;\n-\tsize_t bufsize;\n-\n-\tbufsize = 0x200000;\n-\tiov_kunit_iter_to_sg_init(test, bufsize, false, \u0026data);\n-\n-\tfolioq = iov_kunit_create_folioq(test);\n-\tiov_kunit_load_folioq(test, \u0026iter, READ, folioq, data.pages,\n-\t\t\t data.npages);\n-\n-\tiov_kunit_iter_to_sg_check(test, \u0026iter, bufsize, \u0026data);\n-}\n-\n static void __init iov_kunit_iter_to_sg_xarray(struct kunit *test)\n {\n \tstruct iov_kunit_iter_to_sg_data data;\n@@ -1226,17 +1216,16 @@ static struct kunit_case __refdata iov_kunit_cases[] = {\n \tKUNIT_CASE(iov_kunit_copy_from_kvec),\n \tKUNIT_CASE(iov_kunit_copy_to_bvec),\n \tKUNIT_CASE(iov_kunit_copy_from_bvec),\n-\tKUNIT_CASE(iov_kunit_copy_to_folioq),\n-\tKUNIT_CASE(iov_kunit_copy_from_folioq),\n+\tKUNIT_CASE(iov_kunit_copy_to_bvecq),\n+\tKUNIT_CASE(iov_kunit_copy_from_bvecq),\n \tKUNIT_CASE(iov_kunit_copy_to_xarray),\n \tKUNIT_CASE(iov_kunit_copy_from_xarray),\n \tKUNIT_CASE(iov_kunit_extract_pages_kvec),\n \tKUNIT_CASE(iov_kunit_extract_pages_bvec),\n-\tKUNIT_CASE(iov_kunit_extract_pages_folioq),\n+\tKUNIT_CASE(iov_kunit_extract_pages_bvecq),\n \tKUNIT_CASE(iov_kunit_extract_pages_xarray),\n \tKUNIT_CASE(iov_kunit_iter_to_sg_kvec),\n \tKUNIT_CASE(iov_kunit_iter_to_sg_bvec),\n-\tKUNIT_CASE(iov_kunit_iter_to_sg_folioq),\n \tKUNIT_CASE(iov_kunit_iter_to_sg_xarray),\n \tKUNIT_CASE(iov_kunit_iter_to_sg_ubuf),\n \t{}\ndiff --git a/mm/readahead.c b/mm/readahead.c\nindex 558c92957518b7..069ded56fd8020 100644\n--- a/mm/readahead.c\n+++ b/mm/readahead.c\n@@ -188,6 +188,7 @@ static void read_pages(struct readahead_control *rac)\n \tif (unlikely(rac-\u003e_workingset))\n \t\tpsi_memstall_leave(\u0026rac-\u003e_pflags);\n \trac-\u003e_workingset = false;\n+\trac-\u003e_nr_folios = 0;\n \n \tBUG_ON(readahead_count(rac));\n }\n@@ -303,6 +304,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,\n \t\tif (i == mark)\n \t\t\tfolio_set_readahead(folio);\n \t\tractl-\u003e_workingset |= folio_test_workingset(folio);\n+\t\tractl-\u003e_nr_folios++;\n \t\tractl-\u003e_nr_pages += min_nrpages;\n \t\ti += min_nrpages;\n \t}\n@@ -473,6 +475,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,\n \t\treturn err;\n \t}\n \n+\tractl-\u003e_nr_folios++;\n \tractl-\u003e_nr_pages += 1UL \u003c\u003c order;\n \tractl-\u003e_workingset |= folio_test_workingset(folio);\n \treturn 0;\n@@ -822,6 +825,7 @@ void readahead_expand(struct readahead_control *ractl,\n \t\t\tractl-\u003e_workingset = true;\n \t\t\tpsi_memstall_enter(\u0026ractl-\u003e_pflags);\n \t\t}\n+\t\tractl-\u003e_nr_folios++;\n \t\tractl-\u003e_nr_pages += min_nrpages;\n \t\tractl-\u003e_index = folio-\u003eindex;\n \t}\n@@ -851,6 +855,7 @@ void readahead_expand(struct readahead_control *ractl,\n \t\t\tractl-\u003e_workingset = true;\n \t\t\tpsi_memstall_enter(\u0026ractl-\u003e_pflags);\n \t\t}\n+\t\tractl-\u003e_nr_folios++;\n \t\tractl-\u003e_nr_pages += min_nrpages;\n \t\tif (ra) {\n \t\t\tra-\u003esize += min_nrpages;\ndiff --git a/net/9p/client.c b/net/9p/client.c\nindex ef64546c6d52a1..68d82f677a7a74 100644\n--- a/net/9p/client.c\n+++ b/net/9p/client.c\n@@ -1564,6 +1564,7 @@ void\n p9_client_write_subreq(struct netfs_io_subrequest *subreq)\n {\n \tstruct netfs_io_request *wreq = subreq-\u003erreq;\n+\tstruct iov_iter iter;\n \tstruct p9_fid *fid = wreq-\u003enetfs_priv;\n \tstruct p9_client *clnt = fid-\u003eclnt;\n \tstruct p9_req_t *req;\n@@ -1574,14 +1575,17 @@ p9_client_write_subreq(struct netfs_io_subrequest *subreq)\n \tp9_debug(P9_DEBUG_9P, \"\u003e\u003e\u003e TWRITE fid %d offset %llu len %d\\n\",\n \t\t fid-\u003efid, start, len);\n \n+\tiov_iter_bvec_queue(\u0026iter, ITER_SOURCE, subreq-\u003econtent.bvecq,\n+\t\t\t subreq-\u003econtent.slot, subreq-\u003econtent.offset, subreq-\u003elen);\n+\n \t/* Don't bother zerocopy for small IO (\u003c 1024) */\n \tif (clnt-\u003etrans_mod-\u003ezc_request \u0026\u0026 len \u003e 1024) {\n-\t\treq = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, \u0026subreq-\u003eio_iter,\n+\t\treq = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, \u0026iter,\n \t\t\t\t 0, wreq-\u003elen, P9_ZC_HDR_SZ, \"dqd\",\n \t\t\t\t fid-\u003efid, start, len);\n \t} else {\n \t\treq = p9_client_rpc(clnt, P9_TWRITE, \"dqV\", fid-\u003efid,\n-\t\t\t\t start, len, \u0026subreq-\u003eio_iter);\n+\t\t\t\t start, len, \u0026iter);\n \t}\n \tif (IS_ERR(req)) {\n \t\tnetfs_write_subrequest_terminated(subreq, PTR_ERR(req));\n"
} |
| 2/1 | 2026/07/23 14:19 | agent | patch-evaluator |
1mModel:best-expensive Args: null Results: {
"EnableConfigs": [
"NETFS_SUPPORT",
"FSCACHE",
"CACHEFILES"
],
"FocusSymbols": [
"bvecq_alloc_chain",
"bvecq_expand_buffer",
"bvecq_shorten_buffer",
"bvecq_pos_advance",
"bvecq_zero",
"bvecq_slice",
"bvecq_extract",
"bvecq_load_from_ra",
"netfs_prepare_read_buffer",
"netfs_prepare_write_buffer",
"netfs_extract_iter"
],
"Reasoning": "The patch replaces the `folio_queue` data structure with `bvecq` in the netfs library and updates all dependent filesystems (9p, afs, ceph, nfs, smb) to use it. It also removes the on-demand read mode from cachefiles. This is a major functional change to the core I/O buffering logic in netfs and should be heavily fuzzed.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Do not list too generic configs (we already have them enabled). Only list those that specifically cover the modified code. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit eec393828081ac61b3256a58b75135a47309806f
Author: syz-cluster <triage@syzkaller.com>
Date: Thu Jul 23 14:19:18 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/core-api/folio_queue.rst b/Documentation/core-api/folio_queue.rst
deleted file mode 100644
index b7628896d2b695..00000000000000
--- a/Documentation/core-api/folio_queue.rst
+++ /dev/null
@@ -1,209 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0+
-
-===========
-Folio Queue
-===========
-
-:Author: David Howells <dhowells@redhat.com>
-
-.. Contents:
-
- * Overview
- * Initialisation
- * Adding and removing folios
- * Querying information about a folio
- * Querying information about a folio_queue
- * Folio queue iteration
- * Folio marks
- * Lockless simultaneous production/consumption issues
-
-
-Overview
-========
-
-The folio_queue struct forms a single segment in a segmented list of folios
-that can be used to form an I/O buffer. As such, the list can be iterated over
-using the ITER_FOLIOQ iov_iter type.
-
-The publicly accessible members of the structure are::
-
- struct folio_queue {
- struct folio_queue *next;
- struct folio_queue *prev;
- ...
- };
-
-A pair of pointers are provided, ``next`` and ``prev``, that point to the
-segments on either side of the segment being accessed. Whilst this is a
-doubly-linked list, it is intentionally not a circular list; the outward
-sibling pointers in terminal segments should be NULL.
-
-Each segment in the list also stores:
-
- * an ordered sequence of folio pointers,
- * the size of each folio and
- * three 1-bit marks per folio,
-
-but these should not be accessed directly as the underlying data structure may
-change, but rather the access functions outlined below should be used.
-
-The facility can be made accessible by::
-
- #include <linux/folio_queue.h>
-
-and to use the iterator::
-
- #include <linux/uio.h>
-
-
-Initialisation
-==============
-
-A segment should be initialised by calling::
-
- void folioq_init(struct folio_queue *folioq);
-
-with a pointer to the segment to be initialised. Note that this will not
-necessarily initialise all the folio pointers, so care must be taken to check
-the number of folios added.
-
-
-Adding and removing folios
-==========================
-
-Folios can be set in the next unused slot in a segment struct by calling one
-of::
-
- unsigned int folioq_append(struct folio_queue *folioq,
- struct folio *folio);
-
- unsigned int folioq_append_mark(struct folio_queue *folioq,
- struct folio *folio);
-
-Both functions update the stored folio count, store the folio and note its
-size. The second function also sets the first mark for the folio added. Both
-functions return the number of the slot used. [!] Note that no attempt is made
-to check that the capacity wasn't overrun and the list will not be extended
-automatically.
-
-A folio can be excised by calling::
-
- void folioq_clear(struct folio_queue *folioq, unsigned int slot);
-
-This clears the slot in the array and also clears all the marks for that folio,
-but doesn't change the folio count - so future accesses of that slot must check
-if the slot is occupied.
-
-
-Querying information about a folio
-==================================
-
-Information about the folio in a particular slot may be queried by the
-following function::
-
- struct folio *folioq_folio(const struct folio_queue *folioq,
- unsigned int slot);
-
-If a folio has not yet been set in that slot, this may yield an undefined
-pointer. The size of the folio in a slot may be queried with either of::
-
- unsigned int folioq_folio_order(const struct folio_queue *folioq,
- unsigned int slot);
-
- size_t folioq_folio_size(const struct folio_queue *folioq,
- unsigned int slot);
-
-The first function returns the size as an order and the second as a number of
-bytes.
-
-
-Querying information about a folio_queue
-========================================
-
-Information may be retrieved about a particular segment with the following
-functions::
-
- unsigned int folioq_nr_slots(const struct folio_queue *folioq);
-
- unsigned int folioq_count(struct folio_queue *folioq);
-
- bool folioq_full(struct folio_queue *folioq);
-
-The first function returns the maximum capacity of a segment. It must not be
-assumed that this won't vary between segments. The second returns the number
-of folios added to a segments and the third is a shorthand to indicate if the
-segment has been filled to capacity.
-
-Not that the count and fullness are not affected by clearing folios from the
-segment. These are more about indicating how many slots in the array have been
-initialised, and it assumed that slots won't get reused, but rather the segment
-will get discarded as the queue is consumed.
-
-
-Folio marks
-===========
-
-Folios within a queue can also have marks assigned to them. These marks can be
-used to note information such as if a folio needs folio_put() calling upon it.
-There are three marks available to be set for each folio.
-
-The marks can be set by::
-
- void folioq_mark(struct folio_queue *folioq, unsigned int slot);
- void folioq_mark2(struct folio_queue *folioq, unsigned int slot);
-
-Cleared by::
-
- void folioq_unmark(struct folio_queue *folioq, unsigned int slot);
- void folioq_unmark2(struct folio_queue *folioq, unsigned int slot);
-
-And the marks can be queried by::
-
- bool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot);
- bool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot);
-
-The marks can be used for any purpose and are not interpreted by this API.
-
-
-Folio queue iteration
-=====================
-
-A list of segments may be iterated over using the I/O iterator facility using
-an ``iov_iter`` iterator of ``ITER_FOLIOQ`` type. The iterator may be
-initialised with::
-
- void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq,
- unsigned int first_slot, unsigned int offset,
- size_t count);
-
-This may be told to start at a particular segment, slot and offset within a
-queue. The iov iterator functions will follow the next pointers when advancing
-and prev pointers when reverting when needed.
-
-
-Lockless simultaneous production/consumption issues
-===================================================
-
-If properly managed, the list can be extended by the producer at the head end
-and shortened by the consumer at the tail end simultaneously without the need
-to take locks. The ITER_FOLIOQ iterator inserts appropriate barriers to aid
-with this.
-
-Care must be taken when simultaneously producing and consuming a list. If the
-last segment is reached and the folios it refers to are entirely consumed by
-the IOV iterators, an iov_iter struct will be left pointing to the last segment
-with a slot number equal to the capacity of that segment. The iterator will
-try to continue on from this if there's another segment available when it is
-used again, but care must be taken lest the segment got removed and freed by
-the consumer before the iterator was advanced.
-
-It is recommended that the queue always contain at least one segment, even if
-that segment has never been filled or is entirely spent. This prevents the
-head and tail pointers from collapsing.
-
-
-API Function Reference
-======================
-
-.. kernel-doc:: include/linux/folio_queue.h
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 13769d5c40bf2f..16c529a33ac488 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -39,7 +39,6 @@ Library functionality that is used throughout the kernel.
kref
cleanup
assoc_array
- folio_queue
xarray
maple_tree
idr
diff --git a/Documentation/filesystems/caching/cachefiles.rst b/Documentation/filesystems/caching/cachefiles.rst
index b3ccc782cb3b2b..adfb7d07902710 100644
--- a/Documentation/filesystems/caching/cachefiles.rst
+++ b/Documentation/filesystems/caching/cachefiles.rst
@@ -28,8 +28,6 @@ Cache on Already Mounted Filesystem
(*) Debugging.
- (*) On-demand Read.
-
Overview
========
@@ -483,180 +481,3 @@ the control file. For example::
echo $((1|4|8)) >/sys/module/cachefiles/parameters/debug
will turn on all function entry debugging.
-
-
-On-demand Read
-==============
-
-When working in its original mode, CacheFiles serves as a local cache for a
-remote networking fs - while in on-demand read mode, CacheFiles can boost the
-scenario where on-demand read semantics are needed, e.g. container image
-distribution.
-
-The essential difference between these two modes is seen when a cache miss
-occurs: In the original mode, the netfs will fetch the data from the remote
-server and then write it to the cache file; in on-demand read mode, fetching
-the data and writing it into the cache is delegated to a user daemon.
-
-``CONFIG_CACHEFILES_ONDEMAND`` should be enabled to support on-demand read mode.
-
-
-Protocol Communication
-----------------------
-
-The on-demand read mode uses a simple protocol for communication between kernel
-and user daemon. The protocol can be modeled as::
-
- kernel --[request]--> user daemon --[reply]--> kernel
-
-CacheFiles will send requests to the user daemon when needed. The user daemon
-should poll the devnode ('/dev/cachefiles') to check if there's a pending
-request to be processed. A POLLIN event will be returned when there's a pending
-request.
-
-The user daemon then reads the devnode to fetch a request to process. It should
-be noted that each read only gets one request. When it has finished processing
-the request, the user daemon should write the reply to the devnode.
-
-Each request starts with a message header of the form::
-
- struct cachefiles_msg {
- __u32 msg_id;
- __u32 opcode;
- __u32 len;
- __u32 object_id;
- __u8 data[];
- };
-
-where:
-
- * ``msg_id`` is a unique ID identifying this request among all pending
- requests.
-
- * ``opcode`` indicates the type of this request.
-
- * ``object_id`` is a unique ID identifying the cache file operated on.
-
- * ``data`` indicates the payload of this request.
-
- * ``len`` indicates the whole length of this request, including the
- header and following type-specific payload.
-
-
-Turning on On-demand Mode
--------------------------
-
-An optional parameter becomes available to the "bind" command::
-
- bind [ondemand]
-
-When the "bind" command is given no argument, it defaults to the original mode.
-When it is given the "ondemand" argument, i.e. "bind ondemand", on-demand read
-mode will be enabled.
-
-
-The OPEN Request
-----------------
-
-When the netfs opens a cache file for the first time, a request with the
-CACHEFILES_OP_OPEN opcode, a.k.a an OPEN request will be sent to the user
-daemon. The payload format is of the form::
-
- struct cachefiles_open {
- __u32 volume_key_size;
- __u32 cookie_key_size;
- __u32 fd;
- __u32 flags;
- __u8 data[];
- };
-
-where:
-
- * ``data`` contains the volume_key followed directly by the cookie_key.
- The volume key is a NUL-terminated string; the cookie key is binary
- data.
-
- * ``volume_key_size`` indicates the size of the volume key in bytes.
-
- * ``cookie_key_size`` indicates the size of the cookie key in bytes.
-
- * ``fd`` indicates an anonymous fd referring to the cache file, through
- which the user daemon can perform write/llseek file operations on the
- cache file.
-
-
-The user daemon can use the given (volume_key, cookie_key) pair to distinguish
-the requested cache file. With the given anonymous fd, the user daemon can
-fetch the data and write it to the cache file in the background, even when
-kernel has not triggered a cache miss yet.
-
-Be noted that each cache file has a unique object_id, while it may have multiple
-anonymous fds. The user daemon may duplicate anonymous fds from the initial
-anonymous fd indicated by the @fd field through dup(). Thus each object_id can
-be mapped to multiple anonymous fds, while the usr daemon itself needs to
-maintain the mapping.
-
-When implementing a user daemon, please be careful of RLIMIT_NOFILE,
-``/proc/sys/fs/nr_open`` and ``/proc/sys/fs/file-max``. Typically these needn't
-be huge since they're related to the number of open device blobs rather than
-open files of each individual filesystem.
-
-The user daemon should reply the OPEN request by issuing a "copen" (complete
-open) command on the devnode::
-
- copen <msg_id>,<cache_size>
-
-where:
-
- * ``msg_id`` must match the msg_id field of the OPEN request.
-
- * When >= 0, ``cache_size`` indicates the size of the cache file;
- when < 0, ``cache_size`` indicates any error code encountered by the
- user daemon.
-
-
-The CLOSE Request
------------------
-
-When a cookie withdrawn, a CLOSE request (opcode CACHEFILES_OP_CLOSE) will be
-sent to the user daemon. This tells the user daemon to close all anonymous fds
-associated with the given object_id. The CLOSE request has no extra payload,
-and shouldn't be replied.
-
-
-The READ Request
-----------------
-
-When a cache miss is encountered in on-demand read mode, CacheFiles will send a
-READ request (opcode CACHEFILES_OP_READ) to the user daemon. This tells the user
-daemon to fetch the contents of the requested file range. The payload is of the
-form::
-
- struct cachefiles_read {
- __u64 off;
- __u64 len;
- };
-
-where:
-
- * ``off`` indicates the starting offset of the requested file range.
-
- * ``len`` indicates the length of the requested file range.
-
-
-When it receives a READ request, the user daemon should fetch the requested data
-and write it to the cache file identified by object_id.
-
-When it has finished processing the READ request, the user daemon should reply
-by using the CACHEFILES_IOC_READ_COMPLETE ioctl on one of the anonymous fds
-associated with the object_id given in the READ request. The ioctl is of the
-form::
-
- ioctl(fd, CACHEFILES_IOC_READ_COMPLETE, msg_id);
-
-where:
-
- * ``fd`` is one of the anonymous fds associated with the object_id
- given.
-
- * ``msg_id`` must match the msg_id field of the READ request.
diff --git a/Documentation/filesystems/netfs_library.rst b/Documentation/filesystems/netfs_library.rst
index ddd799df6ce363..18e3c3aae57cda 100644
--- a/Documentation/filesystems/netfs_library.rst
+++ b/Documentation/filesystems/netfs_library.rst
@@ -449,7 +449,7 @@ be called from the writeback code to write the data to the cache, if there is
one.
The inode should be marked ``NETFS_ICTX_SINGLE_NO_UPLOAD`` if this API is to be
-used. The writeback function requires the buffer to be of ITER_FOLIOQ type.
+used.
High-Level VM API
==================
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 1ac0b3dcc0778e..fc5b9c677986e8 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -48,31 +48,70 @@ static void v9fs_begin_writeback(struct netfs_io_request *wreq)
wreq->io_streams[0].avail = true;
}
+/*
+ * Estimate how much data should be accumulated before we start issuing
+ * write subrequests.
+ */
+static int v9fs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ struct p9_fid *fid = wreq->netfs_priv;
+ unsigned long long limit = ULLONG_MAX - stream->issue_from;
+ unsigned long long max_len = fid->clnt->msize - P9_IOHDRSZ;
+
+ estimate->issue_at = stream->issue_from + umin(max_len, limit);
+ return 0;
+}
+
/*
* Issue a subrequest to write to the server.
*/
-static void v9fs_issue_write(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_write(struct netfs_io_subrequest *subreq)
{
+ struct iov_iter iter;
struct p9_fid *fid = subreq->rreq->netfs_priv;
- int err, len;
+ int err, len = 0;
+
+ subreq->len = umin(subreq->len, fid->clnt->msize - P9_IOHDRSZ);
- len = p9_client_write(fid, subreq->start, &subreq->io_iter, &err);
+ err = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ len = p9_client_write(fid, subreq->start, &iter, &err);
if (len > 0)
__set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+
netfs_write_subrequest_terminated(subreq, len ?: err);
+ return 0;
}
/**
* v9fs_issue_read - Issue a read from 9P
* @subreq: The read to make
*/
-static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
+static int v9fs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = rreq->netfs_priv;
char *target;
unsigned long long pos = subreq->start + subreq->transferred;
- int total = 0, err, len, n;
+ size_t len;
+ int total = 0, err;
+
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
if (S_ISLNK(rreq->inode->i_mode)) {
/* p9_client_readlink() must not be called for legacy protocols
@@ -89,12 +128,13 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
err = p9_client_readlink(fid, &target);
if (err != 0)
goto fill_subreq;
- len = strlen(target);
- n = copy_to_iter(target, len, &subreq->io_iter);
+ len = min(strlen(target), subreq->len);
+ total = copy_to_iter(target, len, &iter);
kfree(target);
- total = n;
} else {
- total = p9_client_read(fid, pos, &subreq->io_iter, &err);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
+ total = p9_client_read(fid, pos, &iter, &err);
}
fill_subreq:
@@ -112,6 +152,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
subreq->error = err;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/**
@@ -124,7 +165,6 @@ static int v9fs_init_request(struct netfs_io_request *rreq, struct file *file)
struct p9_fid *fid;
struct dentry *dentry;
bool writing = (rreq->origin == NETFS_READ_FOR_WRITE ||
- rreq->origin == NETFS_WRITETHROUGH ||
rreq->origin == NETFS_UNBUFFERED_WRITE ||
rreq->origin == NETFS_DIO_WRITE);
@@ -186,6 +226,7 @@ const struct netfs_request_ops v9fs_req_ops = {
.free_request = v9fs_free_request,
.issue_read = v9fs_issue_read,
.begin_writeback = v9fs_begin_writeback,
+ .estimate_write = v9fs_estimate_write,
.issue_write = v9fs_issue_write,
};
diff --git a/fs/afs/dir.c b/fs/afs/dir.c
index 6df56fe9163fd7..174114c3963955 100644
--- a/fs/afs/dir.c
+++ b/fs/afs/dir.c
@@ -140,9 +140,9 @@ static void afs_dir_dump(struct afs_vnode *dvnode)
pr_warn("DIR %llx:%llx is=%llx\n",
dvnode->fid.vid, dvnode->fid.vnode, i_size);
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
- iterate_folioq(&iter, iov_iter_count(&iter), NULL, NULL,
- afs_dir_dump_step);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ iterate_bvecq(&iter, iov_iter_count(&iter), NULL, NULL,
+ afs_dir_dump_step);
}
/*
@@ -203,9 +203,9 @@ static int afs_dir_check(struct afs_vnode *dvnode)
if (unlikely(!i_size))
return 0;
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
- checked = iterate_folioq(&iter, iov_iter_count(&iter), dvnode, NULL,
- afs_dir_check_step);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ checked = iterate_bvecq(&iter, iov_iter_count(&iter), dvnode, NULL,
+ afs_dir_check_step);
if (checked != i_size) {
afs_dir_dump(dvnode);
return -EIO;
@@ -250,15 +250,14 @@ static ssize_t afs_do_read_single(struct afs_vnode *dvnode, struct file *file)
if (dvnode->directory_size < i_size) {
size_t cur_size = dvnode->directory_size;
- ret = netfs_alloc_folioq_buffer(NULL,
- &dvnode->directory, &cur_size, i_size,
- mapping_gfp_mask(dvnode->netfs.inode.i_mapping));
+ ret = bvecq_expand_buffer(&dvnode->directory, &cur_size,
+ round_up(i_size, PAGE_SIZE), GFP_KERNEL);
dvnode->directory_size = cur_size;
if (ret < 0)
return ret;
}
- iov_iter_folio_queue(&iter, ITER_DEST, dvnode->directory, 0, 0, dvnode->directory_size);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dvnode->directory, 0, 0, dvnode->directory_size);
/* AFS requires us to perform the read of a directory synchronously as
* a single unit to avoid issues with the directory contents being
@@ -294,8 +293,8 @@ static ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file)
}
/*
- * Read the directory into a folio_queue buffer in one go, scrubbing the
- * previous contents. We return -ESTALE if the caller needs to call us again.
+ * Read the directory into the buffer in one go, scrubbing the previous
+ * contents. We return -ESTALE if the caller needs to call us again.
*/
ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)
__acquires(&dvnode->validate_lock)
@@ -483,7 +482,7 @@ static size_t afs_dir_iterate_step(void *iter_base, size_t progress, size_t len,
}
/*
- * Iterate through the directory folios.
+ * Iterate through the directory content.
*/
static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_ctx)
{
@@ -498,11 +497,11 @@ static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *dir_c
if (i_size <= 0 || dir_ctx->pos >= i_size)
return 0;
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size);
iov_iter_advance(&iter, round_down(dir_ctx->pos, AFS_DIR_BLOCK_SIZE));
- iterate_folioq(&iter, iov_iter_count(&iter), dvnode, &ctx,
- afs_dir_iterate_step);
+ iterate_bvecq(&iter, iov_iter_count(&iter), dvnode, &ctx,
+ afs_dir_iterate_step);
if (ctx.error == -ESTALE)
afs_invalidate_dir(dvnode, afs_dir_invalid_iter_stale);
@@ -2228,11 +2227,10 @@ static int afs_dir_writepages(struct address_space *mapping,
}
if (test_bit(AFS_VNODE_DIR_VALID, &dvnode->flags)) {
- iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
- i_size_read(&dvnode->netfs.inode));
- ret = netfs_writeback_single(mapping, wbc, &iter);
- if (ret == 1)
- ret = 0; /* Skipped write due to lock conflict. */
+ size_t len = i_size_read(&dvnode->netfs.inode);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
+ round_up(len, PAGE_SIZE));
+ ret = netfs_writeback_single(mapping, wbc, &iter, len);
}
up_read(&dvnode->validate_lock);
diff --git a/fs/afs/dir_edit.c b/fs/afs/dir_edit.c
index fd3aa9f97ce64c..5f00c41403e5a5 100644
--- a/fs/afs/dir_edit.c
+++ b/fs/afs/dir_edit.c
@@ -10,7 +10,6 @@
#include <linux/namei.h>
#include <linux/pagemap.h>
#include <linux/iversion.h>
-#include <linux/folio_queue.h>
#include "internal.h"
#include "xdr_fs.h"
@@ -110,9 +109,8 @@ static void afs_clear_contig_bits(union afs_xdr_dir_block *block,
*/
static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, size_t block)
{
- struct folio_queue *fq;
struct afs_vnode *dvnode = iter->dvnode;
- struct folio *folio;
+ struct bvecq *bq;
size_t blpos = block * AFS_DIR_BLOCK_SIZE;
size_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter->fpos;
int ret;
@@ -120,41 +118,38 @@ static union afs_xdr_dir_block *afs_dir_get_block(struct afs_dir_iter *iter, siz
if (dvnode->directory_size < blend) {
size_t cur_size = dvnode->directory_size;
- ret = netfs_alloc_folioq_buffer(
- NULL, &dvnode->directory, &cur_size, blend,
- mapping_gfp_mask(dvnode->netfs.inode.i_mapping));
+ ret = bvecq_expand_buffer(&dvnode->directory, &cur_size, blend,
+ GFP_KERNEL);
dvnode->directory_size = cur_size;
if (ret < 0)
goto fail;
}
- fq = iter->fq;
- if (!fq)
- fq = dvnode->directory;
+ bq = iter->bq;
+ if (!bq)
+ bq = dvnode->directory;
- /* Search the folio queue for the folio containing the block... */
- for (; fq; fq = fq->next) {
- for (int s = iter->fq_slot; s < folioq_count(fq); s++) {
- size_t fsize = folioq_folio_size(fq, s);
+ /* Search the contents for the region containing the block... */
+ for (; bq; bq = bq->next) {
+ for (int s = iter->bq_slot; s < bq->nr_slots; s++) {
+ struct bio_vec *bv = &bq->bv[s];
+ size_t bsize = bv->bv_len;
- if (blend <= fpos + fsize) {
+ if (blend <= fpos + bsize) {
/* ... and then return the mapped block. */
- folio = folioq_folio(fq, s);
- if (WARN_ON_ONCE(folio_pos(folio) != fpos))
- goto fail;
- iter->fq = fq;
- iter->fq_slot = s;
+ iter->bq = bq;
+ iter->bq_slot = s;
iter->fpos = fpos;
- return kmap_local_folio(folio, blpos - fpos);
+ return bvec_kmap_partial(bv, blpos - fpos);
}
- fpos += fsize;
+ fpos += bsize;
}
- iter->fq_slot = 0;
+ iter->bq_slot = 0;
}
fail:
- iter->fq = NULL;
- iter->fq_slot = 0;
+ iter->bq = NULL;
+ iter->bq_slot = 0;
afs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);
return NULL;
}
@@ -415,7 +410,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
if (!afs_dir_init_iter(&iter, name))
return;
- meta = afs_dir_find_block(&iter, 0);
+ meta = afs_dir_get_block(&iter, 0);
if (!meta)
return;
@@ -442,7 +437,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
/* Check and clear the entry. */
de = &block->dirents[slot];
if (de->u.valid != 1)
- goto error_unmap;
+ goto error;
trace_afs_edit_dir(vnode, why, afs_edit_dir_delete, b, slot,
ntohl(de->u.vnode), ntohl(de->u.unique),
@@ -458,7 +453,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
/* Clear the constituent entries. */
next = de->u.hash_next;
memset(de, 0, sizeof(*de) * iter.nr_slots);
- kunmap_local(block);
/* Adjust the hash chain: if iter->prev_entry is 0, the hashtable head
* index is previous; otherwise it's slot number of the previous entry.
@@ -485,7 +479,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
pde = &pblock->dirents[ps];
prev_next = pde->u.hash_next;
if (prev_next != htons(entry)) {
- kunmap_local(pblock);
pr_warn("%llx:%llx:%x: not prev in chain b=%x p=%x,%x e=%x %*s",
vnode->fid.vid, vnode->fid.vnode, vnode->fid.unique,
iter.bucket, iter.prev_entry, prev_next, entry,
@@ -493,7 +486,6 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
goto error;
}
pde->u.hash_next = next;
- kunmap_local(pblock);
}
netfs_single_mark_inode_dirty(&vnode->netfs.inode);
@@ -503,18 +495,16 @@ void afs_edit_dir_remove(struct afs_vnode *vnode,
_debug("Remove %s from %u[%u]", name->name, b, slot);
out_unmap:
+ afs_dir_end_iter(&iter);
kunmap_local(meta);
_leave("");
return;
already_invalidated:
- kunmap_local(block);
trace_afs_edit_dir(vnode, why, afs_edit_dir_delete_inval,
0, 0, 0, 0, name->name);
goto out_unmap;
-error_unmap:
- kunmap_local(block);
error:
trace_afs_edit_dir(vnode, why, afs_edit_dir_delete_error,
0, 0, 0, 0, name->name);
diff --git a/fs/afs/dir_search.c b/fs/afs/dir_search.c
index 104411c0692f57..e1f91b3ac6e25e 100644
--- a/fs/afs/dir_search.c
+++ b/fs/afs/dir_search.c
@@ -66,53 +66,47 @@ bool afs_dir_init_iter(struct afs_dir_iter *iter, const struct qstr *name)
*/
union afs_xdr_dir_block *afs_dir_find_block(struct afs_dir_iter *iter, size_t block)
{
- struct folio_queue *fq = iter->fq;
struct afs_vnode *dvnode = iter->dvnode;
- struct folio *folio;
+ struct bvecq *bq = iter->bq;
size_t blpos = block * AFS_DIR_BLOCK_SIZE;
size_t blend = (block + 1) * AFS_DIR_BLOCK_SIZE, fpos = iter->fpos;
- int slot = iter->fq_slot;
+ int slot = iter->bq_slot;
_enter("%zx,%d", block, slot);
- if (iter->block) {
- kunmap_local(iter->block);
- iter->block = NULL;
- }
+ afs_dir_end_iter(iter);
if (dvnode->directory_size < blend)
goto fail;
- if (!fq || blpos < fpos) {
- fq = dvnode->directory;
+ if (!bq || blpos < fpos) {
+ bq = dvnode->directory;
slot = 0;
fpos = 0;
}
/* Search the folio queue for the folio containing the block... */
- for (; fq; fq = fq->next) {
- for (; slot < folioq_count(fq); slot++) {
- size_t fsize = folioq_folio_size(fq, slot);
+ for (; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ struct bio_vec *bv = &bq->bv[slot];
+ size_t bsize = bv->bv_len;
- if (blend <= fpos + fsize) {
+ if (blend <= fpos + bsize) {
/* ... and then return the mapped block. */
- folio = folioq_folio(fq, slot);
- if (WARN_ON_ONCE(folio_pos(folio) != fpos))
- goto fail;
- iter->fq = fq;
- iter->fq_slot = slot;
+ iter->bq = bq;
+ iter->bq_slot = slot;
iter->fpos = fpos;
- iter->block = kmap_local_folio(folio, blpos - fpos);
+ iter->block = bvec_kmap_partial(bv, blpos - fpos);
return iter->block;
}
- fpos += fsize;
+ fpos += bsize;
}
slot = 0;
}
fail:
- iter->fq = NULL;
- iter->fq_slot = 0;
+ iter->bq = NULL;
+ iter->bq_slot = 0;
afs_invalidate_dir(dvnode, afs_dir_invalid_edit_get_block);
return NULL;
}
@@ -173,12 +167,8 @@ int afs_dir_search_bucket(struct afs_dir_iter *iter, const struct qstr *name,
ret = -ENOENT;
found:
- if (iter->block) {
- kunmap_local(iter->block);
- iter->block = NULL;
- }
-
bad:
+ afs_dir_end_iter(iter);
if (ret == -ESTALE)
afs_invalidate_dir(iter->dvnode, afs_dir_invalid_iter_stale);
_leave(" = %d", ret);
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 0467742bfeee34..476b9f3a776ebf 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -332,11 +332,12 @@ void afs_fetch_data_immediate_cancel(struct afs_call *call)
/*
* Fetch file data from the volume.
*/
-static void afs_issue_read(struct netfs_io_subrequest *subreq)
+static int afs_issue_read(struct netfs_io_subrequest *subreq)
{
struct afs_operation *op;
struct afs_vnode *vnode = AFS_FS_I(subreq->rreq->inode);
struct key *key = subreq->rreq->netfs_priv;
+ int ret;
_enter("%s{%llx:%llu.%u},%x,,,",
vnode->volume->name,
@@ -345,11 +346,15 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
vnode->fid.unique,
key_serial(key));
+ ret = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
op = afs_alloc_operation(key, vnode->volume);
if (IS_ERR(op)) {
- subreq->error = PTR_ERR(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = PTR_ERR(op);
+ goto failed;
}
afs_op_set_vnode(op, 0, vnode);
@@ -364,20 +369,24 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq)
op->flags |= AFS_OPERATION_ASYNC;
if (!afs_begin_vnode_operation(op)) {
- subreq->error = afs_put_operation(op);
- netfs_read_subreq_terminated(subreq);
- return;
+ ret = afs_put_operation(op);
+ goto failed;
}
if (!afs_select_fileserver(op)) {
- afs_end_read(op);
- return;
+ afs_end_read(op); /* Error recorded here. */
+ return 0;
}
afs_issue_read_call(op);
} else {
afs_do_sync_operation(op);
}
+ return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
@@ -400,7 +409,6 @@ static int afs_init_request(struct netfs_io_request *rreq, struct file *file)
}
break;
case NETFS_WRITEBACK:
- case NETFS_WRITETHROUGH:
case NETFS_UNBUFFERED_WRITE:
case NETFS_DIO_WRITE:
if (S_ISREG(rreq->inode->i_mode))
@@ -448,7 +456,6 @@ void afs_set_i_size(struct afs_vnode *vnode, loff_t new_i_size)
}
spin_unlock(&inode->i_lock);
write_sequnlock(&vnode->cb_lock);
- fscache_update_cookie(afs_vnode_cache(vnode), NULL, &new_i_size);
}
static void afs_update_i_size(struct inode *inode, loff_t new_i_size)
@@ -471,7 +478,7 @@ const struct netfs_request_ops afs_req_ops = {
.update_i_size = afs_update_i_size,
.invalidate_cache = afs_netfs_invalidate_cache,
.begin_writeback = afs_begin_writeback,
- .prepare_write = afs_prepare_write,
+ .estimate_write = afs_estimate_write,
.issue_write = afs_issue_write,
.retry_request = afs_retry_request,
};
diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c
index a2ffd60889f89d..c332b733d7a733 100644
--- a/fs/afs/fsclient.c
+++ b/fs/afs/fsclient.c
@@ -339,7 +339,9 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = umin(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1085,7 +1087,7 @@ static void afs_fs_store_data64(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
@@ -1139,7 +1141,7 @@ void afs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index 14f39a9bea6cfe..634fbf8eb21206 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c
@@ -683,7 +683,7 @@ void afs_evict_inode(struct inode *inode)
flush_delayed_work(&vnode->lock_work);
netfs_wait_for_outstanding_io(inode);
truncate_inode_pages_final(&inode->i_data);
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
if (vnode->symlink)
afs_evict_symlink(vnode);
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index 601f01e5c15fcc..0d4e61504ef755 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -710,7 +710,7 @@ struct afs_vnode {
#define AFS_VNODE_MODIFYING 10 /* Set if we're performing a modification op */
#define AFS_VNODE_DIR_READ 11 /* Set if we've read a dir's contents */
- struct folio_queue *directory; /* Directory contents */
+ struct bvecq *directory; /* Directory contents */
struct afs_symlink __rcu *symlink; /* Symlink content */
struct list_head wb_keys; /* List of keys available for writeback */
struct list_head pending_locks; /* locks waiting to be granted */
@@ -914,7 +914,7 @@ struct afs_operation {
afs_lock_type_t type;
} lock;
struct {
- struct iov_iter *write_iter;
+ struct iov_iter write_iter;
loff_t pos;
loff_t size;
loff_t i_size;
@@ -991,9 +991,9 @@ static inline void afs_invalidate_cache(struct afs_vnode *vnode, unsigned int fl
struct afs_dir_iter {
struct afs_vnode *dvnode;
union afs_xdr_dir_block *block;
- struct folio_queue *fq;
+ struct bvecq *bq;
unsigned int fpos;
- int fq_slot;
+ int bq_slot;
unsigned int loop_check;
u8 nr_slots;
u8 bucket;
@@ -1133,6 +1133,14 @@ int afs_dir_search_bucket(struct afs_dir_iter *iter, const struct qstr *name,
int afs_dir_search(struct afs_vnode *dvnode, const struct qstr *name,
struct afs_fid *_fid, afs_dataversion_t *_dir_version);
+static inline void afs_dir_end_iter(struct afs_dir_iter *iter)
+{
+ if (iter->block) {
+ kunmap_local(iter->block);
+ iter->block = NULL;
+ }
+}
+
/*
* dir_silly.c
*/
@@ -1697,8 +1705,10 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *);
/*
* write.c
*/
-void afs_prepare_write(struct netfs_io_subrequest *subreq);
-void afs_issue_write(struct netfs_io_subrequest *subreq);
+int afs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
+int afs_issue_write(struct netfs_io_subrequest *subreq);
void afs_begin_writeback(struct netfs_io_request *wreq);
void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
extern int afs_writepages(struct address_space *, struct writeback_control *);
diff --git a/fs/afs/symlink.c b/fs/afs/symlink.c
index 16b4823cb7b7e7..ddd3307adccf3f 100644
--- a/fs/afs/symlink.c
+++ b/fs/afs/symlink.c
@@ -56,7 +56,6 @@ void afs_evict_symlink(struct afs_vnode *vnode)
void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
{
struct afs_symlink *symlink = op->create.symlink;
- size_t dsize = 0;
size_t size = strlen(symlink->content) + 1;
char *p;
@@ -66,13 +65,15 @@ void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs)))
return;
- if (netfs_alloc_folioq_buffer(NULL, &vnode->directory, &dsize, size,
- mapping_gfp_mask(vnode->netfs.inode.i_mapping)) < 0)
+ vnode->directory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL);
+ if (!vnode->directory)
return;
- vnode->directory_size = dsize;
- p = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+ vnode->directory_size = size;
+ p = bvec_kmap_partial(&vnode->directory->bv[0], 0);
memcpy(p, symlink->content, size);
+ if (size < PAGE_SIZE)
+ memset(p + size, 0, PAGE_SIZE - size);
kunmap_local(p);
netfs_single_mark_inode_dirty(&vnode->netfs.inode);
}
@@ -94,17 +95,12 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
}
if (!vnode->directory) {
- size_t cur_size = 0;
-
- ret = netfs_alloc_folioq_buffer(NULL,
- &vnode->directory, &cur_size, PAGE_SIZE,
- mapping_gfp_mask(vnode->netfs.inode.i_mapping));
- vnode->directory_size = PAGE_SIZE - 1;
- if (ret < 0)
- return ret;
+ vnode->directory = bvecq_alloc_buffer(PAGE_SIZE, GFP_KERNEL);
+ if (!vnode->directory)
+ return -ENOMEM;
}
- iov_iter_folio_queue(&iter, ITER_DEST, vnode->directory, 0, 0, PAGE_SIZE);
+ iov_iter_bvec_queue(&iter, ITER_DEST, vnode->directory, 0, 0, PAGE_SIZE);
/* AFS requires us to perform the read of a symlink as a single unit to
* avoid issues with the content being changed between reads.
@@ -127,7 +123,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
refcount_set(&symlink->ref, 1);
symlink->content[i_size] = 0;
- const char *s = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+ const char *s = bvec_kmap_partial(&vnode->directory->bv[0], 0);
memcpy(symlink->content, s, i_size);
kunmap_local(s);
@@ -136,7 +132,7 @@ static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
}
if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs))) {
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
vnode->directory = NULL;
vnode->directory_size = 0;
}
@@ -249,14 +245,15 @@ int afs_symlink_writepages(struct address_space *mapping,
if (vnode->directory &&
atomic64_read(&vnode->cb_expires_at) != AFS_NO_CB_PROMISE) {
- iov_iter_folio_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
- i_size_read(&vnode->netfs.inode));
- ret = netfs_writeback_single(mapping, wbc, &iter);
+ size_t len = i_size_read(&vnode->netfs.inode);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
+ round_up(len, PAGE_SIZE));
+ ret = netfs_writeback_single(mapping, wbc, &iter, len);
}
if (ret == 0) {
netfs_wb_begin(&vnode->netfs, false);
- netfs_free_folioq_buffer(vnode->directory);
+ bvecq_put(vnode->directory);
vnode->directory = NULL;
vnode->directory_size = 0;
netfs_wb_end(&vnode->netfs);
diff --git a/fs/afs/write.c b/fs/afs/write.c
index 7f34b939706a07..46f81d9e7d7ff1 100644
--- a/fs/afs/write.c
+++ b/fs/afs/write.c
@@ -83,17 +83,20 @@ static const struct afs_operation_ops afs_store_data_operation = {
};
/*
- * Prepare a subrequest to write to the server. This sets the max_len
- * parameter.
+ * Estimate the maximum size of a write we can send to the server.
*/
-void afs_prepare_write(struct netfs_io_subrequest *subreq)
+int afs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
{
- struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
+ unsigned long long limit = ULLONG_MAX - stream->issue_from;
+ unsigned long long max_len = 256 * 1024 * 1024;
//if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags))
- // subreq->max_len = 512 * 1024;
- //else
- stream->sreq_max_len = 256 * 1024 * 1024;
+ // max_len = 512 * 1024;
+
+ estimate->issue_at = stream->issue_from + umin(max_len, limit);
+ return 0;
}
/*
@@ -139,12 +142,15 @@ static void afs_issue_write_worker(struct work_struct *work)
op->flags |= AFS_OPERATION_UNINTR;
op->ops = &afs_store_data_operation;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
afs_begin_vnode_operation(op);
- op->store.write_iter = &subreq->io_iter;
op->store.i_size = umax(pos + len, netfs_read_remote_i_size(&vnode->netfs.inode));
op->mtime = inode_get_mtime(&vnode->netfs.inode);
+ iov_iter_bvec_queue(&op->store.write_iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
afs_wait_for_operation(op);
ret = afs_put_operation(op);
switch (ret) {
@@ -168,11 +174,21 @@ static void afs_issue_write_worker(struct work_struct *work)
netfs_write_subrequest_terminated(subreq, ret < 0 ? ret : subreq->len);
}
-void afs_issue_write(struct netfs_io_subrequest *subreq)
+int afs_issue_write(struct netfs_io_subrequest *subreq)
{
+ int ret;
+
+ if (subreq->len > 256 * 1024 * 1024)
+ subreq->len = 256 * 1024 * 1024;
+ ret = netfs_prepare_write_buffer(subreq, INT_MAX);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
subreq->work.func = afs_issue_write_worker;
if (!queue_work(system_dfl_wq, &subreq->work))
WARN_ON_ONCE(1);
+ return 0;
}
/*
@@ -183,6 +199,8 @@ void afs_begin_writeback(struct netfs_io_request *wreq)
{
if (S_ISREG(wreq->inode->i_mode))
afs_get_writeback_key(wreq);
+
+ wreq->io_streams[0].avail = true;
}
/*
diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c
index d941179730a989..52c588092050de 100644
--- a/fs/afs/yfsclient.c
+++ b/fs/afs/yfsclient.c
@@ -385,7 +385,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)
if (call->remaining == 0)
goto no_more_data;
- call->iter = &subreq->io_iter;
+ iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
call->iov_len = min(call->remaining, subreq->len - subreq->transferred);
call->unmarshall++;
fallthrough;
@@ -1357,7 +1359,7 @@ void yfs_fs_store_data(struct afs_operation *op)
if (!call)
return afs_op_nomem(op);
- call->write_iter = op->store.write_iter;
+ call->write_iter = &op->store.write_iter;
/* marshall the parameters */
bp = call->request;
diff --git a/fs/cachefiles/Kconfig b/fs/cachefiles/Kconfig
index c5a070550ee334..afb25b6af5aa17 100644
--- a/fs/cachefiles/Kconfig
+++ b/fs/cachefiles/Kconfig
@@ -26,15 +26,3 @@ config CACHEFILES_ERROR_INJECTION
help
This permits error injection to be enabled in cachefiles whilst a
cache is in service.
-
-config CACHEFILES_ONDEMAND
- bool "Support for on-demand read"
- depends on CACHEFILES
- default n
- help
- This permits userspace to enable the cachefiles on-demand read mode.
- In this mode, when a cache miss occurs, responsibility for fetching
- the data lies with the cachefiles backend instead of with the netfs
- and is delegated to userspace.
-
- If unsure, say N.
diff --git a/fs/cachefiles/Makefile b/fs/cachefiles/Makefile
index c37a7a9af10b60..16d811f1a2faed 100644
--- a/fs/cachefiles/Makefile
+++ b/fs/cachefiles/Makefile
@@ -16,6 +16,5 @@ cachefiles-y := \
xattr.o
cachefiles-$(CONFIG_CACHEFILES_ERROR_INJECTION) += error_inject.o
-cachefiles-$(CONFIG_CACHEFILES_ONDEMAND) += ondemand.o
obj-$(CONFIG_CACHEFILES) := cachefiles.o
diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c
index 4117b145ac9431..6e0248a62c1397 100644
--- a/fs/cachefiles/daemon.c
+++ b/fs/cachefiles/daemon.c
@@ -76,10 +76,6 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = {
{ "inuse", cachefiles_daemon_inuse },
{ "secctx", cachefiles_daemon_secctx },
{ "tag", cachefiles_daemon_tag },
-#ifdef CONFIG_CACHEFILES_ONDEMAND
- { "copen", cachefiles_ondemand_copen },
- { "restore", cachefiles_ondemand_restore },
-#endif
{ "", NULL }
};
@@ -114,8 +110,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)
INIT_LIST_HEAD(&cache->object_list);
spin_lock_init(&cache->object_list_lock);
refcount_set(&cache->unbind_pincount, 1);
- xa_init_flags(&cache->reqs, XA_FLAGS_ALLOC);
- xa_init_flags(&cache->ondemand_ids, XA_FLAGS_ALLOC1);
/* set default caching limits
* - limit at 1% free space and/or free files
@@ -134,40 +128,6 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file)
return 0;
}
-void cachefiles_flush_reqs(struct cachefiles_cache *cache)
-{
- struct xarray *xa = &cache->reqs;
- struct cachefiles_req *req;
- unsigned long index;
-
- /*
- * Make sure the following two operations won't be reordered.
- * 1) set CACHEFILES_DEAD bit
- * 2) flush requests in the xarray
- * Otherwise the request may be enqueued after xarray has been
- * flushed, leaving the orphan request never being completed.
- *
- * CPU 1 CPU 2
- * ===== =====
- * flush requests in the xarray
- * test CACHEFILES_DEAD bit
- * enqueue the request
- * set CACHEFILES_DEAD bit
- */
- smp_mb();
-
- xa_lock(xa);
- xa_for_each(xa, index, req) {
- req->error = -EIO;
- complete(&req->done);
- __xa_erase(xa, index);
- }
- xa_unlock(xa);
-
- xa_destroy(&cache->reqs);
- xa_destroy(&cache->ondemand_ids);
-}
-
void cachefiles_put_unbind_pincount(struct cachefiles_cache *cache)
{
if (refcount_dec_and_test(&cache->unbind_pincount)) {
@@ -195,9 +155,6 @@ static int cachefiles_daemon_release(struct inode *inode, struct file *file)
set_bit(CACHEFILES_DEAD, &cache->flags);
- if (cachefiles_in_ondemand_mode(cache))
- cachefiles_flush_reqs(cache);
-
/* clean up the control file interface */
cache->cachefilesd = NULL;
file->private_data = NULL;
@@ -266,10 +223,7 @@ static ssize_t cachefiles_daemon_read(struct file *file, char __user *_buffer,
if (!test_bit(CACHEFILES_READY, &cache->flags))
return 0;
- if (cachefiles_in_ondemand_mode(cache))
- return cachefiles_ondemand_daemon_read(cache, _buffer, buflen);
- else
- return cachefiles_do_daemon_read(cache, _buffer, buflen);
+ return cachefiles_do_daemon_read(cache, _buffer, buflen);
}
/*
@@ -358,28 +312,13 @@ static __poll_t cachefiles_daemon_poll(struct file *file,
struct poll_table_struct *poll)
{
struct cachefiles_cache *cache = file->private_data;
- XA_STATE(xas, &cache->reqs, 0);
- struct cachefiles_req *req;
__poll_t mask;
poll_wait(file, &cache->daemon_pollwq, poll);
mask = 0;
- if (cachefiles_in_ondemand_mode(cache)) {
- if (!xa_empty(&cache->reqs)) {
- xas_lock(&xas);
- xas_for_each_marked(&xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {
- if (!cachefiles_ondemand_is_reopening_read(req)) {
- mask |= EPOLLIN;
- break;
- }
- }
- xas_unlock(&xas);
- }
- } else {
- if (test_bit(CACHEFILES_STATE_CHANGED, &cache->flags))
- mask |= EPOLLIN;
- }
+ if (test_bit(CACHEFILES_STATE_CHANGED, &cache->flags))
+ mask |= EPOLLIN;
if (test_bit(CACHEFILES_CULLING, &cache->flags))
mask |= EPOLLOUT;
@@ -779,14 +718,7 @@ static int cachefiles_daemon_bind(struct cachefiles_cache *cache, char *args)
return -EBUSY;
}
- if (IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND)) {
- if (!strcmp(args, "ondemand")) {
- set_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags);
- } else if (*args) {
- pr_err("Invalid argument to the 'bind' command\n");
- return -EINVAL;
- }
- } else if (*args) {
+ if (*args) {
pr_err("'bind' command doesn't take an argument\n");
return -EINVAL;
}
diff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c
index a08250d244ea25..638e00717119c8 100644
--- a/fs/cachefiles/interface.c
+++ b/fs/cachefiles/interface.c
@@ -32,11 +32,6 @@ struct cachefiles_object *cachefiles_alloc_object(struct fscache_cookie *cookie)
if (!object)
return NULL;
- if (cachefiles_ondemand_init_obj_info(object, volume)) {
- kmem_cache_free(cachefiles_object_jar, object);
- return NULL;
- }
-
refcount_set(&object->ref, 1);
spin_lock_init(&object->lock);
@@ -94,7 +89,6 @@ void cachefiles_put_object(struct cachefiles_object *object,
ASSERTCMP(object->file, ==, NULL);
kfree(object->d_name);
- cachefiles_ondemand_deinit_obj_info(object);
cache = object->volume->cache->cache;
fscache_put_cookie(object->cookie, fscache_cookie_put_object);
object->cookie = NULL;
@@ -105,73 +99,6 @@ void cachefiles_put_object(struct cachefiles_object *object,
_leave("");
}
-/*
- * Adjust the size of a cache file if necessary to match the DIO size. We keep
- * the EOF marker a multiple of DIO blocks so that we don't fall back to doing
- * non-DIO for a partial block straddling the EOF, but we also have to be
- * careful of someone expanding the file and accidentally accreting the
- * padding.
- */
-static int cachefiles_adjust_size(struct cachefiles_object *object)
-{
- struct iattr newattrs;
- struct file *file = object->file;
- uint64_t ni_size;
- loff_t oi_size;
- int ret;
-
- ni_size = object->cookie->object_size;
- ni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);
-
- _enter("{OBJ%x},[%llu]",
- object->debug_id, (unsigned long long) ni_size);
-
- if (!file)
- return -ENOBUFS;
-
- oi_size = i_size_read(file_inode(file));
- if (oi_size == ni_size)
- return 0;
-
- inode_lock(file_inode(file));
-
- /* if there's an extension to a partial page at the end of the backing
- * file, we need to discard the partial page so that we pick up new
- * data after it */
- if (oi_size & ~PAGE_MASK && ni_size > oi_size) {
- _debug("discard tail %llx", oi_size);
- newattrs.ia_valid = ATTR_SIZE;
- newattrs.ia_size = oi_size & PAGE_MASK;
- ret = cachefiles_inject_remove_error();
- if (ret == 0)
- ret = notify_change(&nop_mnt_idmap, file->f_path.dentry,
- &newattrs, NULL);
- if (ret < 0)
- goto truncate_failed;
- }
-
- newattrs.ia_valid = ATTR_SIZE;
- newattrs.ia_size = ni_size;
- ret = cachefiles_inject_write_error();
- if (ret == 0)
- ret = notify_change(&nop_mnt_idmap, file->f_path.dentry,
- &newattrs, NULL);
-
-truncate_failed:
- inode_unlock(file_inode(file));
-
- if (ret < 0)
- trace_cachefiles_io_error(NULL, file_inode(file), ret,
- cachefiles_trace_notify_change_error);
- if (ret == -EIO) {
- cachefiles_io_error_obj(object, "Size set failed");
- ret = -ENOBUFS;
- }
-
- _leave(" = %d", ret);
- return ret;
-}
-
/*
* Attempt to look up the nominated node in this cache
*/
@@ -204,7 +131,6 @@ static bool cachefiles_lookup_cookie(struct fscache_cookie *cookie)
spin_lock(&cache->object_list_lock);
list_add(&object->cache_link, &cache->object_list);
spin_unlock(&cache->object_list_lock);
- cachefiles_adjust_size(object);
cachefiles_end_secure(cache, saved_cred);
_leave(" = t");
@@ -238,7 +164,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,
loff_t i_size, dio_size;
int ret;
- dio_size = round_up(new_size, CACHEFILES_DIO_BLOCK_SIZE);
+ dio_size = round_up(new_size, cache->bsize);
i_size = i_size_read(inode);
trace_cachefiles_trunc(object, inode, i_size, dio_size,
@@ -270,6 +196,7 @@ static bool cachefiles_shorten_object(struct cachefiles_object *object,
}
}
+ object->object_size = new_size;
return true;
}
@@ -284,22 +211,31 @@ static void cachefiles_resize_cookie(struct netfs_cache_resources *cres,
struct fscache_cookie *cookie = object->cookie;
const struct cred *saved_cred;
struct file *file = cachefiles_cres_file(cres);
- loff_t old_size = cookie->object_size;
+ unsigned long long i_size = i_size_read(file_inode(file));
- _enter("%llu->%llu", old_size, new_size);
+ _enter("%llu->%llu", object->object_size, new_size);
- if (new_size < old_size) {
+ /* If the file is being shrunk, we need to downsize the backing file
+ * and clear the end of the final block.
+ */
+ if (new_size < object->object_size) {
+ if (new_size >= i_size)
+ goto out;
cachefiles_begin_secure(cache, &saved_cred);
cachefiles_shorten_object(object, file, new_size);
cachefiles_end_secure(cache, saved_cred);
object->cookie->object_size = new_size;
+ if (new_size == 0)
+ object->content_info = CACHEFILES_CONTENT_NO_DATA;
return;
}
/* The file is being expanded. We don't need to do anything
- * particularly. cookie->initial_size doesn't change and so the point
- * at which we have to download before doesn't change.
+ * particularly. The tail of the last block should have been cleared
+ * both when it is written and when it is shrunk.
*/
+out:
+ object->object_size = new_size;
cookie->object_size = new_size;
}
@@ -374,8 +310,6 @@ static void cachefiles_withdraw_cookie(struct fscache_cookie *cookie)
spin_unlock(&cache->object_list_lock);
}
- cachefiles_ondemand_clean_object(object);
-
if (object->file) {
cachefiles_begin_secure(cache, &saved_cred);
cachefiles_clean_up_object(object, cache);
diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h
index b62cd3e9a18e4c..b31d7e93fe992b 100644
--- a/fs/cachefiles/internal.h
+++ b/fs/cachefiles/internal.h
@@ -15,11 +15,8 @@
#include <linux/fscache-cache.h>
#include <linux/cred.h>
#include <linux/security.h>
-#include <linux/xarray.h>
#include <linux/cachefiles.h>
-#define CACHEFILES_DIO_BLOCK_SIZE 4096
-
struct cachefiles_cache;
struct cachefiles_object;
@@ -44,21 +41,6 @@ struct cachefiles_volume {
struct dentry *fanout[256]; /* Fanout subdirs */
};
-enum cachefiles_object_state {
- CACHEFILES_ONDEMAND_OBJSTATE_CLOSE, /* Anonymous fd closed by daemon or initial state */
- CACHEFILES_ONDEMAND_OBJSTATE_OPEN, /* Anonymous fd associated with object is available */
- CACHEFILES_ONDEMAND_OBJSTATE_REOPENING, /* Object that was closed and is being reopened. */
- CACHEFILES_ONDEMAND_OBJSTATE_DROPPING, /* Object is being dropped. */
-};
-
-struct cachefiles_ondemand_info {
- struct work_struct ondemand_work;
- int ondemand_id;
- enum cachefiles_object_state state;
- struct cachefiles_object *object;
- spinlock_t lock;
-};
-
/*
* Backing file state.
*/
@@ -68,19 +50,19 @@ struct cachefiles_object {
struct list_head cache_link; /* Link in cache->*_list */
struct file *file; /* The file representing this object */
char *d_name; /* Backing file name */
+ unsigned long flags;
+#define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */
+ unsigned long long object_size; /* Size of the object stored
+ * (independent of cookie->object_size for
+ * coherency reasons)
+ */
+ atomic64_t read_limit; /* Point beyond which uncommitted writes */
int debug_id;
spinlock_t lock;
refcount_t ref;
- enum cachefiles_content content_info:8; /* Info about content presence */
- unsigned long flags;
-#define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */
-#ifdef CONFIG_CACHEFILES_ONDEMAND
- struct cachefiles_ondemand_info *ondemand;
-#endif
+ enum cachefiles_content content_info; /* Info about content presence */
};
-#define CACHEFILES_ONDEMAND_ID_CLOSED -1
-
/*
* Cache files cache definition
*/
@@ -119,35 +101,13 @@ struct cachefiles_cache {
#define CACHEFILES_DEAD 1 /* T if cache dead */
#define CACHEFILES_CULLING 2 /* T if cull engaged */
#define CACHEFILES_STATE_CHANGED 3 /* T if state changed (poll trigger) */
-#define CACHEFILES_ONDEMAND_MODE 4 /* T if in on-demand read mode */
char *rootdirname; /* name of cache root directory */
char *tag; /* cache binding tag */
refcount_t unbind_pincount;/* refcount to do daemon unbind */
- struct xarray reqs; /* xarray of pending on-demand requests */
- unsigned long req_id_next;
- struct xarray ondemand_ids; /* xarray for ondemand_id allocation */
- u32 ondemand_id_next;
- u32 msg_id_next;
u32 secid; /* LSM security id */
bool have_secid; /* whether "secid" was set */
};
-static inline bool cachefiles_in_ondemand_mode(struct cachefiles_cache *cache)
-{
- return IS_ENABLED(CONFIG_CACHEFILES_ONDEMAND) &&
- test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags);
-}
-
-struct cachefiles_req {
- struct cachefiles_object *object;
- struct completion done;
- refcount_t ref;
- int error;
- struct cachefiles_msg msg;
-};
-
-#define CACHEFILES_REQ_NEW XA_MARK_1
-
#include <trace/events/cachefiles.h>
static inline
@@ -298,90 +258,6 @@ extern struct file *cachefiles_create_tmpfile(struct cachefiles_object *object);
extern bool cachefiles_commit_tmpfile(struct cachefiles_cache *cache,
struct cachefiles_object *object);
-/*
- * ondemand.c
- */
-#ifdef CONFIG_CACHEFILES_ONDEMAND
-extern ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen);
-
-extern int cachefiles_ondemand_copen(struct cachefiles_cache *cache,
- char *args);
-
-extern int cachefiles_ondemand_restore(struct cachefiles_cache *cache,
- char *args);
-
-extern int cachefiles_ondemand_init_object(struct cachefiles_object *object);
-extern void cachefiles_ondemand_clean_object(struct cachefiles_object *object);
-
-extern int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len);
-
-extern int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,
- struct cachefiles_volume *volume);
-extern void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj);
-
-#define CACHEFILES_OBJECT_STATE_FUNCS(_state, _STATE) \
-static inline bool \
-cachefiles_ondemand_object_is_##_state(const struct cachefiles_object *object) \
-{ \
- return object->ondemand->state == CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \
-} \
- \
-static inline void \
-cachefiles_ondemand_set_object_##_state(struct cachefiles_object *object) \
-{ \
- object->ondemand->state = CACHEFILES_ONDEMAND_OBJSTATE_##_STATE; \
-}
-
-CACHEFILES_OBJECT_STATE_FUNCS(open, OPEN);
-CACHEFILES_OBJECT_STATE_FUNCS(close, CLOSE);
-CACHEFILES_OBJECT_STATE_FUNCS(reopening, REOPENING);
-CACHEFILES_OBJECT_STATE_FUNCS(dropping, DROPPING);
-
-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)
-{
- return cachefiles_ondemand_object_is_reopening(req->object) &&
- req->msg.opcode == CACHEFILES_OP_READ;
-}
-
-#else
-static inline ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen)
-{
- return -EOPNOTSUPP;
-}
-
-static inline int cachefiles_ondemand_init_object(struct cachefiles_object *object)
-{
- return 0;
-}
-
-static inline void cachefiles_ondemand_clean_object(struct cachefiles_object *object)
-{
-}
-
-static inline int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len)
-{
- return -EOPNOTSUPP;
-}
-
-static inline int cachefiles_ondemand_init_obj_info(struct cachefiles_object *obj,
- struct cachefiles_volume *volume)
-{
- return 0;
-}
-static inline void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *obj)
-{
-}
-
-static inline bool cachefiles_ondemand_is_reopening_read(struct cachefiles_req *req)
-{
- return false;
-}
-#endif
-
/*
* security.c
*/
@@ -430,8 +306,6 @@ do { \
pr_err("I/O Error: " FMT"\n", ##__VA_ARGS__); \
fscache_io_error((___cache)->cache); \
set_bit(CACHEFILES_DEAD, &(___cache)->flags); \
- if (cachefiles_in_ondemand_mode(___cache)) \
- cachefiles_flush_reqs(___cache); \
} while (0)
#define cachefiles_io_error_obj(object, FMT, ...) \
diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c
index d879b80a0bedc9..f289a3bddd63bb 100644
--- a/fs/cachefiles/io.c
+++ b/fs/cachefiles/io.c
@@ -26,12 +26,17 @@ struct cachefiles_kiocb {
};
struct cachefiles_object *object;
netfs_io_terminated_t term_func;
- void *term_func_priv;
+ union {
+ struct netfs_io_subrequest *subreq;
+ void *term_func_priv;
+ };
bool was_async;
unsigned int inval_counter; /* Copy of cookie->inval_counter */
u64 b_writing;
};
+#define IS_ERR_VALUE_LL(x) unlikely((x) >= (unsigned long long)-MAX_ERRNO)
+
static inline void cachefiles_put_kiocb(struct cachefiles_kiocb *ki)
{
if (refcount_dec_and_test(&ki->ki_refcnt)) {
@@ -193,61 +198,204 @@ static int cachefiles_read(struct netfs_cache_resources *cres,
}
/*
- * Query the occupancy of the cache in a region, returning where the next chunk
- * of data starts and how long it is.
+ * Handle completion of a read from the cache issued by netfslib.
+ */
+static void cachefiles_issue_read_complete(struct kiocb *iocb, long ret)
+{
+ struct cachefiles_kiocb *ki = container_of(iocb, struct cachefiles_kiocb, iocb);
+ struct netfs_io_subrequest *subreq = ki->subreq;
+ struct inode *inode = file_inode(ki->iocb.ki_filp);
+
+ _enter("%ld", ret);
+
+ if (ret < 0) {
+ subreq->error = -ESTALE;
+ trace_cachefiles_io_error(ki->object, inode, ret,
+ cachefiles_trace_read_error);
+ }
+
+ if (ret >= 0) {
+ if (ki->object->cookie->inval_counter == ki->inval_counter) {
+ subreq->error = 0;
+ if (ret > 0) {
+ subreq->transferred += ret;
+ __set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ }
+ } else {
+ subreq->error = -ESTALE;
+ }
+ }
+
+ netfs_read_subreq_terminated(subreq);
+ cachefiles_put_kiocb(ki);
+}
+
+/*
+ * Issue a read operation to the cache.
*/
-static int cachefiles_query_occupancy(struct netfs_cache_resources *cres,
- loff_t start, size_t len, size_t granularity,
- loff_t *_data_start, size_t *_data_len)
+static int cachefiles_issue_read(struct netfs_io_subrequest *subreq)
{
+ struct netfs_cache_resources *cres = &subreq->rreq->cache_resources;
struct cachefiles_object *object;
+ struct cachefiles_kiocb *ki;
+ struct iov_iter iter;
struct file *file;
- loff_t off, off2;
-
- *_data_start = -1;
- *_data_len = 0;
+ unsigned int old_nofs;
+ ssize_t ret = -ENOBUFS;
if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))
return -ENOBUFS;
+ fscache_count_read();
object = cachefiles_cres_object(cres);
file = cachefiles_cres_file(cres);
- granularity = max_t(size_t, object->volume->cache->bsize, granularity);
- _enter("%pD,%llu,%llx,%zx/%llx",
- file, file_inode(file)->i_ino, start, len,
+ _enter("%pD,%lli,%llx,%zx/%llx",
+ file, file_inode(file)->i_ino, subreq->start, subreq->len,
i_size_read(file_inode(file)));
- off = cachefiles_inject_read_error();
- if (off == 0)
- off = vfs_llseek(file, start, SEEK_DATA);
- if (off == -ENXIO)
- return -ENODATA; /* Beyond EOF */
- if (off < 0 && off >= (loff_t)-MAX_ERRNO)
- return -ENOBUFS; /* Error. */
- if (round_up(off, granularity) >= start + len)
- return -ENODATA; /* No data in range */
-
- off2 = cachefiles_inject_read_error();
- if (off2 == 0)
- off2 = vfs_llseek(file, off, SEEK_HOLE);
- if (off2 == -ENXIO)
- return -ENODATA; /* Beyond EOF */
- if (off2 < 0 && off2 >= (loff_t)-MAX_ERRNO)
- return -ENOBUFS; /* Error. */
-
- /* Round away partial blocks */
- off = round_up(off, granularity);
- off2 = round_down(off2, granularity);
- if (off2 <= off)
- return -ENODATA;
-
- *_data_start = off;
- if (off2 > start + len)
- *_data_len = len;
- else
- *_data_len = off2 - off;
+ if (subreq->len > MAX_RW_COUNT)
+ subreq->len = MAX_RW_COUNT;
+
+ ret = netfs_prepare_read_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
+ ret = -ENOMEM;
+ ki = kzalloc_obj(struct cachefiles_kiocb);
+ if (!ki)
+ goto failed;
+
+ refcount_set(&ki->ki_refcnt, 2);
+ ki->iocb.ki_filp = file;
+ ki->iocb.ki_pos = subreq->start;
+ ki->iocb.ki_flags = IOCB_DIRECT;
+ ki->iocb.ki_ioprio = get_current_ioprio();
+ ki->iocb.ki_complete = cachefiles_issue_read_complete;
+ ki->object = object;
+ ki->inval_counter = cres->inval_counter;
+ ki->subreq = subreq;
+ ki->was_async = true;
+
+ get_file(ki->iocb.ki_filp);
+ cachefiles_grab_object(object, cachefiles_obj_get_ioreq);
+
+ trace_cachefiles_read(object, file_inode(file), ki->iocb.ki_pos, subreq->len);
+ old_nofs = memalloc_nofs_save();
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_iocb_iter_read(file, &ki->iocb, &iter);
+ memalloc_nofs_restore(old_nofs);
+
+ switch (ret) {
+ case -EIOCBQUEUED:
+ break;
+
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /* There's no easy way to restart the syscall since other AIO's
+ * may be already running. Just fail this IO with EINTR.
+ */
+ ret = -EINTR;
+ fallthrough;
+ default:
+ ki->was_async = false;
+ cachefiles_issue_read_complete(&ki->iocb, ret);
+ break;
+ }
+
+ cachefiles_put_kiocb(ki);
+ _leave(" = %zd", ret);
return 0;
+failed:
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
+}
+
+/*
+ * Query the occupancy of the cache in a region, returning the extent of the
+ * next two chunks of cached data and the next hole. The occupancy map is
+ * preloaded to show just one giant hole.
+ */
+static void cachefiles_query_occupancy(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ)
+{
+ struct cachefiles_object *object;
+ struct inode *inode;
+ struct file *file;
+ unsigned long long read_limit;
+ loff_t ret;
+ int i;
+
+ if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ))
+ return;
+
+ object = cachefiles_cres_object(cres);
+ file = cachefiles_cres_file(cres);
+ inode = file_inode(file);
+ occ->granularity = object->volume->cache->bsize;
+ /* Read read_limit before content_info. */
+ read_limit = atomic64_read_acquire(&object->read_limit);
+
+ _enter("%pD,%llu,%llx-%llx/%llx",
+ file, inode->i_ino, occ->query_from, occ->query_to, read_limit);
+
+ if (read_limit == 0)
+ goto done;
+
+ switch (READ_ONCE(object->content_info)) {
+ case CACHEFILES_CONTENT_ALL:
+ case CACHEFILES_CONTENT_SINGLE:
+ if (read_limit > occ->query_from) {
+ occ->cached_from[0] = 0;
+ occ->cached_to[0] = read_limit;
+ occ->cached_type[0] = FSCACHE_EXTENT_DATA;
+ occ->query_from = ULLONG_MAX;
+ }
+ goto done;
+ default:
+ break;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(occ->cached_from); i++) {
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_llseek(file, occ->query_from, SEEK_DATA);
+ if (IS_ERR_VALUE_LL(ret)) {
+ if (ret != -ENXIO)
+ goto done;
+ occ->query_from = ULLONG_MAX;
+ goto done;
+ }
+ occ->cached_type[i] = FSCACHE_EXTENT_DATA;
+ occ->cached_from[i] = ret;
+ occ->query_from = ret;
+
+ ret = cachefiles_inject_read_error();
+ if (ret == 0)
+ ret = vfs_llseek(file, occ->query_from, SEEK_HOLE);
+ if (IS_ERR_VALUE_LL(ret)) {
+ if (ret != -ENXIO)
+ goto done;
+ occ->query_from = ULLONG_MAX;
+ goto done;
+ }
+ occ->cached_to[i] = ret;
+ occ->query_from = ret;
+ if (occ->query_from >= occ->query_to)
+ break;
+ }
+
+done:
+ _debug("query[0] %llx-%llx", occ->cached_from[0], occ->cached_to[0]);
+ _debug("query[1] %llx-%llx", occ->cached_from[1], occ->cached_to[1]);
}
/*
@@ -390,7 +538,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
size_t len = *_len;
loff_t off, to;
ino_t ino = file ? file_inode(file)->i_ino : 0;
- int rc;
_enter("%zx @%llx/%llx", len, start, i_size);
@@ -403,8 +550,7 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
if (test_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags)) {
__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);
why = cachefiles_trace_read_no_data;
- if (!test_bit(NETFS_SREQ_ONDEMAND, _flags))
- goto out_no_object;
+ goto out_no_object;
}
/* The object and the file may be being created in the background. */
@@ -421,7 +567,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
object = cachefiles_cres_object(cres);
cache = object->volume->cache;
cachefiles_begin_secure(cache, &saved_cred);
-retry:
off = cachefiles_inject_read_error();
if (off == 0)
off = vfs_llseek(file, start, SEEK_DATA);
@@ -474,14 +619,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
download_and_store:
__set_bit(NETFS_SREQ_COPY_TO_CACHE, _flags);
- if (test_bit(NETFS_SREQ_ONDEMAND, _flags)) {
- rc = cachefiles_ondemand_read(object, start, len);
- if (!rc) {
- __clear_bit(NETFS_SREQ_ONDEMAND, _flags);
- goto retry;
- }
- ret = NETFS_INVALID_READ;
- }
out:
cachefiles_end_secure(cache, saved_cred);
out_no_object:
@@ -489,30 +626,6 @@ cachefiles_do_prepare_read(struct netfs_cache_resources *cres,
return ret;
}
-/*
- * Prepare a read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
-static enum netfs_io_source cachefiles_prepare_read(struct netfs_io_subrequest *subreq,
- unsigned long long i_size)
-{
- return cachefiles_do_prepare_read(&subreq->rreq->cache_resources,
- subreq->start, &subreq->len, i_size,
- &subreq->flags, subreq->rreq->inode->i_ino);
-}
-
-/*
- * Prepare an on-demand read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
-static enum netfs_io_source
-cachefiles_prepare_ondemand_read(struct netfs_cache_resources *cres,
- loff_t start, size_t *_len, loff_t i_size,
- unsigned long *_flags, ino_t ino)
-{
- return cachefiles_do_prepare_read(cres, start, _len, i_size, _flags, ino);
-}
-
/*
* Prepare for a write to occur.
*/
@@ -527,7 +640,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
int ret;
/* Round to DIO size */
- start = round_down(*_start, PAGE_SIZE);
+ start = round_down(*_start, cache->bsize);
if (start != *_start || *_len > upper_len) {
/* Probably asked to cache a streaming write written into the
* pagecache when the cookie was temporarily out of service to
@@ -537,7 +650,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
return -ENOBUFS;
}
- *_len = round_up(len, PAGE_SIZE);
+ *_len = round_up(len, cache->bsize);
/* We need to work out whether there's sufficient disk space to perform
* the write - but we can skip that check if we have space already
@@ -563,7 +676,7 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
* space, we need to see if it's fully allocated. If it's not, we may
* want to cull it.
*/
- if (cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,
+ if (cachefiles_has_space(cache, 0, *_len / cache->bsize,
cachefiles_has_space_check) == 0)
return 0; /* Enough space to simply overwrite the whole block */
@@ -595,13 +708,13 @@ int __cachefiles_prepare_write(struct cachefiles_object *object,
return ret;
check_space:
- return cachefiles_has_space(cache, 0, *_len / PAGE_SIZE,
+ return cachefiles_has_space(cache, 0, *_len / cache->bsize,
cachefiles_has_space_for_write);
}
-static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
- loff_t *_start, size_t *_len, size_t upper_len,
- loff_t i_size, bool no_space_allocated_yet)
+static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres,
+ loff_t *_start, size_t *_len, size_t upper_len,
+ loff_t i_size, bool no_space_allocated_yet)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
@@ -623,92 +736,238 @@ static int cachefiles_prepare_write(struct netfs_cache_resources *cres,
return ret;
}
-static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq)
+static int cachefiles_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
{
- struct netfs_io_request *wreq = subreq->rreq;
- struct netfs_cache_resources *cres = &wreq->cache_resources;
- struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
-
- _enter("W=%x[%x] %llx", wreq->debug_id, subreq->debug_index, subreq->start);
-
- stream->sreq_max_len = MAX_RW_COUNT;
- stream->sreq_max_segs = BIO_MAX_VECS;
-
- if (!cachefiles_cres_file(cres)) {
- if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))
- return netfs_prepare_write_failed(subreq);
- if (!cachefiles_cres_file(cres))
- return netfs_prepare_write_failed(subreq);
- }
+ estimate->issue_at = stream->issue_from + MAX_RW_COUNT;
+ estimate->max_segs = BIO_MAX_VECS;
+ return 0;
}
-static void cachefiles_issue_write(struct netfs_io_subrequest *subreq)
+static int cachefiles_issue_write(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
struct netfs_cache_resources *cres = &wreq->cache_resources;
struct cachefiles_object *object = cachefiles_cres_object(cres);
struct cachefiles_cache *cache = object->volume->cache;
- struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+ struct iov_iter iter;
const struct cred *saved_cred;
- size_t off, pre, post, len = subreq->len;
loff_t start = subreq->start;
- int ret;
+ size_t len = subreq->len;
+ int ret = -EINVAL;
_enter("W=%x[%x] %llx-%llx",
wreq->debug_id, subreq->debug_index, start, start + len - 1);
- /* We need to start on the cache granularity boundary */
- off = start & (CACHEFILES_DIO_BLOCK_SIZE - 1);
- if (off) {
- pre = CACHEFILES_DIO_BLOCK_SIZE - off;
- if (pre >= len) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, len);
- return;
- }
- subreq->transferred += pre;
- start += pre;
- len -= pre;
- iov_iter_advance(&subreq->io_iter, pre);
+ if (!cachefiles_cres_file(cres)) {
+ if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE))
+ return -EINVAL;
+ if (!cachefiles_cres_file(cres))
+ return -EINVAL;
}
- /* We also need to end on the cache granularity boundary */
- if (start + len == wreq->i_size) {
- size_t part = len % CACHEFILES_DIO_BLOCK_SIZE;
- size_t need = CACHEFILES_DIO_BLOCK_SIZE - part;
+ ret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS);
+ if (ret < 0)
+ return ret;
+ /* After this point, must fail by termination. */
- if (part && stream->submit_extendable_to >= need) {
- len += need;
- subreq->len += need;
- subreq->io_iter.count += need;
- }
- }
+ /* The buffer extraction func may round out start and end. */
+ start = subreq->start;
+ len = subreq->len;
- post = len & (CACHEFILES_DIO_BLOCK_SIZE - 1);
- if (post) {
- len -= post;
- if (len == 0) {
- fscache_count_dio_misfit();
- netfs_write_subrequest_terminated(subreq, post);
- return;
- }
- iov_iter_truncate(&subreq->io_iter, len);
+ /* We need to start and end on cache granularity boundaries. */
+ if (WARN_ON_ONCE(start & (cache->bsize - 1)) ||
+ WARN_ON_ONCE(len & (cache->bsize - 1))) {
+ fscache_count_dio_misfit();
+ ret = -EIO;
+ goto failed;
}
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, len);
+
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_prepare);
cachefiles_begin_secure(cache, &saved_cred);
ret = __cachefiles_prepare_write(object, cachefiles_cres_file(cres),
&start, &len, len, true);
cachefiles_end_secure(cache, saved_cred);
- if (ret < 0) {
- netfs_write_subrequest_terminated(subreq, ret);
- return;
- }
+ if (ret < 0)
+ goto failed;
trace_netfs_sreq(subreq, netfs_sreq_trace_cache_write);
- cachefiles_write(&subreq->rreq->cache_resources,
- subreq->start, &subreq->io_iter,
+ cachefiles_write(&subreq->rreq->cache_resources, subreq->start, &iter,
netfs_write_subrequest_terminated, subreq);
+ return 0;
+failed:
+ netfs_write_subrequest_terminated(subreq, ret);
+ return 0;
+}
+
+/*
+ * Collect the result of buffered writeback to the cache. This includes
+ * copying a read to the cache. Netfslib collates the results, which might
+ * occur out of order, and delivers them to the cache so that it can update its
+ * content record.
+ *
+ * block_type is one of:
+ * - NETFS_CACHE_COLLECT_WRITE_DATA for a contiguous block of data
+ * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped
+ * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a hole due to a failed/cancelled write
+ *
+ * The writes we made are all rounded out at both sides to the nearest DIO
+ * block boundary, so if the final block contains the EOF in the middle of it
+ * (rather than at the end), padding will have been written to the file. The
+ * backing file's filesize will have been updated if the write extended the
+ * file; the filesize may still change due to outstanding subreqs.
+ *
+ * The metadata in the cache file xattr records the size of the object we have
+ * stored, but the cache file EOF only goes up to where we've cached data to
+ * and, furthermore, is rounded up to the nearest DIO block boundary.
+ *
+ * Concurrent updates should be protected against by the caller. Netfslib
+ * holds NETFS_ICTX_WB_LOCK as a lock on writeback requests. DIO writes
+ * invalidate the cookie and caching is kept disabled until all users have
+ * unused the cookie.
+ */
+static void cachefiles_collect_write(struct netfs_io_request *wreq,
+ unsigned long long start, size_t len,
+ enum netfs_cache_collect block_type)
+{
+ struct netfs_cache_resources *cres = &wreq->cache_resources;
+ struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct cachefiles_cache *cache = object->volume->cache;
+ struct file *file = cachefiles_cres_file(cres);
+ struct inode *inode = file_inode(file);
+ unsigned long long read_limit;
+ unsigned long long old_size = cres->cache_i_size;
+ unsigned long long new_size = i_size_read(inode);
+ unsigned long long data_to = object->object_size;
+ unsigned long long end = start + len;
+ int ret;
+
+ _enter("%llx,%zx,%x", start, len, cache->bsize);
+
+ if (WARN_ON(old_size & (cache->bsize - 1)) ||
+ WARN_ON(new_size & (cache->bsize - 1)) ||
+ WARN_ON(start & (cache->bsize - 1)) ||
+ WARN_ON(len & (cache->bsize - 1))) {
+ trace_cachefiles_io_error(object, inode, -EIO,
+ cachefiles_trace_alignment_error);
+ cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
+ return;
+ }
+
+ /* If this is recording a gap, due to discontiguous writes or lack of
+ * cache space, then a hole may have been introduced into the backing
+ * file. Treat it as a zero-length data block.
+ */
+ if (block_type == NETFS_CACHE_COLLECT_WRITE_GAP ||
+ block_type == NETFS_CACHE_COLLECT_WRITE_CANCEL) {
+ start = end;
+ len = 0;
+ }
+
+ /* Zeroth case: Single monolithic files are handled specially.
+ */
+ if (wreq->origin == NETFS_WRITEBACK_SINGLE) {
+ object->content_info = CACHEFILES_CONTENT_SINGLE;
+ goto update_sizes;
+ }
+
+ /* First case: The backing file was empty. */
+ if (old_size == 0) {
+ if (start == 0)
+ object->content_info = CACHEFILES_CONTENT_ALL;
+ else
+ object->content_info = CACHEFILES_CONTENT_BACKFS_MAP;
+ goto update_sizes;
+ }
+
+ /* Second case: The backing file is entirely within the old object size
+ * and thus there can be no partial tail block to deal with in the
+ * cache file.
+ */
+ if (old_size <= data_to) {
+ if (start > old_size)
+ goto discontiguous;
+ goto update_sizes;
+ }
+
+ /* Third case: The write happened entirely within the bounds of the
+ * current cache file's size.
+ */
+ if (end <= old_size)
+ goto update_sizes;
+
+ /* Fourth case: The write overwrote the partial tail block and extended
+ * the file. We only need to update the object size because netfslib
+ * rounds out/pads cache writes to whole disk blocks.
+ */
+ if (start < old_size)
+ goto update_sizes;
+
+ /* Fifth case: The write started from the end of the whole tail block
+ * and extended the file. Just extend our notion of the filesize.
+ */
+ if (start == old_size && old_size == data_to)
+ goto update_sizes;
+
+ /* Sixth case: The write continued on from the partial tail block and
+ * extended the file. Need to clear the gap.
+ */
+ if (start == old_size && old_size > data_to)
+ goto clear_gap;
+
+discontiguous:
+ /* Seventh case: The write was beyond the EOF on the cache file, so now
+ * there's a hole in the file and we can no longer say in the metadata
+ * that we can assume we have it all. We may also need to clear the
+ * end of the partial tail block.
+ */
+ /* TODO: For the moment, we will have to use SEEK_HOLE/SEEK_DATA. */
+ if (object->content_info != CACHEFILES_CONTENT_BACKFS_MAP) {
+ object->content_info = CACHEFILES_CONTENT_BACKFS_MAP;
+ trace_cachefiles_coherency(object, inode->i_ino, data_to,
+ be64_to_cpup((__be64 *)object->cookie->inline_aux),
+ CACHEFILES_CONTENT_BACKFS_MAP,
+ cachefiles_coherency_discontiguous);
+ }
+
+clear_gap:
+ /* We need to clear any partial padding that got jumped over. It
+ * *should* be all zeros, but shared-writable mmap exists...
+ */
+ if (old_size > data_to) {
+ trace_cachefiles_trunc(object, inode, data_to, old_size,
+ cachefiles_trunc_clear_padding);
+ ret = cachefiles_inject_write_error();
+ if (ret == 0)
+ ret = vfs_fallocate(file, FALLOC_FL_ZERO_RANGE,
+ data_to, old_size - data_to);
+ if (ret < 0) {
+ trace_cachefiles_io_error(object, inode, ret,
+ cachefiles_trace_fallocate_error);
+ cachefiles_io_error_obj(object, "fallocate zero pad failed %d", ret);
+ cachefiles_remove_object_xattr(cache, object, file->f_path.dentry);
+ return;
+ }
+ }
+
+update_sizes:
+ read_limit = umax(old_size, end);
+ cres->cache_i_size = read_limit;
+
+ /* We need to be careful setting the object_size: we may have written
+ * more to the cache than to the server (due to cache DIO rounding) and
+ * the i_size set on the netfs inode may include unwritten data that
+ * the server doesn't know about yet.
+ */
+ object->object_size = umin(read_limit, wreq->i_size);
+
+ /* Raise the limit at which reads can access the file. */
+ /* Update read_limit after content_info */
+ atomic64_set_release(&object->read_limit, read_limit);
}
/*
@@ -727,12 +986,12 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = {
.end_operation = cachefiles_end_operation,
.read = cachefiles_read,
.write = cachefiles_write,
+ .issue_read = cachefiles_issue_read,
.issue_write = cachefiles_issue_write,
- .prepare_read = cachefiles_prepare_read,
- .prepare_write = cachefiles_prepare_write,
- .prepare_write_subreq = cachefiles_prepare_write_subreq,
- .prepare_ondemand_read = cachefiles_prepare_ondemand_read,
+ .estimate_write = cachefiles_estimate_write,
+ .prepare_write_old = cachefiles_prepare_write_old,
.query_occupancy = cachefiles_query_occupancy,
+ .collect_write = cachefiles_collect_write,
};
/*
@@ -742,13 +1001,19 @@ bool cachefiles_begin_operation(struct netfs_cache_resources *cres,
enum fscache_want_state want_state)
{
struct cachefiles_object *object = cachefiles_cres_object(cres);
+ struct file *file;
if (!cachefiles_cres_file(cres)) {
cres->ops = &cachefiles_netfs_cache_ops;
+ cres->object_id = object->debug_id;
if (object->file) {
spin_lock(&object->lock);
- if (!cres->cache_priv2 && object->file)
- cres->cache_priv2 = get_file(object->file);
+ file = object->file;
+ if (!cres->cache_priv2 && file) {
+ cres->cache_priv2 = get_file(file);
+ cres->cache_i_size = i_size_read(file_inode(file));
+ cres->dio_size = object->volume->cache->bsize;
+ }
spin_unlock(&object->lock);
}
}
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index 8a9f6be15828ba..a30df0f91e5e5d 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -414,7 +414,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
struct dentry *fan = volume->fanout[(u8)object->cookie->key_hash];
struct file *file;
const struct path parentpath = { .mnt = cache->mnt, .dentry = fan };
- uint64_t ni_size;
long ret;
@@ -442,27 +441,6 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
if (!cachefiles_mark_inode_in_use(object, file_inode(file)))
WARN_ON(1);
- ret = cachefiles_ondemand_init_object(object);
- if (ret < 0)
- goto err_unuse;
-
- ni_size = object->cookie->object_size;
- ni_size = round_up(ni_size, CACHEFILES_DIO_BLOCK_SIZE);
-
- if (ni_size > 0) {
- trace_cachefiles_trunc(object, file_inode(file), 0, ni_size,
- cachefiles_trunc_expand_tmpfile);
- ret = cachefiles_inject_write_error();
- if (ret == 0)
- ret = vfs_truncate(&file->f_path, ni_size);
- if (ret < 0) {
- trace_cachefiles_vfs_error(
- object, file_inode(file), ret,
- cachefiles_trace_trunc_error);
- goto err_unuse;
- }
- }
-
ret = -EINVAL;
if (unlikely(!file->f_op->read_iter) ||
unlikely(!file->f_op->write_iter)) {
@@ -471,6 +449,7 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object)
}
out:
cachefiles_end_secure(cache, saved_cred);
+ object->content_info = CACHEFILES_CONTENT_ALL;
return file;
err_unuse:
@@ -546,10 +525,6 @@ static bool cachefiles_open_file(struct cachefiles_object *object,
}
_debug("file -> %pd positive", dentry);
- ret = cachefiles_ondemand_init_object(object);
- if (ret < 0)
- goto error_fput;
-
ret = cachefiles_check_auxdata(object, file);
if (ret < 0)
goto check_failed;
diff --git a/fs/cachefiles/ondemand.c b/fs/cachefiles/ondemand.c
deleted file mode 100644
index 0849eaf583cd27..00000000000000
--- a/fs/cachefiles/ondemand.c
+++ /dev/null
@@ -1,761 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-#include <linux/anon_inodes.h>
-#include <linux/uio.h>
-#include "internal.h"
-
-struct ondemand_anon_file {
- struct file *file;
- int fd;
-};
-
-static inline void cachefiles_req_put(struct cachefiles_req *req)
-{
- if (refcount_dec_and_test(&req->ref))
- kfree(req);
-}
-
-static int cachefiles_ondemand_fd_release(struct inode *inode,
- struct file *file)
-{
- struct cachefiles_object *object = file->private_data;
- struct cachefiles_cache *cache;
- struct cachefiles_ondemand_info *info;
- int object_id;
- struct cachefiles_req *req;
- XA_STATE(xas, NULL, 0);
-
- if (!object)
- return 0;
-
- info = object->ondemand;
- cache = object->volume->cache;
- xas.xa = &cache->reqs;
-
- xa_lock(&cache->reqs);
- spin_lock(&info->lock);
- object_id = info->ondemand_id;
- info->ondemand_id = CACHEFILES_ONDEMAND_ID_CLOSED;
- cachefiles_ondemand_set_object_close(object);
- spin_unlock(&info->lock);
-
- /* Only flush CACHEFILES_REQ_NEW marked req to avoid race with daemon_read */
- xas_for_each_marked(&xas, req, ULONG_MAX, CACHEFILES_REQ_NEW) {
- if (req->msg.object_id == object_id &&
- req->msg.opcode == CACHEFILES_OP_CLOSE) {
- complete(&req->done);
- xas_store(&xas, NULL);
- }
- }
- xa_unlock(&cache->reqs);
-
- xa_erase(&cache->ondemand_ids, object_id);
- trace_cachefiles_ondemand_fd_release(object, object_id);
- cachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);
- cachefiles_put_unbind_pincount(cache);
- return 0;
-}
-
-static ssize_t cachefiles_ondemand_fd_write_iter(struct kiocb *kiocb,
- struct iov_iter *iter)
-{
- struct cachefiles_object *object = kiocb->ki_filp->private_data;
- struct cachefiles_cache *cache = object->volume->cache;
- struct file *file;
- size_t len = iter->count, aligned_len = len;
- loff_t pos = kiocb->ki_pos;
- const struct cred *saved_cred;
- int ret;
-
- spin_lock(&object->lock);
- file = object->file;
- if (!file) {
- spin_unlock(&object->lock);
- return -ENOBUFS;
- }
- get_file(file);
- spin_unlock(&object->lock);
-
- cachefiles_begin_secure(cache, &saved_cred);
- ret = __cachefiles_prepare_write(object, file, &pos, &aligned_len, len, true);
- cachefiles_end_secure(cache, saved_cred);
- if (ret < 0)
- goto out;
-
- trace_cachefiles_ondemand_fd_write(object, file_inode(file), pos, len);
- ret = __cachefiles_write(object, file, pos, iter, NULL, NULL);
- if (ret > 0)
- kiocb->ki_pos += ret;
-
-out:
- fput(file);
- return ret;
-}
-
-static loff_t cachefiles_ondemand_fd_llseek(struct file *filp, loff_t pos,
- int whence)
-{
- struct cachefiles_object *object = filp->private_data;
- struct file *file;
- loff_t ret;
-
- spin_lock(&object->lock);
- file = object->file;
- if (!file) {
- spin_unlock(&object->lock);
- return -ENOBUFS;
- }
- get_file(file);
- spin_unlock(&object->lock);
-
- ret = vfs_llseek(file, pos, whence);
- fput(file);
-
- return ret;
-}
-
-static long cachefiles_ondemand_fd_ioctl(struct file *filp, unsigned int ioctl,
- unsigned long id)
-{
- struct cachefiles_object *object = filp->private_data;
- struct cachefiles_cache *cache = object->volume->cache;
- struct cachefiles_req *req;
- XA_STATE(xas, &cache->reqs, id);
-
- if (ioctl != CACHEFILES_IOC_READ_COMPLETE)
- return -EINVAL;
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- xa_lock(&cache->reqs);
- req = xas_load(&xas);
- if (!req || req->msg.opcode != CACHEFILES_OP_READ ||
- req->object != object) {
- xa_unlock(&cache->reqs);
- return -EINVAL;
- }
- xas_store(&xas, NULL);
- xa_unlock(&cache->reqs);
-
- trace_cachefiles_ondemand_cread(object, id);
- complete(&req->done);
- return 0;
-}
-
-static const struct file_operations cachefiles_ondemand_fd_fops = {
- .owner = THIS_MODULE,
- .release = cachefiles_ondemand_fd_release,
- .write_iter = cachefiles_ondemand_fd_write_iter,
- .llseek = cachefiles_ondemand_fd_llseek,
- .unlocked_ioctl = cachefiles_ondemand_fd_ioctl,
-};
-
-/*
- * OPEN request Completion (copen)
- * - command: "copen <id>,<cache_size>"
- * <cache_size> indicates the object size if >=0, error code if negative
- */
-int cachefiles_ondemand_copen(struct cachefiles_cache *cache, char *args)
-{
- struct cachefiles_req *req;
- struct fscache_cookie *cookie;
- struct cachefiles_ondemand_info *info;
- char *pid, *psize;
- unsigned long id;
- long size;
- int ret;
- XA_STATE(xas, &cache->reqs, 0);
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- if (!*args) {
- pr_err("Empty id specified\n");
- return -EINVAL;
- }
-
- pid = args;
- psize = strchr(args, ',');
- if (!psize) {
- pr_err("Cache size is not specified\n");
- return -EINVAL;
- }
-
- *psize = 0;
- psize++;
-
- ret = kstrtoul(pid, 0, &id);
- if (ret)
- return ret;
-
- xa_lock(&cache->reqs);
- xas.xa_index = id;
- req = xas_load(&xas);
- if (!req || req->msg.opcode != CACHEFILES_OP_OPEN ||
- !req->object->ondemand->ondemand_id) {
- xa_unlock(&cache->reqs);
- return -EINVAL;
- }
- xas_store(&xas, NULL);
- xa_unlock(&cache->reqs);
-
- info = req->object->ondemand;
- /* fail OPEN request if copen format is invalid */
- ret = kstrtol(psize, 0, &size);
- if (ret) {
- req->error = ret;
- goto out;
- }
-
- /* fail OPEN request if daemon reports an error */
- if (size < 0) {
- if (!IS_ERR_VALUE(size)) {
- req->error = -EINVAL;
- ret = -EINVAL;
- } else {
- req->error = size;
- ret = 0;
- }
- goto out;
- }
-
- spin_lock(&info->lock);
- /*
- * The anonymous fd was closed before copen ? Fail the request.
- *
- * t1 | t2
- * ---------------------------------------------------------
- * cachefiles_ondemand_copen
- * req = xa_erase(&cache->reqs, id)
- * // Anon fd is maliciously closed.
- * cachefiles_ondemand_fd_release
- * xa_lock(&cache->reqs)
- * cachefiles_ondemand_set_object_close(object)
- * xa_unlock(&cache->reqs)
- * cachefiles_ondemand_set_object_open
- * // No one will ever close it again.
- * cachefiles_ondemand_daemon_read
- * cachefiles_ondemand_select_req
- *
- * Get a read req but its fd is already closed. The daemon can't
- * issue a cread ioctl with an closed fd, then hung.
- */
- if (info->ondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED) {
- spin_unlock(&info->lock);
- req->error = -EBADFD;
- goto out;
- }
- cookie = req->object->cookie;
- cookie->object_size = size;
- if (size)
- clear_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags);
- else
- set_bit(FSCACHE_COOKIE_NO_DATA_TO_READ, &cookie->flags);
- trace_cachefiles_ondemand_copen(req->object, id, size);
-
- cachefiles_ondemand_set_object_open(req->object);
- spin_unlock(&info->lock);
- wake_up_all(&cache->daemon_pollwq);
-
-out:
- spin_lock(&info->lock);
- /* Need to set object close to avoid reopen status continuing */
- if (info->ondemand_id == CACHEFILES_ONDEMAND_ID_CLOSED)
- cachefiles_ondemand_set_object_close(req->object);
- spin_unlock(&info->lock);
- complete(&req->done);
- return ret;
-}
-
-int cachefiles_ondemand_restore(struct cachefiles_cache *cache, char *args)
-{
- struct cachefiles_req *req;
-
- XA_STATE(xas, &cache->reqs, 0);
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return -EOPNOTSUPP;
-
- /*
- * Reset the requests to CACHEFILES_REQ_NEW state, so that the
- * requests have been processed halfway before the crash of the
- * user daemon could be reprocessed after the recovery.
- */
- xas_lock(&xas);
- xas_for_each(&xas, req, ULONG_MAX)
- xas_set_mark(&xas, CACHEFILES_REQ_NEW);
- xas_unlock(&xas);
-
- wake_up_all(&cache->daemon_pollwq);
- return 0;
-}
-
-static int cachefiles_ondemand_get_fd(struct cachefiles_req *req,
- struct ondemand_anon_file *anon_file)
-{
- struct cachefiles_object *object;
- struct cachefiles_cache *cache;
- struct cachefiles_open *load;
- u32 object_id;
- int ret;
-
- object = cachefiles_grab_object(req->object,
- cachefiles_obj_get_ondemand_fd);
- cache = object->volume->cache;
-
- ret = xa_alloc_cyclic(&cache->ondemand_ids, &object_id, NULL,
- XA_LIMIT(1, INT_MAX),
- &cache->ondemand_id_next, GFP_KERNEL);
- if (ret < 0)
- goto err;
-
- anon_file->fd = get_unused_fd_flags(O_WRONLY);
- if (anon_file->fd < 0) {
- ret = anon_file->fd;
- goto err_free_id;
- }
-
- anon_file->file = anon_inode_getfile_fmode("[cachefiles]",
- &cachefiles_ondemand_fd_fops, object,
- O_WRONLY, FMODE_PWRITE | FMODE_LSEEK);
- if (IS_ERR(anon_file->file)) {
- ret = PTR_ERR(anon_file->file);
- goto err_put_fd;
- }
-
- spin_lock(&object->ondemand->lock);
- if (object->ondemand->ondemand_id > 0) {
- spin_unlock(&object->ondemand->lock);
- /* Pair with check in cachefiles_ondemand_fd_release(). */
- anon_file->file->private_data = NULL;
- ret = -EEXIST;
- goto err_put_file;
- }
-
- load = (void *)req->msg.data;
- load->fd = anon_file->fd;
- object->ondemand->ondemand_id = object_id;
- spin_unlock(&object->ondemand->lock);
-
- cachefiles_get_unbind_pincount(cache);
- trace_cachefiles_ondemand_open(object, &req->msg, load);
- return 0;
-
-err_put_file:
- fput(anon_file->file);
- anon_file->file = NULL;
-err_put_fd:
- put_unused_fd(anon_file->fd);
- anon_file->fd = ret;
-err_free_id:
- xa_erase(&cache->ondemand_ids, object_id);
-err:
- spin_lock(&object->ondemand->lock);
- /* Avoid marking an opened object as closed. */
- if (object->ondemand->ondemand_id <= 0)
- cachefiles_ondemand_set_object_close(object);
- spin_unlock(&object->ondemand->lock);
- cachefiles_put_object(object, cachefiles_obj_put_ondemand_fd);
- return ret;
-}
-
-static void ondemand_object_worker(struct work_struct *work)
-{
- struct cachefiles_ondemand_info *info =
- container_of(work, struct cachefiles_ondemand_info, ondemand_work);
-
- cachefiles_ondemand_init_object(info->object);
-}
-
-/*
- * If there are any inflight or subsequent READ requests on the
- * closed object, reopen it.
- * Skip read requests whose related object is reopening.
- */
-static struct cachefiles_req *cachefiles_ondemand_select_req(struct xa_state *xas,
- unsigned long xa_max)
-{
- struct cachefiles_req *req;
- struct cachefiles_object *object;
- struct cachefiles_ondemand_info *info;
-
- xas_for_each_marked(xas, req, xa_max, CACHEFILES_REQ_NEW) {
- if (req->msg.opcode != CACHEFILES_OP_READ)
- return req;
- object = req->object;
- info = object->ondemand;
- if (cachefiles_ondemand_object_is_close(object)) {
- cachefiles_ondemand_set_object_reopening(object);
- queue_work(fscache_wq, &info->ondemand_work);
- continue;
- }
- if (cachefiles_ondemand_object_is_reopening(object))
- continue;
- return req;
- }
- return NULL;
-}
-
-static inline bool cachefiles_ondemand_finish_req(struct cachefiles_req *req,
- struct xa_state *xas, int err)
-{
- if (unlikely(!xas || !req))
- return false;
-
- if (xa_cmpxchg(xas->xa, xas->xa_index, req, NULL, 0) != req)
- return false;
-
- req->error = err;
- complete(&req->done);
- return true;
-}
-
-ssize_t cachefiles_ondemand_daemon_read(struct cachefiles_cache *cache,
- char __user *_buffer, size_t buflen)
-{
- struct cachefiles_req *req;
- struct cachefiles_msg *msg;
- size_t n;
- int ret = 0;
- struct ondemand_anon_file anon_file;
- XA_STATE(xas, &cache->reqs, cache->req_id_next);
-
- xa_lock(&cache->reqs);
- /*
- * Cyclically search for a request that has not ever been processed,
- * to prevent requests from being processed repeatedly, and make
- * request distribution fair.
- */
- req = cachefiles_ondemand_select_req(&xas, ULONG_MAX);
- if (!req && cache->req_id_next > 0) {
- xas_set(&xas, 0);
- req = cachefiles_ondemand_select_req(&xas, cache->req_id_next - 1);
- }
- if (!req) {
- xa_unlock(&cache->reqs);
- return 0;
- }
-
- msg = &req->msg;
- n = msg->len;
-
- if (n > buflen) {
- xa_unlock(&cache->reqs);
- return -EMSGSIZE;
- }
-
- xas_clear_mark(&xas, CACHEFILES_REQ_NEW);
- cache->req_id_next = xas.xa_index + 1;
- refcount_inc(&req->ref);
- cachefiles_grab_object(req->object, cachefiles_obj_get_read_req);
- xa_unlock(&cache->reqs);
-
- if (msg->opcode == CACHEFILES_OP_OPEN) {
- ret = cachefiles_ondemand_get_fd(req, &anon_file);
- if (ret)
- goto out;
- }
-
- msg->msg_id = xas.xa_index;
- msg->object_id = req->object->ondemand->ondemand_id;
-
- if (copy_to_user(_buffer, msg, n) != 0)
- ret = -EFAULT;
-
- if (msg->opcode == CACHEFILES_OP_OPEN) {
- if (ret < 0) {
- fput(anon_file.file);
- put_unused_fd(anon_file.fd);
- goto out;
- }
- fd_install(anon_file.fd, anon_file.file);
- }
-out:
- cachefiles_put_object(req->object, cachefiles_obj_put_read_req);
- /* Remove error request and CLOSE request has no reply */
- if (ret || msg->opcode == CACHEFILES_OP_CLOSE)
- cachefiles_ondemand_finish_req(req, &xas, ret);
- cachefiles_req_put(req);
- return ret ? ret : n;
-}
-
-typedef int (*init_req_fn)(struct cachefiles_req *req, void *private);
-
-static int cachefiles_ondemand_send_req(struct cachefiles_object *object,
- enum cachefiles_opcode opcode,
- size_t data_len,
- init_req_fn init_req,
- void *private)
-{
- struct cachefiles_cache *cache = object->volume->cache;
- struct cachefiles_req *req = NULL;
- XA_STATE(xas, &cache->reqs, 0);
- int ret;
-
- if (!test_bit(CACHEFILES_ONDEMAND_MODE, &cache->flags))
- return 0;
-
- if (test_bit(CACHEFILES_DEAD, &cache->flags)) {
- ret = -EIO;
- goto out;
- }
-
- req = kzalloc(sizeof(*req) + data_len, GFP_KERNEL);
- if (!req) {
- ret = -ENOMEM;
- goto out;
- }
-
- refcount_set(&req->ref, 1);
- req->object = object;
- init_completion(&req->done);
- req->msg.opcode = opcode;
- req->msg.len = sizeof(struct cachefiles_msg) + data_len;
-
- ret = init_req(req, private);
- if (ret)
- goto out;
-
- do {
- /*
- * Stop enqueuing the request when daemon is dying. The
- * following two operations need to be atomic as a whole.
- * 1) check cache state, and
- * 2) enqueue request if cache is alive.
- * Otherwise the request may be enqueued after xarray has been
- * flushed, leaving the orphan request never being completed.
- *
- * CPU 1 CPU 2
- * ===== =====
- * test CACHEFILES_DEAD bit
- * set CACHEFILES_DEAD bit
- * flush requests in the xarray
- * enqueue the request
- */
- xas_lock(&xas);
-
- if (test_bit(CACHEFILES_DEAD, &cache->flags) ||
- cachefiles_ondemand_object_is_dropping(object)) {
- xas_unlock(&xas);
- ret = -EIO;
- goto out;
- }
-
- /* coupled with the barrier in cachefiles_flush_reqs() */
- smp_mb();
-
- if (opcode == CACHEFILES_OP_CLOSE &&
- !cachefiles_ondemand_object_is_open(object)) {
- WARN_ON_ONCE(object->ondemand->ondemand_id == 0);
- xas_unlock(&xas);
- ret = -EIO;
- goto out;
- }
-
- /*
- * Cyclically find a free xas to avoid msg_id reuse that would
- * cause the daemon to successfully copen a stale msg_id.
- */
- xas.xa_index = cache->msg_id_next;
- xas_find_marked(&xas, UINT_MAX, XA_FREE_MARK);
- if (xas.xa_node == XAS_RESTART) {
- xas.xa_index = 0;
- xas_find_marked(&xas, cache->msg_id_next - 1, XA_FREE_MARK);
- }
- if (xas.xa_node == XAS_RESTART)
- xas_set_err(&xas, -EBUSY);
-
- xas_store(&xas, req);
- if (xas_valid(&xas)) {
- cache->msg_id_next = xas.xa_index + 1;
- xas_clear_mark(&xas, XA_FREE_MARK);
- xas_set_mark(&xas, CACHEFILES_REQ_NEW);
- }
- xas_unlock(&xas);
- } while (xas_nomem(&xas, GFP_KERNEL));
-
- ret = xas_error(&xas);
- if (ret)
- goto out;
-
- wake_up_all(&cache->daemon_pollwq);
-wait:
- ret = wait_for_completion_killable(&req->done);
- if (!ret) {
- ret = req->error;
- } else {
- ret = -EINTR;
- if (!cachefiles_ondemand_finish_req(req, &xas, ret)) {
- /* Someone will complete it soon. */
- cpu_relax();
- goto wait;
- }
- }
- cachefiles_req_put(req);
- return ret;
-out:
- /* Reset the object to close state in error handling path.
- * If error occurs after creating the anonymous fd,
- * cachefiles_ondemand_fd_release() will set object to close.
- */
- if (opcode == CACHEFILES_OP_OPEN &&
- !cachefiles_ondemand_object_is_dropping(object))
- cachefiles_ondemand_set_object_close(object);
- kfree(req);
- return ret;
-}
-
-static int cachefiles_ondemand_init_open_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
- struct fscache_cookie *cookie = object->cookie;
- struct fscache_volume *volume = object->volume->vcookie;
- struct cachefiles_open *load = (void *)req->msg.data;
- size_t volume_key_size, cookie_key_size;
- void *volume_key, *cookie_key;
-
- /*
- * Volume key is a NUL-terminated string. key[0] stores strlen() of the
- * string, followed by the content of the string (excluding '\0').
- */
- volume_key_size = volume->key[0] + 1;
- volume_key = volume->key + 1;
-
- /* Cookie key is binary data, which is netfs specific. */
- cookie_key_size = cookie->key_len;
- cookie_key = fscache_get_key(cookie);
-
- if (!(object->cookie->advice & FSCACHE_ADV_WANT_CACHE_SIZE)) {
- pr_err("WANT_CACHE_SIZE is needed for on-demand mode\n");
- return -EINVAL;
- }
-
- load->volume_key_size = volume_key_size;
- load->cookie_key_size = cookie_key_size;
- memcpy(load->data, volume_key, volume_key_size);
- memcpy(load->data + volume_key_size, cookie_key, cookie_key_size);
-
- return 0;
-}
-
-static int cachefiles_ondemand_init_close_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
-
- if (!cachefiles_ondemand_object_is_open(object))
- return -ENOENT;
-
- trace_cachefiles_ondemand_close(object, &req->msg);
- return 0;
-}
-
-struct cachefiles_read_ctx {
- loff_t off;
- size_t len;
-};
-
-static int cachefiles_ondemand_init_read_req(struct cachefiles_req *req,
- void *private)
-{
- struct cachefiles_object *object = req->object;
- struct cachefiles_read *load = (void *)req->msg.data;
- struct cachefiles_read_ctx *read_ctx = private;
-
- load->off = read_ctx->off;
- load->len = read_ctx->len;
- trace_cachefiles_ondemand_read(object, &req->msg, load);
- return 0;
-}
-
-int cachefiles_ondemand_init_object(struct cachefiles_object *object)
-{
- struct fscache_cookie *cookie = object->cookie;
- struct fscache_volume *volume = object->volume->vcookie;
- size_t volume_key_size, cookie_key_size, data_len;
-
- if (!object->ondemand)
- return 0;
-
- /*
- * CacheFiles will firstly check the cache file under the root cache
- * directory. If the coherency check failed, it will fallback to
- * creating a new tmpfile as the cache file. Reuse the previously
- * allocated object ID if any.
- */
- if (cachefiles_ondemand_object_is_open(object))
- return 0;
-
- volume_key_size = volume->key[0] + 1;
- cookie_key_size = cookie->key_len;
- data_len = sizeof(struct cachefiles_open) +
- volume_key_size + cookie_key_size;
-
- return cachefiles_ondemand_send_req(object, CACHEFILES_OP_OPEN,
- data_len, cachefiles_ondemand_init_open_req, NULL);
-}
-
-void cachefiles_ondemand_clean_object(struct cachefiles_object *object)
-{
- unsigned long index;
- struct cachefiles_req *req;
- struct cachefiles_cache *cache;
-
- if (!object->ondemand)
- return;
-
- cachefiles_ondemand_send_req(object, CACHEFILES_OP_CLOSE, 0,
- cachefiles_ondemand_init_close_req, NULL);
-
- if (!object->ondemand->ondemand_id)
- return;
-
- /* Cancel all requests for the object that is being dropped. */
- cache = object->volume->cache;
- xa_lock(&cache->reqs);
- cachefiles_ondemand_set_object_dropping(object);
- xa_for_each(&cache->reqs, index, req) {
- if (req->object == object) {
- req->error = -EIO;
- complete(&req->done);
- __xa_erase(&cache->reqs, index);
- }
- }
- xa_unlock(&cache->reqs);
-
- /* Wait for ondemand_object_worker() to finish to avoid UAF. */
- cancel_work_sync(&object->ondemand->ondemand_work);
-}
-
-int cachefiles_ondemand_init_obj_info(struct cachefiles_object *object,
- struct cachefiles_volume *volume)
-{
- if (!cachefiles_in_ondemand_mode(volume->cache))
- return 0;
-
- object->ondemand = kzalloc_obj(struct cachefiles_ondemand_info);
- if (!object->ondemand)
- return -ENOMEM;
-
- object->ondemand->object = object;
- spin_lock_init(&object->ondemand->lock);
- INIT_WORK(&object->ondemand->ondemand_work, ondemand_object_worker);
- return 0;
-}
-
-void cachefiles_ondemand_deinit_obj_info(struct cachefiles_object *object)
-{
- kfree(object->ondemand);
- object->ondemand = NULL;
-}
-
-int cachefiles_ondemand_read(struct cachefiles_object *object,
- loff_t pos, size_t len)
-{
- struct cachefiles_read_ctx read_ctx = {pos, len};
-
- return cachefiles_ondemand_send_req(object, CACHEFILES_OP_READ,
- sizeof(struct cachefiles_read),
- cachefiles_ondemand_init_read_req, &read_ctx);
-}
diff --git a/fs/cachefiles/xattr.c b/fs/cachefiles/xattr.c
index f8ae78b3f7b6d3..25f2a906f9841c 100644
--- a/fs/cachefiles/xattr.c
+++ b/fs/cachefiles/xattr.c
@@ -54,7 +54,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
if (!buf)
return -ENOMEM;
- buf->object_size = cpu_to_be64(object->cookie->object_size);
+ buf->object_size = cpu_to_be64(object->object_size);
buf->zero_point = 0;
buf->type = CACHEFILES_COOKIE_TYPE_DATA;
buf->content = object->content_info;
@@ -77,6 +77,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
trace_cachefiles_vfs_error(object, file_inode(file), ret,
cachefiles_trace_setxattr_error);
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
+ object->object_size,
be64_to_cpup((__be64 *)buf->data),
buf->content,
cachefiles_coherency_set_fail);
@@ -86,6 +87,7 @@ int cachefiles_set_object_xattr(struct cachefiles_object *object)
"Failed to set xattr with error %d", ret);
} else {
trace_cachefiles_coherency(object, file_inode(file)->i_ino,
+ object->object_size,
be64_to_cpup((__be64 *)buf->data),
buf->content,
cachefiles_coherency_set_ok);
@@ -103,9 +105,11 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
{
struct cachefiles_xattr *buf;
struct dentry *dentry = file->f_path.dentry;
+ struct inode *inode = file_inode(file);
unsigned int len = object->cookie->aux_len, tlen;
const void *p = fscache_get_aux(object->cookie);
enum cachefiles_coherency_trace why;
+ unsigned long long obj_size;
ssize_t xlen;
int ret = -ESTALE;
@@ -120,36 +124,41 @@ int cachefiles_check_auxdata(struct cachefiles_object *object, struct file *file
if (xlen != tlen) {
if (xlen < 0) {
ret = xlen;
- trace_cachefiles_vfs_error(object, file_inode(file), xlen,
+ trace_cachefiles_vfs_error(object, inode, xlen,
cachefiles_trace_getxattr_error);
}
if (xlen == -EIO)
cachefiles_io_error_obj(
object,
"Failed to read aux with error %zd", xlen);
- why = cachefiles_coherency_check_xattr;
+ trace_cachefiles_coherency(object, inode->i_ino, 0, 0, 0,
+ cachefiles_coherency_check_xattr);
goto out;
}
+ obj_size = be64_to_cpu(buf->object_size);
if (buf->type != CACHEFILES_COOKIE_TYPE_DATA) {
why = cachefiles_coherency_check_type;
} else if (memcmp(buf->data, p, len) != 0) {
why = cachefiles_coherency_check_aux;
- } else if (be64_to_cpu(buf->object_size) != object->cookie->object_size) {
+ } else if (obj_size != object->cookie->object_size) {
why = cachefiles_coherency_check_objsize;
} else if (buf->content == CACHEFILES_CONTENT_DIRTY) {
// TODO: Begin conflict resolution
pr_warn("Dirty object in cache\n");
why = cachefiles_coherency_check_dirty;
} else {
+ object->content_info = buf->content;
+ object->object_size = obj_size;
+ atomic64_set(&object->read_limit, i_size_read(inode));
why = cachefiles_coherency_check_ok;
ret = 0;
}
-out:
- trace_cachefiles_coherency(object, file_inode(file)->i_ino,
+ trace_cachefiles_coherency(object, inode->i_ino, obj_size,
be64_to_cpup((__be64 *)buf->data),
buf->content, why);
+out:
kfree(buf);
return ret;
}
@@ -163,6 +172,9 @@ int cachefiles_remove_object_xattr(struct cachefiles_cache *cache,
{
int ret;
+ trace_cachefiles_coherency(object, d_inode(dentry)->i_ino, 0, 0, 0,
+ cachefiles_coherency_remove);
+
ret = cachefiles_inject_remove_error();
if (ret == 0) {
ret = mnt_want_write(cache->mnt);
diff --git a/fs/ceph/Kconfig b/fs/ceph/Kconfig
index 3d64a316ca31d3..aa6ccd7794d233 100644
--- a/fs/ceph/Kconfig
+++ b/fs/ceph/Kconfig
@@ -4,6 +4,7 @@ config CEPH_FS
depends on INET
select CEPH_LIB
select NETFS_SUPPORT
+ select NETFS_PGPRIV2
select FS_ENCRYPTION_ALGS if FS_ENCRYPTION
default n
help
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 61bd6d92ff257a..d3d1e32a8a4aee 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -274,7 +274,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
ceph_dec_osd_stopping_blocker(fsc->mdsc);
}
-static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -283,7 +283,8 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
struct ceph_mds_request *req;
struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode->i_sb);
struct ceph_inode_info *ci = ceph_inode(inode);
- ssize_t err = 0;
+ struct iov_iter iter;
+ ssize_t err;
size_t len;
int mode;
@@ -292,8 +293,20 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
__clear_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
- if (subreq->start >= inode->i_size)
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ if (subreq->start >= inode->i_size) {
+ __set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ err = 0;
goto out;
+ }
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
/* We need to fetch the inline data. */
mode = ceph_try_to_choose_auth_mds(inode, CEPH_STAT_CAP_INLINE_DATA);
@@ -302,11 +315,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
err = PTR_ERR(req);
goto out;
}
+
req->r_ino1 = ci->i_vino;
req->r_args.getattr.mask = cpu_to_le32(CEPH_STAT_CAP_INLINE_DATA);
req->r_num_caps = 2;
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
err = ceph_mdsc_do_request(mdsc, NULL, req);
if (err < 0)
goto out;
@@ -316,11 +331,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
if (iinfo->inline_version == CEPH_INLINE_NONE) {
/* The data got uninlined */
ceph_mdsc_put_request(req);
- return false;
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ err = -EAGAIN;
+ goto out;
}
len = min_t(size_t, iinfo->inline_len - subreq->start, subreq->len);
- err = copy_to_iter(iinfo->inline_data + subreq->start, len, &subreq->io_iter);
+ err = copy_to_iter(iinfo->inline_data + subreq->start, len, &iter);
if (err == 0) {
err = -EFAULT;
} else {
@@ -333,26 +350,10 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
subreq->error = err;
trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);
netfs_read_subreq_terminated(subreq);
- return true;
-}
-
-static int ceph_netfs_prepare_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct inode *inode = rreq->inode;
- struct ceph_inode_info *ci = ceph_inode(inode);
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
- u64 objno, objoff;
- u32 xlen;
-
- /* Truncate the extent at the end of the current block */
- ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, subreq->len,
- &objno, &objoff, &xlen);
- rreq->io_streams[0].sreq_max_len = umin(xlen, fsc->mount_options->rsize);
return 0;
}
-static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
+static int ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct inode *inode = rreq->inode;
@@ -361,19 +362,18 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
struct ceph_client *cl = fsc->client;
struct ceph_osd_request *req = NULL;
struct ceph_vino vino = ceph_vino(inode);
- int err;
- u64 len;
+ struct iov_iter iter;
+ u64 objno, objoff, len, off = subreq->start;
+ u32 maxlen;
+ int err = -EIO;
bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD);
- u64 off = subreq->start;
int extent_cnt;
- if (ceph_inode_is_shutdown(inode)) {
- err = -EIO;
- goto out;
- }
+ if (ceph_inode_is_shutdown(inode))
+ return -EIO;
- if (ceph_has_inline_data(ci) && ceph_netfs_issue_op_inline(subreq))
- return;
+ if (ceph_has_inline_data(ci))
+ return ceph_netfs_issue_op_inline(subreq);
// TODO: This rounding here is slightly dodgy. It *should* work, for
// now, as the cache only deals in blocks that are a multiple of
@@ -383,26 +383,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
len = subreq->len;
ceph_fscrypt_adjust_off_and_len(inode, &off, &len);
+ /* Truncate the extent at the end of the current block */
+ ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, len,
+ &objno, &objoff, &maxlen);
+ maxlen = min(maxlen, fsc->mount_options->rsize);
+ len = min(len, maxlen);
+ subreq->len = len;
+
+ /* Grab a slice of read buffer. This may shrink the subreq. */
+ err = netfs_prepare_read_buffer(subreq, INT_MAX);
+ if (err < 0)
+ return err;
+ /* After this point, must fail by termination. */
+
+ /* Create a request. In theory, this may shrink the request again, but
+ * it shouldn't since we calculated the object size above and already
+ * shrank to that, but if it does, we'll just end up doing a short read
+ * and retrying to get the rest.
+ */
+ len = subreq->len;
req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino,
off, &len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ,
CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq,
ci->i_truncate_size, false);
if (IS_ERR(req)) {
err = PTR_ERR(req);
- req = NULL;
- goto out;
+ goto failed_noput;
}
if (sparse) {
extent_cnt = __ceph_sparse_read_ext_count(inode, len);
err = ceph_alloc_sparse_ext_map(&req->r_ops[0], extent_cnt);
if (err)
- goto out;
+ goto failed;
}
doutc(cl, "%llx.%llx pos=%llu orig_len=%zu len=%llu\n",
ceph_vinop(inode), subreq->start, subreq->len, len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset,
+ subreq->len);
+
/*
* FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for
* encrypted inodes. We'd need infrastructure that handles an iov_iter
@@ -421,13 +443,11 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
* ceph_msg_data_cursor_init() triggers BUG_ON() in the case
* if msg->sparse_read_total > msg->data_length.
*/
- subreq->io_iter.count = len;
-
- err = iov_iter_get_pages_alloc2(&subreq->io_iter, &pages, len, &page_off);
+ err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off);
if (err < 0) {
doutc(cl, "%llx.%llx failed to allocate pages, %d\n",
ceph_vinop(inode), err);
- goto out;
+ goto eio;
}
/* should always give us a page-aligned read */
@@ -438,12 +458,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false,
false);
} else {
- osd_req_op_extent_osd_iter(req, 0, &subreq->io_iter);
- }
- if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
- err = -EIO;
- goto out;
+ osd_req_op_extent_osd_iter(req, 0, &iter);
}
+ if (!ceph_inc_osd_stopping_blocker(fsc->mdsc))
+ goto eio;
req->r_callback = finish_netfs_read;
req->r_priv = subreq;
req->r_inode = inode;
@@ -451,19 +469,22 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
ceph_osdc_start_request(req->r_osdc, req);
-out:
ceph_osdc_put_request(req);
- if (err) {
- subreq->error = err;
- netfs_read_subreq_terminated(subreq);
- }
- doutc(cl, "%llx.%llx result %d\n", ceph_vinop(inode), err);
+ doutc(cl, "%llx.%llx result -EIOCBQUEUED\n", ceph_vinop(inode));
+ return 0;
+eio:
+ err = -EIO;
+failed:
+ ceph_osdc_put_request(req);
+failed_noput:
+ subreq->error = err;
+ netfs_read_subreq_terminated(subreq);
+ return 0;
}
static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
{
struct inode *inode = rreq->inode;
- struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
struct ceph_client *cl = ceph_inode_to_client(inode);
int got = 0, want = CEPH_CAP_FILE_CACHE;
struct ceph_netfs_request_data *priv;
@@ -515,7 +536,6 @@ static int ceph_init_request(struct netfs_io_request *rreq, struct file *file)
priv->caps = got;
rreq->netfs_priv = priv;
- rreq->io_streams[0].sreq_max_len = fsc->mount_options->rsize;
out:
if (ret < 0) {
@@ -543,7 +563,6 @@ static void ceph_netfs_free_request(struct netfs_io_request *rreq)
const struct netfs_request_ops ceph_netfs_ops = {
.init_request = ceph_init_request,
.free_request = ceph_netfs_free_request,
- .prepare_read = ceph_netfs_prepare_read,
.issue_read = ceph_netfs_issue_read,
.expand_readahead = ceph_netfs_expand_readahead,
.check_write_begin = ceph_netfs_check_write_begin,
diff --git a/fs/netfs/Kconfig b/fs/netfs/Kconfig
index 7701c037c3283f..d0e7b0971fa3f0 100644
--- a/fs/netfs/Kconfig
+++ b/fs/netfs/Kconfig
@@ -22,6 +22,9 @@ config NETFS_STATS
between CPUs. On the other hand, the stats are very useful for
debugging purposes. Saying 'Y' here is recommended.
+config NETFS_PGPRIV2
+ bool
+
config NETFS_DEBUG
bool "Enable dynamic debugging netfslib and FS-Cache"
depends on NETFS_SUPPORT
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index b43188d64bd87e..421dd0be413b3d 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -3,6 +3,7 @@
netfs-y := \
buffered_read.o \
buffered_write.o \
+ bvecq.o \
direct_read.o \
direct_write.o \
iterator.o \
@@ -11,14 +12,13 @@ netfs-y := \
misc.o \
objects.o \
read_collect.o \
- read_pgpriv2.o \
read_retry.o \
read_single.o \
- rolling_buffer.o \
write_collect.o \
write_issue.o \
write_retry.o
+netfs-$(CONFIG_NETFS_PGPRIV2) += read_pgpriv2.o
netfs-$(CONFIG_NETFS_STATS) += stats.o
netfs-$(CONFIG_FSCACHE) += \
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 24a8a5418e3119..c1ff33dcf536ee 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -54,6 +54,42 @@ static void netfs_rreq_expand(struct netfs_io_request *rreq,
}
}
+/*
+ * Drop the folio refs acquired from the readahead API.
+ */
+static void netfs_bulk_drop_ra_refs(struct netfs_io_request *rreq)
+{
+ struct folio_batch fbatch;
+ struct folio *folio;
+ pgoff_t nr_pages = DIV_ROUND_UP(rreq->len, PAGE_SIZE);
+ pgoff_t first = rreq->start / PAGE_SIZE;
+ XA_STATE(xas, &rreq->mapping->i_pages, first);
+
+ folio_batch_init(&fbatch);
+
+ rcu_read_lock();
+
+ xas_for_each(&xas, folio, first + nr_pages - 1) {
+ if (xas_retry(&xas, folio))
+ continue;
+
+ if (!folio_batch_add(&fbatch, folio))
+ folio_batch_release(&fbatch);
+ }
+
+ rcu_read_unlock();
+ folio_batch_release(&fbatch);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_ra_put_ref);
+ clear_bit_unlock(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
+ wake_up(&rreq->waitq);
+}
+
+static void netfs_maybe_bulk_drop_ra_refs(struct netfs_io_request *rreq)
+{
+ if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ netfs_bulk_drop_ra_refs(rreq);
+}
+
/*
* Begin an operation, and fetch the stored zero point value from the cookie if
* available.
@@ -64,103 +100,98 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
}
/*
- * netfs_prepare_read_iterator - Prepare the subreq iterator for I/O
- * @subreq: The subrequest to be set up
- *
- * Prepare the I/O iterator representing the read buffer on a subrequest for
- * the filesystem to use for I/O (it can be passed directly to a socket). This
- * is intended to be called from the ->issue_read() method once the filesystem
- * has trimmed the request to the size it wants.
- *
- * Returns the limited size if successful and -ENOMEM if insufficient memory
- * available.
- *
- * [!] NOTE: This must be run in the same thread as ->issue_read() was called
- * in as we access the readahead_control struct.
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
*/
-static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq,
- struct readahead_control *ractl)
+static int netfs_prepare_buffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
struct netfs_io_request *rreq = subreq->rreq;
- size_t rsize = subreq->len;
-
- if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER)
- rsize = umin(rsize, rreq->io_streams[0].sreq_max_len);
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ ssize_t extracted;
- if (ractl) {
- /* If we don't have sufficient folios in the rolling buffer,
- * extract a folioq's worth from the readahead region at a time
- * into the buffer. Note that this acquires a ref on each page
- * that we will need to release later - but we don't want to do
- * that until after we've started the I/O.
- */
- struct folio_batch put_batch;
+ _enter("R=%08x[%x] l=%zx s=%u",
+ rreq->debug_id, subreq->debug_index, subreq->len, max_segs);
- folio_batch_init(&put_batch);
- while (rreq->submitted < subreq->start + rsize) {
- ssize_t added;
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ extracted = bvecq_slice(&stream->dispatch_cursor, subreq->len,
+ max_segs, &subreq->nr_segs);
- added = rolling_buffer_load_from_ra(&rreq->buffer, ractl,
- &put_batch);
- if (added < 0)
- return added;
- rreq->submitted += added;
- }
- folio_batch_release(&put_batch);
+ if (extracted < subreq->len) {
+ subreq->len = extracted;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
}
+ stream->buffered -= extracted;
+ stream->issue_from = subreq->start + subreq->len;
+ rreq->submitted = stream->issue_from;
- subreq->len = rsize;
- if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
- rreq->io_streams[0].sreq_max_segs);
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
- if (limit < rsize) {
- subreq->len = limit;
- trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
- }
+/**
+ * netfs_prepare_read_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ *
+ * [!] NOTE: This must be run in the same thread as ->issue_read() was called
+ * in as we access the readahead_control struct if there is one.
+ */
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ switch (subreq->rreq->origin) {
+ case NETFS_READAHEAD:
+ case NETFS_READPAGE:
+ case NETFS_READ_FOR_WRITE:
+ if (subreq->retry_count)
+ return netfs_prepare_buffered_read_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_read_buffer(subreq, max_segs);
+
+ case NETFS_UNBUFFERED_READ:
+ case NETFS_DIO_READ:
+ case NETFS_READ_GAPS:
+ return netfs_prepare_unbuffered_read_buffer(subreq, max_segs);
+ case NETFS_READ_SINGLE:
+ return netfs_prepare_read_single_buffer(subreq, max_segs);
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
}
-
- subreq->io_iter = rreq->buffer.iter;
-
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- rolling_buffer_advance(&rreq->buffer, subreq->len);
- return subreq->len;
}
+EXPORT_SYMBOL(netfs_prepare_read_buffer);
-static enum netfs_io_source netfs_cache_prepare_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq,
- loff_t i_size)
+void netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ)
{
struct netfs_cache_resources *cres = &rreq->cache_resources;
- enum netfs_io_source source;
+ occ->granularity = PAGE_SIZE;
+ if (occ->query_from >= occ->query_to)
+ return;
if (!cres->ops)
- return NETFS_DOWNLOAD_FROM_SERVER;
- source = cres->ops->prepare_read(subreq, i_size);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- return source;
-
+ return;
+ occ->query_from = round_up(occ->query_from, occ->granularity);
+ cres->ops->query_occupancy(cres, occ);
}
/*
- * Issue a read against the cache.
- * - Eats the caller's ref on subreq.
+ * Allocate and prepare a read subrequest.
*/
-static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_IGNORE,
- netfs_cache_read_terminated, subreq);
-}
-
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq)
{
+ struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
+ subreq = netfs_alloc_subrequest(rreq);
+ if (!subreq)
+ return subreq;
+
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
/* We add to the end of the list whilst the collector may be walking
@@ -179,26 +210,53 @@ void netfs_queue_read(struct netfs_io_request *rreq,
}
spin_unlock(&rreq->lock);
+ return subreq;
}
static void netfs_issue_read(struct netfs_io_request *rreq,
struct netfs_io_subrequest *subreq)
{
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ int ret;
+
+ _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
+
switch (subreq->source) {
case NETFS_DOWNLOAD_FROM_SERVER:
- rreq->netfs_ops->issue_read(subreq);
- break;
- case NETFS_READ_FROM_CACHE:
- netfs_read_cache_to_pagecache(rreq, subreq);
- break;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
+ case NETFS_READ_FROM_CACHE: {
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
+
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0)
+ goto fail;
+ return;
+ }
default:
- __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
- subreq->error = 0;
- iov_iter_zero(subreq->len, &subreq->io_iter);
+ WARN_ON_ONCE(1);
+ fallthrough;
+ case NETFS_FILL_WITH_ZEROES:
+ stream->issue_from = subreq->start + subreq->len;
+ stream->buffered -= subreq->len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ if (!stream->buffered)
+ netfs_all_subreqs_queued(rreq);
+ bvecq_zero(&stream->dispatch_cursor, subreq->len);
subreq->transferred = subreq->len;
- netfs_read_subreq_terminated(subreq);
- break;
+ subreq->error = 0;
+ return netfs_read_subreq_terminated(subreq);
}
+
+fail:
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
/*
@@ -206,116 +264,141 @@ static void netfs_issue_read(struct netfs_io_request *rreq,
* slicing up the region to be read according to available cache blocks and
* network rsize.
*/
-static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
- struct readahead_control *ractl)
+static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
{
- unsigned long long start = rreq->start;
- ssize_t size = rreq->len;
+ struct fscache_occupancy _occ = {
+ .query_from = rreq->start,
+ .query_to = rreq->start + rreq->len,
+ .cached_from[0] = 0,
+ .cached_to[0] = 0,
+ .cached_from[1] = ULLONG_MAX,
+ .cached_to[1] = ULLONG_MAX,
+ };
+ struct fscache_occupancy *occ = &_occ;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
int ret = 0;
+ _enter("R=%08x", rreq->debug_id);
+
+ bvecq_pos_set(&stream->dispatch_cursor, &rreq->load_cursor);
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+
do {
struct netfs_io_subrequest *subreq;
- enum netfs_io_source source = NETFS_SOURCE_UNKNOWN;
- ssize_t slice;
+ unsigned long long hole_to, cache_to, stop;
+
+ /* If we don't have any, find out the next couple of data
+ * extents from the cache, containing of following the
+ * specified start offset. Holes have to be fetched from the
+ * server; data regions from the cache.
+ */
+ hole_to = occ->cached_from[0];
+ cache_to = occ->cached_to[0];
+ if (stream->issue_from >= cache_to) {
+ /* Extent exhausted; shuffle down. */
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(occ->cached_from) - 1; i++) {
+ occ->cached_from[i] = occ->cached_from[i + 1];
+ occ->cached_to[i] = occ->cached_to[i + 1];
+ occ->cached_type[i] = occ->cached_type[i + 1];
+ }
+ occ->cached_from[i] = ULLONG_MAX;
+ occ->cached_to[i] = ULLONG_MAX;
+
+ if (occ->cached_from[0] != ULLONG_MAX)
+ continue;
- subreq = netfs_alloc_subrequest(rreq);
+ /* Get new extents */
+ netfs_read_query_cache(rreq, occ);
+ continue;
+ }
+
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
ret = -ENOMEM;
break;
}
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
-
- source = netfs_cache_prepare_read(rreq, subreq, rreq->i_size);
- subreq->source = source;
- if (source == NETFS_DOWNLOAD_FROM_SERVER) {
- unsigned long long zero_point = netfs_read_zero_point(rreq->inode);
- unsigned long long zp = umin(zero_point, rreq->i_size);
- size_t len = subreq->len;
-
- if (unlikely(rreq->origin == NETFS_READ_SINGLE))
- zp = rreq->i_size;
- if (subreq->start >= zp) {
- subreq->source = source = NETFS_FILL_WITH_ZEROES;
- goto fill_with_zeroes;
- }
-
- if (len > zp - subreq->start)
- len = zp - subreq->start;
- if (len == 0) {
- pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
- rreq->debug_id, subreq->debug_index,
- subreq->len, size,
- subreq->start, zero_point, rreq->i_size);
- netfs_cancel_read(subreq, ret);
- break;
+ subreq->start = stream->issue_from;
+ stop = stream->issue_from + stream->buffered;
+
+ unsigned long long zero_point = netfs_read_zero_point(rreq->inode);
+ unsigned long long zlimit = umin(zero_point, rreq->i_size);
+
+ _debug("rsub %llx %llx-%llx", subreq->start, hole_to, cache_to);
+
+ if (stream->issue_from >= hole_to && stream->issue_from < cache_to) {
+ /* Overlap with a cached region, where the cache may
+ * record a block of zeroes.
+ */
+ _debug("cached s=%llx c=%llx l=%zx",
+ stream->issue_from, cache_to, stream->buffered);
+ subreq->len = umin(cache_to - stream->issue_from, stream->buffered);
+ if (occ->cached_type[0] == FSCACHE_EXTENT_ZERO) {
+ subreq->source = NETFS_FILL_WITH_ZEROES;
+ netfs_stat(&netfs_n_rh_zero);
+ } else {
+ subreq->len = round_up(subreq->len, occ->granularity);
+ subreq->source = NETFS_READ_FROM_CACHE;
}
- subreq->len = len;
-
- netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- }
- goto issue;
- }
-
- fill_with_zeroes:
- if (source == NETFS_FILL_WITH_ZEROES) {
+ } else if (subreq->start >= zlimit &&
+ subreq->start < stop) {
+ /* If this range lies beyond the zero-point, that part
+ * can just be cleared locally.
+ */
+ _debug("zero %llx-%llx", subreq->start, stop);
+ subreq->len = stream->buffered;
subreq->source = NETFS_FILL_WITH_ZEROES;
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+ if (rreq->cache_resources.ops)
+ __set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
netfs_stat(&netfs_n_rh_zero);
- goto issue;
- }
-
- if (source == NETFS_READ_FROM_CACHE) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- goto issue;
+ } else {
+ /* Read a cache hole from the server. If any part of
+ * this range lies beyond the zero-point or the EOF,
+ * that part can just be cleared locally.
+ */
+ unsigned long long limit = min3(zlimit, stop, hole_to);
+
+ _debug("limit %llx %llx", rreq->i_size, zero_point);
+ _debug("download %llx-%llx", subreq->start, stop);
+ subreq->len = umin(limit - subreq->start, ULONG_MAX);
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
+ if (rreq->cache_resources.ops)
+ __set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags);
+ netfs_stat(&netfs_n_rh_download);
}
- pr_err("Unexpected read source %u\n", source);
- WARN_ON_ONCE(1);
- netfs_cancel_read(subreq, ret);
- break;
-
- issue:
- slice = netfs_prepare_read_iterator(subreq, ractl);
- if (slice < 0) {
- ret = slice;
+ if (subreq->len == 0) {
+ pr_err("ZERO-LEN READ: R=%08x[%x] l=%zx/%zx s=%llx z=%llx i=%llx",
+ rreq->debug_id, subreq->debug_index,
+ subreq->len, stream->buffered,
+ subreq->start, zero_point, rreq->i_size);
netfs_cancel_read(subreq, ret);
break;
}
- start += slice;
- size -= slice;
- if (size <= 0) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- }
netfs_issue_read(rreq, subreq);
+ netfs_maybe_bulk_drop_ra_refs(rreq);
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
netfs_wait_for_paused_read(rreq);
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ if (unlikely(!netfs_are_all_subreqs_queued(rreq))) {
+ netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
/* Defer error return as we may need to wait for outstanding I/O. */
- cmpxchg(&rreq->error, 0, ret);
+ if (ret < 0)
+ cmpxchg(&rreq->error, 0, ret);
+
+ bvecq_pos_unset(&rreq->load_cursor);
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/**
@@ -336,16 +419,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq,
void netfs_readahead(struct readahead_control *ractl)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct netfs_inode *ictx = netfs_inode(ractl->mapping->host);
unsigned long long start = readahead_pos(ractl);
+ ssize_t added;
size_t size = readahead_length(ractl);
int ret;
+ _enter("");
+
rreq = netfs_alloc_request(ractl->mapping, ractl->file, start, size,
NETFS_READAHEAD);
if (IS_ERR(rreq))
return;
+ stream = &rreq->io_streams[0];
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &rreq->flags);
ret = netfs_begin_cache_read(rreq, ictx);
@@ -358,11 +447,25 @@ void netfs_readahead(struct readahead_control *ractl)
netfs_rreq_expand(rreq, ractl);
- rreq->submitted = rreq->start;
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST) < 0)
+ /* Load the folios to be read into a bvecq chain. Note that this
+ * acquires a ref on each folio that we will need to release later -
+ * but we don't want to do that until after we've started the I/O.
+ */
+ added = bvecq_load_from_ra(&rreq->load_cursor, ractl);
+ if (added < 0) {
+ ret = added;
goto cleanup_free;
- netfs_read_to_pagecache(rreq, ractl);
+ }
+ __set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags);
+
+ rreq->submitted = rreq->start + added;
+ rreq->cleaned_to = rreq->start;
+ rreq->front_folio_order = get_order(rreq->load_cursor.bvecq->bv[0].bv_len);
+ stream->issue_from = rreq->start;
+ stream->buffered = added;
+ netfs_read_to_pagecache(rreq);
+ netfs_maybe_bulk_drop_ra_refs(rreq);
return netfs_put_request(rreq, netfs_rreq_trace_put_return);
cleanup_free:
@@ -371,20 +474,23 @@ void netfs_readahead(struct readahead_control *ractl)
EXPORT_SYMBOL(netfs_readahead);
/*
- * Create a rolling buffer with a single occupying folio.
+ * Create a buffer queue with a single occupying folio.
*/
-static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio,
- unsigned int rollbuf_flags)
+static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)
{
- ssize_t added;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ struct bvecq *bq;
+ size_t fsize = folio_size(folio);
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST) < 0)
+ if (bvecq_buffer_init(&rreq->load_cursor, GFP_KERNEL) < 0)
return -ENOMEM;
- added = rolling_buffer_append(&rreq->buffer, folio, rollbuf_flags);
- if (added < 0)
- return added;
- rreq->submitted = rreq->start + added;
+ bq = rreq->load_cursor.bvecq;
+ bvec_set_folio(&bq->bv[0], folio, fsize, 0);
+ bvecq_filled_to(bq, 1);
+ rreq->submitted = rreq->start + fsize;
+ stream->issue_from = rreq->start;
+ stream->buffered = fsize;
return 0;
}
@@ -394,15 +500,16 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
static int netfs_read_gaps(struct file *file, struct folio *folio)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
struct address_space *mapping = folio->mapping;
struct netfs_group *group = netfs_folio_group(folio);
struct netfs_folio *finfo = netfs_folio_info(folio);
struct netfs_inode *ctx = netfs_inode(mapping->host);
- struct folio *sink = NULL;
- struct bio_vec *bvec;
+ struct bvecq *bq = NULL;
+ struct page *sink = NULL;
unsigned int from = finfo->dirty_offset;
unsigned int to = from + finfo->dirty_len;
- unsigned int off = 0, i = 0;
+ unsigned int off = 0, slot = 0;
size_t flen = folio_size(folio);
size_t nr_bvec = flen / PAGE_SIZE + 2;
size_t part;
@@ -415,6 +522,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
ret = PTR_ERR(rreq);
goto alloc_error;
}
+ stream = &rreq->io_streams[0];
ret = netfs_begin_cache_read(rreq, ctx);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)
@@ -427,35 +535,52 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
* end get copied to, but the middle is discarded.
*/
ret = -ENOMEM;
- bvec = kmalloc_objs(*bvec, nr_bvec);
- if (!bvec)
+ bq = bvecq_alloc_chain(nr_bvec, GFP_KERNEL);
+ if (!bq)
goto discard;
+ rreq->load_cursor.bvecq = bq;
- sink = folio_alloc(GFP_KERNEL, 0);
- if (!sink) {
- kfree(bvec);
+ sink = alloc_page(GFP_KERNEL);
+ if (!sink)
goto discard;
- }
trace_netfs_folio(folio, netfs_folio_trace_read_gaps);
- rreq->direct_bv = bvec;
- rreq->direct_bv_count = nr_bvec;
+ for (struct bvecq *p = bq; p; p = p->next)
+ p->mem_type = BVECQ_MEM_PAGECACHE;
+
if (from > 0) {
- bvec_set_folio(&bvec[i++], folio, from, 0);
+ folio_get(folio);
+ bvec_set_folio(&bq->bv[slot++], folio, from, 0);
off = from;
}
while (off < to) {
- part = min_t(size_t, to - off, PAGE_SIZE);
- bvec_set_folio(&bvec[i++], sink, part, 0);
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ slot = 0;
+ }
+ part = min(to - off, PAGE_SIZE);
+ get_page(sink);
+ bvec_set_page(&bq->bv[slot++], sink, part, 0);
off += part;
}
- if (to < flen)
- bvec_set_folio(&bvec[i++], folio, flen - to, to);
- iov_iter_bvec(&rreq->buffer.iter, ITER_DEST, bvec, i, rreq->len);
+ if (to < flen) {
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ slot = 0;
+ }
+ folio_get(folio);
+ bvec_set_folio(&bq->bv[slot++], folio, flen - to, to);
+ }
+ bvecq_filled_to(bq, slot);
+
rreq->submitted = rreq->start + flen;
+ stream->issue_from = rreq->start;
+ stream->buffered = flen;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
if (ret >= 0) {
@@ -469,13 +594,14 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
folio_mark_uptodate(folio);
}
- if (sink)
- folio_put(sink);
+ put_page(sink);
folio_unlock(folio);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
discard:
+ if (sink)
+ put_page(sink);
netfs_put_failed_request(rreq);
alloc_error:
folio_unlock(folio);
@@ -526,11 +652,12 @@ int netfs_read_folio(struct file *file, struct folio *folio)
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_readpage);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, 0);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto discard;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
+
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
@@ -683,11 +810,11 @@ int netfs_write_begin(struct netfs_inode *ctx,
trace_netfs_read(rreq, pos, len, netfs_read_trace_write_begin);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, 0);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto error_put;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
if (ret < 0)
@@ -748,11 +875,11 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
trace_netfs_read(rreq, start, flen, netfs_read_trace_prefetch_for_write);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio, NETFS_ROLLBUF_PAGECACHE_MARK);
+ ret = netfs_create_singular_buffer(rreq, folio);
if (ret < 0)
goto error_put;
- netfs_read_to_pagecache(rreq, NULL);
+ netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret < 0 ? ret : 0;
diff --git a/fs/netfs/buffered_write.c b/fs/netfs/buffered_write.c
index 2cdb68e6b16fff..8e84ec6d26c621 100644
--- a/fs/netfs/buffered_write.c
+++ b/fs/netfs/buffered_write.c
@@ -54,9 +54,6 @@ void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
i_size = i_size_read(inode);
if (end > i_size) {
i_size_write(inode, end);
-#if IS_ENABLED(CONFIG_FSCACHE)
- fscache_update_cookie(ctx->cache, NULL, &end);
-#endif
gap = SECTOR_SIZE - (i_size & (SECTOR_SIZE - 1));
if (copied > gap) {
@@ -91,44 +88,14 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
struct inode *inode = file_inode(file);
struct address_space *mapping = inode->i_mapping;
struct netfs_inode *ctx = netfs_inode(inode);
- struct writeback_control wbc = {
- .sync_mode = WB_SYNC_NONE,
- .for_sync = true,
- .nr_to_write = LONG_MAX,
- .range_start = iocb->ki_pos,
- .range_end = iocb->ki_pos + iter->count,
- };
- struct netfs_io_request *wreq = NULL;
- struct folio *folio = NULL, *writethrough = NULL;
+ struct folio *folio = NULL;
unsigned int bdp_flags = (iocb->ki_flags & IOCB_NOWAIT) ? BDP_ASYNC : 0;
- ssize_t written = 0, ret, ret2;
+ ssize_t written = 0, ret;
loff_t pos = iocb->ki_pos;
size_t max_chunk = mapping_max_folio_size(mapping);
bool maybe_trouble = false;
- if (unlikely(iocb->ki_flags & (IOCB_DSYNC | IOCB_SYNC))
- ) {
- wbc_attach_fdatawrite_inode(&wbc, mapping->host);
-
- ret = filemap_write_and_wait_range(mapping, pos, pos + iter->count);
- if (ret < 0) {
- wbc_detach_inode(&wbc);
- goto out;
- }
-
- wreq = netfs_begin_writethrough(iocb, iter->count);
- if (IS_ERR(wreq)) {
- wbc_detach_inode(&wbc);
- ret = PTR_ERR(wreq);
- wreq = NULL;
- goto out;
- }
- if (!is_sync_kiocb(iocb))
- wreq->iocb = iocb;
- netfs_stat(&netfs_n_wh_writethrough);
- } else {
- netfs_stat(&netfs_n_wh_buffered_write);
- }
+ netfs_stat(&netfs_n_wh_buffered_write);
do {
enum netfs_folio_trace trace;
@@ -390,15 +357,8 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
pos += copied;
written += copied;
- if (likely(!wreq)) {
- folio_mark_dirty(folio);
- folio_unlock(folio);
- } else {
- netfs_advance_writethrough(wreq, &wbc, folio, copied,
- offset + copied == flen,
- &writethrough);
- /* Folio unlocked */
- }
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
retry:
folio_put(folio);
folio = NULL;
@@ -420,15 +380,6 @@ ssize_t netfs_perform_write(struct kiocb *iocb, struct iov_iter *iter,
ctx->ops->post_modify(inode);
}
- if (unlikely(wreq)) {
- ret2 = netfs_end_writethrough(wreq, &wbc, writethrough);
- wbc_detach_inode(&wbc);
- if (ret2 == -EIOCBQUEUED)
- return ret2;
- if (ret == 0 && ret2 < 0)
- ret = ret2;
- }
-
iocb->ki_pos += written;
_leave(" = %zd [%zd]", written, ret);
return written ? written : ret;
diff --git a/fs/netfs/bvecq.c b/fs/netfs/bvecq.c
new file mode 100644
index 00000000000000..830d5199bab7e1
--- /dev/null
+++ b/fs/netfs/bvecq.c
@@ -0,0 +1,768 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Buffering helpers for bvec queues
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#include <linux/bvecq.h>
+#include "internal.h"
+
+void bvecq_dump(const struct bvecq *bq)
+{
+ int b = 0;
+
+ for (; bq; bq = bvecq_next(bq), b++) {
+ int skipz = 0;
+
+ pr_notice("BQ[%u] %u/%u fp=%llx%s\n",
+ b, bq->nr_slots, bq->max_slots, bq->fpos,
+ bq->discontig ? " discontig" : "");
+ for (int s = 0; s < bq->nr_slots; s++) {
+ const struct bio_vec *bv = &bq->bv[s];
+
+ if (!bv->bv_page && !bv->bv_len && skipz < 2) {
+ skipz = 1;
+ continue;
+ }
+ if (skipz == 1)
+ pr_notice("BQ[%u:00-%02u] ...\n", b, s - 1);
+ skipz = 2;
+ pr_notice("BQ[%u:%02u] %10lx %04x %04x %u\n",
+ b, s,
+ bv->bv_page ? page_to_pfn(bv->bv_page) : 0,
+ bv->bv_offset, bv->bv_len,
+ bv->bv_page ? page_count(bv->bv_page) : 0);
+ }
+ }
+}
+EXPORT_SYMBOL(bvecq_dump);
+
+/**
+ * bvecq_alloc_one - Allocate a single bvecq node with unpopulated slots
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ *
+ * Allocate a single bvecq node and initialise the header. A number of inline
+ * slots are also allocated, rounded up to fit after the header in a power-of-2
+ * slab object of up to 512 bytes (up to 29 slots on a 64-bit cpu). The slot
+ * array is not initialised.
+ *
+ * Return: The node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp)
+{
+ struct bvecq *bq;
+ const size_t max_size = 512;
+ const size_t max_slots = (max_size - sizeof(*bq)) / sizeof(bq->__bv[0]);
+ size_t part = umin(nr_slots, max_slots);
+ size_t size = roundup_pow_of_two(struct_size(bq, __bv, part));
+
+ bq = kmalloc(size, gfp & ~GFP_ZONEMASK);
+ if (bq) {
+ *bq = (struct bvecq) {
+ .ref = REFCOUNT_INIT(1),
+ .bv = bq->__bv,
+ .inline_bv = true,
+ .max_slots = (size - sizeof(*bq)) / sizeof(bq->__bv[0]),
+ };
+ netfs_stat(&netfs_n_bvecq);
+ }
+ return bq;
+}
+EXPORT_SYMBOL(bvecq_alloc_one);
+
+/**
+ * bvecq_alloc_chain - Allocate an unpopulated bvecq chain
+ * @nr_slots: Number of slots to allocate
+ * @gfp: The allocation constraints.
+ *
+ * Allocate a chain of bvecq nodes providing at least the requested cumulative
+ * number of slots.
+ *
+ * Return: The first node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp)
+{
+ struct bvecq *head = NULL, *tail = NULL;
+
+ _enter("%zu", nr_slots);
+
+ for (;;) {
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(nr_slots, gfp);
+ if (!bq)
+ goto oom;
+
+ if (tail)
+ bvecq_append(tail, bq);
+ else
+ head = bq;
+ tail = bq;
+ if (tail->max_slots >= nr_slots)
+ break;
+ nr_slots -= tail->max_slots;
+ }
+
+ return head;
+oom:
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_chain);
+
+/**
+ * bvecq_alloc_buffer2 - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @pre_slots: Number of preamble slots to set aside
+ * @gfp: The allocation constraints.
+ *
+ * Allocate a chain of bvecq nodes and populate the slots with sufficient pages
+ * to provide at least the requested amount of space, leaving the first
+ * @pre_slots slots unset. The pre-slots must all fit into the the first
+ * bvecq.
+ *
+ * The pages allocated may be compound pages larger than PAGE_SIZE and thus
+ * occupy fewer slots. The pages have their refcounts set to 1 and can be
+ * passed to MSG_SPLICE_PAGES.
+ *
+ * Return: The first node pointer or NULL on allocation failure.
+ */
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp)
+{
+ struct bvecq *head = NULL, *p = NULL;
+ size_t nr_per_bq = BVECQ_STD_SLOTS;
+ size_t count = pre_slots + DIV_ROUND_UP(size, PAGE_SIZE);
+
+ _enter("%zx,%zx,%u", size, count, pre_slots);
+
+ if (WARN_ON_ONCE(pre_slots > nr_per_bq))
+ return NULL;
+
+ head = bvecq_alloc_chain(count, gfp);
+ if (!head)
+ return NULL;
+
+ p = head;
+ do {
+ struct page **pages;
+ size_t unused, want, got, slot;
+
+ if (!count)
+ break;
+ if (WARN_ON_ONCE(!p))
+ goto oom;
+
+ if (p->nr_slots == 0) {
+ /* Need to clear pre slots and pages[], so just clear all. */
+ memset(p->bv, 0, p->max_slots * sizeof(p->bv[0]));
+ p->mem_type = BVECQ_MEM_ALLOCED;
+ p->nr_slots = pre_slots;
+ count -= pre_slots;
+ pre_slots = 0;
+ }
+
+ if (p->nr_slots >= p->max_slots) {
+ p = p->next;
+ continue;
+ }
+ unused = p->max_slots - p->nr_slots;
+
+ pages = (struct page **)&p->bv[p->max_slots];
+ pages -= unused;
+
+ want = min(count, unused);
+ got = alloc_pages_bulk(gfp, want, pages);
+ if (!got)
+ goto oom;
+
+ slot = p->nr_slots;
+ for (int i = 0; i < got; i++) {
+ set_page_count(pages[i], 1);
+ bvec_set_page(&p->bv[slot++], pages[i], PAGE_SIZE, 0);
+ }
+
+ bvecq_filled_to(p, slot);
+ count -= got;
+ } while (count > 0);
+
+ return head;
+oom:
+ bvecq_put(head);
+ return NULL;
+}
+EXPORT_SYMBOL(bvecq_alloc_buffer2);
+
+/*
+ * Free the page pointed to by a slot as necessary.
+ */
+static void bvecq_free_slot(struct bvecq *bq, unsigned int slot)
+{
+ struct page *page = bq->bv[slot].bv_page;
+
+ if (!page)
+ return;
+
+ switch (bq->mem_type) {
+ case BVECQ_MEM_EXTERNAL:
+ break;
+ case BVECQ_MEM_PAGECACHE:
+ put_page(page);
+ break;
+ case BVECQ_MEM_GUP:
+ unpin_user_page(page);
+ break;
+ case BVECQ_MEM_ALLOCED:
+ __free_pages(page, compound_order(page));
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+}
+
+/**
+ * bvecq_put - Put a ref on a bvec queue
+ * @bq: The start of the folio queue to free
+ *
+ * Put the ref(s) on the nodes in a bvec queue, freeing up the node and the
+ * page fragments it points to as the refcounts become zero.
+ */
+void bvecq_put(struct bvecq *bq)
+{
+ struct bvecq *next;
+
+ for (; bq; bq = next) {
+ if (!refcount_dec_and_test(&bq->ref))
+ break;
+ for (int slot = 0; slot < bq->nr_slots; slot++)
+ bvecq_free_slot(bq, slot);
+ next = bq->next;
+ netfs_stat_d(&netfs_n_bvecq);
+ kfree(bq);
+ }
+}
+EXPORT_SYMBOL(bvecq_put);
+
+/**
+ * bvecq_expand_buffer - Allocate buffer space into a bvec queue
+ * @_buffer: Pointer to the bvecq chain to expand (may point to a NULL; updated).
+ * @_cur_size: Current size of the buffer (updated).
+ * @size: Target size of the buffer.
+ * @gfp: The allocation constraints.
+ *
+ * Append extra pages to a buffer to increase its capacity to the @size
+ * specified. If the current tail has space, but is not of the
+ * BVECQ_MEM_ALLOCED memory type, a separate bvecq will be allocated to hold
+ * the new memory.
+ */
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp)
+{
+ struct bvecq *tail = *_buffer;
+
+ size = round_up(size, PAGE_SIZE);
+ if (tail)
+ while (tail->next)
+ tail = tail->next;
+
+ while (*_cur_size < size) {
+ struct page *page;
+ size_t need = size - *_cur_size;
+ int order = 0;
+
+ if (!tail || bvecq_is_full(tail) || tail->mem_type != BVECQ_MEM_ALLOCED) {
+ struct bvecq *p;
+
+ p = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp);
+ if (!p)
+ return -ENOMEM;
+ if (tail)
+ bvecq_append(tail, p);
+ else
+ *_buffer = p;
+ tail = p;
+ p->mem_type = BVECQ_MEM_ALLOCED;
+ }
+
+ if (need > PAGE_SIZE)
+ order = umin(ilog2(need) - PAGE_SHIFT, MAX_PAGECACHE_ORDER);
+
+ page = alloc_pages(gfp | __GFP_COMP, order);
+ if (!page && order > 0) {
+ page = alloc_pages(gfp | __GFP_COMP, 0);
+ order = 0;
+ }
+ if (!page)
+ return -ENOMEM;
+
+ bvec_set_page(&tail->bv[tail->nr_slots], page, PAGE_SIZE << order, 0);
+ *_cur_size += PAGE_SIZE << order;
+ bvecq_filled_to(tail, tail->nr_slots + 1);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_expand_buffer);
+
+/**
+ * bvecq_shorten_buffer - Shorten a bvec queue buffer
+ * @bq: The start of the buffer to shorten
+ * @slot: The slot to start from
+ * @size: The size to retain
+ *
+ * Shorten the content of a bvec queue down to the minimum number of slots,
+ * starting at the specified slot, to retain the specified size.
+ *
+ * Return: 0 if successful; -EMSGSIZE if there is insufficient content.
+ */
+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size)
+{
+ struct bvecq *next;
+
+ /* Skip through the segments we want to keep. */
+ for (; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ if (size < bq->bv[slot].bv_len)
+ goto found;
+ size -= bq->bv[slot].bv_len;
+ }
+ slot = 0;
+ }
+ if (WARN_ON_ONCE(size > 0))
+ return -EMSGSIZE;
+ return 0;
+
+found:
+ /* Shorten any partial entry and clean the rest of this bvecq. */
+ if (size > 0) {
+ bq->bv[slot].bv_len = size;
+ slot++;
+ }
+ for (int i = slot; i < bq->nr_slots; i++)
+ bvecq_free_slot(bq, i);
+ bq->nr_slots = slot;
+
+ /* Free the queue tail. */
+ next = bq->next;
+ bq->next = NULL;
+ bvecq_put(next);
+ return 0;
+}
+EXPORT_SYMBOL(bvecq_shorten_buffer);
+
+/**
+ * bvecq_buffer_init - Initialise a buffer and set position
+ * @pos: The position to point at the new buffer.
+ * @gfp: The allocation constraints.
+ *
+ * Initialise a rolling buffer. We allocate an unpopulated bvecq node to so
+ * that the pointers can be independently driven by the producer and the
+ * consumer.
+ *
+ * Return 0 if successful; -ENOMEM on allocation failure.
+ */
+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp)
+{
+ struct bvecq *bq;
+
+ bq = bvecq_alloc_one(BVECQ_STD_SLOTS, gfp);
+ if (!bq)
+ return -ENOMEM;
+
+ pos->bvecq = bq; /* Comes with a ref. */
+ pos->slot = 0;
+ pos->offset = 0;
+ return 0;
+}
+
+/**
+ * bvecq_buffer_append - Append a new bvecq node to a buffer
+ * @pos: The position of the last node.
+ * @bq: The buffer to add.
+ *
+ * Add a new node on to the buffer chain at the specified position, either
+ * because the previous one is full or because we have a discontiguity to
+ * contend with, and update @pos to point to it.
+ */
+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq)
+{
+ struct bvecq *head = pos->bvecq;
+
+ pos->bvecq = bvecq_get(bq);
+ pos->slot = 0;
+ pos->offset = 0;
+
+ /* [!] NOTE: After we set head->next, the consumer is at liberty to
+ * immediately delete the old head.
+ */
+ bvecq_append(head, bq);
+ bvecq_put(head);
+}
+
+/**
+ * bvecq_pos_advance - Advance a bvecq position
+ * @pos: The position to advance.
+ * @amount: The amount of bytes to advance by.
+ *
+ * Advance the specified bvecq position by @amount bytes. @pos is updated and
+ * bvecq ref counts may have been manipulated. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ */
+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+ unsigned int slot = pos->slot;
+ size_t offset = pos->offset;
+
+ while (amount) {
+ size_t part;
+
+ if (!bvecq_acquire_slot(bq, slot)) {
+ next = bvecq_next(bq);
+ if (!next) {
+ WARN_ON_ONCE(amount > 0);
+ break;
+ }
+ bq = next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ part = bq->bv[slot].bv_len - offset;
+
+ if (part > amount) {
+ offset += amount;
+ break;
+ }
+ amount -= part;
+ offset = 0;
+ slot++;
+ }
+
+ pos->slot = slot;
+ pos->offset = offset;
+ bvecq_pos_move(pos, bq);
+}
+
+/*
+ * Clear part of the memory pointed to by a bio_vec.
+ */
+static void bvec_zero(const struct bio_vec *bv, size_t offset, size_t len)
+{
+ struct page *page = bv->bv_page;
+
+ offset += bv->bv_offset;
+
+ page += offset / PAGE_SIZE;
+ offset = offset % PAGE_SIZE;
+
+ while (len) {
+ size_t part = umin(len, PAGE_SIZE - offset);
+ char *p = kmap_local_page(page);
+
+ memset(p + offset, 0, part);
+ kunmap_local(p);
+
+ len -= part;
+ offset = 0;
+ page++;
+ }
+}
+
+/**
+ * bvecq_zero - Clear memory starting at the bvecq position.
+ * @pos: The position in the bvecq chain to start clearing.
+ * @amount: The number of bytes to clear.
+ *
+ * Clear memory fragments pointed to by a bvec queue. @pos is updated and
+ * bvecq ref counts may have been manipulated. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * Return: The number of bytes cleared.
+ */
+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+ unsigned int slot = pos->slot;
+ ssize_t cleared = 0;
+ size_t offset = pos->offset;
+
+ while (amount) {
+ const struct bio_vec *bv;
+ size_t part;
+
+ if (!bvecq_acquire_slot(bq, slot)) {
+ next = bvecq_next(bq);
+ if (!next) {
+ WARN_ON_ONCE(amount > 0);
+ break;
+ }
+ bq = next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bq->bv[slot];
+ if (offset >= bv->bv_len) {
+ slot++;
+ offset = 0;
+ continue;
+ }
+
+ part = min(bv->bv_len - offset, amount);
+ bvec_zero(bv, offset, part);
+ cleared += part;
+ offset += part;
+ amount -= part;
+ }
+
+ pos->slot = slot;
+ pos->offset = offset;
+ bvecq_pos_move(pos, bq);
+ return cleared;
+}
+
+/**
+ * bvecq_slice - Find a slice of a bvecq queue
+ * @pos: The position to start at.
+ * @max_size: The maximum size of the slice (or ULONG_MAX).
+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).
+ * @_nr_slots: Where to put the number of slots (updated).
+ *
+ * Determine the size and number of slots that can be obtained the next slice
+ * of bvec queue up to the maximum size and slot count specified. The slice is
+ * also limited if a discontiguity is found.
+ *
+ * @pos is updated to the end of the slice. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * Return: The number of bytes in the slice.
+ */
+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, unsigned int *_nr_slots)
+{
+ struct bvecq *bq, *next;
+ unsigned int slot = pos->slot, nslots = 0;
+ size_t size = 0, offset = pos->offset;
+
+ bq = pos->bvecq;
+ for (;;) {
+ for (; slot < bvecq_nr_slots_acquire(bq); slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+
+ if (offset < bvec->bv_len && bvec->bv_page) {
+ size_t part = umin(bvec->bv_len - offset, max_size);
+
+ size += part;
+ offset += part;
+ max_size -= part;
+ nslots++;
+ if (!max_size || nslots >= max_slots)
+ goto out;
+ }
+ offset = 0;
+ }
+
+ /* pos->bvecq isn't allowed to go NULL as the queue may get
+ * extended and we would lose our place.
+ */
+ next = bvecq_next(bq);
+ if (!next)
+ break;
+ slot = 0;
+ bq = next;
+ if (bq->discontig && size > 0)
+ break;
+ }
+
+out:
+ *_nr_slots = nslots;
+ if (slot == bvecq_nr_slots_acquire(bq)) {
+ next = bvecq_next(bq);
+ if (next) {
+ bq = next;
+ slot = 0;
+ offset = 0;
+ }
+ }
+ bvecq_pos_move(pos, bq);
+ pos->slot = slot;
+ pos->offset = offset;
+ return size;
+}
+
+/**
+ * bvecq_extract - Extract a slice of a bvecq queue into a new bvecq queue
+ * @pos: The position to start at.
+ * @max_size: The maximum size of the slice (or ULONG_MAX).
+ * @max_slots: The maximum number of slots in the slice (or INT_MAX).
+ * @to: Where to put the extraction bvecq chain head (updated).
+ *
+ * Allocate a new bvecq and extract into it memory fragments from a slice of
+ * bvec queue, starting at @pos. The slice is also limited if a discontiguity
+ * is found. No refs are taken on the page.
+ *
+ * @pos is updated to the end of the slice. If the position hits the end of
+ * the queue, then it is left pointing beyond the last slot of the last bvecq
+ * so that it doesn't break the chain.
+ *
+ * If successful, *@to is set to point to the head of the newly allocated chain
+ * and the caller inherits a ref to it.
+ *
+ * Return: The number of bytes extracted; -ENOMEM on allocation failure or -EIO
+ * if no slots were available to extract.
+ */
+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, struct bvecq **to)
+{
+ struct bvecq_pos tmp_pos;
+ struct bvecq *src, *dst = NULL, *next;
+ unsigned int slot = pos->slot, dslot = 0, nslots;
+ ssize_t extracted = 0;
+ size_t offset = pos->offset, amount;
+
+ *to = NULL;
+ if (WARN_ON_ONCE(!max_slots))
+ max_slots = INT_MAX;
+
+ bvecq_pos_set(&tmp_pos, pos);
+ amount = bvecq_slice(&tmp_pos, max_size, max_slots, &nslots);
+ bvecq_pos_unset(&tmp_pos);
+ if (nslots == 0)
+ return -EIO;
+
+ dst = bvecq_alloc_chain(nslots, GFP_NOFS);
+ if (!dst)
+ return -ENOMEM;
+ *to = dst;
+ max_slots = nslots;
+ nslots = 0;
+
+ /* Transcribe the slots */
+ src = pos->bvecq;
+ for (;;) {
+ for (; slot < bvecq_nr_slots_acquire(src); slot++) {
+ const struct bio_vec *sv = &src->bv[slot];
+ struct bio_vec *dv = &dst->bv[dslot];
+
+ _debug("EXTR BQ=%x[%x] off=%zx am=%zx p=%lx",
+ src->priv, slot, offset, amount, page_to_pfn(sv->bv_page));
+
+ if (offset < sv->bv_len && sv->bv_page) {
+ size_t part = umin(sv->bv_len - offset, amount);
+
+ bvec_set_page(dv, sv->bv_page, part,
+ sv->bv_offset + offset);
+ extracted += part;
+ amount -= part;
+ offset += part;
+ trace_netfs_bv_slot(dst, dslot);
+ dslot++;
+ nslots++;
+ if (dslot >= dst->max_slots) {
+ bvecq_filled_to(dst, dslot);
+ dst = dst->next;
+ dslot = 0;
+ }
+ if (nslots >= max_slots)
+ goto out;
+ if (amount == 0)
+ goto out;
+ }
+ offset = 0;
+ }
+
+ /* pos->bvecq isn't allowed to go NULL as the queue may get
+ * extended and we would lose our place.
+ */
+ next = bvecq_next(src);
+ if (!next)
+ break;
+ slot = 0;
+ src = next;
+ if (src->discontig && extracted > 0)
+ break;
+ }
+
+out:
+ if (dst)
+ bvecq_filled_to(dst, dslot);
+ if (slot == bvecq_nr_slots_acquire(src)) {
+ next = bvecq_next(src);
+ if (next) {
+ src = next;
+ slot = 0;
+ offset = 0;
+ }
+ }
+ bvecq_pos_move(pos, src);
+ pos->slot = slot;
+ pos->offset = offset;
+ return extracted;
+}
+
+/**
+ * bvecq_load_from_ra - Allocate a bvecq chain and load from readahead
+ * @pos: Blank position object to attach the new chain to.
+ * @ractl: The readahead control context.
+ *
+ * Decant the set of folios to be read from the readahead context into a bvecq
+ * chain. Each folio occupies one bio_vec element.
+ *
+ * Return: Amount of data loaded or -ENOMEM on allocation failure.
+ */
+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl)
+{
+ XA_STATE(xas, &ractl->mapping->i_pages, ractl->_index);
+ struct folio *folio;
+ struct bvecq *bq;
+ unsigned int slot = 0;
+ size_t loaded = 0;
+
+ bq = bvecq_alloc_chain(ractl->_nr_folios, GFP_NOFS);
+ if (!bq)
+ return -ENOMEM;
+
+ pos->bvecq = bq;
+ pos->slot = 0;
+ pos->offset = 0;
+
+ rcu_read_lock();
+
+ xas_for_each(&xas, folio, ractl->_index + ractl->_nr_pages - 1) {
+ size_t len;
+
+ if (xas_retry(&xas, folio))
+ continue;
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+
+ len = folio_size(folio);
+ bvec_set_folio(&bq->bv[slot], folio, len, 0);
+ loaded += len;
+ slot++;
+ trace_netfs_folio(folio, netfs_folio_trace_read);
+
+ if (slot >= bq->max_slots) {
+ bvecq_filled_to(bq, slot);
+ bq = bq->next;
+ if (!bq)
+ break;
+ slot = 0;
+ }
+ }
+
+ rcu_read_unlock();
+
+ if (bq)
+ bvecq_filled_to(bq, slot);
+
+ ractl->_index += ractl->_nr_pages;
+ ractl->_nr_pages = 0;
+ return loaded;
+}
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 6a8fb0d55e040e..743a2921cdd9ff 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -16,29 +16,30 @@
#include <linux/netfs.h>
#include "internal.h"
-static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
struct netfs_io_request *rreq = subreq->rreq;
- size_t rsize;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+ size_t len;
- rsize = umin(subreq->len, rreq->io_streams[0].sreq_max_len);
- subreq->len = rsize;
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
- if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
- rreq->io_streams[0].sreq_max_segs);
-
- if (limit < rsize) {
- subreq->len = limit;
- trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
- }
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
}
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
+ stream->buffered -= subreq->len;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
- subreq->io_iter = rreq->buffer.iter;
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- iov_iter_advance(&rreq->buffer.iter, subreq->len);
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(rreq);
+ return 0;
}
/*
@@ -47,15 +48,16 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
*/
static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
{
- unsigned long long start = rreq->start;
- ssize_t size = rreq->len;
- int ret;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+
+ bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &rreq->load_cursor);
do {
struct netfs_io_subrequest *subreq;
- ssize_t slice;
+ int ret;
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq) {
/* Stash the error in the request if there's not
* already an error set.
@@ -65,44 +67,32 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = size;
-
- netfs_queue_read(rreq, subreq);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0) {
- netfs_cancel_read(subreq, ret);
- break;
- }
- }
- netfs_prepare_dio_read_iterator(subreq);
- slice = subreq->len;
- size -= slice;
- start += slice;
- rreq->submitted += slice;
- if (size <= 0) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ stream->buffered -= subreq->len;
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ break;
}
- rreq->netfs_ops->issue_read(subreq);
-
if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags))
netfs_wait_for_paused_read(rreq);
if (test_bit(NETFS_RREQ_FAILED, &rreq->flags))
break;
cond_resched();
- } while (size > 0);
+ } while (stream->buffered > 0);
- if (unlikely(size > 0)) {
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ if (unlikely(stream->buffered > 0)) {
+ netfs_all_subreqs_queued(rreq);
netfs_wake_collector(rreq);
}
+
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
/*
@@ -153,6 +143,7 @@ static ssize_t netfs_unbuffered_read(struct netfs_io_request *rreq, bool sync)
ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter)
{
struct netfs_io_request *rreq;
+ struct netfs_io_stream *stream;
ssize_t ret;
size_t orig_count = iov_iter_count(iter);
bool sync = is_sync_kiocb(iocb);
@@ -177,29 +168,25 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
netfs_stat(&netfs_n_rh_dio_read);
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_dio_read);
+ stream = &rreq->io_streams[0];
+
/* If this is an async op, we have to keep track of the destination
* buffer for ourselves as the caller's iterator will be trashed when
* we return.
*
- * In such a case, extract an iterator to represent as much of the the
- * output buffer as we can manage. Note that the extraction might not
- * be able to allocate a sufficiently large bvec array and may shorten
- * the request.
+ * Extract a buffer queue to represent as much of the output buffer as
+ * we can manage. The fragments are extracted into a bvecq which will
+ * have sufficient nodes allocated to hold all the data, though this
+ * may end up truncated if ENOMEM is encountered.
*/
- if (user_backed_iter(iter)) {
- ret = netfs_extract_user_iter(iter, rreq->len, &rreq->buffer.iter, 0);
- if (ret < 0)
- goto error_put;
- rreq->direct_bv = (struct bio_vec *)rreq->buffer.iter.bvec;
- rreq->direct_bv_count = ret;
- rreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
- rreq->len = iov_iter_count(&rreq->buffer.iter);
- } else {
- rreq->buffer.iter = *iter;
- rreq->len = orig_count;
- rreq->direct_bv_unpin = false;
- iov_iter_advance(iter, orig_count);
- }
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX, iocb->ki_pos,
+ &rreq->load_cursor.bvecq, 0);
+ if (ret < 0)
+ goto error_put;
+
+ rreq->len = ret;
+ stream->buffered = ret;
+ stream->issue_from = rreq->start;
// TODO: Set up bounce buffer if needed
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index c16fbad286a177..e5140bdf2a2465 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -9,6 +9,34 @@
#include <linux/uio.h>
#include "internal.h"
+/*
+ * Prepare the buffer for an unbuffered/DIO write.
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(subreq->rreq);
+ return 0;
+}
+
/*
* Perform the cleanup rituals after an unbuffered write is complete.
*/
@@ -73,7 +101,11 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
spin_unlock(&wreq->lock);
wreq->transferred += subreq->transferred;
- iov_iter_advance(&wreq->buffer.iter, subreq->transferred);
+ if (subreq->transferred < subreq->len) {
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
+ }
stream->collected_to = subreq->start + subreq->transferred;
wreq->collected_to = stream->collected_to;
@@ -81,6 +113,7 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq,
trace_netfs_collect_stream(wreq, stream);
trace_netfs_collect_state(wreq, wreq->collected_to, 0);
+ /* TODO: Progressively clean up wreq->direct_bq */
}
/*
@@ -99,53 +132,44 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
_enter("%llx", wreq->len);
+ stream->issue_from = wreq->start;
+ stream->buffered = wreq->len;
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
+ bvecq_pos_set(&wreq->collect_cursor, &stream->dispatch_cursor);
+
if (wreq->origin == NETFS_DIO_WRITE)
inode_dio_begin(wreq->inode);
- stream->collected_to = wreq->start;
-
for (;;) {
bool retry = false;
if (!subreq) {
- netfs_prepare_write(wreq, stream, wreq->start + wreq->transferred);
- subreq = stream->construct;
- stream->construct = NULL;
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq)
+ return -ENOMEM;
}
- /* Check if (re-)preparation failed. */
- if (unlikely(test_bit(NETFS_SREQ_FAILED, &subreq->flags))) {
- netfs_write_subrequest_terminated(subreq, subreq->error);
- wreq->error = subreq->error;
- break;
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ netfs_write_subrequest_terminated(subreq, ret);
}
- iov_iter_truncate(&subreq->io_iter, wreq->len - wreq->transferred);
- if (!iov_iter_count(&subreq->io_iter))
- break;
-
- subreq->len = netfs_limit_iter(&subreq->io_iter, 0,
- stream->sreq_max_len,
- stream->sreq_max_segs);
- iov_iter_truncate(&subreq->io_iter, subreq->len);
- stream->submit_extendable_to = subreq->len;
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
-
- /* Async, need to wait. */
- netfs_wait_for_in_progress_stream(wreq, stream);
-
- if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ ret = netfs_wait_for_in_progress_subreq(wreq, subreq);
+ if (ret < 0) {
+ if (ret != -EAGAIN) {
+ /* Don't need to lock here as the collection is
+ * done in this thread.
+ */
+ list_del_init(&subreq->rreq_link);
+ ret = subreq->error;
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
+ subreq = NULL;
+ goto failed;
+ }
retry = true;
- } else if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) {
- ret = subreq->error;
- wreq->error = ret;
- netfs_see_subrequest(subreq, netfs_sreq_trace_see_failed);
- subreq = NULL;
- break;
}
- ret = 0;
if (!retry) {
netfs_unbuffered_write_collect(wreq, stream, subreq);
@@ -160,15 +184,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
continue;
}
- /* We need to retry the last subrequest, so first reset the
- * iterator, taking into account what, if anything, we managed
- * to transfer.
+ /* We need to retry the last subrequest, so first wind back the
+ * buffer position.
*/
subreq->error = -EAGAIN;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos);
+
+ stream->issue_from -= subreq->len - subreq->transferred;
+ stream->buffered += subreq->len - subreq->transferred;
if (subreq->transferred > 0) {
- iov_iter_advance(&wreq->buffer.iter, subreq->transferred);
- wreq->transferred += subreq->transferred;
+ wreq->transferred += subreq->transferred;
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred);
}
if (stream->source == NETFS_UPLOAD_TO_SERVER &&
@@ -177,24 +207,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq)
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
__clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
- subreq->io_iter = wreq->buffer.iter;
- subreq->start = wreq->start + wreq->transferred;
- subreq->len = wreq->len - wreq->transferred;
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
subreq->transferred = 0;
subreq->retry_count += 1;
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- if (stream->prepare_write)
- stream->prepare_write(subreq);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
netfs_stat(&netfs_n_wh_retry_write_subreq);
}
+failed:
+ bvecq_pos_unset(&stream->dispatch_cursor);
netfs_unbuffered_write_done(wreq);
_leave(" = %d", ret);
return ret;
@@ -213,12 +240,12 @@ static void netfs_unbuffered_write_async(struct work_struct *work)
* encrypted file. This can also be used for direct I/O writes.
*/
ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *iter,
- struct netfs_group *netfs_group)
+ struct netfs_group *netfs_group)
{
struct netfs_io_request *wreq;
unsigned long long start = iocb->ki_pos;
unsigned long long end = start + iov_iter_count(iter);
- ssize_t ret, n;
+ ssize_t ret;
size_t len = iov_iter_count(iter);
bool async = !is_sync_kiocb(iocb);
@@ -238,6 +265,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
if (IS_ERR(wreq))
return PTR_ERR(wreq);
+ wreq->len = iov_iter_count(iter);
wreq->io_streams[0].avail = true;
trace_netfs_write(wreq, (iocb->ki_flags & IOCB_DIRECT ?
netfs_write_trace_dio_write :
@@ -248,33 +276,21 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* we have to save the source buffer as the iterator is only
* good until we return. In such a case, extract an iterator
* to represent as much of the the output buffer as we can
- * manage. Note that the extraction might not be able to
- * allocate a sufficiently large bvec array and may shorten the
- * request.
+ * manage. Note that the extraction may shorten the request.
*/
- if (user_backed_iter(iter)) {
- n = netfs_extract_user_iter(iter, len, &wreq->buffer.iter, 0);
- if (n < 0) {
- ret = n;
- goto error_put;
- }
- wreq->direct_bv = (struct bio_vec *)wreq->buffer.iter.bvec;
- wreq->direct_bv_count = n;
- wreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
- } else {
- /* If this is a kernel-generated async DIO request,
- * assume that any resources the iterator points to
- * (eg. a bio_vec array) will persist till the end of
- * the op.
- */
- wreq->buffer.iter = *iter;
- }
+ ssize_t n = netfs_extract_iter(iter, len, INT_MAX, iocb->ki_pos,
+ &wreq->load_cursor.bvecq, 0);
- wreq->len = iov_iter_count(&wreq->buffer.iter);
+ if (n < 0) {
+ ret = n;
+ goto error_put;
+ }
+ wreq->len = n;
+ _debug("dio-write %zx/%zx %u/%u",
+ n, len, wreq->load_cursor.bvecq->nr_slots,
+ wreq->load_cursor.bvecq->max_slots);
}
- __set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
-
/* Copy the data into the bounce buffer and encrypt it. */
// TODO
diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c
index 37f05b4d34699d..a04cd96cd09ac9 100644
--- a/fs/netfs/fscache_io.c
+++ b/fs/netfs/fscache_io.c
@@ -79,7 +79,7 @@ static int fscache_begin_operation(struct netfs_cache_resources *cres,
cres->ops = NULL;
cres->cache_priv = cookie;
cres->cache_priv2 = NULL;
- cres->debug_id = cookie->debug_id;
+ cres->cookie_id = cookie->debug_id;
cres->inval_counter = cookie->inval_counter;
if (!fscache_begin_cookie_access(cookie, why)) {
@@ -239,7 +239,7 @@ void __fscache_write_to_cache(struct fscache_cookie *cookie,
fscache_access_io_write) < 0)
goto abandon_free;
- ret = cres->ops->prepare_write(cres, &start, &len, len, i_size, false);
+ ret = cres->ops->prepare_write_old(cres, &start, &len, len, i_size, false);
if (ret < 0)
goto abandon_end;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index d889caa401dc24..79695b59e5cd57 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -7,7 +7,6 @@
#include <linux/slab.h>
#include <linux/seq_file.h>
-#include <linux/folio_queue.h>
#include <linux/netfs.h>
#include <linux/fscache.h>
#include <linux/fscache-cache.h>
@@ -23,8 +22,9 @@
/*
* buffered_read.c
*/
-void netfs_queue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq);
+void netfs_read_query_cache(struct netfs_io_request *rreq,
+ struct fscache_occupancy *occ);
+struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq);
void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
int netfs_prefetch_for_write(struct file *file, struct folio *folio,
size_t offset, size_t len);
@@ -35,6 +35,18 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode,
loff_t pos, size_t copied);
+/*
+ * direct_read.c
+ */
+int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
+/*
+ * direct_write.c
+ */
+int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* main.c
*/
@@ -67,17 +79,19 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}
/*
* misc.c
*/
-struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq,
- enum netfs_folioq_trace trace);
-void netfs_reset_iter(struct netfs_io_subrequest *subreq);
+struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq,
+ enum netfs_bvecq_trace trace);
void netfs_wake_collector(struct netfs_io_request *rreq);
void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq);
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq);
void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,
struct netfs_io_stream *stream);
ssize_t netfs_wait_for_read(struct netfs_io_request *rreq);
ssize_t netfs_wait_for_write(struct netfs_io_request *rreq);
void netfs_wait_for_paused_read(struct netfs_io_request *rreq);
void netfs_wait_for_paused_write(struct netfs_io_request *rreq);
+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq);
/*
* objects.c
@@ -116,16 +130,53 @@ void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error);
/*
* read_pgpriv2.c
*/
+#ifdef CONFIG_NETFS_PGPRIV2
+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio);
void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq);
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq);
+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
+{
+ return test_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags);
+}
+#else
+static inline int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ return -EIO;
+}
+static inline void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *folio)
+{
+}
+static inline void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
+{
+}
+static inline bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)
+{
+ return true;
+}
+static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq)
+{
+ return false;
+}
+#endif
/*
* read_retry.c
*/
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq);
void netfs_retry_reads(struct netfs_io_request *rreq);
void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq);
+/*
+ * read_single.c
+ */
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* stats.c
*/
@@ -153,7 +204,6 @@ extern atomic_t netfs_n_rh_write_zskip;
extern atomic_t netfs_n_rh_retry_read_req;
extern atomic_t netfs_n_rh_retry_read_subreq;
extern atomic_t netfs_n_wh_buffered_write;
-extern atomic_t netfs_n_wh_writethrough;
extern atomic_t netfs_n_wh_dio_write;
extern atomic_t netfs_n_wh_writepages;
extern atomic_t netfs_n_wh_copy_to_cache;
@@ -168,7 +218,7 @@ extern atomic_t netfs_n_wh_retry_write_req;
extern atomic_t netfs_n_wh_retry_write_subreq;
extern atomic_t netfs_n_wb_lock_skip;
extern atomic_t netfs_n_wb_lock_wait;
-extern atomic_t netfs_n_folioq;
+extern atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v);
@@ -197,31 +247,19 @@ void netfs_write_collection_worker(struct work_struct *work);
/*
* write_issue.c
*/
+struct netfs_writethrough;
struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct file *file,
loff_t start,
enum netfs_io_origin origin);
-void netfs_prepare_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- loff_t start);
-void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq,
- struct iov_iter *source);
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream);
-size_t netfs_advance_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- loff_t start, size_t len, bool to_eof);
-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len);
-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *folio, size_t copied, bool to_page_end,
- struct folio **writethrough_cache);
-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *writethrough_cache);
+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream);
/*
* write_retry.c
*/
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
void netfs_retry_writes(struct netfs_io_request *wreq);
/*
@@ -316,6 +354,25 @@ static inline bool netfs_check_subreq_in_progress(const struct netfs_io_subreque
return test_bit_acquire(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
}
+/*
+ * Indicate that we've generated and queued all the subrequests we're going to.
+ */
+static inline void netfs_all_subreqs_queued(struct netfs_io_request *rreq)
+{
+ smp_wmb(); /* Write lists before ALL_QUEUED. */
+ set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_all_queued);
+}
+
+/*
+ * Query if all subrequests are queued.
+ */
+static inline bool netfs_are_all_subreqs_queued(const struct netfs_io_request *rreq)
+{
+ /* Read lists after ALL_QUEUED. */
+ return test_bit_acquire(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
+}
+
/*
* fscache-cache.c
*/
diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c
index b375567e0520ed..7dea84c40e5bfc 100644
--- a/fs/netfs/iterator.c
+++ b/fs/netfs/iterator.c
@@ -14,295 +14,143 @@
#include "internal.h"
/**
- * netfs_extract_user_iter - Extract the pages from a user iterator into a bvec
+ * netfs_extract_iter - Extract virtually contiguous pages from an iterator into a bvecq
* @orig: The original iterator
- * @orig_len: The amount of iterator to copy
- * @new: The iterator to be set up
+ * @max_len: Maximum number of bytes to extract
+ * @max_pages: Maximum number of pages to extract
+ * @fpos: Starting file position to label the bvecq with
+ * @_bvecq_head: Where to cache the bvec queue
* @extraction_flags: Flags to qualify the request
*
- * Extract the page fragments from the given amount of the source iterator and
- * build up a second iterator that refers to all of those bits. This allows
- * the original iterator to be disposed of.
+ * Extract virtually contiguous page fragments from the source iterator up to
+ * the given maxima and build bvec queue that refers to all of those bits.
+ * This allows the original iterator to disposed of.
*
* @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA be
* allowed on the pages extracted.
*
- * On success, the number of elements in the bvec is returned, the original
- * iterator will have been advanced by the amount extracted.
+ * On success or partial success, the amount of data in the bvec is returned,
+ * the original iterator will have been advanced by the amount extracted.
*
- * The iov_iter_extract_mode() function should be used to query how cleanup
- * should be performed.
+ * If an error occurs and no pages are extracted, an error will be returned and
+ * any allocated bvecq will be freed. If there is no data to be extracted (or
+ * @max_len or @max_pages are zero), a single empty bvecq will be returned.
+ *
+ * The bvecq segments are marked with indications on how to get clean up the
+ * extracted fragments.
*/
-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,
- struct iov_iter *new,
- iov_iter_extraction_t extraction_flags)
+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
+ unsigned long long fpos, struct bvecq **_bvecq_head,
+ iov_iter_extraction_t extraction_flags)
{
- struct bio_vec *bv = NULL;
- struct page **pages;
- unsigned int cur_npages;
- unsigned int max_pages;
- unsigned int npages = 0;
- unsigned int i;
+ struct bvecq *bq_tail = NULL, *bq;
ssize_t ret = 0;
- size_t count = orig_len, offset, len;
- size_t bv_size, pg_size;
-
- if (WARN_ON_ONCE(!iter_is_ubuf(orig) && !iter_is_iovec(orig)))
- return -EIO;
-
- max_pages = iov_iter_npages(orig, INT_MAX);
- bv_size = array_size(max_pages, sizeof(*bv));
- bv = kvmalloc(bv_size, GFP_KERNEL);
- if (!bv)
- return -ENOMEM;
-
- /* Put the page list at the end of the bvec list storage. bvec
- * elements are larger than page pointers, so as long as we work
- * 0->last, we should be fine.
- */
- pg_size = array_size(max_pages, sizeof(*pages));
- pages = (void *)bv + bv_size - pg_size;
-
- while (count && npages < max_pages) {
- ret = iov_iter_extract_pages(orig, &pages, count,
- max_pages - npages, extraction_flags,
- &offset);
- if (unlikely(ret <= 0)) {
- ret = ret ?: -EIO;
- break;
- }
-
- if (WARN(ret > count,
- "%s: extract_pages overrun %zd > %zu bytes\n",
- __func__, ret, count)) {
- ret = -EIO;
- break;
- }
-
- cur_npages = DIV_ROUND_UP(offset + ret, PAGE_SIZE);
- if (WARN(cur_npages > max_pages - npages,
- "%s: extract_pages overrun %u > %u pages\n",
- __func__, npages + cur_npages, max_pages)) {
- ret = -EIO;
- break;
- }
-
- count -= ret;
- ret += offset;
-
- for (i = 0; i < cur_npages; i++) {
- len = ret > PAGE_SIZE ? PAGE_SIZE : ret;
- bvec_set_page(bv + npages + i, *pages++, len - offset, offset);
- ret -= len;
- offset = 0;
- }
-
- npages += cur_npages;
- }
-
- /* Note: Don't try to clean up after EIO. Either we got no pages, so
- * nothing to clean up, or we got a buffer overrun, memory corruption
- * and can't trust the stuff in the buffer (a WARN was emitted).
- */
-
- if (ret < 0 && (ret == -ENOMEM || npages == 0)) {
- for (i = 0; i < npages; i++)
- unpin_user_page(bv[i].bv_page);
- kvfree(bv);
- return ret;
- }
-
- iov_iter_bvec(new, orig->data_source, bv, npages, orig_len - count);
- return npages;
-}
-EXPORT_SYMBOL_GPL(netfs_extract_user_iter);
-
-/*
- * Select the span of a bvec iterator we're going to use. Limit it by both maximum
- * size and maximum number of segments. Returns the size of the span in bytes.
- */
-static size_t netfs_limit_bvec(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct bio_vec *bvecs = iter->bvec;
- unsigned int nbv = iter->nr_segs, ix = 0, nsegs = 0;
- size_t len, span = 0, n = iter->count;
- size_t skip = iter->iov_offset + start_offset;
-
- if (WARN_ON(!iov_iter_is_bvec(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
-
- while (n && ix < nbv && skip) {
- len = bvecs[ix].bv_len;
- if (skip < len)
- break;
- skip -= len;
- n -= len;
- ix++;
- }
+ size_t extracted = 0;
- while (n && ix < nbv) {
- len = min3(n, bvecs[ix].bv_len - skip, max_size);
- span += len;
- nsegs++;
- ix++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- skip = 0;
- n -= len;
- }
+ _enter("{%u,%zx},%zx", orig->iter_type, orig->count, max_len);
- return min(span, max_size);
-}
+ *_bvecq_head = NULL;
+ if (max_len > orig->count)
+ max_len = orig->count;
+ if (WARN_ON_ONCE(!max_len || !max_pages))
+ goto alloc_empty;
-/*
- * Select the span of a kvec iterator we're going to use. Limit it by both
- * maximum size and maximum number of segments. Returns the size of the span
- * in bytes.
- */
-static size_t netfs_limit_kvec(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct kvec *kvecs = iter->kvec;
- unsigned int nkv = iter->nr_segs, ix = 0, nsegs = 0;
- size_t len, span = 0, n = iter->count;
- size_t skip = iter->iov_offset + start_offset;
+ max_pages = iov_iter_npages(orig, max_pages);
+ if (!max_pages)
+ goto alloc_empty;
- if (WARN_ON(!iov_iter_is_kvec(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
-
- while (n && ix < nkv && skip) {
- len = kvecs[ix].iov_len;
- if (skip < len)
- break;
- skip -= len;
- n -= len;
- ix++;
- }
-
- while (n && ix < nkv) {
- len = min3(n, kvecs[ix].iov_len - skip, max_size);
- span += len;
- nsegs++;
- ix++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- skip = 0;
- n -= len;
- }
-
- return min(span, max_size);
-}
-
-/*
- * Select the span of an xarray iterator we're going to use. Limit it by both
- * maximum size and maximum number of segments. It is assumed that segments
- * can be larger than a page in size, provided they're physically contiguous.
- * Returns the size of the span in bytes.
- */
-static size_t netfs_limit_xarray(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- struct folio *folio;
- unsigned int nsegs = 0;
- loff_t pos = iter->xarray_start + iter->iov_offset;
- pgoff_t index = pos / PAGE_SIZE;
- size_t span = 0, n = iter->count;
-
- XA_STATE(xas, iter->xarray, index);
-
- if (WARN_ON(!iov_iter_is_xarray(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
- max_size = min(max_size, n - start_offset);
-
- rcu_read_lock();
- xas_for_each(&xas, folio, ULONG_MAX) {
- size_t offset, flen, len;
- if (xas_retry(&xas, folio))
- continue;
- if (WARN_ON(xa_is_value(folio)))
- break;
- if (WARN_ON(folio_test_hugetlb(folio)))
- break;
-
- flen = folio_size(folio);
- offset = offset_in_folio(folio, pos);
- len = min(max_size, flen - offset);
- span += len;
- nsegs++;
- if (span >= max_size || nsegs >= max_segs)
- break;
- }
-
- rcu_read_unlock();
- return min(span, max_size);
-}
-
-/*
- * Select the span of a folio queue iterator we're going to use. Limit it by
- * both maximum size and maximum number of segments. Returns the size of the
- * span in bytes.
- */
-static size_t netfs_limit_folioq(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int nsegs = 0;
- unsigned int slot = iter->folioq_slot;
- size_t span = 0, n = iter->count;
-
- if (WARN_ON(!iov_iter_is_folioq(iter)) ||
- WARN_ON(start_offset > n) ||
- n == 0)
- return 0;
- max_size = umin(max_size, n - start_offset);
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- }
-
- start_offset += iter->iov_offset;
do {
- size_t flen = folioq_folio_size(folioq, slot);
-
- if (start_offset < flen) {
- span += flen - start_offset;
- nsegs++;
- start_offset = 0;
- } else {
- start_offset -= flen;
- }
- if (span >= max_size || nsegs >= max_segs)
+ bq = bvecq_alloc_one(max_pages, GFP_NOFS);
+ if (!bq) {
+ ret = -ENOMEM;
break;
-
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
}
- } while (folioq);
+ if (user_backed_iter(orig))
+ bq->mem_type = iov_iter_extract_will_pin(orig) ?
+ BVECQ_MEM_GUP : BVECQ_MEM_PAGECACHE;
+ bq->fpos = fpos + extracted;
+
+ if (bq_tail)
+ bvecq_append(bq_tail, bq);
+ else
+ *_bvecq_head = bq;
+ bq_tail = bq;
+
+ if (max_len == 0)
+ break;
- return umin(span, max_size);
-}
+ struct bio_vec *bv = bq->bv;
+ unsigned int slot = 0;
+ do {
+ struct page **pages;
+ ssize_t got;
+ size_t offset;
+ size_t space = bq->max_slots - slot;
+ size_t bv_size = array_size(bq->max_slots, sizeof(*bv));
+ size_t pg_size = array_size(space, sizeof(*pages));
+
+ /* Put the page list at the end of the bvec list
+ * storage. bvec elements are larger than page
+ * pointers, so as long as we work 0->last, we should
+ * be fine.
+ */
+ pages = (void *)bv + bv_size - pg_size;
+
+ got = iov_iter_extract_pages(orig, &pages, max_len,
+ min(space, max_pages),
+ extraction_flags, &offset);
+ if (got < 0) {
+ ret = got;
+ goto out;
+ }
+
+ if (got == 0) {
+ pr_err("extract_pages gave nothing from %zx, %zx\n",
+ extracted, max_len);
+ ret = -EIO;
+ goto out;
+ }
+
+ if (WARN(got > max_len,
+ "%s: extract_pages overrun %zx > %zx bytes\n",
+ __func__, got, max_len)) {
+ ret = -EIO;
+ goto out;
+ }
+
+ extracted += got;
+ max_len -= got;
+
+ do {
+ size_t len = umin(got, PAGE_SIZE - offset);
+
+ BUG_ON(slot >= bq->max_slots);
+
+ bvec_set_page(&bq->bv[slot], *pages++, len, offset);
+ slot++;
+ max_pages--;
+ got -= len;
+ offset = 0;
+ } while (got > 0);
+
+ bvecq_filled_to(bq, slot);
+ } while (max_len > 0 && max_pages > 0 && !bvecq_is_full(bq));
+
+ } while (max_len > 0 && max_pages > 0);
+
+out:
+ if (extracted || ret == 0)
+ return extracted;
+ bvecq_put(*_bvecq_head);
+ *_bvecq_head = NULL;
+ return ret;
+
+alloc_empty:
+ bq = bvecq_alloc_one(1, GFP_NOFS);
+ if (!bq)
+ return -ENOMEM;
+ *_bvecq_head = bq;
+ return 0;
-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs)
-{
- if (iov_iter_is_folioq(iter))
- return netfs_limit_folioq(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_bvec(iter))
- return netfs_limit_bvec(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_xarray(iter))
- return netfs_limit_xarray(iter, start_offset, max_size, max_segs);
- if (iov_iter_is_kvec(iter))
- return netfs_limit_kvec(iter, start_offset, max_size, max_segs);
- BUG();
}
-EXPORT_SYMBOL(netfs_limit_iter);
+EXPORT_SYMBOL_GPL(netfs_extract_iter);
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 73da6c9f5777cf..6a5a6b1b10a34b 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -43,7 +43,6 @@ static const char *netfs_origins[nr__netfs_io_origin] = {
[NETFS_DIO_READ] = "DR",
[NETFS_WRITEBACK] = "WB",
[NETFS_WRITEBACK_SINGLE] = "W1",
- [NETFS_WRITETHROUGH] = "WT",
[NETFS_UNBUFFERED_WRITE] = "UW",
[NETFS_DIO_WRITE] = "DW",
[NETFS_PGPRIV2_COPY_TO_CACHE] = "2C",
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index 5d554512ed23a2..0af45204fabc80 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -8,120 +8,6 @@
#include <linux/swap.h>
#include "internal.h"
-/**
- * netfs_alloc_folioq_buffer - Allocate buffer space into a folio queue
- * @mapping: Address space to set on the folio (or NULL).
- * @_buffer: Pointer to the folio queue to add to (may point to a NULL; updated).
- * @_cur_size: Current size of the buffer (updated).
- * @size: Target size of the buffer.
- * @gfp: The allocation constraints.
- */
-int netfs_alloc_folioq_buffer(struct address_space *mapping,
- struct folio_queue **_buffer,
- size_t *_cur_size, ssize_t size, gfp_t gfp)
-{
- struct folio_queue *tail = *_buffer, *p;
-
- size = round_up(size, PAGE_SIZE);
- if (*_cur_size >= size)
- return 0;
-
- if (tail)
- while (tail->next)
- tail = tail->next;
-
- do {
- struct folio *folio;
- int order = 0, slot;
-
- if (!tail || folioq_full(tail)) {
- p = netfs_folioq_alloc(0, GFP_NOFS, netfs_trace_folioq_alloc_buffer);
- if (!p)
- return -ENOMEM;
- if (tail) {
- tail->next = p;
- p->prev = tail;
- } else {
- *_buffer = p;
- }
- tail = p;
- }
-
- if (size - *_cur_size > PAGE_SIZE)
- order = umin(ilog2(size - *_cur_size) - PAGE_SHIFT,
- MAX_PAGECACHE_ORDER);
-
- folio = folio_alloc(gfp, order);
- if (!folio && order > 0)
- folio = folio_alloc(gfp, 0);
- if (!folio)
- return -ENOMEM;
-
- folio->mapping = mapping;
- folio->index = *_cur_size / PAGE_SIZE;
- trace_netfs_folio(folio, netfs_folio_trace_alloc_buffer);
- slot = folioq_append_mark(tail, folio);
- *_cur_size += folioq_folio_size(tail, slot);
- } while (*_cur_size < size);
-
- return 0;
-}
-EXPORT_SYMBOL(netfs_alloc_folioq_buffer);
-
-/**
- * netfs_free_folioq_buffer - Free a folio queue.
- * @fq: The start of the folio queue to free
- *
- * Free up a chain of folio_queues and, if marked, the marked folios they point
- * to.
- */
-void netfs_free_folioq_buffer(struct folio_queue *fq)
-{
- struct folio_queue *next;
- struct folio_batch fbatch;
-
- folio_batch_init(&fbatch);
-
- for (; fq; fq = next) {
- for (int slot = 0; slot < folioq_count(fq); slot++) {
- struct folio *folio = folioq_folio(fq, slot);
-
- if (!folio ||
- !folioq_is_marked(fq, slot))
- continue;
-
- trace_netfs_folio(folio, netfs_folio_trace_put);
- if (folio_batch_add(&fbatch, folio))
- folio_batch_release(&fbatch);
- }
-
- netfs_stat_d(&netfs_n_folioq);
- next = fq->next;
- kfree(fq);
- }
-
- folio_batch_release(&fbatch);
-}
-EXPORT_SYMBOL(netfs_free_folioq_buffer);
-
-/*
- * Reset the subrequest iterator to refer just to the region remaining to be
- * read. The iterator may or may not have been advanced by socket ops or
- * extraction ops to an extent that may or may not match the amount actually
- * read.
- */
-void netfs_reset_iter(struct netfs_io_subrequest *subreq)
-{
- struct iov_iter *io_iter = &subreq->io_iter;
- size_t remain = subreq->len - subreq->transferred;
-
- if (io_iter->count > remain)
- iov_iter_advance(io_iter, io_iter->count - remain);
- else if (io_iter->count < remain)
- iov_iter_revert(io_iter, remain - io_iter->count);
- iov_iter_truncate(&subreq->io_iter, remain);
-}
-
/**
* netfs_dirty_folio - Mark folio dirty and pin a cache object for writeback
* @mapping: The mapping the folio belongs to.
@@ -364,6 +250,37 @@ void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq)
netfs_wake_collector(rreq);
}
+/*
+ * Wait for a subrequest to come to completion.
+ */
+int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq,
+ struct netfs_io_subrequest *subreq)
+{
+ if (netfs_check_subreq_in_progress(subreq)) {
+ DEFINE_WAIT(myself);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_wait_quiesce);
+ for (;;) {
+ prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE);
+
+ if (!netfs_check_subreq_in_progress(subreq))
+ break;
+
+ trace_netfs_sreq(subreq, netfs_sreq_trace_wait_for);
+ schedule();
+ }
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_waited_quiesce);
+ finish_wait(&rreq->waitq, &myself);
+ }
+
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
+ return -EAGAIN;
+ if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
+ return subreq->error;
+ return 0;
+}
+
/*
* Wait for all outstanding I/O in a stream to quiesce.
*/
@@ -424,7 +341,7 @@ static int netfs_collect_in_app(struct netfs_io_request *rreq,
need_collect = true;
break;
}
- if (subreq || !test_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags))
+ if (subreq || !netfs_are_all_subreqs_queued(rreq))
done = false;
}
@@ -494,7 +411,7 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq,
case NETFS_UNBUFFERED_WRITE:
break;
default:
- if (rreq->submitted < rreq->len) {
+ if (rreq->transferred < rreq->len) {
trace_netfs_failure(rreq, NULL, ret, netfs_fail_short_read);
ret = -EIO;
}
@@ -563,3 +480,22 @@ void netfs_wait_for_paused_write(struct netfs_io_request *rreq)
{
return netfs_wait_for_pause(rreq, netfs_write_collection);
}
+
+/*
+ * Wait for the readahead-acquired refs to be put.
+ */
+void netfs_wait_for_put_ra_refs(struct netfs_io_request *rreq)
+{
+ DEFINE_WAIT(myself);
+
+ for (;;) {
+ trace_netfs_rreq(rreq, netfs_rreq_trace_wait_put_ra_refs);
+ prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE);
+ if (!test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ break;
+ schedule();
+ }
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_waited_put_ra_refs);
+ finish_wait(&rreq->waitq, &myself);
+}
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index b8c4918d3dcdab..d4a95a462576f4 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -46,8 +46,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
rreq->i_size = i_size_read(inode);
rreq->debug_id = atomic_inc_return(&debug_ids);
rreq->wsize = INT_MAX;
- rreq->io_streams[0].sreq_max_len = ULONG_MAX;
- rreq->io_streams[0].sreq_max_segs = 0;
spin_lock_init(&rreq->lock);
INIT_LIST_HEAD(&rreq->io_streams[0].subrequests);
INIT_LIST_HEAD(&rreq->io_streams[1].subrequests);
@@ -119,7 +117,6 @@ static void netfs_free_request_rcu(struct rcu_head *rcu)
static void netfs_deinit_request(struct netfs_io_request *rreq)
{
struct netfs_inode *ictx = netfs_inode(rreq->inode);
- unsigned int i;
trace_netfs_rreq(rreq, netfs_rreq_trace_free);
@@ -134,16 +131,12 @@ static void netfs_deinit_request(struct netfs_io_request *rreq)
rreq->netfs_ops->free_request(rreq);
if (rreq->cache_resources.ops)
rreq->cache_resources.ops->end_operation(&rreq->cache_resources);
- if (rreq->direct_bv) {
- for (i = 0; i < rreq->direct_bv_count; i++) {
- if (rreq->direct_bv[i].bv_page) {
- if (rreq->direct_bv_unpin)
- unpin_user_page(rreq->direct_bv[i].bv_page);
- }
- }
- kvfree(rreq->direct_bv);
- }
- rolling_buffer_clear(&rreq->buffer);
+ bvecq_pos_unset(&rreq->load_cursor);
+ bvecq_pos_unset(&rreq->collect_cursor);
+ bvecq_pos_unset(&rreq->retry_cursor);
+ bvecq_put(rreq->spare);
+ for (int i = 0; i < NR_IO_STREAMS; i++)
+ bvecq_pos_unset(&rreq->io_streams[i].dispatch_cursor);
if (atomic_dec_and_test(&ictx->io_count))
wake_up_var(&ictx->io_count);
@@ -234,8 +227,11 @@ static void netfs_free_subrequest(struct netfs_io_subrequest *subreq)
struct netfs_io_request *rreq = subreq->rreq;
trace_netfs_sreq(subreq, netfs_sreq_trace_free);
+ WARN_ON_ONCE(!list_empty(&subreq->rreq_link));
if (rreq->netfs_ops->free_subrequest)
rreq->netfs_ops->free_subrequest(subreq);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ bvecq_pos_unset(&subreq->content);
mempool_free(subreq, rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool);
netfs_stat_d(&netfs_n_rh_sreq);
netfs_put_request(rreq, netfs_rreq_trace_put_subreq);
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 23660a59012464..5899ba022fabc2 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -19,19 +19,25 @@
#define MADE_PROGRESS 0x04 /* Made progress cleaning up a stream or the folio set */
#define BUFFERED 0x08 /* The pagecache needs cleaning up */
#define NEED_RETRY 0x10 /* A front op requests retrying */
-#define COPY_TO_CACHE 0x40 /* Need to copy subrequest to cache */
-#define ABANDON_SREQ 0x80 /* Need to abandon untransferred part of subrequest */
+#define COPY_TO_CACHE 0x20 /* Need to copy subrequest to cache */
+#define ABANDON_SREQ 0x40 /* Need to abandon untransferred part of subrequest */
+#define ABANDON_RREQ 0x80 /* Need to abandon the rest of a request */
/*
* Clear the unread part of an I/O request.
*/
static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
{
- netfs_reset_iter(subreq);
- WARN_ON_ONCE(subreq->len - subreq->transferred != iov_iter_count(&subreq->io_iter));
- iov_iter_zero(iov_iter_count(&subreq->io_iter), &subreq->io_iter);
+ struct iov_iter iter;
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+ iov_iter_advance(&iter, subreq->transferred);
+ iov_iter_zero(subreq->len, &iter);
+
if (subreq->start + subreq->transferred >= subreq->rreq->i_size)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
+ trace_netfs_rreq(subreq->rreq, netfs_rreq_trace_zero_unread);
}
/*
@@ -40,11 +46,11 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
* dirty and let writeback handle it.
*/
static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
- struct folio_queue *folioq,
+ struct bvecq *bvecq,
int slot)
{
struct netfs_folio *finfo;
- struct folio *folio = folioq_folio(folioq, slot);
+ struct folio *folio = page_folio(bvecq->bv[slot].bv_page);
if (unlikely(folio_pos(folio) < rreq->abandon_to)) {
trace_netfs_folio(folio, netfs_folio_trace_abandon);
@@ -54,7 +60,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
flush_dcache_folio(folio);
folio_mark_uptodate(folio);
- if (!test_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags)) {
+ if (!netfs_using_pgpriv2(rreq)) {
finfo = netfs_folio_info(folio);
if (finfo) {
trace_netfs_folio(folio, netfs_folio_trace_filled_gaps);
@@ -75,7 +81,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
trace_netfs_folio(folio, netfs_folio_trace_read_done);
}
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
} else {
// TODO: Use of PG_private_2 is deprecated.
if (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags))
@@ -91,7 +97,7 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
folio_unlock(folio);
}
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
}
/*
@@ -100,23 +106,21 @@ static void netfs_unlock_read_folio(struct netfs_io_request *rreq,
static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
unsigned int *notes)
{
- struct folio_queue *folioq = rreq->buffer.tail;
+ struct bvecq *bvecq = rreq->collect_cursor.bvecq;
unsigned long long collected_to = rreq->collected_to;
- unsigned int slot = rreq->buffer.first_tail_slot;
+ unsigned int slot = rreq->collect_cursor.slot;
if (rreq->cleaned_to >= rreq->collected_to)
return;
// TODO: Begin decryption
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&rreq->buffer);
- if (!folioq) {
- rreq->front_folio_order = 0;
- return;
- }
- slot = 0;
- }
+ /* We have to wait for readahead refs to have been released before we
+ * can unlock any folios as the ref-dropper walks i_pages and the only
+ * thing preventing these folios from being removed is the folio lock.
+ */
+ if (test_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags))
+ netfs_wait_for_put_ra_refs(rreq);
for (;;) {
struct folio *folio;
@@ -124,16 +128,30 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
unsigned int order;
size_t fsize;
+ /* Clean up the head bvecq segment. If we clear an entire
+ * segment, then we can get rid of it provided it's not also
+ * the tail segment being filled by the issuer.
+ */
+ if (!bvecq_acquire_slot(bvecq, slot)) {
+ rreq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&rreq->collect_cursor)) {
+ rreq->front_folio_order = 0;
+ return;
+ }
+ bvecq = rreq->collect_cursor.bvecq;
+ slot = rreq->collect_cursor.slot;
+ }
+
if (*notes & COPY_TO_CACHE)
set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
- folio = folioq_folio(folioq, slot);
+ folio = page_folio(bvecq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_locked(folio),
"R=%08x: folio %lx is not locked\n",
rreq->debug_id, folio->index))
trace_netfs_folio(folio, netfs_folio_trace_not_locked);
- order = folioq_folio_order(folioq, slot);
+ order = folio_order(folio);
rreq->front_folio_order = order;
fsize = PAGE_SIZE << order;
fpos = folio_pos(folio);
@@ -145,33 +163,19 @@ static void netfs_read_unlock_folios(struct netfs_io_request *rreq,
if (collected_to < fend)
break;
- netfs_unlock_read_folio(rreq, folioq, slot);
+ netfs_unlock_read_folio(rreq, bvecq, slot);
+ slot++;
WRITE_ONCE(rreq->cleaned_to, fpos + fsize);
*notes |= MADE_PROGRESS;
clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
- */
- folioq_clear(folioq, slot);
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&rreq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- trace_netfs_folioq(folioq, netfs_trace_folioq_read_progress);
- }
-
if (fpos + fsize >= collected_to)
break;
}
- rreq->buffer.tail = folioq;
-done:
- rreq->buffer.first_tail_slot = slot;
+ bvecq_pos_move(&rreq->collect_cursor, bvecq);
+ rreq->collect_cursor.slot = slot;
}
/*
@@ -199,6 +203,8 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
notes = BUFFERED;
else
notes = 0;
+ if (test_bit(NETFS_RREQ_ABANDON_REQ, &rreq->flags))
+ notes |= ABANDON_RREQ;
/* Remove completed subrequests from the front of the stream and
* advance the completion point. We stop when we hit something that's
@@ -224,14 +230,43 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
if (netfs_check_subreq_in_progress(front))
notes |= HIT_PENDING;
smp_rmb(); /* Read counters after IN_PROGRESS flag. */
+
transferred = READ_ONCE(front->transferred);
+ if (unlikely(transferred > front->len)) {
+ /* Ugh... A subreq overran its allotted length. It
+ * may have corrupted the read buffer.
+ */
+ set_bit(NETFS_RREQ_FAILED, &rreq->flags);
+ set_bit(NETFS_RREQ_ABANDON_REQ, &rreq->flags);
+ notes |= ABANDON_RREQ;
+ }
+
+ /* Deal with an abandoned request. Wait for each subreq to
+ * complete before abandoning them.
+ */
+ if (unlikely(notes & ABANDON_RREQ)) {
+ if (notes & HIT_PENDING)
+ break;
+
+ /* The subreq now belongs to us. */
+ stream->error = -EIO;
+ stream->failed = true;
+ rreq->abandon_to = front->start + front->len;
+ rreq->error = -EIO;
+ transferred = 0;
+ trace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);
+ notes |= ABANDON_SREQ;
+ goto sreq_complete;
+ }
/* If we can now collect the next folio, do so. We don't want
* to defer this as we have to decide whether we need to copy
* to the cache or not, and that may differ between adjacent
* subreqs.
*/
- if (notes & BUFFERED) {
+ if (notes & ABANDON_SREQ) {
+ /* Don't collect anything. */
+ } else if (notes & BUFFERED) {
size_t fsize = PAGE_SIZE << rreq->front_folio_order;
/* Clear the tail of a short read. */
@@ -257,8 +292,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
transferred = front->len;
trace_netfs_rreq(rreq, netfs_rreq_trace_set_abandon);
}
- if (front->start + transferred >= rreq->cleaned_to + fsize ||
- test_bit(NETFS_SREQ_HIT_EOF, &front->flags))
+ if (front->start + transferred >= rreq->cleaned_to + fsize)
netfs_read_unlock_folios(rreq, ¬es);
} else {
stream->collected_to = front->start + transferred;
@@ -293,6 +327,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
notes |= MADE_PROGRESS;
}
+sreq_complete:
/* Remove if completely consumed. */
stream->source = front->source;
spin_lock(&rreq->lock);
@@ -317,6 +352,8 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
if (!(notes & BUFFERED))
rreq->cleaned_to = rreq->collected_to;
+ if (notes & ABANDON_RREQ)
+ goto out;
if (notes & NEED_RETRY)
goto need_retry;
if (notes & MADE_PROGRESS) {
@@ -348,12 +385,17 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
if (rreq->origin == NETFS_UNBUFFERED_READ ||
rreq->origin == NETFS_DIO_READ) {
- for (i = 0; i < rreq->direct_bv_count; i++) {
- flush_dcache_page(rreq->direct_bv[i].bv_page);
- // TODO: cifs marks pages in the destination buffer
- // dirty under some circumstances after a read. Do we
- // need to do that too?
- set_page_dirty(rreq->direct_bv[i].bv_page);
+ for (struct bvecq *bq = rreq->collect_cursor.bvecq; bq; bq = bvecq_next(bq)) {
+ unsigned int nr_slots = bvecq_nr_slots_acquire(bq);
+ /* Read the slot count before the slots. */
+
+ for (i = 0; i < nr_slots; i++) {
+ flush_dcache_page(bq->bv[i].bv_page);
+ // TODO: cifs marks pages in the destination buffer
+ // dirty under some circumstances after a read. Do we
+ // need to do that too?
+ set_page_dirty(bq->bv[i].bv_page);
+ }
}
}
@@ -372,31 +414,6 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
inode_dio_end(rreq->inode);
}
-/*
- * Do processing after reading a monolithic single object.
- */
-static void netfs_rreq_assess_single(struct netfs_io_request *rreq)
-{
- struct netfs_io_stream *stream = &rreq->io_streams[0];
-
- if (!rreq->error && stream->source == NETFS_DOWNLOAD_FROM_SERVER &&
- fscache_resources_valid(&rreq->cache_resources)) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
- netfs_single_mark_inode_dirty(rreq->inode);
- }
-
- if (rreq->iocb) {
- rreq->iocb->ki_pos += rreq->transferred;
- if (rreq->iocb->ki_complete) {
- trace_netfs_rreq(rreq, netfs_rreq_trace_ki_complete);
- rreq->iocb->ki_complete(
- rreq->iocb, rreq->error ? rreq->error : rreq->transferred);
- }
- }
- if (rreq->netfs_ops->done)
- rreq->netfs_ops->done(rreq);
-}
-
/*
* Perform the collection of subrequests and folios.
*
@@ -412,9 +429,8 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
/* We're done when the app thread has finished posting subreqs and the
* queue is empty.
*/
- if (!test_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags))
+ if (!netfs_are_all_subreqs_queued(rreq))
return false;
- smp_rmb(); /* Read ALL_QUEUED before subreq lists. */
if (!list_empty(&stream->subrequests))
return false;
@@ -432,7 +448,7 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
netfs_rreq_assess_dio(rreq);
break;
case NETFS_READ_SINGLE:
- netfs_rreq_assess_single(rreq);
+ WARN_ON_ONCE(1);
break;
default:
break;
@@ -444,7 +460,15 @@ bool netfs_read_collection(struct netfs_io_request *rreq)
trace_netfs_rreq(rreq, netfs_rreq_trace_done);
netfs_clear_subrequests(rreq);
- netfs_unlock_abandoned_read_pages(rreq);
+ switch (rreq->origin) {
+ case NETFS_READAHEAD:
+ case NETFS_READPAGE:
+ case NETFS_READ_FOR_WRITE:
+ netfs_unlock_abandoned_read_pages(rreq);
+ break;
+ default:
+ break;
+ }
if (unlikely(rreq->copy_to_cache))
netfs_pgpriv2_end_copy_to_cache(rreq);
return true;
@@ -477,15 +501,15 @@ void netfs_read_collection_worker(struct work_struct *work)
void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
- struct netfs_io_stream *stream = &rreq->io_streams[0];
- size_t fsize = PAGE_SIZE << rreq->front_folio_order;
+ struct netfs_io_stream *stream = &rreq->io_streams[subreq->stream_nr];
+ size_t fsize = umax(PAGE_SIZE << rreq->front_folio_order, 256 * 1024);
trace_netfs_sreq(subreq, netfs_sreq_trace_progress);
/* If we are at the head of the queue, wake up the collector,
* getting a ref to it if we were the ones to do so.
*/
- if (subreq->start + subreq->transferred > rreq->cleaned_to + fsize &&
+ if (subreq->start + subreq->transferred >= stream->collected_to + fsize &&
(rreq->origin == NETFS_READAHEAD ||
rreq->origin == NETFS_READPAGE ||
rreq->origin == NETFS_READ_FOR_WRITE) &&
@@ -528,6 +552,21 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
break;
}
+ /* If the subrequest read more than it was supposed to, abort
+ * the request with EIO as we may have clobbered other parts
+ * of the buffer that are already read.
+ */
+ if (subreq->transferred > subreq->len) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_too_much);
+ __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ subreq->error = -EIO;
+ trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_set_pause);
+ set_bit(NETFS_RREQ_PAUSE, &rreq->flags);
+ goto skip_error_checks;
+ }
+
/* Deal with retry requests, short reads and errors. If we retry
* but don't make progress, we abandon the attempt.
*/
@@ -541,6 +580,11 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
} else if (test_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags)) {
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_partial_read);
+ } else if (subreq->source == NETFS_READ_FROM_CACHE) {
+ netfs_stat(&netfs_n_rh_read_failed);
+ __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ subreq->error = -ENODATA;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_short);
} else {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
subreq->error = -ENODATA;
@@ -559,6 +603,8 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
if (unlikely(subreq->error < 0)) {
trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read);
+ if (subreq->error == -ENOMEM)
+ set_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags);
if (subreq->source == NETFS_READ_FROM_CACHE) {
netfs_stat(&netfs_n_rh_read_failed);
__set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index a1489aa29f782a..491cedc4ed1be9 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -13,15 +13,46 @@
#include <linux/task_io_accounting_ops.h>
#include "internal.h"
+int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *creq = subreq->rreq;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ // TODO: Wait here for completion of prev subreq
+
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ if (stream->buffered == 0)
+ netfs_all_subreqs_queued(creq);
+ return 0;
+}
+
/*
- * [DEPRECATED] Copy a folio to the cache with PG_private_2 set.
+ * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. Note that the
+ * folio won't necessarily be contiguous with the previous one as there might
+ * be a mixture of folios read from the cache and downloaded from the server
+ * (or just zeroed).
*/
static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio)
{
struct netfs_io_stream *cache = &creq->io_streams[1];
+ struct bvecq *queue;
+ unsigned int slot;
+ size_t dio_size = PAGE_SIZE;
size_t fsize = folio_size(folio), flen = fsize;
loff_t fpos = folio_pos(folio), i_size;
- bool to_eof = false;
_enter("");
@@ -41,54 +72,43 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
if (fpos + fsize > creq->i_size)
creq->i_size = i_size;
- if (flen > i_size - fpos) {
+ if (flen > i_size - fpos)
flen = i_size - fpos;
- to_eof = true;
- } else if (flen == i_size - fpos) {
- to_eof = true;
- }
+
+ flen = round_up(flen, dio_size);
_debug("folio %zx %zx", flen, fsize);
trace_netfs_folio(folio, netfs_folio_trace_store_copy);
- /* Attach the folio to the rolling buffer. */
- if (rolling_buffer_append(&creq->buffer, folio, 0) < 0) {
- clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &creq->flags);
- return;
+ /* Institute a new bvec queue segment if the current one is full or if
+ * we encounter a discontiguity. The discontiguity break is important
+ * when it comes to bulk unlocking folios by file range.
+ */
+ queue = creq->load_cursor.bvecq;
+ if (bvecq_is_full(queue) ||
+ (fpos != creq->last_end && creq->last_end > 0 && queue->nr_slots > 0)) {
+ bvecq_buffer_append(&creq->load_cursor, creq->spare);
+ creq->spare = NULL;
+
+ queue = creq->load_cursor.bvecq;
+ queue->fpos = fpos;
+ if (fpos != creq->last_end)
+ queue->discontig = true;
}
- cache->submit_extendable_to = fsize;
- cache->submit_off = 0;
- cache->submit_len = flen;
-
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- do {
- ssize_t part;
-
- creq->buffer.iter.iov_offset = cache->submit_off;
-
- atomic64_set(&creq->issued_to, fpos + cache->submit_off);
- cache->submit_extendable_to = fsize - cache->submit_off;
- part = netfs_advance_write(creq, cache, fpos + cache->submit_off,
- cache->submit_len, to_eof);
- cache->submit_off += part;
- if (part > cache->submit_len)
- cache->submit_len = 0;
- else
- cache->submit_len -= part;
- } while (cache->submit_len > 0);
-
- creq->buffer.iter.iov_offset = 0;
- rolling_buffer_advance(&creq->buffer, fsize);
- atomic64_set(&creq->issued_to, fpos + fsize);
-
- if (flen < fsize)
- netfs_issue_write(creq, cache);
+ /* Attach the folio to the rolling buffer. */
+ slot = queue->nr_slots;
+ bvec_set_folio(&queue->bv[slot], folio, fsize, 0);
+ trace_netfs_bv_slot(queue, slot);
+ slot++;
+ bvecq_filled_to(queue, slot);
+ creq->load_cursor.slot = slot;
+ creq->load_cursor.offset = 0;
+ creq->last_end = fpos + flen;
+ trace_netfs_wback(creq, folio, 0);
+
+ cache->buffered += flen;
}
/*
@@ -98,6 +118,7 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
struct netfs_io_request *rreq, struct folio *folio)
{
struct netfs_io_request *creq;
+ struct netfs_io_stream *cache;
if (!fscache_resources_valid(&rreq->cache_resources))
goto cancel;
@@ -107,9 +128,16 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache(
if (IS_ERR(creq))
goto cancel;
- if (!creq->io_streams[1].avail)
+ cache = &creq->io_streams[1];
+ if (!cache->avail)
+ goto cancel_put;
+
+ if (bvecq_buffer_init(&creq->load_cursor, GFP_KERNEL) < 0)
goto cancel_put;
+ bvecq_pos_set(&cache->dispatch_cursor, &creq->load_cursor);
+ bvecq_pos_set(&creq->collect_cursor, &creq->load_cursor);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &creq->flags);
trace_netfs_copy2cache(rreq, creq);
trace_netfs_write(creq, netfs_write_trace_copy_to_cache);
@@ -138,30 +166,74 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo
if (IS_ERR(creq))
return;
+ if (!creq->spare) {
+ creq->spare = bvecq_alloc_one(BVECQ_STD_SLOTS, GFP_NOFS);
+ if (!creq->spare) {
+ clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &creq->flags);
+ return;
+ }
+ }
+
trace_netfs_folio(folio, netfs_folio_trace_copy_to_cache);
folio_start_private_2(folio);
netfs_pgpriv2_copy_folio(creq, folio);
}
+/*
+ * Issue all pending writes on the cache stream.
+ */
+static void netfs_pgpriv2_issue_stream(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
+{
+ atomic64_set_release(&stream->issued_to, wreq->start);
+
+ do {
+ struct netfs_io_subrequest *subreq;
+ int ret;
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq)
+ break;
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. Punt the
+ * error to the collector and stop the issuance of new
+ * subreqs.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+ /* We no longer own subreq. */
+
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags))
+ break;
+
+ } while (stream->buffered > 0);
+
+ netfs_all_subreqs_queued(wreq);
+}
+
/*
* [DEPRECATED] End writing to the cache, flushing out any outstanding writes.
*/
void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
{
struct netfs_io_request *creq = rreq->copy_to_cache;
+ struct netfs_io_stream *stream = &creq->io_streams[1];
if (IS_ERR_OR_NULL(creq))
return;
- netfs_issue_write(creq, &creq->io_streams[1]);
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &creq->flags);
+ netfs_pgpriv2_issue_stream(creq, stream);
trace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache);
if (list_empty_careful(&creq->io_streams[1].subrequests))
netfs_wake_collector(creq);
netfs_put_request(creq, netfs_rreq_trace_put_return);
- creq->copy_to_cache = NULL;
+ rreq->copy_to_cache = NULL;
}
/*
@@ -170,22 +242,27 @@ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq)
*/
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
{
- struct folio_queue *folioq = creq->buffer.tail;
+ struct bvecq *bq = creq->collect_cursor.bvecq;
unsigned long long collected_to = creq->collected_to;
- unsigned int slot = creq->buffer.first_tail_slot;
+ unsigned int slot;
bool made_progress = false;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&creq->buffer);
- slot = 0;
- }
+ slot = creq->collect_cursor.slot;
for (;;) {
struct folio *folio;
unsigned long long fpos, fend;
size_t fsize, flen;
- folio = folioq_folio(folioq, slot);
+ if (!bvecq_acquire_slot(bq, slot)) {
+ creq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&creq->collect_cursor))
+ goto out;
+ bq = creq->collect_cursor.bvecq;
+ slot = creq->collect_cursor.slot;
+ }
+
+ folio = page_folio(bq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_private_2(folio),
"R=%08x: folio %lx is not marked private_2\n",
creq->debug_id, folio->index))
@@ -208,25 +285,18 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *creq)
creq->cleaned_to = fpos + fsize;
made_progress = true;
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
+ /* Clean up the head segment. If we clear an entire segment,
+ * then we can get rid of it provided it's not also the tail
+ * segment being filled by the issuer.
*/
- folioq_clear(folioq, slot);
+ bq->bv[slot].bv_page = NULL;
slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&creq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- }
if (fpos + fsize >= collected_to)
break;
}
- creq->buffer.tail = folioq;
-done:
- creq->buffer.first_tail_slot = slot;
+ creq->collect_cursor.slot = slot;
+out:
return made_progress;
}
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index 2b42758e01ec94..2f96b6e7a1b606 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -9,14 +9,55 @@
#include <linux/slab.h>
#include "internal.h"
-static void netfs_reissue_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
+/*
+ * Prepare the I/O buffer on a buffered read subrequest for the filesystem to
+ * use as a bvec queue.
+ */
+int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- subreq->error = 0;
+ struct netfs_io_request *rreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->retry_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+ len = bvecq_slice(&rreq->retry_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+ rreq->retry_buffered -= subreq->len;
+ rreq->retry_start += subreq->len;
+ return 0;
+}
+
+/*
+ * Reset the state of the subrequest and discard any buffering so that we can
+ * retry (where this may include sending it to the server instead of the
+ * cache).
+ */
+int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq)
+{
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+
+ if (subreq->retry_count > 3) {
+ trace_netfs_sreq(subreq, netfs_sreq_trace_too_many_retries);
+ return subreq->error;
+ }
+
+ subreq->retry_count++;
__clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
- netfs_stat(&netfs_n_rh_retry_read_subreq);
- subreq->rreq->netfs_ops->issue_read(subreq);
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+ subreq->error = 0;
+ subreq->transferred = 0;
+ netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
+ return 0;
}
/*
@@ -28,6 +69,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
struct netfs_io_subrequest *subreq;
struct netfs_io_stream *stream = &rreq->io_streams[0];
struct list_head *next;
+ int ret;
_enter("R=%x", rreq->debug_id);
@@ -37,47 +79,19 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
if (rreq->netfs_ops->retry_request)
rreq->netfs_ops->retry_request(rreq, NULL);
- /* If there's no renegotiation to do, just resend each retryable subreq
- * up to the first permanently failed one.
- */
- if (!rreq->netfs_ops->prepare_read &&
- !rreq->cache_resources.ops) {
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- subreq->retry_count++;
- netfs_reset_iter(subreq);
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
- }
- }
- return;
- }
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
- /* Okay, we need to renegotiate all the download requests and flip any
- * failed cache reads over to being download requests and negotiate
- * those also. All fully successful subreqs have been removed from the
- * list and any spare data from those has been donated.
- *
- * What we do is decant the list and rebuild it one subreq at a time so
- * that we don't end up with donations jumping over a gap we're busy
- * populating with smaller subrequests. In the event that the subreq
- * we just launched finishes before we insert the next subreq, it'll
- * fill in rreq->prev_donated instead.
- *
- * Note: Alternatively, we could split the tail subrequest right before
- * we reissue it and fix up the donations under lock.
+ /* Renegotiate all the download requests and flip any failed cache
+ * reads over to being download requests and negotiate those also.
*/
- next = stream->subrequests.next;
-
do {
struct netfs_io_subrequest *from, *to, *tmp;
- struct iov_iter source;
- unsigned long long start, len;
- size_t part;
- bool boundary = false, subreq_superfluous = false;
+ unsigned long long start;
+ size_t len;
+ bool subreq_superfluous = false;
+
+ bvecq_pos_unset(&rreq->retry_cursor);
/* Go through the subreqs and find the next span of contiguous
* buffer that we then rejig (cifs, for example, needs the
@@ -92,8 +106,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
rreq->debug_id, from->debug_index,
from->start, from->transferred, from->len);
- if (test_bit(NETFS_SREQ_FAILED, &from->flags) ||
- !test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
+ if (!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) {
subreq = from;
goto abandon;
}
@@ -105,66 +118,55 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
break;
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
- if (subreq->start + subreq->transferred != start + len ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
+ if (subreq->start != start + len ||
+ subreq->transferred > 0 ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
len += to->len;
}
- _debug(" - range: %llx-%llx %llx", start, start + len - 1, len);
+ _debug(" - range: %llx-%llx %zx", start, start + len - 1, len);
/* Determine the set of buffers we're going to use. Each
- * subreq gets a subset of a single overall contiguous buffer.
+ * subreq takes a subset of a single overall contiguous buffer.
+ */
+ bvecq_pos_transfer(&rreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&rreq->retry_cursor, from->transferred);
+ rreq->retry_start = start;
+ rreq->retry_buffered = len;
+ from->transferred = 0;
+
+ /* Work through the sublist. The chain of buffers we're going
+ * to fill is attached to dispatch_cursor and we need to read
+ * 'len' amount of data from 'start'.
*/
- netfs_reset_iter(from);
- source = from->io_iter;
- source.count = len;
-
- /* Work through the sublist. */
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len) {
+ if (rreq->retry_buffered == 0) {
subreq_superfluous = true;
break;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start - subreq->transferred;
- subreq->len = len + subreq->transferred;
- __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- subreq->retry_count++;
-
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
- /* Renegotiate max_len (rsize) */
- stream->sreq_max_len = subreq->len;
- if (rreq->netfs_ops->prepare_read &&
- rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
+ ret = netfs_reset_for_read_retry(subreq);
+ if (ret < 0) {
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ rreq->error = ret;
goto abandon;
}
- part = umin(len, stream->sreq_max_len);
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
- subreq->len = subreq->transferred + part;
- subreq->io_iter = source;
- iov_iter_truncate(&subreq->io_iter, part);
- iov_iter_advance(&source, part);
- len -= part;
- start += part;
- if (!len) {
- if (boundary)
- __set_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
- } else {
- __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags);
+ netfs_stat(&netfs_n_rh_download);
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
-
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_read(rreq, subreq);
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
if (subreq == to) {
subreq_superfluous = false;
break;
@@ -174,7 +176,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
+ if (rreq->retry_buffered == 0) {
if (!subreq_superfluous)
continue;
list_for_each_entry_safe_from(subreq, tmp,
@@ -192,7 +194,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
}
/* We ran out of subrequests, so we need to allocate some more
- * and insert them after.
+ * and insert them after. They must start with being marked
+ * for retry to switch to the retry cursor.
*/
do {
subreq = netfs_alloc_subrequest(rreq);
@@ -201,8 +204,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
goto abandon_after;
}
subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- subreq->start = start;
- subreq->len = len;
+ subreq->start = rreq->retry_start;
+ subreq->len = rreq->retry_buffered;
subreq->stream_nr = stream->stream_nr;
subreq->retry_count = 1;
@@ -210,42 +213,32 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
refcount_read(&subreq->ref),
netfs_sreq_trace_new);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+
spin_lock(&rreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&rreq->lock);
to = subreq;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = umin(len, rreq->rsize);
- stream->sreq_max_segs = 0;
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
-
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read(subreq) < 0) {
- trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- goto abandon;
- }
-
- part = umin(len, stream->sreq_max_len);
- subreq->len = subreq->transferred + part;
- subreq->io_iter = source;
- iov_iter_truncate(&subreq->io_iter, part);
- iov_iter_advance(&source, part);
-
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ goto abandon_after;
}
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags))
+ goto abandon_after;
- netfs_reissue_read(rreq, subreq);
- } while (len);
+ } while (rreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
+out:
+ bvecq_pos_unset(&rreq->retry_cursor);
return;
/* If we hit an error, fail all remaining incomplete subrequests */
@@ -262,6 +255,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq)
__set_bit(NETFS_SREQ_FAILED, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
}
+ goto out;
}
/*
@@ -272,6 +266,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
struct netfs_io_stream *stream = &rreq->io_streams[0];
netfs_stat(&netfs_n_rh_retry_read_req);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_retry_begin);
/* Wait for all outstanding I/O to quiesce before performing retries as
* we may need to renegotiate the I/O sizes.
@@ -282,6 +277,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
trace_netfs_rreq(rreq, netfs_rreq_trace_resubmit);
netfs_retry_read_subrequests(rreq);
+ trace_netfs_rreq(rreq, netfs_rreq_trace_retry_end);
}
/*
@@ -290,23 +286,25 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
*/
void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
{
- struct folio_queue *p;
-
- for (p = rreq->buffer.tail; p; p = p->next) {
- for (int slot = 0; slot < folioq_count(p); slot++) {
- struct folio *folio = folioq_folio(p, slot);
-
- if (folio && !folioq_is_marked2(p, slot)) {
- if (folio == rreq->no_unlock_folio &&
- test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO,
- &rreq->flags)) {
- _debug("no unlock");
- } else {
- trace_netfs_folio(folio,
- netfs_folio_trace_abandon);
- folio_unlock(folio);
- }
+ struct bvecq *p;
+
+ for (p = rreq->collect_cursor.bvecq; p; p = bvecq_next(p)) {
+ unsigned int nr_slots = bvecq_nr_slots_acquire(p);
+
+ for (int slot = 0; slot < nr_slots; slot++) {
+ if (!p->bv[slot].bv_page)
+ continue;
+
+ struct folio *folio = page_folio(p->bv[slot].bv_page);
+
+ if (folio == rreq->no_unlock_folio &&
+ test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO, &rreq->flags)) {
+ _debug("no unlock");
+ continue;
}
+ trace_netfs_folio(folio, netfs_folio_trace_abandon);
+ folio_unlock(folio);
+ p->bv[slot].bv_page = NULL;
}
}
}
diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c
index 8833550d2eb608..447b2e1e050c28 100644
--- a/fs/netfs/read_single.c
+++ b/fs/netfs/read_single.c
@@ -16,6 +16,22 @@
#include <linux/netfs.h>
#include "internal.h"
+int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *rreq = subreq->rreq;
+ struct netfs_io_stream *stream = &rreq->io_streams[0];
+
+ bvecq_pos_set(&subreq->dispatch_pos, &rreq->load_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ stream->buffered = 0;
+ stream->issue_from += subreq->len;
+ rreq->submitted = stream->issue_from;
+ netfs_all_subreqs_queued(rreq);
+ return 0;
+}
+
/**
* netfs_single_mark_inode_dirty - Mark a single, monolithic object inode dirty
* @inode: The inode to mark
@@ -58,87 +74,99 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n
return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx));
}
-static void netfs_single_cache_prepare_read(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- if (!cres->ops) {
- subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
- return;
- }
- subreq->source = cres->ops->prepare_read(subreq, rreq->i_size);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
-
-}
-
-static void netfs_single_read_cache(struct netfs_io_request *rreq,
- struct netfs_io_subrequest *subreq)
-{
- struct netfs_cache_resources *cres = &rreq->cache_resources;
-
- _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index);
- netfs_stat(&netfs_n_rh_read);
- cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_FAIL,
- netfs_cache_read_terminated, subreq);
-}
-
/*
* Perform a read to a buffer from the cache or the server. Only a single
* subreq is permitted as the object must be fetched in a single transaction.
*/
static int netfs_single_dispatch_read(struct netfs_io_request *rreq)
{
+ struct fscache_occupancy occ = {
+ .query_from = 0,
+ .query_to = rreq->len,
+ .cached_from[0] = ULLONG_MAX,
+ .cached_to[0] = ULLONG_MAX,
+ .cached_from[1] = ULLONG_MAX,
+ .cached_to[1] = ULLONG_MAX,
+ };
struct netfs_io_subrequest *subreq;
- int ret = 0;
+ int ret;
+
+ netfs_read_query_cache(rreq, &occ);
- subreq = netfs_alloc_subrequest(rreq);
+ subreq = netfs_alloc_read_subrequest(rreq);
if (!subreq)
return -ENOMEM;
- subreq->source = NETFS_SOURCE_UNKNOWN;
subreq->start = 0;
subreq->len = rreq->len;
- subreq->io_iter = rreq->buffer.iter;
- netfs_queue_read(rreq, subreq);
+ trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- netfs_single_cache_prepare_read(rreq, subreq);
- switch (subreq->source) {
- case NETFS_DOWNLOAD_FROM_SERVER:
+ /* Try to use the cache if the cache content matches the size of the
+ * remote file.
+ */
+ if (occ.cached_from[0] == 0 &&
+ occ.cached_to[0] >= rreq->len) {
+ struct netfs_cache_resources *cres = &rreq->cache_resources;
+
+ subreq->source = NETFS_READ_FROM_CACHE;
+ netfs_stat(&netfs_n_rh_read);
+ ret = cres->ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
+ }
+
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+
+ /* Didn't manage to retrieve from the cache, so toss it to the
+ * server instead.
+ */
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
+ }
+
+ __set_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags);
+
+ /* Try to send it to the cache. */
+ for (;;) {
+ subreq->source = NETFS_DOWNLOAD_FROM_SERVER;
netfs_stat(&netfs_n_rh_download);
- if (rreq->netfs_ops->prepare_read) {
- ret = rreq->netfs_ops->prepare_read(subreq);
- if (ret < 0)
- goto cancel;
+ ret = rreq->netfs_ops->issue_read(subreq);
+ if (ret < 0) {
+ subreq->error = ret;
+ netfs_read_subreq_terminated(subreq);
}
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- rreq->netfs_ops->issue_read(subreq);
- rreq->submitted += subreq->len;
- break;
- case NETFS_READ_FROM_CACHE:
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- netfs_single_read_cache(rreq, subreq);
- rreq->submitted += subreq->len;
- ret = 0;
- break;
- default:
- pr_warn("Unexpected single-read source %u\n", subreq->source);
- WARN_ON_ONCE(true);
- ret = -EIO;
- goto cancel;
+ ret = netfs_wait_for_in_progress_subreq(rreq, subreq);
+ if (ret == 0)
+ goto success;
+ if (ret == -ENOMEM)
+ goto cancel;
+ if (ret != -EAGAIN)
+ goto failed;
+ if (netfs_reset_for_read_retry(subreq) < 0)
+ goto cancel;
}
- return ret;
+success:
+ rreq->transferred = subreq->transferred;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_consumed);
+ return 0;
cancel:
- netfs_cancel_read(subreq, ret);
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &rreq->flags);
- netfs_wake_collector(rreq);
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_cancel);
+ return ret;
+failed:
+ rreq->error = ret;
+ list_del_init(&subreq->rreq_link);
+ netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed);
return ret;
}
@@ -170,6 +198,14 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
if (IS_ERR(rreq))
return PTR_ERR(rreq);
+ ret = netfs_extract_iter(iter, rreq->len, INT_MAX, 0, &rreq->load_cursor.bvecq, 0);
+ if (ret < 0)
+ goto cleanup_free;
+ if (ret < rreq->len) {
+ ret = -EIO;
+ goto cleanup_free;
+ }
+
ret = netfs_single_begin_cache_read(rreq, ictx);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS)
goto cleanup_free;
@@ -177,10 +213,29 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite
netfs_stat(&netfs_n_rh_read_single);
trace_netfs_read(rreq, 0, rreq->len, netfs_read_trace_read_single);
- rreq->buffer.iter = *iter;
- netfs_single_dispatch_read(rreq);
+ ret = netfs_single_dispatch_read(rreq);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_complete);
+ if (ret == 0) {
+ task_io_account_read(rreq->transferred);
+
+ if (test_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &rreq->flags) &&
+ fscache_resources_valid(&rreq->cache_resources)) {
+ trace_netfs_rreq(rreq, netfs_rreq_trace_dirty);
+ netfs_single_mark_inode_dirty(rreq->inode);
+ }
+ ret = rreq->transferred;
+ }
+
+ if (rreq->netfs_ops->done)
+ rreq->netfs_ops->done(rreq);
+
+ netfs_wake_rreq_flag(rreq, NETFS_RREQ_IN_PROGRESS, netfs_rreq_trace_wake_ip);
+ /* As we cleared NETFS_RREQ_IN_PROGRESS, we acquired its ref. */
+ netfs_put_request(rreq, netfs_rreq_trace_put_work_ip);
+
+ trace_netfs_rreq(rreq, netfs_rreq_trace_done);
- ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, netfs_rreq_trace_put_return);
return ret;
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
deleted file mode 100644
index a17fbf9853a443..00000000000000
--- a/fs/netfs/rolling_buffer.c
+++ /dev/null
@@ -1,222 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/* Rolling buffer helpers
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- */
-
-#include <linux/bitops.h>
-#include <linux/pagemap.h>
-#include <linux/rolling_buffer.h>
-#include <linux/slab.h>
-#include "internal.h"
-
-static atomic_t debug_ids;
-
-/**
- * netfs_folioq_alloc - Allocate a folio_queue struct
- * @rreq_id: Associated debugging ID for tracing purposes
- * @gfp: Allocation constraints
- * @trace: Trace tag to indicate the purpose of the allocation
- *
- * Allocate, initialise and account the folio_queue struct and log a trace line
- * to mark the allocation.
- */
-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
- unsigned int /*enum netfs_folioq_trace*/ trace)
-{
- struct folio_queue *fq;
-
- fq = kmalloc_obj(*fq, gfp);
- if (fq) {
- netfs_stat(&netfs_n_folioq);
- folioq_init(fq, rreq_id);
- fq->debug_id = atomic_inc_return(&debug_ids);
- trace_netfs_folioq(fq, trace);
- }
- return fq;
-}
-EXPORT_SYMBOL(netfs_folioq_alloc);
-
-/**
- * netfs_folioq_free - Free a folio_queue struct
- * @folioq: The object to free
- * @trace: Trace tag to indicate which free
- *
- * Free and unaccount the folio_queue struct.
- */
-void netfs_folioq_free(struct folio_queue *folioq,
- unsigned int /*enum netfs_trace_folioq*/ trace)
-{
- trace_netfs_folioq(folioq, trace);
- netfs_stat_d(&netfs_n_folioq);
- kfree(folioq);
-}
-EXPORT_SYMBOL(netfs_folioq_free);
-
-/*
- * Initialise a rolling buffer. We allocate an empty folio queue struct to so
- * that the pointers can be independently driven by the producer and the
- * consumer.
- */
-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction)
-{
- struct folio_queue *fq;
-
- fq = netfs_folioq_alloc(rreq_id, GFP_NOFS, netfs_trace_folioq_rollbuf_init);
- if (!fq)
- return -ENOMEM;
-
- roll->head = fq;
- roll->tail = fq;
- iov_iter_folio_queue(&roll->iter, direction, fq, 0, 0, 0);
- return 0;
-}
-
-/*
- * Add another folio_queue to a rolling buffer if there's no space left.
- */
-int rolling_buffer_make_space(struct rolling_buffer *roll)
-{
- struct folio_queue *fq, *head = roll->head;
-
- if (!folioq_full(head))
- return 0;
-
- fq = netfs_folioq_alloc(head->rreq_id, GFP_NOFS, netfs_trace_folioq_make_space);
- if (!fq)
- return -ENOMEM;
- fq->prev = head;
-
- roll->head = fq;
- if (folioq_full(head)) {
- /* Make sure we don't leave the master iterator pointing to a
- * block that might get immediately consumed.
- */
- if (roll->iter.folioq == head &&
- roll->iter.folioq_slot == folioq_nr_slots(head)) {
- roll->iter.folioq = fq;
- roll->iter.folioq_slot = 0;
- }
- }
-
- /* Make sure the initialisation is stored before the next pointer.
- *
- * [!] NOTE: After we set head->next, the consumer is at liberty to
- * immediately delete the old head.
- */
- smp_store_release(&head->next, fq);
- return 0;
-}
-
-/*
- * Decant the list of folios to read into a rolling buffer.
- */
-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- struct folio_batch *put_batch)
-{
- struct folio_queue *fq;
- struct page **vec;
- int nr, ix, to;
- ssize_t size = 0;
-
- if (rolling_buffer_make_space(roll) < 0)
- return -ENOMEM;
-
- fq = roll->head;
- vec = (struct page **)fq->vec.folios;
- nr = __readahead_batch(ractl, vec + folio_batch_count(&fq->vec),
- folio_batch_space(&fq->vec));
- ix = fq->vec.nr;
- to = ix + nr;
- fq->vec.nr = to;
- for (; ix < to; ix++) {
- struct folio *folio = folioq_folio(fq, ix);
- unsigned int order = folio_order(folio);
-
- fq->orders[ix] = order;
- size += PAGE_SIZE << order;
- trace_netfs_folio(folio, netfs_folio_trace_read);
- if (!folio_batch_add(put_batch, folio))
- folio_batch_release(put_batch);
- }
- WRITE_ONCE(roll->iter.count, roll->iter.count + size);
-
- /* Store the counter after setting the slot. */
- smp_store_release(&roll->next_head_slot, to);
- return size;
-}
-
-/*
- * Append a folio to the rolling buffer.
- */
-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags)
-{
- ssize_t size = folio_size(folio);
- int slot;
-
- if (rolling_buffer_make_space(roll) < 0)
- return -ENOMEM;
-
- slot = folioq_append(roll->head, folio);
- if (flags & ROLLBUF_MARK_1)
- folioq_mark(roll->head, slot);
- if (flags & ROLLBUF_MARK_2)
- folioq_mark2(roll->head, slot);
-
- WRITE_ONCE(roll->iter.count, roll->iter.count + size);
-
- /* Store the counter after setting the slot. */
- smp_store_release(&roll->next_head_slot, slot);
- return size;
-}
-
-/*
- * Delete a spent buffer from a rolling queue and return the next in line. We
- * don't return the last buffer to keep the pointers independent, but return
- * NULL instead.
- */
-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll)
-{
- struct folio_queue *spent = roll->tail, *next = READ_ONCE(spent->next);
-
- if (!next)
- return NULL;
- next->prev = NULL;
- netfs_folioq_free(spent, netfs_trace_folioq_delete);
- roll->tail = next;
- return next;
-}
-
-/*
- * Clear out a rolling queue. Folios that have mark 1 set are put.
- */
-void rolling_buffer_clear(struct rolling_buffer *roll)
-{
- struct folio_batch fbatch;
- struct folio_queue *p;
-
- folio_batch_init(&fbatch);
-
- while ((p = roll->tail)) {
- roll->tail = p->next;
- for (int slot = 0; slot < folioq_count(p); slot++) {
- struct folio *folio = folioq_folio(p, slot);
-
- if (!folio)
- continue;
- if (folioq_is_marked(p, slot)) {
- trace_netfs_folio(folio, netfs_folio_trace_put);
- if (!folio_batch_add(&fbatch, folio))
- folio_batch_release(&fbatch);
- }
- }
-
- netfs_folioq_free(p, netfs_trace_folioq_clear);
- }
-
- folio_batch_release(&fbatch);
-}
diff --git a/fs/netfs/stats.c b/fs/netfs/stats.c
index ab6b916addc443..0ba6ce9295b257 100644
--- a/fs/netfs/stats.c
+++ b/fs/netfs/stats.c
@@ -32,7 +32,6 @@ atomic_t netfs_n_rh_write_zskip;
atomic_t netfs_n_rh_retry_read_req;
atomic_t netfs_n_rh_retry_read_subreq;
atomic_t netfs_n_wh_buffered_write;
-atomic_t netfs_n_wh_writethrough;
atomic_t netfs_n_wh_dio_write;
atomic_t netfs_n_wh_writepages;
atomic_t netfs_n_wh_copy_to_cache;
@@ -47,7 +46,7 @@ atomic_t netfs_n_wh_retry_write_req;
atomic_t netfs_n_wh_retry_write_subreq;
atomic_t netfs_n_wb_lock_skip;
atomic_t netfs_n_wb_lock_wait;
-atomic_t netfs_n_folioq;
+atomic_t netfs_n_bvecq;
int netfs_stats_show(struct seq_file *m, void *v)
{
@@ -58,9 +57,8 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_read_single),
atomic_read(&netfs_n_rh_write_begin),
atomic_read(&netfs_n_rh_write_zskip));
- seq_printf(m, "Writes : BW=%u WT=%u DW=%u WP=%u 2C=%u\n",
+ seq_printf(m, "Writes : BW=%u DW=%u WP=%u 2C=%u\n",
atomic_read(&netfs_n_wh_buffered_write),
- atomic_read(&netfs_n_wh_writethrough),
atomic_read(&netfs_n_wh_dio_write),
atomic_read(&netfs_n_wh_writepages),
atomic_read(&netfs_n_wh_copy_to_cache));
@@ -90,10 +88,10 @@ int netfs_stats_show(struct seq_file *m, void *v)
atomic_read(&netfs_n_rh_retry_read_subreq),
atomic_read(&netfs_n_wh_retry_write_req),
atomic_read(&netfs_n_wh_retry_write_subreq));
- seq_printf(m, "Objs : rr=%u sr=%u foq=%u wsc=%u\n",
+ seq_printf(m, "Objs : rr=%u sr=%u bq=%u wsc=%u\n",
atomic_read(&netfs_n_rh_rreq),
atomic_read(&netfs_n_rh_sreq),
- atomic_read(&netfs_n_folioq),
+ atomic_read(&netfs_n_bvecq),
atomic_read(&netfs_n_wh_wstream_conflict));
seq_printf(m, "WbLock : skip=%u wait=%u\n",
atomic_read(&netfs_n_wb_lock_skip),
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index 210eb8f3958d77..33302cdb08f0f0 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -28,8 +28,8 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
rreq->origin, rreq->error);
pr_err(" st=%llx tsl=%zx/%llx/%llx\n",
rreq->start, rreq->transferred, rreq->submitted, rreq->len);
- pr_err(" cci=%llx/%llx/%llx\n",
- rreq->cleaned_to, rreq->collected_to, atomic64_read(&rreq->issued_to));
+ pr_err(" cci=%llx/%llx\n",
+ rreq->cleaned_to, rreq->collected_to);
pr_err(" iw=%pSR\n", rreq->netfs_ops->issue_write);
for (int i = 0; i < NR_IO_STREAMS; i++) {
const struct netfs_io_subrequest *sreq;
@@ -38,8 +38,9 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
pr_err(" str[%x] s=%x e=%d acnf=%u,%u,%u,%u\n",
s->stream_nr, s->source, s->error,
s->avail, s->active, s->need_retry, s->failed);
- pr_err(" str[%x] ct=%llx t=%zx\n",
- s->stream_nr, s->collected_to, s->transferred);
+ pr_err(" str[%x] it=%llx ct=%llx t=%zx\n",
+ s->stream_nr, atomic64_read(&s->issued_to),
+ s->collected_to, s->transferred);
list_for_each_entry(sreq, &s->subrequests, rreq_link) {
pr_err(" sreq[%x:%x] sc=%u s=%llx t=%zx/%zx r=%d f=%lx\n",
sreq->stream_nr, sreq->debug_index, sreq->source,
@@ -56,7 +57,7 @@ static void netfs_dump_request(const struct netfs_io_request *rreq)
*/
int netfs_folio_written_back(struct folio *folio)
{
- enum netfs_folio_trace why = netfs_folio_trace_clear;
+ enum netfs_folio_trace why = netfs_folio_trace_endwb;
struct inode *inode = folio_inode(folio);
struct netfs_inode *ictx = netfs_inode(inode);
struct netfs_folio *finfo;
@@ -79,13 +80,13 @@ int netfs_folio_written_back(struct folio *folio)
group = finfo->netfs_group;
gcount++;
kfree(finfo);
- why = netfs_folio_trace_clear_s;
+ why = netfs_folio_trace_endwb_s;
goto end_wb;
}
if ((group = netfs_folio_group(folio))) {
if (group == NETFS_FOLIO_COPY_TO_CACHE) {
- why = netfs_folio_trace_clear_cc;
+ why = netfs_folio_trace_endwb_cc;
folio_detach_private(folio);
goto end_wb;
}
@@ -98,7 +99,7 @@ int netfs_folio_written_back(struct folio *folio)
if (!folio_test_dirty(folio)) {
folio_detach_private(folio);
gcount++;
- why = netfs_folio_trace_clear_g;
+ why = netfs_folio_trace_endwb_g;
}
}
@@ -114,12 +115,12 @@ int netfs_folio_written_back(struct folio *folio)
static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
unsigned int *notes)
{
- struct folio_queue *folioq = wreq->buffer.tail;
+ struct bvecq *bvecq = wreq->collect_cursor.bvecq;
unsigned long long collected_to = wreq->collected_to;
- unsigned int slot = wreq->buffer.first_tail_slot;
+ unsigned int slot = wreq->collect_cursor.slot;
- if (WARN_ON_ONCE(!folioq)) {
- pr_err("[!] Writeback unlock found empty rolling buffer!\n");
+ if (WARN_ON_ONCE(!bvecq)) {
+ pr_err("[!] Writeback unlock found empty buffer!\n");
netfs_dump_request(wreq);
return;
}
@@ -130,20 +131,34 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
return;
}
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&wreq->buffer);
- if (!folioq)
- return;
- slot = 0;
- }
-
for (;;) {
struct folio *folio;
struct netfs_folio *finfo;
unsigned long long fpos, fend;
size_t fsize, flen;
- folio = folioq_folio(folioq, slot);
+ /* Try to clean up the head of the queue if it appears to be
+ * used up, but we need to be very careful - the cleanup can
+ * catch the dispatcher, which could lead to us having nothing
+ * left in the queue, causing the front and back pointers to
+ * end up on different tracks. To avoid this, we must always
+ * keep at least one segment in the queue.
+ */
+ if (!bvecq_acquire_slot(bvecq, slot)) {
+ wreq->collect_cursor.slot = slot;
+ if (!bvecq_delete_spent(&wreq->collect_cursor))
+ return;
+ bvecq = wreq->collect_cursor.bvecq;
+ slot = wreq->collect_cursor.slot;
+ }
+
+ if (!bvecq->bv[slot].bv_page) {
+ WARN_ONCE(1, "R=%08x slot already cleared?\n", wreq->debug_id);
+ fsize = bvecq->bv[slot].bv_len;
+ goto skip;
+ }
+
+ folio = page_folio(bvecq->bv[slot].bv_page);
if (WARN_ONCE(!folio_test_writeback(folio),
"R=%08x: folio %lx is not under writeback\n",
wreq->debug_id, folio->index))
@@ -166,26 +181,34 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
wreq->cleaned_to = fpos + fsize;
*notes |= MADE_PROGRESS;
- /* Clean up the head folioq. If we clear an entire folioq, then
- * we can get rid of it provided it's not also the tail folioq
- * being filled by the issuer.
- */
- folioq_clear(folioq, slot);
+ bvecq->bv[slot].bv_page = NULL;
+ skip:
slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = rolling_buffer_delete_spent(&wreq->buffer);
- if (!folioq)
- goto done;
- slot = 0;
- }
-
if (fpos + fsize >= collected_to)
break;
}
- wreq->buffer.tail = folioq;
-done:
- wreq->buffer.first_tail_slot = slot;
+ wreq->collect_cursor.slot = slot;
+}
+
+/*
+ * Collect cache results.
+ */
+static void netfs_cache_collect(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ enum netfs_cache_collect block_type)
+{
+ struct netfs_cache_resources *cres = &wreq->cache_resources;
+
+ if (stream->source != NETFS_WRITE_TO_CACHE ||
+ wreq->cache_coll_to >= stream->collected_to)
+ return;
+
+ if (cres->ops && cres->ops->collect_write)
+ cres->ops->collect_write(wreq, wreq->cache_coll_to,
+ stream->collected_to - wreq->cache_coll_to,
+ block_type);
+ wreq->cache_coll_to = stream->collected_to;
}
/*
@@ -210,11 +233,9 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
trace_netfs_rreq(wreq, netfs_rreq_trace_collect);
reassess_streams:
- issued_to = atomic64_read(&wreq->issued_to);
- smp_rmb();
+ issued_to = ULLONG_MAX;
collected_to = ULLONG_MAX;
if (wreq->origin == NETFS_WRITEBACK ||
- wreq->origin == NETFS_WRITETHROUGH ||
wreq->origin == NETFS_PGPRIV2_COPY_TO_CACHE)
notes = NEED_UNLOCK;
else
@@ -226,23 +247,43 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
* to the tail whilst we're doing this.
*/
for (s = 0; s < NR_IO_STREAMS; s++) {
+ unsigned long long s_issued_to;
+
stream = &wreq->io_streams[s];
- /* Read active flag before list pointers */
+ /* Read active flag before issued_to */
if (!smp_load_acquire(&stream->active))
continue;
- front = list_first_entry_or_null_acquire(&stream->subrequests,
- struct netfs_io_subrequest, rreq_link);
- /* Read first subreq pointer before IN_PROGRESS flag. */
+ for (;;) {
+ bool cancelled;
+
+ /* Order reading the issued_to point before reading the
+ * queue it refers to.
+ */
+ s_issued_to = atomic64_read_acquire(&stream->issued_to);
+ if (s_issued_to < issued_to)
+ issued_to = s_issued_to;
+
+ front = list_first_entry_or_null_acquire(&stream->subrequests,
+ struct netfs_io_subrequest,
+ rreq_link);
+ /* Read first subreq pointer before IN_PROGRESS flag. */
+ if (!front) {
+ if (stream->source == NETFS_UPLOAD_TO_SERVER &&
+ test_bit(NETFS_RREQ_PAUSE, &wreq->flags))
+ notes |= MADE_PROGRESS;
+ break;
+ }
- while (front) {
trace_netfs_collect_sreq(wreq, front);
//_debug("sreq [%x] %llx %zx/%zx",
// front->debug_index, front->start, front->transferred, front->len);
if (stream->collected_to < front->start) {
trace_netfs_collect_gap(wreq, stream, issued_to, 'F');
+ netfs_cache_collect(wreq, stream, NETFS_CACHE_COLLECT_WRITE_DATA);
stream->collected_to = front->start;
+ netfs_cache_collect(wreq, stream, NETFS_CACHE_COLLECT_WRITE_GAP);
}
/* Stall if the front is still undergoing I/O. */
@@ -250,7 +291,6 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
notes |= HIT_PENDING;
break;
}
- smp_rmb(); /* Read counters after I-P flag. */
if (stream->failed) {
stream->collected_to = front->start + front->len;
@@ -263,15 +303,44 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
stream->transferred_valid = true;
notes |= MADE_PROGRESS;
}
- if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
- stream->failed = true;
- stream->error = front->error;
- if (stream->source == NETFS_UPLOAD_TO_SERVER)
- mapping_set_error(wreq->mapping, front->error);
- notes |= NEED_REASSESS | SAW_FAILURE;
+
+ /* Handle failed or cancelled subreqs. Failure of
+ * cache writes are handled differently to upload
+ * failures. Cache writes aren't fatal, provided we're
+ * not doing disconnected operation, and so we can kind
+ * of treat them as if they had succeeded - except that
+ * we need to log any holes they cause.
+ */
+ switch (stream->source) {
+ case NETFS_UPLOAD_TO_SERVER:
+ if (test_bit(NETFS_SREQ_FAILED, &front->flags)) {
+ if (!stream->failed) {
+ stream->failed = true;
+ stream->error = front->error;
+ mapping_set_error(wreq->mapping, front->error);
+ break;
+ }
+ notes |= NEED_REASSESS | SAW_FAILURE;
+ }
+ break;
+
+ case NETFS_WRITE_TO_CACHE:
+ cancelled = test_bit(NETFS_SREQ_CANCELLED, &front->flags);
+ if (cancelled != stream->cancelled) {
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_fail_collect);
+ netfs_cache_collect(wreq, stream,
+ cancelled ?
+ NETFS_CACHE_COLLECT_WRITE_CANCEL :
+ NETFS_CACHE_COLLECT_WRITE_DATA);
+ stream->cancelled = !stream->cancelled;
+ }
+ break;
+
+ default:
+ WARN_ON(1);
break;
}
- if (front->transferred < front->len) {
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &front->flags)) {
stream->need_retry = true;
notes |= NEED_RETRY | MADE_PROGRESS;
break;
@@ -360,6 +429,7 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
*/
bool netfs_write_collection(struct netfs_io_request *wreq)
{
+ struct netfs_io_stream *cstream = &wreq->io_streams[1];
struct netfs_inode *ictx = netfs_inode(wreq->inode);
size_t transferred;
bool transferred_valid = false;
@@ -372,9 +442,8 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
/* We're done when the app thread has finished posting subreqs and all
* the queues in all the streams are empty.
*/
- if (!test_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags))
+ if (!netfs_are_all_subreqs_queued(wreq))
return false;
- smp_rmb(); /* Read ALL_QUEUED before lists. */
transferred = LONG_MAX;
for (s = 0; s < NR_IO_STREAMS; s++) {
@@ -395,13 +464,22 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
wreq->transferred = transferred;
trace_netfs_rreq(wreq, netfs_rreq_trace_write_done);
- if (wreq->io_streams[1].active &&
- wreq->io_streams[1].failed &&
- ictx->ops->invalidate_cache) {
- /* Cache write failure doesn't prevent writeback completion
- * unless we're in disconnected mode.
- */
- ictx->ops->invalidate_cache(wreq);
+ if (cstream->active) {
+ if (test_bit(NETFS_RREQ_CACHE_ERROR, &wreq->flags)) {
+ if (ictx->ops->invalidate_cache) {
+ /* Cache write failure doesn't prevent
+ * writeback completion unless we're in
+ * disconnected mode.
+ */
+ trace_netfs_rreq(wreq, netfs_rreq_trace_inval_cache);
+ ictx->ops->invalidate_cache(wreq);
+ }
+ } else if (!cstream->failed) {
+ netfs_cache_collect(wreq, cstream,
+ cstream->cancelled ?
+ NETFS_CACHE_COLLECT_WRITE_CANCEL :
+ NETFS_CACHE_COLLECT_WRITE_DATA);
+ }
}
_debug("finished");
@@ -411,7 +489,6 @@ bool netfs_write_collection(struct netfs_io_request *wreq)
switch (wreq->origin) {
case NETFS_WRITEBACK:
case NETFS_WRITEBACK_SINGLE:
- case NETFS_WRITETHROUGH:
netfs_wb_end(ictx);
break;
default:
@@ -486,24 +563,57 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error)
if (IS_ERR_VALUE(transferred_or_error)) {
subreq->error = transferred_or_error;
- /* if need retry is set, error should not matter */
- if (!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- trace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);
- }
+ if (transferred_or_error == -ENOMEM)
+ set_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags);
switch (subreq->source) {
case NETFS_WRITE_TO_CACHE:
+ /* We don't mark a cache-write subreq as failed.
+ * Instead we tell the issuer to produce dummy subreqs
+ * instead and make a note if we need to invalidate the
+ * cache at the end. We also don't pause the loop that
+ * grabs pages and launches upload subreqs.
+ *
+ * Note that we need to distinguish between -ENOBUFS
+ * (no space available in the cache) and other errors.
+ * In the former case, we can keep the data we have,
+ * though we might have to change the way the on-disk
+ * data is tracked.
+ */
netfs_stat(&netfs_n_wh_write_failed);
+ if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ /* We don't retry failed cache writes. */
+ __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ if (!subreq->error)
+ subreq->error = -ENOBUFS;
+ }
+
+ trace_netfs_failure(wreq, subreq, transferred_or_error, netfs_fail_write);
+ __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
+ set_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags);
+ if (transferred_or_error == -ENOBUFS)
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_no_space);
+ else if (!test_and_set_bit(NETFS_RREQ_CACHE_ERROR, &wreq->flags))
+ trace_netfs_rreq(wreq, netfs_rreq_trace_cache_failed);
+ subreq->transferred = subreq->len;
break;
+
case NETFS_UPLOAD_TO_SERVER:
+ /* If need_retry is set, error should not matter */
+ if (!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
+ set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ trace_netfs_failure(wreq, subreq, transferred_or_error,
+ netfs_fail_upload);
+ }
+
+ set_bit(NETFS_RREQ_PAUSE, &wreq->flags);
+ trace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);
netfs_stat(&netfs_n_wh_upload_failed);
break;
+
default:
break;
}
- trace_netfs_rreq(wreq, netfs_rreq_trace_set_pause);
- set_bit(NETFS_RREQ_PAUSE, &wreq->flags);
} else {
if (WARN(transferred_or_error > subreq->len - subreq->transferred,
"Subreq excess write: R=%x[%x] %zd > %zu - %zu",
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index f2761c99795a6b..3fa2aae5eaadf5 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -36,6 +36,33 @@
#include <linux/pagemap.h>
#include "internal.h"
+#define NOTE_UPLOAD_AVAIL 0x001 /* Upload is available */
+#define NOTE_CACHE_AVAIL 0x002 /* Local cache is available */
+#define NOTE_CACHE_COPY 0x004 /* Copy folio to cache */
+#define NOTE_UPLOAD 0x008 /* Upload folio to server */
+#define NOTE_UPLOAD_STARTED 0x010 /* Upload started */
+#define NOTE_STREAMW 0x020 /* Folio is from a streaming write */
+#define NOTE_DISCONTIG_BEFORE 0x040 /* Folio discontiguous with the previous folio */
+#define NOTE_DISCONTIG_AFTER 0x080 /* Folio discontiguous with the next folio */
+#define NOTE_TO_EOF 0x100 /* Data in folio ends at EOF */
+#define NOTE_FLUSH_ANYWAY 0x200 /* Flush data, even if not hit estimated limit */
+
+#define NOTES__KEEP_MASK (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL | NOTE_UPLOAD_STARTED)
+
+struct netfs_wb_params {
+ unsigned long long last_end; /* End file pos of previous folio */
+ unsigned long long folio_start; /* File pos of folio */
+ unsigned int folio_len; /* Length of folio */
+ unsigned int dirty_offset; /* Offset of dirty region in folio */
+ unsigned int dirty_len; /* Length of dirty region in folio */
+ unsigned int notes; /* Notes on applicability */
+ struct bvecq_pos dispatch_cursor; /* Folio queue anchor for issue_at */
+ struct netfs_write_estimate estimates[NR_IO_STREAMS];
+};
+
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs);
+
/*
* Kill all dirty folios in the event of an unrecoverable error, starting with
* a locked folio we've already obtained from writeback_iter().
@@ -96,7 +123,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
struct netfs_inode *ictx;
bool is_cacheable = (origin == NETFS_WRITEBACK ||
origin == NETFS_WRITEBACK_SINGLE ||
- origin == NETFS_WRITETHROUGH ||
origin == NETFS_PGPRIV2_COPY_TO_CACHE);
wreq = netfs_alloc_request(mapping, file, start, 0, origin);
@@ -108,83 +134,55 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
ictx = netfs_inode(wreq->inode);
if (is_cacheable)
fscache_begin_write_operation(&wreq->cache_resources, netfs_i_cookie(ictx));
- if (rolling_buffer_init(&wreq->buffer, wreq->debug_id, ITER_SOURCE) < 0)
- goto nomem;
wreq->cleaned_to = wreq->start;
+ if (wreq->cache_resources.dio_size > 1)
+ wreq->cache_coll_to = round_down(wreq->start, wreq->cache_resources.dio_size);
wreq->io_streams[0].stream_nr = 0;
wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER;
- wreq->io_streams[0].prepare_write = ictx->ops->prepare_write;
+ wreq->io_streams[0].applicable = NOTE_UPLOAD;
+ wreq->io_streams[0].estimate_write = ictx->ops->estimate_write;
wreq->io_streams[0].issue_write = ictx->ops->issue_write;
wreq->io_streams[0].collected_to = start;
wreq->io_streams[0].transferred = 0;
wreq->io_streams[1].stream_nr = 1;
wreq->io_streams[1].source = NETFS_WRITE_TO_CACHE;
+ wreq->io_streams[1].applicable = NOTE_CACHE_COPY;
wreq->io_streams[1].collected_to = start;
wreq->io_streams[1].transferred = 0;
if (fscache_resources_valid(&wreq->cache_resources)) {
wreq->io_streams[1].avail = true;
wreq->io_streams[1].active = true;
- wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq;
+ wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write;
wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write;
}
return wreq;
-nomem:
- netfs_put_failed_request(wreq);
- return ERR_PTR(-ENOMEM);
-}
-
-/**
- * netfs_prepare_write_failed - Note write preparation failed
- * @subreq: The subrequest to mark
- *
- * Mark a subrequest to note that preparation for write failed.
- */
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq)
-{
- __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
- trace_netfs_sreq(subreq, netfs_sreq_trace_prep_failed);
}
-EXPORT_SYMBOL(netfs_prepare_write_failed);
/*
- * Prepare a write subrequest. We need to allocate a new subrequest
- * if we don't have one.
+ * Allocate and prepare a write subrequest.
*/
-void netfs_prepare_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- loff_t start)
+struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream)
{
struct netfs_io_subrequest *subreq;
- struct iov_iter *wreq_iter = &wreq->buffer.iter;
-
- /* Make sure we don't point the iterator at a used-up folio_queue
- * struct being used as a placeholder to prevent the queue from
- * collapsing. In such a case, extend the queue.
- */
- if (iov_iter_is_folioq(wreq_iter) &&
- wreq_iter->folioq_slot >= folioq_nr_slots(wreq_iter->folioq))
- rolling_buffer_make_space(&wreq->buffer);
subreq = netfs_alloc_subrequest(wreq);
subreq->source = stream->source;
- subreq->start = start;
+ subreq->start = stream->issue_from;
+ subreq->len = stream->buffered;
subreq->stream_nr = stream->stream_nr;
- subreq->io_iter = *wreq_iter;
_enter("R=%x[%x]", wreq->debug_id, subreq->debug_index);
trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- stream->sreq_max_len = UINT_MAX;
- stream->sreq_max_segs = INT_MAX;
switch (stream->source) {
case NETFS_UPLOAD_TO_SERVER:
netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = wreq->wsize;
break;
case NETFS_WRITE_TO_CACHE:
netfs_stat(&netfs_n_wh_write);
@@ -194,9 +192,6 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
break;
}
- if (stream->prepare_write)
- stream->prepare_write(subreq);
-
__set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
/* We add to the end of the list whilst the collector may be walking
@@ -206,140 +201,377 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
spin_lock(&wreq->lock);
/* Write IN_PROGRESS before pointer to new subreq */
list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
- if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
- if (!stream->active) {
- stream->collected_to = subreq->start;
- /* Write list pointers before active flag */
- smp_store_release(&stream->active, true);
- }
- }
+ if (list_is_first(&subreq->rreq_link, &stream->subrequests) &&
+ stream->collected_to == 0)
+ stream->collected_to = subreq->start;
spin_unlock(&wreq->lock);
-
- stream->construct = subreq;
+ return subreq;
}
/*
- * Set the I/O iterator for the filesystem/cache to use and dispatch the I/O
- * operation. The operation may be asynchronous and should call
- * netfs_write_subrequest_terminated() when complete.
+ * Prepare the buffer for a buffered write.
*/
-static void netfs_do_issue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq)
+static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
struct netfs_io_request *wreq = subreq->rreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+ ssize_t len;
- _enter("R=%x[%x],%zx", wreq->debug_id, subreq->debug_index, subreq->len);
+ _enter("%zx,{,%u,%u},%u",
+ subreq->len, stream->dispatch_cursor.slot, stream->dispatch_cursor.offset, max_segs);
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- return netfs_write_subrequest_terminated(subreq, subreq->error);
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
- trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
- stream->issue_write(subreq);
+ /* If we have a write to the cache, we need to round out the first and
+ * last entries (only those as the data will be on virtually contiguous
+ * folios) to cache DIO boundaries.
+ */
+ if (subreq->source == NETFS_WRITE_TO_CACHE) {
+ struct bio_vec *bv;
+ struct bvecq *bq;
+ size_t dio_size = wreq->cache_resources.dio_size;
+ size_t disp, dlen;
+
+ len = bvecq_extract(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->content.bvecq);
+ if (len < 0)
+ return -ENOMEM;
+
+ _debug("extract %zx/%zx", len, subreq->len);
+ subreq->len = len;
+
+ /* Round the first entry down. We should be able to get away
+ * with this as this path only happens for buffered reads and
+ * writes. As such, a bio_vec must always point to a whole
+ * folio (or part thereof) in the pagecache with writeback set,
+ * so presuming that dio_size < folio size, we should be able
+ * to round out bv_offset and bv_len.
+ *
+ * Further, streaming-write pages don't get sent to the cache
+ * (and aren't normally generated if there is a cache), so we
+ * only see fully uptodate pages here.
+ */
+ bq = subreq->content.bvecq;
+ bv = &bq->bv[0];
+ disp = bv->bv_offset & (dio_size - 1);
+ if (disp) {
+ bv->bv_offset -= disp;
+ bv->bv_len += disp;
+ bq->fpos -= disp;
+ subreq->start -= disp;
+ subreq->len += disp;
+ }
+
+ /* Round the end of the last entry up. */
+ while (bq->next)
+ bq = bq->next;
+ bv = &bq->bv[bq->nr_slots - 1];
+ dlen = round_up(bv->bv_len, dio_size);
+ if (dlen > bv->bv_len) {
+ subreq->len += dlen - bv->bv_len;
+ bv->bv_len = dlen;
+ }
+ } else {
+ bvecq_pos_set(&subreq->content, &stream->dispatch_cursor);
+ len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs,
+ &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+ }
+
+ stream->issue_from += len;
+ stream->buffered -= len;
+ if (stream->buffered == 0) {
+ stream->buffering = false;
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ }
+ /* Order loading the queue before updating the issue_to point */
+ atomic64_set_release(&stream->issued_to, stream->issue_from);
+ return 0;
}
-void netfs_reissue_write(struct netfs_io_stream *stream,
- struct netfs_io_subrequest *subreq,
- struct iov_iter *source)
+/**
+ * netfs_prepare_write_buffer - Get the buffer for a subrequest
+ * @subreq: The subrequest to get the buffer for
+ * @max_segs: Maximum number of segments in buffer (or INT_MAX)
+ *
+ * Extract a slice of buffer from the stream and attach it to the subrequest as
+ * a bio_vec queue. The maximum amount of data attached is set by
+ * @subreq->len, but this may be shortened if @max_segs would be exceeded.
+ */
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- size_t size = subreq->len - subreq->transferred;
+ struct netfs_io_request *rreq = subreq->rreq;
- // TODO: Use encrypted buffer
- subreq->io_iter = *source;
- iov_iter_advance(source, size);
- iov_iter_truncate(&subreq->io_iter, size);
+ switch (rreq->origin) {
+ case NETFS_WRITEBACK:
+ if (test_bit(NETFS_RREQ_RETRYING, &rreq->flags))
+ return netfs_prepare_write_retry_buffer(subreq, max_segs);
+ return netfs_prepare_buffered_write_buffer(subreq, max_segs);
- subreq->retry_count++;
- subreq->error = 0;
- __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
- __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
- netfs_stat(&netfs_n_wh_retry_write_subreq);
- netfs_do_issue_write(stream, subreq);
+ case NETFS_UNBUFFERED_WRITE:
+ case NETFS_DIO_WRITE:
+ return netfs_prepare_unbuffered_write_buffer(subreq, max_segs);
+
+ case NETFS_WRITEBACK_SINGLE:
+ return netfs_prepare_write_single_buffer(subreq, max_segs);
+
+ case NETFS_PGPRIV2_COPY_TO_CACHE:
+ return netfs_prepare_pgpriv2_write_buffer(subreq, max_segs);
+
+ default:
+ WARN_ON_ONCE(1);
+ return -EIO;
+ }
}
+EXPORT_SYMBOL(netfs_prepare_write_buffer);
-void netfs_issue_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream)
+/*
+ * Issue writes for a stream.
+ */
+static int netfs_issue_writes(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_wb_params *params)
{
- struct netfs_io_subrequest *subreq = stream->construct;
+ struct netfs_write_estimate *estimate = ¶ms->estimates[stream->stream_nr];
- if (!subreq)
- return;
- stream->construct = NULL;
- subreq->io_iter.count = subreq->len;
- netfs_do_issue_write(stream, subreq);
+ for (;;) {
+ struct netfs_io_subrequest *subreq;
+ int ret;
+
+ if (test_bit(NETFS_RREQ_PAUSE, &wreq->flags))
+ netfs_wait_for_paused_write(wreq);
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq)
+ return -ENOMEM;
+
+ if (stream->source == NETFS_WRITE_TO_CACHE &&
+ unlikely(test_bit(NETFS_RREQ_CACHE_STOP, &wreq->flags))) {
+ size_t dio_size = wreq->cache_resources.dio_size;
+ size_t len, disp;
+
+ disp = subreq->start & (dio_size - 1);
+ len = round_up(subreq->len + disp, dio_size);
+
+ subreq->start -= disp;
+ subreq->len = len;
+
+ stream->issue_from = subreq->start + len;
+ stream->buffered = 0;
+ stream->buffering = false;
+ bvecq_pos_unset(&stream->dispatch_cursor);
+ estimate->issue_at = subreq->start + len + 16 * 1024 * 1024;
+ estimate->max_segs = INT_MAX;
+ __set_bit(NETFS_SREQ_CANCELLED, &subreq->flags);
+ netfs_write_subrequest_terminated(subreq, len);
+ return 0;
+ }
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ bvecq_pos_advance(&stream->dispatch_cursor, subreq->len);
+ stream->issue_from += subreq->len;
+ stream->buffered -= subreq->len;
+ atomic64_set_release(&stream->issued_to, stream->issue_from);
+ netfs_write_subrequest_terminated(subreq, ret);
+ }
+ /* We no longer own subreq. */
+
+ if (test_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags))
+ return -ENOMEM;
+ if (stream->buffered == 0) {
+ if (stream->stream_nr == 0)
+ params->notes &= ~NOTE_UPLOAD_STARTED;
+ return 0;
+ }
+
+ if (!(params->notes & NOTE_FLUSH_ANYWAY)) {
+ estimate->issue_at = ULLONG_MAX;
+ estimate->max_segs = INT_MAX;
+ stream->estimate_write(wreq, stream, estimate);
+ if (stream->issue_from + stream->buffered < estimate->issue_at &&
+ estimate->max_segs > 0)
+ return 0;
+ }
+ }
+}
+
+/*
+ * Issue pending writes on a stream.
+ */
+static int netfs_issue_stream(struct netfs_io_request *wreq,
+ struct netfs_wb_params *params, int s)
+{
+ struct netfs_write_estimate *estimate = ¶ms->estimates[s];
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+ unsigned long long dirty_start;
+ bool discontig_before = params->notes & NOTE_DISCONTIG_BEFORE;
+ int ret;
+
+ _enter("%x", params->notes);
+
+ /* If the current folio doesn't contribute to this stream, see if we
+ * need to flush it.
+ */
+ if (!(params->notes & stream->applicable)) {
+ if (!stream->buffering) {
+ atomic64_set_release(&stream->issued_to,
+ params->folio_start + params->folio_len);
+ return 0;
+ }
+ discontig_before = true;
+ }
+
+ /* Issue writes if we meet a discontiguity before the current folio.
+ * Even if the filesystem can do sparse/vectored writes, we still
+ * generate a subreq per contiguous region rather than generating
+ * separate extent lists.
+ */
+ if (stream->buffering && discontig_before) {
+ params->notes |= NOTE_FLUSH_ANYWAY;
+ ret = netfs_issue_writes(wreq, stream, params);
+ if (ret < 0)
+ return ret;
+ stream->buffering = false;
+ params->notes &= ~NOTE_FLUSH_ANYWAY;
+ }
+
+ if (!(params->notes & stream->applicable)) {
+ atomic64_set_release(&stream->issued_to,
+ params->folio_start + params->folio_len);
+ return 0;
+ }
+
+ /* If we're not currently buffering on this stream, we need to get an
+ * estimate of when we need to issue a write. It might be within the
+ * starting folio.
+ */
+ dirty_start = params->folio_start + params->dirty_offset;
+ if (!stream->buffering) {
+ stream->buffering = true;
+ stream->issue_from = dirty_start;
+ bvecq_pos_set(&stream->dispatch_cursor, ¶ms->dispatch_cursor);
+ estimate->issue_at = ULLONG_MAX;
+ estimate->max_segs = INT_MAX;
+ stream->estimate_write(wreq, stream, estimate);
+ }
+
+ stream->buffered += params->dirty_len;
+ estimate->max_segs--;
+
+ /* Poke the filesystem to issue writes when we hit the limit it set or
+ * if the data ends before the end of the page.
+ */
+ if (params->notes & NOTE_DISCONTIG_AFTER)
+ params->notes |= NOTE_FLUSH_ANYWAY;
+ _debug("[%u] %llx + %zx >= %llx, %u %x",
+ s, stream->issue_from, stream->buffered, estimate->issue_at,
+ estimate->max_segs, params->notes);
+ if (stream->issue_from + stream->buffered >= estimate->issue_at ||
+ estimate->max_segs <= 0 ||
+ (params->notes & NOTE_FLUSH_ANYWAY)) {
+ ret = netfs_issue_writes(wreq, stream, params);
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
}
/*
- * Add data to the write subrequest, dispatching each as we fill it up or if it
- * is discontiguous with the previous. We only fill one part at a time so that
- * we can avoid overrunning the credits obtained (cifs) and try to parallelise
- * content-crypto preparation with network writes.
+ * See which streams need writes issuing and issue them.
*/
-size_t netfs_advance_write(struct netfs_io_request *wreq,
- struct netfs_io_stream *stream,
- loff_t start, size_t len, bool to_eof)
+static int netfs_issue_streams(struct netfs_io_request *wreq,
+ struct netfs_wb_params *params)
{
- struct netfs_io_subrequest *subreq = stream->construct;
- size_t part;
+ int ret = 0, ret2;
+
+ _enter("%x", params->notes);
- if (!stream->avail) {
- _leave("no write");
- return len;
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ ret2 = netfs_issue_stream(wreq, params, s);
+ if (ret2 < 0)
+ ret = ret2;
}
+ return ret;
+}
- _enter("R=%x[%x]", wreq->debug_id, subreq ? subreq->debug_index : 0);
+/*
+ * End the issuing of writes, let the collector know we're done.
+ */
+static void netfs_end_issue_write(struct netfs_io_request *wreq,
+ struct netfs_wb_params *params)
+{
+ bool needs_poke = true;
- if (subreq && start != subreq->start + subreq->len) {
- netfs_issue_write(wreq, stream);
- subreq = NULL;
+ params->notes |= NOTE_FLUSH_ANYWAY;
+
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+ int ret;
+
+ if (stream->buffering) {
+ ret = netfs_issue_writes(wreq, stream, params);
+ if (ret < 0 && stream->source != NETFS_WRITE_TO_CACHE) {
+ /* Leave the error somewhere the completion
+ * path can pick it up if there isn't already
+ * another error logged.
+ */
+ cmpxchg(&wreq->error, 0, ret);
+ }
+ stream->buffering = false;
+ }
}
- if (!stream->construct)
- netfs_prepare_write(wreq, stream, start);
- subreq = stream->construct;
-
- part = umin(stream->sreq_max_len - subreq->len, len);
- _debug("part %zx/%zx %zx/%zx", subreq->len, stream->sreq_max_len, part, len);
- subreq->len += part;
- subreq->nr_segs++;
- stream->submit_extendable_to -= part;
-
- if (subreq->len >= stream->sreq_max_len ||
- subreq->nr_segs >= stream->sreq_max_segs ||
- to_eof) {
- netfs_issue_write(wreq, stream);
- subreq = NULL;
+ netfs_all_subreqs_queued(wreq);
+
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
+
+ if (!stream->active)
+ continue;
+ if (!list_empty(&stream->subrequests))
+ needs_poke = false;
}
- return part;
+ if (needs_poke)
+ netfs_wake_collector(wreq);
}
/*
- * Write some of a pending folio data back to the server.
+ * Queue a folio for writeback.
*/
-static int netfs_write_folio(struct netfs_io_request *wreq,
- struct writeback_control *wbc,
- struct folio *folio)
+static void netfs_queue_wb_folio(struct netfs_io_request *wreq,
+ struct writeback_control *wbc,
+ struct folio *folio,
+ struct netfs_wb_params *params)
{
- struct netfs_io_stream *upload = &wreq->io_streams[0];
- struct netfs_io_stream *cache = &wreq->io_streams[1];
- struct netfs_io_stream *stream;
struct netfs_group *fgroup; /* TODO: Use this with ceph */
struct netfs_folio *finfo;
- size_t iter_off = 0;
+ struct bvecq *queue = wreq->load_cursor.bvecq;
+ unsigned int slot;
size_t fsize = folio_size(folio), flen = fsize, foff = 0;
loff_t fpos = folio_pos(folio), i_size;
- bool to_eof = false, streamw = false;
- bool debug = false;
- _enter("");
+ _enter("%x", params->notes);
- if (rolling_buffer_make_space(&wreq->buffer) < 0)
- return -ENOMEM;
-
- /* netfs_perform_write() may shift i_size around the page or from out
- * of the page to beyond it, but cannot move i_size into or through the
- * page since we have it locked.
+ /* netfs_perform_write() may shift i_size around the folio or from out
+ * of the folio to beyond it, but cannot move i_size into or through
+ * the folio since we have it locked.
+ *
+ * Truncate could in theory move i_size into or before the folio, but
+ * it should take steps to prevent writeback from happening
+ * concurrently and should wait for any in-progress writebacks before
+ * proceeding.
*/
i_size = i_size_read(wreq->inode);
@@ -351,7 +583,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
wreq->nr_group_rel += netfs_folio_written_back(folio);
netfs_put_group_many(wreq->group, wreq->nr_group_rel);
wreq->nr_group_rel = 0;
- return 0;
+ return;
}
if (fpos + fsize > wreq->i_size)
@@ -362,23 +594,32 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
if (finfo) {
foff = finfo->dirty_offset;
flen = foff + finfo->dirty_len;
- streamw = true;
+ params->notes |= NOTE_STREAMW;
+ if (foff > 0)
+ params->notes |= NOTE_DISCONTIG_BEFORE;
+ if (flen < fsize)
+ params->notes |= NOTE_DISCONTIG_AFTER;
}
- if (wreq->origin == NETFS_WRITETHROUGH) {
- to_eof = false;
- if (flen > i_size - fpos)
- flen = i_size - fpos;
- } else if (flen > i_size - fpos) {
+ if (params->last_end && fpos != params->last_end)
+ params->notes |= NOTE_DISCONTIG_BEFORE;
+ params->last_end = fpos + fsize;
+
+ if (flen > i_size - fpos) {
flen = i_size - fpos;
- if (!streamw)
+ if (!(params->notes & NOTE_STREAMW))
folio_zero_segment(folio, flen, fsize);
- to_eof = true;
+ params->notes |= NOTE_TO_EOF;
} else if (flen == i_size - fpos) {
- to_eof = true;
+ params->notes |= NOTE_TO_EOF;
}
flen -= foff;
+ params->folio_start = fpos;
+ params->folio_len = fsize;
+ params->dirty_offset = foff;
+ params->dirty_len = flen;
+
_debug("folio %zx %zx %zx", foff, flen, fsize);
/* Deal with discontinuities in the stream of dirty pages. These can
@@ -398,146 +639,80 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
* write-back group.
*/
if (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {
- netfs_issue_write(wreq, upload);
- } else if (fgroup != wreq->group) {
- /* We can't write this page to the server yet. */
- kdebug("wrong group");
- folio_redirty_for_writepage(wbc, folio);
- folio_unlock(folio);
- netfs_issue_write(wreq, upload);
- netfs_issue_write(wreq, cache);
- return 0;
- }
-
- if (foff > 0)
- netfs_issue_write(wreq, upload);
- if (streamw)
- netfs_issue_write(wreq, cache);
-
- folio_start_writeback(folio);
- folio_unlock(folio);
-
- if (fgroup == NETFS_FOLIO_COPY_TO_CACHE) {
- if (!cache->avail) {
+ if (!(params->notes & NOTE_CACHE_AVAIL)) {
trace_netfs_folio(folio, netfs_folio_trace_cancel_copy);
- netfs_issue_write(wreq, upload);
- netfs_folio_written_back(folio);
- return 0;
+ goto cancel_folio;
}
+ params->notes |= NOTE_CACHE_COPY;
trace_netfs_folio(folio, netfs_folio_trace_store_copy);
- } else if (!upload->avail && !cache->avail) {
+ } else if (fgroup != wreq->group) {
+ /* We can't write this page to the server yet. */
+ kdebug("wrong group");
+ goto skip_folio;
+ } else if (!(params->notes & (NOTE_UPLOAD_AVAIL | NOTE_CACHE_AVAIL))) {
trace_netfs_folio(folio, netfs_folio_trace_cancel_store);
- netfs_folio_written_back(folio);
- return 0;
- } else if (!upload->construct) {
- trace_netfs_folio(folio, netfs_folio_trace_store);
+ goto cancel_folio_discard;
} else {
- trace_netfs_folio(folio, netfs_folio_trace_store_plus);
- }
-
- /* Attach the folio to the rolling buffer. */
- rolling_buffer_append(&wreq->buffer, folio, 0);
-
- /* Move the submission point forward to allow for write-streaming data
- * not starting at the front of the page. We don't do write-streaming
- * with the cache as the cache requires DIO alignment.
- *
- * Also skip uploading for data that's been read and just needs copying
- * to the cache.
- */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- stream->submit_off = foff;
- stream->submit_len = flen;
- if (!stream->avail ||
- (stream->source == NETFS_WRITE_TO_CACHE && streamw) ||
- (stream->source == NETFS_UPLOAD_TO_SERVER &&
- fgroup == NETFS_FOLIO_COPY_TO_CACHE)) {
- stream->submit_off = UINT_MAX;
- stream->submit_len = 0;
+ if (params->notes & NOTE_UPLOAD_STARTED) {
+ params->notes |= NOTE_UPLOAD;
+ trace_netfs_folio(folio, netfs_folio_trace_store_plus);
+ } else {
+ params->notes |= NOTE_UPLOAD | NOTE_UPLOAD_STARTED;
+ trace_netfs_folio(folio, netfs_folio_trace_store);
}
+ if ((params->notes & NOTE_CACHE_AVAIL) &&
+ !(params->notes & NOTE_STREAMW))
+ params->notes |= NOTE_CACHE_COPY;
}
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- for (;;) {
- ssize_t part;
- size_t lowest_off = ULONG_MAX;
- int choose_s = -1;
-
- /* Always add to the lowest-submitted stream first. */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- if (stream->submit_len > 0 &&
- stream->submit_off < lowest_off) {
- lowest_off = stream->submit_off;
- choose_s = s;
- }
- }
-
- if (choose_s < 0)
- break;
- stream = &wreq->io_streams[choose_s];
-
- /* Advance the iterator(s). */
- if (stream->submit_off > iter_off) {
- rolling_buffer_advance(&wreq->buffer, stream->submit_off - iter_off);
- iter_off = stream->submit_off;
- }
+ folio_start_writeback(folio);
+ folio_unlock(folio);
- atomic64_set(&wreq->issued_to, fpos + stream->submit_off);
- stream->submit_extendable_to = fsize - stream->submit_off;
- part = netfs_advance_write(wreq, stream, fpos + stream->submit_off,
- stream->submit_len, to_eof);
- stream->submit_off += part;
- if (part > stream->submit_len)
- stream->submit_len = 0;
- else
- stream->submit_len -= part;
- if (part > 0)
- debug = true;
+ /* Institute a new bvec queue segment if the current one is full or if
+ * we encounter a discontiguity. The discontiguity break is important
+ * when it comes to bulk unlocking folios by file range.
+ */
+ if (bvecq_is_full(queue) ||
+ ((params->notes & NOTE_DISCONTIG_BEFORE) && queue->nr_slots > 0)) {
+ bvecq_buffer_append(&wreq->load_cursor, wreq->spare);
+ wreq->spare = NULL;
+
+ queue = wreq->load_cursor.bvecq;
+ queue->fpos = fpos;
+ if (params->notes & NOTE_DISCONTIG_BEFORE)
+ queue->discontig = true;
+ bvecq_pos_move(¶ms->dispatch_cursor, queue);
+ params->dispatch_cursor.slot = 0;
}
- if (fsize > iter_off)
- rolling_buffer_advance(&wreq->buffer, fsize - iter_off);
- atomic64_set(&wreq->issued_to, fpos + fsize);
-
- if (!debug)
- kdebug("R=%x: No submit", wreq->debug_id);
-
- if (foff + flen < fsize)
- for (int s = 0; s < NR_IO_STREAMS; s++)
- netfs_issue_write(wreq, &wreq->io_streams[s]);
-
- _leave(" = 0");
- return 0;
-}
-
-/*
- * End the issuing of writes, letting the collector know we're done.
- */
-static void netfs_end_issue_write(struct netfs_io_request *wreq)
-{
- bool needs_poke = true;
-
- smp_wmb(); /* Write subreq lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags);
-
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- struct netfs_io_stream *stream = &wreq->io_streams[s];
+ /* Attach the folio to the rolling buffer. */
+ slot = queue->nr_slots;
+ bvec_set_folio(&queue->bv[slot], folio, flen, foff);
+ trace_netfs_bv_slot(queue, slot);
+ slot++;
+ bvecq_filled_to(queue, slot);
+ wreq->load_cursor.slot = slot;
+ wreq->load_cursor.offset = 0;
+ trace_netfs_wback(wreq, folio, params->notes);
- if (!stream->active)
- continue;
- if (!list_empty(&stream->subrequests))
- needs_poke = false;
- netfs_issue_write(wreq, stream);
- }
+out:
+ _leave(" = %x", params->notes);
+ return;
- if (needs_poke)
- netfs_wake_collector(wreq);
+skip_folio:
+ folio_redirty_for_writepage(wbc, folio);
+ folio_unlock(folio);
+ params->notes |= NOTE_DISCONTIG_BEFORE;
+ goto out;
+cancel_folio_discard:
+ netfs_put_group(fgroup);
+cancel_folio:
+ folio_detach_private(folio);
+ kfree(finfo);
+ folio_unlock(folio);
+ folio_cancel_dirty(folio);
+ params->notes |= NOTE_DISCONTIG_BEFORE;
+ goto out;
}
/*
@@ -548,6 +723,7 @@ int netfs_writepages(struct address_space *mapping,
{
struct netfs_inode *ictx = netfs_inode(mapping->host);
struct netfs_io_request *wreq = NULL;
+ struct netfs_wb_params params = {};
struct folio *folio;
int error = 0;
@@ -565,36 +741,65 @@ int netfs_writepages(struct address_space *mapping,
goto couldnt_start;
}
+ if (bvecq_buffer_init(&wreq->load_cursor, GFP_NOFS) < 0)
+ goto nomem;
+ bvecq_pos_set(¶ms.dispatch_cursor, &wreq->load_cursor);
+ bvecq_pos_set(&wreq->collect_cursor, &wreq->load_cursor);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback);
netfs_stat(&netfs_n_wh_writepages);
- do {
- _debug("wbiter %lx %llx", folio->index, atomic64_read(&wreq->issued_to));
+ if (wreq->io_streams[1].avail)
+ params.notes |= NOTE_CACHE_AVAIL;
- /* It appears we don't have to handle cyclic writeback wrapping. */
- WARN_ON_ONCE(wreq && folio_pos(folio) < atomic64_read(&wreq->issued_to));
+ do {
+ _debug("wbiter %lx", folio->index);
+
+ if (!wreq->spare) {
+ wreq->spare = bvecq_alloc_one(BVECQ_STD_SLOTS, GFP_NOFS);
+ if (!wreq->spare) {
+ folio_redirty_for_writepage(wbc, folio);
+ folio_unlock(folio);
+ error = -ENOMEM;
+ goto end_loop;
+ }
+ }
if (netfs_folio_group(folio) != NETFS_FOLIO_COPY_TO_CACHE &&
unlikely(!test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))) {
set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags);
wreq->netfs_ops->begin_writeback(wreq);
+ if (wreq->io_streams[0].avail) {
+ params.notes |= NOTE_UPLOAD_AVAIL;
+ /* Order setting the active flag after other fields. */
+ smp_store_release(&wreq->io_streams[0].active, true);
+ }
}
- error = netfs_write_folio(wreq, wbc, folio);
- if (error == -ENOMEM) {
- folio_redirty_for_writepage(wbc, folio);
- folio_unlock(folio);
- }
+ params.notes &= NOTES__KEEP_MASK;
+ netfs_queue_wb_folio(wreq, wbc, folio, ¶ms);
+ error = netfs_issue_streams(wreq, ¶ms);
+ if (!error)
+ bvecq_pos_step(¶ms.dispatch_cursor);
+end_loop:
} while ((folio = writeback_iter(mapping, wbc, folio, &error)));
- netfs_end_issue_write(wreq);
+ netfs_end_issue_write(wreq, ¶ms);
+
+ bvecq_pos_unset(&wreq->load_cursor);
+ bvecq_pos_unset(¶ms.dispatch_cursor);
+ for (int i = 0; i < NR_IO_STREAMS; i++)
+ bvecq_pos_unset(&wreq->io_streams[i].dispatch_cursor);
netfs_wake_collector(wreq);
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", error);
return error;
+nomem:
+ error = -ENOMEM;
+ netfs_put_failed_request(wreq);
couldnt_start:
if (error == -ENOMEM) {
folio_redirty_for_writepage(wbc, folio);
@@ -612,212 +817,36 @@ int netfs_writepages(struct address_space *mapping,
EXPORT_SYMBOL(netfs_writepages);
/*
- * Begin a write operation for writing through the pagecache.
- */
-struct netfs_io_request *netfs_begin_writethrough(struct kiocb *iocb, size_t len)
-{
- struct netfs_io_request *wreq = NULL;
- struct netfs_inode *ictx = netfs_inode(file_inode(iocb->ki_filp));
-
- netfs_wb_begin(ictx, false);
-
- wreq = netfs_create_write_req(iocb->ki_filp->f_mapping, iocb->ki_filp,
- iocb->ki_pos, NETFS_WRITETHROUGH);
- if (IS_ERR(wreq)) {
- netfs_wb_end(ictx);
- return wreq;
- }
-
- wreq->io_streams[0].avail = true;
- __set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
- trace_netfs_write(wreq, netfs_write_trace_writethrough);
- return wreq;
-}
-
-/*
- * Advance the state of the write operation used when writing through the
- * pagecache. Data has been copied into the pagecache that we need to append
- * to the request. If we've added more than wsize then we need to create a new
- * subrequest.
+ * Prepare a buffer for a single monolithic write.
*/
-int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *folio, size_t copied, bool to_page_end,
- struct folio **writethrough_cache)
+static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
{
- int ret;
-
- _enter("R=%x ic=%zu ws=%u cp=%zu tp=%u",
- wreq->debug_id, wreq->buffer.iter.count, wreq->wsize, copied, to_page_end);
-
- /* The folio is locked. */
-
- if (*writethrough_cache != folio) {
- if (*writethrough_cache) {
- /* Did the folio get moved? */
- folio_put(*writethrough_cache);
- *writethrough_cache = NULL;
- }
- /* We can make multiple writes to the folio... */
- if (wreq->len == 0)
- trace_netfs_folio(folio, netfs_folio_trace_wthru);
- else
- trace_netfs_folio(folio, netfs_folio_trace_wthru_plus);
- *writethrough_cache = folio;
- folio_get(folio);
- }
-
- wreq->len += copied;
-
- if (!to_page_end) {
- folio_mark_dirty(folio);
- folio_unlock(folio);
- return 0;
- }
-
- ret = netfs_write_folio(wreq, wbc, folio);
- folio_put(*writethrough_cache);
- *writethrough_cache = NULL;
- wreq->submitted = wreq->len;
- return ret;
-}
-
-/*
- * End a write operation used when writing through the pagecache.
- */
-ssize_t netfs_end_writethrough(struct netfs_io_request *wreq, struct writeback_control *wbc,
- struct folio *writethrough_cache)
-{
- ssize_t ret;
-
- _enter("R=%x", wreq->debug_id);
-
- if (writethrough_cache) {
- folio_lock(writethrough_cache);
- netfs_write_folio(wreq, wbc, writethrough_cache);
- folio_put(writethrough_cache);
- wreq->submitted = wreq->len;
- }
-
- netfs_end_issue_write(wreq);
-
- if (wreq->iocb)
- ret = -EIOCBQUEUED;
- else
- ret = netfs_wait_for_write(wreq);
- netfs_put_request(wreq, netfs_rreq_trace_put_return);
- return ret;
-}
-
-/*
- * Write some of a pending folio data back to the server and/or the cache.
- */
-static int netfs_write_folio_single(struct netfs_io_request *wreq,
- struct folio *folio)
-{
- struct netfs_io_stream *upload = &wreq->io_streams[0];
- struct netfs_io_stream *cache = &wreq->io_streams[1];
- struct netfs_io_stream *stream;
- size_t iter_off = 0;
- size_t fsize = folio_size(folio), flen;
- loff_t fpos = folio_pos(folio);
- bool to_eof = false;
- bool no_debug = false;
-
- _enter("");
-
- flen = folio_size(folio);
- if (flen > wreq->i_size - fpos) {
- flen = wreq->i_size - fpos;
- folio_zero_segment(folio, flen, fsize);
- to_eof = true;
- } else if (flen == wreq->i_size - fpos) {
- to_eof = true;
- }
-
- _debug("folio %zx/%zx", flen, fsize);
-
- if (!upload->avail && !cache->avail) {
- trace_netfs_folio(folio, netfs_folio_trace_cancel_store);
- return 0;
- }
-
- if (!upload->construct)
- trace_netfs_folio(folio, netfs_folio_trace_store);
- else
- trace_netfs_folio(folio, netfs_folio_trace_store_plus);
-
- /* Attach the folio to the rolling buffer. */
- folio_get(folio);
- rolling_buffer_append(&wreq->buffer, folio, NETFS_ROLLBUF_PUT_MARK);
-
- /* Move the submission point forward to allow for write-streaming data
- * not starting at the front of the page. We don't do write-streaming
- * with the cache as the cache requires DIO alignment.
- *
- * Also skip uploading for data that's been read and just needs copying
- * to the cache.
- */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- stream->submit_off = 0;
- stream->submit_len = flen;
- if (!stream->avail) {
- stream->submit_off = UINT_MAX;
- stream->submit_len = 0;
- }
- }
-
- /* Attach the folio to one or more subrequests. For a big folio, we
- * could end up with thousands of subrequests if the wsize is small -
- * but we might need to wait during the creation of subrequests for
- * network resources (eg. SMB credits).
- */
- for (;;) {
- ssize_t part;
- size_t lowest_off = ULONG_MAX;
- int choose_s = -1;
-
- /* Always add to the lowest-submitted stream first. */
- for (int s = 0; s < NR_IO_STREAMS; s++) {
- stream = &wreq->io_streams[s];
- if (stream->submit_len > 0 &&
- stream->submit_off < lowest_off) {
- lowest_off = stream->submit_off;
- choose_s = s;
- }
- }
-
- if (choose_s < 0)
- break;
- stream = &wreq->io_streams[choose_s];
-
- /* Advance the iterator(s). */
- if (stream->submit_off > iter_off) {
- rolling_buffer_advance(&wreq->buffer, stream->submit_off - iter_off);
- iter_off = stream->submit_off;
- }
-
- atomic64_set(&wreq->issued_to, fpos + stream->submit_off);
- stream->submit_extendable_to = fsize - stream->submit_off;
- part = netfs_advance_write(wreq, stream, fpos + stream->submit_off,
- stream->submit_len, to_eof);
- stream->submit_off += part;
- if (part > stream->submit_len)
- stream->submit_len = 0;
- else
- stream->submit_len -= part;
- if (part > 0)
- no_debug = true;
+ struct netfs_io_request *wreq = subreq->rreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr];
+ struct bio_vec *bv;
+ struct bvecq *bq;
+ size_t dio_size = wreq->cache_resources.dio_size;
+ size_t dlen;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor);
+ bvecq_pos_set(&subreq->content, &subreq->dispatch_pos);
+
+ /* Round the end of the last entry up. */
+ bq = subreq->content.bvecq;
+ while (bq->next)
+ bq = bq->next;
+ bv = &bq->bv[bq->nr_slots - 1];
+ dlen = round_up(bv->bv_len, dio_size);
+ if (dlen > bv->bv_len) {
+ subreq->len += dlen - bv->bv_len;
+ bv->bv_len = dlen;
}
- wreq->buffer.iter.iov_offset = 0;
- if (fsize > iter_off)
- rolling_buffer_advance(&wreq->buffer, fsize - iter_off);
- atomic64_set(&wreq->issued_to, fpos + fsize);
-
- if (!no_debug)
- kdebug("R=%x: No submit", wreq->debug_id);
- _leave(" = 0");
+ stream->buffered = 0;
+ stream->issue_from = subreq->len;
+ wreq->submitted = subreq->len;
+ netfs_all_subreqs_queued(wreq);
return 0;
}
@@ -825,26 +854,29 @@ static int netfs_write_folio_single(struct netfs_io_request *wreq,
* netfs_writeback_single - Write back a monolithic payload
* @mapping: The mapping to write from
* @wbc: Hints from the VM
- * @iter: Data to write, must be ITER_FOLIOQ.
+ * @iter: Data to write
+ * @len: Amount of data to write
*
* Write a monolithic, non-pagecache object back to the server and/or
- * the cache.
+ * the cache. There's a maximum of one subrequest per stream.
*
* Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE;
* or a negative error code.
+ * the cache. There's a maximum of one subrequest per stream.
*/
int netfs_writeback_single(struct address_space *mapping,
struct writeback_control *wbc,
- struct iov_iter *iter)
+ struct iov_iter *iter,
+ size_t len)
{
struct netfs_io_request *wreq;
struct netfs_inode *ictx = netfs_inode(mapping->host);
- struct folio_queue *fq;
- size_t size = iov_iter_count(iter);
int ret;
- if (WARN_ON_ONCE(!iov_iter_is_folioq(iter)))
- return -EIO;
+ _enter("%zx,%zx", iov_iter_count(iter), len);
+
+ if (!len)
+ return 0;
if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) {
/* The VFS will have undirtied the inode. */
@@ -858,41 +890,72 @@ int netfs_writeback_single(struct address_space *mapping,
goto couldnt_start;
}
+ wreq->len = len;
+
+ ret = netfs_extract_iter(iter, len, INT_MAX, 0, &wreq->load_cursor.bvecq, 0);
+ if (ret < 0)
+ goto cleanup_free;
+ if (ret < len) {
+ ret = -EIO;
+ goto cleanup_free;
+ }
+
+ bvecq_pos_set(&wreq->collect_cursor, &wreq->load_cursor);
+
__set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags);
trace_netfs_write(wreq, netfs_write_trace_writeback_single);
netfs_stat(&netfs_n_wh_writepages);
- if (__test_and_set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
+ if (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags))
wreq->netfs_ops->begin_writeback(wreq);
- for (fq = (struct folio_queue *)iter->folioq; fq; fq = fq->next) {
- for (int slot = 0; slot < folioq_count(fq); slot++) {
- struct folio *folio = folioq_folio(fq, slot);
- size_t part = umin(folioq_folio_size(fq, slot), size);
+ for (int s = 0; s < NR_IO_STREAMS; s++) {
+ struct netfs_io_subrequest *subreq;
+ struct netfs_io_stream *stream = &wreq->io_streams[s];
- _debug("wbiter %lx %llx", folio->index, atomic64_read(&wreq->issued_to));
+ if (!stream->avail)
+ continue;
- ret = netfs_write_folio_single(wreq, folio);
- if (ret < 0)
- goto stop;
- size -= part;
- if (size <= 0)
- goto stop;
+ stream->issue_from = 0;
+ stream->buffered = len;
+
+ subreq = netfs_alloc_write_subreq(wreq, stream);
+ if (!subreq) {
+ ret = -ENOMEM;
+ break;
}
+
+ bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor);
+
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us. */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ stream->buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ /* Punt the error to the collector. */
+ }
+
+ bvecq_pos_unset(&stream->dispatch_cursor);
}
-stop:
- for (int s = 0; s < NR_IO_STREAMS; s++)
- netfs_issue_write(wreq, &wreq->io_streams[s]);
- smp_wmb(); /* Write lists before ALL_QUEUED. */
- set_bit(NETFS_RREQ_ALL_QUEUED, &wreq->flags);
+ wreq->submitted = wreq->len;
+ if (unlikely(!netfs_are_all_subreqs_queued(wreq))) {
+ netfs_all_subreqs_queued(wreq);
+ netfs_wake_collector(wreq);
+ }
- netfs_wake_collector(wreq);
+ /* TODO: Might want to be async here if WB_SYNC_NONE, but then need to
+ * wait before modifying.
+ */
+ ret = netfs_wait_for_write(wreq);
netfs_put_request(wreq, netfs_rreq_trace_put_return);
_leave(" = %d", ret);
return ret;
+cleanup_free:
+ netfs_put_failed_request(wreq);
couldnt_start:
netfs_wb_end(ictx);
_leave(" = %d", ret);
diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c
index 058bc7a166a59f..4baac57f577417 100644
--- a/fs/netfs/write_retry.c
+++ b/fs/netfs/write_retry.c
@@ -12,52 +12,66 @@
#include "internal.h"
/*
- * Perform retries on the streams that need it.
+ * Prepare the write buffer for a retry. We can't necessarily reuse the write
+ * buffer from the previous run of a subrequest because the filesystem is
+ * permitted to modify it (add headers/trailers, encrypt it). Further, the
+ * subrequest may now be a different size (e.g. cifs has to negotiate for
+ * maximum transfer size). Also, we can't look at *stream as that may still
+ * refer to the source material being broken up into original subrequests.
+ */
+int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq,
+ unsigned int max_segs)
+{
+ struct netfs_io_request *wreq = subreq->rreq;
+ size_t len;
+
+ bvecq_pos_set(&subreq->dispatch_pos, &wreq->retry_cursor);
+ bvecq_pos_set(&subreq->content, &wreq->retry_cursor);
+ len = bvecq_slice(&wreq->retry_cursor, subreq->len, max_segs, &subreq->nr_segs);
+
+ if (len < subreq->len) {
+ subreq->len = len;
+ trace_netfs_sreq(subreq, netfs_sreq_trace_limited);
+ }
+
+ wreq->retry_start += len;
+ wreq->retry_buffered -= len;
+ if (wreq->retry_buffered == 0)
+ bvecq_pos_unset(&wreq->retry_cursor);
+ return 0;
+}
+
+/*
+ * Perform retries on the streams that need it. This only has to deal with
+ * buffered writes; unbuffered write retry is handled in direct_write.c.
*/
static void netfs_retry_write_stream(struct netfs_io_request *wreq,
struct netfs_io_stream *stream)
{
struct list_head *next;
+ int ret;
_enter("R=%x[%x:]", wreq->debug_id, stream->stream_nr);
if (list_empty(&stream->subrequests))
return;
+ if (WARN_ON_ONCE(stream->source != NETFS_UPLOAD_TO_SERVER))
+ return; /* Shouldn't be retrying cache writes. */
- if (stream->source == NETFS_UPLOAD_TO_SERVER &&
- wreq->netfs_ops->retry_request)
+ if (wreq->netfs_ops->retry_request)
wreq->netfs_ops->retry_request(wreq, stream);
if (unlikely(stream->failed))
return;
- /* If there's no renegotiation to do, just resend each failed subreq. */
- if (!stream->prepare_write) {
- struct netfs_io_subrequest *subreq;
-
- list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
- if (test_bit(NETFS_SREQ_FAILED, &subreq->flags))
- break;
- if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) {
- struct iov_iter source;
-
- netfs_reset_iter(subreq);
- source = subreq->io_iter;
- netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq, &source);
- }
- }
- return;
- }
-
- next = stream->subrequests.next;
+ /* Read pointer to subreq before reading subreq state. */
+ next = smp_load_acquire(&stream->subrequests.next);
do {
struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp;
- struct iov_iter source;
unsigned long long start, len;
- size_t part;
- bool boundary = false;
+
+ bvecq_pos_unset(&wreq->retry_cursor);
/* Go through the stream and find the next span of contiguous
* data that we then rejig (cifs, for example, needs the wsize
@@ -70,7 +84,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
if (test_bit(NETFS_SREQ_FAILED, &from->flags) ||
!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags))
- return;
+ goto out;
for (;;) {
/* Read pointer to subreq before reading subreq state. */
@@ -79,8 +93,8 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
break;
subreq = list_entry(next, struct netfs_io_subrequest, rreq_link);
- if (subreq->start + subreq->transferred != start + len ||
- test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) ||
+ if (subreq->start != start + len ||
+ subreq->transferred > 0 ||
!test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags))
break;
to = subreq;
@@ -90,38 +104,48 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
/* Determine the set of buffers we're going to use. Each
* subreq gets a subset of a single overall contiguous buffer.
*/
- netfs_reset_iter(from);
- source = from->io_iter;
- source.count = len;
-
- /* Work through the sublist. */
+ bvecq_pos_transfer(&wreq->retry_cursor, &from->dispatch_pos);
+ bvecq_pos_advance(&wreq->retry_cursor, from->transferred);
+ wreq->retry_start = start;
+ wreq->retry_buffered = len;
+
+ /* Work through the sublist. The chain of buffers we're going
+ * to fill is attached to dispatch_cursor and we need to read
+ * 'len' amount of data from 'start'.
+ */
subreq = from;
list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) {
- if (!len)
+ if (!wreq->retry_buffered)
break;
- subreq->start = start;
- subreq->len = len;
+ bvecq_pos_unset(&subreq->content);
+ bvecq_pos_unset(&subreq->dispatch_pos);
+
+ __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
__clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags);
+ __clear_bit(NETFS_SREQ_FAILED, &subreq->flags);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
+ subreq->start = wreq->retry_start;
+ subreq->len = wreq->retry_buffered;
+ subreq->transferred = 0;
+ subreq->retry_count += 1;
+ subreq->error = 0;
+
+ netfs_stat(&netfs_n_wh_retry_write_subreq);
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
-
- /* Renegotiate max_len (wsize) */
- stream->sreq_max_len = len;
- stream->prepare_write(subreq);
-
- part = umin(len, stream->sreq_max_len);
- if (unlikely(stream->sreq_max_segs))
- part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
- subreq->len = part;
- subreq->transferred = 0;
- len -= part;
- start += part;
- if (len && subreq == to &&
- __test_and_clear_bit(NETFS_SREQ_BOUNDARY, &to->flags))
- boundary = true;
-
netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit);
- netfs_reissue_write(stream, subreq, &source);
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
+ break;
+ }
+
if (subreq == to)
break;
}
@@ -129,7 +153,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
/* If we managed to use fewer subreqs, we can discard the
* excess; if we used the same number, then we're done.
*/
- if (!len) {
+ if (!wreq->retry_buffered) {
if (subreq == to)
continue;
list_for_each_entry_safe_from(subreq, tmp,
@@ -152,6 +176,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
subreq = netfs_alloc_subrequest(wreq);
subreq->source = to->source;
subreq->start = start;
+ subreq->len = wreq->retry_buffered;
subreq->stream_nr = to->stream_nr;
subreq->retry_count = 1;
@@ -160,44 +185,40 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
netfs_sreq_trace_new);
trace_netfs_sreq(subreq, netfs_sreq_trace_split);
+ __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags);
spin_lock(&wreq->lock);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ smp_wmb();
list_add(&subreq->rreq_link, &to->rreq_link);
spin_unlock(&wreq->lock);
to = subreq;
- trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- stream->sreq_max_len = len;
- stream->sreq_max_segs = INT_MAX;
switch (stream->source) {
case NETFS_UPLOAD_TO_SERVER:
netfs_stat(&netfs_n_wh_upload);
- stream->sreq_max_len = umin(len, wreq->wsize);
- break;
- case NETFS_WRITE_TO_CACHE:
- netfs_stat(&netfs_n_wh_write);
break;
default:
WARN_ON_ONCE(1);
}
- stream->prepare_write(subreq);
-
- part = umin(len, stream->sreq_max_len);
- subreq->len = subreq->transferred + part;
- len -= part;
- start += part;
- if (!len && boundary) {
- __set_bit(NETFS_SREQ_BOUNDARY, &to->flags);
- boundary = false;
- }
-
- netfs_reissue_write(stream, subreq, &source);
- if (!len)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ ret = stream->issue_write(subreq);
+ if (ret < 0) {
+ /* Ownership of subreq was returned to us.
+ * Expand the subreq to consume the entire
+ * remaining amount to be retried and fail it.
+ */
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
+ wreq->retry_buffered -= subreq->len;
+ netfs_write_subrequest_terminated(subreq, ret);
break;
-
- } while (len);
+ }
+ } while (wreq->retry_buffered > 0);
} while (!list_is_head(next, &stream->subrequests));
+
+out:
+ bvecq_pos_unset(&wreq->retry_cursor);
}
/*
@@ -211,6 +232,7 @@ void netfs_retry_writes(struct netfs_io_request *wreq)
int s;
netfs_stat(&netfs_n_wh_retry_write_req);
+ trace_netfs_rreq(wreq, netfs_rreq_trace_retry_begin);
/* Wait for all outstanding I/O to quiesce before performing retries as
* we may need to renegotiate the I/O sizes.
@@ -235,4 +257,6 @@ void netfs_retry_writes(struct netfs_io_request *wreq)
netfs_retry_write_stream(wreq, stream);
}
}
+
+ trace_netfs_rreq(wreq, netfs_rreq_trace_retry_end);
}
diff --git a/fs/nfs/Kconfig b/fs/nfs/Kconfig
index 6bb30543eff00f..e7862f35b72c8c 100644
--- a/fs/nfs/Kconfig
+++ b/fs/nfs/Kconfig
@@ -174,6 +174,7 @@ config NFS_FSCACHE
bool "Provide NFS client caching support"
depends on NFS_FS
select NETFS_SUPPORT
+ select NETFS_PGPRIV2
select FSCACHE
help
Say Y here if you want NFS data to be cached locally on disc through
diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index 9b7fdad4a92019..fb1441f88661a4 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c
@@ -23,6 +23,7 @@
#include "iostat.h"
#include "fscache.h"
#include "nfstrace.h"
+#include <trace/events/netfs.h>
#define NFS_MAX_KEY_LEN 1000
@@ -273,8 +274,6 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fi
rreq->debug_id = atomic_inc_return(&nfs_netfs_debug_id);
/* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */
__set_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags);
- rreq->io_streams[0].sreq_max_len = NFS_SB(rreq->inode->i_sb)->rsize;
-
return 0;
}
@@ -296,8 +295,9 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sre
return netfs;
}
-static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
+static int nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
{
+ struct netfs_io_request *rreq = sreq->rreq;
struct nfs_netfs_io_data *netfs;
struct nfs_pageio_descriptor pgio;
struct inode *inode = sreq->rreq->inode;
@@ -307,6 +307,16 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
pgoff_t start, last;
int err;
+ if (sreq->len > NFS_SB(rreq->inode->i_sb)->rsize)
+ sreq->len = NFS_SB(rreq->inode->i_sb)->rsize;
+
+ err = netfs_prepare_read_buffer(sreq, INT_MAX);
+ if (err < 0) {
+ sreq->error = err;
+ return err;
+ }
+ /* After this point, must fail by termination. */
+
start = (sreq->start + sreq->transferred) >> PAGE_SHIFT;
last = ((sreq->start + sreq->len - sreq->transferred - 1) >> PAGE_SHIFT);
@@ -316,12 +326,15 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
netfs = nfs_netfs_alloc(sreq);
if (!netfs) {
sreq->error = -ENOMEM;
- return netfs_read_subreq_terminated(sreq);
+ netfs_read_subreq_terminated(sreq);
+ return 0;
}
+ trace_netfs_sreq(sreq, netfs_sreq_trace_submit);
+
pgio.pg_netfs = netfs; /* used in completion */
- xa_for_each_range(&sreq->rreq->mapping->i_pages, idx, page, start, last) {
+ xa_for_each_range(&rreq->mapping->i_pages, idx, page, start, last) {
/* nfs_read_add_folio() may schedule() due to pNFS layout and other RPCs */
err = nfs_read_add_folio(&pgio, ctx, page_folio(page));
if (err < 0) {
@@ -332,6 +345,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
out:
nfs_pageio_complete_read(&pgio);
nfs_netfs_put(netfs);
+ return 0;
}
void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr)
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index 08e94633a9c1e2..e06cae4267dc39 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -289,7 +289,7 @@ struct smb_rqst {
struct kvec *rq_iov; /* array of kvecs */
unsigned int rq_nvec; /* number of kvecs in array */
struct iov_iter rq_iter; /* Data iterator */
- struct folio_queue *rq_buffer; /* Buffer for encryption */
+ struct bvecq *rq_buffer; /* Buffer for encryption */
};
struct mid_q_entry;
diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c
index 40162d5554eacc..fec824c1b3c596 100644
--- a/fs/smb/client/cifssmb.c
+++ b/fs/smb/client/cifssmb.c
@@ -1467,8 +1467,7 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
struct cifs_tcon *tcon = tlink_tcon(rdata->req->cfile->tlink);
struct inode *inode = &ictx->inode;
struct smb_rqst rqst = { .rq_iov = rdata->iov,
- .rq_nvec = 1,
- .rq_iter = rdata->subreq.io_iter };
+ .rq_nvec = 1};
struct cifs_credits credits = {
.value = 1,
.instance = 0,
@@ -1482,6 +1481,11 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
__func__, mid->mid, mid->mid_state, rdata->result,
rdata->subreq.len);
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
switch (mid->mid_state) {
case MID_RESPONSE_RECEIVED:
/* result already set, check signature */
@@ -2003,7 +2007,10 @@ cifs_async_writev(struct cifs_io_subrequest *wdata)
rqst.rq_iov = iov;
rqst.rq_nvec = 1;
- rqst.rq_iter = wdata->subreq.io_iter;
+
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
cifs_dbg(FYI, "async write at %llu %zu bytes\n",
wdata->subreq.start, wdata->subreq.len);
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 968740e7c9c3aa..c7dd10b6de327d 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -43,18 +43,34 @@ static int cifs_reopen_file(struct cifsFileInfo *cfile, bool can_flush);
* Prepare a subrequest to upload to the server. We need to allocate credits
* so that we know the maximum amount of data that we can include in it.
*/
-static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
+static int cifs_estimate_write(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate)
+{
+ struct cifs_sb_info *cifs_sb = CIFS_SB(wreq->inode->i_sb);
+
+ estimate->issue_at = stream->issue_from + cifs_sb->ctx->wsize;
+ return 0;
+}
+
+/*
+ * Issue a subrequest to upload to the server.
+ */
+static int cifs_issue_write(struct netfs_io_subrequest *subreq)
{
struct cifs_io_subrequest *wdata =
container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = wdata->req;
- struct netfs_io_stream *stream = &req->rreq.io_streams[subreq->stream_nr];
struct TCP_Server_Info *server;
struct cifsFileInfo *open_file = req->cfile;
- struct cifs_sb_info *cifs_sb = CIFS_SB(wdata->rreq->inode->i_sb);
- size_t wsize = req->rreq.wsize;
+ struct cifs_sb_info *cifs_sb = CIFS_SB(subreq->rreq->inode->i_sb);
+ unsigned int max_segs = INT_MAX;
+ size_t len;
int rc;
+ if (cifs_forced_shutdown(cifs_sb))
+ return smb_EIO(smb_eio_trace_forced_shutdown);
+
if (!wdata->have_xid) {
wdata->xid = get_xid();
wdata->have_xid = true;
@@ -73,18 +89,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
if (rc < 0) {
if (rc == -EAGAIN)
goto retry;
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
+ return rc;
}
}
- rc = server->ops->wait_mtu_credits(server, wsize, &stream->sreq_max_len,
- &wdata->credits);
- if (rc < 0) {
- subreq->error = rc;
- return netfs_prepare_write_failed(subreq);
- }
+ len = umin(subreq->len, cifs_sb->ctx->wsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &wdata->credits);
+ if (rc < 0)
+ return rc;
+ subreq->len = len;
wdata->credits.rreq_debug_id = subreq->rreq->debug_id;
wdata->credits.rreq_debug_index = subreq->debug_index;
wdata->credits.in_flight_check = 1;
@@ -100,46 +114,34 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- stream->sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
-}
-/*
- * Issue a subrequest to upload to the server.
- */
-static void cifs_issue_write(struct netfs_io_subrequest *subreq)
-{
- struct cifs_io_subrequest *wdata =
- container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_sb_info *sbi = CIFS_SB(subreq->rreq->inode->i_sb);
- int rc;
-
- if (cifs_forced_shutdown(sbi)) {
- rc = smb_EIO(smb_eio_trace_forced_shutdown);
- goto fail;
+ rc = netfs_prepare_write_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
+ return rc;
}
+ /* After this point, must fail by termination. */
- rc = adjust_credits(wdata->server, wdata, cifs_trace_rw_credits_issue_write_adjust);
+ rc = adjust_credits(server, wdata, cifs_trace_rw_credits_issue_write_adjust);
if (rc)
- goto fail;
+ goto fail_with_credits;
rc = -EAGAIN;
if (wdata->req->cfile->invalidHandle)
- goto fail;
+ goto fail_with_credits;
wdata->server->ops->async_writev(wdata);
-out:
- return;
+ return 0;
-fail:
+fail_with_credits:
if (rc == -EAGAIN)
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
- else
- trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
add_credits_and_wake_if(wdata->server, &wdata->credits, 0);
cifs_write_subrequest_terminated(wdata, rc);
- goto out;
+ return 0;
}
static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
@@ -148,17 +150,25 @@ static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq)
}
/*
- * Negotiate the size of a read operation on behalf of the netfs library.
+ * Issue a read operation on behalf of the netfs helper functions. We're asked
+ * to make a read of a certain size at a point in the file. We are permitted
+ * to only read a portion of that, but as long as we read something, the netfs
+ * helper will call us again so that we can issue another read.
*/
-static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
+static int cifs_issue_read(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server;
+ struct TCP_Server_Info *server = rdata->server;
struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb);
- size_t size;
- int rc = 0;
+ unsigned int max_segs = INT_MAX;
+ size_t len;
+ int rc;
+
+ cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
+ __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
+ subreq->transferred, subreq->len);
if (!rdata->have_xid) {
rdata->xid = get_xid();
@@ -172,17 +182,15 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
cifs_negotiate_rsize(server, cifs_sb->ctx,
tlink_tcon(req->cfile->tlink));
- rc = server->ops->wait_mtu_credits(server, cifs_sb->ctx->rsize,
- &size, &rdata->credits);
+ len = umin(subreq->len, cifs_sb->ctx->rsize);
+ rc = server->ops->wait_mtu_credits(server, len, &len, &rdata->credits);
if (rc)
return rc;
- rreq->io_streams[0].sreq_max_len = size;
-
- rdata->credits.in_flight_check = 1;
+ subreq->len = len;
rdata->credits.rreq_debug_id = rreq->debug_id;
rdata->credits.rreq_debug_index = subreq->debug_index;
-
+ rdata->credits.in_flight_check = 1;
trace_smb3_rw_credits(rdata->rreq->debug_id,
rdata->subreq.debug_index,
rdata->credits.value,
@@ -194,40 +202,27 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq)
const struct smbdirect_socket_parameters *sp =
smbd_get_parameters(server->smbd_conn);
- rreq->io_streams[0].sreq_max_segs = sp->max_frmr_depth;
+ max_segs = sp->max_frmr_depth;
}
#endif
- return 0;
-}
-
-/*
- * Issue a read operation on behalf of the netfs helper functions. We're asked
- * to make a read of a certain size at a point in the file. We are permitted
- * to only read a portion of that, but as long as we read something, the netfs
- * helper will call us again so that we can issue another read.
- */
-static void cifs_issue_read(struct netfs_io_subrequest *subreq)
-{
- struct netfs_io_request *rreq = subreq->rreq;
- struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq);
- struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq);
- struct TCP_Server_Info *server = rdata->server;
- int rc = 0;
- cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n",
- __func__, rreq->debug_id, subreq->debug_index, rreq->mapping,
- subreq->transferred, subreq->len);
+ rc = netfs_prepare_read_buffer(subreq, max_segs);
+ if (rc < 0) {
+ add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
+ return rc;
+ }
+ /* After this point, must fail by termination. */
rc = adjust_credits(server, rdata, cifs_trace_rw_credits_issue_read_adjust);
if (rc)
- goto failed;
+ goto fail_with_credits;
if (req->cfile->invalidHandle) {
do {
rc = cifs_reopen_file(req->cfile, true);
} while (rc == -EAGAIN);
if (rc)
- goto failed;
+ goto fail_with_credits;
}
if (subreq->rreq->origin != NETFS_UNBUFFERED_READ &&
@@ -235,15 +230,22 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
trace_netfs_sreq(subreq, netfs_sreq_trace_submit);
+
rc = rdata->server->ops->async_readv(rdata);
if (rc)
goto failed;
- return;
+ return 0;
+fail_with_credits:
+ if (rc == -EAGAIN)
+ trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
+ else
+ trace_netfs_sreq(subreq, netfs_sreq_trace_fail);
failed:
add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
subreq->error = rc;
netfs_read_subreq_terminated(subreq);
+ return 0;
}
/*
@@ -358,11 +360,10 @@ const struct netfs_request_ops cifs_req_ops = {
.init_request = cifs_init_request,
.free_request = cifs_free_request,
.free_subrequest = cifs_free_subrequest,
- .prepare_read = cifs_prepare_read,
.issue_read = cifs_issue_read,
.done = cifs_rreq_done,
.begin_writeback = cifs_begin_writeback,
- .prepare_write = cifs_prepare_write,
+ .estimate_write = cifs_estimate_write,
.issue_write = cifs_issue_write,
.invalidate_cache = cifs_netfs_invalidate_cache,
};
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index cbd51a08e97e6e..28553b74333a50 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -13,7 +13,6 @@
#include <linux/sort.h>
#include <crypto/aead.h>
#include <linux/fiemap.h>
-#include <linux/folio_queue.h>
#include <uapi/linux/magic.h>
#include "cifsfs.h"
#include "cifsglob.h"
@@ -4683,19 +4682,18 @@ crypt_message(struct TCP_Server_Info *server, int num_rqst,
}
/*
- * Copy data from an iterator to the folios in a folio queue buffer.
+ * Copy data from an iterator to the pages in a bvec queue buffer.
*/
-static bool cifs_copy_iter_to_folioq(struct iov_iter *iter, size_t size,
- struct folio_queue *buffer)
+static bool cifs_copy_iter_to_bvecq(struct iov_iter *iter, size_t size,
+ struct bvecq *buffer)
{
for (; buffer; buffer = buffer->next) {
- for (int s = 0; s < folioq_count(buffer); s++) {
- struct folio *folio = folioq_folio(buffer, s);
- size_t part = folioq_folio_size(buffer, s);
+ for (int s = 0; s < buffer->nr_slots; s++) {
+ struct bio_vec *bv = &buffer->bv[s];
+ size_t part = umin(bv->bv_len, size);
- part = umin(part, size);
-
- if (copy_folio_from_iter(folio, 0, part, iter) != part)
+ if (copy_page_from_iter(bv->bv_page, bv->bv_offset,
+ part, iter) != part)
return false;
size -= part;
}
@@ -4707,7 +4705,7 @@ void
smb3_free_compound_rqst(int num_rqst, struct smb_rqst *rqst)
{
for (int i = 0; i < num_rqst; i++)
- netfs_free_folioq_buffer(rqst[i].rq_buffer);
+ bvecq_put(rqst[i].rq_buffer);
}
/*
@@ -4734,7 +4732,7 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
for (int i = 1; i < num_rqst; i++) {
struct smb_rqst *old = &old_rq[i - 1];
struct smb_rqst *new = &new_rq[i];
- struct folio_queue *buffer = NULL;
+ struct bvecq *buffer = NULL;
size_t size = iov_iter_count(&old->rq_iter);
orig_len += smb_rqst_len(server, old);
@@ -4742,17 +4740,16 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
new->rq_nvec = old->rq_nvec;
if (size > 0) {
- size_t cur_size = 0;
- rc = netfs_alloc_folioq_buffer(NULL, &buffer, &cur_size,
- size, GFP_NOFS);
- if (rc < 0)
+ rc = -ENOMEM;
+ buffer = bvecq_alloc_buffer(size, GFP_NOFS);
+ if (!buffer)
goto err_free;
new->rq_buffer = buffer;
- iov_iter_folio_queue(&new->rq_iter, ITER_SOURCE,
- buffer, 0, 0, size);
+ iov_iter_bvec_queue(&new->rq_iter, ITER_SOURCE,
+ buffer, 0, 0, size);
- if (!cifs_copy_iter_to_folioq(&old->rq_iter, size, buffer)) {
+ if (!cifs_copy_iter_to_bvecq(&old->rq_iter, size, buffer)) {
rc = smb_EIO1(smb_eio_trace_tx_copy_iter_to_buf, size);
goto err_free;
}
@@ -4842,22 +4839,20 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,
}
static int
-cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,
- size_t skip, struct iov_iter *iter)
+cifs_copy_bvecq_to_iter(struct bvecq *bq, size_t data_size,
+ size_t skip, struct iov_iter *iter)
{
- for (; folioq; folioq = folioq->next) {
- for (int s = 0; s < folioq_count(folioq); s++) {
- struct folio *folio;
- size_t fsize, n, len;
+ for (; bq; bq = bq->next) {
+ for (int s = 0; s < bq->nr_slots; s++) {
+ struct bio_vec *bv = &bq->bv[s];
+ size_t n, len;
if (data_size == 0)
return 0;
- folio = folioq_folio(folioq, s);
- fsize = folio_size(folio);
- len = umin(fsize - skip, data_size);
+ len = umin(bv->bv_len - skip, data_size);
- n = copy_folio_to_iter(folio, skip, len, iter);
+ n = copy_page_to_iter(bv->bv_page, bv->bv_offset + skip, len, iter);
if (n != len) {
cifs_dbg(VFS, "%s: something went wrong\n", __func__);
return smb_EIO2(smb_eio_trace_rx_copy_to_iter,
@@ -4879,7 +4874,7 @@ cifs_copy_folioq_to_iter(struct folio_queue *folioq, size_t data_size,
static int
handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
- char *buf, unsigned int buf_len, struct folio_queue *buffer,
+ char *buf, unsigned int buf_len, struct bvecq *buffer,
unsigned int buffer_len, bool is_offloaded)
{
unsigned int data_offset;
@@ -4889,6 +4884,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
unsigned int cur_page_idx;
unsigned int pad_len;
struct cifs_io_subrequest *rdata = mid->callback_data;
+ struct iov_iter iter;
struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
size_t copied;
bool use_rdma_mr = false;
@@ -4961,6 +4957,10 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
pad_len = data_offset - server->vals->read_rsp_size;
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+
if (buf_len <= data_offset) {
/* read response payload is in pages */
cur_page_idx = pad_len / PAGE_SIZE;
@@ -4989,8 +4989,8 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
}
/* Copy the data to the output I/O iterator. */
- rdata->result = cifs_copy_folioq_to_iter(buffer, data_len,
- cur_off, &rdata->subreq.io_iter);
+ rdata->result = cifs_copy_bvecq_to_iter(buffer, data_len,
+ cur_off, &iter);
if (rdata->result != 0) {
if (is_offloaded)
mid->mid_state = MID_RESPONSE_MALFORMED;
@@ -5004,7 +5004,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
buf_len >= end_off) {
/* read response payload is in buf */
WARN_ONCE(buffer, "read data can be either in buf or in buffer");
- copied = copy_to_iter(buf + data_offset, data_len, &rdata->subreq.io_iter);
+ copied = copy_to_iter(buf + data_offset, data_len, &iter);
if (copied == 0)
return smb_EIO2(smb_eio_trace_rx_copy_to_iter, copied, data_len);
rdata->got_bytes = copied;
@@ -5029,7 +5029,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid,
struct smb2_decrypt_work {
struct work_struct decrypt;
struct TCP_Server_Info *server;
- struct folio_queue *buffer;
+ struct bvecq *buffer;
char *buf;
unsigned int len;
};
@@ -5043,7 +5043,7 @@ static void smb2_decrypt_offload(struct work_struct *work)
struct mid_q_entry *mid;
struct iov_iter iter;
- iov_iter_folio_queue(&iter, ITER_DEST, dw->buffer, 0, 0, dw->len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dw->buffer, 0, 0, dw->len);
rc = decrypt_raw_data(dw->server, dw->buf, dw->server->vals->read_rsp_size,
&iter, true);
if (rc) {
@@ -5092,7 +5092,7 @@ static void smb2_decrypt_offload(struct work_struct *work)
}
free_pages:
- netfs_free_folioq_buffer(dw->buffer);
+ bvecq_put(dw->buffer);
cifs_small_buf_release(dw->buf);
kfree(dw);
}
@@ -5138,12 +5138,12 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,
dw->len = len;
len = round_up(dw->len, PAGE_SIZE);
- size_t cur_size = 0;
- rc = netfs_alloc_folioq_buffer(NULL, &dw->buffer, &cur_size, len, GFP_NOFS);
- if (rc < 0)
+ rc = -ENOMEM;
+ dw->buffer = bvecq_alloc_buffer(len, GFP_NOFS);
+ if (!dw->buffer)
goto discard_data;
- iov_iter_folio_queue(&iter, ITER_DEST, dw->buffer, 0, 0, len);
+ iov_iter_bvec_queue(&iter, ITER_DEST, dw->buffer, 0, 0, len);
/* Read the data into the buffer and clear excess bufferage. */
rc = cifs_read_iter_from_socket(server, &iter, dw->len);
@@ -5201,7 +5201,7 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid,
}
free_pages:
- netfs_free_folioq_buffer(dw->buffer);
+ bvecq_put(dw->buffer);
free_dw:
kfree(dw);
return rc;
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index 4ce165e40657f3..085fdc7d8bec09 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -4592,9 +4592,13 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
*/
if (rdata && smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
+ rdata->mr = smbd_register_mr(server->smbd_conn, &iter,
true, need_invalidate);
if (!rdata->mr)
return -EAGAIN;
@@ -4658,9 +4662,10 @@ smb2_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid)
unsigned int rreq_debug_id = rdata->rreq->debug_id;
unsigned int subreq_debug_index = rdata->subreq.debug_index;
- if (rdata->got_bytes) {
- rqst.rq_iter = rdata->subreq.io_iter;
- }
+ if (rdata->got_bytes)
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST,
+ rdata->subreq.content.bvecq, rdata->subreq.content.slot,
+ rdata->subreq.content.offset, rdata->subreq.len);
WARN_ONCE(rdata->server != server,
"rdata server %p != mid server %p",
@@ -5148,7 +5153,9 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
goto out;
rqst.rq_iov = iov;
- rqst.rq_iter = wdata->subreq.io_iter;
+ iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
rqst.rq_iov[0].iov_len = total_len - 1;
rqst.rq_iov[0].iov_base = (char *)req;
@@ -5187,9 +5194,14 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
*/
if (smb3_use_rdma_offload(io_parms)) {
struct smbdirect_buffer_descriptor_v1 *v1;
+ struct iov_iter iter;
bool need_invalidate = server->dialect == SMB30_PROT_ID;
- wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter,
+ iov_iter_bvec_queue(&iter, ITER_SOURCE,
+ wdata->subreq.content.bvecq, wdata->subreq.content.slot,
+ wdata->subreq.content.offset, wdata->subreq.len);
+
+ wdata->mr = smbd_register_mr(server->smbd_conn, &iter,
false, need_invalidate);
if (!wdata->mr) {
rc = -EAGAIN;
@@ -5226,8 +5238,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
smb2_set_replay(server, &rqst);
}
- cifs_dbg(FYI, "async write at %llu %u bytes iter=%zx\n",
- io_parms->offset, io_parms->length, iov_iter_count(&wdata->subreq.io_iter));
+ cifs_dbg(FYI, "async write at %llu %u bytes len=%zx\n",
+ io_parms->offset, io_parms->length, wdata->subreq.len);
if (wdata->credits.value > 0) {
shdr->CreditCharge = cpu_to_le16(DIV_ROUND_UP(wdata->subreq.len,
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index fdf4e50c27ceb4..be2f6b909c34bd 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -1267,12 +1267,19 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid)
}
#ifdef CONFIG_CIFS_SMB_DIRECT
- if (rdata->mr)
+ if (rdata->mr) {
length = data_len; /* An RDMA read is already done. */
- else
+ } else {
+#endif
+ struct iov_iter iter;
+
+ iov_iter_bvec_queue(&iter, ITER_DEST, rdata->subreq.content.bvecq,
+ rdata->subreq.content.slot, rdata->subreq.content.offset,
+ data_len);
+ length = cifs_read_iter_from_socket(server, &iter, data_len);
+#ifdef CONFIG_CIFS_SMB_DIRECT
+ }
#endif
- length = cifs_read_iter_from_socket(server, &rdata->subreq.io_iter,
- data_len);
if (length > 0)
rdata->got_bytes += length;
server->total_read += length;
diff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c
index 8adf580975344b..f217cf127cf9ba 100644
--- a/fs/smb/smbdirect/connection.c
+++ b/fs/smb/smbdirect/connection.c
@@ -5,7 +5,7 @@
*/
#include "internal.h"
-#include <linux/folio_queue.h>
+#include <linux/bvecq.h>
struct smbdirect_map_sges {
struct ib_sge *sge;
@@ -2006,6 +2006,69 @@ static ssize_t smbdirect_map_sges_from_bvec(struct iov_iter *iter,
return ret;
}
+/*
+ * Extract memory fragments from a BVECQ-class iterator and add them to an RDMA
+ * list. The fragments are not pinned.
+ */
+static ssize_t smbdirect_map_sges_from_bvecq(struct iov_iter *iter,
+ struct smbdirect_map_sges *state,
+ ssize_t maxsize)
+{
+ const struct bvecq *bq = iter->bvecq, *next;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t extracted = 0;
+ size_t offset = iter->iov_offset;
+
+ maxsize = umin(maxsize, iov_iter_count(iter));
+
+ do {
+ struct bio_vec *bv;
+ size_t bsize;
+
+ while (slot >= bq->nr_slots) {
+ next = bvecq_next(bq);
+ if (!next) {
+ if (WARN_ON_ONCE(maxsize > 0))
+ return -EIO;
+ goto out;
+ }
+ bq = next;
+ slot = 0;
+ }
+
+ bv = &bq->bv[slot];
+ bsize = bv->bv_len;
+
+ if (offset < bsize) {
+ size_t part = umin(maxsize, bsize - offset);
+ bool ok;
+
+ ok = smbdirect_map_sges_single_page(state,
+ bv->bv_page,
+ bv->bv_offset + offset,
+ part);
+ if (!ok)
+ return -EIO;
+
+ offset += part;
+ extracted += part;
+ maxsize -= part;
+ }
+
+ if (offset >= bsize) {
+ offset = 0;
+ slot++;
+ }
+ } while (state->num_sge < state->max_sge && maxsize > 0);
+
+out:
+ iter->bvecq = bq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= extracted;
+ return extracted;
+}
+
/*
* Extract fragments from a KVEC-class iterator and add them to an ib_sge list.
* This can deal with vmalloc'd buffers as well as kmalloc'd or static buffers.
@@ -2067,70 +2130,6 @@ static ssize_t smbdirect_map_sges_from_kvec(struct iov_iter *iter,
return ret;
}
-/*
- * Extract folio fragments from a FOLIOQ-class iterator and add them to an
- * ib_sge list. The folios are not pinned.
- */
-static ssize_t smbdirect_map_sges_from_folioq(struct iov_iter *iter,
- struct smbdirect_map_sges *state,
- ssize_t maxsize)
-{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int slot = iter->folioq_slot;
- ssize_t ret = 0;
- size_t offset = iter->iov_offset;
-
- if (WARN_ON_ONCE(!folioq))
- return -EIO;
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- if (WARN_ON_ONCE(!folioq))
- return -EIO;
- slot = 0;
- }
-
- do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t fsize = folioq_folio_size(folioq, slot);
-
- if (offset < fsize) {
- size_t part = umin(maxsize, fsize - offset);
- bool ok;
-
- ok = smbdirect_map_sges_single_page(state,
- folio_page(folio, 0),
- offset,
- part);
- if (!ok)
- return -EIO;
-
- offset += part;
- ret += part;
- maxsize -= part;
- }
-
- if (offset >= fsize) {
- offset = 0;
- slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- if (!folioq->next) {
- WARN_ON_ONCE(ret < iter->count);
- break;
- }
- folioq = folioq->next;
- slot = 0;
- }
- }
- } while (state->num_sge < state->max_sge && maxsize > 0);
-
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- iter->iov_offset = offset;
- iter->count -= ret;
- return ret;
-}
-
/*
* Extract page fragments from up to the given amount of the source iterator
* and build up an ib_sge list that refers to all of those bits. The ib_sge list
@@ -2155,12 +2154,12 @@ static ssize_t smbdirect_map_sges_from_iter(struct iov_iter *iter, size_t len,
case ITER_BVEC:
ret = smbdirect_map_sges_from_bvec(iter, state, len);
break;
+ case ITER_BVECQ:
+ ret = smbdirect_map_sges_from_bvecq(iter, state, len);
+ break;
case ITER_KVEC:
ret = smbdirect_map_sges_from_kvec(iter, state, len);
break;
- case ITER_FOLIOQ:
- ret = smbdirect_map_sges_from_folioq(iter, state, len);
- break;
default:
WARN_ONCE(1, "iov_iter_type[%u]\n", iov_iter_type(iter));
return -EIO;
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index 92837e2743f19a..53cf36e73967f1 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -343,4 +343,22 @@ static inline phys_addr_t bvec_phys(const struct bio_vec *bvec)
return page_to_phys(bvec->bv_page) + bvec->bv_offset;
}
+/**
+ * bvec_kmap_partial - Map part of a bvec into the kernel virtual address space
+ * @bvec: bvec to map
+ * @offset: Offset into bvec
+ *
+ * Map the page containing the byte at @offset into the kernel virtual address
+ * space. The caller is responsible for making sure this doesn't overrun.
+ *
+ * Call kunmap_local on the returned address to unmap.
+ */
+static inline void *bvec_kmap_partial(struct bio_vec *bvec, size_t offset)
+{
+ offset += bvec->bv_offset;
+
+ return kmap_local_page(bvec->bv_page + (offset >> PAGE_SHIFT)) +
+ (offset & ~PAGE_MASK);
+}
+
#endif /* __LINUX_BVEC_H */
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
new file mode 100644
index 00000000000000..fa46be520649c2
--- /dev/null
+++ b/include/linux/bvecq.h
@@ -0,0 +1,358 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Implementation of a segmented queue of bio_vec[].
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#ifndef _LINUX_BVECQ_H
+#define _LINUX_BVECQ_H
+
+#include <linux/bvec.h>
+
+/*
+ * The type of memory retention used by the elements in bvecq->bv[] and how to
+ * clean it up.
+ */
+enum bvecq_mem {
+ BVECQ_MEM_EXTERNAL, /* Externally retained memory - no freeing */
+ BVECQ_MEM_PAGECACHE, /* Ref'd pagecache pages - must put */
+ BVECQ_MEM_GUP, /* Pinned memory from get_user_pages() - unpin */
+ BVECQ_MEM_ALLOCED, /* Memory alloc'd by bvecq - can be freed/pooled */
+} __mode(byte);
+
+/*
+ * Segmented bio_vec queue.
+ *
+ * These can be linked together to form messages of indefinite length and
+ * iterated over with an ITER_BVECQ iterator. The list is non-circular; next
+ * and prev are NULL at the ends.
+ *
+ * The bv pointer points to the bio_vec array; this may be __bv if allocated
+ * together. The caller is responsible for determining whether or not this is
+ * the case as the array pointed to by bv may be follow on directly from the
+ * bvecq by accident of allocation (ie. ->bv == ->__bv is *not* sufficient to
+ * determine this).
+ *
+ * The file position and discontiguity flag allow non-contiguous data sets to
+ * be chained together, but still teased apart without the need to convert the
+ * info in the bio_vec back into a folio pointer.
+ */
+struct bvecq {
+ struct bvecq *next; /* Next bvec in the list or NULL */
+ struct bvecq *prev; /* Prev bvec in the list or NULL */
+ unsigned long long fpos; /* File position */
+ refcount_t ref;
+ u32 priv; /* Private data */
+ u16 nr_slots; /* Number of elements in bv[] used */
+ u16 max_slots; /* Number of elements allocated in bv[] */
+ enum bvecq_mem mem_type:3; /* What sort of memory and how to free it */
+ bool inline_bv:1; /* T if __bv[] is being used */
+ bool discontig:1; /* T if not contiguous with previous bvecq */
+ struct bio_vec *bv; /* Pointer to array of page fragments */
+ struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
+};
+
+#if BITS_PER_LONG == 64
+/* Number of slots in __bv[] for a bvecq in a 512-byte kmalloc block. */
+#define BVECQ_STD_SLOTS 29 /* 2 words/slot; 32 slots; bvecq is 6 words (3 slots) */
+#elif BITS_PER_LONG == 32
+/* Number of slots in __bv[] for a bvecq in a 256-byte kmalloc block. */
+#define BVECQ_STD_SLOTS 18 /* 3 words/slot; 21 slots; bvecq is 9 words (3 slots) */
+#else
+#error BVECQ_STD_SLOTS undetermined
+#endif
+
+/*
+ * Position in a bio_vec queue. The bvecq holds a ref on the queue segment it
+ * points to.
+ */
+struct bvecq_pos {
+ struct bvecq *bvecq; /* The first bvecq */
+ unsigned int offset; /* The offset within the starting slot */
+ u16 slot; /* The starting slot */
+};
+
+void bvecq_dump(const struct bvecq *bq);
+struct bvecq *bvecq_alloc_one(size_t nr_slots, gfp_t gfp);
+struct bvecq *bvecq_alloc_chain(size_t nr_slots, gfp_t gfp);
+struct bvecq *bvecq_alloc_buffer2(size_t size, unsigned int pre_slots, gfp_t gfp);
+void bvecq_put(struct bvecq *bq);
+int bvecq_expand_buffer(struct bvecq **_buffer, size_t *_cur_size, size_t size, gfp_t gfp);
+int bvecq_shorten_buffer(struct bvecq *bq, unsigned int slot, size_t size);
+int bvecq_buffer_init(struct bvecq_pos *pos, gfp_t gfp);
+void bvecq_buffer_append(struct bvecq_pos *pos, struct bvecq *bq);
+void bvecq_pos_advance(struct bvecq_pos *pos, size_t amount);
+ssize_t bvecq_zero(struct bvecq_pos *pos, size_t amount);
+size_t bvecq_slice(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, unsigned int *_nr_slots);
+ssize_t bvecq_extract(struct bvecq_pos *pos, size_t max_size,
+ unsigned int max_slots, struct bvecq **to);
+ssize_t bvecq_load_from_ra(struct bvecq_pos *pos, struct readahead_control *ractl);
+
+/**
+ * bvecq_alloc_buffer - Allocate a bvecq chain and populate with buffers
+ * @size: Target size of the buffer (can be 0 for an empty buffer)
+ * @gfp: The allocation constraints.
+ *
+ * Wrapper around %bvecq_alloc_buffer2().
+ */
+static inline struct bvecq *bvecq_alloc_buffer(size_t size, gfp_t gfp)
+{
+ return bvecq_alloc_buffer2(size, 0, gfp);
+}
+
+/**
+ * bvecq_get - Get a ref on a bvecq
+ * @bq: The bvecq to get a ref on
+ */
+static inline struct bvecq *bvecq_get(struct bvecq *bq)
+{
+ refcount_inc(&bq->ref);
+ return bq;
+}
+
+/**
+ * bvecq_is_full - Determine if a bvecq is full
+ * @bvecq: The object to query
+ *
+ * Return: true if full; false if not.
+ */
+static inline bool bvecq_is_full(const struct bvecq *bvecq)
+{
+ return bvecq->nr_slots >= bvecq->max_slots;
+}
+
+/**
+ * bvecq_filled_to - Release filled slots with release barrier
+ * @bvecq: The object modified
+ * @to: The latest slot filled + 1
+ */
+static inline void bvecq_filled_to(struct bvecq *bvecq, unsigned int to)
+{
+ /* Set the slot counter after filling the slot */
+ smp_store_release(&bvecq->nr_slots, to);
+}
+
+/**
+ * bvecq_nr_slots_acquire - Get the number of filled slots with acquire barrier
+ * @bvecq: The object to query
+ *
+ * Return: The number of filled slots
+ */
+static inline unsigned int bvecq_nr_slots_acquire(const struct bvecq *bvecq)
+{
+ /* Read the slot counter before looking at the slot */
+ return smp_load_acquire(&bvecq->nr_slots);
+}
+
+/**
+ * bvecq_acquire_slot - Determine if a slot is valid with acquire barrier
+ * @bvecq: The object to query
+ * @slot: The next slot
+ *
+ * Return: true if valid; false if might not be valid
+ */
+static inline bool bvecq_acquire_slot(const struct bvecq *bvecq, unsigned int slot)
+{
+ /* Read the slot counter before looking at the slot */
+ return slot < bvecq_nr_slots_acquire(bvecq);
+}
+
+/**
+ * bvecq_append - Get the next bvecq with appropriate barrier
+ * @to: The bvecq to append to
+ * @add: The bvecq to append
+ *
+ * Attach a new bvecq to a chain using an appropriate barrier to protect the
+ * write.
+ *
+ * [!] Note that this function transfers the caller's ref to the chain.
+ */
+static inline void bvecq_append(struct bvecq *to, struct bvecq *add)
+{
+ add->prev = to;
+
+ /* Make sure the initialisation is stored before the next pointer. */
+ smp_store_release(&to->next, add);
+}
+
+/**
+ * bvecq_next - Get the next bvecq with appropriate barrier
+ * @bq: The bvecq to start from
+ *
+ * Return the next bvecq in a chain, using an appropriate barrier to protect
+ * the access.
+ */
+static inline struct bvecq *bvecq_next(const struct bvecq *bq)
+{
+ /* Read the contents of the next node after the pointer to it. */
+ return smp_load_acquire(&bq->next);
+}
+
+/**
+ * bvecq_pos_set - Set one position to be the same as another
+ * @pos: The position object to set
+ * @at: The source position.
+ *
+ * Set @pos to have the same position as @at. This may take a ref on the
+ * bvecq pointed to.
+ */
+static inline void bvecq_pos_set(struct bvecq_pos *pos, const struct bvecq_pos *at)
+{
+ *pos = *at;
+ bvecq_get(pos->bvecq);
+}
+
+/**
+ * bvecq_pos_unset - Unset a position
+ * @pos: The position object to unset
+ *
+ * Unset @pos. This does any needed ref cleanup.
+ */
+static inline void bvecq_pos_unset(struct bvecq_pos *pos)
+{
+ bvecq_put(pos->bvecq);
+ pos->bvecq = NULL;
+ pos->slot = 0;
+ pos->offset = 0;
+}
+
+/**
+ * bvecq_pos_transfer - Transfer one position to another, clearing the first
+ * @pos: The position object to set
+ * @from: The source position to clear.
+ *
+ * Set @pos to have the same position as @from and then clear @from. This may
+ * transfer a ref on the bvecq pointed to.
+ */
+static inline void bvecq_pos_transfer(struct bvecq_pos *pos, struct bvecq_pos *from)
+{
+ *pos = *from;
+ from->bvecq = NULL;
+ from->slot = 0;
+ from->offset = 0;
+}
+
+/**
+ * bvecq_pos_move - Update a position to a new bvecq
+ * @pos: The position object to update.
+ * @to: The new bvecq to point at.
+ *
+ * Update @pos to point to @to if it doesn't already do so. This may
+ * manipulate refs on the bvecqs pointed to.
+ */
+static inline void bvecq_pos_move(struct bvecq_pos *pos, struct bvecq *to)
+{
+ struct bvecq *old = pos->bvecq;
+
+ if (old != to) {
+ pos->bvecq = bvecq_get(to);
+ bvecq_put(old);
+ }
+}
+
+/**
+ * bvecq_pos_nudge - Nudge a position onto the next segment if current used up
+ * @pos: The position object to nudge.
+ *
+ * Update @pos to point to the next segment in the chain if we've used up the
+ * current segment. This may manipulate refs on the bvecqs pointed to.
+ *
+ * Return: true if found a new segment, false if hit the end.
+ */
+static inline bool bvecq_pos_nudge(struct bvecq_pos *pos)
+{
+ struct bvecq *bq = pos->bvecq;
+
+ for (;;) {
+ if (!bvecq_acquire_slot(bq, pos->slot)) {
+ bq = bvecq_next(bq);
+ if (!bq)
+ return false;
+ bvecq_pos_move(pos, bq);
+ pos->slot = 0;
+ pos->offset = 0;
+ continue;
+ }
+ if (pos->offset >= bq->bv[pos->slot].bv_len) {
+ pos->slot++;
+ pos->offset = 0;
+ continue;
+ }
+ return true;
+ }
+}
+
+/**
+ * bvecq_pos_step - Step a position to the next slot if possible
+ * @pos: The position object to step.
+ *
+ * Update @pos to point to the next slot in the queue if not at the end. This
+ * may manipulate refs on the bvecqs pointed to.
+ *
+ * Return: true if successful, false if was at the end.
+ */
+static inline bool bvecq_pos_step(struct bvecq_pos *pos)
+{
+ struct bvecq *bq = pos->bvecq, *next;
+
+ pos->slot++;
+ pos->offset = 0;
+ if (pos->slot < bq->nr_slots)
+ return true;
+ next = bvecq_next(bq);
+ if (!next)
+ return false;
+ bvecq_pos_move(pos, next);
+ pos->slot = 0;
+ return true;
+}
+
+/**
+ * bvecq_delete_spent - Delete the bvecq at the front if possible
+ * @pos: The position object to update.
+ *
+ * Delete the used up bvecq at the front of the queue that @pos points to if it
+ * is not the last node in the queue; if it is the last node in the queue, it
+ * is kept so that the queue doesn't become detached from the other end. This
+ * may manipulate refs on the bvecqs pointed to. It is also possible that the
+ * producer will fill more slots in the current bvecq.
+ *
+ * Also, we have to be very careful: the consumer can catch the producer, which
+ * could lead to us having nothing left in the queue, causing the front and
+ * back pointers to end up on different tracks. To avoid this, we must always
+ * keep at least one segment in the queue.
+ *
+ * The caller must reload from @pos after calling this.
+ *
+ * Return: true if there's more available; false if not.
+ */
+static inline bool bvecq_delete_spent(struct bvecq_pos *pos)
+{
+ struct bvecq *spent = pos->bvecq;
+ struct bvecq *next;
+ unsigned int slot = pos->slot;
+
+again:
+ /* Read the contents of the queue node after the pointer to it. */
+ next = bvecq_next(spent);
+ if (!next)
+ return false; /* Nothing more to consume at the moment. */
+ if (slot < bvecq_nr_slots_acquire(spent))
+ return true; /* The producer added more. */
+ next->prev = NULL;
+ spent->next = NULL;
+ bvecq_put(spent);
+ pos->bvecq = next; /* We take spent's ref. */
+ pos->slot = 0;
+ pos->offset = 0;
+ if (!bvecq_acquire_slot(next, 0)) {
+ spent = next;
+ slot = 0;
+ goto again;
+ }
+ return true;
+}
+
+#endif /* _LINUX_BVECQ_H */
diff --git a/include/linux/folio_queue.h b/include/linux/folio_queue.h
deleted file mode 100644
index f6d5f1f127c9f3..00000000000000
--- a/include/linux/folio_queue.h
+++ /dev/null
@@ -1,282 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/* Queue of folios definitions
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- *
- * See:
- *
- * Documentation/core-api/folio_queue.rst
- *
- * for a description of the API.
- */
-
-#ifndef _LINUX_FOLIO_QUEUE_H
-#define _LINUX_FOLIO_QUEUE_H
-
-#include <linux/folio_batch.h>
-#include <linux/mm.h>
-
-/*
- * Segment in a queue of running buffers. Each segment can hold a number of
- * folios and a portion of the queue can be referenced with the ITER_FOLIOQ
- * iterator. The possibility exists of inserting non-folio elements into the
- * queue (such as gaps).
- *
- * Explicit prev and next pointers are used instead of a list_head to make it
- * easier to add segments to tail and remove them from the head without the
- * need for a lock.
- */
-struct folio_queue {
- struct folio_batch vec; /* Folios in the queue segment */
- u8 orders[FOLIO_BATCH_SIZE]; /* Order of each folio */
- struct folio_queue *next; /* Next queue segment or NULL */
- struct folio_queue *prev; /* Previous queue segment of NULL */
- unsigned long marks; /* 1-bit mark per folio */
- unsigned long marks2; /* Second 1-bit mark per folio */
-#if FOLIO_BATCH_SIZE > BITS_PER_LONG
-#error marks is not big enough
-#endif
- unsigned int rreq_id;
- unsigned int debug_id;
-};
-
-/**
- * folioq_init - Initialise a folio queue segment
- * @folioq: The segment to initialise
- * @rreq_id: The request identifier to use in tracelines.
- *
- * Initialise a folio queue segment and set an identifier to be used in traces.
- *
- * Note that the folio pointers are left uninitialised.
- */
-static inline void folioq_init(struct folio_queue *folioq, unsigned int rreq_id)
-{
- folio_batch_init(&folioq->vec);
- folioq->next = NULL;
- folioq->prev = NULL;
- folioq->marks = 0;
- folioq->marks2 = 0;
- folioq->rreq_id = rreq_id;
- folioq->debug_id = 0;
-}
-
-/**
- * folioq_nr_slots: Query the capacity of a folio queue segment
- * @folioq: The segment to query
- *
- * Query the number of folios that a particular folio queue segment might hold.
- * [!] NOTE: This must not be assumed to be the same for every segment!
- */
-static inline unsigned int folioq_nr_slots(const struct folio_queue *folioq)
-{
- return FOLIO_BATCH_SIZE;
-}
-
-/**
- * folioq_count: Query the occupancy of a folio queue segment
- * @folioq: The segment to query
- *
- * Query the number of folios that have been added to a folio queue segment.
- * Note that this is not decreased as folios are removed from a segment.
- */
-static inline unsigned int folioq_count(struct folio_queue *folioq)
-{
- return folio_batch_count(&folioq->vec);
-}
-
-/**
- * folioq_full: Query if a folio queue segment is full
- * @folioq: The segment to query
- *
- * Query if a folio queue segment is fully occupied. Note that this does not
- * change if folios are removed from a segment.
- */
-static inline bool folioq_full(struct folio_queue *folioq)
-{
- //return !folio_batch_space(&folioq->vec);
- return folioq_count(folioq) >= folioq_nr_slots(folioq);
-}
-
-/**
- * folioq_is_marked: Check first folio mark in a folio queue segment
- * @folioq: The segment to query
- * @slot: The slot number of the folio to query
- *
- * Determine if the first mark is set for the folio in the specified slot in a
- * folio queue segment.
- */
-static inline bool folioq_is_marked(const struct folio_queue *folioq, unsigned int slot)
-{
- return test_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_mark: Set the first mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Set the first mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_mark(struct folio_queue *folioq, unsigned int slot)
-{
- set_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_unmark: Clear the first mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Clear the first mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_unmark(struct folio_queue *folioq, unsigned int slot)
-{
- clear_bit(slot, &folioq->marks);
-}
-
-/**
- * folioq_is_marked2: Check second folio mark in a folio queue segment
- * @folioq: The segment to query
- * @slot: The slot number of the folio to query
- *
- * Determine if the second mark is set for the folio in the specified slot in a
- * folio queue segment.
- */
-static inline bool folioq_is_marked2(const struct folio_queue *folioq, unsigned int slot)
-{
- return test_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_mark2: Set the second mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Set the second mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_mark2(struct folio_queue *folioq, unsigned int slot)
-{
- set_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_unmark2: Clear the second mark on a folio in a folio queue segment
- * @folioq: The segment to modify
- * @slot: The slot number of the folio to modify
- *
- * Clear the second mark for the folio in the specified slot in a folio queue
- * segment.
- */
-static inline void folioq_unmark2(struct folio_queue *folioq, unsigned int slot)
-{
- clear_bit(slot, &folioq->marks2);
-}
-
-/**
- * folioq_append: Add a folio to a folio queue segment
- * @folioq: The segment to add to
- * @folio: The folio to add
- *
- * Add a folio to the tail of the sequence in a folio queue segment, increasing
- * the occupancy count and returning the slot number for the folio just added.
- * The folio size is extracted and stored in the queue and the marks are left
- * unmodified.
- *
- * Note that it's left up to the caller to check that the segment capacity will
- * not be exceeded and to extend the queue.
- */
-static inline unsigned int folioq_append(struct folio_queue *folioq, struct folio *folio)
-{
- unsigned int slot = folioq->vec.nr++;
-
- folioq->vec.folios[slot] = folio;
- folioq->orders[slot] = folio_order(folio);
- return slot;
-}
-
-/**
- * folioq_append_mark: Add a folio to a folio queue segment
- * @folioq: The segment to add to
- * @folio: The folio to add
- *
- * Add a folio to the tail of the sequence in a folio queue segment, increasing
- * the occupancy count and returning the slot number for the folio just added.
- * The folio size is extracted and stored in the queue, the first mark is set
- * and and the second and third marks are left unmodified.
- *
- * Note that it's left up to the caller to check that the segment capacity will
- * not be exceeded and to extend the queue.
- */
-static inline unsigned int folioq_append_mark(struct folio_queue *folioq, struct folio *folio)
-{
- unsigned int slot = folioq->vec.nr++;
-
- folioq->vec.folios[slot] = folio;
- folioq->orders[slot] = folio_order(folio);
- folioq_mark(folioq, slot);
- return slot;
-}
-
-/**
- * folioq_folio: Get a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the folio in the specified slot from a folio queue segment. Note
- * that no bounds check is made and if the slot hasn't been added into yet, the
- * pointer will be undefined. If the slot has been cleared, NULL will be
- * returned.
- */
-static inline struct folio *folioq_folio(const struct folio_queue *folioq, unsigned int slot)
-{
- return folioq->vec.folios[slot];
-}
-
-/**
- * folioq_folio_order: Get the order of a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the order of the folio in the specified slot from a folio queue
- * segment. Note that no bounds check is made and if the slot hasn't been
- * added into yet, the order returned will be 0.
- */
-static inline unsigned int folioq_folio_order(const struct folio_queue *folioq, unsigned int slot)
-{
- return folioq->orders[slot];
-}
-
-/**
- * folioq_folio_size: Get the size of a folio from a folio queue segment
- * @folioq: The segment to access
- * @slot: The folio slot to access
- *
- * Retrieve the size of the folio in the specified slot from a folio queue
- * segment. Note that no bounds check is made and if the slot hasn't been
- * added into yet, the size returned will be PAGE_SIZE.
- */
-static inline size_t folioq_folio_size(const struct folio_queue *folioq, unsigned int slot)
-{
- return PAGE_SIZE << folioq_folio_order(folioq, slot);
-}
-
-/**
- * folioq_clear: Clear a folio from a folio queue segment
- * @folioq: The segment to clear
- * @slot: The folio slot to clear
- *
- * Clear a folio from a sequence in a folio queue segment and clear its marks.
- * The occupancy count is left unchanged.
- */
-static inline void folioq_clear(struct folio_queue *folioq, unsigned int slot)
-{
- folioq->vec.folios[slot] = NULL;
- folioq_unmark(folioq, slot);
- folioq_unmark2(folioq, slot);
-}
-
-#endif /* _LINUX_FOLIO_QUEUE_H */
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index 58fdb9605425dc..850d20241075bf 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
@@ -147,6 +147,23 @@ struct fscache_cookie {
};
};
+enum fscache_extent_type {
+ FSCACHE_EXTENT_DATA,
+ FSCACHE_EXTENT_ZERO,
+} __mode(byte);
+
+/*
+ * Cache occupancy information.
+ */
+struct fscache_occupancy {
+ unsigned long long query_from; /* Point to query from */
+ unsigned long long query_to; /* Point to query to */
+ unsigned long long cached_from[2]; /* Point at which cache extents start */
+ unsigned long long cached_to[2]; /* Point at which cache extents end */
+ unsigned int granularity; /* Granularity desired */
+ enum fscache_extent_type cached_type[2]; /* Type of cache extent */
+};
+
/*
* slow-path functions for when there is actually caching available, and the
* netfs does actually have a valid token
diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h
index f9a17fbbd3980b..4e7d69b4ace239 100644
--- a/include/linux/iov_iter.h
+++ b/include/linux/iov_iter.h
@@ -10,7 +10,7 @@
#include <linux/uio.h>
#include <linux/bvec.h>
-#include <linux/folio_queue.h>
+#include <linux/bvecq.h>
typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len,
void *priv, void *priv2);
@@ -142,56 +142,65 @@ size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2,
}
/*
- * Handle ITER_FOLIOQ.
+ * Handle ITER_BVECQ.
*/
static __always_inline
-size_t iterate_folioq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
- iov_step_f step)
+size_t iterate_bvecq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
+ iov_step_f step)
{
- const struct folio_queue *folioq = iter->folioq;
- unsigned int slot = iter->folioq_slot;
+ const struct bvecq *bq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
size_t progress = 0, skip = iter->iov_offset;
- if (slot == folioq_nr_slots(folioq)) {
- /* The iterator may have been extended. */
- folioq = folioq->next;
- slot = 0;
- }
-
do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t part, remain = 0, consumed;
- size_t fsize;
+ const struct bio_vec *bvec;
+ struct page *page;
+ size_t poff, plen;
void *base;
- if (!folio)
- break;
+ if (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
+ slot = 0;
+ continue;
+ }
+
+ bvec = &bq->bv[slot];
+ /*
+ * The caller must ensure that a slot with bv_len>0 has a valid
+ * bv_page.
+ */
+ page = bvec->bv_page + (bvec->bv_offset + skip) / PAGE_SIZE;
+ poff = (bvec->bv_offset + skip) % PAGE_SIZE;
+ plen = umin(bvec->bv_len - skip, len);
+
+ while (plen > 0) {
+ size_t part, remain, consumed;
- fsize = folioq_folio_size(folioq, slot);
- if (skip < fsize) {
- base = kmap_local_folio(folio, skip);
- part = umin(len, PAGE_SIZE - skip % PAGE_SIZE);
+ part = umin(plen, PAGE_SIZE - poff);
+ base = kmap_local_page(page) + poff;
remain = step(base, progress, part, priv, priv2);
kunmap_local(base);
+
consumed = part - remain;
- len -= consumed;
progress += consumed;
skip += consumed;
+ len -= consumed;
+ if (!len || remain)
+ goto stop;
+ page++;
+ poff = 0;
+ plen -= consumed;
}
- if (skip >= fsize) {
- skip = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- if (remain)
- break;
+
+ skip = 0;
+ slot++;
} while (len);
- iter->folioq_slot = slot;
- iter->folioq = folioq;
+stop:
+ iter->bvecq_slot = slot;
+ iter->bvecq = bq;
iter->iov_offset = skip;
iter->count -= progress;
return progress;
@@ -306,8 +315,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
- if (iov_iter_is_folioq(iter))
- return iterate_folioq(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
return iterate_xarray(iter, len, priv, priv2, step);
return iterate_discard(iter, len, priv, priv2, step);
@@ -342,7 +351,7 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv,
* buffer is presented in segments, which for kernel iteration are broken up by
* physical pages and mapped, with the mapped address being presented.
*
- * [!] Note This will only handle BVEC, KVEC, FOLIOQ, XARRAY and DISCARD-type
+ * [!] Note This will only handle BVEC, KVEC, BVECQ, XARRAY and DISCARD-type
* iterators; it will not handle UBUF or IOVEC-type iterators.
*
* A step functions, @step, must be provided, one for handling mapped kernel
@@ -370,8 +379,8 @@ size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
- if (iov_iter_is_folioq(iter))
- return iterate_folioq(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
return iterate_xarray(iter, len, priv, priv2, step);
return iterate_discard(iter, len, priv, priv2, step);
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 1bc120d61c5beb..2ed79836ecd6e6 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -17,12 +17,15 @@
#include <linux/workqueue.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
-#include <linux/rolling_buffer.h>
enum netfs_sreq_ref_trace;
typedef struct mempool mempool_t;
-struct folio_queue;
+struct readahead_control;
+struct netfs_io_request;
+struct netfs_io_subrequest;
+struct fscache_occupancy;
/**
* folio_start_private_2 - Start an fscache write on a folio. [DEPRECATED]
@@ -63,7 +66,7 @@ struct netfs_inode {
#endif
struct list_head wb_queue; /* Queue of processes wanting to do writeback */
loff_t _remote_i_size; /* Size of the remote file */
- loff_t _zero_point; /* Size after which we assume there's no data
+ unsigned long long _zero_point; /* Size after which we assume there's no data
* on the server */
spinlock_t lock; /* Lock covering wb_queue */
atomic_t io_count; /* Number of outstanding reqs */
@@ -125,26 +128,39 @@ static inline struct netfs_group *netfs_folio_group(struct folio *folio)
return priv;
}
+/*
+ * Estimate of maximum write subrequest for writeback. The filesystem is
+ * responsible for filling this in when called from ->estimate_write(), though
+ * netfslib will preset infinite defaults.
+ */
+struct netfs_write_estimate {
+ unsigned long long issue_at; /* Point at which we must submit */
+ int max_segs; /* Max number of segments in a single RPC */
+};
+
/*
* Stream of I/O subrequests going to a particular destination, such as the
* server or the local cache. This is mainly intended for writing where we may
* have to write to multiple destinations concurrently.
*/
struct netfs_io_stream {
- /* Submission tracking */
- struct netfs_io_subrequest *construct; /* Op being constructed */
- size_t sreq_max_len; /* Maximum size of a subrequest */
- unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */
- unsigned int submit_off; /* Folio offset we're submitting from */
- unsigned int submit_len; /* Amount of data left to submit */
- unsigned int submit_extendable_to; /* Amount I/O can be rounded up to */
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ /* Submission tracking (main dispatch only; not retry) */
+ struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */
+ unsigned long long issue_from; /* Current issue point */
+ size_t buffered; /* Amount in buffer */
+ u8 applicable; /* What sources are applicable (NOTE_* mask) */
+ bool buffering; /* T if buffering on this stream */
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
+ atomic64_t issued_to; /* Point to which can be considered issued */
+
/* Collection tracking */
struct list_head subrequests; /* Contributory I/O operations */
unsigned long long collected_to; /* Position we've collected results to */
size_t transferred; /* The amount transferred from this stream */
- unsigned short error; /* Aggregate error for the stream */
+ short error; /* Aggregate error for the stream */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* Index of stream in parent table */
bool avail; /* T if stream is available */
@@ -152,6 +168,7 @@ struct netfs_io_stream {
bool need_retry; /* T if this stream needs retrying */
bool failed; /* T if this stream failed */
bool transferred_valid; /* T is ->transferred is valid */
+ bool cancelled; /* T if stream is cancelled */
};
/*
@@ -161,8 +178,11 @@ struct netfs_cache_resources {
const struct netfs_cache_ops *ops;
void *cache_priv;
void *cache_priv2;
- unsigned int debug_id; /* Cookie debug ID */
+ unsigned long long cache_i_size; /* Initial size of cache file */
+ unsigned int cookie_id; /* Cache cookie debug ID */
+ unsigned int object_id; /* Cache object debug ID */
unsigned int inval_counter; /* object->inval_counter at begin_op */
+ unsigned int dio_size; /* DIO block size */
};
/*
@@ -176,14 +196,15 @@ struct netfs_io_subrequest {
struct netfs_io_request *rreq; /* Supervising I/O request */
struct work_struct work;
struct list_head rreq_link; /* Link in rreq->subrequests */
- struct iov_iter io_iter; /* Iterator for this subrequest */
+ struct bvecq_pos dispatch_pos; /* Bookmark in the combined queue of the start */
+ struct bvecq_pos content; /* The (copied) content of the subrequest */
unsigned long long start; /* Where to start the I/O */
size_t len; /* Size of the I/O */
size_t transferred; /* Amount of data transferred */
+ unsigned int nr_segs; /* Number of segments in content */
refcount_t ref;
short error; /* 0 or error that occurred */
unsigned short debug_index; /* Index in list (for debugging output) */
- unsigned int nr_segs; /* Number of segs in io_iter */
u8 retry_count; /* The number of retries (0 on initial pass) */
enum netfs_io_source source; /* Where to read from/write to */
unsigned char stream_nr; /* I/O stream this belongs to */
@@ -191,12 +212,11 @@ struct netfs_io_subrequest {
#define NETFS_SREQ_COPY_TO_CACHE 0 /* Set if should copy the data to the cache */
#define NETFS_SREQ_CLEAR_TAIL 1 /* Set if the rest of the read should be cleared */
#define NETFS_SREQ_MADE_PROGRESS 4 /* Set if we transferred at least some data */
-#define NETFS_SREQ_ONDEMAND 5 /* Set if it's from on-demand read mode */
-#define NETFS_SREQ_BOUNDARY 6 /* Set if ends on hard boundary (eg. ceph object) */
#define NETFS_SREQ_HIT_EOF 7 /* Set if short due to EOF */
#define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */
#define NETFS_SREQ_NEED_RETRY 9 /* Set if the filesystem requests a retry */
#define NETFS_SREQ_FAILED 10 /* Set if the subreq failed unretryably */
+#define NETFS_SREQ_CANCELLED 11 /* Set if the subreq was cancelled by netfslib */
};
enum netfs_io_origin {
@@ -209,7 +229,6 @@ enum netfs_io_origin {
NETFS_DIO_READ, /* This is a direct I/O read */
NETFS_WRITEBACK, /* This write was triggered by writepages */
NETFS_WRITEBACK_SINGLE, /* This monolithic write was triggered by writepages */
- NETFS_WRITETHROUGH, /* This write was made by netfs_perform_write() */
NETFS_UNBUFFERED_WRITE, /* This is an unbuffered write */
NETFS_DIO_WRITE, /* This is a direct I/O write */
NETFS_PGPRIV2_COPY_TO_CACHE, /* [DEPRECATED] This is writing read data to the cache */
@@ -237,25 +256,29 @@ struct netfs_io_request {
struct netfs_io_stream io_streams[2]; /* Streams of parallel I/O operations */
#define NR_IO_STREAMS 2 //wreq->nr_io_streams
struct netfs_group *group; /* Writeback group being written back */
- struct rolling_buffer buffer; /* Unencrypted buffer */
-#define NETFS_ROLLBUF_PUT_MARK ROLLBUF_MARK_1
-#define NETFS_ROLLBUF_PAGECACHE_MARK ROLLBUF_MARK_2
+ struct bvecq *spare; /* Advance allocation of bvecq */
+ struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */
+ struct bvecq_pos collect_cursor; /* Clear-up point of I/O buffer */
+ struct bvecq_pos retry_cursor; /* Point from which retries are dispatched */
wait_queue_head_t waitq; /* Processor waiter */
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
- struct bio_vec *direct_bv; /* DIO buffer list (when handling iovec-iter) */
unsigned long long submitted; /* Amount submitted for I/O so far */
unsigned long long len; /* Length of the request */
size_t transferred; /* Amount to be indicated as transferred */
long error; /* 0 or error that occurred */
unsigned long long i_size; /* Size of the file */
unsigned long long start; /* Start position */
- atomic64_t issued_to; /* Write issuer folio cursor */
unsigned long long collected_to; /* Point we've collected to */
+ unsigned long long cache_coll_to; /* Point the cache has collected to */
unsigned long long cleaned_to; /* Position we've cleaned folios to */
unsigned long long abandon_to; /* Position to abandon folios to */
+#ifdef CONFIG_NETFS_PGPRIV2
+ unsigned long long last_end; /* End of last folio added */
+#endif
+ unsigned long long retry_start; /* Position to retry from */
+ size_t retry_buffered; /* Amount of data to retry */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
- unsigned int direct_bv_count; /* Number of elements in direct_bv[] */
unsigned int debug_id;
unsigned int rsize; /* Maximum read size (0 for none) */
unsigned int wsize; /* Maximum write size (0 for none) */
@@ -264,7 +287,6 @@ struct netfs_io_request {
spinlock_t lock; /* Lock for queuing subreqs */
unsigned char front_folio_order; /* Order (size) of front folio */
enum netfs_io_origin origin; /* Origin of the request */
- bool direct_bv_unpin; /* T if direct_bv[] must be unpinned */
refcount_t ref;
unsigned long flags;
#define NETFS_RREQ_IN_PROGRESS 0 /* Unlocked when the request completes (has ref) */
@@ -273,13 +295,20 @@ struct netfs_io_request {
#define NETFS_RREQ_FAILED 3 /* The request failed */
#define NETFS_RREQ_RETRYING 4 /* Set if we're in the retry path */
#define NETFS_RREQ_SHORT_TRANSFER 5 /* Set if we have a short transfer */
-#define NETFS_RREQ_OFFLOAD_COLLECTION 8 /* Offload collection to workqueue */
-#define NETFS_RREQ_NO_UNLOCK_FOLIO 9 /* Don't unlock no_unlock_folio on completion */
-#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 10 /* Copy current folio to cache from read */
-#define NETFS_RREQ_UPLOAD_TO_SERVER 11 /* Need to write to the server */
-#define NETFS_RREQ_USE_IO_ITER 12 /* Use ->io_iter rather than ->i_pages */
+#define NETFS_RREQ_ABANDON_REQ 6 /* Set if the request is to be abandoned */
+#define NETFS_RREQ_SAW_ENOMEM 7 /* Set if we encounted ENOMEM */
+#define NETFS_RREQ_CACHE_STOP 8 /* Set to stop caching (ENOBUFS or error) */
+#define NETFS_RREQ_CACHE_ERROR 9 /* Set if we got an error from the cache */
+#define NETFS_RREQ_OFFLOAD_COLLECTION 12 /* Offload collection to workqueue */
+#define NETFS_RREQ_NO_UNLOCK_FOLIO 13 /* Don't unlock no_unlock_folio on completion */
+#define NETFS_RREQ_FOLIO_COPY_TO_CACHE 14 /* Copy current folio to cache from read */
+#define NETFS_RREQ_UPLOAD_TO_SERVER 15 /* Need to write to the server */
+#define NETFS_RREQ_USE_IO_ITER 16 /* Use ->io_iter rather than ->i_pages */
+#define NETFS_RREQ_NEED_PUT_RA_REFS 17 /* Need to put the folio refs RA gave us */
+#ifdef CONFIG_NETFS_PGPRIV2
#define NETFS_RREQ_USE_PGPRIV2 31 /* [DEPRECATED] Use PG_private_2 to mark
* write to cache on read */
+#endif
const struct netfs_request_ops *netfs_ops;
};
@@ -295,8 +324,7 @@ struct netfs_request_ops {
/* Read request handling */
void (*expand_readahead)(struct netfs_io_request *rreq);
- int (*prepare_read)(struct netfs_io_subrequest *subreq);
- void (*issue_read)(struct netfs_io_subrequest *subreq);
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
bool (*is_still_valid)(struct netfs_io_request *rreq);
int (*check_write_begin)(struct file *file, loff_t pos, unsigned len,
struct folio **foliop, void **_fsdata);
@@ -308,8 +336,10 @@ struct netfs_request_ops {
/* Write request handling */
void (*begin_writeback)(struct netfs_io_request *wreq);
- void (*prepare_write)(struct netfs_io_subrequest *subreq);
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
void (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream);
void (*invalidate_cache)(struct netfs_io_request *wreq);
};
@@ -322,6 +352,12 @@ enum netfs_read_from_hole {
NETFS_READ_HOLE_FAIL,
};
+enum netfs_cache_collect {
+ NETFS_CACHE_COLLECT_WRITE_DATA,
+ NETFS_CACHE_COLLECT_WRITE_GAP,
+ NETFS_CACHE_COLLECT_WRITE_CANCEL,
+};
+
/*
* Table of operations for access to a cache.
*/
@@ -344,8 +380,16 @@ struct netfs_cache_ops {
netfs_io_terminated_t term_func,
void *term_func_priv);
+ /* Estimate the amount of data that can be written in an op. */
+ int (*estimate_write)(struct netfs_io_request *wreq,
+ struct netfs_io_stream *stream,
+ struct netfs_write_estimate *estimate);
+
+ /* Read data from the cache for a netfs subrequest. */
+ int (*issue_read)(struct netfs_io_subrequest *subreq);
+
/* Write data to the cache from a netfs subrequest. */
- void (*issue_write)(struct netfs_io_subrequest *subreq);
+ int (*issue_write)(struct netfs_io_subrequest *subreq);
/* Expand readahead request */
void (*expand_readahead)(struct netfs_cache_resources *cres,
@@ -353,40 +397,28 @@ struct netfs_cache_ops {
unsigned long long *_len,
unsigned long long i_size);
- /* Prepare a read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
- enum netfs_io_source (*prepare_read)(struct netfs_io_subrequest *subreq,
- unsigned long long i_size);
-
- /* Prepare a write subrequest, working out if we're allowed to do it
- * and finding out the maximum amount of data to gather before
- * attempting to submit. If we're not permitted to do it, the
- * subrequest should be marked failed.
- */
- void (*prepare_write_subreq)(struct netfs_io_subrequest *subreq);
-
/* Prepare a write operation, working out what part of the write we can
* actually do.
*/
- int (*prepare_write)(struct netfs_cache_resources *cres,
- loff_t *_start, size_t *_len, size_t upper_len,
- loff_t i_size, bool no_space_allocated_yet);
-
- /* Prepare an on-demand read operation, shortening it to a cached/uncached
- * boundary as appropriate.
- */
- enum netfs_io_source (*prepare_ondemand_read)(struct netfs_cache_resources *cres,
- loff_t start, size_t *_len,
- loff_t i_size,
- unsigned long *_flags, ino_t ino);
+ int (*prepare_write_old)(struct netfs_cache_resources *cres,
+ loff_t *_start, size_t *_len, size_t upper_len,
+ loff_t i_size, bool no_space_allocated_yet);
/* Query the occupancy of the cache in a region, returning where the
* next chunk of data starts and how long it is.
*/
- int (*query_occupancy)(struct netfs_cache_resources *cres,
- loff_t start, size_t len, size_t granularity,
- loff_t *_data_start, size_t *_data_len);
+ void (*query_occupancy)(struct netfs_cache_resources *cres,
+ struct fscache_occupancy *occ);
+
+ /* Collect the result of buffered writeback to the cache. This
+ * includes copying a read to the cache. block_type is one of:
+ * - NETFS_CACHE_COLLECT_WRITE_DATA for a block of data
+ * - NETFS_CACHE_COLLECT_WRITE_GAP if a discontiguity was skipped
+ * - NETFS_CACHE_COLLECT_WRITE_CANCEL for a cancellation gap
+ */
+ void (*collect_write)(struct netfs_io_request *wreq,
+ unsigned long long start, size_t len,
+ enum netfs_cache_collect block_type);
};
/* High-level read API. */
@@ -410,10 +442,9 @@ void netfs_single_mark_inode_dirty(struct inode *inode);
ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_iter *iter);
int netfs_writeback_single(struct address_space *mapping,
struct writeback_control *wbc,
- struct iov_iter *iter);
+ struct iov_iter *iter, size_t len);
/* Address operations API */
-struct readahead_control;
void netfs_readahead(struct readahead_control *);
int netfs_read_folio(struct file *, struct folio *);
int netfs_write_begin(struct netfs_inode *, struct file *,
@@ -437,12 +468,11 @@ void netfs_get_subrequest(struct netfs_io_subrequest *subreq,
enum netfs_sreq_ref_trace what);
void netfs_put_subrequest(struct netfs_io_subrequest *subreq,
enum netfs_sreq_ref_trace what);
-ssize_t netfs_extract_user_iter(struct iov_iter *orig, size_t orig_len,
- struct iov_iter *new,
- iov_iter_extraction_t extraction_flags);
-size_t netfs_limit_iter(const struct iov_iter *iter, size_t start_offset,
- size_t max_size, size_t max_segs);
-void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq);
+ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages,
+ unsigned long long fpos, struct bvecq **_bvecq_head,
+ iov_iter_extraction_t extraction_flags);
+int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
+int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs);
void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error);
int netfs_start_io_read(struct inode *inode);
@@ -452,18 +482,6 @@ void netfs_end_io_write(struct inode *inode);
int netfs_start_io_direct(struct inode *inode);
void netfs_end_io_direct(struct inode *inode);
-/* Miscellaneous APIs. */
-struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
- unsigned int trace /*enum netfs_folioq_trace*/);
-void netfs_folioq_free(struct folio_queue *folioq,
- unsigned int trace /*enum netfs_trace_folioq*/);
-
-/* Buffer wrangling helpers API. */
-int netfs_alloc_folioq_buffer(struct address_space *mapping,
- struct folio_queue **_buffer,
- size_t *_cur_size, ssize_t size, gfp_t gfp);
-void netfs_free_folioq_buffer(struct folio_queue *fq);
-
/* Writeback exclusion API. */
bool netfs_wb_begin(struct netfs_inode *ictx, bool nowait);
void netfs_wb_end(struct netfs_inode *ictx);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 2c3718d592d6dd..e1e51bace388c9 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1348,6 +1348,7 @@ struct readahead_control {
struct file_ra_state *ra;
/* private: use the readahead_* accessors instead */
pgoff_t _index;
+ unsigned int _nr_folios;
unsigned int _nr_pages;
unsigned int _batch_count;
bool dropbehind;
@@ -1527,6 +1528,15 @@ static inline size_t readahead_batch_length(const struct readahead_control *rac)
return rac->_batch_count * PAGE_SIZE;
}
+/**
+ * readahead_folio_count - Get the number of folios in this readahead request.
+ * @rac: The readahead request.
+ */
+static inline unsigned int readahead_folio_count(const struct readahead_control *rac)
+{
+ return rac->_nr_folios;
+}
+
static inline unsigned long dir_pages(const struct inode *inode)
{
return (unsigned long)(inode->i_size + PAGE_SIZE - 1) >>
diff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h
deleted file mode 100644
index ac15b1ffdd8315..00000000000000
--- a/include/linux/rolling_buffer.h
+++ /dev/null
@@ -1,61 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/* Rolling buffer of folios
- *
- * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells (dhowells@redhat.com)
- */
-
-#ifndef _ROLLING_BUFFER_H
-#define _ROLLING_BUFFER_H
-
-#include <linux/folio_queue.h>
-#include <linux/uio.h>
-
-/*
- * Rolling buffer. Whilst the buffer is live and in use, folios and folio
- * queue segments can be added to one end by one thread and removed from the
- * other end by another thread. The buffer isn't allowed to be empty; it must
- * always have at least one folio_queue in it so that neither side has to
- * modify both queue pointers.
- *
- * The iterator in the buffer is extended as buffers are inserted. It can be
- * snapshotted to use a segment of the buffer.
- */
-struct rolling_buffer {
- struct folio_queue *head; /* Producer's insertion point */
- struct folio_queue *tail; /* Consumer's removal point */
- struct iov_iter iter; /* Iterator tracking what's left in the buffer */
- u8 next_head_slot; /* Next slot in ->head */
- u8 first_tail_slot; /* First slot in ->tail */
-};
-
-/*
- * Snapshot of a rolling buffer.
- */
-struct rolling_buffer_snapshot {
- struct folio_queue *curr_folioq; /* Queue segment in which current folio resides */
- unsigned char curr_slot; /* Folio currently being read */
- unsigned char curr_order; /* Order of folio */
-};
-
-/* Marks to store per-folio in the internal folio_queue structs. */
-#define ROLLBUF_MARK_1 BIT(0)
-#define ROLLBUF_MARK_2 BIT(1)
-
-int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction);
-int rolling_buffer_make_space(struct rolling_buffer *roll);
-ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
- struct readahead_control *ractl,
- struct folio_batch *put_batch);
-ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags);
-struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);
-void rolling_buffer_clear(struct rolling_buffer *roll);
-
-static inline void rolling_buffer_advance(struct rolling_buffer *roll, size_t amount)
-{
- iov_iter_advance(&roll->iter, amount);
-}
-
-#endif /* _ROLLING_BUFFER_H */
diff --git a/include/linux/uio.h b/include/linux/uio.h
index a9bc5b3067e320..e84a0c4f28c6c8 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -11,7 +11,6 @@
#include <uapi/linux/uio.h>
struct page;
-struct folio_queue;
typedef unsigned int __bitwise iov_iter_extraction_t;
@@ -26,7 +25,7 @@ enum iter_type {
ITER_IOVEC,
ITER_BVEC,
ITER_KVEC,
- ITER_FOLIOQ,
+ ITER_BVECQ,
ITER_XARRAY,
ITER_DISCARD,
};
@@ -68,7 +67,7 @@ struct iov_iter {
const struct iovec *__iov;
const struct kvec *kvec;
const struct bio_vec *bvec;
- const struct folio_queue *folioq;
+ const struct bvecq *bvecq;
struct xarray *xarray;
void __user *ubuf;
};
@@ -77,7 +76,7 @@ struct iov_iter {
};
union {
unsigned long nr_segs;
- u8 folioq_slot;
+ u16 bvecq_slot;
loff_t xarray_start;
};
};
@@ -145,9 +144,9 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i)
return iov_iter_type(i) == ITER_DISCARD;
}
-static inline bool iov_iter_is_folioq(const struct iov_iter *i)
+static inline bool iov_iter_is_bvecq(const struct iov_iter *i)
{
- return iov_iter_type(i) == ITER_FOLIOQ;
+ return iov_iter_type(i) == ITER_BVECQ;
}
static inline bool iov_iter_is_xarray(const struct iov_iter *i)
@@ -295,9 +294,9 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec
void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec,
unsigned long nr_segs, size_t count);
void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count);
-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq,
- unsigned int first_slot, unsigned int offset, size_t count);
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq,
+ unsigned int first_slot, unsigned int offset, size_t count);
void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray,
loff_t start, size_t count);
ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
diff --git a/include/trace/events/cachefiles.h b/include/trace/events/cachefiles.h
index 6e3b1424eea4dc..b13322b376e923 100644
--- a/include/trace/events/cachefiles.h
+++ b/include/trace/events/cachefiles.h
@@ -31,8 +31,6 @@ enum cachefiles_obj_ref_trace {
cachefiles_obj_see_lookup_failed,
cachefiles_obj_see_withdraw_cookie,
cachefiles_obj_see_withdrawal,
- cachefiles_obj_get_ondemand_fd,
- cachefiles_obj_put_ondemand_fd,
cachefiles_obj_get_read_req,
cachefiles_obj_put_read_req,
};
@@ -56,6 +54,8 @@ enum cachefiles_coherency_trace {
cachefiles_coherency_check_ok,
cachefiles_coherency_check_type,
cachefiles_coherency_check_xattr,
+ cachefiles_coherency_discontiguous,
+ cachefiles_coherency_remove,
cachefiles_coherency_set_fail,
cachefiles_coherency_set_ok,
cachefiles_coherency_vol_check_cmp,
@@ -67,6 +67,7 @@ enum cachefiles_coherency_trace {
};
enum cachefiles_trunc_trace {
+ cachefiles_trunc_clear_padding,
cachefiles_trunc_dio_adjust,
cachefiles_trunc_expand_tmpfile,
cachefiles_trunc_shrink,
@@ -84,6 +85,7 @@ enum cachefiles_prepare_read_trace {
};
enum cachefiles_error_trace {
+ cachefiles_trace_alignment_error,
cachefiles_trace_fallocate_error,
cachefiles_trace_getxattr_error,
cachefiles_trace_link_error,
@@ -130,8 +132,6 @@ enum cachefiles_error_trace {
EM(cachefiles_obj_see_lookup_failed, "SEE lookup_failed") \
EM(cachefiles_obj_see_withdraw_cookie, "SEE withdraw_cookie") \
EM(cachefiles_obj_see_withdrawal, "SEE withdrawal") \
- EM(cachefiles_obj_get_ondemand_fd, "GET ondemand_fd") \
- EM(cachefiles_obj_put_ondemand_fd, "PUT ondemand_fd") \
EM(cachefiles_obj_get_read_req, "GET read_req") \
E_(cachefiles_obj_put_read_req, "PUT read_req")
@@ -144,6 +144,8 @@ enum cachefiles_error_trace {
EM(cachefiles_coherency_check_ok, "OK ") \
EM(cachefiles_coherency_check_type, "BAD type") \
EM(cachefiles_coherency_check_xattr, "BAD xatt") \
+ EM(cachefiles_coherency_discontiguous, "--- gap ") \
+ EM(cachefiles_coherency_remove, "REMOVE ") \
EM(cachefiles_coherency_set_fail, "SET fail") \
EM(cachefiles_coherency_set_ok, "SET ok ") \
EM(cachefiles_coherency_vol_check_cmp, "VOL BAD cmp ") \
@@ -154,6 +156,7 @@ enum cachefiles_error_trace {
E_(cachefiles_coherency_vol_set_ok, "VOL SET ok ")
#define cachefiles_trunc_traces \
+ EM(cachefiles_trunc_clear_padding, "CLRPAD") \
EM(cachefiles_trunc_dio_adjust, "DIOADJ") \
EM(cachefiles_trunc_expand_tmpfile, "EXPTMP") \
E_(cachefiles_trunc_shrink, "SHRINK")
@@ -169,6 +172,7 @@ enum cachefiles_error_trace {
E_(cachefiles_trace_read_seek_nxio, "seek-enxio")
#define cachefiles_error_traces \
+ EM(cachefiles_trace_alignment_error, "align") \
EM(cachefiles_trace_fallocate_error, "fallocate") \
EM(cachefiles_trace_getxattr_error, "getxattr") \
EM(cachefiles_trace_link_error, "link") \
@@ -379,12 +383,12 @@ TRACE_EVENT(cachefiles_rename,
TRACE_EVENT(cachefiles_coherency,
TP_PROTO(struct cachefiles_object *obj,
- ino_t ino,
+ ino_t ino, unsigned long long obj_size,
u64 disk_aux,
enum cachefiles_content content,
enum cachefiles_coherency_trace why),
- TP_ARGS(obj, ino, disk_aux, content, why),
+ TP_ARGS(obj, ino, obj_size, disk_aux, content, why),
/* Note that obj may be NULL */
TP_STRUCT__entry(
@@ -392,6 +396,7 @@ TRACE_EVENT(cachefiles_coherency,
__field(enum cachefiles_coherency_trace, why)
__field(enum cachefiles_content, content)
__field(u64, ino)
+ __field(u64, obj_size)
__field(u64, aux)
__field(u64, disk_aux)
),
@@ -401,14 +406,16 @@ TRACE_EVENT(cachefiles_coherency,
__entry->why = why;
__entry->content = content;
__entry->ino = ino;
+ __entry->obj_size = obj_size;
__entry->aux = be64_to_cpup((__be64 *)obj->cookie->inline_aux);
__entry->disk_aux = disk_aux;
),
- TP_printk("o=%08x %s B=%llx c=%u aux=%llx dsk=%llx",
+ TP_printk("o=%08x %s B=%llx oz=%llx c=%u aux=%llx dsk=%llx",
__entry->obj,
__print_symbolic(__entry->why, cachefiles_coherency_traces),
__entry->ino,
+ __entry->obj_size,
__entry->content,
__entry->aux,
__entry->disk_aux)
@@ -687,180 +694,6 @@ TRACE_EVENT(cachefiles_io_error,
__entry->error)
);
-TRACE_EVENT(cachefiles_ondemand_open,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,
- struct cachefiles_open *load),
-
- TP_ARGS(obj, msg, load),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- __field(unsigned int, fd)
- __field(unsigned int, flags)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- __entry->fd = load->fd;
- __entry->flags = load->flags;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x fd=%d f=%x",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id,
- __entry->fd,
- __entry->flags)
- );
-
-TRACE_EVENT(cachefiles_ondemand_copen,
- TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id,
- long len),
-
- TP_ARGS(obj, msg_id, len),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(long, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg_id;
- __entry->len = len;
- ),
-
- TP_printk("o=%08x mid=%x l=%lx",
- __entry->obj,
- __entry->msg_id,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_close,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg),
-
- TP_ARGS(obj, msg),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id)
- );
-
-TRACE_EVENT(cachefiles_ondemand_read,
- TP_PROTO(struct cachefiles_object *obj, struct cachefiles_msg *msg,
- struct cachefiles_read *load),
-
- TP_ARGS(obj, msg, load),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- __field(unsigned int, object_id)
- __field(loff_t, start)
- __field(size_t, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg->msg_id;
- __entry->object_id = msg->object_id;
- __entry->start = load->off;
- __entry->len = load->len;
- ),
-
- TP_printk("o=%08x mid=%x oid=%x s=%llx l=%zx",
- __entry->obj,
- __entry->msg_id,
- __entry->object_id,
- __entry->start,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_cread,
- TP_PROTO(struct cachefiles_object *obj, unsigned int msg_id),
-
- TP_ARGS(obj, msg_id),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, msg_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->msg_id = msg_id;
- ),
-
- TP_printk("o=%08x mid=%x",
- __entry->obj,
- __entry->msg_id)
- );
-
-TRACE_EVENT(cachefiles_ondemand_fd_write,
- TP_PROTO(struct cachefiles_object *obj, struct inode *backer,
- loff_t start, size_t len),
-
- TP_ARGS(obj, backer, start, len),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, backer)
- __field(loff_t, start)
- __field(size_t, len)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->backer = backer->i_ino;
- __entry->start = start;
- __entry->len = len;
- ),
-
- TP_printk("o=%08x iB=%x s=%llx l=%zx",
- __entry->obj,
- __entry->backer,
- __entry->start,
- __entry->len)
- );
-
-TRACE_EVENT(cachefiles_ondemand_fd_release,
- TP_PROTO(struct cachefiles_object *obj, int object_id),
-
- TP_ARGS(obj, object_id),
-
- TP_STRUCT__entry(
- __field(unsigned int, obj)
- __field(unsigned int, object_id)
- ),
-
- TP_fast_assign(
- __entry->obj = obj ? obj->debug_id : 0;
- __entry->object_id = object_id;
- ),
-
- TP_printk("o=%08x oid=%x",
- __entry->obj,
- __entry->object_id)
- );
-
#endif /* _TRACE_CACHEFILES_H */
/* This part must be outside protection */
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 082cb03c613160..e6bbf2a09d92e8 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -30,8 +30,7 @@
EM(netfs_write_trace_dio_write, "DIO-WRITE") \
EM(netfs_write_trace_unbuffered_write, "UNB-WRITE") \
EM(netfs_write_trace_writeback, "WRITEBACK") \
- EM(netfs_write_trace_writeback_single, "WB-SINGLE") \
- E_(netfs_write_trace_writethrough, "WRITETHRU")
+ E_(netfs_write_trace_writeback_single, "WB-SINGLE")
#define netfs_rreq_origins \
EM(NETFS_READAHEAD, "RA") \
@@ -43,13 +42,17 @@
EM(NETFS_DIO_READ, "DR") \
EM(NETFS_WRITEBACK, "WB") \
EM(NETFS_WRITEBACK_SINGLE, "W1") \
- EM(NETFS_WRITETHROUGH, "WT") \
EM(NETFS_UNBUFFERED_WRITE, "UW") \
EM(NETFS_DIO_WRITE, "DW") \
E_(NETFS_PGPRIV2_COPY_TO_CACHE, "2C")
#define netfs_rreq_traces \
+ EM(netfs_rreq_trace_all_queued, "ALL-Q ") \
EM(netfs_rreq_trace_assess, "ASSESS ") \
+ EM(netfs_rreq_trace_cache_cancelled, "CA-CNCL") \
+ EM(netfs_rreq_trace_cache_failed, "CA-FAIL") \
+ EM(netfs_rreq_trace_cache_fail_collect, "CA-F-CO") \
+ EM(netfs_rreq_trace_cache_no_space, "CA-NOSP") \
EM(netfs_rreq_trace_collect, "COLLECT") \
EM(netfs_rreq_trace_complete, "COMPLET") \
EM(netfs_rreq_trace_copy, "COPY ") \
@@ -58,10 +61,14 @@
EM(netfs_rreq_trace_end_copy_to_cache, "END-C2C") \
EM(netfs_rreq_trace_free, "FREE ") \
EM(netfs_rreq_trace_intr, "INTR ") \
+ EM(netfs_rreq_trace_inval_cache, "INVL-CA") \
EM(netfs_rreq_trace_ki_complete, "KI-CMPL") \
+ EM(netfs_rreq_trace_ra_put_ref, "RA-PUT ") \
EM(netfs_rreq_trace_recollect, "RECLLCT") \
EM(netfs_rreq_trace_redirty, "REDIRTY") \
EM(netfs_rreq_trace_resubmit, "RESUBMT") \
+ EM(netfs_rreq_trace_retry_begin, "RETRY-BEGIN") \
+ EM(netfs_rreq_trace_retry_end, "RETRY-END") \
EM(netfs_rreq_trace_set_abandon, "S-ABNDN") \
EM(netfs_rreq_trace_set_pause, "PAUSE ") \
EM(netfs_rreq_trace_unlock, "UNLOCK ") \
@@ -70,13 +77,16 @@
EM(netfs_rreq_trace_unpause, "UNPAUSE") \
EM(netfs_rreq_trace_wait_ip, "WAIT-IP") \
EM(netfs_rreq_trace_wait_pause, "--PAUSED--") \
+ EM(netfs_rreq_trace_wait_put_ra_refs, "WAIT-P-RA") \
EM(netfs_rreq_trace_wait_quiesce, "WAIT-QUIESCE") \
EM(netfs_rreq_trace_waited_ip, "DONE-IP") \
EM(netfs_rreq_trace_waited_pause, "--UNPAUSED--") \
+ EM(netfs_rreq_trace_waited_put_ra_refs, "DONE-P-RA") \
EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \
EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \
EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \
- E_(netfs_rreq_trace_write_done, "WR-DONE")
+ EM(netfs_rreq_trace_write_done, "WR-DONE") \
+ E_(netfs_rreq_trace_zero_unread, "ZERO-UR")
#define netfs_sreq_sources \
EM(NETFS_SOURCE_UNKNOWN, "----") \
@@ -124,6 +134,8 @@
EM(netfs_sreq_trace_submit, "SUBMT") \
EM(netfs_sreq_trace_superfluous, "SPRFL") \
EM(netfs_sreq_trace_terminated, "TERM ") \
+ EM(netfs_sreq_trace_too_much, "!TOOM") \
+ EM(netfs_sreq_trace_too_many_retries, "!RETR") \
EM(netfs_sreq_trace_wait_for, "_WAIT") \
EM(netfs_sreq_trace_write, "WRITE") \
EM(netfs_sreq_trace_write_skip, "SKIP ") \
@@ -131,12 +143,12 @@
#define netfs_failures \
EM(netfs_fail_check_write_begin, "check-write-begin") \
- EM(netfs_fail_copy_to_cache, "copy-to-cache") \
EM(netfs_fail_dio_read_short, "dio-read-short") \
EM(netfs_fail_dio_read_zero, "dio-read-zero") \
EM(netfs_fail_read, "read") \
EM(netfs_fail_short_read, "short-read") \
EM(netfs_fail_prepare_write, "prep-write") \
+ EM(netfs_fail_upload, "upload") \
E_(netfs_fail_write, "write")
#define netfs_rreq_ref_traces \
@@ -191,12 +203,12 @@
EM(netfs_folio_trace_alloc_buffer, "alloc-buf") \
EM(netfs_folio_trace_cancel_copy, "cancel-copy") \
EM(netfs_folio_trace_cancel_store, "cancel-store") \
- EM(netfs_folio_trace_clear, "clear") \
- EM(netfs_folio_trace_clear_cc, "clear-cc") \
- EM(netfs_folio_trace_clear_g, "clear-g") \
- EM(netfs_folio_trace_clear_s, "clear-s") \
EM(netfs_folio_trace_copy_to_cache, "mark-copy") \
EM(netfs_folio_trace_end_copy, "end-copy") \
+ EM(netfs_folio_trace_endwb, "endwb") \
+ EM(netfs_folio_trace_endwb_cc, "endwb-cc") \
+ EM(netfs_folio_trace_endwb_g, "endwb-g") \
+ EM(netfs_folio_trace_endwb_s, "endwb-s") \
EM(netfs_folio_trace_filled_gaps, "filled-gaps") \
EM(netfs_folio_trace_invalidate_all, "inval-all") \
EM(netfs_folio_trace_invalidate_front, "inval-front") \
@@ -219,8 +231,8 @@
EM(netfs_folio_trace_store, "store") \
EM(netfs_folio_trace_store_copy, "store-copy") \
EM(netfs_folio_trace_store_plus, "store+") \
- EM(netfs_folio_trace_wthru, "wthru") \
- E_(netfs_folio_trace_wthru_plus, "wthru+")
+ EM(netfs_folio_trace_zero, "zero") \
+ E_(netfs_folio_trace_zero_ra, "zero-ra")
#define netfs_collect_contig_traces \
EM(netfs_contig_trace_collect, "Collect") \
@@ -233,13 +245,13 @@
EM(netfs_trace_donate_to_next, "to-next") \
E_(netfs_trace_donate_to_deferred_next, "defer-next")
-#define netfs_folioq_traces \
- EM(netfs_trace_folioq_alloc_buffer, "alloc-buf") \
- EM(netfs_trace_folioq_clear, "clear") \
- EM(netfs_trace_folioq_delete, "delete") \
- EM(netfs_trace_folioq_make_space, "make-space") \
- EM(netfs_trace_folioq_rollbuf_init, "roll-init") \
- E_(netfs_trace_folioq_read_progress, "r-progress")
+#define netfs_bvecq_traces \
+ EM(netfs_trace_bvecq_alloc_buffer, "alloc-buf") \
+ EM(netfs_trace_bvecq_clear, "clear") \
+ EM(netfs_trace_bvecq_delete, "delete") \
+ EM(netfs_trace_bvecq_make_space, "make-space") \
+ EM(netfs_trace_bvecq_rollbuf_init, "roll-init") \
+ E_(netfs_trace_bvecq_read_progress, "r-progress")
#ifndef __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
#define __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
@@ -259,7 +271,7 @@ enum netfs_sreq_ref_trace { netfs_sreq_ref_traces } __mode(byte);
enum netfs_folio_trace { netfs_folio_traces } __mode(byte);
enum netfs_collect_contig_trace { netfs_collect_contig_traces } __mode(byte);
enum netfs_donate_trace { netfs_donate_traces } __mode(byte);
-enum netfs_folioq_trace { netfs_folioq_traces } __mode(byte);
+enum netfs_bvecq_trace { netfs_bvecq_traces } __mode(byte);
#endif
@@ -283,7 +295,7 @@ netfs_sreq_ref_traces;
netfs_folio_traces;
netfs_collect_contig_traces;
netfs_donate_traces;
-netfs_folioq_traces;
+netfs_bvecq_traces;
/*
* Now redefine the EM() and E_() macros to map the enums to the strings that
@@ -304,6 +316,7 @@ TRACE_EVENT(netfs_read,
TP_STRUCT__entry(
__field(unsigned int, rreq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(loff_t, i_size)
__field(loff_t, start)
__field(size_t, len)
@@ -313,7 +326,8 @@ TRACE_EVENT(netfs_read,
TP_fast_assign(
__entry->rreq = rreq->debug_id;
- __entry->cookie = rreq->cache_resources.debug_id;
+ __entry->cookie = rreq->cache_resources.cookie_id;
+ __entry->object = rreq->cache_resources.object_id;
__entry->i_size = rreq->i_size;
__entry->start = start;
__entry->len = len;
@@ -321,10 +335,10 @@ TRACE_EVENT(netfs_read,
__entry->netfs_inode = rreq->inode->i_ino;
),
- TP_printk("R=%08x %s c=%08x ni=%llx s=%llx l=%zx sz=%llx",
+ TP_printk("R=%08x %s c=%08x o=%08x ni=%llx s=%llx l=%zx sz=%llx",
__entry->rreq,
__print_symbolic(__entry->what, netfs_read_traces),
- __entry->cookie,
+ __entry->cookie, __entry->object,
__entry->netfs_inode,
__entry->start, __entry->len, __entry->i_size)
);
@@ -385,10 +399,10 @@ TRACE_EVENT(netfs_sreq,
__entry->len = sreq->len;
__entry->transferred = sreq->transferred;
__entry->start = sreq->start;
- __entry->slot = sreq->io_iter.folioq_slot;
+ __entry->slot = sreq->content.slot;
),
- TP_printk("R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx s=%u e=%d",
+ TP_printk("R=%08x[%x] %s %s f=%03x s=%llx %zx/%zx bv=%u e=%d",
__entry->rreq, __entry->index,
__print_symbolic(__entry->source, netfs_sreq_sources),
__print_symbolic(__entry->what, netfs_sreq_traces),
@@ -496,6 +510,7 @@ TRACE_EVENT(netfs_folio,
TP_STRUCT__entry(
__field(u64, ino)
__field(pgoff_t, index)
+ __field(unsigned long, pfn)
__field(unsigned int, nr)
__field(enum netfs_folio_trace, why)
),
@@ -506,13 +521,40 @@ TRACE_EVENT(netfs_folio,
__entry->why = why;
__entry->index = folio->index;
__entry->nr = folio_nr_pages(folio);
+ __entry->pfn = folio_pfn(folio);
),
- TP_printk("i=%05llx ix=%05lx-%05lx %s",
+ TP_printk("p=%lx i=%05llx ix=%05lx-%05lx %s",
+ __entry->pfn,
__entry->ino, __entry->index, __entry->index + __entry->nr - 1,
__print_symbolic(__entry->why, netfs_folio_traces))
);
+TRACE_EVENT(netfs_wback,
+ TP_PROTO(struct netfs_io_request *wreq, struct folio *folio, unsigned int notes),
+
+ TP_ARGS(wreq, folio, notes),
+
+ TP_STRUCT__entry(
+ __field(pgoff_t, index)
+ __field(unsigned int, wreq)
+ __field(unsigned int, nr)
+ __field(unsigned int, notes)
+ ),
+
+ TP_fast_assign(
+ __entry->wreq = wreq->debug_id;
+ __entry->notes = notes;
+ __entry->index = folio->index;
+ __entry->nr = folio_nr_pages(folio);
+ ),
+
+ TP_printk("R=%08x ix=%05lx-%05lx n=%02x",
+ __entry->wreq,
+ __entry->index, __entry->index + __entry->nr - 1,
+ __entry->notes)
+ );
+
TRACE_EVENT(netfs_write_iter,
TP_PROTO(const struct kiocb *iocb, const struct iov_iter *from),
@@ -545,6 +587,7 @@ TRACE_EVENT(netfs_write,
TP_STRUCT__entry(
__field(unsigned int, wreq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(unsigned int, ino)
__field(enum netfs_write_trace, what)
__field(unsigned long long, start)
@@ -552,20 +595,19 @@ TRACE_EVENT(netfs_write,
),
TP_fast_assign(
- struct netfs_inode *__ctx = netfs_inode(wreq->inode);
- struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);
__entry->wreq = wreq->debug_id;
- __entry->cookie = __cookie ? __cookie->debug_id : 0;
+ __entry->cookie = wreq->cache_resources.cookie_id;
+ __entry->object = wreq->cache_resources.object_id;
__entry->ino = wreq->inode->i_ino;
__entry->what = what;
__entry->start = wreq->start;
__entry->len = wreq->len;
),
- TP_printk("R=%08x %s c=%08x i=%x by=%llx-%llx",
+ TP_printk("R=%08x %s c=%08x o=%08x i=%x by=%llx-%llx",
__entry->wreq,
__print_symbolic(__entry->what, netfs_write_traces),
- __entry->cookie,
+ __entry->cookie, __entry->object,
__entry->ino,
__entry->start, __entry->start + __entry->len - 1)
);
@@ -580,22 +622,23 @@ TRACE_EVENT(netfs_copy2cache,
__field(unsigned int, rreq)
__field(unsigned int, creq)
__field(unsigned int, cookie)
+ __field(unsigned int, object)
__field(unsigned int, ino)
),
TP_fast_assign(
- struct netfs_inode *__ctx = netfs_inode(rreq->inode);
- struct fscache_cookie *__cookie = netfs_i_cookie(__ctx);
__entry->rreq = rreq->debug_id;
__entry->creq = creq->debug_id;
- __entry->cookie = __cookie ? __cookie->debug_id : 0;
+ __entry->cookie = rreq->cache_resources.cookie_id;
+ __entry->object = rreq->cache_resources.object_id;
__entry->ino = rreq->inode->i_ino;
),
- TP_printk("R=%08x CR=%08x c=%08x i=%x ",
+ TP_printk("R=%08x CR=%08x c=%08x o=%08x i=%x ",
__entry->rreq,
__entry->creq,
__entry->cookie,
+ __entry->object,
__entry->ino)
);
@@ -755,7 +798,7 @@ TRACE_EVENT(netfs_collect_stream,
__entry->wreq = wreq->debug_id;
__entry->stream = stream->stream_nr;
__entry->collected_to = stream->collected_to;
- __entry->issued_to = atomic64_read(&wreq->issued_to);
+ __entry->issued_to = atomic64_read(&stream->issued_to);
),
TP_printk("R=%08x[%x:] cto=%llx ito=%llx",
@@ -763,27 +806,49 @@ TRACE_EVENT(netfs_collect_stream,
__entry->collected_to, __entry->issued_to)
);
-TRACE_EVENT(netfs_folioq,
- TP_PROTO(const struct folio_queue *fq,
- enum netfs_folioq_trace trace),
+TRACE_EVENT(netfs_bvecq,
+ TP_PROTO(const struct bvecq *bq,
+ enum netfs_bvecq_trace trace),
- TP_ARGS(fq, trace),
+ TP_ARGS(bq, trace),
TP_STRUCT__entry(
- __field(unsigned int, rreq)
__field(unsigned int, id)
- __field(enum netfs_folioq_trace, trace)
+ __field(enum netfs_bvecq_trace, trace)
),
TP_fast_assign(
- __entry->rreq = fq ? fq->rreq_id : 0;
- __entry->id = fq ? fq->debug_id : 0;
+ __entry->id = bq ? bq->priv : 0;
__entry->trace = trace;
),
- TP_printk("R=%08x fq=%x %s",
- __entry->rreq, __entry->id,
- __print_symbolic(__entry->trace, netfs_folioq_traces))
+ TP_printk("bq=%x %s",
+ __entry->id,
+ __print_symbolic(__entry->trace, netfs_bvecq_traces))
+ );
+
+TRACE_EVENT(netfs_bv_slot,
+ TP_PROTO(const struct bvecq *bq, int slot),
+
+ TP_ARGS(bq, slot),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, pfn)
+ __field(unsigned int, offset)
+ __field(unsigned int, len)
+ __field(unsigned int, slot)
+ ),
+
+ TP_fast_assign(
+ __entry->slot = slot;
+ __entry->pfn = page_to_pfn(bq->bv[slot].bv_page);
+ __entry->offset = bq->bv[slot].bv_offset;
+ __entry->len = bq->bv[slot].bv_len;
+ ),
+
+ TP_printk("bq[%x] p=%lx %x-%x",
+ __entry->slot,
+ __entry->pfn, __entry->offset, __entry->offset + __entry->len)
);
#undef EM
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index c4673a54c4baf2..0268314ef60ba2 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6749,8 +6749,6 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {
/* amdgpu */
{ "amdgpu_vm_bo_map", 0x1 },
{ "amdgpu_vm_bo_unmap", 0x1 },
- /* netfs */
- { "netfs_folioq", 0x1 },
/* xfs from xfs_defer_pending_class */
{ "xfs_defer_create_intent", 0x1 },
{ "xfs_defer_cancel_list", 0x1 },
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index c2484551a4e863..b1d253f6986294 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -538,37 +538,38 @@ static void iov_iter_iovec_advance(struct iov_iter *i, size_t size)
i->__iov = iov;
}
-static void iov_iter_folioq_advance(struct iov_iter *i, size_t size)
+static void iov_iter_bvecq_advance(struct iov_iter *i, size_t by)
{
- const struct folio_queue *folioq = i->folioq;
- unsigned int slot = i->folioq_slot;
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
if (!i->count)
return;
- i->count -= size;
-
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- }
+ i->count -= by;
- size += i->iov_offset; /* From beginning of current segment. */
+ by += i->iov_offset; /* From beginning of current segment. */
do {
- size_t fsize = folioq_folio_size(folioq, slot);
+ size_t len;
- if (likely(size < fsize))
- break;
- size -= fsize;
- slot++;
- if (slot >= folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
+ if (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
slot = 0;
+ continue;
}
- } while (size);
- i->iov_offset = size;
- i->folioq_slot = slot;
- i->folioq = folioq;
+ len = bq->bv[slot].bv_len;
+
+ if (likely(by < len))
+ break;
+ by -= len;
+ slot++;
+ } while (by);
+
+ i->iov_offset = by;
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
}
void iov_iter_advance(struct iov_iter *i, size_t size)
@@ -583,38 +584,39 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
iov_iter_iovec_advance(i, size);
} else if (iov_iter_is_bvec(i)) {
iov_iter_bvec_advance(i, size);
- } else if (iov_iter_is_folioq(i)) {
- iov_iter_folioq_advance(i, size);
+ } else if (iov_iter_is_bvecq(i)) {
+ iov_iter_bvecq_advance(i, size);
} else if (iov_iter_is_discard(i)) {
i->count -= size;
}
}
EXPORT_SYMBOL(iov_iter_advance);
-static void iov_iter_folioq_revert(struct iov_iter *i, size_t unroll)
+static void iov_iter_bvecq_revert(struct iov_iter *i, size_t unroll)
{
- const struct folio_queue *folioq = i->folioq;
- unsigned int slot = i->folioq_slot;
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
for (;;) {
- size_t fsize;
+ size_t len;
if (slot == 0) {
- folioq = folioq->prev;
- slot = folioq_nr_slots(folioq);
+ bq = bq->prev;
+ slot = bq->nr_slots;
+ continue;
}
slot--;
- fsize = folioq_folio_size(folioq, slot);
- if (unroll <= fsize) {
- i->iov_offset = fsize - unroll;
+ len = bq->bv[slot].bv_len;
+ if (unroll <= len) {
+ i->iov_offset = len - unroll;
break;
}
- unroll -= fsize;
+ unroll -= len;
}
- i->folioq_slot = slot;
- i->folioq = folioq;
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
}
void iov_iter_revert(struct iov_iter *i, size_t unroll)
@@ -648,9 +650,9 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)
}
unroll -= n;
}
- } else if (iov_iter_is_folioq(i)) {
+ } else if (iov_iter_is_bvecq(i)) {
i->iov_offset = 0;
- iov_iter_folioq_revert(i, unroll);
+ iov_iter_bvecq_revert(i, unroll);
} else { /* same logics for iovec and kvec */
const struct iovec *iov = iter_iov(i);
while (1) {
@@ -678,9 +680,28 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return min(i->count, i->bvec->bv_len - i->iov_offset);
}
- if (unlikely(iov_iter_is_folioq(i)))
- return !i->count ? 0 :
- umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
+ if (!i->count)
+ return 0;
+ if (unlikely(iov_iter_is_bvecq(i))) {
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+ size_t offset = i->iov_offset;
+
+ for (;;) {
+ while (slot >= bq->nr_slots) {
+ bq = bq->next;
+ if (!bq)
+ return 0;
+ slot = 0;
+ offset = 0;
+ }
+ if (bq->bv[slot].bv_len > offset)
+ break;
+ slot++;
+ offset = 0;
+ }
+ return umin(i->count, bq->bv[slot].bv_len - offset);
+ }
return i->count;
}
EXPORT_SYMBOL(iov_iter_single_seg_count);
@@ -718,34 +739,33 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction,
EXPORT_SYMBOL(iov_iter_bvec);
/**
- * iov_iter_folio_queue - Initialise an I/O iterator to use the folios in a folio queue
+ * iov_iter_bvec_queue - Initialise an I/O iterator to use a segmented bvec queue
* @i: The iterator to initialise.
* @direction: The direction of the transfer.
- * @folioq: The starting point in the folio queue.
- * @first_slot: The first slot in the folio queue to use
- * @offset: The offset into the folio in the first slot to start at
+ * @bvecq: The starting point in the bvec queue.
+ * @first_slot: The first slot in the bvec queue to use
+ * @offset: The offset into the bvec in the first slot to start at
* @count: The size of the I/O buffer in bytes.
*
- * Set up an I/O iterator to either draw data out of the pages attached to an
- * inode or to inject data into those pages. The pages *must* be prevented
- * from evaporation, either by taking a ref on them or locking them by the
- * caller.
+ * Set up an I/O iterator to either draw data out of the buffers attached to an
+ * inode or to inject data into those buffers. The pages *must* be prevented
+ * from evaporation, either by the caller.
*/
-void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
- const struct folio_queue *folioq, unsigned int first_slot,
- unsigned int offset, size_t count)
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq, unsigned int first_slot,
+ unsigned int offset, size_t count)
{
- BUG_ON(direction & ~1);
+ WARN_ON(direction & ~(READ | WRITE));
*i = (struct iov_iter) {
- .iter_type = ITER_FOLIOQ,
- .data_source = direction,
- .folioq = folioq,
- .folioq_slot = first_slot,
- .count = count,
- .iov_offset = offset,
+ .iter_type = ITER_BVECQ,
+ .data_source = direction,
+ .bvecq = bvecq,
+ .bvecq_slot = first_slot,
+ .count = count,
+ .iov_offset = offset,
};
}
-EXPORT_SYMBOL(iov_iter_folio_queue);
+EXPORT_SYMBOL(iov_iter_bvec_queue);
/**
* iov_iter_xarray - Initialise an I/O iterator to use the pages in an xarray
@@ -839,6 +859,39 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)
return res;
}
+static unsigned long iov_iter_alignment_bvecq(const struct iov_iter *iter)
+{
+ const struct bvecq *bq;
+ unsigned long res = 0;
+ unsigned int slot = iter->bvecq_slot;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ if (!size)
+ return res;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t part = umin(bvec->bv_len - skip, size);
+
+ if (part) {
+ res |= bvec->bv_offset + skip;
+ res |= part;
+ }
+
+ size -= part;
+ if (size == 0)
+ return res;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+
+ return res;
+}
+
unsigned long iov_iter_alignment(const struct iov_iter *i)
{
if (likely(iter_is_ubuf(i))) {
@@ -854,10 +907,10 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return iov_iter_alignment_bvec(i);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_alignment_bvecq(i);
- /* With both xarray and folioq types, we're dealing with whole folios. */
- if (iov_iter_is_folioq(i))
- return i->iov_offset | i->count;
+ /* With the xarray type, we're dealing with whole folios. */
if (iov_iter_is_xarray(i))
return (i->xarray_start + i->iov_offset) | i->count;
@@ -910,118 +963,34 @@ static int want_pages_array(struct page ***res, size_t size,
return count;
}
-static ssize_t iter_folioq_get_pages(struct iov_iter *iter,
+/*
+ * Wrap iov_iter_extract_pages() and then pin the non-slab pages we got back.
+ * This only works for non-user iterator types as get_pages uses get_user_pages
+ * not pin_user_pages.
+ */
+static ssize_t iter_get_kernel_pages(struct iov_iter *iter,
struct page ***ppages, size_t maxsize,
unsigned maxpages, size_t *_start_offset)
{
- const struct folio_queue *folioq = iter->folioq;
struct page **pages;
- unsigned int slot = iter->folioq_slot;
- size_t extracted = 0, count = iter->count, iov_offset = iter->iov_offset;
+ ssize_t ret, done;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- if (WARN_ON(iov_offset != 0))
- return -EIO;
- }
+ ret = iov_iter_extract_pages(iter, ppages, maxsize, maxpages,
+ 0, _start_offset);
+ if (ret <= 0)
+ return ret;
- maxpages = want_pages_array(ppages, maxsize, iov_offset & ~PAGE_MASK, maxpages);
- if (!maxpages)
- return -ENOMEM;
- *_start_offset = iov_offset & ~PAGE_MASK;
pages = *ppages;
+ for (done = ret + *_start_offset; done > 0; done -= PAGE_SIZE) {
+ struct folio *folio = page_folio(*pages);
- for (;;) {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t offset = iov_offset, fsize = folioq_folio_size(folioq, slot);
- size_t part = PAGE_SIZE - offset % PAGE_SIZE;
-
- if (offset < fsize) {
- part = umin(part, umin(maxsize - extracted, fsize - offset));
- count -= part;
- iov_offset += part;
- extracted += part;
-
- *pages = folio_page(folio, offset / PAGE_SIZE);
- get_page(*pages);
- pages++;
- maxpages--;
- }
-
- if (maxpages == 0 || extracted >= maxsize)
- break;
-
- if (iov_offset >= fsize) {
- iov_offset = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- }
-
- iter->count = count;
- iter->iov_offset = iov_offset;
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- return extracted;
-}
-
-static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
- pgoff_t index, unsigned int nr_pages)
-{
- XA_STATE(xas, xa, index);
- struct folio *folio;
- unsigned int ret = 0;
-
- rcu_read_lock();
- for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
- if (xas_retry(&xas, folio))
- continue;
-
- /* Has the folio moved or been split? */
- if (unlikely(folio != xas_reload(&xas))) {
- xas_reset(&xas);
- continue;
- }
-
- pages[ret] = folio_file_page(folio, xas.xa_index);
- folio_get(folio);
- if (++ret == nr_pages)
- break;
+ if (!folio_test_slab(folio))
+ folio_get(folio);
+ pages++;
}
- rcu_read_unlock();
return ret;
}
-static ssize_t iter_xarray_get_pages(struct iov_iter *i,
- struct page ***pages, size_t maxsize,
- unsigned maxpages, size_t *_start_offset)
-{
- unsigned nr, offset, count;
- pgoff_t index;
- loff_t pos;
-
- pos = i->xarray_start + i->iov_offset;
- index = pos >> PAGE_SHIFT;
- offset = pos & ~PAGE_MASK;
- *_start_offset = offset;
-
- count = want_pages_array(pages, maxsize, offset, maxpages);
- if (!count)
- return -ENOMEM;
- nr = iter_xarray_populate_pages(*pages, i->xarray, index, count);
- if (nr == 0)
- return 0;
-
- maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize);
- i->iov_offset += maxsize;
- i->count -= maxsize;
- return maxsize;
-}
-
/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */
static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
{
@@ -1044,22 +1013,6 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
BUG(); // if it had been empty, we wouldn't get called
}
-/* must be done on non-empty ITER_BVEC one */
-static struct page *first_bvec_segment(const struct iov_iter *i,
- size_t *size, size_t *start)
-{
- struct page *page;
- size_t skip = i->iov_offset, len;
-
- len = i->bvec->bv_len - skip;
- if (*size > len)
- *size = len;
- skip += i->bvec->bv_offset;
- page = i->bvec->bv_page + skip / PAGE_SIZE;
- *start = skip % PAGE_SIZE;
- return page;
-}
-
static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
struct page ***pages, size_t maxsize,
unsigned int maxpages, size_t *start)
@@ -1095,36 +1048,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
iov_iter_advance(i, maxsize);
return maxsize;
}
- if (iov_iter_is_bvec(i)) {
- struct page **p;
- struct page *page;
- page = first_bvec_segment(i, &maxsize, start);
- n = want_pages_array(pages, maxsize, *start, maxpages);
- if (!n)
- return -ENOMEM;
- p = *pages;
- for (int k = 0; k < n; k++) {
- struct folio *folio = page_folio(page + k);
- p[k] = page + k;
- if (!folio_test_slab(folio))
- folio_get(folio);
- }
- maxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start);
- i->count -= maxsize;
- i->iov_offset += maxsize;
- if (i->iov_offset == i->bvec->bv_len) {
- i->iov_offset = 0;
- i->bvec++;
- i->nr_segs--;
- }
- return maxsize;
- }
- if (iov_iter_is_folioq(i))
- return iter_folioq_get_pages(i, pages, maxsize, maxpages, start);
- if (iov_iter_is_xarray(i))
- return iter_xarray_get_pages(i, pages, maxsize, maxpages, start);
- return -EFAULT;
+ if (iov_iter_is_kvec(i))
+ return -EFAULT;
+ return iter_get_kernel_pages(i, pages, maxsize, maxpages, start);
}
ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
@@ -1192,6 +1119,38 @@ static int bvec_npages(const struct iov_iter *i, int maxpages)
return npages;
}
+static size_t iov_npages_bvecq(const struct iov_iter *iter, size_t maxpages)
+{
+ const struct bvecq *bq;
+ unsigned int slot = iter->bvecq_slot;
+ size_t npages = 0;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t offs = (bvec->bv_offset + skip) % PAGE_SIZE;
+ size_t part = umin(bvec->bv_len - skip, size);
+
+ if (part) {
+ npages += DIV_ROUND_UP(offs + part, PAGE_SIZE);
+ if (npages >= maxpages)
+ goto out;
+ }
+
+ size -= part;
+ if (!size)
+ goto out;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+out:
+ return umin(npages, maxpages);
+}
+
int iov_iter_npages(const struct iov_iter *i, int maxpages)
{
if (unlikely(!i->count))
@@ -1206,11 +1165,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages)
return iov_npages(i, maxpages);
if (iov_iter_is_bvec(i))
return bvec_npages(i, maxpages);
- if (iov_iter_is_folioq(i)) {
- unsigned offset = i->iov_offset % PAGE_SIZE;
- int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
- return min(npages, maxpages);
- }
+ if (iov_iter_is_bvecq(i))
+ return iov_npages_bvecq(i, maxpages);
if (iov_iter_is_xarray(i)) {
unsigned offset = (i->xarray_start + i->iov_offset) % PAGE_SIZE;
int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
@@ -1493,64 +1449,143 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)
}
/*
- * Extract a list of contiguous pages from an ITER_FOLIOQ iterator. This does
- * not get references on the pages, nor does it get a pin on them.
+ * Count the number of virtually contiguous pages coming up next in an
+ * ITER_BVECQ iterator, up to the specified maxima.
*/
-static ssize_t iov_iter_extract_folioq_pages(struct iov_iter *i,
- struct page ***pages, size_t maxsize,
- unsigned int maxpages,
- iov_iter_extraction_t extraction_flags,
- size_t *offset0)
+static unsigned int iter_count_bvecq_pages(const struct iov_iter *iter,
+ size_t maxsize,
+ unsigned int maxpages)
+{
+ const struct bvecq *bvecq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t remain = umin(maxsize, iter->count);
+ size_t count = 0, offset = iter->iov_offset;
+
+ do {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(remain > 0);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
+ }
+
+ bv = &bvecq->bv[slot++];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ if (unlikely(!bv->bv_page)) {
+ if (blen && count > 0)
+ break;
+ continue;
+ }
+ if (!PAGE_ALIGNED(boff) && count > 0)
+ break;
+
+ boff += offset;
+ blen -= offset;
+ offset = 0;
+ if (!blen)
+ continue;
+
+ blen = umin(blen, remain);
+ remain -= blen;
+ blen += offset_in_page(boff);
+ count += DIV_ROUND_UP(blen, PAGE_SIZE);
+
+ if (!PAGE_ALIGNED(blen))
+ break;
+ } while (remain > 0 && count < maxpages);
+
+ return umin(count, maxpages);
+}
+
+/*
+ * Extract a list of virtually contiguous pages from an ITER_BVECQ iterator.
+ * This does not get references on the pages, nor does it get a pin on them.
+ */
+static ssize_t iov_iter_extract_bvecq_pages(struct iov_iter *iter,
+ struct page ***pages, size_t maxsize,
+ unsigned int maxpages,
+ iov_iter_extraction_t extraction_flags,
+ size_t *offset0)
{
- const struct folio_queue *folioq = i->folioq;
+ const struct bvecq *bvecq;
struct page **p;
- unsigned int nr = 0;
- size_t extracted = 0, offset, slot = i->folioq_slot;
+ unsigned int slot, nr = 0;
+ size_t extracted = 0, offset;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- if (WARN_ON(i->iov_offset != 0))
- return -EIO;
- }
+ /* Count the next run of virtually contiguous pages. */
+ maxpages = iter_count_bvecq_pages(iter, maxsize, maxpages);
- offset = i->iov_offset & ~PAGE_MASK;
- *offset0 = offset;
+ if (!*pages) {
+ *pages = kvmalloc_array(maxpages, sizeof(struct page *), GFP_KERNEL);
+ if (!*pages)
+ return -ENOMEM;
+ }
- maxpages = want_pages_array(pages, maxsize, offset, maxpages);
- if (!maxpages)
- return -ENOMEM;
p = *pages;
- for (;;) {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t offset = i->iov_offset, fsize = folioq_folio_size(folioq, slot);
- size_t part = PAGE_SIZE - offset % PAGE_SIZE;
-
- if (offset < fsize) {
- part = umin(part, umin(maxsize - extracted, fsize - offset));
- i->count -= part;
- i->iov_offset += part;
- extracted += part;
+ /* Now transcribe the page pointers. */
+ extracted = 0;
+ bvecq = iter->bvecq;
+ offset = iter->iov_offset;
+ slot = iter->bvecq_slot;
- p[nr++] = folio_page(folio, offset / PAGE_SIZE);
+ do {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(extracted < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
}
- if (nr >= maxpages || extracted >= maxsize)
- break;
+ bv = &bvecq->bv[slot];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ if (!bv->bv_page)
+ blen = 0;
- if (i->iov_offset >= fsize) {
- i->iov_offset = 0;
+ if (offset < blen) {
+ size_t part = umin(maxsize - extracted, blen - offset);
+ size_t poff = (boff + offset) % PAGE_SIZE;
+ size_t pix = (boff + offset) / PAGE_SIZE;
+
+ if (poff + part > PAGE_SIZE)
+ part = PAGE_SIZE - poff;
+
+ if (!extracted)
+ *offset0 = poff;
+
+ p[nr++] = bv->bv_page + pix;
+ offset += part;
+ extracted += part;
+ }
+
+ if (offset >= blen) {
+ offset = 0;
slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
}
- }
+ } while (nr < maxpages && extracted < maxsize);
- i->folioq = folioq;
- i->folioq_slot = slot;
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= extracted;
return extracted;
}
@@ -1816,8 +1851,8 @@ static ssize_t iov_iter_extract_user_pages(struct iov_iter *i,
* added to the pages, but refs will not be taken.
* iov_iter_extract_will_pin() will return true.
*
- * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_FOLIOQ or ITER_XARRAY, the
- * pages are merely listed; no extra refs or pins are obtained.
+ * (*) If the iterator is ITER_KVEC, ITER_BVEC, ITER_XARRAY, the pages are
+ * merely listed; no extra refs or pins are obtained.
* iov_iter_extract_will_pin() will return 0.
*
* Note also:
@@ -1852,10 +1887,10 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,
return iov_iter_extract_bvec_pages(i, pages, maxsize,
maxpages, extraction_flags,
offset0);
- if (iov_iter_is_folioq(i))
- return iov_iter_extract_folioq_pages(i, pages, maxsize,
- maxpages, extraction_flags,
- offset0);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_extract_bvecq_pages(i, pages, maxsize,
+ maxpages, extraction_flags,
+ offset0);
if (iov_iter_is_xarray(i))
return iov_iter_extract_xarray_pages(i, pages, maxsize,
maxpages, extraction_flags,
diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index 6ea40d2e624755..b9a7298306d95d 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -10,8 +10,8 @@
#include <linux/highmem.h>
#include <linux/kmemleak.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
-#include <linux/folio_queue.h>
/**
* sg_nents - return total count of entries in scatterlist
@@ -1268,61 +1268,59 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter,
}
/*
- * Extract up to sg_max folios from an FOLIOQ-type iterator and add them to
+ * Extract up to sg_max folios from an BVECQ-type iterator and add them to
* the scatterlist. The pages are not pinned.
*/
-static ssize_t extract_folioq_to_sg(struct iov_iter *iter,
+static ssize_t extract_bvecq_to_sg(struct iov_iter *iter,
ssize_t maxsize,
struct sg_table *sgtable,
unsigned int sg_max,
iov_iter_extraction_t extraction_flags)
{
- const struct folio_queue *folioq = iter->folioq;
+ const struct bvecq *bvecq = iter->bvecq;
struct scatterlist *sg = sgtable->sgl + sgtable->nents;
- unsigned int slot = iter->folioq_slot;
+ unsigned int slot = iter->bvecq_slot;
ssize_t ret = 0;
size_t offset = iter->iov_offset;
- BUG_ON(!folioq);
+ maxsize = umin(maxsize, iov_iter_count(iter));
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- if (WARN_ON_ONCE(!folioq))
- return 0;
- slot = 0;
- }
-
- do {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t fsize = folioq_folio_size(folioq, slot);
+ while (sg_max > 0 && ret < maxsize) {
+ const struct bio_vec *bv;
+ size_t blen, part;
- if (offset < fsize) {
- size_t part = umin(maxsize - ret, fsize - offset);
-
- sg_set_page(sg, folio_page(folio, 0), part, offset);
- sgtable->nents++;
- sg++;
- sg_max--;
- offset += part;
- ret += part;
+ if (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(ret < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ offset = 0;
+ continue;
}
- if (offset >= fsize) {
+ bv = &bvecq->bv[slot];
+ blen = bv->bv_len;
+
+ if (offset >= blen) {
offset = 0;
slot++;
- if (slot >= folioq_nr_slots(folioq)) {
- if (!folioq->next) {
- WARN_ON_ONCE(ret < iter->count);
- break;
- }
- folioq = folioq->next;
- slot = 0;
- }
+ continue;
}
- } while (sg_max > 0 && ret < maxsize);
- iter->folioq = folioq;
- iter->folioq_slot = slot;
+ part = umin(maxsize - ret, blen - offset);
+
+ sg_set_page(sg, bv->bv_page, part, bv->bv_offset + offset);
+ sgtable->nents++;
+ sg++;
+ sg_max--;
+ offset += part;
+ ret += part;
+ }
+
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = slot;
iter->iov_offset = offset;
iter->count -= ret;
return ret;
@@ -1391,8 +1389,8 @@ static ssize_t extract_xarray_to_sg(struct iov_iter *iter,
* addition of @sg_max elements.
*
* The pages referred to by UBUF- and IOVEC-type iterators are extracted and
- * pinned; BVEC-, KVEC-, FOLIOQ- and XARRAY-type are extracted but aren't
- * pinned; DISCARD-type is not supported.
+ * pinned; BVEC-, BVECQ-, KVEC-, XARRAY-type are extracted but aren't pinned;
+ * DISCARD-type is not supported.
*
* No end mark is placed on the scatterlist; that's left to the caller.
*
@@ -1424,9 +1422,9 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize,
case ITER_KVEC:
return extract_kvec_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
- case ITER_FOLIOQ:
- return extract_folioq_to_sg(iter, maxsize, sgtable, sg_max,
- extraction_flags);
+ case ITER_BVECQ:
+ return extract_bvecq_to_sg(iter, maxsize, sgtable, sg_max,
+ extraction_flags);
case ITER_XARRAY:
return extract_xarray_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
diff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c
index d9690ba1db887c..1c708523400684 100644
--- a/lib/tests/kunit_iov_iter.c
+++ b/lib/tests/kunit_iov_iter.c
@@ -12,7 +12,7 @@
#include <linux/mm.h>
#include <linux/uio.h>
#include <linux/bvec.h>
-#include <linux/folio_queue.h>
+#include <linux/bvecq.h>
#include <linux/scatterlist.h>
#include <linux/minmax.h>
#include <linux/mman.h>
@@ -382,58 +382,67 @@ static void __init iov_kunit_copy_from_bvec(struct kunit *test)
KUNIT_SUCCEED(test);
}
-static void iov_kunit_destroy_folioq(void *data)
+static void iov_kunit_destroy_bvecq(void *data)
{
- struct folio_queue *folioq, *next;
+ struct bvecq *bq, *next;
- for (folioq = data; folioq; folioq = next) {
- next = folioq->next;
- kfree(folioq);
+ for (bq = data; bq; bq = next) {
+ next = bq->next;
+ /* The pages are freed by vmap with VM_MAP_PUT_PAGES. */
+ kfree(bq);
}
}
-static void __init iov_kunit_load_folioq(struct kunit *test,
+static struct bvecq *iov_kunit_alloc_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = kzalloc(struct_size(bq, __bv, max_slots), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bq);
+ bq->max_slots = max_slots;
+ bq->bv = bq->__bv;
+ bq->inline_bv = true;
+ return bq;
+}
+
+static struct bvecq *iov_kunit_create_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = iov_kunit_alloc_bvecq(test, max_slots);
+ kunit_add_action_or_reset(test, iov_kunit_destroy_bvecq, bq);
+ return bq;
+}
+
+static void __init iov_kunit_load_bvecq(struct kunit *test,
struct iov_iter *iter, int dir,
- struct folio_queue *folioq,
+ struct bvecq *bq_head,
struct page **pages, size_t npages)
{
- struct folio_queue *p = folioq;
+ struct bvecq *bq = bq_head;
size_t size = 0;
- int i;
- for (i = 0; i < npages; i++) {
- if (folioq_full(p)) {
- p->next = kzalloc_obj(struct folio_queue);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, p->next);
- folioq_init(p->next, 0);
- p->next->prev = p;
- p = p->next;
+ for (int i = 0; i < npages; i++) {
+ if (bq->nr_slots >= bq->max_slots) {
+ bq->next = iov_kunit_alloc_bvecq(test, 13);
+ bq->next->prev = bq;
+ bq = bq->next;
}
- folioq_append(p, page_folio(pages[i]));
+ bvec_set_page(&bq->bv[bq->nr_slots], pages[i], PAGE_SIZE, 0);
+ bq->nr_slots++;
size += PAGE_SIZE;
}
- iov_iter_folio_queue(iter, dir, folioq, 0, 0, size);
-}
-
-static struct folio_queue *iov_kunit_create_folioq(struct kunit *test)
-{
- struct folio_queue *folioq;
-
- folioq = kzalloc_obj(struct folio_queue);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folioq);
- kunit_add_action_or_reset(test, iov_kunit_destroy_folioq, folioq);
- folioq_init(folioq, 0);
- return folioq;
+ iov_iter_bvec_queue(iter, dir, bq_head, 0, 0, size);
}
/*
- * Test copying to a ITER_FOLIOQ-type iterator.
+ * Test copying to a ITER_BVECQ-type iterator.
*/
-static void __init iov_kunit_copy_to_folioq(struct kunit *test)
+static void __init iov_kunit_copy_to_bvecq(struct kunit *test)
{
const struct kvec_test_range *pr;
struct iov_iter iter;
- struct folio_queue *folioq;
+ struct bvecq *bq;
struct page **spages, **bpages;
u8 *scratch, *buffer;
size_t bufsize, npages, size, copied;
@@ -442,7 +451,7 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)
bufsize = 0x100000;
npages = bufsize / PAGE_SIZE;
- folioq = iov_kunit_create_folioq(test);
+ bq = iov_kunit_create_bvecq(test, 13);
scratch = iov_kunit_create_buffer(test, &spages, npages);
for (i = 0; i < bufsize; i++)
@@ -451,20 +460,19 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)
buffer = iov_kunit_create_buffer(test, &bpages, npages);
memset(buffer, 0, bufsize);
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
i = 0;
for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
size = pr->to - pr->from;
KUNIT_ASSERT_LE(test, pr->to, bufsize);
- iov_iter_folio_queue(&iter, READ, folioq, 0, 0, pr->to);
+ iov_iter_bvec_queue(&iter, READ, bq, 0, 0, pr->to);
iov_iter_advance(&iter, pr->from);
copied = copy_to_iter(scratch + i, size, &iter);
KUNIT_EXPECT_EQ(test, copied, size);
KUNIT_EXPECT_EQ(test, iter.count, 0);
- KUNIT_EXPECT_EQ(test, iter.iov_offset, pr->to % PAGE_SIZE);
i += size;
if (test->status == KUNIT_FAILURE)
goto stop;
@@ -489,13 +497,13 @@ static void __init iov_kunit_copy_to_folioq(struct kunit *test)
}
/*
- * Test copying from a ITER_FOLIOQ-type iterator.
+ * Test copying from a ITER_BVECQ-type iterator.
*/
-static void __init iov_kunit_copy_from_folioq(struct kunit *test)
+static void __init iov_kunit_copy_from_bvecq(struct kunit *test)
{
const struct kvec_test_range *pr;
struct iov_iter iter;
- struct folio_queue *folioq;
+ struct bvecq *bq;
struct page **spages, **bpages;
u8 *scratch, *buffer;
size_t bufsize, npages, size, copied;
@@ -504,7 +512,7 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)
bufsize = 0x100000;
npages = bufsize / PAGE_SIZE;
- folioq = iov_kunit_create_folioq(test);
+ bq = iov_kunit_create_bvecq(test, 13);
buffer = iov_kunit_create_buffer(test, &bpages, npages);
for (i = 0; i < bufsize; i++)
@@ -513,20 +521,19 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)
scratch = iov_kunit_create_buffer(test, &spages, npages);
memset(scratch, 0, bufsize);
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
i = 0;
for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
size = pr->to - pr->from;
KUNIT_ASSERT_LE(test, pr->to, bufsize);
- iov_iter_folio_queue(&iter, WRITE, folioq, 0, 0, pr->to);
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
iov_iter_advance(&iter, pr->from);
copied = copy_from_iter(scratch + i, size, &iter);
KUNIT_EXPECT_EQ(test, copied, size);
KUNIT_EXPECT_EQ(test, iter.count, 0);
- KUNIT_EXPECT_EQ(test, iter.iov_offset, pr->to % PAGE_SIZE);
i += size;
}
@@ -868,13 +875,13 @@ static void __init iov_kunit_extract_pages_bvec(struct kunit *test)
}
/*
- * Test the extraction of ITER_FOLIOQ-type iterators.
+ * Test the extraction of ITER_BVECQ-type iterators.
*/
-static void __init iov_kunit_extract_pages_folioq(struct kunit *test)
+static void __init iov_kunit_extract_pages_bvecq(struct kunit *test)
{
const struct kvec_test_range *pr;
- struct folio_queue *folioq;
struct iov_iter iter;
+ struct bvecq *bq;
struct page **bpages, *pagelist[8], **pages = pagelist;
ssize_t len;
size_t bufsize, size = 0, npages;
@@ -883,17 +890,17 @@ static void __init iov_kunit_extract_pages_folioq(struct kunit *test)
bufsize = 0x100000;
npages = bufsize / PAGE_SIZE;
- folioq = iov_kunit_create_folioq(test);
+ bq = iov_kunit_create_bvecq(test, 13);
iov_kunit_create_buffer(test, &bpages, npages);
- iov_kunit_load_folioq(test, &iter, READ, folioq, bpages, npages);
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
from = pr->from;
size = pr->to - from;
KUNIT_ASSERT_LE(test, pr->to, bufsize);
- iov_iter_folio_queue(&iter, WRITE, folioq, 0, 0, pr->to);
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
iov_iter_advance(&iter, from);
do {
@@ -1173,23 +1180,6 @@ static void __init iov_kunit_iter_to_sg_bvec(struct kunit *test)
iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data);
}
-static void __init iov_kunit_iter_to_sg_folioq(struct kunit *test)
-{
- struct iov_kunit_iter_to_sg_data data;
- struct folio_queue *folioq;
- struct iov_iter iter;
- size_t bufsize;
-
- bufsize = 0x200000;
- iov_kunit_iter_to_sg_init(test, bufsize, false, &data);
-
- folioq = iov_kunit_create_folioq(test);
- iov_kunit_load_folioq(test, &iter, READ, folioq, data.pages,
- data.npages);
-
- iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data);
-}
-
static void __init iov_kunit_iter_to_sg_xarray(struct kunit *test)
{
struct iov_kunit_iter_to_sg_data data;
@@ -1226,17 +1216,16 @@ static struct kunit_case __refdata iov_kunit_cases[] = {
KUNIT_CASE(iov_kunit_copy_from_kvec),
KUNIT_CASE(iov_kunit_copy_to_bvec),
KUNIT_CASE(iov_kunit_copy_from_bvec),
- KUNIT_CASE(iov_kunit_copy_to_folioq),
- KUNIT_CASE(iov_kunit_copy_from_folioq),
+ KUNIT_CASE(iov_kunit_copy_to_bvecq),
+ KUNIT_CASE(iov_kunit_copy_from_bvecq),
KUNIT_CASE(iov_kunit_copy_to_xarray),
KUNIT_CASE(iov_kunit_copy_from_xarray),
KUNIT_CASE(iov_kunit_extract_pages_kvec),
KUNIT_CASE(iov_kunit_extract_pages_bvec),
- KUNIT_CASE(iov_kunit_extract_pages_folioq),
+ KUNIT_CASE(iov_kunit_extract_pages_bvecq),
KUNIT_CASE(iov_kunit_extract_pages_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_kvec),
KUNIT_CASE(iov_kunit_iter_to_sg_bvec),
- KUNIT_CASE(iov_kunit_iter_to_sg_folioq),
KUNIT_CASE(iov_kunit_iter_to_sg_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_ubuf),
{}
diff --git a/mm/readahead.c b/mm/readahead.c
index 558c92957518b7..069ded56fd8020 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -188,6 +188,7 @@ static void read_pages(struct readahead_control *rac)
if (unlikely(rac->_workingset))
psi_memstall_leave(&rac->_pflags);
rac->_workingset = false;
+ rac->_nr_folios = 0;
BUG_ON(readahead_count(rac));
}
@@ -303,6 +304,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,
if (i == mark)
folio_set_readahead(folio);
ractl->_workingset |= folio_test_workingset(folio);
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
i += min_nrpages;
}
@@ -473,6 +475,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,
return err;
}
+ ractl->_nr_folios++;
ractl->_nr_pages += 1UL << order;
ractl->_workingset |= folio_test_workingset(folio);
return 0;
@@ -822,6 +825,7 @@ void readahead_expand(struct readahead_control *ractl,
ractl->_workingset = true;
psi_memstall_enter(&ractl->_pflags);
}
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
ractl->_index = folio->index;
}
@@ -851,6 +855,7 @@ void readahead_expand(struct readahead_control *ractl,
ractl->_workingset = true;
psi_memstall_enter(&ractl->_pflags);
}
+ ractl->_nr_folios++;
ractl->_nr_pages += min_nrpages;
if (ra) {
ra->size += min_nrpages;
diff --git a/net/9p/client.c b/net/9p/client.c
index ef64546c6d52a1..68d82f677a7a74 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -1564,6 +1564,7 @@ void
p9_client_write_subreq(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *wreq = subreq->rreq;
+ struct iov_iter iter;
struct p9_fid *fid = wreq->netfs_priv;
struct p9_client *clnt = fid->clnt;
struct p9_req_t *req;
@@ -1574,14 +1575,17 @@ p9_client_write_subreq(struct netfs_io_subrequest *subreq)
p9_debug(P9_DEBUG_9P, ">>> TWRITE fid %d offset %llu len %d\n",
fid->fid, start, len);
+ iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq,
+ subreq->content.slot, subreq->content.offset, subreq->len);
+
/* Don't bother zerocopy for small IO (< 1024) */
if (clnt->trans_mod->zc_request && len > 1024) {
- req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &subreq->io_iter,
+ req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &iter,
0, wreq->len, P9_ZC_HDR_SZ, "dqd",
fid->fid, start, len);
} else {
req = p9_client_rpc(clnt, P9_TWRITE, "dqV", fid->fid,
- start, len, &subreq->io_iter);
+ start, len, &iter);
}
if (IS_ERR(req)) {
netfs_write_subrequest_terminated(subreq, PTR_ERR(req));
|
| 3/2 | 2026/07/23 14:19 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 190036 output: 240 thoughts: 1651 Thoughts: **Fuzzing Target Assessment: Major I/O Buffer Restructuring** Alright, let's break this down. My initial assessment is that this patch series introduces significant functional changes, and it's definitely a candidate for fuzzing. The magnitude of this refactoring, particularly the replacement of `folio_queue` with `bvecq` across several filesystems, demands thorough scrutiny. This isn't just a minor tweak; it's a fundamental shift in how these filesystems handle I/O buffers. Looking at the diff, the changes are widespread. We're talking about modifications to several critical filesystems: 9p, AFS, Ceph, NFS, SMB, and the core `netfs` infrastructure. The deletion of `Documentation/core-api/folio_queue.rst` and removal of `include/linux/folio_queue.h` are clear indicators of this major shift. The documentation updates also suggest substantial changes to the API or behavior. The removal of `CACHEFILES_ONDEMAND` support also suggests a change in overall behavior of the filesystem. Functionally, this impacts core data structures used for I/O operations in several filesystems. This is a potential risk area. So, where do we start with the fuzzing? I need to extract key components for targeted fuzzing. I've got to find specific functions, the `FocusSymbols`, that are newly introduced or significantly altered in these filesystems. I'm focusing on those functions involved in `bvecq` allocation, manipulation, and interaction with the filesystem layer, as well as the related helper functions in `netfs`. I need a good list: * `bvecq_alloc_one`, `bvecq_alloc_chain`, `bvecq_alloc_buffer2`, `bvecq_put`, `bvecq_expand_buffer`, `bvecq_shorten_buffer`, `bvecq_buffer_init`, `bvecq_buffer_append`, `bvecq_pos_advance`, `bvecq_zero`, `bvecq_slice`, `bvecq_extract`, `bvecq_load_from_ra` * `netfs_prepare_read_buffer`, `netfs_prepare_write_buffer` * `netfs_extract_iter` I also need to determine the essential kernel configuration options, the `EnableConfigs`, required to build and test this feature. I'm assuming that to properly enable and test these changes, I'll need: * `CONFIG_NETFS_SUPPORT` * `CONFIG_FSCACHE` * `CONFIG_CACHEFILES` * `CONFIG_9P_FS` * `CONFIG_AFS_FS` * `CONFIG_CEPH_FS` * `CONFIG_NFS_FS` * `CONFIG_CIFS` Let's make sure that these configs make sense. Given that `NETFS_SUPPORT` is selected by all the filesystems being changed: `9P_FS`, `AFS_FS`, `CEPH_FS`, `CIFS`, and `NFS_FS`, enabling `NETFS_SUPPORT` makes sense. Similarly, `FSCACHE` is selected by `CACHEFILES`, `NFS_FSCACHE`, `CEPH_FSCACHE`, `CIFS_FSCACHE`, `9P_FSCACHE`, and `AFS_FSCACHE`, so that looks good, too. I'll pick the following focus symbols based on their complexity, criticality, and the likelihood of introducing subtle bugs: * `bvecq_alloc_chain` * `bvecq_expand_buffer` * `bvecq_shorten_buffer` * `bvecq_pos_advance` * `bvecq_zero` * `bvecq_slice` * `bvecq_extract` * `bvecq_load_from_ra` * `netfs_prepare_read_buffer` * `netfs_prepare_write_buffer` * `netfs_extract_iter` Now, let's verify my assumptions. Let's make sure that those symbols are where they're supposed to be. I've verified that `bvecq_alloc_chain` is in `fs/netfs/bvecq.c`. I've also verified that `netfs_extract_iter` is in `fs/netfs/iterator.c`. So, I'm confident about my initial assessment. With these configurations and the list of functions in hand, I'm ready to move forward. Let's feed this information into the system using `set-results`. |
| 4/2 | 2026/07/23 14:20 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"NETFS_SUPPORT",
"FSCACHE",
"CACHEFILES"
],
"FocusSymbols": [
"bvecq_alloc_chain",
"bvecq_expand_buffer",
"bvecq_shorten_buffer",
"bvecq_pos_advance",
"bvecq_zero",
"bvecq_slice",
"bvecq_extract",
"bvecq_load_from_ra",
"netfs_prepare_read_buffer",
"netfs_prepare_write_buffer",
"netfs_extract_iter"
],
"Reasoning": "The patch replaces the `folio_queue` data structure with `bvecq` in the netfs library and updates all dependent filesystems (9p, afs, ceph, nfs, smb) to use it. It also removes the on-demand read mode from cachefiles. This is a major functional change to the core I/O buffering logic in netfs and should be heavily fuzzed.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"NETFS_SUPPORT",
"FSCACHE",
"CACHEFILES"
],
"FocusSymbols": [
"bvecq_alloc_chain",
"bvecq_expand_buffer",
"bvecq_shorten_buffer",
"bvecq_pos_advance",
"bvecq_zero",
"bvecq_slice",
"bvecq_extract",
"bvecq_load_from_ra",
"netfs_prepare_read_buffer",
"netfs_prepare_write_buffer",
"netfs_extract_iter"
],
"Reasoning": "The patch replaces the `folio_queue` data structure with `bvecq` in the netfs library and updates all dependent filesystems (9p, afs, ceph, nfs, smb) to use it. It also removes the on-demand read mode from cachefiles. This is a major functional change to the core I/O buffering logic in netfs and should be heavily fuzzed.",
"WorthFuzzing": true
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|