AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/02 14:37	flow	patch-triage	0m Args: null Results: { "EnableConfigs": [ "DMA_SHARED_BUFFER", "IO_URING", "BLK_DEV_NVME" ], "FocusSymbols": [ "dma_buf_io_ctx_create", "dma_buf_io_ctx_release", "dma_buf_io_create_map", "dma_buf_io_invalidate_mappings", "dma_buf_io_init_map", "bio_iov_iter_set", "__io_import_reg_buf", "io_drop_dmabuf_node", "iov_iter_dmabuf_map" ], "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.", "WorthFuzzing": true }
1/1	2026/08/02 14:37	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit 2cea66d19f5e61596d4d0326aee5752566f75bc7\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sun Aug 2 14:37:35 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/block/biovecs.rst b/Documentation/block/biovecs.rst\nindex 11126ed6f40f7..49da2147b7d86 100644\n--- a/Documentation/block/biovecs.rst\n+++ b/Documentation/block/biovecs.rst\n@@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec.\n In the new scheme of things, everything that must be mutated in order to\n partially complete a bio is segregated into struct bvec_iter: bi_sector,\n bi_size and bi_idx have been moved there; and instead of modifying bv_offset\n-and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of\n+and bv_len, struct bvec_iter has bi_offset, which represents the number of\n bytes completed in the current bvec.\n \n There are a bunch of new helper macros for hiding the gory details - in\n particular, presenting the illusion of partially completed biovecs so that\n-normal code doesn't have to deal with bi_bvec_done.\n+normal code doesn't have to deal with bi_offset.\n \n * Driver code should no longer refer to biovecs directly; we now have\n bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs,\n- constructed from the raw biovecs but taking into account bi_bvec_done and\n+ constructed from the raw biovecs but taking into account bi_offset and\n bi_size.\n \n bio_for_each_segment() has been updated to take a bvec_iter argument\n@@ -101,7 +101,7 @@ Other implications:\n I.e. instead of using bio_iovec_idx() (or bio-\u003ebi_iovec[bio-\u003ebi_idx]), you\n now use bio_iter_iovec(), which takes a bvec_iter and returns a\n literal struct bio_vec - constructed on the fly from the raw biovec but\n- taking into account bi_bvec_done (and bi_size).\n+ taking into account bi_offset (and bi_size).\n \n * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that\n doesn't actually own the bio. The reason is twofold: firstly, it's not\ndiff --git a/block/bio.c b/block/bio.c\nindex 500389f332d99..1602eab05761f 100644\n--- a/block/bio.c\n+++ b/block/bio.c\n@@ -229,7 +229,7 @@ void bio_init(struct bio bio, struct block_device bdev, struct bio_vec table,\n \tbio-\u003ebi_iter.bi_sector = 0;\n \tbio-\u003ebi_iter.bi_size = 0;\n \tbio-\u003ebi_iter.bi_idx = 0;\n-\tbio-\u003ebi_iter.bi_bvec_done = 0;\n+\tbio-\u003ebi_iter.bi_offset = 0;\n \tbio-\u003ebi_end_io = NULL;\n \tbio-\u003ebi_private = NULL;\n #ifdef CONFIG_BLK_CGROUP\n@@ -861,6 +861,11 @@ static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp)\n \tbio-\u003ebi_write_stream = bio_src-\u003ebi_write_stream;\n \tbio-\u003ebi_iter = bio_src-\u003ebi_iter;\n \n+\tif (op_is_dmabuf(bio-\u003ebi_opf))\n+\t\tbio-\u003ebi_dmabuf_map = bio_src-\u003ebi_dmabuf_map;\n+\telse\n+\t\tbio-\u003ebi_io_vec = bio_src-\u003ebi_io_vec;\n+\n \tif (bio-\u003ebi_bdev) {\n \t\tif (bio-\u003ebi_bdev == bio_src-\u003ebi_bdev \u0026\u0026\n \t\t bio_flagged(bio_src, BIO_REMAPPED))\n@@ -902,8 +907,6 @@ struct bio bio_alloc_clone(struct block_device bdev, struct bio bio_src,\n \t\tbio_put(bio);\n \t\treturn NULL;\n \t}\n-\tbio-\u003ebi_io_vec = bio_src-\u003ebi_io_vec;\n-\n \treturn bio;\n }\n EXPORT_SYMBOL(bio_alloc_clone);\n@@ -923,7 +926,7 @@ int bio_init_clone(struct block_device bdev, struct bio bio,\n {\n \tint ret;\n \n-\tbio_init(bio, bdev, bio_src-\u003ebi_io_vec, 0, bio_src-\u003ebi_opf);\n+\tbio_init(bio, bdev, NULL, 0, bio_src-\u003ebi_opf);\n \tret = __bio_clone(bio, bio_src, gfp);\n \tif (ret)\n \t\tbio_uninit(bio);\n@@ -1182,15 +1185,26 @@ void __bio_release_pages(struct bio bio, bool mark_dirty)\n }\n EXPORT_SYMBOL_GPL(__bio_release_pages);\n \n-void bio_iov_bvec_set(struct bio bio, const struct iov_iter iter)\n+bool bio_iov_iter_set(struct bio bio, const struct iov_iter iter)\n {\n+\tif (!iov_iter_is_bvec(iter) \u0026\u0026 !iov_iter_is_dmabuf_map(iter))\n+\t\treturn false;\n+\n \tWARN_ON_ONCE(bio-\u003ebi_max_vecs);\n \n+\tstatic_assert(offsetof(struct bio, bi_io_vec) ==\n+\t\t offsetof(struct bio, bi_dmabuf_map));\n+\tstatic_assert(offsetof(struct iov_iter, bvec) ==\n+\t\t offsetof(struct iov_iter, dmabuf_map));\n+\n \tbio-\u003ebi_io_vec = (struct bio_vec )iter-\u003ebvec;\n \tbio-\u003ebi_iter.bi_idx = 0;\n-\tbio-\u003ebi_iter.bi_bvec_done = iter-\u003eiov_offset;\n+\tbio-\u003ebi_iter.bi_offset = iter-\u003eiov_offset;\n \tbio-\u003ebi_iter.bi_size = iov_iter_count(iter);\n \tbio_set_flag(bio, BIO_CLONED);\n+\tif (iov_iter_is_dmabuf_map(iter))\n+\t\tbio-\u003ebi_opf \|= REQ_NOMERGE \| REQ_DMABUF;\n+\treturn true;\n }\n \n /\n@@ -1285,10 +1299,9 @@ int bio_iov_iter_get_pages(struct bio bio, struct iov_iter iter,\n \tif (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))\n \t\treturn -EIO;\n \n-\tif (iov_iter_is_bvec(iter)) {\n-\t\tbio_iov_bvec_set(bio, iter);\n-\n-\t\tif (!bio_iov_bvec_aligned(bio, mem_align_mask))\n+\tif (bio_iov_iter_set(bio, iter)) {\n+\t\tif (iov_iter_is_bvec(iter) \u0026\u0026\n+\t\t !bio_iov_bvec_aligned(bio, mem_align_mask))\n \t\t\treturn -EINVAL;\n \n \t\tiov_iter_advance(iter, bio-\u003ebi_iter.bi_size);\ndiff --git a/block/blk-map.c b/block/blk-map.c\nindex 615d29bb840eb..9cb9605d1f62d 100644\n--- a/block/blk-map.c\n+++ b/block/blk-map.c\n@@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request rq, const struct iov_iter iter)\n \tbio = blk_rq_map_bio_alloc(rq, 0, GFP_KERNEL);\n \tif (!bio)\n \t\treturn -ENOMEM;\n-\tbio_iov_bvec_set(bio, iter);\n+\tbio_iov_iter_set(bio, iter);\n \n \tret = blk_rq_append_bio(rq, bio);\n \tif (ret)\ndiff --git a/block/blk-merge.c b/block/blk-merge.c\nindex ab1161ca69f1e..1beedc42a85d5 100644\n--- a/block/blk-merge.c\n+++ b/block/blk-merge.c\n@@ -9,6 +9,7 @@\n #include \u003clinux/blk-integrity.h\u003e\n #include \u003clinux/part_stat.h\u003e\n #include \u003clinux/blk-cgroup.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n \n #include \u003ctrace/events/block.h\u003e\n \n@@ -33,7 +34,7 @@ static inline void bio_get_last_bvec(struct bio bio, struct bio_vec bv)\n \n \tbio_advance_iter(bio, \u0026iter, iter.bi_size);\n \n-\tif (!iter.bi_bvec_done)\n+\tif (!iter.bi_offset)\n \t\tidx = iter.bi_idx - 1;\n \telse\t/ in the middle of bvec /\n \t\tidx = iter.bi_idx;\n@@ -41,11 +42,11 @@ static inline void bio_get_last_bvec(struct bio bio, struct bio_vec bv)\n \tbv = bio-\u003ebi_io_vec[idx];\n \n \t/\n-\t iter.bi_bvec_done records actual length of the last bvec\n+\t * iter.bi_offset records actual length of the last bvec\n \t * if this bio ends in the middle of one io vector\n \t /\n-\tif (iter.bi_bvec_done)\n-\t\tbv-\u003ebv_len = iter.bi_bvec_done;\n+\tif (iter.bi_offset)\n+\t\tbv-\u003ebv_len = iter.bi_offset;\n }\n \n static inline bool bio_will_gap(struct request_queue q,\n@@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec bvprv,\n \treturn bv-\u003ebv_offset \| (bvprv-\u003ebv_offset + bvprv-\u003ebv_len);\n }\n \n+static inline int bio_split_io_at_dmabuf(struct bio bio,\n+\t\tconst struct queue_limits lim, unsigned segs,\n+\t\tunsigned max_bytes, unsigned len_align_mask,\n+\t\tunsigned start_align_mask)\n+{\n+\tunsigned bytes = min(bio-\u003ebi_iter.bi_size, max_bytes);\n+\tunsigned seg_shift = bio-\u003ebi_dmabuf_map-\u003eseg_shift;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset \u0026 ((1U \u003c\u003c seg_shift) - 1);\n+\n+\tif ((bio-\u003ebi_iter.bi_offset \u0026 start_align_mask) \|\|\n+\t (bio-\u003ebi_iter.bi_size \u0026 len_align_mask))\n+\t\treturn -EINVAL;\n+\n+\t/* single contiguous range into the dma-buf /\n+\tsegs = 1;\n+\n+\tbytes = min(bytes, ((unsigned)lim-\u003emax_segments \u003c\u003c seg_shift) - offset);\n+\tif (bytes != bio-\u003ebi_iter.bi_size)\n+\t\treturn bytes;\n+\treturn 0;\n+}\n+\n /*\n bio_split_io_at - check if and where to split a bio\n * @bio: [in] bio to be split\n@@ -346,6 +369,19 @@ int bio_split_io_at(struct bio bio, const struct queue_limits lim,\n \t\tlen_align_mask \|= (bc-\u003ebc_key-\u003ecrypto_cfg.data_unit_size - 1);\n \t}\n \n+\tif (op_is_dmabuf(bio-\u003ebi_opf)) {\n+\t\tint ret;\n+\n+\t\tret = bio_split_io_at_dmabuf(bio, lim, \u0026nsegs, max_bytes,\n+\t\t\t\t\t len_align_mask, start_align_mask);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t\tif (!ret)\n+\t\t\tgoto out;\n+\t\tbytes = ret;\n+\t\tgoto split;\n+\t}\n+\n \tbio_for_each_bvec(bv, bio, iter) {\n \t\tif (bv.bv_offset \u0026 start_align_mask \|\|\n \t\t bv.bv_len \u0026 len_align_mask)\n@@ -376,6 +412,7 @@ int bio_split_io_at(struct bio bio, const struct queue_limits lim,\n \t\tbvprvp = \u0026bvprv;\n \t}\n \n+out:\n \tsegs = nsegs;\n \tbio-\u003ebi_bvec_gap_bit = ffs(gaps);\n \treturn 0;\ndiff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c\nindex bfdb9ed707411..88fd9cbc951f4 100644\n--- a/block/blk-mq-dma.c\n+++ b/block/blk-mq-dma.c\n@@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request req, struct blk_map_iter iter,\n \t one could be merged into it. This typically happens when moving to\n \t * the next bio, but some callers also don't pack bvecs tight.\n \t /\n-\twhile (!iter-\u003eiter.bi_size \|\| !iter-\u003eiter.bi_bvec_done) {\n+\twhile (!iter-\u003eiter.bi_size \|\| !iter-\u003eiter.bi_offset) {\n \t\tstruct bio_vec next;\n \n \t\tif (!__blk_map_iter_next(iter))\ndiff --git a/block/blk.h b/block/blk.h\nindex eaac05815cb03..50abfd9328861 100644\n--- a/block/blk.h\n+++ b/block/blk.h\n@@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio bio,\n \t\treturn true;\n \n \tbv = __bvec_iter_bvec(bio-\u003ebi_io_vec, bio-\u003ebi_iter);\n-\tif (bio-\u003ebi_iter.bi_size \u003e bv-\u003ebv_len - bio-\u003ebi_iter.bi_bvec_done)\n+\tif (bio-\u003ebi_iter.bi_size \u003e bv-\u003ebv_len - bio-\u003ebi_iter.bi_offset)\n \t\treturn true;\n \tif ((bv-\u003ebv_offset \| bv-\u003ebv_len) \u0026 lim-\u003edma_alignment)\n \t\treturn true;\ndiff --git a/block/fops.c b/block/fops.c\nindex 3c2099dfef1d5..3c403afb0f646 100644\n--- a/block/fops.c\n+++ b/block/fops.c\n@@ -342,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb iocb,\n \tbio-\u003ebi_end_io = blkdev_bio_end_io_async;\n \tbio-\u003ebi_ioprio = iocb-\u003eki_ioprio;\n \n-\tif (iov_iter_is_bvec(iter)) {\n-\t\t/\n-\t\t * Users don't rely on the iterator being in any particular\n-\t\t * state for async I/O returning -EIOCBQUEUED, hence we can\n-\t\t * avoid expensive iov_iter_advance(). Bypass\n-\t\t * bio_iov_iter_get_pages() and set the bvec directly.\n-\t\t /\n-\t\tbio_iov_bvec_set(bio, iter);\n-\t} else {\n+\t/\n+\t * Users don't rely on the iterator being in any particular\n+\t * state for async I/O returning -EIOCBQUEUED, hence we can\n+\t * avoid expensive iov_iter_advance(). Bypass\n+\t * bio_iov_iter_get_pages() and set the bvec/dmabuf directly.\n+\t /\n+\tif (!bio_iov_iter_set(bio, iter)) {\n \t\tret = blkdev_iov_iter_get_pages(bio, iter, bdev);\n \t\tif (unlikely(ret))\n \t\t\tgoto out_bio_put;\n@@ -769,6 +767,9 @@ static ssize_t blkdev_write_iter(struct kiocb iocb, struct iov_iter from)\n \t\t\tret = direct_write_fallback(iocb, from, ret,\n \t\t\t\t\tblkdev_buffered_write(iocb, from));\n \t} else {\n+\t\tif (unlikely(iov_iter_is_dmabuf_map(from)))\n+\t\t\treturn -EOPNOTSUPP;\n+\n \t\t/\n \t\t * Take i_rwsem and invalidate_lock to avoid racing with\n \t\t * set_blocksize changing i_blkbits/folio order and punching\n@@ -823,6 +824,8 @@ static ssize_t blkdev_read_iter(struct kiocb iocb, struct iov_iter to)\n \t\tif (ret \u003c 0 \|\| !count)\n \t\t\tgoto reexpand;\n \t}\n+\tif (unlikely(iov_iter_is_dmabuf_map(to)))\n+\t\treturn -EOPNOTSUPP;\n \n \t/\n \t Take i_rwsem and invalidate_lock to avoid racing with set_blocksize\n@@ -926,6 +929,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc desc)\n \treturn generic_file_mmap_prepare(desc);\n }\n \n+static int blkdev_init_dma_buf_io_ctx(struct file file,\n+\t\t\t\t struct dma_buf_io_ctx ctx)\n+{\n+\tstruct block_device bdev = file_bdev(file);\n+\tstruct gendisk disk = bdev-\u003ebd_disk;\n+\n+\tif (!(file-\u003ef_flags \u0026 O_DIRECT))\n+\t\treturn -EINVAL;\n+\tif (!disk-\u003efops-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EINVAL;\n+\treturn disk-\u003efops-\u003einit_dma_buf_io_ctx(bdev, ctx);\n+}\n+\n const struct file_operations def_blk_fops = {\n \t.open\t\t= blkdev_open,\n \t.release\t= blkdev_release,\n@@ -944,6 +960,7 @@ const struct file_operations def_blk_fops = {\n \t.fallocate\t= blkdev_fallocate,\n \t.uring_cmd\t= blkdev_uring_cmd,\n \t.fop_flags\t= FOP_BUFFER_RASYNC \| FOP_DONTCACHE,\n+\t.init_dma_buf_io_ctx = blkdev_init_dma_buf_io_ctx,\n };\n \n static __init int blkdev_init(void)\ndiff --git a/drivers/block/loop.c b/drivers/block/loop.c\nindex 26d7130c3f559..8639fa34b8470 100644\n--- a/drivers/block/loop.c\n+++ b/drivers/block/loop.c\n@@ -379,7 +379,7 @@ static int lo_rw_aio(struct loop_device lo, struct loop_cmd cmd,\n \t\tiov_iter_bvec(\u0026iter, rw,\n \t\t\t__bvec_iter_bvec(rq-\u003ebio-\u003ebi_io_vec, rq-\u003ebio-\u003ebi_iter),\n \t\t\tnr_bvec, blk_rq_bytes(rq));\n-\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_bvec_done;\n+\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_offset;\n \t}\n \tatomic_set(\u0026cmd-\u003eref, 2);\n \ndiff --git a/drivers/block/zloop.c b/drivers/block/zloop.c\nindex f97a20cfdb7ce..0fef5db8e5b80 100644\n--- a/drivers/block/zloop.c\n+++ b/drivers/block/zloop.c\n@@ -554,7 +554,7 @@ static int zloop_do_rw(struct zloop_cmd cmd)\n \t\tiov_iter_bvec(\u0026iter, rw,\n \t\t\t__bvec_iter_bvec(rq-\u003ebio-\u003ebi_io_vec, rq-\u003ebio-\u003ebi_iter),\n \t\t\t\t\tnr_bvec, blk_rq_bytes(rq));\n-\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_bvec_done;\n+\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_offset;\n \t}\n \n \tcmd-\u003eiocb.ki_pos = (cmd-\u003esector - zone-\u003estart) \u003c\u003c SECTOR_SHIFT;\ndiff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile\nindex b25d7550bacfd..523731b0f83ee 100644\n--- a/drivers/dma-buf/Makefile\n+++ b/drivers/dma-buf/Makefile\n@@ -1,6 +1,6 @@\n # SPDX-License-Identifier: GPL-2.0-only\n obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \\\n-\t dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o\n+\t dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o\n obj-$(CONFIG_DMABUF_HEAPS)\t+= dma-heap.o\n obj-$(CONFIG_DMABUF_HEAPS)\t+= heaps/\n obj-$(CONFIG_SYNC_FILE)\t\t+= sync_file.o\ndiff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c\nnew file mode 100644\nindex 0000000000000..24f58f80c45c4\n--- /dev/null\n+++ b/drivers/dma-buf/dma-buf-io.c\n@@ -0,0 +1,278 @@\n+/* SPDX-License-Identifier: GPL-2.0 /\n+/\n+ * Common infrastructure for supporing dma-buf in the I/O path.\n+ \n+ Copyright (C) 2026 Pavel Begunkov \u003casml.silence@gmail.com\u003e\n+ /\n+#include \u003clinux/dma-buf-io.h\u003e\n+#include \u003clinux/dma-resv.h\u003e\n+\n+struct dma_buf_io_fence {\n+\tstruct dma_fence base;\n+\tspinlock_t lock;\n+};\n+\n+static const char dma_buf_io_fence_drv_name(struct dma_fence fence)\n+{\n+\t/ default fence release kfree's the base pointer /\n+\tBUILD_BUG_ON(offsetof(struct dma_buf_io_fence, base));\n+\n+\treturn \"dma-buf-io-ctx\";\n+}\n+\n+static const char dma_buf_io_fence_timeline_name(struct dma_fence fence)\n+{\n+\treturn \"dma-buf-io-ctx\";\n+}\n+\n+const struct dma_fence_ops dma_buf_io_fence_ops = {\n+\t.get_driver_name = dma_buf_io_fence_drv_name,\n+\t.get_timeline_name = dma_buf_io_fence_timeline_name,\n+};\n+\n+static void dma_buf_io_ctx_destroy_work(struct work_struct work)\n+{\n+\tstruct dma_buf_io_ctx ctx = container_of(work, struct dma_buf_io_ctx,\n+\t\t\t\t\t\t release_work);\n+\n+\tif (WARN_ON_ONCE(refcount_read(\u0026ctx-\u003erefs)))\n+\t\treturn;\n+\n+\tctx-\u003edev_ops-\u003erelease(ctx);\n+\tdma_buf_put(ctx-\u003edmabuf);\n+\tkfree(ctx);\n+}\n+\n+static void dma_buf_io_map_release_work(struct work_struct work)\n+{\n+\tstruct dma_buf_io_map map = container_of(work, struct dma_buf_io_map,\n+\t\t\t\t\t\t release_work);\n+\tstruct dma_buf_io_fence fence = map-\u003efence;\n+\tstruct dma_buf_io_ctx ctx = map-\u003ectx;\n+\tstruct dma_buf dmabuf = ctx-\u003edmabuf;\n+\n+\t/* the release path must wait for fences /\n+\tif (WARN_ON_ONCE(refcount_read(\u0026ctx-\u003erefs) == 0))\n+\t\treturn;\n+\n+\t/ Prevent from destoying the ctx while unmapping /\n+\trefcount_inc(\u0026ctx-\u003erefs);\n+\n+\t/\n+\t * There are no more requests using the map, we can signal the fence.\n+\t * It should be done before taking the resv lock as someone could be\n+\t * waiting for the fence while holding the lock.\n+\t /\n+\tdma_fence_signal(\u0026fence-\u003ebase);\n+\n+\tdma_resv_lock(dmabuf-\u003eresv, NULL);\n+\tctx-\u003edev_ops-\u003eunmap(ctx, map);\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\n+\tdma_fence_put(\u0026fence-\u003ebase);\n+\tpercpu_ref_exit(\u0026map-\u003erefs);\n+\tkfree(map);\n+\n+\tif (refcount_dec_and_test(\u0026ctx-\u003erefs)) {\n+\t\t/\n+\t\t * Destruction needs to wait for I/O and dma fences. Defer it to\n+\t\t * simplify locking.\n+\t\t /\n+\t\tINIT_WORK(\u0026ctx-\u003erelease_work, dma_buf_io_ctx_destroy_work);\n+\t\tqueue_work(system_wq, \u0026ctx-\u003erelease_work);\n+\t}\n+}\n+\n+static void dma_buf_io_map_refs_release(struct percpu_ref ref)\n+{\n+\tstruct dma_buf_io_map map = container_of(ref, struct dma_buf_io_map, refs);\n+\n+\t/ might sleep, use a worker /\n+\tINIT_WORK(\u0026map-\u003erelease_work, dma_buf_io_map_release_work);\n+\tqueue_work(system_wq, \u0026map-\u003erelease_work);\n+}\n+\n+int dma_buf_io_init_map(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map)\n+{\n+\tstruct dma_buf_io_fence fence = NULL;\n+\tint ret;\n+\n+\tfence = kzalloc(sizeof(fence), GFP_KERNEL);\n+\tif (!fence)\n+\t\treturn -ENOMEM;\n+\n+\tret = percpu_ref_init(\u0026map-\u003erefs, dma_buf_io_map_refs_release, 0, GFP_KERNEL);\n+\tif (ret) {\n+\t\tkfree(fence);\n+\t\treturn ret;\n+\t}\n+\n+\tspin_lock_init(\u0026fence-\u003elock);\n+\tdma_fence_init(\u0026fence-\u003ebase, \u0026dma_buf_io_fence_ops, \u0026fence-\u003elock,\n+\t\t\tctx-\u003efence_ctx, atomic_inc_return(\u0026ctx-\u003efence_seq));\n+\tmap-\u003efence = fence;\n+\tmap-\u003ectx = ctx;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, \"DMA_BUF\");\n+\n+struct dma_buf_io_map dma_buf_io_create_map(struct dma_buf_io_ctx ctx)\n+{\n+\tstruct dma_buf dmabuf = ctx-\u003edmabuf;\n+\tstruct dma_buf_io_map map;\n+\tlong ret;\n+\n+retry:\n+\t/\n+\t * -\u003edmabuf_map() will be calling dma_buf_map_attachment(), for which\n+\t * we'll need to wait for fences. Do a bit nicer and try to wait\n+\t * without the resv lock first.\n+\t /\n+\tret = dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t true, MAX_SCHEDULE_TIMEOUT);\n+\tif (!ret)\n+\t\tret = -EAGAIN;\n+\tif (ret \u003c 0)\n+\t\treturn ERR_PTR(ret);\n+\n+\tret = dma_resv_lock_interruptible(dmabuf-\u003eresv, NULL);\n+\tif (ret)\n+\t\treturn ERR_PTR(ret);\n+\n+\tmap = dma_buf_io_get_map(ctx);\n+\tif (map) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t true, 0) \u003c 0) {\n+\t\tdma_resv_unlock(dmabuf-\u003eresv);\n+\t\tgoto retry;\n+\t}\n+\n+\tmap = ctx-\u003edev_ops-\u003emap(ctx);\n+\tif (IS_ERR(map)) {\n+\t\tret = PTR_ERR(map);\n+\t\tgoto out;\n+\t}\n+\n+\tif (WARN_ON_ONCE(!map-\u003eseg_shift))\n+\t\treturn ERR_PTR(-EFAULT);\n+\n+\tpercpu_ref_get(\u0026map-\u003erefs);\n+\trcu_assign_pointer(ctx-\u003emap, map);\n+out:\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\tif (ret \u003c 0)\n+\t\treturn ERR_PTR(ret);\n+\treturn map;\n+}\n+\n+static void dma_buf_io_drop_map(struct dma_buf_io_ctx ctx)\n+{\n+\tstruct dma_buf dmabuf = ctx-\u003edmabuf;\n+\tstruct dma_buf_io_map map;\n+\tint ret;\n+\n+\tdma_resv_assert_held(dmabuf-\u003eresv);\n+\n+\tmap = rcu_dereference_protected(ctx-\u003emap,\n+\t\t\t\t\tdma_resv_held(dmabuf-\u003eresv));\n+\tif (!map)\n+\t\treturn;\n+\trcu_assign_pointer(ctx-\u003emap, NULL);\n+\n+\tret = dma_resv_reserve_fences(dmabuf-\u003eresv, 1);\n+\tif (WARN_ON_ONCE(ret)) {\n+\t\tstruct dma_fence fence = \u0026map-\u003efence-\u003ebase;\n+\n+\t\tdma_fence_get(fence);\n+\t\tpercpu_ref_kill(\u0026map-\u003erefs);\n+\t\tdma_fence_wait(fence, false);\n+\t\tdma_fence_put(fence);\n+\t\treturn;\n+\t}\n+\n+\tdma_resv_add_fence(dmabuf-\u003eresv, \u0026map-\u003efence-\u003ebase,\n+\t\t\t DMA_RESV_USAGE_KERNEL);\n+\t/\n+\t * Delay destruction until all inflight requests using the map are\n+\t * gone. It'll also signal the fence then.\n+\t /\n+\tpercpu_ref_kill(\u0026map-\u003erefs);\n+}\n+\n+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx ctx)\n+{\n+\tdma_buf_io_drop_map(ctx);\n+}\n+EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, \"DMA_BUF\");\n+\n+static void dma_buf_io_ctx_release_work(struct work_struct work)\n+{\n+\tstruct dma_buf_io_ctx ctx = container_of(work, struct dma_buf_io_ctx,\n+\t\t\t\t\t\t release_work);\n+\tstruct dma_buf dmabuf = ctx-\u003edmabuf;\n+\tlong ret;\n+\n+\tdma_resv_lock(dmabuf-\u003eresv, NULL);\n+\t/ Remove the last map, there should be no new ones going forward. /\n+\tdma_buf_io_drop_map(ctx);\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\n+\t/ Wait until all maps are destroyed. /\n+\tret = dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t false, MAX_SCHEDULE_TIMEOUT);\n+\n+\tif (WARN_ON_ONCE(ret \u003c= 0))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(rcu_dereference_protected(ctx-\u003emap, true)))\n+\t\treturn;\n+\n+\tif (refcount_dec_and_test(\u0026ctx-\u003erefs))\n+\t\tdma_buf_io_ctx_destroy_work(\u0026ctx-\u003erelease_work);\n+}\n+\n+void dma_buf_io_ctx_release(struct dma_buf_io_ctx ctx)\n+{\n+\t/\n+\t Destruction needs to wait for I/O and dma fences. Defer it to\n+\t * simplify locking.\n+\t /\n+\tINIT_WORK(\u0026ctx-\u003erelease_work, dma_buf_io_ctx_release_work);\n+\tqueue_work(system_wq, \u0026ctx-\u003erelease_work);\n+}\n+\n+int dma_buf_io_ctx_create(struct file file,\n+\t\t\t struct dma_buf_io_ctx ctx,\n+\t\t\t struct dma_buf dmabuf,\n+\t\t\t enum dma_data_direction dir)\n+{\n+\tint ret;\n+\n+\tif (!file-\u003ef_op-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EOPNOTSUPP;\n+\n+\tmemset(ctx, 0, sizeof(ctx));\n+\tctx-\u003efence_ctx = dma_fence_context_alloc(1);\n+\tctx-\u003edir = dir;\n+\tctx-\u003edmabuf = dmabuf;\n+\trefcount_set(\u0026ctx-\u003erefs, 1);\n+\tget_dma_buf(dmabuf);\n+\n+\tret = file-\u003ef_op-\u003einit_dma_buf_io_ctx(file, ctx);\n+\tif (ret) {\n+\t\tmemset(ctx, 0, sizeof(ctx));\n+\t\tdma_buf_put(dmabuf);\n+\t\treturn ret;\n+\t}\n+\n+\tif (WARN_ON_ONCE(!ctx-\u003edev_ops \|\|\n+\t\t\t !ctx-\u003edev_ops-\u003emap \|\|\n+\t\t\t !ctx-\u003edev_ops-\u003eunmap \|\|\n+\t\t\t !ctx-\u003edev_ops-\u003erelease))\n+\t\treturn -EINVAL;\n+\n+\treturn ret;\n+}\ndiff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c\nindex 6155b0117c9d6..b22719c6411cf 100644\n--- a/drivers/md/dm-io-rewind.c\n+++ b/drivers/md/dm-io-rewind.c\n@@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec bv,\n \tint idx;\n \n \titer-\u003ebi_size += bytes;\n-\tif (bytes \u003c= iter-\u003ebi_bvec_done) {\n-\t\titer-\u003ebi_bvec_done -= bytes;\n+\tif (bytes \u003c= iter-\u003ebi_offset) {\n+\t\titer-\u003ebi_offset -= bytes;\n \t\treturn true;\n \t}\n \n-\tbytes -= iter-\u003ebi_bvec_done;\n+\tbytes -= iter-\u003ebi_offset;\n \tidx = iter-\u003ebi_idx - 1;\n \n \twhile (idx \u003e= 0 \u0026\u0026 bytes \u0026\u0026 bytes \u003e bv[idx].bv_len) {\n@@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec bv,\n \tif (WARN_ONCE(idx \u003c 0 \u0026\u0026 bytes,\n \t\t \"Attempted to rewind iter beyond bvec's boundaries\\n\")) {\n \t\titer-\u003ebi_size -= bytes;\n-\t\titer-\u003ebi_bvec_done = 0;\n+\t\titer-\u003ebi_offset = 0;\n \t\titer-\u003ebi_idx = 0;\n \t\treturn false;\n \t}\n \n \titer-\u003ebi_idx = idx;\n-\titer-\u003ebi_bvec_done = bv[idx].bv_len - bytes;\n+\titer-\u003ebi_offset = bv[idx].bv_len - bytes;\n \treturn true;\n }\n \n@@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bio bio,\n \titer-\u003ebi_sector -= bytes \u003e\u003e 9;\n \n \t/ No advance means no rewind /\n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size += bytes;\n-\telse\n+\t\titer-\u003ebi_offset -= bytes;\n+\t} else {\n \t\tdm_bvec_iter_rewind(bio-\u003ebi_io_vec, iter, bytes);\n+\t}\n }\n \n /\ndiff --git a/drivers/md/dm-pcache/segment.c b/drivers/md/dm-pcache/segment.c\nindex 7e98187014451..8f8816e1c539f 100644\n--- a/drivers/md/dm-pcache/segment.c\n+++ b/drivers/md/dm-pcache/segment.c\n@@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment segment,\n \n \tiov_iter_bvec(\u0026iter, ITER_DEST, \u0026bio-\u003ebi_io_vec[bio-\u003ebi_iter.bi_idx],\n \t\t\tbio_segments(bio), bio-\u003ebi_iter.bi_size);\n-\titer.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\titer.iov_offset = bio-\u003ebi_iter.bi_offset;\n \tif (bio_off)\n \t\tiov_iter_advance(\u0026iter, bio_off);\n \n@@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment segment,\n \n \tiov_iter_bvec(\u0026iter, ITER_SOURCE, \u0026bio-\u003ebi_io_vec[bio-\u003ebi_iter.bi_idx],\n \t\t\tbio_segments(bio), bio-\u003ebi_iter.bi_size);\n-\titer.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\titer.iov_offset = bio-\u003ebi_iter.bi_offset;\n \tif (bio_off)\n \t\tiov_iter_advance(\u0026iter, bio_off);\n \ndiff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c\nindex 7e1112960d7f8..5d910a64503d3 100644\n--- a/drivers/nvdimm/btt.c\n+++ b/drivers/nvdimm/btt.c\n@@ -1155,7 +1155,7 @@ static int btt_rw_integrity(struct btt btt, struct bio_integrity_payload bip,\n \t\tbv = bvec_iter_bvec(bip-\u003ebip_vec, bip-\u003ebip_iter);\n \t\t/\n \t\t * The 'bv' obtained from bvec_iter_bvec has its .bv_len and\n-\t\t * .bv_offset already adjusted for iter-\u003ebi_bvec_done, and we\n+\t\t * .bv_offset already adjusted for iter-\u003ebi_offset, and we\n \t\t * can use those directly\n \t\t /\n \ndiff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c\nindex 453c1f0b2dd09..8bd407603d928 100644\n--- a/drivers/nvme/host/core.c\n+++ b/drivers/nvme/host/core.c\n@@ -2676,6 +2676,17 @@ static int nvme_report_zones(struct gendisk disk, sector_t sector,\n #define nvme_report_zones\tNULL\n #endif /* CONFIG_BLK_DEV_ZONED /\n \n+static int nvme_init_dma_buf_io_ctx(struct block_device bdev,\n+\t\tstruct dma_buf_io_ctx ctx)\n+{\n+\tstruct nvme_ns ns = bdev-\u003ebd_disk-\u003eprivate_data;\n+\tstruct nvme_ctrl ctrl = ns-\u003ectrl;\n+\n+\tif (!ctrl-\u003eops-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EINVAL;\n+\treturn ctrl-\u003eops-\u003einit_dma_buf_io_ctx(ctrl, ctx);\n+}\n+\n const struct block_device_operations nvme_bdev_ops = {\n \t.owner\t\t= THIS_MODULE,\n \t.ioctl\t\t= nvme_ioctl,\n@@ -2686,6 +2697,7 @@ const struct block_device_operations nvme_bdev_ops = {\n \t.get_unique_id\t= nvme_get_unique_id,\n \t.report_zones\t= nvme_report_zones,\n \t.pr_ops\t\t= \u0026nvme_pr_ops,\n+\t.init_dma_buf_io_ctx = nvme_init_dma_buf_io_ctx,\n };\n \n static int nvme_wait_ready(struct nvme_ctrl ctrl, u32 mask, u32 val,\ndiff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h\nindex 824651cc898db..034c31af8fecf 100644\n--- a/drivers/nvme/host/nvme.h\n+++ b/drivers/nvme/host/nvme.h\n@@ -652,6 +652,8 @@ struct nvme_ctrl_ops {\n \tint (get_address)(struct nvme_ctrl ctrl, char buf, int size);\n \tvoid (print_device_info)(struct nvme_ctrl ctrl);\n \tbool (supports_pci_p2pdma)(struct nvme_ctrl ctrl);\n+\tint (init_dma_buf_io_ctx)(struct nvme_ctrl ctrl,\n+\t\t\t\t struct dma_buf_io_ctx ctx);\n \tunsigned long (get_virt_boundary)(struct nvme_ctrl ctrl, bool is_admin);\n };\n \ndiff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c\nindex 69932d640b537..67e26d6e8f255 100644\n--- a/drivers/nvme/host/pci.c\n+++ b/drivers/nvme/host/pci.c\n@@ -27,6 +27,8 @@\n #include \u003clinux/io-64-nonatomic-lo-hi.h\u003e\n #include \u003clinux/io-64-nonatomic-hi-lo.h\u003e\n #include \u003clinux/sed-opal.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n+#include \u003clinux/dma-resv.h\u003e\n \n #include \"trace.h\"\n #include \"nvme.h\"\n@@ -393,6 +395,13 @@ struct nvme_queue {\n \tstruct completion delete_done;\n };\n \n+struct nvme_dmabuf_map {\n+\tstruct dma_buf_io_map base;\n+\tstruct sg_table sgt;\n+\tunsigned nr_entries;\n+\tdma_addr_t dma_list[];\n+};\n+\n / bits for iod-\u003eflags /\n enum nvme_iod_flags {\n \t/ this command has been aborted by the timeout handler /\n@@ -859,6 +868,138 @@ static void nvme_free_descriptors(struct request req)\n \t}\n }\n \n+static inline struct nvme_dmabuf_map \n+to_nvme_dmabuf_map(struct dma_buf_io_map map)\n+{\n+\treturn container_of(map, struct nvme_dmabuf_map, base);\n+}\n+\n+static void nvme_dmabuf_map_sync_for_cpu(struct nvme_dev nvme_dev,\n+\t\t\t\t\t struct request req)\n+{\n+\tstruct device dev = nvme_dev-\u003edev;\n+\tenum dma_data_direction dma_dir;\n+\tstruct bio bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tdma_addr_t dma_list = map-\u003edma_list;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;\n+\tint length = blk_rq_payload_bytes(req) +\n+\t\t (offset \u0026 (NVME_CTRL_PAGE_SIZE - 1));\n+\n+\tdma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;\n+\n+\twhile (length \u003e 0) {\n+\t\tdma_sync_single_for_cpu(dev, dma_list[map_idx++],\n+\t\t\t\t\tNVME_CTRL_PAGE_SIZE, dma_dir);\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t}\n+}\n+\n+static void nvme_dmabuf_map_sync_for_device(struct nvme_dev nvme_dev,\n+\t\t\t\t\t struct request req)\n+{\n+\tstruct device dev = nvme_dev-\u003edev;\n+\tenum dma_data_direction dma_dir;\n+\tstruct bio bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tdma_addr_t dma_list = map-\u003edma_list;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;\n+\tint length = blk_rq_payload_bytes(req) +\n+\t\t (offset \u0026 (NVME_CTRL_PAGE_SIZE - 1));\n+\n+\tdma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;\n+\n+\twhile (length \u003e 0) {\n+\t\tdma_sync_single_for_device(dev, dma_list[map_idx++],\n+\t\t\t\t\t NVME_CTRL_PAGE_SIZE, dma_dir);\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t}\n+}\n+\n+static void nvme_rq_clean_dmabuf_map(struct nvme_dev dev,\n+\t\t\t\t struct request req)\n+{\n+\tstruct nvme_iod iod = blk_mq_rq_to_pdu(req);\n+\n+\tnvme_dmabuf_map_sync_for_cpu(dev, req);\n+\n+\tif (!(iod-\u003eflags \u0026 IOD_SINGLE_SEGMENT))\n+\t\tnvme_free_descriptors(req);\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf_map(struct request req,\n+\t\t\t\t\t struct nvme_queue nvmeq)\n+{\n+\tstruct nvme_iod iod = blk_mq_rq_to_pdu(req);\n+\tstruct bio bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tunsigned bvec_done = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = bvec_done / NVME_CTRL_PAGE_SIZE;\n+\tunsigned offset = bvec_done \u0026 (NVME_CTRL_PAGE_SIZE - 1);\n+\tint length = blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset);\n+\tdma_addr_t dma_list = map-\u003edma_list;\n+\tu64 prp1_dma = dma_list[map_idx++] + offset;\n+\tu64 dma_addr, prp2_dma;\n+\tdma_addr_t prp_dma;\n+\t__le64 prp_list;\n+\tunsigned i;\n+\n+\tnvme_dmabuf_map_sync_for_device(nvmeq-\u003edev, req);\n+\n+\tif (length \u003c= 0) {\n+\t\tprp2_dma = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tif (length \u003c= NVME_CTRL_PAGE_SIZE) {\n+\t\tprp2_dma = dma_list[map_idx];\n+\t\tgoto done;\n+\t}\n+\n+\tif (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) \u003c=\n+\t NVME_SMALL_POOL_SIZE / sizeof(__le64))\n+\t\tiod-\u003eflags \|= IOD_SMALL_DESCRIPTOR;\n+\n+\tprp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,\n+\t\t\t\u0026prp_dma);\n+\tif (!prp_list)\n+\t\treturn BLK_STS_RESOURCE;\n+\n+\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = prp_list;\n+\tprp2_dma = prp_dma;\n+\ti = 0;\n+\tfor (;;) {\n+\t\tif (i == NVME_CTRL_PAGE_SIZE \u003e\u003e 3) {\n+\t\t\t__le64 old_prp_list = prp_list;\n+\n+\t\t\tprp_list = dma_pool_alloc(nvmeq-\u003edescriptor_pools.large,\n+\t\t\t\t\tGFP_ATOMIC, \u0026prp_dma);\n+\t\t\tif (!prp_list)\n+\t\t\t\tgoto free_prps;\n+\t\t\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = prp_list;\n+\t\t\tprp_list[0] = old_prp_list[i - 1];\n+\t\t\told_prp_list[i - 1] = cpu_to_le64(prp_dma);\n+\t\t\ti = 1;\n+\t\t}\n+\n+\t\tdma_addr = dma_list[map_idx++];\n+\t\tprp_list[i++] = cpu_to_le64(dma_addr);\n+\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t\tif (length \u003c= 0)\n+\t\t\tbreak;\n+\t}\n+done:\n+\tiod-\u003ecmd.common.dptr.prp1 = cpu_to_le64(prp1_dma);\n+\tiod-\u003ecmd.common.dptr.prp2 = cpu_to_le64(prp2_dma);\n+\treturn BLK_STS_OK;\n+free_prps:\n+\tnvme_free_descriptors(req);\n+\treturn BLK_STS_RESOURCE;\n+}\n+\n static void nvme_free_prps(struct request req, unsigned int attrs)\n {\n \tstruct nvme_iod iod = blk_mq_rq_to_pdu(req);\n@@ -937,6 +1078,11 @@ static void nvme_unmap_data(struct request req)\n \tstruct device dma_dev = nvmeq-\u003edev-\u003edev;\n \tunsigned int attrs = 0;\n \n+\tif (blk_mq_rq_is_dmabuf(req)) {\n+\t\tnvme_rq_clean_dmabuf_map(nvmeq-\u003edev, req);\n+\t\treturn;\n+\t}\n+\n \tif (iod-\u003eflags \u0026 IOD_SINGLE_SEGMENT) {\n \t\tstatic_assert(offsetof(union nvme_data_ptr, prp1) ==\n \t\t\t\toffsetof(union nvme_data_ptr, sgl.addr));\n@@ -1142,13 +1288,19 @@ static blk_status_t nvme_pci_setup_data_prp(struct request req,\n }\n \n static void nvme_pci_sgl_set_data(struct nvme_sgl_desc sge,\n-\t\tstruct blk_dma_iter iter)\n+\t\tdma_addr_t addr, u32 len)\n {\n-\tsge-\u003eaddr = cpu_to_le64(iter-\u003eaddr);\n-\tsge-\u003elength = cpu_to_le32(iter-\u003elen);\n+\tsge-\u003eaddr = cpu_to_le64(addr);\n+\tsge-\u003elength = cpu_to_le32(len);\n \tsge-\u003etype = NVME_SGL_FMT_DATA_DESC \u003c\u003c 4;\n }\n \n+static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc sge,\n+\t\tstruct blk_dma_iter iter)\n+{\n+\tnvme_pci_sgl_set_data(sge, iter-\u003eaddr, iter-\u003elen);\n+}\n+\n static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc sge,\n \t\tdma_addr_t dma_addr, int entries)\n {\n@@ -1157,6 +1309,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc sge,\n \tsge-\u003etype = NVME_SGL_FMT_LAST_SEG_DESC \u003c\u003c 4;\n }\n \n+static unsigned int nvme_pci_dmabuf_sgl_nents(struct request req)\n+{\n+\tstruct bio bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tstruct scatterlist sg;\n+\tunsigned long tmp;\n+\tsize_t offset = bio-\u003ebi_iter.bi_offset;\n+\tsize_t remaining = blk_rq_payload_bytes(req);\n+\tunsigned int nents = 0;\n+\n+\tfor_each_sgtable_dma_sg(map-\u003esgt, sg, tmp) {\n+\t\tsize_t sg_len = sg_dma_len(sg);\n+\n+\t\tif (!remaining)\n+\t\t\tbreak;\n+\t\tif (offset \u003e= sg_len) {\n+\t\t\toffset -= sg_len;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tsg_len -= offset;\n+\t\toffset = 0;\n+\n+\t\tdo {\n+\t\t\tsize_t chunk = min(remaining, sg_len);\n+\n+\t\t\tnents++;\n+\t\t\tsg_len -= chunk;\n+\t\t\tremaining -= chunk;\n+\t\t} while (sg_len \u0026\u0026 remaining);\n+\t}\n+\n+\tif (unlikely(remaining))\n+\t\treturn 0;\n+\n+\treturn nents;\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request req,\n+\t\tstruct nvme_queue nvmeq, unsigned int entries)\n+{\n+\tstruct nvme_iod iod = blk_mq_rq_to_pdu(req);\n+\tstruct bio bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tsize_t length = blk_rq_payload_bytes(req);\n+\tstruct nvme_sgl_desc sg_list = \u0026iod-\u003ecmd.common.dptr.sgl;\n+\tbool pooled = entries \u003e 1;\n+\tdma_addr_t sgl_dma = 0;\n+\tunsigned int mapped = 0;\n+\tunsigned long tmp;\n+\tstruct scatterlist sg;\n+\tsize_t offset = bio-\u003ebi_iter.bi_offset;\n+\tsize_t remaining = length;\n+\n+\tif (!entries)\n+\t\treturn BLK_STS_IOERR;\n+\n+\tiod-\u003ecmd.common.flags = NVME_CMD_SGL_METABUF;\n+\tiod-\u003etotal_len = length;\n+\n+\tnvme_dmabuf_map_sync_for_device(nvmeq-\u003edev, req);\n+\n+\t/ entries == 1 fits in the inline descriptor; more needs a pool. /\n+\tif (pooled) {\n+\t\tif (entries \u003c= NVME_SMALL_POOL_SIZE / sizeof(sg_list))\n+\t\t\tiod-\u003eflags \|= IOD_SMALL_DESCRIPTOR;\n+\n+\t\tsg_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,\n+\t\t\t\t\t \u0026sgl_dma);\n+\t\tif (!sg_list)\n+\t\t\treturn BLK_STS_RESOURCE;\n+\t\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = sg_list;\n+\t}\n+\n+\tfor_each_sgtable_dma_sg(map-\u003esgt, sg, tmp) {\n+\t\tsize_t sg_len = sg_dma_len(sg);\n+\t\tdma_addr_t addr = sg_dma_address(sg);\n+\n+\t\tif (!remaining)\n+\t\t\tbreak;\n+\t\tif (offset \u003e= sg_len) {\n+\t\t\toffset -= sg_len;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\taddr += offset;\n+\t\tsg_len -= offset;\n+\t\toffset = 0;\n+\n+\t\tdo {\n+\t\t\tu32 chunk = min_t(size_t, remaining, sg_len);\n+\n+\t\t\tif (WARN_ON_ONCE(mapped == entries))\n+\t\t\t\tgoto err_free;\n+\t\t\tnvme_pci_sgl_set_data(\u0026sg_list[mapped++], addr, chunk);\n+\n+\t\t\taddr += chunk;\n+\t\t\tsg_len -= chunk;\n+\t\t\tremaining -= chunk;\n+\t\t} while (sg_len \u0026\u0026 remaining);\n+\t}\n+\n+\tif (unlikely(remaining))\n+\t\tgoto err_free;\n+\n+\tif (pooled)\n+\t\tnvme_pci_sgl_set_seg(\u0026iod-\u003ecmd.common.dptr.sgl, sgl_dma,\n+\t\t\t\t mapped);\n+\treturn BLK_STS_OK;\n+err_free:\n+\tif (pooled) {\n+\t\tiod-\u003enr_descriptors--;\n+\t\tdma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma);\n+\t}\n+\treturn BLK_STS_IOERR;\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf(struct request req,\n+\t\tstruct nvme_queue nvmeq, enum nvme_use_sgl use_sgl)\n+{\n+\tunsigned int entries;\n+\tsize_t avg_seg;\n+\n+\tif (use_sgl == SGL_UNSUPPORTED)\n+\t\treturn nvme_rq_setup_dmabuf_map(req, nvmeq);\n+\n+\tentries = nvme_pci_dmabuf_sgl_nents(req);\n+\n+\tif (use_sgl == SGL_FORCED)\n+\t\treturn nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);\n+\n+\tavg_seg = entries ?\n+\t\tDIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0;\n+\tif (sgl_threshold \u0026\u0026 avg_seg \u003e= sgl_threshold)\n+\t\treturn nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);\n+\n+\treturn nvme_rq_setup_dmabuf_map(req, nvmeq);\n+}\n+\n static blk_status_t nvme_pci_setup_data_sgl(struct request req,\n \t\tstruct blk_dma_iter iter)\n {\n@@ -1171,7 +1462,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request req,\n \tiod-\u003ecmd.common.flags = NVME_CMD_SGL_METABUF;\n \n \tif (entries == 1 \|\| blk_rq_dma_map_coalesce(\u0026iod-\u003edma_state)) {\n-\t\tnvme_pci_sgl_set_data(\u0026iod-\u003ecmd.common.dptr.sgl, iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026iod-\u003ecmd.common.dptr.sgl, iter);\n \t\tiod-\u003etotal_len += iter-\u003elen;\n \t\treturn BLK_STS_OK;\n \t}\n@@ -1193,7 +1484,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request req,\n \t\t\titer-\u003estatus = BLK_STS_IOERR;\n \t\t\tbreak;\n \t\t}\n-\t\tnvme_pci_sgl_set_data(\u0026sg_list[mapped++], iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026sg_list[mapped++], iter);\n \t\tiod-\u003etotal_len += iter-\u003elen;\n \t} while (blk_rq_dma_map_iter_next(req, nvmeq-\u003edev-\u003edev, iter));\n \n@@ -1251,6 +1542,9 @@ static blk_status_t nvme_map_data(struct request req)\n \tstruct blk_dma_iter iter;\n \tblk_status_t ret;\n \n+\tif (blk_mq_rq_is_dmabuf(req))\n+\t\treturn nvme_rq_setup_dmabuf(req, nvmeq, use_sgl);\n+\n \t/\n \t * Try to skip the DMA iterator for single segment requests, as that\n \t * significantly improves performances for small I/O sizes.\n@@ -1353,13 +1647,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request req)\n \tiod-\u003ecmd.common.metadata = cpu_to_le64(sgl_dma);\n \tif (entries == 1) {\n \t\tiod-\u003emeta_total_len = iter.len;\n-\t\tnvme_pci_sgl_set_data(sg_list, \u0026iter);\n+\t\tnvme_pci_dma_iter_set_sgl(sg_list, \u0026iter);\n \t\treturn BLK_STS_OK;\n \t}\n \n \tsgl_dma += sizeof(sg_list);\n \tdo {\n-\t\tnvme_pci_sgl_set_data(\u0026sg_list[++i], \u0026iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026sg_list[++i], \u0026iter);\n \t\tiod-\u003emeta_total_len += iter.len;\n \t} while (blk_rq_integrity_dma_map_iter_next(req, dev-\u003edev, \u0026iter));\n \n@@ -2269,6 +2563,118 @@ static int nvme_create_queue(struct nvme_queue nvmeq, int qid, bool polled)\n \treturn result;\n }\n \n+#ifdef CONFIG_DMA_SHARED_BUFFER\n+static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment attach)\n+{\n+\tstruct dma_buf_io_ctx ctx = attach-\u003eimporter_priv;\n+\n+\tdma_buf_io_invalidate_mappings(ctx);\n+}\n+\n+const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = {\n+\t.invalidate_mappings\t= nvme_dmabuf_invalidate_mappings,\n+\t.allow_peer2peer\t= true,\n+};\n+\n+static struct dma_buf_io_map nvme_dma_buf_io_map(struct dma_buf_io_ctx ctx)\n+{\n+\tunsigned nr_entries = ctx-\u003edmabuf-\u003esize / NVME_CTRL_PAGE_SIZE;\n+\tstruct dma_buf_attachment attach = ctx-\u003edev_priv;\n+\tunsigned long tmp, i = 0;\n+\tunsigned seg_shift = ~0U;\n+\tstruct nvme_dmabuf_map map;\n+\tstruct scatterlist sg;\n+\tstruct sg_table sgt;\n+\tint ret;\n+\n+\tdma_resv_assert_held(ctx-\u003edmabuf-\u003eresv);\n+\n+\tmap = kmalloc_flex(map, dma_list, nr_entries);\n+\tif (!map)\n+\t\treturn ERR_PTR(-ENOMEM);\n+\n+\tsgt = dma_buf_map_attachment(attach, ctx-\u003edir);\n+\tif (IS_ERR(sgt)) {\n+\t\tret = PTR_ERR(sgt);\n+\t\tsgt = NULL;\n+\t\tgoto err;\n+\t}\n+\n+\tfor_each_sgtable_dma_sg(sgt, sg, tmp) {\n+\t\tdma_addr_t dma_addr = sg_dma_address(sg);\n+\t\tunsigned long sg_len = sg_dma_len(sg);\n+\n+\t\tif (sg_len % NVME_CTRL_PAGE_SIZE) {\n+\t\t\tret = -EINVAL;\n+\t\t\tgoto err;\n+\t\t}\n+\t\tseg_shift = min(seg_shift, __ffs(sg_len));\n+\n+\t\twhile (sg_len) {\n+\t\t\tmap-\u003edma_list[i++] = dma_addr;\n+\t\t\tdma_addr += NVME_CTRL_PAGE_SIZE;\n+\t\t\tsg_len -= NVME_CTRL_PAGE_SIZE;\n+\t\t}\n+\t}\n+\n+\tif (WARN_ON_ONCE(seg_shift \u003c NVME_CTRL_PAGE_SHIFT))\n+\t\treturn ERR_PTR(-EFAULT);\n+\n+\tret = dma_buf_io_init_map(ctx, \u0026map-\u003ebase);\n+\tif (ret)\n+\t\tgoto err;\n+\tmap-\u003ebase.seg_shift = seg_shift;\n+\tmap-\u003enr_entries = nr_entries;\n+\tmap-\u003esgt = sgt;\n+\treturn \u0026map-\u003ebase;\n+err:\n+\tif (sgt)\n+\t\tdma_buf_unmap_attachment(attach, sgt, ctx-\u003edir);\n+\tkfree(map);\n+\treturn ERR_PTR(ret);\n+}\n+\n+static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx ctx,\n+\t\t\t\t struct dma_buf_io_map map_base)\n+{\n+\tstruct dma_buf_attachment attach = ctx-\u003edev_priv;\n+\tstruct nvme_dmabuf_map map = to_nvme_dmabuf_map(map_base);\n+\n+\tdma_resv_assert_held(ctx-\u003edmabuf-\u003eresv);\n+\n+\tdma_buf_unmap_attachment(attach, map-\u003esgt, ctx-\u003edir);\n+}\n+\n+static void nvme_dma_buf_io_release(struct dma_buf_io_ctx ctx)\n+{\n+\tstruct dma_buf_attachment attach = ctx-\u003edev_priv;\n+\n+\tdma_buf_detach(ctx-\u003edmabuf, attach);\n+}\n+\n+const struct dma_buf_io_ops nvme_dma_buf_io_ops = {\n+\t.map\t\t= nvme_dma_buf_io_map,\n+\t.unmap\t\t= nvme_dma_buf_io_unmap,\n+\t.release\t= nvme_dma_buf_io_release,\n+};\n+\n+static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl ctrl,\n+\t\t\t\t\tstruct dma_buf_io_ctx ctx)\n+{\n+\tstruct dma_buf_attachment attach;\n+\tstruct nvme_dev dev = to_nvme_dev(ctrl);\n+\n+\tattach = dma_buf_dynamic_attach(ctx-\u003edmabuf, dev-\u003edev,\n+\t\t\t\t\t\u0026nvme_dmabuf_importer_ops, ctx);\n+\tif (IS_ERR(attach))\n+\t\treturn PTR_ERR(attach);\n+\n+\tctx-\u003edev_priv = attach;\n+\tctx-\u003edev_ops = \u0026nvme_dma_buf_io_ops;\n+\treturn 0;\n+}\n+#endif\n+\n static const struct blk_mq_ops nvme_mq_admin_ops = {\n \t.queue_rq\t= nvme_queue_rq,\n \t.complete\t= nvme_pci_complete_rq,\n@@ -3563,6 +3969,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {\n \t.print_device_info\t= nvme_pci_print_device_info,\n \t.supports_pci_p2pdma\t= nvme_pci_supports_pci_p2pdma,\n \t.get_virt_boundary\t= nvme_pci_get_virt_boundary,\n+#ifdef CONFIG_DMA_SHARED_BUFFER\n+\t.init_dma_buf_io_ctx\t= nvme_pci_init_dma_buf_io_ctx,\n+#endif\n };\n \n static int nvme_dev_map(struct nvme_dev dev)\n@@ -4327,5 +4736,6 @@ MODULE_AUTHOR(\"Matthew Wilcox \u003cwilly@linux.intel.com\u003e\");\n MODULE_LICENSE(\"GPL\");\n MODULE_VERSION(\"1.0\");\n MODULE_DESCRIPTION(\"NVMe host PCIe transport driver\");\n+MODULE_IMPORT_NS(\"DMA_BUF\");\n module_init(nvme_init);\n module_exit(nvme_exit);\ndiff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c\nindex ba5c7b3e2a7c6..ce03a0ea4ded7 100644\n--- a/drivers/nvme/host/tcp.c\n+++ b/drivers/nvme/host/tcp.c\n@@ -357,7 +357,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request req,\n \t\tiov_iter_bvec(\u0026req-\u003eiter, dir,\n \t\t\t__bvec_iter_bvec(bio-\u003ebi_io_vec, bio-\u003ebi_iter), nr_bvec,\n \t\t\tbio-\u003ebi_iter.bi_size);\n-\t\treq-\u003eiter.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\t\treq-\u003eiter.iov_offset = bio-\u003ebi_iter.bi_offset;\n \t}\n }\n \ndiff --git a/fs/btrfs/misc.h b/fs/btrfs/misc.h\nindex 694be6d0562a4..8020609431801 100644\n--- a/fs/btrfs/misc.h\n+++ b/fs/btrfs/misc.h\n@@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio bio)\n \t\t.bi_sector = 0,\n \t\t.bi_size = bio_size,\n \t\t.bi_idx = 0,\n-\t\t.bi_bvec_done = 0,\n+\t\t.bi_offset = 0,\n \t};\n }\n \ndiff --git a/include/linux/bio.h b/include/linux/bio.h\nindex 0445ecba3b247..22ce9deb2febb 100644\n--- a/include/linux/bio.h\n+++ b/include/linux/bio.h\n@@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio bio)\n {\n \treturn bio_op(bio) == REQ_OP_DISCARD \|\|\n \t bio_op(bio) == REQ_OP_SECURE_ERASE \|\|\n-\t bio_op(bio) == REQ_OP_WRITE_ZEROES;\n+\t bio_op(bio) == REQ_OP_WRITE_ZEROES \|\|\n+\t op_is_dmabuf(bio-\u003ebi_opf);\n }\n \n static inline void bio_data(struct bio bio)\n@@ -113,11 +114,13 @@ static inline void bio_advance_iter(const struct bio bio,\n {\n \titer-\u003ebi_sector += bytes \u003e\u003e 9;\n \n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size -= bytes;\n-\telse\n+\t\titer-\u003ebi_offset += bytes;\n+\t} else {\n \t\tbvec_iter_advance(bio-\u003ebi_io_vec, iter, bytes);\n \t\t/ TODO: It is reasonable to complete bio with error here. /\n+\t}\n }\n \n / @bytes should be less or equal to bvec[i-\u003ebi_idx].bv_len /\n@@ -127,10 +130,12 @@ static inline void bio_advance_iter_single(const struct bio bio,\n {\n \titer-\u003ebi_sector += bytes \u003e\u003e 9;\n \n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size -= bytes;\n-\telse\n+\t\titer-\u003ebi_offset += bytes;\n+\t} else {\n \t\tbvec_iter_advance_single(bio-\u003ebi_io_vec, iter, bytes);\n+\t}\n }\n \n void __bio_advance(struct bio , unsigned bytes);\n@@ -434,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio bio)\n \n /\n Calculate number of bvec segments that should be allocated to fit data\n- * pointed by @iter. If @iter is backed by bvec it's going to be reused\n- * instead of allocating a new one.\n+ * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec array /\n+ * the dma map are going to be reused, and so no extra allocation is required.\n /\n static inline int bio_iov_vecs_to_alloc(struct iov_iter iter, int max_segs)\n {\n-\tif (iov_iter_is_bvec(iter))\n+\tif (iov_iter_is_bvec(iter) \|\| iov_iter_is_dmabuf_map(iter))\n \t\treturn 0;\n \treturn iov_iter_npages(iter, max_segs);\n }\n@@ -518,7 +523,7 @@ int bdev_rw_virt(struct block_device bdev, sector_t sector, void data,\n int bio_iov_iter_get_pages(struct bio bio, struct iov_iter iter,\n \t\tunsigned mem_align_mask, unsigned len_align_mask);\n \n-void bio_iov_bvec_set(struct bio bio, const struct iov_iter iter);\n+bool bio_iov_iter_set(struct bio bio, const struct iov_iter iter);\n void __bio_release_pages(struct bio bio, bool mark_dirty);\n extern void bio_set_pages_dirty(struct bio bio);\n extern void bio_check_pages_dirty(struct bio bio);\ndiff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h\nindex af878597afb8c..7c7504c84e090 100644\n--- a/include/linux/blk-mq.h\n+++ b/include/linux/blk-mq.h\n@@ -1017,6 +1017,13 @@ static inline void blk_mq_rq_to_pdu(struct request rq)\n \treturn rq + 1;\n }\n \n+static inline bool blk_mq_rq_is_dmabuf(struct request rq)\n+{\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn false;\n+\treturn rq-\u003ebio \u0026\u0026 op_is_dmabuf(rq-\u003ebio-\u003ebi_opf);\n+}\n+\n static inline struct blk_mq_hw_ctx queue_hctx(struct request_queue q, int id)\n {\n \tstruct blk_mq_hw_ctx hctx;\ndiff --git a/include/linux/blk_types.h b/include/linux/blk_types.h\nindex d49d97a050d0e..a305951f03124 100644\n--- a/include/linux/blk_types.h\n+++ b/include/linux/blk_types.h\n@@ -233,7 +233,12 @@ struct bio {\n \tatomic_t\t\t__bi_remaining;\n \n \t/ The actual vec list, preserved by bio_reset() /\n-\tstruct bio_vec\t\tbi_io_vec;\n+\tunion {\n+\t\tstruct bio_vec\t\tbi_io_vec;\n+\t\t/ Driver specific dma map, valid IFF REQ_DMABUF is set /\n+\t\tstruct dma_buf_io_map\tbi_dmabuf_map;\n+\t};\n+\n \tstruct bvec_iter\tbi_iter;\n \n \tunion {\n@@ -402,6 +407,7 @@ enum req_flag_bits {\n \t__REQ_DRV,\t\t/* for driver use /\n \t__REQ_FS_PRIVATE,\t/ for file system (submitter) use /\n \t__REQ_ATOMIC,\t\t/ for atomic write operations /\n+\t__REQ_DMABUF,\t\t/ Using premmaped dma buffers /\n \t/\n \t * Command specific flags, keep last:\n \t /\n@@ -434,6 +440,7 @@ enum req_flag_bits {\n #define REQ_DRV\t\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_DRV)\n #define REQ_FS_PRIVATE\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_FS_PRIVATE)\n #define REQ_ATOMIC\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_ATOMIC)\n+#define REQ_DMABUF\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_DMABUF)\n \n #define REQ_NOUNMAP\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_NOUNMAP)\n \n@@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op)\n \treturn (op \u0026 REQ_OP_MASK) == REQ_OP_DISCARD;\n }\n \n+static inline bool op_is_dmabuf(blk_opf_t op)\n+{\n+\treturn op \u0026 REQ_DMABUF;\n+}\n+\n /\n * Check if a bio or request operation is a zone management operation.\n /\ndiff --git a/include/linux/blkdev.h b/include/linux/blkdev.h\nindex 9213a5716f95a..9f4c92e51dc95 100644\n--- a/include/linux/blkdev.h\n+++ b/include/linux/blkdev.h\n@@ -1682,6 +1682,8 @@ struct block_device_operations {\n \t/ returns the length of the identifier or a negative errno: /\n \tint (get_unique_id)(struct gendisk disk, u8 id[16],\n \t\t\tenum blk_unique_id id_type);\n+\tint (init_dma_buf_io_ctx)(struct block_device ,\n+\t\t\t\t struct dma_buf_io_ctx );\n \tstruct module owner;\n \tconst struct pr_ops pr_ops;\n \ndiff --git a/include/linux/bvec.h b/include/linux/bvec.h\nindex 92837e2743f19..b63914ff56e33 100644\n--- a/include/linux/bvec.h\n+++ b/include/linux/bvec.h\n@@ -108,9 +108,10 @@ struct bvec_iter {\n \tunsigned int\t\tbi_idx;\n \n \t/\n-\t Current offset in the bvec entry pointed to by `bi_idx`.\n+\t * Current offset in the bvec entry pointed to by `bi_idx` or into\n+\t * a dma-buf map.\n \t /\n-\tunsigned int\t\tbi_bvec_done;\n+\tunsigned int\t\tbi_offset;\n } __packed __aligned(4);\n \n struct bvec_iter_all {\n@@ -135,14 +136,14 @@ mp_bvec_iter_page(const struct bio_vec bvecs, const struct bvec_iter iter)\n static __always_inline unsigned int\n mp_bvec_iter_len(const struct bio_vec bvecs, const struct bvec_iter iter)\n {\n-\treturn min(__bvec_iter_bvec(bvecs, iter)-\u003ebv_len - iter.bi_bvec_done,\n+\treturn min(__bvec_iter_bvec(bvecs, iter)-\u003ebv_len - iter.bi_offset,\n \t\t\titer.bi_size);\n }\n \n static __always_inline unsigned int\n mp_bvec_iter_offset(const struct bio_vec bvecs, const struct bvec_iter iter)\n {\n-\treturn __bvec_iter_bvec(bvecs, iter)-\u003ebv_offset + iter.bi_bvec_done;\n+\treturn __bvec_iter_bvec(bvecs, iter)-\u003ebv_offset + iter.bi_offset;\n }\n \n static __always_inline unsigned int\n@@ -204,7 +205,7 @@ static inline bool bvec_iter_advance(const struct bio_vec bv,\n \t}\n \n \titer-\u003ebi_size -= bytes;\n-\tbytes += iter-\u003ebi_bvec_done;\n+\tbytes += iter-\u003ebi_offset;\n \n \twhile (bytes \u0026\u0026 bytes \u003e= bv[idx].bv_len) {\n \t\tbytes -= bv[idx].bv_len;\n@@ -212,7 +213,7 @@ static inline bool bvec_iter_advance(const struct bio_vec bv,\n \t}\n \n \titer-\u003ebi_idx = idx;\n-\titer-\u003ebi_bvec_done = bytes;\n+\titer-\u003ebi_offset = bytes;\n \treturn true;\n }\n \n@@ -223,13 +224,13 @@ static inline bool bvec_iter_advance(const struct bio_vec bv,\n static inline void bvec_iter_advance_single(const struct bio_vec bv,\n \t\t\t\tstruct bvec_iter iter, unsigned int bytes)\n {\n-\tunsigned int done = iter-\u003ebi_bvec_done + bytes;\n+\tunsigned int done = iter-\u003ebi_offset + bytes;\n \n \tif (done == bv[iter-\u003ebi_idx].bv_len) {\n \t\tdone = 0;\n \t\titer-\u003ebi_idx++;\n \t}\n-\titer-\u003ebi_bvec_done = done;\n+\titer-\u003ebi_offset = done;\n \titer-\u003ebi_size -= bytes;\n }\n \ndiff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h\nnew file mode 100644\nindex 0000000000000..f31c7375ae911\n--- /dev/null\n+++ b/include/linux/dma-buf-io.h\n@@ -0,0 +1,92 @@\n+/ SPDX-License-Identifier: GPL-2.0 /\n+#ifndef __DMA_BUF_IO_H__\n+#define __DMA_BUF_IO_H__\n+\n+#include \u003clinux/dma-buf.h\u003e\n+\n+struct dma_buf_io_fence;\n+struct dma_buf_io_ctx;\n+struct dma_buf_io_map;\n+\n+struct dma_buf_io_ops {\n+\t/\n+\t * Create a new map for the given ctx. Called with the reservation\n+\t * lock held.\n+\t /\n+\tstruct dma_buf_io_map (map)(struct dma_buf_io_ctx ctx);\n+\n+\t/\n+\t Clean up device specific parts of the @map. Called with the\n+\t * reservation lock held.\n+\t /\n+\tvoid (unmap)(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map);\n+\n+\t/\n+\t The user tries to destroy the ctx. Release all device specific\n+\t * parts of the token.\n+\t /\n+\tvoid (release)(struct dma_buf_io_ctx );\n+};\n+\n+struct dma_buf_io_map {\n+\t/\n+\t * Counts attached requests and other users. Device specific unmapping\n+\t * is deferred until all refs are dropped.\n+\t /\n+\tstruct percpu_ref\t\trefs;\n+\tunsigned\t\t\tseg_shift;\n+\n+\tstruct work_struct\t\trelease_work;\n+\tstruct dma_buf_io_fence\t\tfence;\n+\tstruct dma_buf_io_ctx\t\tctx;\n+};\n+\n+struct dma_buf_io_ctx {\n+\tstruct dma_buf_io_map __rcu\t\tmap;\n+\tstruct dma_buf\t\t\t\tdmabuf;\n+\tenum dma_data_direction\t\t\tdir;\n+\n+\tatomic_t\t\t\t\tfence_seq;\n+\tu64\t\t\t\t\tfence_ctx;\n+\tstruct work_struct\t\t\trelease_work;\n+\trefcount_t\t\t\t\trefs;\n+\n+\tvoid\t\t\t\t\tdev_priv;\n+\tconst struct dma_buf_io_ops\t\tdev_ops;\n+};\n+\n+int dma_buf_io_ctx_create(struct file file,\n+\t\t\t struct dma_buf_io_ctx ctx,\n+\t\t\t struct dma_buf dmabuf,\n+\t\t\t enum dma_data_direction dir);\n+void dma_buf_io_ctx_release(struct dma_buf_io_ctx ctx);\n+\n+struct dma_buf_io_map dma_buf_io_create_map(struct dma_buf_io_ctx ctx);\n+\n+static inline struct dma_buf_io_map \n+dma_buf_io_get_map(struct dma_buf_io_ctx ctx)\n+{\n+\tstruct dma_buf_io_map map;\n+\n+\tguard(rcu)();\n+\n+\tmap = rcu_dereference(ctx-\u003emap);\n+\tif (unlikely(!map \|\| !percpu_ref_tryget_live_rcu(\u0026map-\u003erefs)))\n+\t\treturn NULL;\n+\n+\treturn map;\n+}\n+\n+static inline void dma_buf_io_map_drop(struct dma_buf_io_map map)\n+{\n+\tpercpu_ref_put(\u0026map-\u003erefs);\n+}\n+\n+/\n+ * Device API\n+ /\n+\n+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx ctx);\n+int dma_buf_io_init_map(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map);\n+\n+#endif\ndiff --git a/include/linux/fs.h b/include/linux/fs.h\nindex 50ce731a2b78f..e62fa653e1484 100644\n--- a/include/linux/fs.h\n+++ b/include/linux/fs.h\n@@ -1914,6 +1914,7 @@ struct dir_context {\n #define COPY_FILE_SPLICE\t\t(1 \u003c\u003c 0)\n \n struct io_uring_cmd;\n+struct dma_buf_io_ctx;\n struct offset_ctx;\n \n typedef unsigned int __bitwise fop_flags_t;\n@@ -1962,6 +1963,7 @@ struct file_operations {\n \tint (uring_cmd_iopoll)(struct io_uring_cmd , struct io_comp_batch ,\n \t\t\t\tunsigned int poll_flags);\n \tint (mmap_prepare)(struct vm_area_desc );\n+\tint (init_dma_buf_io_ctx)(struct file , struct dma_buf_io_ctx );\n } __randomize_layout;\n \n /* Supports async buffered reads /\ndiff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h\nindex a2c623a67a251..f90586d353c1c 100644\n--- a/include/linux/io_uring_types.h\n+++ b/include/linux/io_uring_types.h\n@@ -10,6 +10,7 @@\n \n struct iou_loop_params;\n struct io_uring_bpf_ops;\n+struct dma_buf_io_map;\n \n enum {\n \t/\n@@ -594,6 +595,7 @@ enum {\n \tREQ_F_IMPORT_BUFFER_BIT,\n \tREQ_F_SQE_COPIED_BIT,\n \tREQ_F_IOPOLL_BIT,\n+\tREQ_F_DROP_DMABUF_BIT,\n \n \t/* not a real bit, just to check we're not overflowing the space /\n \t__REQ_F_LAST_BIT,\n@@ -689,6 +691,8 @@ enum {\n \tREQ_F_SQE_COPIED\t= IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT),\n \t/ request must be iopolled to completion (set in -\u003eissue()) /\n \tREQ_F_IOPOLL\t\t= IO_REQ_FLAG(REQ_F_IOPOLL_BIT),\n+\t/ there is a dma map attached to request that needs to be dropped /\n+\tREQ_F_DROP_DMABUF\t= IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT),\n };\n \n struct io_tw_req {\n@@ -811,6 +815,7 @@ struct io_kiocb {\n \t/ custom credentials, valid IFF REQ_F_CREDS is set /\n \tconst struct cred\t\tcreds;\n \tstruct io_wq_work\t\twork;\n+\tstruct dma_buf_io_map\t\tdmabuf_map;\n \n \tstruct io_big_cqe {\n \t\tu64\t\t\textra1;\ndiff --git a/include/linux/uio.h b/include/linux/uio.h\nindex fe2e985d74d24..638c1116e912e 100644\n--- a/include/linux/uio.h\n+++ b/include/linux/uio.h\n@@ -12,6 +12,7 @@\n \n struct page;\n struct folio_queue;\n+struct dma_buf_io_map;\n \n typedef unsigned int __bitwise iov_iter_extraction_t;\n \n@@ -29,6 +30,7 @@ enum iter_type {\n \tITER_FOLIOQ,\n \tITER_XARRAY,\n \tITER_DISCARD,\n+\tITER_DMABUF_MAP,\n };\n \n #define ITER_SOURCE\t1\t// == WRITE\n@@ -71,6 +73,7 @@ struct iov_iter {\n \t\t\t\tconst struct folio_queue folioq;\n \t\t\t\tstruct xarray xarray;\n \t\t\t\tvoid __user ubuf;\n+\t\t\t\tstruct dma_buf_io_map dmabuf_map;\n \t\t\t};\n \t\t\tsize_t count;\n \t\t};\n@@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter i)\n \treturn iov_iter_type(i) == ITER_XARRAY;\n }\n \n+static inline bool iov_iter_is_dmabuf_map(const struct iov_iter i)\n+{\n+\treturn iov_iter_type(i) == ITER_DMABUF_MAP;\n+}\n+\n static inline unsigned char iov_iter_rw(const struct iov_iter i)\n {\n \treturn i-\u003edata_source ? WRITE : READ;\n@@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter i, unsigned int direction,\n \t\t\t unsigned int first_slot, unsigned int offset, size_t count);\n void iov_iter_xarray(struct iov_iter i, unsigned int direction, struct xarray xarray,\n \t\t loff_t start, size_t count);\n+void iov_iter_dmabuf_map(struct iov_iter i, unsigned int direction,\n+\t\t\tstruct dma_buf_io_map map,\n+\t\t\tloff_t off, size_t count);\n ssize_t iov_iter_get_pages2(struct iov_iter i, struct page *pages,\n \t\t\tsize_t maxsize, unsigned maxpages, size_t start);\n ssize_t iov_iter_get_pages_alloc2(struct iov_iter i, struct page *pages,\ndiff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h\nindex 909fb7aea638e..c39297e8beb6f 100644\n--- a/include/uapi/linux/io_uring.h\n+++ b/include/uapi/linux/io_uring.h\n@@ -790,13 +790,42 @@ struct io_uring_rsrc_update {\n \n struct io_uring_rsrc_update2 {\n \t__u32 offset;\n-\t__u32 resv;\n+\t__u32 flags;\n \t__aligned_u64 data;\n \t__aligned_u64 tags;\n \t__u32 nr;\n \t__u32 resv2;\n };\n \n+/ struct io_uring_rsrc_update2::flags /\n+enum io_uring_rsrc_reg_flags {\n+\t/\n+\t * Use the extended descriptor format for buffer updates,\n+\t * see struct io_uring_regbuf_desc\n+\t /\n+\tIORING_RSRC_UPDATE_EXTENDED\t\t= 1U \u003c\u003c 1,\n+};\n+\n+/ Buffer registration type, passed in struct io_uring_regbuf_desc::type /\n+enum io_uring_regbuf_type {\n+\tIO_REGBUF_TYPE_EMPTY,\n+\tIO_REGBUF_TYPE_UADDR,\n+\tIO_REGBUF_TYPE_DMABUF,\n+\n+\t__IO_REGBUF_TYPE_MAX,\n+};\n+\n+struct io_uring_regbuf_desc {\n+\t__u32 type; / enum io_uring_regbuf_type /\n+\t__u32 flags;\n+\t__u64 size;\n+\t__u64 uaddr;\n+\n+\t__s32 dmabuf_fd;\n+\t__s32 target_fd;\n+\t__u64 __resv[6];\n+};\n+\n / Skip updating fd indexes set to this value in the fd table /\n #define IORING_REGISTER_FILES_SKIP\t(-2)\n \ndiff --git a/io_uring/io_uring.c b/io_uring/io_uring.c\nindex 4c83a94b4bdc3..d1d748b5c5dca 100644\n--- a/io_uring/io_uring.c\n+++ b/io_uring/io_uring.c\n@@ -109,7 +109,7 @@\n \n #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT \| IO_REQ_LINK_FLAGS \| \\\n \t\t\t\t REQ_F_REISSUE \| REQ_F_POLLED \| \\\n-\t\t\t\t IO_REQ_CLEAN_FLAGS)\n+\t\t\t\t IO_REQ_CLEAN_FLAGS \| REQ_F_DROP_DMABUF)\n \n #define IO_TCTX_REFS_CACHE_NR\t(1U \u003c\u003c 10)\n \n@@ -1114,6 +1114,7 @@ static void io_free_batch_list(struct io_ring_ctx ctx,\n \t\t\t\tio_queue_next(req);\n \t\t\tif (unlikely(req-\u003eflags \u0026 IO_REQ_CLEAN_FLAGS))\n \t\t\t\tio_clean_op(req);\n+\t\t\tio_req_drop_dmabuf(req);\n \t\t}\n \t\tio_put_file(req);\n \t\tio_req_put_rsrc_nodes(req);\ndiff --git a/io_uring/net.c b/io_uring/net.c\nindex a74d15f7b7d2b..439c99ad18841 100644\n--- a/io_uring/net.c\n+++ b/io_uring/net.c\n@@ -1471,7 +1471,7 @@ static int io_sg_from_iter(struct sk_buff skb,\n \t\treturn zerocopy_fill_skb_from_iter(skb, from, length);\n \n \tbi.bi_size = min(from-\u003ecount, length);\n-\tbi.bi_bvec_done = from-\u003eiov_offset;\n+\tbi.bi_offset = from-\u003eiov_offset;\n \tbi.bi_idx = 0;\n \n \twhile (bi.bi_size \u0026\u0026 frag \u003c MAX_SKB_FRAGS) {\n@@ -1490,7 +1490,7 @@ static int io_sg_from_iter(struct sk_buff skb,\n \tfrom-\u003ebvec += bi.bi_idx;\n \tfrom-\u003enr_segs -= bi.bi_idx;\n \tfrom-\u003ecount -= copied;\n-\tfrom-\u003eiov_offset = bi.bi_bvec_done;\n+\tfrom-\u003eiov_offset = bi.bi_offset;\n \n \tskb-\u003edata_len += copied;\n \tskb-\u003elen += copied;\ndiff --git a/io_uring/rsrc.c b/io_uring/rsrc.c\nindex 8d0f2ee24e0c2..5074b1e4b0505 100644\n--- a/io_uring/rsrc.c\n+++ b/io_uring/rsrc.c\n@@ -10,6 +10,7 @@\n #include \u003clinux/compat.h\u003e\n #include \u003clinux/io_uring.h\u003e\n #include \u003clinux/io_uring/cmd.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n \n #include \u003cuapi/linux/io_uring.h\u003e\n \n@@ -28,7 +29,7 @@ struct io_rsrc_update {\n };\n \n static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx,\n-\t\t\t\t\t\t struct iovec iov);\n+\t\t\t\t\t\t struct io_uring_regbuf_desc desc);\n \n static int hpage_acct_ref(struct io_ring_ctx ctx, struct page hpage,\n \t\t\t bool acct_new)\n@@ -81,6 +82,18 @@ static bool hpage_acct_unref(struct io_ring_ctx ctx, struct page hpage)\n \n #define IO_CACHED_BVECS_SEGS\t32\n \n+static void io_iov_to_regbuf_desc(const struct iovec iov,\n+\t\t\t\t struct io_uring_regbuf_desc desc)\n+{\n+\tdesc = (struct io_uring_regbuf_desc) {\n+\t\t.type = IO_REGBUF_TYPE_UADDR,\n+\t\t.uaddr = (u64)(uintptr_t)iov-\u003eiov_base,\n+\t\t.size = iov-\u003eiov_len,\n+\t};\n+\tif (!desc-\u003euaddr)\n+\t\tdesc-\u003etype = IO_REGBUF_TYPE_EMPTY;\n+}\n+\n int __io_account_mem(struct user_struct user, unsigned long nr_pages)\n {\n \tunsigned long page_limit, cur_pages, new_pages;\n@@ -309,6 +322,8 @@ static int __io_sqe_files_update(struct io_ring_ctx ctx,\n \t\treturn -ENXIO;\n \tif (up-\u003eoffset + nr_args \u003e ctx-\u003efile_table.data.nr)\n \t\treturn -EINVAL;\n+\tif (up-\u003eflags)\n+\t\treturn -EINVAL;\n \n \tfor (done = 0; done \u003c nr_args; done++) {\n \t\tu64 tag = 0;\n@@ -368,9 +383,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx,\n \t\t\t\t struct io_uring_rsrc_update2 up,\n \t\t\t\t unsigned int nr_args)\n {\n+\tbool extended = up-\u003eflags \u0026 IORING_RSRC_UPDATE_EXTENDED;\n \tu64 __user tags = u64_to_user_ptr(up-\u003etags);\n-\tstruct iovec fast_iov, iov;\n-\tstruct iovec __user uvec;\n \tu64 user_data = up-\u003edata;\n \t__u32 done;\n \tint i, err;\n@@ -379,26 +393,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx,\n \t\treturn -ENXIO;\n \tif (up-\u003eoffset + nr_args \u003e ctx-\u003ebuf_table.nr)\n \t\treturn -EINVAL;\n+\tif (up-\u003eflags \u0026 ~IORING_RSRC_UPDATE_EXTENDED)\n+\t\treturn -EINVAL;\n \n \tfor (done = 0; done \u003c nr_args; done++) {\n+\t\tstruct io_uring_regbuf_desc desc;\n \t\tstruct io_rsrc_node node;\n \t\tu64 tag = 0;\n \n-\t\tuvec = u64_to_user_ptr(user_data);\n-\t\tiov = iovec_from_user(uvec, 1, 1, \u0026fast_iov, io_is_compat(ctx));\n-\t\tif (IS_ERR(iov)) {\n-\t\t\terr = PTR_ERR(iov);\n-\t\t\tbreak;\n-\t\t}\n \t\tif (tags \u0026\u0026 copy_from_user(\u0026tag, \u0026tags[done], sizeof(tag))) {\n \t\t\terr = -EFAULT;\n \t\t\tbreak;\n \t\t}\n-\t\tnode = io_sqe_buffer_register(ctx, iov);\n+\n+\t\tif (extended) {\n+\t\t\tif (copy_from_user(\u0026desc, u64_to_user_ptr(user_data),\n+\t\t\t\t\t sizeof(desc))) {\n+\t\t\t\terr = -EFAULT;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tuser_data += sizeof(desc);\n+\t\t} else {\n+\t\t\tstruct iovec __user uvec = u64_to_user_ptr(user_data);\n+\t\t\tstruct iovec fast_iov, iov;\n+\n+\t\t\tif (io_is_compat(ctx))\n+\t\t\t\tuser_data += sizeof(struct compat_iovec);\n+\t\t\telse\n+\t\t\t\tuser_data += sizeof(struct iovec);\n+\n+\t\t\tiov = iovec_from_user(uvec, 1, 1, \u0026fast_iov, io_is_compat(ctx));\n+\t\t\tif (IS_ERR(iov)) {\n+\t\t\t\terr = PTR_ERR(iov);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tio_iov_to_regbuf_desc(iov, \u0026desc);\n+\t\t}\n+\n+\t\tnode = io_sqe_buffer_register(ctx, \u0026desc);\n \t\tif (IS_ERR(node)) {\n \t\t\terr = PTR_ERR(node);\n \t\t\tbreak;\n \t\t}\n+\n \t\tif (tag) {\n \t\t\tif (!node) {\n \t\t\t\terr = -EINVAL;\n@@ -409,10 +446,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx,\n \t\ti = array_index_nospec(up-\u003eoffset + done, ctx-\u003ebuf_table.nr);\n \t\tio_reset_rsrc_node(ctx, \u0026ctx-\u003ebuf_table, i);\n \t\tctx-\u003ebuf_table.nodes[i] = node;\n-\t\tif (io_is_compat(ctx))\n-\t\t\tuser_data += sizeof(struct compat_iovec);\n-\t\telse\n-\t\t\tuser_data += sizeof(struct iovec);\n \t}\n \treturn done ? done : err;\n }\n@@ -447,7 +480,7 @@ int io_register_files_update(struct io_ring_ctx ctx, void __user arg,\n \tmemset(\u0026up, 0, sizeof(up));\n \tif (copy_from_user(\u0026up, arg, sizeof(struct io_uring_rsrc_update)))\n \t\treturn -EFAULT;\n-\tif (up.resv \|\| up.resv2)\n+\tif (up.resv2)\n \t\treturn -EINVAL;\n \treturn __io_register_rsrc_update(ctx, IORING_RSRC_FILE, \u0026up, nr_args);\n }\n@@ -461,7 +494,7 @@ int io_register_rsrc_update(struct io_ring_ctx ctx, void __user arg,\n \t\treturn -EINVAL;\n \tif (copy_from_user(\u0026up, arg, sizeof(up)))\n \t\treturn -EFAULT;\n-\tif (!up.nr \|\| up.resv \|\| up.resv2)\n+\tif (!up.nr \|\| up.resv2)\n \t\treturn -EINVAL;\n \treturn __io_register_rsrc_update(ctx, type, \u0026up, up.nr);\n }\n@@ -561,12 +594,9 @@ int io_files_update(struct io_kiocb req, unsigned int issue_flags)\n \tstruct io_uring_rsrc_update2 up2;\n \tint ret;\n \n+\tmemset(\u0026up2, 0, sizeof(up2));\n \tup2.offset = up-\u003eoffset;\n \tup2.data = up-\u003earg;\n-\tup2.nr = 0;\n-\tup2.tags = 0;\n-\tup2.resv = 0;\n-\tup2.resv2 = 0;\n \n \tif (up-\u003eoffset == IORING_FILE_INDEX_ALLOC) {\n \t\tret = io_files_update_with_index_alloc(req, issue_flags);\n@@ -852,27 +882,126 @@ bool io_check_coalesce_buffer(struct page page_array, int nr_pages,\n \treturn true;\n }\n \n+struct io_regbuf_dma {\n+\tstruct dma_buf_io_ctx\t\tctx;\n+\tstruct file\t\t\ttarget_file;\n+};\n+\n+static void io_release_reg_dmabuf(void priv)\n+{\n+\tstruct io_regbuf_dma db = priv;\n+\n+\tfput(db-\u003etarget_file);\n+\tdma_buf_io_ctx_release(\u0026db-\u003ectx);\n+}\n+\n+static struct io_rsrc_node io_register_dmabuf(struct io_ring_ctx ctx,\n+\t\t\t\t\t\tstruct io_uring_regbuf_desc desc)\n+{\n+\tstruct io_rsrc_node node = NULL;\n+\tstruct io_mapped_ubuf imu = NULL;\n+\tstruct io_regbuf_dma regbuf = NULL;\n+\tstruct file target_file = NULL;\n+\tstruct dma_buf dmabuf = NULL;\n+\tint ret;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn ERR_PTR(-EOPNOTSUPP);\n+\tif (ctx-\u003eflags \u0026 IORING_SETUP_IOPOLL)\n+\t\treturn ERR_PTR(-EOPNOTSUPP);\n+\tif (desc-\u003euaddr \|\| desc-\u003esize)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tret = -ENOMEM;\n+\tnode = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);\n+\tif (!node)\n+\t\treturn ERR_PTR(-ENOMEM);\n+\timu = io_alloc_imu(ctx, 0);\n+\tif (!imu)\n+\t\tgoto err;\n+\tregbuf = kzalloc(sizeof(regbuf), GFP_KERNEL);\n+\tif (!regbuf)\n+\t\tgoto err;\n+\n+\tret = -EBADF;\n+\ttarget_file = fget(desc-\u003etarget_fd);\n+\tif (!target_file)\n+\t\tgoto err;\n+\n+\tdmabuf = dma_buf_get(desc-\u003edmabuf_fd);\n+\tif (IS_ERR(dmabuf)) {\n+\t\tret = PTR_ERR(dmabuf);\n+\t\tdmabuf = NULL;\n+\t\tgoto err;\n+\t}\n+\tif (dmabuf-\u003esize \u003e SZ_1G) {\n+\t\tret = -EINVAL;\n+\t\tgoto err;\n+\t}\n+\n+\tret = dma_buf_io_ctx_create(target_file, \u0026regbuf-\u003ectx, dmabuf,\n+\t\t\t\t DMA_BIDIRECTIONAL);\n+\tif (ret)\n+\t\tgoto err;\n+\n+\tregbuf-\u003etarget_file = target_file;\n+\timu-\u003enr_bvecs = 0;\n+\timu-\u003eubuf = 0;\n+\timu-\u003elen = dmabuf-\u003esize;\n+\timu-\u003efolio_shift = 0;\n+\timu-\u003erelease = io_release_reg_dmabuf;\n+\timu-\u003epriv = regbuf;\n+\timu-\u003eflags = IO_REGBUF_F_DMABUF;\n+\timu-\u003edir = IO_IMU_DEST \| IO_IMU_SOURCE;\n+\trefcount_set(\u0026imu-\u003erefs, 1);\n+\tnode-\u003ebuf = imu;\n+\tdma_buf_put(dmabuf);\n+\treturn node;\n+err:\n+\tkfree(regbuf);\n+\tif (imu)\n+\t\tio_free_imu(ctx, imu);\n+\tif (node)\n+\t\tio_cache_free(\u0026ctx-\u003enode_cache, node);\n+\tif (target_file)\n+\t\tfput(target_file);\n+\tif (dmabuf)\n+\t\tdma_buf_put(dmabuf);\n+\treturn ERR_PTR(ret);\n+}\n+\n static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx,\n-\t\t\t\t\t\t struct iovec iov)\n+\t\t\t\t\t\t struct io_uring_regbuf_desc desc)\n {\n+\tunsigned long uaddr = (unsigned long)desc-\u003euaddr;\n+\tsize_t size = desc-\u003esize;\n \tstruct io_mapped_ubuf imu = NULL;\n \tstruct page *pages = NULL;\n \tstruct io_rsrc_node node;\n \tunsigned long off;\n-\tsize_t size;\n \tint ret, nr_pages, i;\n \tstruct io_imu_folio_data data;\n \tbool coalesced = false;\n \n-\tif (!iov-\u003eiov_base) {\n-\t\tif (iov-\u003eiov_len)\n+\tif (desc-\u003etype \u003e= __IO_REGBUF_TYPE_MAX)\n+\t\treturn ERR_PTR(-EINVAL);\n+\tif (!mem_is_zero(\u0026desc-\u003e__resv, sizeof(desc-\u003e__resv)) \|\| desc-\u003eflags)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tif (desc-\u003etype == IO_REGBUF_TYPE_DMABUF)\n+\t\treturn io_register_dmabuf(ctx, desc);\n+\n+\tif (desc-\u003edmabuf_fd \|\| desc-\u003etarget_fd)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tif (desc-\u003etype == IO_REGBUF_TYPE_EMPTY) {\n+\t\tif (uaddr \|\| size)\n \t\t\treturn ERR_PTR(-EFAULT);\n \t\t/* remove the buffer without installing a new one /\n \t\treturn NULL;\n \t}\n \n-\tret = io_validate_user_buf_range((unsigned long)iov-\u003eiov_base,\n-\t\t\t\t\t iov-\u003eiov_len);\n+\tret = io_validate_user_buf_range(uaddr, size);\n \tif (ret)\n \t\treturn ERR_PTR(ret);\n \n@@ -881,8 +1010,7 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx,\n \t\treturn ERR_PTR(-ENOMEM);\n \n \tret = -ENOMEM;\n-\tpages = io_pin_pages((unsigned long) iov-\u003eiov_base, iov-\u003eiov_len,\n-\t\t\t\t\u0026nr_pages);\n+\tpages = io_pin_pages(uaddr, size, \u0026nr_pages);\n \tif (IS_ERR(pages)) {\n \t\tret = PTR_ERR(pages);\n \t\tpages = NULL;\n@@ -904,10 +1032,9 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx,\n \tif (ret)\n \t\tgoto done;\n \n-\tsize = iov-\u003eiov_len;\n \t/ store original address for later verification /\n-\timu-\u003eubuf = (unsigned long) iov-\u003eiov_base;\n-\timu-\u003elen = iov-\u003eiov_len;\n+\timu-\u003eubuf = uaddr;\n+\timu-\u003elen = size;\n \timu-\u003efolio_shift = PAGE_SHIFT;\n \timu-\u003erelease = io_release_ubuf;\n \timu-\u003epriv = imu;\n@@ -917,7 +1044,7 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx,\n \t\timu-\u003efolio_shift = data.folio_shift;\n \trefcount_set(\u0026imu-\u003erefs, 1);\n \n-\toff = (unsigned long)iov-\u003eiov_base \u0026 ~PAGE_MASK;\n+\toff = uaddr \u0026 ~PAGE_MASK;\n \tif (coalesced)\n \t\toff += data.first_folio_page_idx \u003c\u003c PAGE_SHIFT;\n \n@@ -969,6 +1096,7 @@ int io_sqe_buffers_register(struct io_ring_ctx ctx, void __user arg,\n \t\tmemset(iov, 0, sizeof(iov));\n \n \tfor (i = 0; i \u003c nr_args; i++) {\n+\t\tstruct io_uring_regbuf_desc desc;\n \t\tstruct io_rsrc_node node;\n \t\tu64 tag = 0;\n \n@@ -992,7 +1120,8 @@ int io_sqe_buffers_register(struct io_ring_ctx ctx, void __user arg,\n \t\t\t}\n \t\t}\n \n-\t\tnode = io_sqe_buffer_register(ctx, iov);\n+\t\tio_iov_to_regbuf_desc(iov, \u0026desc);\n+\t\tnode = io_sqe_buffer_register(ctx, \u0026desc);\n \t\tif (IS_ERR(node)) {\n \t\t\tret = PTR_ERR(node);\n \t\t\tbreak;\n@@ -1136,9 +1265,64 @@ static int io_import_kbuf(int ddir, struct iov_iter iter,\n \treturn 0;\n }\n \n-static int io_import_fixed(int ddir, struct iov_iter iter,\n+void io_drop_dmabuf_node(struct io_kiocb req)\n+{\n+\tstruct io_mapped_ubuf imu;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(req-\u003ebuf_node-\u003etype != IORING_RSRC_BUFFER))\n+\t\treturn;\n+\timu = req-\u003ebuf_node-\u003ebuf;\n+\tif (WARN_ON_ONCE(!(imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF)))\n+\t\treturn;\n+\tdma_buf_io_map_drop(req-\u003edmabuf_map);\n+\treq-\u003eflags \u0026= ~REQ_F_DROP_DMABUF;\n+}\n+\n+static int io_import_dmabuf(struct io_kiocb req,\n+\t\t\t int ddir, struct iov_iter iter,\n+\t\t\t struct io_mapped_ubuf imu,\n+\t\t\t size_t len, size_t offset,\n+\t\t\t unsigned issue_flags)\n+{\n+\tstruct io_regbuf_dma db = imu-\u003epriv;\n+\tstruct dma_buf_io_map map;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn -EOPNOTSUPP;\n+\tif (!len)\n+\t\treturn -EFAULT;\n+\tif (req-\u003efile != db-\u003etarget_file)\n+\t\treturn -EBADF;\n+\n+\tif (req-\u003eflags \u0026 REQ_F_DROP_DMABUF) {\n+\t\tmap = req-\u003edmabuf_map;\n+\t\tgoto init_iter;\n+\t}\n+\n+\tmap = dma_buf_io_get_map(\u0026db-\u003ectx);\n+\tif (unlikely(!map)) {\n+\t\tif (!(issue_flags \u0026 IO_URING_F_IOWQ))\n+\t\t\treturn -EAGAIN;\n+\t\tmap = dma_buf_io_create_map(\u0026db-\u003ectx);\n+\t\tif (IS_ERR(map))\n+\t\t\treturn PTR_ERR(map);\n+\t}\n+\n+\treq-\u003edmabuf_map = map;\n+\treq-\u003eflags \|= REQ_F_DROP_DMABUF;\n+init_iter:\n+\tiov_iter_dmabuf_map(iter, ddir, map, offset, len);\n+\treturn 0;\n+}\n+\n+static int io_import_fixed(struct io_kiocb req,\n+\t\t\t int ddir, struct iov_iter iter,\n \t\t\t struct io_mapped_ubuf imu,\n-\t\t\t u64 buf_addr, size_t len)\n+\t\t\t u64 buf_addr, size_t len,\n+\t\t\t unsigned issue_flags,\n+\t\t\t unsigned import_flags)\n {\n \tconst struct bio_vec bvec;\n \tsize_t folio_mask;\n@@ -1158,6 +1342,12 @@ static int io_import_fixed(int ddir, struct iov_iter iter,\n \n \toffset = buf_addr - imu-\u003eubuf;\n \n+\tif (imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF) {\n+\t\tif (!(import_flags \u0026 IO_REGBUF_IMPORT_ALLOW_DMABUF))\n+\t\t\treturn -EFAULT;\n+\t\treturn io_import_dmabuf(req, ddir, iter, imu, len, offset,\n+\t\t\t\t\tissue_flags);\n+\t}\n \tif (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF)\n \t\treturn io_import_kbuf(ddir, iter, imu, len, offset);\n \n@@ -1211,16 +1401,17 @@ inline struct io_rsrc_node io_find_buf_node(struct io_kiocb req,\n \treturn NULL;\n }\n \n-int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter,\n+int __io_import_reg_buf(struct io_kiocb req, struct iov_iter iter,\n \t\t\tu64 buf_addr, size_t len, int ddir,\n-\t\t\tunsigned issue_flags)\n+\t\t\tunsigned issue_flags, unsigned import_flags)\n {\n \tstruct io_rsrc_node node;\n \n \tnode = io_find_buf_node(req, issue_flags);\n \tif (!node)\n \t\treturn -EFAULT;\n-\treturn io_import_fixed(ddir, iter, node-\u003ebuf, buf_addr, len);\n+\treturn io_import_fixed(req, ddir, iter, node-\u003ebuf, buf_addr, len,\n+\t\t\t\tissue_flags, import_flags);\n }\n \n static int io_buffer_acct_cloned_hpages(struct io_ring_ctx ctx,\n@@ -1347,6 +1538,11 @@ static int io_clone_buffers(struct io_ring_ctx ctx, struct io_ring_ctx src_ctx\n \t\tif (!src_node) {\n \t\t\tdst_node = NULL;\n \t\t} else {\n+\t\t\tif (src_node-\u003ebuf-\u003eflags \u0026 IO_REGBUF_F_UNCLONEABLE) {\n+\t\t\t\tio_rsrc_data_free(ctx, \u0026data);\n+\t\t\t\treturn -ENOMEM;\n+\t\t\t}\n+\n \t\t\tdst_node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);\n \t\t\tif (!dst_node) {\n \t\t\t\tio_rsrc_data_free(ctx, \u0026data);\n@@ -1617,9 +1813,9 @@ static int io_kern_bvec_size(struct iovec iov, unsigned nr_iovs,\n \treturn 0;\n }\n \n-int io_import_reg_vec(int ddir, struct iov_iter iter,\n+int __io_import_reg_vec(int ddir, struct iov_iter iter,\n \t\t\tstruct io_kiocb req, struct iou_vec vec,\n-\t\t\tunsigned nr_iovs, unsigned issue_flags)\n+\t\t\tunsigned nr_iovs, unsigned issue_flags, unsigned import_flags)\n {\n \tstruct io_rsrc_node node;\n \tstruct io_mapped_ubuf imu;\n@@ -1637,7 +1833,9 @@ int io_import_reg_vec(int ddir, struct iov_iter iter,\n \tiovec_off = vec-\u003enr - nr_iovs;\n \tiov = vec-\u003eiovec + iovec_off;\n \n-\tif (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF) {\n+\tif (imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF) {\n+\t\treturn -EOPNOTSUPP;\n+\t} else if (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF) {\n \t\tint ret = io_kern_bvec_size(iov, nr_iovs, imu, \u0026nr_segs);\n \n \t\tif (unlikely(ret))\ndiff --git a/io_uring/rsrc.h b/io_uring/rsrc.h\nindex 98ae8ef51009d..3cb9b57f159b6 100644\n--- a/io_uring/rsrc.h\n+++ b/io_uring/rsrc.h\n@@ -29,7 +29,13 @@ enum {\n };\n \n enum {\n-\tIO_REGBUF_F_KBUF\t\t= 1,\n+\tIO_REGBUF_F_KBUF\t\t= 1 \u003c\u003c 0,\n+\tIO_REGBUF_F_UNCLONEABLE\t\t= 1 \u003c\u003c 1,\n+\tIO_REGBUF_F_DMABUF\t\t= 1 \u003c\u003c 3,\n+};\n+\n+enum {\n+\tIO_REGBUF_IMPORT_ALLOW_DMABUF\t\t= 1 \u003c\u003c 1,\n };\n \n struct io_mapped_ubuf {\n@@ -64,12 +70,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data data, unsigned nr);\n \n struct io_rsrc_node io_find_buf_node(struct io_kiocb req,\n \t\t\t\t unsigned issue_flags);\n-int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter,\n+int __io_import_reg_buf(struct io_kiocb req, struct iov_iter iter,\n \t\t\tu64 buf_addr, size_t len, int ddir,\n-\t\t\tunsigned issue_flags);\n-int io_import_reg_vec(int ddir, struct iov_iter iter,\n+\t\t\tunsigned issue_flags, unsigned import_flags);\n+int __io_import_reg_vec(int ddir, struct iov_iter iter,\n \t\t\tstruct io_kiocb req, struct iou_vec vec,\n-\t\t\tunsigned nr_iovs, unsigned issue_flags);\n+\t\t\tunsigned nr_iovs, unsigned issue_flags,\n+\t\t\tunsigned import_flags);\n+\n+static inline int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter,\n+\t\t\t\t u64 buf_addr, size_t len, int ddir,\n+\t\t\t\t unsigned issue_flags)\n+{\n+\treturn __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0);\n+}\n+\n+static inline int io_import_reg_vec(int ddir, struct iov_iter iter,\n+\t\t\t\t struct io_kiocb req, struct iou_vec vec,\n+\t\t\t\t unsigned nr_iovs, unsigned issue_flags)\n+{\n+\treturn __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0);\n+}\n+\n int io_prep_reg_iovec(struct io_kiocb req, struct iou_vec iv,\n \t\t\tconst struct iovec __user uvec, size_t uvec_segs);\n \n@@ -156,4 +178,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec iv)\n \t\tio_vec_free(iv);\n }\n \n+void io_drop_dmabuf_node(struct io_kiocb req);\n+\n+static inline void io_req_drop_dmabuf(struct io_kiocb req)\n+{\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn;\n+\tif (!(req-\u003eflags \u0026 REQ_F_DROP_DMABUF))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(!(req-\u003eflags \u0026 REQ_F_BUF_NODE)))\n+\t\treturn;\n+\tio_drop_dmabuf_node(req);\n+}\n+\n #endif\ndiff --git a/io_uring/rw.c b/io_uring/rw.c\nindex 95038cfda6153..11ff7b5a11647 100644\n--- a/io_uring/rw.c\n+++ b/io_uring/rw.c\n@@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb req, unsigned int issue_flags,\n \tif (io-\u003ebytes_done)\n \t\treturn 0;\n \n-\tret = io_import_reg_buf(req, \u0026io-\u003eiter, rw-\u003eaddr, rw-\u003elen, ddir,\n-\t\t\t\tissue_flags);\n+\tret = __io_import_reg_buf(req, \u0026io-\u003eiter, rw-\u003eaddr, rw-\u003elen, ddir,\n+\t\t\t\t issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF);\n \tiov_iter_save_state(\u0026io-\u003eiter, \u0026io-\u003eiter_state);\n \treturn ret;\n }\n@@ -591,6 +591,8 @@ static void io_complete_rw(struct kiocb kiocb, long res)\n \tstruct io_rw rw = container_of(kiocb, struct io_rw, kiocb);\n \tstruct io_kiocb req = cmd_to_io_kiocb(rw);\n \n+\tio_req_drop_dmabuf(req);\n+\n \t__io_complete_rw_common(req, res);\n \tio_req_set_res(req, io_fixup_rw_res(req, res), 0);\n \treq-\u003eio_task_work.func = io_req_rw_complete;\ndiff --git a/lib/iov_iter.c b/lib/iov_iter.c\nindex 34a52e9ba9e1b..d6f5d7e180b54 100644\n--- a/lib/iov_iter.c\n+++ b/lib/iov_iter.c\n@@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter i, size_t size)\n {\n \tif (unlikely(i-\u003ecount \u003c size))\n \t\tsize = i-\u003ecount;\n-\tif (likely(iter_is_ubuf(i)) \|\| unlikely(iov_iter_is_xarray(i))) {\n+\tif (likely(iter_is_ubuf(i)) \|\| unlikely(iov_iter_is_xarray(i)) \|\|\n+\t unlikely(iov_iter_is_dmabuf_map(i))) {\n \t\ti-\u003eiov_offset += size;\n \t\ti-\u003ecount -= size;\n \t} else if (likely(iter_is_iovec(i) \|\| iov_iter_is_kvec(i))) {\n@@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter i, size_t unroll)\n \t\treturn;\n \t}\n \tunroll -= i-\u003eiov_offset;\n-\tif (iov_iter_is_xarray(i) \|\| iter_is_ubuf(i)) {\n+\tif (iov_iter_is_xarray(i) \|\| iter_is_ubuf(i) \|\|\n+\t iov_iter_is_dmabuf_map(i)) {\n \t\tBUG(); / We should never go beyond the start of the specified\n \t\t\t* range since we might then be straying into pages that\n \t\t\t* aren't pinned.\n@@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter i, unsigned int direction,\n }\n EXPORT_SYMBOL(iov_iter_xarray);\n \n+void iov_iter_dmabuf_map(struct iov_iter i, unsigned int direction,\n+\t\t\t struct dma_buf_io_map map,\n+\t\t\t loff_t off, size_t count)\n+{\n+\tWARN_ON(direction \u0026 ~(READ \| WRITE));\n+\ti = (struct iov_iter){\n+\t\t.iter_type = ITER_DMABUF_MAP,\n+\t\t.data_source = direction,\n+\t\t.dmabuf_map = map,\n+\t\t.count = count,\n+\t\t.iov_offset = off,\n+\t};\n+}\n+\n /*\n iov_iter_discard - Initialise an I/O iterator that discards data\n * @i: The iterator to initialise.\n@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter i)\n \n unsigned long iov_iter_alignment(const struct iov_iter i)\n {\n-\tif (likely(iter_is_ubuf(i))) {\n+\tif (likely(iter_is_ubuf(i)) \|\| iov_iter_is_dmabuf_map(i)) {\n \t\tsize_t size = i-\u003ecount;\n \t\tif (size)\n \t\t\treturn ((unsigned long)i-\u003eubuf + i-\u003eiov_offset) \| size;\n@@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter i)\n \tsize_t size = i-\u003ecount;\n \tunsigned k;\n \n-\tif (iter_is_ubuf(i))\n+\tif (iter_is_ubuf(i) \|\| iov_iter_is_dmabuf_map(i))\n \t\treturn 0;\n \n \tif (WARN_ON(!iter_is_iovec(i)))\n@@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf);\n void iov_iter_restore(struct iov_iter i, struct iov_iter_state state)\n {\n \tif (WARN_ON_ONCE(!iov_iter_is_bvec(i) \u0026\u0026 !iter_is_iovec(i) \u0026\u0026\n-\t\t\t !iter_is_ubuf(i)) \u0026\u0026 !iov_iter_is_kvec(i))\n+\t\t\t !iter_is_ubuf(i) \u0026\u0026 !iov_iter_is_kvec(i) \u0026\u0026\n+\t\t\t !iov_iter_is_dmabuf_map(i)))\n \t\treturn;\n \ti-\u003eiov_offset = state-\u003eiov_offset;\n \ti-\u003ecount = state-\u003ecount;\n-\tif (iter_is_ubuf(i))\n+\tif (iter_is_ubuf(i) \|\| iov_iter_is_dmabuf_map(i))\n \t\treturn;\n \t/\n \t * For the vec iters, nr_segs + iov is constant - if we increment\n@@ -1634,7 +1651,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter i,\n \t}\n \tbi.bi_idx = 0;\n \tbi.bi_size = maxsize;\n-\tbi.bi_bvec_done = skip;\n+\tbi.bi_offset = skip;\n \n \tmaxpages = want_pages_array(pages, maxsize, skip, maxpages);\n \tif (!maxpages)\ndiff --git a/net/ceph/messenger.c b/net/ceph/messenger.c\nindex 34b3097b4c7b9..9c1b6cf8c36fe 100644\n--- a/net/ceph/messenger.c\n+++ b/net/ceph/messenger.c\n@@ -762,7 +762,7 @@ static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor cursor,\n \tif (!cursor-\u003eresid)\n \t\treturn false; / no more data /\n \n-\tif (!bytes \|\| (it-\u003eiter.bi_size \u0026\u0026 it-\u003eiter.bi_bvec_done \u0026\u0026\n+\tif (!bytes \|\| (it-\u003eiter.bi_size \u0026\u0026 it-\u003eiter.bi_offset \u0026\u0026\n \t\t page == bio_iter_page(it-\u003ebio, it-\u003eiter)))\n \t\treturn false;\t/ more bytes to process in this segment /\n \n@@ -817,7 +817,7 @@ static bool ceph_msg_data_bvecs_advance(struct ceph_msg_data_cursor cursor,\n \tif (!cursor-\u003eresid)\n \t\treturn false; /* no more data /\n \n-\tif (!bytes \|\| (cursor-\u003ebvec_iter.bi_bvec_done \u0026\u0026\n+\tif (!bytes \|\| (cursor-\u003ebvec_iter.bi_offset \u0026\u0026\n \t\t page == bvec_iter_page(bvecs, cursor-\u003ebvec_iter)))\n \t\treturn false;\t/ more bytes to process in this segment */\n \n" }
2/1	2026/08/02 14:37	agent	patch-evaluator	0m Model: best-expensive Args: null Results: { "EnableConfigs": [ "DMA_SHARED_BUFFER", "IO_URING", "BLK_DEV_NVME" ], "FocusSymbols": [ "dma_buf_io_ctx_create", "dma_buf_io_ctx_release", "dma_buf_io_create_map", "dma_buf_io_invalidate_mappings", "dma_buf_io_init_map", "bio_iov_iter_set", "__io_import_reg_buf", "io_drop_dmabuf_node", "iov_iter_dmabuf_map" ], "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit 2cea66d19f5e61596d4d0326aee5752566f75bc7 Author: syz-cluster <triage@syzkaller.com> Date: Sun Aug 2 14:37:35 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/block/biovecs.rst b/Documentation/block/biovecs.rst index 11126ed6f40f7..49da2147b7d86 100644 --- a/Documentation/block/biovecs.rst +++ b/Documentation/block/biovecs.rst @@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec. In the new scheme of things, everything that must be mutated in order to partially complete a bio is segregated into struct bvec_iter: bi_sector, bi_size and bi_idx have been moved there; and instead of modifying bv_offset -and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of +and bv_len, struct bvec_iter has bi_offset, which represents the number of bytes completed in the current bvec. There are a bunch of new helper macros for hiding the gory details - in particular, presenting the illusion of partially completed biovecs so that -normal code doesn't have to deal with bi_bvec_done. +normal code doesn't have to deal with bi_offset. * Driver code should no longer refer to biovecs directly; we now have bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs, - constructed from the raw biovecs but taking into account bi_bvec_done and + constructed from the raw biovecs but taking into account bi_offset and bi_size. bio_for_each_segment() has been updated to take a bvec_iter argument @@ -101,7 +101,7 @@ Other implications: I.e. instead of using bio_iovec_idx() (or bio->bi_iovec[bio->bi_idx]), you now use bio_iter_iovec(), which takes a bvec_iter and returns a literal struct bio_vec - constructed on the fly from the raw biovec but - taking into account bi_bvec_done (and bi_size). + taking into account bi_offset (and bi_size). * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that doesn't actually own the bio. The reason is twofold: firstly, it's not diff --git a/block/bio.c b/block/bio.c index 500389f332d99..1602eab05761f 100644 --- a/block/bio.c +++ b/block/bio.c @@ -229,7 +229,7 @@ void bio_init(struct bio bio, struct block_device bdev, struct bio_vec table, bio->bi_iter.bi_sector = 0; bio->bi_iter.bi_size = 0; bio->bi_iter.bi_idx = 0; - bio->bi_iter.bi_bvec_done = 0; + bio->bi_iter.bi_offset = 0; bio->bi_end_io = NULL; bio->bi_private = NULL; #ifdef CONFIG_BLK_CGROUP @@ -861,6 +861,11 @@ static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp) bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter = bio_src->bi_iter; + if (op_is_dmabuf(bio->bi_opf)) + bio->bi_dmabuf_map = bio_src->bi_dmabuf_map; + else + bio->bi_io_vec = bio_src->bi_io_vec; + if (bio->bi_bdev) { if (bio->bi_bdev == bio_src->bi_bdev && bio_flagged(bio_src, BIO_REMAPPED)) @@ -902,8 +907,6 @@ struct bio bio_alloc_clone(struct block_device bdev, struct bio bio_src, bio_put(bio); return NULL; } - bio->bi_io_vec = bio_src->bi_io_vec; - return bio; } EXPORT_SYMBOL(bio_alloc_clone); @@ -923,7 +926,7 @@ int bio_init_clone(struct block_device bdev, struct bio bio, { int ret; - bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf); + bio_init(bio, bdev, NULL, 0, bio_src->bi_opf); ret = __bio_clone(bio, bio_src, gfp); if (ret) bio_uninit(bio); @@ -1182,15 +1185,26 @@ void __bio_release_pages(struct bio bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(__bio_release_pages); -void bio_iov_bvec_set(struct bio bio, const struct iov_iter iter) +bool bio_iov_iter_set(struct bio bio, const struct iov_iter iter) { + if (!iov_iter_is_bvec(iter) && !iov_iter_is_dmabuf_map(iter)) + return false; + WARN_ON_ONCE(bio->bi_max_vecs); + static_assert(offsetof(struct bio, bi_io_vec) == + offsetof(struct bio, bi_dmabuf_map)); + static_assert(offsetof(struct iov_iter, bvec) == + offsetof(struct iov_iter, dmabuf_map)); + bio->bi_io_vec = (struct bio_vec )iter->bvec; bio->bi_iter.bi_idx = 0; - bio->bi_iter.bi_bvec_done = iter->iov_offset; + bio->bi_iter.bi_offset = iter->iov_offset; bio->bi_iter.bi_size = iov_iter_count(iter); bio_set_flag(bio, BIO_CLONED); + if (iov_iter_is_dmabuf_map(iter)) + bio->bi_opf \|= REQ_NOMERGE \| REQ_DMABUF; + return true; } /* @@ -1285,10 +1299,9 @@ int bio_iov_iter_get_pages(struct bio bio, struct iov_iter iter, if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED))) return -EIO; - if (iov_iter_is_bvec(iter)) { - bio_iov_bvec_set(bio, iter); - - if (!bio_iov_bvec_aligned(bio, mem_align_mask)) + if (bio_iov_iter_set(bio, iter)) { + if (iov_iter_is_bvec(iter) && + !bio_iov_bvec_aligned(bio, mem_align_mask)) return -EINVAL; iov_iter_advance(iter, bio->bi_iter.bi_size); diff --git a/block/blk-map.c b/block/blk-map.c index 615d29bb840eb..9cb9605d1f62d 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request rq, const struct iov_iter iter) bio = blk_rq_map_bio_alloc(rq, 0, GFP_KERNEL); if (!bio) return -ENOMEM; - bio_iov_bvec_set(bio, iter); + bio_iov_iter_set(bio, iter); ret = blk_rq_append_bio(rq, bio); if (ret) diff --git a/block/blk-merge.c b/block/blk-merge.c index ab1161ca69f1e..1beedc42a85d5 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -9,6 +9,7 @@ #include <linux/blk-integrity.h> #include <linux/part_stat.h> #include <linux/blk-cgroup.h> +#include <linux/dma-buf-io.h> #include <trace/events/block.h> @@ -33,7 +34,7 @@ static inline void bio_get_last_bvec(struct bio bio, struct bio_vec bv) bio_advance_iter(bio, &iter, iter.bi_size); - if (!iter.bi_bvec_done) + if (!iter.bi_offset) idx = iter.bi_idx - 1; else /* in the middle of bvec / idx = iter.bi_idx; @@ -41,11 +42,11 @@ static inline void bio_get_last_bvec(struct bio bio, struct bio_vec bv) bv = bio->bi_io_vec[idx]; /* - * iter.bi_bvec_done records actual length of the last bvec + * iter.bi_offset records actual length of the last bvec * if this bio ends in the middle of one io vector / - if (iter.bi_bvec_done) - bv->bv_len = iter.bi_bvec_done; + if (iter.bi_offset) + bv->bv_len = iter.bi_offset; } static inline bool bio_will_gap(struct request_queue q, @@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec bvprv, return bv->bv_offset \| (bvprv->bv_offset + bvprv->bv_len); } +static inline int bio_split_io_at_dmabuf(struct bio bio, + const struct queue_limits lim, unsigned segs, + unsigned max_bytes, unsigned len_align_mask, + unsigned start_align_mask) +{ + unsigned bytes = min(bio->bi_iter.bi_size, max_bytes); + unsigned seg_shift = bio->bi_dmabuf_map->seg_shift; + unsigned offset = bio->bi_iter.bi_offset & ((1U << seg_shift) - 1); + + if ((bio->bi_iter.bi_offset & start_align_mask) \|\| + (bio->bi_iter.bi_size & len_align_mask)) + return -EINVAL; + + /* single contiguous range into the dma-buf / + segs = 1; + + bytes = min(bytes, ((unsigned)lim->max_segments << seg_shift) - offset); + if (bytes != bio->bi_iter.bi_size) + return bytes; + return 0; +} + /** * bio_split_io_at - check if and where to split a bio * @bio: [in] bio to be split @@ -346,6 +369,19 @@ int bio_split_io_at(struct bio bio, const struct queue_limits lim, len_align_mask \|= (bc->bc_key->crypto_cfg.data_unit_size - 1); } + if (op_is_dmabuf(bio->bi_opf)) { + int ret; + + ret = bio_split_io_at_dmabuf(bio, lim, &nsegs, max_bytes, + len_align_mask, start_align_mask); + if (ret < 0) + return ret; + if (!ret) + goto out; + bytes = ret; + goto split; + } + bio_for_each_bvec(bv, bio, iter) { if (bv.bv_offset & start_align_mask \|\| bv.bv_len & len_align_mask) @@ -376,6 +412,7 @@ int bio_split_io_at(struct bio bio, const struct queue_limits lim, bvprvp = &bvprv; } +out: segs = nsegs; bio->bi_bvec_gap_bit = ffs(gaps); return 0; diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c index bfdb9ed707411..88fd9cbc951f4 100644 --- a/block/blk-mq-dma.c +++ b/block/blk-mq-dma.c @@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request req, struct blk_map_iter iter, one could be merged into it. This typically happens when moving to * the next bio, but some callers also don't pack bvecs tight. / - while (!iter->iter.bi_size \|\| !iter->iter.bi_bvec_done) { + while (!iter->iter.bi_size \|\| !iter->iter.bi_offset) { struct bio_vec next; if (!__blk_map_iter_next(iter)) diff --git a/block/blk.h b/block/blk.h index eaac05815cb03..50abfd9328861 100644 --- a/block/blk.h +++ b/block/blk.h @@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio bio, return true; bv = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); - if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done) + if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_offset) return true; if ((bv->bv_offset \| bv->bv_len) & lim->dma_alignment) return true; diff --git a/block/fops.c b/block/fops.c index 3c2099dfef1d5..3c403afb0f646 100644 --- a/block/fops.c +++ b/block/fops.c @@ -342,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb iocb, bio->bi_end_io = blkdev_bio_end_io_async; bio->bi_ioprio = iocb->ki_ioprio; - if (iov_iter_is_bvec(iter)) { - / - * Users don't rely on the iterator being in any particular - * state for async I/O returning -EIOCBQUEUED, hence we can - * avoid expensive iov_iter_advance(). Bypass - * bio_iov_iter_get_pages() and set the bvec directly. - / - bio_iov_bvec_set(bio, iter); - } else { + / + * Users don't rely on the iterator being in any particular + * state for async I/O returning -EIOCBQUEUED, hence we can + * avoid expensive iov_iter_advance(). Bypass + * bio_iov_iter_get_pages() and set the bvec/dmabuf directly. + / + if (!bio_iov_iter_set(bio, iter)) { ret = blkdev_iov_iter_get_pages(bio, iter, bdev); if (unlikely(ret)) goto out_bio_put; @@ -769,6 +767,9 @@ static ssize_t blkdev_write_iter(struct kiocb iocb, struct iov_iter from) ret = direct_write_fallback(iocb, from, ret, blkdev_buffered_write(iocb, from)); } else { + if (unlikely(iov_iter_is_dmabuf_map(from))) + return -EOPNOTSUPP; + / * Take i_rwsem and invalidate_lock to avoid racing with * set_blocksize changing i_blkbits/folio order and punching @@ -823,6 +824,8 @@ static ssize_t blkdev_read_iter(struct kiocb iocb, struct iov_iter to) if (ret < 0 \|\| !count) goto reexpand; } + if (unlikely(iov_iter_is_dmabuf_map(to))) + return -EOPNOTSUPP; /* * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize @@ -926,6 +929,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc desc) return generic_file_mmap_prepare(desc); } +static int blkdev_init_dma_buf_io_ctx(struct file file, + struct dma_buf_io_ctx ctx) +{ + struct block_device bdev = file_bdev(file); + struct gendisk disk = bdev->bd_disk; + + if (!(file->f_flags & O_DIRECT)) + return -EINVAL; + if (!disk->fops->init_dma_buf_io_ctx) + return -EINVAL; + return disk->fops->init_dma_buf_io_ctx(bdev, ctx); +} + const struct file_operations def_blk_fops = { .open = blkdev_open, .release = blkdev_release, @@ -944,6 +960,7 @@ const struct file_operations def_blk_fops = { .fallocate = blkdev_fallocate, .uring_cmd = blkdev_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC \| FOP_DONTCACHE, + .init_dma_buf_io_ctx = blkdev_init_dma_buf_io_ctx, }; static __init int blkdev_init(void) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 26d7130c3f559..8639fa34b8470 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -379,7 +379,7 @@ static int lo_rw_aio(struct loop_device lo, struct loop_cmd cmd, iov_iter_bvec(&iter, rw, __bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter), nr_bvec, blk_rq_bytes(rq)); - iter.iov_offset = rq->bio->bi_iter.bi_bvec_done; + iter.iov_offset = rq->bio->bi_iter.bi_offset; } atomic_set(&cmd->ref, 2); diff --git a/drivers/block/zloop.c b/drivers/block/zloop.c index f97a20cfdb7ce..0fef5db8e5b80 100644 --- a/drivers/block/zloop.c +++ b/drivers/block/zloop.c @@ -554,7 +554,7 @@ static int zloop_do_rw(struct zloop_cmd cmd) iov_iter_bvec(&iter, rw, __bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter), nr_bvec, blk_rq_bytes(rq)); - iter.iov_offset = rq->bio->bi_iter.bi_bvec_done; + iter.iov_offset = rq->bio->bi_iter.bi_offset; } cmd->iocb.ki_pos = (cmd->sector - zone->start) << SECTOR_SHIFT; diff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile index b25d7550bacfd..523731b0f83ee 100644 --- a/drivers/dma-buf/Makefile +++ b/drivers/dma-buf/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o + dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) += heaps/ obj-$(CONFIG_SYNC_FILE) += sync_file.o diff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c new file mode 100644 index 0000000000000..24f58f80c45c4 --- /dev/null +++ b/drivers/dma-buf/dma-buf-io.c @@ -0,0 +1,278 @@ +/* SPDX-License-Identifier: GPL-2.0 / +/ + * Common infrastructure for supporing dma-buf in the I/O path. + * + * Copyright (C) 2026 Pavel Begunkov <asml.silence@gmail.com> + / +#include <linux/dma-buf-io.h> +#include <linux/dma-resv.h> + +struct dma_buf_io_fence { + struct dma_fence base; + spinlock_t lock; +}; + +static const char dma_buf_io_fence_drv_name(struct dma_fence fence) +{ + / default fence release kfree's the base pointer / + BUILD_BUG_ON(offsetof(struct dma_buf_io_fence, base)); + + return "dma-buf-io-ctx"; +} + +static const char dma_buf_io_fence_timeline_name(struct dma_fence fence) +{ + return "dma-buf-io-ctx"; +} + +const struct dma_fence_ops dma_buf_io_fence_ops = { + .get_driver_name = dma_buf_io_fence_drv_name, + .get_timeline_name = dma_buf_io_fence_timeline_name, +}; + +static void dma_buf_io_ctx_destroy_work(struct work_struct work) +{ + struct dma_buf_io_ctx ctx = container_of(work, struct dma_buf_io_ctx, + release_work); + + if (WARN_ON_ONCE(refcount_read(&ctx->refs))) + return; + + ctx->dev_ops->release(ctx); + dma_buf_put(ctx->dmabuf); + kfree(ctx); +} + +static void dma_buf_io_map_release_work(struct work_struct work) +{ + struct dma_buf_io_map map = container_of(work, struct dma_buf_io_map, + release_work); + struct dma_buf_io_fence fence = map->fence; + struct dma_buf_io_ctx ctx = map->ctx; + struct dma_buf dmabuf = ctx->dmabuf; + + /* the release path must wait for fences / + if (WARN_ON_ONCE(refcount_read(&ctx->refs) == 0)) + return; + + / Prevent from destoying the ctx while unmapping / + refcount_inc(&ctx->refs); + + / + * There are no more requests using the map, we can signal the fence. + * It should be done before taking the resv lock as someone could be + * waiting for the fence while holding the lock. + / + dma_fence_signal(&fence->base); + + dma_resv_lock(dmabuf->resv, NULL); + ctx->dev_ops->unmap(ctx, map); + dma_resv_unlock(dmabuf->resv); + + dma_fence_put(&fence->base); + percpu_ref_exit(&map->refs); + kfree(map); + + if (refcount_dec_and_test(&ctx->refs)) { + / + * Destruction needs to wait for I/O and dma fences. Defer it to + * simplify locking. + / + INIT_WORK(&ctx->release_work, dma_buf_io_ctx_destroy_work); + queue_work(system_wq, &ctx->release_work); + } +} + +static void dma_buf_io_map_refs_release(struct percpu_ref ref) +{ + struct dma_buf_io_map map = container_of(ref, struct dma_buf_io_map, refs); + + / might sleep, use a worker / + INIT_WORK(&map->release_work, dma_buf_io_map_release_work); + queue_work(system_wq, &map->release_work); +} + +int dma_buf_io_init_map(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map) +{ + struct dma_buf_io_fence fence = NULL; + int ret; + + fence = kzalloc(sizeof(fence), GFP_KERNEL); + if (!fence) + return -ENOMEM; + + ret = percpu_ref_init(&map->refs, dma_buf_io_map_refs_release, 0, GFP_KERNEL); + if (ret) { + kfree(fence); + return ret; + } + + spin_lock_init(&fence->lock); + dma_fence_init(&fence->base, &dma_buf_io_fence_ops, &fence->lock, + ctx->fence_ctx, atomic_inc_return(&ctx->fence_seq)); + map->fence = fence; + map->ctx = ctx; + return 0; +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, "DMA_BUF"); + +struct dma_buf_io_map dma_buf_io_create_map(struct dma_buf_io_ctx ctx) +{ + struct dma_buf dmabuf = ctx->dmabuf; + struct dma_buf_io_map map; + long ret; + +retry: + / + * ->dmabuf_map() will be calling dma_buf_map_attachment(), for which + * we'll need to wait for fences. Do a bit nicer and try to wait + * without the resv lock first. + / + ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL, + true, MAX_SCHEDULE_TIMEOUT); + if (!ret) + ret = -EAGAIN; + if (ret < 0) + return ERR_PTR(ret); + + ret = dma_resv_lock_interruptible(dmabuf->resv, NULL); + if (ret) + return ERR_PTR(ret); + + map = dma_buf_io_get_map(ctx); + if (map) { + ret = 0; + goto out; + } + + if (dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL, + true, 0) < 0) { + dma_resv_unlock(dmabuf->resv); + goto retry; + } + + map = ctx->dev_ops->map(ctx); + if (IS_ERR(map)) { + ret = PTR_ERR(map); + goto out; + } + + if (WARN_ON_ONCE(!map->seg_shift)) + return ERR_PTR(-EFAULT); + + percpu_ref_get(&map->refs); + rcu_assign_pointer(ctx->map, map); +out: + dma_resv_unlock(dmabuf->resv); + if (ret < 0) + return ERR_PTR(ret); + return map; +} + +static void dma_buf_io_drop_map(struct dma_buf_io_ctx ctx) +{ + struct dma_buf dmabuf = ctx->dmabuf; + struct dma_buf_io_map map; + int ret; + + dma_resv_assert_held(dmabuf->resv); + + map = rcu_dereference_protected(ctx->map, + dma_resv_held(dmabuf->resv)); + if (!map) + return; + rcu_assign_pointer(ctx->map, NULL); + + ret = dma_resv_reserve_fences(dmabuf->resv, 1); + if (WARN_ON_ONCE(ret)) { + struct dma_fence fence = &map->fence->base; + + dma_fence_get(fence); + percpu_ref_kill(&map->refs); + dma_fence_wait(fence, false); + dma_fence_put(fence); + return; + } + + dma_resv_add_fence(dmabuf->resv, &map->fence->base, + DMA_RESV_USAGE_KERNEL); + / + * Delay destruction until all inflight requests using the map are + * gone. It'll also signal the fence then. + / + percpu_ref_kill(&map->refs); +} + +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx ctx) +{ + dma_buf_io_drop_map(ctx); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, "DMA_BUF"); + +static void dma_buf_io_ctx_release_work(struct work_struct work) +{ + struct dma_buf_io_ctx ctx = container_of(work, struct dma_buf_io_ctx, + release_work); + struct dma_buf dmabuf = ctx->dmabuf; + long ret; + + dma_resv_lock(dmabuf->resv, NULL); + / Remove the last map, there should be no new ones going forward. / + dma_buf_io_drop_map(ctx); + dma_resv_unlock(dmabuf->resv); + + / Wait until all maps are destroyed. / + ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL, + false, MAX_SCHEDULE_TIMEOUT); + + if (WARN_ON_ONCE(ret <= 0)) + return; + if (WARN_ON_ONCE(rcu_dereference_protected(ctx->map, true))) + return; + + if (refcount_dec_and_test(&ctx->refs)) + dma_buf_io_ctx_destroy_work(&ctx->release_work); +} + +void dma_buf_io_ctx_release(struct dma_buf_io_ctx ctx) +{ + /* + * Destruction needs to wait for I/O and dma fences. Defer it to + * simplify locking. + / + INIT_WORK(&ctx->release_work, dma_buf_io_ctx_release_work); + queue_work(system_wq, &ctx->release_work); +} + +int dma_buf_io_ctx_create(struct file file, + struct dma_buf_io_ctx ctx, + struct dma_buf dmabuf, + enum dma_data_direction dir) +{ + int ret; + + if (!file->f_op->init_dma_buf_io_ctx) + return -EOPNOTSUPP; + + memset(ctx, 0, sizeof(ctx)); + ctx->fence_ctx = dma_fence_context_alloc(1); + ctx->dir = dir; + ctx->dmabuf = dmabuf; + refcount_set(&ctx->refs, 1); + get_dma_buf(dmabuf); + + ret = file->f_op->init_dma_buf_io_ctx(file, ctx); + if (ret) { + memset(ctx, 0, sizeof(ctx)); + dma_buf_put(dmabuf); + return ret; + } + + if (WARN_ON_ONCE(!ctx->dev_ops \|\| + !ctx->dev_ops->map \|\| + !ctx->dev_ops->unmap \|\| + !ctx->dev_ops->release)) + return -EINVAL; + + return ret; +} diff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c index 6155b0117c9d6..b22719c6411cf 100644 --- a/drivers/md/dm-io-rewind.c +++ b/drivers/md/dm-io-rewind.c @@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec bv, int idx; iter->bi_size += bytes; - if (bytes <= iter->bi_bvec_done) { - iter->bi_bvec_done -= bytes; + if (bytes <= iter->bi_offset) { + iter->bi_offset -= bytes; return true; } - bytes -= iter->bi_bvec_done; + bytes -= iter->bi_offset; idx = iter->bi_idx - 1; while (idx >= 0 && bytes && bytes > bv[idx].bv_len) { @@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec bv, if (WARN_ONCE(idx < 0 && bytes, "Attempted to rewind iter beyond bvec's boundaries\n")) { iter->bi_size -= bytes; - iter->bi_bvec_done = 0; + iter->bi_offset = 0; iter->bi_idx = 0; return false; } iter->bi_idx = idx; - iter->bi_bvec_done = bv[idx].bv_len - bytes; + iter->bi_offset = bv[idx].bv_len - bytes; return true; } @@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bio bio, iter->bi_sector -= bytes >> 9; / No advance means no rewind / - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size += bytes; - else + iter->bi_offset -= bytes; + } else { dm_bvec_iter_rewind(bio->bi_io_vec, iter, bytes); + } } /* diff --git a/drivers/md/dm-pcache/segment.c b/drivers/md/dm-pcache/segment.c index 7e98187014451..8f8816e1c539f 100644 --- a/drivers/md/dm-pcache/segment.c +++ b/drivers/md/dm-pcache/segment.c @@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment segment, iov_iter_bvec(&iter, ITER_DEST, &bio->bi_io_vec[bio->bi_iter.bi_idx], bio_segments(bio), bio->bi_iter.bi_size); - iter.iov_offset = bio->bi_iter.bi_bvec_done; + iter.iov_offset = bio->bi_iter.bi_offset; if (bio_off) iov_iter_advance(&iter, bio_off); @@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment segment, iov_iter_bvec(&iter, ITER_SOURCE, &bio->bi_io_vec[bio->bi_iter.bi_idx], bio_segments(bio), bio->bi_iter.bi_size); - iter.iov_offset = bio->bi_iter.bi_bvec_done; + iter.iov_offset = bio->bi_iter.bi_offset; if (bio_off) iov_iter_advance(&iter, bio_off); diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c index 7e1112960d7f8..5d910a64503d3 100644 --- a/drivers/nvdimm/btt.c +++ b/drivers/nvdimm/btt.c @@ -1155,7 +1155,7 @@ static int btt_rw_integrity(struct btt btt, struct bio_integrity_payload bip, bv = bvec_iter_bvec(bip->bip_vec, bip->bip_iter); /* * The 'bv' obtained from bvec_iter_bvec has its .bv_len and - * .bv_offset already adjusted for iter->bi_bvec_done, and we + * .bv_offset already adjusted for iter->bi_offset, and we * can use those directly / diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 453c1f0b2dd09..8bd407603d928 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2676,6 +2676,17 @@ static int nvme_report_zones(struct gendisk disk, sector_t sector, #define nvme_report_zones NULL #endif /* CONFIG_BLK_DEV_ZONED / +static int nvme_init_dma_buf_io_ctx(struct block_device bdev, + struct dma_buf_io_ctx ctx) +{ + struct nvme_ns ns = bdev->bd_disk->private_data; + struct nvme_ctrl ctrl = ns->ctrl; + + if (!ctrl->ops->init_dma_buf_io_ctx) + return -EINVAL; + return ctrl->ops->init_dma_buf_io_ctx(ctrl, ctx); +} + const struct block_device_operations nvme_bdev_ops = { .owner = THIS_MODULE, .ioctl = nvme_ioctl, @@ -2686,6 +2697,7 @@ const struct block_device_operations nvme_bdev_ops = { .get_unique_id = nvme_get_unique_id, .report_zones = nvme_report_zones, .pr_ops = &nvme_pr_ops, + .init_dma_buf_io_ctx = nvme_init_dma_buf_io_ctx, }; static int nvme_wait_ready(struct nvme_ctrl ctrl, u32 mask, u32 val, diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 824651cc898db..034c31af8fecf 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -652,6 +652,8 @@ struct nvme_ctrl_ops { int (get_address)(struct nvme_ctrl ctrl, char buf, int size); void (print_device_info)(struct nvme_ctrl ctrl); bool (supports_pci_p2pdma)(struct nvme_ctrl ctrl); + int (init_dma_buf_io_ctx)(struct nvme_ctrl ctrl, + struct dma_buf_io_ctx ctx); unsigned long (get_virt_boundary)(struct nvme_ctrl ctrl, bool is_admin); }; diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 69932d640b537..67e26d6e8f255 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -27,6 +27,8 @@ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/io-64-nonatomic-hi-lo.h> #include <linux/sed-opal.h> +#include <linux/dma-buf-io.h> +#include <linux/dma-resv.h> #include "trace.h" #include "nvme.h" @@ -393,6 +395,13 @@ struct nvme_queue { struct completion delete_done; }; +struct nvme_dmabuf_map { + struct dma_buf_io_map base; + struct sg_table sgt; + unsigned nr_entries; + dma_addr_t dma_list[]; +}; + / bits for iod->flags / enum nvme_iod_flags { / this command has been aborted by the timeout handler / @@ -859,6 +868,138 @@ static void nvme_free_descriptors(struct request req) } } +static inline struct nvme_dmabuf_map * +to_nvme_dmabuf_map(struct dma_buf_io_map map) +{ + return container_of(map, struct nvme_dmabuf_map, base); +} + +static void nvme_dmabuf_map_sync_for_cpu(struct nvme_dev nvme_dev, + struct request req) +{ + struct device dev = nvme_dev->dev; + enum dma_data_direction dma_dir; + struct bio bio = req->bio; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio->bi_dmabuf_map); + dma_addr_t dma_list = map->dma_list; + unsigned offset = bio->bi_iter.bi_offset; + unsigned map_idx = offset / NVME_CTRL_PAGE_SIZE; + int length = blk_rq_payload_bytes(req) + + (offset & (NVME_CTRL_PAGE_SIZE - 1)); + + dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE; + + while (length > 0) { + dma_sync_single_for_cpu(dev, dma_list[map_idx++], + NVME_CTRL_PAGE_SIZE, dma_dir); + length -= NVME_CTRL_PAGE_SIZE; + } +} + +static void nvme_dmabuf_map_sync_for_device(struct nvme_dev nvme_dev, + struct request req) +{ + struct device dev = nvme_dev->dev; + enum dma_data_direction dma_dir; + struct bio bio = req->bio; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio->bi_dmabuf_map); + dma_addr_t dma_list = map->dma_list; + unsigned offset = bio->bi_iter.bi_offset; + unsigned map_idx = offset / NVME_CTRL_PAGE_SIZE; + int length = blk_rq_payload_bytes(req) + + (offset & (NVME_CTRL_PAGE_SIZE - 1)); + + dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE; + + while (length > 0) { + dma_sync_single_for_device(dev, dma_list[map_idx++], + NVME_CTRL_PAGE_SIZE, dma_dir); + length -= NVME_CTRL_PAGE_SIZE; + } +} + +static void nvme_rq_clean_dmabuf_map(struct nvme_dev dev, + struct request req) +{ + struct nvme_iod iod = blk_mq_rq_to_pdu(req); + + nvme_dmabuf_map_sync_for_cpu(dev, req); + + if (!(iod->flags & IOD_SINGLE_SEGMENT)) + nvme_free_descriptors(req); +} + +static blk_status_t nvme_rq_setup_dmabuf_map(struct request req, + struct nvme_queue nvmeq) +{ + struct nvme_iod iod = blk_mq_rq_to_pdu(req); + struct bio bio = req->bio; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio->bi_dmabuf_map); + unsigned bvec_done = bio->bi_iter.bi_offset; + unsigned map_idx = bvec_done / NVME_CTRL_PAGE_SIZE; + unsigned offset = bvec_done & (NVME_CTRL_PAGE_SIZE - 1); + int length = blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset); + dma_addr_t dma_list = map->dma_list; + u64 prp1_dma = dma_list[map_idx++] + offset; + u64 dma_addr, prp2_dma; + dma_addr_t prp_dma; + __le64 prp_list; + unsigned i; + + nvme_dmabuf_map_sync_for_device(nvmeq->dev, req); + + if (length <= 0) { + prp2_dma = 0; + goto done; + } + + if (length <= NVME_CTRL_PAGE_SIZE) { + prp2_dma = dma_list[map_idx]; + goto done; + } + + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <= + NVME_SMALL_POOL_SIZE / sizeof(__le64)) + iod->flags \|= IOD_SMALL_DESCRIPTOR; + + prp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) + return BLK_STS_RESOURCE; + + iod->descriptors[iod->nr_descriptors++] = prp_list; + prp2_dma = prp_dma; + i = 0; + for (;;) { + if (i == NVME_CTRL_PAGE_SIZE >> 3) { + __le64 old_prp_list = prp_list; + + prp_list = dma_pool_alloc(nvmeq->descriptor_pools.large, + GFP_ATOMIC, &prp_dma); + if (!prp_list) + goto free_prps; + iod->descriptors[iod->nr_descriptors++] = prp_list; + prp_list[0] = old_prp_list[i - 1]; + old_prp_list[i - 1] = cpu_to_le64(prp_dma); + i = 1; + } + + dma_addr = dma_list[map_idx++]; + prp_list[i++] = cpu_to_le64(dma_addr); + + length -= NVME_CTRL_PAGE_SIZE; + if (length <= 0) + break; + } +done: + iod->cmd.common.dptr.prp1 = cpu_to_le64(prp1_dma); + iod->cmd.common.dptr.prp2 = cpu_to_le64(prp2_dma); + return BLK_STS_OK; +free_prps: + nvme_free_descriptors(req); + return BLK_STS_RESOURCE; +} + static void nvme_free_prps(struct request req, unsigned int attrs) { struct nvme_iod iod = blk_mq_rq_to_pdu(req); @@ -937,6 +1078,11 @@ static void nvme_unmap_data(struct request req) struct device dma_dev = nvmeq->dev->dev; unsigned int attrs = 0; + if (blk_mq_rq_is_dmabuf(req)) { + nvme_rq_clean_dmabuf_map(nvmeq->dev, req); + return; + } + if (iod->flags & IOD_SINGLE_SEGMENT) { static_assert(offsetof(union nvme_data_ptr, prp1) == offsetof(union nvme_data_ptr, sgl.addr)); @@ -1142,13 +1288,19 @@ static blk_status_t nvme_pci_setup_data_prp(struct request req, } static void nvme_pci_sgl_set_data(struct nvme_sgl_desc sge, - struct blk_dma_iter iter) + dma_addr_t addr, u32 len) { - sge->addr = cpu_to_le64(iter->addr); - sge->length = cpu_to_le32(iter->len); + sge->addr = cpu_to_le64(addr); + sge->length = cpu_to_le32(len); sge->type = NVME_SGL_FMT_DATA_DESC << 4; } +static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc sge, + struct blk_dma_iter iter) +{ + nvme_pci_sgl_set_data(sge, iter->addr, iter->len); +} + static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc sge, dma_addr_t dma_addr, int entries) { @@ -1157,6 +1309,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc sge, sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4; } +static unsigned int nvme_pci_dmabuf_sgl_nents(struct request req) +{ + struct bio bio = req->bio; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio->bi_dmabuf_map); + struct scatterlist sg; + unsigned long tmp; + size_t offset = bio->bi_iter.bi_offset; + size_t remaining = blk_rq_payload_bytes(req); + unsigned int nents = 0; + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len = sg_dma_len(sg); + + if (!remaining) + break; + if (offset >= sg_len) { + offset -= sg_len; + continue; + } + + sg_len -= offset; + offset = 0; + + do { + size_t chunk = min(remaining, sg_len); + + nents++; + sg_len -= chunk; + remaining -= chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + return 0; + + return nents; +} + +static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request req, + struct nvme_queue nvmeq, unsigned int entries) +{ + struct nvme_iod iod = blk_mq_rq_to_pdu(req); + struct bio bio = req->bio; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(bio->bi_dmabuf_map); + size_t length = blk_rq_payload_bytes(req); + struct nvme_sgl_desc sg_list = &iod->cmd.common.dptr.sgl; + bool pooled = entries > 1; + dma_addr_t sgl_dma = 0; + unsigned int mapped = 0; + unsigned long tmp; + struct scatterlist sg; + size_t offset = bio->bi_iter.bi_offset; + size_t remaining = length; + + if (!entries) + return BLK_STS_IOERR; + + iod->cmd.common.flags = NVME_CMD_SGL_METABUF; + iod->total_len = length; + + nvme_dmabuf_map_sync_for_device(nvmeq->dev, req); + + /* entries == 1 fits in the inline descriptor; more needs a pool. / + if (pooled) { + if (entries <= NVME_SMALL_POOL_SIZE / sizeof(sg_list)) + iod->flags \|= IOD_SMALL_DESCRIPTOR; + + sg_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &sgl_dma); + if (!sg_list) + return BLK_STS_RESOURCE; + iod->descriptors[iod->nr_descriptors++] = sg_list; + } + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len = sg_dma_len(sg); + dma_addr_t addr = sg_dma_address(sg); + + if (!remaining) + break; + if (offset >= sg_len) { + offset -= sg_len; + continue; + } + + addr += offset; + sg_len -= offset; + offset = 0; + + do { + u32 chunk = min_t(size_t, remaining, sg_len); + + if (WARN_ON_ONCE(mapped == entries)) + goto err_free; + nvme_pci_sgl_set_data(&sg_list[mapped++], addr, chunk); + + addr += chunk; + sg_len -= chunk; + remaining -= chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + goto err_free; + + if (pooled) + nvme_pci_sgl_set_seg(&iod->cmd.common.dptr.sgl, sgl_dma, + mapped); + return BLK_STS_OK; +err_free: + if (pooled) { + iod->nr_descriptors--; + dma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma); + } + return BLK_STS_IOERR; +} + +static blk_status_t nvme_rq_setup_dmabuf(struct request req, + struct nvme_queue nvmeq, enum nvme_use_sgl use_sgl) +{ + unsigned int entries; + size_t avg_seg; + + if (use_sgl == SGL_UNSUPPORTED) + return nvme_rq_setup_dmabuf_map(req, nvmeq); + + entries = nvme_pci_dmabuf_sgl_nents(req); + + if (use_sgl == SGL_FORCED) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + avg_seg = entries ? + DIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0; + if (sgl_threshold && avg_seg >= sgl_threshold) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + return nvme_rq_setup_dmabuf_map(req, nvmeq); +} + static blk_status_t nvme_pci_setup_data_sgl(struct request req, struct blk_dma_iter iter) { @@ -1171,7 +1462,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request req, iod->cmd.common.flags = NVME_CMD_SGL_METABUF; if (entries == 1 \|\| blk_rq_dma_map_coalesce(&iod->dma_state)) { - nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter); + nvme_pci_dma_iter_set_sgl(&iod->cmd.common.dptr.sgl, iter); iod->total_len += iter->len; return BLK_STS_OK; } @@ -1193,7 +1484,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request req, iter->status = BLK_STS_IOERR; break; } - nvme_pci_sgl_set_data(&sg_list[mapped++], iter); + nvme_pci_dma_iter_set_sgl(&sg_list[mapped++], iter); iod->total_len += iter->len; } while (blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter)); @@ -1251,6 +1542,9 @@ static blk_status_t nvme_map_data(struct request req) struct blk_dma_iter iter; blk_status_t ret; + if (blk_mq_rq_is_dmabuf(req)) + return nvme_rq_setup_dmabuf(req, nvmeq, use_sgl); + / * Try to skip the DMA iterator for single segment requests, as that * significantly improves performances for small I/O sizes. @@ -1353,13 +1647,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request req) iod->cmd.common.metadata = cpu_to_le64(sgl_dma); if (entries == 1) { iod->meta_total_len = iter.len; - nvme_pci_sgl_set_data(sg_list, &iter); + nvme_pci_dma_iter_set_sgl(sg_list, &iter); return BLK_STS_OK; } sgl_dma += sizeof(sg_list); do { - nvme_pci_sgl_set_data(&sg_list[++i], &iter); + nvme_pci_dma_iter_set_sgl(&sg_list[++i], &iter); iod->meta_total_len += iter.len; } while (blk_rq_integrity_dma_map_iter_next(req, dev->dev, &iter)); @@ -2269,6 +2563,118 @@ static int nvme_create_queue(struct nvme_queue nvmeq, int qid, bool polled) return result; } +#ifdef CONFIG_DMA_SHARED_BUFFER +static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment attach) +{ + struct dma_buf_io_ctx ctx = attach->importer_priv; + + dma_buf_io_invalidate_mappings(ctx); +} + +const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = { + .invalidate_mappings = nvme_dmabuf_invalidate_mappings, + .allow_peer2peer = true, +}; + +static struct dma_buf_io_map nvme_dma_buf_io_map(struct dma_buf_io_ctx ctx) +{ + unsigned nr_entries = ctx->dmabuf->size / NVME_CTRL_PAGE_SIZE; + struct dma_buf_attachment attach = ctx->dev_priv; + unsigned long tmp, i = 0; + unsigned seg_shift = ~0U; + struct nvme_dmabuf_map map; + struct scatterlist sg; + struct sg_table sgt; + int ret; + + dma_resv_assert_held(ctx->dmabuf->resv); + + map = kmalloc_flex(map, dma_list, nr_entries); + if (!map) + return ERR_PTR(-ENOMEM); + + sgt = dma_buf_map_attachment(attach, ctx->dir); + if (IS_ERR(sgt)) { + ret = PTR_ERR(sgt); + sgt = NULL; + goto err; + } + + for_each_sgtable_dma_sg(sgt, sg, tmp) { + dma_addr_t dma_addr = sg_dma_address(sg); + unsigned long sg_len = sg_dma_len(sg); + + if (sg_len % NVME_CTRL_PAGE_SIZE) { + ret = -EINVAL; + goto err; + } + seg_shift = min(seg_shift, __ffs(sg_len)); + + while (sg_len) { + map->dma_list[i++] = dma_addr; + dma_addr += NVME_CTRL_PAGE_SIZE; + sg_len -= NVME_CTRL_PAGE_SIZE; + } + } + + if (WARN_ON_ONCE(seg_shift < NVME_CTRL_PAGE_SHIFT)) + return ERR_PTR(-EFAULT); + + ret = dma_buf_io_init_map(ctx, &map->base); + if (ret) + goto err; + map->base.seg_shift = seg_shift; + map->nr_entries = nr_entries; + map->sgt = sgt; + return &map->base; +err: + if (sgt) + dma_buf_unmap_attachment(attach, sgt, ctx->dir); + kfree(map); + return ERR_PTR(ret); +} + +static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx ctx, + struct dma_buf_io_map map_base) +{ + struct dma_buf_attachment attach = ctx->dev_priv; + struct nvme_dmabuf_map map = to_nvme_dmabuf_map(map_base); + + dma_resv_assert_held(ctx->dmabuf->resv); + + dma_buf_unmap_attachment(attach, map->sgt, ctx->dir); +} + +static void nvme_dma_buf_io_release(struct dma_buf_io_ctx ctx) +{ + struct dma_buf_attachment attach = ctx->dev_priv; + + dma_buf_detach(ctx->dmabuf, attach); +} + +const struct dma_buf_io_ops nvme_dma_buf_io_ops = { + .map = nvme_dma_buf_io_map, + .unmap = nvme_dma_buf_io_unmap, + .release = nvme_dma_buf_io_release, +}; + +static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl ctrl, + struct dma_buf_io_ctx ctx) +{ + struct dma_buf_attachment attach; + struct nvme_dev dev = to_nvme_dev(ctrl); + + attach = dma_buf_dynamic_attach(ctx->dmabuf, dev->dev, + &nvme_dmabuf_importer_ops, ctx); + if (IS_ERR(attach)) + return PTR_ERR(attach); + + ctx->dev_priv = attach; + ctx->dev_ops = &nvme_dma_buf_io_ops; + return 0; +} +#endif + static const struct blk_mq_ops nvme_mq_admin_ops = { .queue_rq = nvme_queue_rq, .complete = nvme_pci_complete_rq, @@ -3563,6 +3969,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .print_device_info = nvme_pci_print_device_info, .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, .get_virt_boundary = nvme_pci_get_virt_boundary, +#ifdef CONFIG_DMA_SHARED_BUFFER + .init_dma_buf_io_ctx = nvme_pci_init_dma_buf_io_ctx, +#endif }; static int nvme_dev_map(struct nvme_dev dev) @@ -4327,5 +4736,6 @@ MODULE_AUTHOR("Matthew Wilcox <willy@linux.intel.com>"); MODULE_LICENSE("GPL"); MODULE_VERSION("1.0"); MODULE_DESCRIPTION("NVMe host PCIe transport driver"); +MODULE_IMPORT_NS("DMA_BUF"); module_init(nvme_init); module_exit(nvme_exit); diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index ba5c7b3e2a7c6..ce03a0ea4ded7 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -357,7 +357,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request req, iov_iter_bvec(&req->iter, dir, __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter), nr_bvec, bio->bi_iter.bi_size); - req->iter.iov_offset = bio->bi_iter.bi_bvec_done; + req->iter.iov_offset = bio->bi_iter.bi_offset; } } diff --git a/fs/btrfs/misc.h b/fs/btrfs/misc.h index 694be6d0562a4..8020609431801 100644 --- a/fs/btrfs/misc.h +++ b/fs/btrfs/misc.h @@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio bio) .bi_sector = 0, .bi_size = bio_size, .bi_idx = 0, - .bi_bvec_done = 0, + .bi_offset = 0, }; } diff --git a/include/linux/bio.h b/include/linux/bio.h index 0445ecba3b247..22ce9deb2febb 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio bio) { return bio_op(bio) == REQ_OP_DISCARD \|\| bio_op(bio) == REQ_OP_SECURE_ERASE \|\| - bio_op(bio) == REQ_OP_WRITE_ZEROES; + bio_op(bio) == REQ_OP_WRITE_ZEROES \|\| + op_is_dmabuf(bio->bi_opf); } static inline void bio_data(struct bio bio) @@ -113,11 +114,13 @@ static inline void bio_advance_iter(const struct bio bio, { iter->bi_sector += bytes >> 9; - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -= bytes; - else + iter->bi_offset += bytes; + } else { bvec_iter_advance(bio->bi_io_vec, iter, bytes); / TODO: It is reasonable to complete bio with error here. / + } } / @bytes should be less or equal to bvec[i->bi_idx].bv_len / @@ -127,10 +130,12 @@ static inline void bio_advance_iter_single(const struct bio bio, { iter->bi_sector += bytes >> 9; - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -= bytes; - else + iter->bi_offset += bytes; + } else { bvec_iter_advance_single(bio->bi_io_vec, iter, bytes); + } } void __bio_advance(struct bio , unsigned bytes); @@ -434,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio bio) /* * Calculate number of bvec segments that should be allocated to fit data - * pointed by @iter. If @iter is backed by bvec it's going to be reused - * instead of allocating a new one. + * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec array / + * the dma map are going to be reused, and so no extra allocation is required. / static inline int bio_iov_vecs_to_alloc(struct iov_iter iter, int max_segs) { - if (iov_iter_is_bvec(iter)) + if (iov_iter_is_bvec(iter) \|\| iov_iter_is_dmabuf_map(iter)) return 0; return iov_iter_npages(iter, max_segs); } @@ -518,7 +523,7 @@ int bdev_rw_virt(struct block_device bdev, sector_t sector, void data, int bio_iov_iter_get_pages(struct bio bio, struct iov_iter iter, unsigned mem_align_mask, unsigned len_align_mask); -void bio_iov_bvec_set(struct bio bio, const struct iov_iter iter); +bool bio_iov_iter_set(struct bio bio, const struct iov_iter iter); void __bio_release_pages(struct bio bio, bool mark_dirty); extern void bio_set_pages_dirty(struct bio bio); extern void bio_check_pages_dirty(struct bio bio); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index af878597afb8c..7c7504c84e090 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -1017,6 +1017,13 @@ static inline void blk_mq_rq_to_pdu(struct request rq) return rq + 1; } +static inline bool blk_mq_rq_is_dmabuf(struct request rq) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return false; + return rq->bio && op_is_dmabuf(rq->bio->bi_opf); +} + static inline struct blk_mq_hw_ctx queue_hctx(struct request_queue q, int id) { struct blk_mq_hw_ctx hctx; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d49d97a050d0e..a305951f03124 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -233,7 +233,12 @@ struct bio { atomic_t __bi_remaining; / The actual vec list, preserved by bio_reset() / - struct bio_vec bi_io_vec; + union { + struct bio_vec bi_io_vec; + / Driver specific dma map, valid IFF REQ_DMABUF is set / + struct dma_buf_io_map bi_dmabuf_map; + }; + struct bvec_iter bi_iter; union { @@ -402,6 +407,7 @@ enum req_flag_bits { __REQ_DRV, /* for driver use / __REQ_FS_PRIVATE, / for file system (submitter) use / __REQ_ATOMIC, / for atomic write operations / + __REQ_DMABUF, / Using premmaped dma buffers / / * Command specific flags, keep last: / @@ -434,6 +440,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) #define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_DMABUF (__force blk_opf_t)(1ULL << __REQ_DMABUF) #define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP) @@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op) return (op & REQ_OP_MASK) == REQ_OP_DISCARD; } +static inline bool op_is_dmabuf(blk_opf_t op) +{ + return op & REQ_DMABUF; +} + / * Check if a bio or request operation is a zone management operation. / diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 9213a5716f95a..9f4c92e51dc95 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1682,6 +1682,8 @@ struct block_device_operations { / returns the length of the identifier or a negative errno: / int (get_unique_id)(struct gendisk disk, u8 id[16], enum blk_unique_id id_type); + int (init_dma_buf_io_ctx)(struct block_device , + struct dma_buf_io_ctx ); struct module owner; const struct pr_ops pr_ops; diff --git a/include/linux/bvec.h b/include/linux/bvec.h index 92837e2743f19..b63914ff56e33 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -108,9 +108,10 @@ struct bvec_iter { unsigned int bi_idx; /* - * Current offset in the bvec entry pointed to by `bi_idx`. + * Current offset in the bvec entry pointed to by `bi_idx` or into + * a dma-buf map. / - unsigned int bi_bvec_done; + unsigned int bi_offset; } __packed __aligned(4); struct bvec_iter_all { @@ -135,14 +136,14 @@ mp_bvec_iter_page(const struct bio_vec bvecs, const struct bvec_iter iter) static __always_inline unsigned int mp_bvec_iter_len(const struct bio_vec bvecs, const struct bvec_iter iter) { - return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_bvec_done, + return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_offset, iter.bi_size); } static __always_inline unsigned int mp_bvec_iter_offset(const struct bio_vec bvecs, const struct bvec_iter iter) { - return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_bvec_done; + return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_offset; } static __always_inline unsigned int @@ -204,7 +205,7 @@ static inline bool bvec_iter_advance(const struct bio_vec bv, } iter->bi_size -= bytes; - bytes += iter->bi_bvec_done; + bytes += iter->bi_offset; while (bytes && bytes >= bv[idx].bv_len) { bytes -= bv[idx].bv_len; @@ -212,7 +213,7 @@ static inline bool bvec_iter_advance(const struct bio_vec bv, } iter->bi_idx = idx; - iter->bi_bvec_done = bytes; + iter->bi_offset = bytes; return true; } @@ -223,13 +224,13 @@ static inline bool bvec_iter_advance(const struct bio_vec bv, static inline void bvec_iter_advance_single(const struct bio_vec bv, struct bvec_iter iter, unsigned int bytes) { - unsigned int done = iter->bi_bvec_done + bytes; + unsigned int done = iter->bi_offset + bytes; if (done == bv[iter->bi_idx].bv_len) { done = 0; iter->bi_idx++; } - iter->bi_bvec_done = done; + iter->bi_offset = done; iter->bi_size -= bytes; } diff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h new file mode 100644 index 0000000000000..f31c7375ae911 --- /dev/null +++ b/include/linux/dma-buf-io.h @@ -0,0 +1,92 @@ +/ SPDX-License-Identifier: GPL-2.0 / +#ifndef __DMA_BUF_IO_H__ +#define __DMA_BUF_IO_H__ + +#include <linux/dma-buf.h> + +struct dma_buf_io_fence; +struct dma_buf_io_ctx; +struct dma_buf_io_map; + +struct dma_buf_io_ops { + / + * Create a new map for the given ctx. Called with the reservation + * lock held. + / + struct dma_buf_io_map (map)(struct dma_buf_io_ctx ctx); + + /* + * Clean up device specific parts of the @map. Called with the + * reservation lock held. + / + void (unmap)(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map); + + /* + * The user tries to destroy the ctx. Release all device specific + * parts of the token. + / + void (release)(struct dma_buf_io_ctx ); +}; + +struct dma_buf_io_map { + / + * Counts attached requests and other users. Device specific unmapping + * is deferred until all refs are dropped. + / + struct percpu_ref refs; + unsigned seg_shift; + + struct work_struct release_work; + struct dma_buf_io_fence fence; + struct dma_buf_io_ctx ctx; +}; + +struct dma_buf_io_ctx { + struct dma_buf_io_map __rcu map; + struct dma_buf dmabuf; + enum dma_data_direction dir; + + atomic_t fence_seq; + u64 fence_ctx; + struct work_struct release_work; + refcount_t refs; + + void dev_priv; + const struct dma_buf_io_ops dev_ops; +}; + +int dma_buf_io_ctx_create(struct file file, + struct dma_buf_io_ctx ctx, + struct dma_buf dmabuf, + enum dma_data_direction dir); +void dma_buf_io_ctx_release(struct dma_buf_io_ctx ctx); + +struct dma_buf_io_map dma_buf_io_create_map(struct dma_buf_io_ctx ctx); + +static inline struct dma_buf_io_map +dma_buf_io_get_map(struct dma_buf_io_ctx ctx) +{ + struct dma_buf_io_map map; + + guard(rcu)(); + + map = rcu_dereference(ctx->map); + if (unlikely(!map \|\| !percpu_ref_tryget_live_rcu(&map->refs))) + return NULL; + + return map; +} + +static inline void dma_buf_io_map_drop(struct dma_buf_io_map map) +{ + percpu_ref_put(&map->refs); +} + +/ + * Device API + / + +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx ctx); +int dma_buf_io_init_map(struct dma_buf_io_ctx ctx, struct dma_buf_io_map map); + +#endif diff --git a/include/linux/fs.h b/include/linux/fs.h index 50ce731a2b78f..e62fa653e1484 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1914,6 +1914,7 @@ struct dir_context { #define COPY_FILE_SPLICE (1 << 0) struct io_uring_cmd; +struct dma_buf_io_ctx; struct offset_ctx; typedef unsigned int __bitwise fop_flags_t; @@ -1962,6 +1963,7 @@ struct file_operations { int (uring_cmd_iopoll)(struct io_uring_cmd , struct io_comp_batch , unsigned int poll_flags); int (mmap_prepare)(struct vm_area_desc ); + int (init_dma_buf_io_ctx)(struct file , struct dma_buf_io_ctx ); } __randomize_layout; /* Supports async buffered reads / diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index a2c623a67a251..f90586d353c1c 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -10,6 +10,7 @@ struct iou_loop_params; struct io_uring_bpf_ops; +struct dma_buf_io_map; enum { / @@ -594,6 +595,7 @@ enum { REQ_F_IMPORT_BUFFER_BIT, REQ_F_SQE_COPIED_BIT, REQ_F_IOPOLL_BIT, + REQ_F_DROP_DMABUF_BIT, /* not a real bit, just to check we're not overflowing the space / __REQ_F_LAST_BIT, @@ -689,6 +691,8 @@ enum { REQ_F_SQE_COPIED = IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT), / request must be iopolled to completion (set in ->issue()) / REQ_F_IOPOLL = IO_REQ_FLAG(REQ_F_IOPOLL_BIT), + / there is a dma map attached to request that needs to be dropped / + REQ_F_DROP_DMABUF = IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT), }; struct io_tw_req { @@ -811,6 +815,7 @@ struct io_kiocb { / custom credentials, valid IFF REQ_F_CREDS is set / const struct cred creds; struct io_wq_work work; + struct dma_buf_io_map dmabuf_map; struct io_big_cqe { u64 extra1; diff --git a/include/linux/uio.h b/include/linux/uio.h index fe2e985d74d24..638c1116e912e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -12,6 +12,7 @@ struct page; struct folio_queue; +struct dma_buf_io_map; typedef unsigned int __bitwise iov_iter_extraction_t; @@ -29,6 +30,7 @@ enum iter_type { ITER_FOLIOQ, ITER_XARRAY, ITER_DISCARD, + ITER_DMABUF_MAP, }; #define ITER_SOURCE 1 // == WRITE @@ -71,6 +73,7 @@ struct iov_iter { const struct folio_queue folioq; struct xarray xarray; void __user ubuf; + struct dma_buf_io_map dmabuf_map; }; size_t count; }; @@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter i) return iov_iter_type(i) == ITER_XARRAY; } +static inline bool iov_iter_is_dmabuf_map(const struct iov_iter i) +{ + return iov_iter_type(i) == ITER_DMABUF_MAP; +} + static inline unsigned char iov_iter_rw(const struct iov_iter i) { return i->data_source ? WRITE : READ; @@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter i, unsigned int direction, unsigned int first_slot, unsigned int offset, size_t count); void iov_iter_xarray(struct iov_iter i, unsigned int direction, struct xarray xarray, loff_t start, size_t count); +void iov_iter_dmabuf_map(struct iov_iter i, unsigned int direction, + struct dma_buf_io_map map, + loff_t off, size_t count); ssize_t iov_iter_get_pages2(struct iov_iter i, struct page *pages, size_t maxsize, unsigned maxpages, size_t start); ssize_t iov_iter_get_pages_alloc2(struct iov_iter i, struct page *pages, diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 909fb7aea638e..c39297e8beb6f 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -790,13 +790,42 @@ struct io_uring_rsrc_update { struct io_uring_rsrc_update2 { __u32 offset; - __u32 resv; + __u32 flags; __aligned_u64 data; __aligned_u64 tags; __u32 nr; __u32 resv2; }; +/ struct io_uring_rsrc_update2::flags / +enum io_uring_rsrc_reg_flags { + / + * Use the extended descriptor format for buffer updates, + * see struct io_uring_regbuf_desc + / + IORING_RSRC_UPDATE_EXTENDED = 1U << 1, +}; + +/ Buffer registration type, passed in struct io_uring_regbuf_desc::type / +enum io_uring_regbuf_type { + IO_REGBUF_TYPE_EMPTY, + IO_REGBUF_TYPE_UADDR, + IO_REGBUF_TYPE_DMABUF, + + __IO_REGBUF_TYPE_MAX, +}; + +struct io_uring_regbuf_desc { + __u32 type; / enum io_uring_regbuf_type / + __u32 flags; + __u64 size; + __u64 uaddr; + + __s32 dmabuf_fd; + __s32 target_fd; + __u64 __resv[6]; +}; + / Skip updating fd indexes set to this value in the fd table / #define IORING_REGISTER_FILES_SKIP (-2) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4c83a94b4bdc3..d1d748b5c5dca 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -109,7 +109,7 @@ #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT \| IO_REQ_LINK_FLAGS \| \ REQ_F_REISSUE \| REQ_F_POLLED \| \ - IO_REQ_CLEAN_FLAGS) + IO_REQ_CLEAN_FLAGS \| REQ_F_DROP_DMABUF) #define IO_TCTX_REFS_CACHE_NR (1U << 10) @@ -1114,6 +1114,7 @@ static void io_free_batch_list(struct io_ring_ctx ctx, io_queue_next(req); if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS)) io_clean_op(req); + io_req_drop_dmabuf(req); } io_put_file(req); io_req_put_rsrc_nodes(req); diff --git a/io_uring/net.c b/io_uring/net.c index a74d15f7b7d2b..439c99ad18841 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1471,7 +1471,7 @@ static int io_sg_from_iter(struct sk_buff skb, return zerocopy_fill_skb_from_iter(skb, from, length); bi.bi_size = min(from->count, length); - bi.bi_bvec_done = from->iov_offset; + bi.bi_offset = from->iov_offset; bi.bi_idx = 0; while (bi.bi_size && frag < MAX_SKB_FRAGS) { @@ -1490,7 +1490,7 @@ static int io_sg_from_iter(struct sk_buff skb, from->bvec += bi.bi_idx; from->nr_segs -= bi.bi_idx; from->count -= copied; - from->iov_offset = bi.bi_bvec_done; + from->iov_offset = bi.bi_offset; skb->data_len += copied; skb->len += copied; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 8d0f2ee24e0c2..5074b1e4b0505 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -10,6 +10,7 @@ #include <linux/compat.h> #include <linux/io_uring.h> #include <linux/io_uring/cmd.h> +#include <linux/dma-buf-io.h> #include <uapi/linux/io_uring.h> @@ -28,7 +29,7 @@ struct io_rsrc_update { }; static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, - struct iovec iov); + struct io_uring_regbuf_desc desc); static int hpage_acct_ref(struct io_ring_ctx ctx, struct page hpage, bool acct_new) @@ -81,6 +82,18 @@ static bool hpage_acct_unref(struct io_ring_ctx ctx, struct page hpage) #define IO_CACHED_BVECS_SEGS 32 +static void io_iov_to_regbuf_desc(const struct iovec iov, + struct io_uring_regbuf_desc desc) +{ + desc = (struct io_uring_regbuf_desc) { + .type = IO_REGBUF_TYPE_UADDR, + .uaddr = (u64)(uintptr_t)iov->iov_base, + .size = iov->iov_len, + }; + if (!desc->uaddr) + desc->type = IO_REGBUF_TYPE_EMPTY; +} + int __io_account_mem(struct user_struct user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -309,6 +322,8 @@ static int __io_sqe_files_update(struct io_ring_ctx ctx, return -ENXIO; if (up->offset + nr_args > ctx->file_table.data.nr) return -EINVAL; + if (up->flags) + return -EINVAL; for (done = 0; done < nr_args; done++) { u64 tag = 0; @@ -368,9 +383,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx, struct io_uring_rsrc_update2 up, unsigned int nr_args) { + bool extended = up->flags & IORING_RSRC_UPDATE_EXTENDED; u64 __user tags = u64_to_user_ptr(up->tags); - struct iovec fast_iov, iov; - struct iovec __user uvec; u64 user_data = up->data; __u32 done; int i, err; @@ -379,26 +393,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx, return -ENXIO; if (up->offset + nr_args > ctx->buf_table.nr) return -EINVAL; + if (up->flags & ~IORING_RSRC_UPDATE_EXTENDED) + return -EINVAL; for (done = 0; done < nr_args; done++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node node; u64 tag = 0; - uvec = u64_to_user_ptr(user_data); - iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); - if (IS_ERR(iov)) { - err = PTR_ERR(iov); - break; - } if (tags && copy_from_user(&tag, &tags[done], sizeof(tag))) { err = -EFAULT; break; } - node = io_sqe_buffer_register(ctx, iov); + + if (extended) { + if (copy_from_user(&desc, u64_to_user_ptr(user_data), + sizeof(desc))) { + err = -EFAULT; + break; + } + user_data += sizeof(desc); + } else { + struct iovec __user uvec = u64_to_user_ptr(user_data); + struct iovec fast_iov, iov; + + if (io_is_compat(ctx)) + user_data += sizeof(struct compat_iovec); + else + user_data += sizeof(struct iovec); + + iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); + if (IS_ERR(iov)) { + err = PTR_ERR(iov); + break; + } + io_iov_to_regbuf_desc(iov, &desc); + } + + node = io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { err = PTR_ERR(node); break; } + if (tag) { if (!node) { err = -EINVAL; @@ -409,10 +446,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx ctx, i = array_index_nospec(up->offset + done, ctx->buf_table.nr); io_reset_rsrc_node(ctx, &ctx->buf_table, i); ctx->buf_table.nodes[i] = node; - if (io_is_compat(ctx)) - user_data += sizeof(struct compat_iovec); - else - user_data += sizeof(struct iovec); } return done ? done : err; } @@ -447,7 +480,7 @@ int io_register_files_update(struct io_ring_ctx ctx, void __user arg, memset(&up, 0, sizeof(up)); if (copy_from_user(&up, arg, sizeof(struct io_uring_rsrc_update))) return -EFAULT; - if (up.resv \|\| up.resv2) + if (up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, IORING_RSRC_FILE, &up, nr_args); } @@ -461,7 +494,7 @@ int io_register_rsrc_update(struct io_ring_ctx ctx, void __user arg, return -EINVAL; if (copy_from_user(&up, arg, sizeof(up))) return -EFAULT; - if (!up.nr \|\| up.resv \|\| up.resv2) + if (!up.nr \|\| up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, type, &up, up.nr); } @@ -561,12 +594,9 @@ int io_files_update(struct io_kiocb req, unsigned int issue_flags) struct io_uring_rsrc_update2 up2; int ret; + memset(&up2, 0, sizeof(up2)); up2.offset = up->offset; up2.data = up->arg; - up2.nr = 0; - up2.tags = 0; - up2.resv = 0; - up2.resv2 = 0; if (up->offset == IORING_FILE_INDEX_ALLOC) { ret = io_files_update_with_index_alloc(req, issue_flags); @@ -852,27 +882,126 @@ bool io_check_coalesce_buffer(struct page page_array, int nr_pages, return true; } +struct io_regbuf_dma { + struct dma_buf_io_ctx ctx; + struct file target_file; +}; + +static void io_release_reg_dmabuf(void priv) +{ + struct io_regbuf_dma db = priv; + + fput(db->target_file); + dma_buf_io_ctx_release(&db->ctx); +} + +static struct io_rsrc_node io_register_dmabuf(struct io_ring_ctx ctx, + struct io_uring_regbuf_desc desc) +{ + struct io_rsrc_node node = NULL; + struct io_mapped_ubuf imu = NULL; + struct io_regbuf_dma regbuf = NULL; + struct file target_file = NULL; + struct dma_buf dmabuf = NULL; + int ret; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return ERR_PTR(-EOPNOTSUPP); + if (ctx->flags & IORING_SETUP_IOPOLL) + return ERR_PTR(-EOPNOTSUPP); + if (desc->uaddr \|\| desc->size) + return ERR_PTR(-EINVAL); + + ret = -ENOMEM; + node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); + if (!node) + return ERR_PTR(-ENOMEM); + imu = io_alloc_imu(ctx, 0); + if (!imu) + goto err; + regbuf = kzalloc(sizeof(regbuf), GFP_KERNEL); + if (!regbuf) + goto err; + + ret = -EBADF; + target_file = fget(desc->target_fd); + if (!target_file) + goto err; + + dmabuf = dma_buf_get(desc->dmabuf_fd); + if (IS_ERR(dmabuf)) { + ret = PTR_ERR(dmabuf); + dmabuf = NULL; + goto err; + } + if (dmabuf->size > SZ_1G) { + ret = -EINVAL; + goto err; + } + + ret = dma_buf_io_ctx_create(target_file, &regbuf->ctx, dmabuf, + DMA_BIDIRECTIONAL); + if (ret) + goto err; + + regbuf->target_file = target_file; + imu->nr_bvecs = 0; + imu->ubuf = 0; + imu->len = dmabuf->size; + imu->folio_shift = 0; + imu->release = io_release_reg_dmabuf; + imu->priv = regbuf; + imu->flags = IO_REGBUF_F_DMABUF; + imu->dir = IO_IMU_DEST \| IO_IMU_SOURCE; + refcount_set(&imu->refs, 1); + node->buf = imu; + dma_buf_put(dmabuf); + return node; +err: + kfree(regbuf); + if (imu) + io_free_imu(ctx, imu); + if (node) + io_cache_free(&ctx->node_cache, node); + if (target_file) + fput(target_file); + if (dmabuf) + dma_buf_put(dmabuf); + return ERR_PTR(ret); +} + static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, - struct iovec iov) + struct io_uring_regbuf_desc desc) { + unsigned long uaddr = (unsigned long)desc->uaddr; + size_t size = desc->size; struct io_mapped_ubuf imu = NULL; struct page *pages = NULL; struct io_rsrc_node node; unsigned long off; - size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; bool coalesced = false; - if (!iov->iov_base) { - if (iov->iov_len) + if (desc->type >= __IO_REGBUF_TYPE_MAX) + return ERR_PTR(-EINVAL); + if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) \|\| desc->flags) + return ERR_PTR(-EINVAL); + + if (desc->type == IO_REGBUF_TYPE_DMABUF) + return io_register_dmabuf(ctx, desc); + + if (desc->dmabuf_fd \|\| desc->target_fd) + return ERR_PTR(-EINVAL); + + if (desc->type == IO_REGBUF_TYPE_EMPTY) { + if (uaddr \|\| size) return ERR_PTR(-EFAULT); /* remove the buffer without installing a new one / return NULL; } - ret = io_validate_user_buf_range((unsigned long)iov->iov_base, - iov->iov_len); + ret = io_validate_user_buf_range(uaddr, size); if (ret) return ERR_PTR(ret); @@ -881,8 +1010,7 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, return ERR_PTR(-ENOMEM); ret = -ENOMEM; - pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, - &nr_pages); + pages = io_pin_pages(uaddr, size, &nr_pages); if (IS_ERR(pages)) { ret = PTR_ERR(pages); pages = NULL; @@ -904,10 +1032,9 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, if (ret) goto done; - size = iov->iov_len; / store original address for later verification / - imu->ubuf = (unsigned long) iov->iov_base; - imu->len = iov->iov_len; + imu->ubuf = uaddr; + imu->len = size; imu->folio_shift = PAGE_SHIFT; imu->release = io_release_ubuf; imu->priv = imu; @@ -917,7 +1044,7 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, imu->folio_shift = data.folio_shift; refcount_set(&imu->refs, 1); - off = (unsigned long)iov->iov_base & ~PAGE_MASK; + off = uaddr & ~PAGE_MASK; if (coalesced) off += data.first_folio_page_idx << PAGE_SHIFT; @@ -969,6 +1096,7 @@ int io_sqe_buffers_register(struct io_ring_ctx ctx, void __user arg, memset(iov, 0, sizeof(iov)); for (i = 0; i < nr_args; i++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node node; u64 tag = 0; @@ -992,7 +1120,8 @@ int io_sqe_buffers_register(struct io_ring_ctx ctx, void __user arg, } } - node = io_sqe_buffer_register(ctx, iov); + io_iov_to_regbuf_desc(iov, &desc); + node = io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { ret = PTR_ERR(node); break; @@ -1136,9 +1265,64 @@ static int io_import_kbuf(int ddir, struct iov_iter iter, return 0; } -static int io_import_fixed(int ddir, struct iov_iter iter, +void io_drop_dmabuf_node(struct io_kiocb req) +{ + struct io_mapped_ubuf imu; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return; + if (WARN_ON_ONCE(req->buf_node->type != IORING_RSRC_BUFFER)) + return; + imu = req->buf_node->buf; + if (WARN_ON_ONCE(!(imu->flags & IO_REGBUF_F_DMABUF))) + return; + dma_buf_io_map_drop(req->dmabuf_map); + req->flags &= ~REQ_F_DROP_DMABUF; +} + +static int io_import_dmabuf(struct io_kiocb req, + int ddir, struct iov_iter iter, + struct io_mapped_ubuf imu, + size_t len, size_t offset, + unsigned issue_flags) +{ + struct io_regbuf_dma db = imu->priv; + struct dma_buf_io_map map; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + if (!len) + return -EFAULT; + if (req->file != db->target_file) + return -EBADF; + + if (req->flags & REQ_F_DROP_DMABUF) { + map = req->dmabuf_map; + goto init_iter; + } + + map = dma_buf_io_get_map(&db->ctx); + if (unlikely(!map)) { + if (!(issue_flags & IO_URING_F_IOWQ)) + return -EAGAIN; + map = dma_buf_io_create_map(&db->ctx); + if (IS_ERR(map)) + return PTR_ERR(map); + } + + req->dmabuf_map = map; + req->flags \|= REQ_F_DROP_DMABUF; +init_iter: + iov_iter_dmabuf_map(iter, ddir, map, offset, len); + return 0; +} + +static int io_import_fixed(struct io_kiocb req, + int ddir, struct iov_iter iter, struct io_mapped_ubuf imu, - u64 buf_addr, size_t len) + u64 buf_addr, size_t len, + unsigned issue_flags, + unsigned import_flags) { const struct bio_vec bvec; size_t folio_mask; @@ -1158,6 +1342,12 @@ static int io_import_fixed(int ddir, struct iov_iter iter, offset = buf_addr - imu->ubuf; + if (imu->flags & IO_REGBUF_F_DMABUF) { + if (!(import_flags & IO_REGBUF_IMPORT_ALLOW_DMABUF)) + return -EFAULT; + return io_import_dmabuf(req, ddir, iter, imu, len, offset, + issue_flags); + } if (imu->flags & IO_REGBUF_F_KBUF) return io_import_kbuf(ddir, iter, imu, len, offset); @@ -1211,16 +1401,17 @@ inline struct io_rsrc_node io_find_buf_node(struct io_kiocb req, return NULL; } -int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter, +int __io_import_reg_buf(struct io_kiocb req, struct iov_iter iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags) + unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node node; node = io_find_buf_node(req, issue_flags); if (!node) return -EFAULT; - return io_import_fixed(ddir, iter, node->buf, buf_addr, len); + return io_import_fixed(req, ddir, iter, node->buf, buf_addr, len, + issue_flags, import_flags); } static int io_buffer_acct_cloned_hpages(struct io_ring_ctx ctx, @@ -1347,6 +1538,11 @@ static int io_clone_buffers(struct io_ring_ctx ctx, struct io_ring_ctx src_ctx if (!src_node) { dst_node = NULL; } else { + if (src_node->buf->flags & IO_REGBUF_F_UNCLONEABLE) { + io_rsrc_data_free(ctx, &data); + return -ENOMEM; + } + dst_node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); if (!dst_node) { io_rsrc_data_free(ctx, &data); @@ -1617,9 +1813,9 @@ static int io_kern_bvec_size(struct iovec iov, unsigned nr_iovs, return 0; } -int io_import_reg_vec(int ddir, struct iov_iter iter, +int __io_import_reg_vec(int ddir, struct iov_iter iter, struct io_kiocb req, struct iou_vec vec, - unsigned nr_iovs, unsigned issue_flags) + unsigned nr_iovs, unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node node; struct io_mapped_ubuf imu; @@ -1637,7 +1833,9 @@ int io_import_reg_vec(int ddir, struct iov_iter iter, iovec_off = vec->nr - nr_iovs; iov = vec->iovec + iovec_off; - if (imu->flags & IO_REGBUF_F_KBUF) { + if (imu->flags & IO_REGBUF_F_DMABUF) { + return -EOPNOTSUPP; + } else if (imu->flags & IO_REGBUF_F_KBUF) { int ret = io_kern_bvec_size(iov, nr_iovs, imu, &nr_segs); if (unlikely(ret)) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 98ae8ef51009d..3cb9b57f159b6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -29,7 +29,13 @@ enum { }; enum { - IO_REGBUF_F_KBUF = 1, + IO_REGBUF_F_KBUF = 1 << 0, + IO_REGBUF_F_UNCLONEABLE = 1 << 1, + IO_REGBUF_F_DMABUF = 1 << 3, +}; + +enum { + IO_REGBUF_IMPORT_ALLOW_DMABUF = 1 << 1, }; struct io_mapped_ubuf { @@ -64,12 +70,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data data, unsigned nr); struct io_rsrc_node io_find_buf_node(struct io_kiocb req, unsigned issue_flags); -int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter, +int __io_import_reg_buf(struct io_kiocb req, struct iov_iter iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags); -int io_import_reg_vec(int ddir, struct iov_iter iter, + unsigned issue_flags, unsigned import_flags); +int __io_import_reg_vec(int ddir, struct iov_iter iter, struct io_kiocb req, struct iou_vec vec, - unsigned nr_iovs, unsigned issue_flags); + unsigned nr_iovs, unsigned issue_flags, + unsigned import_flags); + +static inline int io_import_reg_buf(struct io_kiocb req, struct iov_iter iter, + u64 buf_addr, size_t len, int ddir, + unsigned issue_flags) +{ + return __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0); +} + +static inline int io_import_reg_vec(int ddir, struct iov_iter iter, + struct io_kiocb req, struct iou_vec vec, + unsigned nr_iovs, unsigned issue_flags) +{ + return __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0); +} + int io_prep_reg_iovec(struct io_kiocb req, struct iou_vec iv, const struct iovec __user uvec, size_t uvec_segs); @@ -156,4 +178,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec iv) io_vec_free(iv); } +void io_drop_dmabuf_node(struct io_kiocb req); + +static inline void io_req_drop_dmabuf(struct io_kiocb req) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return; + if (!(req->flags & REQ_F_DROP_DMABUF)) + return; + if (WARN_ON_ONCE(!(req->flags & REQ_F_BUF_NODE))) + return; + io_drop_dmabuf_node(req); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 95038cfda6153..11ff7b5a11647 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb req, unsigned int issue_flags, if (io->bytes_done) return 0; - ret = io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, - issue_flags); + ret = __io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, + issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF); iov_iter_save_state(&io->iter, &io->iter_state); return ret; } @@ -591,6 +591,8 @@ static void io_complete_rw(struct kiocb kiocb, long res) struct io_rw rw = container_of(kiocb, struct io_rw, kiocb); struct io_kiocb req = cmd_to_io_kiocb(rw); + io_req_drop_dmabuf(req); + __io_complete_rw_common(req, res); io_req_set_res(req, io_fixup_rw_res(req, res), 0); req->io_task_work.func = io_req_rw_complete; diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 34a52e9ba9e1b..d6f5d7e180b54 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter i, size_t size) { if (unlikely(i->count < size)) size = i->count; - if (likely(iter_is_ubuf(i)) \|\| unlikely(iov_iter_is_xarray(i))) { + if (likely(iter_is_ubuf(i)) \|\| unlikely(iov_iter_is_xarray(i)) \|\| + unlikely(iov_iter_is_dmabuf_map(i))) { i->iov_offset += size; i->count -= size; } else if (likely(iter_is_iovec(i) \|\| iov_iter_is_kvec(i))) { @@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter i, size_t unroll) return; } unroll -= i->iov_offset; - if (iov_iter_is_xarray(i) \|\| iter_is_ubuf(i)) { + if (iov_iter_is_xarray(i) \|\| iter_is_ubuf(i) \|\| + iov_iter_is_dmabuf_map(i)) { BUG(); / We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_xarray); +void iov_iter_dmabuf_map(struct iov_iter i, unsigned int direction, + struct dma_buf_io_map map, + loff_t off, size_t count) +{ + WARN_ON(direction & ~(READ \| WRITE)); + i = (struct iov_iter){ + .iter_type = ITER_DMABUF_MAP, + .data_source = direction, + .dmabuf_map = map, + .count = count, + .iov_offset = off, + }; +} + /** * iov_iter_discard - Initialise an I/O iterator that discards data * @i: The iterator to initialise. @@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter i) unsigned long iov_iter_alignment(const struct iov_iter i) { - if (likely(iter_is_ubuf(i))) { + if (likely(iter_is_ubuf(i)) \|\| iov_iter_is_dmabuf_map(i)) { size_t size = i->count; if (size) return ((unsigned long)i->ubuf + i->iov_offset) \| size; @@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter i) size_t size = i->count; unsigned k; - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) \|\| iov_iter_is_dmabuf_map(i)) return 0; if (WARN_ON(!iter_is_iovec(i))) @@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf); void iov_iter_restore(struct iov_iter i, struct iov_iter_state state) { if (WARN_ON_ONCE(!iov_iter_is_bvec(i) && !iter_is_iovec(i) && - !iter_is_ubuf(i)) && !iov_iter_is_kvec(i)) + !iter_is_ubuf(i) && !iov_iter_is_kvec(i) && + !iov_iter_is_dmabuf_map(i))) return; i->iov_offset = state->iov_offset; i->count = state->count; - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) \|\| iov_iter_is_dmabuf_map(i)) return; / * For the vec iters, nr_segs + iov is constant - if we increment @@ -1634,7 +1651,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter i, } bi.bi_idx = 0; bi.bi_size = maxsize; - bi.bi_bvec_done = skip; + bi.bi_offset = skip; maxpages = want_pages_array(pages, maxsize, skip, maxpages); if (!maxpages) diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c index 34b3097b4c7b9..9c1b6cf8c36fe 100644 --- a/net/ceph/messenger.c +++ b/net/ceph/messenger.c @@ -762,7 +762,7 @@ static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor cursor, if (!cursor->resid) return false; / no more data / - if (!bytes \|\| (it->iter.bi_size && it->iter.bi_bvec_done && + if (!bytes \|\| (it->iter.bi_size && it->iter.bi_offset && page == bio_iter_page(it->bio, it->iter))) return false; / more bytes to process in this segment / @@ -817,7 +817,7 @@ static bool ceph_msg_data_bvecs_advance(struct ceph_msg_data_cursor cursor, if (!cursor->resid) return false; /* no more data / - if (!bytes \|\| (cursor->bvec_iter.bi_bvec_done && + if (!bytes \|\| (cursor->bvec_iter.bi_offset && page == bvec_iter_page(bvecs, cursor->bvec_iter))) return false; / more bytes to process in this segment */
3/2	2026/08/02 14:37	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 36098 output: 28 thoughts: 1521 Thoughts: Fuzzing Target Identification: DMA-BUF Integration Okay, let's break this down. My initial assessment is that this patch series absolutely warrants fuzzing. We're looking at a fundamental shift here, integrating `dma-buf` into the core I/O paths, specifically impacting `io_uring` and NVMe. This isn't just a minor tweak; it's a significant functional change within the block layer, NVMe driver, and io_uring components. My expertise tells me these are high-value targets for uncovering potential vulnerabilities. Let's look at the specific changes. The patch modifies `bio`, `blk-mq`, `nvme`, `io_uring`, and `iov_iter` to support `dma-buf` mappings. We're seeing changes like `bio_iov_iter_set` replacing `bio_iov_bvec_set`, with the crucial addition of `iov_iter_is_dmabuf_map` handling. We've got a new file, `dma-buf-io.c`, introducing functions like `dma_buf_io_ctx_create` and `dma_buf_io_create_map`, etc. The NVMe driver gets modifications for `nvme_rq_setup_dmabuf`, `nvme_pci_init_dma_buf_io_ctx`, etc., which are important. And in `io_uring/rsrc.c`, we see `io_register_dmabuf`, `io_import_dmabuf`. And, of course, the introduction of `ITER_DMABUF_MAP` in `iov_iter`. We're dealing with a new I/O mapping infrastructure based on `dma-buf`, and its integration with existing mechanisms. This is prime fuzzing territory. I've identified the configurations that are key: `CONFIG_DMA_SHARED_BUFFER` is essential. Although it seems to be enabled by default in many configs, we'll need to specify it explicitly for a complete fuzzing coverage. `CONFIG_NVME_CORE`, `CONFIG_BLK_DEV_NVME`, and `CONFIG_IO_URING` are the other important ones. Now, let's identify the specific functions. I need to focus on the key symbols to fuzz. Based on the prompt, it is better to fuzz non-static functions over internal static helper functions. This suggests that we should aim for API entrypoints. Therefore, the list needs refinement. So I am looking at: - `dma_buf_io_ctx_create` - `dma_buf_io_ctx_release` - `dma_buf_io_create_map` - `dma_buf_io_invalidate_mappings` - `dma_buf_io_init_map` - `bio_iov_iter_set` - `__io_import_reg_buf` - `io_drop_dmabuf_node` - `iov_iter_dmabuf_map` - `bio_split_io_at` (since `bio_split_io_at_dmabuf` is static inline) - `nvme_map_data` (calls `nvme_rq_setup_dmabuf`) - `nvme_unmap_data` (calls `nvme_rq_clean_dmabuf_map`) - `nvme_pci_init_dma_buf_io_ctx` (static, but maybe `nvme_dev_map` or something? It's a callback in `nvme_pci_ctrl_ops`). So I think a good initial focus would be: - `dma_buf_io_ctx_create` - `dma_buf_io_ctx_release` - `dma_buf_io_create_map` - `dma_buf_io_invalidate_mappings` - `dma_buf_io_init_map` - `bio_iov_iter_set` - `__io_import_reg_buf` - `io_drop_dmabuf_node` - `iov_iter_dmabuf_map` - `bio_split_io_at` - `nvme_map_data` - `nvme_unmap_data` - `nvme_pci_init_dma_buf_io_ctx` The reasoning here is straightforward. This patch series enables `dma-buf` within key I/O paths. It adds new functions to create and manage `dma-buf` mappings and modifies core functions to handle `d
4/2	2026/08/02 14:37	tool	grepper	0m Args: { "Expression": "DMA_SHARED_BUFFER", "PathPrefix": "drivers/Makefile" } Results: { "Output": "drivers/Makefile=78=obj-y\t\t\t\t+= dax/\ndrivers/Makefile:79:obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/\ndrivers/Makefile-80-obj-$(CONFIG_NUBUS)\t\t+= nubus/\n" }
5/2	2026/08/02 14:37	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 38747 output: 27 thoughts: 22 Thoughts: Okay, here's my quick mental rundown: Confirming DMA Buffer and Moving on to IO_URING Alright, yep, that `DMA_SHARED_BUFFER` variable—I'm pretty confident that's the correct one we're working with here. Good. Now, let's pivot and take a look at how this all plays out with `IO_URING`. Time to shift my focus and start assessing the implications there.
6/2	2026/08/02 14:38	tool	grepper	0m Args: { "Expression": "IO_URING", "PathPrefix": "init/Kconfig" } Results: { "Output": "init/Kconfig=1910=config AIO\n--\ninit/Kconfig-1917-\ninit/Kconfig:1918:config IO_URING\ninit/Kconfig-1919-\tbool \"Enable IO uring support\" if EXPERT\n--\ninit/Kconfig=1927=config GCOV_PROFILE_URING\ninit/Kconfig-1928-\tbool \"Enable GCOV profiling on the io_uring subsystem\"\ninit/Kconfig:1929:\tdepends on IO_URING \u0026\u0026 GCOV_KERNEL\ninit/Kconfig-1930-\thelp\n--\ninit/Kconfig-1939-\ninit/Kconfig:1940:config IO_URING_MOCK_FILE\ninit/Kconfig-1941-\ttristate \"Enable io_uring mock files (Experimental)\" if EXPERT\ninit/Kconfig-1942-\tdefault n\ninit/Kconfig:1943:\tdepends on IO_URING\ninit/Kconfig-1944-\thelp\n" }
7/2	2026/08/02 14:38	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 39155 output: 35 thoughts: 23 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: IO_URING and NVMe Considerations Yes, `IO_URING` is definitely the right direction to go here. The inherent benefits are clear and it's where we should focus our attention for this architecture. Now, I want to take a deeper look at `BLK_DEV_NVME`. That's where the real performance gains are going to be found. Let's see how well it's working with this new design.
8/2	2026/08/02 14:38	tool	grepper	0m Args: { "Expression": "BLK_DEV_NVME", "PathPrefix": "drivers/nvme/host/Kconfig" } Results: { "Output": "drivers/nvme/host/Kconfig=2=config NVME_CORE\n--\ndrivers/nvme/host/Kconfig-4-\ndrivers/nvme/host/Kconfig:5:config BLK_DEV_NVME\ndrivers/nvme/host/Kconfig-6-\ttristate \"NVM Express block device\"\n" }
9/2	2026/08/02 14:38	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 39414 output: 243 thoughts: 409 Thoughts: Analyzing DMA-Buf Related Symbols for NVMe Driver Integration Okay, so we've confirmed `BLK_DEV_NVME` is the right path here. Now, let's take a look at the important symbols. I'm going through the list and double-checking that we're covering all the relevant functions related to `dma_buf` interaction. First, I identified `dma_buf_io_ctx_create`, `dma_buf_io_ctx_release`, `dma_buf_io_create_map`, `dma_buf_io_invalidate_mappings`, `dma_buf_io_init_map`, `bio_iov_iter_set`, `__io_import_reg_buf`, `io_drop_dmabuf_node` and `iov_iter_dmabuf_map` as initial candidates. I briefly considered `nvme_map_data`, and `nvme_unmap_data`, but realized those are static, as well as `nvme_pci_init_dma_buf_io_ctx` which is also static. Static functions are likely not directly used from outside the driver, so we're probably fine not including them directly, even if the function itself is a callback. Alright, after reviewing the list again, I'm confident with these candidates. The non-static functions seem to capture the core of the `dma_buf` interaction points for the NVMe driver. Here's the final list: * `dma_buf_io_ctx_create` * `dma_buf_io_ctx_release` * `dma_buf_io_create_map` * `dma_buf_io_invalidate_mappings` * `dma_buf_io_init_map` * `bio_iov_iter_set` * `__io_import_reg_buf` * `io_drop_dmabuf_node` * `iov_iter_dmabuf_map` I believe this is a solid selection. Let's call `set-results` and move forward.
10/2	2026/08/02 14:38	tool	set-results	0m Args: { "EnableConfigs": [ "DMA_SHARED_BUFFER", "IO_URING", "BLK_DEV_NVME" ], "FocusSymbols": [ "dma_buf_io_ctx_create", "dma_buf_io_ctx_release", "dma_buf_io_create_map", "dma_buf_io_invalidate_mappings", "dma_buf_io_init_map", "bio_iov_iter_set", "__io_import_reg_buf", "io_drop_dmabuf_node", "iov_iter_dmabuf_map" ], "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "DMA_SHARED_BUFFER", "IO_URING", "BLK_DEV_NVME" ], "FocusSymbols": [ "dma_buf_io_ctx_create", "dma_buf_io_ctx_release", "dma_buf_io_create_map", "dma_buf_io_invalidate_mappings", "dma_buf_io_init_map", "bio_iov_iter_set", "__io_import_reg_buf", "io_drop_dmabuf_node", "iov_iter_dmabuf_map" ], "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.", "WorthFuzzing": true }

Trajectory: (Expand/Collapse All)

Seq

Timestamp

Type

Name

Duration

0/0

2026/08/02 14:37

flow

patch-triage

Args:

null

Results:

{
  "EnableConfigs": [
    "DMA_SHARED_BUFFER",
    "IO_URING",
    "BLK_DEV_NVME"
  ],
  "FocusSymbols": [
    "dma_buf_io_ctx_create",
    "dma_buf_io_ctx_release",
    "dma_buf_io_create_map",
    "dma_buf_io_invalidate_mappings",
    "dma_buf_io_init_map",
    "bio_iov_iter_set",
    "__io_import_reg_buf",
    "io_drop_dmabuf_node",
    "iov_iter_dmabuf_map"
  ],
  "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.",
  "WorthFuzzing": true
}

1/1

2026/08/02 14:37

action

read-patch-diff

Args:

null

Results:

{
  "PatchDiff": "commit 2cea66d19f5e61596d4d0326aee5752566f75bc7\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Sun Aug 2 14:37:35 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/Documentation/block/biovecs.rst b/Documentation/block/biovecs.rst\nindex 11126ed6f40f7..49da2147b7d86 100644\n--- a/Documentation/block/biovecs.rst\n+++ b/Documentation/block/biovecs.rst\n@@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec.\n In the new scheme of things, everything that must be mutated in order to\n partially complete a bio is segregated into struct bvec_iter: bi_sector,\n bi_size and bi_idx have been moved there; and instead of modifying bv_offset\n-and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of\n+and bv_len, struct bvec_iter has bi_offset, which represents the number of\n bytes completed in the current bvec.\n \n There are a bunch of new helper macros for hiding the gory details - in\n particular, presenting the illusion of partially completed biovecs so that\n-normal code doesn't have to deal with bi_bvec_done.\n+normal code doesn't have to deal with bi_offset.\n \n  * Driver code should no longer refer to biovecs directly; we now have\n    bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs,\n-   constructed from the raw biovecs but taking into account bi_bvec_done and\n+   constructed from the raw biovecs but taking into account bi_offset and\n    bi_size.\n \n    bio_for_each_segment() has been updated to take a bvec_iter argument\n@@ -101,7 +101,7 @@ Other implications:\n    I.e. instead of using bio_iovec_idx() (or bio-\u003ebi_iovec[bio-\u003ebi_idx]), you\n    now use bio_iter_iovec(), which takes a bvec_iter and returns a\n    literal struct bio_vec - constructed on the fly from the raw biovec but\n-   taking into account bi_bvec_done (and bi_size).\n+   taking into account bi_offset (and bi_size).\n \n  * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that\n    doesn't actually own the bio. The reason is twofold: firstly, it's not\ndiff --git a/block/bio.c b/block/bio.c\nindex 500389f332d99..1602eab05761f 100644\n--- a/block/bio.c\n+++ b/block/bio.c\n@@ -229,7 +229,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,\n \tbio-\u003ebi_iter.bi_sector = 0;\n \tbio-\u003ebi_iter.bi_size = 0;\n \tbio-\u003ebi_iter.bi_idx = 0;\n-\tbio-\u003ebi_iter.bi_bvec_done = 0;\n+\tbio-\u003ebi_iter.bi_offset = 0;\n \tbio-\u003ebi_end_io = NULL;\n \tbio-\u003ebi_private = NULL;\n #ifdef CONFIG_BLK_CGROUP\n@@ -861,6 +861,11 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)\n \tbio-\u003ebi_write_stream = bio_src-\u003ebi_write_stream;\n \tbio-\u003ebi_iter = bio_src-\u003ebi_iter;\n \n+\tif (op_is_dmabuf(bio-\u003ebi_opf))\n+\t\tbio-\u003ebi_dmabuf_map = bio_src-\u003ebi_dmabuf_map;\n+\telse\n+\t\tbio-\u003ebi_io_vec = bio_src-\u003ebi_io_vec;\n+\n \tif (bio-\u003ebi_bdev) {\n \t\tif (bio-\u003ebi_bdev == bio_src-\u003ebi_bdev \u0026\u0026\n \t\t    bio_flagged(bio_src, BIO_REMAPPED))\n@@ -902,8 +907,6 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,\n \t\tbio_put(bio);\n \t\treturn NULL;\n \t}\n-\tbio-\u003ebi_io_vec = bio_src-\u003ebi_io_vec;\n-\n \treturn bio;\n }\n EXPORT_SYMBOL(bio_alloc_clone);\n@@ -923,7 +926,7 @@ int bio_init_clone(struct block_device *bdev, struct bio *bio,\n {\n \tint ret;\n \n-\tbio_init(bio, bdev, bio_src-\u003ebi_io_vec, 0, bio_src-\u003ebi_opf);\n+\tbio_init(bio, bdev, NULL, 0, bio_src-\u003ebi_opf);\n \tret = __bio_clone(bio, bio_src, gfp);\n \tif (ret)\n \t\tbio_uninit(bio);\n@@ -1182,15 +1185,26 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)\n }\n EXPORT_SYMBOL_GPL(__bio_release_pages);\n \n-void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter)\n+bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter)\n {\n+\tif (!iov_iter_is_bvec(iter) \u0026\u0026 !iov_iter_is_dmabuf_map(iter))\n+\t\treturn false;\n+\n \tWARN_ON_ONCE(bio-\u003ebi_max_vecs);\n \n+\tstatic_assert(offsetof(struct bio, bi_io_vec) ==\n+\t\t      offsetof(struct bio, bi_dmabuf_map));\n+\tstatic_assert(offsetof(struct iov_iter, bvec) ==\n+\t\t      offsetof(struct iov_iter, dmabuf_map));\n+\n \tbio-\u003ebi_io_vec = (struct bio_vec *)iter-\u003ebvec;\n \tbio-\u003ebi_iter.bi_idx = 0;\n-\tbio-\u003ebi_iter.bi_bvec_done = iter-\u003eiov_offset;\n+\tbio-\u003ebi_iter.bi_offset = iter-\u003eiov_offset;\n \tbio-\u003ebi_iter.bi_size = iov_iter_count(iter);\n \tbio_set_flag(bio, BIO_CLONED);\n+\tif (iov_iter_is_dmabuf_map(iter))\n+\t\tbio-\u003ebi_opf |= REQ_NOMERGE | REQ_DMABUF;\n+\treturn true;\n }\n \n /*\n@@ -1285,10 +1299,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,\n \tif (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))\n \t\treturn -EIO;\n \n-\tif (iov_iter_is_bvec(iter)) {\n-\t\tbio_iov_bvec_set(bio, iter);\n-\n-\t\tif (!bio_iov_bvec_aligned(bio, mem_align_mask))\n+\tif (bio_iov_iter_set(bio, iter)) {\n+\t\tif (iov_iter_is_bvec(iter) \u0026\u0026\n+\t\t    !bio_iov_bvec_aligned(bio, mem_align_mask))\n \t\t\treturn -EINVAL;\n \n \t\tiov_iter_advance(iter, bio-\u003ebi_iter.bi_size);\ndiff --git a/block/blk-map.c b/block/blk-map.c\nindex 615d29bb840eb..9cb9605d1f62d 100644\n--- a/block/blk-map.c\n+++ b/block/blk-map.c\n@@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request *rq, const struct iov_iter *iter)\n \tbio = blk_rq_map_bio_alloc(rq, 0, GFP_KERNEL);\n \tif (!bio)\n \t\treturn -ENOMEM;\n-\tbio_iov_bvec_set(bio, iter);\n+\tbio_iov_iter_set(bio, iter);\n \n \tret = blk_rq_append_bio(rq, bio);\n \tif (ret)\ndiff --git a/block/blk-merge.c b/block/blk-merge.c\nindex ab1161ca69f1e..1beedc42a85d5 100644\n--- a/block/blk-merge.c\n+++ b/block/blk-merge.c\n@@ -9,6 +9,7 @@\n #include \u003clinux/blk-integrity.h\u003e\n #include \u003clinux/part_stat.h\u003e\n #include \u003clinux/blk-cgroup.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n \n #include \u003ctrace/events/block.h\u003e\n \n@@ -33,7 +34,7 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)\n \n \tbio_advance_iter(bio, \u0026iter, iter.bi_size);\n \n-\tif (!iter.bi_bvec_done)\n+\tif (!iter.bi_offset)\n \t\tidx = iter.bi_idx - 1;\n \telse\t/* in the middle of bvec */\n \t\tidx = iter.bi_idx;\n@@ -41,11 +42,11 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)\n \t*bv = bio-\u003ebi_io_vec[idx];\n \n \t/*\n-\t * iter.bi_bvec_done records actual length of the last bvec\n+\t * iter.bi_offset records actual length of the last bvec\n \t * if this bio ends in the middle of one io vector\n \t */\n-\tif (iter.bi_bvec_done)\n-\t\tbv-\u003ebv_len = iter.bi_bvec_done;\n+\tif (iter.bi_offset)\n+\t\tbv-\u003ebv_len = iter.bi_offset;\n }\n \n static inline bool bio_will_gap(struct request_queue *q,\n@@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,\n \treturn bv-\u003ebv_offset | (bvprv-\u003ebv_offset + bvprv-\u003ebv_len);\n }\n \n+static inline int bio_split_io_at_dmabuf(struct bio *bio,\n+\t\tconst struct queue_limits *lim, unsigned *segs,\n+\t\tunsigned max_bytes, unsigned len_align_mask,\n+\t\tunsigned start_align_mask)\n+{\n+\tunsigned bytes = min(bio-\u003ebi_iter.bi_size, max_bytes);\n+\tunsigned seg_shift = bio-\u003ebi_dmabuf_map-\u003eseg_shift;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset \u0026 ((1U \u003c\u003c seg_shift) - 1);\n+\n+\tif ((bio-\u003ebi_iter.bi_offset \u0026 start_align_mask) ||\n+\t    (bio-\u003ebi_iter.bi_size \u0026 len_align_mask))\n+\t\treturn -EINVAL;\n+\n+\t/* single contiguous range into the dma-buf */\n+\t*segs = 1;\n+\n+\tbytes = min(bytes, ((unsigned)lim-\u003emax_segments \u003c\u003c seg_shift) - offset);\n+\tif (bytes != bio-\u003ebi_iter.bi_size)\n+\t\treturn bytes;\n+\treturn 0;\n+}\n+\n /**\n  * bio_split_io_at - check if and where to split a bio\n  * @bio:  [in] bio to be split\n@@ -346,6 +369,19 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,\n \t\tlen_align_mask |= (bc-\u003ebc_key-\u003ecrypto_cfg.data_unit_size - 1);\n \t}\n \n+\tif (op_is_dmabuf(bio-\u003ebi_opf)) {\n+\t\tint ret;\n+\n+\t\tret = bio_split_io_at_dmabuf(bio, lim, \u0026nsegs, max_bytes,\n+\t\t\t\t\t     len_align_mask, start_align_mask);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t\tif (!ret)\n+\t\t\tgoto out;\n+\t\tbytes = ret;\n+\t\tgoto split;\n+\t}\n+\n \tbio_for_each_bvec(bv, bio, iter) {\n \t\tif (bv.bv_offset \u0026 start_align_mask ||\n \t\t    bv.bv_len \u0026 len_align_mask)\n@@ -376,6 +412,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,\n \t\tbvprvp = \u0026bvprv;\n \t}\n \n+out:\n \t*segs = nsegs;\n \tbio-\u003ebi_bvec_gap_bit = ffs(gaps);\n \treturn 0;\ndiff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c\nindex bfdb9ed707411..88fd9cbc951f4 100644\n--- a/block/blk-mq-dma.c\n+++ b/block/blk-mq-dma.c\n@@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,\n \t * one could be merged into it.  This typically happens when moving to\n \t * the next bio, but some callers also don't pack bvecs tight.\n \t */\n-\twhile (!iter-\u003eiter.bi_size || !iter-\u003eiter.bi_bvec_done) {\n+\twhile (!iter-\u003eiter.bi_size || !iter-\u003eiter.bi_offset) {\n \t\tstruct bio_vec next;\n \n \t\tif (!__blk_map_iter_next(iter))\ndiff --git a/block/blk.h b/block/blk.h\nindex eaac05815cb03..50abfd9328861 100644\n--- a/block/blk.h\n+++ b/block/blk.h\n@@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio *bio,\n \t\treturn true;\n \n \tbv = __bvec_iter_bvec(bio-\u003ebi_io_vec, bio-\u003ebi_iter);\n-\tif (bio-\u003ebi_iter.bi_size \u003e bv-\u003ebv_len - bio-\u003ebi_iter.bi_bvec_done)\n+\tif (bio-\u003ebi_iter.bi_size \u003e bv-\u003ebv_len - bio-\u003ebi_iter.bi_offset)\n \t\treturn true;\n \tif ((bv-\u003ebv_offset | bv-\u003ebv_len) \u0026 lim-\u003edma_alignment)\n \t\treturn true;\ndiff --git a/block/fops.c b/block/fops.c\nindex 3c2099dfef1d5..3c403afb0f646 100644\n--- a/block/fops.c\n+++ b/block/fops.c\n@@ -342,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,\n \tbio-\u003ebi_end_io = blkdev_bio_end_io_async;\n \tbio-\u003ebi_ioprio = iocb-\u003eki_ioprio;\n \n-\tif (iov_iter_is_bvec(iter)) {\n-\t\t/*\n-\t\t * Users don't rely on the iterator being in any particular\n-\t\t * state for async I/O returning -EIOCBQUEUED, hence we can\n-\t\t * avoid expensive iov_iter_advance(). Bypass\n-\t\t * bio_iov_iter_get_pages() and set the bvec directly.\n-\t\t */\n-\t\tbio_iov_bvec_set(bio, iter);\n-\t} else {\n+\t/*\n+\t * Users don't rely on the iterator being in any particular\n+\t * state for async I/O returning -EIOCBQUEUED, hence we can\n+\t * avoid expensive iov_iter_advance(). Bypass\n+\t * bio_iov_iter_get_pages() and set the bvec/dmabuf directly.\n+\t */\n+\tif (!bio_iov_iter_set(bio, iter)) {\n \t\tret = blkdev_iov_iter_get_pages(bio, iter, bdev);\n \t\tif (unlikely(ret))\n \t\t\tgoto out_bio_put;\n@@ -769,6 +767,9 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)\n \t\t\tret = direct_write_fallback(iocb, from, ret,\n \t\t\t\t\tblkdev_buffered_write(iocb, from));\n \t} else {\n+\t\tif (unlikely(iov_iter_is_dmabuf_map(from)))\n+\t\t\treturn -EOPNOTSUPP;\n+\n \t\t/*\n \t\t * Take i_rwsem and invalidate_lock to avoid racing with\n \t\t * set_blocksize changing i_blkbits/folio order and punching\n@@ -823,6 +824,8 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\tif (ret \u003c 0 || !count)\n \t\t\tgoto reexpand;\n \t}\n+\tif (unlikely(iov_iter_is_dmabuf_map(to)))\n+\t\treturn -EOPNOTSUPP;\n \n \t/*\n \t * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize\n@@ -926,6 +929,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc *desc)\n \treturn generic_file_mmap_prepare(desc);\n }\n \n+static int blkdev_init_dma_buf_io_ctx(struct file *file,\n+\t\t\t\t      struct dma_buf_io_ctx *ctx)\n+{\n+\tstruct block_device *bdev = file_bdev(file);\n+\tstruct gendisk *disk = bdev-\u003ebd_disk;\n+\n+\tif (!(file-\u003ef_flags \u0026 O_DIRECT))\n+\t\treturn -EINVAL;\n+\tif (!disk-\u003efops-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EINVAL;\n+\treturn disk-\u003efops-\u003einit_dma_buf_io_ctx(bdev, ctx);\n+}\n+\n const struct file_operations def_blk_fops = {\n \t.open\t\t= blkdev_open,\n \t.release\t= blkdev_release,\n@@ -944,6 +960,7 @@ const struct file_operations def_blk_fops = {\n \t.fallocate\t= blkdev_fallocate,\n \t.uring_cmd\t= blkdev_uring_cmd,\n \t.fop_flags\t= FOP_BUFFER_RASYNC | FOP_DONTCACHE,\n+\t.init_dma_buf_io_ctx = blkdev_init_dma_buf_io_ctx,\n };\n \n static __init int blkdev_init(void)\ndiff --git a/drivers/block/loop.c b/drivers/block/loop.c\nindex 26d7130c3f559..8639fa34b8470 100644\n--- a/drivers/block/loop.c\n+++ b/drivers/block/loop.c\n@@ -379,7 +379,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,\n \t\tiov_iter_bvec(\u0026iter, rw,\n \t\t\t__bvec_iter_bvec(rq-\u003ebio-\u003ebi_io_vec, rq-\u003ebio-\u003ebi_iter),\n \t\t\tnr_bvec, blk_rq_bytes(rq));\n-\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_bvec_done;\n+\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_offset;\n \t}\n \tatomic_set(\u0026cmd-\u003eref, 2);\n \ndiff --git a/drivers/block/zloop.c b/drivers/block/zloop.c\nindex f97a20cfdb7ce..0fef5db8e5b80 100644\n--- a/drivers/block/zloop.c\n+++ b/drivers/block/zloop.c\n@@ -554,7 +554,7 @@ static int zloop_do_rw(struct zloop_cmd *cmd)\n \t\tiov_iter_bvec(\u0026iter, rw,\n \t\t\t__bvec_iter_bvec(rq-\u003ebio-\u003ebi_io_vec, rq-\u003ebio-\u003ebi_iter),\n \t\t\t\t\tnr_bvec, blk_rq_bytes(rq));\n-\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_bvec_done;\n+\t\titer.iov_offset = rq-\u003ebio-\u003ebi_iter.bi_offset;\n \t}\n \n \tcmd-\u003eiocb.ki_pos = (cmd-\u003esector - zone-\u003estart) \u003c\u003c SECTOR_SHIFT;\ndiff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile\nindex b25d7550bacfd..523731b0f83ee 100644\n--- a/drivers/dma-buf/Makefile\n+++ b/drivers/dma-buf/Makefile\n@@ -1,6 +1,6 @@\n # SPDX-License-Identifier: GPL-2.0-only\n obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \\\n-\t dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o\n+\t dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o\n obj-$(CONFIG_DMABUF_HEAPS)\t+= dma-heap.o\n obj-$(CONFIG_DMABUF_HEAPS)\t+= heaps/\n obj-$(CONFIG_SYNC_FILE)\t\t+= sync_file.o\ndiff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c\nnew file mode 100644\nindex 0000000000000..24f58f80c45c4\n--- /dev/null\n+++ b/drivers/dma-buf/dma-buf-io.c\n@@ -0,0 +1,278 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/*\n+ * Common infrastructure for supporing dma-buf in the I/O path.\n+ *\n+ * Copyright (C) 2026 Pavel Begunkov \u003casml.silence@gmail.com\u003e\n+ */\n+#include \u003clinux/dma-buf-io.h\u003e\n+#include \u003clinux/dma-resv.h\u003e\n+\n+struct dma_buf_io_fence {\n+\tstruct dma_fence base;\n+\tspinlock_t lock;\n+};\n+\n+static const char *dma_buf_io_fence_drv_name(struct dma_fence *fence)\n+{\n+\t/* default fence release kfree's the base pointer */\n+\tBUILD_BUG_ON(offsetof(struct dma_buf_io_fence, base));\n+\n+\treturn \"dma-buf-io-ctx\";\n+}\n+\n+static const char *dma_buf_io_fence_timeline_name(struct dma_fence *fence)\n+{\n+\treturn \"dma-buf-io-ctx\";\n+}\n+\n+const struct dma_fence_ops dma_buf_io_fence_ops = {\n+\t.get_driver_name = dma_buf_io_fence_drv_name,\n+\t.get_timeline_name = dma_buf_io_fence_timeline_name,\n+};\n+\n+static void dma_buf_io_ctx_destroy_work(struct work_struct *work)\n+{\n+\tstruct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx,\n+\t\t\t\t\t\t  release_work);\n+\n+\tif (WARN_ON_ONCE(refcount_read(\u0026ctx-\u003erefs)))\n+\t\treturn;\n+\n+\tctx-\u003edev_ops-\u003erelease(ctx);\n+\tdma_buf_put(ctx-\u003edmabuf);\n+\tkfree(ctx);\n+}\n+\n+static void dma_buf_io_map_release_work(struct work_struct *work)\n+{\n+\tstruct dma_buf_io_map *map = container_of(work, struct dma_buf_io_map,\n+\t\t\t\t\t\t  release_work);\n+\tstruct dma_buf_io_fence *fence = map-\u003efence;\n+\tstruct dma_buf_io_ctx *ctx = map-\u003ectx;\n+\tstruct dma_buf *dmabuf = ctx-\u003edmabuf;\n+\n+\t/* the release path must wait for fences */\n+\tif (WARN_ON_ONCE(refcount_read(\u0026ctx-\u003erefs) == 0))\n+\t\treturn;\n+\n+\t/* Prevent from destoying the ctx while unmapping */\n+\trefcount_inc(\u0026ctx-\u003erefs);\n+\n+\t/*\n+\t * There are no more requests using the map, we can signal the fence.\n+\t * It should be done before taking the resv lock as someone could be\n+\t * waiting for the fence while holding the lock.\n+\t */\n+\tdma_fence_signal(\u0026fence-\u003ebase);\n+\n+\tdma_resv_lock(dmabuf-\u003eresv, NULL);\n+\tctx-\u003edev_ops-\u003eunmap(ctx, map);\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\n+\tdma_fence_put(\u0026fence-\u003ebase);\n+\tpercpu_ref_exit(\u0026map-\u003erefs);\n+\tkfree(map);\n+\n+\tif (refcount_dec_and_test(\u0026ctx-\u003erefs)) {\n+\t\t/*\n+\t\t * Destruction needs to wait for I/O and dma fences. Defer it to\n+\t\t * simplify locking.\n+\t\t */\n+\t\tINIT_WORK(\u0026ctx-\u003erelease_work, dma_buf_io_ctx_destroy_work);\n+\t\tqueue_work(system_wq, \u0026ctx-\u003erelease_work);\n+\t}\n+}\n+\n+static void dma_buf_io_map_refs_release(struct percpu_ref *ref)\n+{\n+\tstruct dma_buf_io_map *map = container_of(ref, struct dma_buf_io_map, refs);\n+\n+\t/* might sleep, use a worker */\n+\tINIT_WORK(\u0026map-\u003erelease_work, dma_buf_io_map_release_work);\n+\tqueue_work(system_wq, \u0026map-\u003erelease_work);\n+}\n+\n+int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map)\n+{\n+\tstruct dma_buf_io_fence *fence = NULL;\n+\tint ret;\n+\n+\tfence = kzalloc(sizeof(*fence), GFP_KERNEL);\n+\tif (!fence)\n+\t\treturn -ENOMEM;\n+\n+\tret = percpu_ref_init(\u0026map-\u003erefs, dma_buf_io_map_refs_release, 0, GFP_KERNEL);\n+\tif (ret) {\n+\t\tkfree(fence);\n+\t\treturn ret;\n+\t}\n+\n+\tspin_lock_init(\u0026fence-\u003elock);\n+\tdma_fence_init(\u0026fence-\u003ebase, \u0026dma_buf_io_fence_ops, \u0026fence-\u003elock,\n+\t\t\tctx-\u003efence_ctx, atomic_inc_return(\u0026ctx-\u003efence_seq));\n+\tmap-\u003efence = fence;\n+\tmap-\u003ectx = ctx;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, \"DMA_BUF\");\n+\n+struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx)\n+{\n+\tstruct dma_buf *dmabuf = ctx-\u003edmabuf;\n+\tstruct dma_buf_io_map *map;\n+\tlong ret;\n+\n+retry:\n+\t/*\n+\t * -\u003edmabuf_map() will be calling dma_buf_map_attachment(), for which\n+\t * we'll need to wait for fences. Do a bit nicer and try to wait\n+\t * without the resv lock first.\n+\t */\n+\tret = dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t    true, MAX_SCHEDULE_TIMEOUT);\n+\tif (!ret)\n+\t\tret = -EAGAIN;\n+\tif (ret \u003c 0)\n+\t\treturn ERR_PTR(ret);\n+\n+\tret = dma_resv_lock_interruptible(dmabuf-\u003eresv, NULL);\n+\tif (ret)\n+\t\treturn ERR_PTR(ret);\n+\n+\tmap = dma_buf_io_get_map(ctx);\n+\tif (map) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t  true, 0) \u003c 0) {\n+\t\tdma_resv_unlock(dmabuf-\u003eresv);\n+\t\tgoto retry;\n+\t}\n+\n+\tmap = ctx-\u003edev_ops-\u003emap(ctx);\n+\tif (IS_ERR(map)) {\n+\t\tret = PTR_ERR(map);\n+\t\tgoto out;\n+\t}\n+\n+\tif (WARN_ON_ONCE(!map-\u003eseg_shift))\n+\t\treturn ERR_PTR(-EFAULT);\n+\n+\tpercpu_ref_get(\u0026map-\u003erefs);\n+\trcu_assign_pointer(ctx-\u003emap, map);\n+out:\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\tif (ret \u003c 0)\n+\t\treturn ERR_PTR(ret);\n+\treturn map;\n+}\n+\n+static void dma_buf_io_drop_map(struct dma_buf_io_ctx *ctx)\n+{\n+\tstruct dma_buf *dmabuf = ctx-\u003edmabuf;\n+\tstruct dma_buf_io_map *map;\n+\tint ret;\n+\n+\tdma_resv_assert_held(dmabuf-\u003eresv);\n+\n+\tmap = rcu_dereference_protected(ctx-\u003emap,\n+\t\t\t\t\tdma_resv_held(dmabuf-\u003eresv));\n+\tif (!map)\n+\t\treturn;\n+\trcu_assign_pointer(ctx-\u003emap, NULL);\n+\n+\tret = dma_resv_reserve_fences(dmabuf-\u003eresv, 1);\n+\tif (WARN_ON_ONCE(ret)) {\n+\t\tstruct dma_fence *fence = \u0026map-\u003efence-\u003ebase;\n+\n+\t\tdma_fence_get(fence);\n+\t\tpercpu_ref_kill(\u0026map-\u003erefs);\n+\t\tdma_fence_wait(fence, false);\n+\t\tdma_fence_put(fence);\n+\t\treturn;\n+\t}\n+\n+\tdma_resv_add_fence(dmabuf-\u003eresv, \u0026map-\u003efence-\u003ebase,\n+\t\t\t   DMA_RESV_USAGE_KERNEL);\n+\t/*\n+\t * Delay destruction until all inflight requests using the map are\n+\t * gone. It'll also signal the fence then.\n+\t */\n+\tpercpu_ref_kill(\u0026map-\u003erefs);\n+}\n+\n+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx)\n+{\n+\tdma_buf_io_drop_map(ctx);\n+}\n+EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, \"DMA_BUF\");\n+\n+static void dma_buf_io_ctx_release_work(struct work_struct *work)\n+{\n+\tstruct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx,\n+\t\t\t\t\t\t  release_work);\n+\tstruct dma_buf *dmabuf = ctx-\u003edmabuf;\n+\tlong ret;\n+\n+\tdma_resv_lock(dmabuf-\u003eresv, NULL);\n+\t/* Remove the last map, there should be no new ones going forward. */\n+\tdma_buf_io_drop_map(ctx);\n+\tdma_resv_unlock(dmabuf-\u003eresv);\n+\n+\t/* Wait until all maps are destroyed. */\n+\tret = dma_resv_wait_timeout(dmabuf-\u003eresv, DMA_RESV_USAGE_KERNEL,\n+\t\t\t\t    false, MAX_SCHEDULE_TIMEOUT);\n+\n+\tif (WARN_ON_ONCE(ret \u003c= 0))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(rcu_dereference_protected(ctx-\u003emap, true)))\n+\t\treturn;\n+\n+\tif (refcount_dec_and_test(\u0026ctx-\u003erefs))\n+\t\tdma_buf_io_ctx_destroy_work(\u0026ctx-\u003erelease_work);\n+}\n+\n+void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx)\n+{\n+\t/*\n+\t * Destruction needs to wait for I/O and dma fences. Defer it to\n+\t * simplify locking.\n+\t */\n+\tINIT_WORK(\u0026ctx-\u003erelease_work, dma_buf_io_ctx_release_work);\n+\tqueue_work(system_wq, \u0026ctx-\u003erelease_work);\n+}\n+\n+int dma_buf_io_ctx_create(struct file *file,\n+\t\t\t   struct dma_buf_io_ctx *ctx,\n+\t\t\t   struct dma_buf *dmabuf,\n+\t\t\t   enum dma_data_direction dir)\n+{\n+\tint ret;\n+\n+\tif (!file-\u003ef_op-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EOPNOTSUPP;\n+\n+\tmemset(ctx, 0, sizeof(*ctx));\n+\tctx-\u003efence_ctx = dma_fence_context_alloc(1);\n+\tctx-\u003edir = dir;\n+\tctx-\u003edmabuf = dmabuf;\n+\trefcount_set(\u0026ctx-\u003erefs, 1);\n+\tget_dma_buf(dmabuf);\n+\n+\tret = file-\u003ef_op-\u003einit_dma_buf_io_ctx(file, ctx);\n+\tif (ret) {\n+\t\tmemset(ctx, 0, sizeof(*ctx));\n+\t\tdma_buf_put(dmabuf);\n+\t\treturn ret;\n+\t}\n+\n+\tif (WARN_ON_ONCE(!ctx-\u003edev_ops ||\n+\t\t\t !ctx-\u003edev_ops-\u003emap ||\n+\t\t\t !ctx-\u003edev_ops-\u003eunmap ||\n+\t\t\t !ctx-\u003edev_ops-\u003erelease))\n+\t\treturn -EINVAL;\n+\n+\treturn ret;\n+}\ndiff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c\nindex 6155b0117c9d6..b22719c6411cf 100644\n--- a/drivers/md/dm-io-rewind.c\n+++ b/drivers/md/dm-io-rewind.c\n@@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,\n \tint idx;\n \n \titer-\u003ebi_size += bytes;\n-\tif (bytes \u003c= iter-\u003ebi_bvec_done) {\n-\t\titer-\u003ebi_bvec_done -= bytes;\n+\tif (bytes \u003c= iter-\u003ebi_offset) {\n+\t\titer-\u003ebi_offset -= bytes;\n \t\treturn true;\n \t}\n \n-\tbytes -= iter-\u003ebi_bvec_done;\n+\tbytes -= iter-\u003ebi_offset;\n \tidx = iter-\u003ebi_idx - 1;\n \n \twhile (idx \u003e= 0 \u0026\u0026 bytes \u0026\u0026 bytes \u003e bv[idx].bv_len) {\n@@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,\n \tif (WARN_ONCE(idx \u003c 0 \u0026\u0026 bytes,\n \t\t      \"Attempted to rewind iter beyond bvec's boundaries\\n\")) {\n \t\titer-\u003ebi_size -= bytes;\n-\t\titer-\u003ebi_bvec_done = 0;\n+\t\titer-\u003ebi_offset = 0;\n \t\titer-\u003ebi_idx = 0;\n \t\treturn false;\n \t}\n \n \titer-\u003ebi_idx = idx;\n-\titer-\u003ebi_bvec_done = bv[idx].bv_len - bytes;\n+\titer-\u003ebi_offset = bv[idx].bv_len - bytes;\n \treturn true;\n }\n \n@@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bio *bio,\n \titer-\u003ebi_sector -= bytes \u003e\u003e 9;\n \n \t/* No advance means no rewind */\n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size += bytes;\n-\telse\n+\t\titer-\u003ebi_offset -= bytes;\n+\t} else {\n \t\tdm_bvec_iter_rewind(bio-\u003ebi_io_vec, iter, bytes);\n+\t}\n }\n \n /**\ndiff --git a/drivers/md/dm-pcache/segment.c b/drivers/md/dm-pcache/segment.c\nindex 7e98187014451..8f8816e1c539f 100644\n--- a/drivers/md/dm-pcache/segment.c\n+++ b/drivers/md/dm-pcache/segment.c\n@@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment *segment,\n \n \tiov_iter_bvec(\u0026iter, ITER_DEST, \u0026bio-\u003ebi_io_vec[bio-\u003ebi_iter.bi_idx],\n \t\t\tbio_segments(bio), bio-\u003ebi_iter.bi_size);\n-\titer.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\titer.iov_offset = bio-\u003ebi_iter.bi_offset;\n \tif (bio_off)\n \t\tiov_iter_advance(\u0026iter, bio_off);\n \n@@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment *segment,\n \n \tiov_iter_bvec(\u0026iter, ITER_SOURCE, \u0026bio-\u003ebi_io_vec[bio-\u003ebi_iter.bi_idx],\n \t\t\tbio_segments(bio), bio-\u003ebi_iter.bi_size);\n-\titer.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\titer.iov_offset = bio-\u003ebi_iter.bi_offset;\n \tif (bio_off)\n \t\tiov_iter_advance(\u0026iter, bio_off);\n \ndiff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c\nindex 7e1112960d7f8..5d910a64503d3 100644\n--- a/drivers/nvdimm/btt.c\n+++ b/drivers/nvdimm/btt.c\n@@ -1155,7 +1155,7 @@ static int btt_rw_integrity(struct btt *btt, struct bio_integrity_payload *bip,\n \t\tbv = bvec_iter_bvec(bip-\u003ebip_vec, bip-\u003ebip_iter);\n \t\t/*\n \t\t * The 'bv' obtained from bvec_iter_bvec has its .bv_len and\n-\t\t * .bv_offset already adjusted for iter-\u003ebi_bvec_done, and we\n+\t\t * .bv_offset already adjusted for iter-\u003ebi_offset, and we\n \t\t * can use those directly\n \t\t */\n \ndiff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c\nindex 453c1f0b2dd09..8bd407603d928 100644\n--- a/drivers/nvme/host/core.c\n+++ b/drivers/nvme/host/core.c\n@@ -2676,6 +2676,17 @@ static int nvme_report_zones(struct gendisk *disk, sector_t sector,\n #define nvme_report_zones\tNULL\n #endif /* CONFIG_BLK_DEV_ZONED */\n \n+static int nvme_init_dma_buf_io_ctx(struct block_device *bdev,\n+\t\tstruct dma_buf_io_ctx *ctx)\n+{\n+\tstruct nvme_ns *ns = bdev-\u003ebd_disk-\u003eprivate_data;\n+\tstruct nvme_ctrl *ctrl = ns-\u003ectrl;\n+\n+\tif (!ctrl-\u003eops-\u003einit_dma_buf_io_ctx)\n+\t\treturn -EINVAL;\n+\treturn ctrl-\u003eops-\u003einit_dma_buf_io_ctx(ctrl, ctx);\n+}\n+\n const struct block_device_operations nvme_bdev_ops = {\n \t.owner\t\t= THIS_MODULE,\n \t.ioctl\t\t= nvme_ioctl,\n@@ -2686,6 +2697,7 @@ const struct block_device_operations nvme_bdev_ops = {\n \t.get_unique_id\t= nvme_get_unique_id,\n \t.report_zones\t= nvme_report_zones,\n \t.pr_ops\t\t= \u0026nvme_pr_ops,\n+\t.init_dma_buf_io_ctx = nvme_init_dma_buf_io_ctx,\n };\n \n static int nvme_wait_ready(struct nvme_ctrl *ctrl, u32 mask, u32 val,\ndiff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h\nindex 824651cc898db..034c31af8fecf 100644\n--- a/drivers/nvme/host/nvme.h\n+++ b/drivers/nvme/host/nvme.h\n@@ -652,6 +652,8 @@ struct nvme_ctrl_ops {\n \tint (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);\n \tvoid (*print_device_info)(struct nvme_ctrl *ctrl);\n \tbool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);\n+\tint (*init_dma_buf_io_ctx)(struct nvme_ctrl *ctrl,\n+\t\t\t\t   struct dma_buf_io_ctx *ctx);\n \tunsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin);\n };\n \ndiff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c\nindex 69932d640b537..67e26d6e8f255 100644\n--- a/drivers/nvme/host/pci.c\n+++ b/drivers/nvme/host/pci.c\n@@ -27,6 +27,8 @@\n #include \u003clinux/io-64-nonatomic-lo-hi.h\u003e\n #include \u003clinux/io-64-nonatomic-hi-lo.h\u003e\n #include \u003clinux/sed-opal.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n+#include \u003clinux/dma-resv.h\u003e\n \n #include \"trace.h\"\n #include \"nvme.h\"\n@@ -393,6 +395,13 @@ struct nvme_queue {\n \tstruct completion delete_done;\n };\n \n+struct nvme_dmabuf_map {\n+\tstruct dma_buf_io_map base;\n+\tstruct sg_table *sgt;\n+\tunsigned nr_entries;\n+\tdma_addr_t dma_list[];\n+};\n+\n /* bits for iod-\u003eflags */\n enum nvme_iod_flags {\n \t/* this command has been aborted by the timeout handler */\n@@ -859,6 +868,138 @@ static void nvme_free_descriptors(struct request *req)\n \t}\n }\n \n+static inline struct nvme_dmabuf_map *\n+to_nvme_dmabuf_map(struct dma_buf_io_map *map)\n+{\n+\treturn container_of(map, struct nvme_dmabuf_map, base);\n+}\n+\n+static void nvme_dmabuf_map_sync_for_cpu(struct nvme_dev *nvme_dev,\n+\t\t\t\t\t struct request *req)\n+{\n+\tstruct device *dev = nvme_dev-\u003edev;\n+\tenum dma_data_direction dma_dir;\n+\tstruct bio *bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tdma_addr_t *dma_list = map-\u003edma_list;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;\n+\tint length = blk_rq_payload_bytes(req) +\n+\t\t     (offset \u0026 (NVME_CTRL_PAGE_SIZE - 1));\n+\n+\tdma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;\n+\n+\twhile (length \u003e 0) {\n+\t\tdma_sync_single_for_cpu(dev, dma_list[map_idx++],\n+\t\t\t\t\tNVME_CTRL_PAGE_SIZE, dma_dir);\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t}\n+}\n+\n+static void nvme_dmabuf_map_sync_for_device(struct nvme_dev *nvme_dev,\n+\t\t\t\t\t    struct request *req)\n+{\n+\tstruct device *dev = nvme_dev-\u003edev;\n+\tenum dma_data_direction dma_dir;\n+\tstruct bio *bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tdma_addr_t *dma_list = map-\u003edma_list;\n+\tunsigned offset = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;\n+\tint length = blk_rq_payload_bytes(req) +\n+\t\t     (offset \u0026 (NVME_CTRL_PAGE_SIZE - 1));\n+\n+\tdma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;\n+\n+\twhile (length \u003e 0) {\n+\t\tdma_sync_single_for_device(dev, dma_list[map_idx++],\n+\t\t\t\t\t   NVME_CTRL_PAGE_SIZE, dma_dir);\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t}\n+}\n+\n+static void nvme_rq_clean_dmabuf_map(struct nvme_dev *dev,\n+\t\t\t\t      struct request *req)\n+{\n+\tstruct nvme_iod *iod = blk_mq_rq_to_pdu(req);\n+\n+\tnvme_dmabuf_map_sync_for_cpu(dev, req);\n+\n+\tif (!(iod-\u003eflags \u0026 IOD_SINGLE_SEGMENT))\n+\t\tnvme_free_descriptors(req);\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req,\n+\t\t\t\t\t     struct nvme_queue *nvmeq)\n+{\n+\tstruct nvme_iod *iod = blk_mq_rq_to_pdu(req);\n+\tstruct bio *bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tunsigned bvec_done = bio-\u003ebi_iter.bi_offset;\n+\tunsigned map_idx = bvec_done / NVME_CTRL_PAGE_SIZE;\n+\tunsigned offset = bvec_done \u0026 (NVME_CTRL_PAGE_SIZE - 1);\n+\tint length = blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset);\n+\tdma_addr_t *dma_list = map-\u003edma_list;\n+\tu64 prp1_dma = dma_list[map_idx++] + offset;\n+\tu64 dma_addr, prp2_dma;\n+\tdma_addr_t prp_dma;\n+\t__le64 *prp_list;\n+\tunsigned i;\n+\n+\tnvme_dmabuf_map_sync_for_device(nvmeq-\u003edev, req);\n+\n+\tif (length \u003c= 0) {\n+\t\tprp2_dma = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tif (length \u003c= NVME_CTRL_PAGE_SIZE) {\n+\t\tprp2_dma = dma_list[map_idx];\n+\t\tgoto done;\n+\t}\n+\n+\tif (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) \u003c=\n+\t    NVME_SMALL_POOL_SIZE / sizeof(__le64))\n+\t\tiod-\u003eflags |= IOD_SMALL_DESCRIPTOR;\n+\n+\tprp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,\n+\t\t\t\u0026prp_dma);\n+\tif (!prp_list)\n+\t\treturn BLK_STS_RESOURCE;\n+\n+\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = prp_list;\n+\tprp2_dma = prp_dma;\n+\ti = 0;\n+\tfor (;;) {\n+\t\tif (i == NVME_CTRL_PAGE_SIZE \u003e\u003e 3) {\n+\t\t\t__le64 *old_prp_list = prp_list;\n+\n+\t\t\tprp_list = dma_pool_alloc(nvmeq-\u003edescriptor_pools.large,\n+\t\t\t\t\tGFP_ATOMIC, \u0026prp_dma);\n+\t\t\tif (!prp_list)\n+\t\t\t\tgoto free_prps;\n+\t\t\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = prp_list;\n+\t\t\tprp_list[0] = old_prp_list[i - 1];\n+\t\t\told_prp_list[i - 1] = cpu_to_le64(prp_dma);\n+\t\t\ti = 1;\n+\t\t}\n+\n+\t\tdma_addr = dma_list[map_idx++];\n+\t\tprp_list[i++] = cpu_to_le64(dma_addr);\n+\n+\t\tlength -= NVME_CTRL_PAGE_SIZE;\n+\t\tif (length \u003c= 0)\n+\t\t\tbreak;\n+\t}\n+done:\n+\tiod-\u003ecmd.common.dptr.prp1 = cpu_to_le64(prp1_dma);\n+\tiod-\u003ecmd.common.dptr.prp2 = cpu_to_le64(prp2_dma);\n+\treturn BLK_STS_OK;\n+free_prps:\n+\tnvme_free_descriptors(req);\n+\treturn BLK_STS_RESOURCE;\n+}\n+\n static void nvme_free_prps(struct request *req, unsigned int attrs)\n {\n \tstruct nvme_iod *iod = blk_mq_rq_to_pdu(req);\n@@ -937,6 +1078,11 @@ static void nvme_unmap_data(struct request *req)\n \tstruct device *dma_dev = nvmeq-\u003edev-\u003edev;\n \tunsigned int attrs = 0;\n \n+\tif (blk_mq_rq_is_dmabuf(req)) {\n+\t\tnvme_rq_clean_dmabuf_map(nvmeq-\u003edev, req);\n+\t\treturn;\n+\t}\n+\n \tif (iod-\u003eflags \u0026 IOD_SINGLE_SEGMENT) {\n \t\tstatic_assert(offsetof(union nvme_data_ptr, prp1) ==\n \t\t\t\toffsetof(union nvme_data_ptr, sgl.addr));\n@@ -1142,13 +1288,19 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,\n }\n \n static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,\n-\t\tstruct blk_dma_iter *iter)\n+\t\tdma_addr_t addr, u32 len)\n {\n-\tsge-\u003eaddr = cpu_to_le64(iter-\u003eaddr);\n-\tsge-\u003elength = cpu_to_le32(iter-\u003elen);\n+\tsge-\u003eaddr = cpu_to_le64(addr);\n+\tsge-\u003elength = cpu_to_le32(len);\n \tsge-\u003etype = NVME_SGL_FMT_DATA_DESC \u003c\u003c 4;\n }\n \n+static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge,\n+\t\tstruct blk_dma_iter *iter)\n+{\n+\tnvme_pci_sgl_set_data(sge, iter-\u003eaddr, iter-\u003elen);\n+}\n+\n static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,\n \t\tdma_addr_t dma_addr, int entries)\n {\n@@ -1157,6 +1309,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,\n \tsge-\u003etype = NVME_SGL_FMT_LAST_SEG_DESC \u003c\u003c 4;\n }\n \n+static unsigned int nvme_pci_dmabuf_sgl_nents(struct request *req)\n+{\n+\tstruct bio *bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tstruct scatterlist *sg;\n+\tunsigned long tmp;\n+\tsize_t offset = bio-\u003ebi_iter.bi_offset;\n+\tsize_t remaining = blk_rq_payload_bytes(req);\n+\tunsigned int nents = 0;\n+\n+\tfor_each_sgtable_dma_sg(map-\u003esgt, sg, tmp) {\n+\t\tsize_t sg_len = sg_dma_len(sg);\n+\n+\t\tif (!remaining)\n+\t\t\tbreak;\n+\t\tif (offset \u003e= sg_len) {\n+\t\t\toffset -= sg_len;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tsg_len -= offset;\n+\t\toffset = 0;\n+\n+\t\tdo {\n+\t\t\tsize_t chunk = min(remaining, sg_len);\n+\n+\t\t\tnents++;\n+\t\t\tsg_len -= chunk;\n+\t\t\tremaining -= chunk;\n+\t\t} while (sg_len \u0026\u0026 remaining);\n+\t}\n+\n+\tif (unlikely(remaining))\n+\t\treturn 0;\n+\n+\treturn nents;\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request *req,\n+\t\tstruct nvme_queue *nvmeq, unsigned int entries)\n+{\n+\tstruct nvme_iod *iod = blk_mq_rq_to_pdu(req);\n+\tstruct bio *bio = req-\u003ebio;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio-\u003ebi_dmabuf_map);\n+\tsize_t length = blk_rq_payload_bytes(req);\n+\tstruct nvme_sgl_desc *sg_list = \u0026iod-\u003ecmd.common.dptr.sgl;\n+\tbool pooled = entries \u003e 1;\n+\tdma_addr_t sgl_dma = 0;\n+\tunsigned int mapped = 0;\n+\tunsigned long tmp;\n+\tstruct scatterlist *sg;\n+\tsize_t offset = bio-\u003ebi_iter.bi_offset;\n+\tsize_t remaining = length;\n+\n+\tif (!entries)\n+\t\treturn BLK_STS_IOERR;\n+\n+\tiod-\u003ecmd.common.flags = NVME_CMD_SGL_METABUF;\n+\tiod-\u003etotal_len = length;\n+\n+\tnvme_dmabuf_map_sync_for_device(nvmeq-\u003edev, req);\n+\n+\t/* entries == 1 fits in the inline descriptor; more needs a pool. */\n+\tif (pooled) {\n+\t\tif (entries \u003c= NVME_SMALL_POOL_SIZE / sizeof(*sg_list))\n+\t\t\tiod-\u003eflags |= IOD_SMALL_DESCRIPTOR;\n+\n+\t\tsg_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,\n+\t\t\t\t\t \u0026sgl_dma);\n+\t\tif (!sg_list)\n+\t\t\treturn BLK_STS_RESOURCE;\n+\t\tiod-\u003edescriptors[iod-\u003enr_descriptors++] = sg_list;\n+\t}\n+\n+\tfor_each_sgtable_dma_sg(map-\u003esgt, sg, tmp) {\n+\t\tsize_t sg_len = sg_dma_len(sg);\n+\t\tdma_addr_t addr = sg_dma_address(sg);\n+\n+\t\tif (!remaining)\n+\t\t\tbreak;\n+\t\tif (offset \u003e= sg_len) {\n+\t\t\toffset -= sg_len;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\taddr += offset;\n+\t\tsg_len -= offset;\n+\t\toffset = 0;\n+\n+\t\tdo {\n+\t\t\tu32 chunk = min_t(size_t, remaining, sg_len);\n+\n+\t\t\tif (WARN_ON_ONCE(mapped == entries))\n+\t\t\t\tgoto err_free;\n+\t\t\tnvme_pci_sgl_set_data(\u0026sg_list[mapped++], addr, chunk);\n+\n+\t\t\taddr += chunk;\n+\t\t\tsg_len -= chunk;\n+\t\t\tremaining -= chunk;\n+\t\t} while (sg_len \u0026\u0026 remaining);\n+\t}\n+\n+\tif (unlikely(remaining))\n+\t\tgoto err_free;\n+\n+\tif (pooled)\n+\t\tnvme_pci_sgl_set_seg(\u0026iod-\u003ecmd.common.dptr.sgl, sgl_dma,\n+\t\t\t\t     mapped);\n+\treturn BLK_STS_OK;\n+err_free:\n+\tif (pooled) {\n+\t\tiod-\u003enr_descriptors--;\n+\t\tdma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma);\n+\t}\n+\treturn BLK_STS_IOERR;\n+}\n+\n+static blk_status_t nvme_rq_setup_dmabuf(struct request *req,\n+\t\tstruct nvme_queue *nvmeq, enum nvme_use_sgl use_sgl)\n+{\n+\tunsigned int entries;\n+\tsize_t avg_seg;\n+\n+\tif (use_sgl == SGL_UNSUPPORTED)\n+\t\treturn nvme_rq_setup_dmabuf_map(req, nvmeq);\n+\n+\tentries = nvme_pci_dmabuf_sgl_nents(req);\n+\n+\tif (use_sgl == SGL_FORCED)\n+\t\treturn nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);\n+\n+\tavg_seg = entries ?\n+\t\tDIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0;\n+\tif (sgl_threshold \u0026\u0026 avg_seg \u003e= sgl_threshold)\n+\t\treturn nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);\n+\n+\treturn nvme_rq_setup_dmabuf_map(req, nvmeq);\n+}\n+\n static blk_status_t nvme_pci_setup_data_sgl(struct request *req,\n \t\tstruct blk_dma_iter *iter)\n {\n@@ -1171,7 +1462,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req,\n \tiod-\u003ecmd.common.flags = NVME_CMD_SGL_METABUF;\n \n \tif (entries == 1 || blk_rq_dma_map_coalesce(\u0026iod-\u003edma_state)) {\n-\t\tnvme_pci_sgl_set_data(\u0026iod-\u003ecmd.common.dptr.sgl, iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026iod-\u003ecmd.common.dptr.sgl, iter);\n \t\tiod-\u003etotal_len += iter-\u003elen;\n \t\treturn BLK_STS_OK;\n \t}\n@@ -1193,7 +1484,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req,\n \t\t\titer-\u003estatus = BLK_STS_IOERR;\n \t\t\tbreak;\n \t\t}\n-\t\tnvme_pci_sgl_set_data(\u0026sg_list[mapped++], iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026sg_list[mapped++], iter);\n \t\tiod-\u003etotal_len += iter-\u003elen;\n \t} while (blk_rq_dma_map_iter_next(req, nvmeq-\u003edev-\u003edev, iter));\n \n@@ -1251,6 +1542,9 @@ static blk_status_t nvme_map_data(struct request *req)\n \tstruct blk_dma_iter iter;\n \tblk_status_t ret;\n \n+\tif (blk_mq_rq_is_dmabuf(req))\n+\t\treturn nvme_rq_setup_dmabuf(req, nvmeq, use_sgl);\n+\n \t/*\n \t * Try to skip the DMA iterator for single segment requests, as that\n \t * significantly improves performances for small I/O sizes.\n@@ -1353,13 +1647,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request *req)\n \tiod-\u003ecmd.common.metadata = cpu_to_le64(sgl_dma);\n \tif (entries == 1) {\n \t\tiod-\u003emeta_total_len = iter.len;\n-\t\tnvme_pci_sgl_set_data(sg_list, \u0026iter);\n+\t\tnvme_pci_dma_iter_set_sgl(sg_list, \u0026iter);\n \t\treturn BLK_STS_OK;\n \t}\n \n \tsgl_dma += sizeof(*sg_list);\n \tdo {\n-\t\tnvme_pci_sgl_set_data(\u0026sg_list[++i], \u0026iter);\n+\t\tnvme_pci_dma_iter_set_sgl(\u0026sg_list[++i], \u0026iter);\n \t\tiod-\u003emeta_total_len += iter.len;\n \t} while (blk_rq_integrity_dma_map_iter_next(req, dev-\u003edev, \u0026iter));\n \n@@ -2269,6 +2563,118 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)\n \treturn result;\n }\n \n+#ifdef CONFIG_DMA_SHARED_BUFFER\n+static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *attach)\n+{\n+\tstruct dma_buf_io_ctx *ctx = attach-\u003eimporter_priv;\n+\n+\tdma_buf_io_invalidate_mappings(ctx);\n+}\n+\n+const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = {\n+\t.invalidate_mappings\t= nvme_dmabuf_invalidate_mappings,\n+\t.allow_peer2peer\t= true,\n+};\n+\n+static struct dma_buf_io_map *nvme_dma_buf_io_map(struct dma_buf_io_ctx *ctx)\n+{\n+\tunsigned nr_entries = ctx-\u003edmabuf-\u003esize / NVME_CTRL_PAGE_SIZE;\n+\tstruct dma_buf_attachment *attach = ctx-\u003edev_priv;\n+\tunsigned long tmp, i = 0;\n+\tunsigned seg_shift = ~0U;\n+\tstruct nvme_dmabuf_map *map;\n+\tstruct scatterlist *sg;\n+\tstruct sg_table *sgt;\n+\tint ret;\n+\n+\tdma_resv_assert_held(ctx-\u003edmabuf-\u003eresv);\n+\n+\tmap = kmalloc_flex(*map, dma_list, nr_entries);\n+\tif (!map)\n+\t\treturn ERR_PTR(-ENOMEM);\n+\n+\tsgt = dma_buf_map_attachment(attach, ctx-\u003edir);\n+\tif (IS_ERR(sgt)) {\n+\t\tret = PTR_ERR(sgt);\n+\t\tsgt = NULL;\n+\t\tgoto err;\n+\t}\n+\n+\tfor_each_sgtable_dma_sg(sgt, sg, tmp) {\n+\t\tdma_addr_t dma_addr = sg_dma_address(sg);\n+\t\tunsigned long sg_len = sg_dma_len(sg);\n+\n+\t\tif (sg_len % NVME_CTRL_PAGE_SIZE) {\n+\t\t\tret = -EINVAL;\n+\t\t\tgoto err;\n+\t\t}\n+\t\tseg_shift = min(seg_shift, __ffs(sg_len));\n+\n+\t\twhile (sg_len) {\n+\t\t\tmap-\u003edma_list[i++] = dma_addr;\n+\t\t\tdma_addr += NVME_CTRL_PAGE_SIZE;\n+\t\t\tsg_len -= NVME_CTRL_PAGE_SIZE;\n+\t\t}\n+\t}\n+\n+\tif (WARN_ON_ONCE(seg_shift \u003c NVME_CTRL_PAGE_SHIFT))\n+\t\treturn ERR_PTR(-EFAULT);\n+\n+\tret = dma_buf_io_init_map(ctx, \u0026map-\u003ebase);\n+\tif (ret)\n+\t\tgoto err;\n+\tmap-\u003ebase.seg_shift = seg_shift;\n+\tmap-\u003enr_entries = nr_entries;\n+\tmap-\u003esgt = sgt;\n+\treturn \u0026map-\u003ebase;\n+err:\n+\tif (sgt)\n+\t\tdma_buf_unmap_attachment(attach, sgt, ctx-\u003edir);\n+\tkfree(map);\n+\treturn ERR_PTR(ret);\n+}\n+\n+static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx *ctx,\n+\t\t\t\t  struct dma_buf_io_map *map_base)\n+{\n+\tstruct dma_buf_attachment *attach = ctx-\u003edev_priv;\n+\tstruct nvme_dmabuf_map *map = to_nvme_dmabuf_map(map_base);\n+\n+\tdma_resv_assert_held(ctx-\u003edmabuf-\u003eresv);\n+\n+\tdma_buf_unmap_attachment(attach, map-\u003esgt, ctx-\u003edir);\n+}\n+\n+static void nvme_dma_buf_io_release(struct dma_buf_io_ctx *ctx)\n+{\n+\tstruct dma_buf_attachment *attach = ctx-\u003edev_priv;\n+\n+\tdma_buf_detach(ctx-\u003edmabuf, attach);\n+}\n+\n+const struct dma_buf_io_ops nvme_dma_buf_io_ops = {\n+\t.map\t\t= nvme_dma_buf_io_map,\n+\t.unmap\t\t= nvme_dma_buf_io_unmap,\n+\t.release\t= nvme_dma_buf_io_release,\n+};\n+\n+static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl,\n+\t\t\t\t\tstruct dma_buf_io_ctx *ctx)\n+{\n+\tstruct dma_buf_attachment *attach;\n+\tstruct nvme_dev *dev = to_nvme_dev(ctrl);\n+\n+\tattach = dma_buf_dynamic_attach(ctx-\u003edmabuf, dev-\u003edev,\n+\t\t\t\t\t\u0026nvme_dmabuf_importer_ops, ctx);\n+\tif (IS_ERR(attach))\n+\t\treturn PTR_ERR(attach);\n+\n+\tctx-\u003edev_priv = attach;\n+\tctx-\u003edev_ops = \u0026nvme_dma_buf_io_ops;\n+\treturn 0;\n+}\n+#endif\n+\n static const struct blk_mq_ops nvme_mq_admin_ops = {\n \t.queue_rq\t= nvme_queue_rq,\n \t.complete\t= nvme_pci_complete_rq,\n@@ -3563,6 +3969,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {\n \t.print_device_info\t= nvme_pci_print_device_info,\n \t.supports_pci_p2pdma\t= nvme_pci_supports_pci_p2pdma,\n \t.get_virt_boundary\t= nvme_pci_get_virt_boundary,\n+#ifdef CONFIG_DMA_SHARED_BUFFER\n+\t.init_dma_buf_io_ctx\t= nvme_pci_init_dma_buf_io_ctx,\n+#endif\n };\n \n static int nvme_dev_map(struct nvme_dev *dev)\n@@ -4327,5 +4736,6 @@ MODULE_AUTHOR(\"Matthew Wilcox \u003cwilly@linux.intel.com\u003e\");\n MODULE_LICENSE(\"GPL\");\n MODULE_VERSION(\"1.0\");\n MODULE_DESCRIPTION(\"NVMe host PCIe transport driver\");\n+MODULE_IMPORT_NS(\"DMA_BUF\");\n module_init(nvme_init);\n module_exit(nvme_exit);\ndiff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c\nindex ba5c7b3e2a7c6..ce03a0ea4ded7 100644\n--- a/drivers/nvme/host/tcp.c\n+++ b/drivers/nvme/host/tcp.c\n@@ -357,7 +357,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req,\n \t\tiov_iter_bvec(\u0026req-\u003eiter, dir,\n \t\t\t__bvec_iter_bvec(bio-\u003ebi_io_vec, bio-\u003ebi_iter), nr_bvec,\n \t\t\tbio-\u003ebi_iter.bi_size);\n-\t\treq-\u003eiter.iov_offset = bio-\u003ebi_iter.bi_bvec_done;\n+\t\treq-\u003eiter.iov_offset = bio-\u003ebi_iter.bi_offset;\n \t}\n }\n \ndiff --git a/fs/btrfs/misc.h b/fs/btrfs/misc.h\nindex 694be6d0562a4..8020609431801 100644\n--- a/fs/btrfs/misc.h\n+++ b/fs/btrfs/misc.h\n@@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio *bio)\n \t\t.bi_sector = 0,\n \t\t.bi_size = bio_size,\n \t\t.bi_idx = 0,\n-\t\t.bi_bvec_done = 0,\n+\t\t.bi_offset = 0,\n \t};\n }\n \ndiff --git a/include/linux/bio.h b/include/linux/bio.h\nindex 0445ecba3b247..22ce9deb2febb 100644\n--- a/include/linux/bio.h\n+++ b/include/linux/bio.h\n@@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio *bio)\n {\n \treturn bio_op(bio) == REQ_OP_DISCARD ||\n \t       bio_op(bio) == REQ_OP_SECURE_ERASE ||\n-\t       bio_op(bio) == REQ_OP_WRITE_ZEROES;\n+\t       bio_op(bio) == REQ_OP_WRITE_ZEROES ||\n+\t       op_is_dmabuf(bio-\u003ebi_opf);\n }\n \n static inline void *bio_data(struct bio *bio)\n@@ -113,11 +114,13 @@ static inline void bio_advance_iter(const struct bio *bio,\n {\n \titer-\u003ebi_sector += bytes \u003e\u003e 9;\n \n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size -= bytes;\n-\telse\n+\t\titer-\u003ebi_offset += bytes;\n+\t} else {\n \t\tbvec_iter_advance(bio-\u003ebi_io_vec, iter, bytes);\n \t\t/* TODO: It is reasonable to complete bio with error here. */\n+\t}\n }\n \n /* @bytes should be less or equal to bvec[i-\u003ebi_idx].bv_len */\n@@ -127,10 +130,12 @@ static inline void bio_advance_iter_single(const struct bio *bio,\n {\n \titer-\u003ebi_sector += bytes \u003e\u003e 9;\n \n-\tif (bio_no_advance_iter(bio))\n+\tif (bio_no_advance_iter(bio)) {\n \t\titer-\u003ebi_size -= bytes;\n-\telse\n+\t\titer-\u003ebi_offset += bytes;\n+\t} else {\n \t\tbvec_iter_advance_single(bio-\u003ebi_io_vec, iter, bytes);\n+\t}\n }\n \n void __bio_advance(struct bio *, unsigned bytes);\n@@ -434,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio *bio)\n \n /*\n  * Calculate number of bvec segments that should be allocated to fit data\n- * pointed by @iter. If @iter is backed by bvec it's going to be reused\n- * instead of allocating a new one.\n+ * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec array /\n+ * the dma map are going to be reused, and so no extra allocation is required.\n  */\n static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs)\n {\n-\tif (iov_iter_is_bvec(iter))\n+\tif (iov_iter_is_bvec(iter) || iov_iter_is_dmabuf_map(iter))\n \t\treturn 0;\n \treturn iov_iter_npages(iter, max_segs);\n }\n@@ -518,7 +523,7 @@ int bdev_rw_virt(struct block_device *bdev, sector_t sector, void *data,\n int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,\n \t\tunsigned mem_align_mask, unsigned len_align_mask);\n \n-void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter);\n+bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter);\n void __bio_release_pages(struct bio *bio, bool mark_dirty);\n extern void bio_set_pages_dirty(struct bio *bio);\n extern void bio_check_pages_dirty(struct bio *bio);\ndiff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h\nindex af878597afb8c..7c7504c84e090 100644\n--- a/include/linux/blk-mq.h\n+++ b/include/linux/blk-mq.h\n@@ -1017,6 +1017,13 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq)\n \treturn rq + 1;\n }\n \n+static inline bool blk_mq_rq_is_dmabuf(struct request *rq)\n+{\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn false;\n+\treturn rq-\u003ebio \u0026\u0026 op_is_dmabuf(rq-\u003ebio-\u003ebi_opf);\n+}\n+\n static inline struct blk_mq_hw_ctx *queue_hctx(struct request_queue *q, int id)\n {\n \tstruct blk_mq_hw_ctx *hctx;\ndiff --git a/include/linux/blk_types.h b/include/linux/blk_types.h\nindex d49d97a050d0e..a305951f03124 100644\n--- a/include/linux/blk_types.h\n+++ b/include/linux/blk_types.h\n@@ -233,7 +233,12 @@ struct bio {\n \tatomic_t\t\t__bi_remaining;\n \n \t/* The actual vec list, preserved by bio_reset() */\n-\tstruct bio_vec\t\t*bi_io_vec;\n+\tunion {\n+\t\tstruct bio_vec\t\t*bi_io_vec;\n+\t\t/* Driver specific dma map, valid IFF REQ_DMABUF is set */\n+\t\tstruct dma_buf_io_map\t*bi_dmabuf_map;\n+\t};\n+\n \tstruct bvec_iter\tbi_iter;\n \n \tunion {\n@@ -402,6 +407,7 @@ enum req_flag_bits {\n \t__REQ_DRV,\t\t/* for driver use */\n \t__REQ_FS_PRIVATE,\t/* for file system (submitter) use */\n \t__REQ_ATOMIC,\t\t/* for atomic write operations */\n+\t__REQ_DMABUF,\t\t/* Using premmaped dma buffers */\n \t/*\n \t * Command specific flags, keep last:\n \t */\n@@ -434,6 +440,7 @@ enum req_flag_bits {\n #define REQ_DRV\t\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_DRV)\n #define REQ_FS_PRIVATE\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_FS_PRIVATE)\n #define REQ_ATOMIC\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_ATOMIC)\n+#define REQ_DMABUF\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_DMABUF)\n \n #define REQ_NOUNMAP\t(__force blk_opf_t)(1ULL \u003c\u003c __REQ_NOUNMAP)\n \n@@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op)\n \treturn (op \u0026 REQ_OP_MASK) == REQ_OP_DISCARD;\n }\n \n+static inline bool op_is_dmabuf(blk_opf_t op)\n+{\n+\treturn op \u0026 REQ_DMABUF;\n+}\n+\n /*\n  * Check if a bio or request operation is a zone management operation.\n  */\ndiff --git a/include/linux/blkdev.h b/include/linux/blkdev.h\nindex 9213a5716f95a..9f4c92e51dc95 100644\n--- a/include/linux/blkdev.h\n+++ b/include/linux/blkdev.h\n@@ -1682,6 +1682,8 @@ struct block_device_operations {\n \t/* returns the length of the identifier or a negative errno: */\n \tint (*get_unique_id)(struct gendisk *disk, u8 id[16],\n \t\t\tenum blk_unique_id id_type);\n+\tint (*init_dma_buf_io_ctx)(struct block_device *,\n+\t\t\t\t   struct dma_buf_io_ctx *);\n \tstruct module *owner;\n \tconst struct pr_ops *pr_ops;\n \ndiff --git a/include/linux/bvec.h b/include/linux/bvec.h\nindex 92837e2743f19..b63914ff56e33 100644\n--- a/include/linux/bvec.h\n+++ b/include/linux/bvec.h\n@@ -108,9 +108,10 @@ struct bvec_iter {\n \tunsigned int\t\tbi_idx;\n \n \t/*\n-\t * Current offset in the bvec entry pointed to by `bi_idx`.\n+\t * Current offset in the bvec entry pointed to by `bi_idx` or into\n+\t * a dma-buf map.\n \t */\n-\tunsigned int\t\tbi_bvec_done;\n+\tunsigned int\t\tbi_offset;\n } __packed __aligned(4);\n \n struct bvec_iter_all {\n@@ -135,14 +136,14 @@ mp_bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter)\n static __always_inline unsigned int\n mp_bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter)\n {\n-\treturn min(__bvec_iter_bvec(bvecs, iter)-\u003ebv_len - iter.bi_bvec_done,\n+\treturn min(__bvec_iter_bvec(bvecs, iter)-\u003ebv_len - iter.bi_offset,\n \t\t\titer.bi_size);\n }\n \n static __always_inline unsigned int\n mp_bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter)\n {\n-\treturn __bvec_iter_bvec(bvecs, iter)-\u003ebv_offset + iter.bi_bvec_done;\n+\treturn __bvec_iter_bvec(bvecs, iter)-\u003ebv_offset + iter.bi_offset;\n }\n \n static __always_inline unsigned int\n@@ -204,7 +205,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,\n \t}\n \n \titer-\u003ebi_size -= bytes;\n-\tbytes += iter-\u003ebi_bvec_done;\n+\tbytes += iter-\u003ebi_offset;\n \n \twhile (bytes \u0026\u0026 bytes \u003e= bv[idx].bv_len) {\n \t\tbytes -= bv[idx].bv_len;\n@@ -212,7 +213,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,\n \t}\n \n \titer-\u003ebi_idx = idx;\n-\titer-\u003ebi_bvec_done = bytes;\n+\titer-\u003ebi_offset = bytes;\n \treturn true;\n }\n \n@@ -223,13 +224,13 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,\n static inline void bvec_iter_advance_single(const struct bio_vec *bv,\n \t\t\t\tstruct bvec_iter *iter, unsigned int bytes)\n {\n-\tunsigned int done = iter-\u003ebi_bvec_done + bytes;\n+\tunsigned int done = iter-\u003ebi_offset + bytes;\n \n \tif (done == bv[iter-\u003ebi_idx].bv_len) {\n \t\tdone = 0;\n \t\titer-\u003ebi_idx++;\n \t}\n-\titer-\u003ebi_bvec_done = done;\n+\titer-\u003ebi_offset = done;\n \titer-\u003ebi_size -= bytes;\n }\n \ndiff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h\nnew file mode 100644\nindex 0000000000000..f31c7375ae911\n--- /dev/null\n+++ b/include/linux/dma-buf-io.h\n@@ -0,0 +1,92 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef __DMA_BUF_IO_H__\n+#define __DMA_BUF_IO_H__\n+\n+#include \u003clinux/dma-buf.h\u003e\n+\n+struct dma_buf_io_fence;\n+struct dma_buf_io_ctx;\n+struct dma_buf_io_map;\n+\n+struct dma_buf_io_ops {\n+\t/*\n+\t * Create a new map for the given ctx. Called with the reservation\n+\t * lock held.\n+\t */\n+\tstruct dma_buf_io_map *(*map)(struct dma_buf_io_ctx *ctx);\n+\n+\t/*\n+\t * Clean up device specific parts of the @map. Called with the\n+\t * reservation lock held.\n+\t */\n+\tvoid (*unmap)(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map);\n+\n+\t/*\n+\t * The user tries to destroy the ctx. Release all device specific\n+\t * parts of the token.\n+\t */\n+\tvoid (*release)(struct dma_buf_io_ctx *);\n+};\n+\n+struct dma_buf_io_map {\n+\t/*\n+\t * Counts attached requests and other users. Device specific unmapping\n+\t * is deferred until all refs are dropped.\n+\t */\n+\tstruct percpu_ref\t\trefs;\n+\tunsigned\t\t\tseg_shift;\n+\n+\tstruct work_struct\t\trelease_work;\n+\tstruct dma_buf_io_fence\t\t*fence;\n+\tstruct dma_buf_io_ctx\t\t*ctx;\n+};\n+\n+struct dma_buf_io_ctx {\n+\tstruct dma_buf_io_map __rcu\t\t*map;\n+\tstruct dma_buf\t\t\t\t*dmabuf;\n+\tenum dma_data_direction\t\t\tdir;\n+\n+\tatomic_t\t\t\t\tfence_seq;\n+\tu64\t\t\t\t\tfence_ctx;\n+\tstruct work_struct\t\t\trelease_work;\n+\trefcount_t\t\t\t\trefs;\n+\n+\tvoid\t\t\t\t\t*dev_priv;\n+\tconst struct dma_buf_io_ops\t\t*dev_ops;\n+};\n+\n+int dma_buf_io_ctx_create(struct file *file,\n+\t\t\t   struct dma_buf_io_ctx *ctx,\n+\t\t\t   struct dma_buf *dmabuf,\n+\t\t\t   enum dma_data_direction dir);\n+void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx);\n+\n+struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx);\n+\n+static inline struct dma_buf_io_map *\n+dma_buf_io_get_map(struct dma_buf_io_ctx *ctx)\n+{\n+\tstruct dma_buf_io_map *map;\n+\n+\tguard(rcu)();\n+\n+\tmap = rcu_dereference(ctx-\u003emap);\n+\tif (unlikely(!map || !percpu_ref_tryget_live_rcu(\u0026map-\u003erefs)))\n+\t\treturn NULL;\n+\n+\treturn map;\n+}\n+\n+static inline void dma_buf_io_map_drop(struct dma_buf_io_map *map)\n+{\n+\tpercpu_ref_put(\u0026map-\u003erefs);\n+}\n+\n+/*\n+ * Device API\n+ */\n+\n+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx);\n+int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map);\n+\n+#endif\ndiff --git a/include/linux/fs.h b/include/linux/fs.h\nindex 50ce731a2b78f..e62fa653e1484 100644\n--- a/include/linux/fs.h\n+++ b/include/linux/fs.h\n@@ -1914,6 +1914,7 @@ struct dir_context {\n #define COPY_FILE_SPLICE\t\t(1 \u003c\u003c 0)\n \n struct io_uring_cmd;\n+struct dma_buf_io_ctx;\n struct offset_ctx;\n \n typedef unsigned int __bitwise fop_flags_t;\n@@ -1962,6 +1963,7 @@ struct file_operations {\n \tint (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *,\n \t\t\t\tunsigned int poll_flags);\n \tint (*mmap_prepare)(struct vm_area_desc *);\n+\tint (*init_dma_buf_io_ctx)(struct file *, struct dma_buf_io_ctx *);\n } __randomize_layout;\n \n /* Supports async buffered reads */\ndiff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h\nindex a2c623a67a251..f90586d353c1c 100644\n--- a/include/linux/io_uring_types.h\n+++ b/include/linux/io_uring_types.h\n@@ -10,6 +10,7 @@\n \n struct iou_loop_params;\n struct io_uring_bpf_ops;\n+struct dma_buf_io_map;\n \n enum {\n \t/*\n@@ -594,6 +595,7 @@ enum {\n \tREQ_F_IMPORT_BUFFER_BIT,\n \tREQ_F_SQE_COPIED_BIT,\n \tREQ_F_IOPOLL_BIT,\n+\tREQ_F_DROP_DMABUF_BIT,\n \n \t/* not a real bit, just to check we're not overflowing the space */\n \t__REQ_F_LAST_BIT,\n@@ -689,6 +691,8 @@ enum {\n \tREQ_F_SQE_COPIED\t= IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT),\n \t/* request must be iopolled to completion (set in -\u003eissue()) */\n \tREQ_F_IOPOLL\t\t= IO_REQ_FLAG(REQ_F_IOPOLL_BIT),\n+\t/* there is a dma map attached to request that needs to be dropped */\n+\tREQ_F_DROP_DMABUF\t= IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT),\n };\n \n struct io_tw_req {\n@@ -811,6 +815,7 @@ struct io_kiocb {\n \t/* custom credentials, valid IFF REQ_F_CREDS is set */\n \tconst struct cred\t\t*creds;\n \tstruct io_wq_work\t\twork;\n+\tstruct dma_buf_io_map\t\t*dmabuf_map;\n \n \tstruct io_big_cqe {\n \t\tu64\t\t\textra1;\ndiff --git a/include/linux/uio.h b/include/linux/uio.h\nindex fe2e985d74d24..638c1116e912e 100644\n--- a/include/linux/uio.h\n+++ b/include/linux/uio.h\n@@ -12,6 +12,7 @@\n \n struct page;\n struct folio_queue;\n+struct dma_buf_io_map;\n \n typedef unsigned int __bitwise iov_iter_extraction_t;\n \n@@ -29,6 +30,7 @@ enum iter_type {\n \tITER_FOLIOQ,\n \tITER_XARRAY,\n \tITER_DISCARD,\n+\tITER_DMABUF_MAP,\n };\n \n #define ITER_SOURCE\t1\t// == WRITE\n@@ -71,6 +73,7 @@ struct iov_iter {\n \t\t\t\tconst struct folio_queue *folioq;\n \t\t\t\tstruct xarray *xarray;\n \t\t\t\tvoid __user *ubuf;\n+\t\t\t\tstruct dma_buf_io_map *dmabuf_map;\n \t\t\t};\n \t\t\tsize_t count;\n \t\t};\n@@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter *i)\n \treturn iov_iter_type(i) == ITER_XARRAY;\n }\n \n+static inline bool iov_iter_is_dmabuf_map(const struct iov_iter *i)\n+{\n+\treturn iov_iter_type(i) == ITER_DMABUF_MAP;\n+}\n+\n static inline unsigned char iov_iter_rw(const struct iov_iter *i)\n {\n \treturn i-\u003edata_source ? WRITE : READ;\n@@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,\n \t\t\t  unsigned int first_slot, unsigned int offset, size_t count);\n void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray,\n \t\t     loff_t start, size_t count);\n+void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction,\n+\t\t\tstruct dma_buf_io_map *map,\n+\t\t\tloff_t off, size_t count);\n ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,\n \t\t\tsize_t maxsize, unsigned maxpages, size_t *start);\n ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages,\ndiff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h\nindex 909fb7aea638e..c39297e8beb6f 100644\n--- a/include/uapi/linux/io_uring.h\n+++ b/include/uapi/linux/io_uring.h\n@@ -790,13 +790,42 @@ struct io_uring_rsrc_update {\n \n struct io_uring_rsrc_update2 {\n \t__u32 offset;\n-\t__u32 resv;\n+\t__u32 flags;\n \t__aligned_u64 data;\n \t__aligned_u64 tags;\n \t__u32 nr;\n \t__u32 resv2;\n };\n \n+/* struct io_uring_rsrc_update2::flags */\n+enum io_uring_rsrc_reg_flags {\n+\t/*\n+\t * Use the extended descriptor format for buffer updates,\n+\t * see struct io_uring_regbuf_desc\n+\t */\n+\tIORING_RSRC_UPDATE_EXTENDED\t\t= 1U \u003c\u003c 1,\n+};\n+\n+/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */\n+enum io_uring_regbuf_type {\n+\tIO_REGBUF_TYPE_EMPTY,\n+\tIO_REGBUF_TYPE_UADDR,\n+\tIO_REGBUF_TYPE_DMABUF,\n+\n+\t__IO_REGBUF_TYPE_MAX,\n+};\n+\n+struct io_uring_regbuf_desc {\n+\t__u32 type; /* enum io_uring_regbuf_type */\n+\t__u32 flags;\n+\t__u64 size;\n+\t__u64 uaddr;\n+\n+\t__s32 dmabuf_fd;\n+\t__s32 target_fd;\n+\t__u64 __resv[6];\n+};\n+\n /* Skip updating fd indexes set to this value in the fd table */\n #define IORING_REGISTER_FILES_SKIP\t(-2)\n \ndiff --git a/io_uring/io_uring.c b/io_uring/io_uring.c\nindex 4c83a94b4bdc3..d1d748b5c5dca 100644\n--- a/io_uring/io_uring.c\n+++ b/io_uring/io_uring.c\n@@ -109,7 +109,7 @@\n \n #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | IO_REQ_LINK_FLAGS | \\\n \t\t\t\t REQ_F_REISSUE | REQ_F_POLLED | \\\n-\t\t\t\t IO_REQ_CLEAN_FLAGS)\n+\t\t\t\t IO_REQ_CLEAN_FLAGS | REQ_F_DROP_DMABUF)\n \n #define IO_TCTX_REFS_CACHE_NR\t(1U \u003c\u003c 10)\n \n@@ -1114,6 +1114,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,\n \t\t\t\tio_queue_next(req);\n \t\t\tif (unlikely(req-\u003eflags \u0026 IO_REQ_CLEAN_FLAGS))\n \t\t\t\tio_clean_op(req);\n+\t\t\tio_req_drop_dmabuf(req);\n \t\t}\n \t\tio_put_file(req);\n \t\tio_req_put_rsrc_nodes(req);\ndiff --git a/io_uring/net.c b/io_uring/net.c\nindex a74d15f7b7d2b..439c99ad18841 100644\n--- a/io_uring/net.c\n+++ b/io_uring/net.c\n@@ -1471,7 +1471,7 @@ static int io_sg_from_iter(struct sk_buff *skb,\n \t\treturn zerocopy_fill_skb_from_iter(skb, from, length);\n \n \tbi.bi_size = min(from-\u003ecount, length);\n-\tbi.bi_bvec_done = from-\u003eiov_offset;\n+\tbi.bi_offset = from-\u003eiov_offset;\n \tbi.bi_idx = 0;\n \n \twhile (bi.bi_size \u0026\u0026 frag \u003c MAX_SKB_FRAGS) {\n@@ -1490,7 +1490,7 @@ static int io_sg_from_iter(struct sk_buff *skb,\n \tfrom-\u003ebvec += bi.bi_idx;\n \tfrom-\u003enr_segs -= bi.bi_idx;\n \tfrom-\u003ecount -= copied;\n-\tfrom-\u003eiov_offset = bi.bi_bvec_done;\n+\tfrom-\u003eiov_offset = bi.bi_offset;\n \n \tskb-\u003edata_len += copied;\n \tskb-\u003elen += copied;\ndiff --git a/io_uring/rsrc.c b/io_uring/rsrc.c\nindex 8d0f2ee24e0c2..5074b1e4b0505 100644\n--- a/io_uring/rsrc.c\n+++ b/io_uring/rsrc.c\n@@ -10,6 +10,7 @@\n #include \u003clinux/compat.h\u003e\n #include \u003clinux/io_uring.h\u003e\n #include \u003clinux/io_uring/cmd.h\u003e\n+#include \u003clinux/dma-buf-io.h\u003e\n \n #include \u003cuapi/linux/io_uring.h\u003e\n \n@@ -28,7 +29,7 @@ struct io_rsrc_update {\n };\n \n static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,\n-\t\t\t\t\t\t   struct iovec *iov);\n+\t\t\t\t\t\t   struct io_uring_regbuf_desc *desc);\n \n static int hpage_acct_ref(struct io_ring_ctx *ctx, struct page *hpage,\n \t\t\t  bool *acct_new)\n@@ -81,6 +82,18 @@ static bool hpage_acct_unref(struct io_ring_ctx *ctx, struct page *hpage)\n \n #define IO_CACHED_BVECS_SEGS\t32\n \n+static void io_iov_to_regbuf_desc(const struct iovec *iov,\n+\t\t\t\t  struct io_uring_regbuf_desc *desc)\n+{\n+\t*desc = (struct io_uring_regbuf_desc) {\n+\t\t.type = IO_REGBUF_TYPE_UADDR,\n+\t\t.uaddr = (u64)(uintptr_t)iov-\u003eiov_base,\n+\t\t.size = iov-\u003eiov_len,\n+\t};\n+\tif (!desc-\u003euaddr)\n+\t\tdesc-\u003etype = IO_REGBUF_TYPE_EMPTY;\n+}\n+\n int __io_account_mem(struct user_struct *user, unsigned long nr_pages)\n {\n \tunsigned long page_limit, cur_pages, new_pages;\n@@ -309,6 +322,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,\n \t\treturn -ENXIO;\n \tif (up-\u003eoffset + nr_args \u003e ctx-\u003efile_table.data.nr)\n \t\treturn -EINVAL;\n+\tif (up-\u003eflags)\n+\t\treturn -EINVAL;\n \n \tfor (done = 0; done \u003c nr_args; done++) {\n \t\tu64 tag = 0;\n@@ -368,9 +383,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,\n \t\t\t\t   struct io_uring_rsrc_update2 *up,\n \t\t\t\t   unsigned int nr_args)\n {\n+\tbool extended = up-\u003eflags \u0026 IORING_RSRC_UPDATE_EXTENDED;\n \tu64 __user *tags = u64_to_user_ptr(up-\u003etags);\n-\tstruct iovec fast_iov, *iov;\n-\tstruct iovec __user *uvec;\n \tu64 user_data = up-\u003edata;\n \t__u32 done;\n \tint i, err;\n@@ -379,26 +393,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,\n \t\treturn -ENXIO;\n \tif (up-\u003eoffset + nr_args \u003e ctx-\u003ebuf_table.nr)\n \t\treturn -EINVAL;\n+\tif (up-\u003eflags \u0026 ~IORING_RSRC_UPDATE_EXTENDED)\n+\t\treturn -EINVAL;\n \n \tfor (done = 0; done \u003c nr_args; done++) {\n+\t\tstruct io_uring_regbuf_desc desc;\n \t\tstruct io_rsrc_node *node;\n \t\tu64 tag = 0;\n \n-\t\tuvec = u64_to_user_ptr(user_data);\n-\t\tiov = iovec_from_user(uvec, 1, 1, \u0026fast_iov, io_is_compat(ctx));\n-\t\tif (IS_ERR(iov)) {\n-\t\t\terr = PTR_ERR(iov);\n-\t\t\tbreak;\n-\t\t}\n \t\tif (tags \u0026\u0026 copy_from_user(\u0026tag, \u0026tags[done], sizeof(tag))) {\n \t\t\terr = -EFAULT;\n \t\t\tbreak;\n \t\t}\n-\t\tnode = io_sqe_buffer_register(ctx, iov);\n+\n+\t\tif (extended) {\n+\t\t\tif (copy_from_user(\u0026desc, u64_to_user_ptr(user_data),\n+\t\t\t\t\t   sizeof(desc))) {\n+\t\t\t\terr = -EFAULT;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tuser_data += sizeof(desc);\n+\t\t} else {\n+\t\t\tstruct iovec __user *uvec = u64_to_user_ptr(user_data);\n+\t\t\tstruct iovec fast_iov, *iov;\n+\n+\t\t\tif (io_is_compat(ctx))\n+\t\t\t\tuser_data += sizeof(struct compat_iovec);\n+\t\t\telse\n+\t\t\t\tuser_data += sizeof(struct iovec);\n+\n+\t\t\tiov = iovec_from_user(uvec, 1, 1, \u0026fast_iov, io_is_compat(ctx));\n+\t\t\tif (IS_ERR(iov)) {\n+\t\t\t\terr = PTR_ERR(iov);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tio_iov_to_regbuf_desc(iov, \u0026desc);\n+\t\t}\n+\n+\t\tnode = io_sqe_buffer_register(ctx, \u0026desc);\n \t\tif (IS_ERR(node)) {\n \t\t\terr = PTR_ERR(node);\n \t\t\tbreak;\n \t\t}\n+\n \t\tif (tag) {\n \t\t\tif (!node) {\n \t\t\t\terr = -EINVAL;\n@@ -409,10 +446,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,\n \t\ti = array_index_nospec(up-\u003eoffset + done, ctx-\u003ebuf_table.nr);\n \t\tio_reset_rsrc_node(ctx, \u0026ctx-\u003ebuf_table, i);\n \t\tctx-\u003ebuf_table.nodes[i] = node;\n-\t\tif (io_is_compat(ctx))\n-\t\t\tuser_data += sizeof(struct compat_iovec);\n-\t\telse\n-\t\t\tuser_data += sizeof(struct iovec);\n \t}\n \treturn done ? done : err;\n }\n@@ -447,7 +480,7 @@ int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg,\n \tmemset(\u0026up, 0, sizeof(up));\n \tif (copy_from_user(\u0026up, arg, sizeof(struct io_uring_rsrc_update)))\n \t\treturn -EFAULT;\n-\tif (up.resv || up.resv2)\n+\tif (up.resv2)\n \t\treturn -EINVAL;\n \treturn __io_register_rsrc_update(ctx, IORING_RSRC_FILE, \u0026up, nr_args);\n }\n@@ -461,7 +494,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,\n \t\treturn -EINVAL;\n \tif (copy_from_user(\u0026up, arg, sizeof(up)))\n \t\treturn -EFAULT;\n-\tif (!up.nr || up.resv || up.resv2)\n+\tif (!up.nr || up.resv2)\n \t\treturn -EINVAL;\n \treturn __io_register_rsrc_update(ctx, type, \u0026up, up.nr);\n }\n@@ -561,12 +594,9 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)\n \tstruct io_uring_rsrc_update2 up2;\n \tint ret;\n \n+\tmemset(\u0026up2, 0, sizeof(up2));\n \tup2.offset = up-\u003eoffset;\n \tup2.data = up-\u003earg;\n-\tup2.nr = 0;\n-\tup2.tags = 0;\n-\tup2.resv = 0;\n-\tup2.resv2 = 0;\n \n \tif (up-\u003eoffset == IORING_FILE_INDEX_ALLOC) {\n \t\tret = io_files_update_with_index_alloc(req, issue_flags);\n@@ -852,27 +882,126 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,\n \treturn true;\n }\n \n+struct io_regbuf_dma {\n+\tstruct dma_buf_io_ctx\t\tctx;\n+\tstruct file\t\t\t*target_file;\n+};\n+\n+static void io_release_reg_dmabuf(void *priv)\n+{\n+\tstruct io_regbuf_dma *db = priv;\n+\n+\tfput(db-\u003etarget_file);\n+\tdma_buf_io_ctx_release(\u0026db-\u003ectx);\n+}\n+\n+static struct io_rsrc_node *io_register_dmabuf(struct io_ring_ctx *ctx,\n+\t\t\t\t\t\tstruct io_uring_regbuf_desc *desc)\n+{\n+\tstruct io_rsrc_node *node = NULL;\n+\tstruct io_mapped_ubuf *imu = NULL;\n+\tstruct io_regbuf_dma *regbuf = NULL;\n+\tstruct file *target_file = NULL;\n+\tstruct dma_buf *dmabuf = NULL;\n+\tint ret;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn ERR_PTR(-EOPNOTSUPP);\n+\tif (ctx-\u003eflags \u0026 IORING_SETUP_IOPOLL)\n+\t\treturn ERR_PTR(-EOPNOTSUPP);\n+\tif (desc-\u003euaddr || desc-\u003esize)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tret = -ENOMEM;\n+\tnode = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);\n+\tif (!node)\n+\t\treturn ERR_PTR(-ENOMEM);\n+\timu = io_alloc_imu(ctx, 0);\n+\tif (!imu)\n+\t\tgoto err;\n+\tregbuf = kzalloc(sizeof(*regbuf), GFP_KERNEL);\n+\tif (!regbuf)\n+\t\tgoto err;\n+\n+\tret = -EBADF;\n+\ttarget_file = fget(desc-\u003etarget_fd);\n+\tif (!target_file)\n+\t\tgoto err;\n+\n+\tdmabuf = dma_buf_get(desc-\u003edmabuf_fd);\n+\tif (IS_ERR(dmabuf)) {\n+\t\tret = PTR_ERR(dmabuf);\n+\t\tdmabuf = NULL;\n+\t\tgoto err;\n+\t}\n+\tif (dmabuf-\u003esize \u003e SZ_1G) {\n+\t\tret = -EINVAL;\n+\t\tgoto err;\n+\t}\n+\n+\tret = dma_buf_io_ctx_create(target_file, \u0026regbuf-\u003ectx, dmabuf,\n+\t\t\t\t    DMA_BIDIRECTIONAL);\n+\tif (ret)\n+\t\tgoto err;\n+\n+\tregbuf-\u003etarget_file = target_file;\n+\timu-\u003enr_bvecs = 0;\n+\timu-\u003eubuf = 0;\n+\timu-\u003elen = dmabuf-\u003esize;\n+\timu-\u003efolio_shift = 0;\n+\timu-\u003erelease = io_release_reg_dmabuf;\n+\timu-\u003epriv = regbuf;\n+\timu-\u003eflags = IO_REGBUF_F_DMABUF;\n+\timu-\u003edir = IO_IMU_DEST | IO_IMU_SOURCE;\n+\trefcount_set(\u0026imu-\u003erefs, 1);\n+\tnode-\u003ebuf = imu;\n+\tdma_buf_put(dmabuf);\n+\treturn node;\n+err:\n+\tkfree(regbuf);\n+\tif (imu)\n+\t\tio_free_imu(ctx, imu);\n+\tif (node)\n+\t\tio_cache_free(\u0026ctx-\u003enode_cache, node);\n+\tif (target_file)\n+\t\tfput(target_file);\n+\tif (dmabuf)\n+\t\tdma_buf_put(dmabuf);\n+\treturn ERR_PTR(ret);\n+}\n+\n static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,\n-\t\t\t\t\t\t   struct iovec *iov)\n+\t\t\t\t\t\t   struct io_uring_regbuf_desc *desc)\n {\n+\tunsigned long uaddr = (unsigned long)desc-\u003euaddr;\n+\tsize_t size = desc-\u003esize;\n \tstruct io_mapped_ubuf *imu = NULL;\n \tstruct page **pages = NULL;\n \tstruct io_rsrc_node *node;\n \tunsigned long off;\n-\tsize_t size;\n \tint ret, nr_pages, i;\n \tstruct io_imu_folio_data data;\n \tbool coalesced = false;\n \n-\tif (!iov-\u003eiov_base) {\n-\t\tif (iov-\u003eiov_len)\n+\tif (desc-\u003etype \u003e= __IO_REGBUF_TYPE_MAX)\n+\t\treturn ERR_PTR(-EINVAL);\n+\tif (!mem_is_zero(\u0026desc-\u003e__resv, sizeof(desc-\u003e__resv)) || desc-\u003eflags)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tif (desc-\u003etype == IO_REGBUF_TYPE_DMABUF)\n+\t\treturn io_register_dmabuf(ctx, desc);\n+\n+\tif (desc-\u003edmabuf_fd || desc-\u003etarget_fd)\n+\t\treturn ERR_PTR(-EINVAL);\n+\n+\tif (desc-\u003etype == IO_REGBUF_TYPE_EMPTY) {\n+\t\tif (uaddr || size)\n \t\t\treturn ERR_PTR(-EFAULT);\n \t\t/* remove the buffer without installing a new one */\n \t\treturn NULL;\n \t}\n \n-\tret = io_validate_user_buf_range((unsigned long)iov-\u003eiov_base,\n-\t\t\t\t\t iov-\u003eiov_len);\n+\tret = io_validate_user_buf_range(uaddr, size);\n \tif (ret)\n \t\treturn ERR_PTR(ret);\n \n@@ -881,8 +1010,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,\n \t\treturn ERR_PTR(-ENOMEM);\n \n \tret = -ENOMEM;\n-\tpages = io_pin_pages((unsigned long) iov-\u003eiov_base, iov-\u003eiov_len,\n-\t\t\t\t\u0026nr_pages);\n+\tpages = io_pin_pages(uaddr, size, \u0026nr_pages);\n \tif (IS_ERR(pages)) {\n \t\tret = PTR_ERR(pages);\n \t\tpages = NULL;\n@@ -904,10 +1032,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,\n \tif (ret)\n \t\tgoto done;\n \n-\tsize = iov-\u003eiov_len;\n \t/* store original address for later verification */\n-\timu-\u003eubuf = (unsigned long) iov-\u003eiov_base;\n-\timu-\u003elen = iov-\u003eiov_len;\n+\timu-\u003eubuf = uaddr;\n+\timu-\u003elen = size;\n \timu-\u003efolio_shift = PAGE_SHIFT;\n \timu-\u003erelease = io_release_ubuf;\n \timu-\u003epriv = imu;\n@@ -917,7 +1044,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,\n \t\timu-\u003efolio_shift = data.folio_shift;\n \trefcount_set(\u0026imu-\u003erefs, 1);\n \n-\toff = (unsigned long)iov-\u003eiov_base \u0026 ~PAGE_MASK;\n+\toff = uaddr \u0026 ~PAGE_MASK;\n \tif (coalesced)\n \t\toff += data.first_folio_page_idx \u003c\u003c PAGE_SHIFT;\n \n@@ -969,6 +1096,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,\n \t\tmemset(iov, 0, sizeof(*iov));\n \n \tfor (i = 0; i \u003c nr_args; i++) {\n+\t\tstruct io_uring_regbuf_desc desc;\n \t\tstruct io_rsrc_node *node;\n \t\tu64 tag = 0;\n \n@@ -992,7 +1120,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,\n \t\t\t}\n \t\t}\n \n-\t\tnode = io_sqe_buffer_register(ctx, iov);\n+\t\tio_iov_to_regbuf_desc(iov, \u0026desc);\n+\t\tnode = io_sqe_buffer_register(ctx, \u0026desc);\n \t\tif (IS_ERR(node)) {\n \t\t\tret = PTR_ERR(node);\n \t\t\tbreak;\n@@ -1136,9 +1265,64 @@ static int io_import_kbuf(int ddir, struct iov_iter *iter,\n \treturn 0;\n }\n \n-static int io_import_fixed(int ddir, struct iov_iter *iter,\n+void io_drop_dmabuf_node(struct io_kiocb *req)\n+{\n+\tstruct io_mapped_ubuf *imu;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(req-\u003ebuf_node-\u003etype != IORING_RSRC_BUFFER))\n+\t\treturn;\n+\timu = req-\u003ebuf_node-\u003ebuf;\n+\tif (WARN_ON_ONCE(!(imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF)))\n+\t\treturn;\n+\tdma_buf_io_map_drop(req-\u003edmabuf_map);\n+\treq-\u003eflags \u0026= ~REQ_F_DROP_DMABUF;\n+}\n+\n+static int io_import_dmabuf(struct io_kiocb *req,\n+\t\t\t   int ddir, struct iov_iter *iter,\n+\t\t\t   struct io_mapped_ubuf *imu,\n+\t\t\t   size_t len, size_t offset,\n+\t\t\t   unsigned issue_flags)\n+{\n+\tstruct io_regbuf_dma *db = imu-\u003epriv;\n+\tstruct dma_buf_io_map *map;\n+\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn -EOPNOTSUPP;\n+\tif (!len)\n+\t\treturn -EFAULT;\n+\tif (req-\u003efile != db-\u003etarget_file)\n+\t\treturn -EBADF;\n+\n+\tif (req-\u003eflags \u0026 REQ_F_DROP_DMABUF) {\n+\t\tmap = req-\u003edmabuf_map;\n+\t\tgoto init_iter;\n+\t}\n+\n+\tmap = dma_buf_io_get_map(\u0026db-\u003ectx);\n+\tif (unlikely(!map)) {\n+\t\tif (!(issue_flags \u0026 IO_URING_F_IOWQ))\n+\t\t\treturn -EAGAIN;\n+\t\tmap = dma_buf_io_create_map(\u0026db-\u003ectx);\n+\t\tif (IS_ERR(map))\n+\t\t\treturn PTR_ERR(map);\n+\t}\n+\n+\treq-\u003edmabuf_map = map;\n+\treq-\u003eflags |= REQ_F_DROP_DMABUF;\n+init_iter:\n+\tiov_iter_dmabuf_map(iter, ddir, map, offset, len);\n+\treturn 0;\n+}\n+\n+static int io_import_fixed(struct io_kiocb *req,\n+\t\t\t   int ddir, struct iov_iter *iter,\n \t\t\t   struct io_mapped_ubuf *imu,\n-\t\t\t   u64 buf_addr, size_t len)\n+\t\t\t   u64 buf_addr, size_t len,\n+\t\t\t   unsigned issue_flags,\n+\t\t\t   unsigned import_flags)\n {\n \tconst struct bio_vec *bvec;\n \tsize_t folio_mask;\n@@ -1158,6 +1342,12 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,\n \n \toffset = buf_addr - imu-\u003eubuf;\n \n+\tif (imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF) {\n+\t\tif (!(import_flags \u0026 IO_REGBUF_IMPORT_ALLOW_DMABUF))\n+\t\t\treturn -EFAULT;\n+\t\treturn io_import_dmabuf(req, ddir, iter, imu, len, offset,\n+\t\t\t\t\tissue_flags);\n+\t}\n \tif (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF)\n \t\treturn io_import_kbuf(ddir, iter, imu, len, offset);\n \n@@ -1211,16 +1401,17 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,\n \treturn NULL;\n }\n \n-int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,\n+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,\n \t\t\tu64 buf_addr, size_t len, int ddir,\n-\t\t\tunsigned issue_flags)\n+\t\t\tunsigned issue_flags, unsigned import_flags)\n {\n \tstruct io_rsrc_node *node;\n \n \tnode = io_find_buf_node(req, issue_flags);\n \tif (!node)\n \t\treturn -EFAULT;\n-\treturn io_import_fixed(ddir, iter, node-\u003ebuf, buf_addr, len);\n+\treturn io_import_fixed(req, ddir, iter, node-\u003ebuf, buf_addr, len,\n+\t\t\t\tissue_flags, import_flags);\n }\n \n static int io_buffer_acct_cloned_hpages(struct io_ring_ctx *ctx,\n@@ -1347,6 +1538,11 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx\n \t\tif (!src_node) {\n \t\t\tdst_node = NULL;\n \t\t} else {\n+\t\t\tif (src_node-\u003ebuf-\u003eflags \u0026 IO_REGBUF_F_UNCLONEABLE) {\n+\t\t\t\tio_rsrc_data_free(ctx, \u0026data);\n+\t\t\t\treturn -ENOMEM;\n+\t\t\t}\n+\n \t\t\tdst_node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);\n \t\t\tif (!dst_node) {\n \t\t\t\tio_rsrc_data_free(ctx, \u0026data);\n@@ -1617,9 +1813,9 @@ static int io_kern_bvec_size(struct iovec *iov, unsigned nr_iovs,\n \treturn 0;\n }\n \n-int io_import_reg_vec(int ddir, struct iov_iter *iter,\n+int __io_import_reg_vec(int ddir, struct iov_iter *iter,\n \t\t\tstruct io_kiocb *req, struct iou_vec *vec,\n-\t\t\tunsigned nr_iovs, unsigned issue_flags)\n+\t\t\tunsigned nr_iovs, unsigned issue_flags, unsigned import_flags)\n {\n \tstruct io_rsrc_node *node;\n \tstruct io_mapped_ubuf *imu;\n@@ -1637,7 +1833,9 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter,\n \tiovec_off = vec-\u003enr - nr_iovs;\n \tiov = vec-\u003eiovec + iovec_off;\n \n-\tif (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF) {\n+\tif (imu-\u003eflags \u0026 IO_REGBUF_F_DMABUF) {\n+\t\treturn -EOPNOTSUPP;\n+\t} else if (imu-\u003eflags \u0026 IO_REGBUF_F_KBUF) {\n \t\tint ret = io_kern_bvec_size(iov, nr_iovs, imu, \u0026nr_segs);\n \n \t\tif (unlikely(ret))\ndiff --git a/io_uring/rsrc.h b/io_uring/rsrc.h\nindex 98ae8ef51009d..3cb9b57f159b6 100644\n--- a/io_uring/rsrc.h\n+++ b/io_uring/rsrc.h\n@@ -29,7 +29,13 @@ enum {\n };\n \n enum {\n-\tIO_REGBUF_F_KBUF\t\t= 1,\n+\tIO_REGBUF_F_KBUF\t\t= 1 \u003c\u003c 0,\n+\tIO_REGBUF_F_UNCLONEABLE\t\t= 1 \u003c\u003c 1,\n+\tIO_REGBUF_F_DMABUF\t\t= 1 \u003c\u003c 3,\n+};\n+\n+enum {\n+\tIO_REGBUF_IMPORT_ALLOW_DMABUF\t\t= 1 \u003c\u003c 1,\n };\n \n struct io_mapped_ubuf {\n@@ -64,12 +70,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr);\n \n struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,\n \t\t\t\t      unsigned issue_flags);\n-int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,\n+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,\n \t\t\tu64 buf_addr, size_t len, int ddir,\n-\t\t\tunsigned issue_flags);\n-int io_import_reg_vec(int ddir, struct iov_iter *iter,\n+\t\t\tunsigned issue_flags, unsigned import_flags);\n+int __io_import_reg_vec(int ddir, struct iov_iter *iter,\n \t\t\tstruct io_kiocb *req, struct iou_vec *vec,\n-\t\t\tunsigned nr_iovs, unsigned issue_flags);\n+\t\t\tunsigned nr_iovs, unsigned issue_flags,\n+\t\t\tunsigned import_flags);\n+\n+static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,\n+\t\t\t\t    u64 buf_addr, size_t len, int ddir,\n+\t\t\t\t    unsigned issue_flags)\n+{\n+\treturn __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0);\n+}\n+\n+static inline int io_import_reg_vec(int ddir, struct iov_iter *iter,\n+\t\t\t\t    struct io_kiocb *req, struct iou_vec *vec,\n+\t\t\t\t    unsigned nr_iovs, unsigned issue_flags)\n+{\n+\treturn __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0);\n+}\n+\n int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv,\n \t\t\tconst struct iovec __user *uvec, size_t uvec_segs);\n \n@@ -156,4 +178,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv)\n \t\tio_vec_free(iv);\n }\n \n+void io_drop_dmabuf_node(struct io_kiocb *req);\n+\n+static inline void io_req_drop_dmabuf(struct io_kiocb *req)\n+{\n+\tif (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))\n+\t\treturn;\n+\tif (!(req-\u003eflags \u0026 REQ_F_DROP_DMABUF))\n+\t\treturn;\n+\tif (WARN_ON_ONCE(!(req-\u003eflags \u0026 REQ_F_BUF_NODE)))\n+\t\treturn;\n+\tio_drop_dmabuf_node(req);\n+}\n+\n #endif\ndiff --git a/io_uring/rw.c b/io_uring/rw.c\nindex 95038cfda6153..11ff7b5a11647 100644\n--- a/io_uring/rw.c\n+++ b/io_uring/rw.c\n@@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb *req, unsigned int issue_flags,\n \tif (io-\u003ebytes_done)\n \t\treturn 0;\n \n-\tret = io_import_reg_buf(req, \u0026io-\u003eiter, rw-\u003eaddr, rw-\u003elen, ddir,\n-\t\t\t\tissue_flags);\n+\tret = __io_import_reg_buf(req, \u0026io-\u003eiter, rw-\u003eaddr, rw-\u003elen, ddir,\n+\t\t\t\t  issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF);\n \tiov_iter_save_state(\u0026io-\u003eiter, \u0026io-\u003eiter_state);\n \treturn ret;\n }\n@@ -591,6 +591,8 @@ static void io_complete_rw(struct kiocb *kiocb, long res)\n \tstruct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);\n \tstruct io_kiocb *req = cmd_to_io_kiocb(rw);\n \n+\tio_req_drop_dmabuf(req);\n+\n \t__io_complete_rw_common(req, res);\n \tio_req_set_res(req, io_fixup_rw_res(req, res), 0);\n \treq-\u003eio_task_work.func = io_req_rw_complete;\ndiff --git a/lib/iov_iter.c b/lib/iov_iter.c\nindex 34a52e9ba9e1b..d6f5d7e180b54 100644\n--- a/lib/iov_iter.c\n+++ b/lib/iov_iter.c\n@@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size)\n {\n \tif (unlikely(i-\u003ecount \u003c size))\n \t\tsize = i-\u003ecount;\n-\tif (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i))) {\n+\tif (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i)) ||\n+\t    unlikely(iov_iter_is_dmabuf_map(i))) {\n \t\ti-\u003eiov_offset += size;\n \t\ti-\u003ecount -= size;\n \t} else if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) {\n@@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)\n \t\treturn;\n \t}\n \tunroll -= i-\u003eiov_offset;\n-\tif (iov_iter_is_xarray(i) || iter_is_ubuf(i)) {\n+\tif (iov_iter_is_xarray(i) || iter_is_ubuf(i) ||\n+\t    iov_iter_is_dmabuf_map(i)) {\n \t\tBUG(); /* We should never go beyond the start of the specified\n \t\t\t* range since we might then be straying into pages that\n \t\t\t* aren't pinned.\n@@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction,\n }\n EXPORT_SYMBOL(iov_iter_xarray);\n \n+void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction,\n+\t\t\t struct dma_buf_io_map *map,\n+\t\t\t loff_t off, size_t count)\n+{\n+\tWARN_ON(direction \u0026 ~(READ | WRITE));\n+\t*i = (struct iov_iter){\n+\t\t.iter_type = ITER_DMABUF_MAP,\n+\t\t.data_source = direction,\n+\t\t.dmabuf_map = map,\n+\t\t.count = count,\n+\t\t.iov_offset = off,\n+\t};\n+}\n+\n /**\n  * iov_iter_discard - Initialise an I/O iterator that discards data\n  * @i: The iterator to initialise.\n@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)\n \n unsigned long iov_iter_alignment(const struct iov_iter *i)\n {\n-\tif (likely(iter_is_ubuf(i))) {\n+\tif (likely(iter_is_ubuf(i)) || iov_iter_is_dmabuf_map(i)) {\n \t\tsize_t size = i-\u003ecount;\n \t\tif (size)\n \t\t\treturn ((unsigned long)i-\u003eubuf + i-\u003eiov_offset) | size;\n@@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)\n \tsize_t size = i-\u003ecount;\n \tunsigned k;\n \n-\tif (iter_is_ubuf(i))\n+\tif (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i))\n \t\treturn 0;\n \n \tif (WARN_ON(!iter_is_iovec(i)))\n@@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf);\n void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)\n {\n \tif (WARN_ON_ONCE(!iov_iter_is_bvec(i) \u0026\u0026 !iter_is_iovec(i) \u0026\u0026\n-\t\t\t !iter_is_ubuf(i)) \u0026\u0026 !iov_iter_is_kvec(i))\n+\t\t\t !iter_is_ubuf(i) \u0026\u0026 !iov_iter_is_kvec(i) \u0026\u0026\n+\t\t\t !iov_iter_is_dmabuf_map(i)))\n \t\treturn;\n \ti-\u003eiov_offset = state-\u003eiov_offset;\n \ti-\u003ecount = state-\u003ecount;\n-\tif (iter_is_ubuf(i))\n+\tif (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i))\n \t\treturn;\n \t/*\n \t * For the *vec iters, nr_segs + iov is constant - if we increment\n@@ -1634,7 +1651,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i,\n \t}\n \tbi.bi_idx = 0;\n \tbi.bi_size = maxsize;\n-\tbi.bi_bvec_done = skip;\n+\tbi.bi_offset = skip;\n \n \tmaxpages = want_pages_array(pages, maxsize, skip, maxpages);\n \tif (!maxpages)\ndiff --git a/net/ceph/messenger.c b/net/ceph/messenger.c\nindex 34b3097b4c7b9..9c1b6cf8c36fe 100644\n--- a/net/ceph/messenger.c\n+++ b/net/ceph/messenger.c\n@@ -762,7 +762,7 @@ static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor *cursor,\n \tif (!cursor-\u003eresid)\n \t\treturn false;   /* no more data */\n \n-\tif (!bytes || (it-\u003eiter.bi_size \u0026\u0026 it-\u003eiter.bi_bvec_done \u0026\u0026\n+\tif (!bytes || (it-\u003eiter.bi_size \u0026\u0026 it-\u003eiter.bi_offset \u0026\u0026\n \t\t       page == bio_iter_page(it-\u003ebio, it-\u003eiter)))\n \t\treturn false;\t/* more bytes to process in this segment */\n \n@@ -817,7 +817,7 @@ static bool ceph_msg_data_bvecs_advance(struct ceph_msg_data_cursor *cursor,\n \tif (!cursor-\u003eresid)\n \t\treturn false;   /* no more data */\n \n-\tif (!bytes || (cursor-\u003ebvec_iter.bi_bvec_done \u0026\u0026\n+\tif (!bytes || (cursor-\u003ebvec_iter.bi_offset \u0026\u0026\n \t\t       page == bvec_iter_page(bvecs, cursor-\u003ebvec_iter)))\n \t\treturn false;\t/* more bytes to process in this segment */\n \n"
}

2/1

2026/08/02 14:37

agent

patch-evaluator

Model:

best-expensive

Args:

null

Results:

{
  "EnableConfigs": [
    "DMA_SHARED_BUFFER",
    "IO_URING",
    "BLK_DEV_NVME"
  ],
  "FocusSymbols": [
    "dma_buf_io_ctx_create",
    "dma_buf_io_ctx_release",
    "dma_buf_io_create_map",
    "dma_buf_io_invalidate_mappings",
    "dma_buf_io_init_map",
    "bio_iov_iter_set",
    "__io_import_reg_buf",
    "io_drop_dmabuf_node",
    "iov_iter_dmabuf_map"
  ],
  "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.",
  "WorthFuzzing": true
}

Instruction:

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

Prompt:

For your convenience, here is the diff of the changes:
commit 2cea66d19f5e61596d4d0326aee5752566f75bc7
Author: syz-cluster <triage@syzkaller.com>
Date:   Sun Aug 2 14:37:35 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/block/biovecs.rst b/Documentation/block/biovecs.rst
index 11126ed6f40f7..49da2147b7d86 100644
--- a/Documentation/block/biovecs.rst
+++ b/Documentation/block/biovecs.rst
@@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec.
 In the new scheme of things, everything that must be mutated in order to
 partially complete a bio is segregated into struct bvec_iter: bi_sector,
 bi_size and bi_idx have been moved there; and instead of modifying bv_offset
-and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of
+and bv_len, struct bvec_iter has bi_offset, which represents the number of
 bytes completed in the current bvec.
 
 There are a bunch of new helper macros for hiding the gory details - in
 particular, presenting the illusion of partially completed biovecs so that
-normal code doesn't have to deal with bi_bvec_done.
+normal code doesn't have to deal with bi_offset.
 
  * Driver code should no longer refer to biovecs directly; we now have
    bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs,
-   constructed from the raw biovecs but taking into account bi_bvec_done and
+   constructed from the raw biovecs but taking into account bi_offset and
    bi_size.
 
    bio_for_each_segment() has been updated to take a bvec_iter argument
@@ -101,7 +101,7 @@ Other implications:
    I.e. instead of using bio_iovec_idx() (or bio->bi_iovec[bio->bi_idx]), you
    now use bio_iter_iovec(), which takes a bvec_iter and returns a
    literal struct bio_vec - constructed on the fly from the raw biovec but
-   taking into account bi_bvec_done (and bi_size).
+   taking into account bi_offset (and bi_size).
 
  * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that
    doesn't actually own the bio. The reason is twofold: firstly, it's not
diff --git a/block/bio.c b/block/bio.c
index 500389f332d99..1602eab05761f 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -229,7 +229,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
 	bio->bi_iter.bi_sector = 0;
 	bio->bi_iter.bi_size = 0;
 	bio->bi_iter.bi_idx = 0;
-	bio->bi_iter.bi_bvec_done = 0;
+	bio->bi_iter.bi_offset = 0;
 	bio->bi_end_io = NULL;
 	bio->bi_private = NULL;
 #ifdef CONFIG_BLK_CGROUP
@@ -861,6 +861,11 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
 	bio->bi_write_stream = bio_src->bi_write_stream;
 	bio->bi_iter = bio_src->bi_iter;
 
+	if (op_is_dmabuf(bio->bi_opf))
+		bio->bi_dmabuf_map = bio_src->bi_dmabuf_map;
+	else
+		bio->bi_io_vec = bio_src->bi_io_vec;
+
 	if (bio->bi_bdev) {
 		if (bio->bi_bdev == bio_src->bi_bdev &&
 		    bio_flagged(bio_src, BIO_REMAPPED))
@@ -902,8 +907,6 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,
 		bio_put(bio);
 		return NULL;
 	}
-	bio->bi_io_vec = bio_src->bi_io_vec;
-
 	return bio;
 }
 EXPORT_SYMBOL(bio_alloc_clone);
@@ -923,7 +926,7 @@ int bio_init_clone(struct block_device *bdev, struct bio *bio,
 {
 	int ret;
 
-	bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf);
+	bio_init(bio, bdev, NULL, 0, bio_src->bi_opf);
 	ret = __bio_clone(bio, bio_src, gfp);
 	if (ret)
 		bio_uninit(bio);
@@ -1182,15 +1185,26 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)
 }
 EXPORT_SYMBOL_GPL(__bio_release_pages);
 
-void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter)
+bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter)
 {
+	if (!iov_iter_is_bvec(iter) && !iov_iter_is_dmabuf_map(iter))
+		return false;
+
 	WARN_ON_ONCE(bio->bi_max_vecs);
 
+	static_assert(offsetof(struct bio, bi_io_vec) ==
+		      offsetof(struct bio, bi_dmabuf_map));
+	static_assert(offsetof(struct iov_iter, bvec) ==
+		      offsetof(struct iov_iter, dmabuf_map));
+
 	bio->bi_io_vec = (struct bio_vec *)iter->bvec;
 	bio->bi_iter.bi_idx = 0;
-	bio->bi_iter.bi_bvec_done = iter->iov_offset;
+	bio->bi_iter.bi_offset = iter->iov_offset;
 	bio->bi_iter.bi_size = iov_iter_count(iter);
 	bio_set_flag(bio, BIO_CLONED);
+	if (iov_iter_is_dmabuf_map(iter))
+		bio->bi_opf |= REQ_NOMERGE | REQ_DMABUF;
+	return true;
 }
 
 /*
@@ -1285,10 +1299,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 	if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
 		return -EIO;
 
-	if (iov_iter_is_bvec(iter)) {
-		bio_iov_bvec_set(bio, iter);
-
-		if (!bio_iov_bvec_aligned(bio, mem_align_mask))
+	if (bio_iov_iter_set(bio, iter)) {
+		if (iov_iter_is_bvec(iter) &&
+		    !bio_iov_bvec_aligned(bio, mem_align_mask))
 			return -EINVAL;
 
 		iov_iter_advance(iter, bio->bi_iter.bi_size);
diff --git a/block/blk-map.c b/block/blk-map.c
index 615d29bb840eb..9cb9605d1f62d 100644
--- a/block/blk-map.c
+++ b/block/blk-map.c
@@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request *rq, const struct iov_iter *iter)
 	bio = blk_rq_map_bio_alloc(rq, 0, GFP_KERNEL);
 	if (!bio)
 		return -ENOMEM;
-	bio_iov_bvec_set(bio, iter);
+	bio_iov_iter_set(bio, iter);
 
 	ret = blk_rq_append_bio(rq, bio);
 	if (ret)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index ab1161ca69f1e..1beedc42a85d5 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -9,6 +9,7 @@
 #include <linux/blk-integrity.h>
 #include <linux/part_stat.h>
 #include <linux/blk-cgroup.h>
+#include <linux/dma-buf-io.h>
 
 #include <trace/events/block.h>
 
@@ -33,7 +34,7 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)
 
 	bio_advance_iter(bio, &iter, iter.bi_size);
 
-	if (!iter.bi_bvec_done)
+	if (!iter.bi_offset)
 		idx = iter.bi_idx - 1;
 	else	/* in the middle of bvec */
 		idx = iter.bi_idx;
@@ -41,11 +42,11 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)
 	*bv = bio->bi_io_vec[idx];
 
 	/*
-	 * iter.bi_bvec_done records actual length of the last bvec
+	 * iter.bi_offset records actual length of the last bvec
 	 * if this bio ends in the middle of one io vector
 	 */
-	if (iter.bi_bvec_done)
-		bv->bv_len = iter.bi_bvec_done;
+	if (iter.bi_offset)
+		bv->bv_len = iter.bi_offset;
 }
 
 static inline bool bio_will_gap(struct request_queue *q,
@@ -319,6 +320,28 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
 	return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len);
 }
 
+static inline int bio_split_io_at_dmabuf(struct bio *bio,
+		const struct queue_limits *lim, unsigned *segs,
+		unsigned max_bytes, unsigned len_align_mask,
+		unsigned start_align_mask)
+{
+	unsigned bytes = min(bio->bi_iter.bi_size, max_bytes);
+	unsigned seg_shift = bio->bi_dmabuf_map->seg_shift;
+	unsigned offset = bio->bi_iter.bi_offset & ((1U << seg_shift) - 1);
+
+	if ((bio->bi_iter.bi_offset & start_align_mask) ||
+	    (bio->bi_iter.bi_size & len_align_mask))
+		return -EINVAL;
+
+	/* single contiguous range into the dma-buf */
+	*segs = 1;
+
+	bytes = min(bytes, ((unsigned)lim->max_segments << seg_shift) - offset);
+	if (bytes != bio->bi_iter.bi_size)
+		return bytes;
+	return 0;
+}
+
 /**
  * bio_split_io_at - check if and where to split a bio
  * @bio:  [in] bio to be split
@@ -346,6 +369,19 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
 		len_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
 	}
 
+	if (op_is_dmabuf(bio->bi_opf)) {
+		int ret;
+
+		ret = bio_split_io_at_dmabuf(bio, lim, &nsegs, max_bytes,
+					     len_align_mask, start_align_mask);
+		if (ret < 0)
+			return ret;
+		if (!ret)
+			goto out;
+		bytes = ret;
+		goto split;
+	}
+
 	bio_for_each_bvec(bv, bio, iter) {
 		if (bv.bv_offset & start_align_mask ||
 		    bv.bv_len & len_align_mask)
@@ -376,6 +412,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
 		bvprvp = &bvprv;
 	}
 
+out:
 	*segs = nsegs;
 	bio->bi_bvec_gap_bit = ffs(gaps);
 	return 0;
diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c
index bfdb9ed707411..88fd9cbc951f4 100644
--- a/block/blk-mq-dma.c
+++ b/block/blk-mq-dma.c
@@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,
 	 * one could be merged into it.  This typically happens when moving to
 	 * the next bio, but some callers also don't pack bvecs tight.
 	 */
-	while (!iter->iter.bi_size || !iter->iter.bi_bvec_done) {
+	while (!iter->iter.bi_size || !iter->iter.bi_offset) {
 		struct bio_vec next;
 
 		if (!__blk_map_iter_next(iter))
diff --git a/block/blk.h b/block/blk.h
index eaac05815cb03..50abfd9328861 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio *bio,
 		return true;
 
 	bv = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
-	if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done)
+	if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_offset)
 		return true;
 	if ((bv->bv_offset | bv->bv_len) & lim->dma_alignment)
 		return true;
diff --git a/block/fops.c b/block/fops.c
index 3c2099dfef1d5..3c403afb0f646 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -342,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	bio->bi_end_io = blkdev_bio_end_io_async;
 	bio->bi_ioprio = iocb->ki_ioprio;
 
-	if (iov_iter_is_bvec(iter)) {
-		/*
-		 * Users don't rely on the iterator being in any particular
-		 * state for async I/O returning -EIOCBQUEUED, hence we can
-		 * avoid expensive iov_iter_advance(). Bypass
-		 * bio_iov_iter_get_pages() and set the bvec directly.
-		 */
-		bio_iov_bvec_set(bio, iter);
-	} else {
+	/*
+	 * Users don't rely on the iterator being in any particular
+	 * state for async I/O returning -EIOCBQUEUED, hence we can
+	 * avoid expensive iov_iter_advance(). Bypass
+	 * bio_iov_iter_get_pages() and set the bvec/dmabuf directly.
+	 */
+	if (!bio_iov_iter_set(bio, iter)) {
 		ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
 		if (unlikely(ret))
 			goto out_bio_put;
@@ -769,6 +767,9 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
 			ret = direct_write_fallback(iocb, from, ret,
 					blkdev_buffered_write(iocb, from));
 	} else {
+		if (unlikely(iov_iter_is_dmabuf_map(from)))
+			return -EOPNOTSUPP;
+
 		/*
 		 * Take i_rwsem and invalidate_lock to avoid racing with
 		 * set_blocksize changing i_blkbits/folio order and punching
@@ -823,6 +824,8 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		if (ret < 0 || !count)
 			goto reexpand;
 	}
+	if (unlikely(iov_iter_is_dmabuf_map(to)))
+		return -EOPNOTSUPP;
 
 	/*
 	 * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize
@@ -926,6 +929,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc *desc)
 	return generic_file_mmap_prepare(desc);
 }
 
+static int blkdev_init_dma_buf_io_ctx(struct file *file,
+				      struct dma_buf_io_ctx *ctx)
+{
+	struct block_device *bdev = file_bdev(file);
+	struct gendisk *disk = bdev->bd_disk;
+
+	if (!(file->f_flags & O_DIRECT))
+		return -EINVAL;
+	if (!disk->fops->init_dma_buf_io_ctx)
+		return -EINVAL;
+	return disk->fops->init_dma_buf_io_ctx(bdev, ctx);
+}
+
 const struct file_operations def_blk_fops = {
 	.open		= blkdev_open,
 	.release	= blkdev_release,
@@ -944,6 +960,7 @@ const struct file_operations def_blk_fops = {
 	.fallocate	= blkdev_fallocate,
 	.uring_cmd	= blkdev_uring_cmd,
 	.fop_flags	= FOP_BUFFER_RASYNC | FOP_DONTCACHE,
+	.init_dma_buf_io_ctx = blkdev_init_dma_buf_io_ctx,
 };
 
 static __init int blkdev_init(void)
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 26d7130c3f559..8639fa34b8470 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -379,7 +379,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,
 		iov_iter_bvec(&iter, rw,
 			__bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter),
 			nr_bvec, blk_rq_bytes(rq));
-		iter.iov_offset = rq->bio->bi_iter.bi_bvec_done;
+		iter.iov_offset = rq->bio->bi_iter.bi_offset;
 	}
 	atomic_set(&cmd->ref, 2);
 
diff --git a/drivers/block/zloop.c b/drivers/block/zloop.c
index f97a20cfdb7ce..0fef5db8e5b80 100644
--- a/drivers/block/zloop.c
+++ b/drivers/block/zloop.c
@@ -554,7 +554,7 @@ static int zloop_do_rw(struct zloop_cmd *cmd)
 		iov_iter_bvec(&iter, rw,
 			__bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter),
 					nr_bvec, blk_rq_bytes(rq));
-		iter.iov_offset = rq->bio->bi_iter.bi_bvec_done;
+		iter.iov_offset = rq->bio->bi_iter.bi_offset;
 	}
 
 	cmd->iocb.ki_pos = (cmd->sector - zone->start) << SECTOR_SHIFT;
diff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile
index b25d7550bacfd..523731b0f83ee 100644
--- a/drivers/dma-buf/Makefile
+++ b/drivers/dma-buf/Makefile
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \
-	 dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o
+	 dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o
 obj-$(CONFIG_DMABUF_HEAPS)	+= dma-heap.o
 obj-$(CONFIG_DMABUF_HEAPS)	+= heaps/
 obj-$(CONFIG_SYNC_FILE)		+= sync_file.o
diff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c
new file mode 100644
index 0000000000000..24f58f80c45c4
--- /dev/null
+++ b/drivers/dma-buf/dma-buf-io.c
@@ -0,0 +1,278 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Common infrastructure for supporing dma-buf in the I/O path.
+ *
+ * Copyright (C) 2026 Pavel Begunkov <asml.silence@gmail.com>
+ */
+#include <linux/dma-buf-io.h>
+#include <linux/dma-resv.h>
+
+struct dma_buf_io_fence {
+	struct dma_fence base;
+	spinlock_t lock;
+};
+
+static const char *dma_buf_io_fence_drv_name(struct dma_fence *fence)
+{
+	/* default fence release kfree's the base pointer */
+	BUILD_BUG_ON(offsetof(struct dma_buf_io_fence, base));
+
+	return "dma-buf-io-ctx";
+}
+
+static const char *dma_buf_io_fence_timeline_name(struct dma_fence *fence)
+{
+	return "dma-buf-io-ctx";
+}
+
+const struct dma_fence_ops dma_buf_io_fence_ops = {
+	.get_driver_name = dma_buf_io_fence_drv_name,
+	.get_timeline_name = dma_buf_io_fence_timeline_name,
+};
+
+static void dma_buf_io_ctx_destroy_work(struct work_struct *work)
+{
+	struct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx,
+						  release_work);
+
+	if (WARN_ON_ONCE(refcount_read(&ctx->refs)))
+		return;
+
+	ctx->dev_ops->release(ctx);
+	dma_buf_put(ctx->dmabuf);
+	kfree(ctx);
+}
+
+static void dma_buf_io_map_release_work(struct work_struct *work)
+{
+	struct dma_buf_io_map *map = container_of(work, struct dma_buf_io_map,
+						  release_work);
+	struct dma_buf_io_fence *fence = map->fence;
+	struct dma_buf_io_ctx *ctx = map->ctx;
+	struct dma_buf *dmabuf = ctx->dmabuf;
+
+	/* the release path must wait for fences */
+	if (WARN_ON_ONCE(refcount_read(&ctx->refs) == 0))
+		return;
+
+	/* Prevent from destoying the ctx while unmapping */
+	refcount_inc(&ctx->refs);
+
+	/*
+	 * There are no more requests using the map, we can signal the fence.
+	 * It should be done before taking the resv lock as someone could be
+	 * waiting for the fence while holding the lock.
+	 */
+	dma_fence_signal(&fence->base);
+
+	dma_resv_lock(dmabuf->resv, NULL);
+	ctx->dev_ops->unmap(ctx, map);
+	dma_resv_unlock(dmabuf->resv);
+
+	dma_fence_put(&fence->base);
+	percpu_ref_exit(&map->refs);
+	kfree(map);
+
+	if (refcount_dec_and_test(&ctx->refs)) {
+		/*
+		 * Destruction needs to wait for I/O and dma fences. Defer it to
+		 * simplify locking.
+		 */
+		INIT_WORK(&ctx->release_work, dma_buf_io_ctx_destroy_work);
+		queue_work(system_wq, &ctx->release_work);
+	}
+}
+
+static void dma_buf_io_map_refs_release(struct percpu_ref *ref)
+{
+	struct dma_buf_io_map *map = container_of(ref, struct dma_buf_io_map, refs);
+
+	/* might sleep, use a worker */
+	INIT_WORK(&map->release_work, dma_buf_io_map_release_work);
+	queue_work(system_wq, &map->release_work);
+}
+
+int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map)
+{
+	struct dma_buf_io_fence *fence = NULL;
+	int ret;
+
+	fence = kzalloc(sizeof(*fence), GFP_KERNEL);
+	if (!fence)
+		return -ENOMEM;
+
+	ret = percpu_ref_init(&map->refs, dma_buf_io_map_refs_release, 0, GFP_KERNEL);
+	if (ret) {
+		kfree(fence);
+		return ret;
+	}
+
+	spin_lock_init(&fence->lock);
+	dma_fence_init(&fence->base, &dma_buf_io_fence_ops, &fence->lock,
+			ctx->fence_ctx, atomic_inc_return(&ctx->fence_seq));
+	map->fence = fence;
+	map->ctx = ctx;
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, "DMA_BUF");
+
+struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx)
+{
+	struct dma_buf *dmabuf = ctx->dmabuf;
+	struct dma_buf_io_map *map;
+	long ret;
+
+retry:
+	/*
+	 * ->dmabuf_map() will be calling dma_buf_map_attachment(), for which
+	 * we'll need to wait for fences. Do a bit nicer and try to wait
+	 * without the resv lock first.
+	 */
+	ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+				    true, MAX_SCHEDULE_TIMEOUT);
+	if (!ret)
+		ret = -EAGAIN;
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	ret = dma_resv_lock_interruptible(dmabuf->resv, NULL);
+	if (ret)
+		return ERR_PTR(ret);
+
+	map = dma_buf_io_get_map(ctx);
+	if (map) {
+		ret = 0;
+		goto out;
+	}
+
+	if (dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+				  true, 0) < 0) {
+		dma_resv_unlock(dmabuf->resv);
+		goto retry;
+	}
+
+	map = ctx->dev_ops->map(ctx);
+	if (IS_ERR(map)) {
+		ret = PTR_ERR(map);
+		goto out;
+	}
+
+	if (WARN_ON_ONCE(!map->seg_shift))
+		return ERR_PTR(-EFAULT);
+
+	percpu_ref_get(&map->refs);
+	rcu_assign_pointer(ctx->map, map);
+out:
+	dma_resv_unlock(dmabuf->resv);
+	if (ret < 0)
+		return ERR_PTR(ret);
+	return map;
+}
+
+static void dma_buf_io_drop_map(struct dma_buf_io_ctx *ctx)
+{
+	struct dma_buf *dmabuf = ctx->dmabuf;
+	struct dma_buf_io_map *map;
+	int ret;
+
+	dma_resv_assert_held(dmabuf->resv);
+
+	map = rcu_dereference_protected(ctx->map,
+					dma_resv_held(dmabuf->resv));
+	if (!map)
+		return;
+	rcu_assign_pointer(ctx->map, NULL);
+
+	ret = dma_resv_reserve_fences(dmabuf->resv, 1);
+	if (WARN_ON_ONCE(ret)) {
+		struct dma_fence *fence = &map->fence->base;
+
+		dma_fence_get(fence);
+		percpu_ref_kill(&map->refs);
+		dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+		return;
+	}
+
+	dma_resv_add_fence(dmabuf->resv, &map->fence->base,
+			   DMA_RESV_USAGE_KERNEL);
+	/*
+	 * Delay destruction until all inflight requests using the map are
+	 * gone. It'll also signal the fence then.
+	 */
+	percpu_ref_kill(&map->refs);
+}
+
+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx)
+{
+	dma_buf_io_drop_map(ctx);
+}
+EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, "DMA_BUF");
+
+static void dma_buf_io_ctx_release_work(struct work_struct *work)
+{
+	struct dma_buf_io_ctx *ctx = container_of(work, struct dma_buf_io_ctx,
+						  release_work);
+	struct dma_buf *dmabuf = ctx->dmabuf;
+	long ret;
+
+	dma_resv_lock(dmabuf->resv, NULL);
+	/* Remove the last map, there should be no new ones going forward. */
+	dma_buf_io_drop_map(ctx);
+	dma_resv_unlock(dmabuf->resv);
+
+	/* Wait until all maps are destroyed. */
+	ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+				    false, MAX_SCHEDULE_TIMEOUT);
+
+	if (WARN_ON_ONCE(ret <= 0))
+		return;
+	if (WARN_ON_ONCE(rcu_dereference_protected(ctx->map, true)))
+		return;
+
+	if (refcount_dec_and_test(&ctx->refs))
+		dma_buf_io_ctx_destroy_work(&ctx->release_work);
+}
+
+void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx)
+{
+	/*
+	 * Destruction needs to wait for I/O and dma fences. Defer it to
+	 * simplify locking.
+	 */
+	INIT_WORK(&ctx->release_work, dma_buf_io_ctx_release_work);
+	queue_work(system_wq, &ctx->release_work);
+}
+
+int dma_buf_io_ctx_create(struct file *file,
+			   struct dma_buf_io_ctx *ctx,
+			   struct dma_buf *dmabuf,
+			   enum dma_data_direction dir)
+{
+	int ret;
+
+	if (!file->f_op->init_dma_buf_io_ctx)
+		return -EOPNOTSUPP;
+
+	memset(ctx, 0, sizeof(*ctx));
+	ctx->fence_ctx = dma_fence_context_alloc(1);
+	ctx->dir = dir;
+	ctx->dmabuf = dmabuf;
+	refcount_set(&ctx->refs, 1);
+	get_dma_buf(dmabuf);
+
+	ret = file->f_op->init_dma_buf_io_ctx(file, ctx);
+	if (ret) {
+		memset(ctx, 0, sizeof(*ctx));
+		dma_buf_put(dmabuf);
+		return ret;
+	}
+
+	if (WARN_ON_ONCE(!ctx->dev_ops ||
+			 !ctx->dev_ops->map ||
+			 !ctx->dev_ops->unmap ||
+			 !ctx->dev_ops->release))
+		return -EINVAL;
+
+	return ret;
+}
diff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c
index 6155b0117c9d6..b22719c6411cf 100644
--- a/drivers/md/dm-io-rewind.c
+++ b/drivers/md/dm-io-rewind.c
@@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,
 	int idx;
 
 	iter->bi_size += bytes;
-	if (bytes <= iter->bi_bvec_done) {
-		iter->bi_bvec_done -= bytes;
+	if (bytes <= iter->bi_offset) {
+		iter->bi_offset -= bytes;
 		return true;
 	}
 
-	bytes -= iter->bi_bvec_done;
+	bytes -= iter->bi_offset;
 	idx = iter->bi_idx - 1;
 
 	while (idx >= 0 && bytes && bytes > bv[idx].bv_len) {
@@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,
 	if (WARN_ONCE(idx < 0 && bytes,
 		      "Attempted to rewind iter beyond bvec's boundaries\n")) {
 		iter->bi_size -= bytes;
-		iter->bi_bvec_done = 0;
+		iter->bi_offset = 0;
 		iter->bi_idx = 0;
 		return false;
 	}
 
 	iter->bi_idx = idx;
-	iter->bi_bvec_done = bv[idx].bv_len - bytes;
+	iter->bi_offset = bv[idx].bv_len - bytes;
 	return true;
 }
 
@@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bio *bio,
 	iter->bi_sector -= bytes >> 9;
 
 	/* No advance means no rewind */
-	if (bio_no_advance_iter(bio))
+	if (bio_no_advance_iter(bio)) {
 		iter->bi_size += bytes;
-	else
+		iter->bi_offset -= bytes;
+	} else {
 		dm_bvec_iter_rewind(bio->bi_io_vec, iter, bytes);
+	}
 }
 
 /**
diff --git a/drivers/md/dm-pcache/segment.c b/drivers/md/dm-pcache/segment.c
index 7e98187014451..8f8816e1c539f 100644
--- a/drivers/md/dm-pcache/segment.c
+++ b/drivers/md/dm-pcache/segment.c
@@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment *segment,
 
 	iov_iter_bvec(&iter, ITER_DEST, &bio->bi_io_vec[bio->bi_iter.bi_idx],
 			bio_segments(bio), bio->bi_iter.bi_size);
-	iter.iov_offset = bio->bi_iter.bi_bvec_done;
+	iter.iov_offset = bio->bi_iter.bi_offset;
 	if (bio_off)
 		iov_iter_advance(&iter, bio_off);
 
@@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment *segment,
 
 	iov_iter_bvec(&iter, ITER_SOURCE, &bio->bi_io_vec[bio->bi_iter.bi_idx],
 			bio_segments(bio), bio->bi_iter.bi_size);
-	iter.iov_offset = bio->bi_iter.bi_bvec_done;
+	iter.iov_offset = bio->bi_iter.bi_offset;
 	if (bio_off)
 		iov_iter_advance(&iter, bio_off);
 
diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c
index 7e1112960d7f8..5d910a64503d3 100644
--- a/drivers/nvdimm/btt.c
+++ b/drivers/nvdimm/btt.c
@@ -1155,7 +1155,7 @@ static int btt_rw_integrity(struct btt *btt, struct bio_integrity_payload *bip,
 		bv = bvec_iter_bvec(bip->bip_vec, bip->bip_iter);
 		/*
 		 * The 'bv' obtained from bvec_iter_bvec has its .bv_len and
-		 * .bv_offset already adjusted for iter->bi_bvec_done, and we
+		 * .bv_offset already adjusted for iter->bi_offset, and we
 		 * can use those directly
 		 */
 
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 453c1f0b2dd09..8bd407603d928 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2676,6 +2676,17 @@ static int nvme_report_zones(struct gendisk *disk, sector_t sector,
 #define nvme_report_zones	NULL
 #endif /* CONFIG_BLK_DEV_ZONED */
 
+static int nvme_init_dma_buf_io_ctx(struct block_device *bdev,
+		struct dma_buf_io_ctx *ctx)
+{
+	struct nvme_ns *ns = bdev->bd_disk->private_data;
+	struct nvme_ctrl *ctrl = ns->ctrl;
+
+	if (!ctrl->ops->init_dma_buf_io_ctx)
+		return -EINVAL;
+	return ctrl->ops->init_dma_buf_io_ctx(ctrl, ctx);
+}
+
 const struct block_device_operations nvme_bdev_ops = {
 	.owner		= THIS_MODULE,
 	.ioctl		= nvme_ioctl,
@@ -2686,6 +2697,7 @@ const struct block_device_operations nvme_bdev_ops = {
 	.get_unique_id	= nvme_get_unique_id,
 	.report_zones	= nvme_report_zones,
 	.pr_ops		= &nvme_pr_ops,
+	.init_dma_buf_io_ctx = nvme_init_dma_buf_io_ctx,
 };
 
 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u32 mask, u32 val,
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 824651cc898db..034c31af8fecf 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -652,6 +652,8 @@ struct nvme_ctrl_ops {
 	int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
 	void (*print_device_info)(struct nvme_ctrl *ctrl);
 	bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);
+	int (*init_dma_buf_io_ctx)(struct nvme_ctrl *ctrl,
+				   struct dma_buf_io_ctx *ctx);
 	unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin);
 };
 
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b537..67e26d6e8f255 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -27,6 +27,8 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/io-64-nonatomic-hi-lo.h>
 #include <linux/sed-opal.h>
+#include <linux/dma-buf-io.h>
+#include <linux/dma-resv.h>
 
 #include "trace.h"
 #include "nvme.h"
@@ -393,6 +395,13 @@ struct nvme_queue {
 	struct completion delete_done;
 };
 
+struct nvme_dmabuf_map {
+	struct dma_buf_io_map base;
+	struct sg_table *sgt;
+	unsigned nr_entries;
+	dma_addr_t dma_list[];
+};
+
 /* bits for iod->flags */
 enum nvme_iod_flags {
 	/* this command has been aborted by the timeout handler */
@@ -859,6 +868,138 @@ static void nvme_free_descriptors(struct request *req)
 	}
 }
 
+static inline struct nvme_dmabuf_map *
+to_nvme_dmabuf_map(struct dma_buf_io_map *map)
+{
+	return container_of(map, struct nvme_dmabuf_map, base);
+}
+
+static void nvme_dmabuf_map_sync_for_cpu(struct nvme_dev *nvme_dev,
+					 struct request *req)
+{
+	struct device *dev = nvme_dev->dev;
+	enum dma_data_direction dma_dir;
+	struct bio *bio = req->bio;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio->bi_dmabuf_map);
+	dma_addr_t *dma_list = map->dma_list;
+	unsigned offset = bio->bi_iter.bi_offset;
+	unsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;
+	int length = blk_rq_payload_bytes(req) +
+		     (offset & (NVME_CTRL_PAGE_SIZE - 1));
+
+	dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+
+	while (length > 0) {
+		dma_sync_single_for_cpu(dev, dma_list[map_idx++],
+					NVME_CTRL_PAGE_SIZE, dma_dir);
+		length -= NVME_CTRL_PAGE_SIZE;
+	}
+}
+
+static void nvme_dmabuf_map_sync_for_device(struct nvme_dev *nvme_dev,
+					    struct request *req)
+{
+	struct device *dev = nvme_dev->dev;
+	enum dma_data_direction dma_dir;
+	struct bio *bio = req->bio;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio->bi_dmabuf_map);
+	dma_addr_t *dma_list = map->dma_list;
+	unsigned offset = bio->bi_iter.bi_offset;
+	unsigned map_idx = offset / NVME_CTRL_PAGE_SIZE;
+	int length = blk_rq_payload_bytes(req) +
+		     (offset & (NVME_CTRL_PAGE_SIZE - 1));
+
+	dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+
+	while (length > 0) {
+		dma_sync_single_for_device(dev, dma_list[map_idx++],
+					   NVME_CTRL_PAGE_SIZE, dma_dir);
+		length -= NVME_CTRL_PAGE_SIZE;
+	}
+}
+
+static void nvme_rq_clean_dmabuf_map(struct nvme_dev *dev,
+				      struct request *req)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+
+	nvme_dmabuf_map_sync_for_cpu(dev, req);
+
+	if (!(iod->flags & IOD_SINGLE_SEGMENT))
+		nvme_free_descriptors(req);
+}
+
+static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req,
+					     struct nvme_queue *nvmeq)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	struct bio *bio = req->bio;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio->bi_dmabuf_map);
+	unsigned bvec_done = bio->bi_iter.bi_offset;
+	unsigned map_idx = bvec_done / NVME_CTRL_PAGE_SIZE;
+	unsigned offset = bvec_done & (NVME_CTRL_PAGE_SIZE - 1);
+	int length = blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset);
+	dma_addr_t *dma_list = map->dma_list;
+	u64 prp1_dma = dma_list[map_idx++] + offset;
+	u64 dma_addr, prp2_dma;
+	dma_addr_t prp_dma;
+	__le64 *prp_list;
+	unsigned i;
+
+	nvme_dmabuf_map_sync_for_device(nvmeq->dev, req);
+
+	if (length <= 0) {
+		prp2_dma = 0;
+		goto done;
+	}
+
+	if (length <= NVME_CTRL_PAGE_SIZE) {
+		prp2_dma = dma_list[map_idx];
+		goto done;
+	}
+
+	if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <=
+	    NVME_SMALL_POOL_SIZE / sizeof(__le64))
+		iod->flags |= IOD_SMALL_DESCRIPTOR;
+
+	prp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,
+			&prp_dma);
+	if (!prp_list)
+		return BLK_STS_RESOURCE;
+
+	iod->descriptors[iod->nr_descriptors++] = prp_list;
+	prp2_dma = prp_dma;
+	i = 0;
+	for (;;) {
+		if (i == NVME_CTRL_PAGE_SIZE >> 3) {
+			__le64 *old_prp_list = prp_list;
+
+			prp_list = dma_pool_alloc(nvmeq->descriptor_pools.large,
+					GFP_ATOMIC, &prp_dma);
+			if (!prp_list)
+				goto free_prps;
+			iod->descriptors[iod->nr_descriptors++] = prp_list;
+			prp_list[0] = old_prp_list[i - 1];
+			old_prp_list[i - 1] = cpu_to_le64(prp_dma);
+			i = 1;
+		}
+
+		dma_addr = dma_list[map_idx++];
+		prp_list[i++] = cpu_to_le64(dma_addr);
+
+		length -= NVME_CTRL_PAGE_SIZE;
+		if (length <= 0)
+			break;
+	}
+done:
+	iod->cmd.common.dptr.prp1 = cpu_to_le64(prp1_dma);
+	iod->cmd.common.dptr.prp2 = cpu_to_le64(prp2_dma);
+	return BLK_STS_OK;
+free_prps:
+	nvme_free_descriptors(req);
+	return BLK_STS_RESOURCE;
+}
+
 static void nvme_free_prps(struct request *req, unsigned int attrs)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
@@ -937,6 +1078,11 @@ static void nvme_unmap_data(struct request *req)
 	struct device *dma_dev = nvmeq->dev->dev;
 	unsigned int attrs = 0;
 
+	if (blk_mq_rq_is_dmabuf(req)) {
+		nvme_rq_clean_dmabuf_map(nvmeq->dev, req);
+		return;
+	}
+
 	if (iod->flags & IOD_SINGLE_SEGMENT) {
 		static_assert(offsetof(union nvme_data_ptr, prp1) ==
 				offsetof(union nvme_data_ptr, sgl.addr));
@@ -1142,13 +1288,19 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,
 }
 
 static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge,
-		struct blk_dma_iter *iter)
+		dma_addr_t addr, u32 len)
 {
-	sge->addr = cpu_to_le64(iter->addr);
-	sge->length = cpu_to_le32(iter->len);
+	sge->addr = cpu_to_le64(addr);
+	sge->length = cpu_to_le32(len);
 	sge->type = NVME_SGL_FMT_DATA_DESC << 4;
 }
 
+static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge,
+		struct blk_dma_iter *iter)
+{
+	nvme_pci_sgl_set_data(sge, iter->addr, iter->len);
+}
+
 static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
 		dma_addr_t dma_addr, int entries)
 {
@@ -1157,6 +1309,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge,
 	sge->type = NVME_SGL_FMT_LAST_SEG_DESC << 4;
 }
 
+static unsigned int nvme_pci_dmabuf_sgl_nents(struct request *req)
+{
+	struct bio *bio = req->bio;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio->bi_dmabuf_map);
+	struct scatterlist *sg;
+	unsigned long tmp;
+	size_t offset = bio->bi_iter.bi_offset;
+	size_t remaining = blk_rq_payload_bytes(req);
+	unsigned int nents = 0;
+
+	for_each_sgtable_dma_sg(map->sgt, sg, tmp) {
+		size_t sg_len = sg_dma_len(sg);
+
+		if (!remaining)
+			break;
+		if (offset >= sg_len) {
+			offset -= sg_len;
+			continue;
+		}
+
+		sg_len -= offset;
+		offset = 0;
+
+		do {
+			size_t chunk = min(remaining, sg_len);
+
+			nents++;
+			sg_len -= chunk;
+			remaining -= chunk;
+		} while (sg_len && remaining);
+	}
+
+	if (unlikely(remaining))
+		return 0;
+
+	return nents;
+}
+
+static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request *req,
+		struct nvme_queue *nvmeq, unsigned int entries)
+{
+	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+	struct bio *bio = req->bio;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(bio->bi_dmabuf_map);
+	size_t length = blk_rq_payload_bytes(req);
+	struct nvme_sgl_desc *sg_list = &iod->cmd.common.dptr.sgl;
+	bool pooled = entries > 1;
+	dma_addr_t sgl_dma = 0;
+	unsigned int mapped = 0;
+	unsigned long tmp;
+	struct scatterlist *sg;
+	size_t offset = bio->bi_iter.bi_offset;
+	size_t remaining = length;
+
+	if (!entries)
+		return BLK_STS_IOERR;
+
+	iod->cmd.common.flags = NVME_CMD_SGL_METABUF;
+	iod->total_len = length;
+
+	nvme_dmabuf_map_sync_for_device(nvmeq->dev, req);
+
+	/* entries == 1 fits in the inline descriptor; more needs a pool. */
+	if (pooled) {
+		if (entries <= NVME_SMALL_POOL_SIZE / sizeof(*sg_list))
+			iod->flags |= IOD_SMALL_DESCRIPTOR;
+
+		sg_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,
+					 &sgl_dma);
+		if (!sg_list)
+			return BLK_STS_RESOURCE;
+		iod->descriptors[iod->nr_descriptors++] = sg_list;
+	}
+
+	for_each_sgtable_dma_sg(map->sgt, sg, tmp) {
+		size_t sg_len = sg_dma_len(sg);
+		dma_addr_t addr = sg_dma_address(sg);
+
+		if (!remaining)
+			break;
+		if (offset >= sg_len) {
+			offset -= sg_len;
+			continue;
+		}
+
+		addr += offset;
+		sg_len -= offset;
+		offset = 0;
+
+		do {
+			u32 chunk = min_t(size_t, remaining, sg_len);
+
+			if (WARN_ON_ONCE(mapped == entries))
+				goto err_free;
+			nvme_pci_sgl_set_data(&sg_list[mapped++], addr, chunk);
+
+			addr += chunk;
+			sg_len -= chunk;
+			remaining -= chunk;
+		} while (sg_len && remaining);
+	}
+
+	if (unlikely(remaining))
+		goto err_free;
+
+	if (pooled)
+		nvme_pci_sgl_set_seg(&iod->cmd.common.dptr.sgl, sgl_dma,
+				     mapped);
+	return BLK_STS_OK;
+err_free:
+	if (pooled) {
+		iod->nr_descriptors--;
+		dma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma);
+	}
+	return BLK_STS_IOERR;
+}
+
+static blk_status_t nvme_rq_setup_dmabuf(struct request *req,
+		struct nvme_queue *nvmeq, enum nvme_use_sgl use_sgl)
+{
+	unsigned int entries;
+	size_t avg_seg;
+
+	if (use_sgl == SGL_UNSUPPORTED)
+		return nvme_rq_setup_dmabuf_map(req, nvmeq);
+
+	entries = nvme_pci_dmabuf_sgl_nents(req);
+
+	if (use_sgl == SGL_FORCED)
+		return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);
+
+	avg_seg = entries ?
+		DIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0;
+	if (sgl_threshold && avg_seg >= sgl_threshold)
+		return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries);
+
+	return nvme_rq_setup_dmabuf_map(req, nvmeq);
+}
+
 static blk_status_t nvme_pci_setup_data_sgl(struct request *req,
 		struct blk_dma_iter *iter)
 {
@@ -1171,7 +1462,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req,
 	iod->cmd.common.flags = NVME_CMD_SGL_METABUF;
 
 	if (entries == 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) {
-		nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter);
+		nvme_pci_dma_iter_set_sgl(&iod->cmd.common.dptr.sgl, iter);
 		iod->total_len += iter->len;
 		return BLK_STS_OK;
 	}
@@ -1193,7 +1484,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req,
 			iter->status = BLK_STS_IOERR;
 			break;
 		}
-		nvme_pci_sgl_set_data(&sg_list[mapped++], iter);
+		nvme_pci_dma_iter_set_sgl(&sg_list[mapped++], iter);
 		iod->total_len += iter->len;
 	} while (blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter));
 
@@ -1251,6 +1542,9 @@ static blk_status_t nvme_map_data(struct request *req)
 	struct blk_dma_iter iter;
 	blk_status_t ret;
 
+	if (blk_mq_rq_is_dmabuf(req))
+		return nvme_rq_setup_dmabuf(req, nvmeq, use_sgl);
+
 	/*
 	 * Try to skip the DMA iterator for single segment requests, as that
 	 * significantly improves performances for small I/O sizes.
@@ -1353,13 +1647,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request *req)
 	iod->cmd.common.metadata = cpu_to_le64(sgl_dma);
 	if (entries == 1) {
 		iod->meta_total_len = iter.len;
-		nvme_pci_sgl_set_data(sg_list, &iter);
+		nvme_pci_dma_iter_set_sgl(sg_list, &iter);
 		return BLK_STS_OK;
 	}
 
 	sgl_dma += sizeof(*sg_list);
 	do {
-		nvme_pci_sgl_set_data(&sg_list[++i], &iter);
+		nvme_pci_dma_iter_set_sgl(&sg_list[++i], &iter);
 		iod->meta_total_len += iter.len;
 	} while (blk_rq_integrity_dma_map_iter_next(req, dev->dev, &iter));
 
@@ -2269,6 +2563,118 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 	return result;
 }
 
+#ifdef CONFIG_DMA_SHARED_BUFFER
+static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *attach)
+{
+	struct dma_buf_io_ctx *ctx = attach->importer_priv;
+
+	dma_buf_io_invalidate_mappings(ctx);
+}
+
+const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = {
+	.invalidate_mappings	= nvme_dmabuf_invalidate_mappings,
+	.allow_peer2peer	= true,
+};
+
+static struct dma_buf_io_map *nvme_dma_buf_io_map(struct dma_buf_io_ctx *ctx)
+{
+	unsigned nr_entries = ctx->dmabuf->size / NVME_CTRL_PAGE_SIZE;
+	struct dma_buf_attachment *attach = ctx->dev_priv;
+	unsigned long tmp, i = 0;
+	unsigned seg_shift = ~0U;
+	struct nvme_dmabuf_map *map;
+	struct scatterlist *sg;
+	struct sg_table *sgt;
+	int ret;
+
+	dma_resv_assert_held(ctx->dmabuf->resv);
+
+	map = kmalloc_flex(*map, dma_list, nr_entries);
+	if (!map)
+		return ERR_PTR(-ENOMEM);
+
+	sgt = dma_buf_map_attachment(attach, ctx->dir);
+	if (IS_ERR(sgt)) {
+		ret = PTR_ERR(sgt);
+		sgt = NULL;
+		goto err;
+	}
+
+	for_each_sgtable_dma_sg(sgt, sg, tmp) {
+		dma_addr_t dma_addr = sg_dma_address(sg);
+		unsigned long sg_len = sg_dma_len(sg);
+
+		if (sg_len % NVME_CTRL_PAGE_SIZE) {
+			ret = -EINVAL;
+			goto err;
+		}
+		seg_shift = min(seg_shift, __ffs(sg_len));
+
+		while (sg_len) {
+			map->dma_list[i++] = dma_addr;
+			dma_addr += NVME_CTRL_PAGE_SIZE;
+			sg_len -= NVME_CTRL_PAGE_SIZE;
+		}
+	}
+
+	if (WARN_ON_ONCE(seg_shift < NVME_CTRL_PAGE_SHIFT))
+		return ERR_PTR(-EFAULT);
+
+	ret = dma_buf_io_init_map(ctx, &map->base);
+	if (ret)
+		goto err;
+	map->base.seg_shift = seg_shift;
+	map->nr_entries = nr_entries;
+	map->sgt = sgt;
+	return &map->base;
+err:
+	if (sgt)
+		dma_buf_unmap_attachment(attach, sgt, ctx->dir);
+	kfree(map);
+	return ERR_PTR(ret);
+}
+
+static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx *ctx,
+				  struct dma_buf_io_map *map_base)
+{
+	struct dma_buf_attachment *attach = ctx->dev_priv;
+	struct nvme_dmabuf_map *map = to_nvme_dmabuf_map(map_base);
+
+	dma_resv_assert_held(ctx->dmabuf->resv);
+
+	dma_buf_unmap_attachment(attach, map->sgt, ctx->dir);
+}
+
+static void nvme_dma_buf_io_release(struct dma_buf_io_ctx *ctx)
+{
+	struct dma_buf_attachment *attach = ctx->dev_priv;
+
+	dma_buf_detach(ctx->dmabuf, attach);
+}
+
+const struct dma_buf_io_ops nvme_dma_buf_io_ops = {
+	.map		= nvme_dma_buf_io_map,
+	.unmap		= nvme_dma_buf_io_unmap,
+	.release	= nvme_dma_buf_io_release,
+};
+
+static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl,
+					struct dma_buf_io_ctx *ctx)
+{
+	struct dma_buf_attachment *attach;
+	struct nvme_dev *dev = to_nvme_dev(ctrl);
+
+	attach = dma_buf_dynamic_attach(ctx->dmabuf, dev->dev,
+					&nvme_dmabuf_importer_ops, ctx);
+	if (IS_ERR(attach))
+		return PTR_ERR(attach);
+
+	ctx->dev_priv = attach;
+	ctx->dev_ops = &nvme_dma_buf_io_ops;
+	return 0;
+}
+#endif
+
 static const struct blk_mq_ops nvme_mq_admin_ops = {
 	.queue_rq	= nvme_queue_rq,
 	.complete	= nvme_pci_complete_rq,
@@ -3563,6 +3969,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
 	.print_device_info	= nvme_pci_print_device_info,
 	.supports_pci_p2pdma	= nvme_pci_supports_pci_p2pdma,
 	.get_virt_boundary	= nvme_pci_get_virt_boundary,
+#ifdef CONFIG_DMA_SHARED_BUFFER
+	.init_dma_buf_io_ctx	= nvme_pci_init_dma_buf_io_ctx,
+#endif
 };
 
 static int nvme_dev_map(struct nvme_dev *dev)
@@ -4327,5 +4736,6 @@ MODULE_AUTHOR("Matthew Wilcox <willy@linux.intel.com>");
 MODULE_LICENSE("GPL");
 MODULE_VERSION("1.0");
 MODULE_DESCRIPTION("NVMe host PCIe transport driver");
+MODULE_IMPORT_NS("DMA_BUF");
 module_init(nvme_init);
 module_exit(nvme_exit);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ba5c7b3e2a7c6..ce03a0ea4ded7 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -357,7 +357,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
 		iov_iter_bvec(&req->iter, dir,
 			__bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter), nr_bvec,
 			bio->bi_iter.bi_size);
-		req->iter.iov_offset = bio->bi_iter.bi_bvec_done;
+		req->iter.iov_offset = bio->bi_iter.bi_offset;
 	}
 }
 
diff --git a/fs/btrfs/misc.h b/fs/btrfs/misc.h
index 694be6d0562a4..8020609431801 100644
--- a/fs/btrfs/misc.h
+++ b/fs/btrfs/misc.h
@@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio *bio)
 		.bi_sector = 0,
 		.bi_size = bio_size,
 		.bi_idx = 0,
-		.bi_bvec_done = 0,
+		.bi_offset = 0,
 	};
 }
 
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 0445ecba3b247..22ce9deb2febb 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio *bio)
 {
 	return bio_op(bio) == REQ_OP_DISCARD ||
 	       bio_op(bio) == REQ_OP_SECURE_ERASE ||
-	       bio_op(bio) == REQ_OP_WRITE_ZEROES;
+	       bio_op(bio) == REQ_OP_WRITE_ZEROES ||
+	       op_is_dmabuf(bio->bi_opf);
 }
 
 static inline void *bio_data(struct bio *bio)
@@ -113,11 +114,13 @@ static inline void bio_advance_iter(const struct bio *bio,
 {
 	iter->bi_sector += bytes >> 9;
 
-	if (bio_no_advance_iter(bio))
+	if (bio_no_advance_iter(bio)) {
 		iter->bi_size -= bytes;
-	else
+		iter->bi_offset += bytes;
+	} else {
 		bvec_iter_advance(bio->bi_io_vec, iter, bytes);
 		/* TODO: It is reasonable to complete bio with error here. */
+	}
 }
 
 /* @bytes should be less or equal to bvec[i->bi_idx].bv_len */
@@ -127,10 +130,12 @@ static inline void bio_advance_iter_single(const struct bio *bio,
 {
 	iter->bi_sector += bytes >> 9;
 
-	if (bio_no_advance_iter(bio))
+	if (bio_no_advance_iter(bio)) {
 		iter->bi_size -= bytes;
-	else
+		iter->bi_offset += bytes;
+	} else {
 		bvec_iter_advance_single(bio->bi_io_vec, iter, bytes);
+	}
 }
 
 void __bio_advance(struct bio *, unsigned bytes);
@@ -434,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio *bio)
 
 /*
  * Calculate number of bvec segments that should be allocated to fit data
- * pointed by @iter. If @iter is backed by bvec it's going to be reused
- * instead of allocating a new one.
+ * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec array /
+ * the dma map are going to be reused, and so no extra allocation is required.
  */
 static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs)
 {
-	if (iov_iter_is_bvec(iter))
+	if (iov_iter_is_bvec(iter) || iov_iter_is_dmabuf_map(iter))
 		return 0;
 	return iov_iter_npages(iter, max_segs);
 }
@@ -518,7 +523,7 @@ int bdev_rw_virt(struct block_device *bdev, sector_t sector, void *data,
 int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		unsigned mem_align_mask, unsigned len_align_mask);
 
-void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter);
+bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter);
 void __bio_release_pages(struct bio *bio, bool mark_dirty);
 extern void bio_set_pages_dirty(struct bio *bio);
 extern void bio_check_pages_dirty(struct bio *bio);
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index af878597afb8c..7c7504c84e090 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -1017,6 +1017,13 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq)
 	return rq + 1;
 }
 
+static inline bool blk_mq_rq_is_dmabuf(struct request *rq)
+{
+	if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))
+		return false;
+	return rq->bio && op_is_dmabuf(rq->bio->bi_opf);
+}
+
 static inline struct blk_mq_hw_ctx *queue_hctx(struct request_queue *q, int id)
 {
 	struct blk_mq_hw_ctx *hctx;
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index d49d97a050d0e..a305951f03124 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -233,7 +233,12 @@ struct bio {
 	atomic_t		__bi_remaining;
 
 	/* The actual vec list, preserved by bio_reset() */
-	struct bio_vec		*bi_io_vec;
+	union {
+		struct bio_vec		*bi_io_vec;
+		/* Driver specific dma map, valid IFF REQ_DMABUF is set */
+		struct dma_buf_io_map	*bi_dmabuf_map;
+	};
+
 	struct bvec_iter	bi_iter;
 
 	union {
@@ -402,6 +407,7 @@ enum req_flag_bits {
 	__REQ_DRV,		/* for driver use */
 	__REQ_FS_PRIVATE,	/* for file system (submitter) use */
 	__REQ_ATOMIC,		/* for atomic write operations */
+	__REQ_DMABUF,		/* Using premmaped dma buffers */
 	/*
 	 * Command specific flags, keep last:
 	 */
@@ -434,6 +440,7 @@ enum req_flag_bits {
 #define REQ_DRV		(__force blk_opf_t)(1ULL << __REQ_DRV)
 #define REQ_FS_PRIVATE	(__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE)
 #define REQ_ATOMIC	(__force blk_opf_t)(1ULL << __REQ_ATOMIC)
+#define REQ_DMABUF	(__force blk_opf_t)(1ULL << __REQ_DMABUF)
 
 #define REQ_NOUNMAP	(__force blk_opf_t)(1ULL << __REQ_NOUNMAP)
 
@@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op)
 	return (op & REQ_OP_MASK) == REQ_OP_DISCARD;
 }
 
+static inline bool op_is_dmabuf(blk_opf_t op)
+{
+	return op & REQ_DMABUF;
+}
+
 /*
  * Check if a bio or request operation is a zone management operation.
  */
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 9213a5716f95a..9f4c92e51dc95 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1682,6 +1682,8 @@ struct block_device_operations {
 	/* returns the length of the identifier or a negative errno: */
 	int (*get_unique_id)(struct gendisk *disk, u8 id[16],
 			enum blk_unique_id id_type);
+	int (*init_dma_buf_io_ctx)(struct block_device *,
+				   struct dma_buf_io_ctx *);
 	struct module *owner;
 	const struct pr_ops *pr_ops;
 
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index 92837e2743f19..b63914ff56e33 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -108,9 +108,10 @@ struct bvec_iter {
 	unsigned int		bi_idx;
 
 	/*
-	 * Current offset in the bvec entry pointed to by `bi_idx`.
+	 * Current offset in the bvec entry pointed to by `bi_idx` or into
+	 * a dma-buf map.
 	 */
-	unsigned int		bi_bvec_done;
+	unsigned int		bi_offset;
 } __packed __aligned(4);
 
 struct bvec_iter_all {
@@ -135,14 +136,14 @@ mp_bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter)
 static __always_inline unsigned int
 mp_bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter)
 {
-	return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_bvec_done,
+	return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_offset,
 			iter.bi_size);
 }
 
 static __always_inline unsigned int
 mp_bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter)
 {
-	return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_bvec_done;
+	return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_offset;
 }
 
 static __always_inline unsigned int
@@ -204,7 +205,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,
 	}
 
 	iter->bi_size -= bytes;
-	bytes += iter->bi_bvec_done;
+	bytes += iter->bi_offset;
 
 	while (bytes && bytes >= bv[idx].bv_len) {
 		bytes -= bv[idx].bv_len;
@@ -212,7 +213,7 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,
 	}
 
 	iter->bi_idx = idx;
-	iter->bi_bvec_done = bytes;
+	iter->bi_offset = bytes;
 	return true;
 }
 
@@ -223,13 +224,13 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,
 static inline void bvec_iter_advance_single(const struct bio_vec *bv,
 				struct bvec_iter *iter, unsigned int bytes)
 {
-	unsigned int done = iter->bi_bvec_done + bytes;
+	unsigned int done = iter->bi_offset + bytes;
 
 	if (done == bv[iter->bi_idx].bv_len) {
 		done = 0;
 		iter->bi_idx++;
 	}
-	iter->bi_bvec_done = done;
+	iter->bi_offset = done;
 	iter->bi_size -= bytes;
 }
 
diff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h
new file mode 100644
index 0000000000000..f31c7375ae911
--- /dev/null
+++ b/include/linux/dma-buf-io.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __DMA_BUF_IO_H__
+#define __DMA_BUF_IO_H__
+
+#include <linux/dma-buf.h>
+
+struct dma_buf_io_fence;
+struct dma_buf_io_ctx;
+struct dma_buf_io_map;
+
+struct dma_buf_io_ops {
+	/*
+	 * Create a new map for the given ctx. Called with the reservation
+	 * lock held.
+	 */
+	struct dma_buf_io_map *(*map)(struct dma_buf_io_ctx *ctx);
+
+	/*
+	 * Clean up device specific parts of the @map. Called with the
+	 * reservation lock held.
+	 */
+	void (*unmap)(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map);
+
+	/*
+	 * The user tries to destroy the ctx. Release all device specific
+	 * parts of the token.
+	 */
+	void (*release)(struct dma_buf_io_ctx *);
+};
+
+struct dma_buf_io_map {
+	/*
+	 * Counts attached requests and other users. Device specific unmapping
+	 * is deferred until all refs are dropped.
+	 */
+	struct percpu_ref		refs;
+	unsigned			seg_shift;
+
+	struct work_struct		release_work;
+	struct dma_buf_io_fence		*fence;
+	struct dma_buf_io_ctx		*ctx;
+};
+
+struct dma_buf_io_ctx {
+	struct dma_buf_io_map __rcu		*map;
+	struct dma_buf				*dmabuf;
+	enum dma_data_direction			dir;
+
+	atomic_t				fence_seq;
+	u64					fence_ctx;
+	struct work_struct			release_work;
+	refcount_t				refs;
+
+	void					*dev_priv;
+	const struct dma_buf_io_ops		*dev_ops;
+};
+
+int dma_buf_io_ctx_create(struct file *file,
+			   struct dma_buf_io_ctx *ctx,
+			   struct dma_buf *dmabuf,
+			   enum dma_data_direction dir);
+void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx);
+
+struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx);
+
+static inline struct dma_buf_io_map *
+dma_buf_io_get_map(struct dma_buf_io_ctx *ctx)
+{
+	struct dma_buf_io_map *map;
+
+	guard(rcu)();
+
+	map = rcu_dereference(ctx->map);
+	if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs)))
+		return NULL;
+
+	return map;
+}
+
+static inline void dma_buf_io_map_drop(struct dma_buf_io_map *map)
+{
+	percpu_ref_put(&map->refs);
+}
+
+/*
+ * Device API
+ */
+
+void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx);
+int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map);
+
+#endif
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 50ce731a2b78f..e62fa653e1484 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1914,6 +1914,7 @@ struct dir_context {
 #define COPY_FILE_SPLICE		(1 << 0)
 
 struct io_uring_cmd;
+struct dma_buf_io_ctx;
 struct offset_ctx;
 
 typedef unsigned int __bitwise fop_flags_t;
@@ -1962,6 +1963,7 @@ struct file_operations {
 	int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *,
 				unsigned int poll_flags);
 	int (*mmap_prepare)(struct vm_area_desc *);
+	int (*init_dma_buf_io_ctx)(struct file *, struct dma_buf_io_ctx *);
 } __randomize_layout;
 
 /* Supports async buffered reads */
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index a2c623a67a251..f90586d353c1c 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -10,6 +10,7 @@
 
 struct iou_loop_params;
 struct io_uring_bpf_ops;
+struct dma_buf_io_map;
 
 enum {
 	/*
@@ -594,6 +595,7 @@ enum {
 	REQ_F_IMPORT_BUFFER_BIT,
 	REQ_F_SQE_COPIED_BIT,
 	REQ_F_IOPOLL_BIT,
+	REQ_F_DROP_DMABUF_BIT,
 
 	/* not a real bit, just to check we're not overflowing the space */
 	__REQ_F_LAST_BIT,
@@ -689,6 +691,8 @@ enum {
 	REQ_F_SQE_COPIED	= IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT),
 	/* request must be iopolled to completion (set in ->issue()) */
 	REQ_F_IOPOLL		= IO_REQ_FLAG(REQ_F_IOPOLL_BIT),
+	/* there is a dma map attached to request that needs to be dropped */
+	REQ_F_DROP_DMABUF	= IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT),
 };
 
 struct io_tw_req {
@@ -811,6 +815,7 @@ struct io_kiocb {
 	/* custom credentials, valid IFF REQ_F_CREDS is set */
 	const struct cred		*creds;
 	struct io_wq_work		work;
+	struct dma_buf_io_map		*dmabuf_map;
 
 	struct io_big_cqe {
 		u64			extra1;
diff --git a/include/linux/uio.h b/include/linux/uio.h
index fe2e985d74d24..638c1116e912e 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -12,6 +12,7 @@
 
 struct page;
 struct folio_queue;
+struct dma_buf_io_map;
 
 typedef unsigned int __bitwise iov_iter_extraction_t;
 
@@ -29,6 +30,7 @@ enum iter_type {
 	ITER_FOLIOQ,
 	ITER_XARRAY,
 	ITER_DISCARD,
+	ITER_DMABUF_MAP,
 };
 
 #define ITER_SOURCE	1	// == WRITE
@@ -71,6 +73,7 @@ struct iov_iter {
 				const struct folio_queue *folioq;
 				struct xarray *xarray;
 				void __user *ubuf;
+				struct dma_buf_io_map *dmabuf_map;
 			};
 			size_t count;
 		};
@@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter *i)
 	return iov_iter_type(i) == ITER_XARRAY;
 }
 
+static inline bool iov_iter_is_dmabuf_map(const struct iov_iter *i)
+{
+	return iov_iter_type(i) == ITER_DMABUF_MAP;
+}
+
 static inline unsigned char iov_iter_rw(const struct iov_iter *i)
 {
 	return i->data_source ? WRITE : READ;
@@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
 			  unsigned int first_slot, unsigned int offset, size_t count);
 void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray,
 		     loff_t start, size_t count);
+void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction,
+			struct dma_buf_io_map *map,
+			loff_t off, size_t count);
 ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
 			size_t maxsize, unsigned maxpages, size_t *start);
 ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages,
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index 909fb7aea638e..c39297e8beb6f 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -790,13 +790,42 @@ struct io_uring_rsrc_update {
 
 struct io_uring_rsrc_update2 {
 	__u32 offset;
-	__u32 resv;
+	__u32 flags;
 	__aligned_u64 data;
 	__aligned_u64 tags;
 	__u32 nr;
 	__u32 resv2;
 };
 
+/* struct io_uring_rsrc_update2::flags */
+enum io_uring_rsrc_reg_flags {
+	/*
+	 * Use the extended descriptor format for buffer updates,
+	 * see struct io_uring_regbuf_desc
+	 */
+	IORING_RSRC_UPDATE_EXTENDED		= 1U << 1,
+};
+
+/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */
+enum io_uring_regbuf_type {
+	IO_REGBUF_TYPE_EMPTY,
+	IO_REGBUF_TYPE_UADDR,
+	IO_REGBUF_TYPE_DMABUF,
+
+	__IO_REGBUF_TYPE_MAX,
+};
+
+struct io_uring_regbuf_desc {
+	__u32 type; /* enum io_uring_regbuf_type */
+	__u32 flags;
+	__u64 size;
+	__u64 uaddr;
+
+	__s32 dmabuf_fd;
+	__s32 target_fd;
+	__u64 __resv[6];
+};
+
 /* Skip updating fd indexes set to this value in the fd table */
 #define IORING_REGISTER_FILES_SKIP	(-2)
 
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 4c83a94b4bdc3..d1d748b5c5dca 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -109,7 +109,7 @@
 
 #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | IO_REQ_LINK_FLAGS | \
 				 REQ_F_REISSUE | REQ_F_POLLED | \
-				 IO_REQ_CLEAN_FLAGS)
+				 IO_REQ_CLEAN_FLAGS | REQ_F_DROP_DMABUF)
 
 #define IO_TCTX_REFS_CACHE_NR	(1U << 10)
 
@@ -1114,6 +1114,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,
 				io_queue_next(req);
 			if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS))
 				io_clean_op(req);
+			io_req_drop_dmabuf(req);
 		}
 		io_put_file(req);
 		io_req_put_rsrc_nodes(req);
diff --git a/io_uring/net.c b/io_uring/net.c
index a74d15f7b7d2b..439c99ad18841 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1471,7 +1471,7 @@ static int io_sg_from_iter(struct sk_buff *skb,
 		return zerocopy_fill_skb_from_iter(skb, from, length);
 
 	bi.bi_size = min(from->count, length);
-	bi.bi_bvec_done = from->iov_offset;
+	bi.bi_offset = from->iov_offset;
 	bi.bi_idx = 0;
 
 	while (bi.bi_size && frag < MAX_SKB_FRAGS) {
@@ -1490,7 +1490,7 @@ static int io_sg_from_iter(struct sk_buff *skb,
 	from->bvec += bi.bi_idx;
 	from->nr_segs -= bi.bi_idx;
 	from->count -= copied;
-	from->iov_offset = bi.bi_bvec_done;
+	from->iov_offset = bi.bi_offset;
 
 	skb->data_len += copied;
 	skb->len += copied;
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 8d0f2ee24e0c2..5074b1e4b0505 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -10,6 +10,7 @@
 #include <linux/compat.h>
 #include <linux/io_uring.h>
 #include <linux/io_uring/cmd.h>
+#include <linux/dma-buf-io.h>
 
 #include <uapi/linux/io_uring.h>
 
@@ -28,7 +29,7 @@ struct io_rsrc_update {
 };
 
 static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
-						   struct iovec *iov);
+						   struct io_uring_regbuf_desc *desc);
 
 static int hpage_acct_ref(struct io_ring_ctx *ctx, struct page *hpage,
 			  bool *acct_new)
@@ -81,6 +82,18 @@ static bool hpage_acct_unref(struct io_ring_ctx *ctx, struct page *hpage)
 
 #define IO_CACHED_BVECS_SEGS	32
 
+static void io_iov_to_regbuf_desc(const struct iovec *iov,
+				  struct io_uring_regbuf_desc *desc)
+{
+	*desc = (struct io_uring_regbuf_desc) {
+		.type = IO_REGBUF_TYPE_UADDR,
+		.uaddr = (u64)(uintptr_t)iov->iov_base,
+		.size = iov->iov_len,
+	};
+	if (!desc->uaddr)
+		desc->type = IO_REGBUF_TYPE_EMPTY;
+}
+
 int __io_account_mem(struct user_struct *user, unsigned long nr_pages)
 {
 	unsigned long page_limit, cur_pages, new_pages;
@@ -309,6 +322,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
 		return -ENXIO;
 	if (up->offset + nr_args > ctx->file_table.data.nr)
 		return -EINVAL;
+	if (up->flags)
+		return -EINVAL;
 
 	for (done = 0; done < nr_args; done++) {
 		u64 tag = 0;
@@ -368,9 +383,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
 				   struct io_uring_rsrc_update2 *up,
 				   unsigned int nr_args)
 {
+	bool extended = up->flags & IORING_RSRC_UPDATE_EXTENDED;
 	u64 __user *tags = u64_to_user_ptr(up->tags);
-	struct iovec fast_iov, *iov;
-	struct iovec __user *uvec;
 	u64 user_data = up->data;
 	__u32 done;
 	int i, err;
@@ -379,26 +393,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
 		return -ENXIO;
 	if (up->offset + nr_args > ctx->buf_table.nr)
 		return -EINVAL;
+	if (up->flags & ~IORING_RSRC_UPDATE_EXTENDED)
+		return -EINVAL;
 
 	for (done = 0; done < nr_args; done++) {
+		struct io_uring_regbuf_desc desc;
 		struct io_rsrc_node *node;
 		u64 tag = 0;
 
-		uvec = u64_to_user_ptr(user_data);
-		iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx));
-		if (IS_ERR(iov)) {
-			err = PTR_ERR(iov);
-			break;
-		}
 		if (tags && copy_from_user(&tag, &tags[done], sizeof(tag))) {
 			err = -EFAULT;
 			break;
 		}
-		node = io_sqe_buffer_register(ctx, iov);
+
+		if (extended) {
+			if (copy_from_user(&desc, u64_to_user_ptr(user_data),
+					   sizeof(desc))) {
+				err = -EFAULT;
+				break;
+			}
+			user_data += sizeof(desc);
+		} else {
+			struct iovec __user *uvec = u64_to_user_ptr(user_data);
+			struct iovec fast_iov, *iov;
+
+			if (io_is_compat(ctx))
+				user_data += sizeof(struct compat_iovec);
+			else
+				user_data += sizeof(struct iovec);
+
+			iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx));
+			if (IS_ERR(iov)) {
+				err = PTR_ERR(iov);
+				break;
+			}
+			io_iov_to_regbuf_desc(iov, &desc);
+		}
+
+		node = io_sqe_buffer_register(ctx, &desc);
 		if (IS_ERR(node)) {
 			err = PTR_ERR(node);
 			break;
 		}
+
 		if (tag) {
 			if (!node) {
 				err = -EINVAL;
@@ -409,10 +446,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
 		i = array_index_nospec(up->offset + done, ctx->buf_table.nr);
 		io_reset_rsrc_node(ctx, &ctx->buf_table, i);
 		ctx->buf_table.nodes[i] = node;
-		if (io_is_compat(ctx))
-			user_data += sizeof(struct compat_iovec);
-		else
-			user_data += sizeof(struct iovec);
 	}
 	return done ? done : err;
 }
@@ -447,7 +480,7 @@ int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg,
 	memset(&up, 0, sizeof(up));
 	if (copy_from_user(&up, arg, sizeof(struct io_uring_rsrc_update)))
 		return -EFAULT;
-	if (up.resv || up.resv2)
+	if (up.resv2)
 		return -EINVAL;
 	return __io_register_rsrc_update(ctx, IORING_RSRC_FILE, &up, nr_args);
 }
@@ -461,7 +494,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,
 		return -EINVAL;
 	if (copy_from_user(&up, arg, sizeof(up)))
 		return -EFAULT;
-	if (!up.nr || up.resv || up.resv2)
+	if (!up.nr || up.resv2)
 		return -EINVAL;
 	return __io_register_rsrc_update(ctx, type, &up, up.nr);
 }
@@ -561,12 +594,9 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)
 	struct io_uring_rsrc_update2 up2;
 	int ret;
 
+	memset(&up2, 0, sizeof(up2));
 	up2.offset = up->offset;
 	up2.data = up->arg;
-	up2.nr = 0;
-	up2.tags = 0;
-	up2.resv = 0;
-	up2.resv2 = 0;
 
 	if (up->offset == IORING_FILE_INDEX_ALLOC) {
 		ret = io_files_update_with_index_alloc(req, issue_flags);
@@ -852,27 +882,126 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
 	return true;
 }
 
+struct io_regbuf_dma {
+	struct dma_buf_io_ctx		ctx;
+	struct file			*target_file;
+};
+
+static void io_release_reg_dmabuf(void *priv)
+{
+	struct io_regbuf_dma *db = priv;
+
+	fput(db->target_file);
+	dma_buf_io_ctx_release(&db->ctx);
+}
+
+static struct io_rsrc_node *io_register_dmabuf(struct io_ring_ctx *ctx,
+						struct io_uring_regbuf_desc *desc)
+{
+	struct io_rsrc_node *node = NULL;
+	struct io_mapped_ubuf *imu = NULL;
+	struct io_regbuf_dma *regbuf = NULL;
+	struct file *target_file = NULL;
+	struct dma_buf *dmabuf = NULL;
+	int ret;
+
+	if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))
+		return ERR_PTR(-EOPNOTSUPP);
+	if (ctx->flags & IORING_SETUP_IOPOLL)
+		return ERR_PTR(-EOPNOTSUPP);
+	if (desc->uaddr || desc->size)
+		return ERR_PTR(-EINVAL);
+
+	ret = -ENOMEM;
+	node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
+	if (!node)
+		return ERR_PTR(-ENOMEM);
+	imu = io_alloc_imu(ctx, 0);
+	if (!imu)
+		goto err;
+	regbuf = kzalloc(sizeof(*regbuf), GFP_KERNEL);
+	if (!regbuf)
+		goto err;
+
+	ret = -EBADF;
+	target_file = fget(desc->target_fd);
+	if (!target_file)
+		goto err;
+
+	dmabuf = dma_buf_get(desc->dmabuf_fd);
+	if (IS_ERR(dmabuf)) {
+		ret = PTR_ERR(dmabuf);
+		dmabuf = NULL;
+		goto err;
+	}
+	if (dmabuf->size > SZ_1G) {
+		ret = -EINVAL;
+		goto err;
+	}
+
+	ret = dma_buf_io_ctx_create(target_file, &regbuf->ctx, dmabuf,
+				    DMA_BIDIRECTIONAL);
+	if (ret)
+		goto err;
+
+	regbuf->target_file = target_file;
+	imu->nr_bvecs = 0;
+	imu->ubuf = 0;
+	imu->len = dmabuf->size;
+	imu->folio_shift = 0;
+	imu->release = io_release_reg_dmabuf;
+	imu->priv = regbuf;
+	imu->flags = IO_REGBUF_F_DMABUF;
+	imu->dir = IO_IMU_DEST | IO_IMU_SOURCE;
+	refcount_set(&imu->refs, 1);
+	node->buf = imu;
+	dma_buf_put(dmabuf);
+	return node;
+err:
+	kfree(regbuf);
+	if (imu)
+		io_free_imu(ctx, imu);
+	if (node)
+		io_cache_free(&ctx->node_cache, node);
+	if (target_file)
+		fput(target_file);
+	if (dmabuf)
+		dma_buf_put(dmabuf);
+	return ERR_PTR(ret);
+}
+
 static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
-						   struct iovec *iov)
+						   struct io_uring_regbuf_desc *desc)
 {
+	unsigned long uaddr = (unsigned long)desc->uaddr;
+	size_t size = desc->size;
 	struct io_mapped_ubuf *imu = NULL;
 	struct page **pages = NULL;
 	struct io_rsrc_node *node;
 	unsigned long off;
-	size_t size;
 	int ret, nr_pages, i;
 	struct io_imu_folio_data data;
 	bool coalesced = false;
 
-	if (!iov->iov_base) {
-		if (iov->iov_len)
+	if (desc->type >= __IO_REGBUF_TYPE_MAX)
+		return ERR_PTR(-EINVAL);
+	if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) || desc->flags)
+		return ERR_PTR(-EINVAL);
+
+	if (desc->type == IO_REGBUF_TYPE_DMABUF)
+		return io_register_dmabuf(ctx, desc);
+
+	if (desc->dmabuf_fd || desc->target_fd)
+		return ERR_PTR(-EINVAL);
+
+	if (desc->type == IO_REGBUF_TYPE_EMPTY) {
+		if (uaddr || size)
 			return ERR_PTR(-EFAULT);
 		/* remove the buffer without installing a new one */
 		return NULL;
 	}
 
-	ret = io_validate_user_buf_range((unsigned long)iov->iov_base,
-					 iov->iov_len);
+	ret = io_validate_user_buf_range(uaddr, size);
 	if (ret)
 		return ERR_PTR(ret);
 
@@ -881,8 +1010,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 		return ERR_PTR(-ENOMEM);
 
 	ret = -ENOMEM;
-	pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len,
-				&nr_pages);
+	pages = io_pin_pages(uaddr, size, &nr_pages);
 	if (IS_ERR(pages)) {
 		ret = PTR_ERR(pages);
 		pages = NULL;
@@ -904,10 +1032,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 	if (ret)
 		goto done;
 
-	size = iov->iov_len;
 	/* store original address for later verification */
-	imu->ubuf = (unsigned long) iov->iov_base;
-	imu->len = iov->iov_len;
+	imu->ubuf = uaddr;
+	imu->len = size;
 	imu->folio_shift = PAGE_SHIFT;
 	imu->release = io_release_ubuf;
 	imu->priv = imu;
@@ -917,7 +1044,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 		imu->folio_shift = data.folio_shift;
 	refcount_set(&imu->refs, 1);
 
-	off = (unsigned long)iov->iov_base & ~PAGE_MASK;
+	off = uaddr & ~PAGE_MASK;
 	if (coalesced)
 		off += data.first_folio_page_idx << PAGE_SHIFT;
 
@@ -969,6 +1096,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
 		memset(iov, 0, sizeof(*iov));
 
 	for (i = 0; i < nr_args; i++) {
+		struct io_uring_regbuf_desc desc;
 		struct io_rsrc_node *node;
 		u64 tag = 0;
 
@@ -992,7 +1120,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
 			}
 		}
 
-		node = io_sqe_buffer_register(ctx, iov);
+		io_iov_to_regbuf_desc(iov, &desc);
+		node = io_sqe_buffer_register(ctx, &desc);
 		if (IS_ERR(node)) {
 			ret = PTR_ERR(node);
 			break;
@@ -1136,9 +1265,64 @@ static int io_import_kbuf(int ddir, struct iov_iter *iter,
 	return 0;
 }
 
-static int io_import_fixed(int ddir, struct iov_iter *iter,
+void io_drop_dmabuf_node(struct io_kiocb *req)
+{
+	struct io_mapped_ubuf *imu;
+
+	if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))
+		return;
+	if (WARN_ON_ONCE(req->buf_node->type != IORING_RSRC_BUFFER))
+		return;
+	imu = req->buf_node->buf;
+	if (WARN_ON_ONCE(!(imu->flags & IO_REGBUF_F_DMABUF)))
+		return;
+	dma_buf_io_map_drop(req->dmabuf_map);
+	req->flags &= ~REQ_F_DROP_DMABUF;
+}
+
+static int io_import_dmabuf(struct io_kiocb *req,
+			   int ddir, struct iov_iter *iter,
+			   struct io_mapped_ubuf *imu,
+			   size_t len, size_t offset,
+			   unsigned issue_flags)
+{
+	struct io_regbuf_dma *db = imu->priv;
+	struct dma_buf_io_map *map;
+
+	if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))
+		return -EOPNOTSUPP;
+	if (!len)
+		return -EFAULT;
+	if (req->file != db->target_file)
+		return -EBADF;
+
+	if (req->flags & REQ_F_DROP_DMABUF) {
+		map = req->dmabuf_map;
+		goto init_iter;
+	}
+
+	map = dma_buf_io_get_map(&db->ctx);
+	if (unlikely(!map)) {
+		if (!(issue_flags & IO_URING_F_IOWQ))
+			return -EAGAIN;
+		map = dma_buf_io_create_map(&db->ctx);
+		if (IS_ERR(map))
+			return PTR_ERR(map);
+	}
+
+	req->dmabuf_map = map;
+	req->flags |= REQ_F_DROP_DMABUF;
+init_iter:
+	iov_iter_dmabuf_map(iter, ddir, map, offset, len);
+	return 0;
+}
+
+static int io_import_fixed(struct io_kiocb *req,
+			   int ddir, struct iov_iter *iter,
 			   struct io_mapped_ubuf *imu,
-			   u64 buf_addr, size_t len)
+			   u64 buf_addr, size_t len,
+			   unsigned issue_flags,
+			   unsigned import_flags)
 {
 	const struct bio_vec *bvec;
 	size_t folio_mask;
@@ -1158,6 +1342,12 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 
 	offset = buf_addr - imu->ubuf;
 
+	if (imu->flags & IO_REGBUF_F_DMABUF) {
+		if (!(import_flags & IO_REGBUF_IMPORT_ALLOW_DMABUF))
+			return -EFAULT;
+		return io_import_dmabuf(req, ddir, iter, imu, len, offset,
+					issue_flags);
+	}
 	if (imu->flags & IO_REGBUF_F_KBUF)
 		return io_import_kbuf(ddir, iter, imu, len, offset);
 
@@ -1211,16 +1401,17 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
 	return NULL;
 }
 
-int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
 			u64 buf_addr, size_t len, int ddir,
-			unsigned issue_flags)
+			unsigned issue_flags, unsigned import_flags)
 {
 	struct io_rsrc_node *node;
 
 	node = io_find_buf_node(req, issue_flags);
 	if (!node)
 		return -EFAULT;
-	return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
+	return io_import_fixed(req, ddir, iter, node->buf, buf_addr, len,
+				issue_flags, import_flags);
 }
 
 static int io_buffer_acct_cloned_hpages(struct io_ring_ctx *ctx,
@@ -1347,6 +1538,11 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx
 		if (!src_node) {
 			dst_node = NULL;
 		} else {
+			if (src_node->buf->flags & IO_REGBUF_F_UNCLONEABLE) {
+				io_rsrc_data_free(ctx, &data);
+				return -ENOMEM;
+			}
+
 			dst_node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
 			if (!dst_node) {
 				io_rsrc_data_free(ctx, &data);
@@ -1617,9 +1813,9 @@ static int io_kern_bvec_size(struct iovec *iov, unsigned nr_iovs,
 	return 0;
 }
 
-int io_import_reg_vec(int ddir, struct iov_iter *iter,
+int __io_import_reg_vec(int ddir, struct iov_iter *iter,
 			struct io_kiocb *req, struct iou_vec *vec,
-			unsigned nr_iovs, unsigned issue_flags)
+			unsigned nr_iovs, unsigned issue_flags, unsigned import_flags)
 {
 	struct io_rsrc_node *node;
 	struct io_mapped_ubuf *imu;
@@ -1637,7 +1833,9 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter,
 	iovec_off = vec->nr - nr_iovs;
 	iov = vec->iovec + iovec_off;
 
-	if (imu->flags & IO_REGBUF_F_KBUF) {
+	if (imu->flags & IO_REGBUF_F_DMABUF) {
+		return -EOPNOTSUPP;
+	} else if (imu->flags & IO_REGBUF_F_KBUF) {
 		int ret = io_kern_bvec_size(iov, nr_iovs, imu, &nr_segs);
 
 		if (unlikely(ret))
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 98ae8ef51009d..3cb9b57f159b6 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -29,7 +29,13 @@ enum {
 };
 
 enum {
-	IO_REGBUF_F_KBUF		= 1,
+	IO_REGBUF_F_KBUF		= 1 << 0,
+	IO_REGBUF_F_UNCLONEABLE		= 1 << 1,
+	IO_REGBUF_F_DMABUF		= 1 << 3,
+};
+
+enum {
+	IO_REGBUF_IMPORT_ALLOW_DMABUF		= 1 << 1,
 };
 
 struct io_mapped_ubuf {
@@ -64,12 +70,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr);
 
 struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
 				      unsigned issue_flags);
-int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
 			u64 buf_addr, size_t len, int ddir,
-			unsigned issue_flags);
-int io_import_reg_vec(int ddir, struct iov_iter *iter,
+			unsigned issue_flags, unsigned import_flags);
+int __io_import_reg_vec(int ddir, struct iov_iter *iter,
 			struct io_kiocb *req, struct iou_vec *vec,
-			unsigned nr_iovs, unsigned issue_flags);
+			unsigned nr_iovs, unsigned issue_flags,
+			unsigned import_flags);
+
+static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
+				    u64 buf_addr, size_t len, int ddir,
+				    unsigned issue_flags)
+{
+	return __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0);
+}
+
+static inline int io_import_reg_vec(int ddir, struct iov_iter *iter,
+				    struct io_kiocb *req, struct iou_vec *vec,
+				    unsigned nr_iovs, unsigned issue_flags)
+{
+	return __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0);
+}
+
 int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv,
 			const struct iovec __user *uvec, size_t uvec_segs);
 
@@ -156,4 +178,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv)
 		io_vec_free(iv);
 }
 
+void io_drop_dmabuf_node(struct io_kiocb *req);
+
+static inline void io_req_drop_dmabuf(struct io_kiocb *req)
+{
+	if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER))
+		return;
+	if (!(req->flags & REQ_F_DROP_DMABUF))
+		return;
+	if (WARN_ON_ONCE(!(req->flags & REQ_F_BUF_NODE)))
+		return;
+	io_drop_dmabuf_node(req);
+}
+
 #endif
diff --git a/io_uring/rw.c b/io_uring/rw.c
index 95038cfda6153..11ff7b5a11647 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb *req, unsigned int issue_flags,
 	if (io->bytes_done)
 		return 0;
 
-	ret = io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir,
-				issue_flags);
+	ret = __io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir,
+				  issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF);
 	iov_iter_save_state(&io->iter, &io->iter_state);
 	return ret;
 }
@@ -591,6 +591,8 @@ static void io_complete_rw(struct kiocb *kiocb, long res)
 	struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);
 	struct io_kiocb *req = cmd_to_io_kiocb(rw);
 
+	io_req_drop_dmabuf(req);
+
 	__io_complete_rw_common(req, res);
 	io_req_set_res(req, io_fixup_rw_res(req, res), 0);
 	req->io_task_work.func = io_req_rw_complete;
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 34a52e9ba9e1b..d6f5d7e180b54 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
 {
 	if (unlikely(i->count < size))
 		size = i->count;
-	if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i))) {
+	if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i)) ||
+	    unlikely(iov_iter_is_dmabuf_map(i))) {
 		i->iov_offset += size;
 		i->count -= size;
 	} else if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) {
@@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)
 		return;
 	}
 	unroll -= i->iov_offset;
-	if (iov_iter_is_xarray(i) || iter_is_ubuf(i)) {
+	if (iov_iter_is_xarray(i) || iter_is_ubuf(i) ||
+	    iov_iter_is_dmabuf_map(i)) {
 		BUG(); /* We should never go beyond the start of the specified
 			* range since we might then be straying into pages that
 			* aren't pinned.
@@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction,
 }
 EXPORT_SYMBOL(iov_iter_xarray);
 
+void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction,
+			 struct dma_buf_io_map *map,
+			 loff_t off, size_t count)
+{
+	WARN_ON(direction & ~(READ | WRITE));
+	*i = (struct iov_iter){
+		.iter_type = ITER_DMABUF_MAP,
+		.data_source = direction,
+		.dmabuf_map = map,
+		.count = count,
+		.iov_offset = off,
+	};
+}
+
 /**
  * iov_iter_discard - Initialise an I/O iterator that discards data
  * @i: The iterator to initialise.
@@ -841,7 +857,7 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)
 
 unsigned long iov_iter_alignment(const struct iov_iter *i)
 {
-	if (likely(iter_is_ubuf(i))) {
+	if (likely(iter_is_ubuf(i)) || iov_iter_is_dmabuf_map(i)) {
 		size_t size = i->count;
 		if (size)
 			return ((unsigned long)i->ubuf + i->iov_offset) | size;
@@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)
 	size_t size = i->count;
 	unsigned k;
 
-	if (iter_is_ubuf(i))
+	if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i))
 		return 0;
 
 	if (WARN_ON(!iter_is_iovec(i)))
@@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf);
 void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)
 {
 	if (WARN_ON_ONCE(!iov_iter_is_bvec(i) && !iter_is_iovec(i) &&
-			 !iter_is_ubuf(i)) && !iov_iter_is_kvec(i))
+			 !iter_is_ubuf(i) && !iov_iter_is_kvec(i) &&
+			 !iov_iter_is_dmabuf_map(i)))
 		return;
 	i->iov_offset = state->iov_offset;
 	i->count = state->count;
-	if (iter_is_ubuf(i))
+	if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i))
 		return;
 	/*
 	 * For the *vec iters, nr_segs + iov is constant - if we increment
@@ -1634,7 +1651,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i,
 	}
 	bi.bi_idx = 0;
 	bi.bi_size = maxsize;
-	bi.bi_bvec_done = skip;
+	bi.bi_offset = skip;
 
 	maxpages = want_pages_array(pages, maxsize, skip, maxpages);
 	if (!maxpages)
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 34b3097b4c7b9..9c1b6cf8c36fe 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -762,7 +762,7 @@ static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor *cursor,
 	if (!cursor->resid)
 		return false;   /* no more data */
 
-	if (!bytes || (it->iter.bi_size && it->iter.bi_bvec_done &&
+	if (!bytes || (it->iter.bi_size && it->iter.bi_offset &&
 		       page == bio_iter_page(it->bio, it->iter)))
 		return false;	/* more bytes to process in this segment */
 
@@ -817,7 +817,7 @@ static bool ceph_msg_data_bvecs_advance(struct ceph_msg_data_cursor *cursor,
 	if (!cursor->resid)
 		return false;   /* no more data */
 
-	if (!bytes || (cursor->bvec_iter.bi_bvec_done &&
+	if (!bytes || (cursor->bvec_iter.bi_offset &&
 		       page == bvec_iter_page(bvecs, cursor->bvec_iter)))
 		return false;	/* more bytes to process in this segment */

3/2

2026/08/02 14:37

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 36098
							output: 28
							thoughts: 1521

Thoughts:

**Fuzzing Target Identification: DMA-BUF Integration**

Okay, let's break this down. My initial assessment is that this patch series *absolutely* warrants fuzzing. We're looking at a fundamental shift here, integrating `dma-buf` into the core I/O paths, specifically impacting `io_uring` and NVMe. This isn't just a minor tweak; it's a significant functional change within the block layer, NVMe driver, and io_uring components. My expertise tells me these are high-value targets for uncovering potential vulnerabilities.

Let's look at the specific changes. The patch modifies `bio`, `blk-mq`, `nvme`, `io_uring`, and `iov_iter` to support `dma-buf` mappings. We're seeing changes like `bio_iov_iter_set` replacing `bio_iov_bvec_set`, with the crucial addition of `iov_iter_is_dmabuf_map` handling. We've got a new file, `dma-buf-io.c`, introducing functions like `dma_buf_io_ctx_create` and `dma_buf_io_create_map`, etc. The NVMe driver gets modifications for `nvme_rq_setup_dmabuf`, `nvme_pci_init_dma_buf_io_ctx`, etc., which are important. And in `io_uring/rsrc.c`, we see `io_register_dmabuf`, `io_import_dmabuf`. And, of course, the introduction of `ITER_DMABUF_MAP` in `iov_iter`.

We're dealing with a new I/O mapping infrastructure based on `dma-buf`, and its integration with existing mechanisms. This is prime fuzzing territory.

I've identified the configurations that are key: `CONFIG_DMA_SHARED_BUFFER` is essential. Although it seems to be enabled by default in many configs, we'll need to specify it explicitly for a complete fuzzing coverage. `CONFIG_NVME_CORE`, `CONFIG_BLK_DEV_NVME`, and `CONFIG_IO_URING` are the other important ones.

Now, let's identify the specific functions. I need to focus on the key symbols to fuzz.

Based on the prompt, it is better to fuzz non-static functions over internal static helper functions. This suggests that we should aim for API entrypoints. Therefore, the list needs refinement.

So I am looking at:

- `dma_buf_io_ctx_create`
- `dma_buf_io_ctx_release`
- `dma_buf_io_create_map`
- `dma_buf_io_invalidate_mappings`
- `dma_buf_io_init_map`
- `bio_iov_iter_set`
- `__io_import_reg_buf`
- `io_drop_dmabuf_node`
- `iov_iter_dmabuf_map`
- `bio_split_io_at` (since `bio_split_io_at_dmabuf` is static inline)
- `nvme_map_data` (calls `nvme_rq_setup_dmabuf`)
- `nvme_unmap_data` (calls `nvme_rq_clean_dmabuf_map`)
- `nvme_pci_init_dma_buf_io_ctx` (static, but maybe `nvme_dev_map` or something? It's a callback in `nvme_pci_ctrl_ops`).

So I think a good initial focus would be:

- `dma_buf_io_ctx_create`
- `dma_buf_io_ctx_release`
- `dma_buf_io_create_map`
- `dma_buf_io_invalidate_mappings`
- `dma_buf_io_init_map`
- `bio_iov_iter_set`
- `__io_import_reg_buf`
- `io_drop_dmabuf_node`
- `iov_iter_dmabuf_map`
- `bio_split_io_at`
- `nvme_map_data`
- `nvme_unmap_data`
- `nvme_pci_init_dma_buf_io_ctx`

The reasoning here is straightforward. This patch series enables `dma-buf` within key I/O paths. It adds new functions to create and manage `dma-buf` mappings and modifies core functions to handle `d

4/2

2026/08/02 14:37

tool

grepper

Args:

{
  "Expression": "DMA_SHARED_BUFFER",
  "PathPrefix": "drivers/Makefile"
}

Results:

{
  "Output": "drivers/Makefile=78=obj-y\t\t\t\t+= dax/\ndrivers/Makefile:79:obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/\ndrivers/Makefile-80-obj-$(CONFIG_NUBUS)\t\t+= nubus/\n"
}

5/2

2026/08/02 14:37

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 38747
							output: 27
							thoughts: 22

Thoughts:

Okay, here's my quick mental rundown:

**Confirming DMA Buffer and Moving on to IO_URING**

Alright, yep, that `DMA_SHARED_BUFFER` variable—I'm pretty confident that's the correct one we're working with here. Good. Now, let's pivot and take a look at how this all plays out with `IO_URING`. Time to shift my focus and start assessing the implications there.

6/2

2026/08/02 14:38

tool

grepper

Args:

{
  "Expression": "IO_URING",
  "PathPrefix": "init/Kconfig"
}

Results:

{
  "Output": "init/Kconfig=1910=config AIO\n--\ninit/Kconfig-1917-\ninit/Kconfig:1918:config IO_URING\ninit/Kconfig-1919-\tbool \"Enable IO uring support\" if EXPERT\n--\ninit/Kconfig=1927=config GCOV_PROFILE_URING\ninit/Kconfig-1928-\tbool \"Enable GCOV profiling on the io_uring subsystem\"\ninit/Kconfig:1929:\tdepends on IO_URING \u0026\u0026 GCOV_KERNEL\ninit/Kconfig-1930-\thelp\n--\ninit/Kconfig-1939-\ninit/Kconfig:1940:config IO_URING_MOCK_FILE\ninit/Kconfig-1941-\ttristate \"Enable io_uring mock files (Experimental)\" if EXPERT\ninit/Kconfig-1942-\tdefault n\ninit/Kconfig:1943:\tdepends on IO_URING\ninit/Kconfig-1944-\thelp\n"
}

7/2

2026/08/02 14:38

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 39155
							output: 35
							thoughts: 23

Thoughts:

Okay, here's my take on those thoughts, tailored for an expert audience:

**IO_URING and NVMe Considerations**

Yes, `IO_URING` is definitely the right direction to go here. The inherent benefits are clear and it's where we should focus our attention for this architecture. Now, I want to take a deeper look at `BLK_DEV_NVME`. That's where the real performance gains are going to be found. Let's see how well it's working with this new design.

8/2

2026/08/02 14:38

tool

grepper

Args:

{
  "Expression": "BLK_DEV_NVME",
  "PathPrefix": "drivers/nvme/host/Kconfig"
}

Results:

{
  "Output": "drivers/nvme/host/Kconfig=2=config NVME_CORE\n--\ndrivers/nvme/host/Kconfig-4-\ndrivers/nvme/host/Kconfig:5:config BLK_DEV_NVME\ndrivers/nvme/host/Kconfig-6-\ttristate \"NVM Express block device\"\n"
}

9/2

2026/08/02 14:38

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 39414
							output: 243
							thoughts: 409

Thoughts:

**Analyzing DMA-Buf Related Symbols for NVMe Driver Integration**

Okay, so we've confirmed `BLK_DEV_NVME` is the right path here. Now, let's take a look at the important symbols. I'm going through the list and double-checking that we're covering all the relevant functions related to `dma_buf` interaction.

First, I identified `dma_buf_io_ctx_create`, `dma_buf_io_ctx_release`, `dma_buf_io_create_map`, `dma_buf_io_invalidate_mappings`, `dma_buf_io_init_map`, `bio_iov_iter_set`, `__io_import_reg_buf`, `io_drop_dmabuf_node` and `iov_iter_dmabuf_map` as initial candidates.

I briefly considered `nvme_map_data`, and `nvme_unmap_data`, but realized those are static, as well as `nvme_pci_init_dma_buf_io_ctx` which is also static. Static functions are likely not directly used from outside the driver, so we're probably fine not including them directly, even if the function itself is a callback.

Alright, after reviewing the list again, I'm confident with these candidates. The non-static functions seem to capture the core of the `dma_buf` interaction points for the NVMe driver.

Here's the final list:
*   `dma_buf_io_ctx_create`
*   `dma_buf_io_ctx_release`
*   `dma_buf_io_create_map`
*   `dma_buf_io_invalidate_mappings`
*   `dma_buf_io_init_map`
*   `bio_iov_iter_set`
*   `__io_import_reg_buf`
*   `io_drop_dmabuf_node`
*   `iov_iter_dmabuf_map`

I believe this is a solid selection. Let's call `set-results` and move forward.

10/2

2026/08/02 14:38

tool

set-results

Args:

{
  "EnableConfigs": [
    "DMA_SHARED_BUFFER",
    "IO_URING",
    "BLK_DEV_NVME"
  ],
  "FocusSymbols": [
    "dma_buf_io_ctx_create",
    "dma_buf_io_ctx_release",
    "dma_buf_io_create_map",
    "dma_buf_io_invalidate_mappings",
    "dma_buf_io_init_map",
    "bio_iov_iter_set",
    "__io_import_reg_buf",
    "io_drop_dmabuf_node",
    "iov_iter_dmabuf_map"
  ],
  "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.",
  "WorthFuzzing": true
}

Results:

{
  "EnableConfigs": [
    "DMA_SHARED_BUFFER",
    "IO_URING",
    "BLK_DEV_NVME"
  ],
  "FocusSymbols": [
    "dma_buf_io_ctx_create",
    "dma_buf_io_ctx_release",
    "dma_buf_io_create_map",
    "dma_buf_io_invalidate_mappings",
    "dma_buf_io_init_map",
    "bio_iov_iter_set",
    "__io_import_reg_buf",
    "io_drop_dmabuf_node",
    "iov_iter_dmabuf_map"
  ],
  "Reasoning": "The patch introduces dma-buf support in the block layer, NVMe PCI driver, and io_uring. It adds a new dma-buf I/O mapping infrastructure and modifies core bio and iov_iter logic to handle dma-buf maps. This is a significant functional change that exposes new reachable code paths via io_uring and NVMe, which should be fuzzed.",
  "WorthFuzzing": true
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)