Commit 91fb2b6052f7 ("nvme-pci: convert to using dma_map_sgtable()") deliberately mapped unsupported P2PDMA transfers to BLK_STS_TARGET, matching dma_map_sgtable()'s -EREMOTEIO: "When this happens, return BLK_STS_TARGET so the request isn't retried." The conversion to blk_rq_dma_map silently changed the status to BLK_STS_INVAL, which regresses two consumers: - md/raid1 and raid10 ignore BLK_STS_INVAL leg failures (commit f7b24c7b41f2 ("md/raid1,raid10: don't fail devices for invalid IO errors"), where it means a request-shaped error that fails identically on every member). Since commit 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") P2PDMA bios reach md arrays: a peer-memory write to a member the peer cannot reach is counted as written, the master bio reports success, mirrors silently diverge, and on a topology where no member is reachable the write reports success with zero copies on stable storage. - the failure's classification flips: for direct NVMe consumers (nvme advertises BLK_FEAT_PCI_P2PDMA) the mapping failure has surfaced as EINVAL instead of the documented -EREMOTEIO since v6.17, and blk_path_error(BLK_STS_INVAL) is true, so a stacking consumer would treat it as a retryable path error (dm-mpath is the only in-tree blk_path_error() caller; P2P bios cannot currently reach it, but the classification is wrong on its face). Restore BLK_STS_TARGET. md then routes an unreachable leg through its per-device error handling (badblocks, mirror retry for reads); a later patch in this series teaches raid1/raid10 to handle mapping failures without the retry storms that machinery was built around. Fixes: 858299dc6160 ("block: add scatterlist-less DMA mapping helpers") Fixes: 7ce3c1dd78fc ("nvme-pci: convert the data mapping to blk_rq_dma_map") Cc: stable@vger.kernel.org # v6.17 Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan --- block/blk-mq-dma.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c index bfdb9ed70741..2eed06bfe791 100644 --- a/block/blk-mq-dma.c +++ b/block/blk-mq-dma.c @@ -190,7 +190,15 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev, case PCI_P2PDMA_MAP_NONE: break; default: - iter->status = BLK_STS_INVAL; + /* + * P2P transfers that the mapping layer cannot support + * report BLK_STS_TARGET, matching dma_map_sgtable()'s + * -EREMOTEIO and the pre-blk_rq_dma_map nvme behavior: + * the failure is a property of this device pairing, so + * it must not be retried on another path (blk_path_error) + * nor be mistaken for an invalid request. + */ + iter->status = BLK_STS_TARGET; return false; } -- 2.43.0 md_submit_bio() unconditionally strips REQ_NOMERGE before passing the bio to the personality, an optimization from commit 9c573de3283a ("MD: make bio mergeable"): a bio that md has split may become mergeable again below md. For PCI P2PDMA bios the flag is load-bearing, not a hint. The block layer sets REQ_NOMERGE on P2PDMA bios (__bio_add_page(), and the extraction path of bio_iov_iter_get_pages()) because the DMA mapping type of a request is resolved once, from its first segment (blk_dma_map_iter_start()), and request-level merging is prevented only by REQ_NOMERGE. Stripping it allows the member queue to merge a P2PDMA bio with a bio carrying pages of a different pgmap, or host memory, mapping the merged segments with the wrong bus address: silent data corruption on the member. Set the flag for P2PDMA bios instead of merely preserving it. No in-tree path currently submits P2PDMA pages through the bvec-iter path (bio_iov_bvec_set()), which skips the flagging -- but nothing structural prevents one, so setting rather than preserving hardens md against that gap at the cost of one branch. Everything else keeps the original optimization of clearing the flag. This covers every personality that advertises BLK_FEAT_PCI_P2PDMA (raid0, raid1, raid10), which is why the fix lives in the shared md_submit_bio() path. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan --- drivers/md/md.c | 14 ++++++++++++-- drivers/md/md.h | 18 ++++++++++++++++++ 2 files changed, 30 insertions(+), 2 deletions(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index d1465bcd86c8..3ae4fd4ef381 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -451,8 +451,18 @@ static void md_submit_bio(struct bio *bio) return; } - /* bio could be mergeable after passing to underlayer */ - bio->bi_opf &= ~REQ_NOMERGE; + /* + * A bio md split could be mergeable again below md, but for P2PDMA + * bios REQ_NOMERGE is load-bearing: the DMA mapping type of a + * request is resolved once, from its first segment, so requests + * must stay single-provider (see __bio_add_page()). Set the flag + * rather than merely preserve it -- bios built through the + * bvec-iter path arrive without it. + */ + if (md_bio_is_p2pdma(bio)) + bio->bi_opf |= REQ_NOMERGE; + else + bio->bi_opf &= ~REQ_NOMERGE; md_handle_request(mddev, bio); } diff --git a/drivers/md/md.h b/drivers/md/md.h index d8daf0f75cbb..140e2b3670d8 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -11,8 +11,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -22,6 +24,22 @@ #include #define MaxSector (~(sector_t)0) + +/* + * Check if the bio carries PCI P2PDMA (peer device memory) pages. Read + * bi_io_vec directly rather than using bio_first_bvec_all(), which WARNs + * on cloned bios: md routinely handles split clones, which have + * bi_vcnt == 0 but a valid bi_io_vec shared with the parent. P2PDMA and + * host pages must not be mixed within one bio, so the first bvec is + * representative. Only valid before the bio's iterator is consumed: + * bio_has_data() is false at completion time. + */ +static inline bool md_bio_is_p2pdma(struct bio *bio) +{ + return bio_has_data(bio) && bio->bi_io_vec && + is_pci_p2pdma_page(bio->bi_io_vec->bv_page); +} + /* * Number of guaranteed raid bios in case of extreme VM load: */ -- 2.43.0 A write-behind write is acknowledged to the caller while its copy to the write-mostly member is still in flight. wait_for_serialization() exists to order later overlapping writes behind it, but the non-behind clone path only takes it under serialize_policy. A write to a write-behind array skips the behind path whenever the behind queue is full, a reader is waiting on behind completion, or the behind bio allocation fails (raid1_start_write_behind()) -- and then races the in-flight behind write on the write-mostly member without any ordering: if the older behind data lands last, the member keeps stale data for sectors whose newer write the caller has already seen acknowledged. Gate the non-behind serialization on CollisionCheck alone. The flag marks exactly the rdevs that own a serial tree: every rdev present when serialize_policy is enabled, and write-mostly members when write-behind arms serialization, which is the case the current test misses. Take the matching remove_serial() at completion under the same condition. Discards take the same non-behind path and are now ordered as well. For an rdev hot-added while serialize_policy is already enabled, mddev_create_serial_pool() never builds a serial tree, so the old MD_SERIALIZE_POLICY gate sent it into wait_for_serialization() with rdev->serial == NULL -- a latent oops that the per-rdev gate avoids by construction. REQ_NOWAIT writes can wait here when they overlap an in-flight behind write, as they already do on the behind path and under serialize_policy. Fixes: d0d2d8ba0494 ("md/raid1: introduce wait_for_serialization") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index afe2ca96ad8c..8172df882f26 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -564,7 +564,7 @@ static void raid1_end_write_request(struct bio *bio) call_bio_endio(r1_bio); } } - } else if (test_bit(MD_SERIALIZE_POLICY, &rdev->mddev->flags)) + } else if (test_bit(CollisionCheck, &rdev->flags)) remove_serial(rdev, lo, hi); if (r1_bio->bios[mirror] == NULL) rdev_dec_pending(rdev, conf->mddev); @@ -1677,7 +1677,14 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio, mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set); - if (test_bit(MD_SERIALIZE_POLICY, &mddev->flags)) + /* + * Order against in-flight write-behind I/O: a + * behind write is acked early, and an unordered + * overwrite could land first, leaving its stale + * data on the member last. CollisionCheck marks + * every rdev that owns a serial tree. + */ + if (test_bit(CollisionCheck, &rdev->flags)) wait_for_serialization(rdev, r1_bio); } -- 2.43.0 alloc_behind_master_bio() copies the bio's data with bio_copy_data(), a CPU copy. P2PDMA pages are peer device (BAR) memory; generic code must not assume CPU load/store access to them is safe or fast on every architecture, and bouncing peer memory through the CPU defeats the point of a peer-to-peer transfer. Skip write-behind for P2PDMA bios: they are written directly to all members, including write-mostly ones. Ordering against write-behind I/O in flight to overlapping sectors is preserved: the non-behind clone path serializes on CollisionCheck rdevs (see the preceding fix), which covers these bios like any other write that bypasses write-behind. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 8172df882f26..914fb86452c0 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1523,6 +1523,7 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio, bool write_behind = false; bool nowait = bio->bi_opf & REQ_NOWAIT; bool is_discard = op_is_discard(bio->bi_opf); + bool is_p2pdma = md_bio_is_p2pdma(bio); sector_t sector = bio->bi_iter.bi_sector; if (mddev_is_clustered(mddev) && @@ -1575,9 +1576,12 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio, /* * The write-behind io is only attempted on drives marked as * write-mostly, which means we could allocate write behind - * bio later. + * bio later. P2PDMA bios are excluded: write-behind copies + * the data with bio_copy_data(), a CPU copy that cannot be + * assumed safe or fast on P2PDMA (device BAR) pages. */ - if (!is_discard && rdev && test_bit(WriteMostly, &rdev->flags)) + if (!is_discard && !is_p2pdma && rdev && + test_bit(WriteMostly, &rdev->flags)) write_behind = true; r1_bio->bios[i] = NULL; -- 2.43.0 narrow_write_error() re-issues a failed write in badblock-granularity chunks, cloning from the master bio and resetting bi_opf to a bare REQ_OP_WRITE. For a P2PDMA bio that reset drops REQ_NOMERGE, which is the only request-level protection against the member queue merging P2PDMA segments across pgmaps or with host memory (see the preceding md_submit_bio() fix): the retry path would quietly reopen the hole the submission path closes. Restore the flag on P2PDMA retry clones. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 3 +++ drivers/md/raid10.c | 3 +++ 2 files changed, 6 insertions(+) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 914fb86452c0..f562b6bd438b 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2573,6 +2573,9 @@ static void narrow_write_error(struct r1bio *r1_bio, int i) } wbio->bi_opf = REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |= REQ_NOMERGE; wbio->bi_iter.bi_sector = r1_bio->sector; wbio->bi_iter.bi_size = r1_bio->sectors << 9; diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 0a3cfdd3f5df..f7ef903a3d4e 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2831,6 +2831,9 @@ static void narrow_write_error(struct r10bio *r10_bio, int i) wbio->bi_iter.bi_sector = wsector + choose_data_offset(r10_bio, rdev); wbio->bi_opf = REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |= REQ_NOMERGE; if (submit_bio_wait(wbio) && !rdev_set_badblocks(rdev, wsector, sectors, 0)) { -- 2.43.0 Since commit 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") raid1 and raid10 arrays accept P2PDMA (peer device memory) bios, and a member that cannot DMA-map the peer's pages fails its leg bio with BLK_STS_TARGET (from blk_dma_map_iter_start()). That mapping failure is a property of the peer-device/member pairing, not of the medium: re-submitting the same peer pages to the same member cannot succeed, and there is nothing on the disk to repair. Routing it through the stock error machinery misfires on every path: - narrow_write_error() re-issues the failed write in badblock-sized chunks, serializing hundreds of guaranteed-to-fail synchronous bios through raid1d/raid10d per failed write (256 for a 1 MiB write on 4K logical blocks, 2048 on 512e). - the write-error handler sets WantReplacement, so a hot spare is pulled in, rebuilt onto, and the healthy member is then evicted by spare_active() -- and if the spare is also unreachable from the peer, the cycle consumes the next spare. - fix_read_error() probes members with kernel pages, starting with the failing member itself. That probe succeeds (the member is healthy for host memory), so the routine rewrites nothing, records nothing, and logs nothing -- but each invocation costs a full freeze_array() quiesce and a tick of the read-error budget. The budget exists to evict members whose medium keeps producing corrected errors; its hourly decay is sized for sporadic medium errors, while mapping failures are deterministic and arrive at I/O rate. On an asymmetric PCIe topology a mixed host/P2P read workload charges 20 errors to the unreachable leg within a second and kicks a perfectly healthy member. - On FailFast members both paths short-circuit into md_error(), so a single unroutable peer-memory I/O evicts a healthy mirror outright. Track P2PDMA masters with a new r1bio/r10bio state bit, set at submission where bio_has_data() is still meaningful; a leg's mapping failure is then "BLK_STS_TARGET and the master carries P2PDMA pages", cheap to test on every completion path. Handle it explicitly: - Writes: probe the whole range with one retry and record one bad range if it also fails. BLK_STS_TARGET is also produced for device conditions that narrow_write_error()'s retry loop recovers from (e.g. NVME_SC_NS_NOT_READY and NVME_SC_CMD_INTERRUPTED via nvme_error_status()), and md cannot tell the two apart from bi_status alone, so it must not skip the retry outright. A single whole-range probe observes the outcome instead of predicting it. WriteErrorSeen is still set (it gates the write-path badblocks consult), but WantReplacement is not: replacing a member cannot fix a topology property, and spending a spare on it destroys redundancy management for real failures. - Reads: mark the leg IO_BLOCKED so read_balance() picks another mirror, and skip the freeze/fix cycle and the budget charge. The same TARGET ambiguity exists here, but a redirected read leaves nothing to fence, and fix_read_error()'s host-page probe fails the same way under e.g. NVME_SC_NS_NOT_READY -- charging the budget would evict a healthy member for a firmware activation window. A genuinely failing member is still evicted via host reads, writes and BLK_STS_MEDIUM errors. If no mirror can serve the read the master bio fails with EIO as before (md completes masters by Uptodate state, as for any failed mirror I/O), now without kicking healthy members on the way. - FailFast: a mapping failure is rejected at map time and never reaches the wire, so it is no evidence of device unreliability; don't let it trigger the FailFast md_error() -- the same reasoning commit f7b24c7b41f2 ("md/raid1,raid10: don't fail devices for invalid IO errors") applied to BLK_STS_INVAL. raid10 replacement legs keep their stock policy: badblocks are never recorded on a replacement, so failing it is the only outcome that cannot leave a silent hole in a rebuilding replacement. This belongs in the same release as the BLK_STS_TARGET restoration: with that fix alone, an unroutable leg costs the retry storms and the healthy-member evictions above. Measured on QEMU q35 rigs (one CMB provider, two NVMe members), 8KiB peer-memory I/O, reads x90 under concurrent host reads: scenario stock error machinery with this patch asym reads 90/90 ok, healthy far 90/90 ok, no eviction, leg kicked after ~20 no budget charge unreach reads all EIO, one healthy all EIO, no members member kicked kicked p2p TARGET 16 chunk retries from 1 whole-range probe, write raid1d, then badblocks same badblocks, master ok A transient TARGET recovers through the single probe with no badblocks recorded, and a mapping failure no longer trips FailFast eviction while a genuine I/O error still does -- both verified on the same rig. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 56 +++++++++++++++++++++++++++++++----- drivers/md/raid1.h | 2 ++ drivers/md/raid10.c | 69 ++++++++++++++++++++++++++++++++++++++------- drivers/md/raid10.h | 2 ++ 4 files changed, 112 insertions(+), 17 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index f562b6bd438b..61f635463475 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -483,12 +483,22 @@ static void raid1_end_write_request(struct bio *bio) * 'one mirror IO has finished' event handler: */ if (bio->bi_status && !ignore_error) { + /* + * A P2PDMA mapping failure reflects the peer/member + * pairing, not member health: don't pull in a spare + * or trip FailFast for it. + */ + bool p2pdma_unmappable = bio->bi_status == BLK_STS_TARGET && + test_bit(R1BIO_P2PDMA, &r1_bio->state); + set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) + if (!p2pdma_unmappable && + !test_and_set_bit(WantReplacement, &rdev->flags)) set_bit(MD_RECOVERY_NEEDED, & conf->mddev->recovery); - if (test_bit(FailFast, &rdev->flags) && + if (!p2pdma_unmappable && + test_bit(FailFast, &rdev->flags) && (bio->bi_opf & MD_FAILFAST) && /* We never try FailFast to WriteMostly devices */ !test_bit(WriteMostly, &rdev->flags)) { @@ -1378,6 +1388,8 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, else init_r1bio(r1_bio, mddev, bio); r1_bio->sectors = max_read_sectors; + if (md_bio_is_p2pdma(bio)) + set_bit(R1BIO_P2PDMA, &r1_bio->state); /* * make_request() can abort the operation when read-ahead is being @@ -1557,6 +1569,8 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio, r1_bio = alloc_r1bio(mddev, bio); r1_bio->sectors = max_sectors; + if (md_bio_is_p2pdma(bio)) + set_bit(R1BIO_P2PDMA, &r1_bio->state); /* first select target devices under rcu_lock and * inc refcount on their rdev. Record them by setting @@ -2525,7 +2539,7 @@ static void fix_read_error(struct r1conf *conf, struct r1bio *r1_bio) } } -static void narrow_write_error(struct r1bio *r1_bio, int i) +static void narrow_write_error(struct r1bio *r1_bio, int i, bool coarse) { struct mddev *mddev = r1_bio->mddev; struct r1conf *conf = mddev->private; @@ -2539,6 +2553,11 @@ static void narrow_write_error(struct r1bio *r1_bio, int i) * It is conceivable that the bio doesn't exactly align with * blocks. We must handle this somehow. * + * With 'coarse', retry the whole range as one bio and record + * one bad range if it fails: for P2PDMA mapping failures, + * which fail every block identically, while the single retry + * still lets a cleared transient error recover. + * * We currently own a reference on the rdev. */ @@ -2553,9 +2572,12 @@ static void narrow_write_error(struct r1bio *r1_bio, int i) block_sectors = roundup(1 << rdev->badblocks.shift, lbs); sector = r1_bio->sector; - sectors = ((sector + block_sectors) - & ~(sector_t)(block_sectors - 1)) - - sector; + if (coarse) + sectors = sect_to_write; + else + sectors = ((sector + block_sectors) + & ~(sector_t)(block_sectors - 1)) + - sector; while (sect_to_write) { struct bio *wbio; @@ -2636,8 +2658,18 @@ static void handle_write_finished(struct r1conf *conf, struct r1bio *r1_bio) * narrow down and record precise write * errors. */ + bool coarse = r1_bio->bios[m]->bi_status == + BLK_STS_TARGET && + test_bit(R1BIO_P2PDMA, &r1_bio->state); + fail = true; - narrow_write_error(r1_bio, m); + /* + * A P2PDMA mapping failure fails the whole range + * identically: probe it once (coarse) instead of + * narrowing block by block. A transient TARGET + * recovers via the probe with nothing recorded. + */ + narrow_write_error(r1_bio, m, coarse); rdev_dec_pending(conf->mirrors[m].rdev, conf->mddev); } @@ -2664,6 +2696,9 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) { struct md_rdev *rdev = conf->mirrors[r1_bio->read_disk].rdev; struct bio *bio = r1_bio->bios[r1_bio->read_disk]; + /* evaluate before the bio_put() below */ + bool p2pdma_error = bio->bi_status == BLK_STS_TARGET && + test_bit(R1BIO_P2PDMA, &r1_bio->state); struct mddev *mddev = conf->mddev; sector_t sector; @@ -2683,6 +2718,13 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) */ if (mddev->ro) { r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; + } else if (p2pdma_error) { + /* + * The peer cannot reach this member; nothing on the + * medium to fix. Skip the read-error budget and + * FailFast, just keep this leg out of the retry. + */ + r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; } else if (test_bit(FailFast, &rdev->flags)) { md_error(mddev, rdev); } else { diff --git a/drivers/md/raid1.h b/drivers/md/raid1.h index c98d43a7ae99..61b788a99d14 100644 --- a/drivers/md/raid1.h +++ b/drivers/md/raid1.h @@ -184,6 +184,8 @@ enum r1bio_state { R1BIO_MadeGood, R1BIO_WriteError, R1BIO_FailFast, +/* the master bio carries PCI P2PDMA (peer device memory) pages */ + R1BIO_P2PDMA, }; static inline int sector_to_idx(sector_t sector) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index f7ef903a3d4e..144d07b6029c 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -482,13 +482,24 @@ static void raid10_end_write_request(struct bio *bio) */ md_error(rdev->mddev, rdev); else { + /* + * A P2PDMA mapping failure reflects the + * peer/member pairing, not member health: don't + * pull in a spare or trip FailFast for it. + */ + bool p2pdma_unmappable = + bio->bi_status == BLK_STS_TARGET && + test_bit(R10BIO_P2PDMA, &r10_bio->state); + set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) + if (!p2pdma_unmappable && + !test_and_set_bit(WantReplacement, &rdev->flags)) set_bit(MD_RECOVERY_NEEDED, &rdev->mddev->recovery); dec_rdev = 0; - if (test_bit(FailFast, &rdev->flags) && + if (!p2pdma_unmappable && + test_bit(FailFast, &rdev->flags) && (bio->bi_opf & MD_FAILFAST)) { md_error(rdev->mddev, rdev); } @@ -1170,6 +1181,9 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, */ gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; + if (md_bio_is_p2pdma(bio)) + set_bit(R10BIO_P2PDMA, &r10_bio->state); + if (slot >= 0 && r10_bio->devs[slot].rdev) { /* * This is an error retry, but we cannot @@ -1357,6 +1371,9 @@ static bool raid10_write_request(struct mddev *mddev, struct bio *bio, sector_t sectors; int max_sectors; + if (md_bio_is_p2pdma(bio)) + set_bit(R10BIO_P2PDMA, &r10_bio->state); + if ((mddev_is_clustered(mddev) && mddev->cluster_ops->area_resyncing(mddev, WRITE, bio->bi_iter.bi_sector, @@ -2786,7 +2803,7 @@ static void fix_read_error(struct r10conf *conf, struct mddev *mddev, struct r10 } } -static void narrow_write_error(struct r10bio *r10_bio, int i) +static void narrow_write_error(struct r10bio *r10_bio, int i, bool coarse) { struct bio *bio = r10_bio->master_bio; struct mddev *mddev = r10_bio->mddev; @@ -2800,6 +2817,11 @@ static void narrow_write_error(struct r10bio *r10_bio, int i) * It is conceivable that the bio doesn't exactly align with * blocks. We must handle this. * + * With 'coarse', retry the whole range as one bio and record + * one bad range if it fails: for P2PDMA mapping failures, + * which fail every block identically, while the single retry + * still lets a cleared transient error recover. + * * We currently own a reference to the rdev. */ @@ -2814,9 +2836,12 @@ static void narrow_write_error(struct r10bio *r10_bio, int i) block_sectors = roundup(1 << rdev->badblocks.shift, lbs); sector = r10_bio->sector; - sectors = ((r10_bio->sector + block_sectors) - & ~(sector_t)(block_sectors - 1)) - - sector; + if (coarse) + sectors = sect_to_write; + else + sectors = ((r10_bio->sector + block_sectors) + & ~(sector_t)(block_sectors - 1)) + - sector; while (sect_to_write) { struct bio *wbio; @@ -2856,6 +2881,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) { int slot = r10_bio->read_slot; struct bio *bio; + bool p2pdma_error; struct r10conf *conf = mddev->private; struct md_rdev *rdev = r10_bio->devs[slot].rdev; @@ -2868,17 +2894,28 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) * frozen. */ bio = r10_bio->devs[slot].bio; + /* evaluate before the bio_put() below */ + p2pdma_error = bio->bi_status == BLK_STS_TARGET && + test_bit(R10BIO_P2PDMA, &r10_bio->state); bio_put(bio); r10_bio->devs[slot].bio = NULL; if (mddev->ro) r10_bio->devs[slot].bio = IO_BLOCKED; - else if (!test_bit(FailFast, &rdev->flags)) { + else if (p2pdma_error) { + /* + * The peer cannot reach this member; nothing on the + * medium to fix. Skip the read-error budget and + * FailFast, just keep this leg out of the retry. + */ + r10_bio->devs[slot].bio = IO_BLOCKED; + } else if (test_bit(FailFast, &rdev->flags)) { + md_error(mddev, rdev); + } else { freeze_array(conf, 1); fix_read_error(conf, mddev, r10_bio); unfreeze_array(conf); - } else - md_error(mddev, rdev); + } rdev_dec_pending(rdev, mddev); r10_bio->state = 0; @@ -2947,8 +2984,20 @@ static void handle_write_completed(struct r10conf *conf, struct r10bio *r10_bio) r10_bio->sectors, 0); rdev_dec_pending(rdev, conf->mddev); } else if (bio != NULL && bio->bi_status) { + bool coarse = bio->bi_status == + BLK_STS_TARGET && + test_bit(R10BIO_P2PDMA, + &r10_bio->state); + fail = true; - narrow_write_error(r10_bio, m); + /* + * A P2PDMA mapping failure fails the whole + * range identically: probe it once (coarse) + * instead of narrowing block by block. A + * transient TARGET recovers via the probe + * with nothing recorded. + */ + narrow_write_error(r10_bio, m, coarse); rdev_dec_pending(rdev, conf->mddev); } bio = r10_bio->devs[m].repl_bio; diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h index ec79d87fb92f..a2e1554f77db 100644 --- a/drivers/md/raid10.h +++ b/drivers/md/raid10.h @@ -174,6 +174,8 @@ enum r10bio_state { R10BIO_Previous, /* failfast devices did receive failfast requests. */ R10BIO_FailFast, +/* the master bio carries PCI P2PDMA (peer device memory) pages */ + R10BIO_P2PDMA, R10BIO_Discard, }; #endif -- 2.43.0 Since commit 23528aa3320a ("nvme: enable PCI P2PDMA support for RDMA transport") nvme-rdma accepts P2PDMA bios, but a mapping failure for peer memory the HCA cannot reach is misreported as a path error: ib_dma_map_sg() returns 0, discarding the -EREMOTEIO that dma_map_sgtable() documents for exactly this case, and the driver converts it to -EIO -> nvme_host_path_error(). Under the default multipath configuration (the multipath head node advertises BLK_FEAT_PCI_P2PDMA since commit fb0eeeed91f3 ("nvme-multipath: enable PCI P2PDMA for multipath devices")) the path-error status makes nvme_failover_req() requeue the bios with a fresh retry budget each cycle, and since the mapping failure is a deterministic property of the peer/device pairing the I/O simply never completes: a hot requeue livelock, and a stacked md mirror hangs instead of failing over to its other leg. With nvme_core.multipath=N the request burns nvme_max_retries requeues -- nothing ever reaching the wire -- and completes as BLK_STS_TRANSPORT, which blk_path_error() classifies as retryable and md/raid1,raid10 treat as a genuine device error: retry storms on writes, read-error-budget eviction of a healthy member on reads (see the preceding md patches, whose mapping-failure handling keys on BLK_STS_TARGET). Map both the data and the metadata scatterlists with ib_dma_map_sgtable_attrs() so the DMA layer's error code is preserved, and translate -EREMOTEIO to BLK_STS_TARGET: the classification nvme-pci established in commit 91fb2b6052f7 ("nvme-pci: convert to using dma_map_sgtable()") and the preceding blk-mq-dma patch restores. The metadata path needs the same treatment because integrity buffers are not always host memory: bio_integrity_map_user() allows P2P pages whenever the queue advertises P2PDMA, and nvme-pci already maps P2P metadata. Returning BLK_STS_TARGET from queue_rq is a deliberate trade-off: it forecloses failover for a configuration where a sibling path through a different HCA could reach the peer. This matches nvme-pci, which has returned BLK_STS_TARGET for unsupported P2P mappings since that same commit and equally forecloses cross-controller failover; distinguishing this-path-unreachable from all-paths-unreachable at completion time would cost a spurious requeue cycle per I/O, and a stacking consumer (md) routes around the leg one layer up. Start the request only after mapping succeeds, as nvme-pci does: a mapping failure now errors out of queue_rq on a not-yet-started request, so nvme_mpath_start_request() accounting is never taken and cannot leak on the direct blk-mq completion (this also closes the same latent leak for mapping-failure BLK_STS_IOERR returns; the post-send error tail is untouched). A mapping -ENOMEM now takes the existing BLK_STS_RESOURCE branch instead of masquerading as a path error, and a DMA-layer -EINVAL completes as BLK_STS_IOERR instead of a path error; generic -EIO keeps today's host-path-error behavior, and virt-DMA devices are unaffected. Ratelimit the map-failure message -- with an md mirror steering peer-memory I/O around an unreachable leg it fires per redirected I/O, not per rare event. Fixes: 23528aa3320a ("nvme: enable PCI P2PDMA support for RDMA transport") Cc: stable@vger.kernel.org # v7.1 Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/nvme/host/rdma.c | 42 +++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 6909e3542794..9017d927edc4 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1469,6 +1469,7 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, int *count, int *pi_count) { struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); + struct sg_table sgt; int ret; req->data_sgl.sg_table.sgl = (struct scatterlist *)(req + 1); @@ -1480,12 +1481,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, req->data_sgl.nents = blk_rq_map_sg(rq, req->data_sgl.sg_table.sgl); - *count = ib_dma_map_sg(ibdev, req->data_sgl.sg_table.sgl, - req->data_sgl.nents, rq_dma_dir(rq)); - if (unlikely(*count <= 0)) { - ret = -EIO; + sgt = (struct sg_table) { + .sgl = req->data_sgl.sg_table.sgl, + .orig_nents = req->data_sgl.nents, + }; + ret = ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_table; - } + *count = sgt.nents; if (blk_integrity_rq(rq)) { req->metadata_sgl->sg_table.sgl = @@ -1501,14 +1504,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, req->metadata_sgl->nents = blk_rq_map_integrity_sg(rq, req->metadata_sgl->sg_table.sgl); - *pi_count = ib_dma_map_sg(ibdev, - req->metadata_sgl->sg_table.sgl, - req->metadata_sgl->nents, - rq_dma_dir(rq)); - if (unlikely(*pi_count <= 0)) { - ret = -EIO; + sgt = (struct sg_table) { + .sgl = req->metadata_sgl->sg_table.sgl, + .orig_nents = req->metadata_sgl->nents, + }; + ret = ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_pi_table; - } + *pi_count = sgt.nents; } return 0; @@ -2026,8 +2029,6 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) goto unmap_qe; - nvme_start_request(rq); - if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || @@ -2039,11 +2040,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, err = nvme_rdma_map_data(queue, rq, c); if (unlikely(err < 0)) { - dev_err(queue->ctrl->ctrl.device, - "Failed to map data (%d)\n", err); + dev_err_ratelimited(queue->ctrl->ctrl.device, + "Failed to map data (%d)\n", err); goto err; } + nvme_start_request(rq); + sqe->cqe.done = nvme_rdma_send_done; ib_dma_sync_single_for_device(dev, sqe->dma, @@ -2063,6 +2066,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_host_path_error(rq); else if (err == -ENOMEM || err == -EAGAIN) ret = BLK_STS_RESOURCE; + /* + * The DMA layer refused to map peer memory to this device: a + * property of the pairing, not a path failure. Match nvme-pci + * and do not retry (see blk_path_error()). + */ + else if (err == -EREMOTEIO) + ret = BLK_STS_TARGET; else ret = BLK_STS_IOERR; nvme_cleanup_cmd(rq); -- 2.43.0