An unacknowledged write that is retried on another path can still be alive in the fabric. If it reaches the target after a later write to the same LBA has landed, it overwrites it, and a read returns stale data. Nothing in the tree exercises that window. Add a test that builds it deliberately. A ublk loop device backs an nvmet namespace exported through two ports, and the host connects to both, so nvme-multipath has a second path to fail over to. io_timeout on the subsystem drops to 2 seconds, then miniublk's inject command holds one write in the backstore for 4 seconds. The host times that write out, retries it on the other path, and the held write completes at the target afterwards. nvme-ghost-write-detector then writes distinct patterns to a single LBA and reads the block back, so a resurfaced write shows up as the wrong pattern. The test requires nvme_core.multipath=Y and a fabrics transport. As of today it passes on loop, which defines no timeout callback and so never times the write out and never retries it, and fails on tcp, rdma and fc. Signed-off-by: Mohamed Khalfella --- tests/nvme/070 | 98 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nvme/070.out | 35 +++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100755 tests/nvme/070 create mode 100644 tests/nvme/070.out diff --git a/tests/nvme/070 b/tests/nvme/070 new file mode 100755 index 0000000..e9a4690 --- /dev/null +++ b/tests/nvme/070 @@ -0,0 +1,98 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-3.0+ +# Copyright (C) 2026 Mohamed Khalfella + +. tests/nvme/rc +. common/ublk + +DESCRIPTION="Test injecting delay on nvme-target backstore and expect no corruption" + +requires() { + _nvme_requires + _have_loop + _have_ublk + _have_module_param_value nvme_core multipath Y + _require_nvme_trtype_is_fabrics + _have_src_program nvme-ghost-write-detector +} + +set_conditions() { + _set_nvme_trtype "$@" +} + +count_paths_to_subsystem() { + local subsysnqn="$1" + local dev count + + count=0 + for dev in /sys/class/nvme/nvme*; do + [[ -e "${dev}/subsysnqn" ]] || continue + [[ "$(cat "${dev}/subsysnqn")" == "${subsysnqn}" ]] || continue + count=$(( count + 1 )) + done + echo "${count}" +} + +set_io_timeout_of_subsystem() { + local subsysnqn="$1" + local timeout="$2" + local dev + + for dev in /sys/class/nvme/nvme*; do + [[ -e "${dev}/subsysnqn" ]] || continue + [[ "$(cat "${dev}/subsysnqn")" == "${subsysnqn}" ]] || continue + if ! echo "${timeout}" > "${dev}/io_timeout" 2> /dev/null; then + echo "FAIL: can not set io_timeout on ${dev##*/}" + return 1 + fi + done +} + +test() { + echo "Running ${TEST_NAME}" + + local ns port nr_paths + local -a ports + + if ! _init_ublk; then + return 1 + fi + + truncate -s "${NVME_IMG_SIZE}" "${TMPDIR}/ublk-img" + if ! ${UBLK_PROG} add -t loop -f "${TMPDIR}/ublk-img" -n 0 > "$FULL" 2>&1; then + echo "fail to add ublk device" + _exit_ublk + return 1 + fi + udevadm settle + + _setup_nvmet + _nvmet_target_setup --ports 2 --blkdev none + _create_nvmet_ns --blkdev /dev/ublkb0 \ + --uuid "${def_subsys_uuid}" > /dev/null + + _get_nvmet_ports "${def_subsysnqn}" ports + echo "Target ports: ${#ports[@]}" + for port in "${ports[@]}"; do + _nvme_connect_subsys --port "${port}" + done + + nr_paths=$(count_paths_to_subsystem "${def_subsysnqn}") + if (( nr_paths != 2 )); then + echo "FAIL: expected 2 paths, found ${nr_paths}" + fi + + set_io_timeout_of_subsystem "${def_subsysnqn}" 2000 + + if ! ${UBLK_PROG} inject -n 0 -o write -d 4 -c 1 >> "$FULL" 2>&1; then + echo "FAIL: can not inject write delay" + fi + + ns=$(_find_nvme_ns "${def_subsys_uuid}") + "$SRCDIR/nvme-ghost-write-detector" "/dev/${ns}" + + _nvme_disconnect_subsys + _nvmet_target_cleanup + _exit_ublk + echo "Test complete" +} diff --git a/tests/nvme/070.out b/tests/nvme/070.out new file mode 100644 index 0000000..b43fec7 --- /dev/null +++ b/tests/nvme/070.out @@ -0,0 +1,35 @@ +Running nvme/070 +Target ports: 2 +starting nvme-ghost-write-detector test program +iteration number 0, writing data +validating written data +successfully validated +iteration number 1, writing data +validating written data +successfully validated +iteration number 2, writing data +validating written data +successfully validated +iteration number 3, writing data +validating written data +successfully validated +iteration number 4, writing data +validating written data +successfully validated +iteration number 5, writing data +validating written data +successfully validated +iteration number 6, writing data +validating written data +successfully validated +iteration number 7, writing data +validating written data +successfully validated +iteration number 8, writing data +validating written data +successfully validated +iteration number 9, writing data +validating written data +successfully validated +finished nvme-ghost-write-detector test program +Test complete -- 2.55.0