| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/01 22:28 | flow | patch-triage |
1mArgs:null Results: {
"EnableConfigs": [
"VFIO",
"VFIO_PCI",
"VFIO_PCI_CORE",
"PCIEAER"
],
"FocusSymbols": [
"vfio_pci_core_ioctl_feature",
"vfio_pci_try_reset_function",
"vfio_pci_core_aer_err_detected",
"vfio_pci_core_aer_slot_reset",
"vfio_pci_core_aer_resume",
"vfio_pci_core_access_begin",
"vfio_pci_core_access_end",
"vfio_pci_config_rw_single",
"vfio_pci_bar_rw",
"vfio_pci_intx_recovery_start",
"vfio_pci_intx_recovery_finish"
],
"KMSANReasoning": "The patch introduces PCI error recovery support into the VFIO PCI driver, managing locking (recovery_lock vs memory_lock) and synchronization during AER events, resets, and power transitions.\n\nAnalysis of potential uninitialized memory risks:\n1. Kernel structure copied to user space:\n - In `vfio_pci_core_feature_error_recovery()`, `struct vfio_device_pci_error_recovery` is copied to userspace via `copy_to_user()`. The structure consists of `__u32 flags` (4 bytes), `__s32 eventfd` (4 bytes), and `__aligned_u64 sequence` (8 bytes), totaling 16 bytes with zero padding holes. The local variable `state` is initialized with `= { .eventfd = -1 }`, which zeroes all unmentioned fields, and then populated with the recovery state before being copied. No uninitialized bytes or padding are exposed.\n2. Stack/heap variables:\n - `struct vfio_pci_config_deferred deferred = {};` is zero-initialized on the stack.\n - All newly added struct fields in `struct vfio_pci_core_device` and `struct vfio_pci_irq_ctx` are initialized upon device/context allocation.\n3. Control flow and locking:\n - The patch primarily introduces access gating (`vfio_pci_core_access_begin`/`end`), wait queues, and error recovery handlers. Potential concurrency or locking bugs (such as deadlocks or race conditions) are within the domain of LOCKDEP and KASAN.\n\nThere are no uninitialized memory reads or information leak risks that would require a dedicated KMSAN fuzzing session.",
"NeedsKMSAN": false,
"Reasoning": "The patch series adds host PCI error recovery support to the VFIO PCI core driver. It introduces the VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY device feature ioctl, implements PCIe AER error recovery callbacks (.error_detected, .slot_reset, .resume), updates access guards and locking across config/BAR access, IRQ handlers, resets, and huge page fault handling. These changes modify reachable driver/core logic and are suitable for fuzzing.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/01 22:28 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 9245578c8ff1dcd9e2b6e621a377e36f1e75be35\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Sep 1 22:28:34 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c\nindex 830369ff878db..46544dbe70d01 100644\n--- a/drivers/vfio/pci/vfio_pci.c\n+++ b/drivers/vfio/pci/vfio_pci.c\n@@ -139,6 +139,7 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)\n \t */\n \tvdev-\u003enointxmask = nointxmask;\n \tvdev-\u003edisable_idle_d3 = disable_idle_d3;\n+\tvdev-\u003epci_recovery_supported = true;\n #ifdef CONFIG_VFIO_PCI_VGA\n \tvdev-\u003edisable_vga = disable_vga;\n #endif\ndiff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c\nindex 9914f3ac69aef..3365100acf216 100644\n--- a/drivers/vfio/pci/vfio_pci_config.c\n+++ b/drivers/vfio/pci/vfio_pci_config.c\n@@ -99,6 +99,12 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = {\n \t[PCI_EXT_CAP_ID_DVSEC]\t=\t0xFF,\n };\n \n+/* What a config write asked for which has to wait for the access guard. */\n+struct vfio_pci_config_deferred {\n+\tbool\tflr;\t\t/* a function-level reset */\n+\tbool\tpower_up;\t/* a transition to D0 */\n+};\n+\n /*\n * Read/Write Permission Bits - one bit for each bit in capability\n * Any field can be read if it exists, but what is read depends on\n@@ -111,8 +117,17 @@ struct perm_bits {\n \tu8\t*write;\t\t/* writeable bits */\n \tint\t(*readfn)(struct vfio_pci_core_device *vdev, int pos, int count,\n \t\t\t struct perm_bits *perm, int offset, __le32 *val);\n+\t/*\n+\t * @deferred records work the write asked for which a writefn must not\n+\t * do itself. Both a reset method and a transition to D0 acquire\n+\t * pci_bus_sem, which AER already holds when it enters the driver, so\n+\t * doing either here would invert the lock order against recovery_lock.\n+\t * The dispatcher does them after dropping recovery_lock. Callers zero\n+\t * it, and a writefn only sets a field on a success return.\n+\t */\n \tint\t(*writefn)(struct vfio_pci_core_device *vdev, int pos, int count,\n-\t\t\t struct perm_bits *perm, int offset, __le32 val);\n+\t\t\t struct perm_bits *perm, int offset, __le32 val,\n+\t\t\t struct vfio_pci_config_deferred *deferred);\n };\n \n #define\tNO_VIRT\t\t0\n@@ -200,7 +215,8 @@ static int vfio_default_config_read(struct vfio_pci_core_device *vdev, int pos,\n \n static int vfio_default_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \t__le32 virt = 0, write = 0;\n \n@@ -272,7 +288,8 @@ static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos,\n /* Raw access skips any kind of virtualization */\n static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \tint ret;\n \n@@ -299,7 +316,8 @@ static int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,\n /* Virt access uses only virtualization */\n static int vfio_virt_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \tmemcpy(vdev-\u003evconfig + pos, \u0026val, count);\n \treturn count;\n@@ -563,7 +581,8 @@ static bool vfio_need_bar_restore(struct vfio_pci_core_device *vdev)\n \n static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \tstruct pci_dev *pdev = vdev-\u003epdev;\n \t__le16 *virt_cmd;\n@@ -613,7 +632,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\tvfio_bar_restore(vdev);\n \t}\n \n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0) {\n \t\tif (offset == PCI_COMMAND)\n \t\t\tup_write(\u0026vdev-\u003ememory_lock);\n@@ -727,9 +747,11 @@ static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,\n \n static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\tint count, struct perm_bits *perm,\n-\t\t\t\tint offset, __le32 val)\n+\t\t\t\tint offset, __le32 val,\n+\t\t\t\tstruct vfio_pci_config_deferred *deferred)\n {\n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0)\n \t\treturn count;\n \n@@ -738,8 +760,15 @@ static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,\n \n \t\tswitch (le32_to_cpu(val) \u0026 PCI_PM_CTRL_STATE_MASK) {\n \t\tcase 0:\n-\t\t\tstate = PCI_D0;\n-\t\t\tbreak;\n+\t\t\t/*\n+\t\t\t * Going to D0 reaches pci_set_full_power_state(),\n+\t\t\t * which takes pci_bus_sem through\n+\t\t\t * pcie_aspm_pm_state_change(). Leave it to the\n+\t\t\t * dispatcher. The lower states do not, so they run\n+\t\t\t * here.\n+\t\t\t */\n+\t\t\tdeferred-\u003epower_up = true;\n+\t\t\treturn count;\n \t\tcase 1:\n \t\t\tstate = PCI_D1;\n \t\t\tbreak;\n@@ -799,7 +828,8 @@ static int __init init_pci_cap_pm_perm(struct perm_bits *perm)\n \n static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \tstruct pci_dev *pdev = vdev-\u003epdev;\n \t__le16 *paddr = (__le16 *)(vdev-\u003evconfig + pos - offset + PCI_VPD_ADDR);\n@@ -812,7 +842,8 @@ static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t * of PCI_VPD_ADDR, then the PCI_VPD_ADDR_F bit is written and we\n \t * have work to do.\n \t */\n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0 || offset \u003e PCI_VPD_ADDR + 1 ||\n \t offset + count \u003c= PCI_VPD_ADDR + 1)\n \t\treturn count;\n@@ -881,21 +912,24 @@ static int __init init_pci_cap_pcix_perm(struct perm_bits *perm)\n \n static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n \t__le16 *ctrl = (__le16 *)(vdev-\u003evconfig + pos -\n \t\t\t\t offset + PCI_EXP_DEVCTL);\n \tint readrq = le16_to_cpu(*ctrl) \u0026 PCI_EXP_DEVCTL_READRQ;\n \n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0)\n \t\treturn count;\n \n \t/*\n \t * The FLR bit is virtualized, if set and the device supports PCIe\n-\t * FLR, issue a reset_function. Regardless, clear the bit, the spec\n-\t * requires it to be always read as zero. NB, reset_function might\n-\t * not use a PCIe FLR, we don't have that level of granularity.\n+\t * FLR, request a function reset once recovery_lock has been\n+\t * released. Regardless, clear the bit, the spec requires it to be\n+\t * always read as zero. NB, reset_function might not use a PCIe FLR,\n+\t * we don't have that level of granularity.\n \t */\n \tif (*ctrl \u0026 cpu_to_le16(PCI_EXP_DEVCTL_BCR_FLR)) {\n \t\tu32 cap;\n@@ -907,14 +941,8 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t\t\t pos - offset + PCI_EXP_DEVCAP,\n \t\t\t\t\t\t \u0026cap);\n \n-\t\tif (!ret \u0026\u0026 (cap \u0026 PCI_EXP_DEVCAP_FLR)) {\n-\t\t\tvfio_pci_zap_and_down_write_memory_lock(vdev);\n-\t\t\tvfio_pci_dma_buf_move(vdev, true);\n-\t\t\tpci_try_reset_function(vdev-\u003epdev);\n-\t\t\tif (__vfio_pci_memory_enabled(vdev))\n-\t\t\t\tvfio_pci_dma_buf_move(vdev, false);\n-\t\t\tup_write(\u0026vdev-\u003ememory_lock);\n-\t\t}\n+\t\tif (!ret \u0026\u0026 (cap \u0026 PCI_EXP_DEVCAP_FLR))\n+\t\t\tdeferred-\u003eflr = true;\n \t}\n \n \t/*\n@@ -968,19 +996,22 @@ static int __init init_pci_cap_exp_perm(struct perm_bits *perm)\n \n static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\tint count, struct perm_bits *perm,\n-\t\t\t\tint offset, __le32 val)\n+\t\t\t\tint offset, __le32 val,\n+\t\t\t\tstruct vfio_pci_config_deferred *deferred)\n {\n \tu8 *ctrl = vdev-\u003evconfig + pos - offset + PCI_AF_CTRL;\n \n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0)\n \t\treturn count;\n \n \t/*\n \t * The FLR bit is virtualized, if set and the device supports AF\n-\t * FLR, issue a reset_function. Regardless, clear the bit, the spec\n-\t * requires it to be always read as zero. NB, reset_function might\n-\t * not use an AF FLR, we don't have that level of granularity.\n+\t * FLR, request a function reset once recovery_lock has been\n+\t * released. Regardless, clear the bit, the spec requires it to be\n+\t * always read as zero. NB, reset_function might not use an AF FLR,\n+\t * we don't have that level of granularity.\n \t */\n \tif (*ctrl \u0026 PCI_AF_CTRL_FLR) {\n \t\tu8 cap;\n@@ -992,14 +1023,8 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t\t\tpos - offset + PCI_AF_CAP,\n \t\t\t\t\t\t\u0026cap);\n \n-\t\tif (!ret \u0026\u0026 (cap \u0026 PCI_AF_CAP_FLR) \u0026\u0026 (cap \u0026 PCI_AF_CAP_TP)) {\n-\t\t\tvfio_pci_zap_and_down_write_memory_lock(vdev);\n-\t\t\tvfio_pci_dma_buf_move(vdev, true);\n-\t\t\tpci_try_reset_function(vdev-\u003epdev);\n-\t\t\tif (__vfio_pci_memory_enabled(vdev))\n-\t\t\t\tvfio_pci_dma_buf_move(vdev, false);\n-\t\t\tup_write(\u0026vdev-\u003ememory_lock);\n-\t\t}\n+\t\tif (!ret \u0026\u0026 (cap \u0026 PCI_AF_CAP_FLR) \u0026\u0026 (cap \u0026 PCI_AF_CAP_TP))\n+\t\t\tdeferred-\u003eflr = true;\n \t}\n \n \treturn count;\n@@ -1168,9 +1193,11 @@ static int vfio_msi_config_read(struct vfio_pci_core_device *vdev, int pos,\n \n static int vfio_msi_config_write(struct vfio_pci_core_device *vdev, int pos,\n \t\t\t\t int count, struct perm_bits *perm,\n-\t\t\t\t int offset, __le32 val)\n+\t\t\t\t int offset, __le32 val,\n+\t\t\t\t struct vfio_pci_config_deferred *deferred)\n {\n-\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val);\n+\tcount = vfio_default_config_write(vdev, pos, count, perm, offset, val,\n+\t\t\t\t\t deferred);\n \tif (count \u003c 0)\n \t\treturn count;\n \n@@ -1889,6 +1916,8 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,\n \tstruct perm_bits *perm;\n \t__le32 val = 0;\n \tint cap_start = 0, offset;\n+\tint access_ret;\n+\tstruct vfio_pci_config_deferred deferred = {};\n \tu8 cap_id;\n \tssize_t ret;\n \n@@ -1957,14 +1986,42 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,\n \t\tif (copy_from_user(\u0026val, buf, count))\n \t\t\treturn -EFAULT;\n \n-\t\tret = perm-\u003ewritefn(vdev, *ppos, count, perm, offset, val);\n+\t\taccess_ret = vfio_pci_core_access_begin(vdev);\n+\t\tif (access_ret)\n+\t\t\treturn access_ret;\n+\t\tret = perm-\u003ewritefn(vdev, *ppos, count, perm, offset, val,\n+\t\t\t\t \u0026deferred);\n+\t\tvfio_pci_core_access_end(vdev);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t\t/*\n+\t\t * Both of these take pci_bus_sem, so run them with the access\n+\t\t * guard dropped. The reset re-checks the recovery state for\n+\t\t * itself. The power up does not, so check it here.\n+\t\t *\n+\t\t * Both are best effort, as the guest-requested FLR has always\n+\t\t * been. The result is not reported back through the config\n+\t\t * write. Without recovery enabled the only failure is -EAGAIN\n+\t\t * from device lock contention, exactly as before. With it they\n+\t\t * are dropped while a recovery or reset transaction is in\n+\t\t * flight, which leaves the device in D0 and reset anyway.\n+\t\t */\n+\t\tif (deferred.power_up \u0026\u0026\n+\t\t !(vdev-\u003epci_recovery_supported \u0026\u0026\n+\t\t READ_ONCE(vdev-\u003epci_recovery_access_blocked)))\n+\t\t\tvfio_lock_and_set_power_state(vdev, PCI_D0);\n+\t\tif (deferred.flr)\n+\t\t\tvfio_pci_try_reset_function(vdev, false);\n \t} else {\n-\t\tif (perm-\u003ereadfn) {\n+\t\taccess_ret = vfio_pci_core_access_begin(vdev);\n+\t\tif (access_ret)\n+\t\t\treturn access_ret;\n+\t\tif (perm-\u003ereadfn)\n \t\t\tret = perm-\u003ereadfn(vdev, *ppos, count,\n \t\t\t\t\t perm, offset, \u0026val);\n-\t\t\tif (ret \u003c 0)\n-\t\t\t\treturn ret;\n-\t\t}\n+\t\tvfio_pci_core_access_end(vdev);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n \n \t\tif (copy_to_user(buf, \u0026val, count))\n \t\t\treturn -EFAULT;\ndiff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c\nindex 6757054e9d875..c1ea3c868fc5e 100644\n--- a/drivers/vfio/pci/vfio_pci_core.c\n+++ b/drivers/vfio/pci/vfio_pci_core.c\n@@ -372,15 +372,21 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat\n static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,\n \t\t\t\t struct eventfd_ctx *efdctx)\n {\n+\tint ret;\n+\n \t/*\n \t * The vdev power related flags are protected with 'memory_lock'\n \t * semaphore.\n \t */\n+\tret = vfio_pci_core_access_begin(vdev);\n+\tif (ret)\n+\t\treturn ret;\n \tvfio_pci_zap_and_down_write_memory_lock(vdev);\n \tvfio_pci_dma_buf_move(vdev, true);\n \n \tif (vdev-\u003epm_runtime_engaged) {\n \t\tup_write(\u0026vdev-\u003ememory_lock);\n+\t\tvfio_pci_core_access_end(vdev);\n \t\treturn -EINVAL;\n \t}\n \n@@ -388,6 +394,7 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,\n \tvdev-\u003epm_wake_eventfd_ctx = efdctx;\n \tpm_runtime_put_noidle(\u0026vdev-\u003epdev-\u003edev);\n \tup_write(\u0026vdev-\u003ememory_lock);\n+\tvfio_pci_core_access_end(vdev);\n \n \treturn 0;\n }\n@@ -483,7 +490,11 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,\n \t * already signaled the eventfd and exited low power mode itself.\n \t * pm_runtime_engaged protects the redundant call here.\n \t */\n+\tret = vfio_pci_core_access_begin(vdev);\n+\tif (ret)\n+\t\treturn ret;\n \tvfio_pci_runtime_pm_exit(vdev);\n+\tvfio_pci_core_access_end(vdev);\n \treturn 0;\n }\n \n@@ -591,10 +602,23 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {\n int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)\n {\n \tstruct pci_dev *pdev = vdev-\u003epdev;\n+\tbool supported = vdev-\u003epci_recovery_supported;\n \tint ret;\n \tu16 cmd;\n \tu8 msix_pos;\n \n+\tif (supported) {\n+\t\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\t\tif (pci_dev_is_disconnected(pdev)) {\n+\t\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t\t\treturn -ENODEV;\n+\t\t}\n+\n+\t\tvdev-\u003epci_recovery_command_valid = false;\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_device_open, false);\n+\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t}\n+\n \tif (!vdev-\u003edisable_idle_d3) {\n \t\tret = pm_runtime_resume_and_get(\u0026pdev-\u003edev);\n \t\tif (ret \u003c 0)\n@@ -815,7 +839,40 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)\n }\n EXPORT_SYMBOL_GPL(vfio_pci_core_disable);\n \n-void vfio_pci_core_close_device(struct vfio_device *core_vdev)\n+static void vfio_pci_core_prepare_close(struct vfio_pci_core_device *vdev)\n+{\n+\tif (!vdev-\u003epci_recovery_supported)\n+\t\treturn;\n+\n+\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\tWRITE_ONCE(vdev-\u003epci_recovery_enabled, false);\n+\tvdev-\u003epci_recovery_command_valid = false;\n+\t/*\n+\t * Clear access_blocked before device_open, so a lock-free reader\n+\t * never sees it set on a device which is no longer open. A\n+\t * transaction which is still running cannot clear it once\n+\t * device_open is gone, and paths which refuse work on a blocked\n+\t * device would then refuse it for good.\n+\t */\n+\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, false);\n+\tWRITE_ONCE(vdev-\u003epci_recovery_device_open, false);\n+\tWRITE_ONCE(vdev-\u003epci_recovery_flags, 0);\n+\n+\t/*\n+\t * Publish the closing state and drop recovery_lock before any\n+\t * teardown. Recovery is disabled and its state cleared, so\n+\t * slot_reset() and resume() become no-ops and a later\n+\t * error_detected() only follows the legacy notification path.\n+\t * Holding the lock across vfio_pci_core_disable() protects nothing\n+\t * and inverts the lock order. disable() reaches pci_reset_bus(),\n+\t * which takes pci_bus_sem, while error_detected() takes\n+\t * recovery_lock from under pci_bus_sem.\n+\t */\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n+\twake_up_all(\u0026vdev-\u003epci_recovery_wait);\n+}\n+\n+static void vfio_pci_core_finish_close(struct vfio_device *core_vdev)\n {\n \tstruct vfio_pci_core_device *vdev =\n \t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\n@@ -836,8 +893,21 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)\n \tmutex_lock(\u0026vdev-\u003eigate);\n \tvfio_pci_eventfd_replace_locked(vdev, \u0026vdev-\u003eerr_trigger, NULL);\n \tvfio_pci_eventfd_replace_locked(vdev, \u0026vdev-\u003ereq_trigger, NULL);\n+\tif (vdev-\u003epci_recovery_supported)\n+\t\tvfio_pci_eventfd_replace_locked(vdev,\n+\t\t\t\t\t\t\u0026vdev-\u003epci_recovery_trigger,\n+\t\t\t\t\t\tNULL);\n \tmutex_unlock(\u0026vdev-\u003eigate);\n }\n+\n+void vfio_pci_core_close_device(struct vfio_device *core_vdev)\n+{\n+\tstruct vfio_pci_core_device *vdev =\n+\t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\n+\n+\tvfio_pci_core_prepare_close(vdev);\n+\tvfio_pci_core_finish_close(core_vdev);\n+}\n EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);\n \n void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)\n@@ -852,6 +922,18 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)\n \t\tvdev-\u003esriov_pf_core_dev-\u003evf_token-\u003eusers++;\n \t\tmutex_unlock(\u0026vdev-\u003esriov_pf_core_dev-\u003evf_token-\u003elock);\n \t}\n+\n+\tif (vdev-\u003epci_recovery_supported) {\n+\t\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags, 0);\n+\t\tvdev-\u003epci_recovery_sequence = 0;\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_enabled, false);\n+\t\t/* Close clears this too. Start unblocked either way. */\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, false);\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_device_open, true);\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_rom_disable, false);\n+\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t}\n }\n EXPORT_SYMBOL_GPL(vfio_pci_core_finish_enable);\n \n@@ -1101,7 +1183,13 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,\n \t\treturn ret;\n \t}\n \n+\tret = vfio_pci_core_access_begin(vdev);\n+\tif (ret) {\n+\t\tkfree(caps.buf);\n+\t\treturn ret;\n+\t}\n \tret = vfio_pci_info_atomic_cap(vdev, \u0026caps);\n+\tvfio_pci_core_access_end(vdev);\n \tif (ret \u0026\u0026 ret != -ENODEV) {\n \t\tpci_warn(vdev-\u003epdev,\n \t\t\t \"Failed to setup AtomicOps info capability\\n\");\n@@ -1177,6 +1265,9 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,\n \t\t\t * Check ROM content is valid. Need to enable memory\n \t\t\t * decode for ROM access in pci_map_rom().\n \t\t\t */\n+\t\t\tret = vfio_pci_core_access_begin(vdev);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n \t\t\tcmd = vfio_pci_memory_lock_and_enable(vdev);\n \t\t\tio = pci_map_rom(pdev, \u0026size);\n \t\t\tif (io) {\n@@ -1187,6 +1278,7 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,\n \t\t\t\tpci_unmap_rom(pdev, io);\n \t\t\t}\n \t\t\tvfio_pci_memory_unlock_and_restore(vdev, cmd);\n+\t\t\tvfio_pci_core_access_end(vdev);\n \t\t} else if (pdev-\u003erom \u0026\u0026 pdev-\u003eromlen) {\n \t\t\tinfo-\u003eflags = VFIO_REGION_INFO_FLAG_READ;\n \t\t\t/* Report BAR size as power of two. */\n@@ -1242,11 +1334,29 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,\n }\n EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);\n \n+/*\n+ * Which IRQ indexes can reach the device. ERR and REQ are software only.\n+ * An index added later gets no access guard until it is listed here.\n+ */\n+static bool vfio_pci_irq_index_is_device(u32 index)\n+{\n+\tswitch (index) {\n+\tcase VFIO_PCI_INTX_IRQ_INDEX:\n+\tcase VFIO_PCI_MSI_IRQ_INDEX:\n+\tcase VFIO_PCI_MSIX_IRQ_INDEX:\n+\t\treturn true;\n+\tdefault:\n+\t\treturn false;\n+\t}\n+}\n+\n static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,\n \t\t\t\t struct vfio_irq_info __user *arg)\n {\n \tunsigned long minsz = offsetofend(struct vfio_irq_info, count);\n \tstruct vfio_irq_info info;\n+\tbool device_irq;\n+\tint ret;\n \n \tif (copy_from_user(\u0026info, arg, minsz))\n \t\treturn -EFAULT;\n@@ -1265,7 +1375,15 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,\n \n \tinfo.flags = VFIO_IRQ_INFO_EVENTFD;\n \n+\tdevice_irq = vfio_pci_irq_index_is_device(info.index);\n+\tif (device_irq) {\n+\t\tret = vfio_pci_core_access_begin(vdev);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n \tinfo.count = vfio_pci_get_irq_count(vdev, info.index);\n+\tif (device_irq)\n+\t\tvfio_pci_core_access_end(vdev);\n \n \tif (info.index == VFIO_PCI_INTX_IRQ_INDEX)\n \t\tinfo.flags |=\n@@ -1282,13 +1400,23 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,\n \tunsigned long minsz = offsetofend(struct vfio_irq_set, count);\n \tstruct vfio_irq_set hdr;\n \tu8 *data = NULL;\n+\tbool device_irq;\n \tint max, ret = 0;\n \tsize_t data_size = 0;\n \n \tif (copy_from_user(\u0026hdr, arg, minsz))\n \t\treturn -EFAULT;\n \n+\tdevice_irq = vfio_pci_irq_index_is_device(hdr.index);\n+\tif (device_irq) {\n+\t\tret = vfio_pci_core_access_begin(vdev);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n \tmax = vfio_pci_get_irq_count(vdev, hdr.index);\n+\t/* Dropped for the user copy below, which can fault under userfaultfd. */\n+\tif (device_irq)\n+\t\tvfio_pci_core_access_end(vdev);\n \n \tret = vfio_set_irqs_validate_and_prepare(\u0026hdr, max, VFIO_PCI_NUM_IRQS,\n \t\t\t\t\t\t \u0026data_size);\n@@ -1301,25 +1429,107 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,\n \t\t\treturn PTR_ERR(data);\n \t}\n \n+\t/*\n+\t * Interrupt teardown reaches vfio_virqfd_disable(), which flushes the\n+\t * global virqfd cleanup workqueue, so recovery_lock is held here for\n+\t * as long as work queued by any vfio device takes. Shutdown work waits\n+\t * for its inject worker, and an ioeventfd inject takes that device's\n+\t * memory_lock, so the wait can last as long as a reset there. That is\n+\t * only a wait. Nothing on that workqueue takes recovery_lock, which is\n+\t * why the ioeventfd write path reads the recovery state without it. A\n+\t * callback there which used the vfio_pci_core_iowrite*() accessors\n+\t * would break that and deadlock against a queued writer.\n+\t */\n+\tif (device_irq) {\n+\t\tret = vfio_pci_core_access_begin(vdev);\n+\t\tif (ret)\n+\t\t\tgoto out_free;\n+\t}\n \tmutex_lock(\u0026vdev-\u003eigate);\n \n \tret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index, hdr.start,\n \t\t\t\t hdr.count, data);\n \n \tmutex_unlock(\u0026vdev-\u003eigate);\n+\tif (device_irq)\n+\t\tvfio_pci_core_access_end(vdev);\n+out_free:\n \tkfree(data);\n \n \treturn ret;\n }\n \n-static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,\n-\t\t\t\tvoid __user *arg)\n+/*\n+ * Complete a ROM unmap which could not disable decode through config space.\n+ * Call once whatever blocked access has finished. A closed device is skipped.\n+ * It runs without recovery_lock, and close puts the device back through reset\n+ * and config restore.\n+ *\n+ * The IORESOURCE_ROM_ENABLE test is what pci_unmap_rom() would have done.\n+ * A ROM which firmware left enabled is not ours to turn off.\n+ */\n+static void vfio_pci_recovery_rom_disable(struct vfio_pci_core_device *vdev)\n {\n+\tstruct pci_dev *pdev = vdev-\u003epdev;\n+\n+\tlockdep_assert_held_write(\u0026vdev-\u003erecovery_lock);\n+\n+\tif (!vdev-\u003epci_recovery_device_open ||\n+\t !READ_ONCE(vdev-\u003epci_recovery_rom_disable))\n+\t\treturn;\n+\n+\tif (!(pdev-\u003eresource[PCI_ROM_RESOURCE].flags \u0026 IORESOURCE_ROM_ENABLE))\n+\t\tpci_disable_rom(pdev);\n+\tWRITE_ONCE(vdev-\u003epci_recovery_rom_disable, false);\n+}\n+\n+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,\n+\t\t\t\tbool reset_power_state)\n+{\n+\tstruct pci_dev *pdev = vdev-\u003epdev;\n+\tbool enabled = false;\n+\tbool supported = vdev-\u003epci_recovery_supported;\n \tint ret;\n \n-\tif (!vdev-\u003ereset_works)\n-\t\treturn -EINVAL;\n+\t/*\n+\t * Claim the device against recovery before resetting it. The PCI\n+\t * error callbacks hold recovery_lock for their whole body, so taking\n+\t * it for writing here waits for one already running, and\n+\t * access_blocked keeps a later one away while the lock is dropped.\n+\t */\n+\tif (supported) {\n+\t\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\t\tif (!vdev-\u003epci_recovery_device_open) {\n+\t\t\tret = -ENODEV;\n+\t\t\tgoto out_recovery;\n+\t\t}\n+\n+\t\tenabled = vdev-\u003epci_recovery_enabled;\n \n+\t\t/*\n+\t\t * Only claim access_blocked when recovery is enabled.\n+\t\t * error_detected() returns early for a device which has not\n+\t\t * enabled it, so there is nothing to exclude, and claiming it\n+\t\t * anyway would fail the second of two concurrent\n+\t\t * VFIO_DEVICE_RESET calls with -EBUSY.\n+\t\t */\n+\t\tif (enabled) {\n+\t\t\tif (vdev-\u003epci_recovery_access_blocked) {\n+\t\t\t\tret = -EBUSY;\n+\t\t\t\tgoto out_recovery;\n+\t\t\t}\n+\t\t\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, true);\n+\t\t}\n+\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t}\n+\n+\t/*\n+\t * On a device which supports recovery, taking recovery_lock for\n+\t * writing above waited for anything already past its access check,\n+\t * and if recovery is enabled access_blocked keeps new ones out. Do\n+\t * not hold recovery_lock while taking memory_lock or running a reset\n+\t * method, since a reset can take pci_bus_sem.\n+\t */\n \tvfio_pci_zap_and_down_write_memory_lock(vdev);\n \n \t/*\n@@ -1331,17 +1541,54 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,\n \t * reset without restoring the original state (saved locally in\n \t * 'vdev-\u003epm_save').\n \t */\n-\tvfio_pci_set_power_state(vdev, PCI_D0);\n+\tif (reset_power_state)\n+\t\tvfio_pci_set_power_state(vdev, PCI_D0);\n \n \tvfio_pci_dma_buf_move(vdev, true);\n-\tret = pci_try_reset_function(vdev-\u003epdev);\n+\tret = pci_try_reset_function(pdev);\n \tif (__vfio_pci_memory_enabled(vdev))\n \t\tvfio_pci_dma_buf_move(vdev, false);\n \tup_write(\u0026vdev-\u003ememory_lock);\n \n+\tif (enabled) {\n+\t\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\t\t/*\n+\t\t * An error callback can have started an event while the lock\n+\t\t * was down. Leave the state to it. Only unblock access for a\n+\t\t * reset which is still the one holding it.\n+\t\t */\n+\t\tif (vdev-\u003epci_recovery_device_open \u0026\u0026\n+\t\t !(vdev-\u003epci_recovery_flags \u0026 (VFIO_PCI_RECOVERY_IN_PROGRESS |\n+\t\t\t\t\t\t VFIO_PCI_RECOVERY_FAILED))) {\n+\t\t\tvfio_pci_recovery_rom_disable(vdev);\n+\t\t\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, false);\n+\t\t\tvfio_pci_intx_recovery_finish(vdev);\n+\t\t}\n+\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t\t/*\n+\t\t * Access is blocked for the length of the reset, so anything\n+\t\t * waiting for it to clear has to be woken here. A later patch\n+\t\t * adds the BAR fault path which waits on this.\n+\t\t */\n+\t\twake_up_all(\u0026vdev-\u003epci_recovery_wait);\n+\t}\n+\n+\treturn ret;\n+\n+out_recovery:\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n \treturn ret;\n }\n \n+static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,\n+\t\t\t\tvoid __user *arg)\n+{\n+\tif (!vdev-\u003ereset_works)\n+\t\treturn -EINVAL;\n+\n+\treturn vfio_pci_try_reset_function(vdev, true);\n+}\n+\n static int vfio_pci_ioctl_get_pci_hot_reset_info(\n \tstruct vfio_pci_core_device *vdev,\n \tstruct vfio_pci_hot_reset_info __user *arg)\n@@ -1607,6 +1854,106 @@ static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,\n \treturn 0;\n }\n \n+static int\n+vfio_pci_core_feature_error_recovery(struct vfio_pci_core_device *vdev, u32 flags,\n+\t\t\t\t struct vfio_device_pci_error_recovery __user *arg,\n+\t\t\t\t size_t argsz)\n+{\n+\tstruct vfio_device_pci_error_recovery state = { .eventfd = -1 };\n+\tstruct eventfd_ctx *ctx = NULL;\n+\tbool enable;\n+\tint ret;\n+\n+\tif (!vdev-\u003epci_recovery_supported)\n+\t\treturn -ENOTTY;\n+\n+\tret = vfio_check_feature(flags, argsz,\n+\t\t\t\t VFIO_DEVICE_FEATURE_GET |\n+\t\t\t\t VFIO_DEVICE_FEATURE_SET, sizeof(state));\n+\tif (ret != 1)\n+\t\treturn ret;\n+\n+\tif (flags \u0026 VFIO_DEVICE_FEATURE_GET) {\n+\t\tdown_read(\u0026vdev-\u003erecovery_lock);\n+\t\tif (vdev-\u003epci_recovery_enabled)\n+\t\t\tstate.flags |= VFIO_PCI_ERROR_RECOVERY_ENABLED;\n+\t\tif (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_IN_PROGRESS)\n+\t\t\tstate.flags |= VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS;\n+\t\tif (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_FROZEN)\n+\t\t\tstate.flags |=\n+\t\t\t\tVFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN;\n+\t\tif (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_RESET)\n+\t\t\tstate.flags |= VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET;\n+\t\tif (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_FAILED)\n+\t\t\tstate.flags |= VFIO_PCI_ERROR_RECOVERY_FAILED;\n+\t\tstate.sequence = vdev-\u003epci_recovery_sequence;\n+\t\tup_read(\u0026vdev-\u003erecovery_lock);\n+\n+\t\tif (copy_to_user(arg, \u0026state, sizeof(state)))\n+\t\t\treturn -EFAULT;\n+\t\treturn 0;\n+\t}\n+\n+\tif (copy_from_user(\u0026state, arg, sizeof(state)))\n+\t\treturn -EFAULT;\n+\tif (state.flags || state.sequence || state.eventfd \u003c -1)\n+\t\treturn -EINVAL;\n+\n+\tenable = state.eventfd \u003e= 0;\n+\tif (enable) {\n+\t\tctx = eventfd_ctx_fdget(state.eventfd);\n+\t\tif (IS_ERR(ctx))\n+\t\t\treturn PTR_ERR(ctx);\n+\t}\n+\n+\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\tif (!vdev-\u003epci_recovery_device_open) {\n+\t\tret = -ENODEV;\n+\t\tgoto out_unlock;\n+\t}\n+\tif (vdev-\u003epci_recovery_access_blocked) {\n+\t\tret = -EBUSY;\n+\t\tgoto out_unlock;\n+\t}\n+\n+\tif (!enable \u0026\u0026\n+\t (vdev-\u003epci_recovery_flags \u0026\n+\t (VFIO_PCI_RECOVERY_IN_PROGRESS | VFIO_PCI_RECOVERY_FAILED))) {\n+\t\tret = -EBUSY;\n+\t\tgoto out_unlock;\n+\t}\n+\n+\tmutex_lock(\u0026vdev-\u003eigate);\n+\tret = vfio_pci_eventfd_replace_locked(vdev,\n+\t\t\t\t\t \u0026vdev-\u003epci_recovery_trigger,\n+\t\t\t\t\t ctx);\n+\tmutex_unlock(\u0026vdev-\u003eigate);\n+\tif (ret)\n+\t\tgoto out_unlock;\n+\n+\tWRITE_ONCE(vdev-\u003epci_recovery_enabled, enable);\n+\t/*\n+\t * Start each enabled period from a clear state, so a sequence number\n+\t * and the status bits beside it always describe an event this\n+\t * eventfd was notified of. Nothing is in flight to lose. A\n+\t * transaction holds access_blocked, which failed this call with\n+\t * -EBUSY above.\n+\t *\n+\t * access_blocked itself is not cleared here, so userspace can never\n+\t * disable its way out of a block.\n+\t */\n+\tWRITE_ONCE(vdev-\u003epci_recovery_flags, 0);\n+\tvdev-\u003epci_recovery_sequence = 0;\n+\tvdev-\u003epci_recovery_command_valid = false;\n+\n+out_unlock:\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n+\tif (ret \u0026\u0026 ctx)\n+\t\teventfd_ctx_put(ctx);\n+\n+\treturn ret;\n+}\n+\n int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,\n \t\t\t\tvoid __user *arg, size_t argsz)\n {\n@@ -1627,6 +1974,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,\n \t\treturn vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);\n \tcase VFIO_DEVICE_FEATURE_ZPCI_ERROR:\n \t\treturn vfio_pci_zdev_feature_err(device, flags, arg, argsz);\n+\n+\tcase VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY:\n+\t\treturn vfio_pci_core_feature_error_recovery(vdev, flags, arg, argsz);\n \tdefault:\n \t\treturn -ENOTTY;\n \t}\n@@ -1642,6 +1992,24 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \tif (index \u003e= VFIO_PCI_NUM_REGIONS + vdev-\u003enum_regions)\n \t\treturn -EINVAL;\n \n+\tret = vfio_pci_core_access_begin(vdev);\n+\tif (ret)\n+\t\treturn ret;\n+\tvfio_pci_core_access_end(vdev);\n+\n+\t/*\n+\t * Resume with the guard dropped. A resume takes pci_bus_sem, through\n+\t * pcie_aspm_pm_state_change() and, from D3cold, through\n+\t * pci_bridge_wait_for_secondary_bus(). The error callbacks take\n+\t * recovery_lock from under pci_bus_sem, so holding it here would\n+\t * invert the order.\n+\t *\n+\t * The check above only avoids waking a device whose access is already\n+\t * blocked. A recovery which starts in between is not excluded, and\n+\t * does not need to be. pcie_do_recovery() has already resumed every\n+\t * device under the bridge and holds the reference until it finishes.\n+\t * The region access below takes the guard for itself.\n+\t */\n \tret = pm_runtime_resume_and_get(\u0026vdev-\u003epdev-\u003edev);\n \tif (ret) {\n \t\tpci_info_ratelimited(vdev-\u003epdev, \"runtime resume failed %d\\n\",\n@@ -1680,6 +2048,27 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \treturn ret;\n }\n \n+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev)\n+{\n+\tif (!vdev-\u003epci_recovery_supported)\n+\t\treturn 0;\n+\n+\tdown_read(\u0026vdev-\u003erecovery_lock);\n+\tif (unlikely(!vdev-\u003epci_recovery_device_open ||\n+\t\t vdev-\u003epci_recovery_access_blocked)) {\n+\t\tup_read(\u0026vdev-\u003erecovery_lock);\n+\t\treturn -EIO;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev)\n+{\n+\tif (vdev-\u003epci_recovery_supported)\n+\t\tup_read(\u0026vdev-\u003erecovery_lock);\n+}\n+\n ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,\n \t\tsize_t count, loff_t *ppos)\n {\n@@ -1776,6 +2165,127 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,\n }\n EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);\n \n+/*\n+ * Whether a fault which found access blocked is worth retrying. Read\n+ * without recovery_lock, so it is only a hint about when to look again.\n+ * vfio_pci_fault_trylock_once() takes the lock and decides. Read the flags\n+ * once so the two tests below see the same value. Every writer which can\n+ * make this true wakes pci_recovery_wait.\n+ */\n+static bool vfio_pci_recovery_done(struct vfio_pci_core_device *vdev)\n+{\n+\tu32 flags = READ_ONCE(vdev-\u003epci_recovery_flags);\n+\n+\tif (!READ_ONCE(vdev-\u003epci_recovery_device_open))\n+\t\treturn true;\n+\tif (flags \u0026 VFIO_PCI_RECOVERY_IN_PROGRESS)\n+\t\treturn false;\n+\tif (flags \u0026 VFIO_PCI_RECOVERY_FAILED)\n+\t\treturn true;\n+\treturn !READ_ONCE(vdev-\u003epci_recovery_access_blocked);\n+}\n+\n+static int vfio_pci_wait_for_recovery(struct vfio_pci_core_device *vdev)\n+{\n+\treturn wait_event_killable(vdev-\u003epci_recovery_wait,\n+\t\t\t\t vfio_pci_recovery_done(vdev));\n+}\n+\n+/* What one look at the recovery state says the fault should do. */\n+enum vfio_pci_fault_action {\n+\tVFIO_PCI_FAULT_PROCEED,\t/* returns with memory_lock held */\n+\tVFIO_PCI_FAULT_WAIT,\t/* recovery is running, may still recover */\n+\tVFIO_PCI_FAULT_FAIL,\t/* closed, or failed for good */\n+};\n+\n+static enum vfio_pci_fault_action\n+vfio_pci_fault_trylock_once(struct vfio_pci_core_device *vdev)\n+{\n+\tenum vfio_pci_fault_action action;\n+\n+\tdown_read(\u0026vdev-\u003erecovery_lock);\n+\tif (!vdev-\u003epci_recovery_device_open ||\n+\t (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_FAILED)) {\n+\t\taction = VFIO_PCI_FAULT_FAIL;\n+\t} else if (vdev-\u003epci_recovery_access_blocked) {\n+\t\t/*\n+\t\t * Blocked for a reason which still ends: a recovery which has\n+\t\t * not failed, or a function reset. Test FAILED above rather\n+\t\t * than IN_PROGRESS here, so a fault does not fail for good\n+\t\t * while a reset is running, or in the window where\n+\t\t * error_detected() has blocked access but not yet published\n+\t\t * the flags.\n+\t\t */\n+\t\taction = VFIO_PCI_FAULT_WAIT;\n+\t} else {\n+\t\tdown_read(\u0026vdev-\u003ememory_lock);\n+\t\taction = VFIO_PCI_FAULT_PROCEED;\n+\t}\n+\tup_read(\u0026vdev-\u003erecovery_lock);\n+\n+\treturn action;\n+}\n+\n+/*\n+ * Return true with memory_lock held for a fault that may proceed. Otherwise\n+ * return false with @ret set to the result the fault handler should return.\n+ */\n+static bool vfio_pci_core_fault_trylock(struct vfio_pci_core_device *vdev,\n+\t\t\t\t\tstruct vm_fault *vmf,\n+\t\t\t\t\tvm_fault_t *ret)\n+{\n+\tif (!vdev-\u003epci_recovery_supported) {\n+\t\tdown_read(\u0026vdev-\u003ememory_lock);\n+\t\treturn true;\n+\t}\n+\n+\tswitch (vfio_pci_fault_trylock_once(vdev)) {\n+\tcase VFIO_PCI_FAULT_PROCEED:\n+\t\treturn true;\n+\tcase VFIO_PCI_FAULT_FAIL:\n+\t\t*ret = VM_FAULT_SIGBUS;\n+\t\treturn false;\n+\tcase VFIO_PCI_FAULT_WAIT:\n+\t\tbreak;\n+\t}\n+\n+\tif (fault_flag_allow_retry_first(vmf-\u003eflags)) {\n+\t\tif (vmf-\u003eflags \u0026 FAULT_FLAG_RETRY_NOWAIT) {\n+\t\t\t*ret = VM_FAULT_RETRY;\n+\t\t\treturn false;\n+\t\t}\n+\n+\t\tif (!vfio_device_try_get_registration(\u0026vdev-\u003evdev)) {\n+\t\t\t*ret = VM_FAULT_SIGBUS;\n+\t\t\treturn false;\n+\t\t}\n+\n+\t\trelease_fault_lock(vmf);\n+\t\tvfio_pci_wait_for_recovery(vdev);\n+\t\tvfio_device_put_registration(\u0026vdev-\u003evdev);\n+\t\t*ret = VM_FAULT_RETRY;\n+\t\treturn false;\n+\t}\n+\n+\t/*\n+\t * The fault lock cannot be dropped here: either the caller did not\n+\t * allow a retry, or this fault has already used one. So wait with\n+\t * it held. It is not a deadlock. Recovery revokes mappings through\n+\t * unmap_mapping_range(), which never takes mmap_lock. The wait is\n+\t * killable.\n+\t */\n+\tif (vfio_pci_wait_for_recovery(vdev)) {\n+\t\t*ret = VM_FAULT_NOPAGE;\n+\t\treturn false;\n+\t}\n+\n+\tif (vfio_pci_fault_trylock_once(vdev) == VFIO_PCI_FAULT_PROCEED)\n+\t\treturn true;\n+\n+\t*ret = VM_FAULT_SIGBUS;\n+\treturn false;\n+}\n+\n static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,\n \t\t\t\t\t unsigned int order)\n {\n@@ -1787,8 +2297,11 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,\n \tvm_fault_t ret = VM_FAULT_FALLBACK;\n \n \tif (is_aligned_for_order(vma, addr, pfn, order)) {\n-\t\tscoped_guard(rwsem_read, \u0026vdev-\u003ememory_lock)\n-\t\t\tret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);\n+\t\tif (!vfio_pci_core_fault_trylock(vdev, vmf, \u0026ret))\n+\t\t\treturn ret;\n+\n+\t\tret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);\n+\t\tup_read(\u0026vdev-\u003ememory_lock);\n \t}\n \n \tdev_dbg_ratelimited(\u0026vdev-\u003epdev-\u003edev,\n@@ -2196,7 +2709,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)\n \tif (ret \u0026\u0026 ret != -EOPNOTSUPP)\n \t\treturn ret;\n \tINIT_LIST_HEAD(\u0026vdev-\u003edmabufs);\n+\tinit_rwsem(\u0026vdev-\u003erecovery_lock);\n \tinit_rwsem(\u0026vdev-\u003ememory_lock);\n+\tinit_waitqueue_head(\u0026vdev-\u003epci_recovery_wait);\n \txa_init(\u0026vdev-\u003ectx);\n \n \treturn 0;\n@@ -2333,22 +2848,347 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev)\n }\n EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device);\n \n+static void\n+vfio_pci_signal_recovery_event(struct vfio_pci_core_device *vdev)\n+{\n+\tstruct vfio_pci_eventfd *eventfd;\n+\n+\trcu_read_lock();\n+\teventfd = rcu_dereference(vdev-\u003epci_recovery_trigger);\n+\tif (eventfd)\n+\t\teventfd_signal(eventfd-\u003ectx);\n+\trcu_read_unlock();\n+}\n+\n pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,\n \t\t\t\t\t\tpci_channel_state_t state)\n {\n \tstruct vfio_pci_core_device *vdev = dev_get_drvdata(\u0026pdev-\u003edev);\n \tstruct vfio_pci_eventfd *eventfd;\n+\tpci_ers_result_t result = PCI_ERS_RESULT_CAN_RECOVER;\n+\tunsigned long irq_flags;\n+\tbool notify_recovery = false;\n+\tbool terminal = false;\n+\tbool nested;\n+\tu32 flags;\n+\tint ret;\n+\n+\tif (!vdev-\u003epci_recovery_supported ||\n+\t !READ_ONCE(vdev-\u003epci_recovery_enabled))\n+\t\tgoto out;\n+\n+\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\tif (!vdev-\u003epci_recovery_enabled)\n+\t\tgoto out_unlock;\n \n+\t/*\n+\t * A failed device remains blocked until close and a new open have\n+\t * reinitialized it. A later bridge event cannot make the saved VFIO\n+\t * state valid again.\n+\t */\n+\tif (vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_FAILED) {\n+\t\tresult = PCI_ERS_RESULT_NONE;\n+\t\tgoto out_unlock;\n+\t}\n+\n+\tif (!vdev-\u003epci_recovery_device_open) {\n+\t\tresult = PCI_ERS_RESULT_NONE;\n+\t\t/*\n+\t\t * PCI core rebroadcasts permanent failure when subtree\n+\t\t * recovery fails. Complete an event which started before\n+\t\t * close so a later open is not permanently stuck on\n+\t\t * IN_PROGRESS.\n+\t\t */\n+\t\tif (state == pci_channel_io_perm_failure \u0026\u0026\n+\t\t (vdev-\u003epci_recovery_flags \u0026\n+\t\t VFIO_PCI_RECOVERY_IN_PROGRESS)) {\n+\t\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t\t (vdev-\u003epci_recovery_flags |\n+\t\t\t\t VFIO_PCI_RECOVERY_FAILED) \u0026\n+\t\t\t\t ~VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\t\t\tvdev-\u003epci_recovery_command_valid = false;\n+\t\t\tterminal = true;\n+\t\t}\n+\t\tgoto out_unlock;\n+\t}\n+\n+\tnotify_recovery = true;\n+\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, true);\n+\t/*\n+\t * A second event before resume() has finished the first joins the\n+\t * transaction already running rather than starting one. Keep its\n+\t * sequence number, the command word it saved before the device was\n+\t * quiesced, and any reset a slot_reset() in between recorded. Reading\n+\t * the command word again here would save the quiesced value, and\n+\t * restoring that leaves the device with bus mastering off.\n+\t */\n+\tnested = vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_IN_PROGRESS;\n+\tif (!nested)\n+\t\tvdev-\u003epci_recovery_command_valid = false;\n+\tvfio_pci_intx_recovery_start(vdev);\n+\t/*\n+\t * INTx hardirq and virqfd callbacks cannot take recovery_lock.\n+\t * For devices with per-function INTx masking, mask INTx while holding\n+\t * irqlock so a callback which passed its blocked-state check is drained\n+\t * before the temporary command value is installed. Devices without\n+\t * per-function masking were quiesced above through genirq.\n+\t */\n+\tspin_lock_irqsave(\u0026vdev-\u003eirqlock, irq_flags);\n+\tret = 0;\n+\tif (state == pci_channel_io_normal \u0026\u0026 vdev-\u003epci_2_3 \u0026\u0026 !nested) {\n+\t\tu16 command;\n+\n+\t\tret = pci_read_config_word(pdev, PCI_COMMAND,\n+\t\t\t\t\t \u0026vdev-\u003epci_recovery_command);\n+\t\t/*\n+\t\t * A read from a device which has stopped responding succeeds\n+\t\t * and returns all ones. Writing that back would set every\n+\t\t * command bit, and saving it would restore them at the end.\n+\t\t */\n+\t\tif (!ret \u0026\u0026 PCI_POSSIBLE_ERROR(vdev-\u003epci_recovery_command))\n+\t\t\tret = -EIO;\n+\t\tif (!ret) {\n+\t\t\tcommand = (vdev-\u003epci_recovery_command \u0026\n+\t\t\t\t ~PCI_COMMAND_MASTER) |\n+\t\t\t\t PCI_COMMAND_INTX_DISABLE;\n+\t\t\tret = pci_write_config_word(pdev, PCI_COMMAND, command);\n+\t\t}\n+\t\tif (!ret)\n+\t\t\tvdev-\u003epci_recovery_command_valid = true;\n+\t}\n+\tspin_unlock_irqrestore(\u0026vdev-\u003eirqlock, irq_flags);\n+\tvfio_pci_zap_and_down_write_memory_lock(vdev);\n+\tvfio_pci_dma_buf_move(vdev, true);\n+\n+\t/*\n+\t * Allocate a sequence for a new transaction, and drop the flags the\n+\t * previous one left behind for userspace to read. A nested event adds\n+\t * to the flags already there. Each path below publishes the result in\n+\t * one store, so a lock-free reader never observes a cleared state that\n+\t * looks like successful completion.\n+\t */\n+\tflags = vdev-\u003epci_recovery_flags;\n+\tif (!nested) {\n+\t\tif (++vdev-\u003epci_recovery_sequence == 0)\n+\t\t\tvdev-\u003epci_recovery_sequence++;\n+\t\tflags = 0;\n+\t}\n+\n+\tif (state == pci_channel_io_perm_failure) {\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t (flags | VFIO_PCI_RECOVERY_FAILED) \u0026\n+\t\t\t ~VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\t\tvdev-\u003epci_recovery_command_valid = false;\n+\t\tresult = PCI_ERS_RESULT_DISCONNECT;\n+\t\tterminal = true;\n+\t\tgoto out_memory;\n+\t}\n+\n+\tif (state == pci_channel_io_frozen) {\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t flags | VFIO_PCI_RECOVERY_IN_PROGRESS |\n+\t\t\t VFIO_PCI_RECOVERY_FROZEN);\n+\t\tresult = PCI_ERS_RESULT_NEED_RESET;\n+\t\tgoto out_memory;\n+\t}\n+\n+\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t flags | VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\tif (ret)\n+\t\tgoto out_failed;\n+\tif (vdev-\u003epci_2_3 || nested)\n+\t\tgoto out_memory;\n+\n+\tret = pci_read_config_word(pdev, PCI_COMMAND,\n+\t\t\t\t \u0026vdev-\u003epci_recovery_command);\n+\tif (ret)\n+\t\tgoto out_failed;\n+\n+\tif (PCI_POSSIBLE_ERROR(vdev-\u003epci_recovery_command)) {\n+\t\tret = -EIO;\n+\t\tgoto out_failed;\n+\t}\n+\n+\tret = pci_write_config_word(pdev, PCI_COMMAND,\n+\t\t\t\t vdev-\u003epci_recovery_command \u0026\n+\t\t\t\t ~PCI_COMMAND_MASTER);\n+\tif (ret)\n+\t\tgoto out_failed;\n+\n+\tvdev-\u003epci_recovery_command_valid = true;\n+\tgoto out_memory;\n+\n+out_failed:\n+\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t (vdev-\u003epci_recovery_flags | VFIO_PCI_RECOVERY_FAILED) \u0026\n+\t\t ~VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\tresult = PCI_ERS_RESULT_NONE;\n+\tterminal = true;\n+out_memory:\n+\tup_write(\u0026vdev-\u003ememory_lock);\n+out_unlock:\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n+\tif (terminal)\n+\t\twake_up_all(\u0026vdev-\u003epci_recovery_wait);\n+\n+out:\n \trcu_read_lock();\n \teventfd = rcu_dereference(vdev-\u003eerr_trigger);\n \tif (eventfd)\n \t\teventfd_signal(eventfd-\u003ectx);\n \trcu_read_unlock();\n+\tif (notify_recovery)\n+\t\tvfio_pci_signal_recovery_event(vdev);\n \n-\treturn PCI_ERS_RESULT_CAN_RECOVER;\n+\treturn result;\n }\n EXPORT_SYMBOL_GPL(vfio_pci_core_aer_err_detected);\n \n+static pci_ers_result_t vfio_pci_core_aer_slot_reset(struct pci_dev *pdev)\n+{\n+\tstruct vfio_pci_core_device *vdev = dev_get_drvdata(\u0026pdev-\u003edev);\n+\tpci_ers_result_t result = PCI_ERS_RESULT_RECOVERED;\n+\tint ret = 0;\n+\n+\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\tif (!(vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_IN_PROGRESS) ||\n+\t !vdev-\u003epci_recovery_device_open) {\n+\t\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t\treturn PCI_ERS_RESULT_NONE;\n+\t}\n+\n+\t/*\n+\t * Restore first. aer_root_reset() resets the link with\n+\t * PCI_RESET_NO_RESTORE, so on entry the BARs read as zero. Tearing\n+\t * down MSI-X before this would have pci_msix_shutdown() write through\n+\t * the stale table mapping to an address the device no longer decodes.\n+\t */\n+\tpci_restore_state(pdev);\n+\n+\t/*\n+\t * Hold recovery_lock across the interrupt teardown.\n+\t * vfio_pci_core_disable() runs the same teardown on close without\n+\t * taking igate, and running the per-vector teardown twice frees the\n+\t * irq, the name and the eventfd context twice.\n+\t * vfio_pci_core_prepare_close() takes recovery_lock for writing\n+\t * before it, so holding it here keeps the two apart.\n+\t */\n+\tmutex_lock(\u0026vdev-\u003eigate);\n+\tif (vdev-\u003eirq_type \u003c VFIO_PCI_NUM_IRQS)\n+\t\tret = vfio_pci_set_irqs_ioctl(vdev,\n+\t\t\t\t\t VFIO_IRQ_SET_DATA_NONE |\n+\t\t\t\t\t VFIO_IRQ_SET_ACTION_TRIGGER,\n+\t\t\t\t\t vdev-\u003eirq_type, 0, 0, NULL);\n+\tmutex_unlock(\u0026vdev-\u003eigate);\n+\n+\tif (ret) {\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t (vdev-\u003epci_recovery_flags |\n+\t\t\t VFIO_PCI_RECOVERY_FAILED) \u0026\n+\t\t\t ~VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\t\tvdev-\u003epci_recovery_command_valid = false;\n+\t\t/*\n+\t\t * Vote NONE, not DISCONNECT. A DISCONNECT anywhere in the\n+\t\t * domain makes the core skip resume() for every device under\n+\t\t * the bridge and report permanent failure for all of them.\n+\t\t * Our interrupt teardown failing says nothing about the\n+\t\t * others, so record it locally and leave the domain verdict\n+\t\t * alone.\n+\t\t */\n+\t\tresult = PCI_ERS_RESULT_NONE;\n+\t} else {\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t vdev-\u003epci_recovery_flags |\n+\t\t\t VFIO_PCI_RECOVERY_RESET);\n+\t}\n+\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n+\t/*\n+\t * Whoever clears IN_PROGRESS owes the wake and the event. resume()\n+\t * will not do it, since it bails once IN_PROGRESS is clear, and the\n+\t * core skips it altogether if the domain verdict is not RECOVERED.\n+\t * On success the transaction carries on and resume() does both.\n+\t */\n+\tif (ret) {\n+\t\twake_up_all(\u0026vdev-\u003epci_recovery_wait);\n+\t\tvfio_pci_signal_recovery_event(vdev);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static void vfio_pci_core_aer_resume(struct pci_dev *pdev)\n+{\n+\tstruct vfio_pci_core_device *vdev = dev_get_drvdata(\u0026pdev-\u003edev);\n+\tunsigned long irq_flags;\n+\tbool notify_recovery = false;\n+\tu32 flags;\n+\tint ret = 0;\n+\n+\tdown_write(\u0026vdev-\u003erecovery_lock);\n+\n+\t/*\n+\t * Pay any deferred ROM disable before the in-progress check below,\n+\t * which a failed transaction has already cleared, or it would be\n+\t * lost.\n+\t */\n+\tvfio_pci_recovery_rom_disable(vdev);\n+\n+\tif (!(vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_IN_PROGRESS))\n+\t\tgoto out_unlock;\n+\n+\tnotify_recovery = true;\n+\tif (!vdev-\u003epci_recovery_device_open) {\n+\t\tvdev-\u003epci_recovery_command_valid = false;\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_flags,\n+\t\t\t vdev-\u003epci_recovery_flags \u0026\n+\t\t\t ~VFIO_PCI_RECOVERY_IN_PROGRESS);\n+\t\tgoto out_unlock;\n+\t}\n+\n+\tdown_write(\u0026vdev-\u003ememory_lock);\n+\t/*\n+\t * Restore the command word and clear access_blocked under irqlock.\n+\t * The INTx handler writes the same register through\n+\t * pci_check_and_mask_intx(), so it must not interleave with the\n+\t * restore, and it must not see access blocked cleared while the\n+\t * temporary command value is still installed.\n+\t *\n+\t * INTX_DISABLE comes from the INTx state rather than from the saved\n+\t * word, which can be older than the last mask. The replay below is\n+\t * what unmasks the line.\n+\t */\n+\tspin_lock_irqsave(\u0026vdev-\u003eirqlock, irq_flags);\n+\tif (!(vdev-\u003epci_recovery_flags \u0026 VFIO_PCI_RECOVERY_RESET) \u0026\u0026\n+\t vdev-\u003epci_recovery_command_valid) {\n+\t\tu16 cmd = vdev-\u003epci_recovery_command;\n+\n+\t\tcmd = vfio_pci_intx_recovery_command(vdev, cmd);\n+\t\tret = pci_write_config_word(pdev, PCI_COMMAND, cmd);\n+\t}\n+\tif (!ret)\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_access_blocked, false);\n+\tspin_unlock_irqrestore(\u0026vdev-\u003eirqlock, irq_flags);\n+\tif (!ret \u0026\u0026 __vfio_pci_memory_enabled(vdev))\n+\t\tvfio_pci_dma_buf_move(vdev, false);\n+\tup_write(\u0026vdev-\u003ememory_lock);\n+\n+\tvdev-\u003epci_recovery_command_valid = false;\n+\tflags = vdev-\u003epci_recovery_flags \u0026 ~VFIO_PCI_RECOVERY_IN_PROGRESS;\n+\tif (ret)\n+\t\tflags |= VFIO_PCI_RECOVERY_FAILED;\n+\tWRITE_ONCE(vdev-\u003epci_recovery_flags, flags);\n+\tif (!ret)\n+\t\tvfio_pci_intx_recovery_finish(vdev);\n+\n+out_unlock:\n+\tup_write(\u0026vdev-\u003erecovery_lock);\n+\tif (notify_recovery) {\n+\t\twake_up_all(\u0026vdev-\u003epci_recovery_wait);\n+\t\tvfio_pci_signal_recovery_event(vdev);\n+\t}\n+}\n+\n int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,\n \t\t\t\t int nr_virtfn)\n {\n@@ -2421,6 +3261,8 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure);\n \n const struct pci_error_handlers vfio_pci_core_err_handlers = {\n \t.error_detected = vfio_pci_core_aer_err_detected,\n+\t.slot_reset = vfio_pci_core_aer_slot_reset,\n+\t.resume = vfio_pci_core_aer_resume,\n };\n EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);\n \n@@ -2565,6 +3407,27 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,\n \t\t\tbreak;\n \t\t}\n \n+\t\t/*\n+\t\t * Between the AER callbacks memory_lock is not held, so refuse\n+\t\t * the reset on access_blocked as well. Read it rather than\n+\t\t * take recovery_lock, which would have to be released before\n+\t\t * pci_reset_bus() anyway since that reaches pci_bus_sem.\n+\t\t *\n+\t\t * Let a device which has failed for good through. Nothing is\n+\t\t * running on its behalf, and this reset covers the whole set,\n+\t\t * so refusing there would stop a healthy sibling from being\n+\t\t * reset because an unrelated device errored. The failed one\n+\t\t * stays blocked, since the reset does not unblock a device\n+\t\t * with FAILED set.\n+\t\t */\n+\t\tif (vdev-\u003epci_recovery_supported \u0026\u0026\n+\t\t READ_ONCE(vdev-\u003epci_recovery_access_blocked) \u0026\u0026\n+\t\t !(READ_ONCE(vdev-\u003epci_recovery_flags) \u0026\n+\t\t VFIO_PCI_RECOVERY_FAILED)) {\n+\t\t\tret = -EBUSY;\n+\t\t\tbreak;\n+\t\t}\n+\n \t\t/*\n \t\t * Take the memory write lock for each device and zap BAR\n \t\t * mappings to prevent the user accessing the device while in\ndiff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c\nindex c16f460c01d68..a54d199a72c92 100644\n--- a/drivers/vfio/pci/vfio_pci_dmabuf.c\n+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c\n@@ -243,12 +243,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,\n \tif (!get_dma_buf.nr_ranges || get_dma_buf.flags)\n \t\treturn -EINVAL;\n \n-\t/*\n-\t * For PCI the region_index is the BAR number like everything\n-\t * else. Check that PCI resources have been claimed for it.\n-\t */\n-\tif (get_dma_buf.region_index \u003e= VFIO_PCI_ROM_REGION_INDEX ||\n-\t IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index)))\n+\t/* For PCI the region_index is the BAR number like everything else. */\n+\tif (get_dma_buf.region_index \u003e= VFIO_PCI_ROM_REGION_INDEX)\n \t\treturn -ENODEV;\n \n \tdma_ranges = memdup_array_user(\u0026arg-\u003edma_ranges, get_dma_buf.nr_ranges,\n@@ -274,19 +270,30 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,\n \tpriv-\u003evdev = vdev;\n \tpriv-\u003enr_ranges = get_dma_buf.nr_ranges;\n \tpriv-\u003esize = length;\n+\n+\tret = vfio_pci_core_access_begin(vdev);\n+\tif (ret)\n+\t\tgoto err_free_phys;\n+\n+\t/* Check that PCI resources have been claimed for the BAR. */\n+\tif (IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index))) {\n+\t\tret = -ENODEV;\n+\t\tgoto err_access;\n+\t}\n+\n \tret = vdev-\u003epci_ops-\u003eget_dmabuf_phys(vdev, \u0026priv-\u003eprovider,\n \t\t\t\t\t get_dma_buf.region_index,\n \t\t\t\t\t priv-\u003ephys_vec, dma_ranges,\n \t\t\t\t\t priv-\u003enr_ranges);\n \tif (ret)\n-\t\tgoto err_free_phys;\n+\t\tgoto err_access;\n \n \tkfree(dma_ranges);\n \tdma_ranges = NULL;\n \n \tif (!vfio_device_try_get_registration(\u0026vdev-\u003evdev)) {\n \t\tret = -ENODEV;\n-\t\tgoto err_free_phys;\n+\t\tgoto err_access;\n \t}\n \n \texp_info.ops = \u0026vfio_pci_dmabuf_ops;\n@@ -311,6 +318,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,\n \tlist_add_tail(\u0026priv-\u003edmabufs_elm, \u0026vdev-\u003edmabufs);\n \tdma_resv_unlock(priv-\u003edmabuf-\u003eresv);\n \tup_write(\u0026vdev-\u003ememory_lock);\n+\tvfio_pci_core_access_end(vdev);\n \n \t/*\n \t * dma_buf_fd() consumes the reference, when the file closes the dmabuf\n@@ -324,6 +332,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,\n \n err_dev_put:\n \tvfio_device_put_registration(\u0026vdev-\u003evdev);\n+err_access:\n+\tvfio_pci_core_access_end(vdev);\n err_free_phys:\n \tkfree(priv-\u003ephys_vec);\n err_free_priv:\ndiff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c\nindex 64f80f64ff57a..1981a64b6e181 100644\n--- a/drivers/vfio/pci/vfio_pci_intrs.c\n+++ b/drivers/vfio/pci/vfio_pci_intrs.c\n@@ -29,6 +29,8 @@ struct vfio_pci_irq_ctx {\n \tstruct virqfd\t\t\t*mask;\n \tchar\t\t\t\t*name;\n \tbool\t\t\t\tmasked;\n+\tbool\t\t\t\trecovery_masked;\n+\tbool\t\t\t\tunmask_pending;\n \tstruct irq_bypass_producer\tproducer;\n };\n \n@@ -49,6 +51,15 @@ static bool is_irq_none(struct vfio_pci_core_device *vdev)\n \t\t vdev-\u003eirq_type == VFIO_PCI_MSIX_IRQ_INDEX);\n }\n \n+static bool vfio_pci_recovery_blocks_irq(struct vfio_pci_core_device *vdev)\n+{\n+\tif (!vdev-\u003epci_recovery_supported)\n+\t\treturn false;\n+\n+\treturn READ_ONCE(vdev-\u003epci_recovery_enabled) \u0026\u0026\n+\t READ_ONCE(vdev-\u003epci_recovery_access_blocked);\n+}\n+\n static\n struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,\n \t\t\t\t\t unsigned long index)\n@@ -171,6 +182,16 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)\n \tint ret = 0;\n \n \tspin_lock_irqsave(\u0026vdev-\u003eirqlock, flags);\n+\t/*\n+\t * Check for a blocked device before the INTx test below, so a blocked\n+\t * device is not touched through pci_intx(). @ctx is only valid when\n+\t * INTx is in use, so record the request only then.\n+\t */\n+\tif (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {\n+\t\tif (is_intx(vdev))\n+\t\t\tctx-\u003eunmask_pending = true;\n+\t\tgoto out_unlock;\n+\t}\n \n \t/*\n \t * Unmasking comes from ioctl or config, so again, have the\n@@ -182,6 +203,9 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)\n \t\tgoto out_unlock;\n \t}\n \n+\tctx-\u003eunmask_pending = false;\n+\tctx-\u003erecovery_masked = false;\n+\n \tif (ctx-\u003emasked \u0026\u0026 !vdev-\u003evirq_disabled) {\n \t\t/*\n \t\t * A pending interrupt here would immediately trigger,\n@@ -220,6 +244,47 @@ void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev)\n \tmutex_unlock(\u0026vdev-\u003eigate);\n }\n \n+/*\n+ * Mask INTx because recovery has blocked device access. Returns true if this\n+ * call did the masking.\n+ *\n+ * Set @quiesce when recovery is masking the line itself rather than masking\n+ * one delivered interrupt. Nothing is normally asserted at that point, and\n+ * pci_check_and_mask_intx() only writes DisINTx when the status register says\n+ * an interrupt is pending, so it would leave the line alone. The interrupt\n+ * handler wants that test, since a false return there means the interrupt\n+ * belongs to another device on a shared line. @quiesce also records that\n+ * recovery is the one which must unmask. A masked interrupt which was\n+ * delivered is the user's to unmask, exactly as outside recovery.\n+ *\n+ * Masking a pci_2_3 device goes through config space. If the error left\n+ * config space unreadable the write has no effect and the line stays\n+ * asserted, which is no worse than not trying. For a non-fatal error config\n+ * space still works, and this is what keeps a shared line from storming while\n+ * access is blocked.\n+ */\n+static bool vfio_pci_intx_mask_for_recovery(struct vfio_pci_core_device *vdev,\n+\t\t\t\t\t struct vfio_pci_irq_ctx *ctx,\n+\t\t\t\t\t bool quiesce)\n+{\n+\tlockdep_assert_held(\u0026vdev-\u003eirqlock);\n+\n+\tif (ctx-\u003emasked)\n+\t\treturn false;\n+\n+\tif (!vdev-\u003epci_2_3)\n+\t\tdisable_irq_nosync(vdev-\u003epdev-\u003eirq);\n+\telse if (quiesce)\n+\t\tpci_intx(vdev-\u003epdev, 0);\n+\telse if (!pci_check_and_mask_intx(vdev-\u003epdev))\n+\t\treturn false;\n+\n+\tctx-\u003emasked = true;\n+\tif (quiesce)\n+\t\tctx-\u003erecovery_masked = true;\n+\treturn true;\n+}\n+\n static irqreturn_t vfio_intx_handler(int irq, void *dev_id)\n {\n \tstruct vfio_pci_irq_ctx *ctx = dev_id;\n@@ -228,6 +293,19 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)\n \tint ret = IRQ_NONE;\n \n \tspin_lock_irqsave(\u0026vdev-\u003eirqlock, flags);\n+\tif (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {\n+\t\t/*\n+\t\t * Mask rather than return IRQ_NONE with the line still\n+\t\t * asserted. For a shared pci_2_3 line an unhandled level\n+\t\t * interrupt storms until note_interrupt() disables the line\n+\t\t * for every device on it, not just this one.\n+\t\t */\n+\t\tif (vfio_pci_intx_mask_for_recovery(vdev, ctx, false))\n+\t\t\tret = IRQ_HANDLED;\n+\t\telse if (ctx-\u003emasked \u0026\u0026 !vdev-\u003epci_2_3)\n+\t\t\tret = IRQ_HANDLED;\n+\t\tgoto out_unlock;\n+\t}\n \n \tif (!vdev-\u003epci_2_3) {\n \t\tdisable_irq_nosync(vdev-\u003epdev-\u003eirq);\n@@ -239,6 +317,7 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)\n \t\tret = IRQ_HANDLED;\n \t}\n \n+out_unlock:\n \tspin_unlock_irqrestore(\u0026vdev-\u003eirqlock, flags);\n \n \tif (ret == IRQ_HANDLED)\n@@ -247,6 +326,86 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)\n \treturn ret;\n }\n \n+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev)\n+{\n+\tstruct vfio_pci_irq_ctx *ctx;\n+\tunsigned long flags;\n+\n+\tlockdep_assert_held_write(\u0026vdev-\u003erecovery_lock);\n+\n+\tspin_lock_irqsave(\u0026vdev-\u003eirqlock, flags);\n+\tif (!is_intx(vdev))\n+\t\tgoto out_unlock;\n+\n+\tctx = vfio_irq_ctx_get(vdev, 0);\n+\tif (WARN_ON_ONCE(!ctx))\n+\t\tgoto out_unlock;\n+\n+\tvfio_pci_intx_mask_for_recovery(vdev, ctx, true);\n+\n+out_unlock:\n+\tspin_unlock_irqrestore(\u0026vdev-\u003eirqlock, flags);\n+}\n+\n+/*\n+ * Replay the masking recovery did, and any unmask which arrived while it was\n+ * blocked. Call this only after access_blocked has been cleared, or the\n+ * replayed unmask is swallowed and recorded as pending again with nothing\n+ * left to replay it.\n+ */\n+/*\n+ * The command word saved before the quiesce can have INTX_DISABLE clear, but\n+ * the INTx handler may have masked the line since. Keep the bit as the INTx\n+ * state has it, so hardware and ctx-\u003emasked agree until\n+ * vfio_pci_intx_recovery_finish() replays. Restoring the saved bit instead\n+ * would unmask a line the handler still believes is masked, and a shared\n+ * pci_2_3 line would then storm until note_interrupt() disables it.\n+ */\n+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,\n+\t\t\t\t u16 command)\n+{\n+\tstruct vfio_pci_irq_ctx *ctx;\n+\n+\tlockdep_assert_held(\u0026vdev-\u003eirqlock);\n+\n+\tif (!is_intx(vdev))\n+\t\treturn command;\n+\n+\tctx = vfio_irq_ctx_get(vdev, 0);\n+\tif (ctx \u0026\u0026 ctx-\u003emasked)\n+\t\tcommand |= PCI_COMMAND_INTX_DISABLE;\n+\n+\treturn command;\n+}\n+\n+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev)\n+{\n+\tstruct vfio_pci_irq_ctx *ctx;\n+\tunsigned long flags;\n+\tbool replay = false;\n+\n+\tlockdep_assert_held_write(\u0026vdev-\u003erecovery_lock);\n+\n+\tmutex_lock(\u0026vdev-\u003eigate);\n+\tspin_lock_irqsave(\u0026vdev-\u003eirqlock, flags);\n+\tif (!is_intx(vdev))\n+\t\tgoto out_unlock;\n+\n+\tctx = vfio_irq_ctx_get(vdev, 0);\n+\tif (WARN_ON_ONCE(!ctx))\n+\t\tgoto out_unlock;\n+\n+\treplay = ctx-\u003erecovery_masked || ctx-\u003eunmask_pending;\n+\tctx-\u003erecovery_masked = false;\n+\tctx-\u003eunmask_pending = false;\n+\n+out_unlock:\n+\tspin_unlock_irqrestore(\u0026vdev-\u003eirqlock, flags);\n+\tif (replay)\n+\t\t__vfio_pci_intx_unmask(vdev);\n+\tmutex_unlock(\u0026vdev-\u003eigate);\n+}\n+\n static int vfio_intx_enable(struct vfio_pci_core_device *vdev,\n \t\t\t struct eventfd_ctx *trigger)\n {\ndiff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h\nindex 4e7162234a2eb..5598e472da4bf 100644\n--- a/drivers/vfio/pci/vfio_pci_priv.h\n+++ b/drivers/vfio/pci/vfio_pci_priv.h\n@@ -25,6 +25,10 @@ struct vfio_pci_ioeventfd {\n \n bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);\n void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);\n+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev);\n+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev);\n+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,\n+\t\t\t\t u16 command);\n \n int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,\n \t\t\t\t struct vfio_pci_eventfd __rcu **peventfd,\n@@ -41,6 +45,9 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,\n \t\t\t\t char __user *buf, size_t count, loff_t *ppos,\n \t\t\t\t bool iswrite);\n \n+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,\n+\t\t\t\tbool reset_power_state);\n+\n ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \t\t\tsize_t count, loff_t *ppos, bool iswrite);\n \n@@ -73,6 +80,9 @@ u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);\n void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,\n \t\t\t\t\tu16 cmd);\n \n+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev);\n+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev);\n+\n #ifdef CONFIG_VFIO_PCI_IGD\n bool vfio_pci_is_intel_display(struct pci_dev *pdev);\n int vfio_pci_igd_init(struct vfio_pci_core_device *vdev);\ndiff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c\nindex 7f14dd46de17a..86fadc9999621 100644\n--- a/drivers/vfio/pci/vfio_pci_rdwr.c\n+++ b/drivers/vfio/pci/vfio_pci_rdwr.c\n@@ -42,10 +42,17 @@\n int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,\t\\\n \t\t\tbool test_mem, u##size val, void __iomem *io)\t\\\n {\t\t\t\t\t\t\t\t\t\\\n+\tint ret;\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tret = vfio_pci_core_access_begin(vdev);\t\t\t\t\\\n+\tif (ret)\t\t\t\t\t\t\t\\\n+\t\treturn ret;\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n \tif (test_mem) {\t\t\t\t\t\t\t\\\n \t\tdown_read(\u0026vdev-\u003ememory_lock);\t\t\t\t\\\n \t\tif (!__vfio_pci_memory_enabled(vdev)) {\t\t\t\\\n \t\t\tup_read(\u0026vdev-\u003ememory_lock);\t\t\t\\\n+\t\t\tvfio_pci_core_access_end(vdev);\t\t\t\\\n \t\t\treturn -EIO;\t\t\t\t\t\\\n \t\t}\t\t\t\t\t\t\t\\\n \t}\t\t\t\t\t\t\t\t\\\n@@ -54,6 +61,7 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,\t\\\n \t\t\t\t\t\t\t\t\t\\\n \tif (test_mem)\t\t\t\t\t\t\t\\\n \t\tup_read(\u0026vdev-\u003ememory_lock);\t\t\t\t\\\n+\tvfio_pci_core_access_end(vdev);\t\t\t\t\t\\\n \t\t\t\t\t\t\t\t\t\\\n \treturn 0;\t\t\t\t\t\t\t\\\n }\t\t\t\t\t\t\t\t\t\\\n@@ -68,10 +76,17 @@ VFIO_IOWRITE(64)\n int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,\t\\\n \t\t\tbool test_mem, u##size *val, void __iomem *io)\t\\\n {\t\t\t\t\t\t\t\t\t\\\n+\tint ret;\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tret = vfio_pci_core_access_begin(vdev);\t\t\t\t\\\n+\tif (ret)\t\t\t\t\t\t\t\\\n+\t\treturn ret;\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n \tif (test_mem) {\t\t\t\t\t\t\t\\\n \t\tdown_read(\u0026vdev-\u003ememory_lock);\t\t\t\t\\\n \t\tif (!__vfio_pci_memory_enabled(vdev)) {\t\t\t\\\n \t\t\tup_read(\u0026vdev-\u003ememory_lock);\t\t\t\\\n+\t\t\tvfio_pci_core_access_end(vdev);\t\t\t\\\n \t\t\treturn -EIO;\t\t\t\t\t\\\n \t\t}\t\t\t\t\t\t\t\\\n \t}\t\t\t\t\t\t\t\t\\\n@@ -80,6 +95,7 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,\t\\\n \t\t\t\t\t\t\t\t\t\\\n \tif (test_mem)\t\t\t\t\t\t\t\\\n \t\tup_read(\u0026vdev-\u003ememory_lock);\t\t\t\t\\\n+\tvfio_pci_core_access_end(vdev);\t\t\t\t\t\\\n \t\t\t\t\t\t\t\t\t\\\n \treturn 0;\t\t\t\t\t\t\t\\\n }\t\t\t\t\t\t\t\t\t\\\n@@ -198,12 +214,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,\n }\n EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);\n \n+/*\n+ * Undo pci_map_rom(). The iounmap is always safe, but pci_disable_rom() is a\n+ * config space write. If recovery has blocked access, do the iounmap now and\n+ * record the disable, for whichever of resume() or the reset tail unblocks\n+ * access again. recovery_lock spans the decision and the record so recovery\n+ * cannot complete in between.\n+ */\n+static void vfio_pci_unmap_rom(struct vfio_pci_core_device *vdev,\n+\t\t\t void __iomem *io)\n+{\n+\tstruct pci_dev *pdev = vdev-\u003epdev;\n+\n+\tif (!vdev-\u003epci_recovery_supported) {\n+\t\tpci_unmap_rom(pdev, io);\n+\t\treturn;\n+\t}\n+\n+\tdown_read(\u0026vdev-\u003erecovery_lock);\n+\tif (vdev-\u003epci_recovery_device_open \u0026\u0026\n+\t !vdev-\u003epci_recovery_access_blocked) {\n+\t\tpci_unmap_rom(pdev, io);\n+\t} else {\n+\t\tiounmap(io);\n+\t\tWRITE_ONCE(vdev-\u003epci_recovery_rom_disable, true);\n+\t}\n+\tup_read(\u0026vdev-\u003erecovery_lock);\n+}\n+\n ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \t\t\tsize_t count, loff_t *ppos, bool iswrite)\n {\n \tstruct pci_dev *pdev = vdev-\u003epdev;\n \tloff_t pos = *ppos \u0026 VFIO_PCI_OFFSET_MASK;\n \tint bar = VFIO_PCI_OFFSET_TO_INDEX(*ppos);\n+\tint ret;\n \tsize_t x_start = 0, x_end = 0;\n \tresource_size_t end;\n \tvoid __iomem *io;\n@@ -230,7 +275,11 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \t\t * filling large ROM BARs much faster.\n \t\t */\n \t\tif (pci_resource_start(pdev, bar)) {\n+\t\t\tret = vfio_pci_core_access_begin(vdev);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n \t\t\tio = pci_map_rom(pdev, \u0026x_start);\n+\t\t\tvfio_pci_core_access_end(vdev);\n \t\t} else {\n \t\t\tio = ioremap(pdev-\u003erom, pdev-\u003eromlen);\n \t\t\tx_start = pdev-\u003eromlen;\n@@ -269,7 +318,7 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n \n \tif (bar == PCI_ROM_RESOURCE) {\n \t\tif (pci_resource_start(pdev, bar))\n-\t\t\tpci_unmap_rom(pdev, io);\n+\t\t\tvfio_pci_unmap_rom(vdev, io);\n \t\telse\n \t\t\tiounmap(io);\n \t}\n@@ -349,56 +398,85 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,\n }\n #endif\n \n-static void vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,\n-\t\t\t\t\tbool test_mem)\n+static int vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,\n+\t\t\t\t bool trylock)\n {\n+\tstruct vfio_pci_core_device *vdev = ioeventfd-\u003evdev;\n+\n+\tif (ioeventfd-\u003etest_mem) {\n+\t\tif (trylock) {\n+\t\t\tif (!down_read_trylock(\u0026vdev-\u003ememory_lock))\n+\t\t\t\treturn 1; /* Lock contended, use thread */\n+\t\t} else {\n+\t\t\tdown_read(\u0026vdev-\u003ememory_lock);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Read the recovery state lock-free rather than under recovery_lock.\n+\t * This path runs from the virqfd cleanup workqueue, which is flushed\n+\t * from paths that take recovery_lock for reading, so blocking on it\n+\t * here would deadlock behind a queued writer.\n+\t *\n+\t * For a memory BAR, a blocked device still waits for a write already\n+\t * under way, through memory_lock. The lock is taken above before the\n+\t * flag is read, so a write which saw the flag clear is already\n+\t * holding the read side, and the blocker waits for it when it takes\n+\t * memory_lock for writing. An I/O port BAR takes no memory_lock, so\n+\t * a write which saw the flag clear can still land afterwards. Port\n+\t * writes are best effort here.\n+\t *\n+\t * A write can also be dropped for a short while after a reset has\n+\t * finished, since VFIO_DEVICE_RESET releases memory_lock before it\n+\t * retakes recovery_lock to unblock access. Closing that would mean\n+\t * taking recovery_lock inside memory_lock, which is the wrong way\n+\t * round.\n+\t *\n+\t * pci_recovery_device_open records that the recovery machinery is\n+\t * live, so it is only ever set for drivers which advertise support.\n+\t * Testing it unconditionally would drop every write for every other\n+\t * driver.\n+\t *\n+\t * The raw vfio_iowrite*() accessors below are used for the same\n+\t * reason. This path must not take recovery_lock.\n+\t */\n+\tif (vdev-\u003epci_recovery_supported \u0026\u0026\n+\t (!READ_ONCE(vdev-\u003epci_recovery_device_open) ||\n+\t READ_ONCE(vdev-\u003epci_recovery_access_blocked)))\n+\t\tgoto out_memory;\n+\n+\tif (ioeventfd-\u003etest_mem \u0026\u0026 !__vfio_pci_memory_enabled(vdev))\n+\t\tgoto out_memory;\n+\n \tswitch (ioeventfd-\u003ecount) {\n \tcase 1:\n-\t\tvfio_pci_core_iowrite8(ioeventfd-\u003evdev, test_mem,\n-\t\t\t\t ioeventfd-\u003edata, ioeventfd-\u003eaddr);\n+\t\tvfio_iowrite8(ioeventfd-\u003edata, ioeventfd-\u003eaddr);\n \t\tbreak;\n \tcase 2:\n-\t\tvfio_pci_core_iowrite16(ioeventfd-\u003evdev, test_mem,\n-\t\t\t\t\tioeventfd-\u003edata, ioeventfd-\u003eaddr);\n+\t\tvfio_iowrite16(ioeventfd-\u003edata, ioeventfd-\u003eaddr);\n \t\tbreak;\n \tcase 4:\n-\t\tvfio_pci_core_iowrite32(ioeventfd-\u003evdev, test_mem,\n-\t\t\t\t\tioeventfd-\u003edata, ioeventfd-\u003eaddr);\n+\t\tvfio_iowrite32(ioeventfd-\u003edata, ioeventfd-\u003eaddr);\n \t\tbreak;\n \tcase 8:\n-\t\tvfio_pci_core_iowrite64(ioeventfd-\u003evdev, test_mem,\n-\t\t\t\t\tioeventfd-\u003edata, ioeventfd-\u003eaddr);\n+\t\tvfio_iowrite64(ioeventfd-\u003edata, ioeventfd-\u003eaddr);\n \t\tbreak;\n \t}\n-}\n-\n-static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)\n-{\n-\tstruct vfio_pci_ioeventfd *ioeventfd = opaque;\n-\tstruct vfio_pci_core_device *vdev = ioeventfd-\u003evdev;\n-\n-\tif (ioeventfd-\u003etest_mem) {\n-\t\tif (!down_read_trylock(\u0026vdev-\u003ememory_lock))\n-\t\t\treturn 1; /* Lock contended, use thread */\n-\t\tif (!__vfio_pci_memory_enabled(vdev)) {\n-\t\t\tup_read(\u0026vdev-\u003ememory_lock);\n-\t\t\treturn 0;\n-\t\t}\n-\t}\n-\n-\tvfio_pci_ioeventfd_do_write(ioeventfd, false);\n \n+out_memory:\n \tif (ioeventfd-\u003etest_mem)\n \t\tup_read(\u0026vdev-\u003ememory_lock);\n-\n \treturn 0;\n }\n \n-static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)\n+static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)\n {\n-\tstruct vfio_pci_ioeventfd *ioeventfd = opaque;\n+\treturn vfio_pci_ioeventfd_do_write(opaque, true);\n+}\n \n-\tvfio_pci_ioeventfd_do_write(ioeventfd, ioeventfd-\u003etest_mem);\n+static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)\n+{\n+\tvfio_pci_ioeventfd_do_write(opaque, false);\n }\n \n int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,\ndiff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h\nindex 9a1674c152aa2..fe42089be3fc0 100644\n--- a/include/linux/vfio_pci_core.h\n+++ b/include/linux/vfio_pci_core.h\n@@ -95,6 +95,11 @@ static inline int vfio_pci_core_get_dmabuf_phys(\n }\n #endif\n \n+#define VFIO_PCI_RECOVERY_IN_PROGRESS\tBIT(0)\n+#define VFIO_PCI_RECOVERY_FROZEN\tBIT(1)\n+#define VFIO_PCI_RECOVERY_RESET\t\tBIT(2)\n+#define VFIO_PCI_RECOVERY_FAILED\tBIT(3)\n+\n struct vfio_pci_core_device {\n \tstruct vfio_device\tvdev;\n \tstruct pci_dev\t\t*pdev;\n@@ -129,6 +134,7 @@ struct vfio_pci_core_device {\n \tbool\t\t\tdisable_idle_d3:1;\n \tbool\t\t\tnointxmask:1;\n \tbool\t\t\tdisable_vga:1;\n+\tbool\t\t\tpci_recovery_supported:1;\n \t/* Flags modified at runtime - dedicated storage unit */\n \tbool\t\t\tneeds_reset;\n \tbool\t\t\tpm_intx_masked;\n@@ -139,6 +145,7 @@ struct vfio_pci_core_device {\n \tint\t\t\tioeventfds_nr;\n \tstruct vfio_pci_eventfd __rcu *err_trigger;\n \tstruct vfio_pci_eventfd __rcu *req_trigger;\n+\tstruct vfio_pci_eventfd __rcu *pci_recovery_trigger;\n \tstruct eventfd_ctx\t*pm_wake_eventfd_ctx;\n \tstruct list_head\tdummy_resources_list;\n \tstruct mutex\t\tioeventfds_lock;\n@@ -147,7 +154,60 @@ struct vfio_pci_core_device {\n \tstruct list_head\t\tsriov_pfs_item;\n \tstruct vfio_pci_core_device\t*sriov_pf_core_dev;\n \tstruct notifier_block\tnb;\n+\t/*\n+\t * Serializes host PCI error recovery with device access and the\n+\t * open/close lifecycle. recovery_lock nests outside memory_lock.\n+\t */\n+\tstruct rw_semaphore\trecovery_lock;\n \tstruct rw_semaphore\tmemory_lock;\n+\t/*\n+\t * PCI error recovery state, written under recovery_lock held for\n+\t * writing except where noted.\n+\t *\n+\t * Some readers cannot take recovery_lock. An interrupt handler cannot\n+\t * sleep. The ioeventfd write runs on a workqueue which is flushed with\n+\t * the lock held, so it would block behind a queued writer. A\n+\t * wait-queue condition cannot take it either. Some checks would refuse\n+\t * work against ordinary device traffic if they took it. Work deferred\n+\t * past the guard has none to take, since it reaches pci_bus_sem.\n+\t *\n+\t * Those readers use READ_ONCE() on pci_recovery_flags,\n+\t * pci_recovery_enabled, pci_recovery_access_blocked and\n+\t * pci_recovery_device_open instead. All of them fail safe. A stale\n+\t * read costs an extra refusal or retry, never an unguarded access.\n+\t *\n+\t * recovery_lock does not exclude those readers, so publish each field\n+\t * with a single store of its final value rather than clearing and\n+\t * then setting. That keeps the states a reader can observe to ones\n+\t * which are meaningful on their own.\n+\t *\n+\t * device_open and access_blocked mean different things. device_open\n+\t * says the device is open and its per-open state, vconfig included,\n+\t * is allocated. access_blocked says a recovery or reset is blocking\n+\t * access right now.\n+\t *\n+\t * access_blocked is only ever set while device_open is set. Nothing\n+\t * sets it without testing device_open first under recovery_lock, and\n+\t * close clears access_blocked before it clears device_open, so a\n+\t * block never outlives the open which created it. A reader which\n+\t * finds access_blocked set can rely on the per-open state being\n+\t * there with it.\n+\t */\n+\tu32\t\t\tpci_recovery_flags;\n+\tu64\t\t\tpci_recovery_sequence;\n+\t/* PCI_COMMAND value saved before recovery quiesces the device. */\n+\tu16\t\t\tpci_recovery_command;\n+\t/* Userspace enabled recovery for this device open. */\n+\tbool\t\t\tpci_recovery_enabled;\n+\t/* pci_recovery_command contains a restorable value. */\n+\tbool\t\t\tpci_recovery_command_valid;\n+\t/* A recovery or reset transaction is blocking physical access. */\n+\tbool\t\t\tpci_recovery_access_blocked;\n+\t/* Device initialization completed and close teardown has not started. */\n+\tbool\t\t\tpci_recovery_device_open;\n+\t/* May be set while recovery_lock is held for reading during ROM unmap. */\n+\tbool\t\t\tpci_recovery_rom_disable;\n+\twait_queue_head_t\tpci_recovery_wait;\n \tstruct list_head\tdmabufs;\n };\n \ndiff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h\nindex e41437fa17ad0..ce0cda2dcbbb1 100644\n--- a/include/uapi/linux/vfio.h\n+++ b/include/uapi/linux/vfio.h\n@@ -1555,6 +1555,75 @@ struct vfio_device_feature_zpci_err {\n \n #define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13\n \n+/*\n+ * Report host PCI error recovery state for this device.\n+ *\n+ * The sequence number is incremented at the start of each event and remains\n+ * unchanged for its subsequent state changes. Userspace can therefore\n+ * distinguish a new event from completion of the current one and detect\n+ * coalesced notifications. It restarts from zero each time recovery is\n+ * enabled, so it is only meaningful within one enabled period.\n+ *\n+ * ENABLED reports that userspace has enabled recovery.\n+ * CHANNEL_FROZEN records that recovery started with the PCI channel frozen.\n+ * DEVICE_RESET records that the host reset the device. FAILED records that\n+ * recovery did not complete successfully. Event status bits remain set after\n+ * IN_PROGRESS is cleared. A new event supersedes status from a previous\n+ * successful event. FAILED is terminal for the current device open and\n+ * remains set until the device is closed and reopened.\n+ *\n+ * Status bits may also be set while IN_PROGRESS is still set, describing the\n+ * event so far. Act on them once IN_PROGRESS is clear. Device access is\n+ * refused with -EIO until then.\n+ *\n+ * When DEVICE_RESET is reported the host reset the device, which tears down\n+ * the interrupt configuration the user had established. INTx, MSI and MSI-X\n+ * must be re-armed with VFIO_DEVICE_SET_IRQS before interrupts resume.\n+ *\n+ * VFIO_DEVICE_FEATURE_GET returns the current state and -1 in eventfd. GET is\n+ * never refused, including while recovery blocks device access, so that\n+ * userspace can read this state during an event. It can wait for a recovery\n+ * callback which is already running.\n+ *\n+ * IN_PROGRESS is not guaranteed to be observable. A recovery which needs no\n+ * device reset can complete within microseconds of the notification, before\n+ * userspace is scheduled, so a GET which follows the eventfd may already see\n+ * IN_PROGRESS clear. Userspace must treat a notification as \"an event\n+ * occurred\" and read the sequence number and the status bits to learn what\n+ * happened. It must not wait for IN_PROGRESS to appear set.\n+ *\n+ * VFIO_DEVICE_FEATURE_SET with a valid eventfd enables recovery\n+ * and installs the eventfd as a notification for recovery start and terminal\n+ * completion. SET with eventfd -1 disables recovery when none\n+ * is in progress and the latest event has not failed. SET returns -EBUSY when\n+ * any of those restrictions prevents the requested transition, including while\n+ * an explicit VFIO_DEVICE_RESET blocks device access, and -ENODEV if device\n+ * close has begun. flags and sequence must be zero for SET.\n+ *\n+ * This eventfd is separate from VFIO_PCI_ERR_IRQ_INDEX and additional to it.\n+ * VFIO_PCI_ERR_IRQ_INDEX keeps reporting errors as it does today whether or\n+ * not this feature is enabled, so a user of both receives two notifications\n+ * for one event.\n+ *\n+ * Enabling recovery does not recover an event which is already being handled\n+ * for this device. Such an event was declined before it started, so it\n+ * completes without notification and without status, even though the host may\n+ * reset the device as part of it. Enable recovery before errors occur rather\n+ * than in response to one.\n+ */\n+struct vfio_device_pci_error_recovery {\n+\t__u32 flags;\n+#define VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS\t(1U \u003c\u003c 0)\n+#define VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN\t(1U \u003c\u003c 1)\n+#define VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET\t(1U \u003c\u003c 2)\n+#define VFIO_PCI_ERROR_RECOVERY_FAILED\t\t(1U \u003c\u003c 3)\n+#define VFIO_PCI_ERROR_RECOVERY_ENABLED\t\t(1U \u003c\u003c 4)\n+\t__s32 eventfd;\n+\t__aligned_u64 sequence;\n+};\n+\n+#define VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY 14\n+\n /* -------- API for Type1 VFIO IOMMU -------- */\n \n /**\n"
} |
| 2/1 | 2026/09/01 22:28 | agent | patch-evaluator |
0mModel:core Args: null Results: {
"EnableConfigs": [
"VFIO",
"VFIO_PCI",
"VFIO_PCI_CORE",
"PCIEAER"
],
"FocusSymbols": [
"vfio_pci_core_ioctl_feature",
"vfio_pci_try_reset_function",
"vfio_pci_core_aer_err_detected",
"vfio_pci_core_aer_slot_reset",
"vfio_pci_core_aer_resume",
"vfio_pci_core_access_begin",
"vfio_pci_core_access_end",
"vfio_pci_config_rw_single",
"vfio_pci_bar_rw",
"vfio_pci_intx_recovery_start",
"vfio_pci_intx_recovery_finish"
],
"Reasoning": "The patch series adds host PCI error recovery support to the VFIO PCI core driver. It introduces the VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY device feature ioctl, implements PCIe AER error recovery callbacks (.error_detected, .slot_reset, .resume), updates access guards and locking across config/BAR access, IRQ handlers, resets, and huge page fault handling. These changes modify reachable driver/core logic and are suitable for fuzzing.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 9245578c8ff1dcd9e2b6e621a377e36f1e75be35
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 1 22:28:34 2026 +0000
syz-cluster: applied patch under review
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 830369ff878db..46544dbe70d01 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -139,6 +139,7 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
*/
vdev->nointxmask = nointxmask;
vdev->disable_idle_d3 = disable_idle_d3;
+ vdev->pci_recovery_supported = true;
#ifdef CONFIG_VFIO_PCI_VGA
vdev->disable_vga = disable_vga;
#endif
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 9914f3ac69aef..3365100acf216 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -99,6 +99,12 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = {
[PCI_EXT_CAP_ID_DVSEC] = 0xFF,
};
+/* What a config write asked for which has to wait for the access guard. */
+struct vfio_pci_config_deferred {
+ bool flr; /* a function-level reset */
+ bool power_up; /* a transition to D0 */
+};
+
/*
* Read/Write Permission Bits - one bit for each bit in capability
* Any field can be read if it exists, but what is read depends on
@@ -111,8 +117,17 @@ struct perm_bits {
u8 *write; /* writeable bits */
int (*readfn)(struct vfio_pci_core_device *vdev, int pos, int count,
struct perm_bits *perm, int offset, __le32 *val);
+ /*
+ * @deferred records work the write asked for which a writefn must not
+ * do itself. Both a reset method and a transition to D0 acquire
+ * pci_bus_sem, which AER already holds when it enters the driver, so
+ * doing either here would invert the lock order against recovery_lock.
+ * The dispatcher does them after dropping recovery_lock. Callers zero
+ * it, and a writefn only sets a field on a success return.
+ */
int (*writefn)(struct vfio_pci_core_device *vdev, int pos, int count,
- struct perm_bits *perm, int offset, __le32 val);
+ struct perm_bits *perm, int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred);
};
#define NO_VIRT 0
@@ -200,7 +215,8 @@ static int vfio_default_config_read(struct vfio_pci_core_device *vdev, int pos,
static int vfio_default_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
__le32 virt = 0, write = 0;
@@ -272,7 +288,8 @@ static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos,
/* Raw access skips any kind of virtualization */
static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
int ret;
@@ -299,7 +316,8 @@ static int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,
/* Virt access uses only virtualization */
static int vfio_virt_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
memcpy(vdev->vconfig + pos, &val, count);
return count;
@@ -563,7 +581,8 @@ static bool vfio_need_bar_restore(struct vfio_pci_core_device *vdev)
static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
struct pci_dev *pdev = vdev->pdev;
__le16 *virt_cmd;
@@ -613,7 +632,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
vfio_bar_restore(vdev);
}
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0) {
if (offset == PCI_COMMAND)
up_write(&vdev->memory_lock);
@@ -727,9 +747,11 @@ static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
@@ -738,8 +760,15 @@ static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
switch (le32_to_cpu(val) & PCI_PM_CTRL_STATE_MASK) {
case 0:
- state = PCI_D0;
- break;
+ /*
+ * Going to D0 reaches pci_set_full_power_state(),
+ * which takes pci_bus_sem through
+ * pcie_aspm_pm_state_change(). Leave it to the
+ * dispatcher. The lower states do not, so they run
+ * here.
+ */
+ deferred->power_up = true;
+ return count;
case 1:
state = PCI_D1;
break;
@@ -799,7 +828,8 @@ static int __init init_pci_cap_pm_perm(struct perm_bits *perm)
static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
struct pci_dev *pdev = vdev->pdev;
__le16 *paddr = (__le16 *)(vdev->vconfig + pos - offset + PCI_VPD_ADDR);
@@ -812,7 +842,8 @@ static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,
* of PCI_VPD_ADDR, then the PCI_VPD_ADDR_F bit is written and we
* have work to do.
*/
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0 || offset > PCI_VPD_ADDR + 1 ||
offset + count <= PCI_VPD_ADDR + 1)
return count;
@@ -881,21 +912,24 @@ static int __init init_pci_cap_pcix_perm(struct perm_bits *perm)
static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
__le16 *ctrl = (__le16 *)(vdev->vconfig + pos -
offset + PCI_EXP_DEVCTL);
int readrq = le16_to_cpu(*ctrl) & PCI_EXP_DEVCTL_READRQ;
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
/*
* The FLR bit is virtualized, if set and the device supports PCIe
- * FLR, issue a reset_function. Regardless, clear the bit, the spec
- * requires it to be always read as zero. NB, reset_function might
- * not use a PCIe FLR, we don't have that level of granularity.
+ * FLR, request a function reset once recovery_lock has been
+ * released. Regardless, clear the bit, the spec requires it to be
+ * always read as zero. NB, reset_function might not use a PCIe FLR,
+ * we don't have that level of granularity.
*/
if (*ctrl & cpu_to_le16(PCI_EXP_DEVCTL_BCR_FLR)) {
u32 cap;
@@ -907,14 +941,8 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
pos - offset + PCI_EXP_DEVCAP,
&cap);
- if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
- vfio_pci_zap_and_down_write_memory_lock(vdev);
- vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
- if (__vfio_pci_memory_enabled(vdev))
- vfio_pci_dma_buf_move(vdev, false);
- up_write(&vdev->memory_lock);
- }
+ if (!ret && (cap & PCI_EXP_DEVCAP_FLR))
+ deferred->flr = true;
}
/*
@@ -968,19 +996,22 @@ static int __init init_pci_cap_exp_perm(struct perm_bits *perm)
static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
u8 *ctrl = vdev->vconfig + pos - offset + PCI_AF_CTRL;
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
/*
* The FLR bit is virtualized, if set and the device supports AF
- * FLR, issue a reset_function. Regardless, clear the bit, the spec
- * requires it to be always read as zero. NB, reset_function might
- * not use an AF FLR, we don't have that level of granularity.
+ * FLR, request a function reset once recovery_lock has been
+ * released. Regardless, clear the bit, the spec requires it to be
+ * always read as zero. NB, reset_function might not use an AF FLR,
+ * we don't have that level of granularity.
*/
if (*ctrl & PCI_AF_CTRL_FLR) {
u8 cap;
@@ -992,14 +1023,8 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
pos - offset + PCI_AF_CAP,
&cap);
- if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
- vfio_pci_zap_and_down_write_memory_lock(vdev);
- vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
- if (__vfio_pci_memory_enabled(vdev))
- vfio_pci_dma_buf_move(vdev, false);
- up_write(&vdev->memory_lock);
- }
+ if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP))
+ deferred->flr = true;
}
return count;
@@ -1168,9 +1193,11 @@ static int vfio_msi_config_read(struct vfio_pci_core_device *vdev, int pos,
static int vfio_msi_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
@@ -1889,6 +1916,8 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
struct perm_bits *perm;
__le32 val = 0;
int cap_start = 0, offset;
+ int access_ret;
+ struct vfio_pci_config_deferred deferred = {};
u8 cap_id;
ssize_t ret;
@@ -1957,14 +1986,42 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
if (copy_from_user(&val, buf, count))
return -EFAULT;
- ret = perm->writefn(vdev, *ppos, count, perm, offset, val);
+ access_ret = vfio_pci_core_access_begin(vdev);
+ if (access_ret)
+ return access_ret;
+ ret = perm->writefn(vdev, *ppos, count, perm, offset, val,
+ &deferred);
+ vfio_pci_core_access_end(vdev);
+ if (ret < 0)
+ return ret;
+ /*
+ * Both of these take pci_bus_sem, so run them with the access
+ * guard dropped. The reset re-checks the recovery state for
+ * itself. The power up does not, so check it here.
+ *
+ * Both are best effort, as the guest-requested FLR has always
+ * been. The result is not reported back through the config
+ * write. Without recovery enabled the only failure is -EAGAIN
+ * from device lock contention, exactly as before. With it they
+ * are dropped while a recovery or reset transaction is in
+ * flight, which leaves the device in D0 and reset anyway.
+ */
+ if (deferred.power_up &&
+ !(vdev->pci_recovery_supported &&
+ READ_ONCE(vdev->pci_recovery_access_blocked)))
+ vfio_lock_and_set_power_state(vdev, PCI_D0);
+ if (deferred.flr)
+ vfio_pci_try_reset_function(vdev, false);
} else {
- if (perm->readfn) {
+ access_ret = vfio_pci_core_access_begin(vdev);
+ if (access_ret)
+ return access_ret;
+ if (perm->readfn)
ret = perm->readfn(vdev, *ppos, count,
perm, offset, &val);
- if (ret < 0)
- return ret;
- }
+ vfio_pci_core_access_end(vdev);
+ if (ret < 0)
+ return ret;
if (copy_to_user(buf, &val, count))
return -EFAULT;
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6757054e9d875..c1ea3c868fc5e 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -372,15 +372,21 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
struct eventfd_ctx *efdctx)
{
+ int ret;
+
/*
* The vdev power related flags are protected with 'memory_lock'
* semaphore.
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
vfio_pci_zap_and_down_write_memory_lock(vdev);
vfio_pci_dma_buf_move(vdev, true);
if (vdev->pm_runtime_engaged) {
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
return -EINVAL;
}
@@ -388,6 +394,7 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
vdev->pm_wake_eventfd_ctx = efdctx;
pm_runtime_put_noidle(&vdev->pdev->dev);
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
return 0;
}
@@ -483,7 +490,11 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
* already signaled the eventfd and exited low power mode itself.
* pm_runtime_engaged protects the redundant call here.
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
vfio_pci_runtime_pm_exit(vdev);
+ vfio_pci_core_access_end(vdev);
return 0;
}
@@ -591,10 +602,23 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {
int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
{
struct pci_dev *pdev = vdev->pdev;
+ bool supported = vdev->pci_recovery_supported;
int ret;
u16 cmd;
u8 msix_pos;
+ if (supported) {
+ down_write(&vdev->recovery_lock);
+ if (pci_dev_is_disconnected(pdev)) {
+ up_write(&vdev->recovery_lock);
+ return -ENODEV;
+ }
+
+ vdev->pci_recovery_command_valid = false;
+ WRITE_ONCE(vdev->pci_recovery_device_open, false);
+ up_write(&vdev->recovery_lock);
+ }
+
if (!vdev->disable_idle_d3) {
ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0)
@@ -815,7 +839,40 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
}
EXPORT_SYMBOL_GPL(vfio_pci_core_disable);
-void vfio_pci_core_close_device(struct vfio_device *core_vdev)
+static void vfio_pci_core_prepare_close(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return;
+
+ down_write(&vdev->recovery_lock);
+ WRITE_ONCE(vdev->pci_recovery_enabled, false);
+ vdev->pci_recovery_command_valid = false;
+ /*
+ * Clear access_blocked before device_open, so a lock-free reader
+ * never sees it set on a device which is no longer open. A
+ * transaction which is still running cannot clear it once
+ * device_open is gone, and paths which refuse work on a blocked
+ * device would then refuse it for good.
+ */
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ WRITE_ONCE(vdev->pci_recovery_device_open, false);
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+
+ /*
+ * Publish the closing state and drop recovery_lock before any
+ * teardown. Recovery is disabled and its state cleared, so
+ * slot_reset() and resume() become no-ops and a later
+ * error_detected() only follows the legacy notification path.
+ * Holding the lock across vfio_pci_core_disable() protects nothing
+ * and inverts the lock order. disable() reaches pci_reset_bus(),
+ * which takes pci_bus_sem, while error_detected() takes
+ * recovery_lock from under pci_bus_sem.
+ */
+ up_write(&vdev->recovery_lock);
+ wake_up_all(&vdev->pci_recovery_wait);
+}
+
+static void vfio_pci_core_finish_close(struct vfio_device *core_vdev)
{
struct vfio_pci_core_device *vdev =
container_of(core_vdev, struct vfio_pci_core_device, vdev);
@@ -836,8 +893,21 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
mutex_lock(&vdev->igate);
vfio_pci_eventfd_replace_locked(vdev, &vdev->err_trigger, NULL);
vfio_pci_eventfd_replace_locked(vdev, &vdev->req_trigger, NULL);
+ if (vdev->pci_recovery_supported)
+ vfio_pci_eventfd_replace_locked(vdev,
+ &vdev->pci_recovery_trigger,
+ NULL);
mutex_unlock(&vdev->igate);
}
+
+void vfio_pci_core_close_device(struct vfio_device *core_vdev)
+{
+ struct vfio_pci_core_device *vdev =
+ container_of(core_vdev, struct vfio_pci_core_device, vdev);
+
+ vfio_pci_core_prepare_close(vdev);
+ vfio_pci_core_finish_close(core_vdev);
+}
EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);
void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
@@ -852,6 +922,18 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
vdev->sriov_pf_core_dev->vf_token->users++;
mutex_unlock(&vdev->sriov_pf_core_dev->vf_token->lock);
}
+
+ if (vdev->pci_recovery_supported) {
+ down_write(&vdev->recovery_lock);
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+ vdev->pci_recovery_sequence = 0;
+ WRITE_ONCE(vdev->pci_recovery_enabled, false);
+ /* Close clears this too. Start unblocked either way. */
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ WRITE_ONCE(vdev->pci_recovery_device_open, true);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, false);
+ up_write(&vdev->recovery_lock);
+ }
}
EXPORT_SYMBOL_GPL(vfio_pci_core_finish_enable);
@@ -1101,7 +1183,13 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
return ret;
}
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret) {
+ kfree(caps.buf);
+ return ret;
+ }
ret = vfio_pci_info_atomic_cap(vdev, &caps);
+ vfio_pci_core_access_end(vdev);
if (ret && ret != -ENODEV) {
pci_warn(vdev->pdev,
"Failed to setup AtomicOps info capability\n");
@@ -1177,6 +1265,9 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
* Check ROM content is valid. Need to enable memory
* decode for ROM access in pci_map_rom().
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
cmd = vfio_pci_memory_lock_and_enable(vdev);
io = pci_map_rom(pdev, &size);
if (io) {
@@ -1187,6 +1278,7 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
pci_unmap_rom(pdev, io);
}
vfio_pci_memory_unlock_and_restore(vdev, cmd);
+ vfio_pci_core_access_end(vdev);
} else if (pdev->rom && pdev->romlen) {
info->flags = VFIO_REGION_INFO_FLAG_READ;
/* Report BAR size as power of two. */
@@ -1242,11 +1334,29 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
}
EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);
+/*
+ * Which IRQ indexes can reach the device. ERR and REQ are software only.
+ * An index added later gets no access guard until it is listed here.
+ */
+static bool vfio_pci_irq_index_is_device(u32 index)
+{
+ switch (index) {
+ case VFIO_PCI_INTX_IRQ_INDEX:
+ case VFIO_PCI_MSI_IRQ_INDEX:
+ case VFIO_PCI_MSIX_IRQ_INDEX:
+ return true;
+ default:
+ return false;
+ }
+}
+
static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
struct vfio_irq_info __user *arg)
{
unsigned long minsz = offsetofend(struct vfio_irq_info, count);
struct vfio_irq_info info;
+ bool device_irq;
+ int ret;
if (copy_from_user(&info, arg, minsz))
return -EFAULT;
@@ -1265,7 +1375,15 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
info.flags = VFIO_IRQ_INFO_EVENTFD;
+ device_irq = vfio_pci_irq_index_is_device(info.index);
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ }
info.count = vfio_pci_get_irq_count(vdev, info.index);
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
info.flags |=
@@ -1282,13 +1400,23 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
unsigned long minsz = offsetofend(struct vfio_irq_set, count);
struct vfio_irq_set hdr;
u8 *data = NULL;
+ bool device_irq;
int max, ret = 0;
size_t data_size = 0;
if (copy_from_user(&hdr, arg, minsz))
return -EFAULT;
+ device_irq = vfio_pci_irq_index_is_device(hdr.index);
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ }
max = vfio_pci_get_irq_count(vdev, hdr.index);
+ /* Dropped for the user copy below, which can fault under userfaultfd. */
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
ret = vfio_set_irqs_validate_and_prepare(&hdr, max, VFIO_PCI_NUM_IRQS,
&data_size);
@@ -1301,25 +1429,107 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
return PTR_ERR(data);
}
+ /*
+ * Interrupt teardown reaches vfio_virqfd_disable(), which flushes the
+ * global virqfd cleanup workqueue, so recovery_lock is held here for
+ * as long as work queued by any vfio device takes. Shutdown work waits
+ * for its inject worker, and an ioeventfd inject takes that device's
+ * memory_lock, so the wait can last as long as a reset there. That is
+ * only a wait. Nothing on that workqueue takes recovery_lock, which is
+ * why the ioeventfd write path reads the recovery state without it. A
+ * callback there which used the vfio_pci_core_iowrite*() accessors
+ * would break that and deadlock against a queued writer.
+ */
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ goto out_free;
+ }
mutex_lock(&vdev->igate);
ret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index, hdr.start,
hdr.count, data);
mutex_unlock(&vdev->igate);
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
+out_free:
kfree(data);
return ret;
}
-static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
- void __user *arg)
+/*
+ * Complete a ROM unmap which could not disable decode through config space.
+ * Call once whatever blocked access has finished. A closed device is skipped.
+ * It runs without recovery_lock, and close puts the device back through reset
+ * and config restore.
+ *
+ * The IORESOURCE_ROM_ENABLE test is what pci_unmap_rom() would have done.
+ * A ROM which firmware left enabled is not ours to turn off.
+ */
+static void vfio_pci_recovery_rom_disable(struct vfio_pci_core_device *vdev)
{
+ struct pci_dev *pdev = vdev->pdev;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ if (!vdev->pci_recovery_device_open ||
+ !READ_ONCE(vdev->pci_recovery_rom_disable))
+ return;
+
+ if (!(pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_ENABLE))
+ pci_disable_rom(pdev);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, false);
+}
+
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,
+ bool reset_power_state)
+{
+ struct pci_dev *pdev = vdev->pdev;
+ bool enabled = false;
+ bool supported = vdev->pci_recovery_supported;
int ret;
- if (!vdev->reset_works)
- return -EINVAL;
+ /*
+ * Claim the device against recovery before resetting it. The PCI
+ * error callbacks hold recovery_lock for their whole body, so taking
+ * it for writing here waits for one already running, and
+ * access_blocked keeps a later one away while the lock is dropped.
+ */
+ if (supported) {
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open) {
+ ret = -ENODEV;
+ goto out_recovery;
+ }
+
+ enabled = vdev->pci_recovery_enabled;
+ /*
+ * Only claim access_blocked when recovery is enabled.
+ * error_detected() returns early for a device which has not
+ * enabled it, so there is nothing to exclude, and claiming it
+ * anyway would fail the second of two concurrent
+ * VFIO_DEVICE_RESET calls with -EBUSY.
+ */
+ if (enabled) {
+ if (vdev->pci_recovery_access_blocked) {
+ ret = -EBUSY;
+ goto out_recovery;
+ }
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, true);
+ }
+ up_write(&vdev->recovery_lock);
+ }
+
+ /*
+ * On a device which supports recovery, taking recovery_lock for
+ * writing above waited for anything already past its access check,
+ * and if recovery is enabled access_blocked keeps new ones out. Do
+ * not hold recovery_lock while taking memory_lock or running a reset
+ * method, since a reset can take pci_bus_sem.
+ */
vfio_pci_zap_and_down_write_memory_lock(vdev);
/*
@@ -1331,17 +1541,54 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
* reset without restoring the original state (saved locally in
* 'vdev->pm_save').
*/
- vfio_pci_set_power_state(vdev, PCI_D0);
+ if (reset_power_state)
+ vfio_pci_set_power_state(vdev, PCI_D0);
vfio_pci_dma_buf_move(vdev, true);
- ret = pci_try_reset_function(vdev->pdev);
+ ret = pci_try_reset_function(pdev);
if (__vfio_pci_memory_enabled(vdev))
vfio_pci_dma_buf_move(vdev, false);
up_write(&vdev->memory_lock);
+ if (enabled) {
+ down_write(&vdev->recovery_lock);
+ /*
+ * An error callback can have started an event while the lock
+ * was down. Leave the state to it. Only unblock access for a
+ * reset which is still the one holding it.
+ */
+ if (vdev->pci_recovery_device_open &&
+ !(vdev->pci_recovery_flags & (VFIO_PCI_RECOVERY_IN_PROGRESS |
+ VFIO_PCI_RECOVERY_FAILED))) {
+ vfio_pci_recovery_rom_disable(vdev);
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ vfio_pci_intx_recovery_finish(vdev);
+ }
+ up_write(&vdev->recovery_lock);
+ /*
+ * Access is blocked for the length of the reset, so anything
+ * waiting for it to clear has to be woken here. A later patch
+ * adds the BAR fault path which waits on this.
+ */
+ wake_up_all(&vdev->pci_recovery_wait);
+ }
+
+ return ret;
+
+out_recovery:
+ up_write(&vdev->recovery_lock);
return ret;
}
+static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
+ void __user *arg)
+{
+ if (!vdev->reset_works)
+ return -EINVAL;
+
+ return vfio_pci_try_reset_function(vdev, true);
+}
+
static int vfio_pci_ioctl_get_pci_hot_reset_info(
struct vfio_pci_core_device *vdev,
struct vfio_pci_hot_reset_info __user *arg)
@@ -1607,6 +1854,106 @@ static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,
return 0;
}
+static int
+vfio_pci_core_feature_error_recovery(struct vfio_pci_core_device *vdev, u32 flags,
+ struct vfio_device_pci_error_recovery __user *arg,
+ size_t argsz)
+{
+ struct vfio_device_pci_error_recovery state = { .eventfd = -1 };
+ struct eventfd_ctx *ctx = NULL;
+ bool enable;
+ int ret;
+
+ if (!vdev->pci_recovery_supported)
+ return -ENOTTY;
+
+ ret = vfio_check_feature(flags, argsz,
+ VFIO_DEVICE_FEATURE_GET |
+ VFIO_DEVICE_FEATURE_SET, sizeof(state));
+ if (ret != 1)
+ return ret;
+
+ if (flags & VFIO_DEVICE_FEATURE_GET) {
+ down_read(&vdev->recovery_lock);
+ if (vdev->pci_recovery_enabled)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_ENABLED;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FROZEN)
+ state.flags |=
+ VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_RESET)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_FAILED;
+ state.sequence = vdev->pci_recovery_sequence;
+ up_read(&vdev->recovery_lock);
+
+ if (copy_to_user(arg, &state, sizeof(state)))
+ return -EFAULT;
+ return 0;
+ }
+
+ if (copy_from_user(&state, arg, sizeof(state)))
+ return -EFAULT;
+ if (state.flags || state.sequence || state.eventfd < -1)
+ return -EINVAL;
+
+ enable = state.eventfd >= 0;
+ if (enable) {
+ ctx = eventfd_ctx_fdget(state.eventfd);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+ }
+
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open) {
+ ret = -ENODEV;
+ goto out_unlock;
+ }
+ if (vdev->pci_recovery_access_blocked) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+
+ if (!enable &&
+ (vdev->pci_recovery_flags &
+ (VFIO_PCI_RECOVERY_IN_PROGRESS | VFIO_PCI_RECOVERY_FAILED))) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+
+ mutex_lock(&vdev->igate);
+ ret = vfio_pci_eventfd_replace_locked(vdev,
+ &vdev->pci_recovery_trigger,
+ ctx);
+ mutex_unlock(&vdev->igate);
+ if (ret)
+ goto out_unlock;
+
+ WRITE_ONCE(vdev->pci_recovery_enabled, enable);
+ /*
+ * Start each enabled period from a clear state, so a sequence number
+ * and the status bits beside it always describe an event this
+ * eventfd was notified of. Nothing is in flight to lose. A
+ * transaction holds access_blocked, which failed this call with
+ * -EBUSY above.
+ *
+ * access_blocked itself is not cleared here, so userspace can never
+ * disable its way out of a block.
+ */
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+ vdev->pci_recovery_sequence = 0;
+ vdev->pci_recovery_command_valid = false;
+
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (ret && ctx)
+ eventfd_ctx_put(ctx);
+
+ return ret;
+}
+
int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
void __user *arg, size_t argsz)
{
@@ -1627,6 +1974,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
case VFIO_DEVICE_FEATURE_ZPCI_ERROR:
return vfio_pci_zdev_feature_err(device, flags, arg, argsz);
+
+ case VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY:
+ return vfio_pci_core_feature_error_recovery(vdev, flags, arg, argsz);
default:
return -ENOTTY;
}
@@ -1642,6 +1992,24 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,
if (index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
return -EINVAL;
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ vfio_pci_core_access_end(vdev);
+
+ /*
+ * Resume with the guard dropped. A resume takes pci_bus_sem, through
+ * pcie_aspm_pm_state_change() and, from D3cold, through
+ * pci_bridge_wait_for_secondary_bus(). The error callbacks take
+ * recovery_lock from under pci_bus_sem, so holding it here would
+ * invert the order.
+ *
+ * The check above only avoids waking a device whose access is already
+ * blocked. A recovery which starts in between is not excluded, and
+ * does not need to be. pcie_do_recovery() has already resumed every
+ * device under the bridge and holds the reference until it finishes.
+ * The region access below takes the guard for itself.
+ */
ret = pm_runtime_resume_and_get(&vdev->pdev->dev);
if (ret) {
pci_info_ratelimited(vdev->pdev, "runtime resume failed %d\n",
@@ -1680,6 +2048,27 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,
return ret;
}
+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return 0;
+
+ down_read(&vdev->recovery_lock);
+ if (unlikely(!vdev->pci_recovery_device_open ||
+ vdev->pci_recovery_access_blocked)) {
+ up_read(&vdev->recovery_lock);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev)
+{
+ if (vdev->pci_recovery_supported)
+ up_read(&vdev->recovery_lock);
+}
+
ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,
size_t count, loff_t *ppos)
{
@@ -1776,6 +2165,127 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
}
EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
+/*
+ * Whether a fault which found access blocked is worth retrying. Read
+ * without recovery_lock, so it is only a hint about when to look again.
+ * vfio_pci_fault_trylock_once() takes the lock and decides. Read the flags
+ * once so the two tests below see the same value. Every writer which can
+ * make this true wakes pci_recovery_wait.
+ */
+static bool vfio_pci_recovery_done(struct vfio_pci_core_device *vdev)
+{
+ u32 flags = READ_ONCE(vdev->pci_recovery_flags);
+
+ if (!READ_ONCE(vdev->pci_recovery_device_open))
+ return true;
+ if (flags & VFIO_PCI_RECOVERY_IN_PROGRESS)
+ return false;
+ if (flags & VFIO_PCI_RECOVERY_FAILED)
+ return true;
+ return !READ_ONCE(vdev->pci_recovery_access_blocked);
+}
+
+static int vfio_pci_wait_for_recovery(struct vfio_pci_core_device *vdev)
+{
+ return wait_event_killable(vdev->pci_recovery_wait,
+ vfio_pci_recovery_done(vdev));
+}
+
+/* What one look at the recovery state says the fault should do. */
+enum vfio_pci_fault_action {
+ VFIO_PCI_FAULT_PROCEED, /* returns with memory_lock held */
+ VFIO_PCI_FAULT_WAIT, /* recovery is running, may still recover */
+ VFIO_PCI_FAULT_FAIL, /* closed, or failed for good */
+};
+
+static enum vfio_pci_fault_action
+vfio_pci_fault_trylock_once(struct vfio_pci_core_device *vdev)
+{
+ enum vfio_pci_fault_action action;
+
+ down_read(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open ||
+ (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED)) {
+ action = VFIO_PCI_FAULT_FAIL;
+ } else if (vdev->pci_recovery_access_blocked) {
+ /*
+ * Blocked for a reason which still ends: a recovery which has
+ * not failed, or a function reset. Test FAILED above rather
+ * than IN_PROGRESS here, so a fault does not fail for good
+ * while a reset is running, or in the window where
+ * error_detected() has blocked access but not yet published
+ * the flags.
+ */
+ action = VFIO_PCI_FAULT_WAIT;
+ } else {
+ down_read(&vdev->memory_lock);
+ action = VFIO_PCI_FAULT_PROCEED;
+ }
+ up_read(&vdev->recovery_lock);
+
+ return action;
+}
+
+/*
+ * Return true with memory_lock held for a fault that may proceed. Otherwise
+ * return false with @ret set to the result the fault handler should return.
+ */
+static bool vfio_pci_core_fault_trylock(struct vfio_pci_core_device *vdev,
+ struct vm_fault *vmf,
+ vm_fault_t *ret)
+{
+ if (!vdev->pci_recovery_supported) {
+ down_read(&vdev->memory_lock);
+ return true;
+ }
+
+ switch (vfio_pci_fault_trylock_once(vdev)) {
+ case VFIO_PCI_FAULT_PROCEED:
+ return true;
+ case VFIO_PCI_FAULT_FAIL:
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+ case VFIO_PCI_FAULT_WAIT:
+ break;
+ }
+
+ if (fault_flag_allow_retry_first(vmf->flags)) {
+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) {
+ *ret = VM_FAULT_RETRY;
+ return false;
+ }
+
+ if (!vfio_device_try_get_registration(&vdev->vdev)) {
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+ }
+
+ release_fault_lock(vmf);
+ vfio_pci_wait_for_recovery(vdev);
+ vfio_device_put_registration(&vdev->vdev);
+ *ret = VM_FAULT_RETRY;
+ return false;
+ }
+
+ /*
+ * The fault lock cannot be dropped here: either the caller did not
+ * allow a retry, or this fault has already used one. So wait with
+ * it held. It is not a deadlock. Recovery revokes mappings through
+ * unmap_mapping_range(), which never takes mmap_lock. The wait is
+ * killable.
+ */
+ if (vfio_pci_wait_for_recovery(vdev)) {
+ *ret = VM_FAULT_NOPAGE;
+ return false;
+ }
+
+ if (vfio_pci_fault_trylock_once(vdev) == VFIO_PCI_FAULT_PROCEED)
+ return true;
+
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+}
+
static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
unsigned int order)
{
@@ -1787,8 +2297,11 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
vm_fault_t ret = VM_FAULT_FALLBACK;
if (is_aligned_for_order(vma, addr, pfn, order)) {
- scoped_guard(rwsem_read, &vdev->memory_lock)
- ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+ if (!vfio_pci_core_fault_trylock(vdev, vmf, &ret))
+ return ret;
+
+ ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+ up_read(&vdev->memory_lock);
}
dev_dbg_ratelimited(&vdev->pdev->dev,
@@ -2196,7 +2709,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
if (ret && ret != -EOPNOTSUPP)
return ret;
INIT_LIST_HEAD(&vdev->dmabufs);
+ init_rwsem(&vdev->recovery_lock);
init_rwsem(&vdev->memory_lock);
+ init_waitqueue_head(&vdev->pci_recovery_wait);
xa_init(&vdev->ctx);
return 0;
@@ -2333,22 +2848,347 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev)
}
EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device);
+static void
+vfio_pci_signal_recovery_event(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_eventfd *eventfd;
+
+ rcu_read_lock();
+ eventfd = rcu_dereference(vdev->pci_recovery_trigger);
+ if (eventfd)
+ eventfd_signal(eventfd->ctx);
+ rcu_read_unlock();
+}
+
pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,
pci_channel_state_t state)
{
struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
struct vfio_pci_eventfd *eventfd;
+ pci_ers_result_t result = PCI_ERS_RESULT_CAN_RECOVER;
+ unsigned long irq_flags;
+ bool notify_recovery = false;
+ bool terminal = false;
+ bool nested;
+ u32 flags;
+ int ret;
+
+ if (!vdev->pci_recovery_supported ||
+ !READ_ONCE(vdev->pci_recovery_enabled))
+ goto out;
+
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_enabled)
+ goto out_unlock;
+ /*
+ * A failed device remains blocked until close and a new open have
+ * reinitialized it. A later bridge event cannot make the saved VFIO
+ * state valid again.
+ */
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED) {
+ result = PCI_ERS_RESULT_NONE;
+ goto out_unlock;
+ }
+
+ if (!vdev->pci_recovery_device_open) {
+ result = PCI_ERS_RESULT_NONE;
+ /*
+ * PCI core rebroadcasts permanent failure when subtree
+ * recovery fails. Complete an event which started before
+ * close so a later open is not permanently stuck on
+ * IN_PROGRESS.
+ */
+ if (state == pci_channel_io_perm_failure &&
+ (vdev->pci_recovery_flags &
+ VFIO_PCI_RECOVERY_IN_PROGRESS)) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ terminal = true;
+ }
+ goto out_unlock;
+ }
+
+ notify_recovery = true;
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, true);
+ /*
+ * A second event before resume() has finished the first joins the
+ * transaction already running rather than starting one. Keep its
+ * sequence number, the command word it saved before the device was
+ * quiesced, and any reset a slot_reset() in between recorded. Reading
+ * the command word again here would save the quiesced value, and
+ * restoring that leaves the device with bus mastering off.
+ */
+ nested = vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS;
+ if (!nested)
+ vdev->pci_recovery_command_valid = false;
+ vfio_pci_intx_recovery_start(vdev);
+ /*
+ * INTx hardirq and virqfd callbacks cannot take recovery_lock.
+ * For devices with per-function INTx masking, mask INTx while holding
+ * irqlock so a callback which passed its blocked-state check is drained
+ * before the temporary command value is installed. Devices without
+ * per-function masking were quiesced above through genirq.
+ */
+ spin_lock_irqsave(&vdev->irqlock, irq_flags);
+ ret = 0;
+ if (state == pci_channel_io_normal && vdev->pci_2_3 && !nested) {
+ u16 command;
+
+ ret = pci_read_config_word(pdev, PCI_COMMAND,
+ &vdev->pci_recovery_command);
+ /*
+ * A read from a device which has stopped responding succeeds
+ * and returns all ones. Writing that back would set every
+ * command bit, and saving it would restore them at the end.
+ */
+ if (!ret && PCI_POSSIBLE_ERROR(vdev->pci_recovery_command))
+ ret = -EIO;
+ if (!ret) {
+ command = (vdev->pci_recovery_command &
+ ~PCI_COMMAND_MASTER) |
+ PCI_COMMAND_INTX_DISABLE;
+ ret = pci_write_config_word(pdev, PCI_COMMAND, command);
+ }
+ if (!ret)
+ vdev->pci_recovery_command_valid = true;
+ }
+ spin_unlock_irqrestore(&vdev->irqlock, irq_flags);
+ vfio_pci_zap_and_down_write_memory_lock(vdev);
+ vfio_pci_dma_buf_move(vdev, true);
+
+ /*
+ * Allocate a sequence for a new transaction, and drop the flags the
+ * previous one left behind for userspace to read. A nested event adds
+ * to the flags already there. Each path below publishes the result in
+ * one store, so a lock-free reader never observes a cleared state that
+ * looks like successful completion.
+ */
+ flags = vdev->pci_recovery_flags;
+ if (!nested) {
+ if (++vdev->pci_recovery_sequence == 0)
+ vdev->pci_recovery_sequence++;
+ flags = 0;
+ }
+
+ if (state == pci_channel_io_perm_failure) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (flags | VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ result = PCI_ERS_RESULT_DISCONNECT;
+ terminal = true;
+ goto out_memory;
+ }
+
+ if (state == pci_channel_io_frozen) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ flags | VFIO_PCI_RECOVERY_IN_PROGRESS |
+ VFIO_PCI_RECOVERY_FROZEN);
+ result = PCI_ERS_RESULT_NEED_RESET;
+ goto out_memory;
+ }
+
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ flags | VFIO_PCI_RECOVERY_IN_PROGRESS);
+ if (ret)
+ goto out_failed;
+ if (vdev->pci_2_3 || nested)
+ goto out_memory;
+
+ ret = pci_read_config_word(pdev, PCI_COMMAND,
+ &vdev->pci_recovery_command);
+ if (ret)
+ goto out_failed;
+
+ if (PCI_POSSIBLE_ERROR(vdev->pci_recovery_command)) {
+ ret = -EIO;
+ goto out_failed;
+ }
+
+ ret = pci_write_config_word(pdev, PCI_COMMAND,
+ vdev->pci_recovery_command &
+ ~PCI_COMMAND_MASTER);
+ if (ret)
+ goto out_failed;
+
+ vdev->pci_recovery_command_valid = true;
+ goto out_memory;
+
+out_failed:
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags | VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ result = PCI_ERS_RESULT_NONE;
+ terminal = true;
+out_memory:
+ up_write(&vdev->memory_lock);
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (terminal)
+ wake_up_all(&vdev->pci_recovery_wait);
+
+out:
rcu_read_lock();
eventfd = rcu_dereference(vdev->err_trigger);
if (eventfd)
eventfd_signal(eventfd->ctx);
rcu_read_unlock();
+ if (notify_recovery)
+ vfio_pci_signal_recovery_event(vdev);
- return PCI_ERS_RESULT_CAN_RECOVER;
+ return result;
}
EXPORT_SYMBOL_GPL(vfio_pci_core_aer_err_detected);
+static pci_ers_result_t vfio_pci_core_aer_slot_reset(struct pci_dev *pdev)
+{
+ struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+ pci_ers_result_t result = PCI_ERS_RESULT_RECOVERED;
+ int ret = 0;
+
+ down_write(&vdev->recovery_lock);
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS) ||
+ !vdev->pci_recovery_device_open) {
+ up_write(&vdev->recovery_lock);
+ return PCI_ERS_RESULT_NONE;
+ }
+
+ /*
+ * Restore first. aer_root_reset() resets the link with
+ * PCI_RESET_NO_RESTORE, so on entry the BARs read as zero. Tearing
+ * down MSI-X before this would have pci_msix_shutdown() write through
+ * the stale table mapping to an address the device no longer decodes.
+ */
+ pci_restore_state(pdev);
+
+ /*
+ * Hold recovery_lock across the interrupt teardown.
+ * vfio_pci_core_disable() runs the same teardown on close without
+ * taking igate, and running the per-vector teardown twice frees the
+ * irq, the name and the eventfd context twice.
+ * vfio_pci_core_prepare_close() takes recovery_lock for writing
+ * before it, so holding it here keeps the two apart.
+ */
+ mutex_lock(&vdev->igate);
+ if (vdev->irq_type < VFIO_PCI_NUM_IRQS)
+ ret = vfio_pci_set_irqs_ioctl(vdev,
+ VFIO_IRQ_SET_DATA_NONE |
+ VFIO_IRQ_SET_ACTION_TRIGGER,
+ vdev->irq_type, 0, 0, NULL);
+ mutex_unlock(&vdev->igate);
+
+ if (ret) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ /*
+ * Vote NONE, not DISCONNECT. A DISCONNECT anywhere in the
+ * domain makes the core skip resume() for every device under
+ * the bridge and report permanent failure for all of them.
+ * Our interrupt teardown failing says nothing about the
+ * others, so record it locally and leave the domain verdict
+ * alone.
+ */
+ result = PCI_ERS_RESULT_NONE;
+ } else {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_RESET);
+ }
+
+ up_write(&vdev->recovery_lock);
+ /*
+ * Whoever clears IN_PROGRESS owes the wake and the event. resume()
+ * will not do it, since it bails once IN_PROGRESS is clear, and the
+ * core skips it altogether if the domain verdict is not RECOVERED.
+ * On success the transaction carries on and resume() does both.
+ */
+ if (ret) {
+ wake_up_all(&vdev->pci_recovery_wait);
+ vfio_pci_signal_recovery_event(vdev);
+ }
+
+ return result;
+}
+
+static void vfio_pci_core_aer_resume(struct pci_dev *pdev)
+{
+ struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+ unsigned long irq_flags;
+ bool notify_recovery = false;
+ u32 flags;
+ int ret = 0;
+
+ down_write(&vdev->recovery_lock);
+
+ /*
+ * Pay any deferred ROM disable before the in-progress check below,
+ * which a failed transaction has already cleared, or it would be
+ * lost.
+ */
+ vfio_pci_recovery_rom_disable(vdev);
+
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS))
+ goto out_unlock;
+
+ notify_recovery = true;
+ if (!vdev->pci_recovery_device_open) {
+ vdev->pci_recovery_command_valid = false;
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ vdev->pci_recovery_flags &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ goto out_unlock;
+ }
+
+ down_write(&vdev->memory_lock);
+ /*
+ * Restore the command word and clear access_blocked under irqlock.
+ * The INTx handler writes the same register through
+ * pci_check_and_mask_intx(), so it must not interleave with the
+ * restore, and it must not see access blocked cleared while the
+ * temporary command value is still installed.
+ *
+ * INTX_DISABLE comes from the INTx state rather than from the saved
+ * word, which can be older than the last mask. The replay below is
+ * what unmasks the line.
+ */
+ spin_lock_irqsave(&vdev->irqlock, irq_flags);
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_RESET) &&
+ vdev->pci_recovery_command_valid) {
+ u16 cmd = vdev->pci_recovery_command;
+
+ cmd = vfio_pci_intx_recovery_command(vdev, cmd);
+ ret = pci_write_config_word(pdev, PCI_COMMAND, cmd);
+ }
+ if (!ret)
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ spin_unlock_irqrestore(&vdev->irqlock, irq_flags);
+ if (!ret && __vfio_pci_memory_enabled(vdev))
+ vfio_pci_dma_buf_move(vdev, false);
+ up_write(&vdev->memory_lock);
+
+ vdev->pci_recovery_command_valid = false;
+ flags = vdev->pci_recovery_flags & ~VFIO_PCI_RECOVERY_IN_PROGRESS;
+ if (ret)
+ flags |= VFIO_PCI_RECOVERY_FAILED;
+ WRITE_ONCE(vdev->pci_recovery_flags, flags);
+ if (!ret)
+ vfio_pci_intx_recovery_finish(vdev);
+
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (notify_recovery) {
+ wake_up_all(&vdev->pci_recovery_wait);
+ vfio_pci_signal_recovery_event(vdev);
+ }
+}
+
int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
int nr_virtfn)
{
@@ -2421,6 +3261,8 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure);
const struct pci_error_handlers vfio_pci_core_err_handlers = {
.error_detected = vfio_pci_core_aer_err_detected,
+ .slot_reset = vfio_pci_core_aer_slot_reset,
+ .resume = vfio_pci_core_aer_resume,
};
EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
@@ -2565,6 +3407,27 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
break;
}
+ /*
+ * Between the AER callbacks memory_lock is not held, so refuse
+ * the reset on access_blocked as well. Read it rather than
+ * take recovery_lock, which would have to be released before
+ * pci_reset_bus() anyway since that reaches pci_bus_sem.
+ *
+ * Let a device which has failed for good through. Nothing is
+ * running on its behalf, and this reset covers the whole set,
+ * so refusing there would stop a healthy sibling from being
+ * reset because an unrelated device errored. The failed one
+ * stays blocked, since the reset does not unblock a device
+ * with FAILED set.
+ */
+ if (vdev->pci_recovery_supported &&
+ READ_ONCE(vdev->pci_recovery_access_blocked) &&
+ !(READ_ONCE(vdev->pci_recovery_flags) &
+ VFIO_PCI_RECOVERY_FAILED)) {
+ ret = -EBUSY;
+ break;
+ }
+
/*
* Take the memory write lock for each device and zap BAR
* mappings to prevent the user accessing the device while in
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d68..a54d199a72c92 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -243,12 +243,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
if (!get_dma_buf.nr_ranges || get_dma_buf.flags)
return -EINVAL;
- /*
- * For PCI the region_index is the BAR number like everything
- * else. Check that PCI resources have been claimed for it.
- */
- if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX ||
- IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index)))
+ /* For PCI the region_index is the BAR number like everything else. */
+ if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX)
return -ENODEV;
dma_ranges = memdup_array_user(&arg->dma_ranges, get_dma_buf.nr_ranges,
@@ -274,19 +270,30 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
priv->vdev = vdev;
priv->nr_ranges = get_dma_buf.nr_ranges;
priv->size = length;
+
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ goto err_free_phys;
+
+ /* Check that PCI resources have been claimed for the BAR. */
+ if (IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index))) {
+ ret = -ENODEV;
+ goto err_access;
+ }
+
ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
get_dma_buf.region_index,
priv->phys_vec, dma_ranges,
priv->nr_ranges);
if (ret)
- goto err_free_phys;
+ goto err_access;
kfree(dma_ranges);
dma_ranges = NULL;
if (!vfio_device_try_get_registration(&vdev->vdev)) {
ret = -ENODEV;
- goto err_free_phys;
+ goto err_access;
}
exp_info.ops = &vfio_pci_dmabuf_ops;
@@ -311,6 +318,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
dma_resv_unlock(priv->dmabuf->resv);
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
/*
* dma_buf_fd() consumes the reference, when the file closes the dmabuf
@@ -324,6 +332,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
err_dev_put:
vfio_device_put_registration(&vdev->vdev);
+err_access:
+ vfio_pci_core_access_end(vdev);
err_free_phys:
kfree(priv->phys_vec);
err_free_priv:
diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c
index 64f80f64ff57a..1981a64b6e181 100644
--- a/drivers/vfio/pci/vfio_pci_intrs.c
+++ b/drivers/vfio/pci/vfio_pci_intrs.c
@@ -29,6 +29,8 @@ struct vfio_pci_irq_ctx {
struct virqfd *mask;
char *name;
bool masked;
+ bool recovery_masked;
+ bool unmask_pending;
struct irq_bypass_producer producer;
};
@@ -49,6 +51,15 @@ static bool is_irq_none(struct vfio_pci_core_device *vdev)
vdev->irq_type == VFIO_PCI_MSIX_IRQ_INDEX);
}
+static bool vfio_pci_recovery_blocks_irq(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return false;
+
+ return READ_ONCE(vdev->pci_recovery_enabled) &&
+ READ_ONCE(vdev->pci_recovery_access_blocked);
+}
+
static
struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,
unsigned long index)
@@ -171,6 +182,16 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)
int ret = 0;
spin_lock_irqsave(&vdev->irqlock, flags);
+ /*
+ * Check for a blocked device before the INTx test below, so a blocked
+ * device is not touched through pci_intx(). @ctx is only valid when
+ * INTx is in use, so record the request only then.
+ */
+ if (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {
+ if (is_intx(vdev))
+ ctx->unmask_pending = true;
+ goto out_unlock;
+ }
/*
* Unmasking comes from ioctl or config, so again, have the
@@ -182,6 +203,9 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)
goto out_unlock;
}
+ ctx->unmask_pending = false;
+ ctx->recovery_masked = false;
+
if (ctx->masked && !vdev->virq_disabled) {
/*
* A pending interrupt here would immediately trigger,
@@ -220,6 +244,47 @@ void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev)
mutex_unlock(&vdev->igate);
}
+/*
+ * Mask INTx because recovery has blocked device access. Returns true if this
+ * call did the masking.
+ *
+ * Set @quiesce when recovery is masking the line itself rather than masking
+ * one delivered interrupt. Nothing is normally asserted at that point, and
+ * pci_check_and_mask_intx() only writes DisINTx when the status register says
+ * an interrupt is pending, so it would leave the line alone. The interrupt
+ * handler wants that test, since a false return there means the interrupt
+ * belongs to another device on a shared line. @quiesce also records that
+ * recovery is the one which must unmask. A masked interrupt which was
+ * delivered is the user's to unmask, exactly as outside recovery.
+ *
+ * Masking a pci_2_3 device goes through config space. If the error left
+ * config space unreadable the write has no effect and the line stays
+ * asserted, which is no worse than not trying. For a non-fatal error config
+ * space still works, and this is what keeps a shared line from storming while
+ * access is blocked.
+ */
+static bool vfio_pci_intx_mask_for_recovery(struct vfio_pci_core_device *vdev,
+ struct vfio_pci_irq_ctx *ctx,
+ bool quiesce)
+{
+ lockdep_assert_held(&vdev->irqlock);
+
+ if (ctx->masked)
+ return false;
+
+ if (!vdev->pci_2_3)
+ disable_irq_nosync(vdev->pdev->irq);
+ else if (quiesce)
+ pci_intx(vdev->pdev, 0);
+ else if (!pci_check_and_mask_intx(vdev->pdev))
+ return false;
+
+ ctx->masked = true;
+ if (quiesce)
+ ctx->recovery_masked = true;
+ return true;
+}
+
static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
{
struct vfio_pci_irq_ctx *ctx = dev_id;
@@ -228,6 +293,19 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
int ret = IRQ_NONE;
spin_lock_irqsave(&vdev->irqlock, flags);
+ if (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {
+ /*
+ * Mask rather than return IRQ_NONE with the line still
+ * asserted. For a shared pci_2_3 line an unhandled level
+ * interrupt storms until note_interrupt() disables the line
+ * for every device on it, not just this one.
+ */
+ if (vfio_pci_intx_mask_for_recovery(vdev, ctx, false))
+ ret = IRQ_HANDLED;
+ else if (ctx->masked && !vdev->pci_2_3)
+ ret = IRQ_HANDLED;
+ goto out_unlock;
+ }
if (!vdev->pci_2_3) {
disable_irq_nosync(vdev->pdev->irq);
@@ -239,6 +317,7 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
ret = IRQ_HANDLED;
}
+out_unlock:
spin_unlock_irqrestore(&vdev->irqlock, flags);
if (ret == IRQ_HANDLED)
@@ -247,6 +326,86 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
return ret;
}
+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_irq_ctx *ctx;
+ unsigned long flags;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ spin_lock_irqsave(&vdev->irqlock, flags);
+ if (!is_intx(vdev))
+ goto out_unlock;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (WARN_ON_ONCE(!ctx))
+ goto out_unlock;
+
+ vfio_pci_intx_mask_for_recovery(vdev, ctx, true);
+
+out_unlock:
+ spin_unlock_irqrestore(&vdev->irqlock, flags);
+}
+
+/*
+ * Replay the masking recovery did, and any unmask which arrived while it was
+ * blocked. Call this only after access_blocked has been cleared, or the
+ * replayed unmask is swallowed and recorded as pending again with nothing
+ * left to replay it.
+ */
+/*
+ * The command word saved before the quiesce can have INTX_DISABLE clear, but
+ * the INTx handler may have masked the line since. Keep the bit as the INTx
+ * state has it, so hardware and ctx->masked agree until
+ * vfio_pci_intx_recovery_finish() replays. Restoring the saved bit instead
+ * would unmask a line the handler still believes is masked, and a shared
+ * pci_2_3 line would then storm until note_interrupt() disables it.
+ */
+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,
+ u16 command)
+{
+ struct vfio_pci_irq_ctx *ctx;
+
+ lockdep_assert_held(&vdev->irqlock);
+
+ if (!is_intx(vdev))
+ return command;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (ctx && ctx->masked)
+ command |= PCI_COMMAND_INTX_DISABLE;
+
+ return command;
+}
+
+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_irq_ctx *ctx;
+ unsigned long flags;
+ bool replay = false;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ mutex_lock(&vdev->igate);
+ spin_lock_irqsave(&vdev->irqlock, flags);
+ if (!is_intx(vdev))
+ goto out_unlock;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (WARN_ON_ONCE(!ctx))
+ goto out_unlock;
+
+ replay = ctx->recovery_masked || ctx->unmask_pending;
+ ctx->recovery_masked = false;
+ ctx->unmask_pending = false;
+
+out_unlock:
+ spin_unlock_irqrestore(&vdev->irqlock, flags);
+ if (replay)
+ __vfio_pci_intx_unmask(vdev);
+ mutex_unlock(&vdev->igate);
+}
+
static int vfio_intx_enable(struct vfio_pci_core_device *vdev,
struct eventfd_ctx *trigger)
{
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234a2eb..5598e472da4bf 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -25,6 +25,10 @@ struct vfio_pci_ioeventfd {
bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);
void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev);
+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,
+ u16 command);
int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
struct vfio_pci_eventfd __rcu **peventfd,
@@ -41,6 +45,9 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
char __user *buf, size_t count, loff_t *ppos,
bool iswrite);
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,
+ bool reset_power_state);
+
ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
size_t count, loff_t *ppos, bool iswrite);
@@ -73,6 +80,9 @@ u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
u16 cmd);
+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev);
+
#ifdef CONFIG_VFIO_PCI_IGD
bool vfio_pci_is_intel_display(struct pci_dev *pdev);
int vfio_pci_igd_init(struct vfio_pci_core_device *vdev);
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 7f14dd46de17a..86fadc9999621 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -42,10 +42,17 @@
int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \
bool test_mem, u##size val, void __iomem *io) \
{ \
+ int ret; \
+ \
+ ret = vfio_pci_core_access_begin(vdev); \
+ if (ret) \
+ return ret; \
+ \
if (test_mem) { \
down_read(&vdev->memory_lock); \
if (!__vfio_pci_memory_enabled(vdev)) { \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
return -EIO; \
} \
} \
@@ -54,6 +61,7 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \
\
if (test_mem) \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
\
return 0; \
} \
@@ -68,10 +76,17 @@ VFIO_IOWRITE(64)
int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \
bool test_mem, u##size *val, void __iomem *io) \
{ \
+ int ret; \
+ \
+ ret = vfio_pci_core_access_begin(vdev); \
+ if (ret) \
+ return ret; \
+ \
if (test_mem) { \
down_read(&vdev->memory_lock); \
if (!__vfio_pci_memory_enabled(vdev)) { \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
return -EIO; \
} \
} \
@@ -80,6 +95,7 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \
\
if (test_mem) \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
\
return 0; \
} \
@@ -198,12 +214,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
}
EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
+/*
+ * Undo pci_map_rom(). The iounmap is always safe, but pci_disable_rom() is a
+ * config space write. If recovery has blocked access, do the iounmap now and
+ * record the disable, for whichever of resume() or the reset tail unblocks
+ * access again. recovery_lock spans the decision and the record so recovery
+ * cannot complete in between.
+ */
+static void vfio_pci_unmap_rom(struct vfio_pci_core_device *vdev,
+ void __iomem *io)
+{
+ struct pci_dev *pdev = vdev->pdev;
+
+ if (!vdev->pci_recovery_supported) {
+ pci_unmap_rom(pdev, io);
+ return;
+ }
+
+ down_read(&vdev->recovery_lock);
+ if (vdev->pci_recovery_device_open &&
+ !vdev->pci_recovery_access_blocked) {
+ pci_unmap_rom(pdev, io);
+ } else {
+ iounmap(io);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, true);
+ }
+ up_read(&vdev->recovery_lock);
+}
+
ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
size_t count, loff_t *ppos, bool iswrite)
{
struct pci_dev *pdev = vdev->pdev;
loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
int bar = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+ int ret;
size_t x_start = 0, x_end = 0;
resource_size_t end;
void __iomem *io;
@@ -230,7 +275,11 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
* filling large ROM BARs much faster.
*/
if (pci_resource_start(pdev, bar)) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
io = pci_map_rom(pdev, &x_start);
+ vfio_pci_core_access_end(vdev);
} else {
io = ioremap(pdev->rom, pdev->romlen);
x_start = pdev->romlen;
@@ -269,7 +318,7 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
if (bar == PCI_ROM_RESOURCE) {
if (pci_resource_start(pdev, bar))
- pci_unmap_rom(pdev, io);
+ vfio_pci_unmap_rom(vdev, io);
else
iounmap(io);
}
@@ -349,56 +398,85 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
}
#endif
-static void vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,
- bool test_mem)
+static int vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,
+ bool trylock)
{
+ struct vfio_pci_core_device *vdev = ioeventfd->vdev;
+
+ if (ioeventfd->test_mem) {
+ if (trylock) {
+ if (!down_read_trylock(&vdev->memory_lock))
+ return 1; /* Lock contended, use thread */
+ } else {
+ down_read(&vdev->memory_lock);
+ }
+ }
+
+ /*
+ * Read the recovery state lock-free rather than under recovery_lock.
+ * This path runs from the virqfd cleanup workqueue, which is flushed
+ * from paths that take recovery_lock for reading, so blocking on it
+ * here would deadlock behind a queued writer.
+ *
+ * For a memory BAR, a blocked device still waits for a write already
+ * under way, through memory_lock. The lock is taken above before the
+ * flag is read, so a write which saw the flag clear is already
+ * holding the read side, and the blocker waits for it when it takes
+ * memory_lock for writing. An I/O port BAR takes no memory_lock, so
+ * a write which saw the flag clear can still land afterwards. Port
+ * writes are best effort here.
+ *
+ * A write can also be dropped for a short while after a reset has
+ * finished, since VFIO_DEVICE_RESET releases memory_lock before it
+ * retakes recovery_lock to unblock access. Closing that would mean
+ * taking recovery_lock inside memory_lock, which is the wrong way
+ * round.
+ *
+ * pci_recovery_device_open records that the recovery machinery is
+ * live, so it is only ever set for drivers which advertise support.
+ * Testing it unconditionally would drop every write for every other
+ * driver.
+ *
+ * The raw vfio_iowrite*() accessors below are used for the same
+ * reason. This path must not take recovery_lock.
+ */
+ if (vdev->pci_recovery_supported &&
+ (!READ_ONCE(vdev->pci_recovery_device_open) ||
+ READ_ONCE(vdev->pci_recovery_access_blocked)))
+ goto out_memory;
+
+ if (ioeventfd->test_mem && !__vfio_pci_memory_enabled(vdev))
+ goto out_memory;
+
switch (ioeventfd->count) {
case 1:
- vfio_pci_core_iowrite8(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite8(ioeventfd->data, ioeventfd->addr);
break;
case 2:
- vfio_pci_core_iowrite16(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite16(ioeventfd->data, ioeventfd->addr);
break;
case 4:
- vfio_pci_core_iowrite32(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite32(ioeventfd->data, ioeventfd->addr);
break;
case 8:
- vfio_pci_core_iowrite64(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite64(ioeventfd->data, ioeventfd->addr);
break;
}
-}
-
-static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)
-{
- struct vfio_pci_ioeventfd *ioeventfd = opaque;
- struct vfio_pci_core_device *vdev = ioeventfd->vdev;
-
- if (ioeventfd->test_mem) {
- if (!down_read_trylock(&vdev->memory_lock))
- return 1; /* Lock contended, use thread */
- if (!__vfio_pci_memory_enabled(vdev)) {
- up_read(&vdev->memory_lock);
- return 0;
- }
- }
-
- vfio_pci_ioeventfd_do_write(ioeventfd, false);
+out_memory:
if (ioeventfd->test_mem)
up_read(&vdev->memory_lock);
-
return 0;
}
-static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)
+static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)
{
- struct vfio_pci_ioeventfd *ioeventfd = opaque;
+ return vfio_pci_ioeventfd_do_write(opaque, true);
+}
- vfio_pci_ioeventfd_do_write(ioeventfd, ioeventfd->test_mem);
+static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)
+{
+ vfio_pci_ioeventfd_do_write(opaque, false);
}
int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9a1674c152aa2..fe42089be3fc0 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -95,6 +95,11 @@ static inline int vfio_pci_core_get_dmabuf_phys(
}
#endif
+#define VFIO_PCI_RECOVERY_IN_PROGRESS BIT(0)
+#define VFIO_PCI_RECOVERY_FROZEN BIT(1)
+#define VFIO_PCI_RECOVERY_RESET BIT(2)
+#define VFIO_PCI_RECOVERY_FAILED BIT(3)
+
struct vfio_pci_core_device {
struct vfio_device vdev;
struct pci_dev *pdev;
@@ -129,6 +134,7 @@ struct vfio_pci_core_device {
bool disable_idle_d3:1;
bool nointxmask:1;
bool disable_vga:1;
+ bool pci_recovery_supported:1;
/* Flags modified at runtime - dedicated storage unit */
bool needs_reset;
bool pm_intx_masked;
@@ -139,6 +145,7 @@ struct vfio_pci_core_device {
int ioeventfds_nr;
struct vfio_pci_eventfd __rcu *err_trigger;
struct vfio_pci_eventfd __rcu *req_trigger;
+ struct vfio_pci_eventfd __rcu *pci_recovery_trigger;
struct eventfd_ctx *pm_wake_eventfd_ctx;
struct list_head dummy_resources_list;
struct mutex ioeventfds_lock;
@@ -147,7 +154,60 @@ struct vfio_pci_core_device {
struct list_head sriov_pfs_item;
struct vfio_pci_core_device *sriov_pf_core_dev;
struct notifier_block nb;
+ /*
+ * Serializes host PCI error recovery with device access and the
+ * open/close lifecycle. recovery_lock nests outside memory_lock.
+ */
+ struct rw_semaphore recovery_lock;
struct rw_semaphore memory_lock;
+ /*
+ * PCI error recovery state, written under recovery_lock held for
+ * writing except where noted.
+ *
+ * Some readers cannot take recovery_lock. An interrupt handler cannot
+ * sleep. The ioeventfd write runs on a workqueue which is flushed with
+ * the lock held, so it would block behind a queued writer. A
+ * wait-queue condition cannot take it either. Some checks would refuse
+ * work against ordinary device traffic if they took it. Work deferred
+ * past the guard has none to take, since it reaches pci_bus_sem.
+ *
+ * Those readers use READ_ONCE() on pci_recovery_flags,
+ * pci_recovery_enabled, pci_recovery_access_blocked and
+ * pci_recovery_device_open instead. All of them fail safe. A stale
+ * read costs an extra refusal or retry, never an unguarded access.
+ *
+ * recovery_lock does not exclude those readers, so publish each field
+ * with a single store of its final value rather than clearing and
+ * then setting. That keeps the states a reader can observe to ones
+ * which are meaningful on their own.
+ *
+ * device_open and access_blocked mean different things. device_open
+ * says the device is open and its per-open state, vconfig included,
+ * is allocated. access_blocked says a recovery or reset is blocking
+ * access right now.
+ *
+ * access_blocked is only ever set while device_open is set. Nothing
+ * sets it without testing device_open first under recovery_lock, and
+ * close clears access_blocked before it clears device_open, so a
+ * block never outlives the open which created it. A reader which
+ * finds access_blocked set can rely on the per-open state being
+ * there with it.
+ */
+ u32 pci_recovery_flags;
+ u64 pci_recovery_sequence;
+ /* PCI_COMMAND value saved before recovery quiesces the device. */
+ u16 pci_recovery_command;
+ /* Userspace enabled recovery for this device open. */
+ bool pci_recovery_enabled;
+ /* pci_recovery_command contains a restorable value. */
+ bool pci_recovery_command_valid;
+ /* A recovery or reset transaction is blocking physical access. */
+ bool pci_recovery_access_blocked;
+ /* Device initialization completed and close teardown has not started. */
+ bool pci_recovery_device_open;
+ /* May be set while recovery_lock is held for reading during ROM unmap. */
+ bool pci_recovery_rom_disable;
+ wait_queue_head_t pci_recovery_wait;
struct list_head dmabufs;
};
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index e41437fa17ad0..ce0cda2dcbbb1 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1555,6 +1555,75 @@ struct vfio_device_feature_zpci_err {
#define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13
+/*
+ * Report host PCI error recovery state for this device.
+ *
+ * The sequence number is incremented at the start of each event and remains
+ * unchanged for its subsequent state changes. Userspace can therefore
+ * distinguish a new event from completion of the current one and detect
+ * coalesced notifications. It restarts from zero each time recovery is
+ * enabled, so it is only meaningful within one enabled period.
+ *
+ * ENABLED reports that userspace has enabled recovery.
+ * CHANNEL_FROZEN records that recovery started with the PCI channel frozen.
+ * DEVICE_RESET records that the host reset the device. FAILED records that
+ * recovery did not complete successfully. Event status bits remain set after
+ * IN_PROGRESS is cleared. A new event supersedes status from a previous
+ * successful event. FAILED is terminal for the current device open and
+ * remains set until the device is closed and reopened.
+ *
+ * Status bits may also be set while IN_PROGRESS is still set, describing the
+ * event so far. Act on them once IN_PROGRESS is clear. Device access is
+ * refused with -EIO until then.
+ *
+ * When DEVICE_RESET is reported the host reset the device, which tears down
+ * the interrupt configuration the user had established. INTx, MSI and MSI-X
+ * must be re-armed with VFIO_DEVICE_SET_IRQS before interrupts resume.
+ *
+ * VFIO_DEVICE_FEATURE_GET returns the current state and -1 in eventfd. GET is
+ * never refused, including while recovery blocks device access, so that
+ * userspace can read this state during an event. It can wait for a recovery
+ * callback which is already running.
+ *
+ * IN_PROGRESS is not guaranteed to be observable. A recovery which needs no
+ * device reset can complete within microseconds of the notification, before
+ * userspace is scheduled, so a GET which follows the eventfd may already see
+ * IN_PROGRESS clear. Userspace must treat a notification as "an event
+ * occurred" and read the sequence number and the status bits to learn what
+ * happened. It must not wait for IN_PROGRESS to appear set.
+ *
+ * VFIO_DEVICE_FEATURE_SET with a valid eventfd enables recovery
+ * and installs the eventfd as a notification for recovery start and terminal
+ * completion. SET with eventfd -1 disables recovery when none
+ * is in progress and the latest event has not failed. SET returns -EBUSY when
+ * any of those restrictions prevents the requested transition, including while
+ * an explicit VFIO_DEVICE_RESET blocks device access, and -ENODEV if device
+ * close has begun. flags and sequence must be zero for SET.
+ *
+ * This eventfd is separate from VFIO_PCI_ERR_IRQ_INDEX and additional to it.
+ * VFIO_PCI_ERR_IRQ_INDEX keeps reporting errors as it does today whether or
+ * not this feature is enabled, so a user of both receives two notifications
+ * for one event.
+ *
+ * Enabling recovery does not recover an event which is already being handled
+ * for this device. Such an event was declined before it started, so it
+ * completes without notification and without status, even though the host may
+ * reset the device as part of it. Enable recovery before errors occur rather
+ * than in response to one.
+ */
+struct vfio_device_pci_error_recovery {
+ __u32 flags;
+#define VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS (1U << 0)
+#define VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN (1U << 1)
+#define VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET (1U << 2)
+#define VFIO_PCI_ERROR_RECOVERY_FAILED (1U << 3)
+#define VFIO_PCI_ERROR_RECOVERY_ENABLED (1U << 4)
+ __s32 eventfd;
+ __aligned_u64 sequence;
+};
+
+#define VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY 14
+
/* -------- API for Type1 VFIO IOMMU -------- */
/**
|
| 3/2 | 2026/09/01 22:28 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 31515 output: 34 thoughts: 360 Thoughts: **Analyzing the VFIO PCI Patch** Okay, let's dive into this patch. Looks like we're getting PCI error recovery support added to the VFIO PCI core. Seeing modifications across the `drivers/vfio/pci/` directory, specifically in `vfio_pci.c`, `vfio_pci_config.c`, `vfio_pci_core.c`, `vfio_pci_dmabuf.c`, `vfio_pci_intrs.c`, `vfio_pci_priv.h`, and `vfio_pci_rdwr.c`, plus the related headers. This indicates a fairly widespread change, touching many aspects of the VFIO PCI implementation. So, we're enabling `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY`, which means we're adding AER recovery handlers, like `vfio_pci_core_aer_slot_reset` and `vfio_pci_core_aer_resume`, which is good. Also looks like locks, config read/write changes and reset ioctls are being added. Plus a mmap fault handler with retry, ioeventfd write recovery checks, and intx recovery start/finish. All necessary additions to improve the fault tolerance of VFIO PCI. Now, I need to understand the testability of this. I need to figure out how this integrates with QEMU and virtualization. Is this testable and fuzzable in a virtualized environment? What Kconfig options need to be enabled for testing? I'll probably start with `grepper` or scan through the Kconfig files in `drivers/vfio/pci` to figure out the requirements. Gotta make sure we can thoroughly validate this change. |
| 4/2 | 2026/09/01 22:28 | tool | grepper |
0mArgs:{
"Expression": "VFIO_PCI",
"PathPrefix": "drivers/vfio/pci/Kconfig"
}Results: {
"Output": "drivers/vfio/pci/Kconfig=2=menu \"VFIO support for PCI devices\"\n--\ndrivers/vfio/pci/Kconfig-4-\ndrivers/vfio/pci/Kconfig:5:config VFIO_PCI_CORE\ndrivers/vfio/pci/Kconfig-6-\ttristate\n--\ndrivers/vfio/pci/Kconfig-9-\ndrivers/vfio/pci/Kconfig:10:config VFIO_PCI_INTX\ndrivers/vfio/pci/Kconfig-11-\tdef_bool y if !S390\ndrivers/vfio/pci/Kconfig:12:\tdepends on VFIO_PCI_CORE\ndrivers/vfio/pci/Kconfig-13-\ndrivers/vfio/pci/Kconfig:14:config VFIO_PCI\ndrivers/vfio/pci/Kconfig-15-\ttristate \"Generic VFIO support for any PCI device\"\ndrivers/vfio/pci/Kconfig:16:\tselect VFIO_PCI_CORE\ndrivers/vfio/pci/Kconfig-17-\thelp\n--\ndrivers/vfio/pci/Kconfig-22-\ndrivers/vfio/pci/Kconfig:23:if VFIO_PCI\ndrivers/vfio/pci/Kconfig:24:config VFIO_PCI_VGA\ndrivers/vfio/pci/Kconfig-25-\tbool \"Generic VFIO PCI support for VGA devices\"\n--\ndrivers/vfio/pci/Kconfig-33-\ndrivers/vfio/pci/Kconfig:34:config VFIO_PCI_IGD\ndrivers/vfio/pci/Kconfig-35-\tbool \"Generic VFIO PCI extensions for Intel graphics (GVT-d)\"\n--\ndrivers/vfio/pci/Kconfig=45=endif\ndrivers/vfio/pci/Kconfig-46-\ndrivers/vfio/pci/Kconfig:47:config VFIO_PCI_ZDEV_KVM\ndrivers/vfio/pci/Kconfig-48-\tbool \"VFIO PCI extensions for s390x KVM passthrough\"\n--\ndrivers/vfio/pci/Kconfig-57-\ndrivers/vfio/pci/Kconfig:58:config VFIO_PCI_DMABUF\ndrivers/vfio/pci/Kconfig:59:\tdef_bool y if VFIO_PCI_CORE \u0026\u0026 PCI_P2PDMA \u0026\u0026 DMA_SHARED_BUFFER\ndrivers/vfio/pci/Kconfig-60-\n"
} |
| 5/2 | 2026/09/01 22:28 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 32914 output: 33 thoughts: 18 Thoughts: Here's my take on the thought process: **Diving into the VFIO Kconfig Hierarchy** Okay, time to get my bearings. Before I start making any assumptions or diving into the code, I need to understand the structural context of the VFIO driver configuration. My first step is going to be to examine the `drivers/vfio/Kconfig` file. This is where I'll find the Kconfig hierarchy, the relationships between the various VFIO options, and any dependencies they might have on other kernel features or hardware. Knowing this upfront will save a lot of time and potential missteps later. Understanding how all the pieces fit together from a configuration standpoint is crucial for making informed decisions and ensuring that any changes I make are compatible and correctly integrated. Let's see what we can uncover in there. |
| 6/2 | 2026/09/01 22:28 | tool | read-file |
0mArgs:{
"File": "drivers/vfio/Kconfig",
"FirstLine": 1,
"LineCount": 80
}Results: {
"Contents": " 1:\t# SPDX-License-Identifier: GPL-2.0-only\n 2:\tmenuconfig VFIO\n 3:\t\ttristate \"VFIO Non-Privileged userspace driver framework\"\n 4:\t\tselect IOMMU_API\n 5:\t\tdepends on IOMMUFD || !IOMMUFD\n 6:\t\tselect INTERVAL_TREE\n 7:\t\tselect VFIO_GROUP if SPAPR_TCE_IOMMU || IOMMUFD=n\n 8:\t\tselect VFIO_DEVICE_CDEV if !VFIO_GROUP\n 9:\t\tselect VFIO_CONTAINER if IOMMUFD=n\n 10:\t\thelp\n 11:\t\t VFIO provides a framework for secure userspace device drivers.\n 12:\t\t See Documentation/driver-api/vfio.rst for more details.\n 13:\t\n 14:\t\t If you don't know what to do here, say N.\n 15:\t\n 16:\tif VFIO\n 17:\tconfig VFIO_DEVICE_CDEV\n 18:\t\tbool \"Support for the VFIO cdev /dev/vfio/devices/vfioX\"\n 19:\t\tdepends on IOMMUFD \u0026\u0026 !SPAPR_TCE_IOMMU\n 20:\t\tdefault !VFIO_GROUP\n 21:\t\thelp\n 22:\t\t The VFIO device cdev is another way for userspace to get device\n 23:\t\t access. Userspace gets device fd by opening device cdev under\n 24:\t\t /dev/vfio/devices/vfioX, and then bind the device fd with an iommufd\n 25:\t\t to set up secure DMA context for device access.\n 26:\t\n 27:\t\t If you don't know what to do here, say N.\n 28:\t\n 29:\tconfig VFIO_GROUP\n 30:\t\tbool \"Support for the VFIO group /dev/vfio/$group_id\"\n 31:\t\tdefault y\n 32:\t\thelp\n 33:\t\t VFIO group support provides the traditional model for accessing\n 34:\t\t devices through VFIO and is used by the majority of userspace\n 35:\t\t applications and drivers making use of VFIO.\n 36:\t\n 37:\t\t If you don't know what to do here, say Y.\n 38:\t\n 39:\tconfig VFIO_CONTAINER\n 40:\t\tbool \"Support for the VFIO container /dev/vfio/vfio\"\n 41:\t\tselect VFIO_IOMMU_TYPE1 if MMU \u0026\u0026 (X86 || S390 || ARM || ARM64)\n 42:\t\tdepends on VFIO_GROUP\n 43:\t\tdefault y\n 44:\t\thelp\n 45:\t\t The VFIO container is the classic interface to VFIO for establishing\n 46:\t\t IOMMU mappings. If N is selected here then IOMMUFD must be used to\n 47:\t\t manage the mappings.\n 48:\t\n 49:\t\t Unless testing IOMMUFD say Y here.\n 50:\t\n 51:\tif VFIO_CONTAINER\n 52:\tconfig VFIO_IOMMU_TYPE1\n 53:\t\ttristate\n 54:\t\tdefault n\n 55:\t\n 56:\tconfig VFIO_IOMMU_SPAPR_TCE\n 57:\t\ttristate\n 58:\t\tdepends on SPAPR_TCE_IOMMU\n 59:\t\tdefault VFIO\n 60:\tendif\n 61:\t\n 62:\tconfig VFIO_NOIOMMU\n 63:\t\tbool \"VFIO No-IOMMU support\"\n 64:\t\tdepends on VFIO_GROUP || (VFIO_DEVICE_CDEV \u0026\u0026 !GENERIC_ATOMIC64)\n 65:\t\tdepends on !VFIO_GROUP || VFIO_CONTAINER || IOMMUFD_VFIO_CONTAINER\n 66:\t\tselect IOMMUFD_NOIOMMU if VFIO_DEVICE_CDEV \u0026\u0026 !GENERIC_ATOMIC64\n 67:\t\thelp\n 68:\t\t VFIO is built on the ability to isolate devices using the IOMMU.\n 69:\t\t Only with an IOMMU can userspace access to DMA capable devices be\n 70:\t\t considered secure. VFIO No-IOMMU mode enables IOMMU groups for\n 71:\t\t devices without IOMMU backing for the purpose of re-using the VFIO\n 72:\t\t infrastructure in a non-secure mode. Use of this mode will result\n 73:\t\t in an unsupportable kernel and will therefore taint the kernel.\n 74:\t\t Device assignment to virtual machines is also not possible with\n 75:\t\t this mode since there is no IOMMU to provide DMA translation.\n 76:\t\n 77:\t\t If you don't know what to do here, say N.\n 78:\t\n 79:\tconfig VFIO_VIRQFD\n 80:\t\tbool\n"
} |
| 7/2 | 2026/09/01 22:28 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 34388 output: 35 thoughts: 20 Thoughts: Here's my take on those thoughts: **Investigating the Build Configuration for VFIO Drivers** Okay, so I need to understand exactly how the build system handles the VFIO drivers, specifically the PCI-related ones. My instinct is to head straight for the `drivers/vfio/pci/Makefile`. That's where I expect to find the crucial information: the conditional logic, the variables, the `obj-$(CONFIG_...)` lines – everything that dictates which VFIO PCI components get compiled and linked depending on the various configuration options. This Makefile should be my Rosetta Stone for deciphering the build process for this section. I need to figure out what flags control what. |
| 8/2 | 2026/09/01 22:28 | tool | read-file |
0mArgs:{
"File": "drivers/vfio/pci/Makefile",
"FirstLine": 1,
"LineCount": 40
}Results: {
"Contents": " 1:\t# SPDX-License-Identifier: GPL-2.0-only\n 2:\t\n 3:\tvfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o\n 4:\tvfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o\n 5:\tvfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o\n 6:\tobj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o\n 7:\t\n 8:\tvfio-pci-y := vfio_pci.o\n 9:\tvfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o\n 10:\tobj-$(CONFIG_VFIO_PCI) += vfio-pci.o\n 11:\t\n 12:\tobj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/\n 13:\t\n 14:\tobj-$(CONFIG_ISM_VFIO_PCI) += ism/\n 15:\t\n 16:\tobj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/\n 17:\t\n 18:\tobj-$(CONFIG_PDS_VFIO_PCI) += pds/\n 19:\t\n 20:\tobj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/\n 21:\t\n 22:\tobj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/\n 23:\t\n 24:\tobj-$(CONFIG_QAT_VFIO_PCI) += qat/\n 25:\t\n 26:\tobj-$(CONFIG_XE_VFIO_PCI) += xe/\n"
} |
| 9/2 | 2026/09/01 22:28 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 35101 output: 34 thoughts: 1142 Thoughts: **VFIO PCI Analysis: Worthiness and Focus for Fuzzing** Alright, let's break down these changes to VFIO-PCI and assess their fuzzing potential. First, I'm checking the context – VFIO-PCI is core VFIO support for PCI devices, so we're dealing with a fundamental, reachable component used in standard environments, QEMU, and, importantly, Syzkaller. Excellent. It's a key framework for user-space access to PCI devices. Syzkaller is actively targeting vfio ioctls, which immediately piques my interest. My initial scan focuses on what's *new*. The `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY` ioctl handler in `vfio_pci_core_ioctl_feature` catches my eye. Looking deeper, I see a constellation of related functions modified or introduced: `vfio_pci_try_reset_function`, AER-related handlers (`vfio_pci_core_aer_err_detected`, `vfio_pci_core_aer_slot_reset`, `vfio_pci_core_aer_resume`), mmap fault handlers (`vfio_pci_mmap_huge_fault` / `vfio_pci_core_fault_trylock`), config space access (`vfio_pci_config_rw_single`), BAR accesses (`vfio_pci_bar_rw`), and interrupt handling (`vfio_pci_ioctl_set_irqs`, `vfio_pci_intx_unmask_handler`, `vfio_intx_handler`). It's a good chunk of code, and it touches multiple areas. Now, the crucial question: Is this worth fuzzing? Let's check: * Is it modifying reachable core kernel logic? Yes, VFIO-PCI, so, yes! * Is it merely decorative, documentation, or unreachable? No, it's generic and fundamental. * Is it purely teardown/cleanup? No, it *adds* the `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY` feature and affects locking, error handling, reset, interrupts, and memory mapping. Therefore: WorthFuzzing = true. Absolutely. Next, I need to identify the key symbols – the entrypoints, the ones that are exported or distinct. I've got to find the non-static, non-inlined API. I've examined the functions: * `vfio_pci_core_feature_error_recovery` (static, but `vfio_pci_core_ioctl_feature` is an entrypoint) * `vfio_pci_try_reset_function` (non-static, declared in vfio_pci_priv.h, used in multiple files) * `vfio_pci_core_aer_err_detected` (EXPORT_SYMBOL_GPL) * `vfio_pci_core_access_begin` (non-static) * `vfio_pci_core_access_end` (non-static) * `vfio_pci_core_close_device` (EXPORT_SYMBOL_GPL) * `vfio_pci_core_enable` (EXPORT_SYMBOL_GPL) * `vfio_pci_core_finish_enable` (EXPORT_SYMBOL_GPL) * `vfio_pci_core_ioctl_feature` (EXPORT_SYMBOL_GPL) * `vfio_pci_config_rw_single` (non-static) * `vfio_pci_intx_recovery_start` (non-static) * `vfio_pci_intx_recovery_finish` (non-static) * `vfio_pci_intx_recovery_command` (non-static) * `vfio_pci_bar_rw` (non-static) * `vfio_pci_core_feature_dma_buf` (EXPORT_SYMBOL_GPL) I need to confirm which ones are global symbols across the source files. These are the primary targets for fuzzing. We have a solid starting point for a fuzzing campaign! |
| 10/2 | 2026/09/01 22:29 | tool | grepper |
0mArgs:{
"Expression": "vfio_pci_",
"PathPrefix": "drivers/vfio/pci/"
}Results: {
"Output": "\nFull output is too long, showing 500 out of 4425 lines.\nUse more precise expression if possible.\n\ndrivers/vfio/pci/Makefile-2-\ndrivers/vfio/pci/Makefile:3:vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o\ndrivers/vfio/pci/Makefile:4:vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o\ndrivers/vfio/pci/Makefile:5:vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o\ndrivers/vfio/pci/Makefile-6-obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o\n--\ndrivers/vfio/pci/Makefile=8=vfio-pci-y := vfio_pci.o\ndrivers/vfio/pci/Makefile:9:vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o\ndrivers/vfio/pci/Makefile-10-obj-$(CONFIG_VFIO_PCI) += vfio-pci.o\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-13-#include \u003clinux/vfio.h\u003e\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:14:#include \u003clinux/vfio_pci_core.h\u003e\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-15-#include \u003clinux/anon_inodes.h\u003e\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=589=static struct hisi_acc_vf_core_device *hisi_acc_drvdata(struct pci_dev *pdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-590-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:591:\tstruct vfio_pci_core_device *core_device = dev_get_drvdata(\u0026pdev-\u003edev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-592-\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1100=static struct file *\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1101:hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1102-\t\t\t\t enum vfio_device_mig_state new_state)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1133=static int\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1134:hisi_acc_vfio_pci_get_data_size(struct vfio_device *vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1135-\t\t\t\tunsigned long *stop_copy_length)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1141=static int\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1142:hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1143-\t\t\t\t enum vfio_device_mig_state *curr_state)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1188=static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1189-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1190:\tstruct vfio_pci_core_device *vdev = \u0026hisi_acc_vdev-\u003ecore_device;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1191-\tstruct hisi_qm *vf_qm = \u0026hisi_acc_vdev-\u003evf_qm;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1244=static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1273-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1274:static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1275-\t\t\t\t\tunsigned int index)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1296=static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1300-\tunsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1301:\tstruct vfio_pci_core_device *vdev =\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1302:\t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1303-\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1318-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1319:static int hisi_acc_vfio_pci_mmap(struct vfio_device *core_vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1320-\t\t\t\t struct vm_area_struct *vma)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1321-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1322:\tstruct vfio_pci_core_device *vdev =\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1323:\t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1324-\tunsigned int index;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1340-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1341:\treturn vfio_pci_core_mmap(core_vdev, vma);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1342-}\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1343-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1344:static ssize_t hisi_acc_vfio_pci_write(struct vfio_device *core_vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1345-\t\t\t\t const char __user *buf, size_t count,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1354-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1355:\treturn vfio_pci_core_write(core_vdev, buf, new_count, ppos);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1356-}\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1357-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1358:static ssize_t hisi_acc_vfio_pci_read(struct vfio_device *core_vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1359-\t\t\t\t char __user *buf, size_t count,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1368-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1369:\treturn vfio_pci_core_read(core_vdev, buf, new_count, ppos);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1370-}\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1372=static int hisi_acc_vfio_ioctl_get_region(struct vfio_device *core_vdev,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1375-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1376:\tstruct vfio_pci_core_device *vdev =\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1377:\t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1378-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1379-\tif (info-\u003eindex != VFIO_PCI_BAR2_REGION_INDEX)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1380:\t\treturn vfio_pci_ioctl_get_region_info(core_vdev, info, caps);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1381-\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1416=static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1418-\tstruct device *vf_dev = seq-\u003eprivate;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1419:\tstruct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1420-\tstruct vfio_device *vdev = \u0026core_device-\u003evdev;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1445=static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1447-\tstruct device *vf_dev = seq-\u003eprivate;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1448:\tstruct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1449-\tstruct vfio_device *vdev = \u0026core_device-\u003evdev;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1492=static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1494-\tstruct device *vf_dev = seq-\u003eprivate;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1495:\tstruct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1496-\tstruct vfio_device *vdev = \u0026core_device-\u003evdev;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1513-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1514:static int hisi_acc_vfio_pci_open_device(struct vfio_device *core_vdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1515-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1516-\tstruct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(core_vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1517:\tstruct vfio_pci_core_device *vdev = \u0026hisi_acc_vdev-\u003ecore_device;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1518-\tint ret;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1519-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1520:\tret = vfio_pci_core_enable(vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1521-\tif (ret)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1528-\t\t\tmutex_unlock(\u0026hisi_acc_vdev-\u003eopen_mutex);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1529:\t\t\tvfio_pci_core_disable(vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1530-\t\t\treturn ret;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1537-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1538:\tvfio_pci_core_finish_enable(vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1539-\treturn 0;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1541-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1542:static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1543-{\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1552-\tmutex_unlock(\u0026hisi_acc_vdev-\u003eopen_mutex);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1553:\tvfio_pci_core_close_device(core_vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1554-}\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1555-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1556:static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1557:\t.migration_set_state = hisi_acc_vfio_pci_set_device_state,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1558:\t.migration_get_state = hisi_acc_vfio_pci_get_device_state,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1559:\t.migration_get_data_size = hisi_acc_vfio_pci_get_data_size,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1560-};\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1561-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1562:static int hisi_acc_vfio_pci_migrn_init_dev(struct vfio_device *core_vdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1563-{\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1575-\tcore_vdev-\u003emigration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_PRE_COPY;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1576:\tcore_vdev-\u003emig_ops = \u0026hisi_acc_vfio_pci_migrn_state_ops;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1577-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1578:\treturn vfio_pci_core_init_dev(core_vdev);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1579-}\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1580-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1581:static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1582-\t.name = \"hisi-acc-vfio-pci-migration\",\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1583:\t.init = hisi_acc_vfio_pci_migrn_init_dev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1584:\t.release = vfio_pci_core_release_dev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1585:\t.open_device = hisi_acc_vfio_pci_open_device,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1586:\t.close_device = hisi_acc_vfio_pci_close_device,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1587:\t.ioctl = vfio_pci_core_ioctl,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1588-\t.get_region_info_caps = hisi_acc_vfio_ioctl_get_region,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1589:\t.device_feature = vfio_pci_core_ioctl_feature,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1590:\t.read = hisi_acc_vfio_pci_read,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1591:\t.write = hisi_acc_vfio_pci_write,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1592:\t.mmap = hisi_acc_vfio_pci_mmap,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1593:\t.request = vfio_pci_core_request,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1594:\t.match = vfio_pci_core_match,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1595:\t.match_token_uuid = vfio_pci_core_match_token_uuid,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1596-\t.bind_iommufd = vfio_iommufd_physical_bind,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1601-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1602:static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1603-\t.name = \"hisi-acc-vfio-pci\",\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1604:\t.init = vfio_pci_core_init_dev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1605:\t.release = vfio_pci_core_release_dev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1606:\t.open_device = hisi_acc_vfio_pci_open_device,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1607:\t.close_device = vfio_pci_core_close_device,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1608:\t.ioctl = vfio_pci_core_ioctl,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1609:\t.get_region_info_caps = vfio_pci_ioctl_get_region_info,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1610:\t.device_feature = vfio_pci_core_ioctl_feature,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1611:\t.read = vfio_pci_core_read,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1612:\t.write = vfio_pci_core_write,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1613:\t.mmap = vfio_pci_core_mmap,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1614:\t.request = vfio_pci_core_request,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1615:\t.match = vfio_pci_core_match,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1616:\t.match_token_uuid = vfio_pci_core_match_token_uuid,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1617-\t.bind_iommufd = vfio_iommufd_physical_bind,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1623=static void hisi_acc_vfio_debug_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1634-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1635:\tif (vdev-\u003eops != \u0026hisi_acc_vfio_pci_migrn_ops)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1636-\t\treturn;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1662=static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device *hisi_acc_vdev)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1667-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1668:static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1669-{\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1670-\tstruct hisi_acc_vf_core_device *hisi_acc_vdev;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1671:\tconst struct vfio_device_ops *ops = \u0026hisi_acc_vfio_pci_ops;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1672-\tstruct hisi_qm *pf_qm;\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1679-\t\tif (vf_id \u003e= 0)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1680:\t\t\tops = \u0026hisi_acc_vfio_pci_migrn_ops;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1681-\t\telse\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1690-\tdev_set_drvdata(\u0026pdev-\u003edev, \u0026hisi_acc_vdev-\u003ecore_device);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1691:\tret = vfio_pci_core_register_device(\u0026hisi_acc_vdev-\u003ecore_device);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1692-\tif (ret)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1702-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1703:static void hisi_acc_vfio_pci_remove(struct pci_dev *pdev)\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1704-{\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1706-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1707:\tvfio_pci_core_unregister_device(\u0026hisi_acc_vdev-\u003ecore_device);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1708-\thisi_acc_vf_debugfs_exit(hisi_acc_vdev);\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1711-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1712:static const struct pci_device_id hisi_acc_vfio_pci_table[] = {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1713-\t{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HUAWEI_SEC_VF) },\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1718-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1719:MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1720-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1721=static const struct pci_error_handlers hisi_acc_vf_err_handlers = {\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1723-\t.reset_done = hisi_acc_vf_pci_aer_reset_done,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1724:\t.error_detected = vfio_pci_core_aer_err_detected,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1725-};\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1726-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1727:static struct pci_driver hisi_acc_vfio_pci_driver = {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1728-\t.name = KBUILD_MODNAME,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1729:\t.id_table = hisi_acc_vfio_pci_table,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1730:\t.probe = hisi_acc_vfio_pci_probe,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1731:\t.remove = hisi_acc_vfio_pci_remove,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1732-\t.err_handler = \u0026hisi_acc_vf_err_handlers,\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1735-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1736:module_pci_driver(hisi_acc_vfio_pci_driver);\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1737-\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h=129=struct hisi_acc_vf_core_device {\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h:130:\tstruct vfio_pci_core_device core_device;\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h-131-\tu8 match_done;\n--\ndrivers/vfio/pci/ism/main.c-8-#include \u003clinux/slab.h\u003e\ndrivers/vfio/pci/ism/main.c:9:#include \"../vfio_pci_priv.h\"\ndrivers/vfio/pci/ism/main.c-10-\n--\ndrivers/vfio/pci/ism/main.c=43=ISM_READ(8);\ndrivers/vfio/pci/ism/main.c-44-\ndrivers/vfio/pci/ism/main.c:45:struct ism_vfio_pci_core_device {\ndrivers/vfio/pci/ism/main.c:46:\tstruct vfio_pci_core_device core_device;\ndrivers/vfio/pci/ism/main.c-47-\tstruct kmem_cache *store_block_cache;\n--\ndrivers/vfio/pci/ism/main.c-49-\ndrivers/vfio/pci/ism/main.c:50:static int ism_vfio_pci_open_device(struct vfio_device *core_vdev)\ndrivers/vfio/pci/ism/main.c-51-{\ndrivers/vfio/pci/ism/main.c:52:\tstruct ism_vfio_pci_core_device *ivpcd;\ndrivers/vfio/pci/ism/main.c:53:\tstruct vfio_pci_core_device *vdev;\ndrivers/vfio/pci/ism/main.c-54-\tint ret;\ndrivers/vfio/pci/ism/main.c-55-\ndrivers/vfio/pci/ism/main.c:56:\tivpcd = container_of(core_vdev, struct ism_vfio_pci_core_device,\ndrivers/vfio/pci/ism/main.c-57-\t\t\t core_device.vdev);\n--\ndrivers/vfio/pci/ism/main.c-59-\ndrivers/vfio/pci/ism/main.c:60:\tret = vfio_pci_core_enable(vdev);\ndrivers/vfio/pci/ism/main.c-61-\tif (ret)\n--\ndrivers/vfio/pci/ism/main.c-63-\ndrivers/vfio/pci/ism/main.c:64:\tvfio_pci_core_finish_enable(vdev);\ndrivers/vfio/pci/ism/main.c-65-\treturn 0;\n--\ndrivers/vfio/pci/ism/main.c-68-/*\ndrivers/vfio/pci/ism/main.c:69: * ism_vfio_pci_do_io_r()\ndrivers/vfio/pci/ism/main.c-70- *\n--\ndrivers/vfio/pci/ism/main.c-76- */\ndrivers/vfio/pci/ism/main.c:77:static ssize_t ism_vfio_pci_do_io_r(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/ism/main.c-78-\t\t\t\t char __user *buf, loff_t off, size_t count,\n--\ndrivers/vfio/pci/ism/main.c-115-/*\ndrivers/vfio/pci/ism/main.c:116: * ism_vfio_pci_do_io_w()\ndrivers/vfio/pci/ism/main.c-117- *\n--\ndrivers/vfio/pci/ism/main.c-121- */\ndrivers/vfio/pci/ism/main.c:122:static ssize_t ism_vfio_pci_do_io_w(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/ism/main.c-123-\t\t\t\t char __user *buf, loff_t off, size_t count,\n--\ndrivers/vfio/pci/ism/main.c-126-\tstruct zpci_dev *zdev = to_zpci(vdev-\u003epdev);\ndrivers/vfio/pci/ism/main.c:127:\tstruct ism_vfio_pci_core_device *ivpcd;\ndrivers/vfio/pci/ism/main.c-128-\tssize_t ret;\n--\ndrivers/vfio/pci/ism/main.c-136-\ndrivers/vfio/pci/ism/main.c:137:\tivpcd = container_of(vdev, struct ism_vfio_pci_core_device,\ndrivers/vfio/pci/ism/main.c-138-\t\t\t core_device);\n--\ndrivers/vfio/pci/ism/main.c-159-\ndrivers/vfio/pci/ism/main.c:160:static ssize_t ism_vfio_pci_bar_rw(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/ism/main.c-161-\t\t\t\t char __user *buf, size_t count, loff_t *ppos,\n--\ndrivers/vfio/pci/ism/main.c-179-\tif (iswrite)\ndrivers/vfio/pci/ism/main.c:180:\t\tdone = ism_vfio_pci_do_io_w(vdev, buf, pos, count, bar);\ndrivers/vfio/pci/ism/main.c-181-\telse\ndrivers/vfio/pci/ism/main.c:182:\t\tdone = ism_vfio_pci_do_io_r(vdev, buf, pos, count, bar);\ndrivers/vfio/pci/ism/main.c-183-\n--\ndrivers/vfio/pci/ism/main.c-189-\ndrivers/vfio/pci/ism/main.c:190:static ssize_t ism_vfio_pci_config_rw(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/ism/main.c-191-\t\t\t\t char __user *buf, size_t count,\n--\ndrivers/vfio/pci/ism/main.c-204-\t\t */\ndrivers/vfio/pci/ism/main.c:205:\t\tret = vfio_pci_config_rw_single(vdev, buf, count, \u0026pos, iswrite);\ndrivers/vfio/pci/ism/main.c-206-\t\tif (ret \u003c 0)\n--\ndrivers/vfio/pci/ism/main.c-219-\ndrivers/vfio/pci/ism/main.c:220:static ssize_t ism_vfio_pci_rw(struct vfio_device *core_vdev, char __user *buf,\ndrivers/vfio/pci/ism/main.c-221-\t\t\t size_t count, loff_t *ppos, bool iswrite)\n--\ndrivers/vfio/pci/ism/main.c-223-\tunsigned int index = ISM_VFIO_PCI_OFFSET_TO_INDEX(*ppos);\ndrivers/vfio/pci/ism/main.c:224:\tstruct vfio_pci_core_device *vdev;\ndrivers/vfio/pci/ism/main.c-225-\tint ret;\ndrivers/vfio/pci/ism/main.c-226-\ndrivers/vfio/pci/ism/main.c:227:\tvdev = container_of(core_vdev, struct vfio_pci_core_device, vdev);\ndrivers/vfio/pci/ism/main.c-228-\n--\ndrivers/vfio/pci/ism/main.c-233-\tcase VFIO_PCI_CONFIG_REGION_INDEX:\ndrivers/vfio/pci/ism/main.c:234:\t\tret = ism_vfio_pci_config_rw(vdev, buf, count, ppos, iswrite);\ndrivers/vfio/pci/ism/main.c-235-\t\tbreak;\n--\ndrivers/vfio/pci/ism/main.c-237-\tcase VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:\ndrivers/vfio/pci/ism/main.c:238:\t\tret = ism_vfio_pci_bar_rw(vdev, buf, count, ppos, iswrite);\ndrivers/vfio/pci/ism/main.c-239-\t\tbreak;\n--\ndrivers/vfio/pci/ism/main.c-247-\ndrivers/vfio/pci/ism/main.c:248:static ssize_t ism_vfio_pci_read(struct vfio_device *core_vdev,\ndrivers/vfio/pci/ism/main.c-249-\t\t\t\t char __user *buf, size_t count, loff_t *ppos)\ndrivers/vfio/pci/ism/main.c-250-{\ndrivers/vfio/pci/ism/main.c:251:\treturn ism_vfio_pci_rw(core_vdev, buf, count, ppos, false);\ndrivers/vfio/pci/ism/main.c-252-}\ndrivers/vfio/pci/ism/main.c-253-\ndrivers/vfio/pci/ism/main.c:254:static ssize_t ism_vfio_pci_write(struct vfio_device *core_vdev,\ndrivers/vfio/pci/ism/main.c-255-\t\t\t\t const char __user *buf, size_t count,\n--\ndrivers/vfio/pci/ism/main.c-257-{\ndrivers/vfio/pci/ism/main.c:258:\treturn ism_vfio_pci_rw(core_vdev, (char __user *)buf, count, ppos,\ndrivers/vfio/pci/ism/main.c-259-\t\t\t true);\n--\ndrivers/vfio/pci/ism/main.c-261-\ndrivers/vfio/pci/ism/main.c:262:static int ism_vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,\ndrivers/vfio/pci/ism/main.c-263-\t\t\t\t\t struct vfio_region_info *info,\n--\ndrivers/vfio/pci/ism/main.c-265-{\ndrivers/vfio/pci/ism/main.c:266:\tstruct vfio_pci_core_device *vdev =\ndrivers/vfio/pci/ism/main.c:267:\t\tcontainer_of(core_vdev, struct vfio_pci_core_device, vdev);\ndrivers/vfio/pci/ism/main.c-268-\tstruct pci_dev *pdev = vdev-\u003epdev;\n--\ndrivers/vfio/pci/ism/main.c-295-\ndrivers/vfio/pci/ism/main.c:296:static int ism_vfio_pci_init_dev(struct vfio_device *core_vdev)\ndrivers/vfio/pci/ism/main.c-297-{\ndrivers/vfio/pci/ism/main.c-298-\tstruct zpci_dev *zdev = to_zpci(to_pci_dev(core_vdev-\u003edev));\ndrivers/vfio/pci/ism/main.c:299:\tstruct ism_vfio_pci_core_device *ivpcd;\ndrivers/vfio/pci/ism/main.c-300-\tchar cache_name[20];\n--\ndrivers/vfio/pci/ism/main.c-302-\ndrivers/vfio/pci/ism/main.c:303:\tivpcd = container_of(core_vdev, struct ism_vfio_pci_core_device,\ndrivers/vfio/pci/ism/main.c-304-\t\t\t core_device.vdev);\n--\ndrivers/vfio/pci/ism/main.c-318-\ndrivers/vfio/pci/ism/main.c:319:\tret = vfio_pci_core_init_dev(core_vdev);\ndrivers/vfio/pci/ism/main.c-320-\tif (ret)\n--\ndrivers/vfio/pci/ism/main.c-325-\ndrivers/vfio/pci/ism/main.c:326:static void ism_vfio_pci_release_dev(struct vfio_device *core_vdev)\ndrivers/vfio/pci/ism/main.c-327-{\ndrivers/vfio/pci/ism/main.c:328:\tstruct ism_vfio_pci_core_device *ivpcd = container_of(\ndrivers/vfio/pci/ism/main.c:329:\t\tcore_vdev, struct ism_vfio_pci_core_device, core_device.vdev);\ndrivers/vfio/pci/ism/main.c-330-\ndrivers/vfio/pci/ism/main.c-331-\tkmem_cache_destroy(ivpcd-\u003estore_block_cache);\ndrivers/vfio/pci/ism/main.c:332:\tvfio_pci_core_release_dev(core_vdev);\ndrivers/vfio/pci/ism/main.c-333-}\n--\ndrivers/vfio/pci/ism/main.c=335=static const struct vfio_device_ops ism_pci_ops = {\ndrivers/vfio/pci/ism/main.c-336-\t.name = \"ism-vfio-pci\",\ndrivers/vfio/pci/ism/main.c:337:\t.init = ism_vfio_pci_init_dev,\ndrivers/vfio/pci/ism/main.c:338:\t.release = ism_vfio_pci_release_dev,\ndrivers/vfio/pci/ism/main.c:339:\t.open_device = ism_vfio_pci_open_device,\ndrivers/vfio/pci/ism/main.c:340:\t.close_device = vfio_pci_core_close_device,\ndrivers/vfio/pci/ism/main.c:341:\t.ioctl = vfio_pci_core_ioctl,\ndrivers/vfio/pci/ism/main.c:342:\t.get_region_info_caps = ism_vfio_pci_ioctl_get_region_info,\ndrivers/vfio/pci/ism/main.c:343:\t.device_feature = vfio_pci_core_ioctl_feature,\ndrivers/vfio/pci/ism/main.c:344:\t.read = ism_vfio_pci_read,\ndrivers/vfio/pci/ism/main.c:345:\t.write = ism_vfio_pci_write,\ndrivers/vfio/pci/ism/main.c:346:\t.request = vfio_pci_core_request,\ndrivers/vfio/pci/ism/main.c:347:\t.match = vfio_pci_core_match,\ndrivers/vfio/pci/ism/main.c:348:\t.match_token_uuid = vfio_pci_core_match_token_uuid,\ndrivers/vfio/pci/ism/main.c-349-\t.bind_iommufd = vfio_iommufd_physical_bind,\n--\ndrivers/vfio/pci/ism/main.c-354-\ndrivers/vfio/pci/ism/main.c:355:static int ism_vfio_pci_probe(struct pci_dev *pdev,\ndrivers/vfio/pci/ism/main.c-356-\t\t\t const struct pci_device_id *id)\ndrivers/vfio/pci/ism/main.c-357-{\ndrivers/vfio/pci/ism/main.c:358:\tstruct ism_vfio_pci_core_device *ivpcd;\ndrivers/vfio/pci/ism/main.c-359-\tint ret;\ndrivers/vfio/pci/ism/main.c-360-\ndrivers/vfio/pci/ism/main.c:361:\tivpcd = vfio_alloc_device(ism_vfio_pci_core_device, core_device.vdev,\ndrivers/vfio/pci/ism/main.c-362-\t\t\t\t \u0026pdev-\u003edev, \u0026ism_pci_ops);\n--\ndrivers/vfio/pci/ism/main.c-367-\ndrivers/vfio/pci/ism/main.c:368:\tret = vfio_pci_core_register_device(\u0026ivpcd-\u003ecore_device);\ndrivers/vfio/pci/ism/main.c-369-\tif (ret)\n--\ndrivers/vfio/pci/ism/main.c-374-\ndrivers/vfio/pci/ism/main.c:375:static void ism_vfio_pci_remove(struct pci_dev *pdev)\ndrivers/vfio/pci/ism/main.c-376-{\ndrivers/vfio/pci/ism/main.c:377:\tstruct vfio_pci_core_device *core_device;\ndrivers/vfio/pci/ism/main.c:378:\tstruct ism_vfio_pci_core_device *ivpcd;\ndrivers/vfio/pci/ism/main.c-379-\ndrivers/vfio/pci/ism/main.c-380-\tcore_device = dev_get_drvdata(\u0026pdev-\u003edev);\ndrivers/vfio/pci/ism/main.c:381:\tivpcd = container_of(core_device, struct ism_vfio_pci_core_device,\ndrivers/vfio/pci/ism/main.c-382-\t\t\t core_device);\ndrivers/vfio/pci/ism/main.c-383-\ndrivers/vfio/pci/ism/main.c:384:\tvfio_pci_core_unregister_device(\u0026ivpcd-\u003ecore_device);\ndrivers/vfio/pci/ism/main.c-385-\tvfio_put_device(\u0026ivpcd-\u003ecore_device.vdev);\n--\ndrivers/vfio/pci/ism/main.c=393=MODULE_DEVICE_TABLE(pci, ism_device_table);\ndrivers/vfio/pci/ism/main.c-394-\ndrivers/vfio/pci/ism/main.c:395:static struct pci_driver ism_vfio_pci_driver = {\ndrivers/vfio/pci/ism/main.c-396-\t.name = KBUILD_MODNAME,\ndrivers/vfio/pci/ism/main.c-397-\t.id_table = ism_device_table,\ndrivers/vfio/pci/ism/main.c:398:\t.probe = ism_vfio_pci_probe,\ndrivers/vfio/pci/ism/main.c:399:\t.remove = ism_vfio_pci_remove,\ndrivers/vfio/pci/ism/main.c:400:\t.err_handler = \u0026vfio_pci_core_err_handlers,\ndrivers/vfio/pci/ism/main.c-401-\t.driver_managed_dma = true,\n--\ndrivers/vfio/pci/ism/main.c-403-\ndrivers/vfio/pci/ism/main.c:404:module_pci_driver(ism_vfio_pci_driver);\ndrivers/vfio/pci/ism/main.c-405-\n--\ndrivers/vfio/pci/mlx5/cmd.h-9-#include \u003clinux/kernel.h\u003e\ndrivers/vfio/pci/mlx5/cmd.h:10:#include \u003clinux/vfio_pci_core.h\u003e\ndrivers/vfio/pci/mlx5/cmd.h-11-#include \u003clinux/mlx5/driver.h\u003e\n--\ndrivers/vfio/pci/mlx5/cmd.h=172=struct mlx5vf_pci_core_device {\ndrivers/vfio/pci/mlx5/cmd.h:173:\tstruct vfio_pci_core_device core_device;\ndrivers/vfio/pci/mlx5/cmd.h-174-\tint vf_id;\n--\ndrivers/vfio/pci/mlx5/main.c=29=static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev)\ndrivers/vfio/pci/mlx5/main.c-30-{\ndrivers/vfio/pci/mlx5/main.c:31:\tstruct vfio_pci_core_device *core_device = dev_get_drvdata(\u0026pdev-\u003edev);\ndrivers/vfio/pci/mlx5/main.c-32-\n--\ndrivers/vfio/pci/mlx5/main.c=1323=static int mlx5vf_pci_open_device(struct vfio_device *core_vdev)\n--\ndrivers/vfio/pci/mlx5/main.c-1326-\t\tcore_vdev, struct mlx5vf_pci_core_device, core_device.vdev);\ndrivers/vfio/pci/mlx5/main.c:1327:\tstruct vfio_pci_core_device *vdev = \u0026mvdev-\u003ecore_device;\ndrivers/vfio/pci/mlx5/main.c-1328-\tint ret;\ndrivers/vfio/pci/mlx5/main.c-1329-\ndrivers/vfio/pci/mlx5/main.c:1330:\tret = vfio_pci_core_enable(vdev);\ndrivers/vfio/pci/mlx5/main.c-1331-\tif (ret)\n--\ndrivers/vfio/pci/mlx5/main.c-1335-\t\tmvdev-\u003emig_state = VFIO_DEVICE_STATE_RUNNING;\ndrivers/vfio/pci/mlx5/main.c:1336:\tvfio_pci_core_finish_enable(vdev);\ndrivers/vfio/pci/mlx5/main.c-1337-\treturn 0;\n--\ndrivers/vfio/pci/mlx5/main.c=1340=static void mlx5vf_pci_close_device(struct vfio_device *core_vdev)\n--\ndrivers/vfio/pci/mlx5/main.c-1345-\tmlx5vf_cmd_close_migratable(mvdev);\ndrivers/vfio/pci/mlx5/main.c:1346:\tvfio_pci_core_close_device(core_vdev);\ndrivers/vfio/pci/mlx5/main.c-1347-}\n--\ndrivers/vfio/pci/mlx5/main.c=1361=static int mlx5vf_pci_init_dev(struct vfio_device *core_vdev)\n--\ndrivers/vfio/pci/mlx5/main.c-1366-\ndrivers/vfio/pci/mlx5/main.c:1367:\tret = vfio_pci_core_init_dev(core_vdev);\ndrivers/vfio/pci/mlx5/main.c-1368-\tif (ret)\n--\ndrivers/vfio/pci/mlx5/main.c=1377=static void mlx5vf_pci_release_dev(struct vfio_device *core_vdev)\n--\ndrivers/vfio/pci/mlx5/main.c-1382-\tmlx5vf_cmd_remove_migratable(mvdev);\ndrivers/vfio/pci/mlx5/main.c:1383:\tvfio_pci_core_release_dev(core_vdev);\ndrivers/vfio/pci/mlx5/main.c-1384-}\n--\ndrivers/vfio/pci/mlx5/main.c=1386=static const struct vfio_device_ops mlx5vf_pci_ops = {\n--\ndrivers/vfio/pci/mlx5/main.c-1391-\t.close_device = mlx5vf_pci_close_device,\ndrivers/vfio/pci/mlx5/main.c:1392:\t.ioctl = vfio_pci_core_ioctl,\ndrivers/vfio/pci/mlx5/main.c:1393:\t.get_region_info_caps = vfio_pci_ioctl_get_region_info,\ndrivers/vfio/pci/mlx5/main.c:1394:\t.device_feature = vfio_pci_core_ioctl_feature,\ndrivers/vfio/pci/mlx5/main.c:1395:\t.read = vfio_pci_core_read,\ndrivers/vfio/pci/mlx5/main.c:1396:\t.write = vfio_pci_core_write,\ndrivers/vfio/pci/mlx5/main.c:1397:\t.mmap = vfio_pci_core_mmap,\ndrivers/vfio/pci/mlx5/main.c:1398:\t.request = vfio_pci_core_request,\ndrivers/vfio/pci/mlx5/main.c:1399:\t.match = vfio_pci_core_match,\ndrivers/vfio/pci/mlx5/main.c:1400:\t.match_token_uuid = vfio_pci_core_match_token_uuid,\ndrivers/vfio/pci/mlx5/main.c-1401-\t.bind_iommufd = vfio_iommufd_physical_bind,\n--\ndrivers/vfio/pci/mlx5/main.c=1407=static int mlx5vf_pci_probe(struct pci_dev *pdev,\n--\ndrivers/vfio/pci/mlx5/main.c-1418-\tdev_set_drvdata(\u0026pdev-\u003edev, \u0026mvdev-\u003ecore_device);\ndrivers/vfio/pci/mlx5/main.c:1419:\tret = vfio_pci_core_register_device(\u0026mvdev-\u003ecore_device);\ndrivers/vfio/pci/mlx5/main.c-1420-\tif (ret)\n--\ndrivers/vfio/pci/mlx5/main.c=1429=static void mlx5vf_pci_remove(struct pci_dev *pdev)\n--\ndrivers/vfio/pci/mlx5/main.c-1432-\ndrivers/vfio/pci/mlx5/main.c:1433:\tvfio_pci_core_unregister_device(\u0026mvdev-\u003ecore_device);\n\n"
} |
| 11/2 | 2026/09/01 22:29 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 53101 output: 29 thoughts: 667 Thoughts: **My Analysis of VFIO/PCI Driver Modifications** Okay, let's break down these changes in the `drivers/vfio/pci/` subsystem. I need to get a clear picture of the modified functions and how they're exposed. I'm focusing on the exported and non-static functions, as these are the ones that are likely part of the external interface or can have broader scope. First, I see the following functions have been touched: 1. `vfio_pci_core_feature_error_recovery`: This one's static, but it's called from `vfio_pci_core_ioctl_feature`, so it's indirectly part of the exposed functionality. 2. `vfio_pci_core_ioctl_feature`: This one is definitely exported; it's a key entry point. 3. `vfio_pci_try_reset_function`: Non-static and declared in `vfio_pci_priv.h`. This likely means it's accessible within the VFIO/PCI module and used in several places. 4. `vfio_pci_core_aer_err_detected`: Exported, likely related to Advanced Error Reporting. 5. `vfio_pci_core_aer_slot_reset` / `vfio_pci_core_err_handlers`: Exported, more AER-related functionality. 6. `vfio_pci_core_aer_resume`: In `vfio_pci_core.c` - implies AER recovery process. 7. `vfio_pci_core_access_begin` / `vfio_pci_core_access_end`: Non-static and declared in `vfio_pci_priv.h`. This is likely around resource access management for the PCI device. 8. `vfio_pci_config_rw_single`: Non-static, in `vfio_pci_config.c`, declared in `vfio_pci_priv.h`. This likely handles single configuration register reads/writes. 9. `vfio_pci_bar_rw`: Non-static, in `vfio_pci_rdwr.c`, declared in `vfio_pci_priv.h`. This is important; it's related to BAR (Base Address Register) access. 10. `vfio_pci_intx_recovery_start` / `vfio_pci_intx_recovery_finish`: Non-static in `vfio_pci_intrs.c`. Interrupt-related recovery code. Now, let's look at the Kconfig symbols to understand how these features are enabled. I'll need to verify the dependencies. It looks like: * `VFIO` (overall VFIO support) * `VFIO_PCI` (specific VFIO support for PCI devices) * `VFIO_PCI_CORE` (likely the core VFIO/PCI implementation) I need to confirm these in `drivers/vfio/pci/Kconfig` and `drivers/vfio/Kconfig`. Let's check the configurations. In `drivers/vfio/pci/Kconfig`, I see: `config VFIO_PCI` ` tristate "Generic VFIO support for any PCI device"` ` select VFIO_PCI_CORE` And in `drivers/vfio/Kconfig`: `config VFIO` Since `VFIO_PCI` is in the `drivers/vfio/pci/Kconfig` file, which is sourced from the `drivers/vfio/Kconfig` if PCI is enabled, I also need to verify that `PCI` itself is enabled. I should look at `drivers/vfio/Makefile` and `drivers/vfio/Kconfig` to be sure. I will also review any potential interactions with other subsystems in the PCI space, especially those related to error handling or resource management. |
| 12/2 | 2026/09/01 22:29 | tool | grepper |
0mArgs:{
"Expression": "source.*vfio.*pci",
"PathPrefix": "drivers/"
}Results: {
"Output": "drivers/vfio/Kconfig=84=config VFIO_DEBUGFS\n--\ndrivers/vfio/Kconfig-93-\ndrivers/vfio/Kconfig:94:source \"drivers/vfio/pci/Kconfig\"\ndrivers/vfio/Kconfig-95-source \"drivers/vfio/platform/Kconfig\"\n--\ndrivers/vfio/pci/Kconfig=58=config VFIO_PCI_DMABUF\n--\ndrivers/vfio/pci/Kconfig-60-\ndrivers/vfio/pci/Kconfig:61:source \"drivers/vfio/pci/mlx5/Kconfig\"\ndrivers/vfio/pci/Kconfig-62-\ndrivers/vfio/pci/Kconfig:63:source \"drivers/vfio/pci/ism/Kconfig\"\ndrivers/vfio/pci/Kconfig-64-\ndrivers/vfio/pci/Kconfig:65:source \"drivers/vfio/pci/hisilicon/Kconfig\"\ndrivers/vfio/pci/Kconfig-66-\ndrivers/vfio/pci/Kconfig:67:source \"drivers/vfio/pci/pds/Kconfig\"\ndrivers/vfio/pci/Kconfig-68-\ndrivers/vfio/pci/Kconfig:69:source \"drivers/vfio/pci/virtio/Kconfig\"\ndrivers/vfio/pci/Kconfig-70-\ndrivers/vfio/pci/Kconfig:71:source \"drivers/vfio/pci/nvgrace-gpu/Kconfig\"\ndrivers/vfio/pci/Kconfig-72-\ndrivers/vfio/pci/Kconfig:73:source \"drivers/vfio/pci/qat/Kconfig\"\ndrivers/vfio/pci/Kconfig-74-\ndrivers/vfio/pci/Kconfig:75:source \"drivers/vfio/pci/xe/Kconfig\"\ndrivers/vfio/pci/Kconfig-76-\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c=1244=static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev)\n--\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1273-\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c:1274:static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,\ndrivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c-1275-\t\t\t\t\tunsigned int index)\n--\ndrivers/vfio/pci/vfio_pci.c=44=module_param(disable_vga, bool, S_IRUGO);\ndrivers/vfio/pci/vfio_pci.c:45:MODULE_PARM_DESC(disable_vga, \"Disable VGA resource access through vfio-pci\");\ndrivers/vfio/pci/vfio_pci.c-46-#endif\n"
} |
| 13/2 | 2026/09/01 22:29 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 55472 output: 36 thoughts: 18 Thoughts: Here's my take on those thoughts: **Checking the VFIO PCI Configuration** Okay, time to circle back and double-check `drivers/vfio/pci/Kconfig`. I've got a hunch there's something specific in there that I need to confirm or revisit regarding the current implementation, probably a setting I want to be certain about. It's a key configuration file, so it's always worth a thorough look, even if I *think* I already know it inside and out. I'll need to refresh my memory on the interactions and dependencies within this particular part of the tree. Let's see what gems this Kconfig holds today. |
| 14/2 | 2026/09/01 22:29 | tool | read-file |
0mArgs:{
"File": "drivers/vfio/pci/Kconfig",
"FirstLine": 1,
"LineCount": 40
}Results: {
"Contents": " 1:\t# SPDX-License-Identifier: GPL-2.0-only\n 2:\tmenu \"VFIO support for PCI devices\"\n 3:\t\tdepends on PCI\n 4:\t\n 5:\tconfig VFIO_PCI_CORE\n 6:\t\ttristate\n 7:\t\tselect VFIO_VIRQFD\n 8:\t\tselect IRQ_BYPASS_MANAGER\n 9:\t\n 10:\tconfig VFIO_PCI_INTX\n 11:\t\tdef_bool y if !S390\n 12:\t\tdepends on VFIO_PCI_CORE\n 13:\t\n 14:\tconfig VFIO_PCI\n 15:\t\ttristate \"Generic VFIO support for any PCI device\"\n 16:\t\tselect VFIO_PCI_CORE\n 17:\t\thelp\n 18:\t\t Support for the generic PCI VFIO bus driver which can connect any\n 19:\t\t PCI device to the VFIO framework.\n 20:\t\n 21:\t\t If you don't know what to do here, say N.\n 22:\t\n 23:\tif VFIO_PCI\n 24:\tconfig VFIO_PCI_VGA\n 25:\t\tbool \"Generic VFIO PCI support for VGA devices\"\n 26:\t\tdepends on X86 \u0026\u0026 VGA_ARB\n 27:\t\thelp\n 28:\t\t Support for VGA extension to VFIO PCI. This exposes an additional\n 29:\t\t region on VGA devices for accessing legacy VGA addresses used by\n 30:\t\t BIOS and generic video drivers.\n 31:\t\n 32:\t\t If you don't know what to do here, say N.\n 33:\t\n 34:\tconfig VFIO_PCI_IGD\n 35:\t\tbool \"Generic VFIO PCI extensions for Intel graphics (GVT-d)\"\n 36:\t\tdepends on X86\n 37:\t\tdefault y\n 38:\t\thelp\n 39:\t\t Support for Intel IGD specific extensions to enable direct\n 40:\t\t assignment to virtual machines. This includes exposing an IGD\n"
} |
| 15/2 | 2026/09/01 22:29 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 56190 output: 29 thoughts: 60 Thoughts: **My VFIO_PCI Configuration Checklist** Okay, so I'm diving into VFIO_PCI and need to make sure I've got my ducks in a row with the kernel configuration. I know VFIO_PCI is the core, but I need to ensure all the prerequisites are met. First, I need to confirm that `VFIO` itself is enabled in `drivers/vfio/Kconfig`. That's the foundational piece, and then the obvious, `PCI` support should be enabled. Then, and only then, does `VFIO_PCI` become relevant, which also selects `VFIO_PCI_CORE`. Seems logical so far. Now, just to be thorough and because I've run into this before, I need to quickly scan for any other potentially relevant configurations. Specifically, I'll be looking for `PCIEAER` and `PCI_AER`. It's always worth a double check, since I may need advanced error reporting features enabled for optimal performance, and I wouldn't want to miss anything. Time to dig in and make sure I've got everything I need. |
| 16/2 | 2026/09/01 22:29 | tool | grepper |
0mArgs:{
"Expression": "PCIEAER",
"PathPrefix": "drivers/pci/"
}Results: {
"Output": "drivers/pci/controller/dwc/pcie-tegra194.c=783=static void tegra_pcie_enable_intx_interrupts(struct dw_pcie_rp *pp)\n--\ndrivers/pci/controller/dwc/pcie-tegra194.c-799-\tval |= APPL_INTR_EN_L1_8_EDMA_INT_EN;\ndrivers/pci/controller/dwc/pcie-tegra194.c:800:\tif (IS_ENABLED(CONFIG_PCIEAER))\ndrivers/pci/controller/dwc/pcie-tegra194.c-801-\t\tval |= APPL_INTR_EN_L1_8_AER_INT_EN;\n--\ndrivers/pci/controller/pci-host-common.c=345=static void pci_host_recover_root_port(struct pci_dev *port)\ndrivers/pci/controller/pci-host-common.c-346-{\ndrivers/pci/controller/pci-host-common.c:347:#if IS_ENABLED(CONFIG_PCIEAER)\ndrivers/pci/controller/pci-host-common.c-348-\tpcie_do_recovery(port, pci_channel_io_frozen, pci_host_reset_root_port);\n--\ndrivers/pci/pci-driver.c=1611=static int pci_uevent(const struct device *dev, struct kobj_uevent_env *env)\n--\ndrivers/pci/pci-driver.c-1642-\ndrivers/pci/pci-driver.c:1643:#if defined(CONFIG_PCIEAER) || defined(CONFIG_EEH) || defined(CONFIG_S390)\ndrivers/pci/pci-driver.c-1644-/**\n--\ndrivers/pci/pci-sysfs.c=1938=const struct attribute_group *pci_dev_attr_groups[] = {\n--\ndrivers/pci/pci-sysfs.c-1946-\t\u0026pcie_dev_attr_group,\ndrivers/pci/pci-sysfs.c:1947:#ifdef CONFIG_PCIEAER\ndrivers/pci/pci-sysfs.c-1948-\t\u0026aer_stats_attr_group,\n--\ndrivers/pci/pci.c=2270=EXPORT_SYMBOL_GPL(pci_set_pcie_reset_state);\ndrivers/pci/pci.c-2271-\ndrivers/pci/pci.c:2272:#ifdef CONFIG_PCIEAER\ndrivers/pci/pci.c-2273-void pcie_clear_device_status(struct pci_dev *dev)\n--\ndrivers/pci/pci.h=848=static inline bool pci_dev_binding_disallowed(struct pci_dev *dev)\n--\ndrivers/pci/pci.h-852-\ndrivers/pci/pci.h:853:#ifdef CONFIG_PCIEAER\ndrivers/pci/pci.h-854-#include \u003clinux/aer.h\u003e\n--\ndrivers/pci/pci.h=916=void pcie_print_tlp_log(const struct pci_dev *dev,\n--\ndrivers/pci/pci.h-918-\t\t\tconst char *pfx);\ndrivers/pci/pci.h:919:#endif\t/* CONFIG_PCIEAER */\ndrivers/pci/pci.h-920-\n--\ndrivers/pci/pci.h=1329=static inline void of_pci_remove_host_bridge_node(struct pci_host_bridge *bridge) { }\n--\ndrivers/pci/pci.h-1331-\ndrivers/pci/pci.h:1332:#ifdef CONFIG_PCIEAER\ndrivers/pci/pci.h-1333-void pci_no_aer(void);\n--\ndrivers/pci/pcie/Kconfig=16=config HOTPLUG_PCI_PCIE\n--\ndrivers/pci/pcie/Kconfig-27-\ndrivers/pci/pcie/Kconfig:28:config PCIEAER\ndrivers/pci/pcie/Kconfig-29-\tbool \"PCI Express Advanced Error Reporting support\"\n--\ndrivers/pci/pcie/Kconfig-36-\ndrivers/pci/pcie/Kconfig:37:config PCIEAER_INJECT\ndrivers/pci/pcie/Kconfig-38-\ttristate \"PCI Express error injection support\"\ndrivers/pci/pcie/Kconfig:39:\tdepends on PCIEAER\ndrivers/pci/pcie/Kconfig-40-\tselect GENERIC_IRQ_INJECTION\n--\ndrivers/pci/pcie/Kconfig=55=config PCIE_ECRC\ndrivers/pci/pcie/Kconfig-56-\tbool \"PCI Express ECRC settings control\"\ndrivers/pci/pcie/Kconfig:57:\tdepends on PCIEAER\ndrivers/pci/pcie/Kconfig-58-\thelp\n--\ndrivers/pci/pcie/Kconfig=121=config PCIE_DPC\ndrivers/pci/pcie/Kconfig-122-\tbool \"PCI Express Downstream Port Containment support\"\ndrivers/pci/pcie/Kconfig:123:\tdepends on PCIEPORTBUS \u0026\u0026 PCIEAER\ndrivers/pci/pcie/Kconfig-124-\thelp\n--\ndrivers/pci/pcie/Makefile=9=obj-y\t\t\t\t+= aspm.o\ndrivers/pci/pcie/Makefile:10:obj-$(CONFIG_PCIEAER)\t\t+= aer.o err.o tlp.o\ndrivers/pci/pcie/Makefile-11-obj-$(CONFIG_CXL_RAS)\t\t+= aer_cxl_rch.o\ndrivers/pci/pcie/Makefile:12:obj-$(CONFIG_PCIEAER_INJECT)\t+= aer_inject.o\ndrivers/pci/pcie/Makefile-13-obj-$(CONFIG_PCIE_PME)\t\t+= pme.o\n--\ndrivers/pci/pcie/aer.c=962=void aer_print_error(struct aer_err_info *info, int i)\n--\ndrivers/pci/pcie/aer.c-1015-\ndrivers/pci/pcie/aer.c:1016:#ifdef CONFIG_ACPI_APEI_PCIEAER\ndrivers/pci/pcie/aer.c-1017-int cper_severity_to_aer(int cper_severity)\n--\ndrivers/pci/pcie/aer.c=1339=static void handle_error_source(struct pci_dev *dev, struct aer_err_info *info)\n--\ndrivers/pci/pcie/aer.c-1345-\ndrivers/pci/pcie/aer.c:1346:#ifdef CONFIG_ACPI_APEI_PCIEAER\ndrivers/pci/pcie/aer.c-1347-\n--\ndrivers/pci/pcie/portdrv.c=57=static int pcie_message_numbers(struct pci_dev *dev, int mask,\n--\ndrivers/pci/pcie/portdrv.c-76-\ndrivers/pci/pcie/portdrv.c:77:#ifdef CONFIG_PCIEAER\ndrivers/pci/pcie/portdrv.c-78-\tif (mask \u0026 PCIE_PORT_SERVICE_AER) {\n--\ndrivers/pci/pcie/portdrv.c=218=static int get_port_device_capability(struct pci_dev *dev)\n--\ndrivers/pci/pcie/portdrv.c-238-\ndrivers/pci/pcie/portdrv.c:239:#ifdef CONFIG_PCIEAER\ndrivers/pci/pcie/portdrv.c-240-\tif ((pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||\n--\ndrivers/pci/pcie/portdrv.h=28=extern bool pcie_ports_dpc_native;\ndrivers/pci/pcie/portdrv.h-29-\ndrivers/pci/pcie/portdrv.h:30:#ifdef CONFIG_PCIEAER\ndrivers/pci/pcie/portdrv.h-31-int pcie_aer_init(void);\n--\ndrivers/pci/quirks.c=6405=DECLARE_PCI_FIXUP_FINAL(0x5555, 0x0004, pci_fixup_d3cold_delay_1sec);\ndrivers/pci/quirks.c-6406-\ndrivers/pci/quirks.c:6407:#ifdef CONFIG_PCIEAER\ndrivers/pci/quirks.c-6408-static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)\n"
} |
| 17/2 | 2026/09/01 22:29 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 58803 output: 310 thoughts: 357 Thoughts: **My Assessment of the VFIO PCI Error Recovery Implementation** Okay, so I'm digging into this new PCI error recovery functionality within VFIO. It looks like `PCIEAER` is directly intertwined with how we handle AER errors, especially with the `vfio_pci_core_aer_err_detected`, `slot_reset`, and `resume` functions. Also, the error injection mechanism using `PCIEAER_INJECT` is clearly part of the picture. Let's make sure we have a solid grasp of the core symbols involved here. I'm focusing on these key functions: `vfio_pci_core_ioctl_feature`, `vfio_pci_try_reset_function`, `vfio_pci_core_aer_err_detected`, `vfio_pci_core_aer_slot_reset`, `vfio_pci_core_aer_resume`, `vfio_pci_core_access_begin`, `vfio_pci_core_access_end`, `vfio_pci_config_rw_single`, `vfio_pci_bar_rw`, `vfio_pci_intx_recovery_start`, and `vfio_pci_intx_recovery_finish`. I'm making sure all the pieces are accounted for. I'm also double-checking the relevant configuration options: `VFIO`, `VFIO_PCI`, `VFIO_PCI_CORE`, and of course, `PCIEAER`. It's critical to ensure these are set up correctly. In essence, this is a clear and concise implementation of PCI error recovery. It covers the new `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY` feature ioctl, the AER error handler callbacks that have to work together, and any changes in concurrency or fault handling. This deserves a closer inspection and is definitely worth fuzzing. Alright, I think I've got a good handle on this. Let's call `set-results`. |
| 18/2 | 2026/09/01 22:29 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"VFIO",
"VFIO_PCI",
"VFIO_PCI_CORE",
"PCIEAER"
],
"FocusSymbols": [
"vfio_pci_core_ioctl_feature",
"vfio_pci_try_reset_function",
"vfio_pci_core_aer_err_detected",
"vfio_pci_core_aer_slot_reset",
"vfio_pci_core_aer_resume",
"vfio_pci_core_access_begin",
"vfio_pci_core_access_end",
"vfio_pci_config_rw_single",
"vfio_pci_bar_rw",
"vfio_pci_intx_recovery_start",
"vfio_pci_intx_recovery_finish"
],
"Reasoning": "The patch series adds host PCI error recovery support to the VFIO PCI core driver. It introduces the VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY device feature ioctl, implements PCIe AER error recovery callbacks (.error_detected, .slot_reset, .resume), updates access guards and locking across config/BAR access, IRQ handlers, resets, and huge page fault handling. These changes modify reachable driver/core logic and are suitable for fuzzing.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"VFIO",
"VFIO_PCI",
"VFIO_PCI_CORE",
"PCIEAER"
],
"FocusSymbols": [
"vfio_pci_core_ioctl_feature",
"vfio_pci_try_reset_function",
"vfio_pci_core_aer_err_detected",
"vfio_pci_core_aer_slot_reset",
"vfio_pci_core_aer_resume",
"vfio_pci_core_access_begin",
"vfio_pci_core_access_end",
"vfio_pci_config_rw_single",
"vfio_pci_bar_rw",
"vfio_pci_intx_recovery_start",
"vfio_pci_intx_recovery_finish"
],
"Reasoning": "The patch series adds host PCI error recovery support to the VFIO PCI core driver. It introduces the VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY device feature ioctl, implements PCIe AER error recovery callbacks (.error_detected, .slot_reset, .resume), updates access guards and locking across config/BAR access, IRQ handlers, resets, and huge page fault handling. These changes modify reachable driver/core logic and are suitable for fuzzing.",
"WorthFuzzing": true
} |
| 19/1 | 2026/09/01 22:29 | action | If |
0mArgs:{
"WorthFuzzing": true
}Results: null |
| 20/2 | 2026/09/01 22:29 | agent | kmsan-evaluator |
0mModel:core Args: null Results: {
"KMSANReasoning": "The patch introduces PCI error recovery support into the VFIO PCI driver, managing locking (recovery_lock vs memory_lock) and synchronization during AER events, resets, and power transitions.\n\nAnalysis of potential uninitialized memory risks:\n1. Kernel structure copied to user space:\n - In `vfio_pci_core_feature_error_recovery()`, `struct vfio_device_pci_error_recovery` is copied to userspace via `copy_to_user()`. The structure consists of `__u32 flags` (4 bytes), `__s32 eventfd` (4 bytes), and `__aligned_u64 sequence` (8 bytes), totaling 16 bytes with zero padding holes. The local variable `state` is initialized with `= { .eventfd = -1 }`, which zeroes all unmentioned fields, and then populated with the recovery state before being copied. No uninitialized bytes or padding are exposed.\n2. Stack/heap variables:\n - `struct vfio_pci_config_deferred deferred = {};` is zero-initialized on the stack.\n - All newly added struct fields in `struct vfio_pci_core_device` and `struct vfio_pci_irq_ctx` are initialized upon device/context allocation.\n3. Control flow and locking:\n - The patch primarily introduces access gating (`vfio_pci_core_access_begin`/`end`), wait queues, and error recovery handlers. Potential concurrency or locking bugs (such as deadlocks or race conditions) are within the domain of LOCKDEP and KASAN.\n\nThere are no uninitialized memory reads or information leak risks that would require a dedicated KMSAN fuzzing session.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: For your convenience, here is the diff of the changes:
commit 9245578c8ff1dcd9e2b6e621a377e36f1e75be35
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 1 22:28:34 2026 +0000
syz-cluster: applied patch under review
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 830369ff878db..46544dbe70d01 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -139,6 +139,7 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
*/
vdev->nointxmask = nointxmask;
vdev->disable_idle_d3 = disable_idle_d3;
+ vdev->pci_recovery_supported = true;
#ifdef CONFIG_VFIO_PCI_VGA
vdev->disable_vga = disable_vga;
#endif
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 9914f3ac69aef..3365100acf216 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -99,6 +99,12 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = {
[PCI_EXT_CAP_ID_DVSEC] = 0xFF,
};
+/* What a config write asked for which has to wait for the access guard. */
+struct vfio_pci_config_deferred {
+ bool flr; /* a function-level reset */
+ bool power_up; /* a transition to D0 */
+};
+
/*
* Read/Write Permission Bits - one bit for each bit in capability
* Any field can be read if it exists, but what is read depends on
@@ -111,8 +117,17 @@ struct perm_bits {
u8 *write; /* writeable bits */
int (*readfn)(struct vfio_pci_core_device *vdev, int pos, int count,
struct perm_bits *perm, int offset, __le32 *val);
+ /*
+ * @deferred records work the write asked for which a writefn must not
+ * do itself. Both a reset method and a transition to D0 acquire
+ * pci_bus_sem, which AER already holds when it enters the driver, so
+ * doing either here would invert the lock order against recovery_lock.
+ * The dispatcher does them after dropping recovery_lock. Callers zero
+ * it, and a writefn only sets a field on a success return.
+ */
int (*writefn)(struct vfio_pci_core_device *vdev, int pos, int count,
- struct perm_bits *perm, int offset, __le32 val);
+ struct perm_bits *perm, int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred);
};
#define NO_VIRT 0
@@ -200,7 +215,8 @@ static int vfio_default_config_read(struct vfio_pci_core_device *vdev, int pos,
static int vfio_default_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
__le32 virt = 0, write = 0;
@@ -272,7 +288,8 @@ static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos,
/* Raw access skips any kind of virtualization */
static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
int ret;
@@ -299,7 +316,8 @@ static int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos,
/* Virt access uses only virtualization */
static int vfio_virt_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
memcpy(vdev->vconfig + pos, &val, count);
return count;
@@ -563,7 +581,8 @@ static bool vfio_need_bar_restore(struct vfio_pci_core_device *vdev)
static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
struct pci_dev *pdev = vdev->pdev;
__le16 *virt_cmd;
@@ -613,7 +632,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
vfio_bar_restore(vdev);
}
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0) {
if (offset == PCI_COMMAND)
up_write(&vdev->memory_lock);
@@ -727,9 +747,11 @@ static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
@@ -738,8 +760,15 @@ static int vfio_pm_config_write(struct vfio_pci_core_device *vdev, int pos,
switch (le32_to_cpu(val) & PCI_PM_CTRL_STATE_MASK) {
case 0:
- state = PCI_D0;
- break;
+ /*
+ * Going to D0 reaches pci_set_full_power_state(),
+ * which takes pci_bus_sem through
+ * pcie_aspm_pm_state_change(). Leave it to the
+ * dispatcher. The lower states do not, so they run
+ * here.
+ */
+ deferred->power_up = true;
+ return count;
case 1:
state = PCI_D1;
break;
@@ -799,7 +828,8 @@ static int __init init_pci_cap_pm_perm(struct perm_bits *perm)
static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
struct pci_dev *pdev = vdev->pdev;
__le16 *paddr = (__le16 *)(vdev->vconfig + pos - offset + PCI_VPD_ADDR);
@@ -812,7 +842,8 @@ static int vfio_vpd_config_write(struct vfio_pci_core_device *vdev, int pos,
* of PCI_VPD_ADDR, then the PCI_VPD_ADDR_F bit is written and we
* have work to do.
*/
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0 || offset > PCI_VPD_ADDR + 1 ||
offset + count <= PCI_VPD_ADDR + 1)
return count;
@@ -881,21 +912,24 @@ static int __init init_pci_cap_pcix_perm(struct perm_bits *perm)
static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
__le16 *ctrl = (__le16 *)(vdev->vconfig + pos -
offset + PCI_EXP_DEVCTL);
int readrq = le16_to_cpu(*ctrl) & PCI_EXP_DEVCTL_READRQ;
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
/*
* The FLR bit is virtualized, if set and the device supports PCIe
- * FLR, issue a reset_function. Regardless, clear the bit, the spec
- * requires it to be always read as zero. NB, reset_function might
- * not use a PCIe FLR, we don't have that level of granularity.
+ * FLR, request a function reset once recovery_lock has been
+ * released. Regardless, clear the bit, the spec requires it to be
+ * always read as zero. NB, reset_function might not use a PCIe FLR,
+ * we don't have that level of granularity.
*/
if (*ctrl & cpu_to_le16(PCI_EXP_DEVCTL_BCR_FLR)) {
u32 cap;
@@ -907,14 +941,8 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
pos - offset + PCI_EXP_DEVCAP,
&cap);
- if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
- vfio_pci_zap_and_down_write_memory_lock(vdev);
- vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
- if (__vfio_pci_memory_enabled(vdev))
- vfio_pci_dma_buf_move(vdev, false);
- up_write(&vdev->memory_lock);
- }
+ if (!ret && (cap & PCI_EXP_DEVCAP_FLR))
+ deferred->flr = true;
}
/*
@@ -968,19 +996,22 @@ static int __init init_pci_cap_exp_perm(struct perm_bits *perm)
static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
u8 *ctrl = vdev->vconfig + pos - offset + PCI_AF_CTRL;
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
/*
* The FLR bit is virtualized, if set and the device supports AF
- * FLR, issue a reset_function. Regardless, clear the bit, the spec
- * requires it to be always read as zero. NB, reset_function might
- * not use an AF FLR, we don't have that level of granularity.
+ * FLR, request a function reset once recovery_lock has been
+ * released. Regardless, clear the bit, the spec requires it to be
+ * always read as zero. NB, reset_function might not use an AF FLR,
+ * we don't have that level of granularity.
*/
if (*ctrl & PCI_AF_CTRL_FLR) {
u8 cap;
@@ -992,14 +1023,8 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
pos - offset + PCI_AF_CAP,
&cap);
- if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
- vfio_pci_zap_and_down_write_memory_lock(vdev);
- vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
- if (__vfio_pci_memory_enabled(vdev))
- vfio_pci_dma_buf_move(vdev, false);
- up_write(&vdev->memory_lock);
- }
+ if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP))
+ deferred->flr = true;
}
return count;
@@ -1168,9 +1193,11 @@ static int vfio_msi_config_read(struct vfio_pci_core_device *vdev, int pos,
static int vfio_msi_config_write(struct vfio_pci_core_device *vdev, int pos,
int count, struct perm_bits *perm,
- int offset, __le32 val)
+ int offset, __le32 val,
+ struct vfio_pci_config_deferred *deferred)
{
- count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+ count = vfio_default_config_write(vdev, pos, count, perm, offset, val,
+ deferred);
if (count < 0)
return count;
@@ -1889,6 +1916,8 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
struct perm_bits *perm;
__le32 val = 0;
int cap_start = 0, offset;
+ int access_ret;
+ struct vfio_pci_config_deferred deferred = {};
u8 cap_id;
ssize_t ret;
@@ -1957,14 +1986,42 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
if (copy_from_user(&val, buf, count))
return -EFAULT;
- ret = perm->writefn(vdev, *ppos, count, perm, offset, val);
+ access_ret = vfio_pci_core_access_begin(vdev);
+ if (access_ret)
+ return access_ret;
+ ret = perm->writefn(vdev, *ppos, count, perm, offset, val,
+ &deferred);
+ vfio_pci_core_access_end(vdev);
+ if (ret < 0)
+ return ret;
+ /*
+ * Both of these take pci_bus_sem, so run them with the access
+ * guard dropped. The reset re-checks the recovery state for
+ * itself. The power up does not, so check it here.
+ *
+ * Both are best effort, as the guest-requested FLR has always
+ * been. The result is not reported back through the config
+ * write. Without recovery enabled the only failure is -EAGAIN
+ * from device lock contention, exactly as before. With it they
+ * are dropped while a recovery or reset transaction is in
+ * flight, which leaves the device in D0 and reset anyway.
+ */
+ if (deferred.power_up &&
+ !(vdev->pci_recovery_supported &&
+ READ_ONCE(vdev->pci_recovery_access_blocked)))
+ vfio_lock_and_set_power_state(vdev, PCI_D0);
+ if (deferred.flr)
+ vfio_pci_try_reset_function(vdev, false);
} else {
- if (perm->readfn) {
+ access_ret = vfio_pci_core_access_begin(vdev);
+ if (access_ret)
+ return access_ret;
+ if (perm->readfn)
ret = perm->readfn(vdev, *ppos, count,
perm, offset, &val);
- if (ret < 0)
- return ret;
- }
+ vfio_pci_core_access_end(vdev);
+ if (ret < 0)
+ return ret;
if (copy_to_user(buf, &val, count))
return -EFAULT;
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6757054e9d875..c1ea3c868fc5e 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -372,15 +372,21 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
struct eventfd_ctx *efdctx)
{
+ int ret;
+
/*
* The vdev power related flags are protected with 'memory_lock'
* semaphore.
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
vfio_pci_zap_and_down_write_memory_lock(vdev);
vfio_pci_dma_buf_move(vdev, true);
if (vdev->pm_runtime_engaged) {
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
return -EINVAL;
}
@@ -388,6 +394,7 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
vdev->pm_wake_eventfd_ctx = efdctx;
pm_runtime_put_noidle(&vdev->pdev->dev);
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
return 0;
}
@@ -483,7 +490,11 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
* already signaled the eventfd and exited low power mode itself.
* pm_runtime_engaged protects the redundant call here.
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
vfio_pci_runtime_pm_exit(vdev);
+ vfio_pci_core_access_end(vdev);
return 0;
}
@@ -591,10 +602,23 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {
int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
{
struct pci_dev *pdev = vdev->pdev;
+ bool supported = vdev->pci_recovery_supported;
int ret;
u16 cmd;
u8 msix_pos;
+ if (supported) {
+ down_write(&vdev->recovery_lock);
+ if (pci_dev_is_disconnected(pdev)) {
+ up_write(&vdev->recovery_lock);
+ return -ENODEV;
+ }
+
+ vdev->pci_recovery_command_valid = false;
+ WRITE_ONCE(vdev->pci_recovery_device_open, false);
+ up_write(&vdev->recovery_lock);
+ }
+
if (!vdev->disable_idle_d3) {
ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0)
@@ -815,7 +839,40 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
}
EXPORT_SYMBOL_GPL(vfio_pci_core_disable);
-void vfio_pci_core_close_device(struct vfio_device *core_vdev)
+static void vfio_pci_core_prepare_close(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return;
+
+ down_write(&vdev->recovery_lock);
+ WRITE_ONCE(vdev->pci_recovery_enabled, false);
+ vdev->pci_recovery_command_valid = false;
+ /*
+ * Clear access_blocked before device_open, so a lock-free reader
+ * never sees it set on a device which is no longer open. A
+ * transaction which is still running cannot clear it once
+ * device_open is gone, and paths which refuse work on a blocked
+ * device would then refuse it for good.
+ */
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ WRITE_ONCE(vdev->pci_recovery_device_open, false);
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+
+ /*
+ * Publish the closing state and drop recovery_lock before any
+ * teardown. Recovery is disabled and its state cleared, so
+ * slot_reset() and resume() become no-ops and a later
+ * error_detected() only follows the legacy notification path.
+ * Holding the lock across vfio_pci_core_disable() protects nothing
+ * and inverts the lock order. disable() reaches pci_reset_bus(),
+ * which takes pci_bus_sem, while error_detected() takes
+ * recovery_lock from under pci_bus_sem.
+ */
+ up_write(&vdev->recovery_lock);
+ wake_up_all(&vdev->pci_recovery_wait);
+}
+
+static void vfio_pci_core_finish_close(struct vfio_device *core_vdev)
{
struct vfio_pci_core_device *vdev =
container_of(core_vdev, struct vfio_pci_core_device, vdev);
@@ -836,8 +893,21 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
mutex_lock(&vdev->igate);
vfio_pci_eventfd_replace_locked(vdev, &vdev->err_trigger, NULL);
vfio_pci_eventfd_replace_locked(vdev, &vdev->req_trigger, NULL);
+ if (vdev->pci_recovery_supported)
+ vfio_pci_eventfd_replace_locked(vdev,
+ &vdev->pci_recovery_trigger,
+ NULL);
mutex_unlock(&vdev->igate);
}
+
+void vfio_pci_core_close_device(struct vfio_device *core_vdev)
+{
+ struct vfio_pci_core_device *vdev =
+ container_of(core_vdev, struct vfio_pci_core_device, vdev);
+
+ vfio_pci_core_prepare_close(vdev);
+ vfio_pci_core_finish_close(core_vdev);
+}
EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);
void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
@@ -852,6 +922,18 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
vdev->sriov_pf_core_dev->vf_token->users++;
mutex_unlock(&vdev->sriov_pf_core_dev->vf_token->lock);
}
+
+ if (vdev->pci_recovery_supported) {
+ down_write(&vdev->recovery_lock);
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+ vdev->pci_recovery_sequence = 0;
+ WRITE_ONCE(vdev->pci_recovery_enabled, false);
+ /* Close clears this too. Start unblocked either way. */
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ WRITE_ONCE(vdev->pci_recovery_device_open, true);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, false);
+ up_write(&vdev->recovery_lock);
+ }
}
EXPORT_SYMBOL_GPL(vfio_pci_core_finish_enable);
@@ -1101,7 +1183,13 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
return ret;
}
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret) {
+ kfree(caps.buf);
+ return ret;
+ }
ret = vfio_pci_info_atomic_cap(vdev, &caps);
+ vfio_pci_core_access_end(vdev);
if (ret && ret != -ENODEV) {
pci_warn(vdev->pdev,
"Failed to setup AtomicOps info capability\n");
@@ -1177,6 +1265,9 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
* Check ROM content is valid. Need to enable memory
* decode for ROM access in pci_map_rom().
*/
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
cmd = vfio_pci_memory_lock_and_enable(vdev);
io = pci_map_rom(pdev, &size);
if (io) {
@@ -1187,6 +1278,7 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
pci_unmap_rom(pdev, io);
}
vfio_pci_memory_unlock_and_restore(vdev, cmd);
+ vfio_pci_core_access_end(vdev);
} else if (pdev->rom && pdev->romlen) {
info->flags = VFIO_REGION_INFO_FLAG_READ;
/* Report BAR size as power of two. */
@@ -1242,11 +1334,29 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
}
EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);
+/*
+ * Which IRQ indexes can reach the device. ERR and REQ are software only.
+ * An index added later gets no access guard until it is listed here.
+ */
+static bool vfio_pci_irq_index_is_device(u32 index)
+{
+ switch (index) {
+ case VFIO_PCI_INTX_IRQ_INDEX:
+ case VFIO_PCI_MSI_IRQ_INDEX:
+ case VFIO_PCI_MSIX_IRQ_INDEX:
+ return true;
+ default:
+ return false;
+ }
+}
+
static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
struct vfio_irq_info __user *arg)
{
unsigned long minsz = offsetofend(struct vfio_irq_info, count);
struct vfio_irq_info info;
+ bool device_irq;
+ int ret;
if (copy_from_user(&info, arg, minsz))
return -EFAULT;
@@ -1265,7 +1375,15 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
info.flags = VFIO_IRQ_INFO_EVENTFD;
+ device_irq = vfio_pci_irq_index_is_device(info.index);
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ }
info.count = vfio_pci_get_irq_count(vdev, info.index);
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
info.flags |=
@@ -1282,13 +1400,23 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
unsigned long minsz = offsetofend(struct vfio_irq_set, count);
struct vfio_irq_set hdr;
u8 *data = NULL;
+ bool device_irq;
int max, ret = 0;
size_t data_size = 0;
if (copy_from_user(&hdr, arg, minsz))
return -EFAULT;
+ device_irq = vfio_pci_irq_index_is_device(hdr.index);
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ }
max = vfio_pci_get_irq_count(vdev, hdr.index);
+ /* Dropped for the user copy below, which can fault under userfaultfd. */
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
ret = vfio_set_irqs_validate_and_prepare(&hdr, max, VFIO_PCI_NUM_IRQS,
&data_size);
@@ -1301,25 +1429,107 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
return PTR_ERR(data);
}
+ /*
+ * Interrupt teardown reaches vfio_virqfd_disable(), which flushes the
+ * global virqfd cleanup workqueue, so recovery_lock is held here for
+ * as long as work queued by any vfio device takes. Shutdown work waits
+ * for its inject worker, and an ioeventfd inject takes that device's
+ * memory_lock, so the wait can last as long as a reset there. That is
+ * only a wait. Nothing on that workqueue takes recovery_lock, which is
+ * why the ioeventfd write path reads the recovery state without it. A
+ * callback there which used the vfio_pci_core_iowrite*() accessors
+ * would break that and deadlock against a queued writer.
+ */
+ if (device_irq) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ goto out_free;
+ }
mutex_lock(&vdev->igate);
ret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index, hdr.start,
hdr.count, data);
mutex_unlock(&vdev->igate);
+ if (device_irq)
+ vfio_pci_core_access_end(vdev);
+out_free:
kfree(data);
return ret;
}
-static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
- void __user *arg)
+/*
+ * Complete a ROM unmap which could not disable decode through config space.
+ * Call once whatever blocked access has finished. A closed device is skipped.
+ * It runs without recovery_lock, and close puts the device back through reset
+ * and config restore.
+ *
+ * The IORESOURCE_ROM_ENABLE test is what pci_unmap_rom() would have done.
+ * A ROM which firmware left enabled is not ours to turn off.
+ */
+static void vfio_pci_recovery_rom_disable(struct vfio_pci_core_device *vdev)
{
+ struct pci_dev *pdev = vdev->pdev;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ if (!vdev->pci_recovery_device_open ||
+ !READ_ONCE(vdev->pci_recovery_rom_disable))
+ return;
+
+ if (!(pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_ENABLE))
+ pci_disable_rom(pdev);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, false);
+}
+
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,
+ bool reset_power_state)
+{
+ struct pci_dev *pdev = vdev->pdev;
+ bool enabled = false;
+ bool supported = vdev->pci_recovery_supported;
int ret;
- if (!vdev->reset_works)
- return -EINVAL;
+ /*
+ * Claim the device against recovery before resetting it. The PCI
+ * error callbacks hold recovery_lock for their whole body, so taking
+ * it for writing here waits for one already running, and
+ * access_blocked keeps a later one away while the lock is dropped.
+ */
+ if (supported) {
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open) {
+ ret = -ENODEV;
+ goto out_recovery;
+ }
+
+ enabled = vdev->pci_recovery_enabled;
+ /*
+ * Only claim access_blocked when recovery is enabled.
+ * error_detected() returns early for a device which has not
+ * enabled it, so there is nothing to exclude, and claiming it
+ * anyway would fail the second of two concurrent
+ * VFIO_DEVICE_RESET calls with -EBUSY.
+ */
+ if (enabled) {
+ if (vdev->pci_recovery_access_blocked) {
+ ret = -EBUSY;
+ goto out_recovery;
+ }
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, true);
+ }
+ up_write(&vdev->recovery_lock);
+ }
+
+ /*
+ * On a device which supports recovery, taking recovery_lock for
+ * writing above waited for anything already past its access check,
+ * and if recovery is enabled access_blocked keeps new ones out. Do
+ * not hold recovery_lock while taking memory_lock or running a reset
+ * method, since a reset can take pci_bus_sem.
+ */
vfio_pci_zap_and_down_write_memory_lock(vdev);
/*
@@ -1331,17 +1541,54 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
* reset without restoring the original state (saved locally in
* 'vdev->pm_save').
*/
- vfio_pci_set_power_state(vdev, PCI_D0);
+ if (reset_power_state)
+ vfio_pci_set_power_state(vdev, PCI_D0);
vfio_pci_dma_buf_move(vdev, true);
- ret = pci_try_reset_function(vdev->pdev);
+ ret = pci_try_reset_function(pdev);
if (__vfio_pci_memory_enabled(vdev))
vfio_pci_dma_buf_move(vdev, false);
up_write(&vdev->memory_lock);
+ if (enabled) {
+ down_write(&vdev->recovery_lock);
+ /*
+ * An error callback can have started an event while the lock
+ * was down. Leave the state to it. Only unblock access for a
+ * reset which is still the one holding it.
+ */
+ if (vdev->pci_recovery_device_open &&
+ !(vdev->pci_recovery_flags & (VFIO_PCI_RECOVERY_IN_PROGRESS |
+ VFIO_PCI_RECOVERY_FAILED))) {
+ vfio_pci_recovery_rom_disable(vdev);
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ vfio_pci_intx_recovery_finish(vdev);
+ }
+ up_write(&vdev->recovery_lock);
+ /*
+ * Access is blocked for the length of the reset, so anything
+ * waiting for it to clear has to be woken here. A later patch
+ * adds the BAR fault path which waits on this.
+ */
+ wake_up_all(&vdev->pci_recovery_wait);
+ }
+
+ return ret;
+
+out_recovery:
+ up_write(&vdev->recovery_lock);
return ret;
}
+static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
+ void __user *arg)
+{
+ if (!vdev->reset_works)
+ return -EINVAL;
+
+ return vfio_pci_try_reset_function(vdev, true);
+}
+
static int vfio_pci_ioctl_get_pci_hot_reset_info(
struct vfio_pci_core_device *vdev,
struct vfio_pci_hot_reset_info __user *arg)
@@ -1607,6 +1854,106 @@ static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,
return 0;
}
+static int
+vfio_pci_core_feature_error_recovery(struct vfio_pci_core_device *vdev, u32 flags,
+ struct vfio_device_pci_error_recovery __user *arg,
+ size_t argsz)
+{
+ struct vfio_device_pci_error_recovery state = { .eventfd = -1 };
+ struct eventfd_ctx *ctx = NULL;
+ bool enable;
+ int ret;
+
+ if (!vdev->pci_recovery_supported)
+ return -ENOTTY;
+
+ ret = vfio_check_feature(flags, argsz,
+ VFIO_DEVICE_FEATURE_GET |
+ VFIO_DEVICE_FEATURE_SET, sizeof(state));
+ if (ret != 1)
+ return ret;
+
+ if (flags & VFIO_DEVICE_FEATURE_GET) {
+ down_read(&vdev->recovery_lock);
+ if (vdev->pci_recovery_enabled)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_ENABLED;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FROZEN)
+ state.flags |=
+ VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_RESET)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET;
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED)
+ state.flags |= VFIO_PCI_ERROR_RECOVERY_FAILED;
+ state.sequence = vdev->pci_recovery_sequence;
+ up_read(&vdev->recovery_lock);
+
+ if (copy_to_user(arg, &state, sizeof(state)))
+ return -EFAULT;
+ return 0;
+ }
+
+ if (copy_from_user(&state, arg, sizeof(state)))
+ return -EFAULT;
+ if (state.flags || state.sequence || state.eventfd < -1)
+ return -EINVAL;
+
+ enable = state.eventfd >= 0;
+ if (enable) {
+ ctx = eventfd_ctx_fdget(state.eventfd);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+ }
+
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open) {
+ ret = -ENODEV;
+ goto out_unlock;
+ }
+ if (vdev->pci_recovery_access_blocked) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+
+ if (!enable &&
+ (vdev->pci_recovery_flags &
+ (VFIO_PCI_RECOVERY_IN_PROGRESS | VFIO_PCI_RECOVERY_FAILED))) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+
+ mutex_lock(&vdev->igate);
+ ret = vfio_pci_eventfd_replace_locked(vdev,
+ &vdev->pci_recovery_trigger,
+ ctx);
+ mutex_unlock(&vdev->igate);
+ if (ret)
+ goto out_unlock;
+
+ WRITE_ONCE(vdev->pci_recovery_enabled, enable);
+ /*
+ * Start each enabled period from a clear state, so a sequence number
+ * and the status bits beside it always describe an event this
+ * eventfd was notified of. Nothing is in flight to lose. A
+ * transaction holds access_blocked, which failed this call with
+ * -EBUSY above.
+ *
+ * access_blocked itself is not cleared here, so userspace can never
+ * disable its way out of a block.
+ */
+ WRITE_ONCE(vdev->pci_recovery_flags, 0);
+ vdev->pci_recovery_sequence = 0;
+ vdev->pci_recovery_command_valid = false;
+
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (ret && ctx)
+ eventfd_ctx_put(ctx);
+
+ return ret;
+}
+
int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
void __user *arg, size_t argsz)
{
@@ -1627,6 +1974,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
case VFIO_DEVICE_FEATURE_ZPCI_ERROR:
return vfio_pci_zdev_feature_err(device, flags, arg, argsz);
+
+ case VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY:
+ return vfio_pci_core_feature_error_recovery(vdev, flags, arg, argsz);
default:
return -ENOTTY;
}
@@ -1642,6 +1992,24 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,
if (index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
return -EINVAL;
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
+ vfio_pci_core_access_end(vdev);
+
+ /*
+ * Resume with the guard dropped. A resume takes pci_bus_sem, through
+ * pcie_aspm_pm_state_change() and, from D3cold, through
+ * pci_bridge_wait_for_secondary_bus(). The error callbacks take
+ * recovery_lock from under pci_bus_sem, so holding it here would
+ * invert the order.
+ *
+ * The check above only avoids waking a device whose access is already
+ * blocked. A recovery which starts in between is not excluded, and
+ * does not need to be. pcie_do_recovery() has already resumed every
+ * device under the bridge and holds the reference until it finishes.
+ * The region access below takes the guard for itself.
+ */
ret = pm_runtime_resume_and_get(&vdev->pdev->dev);
if (ret) {
pci_info_ratelimited(vdev->pdev, "runtime resume failed %d\n",
@@ -1680,6 +2048,27 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf,
return ret;
}
+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return 0;
+
+ down_read(&vdev->recovery_lock);
+ if (unlikely(!vdev->pci_recovery_device_open ||
+ vdev->pci_recovery_access_blocked)) {
+ up_read(&vdev->recovery_lock);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev)
+{
+ if (vdev->pci_recovery_supported)
+ up_read(&vdev->recovery_lock);
+}
+
ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,
size_t count, loff_t *ppos)
{
@@ -1776,6 +2165,127 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
}
EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
+/*
+ * Whether a fault which found access blocked is worth retrying. Read
+ * without recovery_lock, so it is only a hint about when to look again.
+ * vfio_pci_fault_trylock_once() takes the lock and decides. Read the flags
+ * once so the two tests below see the same value. Every writer which can
+ * make this true wakes pci_recovery_wait.
+ */
+static bool vfio_pci_recovery_done(struct vfio_pci_core_device *vdev)
+{
+ u32 flags = READ_ONCE(vdev->pci_recovery_flags);
+
+ if (!READ_ONCE(vdev->pci_recovery_device_open))
+ return true;
+ if (flags & VFIO_PCI_RECOVERY_IN_PROGRESS)
+ return false;
+ if (flags & VFIO_PCI_RECOVERY_FAILED)
+ return true;
+ return !READ_ONCE(vdev->pci_recovery_access_blocked);
+}
+
+static int vfio_pci_wait_for_recovery(struct vfio_pci_core_device *vdev)
+{
+ return wait_event_killable(vdev->pci_recovery_wait,
+ vfio_pci_recovery_done(vdev));
+}
+
+/* What one look at the recovery state says the fault should do. */
+enum vfio_pci_fault_action {
+ VFIO_PCI_FAULT_PROCEED, /* returns with memory_lock held */
+ VFIO_PCI_FAULT_WAIT, /* recovery is running, may still recover */
+ VFIO_PCI_FAULT_FAIL, /* closed, or failed for good */
+};
+
+static enum vfio_pci_fault_action
+vfio_pci_fault_trylock_once(struct vfio_pci_core_device *vdev)
+{
+ enum vfio_pci_fault_action action;
+
+ down_read(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_device_open ||
+ (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED)) {
+ action = VFIO_PCI_FAULT_FAIL;
+ } else if (vdev->pci_recovery_access_blocked) {
+ /*
+ * Blocked for a reason which still ends: a recovery which has
+ * not failed, or a function reset. Test FAILED above rather
+ * than IN_PROGRESS here, so a fault does not fail for good
+ * while a reset is running, or in the window where
+ * error_detected() has blocked access but not yet published
+ * the flags.
+ */
+ action = VFIO_PCI_FAULT_WAIT;
+ } else {
+ down_read(&vdev->memory_lock);
+ action = VFIO_PCI_FAULT_PROCEED;
+ }
+ up_read(&vdev->recovery_lock);
+
+ return action;
+}
+
+/*
+ * Return true with memory_lock held for a fault that may proceed. Otherwise
+ * return false with @ret set to the result the fault handler should return.
+ */
+static bool vfio_pci_core_fault_trylock(struct vfio_pci_core_device *vdev,
+ struct vm_fault *vmf,
+ vm_fault_t *ret)
+{
+ if (!vdev->pci_recovery_supported) {
+ down_read(&vdev->memory_lock);
+ return true;
+ }
+
+ switch (vfio_pci_fault_trylock_once(vdev)) {
+ case VFIO_PCI_FAULT_PROCEED:
+ return true;
+ case VFIO_PCI_FAULT_FAIL:
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+ case VFIO_PCI_FAULT_WAIT:
+ break;
+ }
+
+ if (fault_flag_allow_retry_first(vmf->flags)) {
+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) {
+ *ret = VM_FAULT_RETRY;
+ return false;
+ }
+
+ if (!vfio_device_try_get_registration(&vdev->vdev)) {
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+ }
+
+ release_fault_lock(vmf);
+ vfio_pci_wait_for_recovery(vdev);
+ vfio_device_put_registration(&vdev->vdev);
+ *ret = VM_FAULT_RETRY;
+ return false;
+ }
+
+ /*
+ * The fault lock cannot be dropped here: either the caller did not
+ * allow a retry, or this fault has already used one. So wait with
+ * it held. It is not a deadlock. Recovery revokes mappings through
+ * unmap_mapping_range(), which never takes mmap_lock. The wait is
+ * killable.
+ */
+ if (vfio_pci_wait_for_recovery(vdev)) {
+ *ret = VM_FAULT_NOPAGE;
+ return false;
+ }
+
+ if (vfio_pci_fault_trylock_once(vdev) == VFIO_PCI_FAULT_PROCEED)
+ return true;
+
+ *ret = VM_FAULT_SIGBUS;
+ return false;
+}
+
static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
unsigned int order)
{
@@ -1787,8 +2297,11 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
vm_fault_t ret = VM_FAULT_FALLBACK;
if (is_aligned_for_order(vma, addr, pfn, order)) {
- scoped_guard(rwsem_read, &vdev->memory_lock)
- ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+ if (!vfio_pci_core_fault_trylock(vdev, vmf, &ret))
+ return ret;
+
+ ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+ up_read(&vdev->memory_lock);
}
dev_dbg_ratelimited(&vdev->pdev->dev,
@@ -2196,7 +2709,9 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
if (ret && ret != -EOPNOTSUPP)
return ret;
INIT_LIST_HEAD(&vdev->dmabufs);
+ init_rwsem(&vdev->recovery_lock);
init_rwsem(&vdev->memory_lock);
+ init_waitqueue_head(&vdev->pci_recovery_wait);
xa_init(&vdev->ctx);
return 0;
@@ -2333,22 +2848,347 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev)
}
EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device);
+static void
+vfio_pci_signal_recovery_event(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_eventfd *eventfd;
+
+ rcu_read_lock();
+ eventfd = rcu_dereference(vdev->pci_recovery_trigger);
+ if (eventfd)
+ eventfd_signal(eventfd->ctx);
+ rcu_read_unlock();
+}
+
pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,
pci_channel_state_t state)
{
struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
struct vfio_pci_eventfd *eventfd;
+ pci_ers_result_t result = PCI_ERS_RESULT_CAN_RECOVER;
+ unsigned long irq_flags;
+ bool notify_recovery = false;
+ bool terminal = false;
+ bool nested;
+ u32 flags;
+ int ret;
+
+ if (!vdev->pci_recovery_supported ||
+ !READ_ONCE(vdev->pci_recovery_enabled))
+ goto out;
+
+ down_write(&vdev->recovery_lock);
+ if (!vdev->pci_recovery_enabled)
+ goto out_unlock;
+ /*
+ * A failed device remains blocked until close and a new open have
+ * reinitialized it. A later bridge event cannot make the saved VFIO
+ * state valid again.
+ */
+ if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED) {
+ result = PCI_ERS_RESULT_NONE;
+ goto out_unlock;
+ }
+
+ if (!vdev->pci_recovery_device_open) {
+ result = PCI_ERS_RESULT_NONE;
+ /*
+ * PCI core rebroadcasts permanent failure when subtree
+ * recovery fails. Complete an event which started before
+ * close so a later open is not permanently stuck on
+ * IN_PROGRESS.
+ */
+ if (state == pci_channel_io_perm_failure &&
+ (vdev->pci_recovery_flags &
+ VFIO_PCI_RECOVERY_IN_PROGRESS)) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ terminal = true;
+ }
+ goto out_unlock;
+ }
+
+ notify_recovery = true;
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, true);
+ /*
+ * A second event before resume() has finished the first joins the
+ * transaction already running rather than starting one. Keep its
+ * sequence number, the command word it saved before the device was
+ * quiesced, and any reset a slot_reset() in between recorded. Reading
+ * the command word again here would save the quiesced value, and
+ * restoring that leaves the device with bus mastering off.
+ */
+ nested = vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS;
+ if (!nested)
+ vdev->pci_recovery_command_valid = false;
+ vfio_pci_intx_recovery_start(vdev);
+ /*
+ * INTx hardirq and virqfd callbacks cannot take recovery_lock.
+ * For devices with per-function INTx masking, mask INTx while holding
+ * irqlock so a callback which passed its blocked-state check is drained
+ * before the temporary command value is installed. Devices without
+ * per-function masking were quiesced above through genirq.
+ */
+ spin_lock_irqsave(&vdev->irqlock, irq_flags);
+ ret = 0;
+ if (state == pci_channel_io_normal && vdev->pci_2_3 && !nested) {
+ u16 command;
+
+ ret = pci_read_config_word(pdev, PCI_COMMAND,
+ &vdev->pci_recovery_command);
+ /*
+ * A read from a device which has stopped responding succeeds
+ * and returns all ones. Writing that back would set every
+ * command bit, and saving it would restore them at the end.
+ */
+ if (!ret && PCI_POSSIBLE_ERROR(vdev->pci_recovery_command))
+ ret = -EIO;
+ if (!ret) {
+ command = (vdev->pci_recovery_command &
+ ~PCI_COMMAND_MASTER) |
+ PCI_COMMAND_INTX_DISABLE;
+ ret = pci_write_config_word(pdev, PCI_COMMAND, command);
+ }
+ if (!ret)
+ vdev->pci_recovery_command_valid = true;
+ }
+ spin_unlock_irqrestore(&vdev->irqlock, irq_flags);
+ vfio_pci_zap_and_down_write_memory_lock(vdev);
+ vfio_pci_dma_buf_move(vdev, true);
+
+ /*
+ * Allocate a sequence for a new transaction, and drop the flags the
+ * previous one left behind for userspace to read. A nested event adds
+ * to the flags already there. Each path below publishes the result in
+ * one store, so a lock-free reader never observes a cleared state that
+ * looks like successful completion.
+ */
+ flags = vdev->pci_recovery_flags;
+ if (!nested) {
+ if (++vdev->pci_recovery_sequence == 0)
+ vdev->pci_recovery_sequence++;
+ flags = 0;
+ }
+
+ if (state == pci_channel_io_perm_failure) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (flags | VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ result = PCI_ERS_RESULT_DISCONNECT;
+ terminal = true;
+ goto out_memory;
+ }
+
+ if (state == pci_channel_io_frozen) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ flags | VFIO_PCI_RECOVERY_IN_PROGRESS |
+ VFIO_PCI_RECOVERY_FROZEN);
+ result = PCI_ERS_RESULT_NEED_RESET;
+ goto out_memory;
+ }
+
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ flags | VFIO_PCI_RECOVERY_IN_PROGRESS);
+ if (ret)
+ goto out_failed;
+ if (vdev->pci_2_3 || nested)
+ goto out_memory;
+
+ ret = pci_read_config_word(pdev, PCI_COMMAND,
+ &vdev->pci_recovery_command);
+ if (ret)
+ goto out_failed;
+
+ if (PCI_POSSIBLE_ERROR(vdev->pci_recovery_command)) {
+ ret = -EIO;
+ goto out_failed;
+ }
+
+ ret = pci_write_config_word(pdev, PCI_COMMAND,
+ vdev->pci_recovery_command &
+ ~PCI_COMMAND_MASTER);
+ if (ret)
+ goto out_failed;
+
+ vdev->pci_recovery_command_valid = true;
+ goto out_memory;
+
+out_failed:
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags | VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ result = PCI_ERS_RESULT_NONE;
+ terminal = true;
+out_memory:
+ up_write(&vdev->memory_lock);
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (terminal)
+ wake_up_all(&vdev->pci_recovery_wait);
+
+out:
rcu_read_lock();
eventfd = rcu_dereference(vdev->err_trigger);
if (eventfd)
eventfd_signal(eventfd->ctx);
rcu_read_unlock();
+ if (notify_recovery)
+ vfio_pci_signal_recovery_event(vdev);
- return PCI_ERS_RESULT_CAN_RECOVER;
+ return result;
}
EXPORT_SYMBOL_GPL(vfio_pci_core_aer_err_detected);
+static pci_ers_result_t vfio_pci_core_aer_slot_reset(struct pci_dev *pdev)
+{
+ struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+ pci_ers_result_t result = PCI_ERS_RESULT_RECOVERED;
+ int ret = 0;
+
+ down_write(&vdev->recovery_lock);
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS) ||
+ !vdev->pci_recovery_device_open) {
+ up_write(&vdev->recovery_lock);
+ return PCI_ERS_RESULT_NONE;
+ }
+
+ /*
+ * Restore first. aer_root_reset() resets the link with
+ * PCI_RESET_NO_RESTORE, so on entry the BARs read as zero. Tearing
+ * down MSI-X before this would have pci_msix_shutdown() write through
+ * the stale table mapping to an address the device no longer decodes.
+ */
+ pci_restore_state(pdev);
+
+ /*
+ * Hold recovery_lock across the interrupt teardown.
+ * vfio_pci_core_disable() runs the same teardown on close without
+ * taking igate, and running the per-vector teardown twice frees the
+ * irq, the name and the eventfd context twice.
+ * vfio_pci_core_prepare_close() takes recovery_lock for writing
+ * before it, so holding it here keeps the two apart.
+ */
+ mutex_lock(&vdev->igate);
+ if (vdev->irq_type < VFIO_PCI_NUM_IRQS)
+ ret = vfio_pci_set_irqs_ioctl(vdev,
+ VFIO_IRQ_SET_DATA_NONE |
+ VFIO_IRQ_SET_ACTION_TRIGGER,
+ vdev->irq_type, 0, 0, NULL);
+ mutex_unlock(&vdev->igate);
+
+ if (ret) {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ (vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_FAILED) &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ vdev->pci_recovery_command_valid = false;
+ /*
+ * Vote NONE, not DISCONNECT. A DISCONNECT anywhere in the
+ * domain makes the core skip resume() for every device under
+ * the bridge and report permanent failure for all of them.
+ * Our interrupt teardown failing says nothing about the
+ * others, so record it locally and leave the domain verdict
+ * alone.
+ */
+ result = PCI_ERS_RESULT_NONE;
+ } else {
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ vdev->pci_recovery_flags |
+ VFIO_PCI_RECOVERY_RESET);
+ }
+
+ up_write(&vdev->recovery_lock);
+ /*
+ * Whoever clears IN_PROGRESS owes the wake and the event. resume()
+ * will not do it, since it bails once IN_PROGRESS is clear, and the
+ * core skips it altogether if the domain verdict is not RECOVERED.
+ * On success the transaction carries on and resume() does both.
+ */
+ if (ret) {
+ wake_up_all(&vdev->pci_recovery_wait);
+ vfio_pci_signal_recovery_event(vdev);
+ }
+
+ return result;
+}
+
+static void vfio_pci_core_aer_resume(struct pci_dev *pdev)
+{
+ struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+ unsigned long irq_flags;
+ bool notify_recovery = false;
+ u32 flags;
+ int ret = 0;
+
+ down_write(&vdev->recovery_lock);
+
+ /*
+ * Pay any deferred ROM disable before the in-progress check below,
+ * which a failed transaction has already cleared, or it would be
+ * lost.
+ */
+ vfio_pci_recovery_rom_disable(vdev);
+
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS))
+ goto out_unlock;
+
+ notify_recovery = true;
+ if (!vdev->pci_recovery_device_open) {
+ vdev->pci_recovery_command_valid = false;
+ WRITE_ONCE(vdev->pci_recovery_flags,
+ vdev->pci_recovery_flags &
+ ~VFIO_PCI_RECOVERY_IN_PROGRESS);
+ goto out_unlock;
+ }
+
+ down_write(&vdev->memory_lock);
+ /*
+ * Restore the command word and clear access_blocked under irqlock.
+ * The INTx handler writes the same register through
+ * pci_check_and_mask_intx(), so it must not interleave with the
+ * restore, and it must not see access blocked cleared while the
+ * temporary command value is still installed.
+ *
+ * INTX_DISABLE comes from the INTx state rather than from the saved
+ * word, which can be older than the last mask. The replay below is
+ * what unmasks the line.
+ */
+ spin_lock_irqsave(&vdev->irqlock, irq_flags);
+ if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_RESET) &&
+ vdev->pci_recovery_command_valid) {
+ u16 cmd = vdev->pci_recovery_command;
+
+ cmd = vfio_pci_intx_recovery_command(vdev, cmd);
+ ret = pci_write_config_word(pdev, PCI_COMMAND, cmd);
+ }
+ if (!ret)
+ WRITE_ONCE(vdev->pci_recovery_access_blocked, false);
+ spin_unlock_irqrestore(&vdev->irqlock, irq_flags);
+ if (!ret && __vfio_pci_memory_enabled(vdev))
+ vfio_pci_dma_buf_move(vdev, false);
+ up_write(&vdev->memory_lock);
+
+ vdev->pci_recovery_command_valid = false;
+ flags = vdev->pci_recovery_flags & ~VFIO_PCI_RECOVERY_IN_PROGRESS;
+ if (ret)
+ flags |= VFIO_PCI_RECOVERY_FAILED;
+ WRITE_ONCE(vdev->pci_recovery_flags, flags);
+ if (!ret)
+ vfio_pci_intx_recovery_finish(vdev);
+
+out_unlock:
+ up_write(&vdev->recovery_lock);
+ if (notify_recovery) {
+ wake_up_all(&vdev->pci_recovery_wait);
+ vfio_pci_signal_recovery_event(vdev);
+ }
+}
+
int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
int nr_virtfn)
{
@@ -2421,6 +3261,8 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure);
const struct pci_error_handlers vfio_pci_core_err_handlers = {
.error_detected = vfio_pci_core_aer_err_detected,
+ .slot_reset = vfio_pci_core_aer_slot_reset,
+ .resume = vfio_pci_core_aer_resume,
};
EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers);
@@ -2565,6 +3407,27 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
break;
}
+ /*
+ * Between the AER callbacks memory_lock is not held, so refuse
+ * the reset on access_blocked as well. Read it rather than
+ * take recovery_lock, which would have to be released before
+ * pci_reset_bus() anyway since that reaches pci_bus_sem.
+ *
+ * Let a device which has failed for good through. Nothing is
+ * running on its behalf, and this reset covers the whole set,
+ * so refusing there would stop a healthy sibling from being
+ * reset because an unrelated device errored. The failed one
+ * stays blocked, since the reset does not unblock a device
+ * with FAILED set.
+ */
+ if (vdev->pci_recovery_supported &&
+ READ_ONCE(vdev->pci_recovery_access_blocked) &&
+ !(READ_ONCE(vdev->pci_recovery_flags) &
+ VFIO_PCI_RECOVERY_FAILED)) {
+ ret = -EBUSY;
+ break;
+ }
+
/*
* Take the memory write lock for each device and zap BAR
* mappings to prevent the user accessing the device while in
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d68..a54d199a72c92 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -243,12 +243,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
if (!get_dma_buf.nr_ranges || get_dma_buf.flags)
return -EINVAL;
- /*
- * For PCI the region_index is the BAR number like everything
- * else. Check that PCI resources have been claimed for it.
- */
- if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX ||
- IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index)))
+ /* For PCI the region_index is the BAR number like everything else. */
+ if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX)
return -ENODEV;
dma_ranges = memdup_array_user(&arg->dma_ranges, get_dma_buf.nr_ranges,
@@ -274,19 +270,30 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
priv->vdev = vdev;
priv->nr_ranges = get_dma_buf.nr_ranges;
priv->size = length;
+
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ goto err_free_phys;
+
+ /* Check that PCI resources have been claimed for the BAR. */
+ if (IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index))) {
+ ret = -ENODEV;
+ goto err_access;
+ }
+
ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
get_dma_buf.region_index,
priv->phys_vec, dma_ranges,
priv->nr_ranges);
if (ret)
- goto err_free_phys;
+ goto err_access;
kfree(dma_ranges);
dma_ranges = NULL;
if (!vfio_device_try_get_registration(&vdev->vdev)) {
ret = -ENODEV;
- goto err_free_phys;
+ goto err_access;
}
exp_info.ops = &vfio_pci_dmabuf_ops;
@@ -311,6 +318,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
dma_resv_unlock(priv->dmabuf->resv);
up_write(&vdev->memory_lock);
+ vfio_pci_core_access_end(vdev);
/*
* dma_buf_fd() consumes the reference, when the file closes the dmabuf
@@ -324,6 +332,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
err_dev_put:
vfio_device_put_registration(&vdev->vdev);
+err_access:
+ vfio_pci_core_access_end(vdev);
err_free_phys:
kfree(priv->phys_vec);
err_free_priv:
diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c
index 64f80f64ff57a..1981a64b6e181 100644
--- a/drivers/vfio/pci/vfio_pci_intrs.c
+++ b/drivers/vfio/pci/vfio_pci_intrs.c
@@ -29,6 +29,8 @@ struct vfio_pci_irq_ctx {
struct virqfd *mask;
char *name;
bool masked;
+ bool recovery_masked;
+ bool unmask_pending;
struct irq_bypass_producer producer;
};
@@ -49,6 +51,15 @@ static bool is_irq_none(struct vfio_pci_core_device *vdev)
vdev->irq_type == VFIO_PCI_MSIX_IRQ_INDEX);
}
+static bool vfio_pci_recovery_blocks_irq(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->pci_recovery_supported)
+ return false;
+
+ return READ_ONCE(vdev->pci_recovery_enabled) &&
+ READ_ONCE(vdev->pci_recovery_access_blocked);
+}
+
static
struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,
unsigned long index)
@@ -171,6 +182,16 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)
int ret = 0;
spin_lock_irqsave(&vdev->irqlock, flags);
+ /*
+ * Check for a blocked device before the INTx test below, so a blocked
+ * device is not touched through pci_intx(). @ctx is only valid when
+ * INTx is in use, so record the request only then.
+ */
+ if (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {
+ if (is_intx(vdev))
+ ctx->unmask_pending = true;
+ goto out_unlock;
+ }
/*
* Unmasking comes from ioctl or config, so again, have the
@@ -182,6 +203,9 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *data)
goto out_unlock;
}
+ ctx->unmask_pending = false;
+ ctx->recovery_masked = false;
+
if (ctx->masked && !vdev->virq_disabled) {
/*
* A pending interrupt here would immediately trigger,
@@ -220,6 +244,47 @@ void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev)
mutex_unlock(&vdev->igate);
}
+/*
+ * Mask INTx because recovery has blocked device access. Returns true if this
+ * call did the masking.
+ *
+ * Set @quiesce when recovery is masking the line itself rather than masking
+ * one delivered interrupt. Nothing is normally asserted at that point, and
+ * pci_check_and_mask_intx() only writes DisINTx when the status register says
+ * an interrupt is pending, so it would leave the line alone. The interrupt
+ * handler wants that test, since a false return there means the interrupt
+ * belongs to another device on a shared line. @quiesce also records that
+ * recovery is the one which must unmask. A masked interrupt which was
+ * delivered is the user's to unmask, exactly as outside recovery.
+ *
+ * Masking a pci_2_3 device goes through config space. If the error left
+ * config space unreadable the write has no effect and the line stays
+ * asserted, which is no worse than not trying. For a non-fatal error config
+ * space still works, and this is what keeps a shared line from storming while
+ * access is blocked.
+ */
+static bool vfio_pci_intx_mask_for_recovery(struct vfio_pci_core_device *vdev,
+ struct vfio_pci_irq_ctx *ctx,
+ bool quiesce)
+{
+ lockdep_assert_held(&vdev->irqlock);
+
+ if (ctx->masked)
+ return false;
+
+ if (!vdev->pci_2_3)
+ disable_irq_nosync(vdev->pdev->irq);
+ else if (quiesce)
+ pci_intx(vdev->pdev, 0);
+ else if (!pci_check_and_mask_intx(vdev->pdev))
+ return false;
+
+ ctx->masked = true;
+ if (quiesce)
+ ctx->recovery_masked = true;
+ return true;
+}
+
static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
{
struct vfio_pci_irq_ctx *ctx = dev_id;
@@ -228,6 +293,19 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
int ret = IRQ_NONE;
spin_lock_irqsave(&vdev->irqlock, flags);
+ if (unlikely(vfio_pci_recovery_blocks_irq(vdev))) {
+ /*
+ * Mask rather than return IRQ_NONE with the line still
+ * asserted. For a shared pci_2_3 line an unhandled level
+ * interrupt storms until note_interrupt() disables the line
+ * for every device on it, not just this one.
+ */
+ if (vfio_pci_intx_mask_for_recovery(vdev, ctx, false))
+ ret = IRQ_HANDLED;
+ else if (ctx->masked && !vdev->pci_2_3)
+ ret = IRQ_HANDLED;
+ goto out_unlock;
+ }
if (!vdev->pci_2_3) {
disable_irq_nosync(vdev->pdev->irq);
@@ -239,6 +317,7 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
ret = IRQ_HANDLED;
}
+out_unlock:
spin_unlock_irqrestore(&vdev->irqlock, flags);
if (ret == IRQ_HANDLED)
@@ -247,6 +326,86 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
return ret;
}
+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_irq_ctx *ctx;
+ unsigned long flags;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ spin_lock_irqsave(&vdev->irqlock, flags);
+ if (!is_intx(vdev))
+ goto out_unlock;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (WARN_ON_ONCE(!ctx))
+ goto out_unlock;
+
+ vfio_pci_intx_mask_for_recovery(vdev, ctx, true);
+
+out_unlock:
+ spin_unlock_irqrestore(&vdev->irqlock, flags);
+}
+
+/*
+ * Replay the masking recovery did, and any unmask which arrived while it was
+ * blocked. Call this only after access_blocked has been cleared, or the
+ * replayed unmask is swallowed and recorded as pending again with nothing
+ * left to replay it.
+ */
+/*
+ * The command word saved before the quiesce can have INTX_DISABLE clear, but
+ * the INTx handler may have masked the line since. Keep the bit as the INTx
+ * state has it, so hardware and ctx->masked agree until
+ * vfio_pci_intx_recovery_finish() replays. Restoring the saved bit instead
+ * would unmask a line the handler still believes is masked, and a shared
+ * pci_2_3 line would then storm until note_interrupt() disables it.
+ */
+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,
+ u16 command)
+{
+ struct vfio_pci_irq_ctx *ctx;
+
+ lockdep_assert_held(&vdev->irqlock);
+
+ if (!is_intx(vdev))
+ return command;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (ctx && ctx->masked)
+ command |= PCI_COMMAND_INTX_DISABLE;
+
+ return command;
+}
+
+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev)
+{
+ struct vfio_pci_irq_ctx *ctx;
+ unsigned long flags;
+ bool replay = false;
+
+ lockdep_assert_held_write(&vdev->recovery_lock);
+
+ mutex_lock(&vdev->igate);
+ spin_lock_irqsave(&vdev->irqlock, flags);
+ if (!is_intx(vdev))
+ goto out_unlock;
+
+ ctx = vfio_irq_ctx_get(vdev, 0);
+ if (WARN_ON_ONCE(!ctx))
+ goto out_unlock;
+
+ replay = ctx->recovery_masked || ctx->unmask_pending;
+ ctx->recovery_masked = false;
+ ctx->unmask_pending = false;
+
+out_unlock:
+ spin_unlock_irqrestore(&vdev->irqlock, flags);
+ if (replay)
+ __vfio_pci_intx_unmask(vdev);
+ mutex_unlock(&vdev->igate);
+}
+
static int vfio_intx_enable(struct vfio_pci_core_device *vdev,
struct eventfd_ctx *trigger)
{
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234a2eb..5598e472da4bf 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -25,6 +25,10 @@ struct vfio_pci_ioeventfd {
bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);
void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev);
+u16 vfio_pci_intx_recovery_command(struct vfio_pci_core_device *vdev,
+ u16 command);
int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
struct vfio_pci_eventfd __rcu **peventfd,
@@ -41,6 +45,9 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
char __user *buf, size_t count, loff_t *ppos,
bool iswrite);
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev,
+ bool reset_power_state);
+
ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
size_t count, loff_t *ppos, bool iswrite);
@@ -73,6 +80,9 @@ u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
u16 cmd);
+int vfio_pci_core_access_begin(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_access_end(struct vfio_pci_core_device *vdev);
+
#ifdef CONFIG_VFIO_PCI_IGD
bool vfio_pci_is_intel_display(struct pci_dev *pdev);
int vfio_pci_igd_init(struct vfio_pci_core_device *vdev);
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 7f14dd46de17a..86fadc9999621 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -42,10 +42,17 @@
int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \
bool test_mem, u##size val, void __iomem *io) \
{ \
+ int ret; \
+ \
+ ret = vfio_pci_core_access_begin(vdev); \
+ if (ret) \
+ return ret; \
+ \
if (test_mem) { \
down_read(&vdev->memory_lock); \
if (!__vfio_pci_memory_enabled(vdev)) { \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
return -EIO; \
} \
} \
@@ -54,6 +61,7 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \
\
if (test_mem) \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
\
return 0; \
} \
@@ -68,10 +76,17 @@ VFIO_IOWRITE(64)
int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \
bool test_mem, u##size *val, void __iomem *io) \
{ \
+ int ret; \
+ \
+ ret = vfio_pci_core_access_begin(vdev); \
+ if (ret) \
+ return ret; \
+ \
if (test_mem) { \
down_read(&vdev->memory_lock); \
if (!__vfio_pci_memory_enabled(vdev)) { \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
return -EIO; \
} \
} \
@@ -80,6 +95,7 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \
\
if (test_mem) \
up_read(&vdev->memory_lock); \
+ vfio_pci_core_access_end(vdev); \
\
return 0; \
} \
@@ -198,12 +214,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
}
EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
+/*
+ * Undo pci_map_rom(). The iounmap is always safe, but pci_disable_rom() is a
+ * config space write. If recovery has blocked access, do the iounmap now and
+ * record the disable, for whichever of resume() or the reset tail unblocks
+ * access again. recovery_lock spans the decision and the record so recovery
+ * cannot complete in between.
+ */
+static void vfio_pci_unmap_rom(struct vfio_pci_core_device *vdev,
+ void __iomem *io)
+{
+ struct pci_dev *pdev = vdev->pdev;
+
+ if (!vdev->pci_recovery_supported) {
+ pci_unmap_rom(pdev, io);
+ return;
+ }
+
+ down_read(&vdev->recovery_lock);
+ if (vdev->pci_recovery_device_open &&
+ !vdev->pci_recovery_access_blocked) {
+ pci_unmap_rom(pdev, io);
+ } else {
+ iounmap(io);
+ WRITE_ONCE(vdev->pci_recovery_rom_disable, true);
+ }
+ up_read(&vdev->recovery_lock);
+}
+
ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
size_t count, loff_t *ppos, bool iswrite)
{
struct pci_dev *pdev = vdev->pdev;
loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
int bar = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+ int ret;
size_t x_start = 0, x_end = 0;
resource_size_t end;
void __iomem *io;
@@ -230,7 +275,11 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
* filling large ROM BARs much faster.
*/
if (pci_resource_start(pdev, bar)) {
+ ret = vfio_pci_core_access_begin(vdev);
+ if (ret)
+ return ret;
io = pci_map_rom(pdev, &x_start);
+ vfio_pci_core_access_end(vdev);
} else {
io = ioremap(pdev->rom, pdev->romlen);
x_start = pdev->romlen;
@@ -269,7 +318,7 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
if (bar == PCI_ROM_RESOURCE) {
if (pci_resource_start(pdev, bar))
- pci_unmap_rom(pdev, io);
+ vfio_pci_unmap_rom(vdev, io);
else
iounmap(io);
}
@@ -349,56 +398,85 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
}
#endif
-static void vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,
- bool test_mem)
+static int vfio_pci_ioeventfd_do_write(struct vfio_pci_ioeventfd *ioeventfd,
+ bool trylock)
{
+ struct vfio_pci_core_device *vdev = ioeventfd->vdev;
+
+ if (ioeventfd->test_mem) {
+ if (trylock) {
+ if (!down_read_trylock(&vdev->memory_lock))
+ return 1; /* Lock contended, use thread */
+ } else {
+ down_read(&vdev->memory_lock);
+ }
+ }
+
+ /*
+ * Read the recovery state lock-free rather than under recovery_lock.
+ * This path runs from the virqfd cleanup workqueue, which is flushed
+ * from paths that take recovery_lock for reading, so blocking on it
+ * here would deadlock behind a queued writer.
+ *
+ * For a memory BAR, a blocked device still waits for a write already
+ * under way, through memory_lock. The lock is taken above before the
+ * flag is read, so a write which saw the flag clear is already
+ * holding the read side, and the blocker waits for it when it takes
+ * memory_lock for writing. An I/O port BAR takes no memory_lock, so
+ * a write which saw the flag clear can still land afterwards. Port
+ * writes are best effort here.
+ *
+ * A write can also be dropped for a short while after a reset has
+ * finished, since VFIO_DEVICE_RESET releases memory_lock before it
+ * retakes recovery_lock to unblock access. Closing that would mean
+ * taking recovery_lock inside memory_lock, which is the wrong way
+ * round.
+ *
+ * pci_recovery_device_open records that the recovery machinery is
+ * live, so it is only ever set for drivers which advertise support.
+ * Testing it unconditionally would drop every write for every other
+ * driver.
+ *
+ * The raw vfio_iowrite*() accessors below are used for the same
+ * reason. This path must not take recovery_lock.
+ */
+ if (vdev->pci_recovery_supported &&
+ (!READ_ONCE(vdev->pci_recovery_device_open) ||
+ READ_ONCE(vdev->pci_recovery_access_blocked)))
+ goto out_memory;
+
+ if (ioeventfd->test_mem && !__vfio_pci_memory_enabled(vdev))
+ goto out_memory;
+
switch (ioeventfd->count) {
case 1:
- vfio_pci_core_iowrite8(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite8(ioeventfd->data, ioeventfd->addr);
break;
case 2:
- vfio_pci_core_iowrite16(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite16(ioeventfd->data, ioeventfd->addr);
break;
case 4:
- vfio_pci_core_iowrite32(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite32(ioeventfd->data, ioeventfd->addr);
break;
case 8:
- vfio_pci_core_iowrite64(ioeventfd->vdev, test_mem,
- ioeventfd->data, ioeventfd->addr);
+ vfio_iowrite64(ioeventfd->data, ioeventfd->addr);
break;
}
-}
-
-static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)
-{
- struct vfio_pci_ioeventfd *ioeventfd = opaque;
- struct vfio_pci_core_device *vdev = ioeventfd->vdev;
-
- if (ioeventfd->test_mem) {
- if (!down_read_trylock(&vdev->memory_lock))
- return 1; /* Lock contended, use thread */
- if (!__vfio_pci_memory_enabled(vdev)) {
- up_read(&vdev->memory_lock);
- return 0;
- }
- }
-
- vfio_pci_ioeventfd_do_write(ioeventfd, false);
+out_memory:
if (ioeventfd->test_mem)
up_read(&vdev->memory_lock);
-
return 0;
}
-static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)
+static int vfio_pci_ioeventfd_handler(void *opaque, void *unused)
{
- struct vfio_pci_ioeventfd *ioeventfd = opaque;
+ return vfio_pci_ioeventfd_do_write(opaque, true);
+}
- vfio_pci_ioeventfd_do_write(ioeventfd, ioeventfd->test_mem);
+static void vfio_pci_ioeventfd_thread(void *opaque, void *unused)
+{
+ vfio_pci_ioeventfd_do_write(opaque, false);
}
int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9a1674c152aa2..fe42089be3fc0 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -95,6 +95,11 @@ static inline int vfio_pci_core_get_dmabuf_phys(
}
#endif
+#define VFIO_PCI_RECOVERY_IN_PROGRESS BIT(0)
+#define VFIO_PCI_RECOVERY_FROZEN BIT(1)
+#define VFIO_PCI_RECOVERY_RESET BIT(2)
+#define VFIO_PCI_RECOVERY_FAILED BIT(3)
+
struct vfio_pci_core_device {
struct vfio_device vdev;
struct pci_dev *pdev;
@@ -129,6 +134,7 @@ struct vfio_pci_core_device {
bool disable_idle_d3:1;
bool nointxmask:1;
bool disable_vga:1;
+ bool pci_recovery_supported:1;
/* Flags modified at runtime - dedicated storage unit */
bool needs_reset;
bool pm_intx_masked;
@@ -139,6 +145,7 @@ struct vfio_pci_core_device {
int ioeventfds_nr;
struct vfio_pci_eventfd __rcu *err_trigger;
struct vfio_pci_eventfd __rcu *req_trigger;
+ struct vfio_pci_eventfd __rcu *pci_recovery_trigger;
struct eventfd_ctx *pm_wake_eventfd_ctx;
struct list_head dummy_resources_list;
struct mutex ioeventfds_lock;
@@ -147,7 +154,60 @@ struct vfio_pci_core_device {
struct list_head sriov_pfs_item;
struct vfio_pci_core_device *sriov_pf_core_dev;
struct notifier_block nb;
+ /*
+ * Serializes host PCI error recovery with device access and the
+ * open/close lifecycle. recovery_lock nests outside memory_lock.
+ */
+ struct rw_semaphore recovery_lock;
struct rw_semaphore memory_lock;
+ /*
+ * PCI error recovery state, written under recovery_lock held for
+ * writing except where noted.
+ *
+ * Some readers cannot take recovery_lock. An interrupt handler cannot
+ * sleep. The ioeventfd write runs on a workqueue which is flushed with
+ * the lock held, so it would block behind a queued writer. A
+ * wait-queue condition cannot take it either. Some checks would refuse
+ * work against ordinary device traffic if they took it. Work deferred
+ * past the guard has none to take, since it reaches pci_bus_sem.
+ *
+ * Those readers use READ_ONCE() on pci_recovery_flags,
+ * pci_recovery_enabled, pci_recovery_access_blocked and
+ * pci_recovery_device_open instead. All of them fail safe. A stale
+ * read costs an extra refusal or retry, never an unguarded access.
+ *
+ * recovery_lock does not exclude those readers, so publish each field
+ * with a single store of its final value rather than clearing and
+ * then setting. That keeps the states a reader can observe to ones
+ * which are meaningful on their own.
+ *
+ * device_open and access_blocked mean different things. device_open
+ * says the device is open and its per-open state, vconfig included,
+ * is allocated. access_blocked says a recovery or reset is blocking
+ * access right now.
+ *
+ * access_blocked is only ever set while device_open is set. Nothing
+ * sets it without testing device_open first under recovery_lock, and
+ * close clears access_blocked before it clears device_open, so a
+ * block never outlives the open which created it. A reader which
+ * finds access_blocked set can rely on the per-open state being
+ * there with it.
+ */
+ u32 pci_recovery_flags;
+ u64 pci_recovery_sequence;
+ /* PCI_COMMAND value saved before recovery quiesces the device. */
+ u16 pci_recovery_command;
+ /* Userspace enabled recovery for this device open. */
+ bool pci_recovery_enabled;
+ /* pci_recovery_command contains a restorable value. */
+ bool pci_recovery_command_valid;
+ /* A recovery or reset transaction is blocking physical access. */
+ bool pci_recovery_access_blocked;
+ /* Device initialization completed and close teardown has not started. */
+ bool pci_recovery_device_open;
+ /* May be set while recovery_lock is held for reading during ROM unmap. */
+ bool pci_recovery_rom_disable;
+ wait_queue_head_t pci_recovery_wait;
struct list_head dmabufs;
};
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index e41437fa17ad0..ce0cda2dcbbb1 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1555,6 +1555,75 @@ struct vfio_device_feature_zpci_err {
#define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13
+/*
+ * Report host PCI error recovery state for this device.
+ *
+ * The sequence number is incremented at the start of each event and remains
+ * unchanged for its subsequent state changes. Userspace can therefore
+ * distinguish a new event from completion of the current one and detect
+ * coalesced notifications. It restarts from zero each time recovery is
+ * enabled, so it is only meaningful within one enabled period.
+ *
+ * ENABLED reports that userspace has enabled recovery.
+ * CHANNEL_FROZEN records that recovery started with the PCI channel frozen.
+ * DEVICE_RESET records that the host reset the device. FAILED records that
+ * recovery did not complete successfully. Event status bits remain set after
+ * IN_PROGRESS is cleared. A new event supersedes status from a previous
+ * successful event. FAILED is terminal for the current device open and
+ * remains set until the device is closed and reopened.
+ *
+ * Status bits may also be set while IN_PROGRESS is still set, describing the
+ * event so far. Act on them once IN_PROGRESS is clear. Device access is
+ * refused with -EIO until then.
+ *
+ * When DEVICE_RESET is reported the host reset the device, which tears down
+ * the interrupt configuration the user had established. INTx, MSI and MSI-X
+ * must be re-armed with VFIO_DEVICE_SET_IRQS before interrupts resume.
+ *
+ * VFIO_DEVICE_FEATURE_GET returns the current state and -1 in eventfd. GET is
+ * never refused, including while recovery blocks device access, so that
+ * userspace can read this state during an event. It can wait for a recovery
+ * callback which is already running.
+ *
+ * IN_PROGRESS is not guaranteed to be observable. A recovery which needs no
+ * device reset can complete within microseconds of the notification, before
+ * userspace is scheduled, so a GET which follows the eventfd may already see
+ * IN_PROGRESS clear. Userspace must treat a notification as "an event
+ * occurred" and read the sequence number and the status bits to learn what
+ * happened. It must not wait for IN_PROGRESS to appear set.
+ *
+ * VFIO_DEVICE_FEATURE_SET with a valid eventfd enables recovery
+ * and installs the eventfd as a notification for recovery start and terminal
+ * completion. SET with eventfd -1 disables recovery when none
+ * is in progress and the latest event has not failed. SET returns -EBUSY when
+ * any of those restrictions prevents the requested transition, including while
+ * an explicit VFIO_DEVICE_RESET blocks device access, and -ENODEV if device
+ * close has begun. flags and sequence must be zero for SET.
+ *
+ * This eventfd is separate from VFIO_PCI_ERR_IRQ_INDEX and additional to it.
+ * VFIO_PCI_ERR_IRQ_INDEX keeps reporting errors as it does today whether or
+ * not this feature is enabled, so a user of both receives two notifications
+ * for one event.
+ *
+ * Enabling recovery does not recover an event which is already being handled
+ * for this device. Such an event was declined before it started, so it
+ * completes without notification and without status, even though the host may
+ * reset the device as part of it. Enable recovery before errors occur rather
+ * than in response to one.
+ */
+struct vfio_device_pci_error_recovery {
+ __u32 flags;
+#define VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS (1U << 0)
+#define VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN (1U << 1)
+#define VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET (1U << 2)
+#define VFIO_PCI_ERROR_RECOVERY_FAILED (1U << 3)
+#define VFIO_PCI_ERROR_RECOVERY_ENABLED (1U << 4)
+ __s32 eventfd;
+ __aligned_u64 sequence;
+};
+
+#define VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY 14
+
/* -------- API for Type1 VFIO IOMMU -------- */
/**
|
| 21/3 | 2026/09/01 22:29 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 31512 output: 20 thoughts: 1350 Thoughts: **KMSAN/KASAN Analysis of VFIO PCI Error Recovery Patch** Okay, let's dive into this patch with a focus on potential memory safety issues that KMSAN and KASAN might flag. First, I need to understand what this patch actually *does*. It's adding PCI error recovery support to vfio-pci, which immediately raises a flag for potential complexities in state management and concurrency. The core changes are in `vfio_pci_config.c`, `vfio_pci_core.c`, `vfio_pci_dmabuf.c`, `vfio_pci_intrs.c`, and `vfio_pci_rdwr.c`. It looks like there's a new structure, `vfio_pci_config_deferred`, and access guards are being introduced using `vfio_pci_core_access_begin` and `vfio_pci_core_access_end`. The use of `recovery_lock` along with `pci_recovery_flags`, `pci_recovery_sequence`, and `pci_recovery_wait` in `vfio_pci_core.c` clearly indicates this is related to error recovery and concurrent access. The new feature `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY` uses a structure `vfio_device_pci_error_recovery`. Now, the critical part for memory safety: the `VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY`'s GET operation. It copies data to userspace using `copy_to_user`. I'm going to scrutinize this very closely. I see that the `state` struct is initialized with `.eventfd = -1`, which is C's default zero initialization, and then the flags and sequence are populated. I also verified the struct's layout to ensure there are no padding holes. That seems okay, the struct's layout is clean. The important thing is that `state` *is* initialized *before* the data is copied to userspace. All the fields are explicitly set *before* the `copy_to_user` call, which is good. The other changes are wrapping IO operations in access begin/end. Okay, let's be thorough. I'll search the diff for *any* other instances of `copy_to_user`, `put_user`, netlink skb usage, or potentially uninitialized variables. Looks like I've found what I was looking for, in the `vfio_pci_core_feature_error_recovery` block, I've confirmed that the struct being sent to the userspace is first zero initialized and that all fields are explicitly populated *before* being sent to the userspace. It seems the `state` variable is initialized before being copied. I'll need to check `vfio_check_feature` though. |
| 22/3 | 2026/09/01 22:29 | tool | grepper |
0mArgs:{
"Expression": "vfio_check_feature"
}Results: {
"Output": "drivers/vfio/cdx/main.c=84=static int vfio_cdx_bm_ctrl(struct vfio_device *core_vdev, u32 flags,\n--\ndrivers/vfio/cdx/main.c-97-\ndrivers/vfio/cdx/main.c:98:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET,\ndrivers/vfio/cdx/main.c-99-\t\t\t\t sizeof(ops));\n--\ndrivers/vfio/pci/vfio_pci_core.c=402=static int vfio_pci_core_pm_entry(struct vfio_pci_core_device *vdev, u32 flags,\n--\ndrivers/vfio/pci/vfio_pci_core.c-406-\ndrivers/vfio/pci/vfio_pci_core.c:407:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);\ndrivers/vfio/pci/vfio_pci_core.c-408-\tif (ret != 1)\n--\ndrivers/vfio/pci/vfio_pci_core.c=420=static int vfio_pci_core_pm_entry_with_wakeup(\n--\ndrivers/vfio/pci/vfio_pci_core.c-428-\ndrivers/vfio/pci/vfio_pci_core.c:429:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET,\ndrivers/vfio/pci/vfio_pci_core.c-430-\t\t\t\t sizeof(entry));\n--\ndrivers/vfio/pci/vfio_pci_core.c=477=static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,\n--\ndrivers/vfio/pci/vfio_pci_core.c-481-\ndrivers/vfio/pci/vfio_pci_core.c:482:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);\ndrivers/vfio/pci/vfio_pci_core.c-483-\tif (ret != 1)\n--\ndrivers/vfio/pci/vfio_pci_core.c=1830=static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,\n--\ndrivers/vfio/pci/vfio_pci_core.c-1842-\t */\ndrivers/vfio/pci/vfio_pci_core.c:1843:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET,\ndrivers/vfio/pci/vfio_pci_core.c-1844-\t\t\t\t sizeof(uuid));\n--\ndrivers/vfio/pci/vfio_pci_core.c=1858=vfio_pci_core_feature_error_recovery(struct vfio_pci_core_device *vdev, u32 flags,\n--\ndrivers/vfio/pci/vfio_pci_core.c-1869-\ndrivers/vfio/pci/vfio_pci_core.c:1870:\tret = vfio_check_feature(flags, argsz,\ndrivers/vfio/pci/vfio_pci_core.c-1871-\t\t\t\t VFIO_DEVICE_FEATURE_GET |\n--\ndrivers/vfio/pci/vfio_pci_dmabuf.c=221=int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,\n--\ndrivers/vfio/pci/vfio_pci_dmabuf.c-234-\ndrivers/vfio/pci/vfio_pci_dmabuf.c:235:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,\ndrivers/vfio/pci/vfio_pci_dmabuf.c-236-\t\t\t\t sizeof(get_dma_buf));\n--\ndrivers/vfio/pci/vfio_pci_zdev.c=145=int vfio_pci_zdev_feature_err(struct vfio_device *device, u32 flags,\n--\ndrivers/vfio/pci/vfio_pci_zdev.c-158-\ndrivers/vfio/pci/vfio_pci_zdev.c:159:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,\ndrivers/vfio/pci/vfio_pci_zdev.c-160-\t\t\t\t sizeof(err));\n--\ndrivers/vfio/vfio_main.c=931=vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-943-\ndrivers/vfio/vfio_main.c:944:\tret = vfio_check_feature(flags, argsz,\ndrivers/vfio/vfio_main.c-945-\t\t\t\t VFIO_DEVICE_FEATURE_SET |\n--\ndrivers/vfio/vfio_main.c=981=vfio_ioctl_device_feature_migration_data_size(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-991-\ndrivers/vfio/vfio_main.c:992:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,\ndrivers/vfio/vfio_main.c-993-\t\t\t\t sizeof(data_size));\n--\ndrivers/vfio/vfio_main.c=1009=vfio_ioctl_device_feature_migration_precopy_info_v2(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-1016-\ndrivers/vfio/vfio_main.c:1017:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);\ndrivers/vfio/vfio_main.c-1018-\tif (ret != 1)\n--\ndrivers/vfio/vfio_main.c=1025=static int vfio_ioctl_device_feature_migration(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-1036-\ndrivers/vfio/vfio_main.c:1037:\tret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,\ndrivers/vfio/vfio_main.c-1038-\t\t\t\t sizeof(mig));\n--\ndrivers/vfio/vfio_main.c=1108=vfio_ioctl_device_feature_logging_start(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-1126-\ndrivers/vfio/vfio_main.c:1127:\tret = vfio_check_feature(flags, argsz,\ndrivers/vfio/vfio_main.c-1128-\t\t\t\t VFIO_DEVICE_FEATURE_SET,\n--\ndrivers/vfio/vfio_main.c=1192=vfio_ioctl_device_feature_logging_stop(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-1200-\ndrivers/vfio/vfio_main.c:1201:\tret = vfio_check_feature(flags, argsz,\ndrivers/vfio/vfio_main.c-1202-\t\t\t\t VFIO_DEVICE_FEATURE_SET, 0);\n--\ndrivers/vfio/vfio_main.c=1219=vfio_ioctl_device_feature_logging_report(struct vfio_device *device,\n--\ndrivers/vfio/vfio_main.c-1233-\ndrivers/vfio/vfio_main.c:1234:\tret = vfio_check_feature(flags, argsz,\ndrivers/vfio/vfio_main.c-1235-\t\t\t\t VFIO_DEVICE_FEATURE_GET,\n--\ninclude/linux/vfio.h=250=struct vfio_log_ops {\n--\ninclude/linux/vfio.h-259-/**\ninclude/linux/vfio.h:260: * vfio_check_feature - Validate user input for the VFIO_DEVICE_FEATURE ioctl\ninclude/linux/vfio.h-261- * @flags: Arg from the device_feature op\n--\ninclude/linux/vfio.h-271- */\ninclude/linux/vfio.h:272:static inline int vfio_check_feature(u32 flags, size_t argsz, u32 supported_ops,\ninclude/linux/vfio.h-273-\t\t\t\t size_t minsz)\n"
} |
| 23/3 | 2026/09/01 22:29 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 35817 output: 394 thoughts: 1585 Thoughts: **VFIO PCI Error Recovery Analysis: No KMSAN Needed** Okay, let's break down this patch and determine if it warrants a KMSAN check. My initial assessment leans towards "no," but let's be methodical. First, I examine how `vfio_pci_config_rw_single` handles read operations. The `deferred` struct and `val` are zero-initialized. The `perm->readfn` is where the config data is read from, the diff doesn't seem to modify the read functions, only adding `access_begin` and `access_end`. Therefore, these functions should be safe. Next, I look at the new variables and structure members introduced within `struct vfio_pci_core_device` and `struct vfio_pci_irq_ctx`. Key point: all new members are either explicitly initialized during device setup (e.g., `pci_recovery_supported`, `recovery_lock`, `pci_recovery_wait`) or zero-initialized (e.g., the `pci_recovery_trigger` pointer in `vfio_pci_core_device` or `recovery_masked` in `vfio_pci_irq_ctx`). This initialization happens with either kzalloc or kcalloc, so no problems are expected here. The flags and sequence variables are explicitly managed with `WRITE_ONCE`. Then I check the `vfio_pci_fault_trylock_once` and `vfio_pci_core_fault_trylock` functions, along with their usage in `vfio_pci_mmap_huge_fault`. The trylock returns a boolean and sets the return value in all branches that would not proceed, therefore the error value should always be set. After that, I'm verifying the standard AER recovery handlers like `vfio_pci_aer_err_detected`, `vfio_pci_core_aer_slot_reset`, and `vfio_pci_core_aer_resume`. These are standard kernel functions. No memory leaks or uninitialized memory reads are apparent. Furthermore, the `struct vfio_device_pci_error_recovery` is fully initialized before being copied to userspace. I then review all the ioctls and user-facing interfaces affected by this patch. Specifically: `vfio_pci_ioctl_get_info`, `vfio_pci_ioctl_get_region_info`, `vfio_pci_ioctl_get_irq_info`, `vfio_pci_ioctl_set_irqs`, `vfio_pci_ioctl_reset`, and `vfio_pci_core_feature_dma_buf`. In each case, added access checks are noted. No uninitialized memory usage, info leaks, or uninitialized padding risks are found. I'm confident because the changes focus on synchronization, lock order inversion fixes, state tracking, and error recovery flow. Bugs here would likely manifest as deadlocks, race conditions, or logic errors, which LOCKDEP, KASAN, and the standard kernel debuggers should catch. To double-check: 1. Are there new kernel structures sent to userspace, where uninitialized fields or padding *could* be problematic? No. `vfio_device_pci_error_recovery` is initialized before copying. Its layout is straightforward, with no padding. 2. Does the patch depend on potentially uninitialized variables? No, all conditional logic checks are initialized properly. 3. Are there complex data structures with uninitialized fields read later in reachable paths? No. 4. Are there bounds checks/lengths that could enable uninitialized buffer accesses? No. Therefore, `NeedsKMSAN` is `false`. Let's prepare that tool call to `set-results`. |
| 24/3 | 2026/09/01 22:29 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch introduces PCI error recovery support into the VFIO PCI driver, managing locking (recovery_lock vs memory_lock) and synchronization during AER events, resets, and power transitions.\n\nAnalysis of potential uninitialized memory risks:\n1. Kernel structure copied to user space:\n - In `vfio_pci_core_feature_error_recovery()`, `struct vfio_device_pci_error_recovery` is copied to userspace via `copy_to_user()`. The structure consists of `__u32 flags` (4 bytes), `__s32 eventfd` (4 bytes), and `__aligned_u64 sequence` (8 bytes), totaling 16 bytes with zero padding holes. The local variable `state` is initialized with `= { .eventfd = -1 }`, which zeroes all unmentioned fields, and then populated with the recovery state before being copied. No uninitialized bytes or padding are exposed.\n2. Stack/heap variables:\n - `struct vfio_pci_config_deferred deferred = {};` is zero-initialized on the stack.\n - All newly added struct fields in `struct vfio_pci_core_device` and `struct vfio_pci_irq_ctx` are initialized upon device/context allocation.\n3. Control flow and locking:\n - The patch primarily introduces access gating (`vfio_pci_core_access_begin`/`end`), wait queues, and error recovery handlers. Potential concurrency or locking bugs (such as deadlocks or race conditions) are within the domain of LOCKDEP and KASAN.\n\nThere are no uninitialized memory reads or information leak risks that would require a dedicated KMSAN fuzzing session.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch introduces PCI error recovery support into the VFIO PCI driver, managing locking (recovery_lock vs memory_lock) and synchronization during AER events, resets, and power transitions.\n\nAnalysis of potential uninitialized memory risks:\n1. Kernel structure copied to user space:\n - In `vfio_pci_core_feature_error_recovery()`, `struct vfio_device_pci_error_recovery` is copied to userspace via `copy_to_user()`. The structure consists of `__u32 flags` (4 bytes), `__s32 eventfd` (4 bytes), and `__aligned_u64 sequence` (8 bytes), totaling 16 bytes with zero padding holes. The local variable `state` is initialized with `= { .eventfd = -1 }`, which zeroes all unmentioned fields, and then populated with the recovery state before being copied. No uninitialized bytes or padding are exposed.\n2. Stack/heap variables:\n - `struct vfio_pci_config_deferred deferred = {};` is zero-initialized on the stack.\n - All newly added struct fields in `struct vfio_pci_core_device` and `struct vfio_pci_irq_ctx` are initialized upon device/context allocation.\n3. Control flow and locking:\n - The patch primarily introduces access gating (`vfio_pci_core_access_begin`/`end`), wait queues, and error recovery handlers. Potential concurrency or locking bugs (such as deadlocks or race conditions) are within the domain of LOCKDEP and KASAN.\n\nThere are no uninitialized memory reads or information leak risks that would require a dedicated KMSAN fuzzing session.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|