Add a VFIO device feature that reports whether host PCI recovery is enabled, whether recovery is in progress, whether the channel was frozen, whether the host reset the device, and whether recovery failed. A sequence number lets userspace distinguish events. Installing a dedicated recovery eventfd enables recovery. The eventfd is additional to VFIO_PCI_ERR_IRQ_INDEX, which keeps reporting errors as it does today either way. Recovery can be disabled only when not in progress and before terminal failure. Reject both enable and disable while device access is blocked so feature changes cannot race lifecycle teardown or an explicit reset. Clear the sequence number and the status bits whichever way the feature is being changed, so a sequence number always describes an event the eventfd holding it was notified of. A user which replaces the eventfd would otherwise read status for an event it never heard about. Variant drivers return -ENOTTY as they do not advertise recovery support. Signed-off-by: Shameer Kolothum --- include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 69 +++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 142 +++++++++++++++++++++++++++++-- 3 files changed, 206 insertions(+), 6 deletions(-) diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 42a77ed6b93c..fe42089be3fc 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -145,6 +145,7 @@ struct vfio_pci_core_device { int ioeventfds_nr; struct vfio_pci_eventfd __rcu *err_trigger; struct vfio_pci_eventfd __rcu *req_trigger; + struct vfio_pci_eventfd __rcu *pci_recovery_trigger; struct eventfd_ctx *pm_wake_eventfd_ctx; struct list_head dummy_resources_list; struct mutex ioeventfds_lock; diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index e41437fa17ad..ce0cda2dcbbb 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -1555,6 +1555,75 @@ struct vfio_device_feature_zpci_err { #define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13 +/* + * Report host PCI error recovery state for this device. + * + * The sequence number is incremented at the start of each event and remains + * unchanged for its subsequent state changes. Userspace can therefore + * distinguish a new event from completion of the current one and detect + * coalesced notifications. It restarts from zero each time recovery is + * enabled, so it is only meaningful within one enabled period. + * + * ENABLED reports that userspace has enabled recovery. + * CHANNEL_FROZEN records that recovery started with the PCI channel frozen. + * DEVICE_RESET records that the host reset the device. FAILED records that + * recovery did not complete successfully. Event status bits remain set after + * IN_PROGRESS is cleared. A new event supersedes status from a previous + * successful event. FAILED is terminal for the current device open and + * remains set until the device is closed and reopened. + * + * Status bits may also be set while IN_PROGRESS is still set, describing the + * event so far. Act on them once IN_PROGRESS is clear. Device access is + * refused with -EIO until then. + * + * When DEVICE_RESET is reported the host reset the device, which tears down + * the interrupt configuration the user had established. INTx, MSI and MSI-X + * must be re-armed with VFIO_DEVICE_SET_IRQS before interrupts resume. + * + * VFIO_DEVICE_FEATURE_GET returns the current state and -1 in eventfd. GET is + * never refused, including while recovery blocks device access, so that + * userspace can read this state during an event. It can wait for a recovery + * callback which is already running. + * + * IN_PROGRESS is not guaranteed to be observable. A recovery which needs no + * device reset can complete within microseconds of the notification, before + * userspace is scheduled, so a GET which follows the eventfd may already see + * IN_PROGRESS clear. Userspace must treat a notification as "an event + * occurred" and read the sequence number and the status bits to learn what + * happened. It must not wait for IN_PROGRESS to appear set. + * + * VFIO_DEVICE_FEATURE_SET with a valid eventfd enables recovery + * and installs the eventfd as a notification for recovery start and terminal + * completion. SET with eventfd -1 disables recovery when none + * is in progress and the latest event has not failed. SET returns -EBUSY when + * any of those restrictions prevents the requested transition, including while + * an explicit VFIO_DEVICE_RESET blocks device access, and -ENODEV if device + * close has begun. flags and sequence must be zero for SET. + * + * This eventfd is separate from VFIO_PCI_ERR_IRQ_INDEX and additional to it. + * VFIO_PCI_ERR_IRQ_INDEX keeps reporting errors as it does today whether or + * not this feature is enabled, so a user of both receives two notifications + * for one event. + * + * Enabling recovery does not recover an event which is already being handled + * for this device. Such an event was declined before it started, so it + * completes without notification and without status, even though the host may + * reset the device as part of it. Enable recovery before errors occur rather + * than in response to one. + */ +struct vfio_device_pci_error_recovery { + __u32 flags; +#define VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS (1U << 0) +#define VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN (1U << 1) +#define VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET (1U << 2) +#define VFIO_PCI_ERROR_RECOVERY_FAILED (1U << 3) +#define VFIO_PCI_ERROR_RECOVERY_ENABLED (1U << 4) + __s32 eventfd; + __aligned_u64 sequence; +}; + +#define VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY 14 + /* -------- API for Type1 VFIO IOMMU -------- */ /** diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2d757d6a5fe1..c1ea3c868fc5 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -893,6 +893,10 @@ static void vfio_pci_core_finish_close(struct vfio_device *core_vdev) mutex_lock(&vdev->igate); vfio_pci_eventfd_replace_locked(vdev, &vdev->err_trigger, NULL); vfio_pci_eventfd_replace_locked(vdev, &vdev->req_trigger, NULL); + if (vdev->pci_recovery_supported) + vfio_pci_eventfd_replace_locked(vdev, + &vdev->pci_recovery_trigger, + NULL); mutex_unlock(&vdev->igate); } @@ -1850,6 +1854,106 @@ static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev, return 0; } +static int +vfio_pci_core_feature_error_recovery(struct vfio_pci_core_device *vdev, u32 flags, + struct vfio_device_pci_error_recovery __user *arg, + size_t argsz) +{ + struct vfio_device_pci_error_recovery state = { .eventfd = -1 }; + struct eventfd_ctx *ctx = NULL; + bool enable; + int ret; + + if (!vdev->pci_recovery_supported) + return -ENOTTY; + + ret = vfio_check_feature(flags, argsz, + VFIO_DEVICE_FEATURE_GET | + VFIO_DEVICE_FEATURE_SET, sizeof(state)); + if (ret != 1) + return ret; + + if (flags & VFIO_DEVICE_FEATURE_GET) { + down_read(&vdev->recovery_lock); + if (vdev->pci_recovery_enabled) + state.flags |= VFIO_PCI_ERROR_RECOVERY_ENABLED; + if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS) + state.flags |= VFIO_PCI_ERROR_RECOVERY_IN_PROGRESS; + if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FROZEN) + state.flags |= + VFIO_PCI_ERROR_RECOVERY_CHANNEL_FROZEN; + if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_RESET) + state.flags |= VFIO_PCI_ERROR_RECOVERY_DEVICE_RESET; + if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED) + state.flags |= VFIO_PCI_ERROR_RECOVERY_FAILED; + state.sequence = vdev->pci_recovery_sequence; + up_read(&vdev->recovery_lock); + + if (copy_to_user(arg, &state, sizeof(state))) + return -EFAULT; + return 0; + } + + if (copy_from_user(&state, arg, sizeof(state))) + return -EFAULT; + if (state.flags || state.sequence || state.eventfd < -1) + return -EINVAL; + + enable = state.eventfd >= 0; + if (enable) { + ctx = eventfd_ctx_fdget(state.eventfd); + if (IS_ERR(ctx)) + return PTR_ERR(ctx); + } + + down_write(&vdev->recovery_lock); + if (!vdev->pci_recovery_device_open) { + ret = -ENODEV; + goto out_unlock; + } + if (vdev->pci_recovery_access_blocked) { + ret = -EBUSY; + goto out_unlock; + } + + if (!enable && + (vdev->pci_recovery_flags & + (VFIO_PCI_RECOVERY_IN_PROGRESS | VFIO_PCI_RECOVERY_FAILED))) { + ret = -EBUSY; + goto out_unlock; + } + + mutex_lock(&vdev->igate); + ret = vfio_pci_eventfd_replace_locked(vdev, + &vdev->pci_recovery_trigger, + ctx); + mutex_unlock(&vdev->igate); + if (ret) + goto out_unlock; + + WRITE_ONCE(vdev->pci_recovery_enabled, enable); + /* + * Start each enabled period from a clear state, so a sequence number + * and the status bits beside it always describe an event this + * eventfd was notified of. Nothing is in flight to lose. A + * transaction holds access_blocked, which failed this call with + * -EBUSY above. + * + * access_blocked itself is not cleared here, so userspace can never + * disable its way out of a block. + */ + WRITE_ONCE(vdev->pci_recovery_flags, 0); + vdev->pci_recovery_sequence = 0; + vdev->pci_recovery_command_valid = false; + +out_unlock: + up_write(&vdev->recovery_lock); + if (ret && ctx) + eventfd_ctx_put(ctx); + + return ret; +} + int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags, void __user *arg, size_t argsz) { @@ -1870,6 +1974,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags, return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz); case VFIO_DEVICE_FEATURE_ZPCI_ERROR: return vfio_pci_zdev_feature_err(device, flags, arg, argsz); + + case VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY: + return vfio_pci_core_feature_error_recovery(vdev, flags, arg, argsz); default: return -ENOTTY; } @@ -2741,6 +2848,18 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) } EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device); +static void +vfio_pci_signal_recovery_event(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_eventfd *eventfd; + + rcu_read_lock(); + eventfd = rcu_dereference(vdev->pci_recovery_trigger); + if (eventfd) + eventfd_signal(eventfd->ctx); + rcu_read_unlock(); +} + pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, pci_channel_state_t state) { @@ -2748,6 +2867,7 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, struct vfio_pci_eventfd *eventfd; pci_ers_result_t result = PCI_ERS_RESULT_CAN_RECOVER; unsigned long irq_flags; + bool notify_recovery = false; bool terminal = false; bool nested; u32 flags; @@ -2792,6 +2912,7 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, goto out_unlock; } + notify_recovery = true; WRITE_ONCE(vdev->pci_recovery_access_blocked, true); /* * A second event before resume() has finished the first joins the @@ -2916,6 +3037,8 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, if (eventfd) eventfd_signal(eventfd->ctx); rcu_read_unlock(); + if (notify_recovery) + vfio_pci_signal_recovery_event(vdev); return result; } @@ -2981,13 +3104,15 @@ static pci_ers_result_t vfio_pci_core_aer_slot_reset(struct pci_dev *pdev) up_write(&vdev->recovery_lock); /* - * Whoever clears IN_PROGRESS owes the wake. resume() will not do it, - * since it bails once IN_PROGRESS is clear, and the core skips it - * altogether if the domain verdict is not RECOVERED. On success the - * transaction carries on and resume() wakes. + * Whoever clears IN_PROGRESS owes the wake and the event. resume() + * will not do it, since it bails once IN_PROGRESS is clear, and the + * core skips it altogether if the domain verdict is not RECOVERED. + * On success the transaction carries on and resume() does both. */ - if (ret) + if (ret) { wake_up_all(&vdev->pci_recovery_wait); + vfio_pci_signal_recovery_event(vdev); + } return result; } @@ -2996,6 +3121,7 @@ static void vfio_pci_core_aer_resume(struct pci_dev *pdev) { struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); unsigned long irq_flags; + bool notify_recovery = false; u32 flags; int ret = 0; @@ -3011,6 +3137,7 @@ static void vfio_pci_core_aer_resume(struct pci_dev *pdev) if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS)) goto out_unlock; + notify_recovery = true; if (!vdev->pci_recovery_device_open) { vdev->pci_recovery_command_valid = false; WRITE_ONCE(vdev->pci_recovery_flags, @@ -3056,7 +3183,10 @@ static void vfio_pci_core_aer_resume(struct pci_dev *pdev) out_unlock: up_write(&vdev->recovery_lock); - wake_up_all(&vdev->pci_recovery_wait); + if (notify_recovery) { + wake_up_all(&vdev->pci_recovery_wait); + vfio_pci_signal_recovery_event(vdev); + } } int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev, -- 2.43.0