| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/24 00:57 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"KMSANReasoning": "",
"NeedsKMSAN": false,
"Reasoning": "The patch series primarily adds an mlx5 selftest driver under tools/testing/selftests/ and modifies userspace test headers under tools/. The only kernel header changes (under include/linux/mlx5/) refactor and move macros, structs, and enum definitions without modifying any executable kernel C code. Furthermore, mlx5 targets vendor-specific Mellanox NICs/HCAs requiring physical PCIe hardware not emulated in standard QEMU environments.",
"WorthFuzzing": false
} |
| 1/1 | 2026/09/24 00:57 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit f344d40ff4714d670cc8789c378c68c22d8901f6\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Thu Sep 24 00:57:40 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h\nindex 9d47cdc727ad0..a1c14479e462c 100644\n--- a/include/linux/mlx5/cq.h\n+++ b/include/linux/mlx5/cq.h\n@@ -81,16 +81,6 @@ enum {\n \n enum {\n \tMLX5_CQE_OWNER_MASK\t= 1,\n-\tMLX5_CQE_REQ\t\t= 0,\n-\tMLX5_CQE_RESP_WR_IMM\t= 1,\n-\tMLX5_CQE_RESP_SEND\t= 2,\n-\tMLX5_CQE_RESP_SEND_IMM\t= 3,\n-\tMLX5_CQE_RESP_SEND_INV\t= 4,\n-\tMLX5_CQE_RESIZE_CQ\t= 5,\n-\tMLX5_CQE_SIG_ERR\t= 12,\n-\tMLX5_CQE_REQ_ERR\t= 13,\n-\tMLX5_CQE_RESP_ERR\t= 14,\n-\tMLX5_CQE_INVALID\t= 15,\n };\n \n enum {\ndiff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h\nindex 8cb321a9fb3d1..37727b5ab78f2 100644\n--- a/include/linux/mlx5/device.h\n+++ b/include/linux/mlx5/device.h\n@@ -36,6 +36,7 @@\n #include \u003clinux/types.h\u003e\n #include \u003crdma/ib_verbs.h\u003e\n #include \u003clinux/mlx5/mlx5_ifc.h\u003e\n+#include \u003clinux/mlx5/mlx5_ifc_macros.h\u003e\n #include \u003clinux/bitfield.h\u003e\n \n #if defined(__LITTLE_ENDIAN)\n@@ -46,122 +47,6 @@\n #error Host endianness not defined\n #endif\n \n-/* helper macros */\n-#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0)\n-#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)-\u003efld)\n-#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))\n-#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16)\n-#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)\n-#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64)\n-#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) \u0026 0xf))\n-#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) \u0026 0x1f))\n-#define __mlx5_mask(typ, fld) ((u32)((1ull \u003c\u003c __mlx5_bit_sz(typ, fld)) - 1))\n-#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) \u003c\u003c __mlx5_dw_bit_off(typ, fld))\n-#define __mlx5_mask16(typ, fld) ((u16)((1ull \u003c\u003c __mlx5_bit_sz(typ, fld)) - 1))\n-#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) \u003c\u003c __mlx5_16_bit_off(typ, fld))\n-#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits)\n-\n-#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8)\n-#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8)\n-#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32)\n-#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64)\n-#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8)\n-#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32)\n-#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)\n-#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld)))\n-\n-/* insert a value to a struct */\n-#define MLX5_SET(typ, p, fld, v) do { \\\n-\tu32 _v = v; \\\n-\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \\\n-\t*((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \\\n-\tcpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) \u0026 \\\n-\t\t (~__mlx5_dw_mask(typ, fld))) | (((_v) \u0026 __mlx5_mask(typ, fld)) \\\n-\t\t \u003c\u003c __mlx5_dw_bit_off(typ, fld))); \\\n-} while (0)\n-\n-#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \\\n-\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \\\n-\tMLX5_SET(typ, p, fld[idx], v); \\\n-} while (0)\n-\n-#define MLX5_SET_TO_ONES(typ, p, fld) do { \\\n-\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \\\n-\t*((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \\\n-\tcpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) \u0026 \\\n-\t\t (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \\\n-\t\t \u003c\u003c __mlx5_dw_bit_off(typ, fld))); \\\n-} while (0)\n-\n-#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\\\n-__mlx5_dw_off(typ, fld))) \u003e\u003e __mlx5_dw_bit_off(typ, fld)) \u0026 \\\n-__mlx5_mask(typ, fld))\n-\n-#define MLX5_GET_PR(typ, p, fld) ({ \\\n-\tu32 ___t = MLX5_GET(typ, p, fld); \\\n-\tpr_debug(#fld \" = 0x%x\\n\", ___t); \\\n-\t___t; \\\n-})\n-\n-#define __MLX5_SET64(typ, p, fld, v) do { \\\n-\tBUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \\\n-\t*((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \\\n-} while (0)\n-\n-#define MLX5_SET64(typ, p, fld, v) do { \\\n-\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \\\n-\t__MLX5_SET64(typ, p, fld, v); \\\n-} while (0)\n-\n-#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \\\n-\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \\\n-\t__MLX5_SET64(typ, p, fld[idx], v); \\\n-} while (0)\n-\n-#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld)))\n-\n-#define MLX5_GET64_PR(typ, p, fld) ({ \\\n-\tu64 ___t = MLX5_GET64(typ, p, fld); \\\n-\tpr_debug(#fld \" = 0x%llx\\n\", ___t); \\\n-\t___t; \\\n-})\n-\n-#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\\\n-__mlx5_16_off(typ, fld))) \u003e\u003e __mlx5_16_bit_off(typ, fld)) \u0026 \\\n-__mlx5_mask16(typ, fld))\n-\n-#define MLX5_SET16(typ, p, fld, v) do { \\\n-\tu16 _v = v; \\\n-\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \\\n-\t*((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \\\n-\tcpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) \u0026 \\\n-\t\t (~__mlx5_16_mask(typ, fld))) | (((_v) \u0026 __mlx5_mask16(typ, fld)) \\\n-\t\t \u003c\u003c __mlx5_16_bit_off(typ, fld))); \\\n-} while (0)\n-\n-/* Big endian getters */\n-#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\\\n-\t__mlx5_64_off(typ, fld)))\n-\n-#define MLX5_GET_BE(type_t, typ, p, fld) ({\t\t\t\t \\\n-\t\ttype_t tmp;\t\t\t\t\t\t \\\n-\t\tswitch (sizeof(tmp)) {\t\t\t\t\t \\\n-\t\tcase sizeof(u8):\t\t\t\t\t \\\n-\t\t\ttmp = (__force type_t)MLX5_GET(typ, p, fld);\t \\\n-\t\t\tbreak;\t\t\t\t\t\t \\\n-\t\tcase sizeof(u16):\t\t\t\t\t \\\n-\t\t\ttmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \\\n-\t\t\tbreak;\t\t\t\t\t\t \\\n-\t\tcase sizeof(u32):\t\t\t\t\t \\\n-\t\t\ttmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \\\n-\t\t\tbreak;\t\t\t\t\t\t \\\n-\t\tcase sizeof(u64):\t\t\t\t\t \\\n-\t\t\ttmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \\\n-\t\t\tbreak;\t\t\t\t\t\t \\\n-\t\t\t}\t\t\t\t\t\t \\\n-\t\ttmp;\t\t\t\t\t\t\t \\\n-\t\t})\n-\n enum mlx5_inline_modes {\n \tMLX5_INLINE_MODE_NONE,\n \tMLX5_INLINE_MODE_L2,\n@@ -172,7 +57,7 @@ enum mlx5_inline_modes {\n enum {\n \tMLX5_MAX_COMMANDS\t\t= 32,\n \tMLX5_CMD_DATA_BLOCK_SIZE\t= 512,\n-\tMLX5_PCI_CMD_XPORT\t\t= 7,\n+\tMLX5_PCI_CMD_XPORT\t\t= MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT,\n \tMLX5_MKEY_BSF_OCTO_SIZE\t\t= 4,\n \tMLX5_MAX_PSVS\t\t\t= 4,\n };\n@@ -308,63 +193,6 @@ enum {\n \tMLX5_EVENT_QUEUE_TYPE_DCT = 6,\n };\n \n-/* mlx5 components can subscribe to any one of these events via\n- * mlx5_eq_notifier_register API.\n- */\n-enum mlx5_event {\n-\t/* Special value to subscribe to any event */\n-\tMLX5_EVENT_TYPE_NOTIFY_ANY\t = 0x0,\n-\t/* HW events enum start: comp events are not subscribable */\n-\tMLX5_EVENT_TYPE_COMP\t\t = 0x0,\n-\t/* HW Async events enum start: subscribable events */\n-\tMLX5_EVENT_TYPE_PATH_MIG\t = 0x01,\n-\tMLX5_EVENT_TYPE_COMM_EST\t = 0x02,\n-\tMLX5_EVENT_TYPE_SQ_DRAINED\t = 0x03,\n-\tMLX5_EVENT_TYPE_SRQ_LAST_WQE\t = 0x13,\n-\tMLX5_EVENT_TYPE_SRQ_RQ_LIMIT\t = 0x14,\n-\n-\tMLX5_EVENT_TYPE_CQ_ERROR\t = 0x04,\n-\tMLX5_EVENT_TYPE_WQ_CATAS_ERROR\t = 0x05,\n-\tMLX5_EVENT_TYPE_PATH_MIG_FAILED\t = 0x07,\n-\tMLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10,\n-\tMLX5_EVENT_TYPE_WQ_ACCESS_ERROR\t = 0x11,\n-\tMLX5_EVENT_TYPE_SRQ_CATAS_ERROR\t = 0x12,\n-\tMLX5_EVENT_TYPE_OBJECT_CHANGE\t = 0x27,\n-\n-\tMLX5_EVENT_TYPE_INTERNAL_ERROR\t = 0x08,\n-\tMLX5_EVENT_TYPE_PORT_CHANGE\t = 0x09,\n-\tMLX5_EVENT_TYPE_GPIO_EVENT\t = 0x15,\n-\tMLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16,\n-\tMLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17,\n-\tMLX5_EVENT_TYPE_XRQ_ERROR\t = 0x18,\n-\tMLX5_EVENT_TYPE_REMOTE_CONFIG\t = 0x19,\n-\tMLX5_EVENT_TYPE_GENERAL_EVENT\t = 0x22,\n-\tMLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24,\n-\tMLX5_EVENT_TYPE_PPS_EVENT = 0x25,\n-\n-\tMLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a,\n-\tMLX5_EVENT_TYPE_STALL_EVENT\t = 0x1b,\n-\n-\tMLX5_EVENT_TYPE_CMD\t\t = 0x0a,\n-\tMLX5_EVENT_TYPE_PAGE_REQUEST\t = 0xb,\n-\n-\tMLX5_EVENT_TYPE_PAGE_FAULT\t = 0xc,\n-\tMLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0xd,\n-\n-\tMLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0xe,\n-\tMLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0xf,\n-\n-\tMLX5_EVENT_TYPE_DCT_DRAINED = 0x1c,\n-\tMLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d,\n-\n-\tMLX5_EVENT_TYPE_FPGA_ERROR = 0x20,\n-\tMLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21,\n-\n-\tMLX5_EVENT_TYPE_DEVICE_TRACER = 0x26,\n-\n-\tMLX5_EVENT_TYPE_MAX = 0x100,\n-};\n-\n enum mlx5_driver_event {\n \tMLX5_DRIVER_EVENT_TYPE_TRAP = 0,\n \tMLX5_DRIVER_EVENT_UPLINK_NETDEV,\n@@ -420,22 +248,6 @@ enum {\n };\n \n enum {\n-\tMLX5_OPCODE_NOP\t\t\t= 0x00,\n-\tMLX5_OPCODE_SEND_INVAL\t\t= 0x01,\n-\tMLX5_OPCODE_RDMA_WRITE\t\t= 0x08,\n-\tMLX5_OPCODE_RDMA_WRITE_IMM\t= 0x09,\n-\tMLX5_OPCODE_SEND\t\t= 0x0a,\n-\tMLX5_OPCODE_SEND_IMM\t\t= 0x0b,\n-\tMLX5_OPCODE_LSO\t\t\t= 0x0e,\n-\tMLX5_OPCODE_RDMA_READ\t\t= 0x10,\n-\tMLX5_OPCODE_ATOMIC_CS\t\t= 0x11,\n-\tMLX5_OPCODE_ATOMIC_FA\t\t= 0x12,\n-\tMLX5_OPCODE_ATOMIC_MASKED_CS\t= 0x14,\n-\tMLX5_OPCODE_ATOMIC_MASKED_FA\t= 0x15,\n-\tMLX5_OPCODE_BIND_MW\t\t= 0x18,\n-\tMLX5_OPCODE_CONFIG_CMD\t\t= 0x1f,\n-\tMLX5_OPCODE_ENHANCED_MPSW\t= 0x29,\n-\n \tMLX5_RECV_OPCODE_RDMA_WRITE_IMM\t= 0x00,\n \tMLX5_RECV_OPCODE_SEND\t\t= 0x01,\n \tMLX5_RECV_OPCODE_SEND_IMM\t= 0x02,\n@@ -443,19 +255,6 @@ enum {\n \n \tMLX5_CQE_OPCODE_ERROR\t\t= 0x1e,\n \tMLX5_CQE_OPCODE_RESIZE\t\t= 0x16,\n-\n-\tMLX5_OPCODE_SET_PSV\t\t= 0x20,\n-\tMLX5_OPCODE_GET_PSV\t\t= 0x21,\n-\tMLX5_OPCODE_CHECK_PSV\t\t= 0x22,\n-\tMLX5_OPCODE_DUMP\t\t= 0x23,\n-\tMLX5_OPCODE_RGET_PSV\t\t= 0x26,\n-\tMLX5_OPCODE_RCHECK_PSV\t\t= 0x27,\n-\n-\tMLX5_OPCODE_UMR\t\t\t= 0x25,\n-\n-\tMLX5_OPCODE_FLOW_TBL_ACCESS\t= 0x2c,\n-\n-\tMLX5_OPCODE_ACCESS_ASO\t\t= 0x2d,\n };\n \n enum {\n@@ -1224,12 +1023,6 @@ enum mlx5_flex_parser_protos {\n \n /* MLX5 DEV CAPs */\n \n-/* TODO: EAT.ME */\n-enum mlx5_cap_mode {\n-\tHCA_CAP_OPMOD_GET_MAX\t= 0,\n-\tHCA_CAP_OPMOD_GET_CUR\t= 1,\n-};\n-\n /* Any new cap addition must update mlx5_hca_caps_alloc() to allocate\n * capability memory.\n */\n@@ -1507,26 +1300,6 @@ enum mlx5_qcam_feature_groups {\n #define MLX5_CAP_PSP(mdev, cap)\\\n \tMLX5_GET(psp_cap, (mdev)-\u003ecaps.hca[MLX5_CAP_PSP]-\u003ecur, cap)\n \n-enum {\n-\tMLX5_CMD_STAT_OK\t\t\t= 0x0,\n-\tMLX5_CMD_STAT_INT_ERR\t\t\t= 0x1,\n-\tMLX5_CMD_STAT_BAD_OP_ERR\t\t= 0x2,\n-\tMLX5_CMD_STAT_BAD_PARAM_ERR\t\t= 0x3,\n-\tMLX5_CMD_STAT_BAD_SYS_STATE_ERR\t\t= 0x4,\n-\tMLX5_CMD_STAT_BAD_RES_ERR\t\t= 0x5,\n-\tMLX5_CMD_STAT_RES_BUSY\t\t\t= 0x6,\n-\tMLX5_CMD_STAT_NOT_READY\t\t\t= 0x7,\n-\tMLX5_CMD_STAT_LIM_ERR\t\t\t= 0x8,\n-\tMLX5_CMD_STAT_BAD_RES_STATE_ERR\t\t= 0x9,\n-\tMLX5_CMD_STAT_IX_ERR\t\t\t= 0xa,\n-\tMLX5_CMD_STAT_NO_RES_ERR\t\t= 0xf,\n-\tMLX5_CMD_STAT_BAD_INP_LEN_ERR\t\t= 0x50,\n-\tMLX5_CMD_STAT_BAD_OUTP_LEN_ERR\t\t= 0x51,\n-\tMLX5_CMD_STAT_BAD_QP_STATE_ERR\t\t= 0x10,\n-\tMLX5_CMD_STAT_BAD_PKT_ERR\t\t= 0x30,\n-\tMLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR\t= 0x40,\n-};\n-\n enum {\n \tMLX5_IEEE_802_3_COUNTERS_GROUP\t = 0x0,\n \tMLX5_RFC_2863_COUNTERS_GROUP\t = 0x1,\ndiff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h\nindex 12b9a81e56214..8247cb0f8dfd6 100644\n--- a/include/linux/mlx5/mlx5_ifc.h\n+++ b/include/linux/mlx5/mlx5_ifc.h\n@@ -5979,6 +5979,108 @@ struct mlx5_ifc_cqe_error_syndrome_bits {\n \tu8 syndrome[0x8];\n };\n \n+struct mlx5_ifc_wqe_ctrl_seg_bits {\n+\tu8 opmod[0x8];\n+\tu8 wqe_index[0x10];\n+\tu8 opcode[0x8];\n+\n+\tu8 qp_or_sq[0x18];\n+\tu8 reserved_at_38[0x2];\n+\tu8 ds[0x6];\n+\n+\tu8 signature[0x8];\n+\tu8 reserved_at_48[0x10];\n+\tu8 fm[0x3];\n+\tu8 reserved_at_5b[0x1];\n+\tu8 ce[0x2];\n+\tu8 se[0x1];\n+\tu8 reserved_at_5f[0x1];\n+\n+\tu8 imm[0x20];\n+};\n+\n+/* Values for wqe_ctrl_seg_bits.opcode */\n+enum {\n+\tMLX5_OPCODE_NOP\t\t\t= 0x00,\n+\tMLX5_OPCODE_SEND_INVAL\t\t= 0x01,\n+\tMLX5_OPCODE_RDMA_WRITE\t\t= 0x08,\n+\tMLX5_OPCODE_RDMA_WRITE_IMM\t= 0x09,\n+\tMLX5_OPCODE_SEND\t\t= 0x0a,\n+\tMLX5_OPCODE_SEND_IMM\t\t= 0x0b,\n+\tMLX5_OPCODE_LSO\t\t\t= 0x0e,\n+\tMLX5_OPCODE_RDMA_READ\t\t= 0x10,\n+\tMLX5_OPCODE_ATOMIC_CS\t\t= 0x11,\n+\tMLX5_OPCODE_ATOMIC_FA\t\t= 0x12,\n+\tMLX5_OPCODE_ATOMIC_MASKED_CS\t= 0x14,\n+\tMLX5_OPCODE_ATOMIC_MASKED_FA\t= 0x15,\n+\tMLX5_OPCODE_BIND_MW\t\t= 0x18,\n+\tMLX5_OPCODE_CONFIG_CMD\t\t= 0x1f,\n+\tMLX5_OPCODE_SET_PSV\t\t= 0x20,\n+\tMLX5_OPCODE_GET_PSV\t\t= 0x21,\n+\tMLX5_OPCODE_CHECK_PSV\t\t= 0x22,\n+\tMLX5_OPCODE_DUMP\t\t= 0x23,\n+\tMLX5_OPCODE_UMR\t\t\t= 0x25,\n+\tMLX5_OPCODE_RGET_PSV\t\t= 0x26,\n+\tMLX5_OPCODE_RCHECK_PSV\t\t= 0x27,\n+\tMLX5_OPCODE_ENHANCED_MPSW\t= 0x29,\n+\tMLX5_OPCODE_FLOW_TBL_ACCESS\t= 0x2c,\n+\tMLX5_OPCODE_ACCESS_ASO\t\t= 0x2d,\n+};\n+\n+/* Values for wqe_ctrl_seg_bits.ce */\n+enum {\n+\tMLX5_WQE_CE_CQE_ALWAYS\t\t= 2,\n+};\n+\n+struct mlx5_ifc_wqe_raddr_seg_bits {\n+\tu8 raddr[0x40];\n+\n+\tu8 rkey[0x20];\n+\tu8 reserved_at_60[0x20];\n+};\n+\n+struct mlx5_ifc_wqe_data_seg_bits {\n+\tu8 reserved_at_0[0x1];\n+\tu8 byte_count[0x1f];\n+\n+\tu8 lkey[0x20];\n+\n+\tu8 addr[0x40];\n+};\n+\n+struct mlx5_ifc_cqe64_bits {\n+\tu8 reserved_at_0[0x1a0];\n+\n+\tunion {\n+\t\tu8 reserved_at_1a0[0x20];\n+\t\tstruct mlx5_ifc_cqe_error_syndrome_bits error_syndrome;\n+\t};\n+\n+\tu8 send_wqe_opcode[0x8];\n+\tu8 qpn_or_dctn_or_flow_tag[0x18];\n+\n+\tu8 wqe_counter[0x10];\n+\tu8 signature[0x8];\n+\tu8 opcode[0x4];\n+\tu8 cqe_format[0x2];\n+\tu8 se[0x1];\n+\tu8 owner[0x1];\n+};\n+\n+/* Values for cqe64_bits.opcode */\n+enum {\n+\tMLX5_CQE_REQ\t\t= 0,\n+\tMLX5_CQE_RESP_WR_IMM\t= 1,\n+\tMLX5_CQE_RESP_SEND\t= 2,\n+\tMLX5_CQE_RESP_SEND_IMM\t= 3,\n+\tMLX5_CQE_RESP_SEND_INV\t= 4,\n+\tMLX5_CQE_RESIZE_CQ\t= 5,\n+\tMLX5_CQE_SIG_ERR\t= 12,\n+\tMLX5_CQE_REQ_ERR\t= 13,\n+\tMLX5_CQE_RESP_ERR\t= 14,\n+\tMLX5_CQE_INVALID\t= 15,\n+};\n+\n struct mlx5_ifc_qp_context_extension_bits {\n \tu8 reserved_at_0[0x60];\n \n@@ -6475,6 +6577,12 @@ struct mlx5_ifc_query_hca_cap_in_bits {\n \tu8 reserved_at_60[0x20];\n };\n \n+/* Values for query_hca_cap_in_bits.op_mod */\n+enum mlx5_cap_mode {\n+\tHCA_CAP_OPMOD_GET_MAX\t= 0,\n+\tHCA_CAP_OPMOD_GET_CUR\t= 1,\n+};\n+\n struct mlx5_ifc_other_hca_cap_bits {\n \tu8 roce[0x1];\n \tu8 reserved_at_1[0x27f];\n@@ -11270,6 +11378,55 @@ struct mlx5_ifc_eqe_bits {\n \tu8 owner[0x1];\n };\n \n+/* Values for eqe_bits.event_type */\n+enum mlx5_event {\n+\t/* Special value to subscribe to any event */\n+\tMLX5_EVENT_TYPE_NOTIFY_ANY\t = 0x0,\n+\t/* HW events enum start: comp events are not subscribable */\n+\tMLX5_EVENT_TYPE_COMP\t\t = 0x0,\n+\t/* HW Async events enum start: subscribable events */\n+\tMLX5_EVENT_TYPE_PATH_MIG\t = 0x01,\n+\tMLX5_EVENT_TYPE_COMM_EST\t = 0x02,\n+\tMLX5_EVENT_TYPE_SQ_DRAINED\t = 0x03,\n+\tMLX5_EVENT_TYPE_SRQ_LAST_WQE\t = 0x13,\n+\tMLX5_EVENT_TYPE_SRQ_RQ_LIMIT\t = 0x14,\n+\n+\tMLX5_EVENT_TYPE_CQ_ERROR\t = 0x04,\n+\tMLX5_EVENT_TYPE_WQ_CATAS_ERROR\t = 0x05,\n+\tMLX5_EVENT_TYPE_PATH_MIG_FAILED\t = 0x07,\n+\tMLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10,\n+\tMLX5_EVENT_TYPE_WQ_ACCESS_ERROR\t = 0x11,\n+\tMLX5_EVENT_TYPE_SRQ_CATAS_ERROR\t = 0x12,\n+\tMLX5_EVENT_TYPE_OBJECT_CHANGE\t = 0x27,\n+\n+\tMLX5_EVENT_TYPE_INTERNAL_ERROR\t = 0x08,\n+\tMLX5_EVENT_TYPE_PORT_CHANGE\t = 0x09,\n+\tMLX5_EVENT_TYPE_CMD\t\t = 0x0a,\n+\tMLX5_EVENT_TYPE_PAGE_REQUEST\t = 0x0b,\n+\tMLX5_EVENT_TYPE_PAGE_FAULT\t = 0x0c,\n+\tMLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0x0d,\n+\tMLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0x0e,\n+\tMLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0x0f,\n+\tMLX5_EVENT_TYPE_GPIO_EVENT\t = 0x15,\n+\tMLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16,\n+\tMLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17,\n+\tMLX5_EVENT_TYPE_XRQ_ERROR\t = 0x18,\n+\tMLX5_EVENT_TYPE_REMOTE_CONFIG\t = 0x19,\n+\tMLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a,\n+\tMLX5_EVENT_TYPE_STALL_EVENT\t = 0x1b,\n+\tMLX5_EVENT_TYPE_DCT_DRAINED = 0x1c,\n+\tMLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d,\n+\tMLX5_EVENT_TYPE_FPGA_ERROR = 0x20,\n+\tMLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21,\n+\tMLX5_EVENT_TYPE_GENERAL_EVENT\t = 0x22,\n+\tMLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24,\n+\tMLX5_EVENT_TYPE_PPS_EVENT = 0x25,\n+\tMLX5_EVENT_TYPE_DEVICE_TRACER = 0x26,\n+\n+\tMLX5_EVENT_TYPE_MAX = 0x100,\n+};\n+\n+/* Values for cmd_queue_entry_bits.type */\n enum {\n \tMLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT = 0x7,\n };\n@@ -11312,6 +11469,27 @@ struct mlx5_ifc_cmd_out_bits {\n \tu8 command_output[0x20];\n };\n \n+/* Values for cmd_out_bits.status */\n+enum {\n+\tMLX5_CMD_STAT_OK\t\t\t= 0x0,\n+\tMLX5_CMD_STAT_INT_ERR\t\t\t= 0x1,\n+\tMLX5_CMD_STAT_BAD_OP_ERR\t\t= 0x2,\n+\tMLX5_CMD_STAT_BAD_PARAM_ERR\t\t= 0x3,\n+\tMLX5_CMD_STAT_BAD_SYS_STATE_ERR\t\t= 0x4,\n+\tMLX5_CMD_STAT_BAD_RES_ERR\t\t= 0x5,\n+\tMLX5_CMD_STAT_RES_BUSY\t\t\t= 0x6,\n+\tMLX5_CMD_STAT_NOT_READY\t\t\t= 0x7,\n+\tMLX5_CMD_STAT_LIM_ERR\t\t\t= 0x8,\n+\tMLX5_CMD_STAT_BAD_RES_STATE_ERR\t\t= 0x9,\n+\tMLX5_CMD_STAT_IX_ERR\t\t\t= 0xa,\n+\tMLX5_CMD_STAT_NO_RES_ERR\t\t= 0xf,\n+\tMLX5_CMD_STAT_BAD_QP_STATE_ERR\t\t= 0x10,\n+\tMLX5_CMD_STAT_BAD_PKT_ERR\t\t= 0x30,\n+\tMLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR\t= 0x40,\n+\tMLX5_CMD_STAT_BAD_INP_LEN_ERR\t\t= 0x50,\n+\tMLX5_CMD_STAT_BAD_OUTP_LEN_ERR\t\t= 0x51,\n+};\n+\n struct mlx5_ifc_cmd_in_bits {\n \tu8 opcode[0x10];\n \tu8 reserved_at_10[0x10];\ndiff --git a/include/linux/mlx5/mlx5_ifc_macros.h b/include/linux/mlx5/mlx5_ifc_macros.h\nnew file mode 100644\nindex 0000000000000..1b9e39a509495\n--- /dev/null\n+++ b/include/linux/mlx5/mlx5_ifc_macros.h\n@@ -0,0 +1,185 @@\n+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */\n+/*\n+ * Copyright (c) 2013-2026, Mellanox Technologies. All rights reserved.\n+ *\n+ * Accessor macros for mlx5 IFC structures.\n+ *\n+ * Extracted from device.h so that code which cannot include device.h\n+ * (e.g. selftests) can still use the MLX5_SET/GET family directly.\n+ */\n+\n+#ifndef MLX5_IFC_MACROS_H\n+#define MLX5_IFC_MACROS_H\n+\n+/* Internal helpers -- 32-bit */\n+#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0)\n+#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)-\u003efld)\n+#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))\n+#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16)\n+#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)\n+#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64)\n+#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) \u0026 0xf))\n+#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) \u0026 0x1f))\n+#define __mlx5_mask(typ, fld) ((u32)((1ull \u003c\u003c __mlx5_bit_sz(typ, fld)) - 1))\n+#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) \u003c\u003c __mlx5_dw_bit_off(typ, fld))\n+#define __mlx5_mask16(typ, fld) ((u16)((1ull \u003c\u003c __mlx5_bit_sz(typ, fld)) - 1))\n+#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) \u003c\u003c __mlx5_16_bit_off(typ, fld))\n+#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits)\n+\n+/* Size and address macros */\n+#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8)\n+#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8)\n+#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32)\n+#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64)\n+#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8)\n+#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32)\n+#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)\n+#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld)))\n+\n+/* insert a value to a struct */\n+#define MLX5_SET(typ, p, fld, v) do { \\\n+\tu32 _v = v; \\\n+\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \\\n+\t*((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \\\n+\tcpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) \u0026 \\\n+\t\t (~__mlx5_dw_mask(typ, fld))) | (((_v) \u0026 __mlx5_mask(typ, fld)) \\\n+\t\t \u003c\u003c __mlx5_dw_bit_off(typ, fld))); \\\n+} while (0)\n+\n+#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \\\n+\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \\\n+\tMLX5_SET(typ, p, fld[idx], v); \\\n+} while (0)\n+\n+#define MLX5_SET_TO_ONES(typ, p, fld) do { \\\n+\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \\\n+\t*((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \\\n+\tcpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) \u0026 \\\n+\t\t (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \\\n+\t\t \u003c\u003c __mlx5_dw_bit_off(typ, fld))); \\\n+} while (0)\n+\n+#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\\\n+__mlx5_dw_off(typ, fld))) \u003e\u003e __mlx5_dw_bit_off(typ, fld)) \u0026 \\\n+__mlx5_mask(typ, fld))\n+\n+#define MLX5_GET_PR(typ, p, fld) ({ \\\n+\tu32 ___t = MLX5_GET(typ, p, fld); \\\n+\tpr_debug(#fld \" = 0x%x\\n\", ___t); \\\n+\t___t; \\\n+})\n+\n+/* 64-bit field accessors */\n+#define __MLX5_SET64(typ, p, fld, v) do { \\\n+\tBUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \\\n+\t*((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \\\n+} while (0)\n+\n+#define MLX5_SET64(typ, p, fld, v) do { \\\n+\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \\\n+\t__MLX5_SET64(typ, p, fld, v); \\\n+} while (0)\n+\n+#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \\\n+\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \\\n+\t__MLX5_SET64(typ, p, fld[idx], v); \\\n+} while (0)\n+\n+#define MLX5_ARRAY_GET64(typ, p, fld, idx) \\\n+\t({ \\\n+\t\tBUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \\\n+\t\tbe64_to_cpu( \\\n+\t\t\t*((__be64 *)(p) + __mlx5_64_off(typ, fld) + (idx))); \\\n+\t})\n+\n+#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld)))\n+\n+#define MLX5_GET64_PR(typ, p, fld) ({ \\\n+\tu64 ___t = MLX5_GET64(typ, p, fld); \\\n+\tpr_debug(#fld \" = 0x%llx\\n\", ___t); \\\n+\t___t; \\\n+})\n+\n+/* 16-bit field accessors */\n+#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\\\n+__mlx5_16_off(typ, fld))) \u003e\u003e __mlx5_16_bit_off(typ, fld)) \u0026 \\\n+__mlx5_mask16(typ, fld))\n+\n+#define MLX5_SET16(typ, p, fld, v) do { \\\n+\tu16 _v = v; \\\n+\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \\\n+\t*((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \\\n+\tcpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) \u0026 \\\n+\t\t (~__mlx5_16_mask(typ, fld))) | (((_v) \u0026 __mlx5_mask16(typ, fld)) \\\n+\t\t \u003c\u003c __mlx5_16_bit_off(typ, fld))); \\\n+} while (0)\n+\n+/* Big endian getters */\n+#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\\\n+\t__mlx5_64_off(typ, fld)))\n+\n+#define MLX5_GET_BE(type_t, typ, p, fld) ({\t\t\t\t \\\n+\t\ttype_t tmp;\t\t\t\t\t\t \\\n+\t\tswitch (sizeof(tmp)) {\t\t\t\t\t \\\n+\t\tcase sizeof(u8):\t\t\t\t\t \\\n+\t\t\ttmp = (__force type_t)MLX5_GET(typ, p, fld);\t \\\n+\t\t\tbreak;\t\t\t\t\t\t \\\n+\t\tcase sizeof(u16):\t\t\t\t\t \\\n+\t\t\ttmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \\\n+\t\t\tbreak;\t\t\t\t\t\t \\\n+\t\tcase sizeof(u32):\t\t\t\t\t \\\n+\t\t\ttmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \\\n+\t\t\tbreak;\t\t\t\t\t\t \\\n+\t\tcase sizeof(u64):\t\t\t\t\t \\\n+\t\t\ttmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \\\n+\t\t\tbreak;\t\t\t\t\t\t \\\n+\t\t\t}\t\t\t\t\t\t \\\n+\t\ttmp;\t\t\t\t\t\t\t \\\n+\t\t})\n+\n+/*\n+ * Use READ_ONCE/WRITE_ONCE for a single field that hardware may read/write\n+ * unpredictably, mostly owner bits. All other bits in the DW must be stable.\n+ * Usually a dma_wmb() will be required before a write and a dma_rmb() after a\n+ * read.\n+ */\n+#define MLX5_GET_ONCE(typ, p, fld) \\\n+\t((be32_to_cpu(READ_ONCE(*((__be32 *)(p) + __mlx5_dw_off(typ, fld)))) \u003e\u003e \\\n+\t __mlx5_dw_bit_off(typ, fld)) \u0026 \\\n+\t __mlx5_mask(typ, fld))\n+\n+#define MLX5_SET_ONCE(typ, p, fld, v) \\\n+\tdo { \\\n+\t\tu32 _v = v; \\\n+\t\t__be32 *_dw = (__be32 *)(p) + __mlx5_dw_off(typ, fld); \\\n+\t\tBUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \\\n+\t\tWRITE_ONCE(*_dw, \\\n+\t\t\t cpu_to_be32((be32_to_cpu(READ_ONCE(*_dw)) \u0026 \\\n+\t\t\t\t\t(~__mlx5_dw_mask(typ, fld))) | \\\n+\t\t\t\t (((_v) \u0026 __mlx5_mask(typ, fld)) \\\n+\t\t\t\t\t\u003c\u003c __mlx5_dw_bit_off(typ, fld)))); \\\n+\t} while (0)\n+\n+/* Access MMIO registers, usually the init segment, using IFC structs. */\n+#define MLX5_GET_MMIO(typ, p, fld) \\\n+\t((ioread32be(((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld))) \u003e\u003e \\\n+\t __mlx5_dw_bit_off(typ, fld)) \u0026 \\\n+\t __mlx5_mask(typ, fld))\n+\n+/* The set is not relaxed so there is an integrated dma_wmb(). */\n+#define MLX5_SET_MMIO(typ, p, fld, v) \\\n+\tdo { \\\n+\t\tu32 _v = v; \\\n+\t\tvoid __iomem *_dw = \\\n+\t\t\t((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld)); \\\n+\t\tif (__mlx5_bit_sz(typ, fld) == 32) \\\n+\t\t\tiowrite32be(_v, _dw); \\\n+\t\telse \\\n+\t\t\tiowrite32be((ioread32be(_dw) \u0026 \\\n+\t\t\t\t (~__mlx5_dw_mask(typ, fld))) | \\\n+\t\t\t\t\t ((_v \u0026 __mlx5_mask(typ, fld)) \\\n+\t\t\t\t\t \u003c\u003c __mlx5_dw_bit_off(typ, fld)), \\\n+\t\t\t\t _dw); \\\n+\t} while (0)\n+\n+#endif /* MLX5_IFC_MACROS_H */\ndiff --git a/tools/arch/arm64/include/asm/barrier.h b/tools/arch/arm64/include/asm/barrier.h\nindex 3b9b41331c4f1..abdc64fc3c70f 100644\n--- a/tools/arch/arm64/include/asm/barrier.h\n+++ b/tools/arch/arm64/include/asm/barrier.h\n@@ -24,6 +24,10 @@\n #define smp_wmb()\tasm volatile(\"dmb ishst\" ::: \"memory\")\n #define smp_rmb()\tasm volatile(\"dmb ishld\" ::: \"memory\")\n \n+/* DMA barriers use outer-shareable (osh) for device visibility */\n+#define dma_rmb()\tasm volatile(\"dmb oshld\" ::: \"memory\")\n+#define dma_wmb()\tasm volatile(\"dmb oshst\" ::: \"memory\")\n+\n #define smp_store_release(p, v)\t\t\t\t\t\t\\\n do {\t\t\t\t\t\t\t\t\t\\\n \tunion { typeof(*p) __val; char __c[1]; } __u =\t\t\t\\\ndiff --git a/tools/arch/x86/include/asm/barrier.h b/tools/arch/x86/include/asm/barrier.h\nindex 0adf295dd5b6a..0b51431fa530e 100644\n--- a/tools/arch/x86/include/asm/barrier.h\n+++ b/tools/arch/x86/include/asm/barrier.h\n@@ -43,4 +43,9 @@ do {\t\t\t\t\t\t\\\n \t___p1;\t\t\t\t\t\\\n })\n #endif /* defined(__x86_64__) */\n+\n+/* x86 is DMA-coherent so DMA barriers are just compiler barriers */\n+#define dma_rmb()\tbarrier()\n+#define dma_wmb()\tbarrier()\n+\n #endif /* _TOOLS_LINUX_ASM_X86_BARRIER_H */\ndiff --git a/tools/include/asm-generic/io.h b/tools/include/asm-generic/io.h\nindex e5a0b07ad452a..0d89decdafb81 100644\n--- a/tools/include/asm-generic/io.h\n+++ b/tools/include/asm-generic/io.h\n@@ -479,4 +479,32 @@ static inline void writesq(volatile void __iomem *addr, const void *buffer,\n }\n #endif\n \n+/*\n+ * ioread/iowrite for big-endian MMIO registers.\n+ */\n+\n+#ifndef ioread32be\n+#define ioread32be ioread32be\n+static inline u32 ioread32be(const volatile void __iomem *addr)\n+{\n+\treturn bswap_32(readl(addr));\n+}\n+#endif\n+\n+#ifndef iowrite32be\n+#define iowrite32be iowrite32be\n+static inline void iowrite32be(u32 value, volatile void __iomem *addr)\n+{\n+\twritel(bswap_32(value), addr);\n+}\n+#endif\n+\n+#ifndef iowrite64be\n+#define iowrite64be iowrite64be\n+static inline void iowrite64be(u64 value, volatile void __iomem *addr)\n+{\n+\twriteq(bswap_64(value), addr);\n+}\n+#endif\n+\n #endif /* _TOOLS_ASM_GENERIC_IO_H */\ndiff --git a/tools/include/asm/barrier.h b/tools/include/asm/barrier.h\nindex 0c21678ac5e65..e7e0c7de5a2ff 100644\n--- a/tools/include/asm/barrier.h\n+++ b/tools/include/asm/barrier.h\n@@ -47,6 +47,14 @@\n # define smp_mb()\tmb()\n #endif\n \n+#ifndef dma_rmb\n+# define dma_rmb()\trmb()\n+#endif\n+\n+#ifndef dma_wmb\n+# define dma_wmb()\twmb()\n+#endif\n+\n #ifndef smp_store_release\n # define smp_store_release(p, v)\t\t\\\n do {\t\t\t\t\t\t\\\ndiff --git a/tools/include/linux/stddef.h b/tools/include/linux/stddef.h\nnew file mode 100644\nindex 0000000000000..99182ea4a1419\n--- /dev/null\n+++ b/tools/include/linux/stddef.h\n@@ -0,0 +1,10 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef _TOOLS_LINUX_STDDEF_H\n+#define _TOOLS_LINUX_STDDEF_H\n+\n+#include_next \u003clinux/stddef.h\u003e\n+\n+#define DECLARE_FLEX_ARRAY(TYPE, NAME) \\\n+\t__DECLARE_FLEX_ARRAY(TYPE, NAME)\n+\n+#endif /* _TOOLS_LINUX_STDDEF_H */\ndiff --git a/tools/testing/selftests/kvm/include/arm64/processor.h b/tools/testing/selftests/kvm/include/arm64/processor.h\nindex b8a902ba8573d..5878a6abdf20e 100644\n--- a/tools/testing/selftests/kvm/include/arm64/processor.h\n+++ b/tools/testing/selftests/kvm/include/arm64/processor.h\n@@ -12,6 +12,7 @@\n \n #include \u003clinux/stringify.h\u003e\n #include \u003clinux/types.h\u003e\n+#include \u003casm/barrier.h\u003e\n #include \u003casm/brk-imm.h\u003e\n #include \u003casm/esr.h\u003e\n #include \u003casm/sysreg.h\u003e\n@@ -191,11 +192,8 @@ static inline void cpu_relax(void)\n #define dsb(opt)\tasm volatile(\"dsb \" #opt : : : \"memory\")\n #define dmb(opt)\tasm volatile(\"dmb \" #opt : : : \"memory\")\n \n-#define dma_wmb()\tdmb(oshst)\n #define __iowmb()\tdma_wmb()\n \n-#define dma_rmb()\tdmb(oshld)\n-\n #define __iormb(v)\t\t\t\t\t\t\t\\\n ({\t\t\t\t\t\t\t\t\t\\\n \tunsigned long tmp;\t\t\t\t\t\t\\\ndiff --git a/tools/testing/selftests/kvm/irq_test.c b/tools/testing/selftests/kvm/irq_test.c\nindex 168c52734fef5..791e576c334cd 100644\n--- a/tools/testing/selftests/kvm/irq_test.c\n+++ b/tools/testing/selftests/kvm/irq_test.c\n@@ -6,7 +6,6 @@\n #include \"proc_util.h\"\n \n #include \u003clibvfio.h\u003e\n-#include \u003clinux/sizes.h\u003e\n #include \u003cstdio.h\u003e\n #include \u003cstdlib.h\u003e\n #include \u003cunistd.h\u003e\n@@ -101,7 +100,6 @@ static int vfio_setup_msi(struct vfio_pci_device *device)\n \tallocator = iova_allocator_init(device-\u003eiommu);\n \n \tregion = \u0026device-\u003edriver.region;\n-\tregion-\u003esize = SZ_2M;\n \tregion-\u003eiova = iova_allocator_alloc(allocator, region-\u003esize);\n \tregion-\u003evaddr = kvm_mmap(region-\u003esize, prot, flags, -1);\n \tTEST_ASSERT(region-\u003evaddr != MAP_FAILED, \"mmap() failed\\n\");\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c\nindex 19d9630b24c23..40b8541b588ee 100644\n--- a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c\n+++ b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c\n@@ -418,6 +418,7 @@ static void dsa_send_msi(struct vfio_pci_device *device)\n \n const struct vfio_pci_driver_ops dsa_ops = {\n \t.name = \"dsa\",\n+\t.region_size = sizeof(struct dsa_state),\n \t.probe = dsa_probe,\n \t.init = dsa_init,\n \t.remove = dsa_remove,\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c\nindex fd9e05d77ea4b..2fafcc9f51c02 100644\n--- a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c\n+++ b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c\n@@ -576,6 +576,7 @@ static void igb_send_msi(struct vfio_pci_device *device)\n \n const struct vfio_pci_driver_ops igb_ops = {\n \t.name = \"igb\",\n+\t.region_size = sizeof(struct igb),\n \t.probe = igb_probe,\n \t.init = igb_init,\n \t.remove = igb_remove,\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c\nindex a871b935542ba..c9b28365c5eb6 100644\n--- a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c\n+++ b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c\n@@ -226,6 +226,7 @@ static void ioat_send_msi(struct vfio_pci_device *device)\n \n const struct vfio_pci_driver_ops ioat_ops = {\n \t.name = \"ioat\",\n+\t.region_size = sizeof(struct ioat_state),\n \t.probe = ioat_probe,\n \t.init = ioat_init,\n \t.remove = ioat_remove,\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c\nnew file mode 100644\nindex 0000000000000..a65b52e183bc6\n--- /dev/null\n+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c\n@@ -0,0 +1,1928 @@\n+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB\n+/*\n+ * mlx5 VFIO selftest driver\n+ *\n+ * Programs mlx5 ConnectX VFs and PFs through the bare-metal command interface\n+ * and RDMA WRITE self-loopback to perform DMA. Implements vfio_pci_driver_ops\n+ * (probe/init/remove) and plugs into the VFIO selftest framework.\n+ */\n+#include \u003csched.h\u003e\n+#include \u003cstdbool.h\u003e\n+#include \u003cstdint.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003ctime.h\u003e\n+#include \u003cunistd.h\u003e\n+\n+#include \u003clinux/errno.h\u003e\n+#include \u003clinux/io.h\u003e\n+#include \u003clinux/log2.h\u003e\n+#include \u003clinux/pci_regs.h\u003e\n+#include \u003clinux/pci_ids.h\u003e\n+\n+#include \u003clibvfio.h\u003e\n+\n+#include \"mlx5_hw.h\"\n+\n+/*\n+ * Driver state — overlaid on device-\u003edriver.region.vaddr.\n+ *\n+ * Contains both software-only state and HW-visible DMA buffers. HW buffers need\n+ * strict IOVA alignment.\n+ */\n+struct mlx5st_device {\n+\t/* Back pointer */\n+\tstruct vfio_pci_device *device;\n+\n+\t/* BAR0 */\n+\tstruct mlx5st_initial_seg __iomem *init_seg;\n+\tvoid __iomem *bar0;\n+\n+\t/* Command interface */\n+\tstruct mlx5st_cmd_queue_entry *cmd_lay;\n+\tstruct mlx5st_cmd_queue_entry *pages_cmd_lay;\n+\tu8 cmd_log_stride;\n+\tunsigned int pages_slot;\n+\tu8 cmd_token;\n+\tbool cmd_sig_enabled;\n+\n+\t/* PD */\n+\tu32 pdn;\n+\n+\t/* Global PA-mode MKEY */\n+\tu32 global_lkey;\n+\tu32 global_rkey;\n+\tu32 mkey_index;\n+\n+\t/* CQ */\n+\tu32 cqn;\n+\tu32 cq_ci;\n+\tu32 cq_arm_sn;\n+\n+\t/* UAR */\n+\tu32 uar_page;\n+\tvoid __iomem *uar_base;\n+\tunsigned int uar_bf_offset;\n+\n+\t/* EQ (cmd/pages events — polled, not interrupt-driven) */\n+\tu32 eqn;\n+\tu32 eq_cons_index;\n+\tbool have_eq;\n+\n+\t/* MSI EQ (CQ completion events — fires MSI-X) */\n+\tu32 msi_eqn;\n+\tu32 msi_eq_cons_index;\n+\tbool have_msi_eq;\n+\n+\t/* Async pages slot state */\n+\tbool pages_slot_in_use;\n+\tbool pages_slot_is_reclaim;\n+\tunsigned int pages_reclaim_npages;\n+\tunsigned int pages_pending_give;\n+\tunsigned int pages_pending_reclaim;\n+\tu16 pages_pending_func_id;\n+\tbool pages_func_id_seen;\n+\n+\t/* QP */\n+\tu32 qpn;\n+\tu32 sq_pi;\n+\tu32 sq_ci;\n+\n+\t/* FW pages bitmap */\n+\tu64 fw_pages_bitmap[MAX_FW_PAGES / 64];\n+\tu32 fw_pages_given;\n+\tu16 fw_func_id;\n+\n+\t/* Capabilities */\n+\tbool fl_supported;\n+\tu8 log_max_msg;\n+\n+\t/* Buffers used by send_msi() to trigger an interrupt */\n+\tu64 send_msi_src;\n+\tu64 send_msi_dst;\n+\n+\t/*\n+\t * HW-visible DMA buffers below — device reads/writes via DMA.\n+\t */\n+\tstruct mlx5st_cmd_queue_entry cmd_queue\n+\t\t[MLX5_HW_PAGE_SIZE / sizeof(struct mlx5st_cmd_queue_entry)]\n+\t\t__aligned(MLX5_HW_PAGE_SIZE);\n+\tstruct mlx5st_send_wqe sq_buf[SQ_WQE_CNT];\n+\tstruct mlx5st_dbrec cq_dbrec;\n+\tstruct mlx5st_dbrec qp_dbrec;\n+\tstruct mlx5st_cqe64 cq_buf[CQ_CQE_CNT];\n+\n+\t/* Slot 0 mailboxes (regular commands) */\n+\tstruct mlx5st_mbox_entry cmd_in_mbox[CMD_MBOX_NENT];\n+\tstruct mlx5st_mbox_entry cmd_out_mbox[CMD_MBOX_NENT];\n+\n+\t/* Pages slot mailboxes (async MANAGE_PAGES) */\n+\tstruct mlx5st_mbox_entry pages_in_mbox[CMD_MBOX_NENT];\n+\tstruct mlx5st_mbox_entry pages_out_mbox[CMD_MBOX_NENT];\n+\n+\t/* EQ does not support page_offset */\n+\tstruct mlx5st_eqe eq_buf[EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE);\n+\n+\t/* MSI EQ buffer — CQ completions generate EQEs here -\u003e MSI-X */\n+\tstruct mlx5st_eqe msi_eq_buf[MSI_EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE);\n+\n+\tu8 fw_pages[MAX_FW_PAGES][MLX5_HW_PAGE_SIZE]\n+\t\t__aligned(MLX5_HW_PAGE_SIZE);\n+};\n+\n+/* Check against HW limits on IOVA alignment */\n+static_assert(offsetof(struct mlx5st_device, cmd_in_mbox) %\n+\t\t\t CMD_MBOX_STRIDE == 0,\n+\t \"cmd_in_mbox must be stride-aligned\");\n+static_assert(offsetof(struct mlx5st_device, pages_in_mbox) %\n+\t\t\t CMD_MBOX_STRIDE == 0,\n+\t \"pages_in_mbox must be stride-aligned\");\n+static_assert(offsetof(struct mlx5st_device, cq_buf) % 64 == 0,\n+\t \"cq_buf must be 64-byte aligned\");\n+static_assert(offsetof(struct mlx5st_device, sq_buf) % 64 == 0,\n+\t \"sq_buf must be 64-byte aligned\");\n+static_assert(offsetof(struct mlx5st_device, cq_dbrec) % 64 == 0,\n+\t \"cq_dbrec must be 64-byte aligned\");\n+static_assert(offsetof(struct mlx5st_device, qp_dbrec) % 64 == 0,\n+\t \"qp_dbrec must be 64-byte aligned\");\n+static_assert(offsetof(struct mlx5st_device, eq_buf) %\n+\t\t\t MLX5_HW_PAGE_SIZE == 0,\n+\t \"eq_buf must be page-aligned\");\n+static_assert(offsetof(struct mlx5st_device, msi_eq_buf) %\n+\t\t\t MLX5_HW_PAGE_SIZE == 0,\n+\t \"msi_eq_buf must be page-aligned\");\n+static_assert(offsetof(struct mlx5st_device, fw_pages) %\n+\t\t\t MLX5_HW_PAGE_SIZE == 0,\n+\t \"fw_pages must be page-aligned\");\n+\n+static struct mlx5st_device *to_mlx5st(struct vfio_pci_device *device)\n+{\n+\treturn device-\u003edriver.region.vaddr;\n+}\n+\n+/*\n+ * Fill a PAS (Physical Address Segment) for a buffer in the driver region.\n+ * Sets pas[0] to the page-aligned IOVA and returns the page_offset (the\n+ * buffer's byte offset within that page, in units of 64 bytes).\n+ */\n+static unsigned int mlx5st_fill_pas(struct vfio_pci_device *device, void *buf,\n+\t\t\t\t __be64 *pas)\n+{\n+\tu64 iova = to_iova(device, buf);\n+\n+\tpas[0] = cpu_to_be64(iova \u0026 ~(u64)(MLX5_HW_PAGE_SIZE - 1));\n+\treturn (iova \u0026 (MLX5_HW_PAGE_SIZE - 1)) / 64;\n+}\n+\n+/*\n+ * Probe — match mlx5 devices by PCI vendor/device ID.\n+ */\n+\n+static int mlx5st_probe(struct vfio_pci_device *device)\n+{\n+\tstatic const u16 mlx5st_pci_ids[] = {\n+\t\t0x1011, /* Connect-IB */\n+\t\t0x1012, /* Connect-IB VF */\n+\t\t0x1013, /* ConnectX-4 */\n+\t\t0x1014, /* ConnectX-4 VF */\n+\t\t0x1015, /* ConnectX-4LX */\n+\t\t0x1016, /* ConnectX-4LX VF */\n+\t\t0x1017, /* ConnectX-5 */\n+\t\t0x1018, /* ConnectX-5 VF */\n+\t\t0x1019, /* ConnectX-5 Ex */\n+\t\t0x101a, /* ConnectX-5 Ex VF */\n+\t\t0x101b, /* ConnectX-6 */\n+\t\t0x101c, /* ConnectX-6 VF */\n+\t\t0x101d, /* ConnectX-6 Dx */\n+\t\t0x101e, /* ConnectX-6 Dx VF */\n+\t\t0x101f, /* ConnectX-6 LX */\n+\t\t0x1021, /* ConnectX-7 */\n+\t\t0x1023, /* ConnectX-8 */\n+\t\t0x1025, /* ConnectX-9 */\n+\t\t0x1027, /* ConnectX-10 */\n+\t\t0x2101, /* ConnectX-10 NVLink-C2C */\n+\t\t0xa2d2, /* BlueField integrated ConnectX-5 */\n+\t\t0xa2d3, /* BlueField integrated ConnectX-5 VF */\n+\t\t0xa2d6, /* BlueField-2 integrated ConnectX-6 Dx */\n+\t\t0xa2dc, /* BlueField-3 integrated ConnectX-7 */\n+\t\t0xa2df, /* BlueField-4 integrated ConnectX-8 */\n+\t};\n+\tunsigned int i;\n+\tu16 did;\n+\n+\tif (vfio_pci_config_readw(device, PCI_VENDOR_ID) !=\n+\t PCI_VENDOR_ID_MELLANOX)\n+\t\treturn -ENODEV;\n+\n+\tdid = vfio_pci_config_readw(device, PCI_DEVICE_ID);\n+\tfor (i = 0; i \u003c ARRAY_SIZE(mlx5st_pci_ids); i++) {\n+\t\tif (mlx5st_pci_ids[i] == did)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn -ENODEV;\n+}\n+\n+/*\n+ * Command interface\n+ */\n+\n+static u8 xor8_buf(const void *buf, size_t offset, size_t len)\n+{\n+\tconst u8 *p = buf;\n+\tu8 sum = 0;\n+\tsize_t i;\n+\n+\tfor (i = offset; i \u003c offset + len; i++)\n+\t\tsum ^= p[i];\n+\treturn sum;\n+}\n+\n+#define CMD_IF_BOX_CTRL_OFF MLX5_BYTE_OFF(cmd_if_box, reserved_at_1000)\n+#define CMD_IF_BOX_CTRL_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, ctrl_signature)\n+#define CMD_IF_BOX_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, signature)\n+\n+static void mlx5st_cmd_calc_block_sig(struct mlx5st_cmd_if_box *blk)\n+{\n+\tMLX5_SET(cmd_if_box, blk, ctrl_signature,\n+\t\t ~xor8_buf(blk, CMD_IF_BOX_CTRL_OFF,\n+\t\t\t CMD_IF_BOX_CTRL_SIG_OFF - CMD_IF_BOX_CTRL_OFF));\n+\tMLX5_SET(cmd_if_box, blk, signature,\n+\t\t ~xor8_buf(blk, 0, CMD_IF_BOX_SIG_OFF));\n+}\n+\n+static int mlx5st_cmd_verify_block_sig(struct mlx5st_cmd_if_box *blk)\n+{\n+\tif (xor8_buf(blk, CMD_IF_BOX_CTRL_OFF,\n+\t\t CMD_IF_BOX_SIG_OFF - CMD_IF_BOX_CTRL_OFF) != 0xff)\n+\t\treturn -1;\n+\tif (xor8_buf(blk, 0, sizeof(struct mlx5st_cmd_if_box)) != 0xff)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+static unsigned int mlx5st_cmd_setup_mbox_chain(struct vfio_pci_device *device,\n+\t\t\t\t\t\tstruct mlx5st_mbox_entry *mbox,\n+\t\t\t\t\t\tunsigned int nblocks, u8 token)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c nblocks; i++) {\n+\t\tstruct mlx5st_cmd_if_box *blk = \u0026mbox[i].block;\n+\t\tu64 next_iova;\n+\n+\t\tmemset(blk, 0, sizeof(struct mlx5st_cmd_if_box));\n+\t\tMLX5_SET(cmd_if_box, blk, block_number, i);\n+\t\tMLX5_SET(cmd_if_box, blk, token, token);\n+\t\tif (i \u003c nblocks - 1) {\n+\t\t\tnext_iova = to_iova(device, \u0026mbox[i + 1]);\n+\t\t\tMLX5_SET(cmd_if_box, blk, next_pointer_63_32,\n+\t\t\t\t next_iova \u003e\u003e 32);\n+\t\t\tMLX5_SET(cmd_if_box, blk, next_pointer_31_10,\n+\t\t\t\t (u32)next_iova \u003e\u003e 10);\n+\t\t}\n+\t}\n+\treturn nblocks;\n+}\n+\n+static void mlx5st_cmd_copy_to_mbox(struct mlx5st_mbox_entry *mbox,\n+\t\t\t\t const void *data, unsigned int len)\n+{\n+\tconst u8 *src = data;\n+\tunsigned int i = 0;\n+\n+\twhile (len \u003e 0) {\n+\t\tunsigned int chunk = len \u003c MLX5_CMD_DATA_BLOCK_SIZE ?\n+\t\t\t\t\t len :\n+\t\t\t\t\t MLX5_CMD_DATA_BLOCK_SIZE;\n+\n+\t\tmemcpy(MLX5_ADDR_OF(cmd_if_box, \u0026mbox[i].block, mailbox_data),\n+\t\t src, chunk);\n+\t\tsrc += chunk;\n+\t\tlen -= chunk;\n+\t\ti++;\n+\t}\n+}\n+\n+static void mlx5st_cmd_copy_from_mbox(void *data,\n+\t\t\t\t const struct mlx5st_mbox_entry *mbox,\n+\t\t\t\t unsigned int len)\n+{\n+\tunsigned int i = 0;\n+\tu8 *dst = data;\n+\n+\twhile (len \u003e 0) {\n+\t\tunsigned int chunk = len \u003c MLX5_CMD_DATA_BLOCK_SIZE ?\n+\t\t\t\t\t len :\n+\t\t\t\t\t MLX5_CMD_DATA_BLOCK_SIZE;\n+\n+\t\tmemcpy(dst,\n+\t\t MLX5_ADDR_OF(cmd_if_box, \u0026mbox[i].block, mailbox_data),\n+\t\t chunk);\n+\t\tdst += chunk;\n+\t\tlen -= chunk;\n+\t\ti++;\n+\t}\n+}\n+\n+/* Forward declaration — cmd_exec polls events during command wait */\n+static void mlx5st_process_events(struct mlx5st_device *dev);\n+\n+static const char *mlx5st_cmd_name(u16 opcode)\n+{\n+\tswitch (opcode) {\n+\tcase MLX5_CMD_OP_QUERY_HCA_CAP: return \"QUERY_HCA_CAP\";\n+\tcase MLX5_CMD_OP_INIT_HCA: return \"INIT_HCA\";\n+\tcase MLX5_CMD_OP_TEARDOWN_HCA: return \"TEARDOWN_HCA\";\n+\tcase MLX5_CMD_OP_ENABLE_HCA: return \"ENABLE_HCA\";\n+\tcase MLX5_CMD_OP_DISABLE_HCA: return \"DISABLE_HCA\";\n+\tcase MLX5_CMD_OP_QUERY_PAGES: return \"QUERY_PAGES\";\n+\tcase MLX5_CMD_OP_MANAGE_PAGES: return \"MANAGE_PAGES\";\n+\tcase MLX5_CMD_OP_SET_HCA_CAP: return \"SET_HCA_CAP\";\n+\tcase MLX5_CMD_OP_SET_ISSI: return \"SET_ISSI\";\n+\tcase MLX5_CMD_OP_CREATE_MKEY: return \"CREATE_MKEY\";\n+\tcase MLX5_CMD_OP_DESTROY_MKEY: return \"DESTROY_MKEY\";\n+\tcase MLX5_CMD_OP_CREATE_EQ: return \"CREATE_EQ\";\n+\tcase MLX5_CMD_OP_DESTROY_EQ: return \"DESTROY_EQ\";\n+\tcase MLX5_CMD_OP_CREATE_CQ: return \"CREATE_CQ\";\n+\tcase MLX5_CMD_OP_DESTROY_CQ: return \"DESTROY_CQ\";\n+\tcase MLX5_CMD_OP_CREATE_QP: return \"CREATE_QP\";\n+\tcase MLX5_CMD_OP_DESTROY_QP: return \"DESTROY_QP\";\n+\tcase MLX5_CMD_OP_RST2INIT_QP: return \"RST2INIT_QP\";\n+\tcase MLX5_CMD_OP_INIT2RTR_QP: return \"INIT2RTR_QP\";\n+\tcase MLX5_CMD_OP_RTR2RTS_QP: return \"RTR2RTS_QP\";\n+\tcase MLX5_CMD_OP_ALLOC_PD: return \"ALLOC_PD\";\n+\tcase MLX5_CMD_OP_DEALLOC_PD: return \"DEALLOC_PD\";\n+\tcase MLX5_CMD_OP_ALLOC_UAR: return \"ALLOC_UAR\";\n+\tcase MLX5_CMD_OP_DEALLOC_UAR: return \"DEALLOC_UAR\";\n+\tdefault: return \"UNKNOWN\";\n+\t}\n+}\n+\n+/*\n+ * Post a command on a given slot: fill the cmd_queue_entry, set up mailbox\n+ * chains, compute signatures, hand ownership to FW, and ring the doorbell.\n+ */\n+static void mlx5st_cmd_post(struct mlx5st_device *dev,\n+\t\t\t struct mlx5st_cmd_queue_entry *cmd,\n+\t\t\t struct mlx5st_mbox_entry *in_mbox,\n+\t\t\t struct mlx5st_mbox_entry *out_mbox,\n+\t\t\t void *in, unsigned int ilen, unsigned int olen,\n+\t\t\t u32 doorbell)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tunsigned int in_remain, out_remain, in_nblk, out_nblk;\n+\tunsigned int i;\n+\tvoid *cin, *cout;\n+\tu8 token;\n+\n+\t/* Rotating non-zero token ties cmd entry to its mailbox blocks */\n+\ttoken = ++dev-\u003ecmd_token;\n+\tif (!token)\n+\t\ttoken = ++dev-\u003ecmd_token;\n+\n+\tin_remain = ilen \u003e MLX5_CMD_INLINE_SZ ? ilen - MLX5_CMD_INLINE_SZ : 0;\n+\tout_remain = olen \u003e MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0;\n+\tin_nblk = (in_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /\n+\t\t MLX5_CMD_DATA_BLOCK_SIZE;\n+\tout_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /\n+\t\t MLX5_CMD_DATA_BLOCK_SIZE;\n+\n+\t/* Set up mailbox chains */\n+\tif (in_nblk \u003e 0) {\n+\t\tmlx5st_cmd_setup_mbox_chain(device, in_mbox, in_nblk, token);\n+\t\tmlx5st_cmd_copy_to_mbox(in_mbox,\n+\t\t\t\t\t(u8 *)in + MLX5_CMD_INLINE_SZ,\n+\t\t\t\t\tin_remain);\n+\t}\n+\tif (out_nblk \u003e 0)\n+\t\tmlx5st_cmd_setup_mbox_chain(device, out_mbox, out_nblk, token);\n+\n+\t/* Copy inline input */\n+\tcin = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_input_inline_data);\n+\tmemset(cin, 0, MLX5_CMD_INLINE_SZ);\n+\tmemcpy(cin, in, ilen \u003c MLX5_CMD_INLINE_SZ ? ilen : MLX5_CMD_INLINE_SZ);\n+\tMLX5_SET(cmd_queue_entry, cmd, input_length, ilen);\n+\tMLX5_SET(cmd_queue_entry, cmd, token, token);\n+\n+\t/* Zero inline output */\n+\tcout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);\n+\tmemset(cout, 0, MLX5_CMD_INLINE_SZ);\n+\tMLX5_SET(cmd_queue_entry, cmd, output_length, olen);\n+\n+\t/*\n+\t * Compute signatures: mailbox blocks first, then cmd_queue_entry.\n+\t * The sig must cover the final state including ownership=0x1, but\n+\t * we must not set ownership until after the sig is in place -\n+\t * XOR in the 0x1 without storing it to memory.\n+\t */\n+\tfor (i = 0; i \u003c in_nblk; i++)\n+\t\tmlx5st_cmd_calc_block_sig(\u0026in_mbox[i].block);\n+\tfor (i = 0; i \u003c out_nblk; i++)\n+\t\tmlx5st_cmd_calc_block_sig(\u0026out_mbox[i].block);\n+\tMLX5_SET(cmd_queue_entry, cmd, signature, 0);\n+\tMLX5_SET(cmd_queue_entry, cmd, signature,\n+\t\t ~(xor8_buf(cmd, 0, sizeof(struct mlx5st_cmd_queue_entry)) ^\n+\t\t 0x1));\n+\n+\t/* Ensure all cmd data (including sig) is visible, then hand to FW */\n+\tdma_wmb();\n+\tMLX5_SET_ONCE(cmd_queue_entry, cmd, ownership, 1);\n+\n+\t/* Ring doorbell */\n+\tMLX5_SET_MMIO(initial_seg, dev-\u003einit_seg, command_doorbell_vector,\n+\t\t doorbell);\n+}\n+\n+static void mlx5st_cmd_exec(struct mlx5st_device *dev, void *in,\n+\t\t\t unsigned int ilen, void *out, unsigned int olen)\n+{\n+\tstruct mlx5st_cmd_queue_entry *cmd = dev-\u003ecmd_lay;\n+\tunsigned int out_remain, out_nblk;\n+\tstruct timespec start, now;\n+\tunsigned int elapsed;\n+\tunsigned int i;\n+\tvoid *cout;\n+\n+\tmlx5st_cmd_post(dev, cmd, dev-\u003ecmd_in_mbox, dev-\u003ecmd_out_mbox, in,\n+\t\t\tilen, olen, 1);\n+\n+\tout_remain = olen \u003e MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0;\n+\tout_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /\n+\t\t MLX5_CMD_DATA_BLOCK_SIZE;\n+\n+\t/* Poll for completion — also process EQ events for PF page requests */\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026start);\n+\tfor (;;) {\n+\t\tif (!MLX5_GET_ONCE(cmd_queue_entry, cmd, ownership))\n+\t\t\tbreak;\n+\t\tif (dev-\u003ehave_eq)\n+\t\t\tmlx5st_process_events(dev);\n+\t\tsched_yield();\n+\t\tclock_gettime(CLOCK_MONOTONIC, \u0026now);\n+\t\telapsed = (now.tv_sec - start.tv_sec) * 1000 +\n+\t\t\t (now.tv_nsec - start.tv_nsec) / 1000000;\n+\t\tif (elapsed \u003e MLX5_CMD_TIMEOUT_MS)\n+\t\t\tVFIO_FAIL(\"cmd timeout after %d ms\", elapsed);\n+\t}\n+\t/* Ensure output data reads happen after ownership is seen clear */\n+\tdma_rmb();\n+\n+\t/* Verify output signatures when FW has checksums enabled */\n+\tif (dev-\u003ecmd_sig_enabled) {\n+\t\tif (xor8_buf(cmd, 0,\n+\t\t\t sizeof(struct mlx5st_cmd_queue_entry)) != 0xff)\n+\t\t\tVFIO_FAIL(\"cmd output signature mismatch\");\n+\t\tfor (i = 0; i \u003c out_nblk; i++) {\n+\t\t\tif (mlx5st_cmd_verify_block_sig(\n+\t\t\t\t \u0026dev-\u003ecmd_out_mbox[i].block))\n+\t\t\t\tVFIO_FAIL(\"cmd output mailbox block %d signature mismatch\",\n+\t\t\t\t\t i);\n+\t\t}\n+\t}\n+\n+\t/* Copy output: inline first */\n+\tcout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);\n+\tmemcpy(out, cout, olen \u003c MLX5_CMD_INLINE_SZ ? olen : MLX5_CMD_INLINE_SZ);\n+\n+\t/* Copy remaining from output mailbox chain */\n+\tif (out_remain \u003e 0)\n+\t\tmlx5st_cmd_copy_from_mbox((u8 *)out + MLX5_CMD_INLINE_SZ,\n+\t\t\t\t\t dev-\u003ecmd_out_mbox, out_remain);\n+\n+\t/* Check command status */\n+\tif (MLX5_GET(enable_hca_out, out, status) != MLX5_CMD_STAT_OK)\n+\t\tVFIO_FAIL(\"%s: status=0x%x syndrome=0x%x\",\n+\t\t\t mlx5st_cmd_name(MLX5_GET(enable_hca_in, in, opcode)),\n+\t\t\t MLX5_GET(enable_hca_out, out, status),\n+\t\t\t MLX5_GET(enable_hca_out, out, syndrome));\n+}\n+\n+static struct mlx5st_cmd_queue_entry *\n+mlx5st_cmd_slot_init(struct mlx5st_device *dev, unsigned int slot,\n+\t\t struct mlx5st_mbox_entry *in_mbox,\n+\t\t struct mlx5st_mbox_entry *out_mbox)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tstruct mlx5st_cmd_queue_entry *cmd =\n+\t\t\u0026dev-\u003ecmd_queue[(slot \u003c\u003c dev-\u003ecmd_log_stride) /\n+\t\t\t\tsizeof(struct mlx5st_cmd_queue_entry)];\n+\tu64 iova;\n+\n+\tMLX5_SET(cmd_queue_entry, cmd, type,\n+\t\t MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT);\n+\tiova = to_iova(device, in_mbox);\n+\tMLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_63_32,\n+\t\t iova \u003e\u003e 32);\n+\tMLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_31_9, iova \u003e\u003e 9);\n+\tiova = to_iova(device, out_mbox);\n+\tMLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_63_32,\n+\t\t iova \u003e\u003e 32);\n+\tMLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_31_9,\n+\t\t iova \u003e\u003e 9);\n+\treturn cmd;\n+}\n+\n+static void mlx5st_cmd_init(struct mlx5st_device *dev)\n+{\n+\tstruct mlx5st_initial_seg __iomem *seg = dev-\u003einit_seg;\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tu16 cmdif_rev;\n+\tu8 log_sz;\n+\tu64 iova;\n+\n+\tcmdif_rev = MLX5_GET_MMIO(initial_seg, seg, cmd_interface_rev);\n+\tVFIO_ASSERT_EQ(cmdif_rev, 5);\n+\n+\t/* Read command queue geometry from BAR */\n+\tlog_sz = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_size);\n+\tdev-\u003ecmd_log_stride = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_stride);\n+\tdev-\u003epages_slot = (1 \u003c\u003c log_sz) - 1;\n+\n+\tVFIO_ASSERT_LE((unsigned int)(1 \u003c\u003c log_sz), 32u);\n+\tVFIO_ASSERT_GE((unsigned int)(1 \u003c\u003c dev-\u003ecmd_log_stride),\n+\t\t (unsigned int)sizeof(struct mlx5st_cmd_queue_entry));\n+\tVFIO_ASSERT_LE((unsigned int)((dev-\u003epages_slot + 1) \u003c\u003c\n+\t\t\t\t dev-\u003ecmd_log_stride),\n+\t\t (unsigned int)sizeof(dev-\u003ecmd_queue));\n+\n+\t/* Set up slot 0 — regular commands */\n+\tdev-\u003ecmd_lay = mlx5st_cmd_slot_init(dev, 0, dev-\u003ecmd_in_mbox,\n+\t\t\t\t\t dev-\u003ecmd_out_mbox);\n+\n+\t/* Set up pages slot — async MANAGE_PAGES */\n+\tdev-\u003epages_cmd_lay = mlx5st_cmd_slot_init(dev, dev-\u003epages_slot,\n+\t\t\t\t\t\t dev-\u003epages_in_mbox,\n+\t\t\t\t\t\t dev-\u003epages_out_mbox);\n+\n+\t/* Write command queue page address to BAR0 */\n+\tiova = to_iova(device, dev-\u003ecmd_queue);\n+\tMLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_63_32, iova \u003e\u003e 32);\n+\tMLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_31_12, iova \u003e\u003e 12);\n+\n+\tdev_dbg(device,\n+\t\t \"Command interface initialized (cmdif_rev=5, log_sz=%u, log_stride=%u, pages_slot=%u)\\n\",\n+\t\t log_sz, dev-\u003ecmd_log_stride, dev-\u003epages_slot);\n+}\n+\n+/*\n+ * FW pages: bitmap allocator + MANAGE_PAGES\n+ */\n+\n+static void mlx5st_fw_pages_alloc(struct mlx5st_device *dev,\n+\t\t\t\t unsigned int npages, u64 *iovas)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tunsigned int found = 0;\n+\tunsigned int w, b;\n+\tu64 word;\n+\n+\tfor (w = 0; w \u003c MAX_FW_PAGES / 64 \u0026\u0026 found \u003c npages; w++) {\n+\t\tword = dev-\u003efw_pages_bitmap[w];\n+\n+\t\tfor (b = 0; b \u003c 64 \u0026\u0026 found \u003c npages; b++) {\n+\t\t\tif (!(word \u0026 (1ULL \u003c\u003c b))) {\n+\t\t\t\tunsigned int idx = w * 64 + b;\n+\n+\t\t\t\tdev-\u003efw_pages_bitmap[w] |= (1ULL \u003c\u003c b);\n+\t\t\t\tiovas[found++] = to_iova(device,\n+\t\t\t\t\t\t\t dev-\u003efw_pages[idx]);\n+\t\t\t}\n+\t\t}\n+\t}\n+\tVFIO_ASSERT_EQ(found, npages);\n+\tdev-\u003efw_pages_given += npages;\n+}\n+\n+static void mlx5st_fw_pages_free(struct mlx5st_device *dev,\n+\t\t\t\t unsigned int npages, const u64 *iovas)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tunsigned int i, idx;\n+\tu64 off;\n+\n+\tfor (i = 0; i \u003c npages; i++) {\n+\t\toff = iovas[i] - to_iova(device, dev-\u003efw_pages);\n+\t\tidx = off / MLX5_HW_PAGE_SIZE;\n+\n+\t\tVFIO_ASSERT_TRUE(idx \u003c MAX_FW_PAGES);\n+\t\tdev-\u003efw_pages_bitmap[idx / 64] \u0026= ~(1ULL \u003c\u003c (idx % 64));\n+\t}\n+\tdev-\u003efw_pages_given -= npages;\n+}\n+\n+static void *mlx5st_build_manage_pages_give(u16 func_id, unsigned int npages,\n+\t\t\t\t\t const u64 *iovas,\n+\t\t\t\t\t unsigned int *out_inlen)\n+{\n+\tunsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in) + npages * 8;\n+\tunsigned int i;\n+\tvoid *in;\n+\n+\tin = calloc(1, inlen);\n+\tVFIO_ASSERT_NOT_NULL(in);\n+\n+\tMLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES);\n+\tMLX5_SET(manage_pages_in, in, op_mod,\n+\t\t MLX5_MANAGE_PAGES_IN_OP_MOD_ALLOCATION_SUCCESS);\n+\tMLX5_SET(manage_pages_in, in, function_id, func_id);\n+\tMLX5_SET(manage_pages_in, in, input_num_entries, npages);\n+\n+\tfor (i = 0; i \u003c npages; i++)\n+\t\tMLX5_ARRAY_SET64(manage_pages_in, in, pas, i, iovas[i]);\n+\n+\t*out_inlen = inlen;\n+\treturn in;\n+}\n+\n+static void mlx5st_fw_pages_give_one(struct mlx5st_device *dev, u16 func_id,\n+\t\t\t\t unsigned int npages, u64 *iovas)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(manage_pages_out)] = {};\n+\tunsigned int inlen;\n+\tvoid *in;\n+\n+\tin = mlx5st_build_manage_pages_give(func_id, npages, iovas, \u0026inlen);\n+\tmlx5st_cmd_exec(dev, in, inlen, out, sizeof(out));\n+\tfree(in);\n+}\n+\n+static void mlx5st_fw_pages_give(struct mlx5st_device *dev, u16 func_id,\n+\t\t\t\t unsigned int npages)\n+{\n+\tunsigned int remaining = npages;\n+\tu64 *iovas;\n+\n+\tif (!npages)\n+\t\treturn;\n+\n+\tiovas = calloc(npages, sizeof(u64));\n+\tVFIO_ASSERT_NOT_NULL(iovas);\n+\n+\tmlx5st_fw_pages_alloc(dev, npages, iovas);\n+\n+\t/* Batch into chunks that fit in one mailbox */\n+\tfor (unsigned int off = 0; remaining \u003e 0;) {\n+\t\tunsigned int batch = remaining \u003c MAX_FW_PAGES_PER_CMD ?\n+\t\t\t\t\t remaining :\n+\t\t\t\t\t MAX_FW_PAGES_PER_CMD;\n+\n+\t\tmlx5st_fw_pages_give_one(dev, func_id, batch, iovas + off);\n+\t\toff += batch;\n+\t\tremaining -= batch;\n+\t}\n+\n+\tdev_dbg(dev-\u003edevice, \"MANAGE_PAGES GIVE: %d pages to func_id=%u\\n\",\n+\t\t npages, func_id);\n+\tfree(iovas);\n+}\n+\n+static void mlx5st_fw_pages_satisfy(struct mlx5st_device *dev, int boot)\n+{\n+\tu32 qo[MLX5_ST_SZ_DW(query_pages_out)] = {};\n+\tu32 qi[MLX5_ST_SZ_DW(query_pages_in)] = {};\n+\tu16 func_id;\n+\tint npages;\n+\n+\tMLX5_SET(query_pages_in, qi, opcode, MLX5_CMD_OP_QUERY_PAGES);\n+\tMLX5_SET(query_pages_in, qi, op_mod, boot ? 0x01 : 0x02);\n+\tmlx5st_cmd_exec(dev, qi, sizeof(qi), qo, sizeof(qo));\n+\n+\tnpages = MLX5_GET(query_pages_out, qo, num_pages);\n+\tfunc_id = MLX5_GET(query_pages_out, qo, function_id);\n+\tdev_dbg(dev-\u003edevice, \"QUERY_PAGES (%s): %d pages (func_id=%u)\\n\",\n+\t\t boot ? \"boot\" : \"init\", npages, func_id);\n+\n+\tif (npages \u003e 0) {\n+\t\tdev-\u003efw_func_id = func_id;\n+\t\tmlx5st_fw_pages_give(dev, func_id, npages);\n+\t}\n+}\n+\n+/*\n+ * Async MANAGE_PAGES on the pages command slot.\n+ *\n+ * On PFs, firmware sends PAGE_REQUEST events via the EQ during command\n+ * execution. We must respond with MANAGE_PAGES on a second command slot\n+ * before the first (regular) command can complete.\n+ */\n+\n+static void mlx5st_pages_slot_post(struct mlx5st_device *dev, void *in,\n+\t\t\t\t unsigned int ilen, unsigned int olen)\n+{\n+\tmlx5st_cmd_post(dev, dev-\u003epages_cmd_lay, dev-\u003epages_in_mbox,\n+\t\t\tdev-\u003epages_out_mbox, in, ilen, olen,\n+\t\t\t1 \u003c\u003c dev-\u003epages_slot);\n+}\n+\n+static void mlx5st_pages_slot_give(struct mlx5st_device *dev, u16 func_id,\n+\t\t\t\t unsigned int npages)\n+{\n+\tunsigned int inlen;\n+\tu64 *iovas;\n+\tvoid *in;\n+\n+\tiovas = calloc(npages, sizeof(u64));\n+\tVFIO_ASSERT_NOT_NULL(iovas);\n+\n+\tmlx5st_fw_pages_alloc(dev, npages, iovas);\n+\n+\tin = mlx5st_build_manage_pages_give(func_id, npages, iovas, \u0026inlen);\n+\tfree(iovas);\n+\n+\tmlx5st_pages_slot_post(dev, in, inlen,\n+\t\t\t MLX5_ST_SZ_BYTES(manage_pages_out));\n+\tdev-\u003epages_slot_in_use = true;\n+\tdev-\u003epages_slot_is_reclaim = false;\n+\tfree(in);\n+\n+\tdev_dbg(dev-\u003edevice,\n+\t\t \"PAGE_REQUEST: %d pages given async to func_id=%u\\n\",\n+\t\t npages, func_id);\n+}\n+\n+static void mlx5st_pages_slot_reclaim(struct mlx5st_device *dev, u16 func_id,\n+\t\t\t\t unsigned int npages)\n+{\n+\tunsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in);\n+\tunsigned int outlen =\n+\t\tMLX5_ST_SZ_BYTES(manage_pages_out) + npages * 8;\n+\tvoid *in;\n+\n+\tin = calloc(1, inlen);\n+\tVFIO_ASSERT_NOT_NULL(in);\n+\n+\tMLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES);\n+\tMLX5_SET(manage_pages_in, in, op_mod,\n+\t\t MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES);\n+\tMLX5_SET(manage_pages_in, in, function_id, func_id);\n+\tMLX5_SET(manage_pages_in, in, input_num_entries, npages);\n+\n+\tmlx5st_pages_slot_post(dev, in, inlen, outlen);\n+\tdev-\u003epages_slot_in_use = true;\n+\tdev-\u003epages_slot_is_reclaim = true;\n+\tdev-\u003epages_reclaim_npages = npages;\n+\tfree(in);\n+\n+\tdev_dbg(dev-\u003edevice,\n+\t\t \"PAGE_REQUEST: reclaim %d pages async from func_id=%u\\n\",\n+\t\t npages, func_id);\n+}\n+\n+static void mlx5st_pages_slot_kick(struct mlx5st_device *dev)\n+{\n+\tunsigned int batch;\n+\n+\tif (dev-\u003epages_slot_in_use)\n+\t\treturn;\n+\n+\tif (dev-\u003epages_pending_give) {\n+\t\tbatch = dev-\u003epages_pending_give \u003c MAX_FW_PAGES_PER_CMD ?\n+\t\t\t\tdev-\u003epages_pending_give :\n+\t\t\t\tMAX_FW_PAGES_PER_CMD;\n+\t\tdev-\u003epages_pending_give -= batch;\n+\t\tmlx5st_pages_slot_give(dev, dev-\u003epages_pending_func_id, batch);\n+\t} else if (dev-\u003epages_pending_reclaim) {\n+\t\tbatch = dev-\u003epages_pending_reclaim \u003c MAX_FW_PAGES_PER_CMD ?\n+\t\t\t\tdev-\u003epages_pending_reclaim :\n+\t\t\t\tMAX_FW_PAGES_PER_CMD;\n+\t\tdev-\u003epages_pending_reclaim -= batch;\n+\t\tmlx5st_pages_slot_reclaim(dev, dev-\u003epages_pending_func_id,\n+\t\t\t\t\t batch);\n+\t}\n+}\n+\n+static void mlx5st_fw_pages_give_async(struct mlx5st_device *dev,\n+\t\t\t\t\tu16 func_id, unsigned int npages)\n+{\n+\tif (!npages)\n+\t\treturn;\n+\n+\tdev-\u003epages_pending_give += npages;\n+\tdev-\u003epages_pending_func_id = func_id;\n+\tmlx5st_pages_slot_kick(dev);\n+}\n+\n+static void mlx5st_fw_pages_reclaim_async(struct mlx5st_device *dev,\n+\t\t\t\t\t u16 func_id, unsigned int npages)\n+{\n+\tdev-\u003epages_pending_reclaim += npages;\n+\tdev-\u003epages_pending_func_id = func_id;\n+\tmlx5st_pages_slot_kick(dev);\n+}\n+\n+static void mlx5st_pages_slot_complete(struct mlx5st_device *dev)\n+{\n+\tstruct mlx5st_cmd_queue_entry *cmd = dev-\u003epages_cmd_lay;\n+\tvoid *cout;\n+\n+\tdma_rmb();\n+\n+\tcout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);\n+\tif (MLX5_GET(enable_hca_out, cout, status) != MLX5_CMD_STAT_OK)\n+\t\tVFIO_FAIL(\"async MANAGE_PAGES failed: status=0x%x syndrome=0x%x\",\n+\t\t\t MLX5_GET(enable_hca_out, cout, status),\n+\t\t\t MLX5_GET(enable_hca_out, cout, syndrome));\n+\n+\tif (dev-\u003epages_slot_is_reclaim) {\n+\t\tunsigned int outlen = MLX5_ST_SZ_BYTES(manage_pages_out) +\n+\t\t\t\t dev-\u003epages_reclaim_npages * 8;\n+\t\tunsigned int num_claimed;\n+\t\tunsigned int i;\n+\t\tvoid *out;\n+\t\tu64 *iovas;\n+\n+\t\tout = calloc(1, outlen);\n+\t\tiovas = calloc(dev-\u003epages_reclaim_npages, sizeof(u64));\n+\t\tVFIO_ASSERT_NOT_NULL(out);\n+\t\tVFIO_ASSERT_NOT_NULL(iovas);\n+\n+\t\t/* Copy inline output */\n+\t\tmemcpy(out, cout, MLX5_CMD_INLINE_SZ);\n+\t\tif (outlen \u003e MLX5_CMD_INLINE_SZ)\n+\t\t\tmlx5st_cmd_copy_from_mbox(\n+\t\t\t\t(u8 *)out + MLX5_CMD_INLINE_SZ,\n+\t\t\t\tdev-\u003epages_out_mbox,\n+\t\t\t\toutlen - MLX5_CMD_INLINE_SZ);\n+\n+\t\tnum_claimed =\n+\t\t\tMLX5_GET(manage_pages_out, out, output_num_entries);\n+\t\tfor (i = 0; i \u003c num_claimed; i++)\n+\t\t\tiovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas,\n+\t\t\t\t\t\t i);\n+\n+\t\tmlx5st_fw_pages_free(dev, num_claimed, iovas);\n+\t\tdev_dbg(dev-\u003edevice, \"PAGE_REQUEST: reclaimed %d pages\\n\",\n+\t\t\t num_claimed);\n+\n+\t\tfree(iovas);\n+\t\tfree(out);\n+\t}\n+\n+\tdev-\u003epages_slot_in_use = false;\n+\tmlx5st_pages_slot_kick(dev);\n+}\n+\n+/*\n+ * UAR alloc/dealloc\n+ */\n+\n+static void mlx5st_alloc_uar(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(alloc_uar_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(alloc_uar_in)] = {};\n+\n+\tMLX5_SET(alloc_uar_in, in, opcode, MLX5_CMD_OP_ALLOC_UAR);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003euar_page = MLX5_GET(alloc_uar_out, out, uar);\n+\tdev-\u003euar_base = (u8 __iomem*)dev-\u003ebar0 + dev-\u003euar_page * MLX5_HW_PAGE_SIZE;\n+\tdev-\u003euar_bf_offset = MLX5_BF_OFFSET;\n+\n+\tdev_dbg(dev-\u003edevice,\n+\t\t \"Allocated UAR page_id=%u, doorbell offset=0x%x\\n\",\n+\t\t dev-\u003euar_page,\n+\t\t dev-\u003euar_page * MLX5_HW_PAGE_SIZE + MLX5_BF_OFFSET);\n+}\n+\n+static void mlx5st_dealloc_uar(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(dealloc_uar_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(dealloc_uar_in)] = {};\n+\n+\tMLX5_SET(dealloc_uar_in, in, opcode, MLX5_CMD_OP_DEALLOC_UAR);\n+\tMLX5_SET(dealloc_uar_in, in, uar, dev-\u003euar_page);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * EQ infrastructure\n+ */\n+\n+static struct mlx5st_eqe *mlx5st_eq_get_eqe(struct mlx5st_device *dev, u32 cc)\n+{\n+\tu32 ci = dev-\u003eeq_cons_index + cc;\n+\tstruct mlx5st_eqe *eqe = \u0026dev-\u003eeq_buf[ci % EQ_NENT];\n+\tu8 owner = MLX5_GET_ONCE(eqe, eqe, owner);\n+\tu8 expected = !!(ci \u0026 EQ_NENT);\n+\n+\tif (owner != expected)\n+\t\treturn NULL;\n+\tdma_rmb();\n+\treturn eqe;\n+}\n+\n+static void mlx5st_eq_update_ci(struct mlx5st_device *dev, u32 cc, bool arm)\n+{\n+\tu32 val;\n+\n+\tdev-\u003eeq_cons_index += cc;\n+\tval = (dev-\u003eeq_cons_index \u0026 0xffffff) | (dev-\u003eeqn \u003c\u003c 24);\n+\tiowrite32be(val, (u8 __iomem *)dev-\u003euar_base + MLX5_EQ_DOORBELL_OFFSET +\n+\t\t\t\t (arm ? 0 : 8));\n+}\n+\n+static void mlx5st_create_eq(struct mlx5st_device *dev)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tu64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {};\n+\tu32 out[MLX5_ST_SZ_DW(create_eq_out)] = {};\n+\tstruct mlx5_ifc_eqc_bits *eqc;\n+\tunsigned int i;\n+\t__be64 *pas;\n+\n+\t/* Initialize EQE owner bits */\n+\tfor (i = 0; i \u003c EQ_NENT; i++) {\n+\t\tstruct mlx5st_eqe *eqe = \u0026dev-\u003eeq_buf[i];\n+\n+\t\tMLX5_SET_ONCE(eqe, eqe, owner, 1);\n+\t}\n+\n+\tMLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ);\n+\n+\t/* Subscribe to CMD completions and PAGE_REQUEST events */\n+\tMLX5_ARRAY_SET64(create_eq_in, in, event_bitmask, 0,\n+\t\t\t (1ULL \u003c\u003c MLX5_EVENT_TYPE_CMD) |\n+\t\t\t\t (1ULL \u003c\u003c MLX5_EVENT_TYPE_PAGE_REQUEST));\n+\n+\teqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);\n+\tMLX5_SET(eqc, eqc, log_eq_size, LOG_EQ_SIZE);\n+\tMLX5_SET(eqc, eqc, uar_page, dev-\u003euar_page);\n+\tpas = MLX5_ADDR_OF(create_eq_in, in, pas);\n+\tVFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev-\u003eeq_buf, pas), 0u);\n+\tMLX5_SET(eqc, eqc, log_page_size, 0);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003eeqn = MLX5_GET(create_eq_out, out, eq_number);\n+\tdev-\u003eeq_cons_index = 0;\n+\tmlx5st_eq_update_ci(dev, 0, 0);\n+\tdev-\u003ehave_eq = true;\n+\n+\tdev_dbg(device, \"Created EQ: eqn=%u, %d entries (CMD+PAGE_REQUEST)\\n\",\n+\t\t dev-\u003eeqn, EQ_NENT);\n+}\n+\n+static void mlx5st_destroy_eq(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {};\n+\n+\tMLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ);\n+\tMLX5_SET(destroy_eq_in, in, eq_number, dev-\u003eeqn);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * Drain all pending EQ events. Dispatches PAGE_REQUEST to the async pages\n+ * slot and CMD completions to the pages slot completion handler.\n+ */\n+static void mlx5st_process_events(struct mlx5st_device *dev)\n+{\n+\tstruct mlx5st_eqe *eqe;\n+\tu32 cc = 0;\n+\n+\twhile ((eqe = mlx5st_eq_get_eqe(dev, cc))) {\n+\t\tu8 type = MLX5_GET(eqe, eqe, event_type);\n+\n+\t\tswitch (type) {\n+\t\tcase MLX5_EVENT_TYPE_PAGE_REQUEST: {\n+\t\t\tvoid *evdata = MLX5_ADDR_OF(eqe, eqe, event_data);\n+\t\t\tu16 func_id = MLX5_GET(pages_req_event, evdata,\n+\t\t\t\t\t function_id);\n+\t\t\ts32 npages = (s32)MLX5_GET(pages_req_event, evdata,\n+\t\t\t\t\t\t num_pages);\n+\n+\t\t\t/*\n+\t\t\t * The selftest doesn't use more than one func_id so a\n+\t\t\t * simple counter approach is possible.\n+\t\t\t */\n+\t\t\tif (dev-\u003epages_func_id_seen)\n+\t\t\t\tVFIO_ASSERT_EQ(func_id,\n+\t\t\t\t\t dev-\u003epages_pending_func_id);\n+\t\t\tdev-\u003epages_func_id_seen = true;\n+\n+\t\t\tif (npages \u003e 0)\n+\t\t\t\tmlx5st_fw_pages_give_async(dev, func_id,\n+\t\t\t\t\t\t\t npages);\n+\t\t\telse if (npages \u003c 0)\n+\t\t\t\tmlx5st_fw_pages_reclaim_async(dev, func_id,\n+\t\t\t\t\t\t\t -npages);\n+\t\t\tbreak;\n+\t\t}\n+\t\tcase MLX5_EVENT_TYPE_CMD: {\n+\t\t\tvoid *evdata = MLX5_ADDR_OF(eqe, eqe, event_data);\n+\t\t\tu32 vector = MLX5_GET(cmd_inter_comp_event, evdata,\n+\t\t\t\t\t command_completion_vector);\n+\n+\t\t\tif (vector \u0026 (1U \u003c\u003c dev-\u003epages_slot))\n+\t\t\t\tmlx5st_pages_slot_complete(dev);\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tbreak;\n+\t\t}\n+\t\tcc++;\n+\t}\n+\n+\tif (cc)\n+\t\tmlx5st_eq_update_ci(dev, cc, 0);\n+}\n+\n+/*\n+ * MSI EQ — dedicated EQ for CQ completion events that fires MSI-X.\n+ * Separate from the cmd/pages EQ so that only CQ completions (from\n+ * send_msi or memcpy) trigger the interrupt vector.\n+ */\n+\n+static void mlx5st_msi_eq_drain(struct mlx5st_device *dev)\n+{\n+\tu32 cc = 0;\n+\tu32 val;\n+\n+\twhile (cc \u003c MSI_EQ_NENT) {\n+\t\tu32 ci = dev-\u003emsi_eq_cons_index + cc;\n+\t\tstruct mlx5st_eqe *eqe =\n+\t\t\t\u0026dev-\u003emsi_eq_buf[ci % MSI_EQ_NENT];\n+\n+\t\tif (MLX5_GET_ONCE(eqe, eqe, owner) != !!(ci \u0026 MSI_EQ_NENT))\n+\t\t\tbreak;\n+\t\tcc++;\n+\t}\n+\n+\t/* Update consumer index and re-arm for next interrupt */\n+\tdev-\u003emsi_eq_cons_index += cc;\n+\tval = (dev-\u003emsi_eq_cons_index \u0026 0xffffff) | (dev-\u003emsi_eqn \u003c\u003c 24);\n+\tiowrite32be(val, (u8 __iomem *)dev-\u003euar_base + MLX5_EQ_DOORBELL_OFFSET);\n+}\n+\n+static void mlx5st_create_msi_eq(struct mlx5st_device *dev)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tu64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {};\n+\tu32 out[MLX5_ST_SZ_DW(create_eq_out)] = {};\n+\tstruct mlx5_ifc_eqc_bits *eqc;\n+\tunsigned int i;\n+\t__be64 *pas;\n+\n+\t/* Initialize EQE owner bits */\n+\tfor (i = 0; i \u003c MSI_EQ_NENT; i++) {\n+\t\tstruct mlx5st_eqe *eqe = \u0026dev-\u003emsi_eq_buf[i];\n+\n+\t\tMLX5_SET_ONCE(eqe, eqe, owner, 1);\n+\t}\n+\n+\tMLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ);\n+\n+\t/*\n+\t * No event_bitmask — completion events are routed to this EQ via\n+\t * the CQ's c_eqn field, not through CREATE_EQ subscription.\n+\t */\n+\teqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);\n+\tMLX5_SET(eqc, eqc, log_eq_size, LOG_MSI_EQ_SIZE);\n+\tMLX5_SET(eqc, eqc, uar_page, dev-\u003euar_page);\n+\tMLX5_SET(eqc, eqc, intr, MSI_VECTOR);\n+\tpas = MLX5_ADDR_OF(create_eq_in, in, pas);\n+\tVFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev-\u003emsi_eq_buf, pas), 0u);\n+\tMLX5_SET(eqc, eqc, log_page_size, 0);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003emsi_eqn = MLX5_GET(create_eq_out, out, eq_number);\n+\tdev-\u003emsi_eq_cons_index = 0;\n+\tdev-\u003ehave_msi_eq = true;\n+\tmlx5st_msi_eq_drain(dev);\n+\n+\tdev_dbg(device,\n+\t\t \"Created MSI EQ: eqn=%u, %d entries (COMP), vector=%d\\n\",\n+\t\t dev-\u003emsi_eqn, MSI_EQ_NENT, MSI_VECTOR);\n+}\n+\n+static void mlx5st_destroy_msi_eq(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {};\n+\n+\tMLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ);\n+\tMLX5_SET(destroy_eq_in, in, eq_number, dev-\u003emsi_eqn);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * HCA init / teardown\n+ */\n+\n+#define FW_INIT_TIMEOUT_MS 120000\n+#define FW_INIT_WAIT_MS 200\n+\n+static void mlx5st_wait_fw_init(struct mlx5st_device *dev)\n+{\n+\tstruct timespec start, now;\n+\tunsigned int elapsed;\n+\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026start);\n+\twhile (MLX5_GET_MMIO(initial_seg, dev-\u003einit_seg, initializing)) {\n+\t\tusleep(FW_INIT_WAIT_MS * 1000);\n+\t\tclock_gettime(CLOCK_MONOTONIC, \u0026now);\n+\t\telapsed = (now.tv_sec - start.tv_sec) * 1000 +\n+\t\t\t (now.tv_nsec - start.tv_nsec) / 1000000;\n+\t\tif (elapsed \u003e FW_INIT_TIMEOUT_MS)\n+\t\t\tVFIO_FAIL(\"FW init timeout after %d ms\", elapsed);\n+\t}\n+}\n+\n+static void mlx5st_set_issi(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(set_issi_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(set_issi_in)] = {};\n+\n+\tMLX5_SET(set_issi_in, in, opcode, MLX5_CMD_OP_SET_ISSI);\n+\tMLX5_SET(set_issi_in, in, current_issi, 1);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"SET_ISSI: OK (issi=1)\\n\");\n+}\n+\n+static void mlx5st_set_hca_caps(struct mlx5st_device *dev)\n+{\n+\tu32 qout[MLX5_ST_SZ_DW(query_hca_cap_out)] = {};\n+\tu32 qin[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};\n+\tu32 sout[MLX5_ST_SZ_DW(set_hca_cap_out)] = {};\n+\tu32 sin[MLX5_ST_SZ_DW(set_hca_cap_in)] = {};\n+\tstruct mlx5_ifc_cmd_hca_cap_bits *set_hca_cap;\n+\tu32 max_checksum;\n+\n+\t/* Query max caps to learn cmdif_checksum support */\n+\tMLX5_SET(query_hca_cap_in, qin, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);\n+\tMLX5_SET(query_hca_cap_in, qin, op_mod,\n+\t\t (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE \u003c\u003c 1));\n+\tmlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout));\n+\n+\tmax_checksum = MLX5_GET(\n+\t\tcmd_hca_cap,\n+\t\tMLX5_ADDR_OF(query_hca_cap_out, qout, capability),\n+\t\tcmdif_checksum);\n+\n+\t/* Query current caps as base for SET */\n+\tmemset(qout, 0, sizeof(qout));\n+\tMLX5_SET(query_hca_cap_in, qin, op_mod,\n+\t\t (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE \u003c\u003c 1) |\n+\t\t\t HCA_CAP_OPMOD_GET_CUR);\n+\tmlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout));\n+\n+\tset_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, sin, capability);\n+\tmemcpy(set_hca_cap,\n+\t MLX5_ADDR_OF(query_hca_cap_out, qout, capability),\n+\t MLX5_ST_SZ_BYTES(cmd_hca_cap));\n+\n+\tMLX5_SET(cmd_hca_cap, set_hca_cap, cmdif_checksum, max_checksum);\n+\tMLX5_SET(cmd_hca_cap, set_hca_cap, log_uar_page_sz, 0);\n+\n+\tMLX5_SET(set_hca_cap_in, sin, opcode, MLX5_CMD_OP_SET_HCA_CAP);\n+\tMLX5_SET(set_hca_cap_in, sin, op_mod,\n+\t\t MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE \u003c\u003c 1);\n+\n+\tmlx5st_cmd_exec(dev, sin, sizeof(sin), sout, sizeof(sout));\n+\n+\tdev-\u003ecmd_sig_enabled = max_checksum == 0x3;\n+\tdev_dbg(dev-\u003edevice, \"SET_HCA_CAP: OK (cmdif_checksum=%u)\\n\",\n+\t\t max_checksum);\n+}\n+\n+static void mlx5st_hca_init(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(enable_hca_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(enable_hca_in)] = {};\n+\n+\tmlx5st_wait_fw_init(dev);\n+\tdev_dbg(dev-\u003edevice, \"Firmware ready\\n\");\n+\n+\tMLX5_SET(enable_hca_in, in, opcode, MLX5_CMD_OP_ENABLE_HCA);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"ENABLE_HCA: OK\\n\");\n+\n+\tmlx5st_set_issi(dev);\n+\tmlx5st_fw_pages_satisfy(dev, 1);\n+\n+\tmlx5st_set_hca_caps(dev);\n+\tmlx5st_fw_pages_satisfy(dev, 0);\n+\n+\tmemset(in, 0, sizeof(in));\n+\tmemset(out, 0, sizeof(out));\n+\tMLX5_SET(init_hca_in, in, opcode, MLX5_CMD_OP_INIT_HCA);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"INIT_HCA: OK\\n\");\n+\n+\t/*\n+\t * Create EQ immediately after INIT_HCA so PAGE_REQUEST events\n+\t * are captured during all subsequent commands.\n+\t */\n+\tmlx5st_alloc_uar(dev);\n+\tmlx5st_create_eq(dev);\n+}\n+\n+static void mlx5st_disable_hca(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(disable_hca_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(disable_hca_in)] = {};\n+\n+\tMLX5_SET(disable_hca_in, in, opcode, MLX5_CMD_OP_DISABLE_HCA);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+static void mlx5st_fw_pages_reclaim(struct mlx5st_device *dev, u16 func_id)\n+{\n+\tunsigned int npages = dev-\u003efw_pages_given;\n+\tunsigned int total_claimed = 0;\n+\n+\twhile (npages \u003e 0) {\n+\t\tunsigned int batch = npages \u003c MAX_FW_PAGES_PER_CMD ?\n+\t\t\t\t\t npages :\n+\t\t\t\t\t MAX_FW_PAGES_PER_CMD;\n+\t\tunsigned int outlen =\n+\t\t\tMLX5_ST_SZ_BYTES(manage_pages_out) + batch * 8;\n+\t\tunsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in);\n+\t\tunsigned int num_claimed;\n+\t\tunsigned int i;\n+\t\tvoid *in, *out;\n+\t\tu64 *iovas;\n+\n+\t\tin = calloc(1, inlen);\n+\t\tout = calloc(1, outlen);\n+\t\tiovas = calloc(batch, sizeof(u64));\n+\t\tVFIO_ASSERT_NOT_NULL(in);\n+\t\tVFIO_ASSERT_NOT_NULL(out);\n+\t\tVFIO_ASSERT_NOT_NULL(iovas);\n+\n+\t\tMLX5_SET(manage_pages_in, in, opcode,\n+\t\t\t MLX5_CMD_OP_MANAGE_PAGES);\n+\t\tMLX5_SET(manage_pages_in, in, op_mod,\n+\t\t\t MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES);\n+\t\tMLX5_SET(manage_pages_in, in, function_id, func_id);\n+\t\tMLX5_SET(manage_pages_in, in, input_num_entries, batch);\n+\n+\t\tmlx5st_cmd_exec(dev, in, inlen, out, outlen);\n+\n+\t\tnum_claimed =\n+\t\t\tMLX5_GET(manage_pages_out, out, output_num_entries);\n+\t\tfor (i = 0; i \u003c num_claimed; i++)\n+\t\t\tiovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas,\n+\t\t\t\t\t\t i);\n+\n+\t\tmlx5st_fw_pages_free(dev, num_claimed, iovas);\n+\t\ttotal_claimed += num_claimed;\n+\t\tnpages -= num_claimed;\n+\n+\t\tfree(iovas);\n+\t\tfree(in);\n+\t\tfree(out);\n+\n+\t\tif (!num_claimed \u0026\u0026 !dev-\u003efw_pages_given)\n+\t\t\tbreak;\n+\t\tif (!num_claimed)\n+\t\t\tVFIO_FAIL(\"MANAGE_PAGES RECLAIM: FW returned 0 but %d pages still given\",\n+\t\t\t\t dev-\u003efw_pages_given);\n+\t}\n+\n+\tdev_dbg(dev-\u003edevice,\n+\t\t \"MANAGE_PAGES RECLAIM: %d pages (%d still given)\\n\",\n+\t\t total_claimed, dev-\u003efw_pages_given);\n+}\n+\n+static void mlx5st_hca_teardown(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(teardown_hca_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(teardown_hca_in)] = {};\n+\n+\t/* Drain async pages slot, then stop EQ processing */\n+\twhile (dev-\u003epages_slot_in_use) {\n+\t\tif (!MLX5_GET_ONCE(cmd_queue_entry, dev-\u003epages_cmd_lay,\n+\t\t\t\t ownership))\n+\t\t\tmlx5st_pages_slot_complete(dev);\n+\t\telse\n+\t\t\tsched_yield();\n+\t}\n+\tdev-\u003ehave_eq = false;\n+\n+\tif (dev-\u003eeqn) {\n+\t\tmlx5st_destroy_eq(dev);\n+\t\tdev-\u003eeqn = 0;\n+\t}\n+\tif (dev-\u003euar_page) {\n+\t\tmlx5st_dealloc_uar(dev);\n+\t\tdev-\u003euar_page = 0;\n+\t}\n+\n+\tdev_dbg(dev-\u003edevice, \" hca_teardown: TEARDOWN_HCA\\n\");\n+\tMLX5_SET(teardown_hca_in, in, opcode, MLX5_CMD_OP_TEARDOWN_HCA);\n+\tMLX5_SET(teardown_hca_in, in, profile,\n+\t\t MLX5_TEARDOWN_HCA_IN_PROFILE_GRACEFUL_CLOSE);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tif (dev-\u003efw_pages_given \u003e 0) {\n+\t\tdev_dbg(dev-\u003edevice, \" hca_teardown: reclaim %d pages\\n\",\n+\t\t\t dev-\u003efw_pages_given);\n+\t\tmlx5st_fw_pages_reclaim(dev, dev-\u003efw_func_id);\n+\t}\n+\n+\tdev_dbg(dev-\u003edevice, \" hca_teardown: DISABLE_HCA\\n\");\n+\tmlx5st_disable_hca(dev);\n+}\n+\n+/*\n+ * Query capabilities\n+ */\n+static void mlx5st_query_fl_caps(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};\n+\tbool fl_roce_en, fl_roce_dis;\n+\n+\t/* Query RoCE capabilities */\n+\tMLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);\n+\tMLX5_SET(query_hca_cap_in, in, op_mod,\n+\t\t (MLX5_SET_HCA_CAP_OP_MOD_ROCE \u003c\u003c 1) | HCA_CAP_OPMOD_GET_CUR);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tfl_roce_en = MLX5_GET(query_hca_cap_out, out,\n+\t\t\t capability.roce_cap.fl_rc_qp_when_roce_enabled);\n+\tfl_roce_dis = MLX5_GET(query_hca_cap_out, out,\n+\t\t\t capability.roce_cap.fl_rc_qp_when_roce_disabled);\n+\n+\t/* Also check general caps */\n+\tmemset(in, 0, sizeof(in));\n+\tmemset(out, 0, sizeof(out));\n+\tMLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);\n+\tMLX5_SET(query_hca_cap_in, in, op_mod,\n+\t\t (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE \u003c\u003c 1) |\n+\t\t\t HCA_CAP_OPMOD_GET_CUR);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tfl_roce_dis |=\n+\t\tMLX5_GET(query_hca_cap_out, out,\n+\t\t\t capability.cmd_hca_cap.fl_rc_qp_when_roce_disabled);\n+\n+\tdev-\u003efl_supported = fl_roce_en || fl_roce_dis;\n+\tdev-\u003elog_max_msg = MLX5_GET(query_hca_cap_out, out,\n+\t\t\t\t capability.cmd_hca_cap.log_max_msg);\n+\tdev_dbg(dev-\u003edevice,\n+\t\t \"HCA capabilities: fl_roce_enabled=%d fl_roce_disabled=%d log_max_msg=%u\\n\",\n+\t\t fl_roce_en, fl_roce_dis, dev-\u003elog_max_msg);\n+\n+\tVFIO_ASSERT_TRUE(dev-\u003efl_supported,\n+\t\t\t \"Force-loopback not supported on this device\");\n+}\n+\n+/*\n+ * Resource allocation\n+ */\n+\n+static void mlx5st_alloc_pd(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(alloc_pd_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(alloc_pd_in)] = {};\n+\n+\tMLX5_SET(alloc_pd_in, in, opcode, MLX5_CMD_OP_ALLOC_PD);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003epdn = MLX5_GET(alloc_pd_out, out, pd);\n+\tdev_dbg(dev-\u003edevice, \"Allocated PD pdn=%u\\n\", dev-\u003epdn);\n+}\n+\n+static void mlx5st_dealloc_pd(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(dealloc_pd_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(dealloc_pd_in)] = {};\n+\n+\tMLX5_SET(dealloc_pd_in, in, opcode, MLX5_CMD_OP_DEALLOC_PD);\n+\tMLX5_SET(dealloc_pd_in, in, pd, dev-\u003epdn);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+static void mlx5st_create_mkey(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(create_mkey_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};\n+\tstruct mlx5_ifc_mkc_bits *mkc;\n+\n+\tMLX5_SET(create_mkey_in, in, opcode, MLX5_CMD_OP_CREATE_MKEY);\n+\n+\tmkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);\n+\tMLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA);\n+\tMLX5_SET(mkc, mkc, length64, 1);\n+\tMLX5_SET(mkc, mkc, pd, dev-\u003epdn);\n+\tMLX5_SET(mkc, mkc, qpn, 0xffffff);\n+\tMLX5_SET(mkc, mkc, lr, 1);\n+\tMLX5_SET(mkc, mkc, lw, 1);\n+\tMLX5_SET(mkc, mkc, rw, 1);\n+\tMLX5_SET(mkc, mkc, rr, 1);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003emkey_index = MLX5_GET(create_mkey_out, out, mkey_index);\n+\tdev-\u003eglobal_lkey = mlx5st_idx_to_mkey(dev-\u003emkey_index);\n+\tdev-\u003eglobal_rkey = dev-\u003eglobal_lkey;\n+\n+\tdev_dbg(dev-\u003edevice, \"Created global PA-mode MKEY: lkey=0x%x\\n\",\n+\t\t dev-\u003eglobal_lkey);\n+}\n+\n+static void mlx5st_destroy_mkey(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(destroy_mkey_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(destroy_mkey_in)] = {};\n+\n+\tMLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY);\n+\tMLX5_SET(destroy_mkey_in, in, mkey_index, dev-\u003emkey_index);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * CQ create/destroy\n+ */\n+\n+static void mlx5st_create_cq(struct mlx5st_device *dev)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tu64 in[MLX5_ST_SZ_QW(create_cq_in) + 1] = {};\n+\tu32 out[MLX5_ST_SZ_DW(create_cq_out)] = {};\n+\tstruct mlx5_ifc_cqc_bits *cqc;\n+\tunsigned int i;\n+\t__be64 *pas;\n+\n+\t/* Initialize CQEs before CREATE_CQ: opcode=0xF, owner=1 */\n+\tfor (i = 0; i \u003c CQ_CQE_CNT; i++) {\n+\t\tstruct mlx5st_cqe64 *cqe = \u0026dev-\u003ecq_buf[i];\n+\n+\t\tMLX5_SET(cqe64, cqe, opcode, 0xF);\n+\t\tMLX5_SET_ONCE(cqe64, cqe, owner, 1);\n+\t}\n+\n+\tMLX5_SET(create_cq_in, in, opcode, MLX5_CMD_OP_CREATE_CQ);\n+\n+\tcqc = MLX5_ADDR_OF(create_cq_in, in, cq_context);\n+\tMLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE);\n+\tMLX5_SET(cqc, cqc, uar_page, dev-\u003euar_page);\n+\tMLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev-\u003emsi_eqn);\n+\tMLX5_SET(cqc, cqc, cqe_sz, 0);\n+\tpas = MLX5_ADDR_OF(create_cq_in, in, pas);\n+\tMLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev-\u003ecq_buf, pas));\n+\tMLX5_SET(cqc, cqc, log_page_size, 0);\n+\tMLX5_SET64(cqc, cqc, dbr_addr, to_iova(device, \u0026dev-\u003ecq_dbrec));\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003ecqn = MLX5_GET(create_cq_out, out, cqn);\n+\tdev-\u003ecq_ci = 0;\n+\tdev_dbg(device, \"Created CQ: cqn=%u, %d entries\\n\", dev-\u003ecqn,\n+\t\t CQ_CQE_CNT);\n+}\n+\n+static void mlx5st_destroy_cq(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(destroy_cq_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {};\n+\n+\tMLX5_SET(destroy_cq_in, in, opcode, MLX5_CMD_OP_DESTROY_CQ);\n+\tMLX5_SET(destroy_cq_in, in, cqn, dev-\u003ecqn);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * Arm CQ for event generation. The CQ event delivery state machine is\n+ * single-shot: after generating one EQE the CQ enters \"Fired\" state and\n+ * won't generate another until re-armed via ARM_NEXT. Both the CQ doorbell\n+ * record and the UAR CQ doorbell register must be written.\n+ */\n+static void mlx5st_arm_cq(struct mlx5st_device *dev)\n+{\n+\tu32 sn = dev-\u003ecq_arm_sn \u0026 3;\n+\tu32 ci = dev-\u003ecq_ci \u0026 0xffffff;\n+\tu64 doorbell;\n+\n+\t/* Update CQ doorbell record arm word */\n+\tWRITE_ONCE(dev-\u003ecq_dbrec.send_counter,\n+\t\t cpu_to_be32(sn \u003c\u003c 28 | ci));\n+\n+\t/* Ring CQ doorbell register, iowrite has an internal dma_wmb() */\n+\tdoorbell = ((u64)(sn \u003c\u003c 28 | ci) \u003c\u003c 32) | dev-\u003ecqn;\n+\tiowrite64be(doorbell,\n+\t\t (u8 __iomem *)dev-\u003euar_base + MLX5_CQ_DOORBELL_OFFSET);\n+\n+\tdev-\u003ecq_arm_sn++;\n+}\n+\n+/*\n+ * QP create/destroy\n+ */\n+\n+static void mlx5st_create_qp(struct mlx5st_device *dev)\n+{\n+\tstruct vfio_pci_device *device = dev-\u003edevice;\n+\tu64 in[MLX5_ST_SZ_QW(create_qp_in) + 1] = {};\n+\tu32 out[MLX5_ST_SZ_DW(create_qp_out)] = {};\n+\tstruct mlx5_ifc_qpc_bits *qpc;\n+\t__be64 *pas;\n+\n+\tMLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);\n+\n+\tqpc = MLX5_ADDR_OF(create_qp_in, in, qpc);\n+\tMLX5_SET(qpc, qpc, st, MLX5_QPC_ST_RC);\n+\tMLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);\n+\tMLX5_SET(qpc, qpc, pd, dev-\u003epdn);\n+\tMLX5_SET(qpc, qpc, uar_page, dev-\u003euar_page);\n+\tMLX5_SET(qpc, qpc, cqn_snd, dev-\u003ecqn);\n+\tMLX5_SET(qpc, qpc, cqn_rcv, dev-\u003ecqn);\n+\tMLX5_SET(qpc, qpc, log_sq_size, LOG_SQ_SIZE);\n+\tMLX5_SET(qpc, qpc, log_msg_max, dev-\u003elog_max_msg);\n+\tMLX5_SET(qpc, qpc, rq_type, 0x3);\n+\tMLX5_SET(qpc, qpc, ts_format, 1);\n+\tpas = MLX5_ADDR_OF(create_qp_in, in, pas);\n+\tMLX5_SET(qpc, qpc, page_offset,\n+\t\t mlx5st_fill_pas(device, dev-\u003esq_buf, pas));\n+\tMLX5_SET(qpc, qpc, log_page_size, 0);\n+\tMLX5_SET64(qpc, qpc, dbr_addr, to_iova(device, \u0026dev-\u003eqp_dbrec));\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\n+\tdev-\u003eqpn = MLX5_GET(create_qp_out, out, qpn);\n+\tdev-\u003esq_pi = 0;\n+\tdev_dbg(device, \"Created QP: qpn=%u, RC, sq=%d wqes\\n\", dev-\u003eqpn,\n+\t\t SQ_WQE_CNT);\n+}\n+\n+static void mlx5st_destroy_qp(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(destroy_qp_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(destroy_qp_in)] = {};\n+\n+\tMLX5_SET(destroy_qp_in, in, opcode, MLX5_CMD_OP_DESTROY_QP);\n+\tMLX5_SET(destroy_qp_in, in, qpn, dev-\u003eqpn);\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+}\n+\n+/*\n+ * QP state transitions\n+ */\n+\n+static void mlx5st_qp_rst2init(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(rst2init_qp_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {};\n+\tstruct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc);\n+\n+\tMLX5_SET(rst2init_qp_in, in, opcode, MLX5_CMD_OP_RST2INIT_QP);\n+\tMLX5_SET(rst2init_qp_in, in, qpn, dev-\u003eqpn);\n+\n+\tMLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1);\n+\tMLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);\n+\tMLX5_SET(qpc, qpc, rre, 1);\n+\tMLX5_SET(qpc, qpc, rwe, 1);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"QP RST-\u003eINIT\\n\");\n+}\n+\n+static void mlx5st_qp_init2rtr(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(init2rtr_qp_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {};\n+\tstruct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc);\n+\n+\tMLX5_SET(init2rtr_qp_in, in, opcode, MLX5_CMD_OP_INIT2RTR_QP);\n+\tMLX5_SET(init2rtr_qp_in, in, qpn, dev-\u003eqpn);\n+\n+\tMLX5_SET(qpc, qpc, mtu, 3);\n+\tMLX5_SET(qpc, qpc, log_msg_max, dev-\u003elog_max_msg);\n+\tMLX5_SET(qpc, qpc, remote_qpn, dev-\u003eqpn);\n+\tMLX5_SET(qpc, qpc, min_rnr_nak, 12);\n+\tMLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1);\n+\tMLX5_SET(qpc, qpc, primary_address_path.fl, 1);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"QP INIT-\u003eRTR (fl=1)\\n\");\n+}\n+\n+static void mlx5st_qp_rtr2rts(struct mlx5st_device *dev)\n+{\n+\tu32 out[MLX5_ST_SZ_DW(rtr2rts_qp_out)] = {};\n+\tu32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {};\n+\tstruct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc);\n+\n+\tMLX5_SET(rtr2rts_qp_in, in, opcode, MLX5_CMD_OP_RTR2RTS_QP);\n+\tMLX5_SET(rtr2rts_qp_in, in, qpn, dev-\u003eqpn);\n+\n+\tMLX5_SET(qpc, qpc, log_ack_req_freq, 0);\n+\tMLX5_SET(qpc, qpc, retry_count, 7);\n+\tMLX5_SET(qpc, qpc, rnr_retry, 7);\n+\tMLX5_SET(qpc, qpc, primary_address_path.ack_timeout, 14);\n+\n+\tmlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));\n+\tdev_dbg(dev-\u003edevice, \"QP RTR-\u003eRTS\\n\");\n+}\n+\n+/*\n+ * Post RDMA Write WQE\n+ */\n+static void mlx5st_post_rdma_write(struct mlx5st_device *dev, u64 src_addr,\n+\t\t\t\t u32 src_lkey, u64 dst_addr, u32 dst_rkey,\n+\t\t\t\t u32 length, bool signaled)\n+{\n+\tstruct mlx5st_send_wqe *wqe;\n+\tunsigned int idx;\n+\n+\tidx = dev-\u003esq_pi % SQ_WQE_CNT;\n+\twqe = \u0026dev-\u003esq_buf[idx];\n+\n+\tmemset(wqe, 0, sizeof(*wqe));\n+\tMLX5_SET(wqe_ctrl_seg, \u0026wqe-\u003ectrl, opcode, MLX5_OPCODE_RDMA_WRITE);\n+\tMLX5_SET(wqe_ctrl_seg, \u0026wqe-\u003ectrl, wqe_index, dev-\u003esq_pi);\n+\tMLX5_SET(wqe_ctrl_seg, \u0026wqe-\u003ectrl, qp_or_sq, dev-\u003eqpn);\n+\tMLX5_SET(wqe_ctrl_seg, \u0026wqe-\u003ectrl, ds, MLX5_RDMA_WRITE_DS);\n+\tif (signaled)\n+\t\tMLX5_SET(wqe_ctrl_seg, \u0026wqe-\u003ectrl, ce, MLX5_WQE_CE_CQE_ALWAYS);\n+\n+\tMLX5_SET64(wqe_raddr_seg, \u0026wqe-\u003eraddr, raddr, dst_addr);\n+\tMLX5_SET(wqe_raddr_seg, \u0026wqe-\u003eraddr, rkey, dst_rkey);\n+\n+\tMLX5_SET(wqe_data_seg, \u0026wqe-\u003edata, byte_count, length);\n+\tMLX5_SET(wqe_data_seg, \u0026wqe-\u003edata, lkey, src_lkey);\n+\tMLX5_SET64(wqe_data_seg, \u0026wqe-\u003edata, addr, src_addr);\n+\n+\tdev-\u003esq_pi++;\n+\n+\t/* Ensure WQE is visible to device before doorbell record */\n+\tdma_wmb();\n+\n+\tWRITE_ONCE(dev-\u003eqp_dbrec.send_counter,\n+\t\t cpu_to_be32(dev-\u003esq_pi \u0026 0xffff));\n+\n+\t/*\n+\t * Ring doorbell: write first 8 bytes of ctrl to UAR BF register,\n+\t * iowrite has an internal dma_wmb() so the doorbell record will be\n+\t * visible.\n+\t */\n+\tiowrite64be(be64_to_cpu(*(__be64 *)wqe),\n+\t\t (u8 __iomem *)dev-\u003euar_base + dev-\u003euar_bf_offset);\n+\tdev-\u003euar_bf_offset ^= MLX5_BF_SIZE;\n+}\n+\n+/*\n+ * Poll CQ\n+ */\n+static int mlx5st_poll_cq_batch(struct mlx5st_device *dev,\n+\t\t\t\tunsigned int max_cqe)\n+{\n+\tunsigned int polled = 0;\n+\n+\twhile (polled \u003c max_cqe) {\n+\t\tunsigned int idx = dev-\u003ecq_ci % CQ_CQE_CNT;\n+\t\tstruct mlx5st_cqe64 *cqe = \u0026dev-\u003ecq_buf[idx];\n+\t\tu8 owner, opcode;\n+\n+\t\towner = MLX5_GET_ONCE(cqe64, cqe, owner);\n+\t\tif (owner != ((dev-\u003ecq_ci \u003e\u003e LOG_CQ_SIZE) \u0026 1))\n+\t\t\tbreak;\n+\n+\t\tdma_rmb();\n+\n+\t\topcode = MLX5_GET(cqe64, cqe, opcode);\n+\n+\t\tdev-\u003ecq_ci++;\n+\t\tWRITE_ONCE(dev-\u003ecq_dbrec.recv_counter,\n+\t\t\t cpu_to_be32(dev-\u003ecq_ci \u0026 0xffffff));\n+\n+\t\tif (opcode == MLX5_CQE_REQ) {\n+\t\t\tdev-\u003esq_ci =\n+\t\t\t\t(u16)(MLX5_GET(cqe64, cqe, wqe_counter) + 1);\n+\t\t\tpolled++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (opcode == MLX5_CQE_REQ_ERR ||\n+\t\t opcode == MLX5_CQE_RESP_ERR) {\n+\t\t\tdev_dbg(dev-\u003edevice,\n+\t\t\t\t\"CQE error: opcode=0x%x syndrome=0x%x vendor=0x%x\\n\",\n+\t\t\t\topcode,\n+\t\t\t\tMLX5_GET(cqe64, cqe, error_syndrome.syndrome),\n+\t\t\t\tMLX5_GET(cqe64, cqe,\n+\t\t\t\t\t error_syndrome.vendor_error_syndrome));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tdev_err(dev-\u003edevice, \"CQE unexpected opcode=0x%x\\n\", opcode);\n+\t\treturn -1;\n+\t}\n+\n+\treturn polled;\n+}\n+\n+static int mlx5st_poll_cq(struct mlx5st_device *dev, unsigned int timeout_ms)\n+{\n+\tstruct timespec start, now;\n+\tunsigned int elapsed;\n+\tint ret;\n+\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026start);\n+\tfor (;;) {\n+\t\tret = mlx5st_poll_cq_batch(dev, 1);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn -1;\n+\t\tif (ret \u003e 0)\n+\t\t\treturn 0;\n+\n+\t\tif (dev-\u003ehave_eq)\n+\t\t\tmlx5st_process_events(dev);\n+\n+\t\tclock_gettime(CLOCK_MONOTONIC, \u0026now);\n+\t\telapsed = (now.tv_sec - start.tv_sec) * 1000 +\n+\t\t\t (now.tv_nsec - start.tv_nsec) / 1000000;\n+\t\tif (elapsed \u003e timeout_ms) {\n+\t\t\tdev_err(dev-\u003edevice, \"CQ poll timeout after %u ms\\n\",\n+\t\t\t\ttimeout_ms);\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Data path setup/teardown helpers\n+ */\n+\n+static void mlx5st_setup_datapath(struct mlx5st_device *dev)\n+{\n+\tmlx5st_create_cq(dev);\n+\tmlx5st_create_qp(dev);\n+\tmlx5st_qp_rst2init(dev);\n+\tmlx5st_qp_init2rtr(dev);\n+\tmlx5st_qp_rtr2rts(dev);\n+}\n+\n+static void mlx5st_teardown_datapath(struct mlx5st_device *dev)\n+{\n+\tif (dev-\u003eqpn) {\n+\t\tmlx5st_destroy_qp(dev);\n+\t\tdev-\u003eqpn = 0;\n+\t}\n+\tif (dev-\u003ecqn) {\n+\t\tmlx5st_destroy_cq(dev);\n+\t\tdev-\u003ecqn = 0;\n+\t}\n+\tdev-\u003esq_pi = 0;\n+\tdev-\u003esq_ci = 0;\n+\tdev-\u003ecq_arm_sn = 0;\n+\tmemset(\u0026dev-\u003eqp_dbrec, 0, sizeof(dev-\u003eqp_dbrec));\n+\tmemset(\u0026dev-\u003ecq_dbrec, 0, sizeof(dev-\u003ecq_dbrec));\n+}\n+\n+/*\n+ * memcpy callbacks\n+ */\n+\n+#define MLX5ST_MEMCPY_TIMEOUT_MS 60000\n+\n+static void mlx5st_memcpy_start(struct vfio_pci_device *device,\n+\t\t\t\t iova_t src, iova_t dst, u64 size, u64 count)\n+{\n+\tstruct mlx5st_device *dev = to_mlx5st(device);\n+\tu64 i;\n+\n+\tfor (i = 0; i \u003c count; i++) {\n+\t\tbool signaled = (i == count - 1);\n+\n+\t\tmlx5st_post_rdma_write(dev, src, dev-\u003eglobal_lkey, dst,\n+\t\t\t\t dev-\u003eglobal_rkey, size, signaled);\n+\t}\n+}\n+\n+static int mlx5st_memcpy_wait(struct vfio_pci_device *device)\n+{\n+\tstruct mlx5st_device *dev = to_mlx5st(device);\n+\tint ret;\n+\n+\tret = mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS);\n+\tif (ret) {\n+\t\t/*\n+\t\t * CQE error puts the QP in error state. Rebuild the data path\n+\t\t * so subsequent operations can succeed.\n+\t\t */\n+\t\tmlx5st_teardown_datapath(dev);\n+\t\tmlx5st_setup_datapath(dev);\n+\t}\n+\treturn ret;\n+}\n+\n+/*\n+ * send_msi callback — trigger CQE -\u003e EQE -\u003e MSI-X via a small RDMA Write.\n+ *\n+ * Both the CQ and MSI EQ use single-shot arming: the CQ must be armed so the\n+ * CQE generates an EQE, and the MSI EQ must be armed so the EQE fires MSI-X.\n+ */\n+static void mlx5st_send_msi(struct vfio_pci_device *device)\n+{\n+\tstruct mlx5st_device *dev = to_mlx5st(device);\n+\n+\t/* Drain accumulated MSI EQ events and re-arm for next interrupt */\n+\tmlx5st_msi_eq_drain(dev);\n+\n+\t/* Arm CQ so the next CQE generates an EQE on the MSI EQ */\n+\tmlx5st_arm_cq(dev);\n+\n+\t/* Post a signaled RDMA Write to trigger CQE -\u003e EQE -\u003e MSI-X */\n+\tmlx5st_post_rdma_write(dev,\n+\t\t\t to_iova(device, \u0026dev-\u003esend_msi_src),\n+\t\t\t dev-\u003eglobal_lkey,\n+\t\t\t to_iova(device, \u0026dev-\u003esend_msi_dst),\n+\t\t\t dev-\u003eglobal_rkey,\n+\t\t\t sizeof(dev-\u003esend_msi_src), true);\n+\n+\t/* Consume the CQE to avoid stale completions */\n+\tVFIO_ASSERT_EQ(mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS), 0);\n+}\n+\n+/*\n+ * Driver ops callbacks\n+ */\n+\n+static void mlx5st_init(struct vfio_pci_device *device)\n+{\n+\tstruct mlx5st_device *dev = to_mlx5st(device);\n+\tiova_t iova_align =\n+\t\tdevice-\u003edriver.region.iova % __alignof__(struct mlx5st_device);\n+\n+\tVFIO_ASSERT_GE(device-\u003edriver.region.size, sizeof(*dev));\n+\tVFIO_ASSERT_EQ(iova_align, 0);\n+\tmemset(dev, 0, sizeof(*dev));\n+\n+\tdev-\u003edevice = device;\n+\tdev-\u003ebar0 = device-\u003ebars[0].vaddr;\n+\tdev-\u003einit_seg = dev-\u003ebar0;\n+\n+\tvfio_pci_config_writew(device, PCI_COMMAND,\n+\t\t\t vfio_pci_config_readw(device, PCI_COMMAND) |\n+\t\t\t\t PCI_COMMAND_MASTER);\n+\n+\tmlx5st_wait_fw_init(dev);\n+\n+\tmlx5st_cmd_init(dev);\n+\tmlx5st_hca_init(dev);\n+\tmlx5st_query_fl_caps(dev);\n+\tmlx5st_alloc_pd(dev);\n+\tmlx5st_create_mkey(dev);\n+\n+\t/* MSI EQ must be created before CQ so CQ can reference its eqn */\n+\tmlx5st_create_msi_eq(dev);\n+\tmlx5st_setup_datapath(dev);\n+\n+\tvfio_pci_msix_enable(device, MSI_VECTOR, 1);\n+\tdevice-\u003edriver.msi = MSI_VECTOR;\n+\n+\tdevice-\u003edriver.max_memcpy_size = 1ULL \u003c\u003c dev-\u003elog_max_msg;\n+\tdevice-\u003edriver.max_memcpy_count = SQ_WQE_CNT - 1;\n+\n+\tdev_dbg(device, \"mlx5 driver initialized\\n\");\n+}\n+\n+static void mlx5st_remove(struct vfio_pci_device *device)\n+{\n+\tstruct mlx5st_device *dev = to_mlx5st(device);\n+\n+\tvfio_pci_msix_disable(device);\n+\tmlx5st_teardown_datapath(dev);\n+\n+\tif (dev-\u003ehave_msi_eq) {\n+\t\tmlx5st_destroy_msi_eq(dev);\n+\t\tdev-\u003ehave_msi_eq = false;\n+\t}\n+\n+\tdev_dbg(device, \"teardown: destroy_mkey\\n\");\n+\tif (dev-\u003emkey_index) {\n+\t\tmlx5st_destroy_mkey(dev);\n+\t\tdev-\u003emkey_index = 0;\n+\t}\n+\n+\tdev_dbg(device, \"teardown: dealloc_pd\\n\");\n+\tif (dev-\u003epdn) {\n+\t\tmlx5st_dealloc_pd(dev);\n+\t\tdev-\u003epdn = 0;\n+\t}\n+\n+\tdev_dbg(device, \"teardown: hca_teardown\\n\");\n+\tmlx5st_hca_teardown(dev);\n+\n+\tvfio_pci_config_writew(device, PCI_COMMAND,\n+\t\t\t vfio_pci_config_readw(device, PCI_COMMAND) \u0026\n+\t\t\t\t ~(u16)PCI_COMMAND_MASTER);\n+\n+\tdev_dbg(device, \"Teardown complete\\n\");\n+}\n+\n+struct vfio_pci_driver_ops mlx5st_ops = {\n+\t.name = \"mlx5\",\n+\t.region_size = sizeof(struct mlx5st_device),\n+\t.probe = mlx5st_probe,\n+\t.init = mlx5st_init,\n+\t.remove = mlx5st_remove,\n+\t.memcpy_start = mlx5st_memcpy_start,\n+\t.memcpy_wait = mlx5st_memcpy_wait,\n+\t.send_msi = mlx5st_send_msi,\n+};\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h\nnew file mode 100644\nindex 0000000000000..2c451e411ec13\n--- /dev/null\n+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h\n@@ -0,0 +1,114 @@\n+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */\n+/*\n+ * mlx5 VFIO selftest driver - HW definitions\n+ *\n+ * Typed wrappers, constants, and helpers for programming mlx5 hardware\n+ * via the VFIO selftest framework. Most HW constants and all MLX5_SET/GET\n+ * macros come from the kernel headers (mlx5_ifc.h, mlx5_ifc_macros.h).\n+ */\n+#ifndef SELFTESTS_VFIO_MLX5_HW_H\n+#define SELFTESTS_VFIO_MLX5_HW_H\n+\n+#include \u003clinux/io.h\u003e\n+#include \u003clinux/build_bug.h\u003e\n+#include \u003cvdso/bits.h\u003e\n+\n+#include \"mlx5_ifc.h\"\n+#include \"mlx5_ifc_macros.h\"\n+\n+/*\n+ * Typed HW object wrappers for driver region arrays.\n+ *\n+ * The IFC _bits structs have sizeof == num_bits (not bytes), so they cannot\n+ * be used as array elements. These wrappers provide byte-sized types.\n+ */\n+#define MLX5ST_MAKE_DATA32(name) \\\n+\tstruct mlx5st_##name { \\\n+\t\tu32 data[MLX5_ST_SZ_DW(name)]; \\\n+\t}\n+#define MLX5ST_MAKE_DATA64(name) \\\n+\tstruct mlx5st_##name { \\\n+\t\tu64 data[MLX5_ST_SZ_QW(name)]; \\\n+\t}\n+\n+MLX5ST_MAKE_DATA32(initial_seg);\n+MLX5ST_MAKE_DATA64(cmd_queue_entry);\n+MLX5ST_MAKE_DATA64(cmd_if_box);\n+MLX5ST_MAKE_DATA64(wqe_ctrl_seg);\n+MLX5ST_MAKE_DATA64(wqe_raddr_seg);\n+MLX5ST_MAKE_DATA64(wqe_data_seg);\n+MLX5ST_MAKE_DATA64(cqe64) __aligned(64);\n+MLX5ST_MAKE_DATA64(eqe);\n+\n+/*\n+ * Mailbox blocks: 512 data + 64 header = 576 bytes, but the\n+ * next_pointer field stores bits [31:10], requiring 1024-byte alignment.\n+ */\n+#define CMD_MBOX_SIZE (2 * MLX5_HW_PAGE_SIZE)\n+#define CMD_MBOX_STRIDE 1024\n+#define CMD_MBOX_NENT (CMD_MBOX_SIZE / CMD_MBOX_STRIDE)\n+/* Stride-aligned mailbox entry — block + padding to 1024 bytes */\n+struct mlx5st_mbox_entry {\n+\tstruct mlx5st_cmd_if_box block;\n+} __aligned(CMD_MBOX_STRIDE);\n+\n+#define MLX5_CMD_INLINE_SZ \\\n+\tMLX5_FLD_SZ_BYTES(cmd_queue_entry, command_input_inline_data)\n+\n+/* Command interface mailbox block (512 data + 64 header) */\n+#define MLX5_CMD_DATA_BLOCK_SIZE MLX5_FLD_SZ_BYTES(cmd_if_box, mailbox_data)\n+\n+/* RDMA Write WQE — one basic block: ctrl + raddr + data + padding */\n+struct mlx5st_send_wqe {\n+\tstruct mlx5st_wqe_ctrl_seg ctrl;\n+\tstruct mlx5st_wqe_raddr_seg raddr;\n+\tstruct mlx5st_wqe_data_seg data;\n+} __aligned(64);\n+static_assert(sizeof(struct mlx5st_send_wqe) == 64,\n+\t \"send WQE segments must fit in one BB\");\n+\n+/* DS = number of 16-byte segments in the WQE (ctrl + raddr + data) */\n+#define MLX5_RDMA_WRITE_DS 3\n+\n+/* Doorbell record — two __be32 in a 64-byte aligned pair */\n+struct mlx5st_dbrec {\n+\t__be32 recv_counter;\n+\t__be32 send_counter;\n+} __aligned(64);\n+\n+/* UAR BlueFlame buffer offsets within a UAR page */\n+#define MLX5_BF_OFFSET 0x800\n+#define MLX5_BF_SIZE 0x100\n+\n+/* CQ doorbell offset within UAR page */\n+#define MLX5_CQ_DOORBELL_OFFSET 0x20\n+\n+/* EQ doorbell offset within UAR page */\n+#define MLX5_EQ_DOORBELL_OFFSET 0x40\n+\n+#define MLX5_HW_PAGE_SIZE 4096\n+\n+/*\n+ * Test parameters\n+ */\n+#define SQ_WQE_CNT 16\n+#define LOG_SQ_SIZE 4\n+#define CQ_CQE_CNT 16\n+#define LOG_CQ_SIZE 4\n+#define EQ_NENT 64\n+#define LOG_EQ_SIZE 6\n+#define MSI_EQ_NENT 16\n+#define LOG_MSI_EQ_SIZE 4\n+#define MSI_VECTOR 0\n+\n+#define MAX_FW_PAGES 8192\n+#define MAX_FW_PAGES_PER_CMD 512\n+\n+#define MLX5_CMD_TIMEOUT_MS 5000\n+\n+static inline u32 mlx5st_idx_to_mkey(u32 mkey_idx)\n+{\n+\treturn mkey_idx \u003c\u003c 8;\n+}\n+\n+#endif /* SELFTESTS_VFIO_MLX5_HW_H */\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h\nnew file mode 120000\nindex 0000000000000..7dcbb79e1af06\n--- /dev/null\n+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h\n@@ -0,0 +1 @@\n+../../../../../../../include/linux/mlx5/mlx5_ifc.h\n\\ No newline at end of file\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h\nnew file mode 120000\nindex 0000000000000..865d99e2aeecd\n--- /dev/null\n+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h\n@@ -0,0 +1 @@\n+../../../../../../../include/linux/mlx5/mlx5_ifc_fpga.h\n\\ No newline at end of file\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h\nnew file mode 120000\nindex 0000000000000..97408c247f06c\n--- /dev/null\n+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h\n@@ -0,0 +1 @@\n+../../../../../../../include/linux/mlx5/mlx5_ifc_macros.h\n\\ No newline at end of file\ndiff --git a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c\nindex c08aa81c44f41..6958877ad4638 100644\n--- a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c\n+++ b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c\n@@ -775,6 +775,7 @@ static int nv_falcon_memcpy_wait(struct vfio_pci_device *device)\n \n const struct vfio_pci_driver_ops nv_falcon_ops = {\n \t.name = \"nv_falcon\",\n+\t.region_size = sizeof(struct gpu_device),\n \t.probe = nv_falcon_probe,\n \t.init = nv_falcon_init,\n \t.remove = nv_falcon_remove,\ndiff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h\nindex e19bd94b8dd2a..007adc322c1ea 100644\n--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h\n+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h\n@@ -40,6 +40,17 @@ struct vfio_pci_device {\n \n struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu);\n void vfio_pci_device_free(struct vfio_pci_device *device);\n+\n+#ifdef DEBUG\n+#define dev_dbg dev_info\n+#else\n+#define dev_dbg(_dev, _fmt, ...) \\\n+\tdo { \\\n+\t\tif (0) \\\n+\t\t\tdev_info(_dev, _fmt, ##__VA_ARGS__); \\\n+\t} while (0)\n+#endif\n+\n struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu);\n void vfio_pci_device_cleanup(struct vfio_pci_device *device);\n \ndiff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h\nindex e5ada209b1d10..547369c5cff95 100644\n--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h\n+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h\n@@ -9,6 +9,12 @@ struct vfio_pci_device;\n struct vfio_pci_driver_ops {\n \tconst char *name;\n \n+\t/*\n+\t * Size of the driver's state structure overlaid on\n+\t * device-\u003edriver.region.vaddr\n+\t */\n+\tu64 region_size;\n+\n \t/**\n \t * @probe() - Check if the driver supports the given device.\n \t *\ndiff --git a/tools/testing/selftests/vfio/lib/iova_allocator.c b/tools/testing/selftests/vfio/lib/iova_allocator.c\nindex 4a660f636f497..a8e5815a1b165 100644\n--- a/tools/testing/selftests/vfio/lib/iova_allocator.c\n+++ b/tools/testing/selftests/vfio/lib/iova_allocator.c\n@@ -13,8 +13,10 @@\n \n #include \u003clinux/iommufd.h\u003e\n #include \u003clinux/limits.h\u003e\n+#include \u003clinux/log2.h\u003e\n #include \u003clinux/mman.h\u003e\n #include \u003clinux/overflow.h\u003e\n+#include \u003clinux/sizes.h\u003e\n #include \u003clinux/types.h\u003e\n #include \u003clinux/vfio.h\u003e\n \n@@ -50,7 +52,10 @@ void iova_allocator_cleanup(struct iova_allocator *allocator)\n iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)\n {\n \tVFIO_ASSERT_GT(size, 0, \"Invalid size arg, zero\\n\");\n-\tVFIO_ASSERT_EQ(size \u0026 (size - 1), 0, \"Invalid size arg, non-power-of-2\\n\");\n+\n+\tVFIO_ASSERT_LE(size, rounddown_pow_of_two(SIZE_MAX),\n+\t\t \"Invalid size arg, too large (%zu)\\n\", size);\n+\tsize = roundup_pow_of_two(size);\n \n \tfor (;;) {\n \t\tstruct iommu_iova_range *range;\ndiff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk\nindex bcfa74ae040e7..eff95828719e0 100644\n--- a/tools/testing/selftests/vfio/lib/libvfio.mk\n+++ b/tools/testing/selftests/vfio/lib/libvfio.mk\n@@ -15,6 +15,7 @@ LIBVFIO_C += drivers/ioat/ioat.c\n LIBVFIO_C += drivers/dsa/dsa.c\n endif\n \n+LIBVFIO_C += drivers/mlx5/mlx5.c\n LIBVFIO_C += drivers/nv_falcon/nv_falcon.c\n LIBVFIO_C += drivers/igb/igb.c\n \ndiff --git a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c\nindex 5e65434d2318b..b31abfe2c7ffc 100644\n--- a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c\n+++ b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c\n@@ -6,6 +6,7 @@\n extern struct vfio_pci_driver_ops dsa_ops;\n extern struct vfio_pci_driver_ops ioat_ops;\n #endif\n+extern struct vfio_pci_driver_ops mlx5st_ops;\n extern struct vfio_pci_driver_ops nv_falcon_ops;\n extern struct vfio_pci_driver_ops igb_ops;\n \n@@ -14,6 +15,7 @@ static struct vfio_pci_driver_ops *driver_ops[] = {\n \t\u0026dsa_ops,\n \t\u0026ioat_ops,\n #endif\n+\t\u0026mlx5st_ops,\n \t\u0026nv_falcon_ops,\n \t\u0026igb_ops,\n };\n@@ -32,6 +34,10 @@ void vfio_pci_driver_probe(struct vfio_pci_device *device)\n \t\t\tcontinue;\n \n \t\tdevice-\u003edriver.ops = ops;\n+\n+\t\tVFIO_ASSERT_NE(ops-\u003eregion_size, 0);\n+\t\tdevice-\u003edriver.region.size =\n+\t\t\tround_up(ops-\u003eregion_size, getpagesize());\n \t}\n }\n \ndiff --git a/tools/testing/selftests/vfio/vfio_pci_driver_test.c b/tools/testing/selftests/vfio/vfio_pci_driver_test.c\nindex 761bf117d624f..2bd949aafd731 100644\n--- a/tools/testing/selftests/vfio/vfio_pci_driver_test.c\n+++ b/tools/testing/selftests/vfio/vfio_pci_driver_test.c\n@@ -87,7 +87,8 @@ FIXTURE_SETUP(vfio_pci_driver_test)\n \tdriver = \u0026self-\u003edevice-\u003edriver;\n \n \tregion_setup(self-\u003eiommu, self-\u003eiova_allocator, \u0026self-\u003ememcpy_region, SZ_1G);\n-\tregion_setup(self-\u003eiommu, self-\u003eiova_allocator, \u0026driver-\u003eregion, SZ_2M);\n+\tregion_setup(self-\u003eiommu, self-\u003eiova_allocator, \u0026driver-\u003eregion,\n+\t\t driver-\u003eregion.size);\n \n \t/* Any IOVA that doesn't overlap memcpy_region and driver-\u003eregion. */\n \tself-\u003eunmapped_iova = iova_allocator_alloc(self-\u003eiova_allocator, SZ_1G);\n"
} |
| 2/1 | 2026/09/24 00:57 | agent | patch-evaluator |
0mModel:core Args: null Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"Reasoning": "The patch series primarily adds an mlx5 selftest driver under tools/testing/selftests/ and modifies userspace test headers under tools/. The only kernel header changes (under include/linux/mlx5/) refactor and move macros, structs, and enum definitions without modifying any executable kernel C code. Furthermore, mlx5 targets vendor-specific Mellanox NICs/HCAs requiring physical PCIe hardware not emulated in standard QEMU environments.",
"WorthFuzzing": false
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit f344d40ff4714d670cc8789c378c68c22d8901f6
Author: syz-cluster <triage@syzkaller.com>
Date: Thu Sep 24 00:57:40 2026 +0000
syz-cluster: applied patch under review
diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
index 9d47cdc727ad0..a1c14479e462c 100644
--- a/include/linux/mlx5/cq.h
+++ b/include/linux/mlx5/cq.h
@@ -81,16 +81,6 @@ enum {
enum {
MLX5_CQE_OWNER_MASK = 1,
- MLX5_CQE_REQ = 0,
- MLX5_CQE_RESP_WR_IMM = 1,
- MLX5_CQE_RESP_SEND = 2,
- MLX5_CQE_RESP_SEND_IMM = 3,
- MLX5_CQE_RESP_SEND_INV = 4,
- MLX5_CQE_RESIZE_CQ = 5,
- MLX5_CQE_SIG_ERR = 12,
- MLX5_CQE_REQ_ERR = 13,
- MLX5_CQE_RESP_ERR = 14,
- MLX5_CQE_INVALID = 15,
};
enum {
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 8cb321a9fb3d1..37727b5ab78f2 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -36,6 +36,7 @@
#include <linux/types.h>
#include <rdma/ib_verbs.h>
#include <linux/mlx5/mlx5_ifc.h>
+#include <linux/mlx5/mlx5_ifc_macros.h>
#include <linux/bitfield.h>
#if defined(__LITTLE_ENDIAN)
@@ -46,122 +47,6 @@
#error Host endianness not defined
#endif
-/* helper macros */
-#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0)
-#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
-#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
-#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16)
-#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)
-#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64)
-#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf))
-#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
-#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
-#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld))
-#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
-#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld))
-#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits)
-
-#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8)
-#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8)
-#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32)
-#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64)
-#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8)
-#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32)
-#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)
-#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld)))
-
-/* insert a value to a struct */
-#define MLX5_SET(typ, p, fld, v) do { \
- u32 _v = v; \
- BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \
- *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \
- cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \
- (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \
- << __mlx5_dw_bit_off(typ, fld))); \
-} while (0)
-
-#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \
- BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \
- MLX5_SET(typ, p, fld[idx], v); \
-} while (0)
-
-#define MLX5_SET_TO_ONES(typ, p, fld) do { \
- BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \
- *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \
- cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \
- (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \
- << __mlx5_dw_bit_off(typ, fld))); \
-} while (0)
-
-#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\
-__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
-__mlx5_mask(typ, fld))
-
-#define MLX5_GET_PR(typ, p, fld) ({ \
- u32 ___t = MLX5_GET(typ, p, fld); \
- pr_debug(#fld " = 0x%x\n", ___t); \
- ___t; \
-})
-
-#define __MLX5_SET64(typ, p, fld, v) do { \
- BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \
- *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \
-} while (0)
-
-#define MLX5_SET64(typ, p, fld, v) do { \
- BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \
- __MLX5_SET64(typ, p, fld, v); \
-} while (0)
-
-#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \
- BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \
- __MLX5_SET64(typ, p, fld[idx], v); \
-} while (0)
-
-#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld)))
-
-#define MLX5_GET64_PR(typ, p, fld) ({ \
- u64 ___t = MLX5_GET64(typ, p, fld); \
- pr_debug(#fld " = 0x%llx\n", ___t); \
- ___t; \
-})
-
-#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\
-__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \
-__mlx5_mask16(typ, fld))
-
-#define MLX5_SET16(typ, p, fld, v) do { \
- u16 _v = v; \
- BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \
- *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \
- cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \
- (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \
- << __mlx5_16_bit_off(typ, fld))); \
-} while (0)
-
-/* Big endian getters */
-#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\
- __mlx5_64_off(typ, fld)))
-
-#define MLX5_GET_BE(type_t, typ, p, fld) ({ \
- type_t tmp; \
- switch (sizeof(tmp)) { \
- case sizeof(u8): \
- tmp = (__force type_t)MLX5_GET(typ, p, fld); \
- break; \
- case sizeof(u16): \
- tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \
- break; \
- case sizeof(u32): \
- tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \
- break; \
- case sizeof(u64): \
- tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \
- break; \
- } \
- tmp; \
- })
-
enum mlx5_inline_modes {
MLX5_INLINE_MODE_NONE,
MLX5_INLINE_MODE_L2,
@@ -172,7 +57,7 @@ enum mlx5_inline_modes {
enum {
MLX5_MAX_COMMANDS = 32,
MLX5_CMD_DATA_BLOCK_SIZE = 512,
- MLX5_PCI_CMD_XPORT = 7,
+ MLX5_PCI_CMD_XPORT = MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT,
MLX5_MKEY_BSF_OCTO_SIZE = 4,
MLX5_MAX_PSVS = 4,
};
@@ -308,63 +193,6 @@ enum {
MLX5_EVENT_QUEUE_TYPE_DCT = 6,
};
-/* mlx5 components can subscribe to any one of these events via
- * mlx5_eq_notifier_register API.
- */
-enum mlx5_event {
- /* Special value to subscribe to any event */
- MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0,
- /* HW events enum start: comp events are not subscribable */
- MLX5_EVENT_TYPE_COMP = 0x0,
- /* HW Async events enum start: subscribable events */
- MLX5_EVENT_TYPE_PATH_MIG = 0x01,
- MLX5_EVENT_TYPE_COMM_EST = 0x02,
- MLX5_EVENT_TYPE_SQ_DRAINED = 0x03,
- MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13,
- MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14,
-
- MLX5_EVENT_TYPE_CQ_ERROR = 0x04,
- MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05,
- MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07,
- MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10,
- MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11,
- MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12,
- MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27,
-
- MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08,
- MLX5_EVENT_TYPE_PORT_CHANGE = 0x09,
- MLX5_EVENT_TYPE_GPIO_EVENT = 0x15,
- MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16,
- MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17,
- MLX5_EVENT_TYPE_XRQ_ERROR = 0x18,
- MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19,
- MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22,
- MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24,
- MLX5_EVENT_TYPE_PPS_EVENT = 0x25,
-
- MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a,
- MLX5_EVENT_TYPE_STALL_EVENT = 0x1b,
-
- MLX5_EVENT_TYPE_CMD = 0x0a,
- MLX5_EVENT_TYPE_PAGE_REQUEST = 0xb,
-
- MLX5_EVENT_TYPE_PAGE_FAULT = 0xc,
- MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0xd,
-
- MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0xe,
- MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0xf,
-
- MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c,
- MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d,
-
- MLX5_EVENT_TYPE_FPGA_ERROR = 0x20,
- MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21,
-
- MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26,
-
- MLX5_EVENT_TYPE_MAX = 0x100,
-};
-
enum mlx5_driver_event {
MLX5_DRIVER_EVENT_TYPE_TRAP = 0,
MLX5_DRIVER_EVENT_UPLINK_NETDEV,
@@ -420,22 +248,6 @@ enum {
};
enum {
- MLX5_OPCODE_NOP = 0x00,
- MLX5_OPCODE_SEND_INVAL = 0x01,
- MLX5_OPCODE_RDMA_WRITE = 0x08,
- MLX5_OPCODE_RDMA_WRITE_IMM = 0x09,
- MLX5_OPCODE_SEND = 0x0a,
- MLX5_OPCODE_SEND_IMM = 0x0b,
- MLX5_OPCODE_LSO = 0x0e,
- MLX5_OPCODE_RDMA_READ = 0x10,
- MLX5_OPCODE_ATOMIC_CS = 0x11,
- MLX5_OPCODE_ATOMIC_FA = 0x12,
- MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14,
- MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15,
- MLX5_OPCODE_BIND_MW = 0x18,
- MLX5_OPCODE_CONFIG_CMD = 0x1f,
- MLX5_OPCODE_ENHANCED_MPSW = 0x29,
-
MLX5_RECV_OPCODE_RDMA_WRITE_IMM = 0x00,
MLX5_RECV_OPCODE_SEND = 0x01,
MLX5_RECV_OPCODE_SEND_IMM = 0x02,
@@ -443,19 +255,6 @@ enum {
MLX5_CQE_OPCODE_ERROR = 0x1e,
MLX5_CQE_OPCODE_RESIZE = 0x16,
-
- MLX5_OPCODE_SET_PSV = 0x20,
- MLX5_OPCODE_GET_PSV = 0x21,
- MLX5_OPCODE_CHECK_PSV = 0x22,
- MLX5_OPCODE_DUMP = 0x23,
- MLX5_OPCODE_RGET_PSV = 0x26,
- MLX5_OPCODE_RCHECK_PSV = 0x27,
-
- MLX5_OPCODE_UMR = 0x25,
-
- MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c,
-
- MLX5_OPCODE_ACCESS_ASO = 0x2d,
};
enum {
@@ -1224,12 +1023,6 @@ enum mlx5_flex_parser_protos {
/* MLX5 DEV CAPs */
-/* TODO: EAT.ME */
-enum mlx5_cap_mode {
- HCA_CAP_OPMOD_GET_MAX = 0,
- HCA_CAP_OPMOD_GET_CUR = 1,
-};
-
/* Any new cap addition must update mlx5_hca_caps_alloc() to allocate
* capability memory.
*/
@@ -1507,26 +1300,6 @@ enum mlx5_qcam_feature_groups {
#define MLX5_CAP_PSP(mdev, cap)\
MLX5_GET(psp_cap, (mdev)->caps.hca[MLX5_CAP_PSP]->cur, cap)
-enum {
- MLX5_CMD_STAT_OK = 0x0,
- MLX5_CMD_STAT_INT_ERR = 0x1,
- MLX5_CMD_STAT_BAD_OP_ERR = 0x2,
- MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3,
- MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4,
- MLX5_CMD_STAT_BAD_RES_ERR = 0x5,
- MLX5_CMD_STAT_RES_BUSY = 0x6,
- MLX5_CMD_STAT_NOT_READY = 0x7,
- MLX5_CMD_STAT_LIM_ERR = 0x8,
- MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9,
- MLX5_CMD_STAT_IX_ERR = 0xa,
- MLX5_CMD_STAT_NO_RES_ERR = 0xf,
- MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50,
- MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51,
- MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10,
- MLX5_CMD_STAT_BAD_PKT_ERR = 0x30,
- MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40,
-};
-
enum {
MLX5_IEEE_802_3_COUNTERS_GROUP = 0x0,
MLX5_RFC_2863_COUNTERS_GROUP = 0x1,
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 12b9a81e56214..8247cb0f8dfd6 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -5979,6 +5979,108 @@ struct mlx5_ifc_cqe_error_syndrome_bits {
u8 syndrome[0x8];
};
+struct mlx5_ifc_wqe_ctrl_seg_bits {
+ u8 opmod[0x8];
+ u8 wqe_index[0x10];
+ u8 opcode[0x8];
+
+ u8 qp_or_sq[0x18];
+ u8 reserved_at_38[0x2];
+ u8 ds[0x6];
+
+ u8 signature[0x8];
+ u8 reserved_at_48[0x10];
+ u8 fm[0x3];
+ u8 reserved_at_5b[0x1];
+ u8 ce[0x2];
+ u8 se[0x1];
+ u8 reserved_at_5f[0x1];
+
+ u8 imm[0x20];
+};
+
+/* Values for wqe_ctrl_seg_bits.opcode */
+enum {
+ MLX5_OPCODE_NOP = 0x00,
+ MLX5_OPCODE_SEND_INVAL = 0x01,
+ MLX5_OPCODE_RDMA_WRITE = 0x08,
+ MLX5_OPCODE_RDMA_WRITE_IMM = 0x09,
+ MLX5_OPCODE_SEND = 0x0a,
+ MLX5_OPCODE_SEND_IMM = 0x0b,
+ MLX5_OPCODE_LSO = 0x0e,
+ MLX5_OPCODE_RDMA_READ = 0x10,
+ MLX5_OPCODE_ATOMIC_CS = 0x11,
+ MLX5_OPCODE_ATOMIC_FA = 0x12,
+ MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14,
+ MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15,
+ MLX5_OPCODE_BIND_MW = 0x18,
+ MLX5_OPCODE_CONFIG_CMD = 0x1f,
+ MLX5_OPCODE_SET_PSV = 0x20,
+ MLX5_OPCODE_GET_PSV = 0x21,
+ MLX5_OPCODE_CHECK_PSV = 0x22,
+ MLX5_OPCODE_DUMP = 0x23,
+ MLX5_OPCODE_UMR = 0x25,
+ MLX5_OPCODE_RGET_PSV = 0x26,
+ MLX5_OPCODE_RCHECK_PSV = 0x27,
+ MLX5_OPCODE_ENHANCED_MPSW = 0x29,
+ MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c,
+ MLX5_OPCODE_ACCESS_ASO = 0x2d,
+};
+
+/* Values for wqe_ctrl_seg_bits.ce */
+enum {
+ MLX5_WQE_CE_CQE_ALWAYS = 2,
+};
+
+struct mlx5_ifc_wqe_raddr_seg_bits {
+ u8 raddr[0x40];
+
+ u8 rkey[0x20];
+ u8 reserved_at_60[0x20];
+};
+
+struct mlx5_ifc_wqe_data_seg_bits {
+ u8 reserved_at_0[0x1];
+ u8 byte_count[0x1f];
+
+ u8 lkey[0x20];
+
+ u8 addr[0x40];
+};
+
+struct mlx5_ifc_cqe64_bits {
+ u8 reserved_at_0[0x1a0];
+
+ union {
+ u8 reserved_at_1a0[0x20];
+ struct mlx5_ifc_cqe_error_syndrome_bits error_syndrome;
+ };
+
+ u8 send_wqe_opcode[0x8];
+ u8 qpn_or_dctn_or_flow_tag[0x18];
+
+ u8 wqe_counter[0x10];
+ u8 signature[0x8];
+ u8 opcode[0x4];
+ u8 cqe_format[0x2];
+ u8 se[0x1];
+ u8 owner[0x1];
+};
+
+/* Values for cqe64_bits.opcode */
+enum {
+ MLX5_CQE_REQ = 0,
+ MLX5_CQE_RESP_WR_IMM = 1,
+ MLX5_CQE_RESP_SEND = 2,
+ MLX5_CQE_RESP_SEND_IMM = 3,
+ MLX5_CQE_RESP_SEND_INV = 4,
+ MLX5_CQE_RESIZE_CQ = 5,
+ MLX5_CQE_SIG_ERR = 12,
+ MLX5_CQE_REQ_ERR = 13,
+ MLX5_CQE_RESP_ERR = 14,
+ MLX5_CQE_INVALID = 15,
+};
+
struct mlx5_ifc_qp_context_extension_bits {
u8 reserved_at_0[0x60];
@@ -6475,6 +6577,12 @@ struct mlx5_ifc_query_hca_cap_in_bits {
u8 reserved_at_60[0x20];
};
+/* Values for query_hca_cap_in_bits.op_mod */
+enum mlx5_cap_mode {
+ HCA_CAP_OPMOD_GET_MAX = 0,
+ HCA_CAP_OPMOD_GET_CUR = 1,
+};
+
struct mlx5_ifc_other_hca_cap_bits {
u8 roce[0x1];
u8 reserved_at_1[0x27f];
@@ -11270,6 +11378,55 @@ struct mlx5_ifc_eqe_bits {
u8 owner[0x1];
};
+/* Values for eqe_bits.event_type */
+enum mlx5_event {
+ /* Special value to subscribe to any event */
+ MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0,
+ /* HW events enum start: comp events are not subscribable */
+ MLX5_EVENT_TYPE_COMP = 0x0,
+ /* HW Async events enum start: subscribable events */
+ MLX5_EVENT_TYPE_PATH_MIG = 0x01,
+ MLX5_EVENT_TYPE_COMM_EST = 0x02,
+ MLX5_EVENT_TYPE_SQ_DRAINED = 0x03,
+ MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13,
+ MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14,
+
+ MLX5_EVENT_TYPE_CQ_ERROR = 0x04,
+ MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05,
+ MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07,
+ MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10,
+ MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11,
+ MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12,
+ MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27,
+
+ MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08,
+ MLX5_EVENT_TYPE_PORT_CHANGE = 0x09,
+ MLX5_EVENT_TYPE_CMD = 0x0a,
+ MLX5_EVENT_TYPE_PAGE_REQUEST = 0x0b,
+ MLX5_EVENT_TYPE_PAGE_FAULT = 0x0c,
+ MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0x0d,
+ MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0x0e,
+ MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0x0f,
+ MLX5_EVENT_TYPE_GPIO_EVENT = 0x15,
+ MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16,
+ MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17,
+ MLX5_EVENT_TYPE_XRQ_ERROR = 0x18,
+ MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19,
+ MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a,
+ MLX5_EVENT_TYPE_STALL_EVENT = 0x1b,
+ MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c,
+ MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d,
+ MLX5_EVENT_TYPE_FPGA_ERROR = 0x20,
+ MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21,
+ MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22,
+ MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24,
+ MLX5_EVENT_TYPE_PPS_EVENT = 0x25,
+ MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26,
+
+ MLX5_EVENT_TYPE_MAX = 0x100,
+};
+
+/* Values for cmd_queue_entry_bits.type */
enum {
MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT = 0x7,
};
@@ -11312,6 +11469,27 @@ struct mlx5_ifc_cmd_out_bits {
u8 command_output[0x20];
};
+/* Values for cmd_out_bits.status */
+enum {
+ MLX5_CMD_STAT_OK = 0x0,
+ MLX5_CMD_STAT_INT_ERR = 0x1,
+ MLX5_CMD_STAT_BAD_OP_ERR = 0x2,
+ MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3,
+ MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4,
+ MLX5_CMD_STAT_BAD_RES_ERR = 0x5,
+ MLX5_CMD_STAT_RES_BUSY = 0x6,
+ MLX5_CMD_STAT_NOT_READY = 0x7,
+ MLX5_CMD_STAT_LIM_ERR = 0x8,
+ MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9,
+ MLX5_CMD_STAT_IX_ERR = 0xa,
+ MLX5_CMD_STAT_NO_RES_ERR = 0xf,
+ MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10,
+ MLX5_CMD_STAT_BAD_PKT_ERR = 0x30,
+ MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40,
+ MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50,
+ MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51,
+};
+
struct mlx5_ifc_cmd_in_bits {
u8 opcode[0x10];
u8 reserved_at_10[0x10];
diff --git a/include/linux/mlx5/mlx5_ifc_macros.h b/include/linux/mlx5/mlx5_ifc_macros.h
new file mode 100644
index 0000000000000..1b9e39a509495
--- /dev/null
+++ b/include/linux/mlx5/mlx5_ifc_macros.h
@@ -0,0 +1,185 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright (c) 2013-2026, Mellanox Technologies. All rights reserved.
+ *
+ * Accessor macros for mlx5 IFC structures.
+ *
+ * Extracted from device.h so that code which cannot include device.h
+ * (e.g. selftests) can still use the MLX5_SET/GET family directly.
+ */
+
+#ifndef MLX5_IFC_MACROS_H
+#define MLX5_IFC_MACROS_H
+
+/* Internal helpers -- 32-bit */
+#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0)
+#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
+#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
+#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16)
+#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)
+#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64)
+#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf))
+#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
+#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
+#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld))
+#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
+#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld))
+#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits)
+
+/* Size and address macros */
+#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8)
+#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8)
+#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32)
+#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64)
+#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8)
+#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32)
+#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8)
+#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld)))
+
+/* insert a value to a struct */
+#define MLX5_SET(typ, p, fld, v) do { \
+ u32 _v = v; \
+ BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \
+ *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \
+ cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \
+ (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \
+ << __mlx5_dw_bit_off(typ, fld))); \
+} while (0)
+
+#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \
+ BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \
+ MLX5_SET(typ, p, fld[idx], v); \
+} while (0)
+
+#define MLX5_SET_TO_ONES(typ, p, fld) do { \
+ BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \
+ *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \
+ cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \
+ (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \
+ << __mlx5_dw_bit_off(typ, fld))); \
+} while (0)
+
+#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\
+__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
+__mlx5_mask(typ, fld))
+
+#define MLX5_GET_PR(typ, p, fld) ({ \
+ u32 ___t = MLX5_GET(typ, p, fld); \
+ pr_debug(#fld " = 0x%x\n", ___t); \
+ ___t; \
+})
+
+/* 64-bit field accessors */
+#define __MLX5_SET64(typ, p, fld, v) do { \
+ BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \
+ *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \
+} while (0)
+
+#define MLX5_SET64(typ, p, fld, v) do { \
+ BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \
+ __MLX5_SET64(typ, p, fld, v); \
+} while (0)
+
+#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \
+ BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \
+ __MLX5_SET64(typ, p, fld[idx], v); \
+} while (0)
+
+#define MLX5_ARRAY_GET64(typ, p, fld, idx) \
+ ({ \
+ BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \
+ be64_to_cpu( \
+ *((__be64 *)(p) + __mlx5_64_off(typ, fld) + (idx))); \
+ })
+
+#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld)))
+
+#define MLX5_GET64_PR(typ, p, fld) ({ \
+ u64 ___t = MLX5_GET64(typ, p, fld); \
+ pr_debug(#fld " = 0x%llx\n", ___t); \
+ ___t; \
+})
+
+/* 16-bit field accessors */
+#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\
+__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \
+__mlx5_mask16(typ, fld))
+
+#define MLX5_SET16(typ, p, fld, v) do { \
+ u16 _v = v; \
+ BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \
+ *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \
+ cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \
+ (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \
+ << __mlx5_16_bit_off(typ, fld))); \
+} while (0)
+
+/* Big endian getters */
+#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\
+ __mlx5_64_off(typ, fld)))
+
+#define MLX5_GET_BE(type_t, typ, p, fld) ({ \
+ type_t tmp; \
+ switch (sizeof(tmp)) { \
+ case sizeof(u8): \
+ tmp = (__force type_t)MLX5_GET(typ, p, fld); \
+ break; \
+ case sizeof(u16): \
+ tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \
+ break; \
+ case sizeof(u32): \
+ tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \
+ break; \
+ case sizeof(u64): \
+ tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \
+ break; \
+ } \
+ tmp; \
+ })
+
+/*
+ * Use READ_ONCE/WRITE_ONCE for a single field that hardware may read/write
+ * unpredictably, mostly owner bits. All other bits in the DW must be stable.
+ * Usually a dma_wmb() will be required before a write and a dma_rmb() after a
+ * read.
+ */
+#define MLX5_GET_ONCE(typ, p, fld) \
+ ((be32_to_cpu(READ_ONCE(*((__be32 *)(p) + __mlx5_dw_off(typ, fld)))) >> \
+ __mlx5_dw_bit_off(typ, fld)) & \
+ __mlx5_mask(typ, fld))
+
+#define MLX5_SET_ONCE(typ, p, fld, v) \
+ do { \
+ u32 _v = v; \
+ __be32 *_dw = (__be32 *)(p) + __mlx5_dw_off(typ, fld); \
+ BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \
+ WRITE_ONCE(*_dw, \
+ cpu_to_be32((be32_to_cpu(READ_ONCE(*_dw)) & \
+ (~__mlx5_dw_mask(typ, fld))) | \
+ (((_v) & __mlx5_mask(typ, fld)) \
+ << __mlx5_dw_bit_off(typ, fld)))); \
+ } while (0)
+
+/* Access MMIO registers, usually the init segment, using IFC structs. */
+#define MLX5_GET_MMIO(typ, p, fld) \
+ ((ioread32be(((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld))) >> \
+ __mlx5_dw_bit_off(typ, fld)) & \
+ __mlx5_mask(typ, fld))
+
+/* The set is not relaxed so there is an integrated dma_wmb(). */
+#define MLX5_SET_MMIO(typ, p, fld, v) \
+ do { \
+ u32 _v = v; \
+ void __iomem *_dw = \
+ ((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld)); \
+ if (__mlx5_bit_sz(typ, fld) == 32) \
+ iowrite32be(_v, _dw); \
+ else \
+ iowrite32be((ioread32be(_dw) & \
+ (~__mlx5_dw_mask(typ, fld))) | \
+ ((_v & __mlx5_mask(typ, fld)) \
+ << __mlx5_dw_bit_off(typ, fld)), \
+ _dw); \
+ } while (0)
+
+#endif /* MLX5_IFC_MACROS_H */
diff --git a/tools/arch/arm64/include/asm/barrier.h b/tools/arch/arm64/include/asm/barrier.h
index 3b9b41331c4f1..abdc64fc3c70f 100644
--- a/tools/arch/arm64/include/asm/barrier.h
+++ b/tools/arch/arm64/include/asm/barrier.h
@@ -24,6 +24,10 @@
#define smp_wmb() asm volatile("dmb ishst" ::: "memory")
#define smp_rmb() asm volatile("dmb ishld" ::: "memory")
+/* DMA barriers use outer-shareable (osh) for device visibility */
+#define dma_rmb() asm volatile("dmb oshld" ::: "memory")
+#define dma_wmb() asm volatile("dmb oshst" ::: "memory")
+
#define smp_store_release(p, v) \
do { \
union { typeof(*p) __val; char __c[1]; } __u = \
diff --git a/tools/arch/x86/include/asm/barrier.h b/tools/arch/x86/include/asm/barrier.h
index 0adf295dd5b6a..0b51431fa530e 100644
--- a/tools/arch/x86/include/asm/barrier.h
+++ b/tools/arch/x86/include/asm/barrier.h
@@ -43,4 +43,9 @@ do { \
___p1; \
})
#endif /* defined(__x86_64__) */
+
+/* x86 is DMA-coherent so DMA barriers are just compiler barriers */
+#define dma_rmb() barrier()
+#define dma_wmb() barrier()
+
#endif /* _TOOLS_LINUX_ASM_X86_BARRIER_H */
diff --git a/tools/include/asm-generic/io.h b/tools/include/asm-generic/io.h
index e5a0b07ad452a..0d89decdafb81 100644
--- a/tools/include/asm-generic/io.h
+++ b/tools/include/asm-generic/io.h
@@ -479,4 +479,32 @@ static inline void writesq(volatile void __iomem *addr, const void *buffer,
}
#endif
+/*
+ * ioread/iowrite for big-endian MMIO registers.
+ */
+
+#ifndef ioread32be
+#define ioread32be ioread32be
+static inline u32 ioread32be(const volatile void __iomem *addr)
+{
+ return bswap_32(readl(addr));
+}
+#endif
+
+#ifndef iowrite32be
+#define iowrite32be iowrite32be
+static inline void iowrite32be(u32 value, volatile void __iomem *addr)
+{
+ writel(bswap_32(value), addr);
+}
+#endif
+
+#ifndef iowrite64be
+#define iowrite64be iowrite64be
+static inline void iowrite64be(u64 value, volatile void __iomem *addr)
+{
+ writeq(bswap_64(value), addr);
+}
+#endif
+
#endif /* _TOOLS_ASM_GENERIC_IO_H */
diff --git a/tools/include/asm/barrier.h b/tools/include/asm/barrier.h
index 0c21678ac5e65..e7e0c7de5a2ff 100644
--- a/tools/include/asm/barrier.h
+++ b/tools/include/asm/barrier.h
@@ -47,6 +47,14 @@
# define smp_mb() mb()
#endif
+#ifndef dma_rmb
+# define dma_rmb() rmb()
+#endif
+
+#ifndef dma_wmb
+# define dma_wmb() wmb()
+#endif
+
#ifndef smp_store_release
# define smp_store_release(p, v) \
do { \
diff --git a/tools/include/linux/stddef.h b/tools/include/linux/stddef.h
new file mode 100644
index 0000000000000..99182ea4a1419
--- /dev/null
+++ b/tools/include/linux/stddef.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _TOOLS_LINUX_STDDEF_H
+#define _TOOLS_LINUX_STDDEF_H
+
+#include_next <linux/stddef.h>
+
+#define DECLARE_FLEX_ARRAY(TYPE, NAME) \
+ __DECLARE_FLEX_ARRAY(TYPE, NAME)
+
+#endif /* _TOOLS_LINUX_STDDEF_H */
diff --git a/tools/testing/selftests/kvm/include/arm64/processor.h b/tools/testing/selftests/kvm/include/arm64/processor.h
index b8a902ba8573d..5878a6abdf20e 100644
--- a/tools/testing/selftests/kvm/include/arm64/processor.h
+++ b/tools/testing/selftests/kvm/include/arm64/processor.h
@@ -12,6 +12,7 @@
#include <linux/stringify.h>
#include <linux/types.h>
+#include <asm/barrier.h>
#include <asm/brk-imm.h>
#include <asm/esr.h>
#include <asm/sysreg.h>
@@ -191,11 +192,8 @@ static inline void cpu_relax(void)
#define dsb(opt) asm volatile("dsb " #opt : : : "memory")
#define dmb(opt) asm volatile("dmb " #opt : : : "memory")
-#define dma_wmb() dmb(oshst)
#define __iowmb() dma_wmb()
-#define dma_rmb() dmb(oshld)
-
#define __iormb(v) \
({ \
unsigned long tmp; \
diff --git a/tools/testing/selftests/kvm/irq_test.c b/tools/testing/selftests/kvm/irq_test.c
index 168c52734fef5..791e576c334cd 100644
--- a/tools/testing/selftests/kvm/irq_test.c
+++ b/tools/testing/selftests/kvm/irq_test.c
@@ -6,7 +6,6 @@
#include "proc_util.h"
#include <libvfio.h>
-#include <linux/sizes.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
@@ -101,7 +100,6 @@ static int vfio_setup_msi(struct vfio_pci_device *device)
allocator = iova_allocator_init(device->iommu);
region = &device->driver.region;
- region->size = SZ_2M;
region->iova = iova_allocator_alloc(allocator, region->size);
region->vaddr = kvm_mmap(region->size, prot, flags, -1);
TEST_ASSERT(region->vaddr != MAP_FAILED, "mmap() failed\n");
diff --git a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c
index 19d9630b24c23..40b8541b588ee 100644
--- a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c
+++ b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c
@@ -418,6 +418,7 @@ static void dsa_send_msi(struct vfio_pci_device *device)
const struct vfio_pci_driver_ops dsa_ops = {
.name = "dsa",
+ .region_size = sizeof(struct dsa_state),
.probe = dsa_probe,
.init = dsa_init,
.remove = dsa_remove,
diff --git a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c
index fd9e05d77ea4b..2fafcc9f51c02 100644
--- a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c
+++ b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c
@@ -576,6 +576,7 @@ static void igb_send_msi(struct vfio_pci_device *device)
const struct vfio_pci_driver_ops igb_ops = {
.name = "igb",
+ .region_size = sizeof(struct igb),
.probe = igb_probe,
.init = igb_init,
.remove = igb_remove,
diff --git a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c
index a871b935542ba..c9b28365c5eb6 100644
--- a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c
+++ b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c
@@ -226,6 +226,7 @@ static void ioat_send_msi(struct vfio_pci_device *device)
const struct vfio_pci_driver_ops ioat_ops = {
.name = "ioat",
+ .region_size = sizeof(struct ioat_state),
.probe = ioat_probe,
.init = ioat_init,
.remove = ioat_remove,
diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c
new file mode 100644
index 0000000000000..a65b52e183bc6
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c
@@ -0,0 +1,1928 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * mlx5 VFIO selftest driver
+ *
+ * Programs mlx5 ConnectX VFs and PFs through the bare-metal command interface
+ * and RDMA WRITE self-loopback to perform DMA. Implements vfio_pci_driver_ops
+ * (probe/init/remove) and plugs into the VFIO selftest framework.
+ */
+#include <sched.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <linux/errno.h>
+#include <linux/io.h>
+#include <linux/log2.h>
+#include <linux/pci_regs.h>
+#include <linux/pci_ids.h>
+
+#include <libvfio.h>
+
+#include "mlx5_hw.h"
+
+/*
+ * Driver state — overlaid on device->driver.region.vaddr.
+ *
+ * Contains both software-only state and HW-visible DMA buffers. HW buffers need
+ * strict IOVA alignment.
+ */
+struct mlx5st_device {
+ /* Back pointer */
+ struct vfio_pci_device *device;
+
+ /* BAR0 */
+ struct mlx5st_initial_seg __iomem *init_seg;
+ void __iomem *bar0;
+
+ /* Command interface */
+ struct mlx5st_cmd_queue_entry *cmd_lay;
+ struct mlx5st_cmd_queue_entry *pages_cmd_lay;
+ u8 cmd_log_stride;
+ unsigned int pages_slot;
+ u8 cmd_token;
+ bool cmd_sig_enabled;
+
+ /* PD */
+ u32 pdn;
+
+ /* Global PA-mode MKEY */
+ u32 global_lkey;
+ u32 global_rkey;
+ u32 mkey_index;
+
+ /* CQ */
+ u32 cqn;
+ u32 cq_ci;
+ u32 cq_arm_sn;
+
+ /* UAR */
+ u32 uar_page;
+ void __iomem *uar_base;
+ unsigned int uar_bf_offset;
+
+ /* EQ (cmd/pages events — polled, not interrupt-driven) */
+ u32 eqn;
+ u32 eq_cons_index;
+ bool have_eq;
+
+ /* MSI EQ (CQ completion events — fires MSI-X) */
+ u32 msi_eqn;
+ u32 msi_eq_cons_index;
+ bool have_msi_eq;
+
+ /* Async pages slot state */
+ bool pages_slot_in_use;
+ bool pages_slot_is_reclaim;
+ unsigned int pages_reclaim_npages;
+ unsigned int pages_pending_give;
+ unsigned int pages_pending_reclaim;
+ u16 pages_pending_func_id;
+ bool pages_func_id_seen;
+
+ /* QP */
+ u32 qpn;
+ u32 sq_pi;
+ u32 sq_ci;
+
+ /* FW pages bitmap */
+ u64 fw_pages_bitmap[MAX_FW_PAGES / 64];
+ u32 fw_pages_given;
+ u16 fw_func_id;
+
+ /* Capabilities */
+ bool fl_supported;
+ u8 log_max_msg;
+
+ /* Buffers used by send_msi() to trigger an interrupt */
+ u64 send_msi_src;
+ u64 send_msi_dst;
+
+ /*
+ * HW-visible DMA buffers below — device reads/writes via DMA.
+ */
+ struct mlx5st_cmd_queue_entry cmd_queue
+ [MLX5_HW_PAGE_SIZE / sizeof(struct mlx5st_cmd_queue_entry)]
+ __aligned(MLX5_HW_PAGE_SIZE);
+ struct mlx5st_send_wqe sq_buf[SQ_WQE_CNT];
+ struct mlx5st_dbrec cq_dbrec;
+ struct mlx5st_dbrec qp_dbrec;
+ struct mlx5st_cqe64 cq_buf[CQ_CQE_CNT];
+
+ /* Slot 0 mailboxes (regular commands) */
+ struct mlx5st_mbox_entry cmd_in_mbox[CMD_MBOX_NENT];
+ struct mlx5st_mbox_entry cmd_out_mbox[CMD_MBOX_NENT];
+
+ /* Pages slot mailboxes (async MANAGE_PAGES) */
+ struct mlx5st_mbox_entry pages_in_mbox[CMD_MBOX_NENT];
+ struct mlx5st_mbox_entry pages_out_mbox[CMD_MBOX_NENT];
+
+ /* EQ does not support page_offset */
+ struct mlx5st_eqe eq_buf[EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE);
+
+ /* MSI EQ buffer — CQ completions generate EQEs here -> MSI-X */
+ struct mlx5st_eqe msi_eq_buf[MSI_EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE);
+
+ u8 fw_pages[MAX_FW_PAGES][MLX5_HW_PAGE_SIZE]
+ __aligned(MLX5_HW_PAGE_SIZE);
+};
+
+/* Check against HW limits on IOVA alignment */
+static_assert(offsetof(struct mlx5st_device, cmd_in_mbox) %
+ CMD_MBOX_STRIDE == 0,
+ "cmd_in_mbox must be stride-aligned");
+static_assert(offsetof(struct mlx5st_device, pages_in_mbox) %
+ CMD_MBOX_STRIDE == 0,
+ "pages_in_mbox must be stride-aligned");
+static_assert(offsetof(struct mlx5st_device, cq_buf) % 64 == 0,
+ "cq_buf must be 64-byte aligned");
+static_assert(offsetof(struct mlx5st_device, sq_buf) % 64 == 0,
+ "sq_buf must be 64-byte aligned");
+static_assert(offsetof(struct mlx5st_device, cq_dbrec) % 64 == 0,
+ "cq_dbrec must be 64-byte aligned");
+static_assert(offsetof(struct mlx5st_device, qp_dbrec) % 64 == 0,
+ "qp_dbrec must be 64-byte aligned");
+static_assert(offsetof(struct mlx5st_device, eq_buf) %
+ MLX5_HW_PAGE_SIZE == 0,
+ "eq_buf must be page-aligned");
+static_assert(offsetof(struct mlx5st_device, msi_eq_buf) %
+ MLX5_HW_PAGE_SIZE == 0,
+ "msi_eq_buf must be page-aligned");
+static_assert(offsetof(struct mlx5st_device, fw_pages) %
+ MLX5_HW_PAGE_SIZE == 0,
+ "fw_pages must be page-aligned");
+
+static struct mlx5st_device *to_mlx5st(struct vfio_pci_device *device)
+{
+ return device->driver.region.vaddr;
+}
+
+/*
+ * Fill a PAS (Physical Address Segment) for a buffer in the driver region.
+ * Sets pas[0] to the page-aligned IOVA and returns the page_offset (the
+ * buffer's byte offset within that page, in units of 64 bytes).
+ */
+static unsigned int mlx5st_fill_pas(struct vfio_pci_device *device, void *buf,
+ __be64 *pas)
+{
+ u64 iova = to_iova(device, buf);
+
+ pas[0] = cpu_to_be64(iova & ~(u64)(MLX5_HW_PAGE_SIZE - 1));
+ return (iova & (MLX5_HW_PAGE_SIZE - 1)) / 64;
+}
+
+/*
+ * Probe — match mlx5 devices by PCI vendor/device ID.
+ */
+
+static int mlx5st_probe(struct vfio_pci_device *device)
+{
+ static const u16 mlx5st_pci_ids[] = {
+ 0x1011, /* Connect-IB */
+ 0x1012, /* Connect-IB VF */
+ 0x1013, /* ConnectX-4 */
+ 0x1014, /* ConnectX-4 VF */
+ 0x1015, /* ConnectX-4LX */
+ 0x1016, /* ConnectX-4LX VF */
+ 0x1017, /* ConnectX-5 */
+ 0x1018, /* ConnectX-5 VF */
+ 0x1019, /* ConnectX-5 Ex */
+ 0x101a, /* ConnectX-5 Ex VF */
+ 0x101b, /* ConnectX-6 */
+ 0x101c, /* ConnectX-6 VF */
+ 0x101d, /* ConnectX-6 Dx */
+ 0x101e, /* ConnectX-6 Dx VF */
+ 0x101f, /* ConnectX-6 LX */
+ 0x1021, /* ConnectX-7 */
+ 0x1023, /* ConnectX-8 */
+ 0x1025, /* ConnectX-9 */
+ 0x1027, /* ConnectX-10 */
+ 0x2101, /* ConnectX-10 NVLink-C2C */
+ 0xa2d2, /* BlueField integrated ConnectX-5 */
+ 0xa2d3, /* BlueField integrated ConnectX-5 VF */
+ 0xa2d6, /* BlueField-2 integrated ConnectX-6 Dx */
+ 0xa2dc, /* BlueField-3 integrated ConnectX-7 */
+ 0xa2df, /* BlueField-4 integrated ConnectX-8 */
+ };
+ unsigned int i;
+ u16 did;
+
+ if (vfio_pci_config_readw(device, PCI_VENDOR_ID) !=
+ PCI_VENDOR_ID_MELLANOX)
+ return -ENODEV;
+
+ did = vfio_pci_config_readw(device, PCI_DEVICE_ID);
+ for (i = 0; i < ARRAY_SIZE(mlx5st_pci_ids); i++) {
+ if (mlx5st_pci_ids[i] == did)
+ return 0;
+ }
+
+ return -ENODEV;
+}
+
+/*
+ * Command interface
+ */
+
+static u8 xor8_buf(const void *buf, size_t offset, size_t len)
+{
+ const u8 *p = buf;
+ u8 sum = 0;
+ size_t i;
+
+ for (i = offset; i < offset + len; i++)
+ sum ^= p[i];
+ return sum;
+}
+
+#define CMD_IF_BOX_CTRL_OFF MLX5_BYTE_OFF(cmd_if_box, reserved_at_1000)
+#define CMD_IF_BOX_CTRL_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, ctrl_signature)
+#define CMD_IF_BOX_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, signature)
+
+static void mlx5st_cmd_calc_block_sig(struct mlx5st_cmd_if_box *blk)
+{
+ MLX5_SET(cmd_if_box, blk, ctrl_signature,
+ ~xor8_buf(blk, CMD_IF_BOX_CTRL_OFF,
+ CMD_IF_BOX_CTRL_SIG_OFF - CMD_IF_BOX_CTRL_OFF));
+ MLX5_SET(cmd_if_box, blk, signature,
+ ~xor8_buf(blk, 0, CMD_IF_BOX_SIG_OFF));
+}
+
+static int mlx5st_cmd_verify_block_sig(struct mlx5st_cmd_if_box *blk)
+{
+ if (xor8_buf(blk, CMD_IF_BOX_CTRL_OFF,
+ CMD_IF_BOX_SIG_OFF - CMD_IF_BOX_CTRL_OFF) != 0xff)
+ return -1;
+ if (xor8_buf(blk, 0, sizeof(struct mlx5st_cmd_if_box)) != 0xff)
+ return -1;
+ return 0;
+}
+
+static unsigned int mlx5st_cmd_setup_mbox_chain(struct vfio_pci_device *device,
+ struct mlx5st_mbox_entry *mbox,
+ unsigned int nblocks, u8 token)
+{
+ unsigned int i;
+
+ for (i = 0; i < nblocks; i++) {
+ struct mlx5st_cmd_if_box *blk = &mbox[i].block;
+ u64 next_iova;
+
+ memset(blk, 0, sizeof(struct mlx5st_cmd_if_box));
+ MLX5_SET(cmd_if_box, blk, block_number, i);
+ MLX5_SET(cmd_if_box, blk, token, token);
+ if (i < nblocks - 1) {
+ next_iova = to_iova(device, &mbox[i + 1]);
+ MLX5_SET(cmd_if_box, blk, next_pointer_63_32,
+ next_iova >> 32);
+ MLX5_SET(cmd_if_box, blk, next_pointer_31_10,
+ (u32)next_iova >> 10);
+ }
+ }
+ return nblocks;
+}
+
+static void mlx5st_cmd_copy_to_mbox(struct mlx5st_mbox_entry *mbox,
+ const void *data, unsigned int len)
+{
+ const u8 *src = data;
+ unsigned int i = 0;
+
+ while (len > 0) {
+ unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ?
+ len :
+ MLX5_CMD_DATA_BLOCK_SIZE;
+
+ memcpy(MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data),
+ src, chunk);
+ src += chunk;
+ len -= chunk;
+ i++;
+ }
+}
+
+static void mlx5st_cmd_copy_from_mbox(void *data,
+ const struct mlx5st_mbox_entry *mbox,
+ unsigned int len)
+{
+ unsigned int i = 0;
+ u8 *dst = data;
+
+ while (len > 0) {
+ unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ?
+ len :
+ MLX5_CMD_DATA_BLOCK_SIZE;
+
+ memcpy(dst,
+ MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data),
+ chunk);
+ dst += chunk;
+ len -= chunk;
+ i++;
+ }
+}
+
+/* Forward declaration — cmd_exec polls events during command wait */
+static void mlx5st_process_events(struct mlx5st_device *dev);
+
+static const char *mlx5st_cmd_name(u16 opcode)
+{
+ switch (opcode) {
+ case MLX5_CMD_OP_QUERY_HCA_CAP: return "QUERY_HCA_CAP";
+ case MLX5_CMD_OP_INIT_HCA: return "INIT_HCA";
+ case MLX5_CMD_OP_TEARDOWN_HCA: return "TEARDOWN_HCA";
+ case MLX5_CMD_OP_ENABLE_HCA: return "ENABLE_HCA";
+ case MLX5_CMD_OP_DISABLE_HCA: return "DISABLE_HCA";
+ case MLX5_CMD_OP_QUERY_PAGES: return "QUERY_PAGES";
+ case MLX5_CMD_OP_MANAGE_PAGES: return "MANAGE_PAGES";
+ case MLX5_CMD_OP_SET_HCA_CAP: return "SET_HCA_CAP";
+ case MLX5_CMD_OP_SET_ISSI: return "SET_ISSI";
+ case MLX5_CMD_OP_CREATE_MKEY: return "CREATE_MKEY";
+ case MLX5_CMD_OP_DESTROY_MKEY: return "DESTROY_MKEY";
+ case MLX5_CMD_OP_CREATE_EQ: return "CREATE_EQ";
+ case MLX5_CMD_OP_DESTROY_EQ: return "DESTROY_EQ";
+ case MLX5_CMD_OP_CREATE_CQ: return "CREATE_CQ";
+ case MLX5_CMD_OP_DESTROY_CQ: return "DESTROY_CQ";
+ case MLX5_CMD_OP_CREATE_QP: return "CREATE_QP";
+ case MLX5_CMD_OP_DESTROY_QP: return "DESTROY_QP";
+ case MLX5_CMD_OP_RST2INIT_QP: return "RST2INIT_QP";
+ case MLX5_CMD_OP_INIT2RTR_QP: return "INIT2RTR_QP";
+ case MLX5_CMD_OP_RTR2RTS_QP: return "RTR2RTS_QP";
+ case MLX5_CMD_OP_ALLOC_PD: return "ALLOC_PD";
+ case MLX5_CMD_OP_DEALLOC_PD: return "DEALLOC_PD";
+ case MLX5_CMD_OP_ALLOC_UAR: return "ALLOC_UAR";
+ case MLX5_CMD_OP_DEALLOC_UAR: return "DEALLOC_UAR";
+ default: return "UNKNOWN";
+ }
+}
+
+/*
+ * Post a command on a given slot: fill the cmd_queue_entry, set up mailbox
+ * chains, compute signatures, hand ownership to FW, and ring the doorbell.
+ */
+static void mlx5st_cmd_post(struct mlx5st_device *dev,
+ struct mlx5st_cmd_queue_entry *cmd,
+ struct mlx5st_mbox_entry *in_mbox,
+ struct mlx5st_mbox_entry *out_mbox,
+ void *in, unsigned int ilen, unsigned int olen,
+ u32 doorbell)
+{
+ struct vfio_pci_device *device = dev->device;
+ unsigned int in_remain, out_remain, in_nblk, out_nblk;
+ unsigned int i;
+ void *cin, *cout;
+ u8 token;
+
+ /* Rotating non-zero token ties cmd entry to its mailbox blocks */
+ token = ++dev->cmd_token;
+ if (!token)
+ token = ++dev->cmd_token;
+
+ in_remain = ilen > MLX5_CMD_INLINE_SZ ? ilen - MLX5_CMD_INLINE_SZ : 0;
+ out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0;
+ in_nblk = (in_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /
+ MLX5_CMD_DATA_BLOCK_SIZE;
+ out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /
+ MLX5_CMD_DATA_BLOCK_SIZE;
+
+ /* Set up mailbox chains */
+ if (in_nblk > 0) {
+ mlx5st_cmd_setup_mbox_chain(device, in_mbox, in_nblk, token);
+ mlx5st_cmd_copy_to_mbox(in_mbox,
+ (u8 *)in + MLX5_CMD_INLINE_SZ,
+ in_remain);
+ }
+ if (out_nblk > 0)
+ mlx5st_cmd_setup_mbox_chain(device, out_mbox, out_nblk, token);
+
+ /* Copy inline input */
+ cin = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_input_inline_data);
+ memset(cin, 0, MLX5_CMD_INLINE_SZ);
+ memcpy(cin, in, ilen < MLX5_CMD_INLINE_SZ ? ilen : MLX5_CMD_INLINE_SZ);
+ MLX5_SET(cmd_queue_entry, cmd, input_length, ilen);
+ MLX5_SET(cmd_queue_entry, cmd, token, token);
+
+ /* Zero inline output */
+ cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);
+ memset(cout, 0, MLX5_CMD_INLINE_SZ);
+ MLX5_SET(cmd_queue_entry, cmd, output_length, olen);
+
+ /*
+ * Compute signatures: mailbox blocks first, then cmd_queue_entry.
+ * The sig must cover the final state including ownership=0x1, but
+ * we must not set ownership until after the sig is in place -
+ * XOR in the 0x1 without storing it to memory.
+ */
+ for (i = 0; i < in_nblk; i++)
+ mlx5st_cmd_calc_block_sig(&in_mbox[i].block);
+ for (i = 0; i < out_nblk; i++)
+ mlx5st_cmd_calc_block_sig(&out_mbox[i].block);
+ MLX5_SET(cmd_queue_entry, cmd, signature, 0);
+ MLX5_SET(cmd_queue_entry, cmd, signature,
+ ~(xor8_buf(cmd, 0, sizeof(struct mlx5st_cmd_queue_entry)) ^
+ 0x1));
+
+ /* Ensure all cmd data (including sig) is visible, then hand to FW */
+ dma_wmb();
+ MLX5_SET_ONCE(cmd_queue_entry, cmd, ownership, 1);
+
+ /* Ring doorbell */
+ MLX5_SET_MMIO(initial_seg, dev->init_seg, command_doorbell_vector,
+ doorbell);
+}
+
+static void mlx5st_cmd_exec(struct mlx5st_device *dev, void *in,
+ unsigned int ilen, void *out, unsigned int olen)
+{
+ struct mlx5st_cmd_queue_entry *cmd = dev->cmd_lay;
+ unsigned int out_remain, out_nblk;
+ struct timespec start, now;
+ unsigned int elapsed;
+ unsigned int i;
+ void *cout;
+
+ mlx5st_cmd_post(dev, cmd, dev->cmd_in_mbox, dev->cmd_out_mbox, in,
+ ilen, olen, 1);
+
+ out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0;
+ out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) /
+ MLX5_CMD_DATA_BLOCK_SIZE;
+
+ /* Poll for completion — also process EQ events for PF page requests */
+ clock_gettime(CLOCK_MONOTONIC, &start);
+ for (;;) {
+ if (!MLX5_GET_ONCE(cmd_queue_entry, cmd, ownership))
+ break;
+ if (dev->have_eq)
+ mlx5st_process_events(dev);
+ sched_yield();
+ clock_gettime(CLOCK_MONOTONIC, &now);
+ elapsed = (now.tv_sec - start.tv_sec) * 1000 +
+ (now.tv_nsec - start.tv_nsec) / 1000000;
+ if (elapsed > MLX5_CMD_TIMEOUT_MS)
+ VFIO_FAIL("cmd timeout after %d ms", elapsed);
+ }
+ /* Ensure output data reads happen after ownership is seen clear */
+ dma_rmb();
+
+ /* Verify output signatures when FW has checksums enabled */
+ if (dev->cmd_sig_enabled) {
+ if (xor8_buf(cmd, 0,
+ sizeof(struct mlx5st_cmd_queue_entry)) != 0xff)
+ VFIO_FAIL("cmd output signature mismatch");
+ for (i = 0; i < out_nblk; i++) {
+ if (mlx5st_cmd_verify_block_sig(
+ &dev->cmd_out_mbox[i].block))
+ VFIO_FAIL("cmd output mailbox block %d signature mismatch",
+ i);
+ }
+ }
+
+ /* Copy output: inline first */
+ cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);
+ memcpy(out, cout, olen < MLX5_CMD_INLINE_SZ ? olen : MLX5_CMD_INLINE_SZ);
+
+ /* Copy remaining from output mailbox chain */
+ if (out_remain > 0)
+ mlx5st_cmd_copy_from_mbox((u8 *)out + MLX5_CMD_INLINE_SZ,
+ dev->cmd_out_mbox, out_remain);
+
+ /* Check command status */
+ if (MLX5_GET(enable_hca_out, out, status) != MLX5_CMD_STAT_OK)
+ VFIO_FAIL("%s: status=0x%x syndrome=0x%x",
+ mlx5st_cmd_name(MLX5_GET(enable_hca_in, in, opcode)),
+ MLX5_GET(enable_hca_out, out, status),
+ MLX5_GET(enable_hca_out, out, syndrome));
+}
+
+static struct mlx5st_cmd_queue_entry *
+mlx5st_cmd_slot_init(struct mlx5st_device *dev, unsigned int slot,
+ struct mlx5st_mbox_entry *in_mbox,
+ struct mlx5st_mbox_entry *out_mbox)
+{
+ struct vfio_pci_device *device = dev->device;
+ struct mlx5st_cmd_queue_entry *cmd =
+ &dev->cmd_queue[(slot << dev->cmd_log_stride) /
+ sizeof(struct mlx5st_cmd_queue_entry)];
+ u64 iova;
+
+ MLX5_SET(cmd_queue_entry, cmd, type,
+ MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT);
+ iova = to_iova(device, in_mbox);
+ MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_63_32,
+ iova >> 32);
+ MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_31_9, iova >> 9);
+ iova = to_iova(device, out_mbox);
+ MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_63_32,
+ iova >> 32);
+ MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_31_9,
+ iova >> 9);
+ return cmd;
+}
+
+static void mlx5st_cmd_init(struct mlx5st_device *dev)
+{
+ struct mlx5st_initial_seg __iomem *seg = dev->init_seg;
+ struct vfio_pci_device *device = dev->device;
+ u16 cmdif_rev;
+ u8 log_sz;
+ u64 iova;
+
+ cmdif_rev = MLX5_GET_MMIO(initial_seg, seg, cmd_interface_rev);
+ VFIO_ASSERT_EQ(cmdif_rev, 5);
+
+ /* Read command queue geometry from BAR */
+ log_sz = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_size);
+ dev->cmd_log_stride = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_stride);
+ dev->pages_slot = (1 << log_sz) - 1;
+
+ VFIO_ASSERT_LE((unsigned int)(1 << log_sz), 32u);
+ VFIO_ASSERT_GE((unsigned int)(1 << dev->cmd_log_stride),
+ (unsigned int)sizeof(struct mlx5st_cmd_queue_entry));
+ VFIO_ASSERT_LE((unsigned int)((dev->pages_slot + 1) <<
+ dev->cmd_log_stride),
+ (unsigned int)sizeof(dev->cmd_queue));
+
+ /* Set up slot 0 — regular commands */
+ dev->cmd_lay = mlx5st_cmd_slot_init(dev, 0, dev->cmd_in_mbox,
+ dev->cmd_out_mbox);
+
+ /* Set up pages slot — async MANAGE_PAGES */
+ dev->pages_cmd_lay = mlx5st_cmd_slot_init(dev, dev->pages_slot,
+ dev->pages_in_mbox,
+ dev->pages_out_mbox);
+
+ /* Write command queue page address to BAR0 */
+ iova = to_iova(device, dev->cmd_queue);
+ MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_63_32, iova >> 32);
+ MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_31_12, iova >> 12);
+
+ dev_dbg(device,
+ "Command interface initialized (cmdif_rev=5, log_sz=%u, log_stride=%u, pages_slot=%u)\n",
+ log_sz, dev->cmd_log_stride, dev->pages_slot);
+}
+
+/*
+ * FW pages: bitmap allocator + MANAGE_PAGES
+ */
+
+static void mlx5st_fw_pages_alloc(struct mlx5st_device *dev,
+ unsigned int npages, u64 *iovas)
+{
+ struct vfio_pci_device *device = dev->device;
+ unsigned int found = 0;
+ unsigned int w, b;
+ u64 word;
+
+ for (w = 0; w < MAX_FW_PAGES / 64 && found < npages; w++) {
+ word = dev->fw_pages_bitmap[w];
+
+ for (b = 0; b < 64 && found < npages; b++) {
+ if (!(word & (1ULL << b))) {
+ unsigned int idx = w * 64 + b;
+
+ dev->fw_pages_bitmap[w] |= (1ULL << b);
+ iovas[found++] = to_iova(device,
+ dev->fw_pages[idx]);
+ }
+ }
+ }
+ VFIO_ASSERT_EQ(found, npages);
+ dev->fw_pages_given += npages;
+}
+
+static void mlx5st_fw_pages_free(struct mlx5st_device *dev,
+ unsigned int npages, const u64 *iovas)
+{
+ struct vfio_pci_device *device = dev->device;
+ unsigned int i, idx;
+ u64 off;
+
+ for (i = 0; i < npages; i++) {
+ off = iovas[i] - to_iova(device, dev->fw_pages);
+ idx = off / MLX5_HW_PAGE_SIZE;
+
+ VFIO_ASSERT_TRUE(idx < MAX_FW_PAGES);
+ dev->fw_pages_bitmap[idx / 64] &= ~(1ULL << (idx % 64));
+ }
+ dev->fw_pages_given -= npages;
+}
+
+static void *mlx5st_build_manage_pages_give(u16 func_id, unsigned int npages,
+ const u64 *iovas,
+ unsigned int *out_inlen)
+{
+ unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in) + npages * 8;
+ unsigned int i;
+ void *in;
+
+ in = calloc(1, inlen);
+ VFIO_ASSERT_NOT_NULL(in);
+
+ MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES);
+ MLX5_SET(manage_pages_in, in, op_mod,
+ MLX5_MANAGE_PAGES_IN_OP_MOD_ALLOCATION_SUCCESS);
+ MLX5_SET(manage_pages_in, in, function_id, func_id);
+ MLX5_SET(manage_pages_in, in, input_num_entries, npages);
+
+ for (i = 0; i < npages; i++)
+ MLX5_ARRAY_SET64(manage_pages_in, in, pas, i, iovas[i]);
+
+ *out_inlen = inlen;
+ return in;
+}
+
+static void mlx5st_fw_pages_give_one(struct mlx5st_device *dev, u16 func_id,
+ unsigned int npages, u64 *iovas)
+{
+ u32 out[MLX5_ST_SZ_DW(manage_pages_out)] = {};
+ unsigned int inlen;
+ void *in;
+
+ in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen);
+ mlx5st_cmd_exec(dev, in, inlen, out, sizeof(out));
+ free(in);
+}
+
+static void mlx5st_fw_pages_give(struct mlx5st_device *dev, u16 func_id,
+ unsigned int npages)
+{
+ unsigned int remaining = npages;
+ u64 *iovas;
+
+ if (!npages)
+ return;
+
+ iovas = calloc(npages, sizeof(u64));
+ VFIO_ASSERT_NOT_NULL(iovas);
+
+ mlx5st_fw_pages_alloc(dev, npages, iovas);
+
+ /* Batch into chunks that fit in one mailbox */
+ for (unsigned int off = 0; remaining > 0;) {
+ unsigned int batch = remaining < MAX_FW_PAGES_PER_CMD ?
+ remaining :
+ MAX_FW_PAGES_PER_CMD;
+
+ mlx5st_fw_pages_give_one(dev, func_id, batch, iovas + off);
+ off += batch;
+ remaining -= batch;
+ }
+
+ dev_dbg(dev->device, "MANAGE_PAGES GIVE: %d pages to func_id=%u\n",
+ npages, func_id);
+ free(iovas);
+}
+
+static void mlx5st_fw_pages_satisfy(struct mlx5st_device *dev, int boot)
+{
+ u32 qo[MLX5_ST_SZ_DW(query_pages_out)] = {};
+ u32 qi[MLX5_ST_SZ_DW(query_pages_in)] = {};
+ u16 func_id;
+ int npages;
+
+ MLX5_SET(query_pages_in, qi, opcode, MLX5_CMD_OP_QUERY_PAGES);
+ MLX5_SET(query_pages_in, qi, op_mod, boot ? 0x01 : 0x02);
+ mlx5st_cmd_exec(dev, qi, sizeof(qi), qo, sizeof(qo));
+
+ npages = MLX5_GET(query_pages_out, qo, num_pages);
+ func_id = MLX5_GET(query_pages_out, qo, function_id);
+ dev_dbg(dev->device, "QUERY_PAGES (%s): %d pages (func_id=%u)\n",
+ boot ? "boot" : "init", npages, func_id);
+
+ if (npages > 0) {
+ dev->fw_func_id = func_id;
+ mlx5st_fw_pages_give(dev, func_id, npages);
+ }
+}
+
+/*
+ * Async MANAGE_PAGES on the pages command slot.
+ *
+ * On PFs, firmware sends PAGE_REQUEST events via the EQ during command
+ * execution. We must respond with MANAGE_PAGES on a second command slot
+ * before the first (regular) command can complete.
+ */
+
+static void mlx5st_pages_slot_post(struct mlx5st_device *dev, void *in,
+ unsigned int ilen, unsigned int olen)
+{
+ mlx5st_cmd_post(dev, dev->pages_cmd_lay, dev->pages_in_mbox,
+ dev->pages_out_mbox, in, ilen, olen,
+ 1 << dev->pages_slot);
+}
+
+static void mlx5st_pages_slot_give(struct mlx5st_device *dev, u16 func_id,
+ unsigned int npages)
+{
+ unsigned int inlen;
+ u64 *iovas;
+ void *in;
+
+ iovas = calloc(npages, sizeof(u64));
+ VFIO_ASSERT_NOT_NULL(iovas);
+
+ mlx5st_fw_pages_alloc(dev, npages, iovas);
+
+ in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen);
+ free(iovas);
+
+ mlx5st_pages_slot_post(dev, in, inlen,
+ MLX5_ST_SZ_BYTES(manage_pages_out));
+ dev->pages_slot_in_use = true;
+ dev->pages_slot_is_reclaim = false;
+ free(in);
+
+ dev_dbg(dev->device,
+ "PAGE_REQUEST: %d pages given async to func_id=%u\n",
+ npages, func_id);
+}
+
+static void mlx5st_pages_slot_reclaim(struct mlx5st_device *dev, u16 func_id,
+ unsigned int npages)
+{
+ unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in);
+ unsigned int outlen =
+ MLX5_ST_SZ_BYTES(manage_pages_out) + npages * 8;
+ void *in;
+
+ in = calloc(1, inlen);
+ VFIO_ASSERT_NOT_NULL(in);
+
+ MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES);
+ MLX5_SET(manage_pages_in, in, op_mod,
+ MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES);
+ MLX5_SET(manage_pages_in, in, function_id, func_id);
+ MLX5_SET(manage_pages_in, in, input_num_entries, npages);
+
+ mlx5st_pages_slot_post(dev, in, inlen, outlen);
+ dev->pages_slot_in_use = true;
+ dev->pages_slot_is_reclaim = true;
+ dev->pages_reclaim_npages = npages;
+ free(in);
+
+ dev_dbg(dev->device,
+ "PAGE_REQUEST: reclaim %d pages async from func_id=%u\n",
+ npages, func_id);
+}
+
+static void mlx5st_pages_slot_kick(struct mlx5st_device *dev)
+{
+ unsigned int batch;
+
+ if (dev->pages_slot_in_use)
+ return;
+
+ if (dev->pages_pending_give) {
+ batch = dev->pages_pending_give < MAX_FW_PAGES_PER_CMD ?
+ dev->pages_pending_give :
+ MAX_FW_PAGES_PER_CMD;
+ dev->pages_pending_give -= batch;
+ mlx5st_pages_slot_give(dev, dev->pages_pending_func_id, batch);
+ } else if (dev->pages_pending_reclaim) {
+ batch = dev->pages_pending_reclaim < MAX_FW_PAGES_PER_CMD ?
+ dev->pages_pending_reclaim :
+ MAX_FW_PAGES_PER_CMD;
+ dev->pages_pending_reclaim -= batch;
+ mlx5st_pages_slot_reclaim(dev, dev->pages_pending_func_id,
+ batch);
+ }
+}
+
+static void mlx5st_fw_pages_give_async(struct mlx5st_device *dev,
+ u16 func_id, unsigned int npages)
+{
+ if (!npages)
+ return;
+
+ dev->pages_pending_give += npages;
+ dev->pages_pending_func_id = func_id;
+ mlx5st_pages_slot_kick(dev);
+}
+
+static void mlx5st_fw_pages_reclaim_async(struct mlx5st_device *dev,
+ u16 func_id, unsigned int npages)
+{
+ dev->pages_pending_reclaim += npages;
+ dev->pages_pending_func_id = func_id;
+ mlx5st_pages_slot_kick(dev);
+}
+
+static void mlx5st_pages_slot_complete(struct mlx5st_device *dev)
+{
+ struct mlx5st_cmd_queue_entry *cmd = dev->pages_cmd_lay;
+ void *cout;
+
+ dma_rmb();
+
+ cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data);
+ if (MLX5_GET(enable_hca_out, cout, status) != MLX5_CMD_STAT_OK)
+ VFIO_FAIL("async MANAGE_PAGES failed: status=0x%x syndrome=0x%x",
+ MLX5_GET(enable_hca_out, cout, status),
+ MLX5_GET(enable_hca_out, cout, syndrome));
+
+ if (dev->pages_slot_is_reclaim) {
+ unsigned int outlen = MLX5_ST_SZ_BYTES(manage_pages_out) +
+ dev->pages_reclaim_npages * 8;
+ unsigned int num_claimed;
+ unsigned int i;
+ void *out;
+ u64 *iovas;
+
+ out = calloc(1, outlen);
+ iovas = calloc(dev->pages_reclaim_npages, sizeof(u64));
+ VFIO_ASSERT_NOT_NULL(out);
+ VFIO_ASSERT_NOT_NULL(iovas);
+
+ /* Copy inline output */
+ memcpy(out, cout, MLX5_CMD_INLINE_SZ);
+ if (outlen > MLX5_CMD_INLINE_SZ)
+ mlx5st_cmd_copy_from_mbox(
+ (u8 *)out + MLX5_CMD_INLINE_SZ,
+ dev->pages_out_mbox,
+ outlen - MLX5_CMD_INLINE_SZ);
+
+ num_claimed =
+ MLX5_GET(manage_pages_out, out, output_num_entries);
+ for (i = 0; i < num_claimed; i++)
+ iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas,
+ i);
+
+ mlx5st_fw_pages_free(dev, num_claimed, iovas);
+ dev_dbg(dev->device, "PAGE_REQUEST: reclaimed %d pages\n",
+ num_claimed);
+
+ free(iovas);
+ free(out);
+ }
+
+ dev->pages_slot_in_use = false;
+ mlx5st_pages_slot_kick(dev);
+}
+
+/*
+ * UAR alloc/dealloc
+ */
+
+static void mlx5st_alloc_uar(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(alloc_uar_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(alloc_uar_in)] = {};
+
+ MLX5_SET(alloc_uar_in, in, opcode, MLX5_CMD_OP_ALLOC_UAR);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->uar_page = MLX5_GET(alloc_uar_out, out, uar);
+ dev->uar_base = (u8 __iomem*)dev->bar0 + dev->uar_page * MLX5_HW_PAGE_SIZE;
+ dev->uar_bf_offset = MLX5_BF_OFFSET;
+
+ dev_dbg(dev->device,
+ "Allocated UAR page_id=%u, doorbell offset=0x%x\n",
+ dev->uar_page,
+ dev->uar_page * MLX5_HW_PAGE_SIZE + MLX5_BF_OFFSET);
+}
+
+static void mlx5st_dealloc_uar(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(dealloc_uar_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(dealloc_uar_in)] = {};
+
+ MLX5_SET(dealloc_uar_in, in, opcode, MLX5_CMD_OP_DEALLOC_UAR);
+ MLX5_SET(dealloc_uar_in, in, uar, dev->uar_page);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * EQ infrastructure
+ */
+
+static struct mlx5st_eqe *mlx5st_eq_get_eqe(struct mlx5st_device *dev, u32 cc)
+{
+ u32 ci = dev->eq_cons_index + cc;
+ struct mlx5st_eqe *eqe = &dev->eq_buf[ci % EQ_NENT];
+ u8 owner = MLX5_GET_ONCE(eqe, eqe, owner);
+ u8 expected = !!(ci & EQ_NENT);
+
+ if (owner != expected)
+ return NULL;
+ dma_rmb();
+ return eqe;
+}
+
+static void mlx5st_eq_update_ci(struct mlx5st_device *dev, u32 cc, bool arm)
+{
+ u32 val;
+
+ dev->eq_cons_index += cc;
+ val = (dev->eq_cons_index & 0xffffff) | (dev->eqn << 24);
+ iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET +
+ (arm ? 0 : 8));
+}
+
+static void mlx5st_create_eq(struct mlx5st_device *dev)
+{
+ struct vfio_pci_device *device = dev->device;
+ u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {};
+ u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {};
+ struct mlx5_ifc_eqc_bits *eqc;
+ unsigned int i;
+ __be64 *pas;
+
+ /* Initialize EQE owner bits */
+ for (i = 0; i < EQ_NENT; i++) {
+ struct mlx5st_eqe *eqe = &dev->eq_buf[i];
+
+ MLX5_SET_ONCE(eqe, eqe, owner, 1);
+ }
+
+ MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ);
+
+ /* Subscribe to CMD completions and PAGE_REQUEST events */
+ MLX5_ARRAY_SET64(create_eq_in, in, event_bitmask, 0,
+ (1ULL << MLX5_EVENT_TYPE_CMD) |
+ (1ULL << MLX5_EVENT_TYPE_PAGE_REQUEST));
+
+ eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);
+ MLX5_SET(eqc, eqc, log_eq_size, LOG_EQ_SIZE);
+ MLX5_SET(eqc, eqc, uar_page, dev->uar_page);
+ pas = MLX5_ADDR_OF(create_eq_in, in, pas);
+ VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->eq_buf, pas), 0u);
+ MLX5_SET(eqc, eqc, log_page_size, 0);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->eqn = MLX5_GET(create_eq_out, out, eq_number);
+ dev->eq_cons_index = 0;
+ mlx5st_eq_update_ci(dev, 0, 0);
+ dev->have_eq = true;
+
+ dev_dbg(device, "Created EQ: eqn=%u, %d entries (CMD+PAGE_REQUEST)\n",
+ dev->eqn, EQ_NENT);
+}
+
+static void mlx5st_destroy_eq(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {};
+
+ MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ);
+ MLX5_SET(destroy_eq_in, in, eq_number, dev->eqn);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * Drain all pending EQ events. Dispatches PAGE_REQUEST to the async pages
+ * slot and CMD completions to the pages slot completion handler.
+ */
+static void mlx5st_process_events(struct mlx5st_device *dev)
+{
+ struct mlx5st_eqe *eqe;
+ u32 cc = 0;
+
+ while ((eqe = mlx5st_eq_get_eqe(dev, cc))) {
+ u8 type = MLX5_GET(eqe, eqe, event_type);
+
+ switch (type) {
+ case MLX5_EVENT_TYPE_PAGE_REQUEST: {
+ void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data);
+ u16 func_id = MLX5_GET(pages_req_event, evdata,
+ function_id);
+ s32 npages = (s32)MLX5_GET(pages_req_event, evdata,
+ num_pages);
+
+ /*
+ * The selftest doesn't use more than one func_id so a
+ * simple counter approach is possible.
+ */
+ if (dev->pages_func_id_seen)
+ VFIO_ASSERT_EQ(func_id,
+ dev->pages_pending_func_id);
+ dev->pages_func_id_seen = true;
+
+ if (npages > 0)
+ mlx5st_fw_pages_give_async(dev, func_id,
+ npages);
+ else if (npages < 0)
+ mlx5st_fw_pages_reclaim_async(dev, func_id,
+ -npages);
+ break;
+ }
+ case MLX5_EVENT_TYPE_CMD: {
+ void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data);
+ u32 vector = MLX5_GET(cmd_inter_comp_event, evdata,
+ command_completion_vector);
+
+ if (vector & (1U << dev->pages_slot))
+ mlx5st_pages_slot_complete(dev);
+ break;
+ }
+ default:
+ break;
+ }
+ cc++;
+ }
+
+ if (cc)
+ mlx5st_eq_update_ci(dev, cc, 0);
+}
+
+/*
+ * MSI EQ — dedicated EQ for CQ completion events that fires MSI-X.
+ * Separate from the cmd/pages EQ so that only CQ completions (from
+ * send_msi or memcpy) trigger the interrupt vector.
+ */
+
+static void mlx5st_msi_eq_drain(struct mlx5st_device *dev)
+{
+ u32 cc = 0;
+ u32 val;
+
+ while (cc < MSI_EQ_NENT) {
+ u32 ci = dev->msi_eq_cons_index + cc;
+ struct mlx5st_eqe *eqe =
+ &dev->msi_eq_buf[ci % MSI_EQ_NENT];
+
+ if (MLX5_GET_ONCE(eqe, eqe, owner) != !!(ci & MSI_EQ_NENT))
+ break;
+ cc++;
+ }
+
+ /* Update consumer index and re-arm for next interrupt */
+ dev->msi_eq_cons_index += cc;
+ val = (dev->msi_eq_cons_index & 0xffffff) | (dev->msi_eqn << 24);
+ iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET);
+}
+
+static void mlx5st_create_msi_eq(struct mlx5st_device *dev)
+{
+ struct vfio_pci_device *device = dev->device;
+ u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {};
+ u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {};
+ struct mlx5_ifc_eqc_bits *eqc;
+ unsigned int i;
+ __be64 *pas;
+
+ /* Initialize EQE owner bits */
+ for (i = 0; i < MSI_EQ_NENT; i++) {
+ struct mlx5st_eqe *eqe = &dev->msi_eq_buf[i];
+
+ MLX5_SET_ONCE(eqe, eqe, owner, 1);
+ }
+
+ MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ);
+
+ /*
+ * No event_bitmask — completion events are routed to this EQ via
+ * the CQ's c_eqn field, not through CREATE_EQ subscription.
+ */
+ eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);
+ MLX5_SET(eqc, eqc, log_eq_size, LOG_MSI_EQ_SIZE);
+ MLX5_SET(eqc, eqc, uar_page, dev->uar_page);
+ MLX5_SET(eqc, eqc, intr, MSI_VECTOR);
+ pas = MLX5_ADDR_OF(create_eq_in, in, pas);
+ VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->msi_eq_buf, pas), 0u);
+ MLX5_SET(eqc, eqc, log_page_size, 0);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->msi_eqn = MLX5_GET(create_eq_out, out, eq_number);
+ dev->msi_eq_cons_index = 0;
+ dev->have_msi_eq = true;
+ mlx5st_msi_eq_drain(dev);
+
+ dev_dbg(device,
+ "Created MSI EQ: eqn=%u, %d entries (COMP), vector=%d\n",
+ dev->msi_eqn, MSI_EQ_NENT, MSI_VECTOR);
+}
+
+static void mlx5st_destroy_msi_eq(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {};
+
+ MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ);
+ MLX5_SET(destroy_eq_in, in, eq_number, dev->msi_eqn);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * HCA init / teardown
+ */
+
+#define FW_INIT_TIMEOUT_MS 120000
+#define FW_INIT_WAIT_MS 200
+
+static void mlx5st_wait_fw_init(struct mlx5st_device *dev)
+{
+ struct timespec start, now;
+ unsigned int elapsed;
+
+ clock_gettime(CLOCK_MONOTONIC, &start);
+ while (MLX5_GET_MMIO(initial_seg, dev->init_seg, initializing)) {
+ usleep(FW_INIT_WAIT_MS * 1000);
+ clock_gettime(CLOCK_MONOTONIC, &now);
+ elapsed = (now.tv_sec - start.tv_sec) * 1000 +
+ (now.tv_nsec - start.tv_nsec) / 1000000;
+ if (elapsed > FW_INIT_TIMEOUT_MS)
+ VFIO_FAIL("FW init timeout after %d ms", elapsed);
+ }
+}
+
+static void mlx5st_set_issi(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(set_issi_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(set_issi_in)] = {};
+
+ MLX5_SET(set_issi_in, in, opcode, MLX5_CMD_OP_SET_ISSI);
+ MLX5_SET(set_issi_in, in, current_issi, 1);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "SET_ISSI: OK (issi=1)\n");
+}
+
+static void mlx5st_set_hca_caps(struct mlx5st_device *dev)
+{
+ u32 qout[MLX5_ST_SZ_DW(query_hca_cap_out)] = {};
+ u32 qin[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};
+ u32 sout[MLX5_ST_SZ_DW(set_hca_cap_out)] = {};
+ u32 sin[MLX5_ST_SZ_DW(set_hca_cap_in)] = {};
+ struct mlx5_ifc_cmd_hca_cap_bits *set_hca_cap;
+ u32 max_checksum;
+
+ /* Query max caps to learn cmdif_checksum support */
+ MLX5_SET(query_hca_cap_in, qin, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
+ MLX5_SET(query_hca_cap_in, qin, op_mod,
+ (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1));
+ mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout));
+
+ max_checksum = MLX5_GET(
+ cmd_hca_cap,
+ MLX5_ADDR_OF(query_hca_cap_out, qout, capability),
+ cmdif_checksum);
+
+ /* Query current caps as base for SET */
+ memset(qout, 0, sizeof(qout));
+ MLX5_SET(query_hca_cap_in, qin, op_mod,
+ (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) |
+ HCA_CAP_OPMOD_GET_CUR);
+ mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout));
+
+ set_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, sin, capability);
+ memcpy(set_hca_cap,
+ MLX5_ADDR_OF(query_hca_cap_out, qout, capability),
+ MLX5_ST_SZ_BYTES(cmd_hca_cap));
+
+ MLX5_SET(cmd_hca_cap, set_hca_cap, cmdif_checksum, max_checksum);
+ MLX5_SET(cmd_hca_cap, set_hca_cap, log_uar_page_sz, 0);
+
+ MLX5_SET(set_hca_cap_in, sin, opcode, MLX5_CMD_OP_SET_HCA_CAP);
+ MLX5_SET(set_hca_cap_in, sin, op_mod,
+ MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1);
+
+ mlx5st_cmd_exec(dev, sin, sizeof(sin), sout, sizeof(sout));
+
+ dev->cmd_sig_enabled = max_checksum == 0x3;
+ dev_dbg(dev->device, "SET_HCA_CAP: OK (cmdif_checksum=%u)\n",
+ max_checksum);
+}
+
+static void mlx5st_hca_init(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(enable_hca_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(enable_hca_in)] = {};
+
+ mlx5st_wait_fw_init(dev);
+ dev_dbg(dev->device, "Firmware ready\n");
+
+ MLX5_SET(enable_hca_in, in, opcode, MLX5_CMD_OP_ENABLE_HCA);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "ENABLE_HCA: OK\n");
+
+ mlx5st_set_issi(dev);
+ mlx5st_fw_pages_satisfy(dev, 1);
+
+ mlx5st_set_hca_caps(dev);
+ mlx5st_fw_pages_satisfy(dev, 0);
+
+ memset(in, 0, sizeof(in));
+ memset(out, 0, sizeof(out));
+ MLX5_SET(init_hca_in, in, opcode, MLX5_CMD_OP_INIT_HCA);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "INIT_HCA: OK\n");
+
+ /*
+ * Create EQ immediately after INIT_HCA so PAGE_REQUEST events
+ * are captured during all subsequent commands.
+ */
+ mlx5st_alloc_uar(dev);
+ mlx5st_create_eq(dev);
+}
+
+static void mlx5st_disable_hca(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(disable_hca_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(disable_hca_in)] = {};
+
+ MLX5_SET(disable_hca_in, in, opcode, MLX5_CMD_OP_DISABLE_HCA);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+static void mlx5st_fw_pages_reclaim(struct mlx5st_device *dev, u16 func_id)
+{
+ unsigned int npages = dev->fw_pages_given;
+ unsigned int total_claimed = 0;
+
+ while (npages > 0) {
+ unsigned int batch = npages < MAX_FW_PAGES_PER_CMD ?
+ npages :
+ MAX_FW_PAGES_PER_CMD;
+ unsigned int outlen =
+ MLX5_ST_SZ_BYTES(manage_pages_out) + batch * 8;
+ unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in);
+ unsigned int num_claimed;
+ unsigned int i;
+ void *in, *out;
+ u64 *iovas;
+
+ in = calloc(1, inlen);
+ out = calloc(1, outlen);
+ iovas = calloc(batch, sizeof(u64));
+ VFIO_ASSERT_NOT_NULL(in);
+ VFIO_ASSERT_NOT_NULL(out);
+ VFIO_ASSERT_NOT_NULL(iovas);
+
+ MLX5_SET(manage_pages_in, in, opcode,
+ MLX5_CMD_OP_MANAGE_PAGES);
+ MLX5_SET(manage_pages_in, in, op_mod,
+ MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES);
+ MLX5_SET(manage_pages_in, in, function_id, func_id);
+ MLX5_SET(manage_pages_in, in, input_num_entries, batch);
+
+ mlx5st_cmd_exec(dev, in, inlen, out, outlen);
+
+ num_claimed =
+ MLX5_GET(manage_pages_out, out, output_num_entries);
+ for (i = 0; i < num_claimed; i++)
+ iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas,
+ i);
+
+ mlx5st_fw_pages_free(dev, num_claimed, iovas);
+ total_claimed += num_claimed;
+ npages -= num_claimed;
+
+ free(iovas);
+ free(in);
+ free(out);
+
+ if (!num_claimed && !dev->fw_pages_given)
+ break;
+ if (!num_claimed)
+ VFIO_FAIL("MANAGE_PAGES RECLAIM: FW returned 0 but %d pages still given",
+ dev->fw_pages_given);
+ }
+
+ dev_dbg(dev->device,
+ "MANAGE_PAGES RECLAIM: %d pages (%d still given)\n",
+ total_claimed, dev->fw_pages_given);
+}
+
+static void mlx5st_hca_teardown(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(teardown_hca_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(teardown_hca_in)] = {};
+
+ /* Drain async pages slot, then stop EQ processing */
+ while (dev->pages_slot_in_use) {
+ if (!MLX5_GET_ONCE(cmd_queue_entry, dev->pages_cmd_lay,
+ ownership))
+ mlx5st_pages_slot_complete(dev);
+ else
+ sched_yield();
+ }
+ dev->have_eq = false;
+
+ if (dev->eqn) {
+ mlx5st_destroy_eq(dev);
+ dev->eqn = 0;
+ }
+ if (dev->uar_page) {
+ mlx5st_dealloc_uar(dev);
+ dev->uar_page = 0;
+ }
+
+ dev_dbg(dev->device, " hca_teardown: TEARDOWN_HCA\n");
+ MLX5_SET(teardown_hca_in, in, opcode, MLX5_CMD_OP_TEARDOWN_HCA);
+ MLX5_SET(teardown_hca_in, in, profile,
+ MLX5_TEARDOWN_HCA_IN_PROFILE_GRACEFUL_CLOSE);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ if (dev->fw_pages_given > 0) {
+ dev_dbg(dev->device, " hca_teardown: reclaim %d pages\n",
+ dev->fw_pages_given);
+ mlx5st_fw_pages_reclaim(dev, dev->fw_func_id);
+ }
+
+ dev_dbg(dev->device, " hca_teardown: DISABLE_HCA\n");
+ mlx5st_disable_hca(dev);
+}
+
+/*
+ * Query capabilities
+ */
+static void mlx5st_query_fl_caps(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {};
+ bool fl_roce_en, fl_roce_dis;
+
+ /* Query RoCE capabilities */
+ MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
+ MLX5_SET(query_hca_cap_in, in, op_mod,
+ (MLX5_SET_HCA_CAP_OP_MOD_ROCE << 1) | HCA_CAP_OPMOD_GET_CUR);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ fl_roce_en = MLX5_GET(query_hca_cap_out, out,
+ capability.roce_cap.fl_rc_qp_when_roce_enabled);
+ fl_roce_dis = MLX5_GET(query_hca_cap_out, out,
+ capability.roce_cap.fl_rc_qp_when_roce_disabled);
+
+ /* Also check general caps */
+ memset(in, 0, sizeof(in));
+ memset(out, 0, sizeof(out));
+ MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
+ MLX5_SET(query_hca_cap_in, in, op_mod,
+ (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) |
+ HCA_CAP_OPMOD_GET_CUR);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ fl_roce_dis |=
+ MLX5_GET(query_hca_cap_out, out,
+ capability.cmd_hca_cap.fl_rc_qp_when_roce_disabled);
+
+ dev->fl_supported = fl_roce_en || fl_roce_dis;
+ dev->log_max_msg = MLX5_GET(query_hca_cap_out, out,
+ capability.cmd_hca_cap.log_max_msg);
+ dev_dbg(dev->device,
+ "HCA capabilities: fl_roce_enabled=%d fl_roce_disabled=%d log_max_msg=%u\n",
+ fl_roce_en, fl_roce_dis, dev->log_max_msg);
+
+ VFIO_ASSERT_TRUE(dev->fl_supported,
+ "Force-loopback not supported on this device");
+}
+
+/*
+ * Resource allocation
+ */
+
+static void mlx5st_alloc_pd(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(alloc_pd_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(alloc_pd_in)] = {};
+
+ MLX5_SET(alloc_pd_in, in, opcode, MLX5_CMD_OP_ALLOC_PD);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->pdn = MLX5_GET(alloc_pd_out, out, pd);
+ dev_dbg(dev->device, "Allocated PD pdn=%u\n", dev->pdn);
+}
+
+static void mlx5st_dealloc_pd(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(dealloc_pd_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(dealloc_pd_in)] = {};
+
+ MLX5_SET(dealloc_pd_in, in, opcode, MLX5_CMD_OP_DEALLOC_PD);
+ MLX5_SET(dealloc_pd_in, in, pd, dev->pdn);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+static void mlx5st_create_mkey(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(create_mkey_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {};
+ struct mlx5_ifc_mkc_bits *mkc;
+
+ MLX5_SET(create_mkey_in, in, opcode, MLX5_CMD_OP_CREATE_MKEY);
+
+ mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
+ MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA);
+ MLX5_SET(mkc, mkc, length64, 1);
+ MLX5_SET(mkc, mkc, pd, dev->pdn);
+ MLX5_SET(mkc, mkc, qpn, 0xffffff);
+ MLX5_SET(mkc, mkc, lr, 1);
+ MLX5_SET(mkc, mkc, lw, 1);
+ MLX5_SET(mkc, mkc, rw, 1);
+ MLX5_SET(mkc, mkc, rr, 1);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->mkey_index = MLX5_GET(create_mkey_out, out, mkey_index);
+ dev->global_lkey = mlx5st_idx_to_mkey(dev->mkey_index);
+ dev->global_rkey = dev->global_lkey;
+
+ dev_dbg(dev->device, "Created global PA-mode MKEY: lkey=0x%x\n",
+ dev->global_lkey);
+}
+
+static void mlx5st_destroy_mkey(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(destroy_mkey_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(destroy_mkey_in)] = {};
+
+ MLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY);
+ MLX5_SET(destroy_mkey_in, in, mkey_index, dev->mkey_index);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * CQ create/destroy
+ */
+
+static void mlx5st_create_cq(struct mlx5st_device *dev)
+{
+ struct vfio_pci_device *device = dev->device;
+ u64 in[MLX5_ST_SZ_QW(create_cq_in) + 1] = {};
+ u32 out[MLX5_ST_SZ_DW(create_cq_out)] = {};
+ struct mlx5_ifc_cqc_bits *cqc;
+ unsigned int i;
+ __be64 *pas;
+
+ /* Initialize CQEs before CREATE_CQ: opcode=0xF, owner=1 */
+ for (i = 0; i < CQ_CQE_CNT; i++) {
+ struct mlx5st_cqe64 *cqe = &dev->cq_buf[i];
+
+ MLX5_SET(cqe64, cqe, opcode, 0xF);
+ MLX5_SET_ONCE(cqe64, cqe, owner, 1);
+ }
+
+ MLX5_SET(create_cq_in, in, opcode, MLX5_CMD_OP_CREATE_CQ);
+
+ cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context);
+ MLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE);
+ MLX5_SET(cqc, cqc, uar_page, dev->uar_page);
+ MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->msi_eqn);
+ MLX5_SET(cqc, cqc, cqe_sz, 0);
+ pas = MLX5_ADDR_OF(create_cq_in, in, pas);
+ MLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev->cq_buf, pas));
+ MLX5_SET(cqc, cqc, log_page_size, 0);
+ MLX5_SET64(cqc, cqc, dbr_addr, to_iova(device, &dev->cq_dbrec));
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->cqn = MLX5_GET(create_cq_out, out, cqn);
+ dev->cq_ci = 0;
+ dev_dbg(device, "Created CQ: cqn=%u, %d entries\n", dev->cqn,
+ CQ_CQE_CNT);
+}
+
+static void mlx5st_destroy_cq(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(destroy_cq_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {};
+
+ MLX5_SET(destroy_cq_in, in, opcode, MLX5_CMD_OP_DESTROY_CQ);
+ MLX5_SET(destroy_cq_in, in, cqn, dev->cqn);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * Arm CQ for event generation. The CQ event delivery state machine is
+ * single-shot: after generating one EQE the CQ enters "Fired" state and
+ * won't generate another until re-armed via ARM_NEXT. Both the CQ doorbell
+ * record and the UAR CQ doorbell register must be written.
+ */
+static void mlx5st_arm_cq(struct mlx5st_device *dev)
+{
+ u32 sn = dev->cq_arm_sn & 3;
+ u32 ci = dev->cq_ci & 0xffffff;
+ u64 doorbell;
+
+ /* Update CQ doorbell record arm word */
+ WRITE_ONCE(dev->cq_dbrec.send_counter,
+ cpu_to_be32(sn << 28 | ci));
+
+ /* Ring CQ doorbell register, iowrite has an internal dma_wmb() */
+ doorbell = ((u64)(sn << 28 | ci) << 32) | dev->cqn;
+ iowrite64be(doorbell,
+ (u8 __iomem *)dev->uar_base + MLX5_CQ_DOORBELL_OFFSET);
+
+ dev->cq_arm_sn++;
+}
+
+/*
+ * QP create/destroy
+ */
+
+static void mlx5st_create_qp(struct mlx5st_device *dev)
+{
+ struct vfio_pci_device *device = dev->device;
+ u64 in[MLX5_ST_SZ_QW(create_qp_in) + 1] = {};
+ u32 out[MLX5_ST_SZ_DW(create_qp_out)] = {};
+ struct mlx5_ifc_qpc_bits *qpc;
+ __be64 *pas;
+
+ MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);
+
+ qpc = MLX5_ADDR_OF(create_qp_in, in, qpc);
+ MLX5_SET(qpc, qpc, st, MLX5_QPC_ST_RC);
+ MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);
+ MLX5_SET(qpc, qpc, pd, dev->pdn);
+ MLX5_SET(qpc, qpc, uar_page, dev->uar_page);
+ MLX5_SET(qpc, qpc, cqn_snd, dev->cqn);
+ MLX5_SET(qpc, qpc, cqn_rcv, dev->cqn);
+ MLX5_SET(qpc, qpc, log_sq_size, LOG_SQ_SIZE);
+ MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg);
+ MLX5_SET(qpc, qpc, rq_type, 0x3);
+ MLX5_SET(qpc, qpc, ts_format, 1);
+ pas = MLX5_ADDR_OF(create_qp_in, in, pas);
+ MLX5_SET(qpc, qpc, page_offset,
+ mlx5st_fill_pas(device, dev->sq_buf, pas));
+ MLX5_SET(qpc, qpc, log_page_size, 0);
+ MLX5_SET64(qpc, qpc, dbr_addr, to_iova(device, &dev->qp_dbrec));
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+
+ dev->qpn = MLX5_GET(create_qp_out, out, qpn);
+ dev->sq_pi = 0;
+ dev_dbg(device, "Created QP: qpn=%u, RC, sq=%d wqes\n", dev->qpn,
+ SQ_WQE_CNT);
+}
+
+static void mlx5st_destroy_qp(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(destroy_qp_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(destroy_qp_in)] = {};
+
+ MLX5_SET(destroy_qp_in, in, opcode, MLX5_CMD_OP_DESTROY_QP);
+ MLX5_SET(destroy_qp_in, in, qpn, dev->qpn);
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
+
+/*
+ * QP state transitions
+ */
+
+static void mlx5st_qp_rst2init(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(rst2init_qp_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {};
+ struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc);
+
+ MLX5_SET(rst2init_qp_in, in, opcode, MLX5_CMD_OP_RST2INIT_QP);
+ MLX5_SET(rst2init_qp_in, in, qpn, dev->qpn);
+
+ MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1);
+ MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED);
+ MLX5_SET(qpc, qpc, rre, 1);
+ MLX5_SET(qpc, qpc, rwe, 1);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "QP RST->INIT\n");
+}
+
+static void mlx5st_qp_init2rtr(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(init2rtr_qp_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {};
+ struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc);
+
+ MLX5_SET(init2rtr_qp_in, in, opcode, MLX5_CMD_OP_INIT2RTR_QP);
+ MLX5_SET(init2rtr_qp_in, in, qpn, dev->qpn);
+
+ MLX5_SET(qpc, qpc, mtu, 3);
+ MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg);
+ MLX5_SET(qpc, qpc, remote_qpn, dev->qpn);
+ MLX5_SET(qpc, qpc, min_rnr_nak, 12);
+ MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1);
+ MLX5_SET(qpc, qpc, primary_address_path.fl, 1);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "QP INIT->RTR (fl=1)\n");
+}
+
+static void mlx5st_qp_rtr2rts(struct mlx5st_device *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(rtr2rts_qp_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {};
+ struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc);
+
+ MLX5_SET(rtr2rts_qp_in, in, opcode, MLX5_CMD_OP_RTR2RTS_QP);
+ MLX5_SET(rtr2rts_qp_in, in, qpn, dev->qpn);
+
+ MLX5_SET(qpc, qpc, log_ack_req_freq, 0);
+ MLX5_SET(qpc, qpc, retry_count, 7);
+ MLX5_SET(qpc, qpc, rnr_retry, 7);
+ MLX5_SET(qpc, qpc, primary_address_path.ack_timeout, 14);
+
+ mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ dev_dbg(dev->device, "QP RTR->RTS\n");
+}
+
+/*
+ * Post RDMA Write WQE
+ */
+static void mlx5st_post_rdma_write(struct mlx5st_device *dev, u64 src_addr,
+ u32 src_lkey, u64 dst_addr, u32 dst_rkey,
+ u32 length, bool signaled)
+{
+ struct mlx5st_send_wqe *wqe;
+ unsigned int idx;
+
+ idx = dev->sq_pi % SQ_WQE_CNT;
+ wqe = &dev->sq_buf[idx];
+
+ memset(wqe, 0, sizeof(*wqe));
+ MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, opcode, MLX5_OPCODE_RDMA_WRITE);
+ MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, wqe_index, dev->sq_pi);
+ MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, qp_or_sq, dev->qpn);
+ MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ds, MLX5_RDMA_WRITE_DS);
+ if (signaled)
+ MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ce, MLX5_WQE_CE_CQE_ALWAYS);
+
+ MLX5_SET64(wqe_raddr_seg, &wqe->raddr, raddr, dst_addr);
+ MLX5_SET(wqe_raddr_seg, &wqe->raddr, rkey, dst_rkey);
+
+ MLX5_SET(wqe_data_seg, &wqe->data, byte_count, length);
+ MLX5_SET(wqe_data_seg, &wqe->data, lkey, src_lkey);
+ MLX5_SET64(wqe_data_seg, &wqe->data, addr, src_addr);
+
+ dev->sq_pi++;
+
+ /* Ensure WQE is visible to device before doorbell record */
+ dma_wmb();
+
+ WRITE_ONCE(dev->qp_dbrec.send_counter,
+ cpu_to_be32(dev->sq_pi & 0xffff));
+
+ /*
+ * Ring doorbell: write first 8 bytes of ctrl to UAR BF register,
+ * iowrite has an internal dma_wmb() so the doorbell record will be
+ * visible.
+ */
+ iowrite64be(be64_to_cpu(*(__be64 *)wqe),
+ (u8 __iomem *)dev->uar_base + dev->uar_bf_offset);
+ dev->uar_bf_offset ^= MLX5_BF_SIZE;
+}
+
+/*
+ * Poll CQ
+ */
+static int mlx5st_poll_cq_batch(struct mlx5st_device *dev,
+ unsigned int max_cqe)
+{
+ unsigned int polled = 0;
+
+ while (polled < max_cqe) {
+ unsigned int idx = dev->cq_ci % CQ_CQE_CNT;
+ struct mlx5st_cqe64 *cqe = &dev->cq_buf[idx];
+ u8 owner, opcode;
+
+ owner = MLX5_GET_ONCE(cqe64, cqe, owner);
+ if (owner != ((dev->cq_ci >> LOG_CQ_SIZE) & 1))
+ break;
+
+ dma_rmb();
+
+ opcode = MLX5_GET(cqe64, cqe, opcode);
+
+ dev->cq_ci++;
+ WRITE_ONCE(dev->cq_dbrec.recv_counter,
+ cpu_to_be32(dev->cq_ci & 0xffffff));
+
+ if (opcode == MLX5_CQE_REQ) {
+ dev->sq_ci =
+ (u16)(MLX5_GET(cqe64, cqe, wqe_counter) + 1);
+ polled++;
+ continue;
+ }
+ if (opcode == MLX5_CQE_REQ_ERR ||
+ opcode == MLX5_CQE_RESP_ERR) {
+ dev_dbg(dev->device,
+ "CQE error: opcode=0x%x syndrome=0x%x vendor=0x%x\n",
+ opcode,
+ MLX5_GET(cqe64, cqe, error_syndrome.syndrome),
+ MLX5_GET(cqe64, cqe,
+ error_syndrome.vendor_error_syndrome));
+ return -1;
+ }
+ dev_err(dev->device, "CQE unexpected opcode=0x%x\n", opcode);
+ return -1;
+ }
+
+ return polled;
+}
+
+static int mlx5st_poll_cq(struct mlx5st_device *dev, unsigned int timeout_ms)
+{
+ struct timespec start, now;
+ unsigned int elapsed;
+ int ret;
+
+ clock_gettime(CLOCK_MONOTONIC, &start);
+ for (;;) {
+ ret = mlx5st_poll_cq_batch(dev, 1);
+ if (ret < 0)
+ return -1;
+ if (ret > 0)
+ return 0;
+
+ if (dev->have_eq)
+ mlx5st_process_events(dev);
+
+ clock_gettime(CLOCK_MONOTONIC, &now);
+ elapsed = (now.tv_sec - start.tv_sec) * 1000 +
+ (now.tv_nsec - start.tv_nsec) / 1000000;
+ if (elapsed > timeout_ms) {
+ dev_err(dev->device, "CQ poll timeout after %u ms\n",
+ timeout_ms);
+ return -1;
+ }
+ }
+}
+
+/*
+ * Data path setup/teardown helpers
+ */
+
+static void mlx5st_setup_datapath(struct mlx5st_device *dev)
+{
+ mlx5st_create_cq(dev);
+ mlx5st_create_qp(dev);
+ mlx5st_qp_rst2init(dev);
+ mlx5st_qp_init2rtr(dev);
+ mlx5st_qp_rtr2rts(dev);
+}
+
+static void mlx5st_teardown_datapath(struct mlx5st_device *dev)
+{
+ if (dev->qpn) {
+ mlx5st_destroy_qp(dev);
+ dev->qpn = 0;
+ }
+ if (dev->cqn) {
+ mlx5st_destroy_cq(dev);
+ dev->cqn = 0;
+ }
+ dev->sq_pi = 0;
+ dev->sq_ci = 0;
+ dev->cq_arm_sn = 0;
+ memset(&dev->qp_dbrec, 0, sizeof(dev->qp_dbrec));
+ memset(&dev->cq_dbrec, 0, sizeof(dev->cq_dbrec));
+}
+
+/*
+ * memcpy callbacks
+ */
+
+#define MLX5ST_MEMCPY_TIMEOUT_MS 60000
+
+static void mlx5st_memcpy_start(struct vfio_pci_device *device,
+ iova_t src, iova_t dst, u64 size, u64 count)
+{
+ struct mlx5st_device *dev = to_mlx5st(device);
+ u64 i;
+
+ for (i = 0; i < count; i++) {
+ bool signaled = (i == count - 1);
+
+ mlx5st_post_rdma_write(dev, src, dev->global_lkey, dst,
+ dev->global_rkey, size, signaled);
+ }
+}
+
+static int mlx5st_memcpy_wait(struct vfio_pci_device *device)
+{
+ struct mlx5st_device *dev = to_mlx5st(device);
+ int ret;
+
+ ret = mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS);
+ if (ret) {
+ /*
+ * CQE error puts the QP in error state. Rebuild the data path
+ * so subsequent operations can succeed.
+ */
+ mlx5st_teardown_datapath(dev);
+ mlx5st_setup_datapath(dev);
+ }
+ return ret;
+}
+
+/*
+ * send_msi callback — trigger CQE -> EQE -> MSI-X via a small RDMA Write.
+ *
+ * Both the CQ and MSI EQ use single-shot arming: the CQ must be armed so the
+ * CQE generates an EQE, and the MSI EQ must be armed so the EQE fires MSI-X.
+ */
+static void mlx5st_send_msi(struct vfio_pci_device *device)
+{
+ struct mlx5st_device *dev = to_mlx5st(device);
+
+ /* Drain accumulated MSI EQ events and re-arm for next interrupt */
+ mlx5st_msi_eq_drain(dev);
+
+ /* Arm CQ so the next CQE generates an EQE on the MSI EQ */
+ mlx5st_arm_cq(dev);
+
+ /* Post a signaled RDMA Write to trigger CQE -> EQE -> MSI-X */
+ mlx5st_post_rdma_write(dev,
+ to_iova(device, &dev->send_msi_src),
+ dev->global_lkey,
+ to_iova(device, &dev->send_msi_dst),
+ dev->global_rkey,
+ sizeof(dev->send_msi_src), true);
+
+ /* Consume the CQE to avoid stale completions */
+ VFIO_ASSERT_EQ(mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS), 0);
+}
+
+/*
+ * Driver ops callbacks
+ */
+
+static void mlx5st_init(struct vfio_pci_device *device)
+{
+ struct mlx5st_device *dev = to_mlx5st(device);
+ iova_t iova_align =
+ device->driver.region.iova % __alignof__(struct mlx5st_device);
+
+ VFIO_ASSERT_GE(device->driver.region.size, sizeof(*dev));
+ VFIO_ASSERT_EQ(iova_align, 0);
+ memset(dev, 0, sizeof(*dev));
+
+ dev->device = device;
+ dev->bar0 = device->bars[0].vaddr;
+ dev->init_seg = dev->bar0;
+
+ vfio_pci_config_writew(device, PCI_COMMAND,
+ vfio_pci_config_readw(device, PCI_COMMAND) |
+ PCI_COMMAND_MASTER);
+
+ mlx5st_wait_fw_init(dev);
+
+ mlx5st_cmd_init(dev);
+ mlx5st_hca_init(dev);
+ mlx5st_query_fl_caps(dev);
+ mlx5st_alloc_pd(dev);
+ mlx5st_create_mkey(dev);
+
+ /* MSI EQ must be created before CQ so CQ can reference its eqn */
+ mlx5st_create_msi_eq(dev);
+ mlx5st_setup_datapath(dev);
+
+ vfio_pci_msix_enable(device, MSI_VECTOR, 1);
+ device->driver.msi = MSI_VECTOR;
+
+ device->driver.max_memcpy_size = 1ULL << dev->log_max_msg;
+ device->driver.max_memcpy_count = SQ_WQE_CNT - 1;
+
+ dev_dbg(device, "mlx5 driver initialized\n");
+}
+
+static void mlx5st_remove(struct vfio_pci_device *device)
+{
+ struct mlx5st_device *dev = to_mlx5st(device);
+
+ vfio_pci_msix_disable(device);
+ mlx5st_teardown_datapath(dev);
+
+ if (dev->have_msi_eq) {
+ mlx5st_destroy_msi_eq(dev);
+ dev->have_msi_eq = false;
+ }
+
+ dev_dbg(device, "teardown: destroy_mkey\n");
+ if (dev->mkey_index) {
+ mlx5st_destroy_mkey(dev);
+ dev->mkey_index = 0;
+ }
+
+ dev_dbg(device, "teardown: dealloc_pd\n");
+ if (dev->pdn) {
+ mlx5st_dealloc_pd(dev);
+ dev->pdn = 0;
+ }
+
+ dev_dbg(device, "teardown: hca_teardown\n");
+ mlx5st_hca_teardown(dev);
+
+ vfio_pci_config_writew(device, PCI_COMMAND,
+ vfio_pci_config_readw(device, PCI_COMMAND) &
+ ~(u16)PCI_COMMAND_MASTER);
+
+ dev_dbg(device, "Teardown complete\n");
+}
+
+struct vfio_pci_driver_ops mlx5st_ops = {
+ .name = "mlx5",
+ .region_size = sizeof(struct mlx5st_device),
+ .probe = mlx5st_probe,
+ .init = mlx5st_init,
+ .remove = mlx5st_remove,
+ .memcpy_start = mlx5st_memcpy_start,
+ .memcpy_wait = mlx5st_memcpy_wait,
+ .send_msi = mlx5st_send_msi,
+};
diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h
new file mode 100644
index 0000000000000..2c451e411ec13
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h
@@ -0,0 +1,114 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * mlx5 VFIO selftest driver - HW definitions
+ *
+ * Typed wrappers, constants, and helpers for programming mlx5 hardware
+ * via the VFIO selftest framework. Most HW constants and all MLX5_SET/GET
+ * macros come from the kernel headers (mlx5_ifc.h, mlx5_ifc_macros.h).
+ */
+#ifndef SELFTESTS_VFIO_MLX5_HW_H
+#define SELFTESTS_VFIO_MLX5_HW_H
+
+#include <linux/io.h>
+#include <linux/build_bug.h>
+#include <vdso/bits.h>
+
+#include "mlx5_ifc.h"
+#include "mlx5_ifc_macros.h"
+
+/*
+ * Typed HW object wrappers for driver region arrays.
+ *
+ * The IFC _bits structs have sizeof == num_bits (not bytes), so they cannot
+ * be used as array elements. These wrappers provide byte-sized types.
+ */
+#define MLX5ST_MAKE_DATA32(name) \
+ struct mlx5st_##name { \
+ u32 data[MLX5_ST_SZ_DW(name)]; \
+ }
+#define MLX5ST_MAKE_DATA64(name) \
+ struct mlx5st_##name { \
+ u64 data[MLX5_ST_SZ_QW(name)]; \
+ }
+
+MLX5ST_MAKE_DATA32(initial_seg);
+MLX5ST_MAKE_DATA64(cmd_queue_entry);
+MLX5ST_MAKE_DATA64(cmd_if_box);
+MLX5ST_MAKE_DATA64(wqe_ctrl_seg);
+MLX5ST_MAKE_DATA64(wqe_raddr_seg);
+MLX5ST_MAKE_DATA64(wqe_data_seg);
+MLX5ST_MAKE_DATA64(cqe64) __aligned(64);
+MLX5ST_MAKE_DATA64(eqe);
+
+/*
+ * Mailbox blocks: 512 data + 64 header = 576 bytes, but the
+ * next_pointer field stores bits [31:10], requiring 1024-byte alignment.
+ */
+#define CMD_MBOX_SIZE (2 * MLX5_HW_PAGE_SIZE)
+#define CMD_MBOX_STRIDE 1024
+#define CMD_MBOX_NENT (CMD_MBOX_SIZE / CMD_MBOX_STRIDE)
+/* Stride-aligned mailbox entry — block + padding to 1024 bytes */
+struct mlx5st_mbox_entry {
+ struct mlx5st_cmd_if_box block;
+} __aligned(CMD_MBOX_STRIDE);
+
+#define MLX5_CMD_INLINE_SZ \
+ MLX5_FLD_SZ_BYTES(cmd_queue_entry, command_input_inline_data)
+
+/* Command interface mailbox block (512 data + 64 header) */
+#define MLX5_CMD_DATA_BLOCK_SIZE MLX5_FLD_SZ_BYTES(cmd_if_box, mailbox_data)
+
+/* RDMA Write WQE — one basic block: ctrl + raddr + data + padding */
+struct mlx5st_send_wqe {
+ struct mlx5st_wqe_ctrl_seg ctrl;
+ struct mlx5st_wqe_raddr_seg raddr;
+ struct mlx5st_wqe_data_seg data;
+} __aligned(64);
+static_assert(sizeof(struct mlx5st_send_wqe) == 64,
+ "send WQE segments must fit in one BB");
+
+/* DS = number of 16-byte segments in the WQE (ctrl + raddr + data) */
+#define MLX5_RDMA_WRITE_DS 3
+
+/* Doorbell record — two __be32 in a 64-byte aligned pair */
+struct mlx5st_dbrec {
+ __be32 recv_counter;
+ __be32 send_counter;
+} __aligned(64);
+
+/* UAR BlueFlame buffer offsets within a UAR page */
+#define MLX5_BF_OFFSET 0x800
+#define MLX5_BF_SIZE 0x100
+
+/* CQ doorbell offset within UAR page */
+#define MLX5_CQ_DOORBELL_OFFSET 0x20
+
+/* EQ doorbell offset within UAR page */
+#define MLX5_EQ_DOORBELL_OFFSET 0x40
+
+#define MLX5_HW_PAGE_SIZE 4096
+
+/*
+ * Test parameters
+ */
+#define SQ_WQE_CNT 16
+#define LOG_SQ_SIZE 4
+#define CQ_CQE_CNT 16
+#define LOG_CQ_SIZE 4
+#define EQ_NENT 64
+#define LOG_EQ_SIZE 6
+#define MSI_EQ_NENT 16
+#define LOG_MSI_EQ_SIZE 4
+#define MSI_VECTOR 0
+
+#define MAX_FW_PAGES 8192
+#define MAX_FW_PAGES_PER_CMD 512
+
+#define MLX5_CMD_TIMEOUT_MS 5000
+
+static inline u32 mlx5st_idx_to_mkey(u32 mkey_idx)
+{
+ return mkey_idx << 8;
+}
+
+#endif /* SELFTESTS_VFIO_MLX5_HW_H */
diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h
new file mode 120000
index 0000000000000..7dcbb79e1af06
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h
@@ -0,0 +1 @@
+../../../../../../../include/linux/mlx5/mlx5_ifc.h
\ No newline at end of file
diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h
new file mode 120000
index 0000000000000..865d99e2aeecd
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h
@@ -0,0 +1 @@
+../../../../../../../include/linux/mlx5/mlx5_ifc_fpga.h
\ No newline at end of file
diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h
new file mode 120000
index 0000000000000..97408c247f06c
--- /dev/null
+++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h
@@ -0,0 +1 @@
+../../../../../../../include/linux/mlx5/mlx5_ifc_macros.h
\ No newline at end of file
diff --git a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c
index c08aa81c44f41..6958877ad4638 100644
--- a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c
+++ b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c
@@ -775,6 +775,7 @@ static int nv_falcon_memcpy_wait(struct vfio_pci_device *device)
const struct vfio_pci_driver_ops nv_falcon_ops = {
.name = "nv_falcon",
+ .region_size = sizeof(struct gpu_device),
.probe = nv_falcon_probe,
.init = nv_falcon_init,
.remove = nv_falcon_remove,
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
index e19bd94b8dd2a..007adc322c1ea 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h
@@ -40,6 +40,17 @@ struct vfio_pci_device {
struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu);
void vfio_pci_device_free(struct vfio_pci_device *device);
+
+#ifdef DEBUG
+#define dev_dbg dev_info
+#else
+#define dev_dbg(_dev, _fmt, ...) \
+ do { \
+ if (0) \
+ dev_info(_dev, _fmt, ##__VA_ARGS__); \
+ } while (0)
+#endif
+
struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu);
void vfio_pci_device_cleanup(struct vfio_pci_device *device);
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h
index e5ada209b1d10..547369c5cff95 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h
@@ -9,6 +9,12 @@ struct vfio_pci_device;
struct vfio_pci_driver_ops {
const char *name;
+ /*
+ * Size of the driver's state structure overlaid on
+ * device->driver.region.vaddr
+ */
+ u64 region_size;
+
/**
* @probe() - Check if the driver supports the given device.
*
diff --git a/tools/testing/selftests/vfio/lib/iova_allocator.c b/tools/testing/selftests/vfio/lib/iova_allocator.c
index 4a660f636f497..a8e5815a1b165 100644
--- a/tools/testing/selftests/vfio/lib/iova_allocator.c
+++ b/tools/testing/selftests/vfio/lib/iova_allocator.c
@@ -13,8 +13,10 @@
#include <linux/iommufd.h>
#include <linux/limits.h>
+#include <linux/log2.h>
#include <linux/mman.h>
#include <linux/overflow.h>
+#include <linux/sizes.h>
#include <linux/types.h>
#include <linux/vfio.h>
@@ -50,7 +52,10 @@ void iova_allocator_cleanup(struct iova_allocator *allocator)
iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)
{
VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n");
- VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n");
+
+ VFIO_ASSERT_LE(size, rounddown_pow_of_two(SIZE_MAX),
+ "Invalid size arg, too large (%zu)\n", size);
+ size = roundup_pow_of_two(size);
for (;;) {
struct iommu_iova_range *range;
diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk
index bcfa74ae040e7..eff95828719e0 100644
--- a/tools/testing/selftests/vfio/lib/libvfio.mk
+++ b/tools/testing/selftests/vfio/lib/libvfio.mk
@@ -15,6 +15,7 @@ LIBVFIO_C += drivers/ioat/ioat.c
LIBVFIO_C += drivers/dsa/dsa.c
endif
+LIBVFIO_C += drivers/mlx5/mlx5.c
LIBVFIO_C += drivers/nv_falcon/nv_falcon.c
LIBVFIO_C += drivers/igb/igb.c
diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c
index 5e65434d2318b..b31abfe2c7ffc 100644
--- a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c
+++ b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c
@@ -6,6 +6,7 @@
extern struct vfio_pci_driver_ops dsa_ops;
extern struct vfio_pci_driver_ops ioat_ops;
#endif
+extern struct vfio_pci_driver_ops mlx5st_ops;
extern struct vfio_pci_driver_ops nv_falcon_ops;
extern struct vfio_pci_driver_ops igb_ops;
@@ -14,6 +15,7 @@ static struct vfio_pci_driver_ops *driver_ops[] = {
&dsa_ops,
&ioat_ops,
#endif
+ &mlx5st_ops,
&nv_falcon_ops,
&igb_ops,
};
@@ -32,6 +34,10 @@ void vfio_pci_driver_probe(struct vfio_pci_device *device)
continue;
device->driver.ops = ops;
+
+ VFIO_ASSERT_NE(ops->region_size, 0);
+ device->driver.region.size =
+ round_up(ops->region_size, getpagesize());
}
}
diff --git a/tools/testing/selftests/vfio/vfio_pci_driver_test.c b/tools/testing/selftests/vfio/vfio_pci_driver_test.c
index 761bf117d624f..2bd949aafd731 100644
--- a/tools/testing/selftests/vfio/vfio_pci_driver_test.c
+++ b/tools/testing/selftests/vfio/vfio_pci_driver_test.c
@@ -87,7 +87,8 @@ FIXTURE_SETUP(vfio_pci_driver_test)
driver = &self->device->driver;
region_setup(self->iommu, self->iova_allocator, &self->memcpy_region, SZ_1G);
- region_setup(self->iommu, self->iova_allocator, &driver->region, SZ_2M);
+ region_setup(self->iommu, self->iova_allocator, &driver->region,
+ driver->region.size);
/* Any IOVA that doesn't overlap memcpy_region and driver->region. */
self->unmapped_iova = iova_allocator_alloc(self->iova_allocator, SZ_1G);
|
| 3/2 | 2026/09/24 00:57 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 49577 output: 30 thoughts: 929 Thoughts: **Analyzing the Patch for Fuzzing Potential** Okay, let's break this patch down and see if it's worth our time with syzkaller. First, I need to get a clear picture of what's been touched. The file list jumps out at me: primarily header files within `include/linux/mlx5/`, along with some modifications under `tools/`. This is interesting. Looking at `include/linux/mlx5/`, it seems like we're dealing with a reorganization and refactoring. It seems that it involves moving around header files and macros; specifically `include/linux/mlx5/cq.h` is removed `enum` values like `MLX5_CQE_REQ`, while `include/linux/mlx5/device.h` has accessor macros moved to a newly added `include/linux/mlx5/mlx5_ifc_macros.h`. Additionally, some enums and structures have been moved to `include/linux/mlx5/mlx5_ifc.h`. It's all about structuring things around for these drivers, for instance defining `mlx5_ifc_wqe_ctrl_seg_bits`, `mlx5_ifc_wqe_raddr_seg_bits`, etc. This looks like some sort of interface code, judging by all the \`ifc\` in the name. Then, `include/linux/mlx5/mlx5_ifc_macros.h` is introduced, and this contains accessor macros. And as for `tools/`, the rest of the changes are under there. But before I get too excited, let's step back and consider rule 2. I'm thinking about the "NEGATIVE CRITERIA" – the things that would automatically make this *not* worth fuzzing. I need to be sure any modifications here don't fall strictly into things that are "Non-kernel and non-executable changes" or "Structurally unreachable hardware." It looks like the changes under `tools/` would likely fall under rule 2: "User-space directories, self-tests, samples, or scripts... that do not affect the compiled kernel image (vmlinux) or kernel modules". Now, what about the changes to `include/linux/mlx5/*`? I need to dig deeper. I need to determine if these header changes are purely refactoring (moving declarations and things around), or if they somehow touch executable kernel code. I need to examine `git diff` more closely or inspect the changes to the `include/linux/mlx5/` files in detail. I'm going to look at the git log or the git diff for these header files. |
| 4/2 | 2026/09/24 00:57 | tool | grepper |
0mError:no matches Args: {
"Expression": "MLX5_GET_ONCE",
"PathPrefix": "drivers/"
}Results: {
"error": "no matches"
} |
| 5/2 | 2026/09/24 00:57 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 51118 output: 30 thoughts: 81 Thoughts: Here's my take on this – a quick mental inventory, you know how it goes. **Quick Scan of mlx5 Related Defines** Okay, so `MLX5_GET_ONCE`… that's only in the selftests for VFIO, specifically within the mlx5 driver testing framework. Interesting, makes me wonder if there's something specific about its usage in that context that's not general across the driver. Let's dig deeper. Gotta check the relatives! I should quickly cross reference that with related macros in that space. I'll need to confirm what's going on with `MLX5_SET_ONCE` and see if there are any obvious connections or patterns there. And while I'm at it, might as well sweep through `MLX5_GET_MMIO`, `MLX5_SET_MMIO`, and `MLX5_ARRAY_GET64` to get the lay of the land and see if there are any obvious relationships to the get/set once calls. No point in missing something! |
| 6/2 | 2026/09/24 00:57 | tool | grepper |
0mError:no matches Args: {
"Expression": "MLX5_SET_MMIO",
"PathPrefix": "drivers/"
}Results: {
"error": "no matches"
} |
| 7/2 | 2026/09/24 00:57 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 51470 output: 125 thoughts: 3673 Thoughts: **Analysis of the Provided Commit** Alright, let's break down this commit. My initial thought is to identify any changes to kernel code, specifically focusing on files ending in `.c` within the `drivers/`, `net/`, or similar directories. I'm examining the diff from the user prompt and it seems the changes are restricted to include files and user-space testing tools. The `diff` output shows modifications across a variety of files. I'm seeing changes in header files (`include/linux/mlx5/*`) and within the `tools/` directory. Notably, the include changes are all within the `mlx5` directory, so they must relate to changes within the same driver subsystem. The `tools/` directory contains various self-tests and utilities for VFIO and KVM, which confirms that these are user-space tools. Looking at the include changes, it appears that the changes involve moving definitions between header files within the `include/linux/mlx5/` directory. - `include/linux/mlx5/cq.h`: Definitions related to completion queue entries have been moved to `include/linux/mlx5/mlx5_ifc.h`. - `include/linux/mlx5/device.h`: This seems to involve macro definitions from `device.h` being moved to `mlx5_ifc_macros.h` which is new. This file also references the new `MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT` macro which is defined in `mlx5_ifc.h`. Some enums are also moved to `mlx5_ifc.h`. - `include/linux/mlx5/mlx5_ifc.h`: This file seems to be accumulating new definitions and structs, as well as enums which were removed from `device.h`. - `include/linux/mlx5/mlx5_ifc_macros.h`: This new file is created to house macros extracted from `device.h` for use by user-space selftests. These changes are clearly refactoring header files, likely to better support the user-space testing within the `tools/` directory. I don't see any executable kernel code being modified here. I'm now checking if any kernel behavior would be changed, even if there are changes inside include files. The code changes are limited to header definitions and macros. This is just a reorganization of header files that does not affect compiled kernel output, with the intention of better support for the userspace tests and drivers. Now, I'm considering the "structurally unreachable hardware" criteria. The mlx5 subsystem involves vendor-specific PCIe cards (ConnectX-4, ConnectX-5, etc.) that are not emulated in standard QEMU. Even if the driver code *were* modified (which it isn't), this hardware is not something I can readily test without the specific hardware. Therefore, this commit falls under the "Non-kernel and non-executable changes" and "Structurally unreachable hardware" negative criteria. Specifically, it involves: 1. User-space code and tools in `tools/` and includes supporting userspace code. 2. Refactoring header definitions, with no compiled kernel code changes. 3. Even if mlx5 driver code *were* modified, mlx5 requires physical ASIC/PCIe cards. This means it's not worth fuzzing. My conclusion is that I should return `WorthFuzzing=false`. The `FocusSymbols` and `EnableConfigs` are not relevant in this case. My reasoning is concise, explaining that the commit is solely related to header refactoring and user-space testing, and that even the involved hardware is un-emulated. I'll double-check this before submitting. It seems that I have addressed all the criteria. I have performed a sanity check of the git diff output. All the files in the diff were examined, and match the conclusions of the above process. I am now confident in calling the set-results function. |
| 8/2 | 2026/09/24 00:58 | tool | set-results |
0mArgs:{
"EnableConfigs": [],
"FocusSymbols": [],
"Reasoning": "The patch series primarily adds an mlx5 selftest driver under tools/testing/selftests/ and modifies userspace test headers under tools/. The only kernel header changes (under include/linux/mlx5/) refactor and move macros, structs, and enum definitions without modifying any executable kernel C code. Furthermore, mlx5 targets vendor-specific Mellanox NICs/HCAs requiring physical PCIe hardware not emulated in standard QEMU environments.",
"WorthFuzzing": false
}Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"Reasoning": "The patch series primarily adds an mlx5 selftest driver under tools/testing/selftests/ and modifies userspace test headers under tools/. The only kernel header changes (under include/linux/mlx5/) refactor and move macros, structs, and enum definitions without modifying any executable kernel C code. Furthermore, mlx5 targets vendor-specific Mellanox NICs/HCAs requiring physical PCIe hardware not emulated in standard QEMU environments.",
"WorthFuzzing": false
} |
| 9/1 | 2026/09/24 00:58 | action | If |
0mArgs:{
"WorthFuzzing": false
}Results: null |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|