| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/08/01 10:16 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": [
"NET_VENDOR_IBM",
"IBMVETH"
],
"FocusSymbols": [
"ibmveth_open",
"ibmveth_close",
"ibmveth_poll",
"ibmveth_start_xmit",
"ibmveth_set_channels",
"ibmveth_probe"
],
"Reasoning": "The patch adds multi-queue RX support to the IBM Power Virtual Ethernet (ibmveth) driver, modifying core driver entry points like open, close, poll, xmit, and ethtool set_channels. This driver is reachable in QEMU when running a PowerPC pseries VM, making it fuzzable.",
"WorthFuzzing": true
} |
| 1/1 | 2026/08/01 10:16 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 92c4ac344e4c00de83a639c43fdc3d7c2d91f40a\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sat Aug 1 10:16:46 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h\nindex dff90a7d7f70b..d79baf2e38273 100644\n--- a/arch/powerpc/include/asm/hvcall.h\n+++ b/arch/powerpc/include/asm/hvcall.h\n@@ -362,7 +362,10 @@\n #define H_GUEST_DELETE\t\t0x488\n #define H_PKS_WRAP_OBJECT\t0x490\n #define H_PKS_UNWRAP_OBJECT\t0x494\n-#define MAX_HCALL_OPCODE\tH_PKS_UNWRAP_OBJECT\n+#define H_REG_LOGICAL_LAN_QUEUE 0x49C\n+#define H_ADD_LOGICAL_LAN_BUFFERS_QUEUE 0x4A0\n+#define H_FREE_LOGICAL_LAN_QUEUE 0x4A8\n+#define MAX_HCALL_OPCODE\tH_FREE_LOGICAL_LAN_QUEUE\n \n /* Scope args for H_SCM_UNBIND_ALL */\n #define H_UNBIND_SCOPE_ALL (0x1)\ndiff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c\nindex 73e051d26b9d8..09e06d86701a5 100644\n--- a/drivers/net/ethernet/ibm/ibmveth.c\n+++ b/drivers/net/ethernet/ibm/ibmveth.c\n@@ -21,6 +21,8 @@\n #include \u003clinux/skbuff.h\u003e\n #include \u003clinux/init.h\u003e\n #include \u003clinux/interrupt.h\u003e\n+#include \u003clinux/irq.h\u003e\n+#include \u003clinux/irqdomain.h\u003e\n #include \u003clinux/mm.h\u003e\n #include \u003clinux/pm.h\u003e\n #include \u003clinux/ethtool.h\u003e\n@@ -28,6 +30,8 @@\n #include \u003clinux/ip.h\u003e\n #include \u003clinux/ipv6.h\u003e\n #include \u003clinux/slab.h\u003e\n+#include \u003clinux/spinlock.h\u003e\n+#include \u003clinux/debugfs.h\u003e\n #include \u003casm/hvcall.h\u003e\n #include \u003clinux/atomic.h\u003e\n #include \u003casm/vio.h\u003e\n@@ -43,7 +47,6 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev);\n \n static struct kobj_type ktype_veth_pool;\n \n-\n static const char ibmveth_driver_name[] = \"ibmveth\";\n static const char ibmveth_driver_string[] = \"IBM Power Virtual Ethernet Driver\";\n #define ibmveth_driver_version \"1.06\"\n@@ -95,49 +98,72 @@ static struct ibmveth_stat ibmveth_stats[] = {\n \t{ \"fw_enabled_ipv6_csum\", IBMVETH_STAT_OFF(fw_ipv6_csum_support) },\n \t{ \"tx_large_packets\", IBMVETH_STAT_OFF(tx_large_packets) },\n \t{ \"rx_large_packets\", IBMVETH_STAT_OFF(rx_large_packets) },\n-\t{ \"fw_enabled_large_send\", IBMVETH_STAT_OFF(fw_large_send_support) }\n+\t{ \"fw_enabled_large_send\", IBMVETH_STAT_OFF(fw_large_send_support) },\n+\t{ \"hcall_reg_lan_queue\", IBMVETH_STAT_OFF(hcall_stats.reg_lan_queue) },\n+\t{ \"hcall_reg_lan\", IBMVETH_STAT_OFF(hcall_stats.reg_lan) },\n+\t{ \"hcall_add_bufs_queue\",\n+\t IBMVETH_STAT_OFF(hcall_stats.add_bufs_queue) },\n+\t{ \"hcall_add_bufs\", IBMVETH_STAT_OFF(hcall_stats.add_bufs) },\n+\t{ \"hcall_add_buf\", IBMVETH_STAT_OFF(hcall_stats.add_buf) },\n+\t{ \"hcall_free_lan_queue\",\n+\t IBMVETH_STAT_OFF(hcall_stats.free_lan_queue) },\n+\t{ \"hcall_free_lan\", IBMVETH_STAT_OFF(hcall_stats.free_lan) },\n+\t{ \"hcall_send_lan\", IBMVETH_STAT_OFF(hcall_stats.send_lan) },\n };\n \n /* simple methods of getting data from the current rxq entry */\n-static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)\n+static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_index)\n {\n-\treturn be32_to_cpu(adapter-\u003erx_queue.queue_addr[adapter-\u003erx_queue.index].flags_off);\n+\tstruct ibmveth_rx_q *rxq = \u0026adapter-\u003erx_queue[queue_index];\n+\n+\treturn be32_to_cpu(rxq-\u003equeue_addr[rxq-\u003eindex].flags_off);\n }\n \n-static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_index)\n {\n-\treturn (ibmveth_rxq_flags(adapter) \u0026 IBMVETH_RXQ_TOGGLE) \u003e\u003e\n-\t\t\tIBMVETH_RXQ_TOGGLE_SHIFT;\n+\treturn (ibmveth_rxq_flags(adapter, queue_index) \u0026 IBMVETH_RXQ_TOGGLE) \u003e\u003e\n+\t\tIBMVETH_RXQ_TOGGLE_SHIFT;\n }\n \n-static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index)\n {\n-\treturn ibmveth_rxq_toggle(adapter) == adapter-\u003erx_queue.toggle;\n+\treturn ibmveth_rxq_toggle(adapter, queue_index) ==\n+\t\tadapter-\u003erx_queue[queue_index].toggle;\n }\n \n-static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index)\n {\n-\treturn ibmveth_rxq_flags(adapter) \u0026 IBMVETH_RXQ_VALID;\n+\treturn ibmveth_rxq_flags(adapter, queue_index) \u0026 IBMVETH_RXQ_VALID;\n }\n \n-static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index)\n {\n-\treturn ibmveth_rxq_flags(adapter) \u0026 IBMVETH_RXQ_OFF_MASK;\n+\treturn ibmveth_rxq_flags(adapter, queue_index) \u0026 IBMVETH_RXQ_OFF_MASK;\n }\n \n-static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index)\n {\n-\treturn ibmveth_rxq_flags(adapter) \u0026 IBMVETH_RXQ_LRG_PKT;\n+\treturn ibmveth_rxq_flags(adapter, queue_index) \u0026 IBMVETH_RXQ_LRG_PKT;\n }\n \n-static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index)\n {\n-\treturn be32_to_cpu(adapter-\u003erx_queue.queue_addr[adapter-\u003erx_queue.index].length);\n+\tstruct ibmveth_rx_q *rxq = \u0026adapter-\u003erx_queue[queue_index];\n+\n+\treturn be32_to_cpu(rxq-\u003equeue_addr[rxq-\u003eindex].length);\n }\n \n-static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)\n+static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,\n+\t\t\t\t\tint queue_index)\n {\n-\treturn ibmveth_rxq_flags(adapter) \u0026 IBMVETH_RXQ_CSUM_GOOD;\n+\treturn ibmveth_rxq_flags(adapter, queue_index) \u0026 IBMVETH_RXQ_CSUM_GOOD;\n }\n \n static unsigned int ibmveth_real_max_tx_queues(void)\n@@ -147,6 +173,560 @@ static unsigned int ibmveth_real_max_tx_queues(void)\n \treturn min(n_cpu, IBMVETH_MAX_QUEUES);\n }\n \n+/**\n+ * ibmveth_alloc_filter_list - Allocate and map filter list\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_alloc_filter_list(struct ibmveth_adapter *adapter)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\n+\tadapter-\u003efilter_list_addr = (void *)get_zeroed_page(GFP_KERNEL);\n+\tif (!adapter-\u003efilter_list_addr) {\n+\t\tnetdev_err(netdev, \"unable to allocate filter pages\\n\");\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\tadapter-\u003efilter_list_dma = dma_map_single(dev,\n+\t\t\t\t\t\t adapter-\u003efilter_list_addr,\n+\t\t\t\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\tif (dma_mapping_error(dev, adapter-\u003efilter_list_dma)) {\n+\t\tnetdev_err(netdev, \"unable to map filter list pages\\n\");\n+\t\tfree_page((unsigned long)adapter-\u003efilter_list_addr);\n+\t\tadapter-\u003efilter_list_addr = NULL;\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\tnetdev_dbg(netdev, \"filter list @ 0x%p (DMA: 0x%llx)\\n\",\n+\t\t adapter-\u003efilter_list_addr,\n+\t\t (unsigned long long)adapter-\u003efilter_list_dma);\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_free_filter_list - Free filter list resources\n+ * @adapter: ibmveth adapter structure\n+ */\n+static void\n+ibmveth_free_filter_list(struct ibmveth_adapter *adapter)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\n+\tif (adapter-\u003efilter_list_dma) {\n+\t\tdma_unmap_single(dev, adapter-\u003efilter_list_dma, 4096,\n+\t\t\t\t DMA_BIDIRECTIONAL);\n+\t\tadapter-\u003efilter_list_dma = 0;\n+\t}\n+\n+\tif (adapter-\u003efilter_list_addr) {\n+\t\tfree_page((unsigned long)adapter-\u003efilter_list_addr);\n+\t\tadapter-\u003efilter_list_addr = NULL;\n+\t}\n+}\n+\n+/**\n+ * ibmveth_alloc_rx_qstats - Allocate per-queue RX statistics\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Return: 0 on success, -ENOMEM on failure\n+ */\n+static int ibmveth_alloc_rx_qstats(struct ibmveth_adapter *adapter)\n+{\n+\tadapter-\u003erx_qstats = kcalloc(IBMVETH_MAX_RX_QUEUES,\n+\t\t\t\t sizeof(*adapter-\u003erx_qstats),\n+\t\t\t\t GFP_KERNEL);\n+\tif (!adapter-\u003erx_qstats)\n+\t\treturn -ENOMEM;\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_free_rx_qstats - Free per-queue RX statistics\n+ * @adapter: ibmveth adapter structure\n+ */\n+static void ibmveth_free_rx_qstats(struct ibmveth_adapter *adapter)\n+{\n+\tkfree(adapter-\u003erx_qstats);\n+\tadapter-\u003erx_qstats = NULL;\n+}\n+\n+/**\n+ * ibmveth_alloc_tx_qstats - Allocate per-queue TX statistics\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Return: 0 on success, -ENOMEM on failure\n+ */\n+static int ibmveth_alloc_tx_qstats(struct ibmveth_adapter *adapter)\n+{\n+\tadapter-\u003etx_qstats = kcalloc(IBMVETH_MAX_QUEUES,\n+\t\t\t\t sizeof(*adapter-\u003etx_qstats),\n+\t\t\t\t GFP_KERNEL);\n+\tif (!adapter-\u003etx_qstats)\n+\t\treturn -ENOMEM;\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_free_tx_qstats - Free per-queue TX statistics\n+ * @adapter: ibmveth adapter structure\n+ */\n+static void ibmveth_free_tx_qstats(struct ibmveth_adapter *adapter)\n+{\n+\tkfree(adapter-\u003etx_qstats);\n+\tadapter-\u003etx_qstats = NULL;\n+}\n+\n+/**\n+ * ibmveth_alloc_rx_queues - Allocate per-queue RX resources\n+ * @adapter: ibmveth adapter structure\n+ * @rxq_entries: Number of entries per RX queue\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i;\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tadapter-\u003ebuffer_list_addr[i] =\n+\t\t\t(void *)get_zeroed_page(GFP_KERNEL);\n+\t\tif (!adapter-\u003ebuffer_list_addr[i]) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"unable to allocate buffer list for queue %d\\n\",\n+\t\t\t\t i);\n+\t\t\tgoto err_cleanup;\n+\t\t}\n+\n+\t\tadapter-\u003erx_queue[i].queue_len =\n+\t\t\tsizeof(struct ibmveth_rx_q_entry) * rxq_entries;\n+\t\tadapter-\u003erx_queue[i].queue_addr =\n+\t\t\tdma_alloc_coherent(dev, adapter-\u003erx_queue[i].queue_len,\n+\t\t\t\t\t \u0026adapter-\u003erx_queue[i].queue_dma,\n+\t\t\t\t\t GFP_KERNEL);\n+\t\tif (!adapter-\u003erx_queue[i].queue_addr) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"unable to allocate RX queue for queue %d\\n\",\n+\t\t\t\t i);\n+\t\t\tgoto err_cleanup;\n+\t\t}\n+\n+\t\tadapter-\u003ebuffer_list_dma[i] =\n+\t\t\tdma_map_single(dev, adapter-\u003ebuffer_list_addr[i],\n+\t\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\t\tif (dma_mapping_error(dev, adapter-\u003ebuffer_list_dma[i])) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"unable to map buffer list for queue %d\\n\",\n+\t\t\t\t i);\n+\t\t\tadapter-\u003ebuffer_list_dma[i] = 0;\n+\t\t\tgoto err_cleanup;\n+\t\t}\n+\n+\t\tadapter-\u003erx_queue[i].index = 0;\n+\t\tadapter-\u003erx_queue[i].num_slots = rxq_entries;\n+\t\tadapter-\u003erx_queue[i].toggle = 1;\n+\t\tspin_lock_init(\u0026adapter-\u003erx_queue[i].replenish_lock);\n+\n+\t\tnetdev_dbg(netdev, \"queue %d: buffer_list @ 0x%p (DMA: 0x%llx), rx_queue @ 0x%p (DMA: 0x%llx), %llu entries\\n\",\n+\t\t\t i, adapter-\u003ebuffer_list_addr[i],\n+\t\t\t (unsigned long long)adapter-\u003ebuffer_list_dma[i],\n+\t\t\t adapter-\u003erx_queue[i].queue_addr,\n+\t\t\t (unsigned long long)adapter-\u003erx_queue[i].queue_dma,\n+\t\t\t (unsigned long long)rxq_entries);\n+\t}\n+\n+\tnetdev_dbg(netdev, \"allocated %d RX queue(s) with %d entries each\\n\",\n+\t\t adapter-\u003enum_rx_queues, rxq_entries);\n+\n+\treturn 0;\n+\n+err_cleanup:\n+\t/* Clean up previously allocated queues */\n+\tfor (; i \u003e= 0; i--) {\n+\t\tif (adapter-\u003ebuffer_list_dma[i]) {\n+\t\t\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma[i],\n+\t\t\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\t\t\tadapter-\u003ebuffer_list_dma[i] = 0;\n+\t\t}\n+\t\tif (adapter-\u003erx_queue[i].queue_addr) {\n+\t\t\tdma_free_coherent(dev, adapter-\u003erx_queue[i].queue_len,\n+\t\t\t\t\t adapter-\u003erx_queue[i].queue_addr,\n+\t\t\t\t\t adapter-\u003erx_queue[i].queue_dma);\n+\t\t\tadapter-\u003erx_queue[i].queue_addr = NULL;\n+\t\t}\n+\t\tif (adapter-\u003ebuffer_list_addr[i]) {\n+\t\t\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr[i]);\n+\t\t\tadapter-\u003ebuffer_list_addr[i] = NULL;\n+\t\t}\n+\t}\n+\n+\treturn -ENOMEM;\n+}\n+\n+/**\n+ * ibmveth_cleanup_rx_resources - Free all RX queue resources\n+ * @adapter: ibmveth adapter structure\n+ */\n+static void\n+ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\tint i;\n+\n+\tnetdev_dbg(adapter-\u003enetdev, \"cleaning up %d RX queue(s)\\n\",\n+\t\t adapter-\u003enum_rx_queues);\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tif (adapter-\u003ebuffer_list_dma[i]) {\n+\t\t\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma[i],\n+\t\t\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\t\t\tadapter-\u003ebuffer_list_dma[i] = 0;\n+\t\t}\n+\n+\t\tif (adapter-\u003erx_queue[i].queue_addr) {\n+\t\t\tdma_free_coherent(dev, adapter-\u003erx_queue[i].queue_len,\n+\t\t\t\t\t adapter-\u003erx_queue[i].queue_addr,\n+\t\t\t\t\t adapter-\u003erx_queue[i].queue_dma);\n+\t\t\tadapter-\u003erx_queue[i].queue_addr = NULL;\n+\t\t}\n+\n+\t\tif (adapter-\u003ebuffer_list_addr[i]) {\n+\t\t\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr[i]);\n+\t\t\tadapter-\u003ebuffer_list_addr[i] = NULL;\n+\t\t}\n+\t}\n+}\n+\n+/**\n+ * ibmveth_toggle_irq - Common helper to enable/disable queue interrupts\n+ * @adapter: ibmveth adapter structure\n+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)\n+ * @enable: true to enable, false to disable\n+ *\n+ * For queue 0 (primary), uses h_vio_signal() as it's registered via\n+ * h_register_logical_lan(). For subordinate queues (1+), uses H_VIOCTL\n+ * with H_ENABLE/DISABLE_VIO_INTERRUPT for per-queue interrupt control.\n+ *\n+ * Return: 0 on success, error code otherwise\n+ */\n+static int\n+ibmveth_toggle_irq(struct ibmveth_adapter *adapter, int queue_index,\n+\t\t bool enable)\n+{\n+\tunsigned long rc;\n+\tunsigned long irq = adapter-\u003equeue_irq[queue_index];\n+\tconst char *action = enable ? \"enable\" : \"disable\";\n+\n+\tif (queue_index == 0) {\n+\t\t/* Primary queue: use h_vio_signal() */\n+\t\trc = h_vio_signal(adapter-\u003evdev-\u003eunit_address,\n+\t\t\t\t enable ? VIO_IRQ_ENABLE : VIO_IRQ_DISABLE);\n+\t} else {\n+\t\t/* Subordinate queues: use H_VIOCTL with hardware IRQ */\n+\t\tstruct irq_data *irq_data = irq_get_irq_data(irq);\n+\t\tirq_hw_number_t hwirq;\n+\t\tu64 vioctl_cmd = enable ? H_ENABLE_VIO_INTERRUPT :\n+\t\t\tH_DISABLE_VIO_INTERRUPT;\n+\n+\t\tif (!irq_data) {\n+\t\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t\t \"Failed to get IRQ data for queue %d (virq=%lu)\\n\",\n+\t\t\t\t queue_index, irq);\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\n+\t\thwirq = irqd_to_hwirq(irq_data);\n+\t\trc = plpar_hcall_norets(H_VIOCTL,\n+\t\t\t\t\tadapter-\u003evdev-\u003eunit_address,\n+\t\t\t\t\tvioctl_cmd,\n+\t\t\t\t\thwirq, 0, 0);\n+\n+\t\tif (rc == H_PARAMETER) {\n+\t\t\t/* H_PARAMETER is non-fatal when IRQ is already in\n+\t\t\t * the requested state.\n+\t\t\t */\n+\t\t\tnetdev_warn_once(adapter-\u003enetdev,\n+\t\t\t\t\t \"H_VIOCTL %s IRQ returned H_PARAMETER for queue %d (hwirq=%lu)\\n\",\n+\t\t\t\t\t action, queue_index, hwirq);\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (rc)\n+\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t \"Failed to %s IRQ for queue %d, rc=%ld\\n\",\n+\t\t\t action, queue_index, rc);\n+\treturn rc;\n+}\n+\n+/**\n+ * ibmveth_disable_irq - Disable interrupt for a specific queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)\n+ *\n+ * Return: 0 on success, error code otherwise\n+ */\n+static int\n+ibmveth_disable_irq(struct ibmveth_adapter *adapter, int queue_index)\n+{\n+\treturn ibmveth_toggle_irq(adapter, queue_index, false);\n+}\n+\n+/**\n+ * ibmveth_enable_irq - Enable interrupt for a specific queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)\n+ *\n+ * Return: 0 on success, error code otherwise\n+ */\n+static int\n+ibmveth_enable_irq(struct ibmveth_adapter *adapter, int queue_index)\n+{\n+\treturn ibmveth_toggle_irq(adapter, queue_index, true);\n+}\n+\n+/**\n+ * ibmveth_dispose_subordinate_irq_mapping - Drop one subordinate virq mapping\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: RX queue index (1..N)\n+ *\n+ * Subordinate queues get mappings from irq_create_mapping() during PHYP\n+ * registration. Queue 0 uses netdev-\u003eirq from device tree and is left alone.\n+ *\n+ * Bound against IBMVETH_MAX_RX_QUEUES, not num_rx_queues: scale-down and\n+ * scale-up fail paths dispose queues that are no longer in the published\n+ * live set but still own a virq in queue_irq[]. The bulk helper still\n+ * iterates only 1..num_rx_queues-1 for close/open-fail cleanup.\n+ *\n+ * Linux virq lifetime is owned by interrupt cleanup helpers. Call this only\n+ * after free_irq() when a handler was installed, or from registration failure\n+ * cleanup before request_irq().\n+ */\n+static void\n+ibmveth_dispose_subordinate_irq_mapping(struct ibmveth_adapter *adapter,\n+\t\t\t\t\tint queue_idx)\n+{\n+\tif (queue_idx \u003c= 0 || queue_idx \u003e= IBMVETH_MAX_RX_QUEUES)\n+\t\treturn;\n+\n+\tif (adapter-\u003equeue_irq[queue_idx]) {\n+\t\tirq_dispose_mapping(adapter-\u003equeue_irq[queue_idx]);\n+\t\tadapter-\u003equeue_irq[queue_idx] = 0;\n+\t}\n+}\n+\n+/**\n+ * ibmveth_dispose_subordinate_irq_mappings - Drop virq mappings for queues 1..N\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Bulk helper for paths that registered subordinate queues but never installed\n+ * Linux IRQ handlers.\n+ */\n+static void\n+ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)\n+{\n+\tint i;\n+\n+\tfor (i = 1; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\tibmveth_dispose_subordinate_irq_mapping(adapter, i);\n+}\n+\n+/**\n+ * ibmveth_setup_rx_interrupts - Register IRQs and enable NAPI\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Registers interrupt handlers for all RX queues, enables NAPI, then\n+ * enables hypervisor interrupt delivery for multi-queue mode after\n+ * every queue has a Linux handler installed. For multi-queue open the\n+ * caller should replenish RX buffers before this helper so traffic\n+ * during open is not dropped (PHYP only interrupts after a successful\n+ * enqueue, which needs buffers). Single-queue open leaves PHYP masked\n+ * here and kicks NAPI afterward (classic path: first poll posts then\n+ * enables).\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i, rc, num = adapter-\u003enum_rx_queues;\n+\n+\tfor (i = 0; i \u003c num; i++) {\n+\t\tif (!adapter-\u003equeue_irq[i]) {\n+\t\t\tnetdev_err(netdev, \"queue %d has invalid IRQ (0)\\n\", i);\n+\t\t\trc = -EINVAL;\n+\t\t\tgoto err_free_irqs;\n+\t\t}\n+\n+\t\trc = request_irq(adapter-\u003equeue_irq[i], ibmveth_interrupt,\n+\t\t\t\t 0, netdev-\u003ename, \u0026adapter-\u003enapi[i]);\n+\t\tif (rc) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"request_irq() failed for irq 0x%x queue %d: %d\\n\",\n+\t\t\t\t adapter-\u003equeue_irq[i], i, rc);\n+\t\t\tgoto err_free_irqs;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i \u003c num; i++)\n+\t\tnapi_enable(\u0026adapter-\u003enapi[i]);\n+\n+\tif (adapter-\u003emulti_queue \u0026\u0026 num \u003e 1) {\n+\t\tfor (i = 0; i \u003c num; i++) {\n+\t\t\trc = ibmveth_enable_irq(adapter, i);\n+\t\t\tif (rc) {\n+\t\t\t\tnetdev_err(netdev,\n+\t\t\t\t\t \"Failed to enable IRQ for queue %d, rc=%d\\n\",\n+\t\t\t\t\t i, rc);\n+\t\t\t\twhile (--i \u003e= 0)\n+\t\t\t\t\tibmveth_disable_irq(adapter, i);\n+\t\t\t\trc = -EIO;\n+\t\t\t\tgoto err_disable_napi;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+\n+err_disable_napi:\n+\t/* PHYP unmask was rolled back above; disable NAPI before free_irq */\n+\tfor (i = 0; i \u003c num; i++)\n+\t\tnapi_disable(\u0026adapter-\u003enapi[i]);\n+\tfor (i = 0; i \u003c num; i++) {\n+\t\tif (adapter-\u003equeue_irq[i])\n+\t\t\tfree_irq(adapter-\u003equeue_irq[i], \u0026adapter-\u003enapi[i]);\n+\t}\n+\tgoto err_dispose_mappings;\n+\n+err_free_irqs:\n+\twhile (--i \u003e= 0)\n+\t\tfree_irq(adapter-\u003equeue_irq[i], \u0026adapter-\u003enapi[i]);\n+err_dispose_mappings:\n+\t/* Both setup failure paths own subordinate virq disposal. */\n+\tibmveth_dispose_subordinate_irq_mappings(adapter);\n+\treturn rc;\n+}\n+\n+/**\n+ * ibmveth_cleanup_rx_interrupts - Mask PHYP, disable NAPI, free IRQs\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Tears down RX interrupt delivery for all queues. Mask PHYP before\n+ * napi_disable so ibmveth_interrupt cannot return IRQ_HANDLED without\n+ * masking (same storm window as scale-down). Safe for close and for\n+ * open failure after setup_rx_interrupts() already unmasked PHYP.\n+ */\n+static void\n+ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tif (adapter-\u003equeue_irq[i]) {\n+\t\t\tibmveth_disable_irq(adapter, i);\n+\t\t\tsynchronize_irq(adapter-\u003equeue_irq[i]);\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\tnapi_disable(\u0026adapter-\u003enapi[i]);\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tif (adapter-\u003equeue_irq[i])\n+\t\t\tfree_irq(adapter-\u003equeue_irq[i], \u0026adapter-\u003enapi[i]);\n+\t}\n+\n+\tibmveth_dispose_subordinate_irq_mappings(adapter);\n+\n+\t/* Queue 0 uses netdev-\u003eirq; leave queue_irq[0] for next open. */\n+}\n+\n+/**\n+ * ibmveth_setup_single_rx_interrupt - Setup interrupt for a single RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to setup\n+ *\n+ * Registers the IRQ handler for one queue. Used during incremental\n+ * scale-up when adding new RX queues. The caller publishes the queue,\n+ * replenishes buffers, enables NAPI, then unmasks PHYP delivery.\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_setup_single_rx_interrupt(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_idx)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint rc;\n+\n+\trc = request_irq(adapter-\u003equeue_irq[queue_idx], ibmveth_interrupt,\n+\t\t\t 0, netdev-\u003ename, \u0026adapter-\u003enapi[queue_idx]);\n+\tif (rc) {\n+\t\tnetdev_err(netdev, \"request_irq() failed for queue %d: %d\\n\",\n+\t\t\t queue_idx, rc);\n+\t\treturn rc;\n+\t}\n+\n+\tnetdev_dbg(netdev, \"Setup IRQ %d for queue %d\\n\",\n+\t\t adapter-\u003equeue_irq[queue_idx], queue_idx);\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_cleanup_single_rx_interrupt - Cleanup interrupt for a single RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to cleanup\n+ *\n+ * Frees the IRQ handler for one queue and releases the subordinate virq\n+ * mapping. Used during incremental scale-down.\n+ */\n+static void\n+ibmveth_cleanup_single_rx_interrupt(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_idx)\n+{\n+\tif (adapter-\u003equeue_irq[queue_idx]) {\n+\t\tfree_irq(adapter-\u003equeue_irq[queue_idx],\n+\t\t\t \u0026adapter-\u003enapi[queue_idx]);\n+\t\tibmveth_dispose_subordinate_irq_mapping(adapter, queue_idx);\n+\t\tnetdev_dbg(adapter-\u003enetdev,\n+\t\t\t \"Freed IRQ for queue %d\\n\", queue_idx);\n+\t}\n+}\n+\n+/**\n+ * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @qindex: RX queue index\n+ *\n+ * Shared by the IRQ handler and process-context kick sites (open, resume,\n+ * pool sysfs, poll_controller).\n+ */\n+static void ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,\n+\t\t\t\t int qindex)\n+{\n+\tstruct napi_struct *napi = \u0026adapter-\u003enapi[qindex];\n+\tunsigned long lpar_rc;\n+\n+\tif (WARN_ON(qindex \u003c 0 || qindex \u003e= adapter-\u003enum_rx_queues))\n+\t\treturn;\n+\n+\tif (napi_schedule_prep(napi)) {\n+\t\tlpar_rc = ibmveth_disable_irq(adapter, qindex);\n+\t\tWARN_ON(lpar_rc != H_SUCCESS);\n+\t\t__napi_schedule(napi);\n+\t}\n+}\n+\n /* setup the initial settings for a buffer pool */\n static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,\n \t\t\t\t u32 pool_index, u32 pool_size,\n@@ -205,11 +785,75 @@ static inline void ibmveth_flush_buffer(void *addr, unsigned long length)\n \t\tasm(\"dcbf %0,%1,1\" :: \"b\" (addr), \"r\" (offset));\n }\n \n+/**\n+ * ibmveth_add_logical_lan_buffers - Add receive buffers to hypervisor\n+ * @adapter: ibmveth adapter structure\n+ * @descs: array of buffer descriptors to add\n+ * @filled: number of valid descriptors in the array\n+ * @buff_size: size of each buffer (multi-queue mode only)\n+ * @queue_index: RX queue index\n+ *\n+ * Return: hypervisor return code\n+ */\n+static long ibmveth_add_logical_lan_buffers(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t union ibmveth_buf_desc *descs,\n+\t\t\t\t\t int filled,\n+\t\t\t\t\t unsigned long buff_size,\n+\t\t\t\t\t int queue_index)\n+{\n+\tstruct vio_dev *vdev = adapter-\u003evdev;\n+\tunsigned long rc;\n+\n+\tif (adapter-\u003emulti_queue) {\n+\t\tunsigned long buffersznum = (buff_size \u003c\u003c 32) | filled;\n+\t\tunsigned long ioba[IBMVETH_MAX_RX_PER_HCALL / 2] = {0};\n+\t\tunsigned long handle = adapter-\u003equeue_handle[queue_index];\n+\t\tint i;\n+\n+\t\t/* Pack descriptor addresses into ioba pairs.\n+\t\t * Each ioba holds two 32-bit addresses packed into 64 bits:\n+\t\t * - Even descriptors (0,2,4...) go in high 32 bits\n+\t\t * - Odd descriptors (1,3,5...) go in low 32 bits\n+\t\t */\n+\t\tfor (i = 0; i \u003c filled \u0026\u0026 i \u003c IBMVETH_MAX_RX_PER_HCALL; i++) {\n+\t\t\tint pair_idx = i / 2;\n+\t\t\tint is_high = (i % 2 == 0);\n+\n+\t\t\tif (is_high)\n+\t\t\t\tioba[pair_idx] = (unsigned long)\n+\t\t\t\t\tdescs[i].fields.address \u003c\u003c 32;\n+\t\t\telse\n+\t\t\t\tioba[pair_idx] |= descs[i].fields.address;\n+\t\t}\n+\n+\t\trc = h_add_logical_lan_buffers_queue(vdev-\u003eunit_address,\n+\t\t\t\t\t\t handle,\n+\t\t\t\t\t\t buffersznum,\n+\t\t\t\t\t\t ioba[0], ioba[1], ioba[2],\n+\t\t\t\t\t\t ioba[3], ioba[4], ioba[5]);\n+\t\tadapter-\u003ehcall_stats.add_bufs_queue++;\n+\t} else if (filled == 1) {\n+\t\trc = h_add_logical_lan_buffer(vdev-\u003eunit_address,\n+\t\t\t\t\t descs[0].desc);\n+\t\tadapter-\u003ehcall_stats.add_buf++;\n+\t} else {\n+\t\trc = h_add_logical_lan_buffers(vdev-\u003eunit_address,\n+\t\t\t\t\t descs[0].desc, descs[1].desc,\n+\t\t\t\t\t descs[2].desc, descs[3].desc,\n+\t\t\t\t\t descs[4].desc, descs[5].desc,\n+\t\t\t\t\t descs[6].desc, descs[7].desc);\n+\t\tadapter-\u003ehcall_stats.add_bufs++;\n+\t}\n+\n+\treturn rc;\n+}\n+\n /* replenish the buffers for a pool. note that we don't need to\n * skb_reserve these since they are used for incoming...\n */\n static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,\n-\t\t\t\t\t struct ibmveth_buff_pool *pool)\n+\t\t\t\t\t struct ibmveth_buff_pool *pool,\n+\t\t\t\t\t int queue_index)\n {\n \tunion ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};\n \tu32 remaining = pool-\u003esize - atomic_read(\u0026pool-\u003eavailable);\n@@ -295,24 +939,16 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,\n \t\tif (!filled)\n \t\t\tbreak;\n \n-\t\t/* single buffer case*/\n-\t\tif (filled == 1)\n-\t\t\tlpar_rc = h_add_logical_lan_buffer(vdev-\u003eunit_address,\n-\t\t\t\t\t\t\t descs[0].desc);\n-\t\telse\n-\t\t\t/* Multi-buffer hcall */\n-\t\t\tlpar_rc = h_add_logical_lan_buffers(vdev-\u003eunit_address,\n-\t\t\t\t\t\t\t descs[0].desc,\n-\t\t\t\t\t\t\t descs[1].desc,\n-\t\t\t\t\t\t\t descs[2].desc,\n-\t\t\t\t\t\t\t descs[3].desc,\n-\t\t\t\t\t\t\t descs[4].desc,\n-\t\t\t\t\t\t\t descs[5].desc,\n-\t\t\t\t\t\t\t descs[6].desc,\n-\t\t\t\t\t\t\t descs[7].desc);\n+\t\tlpar_rc = ibmveth_add_logical_lan_buffers(adapter, descs,\n+\t\t\t\t\t\t\t filled,\n+\t\t\t\t\t\t\t pool-\u003ebuff_size,\n+\t\t\t\t\t\t\t queue_index);\n+\n \t\tif (lpar_rc != H_SUCCESS) {\n \t\t\tdev_warn_ratelimited(dev,\n-\t\t\t\t\t \"RX h_add_logical_lan failed: filled=%u, rc=%lu, batch=%u\\n\",\n+\t\t\t\t\t \"RX h_add_logical_lan %s failed: filled=%u, rc=%lu, batch=%u\\n\",\n+\t\t\t\t\t adapter-\u003emulti_queue ?\n+\t\t\t\t\t \"_queue\" : \"\",\n \t\t\t\t\t filled, lpar_rc, batch);\n \t\t\tgoto hcall_failure;\n \t\t}\n@@ -353,24 +989,23 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,\n \t\t}\n \t\tadapter-\u003ereplenish_add_buff_failure += filled;\n \n-\t\t/*\n-\t\t * If multi rx buffers hcall is no longer supported by FW\n-\t\t * e.g. in the case of Live Partition Migration\n-\t\t */\n-\t\tif (batch \u003e 1 \u0026\u0026 lpar_rc == H_FUNCTION) {\n-\t\t\t/*\n-\t\t\t * Instead of retry submit single buffer individually\n-\t\t\t * here just set the max rx buffer per hcall to 1\n-\t\t\t * buffers will be respleshed next time\n-\t\t\t * when ibmveth_replenish_buffer_pool() is called again\n-\t\t\t * with single-buffer case\n-\t\t\t */\n-\t\t\tnetdev_info(adapter-\u003enetdev,\n-\t\t\t\t \"RX Multi buffers not supported by FW, rc=%lu\\n\",\n-\t\t\t\t lpar_rc);\n-\t\t\tadapter-\u003erx_buffers_per_hcall = 1;\n-\t\t\tnetdev_info(adapter-\u003enetdev,\n-\t\t\t\t \"Next rx replesh will fall back to single-buffer hcall\\n\");\n+\t\tif (lpar_rc == H_FUNCTION) {\n+\t\t\tif (adapter-\u003emulti_queue) {\n+\t\t\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t\t\t \"MQ buffer add H_FUNCTION (q=%d, batch=%d)\\n\",\n+\t\t\t\t\t queue_index, batch);\n+\t\t\t} else if (batch \u003e 1) {\n+\t\t\t\t/*\n+\t\t\t\t * Live Partition Migration may drop multi-\n+\t\t\t\t * buffer support. Fall back to single-buffer\n+\t\t\t\t * on the next replenish; do not continue with\n+\t\t\t\t * a stale local batch size (infinite loop).\n+\t\t\t\t */\n+\t\t\t\tnetdev_warn(adapter-\u003enetdev,\n+\t\t\t\t\t \"Legacy batch add H_FUNCTION (batch=%d), fallback\\n\",\n+\t\t\t\t\t batch);\n+\t\t\t\tadapter-\u003erx_buffers_per_hcall = 1;\n+\t\t\t}\n \t\t}\n \t\tbreak;\n \t}\n@@ -386,27 +1021,51 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,\n */\n static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)\n {\n-\t__be64 *p = adapter-\u003ebuffer_list_addr + 4096 - 8;\n+\tint i;\n \n-\tadapter-\u003erx_no_buffer = be64_to_cpup(p);\n+\tadapter-\u003erx_no_buffer = 0;\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\t__be64 *p = adapter-\u003ebuffer_list_addr[i] + 4096 - 8;\n+\t\tu64 drops = be64_to_cpup(p);\n+\n+\t\tif (adapter-\u003erx_qstats)\n+\t\t\tadapter-\u003erx_qstats[i].no_buffer_drops = drops;\n+\t\tadapter-\u003erx_no_buffer += drops;\n+\t}\n }\n \n /* replenish routine */\n-static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)\n+static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_index)\n {\n+\tstruct ibmveth_rx_q *rxq = \u0026adapter-\u003erx_queue[queue_index];\n+\tunsigned long flags;\n \tint i;\n \n+\tif (queue_index \u003e= adapter-\u003enum_rx_queues) {\n+\t\tnetdev_dbg(adapter-\u003enetdev,\n+\t\t\t \"Skipping replenish for freed queue %d (num_queues=%d)\\n\",\n+\t\t\t queue_index, adapter-\u003enum_rx_queues);\n+\t\treturn;\n+\t}\n+\n \tadapter-\u003ereplenish_task_cycles++;\n \n+\tspin_lock_irqsave(\u0026rxq-\u003ereplenish_lock, flags);\n+\n \tfor (i = (IBMVETH_NUM_BUFF_POOLS - 1); i \u003e= 0; i--) {\n-\t\tstruct ibmveth_buff_pool *pool = \u0026adapter-\u003erx_buff_pool[i];\n+\t\tstruct ibmveth_buff_pool *pool =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[queue_index][i];\n \n-\t\tif (pool-\u003eactive \u0026\u0026\n+\t\tif (pool-\u003eactive \u0026\u0026 pool-\u003efree_map \u0026\u0026\n \t\t (atomic_read(\u0026pool-\u003eavailable) \u003c pool-\u003ethreshold))\n-\t\t\tibmveth_replenish_buffer_pool(adapter, pool);\n+\t\t\tibmveth_replenish_buffer_pool(adapter, pool,\n+\t\t\t\t\t\t queue_index);\n \t}\n \n \tibmveth_update_rx_no_buffer(adapter);\n+\n+\tspin_unlock_irqrestore(\u0026rxq-\u003ereplenish_lock, flags);\n }\n \n /* empty and free ana buffer pool - also used to do cleanup in error paths */\n@@ -443,10 +1102,303 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,\n \t}\n }\n \n+/**\n+ * ibmveth_alloc_queue_buffer_pools - Allocate buffer pools for a single queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue: queue index\n+ *\n+ * Allocates all active buffer pools for the specified queue.\n+ * Pool metadata must be initialized before calling this function.\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int ibmveth_alloc_queue_buffer_pools(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i;\n+\n+\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\tstruct ibmveth_buff_pool *bpool =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[queue][i];\n+\n+\t\tif (!bpool-\u003eactive)\n+\t\t\tcontinue;\n+\n+\t\tif (ibmveth_alloc_buffer_pool(bpool)) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"pool %d/%d alloc failed (size=%u count=%u)\\n\",\n+\t\t\t\t i, queue,\n+\t\t\t\t bpool-\u003ebuff_size,\n+\t\t\t\t bpool-\u003esize);\n+\t\t\tbpool-\u003eactive = 0;\n+\n+\t\t\t/* Free pools allocated so far for this queue */\n+\t\t\twhile (--i \u003e= 0) {\n+\t\t\t\tstruct ibmveth_buff_pool *fpool =\n+\t\t\t\t\t\u0026adapter-\u003erx_buff_pool[queue][i];\n+\n+\t\t\t\tif (fpool-\u003eactive)\n+\t\t\t\t\tibmveth_free_buffer_pool(adapter,\n+\t\t\t\t\t\t\t\t fpool);\n+\t\t\t}\n+\t\t\treturn -ENOMEM;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_free_queue_buffer_pools - Free buffer pools for a single queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue: queue index\n+ *\n+ * Frees all active buffer pools for the specified queue.\n+ */\n+static void ibmveth_free_queue_buffer_pools(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\tstruct ibmveth_buff_pool *pool =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[queue][i];\n+\n+\t\t/* Free pool if it has allocated memory, regardless of\n+\t\t * active flag. Pools may have memory allocated but not\n+\t\t * marked active during queue scale-up, so we must check\n+\t\t * for actual allocations.\n+\t\t */\n+\t\tif (pool-\u003efree_map || pool-\u003edma_addr || pool-\u003eskbuff)\n+\t\t\tibmveth_free_buffer_pool(adapter, pool);\n+\t}\n+}\n+\n+/**\n+ * ibmveth_alloc_buffer_pools - Allocate buffer pools for all queues\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Initializes pool metadata for queues 1-N from queue 0 settings,\n+ * then allocates buffer pools for all queues using the helper function.\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i, q, rc;\n+\n+\t/* Initialize pool metadata for queues 1-15 from queue 0 settings */\n+\tfor (q = 1; q \u003c adapter-\u003enum_rx_queues; q++) {\n+\t\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\t\tstruct ibmveth_buff_pool *src =\n+\t\t\t\t\u0026adapter-\u003erx_buff_pool[0][i];\n+\t\t\tstruct ibmveth_buff_pool *dst =\n+\t\t\t\t\u0026adapter-\u003erx_buff_pool[q][i];\n+\n+\t\t\tdst-\u003esize = src-\u003esize;\n+\t\t\tdst-\u003eindex = src-\u003eindex;\n+\t\t\tdst-\u003ebuff_size = src-\u003ebuff_size;\n+\t\t\tdst-\u003ethreshold = src-\u003ethreshold;\n+\t\t\tdst-\u003eactive = src-\u003eactive;\n+\t\t}\n+\t}\n+\n+\t/* Allocate actual buffers for all queues */\n+\tfor (q = 0; q \u003c adapter-\u003enum_rx_queues; q++) {\n+\t\trc = ibmveth_alloc_queue_buffer_pools(adapter, q);\n+\t\tif (rc) {\n+\t\t\t/* Free pools for all previous queues */\n+\t\t\twhile (--q \u003e= 0)\n+\t\t\t\tibmveth_free_queue_buffer_pools(adapter, q);\n+\t\t\treturn rc;\n+\t\t}\n+\t}\n+\n+\tnetdev_dbg(netdev, \"allocated buffer pools for %d queue(s)\\n\",\n+\t\t adapter-\u003enum_rx_queues);\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_free_buffer_pools - Free buffer pools for all queues\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Frees buffer pools for all queues using the helper function.\n+ */\n+static void\n+ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)\n+{\n+\tint q;\n+\n+\t/* Free buffer pools for all queues */\n+\tfor (q = 0; q \u003c adapter-\u003enum_rx_queues; q++)\n+\t\tibmveth_free_queue_buffer_pools(adapter, q);\n+\n+\tnetdev_dbg(adapter-\u003enetdev, \"freed buffer pools for %d queue(s)\\n\",\n+\t\t adapter-\u003enum_rx_queues);\n+}\n+\n+/**\n+ * ibmveth_alloc_single_rx_queue - Allocate resources for a single RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to allocate\n+ * @rxq_entries: Number of RX queue entries\n+ *\n+ * Allocates buffer list, RX queue, and per-queue buffer pools for one queue.\n+ * Used during incremental scale-up without affecting existing queues.\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_alloc_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx,\n+\t\t\t int rxq_entries)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i, rc = -ENOMEM;\n+\n+\tadapter-\u003ebuffer_list_addr[queue_idx] =\n+\t\t(void *)get_zeroed_page(GFP_KERNEL);\n+\tif (!adapter-\u003ebuffer_list_addr[queue_idx]) {\n+\t\tnetdev_err(netdev, \"unable to allocate buffer list for queue %d\\n\",\n+\t\t\t queue_idx);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\tadapter-\u003erx_queue[queue_idx].queue_len =\n+\t\tsizeof(struct ibmveth_rx_q_entry) * rxq_entries;\n+\tadapter-\u003erx_queue[queue_idx].queue_addr =\n+\t\tdma_alloc_coherent(dev, adapter-\u003erx_queue[queue_idx].queue_len,\n+\t\t\t\t \u0026adapter-\u003erx_queue[queue_idx].queue_dma,\n+\t\t\t\t GFP_KERNEL);\n+\tif (!adapter-\u003erx_queue[queue_idx].queue_addr) {\n+\t\tnetdev_err(netdev, \"unable to allocate RX queue for queue %d\\n\",\n+\t\t\t queue_idx);\n+\t\tgoto out_free_buflist;\n+\t}\n+\n+\tadapter-\u003ebuffer_list_dma[queue_idx] =\n+\t\tdma_map_single(dev, adapter-\u003ebuffer_list_addr[queue_idx],\n+\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\tif (dma_mapping_error(dev, adapter-\u003ebuffer_list_dma[queue_idx])) {\n+\t\tnetdev_err(netdev, \"unable to map buffer list for queue %d\\n\",\n+\t\t\t queue_idx);\n+\t\tgoto out_free_rxq;\n+\t}\n+\n+\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\tstruct ibmveth_buff_pool *src =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[0][i];\n+\t\tstruct ibmveth_buff_pool *dst =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[queue_idx][i];\n+\n+\t\tdst-\u003esize = src-\u003esize;\n+\t\tdst-\u003eindex = src-\u003eindex;\n+\t\tdst-\u003ebuff_size = src-\u003ebuff_size;\n+\t\tdst-\u003ethreshold = src-\u003ethreshold;\n+\t\tdst-\u003eactive = src-\u003eactive;\n+\t}\n+\n+\trc = ibmveth_alloc_queue_buffer_pools(adapter, queue_idx);\n+\tif (rc) {\n+\t\tnetdev_err(netdev,\n+\t\t\t \"Failed to allocate buffer pools for queue %d\\n\",\n+\t\t\t queue_idx);\n+\t\tgoto out_unmap_buflist;\n+\t}\n+\n+\tadapter-\u003erx_queue[queue_idx].index = 0;\n+\tadapter-\u003erx_queue[queue_idx].num_slots = rxq_entries;\n+\tadapter-\u003erx_queue[queue_idx].toggle = 1;\n+\tspin_lock_init(\u0026adapter-\u003erx_queue[queue_idx].replenish_lock);\n+\n+\tnetdev_dbg(netdev,\n+\t\t \"Allocated queue %d: buffer_list @ %p (DMA: 0x%llx), rx_queue @ %p (DMA: 0x%llx), %d entries\\n\",\n+\t\t queue_idx, adapter-\u003ebuffer_list_addr[queue_idx],\n+\t\t (unsigned long long)adapter-\u003ebuffer_list_dma[queue_idx],\n+\t\t adapter-\u003erx_queue[queue_idx].queue_addr,\n+\t\t (unsigned long long)adapter-\u003erx_queue[queue_idx].queue_dma,\n+\t\t rxq_entries);\n+\n+\treturn 0;\n+\n+out_unmap_buflist:\n+\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma[queue_idx],\n+\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\tadapter-\u003ebuffer_list_dma[queue_idx] = 0;\n+out_free_rxq:\n+\tdma_free_coherent(dev, adapter-\u003erx_queue[queue_idx].queue_len,\n+\t\t\t adapter-\u003erx_queue[queue_idx].queue_addr,\n+\t\t\t adapter-\u003erx_queue[queue_idx].queue_dma);\n+\tadapter-\u003erx_queue[queue_idx].queue_addr = NULL;\n+out_free_buflist:\n+\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr[queue_idx]);\n+\tadapter-\u003ebuffer_list_addr[queue_idx] = NULL;\n+\treturn rc;\n+}\n+\n+/**\n+ * ibmveth_free_single_rx_queue - Free resources for a single RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to free\n+ *\n+ * Frees buffer list, RX queue, and per-queue buffer pools for one queue.\n+ * Used during incremental scale-down without affecting remaining queues.\n+ */\n+static void\n+ibmveth_free_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx)\n+{\n+\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n+\n+\tibmveth_free_queue_buffer_pools(adapter, queue_idx);\n+\n+\tif (adapter-\u003ebuffer_list_dma[queue_idx]) {\n+\t\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma[queue_idx],\n+\t\t\t\t 4096, DMA_BIDIRECTIONAL);\n+\t\tadapter-\u003ebuffer_list_dma[queue_idx] = 0;\n+\t}\n+\n+\tif (adapter-\u003erx_queue[queue_idx].queue_addr) {\n+\t\tdma_free_coherent(dev, adapter-\u003erx_queue[queue_idx].queue_len,\n+\t\t\t\t adapter-\u003erx_queue[queue_idx].queue_addr,\n+\t\t\t\t adapter-\u003erx_queue[queue_idx].queue_dma);\n+\t\tadapter-\u003erx_queue[queue_idx].queue_addr = NULL;\n+\t}\n+\n+\tif (adapter-\u003ebuffer_list_addr[queue_idx]) {\n+\t\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr[queue_idx]);\n+\t\tadapter-\u003ebuffer_list_addr[queue_idx] = NULL;\n+\t}\n+\n+\tnetdev_dbg(adapter-\u003enetdev, \"Freed queue %d resources\\n\", queue_idx);\n+}\n+\n+static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_index, u64 correlator)\n+{\n+\tunsigned int pool = correlator \u003e\u003e 32;\n+\tunsigned int index = correlator \u0026 0xffffffffUL;\n+\n+\treturn pool \u003c IBMVETH_NUM_BUFF_POOLS \u0026\u0026\n+\t index \u003c adapter-\u003erx_buff_pool[queue_index][pool].size;\n+}\n+\n+static void ibmveth_rxq_advance(struct ibmveth_rx_q *rxq)\n+{\n+\tif (++rxq-\u003eindex == rxq-\u003enum_slots) {\n+\t\trxq-\u003eindex = 0;\n+\t\trxq-\u003etoggle = !rxq-\u003etoggle;\n+\t}\n+}\n+\n /**\n * ibmveth_remove_buffer_from_pool - remove a buffer from a pool\n * @adapter: adapter instance\n * @correlator: identifies pool and index\n+ * @queue_index: RX queue index (0..num_rx_queues-1)\n * @reuse: whether to reuse buffer\n *\n * Return:\n@@ -455,24 +1407,20 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,\n * * %-EFAULT - pool and index map to null skb\n */\n static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,\n-\t\t\t\t\t u64 correlator, bool reuse)\n+\t\t\t\t\t u64 correlator, int queue_index,\n+\t\t\t\t\t bool reuse)\n {\n \tunsigned int pool = correlator \u003e\u003e 32;\n \tunsigned int index = correlator \u0026 0xffffffffUL;\n \tunsigned int free_index;\n \tstruct sk_buff *skb;\n \n-\tif (WARN_ON(pool \u003e= IBMVETH_NUM_BUFF_POOLS) ||\n-\t WARN_ON(index \u003e= adapter-\u003erx_buff_pool[pool].size)) {\n-\t\tschedule_work(\u0026adapter-\u003ework);\n+\tif (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))\n \t\treturn -EINVAL;\n-\t}\n \n-\tskb = adapter-\u003erx_buff_pool[pool].skbuff[index];\n-\tif (WARN_ON(!skb)) {\n-\t\tschedule_work(\u0026adapter-\u003ework);\n+\tskb = adapter-\u003erx_buff_pool[queue_index][pool].skbuff[index];\n+\tif (!skb)\n \t\treturn -EFAULT;\n-\t}\n \n \t/* if we are going to reuse the buffer then keep the pointers around\n \t * but mark index as available. replenish will see the skb pointer and\n@@ -482,48 +1430,52 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,\n \t\t/* remove the skb pointer to mark free. actual freeing is done\n \t\t * by upper level networking after gro_receive\n \t\t */\n-\t\tadapter-\u003erx_buff_pool[pool].skbuff[index] = NULL;\n+\t\tstruct ibmveth_buff_pool *bpool =\n+\t\t\t\u0026adapter-\u003erx_buff_pool[queue_index][pool];\n+\n+\t\tbpool-\u003eskbuff[index] = NULL;\n \n \t\tdma_unmap_single(\u0026adapter-\u003evdev-\u003edev,\n-\t\t\t\t adapter-\u003erx_buff_pool[pool].dma_addr[index],\n-\t\t\t\t adapter-\u003erx_buff_pool[pool].buff_size,\n+\t\t\t\t bpool-\u003edma_addr[index],\n+\t\t\t\t bpool-\u003ebuff_size,\n \t\t\t\t DMA_FROM_DEVICE);\n \t}\n \n-\tfree_index = adapter-\u003erx_buff_pool[pool].producer_index;\n-\tadapter-\u003erx_buff_pool[pool].producer_index++;\n-\tif (adapter-\u003erx_buff_pool[pool].producer_index \u003e=\n-\t adapter-\u003erx_buff_pool[pool].size)\n-\t\tadapter-\u003erx_buff_pool[pool].producer_index = 0;\n-\tadapter-\u003erx_buff_pool[pool].free_map[free_index] = index;\n+\tfree_index = adapter-\u003erx_buff_pool[queue_index][pool].producer_index;\n+\tadapter-\u003erx_buff_pool[queue_index][pool].producer_index++;\n+\tif (adapter-\u003erx_buff_pool[queue_index][pool].producer_index \u003e=\n+\t adapter-\u003erx_buff_pool[queue_index][pool].size)\n+\t\tadapter-\u003erx_buff_pool[queue_index][pool].producer_index = 0;\n+\tadapter-\u003erx_buff_pool[queue_index][pool].free_map[free_index] = index;\n \n \tmb();\n \n-\tatomic_dec(\u0026(adapter-\u003erx_buff_pool[pool].available));\n+\tatomic_dec(\u0026adapter-\u003erx_buff_pool[queue_index][pool].available);\n \n \treturn 0;\n }\n \n /* get the current buffer on the rx queue */\n-static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter)\n+static inline struct sk_buff *\n+ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,\n+\t\t int queue_index)\n {\n-\tu64 correlator = adapter-\u003erx_queue.queue_addr[adapter-\u003erx_queue.index].correlator;\n+\tstruct ibmveth_rx_q *rxq = \u0026adapter-\u003erx_queue[queue_index];\n+\tu64 correlator = rxq-\u003equeue_addr[rxq-\u003eindex].correlator;\n \tunsigned int pool = correlator \u003e\u003e 32;\n \tunsigned int index = correlator \u0026 0xffffffffUL;\n \n-\tif (WARN_ON(pool \u003e= IBMVETH_NUM_BUFF_POOLS) ||\n-\t WARN_ON(index \u003e= adapter-\u003erx_buff_pool[pool].size)) {\n-\t\tschedule_work(\u0026adapter-\u003ework);\n+\tif (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))\n \t\treturn NULL;\n-\t}\n \n-\treturn adapter-\u003erx_buff_pool[pool].skbuff[index];\n+\treturn adapter-\u003erx_buff_pool[queue_index][pool].skbuff[index];\n }\n \n /**\n * ibmveth_rxq_harvest_buffer - Harvest buffer from pool\n *\n * @adapter: pointer to adapter\n+ * @queue_index: RX queue index to harvest from\n * @reuse: whether to reuse buffer\n *\n * Context: called from ibmveth_poll\n@@ -533,28 +1485,81 @@ static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *ada\n * * other - non-zero return from ibmveth_remove_buffer_from_pool\n */\n static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,\n-\t\t\t\t bool reuse)\n+\t\t\t\t int queue_index, bool reuse)\n {\n+\tstruct ibmveth_rx_q *rxq = \u0026adapter-\u003erx_queue[queue_index];\n \tu64 cor;\n \tint rc;\n \n-\tcor = adapter-\u003erx_queue.queue_addr[adapter-\u003erx_queue.index].correlator;\n-\trc = ibmveth_remove_buffer_from_pool(adapter, cor, reuse);\n-\tif (unlikely(rc))\n+\tcor = rxq-\u003equeue_addr[rxq-\u003eindex].correlator;\n+\trc = ibmveth_remove_buffer_from_pool(adapter, cor, queue_index, reuse);\n+\tif (unlikely(rc)) {\n+\t\tif (rc == -EINVAL || rc == -EFAULT)\n+\t\t\tgoto advance;\n \t\treturn rc;\n-\n-\tif (++adapter-\u003erx_queue.index == adapter-\u003erx_queue.num_slots) {\n-\t\tadapter-\u003erx_queue.index = 0;\n-\t\tadapter-\u003erx_queue.toggle = !adapter-\u003erx_queue.toggle;\n \t}\n \n+advance:\n+\tibmveth_rxq_advance(rxq);\n+\n \treturn 0;\n }\n \n+/**\n+ * ibmveth_drain_rx_queue - Drain pending buffers from an RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_index: Queue index to drain\n+ *\n+ * Recycles all pending buffers back to the per-queue buffer pools.\n+ * Must be called with NAPI disabled for this queue.\n+ *\n+ * Return: Number of buffers drained\n+ */\n+static int\n+ibmveth_drain_rx_queue(struct ibmveth_adapter *adapter, int queue_index)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint drained = 0;\n+\tint limit = adapter-\u003erx_queue[queue_index].num_slots;\n+\tint rc;\n+\n+\tnetdev_dbg(netdev, \"Draining RX queue %d (limit: %d slots)\\n\",\n+\t\t queue_index, limit);\n+\n+\twhile (drained \u003c limit \u0026\u0026\n+\t ibmveth_rxq_pending_buffer(adapter, queue_index)) {\n+\t\t/* Match poll-side order before harvesting completion state. */\n+\t\tsmp_rmb();\n+\t\trc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);\n+\t\tif (rc) {\n+\t\t\tnetdev_err(netdev,\n+\t\t\t\t \"Failed to harvest buffer from queue %d during drain: %d\\n\",\n+\t\t\t\t queue_index, rc);\n+\t\t\tbreak;\n+\t\t}\n+\t\tdrained++;\n+\t}\n+\n+\tif (drained \u003e 0)\n+\t\tnetdev_dbg(netdev, \"Drained %d buffer(s) from RX queue %d\\n\",\n+\t\t\t drained, queue_index);\n+\telse\n+\t\tnetdev_dbg(netdev, \"No buffers to drain from RX queue %d\\n\",\n+\t\t\t queue_index);\n+\n+\treturn drained;\n+}\n+\n static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)\n {\n-\tdma_unmap_single(\u0026adapter-\u003evdev-\u003edev, adapter-\u003etx_ltb_dma[idx],\n-\t\t\t adapter-\u003etx_ltb_size, DMA_TO_DEVICE);\n+\tif (!adapter-\u003etx_ltb_ptr[idx])\n+\t\treturn;\n+\n+\tif (adapter-\u003etx_ltb_dma[idx]) {\n+\t\tdma_unmap_single(\u0026adapter-\u003evdev-\u003edev, adapter-\u003etx_ltb_dma[idx],\n+\t\t\t\t adapter-\u003etx_ltb_size, DMA_TO_DEVICE);\n+\t\tadapter-\u003etx_ltb_dma[idx] = 0;\n+\t}\n \tkfree(adapter-\u003etx_ltb_ptr[idx]);\n \tadapter-\u003etx_ltb_ptr[idx] = NULL;\n }\n@@ -577,16 +1582,63 @@ static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)\n \t\t\t \"unable to DMA map tx long term buffer\\n\");\n \t\tkfree(adapter-\u003etx_ltb_ptr[idx]);\n \t\tadapter-\u003etx_ltb_ptr[idx] = NULL;\n+\t\tadapter-\u003etx_ltb_dma[idx] = 0;\n \t\treturn -ENOMEM;\n \t}\n \n \treturn 0;\n }\n \n+/**\n+ * ibmveth_alloc_tx_resources - Allocate TX resources for all queues\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Allocates TX Long Term Buffers (LTBs) for all TX queues.\n+ *\n+ * Return: 0 on success, -ENOMEM on failure\n+ */\n+static int ibmveth_alloc_tx_resources(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i;\n+\n+\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++) {\n+\t\tif (ibmveth_allocate_tx_ltb(adapter, i))\n+\t\t\tgoto err_free_ltbs;\n+\t}\n+\n+\treturn 0;\n+\n+err_free_ltbs:\n+\twhile (--i \u003e= 0)\n+\t\tibmveth_free_tx_ltb(adapter, i);\n+\treturn -ENOMEM;\n+}\n+\n+/**\n+ * ibmveth_free_tx_resources - Free TX resources for all queues\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Frees TX Long Term Buffers (LTBs) for all TX queues.\n+ */\n+static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tint i;\n+\n+\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++)\n+\t\tibmveth_free_tx_ltb(adapter, i);\n+}\n+\n static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,\n- union ibmveth_buf_desc rxq_desc, u64 mac_address)\n+\t\t\t\t\tunion ibmveth_buf_desc rxq_desc,\n+\t\t\t\t\tu64 mac_address)\n {\n \tint rc, try_again = 1;\n+\tunsigned long ua = adapter-\u003evdev-\u003eunit_address;\n+\tunsigned long buf_dma = adapter-\u003ebuffer_list_dma[0];\n+\tunsigned long filter_dma = adapter-\u003efilter_list_dma;\n+\tu64 *qh0 = \u0026adapter-\u003equeue_handle[0];\n \n \t/*\n \t * After a kexec the adapter will still be open, so our attempt to\n@@ -594,13 +1646,25 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,\n \t * try again, but only once.\n \t */\n retry:\n-\trc = h_register_logical_lan(adapter-\u003evdev-\u003eunit_address,\n-\t\t\t\t adapter-\u003ebuffer_list_dma, rxq_desc.desc,\n-\t\t\t\t adapter-\u003efilter_list_dma, mac_address);\n+\t/* In multi-queue mode, obtain a queue handle for queue 0 so all RX\n+\t * queues can use the same per-queue buffer hypercalls.\n+\t */\n+\tif (adapter-\u003emulti_queue) {\n+\t\trc = h_register_logical_lan_with_handle(ua, buf_dma,\n+\t\t\t\t\t\t\trxq_desc.desc,\n+\t\t\t\t\t\t\tfilter_dma,\n+\t\t\t\t\t\t\tmac_address,\n+\t\t\t\t\t\t\tqh0);\n+\t} else {\n+\t\trc = h_register_logical_lan(ua, buf_dma, rxq_desc.desc,\n+\t\t\t\t\t filter_dma, mac_address);\n+\t}\n+\tadapter-\u003ehcall_stats.reg_lan++;\n \n \tif (rc != H_SUCCESS \u0026\u0026 try_again) {\n \t\tdo {\n \t\t\trc = h_free_logical_lan(adapter-\u003evdev-\u003eunit_address);\n+\t\t\tadapter-\u003ehcall_stats.free_lan++;\n \t\t} while (H_IS_LONG_BUSY(rc) || (rc == H_BUSY));\n \n \t\ttry_again = 0;\n@@ -610,125 +1674,521 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,\n \treturn rc;\n }\n \n+/**\n+ * ibmveth_register_logical_lan_queue - Register subordinate queue with\n+ * hypervisor\n+ * @adapter: ibmveth adapter structure\n+ * @rxq_desc: Receive queue descriptor\n+ * @queue_index: RX queue index (1..N for subordinate queues)\n+ *\n+ * Registers a subordinate receive queue using H_REG_LOGICAL_LAN_QUEUE.\n+ * On success, stores the queue handle and virtual IRQ in the adapter.\n+ * If IRQ mapping fails after a successful hypervisor registration, the\n+ * queue is freed before returning.\n+ *\n+ * Return: H_SUCCESS on success, negative errno on IRQ mapping failure,\n+ * hypervisor error code otherwise\n+ */\n+static int\n+ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,\n+\t\t\t\t union ibmveth_buf_desc rxq_desc,\n+\t\t\t\t int queue_index)\n+{\n+\tunsigned long handle, hwirq;\n+\tunsigned int virq;\n+\tlong lpar_rc;\n+\n+\tnetdev_dbg(adapter-\u003enetdev,\n+\t\t \"Attempting to register queue %d: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\\n\",\n+\t\t queue_index, adapter-\u003evdev-\u003eunit_address,\n+\t\t (unsigned long long)adapter-\u003ebuffer_list_dma[queue_index],\n+\t\t (unsigned long long)rxq_desc.desc);\n+\n+\tlpar_rc = h_reg_logical_lan_queue(adapter-\u003evdev-\u003eunit_address,\n+\t\t\t\t\t adapter-\u003ebuffer_list_dma[queue_index],\n+\t\t\t\t\t rxq_desc.desc, \u0026handle, \u0026hwirq);\n+\tadapter-\u003ehcall_stats.reg_lan_queue++;\n+\n+\tif (lpar_rc == H_SUCCESS) {\n+\t\tvirq = irq_create_mapping(NULL, hwirq);\n+\t\tif (!virq) {\n+\t\t\tunsigned long free_rc;\n+\t\t\tunsigned long ua = adapter-\u003evdev-\u003eunit_address;\n+\n+\t\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t\t \"Failed to map IRQ for queue %d (hwirq=%lu)\\n\",\n+\t\t\t\t queue_index, hwirq);\n+\t\t\tdo {\n+\t\t\t\tfree_rc = h_free_logical_lan_queue(ua, handle);\n+\t\t\t} while (H_IS_LONG_BUSY(free_rc) ||\n+\t\t\t\t (free_rc == H_BUSY));\n+\t\t\tadapter-\u003ehcall_stats.free_lan_queue++;\n+\t\t\tif (free_rc != H_SUCCESS)\n+\t\t\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t\t\t \"h_free_logical_lan_queue failed for queue %d after IRQ map failure: rc=0x%lx\\n\",\n+\t\t\t\t\t queue_index, free_rc);\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\n+\t\tadapter-\u003equeue_handle[queue_index] = handle;\n+\t\tadapter-\u003equeue_irq[queue_index] = virq;\n+\n+\t\tnetdev_dbg(adapter-\u003enetdev,\n+\t\t\t \"queue %d registered: handle=0x%llx irq=%u\\n\",\n+\t\t\t queue_index, adapter-\u003equeue_handle[queue_index],\n+\t\t\t adapter-\u003equeue_irq[queue_index]);\n+\t\treturn H_SUCCESS;\n+\t}\n+\n+\t/*\n+\t * H_FUNCTION means firmware rejected this subordinate register\n+\t * (MQ unsupported). That is a hard open failure: do not clear\n+\t * multi_queue or claim single-queue fallback. Keep a specific\n+\t * log, then the generic failure lines below (no early return).\n+\t */\n+\tif (lpar_rc == H_FUNCTION)\n+\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t \"h_reg_logical_lan_queue H_FUNCTION for queue %d (firmware MQ unsupported)\\n\",\n+\t\t\t queue_index);\n+\n+\tnetdev_err(adapter-\u003enetdev,\n+\t\t \"h_reg_logical_lan_queue failed for queue %d with %ld\\n\",\n+\t\t queue_index, lpar_rc);\n+\tnetdev_err(adapter-\u003enetdev,\n+\t\t \"queue %d params: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\\n\",\n+\t\t queue_index, adapter-\u003evdev-\u003eunit_address,\n+\t\t (unsigned long long)adapter-\u003ebuffer_list_dma[queue_index],\n+\t\t (unsigned long long)rxq_desc.desc);\n+\n+\treturn lpar_rc;\n+}\n+\n+/**\n+ * ibmveth_register_single_rx_queue - Register one subordinate RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to register (1..N)\n+ * @mac_address: MAC address (unused; reserved for API symmetry)\n+ *\n+ * Builds the queue descriptor and registers with the hypervisor via\n+ * ibmveth_register_logical_lan_queue().\n+ *\n+ * Return: 0 on success, -EINVAL if @queue_idx is invalid, -EIO on failure\n+ */\n+static int\n+ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_idx, u64 mac_address)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tunion ibmveth_buf_desc rxq_desc;\n+\tlong lpar_rc;\n+\n+\t(void)mac_address;\n+\n+\tif (WARN_ON(queue_idx \u003c 1 || queue_idx \u003e= IBMVETH_MAX_RX_QUEUES))\n+\t\treturn -EINVAL;\n+\n+\trxq_desc.fields.flags_len = IBMVETH_BUF_VALID |\n+\t\t\t\t adapter-\u003erx_queue[queue_idx].queue_len;\n+\trxq_desc.fields.address = adapter-\u003erx_queue[queue_idx].queue_dma;\n+\n+\tlpar_rc = ibmveth_register_logical_lan_queue(adapter, rxq_desc,\n+\t\t\t\t\t\t queue_idx);\n+\tif (lpar_rc != H_SUCCESS) {\n+\t\tnetdev_err(netdev, \"Failed to register queue %d: rc=0x%lx\\n\",\n+\t\t\t queue_idx, lpar_rc);\n+\t\treturn -EIO;\n+\t}\n+\n+\tnetdev_dbg(netdev, \"Registered queue %d with handle 0x%llx\\n\",\n+\t\t queue_idx, adapter-\u003equeue_handle[queue_idx]);\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * ibmveth_deregister_single_rx_queue - Deregister one subordinate RX queue\n+ * @adapter: ibmveth adapter structure\n+ * @queue_idx: Queue index to deregister (1..N)\n+ *\n+ * Deregisters a single queue via H_FREE_LOGICAL_LAN_QUEUE. Linux IRQ handler\n+ * teardown and subordinate virq mapping disposal are owned by interrupt\n+ * cleanup helpers; queue 0 is freed only through ibmveth_free_all_queues()\n+ * (H_FREE_LOGICAL_LAN).\n+ */\n+static void\n+ibmveth_deregister_single_rx_queue(struct ibmveth_adapter *adapter,\n+\t\t\t\t int queue_idx)\n+{\n+\tunsigned long lpar_rc;\n+\tunsigned long ua = adapter-\u003evdev-\u003eunit_address;\n+\tunsigned long qh = adapter-\u003equeue_handle[queue_idx];\n+\n+\tif (!qh)\n+\t\treturn;\n+\n+\tdo {\n+\t\tlpar_rc = h_free_logical_lan_queue(ua, qh);\n+\t} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));\n+\n+\tadapter-\u003ehcall_stats.free_lan_queue++;\n+\n+\tif (lpar_rc != H_SUCCESS) {\n+\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t \"h_free_logical_lan_queue failed for queue %d: rc=0x%lx\\n\",\n+\t\t\t queue_idx, lpar_rc);\n+\t}\n+\n+\tadapter-\u003equeue_handle[queue_idx] = 0;\n+\n+\tnetdev_dbg(adapter-\u003enetdev, \"Deregistered queue %d\\n\", queue_idx);\n+}\n+\n+/**\n+ * ibmveth_resize_rx_queues_incremental - Resize RX queue count incrementally\n+ * @adapter: ibmveth adapter structure\n+ * @new_count: Target number of RX queues\n+ * @rxq_entries: Number of entries per RX queue\n+ *\n+ * Adds or removes RX queues without tearing down the entire adapter.\n+ * Active queues continue receiving during scale-up; scale-down drains\n+ * excess queues before deregistering them with the hypervisor.\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int\n+ibmveth_resize_rx_queues_incremental(struct ibmveth_adapter *adapter,\n+\t\t\t\t int new_count, int rxq_entries)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tu64 mac_address = ether_addr_to_u64(netdev-\u003edev_addr);\n+\tint old_count = adapter-\u003enum_rx_queues;\n+\tint failed_queue;\n+\tint rc, i;\n+\n+\tif (old_count == new_count) {\n+\t\tnetdev_dbg(netdev, \"RX queue count unchanged (%d), nothing to do\\n\",\n+\t\t\t old_count);\n+\t\treturn 0;\n+\t}\n+\n+\tif (new_count \u003c 1 || new_count \u003e IBMVETH_MAX_RX_QUEUES) {\n+\t\tnetdev_err(netdev, \"Invalid RX queue count %d (must be 1-%d)\\n\",\n+\t\t\t new_count, IBMVETH_MAX_RX_QUEUES);\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tnetdev_info(netdev, \"Incrementally resizing RX queues: %d to %d\\n\",\n+\t\t old_count, new_count);\n+\n+\tif (new_count \u003e old_count) {\n+\t\tnetdev_dbg(netdev, \"Scale-up: adding queues %d-%d\\n\",\n+\t\t\t old_count, new_count - 1);\n+\n+\t\tfor (i = old_count; i \u003c new_count; i++) {\n+\t\t\trc = ibmveth_alloc_single_rx_queue(adapter, i,\n+\t\t\t\t\t\t\t rxq_entries);\n+\t\t\tif (rc) {\n+\t\t\t\tnetdev_err(netdev, \"Failed to allocate queue %d: %d\\n\",\n+\t\t\t\t\t i, rc);\n+\t\t\t\tgoto cleanup_new_queues;\n+\t\t\t}\n+\n+\t\t\trc = ibmveth_register_single_rx_queue(adapter, i,\n+\t\t\t\t\t\t\t mac_address);\n+\t\t\tif (rc) {\n+\t\t\t\tnetdev_err(netdev, \"Failed to register queue %d: %d\\n\",\n+\t\t\t\t\t i, rc);\n+\t\t\t\tibmveth_free_single_rx_queue(adapter, i);\n+\t\t\t\tgoto cleanup_new_queues;\n+\t\t\t}\n+\n+\t\t\trc = ibmveth_setup_single_rx_interrupt(adapter, i);\n+\t\t\tif (rc) {\n+\t\t\t\tnetdev_err(netdev,\n+\t\t\t\t\t \"Failed to setup IRQ for queue %d: %d\\n\",\n+\t\t\t\t\t i, rc);\n+\t\t\t\t/* request_irq failed: mapped but no handler */\n+\t\t\t\tibmveth_dispose_subordinate_irq_mapping(adapter,\n+\t\t\t\t\t\t\t\t\ti);\n+\t\t\t\tibmveth_deregister_single_rx_queue(adapter, i);\n+\t\t\t\tibmveth_free_single_rx_queue(adapter, i);\n+\t\t\t\tgoto cleanup_new_queues;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Fully ready before PHYP delivery, matching open():\n+\t\t\t * publish -\u003e replenish -\u003e napi_enable -\u003e enable_irq.\n+\t\t\t * That way ibmveth_interrupt() cannot run on an\n+\t\t\t * unpublished, empty, or NAPI-disabled queue.\n+\t\t\t */\n+\t\t\tadapter-\u003enum_rx_queues = i + 1;\n+\t\t\tibmveth_replenish_task(adapter, i);\n+\t\t\tnapi_enable(\u0026adapter-\u003enapi[i]);\n+\n+\t\t\trc = ibmveth_enable_irq(adapter, i);\n+\t\t\tif (rc) {\n+\t\t\t\tnetdev_err(netdev,\n+\t\t\t\t\t \"Failed to enable IRQ for queue %d: %d\\n\",\n+\t\t\t\t\t i, rc);\n+\t\t\t\tadapter-\u003enum_rx_queues = i;\n+\t\t\t\tnapi_disable(\u0026adapter-\u003enapi[i]);\n+\t\t\t\tibmveth_cleanup_single_rx_interrupt(adapter, i);\n+\t\t\t\tibmveth_deregister_single_rx_queue(adapter, i);\n+\t\t\t\tibmveth_free_single_rx_queue(adapter, i);\n+\t\t\t\tgoto cleanup_new_queues;\n+\t\t\t}\n+\t\t}\n+\n+\t\trc = netif_set_real_num_rx_queues(netdev, new_count);\n+\t\tif (rc) {\n+\t\t\tnetdev_err(netdev, \"Failed to set real RX queues to %d: %d\\n\",\n+\t\t\t\t new_count, rc);\n+\t\t\tgoto cleanup_new_queues;\n+\t\t}\n+\t} else {\n+\t\tnetdev_dbg(netdev, \"Scale-down: removing queues %d-%d\\n\",\n+\t\t\t new_count, old_count - 1);\n+\n+\t\t/*\n+\t\t * Mask PHYP delivery before napi_disable/drain. Otherwise\n+\t\t * ibmveth_interrupt returns IRQ_HANDLED without masking when\n+\t\t * NAPI is disabled, and the HV can storm during drain.\n+\t\t */\n+\t\tfor (i = new_count; i \u003c old_count; i++) {\n+\t\t\tibmveth_disable_irq(adapter, i);\n+\t\t\tsynchronize_irq(adapter-\u003equeue_irq[i]);\n+\t\t}\n+\n+\t\tfor (i = new_count; i \u003c old_count; i++)\n+\t\t\tnapi_disable(\u0026adapter-\u003enapi[i]);\n+\n+\t\tfor (i = new_count; i \u003c old_count; i++)\n+\t\t\tibmveth_drain_rx_queue(adapter, i);\n+\n+\t\tsynchronize_net();\n+\n+\t\trc = netif_set_real_num_rx_queues(netdev, new_count);\n+\t\tif (rc) {\n+\t\t\tnetdev_err(netdev, \"Failed to set real RX queues to %d: %d\\n\",\n+\t\t\t\t new_count, rc);\n+\t\t\tfor (i = new_count; i \u003c old_count; i++) {\n+\t\t\t\tibmveth_replenish_task(adapter, i);\n+\t\t\t\tnapi_enable(\u0026adapter-\u003enapi[i]);\n+\t\t\t\tibmveth_enable_irq(adapter, i);\n+\t\t\t}\n+\t\t\treturn rc;\n+\t\t}\n+\n+\t\tadapter-\u003enum_rx_queues = new_count;\n+\n+\t\tfor (i = new_count; i \u003c old_count; i++) {\n+\t\t\tibmveth_cleanup_single_rx_interrupt(adapter, i);\n+\t\t\tibmveth_deregister_single_rx_queue(adapter, i);\n+\t\t\tibmveth_free_single_rx_queue(adapter, i);\n+\t\t}\n+\t}\n+\n+\tnetdev_info(netdev, \"Successfully resized to %d RX queues (incremental)\\n\",\n+\t\t adapter-\u003enum_rx_queues);\n+\n+\tif (firmware_has_feature(FW_FEATURE_CMO))\n+\t\tvio_cmo_set_dev_desired(adapter-\u003evdev,\n+\t\t\t\t\tibmveth_get_desired_dma(adapter-\u003evdev));\n+\n+\treturn 0;\n+\n+cleanup_new_queues:\n+\tfailed_queue = i;\n+\tnetdev_err(netdev,\n+\t\t \"Scale-up failed at queue %d, cleaning up queues %d-%d\\n\",\n+\t\t failed_queue, old_count, failed_queue - 1);\n+\tfor (i = old_count; i \u003c failed_queue; i++) {\n+\t\tibmveth_disable_irq(adapter, i);\n+\t\tsynchronize_irq(adapter-\u003equeue_irq[i]);\n+\t}\n+\n+\tfor (i = old_count; i \u003c failed_queue; i++)\n+\t\tnapi_disable(\u0026adapter-\u003enapi[i]);\n+\n+\tfor (i = old_count; i \u003c failed_queue; i++)\n+\t\tibmveth_drain_rx_queue(adapter, i);\n+\n+\tsynchronize_net();\n+\n+\tfor (i = old_count; i \u003c failed_queue; i++) {\n+\t\tibmveth_cleanup_single_rx_interrupt(adapter, i);\n+\t\tibmveth_deregister_single_rx_queue(adapter, i);\n+\t\tibmveth_free_single_rx_queue(adapter, i);\n+\t}\n+\tadapter-\u003enum_rx_queues = old_count;\n+\tnetdev_warn(netdev, \"Keeping %d queues after scale-up failure\\n\",\n+\t\t old_count);\n+\treturn rc;\n+}\n+\n+/**\n+ * ibmveth_free_all_queues - Free all RX queues at once\n+ * @adapter: ibmveth adapter structure\n+ *\n+ * Uses H_FREE_LOGICAL_LAN to free all queues in one hypercall.\n+ * Used during interface close and registration error cleanup.\n+ *\n+ * Clears queue handles only; queue_irq[] is released by\n+ * ibmveth_cleanup_rx_interrupts() on close, or by\n+ * ibmveth_dispose_subordinate_irq_mappings() on partial register failure.\n+ */\n+static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)\n+{\n+\tunsigned long lpar_rc;\n+\tint i;\n+\n+\tnetdev_dbg(adapter-\u003enetdev, \"freeing all RX queues at once\\n\");\n+\n+\tdo {\n+\t\tlpar_rc = h_free_logical_lan(adapter-\u003evdev-\u003eunit_address);\n+\t\tadapter-\u003ehcall_stats.free_lan++;\n+\t} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));\n+\n+\tif (lpar_rc != H_SUCCESS) {\n+\t\tnetdev_err(adapter-\u003enetdev,\n+\t\t\t \"h_free_logical_lan failed: %ld\\n\", lpar_rc);\n+\t}\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\tadapter-\u003equeue_handle[i] = 0;\n+}\n+\n+/**\n+ * ibmveth_register_rx_queues - Register RX queues with hypervisor\n+ * @adapter: ibmveth adapter structure\n+ * @mac_address: MAC address for device registration\n+ *\n+ * Registers queue 0 via ibmveth_register_logical_lan(), then subordinate\n+ * queues 1..N when multi-queue mode is enabled.\n+ *\n+ * Return: 0 on success, -ENONET if queue 0 registration fails, -EIO on\n+ * subordinate queue registration failure\n+ */\n+static int\n+ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tunion ibmveth_buf_desc rxq_desc;\n+\tunsigned long lpar_rc;\n+\tint i, rc;\n+\n+\trxq_desc.fields.flags_len = IBMVETH_BUF_VALID |\n+\t\t\t\t adapter-\u003erx_queue[0].queue_len;\n+\trxq_desc.fields.address = adapter-\u003erx_queue[0].queue_dma;\n+\tadapter-\u003equeue_irq[0] = netdev-\u003eirq;\n+\n+\trc = ibmveth_disable_irq(adapter, 0);\n+\tif (rc != H_SUCCESS)\n+\t\tnetdev_dbg(netdev,\n+\t\t\t \"Failed to disable IRQ for queue 0 before registration, rc=%d\\n\",\n+\t\t\t rc);\n+\n+\tlpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);\n+\tif (lpar_rc != H_SUCCESS) {\n+\t\tnetdev_err(netdev,\n+\t\t\t \"h_register_logical_lan failed: %ld\\n\", lpar_rc);\n+\t\tnetdev_err(netdev,\n+\t\t\t \"buffer TCE:0x%llx filter TCE:0x%llx rxq desc:0x%llx MAC:0x%llx\\n\",\n+\t\t\t adapter-\u003ebuffer_list_dma[0],\n+\t\t\t adapter-\u003efilter_list_dma,\n+\t\t\t rxq_desc.desc, mac_address);\n+\t\treturn -ENONET;\n+\t}\n+\n+\tif (adapter-\u003enum_rx_queues == 1 || !adapter-\u003emulti_queue) {\n+\t\tnetdev_dbg(netdev,\n+\t\t\t \"registered 1 RX queue with hypervisor (single-queue mode)\\n\");\n+\t\treturn 0;\n+\t}\n+\n+\tnetdev_dbg(netdev, \"Registering %d subordinate queues (1-%d)\\n\",\n+\t\t adapter-\u003enum_rx_queues - 1, adapter-\u003enum_rx_queues - 1);\n+\n+\tfor (i = 1; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\trc = ibmveth_register_single_rx_queue(adapter, i, mac_address);\n+\t\tif (rc)\n+\t\t\tgoto err_unregister;\n+\t}\n+\n+\tnetdev_dbg(netdev,\n+\t\t \"registered %d RX queues with hypervisor (multi-queue mode)\\n\",\n+\t\t adapter-\u003enum_rx_queues);\n+\n+\treturn 0;\n+\n+err_unregister:\n+\tibmveth_dispose_subordinate_irq_mappings(adapter);\n+\tibmveth_free_all_queues(adapter);\n+\treturn rc;\n+}\n+\n static int ibmveth_open(struct net_device *netdev)\n {\n \tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n-\tu64 mac_address;\n+\tu64 mac_address = ether_addr_to_u64(netdev-\u003edev_addr);\n \tint rxq_entries = 1;\n-\tunsigned long lpar_rc;\n \tint rc;\n-\tunion ibmveth_buf_desc rxq_desc;\n \tint i;\n-\tstruct device *dev;\n \n \tnetdev_dbg(netdev, \"open starting\\n\");\n \n-\tnapi_enable(\u0026adapter-\u003enapi);\n-\n-\tfor(i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\trxq_entries += adapter-\u003erx_buff_pool[i].size;\n+\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n+\t\trxq_entries += adapter-\u003erx_buff_pool[0][i].size;\n \n-\trc = -ENOMEM;\n-\tadapter-\u003ebuffer_list_addr = (void*) get_zeroed_page(GFP_KERNEL);\n-\tif (!adapter-\u003ebuffer_list_addr) {\n-\t\tnetdev_err(netdev, \"unable to allocate list pages\\n\");\n+\trc = ibmveth_alloc_filter_list(adapter);\n+\tif (rc)\n \t\tgoto out;\n-\t}\n-\n-\tadapter-\u003efilter_list_addr = (void*) get_zeroed_page(GFP_KERNEL);\n-\tif (!adapter-\u003efilter_list_addr) {\n-\t\tnetdev_err(netdev, \"unable to allocate filter pages\\n\");\n-\t\tgoto out_free_buffer_list;\n-\t}\n-\n-\tdev = \u0026adapter-\u003evdev-\u003edev;\n \n-\tadapter-\u003erx_queue.queue_len = sizeof(struct ibmveth_rx_q_entry) *\n-\t\t\t\t\t\trxq_entries;\n-\tadapter-\u003erx_queue.queue_addr =\n-\t\tdma_alloc_coherent(dev, adapter-\u003erx_queue.queue_len,\n-\t\t\t\t \u0026adapter-\u003erx_queue.queue_dma, GFP_KERNEL);\n-\tif (!adapter-\u003erx_queue.queue_addr)\n+\trc = ibmveth_alloc_rx_queues(adapter, rxq_entries);\n+\tif (rc)\n \t\tgoto out_free_filter_list;\n \n-\tadapter-\u003ebuffer_list_dma = dma_map_single(dev,\n-\t\t\tadapter-\u003ebuffer_list_addr, 4096, DMA_BIDIRECTIONAL);\n-\tif (dma_mapping_error(dev, adapter-\u003ebuffer_list_dma)) {\n-\t\tnetdev_err(netdev, \"unable to map buffer list pages\\n\");\n+\trc = ibmveth_alloc_buffer_pools(adapter);\n+\tif (rc)\n \t\tgoto out_free_queue_mem;\n-\t}\n-\n-\tadapter-\u003efilter_list_dma = dma_map_single(dev,\n-\t\t\tadapter-\u003efilter_list_addr, 4096, DMA_BIDIRECTIONAL);\n-\tif (dma_mapping_error(dev, adapter-\u003efilter_list_dma)) {\n-\t\tnetdev_err(netdev, \"unable to map filter list pages\\n\");\n-\t\tgoto out_unmap_buffer_list;\n-\t}\n-\n-\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++) {\n-\t\tif (ibmveth_allocate_tx_ltb(adapter, i))\n-\t\t\tgoto out_free_tx_ltb;\n-\t}\n-\n-\tadapter-\u003erx_queue.index = 0;\n-\tadapter-\u003erx_queue.num_slots = rxq_entries;\n-\tadapter-\u003erx_queue.toggle = 1;\n-\n-\tmac_address = ether_addr_to_u64(netdev-\u003edev_addr);\n \n-\trxq_desc.fields.flags_len = IBMVETH_BUF_VALID |\n-\t\t\t\t\tadapter-\u003erx_queue.queue_len;\n-\trxq_desc.fields.address = adapter-\u003erx_queue.queue_dma;\n-\n-\tnetdev_dbg(netdev, \"buffer list @ 0x%p\\n\", adapter-\u003ebuffer_list_addr);\n-\tnetdev_dbg(netdev, \"filter list @ 0x%p\\n\", adapter-\u003efilter_list_addr);\n-\tnetdev_dbg(netdev, \"receive q @ 0x%p\\n\", adapter-\u003erx_queue.queue_addr);\n-\n-\th_vio_signal(adapter-\u003evdev-\u003eunit_address, VIO_IRQ_DISABLE);\n-\n-\tlpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);\n+\trc = ibmveth_register_rx_queues(adapter, mac_address);\n+\tif (rc)\n+\t\tgoto out_free_buffer_pools;\n \n-\tif (lpar_rc != H_SUCCESS) {\n-\t\tnetdev_err(netdev, \"h_register_logical_lan failed with %ld\\n\",\n-\t\t\t lpar_rc);\n-\t\tnetdev_err(netdev, \"buffer TCE:0x%llx filter TCE:0x%llx rxq \"\n-\t\t\t \"desc:0x%llx MAC:0x%llx\\n\",\n-\t\t\t\t adapter-\u003ebuffer_list_dma,\n-\t\t\t\t adapter-\u003efilter_list_dma,\n-\t\t\t\t rxq_desc.desc,\n-\t\t\t\t mac_address);\n-\t\trc = -ENONET;\n-\t\tgoto out_unmap_filter_list;\n+\trc = netif_set_real_num_rx_queues(netdev, adapter-\u003enum_rx_queues);\n+\tif (rc) {\n+\t\tnetdev_err(netdev, \"failed to set number of rx queues\\n\");\n+\t\tgoto out_unregister_queues;\n \t}\n \n-\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n-\t\tif (!adapter-\u003erx_buff_pool[i].active)\n-\t\t\tcontinue;\n-\t\tif (ibmveth_alloc_buffer_pool(\u0026adapter-\u003erx_buff_pool[i])) {\n-\t\t\tnetdev_err(netdev, \"unable to alloc pool\\n\");\n-\t\t\tadapter-\u003erx_buff_pool[i].active = 0;\n-\t\t\trc = -ENOMEM;\n-\t\t\tgoto out_free_buffer_pools;\n+\t/*\n+\t * MQ: post buffers before setup_rx_interrupts() unmasks PHYP\n+\t * (avoids drops if traffic arrives during open; PHYP allows\n+\t * either order). Single-queue keeps the classic kick: setup\n+\t * (no unmask) then schedule_rx_queue() so the first poll\n+\t * replenishes and enables.\n+\t */\n+\tif (adapter-\u003emulti_queue \u0026\u0026 adapter-\u003enum_rx_queues \u003e 1) {\n+\t\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\t\tnetdev_dbg(netdev,\n+\t\t\t\t \"initial replenish cycle for queue %d\\n\", i);\n+\t\t\tibmveth_replenish_task(adapter, i);\n \t\t}\n \t}\n \n-\tnetdev_dbg(netdev, \"registering irq 0x%x\\n\", netdev-\u003eirq);\n-\trc = request_irq(netdev-\u003eirq, ibmveth_interrupt, 0, netdev-\u003ename,\n-\t\t\t netdev);\n-\tif (rc != 0) {\n-\t\tnetdev_err(netdev, \"unable to request irq 0x%x, rc %d\\n\",\n-\t\t\t netdev-\u003eirq, rc);\n-\t\tdo {\n-\t\t\tlpar_rc = h_free_logical_lan(adapter-\u003evdev-\u003eunit_address);\n-\t\t} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));\n+\trc = ibmveth_setup_rx_interrupts(adapter);\n+\tif (rc)\n+\t\tgoto out_free_all_queues; /* setup already disposed IRQs */\n \n-\t\tgoto out_free_buffer_pools;\n+\tif (!(adapter-\u003emulti_queue \u0026\u0026 adapter-\u003enum_rx_queues \u003e 1)) {\n+\t\tnetdev_dbg(netdev, \"initial replenish cycle\\n\");\n+\t\tibmveth_schedule_rx_queue(adapter, 0);\n \t}\n \n-\trc = -ENOMEM;\n-\n-\tnetdev_dbg(netdev, \"initial replenish cycle\\n\");\n-\tibmveth_interrupt(netdev-\u003eirq, netdev);\n+\trc = ibmveth_alloc_tx_resources(adapter);\n+\tif (rc)\n+\t\tgoto out_cleanup_rx_interrupts;\n \n \tnetif_tx_start_all_queues(netdev);\n \n@@ -736,84 +2196,39 @@ static int ibmveth_open(struct net_device *netdev)\n \n \treturn 0;\n \n+out_cleanup_rx_interrupts:\n+\tibmveth_cleanup_rx_interrupts(adapter);\n+\tgoto out_free_all_queues; /* cleanup already disposed IRQs */\n+out_unregister_queues:\n+\tibmveth_dispose_subordinate_irq_mappings(adapter);\n+out_free_all_queues:\n+\tibmveth_free_all_queues(adapter);\n out_free_buffer_pools:\n-\twhile (--i \u003e= 0) {\n-\t\tif (adapter-\u003erx_buff_pool[i].active)\n-\t\t\tibmveth_free_buffer_pool(adapter,\n-\t\t\t\t\t\t \u0026adapter-\u003erx_buff_pool[i]);\n-\t}\n-out_unmap_filter_list:\n-\tdma_unmap_single(dev, adapter-\u003efilter_list_dma, 4096,\n-\t\t\t DMA_BIDIRECTIONAL);\n-\n-out_free_tx_ltb:\n-\twhile (--i \u003e= 0) {\n-\t\tibmveth_free_tx_ltb(adapter, i);\n-\t}\n-\n-out_unmap_buffer_list:\n-\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma, 4096,\n-\t\t\t DMA_BIDIRECTIONAL);\n+\tibmveth_free_buffer_pools(adapter);\n out_free_queue_mem:\n-\tdma_free_coherent(dev, adapter-\u003erx_queue.queue_len,\n-\t\t\t adapter-\u003erx_queue.queue_addr,\n-\t\t\t adapter-\u003erx_queue.queue_dma);\n+\tibmveth_cleanup_rx_resources(adapter);\n out_free_filter_list:\n-\tfree_page((unsigned long)adapter-\u003efilter_list_addr);\n-out_free_buffer_list:\n-\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr);\n+\tibmveth_free_filter_list(adapter);\n out:\n-\tnapi_disable(\u0026adapter-\u003enapi);\n \treturn rc;\n }\n \n static int ibmveth_close(struct net_device *netdev)\n {\n \tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n-\tstruct device *dev = \u0026adapter-\u003evdev-\u003edev;\n-\tlong lpar_rc;\n-\tint i;\n \n \tnetdev_dbg(netdev, \"close starting\\n\");\n \n-\tnapi_disable(\u0026adapter-\u003enapi);\n-\n \tnetif_tx_stop_all_queues(netdev);\n \n-\th_vio_signal(adapter-\u003evdev-\u003eunit_address, VIO_IRQ_DISABLE);\n-\n-\tdo {\n-\t\tlpar_rc = h_free_logical_lan(adapter-\u003evdev-\u003eunit_address);\n-\t} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));\n-\n-\tif (lpar_rc != H_SUCCESS) {\n-\t\tnetdev_err(netdev, \"h_free_logical_lan failed with %lx, \"\n-\t\t\t \"continuing with close\\n\", lpar_rc);\n-\t}\n-\n-\tfree_irq(netdev-\u003eirq, netdev);\n-\n+\t/* PHYP mask + napi_disable + free_irq live in cleanup_rx_interrupts */\n+\tibmveth_free_tx_resources(adapter);\n+\tibmveth_cleanup_rx_interrupts(adapter);\n \tibmveth_update_rx_no_buffer(adapter);\n-\n-\tdma_unmap_single(dev, adapter-\u003ebuffer_list_dma, 4096,\n-\t\t\t DMA_BIDIRECTIONAL);\n-\tfree_page((unsigned long)adapter-\u003ebuffer_list_addr);\n-\n-\tdma_unmap_single(dev, adapter-\u003efilter_list_dma, 4096,\n-\t\t\t DMA_BIDIRECTIONAL);\n-\tfree_page((unsigned long)adapter-\u003efilter_list_addr);\n-\n-\tdma_free_coherent(dev, adapter-\u003erx_queue.queue_len,\n-\t\t\t adapter-\u003erx_queue.queue_addr,\n-\t\t\t adapter-\u003erx_queue.queue_dma);\n-\n-\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\tif (adapter-\u003erx_buff_pool[i].active)\n-\t\t\tibmveth_free_buffer_pool(adapter,\n-\t\t\t\t\t\t \u0026adapter-\u003erx_buff_pool[i]);\n-\n-\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++)\n-\t\tibmveth_free_tx_ltb(adapter, i);\n+\tibmveth_free_all_queues(adapter);\n+\tibmveth_free_buffer_pools(adapter);\n+\tibmveth_cleanup_rx_resources(adapter);\n+\tibmveth_free_filter_list(adapter);\n \n \tnetdev_dbg(netdev, \"close complete\\n\");\n \n@@ -1073,22 +2488,136 @@ static int ibmveth_set_features(struct net_device *dev,\n \treturn rc1 ? rc1 : rc2;\n }\n \n+/*\n+ * Sum per-queue counters for rare ethtool reads. Do not write adapter\n+ * globals on the hot path (ibmvnic-style); with qstats allocated for the\n+ * adapter lifetime, these sums remain meaningful across ifdown/up.\n+ */\n+static u64 ibmveth_sum_rx_invalid_buffers(struct ibmveth_adapter *adapter)\n+{\n+\tu64 total = 0;\n+\tint i;\n+\n+\tif (!adapter-\u003erx_qstats)\n+\t\treturn adapter-\u003erx_invalid_buffer;\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\ttotal += adapter-\u003erx_qstats[i].invalid_buffers;\n+\n+\treturn total;\n+}\n+\n+static u64 ibmveth_sum_rx_large_packets(struct ibmveth_adapter *adapter)\n+{\n+\tu64 total = 0;\n+\tint i;\n+\n+\tif (!adapter-\u003erx_qstats)\n+\t\treturn adapter-\u003erx_large_packets;\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\ttotal += adapter-\u003erx_qstats[i].large_packets;\n+\n+\treturn total;\n+}\n+\n+static u64 ibmveth_sum_tx_large_packets(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tu64 total = 0;\n+\tint i;\n+\n+\tif (!adapter-\u003etx_qstats)\n+\t\treturn adapter-\u003etx_large_packets;\n+\n+\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++)\n+\t\ttotal += adapter-\u003etx_qstats[i].large_packets;\n+\n+\treturn total;\n+}\n+\n+static u64 ibmveth_sum_tx_send_failed(struct ibmveth_adapter *adapter)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tu64 total = 0;\n+\tint i;\n+\n+\tif (!adapter-\u003etx_qstats)\n+\t\treturn adapter-\u003etx_send_failed;\n+\n+\tfor (i = 0; i \u003c netdev-\u003ereal_num_tx_queues; i++)\n+\t\ttotal += adapter-\u003etx_qstats[i].send_failures;\n+\n+\treturn total;\n+}\n+\n+static u64 ibmveth_ethtool_adapter_stat(struct ibmveth_adapter *adapter,\n+\t\t\t\t\tint index)\n+{\n+\tunsigned long offset = ibmveth_stats[index].offset;\n+\n+\tif (offset == IBMVETH_STAT_OFF(rx_invalid_buffer))\n+\t\treturn ibmveth_sum_rx_invalid_buffers(adapter);\n+\tif (offset == IBMVETH_STAT_OFF(rx_large_packets))\n+\t\treturn ibmveth_sum_rx_large_packets(adapter);\n+\tif (offset == IBMVETH_STAT_OFF(tx_large_packets))\n+\t\treturn ibmveth_sum_tx_large_packets(adapter);\n+\tif (offset == IBMVETH_STAT_OFF(tx_send_failed))\n+\t\treturn ibmveth_sum_tx_send_failed(adapter);\n+\n+\treturn IBMVETH_GET_STAT(adapter, offset);\n+}\n+\n static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)\n {\n+\tstruct ibmveth_adapter *adapter = netdev_priv(dev);\n+\tu8 *p = data;\n \tint i;\n \n \tif (stringset != ETH_SS_STATS)\n \t\treturn;\n \n-\tfor (i = 0; i \u003c ARRAY_SIZE(ibmveth_stats); i++, data += ETH_GSTRING_LEN)\n-\t\tmemcpy(data, ibmveth_stats[i].name, ETH_GSTRING_LEN);\n+\tfor (i = 0; i \u003c ARRAY_SIZE(ibmveth_stats); i++) {\n+\t\tmemcpy(p, ibmveth_stats[i].name, ETH_GSTRING_LEN);\n+\t\tp += ETH_GSTRING_LEN;\n+\t}\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tethtool_sprintf(\u0026p, \"rx%d_packets\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_bytes\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_interrupts\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_polls\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_large_packets\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_invalid_buffers\", i);\n+\t\tethtool_sprintf(\u0026p, \"rx%d_no_buffer_drops\", i);\n+\t}\n+\n+\tfor (i = 0; i \u003c dev-\u003ereal_num_tx_queues; i++) {\n+\t\tethtool_sprintf(\u0026p, \"tx%d_packets\", i);\n+\t\tethtool_sprintf(\u0026p, \"tx%d_bytes\", i);\n+\t\tethtool_sprintf(\u0026p, \"tx%d_large_packets\", i);\n+\t\tethtool_sprintf(\u0026p, \"tx%d_dropped_packets\", i);\n+\t\tethtool_sprintf(\u0026p, \"tx%d_send_failures\", i);\n+\t\tethtool_sprintf(\u0026p, \"tx%d_checksum_offload\", i);\n+\t}\n+\n+\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\tethtool_sprintf(\u0026p, \"pool%d_size\", i);\n+\t\tethtool_sprintf(\u0026p, \"pool%d_active\", i);\n+\t\tethtool_sprintf(\u0026p, \"pool%d_available\", i);\n+\t}\n }\n \n static int ibmveth_get_sset_count(struct net_device *dev, int sset)\n {\n+\tstruct ibmveth_adapter *adapter = netdev_priv(dev);\n+\n \tswitch (sset) {\n \tcase ETH_SS_STATS:\n-\t\treturn ARRAY_SIZE(ibmveth_stats);\n+\t\treturn ARRAY_SIZE(ibmveth_stats) +\n+\t\t adapter-\u003enum_rx_queues * IBMVETH_NUM_RX_QSTATS +\n+\t\t dev-\u003ereal_num_tx_queues * IBMVETH_NUM_TX_QSTATS +\n+\t\t IBMVETH_NUM_BUFF_POOLS * 3;\n \tdefault:\n \t\treturn -EOPNOTSUPP;\n \t}\n@@ -1097,44 +2626,154 @@ static int ibmveth_get_sset_count(struct net_device *dev, int sset)\n static void ibmveth_get_ethtool_stats(struct net_device *dev,\n \t\t\t\t struct ethtool_stats *stats, u64 *data)\n {\n-\tint i;\n \tstruct ibmveth_adapter *adapter = netdev_priv(dev);\n+\tint i, j;\n \n \tfor (i = 0; i \u003c ARRAY_SIZE(ibmveth_stats); i++)\n-\t\tdata[i] = IBMVETH_GET_STAT(adapter, ibmveth_stats[i].offset);\n+\t\tdata[i] = ibmveth_ethtool_adapter_stat(adapter, i);\n+\n+\tfor (j = 0; j \u003c adapter-\u003enum_rx_queues; j++) {\n+\t\tif (adapter-\u003erx_qstats) {\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].packets;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].bytes;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].interrupts;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].polls;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].large_packets;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].invalid_buffers;\n+\t\t\tdata[i++] = adapter-\u003erx_qstats[j].no_buffer_drops;\n+\t\t} else {\n+\t\t\ti += IBMVETH_NUM_RX_QSTATS;\n+\t\t}\n+\t}\n+\n+\tfor (j = 0; j \u003c dev-\u003ereal_num_tx_queues; j++) {\n+\t\tif (adapter-\u003etx_qstats) {\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].packets;\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].bytes;\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].large_packets;\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].dropped_packets;\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].send_failures;\n+\t\t\tdata[i++] = adapter-\u003etx_qstats[j].checksum_offload;\n+\t\t} else {\n+\t\t\ti += IBMVETH_NUM_TX_QSTATS;\n+\t\t}\n+\t}\n+\n+\tfor (j = 0; j \u003c IBMVETH_NUM_BUFF_POOLS; j++) {\n+\t\tdata[i++] = adapter-\u003erx_buff_pool[0][j].size;\n+\t\tdata[i++] = adapter-\u003erx_buff_pool[0][j].active;\n+\t\tdata[i++] = atomic_read(\u0026adapter-\u003erx_buff_pool[0][j].available);\n+\t}\n }\n \n static void ibmveth_get_channels(struct net_device *netdev,\n \t\t\t\t struct ethtool_channels *channels)\n {\n+\tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n+\n \tchannels-\u003emax_tx = ibmveth_real_max_tx_queues();\n \tchannels-\u003etx_count = netdev-\u003ereal_num_tx_queues;\n \n-\tchannels-\u003emax_rx = netdev-\u003ereal_num_rx_queues;\n-\tchannels-\u003erx_count = netdev-\u003ereal_num_rx_queues;\n+\tif (adapter-\u003emulti_queue)\n+\t\tchannels-\u003emax_rx = IBMVETH_MAX_RX_QUEUES;\n+\telse\n+\t\tchannels-\u003emax_rx = 1;\n+\tchannels-\u003erx_count = adapter-\u003enum_rx_queues;\n+}\n+\n+/**\n+ * ibmveth_resize_rx_channels - Validate and apply a new RX queue count\n+ * @adapter: ibmveth adapter structure\n+ * @goal_rx: desired RX queue count\n+ *\n+ * When the interface is up, resize live queues via\n+ * ibmveth_resize_rx_queues_incremental(). When down, only stash\n+ * adapter-\u003enum_rx_queues for the next open().\n+ *\n+ * Return: 0 on success, negative error code on failure\n+ */\n+static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,\n+\t\t\t\t unsigned int goal_rx)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tunsigned int old_rx = adapter-\u003enum_rx_queues;\n+\tint rxq_entries;\n+\tint rc;\n+\n+\tif (goal_rx \u003e 1 \u0026\u0026 !adapter-\u003emulti_queue) {\n+\t\tnetdev_err(netdev,\n+\t\t\t \"Cannot resize to %u RX queues: multi-queue mode not supported by firmware\\n\",\n+\t\t\t goal_rx);\n+\t\treturn -EOPNOTSUPP;\n+\t}\n+\n+\tif (goal_rx \u003c 1 || goal_rx \u003e IBMVETH_MAX_RX_QUEUES) {\n+\t\tnetdev_err(netdev,\n+\t\t\t \"Invalid RX queue count %u (must be 1-%d)\\n\",\n+\t\t\t goal_rx, IBMVETH_MAX_RX_QUEUES);\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tif (goal_rx == old_rx)\n+\t\treturn 0;\n+\n+\tif (!(netdev-\u003eflags \u0026 IFF_UP)) {\n+\t\tadapter-\u003enum_rx_queues = goal_rx;\n+\t\treturn 0;\n+\t}\n+\n+\trxq_entries = adapter-\u003erx_queue[0].num_slots;\n+\trc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,\n+\t\t\t\t\t\t rxq_entries);\n+\tif (rc)\n+\t\tnetdev_err(netdev, \"Failed to resize RX queues: %d\\n\", rc);\n+\treturn rc;\n }\n \n static int ibmveth_set_channels(struct net_device *netdev,\n \t\t\t\tstruct ethtool_channels *channels)\n {\n \tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n-\tunsigned int old = netdev-\u003ereal_num_tx_queues,\n-\t\t goal = channels-\u003etx_count;\n+\tunsigned int goal_rx = channels-\u003erx_count;\n+\tunsigned int old_tx = netdev-\u003ereal_num_tx_queues;\n+\tunsigned int goal_tx = channels-\u003etx_count;\n \tint rc, i;\n \n \t/* If ndo_open has not been called yet then don't allocate, just set\n \t * desired netdev_queue's and return\n \t */\n-\tif (!(netdev-\u003eflags \u0026 IFF_UP))\n-\t\treturn netif_set_real_num_tx_queues(netdev, goal);\n+\tif (!(netdev-\u003eflags \u0026 IFF_UP)) {\n+\t\tif (goal_tx != old_tx) {\n+\t\t\trc = netif_set_real_num_tx_queues(netdev, goal_tx);\n+\t\t\tif (rc)\n+\t\t\t\treturn rc;\n+\t\t}\n+\n+\t\t/* Stash desired RX count only after TX succeeds (or was\n+\t\t * already correct); open() publishes it via\n+\t\t * netif_set_real_num_rx_queues().\n+\t\t */\n+\t\treturn ibmveth_resize_rx_channels(adapter, goal_rx);\n+\t}\n+\n+\trc = ibmveth_resize_rx_channels(adapter, goal_rx);\n+\tif (rc)\n+\t\treturn rc;\n \n \t/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated\n \t * but we may need to alloc/free the ltb's.\n \t */\n+\tif (goal_tx == old_tx)\n+\t\treturn 0;\n+\n \tnetif_tx_stop_all_queues(netdev);\n \n-\t/* Allocate any queue that we need */\n-\tfor (i = old; i \u003c goal; i++) {\n+\t/* Allocate any queue that we need. Initialize i to old_tx so a\n+\t * scale-down path that never enters the loop still has defined\n+\t * bounds if set_real_num_tx_queues() fails.\n+\t */\n+\ti = old_tx;\n+\tfor (; i \u003c goal_tx; i++) {\n \t\tif (adapter-\u003etx_ltb_ptr[i])\n \t\t\tcontinue;\n \n@@ -1144,20 +2783,20 @@ static int ibmveth_set_channels(struct net_device *netdev,\n \n \t\t/* if something goes wrong, free everything we just allocated */\n \t\tnetdev_err(netdev, \"Failed to allocate more tx queues, returning to %d queues\\n\",\n-\t\t\t old);\n-\t\tgoal = old;\n-\t\told = i;\n+\t\t\t old_tx);\n+\t\tgoal_tx = old_tx;\n+\t\told_tx = i;\n \t\tbreak;\n \t}\n-\trc = netif_set_real_num_tx_queues(netdev, goal);\n+\trc = netif_set_real_num_tx_queues(netdev, goal_tx);\n \tif (rc) {\n \t\tnetdev_err(netdev, \"Failed to set real tx queues, returning to %d queues\\n\",\n-\t\t\t old);\n-\t\tgoal = old;\n-\t\told = i;\n+\t\t\t old_tx);\n+\t\tgoal_tx = old_tx;\n+\t\told_tx = i;\n \t}\n \t/* Free any that are no longer needed */\n-\tfor (i = old; i \u003e goal; i--) {\n+\tfor (i = old_tx; i \u003e goal_tx; i--) {\n \t\tif (adapter-\u003etx_ltb_ptr[i - 1])\n \t\t\tibmveth_free_tx_ltb(adapter, i - 1);\n \t}\n@@ -1209,12 +2848,15 @@ static int ibmveth_send(struct ibmveth_adapter *adapter,\n \t\treturn 1;\n \t}\n \n+\tadapter-\u003ehcall_stats.send_lan++;\n \treturn 0;\n }\n \n static int ibmveth_is_packet_unsupported(struct sk_buff *skb,\n-\t\t\t\t\t struct net_device *netdev)\n+\t\t\t\t\t struct ibmveth_adapter *adapter,\n+\t\t\t\t\t int queue_num)\n {\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n \tstruct ethhdr *ether_header;\n \tint ret = 0;\n \n@@ -1222,7 +2864,8 @@ static int ibmveth_is_packet_unsupported(struct sk_buff *skb,\n \n \tif (ether_addr_equal(ether_header-\u003eh_dest, netdev-\u003edev_addr)) {\n \t\tnetdev_dbg(netdev, \"veth doesn't support loopback packets, dropping packet.\\n\");\n-\t\tnetdev-\u003estats.tx_dropped++;\n+\t\tif (adapter-\u003etx_qstats)\n+\t\t\tadapter-\u003etx_qstats[queue_num].dropped_packets++;\n \t\tret = -EOPNOTSUPP;\n \t}\n \n@@ -1238,7 +2881,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \tint i, queue_num = skb_get_queue_mapping(skb);\n \tunsigned long mss = 0;\n \n-\tif (ibmveth_is_packet_unsupported(skb, netdev))\n+\tif (ibmveth_is_packet_unsupported(skb, adapter, queue_num))\n \t\tgoto out;\n \t/* veth can't checksum offload UDP */\n \tif (skb-\u003eip_summed == CHECKSUM_PARTIAL \u0026\u0026\n@@ -1249,7 +2892,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \t skb_checksum_help(skb)) {\n \n \t\tnetdev_err(netdev, \"tx: failed to checksum packet\\n\");\n-\t\tnetdev-\u003estats.tx_dropped++;\n+\t\tadapter-\u003etx_qstats[queue_num].dropped_packets++;\n \t\tgoto out;\n \t}\n \n@@ -1261,6 +2904,8 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \n \t\tdesc_flags |= (IBMVETH_BUF_NO_CSUM | IBMVETH_BUF_CSUM_GOOD);\n \n+\t\tadapter-\u003etx_qstats[queue_num].checksum_offload++;\n+\n \t\t/* Need to zero out the checksum */\n \t\tbuf[0] = 0;\n \t\tbuf[1] = 0;\n@@ -1272,7 +2917,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \tif (skb-\u003eip_summed == CHECKSUM_PARTIAL \u0026\u0026 skb_is_gso(skb)) {\n \t\tif (adapter-\u003efw_large_send_support) {\n \t\t\tmss = (unsigned long)skb_shinfo(skb)-\u003egso_size;\n-\t\t\tadapter-\u003etx_large_packets++;\n+\t\t\tadapter-\u003etx_qstats[queue_num].large_packets++;\n \t\t} else if (!skb_is_gso_v6(skb)) {\n \t\t\t/* Put -1 in the IP checksum to tell phyp it\n \t\t\t * is a largesend packet. Put the mss in\n@@ -1281,7 +2926,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \t\t\tip_hdr(skb)-\u003echeck = 0xffff;\n \t\t\ttcp_hdr(skb)-\u003echeck =\n \t\t\t\tcpu_to_be16(skb_shinfo(skb)-\u003egso_size);\n-\t\t\tadapter-\u003etx_large_packets++;\n+\t\t\tadapter-\u003etx_qstats[queue_num].large_packets++;\n \t\t}\n \t}\n \n@@ -1289,7 +2934,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \tif (unlikely(skb-\u003elen \u003e adapter-\u003etx_ltb_size)) {\n \t\tnetdev_err(adapter-\u003enetdev, \"tx: packet size (%u) exceeds ltb (%u)\\n\",\n \t\t\t skb-\u003elen, adapter-\u003etx_ltb_size);\n-\t\tnetdev-\u003estats.tx_dropped++;\n+\t\tadapter-\u003etx_qstats[queue_num].dropped_packets++;\n \t\tgoto out;\n \t}\n \tmemcpy(adapter-\u003etx_ltb_ptr[queue_num], skb-\u003edata, skb_headlen(skb));\n@@ -1306,7 +2951,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \tif (unlikely(total_bytes != skb-\u003elen)) {\n \t\tnetdev_err(adapter-\u003enetdev, \"tx: incorrect packet len copied into ltb (%u != %u)\\n\",\n \t\t\t skb-\u003elen, total_bytes);\n-\t\tnetdev-\u003estats.tx_dropped++;\n+\t\tadapter-\u003etx_qstats[queue_num].dropped_packets++;\n \t\tgoto out;\n \t}\n \tdesc.fields.flags_len = desc_flags | skb-\u003elen;\n@@ -1315,18 +2960,17 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,\n \tdma_wmb();\n \n \tif (ibmveth_send(adapter, desc.desc, mss)) {\n-\t\tadapter-\u003etx_send_failed++;\n-\t\tnetdev-\u003estats.tx_dropped++;\n+\t\tadapter-\u003etx_qstats[queue_num].send_failures++;\n+\t\tadapter-\u003etx_qstats[queue_num].dropped_packets++;\n \t} else {\n-\t\tnetdev-\u003estats.tx_packets++;\n-\t\tnetdev-\u003estats.tx_bytes += skb-\u003elen;\n+\t\tadapter-\u003etx_qstats[queue_num].packets++;\n+\t\tadapter-\u003etx_qstats[queue_num].bytes += skb-\u003elen;\n \t}\n \n out:\n \tdev_consume_skb_any(skb);\n \treturn NETDEV_TX_OK;\n \n-\n }\n \n static void ibmveth_rx_mss_helper(struct sk_buff *skb, u16 mss, int lrg_pkt)\n@@ -1447,36 +3091,99 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,\n \n static int ibmveth_poll(struct napi_struct *napi, int budget)\n {\n-\tstruct ibmveth_adapter *adapter =\n-\t\t\tcontainer_of(napi, struct ibmveth_adapter, napi);\n-\tstruct net_device *netdev = adapter-\u003enetdev;\n+\tstruct net_device *netdev = napi-\u003edev;\n+\tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n \tint frames_processed = 0;\n \tunsigned long lpar_rc;\n+\tint queue_index, rc;\n \tu16 mss = 0;\n \n+\tqueue_index = napi - adapter-\u003enapi;\n+\n+\tif (WARN_ON(queue_index \u003c 0 || queue_index \u003e= adapter-\u003enum_rx_queues))\n+\t\treturn 0;\n+\n+\tif (!netif_running(netdev) || napi_disable_pending(napi)) {\n+\t\tnapi_complete_done(napi, 0);\n+\t\treturn 0;\n+\t}\n+\n+\tif (adapter-\u003erx_qstats)\n+\t\tadapter-\u003erx_qstats[queue_index].polls++;\n+\n restart_poll:\n \twhile (frames_processed \u003c budget) {\n-\t\tif (!ibmveth_rxq_pending_buffer(adapter))\n+\t\tif (!netif_running(netdev) || napi_disable_pending(napi))\n+\t\t\tbreak;\n+\n+\t\tif (!ibmveth_rxq_pending_buffer(adapter, queue_index))\n \t\t\tbreak;\n \n \t\tsmp_rmb();\n-\t\tif (!ibmveth_rxq_buffer_valid(adapter)) {\n+\t\tif (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {\n \t\t\twmb(); /* suggested by larson1 */\n-\t\t\tadapter-\u003erx_invalid_buffer++;\n+\t\t\tif (adapter-\u003erx_qstats)\n+\t\t\t\tadapter-\u003erx_qstats[queue_index]\n+\t\t\t\t\t.invalid_buffers++;\n+\t\t\telse\n+\t\t\t\tadapter-\u003erx_invalid_buffer++;\n \t\t\tnetdev_dbg(netdev, \"recycling invalid buffer\\n\");\n-\t\t\tif (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))\n+\t\t\trc = ibmveth_rxq_harvest_buffer(adapter,\n+\t\t\t\t\t\t\tqueue_index, true);\n+\t\t\tif (unlikely(rc))\n \t\t\t\tbreak;\n \t\t} else {\n \t\t\tstruct sk_buff *skb, *new_skb;\n-\t\t\tint length = ibmveth_rxq_frame_length(adapter);\n-\t\t\tint offset = ibmveth_rxq_frame_offset(adapter);\n-\t\t\tint csum_good = ibmveth_rxq_csum_good(adapter);\n-\t\t\tint lrg_pkt = ibmveth_rxq_large_packet(adapter);\n+\t\t\tint length = ibmveth_rxq_frame_length(adapter,\n+\t\t\t\t\t\t\t queue_index);\n+\t\t\tint offset = ibmveth_rxq_frame_offset(adapter,\n+\t\t\t\t\t\t\t queue_index);\n+\t\t\tint csum_good = ibmveth_rxq_csum_good(adapter,\n+\t\t\t\t\t\t\t queue_index);\n+\t\t\tint lrg_pkt = ibmveth_rxq_large_packet(adapter,\n+\t\t\t\t\t\t\t queue_index);\n \t\t\t__sum16 iph_check = 0;\n \n-\t\t\tskb = ibmveth_rxq_get_buffer(adapter);\n-\t\t\tif (unlikely(!skb))\n-\t\t\t\tbreak;\n+\t\t\tskb = ibmveth_rxq_get_buffer(adapter, queue_index);\n+\t\t\tif (unlikely(!skb)) {\n+\t\t\t\tif (net_ratelimit())\n+\t\t\t\t\tnetdev_err(netdev,\n+\t\t\t\t\t\t \"bad correlator on queue %d, skipping slot\\n\",\n+\t\t\t\t\t\t queue_index);\n+\t\t\t\tif (adapter-\u003erx_qstats)\n+\t\t\t\t\tadapter-\u003erx_qstats[queue_index]\n+\t\t\t\t\t\t.invalid_buffers++;\n+\t\t\t\telse\n+\t\t\t\t\tadapter-\u003erx_invalid_buffer++;\n+\t\t\t\trc = ibmveth_rxq_harvest_buffer(adapter,\n+\t\t\t\t\t\t\t\tqueue_index,\n+\t\t\t\t\t\t\t\ttrue);\n+\t\t\t\tif (unlikely(rc))\n+\t\t\t\t\tbreak;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (unlikely((unsigned int)offset +\n+\t\t\t\t (unsigned int)length \u003e\n+\t\t\t\t skb_tailroom(skb))) {\n+\t\t\t\tif (net_ratelimit())\n+\t\t\t\t\tnetdev_err(netdev,\n+\t\t\t\t\t\t \"RX frame %u+%u exceeds buffer %u on queue %d, dropping\\n\",\n+\t\t\t\t\t\t offset, length,\n+\t\t\t\t\t\t skb_tailroom(skb),\n+\t\t\t\t\t\t queue_index);\n+\t\t\t\tif (adapter-\u003erx_qstats)\n+\t\t\t\t\tadapter-\u003erx_qstats[queue_index]\n+\t\t\t\t\t\t.invalid_buffers++;\n+\t\t\t\telse\n+\t\t\t\t\tadapter-\u003erx_invalid_buffer++;\n+\t\t\t\trc = ibmveth_rxq_harvest_buffer(adapter,\n+\t\t\t\t\t\t\t\tqueue_index,\n+\t\t\t\t\t\t\t\ttrue);\n+\t\t\t\tif (unlikely(rc))\n+\t\t\t\t\tbreak;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \n \t\t\t/* if the large packet bit is set in the rx queue\n \t\t\t * descriptor, the mss will be written by PHYP eight\n@@ -1499,12 +3206,18 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)\n \t\t\t\t\t\t\tlength);\n \t\t\t\tif (rx_flush)\n \t\t\t\t\tibmveth_flush_buffer(skb-\u003edata,\n-\t\t\t\t\t\tlength + offset);\n-\t\t\t\tif (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))\n+\t\t\t\t\t\t\t length + offset);\n+\t\t\t\trc = ibmveth_rxq_harvest_buffer(adapter,\n+\t\t\t\t\t\t\t\tqueue_index,\n+\t\t\t\t\t\t\t\ttrue);\n+\t\t\t\tif (unlikely(rc))\n \t\t\t\t\tbreak;\n \t\t\t\tskb = new_skb;\n \t\t\t} else {\n-\t\t\t\tif (unlikely(ibmveth_rxq_harvest_buffer(adapter, false)))\n+\t\t\t\trc = ibmveth_rxq_harvest_buffer(adapter,\n+\t\t\t\t\t\t\t\tqueue_index,\n+\t\t\t\t\t\t\t\tfalse);\n+\t\t\t\tif (unlikely(rc))\n \t\t\t\t\tbreak;\n \t\t\t\tskb_reserve(skb, offset);\n \t\t\t}\n@@ -1524,7 +3237,11 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)\n \t\t\tif ((length \u003e netdev-\u003emtu + ETH_HLEN) ||\n \t\t\t lrg_pkt || iph_check == 0xffff) {\n \t\t\t\tibmveth_rx_mss_helper(skb, mss, lrg_pkt);\n-\t\t\t\tadapter-\u003erx_large_packets++;\n+\t\t\t\tif (adapter-\u003erx_qstats)\n+\t\t\t\t\tadapter-\u003erx_qstats[queue_index]\n+\t\t\t\t\t\t.large_packets++;\n+\t\t\t\telse\n+\t\t\t\t\tadapter-\u003erx_large_packets++;\n \t\t\t}\n \n \t\t\tif (csum_good) {\n@@ -1534,16 +3251,25 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)\n \n \t\t\tnapi_gro_receive(napi, skb);\t/* send it up */\n \n-\t\t\tnetdev-\u003estats.rx_packets++;\n-\t\t\tnetdev-\u003estats.rx_bytes += length;\n+\t\t\tif (adapter-\u003erx_qstats) {\n+\t\t\t\tadapter-\u003erx_qstats[queue_index].packets++;\n+\t\t\t\tadapter-\u003erx_qstats[queue_index].bytes += length;\n+\t\t\t}\n+\n \t\t\tframes_processed++;\n \t\t}\n \t}\n \n-\tibmveth_replenish_task(adapter);\n+\tibmveth_replenish_task(adapter, queue_index);\n \n-\tif (frames_processed == budget)\n+\tif (frames_processed == budget) {\n+\t\tif (!netif_running(netdev) || napi_disable_pending(napi)) {\n+\t\t\tnapi_complete_done(napi, frames_processed);\n+\t\t\t/* After complete_done, must not return full budget. */\n+\t\t\treturn frames_processed ? frames_processed - 1 : 0;\n+\t\t}\n \t\tgoto out;\n+\t}\n \n \tif (!napi_complete_done(napi, frames_processed))\n \t\tgoto out;\n@@ -1551,15 +3277,21 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)\n \t/* We think we are done - reenable interrupts,\n \t * then check once more to make sure we are done.\n \t */\n-\tlpar_rc = h_vio_signal(adapter-\u003evdev-\u003eunit_address, VIO_IRQ_ENABLE);\n-\tif (WARN_ON(lpar_rc != H_SUCCESS)) {\n+\tlpar_rc = ibmveth_enable_irq(adapter, queue_index);\n+\tif (lpar_rc != H_SUCCESS) {\n+\t\tnetdev_err(netdev,\n+\t\t\t \"Failed to enable IRQ for queue %d (rc=0x%lx), scheduling reset\\n\",\n+\t\t\t queue_index, lpar_rc);\n \t\tschedule_work(\u0026adapter-\u003ework);\n \t\tgoto out;\n \t}\n \n-\tif (ibmveth_rxq_pending_buffer(adapter) \u0026\u0026 napi_schedule(napi)) {\n-\t\tlpar_rc = h_vio_signal(adapter-\u003evdev-\u003eunit_address,\n-\t\t\t\t VIO_IRQ_DISABLE);\n+\tif (ibmveth_rxq_pending_buffer(adapter, queue_index) \u0026\u0026\n+\t netif_running(netdev) \u0026\u0026\n+\t !napi_disable_pending(napi) \u0026\u0026\n+\t napi_schedule(napi)) {\n+\t\tlpar_rc = ibmveth_disable_irq(adapter, queue_index);\n+\t\tWARN_ON(lpar_rc != H_SUCCESS);\n \t\tgoto restart_poll;\n \t}\n \n@@ -1569,16 +3301,19 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)\n \n static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)\n {\n-\tstruct net_device *netdev = dev_instance;\n+\tstruct napi_struct *napi = dev_instance;\n+\tstruct net_device *netdev = napi-\u003edev;\n \tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n-\tunsigned long lpar_rc;\n+\tint qindex;\n \n-\tif (napi_schedule_prep(\u0026adapter-\u003enapi)) {\n-\t\tlpar_rc = h_vio_signal(adapter-\u003evdev-\u003eunit_address,\n-\t\t\t\t VIO_IRQ_DISABLE);\n-\t\tWARN_ON(lpar_rc != H_SUCCESS);\n-\t\t__napi_schedule(\u0026adapter-\u003enapi);\n-\t}\n+\tqindex = napi - adapter-\u003enapi;\n+\tif (WARN_ON(qindex \u003c 0 || qindex \u003e= adapter-\u003enum_rx_queues))\n+\t\treturn IRQ_NONE;\n+\n+\tif (adapter-\u003erx_qstats)\n+\t\tadapter-\u003erx_qstats[qindex].interrupts++;\n+\n+\tibmveth_schedule_rx_queue(adapter, qindex);\n \treturn IRQ_HANDLED;\n }\n \n@@ -1645,7 +3380,7 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)\n \tint need_restart = 0;\n \n \tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\tif (new_mtu_oh \u003c= adapter-\u003erx_buff_pool[i].buff_size)\n+\t\tif (new_mtu_oh \u003c= adapter-\u003erx_buff_pool[0][i].buff_size)\n \t\t\tbreak;\n \n \tif (i == IBMVETH_NUM_BUFF_POOLS)\n@@ -1660,9 +3395,9 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)\n \n \t/* Look for an active buffer pool that can hold the new MTU */\n \tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n-\t\tadapter-\u003erx_buff_pool[i].active = 1;\n+\t\tadapter-\u003erx_buff_pool[0][i].active = 1;\n \n-\t\tif (new_mtu_oh \u003c= adapter-\u003erx_buff_pool[i].buff_size) {\n+\t\tif (new_mtu_oh \u003c= adapter-\u003erx_buff_pool[0][i].buff_size) {\n \t\t\tWRITE_ONCE(dev-\u003emtu, new_mtu);\n \t\t\tvio_cmo_set_dev_desired(viodev,\n \t\t\t\t\t\tibmveth_get_desired_dma\n@@ -1683,8 +3418,14 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)\n #ifdef CONFIG_NET_POLL_CONTROLLER\n static void ibmveth_poll_controller(struct net_device *dev)\n {\n-\tibmveth_replenish_task(netdev_priv(dev));\n-\tibmveth_interrupt(dev-\u003eirq, dev);\n+\tstruct ibmveth_adapter *adapter = netdev_priv(dev);\n+\tint i;\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\tibmveth_replenish_task(adapter, i);\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++)\n+\t\tibmveth_schedule_rx_queue(adapter, i);\n }\n #endif\n \n@@ -1693,8 +3434,7 @@ static void ibmveth_poll_controller(struct net_device *dev)\n *\n * @vdev: struct vio_dev for the device whose desired IO mem is to be returned\n *\n- * Return value:\n- *\tNumber of bytes of IO data the driver will need to perform well.\n+ * Return: Number of bytes of IO data the driver will need to perform well.\n */\n static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)\n {\n@@ -1702,8 +3442,7 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)\n \tstruct ibmveth_adapter *adapter;\n \tstruct iommu_table *tbl;\n \tunsigned long ret;\n-\tint i;\n-\tint rxqentries = 1;\n+\tint i, q;\n \n \ttbl = get_iommu_table_base(\u0026vdev-\u003edev);\n \n@@ -1718,18 +3457,25 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)\n \t/* add size of mapped tx buffers */\n \tret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);\n \n-\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n-\t\t/* add the size of the active receive buffers */\n-\t\tif (adapter-\u003erx_buff_pool[i].active)\n-\t\t\tret +=\n-\t\t\t adapter-\u003erx_buff_pool[i].size *\n-\t\t\t IOMMU_PAGE_ALIGN(adapter-\u003erx_buff_pool[i].\n-\t\t\t\t\t buff_size, tbl);\n-\t\trxqentries += adapter-\u003erx_buff_pool[i].size;\n-\t}\n-\t/* add the size of the receive queue entries */\n-\tret += IOMMU_PAGE_ALIGN(\n-\t\trxqentries * sizeof(struct ibmveth_rx_q_entry), tbl);\n+\tfor (q = 0; q \u003c adapter-\u003enum_rx_queues; q++) {\n+\t\tint rxqentries = 1;\n+\n+\t\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n+\t\t\t/* add the size of the active receive buffers */\n+\t\t\tstruct ibmveth_buff_pool *bpool =\n+\t\t\t\t\u0026adapter-\u003erx_buff_pool[q][i];\n+\n+\t\t\t/* add the size of the active receive buffers */\n+\t\t\tif (bpool-\u003eactive)\n+\t\t\t\tret += bpool-\u003esize *\n+\t\t\t\t\tIOMMU_PAGE_ALIGN(bpool-\u003ebuff_size, tbl);\n+\t\t\trxqentries += bpool-\u003esize;\n+\t\t}\n+\n+\t\t/* add the size of the receive queue entries */\n+\t\tret += IOMMU_PAGE_ALIGN(rxqentries *\n+\t\t\t\t\tsizeof(struct ibmveth_rx_q_entry), tbl);\n+\t}\n \n \treturn ret;\n }\n@@ -1777,6 +3523,40 @@ static netdev_features_t ibmveth_features_check(struct sk_buff *skb,\n \treturn vlan_features_check(skb, features);\n }\n \n+/**\n+ * ibmveth_get_stats64 - Return aggregated per-queue statistics\n+ * @dev: network device\n+ * @stats: rtnl link statistics storage\n+ *\n+ * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.\n+ * Callers use ndo_get_stats64(); avoid updating netdev-\u003estats on the\n+ * xmit/poll paths to keep per-queue counters off the hot cache line.\n+ */\n+static void ibmveth_get_stats64(struct net_device *dev,\n+\t\t\t\tstruct rtnl_link_stats64 *stats)\n+{\n+\tstruct ibmveth_adapter *adapter = netdev_priv(dev);\n+\tint i;\n+\n+\tif (adapter-\u003erx_qstats) {\n+\t\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\t\tstats-\u003erx_packets += adapter-\u003erx_qstats[i].packets;\n+\t\t\tstats-\u003erx_bytes += adapter-\u003erx_qstats[i].bytes;\n+\t\t}\n+\t}\n+\n+\tif (adapter-\u003etx_qstats) {\n+\t\tfor (i = 0; i \u003c dev-\u003ereal_num_tx_queues; i++) {\n+\t\t\tstats-\u003etx_packets += adapter-\u003etx_qstats[i].packets;\n+\t\t\tstats-\u003etx_bytes += adapter-\u003etx_qstats[i].bytes;\n+\t\t\tstats-\u003etx_dropped +=\n+\t\t\t\tadapter-\u003etx_qstats[i].dropped_packets;\n+\t\t}\n+\t}\n+\n+\tstats-\u003etx_errors = dev-\u003estats.tx_errors;\n+}\n+\n static const struct net_device_ops ibmveth_netdev_ops = {\n \t.ndo_open\t\t= ibmveth_open,\n \t.ndo_stop\t\t= ibmveth_close,\n@@ -1789,14 +3569,75 @@ static const struct net_device_ops ibmveth_netdev_ops = {\n \t.ndo_validate_addr\t= eth_validate_addr,\n \t.ndo_set_mac_address = ibmveth_set_mac_addr,\n \t.ndo_features_check\t= ibmveth_features_check,\n+\t.ndo_get_stats64\t= ibmveth_get_stats64,\n #ifdef CONFIG_NET_POLL_CONTROLLER\n \t.ndo_poll_controller\t= ibmveth_poll_controller,\n #endif\n };\n \n+static int ibmveth_buffer_pools_show(struct seq_file *m, void *v)\n+{\n+\tstruct ibmveth_adapter *adapter = m-\u003eprivate;\n+\tint i, j;\n+\n+\tseq_puts(m, \"Queue Pool Size BuffSize Active Available\\n\");\n+\tseq_puts(m, \"----- ---- ---- -------- ------ ---------\\n\");\n+\n+\tfor (i = 0; i \u003c adapter-\u003enum_rx_queues; i++) {\n+\t\tfor (j = 0; j \u003c IBMVETH_NUM_BUFF_POOLS; j++) {\n+\t\t\tstruct ibmveth_buff_pool *pool =\n+\t\t\t\t\u0026adapter-\u003erx_buff_pool[i][j];\n+\n+\t\t\tseq_printf(m, \"%5d %4d %4u %8u %6d %9d\\n\",\n+\t\t\t\t i, j, pool-\u003esize, pool-\u003ebuff_size,\n+\t\t\t\t pool-\u003eactive,\n+\t\t\t\t atomic_read(\u0026pool-\u003eavailable));\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+DEFINE_SHOW_ATTRIBUTE(ibmveth_buffer_pools);\n+\n+static void ibmveth_debugfs_init(struct ibmveth_adapter *adapter)\n+{\n+\tadapter-\u003edebugfs_dir = debugfs_create_dir(adapter-\u003enetdev-\u003ename,\n+\t\t\t\t\t\t NULL);\n+\tdebugfs_create_file(\"buffer_pools\", 0400, adapter-\u003edebugfs_dir,\n+\t\t\t adapter, \u0026ibmveth_buffer_pools_fops);\n+}\n+\n+static void ibmveth_debugfs_exit(struct ibmveth_adapter *adapter)\n+{\n+\tdebugfs_remove_recursive(adapter-\u003edebugfs_dir);\n+\tadapter-\u003edebugfs_dir = NULL;\n+}\n+\n+static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,\n+\t\t\t\t int pools_ready)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i \u003c pools_ready; i++)\n+\t\tkobject_put(\u0026adapter-\u003erx_buff_pool[0][i].kobj);\n+}\n+\n+static void ibmveth_probe_cleanup(struct ibmveth_adapter *adapter,\n+\t\t\t\t int pools_ready)\n+{\n+\tstruct net_device *netdev = adapter-\u003enetdev;\n+\n+\tcancel_work_sync(\u0026adapter-\u003ework);\n+\tibmveth_put_pool_kobjs(adapter, pools_ready);\n+\n+\tibmveth_free_tx_qstats(adapter);\n+\tibmveth_free_rx_qstats(adapter);\n+\tfree_netdev(netdev);\n+}\n+\n static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n {\n-\tint rc, i, mac_len;\n+\tint rc, i, mac_len, pools_ready = 0;\n \tstruct net_device *netdev;\n \tstruct ibmveth_adapter *adapter;\n \tunsigned char *mac_addr_p;\n@@ -1831,7 +3672,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \t\treturn -EINVAL;\n \t}\n \n-\tnetdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter), IBMVETH_MAX_QUEUES, 1);\n+\tnetdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter),\n+\t\t\t\t IBMVETH_MAX_QUEUES, IBMVETH_MAX_RX_QUEUES);\n \tif (!netdev)\n \t\treturn -ENOMEM;\n \n@@ -1844,7 +3686,15 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \tadapter-\u003emcastFilterSize = be32_to_cpu(*mcastFilterSize_p);\n \tibmveth_init_link_settings(netdev);\n \n-\tnetif_napi_add_weight(netdev, \u0026adapter-\u003enapi, ibmveth_poll, 16);\n+\tfor (i = 0; i \u003c IBMVETH_MAX_RX_QUEUES; i++)\n+\t\tnetif_napi_add_weight(netdev, \u0026adapter-\u003enapi[i],\n+\t\t\t\t ibmveth_poll, 16);\n+\n+\tif (ibmveth_alloc_rx_qstats(adapter) ||\n+\t ibmveth_alloc_tx_qstats(adapter)) {\n+\t\tibmveth_probe_cleanup(adapter, 0);\n+\t\treturn -ENOMEM;\n+\t}\n \n \tnetdev-\u003eirq = dev-\u003eirq;\n \tnetdev-\u003enetdev_ops = \u0026ibmveth_netdev_ops;\n@@ -1876,12 +3726,27 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \t\tnetdev-\u003efeatures |= NETIF_F_FRAGLIST;\n \t}\n \n+\tif (ret == H_SUCCESS \u0026\u0026\n+\t (ret_attr \u0026 IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT)) {\n+\t\tadapter-\u003emulti_queue = 1;\n+\t\tadapter-\u003enum_rx_queues = min(num_online_cpus(),\n+\t\t\t\t\t IBMVETH_DEFAULT_QUEUES);\n+\t\tnetdev_dbg(netdev, \"RX multi queue mode enabled: %d queues\\n\",\n+\t\t\t adapter-\u003enum_rx_queues);\n+\t} else {\n+\t\tadapter-\u003emulti_queue = 0;\n+\t\tadapter-\u003enum_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;\n+\t}\n+\n \tif (ret == H_SUCCESS \u0026\u0026\n \t (ret_attr \u0026 IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {\n-\t\tadapter-\u003erx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;\n+\t\tif (adapter-\u003emulti_queue)\n+\t\t\tadapter-\u003erx_buffers_per_hcall = IBMVETH_MAX_RX_QUEUE;\n+\t\telse\n+\t\t\tadapter-\u003erx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;\n \t\tnetdev_dbg(netdev,\n \t\t\t \"RX Multi-buffer hcall supported by FW, batch set to %u\\n\",\n-\t\t\t adapter-\u003erx_buffers_per_hcall);\n+\t\t\t adapter-\u003erx_buffers_per_hcall);\n \t} else {\n \t\tadapter-\u003erx_buffers_per_hcall = 1;\n \t\tnetdev_dbg(netdev,\n@@ -1898,16 +3763,24 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \t\tmemcpy(pool_count, pool_count_cmo, sizeof(pool_count));\n \n \tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n-\t\tstruct kobject *kobj = \u0026adapter-\u003erx_buff_pool[i].kobj;\n-\t\tint error;\n+\t\tstruct kobject *kobj = \u0026adapter-\u003erx_buff_pool[0][i].kobj;\n \n-\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[i], i,\n+\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[0][i], i,\n \t\t\t\t\t pool_count[i], pool_size[i],\n \t\t\t\t\t pool_active[i]);\n-\t\terror = kobject_init_and_add(kobj, \u0026ktype_veth_pool,\n-\t\t\t\t\t \u0026dev-\u003edev.kobj, \"pool%d\", i);\n-\t\tif (!error)\n-\t\t\tkobject_uevent(kobj, KOBJ_ADD);\n+\t\trc = kobject_init_and_add(kobj, \u0026ktype_veth_pool,\n+\t\t\t\t\t \u0026dev-\u003edev.kobj, \"pool%d\", i);\n+\t\tif (rc) {\n+\t\t\tdev_err(\u0026dev-\u003edev,\n+\t\t\t\t\"failed to create pool%d kobject: %d\\n\", i, rc);\n+\t\t\t/* init_and_add takes a ref even on failure */\n+\t\t\tkobject_put(kobj);\n+\t\t\tibmveth_probe_cleanup(adapter, pools_ready);\n+\t\t\treturn rc;\n+\t\t}\n+\n+\t\tpools_ready++;\n+\t\tkobject_uevent(kobj, KOBJ_ADD);\n \t}\n \n \trc = netif_set_real_num_tx_queues(netdev, min(num_online_cpus(),\n@@ -1915,7 +3788,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \tif (rc) {\n \t\tnetdev_dbg(netdev, \"failed to set number of tx queues rc=%d\\n\",\n \t\t\t rc);\n-\t\tfree_netdev(netdev);\n+\t\tibmveth_probe_cleanup(adapter, pools_ready);\n \t\treturn rc;\n \t}\n \tadapter-\u003etx_ltb_size = PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE);\n@@ -1931,12 +3804,14 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)\n \n \tif (rc) {\n \t\tnetdev_dbg(netdev, \"failed to register netdev rc=%d\\n\", rc);\n-\t\tfree_netdev(netdev);\n+\t\tibmveth_probe_cleanup(adapter, pools_ready);\n \t\treturn rc;\n \t}\n \n \tnetdev_dbg(netdev, \"registered\\n\");\n \n+\tibmveth_debugfs_init(adapter);\n+\n \treturn 0;\n }\n \n@@ -1948,11 +3823,16 @@ static void ibmveth_remove(struct vio_dev *dev)\n \n \tcancel_work_sync(\u0026adapter-\u003ework);\n \n+\tibmveth_debugfs_exit(adapter);\n+\n \tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\tkobject_put(\u0026adapter-\u003erx_buff_pool[i].kobj);\n+\t\tkobject_put(\u0026adapter-\u003erx_buff_pool[0][i].kobj);\n \n \tunregister_netdev(netdev);\n \n+\tibmveth_free_tx_qstats(adapter);\n+\tibmveth_free_rx_qstats(adapter);\n+\n \tfree_netdev(netdev);\n \tdev_set_drvdata(\u0026dev-\u003edev, NULL);\n }\n@@ -2035,11 +3915,12 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,\n \t\t\t/* Make sure there is a buffer pool with buffers that\n \t\t\t can hold a packet of the size of the MTU */\n \t\t\tfor (i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++) {\n-\t\t\t\tif (pool == \u0026adapter-\u003erx_buff_pool[i])\n+\t\t\t\tif (pool == \u0026adapter-\u003erx_buff_pool[0][i])\n \t\t\t\t\tcontinue;\n-\t\t\t\tif (!adapter-\u003erx_buff_pool[i].active)\n+\t\t\t\tif (!adapter-\u003erx_buff_pool[0][i].active)\n \t\t\t\t\tcontinue;\n-\t\t\t\tif (mtu \u003c= adapter-\u003erx_buff_pool[i].buff_size)\n+\t\t\t\tif (mtu \u003c=\n+\t\t\t\t adapter-\u003erx_buff_pool[0][i].buff_size)\n \t\t\t\t\tbreak;\n \t\t\t}\n \n@@ -2092,8 +3973,8 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,\n \t}\n \trtnl_unlock();\n \n-\t/* kick the interrupt handler to allocate/deallocate pools */\n-\tibmveth_interrupt(netdev-\u003eirq, netdev);\n+\t/* kick RX processing to allocate/deallocate pools */\n+\tibmveth_schedule_rx_queue(adapter, 0);\n \treturn count;\n \n unlock_err:\n@@ -2101,7 +3982,6 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,\n \treturn rc;\n }\n \n-\n #define ATTR(_name, _mode)\t\t\t\t\\\n \tstruct attribute veth_##_name##_attr = {\t\\\n \t.name = __stringify(_name), .mode = _mode,\t\\\n@@ -2133,7 +4013,9 @@ static struct kobj_type ktype_veth_pool = {\n static int ibmveth_resume(struct device *dev)\n {\n \tstruct net_device *netdev = dev_get_drvdata(dev);\n-\tibmveth_interrupt(netdev-\u003eirq, netdev);\n+\tstruct ibmveth_adapter *adapter = netdev_priv(netdev);\n+\n+\tibmveth_schedule_rx_queue(adapter, 0);\n \treturn 0;\n }\n \n@@ -2196,8 +4078,7 @@ static void ibmveth_reset_kunit(struct work_struct *w)\n * @test: pointer to kunit structure\n *\n * Tests the error returns from ibmveth_remove_buffer_from_pool.\n- * ibmveth_remove_buffer_from_pool also calls WARN_ON, so dmesg should be\n- * checked to see that these warnings happened.\n+ * Bad correlators return -EINVAL/-EFAULT (no WARN_ON).\n *\n * Return: void\n */\n@@ -2213,26 +4094,38 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)\n \n \t/* Set sane values for buffer pools */\n \tfor (int i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[i], i,\n+\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[0][i], i,\n \t\t\t\t\t pool_count[i], pool_size[i],\n \t\t\t\t\t pool_active[i]);\n \n-\tpool = \u0026adapter-\u003erx_buff_pool[0];\n+\tpool = \u0026adapter-\u003erx_buff_pool[0][0];\n \tpool-\u003eskbuff = kunit_kcalloc(test, pool-\u003esize, sizeof(void *), GFP_KERNEL);\n \tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool-\u003eskbuff);\n \n \tcorrelator = ((u64)IBMVETH_NUM_BUFF_POOLS \u003c\u003c 32) | 0;\n-\tKUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));\n-\tKUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));\n-\n-\tcorrelator = ((u64)0 \u003c\u003c 32) | adapter-\u003erx_buff_pool[0].size;\n-\tKUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));\n-\tKUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));\n+\tKUNIT_EXPECT_EQ(test, -EINVAL,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, false));\n+\tKUNIT_EXPECT_EQ(test, -EINVAL,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, true));\n+\n+\tcorrelator = ((u64)0 \u003c\u003c 32) | adapter-\u003erx_buff_pool[0][0].size;\n+\tKUNIT_EXPECT_EQ(test, -EINVAL,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, false));\n+\tKUNIT_EXPECT_EQ(test, -EINVAL,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, true));\n \n \tcorrelator = (u64)0 | 0;\n \tpool-\u003eskbuff[0] = NULL;\n-\tKUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, false));\n-\tKUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, true));\n+\tKUNIT_EXPECT_EQ(test, -EFAULT,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, false));\n+\tKUNIT_EXPECT_EQ(test, -EFAULT,\n+\t\t\tibmveth_remove_buffer_from_pool(adapter,\n+\t\t\t\t\t\t\tcorrelator, 0, true));\n \n \tflush_work(\u0026adapter-\u003ework);\n }\n@@ -2241,9 +4134,7 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)\n * ibmveth_rxq_get_buffer_test - unit test for ibmveth_rxq_get_buffer\n * @test: pointer to kunit structure\n *\n- * Tests ibmveth_rxq_get_buffer. ibmveth_rxq_get_buffer also calls WARN_ON for\n- * the NULL returns, so dmesg should be checked to see that these warnings\n- * happened.\n+ * Tests ibmveth_rxq_get_buffer invalid correlator returns NULL without WARN.\n *\n * Return: void\n */\n@@ -2258,31 +4149,34 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)\n \n \tINIT_WORK(\u0026adapter-\u003ework, ibmveth_reset_kunit);\n \n-\tadapter-\u003erx_queue.queue_len = 1;\n-\tadapter-\u003erx_queue.index = 0;\n-\tadapter-\u003erx_queue.queue_addr = kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),\n-\t\t\t\t\t\t GFP_KERNEL);\n-\tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter-\u003erx_queue.queue_addr);\n+\tadapter-\u003erx_queue[0].queue_len = 1;\n+\tadapter-\u003erx_queue[0].index = 0;\n+\tadapter-\u003erx_queue[0].queue_addr =\n+\t\tkunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),\n+\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter-\u003erx_queue[0].queue_addr);\n \n \t/* Set sane values for buffer pools */\n \tfor (int i = 0; i \u003c IBMVETH_NUM_BUFF_POOLS; i++)\n-\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[i], i,\n+\t\tibmveth_init_buffer_pool(\u0026adapter-\u003erx_buff_pool[0][i], i,\n \t\t\t\t\t pool_count[i], pool_size[i],\n \t\t\t\t\t pool_active[i]);\n \n-\tpool = \u0026adapter-\u003erx_buff_pool[0];\n+\tpool = \u0026adapter-\u003erx_buff_pool[0][0];\n \tpool-\u003eskbuff = kunit_kcalloc(test, pool-\u003esize, sizeof(void *), GFP_KERNEL);\n \tKUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool-\u003eskbuff);\n \n-\tadapter-\u003erx_queue.queue_addr[0].correlator = (u64)IBMVETH_NUM_BUFF_POOLS \u003c\u003c 32 | 0;\n-\tKUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));\n+\tadapter-\u003erx_queue[0].queue_addr[0].correlator =\n+\t\t(u64)IBMVETH_NUM_BUFF_POOLS \u003c\u003c 32 | 0;\n+\tKUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));\n \n-\tadapter-\u003erx_queue.queue_addr[0].correlator = (u64)0 \u003c\u003c 32 | adapter-\u003erx_buff_pool[0].size;\n-\tKUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));\n+\tadapter-\u003erx_queue[0].queue_addr[0].correlator =\n+\t\t(u64)0 \u003c\u003c 32 | adapter-\u003erx_buff_pool[0][0].size;\n+\tKUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));\n \n \tpool-\u003eskbuff[0] = skb;\n-\tadapter-\u003erx_queue.queue_addr[0].correlator = (u64)0 \u003c\u003c 32 | 0;\n-\tKUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter));\n+\tadapter-\u003erx_queue[0].queue_addr[0].correlator = (u64)0 \u003c\u003c 32 | 0;\n+\tKUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter, 0));\n \n \tflush_work(\u0026adapter-\u003ework);\n }\ndiff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h\nindex d87713668ed30..17faaa4c2d33d 100644\n--- a/drivers/net/ethernet/ibm/ibmveth.h\n+++ b/drivers/net/ethernet/ibm/ibmveth.h\n@@ -14,6 +14,8 @@\n #ifndef _IBMVETH_H\n #define _IBMVETH_H\n \n+#include \u003clinux/spinlock_types.h\u003e\n+\n /* constants for H_MULTICAST_CTRL */\n #define IbmVethMcastReceptionModifyBit 0x80000UL\n #define IbmVethMcastReceptionEnableBit 0x20000UL\n@@ -28,6 +30,7 @@\n #define IbmVethMcastRemoveFilter 0x2UL\n #define IbmVethMcastClearFilterTable 0x3UL\n \n+#define IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT\t0x0000000000080000UL\n #define IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT\t0x0000000000040000UL\n #define IBMVETH_ILLAN_LRG_SR_ENABLED\t0x0000000000010000UL\n #define IBMVETH_ILLAN_LRG_SND_SUPPORT\t0x0000000000008000UL\n@@ -66,6 +69,143 @@ static inline long h_add_logical_lan_buffers(unsigned long unit_address,\n \t\t\t desc5, desc6, desc7, desc8);\n }\n \n+/**\n+ * h_reg_logical_lan_queue - Register a subordinate receive queue\n+ * @unit_address: Device unit address\n+ * @buffer_list: DMA address of 4KB page for tracking registered buffers\n+ * @rec_queue: Buffer descriptor of receive queue\n+ * @queue_handle: Output queue handle on success (may be NULL)\n+ * @irq: Output hypervisor IRQ number on success (may be NULL)\n+ *\n+ * Registers a subordinate receive queue with the hypervisor.\n+ *\n+ * Return:\n+ * H_SUCCESS (0) on success\n+ * H_PARAMETER if parameters are invalid\n+ *\n+ * On success, hypervisor returns:\n+ * R3: H_SUCCESS\n+ * R4: Queue handle\n+ * R5: IRQ number for this queue\n+ */\n+static inline long h_reg_logical_lan_queue(unsigned long unit_address,\n+\t\t\t\t\t unsigned long buffer_list,\n+\t\t\t\t\t unsigned long rec_queue,\n+\t\t\t\t\t unsigned long *queue_handle,\n+\t\t\t\t\t unsigned long *irq)\n+{\n+\tunsigned long retbuf[PLPAR_HCALL9_BUFSIZE];\n+\tlong rc;\n+\n+\trc = plpar_hcall9(H_REG_LOGICAL_LAN_QUEUE,\n+\t\t\t retbuf, unit_address,\n+\t\t\t buffer_list, rec_queue);\n+\n+\tif (rc == H_SUCCESS) {\n+\t\tif (queue_handle)\n+\t\t\t*queue_handle = retbuf[0];\n+\t\tif (irq)\n+\t\t\t*irq = retbuf[1];\n+\t}\n+\n+\treturn rc;\n+}\n+\n+/**\n+ * h_add_logical_lan_buffers_queue - Add buffers to subordinate queue\n+ * @unit_address: Device unit address\n+ * @queue_handle: Queue handle from h_reg_logical_lan_queue()\n+ * @buffersznum: Buffer size (upper 32 bits) | count (lower 32 bits)\n+ * @ioba12: Buffer addresses 1 and 2 packed ((addr1 \u003c\u003c 32) | addr2)\n+ * @ioba34: Buffer addresses 3 and 4 packed\n+ * @ioba56: Buffer addresses 5 and 6 packed\n+ * @ioba78: Buffer addresses 7 and 8 packed\n+ * @ioba910: Buffer addresses 9 and 10 packed\n+ * @ioba1112: Buffer addresses 11 and 12 packed\n+ *\n+ * Return:\n+ * H_SUCCESS - All buffers added successfully\n+ * H_PARAMETER - Invalid parameters\n+ * H_HARDWARE - Hardware error\n+ */\n+static inline long h_add_logical_lan_buffers_queue(unsigned long unit_address,\n+\t\t\t\t\t\t unsigned long queue_handle,\n+\t\t\t\t\t\t unsigned long buffersznum,\n+\t\t\t\t\t\t unsigned long ioba12,\n+\t\t\t\t\t\t unsigned long ioba34,\n+\t\t\t\t\t\t unsigned long ioba56,\n+\t\t\t\t\t\t unsigned long ioba78,\n+\t\t\t\t\t\t unsigned long ioba910,\n+\t\t\t\t\t\t unsigned long ioba1112)\n+{\n+\tunsigned long retbuf[PLPAR_HCALL9_BUFSIZE];\n+\n+\treturn plpar_hcall9(H_ADD_LOGICAL_LAN_BUFFERS_QUEUE,\n+\t\t\t retbuf, unit_address,\n+\t\t\t queue_handle, buffersznum,\n+\t\t\t ioba12, ioba34, ioba56,\n+\t\t\t ioba78, ioba910, ioba1112);\n+}\n+\n+/**\n+ * h_free_logical_lan_queue - Deregister subordinate receive queue\n+ * @unit_address: Device unit address\n+ * @queue_handle: Queue handle from h_reg_logical_lan_queue()\n+ *\n+ * Deregisters and frees all structures associated with the subordinate queue.\n+ *\n+ * Return:\n+ * H_SUCCESS - Queue freed successfully\n+ * H_PARAMETER - Invalid parameters\n+ * H_HARDWARE - Hardware error\n+ * H_STATE - VIOA not in valid state\n+ * H_BUSY / H_LONG_BUSY_* - Resource busy, retry\n+ */\n+static inline long h_free_logical_lan_queue(unsigned long unit_address,\n+\t\t\t\t\t unsigned long queue_handle)\n+{\n+\tunsigned long retbuf[PLPAR_HCALL9_BUFSIZE];\n+\n+\treturn plpar_hcall9(H_FREE_LOGICAL_LAN_QUEUE,\n+\t\t\t retbuf, unit_address, queue_handle);\n+}\n+\n+/**\n+ * h_register_logical_lan_with_handle - Register primary queue and get handle\n+ * @unit_address: Device unit address\n+ * @buffer_list: DMA address of buffer list\n+ * @rec_queue: Buffer descriptor of receive queue\n+ * @filter_list: DMA address of filter list\n+ * @mac_address: MAC address\n+ * @queue_handle: Output parameter for queue handle\n+ *\n+ * Registers the primary receive queue (queue 0) with the hypervisor and\n+ * returns the queue handle. This is needed in multi-queue mode to use\n+ * h_add_logical_lan_buffers_queue() for all queues including queue 0.\n+ *\n+ * Return: H_SUCCESS (0) on success, error code otherwise\n+ */\n+static inline long\n+h_register_logical_lan_with_handle(unsigned long unit_address,\n+\t\t\t\t unsigned long buffer_list,\n+\t\t\t\t unsigned long rec_queue,\n+\t\t\t\t unsigned long filter_list,\n+\t\t\t\t unsigned long mac_address,\n+\t\t\t\t u64 *queue_handle)\n+{\n+\tunsigned long retbuf[PLPAR_HCALL9_BUFSIZE];\n+\tlong rc;\n+\n+\trc = plpar_hcall9(H_REGISTER_LOGICAL_LAN, retbuf,\n+\t\t\t unit_address, buffer_list, rec_queue,\n+\t\t\t filter_list, mac_address);\n+\n+\tif (rc == H_SUCCESS \u0026\u0026 queue_handle)\n+\t\t*queue_handle = retbuf[0];\n+\n+\treturn rc;\n+}\n+\n /* FW allows us to send 6 descriptors but we only use one so mark\n * the other 5 as unused (0)\n */\n@@ -121,15 +261,62 @@ static inline long h_illan_attributes(unsigned long unit_address,\n #define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)\n #define IBMVETH_MAX_QUEUES 16U\n #define IBMVETH_DEFAULT_QUEUES 8U\n-#define IBMVETH_MAX_RX_PER_HCALL 8U\n+#define IBMVETH_MAX_RX_QUEUES 16U\n+#define IBMVETH_DEFAULT_RX_QUEUES 1U\n+#define IBMVETH_MAX_RX_REGULAR 8U\n+#define IBMVETH_MAX_RX_QUEUE 12U\n+#define IBMVETH_MAX_RX_PER_HCALL 12U\n \n static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };\n static int pool_count[] = { 256, 512, 256, 256, 256 };\n static int pool_count_cmo[] = { 256, 512, 256, 256, 64 };\n-static int pool_active[] = { 1, 1, 0, 0, 1};\n+static int pool_active[] = { 1, 1, 0, 0, 0};\n \n #define IBM_VETH_INVALID_MAP ((u16)0xffff)\n \n+struct ibmveth_hcall_stats {\n+\tu64 reg_lan_queue;\t/* H_REG_LOGICAL_LAN_QUEUE */\n+\tu64 reg_lan;\t\t/* H_REGISTER_LOGICAL_LAN */\n+\tu64 add_bufs_queue;\t/* H_ADD_LOGICAL_LAN_BUFFERS_QUEUE */\n+\tu64 add_bufs;\t\t/* H_ADD_LOGICAL_LAN_BUFFERS */\n+\tu64 add_buf;\t\t/* H_ADD_LOGICAL_LAN_BUFFER */\n+\tu64 free_lan_queue;\t/* H_FREE_LOGICAL_LAN_QUEUE */\n+\tu64 free_lan;\t\t/* H_FREE_LOGICAL_LAN */\n+\tu64 send_lan;\t\t/* H_SEND_LOGICAL_LAN */\n+};\n+\n+struct ibmveth_rx_queue_stats {\n+\tu64 packets;\n+\tu64 bytes;\n+\tu64 interrupts;\n+\tu64 polls;\n+\tu64 large_packets;\n+\tu64 invalid_buffers;\n+\tu64 no_buffer_drops;\n+} ____cacheline_aligned_in_smp;\n+\n+struct ibmveth_tx_queue_stats {\n+\tu64 packets;\n+\tu64 bytes;\n+\tu64 large_packets;\n+\tu64 dropped_packets;\n+\tu64 send_failures;\n+\tu64 checksum_offload;\n+} ____cacheline_aligned_in_smp;\n+\n+/*\n+ * ethtool string count: use offsetof of the last counter so alignment\n+ * padding from ____cacheline_aligned_in_smp is not counted. When adding\n+ * a new counter at the end, point these at the new last field (same idea\n+ * as sizeof(struct)/sizeof(u64) before alignment was added).\n+ */\n+#define IBMVETH_NUM_RX_QSTATS \\\n+\t(offsetof(struct ibmveth_rx_queue_stats, no_buffer_drops) / \\\n+\t\tsizeof(u64) + 1)\n+#define IBMVETH_NUM_TX_QSTATS \\\n+\t(offsetof(struct ibmveth_tx_queue_stats, checksum_offload) / \\\n+\t\tsizeof(u64) + 1)\n+\n struct ibmveth_buff_pool {\n u32 size;\n u32 index;\n@@ -152,23 +339,29 @@ struct ibmveth_rx_q {\n dma_addr_t queue_dma;\n u32 queue_len;\n struct ibmveth_rx_q_entry *queue_addr;\n+\tspinlock_t\treplenish_lock;\t/* per-queue buffer replenish */\n };\n \n struct ibmveth_adapter {\n \tstruct vio_dev *vdev;\n \tstruct net_device *netdev;\n-\tstruct napi_struct napi;\n+\tstruct napi_struct napi[IBMVETH_MAX_RX_QUEUES];\n \tstruct work_struct work;\n \tunsigned int mcastFilterSize;\n-\tvoid *buffer_list_addr;\n+\tvoid *buffer_list_addr[IBMVETH_MAX_RX_QUEUES];\n \tvoid *filter_list_addr;\n \tvoid *tx_ltb_ptr[IBMVETH_MAX_QUEUES];\n \tunsigned int tx_ltb_size;\n \tdma_addr_t tx_ltb_dma[IBMVETH_MAX_QUEUES];\n-\tdma_addr_t buffer_list_dma;\n+\tdma_addr_t buffer_list_dma[IBMVETH_MAX_RX_QUEUES];\n \tdma_addr_t filter_list_dma;\n-\tstruct ibmveth_buff_pool rx_buff_pool[IBMVETH_NUM_BUFF_POOLS];\n-\tstruct ibmveth_rx_q rx_queue;\n+\tstruct ibmveth_buff_pool\n+\t\trx_buff_pool[IBMVETH_MAX_RX_QUEUES][IBMVETH_NUM_BUFF_POOLS];\n+\tstruct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];\n+\tu64 queue_handle[IBMVETH_MAX_RX_QUEUES];\n+\tunsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];\n+\tint multi_queue;\n+\tunsigned int num_rx_queues;\n \tint rx_csum;\n \tint large_send;\n \tbool is_active_trunk;\n@@ -188,6 +381,14 @@ struct ibmveth_adapter {\n \tu64 tx_send_failed;\n \tu64 tx_large_packets;\n \tu64 rx_large_packets;\n+\n+\t/* Multi-queue statistics */\n+\tstruct ibmveth_hcall_stats hcall_stats;\n+\tstruct ibmveth_rx_queue_stats *rx_qstats;\n+\tstruct ibmveth_tx_queue_stats *tx_qstats;\n+\n+\tstruct dentry *debugfs_dir;\n+\n \t/* Ethtool settings */\n \tu8 duplex;\n \tu32 speed;\n"
} |
| 2/1 | 2026/08/01 10:16 | agent | patch-evaluator |
0mModel:best-expensive Args: null Results: {
"EnableConfigs": [
"NET_VENDOR_IBM",
"IBMVETH"
],
"FocusSymbols": [
"ibmveth_open",
"ibmveth_close",
"ibmveth_poll",
"ibmveth_start_xmit",
"ibmveth_set_channels",
"ibmveth_probe"
],
"Reasoning": "The patch adds multi-queue RX support to the IBM Power Virtual Ethernet (ibmveth) driver, modifying core driver entry points like open, close, poll, xmit, and ethtool set_channels. This driver is reachable in QEMU when running a PowerPC pseries VM, making it fuzzable.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 92c4ac344e4c00de83a639c43fdc3d7c2d91f40a
Author: syz-cluster <triage@syzkaller.com>
Date: Sat Aug 1 10:16:46 2026 +0000
syz-cluster: applied patch under review
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index dff90a7d7f70b..d79baf2e38273 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -362,7 +362,10 @@
#define H_GUEST_DELETE 0x488
#define H_PKS_WRAP_OBJECT 0x490
#define H_PKS_UNWRAP_OBJECT 0x494
-#define MAX_HCALL_OPCODE H_PKS_UNWRAP_OBJECT
+#define H_REG_LOGICAL_LAN_QUEUE 0x49C
+#define H_ADD_LOGICAL_LAN_BUFFERS_QUEUE 0x4A0
+#define H_FREE_LOGICAL_LAN_QUEUE 0x4A8
+#define MAX_HCALL_OPCODE H_FREE_LOGICAL_LAN_QUEUE
/* Scope args for H_SCM_UNBIND_ALL */
#define H_UNBIND_SCOPE_ALL (0x1)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 73e051d26b9d8..09e06d86701a5 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -21,6 +21,8 @@
#include <linux/skbuff.h>
#include <linux/init.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
#include <linux/mm.h>
#include <linux/pm.h>
#include <linux/ethtool.h>
@@ -28,6 +30,8 @@
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/debugfs.h>
#include <asm/hvcall.h>
#include <linux/atomic.h>
#include <asm/vio.h>
@@ -43,7 +47,6 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev);
static struct kobj_type ktype_veth_pool;
-
static const char ibmveth_driver_name[] = "ibmveth";
static const char ibmveth_driver_string[] = "IBM Power Virtual Ethernet Driver";
#define ibmveth_driver_version "1.06"
@@ -95,49 +98,72 @@ static struct ibmveth_stat ibmveth_stats[] = {
{ "fw_enabled_ipv6_csum", IBMVETH_STAT_OFF(fw_ipv6_csum_support) },
{ "tx_large_packets", IBMVETH_STAT_OFF(tx_large_packets) },
{ "rx_large_packets", IBMVETH_STAT_OFF(rx_large_packets) },
- { "fw_enabled_large_send", IBMVETH_STAT_OFF(fw_large_send_support) }
+ { "fw_enabled_large_send", IBMVETH_STAT_OFF(fw_large_send_support) },
+ { "hcall_reg_lan_queue", IBMVETH_STAT_OFF(hcall_stats.reg_lan_queue) },
+ { "hcall_reg_lan", IBMVETH_STAT_OFF(hcall_stats.reg_lan) },
+ { "hcall_add_bufs_queue",
+ IBMVETH_STAT_OFF(hcall_stats.add_bufs_queue) },
+ { "hcall_add_bufs", IBMVETH_STAT_OFF(hcall_stats.add_bufs) },
+ { "hcall_add_buf", IBMVETH_STAT_OFF(hcall_stats.add_buf) },
+ { "hcall_free_lan_queue",
+ IBMVETH_STAT_OFF(hcall_stats.free_lan_queue) },
+ { "hcall_free_lan", IBMVETH_STAT_OFF(hcall_stats.free_lan) },
+ { "hcall_send_lan", IBMVETH_STAT_OFF(hcall_stats.send_lan) },
};
/* simple methods of getting data from the current rxq entry */
-static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)
+static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].flags_off);
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+
+ return be32_to_cpu(rxq->queue_addr[rxq->index].flags_off);
}
-static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return (ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_TOGGLE) >>
- IBMVETH_RXQ_TOGGLE_SHIFT;
+ return (ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_TOGGLE) >>
+ IBMVETH_RXQ_TOGGLE_SHIFT;
}
-static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_toggle(adapter) == adapter->rx_queue.toggle;
+ return ibmveth_rxq_toggle(adapter, queue_index) ==
+ adapter->rx_queue[queue_index].toggle;
}
-static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_VALID;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_VALID;
}
-static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_OFF_MASK;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_OFF_MASK;
}
-static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_LRG_PKT;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_LRG_PKT;
}
-static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].length);
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+
+ return be32_to_cpu(rxq->queue_addr[rxq->index].length);
}
-static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)
+static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_CSUM_GOOD;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_CSUM_GOOD;
}
static unsigned int ibmveth_real_max_tx_queues(void)
@@ -147,6 +173,560 @@ static unsigned int ibmveth_real_max_tx_queues(void)
return min(n_cpu, IBMVETH_MAX_QUEUES);
}
+/**
+ * ibmveth_alloc_filter_list - Allocate and map filter list
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_filter_list(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+
+ adapter->filter_list_addr = (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->filter_list_addr) {
+ netdev_err(netdev, "unable to allocate filter pages\n");
+ return -ENOMEM;
+ }
+
+ adapter->filter_list_dma = dma_map_single(dev,
+ adapter->filter_list_addr,
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->filter_list_dma)) {
+ netdev_err(netdev, "unable to map filter list pages\n");
+ free_page((unsigned long)adapter->filter_list_addr);
+ adapter->filter_list_addr = NULL;
+ return -ENOMEM;
+ }
+
+ netdev_dbg(netdev, "filter list @ 0x%p (DMA: 0x%llx)\n",
+ adapter->filter_list_addr,
+ (unsigned long long)adapter->filter_list_dma);
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_filter_list - Free filter list resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+
+ if (adapter->filter_list_dma) {
+ dma_unmap_single(dev, adapter->filter_list_dma, 4096,
+ DMA_BIDIRECTIONAL);
+ adapter->filter_list_dma = 0;
+ }
+
+ if (adapter->filter_list_addr) {
+ free_page((unsigned long)adapter->filter_list_addr);
+ adapter->filter_list_addr = NULL;
+ }
+}
+
+/**
+ * ibmveth_alloc_rx_qstats - Allocate per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->rx_qstats = kcalloc(IBMVETH_MAX_RX_QUEUES,
+ sizeof(*adapter->rx_qstats),
+ GFP_KERNEL);
+ if (!adapter->rx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_rx_qstats - Free per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->rx_qstats);
+ adapter->rx_qstats = NULL;
+}
+
+/**
+ * ibmveth_alloc_tx_qstats - Allocate per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->tx_qstats = kcalloc(IBMVETH_MAX_QUEUES,
+ sizeof(*adapter->tx_qstats),
+ GFP_KERNEL);
+ if (!adapter->tx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_tx_qstats - Free per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->tx_qstats);
+ adapter->tx_qstats = NULL;
+}
+
+/**
+ * ibmveth_alloc_rx_queues - Allocate per-queue RX resources
+ * @adapter: ibmveth adapter structure
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ adapter->buffer_list_addr[i] =
+ (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->buffer_list_addr[i]) {
+ netdev_err(netdev,
+ "unable to allocate buffer list for queue %d\n",
+ i);
+ goto err_cleanup;
+ }
+
+ adapter->rx_queue[i].queue_len =
+ sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+ adapter->rx_queue[i].queue_addr =
+ dma_alloc_coherent(dev, adapter->rx_queue[i].queue_len,
+ &adapter->rx_queue[i].queue_dma,
+ GFP_KERNEL);
+ if (!adapter->rx_queue[i].queue_addr) {
+ netdev_err(netdev,
+ "unable to allocate RX queue for queue %d\n",
+ i);
+ goto err_cleanup;
+ }
+
+ adapter->buffer_list_dma[i] =
+ dma_map_single(dev, adapter->buffer_list_addr[i],
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->buffer_list_dma[i])) {
+ netdev_err(netdev,
+ "unable to map buffer list for queue %d\n",
+ i);
+ adapter->buffer_list_dma[i] = 0;
+ goto err_cleanup;
+ }
+
+ adapter->rx_queue[i].index = 0;
+ adapter->rx_queue[i].num_slots = rxq_entries;
+ adapter->rx_queue[i].toggle = 1;
+ spin_lock_init(&adapter->rx_queue[i].replenish_lock);
+
+ netdev_dbg(netdev, "queue %d: buffer_list @ 0x%p (DMA: 0x%llx), rx_queue @ 0x%p (DMA: 0x%llx), %llu entries\n",
+ i, adapter->buffer_list_addr[i],
+ (unsigned long long)adapter->buffer_list_dma[i],
+ adapter->rx_queue[i].queue_addr,
+ (unsigned long long)adapter->rx_queue[i].queue_dma,
+ (unsigned long long)rxq_entries);
+ }
+
+ netdev_dbg(netdev, "allocated %d RX queue(s) with %d entries each\n",
+ adapter->num_rx_queues, rxq_entries);
+
+ return 0;
+
+err_cleanup:
+ /* Clean up previously allocated queues */
+ for (; i >= 0; i--) {
+ if (adapter->buffer_list_dma[i]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[i],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[i] = 0;
+ }
+ if (adapter->rx_queue[i].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+ adapter->rx_queue[i].queue_addr,
+ adapter->rx_queue[i].queue_dma);
+ adapter->rx_queue[i].queue_addr = NULL;
+ }
+ if (adapter->buffer_list_addr[i]) {
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ }
+ }
+
+ return -ENOMEM;
+}
+
+/**
+ * ibmveth_cleanup_rx_resources - Free all RX queue resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ int i;
+
+ netdev_dbg(adapter->netdev, "cleaning up %d RX queue(s)\n",
+ adapter->num_rx_queues);
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->buffer_list_dma[i]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[i],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[i] = 0;
+ }
+
+ if (adapter->rx_queue[i].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+ adapter->rx_queue[i].queue_addr,
+ adapter->rx_queue[i].queue_dma);
+ adapter->rx_queue[i].queue_addr = NULL;
+ }
+
+ if (adapter->buffer_list_addr[i]) {
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ }
+ }
+}
+
+/**
+ * ibmveth_toggle_irq - Common helper to enable/disable queue interrupts
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ * @enable: true to enable, false to disable
+ *
+ * For queue 0 (primary), uses h_vio_signal() as it's registered via
+ * h_register_logical_lan(). For subordinate queues (1+), uses H_VIOCTL
+ * with H_ENABLE/DISABLE_VIO_INTERRUPT for per-queue interrupt control.
+ *
+ * Return: 0 on success, error code otherwise
+ */
+static int
+ibmveth_toggle_irq(struct ibmveth_adapter *adapter, int queue_index,
+ bool enable)
+{
+ unsigned long rc;
+ unsigned long irq = adapter->queue_irq[queue_index];
+ const char *action = enable ? "enable" : "disable";
+
+ if (queue_index == 0) {
+ /* Primary queue: use h_vio_signal() */
+ rc = h_vio_signal(adapter->vdev->unit_address,
+ enable ? VIO_IRQ_ENABLE : VIO_IRQ_DISABLE);
+ } else {
+ /* Subordinate queues: use H_VIOCTL with hardware IRQ */
+ struct irq_data *irq_data = irq_get_irq_data(irq);
+ irq_hw_number_t hwirq;
+ u64 vioctl_cmd = enable ? H_ENABLE_VIO_INTERRUPT :
+ H_DISABLE_VIO_INTERRUPT;
+
+ if (!irq_data) {
+ netdev_err(adapter->netdev,
+ "Failed to get IRQ data for queue %d (virq=%lu)\n",
+ queue_index, irq);
+ return -EINVAL;
+ }
+
+ hwirq = irqd_to_hwirq(irq_data);
+ rc = plpar_hcall_norets(H_VIOCTL,
+ adapter->vdev->unit_address,
+ vioctl_cmd,
+ hwirq, 0, 0);
+
+ if (rc == H_PARAMETER) {
+ /* H_PARAMETER is non-fatal when IRQ is already in
+ * the requested state.
+ */
+ netdev_warn_once(adapter->netdev,
+ "H_VIOCTL %s IRQ returned H_PARAMETER for queue %d (hwirq=%lu)\n",
+ action, queue_index, hwirq);
+ return 0;
+ }
+ }
+
+ if (rc)
+ netdev_err(adapter->netdev,
+ "Failed to %s IRQ for queue %d, rc=%ld\n",
+ action, queue_index, rc);
+ return rc;
+}
+
+/**
+ * ibmveth_disable_irq - Disable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, error code otherwise
+ */
+static int
+ibmveth_disable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+ return ibmveth_toggle_irq(adapter, queue_index, false);
+}
+
+/**
+ * ibmveth_enable_irq - Enable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, error code otherwise
+ */
+static int
+ibmveth_enable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+ return ibmveth_toggle_irq(adapter, queue_index, true);
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mapping - Drop one subordinate virq mapping
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: RX queue index (1..N)
+ *
+ * Subordinate queues get mappings from irq_create_mapping() during PHYP
+ * registration. Queue 0 uses netdev->irq from device tree and is left alone.
+ *
+ * Bound against IBMVETH_MAX_RX_QUEUES, not num_rx_queues: scale-down and
+ * scale-up fail paths dispose queues that are no longer in the published
+ * live set but still own a virq in queue_irq[]. The bulk helper still
+ * iterates only 1..num_rx_queues-1 for close/open-fail cleanup.
+ *
+ * Linux virq lifetime is owned by interrupt cleanup helpers. Call this only
+ * after free_irq() when a handler was installed, or from registration failure
+ * cleanup before request_irq().
+ */
+static void
+ibmveth_dispose_subordinate_irq_mapping(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ if (queue_idx <= 0 || queue_idx >= IBMVETH_MAX_RX_QUEUES)
+ return;
+
+ if (adapter->queue_irq[queue_idx]) {
+ irq_dispose_mapping(adapter->queue_irq[queue_idx]);
+ adapter->queue_irq[queue_idx] = 0;
+ }
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mappings - Drop virq mappings for queues 1..N
+ * @adapter: ibmveth adapter structure
+ *
+ * Bulk helper for paths that registered subordinate queues but never installed
+ * Linux IRQ handlers.
+ */
+static void
+ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ for (i = 1; i < adapter->num_rx_queues; i++)
+ ibmveth_dispose_subordinate_irq_mapping(adapter, i);
+}
+
+/**
+ * ibmveth_setup_rx_interrupts - Register IRQs and enable NAPI
+ * @adapter: ibmveth adapter structure
+ *
+ * Registers interrupt handlers for all RX queues, enables NAPI, then
+ * enables hypervisor interrupt delivery for multi-queue mode after
+ * every queue has a Linux handler installed. For multi-queue open the
+ * caller should replenish RX buffers before this helper so traffic
+ * during open is not dropped (PHYP only interrupts after a successful
+ * enqueue, which needs buffers). Single-queue open leaves PHYP masked
+ * here and kicks NAPI afterward (classic path: first poll posts then
+ * enables).
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i, rc, num = adapter->num_rx_queues;
+
+ for (i = 0; i < num; i++) {
+ if (!adapter->queue_irq[i]) {
+ netdev_err(netdev, "queue %d has invalid IRQ (0)\n", i);
+ rc = -EINVAL;
+ goto err_free_irqs;
+ }
+
+ rc = request_irq(adapter->queue_irq[i], ibmveth_interrupt,
+ 0, netdev->name, &adapter->napi[i]);
+ if (rc) {
+ netdev_err(netdev,
+ "request_irq() failed for irq 0x%x queue %d: %d\n",
+ adapter->queue_irq[i], i, rc);
+ goto err_free_irqs;
+ }
+ }
+
+ for (i = 0; i < num; i++)
+ napi_enable(&adapter->napi[i]);
+
+ if (adapter->multi_queue && num > 1) {
+ for (i = 0; i < num; i++) {
+ rc = ibmveth_enable_irq(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d, rc=%d\n",
+ i, rc);
+ while (--i >= 0)
+ ibmveth_disable_irq(adapter, i);
+ rc = -EIO;
+ goto err_disable_napi;
+ }
+ }
+ }
+
+ return 0;
+
+err_disable_napi:
+ /* PHYP unmask was rolled back above; disable NAPI before free_irq */
+ for (i = 0; i < num; i++)
+ napi_disable(&adapter->napi[i]);
+ for (i = 0; i < num; i++) {
+ if (adapter->queue_irq[i])
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+ }
+ goto err_dispose_mappings;
+
+err_free_irqs:
+ while (--i >= 0)
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+err_dispose_mappings:
+ /* Both setup failure paths own subordinate virq disposal. */
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+ return rc;
+}
+
+/**
+ * ibmveth_cleanup_rx_interrupts - Mask PHYP, disable NAPI, free IRQs
+ * @adapter: ibmveth adapter structure
+ *
+ * Tears down RX interrupt delivery for all queues. Mask PHYP before
+ * napi_disable so ibmveth_interrupt cannot return IRQ_HANDLED without
+ * masking (same storm window as scale-down). Safe for close and for
+ * open failure after setup_rx_interrupts() already unmasked PHYP.
+ */
+static void
+ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->queue_irq[i]) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ napi_disable(&adapter->napi[i]);
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->queue_irq[i])
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+ }
+
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+
+ /* Queue 0 uses netdev->irq; leave queue_irq[0] for next open. */
+}
+
+/**
+ * ibmveth_setup_single_rx_interrupt - Setup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to setup
+ *
+ * Registers the IRQ handler for one queue. Used during incremental
+ * scale-up when adding new RX queues. The caller publishes the queue,
+ * replenishes buffers, enables NAPI, then unmasks PHYP delivery.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ struct net_device *netdev = adapter->netdev;
+ int rc;
+
+ rc = request_irq(adapter->queue_irq[queue_idx], ibmveth_interrupt,
+ 0, netdev->name, &adapter->napi[queue_idx]);
+ if (rc) {
+ netdev_err(netdev, "request_irq() failed for queue %d: %d\n",
+ queue_idx, rc);
+ return rc;
+ }
+
+ netdev_dbg(netdev, "Setup IRQ %d for queue %d\n",
+ adapter->queue_irq[queue_idx], queue_idx);
+ return 0;
+}
+
+/**
+ * ibmveth_cleanup_single_rx_interrupt - Cleanup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to cleanup
+ *
+ * Frees the IRQ handler for one queue and releases the subordinate virq
+ * mapping. Used during incremental scale-down.
+ */
+static void
+ibmveth_cleanup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ if (adapter->queue_irq[queue_idx]) {
+ free_irq(adapter->queue_irq[queue_idx],
+ &adapter->napi[queue_idx]);
+ ibmveth_dispose_subordinate_irq_mapping(adapter, queue_idx);
+ netdev_dbg(adapter->netdev,
+ "Freed IRQ for queue %d\n", queue_idx);
+ }
+}
+
+/**
+ * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
+ * @adapter: ibmveth adapter structure
+ * @qindex: RX queue index
+ *
+ * Shared by the IRQ handler and process-context kick sites (open, resume,
+ * pool sysfs, poll_controller).
+ */
+static void ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
+ int qindex)
+{
+ struct napi_struct *napi = &adapter->napi[qindex];
+ unsigned long lpar_rc;
+
+ if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+ return;
+
+ if (napi_schedule_prep(napi)) {
+ lpar_rc = ibmveth_disable_irq(adapter, qindex);
+ WARN_ON(lpar_rc != H_SUCCESS);
+ __napi_schedule(napi);
+ }
+}
+
/* setup the initial settings for a buffer pool */
static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
u32 pool_index, u32 pool_size,
@@ -205,11 +785,75 @@ static inline void ibmveth_flush_buffer(void *addr, unsigned long length)
asm("dcbf %0,%1,1" :: "b" (addr), "r" (offset));
}
+/**
+ * ibmveth_add_logical_lan_buffers - Add receive buffers to hypervisor
+ * @adapter: ibmveth adapter structure
+ * @descs: array of buffer descriptors to add
+ * @filled: number of valid descriptors in the array
+ * @buff_size: size of each buffer (multi-queue mode only)
+ * @queue_index: RX queue index
+ *
+ * Return: hypervisor return code
+ */
+static long ibmveth_add_logical_lan_buffers(struct ibmveth_adapter *adapter,
+ union ibmveth_buf_desc *descs,
+ int filled,
+ unsigned long buff_size,
+ int queue_index)
+{
+ struct vio_dev *vdev = adapter->vdev;
+ unsigned long rc;
+
+ if (adapter->multi_queue) {
+ unsigned long buffersznum = (buff_size << 32) | filled;
+ unsigned long ioba[IBMVETH_MAX_RX_PER_HCALL / 2] = {0};
+ unsigned long handle = adapter->queue_handle[queue_index];
+ int i;
+
+ /* Pack descriptor addresses into ioba pairs.
+ * Each ioba holds two 32-bit addresses packed into 64 bits:
+ * - Even descriptors (0,2,4...) go in high 32 bits
+ * - Odd descriptors (1,3,5...) go in low 32 bits
+ */
+ for (i = 0; i < filled && i < IBMVETH_MAX_RX_PER_HCALL; i++) {
+ int pair_idx = i / 2;
+ int is_high = (i % 2 == 0);
+
+ if (is_high)
+ ioba[pair_idx] = (unsigned long)
+ descs[i].fields.address << 32;
+ else
+ ioba[pair_idx] |= descs[i].fields.address;
+ }
+
+ rc = h_add_logical_lan_buffers_queue(vdev->unit_address,
+ handle,
+ buffersznum,
+ ioba[0], ioba[1], ioba[2],
+ ioba[3], ioba[4], ioba[5]);
+ adapter->hcall_stats.add_bufs_queue++;
+ } else if (filled == 1) {
+ rc = h_add_logical_lan_buffer(vdev->unit_address,
+ descs[0].desc);
+ adapter->hcall_stats.add_buf++;
+ } else {
+ rc = h_add_logical_lan_buffers(vdev->unit_address,
+ descs[0].desc, descs[1].desc,
+ descs[2].desc, descs[3].desc,
+ descs[4].desc, descs[5].desc,
+ descs[6].desc, descs[7].desc);
+ adapter->hcall_stats.add_bufs++;
+ }
+
+ return rc;
+}
+
/* replenish the buffers for a pool. note that we don't need to
* skb_reserve these since they are used for incoming...
*/
static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
- struct ibmveth_buff_pool *pool)
+ struct ibmveth_buff_pool *pool,
+ int queue_index)
{
union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
u32 remaining = pool->size - atomic_read(&pool->available);
@@ -295,24 +939,16 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
if (!filled)
break;
- /* single buffer case*/
- if (filled == 1)
- lpar_rc = h_add_logical_lan_buffer(vdev->unit_address,
- descs[0].desc);
- else
- /* Multi-buffer hcall */
- lpar_rc = h_add_logical_lan_buffers(vdev->unit_address,
- descs[0].desc,
- descs[1].desc,
- descs[2].desc,
- descs[3].desc,
- descs[4].desc,
- descs[5].desc,
- descs[6].desc,
- descs[7].desc);
+ lpar_rc = ibmveth_add_logical_lan_buffers(adapter, descs,
+ filled,
+ pool->buff_size,
+ queue_index);
+
if (lpar_rc != H_SUCCESS) {
dev_warn_ratelimited(dev,
- "RX h_add_logical_lan failed: filled=%u, rc=%lu, batch=%u\n",
+ "RX h_add_logical_lan %s failed: filled=%u, rc=%lu, batch=%u\n",
+ adapter->multi_queue ?
+ "_queue" : "",
filled, lpar_rc, batch);
goto hcall_failure;
}
@@ -353,24 +989,23 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
}
adapter->replenish_add_buff_failure += filled;
- /*
- * If multi rx buffers hcall is no longer supported by FW
- * e.g. in the case of Live Partition Migration
- */
- if (batch > 1 && lpar_rc == H_FUNCTION) {
- /*
- * Instead of retry submit single buffer individually
- * here just set the max rx buffer per hcall to 1
- * buffers will be respleshed next time
- * when ibmveth_replenish_buffer_pool() is called again
- * with single-buffer case
- */
- netdev_info(adapter->netdev,
- "RX Multi buffers not supported by FW, rc=%lu\n",
- lpar_rc);
- adapter->rx_buffers_per_hcall = 1;
- netdev_info(adapter->netdev,
- "Next rx replesh will fall back to single-buffer hcall\n");
+ if (lpar_rc == H_FUNCTION) {
+ if (adapter->multi_queue) {
+ netdev_err(adapter->netdev,
+ "MQ buffer add H_FUNCTION (q=%d, batch=%d)\n",
+ queue_index, batch);
+ } else if (batch > 1) {
+ /*
+ * Live Partition Migration may drop multi-
+ * buffer support. Fall back to single-buffer
+ * on the next replenish; do not continue with
+ * a stale local batch size (infinite loop).
+ */
+ netdev_warn(adapter->netdev,
+ "Legacy batch add H_FUNCTION (batch=%d), fallback\n",
+ batch);
+ adapter->rx_buffers_per_hcall = 1;
+ }
}
break;
}
@@ -386,27 +1021,51 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
*/
static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
{
- __be64 *p = adapter->buffer_list_addr + 4096 - 8;
+ int i;
- adapter->rx_no_buffer = be64_to_cpup(p);
+ adapter->rx_no_buffer = 0;
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ __be64 *p = adapter->buffer_list_addr[i] + 4096 - 8;
+ u64 drops = be64_to_cpup(p);
+
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[i].no_buffer_drops = drops;
+ adapter->rx_no_buffer += drops;
+ }
}
/* replenish routine */
-static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
+static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
+ int queue_index)
{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+ unsigned long flags;
int i;
+ if (queue_index >= adapter->num_rx_queues) {
+ netdev_dbg(adapter->netdev,
+ "Skipping replenish for freed queue %d (num_queues=%d)\n",
+ queue_index, adapter->num_rx_queues);
+ return;
+ }
+
adapter->replenish_task_cycles++;
+ spin_lock_irqsave(&rxq->replenish_lock, flags);
+
for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
- struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[i];
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[queue_index][i];
- if (pool->active &&
+ if (pool->active && pool->free_map &&
(atomic_read(&pool->available) < pool->threshold))
- ibmveth_replenish_buffer_pool(adapter, pool);
+ ibmveth_replenish_buffer_pool(adapter, pool,
+ queue_index);
}
ibmveth_update_rx_no_buffer(adapter);
+
+ spin_unlock_irqrestore(&rxq->replenish_lock, flags);
}
/* empty and free ana buffer pool - also used to do cleanup in error paths */
@@ -443,10 +1102,303 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
}
}
+/**
+ * ibmveth_alloc_queue_buffer_pools - Allocate buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Allocates all active buffer pools for the specified queue.
+ * Pool metadata must be initialized before calling this function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int ibmveth_alloc_queue_buffer_pools(struct ibmveth_adapter *adapter,
+ int queue)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[queue][i];
+
+ if (!bpool->active)
+ continue;
+
+ if (ibmveth_alloc_buffer_pool(bpool)) {
+ netdev_err(netdev,
+ "pool %d/%d alloc failed (size=%u count=%u)\n",
+ i, queue,
+ bpool->buff_size,
+ bpool->size);
+ bpool->active = 0;
+
+ /* Free pools allocated so far for this queue */
+ while (--i >= 0) {
+ struct ibmveth_buff_pool *fpool =
+ &adapter->rx_buff_pool[queue][i];
+
+ if (fpool->active)
+ ibmveth_free_buffer_pool(adapter,
+ fpool);
+ }
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_queue_buffer_pools - Free buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Frees all active buffer pools for the specified queue.
+ */
+static void ibmveth_free_queue_buffer_pools(struct ibmveth_adapter *adapter,
+ int queue)
+{
+ int i;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[queue][i];
+
+ /* Free pool if it has allocated memory, regardless of
+ * active flag. Pools may have memory allocated but not
+ * marked active during queue scale-up, so we must check
+ * for actual allocations.
+ */
+ if (pool->free_map || pool->dma_addr || pool->skbuff)
+ ibmveth_free_buffer_pool(adapter, pool);
+ }
+}
+
+/**
+ * ibmveth_alloc_buffer_pools - Allocate buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Initializes pool metadata for queues 1-N from queue 0 settings,
+ * then allocates buffer pools for all queues using the helper function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i, q, rc;
+
+ /* Initialize pool metadata for queues 1-15 from queue 0 settings */
+ for (q = 1; q < adapter->num_rx_queues; q++) {
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *src =
+ &adapter->rx_buff_pool[0][i];
+ struct ibmveth_buff_pool *dst =
+ &adapter->rx_buff_pool[q][i];
+
+ dst->size = src->size;
+ dst->index = src->index;
+ dst->buff_size = src->buff_size;
+ dst->threshold = src->threshold;
+ dst->active = src->active;
+ }
+ }
+
+ /* Allocate actual buffers for all queues */
+ for (q = 0; q < adapter->num_rx_queues; q++) {
+ rc = ibmveth_alloc_queue_buffer_pools(adapter, q);
+ if (rc) {
+ /* Free pools for all previous queues */
+ while (--q >= 0)
+ ibmveth_free_queue_buffer_pools(adapter, q);
+ return rc;
+ }
+ }
+
+ netdev_dbg(netdev, "allocated buffer pools for %d queue(s)\n",
+ adapter->num_rx_queues);
+ return 0;
+}
+
+/**
+ * ibmveth_free_buffer_pools - Free buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees buffer pools for all queues using the helper function.
+ */
+static void
+ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
+{
+ int q;
+
+ /* Free buffer pools for all queues */
+ for (q = 0; q < adapter->num_rx_queues; q++)
+ ibmveth_free_queue_buffer_pools(adapter, q);
+
+ netdev_dbg(adapter->netdev, "freed buffer pools for %d queue(s)\n",
+ adapter->num_rx_queues);
+}
+
+/**
+ * ibmveth_alloc_single_rx_queue - Allocate resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to allocate
+ * @rxq_entries: Number of RX queue entries
+ *
+ * Allocates buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used during incremental scale-up without affecting existing queues.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx,
+ int rxq_entries)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+ int i, rc = -ENOMEM;
+
+ adapter->buffer_list_addr[queue_idx] =
+ (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->buffer_list_addr[queue_idx]) {
+ netdev_err(netdev, "unable to allocate buffer list for queue %d\n",
+ queue_idx);
+ return -ENOMEM;
+ }
+
+ adapter->rx_queue[queue_idx].queue_len =
+ sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+ adapter->rx_queue[queue_idx].queue_addr =
+ dma_alloc_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ &adapter->rx_queue[queue_idx].queue_dma,
+ GFP_KERNEL);
+ if (!adapter->rx_queue[queue_idx].queue_addr) {
+ netdev_err(netdev, "unable to allocate RX queue for queue %d\n",
+ queue_idx);
+ goto out_free_buflist;
+ }
+
+ adapter->buffer_list_dma[queue_idx] =
+ dma_map_single(dev, adapter->buffer_list_addr[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->buffer_list_dma[queue_idx])) {
+ netdev_err(netdev, "unable to map buffer list for queue %d\n",
+ queue_idx);
+ goto out_free_rxq;
+ }
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *src =
+ &adapter->rx_buff_pool[0][i];
+ struct ibmveth_buff_pool *dst =
+ &adapter->rx_buff_pool[queue_idx][i];
+
+ dst->size = src->size;
+ dst->index = src->index;
+ dst->buff_size = src->buff_size;
+ dst->threshold = src->threshold;
+ dst->active = src->active;
+ }
+
+ rc = ibmveth_alloc_queue_buffer_pools(adapter, queue_idx);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to allocate buffer pools for queue %d\n",
+ queue_idx);
+ goto out_unmap_buflist;
+ }
+
+ adapter->rx_queue[queue_idx].index = 0;
+ adapter->rx_queue[queue_idx].num_slots = rxq_entries;
+ adapter->rx_queue[queue_idx].toggle = 1;
+ spin_lock_init(&adapter->rx_queue[queue_idx].replenish_lock);
+
+ netdev_dbg(netdev,
+ "Allocated queue %d: buffer_list @ %p (DMA: 0x%llx), rx_queue @ %p (DMA: 0x%llx), %d entries\n",
+ queue_idx, adapter->buffer_list_addr[queue_idx],
+ (unsigned long long)adapter->buffer_list_dma[queue_idx],
+ adapter->rx_queue[queue_idx].queue_addr,
+ (unsigned long long)adapter->rx_queue[queue_idx].queue_dma,
+ rxq_entries);
+
+ return 0;
+
+out_unmap_buflist:
+ dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[queue_idx] = 0;
+out_free_rxq:
+ dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ adapter->rx_queue[queue_idx].queue_addr,
+ adapter->rx_queue[queue_idx].queue_dma);
+ adapter->rx_queue[queue_idx].queue_addr = NULL;
+out_free_buflist:
+ free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+ adapter->buffer_list_addr[queue_idx] = NULL;
+ return rc;
+}
+
+/**
+ * ibmveth_free_single_rx_queue - Free resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to free
+ *
+ * Frees buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used during incremental scale-down without affecting remaining queues.
+ */
+static void
+ibmveth_free_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx)
+{
+ struct device *dev = &adapter->vdev->dev;
+
+ ibmveth_free_queue_buffer_pools(adapter, queue_idx);
+
+ if (adapter->buffer_list_dma[queue_idx]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[queue_idx] = 0;
+ }
+
+ if (adapter->rx_queue[queue_idx].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ adapter->rx_queue[queue_idx].queue_addr,
+ adapter->rx_queue[queue_idx].queue_dma);
+ adapter->rx_queue[queue_idx].queue_addr = NULL;
+ }
+
+ if (adapter->buffer_list_addr[queue_idx]) {
+ free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+ adapter->buffer_list_addr[queue_idx] = NULL;
+ }
+
+ netdev_dbg(adapter->netdev, "Freed queue %d resources\n", queue_idx);
+}
+
+static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
+ int queue_index, u64 correlator)
+{
+ unsigned int pool = correlator >> 32;
+ unsigned int index = correlator & 0xffffffffUL;
+
+ return pool < IBMVETH_NUM_BUFF_POOLS &&
+ index < adapter->rx_buff_pool[queue_index][pool].size;
+}
+
+static void ibmveth_rxq_advance(struct ibmveth_rx_q *rxq)
+{
+ if (++rxq->index == rxq->num_slots) {
+ rxq->index = 0;
+ rxq->toggle = !rxq->toggle;
+ }
+}
+
/**
* ibmveth_remove_buffer_from_pool - remove a buffer from a pool
* @adapter: adapter instance
* @correlator: identifies pool and index
+ * @queue_index: RX queue index (0..num_rx_queues-1)
* @reuse: whether to reuse buffer
*
* Return:
@@ -455,24 +1407,20 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
* * %-EFAULT - pool and index map to null skb
*/
static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
- u64 correlator, bool reuse)
+ u64 correlator, int queue_index,
+ bool reuse)
{
unsigned int pool = correlator >> 32;
unsigned int index = correlator & 0xffffffffUL;
unsigned int free_index;
struct sk_buff *skb;
- if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
- WARN_ON(index >= adapter->rx_buff_pool[pool].size)) {
- schedule_work(&adapter->work);
+ if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))
return -EINVAL;
- }
- skb = adapter->rx_buff_pool[pool].skbuff[index];
- if (WARN_ON(!skb)) {
- schedule_work(&adapter->work);
+ skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
+ if (!skb)
return -EFAULT;
- }
/* if we are going to reuse the buffer then keep the pointers around
* but mark index as available. replenish will see the skb pointer and
@@ -482,48 +1430,52 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
/* remove the skb pointer to mark free. actual freeing is done
* by upper level networking after gro_receive
*/
- adapter->rx_buff_pool[pool].skbuff[index] = NULL;
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[queue_index][pool];
+
+ bpool->skbuff[index] = NULL;
dma_unmap_single(&adapter->vdev->dev,
- adapter->rx_buff_pool[pool].dma_addr[index],
- adapter->rx_buff_pool[pool].buff_size,
+ bpool->dma_addr[index],
+ bpool->buff_size,
DMA_FROM_DEVICE);
}
- free_index = adapter->rx_buff_pool[pool].producer_index;
- adapter->rx_buff_pool[pool].producer_index++;
- if (adapter->rx_buff_pool[pool].producer_index >=
- adapter->rx_buff_pool[pool].size)
- adapter->rx_buff_pool[pool].producer_index = 0;
- adapter->rx_buff_pool[pool].free_map[free_index] = index;
+ free_index = adapter->rx_buff_pool[queue_index][pool].producer_index;
+ adapter->rx_buff_pool[queue_index][pool].producer_index++;
+ if (adapter->rx_buff_pool[queue_index][pool].producer_index >=
+ adapter->rx_buff_pool[queue_index][pool].size)
+ adapter->rx_buff_pool[queue_index][pool].producer_index = 0;
+ adapter->rx_buff_pool[queue_index][pool].free_map[free_index] = index;
mb();
- atomic_dec(&(adapter->rx_buff_pool[pool].available));
+ atomic_dec(&adapter->rx_buff_pool[queue_index][pool].available);
return 0;
}
/* get the current buffer on the rx queue */
-static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter)
+static inline struct sk_buff *
+ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- u64 correlator = adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator;
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+ u64 correlator = rxq->queue_addr[rxq->index].correlator;
unsigned int pool = correlator >> 32;
unsigned int index = correlator & 0xffffffffUL;
- if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
- WARN_ON(index >= adapter->rx_buff_pool[pool].size)) {
- schedule_work(&adapter->work);
+ if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))
return NULL;
- }
- return adapter->rx_buff_pool[pool].skbuff[index];
+ return adapter->rx_buff_pool[queue_index][pool].skbuff[index];
}
/**
* ibmveth_rxq_harvest_buffer - Harvest buffer from pool
*
* @adapter: pointer to adapter
+ * @queue_index: RX queue index to harvest from
* @reuse: whether to reuse buffer
*
* Context: called from ibmveth_poll
@@ -533,28 +1485,81 @@ static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *ada
* * other - non-zero return from ibmveth_remove_buffer_from_pool
*/
static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
- bool reuse)
+ int queue_index, bool reuse)
{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
u64 cor;
int rc;
- cor = adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator;
- rc = ibmveth_remove_buffer_from_pool(adapter, cor, reuse);
- if (unlikely(rc))
+ cor = rxq->queue_addr[rxq->index].correlator;
+ rc = ibmveth_remove_buffer_from_pool(adapter, cor, queue_index, reuse);
+ if (unlikely(rc)) {
+ if (rc == -EINVAL || rc == -EFAULT)
+ goto advance;
return rc;
-
- if (++adapter->rx_queue.index == adapter->rx_queue.num_slots) {
- adapter->rx_queue.index = 0;
- adapter->rx_queue.toggle = !adapter->rx_queue.toggle;
}
+advance:
+ ibmveth_rxq_advance(rxq);
+
return 0;
}
+/**
+ * ibmveth_drain_rx_queue - Drain pending buffers from an RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Queue index to drain
+ *
+ * Recycles all pending buffers back to the per-queue buffer pools.
+ * Must be called with NAPI disabled for this queue.
+ *
+ * Return: Number of buffers drained
+ */
+static int
+ibmveth_drain_rx_queue(struct ibmveth_adapter *adapter, int queue_index)
+{
+ struct net_device *netdev = adapter->netdev;
+ int drained = 0;
+ int limit = adapter->rx_queue[queue_index].num_slots;
+ int rc;
+
+ netdev_dbg(netdev, "Draining RX queue %d (limit: %d slots)\n",
+ queue_index, limit);
+
+ while (drained < limit &&
+ ibmveth_rxq_pending_buffer(adapter, queue_index)) {
+ /* Match poll-side order before harvesting completion state. */
+ smp_rmb();
+ rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to harvest buffer from queue %d during drain: %d\n",
+ queue_index, rc);
+ break;
+ }
+ drained++;
+ }
+
+ if (drained > 0)
+ netdev_dbg(netdev, "Drained %d buffer(s) from RX queue %d\n",
+ drained, queue_index);
+ else
+ netdev_dbg(netdev, "No buffers to drain from RX queue %d\n",
+ queue_index);
+
+ return drained;
+}
+
static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
- dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
- adapter->tx_ltb_size, DMA_TO_DEVICE);
+ if (!adapter->tx_ltb_ptr[idx])
+ return;
+
+ if (adapter->tx_ltb_dma[idx]) {
+ dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
+ adapter->tx_ltb_size, DMA_TO_DEVICE);
+ adapter->tx_ltb_dma[idx] = 0;
+ }
kfree(adapter->tx_ltb_ptr[idx]);
adapter->tx_ltb_ptr[idx] = NULL;
}
@@ -577,16 +1582,63 @@ static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
"unable to DMA map tx long term buffer\n");
kfree(adapter->tx_ltb_ptr[idx]);
adapter->tx_ltb_ptr[idx] = NULL;
+ adapter->tx_ltb_dma[idx] = 0;
return -ENOMEM;
}
return 0;
}
+/**
+ * ibmveth_alloc_tx_resources - Allocate TX resources for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Allocates TX Long Term Buffers (LTBs) for all TX queues.
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_resources(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++) {
+ if (ibmveth_allocate_tx_ltb(adapter, i))
+ goto err_free_ltbs;
+ }
+
+ return 0;
+
+err_free_ltbs:
+ while (--i >= 0)
+ ibmveth_free_tx_ltb(adapter, i);
+ return -ENOMEM;
+}
+
+/**
+ * ibmveth_free_tx_resources - Free TX resources for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees TX Long Term Buffers (LTBs) for all TX queues.
+ */
+static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++)
+ ibmveth_free_tx_ltb(adapter, i);
+}
+
static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
- union ibmveth_buf_desc rxq_desc, u64 mac_address)
+ union ibmveth_buf_desc rxq_desc,
+ u64 mac_address)
{
int rc, try_again = 1;
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long buf_dma = adapter->buffer_list_dma[0];
+ unsigned long filter_dma = adapter->filter_list_dma;
+ u64 *qh0 = &adapter->queue_handle[0];
/*
* After a kexec the adapter will still be open, so our attempt to
@@ -594,13 +1646,25 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
* try again, but only once.
*/
retry:
- rc = h_register_logical_lan(adapter->vdev->unit_address,
- adapter->buffer_list_dma, rxq_desc.desc,
- adapter->filter_list_dma, mac_address);
+ /* In multi-queue mode, obtain a queue handle for queue 0 so all RX
+ * queues can use the same per-queue buffer hypercalls.
+ */
+ if (adapter->multi_queue) {
+ rc = h_register_logical_lan_with_handle(ua, buf_dma,
+ rxq_desc.desc,
+ filter_dma,
+ mac_address,
+ qh0);
+ } else {
+ rc = h_register_logical_lan(ua, buf_dma, rxq_desc.desc,
+ filter_dma, mac_address);
+ }
+ adapter->hcall_stats.reg_lan++;
if (rc != H_SUCCESS && try_again) {
do {
rc = h_free_logical_lan(adapter->vdev->unit_address);
+ adapter->hcall_stats.free_lan++;
} while (H_IS_LONG_BUSY(rc) || (rc == H_BUSY));
try_again = 0;
@@ -610,125 +1674,521 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
return rc;
}
+/**
+ * ibmveth_register_logical_lan_queue - Register subordinate queue with
+ * hypervisor
+ * @adapter: ibmveth adapter structure
+ * @rxq_desc: Receive queue descriptor
+ * @queue_index: RX queue index (1..N for subordinate queues)
+ *
+ * Registers a subordinate receive queue using H_REG_LOGICAL_LAN_QUEUE.
+ * On success, stores the queue handle and virtual IRQ in the adapter.
+ * If IRQ mapping fails after a successful hypervisor registration, the
+ * queue is freed before returning.
+ *
+ * Return: H_SUCCESS on success, negative errno on IRQ mapping failure,
+ * hypervisor error code otherwise
+ */
+static int
+ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,
+ union ibmveth_buf_desc rxq_desc,
+ int queue_index)
+{
+ unsigned long handle, hwirq;
+ unsigned int virq;
+ long lpar_rc;
+
+ netdev_dbg(adapter->netdev,
+ "Attempting to register queue %d: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\n",
+ queue_index, adapter->vdev->unit_address,
+ (unsigned long long)adapter->buffer_list_dma[queue_index],
+ (unsigned long long)rxq_desc.desc);
+
+ lpar_rc = h_reg_logical_lan_queue(adapter->vdev->unit_address,
+ adapter->buffer_list_dma[queue_index],
+ rxq_desc.desc, &handle, &hwirq);
+ adapter->hcall_stats.reg_lan_queue++;
+
+ if (lpar_rc == H_SUCCESS) {
+ virq = irq_create_mapping(NULL, hwirq);
+ if (!virq) {
+ unsigned long free_rc;
+ unsigned long ua = adapter->vdev->unit_address;
+
+ netdev_err(adapter->netdev,
+ "Failed to map IRQ for queue %d (hwirq=%lu)\n",
+ queue_index, hwirq);
+ do {
+ free_rc = h_free_logical_lan_queue(ua, handle);
+ } while (H_IS_LONG_BUSY(free_rc) ||
+ (free_rc == H_BUSY));
+ adapter->hcall_stats.free_lan_queue++;
+ if (free_rc != H_SUCCESS)
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan_queue failed for queue %d after IRQ map failure: rc=0x%lx\n",
+ queue_index, free_rc);
+ return -EINVAL;
+ }
+
+ adapter->queue_handle[queue_index] = handle;
+ adapter->queue_irq[queue_index] = virq;
+
+ netdev_dbg(adapter->netdev,
+ "queue %d registered: handle=0x%llx irq=%u\n",
+ queue_index, adapter->queue_handle[queue_index],
+ adapter->queue_irq[queue_index]);
+ return H_SUCCESS;
+ }
+
+ /*
+ * H_FUNCTION means firmware rejected this subordinate register
+ * (MQ unsupported). That is a hard open failure: do not clear
+ * multi_queue or claim single-queue fallback. Keep a specific
+ * log, then the generic failure lines below (no early return).
+ */
+ if (lpar_rc == H_FUNCTION)
+ netdev_err(adapter->netdev,
+ "h_reg_logical_lan_queue H_FUNCTION for queue %d (firmware MQ unsupported)\n",
+ queue_index);
+
+ netdev_err(adapter->netdev,
+ "h_reg_logical_lan_queue failed for queue %d with %ld\n",
+ queue_index, lpar_rc);
+ netdev_err(adapter->netdev,
+ "queue %d params: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\n",
+ queue_index, adapter->vdev->unit_address,
+ (unsigned long long)adapter->buffer_list_dma[queue_index],
+ (unsigned long long)rxq_desc.desc);
+
+ return lpar_rc;
+}
+
+/**
+ * ibmveth_register_single_rx_queue - Register one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to register (1..N)
+ * @mac_address: MAC address (unused; reserved for API symmetry)
+ *
+ * Builds the queue descriptor and registers with the hypervisor via
+ * ibmveth_register_logical_lan_queue().
+ *
+ * Return: 0 on success, -EINVAL if @queue_idx is invalid, -EIO on failure
+ */
+static int
+ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
+ int queue_idx, u64 mac_address)
+{
+ struct net_device *netdev = adapter->netdev;
+ union ibmveth_buf_desc rxq_desc;
+ long lpar_rc;
+
+ (void)mac_address;
+
+ if (WARN_ON(queue_idx < 1 || queue_idx >= IBMVETH_MAX_RX_QUEUES))
+ return -EINVAL;
+
+ rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+ adapter->rx_queue[queue_idx].queue_len;
+ rxq_desc.fields.address = adapter->rx_queue[queue_idx].queue_dma;
+
+ lpar_rc = ibmveth_register_logical_lan_queue(adapter, rxq_desc,
+ queue_idx);
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(netdev, "Failed to register queue %d: rc=0x%lx\n",
+ queue_idx, lpar_rc);
+ return -EIO;
+ }
+
+ netdev_dbg(netdev, "Registered queue %d with handle 0x%llx\n",
+ queue_idx, adapter->queue_handle[queue_idx]);
+
+ return 0;
+}
+
+/**
+ * ibmveth_deregister_single_rx_queue - Deregister one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to deregister (1..N)
+ *
+ * Deregisters a single queue via H_FREE_LOGICAL_LAN_QUEUE. Linux IRQ handler
+ * teardown and subordinate virq mapping disposal are owned by interrupt
+ * cleanup helpers; queue 0 is freed only through ibmveth_free_all_queues()
+ * (H_FREE_LOGICAL_LAN).
+ */
+static void
+ibmveth_deregister_single_rx_queue(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ unsigned long lpar_rc;
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long qh = adapter->queue_handle[queue_idx];
+
+ if (!qh)
+ return;
+
+ do {
+ lpar_rc = h_free_logical_lan_queue(ua, qh);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+ adapter->hcall_stats.free_lan_queue++;
+
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan_queue failed for queue %d: rc=0x%lx\n",
+ queue_idx, lpar_rc);
+ }
+
+ adapter->queue_handle[queue_idx] = 0;
+
+ netdev_dbg(adapter->netdev, "Deregistered queue %d\n", queue_idx);
+}
+
+/**
+ * ibmveth_resize_rx_queues_incremental - Resize RX queue count incrementally
+ * @adapter: ibmveth adapter structure
+ * @new_count: Target number of RX queues
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Adds or removes RX queues without tearing down the entire adapter.
+ * Active queues continue receiving during scale-up; scale-down drains
+ * excess queues before deregistering them with the hypervisor.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_resize_rx_queues_incremental(struct ibmveth_adapter *adapter,
+ int new_count, int rxq_entries)
+{
+ struct net_device *netdev = adapter->netdev;
+ u64 mac_address = ether_addr_to_u64(netdev->dev_addr);
+ int old_count = adapter->num_rx_queues;
+ int failed_queue;
+ int rc, i;
+
+ if (old_count == new_count) {
+ netdev_dbg(netdev, "RX queue count unchanged (%d), nothing to do\n",
+ old_count);
+ return 0;
+ }
+
+ if (new_count < 1 || new_count > IBMVETH_MAX_RX_QUEUES) {
+ netdev_err(netdev, "Invalid RX queue count %d (must be 1-%d)\n",
+ new_count, IBMVETH_MAX_RX_QUEUES);
+ return -EINVAL;
+ }
+
+ netdev_info(netdev, "Incrementally resizing RX queues: %d to %d\n",
+ old_count, new_count);
+
+ if (new_count > old_count) {
+ netdev_dbg(netdev, "Scale-up: adding queues %d-%d\n",
+ old_count, new_count - 1);
+
+ for (i = old_count; i < new_count; i++) {
+ rc = ibmveth_alloc_single_rx_queue(adapter, i,
+ rxq_entries);
+ if (rc) {
+ netdev_err(netdev, "Failed to allocate queue %d: %d\n",
+ i, rc);
+ goto cleanup_new_queues;
+ }
+
+ rc = ibmveth_register_single_rx_queue(adapter, i,
+ mac_address);
+ if (rc) {
+ netdev_err(netdev, "Failed to register queue %d: %d\n",
+ i, rc);
+ ibmveth_free_single_rx_queue(adapter, i);
+ goto cleanup_new_queues;
+ }
+
+ rc = ibmveth_setup_single_rx_interrupt(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to setup IRQ for queue %d: %d\n",
+ i, rc);
+ /* request_irq failed: mapped but no handler */
+ ibmveth_dispose_subordinate_irq_mapping(adapter,
+ i);
+ ibmveth_deregister_single_rx_queue(adapter, i);
+ ibmveth_free_single_rx_queue(adapter, i);
+ goto cleanup_new_queues;
+ }
+
+ /*
+ * Fully ready before PHYP delivery, matching open():
+ * publish -> replenish -> napi_enable -> enable_irq.
+ * That way ibmveth_interrupt() cannot run on an
+ * unpublished, empty, or NAPI-disabled queue.
+ */
+ adapter->num_rx_queues = i + 1;
+ ibmveth_replenish_task(adapter, i);
+ napi_enable(&adapter->napi[i]);
+
+ rc = ibmveth_enable_irq(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d: %d\n",
+ i, rc);
+ adapter->num_rx_queues = i;
+ napi_disable(&adapter->napi[i]);
+ ibmveth_cleanup_single_rx_interrupt(adapter, i);
+ ibmveth_deregister_single_rx_queue(adapter, i);
+ ibmveth_free_single_rx_queue(adapter, i);
+ goto cleanup_new_queues;
+ }
+ }
+
+ rc = netif_set_real_num_rx_queues(netdev, new_count);
+ if (rc) {
+ netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
+ new_count, rc);
+ goto cleanup_new_queues;
+ }
+ } else {
+ netdev_dbg(netdev, "Scale-down: removing queues %d-%d\n",
+ new_count, old_count - 1);
+
+ /*
+ * Mask PHYP delivery before napi_disable/drain. Otherwise
+ * ibmveth_interrupt returns IRQ_HANDLED without masking when
+ * NAPI is disabled, and the HV can storm during drain.
+ */
+ for (i = new_count; i < old_count; i++) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = new_count; i < old_count; i++)
+ napi_disable(&adapter->napi[i]);
+
+ for (i = new_count; i < old_count; i++)
+ ibmveth_drain_rx_queue(adapter, i);
+
+ synchronize_net();
+
+ rc = netif_set_real_num_rx_queues(netdev, new_count);
+ if (rc) {
+ netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
+ new_count, rc);
+ for (i = new_count; i < old_count; i++) {
+ ibmveth_replenish_task(adapter, i);
+ napi_enable(&adapter->napi[i]);
+ ibmveth_enable_irq(adapter, i);
+ }
+ return rc;
+ }
+
+ adapter->num_rx_queues = new_count;
+
+ for (i = new_count; i < old_count; i++) {
+ ibmveth_cleanup_single_rx_interrupt(adapter, i);
+ ibmveth_deregister_single_rx_queue(adapter, i);
+ ibmveth_free_single_rx_queue(adapter, i);
+ }
+ }
+
+ netdev_info(netdev, "Successfully resized to %d RX queues (incremental)\n",
+ adapter->num_rx_queues);
+
+ if (firmware_has_feature(FW_FEATURE_CMO))
+ vio_cmo_set_dev_desired(adapter->vdev,
+ ibmveth_get_desired_dma(adapter->vdev));
+
+ return 0;
+
+cleanup_new_queues:
+ failed_queue = i;
+ netdev_err(netdev,
+ "Scale-up failed at queue %d, cleaning up queues %d-%d\n",
+ failed_queue, old_count, failed_queue - 1);
+ for (i = old_count; i < failed_queue; i++) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = old_count; i < failed_queue; i++)
+ napi_disable(&adapter->napi[i]);
+
+ for (i = old_count; i < failed_queue; i++)
+ ibmveth_drain_rx_queue(adapter, i);
+
+ synchronize_net();
+
+ for (i = old_count; i < failed_queue; i++) {
+ ibmveth_cleanup_single_rx_interrupt(adapter, i);
+ ibmveth_deregister_single_rx_queue(adapter, i);
+ ibmveth_free_single_rx_queue(adapter, i);
+ }
+ adapter->num_rx_queues = old_count;
+ netdev_warn(netdev, "Keeping %d queues after scale-up failure\n",
+ old_count);
+ return rc;
+}
+
+/**
+ * ibmveth_free_all_queues - Free all RX queues at once
+ * @adapter: ibmveth adapter structure
+ *
+ * Uses H_FREE_LOGICAL_LAN to free all queues in one hypercall.
+ * Used during interface close and registration error cleanup.
+ *
+ * Clears queue handles only; queue_irq[] is released by
+ * ibmveth_cleanup_rx_interrupts() on close, or by
+ * ibmveth_dispose_subordinate_irq_mappings() on partial register failure.
+ */
+static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
+{
+ unsigned long lpar_rc;
+ int i;
+
+ netdev_dbg(adapter->netdev, "freeing all RX queues at once\n");
+
+ do {
+ lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+ adapter->hcall_stats.free_lan++;
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan failed: %ld\n", lpar_rc);
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ adapter->queue_handle[i] = 0;
+}
+
+/**
+ * ibmveth_register_rx_queues - Register RX queues with hypervisor
+ * @adapter: ibmveth adapter structure
+ * @mac_address: MAC address for device registration
+ *
+ * Registers queue 0 via ibmveth_register_logical_lan(), then subordinate
+ * queues 1..N when multi-queue mode is enabled.
+ *
+ * Return: 0 on success, -ENONET if queue 0 registration fails, -EIO on
+ * subordinate queue registration failure
+ */
+static int
+ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
+{
+ struct net_device *netdev = adapter->netdev;
+ union ibmveth_buf_desc rxq_desc;
+ unsigned long lpar_rc;
+ int i, rc;
+
+ rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+ adapter->rx_queue[0].queue_len;
+ rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
+ adapter->queue_irq[0] = netdev->irq;
+
+ rc = ibmveth_disable_irq(adapter, 0);
+ if (rc != H_SUCCESS)
+ netdev_dbg(netdev,
+ "Failed to disable IRQ for queue 0 before registration, rc=%d\n",
+ rc);
+
+ lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(netdev,
+ "h_register_logical_lan failed: %ld\n", lpar_rc);
+ netdev_err(netdev,
+ "buffer TCE:0x%llx filter TCE:0x%llx rxq desc:0x%llx MAC:0x%llx\n",
+ adapter->buffer_list_dma[0],
+ adapter->filter_list_dma,
+ rxq_desc.desc, mac_address);
+ return -ENONET;
+ }
+
+ if (adapter->num_rx_queues == 1 || !adapter->multi_queue) {
+ netdev_dbg(netdev,
+ "registered 1 RX queue with hypervisor (single-queue mode)\n");
+ return 0;
+ }
+
+ netdev_dbg(netdev, "Registering %d subordinate queues (1-%d)\n",
+ adapter->num_rx_queues - 1, adapter->num_rx_queues - 1);
+
+ for (i = 1; i < adapter->num_rx_queues; i++) {
+ rc = ibmveth_register_single_rx_queue(adapter, i, mac_address);
+ if (rc)
+ goto err_unregister;
+ }
+
+ netdev_dbg(netdev,
+ "registered %d RX queues with hypervisor (multi-queue mode)\n",
+ adapter->num_rx_queues);
+
+ return 0;
+
+err_unregister:
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+ ibmveth_free_all_queues(adapter);
+ return rc;
+}
+
static int ibmveth_open(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- u64 mac_address;
+ u64 mac_address = ether_addr_to_u64(netdev->dev_addr);
int rxq_entries = 1;
- unsigned long lpar_rc;
int rc;
- union ibmveth_buf_desc rxq_desc;
int i;
- struct device *dev;
netdev_dbg(netdev, "open starting\n");
- napi_enable(&adapter->napi);
-
- for(i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- rxq_entries += adapter->rx_buff_pool[i].size;
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
+ rxq_entries += adapter->rx_buff_pool[0][i].size;
- rc = -ENOMEM;
- adapter->buffer_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
- if (!adapter->buffer_list_addr) {
- netdev_err(netdev, "unable to allocate list pages\n");
+ rc = ibmveth_alloc_filter_list(adapter);
+ if (rc)
goto out;
- }
-
- adapter->filter_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
- if (!adapter->filter_list_addr) {
- netdev_err(netdev, "unable to allocate filter pages\n");
- goto out_free_buffer_list;
- }
-
- dev = &adapter->vdev->dev;
- adapter->rx_queue.queue_len = sizeof(struct ibmveth_rx_q_entry) *
- rxq_entries;
- adapter->rx_queue.queue_addr =
- dma_alloc_coherent(dev, adapter->rx_queue.queue_len,
- &adapter->rx_queue.queue_dma, GFP_KERNEL);
- if (!adapter->rx_queue.queue_addr)
+ rc = ibmveth_alloc_rx_queues(adapter, rxq_entries);
+ if (rc)
goto out_free_filter_list;
- adapter->buffer_list_dma = dma_map_single(dev,
- adapter->buffer_list_addr, 4096, DMA_BIDIRECTIONAL);
- if (dma_mapping_error(dev, adapter->buffer_list_dma)) {
- netdev_err(netdev, "unable to map buffer list pages\n");
+ rc = ibmveth_alloc_buffer_pools(adapter);
+ if (rc)
goto out_free_queue_mem;
- }
-
- adapter->filter_list_dma = dma_map_single(dev,
- adapter->filter_list_addr, 4096, DMA_BIDIRECTIONAL);
- if (dma_mapping_error(dev, adapter->filter_list_dma)) {
- netdev_err(netdev, "unable to map filter list pages\n");
- goto out_unmap_buffer_list;
- }
-
- for (i = 0; i < netdev->real_num_tx_queues; i++) {
- if (ibmveth_allocate_tx_ltb(adapter, i))
- goto out_free_tx_ltb;
- }
-
- adapter->rx_queue.index = 0;
- adapter->rx_queue.num_slots = rxq_entries;
- adapter->rx_queue.toggle = 1;
-
- mac_address = ether_addr_to_u64(netdev->dev_addr);
- rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
- adapter->rx_queue.queue_len;
- rxq_desc.fields.address = adapter->rx_queue.queue_dma;
-
- netdev_dbg(netdev, "buffer list @ 0x%p\n", adapter->buffer_list_addr);
- netdev_dbg(netdev, "filter list @ 0x%p\n", adapter->filter_list_addr);
- netdev_dbg(netdev, "receive q @ 0x%p\n", adapter->rx_queue.queue_addr);
-
- h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
-
- lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
+ rc = ibmveth_register_rx_queues(adapter, mac_address);
+ if (rc)
+ goto out_free_buffer_pools;
- if (lpar_rc != H_SUCCESS) {
- netdev_err(netdev, "h_register_logical_lan failed with %ld\n",
- lpar_rc);
- netdev_err(netdev, "buffer TCE:0x%llx filter TCE:0x%llx rxq "
- "desc:0x%llx MAC:0x%llx\n",
- adapter->buffer_list_dma,
- adapter->filter_list_dma,
- rxq_desc.desc,
- mac_address);
- rc = -ENONET;
- goto out_unmap_filter_list;
+ rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
+ if (rc) {
+ netdev_err(netdev, "failed to set number of rx queues\n");
+ goto out_unregister_queues;
}
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- if (!adapter->rx_buff_pool[i].active)
- continue;
- if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[i])) {
- netdev_err(netdev, "unable to alloc pool\n");
- adapter->rx_buff_pool[i].active = 0;
- rc = -ENOMEM;
- goto out_free_buffer_pools;
+ /*
+ * MQ: post buffers before setup_rx_interrupts() unmasks PHYP
+ * (avoids drops if traffic arrives during open; PHYP allows
+ * either order). Single-queue keeps the classic kick: setup
+ * (no unmask) then schedule_rx_queue() so the first poll
+ * replenishes and enables.
+ */
+ if (adapter->multi_queue && adapter->num_rx_queues > 1) {
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ netdev_dbg(netdev,
+ "initial replenish cycle for queue %d\n", i);
+ ibmveth_replenish_task(adapter, i);
}
}
- netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
- rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
- netdev);
- if (rc != 0) {
- netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
- netdev->irq, rc);
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+ rc = ibmveth_setup_rx_interrupts(adapter);
+ if (rc)
+ goto out_free_all_queues; /* setup already disposed IRQs */
- goto out_free_buffer_pools;
+ if (!(adapter->multi_queue && adapter->num_rx_queues > 1)) {
+ netdev_dbg(netdev, "initial replenish cycle\n");
+ ibmveth_schedule_rx_queue(adapter, 0);
}
- rc = -ENOMEM;
-
- netdev_dbg(netdev, "initial replenish cycle\n");
- ibmveth_interrupt(netdev->irq, netdev);
+ rc = ibmveth_alloc_tx_resources(adapter);
+ if (rc)
+ goto out_cleanup_rx_interrupts;
netif_tx_start_all_queues(netdev);
@@ -736,84 +2196,39 @@ static int ibmveth_open(struct net_device *netdev)
return 0;
+out_cleanup_rx_interrupts:
+ ibmveth_cleanup_rx_interrupts(adapter);
+ goto out_free_all_queues; /* cleanup already disposed IRQs */
+out_unregister_queues:
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+out_free_all_queues:
+ ibmveth_free_all_queues(adapter);
out_free_buffer_pools:
- while (--i >= 0) {
- if (adapter->rx_buff_pool[i].active)
- ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[i]);
- }
-out_unmap_filter_list:
- dma_unmap_single(dev, adapter->filter_list_dma, 4096,
- DMA_BIDIRECTIONAL);
-
-out_free_tx_ltb:
- while (--i >= 0) {
- ibmveth_free_tx_ltb(adapter, i);
- }
-
-out_unmap_buffer_list:
- dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
- DMA_BIDIRECTIONAL);
+ ibmveth_free_buffer_pools(adapter);
out_free_queue_mem:
- dma_free_coherent(dev, adapter->rx_queue.queue_len,
- adapter->rx_queue.queue_addr,
- adapter->rx_queue.queue_dma);
+ ibmveth_cleanup_rx_resources(adapter);
out_free_filter_list:
- free_page((unsigned long)adapter->filter_list_addr);
-out_free_buffer_list:
- free_page((unsigned long)adapter->buffer_list_addr);
+ ibmveth_free_filter_list(adapter);
out:
- napi_disable(&adapter->napi);
return rc;
}
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- struct device *dev = &adapter->vdev->dev;
- long lpar_rc;
- int i;
netdev_dbg(netdev, "close starting\n");
- napi_disable(&adapter->napi);
-
netif_tx_stop_all_queues(netdev);
- h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
-
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
- if (lpar_rc != H_SUCCESS) {
- netdev_err(netdev, "h_free_logical_lan failed with %lx, "
- "continuing with close\n", lpar_rc);
- }
-
- free_irq(netdev->irq, netdev);
-
+ /* PHYP mask + napi_disable + free_irq live in cleanup_rx_interrupts */
+ ibmveth_free_tx_resources(adapter);
+ ibmveth_cleanup_rx_interrupts(adapter);
ibmveth_update_rx_no_buffer(adapter);
-
- dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
- DMA_BIDIRECTIONAL);
- free_page((unsigned long)adapter->buffer_list_addr);
-
- dma_unmap_single(dev, adapter->filter_list_dma, 4096,
- DMA_BIDIRECTIONAL);
- free_page((unsigned long)adapter->filter_list_addr);
-
- dma_free_coherent(dev, adapter->rx_queue.queue_len,
- adapter->rx_queue.queue_addr,
- adapter->rx_queue.queue_dma);
-
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- if (adapter->rx_buff_pool[i].active)
- ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[i]);
-
- for (i = 0; i < netdev->real_num_tx_queues; i++)
- ibmveth_free_tx_ltb(adapter, i);
+ ibmveth_free_all_queues(adapter);
+ ibmveth_free_buffer_pools(adapter);
+ ibmveth_cleanup_rx_resources(adapter);
+ ibmveth_free_filter_list(adapter);
netdev_dbg(netdev, "close complete\n");
@@ -1073,22 +2488,136 @@ static int ibmveth_set_features(struct net_device *dev,
return rc1 ? rc1 : rc2;
}
+/*
+ * Sum per-queue counters for rare ethtool reads. Do not write adapter
+ * globals on the hot path (ibmvnic-style); with qstats allocated for the
+ * adapter lifetime, these sums remain meaningful across ifdown/up.
+ */
+static u64 ibmveth_sum_rx_invalid_buffers(struct ibmveth_adapter *adapter)
+{
+ u64 total = 0;
+ int i;
+
+ if (!adapter->rx_qstats)
+ return adapter->rx_invalid_buffer;
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ total += adapter->rx_qstats[i].invalid_buffers;
+
+ return total;
+}
+
+static u64 ibmveth_sum_rx_large_packets(struct ibmveth_adapter *adapter)
+{
+ u64 total = 0;
+ int i;
+
+ if (!adapter->rx_qstats)
+ return adapter->rx_large_packets;
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ total += adapter->rx_qstats[i].large_packets;
+
+ return total;
+}
+
+static u64 ibmveth_sum_tx_large_packets(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ u64 total = 0;
+ int i;
+
+ if (!adapter->tx_qstats)
+ return adapter->tx_large_packets;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++)
+ total += adapter->tx_qstats[i].large_packets;
+
+ return total;
+}
+
+static u64 ibmveth_sum_tx_send_failed(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ u64 total = 0;
+ int i;
+
+ if (!adapter->tx_qstats)
+ return adapter->tx_send_failed;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++)
+ total += adapter->tx_qstats[i].send_failures;
+
+ return total;
+}
+
+static u64 ibmveth_ethtool_adapter_stat(struct ibmveth_adapter *adapter,
+ int index)
+{
+ unsigned long offset = ibmveth_stats[index].offset;
+
+ if (offset == IBMVETH_STAT_OFF(rx_invalid_buffer))
+ return ibmveth_sum_rx_invalid_buffers(adapter);
+ if (offset == IBMVETH_STAT_OFF(rx_large_packets))
+ return ibmveth_sum_rx_large_packets(adapter);
+ if (offset == IBMVETH_STAT_OFF(tx_large_packets))
+ return ibmveth_sum_tx_large_packets(adapter);
+ if (offset == IBMVETH_STAT_OFF(tx_send_failed))
+ return ibmveth_sum_tx_send_failed(adapter);
+
+ return IBMVETH_GET_STAT(adapter, offset);
+}
+
static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)
{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ u8 *p = data;
int i;
if (stringset != ETH_SS_STATS)
return;
- for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++, data += ETH_GSTRING_LEN)
- memcpy(data, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++) {
+ memcpy(p, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ ethtool_sprintf(&p, "rx%d_packets", i);
+ ethtool_sprintf(&p, "rx%d_bytes", i);
+ ethtool_sprintf(&p, "rx%d_interrupts", i);
+ ethtool_sprintf(&p, "rx%d_polls", i);
+ ethtool_sprintf(&p, "rx%d_large_packets", i);
+ ethtool_sprintf(&p, "rx%d_invalid_buffers", i);
+ ethtool_sprintf(&p, "rx%d_no_buffer_drops", i);
+ }
+
+ for (i = 0; i < dev->real_num_tx_queues; i++) {
+ ethtool_sprintf(&p, "tx%d_packets", i);
+ ethtool_sprintf(&p, "tx%d_bytes", i);
+ ethtool_sprintf(&p, "tx%d_large_packets", i);
+ ethtool_sprintf(&p, "tx%d_dropped_packets", i);
+ ethtool_sprintf(&p, "tx%d_send_failures", i);
+ ethtool_sprintf(&p, "tx%d_checksum_offload", i);
+ }
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ ethtool_sprintf(&p, "pool%d_size", i);
+ ethtool_sprintf(&p, "pool%d_active", i);
+ ethtool_sprintf(&p, "pool%d_available", i);
+ }
}
static int ibmveth_get_sset_count(struct net_device *dev, int sset)
{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+
switch (sset) {
case ETH_SS_STATS:
- return ARRAY_SIZE(ibmveth_stats);
+ return ARRAY_SIZE(ibmveth_stats) +
+ adapter->num_rx_queues * IBMVETH_NUM_RX_QSTATS +
+ dev->real_num_tx_queues * IBMVETH_NUM_TX_QSTATS +
+ IBMVETH_NUM_BUFF_POOLS * 3;
default:
return -EOPNOTSUPP;
}
@@ -1097,44 +2626,154 @@ static int ibmveth_get_sset_count(struct net_device *dev, int sset)
static void ibmveth_get_ethtool_stats(struct net_device *dev,
struct ethtool_stats *stats, u64 *data)
{
- int i;
struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i, j;
for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++)
- data[i] = IBMVETH_GET_STAT(adapter, ibmveth_stats[i].offset);
+ data[i] = ibmveth_ethtool_adapter_stat(adapter, i);
+
+ for (j = 0; j < adapter->num_rx_queues; j++) {
+ if (adapter->rx_qstats) {
+ data[i++] = adapter->rx_qstats[j].packets;
+ data[i++] = adapter->rx_qstats[j].bytes;
+ data[i++] = adapter->rx_qstats[j].interrupts;
+ data[i++] = adapter->rx_qstats[j].polls;
+ data[i++] = adapter->rx_qstats[j].large_packets;
+ data[i++] = adapter->rx_qstats[j].invalid_buffers;
+ data[i++] = adapter->rx_qstats[j].no_buffer_drops;
+ } else {
+ i += IBMVETH_NUM_RX_QSTATS;
+ }
+ }
+
+ for (j = 0; j < dev->real_num_tx_queues; j++) {
+ if (adapter->tx_qstats) {
+ data[i++] = adapter->tx_qstats[j].packets;
+ data[i++] = adapter->tx_qstats[j].bytes;
+ data[i++] = adapter->tx_qstats[j].large_packets;
+ data[i++] = adapter->tx_qstats[j].dropped_packets;
+ data[i++] = adapter->tx_qstats[j].send_failures;
+ data[i++] = adapter->tx_qstats[j].checksum_offload;
+ } else {
+ i += IBMVETH_NUM_TX_QSTATS;
+ }
+ }
+
+ for (j = 0; j < IBMVETH_NUM_BUFF_POOLS; j++) {
+ data[i++] = adapter->rx_buff_pool[0][j].size;
+ data[i++] = adapter->rx_buff_pool[0][j].active;
+ data[i++] = atomic_read(&adapter->rx_buff_pool[0][j].available);
+ }
}
static void ibmveth_get_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
channels->max_tx = ibmveth_real_max_tx_queues();
channels->tx_count = netdev->real_num_tx_queues;
- channels->max_rx = netdev->real_num_rx_queues;
- channels->rx_count = netdev->real_num_rx_queues;
+ if (adapter->multi_queue)
+ channels->max_rx = IBMVETH_MAX_RX_QUEUES;
+ else
+ channels->max_rx = 1;
+ channels->rx_count = adapter->num_rx_queues;
+}
+
+/**
+ * ibmveth_resize_rx_channels - Validate and apply a new RX queue count
+ * @adapter: ibmveth adapter structure
+ * @goal_rx: desired RX queue count
+ *
+ * When the interface is up, resize live queues via
+ * ibmveth_resize_rx_queues_incremental(). When down, only stash
+ * adapter->num_rx_queues for the next open().
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,
+ unsigned int goal_rx)
+{
+ struct net_device *netdev = adapter->netdev;
+ unsigned int old_rx = adapter->num_rx_queues;
+ int rxq_entries;
+ int rc;
+
+ if (goal_rx > 1 && !adapter->multi_queue) {
+ netdev_err(netdev,
+ "Cannot resize to %u RX queues: multi-queue mode not supported by firmware\n",
+ goal_rx);
+ return -EOPNOTSUPP;
+ }
+
+ if (goal_rx < 1 || goal_rx > IBMVETH_MAX_RX_QUEUES) {
+ netdev_err(netdev,
+ "Invalid RX queue count %u (must be 1-%d)\n",
+ goal_rx, IBMVETH_MAX_RX_QUEUES);
+ return -EINVAL;
+ }
+
+ if (goal_rx == old_rx)
+ return 0;
+
+ if (!(netdev->flags & IFF_UP)) {
+ adapter->num_rx_queues = goal_rx;
+ return 0;
+ }
+
+ rxq_entries = adapter->rx_queue[0].num_slots;
+ rc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,
+ rxq_entries);
+ if (rc)
+ netdev_err(netdev, "Failed to resize RX queues: %d\n", rc);
+ return rc;
}
static int ibmveth_set_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- unsigned int old = netdev->real_num_tx_queues,
- goal = channels->tx_count;
+ unsigned int goal_rx = channels->rx_count;
+ unsigned int old_tx = netdev->real_num_tx_queues;
+ unsigned int goal_tx = channels->tx_count;
int rc, i;
/* If ndo_open has not been called yet then don't allocate, just set
* desired netdev_queue's and return
*/
- if (!(netdev->flags & IFF_UP))
- return netif_set_real_num_tx_queues(netdev, goal);
+ if (!(netdev->flags & IFF_UP)) {
+ if (goal_tx != old_tx) {
+ rc = netif_set_real_num_tx_queues(netdev, goal_tx);
+ if (rc)
+ return rc;
+ }
+
+ /* Stash desired RX count only after TX succeeds (or was
+ * already correct); open() publishes it via
+ * netif_set_real_num_rx_queues().
+ */
+ return ibmveth_resize_rx_channels(adapter, goal_rx);
+ }
+
+ rc = ibmveth_resize_rx_channels(adapter, goal_rx);
+ if (rc)
+ return rc;
/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated
* but we may need to alloc/free the ltb's.
*/
+ if (goal_tx == old_tx)
+ return 0;
+
netif_tx_stop_all_queues(netdev);
- /* Allocate any queue that we need */
- for (i = old; i < goal; i++) {
+ /* Allocate any queue that we need. Initialize i to old_tx so a
+ * scale-down path that never enters the loop still has defined
+ * bounds if set_real_num_tx_queues() fails.
+ */
+ i = old_tx;
+ for (; i < goal_tx; i++) {
if (adapter->tx_ltb_ptr[i])
continue;
@@ -1144,20 +2783,20 @@ static int ibmveth_set_channels(struct net_device *netdev,
/* if something goes wrong, free everything we just allocated */
netdev_err(netdev, "Failed to allocate more tx queues, returning to %d queues\n",
- old);
- goal = old;
- old = i;
+ old_tx);
+ goal_tx = old_tx;
+ old_tx = i;
break;
}
- rc = netif_set_real_num_tx_queues(netdev, goal);
+ rc = netif_set_real_num_tx_queues(netdev, goal_tx);
if (rc) {
netdev_err(netdev, "Failed to set real tx queues, returning to %d queues\n",
- old);
- goal = old;
- old = i;
+ old_tx);
+ goal_tx = old_tx;
+ old_tx = i;
}
/* Free any that are no longer needed */
- for (i = old; i > goal; i--) {
+ for (i = old_tx; i > goal_tx; i--) {
if (adapter->tx_ltb_ptr[i - 1])
ibmveth_free_tx_ltb(adapter, i - 1);
}
@@ -1209,12 +2848,15 @@ static int ibmveth_send(struct ibmveth_adapter *adapter,
return 1;
}
+ adapter->hcall_stats.send_lan++;
return 0;
}
static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
- struct net_device *netdev)
+ struct ibmveth_adapter *adapter,
+ int queue_num)
{
+ struct net_device *netdev = adapter->netdev;
struct ethhdr *ether_header;
int ret = 0;
@@ -1222,7 +2864,8 @@ static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
if (ether_addr_equal(ether_header->h_dest, netdev->dev_addr)) {
netdev_dbg(netdev, "veth doesn't support loopback packets, dropping packet.\n");
- netdev->stats.tx_dropped++;
+ if (adapter->tx_qstats)
+ adapter->tx_qstats[queue_num].dropped_packets++;
ret = -EOPNOTSUPP;
}
@@ -1238,7 +2881,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
int i, queue_num = skb_get_queue_mapping(skb);
unsigned long mss = 0;
- if (ibmveth_is_packet_unsupported(skb, netdev))
+ if (ibmveth_is_packet_unsupported(skb, adapter, queue_num))
goto out;
/* veth can't checksum offload UDP */
if (skb->ip_summed == CHECKSUM_PARTIAL &&
@@ -1249,7 +2892,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
skb_checksum_help(skb)) {
netdev_err(netdev, "tx: failed to checksum packet\n");
- netdev->stats.tx_dropped++;
+ adapter->tx_qstats[queue_num].dropped_packets++;
goto out;
}
@@ -1261,6 +2904,8 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
desc_flags |= (IBMVETH_BUF_NO_CSUM | IBMVETH_BUF_CSUM_GOOD);
+ adapter->tx_qstats[queue_num].checksum_offload++;
+
/* Need to zero out the checksum */
buf[0] = 0;
buf[1] = 0;
@@ -1272,7 +2917,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (skb->ip_summed == CHECKSUM_PARTIAL && skb_is_gso(skb)) {
if (adapter->fw_large_send_support) {
mss = (unsigned long)skb_shinfo(skb)->gso_size;
- adapter->tx_large_packets++;
+ adapter->tx_qstats[queue_num].large_packets++;
} else if (!skb_is_gso_v6(skb)) {
/* Put -1 in the IP checksum to tell phyp it
* is a largesend packet. Put the mss in
@@ -1281,7 +2926,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
ip_hdr(skb)->check = 0xffff;
tcp_hdr(skb)->check =
cpu_to_be16(skb_shinfo(skb)->gso_size);
- adapter->tx_large_packets++;
+ adapter->tx_qstats[queue_num].large_packets++;
}
}
@@ -1289,7 +2934,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(skb->len > adapter->tx_ltb_size)) {
netdev_err(adapter->netdev, "tx: packet size (%u) exceeds ltb (%u)\n",
skb->len, adapter->tx_ltb_size);
- netdev->stats.tx_dropped++;
+ adapter->tx_qstats[queue_num].dropped_packets++;
goto out;
}
memcpy(adapter->tx_ltb_ptr[queue_num], skb->data, skb_headlen(skb));
@@ -1306,7 +2951,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(total_bytes != skb->len)) {
netdev_err(adapter->netdev, "tx: incorrect packet len copied into ltb (%u != %u)\n",
skb->len, total_bytes);
- netdev->stats.tx_dropped++;
+ adapter->tx_qstats[queue_num].dropped_packets++;
goto out;
}
desc.fields.flags_len = desc_flags | skb->len;
@@ -1315,18 +2960,17 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
dma_wmb();
if (ibmveth_send(adapter, desc.desc, mss)) {
- adapter->tx_send_failed++;
- netdev->stats.tx_dropped++;
+ adapter->tx_qstats[queue_num].send_failures++;
+ adapter->tx_qstats[queue_num].dropped_packets++;
} else {
- netdev->stats.tx_packets++;
- netdev->stats.tx_bytes += skb->len;
+ adapter->tx_qstats[queue_num].packets++;
+ adapter->tx_qstats[queue_num].bytes += skb->len;
}
out:
dev_consume_skb_any(skb);
return NETDEV_TX_OK;
-
}
static void ibmveth_rx_mss_helper(struct sk_buff *skb, u16 mss, int lrg_pkt)
@@ -1447,36 +3091,99 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
static int ibmveth_poll(struct napi_struct *napi, int budget)
{
- struct ibmveth_adapter *adapter =
- container_of(napi, struct ibmveth_adapter, napi);
- struct net_device *netdev = adapter->netdev;
+ struct net_device *netdev = napi->dev;
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
int frames_processed = 0;
unsigned long lpar_rc;
+ int queue_index, rc;
u16 mss = 0;
+ queue_index = napi - adapter->napi;
+
+ if (WARN_ON(queue_index < 0 || queue_index >= adapter->num_rx_queues))
+ return 0;
+
+ if (!netif_running(netdev) || napi_disable_pending(napi)) {
+ napi_complete_done(napi, 0);
+ return 0;
+ }
+
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[queue_index].polls++;
+
restart_poll:
while (frames_processed < budget) {
- if (!ibmveth_rxq_pending_buffer(adapter))
+ if (!netif_running(netdev) || napi_disable_pending(napi))
+ break;
+
+ if (!ibmveth_rxq_pending_buffer(adapter, queue_index))
break;
smp_rmb();
- if (!ibmveth_rxq_buffer_valid(adapter)) {
+ if (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {
wmb(); /* suggested by larson1 */
- adapter->rx_invalid_buffer++;
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[queue_index]
+ .invalid_buffers++;
+ else
+ adapter->rx_invalid_buffer++;
netdev_dbg(netdev, "recycling invalid buffer\n");
- if (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))
+ rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index, true);
+ if (unlikely(rc))
break;
} else {
struct sk_buff *skb, *new_skb;
- int length = ibmveth_rxq_frame_length(adapter);
- int offset = ibmveth_rxq_frame_offset(adapter);
- int csum_good = ibmveth_rxq_csum_good(adapter);
- int lrg_pkt = ibmveth_rxq_large_packet(adapter);
+ int length = ibmveth_rxq_frame_length(adapter,
+ queue_index);
+ int offset = ibmveth_rxq_frame_offset(adapter,
+ queue_index);
+ int csum_good = ibmveth_rxq_csum_good(adapter,
+ queue_index);
+ int lrg_pkt = ibmveth_rxq_large_packet(adapter,
+ queue_index);
__sum16 iph_check = 0;
- skb = ibmveth_rxq_get_buffer(adapter);
- if (unlikely(!skb))
- break;
+ skb = ibmveth_rxq_get_buffer(adapter, queue_index);
+ if (unlikely(!skb)) {
+ if (net_ratelimit())
+ netdev_err(netdev,
+ "bad correlator on queue %d, skipping slot\n",
+ queue_index);
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[queue_index]
+ .invalid_buffers++;
+ else
+ adapter->rx_invalid_buffer++;
+ rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
+ true);
+ if (unlikely(rc))
+ break;
+ continue;
+ }
+
+ if (unlikely((unsigned int)offset +
+ (unsigned int)length >
+ skb_tailroom(skb))) {
+ if (net_ratelimit())
+ netdev_err(netdev,
+ "RX frame %u+%u exceeds buffer %u on queue %d, dropping\n",
+ offset, length,
+ skb_tailroom(skb),
+ queue_index);
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[queue_index]
+ .invalid_buffers++;
+ else
+ adapter->rx_invalid_buffer++;
+ rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
+ true);
+ if (unlikely(rc))
+ break;
+ continue;
+ }
/* if the large packet bit is set in the rx queue
* descriptor, the mss will be written by PHYP eight
@@ -1499,12 +3206,18 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
length);
if (rx_flush)
ibmveth_flush_buffer(skb->data,
- length + offset);
- if (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))
+ length + offset);
+ rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
+ true);
+ if (unlikely(rc))
break;
skb = new_skb;
} else {
- if (unlikely(ibmveth_rxq_harvest_buffer(adapter, false)))
+ rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
+ false);
+ if (unlikely(rc))
break;
skb_reserve(skb, offset);
}
@@ -1524,7 +3237,11 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
if ((length > netdev->mtu + ETH_HLEN) ||
lrg_pkt || iph_check == 0xffff) {
ibmveth_rx_mss_helper(skb, mss, lrg_pkt);
- adapter->rx_large_packets++;
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[queue_index]
+ .large_packets++;
+ else
+ adapter->rx_large_packets++;
}
if (csum_good) {
@@ -1534,16 +3251,25 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
napi_gro_receive(napi, skb); /* send it up */
- netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += length;
+ if (adapter->rx_qstats) {
+ adapter->rx_qstats[queue_index].packets++;
+ adapter->rx_qstats[queue_index].bytes += length;
+ }
+
frames_processed++;
}
}
- ibmveth_replenish_task(adapter);
+ ibmveth_replenish_task(adapter, queue_index);
- if (frames_processed == budget)
+ if (frames_processed == budget) {
+ if (!netif_running(netdev) || napi_disable_pending(napi)) {
+ napi_complete_done(napi, frames_processed);
+ /* After complete_done, must not return full budget. */
+ return frames_processed ? frames_processed - 1 : 0;
+ }
goto out;
+ }
if (!napi_complete_done(napi, frames_processed))
goto out;
@@ -1551,15 +3277,21 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
/* We think we are done - reenable interrupts,
* then check once more to make sure we are done.
*/
- lpar_rc = h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_ENABLE);
- if (WARN_ON(lpar_rc != H_SUCCESS)) {
+ lpar_rc = ibmveth_enable_irq(adapter, queue_index);
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d (rc=0x%lx), scheduling reset\n",
+ queue_index, lpar_rc);
schedule_work(&adapter->work);
goto out;
}
- if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule(napi)) {
- lpar_rc = h_vio_signal(adapter->vdev->unit_address,
- VIO_IRQ_DISABLE);
+ if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
+ netif_running(netdev) &&
+ !napi_disable_pending(napi) &&
+ napi_schedule(napi)) {
+ lpar_rc = ibmveth_disable_irq(adapter, queue_index);
+ WARN_ON(lpar_rc != H_SUCCESS);
goto restart_poll;
}
@@ -1569,16 +3301,19 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
{
- struct net_device *netdev = dev_instance;
+ struct napi_struct *napi = dev_instance;
+ struct net_device *netdev = napi->dev;
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- unsigned long lpar_rc;
+ int qindex;
- if (napi_schedule_prep(&adapter->napi)) {
- lpar_rc = h_vio_signal(adapter->vdev->unit_address,
- VIO_IRQ_DISABLE);
- WARN_ON(lpar_rc != H_SUCCESS);
- __napi_schedule(&adapter->napi);
- }
+ qindex = napi - adapter->napi;
+ if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+ return IRQ_NONE;
+
+ if (adapter->rx_qstats)
+ adapter->rx_qstats[qindex].interrupts++;
+
+ ibmveth_schedule_rx_queue(adapter, qindex);
return IRQ_HANDLED;
}
@@ -1645,7 +3380,7 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
int need_restart = 0;
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size)
+ if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size)
break;
if (i == IBMVETH_NUM_BUFF_POOLS)
@@ -1660,9 +3395,9 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
/* Look for an active buffer pool that can hold the new MTU */
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- adapter->rx_buff_pool[i].active = 1;
+ adapter->rx_buff_pool[0][i].active = 1;
- if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size) {
+ if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size) {
WRITE_ONCE(dev->mtu, new_mtu);
vio_cmo_set_dev_desired(viodev,
ibmveth_get_desired_dma
@@ -1683,8 +3418,14 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
#ifdef CONFIG_NET_POLL_CONTROLLER
static void ibmveth_poll_controller(struct net_device *dev)
{
- ibmveth_replenish_task(netdev_priv(dev));
- ibmveth_interrupt(dev->irq, dev);
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i;
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ ibmveth_replenish_task(adapter, i);
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ ibmveth_schedule_rx_queue(adapter, i);
}
#endif
@@ -1693,8 +3434,7 @@ static void ibmveth_poll_controller(struct net_device *dev)
*
* @vdev: struct vio_dev for the device whose desired IO mem is to be returned
*
- * Return value:
- * Number of bytes of IO data the driver will need to perform well.
+ * Return: Number of bytes of IO data the driver will need to perform well.
*/
static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
{
@@ -1702,8 +3442,7 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
struct ibmveth_adapter *adapter;
struct iommu_table *tbl;
unsigned long ret;
- int i;
- int rxqentries = 1;
+ int i, q;
tbl = get_iommu_table_base(&vdev->dev);
@@ -1718,18 +3457,25 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
/* add size of mapped tx buffers */
ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- /* add the size of the active receive buffers */
- if (adapter->rx_buff_pool[i].active)
- ret +=
- adapter->rx_buff_pool[i].size *
- IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[i].
- buff_size, tbl);
- rxqentries += adapter->rx_buff_pool[i].size;
- }
- /* add the size of the receive queue entries */
- ret += IOMMU_PAGE_ALIGN(
- rxqentries * sizeof(struct ibmveth_rx_q_entry), tbl);
+ for (q = 0; q < adapter->num_rx_queues; q++) {
+ int rxqentries = 1;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ /* add the size of the active receive buffers */
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[q][i];
+
+ /* add the size of the active receive buffers */
+ if (bpool->active)
+ ret += bpool->size *
+ IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
+ rxqentries += bpool->size;
+ }
+
+ /* add the size of the receive queue entries */
+ ret += IOMMU_PAGE_ALIGN(rxqentries *
+ sizeof(struct ibmveth_rx_q_entry), tbl);
+ }
return ret;
}
@@ -1777,6 +3523,40 @@ static netdev_features_t ibmveth_features_check(struct sk_buff *skb,
return vlan_features_check(skb, features);
}
+/**
+ * ibmveth_get_stats64 - Return aggregated per-queue statistics
+ * @dev: network device
+ * @stats: rtnl link statistics storage
+ *
+ * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.
+ * Callers use ndo_get_stats64(); avoid updating netdev->stats on the
+ * xmit/poll paths to keep per-queue counters off the hot cache line.
+ */
+static void ibmveth_get_stats64(struct net_device *dev,
+ struct rtnl_link_stats64 *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i;
+
+ if (adapter->rx_qstats) {
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ stats->rx_packets += adapter->rx_qstats[i].packets;
+ stats->rx_bytes += adapter->rx_qstats[i].bytes;
+ }
+ }
+
+ if (adapter->tx_qstats) {
+ for (i = 0; i < dev->real_num_tx_queues; i++) {
+ stats->tx_packets += adapter->tx_qstats[i].packets;
+ stats->tx_bytes += adapter->tx_qstats[i].bytes;
+ stats->tx_dropped +=
+ adapter->tx_qstats[i].dropped_packets;
+ }
+ }
+
+ stats->tx_errors = dev->stats.tx_errors;
+}
+
static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_open = ibmveth_open,
.ndo_stop = ibmveth_close,
@@ -1789,14 +3569,75 @@ static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = ibmveth_set_mac_addr,
.ndo_features_check = ibmveth_features_check,
+ .ndo_get_stats64 = ibmveth_get_stats64,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = ibmveth_poll_controller,
#endif
};
+static int ibmveth_buffer_pools_show(struct seq_file *m, void *v)
+{
+ struct ibmveth_adapter *adapter = m->private;
+ int i, j;
+
+ seq_puts(m, "Queue Pool Size BuffSize Active Available\n");
+ seq_puts(m, "----- ---- ---- -------- ------ ---------\n");
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (j = 0; j < IBMVETH_NUM_BUFF_POOLS; j++) {
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[i][j];
+
+ seq_printf(m, "%5d %4d %4u %8u %6d %9d\n",
+ i, j, pool->size, pool->buff_size,
+ pool->active,
+ atomic_read(&pool->available));
+ }
+ }
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ibmveth_buffer_pools);
+
+static void ibmveth_debugfs_init(struct ibmveth_adapter *adapter)
+{
+ adapter->debugfs_dir = debugfs_create_dir(adapter->netdev->name,
+ NULL);
+ debugfs_create_file("buffer_pools", 0400, adapter->debugfs_dir,
+ adapter, &ibmveth_buffer_pools_fops);
+}
+
+static void ibmveth_debugfs_exit(struct ibmveth_adapter *adapter)
+{
+ debugfs_remove_recursive(adapter->debugfs_dir);
+ adapter->debugfs_dir = NULL;
+}
+
+static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
+ int pools_ready)
+{
+ int i;
+
+ for (i = 0; i < pools_ready; i++)
+ kobject_put(&adapter->rx_buff_pool[0][i].kobj);
+}
+
+static void ibmveth_probe_cleanup(struct ibmveth_adapter *adapter,
+ int pools_ready)
+{
+ struct net_device *netdev = adapter->netdev;
+
+ cancel_work_sync(&adapter->work);
+ ibmveth_put_pool_kobjs(adapter, pools_ready);
+
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+ free_netdev(netdev);
+}
+
static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
{
- int rc, i, mac_len;
+ int rc, i, mac_len, pools_ready = 0;
struct net_device *netdev;
struct ibmveth_adapter *adapter;
unsigned char *mac_addr_p;
@@ -1831,7 +3672,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
return -EINVAL;
}
- netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter), IBMVETH_MAX_QUEUES, 1);
+ netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter),
+ IBMVETH_MAX_QUEUES, IBMVETH_MAX_RX_QUEUES);
if (!netdev)
return -ENOMEM;
@@ -1844,7 +3686,15 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
ibmveth_init_link_settings(netdev);
- netif_napi_add_weight(netdev, &adapter->napi, ibmveth_poll, 16);
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+ netif_napi_add_weight(netdev, &adapter->napi[i],
+ ibmveth_poll, 16);
+
+ if (ibmveth_alloc_rx_qstats(adapter) ||
+ ibmveth_alloc_tx_qstats(adapter)) {
+ ibmveth_probe_cleanup(adapter, 0);
+ return -ENOMEM;
+ }
netdev->irq = dev->irq;
netdev->netdev_ops = &ibmveth_netdev_ops;
@@ -1876,12 +3726,27 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
netdev->features |= NETIF_F_FRAGLIST;
}
+ if (ret == H_SUCCESS &&
+ (ret_attr & IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT)) {
+ adapter->multi_queue = 1;
+ adapter->num_rx_queues = min(num_online_cpus(),
+ IBMVETH_DEFAULT_QUEUES);
+ netdev_dbg(netdev, "RX multi queue mode enabled: %d queues\n",
+ adapter->num_rx_queues);
+ } else {
+ adapter->multi_queue = 0;
+ adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
+ }
+
if (ret == H_SUCCESS &&
(ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
- adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;
+ if (adapter->multi_queue)
+ adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_QUEUE;
+ else
+ adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
netdev_dbg(netdev,
"RX Multi-buffer hcall supported by FW, batch set to %u\n",
- adapter->rx_buffers_per_hcall);
+ adapter->rx_buffers_per_hcall);
} else {
adapter->rx_buffers_per_hcall = 1;
netdev_dbg(netdev,
@@ -1898,16 +3763,24 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
memcpy(pool_count, pool_count_cmo, sizeof(pool_count));
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- struct kobject *kobj = &adapter->rx_buff_pool[i].kobj;
- int error;
+ struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- error = kobject_init_and_add(kobj, &ktype_veth_pool,
- &dev->dev.kobj, "pool%d", i);
- if (!error)
- kobject_uevent(kobj, KOBJ_ADD);
+ rc = kobject_init_and_add(kobj, &ktype_veth_pool,
+ &dev->dev.kobj, "pool%d", i);
+ if (rc) {
+ dev_err(&dev->dev,
+ "failed to create pool%d kobject: %d\n", i, rc);
+ /* init_and_add takes a ref even on failure */
+ kobject_put(kobj);
+ ibmveth_probe_cleanup(adapter, pools_ready);
+ return rc;
+ }
+
+ pools_ready++;
+ kobject_uevent(kobj, KOBJ_ADD);
}
rc = netif_set_real_num_tx_queues(netdev, min(num_online_cpus(),
@@ -1915,7 +3788,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
if (rc) {
netdev_dbg(netdev, "failed to set number of tx queues rc=%d\n",
rc);
- free_netdev(netdev);
+ ibmveth_probe_cleanup(adapter, pools_ready);
return rc;
}
adapter->tx_ltb_size = PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE);
@@ -1931,12 +3804,14 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
if (rc) {
netdev_dbg(netdev, "failed to register netdev rc=%d\n", rc);
- free_netdev(netdev);
+ ibmveth_probe_cleanup(adapter, pools_ready);
return rc;
}
netdev_dbg(netdev, "registered\n");
+ ibmveth_debugfs_init(adapter);
+
return 0;
}
@@ -1948,11 +3823,16 @@ static void ibmveth_remove(struct vio_dev *dev)
cancel_work_sync(&adapter->work);
+ ibmveth_debugfs_exit(adapter);
+
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- kobject_put(&adapter->rx_buff_pool[i].kobj);
+ kobject_put(&adapter->rx_buff_pool[0][i].kobj);
unregister_netdev(netdev);
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+
free_netdev(netdev);
dev_set_drvdata(&dev->dev, NULL);
}
@@ -2035,11 +3915,12 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
/* Make sure there is a buffer pool with buffers that
can hold a packet of the size of the MTU */
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- if (pool == &adapter->rx_buff_pool[i])
+ if (pool == &adapter->rx_buff_pool[0][i])
continue;
- if (!adapter->rx_buff_pool[i].active)
+ if (!adapter->rx_buff_pool[0][i].active)
continue;
- if (mtu <= adapter->rx_buff_pool[i].buff_size)
+ if (mtu <=
+ adapter->rx_buff_pool[0][i].buff_size)
break;
}
@@ -2092,8 +3973,8 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
}
rtnl_unlock();
- /* kick the interrupt handler to allocate/deallocate pools */
- ibmveth_interrupt(netdev->irq, netdev);
+ /* kick RX processing to allocate/deallocate pools */
+ ibmveth_schedule_rx_queue(adapter, 0);
return count;
unlock_err:
@@ -2101,7 +3982,6 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
return rc;
}
-
#define ATTR(_name, _mode) \
struct attribute veth_##_name##_attr = { \
.name = __stringify(_name), .mode = _mode, \
@@ -2133,7 +4013,9 @@ static struct kobj_type ktype_veth_pool = {
static int ibmveth_resume(struct device *dev)
{
struct net_device *netdev = dev_get_drvdata(dev);
- ibmveth_interrupt(netdev->irq, netdev);
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
+ ibmveth_schedule_rx_queue(adapter, 0);
return 0;
}
@@ -2196,8 +4078,7 @@ static void ibmveth_reset_kunit(struct work_struct *w)
* @test: pointer to kunit structure
*
* Tests the error returns from ibmveth_remove_buffer_from_pool.
- * ibmveth_remove_buffer_from_pool also calls WARN_ON, so dmesg should be
- * checked to see that these warnings happened.
+ * Bad correlators return -EINVAL/-EFAULT (no WARN_ON).
*
* Return: void
*/
@@ -2213,26 +4094,38 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- pool = &adapter->rx_buff_pool[0];
+ pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
correlator = ((u64)IBMVETH_NUM_BUFF_POOLS << 32) | 0;
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
-
- correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0].size;
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
+
+ correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0][0].size;
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
correlator = (u64)0 | 0;
pool->skbuff[0] = NULL;
- KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+ KUNIT_EXPECT_EQ(test, -EFAULT,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EFAULT,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
flush_work(&adapter->work);
}
@@ -2241,9 +4134,7 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
* ibmveth_rxq_get_buffer_test - unit test for ibmveth_rxq_get_buffer
* @test: pointer to kunit structure
*
- * Tests ibmveth_rxq_get_buffer. ibmveth_rxq_get_buffer also calls WARN_ON for
- * the NULL returns, so dmesg should be checked to see that these warnings
- * happened.
+ * Tests ibmveth_rxq_get_buffer invalid correlator returns NULL without WARN.
*
* Return: void
*/
@@ -2258,31 +4149,34 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
INIT_WORK(&adapter->work, ibmveth_reset_kunit);
- adapter->rx_queue.queue_len = 1;
- adapter->rx_queue.index = 0;
- adapter->rx_queue.queue_addr = kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
- GFP_KERNEL);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue.queue_addr);
+ adapter->rx_queue[0].queue_len = 1;
+ adapter->rx_queue[0].index = 0;
+ adapter->rx_queue[0].queue_addr =
+ kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue[0].queue_addr);
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- pool = &adapter->rx_buff_pool[0];
+ pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
- adapter->rx_queue.queue_addr[0].correlator = (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
- KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
+ adapter->rx_queue[0].queue_addr[0].correlator =
+ (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
+ KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));
- adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | adapter->rx_buff_pool[0].size;
- KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
+ adapter->rx_queue[0].queue_addr[0].correlator =
+ (u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
+ KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));
pool->skbuff[0] = skb;
- adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | 0;
- KUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter));
+ adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
+ KUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter, 0));
flush_work(&adapter->work);
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index d87713668ed30..17faaa4c2d33d 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -14,6 +14,8 @@
#ifndef _IBMVETH_H
#define _IBMVETH_H
+#include <linux/spinlock_types.h>
+
/* constants for H_MULTICAST_CTRL */
#define IbmVethMcastReceptionModifyBit 0x80000UL
#define IbmVethMcastReceptionEnableBit 0x20000UL
@@ -28,6 +30,7 @@
#define IbmVethMcastRemoveFilter 0x2UL
#define IbmVethMcastClearFilterTable 0x3UL
+#define IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT 0x0000000000080000UL
#define IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT 0x0000000000040000UL
#define IBMVETH_ILLAN_LRG_SR_ENABLED 0x0000000000010000UL
#define IBMVETH_ILLAN_LRG_SND_SUPPORT 0x0000000000008000UL
@@ -66,6 +69,143 @@ static inline long h_add_logical_lan_buffers(unsigned long unit_address,
desc5, desc6, desc7, desc8);
}
+/**
+ * h_reg_logical_lan_queue - Register a subordinate receive queue
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of 4KB page for tracking registered buffers
+ * @rec_queue: Buffer descriptor of receive queue
+ * @queue_handle: Output queue handle on success (may be NULL)
+ * @irq: Output hypervisor IRQ number on success (may be NULL)
+ *
+ * Registers a subordinate receive queue with the hypervisor.
+ *
+ * Return:
+ * H_SUCCESS (0) on success
+ * H_PARAMETER if parameters are invalid
+ *
+ * On success, hypervisor returns:
+ * R3: H_SUCCESS
+ * R4: Queue handle
+ * R5: IRQ number for this queue
+ */
+static inline long h_reg_logical_lan_queue(unsigned long unit_address,
+ unsigned long buffer_list,
+ unsigned long rec_queue,
+ unsigned long *queue_handle,
+ unsigned long *irq)
+{
+ unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+ long rc;
+
+ rc = plpar_hcall9(H_REG_LOGICAL_LAN_QUEUE,
+ retbuf, unit_address,
+ buffer_list, rec_queue);
+
+ if (rc == H_SUCCESS) {
+ if (queue_handle)
+ *queue_handle = retbuf[0];
+ if (irq)
+ *irq = retbuf[1];
+ }
+
+ return rc;
+}
+
+/**
+ * h_add_logical_lan_buffers_queue - Add buffers to subordinate queue
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_reg_logical_lan_queue()
+ * @buffersznum: Buffer size (upper 32 bits) | count (lower 32 bits)
+ * @ioba12: Buffer addresses 1 and 2 packed ((addr1 << 32) | addr2)
+ * @ioba34: Buffer addresses 3 and 4 packed
+ * @ioba56: Buffer addresses 5 and 6 packed
+ * @ioba78: Buffer addresses 7 and 8 packed
+ * @ioba910: Buffer addresses 9 and 10 packed
+ * @ioba1112: Buffer addresses 11 and 12 packed
+ *
+ * Return:
+ * H_SUCCESS - All buffers added successfully
+ * H_PARAMETER - Invalid parameters
+ * H_HARDWARE - Hardware error
+ */
+static inline long h_add_logical_lan_buffers_queue(unsigned long unit_address,
+ unsigned long queue_handle,
+ unsigned long buffersznum,
+ unsigned long ioba12,
+ unsigned long ioba34,
+ unsigned long ioba56,
+ unsigned long ioba78,
+ unsigned long ioba910,
+ unsigned long ioba1112)
+{
+ unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+
+ return plpar_hcall9(H_ADD_LOGICAL_LAN_BUFFERS_QUEUE,
+ retbuf, unit_address,
+ queue_handle, buffersznum,
+ ioba12, ioba34, ioba56,
+ ioba78, ioba910, ioba1112);
+}
+
+/**
+ * h_free_logical_lan_queue - Deregister subordinate receive queue
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_reg_logical_lan_queue()
+ *
+ * Deregisters and frees all structures associated with the subordinate queue.
+ *
+ * Return:
+ * H_SUCCESS - Queue freed successfully
+ * H_PARAMETER - Invalid parameters
+ * H_HARDWARE - Hardware error
+ * H_STATE - VIOA not in valid state
+ * H_BUSY / H_LONG_BUSY_* - Resource busy, retry
+ */
+static inline long h_free_logical_lan_queue(unsigned long unit_address,
+ unsigned long queue_handle)
+{
+ unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+
+ return plpar_hcall9(H_FREE_LOGICAL_LAN_QUEUE,
+ retbuf, unit_address, queue_handle);
+}
+
+/**
+ * h_register_logical_lan_with_handle - Register primary queue and get handle
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of buffer list
+ * @rec_queue: Buffer descriptor of receive queue
+ * @filter_list: DMA address of filter list
+ * @mac_address: MAC address
+ * @queue_handle: Output parameter for queue handle
+ *
+ * Registers the primary receive queue (queue 0) with the hypervisor and
+ * returns the queue handle. This is needed in multi-queue mode to use
+ * h_add_logical_lan_buffers_queue() for all queues including queue 0.
+ *
+ * Return: H_SUCCESS (0) on success, error code otherwise
+ */
+static inline long
+h_register_logical_lan_with_handle(unsigned long unit_address,
+ unsigned long buffer_list,
+ unsigned long rec_queue,
+ unsigned long filter_list,
+ unsigned long mac_address,
+ u64 *queue_handle)
+{
+ unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+ long rc;
+
+ rc = plpar_hcall9(H_REGISTER_LOGICAL_LAN, retbuf,
+ unit_address, buffer_list, rec_queue,
+ filter_list, mac_address);
+
+ if (rc == H_SUCCESS && queue_handle)
+ *queue_handle = retbuf[0];
+
+ return rc;
+}
+
/* FW allows us to send 6 descriptors but we only use one so mark
* the other 5 as unused (0)
*/
@@ -121,15 +261,62 @@ static inline long h_illan_attributes(unsigned long unit_address,
#define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)
#define IBMVETH_MAX_QUEUES 16U
#define IBMVETH_DEFAULT_QUEUES 8U
-#define IBMVETH_MAX_RX_PER_HCALL 8U
+#define IBMVETH_MAX_RX_QUEUES 16U
+#define IBMVETH_DEFAULT_RX_QUEUES 1U
+#define IBMVETH_MAX_RX_REGULAR 8U
+#define IBMVETH_MAX_RX_QUEUE 12U
+#define IBMVETH_MAX_RX_PER_HCALL 12U
static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };
static int pool_count[] = { 256, 512, 256, 256, 256 };
static int pool_count_cmo[] = { 256, 512, 256, 256, 64 };
-static int pool_active[] = { 1, 1, 0, 0, 1};
+static int pool_active[] = { 1, 1, 0, 0, 0};
#define IBM_VETH_INVALID_MAP ((u16)0xffff)
+struct ibmveth_hcall_stats {
+ u64 reg_lan_queue; /* H_REG_LOGICAL_LAN_QUEUE */
+ u64 reg_lan; /* H_REGISTER_LOGICAL_LAN */
+ u64 add_bufs_queue; /* H_ADD_LOGICAL_LAN_BUFFERS_QUEUE */
+ u64 add_bufs; /* H_ADD_LOGICAL_LAN_BUFFERS */
+ u64 add_buf; /* H_ADD_LOGICAL_LAN_BUFFER */
+ u64 free_lan_queue; /* H_FREE_LOGICAL_LAN_QUEUE */
+ u64 free_lan; /* H_FREE_LOGICAL_LAN */
+ u64 send_lan; /* H_SEND_LOGICAL_LAN */
+};
+
+struct ibmveth_rx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 interrupts;
+ u64 polls;
+ u64 large_packets;
+ u64 invalid_buffers;
+ u64 no_buffer_drops;
+} ____cacheline_aligned_in_smp;
+
+struct ibmveth_tx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 large_packets;
+ u64 dropped_packets;
+ u64 send_failures;
+ u64 checksum_offload;
+} ____cacheline_aligned_in_smp;
+
+/*
+ * ethtool string count: use offsetof of the last counter so alignment
+ * padding from ____cacheline_aligned_in_smp is not counted. When adding
+ * a new counter at the end, point these at the new last field (same idea
+ * as sizeof(struct)/sizeof(u64) before alignment was added).
+ */
+#define IBMVETH_NUM_RX_QSTATS \
+ (offsetof(struct ibmveth_rx_queue_stats, no_buffer_drops) / \
+ sizeof(u64) + 1)
+#define IBMVETH_NUM_TX_QSTATS \
+ (offsetof(struct ibmveth_tx_queue_stats, checksum_offload) / \
+ sizeof(u64) + 1)
+
struct ibmveth_buff_pool {
u32 size;
u32 index;
@@ -152,23 +339,29 @@ struct ibmveth_rx_q {
dma_addr_t queue_dma;
u32 queue_len;
struct ibmveth_rx_q_entry *queue_addr;
+ spinlock_t replenish_lock; /* per-queue buffer replenish */
};
struct ibmveth_adapter {
struct vio_dev *vdev;
struct net_device *netdev;
- struct napi_struct napi;
+ struct napi_struct napi[IBMVETH_MAX_RX_QUEUES];
struct work_struct work;
unsigned int mcastFilterSize;
- void *buffer_list_addr;
+ void *buffer_list_addr[IBMVETH_MAX_RX_QUEUES];
void *filter_list_addr;
void *tx_ltb_ptr[IBMVETH_MAX_QUEUES];
unsigned int tx_ltb_size;
dma_addr_t tx_ltb_dma[IBMVETH_MAX_QUEUES];
- dma_addr_t buffer_list_dma;
+ dma_addr_t buffer_list_dma[IBMVETH_MAX_RX_QUEUES];
dma_addr_t filter_list_dma;
- struct ibmveth_buff_pool rx_buff_pool[IBMVETH_NUM_BUFF_POOLS];
- struct ibmveth_rx_q rx_queue;
+ struct ibmveth_buff_pool
+ rx_buff_pool[IBMVETH_MAX_RX_QUEUES][IBMVETH_NUM_BUFF_POOLS];
+ struct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];
+ u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
+ unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
+ int multi_queue;
+ unsigned int num_rx_queues;
int rx_csum;
int large_send;
bool is_active_trunk;
@@ -188,6 +381,14 @@ struct ibmveth_adapter {
u64 tx_send_failed;
u64 tx_large_packets;
u64 rx_large_packets;
+
+ /* Multi-queue statistics */
+ struct ibmveth_hcall_stats hcall_stats;
+ struct ibmveth_rx_queue_stats *rx_qstats;
+ struct ibmveth_tx_queue_stats *tx_qstats;
+
+ struct dentry *debugfs_dir;
+
/* Ethtool settings */
u8 duplex;
u32 speed;
|
| 3/2 | 2026/08/01 10:16 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 49353 output: 35 thoughts: 926 Thoughts: **Fuzzing the IBM Power Virtual Ethernet Driver (ibmveth)** Okay, so I've got this patch – it's touching `drivers/net/ethernet/ibm/ibmveth.c` and `arch/powerpc/include/asm/hvcall.h`. It looks like they're adding multi-queue RX support to the `ibmveth` driver, which is, of course, the IBM Power Virtual Ethernet driver. New hypercalls are being introduced: `H_REG_LOGICAL_LAN_QUEUE`, `H_ADD_LOGICAL_LAN_BUFFERS_QUEUE`, `H_FREE_LOGICAL_LAN_QUEUE`. And it's modifying a bunch of crucial functions like `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, and `ibmveth_probe`, amongst others. Now, `ibmveth` – right, that's specific to the PowerPC architecture, specifically pseries. My immediate thought is, can this even *be* reached in a standard virtualized environment? Is this reachable in standard QEMU or GCE? Thankfully, QEMU supports the `pseries` machine type (using `qemu-system-ppc64 -M pseries`), which emulates the VIO bus and, critically, `ibmveth` (or `spapr-vlan` as it's also known). Okay, can Syzkaller fuzz this? Syzkaller has *some* support for `spapr-vlan`, but it's usually focused on x86_64. But hang on, the prompt specifically says to consider reachability in environments like QEMU. And, in this case, QEMU *does* emulate `spapr-vlan` if running a ppc64 VM. And, wait, I recall Syzkaller *does* have ppc64le support! Perfect. This is starting to look promising. I'm going to quickly confirm this: yes, QEMU does have `hw/net/spapr_llan.c` and thus emulates it. So, is it worth fuzzing? Yes, this is a *network driver*. That's fuzzable if it's enabled. Now, let's nail down the configuration. Looking at `drivers/net/ethernet/ibm/Kconfig`, I see: ``` config IBMVETH tristate "IBM Logical LAN (Virtual Ethernet) support" depends on PPC_PSERIES ``` Alright, so it *depends* on `PPC_PSERIES`. Good. The prompt says to check if the code paths are reachable, so since QEMU *does* emulate `spapr-vlan` (ibmveth) on `pseries` machines, it *is* reachable in QEMU. And since Syzkaller has a `qemu-ppc64le` target, we're good to go. Here's a quick run-down of the key functions being modified: `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, `ibmveth_set_channels`, `ibmveth_resize_rx_channels`, `ibmveth_probe`. So the critical config to enable for fuzzing is `IBMVETH`, and `PPC_PSERIES` (although usually specifying `IBMVETH` will be enough). It's worth a quick check to verify the `Kconfig` for `IBMVETH` is correct. WorthFuzzing=true. |
| 4/2 | 2026/08/01 10:17 | tool | grepper |
0mArgs:{
"Expression": "config IBMVETH",
"PathPrefix": "drivers/net/ethernet/ibm/Kconfig"
}Results: {
"Output": "drivers/net/ethernet/ibm/Kconfig=18=if NET_VENDOR_IBM\ndrivers/net/ethernet/ibm/Kconfig-19-\ndrivers/net/ethernet/ibm/Kconfig:20:config IBMVETH\ndrivers/net/ethernet/ibm/Kconfig-21-\ttristate \"IBM LAN Virtual Ethernet support\"\n--\ndrivers/net/ethernet/ibm/Kconfig-29-\ndrivers/net/ethernet/ibm/Kconfig:30:config IBMVETH_KUNIT_TEST\ndrivers/net/ethernet/ibm/Kconfig-31-\tbool \"KUnit test for IBM LAN Virtual Ethernet support\" if !KUNIT_ALL_TESTS\n"
} |
| 5/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 51287 output: 36 thoughts: 12 Thoughts: Here's my take on the dependencies, as if I were thinking it: **Digging into `IBMVETH`'s Dependencies** Alright, let's get down to brass tacks and dissect the dependency tree for `IBMVETH`. I need to understand its building blocks before I can fully assess its behavior or troubleshoot any issues. I'll need to identify not just the direct dependencies, but also consider the transitive ones. Are there any lurking components that might introduce unexpected complexities or vulnerabilities? Understanding this is key to performance optimization and security hardening. I'm going to start by pulling up the dependency graph and methodically reviewing each connection. I'll be looking for potential conflicts, version mismatches, and anything that screams "potential problem." My experience tells me that a well-understood dependency graph is the foundation of robust code, and this inspection is a critical first step. |
| 6/2 | 2026/08/01 10:17 | tool | grepper |
0mArgs:{
"Expression": "config IBMVETH$",
"PathPrefix": "drivers/net/ethernet/ibm/Kconfig"
}Results: {
"Output": "drivers/net/ethernet/ibm/Kconfig=18=if NET_VENDOR_IBM\ndrivers/net/ethernet/ibm/Kconfig-19-\ndrivers/net/ethernet/ibm/Kconfig:20:config IBMVETH\ndrivers/net/ethernet/ibm/Kconfig-21-\ttristate \"IBM LAN Virtual Ethernet support\"\n"
} |
| 7/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 51615 output: 38 thoughts: 15 Thoughts: Here's my thought process, summarized as if I were considering the given prompt: **Analyzing the Configuration Request** Okay, so I see the prompt. It's asking for a summary of a thought process, presented as if the thoughts were my own. The core request seems to be about looking at some configuration data, specifically related to something called `IBMVETH`. My initial instinct is to treat this as a potentially complex request. It suggests an examination of configuration lines, probably related to a network interface or virtual Ethernet device within an IBM environment (given the "IBMVETH" identifier). The prompt tells me I need to be sure to check what's going on *after* the `config IBMVETH` statement. I need to make sure I understand what details that line provides. This context gives me a starting point. I'll need to carefully interpret the lines immediately following that identifier to give the best summary. |
| 8/2 | 2026/08/01 10:17 | tool | read-file |
0mArgs:{
"File": "drivers/net/ethernet/ibm/Kconfig",
"FirstLine": 18,
"LineCount": 20
}Results: {
"Contents": " 18:\tif NET_VENDOR_IBM\n 19:\t\n 20:\tconfig IBMVETH\n 21:\t\ttristate \"IBM LAN Virtual Ethernet support\"\n 22:\t\tdepends on PPC_PSERIES\n 23:\t\thelp\n 24:\t\t This driver supports virtual ethernet adapters on newer IBM iSeries\n 25:\t\t and pSeries systems.\n 26:\t\n 27:\t\t To compile this driver as a module, choose M here. The module will\n 28:\t\t be called ibmveth.\n 29:\t\n 30:\tconfig IBMVETH_KUNIT_TEST\n 31:\t\tbool \"KUnit test for IBM LAN Virtual Ethernet support\" if !KUNIT_ALL_TESTS\n 32:\t\tdepends on KUNIT\n 33:\t\tdepends on KUNIT=y \u0026\u0026 IBMVETH=y\n 34:\t\tdefault KUNIT_ALL_TESTS\n 35:\t\thelp\n 36:\t\t This builds unit tests for the IBM LAN Virtual Ethernet driver.\n 37:\t\n"
} |
| 9/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 52129 output: 37 thoughts: 42 Thoughts: Okay, here's my interpretation of those thoughts, crafted with the understanding that I'm talking to a fellow expert: **Tracing Dependencies in IBM Networking Code** Alright, let's see, so `IBMVETH` -- that's an IBM-specific Ethernet driver, I believe -- is built on the foundation of `PPC_PSERIES`. That makes sense; it's likely a PowerPC architecture thing. Now, where does this all live in the wider codebase? Ah, right, it's gated by `NET_VENDOR_IBM`. So, if `NET_VENDOR_IBM` is enabled, *then* we get the `IBMVETH` driver, which, in turn, hinges on the PowerPC platform definition. Good, good. I need to now zero in on `NET_VENDOR_IBM` and understand how that particular configuration option is set and what triggers it. That should give me the larger picture of what's being supported, and under what circumstances. Let's dig in and figure out the parent and implications of that configuration option. |
| 10/2 | 2026/08/01 10:17 | tool | grepper |
0mArgs:{
"Expression": "config NET_VENDOR_IBM",
"PathPrefix": "drivers/net/ethernet/ibm/Kconfig"
}Results: {
"Output": "drivers/net/ethernet/ibm/Kconfig-5-\ndrivers/net/ethernet/ibm/Kconfig:6:config NET_VENDOR_IBM\ndrivers/net/ethernet/ibm/Kconfig-7-\tbool \"IBM devices\"\n"
} |
| 11/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 52503 output: 40 thoughts: 411 Thoughts: **Analyzing the ibmveth Driver Entry Points** Okay, so I need to make sure `NET_VENDOR_IBM` is enabled. That's the starting point. Now, let's focus on the critical functions. From the problem description, I'm guessing these are the primary entry points: `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, `ibmveth_set_channels`, `ibmveth_probe`, and `ibmveth_remove`. Right, the prompt is pushing me towards non-static, non-inlined API entrypoints over internal static helper functions. That makes sense, keeps things cleaner and potentially allows for future extensibility or external use. However, I'm pretty sure these listed functions *are* the entry points for this driver, used as callbacks in `net_device_ops` or the `vio_driver` structure. Let me confirm if they're actually static. Quick check in `ibmveth.c`: Yep, they are. They are declared as `static int ibmveth_open(...)`, `static int ibmveth_close(...)`, and so on. Okay, that's a problem. Being static implies they're not explicitly exported. However, since they're the driver's interface to the kernel, they *are* effectively entry points, albeit not in the standard, export-symbol-style way. Is there *any* non-static functions within that scope? Let's keep digging. |
| 12/2 | 2026/08/01 10:17 | tool | grepper |
0mError:no matches Args: {
"Expression": "^int ibmveth_",
"PathPrefix": "drivers/net/ethernet/ibm/ibmveth.c"
}Results: {
"error": "no matches"
} |
| 13/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 53299 output: 41 thoughts: 95 Thoughts: **My Initial Assessment of `ibmveth.c` Entry Points** Okay, so I'm looking at `ibmveth.c`. The first thing that jumps out is that all the functions within this module are declared as `static`. This is pretty standard for kernel drivers, so no surprises there. My next step is to get a solid overview of its interaction points. Therefore, I need to enumerate the publicly exposed functionality. From what I can tell, these are the entry points: `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, and `ibmveth_probe`. I'll need to remember these for tracing the driver's execution path. Just to be absolutely sure, I'll quickly verify that `ibmveth_set_channels` is also, in fact, declared as `static`. It's always worth confirming these assumptions to avoid future headaches. |
| 14/2 | 2026/08/01 10:17 | tool | grepper |
0mArgs:{
"Expression": "ibmveth_set_channels",
"PathPrefix": "drivers/net/ethernet/ibm/ibmveth.c"
}Results: {
"Output": "drivers/net/ethernet/ibm/ibmveth.c=2695=static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,\n--\ndrivers/net/ethernet/ibm/ibmveth.c-2732-\ndrivers/net/ethernet/ibm/ibmveth.c:2733:static int ibmveth_set_channels(struct net_device *netdev,\ndrivers/net/ethernet/ibm/ibmveth.c-2734-\t\t\t\tstruct ethtool_channels *channels)\n--\ndrivers/net/ethernet/ibm/ibmveth.c=2809=static const struct ethtool_ops netdev_ethtool_ops = {\n--\ndrivers/net/ethernet/ibm/ibmveth.c-2817-\t.get_channels\t\t\t = ibmveth_get_channels,\ndrivers/net/ethernet/ibm/ibmveth.c:2818:\t.set_channels\t\t\t = ibmveth_set_channels\ndrivers/net/ethernet/ibm/ibmveth.c-2819-};\n"
} |
| 15/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 53921 output: 171 thoughts: 489 Thoughts: **My Assessment of the ibmveth Multi-Queue Patch** Okay, this looks like it's pretty straightforward, the code is static, which simplifies things. My plan is to start by looking at `ibmveth_resize_rx_channels` because it's a static helper function and is called by `ibmveth_set_channels`. I'll be including `ibmveth_set_channels` in the analysis. Next, I need to understand how the receive queues are registered, so I'll check `ibmveth_register_rx_queues`, which is called by `ibmveth_open`. Therefore I need to look at `ibmveth_open`. I'll also check `ibmveth_resize_rx_queues_incremental`, called by `ibmveth_resize_rx_channels`, as well as `ibmveth_register_logical_lan_queue`, called by `ibmveth_register_single_rx_queue`. I'll also need to consider the core data paths, so that means I need to look at `ibmveth_poll` and `ibmveth_start_xmit`. And, of course, the entry point, `ibmveth_probe`. So, the key functions to focus on are: `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, `ibmveth_set_channels` and `ibmveth_probe`. I need to make sure I've got the necessary configurations in place. I need `NET_VENDOR_IBM`, `IBMVETH`, and `PPC_PSERIES`. Wait, `PPC_PSERIES` is an architecture config, so it's not something we can just enable on x86, but it is fine because we're targetting the `qemu-ppc64le` which uses `pseries` machine type. So that's all good. Now, let's go back and review the patch thoroughly. It's adding multi-queue support to `ibmveth`, and from my initial look it modifies `ibmveth_open`, `ibmveth_close`, `ibmveth_poll`, `ibmveth_start_xmit`, `ibmveth_probe`, `ibmveth_set_channels`, and so on. It looks like it's introducing new hypercalls, namely `H_REG_LOGICAL_LAN_QUEUE`, `H_ADD_LOGICAL_LAN_BUFFERS_QUEUE`, and `H_FREE_LOGICAL_LAN_QUEUE`. Yes, this is definitely worth fuzzing and looking at. Okay, let's call `set-results`. |
| 16/2 | 2026/08/01 10:17 | tool | set-results |
0mError:the following configs do not exist in the kernel tree: PPC_PSERIES Args: {
"EnableConfigs": [
"NET_VENDOR_IBM",
"IBMVETH",
"PPC_PSERIES"
],
"FocusSymbols": [
"ibmveth_open",
"ibmveth_close",
"ibmveth_poll",
"ibmveth_start_xmit",
"ibmveth_set_channels",
"ibmveth_probe"
],
"Reasoning": "The patch adds multi-queue RX support to the IBM Power Virtual Ethernet (ibmveth) driver, modifying core driver entry points like open, close, poll, xmit, and ethtool set_channels. This driver is reachable in QEMU when running a PowerPC pseries VM, making it fuzzable.",
"WorthFuzzing": true
}Results: {
"error": "the following configs do not exist in the kernel tree: PPC_PSERIES"
} |
| 17/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 55203 output: 38 thoughts: 22 Thoughts: Okay, here's my attempt at summarizing those thoughts from a first-person, expert perspective, focusing on the assumed knowledge the user possesses: **Checking Architecture Configuration** Right, so `PPC_PSERIES`… that's clearly an architecture configuration, likely pertaining to PowerPC processors within the pSeries (or, more broadly, Power Systems) line. Given my experience, I'm assuming this refers to a particular architecture definition within the kernel or build system. The logical next step is to quickly determine its actual status. So, let me just verify if it's defined and what the associated parameters are. I'll need to confirm its existence before I move forward, so I can ensure I'm handling the configuration correctly. |
| 18/2 | 2026/08/01 10:17 | tool | grepper |
0mArgs:{
"Expression": "config PPC_PSERIES",
"PathPrefix": "arch/powerpc/platforms/pseries/Kconfig"
}Results: {
"Output": "arch/powerpc/platforms/pseries/Kconfig-1-# SPDX-License-Identifier: GPL-2.0\narch/powerpc/platforms/pseries/Kconfig:2:config PPC_PSERIES\narch/powerpc/platforms/pseries/Kconfig-3-\tdepends on PPC64 \u0026\u0026 PPC_BOOK3S\n--\narch/powerpc/platforms/pseries/Kconfig=92=config LPARCFG\n--\narch/powerpc/platforms/pseries/Kconfig-98-\narch/powerpc/platforms/pseries/Kconfig:99:config PPC_PSERIES_DEBUG\narch/powerpc/platforms/pseries/Kconfig-100-\tdepends on PPC_PSERIES \u0026\u0026 PPC_EARLY_DEBUG\n"
} |
| 19/2 | 2026/08/01 10:17 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 55608 output: 162 thoughts: 237 Thoughts: **Debugging a Configuration Validation Tool** Okay, so the tool is reporting that `PPC_PSERIES` doesn't exist in the kernel, which is strange. I know it does exist; I can see it in `arch/powerpc/platforms/pseries/Kconfig`. I'm starting to think it's an architecture-specific config, which might be the issue. The tool is supposed to check the entire kernel tree, not just the x86-specific parts, but maybe there's a problem there. Let me double-check the exact name, just in case. Yup, `PPC_PSERIES`, exactly as I thought. Still, the tool insists it's not there. Very odd. I need to figure out why the tool is failing. Is it only looking at Kconfig files that are included in the x86 build? No, the tool is designed to search all `Kconfig` files. Right, let me just confirm *again* that `PPC_PSERIES` is defined correctly within the file specified. I'm going to quickly check `arch/powerpc/platforms/pseries/Kconfig` to be sure. Yep, it's there. I'm betting the tool has a script that parses the `Kconfig` files. I should probably dig into the tool's validation process. Let me simplify things for now. I'll omit `PPC_PSERIES` and just focus on `NET_VENDOR_IBM` and `IBMVETH` for now to see if the problem persists. |
| 20/2 | 2026/08/01 10:17 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"NET_VENDOR_IBM",
"IBMVETH"
],
"FocusSymbols": [
"ibmveth_open",
"ibmveth_close",
"ibmveth_poll",
"ibmveth_start_xmit",
"ibmveth_set_channels",
"ibmveth_probe"
],
"Reasoning": "The patch adds multi-queue RX support to the IBM Power Virtual Ethernet (ibmveth) driver, modifying core driver entry points like open, close, poll, xmit, and ethtool set_channels. This driver is reachable in QEMU when running a PowerPC pseries VM, making it fuzzable.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"NET_VENDOR_IBM",
"IBMVETH"
],
"FocusSymbols": [
"ibmveth_open",
"ibmveth_close",
"ibmveth_poll",
"ibmveth_start_xmit",
"ibmveth_set_channels",
"ibmveth_probe"
],
"Reasoning": "The patch adds multi-queue RX support to the IBM Power Virtual Ethernet (ibmveth) driver, modifying core driver entry points like open, close, poll, xmit, and ethtool set_channels. This driver is reachable in QEMU when running a PowerPC pseries VM, making it fuzzable.",
"WorthFuzzing": true
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|