airoha_qdma_hw_init() unmasks the per-ring NO_CPU_DSCP interrupt, which fires when an RX ring runs out of free CPU descriptors, but airoha_irq_handler() only extracts the RX_DONE bits from the same status word. The NO_CPU_DSCP bits are acknowledged and dropped. Once a ring is drained to zero posted descriptors no further RX_DONE can fire for it - nothing is left for hw to receive into - so NAPI is never rescheduled, airoha_qdma_fill_rx_queue() is never called again, and the ring stays dead until the interface is reconfigured. This is reachable on RX ring 4, which airoha_fe_vip_setup() force-routes ~15 VIP-classified protocols onto (BOOTP, PPPoE Discovery, ISAKMP, DHCPv6, SIP, LLDP, PPP LCP/IPCP/CHAP/PAP/IPv6CP, ...) while it sits on the 16-descriptor RX_DSCP_NUM() default. A DHCP renewal or a PPPoE negotiation burst drains it faster than the CPU reposts, after which all of those protocols silently stop being received. Treat NO_CPU_DSCP like RX_DONE for scheduling NAPI: airoha_qdma_rx_process() already calls airoha_qdma_fill_rx_queue() unconditionally at the end of every poll, so scheduling NAPI is enough to make an emptied ring refill itself. Re-enable the bit alongside RX_DONE in airoha_qdma_rx_napi_poll() so it is not left masked after first use. Whether NO_CPU_DSCP re-latches while the ring is still empty is not documented. If it does, a ring that cannot be refilled (page_pool returning NULL) keeps reasserting it. Masking it until a refill succeeds is worse: a starved ring never fires RX_DONE either, so it would stay dead after the memory pressure clears. Fixes: f252493e1835 ("net: airoha: Enable multiple IRQ lines support in airoha_eth driver.") Link: https://github.com/openwrt/openwrt/issues/24715 Acked-by: Lorenzo Bianconi Signed-off-by: Vitaliy Sochnev --- drivers/net/ethernet/airoha/airoha_eth.c | 16 +++++++++++----- drivers/net/ethernet/airoha/airoha_regs.h | 2 ++ 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c index 64619e9a704d..c59201aded26 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.c +++ b/drivers/net/ethernet/airoha/airoha_eth.c @@ -784,13 +784,16 @@ static int airoha_qdma_rx_napi_poll(struct napi_struct *napi, int budget) int i, qid = q - &qdma->q_rx[0]; int intr_reg = qid < RX_DONE_HIGH_OFFSET ? QDMA_INT_REG_IDX1 : QDMA_INT_REG_IDX2; + u32 irq_id = qid % RX_DONE_HIGH_OFFSET; + u32 intr_mask = BIT(irq_id) | + BIT(irq_id + RX_NO_CPU_DSCP_LOW_OFFSET); for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) { if (!(BIT(qid) & RX_IRQ_BANK_PIN_MASK(i))) continue; airoha_qdma_irq_enable(&qdma->irq_banks[i], intr_reg, - BIT(qid % RX_DONE_HIGH_OFFSET)); + intr_mask); } } @@ -1468,16 +1471,19 @@ static irqreturn_t airoha_irq_handler(int irq, void *dev_instance) if (!test_bit(DEV_STATE_INITIALIZED, &qdma->eth->state)) return IRQ_NONE; - rx_intr1 = intr[1] & RX_DONE_LOW_INT_MASK; + rx_intr1 = intr[1] & (RX_DONE_LOW_INT_MASK | RX_NO_CPU_DSCP_LOW_INT_MASK); if (rx_intr1) { airoha_qdma_irq_disable(irq_bank, QDMA_INT_REG_IDX1, rx_intr1); - rx_intr_mask |= rx_intr1; + rx_intr_mask |= (rx_intr1 & RX_DONE_LOW_INT_MASK) | + RX_NO_CPU_DSCP_INT_RX1_MASK(rx_intr1); } - rx_intr2 = intr[2] & RX_DONE_HIGH_INT_MASK; + rx_intr2 = intr[2] & (RX_DONE_HIGH_INT_MASK | RX_NO_CPU_DSCP_HIGH_INT_MASK); if (rx_intr2) { airoha_qdma_irq_disable(irq_bank, QDMA_INT_REG_IDX2, rx_intr2); - rx_intr_mask |= (rx_intr2 << 16); + rx_intr_mask |= ((rx_intr2 & RX_DONE_HIGH_INT_MASK) << + RX_DONE_HIGH_OFFSET) | + (rx_intr2 & RX_NO_CPU_DSCP_HIGH_INT_MASK); } for (i = 0; rx_intr_mask && i < ARRAY_SIZE(qdma->q_rx); i++) { diff --git a/drivers/net/ethernet/airoha/airoha_regs.h b/drivers/net/ethernet/airoha/airoha_regs.h index 442b48c9b991..bde3d599d8b7 100644 --- a/drivers/net/ethernet/airoha/airoha_regs.h +++ b/drivers/net/ethernet/airoha/airoha_regs.h @@ -549,6 +549,8 @@ #define INT_RX1_MASK(_n) \ ((((_n) << RX_NO_CPU_DSCP_LOW_OFFSET) & RX_NO_CPU_DSCP_LOW_INT_MASK) | \ (RX_DONE_LOW_INT_MASK & (_n))) +#define RX_NO_CPU_DSCP_INT_RX1_MASK(_n) \ + (((_n) & RX_NO_CPU_DSCP_LOW_INT_MASK) >> RX_NO_CPU_DSCP_LOW_OFFSET) /* QDMA_CSR_INT_ENABLE3 */ #define RX31_NO_CPU_DSCP_INT_MASK BIT(31) -- 2.55.0 On AN7583 hw can stop advancing the descriptor the sequential consumer in airoha_qdma_rx_process() is waiting on, while its own completion counter keeps moving. The ring is then dead: NAPI is scheduled, finds DONE clear at q->tail, and does nothing, forever. Observed directly on ring 4 at its 16-descriptor default (devmem, qdma0): REG_RX_CPU_IDX frozen at 15 for over an hour while REG_RX_DMA_IDX advanced 29 -> 96, with a 60-byte frame left stranded in the ring. QDMA_DESC_DROP_MASK was never set. In practice this is hit during PPPoE negotiation bursts on the shared "force to CPU" ring, where it stops the dial-up from ever completing. Detect it without trusting ring content: REG_RX_DMA_IDX is hw's own counter, independent of what sw posted. If it advances across polls while q->tail does not, hw is making progress the consumer cannot observe. Idle rings, where hw does not advance either, are left alone. An earlier version scanned the ring for a DONE descriptor and trusted its content; that OOMed once it reached uninitialised DMA memory that happened to have the bit set. A register cannot misfire that way. Recovery is deferred to a work item, since the register access can sleep. It drops what is in flight and re-arms the ring through the existing cleanup_rx_queue()/fill_rx_queue() pair, which only touch the sw-owned [tail, head) window and rewrite both indices from it. Trying instead to identify and keep the descriptor hw used caused a page_pool double free. GLOBAL_CFG_RX_DMA_EN_MASK is per-QDMA, not per-ring, so this briefly pauses every ring behind that instance; there is no per-ring equivalent. The measured pause is 986-1131 us over 13 recoveries, not the 50 ms read_poll_timeout() ceiling, so the logged value is worth having. Fixes: 23020f049327 ("net: airoha: Introduce ethernet support for EN7581 SoC") Signed-off-by: Vitaliy Sochnev --- drivers/net/ethernet/airoha/airoha_eth.c | 98 +++++++++++++++++++++++- drivers/net/ethernet/airoha/airoha_eth.h | 8 ++ 2 files changed, 105 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c index c59201aded26..177a0e10e372 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.c +++ b/drivers/net/ethernet/airoha/airoha_eth.c @@ -3,12 +3,14 @@ * Copyright (c) 2024 AIROHA Inc * Author: Lorenzo Bianconi */ +#include #include #include #include #include #include #include +#include #include #include #include @@ -657,6 +659,34 @@ airoha_qdma_get_gdm_dev(struct airoha_eth *eth, struct airoha_qdma_desc *desc) return port->devs[d] ? port->devs[d] : ERR_PTR(-ENODEV); } +#define AIROHA_RX_STALL_THRESHOLD 3 + +/* REG_RX_DMA_IDX is hw's own completion counter, independent of what sw has + * posted, so comparing it against q->tail spots the stall without trusting + * ring content: if hw keeps advancing while the strictly sequential consumer + * does not, it is completing descriptors that consumer can never reach. + */ +static void airoha_qdma_rx_check_stall(struct airoha_queue *q) +{ + struct airoha_qdma *qdma = q->qdma; + int qid = q - &qdma->q_rx[0]; + u32 dma_idx; + + dma_idx = airoha_qdma_get(qdma, REG_RX_DMA_IDX(qid), + RX_RING_DMA_IDX_MASK); + + if (q->stall_tail == q->tail && dma_idx != q->stall_dma_idx) { + if (++q->stall_count >= AIROHA_RX_STALL_THRESHOLD && + !test_and_set_bit(qid, qdma->rx_recover_mask)) + schedule_work(&qdma->rx_recover_work); + } else { + q->stall_count = 0; + } + + q->stall_tail = q->tail; + q->stall_dma_idx = dma_idx; +} + static int airoha_qdma_rx_process(struct airoha_queue *q, int budget) { enum dma_data_direction dir = page_pool_get_dma_dir(q->page_pool); @@ -675,8 +705,10 @@ static int airoha_qdma_rx_process(struct airoha_queue *q, int budget) struct page *page; desc_ctrl = le32_to_cpu(READ_ONCE(desc->ctrl)); - if (!(desc_ctrl & QDMA_DESC_DONE_MASK)) + if (!(desc_ctrl & QDMA_DESC_DONE_MASK)) { + airoha_qdma_rx_check_stall(q); break; + } dma_rmb(); @@ -895,6 +927,56 @@ static void airoha_qdma_cleanup_rx_queue(struct airoha_queue *q) FIELD_PREP(RX_RING_DMA_IDX_MASK, q->tail)); } +static void airoha_qdma_rx_recover_work(struct work_struct *work) +{ + struct airoha_qdma *qdma = container_of(work, struct airoha_qdma, + rx_recover_work); + int qid; + + for_each_set_bit(qid, qdma->rx_recover_mask, AIROHA_NUM_RX_RING) { + struct airoha_queue *q = &qdma->q_rx[qid]; + ktime_t rx_dma_off_ts; + s64 rx_dma_off_us; + u32 status; + + napi_disable(&q->napi); + + /* per-QDMA, not per-ring: this pauses every RX ring behind + * this instance, hence the measured duration below + */ + rx_dma_off_ts = ktime_get(); + airoha_qdma_clear(qdma, REG_QDMA_GLOBAL_CFG, + GLOBAL_CFG_RX_DMA_EN_MASK); + if (read_poll_timeout(airoha_qdma_rr, status, + !(status & GLOBAL_CFG_RX_DMA_BUSY_MASK), + USEC_PER_MSEC, 50 * USEC_PER_MSEC, true, + qdma, REG_QDMA_GLOBAL_CFG)) + dev_warn(qdma->eth->dev, + "qid=%d RX DMA busy timeout during recovery\n", + qid); + + airoha_qdma_cleanup_rx_queue(q); + if (q->skb) { + dev_kfree_skb(q->skb); + q->skb = NULL; + } + airoha_qdma_fill_rx_queue(q); + + airoha_qdma_set(qdma, REG_QDMA_GLOBAL_CFG, + GLOBAL_CFG_RX_DMA_EN_MASK); + rx_dma_off_us = ktime_us_delta(ktime_get(), rx_dma_off_ts); + + q->stall_count = 0; + napi_enable(&q->napi); + napi_schedule(&q->napi); + + dev_warn_ratelimited(qdma->eth->dev, + "qid=%d RX ring recovered after hw stall (RX DMA paused %lld us)\n", + qid, rx_dma_off_us); + clear_bit(qid, qdma->rx_recover_mask); + } +} + static int airoha_qdma_init_rx(struct airoha_qdma *qdma) { int i; @@ -1582,6 +1664,8 @@ static void airoha_qdma_cleanup(struct airoha_eth *eth, { int i; + cancel_work_sync(&qdma->rx_recover_work); + if (test_bit(DEV_STATE_INITIALIZED, ð->state)) { u32 status; @@ -1639,6 +1723,13 @@ static int airoha_hw_init(struct platform_device *pdev, if (err) return err; + /* init every instance up front: the error path below tears down all + * of eth->qdma[], including entries the init loop never reached + */ + for (i = 0; i < ARRAY_SIZE(eth->qdma); i++) + INIT_WORK(ð->qdma[i].rx_recover_work, + airoha_qdma_rx_recover_work); + for (i = 0; i < ARRAY_SIZE(eth->qdma); i++) { err = airoha_qdma_init(pdev, eth, ð->qdma[i]); if (err) @@ -1687,6 +1778,11 @@ static void airoha_qdma_stop_napi(struct airoha_qdma *qdma) { int i; + /* must not run or re-arm past this point: the work calls + * napi_disable() too, and doing that twice spins forever + */ + disable_work_sync(&qdma->rx_recover_work); + for (i = 0; i < ARRAY_SIZE(qdma->q_tx_irq); i++) napi_disable(&qdma->q_tx_irq[i].napi); diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/ethernet/airoha/airoha_eth.h index fa9a8edce22f..c195dad5ed58 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.h +++ b/drivers/net/ethernet/airoha/airoha_eth.h @@ -207,6 +207,11 @@ struct airoha_queue { bool txq_stopped; bool flushing; + /* see airoha_qdma_rx_check_stall() */ + u32 stall_dma_idx; + u16 stall_tail; + u8 stall_count; + struct napi_struct napi; struct page_pool *page_pool; struct sk_buff *skb; @@ -567,6 +572,9 @@ struct airoha_qdma { struct airoha_queue q_tx[AIROHA_NUM_TX_RING]; struct airoha_queue q_rx[AIROHA_NUM_RX_RING]; + struct work_struct rx_recover_work; + DECLARE_BITMAP(rx_recover_mask, AIROHA_NUM_RX_RING); + DECLARE_BITMAP(qos_channel_map, AIROHA_NUM_QOS_CHANNELS); }; -- 2.55.0 Raise the RX_DSCP_NUM() fallback from 16 to 32, the vendor SDK default, and give ring 4 the 128 descriptors already used for rings 2/11/15. Ring 4 is the shared "force to CPU" ring that airoha_fe_vip_setup() routes BOOTP, PPPoE Discovery, PPP LCP/IPCP/CHAP/PAP/IPv6CP, ISAKMP, DHCPv6, SIP and LLDP onto, so it sees the same bursty non-hashed traffic as the rings already on 128 while sitting on the smallest allowance. At 16 descriptors the completion stall handled by the previous patch is readily reachable: on a Nokia XG-040G-MF the recovery fires roughly every 35 s under repeated PPPoE dial-up, and negotiation never completes. At 128 it did not trigger once across 500 forced PPPoE reconnects over 20 h. Ring 4 and the VIP classification feeding it are shared driver-wide, with no DT or hardware property distinguishing one variant's ring 4 from another's, so this is not scoped to a particular SoC. Fixes: 23020f049327 ("net: airoha: Introduce ethernet support for EN7581 SoC") Signed-off-by: Vitaliy Sochnev --- drivers/net/ethernet/airoha/airoha_eth.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/ethernet/airoha/airoha_eth.h index c195dad5ed58..0b7c90940272 100644 --- a/drivers/net/ethernet/airoha/airoha_eth.h +++ b/drivers/net/ethernet/airoha/airoha_eth.h @@ -41,9 +41,10 @@ #define TX_DSCP_NUM 1024 #define RX_DSCP_NUM(_n) \ ((_n) == 2 ? 128 : \ + (_n) == 4 ? 128 : \ (_n) == 11 ? 128 : \ (_n) == 15 ? 128 : \ - (_n) == 0 ? 1024 : 16) + (_n) == 0 ? 1024 : 32) #define PSE_RSV_PAGES 128 #define PSE_QUEUE_RSV_PAGES 64 -- 2.55.0