From: Yael Chemla mlx5e_params keeps two representations of CQE moderation: the private flags (MLX5E_PFLAG_RX_CQE_BASED_MODER, MLX5E_PFLAG_TX_CQE_BASED_MODER) and the fields rx_moder_use_cqe_mode / tx_moder_use_cqe_mode, which are used by ethtool get/set coalesce. Both are used by the driver (ethtool get/set coalesce, channel open, moderation apply). When they disagree, CQE moderation reported value is wrong. Such an issue can be seen on channel attach or reopen: mlx5e_build_nic_params() runs and sets rx_moder_use_cqe_mode and tx_moder_use_cqe_mode from device capability but did not set the corresponding pflags. So after attach/reopen, pflags could remain zero while the fields reflected hardware configuration. Fix by syncing pflags in build_nic_params from the capability-derived fields. Reproducer: ethtool --show-priv-flags eth2 | grep rx_cqe_moder rx_cqe_moder : off ethtool --coalesce eth2 adaptive-tx off ethtool --show-priv-flags eth2 | grep rx_cqe_moder rx_cqe_moder : on Fixes: 445a25f6e1a2 ("net/mlx5e: Support updating coalescing configuration without resetting channels") Signed-off-by: Yael Chemla Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 4c6060d54fbe..9fcea0395137 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -5407,6 +5407,10 @@ void mlx5e_build_nic_params(struct mlx5e_priv *priv, struct mlx5e_xsk *xsk, u16 MLX5_CAP_GEN(mdev, cq_period_mode_modify); params->rx_moder_use_cqe_mode = !!MLX5_CAP_GEN(mdev, cq_period_start_from_cqe); params->tx_moder_use_cqe_mode = false; + MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_CQE_BASED_MODER, + params->rx_moder_use_cqe_mode); + MLX5E_SET_PFLAG(params, MLX5E_PFLAG_TX_CQE_BASED_MODER, + params->tx_moder_use_cqe_mode); mlx5e_reset_rx_moderation(¶ms->rx_cq_moderation, params->rx_moder_use_cqe_mode, params->rx_dim_enabled); mlx5e_reset_tx_moderation(¶ms->tx_cq_moderation, params->tx_moder_use_cqe_mode, -- 2.44.0 From: Yael Chemla When CQE-based moderation is toggled via the priv-flag path (ethtool --set-priv-flags ... rx_cqe_moder / tx_cqe_moder), set_pflag_cqe_based_moder() updates only params->pflags and does not update rx_moder_use_cqe_mode / tx_moder_use_cqe_mode. After the toggle, --show-priv-flags reflects the new state but get_coalesce (and thus ethtool -c) still reports the old state because it reads from the params fields. Fix by setting new_params.rx_moder_use_cqe_mode or new_params.tx_moder_use_cqe_mode in set_pflag_cqe_based_moder() when toggling the corresponding priv flag, so both representations stay in sync. Additionally, new_params.rx_cq_moderation / new_params.tx_cq_moderation are left as a copy of the old params, so ethtool -c reports stale rx_coalesce_usecs after the toggle. Fix by calling mlx5e_reset_rx/tx_moderation() on new_params to overwrite usec with the default for the new period mode before storing it into priv->channels.params. Reproducer: ethtool -C eth2 adaptive-rx off ethtool --show-priv-flags eth2 | grep -E 'rx_cqe_moder|tx_cqe_moder' rx_cqe_moder : on tx_cqe_moder : off ethtool -c eth2 | grep -E 'rx-usecs:|CQE mode' rx-usecs: 3 CQE mode RX: on TX: off ethtool --set-priv-flags eth2 rx_cqe_moder off ethtool --set-priv-flags eth2 tx_cqe_moder on ethtool --show-priv-flags eth2 | grep -E 'rx_cqe_moder|tx_cqe_moder' rx_cqe_moder : off tx_cqe_moder : on ethtool -c eth2 | grep -E 'rx-usecs:|CQE mode' rx-usecs: 3 # stale: EQE default is 16 CQE mode RX: on TX: off # stale: priv-flags changed, CQE mode did not Fixes: 445a25f6e1a2 ("net/mlx5e: Support updating coalescing configuration without resetting channels") Signed-off-by: Yael Chemla Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c index 3ed59ced0407..c2bee69c26af 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c @@ -2193,12 +2193,20 @@ static int set_pflag_cqe_based_moder(struct net_device *netdev, bool enable, mlx5e_channels_rx_toggle_dim(&priv->channels); MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_RX_CQE_BASED_MODER, cq_period_mode); + new_params.rx_moder_use_cqe_mode = enable; + mlx5e_reset_rx_moderation(&new_params.rx_cq_moderation, + cq_period_mode, + new_params.rx_dim_enabled); } else { mlx5e_reset_tx_channels_moderation(&priv->channels, cq_period_mode, false, true); mlx5e_channels_tx_toggle_dim(&priv->channels); MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_TX_CQE_BASED_MODER, cq_period_mode); + new_params.tx_moder_use_cqe_mode = enable; + mlx5e_reset_tx_moderation(&new_params.tx_cq_moderation, + cq_period_mode, + new_params.tx_dim_enabled); } /* Update pflags of existing channels without resetting them */ -- 2.44.0 From: Yael Chemla commit bc541621f8ba ("net/mlx5e: Support ethtool cq mode") added ETHTOOL_COALESCE_USE_CQE to the NIC ethtool ops but omitted it from mlx5e_rep_ethtool_ops.supported_coalesce_params. The ethtool netlink layer rejects any attempt to set cqe-mode-rx/tx on a representor with EINVAL. Add ETHTOOL_COALESCE_USE_CQE to mlx5e_rep_ethtool_ops. mlx5e_ethtool_set_coalesce() already handles it correctly. Fixes: bc541621f8ba ("net/mlx5e: Support ethtool cq mode") Signed-off-by: Yael Chemla Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index 88a170e40bd9..4fb132a57bca 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -416,7 +416,8 @@ static u32 mlx5e_rep_get_rxfh_indir_size(struct net_device *netdev) static const struct ethtool_ops mlx5e_rep_ethtool_ops = { .supported_coalesce_params = ETHTOOL_COALESCE_USECS | ETHTOOL_COALESCE_MAX_FRAMES | - ETHTOOL_COALESCE_USE_ADAPTIVE, + ETHTOOL_COALESCE_USE_ADAPTIVE | + ETHTOOL_COALESCE_USE_CQE, .op_needs_rtnl = ETHTOOL_OP_NEEDS_RTNL_SCHANNELS | ETHTOOL_OP_NEEDS_RTNL_SRINGPARAM | ETHTOOL_OP_NEEDS_RTNL_GLINK, -- 2.44.0