When a nexthop is removed while other members keep its group alive, only the group array changes: the routes through the group stay in the FIB, and the dsts they cached through the removed nexthop are released when that nexthop is freed, from an RCU callback. Until then those dsts stay valid. IPv6 route exceptions, such as the ones created on Packet Too Big, are checked against the cookie of their FIB node, which does not change, so a socket keeps sending through the removed nexthop. When the nexthop goes away with its device, removing the device's IPv6 addresses usually flushes the routing tree first, but not when the device carries no IPv6 address. Once all readers see the new group arrays, release the per-cpu routes and exceptions of a removed IPv6 nexthop, as replacing a group already does for its old members, and flush the IPv4 route cache once if any of the groups carries IPv4 routes. The routes using the nexthop directly are already gone at that point. A connected UDP socket in netns A sends to a route through group {nh 1, nh 2}, on an IPv6 exception of nh 1 created by Packet Too Big, while nh 1 is deleted. Datagrams still sent through nh 1 after the new group is published and before nh 1 is freed, 10 trials, median and range, the ones left after the change being sent before the dsts are released: before (datagrams) after (datagrams) change per trial 1140 (367-3788) 1 (1-3) -99.9% Signed-off-by: Qiliang Yuan --- net/ipv4/nexthop.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c index 42e55b5a755ee..3fccd4f2f916b 100644 --- a/net/ipv4/nexthop.c +++ b/net/ipv4/nexthop.c @@ -2116,6 +2116,7 @@ remove_nexthop_from_groups(struct net *net, struct nexthop *nh, struct nh_grp_entry *nhge, *tmp; LIST_HEAD(deferred_free); bool need_flush = false; + struct nh_info *nhi; /* If there is nothing to do, let's avoid the costly call to * synchronize_net() @@ -2130,6 +2131,20 @@ remove_nexthop_from_groups(struct net *net, struct nexthop *nh, /* make sure all see the newly published array before releasing rtnl */ synchronize_net(); + /* Routes through the groups can still hold dsts cached through @nh, + * which are only released when @nh is freed. Release them now that + * no reader can pick @nh from the old groups any more. + */ + nhi = rtnl_dereference(nh->nh_info); + if (nhi->family == AF_INET6) + fib6_nh_release_dsts(&nhi->fib6_nh); + list_for_each_entry(nhge, &deferred_free, nh_list) { + if (!list_empty(&nhge->nh_parent->fi_list)) { + rt_cache_flush(net); + break; + } + } + /* Now safe to free percpu stats — all RCU readers have finished */ list_for_each_entry_safe(nhge, tmp, &deferred_free, nh_list) { list_del(&nhge->nh_list); -- 2.43.0