From patchwork Sun Oct 6 06:47:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gilad Naaman X-Patchwork-Id: 13823685 X-Patchwork-Delegate: kuba@kernel.org Received: from dispatch1-eu1.ppe-hosted.com (dispatch1-eu1.ppe-hosted.com [185.132.181.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98D3A166F3A for ; Sun, 6 Oct 2024 06:48:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=185.132.181.6 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728197307; cv=fail; b=i/TZi9n0vjMKrLrAKimvYF+yCydOoxu1tOd+/KXupDQvHEKob6zMu1bhA8TG+kX2SX3amNG+ko+eY8nVvcvnLCjC6+GDSSL0gQ6zenPYjLAVWy+e9QhnaLHYKwKAmLiTcwhBKL1v4JHFoNiVYIn9axfj1Jv5A1wgAiucEkfnTcY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728197307; c=relaxed/simple; bh=6zrPdnJSaOB18zh1gWu3etTXSvAJWWbhmr7A8vagfDQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=jm1xyOD/IatpsFFjwE4a5qNEnH5BoBcgsUUituYg5Cq6MvwwhHn8rFMYpvcFIJXNsjI7M6IIsCwcwIEgfNxOdjlPy1hmDsTDP+7qN6uYAft0gB84rmlzqUnDlN0xh8Qen5bKixoI5/AURTtCF47PHxMjgD23vWSy9VxHeePhvmY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=drivenets.com; spf=pass smtp.mailfrom=drivenets.com; dkim=pass (1024-bit key) header.d=drivenets.onmicrosoft.com header.i=@drivenets.onmicrosoft.com header.b=m94177Zz; arc=fail smtp.client-ip=185.132.181.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=drivenets.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=drivenets.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=drivenets.onmicrosoft.com header.i=@drivenets.onmicrosoft.com header.b="m94177Zz" X-Virus-Scanned: Proofpoint Essentials engine Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05lp2172.outbound.protection.outlook.com [104.47.17.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1-eu1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id C350F740051; Sun, 6 Oct 2024 06:48:16 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Kgf3zPcnH1fh1sD2MjNz3mRa523ZZhxENJhmpNiHYZ9MAhh8aoRFbtKp7n/bRj/9IgCaZZRl86CePt7DfzDKebZ+c8MuTn2WWWTGTXqps8JVIaU8iH8oX+pEyik8xyCLyx8L/wGu3pI+hGWTHn9SUfst779xxTBB6ZyOlMrPnvoMk/yOuyrp0c41vMOa5uEw6WSLEDCu5a3xoBtUvTtmLEsTz/UFHVN6EttsGdpNS2JJ/ePyF3S+Gpz9xIee3dps1fp0LeYcD2MIDQ/TJGDDDHYgSWRTc7h8o1wp/7nFH1GQAvdeg1Kq0n9146E/zuoNm9ufi+Ay/VdxyQXaAbgxiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S9sm9WQ47iK0f5AGUcxm+4XDJfTzdsSG7fioEiSKrBU=; b=LLDdIXHPX0BOMCJ82CqEonIWi4kkYYCH9IQlNE2cWZBwHF8xagpGIcCPanQvF+GHOCt566Jx92HMstt8q4oTxVGar4abpcfJ4IN45opg8yUltcX/wsrVyTOy3XKDiD6w33Sf/egMriLu/GRPkLuQSMj5oLeAx3Fg1RbErpDIyY/1pVQ9+tdwq2Fj2eVfs0/qZV/4aLUGYf06H63od3CQFHlNL1dcp2ogAE4JFO6f1nw2wktQ+Ip8WwMMxHBaj+OFfvYduuTg/sZU+xmiIWAlSgWWdZXaWnTsgy/j79piB3VAAZ7S0t0AhNvDmoz1n+jynw/oKi84kcyP/ginZCl2yA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=drivenets.com; dmarc=pass action=none header.from=drivenets.com; dkim=pass header.d=drivenets.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=drivenets.onmicrosoft.com; s=selector2-drivenets-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S9sm9WQ47iK0f5AGUcxm+4XDJfTzdsSG7fioEiSKrBU=; b=m94177ZzLQM72ymxVHDC7HTOhuE262JJn9Ph8jBgDI4oofbXo158KQ2QjTkwbwPA9XK2q9Qbeh6n07uGReNdqC11oYZom7rDDyjv+8v7PIwPzdSnAOrTZKSko/sFi++IMXMx7QfZnoY6QGebHBaulscm5OmjRHQHwrnE3C0xoic= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=drivenets.com; Received: from DB8PR08MB5388.eurprd08.prod.outlook.com (2603:10a6:10:11c::7) by AM8PR08MB6324.eurprd08.prod.outlook.com (2603:10a6:20b:315::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.20; Sun, 6 Oct 2024 06:48:15 +0000 Received: from DB8PR08MB5388.eurprd08.prod.outlook.com ([fe80::29dd:6773:4977:dc4e]) by DB8PR08MB5388.eurprd08.prod.outlook.com ([fe80::29dd:6773:4977:dc4e%5]) with mapi id 15.20.8026.020; Sun, 6 Oct 2024 06:48:15 +0000 From: Gilad Naaman To: netdev , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Gilad Naaman Subject: [PATCH net-next v2 1/2] Convert neighbour-table to use hlist Date: Sun, 6 Oct 2024 06:47:42 +0000 Message-ID: <20241006064747.201773-2-gnaaman@drivenets.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241006064747.201773-1-gnaaman@drivenets.com> References: <20241006064747.201773-1-gnaaman@drivenets.com> X-ClientProxiedBy: LO2P123CA0071.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1::35) To DB8PR08MB5388.eurprd08.prod.outlook.com (2603:10a6:10:11c::7) Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB8PR08MB5388:EE_|AM8PR08MB6324:EE_ X-MS-Office365-Filtering-Correlation-Id: dfdb6785-4a8d-4e3b-a81d-08dce5d2dab2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|52116014|366016|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: ARKR8seuFpInFTbOjAGUMDl1TjLYacpk68SAdMBmHMlI9TrT8J92FD4aUjTadqhiQWQYX590luyg7NfhYtmzSz6Su+iIpIc+anR5P9Eu+6PrTnAkEsC36G5sqO2APVYP3olFSTVnVuRS5aoJWWdajabKAsFM9o4sATHLmGZ3z31mfQy614VXak0EIaEexjeXqxtkT5aIokh8IR/AvZRaNa0yAAoFV+XsCi7E1CUbhwOaCUwW+ap4ENDxrIgW0XJabuvTMaz9OVKhprooaajdDYF0cvoA6P9ZpLsstNZu+FrEnY75vq1yK1DowxDy95ETCDTKb7YJ3aBUZS3N+u2mTg/neP8g09KULzRRPVntjb12hMGiAvH4HzA5QomS0RP/8v8g5fqRG6fEjZzx8UqGpDWlTOoqTvlrRogCpYZlJxumOquIzQPL6s61yNFb+NefvloFh4FBi8tFA10RplAkBp+zuWA0hDcAL2otE+IG3pClyAA0M5bQ/1yFxssWF7FGEmP4g5euNhsUCMg8huo4FstSkTyPqr2D2G/icuayyf7vyDb4k7ezPZDAs5JfTPDkRj8Oq9toRxivTUPJdjIJ+I/3N8NEuPcZJnFfcMrolekCaEH2HVe0XZVApR6QrigFEd2dXTJun916oVZWIbSbV9nhiKR07AKBPDQtdhdJXDfaTGdg9J+i5BQkoQ/NjmhWd74MSilO2h+STp0J8SDYm8AN3OO/kgRs3nKzfL2L3FlIG4h9XtQLcIu0OeKlwGjie6m6T8wv2xbu75m20VOZGgnLZ9ITxTksJGq2j3pbuE+myoP+ryAmEUdyVkgma3jrIm65bcHp+wEgRCeYwXfdBCHdlJoO65dt8NHsFi8lZ+fHYRGxDmVfJB1nrkEob6t9Zb/A0jdw8epsTnElTODVMqKKQ6B7L7F6BxBGNMYPjAik6qVFMTViewlxzvyB83A+rBQ7SsWu0BdbEKWTijZdKAwqZJw+WYAMbsVNQ2E+CTDDGKyeDzi7gpzjLKVTkUTzLAkCr56RWlEerOGswazHIXF9YIcLr4l6Mh8J2jXb4fP80SEEFd9C3xHQ0IXmYj+Tk/eMeQwMkMNEg+YBjrMwa4lEx+fPSVDFHNH5Vf+IiO4aUupfIDJOb2T0mMPV93z4qahvuSCLWzullGSun6U6k+qRaqaQbO7Ia3XcNhCM2ZGtpCNUJuAGS7MfXHtgs1UZWJ4iJQXzytV15zpx8lT1VHefKavVYmp99sPpWV+dvvptrLqC+sZQQ9yk2OlSxJJ8Mn0mjH/wAmNraLciiCYphC99/8I+fuB/dFBVblst6xY+x7DnHolZWApt8uB49YR3CxWQUhuRJxdtl0pyX1RxFw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DB8PR08MB5388.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(52116014)(366016)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: M12+km5ezOG+0AGx/27u6VWL5+qmlgLbPmIEI+RNm3Wa77Igkbc4bi5/B7LEagri4Mkg7ueAU+Eb2qbY+TembraOyTUKoV3yKrUxSPoStWEBG7VelObaqoDow9r4bVeJDIdorjpXz/rQDE5/jwg94P7rOtGmBocX1Kqy1tFWhOzvSCHQNzM9c3NNtFiLDTiwocGvHV7lO7/1pECFwOOK+2CnefPDVrGJKV9hezAE9u7VVLRWHG5TLYO8OyMZTPGKbX+VS5/lzgugCY4NPQkEsol+NJRoEKy0JJ8l16AHLmzEjAl7489KiW6nsEYAZGO/a0XMA6JyfnPRUO0qvyXcRT+NrJfy+/PbfvWuwEC7XS9iLhRmL3J818PGlhnwVpnL812j6cDzmj0gBVh+GyHqVYKvnCV+FHLw/aDZmt9WjGlqsZozm/+MpuTrt6MRfKr47mzd6b3jpuMlbSNGy6yrmqniIjJaCSQ/Cpyu3VePurG3ddcBO2L/a8lDvirRwxSeEdibweYe7crQctkQDYgg4dIEo8B+vcIfDw/wJM6khyCYsYEq4EbN4bHDGEGrM0OjgeIaq8+NYfX3K/+eXx7FMVoJ96+R1a7bqcSZnAckqGj+golSl7myNMt1rBXEHgvoXucujEd9dqx0IQVWV0DtnYlWfaDBZ6rBLCRdKa1LUOWvpSZwpEAdeSg43DevRw4ryrvdOG/qtccLUyDGG0HMpIBuWTPgDdmSWrPJxkXpzVx99tumDMoymJXrHgxMJ7C5I+ay+qPr1pHBNgrGj/Bja0BzGrWAPcy7hEK1ojypWrDX69z/QTXCWuhVlWFw5tBwZTXfNE1joadSU7CBN6aLFqChNwAJ5ii/GhgFpf0bkb3DyKFmFCLpjA0bbImKc80q5G7xgUs7RcUSdF/8H7Cwedf91Vlp5ixEpODf3LhfOf8JhvlmQ1ApIMYeKiYLYKU2LFEwXc5MqQN1P2fpdoUrzkPZAQcFWdJ9qm3BSW6/TuYXejOFp/Qf/jx75fKB/YCPUjIZr0Vk3388wxw0SAIZt42PgNKU384H1BTFVLowyaAFZz8VTyvVimoicUOcL29Yz6Ni5BNLE6UvsR0dKcYFCmwkEpOvAIqb42M3y3b5bVOCwJn7+LMSYel4cz9fb5UZLzQwPXFloMlHr34YVc6N/XgFwPNaafc+5ZpToh0LKZBffgqeOzRpVFo/J8yyuMCA11kj48PRof2D+Bchd8DvbJJL1//7RrDWFJ0h3VJCkBHR4fGUm9t847s1Ko+H7S2pqHBWWasIJPoWfuFOo1GsDrVVdsffQcZHpNsBNT1PIY8EonaS/s/4Zd/qqgzC5dxwA1bPOh/GN2D5CjnHSToZjV0/prgqMWu9zIEG9G0dGfYLR0WiuKZECkk7kADXTG9xUqCLnh46YSvFLSLDqom3CJxZJu+EXIMvUIrkLUjYHjL6ZXbcBuTrK3SpMMjG86hDcZ4sY/pxDmJO055MyyGLbTYMpFI9IIbhDDibKQ4fGPuGdhZpYksXPSnzlIkmZJIcK/QGuIsxsBjNwyWdwglYUbz4RhY6PMG5M3hDQcFyYcqc+BJ++Elksj7RMMNAtqqRIPQqrVOWA/G6QQpKEceUag== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: AnYxlGtb1CXy4u89/DV8xbbBwhJun70th9ldiNpY+qml37Gn0DCtrrtV0sqLjs9RMVVw2G3Mda6G8g1aPZ+l1Q+wBqmi5dkckuQZt4fMXc9KCbMY6flZd4Bki1dnolB369WOMufUP32rFKLDh5gTUXbrSIkoNIfFzBtqDQ1E1n+TwGUewzA0GTdjYEzP4m2C957EH2D1bcEYBOnHvO90x32SOlgodwfhUA5yfO5UVdoJPm54CZQ8MbXpwlMODP6NJRXdp4UHZCiwY0elzWQgplevkV1sW4vBFrWWuEMI3wa/uKqMTvModw82E0CUiavHfjKDp3X/OJwnPfnPAr2taki1+bRiss6VF4hF/oTTHS63OP/vX03Rva4FlF0nP4TEP0lHFa7oEAQvp80JW07QQrPjJH6i32CpuTtd4AwUOPGPPu+ypwsjPSo/n3XAu+dvu1ZEd+u9Li/XIvRQ7Skx+FZCZarfUxfiactA+jpKQP2g2J4TV7trP6vGyyjGSsdfU6gHYUcZoLK2RyGfrpXCMJZPS2/N6hTk63BVztMEXi4DuUQjUyz1Qki71P+CB3iILYPtYc8lSjbLVHvMF4fFxvZxr3tiVxmQBwnJdib+d9agNJystcWfrG0B7/8rLxjq X-OriginatorOrg: drivenets.com X-MS-Exchange-CrossTenant-Network-Message-Id: dfdb6785-4a8d-4e3b-a81d-08dce5d2dab2 X-MS-Exchange-CrossTenant-AuthSource: DB8PR08MB5388.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2024 06:48:15.6092 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 662f82da-cf45-4bdf-b295-33b083f5d229 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: n1gW5oA6RwssmzKkpoD1I0yVCesYnowXXblgmZU9ddQJFxxMBr4/Bq/msy5oYoatBejUrWcJjg+nyWJ1FCT1/g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB6324 X-MDID: 1728197297-Z8IKJYolTCJc X-MDID-O: eu1;fra;1728197297;Z8IKJYolTCJc;;18cd01b0b368a0fec4275fdb61cf0c87 X-PPE-TRUSTED: V=1;DIR=OUT; X-Patchwork-Delegate: kuba@kernel.org Use doubly-linked instead of singly-linked list when linking neighbours, so that it is possible to remove neighbours without traversing the entire table. Signed-off-by: Gilad Naaman --- include/net/neighbour.h | 8 +-- net/core/neighbour.c | 124 ++++++++++++++-------------------------- 2 files changed, 46 insertions(+), 86 deletions(-) diff --git a/include/net/neighbour.h b/include/net/neighbour.h index a44f262a7384..5dde118323e3 100644 --- a/include/net/neighbour.h +++ b/include/net/neighbour.h @@ -135,7 +135,7 @@ struct neigh_statistics { #define NEIGH_CACHE_STAT_INC(tbl, field) this_cpu_inc((tbl)->stats->field) struct neighbour { - struct neighbour __rcu *next; + struct hlist_node list; struct neigh_table *tbl; struct neigh_parms *parms; unsigned long confirmed; @@ -190,7 +190,7 @@ struct pneigh_entry { #define NEIGH_NUM_HASH_RND 4 struct neigh_hash_table { - struct neighbour __rcu **hash_buckets; + struct hlist_head *hash_buckets; unsigned int hash_shift; __u32 hash_rnd[NEIGH_NUM_HASH_RND]; struct rcu_head rcu; @@ -304,9 +304,9 @@ static inline struct neighbour *___neigh_lookup_noref( u32 hash_val; hash_val = hash(pkey, dev, nht->hash_rnd) >> (32 - nht->hash_shift); - for (n = rcu_dereference(nht->hash_buckets[hash_val]); + for (n = (struct neighbour *)rcu_dereference(hlist_first_rcu(&nht->hash_buckets[hash_val])); n != NULL; - n = rcu_dereference(n->next)) { + n = (struct neighbour *)rcu_dereference(hlist_next_rcu(&n->list))) { if (n->dev == dev && key_eq(n, pkey)) return n; } diff --git a/net/core/neighbour.c b/net/core/neighbour.c index 77b819cd995b..86b174baae27 100644 --- a/net/core/neighbour.c +++ b/net/core/neighbour.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -205,18 +206,13 @@ static void neigh_update_flags(struct neighbour *neigh, u32 flags, int *notify, } } -static bool neigh_del(struct neighbour *n, struct neighbour __rcu **np, - struct neigh_table *tbl) +static bool neigh_del(struct neighbour *n, struct neigh_table *tbl) { bool retval = false; write_lock(&n->lock); if (refcount_read(&n->refcnt) == 1) { - struct neighbour *neigh; - - neigh = rcu_dereference_protected(n->next, - lockdep_is_held(&tbl->lock)); - rcu_assign_pointer(*np, neigh); + hlist_del_rcu(&n->list); neigh_mark_dead(n); retval = true; } @@ -228,25 +224,7 @@ static bool neigh_del(struct neighbour *n, struct neighbour __rcu **np, bool neigh_remove_one(struct neighbour *ndel, struct neigh_table *tbl) { - struct neigh_hash_table *nht; - void *pkey = ndel->primary_key; - u32 hash_val; - struct neighbour *n; - struct neighbour __rcu **np; - - nht = rcu_dereference_protected(tbl->nht, - lockdep_is_held(&tbl->lock)); - hash_val = tbl->hash(pkey, ndel->dev, nht->hash_rnd); - hash_val = hash_val >> (32 - nht->hash_shift); - - np = &nht->hash_buckets[hash_val]; - while ((n = rcu_dereference_protected(*np, - lockdep_is_held(&tbl->lock)))) { - if (n == ndel) - return neigh_del(n, np, tbl); - np = &n->next; - } - return false; + return neigh_del(ndel, tbl); } static int neigh_forced_gc(struct neigh_table *tbl) @@ -388,21 +366,20 @@ static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev, for (i = 0; i < (1 << nht->hash_shift); i++) { struct neighbour *n; - struct neighbour __rcu **np = &nht->hash_buckets[i]; + struct neighbour __rcu **np = + (struct neighbour __rcu **)&nht->hash_buckets[i].first; while ((n = rcu_dereference_protected(*np, lockdep_is_held(&tbl->lock))) != NULL) { if (dev && n->dev != dev) { - np = &n->next; + np = (struct neighbour __rcu **)&n->list.next; continue; } if (skip_perm && n->nud_state & NUD_PERMANENT) { - np = &n->next; + np = (struct neighbour __rcu **)&n->list.next; continue; } - rcu_assign_pointer(*np, - rcu_dereference_protected(n->next, - lockdep_is_held(&tbl->lock))); + hlist_del_rcu(&n->list); write_lock(&n->lock); neigh_del_timer(n); neigh_mark_dead(n); @@ -530,9 +507,9 @@ static void neigh_get_hash_rnd(u32 *x) static struct neigh_hash_table *neigh_hash_alloc(unsigned int shift) { - size_t size = (1 << shift) * sizeof(struct neighbour *); + size_t size = (1 << shift) * sizeof(struct hlist_head); struct neigh_hash_table *ret; - struct neighbour __rcu **buckets; + struct hlist_head *buckets; int i; ret = kmalloc(sizeof(*ret), GFP_ATOMIC); @@ -541,7 +518,7 @@ static struct neigh_hash_table *neigh_hash_alloc(unsigned int shift) if (size <= PAGE_SIZE) { buckets = kzalloc(size, GFP_ATOMIC); } else { - buckets = (struct neighbour __rcu **) + buckets = (struct hlist_head *) __get_free_pages(GFP_ATOMIC | __GFP_ZERO, get_order(size)); kmemleak_alloc(buckets, size, 1, GFP_ATOMIC); @@ -562,8 +539,8 @@ static void neigh_hash_free_rcu(struct rcu_head *head) struct neigh_hash_table *nht = container_of(head, struct neigh_hash_table, rcu); - size_t size = (1 << nht->hash_shift) * sizeof(struct neighbour *); - struct neighbour __rcu **buckets = nht->hash_buckets; + size_t size = (1 << nht->hash_shift) * sizeof(struct hlist_head); + struct hlist_head *buckets = nht->hash_buckets; if (size <= PAGE_SIZE) { kfree(buckets); @@ -591,22 +568,18 @@ static struct neigh_hash_table *neigh_hash_grow(struct neigh_table *tbl, for (i = 0; i < (1 << old_nht->hash_shift); i++) { struct neighbour *n, *next; - for (n = rcu_dereference_protected(old_nht->hash_buckets[i], - lockdep_is_held(&tbl->lock)); + for (n = (struct neighbour *) + rcu_dereference_protected(hlist_first_rcu(&old_nht->hash_buckets[i]), + lockdep_is_held(&tbl->lock)); n != NULL; n = next) { hash = tbl->hash(n->primary_key, n->dev, new_nht->hash_rnd); hash >>= (32 - new_nht->hash_shift); - next = rcu_dereference_protected(n->next, - lockdep_is_held(&tbl->lock)); - - rcu_assign_pointer(n->next, - rcu_dereference_protected( - new_nht->hash_buckets[hash], - lockdep_is_held(&tbl->lock))); - rcu_assign_pointer(new_nht->hash_buckets[hash], n); + next = (struct neighbour *)rcu_dereference(hlist_next_rcu(&n->list)); + hlist_del_rcu(&n->list); + hlist_add_head_rcu(&n->list, &new_nht->hash_buckets[hash]); } } @@ -693,11 +666,10 @@ ___neigh_create(struct neigh_table *tbl, const void *pkey, goto out_tbl_unlock; } - for (n1 = rcu_dereference_protected(nht->hash_buckets[hash_val], - lockdep_is_held(&tbl->lock)); - n1 != NULL; - n1 = rcu_dereference_protected(n1->next, - lockdep_is_held(&tbl->lock))) { + hlist_for_each_entry_rcu(n1, + &nht->hash_buckets[hash_val], + list, + lockdep_is_held(&tbl->lock)) { if (dev == n1->dev && !memcmp(n1->primary_key, n->primary_key, key_len)) { if (want_ref) neigh_hold(n1); @@ -713,10 +685,7 @@ ___neigh_create(struct neigh_table *tbl, const void *pkey, list_add_tail(&n->managed_list, &n->tbl->managed_list); if (want_ref) neigh_hold(n); - rcu_assign_pointer(n->next, - rcu_dereference_protected(nht->hash_buckets[hash_val], - lockdep_is_held(&tbl->lock))); - rcu_assign_pointer(nht->hash_buckets[hash_val], n); + hlist_add_head_rcu(&n->list, &nht->hash_buckets[hash_val]); write_unlock_bh(&tbl->lock); neigh_dbg(2, "neigh %p is created\n", n); rc = n; @@ -976,7 +945,7 @@ static void neigh_periodic_work(struct work_struct *work) goto out; for (i = 0 ; i < (1 << nht->hash_shift); i++) { - np = &nht->hash_buckets[i]; + np = (struct neighbour __rcu **)&nht->hash_buckets[i].first; while ((n = rcu_dereference_protected(*np, lockdep_is_held(&tbl->lock))) != NULL) { @@ -999,9 +968,7 @@ static void neigh_periodic_work(struct work_struct *work) (state == NUD_FAILED || !time_in_range_open(jiffies, n->used, n->used + NEIGH_VAR(n->parms, GC_STALETIME)))) { - rcu_assign_pointer(*np, - rcu_dereference_protected(n->next, - lockdep_is_held(&tbl->lock))); + hlist_del_rcu(&n->list); neigh_mark_dead(n); write_unlock(&n->lock); neigh_cleanup_and_release(n); @@ -1010,7 +977,7 @@ static void neigh_periodic_work(struct work_struct *work) write_unlock(&n->lock); next_elt: - np = &n->next; + np = (struct neighbour __rcu **)&n->list.next; } /* * It's fine to release lock here, even if hash table @@ -2728,9 +2695,7 @@ static int neigh_dump_table(struct neigh_table *tbl, struct sk_buff *skb, for (h = s_h; h < (1 << nht->hash_shift); h++) { if (h > s_h) s_idx = 0; - for (n = rcu_dereference(nht->hash_buckets[h]), idx = 0; - n != NULL; - n = rcu_dereference(n->next)) { + hlist_for_each_entry_rcu(n, &nht->hash_buckets[h], list) { if (idx < s_idx || !net_eq(dev_net(n->dev), net)) goto next; if (neigh_ifindex_filtered(n->dev, filter->dev_idx) || @@ -3097,9 +3062,7 @@ void neigh_for_each(struct neigh_table *tbl, void (*cb)(struct neighbour *, void for (chain = 0; chain < (1 << nht->hash_shift); chain++) { struct neighbour *n; - for (n = rcu_dereference(nht->hash_buckets[chain]); - n != NULL; - n = rcu_dereference(n->next)) + hlist_for_each_entry_rcu(n, &nht->hash_buckets[chain], list) cb(n, cookie); } read_unlock_bh(&tbl->lock); @@ -3120,7 +3083,7 @@ void __neigh_for_each_release(struct neigh_table *tbl, struct neighbour *n; struct neighbour __rcu **np; - np = &nht->hash_buckets[chain]; + np = (struct neighbour __rcu **)&nht->hash_buckets[chain].first; while ((n = rcu_dereference_protected(*np, lockdep_is_held(&tbl->lock))) != NULL) { int release; @@ -3128,12 +3091,10 @@ void __neigh_for_each_release(struct neigh_table *tbl, write_lock(&n->lock); release = cb(n); if (release) { - rcu_assign_pointer(*np, - rcu_dereference_protected(n->next, - lockdep_is_held(&tbl->lock))); + hlist_del_rcu(&n->list); neigh_mark_dead(n); } else - np = &n->next; + np = (struct neighbour __rcu **)&n->list.next; write_unlock(&n->lock); if (release) neigh_cleanup_and_release(n); @@ -3200,25 +3161,21 @@ static struct neighbour *neigh_get_first(struct seq_file *seq) state->flags &= ~NEIGH_SEQ_IS_PNEIGH; for (bucket = 0; bucket < (1 << nht->hash_shift); bucket++) { - n = rcu_dereference(nht->hash_buckets[bucket]); - - while (n) { + hlist_for_each_entry_rcu(n, &nht->hash_buckets[bucket], list) { if (!net_eq(dev_net(n->dev), net)) - goto next; + continue; if (state->neigh_sub_iter) { loff_t fakep = 0; void *v; v = state->neigh_sub_iter(state, n, &fakep); if (!v) - goto next; + continue; } if (!(state->flags & NEIGH_SEQ_SKIP_NOARP)) break; if (READ_ONCE(n->nud_state) & ~NUD_NOARP) break; -next: - n = rcu_dereference(n->next); } if (n) @@ -3242,7 +3199,8 @@ static struct neighbour *neigh_get_next(struct seq_file *seq, if (v) return n; } - n = rcu_dereference(n->next); + + n = (struct neighbour *)rcu_dereference(hlist_next_rcu(&n->list)); while (1) { while (n) { @@ -3260,7 +3218,8 @@ static struct neighbour *neigh_get_next(struct seq_file *seq, if (READ_ONCE(n->nud_state) & ~NUD_NOARP) break; next: - n = rcu_dereference(n->next); + + n = (struct neighbour *)rcu_dereference(hlist_next_rcu(&n->list)); } if (n) @@ -3269,7 +3228,8 @@ static struct neighbour *neigh_get_next(struct seq_file *seq, if (++state->bucket >= (1 << nht->hash_shift)) break; - n = rcu_dereference(nht->hash_buckets[state->bucket]); + n = (struct neighbour *) + rcu_dereference(hlist_first_rcu(&nht->hash_buckets[state->bucket])); } if (n && pos) From patchwork Sun Oct 6 06:47:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gilad Naaman X-Patchwork-Id: 13823686 X-Patchwork-Delegate: kuba@kernel.org Received: from dispatch1-eu1.ppe-hosted.com (dispatch1-eu1.ppe-hosted.com [185.183.29.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9709B54F95 for ; Sun, 6 Oct 2024 06:48:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=185.183.29.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728197311; cv=fail; b=FMyBpUDY3Ki2p4ClaUC9Kjpu7EAvPQQFui4MNvwkdpm9it6rk9m4ECZpZ4okNEqIGNPVLKAFu9Xhp557V3Xo8tcoaKC7Y8oolhqM9ky96dszqGkbF1DAFHVJNKb+iaGWSIcM13jx5Htpm4CAX994PBti+tzJSthwjjvU6rnt6b8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728197311; c=relaxed/simple; bh=JvphN2tHvGEP4l5B6iOwskSoRZwepwlt27mJn6Db2yg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Blk3P5GL0pIQK+b+WOdxCqzdyBFv8qcCG3gwVsplO50r/hozbLfZWvte0wC1RFX30sWm/41BSUMhy9iLizTFJ8Rea/TIRhUv+fU3lbhhg4b4OxmdRIy/Mhab0Yzlqpk3uRUZpC7xpJ0llwhVaHMowTxDInsccsrbHFDSzH9fh7E= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=drivenets.com; spf=pass smtp.mailfrom=drivenets.com; dkim=pass (1024-bit key) header.d=drivenets.onmicrosoft.com header.i=@drivenets.onmicrosoft.com header.b=pPrR9mPl; arc=fail smtp.client-ip=185.183.29.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=drivenets.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=drivenets.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=drivenets.onmicrosoft.com header.i=@drivenets.onmicrosoft.com header.b="pPrR9mPl" X-Virus-Scanned: Proofpoint Essentials engine Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05lp2104.outbound.protection.outlook.com [104.47.18.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1-eu1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id 9A845340055; Sun, 6 Oct 2024 06:48:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=v2ZdXJocV17ySsK+AnCMrP3SXbaWAM7MYur0TQWfWdHcPLOXTUZG5b8dXuCXL5FieoZZ+oypA6hjJEKv/DCuwS72fMyDxvSnYCaw+A7nD/mvzeEK7Jxx/RhwfqMRYS9KGJjkspBW3YI3ktOqmQuaW1zOkukU8EqpHHIjQEo+inqLAClysDoodm6/sN+xyjKIBWVaVNh49pSaEh/PHx9icdS/qOxB4O8gpOxa7vleYhWm4bzZIw4+aJlBTCjAH3Sw0fSoG8v1Cp/jOyHkZvFh4W+NlPaTQX6SKV8QugH3mye1b8kkFOQ2v10hmkyYB4qB8TZXkMolxvqP/ayyKp+AGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f+6xHYszBPQ1eYv4BVZFNZOqgd5UmAdnuIjA3StBy98=; b=KraoI2ntflIDDjhoUlRjJ5/TJiDJNNc9FOsrnSt9GbZzaA7iDG2y9cAJY7tqJViN8NAIEJMCG35SFWn8IFblI9kCpbOXSUez6yTlytWhIWNbpbxn5zx/HonGnHEnw+U/Z/bmsa3SFQuxfRyPHrYc743YXjvCF412+VUh4h4SESh40rBFPS94dMDm0N9joreJ6QI3tYJ9Ut9ZoSs2lCiRwjl8EvkasMHeNSBC5lb8dIaD4mao01sXAs9PaU4EKyuYi4PJ6TLRbE1voe5x7GgbBo3L5IRm1JTDI6/nba8rzD5GcLZBa+1qqttksXOZwjFKh8CucUpDZ8QwG3Ko8Db4mQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=drivenets.com; dmarc=pass action=none header.from=drivenets.com; dkim=pass header.d=drivenets.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=drivenets.onmicrosoft.com; s=selector2-drivenets-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f+6xHYszBPQ1eYv4BVZFNZOqgd5UmAdnuIjA3StBy98=; b=pPrR9mPlvmKbaw/fJARZRc5mJlGCpt68Kyt3vQMNfcoF7+KztGS8IVkIGqPQJOEb2zscsUNoq9nB7UV//G1ewHB1AcYHqfwze3tPYuqPGJzYUKni+3wvAUTUPx2N9/9Z9tOfss893SQBiphFz7y/b/jMd8Ixrbp5K3Y88QqasOA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=drivenets.com; Received: from DB8PR08MB5388.eurprd08.prod.outlook.com (2603:10a6:10:11c::7) by AM8PR08MB6324.eurprd08.prod.outlook.com (2603:10a6:20b:315::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.20; Sun, 6 Oct 2024 06:48:16 +0000 Received: from DB8PR08MB5388.eurprd08.prod.outlook.com ([fe80::29dd:6773:4977:dc4e]) by DB8PR08MB5388.eurprd08.prod.outlook.com ([fe80::29dd:6773:4977:dc4e%5]) with mapi id 15.20.8026.020; Sun, 6 Oct 2024 06:48:16 +0000 From: Gilad Naaman To: netdev , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Gilad Naaman Subject: [PATCH net-next v2 2/2] Create netdev->neighbour association Date: Sun, 6 Oct 2024 06:47:43 +0000 Message-ID: <20241006064747.201773-3-gnaaman@drivenets.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241006064747.201773-1-gnaaman@drivenets.com> References: <20241006064747.201773-1-gnaaman@drivenets.com> X-ClientProxiedBy: LO2P123CA0071.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1::35) To DB8PR08MB5388.eurprd08.prod.outlook.com (2603:10a6:10:11c::7) Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB8PR08MB5388:EE_|AM8PR08MB6324:EE_ X-MS-Office365-Filtering-Correlation-Id: d239fed3-b445-4f6f-dbc8-08dce5d2db39 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|52116014|366016|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: YGP48KOss6jwWzQsupsxTk3TeOuRrsY5v4nNGLiIVmRFyArVY0t0U3k/Kb5ZnAEAP4yIaxEvTIlFwejRJozgk2fKX5tcp7c3Osf+bEbKxOSpnFnSngngjSEtvS2utOQUY44Q1TlkO6WfAwlFNeqKv/tSveDbbvdFX2+oVWcrHzoi/hrpK0yS3wzcWkOcvE00Ifon1xGlg5qVTnw+leTA8F7kwwB/j9MEnMk/gTkFxfj0zekvMnfN4UVmZskn1hLjiXA+e7sfVqNkGidjmozN79KTeW72BjfI5xKmIHZZbI2btxw+Kw+d4z8/XbODFRz/Fg3A3gz3Ahj48zkVKIxr9ndUfz83I0UapEAs7HHtPMz/kX7BM3D7nXPFFLlomlMANAX2Ckv9jm8IdQq9I4QvkWbUiGfOwpsAzBVlRYYQZrCExAObWbAy5LhBiIYefVz0s5HxCjk7KDA/gWTZSrjTyV7uDKtBl7pxhYwyWNCpxMllTJcdMwxmS/AqGqylRb2U3itt8z67dROCEquu5rAFFFaaNNdx0ExWnjqOwvpiZKiyGE/2LizvjWczQn3hhiDhYcsqUyWMfGZtIfS1qP2Wkuk86V0hDPDz9tEZ6v39tCVDVvrJTXuPoc3ebC349yTdp88Pbl53T+EGpEpVcMToemVmTwTz304fYkpJZXb7FkoniUflvN6fFGJv24YjZ13TtFdiII7GE1mjxSG1qHYNaaUW3OGd3zPMUdVFGAC+Y8ZQyqreDKlv7V/Bp9ScCJZDdYMtTURV1os3a3ox4fmhEGYsytWWH+cisxxqUh+d6fdoEn60IvEoIlyBCqFf4+2MhcCXCOnk0aypya5JxFRPaWJmc+UWZAg38xM4wMjEsJ/iWxjZWEEROxsJKn+CR2Mzr2bYlYmbkmENBp7rjuIRen3eG4owD5xo4/vC/+TduZ+pIdxudWv2RirVE/ykflvYa1p/GK0kEJMJg1PmxxYhB0wtmXDf0g8y9Ohw8htRIODwnPlvrUYxK150nmc9lhl1EoDxt+PBlFqfGWMTIzu7HTAu7iCPSfiAtGMO8mlEmC+ha8gLdoTdcrf7whCnb7BsLLWHV3fLYdIsoTZUa8QEgINFaMUQNfsAhEHUC52pJrDzIZAoyG3J7MBWopd1IvmohgzI6Epz/I0AG0rPI1qIRZ3NiFxerx1Najxf26OqazU0vp99+FBXUtW3YU6YKkMeB1k0nGB7rDTzzDafYw4Jrre+qZ1jW0MtzdNaFs6Ryik/WUJ/jigoPIWtQ7UpVSlfQshHAkgSIFApw/SKJLEl98M9y0rq3H8cVQ+f1ICJBcq3UyeWTHxG0nWER2dAKjNkexKrbOZPVhzeLsn/uAXZTw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DB8PR08MB5388.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(52116014)(366016)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Dh0/tdPt4ID1EoKIXyQudbdi4enezgKsxmJ+M/LHdUsIwgUDTn9bu0x33Vsf1VeWhPOrZ9ir2UwdKxFysbL4ItJES/ucvRmE1KrTkrRM+xPQoevuBePwMUk5ZD9+gfD68jNjCo84TK2KbpgiSHI8cJjd92IB+DLIwjon/wxJo7DmQ8EVkih9pkc0maoN+i8u+CWMmgV6/e3I6WJ0qABjmcBPCNlBYiblyziSJ8/1igYlSBnYHmgxF30RX1Rectbk/isNAGaWCjdehj6pL+Sjk4YL2fgZi7HMnBiUhk1Lo8v6CY7g4p3TO1FRzmv0vvoqrQ89fG1UDR7k4KzTnSCI7/NRiQxYc9mo/fvHm87zLFRZifDJIiKGU5Y7JK4SIHUwCb4HmsZz/U5PRMY+7ScyMbgod2R/yepmo521DNJYVcDAbMZ5p0vZXC3ywT+zGscd0ArEEsopoim+MUeZCZovWM+2lNvPs6/aBR+RZVZ1vsziPddlIp/1bOANqHxOVp3oxCSlVYjAecnD6qWLugmo+xYhmPROrtIPmVD0P9iFYL7RWtvXzrye3l4GUcdDjZhPHnnR8IecoEf7u3cHKrJXTbjgvMLXHVhGLSG3a8DjrV2otcsMd/fR5lzMqK8TsZ1UJC8BId11CppZ6L36xzwPccx6rZv+h0HJd4brI5J8yHHN1Zonlk+q9wRhzO36VJ5Gev9k6wKEHSwlp9OlVtw0NQe2NpRt+5Aebc2Tc9d2WzZe6PGyZ3seJgPMLZ7D9U4+VAr/y9Q3CANcHOIscTr/lfkW2hydMLNgjt1pDEu3ni+bVU1UDjAjEFeosJwGuxshck8sOtYnQG4xUsubZIL5Ix79jT8KJpV0BTRAvF7m3tkbuUXc67HJ9hmF7ADRGMeJrkfKb/oRgIyRlhXQQc122zFQBD/dICJ5+5F/B8oZt1Q82Crm3iDU/YspMSS29nxnlwsh+4ulYESJXdAPBW8dB/IDnkUA5IrG7Ssq3BGnnwnFMF0gn/8ElC6yYNFjaAJB6tMgKiwlr2gmktVsCqDDjIlZohk0cLYN393VMcCzZqXh0Q2O95BM3q02GtgUJlZ0/qnnkNA8ufc6ZYHrbTWAXBZGSMruLUaRwnfQU6HAhv7HuVHnjfWDGnKqYwf2fy1Dkh8LhMJeoyJyU6ottNPrfeEmgyRzA7JDQK1DfzVisew5wFrT9NOuxvZD56MFWL5TP9nAMPHxRn9bbZ9CgIj/9hvHrMbelY21bIH6D97jeBNFJB2qlbSchDG40DY5TKXzbHtxQItf6LUh3PZq99+hZB/zizO9SZIta7rZ6blt8gEb4cpSsGDOfROuzP1kcKYY4DsQRfPT6UbQoNL+wL3h+/bI3OYadS/ZI1PoiE9I3aOigg3UBGhb49O4hETD7YNJBRrTDr2sulzLf4FmGBSnnn2GpK6IkdidbL8QFVY6ci2uPCF/Z1oUMHC6klt3P4VFUkvqxoQ0fjslI4QEWNGsOpRHF0n0mjgYM8VNBpbe2otaxTUpYnQ9abbrrMFXpF2nWb3MVTEVR+GVrhHW/4EWLdM+vixdz/I5QnRFE01rE1XEvy/WEyrDUfRXOucVDC0YvoDNIqtvQlebKQjwuSQ/Ow== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: zWTCukFrAF+avADgZ1WpSmTAp5UaL5H2HH76HFcXc2PuDz2w4GxLb7dK/5KQhmf/BKNpI89SjQ2HYPqUgGLyk5bDrCJhqYDeheZDEevgE4p553Tz+idYhFKcA8GuVh8aMTzbylbs4Btzqub5fRVX9G9qeiAE896r8CBCwET2hb1cw1/L+0tAf7e4V5RxoLTgl03HTXcE1D+FMvHIXW/3ffy0QT5eSc0oLOrxi1nP0T/F/ssIikbeeKOlUq75F+nEQ6WnZPRwfsAKbO1pxoXOKWi9lCQrygiHMUkh5mS8PWV3J4+UbrKFLnsLi9rnz49e9M9A5iEA1AXm6vwu0d502KUH+/KQPn1CcV1n7w52y/oHKxITwAsSlz6WS6QAZ760Da5gR8zRvSDyKuHv7LVLYJc6iBcTTlMS6x7yDehpffP6Ji6opH0TKMbhhdBAqPIla/fECzLjJWkl1+X+onYbeuZv5sYPEc/rETXcS2KuPUzE9kEyelhrKQmPwjoqXZmh0KZA99e6YGnOf8QhB5QXUOQLT7tmYyH8hkg++5hoFW1gUKDLmckaVsdpa0lf93Z5e6um2rqR9MDDgJfwQFM25r2fnXacIgFsZ9q2uJeGsgnLL/2nhjPUydCjUTa2kt9+ X-OriginatorOrg: drivenets.com X-MS-Exchange-CrossTenant-Network-Message-Id: d239fed3-b445-4f6f-dbc8-08dce5d2db39 X-MS-Exchange-CrossTenant-AuthSource: DB8PR08MB5388.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2024 06:48:16.4959 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 662f82da-cf45-4bdf-b295-33b083f5d229 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7pUyy8WzMcgEwTGJopYhGznHr7QDDRXvazmH5hWifw43joAmgjtrw0aA2VH/6mhY8Po/BdEBdbbHOjtePMQvOQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB6324 X-MDID: 1728197298-KoP6iQgRSDeU X-MDID-O: eu1;ams;1728197298;KoP6iQgRSDeU;;18cd01b0b368a0fec4275fdb61cf0c87 X-PPE-TRUSTED: V=1;DIR=OUT; X-Patchwork-Delegate: kuba@kernel.org Create a mapping between a netdev and its neighoburs, allowing for much cheaper flushes. Signed-off-by: Gilad Naaman --- .../networking/net_cachelines/net_device.rst | 1 + include/linux/netdevice.h | 6 ++ include/net/neighbour.h | 10 +- include/net/neighbour_tables.h | 13 +++ net/core/neighbour.c | 102 +++++++++++++----- 5 files changed, 99 insertions(+), 33 deletions(-) create mode 100644 include/net/neighbour_tables.h diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst index 22b07c814f4a..510c407d7268 100644 --- a/Documentation/networking/net_cachelines/net_device.rst +++ b/Documentation/networking/net_cachelines/net_device.rst @@ -183,3 +183,4 @@ struct_devlink_port* devlink_port struct_dpll_pin* dpll_pin struct hlist_head page_pools struct dim_irq_moder* irq_moder +struct hlist_head neighbours[3] diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e87b5e488325..071b89c56e07 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -52,6 +52,7 @@ #include #include #include +#include struct netpoll_info; struct device; @@ -2009,6 +2010,9 @@ enum netdev_reg_state { * @dpll_pin: Pointer to the SyncE source pin of a DPLL subsystem, * where the clock is recovered. * + * @neighbours: List heads pointing to this device's neighbours' + * dev_list, one per address-family. + * * FIXME: cleanup struct net_device such that network protocol info * moves out. */ @@ -2399,6 +2403,8 @@ struct net_device { /** @irq_moder: dim parameters used if IS_ENABLED(CONFIG_DIMLIB). */ struct dim_irq_moder *irq_moder; + struct hlist_head neighbours[NEIGH_NR_TABLES]; + u8 priv[] ____cacheline_aligned __counted_by(priv_len); } ____cacheline_aligned; diff --git a/include/net/neighbour.h b/include/net/neighbour.h index 5dde118323e3..dee5c6337fb5 100644 --- a/include/net/neighbour.h +++ b/include/net/neighbour.h @@ -29,6 +29,7 @@ #include #include #include +#include /* * NUD stands for "neighbor unreachability detection" @@ -136,6 +137,7 @@ struct neigh_statistics { struct neighbour { struct hlist_node list; + struct hlist_node dev_list; struct neigh_table *tbl; struct neigh_parms *parms; unsigned long confirmed; @@ -236,14 +238,6 @@ struct neigh_table { struct pneigh_entry **phash_buckets; }; -enum { - NEIGH_ARP_TABLE = 0, - NEIGH_ND_TABLE = 1, - NEIGH_DN_TABLE = 2, - NEIGH_NR_TABLES, - NEIGH_LINK_TABLE = NEIGH_NR_TABLES /* Pseudo table for neigh_xmit */ -}; - static inline int neigh_parms_family(struct neigh_parms *p) { return p->tbl->family; diff --git a/include/net/neighbour_tables.h b/include/net/neighbour_tables.h new file mode 100644 index 000000000000..ad98b49d58db --- /dev/null +++ b/include/net/neighbour_tables.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _NET_NEIGHBOUR_TABLES_H +#define _NET_NEIGHBOUR_TABLES_H + +enum { + NEIGH_ARP_TABLE = 0, + NEIGH_ND_TABLE = 1, + NEIGH_DN_TABLE = 2, + NEIGH_NR_TABLES, + NEIGH_LINK_TABLE = NEIGH_NR_TABLES /* Pseudo table for neigh_xmit */ +}; + +#endif diff --git a/net/core/neighbour.c b/net/core/neighbour.c index 86b174baae27..608f001e320f 100644 --- a/net/core/neighbour.c +++ b/net/core/neighbour.c @@ -62,6 +62,20 @@ static int pneigh_ifdown_and_unlock(struct neigh_table *tbl, static const struct seq_operations neigh_stat_seq_ops; #endif +static int family_to_neightbl_index(int family) +{ + switch (family) { + case AF_INET: + return NEIGH_ARP_TABLE; + case AF_INET6: + return NEIGH_ND_TABLE; + case AF_DECnet: + return NEIGH_DN_TABLE; + default: + return -1; + } +} + /* Neighbour hash table buckets are protected with rwlock tbl->lock. @@ -213,6 +227,7 @@ static bool neigh_del(struct neighbour *n, struct neigh_table *tbl) write_lock(&n->lock); if (refcount_read(&n->refcnt) == 1) { hlist_del_rcu(&n->list); + hlist_del_rcu(&n->dev_list); neigh_mark_dead(n); retval = true; } @@ -355,12 +370,65 @@ static void pneigh_queue_purge(struct sk_buff_head *list, struct net *net, } } +static void _neigh_flush_free_neigh(struct neighbour *n) +{ + hlist_del_rcu(&n->list); + hlist_del_rcu(&n->dev_list); + write_lock(&n->lock); + neigh_del_timer(n); + neigh_mark_dead(n); + if (refcount_read(&n->refcnt) != 1) { + /* The most unpleasant situation. + * We must destroy neighbour entry, + * but someone still uses it. + * + * The destroy will be delayed until + * the last user releases us, but + * we must kill timers etc. and move + * it to safe state. + */ + __skb_queue_purge(&n->arp_queue); + n->arp_queue_len_bytes = 0; + WRITE_ONCE(n->output, neigh_blackhole); + if (n->nud_state & NUD_VALID) + n->nud_state = NUD_NOARP; + else + n->nud_state = NUD_NONE; + neigh_dbg(2, "neigh %p is stray\n", n); + } + write_unlock(&n->lock); + neigh_cleanup_and_release(n); +} + +static void neigh_flush_dev_fast(struct neigh_table *tbl, struct hlist_node *next, + bool skip_perm) +{ + struct neighbour *n; + + while (next) { + n = container_of(next, struct neighbour, dev_list); + next = rcu_dereference(hlist_next_rcu(next)); + if (skip_perm && n->nud_state & NUD_PERMANENT) + continue; + + _neigh_flush_free_neigh(n); + } +} + static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev, bool skip_perm) { int i; struct neigh_hash_table *nht; + i = family_to_neightbl_index(tbl->family); + if (i != -1) { + neigh_flush_dev_fast(tbl, + rcu_dereference(hlist_first_rcu(&dev->neighbours[i])), + skip_perm); + return; + } + nht = rcu_dereference_protected(tbl->nht, lockdep_is_held(&tbl->lock)); @@ -379,31 +447,8 @@ static void neigh_flush_dev(struct neigh_table *tbl, struct net_device *dev, np = (struct neighbour __rcu **)&n->list.next; continue; } - hlist_del_rcu(&n->list); - write_lock(&n->lock); - neigh_del_timer(n); - neigh_mark_dead(n); - if (refcount_read(&n->refcnt) != 1) { - /* The most unpleasant situation. - We must destroy neighbour entry, - but someone still uses it. - - The destroy will be delayed until - the last user releases us, but - we must kill timers etc. and move - it to safe state. - */ - __skb_queue_purge(&n->arp_queue); - n->arp_queue_len_bytes = 0; - WRITE_ONCE(n->output, neigh_blackhole); - if (n->nud_state & NUD_VALID) - n->nud_state = NUD_NOARP; - else - n->nud_state = NUD_NONE; - neigh_dbg(2, "neigh %p is stray\n", n); - } - write_unlock(&n->lock); - neigh_cleanup_and_release(n); + + _neigh_flush_free_neigh(n); } } } @@ -686,6 +731,11 @@ ___neigh_create(struct neigh_table *tbl, const void *pkey, if (want_ref) neigh_hold(n); hlist_add_head_rcu(&n->list, &nht->hash_buckets[hash_val]); + + error = family_to_neightbl_index(tbl->family); + if (error != -1) + hlist_add_head_rcu(&n->dev_list, &dev->neighbours[error]); + write_unlock_bh(&tbl->lock); neigh_dbg(2, "neigh %p is created\n", n); rc = n; @@ -969,6 +1019,7 @@ static void neigh_periodic_work(struct work_struct *work) !time_in_range_open(jiffies, n->used, n->used + NEIGH_VAR(n->parms, GC_STALETIME)))) { hlist_del_rcu(&n->list); + hlist_del_rcu(&n->dev_list); neigh_mark_dead(n); write_unlock(&n->lock); neigh_cleanup_and_release(n); @@ -3092,6 +3143,7 @@ void __neigh_for_each_release(struct neigh_table *tbl, release = cb(n); if (release) { hlist_del_rcu(&n->list); + hlist_del_rcu(&n->dev_list); neigh_mark_dead(n); } else np = (struct neighbour __rcu **)&n->list.next;