From patchwork Thu Jul 6 10:28:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Huang X-Patchwork-Id: 13303489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A0F0EB64D9 for ; Thu, 6 Jul 2023 10:29:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229528AbjGFK3Y (ORCPT ); Thu, 6 Jul 2023 06:29:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230453AbjGFK3X (ORCPT ); Thu, 6 Jul 2023 06:29:23 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 958C01BCC; Thu, 6 Jul 2023 03:29:22 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-668704a5b5bso484579b3a.0; Thu, 06 Jul 2023 03:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688639362; x=1691231362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TcBm3tIPDYjaiJYtej3V+EGlEbrFWBZv2rFb8nvh3TM=; b=XtYlo7GiKeg0vttl4mWNuLr/TzVJOKTEj+YFuVmcPXjvMjK+MpVDj7DA/JalO3GIxR hTVearElLUpntFjHD20Mn3dyu4+tALjwErzxTJnn1a9g+c9LinhYHeF8DblrVQFt4uNJ 2L7tirS90lVgaHOFKMK2frzJzTuiqHeJXBj8RbTh8IhYEwzURUM9hFaUUAwI57rpnRqc Mz2AOQE+YmwJykf+t6BW0i+pipfliIWKgu8WCaqsVilBDMJ3Im5BwzGGsab7HD9nk3wq ufVgvfUYuP4ymhBge3lawMcO1u9P6pr5gNzNwFElnRu1FZkEozhf/EvVHtXNnuqYZhj5 NoMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688639362; x=1691231362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TcBm3tIPDYjaiJYtej3V+EGlEbrFWBZv2rFb8nvh3TM=; b=KrKQfPzdLr2XtIMi7eMNdPG3vcgc4rhnJHQlx/Nb/erSB02nVdUQo7ieWv1RL6I3sX YNWZnFlAk9MvtNkaGJl0GHRbnajnizOCPqQ8fHknDN1CPsX04WGCHn21L8Oj5u3eex36 rUKZ/hAYjA2246QDq1t1zyoHRwWUgmhX3DnkV9mhagFjPnvQ1gq6Gn/ZtZxXRPZNgtVU AqUKvbVLhWLWR60U0qmdoM4FUy69oJ8H5Ty7K+/qNi212CfP/N1NH6fRQ+SaawNa3H6f Fcp8RztEHMDU+75lm3xChF94eUwqQMw26TSXm8azjf8q3MS53s//plsNnaUQm+s5gSWe ulCw== X-Gm-Message-State: ABy/qLZEnn/O7/3QviYuNcTf3aOYOmPf+IK6NiQGaiGDbI6Fy9Gh6sL+ gHBpjkx7BpMWSlTz+RRqhPc= X-Google-Smtp-Source: APBJJlHEsb5IoJZ00ytZJiqM5dS8+VVGstH4ybAo0LHSmC6/cSczMkE1jRCATMAVN94SoLy0iShd/g== X-Received: by 2002:a05:6a00:1790:b0:681:c372:5aa4 with SMTP id s16-20020a056a00179000b00681c3725aa4mr1778465pfg.27.1688639361080; Thu, 06 Jul 2023 03:29:21 -0700 (PDT) Received: from localhost ([2402:d0c0:2:a2a::1]) by smtp.gmail.com with ESMTPSA id l3-20020a62be03000000b0067aea93af40sm1008849pff.2.2023.07.06.03.29.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 03:29:20 -0700 (PDT) From: Alan Huang To: paulmck@kernel.org, frederic@kernel.org, quic_neeraju@quicinc.com, joel@joelfernandes.org, josh@joshtriplett.org, boqun.feng@gmail.com, corbet@lwn.net Cc: rcu@vger.kernel.org, linux-doc@vger.kernel.org, Alan Huang Subject: [PATCH 1/2] rcu: Use WRITE_ONCE() for assignments to ->next for rculist_nulls Date: Thu, 6 Jul 2023 10:28:48 +0000 Message-Id: <20230706102849.437687-2-mmpgouride@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230706102849.437687-1-mmpgouride@gmail.com> References: <20230706102849.437687-1-mmpgouride@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org When the objects managed by rculist_nulls are allocated with SLAB_TYPESAFE_BY_RCU, readers may still hold references to this object that is being added, which means the modification of ->next is visible to readers. So, this patch uses WRITE_ONCE() for assignments to ->next. Signed-off-by: Alan Huang --- include/linux/rculist_nulls.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h index ba4c00dd8005..89186c499dd4 100644 --- a/include/linux/rculist_nulls.h +++ b/include/linux/rculist_nulls.h @@ -101,7 +101,7 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n, { struct hlist_nulls_node *first = h->first; - n->next = first; + WRITE_ONCE(n->next, first); WRITE_ONCE(n->pprev, &h->first); rcu_assign_pointer(hlist_nulls_first_rcu(h), n); if (!is_a_nulls(first)) @@ -137,7 +137,7 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n, last = i; if (last) { - n->next = last->next; + WRITE_ONCE(n->next, last->next); n->pprev = &last->next; rcu_assign_pointer(hlist_nulls_next_rcu(last), n); } else { From patchwork Thu Jul 6 10:28:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Huang X-Patchwork-Id: 13303490 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22D39EB64D9 for ; Thu, 6 Jul 2023 10:29:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbjGFK3h (ORCPT ); Thu, 6 Jul 2023 06:29:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231236AbjGFK3h (ORCPT ); Thu, 6 Jul 2023 06:29:37 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 544D41BD4; Thu, 6 Jul 2023 03:29:30 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6689430d803so359431b3a.0; Thu, 06 Jul 2023 03:29:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688639370; x=1691231370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wXa/kRvaivKsZJZ49tsL8TqteMRykn/WK5NqJztc71Y=; b=llwTDWRS9JBUk6c7OcaMj4IvgMMFXCbhfdRIINelHzV0KP8UPCOziMrQ3casKnc4TW N6OQzRG0YqY7tT6GAPjF//gFR61B6wjnXCSnKqFY5ngsbcj23WYHboexvMPNbEy+3G4O zWR4sgv+Dycp72mBDw2msFKFbFCGs2GTmQQPv2sI8NCCt/KKjNkYmvFG3lwmVU2HSMDi WmTNOtlMtEkalkyCd485YwZFItqvPRoaDULzqed1uholrRCpkjEYBEBa6snrASHv6ApE cIan3TOXaXbWn2bHMqAvdnv3i7dMkUneLW7/kqWZG7O2PAbD5PXUmoZBFCO/hOmXTtMZ Nx1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688639370; x=1691231370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wXa/kRvaivKsZJZ49tsL8TqteMRykn/WK5NqJztc71Y=; b=EQ5IxC2IhAOq6Ht1qNGdyS2mf2JP1r6+1kVABzj0RqJ2N1sCSPTIsVh4PAN96dKQiZ AfGWY2feo9n8xYLMJb2m4ya9K4TZxzZ6R59XBKXTj2kLquYfMreTu96d9oOUyF5oM2d+ XEsRhwOdfJCy3gExxWHFI58ONSx2yU0WlhA4P6aaPXaRZbVLo7i6J7SmLuCxSmrFoAG4 B2VddGhi6IdjCRt5FRWmDyzMk8th//hN1R/wt9EdwFu5WceeVVXWXJfpCsfiuzvBHQlK NmbaT6e3dJIXtNMCB+LODG4IvSJOYrD9v7AspkLfRXYkmkWwwDPZMwo+oucjZRoVnm28 tWNg== X-Gm-Message-State: ABy/qLYohNQ5TkTDIqJrW7rkYxJw31BcNtKktPk3TkR3hwXKthWt4U9Q dyNInkZmoQv+IjlCBtutOyI= X-Google-Smtp-Source: APBJJlH6rwCxUvgQcUF8OulBWj+asdlkhdqjegBBwX3eqsbhb8l3eL8LS58+a1EA9RUpcs3cI2fatA== X-Received: by 2002:a05:6a00:1a89:b0:682:562a:39ae with SMTP id e9-20020a056a001a8900b00682562a39aemr1176651pfv.32.1688639369627; Thu, 06 Jul 2023 03:29:29 -0700 (PDT) Received: from localhost ([2402:d0c0:2:a2a::1]) by smtp.gmail.com with ESMTPSA id u4-20020aa78484000000b00663ab37ff74sm994162pfn.72.2023.07.06.03.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 03:29:29 -0700 (PDT) From: Alan Huang To: paulmck@kernel.org, frederic@kernel.org, quic_neeraju@quicinc.com, joel@joelfernandes.org, josh@joshtriplett.org, boqun.feng@gmail.com, corbet@lwn.net Cc: rcu@vger.kernel.org, linux-doc@vger.kernel.org, Alan Huang Subject: [PATCH 2/2] docs/RCU: Bring back smp_wmb() Date: Thu, 6 Jul 2023 10:28:49 +0000 Message-Id: <20230706102849.437687-3-mmpgouride@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230706102849.437687-1-mmpgouride@gmail.com> References: <20230706102849.437687-1-mmpgouride@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The objects are allocated with SLAB_TYPESAFE_BY_RCU, and there is n->next = first within hlist_add_head_rcu() before rcu_assign_pointer(), which modifies obj->obj_node.next. There may be readers holding the reference of obj in lockless_lookup, and when updater modifies ->next, readers can see the change immediately because of SLAB_TYPESAFE_BY_RCU. There are two memory ordering required in the insertion algorithm, we need to make sure obj->key is updated before obj->obj_node.next and obj->refcnt, atomic_set_release is not enough to provide the required memory barrier. Signed-off-by: Alan Huang --- Documentation/RCU/rculist_nulls.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Documentation/RCU/rculist_nulls.rst b/Documentation/RCU/rculist_nulls.rst index 21e40fcc08de..fa3729dc7e74 100644 --- a/Documentation/RCU/rculist_nulls.rst +++ b/Documentation/RCU/rculist_nulls.rst @@ -64,7 +64,7 @@ but a version with an additional memory barrier (smp_rmb()) { struct hlist_node *node, *next; for (pos = rcu_dereference((head)->first); - pos && ({ next = pos->next; smp_rmb(); prefetch(next); 1; }) && + pos && ({ next = READ_ONCE(pos->next); smp_rmb(); prefetch(next); 1; }) && ({ obj = hlist_entry(pos, typeof(*obj), obj_node); 1; }); pos = rcu_dereference(next)) if (obj->key == key) @@ -112,7 +112,12 @@ detect the fact that it missed following items in original chain. obj = kmem_cache_alloc(...); lock_chain(); // typically a spin_lock() obj->key = key; - atomic_set_release(&obj->refcnt, 1); // key before refcnt + /* + * We need to make sure obj->key is updated before obj->obj_node.next + * and obj->refcnt. + */ + smp_wmb(); + atomic_set(&obj->refcnt, 1); hlist_add_head_rcu(&obj->obj_node, list); unlock_chain(); // typically a spin_unlock()