From patchwork Fri Mar 4 06:34:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12768535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB6DC433EF for ; Fri, 4 Mar 2022 06:34:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 776E18D0008; Fri, 4 Mar 2022 01:34:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FDA18D0001; Fri, 4 Mar 2022 01:34:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5777B8D0008; Fri, 4 Mar 2022 01:34:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 4998D8D0001 for ; Fri, 4 Mar 2022 01:34:58 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 2248580727 for ; Fri, 4 Mar 2022 06:34:58 +0000 (UTC) X-FDA: 79205741076.06.F0D8823 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf09.hostedemail.com (Postfix) with ESMTP id 99AC4140008 for ; Fri, 4 Mar 2022 06:34:56 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id p3-20020a17090a680300b001bbfb9d760eso9862265pjj.2 for ; Thu, 03 Mar 2022 22:34:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PgzWIJiGbElIzAA/+g3O4R+XIWRWlKNYEXwjILpxSlg=; b=VcsP531XOAZHoSzYutr+YrhPB/02f1REI7a2paVkswi+VQ1RJWPGg8afSzCgogHzyW dOUmDbKrKlQ2pIGPipEijTjuCJ2N6uNcjHbyMswPnO5gSGm6KKC8lgnMd5CDOze45fm6 ox2otFm8rAec4bBP8ZSzRbgS3yv+isnJSeAwbZSwGe0Xkpa/AIWIQs8BoVspdQDH/ejc RRFaT/TeqUqitkGA1WFWDZ75k4/o1UNUPoYyOumZB47EYuTWitcDPlm20xlJHkYfJCpF UAvLaQjJ1EFDEDIzDPfSx00I4XvsOPzMJ4rQuu9LeeLw8WyY2JgBU6nlJxw2wLfrXsCY DdbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PgzWIJiGbElIzAA/+g3O4R+XIWRWlKNYEXwjILpxSlg=; b=KIRVeN91qzs1vx6ncY0p8nYKlmzL4adMCZQD8X8nvmCZDtCS736pMF04dRjfXYSP+E x7yW8veijA3phvmeEJO110/T0rUv/lXpXR/FAVJJK+mOH7Fn1IEMGY1tVdNRUJrY+DIn /jV2ra7tsob74E1S8L8VvZNoMtvlUNZxqlZQPTriB5FEClvpuGmJwgd6mZ45iB/KK9an hZZTIV4t1w/Vmwaqzci4rS44fTWDRqyCseYeB5tsbBchoUrM4F8k4G9ihF39ZI1GLgte jVmTmhxsSET4S/EPTuPR63/CLXknKXLh5hVDBXF5tL3MWdYcNz1AJ2IW+cbdQHFNzG9F U+hA== X-Gm-Message-State: AOAM532v0kriFdl+EbGL+b95HUp9XcgZpoekk5tiRu0uUNKO5bo3jmKT BA0L/edOSdyEx5M6x+fOzf44dBARWBzSmA== X-Google-Smtp-Source: ABdhPJxUr2XDiJRBOKg+kXq0iUgKNnV8N4aWOKmy0bwGTHBAdBDB2wi3hQmPcbrds4di6UaI00Dbuw== X-Received: by 2002:a17:902:c401:b0:14f:b686:e6ff with SMTP id k1-20020a170902c40100b0014fb686e6ffmr15079443plk.45.1646375695469; Thu, 03 Mar 2022 22:34:55 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id v10-20020a056a00148a00b004e0f420dd90sm4900007pfu.40.2022.03.03.22.34.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 22:34:55 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Marco Elver , Matthew WilCox , Roman Gushchin , linux-kernel@vger.kernel.org, 42.hyeyoo@gmail.com Subject: [PATCH v2 5/5] mm/slub: refactor deactivate_slab() Date: Fri, 4 Mar 2022 06:34:27 +0000 Message-Id: <20220304063427.372145-6-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220304063427.372145-1-42.hyeyoo@gmail.com> References: <20220304063427.372145-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 99AC4140008 X-Stat-Signature: p4mexj3ctwz1yfa3wrbhefa5tptxqcuy Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VcsP531X; spf=pass (imf09.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-HE-Tag: 1646375696-837882 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simplify deactivate_slab() by unlocking n->list_lock and retrying cmpxchg_double() when cmpxchg_double() fails, and perform add_{partial,full} only when it succeed. Releasing and taking n->list_lock again here is not harmful as SLUB avoids deactivating slabs as much as possible. [ vbabka@suse.cz: perform add_{partial,full} when cmpxchg_double() succeed. ] Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 81 ++++++++++++++++++++++--------------------------------- 1 file changed, 32 insertions(+), 49 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f9ae983a3dc6..c1a693ec5874 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2344,8 +2344,8 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, { enum slab_modes { M_NONE, M_PARTIAL, M_FULL, M_FREE }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); - int lock = 0, free_delta = 0; - enum slab_modes l = M_NONE, m = M_NONE; + int free_delta = 0; + enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2387,14 +2387,10 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, * Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. * - * We setup the list membership and then perform a cmpxchg - * with the count. If there is a mismatch then the slab - * is not unfrozen but the slab is on the wrong list. - * - * Then we restart the process which may have to remove - * the slab from the list that we just put it on again - * because the number of objects in the slab may have - * changed. + * We first perform cmpxchg holding lock and insert to list + * when it succeed. If there is mismatch then slub is not + * unfrozen and number of objects in the slab may have changed. + * Then release lock and retry cmpxchg again. */ redo: @@ -2414,57 +2410,44 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen = 0; if (!new.inuse && n->nr_partial >= s->min_partial) - m = M_FREE; + mode = M_FREE; else if (new.freelist) { - m = M_PARTIAL; - if (!lock) { - lock = 1; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ - spin_lock_irqsave(&n->list_lock, flags); - } - } else { - m = M_FULL; - if (kmem_cache_debug_flags(s, SLAB_STORE_USER) && !lock) { - lock = 1; - /* - * This also ensures that the scanning of full - * slabs from diagnostic functions will not see - * any frozen slabs. - */ - spin_lock_irqsave(&n->list_lock, flags); - } + mode = M_PARTIAL; + /* + * Taking the spinlock removes the possibility that + * acquire_slab() will see a slab that is frozen + */ + spin_lock_irqsave(&n->list_lock, flags); + } else if (kmem_cache_debug_flags(s, SLAB_STORE_USER)) { + mode = M_FULL; + /* + * This also ensures that the scanning of full + * slabs from diagnostic functions will not see + * any frozen slabs. + */ + spin_lock_irqsave(&n->list_lock, flags); } - if (l != m) { - if (l == M_PARTIAL) - remove_partial(n, slab); - else if (l == M_FULL) - remove_full(s, n, slab); - - if (m == M_PARTIAL) - add_partial(n, slab, tail); - else if (m == M_FULL) - add_full(s, n, slab); - } - l = m; if (!cmpxchg_double_slab(s, slab, old.freelist, old.counters, new.freelist, new.counters, - "unfreezing slab")) + "unfreezing slab")) { + if (mode == M_PARTIAL || mode == M_FULL) + spin_unlock_irqrestore(&n->list_lock, flags); goto redo; + } - if (lock) - spin_unlock_irqrestore(&n->list_lock, flags); - if (m == M_PARTIAL) + if (mode == M_PARTIAL) { + add_partial(n, slab, tail); + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - else if (m == M_FULL) + } else if (mode == M_FULL) { + add_full(s, n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, DEACTIVATE_FULL); - else if (m == M_FREE) { + } else if (mode == M_FREE) { stat(s, DEACTIVATE_EMPTY); discard_slab(s, slab); stat(s, FREE_SLAB);