From patchwork Mon Feb 21 10:53:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1202AC433FE for ; Mon, 21 Feb 2022 10:54:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C95E8D0008; Mon, 21 Feb 2022 05:54:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 854308D0001; Mon, 21 Feb 2022 05:54:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67F0A8D0008; Mon, 21 Feb 2022 05:54:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 5249F8D0001 for ; Mon, 21 Feb 2022 05:54:10 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EEBC8180E8950 for ; Mon, 21 Feb 2022 10:54:09 +0000 (UTC) X-FDA: 79166477418.30.3E9CA43 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf17.hostedemail.com (Postfix) with ESMTP id 67D1E40009 for ; Mon, 21 Feb 2022 10:54:09 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id gi6so5232237pjb.1 for ; Mon, 21 Feb 2022 02:54:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mNqSGZPctSWXqZ6gYN0FXGhiJj1MEcwtFP+jmJt2lAM=; b=RFTPXk5DrwGPw8J5yfqw2l+chmPeFSAdBQcKMCI2hS+QHbqTp47fbGdwOgCxkQjL/5 s4KOqNYdUwQnAXGGSon/S4gwtsNgaUQm+hPClmpa6dsyOiUsk4bIbjjXDMN/Mr2LxJbF 8iglITyjVsZYvBOGkg8UpLRM/fXaOMnQJleiLza6uMPjkMb/zZEeyzuW3RjXegMQEwQW 5MfkgPkZqcSZ8OXKsXauuKkYUDFECPq7K/51WtVTzGXzDDDE5FHKMFo7H6Q04KmMXAxr TmgkL1ED1/zGJod/kfDAgFV7FFH/oMNp1EaundBNuAFgPhyWacQ4ZfNwxfK6yHdTy9nY vRxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mNqSGZPctSWXqZ6gYN0FXGhiJj1MEcwtFP+jmJt2lAM=; b=GoeLz9u8CxuW8usOu8ctCPcWdtEVsL9NUNnmia0AHH1tWSeor/qkhFL0d57jhBi537 jJ3jW1d98TtSlUoBK2WiUmvWZO2vde8zWpZR68EC5cGi2Ug4XHZAEHL5AtWe04J+TjKD VKTfMijAiGzza5pO0+ZbhM+s3d4KUKSLHw+AT+dDjeX9SU9U7FG/2PSf1dQMWfp22+KI JyjAdvB8IvZ+aTr2SKxXF2i20Pk1xWOWSYP+EfXENzlWPLfkWbmFxkwt98ElvBPZY20G dUxzNK6jZXk2KqhH7xx8ngB+/OMtBG1ypT8ro+Ny92n4IeiPHugPfgX9TPXjBNo6NWpO nkkg== X-Gm-Message-State: AOAM531zLQ2PWT6VDuI3TkY0ZPBm6TUD2QDrpwFEtm/fMtjMd1pI052u O/tXAJZmqOBiJWiJdBkGwd1dTyf5k0z9dA== X-Google-Smtp-Source: ABdhPJywNTHU63XkzHt51xLMmCuOt0KCbfUmRMm0QdINnoAQ9l+TE+h6FliiSaxwldW7bQVaCkRlDQ== X-Received: by 2002:a17:90a:8048:b0:1b9:55dd:b72d with SMTP id e8-20020a17090a804800b001b955ddb72dmr20733044pjw.90.1645440848421; Mon, 21 Feb 2022 02:54:08 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.54.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:54:08 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 5/5] mm/slub: Refactor deactivate_slab() Date: Mon, 21 Feb 2022 10:53:36 +0000 Message-Id: <20220221105336.522086-6-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 9qx3sodqrcb8qobyupj1oux4m6g1erm4 X-Rspam-User: Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RFTPXk5D; spf=pass (imf17.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67D1E40009 X-HE-Tag: 1645440849-969872 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simply deactivate_slab() by removing variable 'lock' and replacing 'l' and 'm' with 'mode'. Instead, remove slab from list and unlock n->list_lock when cmpxchg_double() fails, and then retry. One slight functional change is releasing and taking n->list_lock again when cmpxchg_double() fails. This is not harmful because SLUB avoids deactivating slabs as much as possible. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 74 +++++++++++++++++++++++++------------------------------ 1 file changed, 33 insertions(+), 41 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a4964deccb61..2d0663befb9e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2350,8 +2350,8 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, { enum slab_modes { M_NONE, M_PARTIAL, M_FULL, M_FREE }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); - int lock = 0, free_delta = 0; - enum slab_modes l = M_NONE, m = M_NONE; + int free_delta = 0; + enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2420,57 +2420,49 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen = 0; if (!new.inuse && n->nr_partial >= s->min_partial) - m = M_FREE; + mode = M_FREE; else if (new.freelist) { - m = M_PARTIAL; - if (!lock) { - lock = 1; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ - spin_lock_irqsave(&n->list_lock, flags); - } - } else { - m = M_FULL; - if (kmem_cache_debug_flags(s, SLAB_STORE_USER) && !lock) { - lock = 1; - /* - * This also ensures that the scanning of full - * slabs from diagnostic functions will not see - * any frozen slabs. - */ - spin_lock_irqsave(&n->list_lock, flags); - } + mode = M_PARTIAL; + /* + * Taking the spinlock removes the possibility that + * acquire_slab() will see a slab that is frozen + */ + spin_lock_irqsave(&n->list_lock, flags); + add_partial(n, slab, tail); + } else if (kmem_cache_debug_flags(s, SLAB_STORE_USER)) { + mode = M_FULL; + /* + * This also ensures that the scanning of full + * slabs from diagnostic functions will not see + * any frozen slabs. + */ + spin_lock_irqsave(&n->list_lock, flags); + add_full(s, n, slab); } - if (l != m) { - if (l == M_PARTIAL) - remove_partial(n, slab); - else if (l == M_FULL) - remove_full(s, n, slab); - if (m == M_PARTIAL) - add_partial(n, slab, tail); - else if (m == M_FULL) - add_full(s, n, slab); - } - - l = m; if (!cmpxchg_double_slab(s, slab, old.freelist, old.counters, new.freelist, new.counters, - "unfreezing slab")) + "unfreezing slab")) { + if (mode == M_PARTIAL) { + remove_partial(n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); + } else if (mode == M_FULL) { + remove_full(s, n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); + } goto redo; + } - if (lock) - spin_unlock_irqrestore(&n->list_lock, flags); - if (m == M_PARTIAL) + if (mode == M_PARTIAL) { + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - else if (m == M_FULL) + } else if (mode == M_FULL) { + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, DEACTIVATE_FULL); - else if (m == M_FREE) { + } else if (mode == M_FREE) { stat(s, DEACTIVATE_EMPTY); discard_slab(s, slab); stat(s, FREE_SLAB);