From patchwork Mon Apr 19 22:50:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 12212779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCAC7C433B4 for ; Mon, 19 Apr 2021 22:51:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6083861090 for ; Mon, 19 Apr 2021 22:51:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6083861090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6C3AA6B006E; Mon, 19 Apr 2021 18:51:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64D506B0070; Mon, 19 Apr 2021 18:51:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DB706B0071; Mon, 19 Apr 2021 18:51:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id 149336B006E for ; Mon, 19 Apr 2021 18:51:01 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CC4731E1E for ; Mon, 19 Apr 2021 22:51:00 +0000 (UTC) X-FDA: 78050613480.09.8E98F9C Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) by imf24.hostedemail.com (Postfix) with ESMTP id 7DEC7A00038D for ; Mon, 19 Apr 2021 22:50:52 +0000 (UTC) Received: by mail-io1-f43.google.com with SMTP id k25so36571306iob.6 for ; Mon, 19 Apr 2021 15:51:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lGtwy4gezzBoO9ZhoFC/C6pqVb7Mu05Yezx/T7Ux99c=; b=GXleaH2ftAwQryd1BZutzaGSmlQFVKc07pJvGxGqnj6MGlldZ3DW/RxvF79+yiSl/h LvtmLs0W3bnTsFpJLzXZR7b8XbaJNTsNfa4d/hlAb+tVy7nCdtyGLk3wc8uhpM66WaVD /13mv+jbtuGpzJvBb4zqk8BgQCJE+K7HdsA+eNn0zvpgWlVAxA/DJYh4iqtW1J/kueh4 CUvrjXRGS9q2RioDeQjZ0J0gTcY46jsq3WvJV1v2oyMRBKYzEjV5Xa6OJT2LZimDLzn5 1gCnNmfgAIa5xtf1gIpX/61gXpr3nTNphWhlbWxavdvib4KkvQOSW0p2L1Gv4VKTsxfz gKag== X-Gm-Message-State: AOAM532aOUAOs/1nWziYU/yGV7OJ93Sz5/EArWvh8K0vKh8aht4LfdSL 1h4C8Kw+gmzchZASxesp/59t4l96k58= X-Google-Smtp-Source: ABdhPJzVpah/mlRGRhTqneMMEk4zOmZqnaajjyfU5u+TLUhwEMzbasZ2Cl5sAye8L7uMex41uoQHiA== X-Received: by 2002:a02:a06:: with SMTP id 6mr13915809jaw.112.1618872659936; Mon, 19 Apr 2021 15:50:59 -0700 (PDT) Received: from abasin.c.googlers.com.com (243.199.238.35.bc.googleusercontent.com. [35.238.199.243]) by smtp.gmail.com with ESMTPSA id d7sm7566967ion.39.2021.04.19.15.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Apr 2021 15:50:59 -0700 (PDT) From: Dennis Zhou To: Tejun Heo , Christoph Lameter , Roman Gushchin Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dennis Zhou Subject: [PATCH 2/4] percpu: use pcpu_free_slot instead of pcpu_nr_slots - 1 Date: Mon, 19 Apr 2021 22:50:45 +0000 Message-Id: <20210419225047.3415425-3-dennis@kernel.org> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog In-Reply-To: <20210419225047.3415425-1-dennis@kernel.org> References: <20210419225047.3415425-1-dennis@kernel.org> MIME-Version: 1.0 X-Stat-Signature: aghqo9bp9pameeap56nnoy6gtwrdrqpi X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7DEC7A00038D Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-io1-f43.google.com; client-ip=209.85.166.43 X-HE-DKIM-Result: none/none X-HE-Tag: 1618872652-305880 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This prepares for adding a to_depopulate list and sidelined list after the free slot in the set of lists in pcpu_slot. Signed-off-by: Dennis Zhou Acked-by: Roman Gushchin --- mm/percpu.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 5edc7bd88133..d462222f4adc 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -135,6 +135,7 @@ static int pcpu_unit_size __ro_after_init; static int pcpu_nr_units __ro_after_init; static int pcpu_atom_size __ro_after_init; int pcpu_nr_slots __ro_after_init; +int pcpu_free_slot __ro_after_init; static size_t pcpu_chunk_struct_size __ro_after_init; /* cpus with the lowest and highest unit addresses */ @@ -237,7 +238,7 @@ static int __pcpu_size_to_slot(int size) static int pcpu_size_to_slot(int size) { if (size == pcpu_unit_size) - return pcpu_nr_slots - 1; + return pcpu_free_slot; return __pcpu_size_to_slot(size); } @@ -1806,7 +1807,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, goto fail; } - if (list_empty(&pcpu_slot[pcpu_nr_slots - 1])) { + if (list_empty(&pcpu_slot[pcpu_free_slot])) { chunk = pcpu_create_chunk(type, pcpu_gfp); if (!chunk) { err = "failed to allocate new chunk"; @@ -1958,7 +1959,7 @@ static void pcpu_balance_free(enum pcpu_chunk_type type) { LIST_HEAD(to_free); struct list_head *pcpu_slot = pcpu_chunk_list(type); - struct list_head *free_head = &pcpu_slot[pcpu_nr_slots - 1]; + struct list_head *free_head = &pcpu_slot[pcpu_free_slot]; struct pcpu_chunk *chunk, *next; /* @@ -2033,7 +2034,7 @@ static void pcpu_balance_populated(enum pcpu_chunk_type type) 0, PCPU_EMPTY_POP_PAGES_HIGH); } - for (slot = pcpu_size_to_slot(PAGE_SIZE); slot < pcpu_nr_slots; slot++) { + for (slot = pcpu_size_to_slot(PAGE_SIZE); slot <= pcpu_free_slot; slot++) { unsigned int nr_unpop = 0, rs, re; if (!nr_to_pop) @@ -2140,7 +2141,7 @@ void free_percpu(void __percpu *ptr) if (chunk->free_bytes == pcpu_unit_size) { struct pcpu_chunk *pos; - list_for_each_entry(pos, &pcpu_slot[pcpu_nr_slots - 1], list) + list_for_each_entry(pos, &pcpu_slot[pcpu_free_slot], list) if (pos != chunk) { need_balance = true; break; @@ -2562,7 +2563,8 @@ void __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai, * Allocate chunk slots. The additional last slot is for * empty chunks. */ - pcpu_nr_slots = __pcpu_size_to_slot(pcpu_unit_size) + 2; + pcpu_free_slot = __pcpu_size_to_slot(pcpu_unit_size) + 1; + pcpu_nr_slots = pcpu_free_slot + 1; pcpu_chunk_lists = memblock_alloc(pcpu_nr_slots * sizeof(pcpu_chunk_lists[0]) * PCPU_NR_CHUNK_TYPES,