From patchwork Wed Jul 7 13:00:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 12362547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,INCLUDES_PATCH, INCLUDES_PULL_REQUEST,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B81B7C07E95 for ; Wed, 7 Jul 2021 13:01:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 46FE5601FF for ; Wed, 7 Jul 2021 13:01:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 46FE5601FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0FA396B0036; Wed, 7 Jul 2021 09:01:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AB386B005D; Wed, 7 Jul 2021 09:01:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8C706B006C; Wed, 7 Jul 2021 09:01:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id C105B6B0036 for ; Wed, 7 Jul 2021 09:01:08 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1945E183E46F2 for ; Wed, 7 Jul 2021 13:01:08 +0000 (UTC) X-FDA: 78335802216.16.109B990 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) by imf15.hostedemail.com (Postfix) with ESMTP id C07CBD0000B6 for ; Wed, 7 Jul 2021 13:01:07 +0000 (UTC) Received: by mail-io1-f47.google.com with SMTP id v3so3211332ioq.9 for ; Wed, 07 Jul 2021 06:01:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition; bh=2kmKBpXwA1RSdHcldobA4OeqQNfsQiFqigDRPNTPVWI=; b=lwP3K2ONY7MAJyQatWDAzK3rbM8orVAVxKI20sg+qG03xoGpOcvV+zfzuyBJ+8zbnA QHaoDqfsSAciQIELtnEvTWWLoKG3xmfhuVRAlHdIRq4I0rU5UWvXCMBet4vkfTrN3GGW VJ8+gtgBRvkGfSQfpKm/PYH5QuUAXN7k0SYJWT+yHQh8KrmJ0tAe+0vQ1NFjyGH33j61 CXQqPaYjTQMToX57DkjBp4Ayq2Q/E8/YuTJcJxW2S/7YbI2HTfkxZzAKjEj3V5mqg8WJ ZRVY9XOWBlM9EDvnbHOkl7GoRGYfk+sowlHdGbRQZt9//2EqjWlSOZfPW4IxIi608NyE 7Rrw== X-Gm-Message-State: AOAM530Oswt+ZPSwD1e9XQ7Qi0SAcwqnCsioX9lEq272SC5mSa56AwwO ESezGVFkTS3tJDPLJjFgWXQ= X-Google-Smtp-Source: ABdhPJzxIK3zkvRIMZSpAYrwTyFFWkfTGRkQnBpOtJJrhD6wgtymb/nFS+COLfCq5hpGiD6nsKRu3g== X-Received: by 2002:a5e:9703:: with SMTP id w3mr19707384ioj.118.1625662840936; Wed, 07 Jul 2021 06:00:40 -0700 (PDT) Received: from google.com (243.199.238.35.bc.googleusercontent.com. [35.238.199.243]) by smtp.gmail.com with ESMTPSA id a1sm9653175ilj.54.2021.07.07.06.00.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Jul 2021 06:00:40 -0700 (PDT) Date: Wed, 7 Jul 2021 13:00:39 +0000 From: Dennis Zhou To: Linus Torvalds Cc: Tejun Heo , Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [GIT PULL] percpu fixes for v5.14-rc1 Message-ID: MIME-Version: 1.0 Content-Disposition: inline X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C07CBD0000B6 X-Stat-Signature: wzn6y7aa8ht3sobw87ndon3g45eyjqhj Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.166.47 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none) X-HE-Tag: 1625662867-167094 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Linus, This is just a single change to fix percpu depopulation. The code relied on depopulation code written specifically for the free path and relied on vmalloc to do the tlb flush lazily. As we're modifying the backing pages during the lifetime of a chunk, we need to also flush the tlb accordingly. Guenter Roeck reported this issue in [1] on mips. I believe we just happen to be lucky given the much larger chunk sizes on x86 and consequently less churning of this memory. [1] https://lore.kernel.org/lkml/20210702191140.GA3166599@roeck-us.net/ Thanks, Dennis The following changes since commit d6b63b5b7d7f363c6a54421533791e9849adf2e0: Merge tag 'sound-5.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound (2021-07-02 15:25:23 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/dennis/percpu.git for-5.14-fixes for you to fetch changes up to 93274f1dd6b0a615b299beddf99871fe81f91275: percpu: flush tlb in pcpu_reclaim_populated() (2021-07-04 18:30:17 +0000) ---------------------------------------------------------------- Dennis Zhou (1): percpu: flush tlb in pcpu_reclaim_populated() mm/percpu-km.c | 6 ++++++ mm/percpu-vm.c | 5 +++-- mm/percpu.c | 32 ++++++++++++++++++++++++++------ 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/mm/percpu-km.c b/mm/percpu-km.c index c9d529dc7651..fe31aa19db81 100644 --- a/mm/percpu-km.c +++ b/mm/percpu-km.c @@ -32,6 +32,12 @@ #include +static void pcpu_post_unmap_tlb_flush(struct pcpu_chunk *chunk, + int page_start, int page_end) +{ + /* nothing */ +} + static int pcpu_populate_chunk(struct pcpu_chunk *chunk, int page_start, int page_end, gfp_t gfp) { diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index ee5d89fcd66f..2054c9213c43 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -303,6 +303,9 @@ static int pcpu_populate_chunk(struct pcpu_chunk *chunk, * For each cpu, depopulate and unmap pages [@page_start,@page_end) * from @chunk. * + * Caller is required to call pcpu_post_unmap_tlb_flush() if not returning the + * region back to vmalloc() which will lazily flush the tlb. + * * CONTEXT: * pcpu_alloc_mutex. */ @@ -324,8 +327,6 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, pcpu_unmap_pages(chunk, pages, page_start, page_end); - /* no need to flush tlb, vmalloc will handle it lazily */ - pcpu_free_pages(chunk, pages, page_start, page_end); } diff --git a/mm/percpu.c b/mm/percpu.c index b4cebeca4c0c..7f2e0151c4e2 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1572,6 +1572,7 @@ static void pcpu_chunk_depopulated(struct pcpu_chunk *chunk, * * pcpu_populate_chunk - populate the specified range of a chunk * pcpu_depopulate_chunk - depopulate the specified range of a chunk + * pcpu_post_unmap_tlb_flush - flush tlb for the specified range of a chunk * pcpu_create_chunk - create a new chunk * pcpu_destroy_chunk - destroy a chunk, always preceded by full depop * pcpu_addr_to_page - translate address to physical address @@ -1581,6 +1582,8 @@ static int pcpu_populate_chunk(struct pcpu_chunk *chunk, int page_start, int page_end, gfp_t gfp); static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, int page_start, int page_end); +static void pcpu_post_unmap_tlb_flush(struct pcpu_chunk *chunk, + int page_start, int page_end); static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp); static void pcpu_destroy_chunk(struct pcpu_chunk *chunk); static struct page *pcpu_addr_to_page(void *addr); @@ -2137,11 +2140,12 @@ static void pcpu_reclaim_populated(void) { struct pcpu_chunk *chunk; struct pcpu_block_md *block; + int freed_page_start, freed_page_end; int i, end; + bool reintegrate; lockdep_assert_held(&pcpu_lock); -restart: /* * Once a chunk is isolated to the to_depopulate list, the chunk is no * longer discoverable to allocations whom may populate pages. The only @@ -2157,6 +2161,9 @@ static void pcpu_reclaim_populated(void) * Scan chunk's pages in the reverse order to keep populated * pages close to the beginning of the chunk. */ + freed_page_start = chunk->nr_pages; + freed_page_end = 0; + reintegrate = false; for (i = chunk->nr_pages - 1, end = -1; i >= 0; i--) { /* no more work to do */ if (chunk->nr_empty_pop_pages == 0) @@ -2164,8 +2171,8 @@ static void pcpu_reclaim_populated(void) /* reintegrate chunk to prevent atomic alloc failures */ if (pcpu_nr_empty_pop_pages < PCPU_EMPTY_POP_PAGES_HIGH) { - pcpu_reintegrate_chunk(chunk); - goto restart; + reintegrate = true; + goto end_chunk; } /* @@ -2194,16 +2201,29 @@ static void pcpu_reclaim_populated(void) spin_lock_irq(&pcpu_lock); pcpu_chunk_depopulated(chunk, i + 1, end + 1); + freed_page_start = min(freed_page_start, i + 1); + freed_page_end = max(freed_page_end, end + 1); /* reset the range and continue */ end = -1; } - if (chunk->free_bytes == pcpu_unit_size) +end_chunk: + /* batch tlb flush per chunk to amortize cost */ + if (freed_page_start < freed_page_end) { + spin_unlock_irq(&pcpu_lock); + pcpu_post_unmap_tlb_flush(chunk, + freed_page_start, + freed_page_end); + cond_resched(); + spin_lock_irq(&pcpu_lock); + } + + if (reintegrate || chunk->free_bytes == pcpu_unit_size) pcpu_reintegrate_chunk(chunk); else - list_move(&chunk->list, - &pcpu_chunk_lists[pcpu_sidelined_slot]); + list_move_tail(&chunk->list, + &pcpu_chunk_lists[pcpu_sidelined_slot]); } }