From patchwork Tue Jun 20 07:47:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13285278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D43CBEB64D7 for ; Tue, 20 Jun 2023 07:48:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B8A98D0003; Tue, 20 Jun 2023 03:48:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63F008D0001; Tue, 20 Jun 2023 03:48:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B8E98D0003; Tue, 20 Jun 2023 03:48:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 397578D0001 for ; Tue, 20 Jun 2023 03:48:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E9FB81206B3 for ; Tue, 20 Jun 2023 07:48:01 +0000 (UTC) X-FDA: 80922347562.30.B3F7E28 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf11.hostedemail.com (Postfix) with ESMTP id 14C1C40003 for ; Tue, 20 Jun 2023 07:47:59 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FTDl5iE6; spf=pass (imf11.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687247280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VQ3G/mOw1YDakj6K/QDY3ISJ4AmLryMvPDR0lMuqBuw=; b=VzaE63j6S0EYcsYd3+AalrgT4NAaXQZvvsjgZINubyaro8QjnaoqTPZWBMecDUimrOM9Gi NM3VMy/PLPFJYr47spVWkvhi3DcIc3LAyc9u/4ugCYEqxkRffyPqkh+9xL1R/5Jxc8p3H5 taDGsi15QdHUFdij9pQrizXXDJocyPo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687247280; a=rsa-sha256; cv=none; b=b47RTTVyvxwMzj0BzoDJDYN33RRCJzaX69n6WnxF8dTyMaVcoQufROCVs4s6si1OVVl0T8 VVEGUJO6osc9xbvHE5cVFEgwHwqaFrhDCCCENOReLX/AaxxCLsLEhIaU2asNKXt8uJcjgU I9A9vLQiiFun+y7HHaLojJG6pH4UFf0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FTDl5iE6; spf=pass (imf11.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-5701e8f2b79so49618617b3.0 for ; Tue, 20 Jun 2023 00:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687247279; x=1689839279; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=VQ3G/mOw1YDakj6K/QDY3ISJ4AmLryMvPDR0lMuqBuw=; b=FTDl5iE634tOzkrqSCVEu9Nf5D7hLqctOZFWgKMkQ3RqUmNsUi/5jC5UBELVK+62a/ tppQbX64R9NP8cj1gGJ6SvM5CU9vfUE39QVPjAKeWzgQ/S5s5xCa2DCQNDmPIUw76uDU docHziNctEA/rKYRNqAAe9o2ycs/Al7k/GvMXYgsm/K5xYfYahKx2SHDAiqufCA//I5i fpqnLbMwekkTgKZJTegbsUNCWCDlJKVeWVzpAxM2KcHVartgtsYK5L6lD7NGrGn8gZ8V H4EcXMPG6hTq4PUDvh8MLdynIDaaJQky/EdiXCX827BTAn0GvFQ+WeIvsKvPBLKBi8Cj Dm0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687247279; x=1689839279; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VQ3G/mOw1YDakj6K/QDY3ISJ4AmLryMvPDR0lMuqBuw=; b=RgOzYAiKUI6OJKg+cmaDAG8FiqQVCIuqhcL+DuJCoFLhNoPi4XM4iOZEqodxBAu5xZ B9zi/HC9cGxv5pq4I0P58QYiDXSNxstdyMUrrBgJTbN3D0vOFgxqGFBqkZkU7j/kB0y+ CAay7X7Fb+HYJf/ShkELyOvHH6DxhDW+PYwQhJCcpIAnIBMS339yJkOKcGqKZt6tbQKw oAH5pzAS6gYwaeFdacCr2z6gbs/oEmDgGFskEi3ge6Ug/31+Vb7SxYsHQ5HClF8R1rok D3qhOeoSJGEFFolk0FcM6PBph9MyxamUWK6cVDm+u+Ql0ul0VomX4qSFvdCqE/BIVfnG w1tA== X-Gm-Message-State: AC+VfDwOEMNQ4Wc1tfljDvHHgyHRjbJhsyU4giX1OgFvDvSyr/ktyADs McQQY1cnLuNNcxzvMFptAKTw5A== X-Google-Smtp-Source: ACHHUZ4S0WdBV687zylMKJcs5BTJOEB/HsGWCswlmI7GBgVKK08qTW3e1TZ6XRwEWVZtJ7HVB5KlNQ== X-Received: by 2002:a0d:e64d:0:b0:569:74f3:f3e1 with SMTP id p74-20020a0de64d000000b0056974f3f3e1mr11385428ywe.0.1687247279055; Tue, 20 Jun 2023 00:47:59 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j127-20020a0df985000000b005612fc707bfsm364068ywf.120.2023.06.20.00.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Jun 2023 00:47:58 -0700 (PDT) Date: Tue, 20 Jun 2023 00:47:54 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Gerald Schaefer , Vasily Gorbik , Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David Sc. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , Vishal Moola , Vlastimil Babka , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 05/12] powerpc: add pte_free_defer() for pgtables sharing page In-Reply-To: <54cb04f-3762-987f-8294-91dafd8ebfb0@google.com> Message-ID: <5cd9f442-61da-4c3d-eca-b7f44d22aa5f@google.com> References: <54cb04f-3762-987f-8294-91dafd8ebfb0@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 14C1C40003 X-Rspam-User: X-Stat-Signature: j6igrq3a65oom3d4amg8hpm8uk1y1bcg X-Rspamd-Server: rspam03 X-HE-Tag: 1687247279-428892 X-HE-Meta: U2FsdGVkX1/EO83QnqajjNgc3zvyXh4gcV8JN6PsUuI7QcJhShqbTjGHl3SF1uBpwkwLgLfRlSIEzMMcQfKVZ33dUY4tPzB9r+q4FD2oBOhsgf/YZlq9arApMNkwyopSX2HjrBTTrD0TEYGJ/viU1c5wyiG6wEgFODLoaxsXTXWCvX+tVCYqNM+NHtf0yllQfC6aPIf0tM84m+NQ92On98YhxSKzCRyepiMojqK5gFfOS3XtnNvr20rLdcfdfwP50KwWC2k8eQLvx1Us//d7el7/z5a1Czsm1iuDNiM9tKm20kWqyxQaJxav35N8uf1C+hI2C+EkzMs+V9vHqcl5M4iNQTRx+Y8cZl6DL/l7d0HCU6AGxG0w74lXaSVSOdXzFXBvpfXcXSLLBXVbS7LsXDkiT84RNRO318zYNoydxye+jnAiqId+Aarc11HZRdNfP2uyw9S5GzoxctSrtNyk5FGtaqU6RFq/RTv3dgUVwYaT6KDeinA74oVRf/N7iS+F885aECPPT7oI91We8jN+zZCtvq7pBnj15CyBd/8Z0NKigdQn2VH+QHgbyh2dfNsNgULvmHqx6o5BJuIYSGrq5FMztXwxq1qf5JXKpEQJTTLm54ZbcyPPpPSR/YuVLdInWbDWtSQ3IfAN0lSXd5/avElfOcTC+t1xfkwM7RRWG7l/w9glCnFVBv9nb/VQ0kocNayZ2Q6VjXzwxXrm0erquofvo07xc7wisJlAZPG1wnDOO7UMITi5MZ+gWPbrVrqL966Tm0VQs3sw7ehd5I8Rx5ZuUNROw4YwF7TEiClVPxSwx+RNYmR3JvGzK72L/dZ7eO9FvglyRsQOQ+63sEftKWAvEI5W6tV9Z1IiKdC0J3LiZuHr6XXDRJHJc89FbyIpRya3nS4ZBPM/ZESjVj5q0CCRzMA9OoGyLxgjodYuMT95cRdThmNd39IIm77MILA+B6tqIfmuI2vj9fiyxCE 72yBVZnW CRTBwk+1zZFuYvg2I+5ssMgpbgAhOObKdfTWugf4AW6NXpn1VKyLe11jQrYcRB3v2jwIYadMU+XN2hwwN9RFOcWGjMdmGB57ayoo0Ks0tv2R8h1SsgAy0N+KhHvM1RC+iVw1ZDF5YwjZOmTN/MKHb87DmB7EYGrOpgWLt7KpWD4T024WWldf8acJWM6pQA0asd4bjwecllUBEb9UDKO9ZkpLhxg0M66iCeVY3KzIbeXATbewJiEK6PWHQm4fEk/kNZrhsf8B0hsM4W4o3jsMGFWHbHbFRApcV9tXewNc8AUqjKF3Jpk/9eL3MKC6Bx7EFVEB0bOxfKl2h2SPcMN+MUZth29VJgrx5JdnyUHplsnrYKCdiwnIGWrkjEET9qKlyWgEaDtLRjX1I1LufIXwqYHP7iDoCayGPCVe81R6x6TCZj+LMrIs2hWx97q8dyA0fFiPgW73bOBC+3NwT2dBmwjmj4j7saayNygPWvNtKnfy1MVEtr6wtKrsnwmc60hwMUReF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add powerpc-specific pte_free_defer(), to call pte_free() via call_rcu(). pte_free_defer() will be called inside khugepaged's retract_page_tables() loop, where allocating extra memory cannot be relied upon. This precedes the generic version to avoid build breakage from incompatible pgtable_t. This is awkward because the struct page contains only one rcu_head, but that page may be shared between PTE_FRAG_NR pagetables, each wanting to use the rcu_head at the same time: account concurrent deferrals with a heightened refcount, only the first making use of the rcu_head, but re-deferring if more deferrals arrived during its grace period. Signed-off-by: Hugh Dickins Signed-off-by: Hugh Dickins --- arch/powerpc/include/asm/pgalloc.h | 4 +++ arch/powerpc/mm/pgtable-frag.c | 51 ++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h index 3360cad78ace..3a971e2a8c73 100644 --- a/arch/powerpc/include/asm/pgalloc.h +++ b/arch/powerpc/include/asm/pgalloc.h @@ -45,6 +45,10 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) pte_fragment_free((unsigned long *)ptepage, 0); } +/* arch use pte_free_defer() implementation in arch/powerpc/mm/pgtable-frag.c */ +#define pte_free_defer pte_free_defer +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); + /* * Functions that deal with pagetables that could be at any level of * the table need to be passed an "index_size" so they know how to diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index 20652daa1d7e..e4f58c5fc2ac 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -120,3 +120,54 @@ void pte_fragment_free(unsigned long *table, int kernel) __free_page(page); } } + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define PTE_FREE_DEFERRED 0x10000 /* beyond any PTE_FRAG_NR */ + +static void pte_free_now(struct rcu_head *head) +{ + struct page *page; + int refcount; + + page = container_of(head, struct page, rcu_head); + refcount = atomic_sub_return(PTE_FREE_DEFERRED - 1, + &page->pt_frag_refcount); + if (refcount < PTE_FREE_DEFERRED) { + pte_fragment_free((unsigned long *)page_address(page), 0); + return; + } + /* + * One page may be shared between PTE_FRAG_NR pagetables. + * At least one more call to pte_free_defer() came in while we + * were already deferring, so the free must be deferred again; + * but just for one grace period, however many calls came in. + */ + while (refcount >= PTE_FREE_DEFERRED + PTE_FREE_DEFERRED) { + refcount = atomic_sub_return(PTE_FREE_DEFERRED, + &page->pt_frag_refcount); + } + /* Remove that refcount of 1 left for fragment freeing above */ + atomic_dec(&page->pt_frag_refcount); + call_rcu(&page->rcu_head, pte_free_now); +} + +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) +{ + struct page *page; + + page = virt_to_page(pgtable); + /* + * One page may be shared between PTE_FRAG_NR pagetables: only queue + * it once for freeing, but note whenever the free must be deferred. + * + * (This would be much simpler if the struct page had an rcu_head for + * each fragment, or if we could allocate a separate array for that.) + * + * Convert our refcount of 1 to a refcount of PTE_FREE_DEFERRED, and + * proceed to call_rcu() only when the rcu_head is not already in use. + */ + if (atomic_add_return(PTE_FREE_DEFERRED - 1, &page->pt_frag_refcount) < + PTE_FREE_DEFERRED + PTE_FREE_DEFERRED) + call_rcu(&page->rcu_head, pte_free_now); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */