From patchwork Mon May 29 06:14:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8176C77B7A for ; Mon, 29 May 2023 06:15:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A22A900003; Mon, 29 May 2023 02:15:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72B4F900002; Mon, 29 May 2023 02:15:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57FA5900003; Mon, 29 May 2023 02:15:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 44B63900002 for ; Mon, 29 May 2023 02:15:04 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1AC471A01FB for ; Mon, 29 May 2023 06:15:04 +0000 (UTC) X-FDA: 80842279728.13.512A0D8 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by imf27.hostedemail.com (Postfix) with ESMTP id 4918C40002 for ; Mon, 29 May 2023 06:15:01 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="j75/ucEx"; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685340902; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2Xgtva00uaRjPQjQYWCbyYlUVerqtzlz4H3/JM5nRsA=; b=3/Q/WIMpILHLL4UPsFi+amyLzbfikCPadweRkcWTeyIqIlUewVKgMmeR8/TuS0bHFl4x+L +WVLQavSvZ2w0D6X2KL021ELI85RwonlgRRD66kEeBJU0ILFy5XQm7t3f/kkmu42sHx/FG hHrhWwVUh4DC0nqsDIjl2IAOb9GdqdY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="j75/ucEx"; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685340902; a=rsa-sha256; cv=none; b=jVwSrARR5eCI9gG534II8XQe4utsIBQqFCtSExuHje2nXg8H4R+l02ndMyGpecCWh4bvcy pAeeW2G3k2dQVSlksuD4nQWqRaBvd56p2aYolB2Z6UbcRTgaxyEl/hwzgRTqxCjxs79ips NV/DWE7DLjFnqv7GOl88Ci2Uml1j650= Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-5658875abfaso41767357b3.1 for ; Sun, 28 May 2023 23:15:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685340901; x=1687932901; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=2Xgtva00uaRjPQjQYWCbyYlUVerqtzlz4H3/JM5nRsA=; b=j75/ucEx6oo4OkFwpLoopPUGefT5zRQ/GQSssdYFJ4lLRni3rLEVhCMo1e8jwYMVcb YP6Bm+ySwzQyzGGQM2rSRov76G9UQ/Tn5RA33SDKqUYCIvPra4wwNlCuRu98iddHbdYN EF4ZzSOFNLEo4zXasuFIAULbNX6r1iMaUqjVeX4Zi3vuHqsR+CvgEyNVfPdXCJfUki2B 3dBz1c7QLHmuXQ+aR4qL+wJWgzXAMcn6OOeaRV4ZdMMOf1AGowm+XIRkGpuH6QAGrUcE pfw/89xaKUYoO8gOTy9XVNliAthq+Njc/D/eHcTCmEi6RvCXrpEvwwxwM3i0HwRhfzXq 7JqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685340901; x=1687932901; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2Xgtva00uaRjPQjQYWCbyYlUVerqtzlz4H3/JM5nRsA=; b=PM3mJTboVe5yPEm0GZyH4+BlvwWpKJEjKZAx/6v73s4eQrFO8K0ISvm42a8A1APa4x gNr0pn2sSZxxx2KKykXRZGml8ACP9zOksRmmNo6GE/LhsnGhGe+yEbOZFfb1RHmaNg2e 8UlEefU+dj3BzEajMdY8CEDuOkCdlnWMTVrYbe1G330xQqe4mHjFsS65wD7lEV5DMjSS WSw3izrFgbgZMaCaiCDaBSZnjUj8Vf+HJpmML9YC//JswcMQPb6vGNuvIR7ObRNTDtVj hPMevZxoxS6XCIRFZfudQL/GhBEQ46mRwcTZFaV2Gd1dlXbO9MuDw9Sj6CjfS2jbKwLy qOGw== X-Gm-Message-State: AC+VfDxRqVz1USJ2hS3wbwhTkq2t7K2iXdN4zcaFezA+6D1R+r1KRfSs FU3DaTQLvQUJxxjIfHYZ5a/clA== X-Google-Smtp-Source: ACHHUZ676xJPstkLOqdersuTwpjuuB4a5YN7mGCxCUJRDLUIc17B3O2mtdL/MxS8/JJgtz6rWj3hTw== X-Received: by 2002:a0d:e685:0:b0:55a:30f5:3d65 with SMTP id p127-20020a0de685000000b0055a30f53d65mr12605725ywe.41.1685340901250; Sun, 28 May 2023 23:15:01 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id z7-20020a81a247000000b00560c2e3ec63sm3404765ywg.77.2023.05.28.23.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:15:00 -0700 (PDT) Date: Sun, 28 May 2023 23:14:48 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 01/12] mm/pgtable: add rcu_read_lock() and rcu_read_unlock()s In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <88c445ae-552-5243-31a4-2674bac62d4d@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4918C40002 X-Rspam-User: X-Stat-Signature: s5uh9smxheyjzenqbh8g8wr9zu1emarx X-Rspamd-Server: rspam01 X-HE-Tag: 1685340901-786411 X-HE-Meta: U2FsdGVkX1+aVQ+3IqeJzHcYOrseJs8eLsiOkNPx8A0QQ2WoAKy5BM6oGJZBvQbnhjD3Il19BfmE9959Xi2XieR06G5SbWQJpt1dR5v3JriMvNGfYQb+TXE6O4UgIb1mCL/CiYefxiyktbzbG45pS4U60DFekh4NzN8UzJgV2lSWwba/g24riE9PVY8TdIRxOQRU1k0ETNcKQRdXyzNQzv6UdfwEwSjo/qUCvzyiPivzlFyd5SsDcgNR7XA0DCtWmBx8G+FMqtLLIt/RJv/bWHLvzr+3eVZdT4QEFb9kNlWtLF20nCss5X7X78M+V/5bv8ZI9AzS+COjdE5uhAJSfHnj1ac7od1LdKWNE1FCb3gr0h489Wc8sRw6RRO38fFgFb7C55qR7mw1YOkRHr8cey0pcOGyAaY7uZ7jdXCoOiefcm1ghmcGLjggFRgYM+8uC8LjY3KARTsd1E8NBOyHgFJAFpdhqp1jKYx0qjAgIPYwpOkN8g4HtBuVZabBtxIjW7U1dJN00ZFHdWRepDUPK5M/lOvTUs5QA4aJL+vX91fPU0fHl4HwJmkcFiATciCBh5GJXJ3mSkcdDRaDtXQzTPLh+AvoBN7PeOvasXLo8urspL/oSgTgVI8cTdAnhdo2LYRGmi0aE+87tqX+E2YOJNCVg2OEcZQxCct4NJsSz1Netqi4Xe/NXgetnuFJT77C+OQD3FxmmQRUhwoMLXyl24811GywV6PWEi+6o5mzzsJkCD2MrFsfsK27D7c3HVUQALJ6/4kVs0/2oXqUH8HJuslXNyPZCTrVztniQtQB26PcOxS9KslzpyHAvnTAdjU8oSyrNNuvV5JBgr7ppVkomHGzUc+L2CmUBqo3hln9W8Br2iW9l03ryxf900tejeIuvd45OZmwHJFzStq6nMI+Vi2wMYs0WELLzQK+ouExfE97THkvd6ChPdjwHDMsT5ZS8J74X4xTSoHzUEKcoKE Vl8aisi4 CLVJJCA2BN5ZESgph16htpKm2nwCruVtsYewxbPkURWN14XXQdNrae0cw3e5VWxAu8ZuKdo8BnBEQbQs60VJUAfD8PZFW6euIA8h9tnQURaIquThm/AX+/INCN5lRG058idFyQcHMNonbRj8zVUsdXaIWgczdYvXo1KGcEpA8ffftK7Icv+UKjrW5pnXtpUbQkOchGysTEEw0zlmy+Eh6csCtvTWnPfjNyLSOrSVwjmhgyZKSaAK1Svn1ZE+14PQlLPKMAVMMybgrLcC4HerOhgPqcVSfB7vytmcHysaJlR9oHoIWQuuqxzxMPKluyyOkMc9/DsiDdAssVTUjATs8/8uVZJo18hsL5lOJGtVVv4UnVzXf2cFHqQsH3GHBSM60tb3p/h/wu/VV2m4KXCitGtCfVLOLTQb/9NFE8D6FY4P2WPLXlV+yADwYyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before putting them to use (several commits later), add rcu_read_lock() to pte_offset_map(), and rcu_read_unlock() to pte_unmap(). Make this a separate commit, since it risks exposing imbalances: prior commits have fixed all the known imbalances, but we may find some have been missed. Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 4 ++-- mm/pgtable-generic.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a1326e61d7ee..8b0fc7fdc46f 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -99,7 +99,7 @@ static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) ((pte_t *)kmap_local_page(pmd_page(*(pmd))) + pte_index((address))) #define pte_unmap(pte) do { \ kunmap_local((pte)); \ - /* rcu_read_unlock() to be added later */ \ + rcu_read_unlock(); \ } while (0) #else static inline pte_t *__pte_map(pmd_t *pmd, unsigned long address) @@ -108,7 +108,7 @@ static inline pte_t *__pte_map(pmd_t *pmd, unsigned long address) } static inline void pte_unmap(pte_t *pte) { - /* rcu_read_unlock() to be added later */ + rcu_read_unlock(); } #endif diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index c7ab18a5fb77..674671835631 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -236,7 +236,7 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) { pmd_t pmdval; - /* rcu_read_lock() to be added later */ + rcu_read_lock(); pmdval = pmdp_get_lockless(pmd); if (pmdvalp) *pmdvalp = pmdval; @@ -250,7 +250,7 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) } return __pte_map(&pmdval, addr); nomap: - /* rcu_read_unlock() to be added later */ + rcu_read_unlock(); return NULL; } From patchwork Mon May 29 06:16:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9936C77B7A for ; Mon, 29 May 2023 06:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57F6F900003; Mon, 29 May 2023 02:16:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50848900002; Mon, 29 May 2023 02:16:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38207900003; Mon, 29 May 2023 02:16:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 27643900002 for ; Mon, 29 May 2023 02:16:24 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E1DBB1201FF for ; Mon, 29 May 2023 06:16:23 +0000 (UTC) X-FDA: 80842283046.23.183FA4A Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf30.hostedemail.com (Postfix) with ESMTP id 12A3880010 for ; Mon, 29 May 2023 06:16:21 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TyhMHkB8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685340982; a=rsa-sha256; cv=none; b=UIZFScnd5LKRgT72cW7CavusW4FwIufP9IjnzMuwLYsZKElA8SxZ/mKjKnZrFfBNSckxxB xSyvLk/IEvUpNJo+lA/bWk/779M/RbkUjhNWYZlbXn9P9cl1lgaq+k2liuNtjE0Cns5Ccn vZ1lz3fMAiuoANrv/cxAoInxkiVtimQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TyhMHkB8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685340982; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xPq2eB/hW7ShnFytnKcTY7aukisfg/jLLLPKQ4lP3rM=; b=IckPXs4wYseHHwlq822VbVI41WnHilUnB5H6uPFmB3WKycKjB+N2qUopT6AgC1c6BDsxRQ XMa57OOdgD3/t9b89SQNd8SA2SUUtqDFntbZ8wfcT4xebTbaiqv3/iVglvqKjBR0nt2Je2 Sa8QcEyVHRLgnNadulnZgm8vcaTgVs4= Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-ba8afcc82c0so5784317276.2 for ; Sun, 28 May 2023 23:16:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685340981; x=1687932981; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=xPq2eB/hW7ShnFytnKcTY7aukisfg/jLLLPKQ4lP3rM=; b=TyhMHkB8EAAdy+hUHa48C2vLWt0oIXv3HPWA7oGBtoKurezV9dmnaHp/Ug08+YXBMJ q7/doDqEaB13J8DWMDOsmxCRPRcN6V2SXOTsYe+dqJjrxA7f0LVq1fNiXr9IMn2dnMMI ESckYbJKgJgrbJ8T+uYkbBEglbSci8JLRkQ5xTjOVaaAsg2STz0pP4v2Ide+0MWDvyHe y1hUI8uDpAzlLTOdYFAa9G5upyPvxazDxgjqrZ/FkHni8ZrS3oyiM+Gr8t/oORl7+XNi bEMYjRBzoNKzLiTXOPBKXj7pF37UgxQaCI1yMdVBb74YQf5DcZFdirdn0N62WO3feJu6 qWEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685340981; x=1687932981; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xPq2eB/hW7ShnFytnKcTY7aukisfg/jLLLPKQ4lP3rM=; b=CrYk/nKBtSSOCSvbicVhay2q/zh9wuFdbjO8azBQ4aERa8KsK1zaVgeSXY+Uj1er4+ u9+Ulnc3zTmOMzVo0WAEbA2+LI/gDrowxT2Jak5FKXHK72xcOUuWdvaPBiQdrp1hnWOq XLvnj5gpTY57KmTSctbvndp034xd6A1vkQPO395m4yAG9WZxUYz2iVSKOWrF7vPbvo4s baxIV1BBNJv8N92CVcUKCEfLpq/SuJT4e+9OvMIG3MymaUNzA1GCimBzthrMMpNJ3+Hj zBWsC5ubedL2eJFxHhTlFSyU0vyU3/uSxN6yt7vDvByEfeBaYTyaEJnJzuxJ+tzR5GS1 L0KA== X-Gm-Message-State: AC+VfDyjfXQuA7yW+cmSsK4pDISnIRWX+h7K4WT+zeuxNQLmY7c5tmdU oSqtUcf3Kp1TltgaEE9hoTdZ5w== X-Google-Smtp-Source: ACHHUZ5ZacpUNHnx35XFkkfr+0IABaEA7Bz0HX1XL9bgqfNu7gBbTpGfMWgF9E5JOg8lgzwGl1APAA== X-Received: by 2002:a0d:f003:0:b0:565:a0c8:7e66 with SMTP id z3-20020a0df003000000b00565a0c87e66mr11630388ywe.0.1685340981021; Sun, 28 May 2023 23:16:21 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id n5-20020a819c45000000b00545a081847fsm3407533ywa.15.2023.05.28.23.16.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:16:20 -0700 (PDT) Date: Sun, 28 May 2023 23:16:16 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 02/12] mm/pgtable: add PAE safety to __pte_offset_map() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <923480d5-35ab-7cac-79d0-343d16e29318@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 12A3880010 X-Stat-Signature: i51xoy87tmqbpy6bzjk1cerihhpjg5y8 X-HE-Tag: 1685340981-2456 X-HE-Meta: U2FsdGVkX18vQ+cRIc7o8ivf6155b2NsLLzGKjHl3k/28z1bvz84lNuC9EoJ+CZ33vGM6PJEIDtjx5jSmndispyha8Irw8kSo2ISo0ox+JguOVxH9wQNHdigVf458QEUfTXjpztqgvsYcMMLZgFIeye1kLuX6J0tHPiEZMMEvNw5mOtmM3Rr1FSnmUZbdUJoScYCskgJJIzJUHoZYgHOJhlnyF73QjIHlt5b8NyMYvBxPemZhjidhKgogwBScowUMPsjV3xkEif0mdNOcwmgZP6Nq3YQMXdqKKqcgL+X0Pm23txBN3o1lzPH8uu3mf/GWNp1I/vE9eUPAVmdeFgAwkJb6CntGTV4W0mo+fsKNcoauu9i8J8RY2THByTbDOW/QXoSlqqWEIELUGxyo0WtosUWE8RaN24WK18lcToT2te+S+efLXRgVLo/yQ7LHiqOE3/w0ig3WcEolRc15zs1gEs4zaqkfdWWj3L2uv1O/PN9/r2OJB5BlFsnv9IxgsQ6CDM/t2sm8lCan1oYfNm8mHwHqJyKOvJbFIII1M4Sqz78ie2oo8OOVEBJHzB0W4na2UUcZOYC5ibhfJA9LWLOa5wZp//+1UyaUWPPxbkgrPr999745bPmemt3mHG9eM7nr0yeBJx4RyQyUjHfuI9xDpOz8J1h5oigElqcMo6wh5CXzLHziKRpb+0UHtOwRVAo9ZsIXol/i0Ny79koaqInpyVoiXzBrQA6mkcBi8EtMOsNxwzWL8Pkc2r4wjEyLZQ6D4pSWwLRa5vN0zPbwU8iYuX4a4YuyJOntGkggI/Q19uPPWfQd0IZoQAkNv8GiOtVbpBfJ0n6q5uPcZPXOcKXufSVo7bmN7ftmaJKqkMiHEY1zROdunH9+1VXQGRcT8q7RdRRt9bOxfyi7ySG7UhzZF5pjeJd9lxzsGLJ9GT4noY9+CEzGKJN4hnLK0LbcqieWNwZHXq9p1nfL25/I7O HWtJmCrr 0ABgkwnLfbw3gLj55LxTOydSdDikOFjldcZ4oSUiqkQwK4uom0fHsWtfuXBaYl2WAZojIidaxhcjx7qnQFvgi1o3JEoUMxSXzkg/OEyMCkrcgjLqJJXzv55jySfZ6ei9ZWjvbhDhApyBsw7ok9hbK9k0eBBu9Z6Gc6FQ2GxtKOHRsbQ+sY5xaFU3YXX0gFiVp77i+DSRUUPcG7VpP9BccfhqgwTWINaZGsfQRq0Eok8PkizcrAvuhys0bieSVUfqvtpslpAztWgIB7/aMdTbhNEENE1Tg9yb0roFpD9QvUeD6BIarTuNByZMI36g+3ZmHiVtoQo45m0tqr98uQs3c9Nxr8MSdnc571TQylSwSu1pffQHes1IfElYUiqMTy6Aqd577zv82evbm4jZP/8WCobvGv1nlHx51fdC28AnLlZ55X09xc0Im90anVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a faint risk that __pte_offset_map(), on a 32-bit architecture with a 64-bit pmd_t e.g. x86-32 with CONFIG_X86_PAE=y, would succeed on a pmdval assembled from a pmd_low and a pmd_high which never belonged together: their combination not pointing to a page table at all, perhaps not even a valid pfn. pmdp_get_lockless() is not enough to prevent that. Guard against that (on such configs) by local_irq_save() blocking TLB flush between present updates, as linux/pgtable.h suggests. It's only needed around the pmdp_get_lockless() in __pte_offset_map(): a race when __pte_offset_map_lock() repeats the pmdp_get_lockless() after getting the lock, would just send it back to __pte_offset_map() again. CONFIG_GUP_GET_PXX_LOW_HIGH is enabled when required by mips, sh and x86. It is not enabled by arm-32 CONFIG_ARM_LPAE: my understanding is that Will Deacon's 2020 enhancements to READ_ONCE() are sufficient for arm. It is not enabled by arc, but its pmd_t is 32-bit even when pte_t 64-bit. Limit the IRQ disablement to CONFIG_HIGHPTE? Perhaps, but would need a little more work, to retry if pmd_low good for page table, but pmd_high non-zero from THP (and that might be making x86-specific assumptions). Signed-off-by: Hugh Dickins --- mm/pgtable-generic.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 674671835631..d28b63386cef 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -232,12 +232,32 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, #endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#if defined(CONFIG_GUP_GET_PXX_LOW_HIGH) && \ + (defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RCU)) +/* + * See the comment above ptep_get_lockless() in include/linux/pgtable.h: + * the barriers in pmdp_get_lockless() cannot guarantee that the value in + * pmd_high actually belongs with the value in pmd_low; but holding interrupts + * off blocks the TLB flush between present updates, which guarantees that a + * successful __pte_offset_map() points to a page from matched halves. + */ +#define config_might_irq_save(flags) local_irq_save(flags) +#define config_might_irq_restore(flags) local_irq_restore(flags) +#else +#define config_might_irq_save(flags) +#define config_might_irq_restore(flags) +#endif + pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) { + unsigned long __maybe_unused flags; pmd_t pmdval; rcu_read_lock(); + config_might_irq_save(flags); pmdval = pmdp_get_lockless(pmd); + config_might_irq_restore(flags); + if (pmdvalp) *pmdvalp = pmdval; if (unlikely(pmd_none(pmdval) || is_pmd_migration_entry(pmdval))) From patchwork Mon May 29 06:17:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98CFDC77B7A for ; Mon, 29 May 2023 06:17:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15F7F900003; Mon, 29 May 2023 02:17:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 11024900002; Mon, 29 May 2023 02:17:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1A34900003; Mon, 29 May 2023 02:17:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E1393900002 for ; Mon, 29 May 2023 02:17:32 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B59BB80218 for ; Mon, 29 May 2023 06:17:32 +0000 (UTC) X-FDA: 80842285944.14.3BFB4E2 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by imf02.hostedemail.com (Postfix) with ESMTP id E180C80003 for ; Mon, 29 May 2023 06:17:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PnX8u4a7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341050; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QJ3N9RNUACQk0rp/xTQEOUK2bEy9LceuxJNDbuu7Hdo=; b=3uE9iN86j7Ji83ZolFP7+hQ3kld+MoVTC9h97FDQyLWZP8ZN8m+9tTj9cAqINPttRSiT7m Ln5Py34vWp8l3+cgtWtGLm5/9rSjJUPIGK04HzXgS8PubJUdUN2cO/RLRI3aFANuC254xS 8oLm4lQ99vMM0NjPNzqz2tUG6BIdpIU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PnX8u4a7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341050; a=rsa-sha256; cv=none; b=w2cd43WdBKLm2UP2/te2LpgUTyb/ZYg3aeG9BUcxHJIiHy1Q3hiGIXeRUX4mGLno5tgJTz IlpezpG7l2cTdqz7PJpTZOWnwfGOKavzY7xQngxPKt6u8JBg+oVeDEJDDYvG8YmlVEBKYK s2csrGZa6uykhuqIzIXrJ76ndb9h0JY= Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-ba829e17aacso4343552276.0 for ; Sun, 28 May 2023 23:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341050; x=1687933050; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=QJ3N9RNUACQk0rp/xTQEOUK2bEy9LceuxJNDbuu7Hdo=; b=PnX8u4a7nFzrI9AMI1PfJ2JIEBsc/6HUoh5hn/jeCJld2cHVyPh5B3mpoBHYP4HlDt huCzuvd10XbNys3q964U8F3Pwm5I1086lfw4K2KXsSdCwoHho0iYP2E/vuoTg9TJPXAw 9Ao+OBlyjIf+ii9nemOKDnm34wjGWTWJPqiOtaDLYpgSbKG3NeKVJfh7S3pXb1pULdQN cjURL7J2sqlVPM4/ao36PpQzMPvierIr1dk2aTiiL2s04dhzXCN3987Ah17CrCgicXI7 uLzuDFZr6ZrjJPk0lh0dr69aYCwSbUbiqg2JiJfrZxUW7xsSohaWtB46bIclcU6whmCj +nDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341050; x=1687933050; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QJ3N9RNUACQk0rp/xTQEOUK2bEy9LceuxJNDbuu7Hdo=; b=GpB+yGFbnl/+ZsXzZAwu0YLzsFV4TPpi6kZugihB2GPALXTqXoojs57of1mx+QAgLw U0Xurupqz3biokWTnWdYVrJGKXgDeLF9AgtsPGR9CweMdGHQ0+Pis3RJKctdPwUYgSZP MghH5UPjF0h2LcavcjevoveWvvBjLGPtKWej+5cDKmnlD7n2K9bnuSmyN+ozcZb6JKFa SI0eFTgCteo/jEnaUtawLFAgCLuRpa46lggKYG6unWvcoQqrgV8+yDAD/wyPmDGG86k7 y3IH20OEnKXK/cO7CxPDK9HlN/4c8yOOzDFeqgHzExEK97M3QyScv/lBfVu0WyCzO/Gk XiMA== X-Gm-Message-State: AC+VfDxqyCtw0HdLp8pMDwYve/GryZvZnq/YgRvIWVhA+/Zv0+uWVwur yjuRlSrSMHV2IaXOkH4uelWZyA== X-Google-Smtp-Source: ACHHUZ6g2ncsiViAbUymnL1kKwlhN1cywz3Q5/8UINRdk6/z7ELyukBfTa2iPWzHzMx63rqo3lZAZA== X-Received: by 2002:a25:fc19:0:b0:ba1:e06b:bc57 with SMTP id v25-20020a25fc19000000b00ba1e06bbc57mr9201429ybd.64.1685341049679; Sun, 28 May 2023 23:17:29 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t62-20020a814641000000b0054f8b201c70sm3381110ywa.108.2023.05.28.23.17.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:17:29 -0700 (PDT) Date: Sun, 28 May 2023 23:17:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 03/12] arm: adjust_pte() use pte_offset_map_nolock() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <94c2ebe1-6b23-1cee-4aae-22cb835776ff@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E180C80003 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: h4nbkfocmcc9a78fijmhau9kokhb871r X-HE-Tag: 1685341050-563270 X-HE-Meta: U2FsdGVkX18ek8qOgE0l5SZRZGvxuyDsqjEsBnPT2CA+MdcqsLEcxZUgd65J0L9HZPYdIOGw2O4FWAixVddmq9R5BMh51w59cY2a4tqCXjNZGUzdx/1QhjDXkuaBgqjmRfu9cDl4ZrOsQ5MH5H0i7NXb8Xky3oumBgvv/KXQl+5jP/YJQUxHuY6FqWfTKBRglAx3wjmI0XWmKU3ynxhXu1kFK/UCPyE80EyDf4SZFnuA1+Bq8TDVXT4YT+RngY/namRxWQW/IuQeNOnuFJp465DAhKqRlMu4PuSt9FJEE4HlC0FKeUspMG6VciftXXo6oK7nzEOHsj64vQ1/f3TLYPEgDmRvQ4aKN/Bk4vHU22oo6wr/yXwPmVatyj4ibe2VuuH6oplY1xCOJEXw16wKB0mbpd886W6/FkAMoy/So59WRNKXxIloAdQyru/BARvhUC1+zjHBryVm1MeXkXHC50GoL/b9aBnfTbi6U2XXwnj83ld/FBJY8b5MzLeDlKr3g8FSvcE3gkC4oBijq/626ZM0nzgdBEleKZoSgkMOtLwwa/ta8odzHe8YEU8Nrim6Li05OIvslD5b0+/CBhrKubmm4aMVJ6cuQnxxrvOM8vTBJhKduS82MqTFGATTkr34F0nAVdozPWNbXkVYoTjzbxSZMqqIRk1nv9IPXIN8wGJh5CpujkVo+AIFMEVe5zdyn7Msrs6yE9TpSwEf+2OCYkOp03fNZN7uP3NbN2kgPnOsyIXq0cAYq10ZdJPb8Wx+yHpw9RIitKrGc1KzXtOxpLx42SL1d7t0daaseZb/f+GzKWvsh1fJgmwJyis436MH6xeRAz8aGJtDxAWAFtmVx2om+BhmoFcghXHFJMHGBnsH5jnvUdtDZ5hb3dCAyIgPQBcRPSxNwAypsihUtpQTzgvOxyLYOOz2FYKnCJwr3bZGlcvrI18WHgretrvU6eHqalWprX9eAsK2xa9yIqq HUoIQhQP dZpeifgP7VlJN6x6TwjnsvWKcbTTun/TbQH2g77k8rSY7TTiO5naXNu/tyfRIFh8CdtjAQT02MkdJZWZ/+V+pWivMVyQyTmqP8p1JGNwfFddgyL2ITA9Na/BBR+Gbcq9JpnEeVbQ9RqghLNHYX04lHXqYN00jI2F3pKOFJCAlDJqeOnxgrSmMW9x7kw0+IKeGoMuKj/O/una2uHMO80MvbJI5RLYBXPNrU8WQB9u1rGYcl0c790SJKuCpRz7IlUyyXsqio4kjbf+52tDcAn+asn/Wd37QkShlfx/SYOLTRdjijuCN66lvH5VSoTLQ3hXqQ4url+HDbPxTifUwOorHtfXQ4iElof3Tu1Y+fBSqu6RetnuAYoR8mZD9rVOv8wm3nHbH+/2D55XwC/yWWKt9pxwn6yC3Ncl7bbhthN2gAu6Uc2uC1La3rCcAwQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of pte_lockptr(), use the recently added pte_offset_map_nolock() in adjust_pte(): because it gives the not-locked ptl for precisely that pte, which the caller can then safely lock; whereas pte_lockptr() is not so tightly coupled, because it dereferences the pmd pointer again. Signed-off-by: Hugh Dickins --- arch/arm/mm/fault-armv.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index ca5302b0b7ee..7cb125497976 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -117,11 +117,10 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map(pmd, address); + pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); if (!pte) return 0; - ptl = pte_lockptr(vma->vm_mm, pmd); do_pte_lock(ptl); ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Mon May 29 06:18:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 926C6C77B7A for ; Mon, 29 May 2023 06:18:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 297CB900003; Mon, 29 May 2023 02:18:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26EB3900002; Mon, 29 May 2023 02:18:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10FF1900003; Mon, 29 May 2023 02:18:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F2A86900002 for ; Mon, 29 May 2023 02:18:50 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BE84F1201FB for ; Mon, 29 May 2023 06:18:50 +0000 (UTC) X-FDA: 80842289220.26.F5AD194 Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 0691810000A for ; Mon, 29 May 2023 06:18:48 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1iA4YfG+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.219.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7tbEFbFk5pzLjwVra+37SCseEW8dJexk2Np2YDtZH+c=; b=x6pvB8vEwME37/dKf0c+n3UH+vzpEw7PkAji8EDG15fl4/dcC2VjjZXpI1bc4nt2U5xsC7 DNsQYYM+uSiAOUJnHXUTGUW+SNg8AwGkfeepTR3bjsQbuYpze5I8yRERYO7cyMCwsdoWxq jIp/9q3dRhEgqPP7+ZvZ7PWjFw10tRU= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1iA4YfG+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.219.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341129; a=rsa-sha256; cv=none; b=MvlmDgoL1JgPcszLWrDpoQbNIJ00SIGYAT1OLuUkE105l65ZKN/jOwZH8snYSAAq0EIwXA VUaExSVxi3Tr8FzhjQTNNq+dOjI3+q5VhPQ6kWJ8zTSfX2wY1XBVFJtaTbtJWqeuleUArD tmhRMzGsZfw1LWkX6h9fN5kc5X430tI= Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-bacbc7a2998so4349823276.3 for ; Sun, 28 May 2023 23:18:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341128; x=1687933128; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7tbEFbFk5pzLjwVra+37SCseEW8dJexk2Np2YDtZH+c=; b=1iA4YfG+l9uCrVK/VfGes1VI6E6dNElNmfH0Ym6MNMAOfLc3iDqikpG9Fh1kd0GZyN S2vDJSr0c2W4FoH5DTwBlYn9D4ysTNQfO/tuIIGYNqiQ41PKjRspMrJoqD8nN+1AW+Ll ZG2b3ePp7JtiTJ6Eed3Cy8FX00RyB15yPBCU8A7XIPIx+RIeQ7MpGS9HzfE/9OdJpx+S pAKYTbnfjluypjHVFpFATeSkHhnhYcVPl6qM+K8dkTFH6eXbpsYJ5LutgoPOrieOlSV3 oZuduBLXP+Ip6U8GW3EdevUfmC5f9cH9uaENaWd8adKsUbZ/byUTAk7/38rOHk1lVCDe TGTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341128; x=1687933128; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7tbEFbFk5pzLjwVra+37SCseEW8dJexk2Np2YDtZH+c=; b=E3PdroqpWyUhWD0/3tp3fh4Afx3Jd5PhKshpSWxsZHoXf+7MHtFBlDbui4/peR4sUo x2DkvauEg5hB3EN7qgyq8yGBTytHcvEh5IcMk/kt7lrBooGxTWNipEQnKu4bYIDXEBJx uxQU3+8oO+ToY6ziX8TpM+6JiLqxyb/HJ/+ncjIJbB3FPdfX2W7Bz0gGBg9KVxE73sT+ aSBiu6zDprzvcW7ueP4SxccuwAx3fKVIE9k43L0x27TCCh3o4QGaYZtZ7OrsQnlL4aJL rBlntL2o5sQzwTAqZIELvbuWsv4DrH1pTdrQv8o7S9pQ/6RcGKFA72jHlH1wrIB4WXC6 xv6A== X-Gm-Message-State: AC+VfDxjQyK/PgOrRn/n8MZAQx6StZSkRHiSu8UWPhguwlVIMrJHrFWN R/sLIsCktMKmj4Enp35+NsZ2ww== X-Google-Smtp-Source: ACHHUZ5wa1HnlGmpMM6V73fepd9odIlgE0K/gNhr4bUAPSedVjgsksTyRpOUN3YO0dkzYJPD3+qKCw== X-Received: by 2002:a81:6b09:0:b0:561:c147:1d46 with SMTP id g9-20020a816b09000000b00561c1471d46mr12682711ywc.9.1685341127900; Sun, 28 May 2023 23:18:47 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m19-20020a819e13000000b00560c648ef1esm3382356ywj.72.2023.05.28.23.18.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:18:47 -0700 (PDT) Date: Sun, 28 May 2023 23:18:43 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 04/12] powerpc: assert_pte_locked() use pte_offset_map_nolock() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: ecbmo3md7o198rq9tmr4gqczbgt39gck X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0691810000A X-HE-Tag: 1685341128-186715 X-HE-Meta: U2FsdGVkX18PfBPYCcmxeX4CIxl04FMg516qNqa1jGQ91ox5dDqs/yiyPvPa0xs1EQ1egQxQvi5K7s1HycVGzYtZPJn/gGNlIXEJQQtQi4gmfPVMPSsXrxWD0XPTTMRBFy9oQ0q5d1uTZbdXmgvNxYOvGolzDnDhvgDP18suUWwAL9v0w311f+qL49EmdScHj3lQJ4pUbnep+HrdD/KFWmDQ+NoNSTAEn1gKAqWGnbEfqikHH5JdyoLvHUPebySefBRyyGWyw3dnNfw3R8YyU8UvD8ObFtjFjGN1t/JTK7Xbk/1C9J2KCXdM8RsxlbKB6vDeD+kiae8NvHUK6YBTv+VrJhcwi5VeMFeN4dG0D/YO6qG8hED2U4mclDzZdYLDfnNHRFjRo8vvShP4PRZfKuD0Ez5PMhTlTp0o0EGRDy2Al8TKvDOk/b1r7NefCb08kofM8e5Tq5jeZSzignyHlRa3PyMlDsD5C0naoD6xlXcYXzflvIScXYn1fYx797iDAUSonHCIGtAcbJTusxa4Ef3EH3qaCP7cl95vc0sJfzekmOU9ej1QPfqnsBYBvWDyDuyNlOgnOiMjwjtVrMwHagKrd/9okCb2Q7Q/ZO/QHfp2G4K7IzJKH+3rbzq+7LxOuqbMF+tcziWgZ2aqemhvcxT7zGxrsZBrPdtD8/q6oCD41ncPX86ycXIsZhnM4Itd8vzYM42ZaB+/47SRIvOQ3mBStc9tclF9LV22GagMXIeKUe7bZ7ZHYTCFFKRZckDqmiMqsY0/lDlxSNrhvA9sELdwyQ/P5scPuM/2D2ZeAgJEPFync7JoPp1Ow6Y6Y+rdn8B2MghVg72uZjkxyd/cHb37TnsPVHLVVUdJvQJ4+uuhWPDOZ6w2nDoePWFStb7z5zrSDDWNzcNOSCvfZS4TRvrRlPF8wiWskt8LnDwQGP+lpjSEQ6ziE+4f7qkf0B+RwSp3dX/xhRnjFo1pYit XptD8E9B CwnyBC/d9336e1aX3371/oIyVtL/CC8iVtRe3WLgaEARURiXzoVNA1wUzqpYQD2tpUX2iwXkwAPRRe6UFthRpK2/fSdPyTIdmRW3Opx7petxRhp2Jjh3EK+yD+eTGVc2jn/r+cxYnGpI6ZnMSe6YSl44Y4Yaw9FWqbnmG5MvhS+PpGrEQgEFBTkWPv5yX3Vq8HV4dP83OwubCztmdfHYZv/YM0XXnPQD2Iwsf94iqHZZyFCEBdbopWENDGyzFuoWQrLLO7aFFNCDu65+xb9t9DKY94HpS/5ABZo9NoEWBoQ3ZjNMk5hgvHdJF6vRB+zHyi7m6LTM+WgqCG//J60I94lnAdD5oU8tDGsY5FTrkJ+daUlTDE3pAh6GukxVxUgZ0eR7VB/mvyH4SbTYlSofHU5OX/4LJ8g2lUSch5dtXtVDl9s/aGzRSyA+QZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of pte_lockptr(), use the recently added pte_offset_map_nolock() in assert_pte_locked(). BUG if pte_offset_map_nolock() fails: this is stricter than the previous implementation, which skipped when pmd_none() (with a comment on khugepaged collapse transitions): but wouldn't we want to know, if an assert_pte_locked() caller can be racing such transitions? This mod might cause new crashes: which either expose my ignorance, or indicate issues to be fixed, or limit the usage of assert_pte_locked(). Signed-off-by: Hugh Dickins --- arch/powerpc/mm/pgtable.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index cb2dcdb18f8e..16b061af86d7 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -311,6 +311,8 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) p4d_t *p4d; pud_t *pud; pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; if (mm == &init_mm) return; @@ -321,16 +323,10 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) pud = pud_offset(p4d, addr); BUG_ON(pud_none(*pud)); pmd = pmd_offset(pud, addr); - /* - * khugepaged to collapse normal pages to hugepage, first set - * pmd to none to force page fault/gup to take mmap_lock. After - * pmd is set to none, we do a pte_clear which does this assertion - * so if we find pmd none, return. - */ - if (pmd_none(*pmd)) - return; - BUG_ON(!pmd_present(*pmd)); - assert_spin_locked(pte_lockptr(mm, pmd)); + pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + BUG_ON(!pte); + assert_spin_locked(ptl); + pte_unmap(pte); } #endif /* CONFIG_DEBUG_VM */ From patchwork Mon May 29 06:20:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10096C77B7A for ; Mon, 29 May 2023 06:20:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B1A4900003; Mon, 29 May 2023 02:20:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 861D7900002; Mon, 29 May 2023 02:20:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 729F4900003; Mon, 29 May 2023 02:20:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 645C6900002 for ; Mon, 29 May 2023 02:20:29 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 33BEB1401F5 for ; Mon, 29 May 2023 06:20:29 +0000 (UTC) X-FDA: 80842293378.28.5069AAD Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf22.hostedemail.com (Postfix) with ESMTP id 68E55C0011 for ; Mon, 29 May 2023 06:20:27 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lDd5t1DN; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=osELeF1qrpmk7tMBwVJZ0+Hn0dD614B/zeb//sVH2O8=; b=QF2S5BA4vCv0RnxQPGeEBl0ut0V6LyNQyPQeyJKwa0LOkw6tHHaI+9PODOwfx/urUJYOTc Jsnp1NgMzOOKq0DoHzViCEPfqMJh9jYu0/KSFWhEGIdWzz+vpECvia76neaDDg/7b5Zw6I KDAZzmKJQiJ7M4f7rpWi1TLEpCmNd1Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341227; a=rsa-sha256; cv=none; b=luCtoC3CGjOL/GAe9vepHVKpdG6C8Xen4iqaQYe2dgMWRuODChn52GCYCdHQorkF8QrrRL 4PW23R2lDQLlweSjvOM9V60fds90QqO3Yvl1n7dh26m5yMyuDPfgpFnFXXl5IaW4KMH20W HBRn94sKHUkFNo09WKHLqQgHn6+3DhA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lDd5t1DN; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-bad041bf313so4417900276.0 for ; Sun, 28 May 2023 23:20:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341226; x=1687933226; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=osELeF1qrpmk7tMBwVJZ0+Hn0dD614B/zeb//sVH2O8=; b=lDd5t1DNHYrERHggJZt1tFe/2MuojYDB8DYuX4saUcSsWZjN9TJvB4D3jOc40ZJzuw KXTyU64u/loqgfjTny6vyEEHc7I3ap+5KBFfNEbc+et5E0P2BRIz5MO59vg+RyUV5HGF LQIvE8HoWeX7dfHrfHrThINSr2Jkgw1dNwKUT5ElKyxmqFpo9JxBWX275kcmdKdX3BCO TKmGDZP/FC8QUdgVXd4u26ydd2mkiG9rdEZxkd6dJlJez+Ye+8eFZqq1OHv/Cc+ig/ka 6L20gyN+7z7wp1ikvxHZmZH1RAvGKKb0++xvnFOn6a1yNDE6E2enDzRJirlL0XfSo/Zh 2JMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341226; x=1687933226; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=osELeF1qrpmk7tMBwVJZ0+Hn0dD614B/zeb//sVH2O8=; b=HqtB/hF8FREYCqqpC1frSeC2Qgpa6Bdiq8miY9wbaiE1VGRhQm19Tl+1gtFfePt6fH 6K/p0R0VDnBtEJIZwCKE2nSVIazpZjVRBLZdtA1iMFuP1lQoFYA69/W+yLC8Hekq3HcR 6jT10uObmNGsHW+fg6bHyjNKFnlaZX5hL+c8mtOIzupFaXsukf1aGdb8cekAGBHywu9y BnfdbwVftLCC4/GfpwIjsdMTrYUPwxpWsUP+xKSsNzo0qke65+veJFuO8Wd8HYG4co8w uwiWlSHqGtOW1imJqXroozQt6rBo+6c8pftfjrUAManTSlHtS1sugCxm1nllmYjIDufA 66Fg== X-Gm-Message-State: AC+VfDxyGm9hjf5qCZ4ha4wWjv/TxQxaUDKpiHsRVZrWLVWbYgib4vmh QNYJQ5Ar+NZHbgMTdVI6584kpA== X-Google-Smtp-Source: ACHHUZ5aMde8BUNH3d96u3IOrLaqOFrYAlrmuVUlWt4ugn0e7ueCBkueldmdmxLB7UIM6n20FuWD7w== X-Received: by 2002:a81:8304:0:b0:565:c888:1d09 with SMTP id t4-20020a818304000000b00565c8881d09mr8471629ywf.30.1685341226458; Sun, 28 May 2023 23:20:26 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y184-20020a0dd6c1000000b00565e57e6662sm1530559ywd.55.2023.05.28.23.20.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:20:25 -0700 (PDT) Date: Sun, 28 May 2023 23:20:21 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 05/12] powerpc: add pte_free_defer() for pgtables sharing page In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <28eb289f-ea2c-8eb9-63bb-9f7d7b9ccc11@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 68E55C0011 X-Rspam-User: X-Stat-Signature: 9y5knhq4ac5zbcyixz6hyg1fy9h5t8ce X-Rspamd-Server: rspam03 X-HE-Tag: 1685341227-160184 X-HE-Meta: U2FsdGVkX1+pHMU23/0/PyWUEe8NAs9dPEbvHqH1acwpphGUsL/MhkvoJAan0ttF5ZL0TTnZPvyCU0Ky5Nn7Lna/fhmr+UApLD+DQa05F9pIx6V7j+1jLT3VKbZs9u3Y6HsGI61oWniRkC3pUnq/SCIito6Hm0dS7Bv9cd+NEha/C+S+oViBbDBqLcY2ygqJNdRyoWGMhsutRnJoFvlxYrwvS6s+LTv7mxFVerIOffOm4NGY91TslN+RrAHReRO7uXI+GYxozh+aatX3gRVKZG3uo51KmAFwuAUn4zqabBjipnEOep3zivDP+WfMBi/nXW361/qLWCzE7kn0eIuKmE8Vgn0b87doPuX40Ip7eIewaIJvopygHtBWu9S2NWX7TZA1QFp4to/njZLrEp4Hp/IvTtAH7MINtJqMZISzBAyonLhCoBw3nH1tJ14z/g3gfBnQsmZ7lq5p/yKlVVXjf5LpLigGM2KMh8nS1crvXUFFbizqxgA8u7Jm83d2ZeQRD8G/TJ8ie4Y6oMxt9Sco3LI6l0eOWlBJSX8Abh4NqSC8MnAMut1gQ7peG0k5ICGE4yFoGv8+ybDk6IJF9gA8Ab5SoTURSmegWhDy/HAVpvj4yEOTMKxDfXh0aVCx/2bpMxZ5eNmeeEFMcBBdXbd96BdV0St/bGZZQnhh5ThRBlFSixru1wWoPJCHDHMh/ysONFMeNU+NoJU7w6+VYQfcz42WwMp1yl4I2PCH17qjst/vUDrCCiF4gXDioLHc4kuW7PrgDGbYrIMfW0EzhiZcKOulzCkGf7hWlHB5Ejq1wlFfPv3tI1O8ob6TPx2odphftld2+qhCd/ngHt8ue9hcxp+FPtbt6FyjLzhHQnrBZW+2eBrEJRf0HDYuii2/+BPrJbj2nwcy7Er9YOVQ6sNlMfL1uk9r0+6zozisGlq/u1iDIPHbIVHtqed25riK/N0Yvaab7bkLaS8zocB4fZS 5jepfUIV rK6j3Lt/XbaWjOkHJ0Eb+LxtbqxvN9Pm8JyJFOdZF5grtyVtwPM1fwbXqtGIfLr7xm3kKmxUf/BlPtJqozZuO6zTaVkXKXnw5ukCuOzuo2uwqEmIdmcyBCUfOcMjQy8ytZdDI6bXZ3dZBMDvkWWITsr5U8uMQRx/WyY0ppxEJR7mEJAVLs9SmpVUHDGQvIna3fnG4p/0MxLBn4fW0An+euGv1N+A3CXnu4fXeT8FYAyk/gDRFCWwQQ0gCBqvGWUyVF1xLx+qvZVEQQ5lTC0KDdcIxh56uHaBu6RzCf8t3Df+RGTEIXDyc1WcoNTeU5wLNZ8wmnE+B7C2vI/xLRk9z1SnMpA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add powerpc-specific pte_free_defer(), to call pte_free() via call_rcu(). pte_free_defer() will be called inside khugepaged's retract_page_tables() loop, where allocating extra memory cannot be relied upon. This precedes the generic version to avoid build breakage from incompatible pgtable_t. Signed-off-by: Hugh Dickins --- arch/powerpc/include/asm/pgalloc.h | 4 ++++ arch/powerpc/mm/pgtable-frag.c | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h index 3360cad78ace..3a971e2a8c73 100644 --- a/arch/powerpc/include/asm/pgalloc.h +++ b/arch/powerpc/include/asm/pgalloc.h @@ -45,6 +45,10 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) pte_fragment_free((unsigned long *)ptepage, 0); } +/* arch use pte_free_defer() implementation in arch/powerpc/mm/pgtable-frag.c */ +#define pte_free_defer pte_free_defer +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); + /* * Functions that deal with pagetables that could be at any level of * the table need to be passed an "index_size" so they know how to diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index 20652daa1d7e..3a3dac77faf2 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -120,3 +120,21 @@ void pte_fragment_free(unsigned long *table, int kernel) __free_page(page); } } + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void pte_free_now(struct rcu_head *head) +{ + struct page *page; + + page = container_of(head, struct page, rcu_head); + pte_fragment_free((unsigned long *)page_to_virt(page), 0); +} + +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) +{ + struct page *page; + + page = virt_to_page(pgtable); + call_rcu(&page->rcu_head, pte_free_now); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Mon May 29 06:21:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA9FDC7EE23 for ; Mon, 29 May 2023 06:21:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52EF8900004; Mon, 29 May 2023 02:21:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DF32900003; Mon, 29 May 2023 02:21:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37FCA900004; Mon, 29 May 2023 02:21:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 295BB900003 for ; Mon, 29 May 2023 02:21:36 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E67521A0203 for ; Mon, 29 May 2023 06:21:35 +0000 (UTC) X-FDA: 80842296150.27.62DD18F Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf20.hostedemail.com (Postfix) with ESMTP id 09A891C0014 for ; Mon, 29 May 2023 06:21:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=rWUgEbcR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341294; a=rsa-sha256; cv=none; b=zB9iWj7cd9W3hFhHsDr5D+2npGpWx80/1ugEPMRdxSnF6QGSInpqq9rrEAE8BNTfm5NuT+ pWzFg1WMWeW4vH2XLefUx9e6ik62SVRYWm0Wb7YfETID3NiU5O43OU1eteMAr/9H1/GeKv TtWFlD14AU0hw8mq7rbftb1aLVeJD3Q= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=rWUgEbcR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341294; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kjV6xBD8+Z39YuH4hgrLfpItULNEjB3if/M/mkAbsVQ=; b=219MyfCJI/ZA/JuqSpqLoHs7YhJLnURrgrY4AN3XDZmoM7TVFGpsWI4Y0tgzOgmzsmj2o6 6sRq8IxW2U137928BVV1CowL/DrWTNiT8cbdJLX/8JTSWdwcH0W2Uo2B/2qfRDa964S8x3 gdpedrqDAf9wyMUzzPK1hNgZcAwePOg= Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-ba86ec8047bso4309361276.3 for ; Sun, 28 May 2023 23:21:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341293; x=1687933293; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=kjV6xBD8+Z39YuH4hgrLfpItULNEjB3if/M/mkAbsVQ=; b=rWUgEbcRxN+0lh7aV7n4UOuWoW/akkZOM/gqoHjVcxn/6hheszcuPJOLVJ4sguWyP8 nkQlUMJmy6e/Jx9YVzvAF7u4UBrtQ7Gt8z6ETj5pN1DYiBGqn6irZMUv2C9TbfCBC78I xaCU0qKIPP/XwpIBhRQPgPdjxQJW6d4VJskC/kyJTtslu2euOqK8ik9oIKBYFLT0OWAQ hthSfePHY1Fj36lFOt87PpfGm9V25DdJkUjrx5h6k5n/1x83JounXfxF7b0VnDNmT3pg /B3jbAdtHb1fjyAf9CqPVGMMMvFfo7sNqLyn5ELpZamilbHqPMk5v4bK33lqW9qB8lmS adrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341293; x=1687933293; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kjV6xBD8+Z39YuH4hgrLfpItULNEjB3if/M/mkAbsVQ=; b=kon6uXZu04DUIrkW88DJfu3fXyO0L2uarPfzPrdJcDGoUJeN5+momGZ/N8Q4NI0iai T0wplzEezpCDNyNTvWiQVYgWv3K4oQiZsGEp1UYRIkDeoSOkNxYiTtIbWmy5nTc+joyI 8o13XZ4TEBsfTziqyZVRVIf/JnLaU3wOqZNT4/A3XFRHefSnXAgKjJtNwKQ0/5Sk5336 FgkTYTKQKTumXCPDztW1vlvpYhdAyN0PeP+ulhx2j6TG+KQXBwsrepuoEyiguSvTF2X1 rwr24/hgMqJzZiGBH4ID46anSp668F9iLtadE2BHX4+2wFxjXRSbXpU4c64vxqoKk/od RsYw== X-Gm-Message-State: AC+VfDxf5Ynxe5tHWBftuYj4GdHCqc3giKGbY5WRa0toYObHd2BzJBRE OTWIckdUbHbejokw1pBrK+8saQ== X-Google-Smtp-Source: ACHHUZ6EPD4pm5LL+EWBEb/p2RZtAUzDMPbUfE7F4FXgHJQ8XYwzEv8l0puT9pEGX6+B6Mzd+dzTzw== X-Received: by 2002:a25:d796:0:b0:ba8:3e89:bd69 with SMTP id o144-20020a25d796000000b00ba83e89bd69mr9536233ybg.12.1685341292952; Sun, 28 May 2023 23:21:32 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m205-20020a2571d6000000b00ba87bc06fe5sm2712527ybc.52.2023.05.28.23.21.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:21:32 -0700 (PDT) Date: Sun, 28 May 2023 23:21:27 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 06/12] sparc: add pte_free_defer() for pgtables sharing page In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 09A891C0014 X-Stat-Signature: 3qta5wkwhqu981a4q6t3jqj8xx6oboow X-HE-Tag: 1685341293-530967 X-HE-Meta: U2FsdGVkX1+vUYzWVH0od5e4Es17p0Ktq8iCQxD/ZiFsIbt9eZkbZonY8u4ed31ZptDq8xn9jem0+gDEM2E8LhqX6O2GIrDTlbCSMKxLkIFYH3L/iDy3WrRpWA/6krfMKsiKdIVFXsRPm76L8sB3LWRqy0qh2BVKEtVCVpCegwBz0VNQAVRRF+oJ+qdZZLpccIdtJHmgXoWx+LkDa3LbeylP35JOT/+6/12vO0ZS3q3ahFMOBDeNbNbTupZ6EU9hyugH+F3tWL3Wf67dJIpLJjYhHJMIK/N/CNESN8XvEhq6DJScptyXJ7W2AHF5pJXBtWrjw0yBqJHdbzJlXIuA693i6tAw2I7+3qZlMZDv3pGvuR7wbtiCD4H8Sfb7KhhsSnQK8K/osK2Ux6aI6KdIoHA5r8NQPzUVGLDwPo4TqOzOpe8pqA9Wqu/0hLawHKJjiU9x3OdCGv1/s8aIcWlDWmkFNqjs4ajDZWD+nujs3C5Q3KsPwoNKM1GJ7liAuDcTOA7dsP6G5VTvypdKSiAefpd18d7/X8AMkTUT99EfpbXRZGORuJjl3zgMLcnAQRIn7X45R3njnhbAUSi7XnVcvcFqgFnB3Tyl79D1Ahcwy/BSKunQThs9RNGdFIAQoYPQ+hqOxMA/526J3dulJVuS7xg13KwiP0lEYwxDPmMgYf/RP3ct7R5IP7uPjr4uG6jB3ABxzkbBCEUbEmAPCspenDgpVUmNlpUwouxhtobX3zKz5vyttdgMijdzqLWbtsTnO40a03cJGWoMHSIHIWvBNlE2ye8a52s6Ib5iHz8LmpFFwAKiYZZmNx/Jd+1IoO3ZDmAmmx1wr6KCYxqhf57fsxG01jQkcKZSr8149mPrLROZKwWJa+LQ4nSZvzocMQznN9Hb6D8KNkOCAqGmHRjMyKzXINcdWjxuJJqR8DrnJJML0+LLOfMhM6/Eiws4is/cmCQy4HjUJZYv9fa3AZg qh3SeFdw HY56h/Q01hrjt+LSRdcTfPqWkgxjlDhFMX3qR59zsz/QH904k/8SonuUrrM0iTKvss34rKQFdNENHHbPWKpjCPT/Ct/2jqKzv3JQr6+qYnuLlMrBT/2/xsyCcRwC4QXmSxLHGQ7yQ4aTZRCci2WiNNTCmnzcnEqQvQEuc74FdHuW9xgyASnxM5uyCAN4K4C6jYPSuB6h8QMSEseTliYdJ0V/H6p1HRNPfAwhunMdBL6zi4GjayHzPoHAt8UQ1cYraASrE+LkdfA57yc5LzwgaFZuQ0f4+ZJx4VqE6G8H3SFyOk6FJjWR4vTW94FThttM+fSZZggDw9ZGKkLCpRLMVvfcpk4rj1f4XAvYfiQ073LW/W/7FhUSh7jsuPAsgEk1VilnyY9Gmhb7cYpCcKo2T33FeZ9PxDyUqZ3k4HPGG2rV4vL8Ks+JSf2aMUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add sparc-specific pte_free_defer(), to call pte_free() via call_rcu(). pte_free_defer() will be called inside khugepaged's retract_page_tables() loop, where allocating extra memory cannot be relied upon. This precedes the generic version to avoid build breakage from incompatible pgtable_t. Signed-off-by: Hugh Dickins --- arch/sparc/include/asm/pgalloc_64.h | 4 ++++ arch/sparc/mm/init_64.c | 16 ++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/pgalloc_64.h index 7b5561d17ab1..caa7632be4c2 100644 --- a/arch/sparc/include/asm/pgalloc_64.h +++ b/arch/sparc/include/asm/pgalloc_64.h @@ -65,6 +65,10 @@ pgtable_t pte_alloc_one(struct mm_struct *mm); void pte_free_kernel(struct mm_struct *mm, pte_t *pte); void pte_free(struct mm_struct *mm, pgtable_t ptepage); +/* arch use pte_free_defer() implementation in arch/sparc/mm/init_64.c */ +#define pte_free_defer pte_free_defer +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); + #define pmd_populate_kernel(MM, PMD, PTE) pmd_set(MM, PMD, PTE) #define pmd_populate(MM, PMD, PTE) pmd_set(MM, PMD, PTE) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 04f9db0c3111..b7c6aa085ef6 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2930,6 +2930,22 @@ void pgtable_free(void *table, bool is_page) } #ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void pte_free_now(struct rcu_head *head) +{ + struct page *page; + + page = container_of(head, struct page, rcu_head); + __pte_free((pgtable_t)page_to_virt(page)); +} + +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) +{ + struct page *page; + + page = virt_to_page(pgtable); + call_rcu(&page->rcu_head, pte_free_now); +} + void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { From patchwork Mon May 29 06:22:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFE9BC77B7A for ; Mon, 29 May 2023 06:22:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C5AD900003; Mon, 29 May 2023 02:22:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77651900002; Mon, 29 May 2023 02:22:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63E93900003; Mon, 29 May 2023 02:22:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 54EE7900002 for ; Mon, 29 May 2023 02:22:49 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 371CD1601ED for ; Mon, 29 May 2023 06:22:49 +0000 (UTC) X-FDA: 80842299258.11.128794E Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 0F262A0014 for ; Mon, 29 May 2023 06:22:46 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=JnCv5o3g; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MHjeymamXVEDQ3Kw25JCka7yLmkzZOMAvMVAEydeLzM=; b=0/60yBa5ztcBLJC69KiDS03bY5RDLQ6/o+9kXSPizU6upWz7ftI0q2wuLhOr47W3tMtLzD KUpVGiVvfWEKfc/22bbD7a4DN08Fdlt8TrvtaJNSj2sI5Cxb2MPfoDqdGW4+rqRBIDjMdC 1oon9P99kTPillvto+GIia5VsKAYnz4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=JnCv5o3g; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341367; a=rsa-sha256; cv=none; b=3jzM0M7BFdE2HjNMi3FBV7tgI7JUMBSleVm9NlqUxMmpz16LlfTjzBeZBm+7JTb2ufvj6r 4klx7i6+tnVFdgeCd/jfXJ3eysvaYWlDGksaUhg+ZbjdkCppgDXKCGY/eU4u5qnFXr+6/y egRfCCn5DCZANgq2LKnFp4S8VSvdIoY= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-565a63087e9so40117937b3.2 for ; Sun, 28 May 2023 23:22:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341366; x=1687933366; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=MHjeymamXVEDQ3Kw25JCka7yLmkzZOMAvMVAEydeLzM=; b=JnCv5o3gZCSjIaRpRDj860LJkDOANHqriyLGSv5pPF/RAyg9m3lQdupCXaTyei8DMf e3KqZBCpZIzZeMjk/yaxPctH6c49veiWRbUeGgx17gSNxkwyxqN1QFATHf6q6YutmR3t HvT+zDNBDGXXzqUl/LmcNCNapCFXfgc8yyTJIwdBnMkvFfp9LdRJqcRYSLqq8ynJ3LiF LXxpx3gFYAZnedKKq/SPuYQMj/8ol6u4/3db6BOFo/ME2oJub0w8Z8IEMye4507Zx0Ui smMsg0FV9ON+iRDLGBbSQlzdjRNF2P0/vV32Ee1vXvhsjJhpzkmQMG6hoZRiP5zjDNGY IC9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341366; x=1687933366; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MHjeymamXVEDQ3Kw25JCka7yLmkzZOMAvMVAEydeLzM=; b=fE9VlUXjcvipnA3b8XYBpV/j5kYIuXtGoRgWvyO2slFYWMoY1qLW0ljt+JC7MNduvu jSzQ8yspbADY0swPKgckTmiPUFKIjdKRFhw0iHcbOkrcgZa7MhBo4+vko8nKoZUFuvQm keRuj5CeJL/8F/GveXvXeLqKi69dSGcJow7A2MvrvA1GDsjjRc1BIdg06zV07MJkAK5j bjrq5sdFvWiK3PBecRymulqyN5yGfPDHRlB0lX7YH2bCsH7b4NAYjA5NZN1Q5IYIJRpJ DxKXrE+h53Tg4xjFyusv8AbOV/wjK9ExQphMag1eY9nZqe/lmM4N5ottaMGmmM6j8oZD H+Qg== X-Gm-Message-State: AC+VfDyv9li8PXEuc2Wm1vBaApvjTxa7a3nsQLLS5ugbpJPQZ0rY5WMN kTzJMsG4BbkvHKJTJyuZ0IxVpA== X-Google-Smtp-Source: ACHHUZ4SIMI4er/AF0deXgjHIDsOF79qc5oA7wlkFnZQpWYe4q0dmwzIjbUF/Ql+fhbfUc3dtFgx4A== X-Received: by 2002:a81:a043:0:b0:560:beeb:6fc1 with SMTP id x64-20020a81a043000000b00560beeb6fc1mr13114394ywg.16.1685341365991; Sun, 28 May 2023 23:22:45 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t66-20020a818345000000b00568938ca41bsm405426ywf.53.2023.05.28.23.22.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:22:45 -0700 (PDT) Date: Sun, 28 May 2023 23:22:40 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 07/12] s390: add pte_free_defer(), with use of mmdrop_async() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <6dd63b39-e71f-2e8b-7e0-83e02f3bcb39@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0F262A0014 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zqbknmtu3c7ouc4doy1rrrexjshferkr X-HE-Tag: 1685341366-453062 X-HE-Meta: U2FsdGVkX1/tGWicXTNAkRwTjkcLShLEFChzoEAkEV6hYRJdqSZupdU9T+oSnN6UxEJb1aLXXVI8l3ZJ3DHndJc3FUQZomZgfnatBvqr8Z05ug2r2AVRjyRhVNwSKsQ3/4mcfRvkPU1g8UbYKEze3ETT9WgTUcwXkAXotQA7qAjsfZm2e6vtnIskrN8rnn6gOIrWgC3/0WVYWALrinR2wjHOZnD5QqNqAWVbRBCGvC4V9UcgIkYz2cV+O96gTYYZIR9c6Jf/aFRDkvxKxvNPXAzcryf25NUwmJxgqlCUQrJ0N7lfyKkEN1fls0017FXLnZgrQveepbdB/fYtGGB+6qGD7qgFT7AJ3maWyfRofGzesmT4DMDm9CqMOGvuM1KgOUy1qH0VK8WpvhZckPrhAzloGhxD4n4y0QzjyPT8zIMMLf7eIvxilBj0G8IL6H2A2RFcuUFu/sn9tQ/TwOKDay4JF7cwMyHeZjdCxV34x2huQ/bAKYHn8GddM7ys3Jb5FZF9g5+Bf4zJtndMMuyM9ZX1qtmC9QG3SCUtuKFzS5zAarYtzxKgk0wnkT2y8g9ycuVQh3a4ADIjSPD8JUVlofKr3eYJQ0bUi1DrxCVC57E33aqdMR+12o10y+10foxda9m5d61Nq42vfHyUVhznpFtvQyma3TWk7SKpRQJ2fzfWpJ51ytK+C7kMlr4zo7bF9kjqf+HkTk3aJJoA2QDTEc/VJCHmlSyUYgimGaPFqsj4l327k8bZzW3/2O7sHRalrd9HQrpExp33AtPsLcy+pGvy8gPbScU8X8TwVfBP/9TUe4jPL9a+A/dSR3QlFDYo05MbX4A/nLHwTwUOdovdG9eaJR0lSiQlm8wUKXfHW7WLN59kvO+V2P34hA7Q5BTshsNrvLmYkorODxjnGPC2IZDxvYA85BZXnK4yAm3teT7y8AWmN/C32wyURctYny3Ab//ICVUpvaEct0dNwwe LtwILAoZ Uk9htXNuXngr4nrXYsXC7AopLJClrwf6diDg4DaAcYqCObCZptgSwpn1mygLkDR7U1na0jCyrbCFZBK5/yox4nsH0CIz2aFfpZAxRaVMytbXgXPwJooLymrFUXadZk7vY2xisZrFlLiiIy2j2JGywURmAvgxmIbcfTLILKEl5JvsZpOzrl3BViGHjPmgDvQJh35JcVoVJtLCJfxsE0Ovq2zE9TCwvDWGEUW+E4drfqfHbQCyePlZlNOyAnd7/Lkg8fhk4VHF0ywF3z7uCfTwpFABR6RpyYsM2B35cdSQyPI1hNNGK3TjMfxwxIASw6bVUg7PiceDw6SoraP4fjid9rjGK909JDT7mvb0uBihE2Hs9sT84nzNHC6opDDswrNUIrbwVsj28xRkLB2sQ2Acp8v2DOpbw7Dd5HzekaXjspRHTBU2xFfbqVfgQPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add s390-specific pte_free_defer(), to call pte_free() via call_rcu(). pte_free_defer() will be called inside khugepaged's retract_page_tables() loop, where allocating extra memory cannot be relied upon. This precedes the generic version to avoid build breakage from incompatible pgtable_t. This version is more complicated than others: because page_table_free() needs to know which fragment is being freed, and which mm to link it to. page_table_free()'s fragment handling is clever, but I could too easily break it: what's done here in pte_free_defer() and pte_free_now() might be better integrated with page_table_free()'s cleverness, but not by me! By the time that page_table_free() gets called via RCU, it's conceivable that mm would already have been freed: so mmgrab() in pte_free_defer() and mmdrop() in pte_free_now(). No, that is not a good context to call mmdrop() from, so make mmdrop_async() public and use that. Signed-off-by: Hugh Dickins Reviewed-by: Gerald Schaefer --- arch/s390/include/asm/pgalloc.h | 4 ++++ arch/s390/mm/pgalloc.c | 34 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 2 +- include/linux/sched/mm.h | 1 + kernel/fork.c | 2 +- 5 files changed, 41 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 17eb618f1348..89a9d5ef94f8 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -143,6 +143,10 @@ static inline void pmd_populate(struct mm_struct *mm, #define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte) #define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte) +/* arch use pte_free_defer() implementation in arch/s390/mm/pgalloc.c */ +#define pte_free_defer pte_free_defer +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); + void vmem_map_init(void); void *vmem_crst_alloc(unsigned long val); pte_t *vmem_pte_alloc(void); diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 66ab68db9842..0129de9addfd 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -346,6 +346,40 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) __free_page(page); } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void pte_free_now(struct rcu_head *head) +{ + struct page *page; + unsigned long mm_bit; + struct mm_struct *mm; + unsigned long *table; + + page = container_of(head, struct page, rcu_head); + table = (unsigned long *)page_to_virt(page); + mm_bit = (unsigned long)page->pt_mm; + /* 4K page has only two 2K fragments, but alignment allows eight */ + mm = (struct mm_struct *)(mm_bit & ~7); + table += PTRS_PER_PTE * (mm_bit & 7); + page_table_free(mm, table); + mmdrop_async(mm); +} + +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) +{ + struct page *page; + unsigned long mm_bit; + + mmgrab(mm); + page = virt_to_page(pgtable); + /* Which 2K page table fragment of a 4K page? */ + mm_bit = ((unsigned long)pgtable & ~PAGE_MASK) / + (PTRS_PER_PTE * sizeof(pte_t)); + mm_bit += (unsigned long)mm; + page->pt_mm = (struct mm_struct *)mm_bit; + call_rcu(&page->rcu_head, pte_free_now); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + void page_table_free_rcu(struct mmu_gather *tlb, unsigned long *table, unsigned long vmaddr) { diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..1667a1bdb8a8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -146,7 +146,7 @@ struct page { pgtable_t pmd_huge_pte; /* protected by page->ptl */ unsigned long _pt_pad_2; /* mapping */ union { - struct mm_struct *pt_mm; /* x86 pgds only */ + struct mm_struct *pt_mm; /* x86 pgd, s390 */ atomic_t pt_frag_refcount; /* powerpc */ }; #if ALLOC_SPLIT_PTLOCKS diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 8d89c8c4fac1..a9043d1a0d55 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -41,6 +41,7 @@ static inline void smp_mb__after_mmgrab(void) smp_mb__after_atomic(); } +extern void mmdrop_async(struct mm_struct *mm); extern void __mmdrop(struct mm_struct *mm); static inline void mmdrop(struct mm_struct *mm) diff --git a/kernel/fork.c b/kernel/fork.c index ed4e01daccaa..fa4486b65c56 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -942,7 +942,7 @@ static void mmdrop_async_fn(struct work_struct *work) __mmdrop(mm); } -static void mmdrop_async(struct mm_struct *mm) +void mmdrop_async(struct mm_struct *mm) { if (unlikely(atomic_dec_and_test(&mm->mm_count))) { INIT_WORK(&mm->async_put_work, mmdrop_async_fn); From patchwork Mon May 29 06:23:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 076A5C7EE23 for ; Mon, 29 May 2023 06:23:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94C9C900003; Mon, 29 May 2023 02:23:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D60F900002; Mon, 29 May 2023 02:23:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74FC4900003; Mon, 29 May 2023 02:23:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5D78B900002 for ; Mon, 29 May 2023 02:23:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 25E4E8021D for ; Mon, 29 May 2023 06:23:54 +0000 (UTC) X-FDA: 80842301988.09.9476F75 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by imf21.hostedemail.com (Postfix) with ESMTP id 4DBC71C000D for ; Mon, 29 May 2023 06:23:52 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=mUjO4pWK; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341432; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2gSRrpVaV+v6oTVvWHOeguB0KnrQ/fwH26LJ25rfvoA=; b=coechnjWCEFWjdqwrEbXOnpCwzwjLKyjqf+0C+xPEPMs1+bG7OJHc+3HgFlnBJqcsan4QS cNm5wnqmHDGsjW1dyDuAbIK6gB3afvMtW90yVVO6iN/4REOWJm/fuRFkhFNCf04CQrwktu dyvUlTaRsb/3Y+Z+POuV9t2RDGLYBMs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341432; a=rsa-sha256; cv=none; b=clsWe9eEN3K3NmAhSUBDAbCyQFGGE/PfU6GhuAA8dPq2sqU7Ojx5GWIc7B2LUcq2bWpJ80 35TD1S2w3LKTnJpdelXJOXB98LK0szovY9M1PVJGePT3TUUZjfgvuM6rvK/0Bt9a1Xoo5J lgsxI2uvWBiM+SOTdbDG/l9EsbumIw8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=mUjO4pWK; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-ba81f71dfefso4296191276.0 for ; Sun, 28 May 2023 23:23:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341431; x=1687933431; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=2gSRrpVaV+v6oTVvWHOeguB0KnrQ/fwH26LJ25rfvoA=; b=mUjO4pWKJgDDKZpKrtcPe8FWu0b2JY0+TgvW1MX4OtSul/CzSbQKQ3pqCcMlWFJNGp gWOyHbCB9BXc3wVa0KzB77Yil0KZvLHMQCnsIsQVZ9fFzAnqckLLbO2LnWVC3QgXUA2m AKj7k1CX2prs/vdJ8Dux8TrJisS6znNjC1Vtrv/PsdyCPjLHWieMNw6R9XprNyEWjRCZ 4j0MNun5Zv7iQwiFWh+o2sHynbvf1kUMSyoEE1HC9cxss3LWR7d6E9drnyGo/GZyrbJL X1vYzNNY1HAUAQnzj74UQMsBrtflmgqlc89omMQkyN/nWA1df9Rcpj56XseLcz9CDb7K rRHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341431; x=1687933431; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2gSRrpVaV+v6oTVvWHOeguB0KnrQ/fwH26LJ25rfvoA=; b=iXjWaukU4g5c6fG3R5LandFcTElcJlYek4BPKHaDehu+LurAjSSS01KUkEdCE6pBot xx16QsSmUutr9LhIGsEMDFv7V9MaU4yuc77Vqos/j6PHkXqT6qnRr+r7bSiZM7iPOCg3 DMgPenRmtNOOHfEo967egy3g2sm1rSrdcvTMGNCFSZLCbl+zyHkwCYCscQFgcVT+O6p+ QoCAgyfZ7W5KwVIfu5GUH/kIzlGH7dKbZvJle7oa3vKFLjTs07IrNW+VmPGCT+RfBFzA BpsNre1sAepRJvLv578gbNAQdVQ4DgcMNaae/R4q/1Yoent9DPPq2unsQHF+yLDN4Dvp f/gA== X-Gm-Message-State: AC+VfDwqwqPcG8fXZef4OiG9ZLwiiOqKvvMFApcRcO+nY6ud4kGuIEEg lqpMdrRfgG0G6dDBxzc+fM2JdQ== X-Google-Smtp-Source: ACHHUZ5CcwvoGgzDzBbk+Lv5ieVtVZp8uUiYjI1VJBrm/lel9oSDffzWyg8Wn1mT8OaCxHwZBaPMhA== X-Received: by 2002:a25:aae2:0:b0:bab:fdb3:7b56 with SMTP id t89-20020a25aae2000000b00babfdb37b56mr13795477ybi.24.1685341431250; Sun, 28 May 2023 23:23:51 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j73-20020a25d24c000000b00bb064767a4esm503449ybg.38.2023.05.28.23.23.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:23:50 -0700 (PDT) Date: Sun, 28 May 2023 23:23:47 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 08/12] mm/pgtable: add pte_free_defer() for pgtable as page In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <739964d-c535-4db4-90ec-2166285b4d47@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DBC71C000D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: o51a55inf8rrkw5wcc3antaddipau3r5 X-HE-Tag: 1685341432-604033 X-HE-Meta: U2FsdGVkX19ZCcQrsfde98kBn4NYQNRfLNHCAD9leGlxJStdk6qEZuT6wTWSFJcN1nVhvkmY3a7S1uApekJjheh2ZmSDtzLL/RdaEpS1iI7HitglpFDmuZPskOjn7Po+zyP7p1QsCVN6ZJ+t2J1tzxUTkEsO6OTPZrKS2cODxs78EDBfAFHi3tSeCzHWZa7z5aNPazdPm5QAgYjUTAd5nx65X71/bV5J+J+m9Bh6oG6p6oR1S9YpC5yeqUgeLXQMxeMxyKrgcC4TD3W8TyWIE944+JOdPKBRCvOuC85yAGU7LnjaNtb1iSzDnJ6sz1AOcSaMEGHqZOpTIC8/V7HYpadmHojnfkLKfqnC1+pJuHO0kLoQswPGIipq1/gj4Iow0n4+QVW6ZAEcqz6HM3U9+tznnZHrF18ipg7dz4XE7XleVApcNyq/26ZMAxH/dQ2BVZ7SeOW2XWVP6EwJsfTPw8DkpCO9Lzf2xFQ1OphhKSPMPQ0G61DbCopLp1XtW00gCQZjz9Z4UZ9IgeuohBhcGEMA0TSnpe83cbNROpNjYS/iMKrDWz5xME05lLndBq62o0Be++eYAMmqC/puL6EiSk6+2V8RAMbLbGPt52FPukNsgwZKJPGly3zv++sV/b7uyiZPe3rPOG1gHFSVwsmVHafc4O7oBrxmE8h/0+zjpEHe7iEu04E6CmcmTuBcNOJdZ1Mgs/Om1OIssVwDwWXlDdSJx5lWQB+di2SnkGJ+sr5Q45OjGS8NFFXYqO5Ir0eVie7lafm7eg9wsEAnyCPJ2XW5OQGxrg8AOECOZkNHRYVVP4E4Hvw8mDvaLnRUHrnu4nZ+OGqQkfRA43vpAcDRIvUh97q1+8qPZJ/bi6tsyp5plPuNn1iHCu2U5FxR1HZx27myYqOIKEYUPpSoisehWYy+y0DiLO3qmppXVAYur4PGwXnW9gSxcHLywmnZneX29QvpzCCPoqqwxgxRQLe 7LUSWsTl 0SFDF5Kg6/yhqHp9xxHx9105sEHbCTScyxNek5ZTpVrVXD8PRRZa00OPGB4x5XoEhfmV67dci4T5xhmYi0Xj63YtA/AA8hf6ixls7fuywYcauCt6kGx2V3ZJLoEgLlSbfWP2PPem3E/7XyrNDjK3gX23/2B9QPtoZS+ueY1bkDN1GgozPuz5HOSt6ZAj10E7GSdCuPRyAI8tSm0MyvMUEQWJl8goOCdiqOcjD1KFDMCVaFaZNdbWVXtUALBZSYLa26FZSv8pWMmsj+vQbN8y4yjQVQtjet5A97OvXqnJzlCxTY/b3FRzTIkACQwr5DGOqOjRR1cNKxf5NgEZQY62JbTYZw1BOyO8qWPehgUTHxrLPLUY6EMCz0Qq6gAcUBco3noN2c9PlqyhOXf8cjU/1WCTtfgbjupbPijXYNNwfFrDTCK795hQdWgJPiSTeLRRiWq/WmVI9WMaat1nIYBlTCl94pg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add the generic pte_free_defer(), to call pte_free() via call_rcu(). pte_free_defer() will be called inside khugepaged's retract_page_tables() loop, where allocating extra memory cannot be relied upon. This version suits all those architectures which use an unfragmented page for one page table (none of whose pte_free()s use the mm arg which was passed to it). Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 2 ++ mm/pgtable-generic.c | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8b0fc7fdc46f..62a8732d92f0 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -112,6 +112,8 @@ static inline void pte_unmap(pte_t *pte) } #endif +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); + /* Find an entry in the second-level page table.. */ #ifndef pmd_offset static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index d28b63386cef..471697dcb244 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -13,6 +13,7 @@ #include #include #include +#include #include /* @@ -230,6 +231,25 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, return pmd; } #endif + +/* arch define pte_free_defer in asm/pgalloc.h for its own implementation */ +#ifndef pte_free_defer +static void pte_free_now(struct rcu_head *head) +{ + struct page *page; + + page = container_of(head, struct page, rcu_head); + pte_free(NULL /* mm not passed and not used */, (pgtable_t)page); +} + +void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) +{ + struct page *page; + + page = pgtable; + call_rcu(&page->rcu_head, pte_free_now); +} +#endif /* pte_free_defer */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #if defined(CONFIG_GUP_GET_PXX_LOW_HIGH) && \ From patchwork Mon May 29 06:25:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5014C7EE32 for ; Mon, 29 May 2023 06:25:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 442EA900003; Mon, 29 May 2023 02:25:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CC81900002; Mon, 29 May 2023 02:25:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24620900003; Mon, 29 May 2023 02:25:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 100DC900002 for ; Mon, 29 May 2023 02:25:23 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D62FB80224 for ; Mon, 29 May 2023 06:25:22 +0000 (UTC) X-FDA: 80842305684.05.1855A68 Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf01.hostedemail.com (Postfix) with ESMTP id EF88740017 for ; Mon, 29 May 2023 06:25:20 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=vy4bOZeB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JHIvVFDDeb0UmLqYECNwvBDzrNYvPV2mAJt9DuZsGqI=; b=OgSQ2K8sm8cBp/eEC/d7jbtrhJjIGrIlfnvKDlugHnoL9kZpxWXDAYZimHR8tE/BjQnRFK 4+fkXc593CFrDhr6L/0d81hw9Od66doNzDIdA+UlMOYhkVQePYKV3IRA+1S1/Rq5s/Q72M 16GL1vjs1sTHePoVhx4BNxlsuXUPszk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=vy4bOZeB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341521; a=rsa-sha256; cv=none; b=j5PllYI+go42NggITz82NR0uTqVdcGHiTHqWD0PEuOcA3TfFq6TDaYv2tpn4EDYWpqfRaT 45Om/M5MMk0UcU40HhTm424rsY67mdIkdu88aXsRxEJ/2MfO37GGfeyZaVR9bu88zntiyX pvbIlqFGychYYRfGXMlF8PuUc9AJ3Rc= Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-ba829e17aacso4350524276.0 for ; Sun, 28 May 2023 23:25:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341520; x=1687933520; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=JHIvVFDDeb0UmLqYECNwvBDzrNYvPV2mAJt9DuZsGqI=; b=vy4bOZeB/A66J4j9HnNgzhCNnvZdPQ/qYVsp6cHRESlm/+NNE5/wIBuX4BQPZZXi+U yfOv5KIuNFZIGxmT2J41Omwq2WuOLtYXkDCRVQFlDnKL7Sb2EkLjAGyVqgxk6YwVvXsq 4Pf+i37X+4QiOjtG5wU2Llmv4Y51UqbqrD37XCP8fC+Il7S+6dvF57+wSKT3pOZKo9Wq XS9k4OAS6EFviHg6GY7NWmUx1fP6lsswTyubgGRjqDpTcAziJzjScdUEu1t3l5BKVD8B YHHYTZOtafVUcsfZZNCTBvgZT4szHnXqQAoJ2bUIgqmZt8ZRwlANX5NxU9BJDyCisuqK 1k1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341520; x=1687933520; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JHIvVFDDeb0UmLqYECNwvBDzrNYvPV2mAJt9DuZsGqI=; b=gEmNqLI8KyY19d2pgBcxm224ShLS5qEMhBiCEF9Q6X6m2gHVP1UJJtk9y0eIBARmes SuiVnYdn5iUjV3NfL8o6UPZQGJDDII9V6b3McTfGPWZsq/WpToxGOGCoHzmltIOgKeHN 9DdXxxg6UmfyQFnV5MTvl9Ecw36P3ZfTFWWLO278B981/R5vsQzsK+IilfEKFu5EuoFg fyLBJ73N2Un2Aaq9ZS6y6B8YnEovGoj3opSoeKWbd57zKEr2VqHiRJnnfDEssabI7pFa AnEZn8ZvXv2VwKF25rliZ2cTqfikR6GbK0zoNplkRS4LxubnV1VmDYKoddKJxgceHp+U 6ubw== X-Gm-Message-State: AC+VfDzjERYZVDS9NXRWnGL8kvZaKgSc2j1Z5JZPwfBViv3S2nNyTF/Y MITPyB1e80fPMpiKWm//g6+AFQ== X-Google-Smtp-Source: ACHHUZ760oJ9OUejnylCEVSFBks584YyNq1hzCWCB4cpQetH+OHnL70ZjQyryA07DRyxberP/3hpcA== X-Received: by 2002:a25:ae87:0:b0:b9e:7082:971e with SMTP id b7-20020a25ae87000000b00b9e7082971emr9144162ybj.45.1685341519867; Sun, 28 May 2023 23:25:19 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x18-20020a258592000000b00ba88763e5b5sm2667181ybk.2.2023.05.28.23.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:25:19 -0700 (PDT) Date: Sun, 28 May 2023 23:25:15 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 09/12] mm/khugepaged: retract_page_tables() without mmap or vma lock In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <2e9996fa-d238-e7c-1194-834a2bd1f60@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: ifcdpm7g6nob9akue56cz953oiy8qd1x X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EF88740017 X-HE-Tag: 1685341520-623447 X-HE-Meta: U2FsdGVkX19z57we+q/sGVKFqUoNtqgDbxUzyfYHWVMdT5jZajLXsiiotzQFdGaBxP83ApuHF0p8yDuEzWxr/oM+tNBCo/GNIBIZEoE3CpQygO+zQAkWC+tKRYXF7vebk2ClwAPEtib56TqxbAjduVTKDl0ako4cfVknHZ7cyf46OmXh0oYjvjILJkvk92K7K8tHPZU9hsvER7rhYw6Wxjx+HDiJQzvn22IFDDkz5f+ScV3ZZ/xuI7K4F8HSbfKxGJ2z7JeGY9at/SFJXDc2D9J4WG0LECQSNgNszS9e3ptvEijyudkxWBmkgml9IqAXxjCUFbawM3pj6NwrEfs4krC7delf+fm8KohyU7v11rIERnA88wqCROelkeJj9hnDWcrZF8fdtVRUBU1cZrXYKXuTaYYRThJnl4R3q5VWPaYW9czt7nId3tjlNEB/78fGoHJUB7C6y5Tbu+4coZNUgJbpK8/zGZBYW9477ToW+nSuZR7aV7Koop0eNCrOVbBPAX8cXst56VKbYQx+aiGBsIeZzjIokLIGczXKuuLTYerLpePQTy2UPPF32E7APvcx30A7adAl6nYWIC0VJaSBZsKMfG2XuGaZvkjgt70zEeDGZEq1l8Vbm78wpq9Gu6uesGZ2nbqEYeC6eDwELoYUmfRPjVW5sRloRf64eI4bEk4I4sbk5+C7gU+Rr9zclgX9NWu1XkpFfA+myle55xAXq5UsjOvRVmcH6oeIkDo2RLlySmg9sl0k1Zl6KuVpaUMWr0I3RdPjYBooqMGyhvENGBQizLrXMi7OeUrjejuNvtkKIadb20jZDCd/5u2oE1PvWsSsE36n3rieolyOOsfCltY5+HYYNhKTkgc/wIxJyvI67IrOpTccah4zRDMEKsiDI63MlT/DNV0oVpczBZ7DdII+x0+mvbLT61Xb64auWV2lWJP0LF1APV2eYOW71SmC8IhndQz/Kd1eu5rSgIg pgARqghf M00a26poyanWlk+Mzv0I+qC0qw6EHYTMcdy7cylRZOUAijt8E8gece22llO2W3PbrstusHWAYX3FnGIHXlaettJsqQyarYkL+zsNJ96ccT4NlumZxoKcoOoiurvA8APlqok89WkaWx6h5ZX3H5jnUgiWebKj4HNqUyiWuMjYoN2tPqiTZ+gwI2vbLTORPI6FsBGrcZBJ/ZqO6C3XxKwvPZoHgfpe2NulK1nbClq/nRI3SzcQ+Anu1VqXfN/XjTpatJPFDlgqYGxWjookpuNTmzQ3s7gcXZt7YjBLghEI8NBPTzTMVg58R0ffqbVw/OVqL9pgNIPQg5U/ROwZJQ+zFJSZlioH/1UdpZ0TR4DgkQjO86MuFm5wutWkz1kmFV6VHK0d1KmZJvtaEPTlih2R5kCjCXe1WvHYZ+0+xiV+VLNqW74K6F4AUZq5UjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simplify shmem and file THP collapse's retract_page_tables(), and relax its locking: to improve its success rate and to lessen impact on others. Instead of its MADV_COLLAPSE case doing set_huge_pmd() at target_addr of target_mm, leave that part of the work to madvise_collapse() calling collapse_pte_mapped_thp() afterwards: just adjust collapse_file()'s result code to arrange for that. That spares retract_page_tables() four arguments; and since it will be successful in retracting all of the page tables expected of it, no need to track and return a result code itself. It needs i_mmap_lock_read(mapping) for traversing the vma interval tree, but it does not need i_mmap_lock_write() for that: page_vma_mapped_walk() allows for pte_offset_map_lock() etc to fail, and uses pmd_lock() for THPs. retract_page_tables() just needs to use those same spinlocks to exclude it briefly, while transitioning pmd from page table to none: so restore its use of pmd_lock() inside of which pte lock is nested. Users of pte_offset_map_lock() etc all now allow for them to fail: so retract_page_tables() now has no use for mmap_write_trylock() or vma_try_start_write(). In common with rmap and page_vma_mapped_walk(), it does not even need the mmap_read_lock(). But those users do expect the page table to remain a good page table, until they unlock and rcu_read_unlock(): so the page table cannot be freed immediately, but rather by the recently added pte_free_defer(). retract_page_tables() can be enhanced to replace_page_tables(), which inserts the final huge pmd without mmap lock: going through an invalid state instead of pmd_none() followed by fault. But that does raise some questions, and requires a more complicated pte_free_defer() for powerpc (when its arch_needs_pgtable_deposit() for shmem and file THPs). Leave that enhancement to a later release. Signed-off-by: Hugh Dickins --- mm/khugepaged.c | 169 +++++++++++++++++------------------------------- 1 file changed, 60 insertions(+), 109 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 1083f0e38a07..4fd408154692 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1617,9 +1617,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, break; case SCAN_PMD_NONE: /* - * In MADV_COLLAPSE path, possible race with khugepaged where - * all pte entries have been removed and pmd cleared. If so, - * skip all the pte checks and just update the pmd mapping. + * All pte entries have been removed and pmd cleared. + * Skip all the pte checks and just update the pmd mapping. */ goto maybe_install_pmd; default: @@ -1748,123 +1747,73 @@ static void khugepaged_collapse_pte_mapped_thps(struct khugepaged_mm_slot *mm_sl mmap_write_unlock(mm); } -static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, - struct mm_struct *target_mm, - unsigned long target_addr, struct page *hpage, - struct collapse_control *cc) +static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) { struct vm_area_struct *vma; - int target_result = SCAN_FAIL; - i_mmap_lock_write(mapping); + i_mmap_lock_read(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) { - int result = SCAN_FAIL; - struct mm_struct *mm = NULL; - unsigned long addr = 0; - pmd_t *pmd; - bool is_target = false; + struct mm_struct *mm; + unsigned long addr; + pmd_t *pmd, pgt_pmd; + spinlock_t *pml; + spinlock_t *ptl; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that - * got written to. These VMAs are likely not worth investing - * mmap_write_lock(mm) as PMD-mapping is likely to be split - * later. + * got written to. These VMAs are likely not worth removing + * page tables from, as PMD-mapping is likely to be split later. * - * Note that vma->anon_vma check is racy: it can be set up after - * the check but before we took mmap_lock by the fault path. - * But page lock would prevent establishing any new ptes of the - * page, so we are safe. - * - * An alternative would be drop the check, but check that page - * table is clear before calling pmdp_collapse_flush() under - * ptl. It has higher chance to recover THP for the VMA, but - * has higher cost too. It would also probably require locking - * the anon_vma. + * Note that vma->anon_vma check is racy: it can be set after + * the check, but page locks (with XA_RETRY_ENTRYs in holes) + * prevented establishing new ptes of the page. So we are safe + * to remove page table below, without even checking it's empty. */ - if (READ_ONCE(vma->anon_vma)) { - result = SCAN_PAGE_ANON; - goto next; - } + if (READ_ONCE(vma->anon_vma)) + continue; + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); if (addr & ~HPAGE_PMD_MASK || - vma->vm_end < addr + HPAGE_PMD_SIZE) { - result = SCAN_VMA_CHECK; - goto next; - } - mm = vma->vm_mm; - is_target = mm == target_mm && addr == target_addr; - result = find_pmd_or_thp_or_none(mm, addr, &pmd); - if (result != SCAN_SUCCEED) - goto next; - /* - * We need exclusive mmap_lock to retract page table. - * - * We use trylock due to lock inversion: we need to acquire - * mmap_lock while holding page lock. Fault path does it in - * reverse order. Trylock is a way to avoid deadlock. - * - * Also, it's not MADV_COLLAPSE's job to collapse other - * mappings - let khugepaged take care of them later. - */ - result = SCAN_PTE_MAPPED_HUGEPAGE; - if ((cc->is_khugepaged || is_target) && - mmap_write_trylock(mm)) { - /* trylock for the same lock inversion as above */ - if (!vma_try_start_write(vma)) - goto unlock_next; - - /* - * Re-check whether we have an ->anon_vma, because - * collapse_and_free_pmd() requires that either no - * ->anon_vma exists or the anon_vma is locked. - * We already checked ->anon_vma above, but that check - * is racy because ->anon_vma can be populated under the - * mmap lock in read mode. - */ - if (vma->anon_vma) { - result = SCAN_PAGE_ANON; - goto unlock_next; - } - /* - * When a vma is registered with uffd-wp, we can't - * recycle the pmd pgtable because there can be pte - * markers installed. Skip it only, so the rest mm/vma - * can still have the same file mapped hugely, however - * it'll always mapped in small page size for uffd-wp - * registered ranges. - */ - if (hpage_collapse_test_exit(mm)) { - result = SCAN_ANY_PROCESS; - goto unlock_next; - } - if (userfaultfd_wp(vma)) { - result = SCAN_PTE_UFFD_WP; - goto unlock_next; - } - collapse_and_free_pmd(mm, vma, addr, pmd); - if (!cc->is_khugepaged && is_target) - result = set_huge_pmd(vma, addr, pmd, hpage); - else - result = SCAN_SUCCEED; - -unlock_next: - mmap_write_unlock(mm); - goto next; - } - /* - * Calling context will handle target mm/addr. Otherwise, let - * khugepaged try again later. - */ - if (!is_target) { - khugepaged_add_pte_mapped_thp(mm, addr); + vma->vm_end < addr + HPAGE_PMD_SIZE) continue; - } -next: - if (is_target) - target_result = result; + + mm = vma->vm_mm; + if (find_pmd_or_thp_or_none(mm, addr, &pmd) != SCAN_SUCCEED) + continue; + + if (hpage_collapse_test_exit(mm)) + continue; + /* + * When a vma is registered with uffd-wp, we cannot recycle + * the page table because there may be pte markers installed. + * Other vmas can still have the same file mapped hugely, but + * skip this one: it will always be mapped in small page size + * for uffd-wp registered ranges. + * + * What if VM_UFFD_WP is set a moment after this check? No + * problem, huge page lock is still held, stopping new mappings + * of page which might then get replaced by pte markers: only + * existing markers need to be protected here. (We could check + * after getting ptl below, but this comment distracting there!) + */ + if (userfaultfd_wp(vma)) + continue; + + /* Huge page lock is still held, so page table must be empty */ + pml = pmd_lock(mm, pmd); + ptl = pte_lockptr(mm, pmd); + if (ptl != pml) + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + pgt_pmd = pmdp_collapse_flush(vma, addr, pmd); + if (ptl != pml) + spin_unlock(ptl); + spin_unlock(pml); + + mm_dec_nr_ptes(mm); + page_table_check_pte_clear_range(mm, addr, pgt_pmd); + pte_free_defer(mm, pmd_pgtable(pgt_pmd)); } - i_mmap_unlock_write(mapping); - return target_result; + i_mmap_unlock_read(mapping); } /** @@ -2261,9 +2210,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* * Remove pte page tables, so we can re-fault the page as huge. + * If MADV_COLLAPSE, adjust result to call collapse_pte_mapped_thp(). */ - result = retract_page_tables(mapping, start, mm, addr, hpage, - cc); + retract_page_tables(mapping, start); + if (cc && !cc->is_khugepaged) + result = SCAN_PTE_MAPPED_HUGEPAGE; unlock_page(hpage); /* From patchwork Mon May 29 06:26:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F43C77B7A for ; Mon, 29 May 2023 06:26:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 812C0900003; Mon, 29 May 2023 02:26:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C317900002; Mon, 29 May 2023 02:26:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6161B900003; Mon, 29 May 2023 02:26:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 51CEC900002 for ; Mon, 29 May 2023 02:26:55 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 277251C6B11 for ; Mon, 29 May 2023 06:26:55 +0000 (UTC) X-FDA: 80842309590.13.102D84A Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by imf05.hostedemail.com (Postfix) with ESMTP id 3CE0D100010 for ; Mon, 29 May 2023 06:26:53 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Ytf3pJx5; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OPZMX1+LA/FvHP0/JC5qx50WzIWIXFewxM+MGzYgG4M=; b=tvtJzOKQqtZhd61F1MiLy68BU+YcaXuY/+kvoczXQlxS4MxGKGs22ViXT6sjGciQtV905d bqwLYx2Iu7DRZQ0qOol2GqBiR4TL7GiT/WJgl56rTqIBfKppGtbv52TgJOtZ/f54Z5yCLU ay9gIgada7Y1/sUb+s9+XYG8NHHGZJY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341613; a=rsa-sha256; cv=none; b=xWphZ7LSJQzwE+3AVUOTUwccTL/pEt5MbEhx0VxLOnzwMhk1KtrgdgntDbGX/czpmn9gLE 79gWs8XestkRPeCyLAXhgQrk71nl6syHgtO8QNCw/w9uLA68tFWWXvCM4ptUDOo4g/MsgW j8dHuM8LwTj+4z5nW4EzJ8qKm4B8Y40= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Ytf3pJx5; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-ba8afcc82c0so5796218276.2 for ; Sun, 28 May 2023 23:26:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341612; x=1687933612; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=OPZMX1+LA/FvHP0/JC5qx50WzIWIXFewxM+MGzYgG4M=; b=Ytf3pJx5ZO2h8JphUMEUM+NgVIXTwIVhtYTQ/lA5Y8cir60w0OXuWtSllgiDhinPaR Iy4j/+CUvFlAp4R/CjSOblw5I+znP0kaNSqJXR4VKYKvTUfPb3oYvzoFoZFf0g3HTRPQ tAY4R1a5sWTcuBd7Oz6cpvBBwVDFmQiqte1C3YTUpMO9AoANcfZAyiW/btELN35Kv490 fFp3EQeG93PsokK2axn0N0nvmbnw91onFgB1MigUnnJSNnNJsoUz87zNfUFbsCL7jZ72 cyBArEZpOY67dIbt6WH+/9VlUHqVGsY29EI6EqLyGxDNiZ8V+VTllfG1NJsbPWUsuh6Y 4UjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341612; x=1687933612; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OPZMX1+LA/FvHP0/JC5qx50WzIWIXFewxM+MGzYgG4M=; b=ZzXhlPtdcbmB23MpbezdVAvRccUur3pHlCYziWGsQT3TJrV1tH/I71CaKQ4ukymTn6 791HAnjp+QPikpGtHEul77PdA1UXU0VpB6RoENN8rd1EFkToXYi/agNO7ybOazD7fjLN qcGe8WjfhjCXFctC5UHFiE3v2Zj8T0CX4P4gT+OaXI0rGh7HZPv9CxV0ENK7+4xq8Ez3 gVcYw/Jcc8wnB/NCkWsSbafK9ng2ZpfIC+/mjmuAsAJeI2Yv5gQwx4wiLUeOZAmYNdud Zk5ljsRpISEoVhgFOA2Wpx1LkS2AeZ4OFYqMWSU0rNADRI6FgX+oqcrI6jRVTDXx1s2M tN1w== X-Gm-Message-State: AC+VfDxmHKzf/8fQ+tsYupLy8Bxg9CQXKC76VEYTB/U+AZqj3uCOC6pe g+NV28Z+qNJtJSgFpmqt2KmuIQ== X-Google-Smtp-Source: ACHHUZ5aDBGSE8V0e3lWDnHjHjCYn6fQF4N/hxkcmf1ygogwgzLWM5AgQMYGVJCj77dNUpkBtnYW7w== X-Received: by 2002:a81:5b55:0:b0:565:ec67:18f4 with SMTP id p82-20020a815b55000000b00565ec6718f4mr4108617ywb.32.1685341612167; Sun, 28 May 2023 23:26:52 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m6-20020a0de306000000b0055a486140b6sm3427593ywe.36.2023.05.28.23.26.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:26:51 -0700 (PDT) Date: Sun, 28 May 2023 23:26:48 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 10/12] mm/khugepaged: collapse_pte_mapped_thp() with mmap_read_lock() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <563340a4-7ac9-7cc8-33d8-f7cc6ef19ea6@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Stat-Signature: mo84bor9keoruaaut573k3he3gysywp7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3CE0D100010 X-Rspam-User: X-HE-Tag: 1685341613-554641 X-HE-Meta: U2FsdGVkX1/O9yfLKpgwAkTQYuZ4x8j+DF8RdIkyWtmUopK78FtlV10rgtg3mWTm2okbIIFdrkME9awke4L8P0g+f0pAgefVo7hfDiarqQmbrOkXgsmSenamwcEyUatYgoVeGK/RIRmdcv3yjSKdhTKqUy3oa3Uka7oWpDlcNO2UmUGbbx91MkFeWBskmmuyKyii4KOl61OcOM1Ja5KXB10b5h9moEuBEj5lGyG76q7HL33NReiuAzBHBTkSjtiOBJyw7LxZurB4SgZ4ruCFKyfeKXBkn70cuc+yWUoOA6qyH0B68iQdQ81P+flGC07pHQWqof/BRpIgkJE7PzXsA+3eZBhb0qeOMGrm5humrQFsTr6YG9MxvAzLxHlEOjSaeFFsMKCRm/Vn1bmHlR3pyrU2sQs/OknmVEyhPuxlf4+pZdOLnZe5koPpLBX/DHWppU/sMaQO0b3E3v/mjjjG+ZEdB9fJxaJ7BoH8dGUsJfrj03/mOgiPtV0fFu5bVHC27XqgJz3zmmOb1TzHEh49RJi9yfGolZtXnd5kyP0X4W9xlTtTrMlJReAy7QsmyqhA5g9GKQDy1GEfIj7w7RrsbYpIOx3sZoNXqgXC48RFMIiKgabNZpgT8bUMpjrtltg9HzrzP5I3EYjlKRndiwp8KXnfbzvINZQe6EMFIn+y0Ez0/VFLgwnXfWL8Ew8S+Ke3Cur/4G8tWX0k6kyMfJ4L87P8eoC5lsAFAkkVH/4wO7fMbJwEhQIh5Jdzts23RdHqfQrGkuuATIfV8y+liCwbpBTaNVPCRIZtzcs0Hm/fCSS8ROVK+zXiS09M1gBDWVrlRHZk1HNKWBU7PUyfAHgcgkGkYmdzGD/Ar9l0UnISIjxN1fMrQrh/NGXuRze8ae1CPhwVgcCLOFk/fJz8+qmf9zgPJYwjHK911PIPECyziy7Gn3xJT8LkaxTTWZ6KbgxoRushI93SJ+vBMXWHREj lAG0Dk2p ci3hzbUtfRuO1OJdr0Q1SfD0tiAKwVJ4XGmQhh/bb/lNmdLGV7qyJOHKqekb+l3hzhpflVLrdbOny7t3u3evka6OcDaQS7My5ddpuwEYPfftpandYCKPf+GMhlF3Qq4ec5OUrHf1Ou1fQGsIEmkj8BJv4R/cPWGc8hQkjeLAuHvlOqwhKkL5f/r5VSR11kax0rva7is2ObtvpNXTMsr17H+PdNlFsgpENnawcsot3vR5w0gnIGajWQj0dx+vjXS0IiUKon4RDipMxTUNgwxNpa/Mv2O3viS8hP/zY+pJwTuGyCYCi4CRw9ekNBjhjJY6WgbFbYQtCVJABBdoo7DFHQAB9iOh5TWTXiro4Y7soyDLXhOioIrJphbimvnUwEBB/QPBiXC+7cfAOmmNjVWxGegzD66I7Zhn3cuqEw8I7M7CDFgp7FxUdkP4z3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Bring collapse_and_free_pmd() back into collapse_pte_mapped_thp(). It does need mmap_read_lock(), but it does not need mmap_write_lock(), nor vma_start_write() nor i_mmap lock nor anon_vma lock. All racing paths are relying on pte_offset_map_lock() and pmd_lock(), so use those. Follow the pattern in retract_page_tables(); and using pte_free_defer() removes the need for tlb_remove_table_sync_one() here. Confirm the preliminary find_pmd_or_thp_or_none() once page lock has been acquired and the page looks suitable: from then on its state is stable. However, collapse_pte_mapped_thp() was doing something others don't: freeing a page table still containing "valid" entries. i_mmap lock did stop a racing truncate from double-freeing those pages, but we prefer collapse_pte_mapped_thp() to clear the entries as usual. Their TLB flush can wait until the pmdp_collapse_flush() which follows, but the mmu_notifier_invalidate_range_start() has to be done earlier. Some cleanup while rearranging: rename "count" to "nr_ptes"; and "step 2" does not need to duplicate the checks in "step 1". Signed-off-by: Hugh Dickins --- mm/khugepaged.c | 131 +++++++++++++++--------------------------------- 1 file changed, 41 insertions(+), 90 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4fd408154692..2999500abdd5 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1485,7 +1485,7 @@ static bool khugepaged_add_pte_mapped_thp(struct mm_struct *mm, return ret; } -/* hpage must be locked, and mmap_lock must be held in write */ +/* hpage must be locked, and mmap_lock must be held */ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, struct page *hpage) { @@ -1497,7 +1497,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, }; VM_BUG_ON(!PageTransHuge(hpage)); - mmap_assert_write_locked(vma->vm_mm); + mmap_assert_locked(vma->vm_mm); if (do_set_pmd(&vmf, hpage)) return SCAN_FAIL; @@ -1506,48 +1506,6 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, return SCAN_SUCCEED; } -/* - * A note about locking: - * Trying to take the page table spinlocks would be useless here because those - * are only used to synchronize: - * - * - modifying terminal entries (ones that point to a data page, not to another - * page table) - * - installing *new* non-terminal entries - * - * Instead, we need roughly the same kind of protection as free_pgtables() or - * mm_take_all_locks() (but only for a single VMA): - * The mmap lock together with this VMA's rmap locks covers all paths towards - * the page table entries we're messing with here, except for hardware page - * table walks and lockless_pages_from_mm(). - */ -static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmdp) -{ - pmd_t pmd; - struct mmu_notifier_range range; - - mmap_assert_write_locked(mm); - if (vma->vm_file) - lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); - /* - * All anon_vmas attached to the VMA have the same root and are - * therefore locked by the same lock. - */ - if (vma->anon_vma) - lockdep_assert_held_write(&vma->anon_vma->root->rwsem); - - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, addr, - addr + HPAGE_PMD_SIZE); - mmu_notifier_invalidate_range_start(&range); - pmd = pmdp_collapse_flush(vma, addr, pmdp); - tlb_remove_table_sync_one(); - mmu_notifier_invalidate_range_end(&range); - mm_dec_nr_ptes(mm); - page_table_check_pte_clear_range(mm, addr, pmd); - pte_free(mm, pmd_pgtable(pmd)); -} - /** * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at * address haddr. @@ -1563,16 +1521,17 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool install_pmd) { + struct mmu_notifier_range range; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); struct page *hpage; pte_t *start_pte, *pte; - pmd_t *pmd; - spinlock_t *ptl; - int count = 0, result = SCAN_FAIL; + pmd_t *pmd, pgt_pmd; + spinlock_t *pml, *ptl; + int nr_ptes = 0, result = SCAN_FAIL; int i; - mmap_assert_write_locked(mm); + mmap_assert_locked(mm); /* Fast check before locking page if already PMD-mapped */ result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1612,6 +1571,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + result = find_pmd_or_thp_or_none(mm, haddr, &pmd); switch (result) { case SCAN_SUCCEED: break; @@ -1625,27 +1585,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } - /* Lock the vma before taking i_mmap and page table locks */ - vma_start_write(vma); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, + haddr, haddr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); - /* - * We need to lock the mapping so that from here on, only GUP-fast and - * hardware page walks can access the parts of the page tables that - * we're operating on. - * See collapse_and_free_pmd(). - */ - i_mmap_lock_write(vma->vm_file->f_mapping); - - /* - * This spinlock should be unnecessary: Nobody else should be accessing - * the page tables under spinlock protection here, only - * lockless_pages_from_mm() and the hardware page walker can access page - * tables while all the high-level locks are held in write mode. - */ result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); - if (!start_pte) - goto drop_immap; + if (!start_pte) /* mmap_lock + page lock should prevent this */ + goto abort; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1671,40 +1618,44 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ if (hpage + i != page) goto abort; - count++; + nr_ptes++; } - /* step 2: adjust rmap */ + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { - struct page *page; - if (pte_none(*pte)) continue; - page = vm_normal_page(vma, addr, *pte); - if (WARN_ON_ONCE(page && is_zone_device_page(page))) - goto abort; - page_remove_rmap(page, vma, false); + + /* Must clear entry, or a racing truncate may re-remove it */ + pte_clear(mm, addr, pte); + page_remove_rmap(hpage + i, vma, false); } pte_unmap_unlock(start_pte, ptl); /* step 3: set proper refcount and mm_counters. */ - if (count) { - page_ref_sub(hpage, count); - add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -count); + if (nr_ptes) { + page_ref_sub(hpage, nr_ptes); + add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -nr_ptes); } - /* step 4: remove pte entries */ - /* we make no change to anon, but protect concurrent anon page lookup */ - if (vma->anon_vma) - anon_vma_lock_write(vma->anon_vma); + /* step 4: remove page table */ - collapse_and_free_pmd(mm, vma, haddr, pmd); + /* Huge page lock is still held, so page table must remain empty */ + pml = pmd_lock(mm, pmd); + if (ptl != pml) + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); + if (ptl != pml) + spin_unlock(ptl); + spin_unlock(pml); - if (vma->anon_vma) - anon_vma_unlock_write(vma->anon_vma); - i_mmap_unlock_write(vma->vm_file->f_mapping); + mmu_notifier_invalidate_range_end(&range); + + mm_dec_nr_ptes(mm); + page_table_check_pte_clear_range(mm, haddr, pgt_pmd); + pte_free_defer(mm, pmd_pgtable(pgt_pmd)); maybe_install_pmd: /* step 5: install pmd entry */ @@ -1718,9 +1669,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, return result; abort: - pte_unmap_unlock(start_pte, ptl); -drop_immap: - i_mmap_unlock_write(vma->vm_file->f_mapping); + if (start_pte) + pte_unmap_unlock(start_pte, ptl); + mmu_notifier_invalidate_range_end(&range); goto drop_hpage; } @@ -2842,9 +2793,9 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, case SCAN_PTE_MAPPED_HUGEPAGE: BUG_ON(mmap_locked); BUG_ON(*prev); - mmap_write_lock(mm); + mmap_read_lock(mm); result = collapse_pte_mapped_thp(mm, addr, true); - mmap_write_unlock(mm); + mmap_locked = true; goto handle_result; /* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL: From patchwork Mon May 29 06:28:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05339C7EE23 for ; Mon, 29 May 2023 06:29:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D5B2900003; Mon, 29 May 2023 02:29:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8859B900002; Mon, 29 May 2023 02:29:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72692900003; Mon, 29 May 2023 02:29:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 62016900002 for ; Mon, 29 May 2023 02:29:00 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 314391601FD for ; Mon, 29 May 2023 06:29:00 +0000 (UTC) X-FDA: 80842314840.27.3ABEF8B Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by imf17.hostedemail.com (Postfix) with ESMTP id 60B0740016 for ; Mon, 29 May 2023 06:28:58 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=JvvTudiJ; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xT/EEswLF0xV48TACSfwsUMzv4W28WePE24GrL1l/5o=; b=r9fJE8QONT+ns2XUL4lEDUvO6OOJyhLxCKKwxjtl6hbNoeSrZYoEybYiSNS8EwvbNgl7mJ nRtauAKedqMbhMEsSTixWBnZc7zCSGkIbya8X37IJq1emzdCVXUsEq5KT/zf1W4Gvb5u7g C5MXwYyE62P7De6S3G0LapVNOksXjrA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=JvvTudiJ; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341738; a=rsa-sha256; cv=none; b=hNxpUY7bSFYoTDcX2Su0SNpYeO4FAGvOY+ZHEEWrjlwixvdYsOpTydt2pCLzQ7mkCSnUQ9 87XdPh8jCZwuPMPGPUnoYJsDIzNNTTbiKWNdA1DJOXw/rUYs8SEKCEXiRjuxF+R1OPbSTJ bR9KcjyJZDLwAulB+PyYpSxhUvAWD4M= Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-ba818eb96dcso2257802276.0 for ; Sun, 28 May 2023 23:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341737; x=1687933737; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=xT/EEswLF0xV48TACSfwsUMzv4W28WePE24GrL1l/5o=; b=JvvTudiJ2ueEboa73P4a3lGmzg34HxMFolFy18ddYVZR38ryyurom4tpA+pKKdAKm+ yYAkrdkmojRae9uCGOc+xBDnB7qvKDF9c9FFPnpSBpjBiwbOJHOHU1+piNik8nuJHIHQ vpxQvD6IDQlSrCbcFB+Lk4wcPVE2ZDCw/XyBpkTiIDI1EAgr2XJxMGctVaCjxy4tlymz LrAusDPYqGDSES0yI8fnkSztcqZEaAtqHPYgPXvbz0LuLLq9bawcEEKWwmIcx4F6CA7T FILNf+ZbNWUKWLzPfTzPIQGR9twFhP7psJ5t7QdrhgKtJ5xSlV25NIlSqIAjV0HdJc5K 2WJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341737; x=1687933737; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xT/EEswLF0xV48TACSfwsUMzv4W28WePE24GrL1l/5o=; b=REwKN3e7Q4MTnmGil7zkKfuk4QGBSz+lU8+apKT7Dm2wjbJR8QwKpWe/RoPRBnMDqB khhWcH7kQhZ9DaUwq8Y3fg028m0ffJDOL4WHVfxGzmMdWnXKToqSuFMBQt5eAkwtkd1h VOfhJEGju7HaXKnW/uX1xBRj2F/qP0ApzKdeG9Hby8QdOnjqC9wZvhRpGQzOHo5Gw58q sk2CDA7CqGqVNuVTBPF+2jOvBbm3DQFgc7+2k5nHdMxcqwl47jDAgW9bqkJN1x1sFqDz xHECPUFl0lYPaKwf1oishIxjcaF6Xy+Z5O7nK+ZZbAa4v6r7013FhiPi0dQlS0ubqCXJ 5xsA== X-Gm-Message-State: AC+VfDwdP/Tgs0IX5VqxyI3QEQER/0fqw0IgMRFnbHGxqiiBxZzb9XTx B/ZczzF7g1NuF6BBwfXbsbJIcA== X-Google-Smtp-Source: ACHHUZ7HfnnGjkEX5sGUsgNAdc3kbPNe/FVMe5HYLThd9cglb9ZMTh0FR6+KstwImWlJK0i5/3xaIA== X-Received: by 2002:a25:d391:0:b0:bac:f582:eefd with SMTP id e139-20020a25d391000000b00bacf582eefdmr10483734ybf.35.1685341737360; Sun, 28 May 2023 23:28:57 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 63-20020a251142000000b00ba7cb887380sm2723779ybr.14.2023.05.28.23.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:28:57 -0700 (PDT) Date: Sun, 28 May 2023 23:28:52 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 11/12] mm/khugepaged: delete khugepaged_collapse_pte_mapped_thps() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: <1bf6f10-1f8d-d410-98b9-66cbf9a45c2@google.com> References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 60B0740016 X-Rspam-User: X-Stat-Signature: 73t34otys7k94zbanpy51q1o8sqk41xr X-Rspamd-Server: rspam01 X-HE-Tag: 1685341738-908118 X-HE-Meta: U2FsdGVkX18AXRCVhiExgSDRI/Q3CWhrfEsBM0z/J6n3nPgjMcH7j3dJUnYPFzMOZbDbCZ1mnG6Wv4ge3RMUjfs6DA1dROjxYtLquf9N2QyA3hZ0KrXNQcew+t+K5dtEA+Bz5RymyoZk9MXjpGUE+LfH39T0/dagE9hAYL7WR3M+ttEVH0kh6cjkbnL4P/4Ni/HnFxHqM64wgbmE9BpAj1aw6U0jIVmkkhMkAa3C3+BwCVVnvCcW51y9ZoJ1kO0ThBJxfN35TBeHmZK3GVJ9jV9L+JeBneIfPqzp0Gr/G6CpcoZHf3Npl0aJ9d/xPZUoJ12fJhZhov6yasA5bPobtfBlFWC3QeVZUbym/UAKLE5mMK94Vt4/jWtxs6DwXG9/lXTV168cnGHVsIjrqvrmkpMLTjhBnjtUGZFVe2HPRb8h0kT8AWkn84lO2Z8Wi+uWSM81YoYGgbD7VvU28ZyJmC9zx3kAASXgJFcz5goEgr3Fy+iaOjITQr49fNGl827kBv5MDkDt1gKXJFyeFVWVQPAZb1GH5mVEiht0wqmcuFPPcYUtsWrUfDLUXgqS4DCJwOpt76zh4mGfAwsPxocKnPko+iFyWxg3qIkq0/ppU4vh6ZV9ysGpCwCFZWmBVji1IT46u12jwZS1Yy9SC4PPrTfBcQwA8OrjrSIJWMA2BGn5wyqlnIjbMJCV+fOIPfmZyfd9D2ym3ExjVQNTb4kgdkvi15V7DnJtE81A5s5OnCxNB5qx3Cqe8pljKrtnXKR+170FkR2VaOxxOp5dvRxJJF1MgJn5PRM1rjGA9a59GDhPONUFMHvOU0yijj8XzFTh5TYFIuOCdy/c2YZg8KdxhXJwtaNdpfBS7SxPXxGPCrxX6Mrr/kney0JZ793dTodkKbg4/BF5lYrjgl/0qHD83b89NTWGvPrBZfUfGBr9OoOmisD0ySfQhwZBbTPVxnw+kX6t5fjvv8lNiFa1kEb 2YNdZ4pv ycZ5Ef6O+vcH4DJA/BI6FhMXtqTpz1CE3yAxrCWXJxLH3opLQtpWxPyvxf8OGX9gt+vTk++wXytJfKw/vbQpHY0IqEI5/QeQVcrQnliILaorM8xJxVrbCEwU+hbIjUrVhiNNtudJn3A167XU9I5gnLacQ6sNDTxIni8x9C/nw8l2bOBjAvoqqLFve6Y46KGzIIp/UGcVrZms3hlqTZDU3nK5CdQUQHuTbOdGZNWSZ7hevIizznyXpoSVCay1BRnHxzqoVGI4ew+8L0+KaiDXWIKlmFKJTl9eDOBNQQUERuZtrZXeQAw6NR2AA05161SomYYcFDHqxtlxOW+PQl2Fq1XaCHT/lkroArHkdULqzLeMKQbvL5nXzWArk5R0Z2CGYEAcWap4z3ruAsrttiaUIBFwM1spksp3wqg5WoxhHvpsu/rgTo4IijcsB8Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that retract_page_tables() can retract page tables reliably, without depending on trylocks, delete all the apparatus for khugepaged to try again later: khugepaged_collapse_pte_mapped_thps() etc; and free up the per-mm memory which was set aside for that in the khugepaged_mm_slot. But one part of that is worth keeping: when hpage_collapse_scan_file() found SCAN_PTE_MAPPED_HUGEPAGE, that address was noted in the mm_slot to be tried for retraction later - catching, for example, page tables where a reversible mprotect() of a portion had required splitting the pmd, but now it can be recollapsed. Call collapse_pte_mapped_thp() directly in this case (why was it deferred before? I assume an issue with needing mmap_lock for write, but now it's only needed for read). Signed-off-by: Hugh Dickins --- mm/khugepaged.c | 125 +++++++----------------------------------------- 1 file changed, 16 insertions(+), 109 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2999500abdd5..301c0e54a2ef 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -92,8 +92,6 @@ static __read_mostly DEFINE_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS); static struct kmem_cache *mm_slot_cache __read_mostly; -#define MAX_PTE_MAPPED_THP 8 - struct collapse_control { bool is_khugepaged; @@ -107,15 +105,9 @@ struct collapse_control { /** * struct khugepaged_mm_slot - khugepaged information per mm that is being scanned * @slot: hash lookup from mm to mm_slot - * @nr_pte_mapped_thp: number of pte mapped THP - * @pte_mapped_thp: address array corresponding pte mapped THP */ struct khugepaged_mm_slot { struct mm_slot slot; - - /* pte-mapped THP in this mm */ - int nr_pte_mapped_thp; - unsigned long pte_mapped_thp[MAX_PTE_MAPPED_THP]; }; /** @@ -1441,50 +1433,6 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot) } #ifdef CONFIG_SHMEM -/* - * Notify khugepaged that given addr of the mm is pte-mapped THP. Then - * khugepaged should try to collapse the page table. - * - * Note that following race exists: - * (1) khugepaged calls khugepaged_collapse_pte_mapped_thps() for mm_struct A, - * emptying the A's ->pte_mapped_thp[] array. - * (2) MADV_COLLAPSE collapses some file extent with target mm_struct B, and - * retract_page_tables() finds a VMA in mm_struct A mapping the same extent - * (at virtual address X) and adds an entry (for X) into mm_struct A's - * ->pte-mapped_thp[] array. - * (3) khugepaged calls khugepaged_collapse_scan_file() for mm_struct A at X, - * sees a pte-mapped THP (SCAN_PTE_MAPPED_HUGEPAGE) and adds an entry - * (for X) into mm_struct A's ->pte-mapped_thp[] array. - * Thus, it's possible the same address is added multiple times for the same - * mm_struct. Should this happen, we'll simply attempt - * collapse_pte_mapped_thp() multiple times for the same address, under the same - * exclusive mmap_lock, and assuming the first call is successful, subsequent - * attempts will return quickly (without grabbing any additional locks) when - * a huge pmd is found in find_pmd_or_thp_or_none(). Since this is a cheap - * check, and since this is a rare occurrence, the cost of preventing this - * "multiple-add" is thought to be more expensive than just handling it, should - * it occur. - */ -static bool khugepaged_add_pte_mapped_thp(struct mm_struct *mm, - unsigned long addr) -{ - struct khugepaged_mm_slot *mm_slot; - struct mm_slot *slot; - bool ret = false; - - VM_BUG_ON(addr & ~HPAGE_PMD_MASK); - - spin_lock(&khugepaged_mm_lock); - slot = mm_slot_lookup(mm_slots_hash, mm); - mm_slot = mm_slot_entry(slot, struct khugepaged_mm_slot, slot); - if (likely(mm_slot && mm_slot->nr_pte_mapped_thp < MAX_PTE_MAPPED_THP)) { - mm_slot->pte_mapped_thp[mm_slot->nr_pte_mapped_thp++] = addr; - ret = true; - } - spin_unlock(&khugepaged_mm_lock); - return ret; -} - /* hpage must be locked, and mmap_lock must be held */ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, struct page *hpage) @@ -1675,29 +1623,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } -static void khugepaged_collapse_pte_mapped_thps(struct khugepaged_mm_slot *mm_slot) -{ - struct mm_slot *slot = &mm_slot->slot; - struct mm_struct *mm = slot->mm; - int i; - - if (likely(mm_slot->nr_pte_mapped_thp == 0)) - return; - - if (!mmap_write_trylock(mm)) - return; - - if (unlikely(hpage_collapse_test_exit(mm))) - goto out; - - for (i = 0; i < mm_slot->nr_pte_mapped_thp; i++) - collapse_pte_mapped_thp(mm, mm_slot->pte_mapped_thp[i], false); - -out: - mm_slot->nr_pte_mapped_thp = 0; - mmap_write_unlock(mm); -} - static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) { struct vm_area_struct *vma; @@ -2326,16 +2251,6 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, { BUILD_BUG(); } - -static void khugepaged_collapse_pte_mapped_thps(struct khugepaged_mm_slot *mm_slot) -{ -} - -static bool khugepaged_add_pte_mapped_thp(struct mm_struct *mm, - unsigned long addr) -{ - return false; -} #endif static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, @@ -2365,7 +2280,6 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, khugepaged_scan.mm_slot = mm_slot; } spin_unlock(&khugepaged_mm_lock); - khugepaged_collapse_pte_mapped_thps(mm_slot); mm = slot->mm; /* @@ -2418,36 +2332,29 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, khugepaged_scan.address); mmap_read_unlock(mm); - *result = hpage_collapse_scan_file(mm, - khugepaged_scan.address, - file, pgoff, cc); mmap_locked = false; + *result = hpage_collapse_scan_file(mm, + khugepaged_scan.address, file, pgoff, cc); + if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { + mmap_read_lock(mm); + mmap_locked = true; + if (hpage_collapse_test_exit(mm)) { + fput(file); + goto breakouterloop; + } + *result = collapse_pte_mapped_thp(mm, + khugepaged_scan.address, false); + if (*result == SCAN_PMD_MAPPED) + *result = SCAN_SUCCEED; + } fput(file); } else { *result = hpage_collapse_scan_pmd(mm, vma, - khugepaged_scan.address, - &mmap_locked, - cc); + khugepaged_scan.address, &mmap_locked, cc); } - switch (*result) { - case SCAN_PTE_MAPPED_HUGEPAGE: { - pmd_t *pmd; - *result = find_pmd_or_thp_or_none(mm, - khugepaged_scan.address, - &pmd); - if (*result != SCAN_SUCCEED) - break; - if (!khugepaged_add_pte_mapped_thp(mm, - khugepaged_scan.address)) - break; - } fallthrough; - case SCAN_SUCCEED: + if (*result == SCAN_SUCCEED) ++khugepaged_pages_collapsed; - break; - default: - break; - } /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; From patchwork Mon May 29 06:30:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C888FC77B7A for ; Mon, 29 May 2023 06:30:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41A41900003; Mon, 29 May 2023 02:30:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CA34900002; Mon, 29 May 2023 02:30:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B5F900003; Mon, 29 May 2023 02:30:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 184C7900002 for ; Mon, 29 May 2023 02:30:32 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D48F2C0210 for ; Mon, 29 May 2023 06:30:31 +0000 (UTC) X-FDA: 80842318662.22.67D88CE Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by imf17.hostedemail.com (Postfix) with ESMTP id D8DED40020 for ; Mon, 29 May 2023 06:30:29 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UNKL1Q0P; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685341829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Aa5HWe45UW6rQUJiJNVLC4MPu6AngULI5k321Sbm9Ew=; b=Odeb20p5/bWLraN6uwmg+5x/7ixsx6dmd5QessBSA4bLZsfgY/ZihkM1o3Xqx9hqwCaMsT zBDKEso3Trkwg+COyEm1vMPHEDf+tF+HRhZTfS/A03dwIkIrlgoumv6ZyuRTCAn3wEH8x7 B9TOdKplCPvdXDcohDywnBg3ooR5+l8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UNKL1Q0P; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685341829; a=rsa-sha256; cv=none; b=tBE43Syb2LN1+dDs0zedLC//Hj0MjzICWIBPb6ON51q0KAAPjkqGsXdaT4ekBbAQh60AKB 6siWIgrGz0jcqvrNjcPN48ACztq47Y5SIbq3LXQTaI/OVoiC1GHSgvrzTgy+1S82OfXTsQ f7IhGf8oM2lh7lOkvl3g1vy0IpaoIqQ= Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-ba86ea269e0so4431137276.1 for ; Sun, 28 May 2023 23:30:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685341829; x=1687933829; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Aa5HWe45UW6rQUJiJNVLC4MPu6AngULI5k321Sbm9Ew=; b=UNKL1Q0Pec+L5oOba/99kxWNjcCwq8SenPXFdijAb/ObswMKjM7OUUmz2ZEICNHgFi Qv4ppRRcA3yebk/gcCjK1B/b31tItw1kcbI5MNnORL3Ys8OpaxJh/WEpZJ/SInbs7OtQ pID86tFe1+AG4qmsZJbOnM+7iKwXNWky09aARIe2tetg2zR8PTrceT/1TSWkarzFIet1 8YIKNPCa18G6ke7Me2QA0b5ztYnbCNDIOf/BDmkOzEl5UNMMd2c3BAGItXCN3K8DkXAw E9u7Ntp2cVwujFtJI7hoKruoONwfQaLYMkiR2S/Hp/B93RwLn01nkmija5yTld6gsNK0 YpVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685341829; x=1687933829; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Aa5HWe45UW6rQUJiJNVLC4MPu6AngULI5k321Sbm9Ew=; b=B2pJ+mlMCE6O5iLr2Mq5aS0XByUZa85mTvmedu8cRbYSNwj6UYcE0ISDqLCX3iu8qc 6fwBipMPVqTnSwCSI3XFV/vC0hdicr13lZvez0T2SRHEloiyCI2egwigDRE+XZtK0XYD dozbkcGNaF6cuVyLkGoQt3+JT6n0QLfGskBRiwgbmQfwTQ+mRz3/HfOi+AMp0pPTEW56 7TqDvttIJFVOIYRKpxOp456ta+T1bq/fvhN4OlCUCkc8zSSLannOCoO+gm+UJ7Bjagj6 faWcM5mRcDEqzzPnXa+6eGdqzvGLpC8djdJ1QHRHu8s8tEmXBPvGp0pDMZrUrtXXrzIp rakA== X-Gm-Message-State: AC+VfDzYqY2w4yQCbeh8RpgMOnTrPyoqu2Cz9pqsq2TiIlLx5WJz0CDR +d0dsJQElWcvaLzRB3yjhZ1jbQ== X-Google-Smtp-Source: ACHHUZ6M1tOYR5IzxCkUxy+sUt0Djeqr4hN+mskEbw3iamyUciaGMEXp/45S6Zcf95QTtDJx+FPlXA== X-Received: by 2002:a81:b40c:0:b0:544:9180:3104 with SMTP id h12-20020a81b40c000000b0054491803104mr11920044ywi.34.1685341828878; Sun, 28 May 2023 23:30:28 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a7-20020a818a07000000b00555c30ec361sm3363238ywg.143.2023.05.28.23.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:30:28 -0700 (PDT) Date: Sun, 28 May 2023 23:30:24 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 12/12] mm: delete mmap_write_trylock() and vma_try_start_write() In-Reply-To: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> Message-ID: References: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: njknqb7eyu7t4ffnqcxybgqg78nbuue3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D8DED40020 X-HE-Tag: 1685341829-774568 X-HE-Meta: U2FsdGVkX18guvZ9XotERFOjqnd5pD6/9jbLwoKZc9bo4kkzUayD3cJXn6u/Sm/gYKwgGC6O7Q0RZFB1DwdtMrCFyWHrTVEd7oWjIQvWXPxMa9vfxVlDxLzl9lvZemGqz5JrXkmuTM/p3NyLaMbFdrLzYdFuoi2s49dXztDPV0mJ5mB8FUhtCEY4Lr5u0+wxjlJLro6Qry7POHlDybs2O+1XNClvNuW0GySqiK7/6qvj5xuJUHKoNH/E8a49KeKpsjddS3pF7z/SZmvdLLGkEco86P3HR78zXFnWCHZYb7hcwgzP8qCdhaF186wg9TfNDuuB9J7WEVq+w42Q1EHk+CWGtHjtPaVxBb9g2fKMbSckqZK8v38mW2sf62BSQehcAqLnxv3ZFKkzmSAYHf/uMMYtHf9nnj0xsEY6HJN+HIQLhArseMmA15ia9uQs2D0cL6/r+q8mvon3IgUkJhSklFEA+fj0jbimIljGDVKUFZewRNUDqUalXdEgUznM+ai4oGa9f9QqbTw8HawLNugvalbjQI/NvoL7FgEc684yJaZiaLPaOKjSzriBJSI+xj4MIbTy0P0lqhZGzBcgC7a8LSgBoGc0UNkUlXsbEmzkOkOhMvaeRAs9U5WC4agDWARbIGh2bJaQ4pUt3cORaJbiWCm1Jb76J9/yHwracaFo7FLHp7hw1CU7z76SKJJspVdpZAYCnwIzCayRjCvZj7yJxJ+MMXp+s0YBsZ9ojuJ0NpVc1GNJlueSN74jvEwhR05XgMvUf2V6Bb1yTLJvA40Z26wIwyEYD1wkqgWDLdAbQt1FM5w3nB5WsTKGWnuOkmApSV8JQAQSapsKwGYSZqZwgFF8mBC3+mi1iEL7Fhxfb0DT4mdY7PDUSxi+F1e2562XlzF94nRE2aIseYNqiNxYlL1t9XvgIvruDknFx5YqyVO+jnpLt5VA5sHIOlQ4oKAaZ4yYb4Gpe/dURxPmXQd C+ohGfvq WxpGGOcK2KSKK+eegiSwV8OHyD6BU6qBG3vFLAlV7vyFOeTcSnbQr2vCqeE861AzAzWOKzBy7Sza5cKcC5xYWHIwAikwVgrdfkGF3XMb1lu6YT52D6UU1xgtmjU1NDAl2YQTtP2SImQUoXWYO8KNfP3mAI//2lHY2pMmRGpE9L5y5fwJskhFv4UyrApou9U1lNLWjRBjX3wgKi4th9PeJ5h8SJEpgeCnOkqDcHVRzHAFkTeDbccIYfWuXhA7P68HTikSYGGPMIpCINma8/YWSjxrV8KrAKbpmphxwBrA6Y+YK51+lxK/aYFvaoc5dVlcZEiN4QcMuK2Ar4hPBfVAb2ytBKIL+BzT1cng8KhGlTYLxq3bZoSlD/FibhMDTGEsDy7qIGCsdGCEx/EpcrT1MpCmXtsRkFeaiyywNSV1hDAzlsgAuJBG7ft7kyw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mmap_write_trylock() and vma_try_start_write() were added just for khugepaged, but now it has no use for them: delete. Signed-off-by: Hugh Dickins --- include/linux/mm.h | 17 ----------------- include/linux/mmap_lock.h | 10 ---------- 2 files changed, 27 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3c2e56980853..9b24f8fbf899 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -690,21 +690,6 @@ static inline void vma_start_write(struct vm_area_struct *vma) up_write(&vma->vm_lock->lock); } -static inline bool vma_try_start_write(struct vm_area_struct *vma) -{ - int mm_lock_seq; - - if (__is_vma_write_locked(vma, &mm_lock_seq)) - return true; - - if (!down_write_trylock(&vma->vm_lock->lock)) - return false; - - vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->vm_lock->lock); - return true; -} - static inline void vma_assert_write_locked(struct vm_area_struct *vma) { int mm_lock_seq; @@ -730,8 +715,6 @@ static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} -static inline bool vma_try_start_write(struct vm_area_struct *vma) - { return true; } static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index aab8f1b28d26..d1191f02c7fa 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -112,16 +112,6 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) return ret; } -static inline bool mmap_write_trylock(struct mm_struct *mm) -{ - bool ret; - - __mmap_lock_trace_start_locking(mm, true); - ret = down_write_trylock(&mm->mmap_lock) != 0; - __mmap_lock_trace_acquire_returned(mm, true, ret); - return ret; -} - static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true);