From patchwork Mon May 22 05:12:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE46C7EE23 for ; Mon, 22 May 2023 05:12:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2B426B0074; Mon, 22 May 2023 01:12:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB23E6B0075; Mon, 22 May 2023 01:12:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D53B7900002; Mon, 22 May 2023 01:12:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BD5BC6B0074 for ; Mon, 22 May 2023 01:12:14 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 973C6ADAB8 for ; Mon, 22 May 2023 05:12:14 +0000 (UTC) X-FDA: 80816719788.06.702FA77 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf19.hostedemail.com (Postfix) with ESMTP id BD3621A0010 for ; Mon, 22 May 2023 05:12:12 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Xury70gJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732332; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=DUChmWKKslp1V9N0xCTT0FkZkipI7Bb1yjTvJT4uIPKPUgUeptDdBaaST5ZDDj4yUyElFQ t5Y9vZYeO1c8ScoK1m1iR+mnW/oYVvvzlm/2ZmoTfhs7lquxLmcuA30xW/dVd3F0MkLvtJ JyXMIMt/yNiH8G5zMVR4BfX4umi6Ro4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Xury70gJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732332; a=rsa-sha256; cv=none; b=YW2rTwC+GzwuNUnqVIXzk4Zto+l7Y01aqTXWYjpN8tGQhIUkulZvuWLKbweZ+7RNAmAx30 mfdpRlvKRpdzCwChlqmTeSSVY2gxugy2e15d5GIztUZf5PB0uT/dQDQYfhCcYjCb0SWpS4 Fija6Zvb9TvmHVVHdsuLhBKTMCqCaTA= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-561d611668eso71645947b3.0 for ; Sun, 21 May 2023 22:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732332; x=1687324332; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=Xury70gJk/Ie7HEmpym3Ju1gfiqnvb3lYbEkOk+qdASbHtYcBtM6f7yvCnD5Qi5hF5 Xqo4lAQiP6+GN4sGJVvV6KOkXQkulHhobtYfAhUe9/l/2KsHbHtH2T86CZpNKlWRl63d IzMkj8FLns70Ingn1LKVytPrfZjfNvxgPvDTj2KhwHoFQ8Kx+CJ4VG6UzypdkQqqJSvO YWMudU4lJeyeTAt+pPO1YgidnSwVn5tO3duE8/EWyOVS7vVgZwJSIiPPVnzLsKahA0qM ZlEnIC4jzz74OlvLVDlS2OOEGCjrtDJzReiTZcHJL2Ft7dpNZsKSmX1OuM4OkIMXn0/q hnaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732332; x=1687324332; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=NjxRjC7l2FGfqWeAvysRVGodEdui+8vVIINo7SnKBS+s84NwQNG2FqD2gfxv2AnakI pI5LhRAgem3EviJFYRKvSndmtBaPd+xWQobXrjOjdqJZ3PkXRHoHGJzNRiOFMaARwNj/ AUtngGuyihCelyihgt399mryVZRUSAwlVlDF051C8SSqkK5iYJIxuj2v3DPkwCE5olYc 7OKp3iNAuFEirnv6pAAP2GjVo7xXEUK6ADSdAcTVzXEVX2kUMYx3wEuldIpMTiSRL2za zS0S1uy2wCakzg8B9bb6z/e1/YSF2db5//wBGi884qjtHmCpvVPPzf23ciz0nmD7ERJv 6DDA== X-Gm-Message-State: AC+VfDwx3vptmnuNDayNHXj6kEJhPW/hcGreBw58Z8OtpDMSYBe3e7cQ wBS6HepgM8wyIF807zsYlO5J/A== X-Google-Smtp-Source: ACHHUZ5YOFvYflvdD4FMdNwpV45kv2pl6g6+I95eOvlzunuPKVoSDfQex4to1CKCQVwtCrNkcGYjoQ== X-Received: by 2002:a0d:d107:0:b0:561:b4e3:5fc8 with SMTP id t7-20020a0dd107000000b00561b4e35fc8mr9743269ywd.37.1684732331752; Sun, 21 May 2023 22:12:11 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id o80-20020a0dcc53000000b00559f03541c6sm1814009ywd.132.2023.05.21.22.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:12:11 -0700 (PDT) Date: Sun, 21 May 2023 22:12:08 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 18/31] mm/mprotect: delete pmd_none_or_clear_bad_unless_trans_huge() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <4a834932-9064-9ed7-3cd1-99466f549486@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BD3621A0010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qhzkj1abxce98ae6n837wjm4goyhxqna X-HE-Tag: 1684732332-143198 X-HE-Meta: U2FsdGVkX18u1LSdiZqb6nZ5hEIr4xADKULHAjGHyBlAHDkJE1fNKYcf3Cs4Bllq+zuCrwGFrW5+7TrTNuSHyzAxgWfHkm1rlV2f7Q9HW4ZmZtuuuH03y68a/vfWqHm1zTEo0nRh2xU4zRh+s+1JxohrM1lvNL18J6GWljUi2K2bgmRi0XSSTi6X18YDYawx0BQZ5ZVCZdvxePBMpu8/CHZSu+pCOYQrX9rKV4hH0y5K3LaXf//EkGUABeELYHyZQ0M7UZT0QRCqmPt6HIs0PJ4FGxY2kWiM08Xttc0GFKExAAlNhUpcZGmnBL05xRoAcffqJTb01m2Cab6shhXKFF2jE1W4lrWbDsYM4nQcNZh7RI8mXRMt5ahSh3cdM/6rDHTdXyxPdLDIDQHNzUbuZ2FxDkqIFbcfAWgTHP06irnr7usiKgK2NCj3xJbSetOA1K/ukiGa9aVKQfyb/Tl8ZVXffSwDw/IZWJW31atrBunU2Bm8a+oIkCwnU/bv9OinzCLGjC3b6vMuu1nH0CwI5e4/1QGvYr3wJ33csXkcZcNpwgkW2lnsCu+blDUlzDBqf2gulvHfXtrI7djErZPiHzRplsOeNpLOj3jEl087zwu4zS2ISbSzTg98YyYXZR4h5N+Oe1EeETHDpmFTOK1Mb5ovjB7hkmukijCDTts+Rxl2Wp6IKuRWoUNLvqPfSRP/o2PfKZp/H0/LG2rOQakPiSGCMJLZEkZ0Hx09fFDQZ4GYAJB3NUg9IRLROoD28w3FEhSNn4lPrY3Jm+L1pDF9TkOcdFz5rIuvTMu//dlBFJSdSqr4kAvG1FhrEhbI1hir+AEaagb6+t873QNvK9nB3Y5o6q6X9+7QcbUv+gbZN+ZAGDqpakDM5vJi9JRIUWmVG/xUnT5MilO36JX7s9952JsyofglpByoM2zNm4ve2z4FYeJgDMlcqijcABvFFO6DpOtIzOOzNTXn9W52Fx6 5g66K+cF vUpiWkWY4mAc3L0hcV/7LxUG+3cNh+3PjyDDDPsi2OVDU6mCtUwvt2X2m+Oz6blD1S0kE8mCjDg1nqfNXgA2+lQslhPIShdisBc3axfbzPmmcLCVUaoYDPYmGUPkUXlM7XYWytCSUX9u6aZMYL/KByqNS7ZMc6sJ0M/HzXrUeW95OkF2X2Yq2psKudVI+LqhWstA+jAozBK+KmiFtwF7uE8VSeE0B4ymp9C6QHBSiywLsD2pM7oCH3+hcKCaz71RiWzVn8acMt380NOEEm04BTlNOr35hHMA9wrHHqu3YZyKP3Js9srOb10kn1Y/ytu9Dnx2kYx68S8/YVFnlULG/t8yRRe1QZkpEMGNJ7ahcbWeiRP8hn4SqTtH8wj8rI6hKu5hHoimYVJu8uxtg2+dQmIGdDqeBAyyI0vmQoNBr0T50xdCjK4OiXONp3g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: change_pmd_range() had special pmd_none_or_clear_bad_unless_trans_huge(), required to avoid "bad" choices when setting automatic NUMA hinting under mmap_read_lock(); but most of that is already covered in pte_offset_map() now. change_pmd_range() just wants a pmd_none() check before wasting time on MMU notifiers, then checks on the read-once _pmd value to work out what's needed for huge cases. If change_pte_range() returns -EAGAIN to retry if pte_offset_map_lock() fails, nothing more special is needed. Signed-off-by: Hugh Dickins --- mm/mprotect.c | 74 ++++++++++++--------------------------------------- 1 file changed, 17 insertions(+), 57 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index c5a13c0f1017..64e1df0af514 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -93,22 +93,9 @@ static long change_pte_range(struct mmu_gather *tlb, bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; tlb_change_page_size(tlb, PAGE_SIZE); - - /* - * Can be called with only the mmap_lock for reading by - * prot_numa so we must check the pmd isn't constantly - * changing from under us from pmd_none to pmd_trans_huge - * and/or the other way around. - */ - if (pmd_trans_unstable(pmd)) - return 0; - - /* - * The pmd points to a regular pte so the pmd can't change - * from under us even if the mmap_lock is only hold for - * reading. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return -EAGAIN; /* Get target node for single threaded private VMAs */ if (prot_numa && !(vma->vm_flags & VM_SHARED) && @@ -301,26 +288,6 @@ static long change_pte_range(struct mmu_gather *tlb, return pages; } -/* - * Used when setting automatic NUMA hinting protection where it is - * critical that a numa hinting PMD is not confused with a bad PMD. - */ -static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) -{ - pmd_t pmdval = pmdp_get_lockless(pmd); - - if (pmd_none(pmdval)) - return 1; - if (pmd_trans_huge(pmdval)) - return 0; - if (unlikely(pmd_bad(pmdval))) { - pmd_clear_bad(pmd); - return 1; - } - - return 0; -} - /* * Return true if we want to split THPs into PTE mappings in change * protection procedure, false otherwise. @@ -398,7 +365,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd = pmd_offset(pud, addr); do { long ret; - + pmd_t _pmd; +again: next = pmd_addr_end(addr, end); ret = change_pmd_prepare(vma, pmd, cp_flags); @@ -406,16 +374,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pages = ret; break; } - /* - * Automatic NUMA balancing walks the tables with mmap_lock - * held for read. It's possible a parallel update to occur - * between pmd_trans_huge() and a pmd_none_or_clear_bad() - * check leading to a false positive and clearing. - * Hence, it's necessary to atomically read the PMD value - * for all the checks. - */ - if (!is_swap_pmd(*pmd) && !pmd_devmap(*pmd) && - pmd_none_or_clear_bad_unless_trans_huge(pmd)) + + if (pmd_none(*pmd)) goto next; /* invoke the mmu notifier if the pmd is populated */ @@ -426,7 +386,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, mmu_notifier_invalidate_range_start(&range); } - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { + _pmd = pmdp_get_lockless(pmd); + if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || pgtable_split_needed(vma, cp_flags)) { __split_huge_pmd(vma, pmd, addr, false, NULL); @@ -441,15 +402,10 @@ static inline long change_pmd_range(struct mmu_gather *tlb, break; } } else { - /* - * change_huge_pmd() does not defer TLB flushes, - * so no need to propagate the tlb argument. - */ - int nr_ptes = change_huge_pmd(tlb, vma, pmd, + ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); - - if (nr_ptes) { - if (nr_ptes == HPAGE_PMD_NR) { + if (ret) { + if (ret == HPAGE_PMD_NR) { pages += HPAGE_PMD_NR; nr_huge_updates++; } @@ -460,8 +416,12 @@ static inline long change_pmd_range(struct mmu_gather *tlb, } /* fall through, the trans huge pmd just split */ } - pages += change_pte_range(tlb, vma, pmd, addr, next, - newprot, cp_flags); + + ret = change_pte_range(tlb, vma, pmd, addr, next, newprot, + cp_flags); + if (ret < 0) + goto again; + pages += ret; next: cond_resched(); } while (pmd++, addr = next, addr != end);