From patchwork Tue Oct 3 09:15:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7BA7E7543A for ; Tue, 3 Oct 2023 09:15:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AB766B013A; Tue, 3 Oct 2023 05:15:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 334A76B013B; Tue, 3 Oct 2023 05:15:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AEAC6B013C; Tue, 3 Oct 2023 05:15:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 068376B013A for ; Tue, 3 Oct 2023 05:15:22 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BF72EC0212 for ; Tue, 3 Oct 2023 09:15:21 +0000 (UTC) X-FDA: 81303591642.05.CD6032D Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf17.hostedemail.com (Postfix) with ESMTP id 020CE4000B for ; Tue, 3 Oct 2023 09:15:19 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hSWdFDpz; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324520; a=rsa-sha256; cv=none; b=koP1lwJCrDwQofODEn4rlCNTOwYbCm01NOBEPwPv8/pODPPVJvuH+VvuSP0LsJ5akQWOMj 4PayfesFe7aZJoMDXTGXBc8c67N11kUjhhvH2RCvUo/GQtfzT61ZOCz+kAygeoYV9p0cw0 2m8bKBrQhayB28WwE8/TkmJomVLsVfo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hSWdFDpz; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VwYZSOOB/glvz2EnW/qnimnVXLIQ+wJ0Ri+lf09J6gI=; b=G5Ls1jy0XGCr/86+W/Hcss6j+djyWFcFDOHxDl0D8ZAEgYTY/y2QTgLNTfZlcpTlNwH8DH oBnstIXH0RohrDCBXJuPG6GtdFKmktggJzV43yhGtIf7dH3k1Js8aEvewpC5hVkAgsoNab 2Ai/dLZp2B37AOufgdl3B9Z6vwiWQc8= Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-579de633419so8376157b3.3 for ; Tue, 03 Oct 2023 02:15:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324519; x=1696929319; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=VwYZSOOB/glvz2EnW/qnimnVXLIQ+wJ0Ri+lf09J6gI=; b=hSWdFDpzGJ5Q0ktLUZdTcN9z3cxmRV5BuT+OYJpNflB0qlYfyrDehgOMp3SbhbZ8JT aH22M1PdjntdRfUpVAny7NtG4/H+74/ckVEriTTLfGtb+ayaiPJbw2YvfVexR7/E3gsb 3dVKKfmG3pIUXqFrdpno/DpSqZeV7ItQfgG8zIKZtBuFwSo0C/przEZUMO1YvO4GZRJH qNAY5IbXRVRmOgpGMKDT1UUsdOEeVQLMQKuriwfwirpjT3US3MP6H9l7I05wOpZ9V/71 +JwmQQoPlpk8/5cC1eMDeGHdYpVEKI+aXXC8V4U6F5Bf2fb4LyJz19IMfpMRddcFKxSV YrDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324519; x=1696929319; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VwYZSOOB/glvz2EnW/qnimnVXLIQ+wJ0Ri+lf09J6gI=; b=TuGGWErpJ8ZtJSc12kyDmt40uo8vjDyB1oXdCmPZHmP6pbU10Vhl6bxA7gD8T4vPr4 E+2LfvZI+oJmTyWLSJ6zRfN3JmkYheIuv/d4qaiZd29tgDS4WBI3vN77OBW1KrvVUKsu dk51hNSKTwlJ10+5bOXcJ4aEDLhR7lg2mSLo4OQLVmSAAfpw7INgc3HOKq4EOTBGe3Pa CyD8YtYjV0vLj2wm+hJFoFNHN7drWW3Jb7AhZfMmXd9AHSPrSdS2e/OBzr+R/VOTp1OD E1y2H5PyAyuugx3B93ig+oOU2/nP/xqi69aHSSy0lOxXHKThbTP+lAnN8MnnKLvvz+bg yyjg== X-Gm-Message-State: AOJu0Yw7q6QAMbtoKZq4KVfUIO0SGTa4tez5SSHBnNDXxMoulfKTt6G6 DlDuwJBOWTd4bnFCmjPXMThHgQ== X-Google-Smtp-Source: AGHT+IHGzltakHVdO+OL/afC9cmnMy82Tb0Z7wP4Wxec5m1WelVAghemXtF1xjfUCVDqbZcaUXYemA== X-Received: by 2002:a81:8907:0:b0:59b:518a:639c with SMTP id z7-20020a818907000000b0059b518a639cmr14262844ywf.36.1696324518906; Tue, 03 Oct 2023 02:15:18 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m131-20020a817189000000b005950e1bbf11sm244244ywc.60.2023.10.03.02.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:15:18 -0700 (PDT) Date: Tue, 3 Oct 2023 02:15:09 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 01/12] hugetlbfs: drop shared NUMA mempolicy pretence In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 020CE4000B X-Stat-Signature: negotmq97yoaaymm3jxr48qqhnw4hzsc X-Rspam-User: X-HE-Tag: 1696324519-491621 X-HE-Meta: U2FsdGVkX19t032c8jrnNmGXDD6J8oXV2Jy3DI4VyjmE6IrAq+HwK2eQUK1DFOPpFbnv4O/HbD+xpaaD4rg8HnWEvp9aulDatkbboH7srR9PFKrGdLnl3y35I0OzVpQJ6JMg0pbl6SYJIi5ZME2XE15D/dU0AX+2eFyMFmbl1aRpe7CnVIOsS4iI4RbjwKSIzPJUtlGcZ+UTC9rqsUaKa4I2Kp6mNZDCRkTEcjMnP/eIRgwcJWmEu7E9GfKASIaTpPw9PhtHs5imB8GNMTJK4XfpJaj0gaQkW8WSXC3ZtLN3CDNTfZMNfN920hBbyjEpnLp1Cs9tgNI8m5pJKvl+Quixl/AnIHogv0JkUXdQI9giMc+MrHa2U/VoG5Sg6FC0PTUJWvP6XS7mPNtx6BMojINWuZVTwyX6SEgw8Js1ih8Z7iv9HANuSQ3BYYxVmzK4xHHpyVtfnGdNBMC0tR76wMSLfZfkvN0YbWtGuFB1pqWEkYPmtWf7OSFWz9mJ0q7+NbkCl6nEbVOUl5Pt/U6Ajvw+PR8szsj70tmLYqHpSs44lfFW7WLlVKQVFtT6VH4SbESDTMsYLKXow4mo0jzZthZVArYmqwO/eYdIK1yZYv4M97ivzU2xHeMIig4Cf8T+8K1+7QvrmKdRsIa3ovYCcEU4IYzyi1mbZFnAtX4eDz8jPddRaQf/FZ1b9EQOzi+YHgl9fvig1yhHhVcTWUh0pm+Nv38CHs0U7Q7Qd3Fk61lYrvidaUhadbzCwi6HPDZkNnJzA4u+IdW2MLwn803OztDBBWXoYWog23KnTVcvh8+dPWlU+4u3c8OfToLZi8xBW5AuK2bzW7UVTqWoEYM0bZT2S2O7dflppft6RlgGyndM1WcvRLhf5JALKwHknvZDT5yyyanOlQsu4ALV3iZDEtzCov4PRliGb0qnEnL8lbo7ih8Nfun11yXYSvmdj93jOf5C1bFUzTHI0gn0bML /xD2z0kp 0ID1FcOXh9NxKIxFrXCYIMNfrs//E6DisH6t5mJ0uSQx6VjspjTq+O/VUHoykScakpvZN3DeX0Ga7x5uWLLdlewIeFlZa3f5JkD7I4Mg6ki1LjnkA3YOexkQE13gX4wJNbYhva/TBIBtsA3TxBSsfjDxHLlhsdctvYBFZFcBsGC6EpiHXgmGy6DODOdsrUix4T7Py5gyMNPjVYmwSWnNnObrG9pnUGZ8pSHtSh/STPGaS+IMO5IuLutYjJPR1lUmtDdY3aan//+vMKeztoN4Oe5m1XyUHtZ+BoLN0pr2N8rKHzhnhzQRn05TUngG6tN8IvYku1Yaw5OOTFpSkMXuX3ByUWcPgU3mhj+z/+VeephWEV1vS3+1mz0m28jOjiSheTetv3n8ECDi5nRm+UgerIoxfRlt08P+KWNgZZfgl+srqlrvYVc7IlETI+tfUMSdAOvfEXU9jatXAUkqlXNoJmjcCxF3yXQyiLM4UqZ/23OypwE6YKVglXM3rLNSjKYDpXKycXxmhR4TVUVs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hugetlbfs_fallocate() goes through the motions of pasting a shared NUMA mempolicy onto its pseudo-vma, but how could there ever be a shared NUMA mempolicy for this file? hugetlb_vm_ops has never offered a set_policy method, and hugetlbfs_parse_param() has never supported any mpol options for a mount-wide default policy. It's just an illusion: clean it away so as not to confuse others, giving us more freedom to adjust shmem's set_policy/get_policy implementation. But hugetlbfs_inode_info is still required, just to accommodate seals. Yes, shared NUMA mempolicy support could be added to hugetlbfs, with a set_policy method and/or mpol mount option (Andi's first posting did include an admitted-unsatisfactory hugetlb_set_policy()); but it seems that nobody has bothered to add that in the nineteen years since v2.6.7 made it possible, and there is at least one company that has invested enough into hugetlbfs, that I guess they have learnt well enough how to manage its NUMA, without needing shared mempolicy. Remove linux/mempolicy.h from linux/hugetlb.h: include linux/pagemap.h in its place, because hugetlb.h's recently added use of filemap_lock_folio() requires that (although most .configs and .c's get it in some other way). Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) --- fs/hugetlbfs/inode.c | 41 +---------------------------------------- include/linux/hugetlb.h | 3 +-- 2 files changed, 2 insertions(+), 42 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 926d01c493fb..0586c90cb9a5 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -83,29 +83,6 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = { {} }; -#ifdef CONFIG_NUMA -static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, - struct inode *inode, pgoff_t index) -{ - vma->vm_policy = mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy, - index); -} - -static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma) -{ - mpol_cond_put(vma->vm_policy); -} -#else -static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, - struct inode *inode, pgoff_t index) -{ -} - -static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma) -{ -} -#endif - /* * Mask used when checking the page offset value passed in via system * calls. This value will be converted to a loff_t which is signed. @@ -853,8 +830,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, /* * Initialize a pseudo vma as this is required by the huge page - * allocation routines. If NUMA is configured, use page index - * as input to create an allocation policy. + * allocation routines. */ vma_init(&pseudo_vma, mm); vm_flags_init(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED); @@ -902,9 +878,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, * folios in these areas, we need to consume the reserves * to keep reservation accounting consistent. */ - hugetlb_set_vma_policy(&pseudo_vma, inode, index); folio = alloc_hugetlb_folio(&pseudo_vma, addr, 0); - hugetlb_drop_vma_policy(&pseudo_vma); if (IS_ERR(folio)) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); error = PTR_ERR(folio); @@ -1283,18 +1257,6 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) hugetlbfs_inc_free_inodes(sbinfo); return NULL; } - - /* - * Any time after allocation, hugetlbfs_destroy_inode can be called - * for the inode. mpol_free_shared_policy is unconditionally called - * as part of hugetlbfs_destroy_inode. So, initialize policy here - * in case of a quick call to destroy. - * - * Note that the policy is initialized even if we are creating a - * private inode. This simplifies hugetlbfs_destroy_inode. - */ - mpol_shared_policy_init(&p->policy, NULL); - return &p->vfs_inode; } @@ -1306,7 +1268,6 @@ static void hugetlbfs_free_inode(struct inode *inode) static void hugetlbfs_destroy_inode(struct inode *inode) { hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb)); - mpol_free_shared_policy(&HUGETLBFS_I(inode)->policy); } static const struct address_space_operations hugetlbfs_aops = { diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3c4427a2396d..a574e26e18a2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -30,7 +30,7 @@ void free_huge_folio(struct folio *folio); #ifdef CONFIG_HUGETLB_PAGE -#include +#include #include #include @@ -513,7 +513,6 @@ static inline struct hugetlbfs_sb_info *HUGETLBFS_SB(struct super_block *sb) } struct hugetlbfs_inode_info { - struct shared_policy policy; struct inode vfs_inode; unsigned int seals; }; From patchwork Tue Oct 3 09:16:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3C83E75438 for ; Tue, 3 Oct 2023 09:16:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 384C26B013C; Tue, 3 Oct 2023 05:16:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 335116B013D; Tue, 3 Oct 2023 05:16:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AFC36B013E; Tue, 3 Oct 2023 05:16:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 08C9E6B013C for ; Tue, 3 Oct 2023 05:16:35 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D54A01C9FC9 for ; Tue, 3 Oct 2023 09:16:34 +0000 (UTC) X-FDA: 81303594708.09.609BA14 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by imf06.hostedemail.com (Postfix) with ESMTP id E9506180019 for ; Tue, 3 Oct 2023 09:16:32 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=s3VgY5Zq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6szECgvu3nTPHPzEk6DBfoYWo45M6R50iawIs9FUBJg=; b=r9R+Ap/8UTyR9uYEwp0AmWBlZB54Srnxn5CEVpRhGnh0hRfuL3QcdVRJI9qpFJHAWj61YY /fRM2PMMcuoTgBEKw/6s/3/uDgdZ4x0XGNghFOTBAozN5ppQhoVnqVnFwcfHtv4vUUgzih Q0MGEWUYupxpzYjadgIRbsPPw65/a4I= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=s3VgY5Zq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324593; a=rsa-sha256; cv=none; b=NlsBHCwkDaqK3kGDQXTV1uC+fQegbQFjRCpRTA2QoHfestJKMEdxL6rSM7Wl9WP/I5Mfel GdMMVWoshj3Zy6OqaKv/dstxhUKRxs31+fvR9ESDzWO6WcG/oDgLaLMEokTKeYzHQ1yKmQ 8apzy+5Ii/p03B4yZnEHzXxSNDwzNag= Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-5a1d0fee86aso8520747b3.2 for ; Tue, 03 Oct 2023 02:16:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324592; x=1696929392; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=6szECgvu3nTPHPzEk6DBfoYWo45M6R50iawIs9FUBJg=; b=s3VgY5ZqTO2z2Phmw0XWlCPr7NFvck9vevctQ0iLVj4YFObAlvnKFs9TTDfdPOILSC UcIFkcDLWSQpNerm8mC7e3FGcdnllk9MW+ETzCgnM4IkN5nApW2qGOqRBzJkDfJBMmOi OFLR+bzqnZ/Ky5rtJ8PnYpj446+gRCnUCRMwE+gft3wUZn36xrKEzA3TNCgNK2NTeH1D PgOEZnym3msfFwxAHi6gCV51VijnOFmAxAS89APlmtQcoTKglr0PXsGJtPUXmZY1HoG3 0gg3jGQRelC6AZ1K540SPzgci9K4DnUoWFeGrqFldJIvZeNmhOywNOTSIZEk89Wo7E34 TbAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324592; x=1696929392; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6szECgvu3nTPHPzEk6DBfoYWo45M6R50iawIs9FUBJg=; b=uWcmf0Rq8RHJk48G5KCZXyp+Fr9EAxQ/lOMYM6XsgKOxqoFKQhtWkjHv/iyFgv1wYP 1AWvQDBI77r6qjG7M+gjJj2vghFzYOx+7eHB72Cbtvpo9TF/45qzgDatq7daTeXBl6B4 mZ+VyTeAG0ORjnCFrE38OR9HjXeCKS03Q4xdXQ/Qqs3+ltNtY/3dr2XO6SLRTIpb7Zn2 FM/ZSWqb8lW+vgkbWrcUqVgYSdSZkFWs3i1DNTFfJWqTEYhiZmIhbAdc+A99xDCeCPWx bLmzRHtsXhX2IohUhgi5O8DOy/hTSr/T1dW6sEsc/Rw7emLS8/Aw2nU6WCJleCJxcGuh teJw== X-Gm-Message-State: AOJu0YxDGtBwZaJYXwYnf3l7Cj54x9achyF0uF14kRpYmT/vgP3QZ7EU wDlza5rDvjhkz6pxgWB3QISu2w== X-Google-Smtp-Source: AGHT+IF7TKqrMTfEG37/bwFMCSN9pzNnHHphoLkIAInKebAgyaWY8C3y19zZ2lbUvYlV8i48+gNb6Q== X-Received: by 2002:a0d:ee46:0:b0:5a1:635e:e68 with SMTP id x67-20020a0dee46000000b005a1635e0e68mr11991552ywe.46.1696324591927; Tue, 03 Oct 2023 02:16:31 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m14-20020a819c0e000000b00583f8f41cb8sm244237ywa.63.2023.10.03.02.16.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:16:31 -0700 (PDT) Date: Tue, 3 Oct 2023 02:16:29 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 02/12] kernfs: drop shared NUMA mempolicy hooks In-Reply-To: Message-ID: <302164-a760-4a9e-879b-6870c9b4013@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E9506180019 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 79eaz9i553i65zintx3oafd4cy1xrzoc X-HE-Tag: 1696324592-240653 X-HE-Meta: U2FsdGVkX1/Ri4QfYfsNfbuldfxuNTX+g5PyNpRBdj3WKsfkZF0+GgtLzgY/KxX9RlMgxg0UF6BtmZaKnGytjNDx1vpJTrLg++Wn33K4aEJHvjFNktz6oS+M53vE6FEegtX65WvWdonBDDynkwCC49jxZbgK8cDa3QNnsUAYdFrux33x8B5E1CNBOnRogYWw4yw+SJC0uFiYMaE5yVT+ghGF+m1IZmAfhwd0M54WQxvHZLHMOWPkNMPLl2QYQedtbYiRq7/1eYM9Kc7yRc7ot8BQehFxh0LoR+RrabJ+nZjKNKsRpJNPBQkxfeyi1QlFI6CPm3Vqnn2l0xDc7Xeh2+CIOZL0WNQCm26mg3NIXobOUNnBkjZDhS+OHL0bg2xpOI5ICaab/DOhq0Ivq0Ma89dfXg7C0CZvOXxDRn2j9K7tkrfp2PYiWVxAvw3uoPRENDhoTYJ4aZAQ2QQhUsh/PVcRVObeARmRQplvBAT4paC9jsQjX4ZnReBiBz/+DWnFLKK++lokOyzdcRtWSzrcYOF1/CiWNjBQtPgn1MUkL68EMmlRYOM+E2fck+Zh7MHlbhbte8pCDQk8Z7qDhT8xpDcwolIJZBkkhmkWy++FJ1ExQ2fCPMa2uJa19AXVM1wyWQDz30ZgnQONw6Ya866NZAtasZhyowDREDqMmboT0gzZR9qmfcNluai6YkxG+XnO0uCTEyU2fVavzRTkA3tZ/ZSJRvSDtk2JZmIkkaXYEdv2PCUG94ZOinvLdvcHGJRu4RTMJMkdlHdtPPjYOlKF/uBMT+f1WXQcDydY6Z/QMCcHXyrYu13YV3VeHebWyG62tBh1/cieaAYEX00OAbVKVFIgk8LWPMEl8ii/f3QpUe4Gv3sPw+TvGMd7ch/cc7ysxa5x1EnT3gULnWsgBsjLGCYdWz1xkegxcYu/acWLcaVyRG6z0iPI8ps5/SVMeUjHPLtxWNAszpThduHqpND YERQqL5w Pq7xgZo+4vVCSkjCpmgrhayEfMhsi2HaFVf39LTD8f7b0e0LJFqt/ZzoxZ9SVvpTCrk7SQkwwhOymvts15EGmAsRHbOxT5TfDtQ4GdakPo2IEn8pSDUuKtc1+9HJoaDiTl7fWGBUVY0XgDHo6Xe3beXLshkmB0igd5UXeSubSQ8INTrOnTboDde8OBGGzxYVHwDZEsU6oXz+N8wOpBIa4FzczWLWhvESdtjAPu53uLNFhqtsp/iZcWIkWurtIGCL/ONjytWFbSv4Lzl/CB3VhrJr2Kw4ThKoj04l2iFFQefmzmRpIooBEeZYdCt7PkVx1AQLD5cDhTa+/SvLEp4kVjf+qByEMu+VopLNNcXBklwoUwc2djvOXPQfmiknTJw9YyiD//G9t0/xIcrkV8AGs8H8VjZLg87dytuaT7ZU57rXBzAMmLylYAbg9aK2LjDYp3xejnDDsExWDCpIbVIVZGHl48YcH0lHKlsvb3SFI9C6OXgKNyJH0ke9H2+DoXOaqxGVv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It seems strange that kernfs should be an outlier with a set_policy and get_policy in its kernfs_vm_ops. Ah, it dates back to v2.6.30's commit 095160aee954 ("sysfs: fix some bin_vm_ops errors"), when I had crashed on powerpc's pci_mmap_legacy_page_range() fallback to shmem_zero_setup(). Well, that was commendably thorough, to give sysfs-bin a set_policy and get_policy, just to avoid the way it was coded resulting in EINVAL from mmap when CONFIG_NUMA; but somehow feels a bit over-the-top to me now. It's easier to say that nobody should expect to manage a shmem object's shared NUMA mempolicy via some kernfs backdoor to that object: delete that code (and there's no longer an EINVAL from mmap in the NUMA case). This then leaves set_policy/get_policy as implemented only by shmem - though importantly also by SysV SHM, which has to interface with shmem which implements them, and with SHM_HUGETLB which does not. Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) --- fs/kernfs/file.c | 49 ------------------------------------------------- 1 file changed, 49 deletions(-) diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c index 180906c36f51..aaa76410e550 100644 --- a/fs/kernfs/file.c +++ b/fs/kernfs/file.c @@ -429,60 +429,11 @@ static int kernfs_vma_access(struct vm_area_struct *vma, unsigned long addr, return ret; } -#ifdef CONFIG_NUMA -static int kernfs_vma_set_policy(struct vm_area_struct *vma, - struct mempolicy *new) -{ - struct file *file = vma->vm_file; - struct kernfs_open_file *of = kernfs_of(file); - int ret; - - if (!of->vm_ops) - return 0; - - if (!kernfs_get_active(of->kn)) - return -EINVAL; - - ret = 0; - if (of->vm_ops->set_policy) - ret = of->vm_ops->set_policy(vma, new); - - kernfs_put_active(of->kn); - return ret; -} - -static struct mempolicy *kernfs_vma_get_policy(struct vm_area_struct *vma, - unsigned long addr) -{ - struct file *file = vma->vm_file; - struct kernfs_open_file *of = kernfs_of(file); - struct mempolicy *pol; - - if (!of->vm_ops) - return vma->vm_policy; - - if (!kernfs_get_active(of->kn)) - return vma->vm_policy; - - pol = vma->vm_policy; - if (of->vm_ops->get_policy) - pol = of->vm_ops->get_policy(vma, addr); - - kernfs_put_active(of->kn); - return pol; -} - -#endif - static const struct vm_operations_struct kernfs_vm_ops = { .open = kernfs_vma_open, .fault = kernfs_vma_fault, .page_mkwrite = kernfs_vma_page_mkwrite, .access = kernfs_vma_access, -#ifdef CONFIG_NUMA - .set_policy = kernfs_vma_set_policy, - .get_policy = kernfs_vma_get_policy, -#endif }; static int kernfs_fop_mmap(struct file *file, struct vm_area_struct *vma) From patchwork Tue Oct 3 09:17:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21E50E7543A for ; Tue, 3 Oct 2023 09:17:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD8766B013E; Tue, 3 Oct 2023 05:17:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A89A46B013F; Tue, 3 Oct 2023 05:17:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 904326B0141; Tue, 3 Oct 2023 05:17:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7A6D06B013E for ; Tue, 3 Oct 2023 05:17:49 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2601F140208 for ; Tue, 3 Oct 2023 09:17:49 +0000 (UTC) X-FDA: 81303597858.02.0132CA8 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf13.hostedemail.com (Postfix) with ESMTP id 57F9C2000F for ; Tue, 3 Oct 2023 09:17:47 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4DXkl7Wa; spf=pass (imf13.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324667; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zn2NsWuFnhQFrO/gKSp+dM0OPa5MQSJrmStGl/lqBlA=; b=7gBbJb7r6OlgitRZeSfcwlG/72DmrLmrMv+aFvCtJVGi+jFh+AiO2EaftqqoRYSsg68HVl pKYRuwMV4ust27OUEmb8j3klaBPXMjGCqd3ZtctT2DbZi6PeV3QgTLV1X4ZcZAUuR9VvSO 2/atvYDZwS1quX8PC3zYbGfpGkazsr8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4DXkl7Wa; spf=pass (imf13.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324667; a=rsa-sha256; cv=none; b=2CCA4S71dwtkM2YIUZ6/TkPGg4K5+srmEwwK/yUtjm3eJ1SnivHtK0Z5EWdvEZ+s4iM5OS vKazCRXijkecQSy8IFONSRw5mtW6sTCurwcQl9ZJD0jVvCBjLxpl7fJFQDW0fMM5qfq576 hZKk1O9IwhoVvx9pR09AXJX+UO4Bw6c= Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-d84f18e908aso713398276.1 for ; Tue, 03 Oct 2023 02:17:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324666; x=1696929466; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Zn2NsWuFnhQFrO/gKSp+dM0OPa5MQSJrmStGl/lqBlA=; b=4DXkl7WaE0UDelaSItCvCNDDZLNnee3Fc6QKM91VK1P9iq7gULv8XRd8yj0jyRJE2H Nyx0KnMdbJq6oUN6WJLpi/PlHvvq+uMiKZJCPrQSiFCkoUss4uSE44iAnfDf4Fybr9+G wMM+4a/wBrGNEJKT1/+w3oAPUZJi7Z7+dUw2ldPqyagXCuSTS6hFMJ/7941fXtoRJ7lQ Ld2a12LF9psnY0KDZqgVOmg7vq1OABx+bgLGbKp68S5P+rYFQ13txEzJagaZapoC80wx 8pSIeoab19AaetRS1Ug1FEIXeAUj4XA6i1WYQXgOA3VsXcCuNRYAP1scCsyLypqjdTFn HHhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324666; x=1696929466; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zn2NsWuFnhQFrO/gKSp+dM0OPa5MQSJrmStGl/lqBlA=; b=VoC5lh6FtbF9sUljJfm8iEYgHfz0Xg8JhMeneqVRU3dGgbv4HtUPCDN76/llenivDS 9DjZ2OrWnk534STgMmV99lIlG0GAxyGqEfBwYmYmFDEtfjS+u43zLRMY1+yt+ahax2OQ CUO55rDXbv17c5udHFnCN2jBjymVrLEK1ghIDQF22l/J4MbqDmZHo1M1vL7t6SkbGeiA HTAWygjWVorcPCK6oypZcRH7QLQyukog32VSlPQV67lXc6RGuVjqZRpjWYAPLP0YTzKK PRbvmMHKhgOweVWa3eXcafwLBtO4rHRpsbJ1+x1+q2agep1hlZ+8ieHrmaMICV/u1/Cr anTQ== X-Gm-Message-State: AOJu0YylYEBQI3YpVsH/rEr/LIGJLm+eksSJFNtgwLNIhtvSOeaAEFLP CHYpHwsW1YWzbUQdi3rGhDgymA== X-Google-Smtp-Source: AGHT+IE3Svm4U0UI94crkF3Xvc7SgRRIcjGyQQz387KOabSZ66gJRpb7rp90OkiPibu4jPkgupThZg== X-Received: by 2002:a25:b10f:0:b0:d7e:dd21:9b16 with SMTP id g15-20020a25b10f000000b00d7edd219b16mr12705693ybj.8.1696324666069; Tue, 03 Oct 2023 02:17:46 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a9-20020a259389000000b00d8128f9a46bsm279851ybm.37.2023.10.03.02.17.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:17:45 -0700 (PDT) Date: Tue, 3 Oct 2023 02:17:43 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 03/12] mempolicy: fix migrate_pages(2) syscall return nr_failed In-Reply-To: Message-ID: <9a6b0b9-3bb-dbef-8adf-efab4397b8d@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 57F9C2000F X-Rspam-User: X-Stat-Signature: dc11967zeuykfaucigp6ou7fm9nbco7c X-Rspamd-Server: rspam01 X-HE-Tag: 1696324667-725459 X-HE-Meta: U2FsdGVkX191SnojNGe+RUeiH89GGAo8G3dH0Gx/soEWXmd8KdrSNE8TARdkh2GuJoc/NTMp2MCrltGkyoqtLGA/R0FwcGc2wYgP+ozcTY/djj754ubQdMEy8F8W8a4cC6xP80YpxMDcbsOii9d1azGy1mfphFNuh9P1SZ8sGPCDfTHPO0DPwizAwPLham82VoA9yY7WLL7OQloSp8bHp/aMwHZFFkgXEDi4ZW8U/gdz7uxknnvHXXY23GbpBMFaoxDtgx+BcFZk6hWQFJZfRyg61XKOiqH8MkdFZijpdgF5XE4Mhli3oMsFWalmrbW+qA3JoU3wO0ytpo3zZhhToKbnc86FEdzHCESCDFgV8EzAkkPJHFwU6h5F1BCzWcDab+aL6X8tbFyEwPg36n5OGtV/sDrIemkjCppF75vyR130Uav/5vCRMfHOJbEJ0mNARM/HRMG4eSvMJnTeUfzNl+WvuSAWeFWIcLPEja0DLXpy78hTt1YRgbG9KSjuPdGToOLvJQOkDZ7qt1d9l0zvQutDtMDzEzTjJ9Naht++5a9AHYqC5Z2qPs5FpEP/QW9FEth0g6p/SxeByWRQXCvUSynrNAv/oR2QeZ3AbXlp0rrRsO3rPwjAVZN04OXdEWS1443St+sZ4uexsH7lRg7HTBN9KTXqdU6b2yxAb5OV+dszZ1T8RPyONfH62p+CfKOm0rCRGYLKqV1VKLl1rsw1WY7M7YMOAuWLmULTnyeh8vgQZmyANpz0gx7JTKtnKrqL2NxAamoER6vcI1zC9KmJrH5qbmbgOwr0mYTPRGBrvx6RScyiZFo/cMYACiN17AURqmzMo15158gaqZGcS1uftxRWPO+dh4nciBbPQT378tuDyGTn4kyEMRF+Ryp3vBjpoKPRKGp2XCqDRyXQvXGPTfnGW7ib7QYq4f8lSe6VVBnvHq8AxVFPSXRW8odFrgqlDaYsWX4HwRgluxoHFKv LVvwBZ+o 9oyjQc/yx6UMgz21JJQ91kYeFGjV0Zdd7uQA8+478HIgs6Y58g6K0qFpjA6pjDtBTlsy9RjBbrVKD90Rd6hE+rf5DX/b0OxRSxdK4GJiNRJiKJyp613hMwe14UFVBlgTU6KRlEk6h4VbgwGOC5rXR81x8FIhrXVFO4mXzHXPclEdBCMVOYoFeKAgdQmb9eluQ1aPHtB8/ZQoKj61e7tt6RBEn37VIYp8rtG4DVhS6yDsNGt/qF022xgaTHXFPN7BBUhapTxJajOv3ARFFwMfCKKdK9rZCoHWz0+UeFrMpeIFxAV3YqrLr+WePp2VLbaY9It/s7UNFZbzRpDZHB2Zs4kqDRjn6tbiR7gTxMVnvxySk+UokihGLxEPSInPQmi37R+ae51ptyqwwSEsZJ1NpCeqn4556BHhPPDf93hYpxtFUSnXp4tfvkkDZyISyxg/0llMRAK01A58UapW31Qyjz9KPrucRfs98DVTghTjQMiKyP3H6AUkiDxP9e0V+0HbaUmxI4GdkhgxPXlujW1yZNGc3Tjd4zntWvH16gpEJcEwQYvCsJwU+q5dDcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: "man 2 migrate_pages" says "On success migrate_pages() returns the number of pages that could not be moved". Although 5.3 and 5.4 commits fixed mbind(MPOL_MF_STRICT|MPOL_MF_MOVE*) to fail with EIO when not all pages could be moved (because some could not be isolated for migration), migrate_pages(2) was left still reporting only those pages failing at the migration stage, forgetting those failing at the earlier isolation stage. Fix that by accumulating a long nr_failed count in struct queue_pages, returned by queue_pages_range() when it's not returning an error, for adding on to the nr_failed count from migrate_pages() in mm/migrate.c. A count of pages? It's more a count of folios, but changing it to pages would entail more work (also in mm/migrate.c): does not seem justified. queue_pages_range() itself should only return -EIO in the "strictly unmovable" case (STRICT without any MOVEs): in that case it's best to break out as soon as nr_failed gets set; but otherwise it should continue to isolate pages for MOVing even when nr_failed - as the mbind(2) manpage promises. There's a case when nr_failed should be incremented when it was missed: queue_folios_pte_range() and queue_folios_hugetlb() count the transient migration entries, like queue_folios_pmd() already did. And there's a case when nr_failed should not be incremented when it would have been: in meeting later PTEs of the same large folio, which can only be isolated once: fixed by recording the current large folio in struct queue_pages. Clean up the affected functions, fixing or updating many comments. Bool migrate_folio_add(), without -EIO: true if adding, or if skipping shared (but its arguable folio_estimated_sharers() heuristic left unchanged). Use MPOL_MF_WRLOCK flag to queue_pages_range(), instead of bool lock_vma. Use explicit STRICT|MOVE* flags where queue_pages_test_walk() checks for skipping, instead of hiding them behind MPOL_MF_VALID. Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: "Huang, Ying" --- mm/mempolicy.c | 342 ++++++++++++++++++++++++--------------------------- 1 file changed, 161 insertions(+), 181 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 38a47fa33ef4..752d880dcdf8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -111,7 +111,8 @@ /* Internal flags */ #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0) /* Skip checks for continuous vmas */ -#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */ +#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */ +#define MPOL_MF_WRLOCK (MPOL_MF_INTERNAL << 2) /* Write-lock walked vmas */ static struct kmem_cache *policy_cache; static struct kmem_cache *sn_cache; @@ -416,9 +417,19 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { }, }; -static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, unsigned long flags); +static bool strictly_unmovable(unsigned long flags) +{ + /* + * STRICT without MOVE flags lets do_mbind() fail immediately with -EIO + * if any misplaced page is found. + */ + return (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) == + MPOL_MF_STRICT; +} + struct queue_pages { struct list_head *pagelist; unsigned long flags; @@ -426,7 +437,8 @@ struct queue_pages { unsigned long start; unsigned long end; struct vm_area_struct *first; - bool has_unmovable; + struct folio *large; /* note last large folio encountered */ + long nr_failed; /* could not be isolated at this time */ }; /* @@ -444,61 +456,37 @@ static inline bool queue_folio_required(struct folio *folio, return node_isset(nid, *qp->nmask) == !(flags & MPOL_MF_INVERT); } -/* - * queue_folios_pmd() has three possible return values: - * 0 - folios are placed on the right node or queued successfully, or - * special page is met, i.e. zero page, or unmovable page is found - * but continue walking (indicated by queue_pages.has_unmovable). - * -EIO - is migration entry or only MPOL_MF_STRICT was specified and an - * existing folio was already on a node that does not follow the - * policy. - */ -static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, - unsigned long end, struct mm_walk *walk) - __releases(ptl) +static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) { - int ret = 0; struct folio *folio; struct queue_pages *qp = walk->private; - unsigned long flags; if (unlikely(is_pmd_migration_entry(*pmd))) { - ret = -EIO; - goto unlock; + qp->nr_failed++; + return; } folio = pfn_folio(pmd_pfn(*pmd)); if (is_huge_zero_page(&folio->page)) { walk->action = ACTION_CONTINUE; - goto unlock; + return; } if (!queue_folio_required(folio, qp)) - goto unlock; - - flags = qp->flags; - /* go to folio migration */ - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { - if (!vma_migratable(walk->vma) || - migrate_folio_add(folio, qp->pagelist, flags)) { - qp->has_unmovable = true; - goto unlock; - } - } else - ret = -EIO; -unlock: - spin_unlock(ptl); - return ret; + return; + if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || + !vma_migratable(walk->vma) || + !migrate_folio_add(folio, qp->pagelist, qp->flags)) + qp->nr_failed++; } /* - * Scan through pages checking if pages follow certain conditions, - * and move them to the pagelist if they do. + * Scan through folios, checking if they satisfy the required conditions, + * moving them from LRU to local pagelist for migration if they do (or not). * - * queue_folios_pte_range() has three possible return values: - * 0 - folios are placed on the right node or queued successfully, or - * special page is met, i.e. zero page, or unmovable page is found - * but continue walking (indicated by queue_pages.has_unmovable). - * -EIO - only MPOL_MF_STRICT was specified and an existing folio was already - * on a node that does not follow the policy. + * queue_folios_pte_range() has two possible return values: + * 0 - continue walking to scan for more, even if an existing folio on the + * wrong node could not be isolated and queued for migration. + * -EIO - only MPOL_MF_STRICT was specified, without MPOL_MF_MOVE or ..._ALL, + * and an existing folio was on a node that does not follow the policy. */ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -512,8 +500,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, spinlock_t *ptl; ptl = pmd_trans_huge_lock(pmd, vma); - if (ptl) - return queue_folios_pmd(pmd, ptl, addr, end, walk); + if (ptl) { + queue_folios_pmd(pmd, walk); + spin_unlock(ptl); + goto out; + } mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); if (!pte) { @@ -522,8 +513,13 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = ptep_get(pte); - if (!pte_present(ptent)) + if (pte_none(ptent)) continue; + if (!pte_present(ptent)) { + if (is_migration_entry(pte_to_swp_entry(ptent))) + qp->nr_failed++; + continue; + } folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; @@ -535,95 +531,87 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, continue; if (!queue_folio_required(folio, qp)) continue; - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { + if (folio_test_large(folio)) { /* - * MPOL_MF_STRICT must be specified if we get here. - * Continue walking vmas due to MPOL_MF_MOVE* flags. + * A large folio can only be isolated from LRU once, + * but may be mapped by many PTEs (and Copy-On-Write may + * intersperse PTEs of other, order 0, folios). This is + * a common case, so don't mistake it for failure (but + * there can be other cases of multi-mapped pages which + * this quick check does not help to filter out - and a + * search of the pagelist might grow to be prohibitive). + * + * migrate_pages(&pagelist) returns nr_failed folios, so + * check "large" now so that queue_pages_range() returns + * a comparable nr_failed folios. This does imply that + * if folio could not be isolated for some racy reason + * at its first PTE, later PTEs will not give it another + * chance of isolation; but keeps the accounting simple. */ - if (!vma_migratable(vma)) - qp->has_unmovable = true; - - /* - * Do not abort immediately since there may be - * temporary off LRU pages in the range. Still - * need migrate other LRU pages. - */ - if (migrate_folio_add(folio, qp->pagelist, flags)) - qp->has_unmovable = true; - } else - break; + if (folio == qp->large) + continue; + qp->large = folio; + } + if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || + !vma_migratable(vma) || + !migrate_folio_add(folio, qp->pagelist, flags)) { + qp->nr_failed++; + if (strictly_unmovable(flags)) + break; + } } pte_unmap_unlock(mapped_pte, ptl); cond_resched(); - - return addr != end ? -EIO : 0; +out: + if (qp->nr_failed && strictly_unmovable(flags)) + return -EIO; + return 0; } static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) { - int ret = 0; #ifdef CONFIG_HUGETLB_PAGE struct queue_pages *qp = walk->private; - unsigned long flags = (qp->flags & MPOL_MF_VALID); + unsigned long flags = qp->flags; struct folio *folio; spinlock_t *ptl; pte_t entry; ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte); entry = huge_ptep_get(pte); - if (!pte_present(entry)) + if (!pte_present(entry)) { + if (unlikely(is_hugetlb_entry_migration(entry))) + qp->nr_failed++; goto unlock; + } folio = pfn_folio(pte_pfn(entry)); if (!queue_folio_required(folio, qp)) goto unlock; - - if (flags == MPOL_MF_STRICT) { - /* - * STRICT alone means only detecting misplaced folio and no - * need to further check other vma. - */ - ret = -EIO; + if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || + !vma_migratable(walk->vma)) { + qp->nr_failed++; goto unlock; } - - if (!vma_migratable(walk->vma)) { - /* - * Must be STRICT with MOVE*, otherwise .test_walk() have - * stopped walking current vma. - * Detecting misplaced folio but allow migrating folios which - * have been queued. - */ - qp->has_unmovable = true; - goto unlock; - } - /* - * With MPOL_MF_MOVE, we try to migrate only unshared folios. If it - * is shared it is likely not worth migrating. + * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. + * Choosing not to migrate a shared folio is not counted as a failure. * * To check if the folio is shared, ideally we want to make sure * every page is mapped to the same process. Doing that is very - * expensive, so check the estimated mapcount of the folio instead. + * expensive, so check the estimated sharers of the folio instead. */ - if (flags & (MPOL_MF_MOVE_ALL) || - (flags & MPOL_MF_MOVE && folio_estimated_sharers(folio) == 1 && - !hugetlb_pmd_shared(pte))) { - if (!isolate_hugetlb(folio, qp->pagelist) && - (flags & MPOL_MF_STRICT)) - /* - * Failed to isolate folio but allow migrating pages - * which have been queued. - */ - qp->has_unmovable = true; - } + if ((flags & MPOL_MF_MOVE_ALL) || + (folio_estimated_sharers(folio) == 1 && !hugetlb_pmd_shared(pte))) + if (!isolate_hugetlb(folio, qp->pagelist)) + qp->nr_failed++; unlock: spin_unlock(ptl); -#else - BUG(); + if (qp->nr_failed && strictly_unmovable(flags)) + return -EIO; #endif - return ret; + return 0; } #ifdef CONFIG_NUMA_BALANCING @@ -704,8 +692,11 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, return 1; } - /* queue pages from current vma */ - if (flags & MPOL_MF_VALID) + /* + * Check page nodes, and queue pages to move, in the current vma. + * But if no moving, and no strict checking, the scan can be skipped. + */ + if (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) return 0; return 1; } @@ -727,22 +718,21 @@ static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { /* * Walk through page tables and collect pages to be migrated. * - * If pages found in a given range are on a set of nodes (determined by - * @nodes and @flags,) it's isolated and queued to the pagelist which is - * passed via @private. + * If pages found in a given range are not on the required set of @nodes, + * and migration is allowed, they are isolated and queued to @pagelist. * - * queue_pages_range() has three possible return values: - * 1 - there is unmovable page, but MPOL_MF_MOVE* & MPOL_MF_STRICT were - * specified. - * 0 - queue pages successfully or no misplaced page. - * errno - i.e. misplaced pages with MPOL_MF_STRICT specified (-EIO) or - * memory range specified by nodemask and maxnode points outside - * your accessible address space (-EFAULT) + * queue_pages_range() may return: + * 0 - all pages already on the right node, or successfully queued for moving + * (or neither strict checking nor moving requested: only range checking). + * >0 - this number of misplaced folios could not be queued for moving + * (a hugetlbfs page or a transparent huge page being counted as 1). + * -EIO - a misplaced page found, when MPOL_MF_STRICT specified without MOVEs. + * -EFAULT - a hole in the memory range, when MPOL_MF_DISCONTIG_OK unspecified. */ -static int +static long queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, nodemask_t *nodes, unsigned long flags, - struct list_head *pagelist, bool lock_vma) + struct list_head *pagelist) { int err; struct queue_pages qp = { @@ -752,20 +742,17 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .start = start, .end = end, .first = NULL, - .has_unmovable = false, }; - const struct mm_walk_ops *ops = lock_vma ? + const struct mm_walk_ops *ops = (flags & MPOL_MF_WRLOCK) ? &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops; err = walk_page_range(mm, start, end, ops, &qp); - if (qp.has_unmovable) - err = 1; if (!qp.first) /* whole range in hole */ err = -EFAULT; - return err; + return err ? : qp.nr_failed; } /* @@ -1028,16 +1015,16 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, } #ifdef CONFIG_MIGRATION -static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, unsigned long flags) { /* - * We try to migrate only unshared folios. If it is shared it - * is likely not worth migrating. + * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. + * Choosing not to migrate a shared folio is not counted as a failure. * * To check if the folio is shared, ideally we want to make sure * every page is mapped to the same process. Doing that is very - * expensive, so check the estimated mapcount of the folio instead. + * expensive, so check the estimated sharers of the folio instead. */ if ((flags & MPOL_MF_MOVE_ALL) || folio_estimated_sharers(folio) == 1) { if (folio_isolate_lru(folio)) { @@ -1045,32 +1032,31 @@ static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), folio_nr_pages(folio)); - } else if (flags & MPOL_MF_STRICT) { + } else { /* * Non-movable folio may reach here. And, there may be * temporary off LRU folios or non-LRU movable folios. * Treat them as unmovable folios since they can't be - * isolated, so they can't be moved at the moment. It - * should return -EIO for this case too. + * isolated, so they can't be moved at the moment. */ - return -EIO; + return false; } } - - return 0; + return true; } /* * Migrate pages from one node to a target node. * Returns error or the number of pages not migrated. */ -static int migrate_to_node(struct mm_struct *mm, int source, int dest, - int flags) +static long migrate_to_node(struct mm_struct *mm, int source, int dest, + int flags) { nodemask_t nmask; struct vm_area_struct *vma; LIST_HEAD(pagelist); - int err = 0; + long nr_failed; + long err = 0; struct migration_target_control mtc = { .nid = dest, .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE, @@ -1079,23 +1065,27 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, nodes_clear(nmask); node_set(source, nmask); - /* - * This does not "check" the range but isolates all pages that - * need migration. Between passing in the full user address - * space range and MPOL_MF_DISCONTIG_OK, this call can not fail. - */ - vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); - queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, - flags | MPOL_MF_DISCONTIG_OK, &pagelist, false); + vma = find_vma(mm, 0); + + /* + * This does not migrate the range, but isolates all pages that + * need migration. Between passing in the full user address + * space range and MPOL_MF_DISCONTIG_OK, this call cannot fail, + * but passes back the count of pages which could not be isolated. + */ + nr_failed = queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, + flags | MPOL_MF_DISCONTIG_OK, &pagelist); if (!list_empty(&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL, - (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL); + (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL); if (err) putback_movable_pages(&pagelist); } + if (err >= 0) + err += nr_failed; return err; } @@ -1108,8 +1098,8 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, const nodemask_t *to, int flags) { - int busy = 0; - int err = 0; + long nr_failed = 0; + long err = 0; nodemask_t tmp; lru_cache_disable(); @@ -1191,7 +1181,7 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, node_clear(source, tmp); err = migrate_to_node(mm, source, dest, flags); if (err > 0) - busy += err; + nr_failed += err; if (err < 0) break; } @@ -1200,8 +1190,7 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, lru_cache_enable(); if (err < 0) return err; - return busy; - + return (nr_failed < INT_MAX) ? nr_failed : INT_MAX; } /* @@ -1240,10 +1229,10 @@ static struct folio *new_folio(struct folio *src, unsigned long start) } #else -static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, unsigned long flags) { - return -EIO; + return false; } int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, @@ -1267,8 +1256,8 @@ static long do_mbind(unsigned long start, unsigned long len, struct vma_iterator vmi; struct mempolicy *new; unsigned long end; - int err; - int ret; + long err; + long nr_failed; LIST_HEAD(pagelist); if (flags & ~(unsigned long)MPOL_MF_VALID) @@ -1308,10 +1297,8 @@ static long do_mbind(unsigned long start, unsigned long len, start, start + len, mode, mode_flags, nmask ? nodes_addr(*nmask)[0] : NUMA_NO_NODE); - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { - + if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) lru_cache_disable(); - } { NODEMASK_SCRATCH(scratch); if (scratch) { @@ -1327,44 +1314,37 @@ static long do_mbind(unsigned long start, unsigned long len, goto mpol_out; /* - * Lock the VMAs before scanning for pages to migrate, to ensure we don't - * miss a concurrently inserted page. + * Lock the VMAs before scanning for pages to migrate, + * to ensure we don't miss a concurrently inserted page. */ - ret = queue_pages_range(mm, start, end, nmask, - flags | MPOL_MF_INVERT, &pagelist, true); + nr_failed = queue_pages_range(mm, start, end, nmask, + flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist); - if (ret < 0) { - err = ret; - goto up_out; - } - - vma_iter_init(&vmi, mm, start); - prev = vma_prev(&vmi); - for_each_vma_range(vmi, vma, end) { - err = mbind_range(&vmi, vma, &prev, start, end, new); - if (err) - break; + if (nr_failed < 0) { + err = nr_failed; + } else { + vma_iter_init(&vmi, mm, start); + prev = vma_prev(&vmi); + for_each_vma_range(vmi, vma, end) { + err = mbind_range(&vmi, vma, &prev, start, end, new); + if (err) + break; + } } if (!err) { - int nr_failed = 0; - if (!list_empty(&pagelist)) { WARN_ON_ONCE(flags & MPOL_MF_LAZY); - nr_failed = migrate_pages(&pagelist, new_folio, NULL, + nr_failed |= migrate_pages(&pagelist, new_folio, NULL, start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); - if (nr_failed) - putback_movable_pages(&pagelist); } - - if (((ret > 0) || nr_failed) && (flags & MPOL_MF_STRICT)) + if (nr_failed && (flags & MPOL_MF_STRICT)) err = -EIO; - } else { -up_out: - if (!list_empty(&pagelist)) - putback_movable_pages(&pagelist); } + if (!list_empty(&pagelist)) + putback_movable_pages(&pagelist); + mmap_write_unlock(mm); mpol_out: mpol_put(new); From patchwork Tue Oct 3 09:19:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82CFCE7543A for ; Tue, 3 Oct 2023 09:19:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2007A6B0143; Tue, 3 Oct 2023 05:19:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B02E6B0144; Tue, 3 Oct 2023 05:19:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 078B86B0146; Tue, 3 Oct 2023 05:19:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EAD0E6B0143 for ; Tue, 3 Oct 2023 05:19:05 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C40AA1C9FCE for ; Tue, 3 Oct 2023 09:19:05 +0000 (UTC) X-FDA: 81303601050.16.887C843 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf04.hostedemail.com (Postfix) with ESMTP id 008994000F for ; Tue, 3 Oct 2023 09:19:03 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XjuhlDzd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324744; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LR8b7je7NOs49VXpiW5eNLJIFvtfHAwACqkCzwughIA=; b=mnuVaG3hGXjynSmvQEDXe/Sg2DACwv2aNDmn66d7juEt85ShmOkwWEjauEMJXlCNETmjG2 g1w9Zre/2ZqvigFrpSH7u4yRFCtwoER666EdWtaxuvh4w/fP4Q5r9TCE3WvGacSQZN0+U4 mW7UKKUcuPEXr9FrtHNMcaVTLfUpzQo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XjuhlDzd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324744; a=rsa-sha256; cv=none; b=6jW8lU32YigA4cjrFeNK40PpPHhtH2W+vVgsYbsTaXc14wRK8TBGlKlAK0jr+qufCbsQLl mTVWoWY+KcbK3xWEzpfxVvoJe4bf6zbUhKAWYv8qPEV6Egqmv9mRwuxgOalekHCtyfMw1V 5RtB2qZYyvYDrBSCDNTDLp4lyGzjH8g= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-5a24b03e22eso8733687b3.0 for ; Tue, 03 Oct 2023 02:19:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324743; x=1696929543; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=LR8b7je7NOs49VXpiW5eNLJIFvtfHAwACqkCzwughIA=; b=XjuhlDzdb4/eeJ6NuAIT1Tmdcmr+jnyaNuq/zlwpY3J3jHLYayeDA78pb21+1W7XWV U+lnUvGrhg38UFwLmDHHaD+zp2dS4LWR+2U/gAYZSlnE952FJq2FCdsvOJCiG6AlUVCx tereibblgT76NDfPLKnc0SdBWCxJctyWeR6m5WQ9Heacg0+OxaoARU+GiZnenQgmGbGd fdXlw34EaOgR1OiIdEc769OtS0szsBBTvEEIddfdiZ14LfDHlN7TRInj/QzwRz2QALqX O8g9MxfAB0c6Z3JDsfejHXRz7LpAEFCi5ZpQ2H8OEHrxeRhQkNBZaEf2bFzbZH1bwDqs RqWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324743; x=1696929543; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LR8b7je7NOs49VXpiW5eNLJIFvtfHAwACqkCzwughIA=; b=fu354kl6YyVGpHdtRiThuSQmIQmcb5kaaLKjLATIop1zL0bE4aSZcgYP9VW8zL3LTl 0weWy1cmUs3bo8ZTeowTydaq5eImnv/xNrR0BFxZ+cbZxAwxYlgBRQuavgFCyZ1lhbOt sHhNDG0J6BZbTAoYJdXKaQS3PznJAzHQU3TM21+87AgtvtLpXdbxd9noeITclafpxZBM yY7510l9kSnI16QAwDxc+r7sxkKXg/V8Azst3Jsr3VAw6wCXH4Vo7oaS02xEQ5AnytPq 3yhCYydw2J1A/jB6ZHwENzkJdcDto274KGx23iqqULf5IUwA02uPP0d4ke16tmAdWMNo GJSw== X-Gm-Message-State: AOJu0YyfwtyEaTbvZCTSa/OFUrDT9QJjacMTdZJi/kapNeD4tGxRm3cn GJhtxAG43g5oEwGbTcseK07j4w== X-Google-Smtp-Source: AGHT+IFH85rdPCXOMMxGLSgYctSHwOqC5dIVzsLz9Oex7w6UXQP98Odac2vWY0r35FgWvr6pMI/Gjg== X-Received: by 2002:a81:a212:0:b0:586:a170:7dbe with SMTP id w18-20020a81a212000000b00586a1707dbemr15010102ywg.13.1696324743080; Tue, 03 Oct 2023 02:19:03 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j13-20020a81920d000000b0059f61be458esm245153ywg.82.2023.10.03.02.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:19:02 -0700 (PDT) Date: Tue, 3 Oct 2023 02:19:00 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 04/12] mempolicy trivia: delete those ancient pr_debug()s In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: urh3i598ce8htwxc9cqja6e67u7a3s34 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 008994000F X-HE-Tag: 1696324743-976175 X-HE-Meta: U2FsdGVkX1+LxdVqznoZRZrqknHzBzKnS+um7MANutzMl1kYWfcrQN/ooktJyRU7sJawGgMx7Tao1we24T46FdiK5g51SWJz636BnMoL7Mgg7uqa49Ntj0MiazZIqKoFwtjcknPfJmLWiJnS3+6s5/Yj5QIelgehZGxyCLwVdSW3waxIzZNeNDzYKFCpcnqyza4RDRlLgAhIVKou45QJvO5B74OcJWe2nITSGrU2IFFTh5pa/trIzy13PaQjJPbAUVndfKuzniysrQD1B0rraMqt03I85UYer8SVaqJDwi0pQw6reDdmC7WpovooQoUIqRiycDHMq3lRJTdBGbn5y2KqVWmibHUFXXnaKwRXFo6QQASYM381WbIxN47eXR1TjbZ1wSPoKDIzDCaWCroXVmq0vsUKqN09TWgMWD26pd2MOzycze5RaKm8Eoed0HJ8/iMlPO9M+n9UPIFXZDc+ulivvEhAkGUu1ojE7PeeUcfT6yeylN85RY6Mqc09zXGXJnBAKLiN7vzTaYV/Yjnh+UklqomqMnE7fqfVcQtl9i2LVVSMd0PdaCRwun8Muw3kOAIiu0s0s6bwG8j1dnL9m60Ag8NzI4gCNdq+r1y8WdTlcoko8c3ejkNVpYhaL5liHiujxihKI0DrY+7FF1iAyM1RHlyPYVfG4sphaTeQQ6Oph/9Jy3Z85ss6rla6qYw8jgznS12yximEtlMfrbX05YvbTuBMUoGXF6ue7r36a8UDKArXT9Qb4h0ZvykwixN/l9XIOX7ysvmZbgwmgiXdN4LGIkX3xcRwEJq4NkLBBLKOW4lCMLt8c3dfZNwXnPbrqY9l5FtiOKm5ZDL+pbOBc8xmegDAtr2IIU3gJBEekmPJDlyZgQ9mloPyWzla/BjC7ZnEyA1XmKXnfSOtCX9GE2owHaS0MeVpul4C4R24A9ZCnHUY7s+awC8O+OOKo07SytC1RZLMbJzMWO62y1p E+9bPKIh RbnVqQys//W3X9eTrjLMbyVwYVx7bzE0ojRLCI8hooKBY9Byqup/07+bbaiHZd3NPjl5DcKL8xdDC4HcnPd7fDOm5WtTMJxGDkC4XBHqnOCtut/DeYc55IoHPImNaHCXhVwQ5kYKwvrSAWGaDUapinXYCKo+7ICi3LhuA7YG7Uvv14W8d+uKFTKyo8hpP36+rc0dJRaiussBUJwuuSAkU8Pu9kAMcombh2fCpNwvupHpTQ0KQYl01dFeo/M7AQ9IjLCr499DC0+VnGEidGUO1MCH/kvdSZcrK1/6XiKuBFWWY58ww1hemJwIAF5FaQ6CVNEa+OPL8t/muKtfScEFMM/SxJIEhN8VL/+SGhKG7QsHwaGfir786+/7pC8kJQAjA4tXNE6mSbG7KK9xuOM1+cvsgMafk7jqhkdftcvNB4EBfTh/vpDNbYffaLlE8vCnAMcsmBegcEU+2moQfyk8uVZSKkx/NUTDxm4sEOI7t30mfqD+WywhE4R2VeyZ7Y8bHcUe6X8b+M4lWk8bnPZ79FaaAV9OzdUH0bF9YcDPduDSGje0tXZJ62qeNeg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Delete those ancient pr_debug()s - PDprintk()s in Andi Kleen's original submission of core NUMA API, and useful when debugging shared mempolicy lifetime back then, but not used recently. Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) --- mm/mempolicy.c | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 752d880dcdf8..780498662b75 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -264,9 +264,6 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, { struct mempolicy *policy; - pr_debug("setting mode %d flags %d nodes[0] %lx\n", - mode, flags, nodes ? nodes_addr(*nodes)[0] : NUMA_NO_NODE); - if (mode == MPOL_DEFAULT) { if (nodes && !nodes_empty(*nodes)) return ERR_PTR(-EINVAL); @@ -768,11 +765,6 @@ static int vma_replace_policy(struct vm_area_struct *vma, vma_assert_write_locked(vma); - pr_debug("vma %lx-%lx/%lx vm_ops %p vm_file %p set_policy %p\n", - vma->vm_start, vma->vm_end, vma->vm_pgoff, - vma->vm_ops, vma->vm_file, - vma->vm_ops ? vma->vm_ops->set_policy : NULL); - new = mpol_dup(pol); if (IS_ERR(new)) return PTR_ERR(new); @@ -1293,10 +1285,6 @@ static long do_mbind(unsigned long start, unsigned long len, if (!new) flags |= MPOL_MF_DISCONTIG_OK; - pr_debug("mbind %lx-%lx mode:%d flags:%d nodes:%lx\n", - start, start + len, mode, mode_flags, - nmask ? nodes_addr(*nmask)[0] : NUMA_NO_NODE); - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) lru_cache_disable(); { @@ -2516,8 +2504,6 @@ static void sp_insert(struct shared_policy *sp, struct sp_node *new) } rb_link_node(&new->nd, parent, p); rb_insert_color(&new->nd, &sp->root); - pr_debug("inserting %lx-%lx: %d\n", new->start, new->end, - new->policy ? new->policy->mode : 0); } /* Find shared policy intersecting idx */ @@ -2656,7 +2642,6 @@ void mpol_put_task_policy(struct task_struct *task) static void sp_delete(struct shared_policy *sp, struct sp_node *n) { - pr_debug("deleting %lx-l%lx\n", n->start, n->end); rb_erase(&n->nd, &sp->root); sp_free(n); } @@ -2813,12 +2798,6 @@ int mpol_set_shared_policy(struct shared_policy *info, struct sp_node *new = NULL; unsigned long sz = vma_pages(vma); - pr_debug("set_shared_policy %lx sz %lu %d %d %lx\n", - vma->vm_pgoff, - sz, npol ? npol->mode : -1, - npol ? npol->flags : -1, - npol ? nodes_addr(npol->nodes)[0] : NUMA_NO_NODE); - if (npol) { new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, npol); if (!new) From patchwork Tue Oct 3 09:20:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB60AE7543A for ; Tue, 3 Oct 2023 09:20:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AB706B0146; Tue, 3 Oct 2023 05:20:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55B0C6B0147; Tue, 3 Oct 2023 05:20:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4239B6B015D; Tue, 3 Oct 2023 05:20:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 32F866B0146 for ; Tue, 3 Oct 2023 05:20:21 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E747380211 for ; Tue, 3 Oct 2023 09:20:20 +0000 (UTC) X-FDA: 81303604200.23.1BC0616 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf23.hostedemail.com (Postfix) with ESMTP id D6FE0140026 for ; Tue, 3 Oct 2023 09:20:18 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OLnR31BD; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WpSDmja3k+xJDYdjjbygGD+ZOvEonrCZy7oEBpIDr3Y=; b=AbLKxeKmRRtBilTKN/8vRf2+Z/7AifyW7F3HiZMa6l7tWoKhHcpN7RWeYjEj4V/25224VB Fwv3s1B14CnmjyR88mh2A4gVedoG14KsBDtrll3Ng9f+sAfjiV51uyBecPK29uTZylQT1V 77XB6vXWJhNbmWUXcKoMd/zivEfrXyA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OLnR31BD; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324818; a=rsa-sha256; cv=none; b=nhi5JyQ3EWwooz29hgLJOoSQ17jfWoIa1iSjCtzbCW8goAwZI+HyE6Tv0ph835jdoBh0AA wxjBmK7ggpA0auxnPcjUfODjRVXwnsFcT8pwBoLx5ZYGWPGsWVeycSBJ7dWr0ouGVYJ82d JMatoyZY1bJ147M4ycrS9greSc2Fec8= Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-d865854ef96so708377276.2 for ; Tue, 03 Oct 2023 02:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324818; x=1696929618; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=WpSDmja3k+xJDYdjjbygGD+ZOvEonrCZy7oEBpIDr3Y=; b=OLnR31BDeL0dBpSlrD/AMxSm1wP2vZy5T0gKF7OUgSkHLFMn1PGNE+UGbLW5LP/Az4 cBrz/OMJjF884VQiMMLWCSlSP2RcvnHQWtbSccvAOf/x08+nPFW+C+kl7DQ3sfPMsKte kkKwQIDZXtTL84qIrwai7DOYyj761fAisXNGwq6KqLWTqH8r1VUcQU6raWORUNn5+TNm F+fyTznTs+AH8XNJjFilIiAX8HELBDGMABI5+PluZU8MIAH+9ba+A17TIuG3cQzk1C1j eccciW0J57qn3yZcSH354j8ov++/WZyGzgvxhfiueAd3UG4FtD0BLszJQRCxnZ3R2BFD O5Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324818; x=1696929618; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WpSDmja3k+xJDYdjjbygGD+ZOvEonrCZy7oEBpIDr3Y=; b=wOHs+ocl6o87NHmaE7MPwd377a7h1Gbw2UbMjWr2sWa0t02G/3IdD0ZA3+bOWfd/sb lXE8Nt7mm0f0TDp2xdVQS4sDdmNaNB1YU12gq7IOFzapplx4omUcrMmJlAKhava+6kIM 9aikGtt+391gcgkWIkE4YgTs+SpDNQi932/zrH6ddz1DvnQycnYaHa2602gsc0/nGfam 7n/G6CVsnCsUNBRTKe+ljy7RS7s1tjJXzuxN0pIp6QhhbbRiMRND+NaFip0crqKHniwF XRT5VKbBvM7CzTkeqiExv6+uGNUHSvwLnfgBES3jC1LzOOT+qUf3G0LI93aUpvG83Nw+ vdKA== X-Gm-Message-State: AOJu0Yw6QGO6rVSFzqfeHc+GOTbTN/AnW9j+l7XShajFvcGpfEzKmEJ/ GOGgXUe+5U8RU21DfBWI3GZi+A== X-Google-Smtp-Source: AGHT+IEft503DS0lDDt3TXmtWRBCS6YBAa+I/5dVhW15BOv+4PDW9asRrQ3b3VJqUbGfVYzkZaPOgg== X-Received: by 2002:a25:50ce:0:b0:d90:b45d:6e6a with SMTP id e197-20020a2550ce000000b00d90b45d6e6amr5801772ybb.2.1696324817790; Tue, 03 Oct 2023 02:20:17 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j2-20020a258b82000000b00d8674371317sm279506ybl.36.2023.10.03.02.20.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:20:17 -0700 (PDT) Date: Tue, 3 Oct 2023 02:20:14 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 05/12] mempolicy trivia: slightly more consistent naming In-Reply-To: Message-ID: <68287974-b6ae-7df-4ba-d19ddd69cbf@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: D6FE0140026 X-Rspam-User: X-Stat-Signature: c9kkyrft8bd4o4eu8awm4mh9eio4mi9z X-Rspamd-Server: rspam01 X-HE-Tag: 1696324818-82369 X-HE-Meta: U2FsdGVkX1/r5thALwFMHt8NlVqLn2n/HDsNvlr6YkfqmQuhjoVFIJ49ucc5LuwsKFIWVuv9hGh5+Gp9IXuaO0v4ZjbyN+5MMU8wKu3uh62X+4BU9Lhmqq4o1h21YSOev2IzRFrXR6eN9gyaXWeb/YyU7L+qjexUa0lAI/GsyFL8fHVKai1lJDj+gw/6uFrjnC+wo2l34wScjC1PYnUe1P1S8z5JUrSL2e01aP/EBJ+pZxb5cJzkSLY2lVv3MJwhUvFNjb3U2vwnFkDRG0E7Kr9BfwhemIjvmvJ1kKVaG7uQxeh664U1R+g+O37Dz/A2odoN6OBLBOONIRxcWO9hMvoFjZll2iTpn3zEi/rx2oqHWN2iUkxBs8QhO0/qbASlD6AgEasOc2x7Xgx2IQSylm6vlFVBD1+1qyPOFDD13lNWy3hogAuFAzxAWc4ChKL7lE9TcTyxNko2tMzHMjGm/6OfBQoG7zxmrakOj+plxJZ6dclE4spgnjbQ4vaOp1IV8PR9Snivh9l4JirnoSMhQ24RUrF22ZnVJm82ATPkh9QKnk3G7eDjeRZBrQGOgzO03hmtt53nIZ9V64L7jEO6TGldd3ma4MZvuS9qYLlS54zldCerruiOHWOyy/Tl4J0/zwNwCy9AAU5tdiB4ySWrR2WEe7KkXBi1mogbah2qK3rEZcOKAGru9Owm64+BKN08m+1vqKZERpHRHyQ6cs27I5ASHOHV0dOdDwhHxYSvg6NDTutu4wVxfdPWPJYlTav71809YIW79JsKDTqTzZtnq5CAzhqixFGWuXPwrIU/rT0AAKnG7dzymEwV4svMe1qrNnzqXgt59bWGUWnkG14wKq5V28Ar1C5dE0cVW1Fh3WICMBEnKNjnHu4FzodV2I1RKPA6GbLXxzp+rz+7aFBxe8eM/UjOIaMRadw5L1Wyj+27nh+tDNIwrzqaGhYma0pVK4wHixnww3EkD3Ook7L tt0c2GeI IkrMJ9OU0cU+RFwVMkEFO8sHFdZmOko4gJJ9ayUxrXiJHjPQvZwmHcNcQO/5dcLkeiyZSkG8C3BeaEco3km40Mif3HXsFgh0/FbSm9LyLWSU7VHw2UzWRzpoQ7oBBivb360P+IbzLuemFyqrHJRaFcsPDZv6gNkrNbIXgjbzkxgn7ssb7z6Uc+gB7DQoq7qBYxUgLd+f02oNbXbDn78uJrad56jyZgEX5Y2SdTV8zdWSBSHbozV0wJVecasPp/2JRjzgQhVPUVbF5FEWpB4q3bNy3eq15IIpE/NxZ7of5DQM+kjVorSxrucySII3qyKWq7diQpJxf30K3Ex61VQfIiYYdnYaKbp93o/tJBKI5HYosAQzyz6qBEpowRqdV2QrJtt7zLvXVxmTKrPU07Klj4lOzoYE8MLIV8bq9LdihNpZ+r6u2wa1+67I5QGDPyHYc/Cy2G7nfkdhUMw0A12EfToJdYZZVqgTe13YS3R92luYS8qtQJWp34xpp7KZBKFuFi9SjGtocM0kNFok= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before getting down to work, do a little cleanup, mainly of inconsistent variable naming. I gave up trying to rationalize mpol versus pol versus policy, and node versus nid, but let's avoid p and nd. Remove a few superfluous blank lines, but add one; and here prefer vma->vm_policy to vma_policy(vma) - the latter being appropriate in other sources, which have to allow for !CONFIG_NUMA. That intriguing line about KERNEL_DS? should have gone in v2.6.15, when numa_policy_init() stopped using set_mempolicy(2)'s system call handler. Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/mempolicy.h | 11 +++---- mm/mempolicy.c | 73 +++++++++++++++++++---------------------- 2 files changed, 38 insertions(+), 46 deletions(-) diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index 6c2754d7bfed..325b7200c311 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -126,10 +126,9 @@ struct shared_policy { int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst); void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol); -int mpol_set_shared_policy(struct shared_policy *info, - struct vm_area_struct *vma, - struct mempolicy *new); -void mpol_free_shared_policy(struct shared_policy *p); +int mpol_set_shared_policy(struct shared_policy *sp, + struct vm_area_struct *vma, struct mempolicy *mpol); +void mpol_free_shared_policy(struct shared_policy *sp); struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx); @@ -193,7 +192,7 @@ static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) return true; } -static inline void mpol_put(struct mempolicy *p) +static inline void mpol_put(struct mempolicy *pol) { } @@ -212,7 +211,7 @@ static inline void mpol_shared_policy_init(struct shared_policy *sp, { } -static inline void mpol_free_shared_policy(struct shared_policy *p) +static inline void mpol_free_shared_policy(struct shared_policy *sp) { } diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 780498662b75..c7906a034959 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -25,7 +25,7 @@ * to the last. It would be better if bind would truly restrict * the allocation to memory nodes instead * - * preferred Try a specific node first before normal fallback. + * preferred Try a specific node first before normal fallback. * As a special case NUMA_NO_NODE here means do the allocation * on the local CPU. This is normally identical to default, * but useful to set in a VMA when you have a non default @@ -52,7 +52,7 @@ * on systems with highmem kernel lowmem allocation don't get policied. * Same with GFP_DMA allocations. * - * For shmfs/tmpfs/hugetlbfs shared memory the policy is shared between + * For shmem/tmpfs shared memory the policy is shared between * all users and remembered even when nobody has memory mapped. */ @@ -291,6 +291,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, return ERR_PTR(-EINVAL); } else if (nodes_empty(*nodes)) return ERR_PTR(-EINVAL); + policy = kmem_cache_alloc(policy_cache, GFP_KERNEL); if (!policy) return ERR_PTR(-ENOMEM); @@ -303,11 +304,11 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, } /* Slow path of a mpol destructor. */ -void __mpol_put(struct mempolicy *p) +void __mpol_put(struct mempolicy *pol) { - if (!atomic_dec_and_test(&p->refcnt)) + if (!atomic_dec_and_test(&pol->refcnt)) return; - kmem_cache_free(policy_cache, p); + kmem_cache_free(policy_cache, pol); } static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes) @@ -364,7 +365,6 @@ static void mpol_rebind_policy(struct mempolicy *pol, const nodemask_t *newmask) * * Called with task's alloc_lock held. */ - void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) { mpol_rebind_policy(tsk->mempolicy, new); @@ -375,7 +375,6 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) * * Call holding a reference to mm. Takes mm->mmap_lock during call. */ - void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) { struct vm_area_struct *vma; @@ -757,7 +756,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, * This must be called with the mmap_lock held for writing. */ static int vma_replace_policy(struct vm_area_struct *vma, - struct mempolicy *pol) + struct mempolicy *pol) { int err; struct mempolicy *old; @@ -803,7 +802,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, vmstart = vma->vm_start; } - if (mpol_equal(vma_policy(vma), new_pol)) { + if (mpol_equal(vma->vm_policy, new_pol)) { *prev = vma; return 0; } @@ -875,18 +874,18 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags, * * Called with task's alloc_lock held */ -static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) +static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes) { nodes_clear(*nodes); - if (p == &default_policy) + if (pol == &default_policy) return; - switch (p->mode) { + switch (pol->mode) { case MPOL_BIND: case MPOL_INTERLEAVE: case MPOL_PREFERRED: case MPOL_PREFERRED_MANY: - *nodes = p->nodes; + *nodes = pol->nodes; break; case MPOL_LOCAL: /* return empty node mask for local allocation */ @@ -1654,7 +1653,6 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, out_put: put_task_struct(task); goto out; - } SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, @@ -1664,7 +1662,6 @@ SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes); } - /* Retrieve NUMA policy */ static int kernel_get_mempolicy(int __user *policy, unsigned long __user *nmask, @@ -1847,10 +1844,10 @@ nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy) * policy_node() is always coupled with policy_nodemask(), which * secures the nodemask limit for 'bind' and 'prefer-many' policy. */ -static int policy_node(gfp_t gfp, struct mempolicy *policy, int nd) +static int policy_node(gfp_t gfp, struct mempolicy *policy, int nid) { if (policy->mode == MPOL_PREFERRED) { - nd = first_node(policy->nodes); + nid = first_node(policy->nodes); } else { /* * __GFP_THISNODE shouldn't even be used with the bind policy @@ -1865,19 +1862,18 @@ static int policy_node(gfp_t gfp, struct mempolicy *policy, int nd) policy->home_node != NUMA_NO_NODE) return policy->home_node; - return nd; + return nid; } /* Do dynamic interleaving for a process */ -static unsigned interleave_nodes(struct mempolicy *policy) +static unsigned int interleave_nodes(struct mempolicy *policy) { - unsigned next; - struct task_struct *me = current; + unsigned int nid; - next = next_node_in(me->il_prev, policy->nodes); - if (next < MAX_NUMNODES) - me->il_prev = next; - return next; + nid = next_node_in(current->il_prev, policy->nodes); + if (nid < MAX_NUMNODES) + current->il_prev = nid; + return nid; } /* @@ -2367,7 +2363,7 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) { - struct mempolicy *pol = mpol_dup(vma_policy(src)); + struct mempolicy *pol = mpol_dup(src->vm_policy); if (IS_ERR(pol)) return PTR_ERR(pol); @@ -2791,40 +2787,40 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) } } -int mpol_set_shared_policy(struct shared_policy *info, - struct vm_area_struct *vma, struct mempolicy *npol) +int mpol_set_shared_policy(struct shared_policy *sp, + struct vm_area_struct *vma, struct mempolicy *pol) { int err; struct sp_node *new = NULL; unsigned long sz = vma_pages(vma); - if (npol) { - new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, npol); + if (pol) { + new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, pol); if (!new) return -ENOMEM; } - err = shared_policy_replace(info, vma->vm_pgoff, vma->vm_pgoff+sz, new); + err = shared_policy_replace(sp, vma->vm_pgoff, vma->vm_pgoff + sz, new); if (err && new) sp_free(new); return err; } /* Free a backing policy store on inode delete. */ -void mpol_free_shared_policy(struct shared_policy *p) +void mpol_free_shared_policy(struct shared_policy *sp) { struct sp_node *n; struct rb_node *next; - if (!p->root.rb_node) + if (!sp->root.rb_node) return; - write_lock(&p->lock); - next = rb_first(&p->root); + write_lock(&sp->lock); + next = rb_first(&sp->root); while (next) { n = rb_entry(next, struct sp_node, nd); next = rb_next(&n->nd); - sp_delete(p, n); + sp_delete(sp, n); } - write_unlock(&p->lock); + write_unlock(&sp->lock); } #ifdef CONFIG_NUMA_BALANCING @@ -2874,7 +2870,6 @@ static inline void __init check_numabalancing_enable(void) } #endif /* CONFIG_NUMA_BALANCING */ -/* assumes fs == KERNEL_DS */ void __init numa_policy_init(void) { nodemask_t interleave_nodes; @@ -2937,7 +2932,6 @@ void numa_default_policy(void) /* * Parse and format mempolicy from/to strings */ - static const char * const policy_modes[] = { [MPOL_DEFAULT] = "default", @@ -2948,7 +2942,6 @@ static const char * const policy_modes[] = [MPOL_PREFERRED_MANY] = "prefer (many)", }; - #ifdef CONFIG_TMPFS /** * mpol_parse_str - parse string to mempolicy, for tmpfs mpol mount option. From patchwork Tue Oct 3 09:21:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A79DE7543A for ; Tue, 3 Oct 2023 09:21:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B3936B015D; Tue, 3 Oct 2023 05:21:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 164986B015E; Tue, 3 Oct 2023 05:21:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02C8D6B015F; Tue, 3 Oct 2023 05:21:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E87D96B015D for ; Tue, 3 Oct 2023 05:21:39 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C3972B3B51 for ; Tue, 3 Oct 2023 09:21:39 +0000 (UTC) X-FDA: 81303607518.27.6AC2E19 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf03.hostedemail.com (Postfix) with ESMTP id 0C11D20010 for ; Tue, 3 Oct 2023 09:21:37 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FallctNc; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324898; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TKEb9Cd3xCSrgbL40B7c6RQSPbdbractxdPzCveSCj4=; b=u1RWTA/kisgrItYu5I9W8dq2UXN3/YwdHwTGG+FYQsOEFHffMcF0A324dn1mnuVtVcLcu+ ZeKnuvoVrSdA1hskiDOpaNWRz8TMN8eURokHtyL+cYFaRheXW1Lo1yJnUbRYC39JniN52L W2KfJxsTh3j1QB+LNm2vLzN7bMoGDG4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FallctNc; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324898; a=rsa-sha256; cv=none; b=cmaR6f5S4qHyY7fPyCSE4E4eF6rDttvZIozB0fH6XWmC1g5tm0gWehZrvG76Tofc8RCx7a TNfC2Saunrd2PTc084kx4X//7F9b2htjjm5g66Dse4Pd2o/lcjdKn+gE3ZeD4dtcxkrReX XC1OL/JHZyDXIb0VQH3Y9QbpXhiCJXo= Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-77433e7a876so54160285a.3 for ; Tue, 03 Oct 2023 02:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324897; x=1696929697; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=TKEb9Cd3xCSrgbL40B7c6RQSPbdbractxdPzCveSCj4=; b=FallctNcGWoeABOLkfUG1a3SDZzm2y6slUClTFndE1ABJigRmJE/f2+QNitwhQkvU2 Jx0x5QsYTK48NEUIlmQWynDQHFlVAeoqiTq51Yy7ncMWnrRtUna6Xng1Us9ZRP8kzRG9 DIriomX8CHowCty9/yPxyGqeC7+i7fIrORIuvo8qC0ybdikcaRO+4EW3s21NWF1jyxx9 KqraOKOaBxov0+y68QH3hFwoWsDQfQo7d5rI8Rsvf7G1pZXsKRDrFOtag81yGNxQSvyg InpqBFY9R7VuEAGf9Hv1OL1yyF3DQAhjzcjs3qGI0noYsamuz2MVBEhetAaRztHWaHNn I6cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324897; x=1696929697; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TKEb9Cd3xCSrgbL40B7c6RQSPbdbractxdPzCveSCj4=; b=meyRbaI/6pQuxCksLR0O+dH8h+uV7zycxce0RDn2Z5ehkE80PBe9zxL7wCDIKJYe2X weLI6dSxLtm8icXJWLP7cOxaCIXWQKy6V36f620mtRIJZ3Ir8/4HOtksgWBBt/FQ5Pn2 qJjixzxCEZpK8/EUl75hyfoohQBZ5fCaM/w6GHe6a/oQgTwn9ZfcdFR8fzk/OHejbTSC eoisAT06M9hE/DwKT4b/3lEaub24A9tvxFCMSY8T91jk6PR985xOl1iF3klNeOv6ibUm hUqS3ajioOrnjtlssJfM7eKe4Ab2PtQZh4N01Vs8ZsMlxqHWF4PuvOM/iz1B0RrbUHaR qNOw== X-Gm-Message-State: AOJu0YzwRoUAPnRtYls48Ts0uFxNwpElfAtOZuODpohwRPbGyJHqrqRB j2FojI4buxVLOd1s6LKrk7OkLg== X-Google-Smtp-Source: AGHT+IGmyNX1mGOvsxjDJ+VgdVzrziqLJLl2hhCUT+uZ6s1yuT2hqDhi9cPwKTZ3QGcrdlo0VxUawQ== X-Received: by 2002:a05:620a:95c:b0:774:1d91:e41a with SMTP id w28-20020a05620a095c00b007741d91e41amr13254046qkw.77.1696324897011; Tue, 03 Oct 2023 02:21:37 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a14-20020a81bb4e000000b0059be6a5fcffsm247867ywl.44.2023.10.03.02.21.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:21:36 -0700 (PDT) Date: Tue, 3 Oct 2023 02:21:34 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 06/12] mempolicy trivia: use pgoff_t in shared mempolicy tree In-Reply-To: Message-ID: <5451157-3818-4af5-fd2c-5d26a5d1dc53@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 0C11D20010 X-Rspam-User: X-Stat-Signature: dyzzgse3t1tcqdwjw57d6anp1ni74nm3 X-Rspamd-Server: rspam01 X-HE-Tag: 1696324897-605627 X-HE-Meta: U2FsdGVkX18COa8sOkA0HRUnazJplAObt6QJdkHAdvri3PRVRSA6I6yl37dn8hSMIe7HS6bfPVU6Vzkp0wa4d71grTdbARcvAmSv2UoAXX7txX9PSa2/MYG+N2Ll3TQoCOWQD+xdgia11clsv0S2Msm4P1aGFo/OXWjjf/4rbHqqiLX1WaF8/1eGYJjcYmLWJsQFNW1lrJH2Ih3VJMBPLQOW73ZUQlFHX06Og/6S47G+Jc5P23c9Xm/zBbRuvgNT8m8Iiuy+RuElK/bqzINIi6WwVWaMGEu3dxhnL6qmMaXFR1GCGPYp63kpkfJIoQlB32A0kuLEAkOjlgjp4UKG4btFZ4cH6G4NXrFOSkLVwPCKW0NPjgR46SbS7CvGYNIj4/jMMc4WP697cXKVw0Ic38txwKRsxC6fPArnjG2q4lx0Dy+w6S+8EY8RITpF5rd5N6SttY0tdaAKG+mOSEXh4PBD2cNRx/NgNz+mlq1wQa8/EWy7hd+O7G86si3Z/3L9fbsTW52vJeO0cj4ZErPEBDIuqDavF7I/Xy3z3xm5XIjSo3z3CCTy4XBUfSNlqYmgZABUdmuh8hMrER+PdOLcVu1gtjDGP3QW0mthD6O//jgEK8A/ZO4dq5UeRfmUQUDxrVxT2UoSfhESPuLS59DSrp87o+3HQd2SboT9Hz1p7cT4kSnZ7qtoZ7PsA5VDfIerwyfI32Qhit8XSUNzs5IOehTRxn9q3Z6Eqs/FjfSAlB5tEG7AFsynaLMowHauAA6rj0pH3i2Uf66NzbMAgaEWo+w6E/N1caLG/pyE/qqwjLwC1q09kwrf8/Q4Q7EwVVO1njZO9yV8mQiNIYU5cNRbQ/HD6dMkjMrAOjnLkofV78NSOVpFIh/EzZRr/oHgjs+GO13bJTNb/Jw28gwNFXJNxtosroBijr330E5pIbj8K3Dvts/5a7O4icg08XU4cwWn2uoNNPoHvTi1I7UqbK4 m7wCYQJ9 cpuCNyTEO4ld/Y1cWM7WF+YHheelKqZ8Mgu06BkDnfryH4VESM4nOzUz6TcCxqrMfL1ibgIxWEF2+GqZ55dJC+/3GU42USDXwKhblnliXa8/TmNzW8Ov4qCfDSpj9C4WTLdjfHT20hgWiv3a3BR3d37x/nCmm2e6ts97QTfUls4Lui4RIEJB9IxZqzrq3N4CGfMtms5urjgFBLW5q/aSdNjYCoHcb6MkaGTtlnL9HpzzmrbnKaQdDqhMIXntksA9HjIKaXYk5pKChL1IbesfExcnl2c89DyAil/MCG9CQvhPmxjigaS2TWWk/1iO2v/GVPhrI1qRw8dZgDArVf0U28qGvQojF75daryb85N3s4muiUe8bLy6EyPOkbz5S15FsmKza77u+y/C6BPMb2QbkYgRK4UvzmJ3HVpBDAXi61bBLuRvIJ0Ky9sPasAqry76cn/UPo7Et3QYysgwtx8/y+w1yVJn0UDcA9wavP4WbhDp9szAFoOPyVK9Uri0D2Fexxueq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Prefer the more explicit "pgoff_t" to "unsigned long" when dealing with a shared mempolicy tree. Delete confusing comment about pseudo mm vmas. Signed-off-by: Hugh Dickins --- include/linux/mempolicy.h | 20 +++++++------------- mm/mempolicy.c | 12 ++++++------ 2 files changed, 13 insertions(+), 19 deletions(-) diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index 325b7200c311..c69f9480d5e4 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -107,22 +107,16 @@ static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) /* * Tree of shared policies for a shared memory region. - * Maintain the policies in a pseudo mm that contains vmas. The vmas - * carry the policy. As a special twist the pseudo mm is indexed in pages, not - * bytes, so that we can work with shared memory segments bigger than - * unsigned long. */ - -struct sp_node { - struct rb_node nd; - unsigned long start, end; - struct mempolicy *policy; -}; - struct shared_policy { struct rb_root root; rwlock_t lock; }; +struct sp_node { + struct rb_node nd; + pgoff_t start, end; + struct mempolicy *policy; +}; int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst); void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol); @@ -130,7 +124,7 @@ int mpol_set_shared_policy(struct shared_policy *sp, struct vm_area_struct *vma, struct mempolicy *mpol); void mpol_free_shared_policy(struct shared_policy *sp); struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, - unsigned long idx); + pgoff_t idx); struct mempolicy *get_task_policy(struct task_struct *p); struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, @@ -216,7 +210,7 @@ static inline void mpol_free_shared_policy(struct shared_policy *sp) } static inline struct mempolicy * -mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx) +mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx) { return NULL; } diff --git a/mm/mempolicy.c b/mm/mempolicy.c index c7906a034959..1d3f9e1ecbb8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2448,8 +2448,8 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) * lookup first element intersecting start-end. Caller holds sp->lock for * reading or for writing */ -static struct sp_node * -sp_lookup(struct shared_policy *sp, unsigned long start, unsigned long end) +static struct sp_node *sp_lookup(struct shared_policy *sp, + pgoff_t start, pgoff_t end) { struct rb_node *n = sp->root.rb_node; @@ -2503,8 +2503,8 @@ static void sp_insert(struct shared_policy *sp, struct sp_node *new) } /* Find shared policy intersecting idx */ -struct mempolicy * -mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx) +struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, + pgoff_t idx) { struct mempolicy *pol = NULL; struct sp_node *sn; @@ -2672,8 +2672,8 @@ static struct sp_node *sp_alloc(unsigned long start, unsigned long end, } /* Replace a policy range. */ -static int shared_policy_replace(struct shared_policy *sp, unsigned long start, - unsigned long end, struct sp_node *new) +static int shared_policy_replace(struct shared_policy *sp, pgoff_t start, + pgoff_t end, struct sp_node *new) { struct sp_node *n; struct sp_node *n_new = NULL; From patchwork Tue Oct 3 09:22:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCD82E7543A for ; Tue, 3 Oct 2023 09:23:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 781216B015F; Tue, 3 Oct 2023 05:23:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7327A6B0160; Tue, 3 Oct 2023 05:23:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F9976B0162; Tue, 3 Oct 2023 05:23:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4F2546B015F for ; Tue, 3 Oct 2023 05:23:05 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2C0FDA021C for ; Tue, 3 Oct 2023 09:23:05 +0000 (UTC) X-FDA: 81303611130.26.517221C Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf10.hostedemail.com (Postfix) with ESMTP id 5B329C0002 for ; Tue, 3 Oct 2023 09:23:03 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JqcSOboz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696324983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Mzr/4uL62aOMXHcIAqfjDQcxRGe9jlWGjvKm9366txw=; b=6mbXda+eACbazk6btZRBz4njyPlVXWCWYOs0GLUXEpbfvNqRDKoyrpEB3xhAFm9l4ywfbY +LQGt95nEIBCwG6WXw/GJ5plyQtWwUhxLb3Iz4Ql/O92sHOpWfaAGNHt+LEo85i/UgWX5L zMLG2WY1l6zOQw4QK5b3lz40BZwnXY8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JqcSOboz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696324983; a=rsa-sha256; cv=none; b=bEtbjsjDFQmguEVrFyctnRgLQbYODyN4FgYyzd0EX/rPo151F5G5GN9h2DpdO5gt022f+h u5+YbiFTQajfx+ib5hEJHcCmIMP1qZLP5tk7qUlb3sBEZJddYiDwlDcdQ1UXWyd/J37nXO UJiMFkNoOgYb3rpuPLSQ6lq+Mi9Aamk= Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-d865c441a54so734842276.1 for ; Tue, 03 Oct 2023 02:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696324982; x=1696929782; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Mzr/4uL62aOMXHcIAqfjDQcxRGe9jlWGjvKm9366txw=; b=JqcSOboziOaz1RhDZa8GQqho5kUUhuanF8M7SEtTrtE0ohUxk771I4hFVCb0BpsAAM ALXHZESU7TkbAY7HjwiXjHiD0IxUF8AapyNesJPrWG4gr65letMTiecImyUvIVslVv4F 2xVLEVLJchnm1Jow2RcCNLlYS3H0iWrTxqBCW/JJqtITqysnGoQzbdFTiK7A36W9YBdF Y+iBxi0rlx9ZYy105+YWq4X5pgXKpCPKGFy+UVljXUUD/yJLeCRWeXKxDh9r3/q8mMnl H+46dSEbVsF86Wasp1I8nh2WG3IdoM92za6K9wVH5olv2jw6pcut21hgCmbAAUGZ8iL7 LvDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696324982; x=1696929782; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Mzr/4uL62aOMXHcIAqfjDQcxRGe9jlWGjvKm9366txw=; b=ZulKae+sLXyIr61eDi/lCqhxbXG5jFNgek9eVfsxhGAVMMgOYccpJ5FSqIxUT9s4JC UO7hDLDIx3xgQvGH44Xe03UJpMH0pwTyKA3u+UV/+Z1hl4iqtwQCELhLVd0aJKP1iW51 ObQ8m9763z5CimhLnMhZkOKG7Yh4HQf6QeUL3D72U35sgCJ9lTQTDSL0EVBY+PTzhfRk 9r2P2rKEZK6YrtdORslUr04E2WvomzRvpOPQ2KwPo9fMQdjZPonyj4HwPZiKNPuPep2I 6FEd4ZST6fJO+W/FWcUNDsOsO0j7hMa8BX2IzTgG9KtCtu5PrDQzFQOM9sIwW9r1QKIY EEaQ== X-Gm-Message-State: AOJu0Yz8sk6z5cglyBDnN5UtvlzaBe8eBuUvMVrmbwquV/ufg6y+SfqE rZ5kMrD/z7b9o3TpDfBiWsQQcg== X-Google-Smtp-Source: AGHT+IFxboDdNa1UhQmoXRYSqnKITzmYFNT7NTostAtgCQUNJAxTZwIq6gRE4Eb16MSOzmlZgLi7Uw== X-Received: by 2002:a5b:c89:0:b0:d81:599f:a538 with SMTP id i9-20020a5b0c89000000b00d81599fa538mr10809754ybq.51.1696324982333; Tue, 03 Oct 2023 02:23:02 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id c6-20020a259c06000000b00d816fa23bd4sm280649ybo.26.2023.10.03.02.23.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:23:01 -0700 (PDT) Date: Tue, 3 Oct 2023 02:22:59 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 07/12] mempolicy: mpol_shared_policy_init() without pseudo-vma In-Reply-To: Message-ID: <3bef62d8-ae78-4c2-533-56a44ae425c@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5B329C0002 X-Stat-Signature: 5jjkn6c31wrwg8wrtmoz94on66w6h1er X-Rspam-User: X-HE-Tag: 1696324983-300103 X-HE-Meta: U2FsdGVkX1/hpjOqNt8NRwA3xrqKJ8vMlf8R9rzSige1iQ4XDhrkLLP124TXkc5odgxdcFbx4B4i1yil4q8/033lK5QnSoEVjVoRAuW+MYFTKZWb5e0wBEOjuwkGREs5WMMZbTaxWaQo78m7npx3nMSRNKswE6iEiQRSXlAV8HNqiRYdAPR1a+SJbQMVXcDBNRlVb4IDAe1P9TFjqnS0WyEnUd1fdQSSBfGmY91m/pnoviNU+qq19kJe6xJU5v8efj79lJ3LazktadPQsWc0hNe2xMCeIUW3pLimSRZvy1C6E8gbewNq4bBF36Qeo3StagUH/uaHwecmcoGVbkk900hO9mhV1Bqt06MbyBzTWDwQ5FEcQyHGMWKegGdHikex8EQUdARTBnzH51j1Xr+oO2IrmZ6nZYRLASvrsu7tdWhniW+dPIrQuprKRICZ9Z8odZ2PRyuZ+xjE17htaMcGS8SpYOxiEPD4HR3t8XJ9/usAkS3gIbBnKp6iVByNIVpa5db3CltUxuawGmkyycDv4XvssyrT3ZONbUsUZsxON6DuVUK1Mkm4JEyc+A2xZe0zHW+HvZzfH/laLWSkyR8/DbmOzaRLkYZKy8icrNOyilXq0U5UqkWha3z5SAlmOSH4A3IMBhaAVgj33IVCm7+P7wg8m/JSqHnF/YSIOxplfevf+fvxtJdmsOkCuKcly6GnHd0/YSVK1AEjjOy1JAwXS3jHRIR9wEbhc7dEalQjJkRVW226AKYoFPlcS9iRr19vwEhVTJ6PHLQTtbMlhRybJByR1pBfB7jWHYb1nNgfZSewxkE9zvqXWptfNpKbUi9KBD7FXlN1qbu3Wj582XUoa2sg6aNr5/1CR8BM0RuqxjEIj5ZyRhWID/GB2VQ/e4tdvO/N1BFljC3vs63cm42LlVi/ukaWUB6hK5oknBxJVkQDbHqU+xYbsH/eIVD0DOwottTgOwcSFQFjzexdM7b g183l7s0 rlh9srvYGk7YAflejLfsg4jRo/DkraKaWlBilRVMSMttOyFAO1YDjsAVljoqQhYnHVqh+r1KFWCRaKp294NWA9NfmIL63WEq6EA4NUEO3LG5fNBEq8X92aAZnlEfxHQNPPt2RuVq5zye8TomTuqOt7HwoFecoZzK6rhclUuxS9FPNU4QmFz5Bk2G0foCdeYzArJpB6cSLl4HxhL7VJiY4j8X39p5nQ+PSbFOLso142BYq4VVGRpGA4DOJwcFEalC9yx6cKJ2MrvXkc1dtLWUJuaxfQdVKHgkl6p6y0YGR4d4SoyNB1hUPCQjocgb8aNk9fwr967EqqSFWD5CN3QHItn81NiGzXLVBsXfRWX1Rlne7Bp+hZUsY2znBPr6yCKtceoW+hDC7j6SX5uO2qQcK5pl4+EoIGz9dxggibKdGK87aVL7xNjMv8Kx9Ta03QtSeVD+lxrUZ6ESxLe9ZfgGJnRopVPqXJ1IzDcQawUlqAOxqp1RW3nPlzUBWpdA4ZfdVgIpJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mpol_shared_policy_init() does not need to use a pseudo-vma: it can use sp_alloc() and sp_insert() directly, since the object's shared policy tree is empty and inaccessible (needing no lock) at get_inode() time. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1d3f9e1ecbb8..5d99fd5cd60b 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2756,30 +2756,30 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) rwlock_init(&sp->lock); if (mpol) { - struct vm_area_struct pvma; - struct mempolicy *new; + struct sp_node *sn; + struct mempolicy *npol; NODEMASK_SCRATCH(scratch); if (!scratch) goto put_mpol; - /* contextualize the tmpfs mount point mempolicy */ - new = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask); - if (IS_ERR(new)) + + /* contextualize the tmpfs mount point mempolicy to this file */ + npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask); + if (IS_ERR(npol)) goto free_scratch; /* no valid nodemask intersection */ task_lock(current); - ret = mpol_set_nodemask(new, &mpol->w.user_nodemask, scratch); + ret = mpol_set_nodemask(npol, &mpol->w.user_nodemask, scratch); task_unlock(current); if (ret) - goto put_new; + goto put_npol; - /* Create pseudo-vma that contains just the policy */ - vma_init(&pvma, NULL); - pvma.vm_end = TASK_SIZE; /* policy covers entire file */ - mpol_set_shared_policy(sp, &pvma, new); /* adds ref */ - -put_new: - mpol_put(new); /* drop initial ref */ + /* alloc node covering entire file; adds ref to file's npol */ + sn = sp_alloc(0, MAX_LFS_FILESIZE >> PAGE_SHIFT, npol); + if (sn) + sp_insert(sp, sn); +put_npol: + mpol_put(npol); /* drop initial ref on file's npol */ free_scratch: NODEMASK_SCRATCH_FREE(scratch); put_mpol: From patchwork Tue Oct 3 09:24:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D12CE75438 for ; Tue, 3 Oct 2023 09:24:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D8CC6B0162; Tue, 3 Oct 2023 05:24:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 388D16B0164; Tue, 3 Oct 2023 05:24:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2512E6B0165; Tue, 3 Oct 2023 05:24:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 174676B0162 for ; Tue, 3 Oct 2023 05:24:24 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E4E701C9FEC for ; Tue, 3 Oct 2023 09:24:23 +0000 (UTC) X-FDA: 81303614406.13.8A90C74 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf28.hostedemail.com (Postfix) with ESMTP id 2D9DEC000C for ; Tue, 3 Oct 2023 09:24:22 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E1GCEx0U; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325062; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ooRbfvgflrDJY0S/ihm6kiwAA3KtmVPaxx1CoMlS90Q=; b=GsGGl72RYYOW8NNWNURQ/dtSJMtsAWh4DCozQTKFSB6faw4OwcFhibntAdc1p9I9mGqX+0 CrcG5sGHkZdwkmU7UppMJDcbgzV2ubTO5p4dgwLEVmTwGQ6wg+jT1WjDE27458YmsY+Z9B BlZBgQXZLZk/lcuVPkyivcXzM7jDq1A= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E1GCEx0U; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325062; a=rsa-sha256; cv=none; b=UA8xOSc7v5FSia+Yho4yE+g1YN+8oITFu3jn3zk/9B1PvM0M2Qwq3LzQJfmvI5UrAyd5ec GK9OwULUJWAjnYgjtJugHsdY/zFvzZgyOf/210RwH/d9S4jEjj+XCQBsLzz6xu6uISjf3N I7ho++oLaeKCMLLlecDzFL24eLu0ZCU= Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-5a21ea6baccso8561607b3.1 for ; Tue, 03 Oct 2023 02:24:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325061; x=1696929861; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ooRbfvgflrDJY0S/ihm6kiwAA3KtmVPaxx1CoMlS90Q=; b=E1GCEx0U+akYpyD58nzae/DFw5IH+roqhL9f0nBJfTDn0d35GohOXfW617tLq/eQ7B 8fyp2W2R4o4otZbgBz/DyzVo0l9cya9GIWJsgnPoYtelREoutG/GSTUUp15jVpL+NgMh tTZrYxw4V+/rL8duxxxMj7b5dbn9GShBYbICxZqzeNIA092FIL4Q1cr25pCGRe9JoLJu 1WF2Sl7NpwDEROZTjCv4JaOhOznGawNJ38XnqFHGutz08HWpEjSFO5QaHXjxGtnzCnI8 kUL35vDuaRoLUczE7tUMjCrabL7Yu7bufB54ULmRc7yY6KIRW+P4C5yV+kA/1GyhgLY/ Ys/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325061; x=1696929861; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ooRbfvgflrDJY0S/ihm6kiwAA3KtmVPaxx1CoMlS90Q=; b=uUuAkUxP2dy+ZXJcaLq0xFD2XIwZCMsajvvAXjUtzemIC81pT5FzhWY/PwePI8Cr4O tNnhgSmB7PE6hFHr/UCmr9oSihp2kUFJieAxLQ3t7U/P887TLa+BdDtZZ50WMX/ZhHmQ FUjXrlWAKbPDVY+kkENG6HrR1ogvEDf5TkKPzS0chfmuaD1MfooZaxrg+kMADjvz5mjz OJY4phwJZnazM8smZ+abaEUXTdm8EzIJZr2S4lDQrk4jXrk+D5unqoXtR9flqPdd3fJg ItJ4bA/J014+DexwDLoRfT/A43NgXAX5QNCD/qMyMlRMSW4g04LgFjxLWzqwxv5HtCRP Erng== X-Gm-Message-State: AOJu0YzIxW0M78i1iu7RofWwm3QEcS5DBok7xw3wJyca7cqJicC1oqvX RHBmLGO306HwjYukiPIc6AQxXw== X-Google-Smtp-Source: AGHT+IF7dS6FmToNAISxPIKOdrXv16fI38OSw83FP5I+O2/LMzV34kyuZxVB4HUMbC6kZMjwwcoTsQ== X-Received: by 2002:a81:a08b:0:b0:58f:ae13:462b with SMTP id x133-20020a81a08b000000b0058fae13462bmr14655078ywg.4.1696325061140; Tue, 03 Oct 2023 02:24:21 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x9-20020a814a09000000b005463e45458bsm251441ywa.123.2023.10.03.02.24.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:24:20 -0700 (PDT) Date: Tue, 3 Oct 2023 02:24:18 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 08/12] mempolicy: remove confusing MPOL_MF_LAZY dead code In-Reply-To: Message-ID: <80c9665c-1c3f-17ba-21a3-f6115cebf7d@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2D9DEC000C X-Stat-Signature: ouwbudbz9op7dk4tfmufar34f6id8u7q X-HE-Tag: 1696325061-92440 X-HE-Meta: U2FsdGVkX1+UdP0Ku36wETYHff2iniRxTZRXPUMUgdnJXeZ7kHDIRPJKyG0jAOkmL7JQF/Yr9gqaoUS4UgDy1F0PbWkej2ArVhsPkQGZoTzVmQimFf4wp84EdJlM4OOTPDGAwRRf32lsrBPzlEFEM5JIfwCuB20UEkhGeH8u8KR61yZDBv46J5WL7MM+1ewCHBjQvz4cm7KwmtVZ8EeB9asejND8JnXKT/lDIub8S9CX2u5oPTIvZksVrt82nKsQXceceDL7sLHgcmYF3O1t79wgtoLIPmhi9z7WjvOwnBZ6QOkKWmdZLmhG6rIziGdoqTpNO1q7WNawe5Qvdfq/XQGMqaWIBowGSO9s6aATFOC3yp7niYXjht2pCp9VnWJIk8cr2VbeRCUBPERkHxZAXmilrEDaBMTA1mOS9N3OMy4Zj2rudj5UykztgAa3MqqUkqQDKH/O91d/GW/sYuoAZ6ctnflRrVyUzepSOiE8jaX0tzGeZdoDxi0PVVoolAvuVzfzrbjLHOgkaXxKHpi1m8ciYQaoyz14UxJXQLO4mD/5WGVDr7IFw63RXiLX01SqzeyvY2wIJIDMksAHzYCSZ86zz1r2x2ug5KV95ZQNvSH7FjdRabk93TKQXQF7FJ5VJLsQxumcI9nGbu1gjL3JP4AlXqVvTFJteG4eXZhbethmgpT9qFXxiUGvJT7CnpzDgV3+ES3Dw+A5VU1G9VseCWihXam4oLZQB8uEs6WngHvrNS3NqDadwSXl8tMQTsQJFH6FuP7UtolkHh8qYCfYTEui03JjCh9Du0AVKDMIkx6TWj8dkVYpec4KL+6Zz+L4ws2GXa5GuEoN3cdW7w3/unBneFwv1Y8ORZFq9AOl/auR95W4UbpgrzZ7ADMpSgOKLF3Id5FR/frrEpwuxVaZXkfFgx8nd+gteaCnIEQwxSA2Lpk5pwX1fdSWzASrhEIxIkM9uSIl85vEmDg1TLd 2/JJgFyY 2jAm/CXUo7x335DhmtLxP6s5wxb5KlxlKsVhWSq24pvqXYBDsTYcCt7qYxKgaseLr6itK4/IZgivkvKCEQlijRB9Vv68eUj0cwE0FdGWsvMueQNpfB8ezhcqjMGu7XztQVA0tvefYhp0cnrWfyiON28ngThMJg6MUFi/OdDyAW1wpwe6/JjV06Wgz8WXtZjRNeFyWlKwS+5Pnuu7OVTB5iC64TMq+OavkOb57XXWRrEOD6byRbJKKpKLXhuapyQy+sCDQHXoyxcTVv8unEJlpJUEEcqKedfRB92+lH/cqb8GHCcVLrQmZaD2kbPoNacwxVR3pyOobrlOrbHz5iIkRVMDDdiKQxh9I5oz9qzeRry4qEC3rWdgUQfiUvclTVmzthSPvDHZ0pdyn+SsC9Ls4mcTa6A8AbOSvZZ2k32NB4fAPN9SsUkt8jPThBYuhnsSPmC7QD797ZdqaUc99ztqBbBhSidYV2FulxmnEJqQ9YrlwqDZ3zV4wBEdJDfzsfMWXMG3iRscQ1rnmH7u/ncN84QVyNLjUMFyuhQp3gLY5Q+mT1l7Ys07bpNPunsVo3puqygvnibLvceJ+TSA4FqybG3L6Ucr5N0hY54mw+CRtrs4Qnaatiy2z57Mt72UM9SsSWagzCXzO12QTzWU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: v3.8 commit b24f53a0bea3 ("mm: mempolicy: Add MPOL_MF_LAZY") introduced MPOL_MF_LAZY, and included it in the MPOL_MF_VALID flags; but a720094ded8 ("mm: mempolicy: Hide MPOL_NOOP and MPOL_MF_LAZY from userspace for now") immediately removed it from MPOL_MF_VALID flags, pending further review. "This will need to be revisited", but it has not been reinstated. The present state is confusing: there is dead code in mm/mempolicy.c to handle MPOL_MF_LAZY cases which can never occur. Remove that: it can be resurrected later if necessary. But keep the definition of MPOL_MF_LAZY, which must remain in the UAPI, even though it always fails with EINVAL. https://lore.kernel.org/linux-mm/1553041659-46787-1-git-send-email-yang.shi@linux.alibaba.com/ links to a previous request to remove MPOL_MF_LAZY. Signed-off-by: Hugh Dickins Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Yang Shi --- include/uapi/linux/mempolicy.h | 2 +- mm/mempolicy.c | 18 ------------------ 2 files changed, 1 insertion(+), 19 deletions(-) diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index 046d0ccba4cd..a8963f7ef4c2 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -48,7 +48,7 @@ enum { #define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform to policy */ #define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */ -#define MPOL_MF_LAZY (1<<3) /* Modifies '_MOVE: lazy migrate on fault */ +#define MPOL_MF_LAZY (1<<3) /* UNSUPPORTED FLAG: Lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 5d99fd5cd60b..f3224a8b0f6c 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -636,12 +636,6 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, return nr_updated; } -#else -static unsigned long change_prot_numa(struct vm_area_struct *vma, - unsigned long addr, unsigned long end) -{ - return 0; -} #endif /* CONFIG_NUMA_BALANCING */ static int queue_pages_test_walk(unsigned long start, unsigned long end, @@ -680,14 +674,6 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, if (endvma > end) endvma = end; - if (flags & MPOL_MF_LAZY) { - /* Similar to task_numa_work, skip inaccessible VMAs */ - if (!is_vm_hugetlb_page(vma) && vma_is_accessible(vma) && - !(vma->vm_flags & VM_MIXEDMAP)) - change_prot_numa(vma, start, endvma); - return 1; - } - /* * Check page nodes, and queue pages to move, in the current vma. * But if no moving, and no strict checking, the scan can be skipped. @@ -1274,9 +1260,6 @@ static long do_mbind(unsigned long start, unsigned long len, if (IS_ERR(new)) return PTR_ERR(new); - if (flags & MPOL_MF_LAZY) - new->flags |= MPOL_F_MOF; - /* * If we are using the default policy then operation * on discontinuous address spaces is okay after all @@ -1321,7 +1304,6 @@ static long do_mbind(unsigned long start, unsigned long len, if (!err) { if (!list_empty(&pagelist)) { - WARN_ON_ONCE(flags & MPOL_MF_LAZY); nr_failed |= migrate_pages(&pagelist, new_folio, NULL, start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); } From patchwork Tue Oct 3 09:25:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14A84E75438 for ; Tue, 3 Oct 2023 09:25:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86E798D0066; Tue, 3 Oct 2023 05:25:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 81DB28D0003; Tue, 3 Oct 2023 05:25:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E8C68D0066; Tue, 3 Oct 2023 05:25:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5F1C38D0003 for ; Tue, 3 Oct 2023 05:25:39 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3488B16021B for ; Tue, 3 Oct 2023 09:25:39 +0000 (UTC) X-FDA: 81303617598.22.3EA884E Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf28.hostedemail.com (Postfix) with ESMTP id 65F58C000C for ; Tue, 3 Oct 2023 09:25:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="tDY/wBKO"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nzmHY2Ka5cz6XnAsRZastGfanbfs6f5GjSGMfdpW808=; b=Sqn4dzDxr474uJX/UD24GPtqLo1pqPL7lN6PBFrJN0PPB6HjmgvoWCCRwPr0+T5k3RniS4 mbdCMWbg8kwqyz4AWwZAcKaIJrJr/eiBJg9+jZVKHw3om/JTqRbvB/nZ4/QzWVsF56uKXD G+e4nDQo0iXoqhDuV7t3BAc+sI7zszE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="tDY/wBKO"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325137; a=rsa-sha256; cv=none; b=s6WAEqD9xE3rlJeUvogqrv3bYp2Ty36pI06/anOBRjewHZK0AJ+H2fNqk95z7VEcwyUj/S SoDbR7aptq316HsIYOAvWMTNAxCrnLP0ojZr+D544RVfCTuvvqVIuzcloGkPD/b2Zb1iI3 P2/CJ89dwb02xWVUrHOUB052SrrnQi0= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-59bf1dde73fso8538027b3.3 for ; Tue, 03 Oct 2023 02:25:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325136; x=1696929936; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=nzmHY2Ka5cz6XnAsRZastGfanbfs6f5GjSGMfdpW808=; b=tDY/wBKON9gX8Uuj8KrGBteW4/OwHiPUo93bRbtNj6tmUbSi/0yoXYnTxIv0Q5ddvr bHobaTqE7Ad+vl6hGuey+gtmXWB9jTDqUcn2+IDxVWoyBXDOJtqSPaCEub+xA4h+iKfd gzPoi78za8Ca4c3TMqtnt3PXcNG2D3HfBv8+8SwvET4pQ4jm0p7bHlFDNcR8/0WWPe5w nr7wiESXkKnV6Gqr0HG0sBwjRXPGHXGTiIr3EVBm5yenuMV7QxaOqJtR7eiDIq9MxzSh FLzYcVD9K52R6FDv7wmz9KrambfiJCeLPCPAapEwOTt/dU9+vewBqsFAcmzOTO1QANjf 0gww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325136; x=1696929936; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nzmHY2Ka5cz6XnAsRZastGfanbfs6f5GjSGMfdpW808=; b=D3auQHHHqLMnAfRXocGQHRwH0fmVCyO/av1HjYy+qSTl4sBmudxPDor6nT00tgFQF6 0nnusA1zUNSoZVZtIHPkX1C9zOXzRaDnayNkG1Q93csv/U3O/GVrAr/NVN81MZCwRKGK oKyxYmbdzOnPT7l/IcPPg1kiSxeL3wIJspbKqJ+Y8h/GJ/kc4oUSGfwJzQb2U1XrgwKx EMYEQKW4iCkbu+eKwLVmOFe0yJvCM3oz1Nz/bZFbGOinBOPw62bgo+XlIKBSSl7+gLRs uTpe4/VIyeajJzdsFUs+k7BQvmeS+v/kCSqskedCGKX34PeuFwyEa8t/pbGLE5PP4vKn v2sQ== X-Gm-Message-State: AOJu0YxTHB0pYGs94sbDq+Le6nBJa03R4vwg78vUYNHraoGAdy4rg3bT y8h4A8z3+KxB/lymvulygEFafA== X-Google-Smtp-Source: AGHT+IGmeBMAQui7nSclUIqDA2Y4gv+lO5peTi3URjoKJb2fITfIn+IGiCa25QjXZ2e92Ix5/drY/A== X-Received: by 2002:a81:4f52:0:b0:599:da80:e1eb with SMTP id d79-20020a814f52000000b00599da80e1ebmr14242966ywb.24.1696325136488; Tue, 03 Oct 2023 02:25:36 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id o19-20020a81de53000000b005925765aa30sm245713ywl.135.2023.10.03.02.25.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:25:35 -0700 (PDT) Date: Tue, 3 Oct 2023 02:25:33 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 09/12] mm: add page_rmappable_folio() wrapper In-Reply-To: Message-ID: <8d92c6cf-eebe-748-e29c-c8ab224c741@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 65F58C000C X-Stat-Signature: 4p6ntcn5y1k1tri4geu16ufejhwrxd6b X-Rspam-User: X-HE-Tag: 1696325137-266751 X-HE-Meta: U2FsdGVkX1/TdbpiDTc2W3ecHgnvsHOcwYaECERe8xS4d9ureOdXigWxfkmL8Y4cm3HSpteiYOkszMFXxJy5c2H/zEMq1NMCZQrWRMHGw+OG/eQCS0qpKcLL7+TIiT/7y66c3T4WHYOt3ltL6CXBpAIloGHG/yLbR/W7z2PNe5d1+mkfKU5tsiLrV8ukCgVa6ROTLQLLTTLy5gfJYyfn4Ry1MBodq7aHz9sZAnDeRHr5GD+BYVTHRlekkcteCqoG6Ep/u8fdOv/BAPj1OYnGacWNyuUY0UIy/mURCTGR6c2gYJ/cIXvWi5ZlIFL6AVboZDhfBoXGueQzBO5apW8IzvNiI24p3sGAj0cJ0vKEKQ6JcDGh3fjW/c2wB6oVSMZF0/SYalhTBNDhzwY3qG0hhox0x/M7mN6C3MNEH3ugJfrocf9FmG9dBe5CO5TOwRAmI+QHdKl61uIl9rg2xhM+z5FeD5HpqvmeXcW3pdCr4GrcxV7KSyOASam3mNpoOyIZv6T4xwJvlITvu0g0RCb/eQXXD76FqYV2XBIafUwS9Hk4qnZD/9gSv0JHsttRFS0hfeJapnO6Iny8lvYXkk9arp5cil1Ocx3MoiEd2GAyWDUprQTzv98p8ocfN/HHixnqGrbYx0g+ej7Q4cU1gwpde/7OtqskxC8Un2ENStccLI729Vv1en48vtnb7qyzioh/Gbe6QjQEc3hmjdA1i1ZIa8WZiM5lZ8IkijuvOXQ+PjTe1rtU5d/eUWw8kNumKCKjgnAGrLB9Mj0HoPvAcwIs6lu7yh8Ah0qjmxroa3m3OX82az0BGYenW4IJvxjJn9Y06npv2mUOv16wOyQ8Rqqklhrpp8OPexUaBX9/00xVrtRz/5GA2ilITDjdmvd3S1mEpRpDAa4SRmvys/jpLGyOYYUaM1iGSo3ILYwOPm87PcZxmpZH4Ok27BIcspX/VMYBf1QxnEHbcIT7EO/01je m4QSUIb6 NB3lkj90NlcXWtpkXkUROtQgySW7hPLYOsOp7xH7wXqPw0VhI95wERVIfOR7Um07GowKrlGZkDqVhxeMe5QLO/1kdPKI1ZuyHHFhj3rSzQgkCrz16df+EOsQ++W44RrbibqqTfjx7qkHEX8q+0250y/JQ/ntlrHP87fiY8yBrg0jAJdnuZLJ/nzIJ3VQCWRfqapey2X5vzzR5e2Uu0EhKWhhraeqftTaM9tlzNMDnKcZZFm/q+yNv20VSTEJ+HNr+dQ4zkcZdNoQR0Rg1CCSV2TjOexQ7nUghIwPLbyQgthvFsrnAC4yGV00prLwImIoEWrbvSZdOkdniGydgkwrN20kVZj4tYMPTuNhCjZXEQIx/zsuaGtjDvwMU7fT+e+/LsQrd8vHNsPyqV2GSwXXsxF73MingDLZG+Dsvpf4yHUgD/tjtkevF4MS1rp1E8w6+oBzTeD2NkZPEVvqAUuljTgrhqA5YGIGJBkurBB5iXIyLuHUV3cqrPYMM8JMmOMC3gtS5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_prep_large_rmappable() is being used repeatedly along with a conversion from page to folio, a check non-NULL, a check order > 1: wrap it all up into struct folio *page_rmappable_folio(struct page *). Signed-off-by: Hugh Dickins --- mm/internal.h | 9 +++++++++ mm/mempolicy.c | 17 +++-------------- mm/page_alloc.c | 8 ++------ 3 files changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index d7916f1e9e98..b2b3716d1df6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -415,6 +415,15 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) void folio_undo_large_rmappable(struct folio *folio); +static inline struct folio *page_rmappable_folio(struct page *page) +{ + struct folio *folio = (struct folio *)page; + + if (folio && folio_order(folio) > 1) + folio_prep_large_rmappable(folio); + return folio; +} + static inline void prep_compound_head(struct page *page, unsigned int order) { struct folio *folio = (struct folio *)page; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index f3224a8b0f6c..bfcc523a2860 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2142,10 +2142,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, mpol_cond_put(pol); gfp |= __GFP_COMP; page = alloc_page_interleave(gfp, order, nid); - folio = (struct folio *)page; - if (folio && order > 1) - folio_prep_large_rmappable(folio); - goto out; + return page_rmappable_folio(page); } if (pol->mode == MPOL_PREFERRED_MANY) { @@ -2155,10 +2152,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, gfp |= __GFP_COMP; page = alloc_pages_preferred_many(gfp, order, node, pol); mpol_cond_put(pol); - folio = (struct folio *)page; - if (folio && order > 1) - folio_prep_large_rmappable(folio); - goto out; + return page_rmappable_folio(page); } if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) { @@ -2252,12 +2246,7 @@ EXPORT_SYMBOL(alloc_pages); struct folio *folio_alloc(gfp_t gfp, unsigned order) { - struct page *page = alloc_pages(gfp | __GFP_COMP, order); - struct folio *folio = (struct folio *)page; - - if (folio && order > 1) - folio_prep_large_rmappable(folio); - return folio; + return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); } EXPORT_SYMBOL(folio_alloc); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7df77b58a961..00f94dd88355 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4619,12 +4619,8 @@ struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page = __alloc_pages(gfp | __GFP_COMP, order, - preferred_nid, nodemask); - struct folio *folio = (struct folio *)page; - - if (folio && order > 1) - folio_prep_large_rmappable(folio); - return folio; + preferred_nid, nodemask); + return page_rmappable_folio(page); } EXPORT_SYMBOL(__folio_alloc); From patchwork Tue Oct 3 09:26:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C2DCE75438 for ; Tue, 3 Oct 2023 09:26:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D779B8E0002; Tue, 3 Oct 2023 05:26:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D27128D0003; Tue, 3 Oct 2023 05:26:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA23A8E0002; Tue, 3 Oct 2023 05:26:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A858E8D0003 for ; Tue, 3 Oct 2023 05:26:48 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6847012021D for ; Tue, 3 Oct 2023 09:26:48 +0000 (UTC) X-FDA: 81303620496.24.E92992A Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 8D0121C0017 for ; Tue, 3 Oct 2023 09:26:46 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jUehyPB5; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325206; a=rsa-sha256; cv=none; b=Sf3qLg4VjqWnQn1KWkEZstjcC3h0HQVjcG89fveUhM/7uDHuhrGBazZtXCK00vEcIttgnC +aQtJUZktOsmsvDnVRPendpi7ruwHIrrgg1owkT4n8bKyI8S4VMiEsbho+nKmm2zlbxQrV 3/cYir5uYm06H4LzjauQC7YfJLtdw9Y= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jUehyPB5; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3qvCe1A/3kG45e5iAe6KCdJKaMNhAMxBLRsWH7ioZ0U=; b=pcmm2IlLKvbMxQTkVvA1TpcGN/mUNXh9xi7hruGL3Wh7BazUqlOvWkYl1XnpZegnYPSp80 68Iz+LcAtJNio54uvTd6yuL3o2eb8HWlJnqD3dQ2JBLCb4cJvErqrNIDBHzcWY6BHzmFvN WFgEkIFpaSplJVYiOyrj3QusIsakl0Q= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-5a2379a8b69so8571707b3.2 for ; Tue, 03 Oct 2023 02:26:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325205; x=1696930005; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=3qvCe1A/3kG45e5iAe6KCdJKaMNhAMxBLRsWH7ioZ0U=; b=jUehyPB5S9y63do690JLc1pv6VPPy4pH5MA70V1K7VXvfHo5G3BJA5PJsdtT8n1o7G iWYWutcbUOW3Lifj9QD7v/VRNi3JWCiiuREm+x2qq5+eiDamOaUr9HcQJq9Os+Z9+Pmt 2Phvv8OaqIVhGviLNSmbAkkFXddEq0MAtKS9ydgUEYHaGRXbh/RkOIU1oqmGljv4r6Qv qXLt+0zJoJu7lJO2kWTuxQ2srAORv5iam36Kcb5wHTOMYEKMCxPVOGVVNrUr2rmmDpid u0wU3uI+yAb4vhraMUbPJQWxy+Dn+N3946rmq50FA93oM2DXUrCq6qdIbLX9KzKH1UNB ceFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325205; x=1696930005; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3qvCe1A/3kG45e5iAe6KCdJKaMNhAMxBLRsWH7ioZ0U=; b=rIPighUCHh1aqbFhWvrbzVvg32oPerEoQArwp8CuZnvNrNzBfMNgfOXi55MiFEi/AO NMUo2h5Ckb5NFNnMrete37TaeJKRRvqT7/U50w1gY2hp/GRxw9Xmt4VRyQgb+xVcOo/3 poIjeu1VYlnCw4jtUcRkYC2osLCyRpdtqrRpeza79itaI1twTPQWnTmtlkvhL8dSwOKb bRXh8lkWHpSUzptP7cIl1M3wi12H0IkDPU9mwRj2mIU+fBabtGcxn0d+HhZBmSqcmREv f3AvpG/fwjM7K6oKuoUatpu1NX2enKXMrsquSqLJ0sN/xfG0cukf2L3I/BInRmfxfUjD mr2g== X-Gm-Message-State: AOJu0Yx99UqlJpk2IKICzHdcQZsSpe6Kdnw6EzRlGS4DtLyaAENLCiNK i0EJbqVZbPd86nWT0Bw3/pkZdg== X-Google-Smtp-Source: AGHT+IEiJBMPsx5lZyUfrAW2d97Mf6lLD0VXN7F7UoDg63fILqX2qt+2gRYWSAcMyi0o5mfws8q+uA== X-Received: by 2002:a0d:eb16:0:b0:59c:8b3:6850 with SMTP id u22-20020a0deb16000000b0059c08b36850mr14504709ywe.4.1696325205371; Tue, 03 Oct 2023 02:26:45 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id h4-20020a0df704000000b0058fc7604f45sm252385ywf.130.2023.10.03.02.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:26:44 -0700 (PDT) Date: Tue, 3 Oct 2023 02:26:42 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 10/12] mempolicy: alloc_pages_mpol() for NUMA policy without vma In-Reply-To: Message-ID: <74e34633-6060-f5e3-aee-7040d43f2e93@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8D0121C0017 X-Stat-Signature: x5kfhwyn7nsk1s9nzwcbuqhxxbx37wn9 X-Rspam-User: X-HE-Tag: 1696325206-105242 X-HE-Meta: U2FsdGVkX1+MFX/aaCZ7GD3QSoRHCDlCzieVI7W3N1HH9UboqHO1DvE2M3t6Pya5L3k1Q8OOg665CEXVmaVZSzaPUjB/I+sRpdZTf+eAiWwdp1Xoau8UHbfNZBPDkcLeHMkysKVBaOWASS963+m+AUP34oE8VlWZ8YektQSVBCc1wpongCjsBQdhZxzWPWzn+HghltDTih8RBzwVjy5jvMsb8cbO+MD1BBWIpXDTBas+JxhmeLHMolYxDAyTVQjt2NzQwixqB4rm1gfYv2nIU6X8ywA/UtIdjQwCKIy0170XqXIr1TvfO4c7v5RyHc38W5BkIFgemCSaADf/5Ckxz5mK5ANBJfgYe67pJpuvCVjudU94cshhElseED89nPTIldZvcf9sfbv1e39d7HlLPlaYVnl9eAc4NR5cb11t0LCN+XnFudTFCQGCdlRYNJWCxZe2LfZtNtsdCHlKHnCtnRrime6OdVYKrD5Hv47KufbbU3MWp9/lNpQhLjFMCvMFuFeouDgYPIjUdQNr355/lFN3m5abYt/kVlqd/PaC1LlnVbq7NIBP4yBSF+SGAvbgifKBeD34M0F0AUihgRAIOj39dbxYAGRHwAmXW3itsYC4GuABY+sZXA3MM3qkTQta5d+nYGeJbfduGCBUBfwvI1/WnT+eOPaMXJQ/Dz0BJwlIL3yCln5ssrknf6DgpUxeXPXXKmHSZisaA2QCaex9jJ4Ud7QttfMxQNMlZ+7fT/TuDO0lpOkUSFm1/OU3XsftDhRcDUtOF4W6M1F1AHfGtUPNe6QGhZmfutnp+6//nMptoJgY1oZjRR5/hffdeBXaI6Igak/daYvspqq/vxCKin753RDuH5TC878PCbVCdVkcc7f6wsG3ghBAkYTaOjERvdeSAVlCQM2nr244RD35d1PILlYhEAx4RXsZsHyLmWktLQ6a86amzQPf699d6CRjcDS9vuHTAW15Xs2psVM nQd0k4E+ V2TRBsQUQVicnhLqISHGfYYQV1hmr+2T/5oqVd47e7jOimAZJxGFh7MhAVLMicFra6CpxY81bFk3HnlLxEsFxyvRAO8muuQntGwPLZFf/v+oVijfe/QyqnpY2QmwWrNXFYXgGombBe1GLHLQASoli5QDiznQbiPaJ8LFDd60rPltbbzCpav/1OZtpfOZRW0M7GvzmZ4HdjnUCbfHb9hwFNINu/+88J35ftsiU9gVknV6Cae8JvzvVC8YaeGgbB6qjiUtimvTxPmPGtB26xWHCwYnLCAZLG2xJG198r9yrGZJ3T7obz+5wc2BsyhAZgjo8gb3/6WW7/kv0qzv0a9sPSVpgKO01oHCYJpnRAZIQlwIBWlD2u9Ptcdjs0VgLqVL4emR9gZpYCEXxRB0A7ca7tUaevOmT/rXBTqmr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Shrink shmem's stack usage by eliminating the pseudo-vma from its folio allocation. alloc_pages_mpol(gfp, order, pol, ilx, nid) becomes the principal actor for passing mempolicy choice down to __alloc_pages(), rather than vma_alloc_folio(gfp, order, vma, addr, hugepage). vma_alloc_folio() and alloc_pages() remain, but as wrappers around alloc_pages_mpol(). alloc_pages_bulk_*() untouched, except to provide the additional args to policy_nodemask(), which subsumes policy_node(). Cleanup throughout, cutting out some unhelpful "helpers". It would all be much simpler without MPOL_INTERLEAVE, but that adds a dynamic to the constant mpol: complicated by v3.6 commit 09c231cb8bfd ("tmpfs: distribute interleave better across nodes"), which added ino bias to the interleave, hidden from mm/mempolicy.c until this commit. Hence "ilx" throughout, the "interleave index". Originally I thought it could be done just with nid, but that's wrong: the nodemask may come from the shared policy layer below a shmem vma, or it may come from the task layer above a shmem vma; and without the final nodemask then nodeid cannot be decided. And how ilx is applied depends also on page order. The interleave index is almost always irrelevant unless MPOL_INTERLEAVE: with one exception in alloc_pages_mpol(), where the NO_INTERLEAVE_INDEX passed down from vma-less alloc_pages() is also used as hint not to use THP-style hugepage allocation - to avoid the overhead of a hugepage arg (though I don't understand why we never just added a GFP bit for THP - if it actually needs a different allocation strategy from other pages of the same order). vma_alloc_folio() still carries its hugepage arg here, but it is not used, and should be removed when agreed. get_vma_policy() no longer allows a NULL vma: over time I believe we've eradicated all the places which used to need it e.g. swapoff and madvise used to pass NULL vma to read_swap_cache_async(), but now know the vma. Signed-off-by: Hugh Dickins --- fs/proc/task_mmu.c | 5 +- include/linux/gfp.h | 10 +- include/linux/mempolicy.h | 13 +- include/linux/mm.h | 2 +- ipc/shm.c | 21 +-- mm/mempolicy.c | 383 +++++++++++++++++----------------------- mm/shmem.c | 92 +++++----- mm/swap.h | 9 +- mm/swap_state.c | 83 +++++---- 9 files changed, 297 insertions(+), 321 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5a302f1b8d68..6cc4720eabf0 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2637,8 +2637,9 @@ static int show_numa_map(struct seq_file *m, void *v) struct numa_maps *md = &numa_priv->md; struct file *file = vma->vm_file; struct mm_struct *mm = vma->vm_mm; - struct mempolicy *pol; char buffer[64]; + struct mempolicy *pol; + pgoff_t ilx; int nid; if (!mm) @@ -2647,7 +2648,7 @@ static int show_numa_map(struct seq_file *m, void *v) /* Ensure we start with an empty set of numa_maps statistics. */ memset(md, 0, sizeof(*md)); - pol = __get_vma_policy(vma, vma->vm_start); + pol = __get_vma_policy(vma, vma->vm_start, &ilx); if (pol) { mpol_to_str(buffer, sizeof(buffer), pol); mpol_cond_put(pol); diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 5b917e5b9350..de292a007138 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -8,6 +8,7 @@ #include struct vm_area_struct; +struct mempolicy; /* Convert GFP flags to their corresponding migrate type */ #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) @@ -262,7 +263,9 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, #ifdef CONFIG_NUMA struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct folio *folio_alloc(gfp_t gfp, unsigned order); +struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, + struct mempolicy *mpol, pgoff_t ilx, int nid); +struct folio *folio_alloc(gfp_t gfp, unsigned int order); struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else @@ -270,6 +273,11 @@ static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) { return alloc_pages_node(numa_node_id(), gfp_mask, order); } +static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, + struct mempolicy *mpol, pgoff_t ilx, int nid) +{ + return alloc_pages(gfp, order); +} static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index c69f9480d5e4..3c208d4f0ee9 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -128,7 +128,9 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, struct mempolicy *get_task_policy(struct task_struct *p); struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, - unsigned long addr); + unsigned long addr, pgoff_t *ilx); +struct mempolicy *get_vma_policy(struct vm_area_struct *vma, + unsigned long addr, int order, pgoff_t *ilx); bool vma_policy_mof(struct vm_area_struct *vma); extern void numa_default_policy(void); @@ -142,8 +144,6 @@ extern int huge_node(struct vm_area_struct *vma, extern bool init_nodemask_of_mempolicy(nodemask_t *mask); extern bool mempolicy_in_oom_domain(struct task_struct *tsk, const nodemask_t *mask); -extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy); - extern unsigned int mempolicy_slab_node(void); extern enum zone_type policy_zone; @@ -215,6 +215,13 @@ mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx) return NULL; } +static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma, + unsigned long addr, int order, pgoff_t *ilx) +{ + *ilx = 0; + return NULL; +} + #define vma_policy(vma) NULL static inline int diff --git a/include/linux/mm.h b/include/linux/mm.h index 52c40b3d0813..9b86a9c35427 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -619,7 +619,7 @@ struct vm_operations_struct { * policy. */ struct mempolicy *(*get_policy)(struct vm_area_struct *vma, - unsigned long addr); + unsigned long addr, pgoff_t *ilx); #endif /* * Called by vm_normal_page() for special PTEs to find the diff --git a/ipc/shm.c b/ipc/shm.c index 576a543b7cff..222aaf035afb 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -562,30 +562,25 @@ static unsigned long shm_pagesize(struct vm_area_struct *vma) } #ifdef CONFIG_NUMA -static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *new) +static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol) { - struct file *file = vma->vm_file; - struct shm_file_data *sfd = shm_file_data(file); + struct shm_file_data *sfd = shm_file_data(vma->vm_file); int err = 0; if (sfd->vm_ops->set_policy) - err = sfd->vm_ops->set_policy(vma, new); + err = sfd->vm_ops->set_policy(vma, mpol); return err; } static struct mempolicy *shm_get_policy(struct vm_area_struct *vma, - unsigned long addr) + unsigned long addr, pgoff_t *ilx) { - struct file *file = vma->vm_file; - struct shm_file_data *sfd = shm_file_data(file); - struct mempolicy *pol = NULL; + struct shm_file_data *sfd = shm_file_data(vma->vm_file); + struct mempolicy *mpol = vma->vm_policy; if (sfd->vm_ops->get_policy) - pol = sfd->vm_ops->get_policy(vma, addr); - else if (vma->vm_policy) - pol = vma->vm_policy; - - return pol; + mpol = sfd->vm_ops->get_policy(vma, addr, ilx); + return mpol; } #endif diff --git a/mm/mempolicy.c b/mm/mempolicy.c index bfcc523a2860..8cf76de12acd 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -114,6 +114,8 @@ #define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */ #define MPOL_MF_WRLOCK (MPOL_MF_INTERNAL << 2) /* Write-lock walked vmas */ +#define NO_INTERLEAVE_INDEX (-1UL) + static struct kmem_cache *policy_cache; static struct kmem_cache *sn_cache; @@ -918,6 +920,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, } if (flags & MPOL_F_ADDR) { + pgoff_t ilx; /* ignored here */ /* * Do NOT fall back to task policy if the * vma/shared policy at addr is NULL. We @@ -929,10 +932,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, mmap_read_unlock(mm); return -EFAULT; } - if (vma->vm_ops && vma->vm_ops->get_policy) - pol = vma->vm_ops->get_policy(vma, addr); - else - pol = vma->vm_policy; + pol = __get_vma_policy(vma, addr, &ilx); } else if (addr) return -EINVAL; @@ -1190,6 +1190,15 @@ static struct folio *new_folio(struct folio *src, unsigned long start) break; } + /* + * __get_vma_policy() now expects a genuine non-NULL vma. Return NULL + * when the page can no longer be located in a vma: that is not ideal + * (migrate_pages() will give up early, presuming ENOMEM), but good + * enough to avoid a crash by syzkaller or concurrent holepunch. + */ + if (!vma) + return NULL; + if (folio_test_hugetlb(src)) { return alloc_hugetlb_folio_vma(folio_hstate(src), vma, address); @@ -1198,9 +1207,6 @@ static struct folio *new_folio(struct folio *src, unsigned long start) if (folio_test_large(src)) gfp = GFP_TRANSHUGE; - /* - * if !vma, vma_alloc_folio() will use task or system default policy - */ return vma_alloc_folio(gfp, folio_order(src), vma, address, folio_test_large(src)); } @@ -1710,34 +1716,19 @@ bool vma_migratable(struct vm_area_struct *vma) } struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, - unsigned long addr) + unsigned long addr, pgoff_t *ilx) { - struct mempolicy *pol = NULL; - - if (vma) { - if (vma->vm_ops && vma->vm_ops->get_policy) { - pol = vma->vm_ops->get_policy(vma, addr); - } else if (vma->vm_policy) { - pol = vma->vm_policy; - - /* - * shmem_alloc_page() passes MPOL_F_SHARED policy with - * a pseudo vma whose vma->vm_ops=NULL. Take a reference - * count on these policies which will be dropped by - * mpol_cond_put() later - */ - if (mpol_needs_cond_ref(pol)) - mpol_get(pol); - } - } - - return pol; + *ilx = 0; + return (vma->vm_ops && vma->vm_ops->get_policy) ? + vma->vm_ops->get_policy(vma, addr, ilx) : vma->vm_policy; } /* - * get_vma_policy(@vma, @addr) + * get_vma_policy(@vma, @addr, @order, @ilx) * @vma: virtual memory area whose policy is sought * @addr: address in @vma for shared policy lookup + * @order: 0, or appropriate huge_page_order for interleaving + * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE * * Returns effective policy for a VMA at specified address. * Falls back to current->mempolicy or system default policy, as necessary. @@ -1746,14 +1737,18 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, * freeing by another task. It is the caller's responsibility to free the * extra reference for shared policies. */ -static struct mempolicy *get_vma_policy(struct vm_area_struct *vma, - unsigned long addr) +struct mempolicy *get_vma_policy(struct vm_area_struct *vma, + unsigned long addr, int order, pgoff_t *ilx) { - struct mempolicy *pol = __get_vma_policy(vma, addr); + struct mempolicy *pol; + pol = __get_vma_policy(vma, addr, ilx); if (!pol) pol = get_task_policy(current); - + if (pol->mode == MPOL_INTERLEAVE) { + *ilx += vma->vm_pgoff >> order; + *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); + } return pol; } @@ -1763,8 +1758,9 @@ bool vma_policy_mof(struct vm_area_struct *vma) if (vma->vm_ops && vma->vm_ops->get_policy) { bool ret = false; + pgoff_t ilx; /* ignored here */ - pol = vma->vm_ops->get_policy(vma, vma->vm_start); + pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx); if (pol && (pol->flags & MPOL_F_MOF)) ret = true; mpol_cond_put(pol); @@ -1799,54 +1795,6 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) return zone >= dynamic_policy_zone; } -/* - * Return a nodemask representing a mempolicy for filtering nodes for - * page allocation - */ -nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy) -{ - int mode = policy->mode; - - /* Lower zones don't get a nodemask applied for MPOL_BIND */ - if (unlikely(mode == MPOL_BIND) && - apply_policy_zone(policy, gfp_zone(gfp)) && - cpuset_nodemask_valid_mems_allowed(&policy->nodes)) - return &policy->nodes; - - if (mode == MPOL_PREFERRED_MANY) - return &policy->nodes; - - return NULL; -} - -/* - * Return the preferred node id for 'prefer' mempolicy, and return - * the given id for all other policies. - * - * policy_node() is always coupled with policy_nodemask(), which - * secures the nodemask limit for 'bind' and 'prefer-many' policy. - */ -static int policy_node(gfp_t gfp, struct mempolicy *policy, int nid) -{ - if (policy->mode == MPOL_PREFERRED) { - nid = first_node(policy->nodes); - } else { - /* - * __GFP_THISNODE shouldn't even be used with the bind policy - * because we might easily break the expectation to stay on the - * requested node and not break the policy. - */ - WARN_ON_ONCE(policy->mode == MPOL_BIND && (gfp & __GFP_THISNODE)); - } - - if ((policy->mode == MPOL_BIND || - policy->mode == MPOL_PREFERRED_MANY) && - policy->home_node != NUMA_NO_NODE) - return policy->home_node; - - return nid; -} - /* Do dynamic interleaving for a process */ static unsigned int interleave_nodes(struct mempolicy *policy) { @@ -1906,11 +1854,11 @@ unsigned int mempolicy_slab_node(void) } /* - * Do static interleaving for a VMA with known offset @n. Returns the n'th - * node in pol->nodes (starting from n=0), wrapping around if n exceeds the - * number of present nodes. + * Do static interleaving for interleave index @ilx. Returns the ilx'th + * node in pol->nodes (starting from ilx=0), wrapping around if ilx + * exceeds the number of present nodes. */ -static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) +static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx) { nodemask_t nodemask = pol->nodes; unsigned int target, nnodes; @@ -1928,33 +1876,54 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) nnodes = nodes_weight(nodemask); if (!nnodes) return numa_node_id(); - target = (unsigned int)n % nnodes; + target = ilx % nnodes; nid = first_node(nodemask); for (i = 0; i < target; i++) nid = next_node(nid, nodemask); return nid; } -/* Determine a node number for interleave */ -static inline unsigned interleave_nid(struct mempolicy *pol, - struct vm_area_struct *vma, unsigned long addr, int shift) +/* + * Return a nodemask representing a mempolicy for filtering nodes for + * page allocation, together with preferred node id (or the input node id). + */ +static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, + pgoff_t ilx, int *nid) { - if (vma) { - unsigned long off; + nodemask_t *nodemask = NULL; + switch (pol->mode) { + case MPOL_PREFERRED: + /* Override input node id */ + *nid = first_node(pol->nodes); + break; + case MPOL_PREFERRED_MANY: + nodemask = &pol->nodes; + if (pol->home_node != NUMA_NO_NODE) + *nid = pol->home_node; + break; + case MPOL_BIND: + /* Restrict to nodemask (but not on lower zones) */ + if (apply_policy_zone(pol, gfp_zone(gfp)) && + cpuset_nodemask_valid_mems_allowed(&pol->nodes)) + nodemask = &pol->nodes; + if (pol->home_node != NUMA_NO_NODE) + *nid = pol->home_node; /* - * for small pages, there is no difference between - * shift and PAGE_SHIFT, so the bit-shift is safe. - * for huge pages, since vm_pgoff is in units of small - * pages, we need to shift off the always 0 bits to get - * a useful offset. + * __GFP_THISNODE shouldn't even be used with the bind policy + * because we might easily break the expectation to stay on the + * requested node and not break the policy. */ - BUG_ON(shift < PAGE_SHIFT); - off = vma->vm_pgoff >> (shift - PAGE_SHIFT); - off += (addr - vma->vm_start) >> shift; - return offset_il_node(pol, off); - } else - return interleave_nodes(pol); + WARN_ON_ONCE(gfp & __GFP_THISNODE); + break; + case MPOL_INTERLEAVE: + /* Override input node id */ + *nid = (ilx == NO_INTERLEAVE_INDEX) ? + interleave_nodes(pol) : interleave_nid(pol, ilx); + break; + } + + return nodemask; } #ifdef CONFIG_HUGETLBFS @@ -1970,27 +1939,16 @@ static inline unsigned interleave_nid(struct mempolicy *pol, * to the struct mempolicy for conditional unref after allocation. * If the effective policy is 'bind' or 'prefer-many', returns a pointer * to the mempolicy's @nodemask for filtering the zonelist. - * - * Must be protected by read_mems_allowed_begin() */ int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags, - struct mempolicy **mpol, nodemask_t **nodemask) + struct mempolicy **mpol, nodemask_t **nodemask) { + pgoff_t ilx; int nid; - int mode; - *mpol = get_vma_policy(vma, addr); - *nodemask = NULL; - mode = (*mpol)->mode; - - if (unlikely(mode == MPOL_INTERLEAVE)) { - nid = interleave_nid(*mpol, vma, addr, - huge_page_shift(hstate_vma(vma))); - } else { - nid = policy_node(gfp_flags, *mpol, numa_node_id()); - if (mode == MPOL_BIND || mode == MPOL_PREFERRED_MANY) - *nodemask = &(*mpol)->nodes; - } + nid = numa_node_id(); + *mpol = get_vma_policy(vma, addr, hstate_vma(vma)->order, &ilx); + *nodemask = policy_nodemask(gfp_flags, *mpol, ilx, &nid); return nid; } @@ -2068,27 +2026,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk, return ret; } -/* Allocate a page in interleaved policy. - Own path because it needs to do special accounting. */ -static struct page *alloc_page_interleave(gfp_t gfp, unsigned order, - unsigned nid) -{ - struct page *page; - - page = __alloc_pages(gfp, order, nid, NULL); - /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */ - if (!static_branch_likely(&vm_numa_stat_key)) - return page; - if (page && page_to_nid(page) == nid) { - preempt_disable(); - __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT); - preempt_enable(); - } - return page; -} - static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, - int nid, struct mempolicy *pol) + int nid, nodemask_t *nodemask) { struct page *page; gfp_t preferred_gfp; @@ -2101,7 +2040,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes); + page = __alloc_pages(preferred_gfp, order, nid, nodemask); if (!page) page = __alloc_pages(gfp, order, nid, NULL); @@ -2109,55 +2048,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. - * @order: Order of the folio. - * @vma: Pointer to VMA or NULL if not available. - * @addr: Virtual address of the allocation. Must be inside @vma. - * @hugepage: For hugepages try only the preferred node if possible. + * @order: Order of the page allocation. + * @pol: Pointer to the NUMA mempolicy. + * @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()). + * @nid: Preferred node (usually numa_node_id() but @mpol may override it). * - * Allocate a folio for a specific address in @vma, using the appropriate - * NUMA policy. When @vma is not NULL the caller must hold the mmap_lock - * of the mm_struct of the VMA to prevent it from going away. Should be - * used for all allocations for folios that will be mapped into user space. - * - * Return: The folio on success or NULL if allocation fails. + * Return: The page on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, - unsigned long addr, bool hugepage) +struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, + struct mempolicy *pol, pgoff_t ilx, int nid) { - struct mempolicy *pol; - int node = numa_node_id(); - struct folio *folio; - int preferred_nid; - nodemask_t *nmask; + nodemask_t *nodemask; + struct page *page; - pol = get_vma_policy(vma, addr); + nodemask = policy_nodemask(gfp, pol, ilx, &nid); - if (pol->mode == MPOL_INTERLEAVE) { - struct page *page; - unsigned nid; - - nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order); - mpol_cond_put(pol); - gfp |= __GFP_COMP; - page = alloc_page_interleave(gfp, order, nid); - return page_rmappable_folio(page); - } - - if (pol->mode == MPOL_PREFERRED_MANY) { - struct page *page; - - node = policy_node(gfp, pol, node); - gfp |= __GFP_COMP; - page = alloc_pages_preferred_many(gfp, order, node, pol); - mpol_cond_put(pol); - return page_rmappable_folio(page); - } - - if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) { - int hpage_node = node; + if (pol->mode == MPOL_PREFERRED_MANY) + return alloc_pages_preferred_many(gfp, order, nid, nodemask); + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + /* filter "hugepage" allocation, unless from alloc_pages() */ + order == HPAGE_PMD_ORDER && ilx != NO_INTERLEAVE_INDEX) { /* * For hugepage allocation and non-interleave policy which * allows the current node (or other explicitly preferred @@ -2168,39 +2081,68 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, * If the policy is interleave or does not allow the current * node in its nodemask, we allocate the standard way. */ - if (pol->mode == MPOL_PREFERRED) - hpage_node = first_node(pol->nodes); - - nmask = policy_nodemask(gfp, pol); - if (!nmask || node_isset(hpage_node, *nmask)) { - mpol_cond_put(pol); + if (pol->mode != MPOL_INTERLEAVE && + (!nodemask || node_isset(nid, *nodemask))) { /* * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - folio = __folio_alloc_node(gfp | __GFP_THISNODE | - __GFP_NORETRY, order, hpage_node); - + page = __alloc_pages_node(nid, + gfp | __GFP_THISNODE | __GFP_NORETRY, order); + if (page || !(gfp & __GFP_DIRECT_RECLAIM)) + return page; /* * If hugepage allocations are configured to always * synchronous compact or the vma has been madvised * to prefer hugepage backing, retry allowing remote * memory with both reclaim and compact as well. */ - if (!folio && (gfp & __GFP_DIRECT_RECLAIM)) - folio = __folio_alloc(gfp, order, hpage_node, - nmask); - - goto out; } } - nmask = policy_nodemask(gfp, pol); - preferred_nid = policy_node(gfp, pol, node); - folio = __folio_alloc(gfp, order, preferred_nid, nmask); + page = __alloc_pages(gfp, order, nid, nodemask); + + if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { + /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ + if (static_branch_likely(&vm_numa_stat_key) && + page_to_nid(page) == nid) { + preempt_disable(); + __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT); + preempt_enable(); + } + } + + return page; +} + +/** + * vma_alloc_folio - Allocate a folio for a VMA. + * @gfp: GFP flags. + * @order: Order of the folio. + * @vma: Pointer to VMA. + * @addr: Virtual address of the allocation. Must be inside @vma. + * @hugepage: Unused (was: For hugepages try only preferred node if possible). + * + * Allocate a folio for a specific address in @vma, using the appropriate + * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the + * VMA to prevent it from going away. Should be used for all allocations + * for folios that will be mapped into user space, excepting hugetlbfs, and + * excepting where direct use of alloc_pages_mpol() is more appropriate. + * + * Return: The folio on success or NULL if allocation fails. + */ +struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, + unsigned long addr, bool hugepage) +{ + struct mempolicy *pol; + pgoff_t ilx; + struct page *page; + + pol = get_vma_policy(vma, addr, order, &ilx); + page = alloc_pages_mpol(gfp | __GFP_COMP, order, + pol, ilx, numa_node_id()); mpol_cond_put(pol); -out: - return folio; + return page_rmappable_folio(page); } EXPORT_SYMBOL(vma_alloc_folio); @@ -2218,33 +2160,23 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned order) +struct page *alloc_pages(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; - struct page *page; - - if (!in_interrupt() && !(gfp & __GFP_THISNODE)) - pol = get_task_policy(current); /* * No reference counting needed for current->mempolicy * nor system default_policy */ - if (pol->mode == MPOL_INTERLEAVE) - page = alloc_page_interleave(gfp, order, interleave_nodes(pol)); - else if (pol->mode == MPOL_PREFERRED_MANY) - page = alloc_pages_preferred_many(gfp, order, - policy_node(gfp, pol, numa_node_id()), pol); - else - page = __alloc_pages(gfp, order, - policy_node(gfp, pol, numa_node_id()), - policy_nodemask(gfp, pol)); + if (!in_interrupt() && !(gfp & __GFP_THISNODE)) + pol = get_task_policy(current); - return page; + return alloc_pages_mpol(gfp, order, + pol, NO_INTERLEAVE_INDEX, numa_node_id()); } EXPORT_SYMBOL(alloc_pages); -struct folio *folio_alloc(gfp_t gfp, unsigned order) +struct folio *folio_alloc(gfp_t gfp, unsigned int order) { return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); } @@ -2315,6 +2247,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; + nodemask_t *nodemask; + int nid; if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); @@ -2327,9 +2261,10 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, return alloc_pages_bulk_array_preferred_many(gfp, numa_node_id(), pol, nr_pages, page_array); - return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), - policy_nodemask(gfp, pol), nr_pages, NULL, - page_array); + nid = numa_node_id(); + nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); + return __alloc_pages_bulk(gfp, nid, nodemask, + nr_pages, NULL, page_array); } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) @@ -2516,23 +2451,21 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma, unsigned long addr) { struct mempolicy *pol; + pgoff_t ilx; struct zoneref *z; int curnid = folio_nid(folio); - unsigned long pgoff; int thiscpu = raw_smp_processor_id(); int thisnid = cpu_to_node(thiscpu); int polnid = NUMA_NO_NODE; int ret = NUMA_NO_NODE; - pol = get_vma_policy(vma, addr); + pol = get_vma_policy(vma, addr, folio_order(folio), &ilx); if (!(pol->flags & MPOL_F_MOF)) goto out; switch (pol->mode) { case MPOL_INTERLEAVE: - pgoff = vma->vm_pgoff; - pgoff += (addr - vma->vm_start) >> PAGE_SHIFT; - polnid = offset_il_node(pol, pgoff); + polnid = interleave_nid(pol, ilx); break; case MPOL_PREFERRED: diff --git a/mm/shmem.c b/mm/shmem.c index a3ec5d2dda9a..6503910b0f54 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1561,38 +1561,20 @@ static inline struct mempolicy *shmem_get_sbmpol(struct shmem_sb_info *sbinfo) return NULL; } #endif /* CONFIG_NUMA && CONFIG_TMPFS */ -#ifndef CONFIG_NUMA -#define vm_policy vm_private_data -#endif -static void shmem_pseudo_vma_init(struct vm_area_struct *vma, - struct shmem_inode_info *info, pgoff_t index) -{ - /* Create a pseudo vma that just contains the policy */ - vma_init(vma, NULL); - /* Bias interleave by inode number to distribute better across nodes */ - vma->vm_pgoff = index + info->vfs_inode.i_ino; - vma->vm_policy = mpol_shared_policy_lookup(&info->policy, index); -} +static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info, + pgoff_t index, unsigned int order, pgoff_t *ilx); -static void shmem_pseudo_vma_destroy(struct vm_area_struct *vma) -{ - /* Drop reference taken by mpol_shared_policy_lookup() */ - mpol_cond_put(vma->vm_policy); -} - -static struct folio *shmem_swapin(swp_entry_t swap, gfp_t gfp, +static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct vm_area_struct pvma; + struct mempolicy *mpol; + pgoff_t ilx; struct page *page; - struct vm_fault vmf = { - .vma = &pvma, - }; - shmem_pseudo_vma_init(&pvma, info, index); - page = swap_cluster_readahead(swap, gfp, &vmf); - shmem_pseudo_vma_destroy(&pvma); + mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); + page = swap_cluster_readahead(swap, gfp, mpol, ilx); + mpol_cond_put(mpol); if (!page) return NULL; @@ -1626,27 +1608,29 @@ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) static struct folio *shmem_alloc_hugefolio(gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct vm_area_struct pvma; - struct folio *folio; + struct mempolicy *mpol; + pgoff_t ilx; + struct page *page; - shmem_pseudo_vma_init(&pvma, info, index); - folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, &pvma, 0, true); - shmem_pseudo_vma_destroy(&pvma); + mpol = shmem_get_pgoff_policy(info, index, HPAGE_PMD_ORDER, &ilx); + page = alloc_pages_mpol(gfp, HPAGE_PMD_ORDER, mpol, ilx, numa_node_id()); + mpol_cond_put(mpol); - return folio; + return page_rmappable_folio(page); } static struct folio *shmem_alloc_folio(gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct vm_area_struct pvma; - struct folio *folio; + struct mempolicy *mpol; + pgoff_t ilx; + struct page *page; - shmem_pseudo_vma_init(&pvma, info, index); - folio = vma_alloc_folio(gfp, 0, &pvma, 0, false); - shmem_pseudo_vma_destroy(&pvma); + mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); + page = alloc_pages_mpol(gfp, 0, mpol, ilx, numa_node_id()); + mpol_cond_put(mpol); - return folio; + return (struct folio *)page; } static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, @@ -1900,7 +1884,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, count_memcg_event_mm(fault_mm, PGMAJFAULT); } /* Here we actually start the io */ - folio = shmem_swapin(swap, gfp, info, index); + folio = shmem_swapin_cluster(swap, gfp, info, index); if (!folio) { error = -ENOMEM; goto failed; @@ -2351,15 +2335,41 @@ static int shmem_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol) } static struct mempolicy *shmem_get_policy(struct vm_area_struct *vma, - unsigned long addr) + unsigned long addr, pgoff_t *ilx) { struct inode *inode = file_inode(vma->vm_file); pgoff_t index; + /* + * Bias interleave by inode number to distribute better across nodes; + * but this interface is independent of which page order is used, so + * supplies only that bias, letting caller apply the offset (adjusted + * by page order, as in shmem_get_pgoff_policy() and get_vma_policy()). + */ + *ilx = inode->i_ino; index = ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff; return mpol_shared_policy_lookup(&SHMEM_I(inode)->policy, index); } -#endif + +static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info, + pgoff_t index, unsigned int order, pgoff_t *ilx) +{ + struct mempolicy *mpol; + + /* Bias interleave by inode number to distribute better across nodes */ + *ilx = info->vfs_inode.i_ino + (index >> order); + + mpol = mpol_shared_policy_lookup(&info->policy, index); + return mpol ? mpol : get_task_policy(current); +} +#else +static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info, + pgoff_t index, unsigned int order, pgoff_t *ilx) +{ + *ilx = 0; + return NULL; +} +#endif /* CONFIG_NUMA */ int shmem_lock(struct file *file, int lock, struct ucounts *ucounts) { diff --git a/mm/swap.h b/mm/swap.h index 8a3c7a0ace4f..73c332ee4d91 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -2,6 +2,8 @@ #ifndef _MM_SWAP_H #define _MM_SWAP_H +struct mempolicy; + #ifdef CONFIG_SWAP #include /* for bio_end_io_t */ @@ -48,11 +50,10 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, unsigned long addr, struct swap_iocb **plug); struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct vm_area_struct *vma, - unsigned long addr, + struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated); struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); + struct mempolicy *mpol, pgoff_t ilx); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); @@ -80,7 +81,7 @@ static inline void show_swap_cache_info(void) } static inline struct page *swap_cluster_readahead(swp_entry_t entry, - gfp_t gfp_mask, struct vm_fault *vmf) + gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index 788e36a06c34..4afa4ed464d2 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -411,8 +412,8 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, } struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct vm_area_struct *vma, unsigned long addr, - bool *new_page_allocated) + struct mempolicy *mpol, pgoff_t ilx, + bool *new_page_allocated) { struct swap_info_struct *si; struct folio *folio; @@ -455,7 +456,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will * cause any racers to loop around until we add it to cache. */ - folio = vma_alloc_folio(gfp_mask, 0, vma, addr, false); + folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); if (!folio) goto fail_put_swap; @@ -547,14 +549,19 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, struct swap_iocb **plug) { - bool page_was_allocated; - struct page *retpage = __read_swap_cache_async(entry, gfp_mask, - vma, addr, &page_was_allocated); + bool page_allocated; + struct mempolicy *mpol; + pgoff_t ilx; + struct page *page; - if (page_was_allocated) - swap_readpage(retpage, false, plug); + mpol = get_vma_policy(vma, addr, 0, &ilx); + page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + &page_allocated); + mpol_cond_put(mpol); - return retpage; + if (page_allocated) + swap_readpage(page, false, plug); + return page; } static unsigned int __swapin_nr_pages(unsigned long prev_offset, @@ -638,7 +645,8 @@ static void inc_nr_protected(struct page *page) * swap_cluster_readahead - swap in pages in hope we need them soon * @entry: swap entry of this memory * @gfp_mask: memory allocation flags - * @vmf: fault information + * @mpol: NUMA memory allocation policy to be applied + * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE * * Returns the struct page for entry and addr, after queueing swapin. * @@ -647,13 +655,12 @@ static void inc_nr_protected(struct page *page) * because it doesn't cost us any seek time. We also make sure to queue * the 'original' request together with the readahead ones... * - * This has been extended to use the NUMA policies from the mm triggering - * the readahead. - * - * Caller must hold read mmap_lock if vmf->vma is not NULL. + * Note: it is intentional that the same NUMA policy and interleave index + * are used for every page of the readahead: neighbouring pages on swap + * are fairly likely to have been swapped out from the same node. */ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct mempolicy *mpol, pgoff_t ilx) { struct page *page; unsigned long entry_offset = swp_offset(entry); @@ -664,8 +671,6 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, struct blk_plug plug; struct swap_iocb *splug = NULL; bool page_allocated; - struct vm_area_struct *vma = vmf->vma; - unsigned long addr = vmf->address; mask = swapin_nr_pages(offset) - 1; if (!mask) @@ -683,8 +688,8 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ page = __read_swap_cache_async( - swp_entry(swp_type(entry), offset), - gfp_mask, vma, addr, &page_allocated); + swp_entry(swp_type(entry), offset), + gfp_mask, mpol, ilx, &page_allocated); if (!page) continue; if (page_allocated) { @@ -698,11 +703,13 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, } blk_finish_plug(&plug); swap_read_unplug(splug); - lru_add_drain(); /* Push any new pages onto the LRU now */ skip: /* The page was likely read above, so no need for plugging here */ - page = read_swap_cache_async(entry, gfp_mask, vma, addr, NULL); + page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + &page_allocated); + if (unlikely(page_allocated)) + swap_readpage(page, false, NULL); #ifdef CONFIG_ZSWAP if (page) inc_nr_protected(page); @@ -805,8 +812,10 @@ static void swap_ra_info(struct vm_fault *vmf, /** * swap_vma_readahead - swap in pages in hope we need them soon - * @fentry: swap entry of this memory + * @targ_entry: swap entry of the targeted memory * @gfp_mask: memory allocation flags + * @mpol: NUMA memory allocation policy to be applied + * @targ_ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE * @vmf: fault information * * Returns the struct page for entry and addr, after queueing swapin. @@ -817,16 +826,17 @@ static void swap_ra_info(struct vm_fault *vmf, * Caller must hold read mmap_lock if vmf->vma is not NULL. * */ -static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, +static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf) { struct blk_plug plug; struct swap_iocb *splug = NULL; - struct vm_area_struct *vma = vmf->vma; struct page *page; pte_t *pte = NULL, pentry; unsigned long addr; swp_entry_t entry; + pgoff_t ilx; unsigned int i; bool page_allocated; struct vma_swap_readahead ra_info = { @@ -838,9 +848,10 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, goto skip; addr = vmf->address - (ra_info.offset * PAGE_SIZE); + ilx = targ_ilx - ra_info.offset; blk_start_plug(&plug); - for (i = 0; i < ra_info.nr_pte; i++, addr += PAGE_SIZE) { + for (i = 0; i < ra_info.nr_pte; i++, ilx++, addr += PAGE_SIZE) { if (!pte++) { pte = pte_offset_map(vmf->pmd, addr); if (!pte) @@ -854,8 +865,8 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, continue; pte_unmap(pte); pte = NULL; - page = __read_swap_cache_async(entry, gfp_mask, vma, - addr, &page_allocated); + page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + &page_allocated); if (!page) continue; if (page_allocated) { @@ -874,7 +885,10 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, lru_add_drain(); skip: /* The page was likely read above, so no need for plugging here */ - page = read_swap_cache_async(fentry, gfp_mask, vma, vmf->address, NULL); + page = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, + &page_allocated); + if (unlikely(page_allocated)) + swap_readpage(page, false, NULL); #ifdef CONFIG_ZSWAP if (page) inc_nr_protected(page); @@ -897,9 +911,16 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, struct vm_fault *vmf) { - return swap_use_vma_readahead() ? - swap_vma_readahead(entry, gfp_mask, vmf) : - swap_cluster_readahead(entry, gfp_mask, vmf); + struct mempolicy *mpol; + pgoff_t ilx; + struct page *page; + + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); + page = swap_use_vma_readahead() ? + swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : + swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + mpol_cond_put(mpol); + return page; } #ifdef CONFIG_SYSFS From patchwork Tue Oct 3 09:27:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C8ABE75438 for ; Tue, 3 Oct 2023 09:27:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A2FD940007; Tue, 3 Oct 2023 05:27:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 22C518D0003; Tue, 3 Oct 2023 05:27:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07F19940007; Tue, 3 Oct 2023 05:27:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E66508D0003 for ; Tue, 3 Oct 2023 05:27:52 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B67B680210 for ; Tue, 3 Oct 2023 09:27:52 +0000 (UTC) X-FDA: 81303623184.16.CAC4BAA Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf27.hostedemail.com (Postfix) with ESMTP id DE61C40008 for ; Tue, 3 Oct 2023 09:27:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=w78WRgqF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325270; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mU56FlmjTv5eFSUjfveUTDJtj2QxEcbZGZN96w5K2oc=; b=KJrkwGuvzuLYEJpVW8cOsvb8NbD7AMxGv8/NTTnOQC8qWFMwEg9jqeQNVjc2rnx+gyr0JI xuYMYqccNY5GckukJ2aHv5ofZ+VEqkimpCfGnH27IeW6H4pg8t/N7uH1x7i9TkmcydbX0K mxM2tZwsZ11pgJokOCUgGO4kEnV//wg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=w78WRgqF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325270; a=rsa-sha256; cv=none; b=bfENp3HeJGnXacKrhp+/Ip5QYe7mKfk2ErvLN/pgp96Nz6DAmDHQfKwfjwBKAC6uyUhops ZmC3w+/yc/M3RO4hvfzCpxM8m4HDQuw1+RCzSuvd1BA9hJ6YuYx/Tp9kK1e06TpZI3KgDn ScotETkTGeixT6hrsDaKNXY+FPPyAmM= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-5a1d0fee86aso8617417b3.2 for ; Tue, 03 Oct 2023 02:27:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325270; x=1696930070; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=mU56FlmjTv5eFSUjfveUTDJtj2QxEcbZGZN96w5K2oc=; b=w78WRgqFleH6+imFqmjOZ48EhoD0nZm9aE+QQ40B4KuCyrQzx0nEIM733VXQXqL0f4 2sZVJmXA3fxS+/5blgx6P7R6cAF6aasLB1IHXbH4ktnQ9ze+NEbWrMY71mo5e6WXoXvb O2RVjTmsxbLo4WYMtlhhjtOBICkOu2omvVO6RUpIlFBkn44g8bhn5pZETobe6A99LXcx RvsR/nob+ZekuzrT0CSEKetoXVHtndGMdCLptVbQ+/sdks40+0//HB/c9V2Fjr5OBpYb cbLQIGfl+99j1ZNaOrDqNZt37zwuaFh8tbziCBh/fD2GP2O+OJaR5uz5syfyFePImKH6 yzMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325270; x=1696930070; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mU56FlmjTv5eFSUjfveUTDJtj2QxEcbZGZN96w5K2oc=; b=MoyeCmF/BmHSJk+RR8vDN2lrow8j5U7mOn4dusQE7D5qM4OOq9sgAo4bJWoXEon0hr nFFO4GKACId93IKUhxbDHgZt1CA3fgxNrFrD7yScWgWLsMm46/MsoGP0fFsWlOQoLzwV y5xO1QBRp8rp2wWWEDo4dY0iPugsHJ29Qqumpq7YkvtOL3pQQ5F0Wb2Zx7QxMGaF0pbG 7bG3VHECbGLMbI4RyzsIaZ5ZWEtqVE/1O7mtnocg4seulEGH5VFgDWLBXJeufNND6ZJs AvFM4Et5rE/7x3qvtgMIkfyi9mhIhk300nZLTVUrPSlsVAIgn7moZuz0Z3oDB6gxqFr1 mZNw== X-Gm-Message-State: AOJu0YwK6aOyhnqV2T2AExHNy57ajvZ/HY70nQyDcbnE1SYbEWRl4bBu nxMjWC/gM6cAmJNdSm6mxIyhFw== X-Google-Smtp-Source: AGHT+IH+RBdvJ2zPjHqn2yE7GeZTf5dQ/uq/KjJe+YyhinlFoMPpIY/IVB5OPsNP+GzZ5W6FH4sY9g== X-Received: by 2002:a0d:ddc1:0:b0:5a1:d4bc:7faa with SMTP id g184-20020a0dddc1000000b005a1d4bc7faamr14296514ywe.18.1696325269879; Tue, 03 Oct 2023 02:27:49 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id k6-20020a0dc806000000b0058038e6609csm256304ywd.74.2023.10.03.02.27.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:27:49 -0700 (PDT) Date: Tue, 3 Oct 2023 02:27:47 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 11/12] mempolicy: mmap_lock is not needed while migrating folios In-Reply-To: Message-ID: <21e564e8-269f-6a89-7ee2-fd612831c289@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DE61C40008 X-Stat-Signature: omhzf8pp9o8au53xanzdg793c3azx6dt X-HE-Tag: 1696325270-615309 X-HE-Meta: U2FsdGVkX18US+dCjKPG4nkjJfhnj9Kckq7rPrWmT1AqqjK8wwkpKps/95yuPuo32MCwjxpDMosEXKr+NYcHzxMIlIoem4uOrZOpjSTJAL4VhnvBRGRdVDGtUiN12eZ1LKRWIYilHDYHX8JZhJLcxh+CnDJ/pVo64pNxmXqL5C1mXvJM/ueA0T6auY9nx6U48mrsu24em8uCTJ8vZWMH2+pIK4DRagyZLGNbVVnvQ3ryRZju7H7DJcqdybdJX+iNkiJhWLscbL5M4xC3d3ykdaSVtHIh3FGkUmC7D1+dk8ZNTYk77zdF5e6rjHKFPnj80maHddjTCVun7dG70btPh4Gb93p37D0SKmwhQquQbx9cgE4jV4HAT16mmTX/MluG56KtoT9tme4OeakgKgaXBAx75xeoigyWkpQDagRFMfoFE12DEQim/229oCEiLq1ug0Djze4Jp6aOVIFXDnS/dDs7BDSZVqBUgHRqczD/qu+qg2Q+Xu3tU4wDsgh4fqUAQdlqI+AfT3axwaIW5xWGFFZy1SVXR1lJiRL5Dn/LnkHrEqJeD71qmQnwjH/K/d2xOAkgWhDtElGTeeDwSXO5/ietHei1QR6siz0hpC+YE2nyKGOLWJDvCDU2lEJuXQXbc4O238ptYbmSpp4UIFPlB6OxCbSSvIfAsfgOZ7tDIR99rIIjC51twtPwyM9OBX0YO1UkL/sMVfP+L23XZ5eYHELaGZJbyhtkHUSA3BI7d+XPgrSag3iQ1a6DBsCoJPtSJKggrL6p605cOMp3yixh6rFOT5aXHOHd9/jgrYFHR94Ep09x37SunB0ceeOsDx48jpgltKYMrdqTwJ+xMRe0mRVWsGluEoPwEq/ytMTvsEk8omZuogAodvrh4wDX+dfMK7a8UrImtLuznoI6eBIsnjVdlCSr4m6Ch/pI7aFS3txq2yr1JVg4B3MGVgP2BNV86ff7RAixHnzIlCRjyLe iQlk0SBy n/4I3nQiekbM3xP0UFOi6V136KFBmrbZrMSPQCD4xwUqoC9gVDslYEu93t4z6Q7LwlMLEOX1oNE8FK55+bWRmfG0tFS4POPCKxrX5GoOwGlnMhW24oWRAMywSmmyDAKC5WRsg7jx1nxdpQlwgRWOSiLwUyDKwx6+VIHFBPLsMri8KmjdwSq0qrl21lA89FbxCY4IxXZMlZi82WmTFtSnll+21OAczb4/I5ErhJShZZcJo1+BqZFWAGc5Q06JkZ1LqcRPQMsxNrj0dOr2i+2GEG8T7matvp/NZx5iZq0SbLzwiW3Y6Az3TPF5RHPwfRucSWvItcpzJEAghmvyZsYUoJu79uJPMVT4eRbokpJroi6tLbh+KVUjOFxB+lTB7v67UeG2g2Svkzm04LxTJVIfSOoYwcw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mbind(2) holds down_write of current task's mmap_lock throughout (exclusive because it needs to set the new mempolicy on the vmas); migrate_pages(2) holds down_read of pid's mmap_lock throughout. They both hold mmap_lock across the internal migrate_pages(), under which all new page allocations (huge or small) are made. I'm nervous about it; and migrate_pages() certainly does not need mmap_lock itself. It's done this way for mbind(2), because its page allocator is vma_alloc_folio() or alloc_hugetlb_folio_vma(), both of which depend on vma and address. Now that we have alloc_pages_mpol(), depending on (refcounted) memory policy and interleave index, mbind(2) can be modified to use that or alloc_hugetlb_folio_nodemask(), and then not need mmap_lock across the internal migrate_pages() at all: add alloc_migration_target_by_mpol() to replace mbind's new_page(). (After that change, alloc_hugetlb_folio_vma() is used by nothing but a userfaultfd function: move it out of hugetlb.h and into the #ifdef.) migrate_pages(2) has chosen its target node before migrating, so can continue to use the standard alloc_migration_target(); but let it take and drop mmap_lock just around migrate_to_node()'s queue_pages_range(): neither the node-to-node calculations nor the page migrations need it. It seems unlikely, but it is conceivable that some userspace depends on the kernel's mmap_lock exclusion here, instead of doing its own locking: more likely in a testsuite than in real life. It is also possible, of course, that some pages on the list will be munmapped by another thread before they are migrated, or a newer memory policy applied to the range by that time: but such races could happen before, as soon as mmap_lock was dropped, so it does not appear to be a concern. Signed-off-by: Hugh Dickins --- include/linux/hugetlb.h | 9 ----- mm/hugetlb.c | 38 ++++++++++---------- mm/mempolicy.c | 83 ++++++++++++++++++++++--------------------- 3 files changed, 63 insertions(+), 67 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a574e26e18a2..7c6faee07b42 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -716,8 +716,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask); -struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma, - unsigned long address); int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping, pgoff_t idx); void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, @@ -1040,13 +1038,6 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, return NULL; } -static inline struct folio *alloc_hugetlb_folio_vma(struct hstate *h, - struct vm_area_struct *vma, - unsigned long address) -{ - return NULL; -} - static inline int __alloc_bootmem_huge_page(struct hstate *h) { return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9d5b7f208dac..68ff79061f88 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2458,24 +2458,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); } -/* mempolicy aware migration callback */ -struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma, - unsigned long address) -{ - struct mempolicy *mpol; - nodemask_t *nodemask; - struct folio *folio; - gfp_t gfp_mask; - int node; - - gfp_mask = htlb_alloc_mask(h); - node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); - folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask); - mpol_cond_put(mpol); - - return folio; -} - /* * Increase the hugetlb pool such that it can accommodate a reservation * of size 'delta'. @@ -6279,6 +6261,26 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } #ifdef CONFIG_USERFAULTFD +/* + * Can probably be eliminated, but still used by hugetlb_mfill_atomic_pte(). + */ +static struct folio *alloc_hugetlb_folio_vma(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) +{ + struct mempolicy *mpol; + nodemask_t *nodemask; + struct folio *folio; + gfp_t gfp_mask; + int node; + + gfp_mask = htlb_alloc_mask(h); + node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); + folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask); + mpol_cond_put(mpol); + + return folio; +} + /* * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte * with modifications for hugetlb pages. diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 8cf76de12acd..a7b34b9c00ef 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -417,6 +417,8 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, unsigned long flags); +static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, + pgoff_t ilx, int *nid); static bool strictly_unmovable(unsigned long flags) { @@ -1043,6 +1045,8 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest, node_set(source, nmask); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); + + mmap_read_lock(mm); vma = find_vma(mm, 0); /* @@ -1053,6 +1057,7 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest, */ nr_failed = queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, flags | MPOL_MF_DISCONTIG_OK, &pagelist); + mmap_read_unlock(mm); if (!list_empty(&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL, @@ -1081,8 +1086,6 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, lru_cache_disable(); - mmap_read_lock(mm); - /* * Find a 'source' bit set in 'tmp' whose corresponding 'dest' * bit in 'to' is not also set in 'tmp'. Clear the found 'source' @@ -1162,7 +1165,6 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, if (err < 0) break; } - mmap_read_unlock(mm); lru_cache_enable(); if (err < 0) @@ -1171,44 +1173,38 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, } /* - * Allocate a new page for page migration based on vma policy. - * Start by assuming the page is mapped by the same vma as contains @start. - * Search forward from there, if not. N.B., this assumes that the - * list of pages handed to migrate_pages()--which is how we get here-- - * is in virtual address order. + * Allocate a new folio for page migration, according to NUMA mempolicy. */ -static struct folio *new_folio(struct folio *src, unsigned long start) +static struct folio *alloc_migration_target_by_mpol(struct folio *src, + unsigned long private) { - struct vm_area_struct *vma; - unsigned long address; - VMA_ITERATOR(vmi, current->mm, start); - gfp_t gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL; + struct mempolicy *pol = (struct mempolicy *)private; + pgoff_t ilx = 0; /* improve on this later */ + struct page *page; + unsigned int order; + int nid = numa_node_id(); + gfp_t gfp; - for_each_vma(vmi, vma) { - address = page_address_in_vma(&src->page, vma); - if (address != -EFAULT) - break; - } - - /* - * __get_vma_policy() now expects a genuine non-NULL vma. Return NULL - * when the page can no longer be located in a vma: that is not ideal - * (migrate_pages() will give up early, presuming ENOMEM), but good - * enough to avoid a crash by syzkaller or concurrent holepunch. - */ - if (!vma) - return NULL; + order = folio_order(src); + ilx += src->index >> order; if (folio_test_hugetlb(src)) { - return alloc_hugetlb_folio_vma(folio_hstate(src), - vma, address); + nodemask_t *nodemask; + struct hstate *h; + + h = folio_hstate(src); + gfp = htlb_alloc_mask(h); + nodemask = policy_nodemask(gfp, pol, ilx, &nid); + return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp); } if (folio_test_large(src)) gfp = GFP_TRANSHUGE; + else + gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP; - return vma_alloc_folio(gfp, folio_order(src), vma, address, - folio_test_large(src)); + page = alloc_pages_mpol(gfp, order, pol, ilx, nid); + return page_rmappable_folio(page); } #else @@ -1224,7 +1220,8 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, return -ENOSYS; } -static struct folio *new_folio(struct folio *src, unsigned long start) +static struct folio *alloc_migration_target_by_mpol(struct folio *src, + unsigned long private) { return NULL; } @@ -1298,6 +1295,7 @@ static long do_mbind(unsigned long start, unsigned long len, if (nr_failed < 0) { err = nr_failed; + nr_failed = 0; } else { vma_iter_init(&vmi, mm, start); prev = vma_prev(&vmi); @@ -1308,19 +1306,24 @@ static long do_mbind(unsigned long start, unsigned long len, } } - if (!err) { - if (!list_empty(&pagelist)) { - nr_failed |= migrate_pages(&pagelist, new_folio, NULL, - start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); + mmap_write_unlock(mm); + + if (!err && !list_empty(&pagelist)) { + /* Convert MPOL_DEFAULT's NULL to task or default policy */ + if (!new) { + new = get_task_policy(current); + mpol_get(new); } - if (nr_failed && (flags & MPOL_MF_STRICT)) - err = -EIO; + nr_failed |= migrate_pages(&pagelist, + alloc_migration_target_by_mpol, NULL, + (unsigned long)new, MIGRATE_SYNC, + MR_MEMPOLICY_MBIND, NULL); } + if (nr_failed && (flags & MPOL_MF_STRICT)) + err = -EIO; if (!list_empty(&pagelist)) putback_movable_pages(&pagelist); - - mmap_write_unlock(mm); mpol_out: mpol_put(new); if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) From patchwork Tue Oct 3 09:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2720E75438 for ; Tue, 3 Oct 2023 09:29:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EDD1940008; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 476C58D0003; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F0EE940008; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1A15C8D0003 for ; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E0B89A01D3 for ; Tue, 3 Oct 2023 09:29:06 +0000 (UTC) X-FDA: 81303626292.27.9944507 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf25.hostedemail.com (Postfix) with ESMTP id 167F2A000E for ; Tue, 3 Oct 2023 09:29:03 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nk13zoZs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=or+htEXIJCxYp3JS4YiDzaqweWJblWmcMiW11bdtnSl6O82O+bJBD1bV9WRTcU3hpT336H ywLbQD0Kss5cjFb3BRM6igeqmhQ9Jzduh6MW6u5x4wwVKfIVL4//Hpb6D1zNGYnbCPA/PJ CqJGZdjg4PBlsMG3sVFPymqWMTm4y2A= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nk13zoZs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325344; a=rsa-sha256; cv=none; b=oIk2iPzySnDQmTDF2+WtiM6UQrgB3DLU6DIyUHY5rehpfjzXynBkCvLZNthD3Snnpi05Lf bGB39mHD9Sb+mu4Sc1WBt6LN9iBWo7YPpYi9QApqp0CbRbqWFqqMYInH+wNEXFc0p/bF0J 8uVefV68ZM1zaOwfF+ooWoBq2bcbhDQ= Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-d89491dab33so736023276.0 for ; Tue, 03 Oct 2023 02:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325343; x=1696930143; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=nk13zoZsUYFof7LjGrJwvOgBHed/Olg6MhjReFQ3CBvIVtQtdoFFLb6NJsyoCY//kF qQf+bOnZYsr93ZbeoOzI5R+qB67H1AB6V5vdlQyQBwI2bVpExRQmfyZJF0/CgPl9reWV i/LjfErOD36974WUjrGXMk2pdnnVfvxM37LYs90+zI0b+Buw8jTEhPfznqtEq6yTX4uA XVpQZRnGLBsvTA/fWYc5nUPkAQDuwgjHnEAT9ZL+hDY6wwJzj8ShdfO5ekNBZ60ekepr 3YXsiJs+tpluLcVV8hrmNHv1YXItabP5zNNuyuaYuhEXCSfFY3Bg64vwtxpIlwCkiW/I 9eKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325343; x=1696930143; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=nJQ3U69mZyL4hPiRqEwlovvYCRCgsDQhv+Ks2F0bd6ovwuvTMTSEeM6lec4KJThRSF JnCeYvSwnuXUt20/+jOyYCPFYFiTblQOWspgelFGeDjg8e3jIXOQNJ9Oe8Nq5mFgMjqt lh/ca0rXXOAmJ7W+HPHu5c9ARpmGFUYEWrAC9NxeHauIssQkvR5HtvepDkEEZYLLR65A O1UtUiXQ2AkGsBCbu5LhJvDLB2IqIwq+gFIdZnCCon388q8wDYHaNBMu+Injr4WMng5N hZ5M4fAlDEYCkiHlykUPTx1mIQcXRApsLHNVzxcyzsi2rpmqj/V83yG0bC1dtHWcBUGS dqiQ== X-Gm-Message-State: AOJu0Yxj6ZQ/ZSvY+XZHC1nztaoeoV6khqSm/hmYMoxAhcqoscfJ6hoQ w1lJpjsFZcjZVs6p69g71oc96g== X-Google-Smtp-Source: AGHT+IG82Q8rMpJY3BZE97NKeDHGMxciJo3XfpIVLFLHZCNcKqiXapr+aAI39Q2ReaCSAdFAa0T/Yg== X-Received: by 2002:a25:da4e:0:b0:d7a:d628:f69d with SMTP id n75-20020a25da4e000000b00d7ad628f69dmr13139965ybf.32.1696325343062; Tue, 03 Oct 2023 02:29:03 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id w196-20020a25c7cd000000b00d1b86efc0ffsm291695ybe.6.2023.10.03.02.29.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:29:02 -0700 (PDT) Date: Tue, 3 Oct 2023 02:29:00 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 12/12] mempolicy: migration attempt to match interleave nodes In-Reply-To: Message-ID: <77954a5-9c9b-1c11-7d5c-3262c01b895f@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 167F2A000E X-Stat-Signature: 9nhzo13pk15ez69s7bkk9rw5ed8hrkq7 X-HE-Tag: 1696325343-119406 X-HE-Meta: U2FsdGVkX1/G970KLd/5NrkBdE1K239kv3vxDCzc6ZyOen4oKgZl9fMWYBtdDyKQOX9zE2l3UJ1xCcScsu+EM+19F1ei/jLzvLEbXoOAm5yJNcQz1lg1xRR9lCUeskIHxcDnqTJuTEKaYEY3kVy76ZJoXPi3C5TiszmBYL2HWa9sCsfUpyrG3xb+5T8nsDaTk0SGA3RlGEJR+me2xnWwYg7Q+Q8Jy+vGdUm+qgXYd9SZi1aIreDfkXDnP/3fDiXWK+PhyS8Ul5g0apkHwXT3xwe9I0YPMlSo0OJ7Fo6qOawMMUlv1yal5A0IcgkZIfNRiUVQpyMC+1sQf4/zH3ZNsou9sRxPbpShLvyqEuwLYi2BSOjLMxDJwL+DskJn8wMIy195NCn8CebBJjM5OYVKGqoUXnOzf0xJ3I9pUynV5DIAIxrCqWsuTa/+zdXLwvak01efr54tMvFCbRHTaghY6TPEUS6ZvtwY+S3v1mCgZJhsMbDhhWCk6sOcNRiP7zC5cJ/FctW4j2m/z5A4AytZLucyQpSbAoZz1HhI90C3ZNI/shIUY6GTnKTe/lO13hfQJMOJWV0dTF2qijq8gjhaZSimQxOGdNLUf3PWMeQ1tbqH3b+UR5PSMlGA+4Wp7ZiB2ruzQoGt0BHtSX+n5XbfrILkYt2ZTU8peEKlkhGapJ59v+4tlYA3DQHyNvf/etIvLg7Uq446YqxkkCt/txU0JlOEIfRniQay8/hv1RYePcBNxpglvUTBr9drgBTvjXuFP8/7dZj3nCNkwec2nZtQBZVKHzE5hF6gjYrk6ivOy2O0qzQ4+SDvT3kVkwrZ5AglJn17QI+0e++v4NE/neW0BM3VZgQRNgmWSJeWAlNQpxQ7/5IM0L/23TKXrwhMmLvBTvIMKiuptWakYeTswQDt1VQnIxrtcFyuwm5YGQ5rnASKgnfmJYGugAW78Mr+qX1j2p9O3cvmR1ccVgwc0qP 441OgCIE sd5Fer0nnzsjRqktpHOArynhofOhT4uJBuIFDypNnLxChiI7elOH66i5Av6acS0yXMOaoeXIvtfCYEijWNUny7Y6txV468QSc6HMzTKwzy2iHnn8MxIgTS9Ssci+EGeCzW9RYfnHKiqc34zDXr9qAvu4KKihk6tpdh/FzWx3XwOTW0m3jEEJW9fmxcmnUZKGCmN8bOlsgpYgk2qT8Wh3RgK0p5w+TLmtPkhZDennS4oWd97p70X3112OTVd4Pc9fU66l9NeffPIS1yk6w6CuGPXEG8WidA6dIKFbXz3W/Sg/SpV9Dh/w1bTnXh2EGF6wPCJDnKkOY4YWquID9DT0wHE5HHQynNoG4AM7INDH8hCFzlwWGGcNXxAiIIFerI1aLo8r+I/xA/sXh9wU2H3jbTYO+r+oHyo4QlS5ddcpHNqywYQzhjcT2cUHbBkXaGLAQ2Io6a0fExoor8+k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Improve alloc_migration_target_by_mpol()'s treatment of MPOL_INTERLEAVE. Make an effort in do_mbind(), to identify the correct interleave index for the first page to be migrated, so that it and all subsequent pages from the same vma will be targeted to precisely their intended nodes. Pages from following vmas will still be interleaved from the requested nodemask, but perhaps starting from a different base. Whether this is worth doing at all, or worth improving further, is arguable: queue_folio_required() is right not to care about the precise placement on interleaved nodes; but this little effort seems appropriate. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index a7b34b9c00ef..b01922e88548 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -430,6 +430,11 @@ static bool strictly_unmovable(unsigned long flags) MPOL_MF_STRICT; } +struct migration_mpol { /* for alloc_migration_target_by_mpol() */ + struct mempolicy *pol; + pgoff_t ilx; +}; + struct queue_pages { struct list_head *pagelist; unsigned long flags; @@ -1178,8 +1183,9 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, static struct folio *alloc_migration_target_by_mpol(struct folio *src, unsigned long private) { - struct mempolicy *pol = (struct mempolicy *)private; - pgoff_t ilx = 0; /* improve on this later */ + struct migration_mpol *mmpol = (struct migration_mpol *)private; + struct mempolicy *pol = mmpol->pol; + pgoff_t ilx = mmpol->ilx; struct page *page; unsigned int order; int nid = numa_node_id(); @@ -1234,6 +1240,7 @@ static long do_mbind(unsigned long start, unsigned long len, struct mm_struct *mm = current->mm; struct vm_area_struct *vma, *prev; struct vma_iterator vmi; + struct migration_mpol mmpol; struct mempolicy *new; unsigned long end; long err; @@ -1314,9 +1321,45 @@ static long do_mbind(unsigned long start, unsigned long len, new = get_task_policy(current); mpol_get(new); } + mmpol.pol = new; + mmpol.ilx = 0; + + /* + * In the interleaved case, attempt to allocate on exactly the + * targeted nodes, for the first VMA to be migrated; for later + * VMAs, the nodes will still be interleaved from the targeted + * nodemask, but one by one may be selected differently. + */ + if (new->mode == MPOL_INTERLEAVE) { + struct page *page; + unsigned int order; + unsigned long addr = -EFAULT; + + list_for_each_entry(page, &pagelist, lru) { + if (!PageKsm(page)) + break; + } + if (!list_entry_is_head(page, &pagelist, lru)) { + vma_iter_init(&vmi, mm, start); + for_each_vma_range(vmi, vma, end) { + addr = page_address_in_vma(page, vma); + if (addr != -EFAULT) + break; + } + } + if (addr != -EFAULT) { + order = compound_order(page); + /* We already know the pol, but not the ilx */ + mpol_cond_put(get_vma_policy(vma, addr, order, + &mmpol.ilx)); + /* Set base from which to increment by index */ + mmpol.ilx -= page->index >> order; + } + } + nr_failed |= migrate_pages(&pagelist, alloc_migration_target_by_mpol, NULL, - (unsigned long)new, MIGRATE_SYNC, + (unsigned long)&mmpol, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); }