From patchwork Mon Jun 6 20:20:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870830 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03BB9C43334 for ; Mon, 6 Jun 2022 20:22:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7802C6B0072; Mon, 6 Jun 2022 16:22:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72DCE6B0073; Mon, 6 Jun 2022 16:22:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 559256B0074; Mon, 6 Jun 2022 16:22:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3E28A6B0072 for ; Mon, 6 Jun 2022 16:22:02 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0C0CEBB2 for ; Mon, 6 Jun 2022 20:22:02 +0000 (UTC) X-FDA: 79548932484.05.95764C9 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf15.hostedemail.com (Postfix) with ESMTP id 4443BA0005 for ; Mon, 6 Jun 2022 20:21:35 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256Jis0W010073; Mon, 6 Jun 2022 20:21:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=FP0i4KiK6HfUfgeKgIQGku+//s7C/GJB65ZftbQP0Qs=; b=oJZJCmtpHa3GWfNtEhCUZC7LndreB3SVqq6DS6vEcWUwx14HAwnjg8fxxu26wcMD9llp W7Z/GU8BXtxJ748HwFP48VNBZ9O1P6FA8AlSUCFUxjBnvUNkrRCG8YE3PXebiS9N4DkL h1+ME7qRa2lH1JFrhD1DrVUjHBUhGpRkarIb++5JuN2aWAIzQVG2/s/3Qxy2pbnS8uy9 9PAEhSB5xkuV491T4LtJ/DUtzoxF02jhorGq1gli7P3InhUvi6DoFQi+MnJDbP3Hb2v+ QlDwB82JyDPnWklsZUJbPuM/LnWCb+yWU5KC9aQ9MVJ3lceG1IwA/nqB7SHnEU5CsFjs TQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfydqm281-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:21:39 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGTaf035671; Mon, 6 Jun 2022 20:21:38 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2172.outbound.protection.outlook.com [104.47.58.172]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1v4tk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:21:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OAHhNH/grKRzShckzIBzsnFMw76lSaM22b8/ite5vlpylBDSRfgTvfIiALflE2V2A4mIGkBZU2Sq+62A2mzKoTzn/8Yanaq9HunteDsZCt8cwmq11bHsdiaxc/U1spwcmLMDy6KtXalxBTHaUwC20xAIyd2QotlKHS5axc64w7GNZS3xdxn1oszKahpUQVouAXh/rCj2XgEADrJIcU1xLo3NwuQt9/gyMSU3GnxORRlcz6SVlLZwGmXaGXBK8B6BteO3J9nRIyv+Tkbk4YgJLzMdihpQZPsi0TDl/3nI0TxNLD5JuanomDTtozHYlUlnhO0jlnt4adItKDOv5KIBag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FP0i4KiK6HfUfgeKgIQGku+//s7C/GJB65ZftbQP0Qs=; b=gOj4JDfeGd8ljuNcEiGR1L0cU9Jx2Ipnf9K5bYUm6r+X4dy1WAHTq+Cs3GzSS/yf11gDKslPp/F0RJ5DE7eJ2XU5vcW23nT0m00t6dc+DNrOKVikVTBWgBBVWiX2CDrWBYyfxW31KanzKMZVAPSOen3TNX7FAjLCO931kLs4l+YATLQrYkv/6lcEfsEaa0VVhmoYndJAfmbaXNK140iwjRUvg9D6+PnAaEy/ud8R0wKjiXyiWNrgTuV3hX/N6Oew8hTkdEuR/1IfrkT+1kTkEKg/D2rPG7FfNMLSHa+Jb+/VdejEy0KaAj6dbA5w5LaRyh+g4cQQlNmn7KT9Ru3nZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FP0i4KiK6HfUfgeKgIQGku+//s7C/GJB65ZftbQP0Qs=; b=WSSZ6hoJJACVvyPzuT+pe+gFMhaIJk0Yp+l8NJjZFEQBL9tOAaXaJmv/G0So0P/5dfdGtLCSdCSZXP+IkcOQR86NaNoTb4UJp4Sge2qGXGFPJ9m8rSVyfoZlVEE2r6bU1RC8cu/4oRVEc+iVRw/3ko9evUNIzoi1NOOxhegjMts= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by CY5PR10MB6046.namprd10.prod.outlook.com (2603:10b6:930:3d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.19; Mon, 6 Jun 2022 20:21:36 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:21:36 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 01/21] mm, huge-page: reorder arguments to process_huge_page() Date: Mon, 6 Jun 2022 20:20:49 +0000 Message-Id: <20220606202109.1306034-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR06CA0036.namprd06.prod.outlook.com (2603:10b6:a03:d4::49) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8fa56c10-06d5-488b-e706-08da47fa27b2 X-MS-TrafficTypeDiagnostic: CY5PR10MB6046:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: usnlB/cSgVyiGz0kmuERZCrrsl1lH3neIz+pVZQsxx/4OmTDkqZkNBI3i1ClRkNPjIXLENswEnj4/5+DBJlwSyw56edrqXpr36XDtVMhkGfQrHeNlLaNBwhlrvC8jLkHbDNz+f8XhM9XXDAtfeqUR7ZcBJqjx8kCl+8VvOYrRTvRiuGOKyZP2NSzpaOd4rDgXDeUVAuTmpstFRxN106iNFP1W/9wUXyibjDR1Zhe0EcSZas5gmliVwFpTmbtBqpTN0oa12A8kdZUb519yxNs6Yir6+XmFerk04trjOQfVNs1to7+71+L8m0DarwsR0xkjj88sc4YigsspQmkzzrTryo2sXpktAlSGoDRA2UihUbIHx7Xvex7b4gsM4sajZbKnibHL3S2XbpVD+lv6ja/jlyzZa7Ib+i+8sWXDi5yN85YHRjjdJWUaYuuNLHaIrPGjgI3N2XlDpz3knzy9KR9u0N+H21Z6wTyrwpWFD8QpZgzgtIbP87DVYvaJY5TCWF1cr/tWfNOSlUj+pqGggORUFHD+HiQrmSWBriooswwm3xf0SIR2R8R1W2Hy/mlFUr+7e65HeSF/U381AI8d9t5NE8ZEg4jJAb9uRzN0w7ntJ/SqZ2fKNCgWV2NswefyU3idN2tWFFIYJ+QjTf/ljKoM7uRi+Md3E4edsXiTbdqhhg6J0e0YJx1+8UhlWTIdCvzYm4H8GeNtj8de5hMb3ywckYQkfvSvhIbmhPUq8e+ZsM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(83380400001)(86362001)(36756003)(103116003)(8936002)(38350700002)(38100700002)(508600001)(6486002)(6666004)(8676002)(4326008)(5660300002)(66946007)(66556008)(66476007)(7416002)(2906002)(316002)(26005)(6512007)(186003)(52116002)(2616005)(107886003)(1076003)(6506007)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: u2Xp7hCRiz46iEXmi2hr1PsoM6r6POf82STVnA+/K2S0psladGvhYoGZgH/x8ekjld0QEQjXnx4FWP61OgujG+QdffuFSve/L1bwFQvkfr+g6fckwA2QT9N2l65EDLSw4kVI1VBOcKB8WAo3QkC4C9F5I08Io0Gde+Yq4TsiU9sXqDZlDUcHLG5ISavgZi75+5uT0+IPokjo3MbUDFaBXeOOd268KSQYmhwjjiwoRBXp7MbinD6OvvXhmlO010z0UygsXTDH8bm4hquLY4TeKkzqBIyGMdQKkJCB9juRRx3x2zPfS+Q/jqhbMdTbjtBpHejmyOwK1tAavjTVheEs9cN9qlDFHMKFLft7yTdDU4pBEt0KjusMdAn0eP9n2c3IpNvoUl1dVHtcuysdqWgVDlIGwr8/4Uf/Wr4RWHgJuusIdqgEGMdFQCqJG1p4qi328KSK7nPmIRrIytbny81eYfv5o+dH8rnKXPBc85bH+Dn+RecVnRaa8byTLwnZ/1TLZLfU6z7GF6AVRjjYOtA9VlVSJg8Z6JcjY7mRLgfXHDaYHnRYXAvyMQiTyrrfV3klgeTCPY+wd5g7kX37RtuOcZ0H2rmVf7Oz1bTLZ/38W30zb1JgTcaGltS010Fnt7JE6GSJvIi+yA5lFXys8ydE6AyMHqpBe+3A8HZfA9eba1wQzNXjwXA40HNPSfjda9TNjmpTCWG1C9PQfSSDILMX4W6zHcQ/xMVTTc7EG6ZKLSfLie6LU4tPn2v7p72y/n07qAvULfb6sc9jDHvKEjd5xXo1eZrn6wbS9Sf+xSQQRs8PF6XfXCcx+enf0tPeMuzpPj6EIzh0dLDxhTZ1P2Kxi34ZtlBfjry1VknxQQM7viGs4niUfsbjliFf4ZazzuhNpBVN+F4p7N8BpvDOmABI8puCSbZ2Ta/rvDiF7S3RoLXM/xqLftTt4LoWAN/sMJJOHqM0H+TBmTjxzAHqaQAfBMWswziV4tQScMq+MDuZj1zHo6k9oDokdf19oodJlcF65EoHpY4iy130k+CCKafn2ehOfpRToOrLOJYAdoloLs+MC04PYFx8Kw3h9frvcIns8TTGvynrPD43Ph6NgwTFbvnYVPiVk9iUhMUGVMJZ6dM80KI23lKpozohd7/1fqqA5ev+23JGf5YDbSNNgsqjpnDAmvJI2NU/JTD+D8jswNOSaxKx+vp37CxB16e/tpnjYQprGR6u3tETPO337V9C+i0tWmgIZTR9+jVViVCi1UYafrynHlvBmP1wtOVCAw+RZowVcGbTWMuEUnmD/jtqv4T+Mb6qTe7S0fSzvF28iKiAqRqu4Hzg3sBr4MjxsUUFJnAhuiElsy9kiPL/y6/G9oGi67wB/ceXI4K+nuu0eujhTXlZ1AtKr9fdVKLpJXQS/E5WjTeC6KqP+SYu86YIgT7M98FvKfu9urXOQQ4CYduFK2WEra3siZGPnavAnV4GozQqVKDbUrx+k4UvkyKQ7rVPUJkX9hpjhDlLDiGpiX603Mw2qYYL2cfMNBs2+xU1O1QANxJXj/gk49Oxq6OJ1XGxH1cJV4v9iDIPryERKI8oRt/2s/0GbMlMUhn85b27MeUV5+WPtkVT6aWGxAPpKpgAqu2/UZobds12eXtSYLKKG7L8VtPyCAwoitgcvK5Ld7jtoED36JYrMWOtyd0V8aXTlijNi/KgNntzxMyzkrkrm+/TkU1b3kX9j/3rhHGPPHbRQdYABzjx1HL3SvmK8WYzlxnIkhRvQ3/VBPy29rM= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8fa56c10-06d5-488b-e706-08da47fa27b2 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:21:36.2434 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +EF+LfLXrk1AFY+NzYpyOU0mNkC5ZdSPq7qqerjhcZl4OR/j2Giq7uX1NorFY7//OVl03YSPpQNikX/Vxo3H+QCPAeKMjg0eLHNFIed169w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR10MB6046 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: xmSxS7VtaBXprgCiMcIQOlFXOWWwyHAa X-Proofpoint-ORIG-GUID: xmSxS7VtaBXprgCiMcIQOlFXOWWwyHAa X-Stat-Signature: abwrw78j7kod7axr7mgg4pr981god9r5 X-Rspam-User: Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=oJZJCmtp; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=WSSZ6hoJ; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf15.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4443BA0005 X-HE-Tag: 1654546895-662952 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mechanical change to process_huge_page() to pass subpage clear/copy args via struct subpage_arg * instead of passing an opaque pointer around. No change in generated code. Signed-off-by: Ankur Arora --- mm/memory.c | 47 ++++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 21dadf03f089..c33aacdaaf11 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5562,15 +5562,22 @@ EXPORT_SYMBOL(__might_fault); #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) + +struct subpage_arg { + struct page *dst; + struct page *src; + struct vm_area_struct *vma; +}; + /* * Process all subpages of the specified huge page with the specified * operation. The target subpage will be processed last to keep its * cache lines hot. */ -static inline void process_huge_page( +static inline void process_huge_page(struct subpage_arg *sa, unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) + void (*process_subpage)(struct subpage_arg *sa, + unsigned long addr, int idx)) { int i, n, base, l; unsigned long addr = addr_hint & @@ -5586,7 +5593,7 @@ static inline void process_huge_page( /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + process_subpage(sa, addr + i * PAGE_SIZE, i); } } else { /* If target subpage in second half of huge page */ @@ -5595,7 +5602,7 @@ static inline void process_huge_page( /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); + process_subpage(sa, addr + i * PAGE_SIZE, i); } } /* @@ -5607,9 +5614,9 @@ static inline void process_huge_page( int right_idx = base + 2 * l - 1 - i; cond_resched(); - process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + process_subpage(sa, addr + left_idx * PAGE_SIZE, left_idx); cond_resched(); - process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + process_subpage(sa, addr + right_idx * PAGE_SIZE, right_idx); } } @@ -5628,9 +5635,9 @@ static void clear_gigantic_page(struct page *page, } } -static void clear_subpage(unsigned long addr, int idx, void *arg) +static void clear_subpage(struct subpage_arg *sa, unsigned long addr, int idx) { - struct page *page = arg; + struct page *page = sa->dst; clear_user_highpage(page + idx, addr); } @@ -5640,13 +5647,18 @@ void clear_huge_page(struct page *page, { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + struct subpage_arg sa = { + .dst = page, + .src = NULL, + .vma = NULL, + }; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { clear_gigantic_page(page, addr, pages_per_huge_page); return; } - process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); + process_huge_page(&sa, addr_hint, pages_per_huge_page, clear_subpage); } static void copy_user_gigantic_page(struct page *dst, struct page *src, @@ -5668,16 +5680,9 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -struct copy_subpage_arg { - struct page *dst; - struct page *src; - struct vm_area_struct *vma; -}; - -static void copy_subpage(unsigned long addr, int idx, void *arg) +static void copy_subpage(struct subpage_arg *copy_arg, + unsigned long addr, int idx) { - struct copy_subpage_arg *copy_arg = arg; - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, addr, copy_arg->vma); } @@ -5688,7 +5693,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - struct copy_subpage_arg arg = { + struct subpage_arg sa = { .dst = dst, .src = src, .vma = vma, @@ -5700,7 +5705,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, return; } - process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); + process_huge_page(&sa, addr_hint, pages_per_huge_page, copy_subpage); } long copy_huge_page_from_user(struct page *dst_page, From patchwork Mon Jun 6 20:20:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73538CCA473 for ; Mon, 6 Jun 2022 20:22:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02C236B0073; Mon, 6 Jun 2022 16:22:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F1D656B0074; Mon, 6 Jun 2022 16:22:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4A416B0075; Mon, 6 Jun 2022 16:22:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BFB3C6B0073 for ; Mon, 6 Jun 2022 16:22:04 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 98BC920C1A for ; Mon, 6 Jun 2022 20:22:04 +0000 (UTC) X-FDA: 79548932568.16.8D2357C Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf16.hostedemail.com (Postfix) with ESMTP id A980618001A for ; Mon, 6 Jun 2022 20:21:44 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256Ivt3s017195; Mon, 6 Jun 2022 20:21:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=0nZQslULxAhoD/bW6aGIxZoSbrQ8Jnb10JGFAIP1T/A=; b=tNejYPCWFGBvEswK0jw1T7MNUM1DeU+E6u78lKNqwkafE/P0ituy1yBd9/j8bQLae93k HrD3lGy7uHeX+8S9NNbyMIRPLdnxnxhiZMwAW1b+/FDXaA5vq4qKx8JtqMp4dtC7MbPz Lg+xSNp7yJ4psiJZiy6qqBzyT8sMxoS4rWLcamHvWK8YYhsV0+lvbcPSqFTp0uutWALX 6eN0oQIYAaMUTuoDuZWPqDBZq3Pq6q4RkcYB2NFHhe+7MBYiBYAVBoGZFlj8Iu6RXSPZ izpRi4XrewXLHxfHacanBLGz2BWZ1QwHsQJ0B2odptS1WhYn3e16VLHxhUycnSlfk7Uk 1w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmteku-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:21:45 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGOds037870; Mon, 6 Jun 2022 20:21:45 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2173.outbound.protection.outlook.com [104.47.58.173]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1ua9d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:21:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hOYwNwh9PJ2iuLXYc2H4XkM8KrdJpsMlGxlJPVqB838capwPO/MNHOwJLkMYwQBHWgvxtdwvr3Iu5vK7UZ+HOiZsoMboZn5CmZp0cAnpS7/Mrl5dbflBEo/EbZ0T8Nnr/dOctsZJwKHJYrfAwgbbBShcx5uWIpz9OZSxmI+vksubODZKjS0H8pswlzurlG8kPtjvWCkSDYbDqFQS+MRwO1qPoxYquWcg0dGA4bRM74wXmwKSulmwX/gGM3NId6djK6fIbVA02eyoslC4k7iIvlk7FUZmrFF7bugr9JHC3eW79jg9poEgVJsobLFVcIrC6AIf7H4uL1qvPYYOAFwBTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0nZQslULxAhoD/bW6aGIxZoSbrQ8Jnb10JGFAIP1T/A=; b=klI4bZ7DGzT/5Ta+IXtNsefvSSkirQGQQAvqUs75z8CgXv9ASRV8COWns7suvBqhouzdZ62f2gEYhrNrGbDZ021qCpvuLKiYXmsd3OTrkX+b/lyLRBcRiNWyjPr+FI3ZPFWhr8Hiu916Y+hOwZ52gTbhIoYzKXgoOIGdfghFbWGoVjjcUwVamKkFrYKF4qTvqCH+5PM6sRhmga4Hv36q1X4UKFodp6Ou0B4a5bIkSpfw2IMw+lRkca7pnjU+3bdUVrisIQn+Z12oDR1FR1lHfFgWd4koeTkIF+WRpGDvph3xkJtoXar2tOS9amAjKbBkiQy1WFIe33XL6WRzA0qVvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0nZQslULxAhoD/bW6aGIxZoSbrQ8Jnb10JGFAIP1T/A=; b=BYCaTh+ZIlXrOg7SEsrkA5ebwX0pQ3NrPHpZ4X/fltBRiL4uaYVCqVirwSHEd/JlngbdumYR9XW23o5E1CFPsZ0gvsbIZePoJUit5p/q01Vg7Pl0ua2bTxfBI941oj8QwJbbWZsMt7MR/Q7cRf2S4XIrJekFotBAVU4iMJ6jzvk= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by CY5PR10MB6046.namprd10.prod.outlook.com (2603:10b6:930:3d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.19; Mon, 6 Jun 2022 20:21:42 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:21:42 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 02/21] mm, huge-page: refactor process_subpage() Date: Mon, 6 Jun 2022 20:20:50 +0000 Message-Id: <20220606202109.1306034-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR06CA0036.namprd06.prod.outlook.com (2603:10b6:a03:d4::49) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 670c3c40-e8ce-4b2f-b6c6-08da47fa2b9c X-MS-TrafficTypeDiagnostic: CY5PR10MB6046:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LVFb+q6EcTOoNFuGvg+j0xnEFSea6ZQWISdgVXVnCywlX//twnY6bQf+XldV8NT4netvHAJLhg+iwiFtKbErIQ5fR8J1pq+3dY0inyMedLKu6UTZFBs2exBk1oYEAmBuT4Q0EkRt1wN912FtemQHggqZBrE30ZLFg8CNBb2WW1wCeYN57p9N0WzXn5wWXCq2Gyb2pifu5ldP6zkSEIEnj/OMf7wh4FnBfm7EeInarcz8x7Gob6THHL9oaT57dWbkgc3T3PVLH88EuaV37E20szOGDe7tpkpcZP251P3IgFvfHED2SRRFML4BklM/iS5dHABMbQvakDBTuXmE4XJtE7f5nishGWDrT4fV0bBEd/o1Hzs23fPu2Ulwpui9Aha3Wq1Qo74vLIeh8GePd2Cwfa2bzljtZtrxdecFUOaFgXBfCluJkC4nuiYPy+ih+s8cOPHfgkFB2hj+QYKAdiPIpH3WFJpe65jWvnt4myvWPHPKGsz0VWo/+7+xU0vHICSwd41J7+VcfBnbLH/A+VP069n/cNnPaz1EWKm99UmATBC0Q2/VJU3SjOwXHDmHapHFjZVYb9D+dCYQ+9QX7KvRCAHDqVujVM8t4UnDXDKp1uozJOoixTV1ItDMhC00tVuxIOYY3uLvN9apyx+Ek3Q5LLdPS34ERfT4mTdzJuYGz7J1C5uKUwpxun7C1QYtuWyDfDFFhgFVHQ3HBd8qm4eDLcuz5M/AIpBr6WlkGAzHkI4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(83380400001)(86362001)(36756003)(103116003)(8936002)(38350700002)(38100700002)(508600001)(6486002)(6666004)(8676002)(4326008)(5660300002)(66946007)(66556008)(66476007)(7416002)(2906002)(316002)(26005)(6512007)(186003)(52116002)(2616005)(107886003)(1076003)(6506007)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ym7dv3/nk0tK2cYAAvUU9neh7WOj2DVPXyUKJcn/jY0VAqjoJC4SUitkUNu+Uv7cXDRhcKWtP7p5+QXe+opKjXzsmj4YFSIKEoDGJzYNFFFOKDV2L+IcA+9nl0KygK3MxjLDNWtBxko5L5a7MW95BQbgzRUmlMuJMqyIxNGZGoUlmrMA2fdm7YvuJGsBLBykCTzTIT2sDXSEgNm5aWyFxd9sKdMCqx9fUoT2/edN8rrIKo/5r/tWCWfXH+95AvBILoUQTHS6TGd2GZdzmH3IwjJnEjVzxkcVsB4pHwMIhFZ85pIPb6UpxkbdnQxbAOosLP2kEINlQpFNf41vQmLRkkady7SQ9S+RThD28nHvV9E+Q1U5PkDO8fxbMuyN6Ww3WMuicX3Kc8vmeMh4C1WnOoQrfYzewkRYJRqpJSiZiY1m0ecZnu3kCl2UJxODYv4zRfrcyb0hTKKzRsyv2CIDFiYjAMlwadAplzVC+Wzbxky9Bf1xUAic0qy2TF8ZDInab4WGkp29Q45o6kc0e3CuczxpAkbN5C/gzgaFFpLU6dFUWOo+LMfv9WqNPYU8qFlkbXLJBI5WW2KO+85+gLECWo5ksltjRL/7xBvW2EaLkKBcPtb6+dpSeEp8S149LpDGfgqfS4I8JPQ8oHqBOACBgM4paCduTvUA5wh7UijCGurPk0zL0LV8b+bVLVLjwf1S0axBSxB2IOpPF4428pkkL0/J1FutryIvjSTE17iDRxKokDZ8MoYaXym1Hy1bFrtHfdHAebjoepdHxp7j6pQwtoZ9jQYOkQh6dRVqPGS7V5R24794iinEqxru+9a0s31rYqa6QPB64Chy9tiRS01mEZHi4gUseWJn3+RRAZXmdSEZHdwJzSV1nq2cAokrvSFo/kZxakbRTPP+7Zn32A9AYiupZ8tbLhY1WAX7CLLjHmxWB+BEcChrvtxQVK4maN0BZaGF4n/kiPqEXTvpE95svGl9jHVXPwCf12xubJHvrpQPCQ51iwoW4TPejaa8+EXcXkx8WKhdvoVUrzcGb/WdRjkcGMBid5QPHSBvWMTnybgkKTBF0iGYvpSNN7dBwn24EFIjBy8aOSK9Y0ei1SXPMCaQYuePAh9y9GHINM/OV6ufh9zNOhlELRVNiJArMaI7IOpZ9FEco28elmwe6oXLt9dp8mQn+3SLV83/w9c+OJ8XY+YVzB9FJHrMtmZTG7Eq5L80n2d4EgIt4BtX79vSbhd08PktMCNRtIXTONRLzo0vs+7nHTB+Gpx3ZVdPa79jws+cglRxEhJeCw+WdoJ6gu62NdAoc/jSywPF5K5xMHyJujDrVs5ZGUW/8+xCEpIAVrhsBIS35agHs5Sndg1Ge3N7EdgfeBI/b18LO43acE8bgVodVzatLPj2cWgdX00yz9crcU2xrhVMOk1bJ5EaYkeKqQMSWwugfIZDMsbSs3G5VpeE5tpKCvj2wv3xDwFibsWVdWorJGGJN7MyikyTeQT/R2LMgpHJv5I1qaBqkNJ2qXRWHg5pEEMVGI22ZPrfFMC1kSfNmkzch4zKLkGQH6Zp14IDHhkA1A/gi+SV3yuyfm5zC4HjUL26lEr6TtTJ6cRNV+Lvm5+GzhQUYKXtT8HigRj6YYfyuISC4zsfrHMGAOCb8PVo2tZE4SoCESnCO9D8kQtzpg2272ULDIdxkv6Gky0oZTrcRdpG1BF0d15hf1xaIqEZp8Hro4lGUoCLFXbwP0iHukdveTQJ2KZCs6ICnaonxomBemtzgMVkumA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 670c3c40-e8ce-4b2f-b6c6-08da47fa2b9c X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:21:42.6852 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sO4UK8aEDZaFr6xDKiB8/34ptkbPSNAoWptdylMs2O+YLNyii4yECzvl0rJaVvlxeaXNOfbQcDQS5PWeqsbodnu7E1q4eXp/kNKNySDxSII= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR10MB6046 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: LBOVjsWmQfqtDZ-N8b-Cdu1GatVx44mh X-Proofpoint-ORIG-GUID: LBOVjsWmQfqtDZ-N8b-Cdu1GatVx44mh Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=tNejYPCW; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=BYCaTh+Z; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf16.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Stat-Signature: khh9t1q1rjfzeonsugmfh7ikugqnsmr9 X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A980618001A X-HE-Tag: 1654546904-136103 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: process_subpage() takes an absolute address and an index, both referencing the same region. Change this so process_huge_page() deals only with the huge-page region, offloading the indexing to process_subpage(). Signed-off-by: Ankur Arora --- mm/memory.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c33aacdaaf11..2c86d79c9d98 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5577,7 +5577,7 @@ struct subpage_arg { static inline void process_huge_page(struct subpage_arg *sa, unsigned long addr_hint, unsigned int pages_per_huge_page, void (*process_subpage)(struct subpage_arg *sa, - unsigned long addr, int idx)) + unsigned long base_addr, int idx)) { int i, n, base, l; unsigned long addr = addr_hint & @@ -5593,7 +5593,7 @@ static inline void process_huge_page(struct subpage_arg *sa, /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - process_subpage(sa, addr + i * PAGE_SIZE, i); + process_subpage(sa, addr, i); } } else { /* If target subpage in second half of huge page */ @@ -5602,7 +5602,7 @@ static inline void process_huge_page(struct subpage_arg *sa, /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - process_subpage(sa, addr + i * PAGE_SIZE, i); + process_subpage(sa, addr, i); } } /* @@ -5614,9 +5614,9 @@ static inline void process_huge_page(struct subpage_arg *sa, int right_idx = base + 2 * l - 1 - i; cond_resched(); - process_subpage(sa, addr + left_idx * PAGE_SIZE, left_idx); + process_subpage(sa, addr, left_idx); cond_resched(); - process_subpage(sa, addr + right_idx * PAGE_SIZE, right_idx); + process_subpage(sa, addr, right_idx); } } @@ -5635,11 +5635,12 @@ static void clear_gigantic_page(struct page *page, } } -static void clear_subpage(struct subpage_arg *sa, unsigned long addr, int idx) +static void clear_subpage(struct subpage_arg *sa, + unsigned long base_addr, int idx) { struct page *page = sa->dst; - clear_user_highpage(page + idx, addr); + clear_user_highpage(page + idx, base_addr + idx * PAGE_SIZE); } void clear_huge_page(struct page *page, @@ -5681,10 +5682,10 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } static void copy_subpage(struct subpage_arg *copy_arg, - unsigned long addr, int idx) + unsigned long base_addr, int idx) { copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma); + base_addr + idx * PAGE_SIZE, copy_arg->vma); } void copy_user_huge_page(struct page *dst, struct page *src, From patchwork Mon Jun 6 20:20:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44171C43334 for ; Mon, 6 Jun 2022 20:22:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A174A6B0075; Mon, 6 Jun 2022 16:22:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9ECCA6B0078; Mon, 6 Jun 2022 16:22:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F22E6B007B; Mon, 6 Jun 2022 16:22:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 661F96B0075 for ; Mon, 6 Jun 2022 16:22:23 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 34285343E1 for ; Mon, 6 Jun 2022 20:22:23 +0000 (UTC) X-FDA: 79548933366.26.7036044 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf04.hostedemail.com (Postfix) with ESMTP id 1C0B640039 for ; Mon, 6 Jun 2022 20:22:00 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256JP4tu002554; Mon, 6 Jun 2022 20:22:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=hEvEoeSig8HHMEjaixIlVBLZWPY91AfXXyj9GOsNcyk=; b=V2t32qvfAGjRFZSOnyB6sQ8VbMydLdjZtKZ+hxG3DZIfgj8X4dvW1ZMOYHMfiGtgmuf0 OAm4WDt8y/I36qLkIgYIWzep1smhu+A+XE9sRWdhJtG+ihjqI53Et224kMJXQesQr/Kj JOXdoqwyHVPSOFtKVqfY/1JZU1YGZm5lz4/zBaDpbstEa7iqKVPVQL4x7ZtbBLl6FHMp ItWdhT/Jgz5OOowrmHD+cOenpnEvs+BqgnAffZRoyuZp3Jq7NjErCM9NuvX/DjDpPRuE 6tmZK2Or6JGipOGGgWb1xJfW4GoVh/VL1Ysn03D5PMit8wbma2oEce3TZHwZGCoVH+FT AQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfyxsc5fh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:04 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KF83w001176; Mon, 6 Jun 2022 20:22:03 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2168.outbound.protection.outlook.com [104.47.58.168]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu9123f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=is+0HuNgrhDJHTZn58JZOw+pTBMBAn/cq02PdcNMmT/Sm5eetg7nIrSGFJnlV+sAiX/HPUrCxmud5X23b57nEY3srUipjvwwcr8QrvgfJO0KSQirU9fXCJ97tDblQUzMJu8E+GRQKy+/N7WDkJ26BS5DI4vtfYpGKs89bG3cJtzFhFjXmKz8r76XvKqvcBquRulJ79T1VaFdlaX+Zi7JVkS+GEGJw3eAPdijNPVXnGcN5t5bPWhT89/F6uEezHmkBVdhw/BYzJl+9P+xO5CjAfoXcdPPGRNgyPyqK9zUc8Hs4qyc+pKxnLh2Mtmx105UGbji+I+HGw0klp4+3uMEqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hEvEoeSig8HHMEjaixIlVBLZWPY91AfXXyj9GOsNcyk=; b=XoPPxaGwemCSsxG9IQsNewHNJm98u57VFnVLv341yNqFKPfLcfXTCmHApolZwoYRihO4giFU4tY9Po1pzuMG6ru5U90SMi/cw/22pVsllgFOc0UA+j3xD192darQQVlKexnHHPVuaepULbcNITlC4N/qT79Brf+kwjA9nB2160YdQez8OKMPqDUvP87XEGGP0Z+C1Y/1Axl/qOvsuhkibLFY9HRsUoos3iGlTWGA25qEt2OHtt4pUh99quO9NVOLkR8AKinty3NjrLGut7kk8V15qqSHQZc9WsS8bFaWitYre+ikGvLc4N0KoGSs+ytP6cm5Y1VosbT3u6Rz85Hncg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hEvEoeSig8HHMEjaixIlVBLZWPY91AfXXyj9GOsNcyk=; b=G+qxufo/kiRs7Ei+1hGOkcVgZilD8jSmc6a7AM4NTjPseeVqtoahJA/+Tixh3BITYQ2K4sJ0AIBb5xdwHJQDXqW75P0iumDkmpQXYvNf3fgMEbnQnQtT8IfzU5LvRGe3QFYEsAAvYu9XzVJSYdpvmB3nBE6v0cDR6UvLJwuUTdg= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by DM5PR10MB1772.namprd10.prod.outlook.com (2603:10b6:4:11::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 20:22:01 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:22:01 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 03/21] clear_page: add generic clear_user_pages() Date: Mon, 6 Jun 2022 20:20:51 +0000 Message-Id: <20220606202109.1306034-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0034.prod.exchangelabs.com (2603:10b6:a02:80::47) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 95c6d1f2-f12f-4f34-4e99-08da47fa3658 X-MS-TrafficTypeDiagnostic: DM5PR10MB1772:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OYg15oXLjxoAFPvRRajQMYLdp+8Xe04grDw94LCwxeSdabQFra+Sb066b1Yl8+H8tVNFYp7bpgtlqOWBmRkTnNsBhren1HV951PlIP/HRqTZGLRBfyUnCR0K0bFfX0I5GlyiKzXEiiMB9QoOvBCRPIWi88erLSscDzI2l+hTuRlXr5SzxX4fT7KT4rbHVJdT+P2r0kfQmjjub/ZM68Q/g194jOgpdxRkrIUHpBldJ3MlOFaRnDSjV7Vn8G8Jy4t8FzCTcOQchDxr7iMUNQOMo1vdvXP0s5dXx4GBXaNEXRJ4MqFwTQRXDGunirM3VVpuTgtSfeHDkgiBcOqSgZNohcuJotFA+8oBbdBsJP6trhIyMl8VNP9U+5EyM1nlDT1LcCeeT8Q82JmBQ198QuBlJxJWZF+BUHddxOKlnlBXAGYUqsUJ8WwwzeEkoWwn/mYCNx+EQwAOFpu0T2F2xwtuEq5yHkckiTyhhKGrvLDHRRZGqf1VFqxsIOaZtktftD2nbHg3fSirNWpaGsQbQ0bLc5deQCxp6mH1wNVEEw6NEJYJlCAaVUZKl+WGCVnSyjl1KpeyK7uA3wwQzVAjbbtJhWVWDHOGhSJ6FeTE3b0G3vvypWbzbnwvyebtjtAaQ3xObGvSHOin+8BklWhKDGLGUK3Lbvq6pKsCqpisEOJndvNhuX9slyjZt/K0YJduTlK4UZhoZIGnTkrcsdJNWdzAe6Tqr2kmJWBZPzEhWQGqkkt4scb8ps4esSMDvS0+rNV3 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(30864003)(8936002)(7416002)(6506007)(83380400001)(107886003)(66556008)(36756003)(1076003)(38100700002)(186003)(6666004)(66946007)(508600001)(6512007)(26005)(2616005)(5660300002)(38350700002)(4326008)(8676002)(316002)(103116003)(2906002)(66476007)(86362001)(52116002)(6486002)(41533002)(21314003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9j//PY9j/QZ9WtKluHN6V3uNY4IFNZsk8RyAh5JQbz2TiR6YNsC0uvxH1Rp6t9YJ96P60Ip4mLzfHrFQNw3ByJe0dJAnFe/YomFgzDeJ6yrLOHH8GJJJoiQrrbpDtunFO41Wh2wPEEmi1lk7y23NYOSwKIA/5gpcbHpOfYxP9Xvx2/wGcnmxO9LQyqV653dqiW6mmjp8shXo7FfJhXTBRCkCoOvMmSKNBp6HlcK2pm+Acy+kXoOiRbLSJYNrcAx0HdWsCIFRtw3tyCdmeLzobsaJxHAyGZueDH1hX8v4WhlVKRh7lWC1dUff8XPIa6afNm/kVrz8NI2RjR/0d3GWcventkJQMhvZoyErqfdUJLPRxDs2LggIXVtyn3upY3CfLOPGour3ZmBHaoxvsGbrkBMRJjzGly2aW0JLvSsav11bNOWE0UvMB9jCoJvi2QGWLCHDNA/ouf5yipKqj+GJl6CB7zDiWikr/NUKXV+a2zndv1NqA1yoNzPOovTwa/rgDhVkHlMGdEZnIWqmKT7E2B7z1Kysag9oZGpCS2Bs0vgLoKM+KNWvWP6LIWv1hTUzwR4Lzrv0aKY+ylGRljV6XLAXU5l9qlJmbZte1Z72Cyy6o6vSIWH4OH6NYo90MVjBa6cd2VqXZJ0vvjHUAbkxGHFRGF57Xf4jLNWDesFtoUTWCq/cKwllPDwtAkXWcLC/4/rr9glVBULUr/K7t7Kq6zW4r2BRHfKOi7d1n8m4n/3el6BSfqeuy6OE6GHO+kXiLSHXVOWl7SslXCnWU4cUh3n4ampbegp4SFv1KJmseoZoygcKC5jswct/MyQcLVXTjEbjbKoIyi6CNj9DZ46u8eLs3fcKZ9BzAZD51doKboUU1REs+QszmeNsIU/i1g1vmpZXO/U0U5Bjg57JzLu+z8hVdRgP8zlnqhl46TuHLoGYYnImrk0tSHXiX+TNTvKQxVUqBEI+rxwPWziX2TYZbC7LWSVa3+aiOC2wwV4Ur537NLJ6lnuKbWixXuXGAY4o7TEyOdw9IvkbwfPKU3vK09K+iWUHIYcoQOGqr37kKaoCsKgTixiMn59lb2Q/dZrC67Ij91CtjeGdr8fRhvYdg1WEBGa53+xkZiBk5YSI2petfzjByIR/w2q95fPFmi1H8+VV1bhlOzYFAhyLk+BlUra2TwUqaxmO7nPhlB82arOo/4Gf4TuvE0R4/iME+Y0SwYuE0cnGq7A64105IjiMH+FyRZN3keQiSQTpZpcIOvN4+PGpuIfPxVqdjWeGEKwe/rLvcWGz/uOvZOOaf+k+xmHzbZNPD4kiIfLZecUfX4aP/6xQclaefiR4pMIsWbsCoIMl54fzZixImzoyIIA4epqKgwVCZxhSxvVwrm0gm/rLf1sUKi1rqUpLL9Orivdx64SbaxRx9Rc3WMQmxI+JukKw+z9p05Y/StaFNX5GsF8QY8UMa1oosMpUBRw4YSaszg29DgUwTAycu8PdLYTZv4cG9usYmcY20a31XAn8vnfweenu0mRDCVflD4WUXpvG5A98jevFyTY7EFxZgCNrmKCFSMYJ1tDjQC4vscrnWYglXJaFCz96o6WfZOZz9JhtMivy8vNx8L+8ftk3Y2IYlEDBdjjUmLPmgDtDtkahtY5/shAGWY5fPz4raxWjDbClWEFFoFV5JS7TMshC/VxBnF3koRuggeA0OOJLvL3pO+c5WqfgRUEPMG4LqZCtpK0XQoWj4H/3AnRg670A9oxgdwl6dFmhnhP7gNBaX33QwH4= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95c6d1f2-f12f-4f34-4e99-08da47fa3658 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:22:00.7231 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UxDC99v9v0IUM/q3lge/Sai9x3yAHUpZVXXe3CpA9lE94KEWcOTqtANr1pTwO8Zt4YHaiNxuhrah4JAjLPl8rOa7OWGVyBIZT0WRDIk2yMo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1772 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: slYONR98P7rk9SvASI4CRCxuMoYz3CXJ X-Proofpoint-GUID: slYONR98P7rk9SvASI4CRCxuMoYz3CXJ X-Stat-Signature: eeiahwub7d7ojf68zgg7tnkcnwfb6haz Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=V2t32qvf; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="G+qxufo/"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf04.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 1C0B640039 X-HE-Tag: 1654546920-381638 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add generic clear_user_pages() which operates on contiguous PAGE_SIZE'd chunks via an arch defined primitive. The generic version defines: #define ARCH_MAX_CLEAR_PAGES_ORDER 0 so clear_user_pages() would fallback to clear_user_page(). An arch can expose this by defining __HAVE_ARCH_CLEAR_USER_PAGES. Also add clear_user_highpages() which, either funnels through to clear_user_pages() or does the clearing page-at-a-time. Signed-off-by: Ankur Arora --- Notes: 1. I'm not sure that a new header asm-generic/clear_page.h is ideal. The logical place for this is asm-generic/page.h itself. However, only H8300 includes that and so this (and the next few patches) would need a stub everywhere else. (Just rechecked and looks like arch/h8300 is no more.) If adding a new header looks reasonable to the community, I'm happy to move clear_user_page(), copy_user_page() stubs out to this file. (Note that patches further on add non-caching clear_user_pages() as well.) Or, if asm-generic/page.h is the best place, then add stubs everywhere else. 2. Shoehorning a multiple page operation in CONFIG_HIGHMEM seems ugly but, seemed like the best choice of a bad set of options. Is there a better way of doing this? arch/alpha/include/asm/page.h | 1 + arch/arc/include/asm/page.h | 1 + arch/arm/include/asm/page.h | 1 + arch/arm64/include/asm/page.h | 1 + arch/csky/include/asm/page.h | 1 + arch/hexagon/include/asm/page.h | 1 + arch/ia64/include/asm/page.h | 1 + arch/m68k/include/asm/page.h | 1 + arch/microblaze/include/asm/page.h | 1 + arch/mips/include/asm/page.h | 1 + arch/nios2/include/asm/page.h | 2 ++ arch/openrisc/include/asm/page.h | 1 + arch/parisc/include/asm/page.h | 1 + arch/powerpc/include/asm/page.h | 1 + arch/riscv/include/asm/page.h | 1 + arch/s390/include/asm/page.h | 1 + arch/sh/include/asm/page.h | 1 + arch/sparc/include/asm/page_32.h | 1 + arch/sparc/include/asm/page_64.h | 1 + arch/um/include/asm/page.h | 1 + arch/x86/include/asm/page.h | 1 + arch/xtensa/include/asm/page.h | 1 + include/asm-generic/clear_page.h | 44 ++++++++++++++++++++++++++++++ include/asm-generic/page.h | 1 + include/linux/highmem.h | 23 ++++++++++++++++ 25 files changed, 91 insertions(+) create mode 100644 include/asm-generic/clear_page.h diff --git a/arch/alpha/include/asm/page.h b/arch/alpha/include/asm/page.h index 8f3f5eecba28..2d3b099e165c 100644 --- a/arch/alpha/include/asm/page.h +++ b/arch/alpha/include/asm/page.h @@ -93,5 +93,6 @@ typedef struct page *pgtable_t; #include #include +#include #endif /* _ALPHA_PAGE_H */ diff --git a/arch/arc/include/asm/page.h b/arch/arc/include/asm/page.h index 9a62e1d87967..abdbef6897bf 100644 --- a/arch/arc/include/asm/page.h +++ b/arch/arc/include/asm/page.h @@ -133,6 +133,7 @@ extern int pfn_valid(unsigned long pfn); #include /* page_to_pfn, pfn_to_page */ #include +#include #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm/include/asm/page.h b/arch/arm/include/asm/page.h index 5fcc8a600e36..ba244baca1fa 100644 --- a/arch/arm/include/asm/page.h +++ b/arch/arm/include/asm/page.h @@ -167,5 +167,6 @@ extern int pfn_valid(unsigned long); #define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC #include +#include #endif diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 993a27ea6f54..8407ac2b5d68 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -50,5 +50,6 @@ int pfn_is_map_memory(unsigned long pfn); #define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) #include +#include #endif diff --git a/arch/csky/include/asm/page.h b/arch/csky/include/asm/page.h index ed7451478b1b..47cc27d4ede1 100644 --- a/arch/csky/include/asm/page.h +++ b/arch/csky/include/asm/page.h @@ -89,6 +89,7 @@ extern unsigned long va_pa_offset; #include #include +#include #endif /* !__ASSEMBLY__ */ #endif /* __ASM_CSKY_PAGE_H */ diff --git a/arch/hexagon/include/asm/page.h b/arch/hexagon/include/asm/page.h index 7cbf719c578e..e7a8edd6903a 100644 --- a/arch/hexagon/include/asm/page.h +++ b/arch/hexagon/include/asm/page.h @@ -142,6 +142,7 @@ static inline void clear_page(void *page) #include /* XXX Todo: implement assembly-optimized version of getorder. */ #include +#include #endif /* ifdef __ASSEMBLY__ */ #endif /* ifdef __KERNEL__ */ diff --git a/arch/ia64/include/asm/page.h b/arch/ia64/include/asm/page.h index 1b990466d540..1feae333e250 100644 --- a/arch/ia64/include/asm/page.h +++ b/arch/ia64/include/asm/page.h @@ -96,6 +96,7 @@ do { \ #define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT) #include +#include #ifdef CONFIG_FLATMEM # define pfn_valid(pfn) ((pfn) < max_mapnr) diff --git a/arch/m68k/include/asm/page.h b/arch/m68k/include/asm/page.h index 2f1c54e4725d..1aeaae820670 100644 --- a/arch/m68k/include/asm/page.h +++ b/arch/m68k/include/asm/page.h @@ -68,5 +68,6 @@ extern unsigned long _ramend; #endif #include +#include #endif /* _M68K_PAGE_H */ diff --git a/arch/microblaze/include/asm/page.h b/arch/microblaze/include/asm/page.h index 4b8b2fa78fc5..baa03569477a 100644 --- a/arch/microblaze/include/asm/page.h +++ b/arch/microblaze/include/asm/page.h @@ -137,5 +137,6 @@ extern int page_is_ram(unsigned long pfn); #include #include +#include #endif /* _ASM_MICROBLAZE_PAGE_H */ diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h index 96bc798c1ec1..3dde03bf99f3 100644 --- a/arch/mips/include/asm/page.h +++ b/arch/mips/include/asm/page.h @@ -269,5 +269,6 @@ static inline unsigned long kaslr_offset(void) #include #include +#include #endif /* _ASM_PAGE_H */ diff --git a/arch/nios2/include/asm/page.h b/arch/nios2/include/asm/page.h index 6a989819a7c1..9763048bd3ed 100644 --- a/arch/nios2/include/asm/page.h +++ b/arch/nios2/include/asm/page.h @@ -104,6 +104,8 @@ static inline bool pfn_valid(unsigned long pfn) #include +#include + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_NIOS2_PAGE_H */ diff --git a/arch/openrisc/include/asm/page.h b/arch/openrisc/include/asm/page.h index aab6e64d6db4..879419c00cd4 100644 --- a/arch/openrisc/include/asm/page.h +++ b/arch/openrisc/include/asm/page.h @@ -88,5 +88,6 @@ typedef struct page *pgtable_t; #include #include +#include #endif /* __ASM_OPENRISC_PAGE_H */ diff --git a/arch/parisc/include/asm/page.h b/arch/parisc/include/asm/page.h index 6faaaa3ebe9b..961f88d6ff63 100644 --- a/arch/parisc/include/asm/page.h +++ b/arch/parisc/include/asm/page.h @@ -184,6 +184,7 @@ extern int npmem_ranges; #include #include +#include #include #define PAGE0 ((struct zeropage *)absolute_pointer(__PAGE_OFFSET)) diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index e5f75c70eda8..4742b1f99a3e 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -335,6 +335,7 @@ static inline unsigned long kaslr_offset(void) } #include +#include #endif /* __ASSEMBLY__ */ #endif /* _ASM_POWERPC_PAGE_H */ diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 1526e410e802..ce9005ffccb0 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -188,5 +188,6 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #include #include +#include #endif /* _ASM_RISCV_PAGE_H */ diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h index 61dea67bb9c7..7a598f86ae39 100644 --- a/arch/s390/include/asm/page.h +++ b/arch/s390/include/asm/page.h @@ -207,5 +207,6 @@ int arch_make_page_accessible(struct page *page); #include #include +#include #endif /* _S390_PAGE_H */ diff --git a/arch/sh/include/asm/page.h b/arch/sh/include/asm/page.h index eca5daa43b93..5e49bb342c2c 100644 --- a/arch/sh/include/asm/page.h +++ b/arch/sh/include/asm/page.h @@ -176,6 +176,7 @@ typedef struct page *pgtable_t; #include #include +#include /* * Some drivers need to perform DMA into kmalloc'ed buffers diff --git a/arch/sparc/include/asm/page_32.h b/arch/sparc/include/asm/page_32.h index fff8861df107..2f061d9a5a30 100644 --- a/arch/sparc/include/asm/page_32.h +++ b/arch/sparc/include/asm/page_32.h @@ -135,5 +135,6 @@ extern unsigned long pfn_base; #include #include +#include #endif /* _SPARC_PAGE_H */ diff --git a/arch/sparc/include/asm/page_64.h b/arch/sparc/include/asm/page_64.h index 254dffd85fb1..2026bf92e3e7 100644 --- a/arch/sparc/include/asm/page_64.h +++ b/arch/sparc/include/asm/page_64.h @@ -159,5 +159,6 @@ extern unsigned long PAGE_OFFSET; #endif /* !(__ASSEMBLY__) */ #include +#include #endif /* _SPARC64_PAGE_H */ diff --git a/arch/um/include/asm/page.h b/arch/um/include/asm/page.h index 95af12e82a32..79768ad6069c 100644 --- a/arch/um/include/asm/page.h +++ b/arch/um/include/asm/page.h @@ -113,6 +113,7 @@ extern unsigned long uml_physmem; #include #include +#include #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 9cc82f305f4b..5a246a2a66aa 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -85,6 +85,7 @@ static __always_inline u64 __is_canonical_address(u64 vaddr, u8 vaddr_bits) #include #include +#include #define HAVE_ARCH_HUGETLB_UNMAPPED_AREA diff --git a/arch/xtensa/include/asm/page.h b/arch/xtensa/include/asm/page.h index 493eb7083b1a..2812f2bea844 100644 --- a/arch/xtensa/include/asm/page.h +++ b/arch/xtensa/include/asm/page.h @@ -200,4 +200,5 @@ static inline unsigned long ___pa(unsigned long va) #endif /* __ASSEMBLY__ */ #include +#include #endif /* _XTENSA_PAGE_H */ diff --git a/include/asm-generic/clear_page.h b/include/asm-generic/clear_page.h new file mode 100644 index 000000000000..f827d661519c --- /dev/null +++ b/include/asm-generic/clear_page.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_CLEAR_PAGE_H +#define __ASM_GENERIC_CLEAR_PAGE_H + +/* + * clear_user_pages() operates on contiguous pages and does the clearing + * operation in a single arch defined primitive. + * + * To do this, arch code defines clear_user_pages() and the max granularity + * it can handle via ARCH_MAX_CLEAR_PAGES_ORDER. + * + * Note that given the need for contiguity, __HAVE_ARCH_CLEAR_USER_PAGES + * and CONFIG_HIGHMEM are mutually exclusive. + */ + +#if defined(CONFIG_HIGHMEM) && defined(__HAVE_ARCH_CLEAR_USER_PAGES) +#error CONFIG_HIGHMEM is incompatible with __HAVE_ARCH_CLEAR_USER_PAGES +#endif + +#ifndef __HAVE_ARCH_CLEAR_USER_PAGES + +/* + * For architectures that do not expose __HAVE_ARCH_CLEAR_USER_PAGES, set + * the granularity to be identical to clear_user_page(). + */ +#define ARCH_MAX_CLEAR_PAGES_ORDER 0 + +#ifndef __ASSEMBLY__ + +/* + * With ARCH_MAX_CLEAR_PAGES_ORDER == 0, all callers should be specifying + * npages == 1 and so we just fallback to clear_user_page(). + */ +static inline void clear_user_pages(void *page, unsigned long vaddr, + struct page *start_page, unsigned int npages) +{ + clear_user_page(page, vaddr, start_page); +} +#endif /* __ASSEMBLY__ */ +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES */ + +#define ARCH_MAX_CLEAR_PAGES (1 << ARCH_MAX_CLEAR_PAGES_ORDER) + +#endif /* __ASM_GENERIC_CLEAR_PAGE_H */ diff --git a/include/asm-generic/page.h b/include/asm-generic/page.h index 6fc47561814c..060094e7f964 100644 --- a/include/asm-generic/page.h +++ b/include/asm-generic/page.h @@ -93,5 +93,6 @@ extern unsigned long memory_end; #include #include +#include #endif /* __ASM_GENERIC_PAGE_H */ diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 3af34de54330..08781d7693e7 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -208,6 +208,29 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr) } #endif +#ifdef __HAVE_ARCH_CLEAR_USER_PAGES +static inline void clear_user_highpages(struct page *page, unsigned long vaddr, + unsigned int npages) +{ + void *addr = page_address(page); + + clear_user_pages(addr, vaddr, page, npages); +} +#else +static inline void clear_user_highpages(struct page *page, unsigned long vaddr, + unsigned int npages) +{ + void *addr; + unsigned int i; + + for (i = 0; i < npages; i++, page++, vaddr += PAGE_SIZE) { + addr = kmap_local_page(page); + clear_user_page(addr, vaddr, page); + kunmap_local(addr); + } +} +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES */ + #ifndef __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE /** * alloc_zeroed_user_highpage_movable - Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move From patchwork Mon Jun 6 20:20:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A983C433EF for ; Mon, 6 Jun 2022 20:22:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE70E6B0078; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D96506B007B; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B74D56B007D; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A1AC26B0078 for ; Mon, 6 Jun 2022 16:22:24 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8436560A4F for ; Mon, 6 Jun 2022 20:22:24 +0000 (UTC) X-FDA: 79548933408.08.D00577E Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 92559180013 for ; Mon, 6 Jun 2022 20:22:19 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KJjoC030086; Mon, 6 Jun 2022 20:22:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=XSThaI5efd5j+aWpu4xuZcVqnFgJClYUuiZswgGeBPUfdrF/YofPWTV4RNIiJSSy7cSG zpK7LZCFmPjxl+CpnmH0sBqCSZGEzCR9/KUsEDv7BN09bOW/DnKxFuf7CAJ3h3rkgKn2 /+f8aDK8vBnrxywjjo3XyZGLxmZrvliOBsun3xid3nAwh9zaAeSxYOirOS1fOJKQ6Udz UeX0VEdyk1TPm8WR8zSQCbFW38zQrsBJQqBQMUocgAp0QSO5mBOpPz7eZsrhcx7lnXle 4JlxJ86umZkiuBJwfz2W4sWnGgmVH3AvlwKOrh7fhcYMqP1WJAbq6mV3wav7x/jc8q2z WA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad84qy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:08 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGEXr003280; Mon, 6 Jun 2022 20:22:08 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu2bt3r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:22:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=md3YSVD2WL410xId5KsVGrrgBh97VuNC4dmOa2zf9YTTxREVkajpdWa5ZHze3j87NVldkEbxTYzVRkvgYcW4KpLvZua2weH5E5h+BsjsSgg8uZeaxmZagdN62FEygA46+7K4yYnsjX9xVmAvHINThEI6oyCFkO/MYWZt08PtS989fGNX0WoQIaNPGdcefVMnyskVVt34YgyR/vIxVJd8rfvdYQ+OnH4paxkZgoemW3kai5XjIaN8HQnC3tDQYHq87fW/q5LsFccLt9F4qOaX6qXREfHVKqEbLbPHc7GNzTBX/RRiuwjk66TgU14uOXX7DKXqeEYMBudaeyuzr6eR2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=Pw5xFKv53wTk0iYxs1/f0XOgTWpgQzIN+MAGLPQl2TM2DfdDL+GC1qeatZaRO/JQtme1X+XAEFSLYCNU/iVV+wgR2HUx2EMZ1EtekYk0rAK0H68xfTmN3OpCJ3poF61dzLKVH6qvLGi2MUtAqDBu2mb93O2tMul6X01GEV8ho0V6yIHCa1VUs/6opGiOYPzhhYFmvazCXAw/7dPclcnTwgJZOA5JvEOx9N0dMRyOiRjHK4wkMyufWKX4dpFZy5aFtOcciDOcmrUU43vMLw6bxwVMYgeKA1f3ibJ8/8RIg3d4OUCgWF40N8+Vdb0ktGV4MQCW1Vbt1TPWJeWQuA9IWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6BJ963cSHLwoWXx+Sn1JXEa7kfrO5qnc4QbhLDXzJfM=; b=NWdT/+Y2K4pGMYi7xbHKmQSeTL8tkQm2A6pnou0u4mKZ/XUzr0egvjkNt/oACen0CpUF0vZZ4js+1d5YmW+hqbcyKKyxSYUO2kIeTYtn85v8pICBC+3TOORE1Z2fz1Z6ijq8UJNiB3m2hT9+kRPZebINKfKIK1fIaku+g94xysg= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by DM5PR10MB1772.namprd10.prod.outlook.com (2603:10b6:4:11::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 20:22:05 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:22:05 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 04/21] mm, clear_huge_page: support clear_user_pages() Date: Mon, 6 Jun 2022 20:20:52 +0000 Message-Id: <20220606202109.1306034-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0034.prod.exchangelabs.com (2603:10b6:a02:80::47) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e76cfaf1-db89-4c79-6177-08da47fa3948 X-MS-TrafficTypeDiagnostic: DM5PR10MB1772:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YK6dU+WVvxawlCO6baPU3XgPpQMrOkN04BqU6vXolYAQoDtuGld7NzufJDLr5yhvjszGJzxhWeeWPFbb/yUl4OKB9VY9hX1NVcGfm+/R/3okMufB9H/jV03dStNZ8QIDmnXLyjctOKIzRwc/EbT4Y8TToBdYDAZuf7T4F5v/e2A7EgQ36UbKct0QPgHmV9+ikmRO31qMIwD7DlhYKOlJI5fF1iGM1WilcKPO8kSp6JWJ4B53oHEEFOWWUcS3QFz+//SUUD3s48d7EZnocmyXjE8iSJZKpiHO7r7/Zfg8zflIKMblPms8YrE/SABwiXz51HlBNdfnfvsxsxwof7E3NK27VDsv/4vgGGgcgSQuHZj1tbDqvCAgB0jL1K3LBlreVPRsIpn1glAyLNOaoU4F0rKayT9K2R6mZnBND6fKqBfNBcKT9pS9RuBUs8Pp4k+XSHUEuRQioGyDXlA+Y5VHkUsK4VGro8o/L2ZPBessZv1bxePs75ZwXfD3rr/wHa07lF2fsBRSBg7ce/2unWtYu2V84bDppElzPvQ1hDmZFf6bKWapEiHV3nYt2N8D84VfwqD4AEASHufUPYO32MhRDp8ZeXPxCpEpoq/xuseKhOzKoLa3yTZ4+ks4RjDZfEK361dtbcrP75GKWCFKXFvW8kiS1A1qcnEynyVzNiYkvPXHXN1bXR6bLeUY2XKSVBX/8Nx9/aptu9904jGluI73zSVJWJSp7snF1g1Z00NRcRw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(7416002)(6506007)(83380400001)(107886003)(66556008)(36756003)(1076003)(38100700002)(186003)(6666004)(66946007)(508600001)(6512007)(26005)(2616005)(5660300002)(38350700002)(4326008)(8676002)(316002)(103116003)(2906002)(66476007)(86362001)(52116002)(6486002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eIGiphxJVBJHg/dOZ2mWIy7aJflRUe16uqgUYp2oRx1+OguaDfVQb5GpbijpSyOn9K8kE32kRVPdYm6P0MQUUYIbPDi+ptKgi2EKjzd2rcsxvRYMwCnBNslvrf0CrxTW1gRat3cLw3jJxpLt+JhfIZEsCeciLK4zN4uE77KEQxI1KUy8bESLGxSWbuaFW1ynsZIrOtGysd6Egd0svED8CjrliSXq9ibLv44n3g43jxclm3LkR6BeVer3XM5olK0AEBZBOEo/mvx1BBIq5zBZgxU6tOmGxMQtMuYeCrMlaf+KfYe1Xc5fDgyWwaQC6otl30HTUPTtCdOHJyGmpMNhMW25Ol2s/dPD1mnPEzwVT3HIs/ARaIB849XjMH779UiLjPy6T1QD/9bY/bRtUtAC8UzxjXIp8LTY3+aliAb5rGqpv9ydkLOvwdKbeDZK0pPW67nAlaYwh/9KLhx9umSPO76KG8Po9NTPSLGr6dRiqR6xQr7qUa9ZBwUuK+JSGOWeY+vyOcZDpApbuN9DmQo+SO6rgRZ+bm5ez4mplu41VTJHYmcUss0CnZA5EeBf6dMtNf8knP/KFCh/AAcZfNuQeZahgXE3tsQTqvyf0J+FUMk7X9Tvjbp+RePisQGTj/f/C6NwCkqoJU/AYB1nWeadJogEDmAJ/aPjRFifkWfabdu7O9JZ8+K3Ur+oBr0pW6UUHx+30rdG+3KMTXXS237WXk6KZHmGt0bsc8YV4jRB+IMeJCBIcioFlYVxQcChLiDophEu62tFNhW5J8xo6j/wbfa9GqPB6v3dxWd2TDKimKvwL7us/0L2DsXlD33T2DX/r+E2Vie7r+Eg1sW7JV3c6JipahUEDkJ9vXxS3G4X3bvFCQkN4EYFO6MFBNb/ckgLRu6QgLnDfzZZrqQIn4bO4mPR4IRAEAGIf/MaKeEbyjavilcXtKvqXIyKRud1C+lRgte3FmHHswbXr+mUTWWqloCFtQ3tQH8zIA87W8b6QlKAGzyWPIBOTb9tkFKUihng1hnWBNqCx+UqHzznEqIGCDVL8LrKSgz+AZZ2oxXpx9ilx5LBJr5wPbIsCJMmq1oWqeyOXVnHPr7SFpZ1PKmdeSDRjybuKdtFXbUS+ai417OUofCU65dRELG19Vgxx56cLu0HmTmauB8B7kKpo6s7QyW7eXBdsJjZ+VjJIA6KnofR+lBBiP2MX4OV9gbVzGMKmf0SdQPowjn5ae8lziBZblPAMlDFUJtk/Vjy9TXUoTh89U06VxYopXPR4r88OcFWSv3aDFYt1tTZtwYGJLHbHw4NArB63AKQ3IE454jD91QKmXdqDvw8AZNfdolXMLX02TiiKrOAOfcdkb5aMArRzMmdmXOcfdNWGSj6AKs8hFASQrlKlNijYDHI3cebl81/scN59fba1/p4lUDtHC7B/CiqVd4mT8Nx0RwSa5w/KLAgA8+E1J8lwkmWcMDLMJ+pTaSnZWqiNowQGinjiSBK5+eiLGmcBZj3E2WCawc/dHg+mUJKWXW8EJcuc6ci2C+jI5c7QBMEtEuswfS4CtVceBEAsRse0oIwolO28OMg9/MMj1MY9VVsI/HP0Lf3J5g5MH7xXD+w3OCadQD17uJ3RMpJ9bKqyHtIvesUnwhQh3+Hy+5gnndSN/3SDPuQtzQMx9a9FLn2tavfHIPluvVPiGpuRaW54yiGqyih6oV2CL8ja4yHDJbGPQrh3ITIZZr/Xhg+J348H4WJAQqmCV/xe7DoGZLM+Y5HEN/IGx5vq8U= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e76cfaf1-db89-4c79-6177-08da47fa3948 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:22:05.6818 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: t8vkoxPS9odXsOp5gm7LcQuVAP+g6fvtqh/MEbErxdXyO88H9Ob/o7DtsghT3horY+0fKfZNgQE6GQIc9QcHQ16PsGb6gaWZaT9tnSiJMPI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1772 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: qqGtsA2CUbLUkubRmSxzRBXAEcra_0nE X-Proofpoint-GUID: qqGtsA2CUbLUkubRmSxzRBXAEcra_0nE X-Stat-Signature: o47rkabksfarccikgb438u59k7u1bwzn X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=XSThaI5e; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="NWdT/+Y2"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf06.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 92559180013 X-HE-Tag: 1654546939-446670 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: process_huge_page() now handles page extents with process_subpages() handling the individual page level operation. process_subpages() workers, clear_subpages() and copy_subpages() chunk the clearing in units of clear_page_unit, or continue to copy using a single page operation. Relatedly, define clear_user_extent() which uses clear_user_highpages() to funnel through to clear_user_pages() or falls back to page-at-a-time clearing via clear_user_highpage(). clear_page_unit, the clearing unit size, is defined to be: 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER). Signed-off-by: Ankur Arora --- mm/memory.c | 95 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 69 insertions(+), 26 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2c86d79c9d98..fbc7bc70dc3d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5563,6 +5563,31 @@ EXPORT_SYMBOL(__might_fault); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +static unsigned int __ro_after_init clear_page_unit = 1; +static int __init setup_clear_page_params(void) +{ + clear_page_unit = 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER); + return 0; +} + +/* + * cacheinfo is setup via device_initcall and we want to get set after + * that. Use the default value until then. + */ +late_initcall(setup_clear_page_params); + +/* + * Clear a page extent. + * + * With ARCH_MAX_CLEAR_PAGES == 1, clear_user_highpages() drops down + * to page-at-a-time mode. Or, funnels through to clear_user_pages(). + */ +static void clear_user_extent(struct page *start_page, unsigned long vaddr, + unsigned int npages) +{ + clear_user_highpages(start_page, vaddr, npages); +} + struct subpage_arg { struct page *dst; struct page *src; @@ -5576,34 +5601,29 @@ struct subpage_arg { */ static inline void process_huge_page(struct subpage_arg *sa, unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(struct subpage_arg *sa, - unsigned long base_addr, int idx)) + void (*process_subpages)(struct subpage_arg *sa, + unsigned long base_addr, int lidx, int ridx)) { int i, n, base, l; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); /* Process target subpage last to keep its cache lines hot */ - might_sleep(); n = (addr_hint - addr) / PAGE_SIZE; + if (2 * n <= pages_per_huge_page) { /* If target subpage in first half of huge page */ base = 0; l = n; /* Process subpages at the end of huge page */ - for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { - cond_resched(); - process_subpage(sa, addr, i); - } + process_subpages(sa, addr, 2*n, pages_per_huge_page-1); } else { /* If target subpage in second half of huge page */ base = pages_per_huge_page - 2 * (pages_per_huge_page - n); l = pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ - for (i = 0; i < base; i++) { - cond_resched(); - process_subpage(sa, addr, i); - } + process_subpages(sa, addr, 0, base); } /* * Process remaining subpages in left-right-left-right pattern @@ -5613,15 +5633,13 @@ static inline void process_huge_page(struct subpage_arg *sa, int left_idx = base + i; int right_idx = base + 2 * l - 1 - i; - cond_resched(); - process_subpage(sa, addr, left_idx); - cond_resched(); - process_subpage(sa, addr, right_idx); + process_subpages(sa, addr, left_idx, left_idx); + process_subpages(sa, addr, right_idx, right_idx); } } static void clear_gigantic_page(struct page *page, - unsigned long addr, + unsigned long base_addr, unsigned int pages_per_huge_page) { int i; @@ -5629,18 +5647,35 @@ static void clear_gigantic_page(struct page *page, might_sleep(); for (i = 0; i < pages_per_huge_page; - i++, p = mem_map_next(p, page, i)) { + i += clear_page_unit, p = mem_map_offset(page, i)) { + /* + * clear_page_unit is a factor of 1<dst; + int i, n; - clear_user_highpage(page + idx, base_addr + idx * PAGE_SIZE); + might_sleep(); + + for (i = lidx; i <= ridx; ) { + unsigned int remaining = (unsigned int) ridx - i + 1; + + n = min(clear_page_unit, remaining); + + clear_user_extent(page + i, base_addr + i * PAGE_SIZE, n); + i += n; + + cond_resched(); + } } void clear_huge_page(struct page *page, @@ -5659,7 +5694,7 @@ void clear_huge_page(struct page *page, return; } - process_huge_page(&sa, addr_hint, pages_per_huge_page, clear_subpage); + process_huge_page(&sa, addr_hint, pages_per_huge_page, clear_subpages); } static void copy_user_gigantic_page(struct page *dst, struct page *src, @@ -5681,11 +5716,19 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -static void copy_subpage(struct subpage_arg *copy_arg, - unsigned long base_addr, int idx) +static void copy_subpages(struct subpage_arg *copy_arg, + unsigned long base_addr, int lidx, int ridx) { - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, + int idx; + + might_sleep(); + + for (idx = lidx; idx <= ridx; idx++) { + copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, base_addr + idx * PAGE_SIZE, copy_arg->vma); + + cond_resched(); + } } void copy_user_huge_page(struct page *dst, struct page *src, @@ -5706,7 +5749,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, return; } - process_huge_page(&sa, addr_hint, pages_per_huge_page, copy_subpage); + process_huge_page(&sa, addr_hint, pages_per_huge_page, copy_subpages); } long copy_huge_page_from_user(struct page *dst_page, From patchwork Mon Jun 6 20:37:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76B7CC433EF for ; Mon, 6 Jun 2022 20:37:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEF186B0072; Mon, 6 Jun 2022 16:37:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9E2C6B0073; Mon, 6 Jun 2022 16:37:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EFA26B0074; Mon, 6 Jun 2022 16:37:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8DEB06B0072 for ; Mon, 6 Jun 2022 16:37:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 65E92A43 for ; Mon, 6 Jun 2022 20:37:58 +0000 (UTC) X-FDA: 79548972636.18.CAD33F2 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 897EB40004 for ; Mon, 6 Jun 2022 20:37:43 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256JKhYm027011; Mon, 6 Jun 2022 20:37:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=KPBhCsdqcunFMUR4wQ/KR9WecNlsalNLN40bYtWiujs=; b=cPBNB/88tpFAXXcRDRYKz8WyUNpALq5CLBP3F77M/vk8tIW8RybCzqEgc0uRx7U1ZUhI ZS+3HJc1IeASagIy4CDM9c2Q9uDN4WVW/pWUO4ZjW7AYRgLmagHf+Izs75lKBHpDegg5 TN1AImdzsM3z2IUB/5gx6xB9wXjCT0TMt6QZZo0dwCyy71OEXTn+hUIaqt1gNwsJEI3t wolZUKf0tNj6ydjsd/XD2oaZ2HWCo1nYLT59arzhB12KvnKrFe1ZlVh9uoThk+xg28HG LdFyIBwjBnVvhvwxjr0ZaAUK3ULNRSA1xrp4AeVBBJZxImE5IYauhHcPKkqVP35wiWN6 qw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfydqm37a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:41 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFt6X001285; Mon, 6 Jun 2022 20:37:40 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2169.outbound.protection.outlook.com [104.47.59.169]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu20qv1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LcdekasnLg/gwFnMu3WXDX0SYt6N2AEp9dcietVpJtACkE5PvT0HtAXA2PIxxdD3H1szzJJbKZGjpYImMMogvpvnjjHxTWjOyeloTK56+WYeawBncOxnRZDeoxoJ1GkvdEblVsZrXcMqpezY+cEFpsQ+POkYrqPQ/i3sN8eAYGM7o5qTNi76/XjzUyUSt4hgwCyKyeWNjhvfAPFdntFzDVvT+xOzi+lMgFW0OOn1wfk8hOSe359eQyYGZEB/OgtP4ZRvmShsK6VeLNsBEQb2L4BnXEqYs5AV6DnFbnnMPzkiubFDxByjiK1ILj/V7sLzLp5QBd7troMkoeplPWTcFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KPBhCsdqcunFMUR4wQ/KR9WecNlsalNLN40bYtWiujs=; b=gOi0RMNxJ2x0SQ2SHx08udihDsT6fmKipm5UIrb0Y8nbrwR4b2HQE9Y4SOXTPo+8g3iG+Bto4B1hxqQu5q6jzKw4mro7xsqc0lN7fR1GmTpWjCWjyhvIt1qrKxytDkk8yxLZ2TPMZzXd5QjYwqLYYHXqm3nQ8E1Ddvo2v78iAFSSOR19H8vd/tNWwG0a/FCNL5apCasp6pQmc5FBlvYUCllUWTUmYr+Muyy66aynG4h9nhQsQWdu9z3/35jFesn9dvkw1DTjzAZn8EkL/kvyG2cs8o10uint0M7ZHhEyfL5Wlhwb1DOV4H0xrRQD3dnmKkrAKF1hbR/rYUFaXgbQjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KPBhCsdqcunFMUR4wQ/KR9WecNlsalNLN40bYtWiujs=; b=EubWIW0q+gw05ckGX4dapBJSy8c7qybKFXZjM5r/ihzLyUyoNwc+7F3SzX3wb2DCfWxf3rBL0l5Gc186abhNtXQ7i2sv5626eutqaHt7KV2x+Gt+OyYYexVo0EtMLGbuz77wJ6Vz6OQ8Uv+2M4d7CxMD+xX/PxcDrQXytrM8T/w= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:37:38 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:37:38 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 05/21] mm/huge_page: generalize process_huge_page() Date: Mon, 6 Jun 2022 20:37:09 +0000 Message-Id: <20220606203725.1313715-1-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR04CA0008.namprd04.prod.outlook.com (2603:10b6:a03:40::21) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4e0a22b6-78fb-43f2-5a1c-08da47fc652c X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +UYS+12ADzu1QhLVveKhBt3Tx6AvX1ZqxwZWwQGz6ZdEmPpREPkXx2JdCCriz6nfgyrN7YSEam9pYcd6WraGfUr0IJJ4ylJV3WkEjCJC8WFq4oPyWo+xH19o45GaPhTY1T4xnxn4CGdZusPZuEF8FVzyPnk8nDEw0Z8b7sLX0BAXDuZ0BqB8iQwZElnvZ6Ok1GDkWevGemCYRCT6+Jty3jOwq3kY16KbCXP1eY23gPlz7Fp6IcTnCFENfyUDy68gvsPDD9YL+8+0aKSDsU+Mm8WsuvgJYzBvDur1OrTAAs8+3uTsrtJUpqNTcs4qI+wJCRaXjqEj7RNu9AZaSn46S5lHFJlEPErl8HPMuKw+xyfybD4nQtN1FUA4RD3dwF9i5Lu86J1lDka4auXoSrWYs9rBtyw7Ku6sqcPAPD6dJyTWbpBidOKvKqGBhyuYiZL+DXAxuM65BdMvmy+cIbraiaDq+Y6OUu3F89zlya4Z9sQZYv6mYYFVPDeXNQ9etuuZesuQqGEkbkhl9oshQw70KkKzXF5vcMLrDyHG7Z5je1x1fXssn/LHDal/TCPYr7c/zSZtnVwJdF0GNZ5Vp7QkIfwqS/YOKGv/6Hf5H6GUZ8KnqzT7BgWDHMvgs96ZJoCpN+wTpFNosCrpLO/6FJDOyAtuz5vH7cSYB1Nvm9c7X2I/QIP6ZdNpMxbSShhdX1TMCJE3wNuWWSVcjEKQ+2y7cBmFifMB+6HklA1PoEXQ9kM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iEynmrX3Gfm9ff/LoNbXirNECG+fb2ioDvJH8gAXvob39soiIx0dvV4R8Z0zdtwLJZIHtNKKvq2O44240sppJ4VEkl01XP0BCBzSfHXe4GrhC+8D+6pQEDJAcd3F6u6XXIstvW0+L6JDq7j0tVjaJcKppS3Y7moOmaGm01OJpXRLusneGjwfbzwkJaGluoKGzp9wbRDUh8uZsW7eCJJdX+idWOy/ab4HWRBN6orRAJmwFyFy1P2DICTaRZexTTDhOx6Nm7d8f+VAOWr8qfvlj0YgAW0jVzT5bd4bpptNTdS08lmgN2UAj3tYgnEbo9RdBvQfiyUf/I4ARD/Mq1Eg2XtlLGyP/vnjv2qATaVPKNShNHr6TDNqlktDCr+CSr+OvQKC+yy8oRSwoAY0mAGEzPiOlzea7ItW5khd1HHjQZSAg+6qkPNMWpwKc+VJH6Dz7Ig8AotiAv7G6T59ISZB3bnYgTWqh6tU4zlQ7Z+B61/OZx0OQDsYSxgIdkskhqjFkangv1efy3Y3P2ro0UTK92Uql3fncOY5NptBD/4P2DbOPqKVZdX76PjwGcT55D2o5I+f929w0lk4T7xj1Pw3Km3Ei+tI57JkZ+bn8yP4Hzea+x6PxQOwiuZbhyzVyzT3wo1qu2D2hb32J69kznAQTZ/yc6v6+Ep8oym7++xSMz/Fj2tDO5+860odxN9BHPwQSCgRuTlwnKKbHzN+1/uLDp7S+OiLDQG+eQs5Kh+nDW5cte6IoYqRjhayKnF5DtyNPbQ1BVw2cStzTgGwvkUpK/hDBslZTskO8LCL3dPk04XM3j2gM0qyvkdRYMsUc/dfLjujRNH9Z1WfRQyKphUnAdd8GCRwJst+1gErxJh7s6w3TLHsrKfq9t3yH5EXodGqWglEzyReTxgA6qtDSdimkChfWJtvMqZHTGOSv13/iAmWMIwoEKMfUqwQdv6BD9klVQIky7mhn3ENKvwTEXKELtgR4yBZeoOqHDKeAMvrCzumFHuW3xUTAxkJkz5ZLAROvTcafQEL+sBh0uZJ+aJHdU1uteWMlexY/VuRyCZCm63fJULc7yQ091aagFcOR837IY2N/OCJKD7qrPMkp9VzKG3R0Iy76u4laoPNquWO24Cf/GhPFD3uCzfpLpGxs0cJL1mbP82HX8Fvy4Ut/o9MWVyAVntAHtrqM0up/8Iudyxxp5wXoF5h/jSBgW2edsyGTyJwLbRs2OgXD43c3SP9fv6nAifGdKTyi2rM0zgWArK7p4mkdvPjN7E017jptZxlnsHvCr2txzgqtNplWfLwAJHt9tJlAJK6saTCTxxwPFG2qkXWJ2LL32C3fRH7GEjsOdQZZtFTgEPCpRLFxzzEA855cqKOc4+gjE6rL5yLAtH6oHTvOWaDVbghBN8Ir05UXB+3pNgklBavdVa/7dRnxk5eszuPY3LGzQ3VPZxARFWDjE/u87BUCWefJXiQ5NDW55r3fFe6Z8zF2955Em5/bsFWLA2xg3YVEKiP8BsCq2+ks/KUg5Yf/FNnOj5clCHBcAQj5jK6BNbHfCyHDjFSudtttUw8jg6/lYf+Me2xaBkGS6vEU81LNbwYusylDIhwROdEXkxwynMUKyECOK2VzOXThJbLMTNqJwBzQXDwdUfViA76pCod0uuI3GO0Dg4GlfNc/qHbUn3BrMBuMpuza6mAag9rof5h0yCaPgpXlb5+GtgwdU85bA/PL3vmN/XiOWW/+5OVccuUa6L2gJsYM/RKZ/LQbGJC4CSFeRtXSmw= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e0a22b6-78fb-43f2-5a1c-08da47fc652c X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:37:38.2176 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: szp2R5zo01h1H6SbE25XM03gpOmgUnwUVrztP234Rz16w0euAa5EXQ7GGkOeShRz5eSwn8zfEska6qbIHF7BFIE0xgcWM7nR46zgVmDNGOA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 adultscore=0 spamscore=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: 7TaTm2LlHJTx879NupUrgluiinq3JVqj X-Proofpoint-ORIG-GUID: 7TaTm2LlHJTx879NupUrgluiinq3JVqj X-Stat-Signature: f5wyfytph636zzfomxhmro1muxn1uhxh X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="cPBNB/88"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=EubWIW0q; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf07.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 897EB40004 X-HE-Tag: 1654547863-550946 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: process_huge_page() processes subpages left-right, narrowing towards the direction of the faulting subpage to keep spatially close cachelines hot. This is done, however, page-at-a-time. Retain the left-right narrowing logic while using larger chunks for page regions farther away from the target, and smaller chunks approaching the target. Clearing in large chunks allows for uarch specific optimizations. Do this, however, only for far away subpages because we don't care about keeping those cachelines hot. In addition, while narrowing towards the target, access both the left and right chunks in the forward direction instead of the reverse -- x86 string instructions perform better that way. Signed-off-by: Ankur Arora --- mm/memory.c | 86 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 22 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fbc7bc70dc3d..04c6bb5d75f6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5592,8 +5592,10 @@ struct subpage_arg { struct page *dst; struct page *src; struct vm_area_struct *vma; + int page_unit; }; +#define NWIDTH 4 /* * Process all subpages of the specified huge page with the specified * operation. The target subpage will be processed last to keep its @@ -5604,37 +5606,75 @@ static inline void process_huge_page(struct subpage_arg *sa, void (*process_subpages)(struct subpage_arg *sa, unsigned long base_addr, int lidx, int ridx)) { - int i, n, base, l; + int n, lbound, rbound; + int remaining, unit = sa->page_unit; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + lbound = 0; + rbound = pages_per_huge_page - 1; + remaining = pages_per_huge_page; + /* Process target subpage last to keep its cache lines hot */ n = (addr_hint - addr) / PAGE_SIZE; - if (2 * n <= pages_per_huge_page) { - /* If target subpage in first half of huge page */ - base = 0; - l = n; - /* Process subpages at the end of huge page */ - process_subpages(sa, addr, 2*n, pages_per_huge_page-1); - } else { - /* If target subpage in second half of huge page */ - base = pages_per_huge_page - 2 * (pages_per_huge_page - n); - l = pages_per_huge_page - n; - - /* Process subpages at the begin of huge page */ - process_subpages(sa, addr, 0, base); - } /* - * Process remaining subpages in left-right-left-right pattern - * towards the target subpage + * Process subpages in a left-right-left-right pattern towards the + * faulting subpage to keep spatially close cachelines hot. + * + * If the architecture advertises multi-page clearing/copying, use + * the largest extent available, process it in the forward direction, + * while iteratively narrowing as the target gets closer. + * + * Clearing in large chunks allows for uarch specific optimizations. + * Do this, however, only for far away subpages because we don't + * care about keeping those cachelines hot. + * + * In addition, while narrowing towards the target, access both the + * left and right chunks in the forward direction instead of the + * reverse -- x86 string instructions perform better that way. */ - for (i = 0; i < l; i++) { - int left_idx = base + i; - int right_idx = base + 2 * l - 1 - i; + while (remaining) { + int left_gap = n - lbound; + int right_gap = rbound - n; + int neighbourhood; - process_subpages(sa, addr, left_idx, left_idx); - process_subpages(sa, addr, right_idx, right_idx); + /* + * We want to defer processing of the immediate neighbourhood of + * the target until rest of the huge-page is exhausted. + */ + neighbourhood = NWIDTH * (left_gap > NWIDTH || + right_gap > NWIDTH); + + /* + * Width of the remaining region on the left: n - lbound + 1. + * In addition hold an additional neighbourhood region, which is + * non-zero until the left, right gaps have been cleared. + * + * [ddddd....xxxxN + * ^ | `---- target + * `---|-- lbound + * `------------ left neighbourhood edge + */ + if ((n - lbound + 1) >= unit + neighbourhood) { + process_subpages(sa, addr, lbound, lbound + unit - 1); + lbound += unit; + remaining -= unit; + } + + /* + * Similarly the right: + * Nxxxx....ddd] + */ + if ((rbound - n) >= (unit + neighbourhood)) { + process_subpages(sa, addr, rbound - unit + 1, rbound); + rbound -= unit; + remaining -= unit; + } + + unit = min(sa->page_unit, unit >> 1); + if (unit == 0) + unit = 1; } } @@ -5687,6 +5727,7 @@ void clear_huge_page(struct page *page, .dst = page, .src = NULL, .vma = NULL, + .page_unit = clear_page_unit, }; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { @@ -5741,6 +5782,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, .dst = dst, .src = src, .vma = vma, + .page_unit = 1, }; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { From patchwork Mon Jun 6 20:37:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71382C43334 for ; Mon, 6 Jun 2022 20:38:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 008116B0073; Mon, 6 Jun 2022 16:38:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF8FE6B0074; Mon, 6 Jun 2022 16:38:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D24836B0075; Mon, 6 Jun 2022 16:38:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BE4166B0073 for ; Mon, 6 Jun 2022 16:38:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 85DB820BBD for ; Mon, 6 Jun 2022 20:38:02 +0000 (UTC) X-FDA: 79548972804.02.EB23AB5 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf04.hostedemail.com (Postfix) with ESMTP id 242FE40041 for ; Mon, 6 Jun 2022 20:37:39 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KJjsL030086; Mon, 6 Jun 2022 20:37:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=O1fUpa0v26fvWXiIeQnktX7swG7rLE95g+TWDuxMvA4=; b=f+pGRyCFHhMHM4+MidG2NoM8/6ysbJzyJq3EjVFLirlXTsa7xayogpKjXsIREmMojHZp cxT8VwhcdvbbRxFCrvhsJYF7XJ8fwC7wsNMSZy+5wIrAVCut0TFzY8YtdB5ephm8t8Av laNOZ/zmJ7WFxttu+WjbMwjZDPgqw6LG20A0s59XBrBav+EfY4Vx+9cOnx2pR4+grziG fj1N6WIGn+rtc54dqbtF2TjrXjcWvLr+W2nCZviYJwMY4G7bV8EuxHCv2gN9SOa0Fuat pOg2ovxKyW16NCEJV57u8r1oY7w8E/6anRHVi4DG5wowfOvJ+oHM+pdIi0uPjikXmRV/ EA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad85nc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:44 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFk2K015941; Mon, 6 Jun 2022 20:37:43 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2105.outbound.protection.outlook.com [104.47.70.105]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu809rh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U/6GkZH6mjHh+Mvg7ZtddYtZYi7vAX7Yt69GiTP3a705bwLHg12QBNRnYkDMUpzr8pqDi2/DdphBOX4r9mIQp2D94dftaBz3cqSNWvfXBm7wKaQk3nS5RVyQ7RZphZ8RgDwa4cMfzhcARP+zuDNu/fnVGf+STU2Ptyp63cKYdtnocSoiQOPUEeo3Z14qEwABBnoOwzOO0Hi7zNdrWsKochugN8jNX9Cw0owcWaROfUSaJPOpFv99xgtN+We5Xq9T5j5S0aoD6MvTPeWffnxdEepEi/Wa+Ucd+GV+5hTIMko96J8uklVoBv+rykfdJfcPeU035QDESdRvWN90ls6Oqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O1fUpa0v26fvWXiIeQnktX7swG7rLE95g+TWDuxMvA4=; b=cKiXJIQnCHIJStur5a1i1WkKlwmfIZ4jmoDrsXRPZYiLojcnKg6Vo1I1Mu49SOnj12ON6MxsIzejprd+FjbVOD77G3cvpbK5p7hwjlAncGZ9OsDmK2xfSkdJfQAdRyU51flt2du+4YTsKRLF8rMQBRDuW0D+/CEahpUGWqSatfeOTqKuoyXPKXlU+PBwHdo1oyrNbfWZJIERNzf2okMpG8ECysfWAb9bjsxTNCGat+Bi6p8ISq7QxeaU2wZ4C7Tx5JkNsrOWahYjdht5BXvZrglA9QJv1Ge6Gds3e4lYScO7zl/vYCl4+RqcS/ynzZHUb56g5BatVmy8VhQukxxhwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O1fUpa0v26fvWXiIeQnktX7swG7rLE95g+TWDuxMvA4=; b=hMDkNjQ0FpubwqFY/gtzX4vdsBP3pikWj0j967H1chMmjETnSZ3Csjt10Xo/NknJdaikLqYKX7+X0GhNlrmsevjNqrIdjqHi3OVjDhU7qO+iUe+Bsnp9k1YZbt4UfVEWrJet5xrI3fILDsOrmh4/fkpXnDWQQPMOGakZOSQfhpo= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:37:41 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:37:41 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 06/21] x86/clear_page: add clear_pages() Date: Mon, 6 Jun 2022 20:37:10 +0000 Message-Id: <20220606203725.1313715-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR04CA0008.namprd04.prod.outlook.com (2603:10b6:a03:40::21) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b8649870-19c8-4f89-96ee-08da47fc66de X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bzplH6Yfy9yxUlR5A4rX0NJVSzqDsUrRolH7XwA9QzihvHuYfDKcxrkKeZT6eTDyHBRKa6Zu5aYo9sjS6DFAtr8BdAIt0fU/qIWWGMpemQ+Wveny4E6FEJJGNiUcMvj7Ni5YUTMDPLrwUn+fZ4kXlAK/PpqJcvIUNiI2wWkWbDPSzF7+VQubJCtFwc6o0eEvrV2+vCOQqNZ4uDpCqiyErTM/HcRILy8IzZiEbXTm8furer8cLkan/Rwst+sym9KI2n7Z/Y/YGE2m8esr26I6r66Npn3YkbNrHpHtIGN0EZ2gxyRZseqAe9VnVg8QxvE1CZJCfUvFNiL1oisQI34inoCWD4lGzBWJcow91Di6M6yyWG1okT6cjqJeJIMWUW/uJ0VMT5D0HYIXO9TrlOa5FkbANs3T9Lo/aE32Q29x5VsuCtPE0MIMQxbUa+nXWsJBywp53W9UM64gmBT+/J2IK69aqJXXXDhGoDgNiVH7GSjGccs9VHWEYhOLZnDpjlFQz/LWdaShsWgSkcm2gJHX6t99cDux+cV/rJDnmtAZOW/Akprcn4f7o9i/FS4MFnjJbaAwu6UXZG2nbHHWc88sm9+nScJANPxjJFVfi6ycXds48fxjP2eDccGo7G7AE3zGv5xEjUvi2BwKjroN5L5Gy1p9sB91EwX00IToEG0PLb1TfTdWBFWDVkbezWY5YlIvfKXHl5jnKXrwgFk0Vazx/A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(30864003)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uITih3FbBUkzmBRzKtPbzcHvSX4l4tZzvM5zLLuEhJqx01L1CmVLff1dZhTMVv+Wi7gNPWrmelg+5K9NMW2pkVMl0uX7rOMTz110XyzltK8nz6xptXXgDPX1ZNdim7sLoKKH4xC7FSedJTdBoRDKurrpVn5BfR1PSTxgzVNPBRgvYrVV3XtwV33GSQf14T0VFN61t2pEL29RcU4grRfFx4BW9leURYFtM83YhVHTbPTbvjNdE9ym0p8O1RyP3FLgp0vlGFb8SRrqu5LeUYApGF8rXnYgcHavjkPkwMTrporR2tWuA+xVuLX2ipmwjpn8WT//S1z5zwJofMwZF57Wnj/JD5qx3vb46vKlWKThMTgjGyQfhTP0sr8PD95ZK1LYqk90KebRJHk9Uw/f3e6af65BXtPOcFFZ1T75fk6Ju7vX2LDZHruGZX19UJzOLrqL17O1D5u32YvM37IUSjtoNcqzHQ3n203iWtToqjliUjqwHquqP7xAp7ZoxGD4b4YQRLz/OsPi6r7agG0uIMWgivrNaMHqdzdNIRiQavbXvnxoikDVvGe5yGSdX0PSRc9hqIPQo8LxodkbysVWUBz5i7cWtCN4B6Tn4jvj5/mr1AiG2JgavrjANfem8oceiCmEuARxGy7C8NDgMhFyh44VLDN74vfFxtl4AA7tQYoshtUcGvoHotf/8beCSwDZzLzpW8P8xudLo3CHfrZTOgvsvbAGTDRT3tpy3igJr+9rGduid3VJQA5H3vPDUh3Zqb5f9084Ht3DO7+9eBhZgyo7uzI6jmZsdWBIKaC1jaIiHR+Aq+cqZlBSlxDPxAjkt1PPnr+Ey2mUuPKFpeE7Dfm28u1oU4O4F1az0qrXFEuWGrjTokdTLq5rEcZg1uC9bBlIPfqodx7sDLysTWeYT0zBGYRwHk0tN3thQVOL95Mm/deFBRUtix1en+oBoAP17qLOZVmoGu55sD5B+yW+TAKACFOeEj2dGrsvlEtLjE+RDhIIpV2iRNI334BNiaXU/6AhYBTwn80pQ+15PtEphRRQqhU1aLLo0L/U81khf5AnxEp/bZuHRJuHqIJZrn3qauzJYO20RxVsHsmiGMhRsIoPl+CmCTE2k+kSV5lERV0OD1ZeesxCDuLeLFvY9Ly1HGQWAm1FABa0iJRzNRnrZloxetYvFto8fPxkoCqrcCLbANCMda4+cs3m80PSbkrWbbfuiXHYdGTOOyNTMfygOF/YDoXCxrTQpUqTelabsxd2wv6PHIuSYGoO7t0cjR5O99X4iWMefsA1wa+LqhSzC9YZsbZ3i6sRNSzA77qm60H9CW6LrALXbOEK9R/XGZPzLs/Ntj4lPXK+pggzIBaR/G7dYgfDXhNxRBM7TDf5g4N9zcS5FY4lWp2ND8VW6ugjRZx+95MAwl8etPx8oY341ivF4yPSq0TGbfqB+jAfR1d07VHxGWNow5dsEMIWaYRGh/UUYflw9q1U5UKuyiz7fH0KOckRnhZeIQVn6vVh21et18iCkTOkrNU3F0cKIv0+BZ/TmJNgHTXr1O5bI35Lle74Dkf11yzT+WhMt+lN0/JQSX3FhjvvW8nxa4SBlDUPFxkrMvy1gEl2HUm5/DTaYRTpFiShqtVHeDZ7RibKDOVh/KTzEvQawhpJXUOuXLnPkH8nqqO6BDi4feKALh+ayVIBORzbOMXhHraLplx0BgzOl8PJvBdxiBSGio338eVx68GlrY4WFr9oAmKU7vMgVQQ1JoFdyxpfjEdJZAVSbYbjZCE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b8649870-19c8-4f89-96ee-08da47fc66de X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:37:41.0946 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s0zXKWVMmhCv3+9YzMl8/5jR5msHvKMWsOop2Na5gW359AlkDYX7mvnsKx7Mwea252TNsV0QzO06M9eqauzBuTzOiDGneqp6SkWz7/PS86s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: SP5jMbWF8uYU6N0qBfVUDpoHi2DzUqi0 X-Proofpoint-GUID: SP5jMbWF8uYU6N0qBfVUDpoHi2DzUqi0 X-Stat-Signature: mriojnn9hmdxfmgqd8rpz67157dwn1id Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=f+pGRyCF; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hMDkNjQ0; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf04.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 242FE40041 X-HE-Tag: 1654547859-25935 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add clear_pages(), with ARCH_MAX_CLEAR_PAGES_ORDER=8, so we can clear in chunks of upto 1024KB. The case for doing this is to expose huge or gigantic page clearing as a few long strings of zeroes instead of many PAGE_SIZE'd operations. Processors could take advantage of this hint by foregoing cacheline allocation. Unfortunately current generation CPUs generally do not do this optimization: among CPUs tested, Intel Skylake, Icelakex don't at all; AMD Milan does for extents > ~LLC-size. (Note, however, numbers below do show a ~25% increase in clearing BW -- just that they aren't due to foregoing cacheline allocation.) One hope for this change is that it might provide enough of a hint that future uarchs could optimize for. A minor negative with this change is that calls to clear_page() (which now calls clear_pages()) clobber an additional register. Performance === System: Oracle X9-2c (2 nodes * 32 cores * 2 threads) Processor: Intel Xeon(R) Platinum 8358 CPU @ 2.60GHz (Icelakex, 6:106:6) Memory: 1024 GB evenly split between nodes LLC-size: 48MB for each node (32-cores * 2-threads) no_turbo: 1, Microcode: 0xd0002c1, scaling-governor: performance System: Oracle E4-2c (2 nodes * 8 CCXs * 8 cores * 2 threads) Processor: AMD EPYC 7J13 64-Core Processor (Milan, 25:1:1) Memory: 512 GB evenly split between nodes LLC-size: 32MB for each CCX (8-cores * 2-threads) boost: 1, Microcode: 0xa00115d, scaling-governor: performance Workload: create a 192GB qemu-VM (backed by preallocated 2MB pages on the local node) == Icelakex --- Time (s) Delta (%) clear_page_erms() 22.37 ( +- 0.14s ) # 9.21 bytes/ns clear_pages_erms() 16.49 ( +- 0.06s ) -26.28% # 12.50 bytes/ns Looking at the perf stats [1] [2], it's not obvious where the improvement is coming from. For clear_pages_erms(), we do execute fewer instructions and branches (multiple pages per call to clear_pages_erms(), and fewer cond_resched() calls) but since this code isn't frontend bound (though there is a marginal improvement in topdown-fe-bound), not clear if that's the cause for the ~25% improvement. The topdown-be-bound numbers are significantly better but they are in a similar proportion to the total slots in both cases. Milan -- Time (s) Delta (%) clear_page_erms() 16.49 ( +- 0.06s ) # 12.50 bytes/ns clear_pages_erms() 11.82 ( +- 0.06s ) -28.32% # 17.44 bytes/ns Similar to the Icelakex case above, from the perf stats [3], [4] it's unclear where the improvement is coming from. We do somewhat better for L1-dcache-loads and marginally better for stalled-cycles-backend but nothing obvious stands out. Workload: vm-scalability hugetlb tests (on Icelakex) == For case-anon-w-seq-hugetlb, there is a ~19.49% improvement in cpu-cycles expended. As above, from perf stats there isn't a clear reason why. No significant differences in user/kernel cache misses. case-anon-w-seq-hugetlb: - 2,632,688,342,385 cpu-cycles # 2.301 GHz ( +- 6.76% ) (33.29%) + 2,119,058,504,338 cpu-cycles # 1.654 GHz ( +- 4.63% ) (33.37%) Other hugetlb tests are flat. case-anon-w-rand-hugetlb: - 14,423,774,217,911 cpu-cycles # 2.452 GHz ( +- 0.55% ) (33.30%) + 14,009,785,056,082 cpu-cycles # 2.428 GHz ( +- 3.11% ) (33.32%) case-anon-cow-seq-hugetlb: - 2,689,994,027,601 cpu-cycles # 2.220 GHz ( +- 1.91% ) (33.27%) + 2,735,414,889,894 cpu-cycles # 2.262 GHz ( +- 1.82% ) (27.73%) case-anon-cow-rand-hugetlb: - 16,130,147,328,192 cpu-cycles # 2.482 GHz ( +- 1.07% ) (33.30%) + 15,815,163,909,204 cpu-cycles # 2.432 GHz ( +- 0.64% ) (33.32%) cache-references, cache-misses are within margin of error across all the tests. [1] Icelakex, create 192GB qemu-VM, clear_page_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 22,378.31 msec task-clock # 1.000 CPUs utilized ( +- 0.67% ) 153 context-switches # 6.844 /sec ( +- 0.57% ) 8 cpu-migrations # 0.358 /sec ( +- 16.49% ) 116 page-faults # 5.189 /sec ( +- 0.17% ) 57,290,131,280 cycles # 2.563 GHz ( +- 0.66% ) (38.46%) 3,077,416,348 instructions # 0.05 insn per cycle ( +- 0.30% ) (46.14%) 631,473,780 branches # 28.246 M/sec ( +- 0.18% ) (53.83%) 1,167,792 branch-misses # 0.19% of all branches ( +- 0.79% ) (61.52%) 286,600,215,705 slots # 12.820 G/sec ( +- 0.66% ) (69.20%) 11,435,999,662 topdown-retiring # 3.9% retiring ( +- 1.56% ) (69.20%) 19,428,489,213 topdown-bad-spec # 6.2% bad speculation ( +- 3.23% ) (69.20%) 3,504,763,769 topdown-fe-bound # 1.2% frontend bound ( +- 0.67% ) (69.20%) 258,517,960,428 topdown-be-bound # 88.7% backend bound ( +- 0.58% ) (69.20%) 749,211,322 L1-dcache-loads # 33.513 M/sec ( +- 0.13% ) (69.18%) 3,244,380,956 L1-dcache-load-misses # 433.32% of all L1-dcache accesses ( +- 0.00% ) (69.20%) 11,441,841 LLC-loads # 511.805 K/sec ( +- 0.30% ) (69.23%) 839,878 LLC-load-misses # 7.32% of all LL-cache accesses ( +- 1.28% ) (69.24%) L1-icache-loads 23,091,397 L1-icache-load-misses ( +- 0.72% ) (30.82%) 772,619,434 dTLB-loads # 34.560 M/sec ( +- 0.31% ) (30.82%) 49,750 dTLB-load-misses # 0.01% of all dTLB cache accesses ( +- 3.21% ) (30.80%) iTLB-loads 503,570 iTLB-load-misses ( +- 0.44% ) (30.78%) L1-dcache-prefetches L1-dcache-prefetch-misses 22.374 +- 0.149 seconds time elapsed ( +- 0.66% ) [2] Icelakex, create 192GB qemu-VM, clear_pages_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 16,329.41 msec task-clock # 0.990 CPUs utilized ( +- 0.42% ) 143 context-switches # 8.681 /sec ( +- 0.93% ) 1 cpu-migrations # 0.061 /sec ( +- 63.25% ) 118 page-faults # 7.164 /sec ( +- 0.27% ) 41,735,523,673 cycles # 2.534 GHz ( +- 0.42% ) (38.46%) 1,454,116,543 instructions # 0.03 insn per cycle ( +- 0.49% ) (46.16%) 266,749,920 branches # 16.194 M/sec ( +- 0.41% ) (53.86%) 928,726 branch-misses # 0.35% of all branches ( +- 0.38% ) (61.54%) 208,805,754,709 slots # 12.676 G/sec ( +- 0.41% ) (69.23%) 5,355,889,366 topdown-retiring # 2.5% retiring ( +- 0.50% ) (69.23%) 12,720,749,784 topdown-bad-spec # 6.1% bad speculation ( +- 1.38% ) (69.23%) 998,710,552 topdown-fe-bound # 0.5% frontend bound ( +- 0.85% ) (69.23%) 192,653,197,875 topdown-be-bound # 90.9% backend bound ( +- 0.38% ) (69.23%) 407,619,058 L1-dcache-loads # 24.746 M/sec ( +- 0.17% ) (69.20%) 3,245,399,461 L1-dcache-load-misses # 801.49% of all L1-dcache accesses ( +- 0.01% ) (69.22%) 10,805,747 LLC-loads # 656.009 K/sec ( +- 0.37% ) (69.25%) 804,475 LLC-load-misses # 7.44% of all LL-cache accesses ( +- 2.73% ) (69.26%) L1-icache-loads 18,134,527 L1-icache-load-misses ( +- 1.24% ) (30.80%) 435,474,462 dTLB-loads # 26.437 M/sec ( +- 0.28% ) (30.80%) 41,187 dTLB-load-misses # 0.01% of all dTLB cache accesses ( +- 4.06% ) (30.79%) iTLB-loads 440,135 iTLB-load-misses ( +- 1.07% ) (30.78%) L1-dcache-prefetches L1-dcache-prefetch-misses 16.4906 +- 0.0676 seconds time elapsed ( +- 0.41% ) [3] Milan, create 192GB qemu-VM, clear_page_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 16,321.98 msec task-clock # 0.989 CPUs utilized ( +- 0.42% ) 104 context-switches # 6.312 /sec ( +- 0.47% ) 0 cpu-migrations # 0.000 /sec 109 page-faults # 6.616 /sec ( +- 0.41% ) 39,430,057,963 cycles # 2.393 GHz ( +- 0.42% ) (33.33%) 252,874,009 stalled-cycles-frontend # 0.64% frontend cycles idle ( +- 17.81% ) (33.34%) 7,240,041 stalled-cycles-backend # 0.02% backend cycles idle ( +-245.73% ) (33.34%) 3,031,754,124 instructions # 0.08 insn per cycle # 0.11 stalled cycles per insn ( +- 0.41% ) (33.35%) 711,675,976 branches # 43.197 M/sec ( +- 0.15% ) (33.34%) 52,470,018 branch-misses # 7.38% of all branches ( +- 0.21% ) (33.36%) 7,744,057,748 L1-dcache-loads # 470.041 M/sec ( +- 0.05% ) (33.36%) 3,241,880,079 L1-dcache-load-misses # 41.92% of all L1-dcache accesses ( +- 0.01% ) (33.35%) LLC-loads LLC-load-misses 155,312,115 L1-icache-loads # 9.427 M/sec ( +- 0.23% ) (33.34%) 1,573,793 L1-icache-load-misses # 1.01% of all L1-icache accesses ( +- 3.74% ) (33.36%) 3,521,392 dTLB-loads # 213.738 K/sec ( +- 4.97% ) (33.35%) 346,337 dTLB-load-misses # 9.31% of all dTLB cache accesses ( +- 5.54% ) (33.35%) 725 iTLB-loads # 44.005 /sec ( +- 8.75% ) (33.34%) 115,723 iTLB-load-misses # 19261.48% of all iTLB cache accesses ( +- 1.20% ) (33.34%) 139,229,403 L1-dcache-prefetches # 8.451 M/sec ( +- 10.97% ) (33.34%) L1-dcache-prefetch-misses 16.4962 +- 0.0665 seconds time elapsed ( +- 0.40% ) [4] Milan, create 192GB qemu-VM, clear_pages_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 11,676.79 msec task-clock # 0.987 CPUs utilized ( +- 0.68% ) 96 context-switches # 8.131 /sec ( +- 0.78% ) 2 cpu-migrations # 0.169 /sec ( +- 18.71% ) 106 page-faults # 8.978 /sec ( +- 0.23% ) 28,161,726,414 cycles # 2.385 GHz ( +- 0.69% ) (33.33%) 141,032,827 stalled-cycles-frontend # 0.50% frontend cycles idle ( +- 52.44% ) (33.35%) 796,792,139 stalled-cycles-backend # 2.80% backend cycles idle ( +- 23.73% ) (33.35%) 1,140,172,646 instructions # 0.04 insn per cycle # 0.50 stalled cycles per insn ( +- 0.89% ) (33.35%) 219,864,061 branches # 18.622 M/sec ( +- 1.06% ) (33.36%) 1,407,446 branch-misses # 0.63% of all branches ( +- 10.66% ) (33.40%) 6,882,968,897 L1-dcache-loads # 582.960 M/sec ( +- 0.03% ) (33.38%) 3,267,546,914 L1-dcache-load-misses # 47.45% of all L1-dcache accesses ( +- 0.02% ) (33.37%) LLC-loads LLC-load-misses 146,901,513 L1-icache-loads # 12.442 M/sec ( +- 0.78% ) (33.36%) 1,462,155 L1-icache-load-misses # 0.99% of all L1-icache accesses ( +- 0.83% ) (33.34%) 2,055,805 dTLB-loads # 174.118 K/sec ( +- 22.56% ) (33.33%) 136,260 dTLB-load-misses # 4.69% of all dTLB cache accesses ( +- 23.13% ) (33.35%) 941 iTLB-loads # 79.699 /sec ( +- 5.54% ) (33.35%) 115,444 iTLB-load-misses # 14051.12% of all iTLB cache accesses ( +- 21.17% ) (33.34%) 95,438,373 L1-dcache-prefetches # 8.083 M/sec ( +- 19.99% ) (33.34%) L1-dcache-prefetch-misses 11.8296 +- 0.0805 seconds time elapsed ( +- 0.68% ) Signed-off-by: Ankur Arora --- arch/x86/include/asm/page.h | 12 +++++++++++ arch/x86/include/asm/page_64.h | 28 ++++++++++++++++++------- arch/x86/lib/clear_page_64.S | 38 ++++++++++++++++++++-------------- 3 files changed, 55 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 5a246a2a66aa..045eaab08f43 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -22,6 +22,18 @@ struct page; extern struct range pfn_mapped[]; extern int nr_pfn_mapped; +#ifdef __HAVE_ARCH_CLEAR_USER_PAGES /* x86_64 */ + +#define clear_page(page) clear_pages(page, 1) + +static inline void clear_user_pages(void *page, unsigned long vaddr, + struct page *pg, unsigned int npages) +{ + clear_pages(page, npages); +} + +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES */ + static inline void clear_user_page(void *page, unsigned long vaddr, struct page *pg) { diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index baa70451b8df..a88a3508888a 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -41,16 +41,28 @@ extern unsigned long __phys_addr_symbol(unsigned long); #define pfn_valid(pfn) ((pfn) < max_pfn) #endif -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +/* + * Clear in chunks of 256 pages/1024KB. + * + * Assuming a clearing BW of 3b/cyc (recent generation processors have + * more), this amounts to around 400K cycles for each chunk. + * + * With a cpufreq of ~2.5GHz, this amounts to ~160us for each chunk + * (which would also be the interval between calls to cond_resched().) + */ +#define ARCH_MAX_CLEAR_PAGES_ORDER 8 -static inline void clear_page(void *page) +void clear_pages_orig(void *page, unsigned long npages); +void clear_pages_rep(void *page, unsigned long npages); +void clear_pages_erms(void *page, unsigned long npages); + +#define __HAVE_ARCH_CLEAR_USER_PAGES +static inline void clear_pages(void *page, unsigned int npages) { - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, - "=D" (page), + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, + "=D" (page), "S" ((unsigned long) npages), "0" (page) : "cc", "memory", "rax", "rcx"); } diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index fe59b8ac4fcc..2cc3b681734a 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0-only */ #include #include +#include /* * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is @@ -10,23 +11,29 @@ */ /* - * Zero a page. - * %rdi - page + * Zero pages. + * %rdi - base page + * %rsi - number of pages + * + * Note: clear_pages_*() have differing alignments restrictions + * but callers are always expected to page align. */ -SYM_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_FUNC_START(clear_pages_rep) + movq %rsi,%rcx + shlq $(PAGE_SHIFT - 3),%rcx xorl %eax,%eax rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) -SYM_FUNC_START(clear_page_orig) +SYM_FUNC_START(clear_pages_orig) xorl %eax,%eax - movl $4096/64,%ecx + movq %rsi,%rcx + shlq $(PAGE_SHIFT - 6),%rcx .p2align 4 .Lloop: - decl %ecx + decq %rcx #define PUT(x) movq %rax,x*8(%rdi) movq %rax,(%rdi) PUT(1) @@ -40,13 +47,14 @@ SYM_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) -SYM_FUNC_START(clear_page_erms) - movl $4096,%ecx +SYM_FUNC_START(clear_pages_erms) + movq %rsi,%rcx + shlq $PAGE_SHIFT, %rcx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) From patchwork Mon Jun 6 20:37:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57DB3C433EF for ; Mon, 6 Jun 2022 20:38:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD5558D0001; Mon, 6 Jun 2022 16:38:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2B3B6B0075; Mon, 6 Jun 2022 16:38:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DF8D8D0001; Mon, 6 Jun 2022 16:38:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7CF3A6B0074 for ; Mon, 6 Jun 2022 16:38:03 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 59B5C60BA3 for ; Mon, 6 Jun 2022 20:38:03 +0000 (UTC) X-FDA: 79548972846.30.2D5EE14 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 8A44B40017 for ; Mon, 6 Jun 2022 20:37:48 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256HqxqK028290; Mon, 6 Jun 2022 20:37:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=RzvU0nEnKpie6gkEUitVhL+sux82w413s8CR0J7uhK0=; b=TL+FwDIjew6HgPmAHPVzx8CgcXSMA1Ni5LWfdhBuCyNfaobxTFMoxvtThPk4V7Fj0Y2z PJ4d62cJHvIT5cxIjQ1YYqT9eR2KXgcYwSWkdphs6lv0hz1RPs4lVqxVxU2NC4hdfB/J YY9yhTAsmqUiXuw3OsVdfB+Veq1EzN0PvkWoXLSUEqnH7ffbBNokIoaW9ubMpjRJOBdK i8ZwPNg3IJvXbkDGyWxOmPmPjm35s8AlPTwi+EAkaLD/eKgoWORyaHTR6uB8YMR16ZMk vWHPWvH8qsyEL4yQMU3lUVDEQu9zExN+AaBWZ648e4rmRXsqropcGvOvMnhccgfQ+FI4 Yg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfyxsc6f2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:47 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGPjv037954; Mon, 6 Jun 2022 20:37:46 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2177.outbound.protection.outlook.com [104.47.59.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1upy2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:37:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Levzntp8Qd+GBPAoY1azEitCoQRZV2uw8++30HjvoKvzAkr1KBP2iRe3RTEZtMyMu+nxsZaR8oTTxgBtW4j4BtucXA0FE+T8GS5+AhVDVgTpuL6PtFaywkBbKjfb5BS99WNfdHrqDqo80AEsQtbcX/+c5JHurEyjgF0XP2YQEvxHcVzQFAdcT1LzuN4tsd/A1pqiZq1LXCDzQ5/JlwlOnjnMpOVKrAkhRGFJ3wtTkUiXRLTyQko4L5xYM8C63Ueq2ayazzJg8Dd5SonSLJLUp/KLQtD2ISrCq+0s/ncfYKN5bKqdYwNrWVvpMavGfxqPncC1cldN1Wega1c8x+hxTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RzvU0nEnKpie6gkEUitVhL+sux82w413s8CR0J7uhK0=; b=GDNu+no9m9UhX29pgzm26CemU+Jy1fLGPKEuXao3GW7RtuXtLIEboEEqoNarf0jyAtI8OdPyB8ixidQjw6xl5B6ze7i/aDD3eKPjhDN54wXREhJ6KwCDhDrHyTssty425ZEzHO4jZ1hBxp+tKUaxxMjRAx/gmP+YPdX8YpagW3sbgD3jSLps40TItjmeqMUYS8F8j5vmqWf5LyMLiCE75avctPX8lHEAIJ5RRnzkebeWrTLuwJ6Ac0zSzFeFAoqA5nSdI56zKXfJWc8fNFeTkmRW/zpjyXqJHNnI28okC10WpGjLV6KEhxugQ7cX4K6TasXHmuBGd9YTbPSea5EvPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RzvU0nEnKpie6gkEUitVhL+sux82w413s8CR0J7uhK0=; b=pkIU7uUMdqyvaiUlt3BD9ognBTE0KmdHPL8t6SprrOaE4hr+5VhfEUv0YTLStiz/rtyJUtMNjVBBvrfqOdddTI3RAjXRqO6Z/yk1OWsgZeY73/zA9Cb+RsZdwYj+rH5oBpwoAWsaPul2iKkLS2WCKBcnee5S4dxtGmLSC9GXE7c= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:37:44 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:37:44 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 07/21] x86/asm: add memset_movnti() Date: Mon, 6 Jun 2022 20:37:11 +0000 Message-Id: <20220606203725.1313715-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR04CA0008.namprd04.prod.outlook.com (2603:10b6:a03:40::21) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 49827a37-a8eb-4a11-7508-08da47fc6895 X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1CM5+389ZXDsRcSLh12K8bvukDD+a7Z4gikJHyUtZI9LGhVC/bmtqXQA5cK2pS27ajHyrxb5mUmfdQ66VJdYU/bxNpEWXMwhJqCAIVYolqYxKuaokTgu37ihPF/VHGH2RFoCDuMYR4Ff5steC6x0E2XwgUCSUUSwnByouM1FWFzpct95SZLPIsz6cLfiPRMCDFKe7MyAr0c5iH0X+eJkrucNjbMPtmeUNe7lM/GZIruveXGKtqdIJwJEsQO4RHZdPSz32bMme8ITwZZ3xHgwVM3OQlnSqteioi79CJT9Dp6IekCf+5REfKMryKe9CUI1RwfUPat8BwbHxbdXUNTYH0xdShqA1Padqpol67c3fqJt2IZ6PyGD4k6S17cne1hq3RYN8OB3GtHKaxS/n+uXPXXmmh/M/P0Cg6dP2D7eChnLOa96KLy/+Ukdn26X48Wy7R/0j79r2SDaiUDvCoRy7IIOXSWiVH3MQE+rJ/dw5rovMJzeMM/5JB1e/8jorTmGclXbVjm8q/FaopAM3risU2uSAB7bPq2pqhPmY+6G+QGDNhQ8/3vXPHa9Qy0EzMZ21bCQuCVzFW+1kFs2PUEmnppBE97Nbid7U4+ZZPV5ubl0w/+uhH1ycUd8QZNyiaVgZd+ZSqyY7HIfb0zrqjNQFs3X0PZE/nkO+tYU+5UYsONtbBRRwdRz3lzM+xVSL8jlZsJEyIZslr4OqwkWjcx9JA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Gz2kUE45r+zp3R/qYBjUfVE9yWALT1+phKSOq8XLMHCf5bcwbSwBTrCl8346z5Cop3sALdh8S/sB1Hs2oiDjP4FpnZTPbqA3zKvV8EMhEjTJuXM8GWPd7WrwKZXI1HDJoFmCK44WmyY+gJ8oqjq71KWbEwXt7PqNO9D1XaZ0Uuo6VtgB88Dw73P/K6+fKBvPG+PQZ3drAukZOGu1DmMCODnHV0kpwfsv1Pjyk+48xTsK7nhuYp4RlIgEZKkC2AFDJHh0OyGKJQ1KyBeFxBhDZdbpiQ4LVP0PP3qADgu9NhOS6UURIiDYANVQKm07H+qGV6om/nbND+ZVybTfDdG+W42GaFSOXLLUHsJS2Rdx7PoOrlMV9mJs44qIABf7igiBPmj+XIdRFa4NSbbSFxRKowBG1/zj5RMLiAh9qQSq22ZT8ycDvjwUzUNmWDpp9tFbcixkNERsNXbdIirBk+QjaVuBN91Q2EPMMqjcPhNbQp7n24J24hsk2Zb+/CnyTbsPdK23NfLtcV+X54uXfKOpyxa9jZ/jPxLUZT8E97bSs2soDXBBCEgdnefxLU/fZ2ZDtB027FckCnofHQXLnnJ5V9a+Z0FgsQAdfQP5iIXjZVdfvWH5YU0+c5NqcHDTLG+1opfkaWliGRPIwF3G45QXhP5WsllxGGVlzTzGWzhxqa4sEVEkzTmy8E1O5V2xXAkjwHZT/5yBEWgdetSlKvjgMKBXmp2806p7CJh1kFOnGdoQSs34/E0n0Ir7tC7xEW/0QyoAE/1/Oe4wpPjD5D2CtgxAsPWFZFtoENlMh7ulk/XNkFe26nWIbslMbABo/hY56h0rNlr1etjkCKfWYSKFcT4wAYLHdHmBzhqn90y5vkMo/5bpq6dTBRovvnm/jxh9R2C+cQuE90fkrTbf3Ju21z597bHsAagiLfHTvVH19Ye6kGMHsHOhsZctoULP5kMyymX9FMICHXZj2ed9LaOGuYjVTO8dnMgw01tF+wzB339Dp9o6Dn8+kuORvDWAfS9s8B0xGII6o3Ba+rKyM6paaTxeytJa6EzvnaBGLVZHYYVn9sDd24L9q1U0413lZnUPIdShA1pi5W2UuBB7j4N5ZENb8B7UInKk4Gxl17z6iN3SsZ3s1Y9Z7LwwiriAvpeOG4ADHKicKCisrQWZyuoeMRzyhmjRlWG9/0ulBiiGMeGQwR2xxDTvV3z9munjMkqHYzd+ravgPmyVQT0wwomx2K1+rFTE0jBCAzZh0ePg/sE+tNTBUoBmKrFUt/WXFnBI5PawXmCEcWXHymomEAYeYVUSXkDVCAphCTxcxv/fnYuV03rHOTL6V0Wf8h2KzaP3qTXbKJJ4ZHyuKt2yy+r4lqWNduuX36naFWXWrlgK5qplGWuvl4eu/vtssqhmu081WUWJegkDWpU2N5vXpp5QR3hGAuhH7m6IPAgJnhpn1t4j2P/zhIRSeDtsjwHeZylPifMWGmJbK/1VLbtWsHrXXsvolknsyEh0RXrDTqNj8JQwv7850qlL8xRIGzyiyephnak2IQkGLSVS9d+hzT5d2/SzR4+UKhWJ16Z9LcozJSZ6DDBIYVk1oAK58TBBopmXtrmhJl9FEgD5itRfcJ72gLKKh1XRLcGWLcMOSwtPUyUnM3BCeZQVWwRQW2PiDAcJNnis8mKMHAjmAcAAoukMnulvyx7vq425u6J0DicDgR1IcDhVyyfhd+TlqCzYzn6ZpTHIYbqqdxeNNetOszvxSJ5DHWppb/K93AtXQW/jDlQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49827a37-a8eb-4a11-7508-08da47fc6895 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:37:43.9864 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0uA6Z3qwHbRCum6xEv2e8SqJ3t7vgIBuCCheXqGf38C40c0QPsluwm3FJLD8T9ZvW0r5PyMJBdUONPt+oZPH7oeSJbr/0MC5xPEYG+7z5rc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: cX51BN8OkZp2Ovz9-xFCzkdwZQLc8vYE X-Proofpoint-GUID: cX51BN8OkZp2Ovz9-xFCzkdwZQLc8vYE X-Stat-Signature: 9w98pnsdotj1chwcrsd1iwqz6qxuo8nq X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=TL+FwDIj; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=pkIU7uUM; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf07.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8A44B40017 X-HE-Tag: 1654547868-155799 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a MOVNTI based non-caching implementation of memset(). memset_movnti() only needs to differ from memset_orig() in the opcode used in the inner loop, so move the memset_orig() logic into a macro, and use that to generate memset_orig() (now memset_movq()) and memset_movnti(). Signed-off-by: Ankur Arora --- arch/x86/lib/memset_64.S | 68 ++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/arch/x86/lib/memset_64.S b/arch/x86/lib/memset_64.S index fc9ffd3ff3b2..307b753ca03a 100644 --- a/arch/x86/lib/memset_64.S +++ b/arch/x86/lib/memset_64.S @@ -24,7 +24,7 @@ SYM_FUNC_START(__memset) * * Otherwise, use original memset function. */ - ALTERNATIVE_2 "jmp memset_orig", "", X86_FEATURE_REP_GOOD, \ + ALTERNATIVE_2 "jmp memset_movq", "", X86_FEATURE_REP_GOOD, \ "jmp memset_erms", X86_FEATURE_ERMS movq %rdi,%r9 @@ -66,7 +66,8 @@ SYM_FUNC_START_LOCAL(memset_erms) RET SYM_FUNC_END(memset_erms) -SYM_FUNC_START_LOCAL(memset_orig) +.macro MEMSET_MOV OP fence +SYM_FUNC_START_LOCAL(memset_\OP) movq %rdi,%r10 /* expand byte value */ @@ -77,64 +78,71 @@ SYM_FUNC_START_LOCAL(memset_orig) /* align dst */ movl %edi,%r9d andl $7,%r9d - jnz .Lbad_alignment -.Lafter_bad_alignment: + jnz .Lbad_alignment_\@ +.Lafter_bad_alignment_\@: movq %rdx,%rcx shrq $6,%rcx - jz .Lhandle_tail + jz .Lhandle_tail_\@ .p2align 4 -.Lloop_64: +.Lloop_64_\@: decq %rcx - movq %rax,(%rdi) - movq %rax,8(%rdi) - movq %rax,16(%rdi) - movq %rax,24(%rdi) - movq %rax,32(%rdi) - movq %rax,40(%rdi) - movq %rax,48(%rdi) - movq %rax,56(%rdi) + \OP %rax,(%rdi) + \OP %rax,8(%rdi) + \OP %rax,16(%rdi) + \OP %rax,24(%rdi) + \OP %rax,32(%rdi) + \OP %rax,40(%rdi) + \OP %rax,48(%rdi) + \OP %rax,56(%rdi) leaq 64(%rdi),%rdi - jnz .Lloop_64 + jnz .Lloop_64_\@ /* Handle tail in loops. The loops should be faster than hard to predict jump tables. */ .p2align 4 -.Lhandle_tail: +.Lhandle_tail_\@: movl %edx,%ecx andl $63&(~7),%ecx - jz .Lhandle_7 + jz .Lhandle_7_\@ shrl $3,%ecx .p2align 4 -.Lloop_8: +.Lloop_8_\@: decl %ecx - movq %rax,(%rdi) + \OP %rax,(%rdi) leaq 8(%rdi),%rdi - jnz .Lloop_8 + jnz .Lloop_8_\@ -.Lhandle_7: +.Lhandle_7_\@: andl $7,%edx - jz .Lende + jz .Lende_\@ .p2align 4 -.Lloop_1: +.Lloop_1_\@: decl %edx movb %al,(%rdi) leaq 1(%rdi),%rdi - jnz .Lloop_1 + jnz .Lloop_1_\@ -.Lende: +.Lende_\@: + .if \fence + sfence + .endif movq %r10,%rax RET -.Lbad_alignment: +.Lbad_alignment_\@: cmpq $7,%rdx - jbe .Lhandle_7 + jbe .Lhandle_7_\@ movq %rax,(%rdi) /* unaligned store */ movq $8,%r8 subq %r9,%r8 addq %r8,%rdi subq %r8,%rdx - jmp .Lafter_bad_alignment -.Lfinal: -SYM_FUNC_END(memset_orig) + jmp .Lafter_bad_alignment_\@ +.Lfinal_\@: +SYM_FUNC_END(memset_\OP) +.endm + +MEMSET_MOV OP=movq fence=0 +MEMSET_MOV OP=movnti fence=1 From patchwork Mon Jun 6 20:37:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AC41C43334 for ; Mon, 6 Jun 2022 20:38:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B5756B0074; Mon, 6 Jun 2022 16:38:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 165918D0002; Mon, 6 Jun 2022 16:38:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED25E6B0078; Mon, 6 Jun 2022 16:38:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DC7986B0074 for ; Mon, 6 Jun 2022 16:38:19 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B353760CA5 for ; Mon, 6 Jun 2022 20:38:19 +0000 (UTC) X-FDA: 79548973518.12.11E2463 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf30.hostedemail.com (Postfix) with ESMTP id BE4A480032 for ; Mon, 6 Jun 2022 20:37:39 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KJjkU030080; Mon, 6 Jun 2022 20:38:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=xs1FcMyh1+cEu2F4KtfsVd2nLcKLmWoODKCscmnpE4Y=; b=vw1/ZGM2qjIcmbBDh+V3x40NuWLEfWFCiIKr+VtqLd31me8HBf52u0Ysb3WiTLhgATZb i6O1xnfZnkEVlBHje157sMPbV7rggkg0fUhfi4a9UTO2AOYafzcFUZqzgl/n0CJmTUPM 5pBT+a3Zb+H6+PyRcWpP6XypNJvTbovaDGFzmcbj9hR+XA7GBVKvez0jB21kWo7pVO3/ wDY2sIUECuKkrAE0CH0nWKKxF2kpRK5mFzlq3mqtF5i1mQCmymfsZMYnUoz7iL1TsAZU 3AaUYBF1uxig6le8XI2wRbovVFbdmvJUZJvgpNh6umiKGAbEmFqNEFANax0TWBQruN6+ dQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad85qp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:02 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGEiq003232; Mon, 6 Jun 2022 20:38:01 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2107.outbound.protection.outlook.com [104.47.70.107]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu2c3xp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JOOXYVyLTz1/semgq+DTfXluDjUuSH9Avv0SUJM2HYqpcy5NRn4esBT+Ab6CVHDh12EcOAA6/XfK1c/mkNcg+8PfbkHi33raNd15WuwpBQ+f/FbeKPi22Xv5XF32XyfrOkcnc3Ux+BMyS335plWAaFGqRBkU0ERdHISQuR47yyQYkvKUZnFu1vavj+VeUndMPah9QgsMW7TQMXIwwslnUkkhl2GZIVAaaZtiNZc4XkjO3zW7E8RIqh2t+EBFNhtBd7NgASdTz+NegHKg3S7IX43ssQdzZX5wePzHJkPVe2U0FImYmdWnlFfNvPappoM3wcZ5p0PpYJ7gFGgRfNmvtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xs1FcMyh1+cEu2F4KtfsVd2nLcKLmWoODKCscmnpE4Y=; b=EjngPIg1IwCU2WolRv5TKgDe7dYTiOu0MYIOl4yqvHAeDJYgysg8FLJmSO/X5+4h4BB4zRXgeYBrgKMEw1If3onRteyaLBgEuyXjCGrlaUjs5moQyQQnSuEbuM1Otu4xrmRLDZqthnhpuUYka8GXk5AEfBgDPMfd8D0eFW3dKykdW6+Rjy2Y8/DMXOLPd1PPckNdzQlF26JWgensFZmIhjy1bo3a6hltpr8VH8TFcYb0q6NWuFvRh4PmpMshGaNG7nU/yoC3tDuX0nq1f7kYLGB8WprCiPfgDPcxZtkQeEpdyXIOjfJJkdsC5LRYtPoHTlpg1yx2/zGyMFcTXTbikw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xs1FcMyh1+cEu2F4KtfsVd2nLcKLmWoODKCscmnpE4Y=; b=frEqrL4xdqBFzqzF2eG5VNtDfAYbpqM1pGIxGsOg4FjPuJGVPhOTMYwA2xC9Me97H4lE3JRHvYlY9kyAE0MpIoovPh93LXs473DSDOfy+cZ2soPPUu8gDLh7eBMgRlOHbmbqrDeWNyL2GSqJw60Wnvm2fwZ0Y6v7nSr6EPAE22I= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:37:59 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:37:59 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 08/21] perf bench: add memset_movnti() Date: Mon, 6 Jun 2022 20:37:12 +0000 Message-Id: <20220606203725.1313715-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BY5PR03CA0010.namprd03.prod.outlook.com (2603:10b6:a03:1e0::20) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4ad4c885-5b78-43be-7f2b-08da47fc7189 X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MnyJaBJb31Bjh/Cq5C9hbYebwZcZz8T3s82k1eB2KKdX5TuGY9OlcAvo+O0yxdPGtjMssitNJaqsHJUc/MsY+bsG2lz4zfUVXhzhODZ1ZN67qiALVC18OMrvORq6RVfN5pCVSPV7RArt0KIQ/0jJ2cPWJUw2DRygES194SOEETrfX7R/9tx4PKndaTU3bDq/s669Yf31XQL0cTHE2m5MwQAd2HAecXxu2ih4Enc3YUBh2l3hyc6EDiw4IWC1SHf0uoNRo1bAe6NFGH039V37+A/H7E+pyT//1GgOs7x/hHqsmKIBF6Fnjjr76WAK42XdhBkA3Izm14Yji0qM/uV6ytqifAVU1rvxQV49+Spxvx/F0FcY81y4cQOBrX2Vhpddogjdeyfb8P2i8NCV2DLYwVYyM2m5IXKzguuwl4u2s56cphBSODMP7nOJwKzA+/cKcAODaLUKGfBTXNR6HOqeVAnwK2dJWlP+lJ3JFTZFNzt1T2l1/RiAAD300Zpv54Qfci8Z2mmqQQ6Xyiq+vTBxzCnJLIEiLh1fgq5Ov0Rz6q9TnpuuL7hVGvwiN9qBXs8lNzbj/qsV2l2yDNa6HJmsHGo1W7CaNGUBxl5Zce3d3r3rkzfXnVF2bzVbWJZct3p0rr+scqHL1dDnv0nKvqAPBkAVhI1FNcUCXWt0wJPMeVp7jLvVbOObb0ML9lxKLfLiPx5Jm4AEk6JJi3++KsUwqw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +csAo9DkQOd7CHL7ByfpLE2uvEmjzIY/eHavsgcYpeHIesvxtL8htaQEVpAdFOXBdG2Hn69y+OJR23aaiOcUj25/qfposfrqdCECKMU1KZjGjNvclLBNrJNwOVNQYOzR7iwswQinzFMXpzJFNPu8J1N53rn1/dmqNJglhtfh0gV/5Fmo9W8mLP3+/H3ii0QlAL1uBoVROTBhJiS0Gt1afL7o3a8JwHGVBwfI5xRnLRuYv1Wxqz60US2MPf1MgIsSlTFZl8Vwvu7oyO6FDUWgXMW2rB+4IzuhChTSKe+oDtJo4HXP37x9vFGjWQH8ttlhwpuCNhjR3MWAg8lggWmeRXohtpitTLu3T22w638JDxyZxwSU4e/56ael5m09aNjDdPbOhK7w076jvfwtcqFdbudnqDZMylvxHLXR6+hbtcdsEYVRkTIHc+o9Zy8SaTFRw8MlXhOfOwCUfdKKMJiUv3LQXyOEqZwoTIl+JzLO46NLh8qnB8rYvCt5+seAcO4V4mxee4tfRUxTwKG/tnajld9xnsC8CgI1iIxq2gVZ3i8KHscDtvq0kWORX25pYQ4uHgE2pHSRv9oNQwwqZkHxQdthkEsi3/xhTtPlVSqM49BKISq2IW6iSOVfm367snhTcR9jhia3hjekCKEBeDZbIg/PtDg18KJm/BEUFCUnrHvyQ3qUVUPHlXcwO3xqQMEbZxK0GmuILzuwrlUhV6KITecTwxj6frQePx1IUZ3MtesWCIlKSwr8r/nF1koZ0O9hdyeSvJEsGD/b2JMFHkm/Mszsl/Nd0RZqPdDWhIWk5z+dY8LZ7fW0rImt6Fpwun+2j8wZrTF7oFvG33bBnaPsSOBi51TUkbjbdo71xrG5wyGU9AnxnUM3wxH6+j/HG2CB4pwKcOHHHTNRZ/RiIWc+AexfRk7LVjge8IQDB4Dtxee8kIdXPoauBXUkuZNHEeomGqL2YVIB9odhupefsBvF94Sm/YaOwnowZwaEEXX1Ohnazn0VSx2t0Wd7IvFI8AndfNttdjm9z2BOMtkl1WFL8M6JDloyWLqrUcbvBw65A4lCWHZuvwTfagwbW3A+ikXMxuXDvQeG8P/Ve325XexKRtr4AtHpnHV3pWOeZRCHdykns2wDG+V/o5ExE2D7jbL1QO71WPv6dICOmyaAdYDcsSto+DRFnvd35EsXQt3g+7SENx0k75smeLcIdhDBYRxKoOE89XmsOVdLN0+t79QNLQ81IBalvnLkf420SdamIhSbcwE4mycSnMAiBJMvDb+pTMVHirM33hGrLJrcuK0m2zNnDPnwKkyX812SgZg0K8Soy+rta/QZqgzER5lPcdlJo2oH+5sjEzUIM/exbc3JqlvRqGBiXo6ENZlLjAcYhG/hCCa9CPlyJdcFhAIX3RJIgSuQKfdINGcavcPmRgGUb1HJiCQehk3kvDNoJRlALkI/ix+Kvir5K06Aw+TWfveEBUY4BZ8nHpsgYecColE0Wng+2tt4jOXWoTBx5MJYbwnGeEyzXEgWzrf0Iggbu9C487nYG6lyT2W9JPBnWOqjZlmcX5CC726l0fHeOV/qZ2d8+LPV9cIkw8CJEIbiD9Buam/IC0u7vo98sR9GvVLYdWyHv3m2Qaby6UbIecMnzfjRvz8dH9jxze0iYNNV7zsDJZk5X/16pFC2lNfrabJwvjUdrjOv8w1K6zdVHkTy+a/VjH//wJoAFOStnCVHE8GAKdtqWhQHn4xI29nMTAOaIJjrazWh94x7nBiWIBxaQrQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ad4c885-5b78-43be-7f2b-08da47fc7189 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:37:58.9607 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WU946s7Jv3ycaAKcjqPqM6Wv14iO4/ZuDqjIPBx+l64fMRdX3ZxkLexQ/Wit0PC6I4yLfcj6MkpaKwWKj9OMYJmII9SiPEOse3uxzh/b7jU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: SQc2V6tU3oHxamsZOwoXebM7oJbWV7Q1 X-Proofpoint-GUID: SQc2V6tU3oHxamsZOwoXebM7oJbWV7Q1 X-Stat-Signature: uiyz6r8gafp5spk4oxiezxqbrut54am5 X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="vw1/ZGM2"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=frEqrL4x; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf30.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BE4A480032 X-HE-Tag: 1654547859-1163 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clone memset_movnti() from arch/x86/lib/memset_64.S. perf bench mem memset -f x86-64-movnt on Intel Icelakex, AMD Milan: # Intel Icelakex $ for i in 8 32 128 512; do perf bench mem memset -f x86-64-movnt -s ${i}MB -l 5 done # Output pruned. # Running 'mem/memset' benchmark: # function 'x86-64-movnt' (movnt-based memset() in arch/x86/lib/memset_64.S) # Copying 8MB bytes ... 12.896170 GB/sec # Copying 32MB bytes ... 15.879065 GB/sec # Copying 128MB bytes ... 20.813214 GB/sec # Copying 512MB bytes ... 24.190817 GB/sec # AMD Milan $ for i in 8 32 128 512; do perf bench mem memset -f x86-64-movnt -s ${i}MB -l 5 done # Output pruned. # Running 'mem/memset' benchmark: # function 'x86-64-movnt' (movnt-based memset() in arch/x86/lib/memset_64.S) # Copying 8MB bytes ... 22.372566 GB/sec # Copying 32MB bytes ... 22.507923 GB/sec # Copying 128MB bytes ... 22.492532 GB/sec # Copying 512MB bytes ... 22.434603 GB/sec Signed-off-by: Ankur Arora --- tools/arch/x86/lib/memset_64.S | 68 +++++++++++--------- tools/perf/bench/mem-memset-x86-64-asm-def.h | 6 +- 2 files changed, 43 insertions(+), 31 deletions(-) diff --git a/tools/arch/x86/lib/memset_64.S b/tools/arch/x86/lib/memset_64.S index fc9ffd3ff3b2..307b753ca03a 100644 --- a/tools/arch/x86/lib/memset_64.S +++ b/tools/arch/x86/lib/memset_64.S @@ -24,7 +24,7 @@ SYM_FUNC_START(__memset) * * Otherwise, use original memset function. */ - ALTERNATIVE_2 "jmp memset_orig", "", X86_FEATURE_REP_GOOD, \ + ALTERNATIVE_2 "jmp memset_movq", "", X86_FEATURE_REP_GOOD, \ "jmp memset_erms", X86_FEATURE_ERMS movq %rdi,%r9 @@ -66,7 +66,8 @@ SYM_FUNC_START_LOCAL(memset_erms) RET SYM_FUNC_END(memset_erms) -SYM_FUNC_START_LOCAL(memset_orig) +.macro MEMSET_MOV OP fence +SYM_FUNC_START_LOCAL(memset_\OP) movq %rdi,%r10 /* expand byte value */ @@ -77,64 +78,71 @@ SYM_FUNC_START_LOCAL(memset_orig) /* align dst */ movl %edi,%r9d andl $7,%r9d - jnz .Lbad_alignment -.Lafter_bad_alignment: + jnz .Lbad_alignment_\@ +.Lafter_bad_alignment_\@: movq %rdx,%rcx shrq $6,%rcx - jz .Lhandle_tail + jz .Lhandle_tail_\@ .p2align 4 -.Lloop_64: +.Lloop_64_\@: decq %rcx - movq %rax,(%rdi) - movq %rax,8(%rdi) - movq %rax,16(%rdi) - movq %rax,24(%rdi) - movq %rax,32(%rdi) - movq %rax,40(%rdi) - movq %rax,48(%rdi) - movq %rax,56(%rdi) + \OP %rax,(%rdi) + \OP %rax,8(%rdi) + \OP %rax,16(%rdi) + \OP %rax,24(%rdi) + \OP %rax,32(%rdi) + \OP %rax,40(%rdi) + \OP %rax,48(%rdi) + \OP %rax,56(%rdi) leaq 64(%rdi),%rdi - jnz .Lloop_64 + jnz .Lloop_64_\@ /* Handle tail in loops. The loops should be faster than hard to predict jump tables. */ .p2align 4 -.Lhandle_tail: +.Lhandle_tail_\@: movl %edx,%ecx andl $63&(~7),%ecx - jz .Lhandle_7 + jz .Lhandle_7_\@ shrl $3,%ecx .p2align 4 -.Lloop_8: +.Lloop_8_\@: decl %ecx - movq %rax,(%rdi) + \OP %rax,(%rdi) leaq 8(%rdi),%rdi - jnz .Lloop_8 + jnz .Lloop_8_\@ -.Lhandle_7: +.Lhandle_7_\@: andl $7,%edx - jz .Lende + jz .Lende_\@ .p2align 4 -.Lloop_1: +.Lloop_1_\@: decl %edx movb %al,(%rdi) leaq 1(%rdi),%rdi - jnz .Lloop_1 + jnz .Lloop_1_\@ -.Lende: +.Lende_\@: + .if \fence + sfence + .endif movq %r10,%rax RET -.Lbad_alignment: +.Lbad_alignment_\@: cmpq $7,%rdx - jbe .Lhandle_7 + jbe .Lhandle_7_\@ movq %rax,(%rdi) /* unaligned store */ movq $8,%r8 subq %r9,%r8 addq %r8,%rdi subq %r8,%rdx - jmp .Lafter_bad_alignment -.Lfinal: -SYM_FUNC_END(memset_orig) + jmp .Lafter_bad_alignment_\@ +.Lfinal_\@: +SYM_FUNC_END(memset_\OP) +.endm + +MEMSET_MOV OP=movq fence=0 +MEMSET_MOV OP=movnti fence=1 diff --git a/tools/perf/bench/mem-memset-x86-64-asm-def.h b/tools/perf/bench/mem-memset-x86-64-asm-def.h index dac6d2b7c39b..53ead7f91313 100644 --- a/tools/perf/bench/mem-memset-x86-64-asm-def.h +++ b/tools/perf/bench/mem-memset-x86-64-asm-def.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ -MEMSET_FN(memset_orig, +MEMSET_FN(memset_movq, "x86-64-unrolled", "unrolled memset() in arch/x86/lib/memset_64.S") @@ -11,3 +11,7 @@ MEMSET_FN(__memset, MEMSET_FN(memset_erms, "x86-64-stosb", "movsb-based memset() in arch/x86/lib/memset_64.S") + +MEMSET_FN(memset_movnti, + "x86-64-movnt", + "movnt-based memset() in arch/x86/lib/memset_64.S") From patchwork Mon Jun 6 20:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CE15C433EF for ; Mon, 6 Jun 2022 20:38:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 166E08D0003; Mon, 6 Jun 2022 16:38:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 117B08D0002; Mon, 6 Jun 2022 16:38:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED3918D0003; Mon, 6 Jun 2022 16:38:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D96498D0002 for ; Mon, 6 Jun 2022 16:38:21 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B789833591 for ; Mon, 6 Jun 2022 20:38:21 +0000 (UTC) X-FDA: 79548973602.23.722FD9D Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 3C746100040 for ; Mon, 6 Jun 2022 20:38:20 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256IwlxY031341; Mon, 6 Jun 2022 20:38:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=eL5qcsZiSIZ5J1kWSXblYsbuTU2Y1/hUPdYwh+A3GPQ=; b=yevygIVZZj/MEFsAC58tEZpjvCxoFM9rJbdQNDb2IuoAtcR2n2Njn9O0+RekXLzynZtW qwsVbvky+vFZfhEp3tHIj+eaNd8iheVGoaSRGVfuC9xR740VtPjOXlEF5vbEdghgNymP OFXGUHPGxoBqLD+1HqRVQ/IpMce+ynHQBo2NHzJKz3uQxY9w3ffyAdIK7scLH/ljT+98 Tjrvk8RcSaQKhGnsqt9/lnHZsxD6Me+gZY2hKIF8WrCeNCzFh7p8oHFT6B2nzeSHzvCL dzjm6IJD10rzLt1YcOBA/uulSQXGTeRrPUzl/3oXjgMx9QdG7b7eoVyLnpYYhAu1cMRj RQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfyxsc6j0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:05 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFtm5001319; Mon, 6 Jun 2022 20:38:03 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2102.outbound.protection.outlook.com [104.47.70.102]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu20r2u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WUS+UxHoZSp9h0kOscwe56f8fHC21FeYELu6BkqsDCZTIQZ5lm/pzqrmiKCuJwTveB+yTzPet9syNUta5Rabzk5r4zmby8raNn8+LjRSyM9K/ICwIB+MrfVRKEheqPENE1ZY1Jn1Q7BNVKofmq6mbfynojn7ktfDiFmwbqTHDPM3vrYSnMxBh/jCWG33W5i/YQwKG1ggkmGDc4AGcM+jSgyHNzhkN6ILrnc+HKPFGiVDzs+mQjgR7oqxenTMQ4gCdJRu9Q8Ywd0BlHbIEjwasIgKitvk2Ao4vwI0WA8YnvUJLKI+NY46i9hz1eMCN1TeK/OPVwgTboLgSG27oKd2kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eL5qcsZiSIZ5J1kWSXblYsbuTU2Y1/hUPdYwh+A3GPQ=; b=FDlPhzGNaRRcoZeMi0mI2HDg9p1ZknZBNA7WdvsMp0M7eMiROtk3qWYCmY6T0btYt++t8wF8wI/QAOCHiwAH1Fu0bgZqmZSUmmX8Ji/lN8oEn8m4cngP6TA4deaine8FspcODm8cr3IFzUkKMTfUhtc4xOzbiAvLTv7yqUjf1ugwElptc9N+An2uXatZmoIv+MzNTwaWYqVzqg2oXLuSC/9nZulyhAZWR0zZXwD2eCmsFdZEXgfqPX5US0sCisL5V19NPcbvSi2GuzWiZ0OxksBQc4mKYk2h2Zom3lVQbUabWoUtFDGp/XVqLNcY8uiLICxEQfe8wyeAIOOjbeQeTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eL5qcsZiSIZ5J1kWSXblYsbuTU2Y1/hUPdYwh+A3GPQ=; b=mWzbcMX6y14+OF0ZV+wmhQb3wU+5DuT580QltXT9n3WyTDapv0TQN7LKdZRm0XeHJryN9gQmtecPGtLTXpC0b6QZi/Hv5nFmWTpnxXyRBMNMcJEpvM32iRCCti4B1ecuu80lJZy8ppzVD70HEi2Ubn7vyO43PoAc4s2JAGHN898= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:02 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:02 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 09/21] x86/asm: add clear_pages_movnt() Date: Mon, 6 Jun 2022 20:37:13 +0000 Message-Id: <20220606203725.1313715-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BY5PR03CA0010.namprd03.prod.outlook.com (2603:10b6:a03:1e0::20) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 64c30f3d-a8c2-46cf-5107-08da47fc735f X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XSH8WRZJ2kak/iLHg3MfNXvBR4DWMZ0tNBac0GQ85xhKN1M2SpW2VD7FApM/qmxBiF5pYNgDVEqYyQ6/Yyjcplu1DkRCkYrJ1mkzqPOsUAf56k1P3N3QyUCnODGZ4r8WxmCpf5bNBc62Y8RpyGy1Mb3cASBxUlQoR+lauRMUzvHCPgvDyfRaJMJP76wv3FdM4+ghfkRgGBgaDp35g5PoZB7bLpPxJKd0r12kCIyhbk3WNdyr8ASKMRkB77cR6PfSFWY0pWc+Kx7kMdVE8t7mA0FlhK8jRs1WZQoH+gMWjJDm+cpd/Z7SToxSGtk1/nFNFwwXVjVSNLf2hqI0aeSO3gCdhqOIwL8IwsQvlfgg/C3qYOaieH0pQYgGN0JnZuwNzqlXHNlWLBIAdjRIbWx3U4TzDh0fn1bOoETR89m3w64F+oqi8ParHO/7AVw46UJjyUAqAm0n76w5MDtPeqAXUc/FGGNeja8QZ4R28gIMpPA0WufATmV/vgkUULsLHvwO4msHAp9mztwh4yOYeXQw0EnEU2ix1BbR9RDlxoIuG/yqqPJl90wknjiVJbsMU3VNAxWf0t4TENP9A3mXkFesXmG5Q4wF/5at3adQnCZACBCchNIO/TM6JfLhZJy6HMbByW3lOiOStATnPpQcYmUwcs22aIsLpm9MtzzQSRPkGSBun8y+CAX6k0/5Rde9e3PRIFzEjw2W5pxMrZI0/GhSGQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PYHJgaeRKuns1B3PpeFt/3ZAk9cmcBMG/pVs6jNm7E0xXH+S+4lOmq0izUwzvzzfwGg+OC1lyXJLRgF3e4tLzvzwOcBqmxVyKzrNQga9lpQEimSfarMOPx95zFPn5NDW8IZpy51825jkHeYxR4ZoFnX7gyaFqiFpeFzoubE6c+4PSNnhtjoJcrfTx6V9xD+eBxn/5lEunVJRLyaz03+JbKBf1EwI7tcvkoaMFPlIExNtnwvlU6h4o4sSQI0KvcPzDeAukQOn3vnKnh5OaNSN3DZPOzYloQwYZT0hejDS32THkvxzf9u++thjEypvyppZ26t0KNUvMyfgXI+HVCV16QiK+kDBtztnRPNdzZ35oRasz6xSKOsQ7FBo9D7VMgKFBpyeXWeAOIJsgU8OOs+liMdpugqqSrxzBH3o55d7L+VfeZPAjZ3QTwBT/3uFrYBJYgq6cV5HX5iDv2frRmSLk2nw+oV1tldatf7zG+lQEwXO+2LWY6paK4C0UHltOTBmefLnqUJKjpyF2AqqYWuHQa4BDZTArpk1jBsKd3PcJsWC1u0rhPCv1EHXK4HtqI3gFE5Dz/1VoPCr8IHOMZTdHJejGwpx0LkH+gN4I2iajXy1v0X/odV4y2gBmVKEfqVlLavOkBQ0qQXM6hXmUevMh1YH6vdDLkNd34L8grGzTfp0ynM5A+CjmeW4H6U9gxAxflz1EBbgqDYU5IeRewFrpksTW+FyU8ma6BZ9xzrg/lyEi7Gx2n1l8da6haNsy2jOf1qTkuTq6Tlp6ZUMdC+XpwxB37vPXmuRDDp0iJwX4RndCeA2GTvgwqwG6KUxXUMWRdqx4jSUAwyQfS4fTzrXkEZqCuU2a/Jgl3SP63XM8MfQDp3yBH+szsq/FMxlyWl668Y8J2ZWxCJnEbDUByFwoPUQFa9D24BY6293gm2QHFsFk0Vx+7Z3tbMKmC+P3THW7eJWp8+63RWQIUwfFOquNsizf60ZOstBx6+E7e3V6k6OVFGWxHSrkZrRvqDYRNgDI4ODszpWxwFMpV5tSKl/cCgySeYaC6Ost6ehMDHc39qYr1jddpP1BJXkzAfgM7Hkofh+zifmld4bzEi3UZbaU3b0FIJb/vyPNReSzIV2WhimyaftCsTIE3KIqWIq7zSLouWOGiDdj/u3kCxZ3ASsm5YHzCzNs7Mci3T8+MyrPJVuqt+uULIEKa3fB/cVLhq8dmIe4SxY7YGrKTdbHvbyhEM+1kCH7YVVSNF8j69H9pzQKyO1mq31mZnnf3CDueNNW4pWLDlXpBjbvtuDT04l0ah1+JNMnbAzsCG0n3F7EJkKn5anbE57mayE5jDQjy8PGRUIpDuChkOiPj9eUPj22ipaySTniKjH4Byh1b57d24FaUwAoyewFTeYlNv5hiUVskUfgwOSDhIkewSj6Y9eU0eLZmLpfkWTxvovhpS2Ifotm2CFluh55oLUjn2Vg/T+sZyUXGvRYLJ9+B82zJxcyhrz4QWinF9J5xAQIyI6t/+5d0PVf4F9u4oZQprroQDKrZKlkZ7U+HMs1xNcvXwJv8SGgi2s4RNXiPkUZfZzZ5RKg12qYtenJVBZyENiU+qWiryXZS6PSSSHoEIfEEbjoDO/VUkgO0kNUff7YrIi51sxYI6F8ptan9rovH7+jNnZDLm64Czim8HPomQVzgAuTYSTaHWG3MweT6t0J10b89vy94vSrqPciAIs6jhngFWtr/3AjFDK+fQVKqX5a8ed2S9hy7/mMz18o3XcAKSYQBE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 64c30f3d-a8c2-46cf-5107-08da47fc735f X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:02.1178 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GJf0u9W4+IQ9gbojQpnXOy6HjyDIo3nvSoONtZEmjfFpncg8VrnZEMbrD+DVgNu9H6Q5uJeedGoQ7SLwYf2BeOzD5zziymbXfTKap6SHISQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 adultscore=0 spamscore=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: PzDcp0hrTT5fkCfnLjFQGqmyc9VxQ1SN X-Proofpoint-GUID: PzDcp0hrTT5fkCfnLjFQGqmyc9VxQ1SN Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=yevygIVZ; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=mWzbcMX6; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf14.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3C746100040 X-Rspam-User: X-Stat-Signature: k617dgyjaxe5rnw4tua87inewxkswb9n X-HE-Tag: 1654547900-377730 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add clear_pages_movnt(), which uses MOVNTI as the underlying primitive. With this, page-clearing can skip the memory hierarchy, thus providing a non cache-polluting implementation of clear_pages(). MOVNTI, from the Intel SDM, Volume 2B, 4-101: "The non-temporal hint is implemented by using a write combining (WC) memory type protocol when writing the data to memory. Using this protocol, the processor does not write the data into the cache hierarchy, nor does it fetch the corresponding cache line from memory into the cache hierarchy." The AMD Arch Manual has something similar to say as well. One use-case is to zero large extents without bringing in never-to-be- accessed cachelines. Also, often clear_pages_movnt() based clearing is faster once extent sizes are O(LLC-size). As the excerpt notes, MOVNTI is weakly ordered with respect to other instructions operating on the memory hierarchy. This needs to be handled by the caller by executing an SFENCE when done. The implementation is straight-forward: unroll the inner loop to keep the code similar to memset_movnti(), so that we can gauge clear_pages_movnt() performance via perf bench mem memset. # Intel Icelakex # Performance comparison of 'perf bench mem memset -l 1' for x86-64-stosb # (X86_FEATURE_ERMS) and x86-64-movnt: System: Oracle X9-2 (2 nodes * 32 cores * 2 threads) Processor: Intel Xeon(R) Platinum 8358 CPU @ 2.60GHz (Icelakex, 6:106:6) Memory: 512 GB evenly split between nodes LLC-size: 48MB for each node (32-cores * 2-threads) no_turbo: 1, Microcode: 0xd0001e0, scaling-governor: performance x86-64-stosb (5 runs) x86-64-movnt (5 runs) Delta(%) ---------------------- --------------------- -------- size BW ( stdev) BW ( stdev) 2MB 14.37 GB/s ( +- 1.55) 12.59 GB/s ( +- 1.20) -12.38% 16MB 16.93 GB/s ( +- 2.61) 15.91 GB/s ( +- 2.74) -6.02% 128MB 12.12 GB/s ( +- 1.06) 22.33 GB/s ( +- 1.84) +84.24% 1024MB 12.12 GB/s ( +- 0.02) 23.92 GB/s ( +- 0.14) +97.35% 4096MB 12.08 GB/s ( +- 0.02) 23.98 GB/s ( +- 0.18) +98.50% Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_64.h | 1 + arch/x86/lib/clear_page_64.S | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index a88a3508888a..3affc4ecb8da 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -55,6 +55,7 @@ extern unsigned long __phys_addr_symbol(unsigned long); void clear_pages_orig(void *page, unsigned long npages); void clear_pages_rep(void *page, unsigned long npages); void clear_pages_erms(void *page, unsigned long npages); +void clear_pages_movnt(void *page, unsigned long npages); #define __HAVE_ARCH_CLEAR_USER_PAGES static inline void clear_pages(void *page, unsigned int npages) diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index 2cc3b681734a..83d14f1c9f57 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -58,3 +58,24 @@ SYM_FUNC_START(clear_pages_erms) RET SYM_FUNC_END(clear_pages_erms) EXPORT_SYMBOL_GPL(clear_pages_erms) + +SYM_FUNC_START(clear_pages_movnt) + xorl %eax,%eax + movq %rsi,%rcx + shlq $PAGE_SHIFT, %rcx + + .p2align 4 +.Lstart: + movnti %rax, 0x00(%rdi) + movnti %rax, 0x08(%rdi) + movnti %rax, 0x10(%rdi) + movnti %rax, 0x18(%rdi) + movnti %rax, 0x20(%rdi) + movnti %rax, 0x28(%rdi) + movnti %rax, 0x30(%rdi) + movnti %rax, 0x38(%rdi) + addq $0x40, %rdi + subl $0x40, %ecx + ja .Lstart + RET +SYM_FUNC_END(clear_pages_movnt) From patchwork Mon Jun 6 20:37:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02017C43334 for ; Mon, 6 Jun 2022 20:38:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86D448D0005; Mon, 6 Jun 2022 16:38:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F3D08D0002; Mon, 6 Jun 2022 16:38:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6483C8D0005; Mon, 6 Jun 2022 16:38:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 51C708D0002 for ; Mon, 6 Jun 2022 16:38:23 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2C77760CAD for ; Mon, 6 Jun 2022 20:38:23 +0000 (UTC) X-FDA: 79548973686.22.510727B Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf21.hostedemail.com (Postfix) with ESMTP id 770B21C0061 for ; Mon, 6 Jun 2022 20:38:06 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256Ivt8D017195; Mon, 6 Jun 2022 20:38:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=V4OQngUJPfD/jfIBUdHLpS1gprSeJlR6H2KU3mtWXFc=; b=t70D2vlaOiAXE2XYjLpUxt7PBHv3/BosecBuYFnQhpF5dv0gnq5YMjxWB9hnDmPnwUd5 7zzgJNXIcm3xFpnmYUVl1P12g8aq4Y0e3lapmMK22WeaJtgR3BWUriHyqKY4mM0+a+B0 lfrTC0xlmi4k6XKDm7++/bf5GuxS6MPQuz6MvLORRAkb+xzznZDWXU8eJ48SJP3gg5t2 9d4qohtjAeOd3rUiGqr/FXSg4f7K4fG9XVTNrjFeboTVm9N7pjAfX+Wa72ujshgWyTDZ Hp5jUtHZkoEFhsnrsvL9mW4L4p55sHQpzVBN17Scswwu8ZEJ/Qo0Q/OuOPpWeAoLZW/j OA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtfrm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:08 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGV1h035820; Mon, 6 Jun 2022 20:38:07 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2107.outbound.protection.outlook.com [104.47.70.107]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1vep0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jk0Ovu7PTODKSXg3TicoN0t3hFXeS7fd6L+QOhGsstLtrUFxIZvViJ2J9+xRecgQIeN0DS4PXBSKiTMWDwM+YLls6+HukeXKGTv7VxKY2xyvHA18vx7dP+dlkr9wrm1i2qXCG2gaw5fzw169b0cBadNm1xtCpEPWpNXsNI+1Ug40O7HoDMLIn/RY9SVChLh3c5vceqE2+ASkuwREGzoVy6O8okdfdGCwp1okyr8poKZGMGdQkVmf/sEaiwR7+fgiXMeb6yqGkcjedPcXyse3AhTowi9LCcIJ7kGWS2sHBPtnpIBMHHig5vmFQoXJG7wqXyy16aqnEth4JAR2+4BV9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V4OQngUJPfD/jfIBUdHLpS1gprSeJlR6H2KU3mtWXFc=; b=Ac55vMHkV8VeO0dOtdfFsPXxOOpV5ne2gADNfMLa2GE5VtPXfP1m1RXTMGkViLGCvRvmUmvpFInj/CuAnueWSjOSWV7Wg5yGvc0B8FxwVudkqixHTrxip4xJvYTyqYy9v/1hlm854Svw++lLu6eZYDrOgaZmNc8H3+WEG0/pF3O3E+FvptaTDsOeNgu8O9+9Nppvhwa/NPXJSTiFUSB7Ly4VNSCN/dBgqFxZONjtizORxUmId1Z2IpnXyubUlNGbiAgEyAlcMEO6NnE+GU6Bym86z8fMuF4AiqAD2mpOoYDdixn8KQ9Yxytir9IRIig41o85STkcAJe7B9gy/hYJHg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V4OQngUJPfD/jfIBUdHLpS1gprSeJlR6H2KU3mtWXFc=; b=k3jDP/I+YiEGjcjBkMvzn5I6xpGfUHIkbnGhCzZOsO8u/yNxSCtv0im0zK8QzpcofCiFe/4YHW9XHBuSa4FfOvRkl7gI/+gA+fS0hUd9rKcJJDI/ozPP+GKVn3e5lReZEPNG14LV2XGfb0R69pOUFbC46N/oxLt4qtJX2bFkpLM= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:05 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:05 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 10/21] x86/asm: add clear_pages_clzero() Date: Mon, 6 Jun 2022 20:37:14 +0000 Message-Id: <20220606203725.1313715-6-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BY5PR03CA0010.namprd03.prod.outlook.com (2603:10b6:a03:1e0::20) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: be1ccfdc-dd40-45d8-af12-08da47fc752c X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6O89qAp7iVFwQtpgejyUMmrXbbiE4zkSCUSWwJX3S4kdXqyS2ctwdn+SGbQkDPH9Vc/5L2kb5w2A2TvF5qQXjPs+lyVrzMn1OLvG3N7N6TZnRE0g5nK9mRXBqrwQlGYk0xdhKQVDUPf3eGFlfNDnqeGF0IyTx+OprPvtiyl8RKxMj/GA5NyH4iZmO0MY4M23SXngI3bOOysbtOEajtew2xVL+HpKM+uiI9c1/TNDXZWGg23rR4mqr8e3g45lGW7uIGaeRmY8O/+AaPH44ZNwXYlxp0YJn2GI7P2CKZpABIkiT035fSJpAqDL3cJ7LxwrwOmWjI6nd8/FouEyobDz4vgUyE4IJFNM/1nsL/7jayUtjAiYOTNmJkRYpjRxd/eXPwBn43+bCc3J0U/m9eskue4x9SlkLOSMN/0f5gAkTLgNa8FmyGYLiRt333XyX/v+iAcFBru+lfeYtJ0kxVhy+8t2xsgUpo3g3qxUD8dwoZ2IMgpRoiHPqy9ZA0JBnqa4TebLQjoByrz3mVTkkB7dfLMMLN5aG1nim4xBJ5j/nYtrrgKTABtI1GHVjjlLSUvM/fNePCIJ7XmfJOmZvaLH/G6d325wdFJ0YBv7s/7arP+0f440nU0ni5C4aLMBMDEKlwVm+lRr6esEhdOtR/EUjjUPbwi224v8wgF91THusgTpjm1XkTmqDZVsFw/FQ5Qi5J716waWbIOAR7XFhfxV3g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BVBAT0yY6X9AH+HLUCnHahEdIsaxDmR/AGINUVDVBENUyY4fx+q91HOZCT72J0l4ZkHVIjd8Kh9QUCtc/1WvNXYg7NLYc0pHCNKx+/icSA7Oemku4WGhA5DP0CK11dez8BrVzGUawyZHEE8plyIkuwZpAu2Le9zMo7UWhQzMMtlZRZeUc9dtsvIApn1IdLhCNxgdZGqyy1U+CCFDl5pAi7wubohgCzSwNonFGtyaQHichNPoErNh6lTbE3jnJZgDE8vAKAqz6vteDtV/qrlRoNFo6y2nIGW1aGlTya/ah8WBn1bvmEiWc200L/kkYIjFmvmIt/peLAVlGrLAyQEuvxxXf5mlDWTybemSP+cpp3K8GEnyTFCaDd1rdJ6u5vrQuiT3mKPR2t84XQnqE7UWrUCPGVDXUw5JT3KwBCvVCyQwgvoHpOVIcH8vRgTzVDlAxDveKN2Y95u7CaTNq+ulepVGoDUu6KZM8nEscrxXjFCHd5KJR4kHQtrUucv9h8jdhREgT1RRMcb1Jkl1VsZb3uGBm7bEDT9FynYCJ1gYl9wlRQyTArrjjpFKK1IS2WycYxBaOQQ0Hs+oM67f0zHgYvWWHTDcEvsb80iAhR7eZimQ8I413bjC2O0Zcw542BwyEpTqMG8F6o1PBL3BuFv722BwhVtcgzM10x9xnEPeXHzmU1LPIBi3XGhjCO95ph1eDPpzr11A+U2PPpHvu83g5KVT3iB/T6J7TQ4pnVX/YHoHyhjf1zu4ail47AdsEm2oGSLOm1lkk9S1epPXU8riv/nvgKpBK2vvgWW8OYkj0ZCVUPo/V0S9P8xR3ngofiEjKFIzvwuvwaAALQc8msZyHaMafbIFWYtcTUstuzkk1FeJ848F0fxXjElEk1zw2TP5uXaukN86aHpOw2S8gfyjCJli7Vytu/05gi2XJe+x7OaCMrqGTmzNgGaMKmqItUIiljSSOoJfdLrDZxmoAHl+D/St/7TXAshMb70g+LsBY4vFtHiu1DfN1lFBxwJHCG22HSyHPgSRVNYnJc9SneDe2gJoEGNUgFxaEAcyuWsa98d9ZekLPcQ8XM1fGlNx0y9WYwaVyjeosgn4hvuP2f3Lpkw0toRk0cDiPqv3frAMFsub4yoiwOWKCMbiXlDXsa63NRNc5PM8C4Xg999N0nn4QXbY8XP/ztBlTATFZk8Az4xYFkjV0nhfef0GMU47QWX4r9UfStzXxf9ffLA2hGjmH/4CoEqxYSaVrV5OasWcgFCL2fxvOZ5JPOhMVVadoH9cact2TbnVKKKmGuhWgMs5bOtEwowl82KNOhqDJaPhjCDygf04OGNCgMwFZJdK/6hPhVcCh2/BenuVa3ax8939WLKCPSSQkuHWKibX/gMZKMbOjgGnASZIWCaueEGfBBVuKlFX7FXFcue4cYdPAi4o/Io+FAJG1c54g0wRuG/Ns+mIXNwtioloL3gGh/swHzFefQW3XrVtWuMqAMHOANmDgzufo3XS6+ZgXif6JZw+/U823N/L06ud6n7lmD1jOdo3GlpImri7StWiblxdGHTemWnK3rdNzbUG8tEr+NkzlNWtfel5eBz9T4unMWG6k1O/lVw39HUO/VqCdbnGpHNS+xs3PsGMncHIObe9thl14lOpXK9nB8WL5WK5dv/ownx7YtL1ttPy2c3jqO8ubUgrM8ymOWmK3c7lWHp/OkyVYtOsjZSAf/F63qyvmLjrSYYqzP4Z9TCYYsYoqY9dCtsY2Qjo/MMnF8iNx8izhy6Q/zQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: be1ccfdc-dd40-45d8-af12-08da47fc752c X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:05.1102 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eTfZpQyXi4kGQkBQ2pSxGoFLvNaTxZPceFK9/L2GDb1WXHBZwIPJxkN8MReB2ffrl/IAf+A3Sgr4FhTAWVIFLZUM96gEnGRiXLs6k11hNbs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: -PXLBj47JlhPQ80ioHpcofYKmtgt8nqQ X-Proofpoint-ORIG-GUID: -PXLBj47JlhPQ80ioHpcofYKmtgt8nqQ X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 770B21C0061 X-Stat-Signature: 6h4nzabfwwocbmjqdf6t6z1zhrqdcb1u X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=t70D2vla; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="k3jDP/I+"; spf=none (imf21.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-HE-Tag: 1654547886-915708 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add clear_pages_clzero(), which uses CLZERO as the clearing primitive. CLZERO skips the memory hierarchy, so this provides a non-polluting implementation of clear_page(). Available if X86_FEATURE_CLZERO is set. CLZERO, from the AMD architecture guide (Vol 3, Rev 3.30): "Clears the cache line specified by the logical address in rAX by writing a zero to every byte in the line. The instruction uses an implied non temporal memory type, similar to a streaming store, and uses the write combining protocol to minimize cache pollution. CLZERO is weakly-ordered with respect to other instructions that operate on memory. Software should use an SFENCE or stronger to enforce memory ordering of CLZERO with respect to other store instructions. The CLZERO instruction executes at any privilege level. CLZERO performs all the segmentation and paging checks that a store of the specified cache line would perform." The use-case is similar to clear_page_movnt(), except that clear_pages_clzero() is expected to be more performant. Cc: jon.grimm@amd.com Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_64.h | 1 + arch/x86/lib/clear_page_64.S | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 3affc4ecb8da..e8d4698fda65 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -56,6 +56,7 @@ void clear_pages_orig(void *page, unsigned long npages); void clear_pages_rep(void *page, unsigned long npages); void clear_pages_erms(void *page, unsigned long npages); void clear_pages_movnt(void *page, unsigned long npages); +void clear_pages_clzero(void *page, unsigned long npages); #define __HAVE_ARCH_CLEAR_USER_PAGES static inline void clear_pages(void *page, unsigned int npages) diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index 83d14f1c9f57..00203103cf77 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -79,3 +79,22 @@ SYM_FUNC_START(clear_pages_movnt) ja .Lstart RET SYM_FUNC_END(clear_pages_movnt) + +/* + * Zero a page using clzero (On AMD, with CPU_FEATURE_CLZERO.) + * + * Caller needs to issue a sfence at the end. + */ +SYM_FUNC_START(clear_pages_clzero) + movq %rdi,%rax + movq %rsi,%rcx + shlq $PAGE_SHIFT, %rcx + + .p2align 4 +.Liter: + clzero + addq $0x40, %rax + subl $0x40, %ecx + ja .Liter + RET +SYM_FUNC_END(clear_pages_clzero) From patchwork Mon Jun 6 20:37:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06E6EC43334 for ; Mon, 6 Jun 2022 20:38:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92E6A6B0072; Mon, 6 Jun 2022 16:38:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DE4C6B0073; Mon, 6 Jun 2022 16:38:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 757696B0075; Mon, 6 Jun 2022 16:38:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 645736B0072 for ; Mon, 6 Jun 2022 16:38:38 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1B22D60BA3 for ; Mon, 6 Jun 2022 20:38:38 +0000 (UTC) X-FDA: 79548974316.10.64A420D Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf14.hostedemail.com (Postfix) with ESMTP id A664E100053 for ; Mon, 6 Jun 2022 20:38:36 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256JqXN6009827; Mon, 6 Jun 2022 20:38:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=QCSv9wnBM6566kGvcpjLE6r0FswLPnX6HxMKaM9sTGo=; b=z0GBVLX8NQiOUYuuTmUeHVm97mCUehinjq881jRw4dNaHqsOs9JhOJsjEvM5pzl7b/OG URlbjLqA8DKXcDI7lj0VptHJxy+54ZEf6voC6kMxuNkk3rTGArgypxE013pKw4/H2544 ynNuJzpuHfrT0DeFl4b1i/afUEPrQyOWiBgRDHuqtfHoKGzq1hzfrYs/2tvWRzj0KUQb J5ML/cJZfjIgezDahDtebpHIYqtkhxWrKvusA2XloeEWIFxAbgTB8kXhfW8UnBB4tLXr lOrthJYzzafVoR/k0VINvv4Sd77wuYFovBhpBfEn/s8Lti1NSs8uv3/8dnxWhFE6l+aS 3w== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtfw8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:22 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFw5L001423; Mon, 6 Jun 2022 20:38:21 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2103.outbound.protection.outlook.com [104.47.70.103]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu20r8p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eHbYh3wx0IpZPNb+dn2qHQmIfkahLHjKBjSF7eTL2rNsK940s4w6I7KO3YN5iXMuJU2MAEpoKwxVL0PtlaLR/xRXM6KM1SrdHXyTHNNQYB8V/xeQ/p2dUAUZwrmm9yTGBiVxfpc4IvEWHc4LKzWgWEMBBzzJcKzdMaw5FeCSDfi64UAQOha7/N/VG79oGkK0UFDQTBHQlBdU3XmYFfhya2CXPeMOP3f1Sl0ihbZ+dRYhPIe3PbfNUpuZhO5fPVst+VkdlUn/MLiqgUsRQAN+oO7HzlsUaQHDuIHv5Pqzfvg2lWmdDCwJXPDg7AchfXX4+6hpIWx51fQGiLZCZcibDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QCSv9wnBM6566kGvcpjLE6r0FswLPnX6HxMKaM9sTGo=; b=PewCIefgFMVXL+FWIqetLf1DhRHj7JqeeNM7tbJroYQmKgwyFZuIaFUhX9AstliwEH/qbZwFZB68r8xCOqqj7UqfQORtV3FVVvmolAx138NcXfaNDnaCvvHDxFNOyp6Nu9iCgobNuBEaGuqwqydAP0rvzvWwp2xGg8mePs55bl3YBdo6vv8eJfqN2WIzOpH3U7KsiB9P3L27ooepuFt1t7v/ue4ypEyzEY9ARWWHmHua458zYKlc62JgpLMtmcxW0dESjs2YLuESwTkJ7vkrVUZGd79eLq0DcyDH1bloxGkdznDz0f40yWQQlp3PDHp8kAtxHV2la94DG5sd8XeXZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QCSv9wnBM6566kGvcpjLE6r0FswLPnX6HxMKaM9sTGo=; b=OWG1T7UNpXjvNArSeyR05xiarpaWLmY6bJslmpoOC8GOk6MaCwvimDRlSmeIL7N6a8ziJ1OxFuq8SF7Ahg1D9nIvtXKf/FCHgSzk5BNtLC0Ry42AG/qT3Z97t/K9zqJReBYl1kKK7hd5Z7iSaEOxetsfhMsqq+m1fRxRbVVEblM= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:20 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:20 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 11/21] x86/cpuid: add X86_FEATURE_MOVNT_SLOW Date: Mon, 6 Jun 2022 20:37:15 +0000 Message-Id: <20220606203725.1313715-7-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SJ0PR13CA0130.namprd13.prod.outlook.com (2603:10b6:a03:2c6::15) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1ac21b89-53d4-4d3b-8157-08da47fc7df1 X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g+cmf01KNkOgAF7D0ZmwCc5R+yXM/e3vX84PjhYsnqd/np8a4jGL0zgC6pjgxuO/k8z9whCyYsk2v3SjJbw3TkNVCowI1WKPE6RfA55mb1vU+lYFG/eEyC1xja0x4+W0hU9sjoTSQvqJmKN2B9mzNa4qh+xFvTntKA78nuIGQXWk/R4LRM7hEjZlKjAG+YhsFfE4acIKyrSGYbyeBLQRnLPmqkglkbXTTtvvA/GWqXy5qECITcNQMPamWfNE1yVR8ki0IOosf8waaGQhWtXMYYdg+pD12T6xS9zYYMq9zTLxsooAqXxdQ6s8gA8EKMook+iNqHGMz+M6Y3yzBHdfIBbEOszxARlinCz+gYCH4F/2rb6rEpMwHBVTEerkJ8xMsoTwXDX9C3nqKaWTWyYnydOb5F5NMcz9cIphiiTrY0OO0o5swXvUyc10IC5l7lq1ElNlqqcD0QqU0Vo53a1Wreozz3xR2IbDcHcYpsU0rVTxQleotQYmb756yKyXk4Vd15O5sz/8DO+3mIooi2+LJ6iC0+uAIoA7ZWfsJQERRDl7W/TzyyJ7j9RxMJdJzhqqr3PJUsxR04D/ZAjfEP9/3whKcmPLV2VXAI0klEhGiIFjso+QPkaNIQ1mI988B+tDBMIdJYiHjd/jDcMRiAbT3EKLejz74XFgDz5Wfu5RcuxW6LATx46gkT99mI+CgRBOwvEcGtK2flFZS78x/eyW4A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: h/3cWwFpJYC7TQfsJeJD4sPvPRNkMRxHwIl5zk7Uwgb+Wuh8Jc/CtWXJgcV6uk9ZQ/xPkwRM9Dz1SnR/IXEjRiR6d3BQ80fwxKtaUgWc0vCfitbLwlbrKbvm3sISbuWeeAAXbW+YheykXEvpE8ywpvSt2At5vLuntrYYizqpl8Apd9nYGsiO86IWbkk7X2sit1nQ2ducl3tRaX2DHSD01ay+1rOM7CfB5BDVqjGk/bowhTS8EM+MjO04c55LKRx1d98GI0rfLhrWvwrxZCpn/hyD8eRxHdSEGfHaVp2flLrZpHtaxJJPhqttafKh+t3LbU6ZW/m9FYtPqHe/A3icg46/2mxjvm/7i+RfCF3WM4LkDpbe/yu5DFxTiiuziygAwhLAju2ohhWXwnp2ui5zMkrfoFSGJ5orqaeDDFTRuLrX3uxQsA1TxMHITrfXiRLnMGur5M4T4cWEmieNTzKiO4GZFU7UPYlpIxAIgWqf1WsFRKK7GOqt6VjRgCGq+3Fqp1WMfRvv5l0s6b6eaOCik8MPYzqiI4RUiOXuSspCKgsWG8E/vZyBQFe9lv8eABJYgitN7DRGeeLicRMZNJkraZCnT0OxxkLDMdkZS1v4R7kyNT6ZDXmzxW6pcP3zOdHgS/3XUW1honMh4uHmURSBbpaIa0GZB0+EZjFVqLQzxrRYHAXiusF6B43QCIobNqfGhSifnNlA0s447Xck4sy7+UFhEOevO0TP6D65tBs4J1m141jCb8AUdKEA6gBe8lzCZT/WdWczWqtdcV/l7T9ghV9u6/pcBMVZNB7kUFenDG3S2+esYRwT4EwfbBWG2hTklEbHf1P+lWQQOY1vZUoCYBhQ8UXcfMid9cYuNjrKOju1oLnleDhqVLjxTPpJLnQVNZO0MW/+rglbUcVDlnz7IYAuVEBpch/AD/m+U8KvRfxfC6BBoT+ND5Dhqw8aWMVV1ge8nlrvnMLDUSKwSIG6Q+DGpsjaZJ9564DwqGmdDhPRe/002B983V0bCaSuYm1pKZXuMrYICP+WMQIPBoqLjLgP88ntzFBnzbnsV3qxPrXrAij8aCEE6WsLI4HS6Ppv0mvH0I+76sfq4ys6PD6C5BPoGtfepFKenOXlMO99BoM27Tzw6/wuLYPTK1HSYgAtxEe3YT3taozeIs5XF3WcGc5auRJfIGMauL9J5nnrUM71gLPgaqMYlsfyRg+2jMUR8606ijAg7tmBcyRAlKYXd1JbZRN8jzJFLa/89mBsvevKSdbs7NG9KCHcqp/BF9yO2Pr6+54niEnBKdxOrJs40QyzUyAWfSzBqIyMLiNnpBgUa71hkpqB3IN2SHuLd/qG6TaSmMlBamjjzHwA5YiXu8a23cterRk9xuISczg4ZHFX3rB71wCUl1HHjYawRK6MsjUXV4eUP+/APoMT9g3eUtHDH8eMDZ0Opk3LQ4Qqo+NJCvMfbj0MOEKgni1NWalXZYvaXFnrZ+YmEeR/ZtFpz9+Zq7Oz8/q8eLYEtOY79mdVPVKYekPzFCI+q9JF2lP36pdh9b3JNZ+esWLymWtVmv2ANE3tfC9i2zQoBbbvA9hKu5q7tgFeWmhfCx3QbXR3Zn6TJVGbgu670DgIiVwV8KwfDpc0Mkn/mMF7GT1KaQ9JONwh/yf3YmNTGlH5oAwE6aXFxdo8CXIVsrmm8kpsHneCF1FQu5ZuZ3APWQe6TXOGUecf1sB7v/Hhxv7rmQAOvdb+hxAILGw9a5CEtWa/OWzF3du9SMHgEhkTWEDcK2E= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1ac21b89-53d4-4d3b-8157-08da47fc7df1 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:19.9156 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8PAQkE8ZPn5tzPRPquZNoFLVv1xNnsd9zkaaBwGPrKmhgULu19uWUJ5m75w0DNYZN8MoFy9GcYIn6edQDj4OPHVW5gc01SKBprBAULrl31I= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 adultscore=0 spamscore=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: jy1ErhQ04z4lTQxqVzK7vvvoIdNHbNrX X-Proofpoint-ORIG-GUID: jy1ErhQ04z4lTQxqVzK7vvvoIdNHbNrX X-Stat-Signature: 9hyj5smntici17srjr434wjm618o1owr X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=z0GBVLX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=OWG1T7UN; spf=none (imf14.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A664E100053 X-HE-Tag: 1654547916-123224 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X86_FEATURE_MOVNT_SLOW denotes that clear_pages_movnti() is slower for bulk page clearing (defined as LLC-sized or larger) than the standard cached clear_page() idiom. Microarchs where this is true would set this via check_movnt_quirks(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/amd.c | 2 ++ arch/x86/kernel/cpu/bugs.c | 16 ++++++++++++++++ arch/x86/kernel/cpu/cpu.h | 2 ++ arch/x86/kernel/cpu/intel.c | 2 ++ 5 files changed, 23 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 393f2bbb5e3a..824bdb1d0da1 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -296,6 +296,7 @@ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7) /* "" Per-thread Memory Bandwidth Allocation */ #define X86_FEATURE_SGX1 (11*32+ 8) /* "" Basic SGX */ #define X86_FEATURE_SGX2 (11*32+ 9) /* "" SGX Enclave Dynamic Memory Management (EDMM) */ +#define X86_FEATURE_MOVNT_SLOW (11*32+10) /* MOVNT is slow. (see check_movnt_quirks()) */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 0c0b09796ced..a5fe1420388d 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -891,6 +891,8 @@ static void init_amd(struct cpuinfo_x86 *c) if (c->x86 >= 0x10) set_cpu_cap(c, X86_FEATURE_REP_GOOD); + check_movnt_quirks(c); + /* get apicid instead of initial apic id from cpuid */ c->apicid = hard_smp_processor_id(); diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index d879a6c93609..16e293654d34 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -85,6 +85,22 @@ EXPORT_SYMBOL_GPL(mds_idle_clear); */ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); +/* + * check_movnt_quirks() sets X86_FEATURE_MOVNT_SLOW for uarchs where + * clear_pages_movnti() is slower for bulk page clearing than the standard + * cached clear_page() idiom (typically rep-stosb/rep-stosq.) + * + * (Bulk clearing defined as LLC-sized or larger.) + * + * x86_64 only since clear_pages_movnti() is only defined there. + */ +void check_movnt_quirks(struct cpuinfo_x86 *c) +{ +#ifdef CONFIG_X86_64 + +#endif +} + void __init check_bugs(void) { identify_boot_cpu(); diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h index 2a8e584fc991..f53f07bf706f 100644 --- a/arch/x86/kernel/cpu/cpu.h +++ b/arch/x86/kernel/cpu/cpu.h @@ -83,4 +83,6 @@ extern void update_srbds_msr(void); extern u64 x86_read_arch_cap_msr(void); +void check_movnt_quirks(struct cpuinfo_x86 *c); + #endif /* ARCH_X86_CPU_H */ diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c index fd5dead8371c..f0dc9b97dc8f 100644 --- a/arch/x86/kernel/cpu/intel.c +++ b/arch/x86/kernel/cpu/intel.c @@ -701,6 +701,8 @@ static void init_intel(struct cpuinfo_x86 *c) c->x86_cache_alignment = c->x86_clflush_size * 2; if (c->x86 == 6) set_cpu_cap(c, X86_FEATURE_REP_GOOD); + + check_movnt_quirks(c); #else /* * Names for the Pentium II/Celeron processors From patchwork Mon Jun 6 20:37:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76BE4CCA483 for ; Mon, 6 Jun 2022 20:38:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9C808D0006; Mon, 6 Jun 2022 16:38:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4C678D0002; Mon, 6 Jun 2022 16:38:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADBBA8D0006; Mon, 6 Jun 2022 16:38:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9E8A88D0002 for ; Mon, 6 Jun 2022 16:38:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7F18E351E2 for ; Mon, 6 Jun 2022 20:38:39 +0000 (UTC) X-FDA: 79548974358.19.95481F5 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf05.hostedemail.com (Postfix) with ESMTP id 46C9610000E for ; Mon, 6 Jun 2022 20:38:00 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256JupQx010668; Mon, 6 Jun 2022 20:38:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=DOjPtUnicS/yGBvgkglQQSNIzS9oMVReVWssGx1+fl8=; b=Xn3ivHJEr2fF9i1iW1yIXQS+7oqtsjy1T/vXnuPHn7t0Y7eMA7o+M+QXQk9EB+qThRL5 mciHQsQHUfKVDoNsc6OKiz0T1WZijk919dD+tEWPEda96edValh+z8BUO61H7adVLAOX MAD/M1/VoUWDAQ2GpXjrYkJK1vt52F6n5zXxBqUfdL2vTbWe7krd5x+4cmn+nSIzcPsm HnoFyYg8WH2npLNxE7rHlqtNDtlrGFEpdI67mhCephS122S2xx2dtrLltF/8XjGkeT8z 26ef3eiRZocmree9SDKkyPtK1yZJ51mERFa7HcH/dREc4n9M9+PXXUPPxyNLovZXeylV xA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfyekc25b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:26 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGODo037881; Mon, 6 Jun 2022 20:38:25 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2108.outbound.protection.outlook.com [104.47.70.108]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1uqjb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WEyeOt1yqx/m2DiDf+7BaIKokLEughZdzxpDqRQ7boTcQEILKJ+M1eQCZdF/RftNRIjeOYsFk7as8NGjPuccXGgQSdkC3wHUj5ZVsIDOfvL8vKWgCjtJWlbTYCdKF3RjVkuykr1Od5UGE3a3i49tKhvj9KD2FdPpuHmjRIoJDET5mmw9vKLLIanOzg6sSgRSKrGsAiMY3LdcL9eNIeur/M3BQJZhz3kKTYEOgnPtXkUcdkJCnpMJb+i1jx0cb9eKwT8kHWJ7KNdMhJlGvl+wVGIlbjh4vgcByd9G3HcJu7TUas1qnpr6nxVseWVV30pkWuzSun1zEhtbD+kyWR0p/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DOjPtUnicS/yGBvgkglQQSNIzS9oMVReVWssGx1+fl8=; b=f4umnED4Qneu0VpOMXbg8unv1jxpiM9Uzpbv96cV3UR724vud469LdUo8ftaa8jWslGUpEhKKofv1Izh33BMm4VG125rUY/huAUA5icnbXSEiDt360tXWubpZpXnNOQDeepi5GdpvUwR+zE51hquoGpvk+gy6+sU1RIzUFsAlGnCa8O8SiA7uWPtD96RsNM/SqOT16hcRZGfThTuSaOA33S/YZzYMzZ+fvpked1ymaJewlI7AH8EllAAszItVSmOuvLhlEnPU/HlOuU+Nq5fFh7vwEWh9dziHckfIf9D1DA4Y6cK3gJGG8HBVuCUytGqA3gT6Rbmi6i2IfymdLQkLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DOjPtUnicS/yGBvgkglQQSNIzS9oMVReVWssGx1+fl8=; b=sIHWEdFkBvJKr9W4NyhSBx3PJlBOUN8RMhcVg6ICbOQ8ighl8yb+FyFeQu1410hsg+Xl2k3aHMKMGbftrZ40nD6kQOaw+Q+/InMUWP3USMDg2Vqo+mdZFx3nV8d3GQnuUAATB8940RQW9kMWTBG9n5vyFswgZd1n9we5vZkLVS8= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by BN7PR10MB2596.namprd10.prod.outlook.com (2603:10b6:406:ca::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:22 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:22 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 12/21] sparse: add address_space __incoherent Date: Mon, 6 Jun 2022 20:37:16 +0000 Message-Id: <20220606203725.1313715-8-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SJ0PR13CA0130.namprd13.prod.outlook.com (2603:10b6:a03:2c6::15) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 65a770ac-82d7-45e6-7467-08da47fc7f9c X-MS-TrafficTypeDiagnostic: BN7PR10MB2596:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WJ+HYBu178ymT2w+Ta6HCYV2dP2Byhh28RwOMr33ZpK7esQyYr6JRYxJ/t9Jr5D0D93I7k+cPjVkh/qxrVa2bNdl1uI4WJmTknNRzQNaBVvpDONtnt0AtxPg5JXRWcqbyxsc6Hu0XEqKMsk9Ru9TVDTMW3w+S9Yua/QEZmwODUBVjiDUZwho9/f2H5rl8yYODcvgy7KrU1lSc19EBtiwoJ0vWEHdU+6cfnv13BIAuX8Tx357HE14r3PP+1uPmUx2WhO90WTfODV6WfpUKeMl62eX2Ox1XWlRRU6ymqAbXrFYZ6CiGtBRRuun6EjSY10f7/B8wJ5XsUGho6K9yVTiSp03vjgu/gHa7pZH+a74awBcRIKCe4AHVA5dNHS5eNZKs+1kWhjh6MBWsTUb+XqmDDJT4pCKOMXOX8YsJbxTaTZc2M9L/UZp+H2HyAYYSf9wbyTu+/QZUN370O/sIPc6C7aGhNt5YS6g0juW/D+GFYMOXWdlFqAJba7G0ZNpc+tehVq9BLpnYYC1LNRAlYfQJaIMFyDVlW4BpY1tX6yHyTzdmjaDvjv8VqVaPEzQKEVzCAZpmyqFiS3Ip8QNvo48NZx3y0F9+SUBayqQfTzMf76QPlTZC4DNNxFWSZ5NdQEbJarJSUPsXQG8m+hOQzNaFoM/J44o54E5eI2DhOg5j47C4AGjhdvuvFh0Kl9K3HtucV1euQuziTtsuh8JP6h/Zg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(83380400001)(66476007)(86362001)(4326008)(52116002)(66946007)(66556008)(8676002)(103116003)(26005)(6512007)(6506007)(508600001)(8936002)(6486002)(5660300002)(1076003)(107886003)(36756003)(2616005)(7416002)(316002)(38100700002)(6666004)(38350700002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FQ3tlD/fcjkeidSBamAxSKtNmiDvG2098wLcdYBgM2CM7fxuJc5AAEmcEuVZ3vXmTQsEWz34Xl13u3K/YL7z4lCaM68/iX93REDUXJMSS29rFlx/sxJ0Kik9WwDivgd5tB6s30YQPZARmbejVbTrDqskG0ujwn+udh8/465maY/0y/fRQLBmPUCseuqKmfzQQUFhbnmYbjo//JtKGNAQxiEgnebkFQqHDevmWtcZKjs1tuL4PfWQhgbskAmnL/iq06+DXlJAS5N/fUOrbEd/wieszcDMRbLfOFIWTlf0bQT4gyPquE0z2kmN7v8EwB+wSyzZmu4ZaRN2BcAX9Azhpjwd08gYOY4KSQpsFeCpSipbfe/vSZZtfNyW/gvk35xUGC5haBb3nMcGYg7GrBDlDIUl1T902neq2ZbafeaIE8pqBKLpkdekYi9Y4qduuVCnAvW2gzidoHwxiyXB31iZwiwewefYCisayvmJyu8boHD1iJsticxMDv2cfzcbDB6PmYZJWwZqNY0P5XO5DLMEp4jlekm+Pml0BYQgCbr4OIu/zHEYDcuxRMgPt9KvaCOvDq7x62trEMfdU4V8qP3y9IHsy8t3ZpwW7KRzATRiNLli0QIbUeEIErK7Z7SK6fJgchE8giXEXJQXiicHoLQ3wVuFGOfCHdKvzZR0k4lESgQJhEJ5UUrHAnM69i92S8YLs58FBGMGiPiPRgx4gmEuWtELMeZRXnYWbQUd6N/ZWi8y8gSitCkxQ8Eslr3OdALEVPyg2T9DiQ6vHkeRha8Hncu82EPrCao8o2W/f+dwTFYq2V66Rfk6ZiyfgN8N+X5jPJokbRScPlsCUkFYVGFJwyu3TIz/j5C3b0gfEnahsYtAU5Dsgxok0AG30p4SKihr0MzOy7q74wRFDHh1CXbewrjsUOlrTGozIRJdP0MkwcpdrKivV1VJ0hMg4nax2ZxA+6yFJ13/yAPqviXjNEtf8TPiWUAKCcc/8AFmGBIdQVFXzEohGHDvg6vUNnS59/FDQXe+r7218x2CvZ7PAYWoxxjl2Rw4s22GUk9U6pIHSp1Hzma1VxhZctVIgpH28S83QhMRw1wXhkRSnbESO4LbaYqNEE0O3ZVFtLRfPZlS10MpxmFOpkX1gDhG8OTgM2HnH7PpT+waRUsJMZtPXyavP7VYLSU2aqUZvk3bk3A7ZzQzsNREMgnKUUbuqpxUj8TCjdHjUHYdFfoM3v8fW5LoI0T4cwj9O2AjAkQYkd3RrjyCf21Lomw1ZIA2YBJYnW7WXg986p5jIhDtOKyp35b5DsB4vOgLl6O+mKtkS5jNeC5m585jxUwERTnvILlWJ5L3shHgWzaLNLLPbh3ey1Zl2MgcnWKoVstC89fZQGTZT4Cc1/DnmtIoG0ZcwjBQE9/Qjg5WpKYeQ17LdLBhcpUNOGpQAkPm2qy33PeShVFD4kIwgN/5oyR7ko/YWf0SinO60AUznj0I7Ltyw3onRFRu7K46346PehgWPIEY8mj7j/lH6uvcH5q6GS9xbaWRuX4KqBdq/yXqeTyM8qiUtWaeK+7wbfJpsfygb1RxSu4EOuqyW0d/UlTAnuH4P7QMNBeL2eylwF60odrMRdryxMBIX1/hcYlPv+ddfFkBvuETgc7t3X4ZtU8lCsxWVfLlwSY8NaDS+YJL4H3xlPdusucBhhwYuHeXQTQ9g5nIfA2BjXkUxB3Or0h+KPy6/+k/I31OCBWUvL4LYh6jhlZJOLVT8ZOeGtjqSlVCrF+IqLhC7hY= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 65a770ac-82d7-45e6-7467-08da47fc7f9c X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:22.7761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CZSoAQjMiB36baOBZln7ysobSOj9M0lXEbL2CBYkAHHWsRFLxDGH1lat5km//6rs0wF5I17wIeouLhue/uQ43WlmQYFVV8T8h8kjwm7ZnDU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=961 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: 94ZBBeZywuUSEb6fxQoQGBa827SWedSz X-Proofpoint-ORIG-GUID: 94ZBBeZywuUSEb6fxQoQGBa827SWedSz Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=Xn3ivHJE; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=sIHWEdFk; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf05.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 46C9610000E X-Rspam-User: X-Stat-Signature: qq55ozgw88xrhrgjqotdxiixn8qydkcd X-HE-Tag: 1654547880-483430 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some CPU architectures provide store instructions that are weakly ordered with respect to rest of the local instruction stream. Add sparse address_space __incoherent to denote regions used such. Signed-off-by: Ankur Arora --- include/linux/compiler_types.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index d08dfcb0ac68..8e3e736fc82f 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -19,6 +19,7 @@ # define __iomem __attribute__((noderef, address_space(__iomem))) # define __percpu __attribute__((noderef, address_space(__percpu))) # define __rcu __attribute__((noderef, address_space(__rcu))) +# define __incoherent __attribute__((noderef, address_space(__incoherent))) static inline void __chk_user_ptr(const volatile void __user *ptr) { } static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } /* context/locking */ @@ -45,6 +46,7 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } # define __iomem # define __percpu BTF_TYPE_TAG(percpu) # define __rcu +# define __incoherent # define __chk_user_ptr(x) (void)0 # define __chk_io_ptr(x) (void)0 /* context/locking */ From patchwork Mon Jun 6 20:37:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9250CCCA47F for ; Mon, 6 Jun 2022 20:38:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 33D9F8D0007; Mon, 6 Jun 2022 16:38:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2EABE8D0002; Mon, 6 Jun 2022 16:38:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 117658D0007; Mon, 6 Jun 2022 16:38:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F3C0F8D0002 for ; Mon, 6 Jun 2022 16:38:45 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C1E98963 for ; Mon, 6 Jun 2022 20:38:45 +0000 (UTC) X-FDA: 79548974610.27.4B7B4B1 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 62F4210001A for ; Mon, 6 Jun 2022 20:38:44 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256Ivt91017195; Mon, 6 Jun 2022 20:38:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=8gpbcHqofa3ZUH2AYI8aFIUjIzCV/nk6Swr3kdhQFQw=; b=gP1S49aqutBnkug2WTKe77ZF+6USRz1EaoVV6SYPDRNYFHwCnyAbbsH7Bv9IgYV4cbya Bn1zVDcrxj6SgW1arhVPPKM9mWRyhWOQl2zn1McHdOEijMFa9Dx/3cKU/2vT1clKQ+S9 C695Idqa7et0mQ7BtHgSbXtBInbjoS7xiMAScs4ElDJgMA0WdeSrDcKhzPVp04b9unKI VXM+z4QxleMiqqRD8+1dvSp0lGVWrsIEL82Fa575UMBWpJkEkWAn7vp2GswHuugpTwKQ WIlBeioanDyMpKrp3vrTLQ5JpZ+gbcBWHVFKswVJwB4L4XqvGseKNPmzbh4vIVdb4lCy cQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtfxg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:28 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KF8PG001195; Mon, 6 Jun 2022 20:38:28 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2103.outbound.protection.outlook.com [104.47.55.103]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu91cqj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AIa45NaG5SjyzHadejagKSY9jDkv0quHDQSMkKXIdT1PAIB+DnnD+cR1aGVa3xVC8fquq1PjdSnYB/DHQmDFM9ESeUUHAcJQvxSf5kNjMeEioEiwDrpzD3JuKGpu02NNhAM3Ku6f01tBOR/hP7gQbmwIp2qXarrB2ADPxycKtCyN0kqMLWpWr0yUARS0SwdIiNUFRGZfmzf98CvO/5MGIY4Y14ufzwa7KBcDsnwqWcmW61bmsy7D00jRodXgZjRQR234d5Md1yYsFEQbqNnJUOQUEVv/O6wZx9Np/27bSamKeQJ1fBB4ao8Yr92zJ4gBQjm3m9qBwLG6lK96snHQ7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8gpbcHqofa3ZUH2AYI8aFIUjIzCV/nk6Swr3kdhQFQw=; b=aN+7BqUBXzVC0/Rvf8A4x9AbXFzTsc4vMUMC8AlAx1mXYYR6p2/knh6mIMnYmFmZ8vYp4yE3WvDY9+lGmg+2/VA0Haf8crOQ4lI6shf3TufveuZ0z4ZCHbTNyrQPl8V/WIyLCh2PsiYcKee3Rgmo8byboy0tKO02+ApKg+YGNr0d7F50pmdchZ3L4WuX2IXhJwKJDl5SYBbGMacsJz08lhpPymTOK1MhqUaJYkDLX7mmeIy9wnvj7cVimvyHroP1kXWNP83n+vr4uZCsV7nLor6hJd4R117RbCSh5VGGIaejj0EI/B6P6ziYoRWdK4dJCl5rQhPWfSro2bpugq8ggw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8gpbcHqofa3ZUH2AYI8aFIUjIzCV/nk6Swr3kdhQFQw=; b=Cfl+OMJVAgMPA88U5YxSf0pNIApj8v59nXdrRh+qA01aYelwfCykCLM5KyfhC60dXJTS454DsTJOvrVuWOWBGfZzdbuSDiBcuFdfK/yjsdvs1BVNLtH8zSlVtWKi5Tepa64vqSE7tBBGAxjpYDALXhrzRZaiSQOSuFoIYwXu2UA= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:25 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:25 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 13/21] clear_page: add generic clear_user_pages_incoherent() Date: Mon, 6 Jun 2022 20:37:17 +0000 Message-Id: <20220606203725.1313715-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SJ0PR13CA0130.namprd13.prod.outlook.com (2603:10b6:a03:2c6::15) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 428ac945-9487-4c41-ed70-08da47fc8168 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GSwNndU3lPthop0EGjapoQ9syHlR7WpYqk/Ng0h99k9e8nUH2R5Ed1V+5QZa3WvOsKMABAjp9eYnHxxlhTFXu+oZyxyzMX91GcahrxK4o5qNHfHZl0doGTC4r/PN42MwyGEb9Mu9P8c+KJ4uozYVfMulzZfx1Zp6vNW5dd1Z0R9mFvleVLQ30hLwM8uyfVB9ontkr1lU0oRsSfB2yWD+U9o2Jrx1F2Mc3KBfrtlU6NbVgP0RET8aphsUBmfHnkaO6HzJNk4l/yTkcdtmpxmm2lLpjMqXxHWYFtK5jZ3C57kDaTtHw0VexEK0se6yB6c+dQdRVDqYfXK6jofQjCYwO9EBL0Hfv2zRS5c0jhOBZbjkOHoM6AcBHk9z1gHNMJCH+NsovxoqS1iHvVcWqBZ8ek43ayR3vjbsPuMR8mu6V7qa8zhOEKDnMFNems/hOmDsF5TPuhyrEXN2+bGk/5NsyOgQ7yWZ7ued1yv4M7A40z9bvhhgDjtM4F1jSeVyVIPPythS6hW1HW6sJ2l/mvg4azgKO/98hrW3gigUinb+a0T0R5R9TKAPi5SAsWArHWqnaxlA0si6CvBTPq3GH1g3/TvS1D75KRHI9Pk/zl7DJx/FbTnvKCByk34IWNSXQ4B8jpWAM6KCc3P1FO4w4tt1v2sN6oJklhlJGSVbt0I0LzjIRDVomGRLGqxeQdZ8Q82N/WBxbmQnbufFBojLXbuKm3QJEkp63QTJSPQCqKpXN1A= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007)(41533002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6lZzxuLLlOn4YNgtqQrxHoEmuDXAyXAfqX3oMwN989b7dCeE5yli9QJpkLudvXyH20EY4f9Wits7T1brB+R11jL2idPaELfT2XlaJRU1PYlZYZej9Ocok/gYr2ZTt1gBUTtpT67dWVoX5wmZyUBskAsiSeP6haidkKYXE0lCyoV8r4BKAkHkZvXNJpRafsb2eXtefl2eTTGf1oWHUB3HGVkCA5MJppYB8uLAgmZJjg5Fi1tGDeC+sPzKvsOTBPpQachPlgCuCDHrnm+qLPsXYYTNZ5hSzcUvwafNUq8fr8LXUF+7u4MvHEW90C+0m2NbCODvY098mYWM45sMFc0VA3uRuRvnaIRo9CikWUYIppOMWXSMXKVyh6Hnx0NLWpzCRfurd0mZUamwdUJKu4614Cj6OAM2hSbtuw2/VWrdfdusP2p5lsiCzaiizCg4HTrkg4SgeBDMg4hQ+61UaZKV4SOV17+YmdrljQ/qj4GD0cKZN/MaRS7WVVt16MULbpCIrVQDJZxbe8dQ6Wfk8XSiP8YzgE3mIaIDWXT/HY+ZcBzQr8mJW1qJHP8CNKU/1Cujqvt8e1dBhAAiFGh0gS6pF+T3VOKlnfmczOn98Mec072Sq79QOT44AjOP39vPS0XZgV520PNZCLtn0gPipsXCuyGv+r8nsmJn13kfiP249qPRt8+mZyrp+kWq0aAdNn7t7pqVdBeO7fCRTnC4bCKwNcsFmLEC9W3+gZ0ynX1eFqgE5dtTsTaIctQt1pppHdDS+J+UnDJAdNY88FGWkWU+0ouO9lKCqFEkHo+0M7OvksOGdynYmB0SFWv7rtOkB2+Uv+p7OBomyEhG0um+pp4TXFhpUDk+w/vvbgniHIfQodg6bGvXapn6qvMdW8pexJAz1GbnX7WHbY2fzK97y0H3pt04L7rzcXMX7d3Y2LGaDdVrcBl1Uk+zj8CR63B+agY0HXr5ORRXJPAhPyYzHw4s2Ew/1cDI4H5/Eyw+h1RigvDReuqmpgW+RmNECtnFyfhEb5WQs7NVuc8SKIbxPuuMRGW4xGPU+cJi/Oj61+FwEQJJNqUbYnpgJ4qrXXTDrefs++eNO7etBeUr0dRGa/lOYTvaNZ2FbkhrIUg9wuPh9xJxB7o/2+wgrc0kP0DADOAs98EZaKWNfVFGUW1MYXtJUjuymFS7weQXX6eOTGV4bfqLk7i9bnEDRyHGKXydiGYD97zPS4bQpCJ9qwsl/xiHiuwGO/H2uEPIs/8p+ZCBImhKlafap1vMzR+Y1WeGv4C1rpBcU0Jts//ASQpUi9dAN0YurzpwlVVeOftkI/6mnVGYBQnYGrh1NNj/LLMwBBCFsWDKpk88ayk8Y0/QOa9SOpl2gisOF7ZINPIy1wAsDhEiv8wepYIaY8YlwTQTdDaVY8Ec4VCqkapwh364TYt5/jdYSmKMWpyuUrEtJzN9hSy/WasM+tK5rW1MIk3k5z4uRGdsqBWCYNFVmxMT0e3e2JceAjtTr+xW8qqKNI6xed9aj7yTRridTtXlFLJTCdFw1hcUtkTudxzy0YgMLLeVV/Z8s6Nps0JYurfWOEOsnXN2XGYj45Q97/SSeDLguggHxF2yqPD6NAQ/sqHQOw8zCFHqaVKs7qIyFwFWFyMgs9nKb4iHIK1Cf7pQ/j0FspFNelQrIg4ds+9Ad7qUJj3ZnwZg8GhTTKbzp4l5+YEkggG3TxtftqYBWsAFemkqzXa4Hf56oCRkA76mBxzFPFZ5rddU4vZd0kaA5/NsI1ygsQI= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 428ac945-9487-4c41-ed70-08da47fc8168 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:25.6220 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7UfeZi4kKIHq24wYNcDqc+nnP/L9mdcmxBB4Z1YQb/HMSOaiCVhgZoNwdMJQLhA8eV9pnhEmz1qzDQO6ujHBBINlhH5tjA6QSpDcMXJfOh4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: ddmajI0b9RWhluUJBhCtUE6uJti-FrxO X-Proofpoint-ORIG-GUID: ddmajI0b9RWhluUJBhCtUE6uJti-FrxO X-Stat-Signature: 158z1jbn6i9t8smp7ixoz96bqsk1yba9 X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=gP1S49aq; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Cfl+OMJV; spf=none (imf14.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 62F4210001A X-HE-Tag: 1654547924-78576 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add generic primitives for clear_user_pages_incoherent() and clear_page_make_coherent(). To ensure that callers don't mix accesses to different types of address_spaces, annotate clear_user_pages_incoherent() as taking an __incoherent pointer as argument. Also add clear_user_highpages_incoherent() which either calls clear_user_pages_incoherent() or falls back to clear_user_highpages() Signed-off-by: Ankur Arora --- Notes: clear_user_highpages_incoherent() operates on an __incoherent region and expects the caller to call clear_page_make_coherent(). It should, however be taking an __incoherent * as argument -- this it does not do because I couldn't see a clean way of doing that with highmem. Suggestions? include/asm-generic/clear_page.h | 21 +++++++++++++++++++++ include/linux/highmem.h | 23 +++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/include/asm-generic/clear_page.h b/include/asm-generic/clear_page.h index f827d661519c..0ebff70a60a9 100644 --- a/include/asm-generic/clear_page.h +++ b/include/asm-generic/clear_page.h @@ -16,6 +16,9 @@ #if defined(CONFIG_HIGHMEM) && defined(__HAVE_ARCH_CLEAR_USER_PAGES) #error CONFIG_HIGHMEM is incompatible with __HAVE_ARCH_CLEAR_USER_PAGES #endif +#if defined(CONFIG_HIGHMEM) && defined(__HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT) +#error CONFIG_HIGHMEM is incompatible with __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT +#endif #ifndef __HAVE_ARCH_CLEAR_USER_PAGES @@ -41,4 +44,22 @@ static inline void clear_user_pages(void *page, unsigned long vaddr, #define ARCH_MAX_CLEAR_PAGES (1 << ARCH_MAX_CLEAR_PAGES_ORDER) +#ifndef __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT +#ifndef __ASSEMBLY__ +/* + * Fallback path (via clear_user_pages()) if the architecture does not + * support incoherent clearing. + */ +static inline void clear_user_pages_incoherent(__incoherent void *page, + unsigned long vaddr, + struct page *pg, + unsigned int npages) +{ + clear_user_pages((__force void *)page, vaddr, pg, npages); +} + +static inline void clear_page_make_coherent(void) { } +#endif /* __ASSEMBLY__ */ +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT */ + #endif /* __ASM_GENERIC_CLEAR_PAGE_H */ diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 08781d7693e7..90179f623c3b 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -231,6 +231,29 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr, } #endif /* __HAVE_ARCH_CLEAR_USER_PAGES */ +#ifdef __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT +static inline void clear_user_highpages_incoherent(struct page *page, + unsigned long vaddr, + unsigned int npages) +{ + __incoherent void *addr = (__incoherent void *) page_address(page); + + clear_user_pages_incoherent(addr, vaddr, page, npages); +} +#else +static inline void clear_user_highpages_incoherent(struct page *page, + unsigned long vaddr, + unsigned int npages) +{ + /* + * We fallback to clear_user_highpages() for the CONFIG_HIGHMEM + * configs. + * For !CONFIG_HIGHMEM, this will get translated to clear_user_pages(). + */ + clear_user_highpages(page, vaddr, npages); +} +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT */ + #ifndef __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE /** * alloc_zeroed_user_highpage_movable - Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move From patchwork Mon Jun 6 20:37:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56594C43334 for ; Mon, 6 Jun 2022 20:38:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC3368D0008; Mon, 6 Jun 2022 16:38:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D728A8D0002; Mon, 6 Jun 2022 16:38:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC5C58D0008; Mon, 6 Jun 2022 16:38:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AA9B68D0002 for ; Mon, 6 Jun 2022 16:38:58 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 86DB135224 for ; Mon, 6 Jun 2022 20:38:58 +0000 (UTC) X-FDA: 79548975156.17.13C6326 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id A2796180036 for ; Mon, 6 Jun 2022 20:38:53 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256IH5p6010190; Mon, 6 Jun 2022 20:38:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=5msmGPhnYMpzXeIEVJ2L1ZIQB+VFf1s7vKE0VKeA+bU=; b=cYWtv5AQMwE4FfKlHDzfVxAKC7qlHXnZnnA0KgaFVKGaRxKsfRoa+rL6ChPlZpSAHUxa twJmnBiknV/VraZ4i2ZRTIaXX9CCrQxemKF1tq6/CSqc8RFT0mDdGnqwuzwGteMkVThQ WTW7X1H/KLyJ6yhVUG2TWNYi54xneUm1g/9seoxIOMPTxmgQa4n7rFTUIZAN5XMIIRUE TeKLYmibsDW31WZKf7S1HNPZ8JvgfdPeGY70SoXrLm2HnVACx9xXkbcacmKGYyYtItLX vhIdZMpFs+docj5W11uHw315U9vHs6O1XLAiRnppgtSYSqD+uYZyOg4PheX872JDQBpv CA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtfya-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:42 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFBbK001327; Mon, 6 Jun 2022 20:38:42 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2109.outbound.protection.outlook.com [104.47.55.109]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu91cum-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IF6NHWn7Yw4jNnwkGPf0zw9AAI6jW+r9KUCmOfxx0E3x+TCAD8z60UzffhB5MoD4hapMhlrB9zCowFyJmBiT4/qf2Ns9vOc/Q5mvQhTR24xIv5Lr7r3kAWE4uOGiejFI0wn1O16/LwtpCv6dKgL9O3kPHss6KfAFkkPjYWY+QDrUKMRSGU6ruBeOsJcosymz0BfKRyfceLuRlUqJp+9zsrZTeQw13sX46rna645SUTKZP2spcfFC3vIUvIt5wiuqB3sH/qU1EmwNsEeklAWcK2IECTCv/6q78IjJ+9U/U7LPXAoWDi0ANER9bnAdr/qy3PktS+w7BpLQfz8nC3gxcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5msmGPhnYMpzXeIEVJ2L1ZIQB+VFf1s7vKE0VKeA+bU=; b=Qds31ICh0CgIOG7l6WjsXzr1C/gJxdl8/tFsBpjAPWE6/DC0NUvNkMJd7t034uBPAq8K/EaWLqoA/HZl2fahHcica4WkaFUja0f7Q8ThWOiHTJ2vMg0XeugepDCHq43dn6fAiNOD/SLt2sNJphOyEyKnFp898WB/3zQlUb0X1SDSNb5lggmXQq/loZ7SwY00mbzoq2KIjxtbPtKvgl3XSkhhhJl2cR2mRoLr3XarCXXo1xXmacHROAdPF9K7f16R17Ne1ImdWC7j1R5+VG3+SlBI/G447O69NIJeR3WoAZin4R6NeHyW7KrBzOuvNi1lDtiK5/4tdVrc6SOVxl/JtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5msmGPhnYMpzXeIEVJ2L1ZIQB+VFf1s7vKE0VKeA+bU=; b=fsZ8L/SRA7Cqa9nXATo3nh/qqiBxRmEJzY3rCQc4ctB0W70cH0OppbEzDSHlG/8/3HU2itnSzCMXUnZO9mEJHSga4M4uRA6smoYxNKb4GNlAnOMykdJjysYM94KhDoGIv1hjf/cGGvWMp/5bUmfbp4//KedeL20bFC+VFSFGAdA= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:40 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:40 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 14/21] x86/clear_page: add clear_pages_incoherent() Date: Mon, 6 Jun 2022 20:37:18 +0000 Message-Id: <20220606203725.1313715-10-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0016.prod.exchangelabs.com (2603:10b6:a02:80::29) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 700ea80c-dcf1-48e0-d256-08da47fc89ec X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IkHUdo+6QvnZeOboqCop7CpzCaQl4JIr4UOoKECe+B/xgP6SAkbhm7n+ZQLeAig/nOlb/7E+kbvmAv4osVxP2f+nxdQBDsChp41kNF6/xgo6CvjA29YIpOzpN3ghVu8SP/hb5Hdn3fdY2aL1G+/yrxWfuBR2DcHwsUBwqy62jaHz1QG5OXeTdEPMY6lI0VkHDcdM4XjyS3wDqNp5I44EKhH8s0nOMxXldteDjk7gxOL8NMcl8Q5s184HmWqQnRLax3k23cn4wjSEimmz0d4tO+UAKqUe+J1vwkuJmZpv/5Q8w1kcVDh6+dAb4isIu9Nu1apN+NjBNavwHDGilkfWaf+BO+m826W5UcR7l6UKqfn11SuCHqAnwSyKyzqBjTDVRCVcfz6sKxVQUymokWMnjRnDAwCgudUFV/tLrnqq24pYYPS5Y/YK2Td2ZEGjZKB+ja67M9yshlt5YpZ/IH2E3gDBAY+DahjdK+/TkbPtFTo3PC/xplrxalxT1FVJGCGuhdCec0LQG/rjf9bn4CsqAFjBqn6xH/vWkDxcqc/uNL3lh7EeX6pHge6BioVYHQyddngqVmmBvKwWUKcMT8dADI4N1BmMqy1DINJYcDGl9YkkNQK3VmdnCzVI6OMHApbH63c2DT33rGcfpAjP8+Zg9n+vUrvYaqYOQBIuzHEXPYdwz0jkLskvXuw7vGuIckk+fB84TbJ76RE/t3853sOXCA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9TUNT//ISlVbLVcWbXI9kQSaU1mig1mW4CthFRIKhVltPigKe71+ialwW5TK0VDIIwtX80KxJLdJaup9u6SpzCHGkODbXVnrUvvSyd6Lpjz4VviTRY5A0z1ZU36c5X2s17JJOxLLpWVaVfYTDfTn5YtoitMWHMZa+GzY8jBKu2BUfhPN6+Kss9INQcsAmR3dhLym8Zcwm/SH0wKR6Yxoq9Dlw3h4rqLE4cvlaft7VBqBIne1S/8j5l54qb5FfiNnNE7QVebHrJyHiZTy+dkPV4BPXiSaM5ftOIlV5nIKm1yQ5ilKCR63P4eK7EBx0y6i9AI9kd24+3I1IPS9QCg7Y8UnIimWbIryoMJIPI4n+Gbem/LApPMTSfGBBbDYEV/VvpQiq6bwBACyP5T5P1yxazcjKLNjPongGbbQj7xj5UpKZjXKgAg4y04La5dWNzwo9Qa4Cj4cxv2lUaGXOKuOB06agyeVtcz085tdhj/87PblnC7vv/sR+1zd7ECjGQvEWqcbMnupUiVTsFzFUtW5F2jfcLeX88W0vZAKqcqdz3dYKKqsql4dzECZlVfGg5tFHHE6AvySDz/f4MHGv+hGzJX9GZCjWKVnAr2Gu1G+fcgBDJ4C2oy7HMpE88o2mJcDTeUjLQq2DkkZzf2y2JNGo1/dq2gfOpqLJgmnBuQSyfR3tP3Pzs0D/pbkrLdN+0Tac30U1SqLz21KcuIuXkU1kjsSRh7y8FNhVOFY7CtGWAr+nrZtYVFQZXL84aN5kbq7blcdlNabG8ObTAWk0EvJjzdkEJ4M7s8/kFWEGUfIW3oqRaoryhxH3iQEvoAKLYZzBpGV8s37/B91P6qCEcp/JuoeafsBkJdwjwhiyQqMJWS2KbitN7e/UFG/12nKqKa51j/kclZCGJBD3l1M+jUl2DP2ovqC4GIcpgCLl4r+4xEBSC+EKgf/xFp2HiJeYF27EcFjPJVSwki5FjKIMAsTMLzERd8pyU7tn0/mDLQzficfHnk4sFoQrEPD/hcvKSTgtbF8SGE/sAMYZPmQts9OcAGHY0dIKthk2kvncy1HAorJI1PX2D3A8ZTXRC1j7OBN9TZfXN0ZoeNYloOsHJenyu05RsaiqUhg2U5ERFRfHndxHPX7AbeQPbq+NzbSRqXZWNnIc0Y9xrDCkwAgF1r2hIIeKx1aTuC/TaPr2Pvd8KmCBKTMrCyULVOkDEsnj8FVeK82EqXZhLIXu/YkA6PcUByvHMB4qrWCKcW5m09O67A/QRB/g+JviODs0Yql0Xbal6IpN9JF5lvc3z2aWnFdcIMl4syY/oMcj0UNuwKyr0tVWBRm2rOMzhhWJ8MzolSb05xMHyj2mX7djsIM8HbFV5SiPn9my5PGmL2b1AQaTJriU6hGO0KsfL9HWQ2S8m8NL8sI+QKoX+QHMefynzD9UDRhA747n08jBLf3R2lmuIiHdzYaTtJQHOIClfAZEViqP/ewn2czsBiVewjpcARiqXA8INvgCJb18SD3+782yHoQTtD95kVo3W04jaeV+q1S+asQuGs9xI5Qfl8laHK0xakSLiWTCg9hKeecdU44vKieWXDjpQHkcpSSF80lVNefVHeOX2lATdykjIgdGK474ot+hc7+8l91BYBpp3upbsGxpF7aDO0ttHI1n7ALEy5FzLqvwe2bsPAcka4ePSwibsv7i5D9qqxm9/b9spcWU7OUETE56uXnO+y/hYHV7LK0+q831C2GmEkYW1le8Tu00cyjnGpVBYmLHDlOtkqe0G0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 700ea80c-dcf1-48e0-d256-08da47fc89ec X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:39.9863 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Dr5OVaZkqmZfCzObBwp9zC0UUXsCZVstX6w6IiJ33eg5W2MczgQOBSQBl9kIeq2SkOQn78jMYisyPKcQTVpVWBSn54ZgNr3rGvwZFQiX24I= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: mgRmUiLOFcvQLp-U8vHLaSwJPE3wULpV X-Proofpoint-ORIG-GUID: mgRmUiLOFcvQLp-U8vHLaSwJPE3wULpV X-Stat-Signature: ix8gneu5u9gminpdqfafn1gfedqyp79s Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=cYWtv5AQ; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="fsZ8L/SR"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf06.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A2796180036 X-HE-Tag: 1654547933-536920 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Expose incoherent clearing primitives (clear_pages_movnt(), clear_pages_clzero()) as alternatives via clear_pages_incoherent(). Fallback to clear_pages() if, X86_FEATURE_MOVNT_SLOW is set and the CPU does not have X86_FEATURE_CLZERO. Both these primitives use weakly-ordered stores. To ensure that callers don't mix accesses to different types of address_spaces, annotate clear_user_pages_incoherent(), and clear_pages_incoherent() as taking __incoherent pointers as arguments. Also add clear_page_make_coherent() which provides the necessary store fence to make access to these __incoherent regions safe. Signed-off-by: Ankur Arora --- arch/x86/include/asm/page.h | 13 +++++++++++++ arch/x86/include/asm/page_64.h | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 045eaab08f43..8fc6cc6759b9 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -40,6 +40,19 @@ static inline void clear_user_page(void *page, unsigned long vaddr, clear_page(page); } +#ifdef __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT /* x86_64 */ +/* + * clear_pages_incoherent: valid on only __incoherent memory regions. + */ +static inline void clear_user_pages_incoherent(__incoherent void *page, + unsigned long vaddr, + struct page *pg, + unsigned int npages) +{ + clear_pages_incoherent(page, npages); +} +#endif /* __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT */ + static inline void copy_user_page(void *to, void *from, unsigned long vaddr, struct page *topage) { diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index e8d4698fda65..78417f63f522 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -69,6 +69,40 @@ static inline void clear_pages(void *page, unsigned int npages) : "cc", "memory", "rax", "rcx"); } +#define __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT +/* + * clear_pages_incoherent: only allowed on __incoherent memory regions. + */ +static inline void clear_pages_incoherent(__incoherent void *page, + unsigned int npages) +{ + alternative_call_2(clear_pages_movnt, + clear_pages, X86_FEATURE_MOVNT_SLOW, + clear_pages_clzero, X86_FEATURE_CLZERO, + "=D" (page), "S" ((unsigned long) npages), + "0" (page) + : "cc", "memory", "rax", "rcx"); +} + +/* + * clear_page_make_coherent: execute the necessary store fence + * after which __incoherent regions can be safely accessed. + */ +static inline void clear_page_make_coherent(void) +{ + /* + * Keep the sfence for oldinstr and clzero separate to guard against + * the possibility that a CPU has both X86_FEATURE_MOVNT_SLOW and + * X86_FEATURE_CLZERO. + * + * The alternatives need to be in the same order as the ones + * in clear_pages_incoherent(). + */ + alternative_2("sfence", + "", X86_FEATURE_MOVNT_SLOW, + "sfence", X86_FEATURE_CLZERO); +} + void copy_page(void *to, void *from); #ifdef CONFIG_X86_5LEVEL From patchwork Mon Jun 6 20:37:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C31FC433EF for ; Mon, 6 Jun 2022 20:39:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C7A68D0009; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 275CA8D0002; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F04C8D0009; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EE8818D0002 for ; Mon, 6 Jun 2022 16:39:03 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id CF9C560A50 for ; Mon, 6 Jun 2022 20:39:03 +0000 (UTC) X-FDA: 79548975366.22.0DC6941 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf27.hostedemail.com (Postfix) with ESMTP id 0008F4005C for ; Mon, 6 Jun 2022 20:38:58 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KC7oI032448; Mon, 6 Jun 2022 20:38:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=sf8L2jPN0S+qcGllsHo55AVeANBqjQC5Q48G9fkQtbQI2IR9KyFH0GcYu654peq00Ctd mUMvzXYX25e7onDYvk2hIwfMMcFKYQXMaLkesHmuL6kNo88vPSO9qkhwVo4lEI0E2ZQi 5RraKegRbBGwna20VjoKLGnO/yC+yNh8/LCkZg9PXcxmGO7KfeEHbgu4OIkZY4rncWYr Qu17QDVFGULEo73tYoC7gMdfy7Nx9ImzGzHSf1EnQdepiiWl5bIO7UiD8RIVRMnL7lgs 1CdRqyh0cZIVKq64exhifKGWQxiucNQjJyu8XJf+2wUhNV6ReHCRZFwsBiRhEkoR5TbG nw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghexe9gfq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:45 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFBbL001327; Mon, 6 Jun 2022 20:38:44 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2104.outbound.protection.outlook.com [104.47.55.104]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu91cv3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cD3IRHRoI4wKMmegAhx/yZE0QjkKLxSzGk7dUJ+5Ml8FgsboM5aW7SSPD6AO/AIndyVL4twTFOq5yGBv3nMNY5hIvSBSTYQrhMwgrQzCVfZupbEp+oIut0hClntgBkF9yq8ylihNBVs4vwY3SnjIiv+io/WDGV4hmn0M3jEzJQeFh3v6kJxfXMgfJ/ilF/1SRY+deeDlUoJUqxfI24swgOhwy9wVkofbx8Rxp13kT6vZp8ghlbV9pTMvBQEW4QCVH8OBxKD89Dt84agyQULF71rT3YUijgQ086WYIBQXA3RZlYgc3LDj4t8v/+NwoQtuHXBH5jxUPjlrsNXn0nDffw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=bEGY+LLjovOnM/8sIt/pas3QWdarhy/EyZ+K5unMz3ehaJrjoGA9TlSXIGPYeWS1441wBeut/5PrUBZ9awFri2vTxDUQb2/AfjEfOnUOh/LOadgNm8LjaffV3l4/FPqJrIWnWoHQ96CiIp/rwY4ZJa2Ek+1EgvzktyOsXPjBAW8Dff+3cjDPcgF+dhn10mQuOl2QyIiLJ2eeb9RqLzn87NWl2q9yQVkZHUqTCCgb6Dic2bHwgy+SnkuPcrCqz3Kgg5N+jXG0+AvyepLNQPL/aX7BY5ZePAlZeJCm0ESRdcA3EuwOiTh7oqedo/PEMGe5UW98Wj/whEEiUUPe1OuyqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=a45EdSDh0pZXgt5jaG5yKucGm2VLPTjFRWC1i6UZo+RdDZC8BUQIkvoVnkCsYEuWj2cq8n7tMddVweAi0gn5oouuTObPG2YEn9RAV02GlkhSZP10SSyh27SQ0G17tyeC7Sks167tGqFFvISV7+Wg1C1VaVwU264JOMe5bCk0IYc= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:42 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:42 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 15/21] mm/clear_page: add clear_page_non_caching_threshold() Date: Mon, 6 Jun 2022 20:37:19 +0000 Message-Id: <20220606203725.1313715-11-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0016.prod.exchangelabs.com (2603:10b6:a02:80::29) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4d5116e2-1d14-4430-1031-08da47fc8b80 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YzsBHFeVXkvQRRmznNMBorsNgAIPb6kSSJQGxZqUkmRXY26Dqni/YK+QLIxfZXc9sclvkEuMRdd9hX8/8ddVM3DnZLyQVl/NjCu6HMZmQ/gavPdpgtnXeN3ZF8vQq29nxn1X2eOIsiFdVaQm7VWArXv5BbiVf85++t4Q5MfT6sZev47z7BpBgjhcHLC7oqc09nIw5M5UJGvSuzNxvtVImbptyOS/BJOJzIk8vh/TF1t5S9IKL8ub8a9yaBwwCg51bOALsCecth9LHT/SuSEfow1XATz/rd7BGVv6qLFPeVqVG/WJRc8D75GSXvcIBSgYwnEF2FW318nbfNYRqU5NqPHhdF/LNR5DOuW+OnTJP/s2ahTmIbDAbtN4S09kF6iXNovV3DpBZkG6ILnU8eeqmgAw6Xa34wxY1PnRHn6SYkrXCetcacwAySwz77/uGUqUWqGa4/0tE/PIl+J1wCl4iS1GHCRw0WvH+dxQlAbaz2RAHhhte1k9DgZuSeLIRTyFooSH/UQoaNBp0XKIwjBZBXz7HfFgkRXwGRJ5gPgU5X0cUJr+T6f+gDm4/+x9p9XOm161TIMnSnnBCnrw9ZzdaFSTemPMBdj/wiVIaMb1yRFGPKMIG58uYZbTV/Hoig6Kap6pcpY4Jzwtkj9Zjh2hWu7F55Az2axJdyakXgvOJiGmry0byjdqK7GuT6wulw3mdPYJ7gTKXSxmaB26ThD2nw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QoJeFBYdwr+6StVXjjP+05MLL/ah1YBFGVZtJ+d1l38tRYYCb4myENnmq/EQSlizsCs9m1ulgB/4ulAKKrHHgA40jTqZQXamdJC/LWEu7bKzhnAtH+cz5+8bDqxwOinKqesde9jycwFgfIPE7q87WQpBbsHFP8U1RRHqbTjIXhvN1k08UEXdQD7oY32YBqbgLiqGM4kwdB4LmCQaHqi2qRuvniM2govKNmSqYABHK9Y6Pqa9baTut4oT8swe0Dt1+pOxwWIVP0MeXr5QXiQP1BhY6uqF5F3vzJbOKFa2PLYx/QFsZc3E8+MCGuhUGhKRl66OP7Fh0sg+LRsPGsqLyobj7XO32XOoJ01xzCQF1VFoKp5wOqDDKTf6uWAFG1VHN3Rj2dC1zF/1eoc3zTeqtWwD8ZLF6wyL9BKTlFEkZbTyHmY6lJ7K5/vPtMND0dSRPtYTB8+etS+se2fiSKZNbcJSpHn9V/SepC1olRK6q9KZ8GC2PHDYC/h3dAAApUCjxH1HOYDvm9ZNSO4N4mlYKHdowI3S1k6bUJpYmbDDeCzk8EwcVccqQ1iEbDI7yhqvUkagrWeb9q5KxftQraJbyrRbMqGXZ+VuLV2PxdmVkphgrknVG6+8r2KF51nbaRj/iIaRrqJYsoJt/YdRoVYuA8058oDTMg06R4VKFhuoTM2jPhBaP0Ma2iNtMoymBGM0pmAUzOTsZz5CW2+RL9rt13U1R66p9ok94YIydeK839OIhX7o1ADaLodv/JuJ8lcoLW8mL39x7glJDlU9+925vBd3IBlljXfchpFs7fDA+9xaRdR2hPITjVFDiCtm/NX4vezXrSJLZdNm6mJiRsSy8kYsXa4A0W4mEgt1S1/XXDS9E4sAk5lDAF16eG0xbxKsjyJHMkP7zPdfHS9CqLHCwY52pxfRfQl6C6JFZm7Vq21FvJx7/KAhcvRw7KNj4eE50xoFIvI+Gxr8kd2Pp9gAGsm2/3KSy2+1vIZe/m0o5EywWODaONMDUaA7bbfNV89P/IpQXmloYAj8eR3+b1kvNcuvnK+VYK9TqFCkakqxqyMZVSX6N9UrnP9sta7boCaZQzD2D0wloDoB703Qw+Q0+Q7EgQcM+XnbUBfbyr/8ujVOFqxi/x1dVnQTrPli3kp4eNvWjgrLo3yu80nANVsB9tO9mY1vOBxqF+b8uWix8F7mroIGNBkTz/T4amoF0Dv155pXVn5YEfzg1+gMUu4Auedn4WVe+l5umbvqr0k+pcgvDHWD7zic68MHnnRPH7EDQ7n7w7QtExZUrO/3tiY5TzR0TwhqOOYtBPxbEXus3l1kYNYkoiyuHn8YRBwE43xArcQ2RE48TBZwTDEIAAuefeXWr/JaL1+r2nCKsuRQpncsmDarCm8I7T3CQCHiISE8ofqGCY4r5VANANxaLX9MdmvhqGLxiPeDr/U3XRpOy/gRL68Scyu0bcSw1ccL5Y0MOWHK9jfvsRp6r+efRdikgOVehxavKqFJ3skL0zfEHnBYnC/kyM/qWYGBiTFZUQie/ugtuzZI1ZgWSvALqslDitFPKN/CiKhkX+uZD9aDRsurmJVe1TGiLlPOO5+TY8ukNGbf3oxBjc5gkqx7MJ+fmtLOcQO/qiqLZ4QlEZA+hb/n/iuEjdrnl/9Uti3pMaLlITx1HWJHdVoAkUYglxJzdOQ40HccRJir8qtobqTNeDD29PEd63bytlWadYoZl9akSh93p98TqiriV4mYMOBb4R9m66G6wDqSZBNzwIyX5KA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d5116e2-1d14-4430-1031-08da47fc8b80 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:42.5352 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FWb+RWeYbgDQnonL6o4cNdq8JO5cYxgxpyQpwNKHe+ezdEHMHGr0dakaDrBYU6iymu/wa/QD40YllAjPs5S5fzhQYmsdfU7l3/TSuiVeOu4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: KGXfBLqyjjy9W3PpTADtgg0MsBKYboGE X-Proofpoint-ORIG-GUID: KGXfBLqyjjy9W3PpTADtgg0MsBKYboGE X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=sf8L2jPN; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=a45EdSDh; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf27.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 0008F4005C X-Stat-Signature: o3sfnqxuistmxwfmq7tyz3kq5u9h89wr X-HE-Tag: 1654547938-587765 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce clear_page_non_caching_threshold_pages which specifies the threshold above which clear_page_incoherent() is used. The ideal threshold value depends on the CPU uarch and where the performance curves for cached and non-cached stores intersect. Typically this would depend on microarchitectural details and the LLC-size. Here, we arbitrarily choose a default value of 8MB (CLEAR_PAGE_NON_CACHING_THRESHOLD), a reasonably large LLC. Also define clear_page_prefer_incoherent() which provides the interface for querying this. Signed-off-by: Ankur Arora --- include/asm-generic/clear_page.h | 4 ++++ include/linux/mm.h | 6 ++++++ mm/memory.c | 25 +++++++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/include/asm-generic/clear_page.h b/include/asm-generic/clear_page.h index 0ebff70a60a9..b790000661ce 100644 --- a/include/asm-generic/clear_page.h +++ b/include/asm-generic/clear_page.h @@ -62,4 +62,8 @@ static inline void clear_page_make_coherent(void) { } #endif /* __ASSEMBLY__ */ #endif /* __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT */ +#ifndef __ASSEMBLY__ +extern unsigned long __init arch_clear_page_non_caching_threshold(void); +#endif + #endif /* __ASM_GENERIC_CLEAR_PAGE_H */ diff --git a/include/linux/mm.h b/include/linux/mm.h index bc8f326be0ce..5084571b2fb6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3328,6 +3328,12 @@ static inline bool vma_is_special_huge(const struct vm_area_struct *vma) (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))); } +extern bool clear_page_prefer_non_caching(unsigned long extent); +#else /* !(CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS) */ +static inline bool clear_page_prefer_non_caching(unsigned long extent) +{ + return false; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ #ifdef CONFIG_DEBUG_PAGEALLOC diff --git a/mm/memory.c b/mm/memory.c index 04c6bb5d75f6..b78b32a3e915 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5563,10 +5563,28 @@ EXPORT_SYMBOL(__might_fault); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +/* + * Default size beyond which huge page clearing uses the non-caching + * path. Size it for a reasonable sized LLC. + */ +#define CLEAR_PAGE_NON_CACHING_THRESHOLD (8 << 20) static unsigned int __ro_after_init clear_page_unit = 1; + +static unsigned long __read_mostly clear_page_non_caching_threshold_pages = + CLEAR_PAGE_NON_CACHING_THRESHOLD / PAGE_SIZE; + +/* Arch code can override for a machine specific value. */ +unsigned long __weak __init arch_clear_page_non_caching_threshold(void) +{ + return CLEAR_PAGE_NON_CACHING_THRESHOLD; +} + static int __init setup_clear_page_params(void) { clear_page_unit = 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER); + + clear_page_non_caching_threshold_pages = + arch_clear_page_non_caching_threshold() / PAGE_SIZE; return 0; } @@ -5576,6 +5594,13 @@ static int __init setup_clear_page_params(void) */ late_initcall(setup_clear_page_params); +bool clear_page_prefer_non_caching(unsigned long extent) +{ + unsigned long pages = extent / PAGE_SIZE; + + return pages >= clear_page_non_caching_threshold_pages; +} + /* * Clear a page extent. * From patchwork Mon Jun 6 20:37:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 039BDC433EF for ; Mon, 6 Jun 2022 20:39:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 908D38D000A; Mon, 6 Jun 2022 16:39:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B7518D0002; Mon, 6 Jun 2022 16:39:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732228D000A; Mon, 6 Jun 2022 16:39:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5FF9F8D0002 for ; Mon, 6 Jun 2022 16:39:07 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 39E1180924 for ; Mon, 6 Jun 2022 20:39:07 +0000 (UTC) X-FDA: 79548975534.19.5C86D13 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf29.hostedemail.com (Postfix) with ESMTP id DDB76120068 for ; Mon, 6 Jun 2022 20:38:51 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256JgLu8032275; Mon, 6 Jun 2022 20:38:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=fYmmVnaSVN/2hFPfnmGyw4rMWEDLHhxpyLeVlQ1iypM=; b=luZGfsX8i+CJ/WNZWgKSNMXfrS/9ZkW6BQx8Qs0xBtUYNGl3zOhMGgKwPRff+mAEFQyw uRs+aadfyMCiz0057Ar6denNeZkcQomBfnHzXAMxupsEJtwlH6JgxOuim5hPQzKEBZKA 91IaU38gcftDQa1i1/7oor308PIuVXA0IUP4wgUbSI+frgF1YYPRC+zscCo/IED0xwMG ZxSYAqDI2FxxgmOgEfyNZ4QyFGotGmlTGMrDKqhuoGqm2i6mnx+y8ZFlJdJpIO0VMPZ3 mCHNhYajdEQd/Q2gQzYbl9BXfzPg9IcLeotp9Dt/RWF5uH9II0MUG9oxXYdp7ke+E7Qm Og== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gfyxsc6k8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:48 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFkOw015917; Mon, 6 Jun 2022 20:38:47 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2108.outbound.protection.outlook.com [104.47.55.108]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu80a9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HVH+DOxnJ5bIDIJ3NblkxL8DAMRaolvjfuWOwAsiQP1OL4f6ji47lPh6601SMxAY/nZmeB+e9kaQnNwmMbd2npwTfkeFBmD1nxOyT2bkVssF5EnaOL47xEwLjyguaT3VrWh/YgvhcA8XcdU136ORUUv5xLqEAQ/j/JpD9LHtkn6rly9Pl7DkMO0bbO+Bfk3YMSWDfMjTRpOKgg+04YwkbdJmc0Hlm62WkhY9Spmjkkr5w8BgUAL9rb8C49+Td8ICvqQg4KGde1grn3oC+7QfZQ5yOoVAJUzZljjX/nOptlbAFg/glMJjoB3ODo8d3pP96Askeo70YYyctPRwYBvFgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fYmmVnaSVN/2hFPfnmGyw4rMWEDLHhxpyLeVlQ1iypM=; b=W056rwGL4lFJJmBKU9+FiDaieVXcWsv79We/vkfIZM0EA8cs1lUqF90i/AUS2eX3kZNJcLmYZ96QWh5gLoPX33c0BUikbSTeEtqZMyn9y4vQy3YlZWwtp81Ct7FQle66n6Gl8SJ/m1SqDAo9aS3VmhvrU53Wb1bCwxoBK0ikl7P+zoFnlKKBIM8Y1Em5aNCWV8YVDBuVzuyAo/hZzKhkAXDRFPD+WQYyyb9GU3PI7mDGT+BoyxX0v4do7LSGjYtgGfq+KkAxsGE39AecqysVM0RziPoyjFiWJwhgDwe5Im61C3T4AW02GjLbGoEHkUFgeDIcevkXzVWn92+CJiPPYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fYmmVnaSVN/2hFPfnmGyw4rMWEDLHhxpyLeVlQ1iypM=; b=cPQl99u/eJ2elBh8pTo+0AdxBuQ/bxM0B+OwqWGFI32SId/V2rPgMeVhBZpepyrmfDBgtu+fr0z+TBreTqLPGku/C+7MgHB3IkWdXN/iKI9eGbgA9AIiwy2XMl6sZEJWPvOrYUe+Wg+yaDmFC+aTp+wGHsqBRxlcn7+VdVlnIgw= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:45 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:45 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 16/21] x86/clear_page: add arch_clear_page_non_caching_threshold() Date: Mon, 6 Jun 2022 20:37:20 +0000 Message-Id: <20220606203725.1313715-12-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0016.prod.exchangelabs.com (2603:10b6:a02:80::29) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 11f4a2ea-7de7-4799-5f4b-08da47fc8d07 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 56A3Iuix3qbn8epfGKllSCn6VCjcOCtUARO3sXsax+xMLk3PQrag0jd4nED33vFeAUSD6KVLvSZ3quaibbDxPMlAabxGqg7RhmoMJCYIouO6huy+lkYO/+qgVB/hQD2362QBE1z+G6zR/HuNkLBcykO8QTD0gs993qhJpoG8Wny/3uPfsOmFoGYgETXNa11lQkej8YlNEDcAXc+1m4iJRk8gjIXzE+enHCoc4PPVjeM2GPlynLUrFQcedsypjqg5mR2d+iPdqxDzDIy05HoJ6VysnxQWT3Ge5l2Ran4dQ/ZSVMsCiWJiEfo5VLUYQWFkdBiOZaJHAEZV5AJGUPaieVj7Eqw6dghMrGsOrF/AXF2nhBo00JxK2CxUMiIWpJOI9urQbBIi+yxUUu8fMjyQeMZ7tND45KYt8jKRIfJEAYRpKs15BhRpUkOpdzoL0cxv2+g/4e1Nk6r9j9mkG2PMFFI4N8x8QVHQRmiToOj9UN0WWNxBDsdMNeYB3LWsnik2aKWC7d9R4QvRTMn1uGWCGtO7yKoXo1cLHft3wLqa1V3TwWPQtx18Txxh4hnfXVbKEO82r6pjpAvFNdBh5Av8sTcGbsCke7VR+QCsyDu0l702ACSsQ8zCvHupTb0KdUs3iHloa1+dabkZ7iqelpAgFvyNd7oVWAz2xZcWnOKB5/y5JYAxYuvc76RXSPC4oyBxBMoSdPiZ7g77UMiDt1PcTA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: REJDy9VIA8kxa1DNULchMa/g+t59pHpLwzy62XpvUNZEDp+GJvY9Qql5e9CF0kiwh5fXyg8QE0pi4nFxzrygi4xY/QJEagyePlsmIfcSM6wwR5h6KTQZLuEKxyjymsTOHIw+7e/+xhwqnRolT5E6Iakp55YOklVQT4BhQPqycC/F6SchcKEvKFDpy4H3gmjGuaxEHyLwhMozF9XK2qrp6kymWKqAkbuK4Hw+72VkouX06wzSqro0g42ZuzLeAcIH4+ypB16IU4FTuDybsM9h+BCGrHv5akyi6Zab756Dv7Oc/wn7f4eo9usSYgAgpyi4wJk5ydZSjY6tygiflZ4an0s32qv3cTVU6uPy4nc9rSzZhGPvhtUzUH2Myzev7AzTEdl7wleQmV5AFOUFNWVeQlMSYR69ERashcV/9l7utmq/7GQjesrBT4vu8+HqINlEl6WmkYU0VjSkn+jJ8z7bZyVOT19NAoew6Rv/FJ+QKpgZwAAuR8io5xjuTvkjuVelpuGkeWZahWTikc1lFTV0XwwJ7gQb0LtyYm3fT0paN1ZdKYzXdPiHu7UPMdq8ZvABPx2vGgUODC3oDhR807ZkmQC8q4AFYMcE4nXITMLHArZ3Y3tnFPz8tux1hrT3u8BBfnPwPObww1BJUGb5aFB8fYl+wvjVCzxAani6/4cuwm+wsrgZI1gSZfsDprcMfa4bRl3bvEkZ5Z5CsLI52tdS+oNavrKPxEjwMq5M2MU9LZJopRYeBRs+1O1jhJvNOhyAkLK6ZAMGobTnZOfqkEDGiM+qgH89b9rLVsIuVherzHOvtcmp7U1xwwtL80iQha9eW14N/TGGnay+Izsz4RIIXckH+LsR+bP8PxBX7u5ZJFm5MLh9F43kGWTf1WJrlT/ERJofv8whyjrsBjHHQqFdsp5ueJ5ND9d8zG2CHea8K635InMWowi/LTtPfXZSajawKYaEEtsRe59XfMVzu5fYfK26r5WS/UcqFPhvJSCRWyDCBmfVpAfNg1jwUwyGftaQuCKcpcYs51KskWx2nP9xpVJZFYbfkCDjrqxIq+GLf4vgiXBPkaLUcyGXL73SbxTd5KSgfdrR3ebE3EFiWYkzWGxqGkb5KivmiygctJbrOeN22kp0f2Cebat/GOdzza/OqGPxtPFHF1w+2fruDLQU7JSuv3S61xgkTZ1qRKzixlUjm5xo7w5pMVlQpiuhSh8+8aw+vOnIEh+LNslu4R4ve7me89jWgNDAZ+e7T3N2UFH7QX/bX0KrQg0N16GiXw1KZUMoO1H8Kfl1jWUIDLSPYezO4uZftUssfjJvEhFP+w0x5S5eDQCyHqW5Neqs8XcUsAil/z5hyrHE0qZXGTvlWIRQkAynqAAJWB/YV/tX4dkq3u9OdTDMnuxfOw36zQR1N18GvhU7GDTOaIaMCOjSuDgZL95RoYBa+yLBfRK0Ih8rmF9YDiqb4ltYLYeRi+u0N3UTcr3+H4FaPnfqa6spTjIdB3CHYWNm0ZzRrxVqFNTld0PbEqwCcPDPL3MgKotCUfm0/dpkluKb6eJmOoIFxXUTnWhGzbcUvAjQxFu08yKGvD+oecClcQmLOa+cFnmrFwFXf2S8yh8/7lk9Sl+lRo3/HCVYRIA8fkPAssj6GzXsq5VWoCp3CvD87SBhRPCSRcj6nx2XEyjmgKzjWkTiMAm8ABuqAe+jh/4nFeUo468MW5wFBjuCvBfYHI4AHRfrbUEFhydfgiq347kGNULE75cAbn/HSiveqOz86j+7jbY= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 11f4a2ea-7de7-4799-5f4b-08da47fc8d07 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:45.0841 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: t7oe/4j3G9Jq1oDjdG2ks1SYekl4AZxr3KP5aFBtiCeFK2vjLBQ8Ysejy7WcUjK8fmsfkx5Gb418FaPB+sjYYQdpH5lc9nagIYVu2JGWrFo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: bFyncl2rKsmgwuWKnXmiTtNg13Gu6Q1D X-Proofpoint-GUID: bFyncl2rKsmgwuWKnXmiTtNg13Gu6Q1D X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: DDB76120068 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=luZGfsX8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="cPQl99u/"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf29.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Stat-Signature: 135dpsuhy81hqpdp9s3mfh5o3mha7g84 X-Rspam-User: X-HE-Tag: 1654547931-469990 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add arch_clear_page_non_caching_threshold() for a machine specific value above which clear_page_incoherent() would be used. The ideal threshold value depends on the CPU model and where the performance curves for caching and non-caching stores intersect. A safe value is LLC-size, so we use that of the boot_cpu. Signed-off-by: Ankur Arora --- arch/x86/include/asm/cacheinfo.h | 1 + arch/x86/kernel/cpu/cacheinfo.c | 13 +++++++++++++ arch/x86/kernel/setup.c | 6 ++++++ 3 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cacheinfo.h index 86b2e0dcc4bf..5c6045699e94 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -4,5 +4,6 @@ void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu); void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, int cpu); +int cacheinfo_lookup_max_size(int cpu); #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c index fe98a1465be6..6fb0cb868099 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -1034,3 +1034,16 @@ int populate_cache_leaves(unsigned int cpu) return 0; } + +int cacheinfo_lookup_max_size(int cpu) +{ + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + struct cacheinfo *this_leaf = this_cpu_ci->info_list; + struct cacheinfo *max_leaf; + + /* + * Assume that cache sizes always increase with level. + */ + max_leaf = this_leaf + this_cpu_ci->num_leaves - 1; + return max_leaf->size; +} diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 249981bf3d8a..701825a22863 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -50,6 +50,7 @@ #include #include #include +#include #include /* @@ -1293,3 +1294,8 @@ static int __init register_kernel_offset_dumper(void) return 0; } __initcall(register_kernel_offset_dumper); + +unsigned long __init arch_clear_page_non_caching_threshold(void) +{ + return cacheinfo_lookup_max_size(0); +} From patchwork Mon Jun 6 20:37:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79463C43334 for ; Mon, 6 Jun 2022 20:42:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0CD776B0073; Mon, 6 Jun 2022 16:42:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 053EB6B0078; Mon, 6 Jun 2022 16:42:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9AF86B007B; Mon, 6 Jun 2022 16:42:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C449A6B0073 for ; Mon, 6 Jun 2022 16:42:49 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id AB732809F9 for ; Mon, 6 Jun 2022 20:42:49 +0000 (UTC) X-FDA: 79548984858.16.F9D4A22 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf15.hostedemail.com (Postfix) with ESMTP id E8E3DA0005 for ; Mon, 6 Jun 2022 20:42:22 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KC7oX032448; Mon, 6 Jun 2022 20:42:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=8mBclENT70Amq1G/+YFQ5vVgG1jGKH43y17kCMnkpHg=; b=ssKiQm0+jLmxy7t0irrPapbTKT7NRqoyKAQj4O30Rx0Jk0AScwuceucIPqbmDvT+If88 UXnx93iKMuu1LBai1zfSAXlIS4eCn6kRZO2VrymARX5qhFCiqr5ysH/k73ha1SGwsUHx jxEl75iuQajNmJHB3E0oAf0pQPpOVLiY/qLxFRAQy9BClB3ggqI/hlJkpMv8daWRCkmb fcw2smnVZjyManqMsHQTyXtPiITU2qcJvqzLbxjPJCen+CSEUMv4Fm9hMNhcXZbmKMKg +KqYYLOm2gvVX54Rl1N/0c+2KjzJcva0me7yzk2ihlfHnCZRGNBGiXHQ8zUUiHsiWL4O 3Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghexe9gg7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:42:33 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFlFe015993; Mon, 6 Jun 2022 20:39:00 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2106.outbound.protection.outlook.com [104.47.55.106]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu80abu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KfKUD6r8+ix0nGpxtD4fgYEuOQauraMjypQKyG+s5CR5sTYJNUtyBNPrGF8VF8xV7/9VlApSpQ/l70OKhLJ12Cgw1NQyzI6IZ9F0/n2ofGRPOx/ie9NYdAGZsHsIwEI9fYiQGCeMyJotK0gtqcYI9XnPcVmgMCQlnoaKAjnB0w5i6lDNJOKfgEFrqx9YIyAS8Juai7JhJ5Bnq84Vpb/Ja6PwpEuZaEuOggA6Uqza4F71lIPIV/I6OIJHlVvUHzeBCA8mTLCCoATVlA2AaBY3p2Th9sRqtZz1mXr/UuOK/ges2sybDMdsa3O9XvmidQeWWt2th2dwXcL3i+/c3XI41Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8mBclENT70Amq1G/+YFQ5vVgG1jGKH43y17kCMnkpHg=; b=FNFlwRhCTOhRMIn9QRIbtqa190G0xvwQkW/9ASmyt3aCo07cYGe2gklAqRMMduNr518WvkRRgBRHVEleak+TCDzR/5NgQByA7Wh6ez9bfx49+XdMp4fv+TAVjNmAdXU0s21uOXOzofKcm4DZwvTXOow70g91jN5iHbB05d5bXSObVxF/z6TuwwPN/d3sABHv3KCf4Yf3o8NUvh/gI1retxIOldZjsQchGdyDmB8LF8JyodRMyvr69csHKkjFiDbzz1ZAIXkDU0YsyQHV66JKAbXyUPecjlsG1/IqRlqAoX5E6wespnpycKRMopHNPWVV1FWs3fVRP8rju5jMEjLsZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8mBclENT70Amq1G/+YFQ5vVgG1jGKH43y17kCMnkpHg=; b=M2XDHhDE7NOByRhz2oPJhq2uu91cD+/pXwfTQJjCsaaCez5xNG7dqdDvUvUE2zR9OSlGAbeYSzIasBmfjQ/PcsZYY8oVNuHqKmnHz/7ekX3DX96ezUe1sgl15i0TCkdVmu7jZ7jfBmOFMA5t4r0Qg3c6HhQ4hvEDzdQJ4Rz4q6Q= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:58 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:58 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 17/21] clear_huge_page: use non-cached clearing Date: Mon, 6 Jun 2022 20:37:21 +0000 Message-Id: <20220606203725.1313715-13-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SA0PR11CA0141.namprd11.prod.outlook.com (2603:10b6:806:131::26) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fa14bfc6-43b4-4304-dc61-08da47fc94a3 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xos1bSqd+WLMESSmWbeKrIwpXVgDzRjkvhFLLeYezc6ihED1DRg4+5ztmSYjh9iHSmpwPB/JgDUnC6gFXjuV1OgscB8guAb99N+Ovbd7dBMzQ9Y1omkZVD5EETSfEnAwtn/M0YybcNfYSiIg6mNY3QWzrCHAqZVYilnGD5DqLWI22Yfg5dt0Vy/Og19NOXnM9PEVXMlDBR27MhD8Gk5+H7AkJCQKLMSGCqC+/wL02XnBdO8nI6cx6e7vw6eXdHRKWutDw+Bib7lWl3oQVbZ6R5ZgxtNeXHiYBHiytb19Aq6j+ecJ/tXojeyc/iHALRYY+vTYKjD6WvK2KTo+Yamo1D4GWJPn+C5Y5j4C3tAZqlpcwFKBdECyL2FY42g2E7nZLvi4Me3B6W8ZqvcJZdzj2Va0HgBvc5MjhYh17Se6ft56MMU24vb77GhhGVajCLy/QDt5M6c2TDeRml7gA4ZpuImUh3mLULShT2jbT6ddFTo6OoWHfmiBbDaTzrNfuideVpNRKhIPrEJigLmX1ho14JEHpDcoDzI28s8veNTuRueuP7FgRES8AXU1bzx8tc0p0v1O5IpyfXns1EFJuStJUr46OwXLxY+OBD1A7ec/wHUNsP68uNMbtEx8cSfljLu9g5Hda02fEPqMlQoQKhGW7zmLySDhjEDBpqAGBTYKqF+wo2ahY/0rdOxReGQdFNVUee7UdRuWZ154QzBMIP8dERCUAmIZcwOwLAa7YOBxsV8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EWz5DL1GJZLHk3Nyvpu6h2mAj2K9Jt8bL6wBRvFXNCtfAX6i6i30OsHZ9Yn7+iUf4bQJYE5szeVykN+22SP9mSd0enRQfH2lVTPfqc2QRYX9E1J9Ds5Rx7puFrbR9m0GdgT5HdfCVAscnhcDmXf/elhrXs3sFdl12pOyw/Vm5hfpxLKwIJt7HcKsPuWovx6mlUn4t4zzfRoojw+ktmRlbswmLY/NU3LDcm+Wevcfujy9xBeJb7g0hxRnXEwKeuL1q8gYlrX9Ans4VcsLGO1OMh4HjGdfBzqI48p1X1IFRy7vjkeVgVe5yqdJ+gZFX5Br8Xn5YCRQhT0hJaTPtO4DFquH/RO0srZxWpkbszecvb5WyHuVMwfxAIQw8aFa4+wOPwe1mIaa4rP/TZbOKiAblAwg8oItgLkCGu5ajOmGVj2KYS0VnU241ZKiSgQtJQL9fAb027OzElsBcHRbOZ0a0r1oi8QJgbqBmxHGq2gO8a0L2tjesy+LhI5bqd8qWZugaU935C8pLPD6AgzJuTihjD05N9/EnO8MHNC6NyfP2CMIP51wmqV65qFc7TFS8Te26wh0KNbv/Kp5n9RS3c95dp/bX3tHtsYApv/rfDGswDnhjnLxZcAeU0lMRJNfzhRbteYt1JOyuLI+MW8hVsihH4KvGYrHMca3ok55QqzRF52rhaFcfFokXYeV9NyI5nUEbRTOGcDoqWuOofNuJhMD8sNTckFpmFuA/fuN/wYNXVwC1ZFdKLYmCtmBd5UyRynm3AZ0ADN+wECG+iXy4umBXEyWXnmfgvDgEXIP2fJ4EaejWuX7m6KA3OhINoWU62KtAPTIc1awvZjgbOZJOAY1jFwyANO4lpTQwM0lseDGFhh4zruSCeq8FtVeAa0q+HwX/h7HdxCDrCJPTGIVVd66M46YS0wRmi/wcKmv+ACLXJ5wueqRIGjARbWxHcaeOSoCFw6xKXYHMf/gIlL+DLSqAKIAnqGOmJGe38BKVylSR9V7Y2vDUNfaApEfBfEkANJ+TaxolVvFM50+57hefKNbukCjZ3GBccDf8j9K/nhhNsJokoMHutPZMCfzzk0ReM5j6Hi3QP58NyPgHVUwBm+3Ix9Vk4dqud8FUiWsNHFcLoA2/XStXBGOmz8iRaMJnbn2QFI1+5F+crCtcBE+0+NLoOG0yhAU1ljKKMZTh1lP8NcEjzS7vM+ib2bvL+XHGKUMOj1e0pyG7HiGlHX7oq/OiM1+c+hksjXUBw5bw3/rmjGTCGCXTVWsE6B8IFAF8AijZU0M6B7DrSFL0V1YtFB40KnDvPm6DYiBPMQ88U2hiWGUMQiVtWF8oIiUHHgpxsia822sHcQ10DZ2a6kPIjPgdLzxb5Eu8DZ6eV3ECvJ7bXLy0/cKBUobJtIO3o80G4oDu88fGvDv3gY/bhcyEYu2BOZxznjrqxSnJgWfSq5xxQ2xnLOvw+/7cOwjJdYpNzd5Fa6e7gDf9MCTgIixRDVj0928uqxHKtjkGFsEwDVj77koM8gMZ/9iRg+nh6/NeqUL362Tu/M7EomDHqNSsgBo1DUX360XgnCJyBnFuC9hDk6fCIDfNGorWvzrnYQVisnfKZ1WWyfTsPFN6eE2fQvMb9uQE0GMt6eFaQAB20DwnsqY3e9qstltUm10KX3Kd9/hxLg8pj9etHFBrQjCMXqxzH3b1GBqc8lTsgcP5mZYUuvAtjQ5kBPWSjgnQpJ3Km1U+gmVMaSajtbRIFnabYJpdyFi3crGuq/WZJcRvxdeK9k= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: fa14bfc6-43b4-4304-dc61-08da47fc94a3 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:57.9162 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wFRqj3hGiXwIyqp9zTxNHE6pNvpRauLgh79wol6Q6dUWlQQqryuPfYyhRR/qEcKBZaM2Gjg2szs/1xEDipl2m5IOJmUFx7UX6xBQgpuZnP8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: djNFwhlMr1ZTXJLmPx9yzZSzZ4CaudUo X-Proofpoint-ORIG-GUID: djNFwhlMr1ZTXJLmPx9yzZSzZ4CaudUo X-Stat-Signature: 4wydta95kawggswtoi664cr1c5j1rrmi Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=ssKiQm0+; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=M2XDHhDE; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf15.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E8E3DA0005 X-HE-Tag: 1654548142-432697 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Non-caching stores are suitable for circumstances where the destination region is unlikely to be read again soon, or is large enough that there's no expectation that we will find the data in the cache. Add a new parameter to clear_user_extent(), which handles the non-caching clearing path for huge and gigantic pages. This needs a final clear_page_make_coherent() operation since non-cached clearing typically involves weakly ordered stores that are incoherent wrt other operations in the memory hierarchy. This path is always invoked for gigantic pages, for huge pages only if pages_per_huge_page is greater than an architectural threshold, or if the user gives an explicit hint (if for instance, this call is part of a larger clearing operation.) Signed-off-by: Ankur Arora --- include/linux/mm.h | 3 ++- mm/huge_memory.c | 3 ++- mm/hugetlb.c | 3 ++- mm/memory.c | 50 +++++++++++++++++++++++++++++++++++++++------- 4 files changed, 49 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5084571b2fb6..a9b0c1889348 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3302,7 +3302,8 @@ enum mf_action_page_type { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) extern void clear_huge_page(struct page *page, unsigned long addr_hint, - unsigned int pages_per_huge_page); + unsigned int pages_per_huge_page, + bool non_cached); extern void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr_hint, struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a77c78a2b6b5..73654db77a1c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -594,6 +594,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, pgtable_t pgtable; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; vm_fault_t ret = 0; + bool non_cached = false; VM_BUG_ON_PAGE(!PageCompound(page), page); @@ -611,7 +612,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, goto release; } - clear_huge_page(page, vmf->address, HPAGE_PMD_NR); + clear_huge_page(page, vmf->address, HPAGE_PMD_NR, non_cached); /* * The memory barrier inside __SetPageUptodate makes sure that * clear_huge_page writes become visible before the set_pmd_at() diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7c468ac1d069..0c4a31b5c1e9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5481,6 +5481,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spinlock_t *ptl; unsigned long haddr = address & huge_page_mask(h); bool new_page, new_pagecache_page = false; + bool non_cached = false; /* * Currently, we are forced to kill the process in the event the @@ -5536,7 +5537,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spin_unlock(ptl); goto out; } - clear_huge_page(page, address, pages_per_huge_page(h)); + clear_huge_page(page, address, pages_per_huge_page(h), non_cached); __SetPageUptodate(page); new_page = true; diff --git a/mm/memory.c b/mm/memory.c index b78b32a3e915..0638dc56828f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5606,11 +5606,18 @@ bool clear_page_prefer_non_caching(unsigned long extent) * * With ARCH_MAX_CLEAR_PAGES == 1, clear_user_highpages() drops down * to page-at-a-time mode. Or, funnels through to clear_user_pages(). + * + * With coherent == false, we use incoherent stores and the caller is + * responsible for making the region coherent again by calling + * clear_page_make_coherent(). */ static void clear_user_extent(struct page *start_page, unsigned long vaddr, - unsigned int npages) + unsigned int npages, bool coherent) { - clear_user_highpages(start_page, vaddr, npages); + if (coherent) + clear_user_highpages(start_page, vaddr, npages); + else + clear_user_highpages_incoherent(start_page, vaddr, npages); } struct subpage_arg { @@ -5709,6 +5716,13 @@ static void clear_gigantic_page(struct page *page, { int i; struct page *p = page; + bool coherent; + + /* + * Gigantic pages are large enough, that there are no cache + * expectations. Use the incoherent path. + */ + coherent = false; might_sleep(); for (i = 0; i < pages_per_huge_page; @@ -5718,9 +5732,16 @@ static void clear_gigantic_page(struct page *page, * guarantees that p[0] and p[clear_page_unit-1] * never straddle a mem_map discontiguity. */ - clear_user_extent(p, base_addr + i * PAGE_SIZE, clear_page_unit); + clear_user_extent(p, base_addr + i * PAGE_SIZE, + clear_page_unit, coherent); cond_resched(); } + + /* + * We need to make sure that writes above are ordered before + * updating the PTE and marking SetPageUptodate(). + */ + clear_page_make_coherent(); } static void clear_subpages(struct subpage_arg *sa, @@ -5736,15 +5757,16 @@ static void clear_subpages(struct subpage_arg *sa, n = min(clear_page_unit, remaining); - clear_user_extent(page + i, base_addr + i * PAGE_SIZE, n); + clear_user_extent(page + i, base_addr + i * PAGE_SIZE, + n, true); i += n; cond_resched(); } } -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) +void clear_huge_page(struct page *page, unsigned long addr_hint, + unsigned int pages_per_huge_page, bool non_cached) { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); @@ -5755,7 +5777,21 @@ void clear_huge_page(struct page *page, .page_unit = clear_page_unit, }; - if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { + /* + * The non-caching path is typically slower for small extents so use + * it only if the caller explicitly hints it or if the extent is + * large enough that there are no cache expectations. + * + * We let the gigantic page path handle the details. + */ + non_cached |= + clear_page_prefer_non_caching(pages_per_huge_page * PAGE_SIZE); + + if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES || non_cached)) { + /* + * Gigantic page clearing always uses incoherent clearing + * internally. + */ clear_gigantic_page(page, addr, pages_per_huge_page); return; } From patchwork Mon Jun 6 20:37:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B99DAC43334 for ; Mon, 6 Jun 2022 20:39:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 588088D0002; Mon, 6 Jun 2022 16:39:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5360C8D0001; Mon, 6 Jun 2022 16:39:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D6A58D0002; Mon, 6 Jun 2022 16:39:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2756E8D0001 for ; Mon, 6 Jun 2022 16:39:18 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 00ECF34964 for ; Mon, 6 Jun 2022 20:39:17 +0000 (UTC) X-FDA: 79548975996.07.9C09488 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 96F30120060 for ; Mon, 6 Jun 2022 20:39:02 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KJjq1030073; Mon, 6 Jun 2022 20:39:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=P0ccEylMYeesKNW22zmkv64iuKviZD4DB7kgxwKe7f0=; b=A0VOplnIDZcaAiDsGAaDKnPJ+tI9pjVoUxpiyPxVDI9vabpufutAq6n4S9dnVAPXzpa3 S7PsiFMT5cN6jcFcr7xRSZ030LApIsjfrRnk2EKddGPzBQUpna3Iyj5jtyaW8wpjdosW HBUhjohxjL70vt8yTzBx8nvMiR/L0sHqKNpn0zDbzvgpjJI4TKRh/WE3xbZ7JQCELzH/ cR3aPXEh6cFR0wlH3jWvEXcwXmEr7LjoTTSka/lrsIgYIBlRZ1NZF5258B00ts/Hkyet RfS41WVS61Ie2FQwcLm88+6w5Em8Kdd8lTEee/rDLkkXdiVEahqO2HZ//NpiZTjng1sl OA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad85se-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:02 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFlFf015993; Mon, 6 Jun 2022 20:39:01 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2106.outbound.protection.outlook.com [104.47.55.106]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu80abu-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PwOmQ09qSRCdzkgXz/Xbya1/bRW/0qkN640dZykX6qvtn10UFuVPCzsalHvmYvIjb84V7sVU+GyFTlkBlVoBG4Xi24L70g6yzNQRox859lehBQMZ7StMewOr4gN+tWO5JWDxz/D5aC6b4i9dIa8pOiaQPTZ9kldlCioZjwnPT3vLZcO0ksVp9pziotZqc4WsRxZyPzIwIxJeEBdfKk4i+rSRZwgHLsZUB8hQfprJK3D5TVEgX+YFSgn2b/c+MKpC/eq+XCxrZHfUhgNUjV6hAM3+RdABECYH4HqOFZoVSqepOOLXwfN0K7g5/GR3/wpCIoeh5ePT9uocrM2OfKVeiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P0ccEylMYeesKNW22zmkv64iuKviZD4DB7kgxwKe7f0=; b=e0cjJDbMUoFf6O1TmcWHXN0uM66myszYV+d57QZmGxzHTMQk8lIh/VGYJgMLBBOrfc/rlvpi4/DwDWtlMg/mpuOzt/qAPTdcbnYM1U3NYRagcOVkrw9ZDXF1XcqzcRxvGICz8rRu49roTOQAAloGlFvQCMrb9cZ05kQbfWwJtjDYScGWQEBwESK9VhJvle6W3THklTk6J/Z6YnXyxiqzj2n0KQvws9+JlWlmRdOGGNC/hc0iNxY1I81YdJqqUAu6Fp89aBOhzHo/OPo19GhXf/VqejUcwVD2iLWtjw6eR5n0jbipLiN5/AblrkAZ/HLriISZYuERTBpO8T7WFS7fuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P0ccEylMYeesKNW22zmkv64iuKviZD4DB7kgxwKe7f0=; b=WRdnMF3vN4S459GfUb+YTggn3ASvGpXU9U9o5ERZiErZzdxQAgfkV7pQZQ6pbIAeIuNrJtQa1mQwUZDK2euwztdrdcjcoWhqEW+4R+go8R6oGu8yh9Qk52i+NoyDeipJYazVG4mIYrgrnFmdS4b7TrM3aF4Zwv/Kg1r9PCQ+Gmg= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:59 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:59 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 18/21] gup: add FOLL_HINT_BULK, FAULT_FLAG_NON_CACHING Date: Mon, 6 Jun 2022 20:37:22 +0000 Message-Id: <20220606203725.1313715-14-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SA0PR11CA0141.namprd11.prod.outlook.com (2603:10b6:806:131::26) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ba9328a4-7a66-40ea-fa45-08da47fc959b X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pMu3vvOQlB46mixzUtWPM60u+rNpjVxclunPIAiX2/V4fOjIusKZ8FMt8THOuqbSD2DekJM/QqkXvjcrkwPr2EkgU8LCLRt3cQeOWgA64oHVGJTKjZXx4VTzDs1R5XsPu/ZW5tLj78yjczUPakeqiHbyST2/VZPJ15o5oACQdiShf4p/AosfiPQV5dHWKB5LHXyKDRCsYF7eVuUvU4lkYzu0GS+PNZ8Aljch6sWXf7gD7gkI3tmxvsEhDioJ8G3cktKrV/H5To7Cv4wjzlibs2S11FKi15N06eXHfcGEo7eCl98R+hBYYqsyGEIKDKH/75iO0C/BwR8EUuPgaP0IrS4bYlxCblq116d20KSytdn7AHYrXgPauO0sarnFSSWdfxNyIO575PhgOLD8ANeqYubQgq4Twf27GnYgq1Eetwce9qFB74L72ENI8U03sLiNX+S8eVg3HYPla6GHJawmbSzJlu7XdEWUfJwzoglTHsCT0baimFXLDf2+wGfCJMSbxoXWPeqB+g/F1zDRZKjb57WC+8onwMwLBcokQxff5taGERnALogaBQ4YazuG0I6vMR0Ps8tksPfpIbRxksKVpJgC1/x/TipurbyzRO3KwbVKiijdyZZFhSUV8orIto7XCxYkN2ifB3120W87+8ePP7u1wvTeHX4fZ2LfX5rX29Q7LB08vYRwgEBMIPaUfvVLkpWySYqV7JEGdXWfGrYyng== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sGs/jAxV3ojkVqAYaZtES/dfk/cL36qWEOBJy8yOtEM1H7Rx//y3MTK4nUEQgQJJzxx6xUefMa2BLDNo09w+WvWQWnMUuyhpWj1HkZewhvrCaDVO7SVQYj1bAoJ2jjY30/szZGXFLbghUFUbZ5HdnCZR9LpP3331KdgioYNJdVCpar/9gTvGDVlpPIkaiA2/OaRaX9kA8uyCyEwalr2rEah5JZGFm0M07ix0rrNBI0HI9g2mg8PafJOtrHyP85KmTpo+s8iG9r9hLyb7PRtbmS5Lgh5WYbNE2KFJPR+mWswPlHDnUKugJU0gs0DHj1ZeSwyXdJ1tctkLY9SrNqd4Hy85UcytPyNxHL3U+9b3rqtovnbUbtDMY9BvJsjXB99Quk8Oj0/RjWUKx0lOR52kIU0opeK7CoifmwpLwLiQZG9azMLLa9Tk7+Oyz8sLV0QIO3C7ZcEE2nkqDX3lCETO9dQ0CRgtJ+h9f5B+uzXIOAMKZixoACPYhMzk7BmMZfM3kTC2d7Jc1u/ClX+Fr/mmoxcBscAg/Qil0Xxzf0PvV7rWM5HOcFdFOIzxyTFKeUGUHS5rlp9Dj9ReN5B2PFmz4XW0zwuUfT9FoPnTuaoewXa3v47ckUyYnkZjibww/qFS51/3kSCQRaBjOJO828ZKruPAXY+VM15xHHNvrS1u0tjUBzczUwdd7Edwcr9+hUuHl32Q2vd/FSpWi3iuAodLVdotpTDjSfYDSstM0XJKtk3jS3W5RrJvFbZKfg7ouwE/JSITke8+TZNbebyc1rAph1wCovjB/jmb7bPMvVHHJrrH1v/thZKqQdRJ6MgYJYmSN2bHkoQA0l9J7HSKaOa/8MhIwfh1DcLxC3RYDj9XqrVCXBQmRwKlzeqfNBNSp0vsjGIsFOPqLHV0exj1CvPR4ocPcDc8JwY8skfohYzg9p4q2CLPTsfaK1paSOI7O/l9Zr4RJ7vWLuKRe34QlTRep1VSOjCAVuM27gdIflMAWT6JISRGJKRTfL+2D1hBGMQH/oZeQq1l7UWf7lyGD1p1L9qgIF9fW0ZvV2QVmhtibpVoh6h/z7xU77cfjxiwYDh7/CdOzqApqc6z0oxtql81SUIsuzLpqp0KRvTNiibeiOLs5nzdfosbBMeNsLitBdJpWTquJO9cCSoyJI/4jnjvUhbt8zK9iK7AkXfnejHAC4hvGOIGKIRghXlL6WTUvEfJgU4fZx2l/OovMhmcrhSTelvPMjo1Ibx4CF1U7phVY7RZ/bXanB5ezaBqzCditoy2PI3cAjdPYpfN5a1+u0YxxqBXKsx25Ahfn6E0/dyaG5qmoVQCBcjA5Fkg7N4Wt7lS1A7Rlj32e+dPvheK2BGLcXNCp5ArQpaihknPeQfKtpdlq8Tv21nneiVvswHtj+9wvwcputxssZDdh8/1kBXpd/JjLy2iYqMllCg5F0XtjG/hCYoZFkQqbrxXUgf9AejhqiiyljGWsL2UBXxN4y8VxeOz+5AF1r6KPURnVWmrQe/8QWtcsmJj1mAW9HYsJhbNMgkA1TJTYSsvDbm3j1iNTD05JO73Ek0SwcPYfVb7Cynd7yhkiHjotx9uqMsgxQMf1MG6GTlmqVvIvPUffydeYf5GZaduYVhD92/5fT86cOb56u5Mjmq/sNMg1PwefqOF7dElvxJxSwqpQ0gIy6HXJarZdegZqJ9Gj331NWxWd7thit6XBEwnvCeou22QLwctJfqfovXUD5QWT6cKUgFtWGxFJ31uxvkI95+4BRsjSb0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba9328a4-7a66-40ea-fa45-08da47fc959b X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:59.4641 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JTkTZWdsZjdi7WP0GZbdkyhnfc/dGm4+J2+1utK8o5QXGRirzaSpIC2DSSRxTwz0D1cy97iy42sD5THZ5d/IwDVhGbQZWJlPkkwZx09Tr/I= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: rqMvZUQdfD8H74o1BQKPC9mDUMqsqEkV X-Proofpoint-GUID: rqMvZUQdfD8H74o1BQKPC9mDUMqsqEkV X-Stat-Signature: s85bcw47xyh3iripcatborde71mo9mbk X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=A0VOplnI; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=WRdnMF3v; spf=none (imf29.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 96F30120060 X-HE-Tag: 1654547942-965660 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add FOLL_HINT_BULK, which callers of get_user_pages(), pin_user_pages() can use to signal that this call is one of many, allowing get_user_pages() to optimize accordingly. Additionally, add FAULT_FLAG_NON_CACHING, which in the fault handling path signals that the underlying logic can use non-caching primitives. This is a possible optimization for FOLL_HINT_BULK calls. Signed-off-by: Ankur Arora --- include/linux/mm.h | 1 + include/linux/mm_types.h | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a9b0c1889348..dbd8b7344dfc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2941,6 +2941,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ #define FOLL_PIN 0x40000 /* pages must be released via unpin_user_page */ #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ +#define FOLL_HINT_BULK 0x100000 /* part of a larger extent being gup'd */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b34ff2cdbc4f..287b3018c14d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -824,6 +824,7 @@ typedef struct { * mapped R/O. * @FAULT_FLAG_ORIG_PTE_VALID: whether the fault has vmf->orig_pte cached. * We should only access orig_pte if this flag set. + * @FAULT_FLAG_NON_CACHING: Avoid polluting the cache if possible. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two @@ -861,6 +862,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_NON_CACHING = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Jun 6 20:37:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94CD2C433EF for ; Mon, 6 Jun 2022 20:39:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 362976B0073; Mon, 6 Jun 2022 16:39:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 312DD8D0001; Mon, 6 Jun 2022 16:39:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1652F6B0075; Mon, 6 Jun 2022 16:39:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F30E26B0073 for ; Mon, 6 Jun 2022 16:39:23 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D1C9D35224 for ; Mon, 6 Jun 2022 20:39:23 +0000 (UTC) X-FDA: 79548976206.05.B02F6BD Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf28.hostedemail.com (Postfix) with ESMTP id 6AE06C0070 for ; Mon, 6 Jun 2022 20:38:38 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KTBDS014043; Mon, 6 Jun 2022 20:39:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=jRb62iV3P9PpV4yoLcVaxwYDz8c9aQiErYxdrxBDqWw=; b=cmeIra3qJnDkYOKdJ7TGByZa7V3UXvPd6hAGrajBW13N20VpS+XcLjdBSarDvX0mnfce AFMb/0r1KGRPIS/Sqv61xR4zvuwh/OiXNoupWxbZBBIPJymYfQqvK/qJnL6EY3XnSD6X 9bly7fITASTdGcjROo18dYKCBR/fpDrNPinEF8255FFxeOijCp4YaCkePpEYLw/dEiXy jG/GD3X3VdHXkepC4SPbNASy8CMih5hNb71US0ppZ4Mqx6QRQdldq7M76wqQs4GX4X11 zBZIKyl453NCP7hKhv4z4GelqP/RRETZAiJdQZdqIbl+gJ9T4pqgwi1edEhmGDcoKamX TA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghqad85sg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:04 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KGY1M035885; Mon, 6 Jun 2022 20:39:04 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2100.outbound.protection.outlook.com [104.47.55.100]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gfwu1vf4a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SaDxmUbpv+lwsHSquagznA/Gwz2BYzQy/jg5e5yn98bskrYPbu0eT9eJiZOA+r3LyLDyV0hgkFUTF3lUttPW/na7SdPlbQP3PeKy/clF5KphOu0BtqUjROUzbVbmob9tQ42HwjCxDz9TaWywvhkcPuzdo72RN97CmZG/K3by02GtIdPuKVgLkvjzOD2uZlssWrBTbTUBGrboh+U/aeNEJ8O+hmtb87MP8Gzi2s5Qy5xktL5Ft3Ua8lt+2zf+azt9j6hFeALWjnRIYh3sb0CVl29zPL0Swb25aMM0oON6pk8wyv5xvz/DiSu6DOto8HW+/C22XWRaf7P77kUTH27/vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jRb62iV3P9PpV4yoLcVaxwYDz8c9aQiErYxdrxBDqWw=; b=W3v/M5Gfi9wOUb+EkdXZ4LQ5c0LAd67YNGx8tPJUyF0GNscJJ2wvQwslbAZ/tq5uI9GK2V5A9GK8ReDZtUvUqO5pcJF3JFY9ViDnvhBXYXgxkT5QjKI3EHOeUhcH2W3SkVJlpGR46i1BH6cpcfmM/pmCa0x4e9RBuzVfBMaPdlDk/elpfzaOm4lU1Iy9iMFGNMqFs+BDSkmUkIHbHPngmXu7c56ZK0ILQBrPOkff9iDXM/X/45EY3elnaRBCFc2ZTy/oSU+s6JoClv/5BziIDy5m1CsQs17exRAmIE6HVkCdhlsYRDpExHXJBuIGyZ5SA3LHV8RQEbupYPSneKHmuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jRb62iV3P9PpV4yoLcVaxwYDz8c9aQiErYxdrxBDqWw=; b=tfZjyoQJG1FZ+C8k9aZZdHpC36iU1klae3k46+qIUGHwcgTQx+QlZAi0QnzGXn2DdomAMV9+HPSLAiDbStd7m4hjiEo2rFAFq2JkJQ/3vvWK5Wqvjjtn4qxtH610VYch+FuNjWkKTDvIfKXUl3P3cDKuwUtVhcxkn6uzS6/9Iys= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:39:01 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:39:01 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 19/21] gup: hint non-caching if clearing large regions Date: Mon, 6 Jun 2022 20:37:23 +0000 Message-Id: <20220606203725.1313715-15-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SA0PR11CA0141.namprd11.prod.outlook.com (2603:10b6:806:131::26) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 47c397d6-003f-4246-10b9-08da47fc96b3 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZbnFCVErSmkrda3KOVHivYwVP4LXwtfdWBVwonaYdfzeonRH8spl9w6rTVWNHml4Pnwvmir+Sa8pPj0N67lB9QBWCcnYbpqEqSkQsoQtWWG10TUUgik79UAuK3G+Pd6wpRh2yvGHJUTyApn02o9/qVpSl83Lp7lwhWUQwS4sm1jhDgy6xUDc/i6M5sOEjfHDViEfRTWm25IK8ba37epENRhp7PVjwKBJwFsm9G/SHT5OeDvfXGxrpVHHKDyvcsOUOnne7xFgLyiFKXEKw5Q5dBFAV25Ukupg/b65ydgysPvugl7dBhnyfIK4vfGetexJCKpMB0L/E6nPALMNmdjKBNnRYqPtZL1QhUEraPJdkoyEA98LP+rXirqbuBu3jn1nQ4jvW+CsRrHQq4Zv1uIhYUTiy+Gg5HjG69sEqqmAGtA2EO73acJ81n+hTwO38ddLM+ozEBme0rWoOfdx2YuLo4difXkInQRMBlRDRvbaGpO/bgARs6QRcv+JedsT7/1riPbFPGc7+cRMww+kMWJCb5eTbxj0f6HCWkGoC8361fTdRDjZeqXfjSu6TStXS6bedhAYc24v4xP6pbXU+M8bGOeZS0eAQi1U34x3PwpCTTfstG2JNwx3HUKsYZxCtNVhb3hSF078JdwDeiItNd3k88kKWiqO60Vb0udNMQlM1hH8uuaxxtoVJz7SaBDUbewv+pJmT4uwqiRp69S46BYnCw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(30864003)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xZXRejXXLeMvuyacK+1Cpo/3oyyPji1q8Gcg3M6/u/fqKewCOsEbNPRFH76YoxRLFQBYc0bF0KV8HmHT/SJM+zNYhBDaf3uQWOHYuzEluR5VLgwqpjbLXnBigXVrl7sSKdHxbjWybf9Df4yH4+7NHQsO2pmI/XUkG91iBFqhCj4P3/UkFmIPjIlxUhho9RkaM3hWjxL9UQlnNj1O0L3j/0sufnjrpJ6KK0LDExg1n/vWcumIb4cimNY0GDilHUfHfk7daMgpg93LurPfJlsm0ydTWL7PBAtoRdCs/QWeDACfW5Kc536HyWRmIqtZSuug3G8TroAEnPaNdrztxOwmIw18krUGNBKhfdefZ4oJ0BENdBxHM+lYew/iq4iFfbRaq4it8SjODk5PJr4/yfB+H4AikZ2eovg9OpTwP+YeizuFc8AaNNJ8yrKXHgyoY7ulktZbxLIs6SPsHMurDjVX2sz4McV5guhQ78eSRc7dqFd6TCXsMY+A/hvjInqiAeWu6c5Qd9JLEqgDn8XMi9wrSXXKslKj+43oVPeslb/gU/Kuc6WSQRoeCskLv5MTOPo91SsC0BSFRwp9pzB2ll+wUPOqYdDiYwPYo28g1+a2Y3SF40jcGwUWQ9HHu1r9xzmcaeJpkGbwNJV2rQwdKJfs3k3oRg0t2yjhMWAUjNO3vBNSwtjaub+TRmqppQsPYTWwafLJtj4PqqijOyWw0XJTebk8lrK8eA81uMHQYpqQgn55zuX94MAtcGIzO76jqWMJTVta9LiXwOLJat7SEBdqO7FEQ52nI84jQcSrIw54S2iQ8hs2mBU1DQN9JM8j70/aHpfzQMEFC2ZSMiQttzWfzdJ8piVfLmgGqrNGdmKYlSzrwkLxoaZKlD6sJngkVw8g13eg/QhRELQl5cqYQfTxWRgZk2SLPf0WmoTG+Y9Xc/MCJPV7ZNiG9drCHTZ9ue8zgkvb5zcbOzaOUZbhOl/i3TMc5NWt171sTAuN1pe8FNCy4aRclFFE0EZb+ZKks+dQBi+QLe2BqtaIPOqGBTlrqBsHO+51qJFmqgnipnzmjzqyTpSzXSDF7IS43nmzEJrQwtlO0e2ZbcxgAhpRqtv5S8N4vpsAKvzZFescBvpg3ZzQZgPOt5e3CPNIcckgzlvvZFfsx/QOwU3Jc0CZPnQQl/xs7BlS4A3XmM293gIoHt7rCKRBdYCNUBwlYH+DXb4H1xplQS5PXUtziXsHcUU7s+WS2JftjEQ/Ax6/vIQd5oufNYos4y80yH3u6KhNX0ey3p4Y3cNuzhnVrGUgwc2rm4I6WODKZ9jxv/hFHj5YXlmtzhH1Glp/OaAl7rqvRnU31XM38UWEIFf50aa8JY4pVlgBmtETsb5a1q6JFDtbJoS3YK8ehnibNA3EINnPxwN0ogxbXj74QvjoAhXkToedPhtg2avI+BdJmlh6PpeBhFCuXo3v0g+l3K2gyozp5nEix7Dtx/vAVmEuJp5Mswi7inKp4yvevKrFSxKwE2JetyQVoDhl33jT5yqaI9WZeH25gQ2iyvJj4/WHTXbtud5xYGMy6uwieTzr7DBFzMCStIVBQyp3eAxfgCVH5oWErrmYMpmLeTEuznKCQHlmRr12/FrZQIOw47lSnPwNwikHcSBKdgkxG4rqh5OYKQjpsIUwVYadbkjJIIiqQe6UYvzIku9miPTjP3fWx62F4BIUpBPXrgoX4Xj+EKOiXUFJEjX+1s6c3bG+Yyn4uwTNHsSXSqAiZ7/PAywsfxDucKaRJGw= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 47c397d6-003f-4246-10b9-08da47fc96b3 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:39:01.3400 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FyCUgTHmE+x4212QRpM9WnBcVSsdMZACkyS3au+l+52TB4PKjst49HIxDsu3H1WRy/bbwPKn4igZTSQlq3NNlBjhJOoLESf6iXDCVO60X04= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-ORIG-GUID: jJd2z4J5k21SRI9vZmNXTzw5_sKjqLAy X-Proofpoint-GUID: jJd2z4J5k21SRI9vZmNXTzw5_sKjqLAy X-Stat-Signature: 13amyrgxutc93qq8xukjmjch31auug6a X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=cmeIra3q; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=tfZjyoQJ; spf=none (imf28.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6AE06C0070 X-HE-Tag: 1654547918-961988 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When clearing a large region, or when the user explicitly hints via FOLL_HINT_BULK that a call to get_user_pages() is part of a larger region being gup'd, take the non-caching path. One notable limitation is that this is only done when the underlying pages are huge or gigantic, even if a large region composed of PAGE_SIZE pages is being cleared. This is because non-caching stores are generally weakly ordered and need some kind of store fence -- at PTE write granularity -- to avoid data leakage. This is expensive enough to negate any performance advantage. Performance == System: Oracle X9-2c (2 nodes * 32 cores * 2 threads) Processor: Intel Xeon(R) Platinum 8358 CPU @ 2.60GHz (Icelakex, 6:106:6) Memory: 1024 GB evenly split between nodes LLC-size: 48MB for each node (32-cores * 2-threads) no_turbo: 1, Microcode: 0xd0002c1, scaling-governor: performance System: Oracle E4-2c (2 nodes * 8 CCXes * 8 cores * 2 threads) Processor: AMD EPYC 7J13 64-Core Processor (Milan, 25:1:1) Memory: 512 GB evenly split between nodes LLC-size: 32MB for each CCX (8-cores * 2-threads) boost: 1, Microcode: 0xa00115d, scaling-governor: performance Two workloads: qemu VM creation where that is the exclusive load and, to probe the cache interference with unrelated processes aspect of these changes, a kbuild with a background page clearing workload. Workload: create a 192GB qemu-VM (backed by preallocated 2MB pages on the local node) == Icelakex --- Time (s) Delta (%) clear_pages_erms() 16.49 ( +- 0.06s ) # 12.50 bytes/ns clear_pages_movnt() 9.42 ( +- 0.20s ) -42.87% # 21.88 bytes/ns It is easy enough to see where the improvement is coming from -- given the non-caching stores, the CPU does not need to do any RFOs ending up with way fewer L1-dcache-load-misses: - 407,619,058 L1-dcache-loads # 24.746 M/sec ( +- 0.17% ) (69.20%) - 3,245,399,461 L1-dcache-load-misses # 801.49% of all L1-dcache accesses ( +- 0.01% ) (69.22%) + 393,160,148 L1-dcache-loads # 41.786 M/sec ( +- 0.80% ) (69.22%) + 5,790,543 L1-dcache-load-misses # 1.50% of all L1-dcache accesses ( +- 1.55% ) (69.26%) (Fuller perf stat output, at [1], [2].) Milan -- Time (s) Delta clear_pages_erms() 11.83 ( +- 0.08s ) # 17.42 bytes/ns clear_pages_clzero() 4.91 ( +- 0.27s ) -58.49% # 41.98 bytes/ns Milan does significantly fewer RFO, as well. - 6,882,968,897 L1-dcache-loads # 582.960 M/sec ( +- 0.03% ) (33.38%) - 3,267,546,914 L1-dcache-load-misses # 47.45% of all L1-dcache accesses ( +- 0.02% ) (33.37%) + 418,489,450 L1-dcache-loads # 85.611 M/sec ( +- 1.19% ) (33.46%) + 5,406,557 L1-dcache-load-misses # 1.35% of all L1-dcache accesses ( +- 1.07% ) (33.45%) (Fuller perf stat output, at [3], [4].) Workload: Kbuild with background clear_huge_page() == Probe the cache-pollution aspect of this commit with a kbuild (make -j 32 bzImage) alongside a background process doing clear_huge_page() via mmap(length=64GB, flags=MAP_POPULATE|MAP_HUGE_2MB) in a loop. The expectation -- assuming kbuild performance is partly cache limited -- is that the clear_huge_page() -> clear_pages_erms() background load would show a greater slowdown than, clear_huge_page() -> clear_pages_movnt(). The kbuild itself does not use THP or similar, so any performance changes are due to the background load. Icelakex -- # kbuild: 16 cores, 32 threads # clear_huge_page() load: single thread bound to the same CPUset # taskset -c 16-31,80-95 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage - 8,226,884,900,694 instructions # 1.09 insn per cycle ( +- 0.02% ) (47.27%) + 8,223,413,950,371 instructions # 1.12 insn per cycle ( +- 0.03% ) (47.31%) - 20,016,410,480,886 slots # 6.565 G/sec ( +- 0.01% ) (69.84%) - 1,310,070,777,023 topdown-be-bound # 6.1% backend bound ( +- 0.28% ) (69.84%) + 19,328,950,611,944 slots # 6.494 G/sec ( +- 0.02% ) (69.87%) + 1,043,408,291,623 topdown-be-bound # 5.0% backend bound ( +- 0.33% ) (69.87%) - 10,747,834,729 LLC-loads # 3.525 M/sec ( +- 0.05% ) (69.68%) - 4,841,355,743 LLC-load-misses # 45.02% of all LL-cache accesses ( +- 0.06% ) (69.70%) + 10,466,865,056 LLC-loads # 3.517 M/sec ( +- 0.08% ) (69.68%) + 4,206,944,783 LLC-load-misses # 40.21% of all LL-cache accesses ( +- 0.06% ) (69.71%) The LLC-load-misses show a significant improvement (-13.11%) which is borne out in the (-20.35%) reduction in topdown-be-bound and a (2.7%) improvement in IPC. - 7,521,157,276,899 cycles # 2.467 GHz ( +- 0.02% ) (39.65%) + 7,348,971,235,549 cycles # 2.469 GHz ( +- 0.04% ) (39.68%) The ends up with an overall improvement in cycles of (-2.28%). (Fuller perf stat output, at [5], [6].) Milan -- # kbuild: 2 CCxes, 16 cores, 32 threads # clear_huge_page() load: single thread bound to the same CPUset # taskset -c 16-31,144-159 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage - 302,739,130,717 stalled-cycles-backend # 3.82% backend cycles idle ( +- 0.10% ) (41.11%) + 287,703,667,307 stalled-cycles-backend # 3.74% backend cycles idle ( +- 0.04% ) (41.11%) - 8,981,403,534,446 instructions # 1.13 insn per cycle + 8,969,062,192,998 instructions # 1.16 insn per cycle Milan sees a (-4.96%) improvement in stalled-cycles-backend and a (-2.65%) improvement in IPC. - 7,930,842,057,103 cycles # 2.338 GHz ( +- 0.04% ) (41.09%) + 7,705,812,395,365 cycles # 2.339 GHz ( +- 0.01% ) (41.11%) The ends up with an overall improvement in cycles of (-2.83%). (Fuller perf stat output, at [7], [8].) [1] Icelakex, clear_pages_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 16,329.41 msec task-clock # 0.990 CPUs utilized ( +- 0.42% ) 143 context-switches # 8.681 /sec ( +- 0.93% ) 1 cpu-migrations # 0.061 /sec ( +- 63.25% ) 118 page-faults # 7.164 /sec ( +- 0.27% ) 41,735,523,673 cycles # 2.534 GHz ( +- 0.42% ) (38.46%) 1,454,116,543 instructions # 0.03 insn per cycle ( +- 0.49% ) (46.16%) 266,749,920 branches # 16.194 M/sec ( +- 0.41% ) (53.86%) 928,726 branch-misses # 0.35% of all branches ( +- 0.38% ) (61.54%) 208,805,754,709 slots # 12.676 G/sec ( +- 0.41% ) (69.23%) 5,355,889,366 topdown-retiring # 2.5% retiring ( +- 0.50% ) (69.23%) 12,720,749,784 topdown-bad-spec # 6.1% bad speculation ( +- 1.38% ) (69.23%) 998,710,552 topdown-fe-bound # 0.5% frontend bound ( +- 0.85% ) (69.23%) 192,653,197,875 topdown-be-bound # 90.9% backend bound ( +- 0.38% ) (69.23%) 407,619,058 L1-dcache-loads # 24.746 M/sec ( +- 0.17% ) (69.20%) 3,245,399,461 L1-dcache-load-misses # 801.49% of all L1-dcache accesses ( +- 0.01% ) (69.22%) 10,805,747 LLC-loads # 656.009 K/sec ( +- 0.37% ) (69.25%) 804,475 LLC-load-misses # 7.44% of all LL-cache accesses ( +- 2.73% ) (69.26%) L1-icache-loads 18,134,527 L1-icache-load-misses ( +- 1.24% ) (30.80%) 435,474,462 dTLB-loads # 26.437 M/sec ( +- 0.28% ) (30.80%) 41,187 dTLB-load-misses # 0.01% of all dTLB cache accesses ( +- 4.06% ) (30.79%) iTLB-loads 440,135 iTLB-load-misses ( +- 1.07% ) (30.78%) L1-dcache-prefetches L1-dcache-prefetch-misses 16.4906 +- 0.0676 seconds time elapsed ( +- 0.41% ) [2] Icelakex, clear_pages_movnt() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 9,896.77 msec task-clock # 1.050 CPUs utilized ( +- 2.08% ) 135 context-switches # 14.348 /sec ( +- 0.74% ) 0 cpu-migrations # 0.000 /sec 116 page-faults # 12.329 /sec ( +- 0.50% ) 25,239,642,558 cycles # 2.683 GHz ( +- 2.11% ) (38.43%) 36,791,658,500 instructions # 1.54 insn per cycle ( +- 0.06% ) (46.12%) 3,475,279,229 branches # 369.361 M/sec ( +- 0.09% ) (53.82%) 1,987,098 branch-misses # 0.06% of all branches ( +- 0.71% ) (61.51%) 126,256,220,768 slots # 13.419 G/sec ( +- 2.10% ) (69.21%) 57,705,186,453 topdown-retiring # 47.8% retiring ( +- 0.28% ) (69.21%) 5,934,729,245 topdown-bad-spec # 4.3% bad speculation ( +- 5.91% ) (69.21%) 4,089,990,217 topdown-fe-bound # 3.1% frontend bound ( +- 2.11% ) (69.21%) 60,298,426,167 topdown-be-bound # 44.8% backend bound ( +- 4.21% ) (69.21%) 393,160,148 L1-dcache-loads # 41.786 M/sec ( +- 0.80% ) (69.22%) 5,790,543 L1-dcache-load-misses # 1.50% of all L1-dcache accesses ( +- 1.55% ) (69.26%) 1,069,049 LLC-loads # 113.621 K/sec ( +- 1.25% ) (69.27%) 728,260 LLC-load-misses # 70.65% of all LL-cache accesses ( +- 2.63% ) (69.30%) L1-icache-loads 14,620,549 L1-icache-load-misses ( +- 1.27% ) (30.80%) 404,962,421 dTLB-loads # 43.040 M/sec ( +- 1.13% ) (30.80%) 31,916 dTLB-load-misses # 0.01% of all dTLB cache accesses ( +- 4.61% ) (30.77%) iTLB-loads 396,984 iTLB-load-misses ( +- 2.23% ) (30.74%) L1-dcache-prefetches L1-dcache-prefetch-misses 9.428 +- 0.206 seconds time elapsed ( +- 2.18% ) [3] Milan, clear_pages_erms() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 11,676.79 msec task-clock # 0.987 CPUs utilized ( +- 0.68% ) 96 context-switches # 8.131 /sec ( +- 0.78% ) 2 cpu-migrations # 0.169 /sec ( +- 18.71% ) 106 page-faults # 8.978 /sec ( +- 0.23% ) 28,161,726,414 cycles # 2.385 GHz ( +- 0.69% ) (33.33%) 141,032,827 stalled-cycles-frontend # 0.50% frontend cycles idle ( +- 52.44% ) (33.35%) 796,792,139 stalled-cycles-backend # 2.80% backend cycles idle ( +- 23.73% ) (33.35%) 1,140,172,646 instructions # 0.04 insn per cycle # 0.50 stalled cycles per insn ( +- 0.89% ) (33.35%) 219,864,061 branches # 18.622 M/sec ( +- 1.06% ) (33.36%) 1,407,446 branch-misses # 0.63% of all branches ( +- 10.66% ) (33.40%) 6,882,968,897 L1-dcache-loads # 582.960 M/sec ( +- 0.03% ) (33.38%) 3,267,546,914 L1-dcache-load-misses # 47.45% of all L1-dcache accesses ( +- 0.02% ) (33.37%) LLC-loads LLC-load-misses 146,901,513 L1-icache-loads # 12.442 M/sec ( +- 0.78% ) (33.36%) 1,462,155 L1-icache-load-misses # 0.99% of all L1-icache accesses ( +- 0.83% ) (33.34%) 2,055,805 dTLB-loads # 174.118 K/sec ( +- 22.56% ) (33.33%) 136,260 dTLB-load-misses # 4.69% of all dTLB cache accesses ( +- 23.13% ) (33.35%) 941 iTLB-loads # 79.699 /sec ( +- 5.54% ) (33.35%) 115,444 iTLB-load-misses # 14051.12% of all iTLB cache accesses ( +- 21.17% ) (33.34%) 95,438,373 L1-dcache-prefetches # 8.083 M/sec ( +- 19.99% ) (33.34%) L1-dcache-prefetch-misses 11.8296 +- 0.0805 seconds time elapsed ( +- 0.68% ) [4] Milan, clear_pages_clzero() # perf stat -r 5 --all-kernel -ddd ./qemu.sh Performance counter stats for './qemu.sh' (5 runs): 4,599.00 msec task-clock # 0.937 CPUs utilized ( +- 5.93% ) 91 context-switches # 18.616 /sec ( +- 0.92% ) 0 cpu-migrations # 0.000 /sec 107 page-faults # 21.889 /sec ( +- 0.19% ) 10,975,453,059 cycles # 2.245 GHz ( +- 6.02% ) (33.28%) 14,193,355 stalled-cycles-frontend # 0.12% frontend cycles idle ( +- 1.90% ) (33.35%) 38,969,144 stalled-cycles-backend # 0.33% backend cycles idle ( +- 23.92% ) (33.34%) 13,951,880,530 instructions # 1.20 insn per cycle # 0.00 stalled cycles per insn ( +- 0.11% ) (33.33%) 3,426,708,418 branches # 701.003 M/sec ( +- 0.06% ) (33.36%) 2,350,619 branch-misses # 0.07% of all branches ( +- 0.61% ) (33.45%) 418,489,450 L1-dcache-loads # 85.611 M/sec ( +- 1.19% ) (33.46%) 5,406,557 L1-dcache-load-misses # 1.35% of all L1-dcache accesses ( +- 1.07% ) (33.45%) LLC-loads LLC-load-misses 90,088,059 L1-icache-loads # 18.429 M/sec ( +- 0.36% ) (33.44%) 1,081,035 L1-icache-load-misses # 1.20% of all L1-icache accesses ( +- 3.67% ) (33.42%) 4,017,464 dTLB-loads # 821.854 K/sec ( +- 1.02% ) (33.40%) 204,096 dTLB-load-misses # 5.22% of all dTLB cache accesses ( +- 9.77% ) (33.36%) 770 iTLB-loads # 157.519 /sec ( +- 5.12% ) (33.36%) 209,834 iTLB-load-misses # 29479.35% of all iTLB cache accesses ( +- 0.17% ) (33.34%) 1,596,265 L1-dcache-prefetches # 326.548 K/sec ( +- 1.55% ) (33.31%) L1-dcache-prefetch-misses 4.908 +- 0.272 seconds time elapsed ( +- 5.54% ) [5] Icelakex, kbuild + bg:clear_pages_erms() load. # taskset -c 16-31,80-95 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage Performance counter stats for 'make -C .. -j 32 O=b2 clean bzImage' (5 runs): 3,047,329.07 msec task-clock # 19.520 CPUs utilized ( +- 0.02% ) 1,675,061 context-switches # 549.415 /sec ( +- 0.43% ) 89,232 cpu-migrations # 29.268 /sec ( +- 2.34% ) 85,752,972 page-faults # 28.127 K/sec ( +- 0.00% ) 7,521,157,276,899 cycles # 2.467 GHz ( +- 0.02% ) (39.65%) 8,226,884,900,694 instructions # 1.09 insn per cycle ( +- 0.02% ) (47.27%) 1,744,557,848,503 branches # 572.209 M/sec ( +- 0.02% ) (54.83%) 36,252,079,075 branch-misses # 2.08% of all branches ( +- 0.02% ) (62.35%) 20,016,410,480,886 slots # 6.565 G/sec ( +- 0.01% ) (69.84%) 6,518,990,385,998 topdown-retiring # 30.5% retiring ( +- 0.02% ) (69.84%) 7,821,817,193,732 topdown-bad-spec # 36.7% bad speculation ( +- 0.29% ) (69.84%) 5,714,082,318,274 topdown-fe-bound # 26.7% frontend bound ( +- 0.10% ) (69.84%) 1,310,070,777,023 topdown-be-bound # 6.1% backend bound ( +- 0.28% ) (69.84%) 2,270,017,283,501 L1-dcache-loads # 744.558 M/sec ( +- 0.02% ) (69.60%) 103,295,556,544 L1-dcache-load-misses # 4.55% of all L1-dcache accesses ( +- 0.02% ) (69.64%) 10,747,834,729 LLC-loads # 3.525 M/sec ( +- 0.05% ) (69.68%) 4,841,355,743 LLC-load-misses # 45.02% of all LL-cache accesses ( +- 0.06% ) (69.70%) L1-icache-loads 180,672,238,145 L1-icache-load-misses ( +- 0.03% ) (31.18%) 2,216,149,664,522 dTLB-loads # 726.890 M/sec ( +- 0.03% ) (31.83%) 2,000,781,326 dTLB-load-misses # 0.09% of all dTLB cache accesses ( +- 0.08% ) (31.79%) iTLB-loads 1,938,124,234 iTLB-load-misses ( +- 0.04% ) (31.76%) L1-dcache-prefetches L1-dcache-prefetch-misses 156.1136 +- 0.0785 seconds time elapsed ( +- 0.05% ) [6] Icelakex, kbuild + bg:clear_pages_movnt() load. # taskset -c 16-31,80-95 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage Performance counter stats for 'make -C .. -j 32 O=b2 clean bzImage' (5 runs): 2,978,535.47 msec task-clock # 19.471 CPUs utilized ( +- 0.05% ) 1,637,295 context-switches # 550.105 /sec ( +- 0.89% ) 91,635 cpu-migrations # 30.788 /sec ( +- 1.88% ) 85,754,138 page-faults # 28.812 K/sec ( +- 0.00% ) 7,348,971,235,549 cycles # 2.469 GHz ( +- 0.04% ) (39.68%) 8,223,413,950,371 instructions # 1.12 insn per cycle ( +- 0.03% ) (47.31%) 1,743,914,970,674 branches # 585.928 M/sec ( +- 0.01% ) (54.87%) 36,188,623,655 branch-misses # 2.07% of all branches ( +- 0.05% ) (62.39%) 19,328,950,611,944 slots # 6.494 G/sec ( +- 0.02% ) (69.87%) 6,508,801,041,075 topdown-retiring # 31.7% retiring ( +- 0.35% ) (69.87%) 7,581,383,615,462 topdown-bad-spec # 36.4% bad speculation ( +- 0.43% ) (69.87%) 5,521,686,808,149 topdown-fe-bound # 26.8% frontend bound ( +- 0.14% ) (69.87%) 1,043,408,291,623 topdown-be-bound # 5.0% backend bound ( +- 0.33% ) (69.87%) 2,269,475,492,575 L1-dcache-loads # 762.507 M/sec ( +- 0.03% ) (69.63%) 101,544,979,642 L1-dcache-load-misses # 4.47% of all L1-dcache accesses ( +- 0.05% ) (69.66%) 10,466,865,056 LLC-loads # 3.517 M/sec ( +- 0.08% ) (69.68%) 4,206,944,783 LLC-load-misses # 40.21% of all LL-cache accesses ( +- 0.06% ) (69.71%) L1-icache-loads 180,267,126,923 L1-icache-load-misses ( +- 0.07% ) (31.17%) 2,216,010,317,050 dTLB-loads # 744.544 M/sec ( +- 0.03% ) (31.82%) 1,979,801,744 dTLB-load-misses # 0.09% of all dTLB cache accesses ( +- 0.10% ) (31.79%) iTLB-loads 1,925,390,304 iTLB-load-misses ( +- 0.08% ) (31.77%) L1-dcache-prefetches L1-dcache-prefetch-misses 152.972 +- 0.309 seconds time elapsed ( +- 0.20% ) [7] Milan, clear_pages_erms() # taskset -c 16-31,144-159 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage Performance counter stats for 'make -C .. -j 32 O=b2 clean bzImage' (5 runs): 3,390,130.53 msec task-clock # 18.241 CPUs utilized ( +- 0.04% ) 1,720,283 context-switches # 507.160 /sec ( +- 0.27% ) 96,694 cpu-migrations # 28.507 /sec ( +- 1.41% ) 75,872,994 page-faults # 22.368 K/sec ( +- 0.00% ) 7,930,842,057,103 cycles # 2.338 GHz ( +- 0.04% ) (41.09%) 39,974,518,172 stalled-cycles-frontend # 0.50% frontend cycles idle ( +- 0.05% ) (41.10%) 302,739,130,717 stalled-cycles-backend # 3.82% backend cycles idle ( +- 0.10% ) (41.11%) 8,981,403,534,446 instructions # 1.13 insn per cycle # 0.03 stalled cycles per insn ( +- 0.03% ) (41.10%) 1,909,303,327,220 branches # 562.886 M/sec ( +- 0.02% ) (41.10%) 50,324,935,298 branch-misses # 2.64% of all branches ( +- 0.02% ) (41.09%) 3,563,297,595,796 L1-dcache-loads # 1.051 G/sec ( +- 0.03% ) (41.08%) 129,901,339,258 L1-dcache-load-misses # 3.65% of all L1-dcache accesses ( +- 0.10% ) (41.07%) LLC-loads LLC-load-misses 809,770,606,566 L1-icache-loads # 238.730 M/sec ( +- 0.03% ) (41.07%) 12,403,758,671 L1-icache-load-misses # 1.53% of all L1-icache accesses ( +- 0.08% ) (41.07%) 60,010,026,089 dTLB-loads # 17.692 M/sec ( +- 0.04% ) (41.07%) 3,254,066,681 dTLB-load-misses # 5.42% of all dTLB cache accesses ( +- 0.09% ) (41.07%) 5,195,070,952 iTLB-loads # 1.532 M/sec ( +- 0.03% ) (41.08%) 489,196,395 iTLB-load-misses # 9.42% of all iTLB cache accesses ( +- 0.10% ) (41.09%) 39,920,161,716 L1-dcache-prefetches # 11.769 M/sec ( +- 0.03% ) (41.09%) L1-dcache-prefetch-misses 185.852 +- 0.501 seconds time elapsed ( +- 0.27% ) [8] Milan, clear_pages_clzero() # taskset -c 16-31,144-159 perf stat -r 5 -ddd \ make -C .. -j 32 O=b2 clean bzImage Performance counter stats for 'make -C .. -j 32 O=b2 clean bzImage' (5 runs): 3,296,677.12 msec task-clock # 18.051 CPUs utilized ( +- 0.02% ) 1,713,645 context-switches # 520.062 /sec ( +- 0.26% ) 91,883 cpu-migrations # 27.885 /sec ( +- 0.83% ) 75,877,740 page-faults # 23.028 K/sec ( +- 0.00% ) 7,705,812,395,365 cycles # 2.339 GHz ( +- 0.01% ) (41.11%) 38,866,265,031 stalled-cycles-frontend # 0.50% frontend cycles idle ( +- 0.09% ) (41.10%) 287,703,667,307 stalled-cycles-backend # 3.74% backend cycles idle ( +- 0.04% ) (41.11%) 8,969,062,192,998 instructions # 1.16 insn per cycle # 0.03 stalled cycles per insn ( +- 0.01% ) (41.11%) 1,906,857,866,689 branches # 578.699 M/sec ( +- 0.01% ) (41.10%) 50,155,411,444 branch-misses # 2.63% of all branches ( +- 0.03% ) (41.11%) 3,552,652,190,906 L1-dcache-loads # 1.078 G/sec ( +- 0.01% ) (41.13%) 127,238,478,917 L1-dcache-load-misses # 3.58% of all L1-dcache accesses ( +- 0.04% ) (41.13%) LLC-loads LLC-load-misses 808,024,730,682 L1-icache-loads # 245.222 M/sec ( +- 0.03% ) (41.13%) 7,773,178,107 L1-icache-load-misses # 0.96% of all L1-icache accesses ( +- 0.11% ) (41.13%) 59,684,355,294 dTLB-loads # 18.113 M/sec ( +- 0.04% ) (41.12%) 3,247,521,154 dTLB-load-misses # 5.44% of all dTLB cache accesses ( +- 0.04% ) (41.12%) 5,064,547,530 iTLB-loads # 1.537 M/sec ( +- 0.09% ) (41.12%) 462,977,175 iTLB-load-misses # 9.13% of all iTLB cache accesses ( +- 0.07% ) (41.12%) 39,307,810,241 L1-dcache-prefetches # 11.929 M/sec ( +- 0.06% ) (41.11%) L1-dcache-prefetch-misses 182.630 +- 0.365 seconds time elapsed ( +- 0.20% ) Signed-off-by: Ankur Arora --- Notes: Not sure if this wall of perf-stats (or indeed the whole kbuild test) is warranted here. To my eyes, there's no non-obvious information in the performance results (reducing cache usage should and does lead to other processes getting a small bump in performance), so is there any value in keeping this in the commit message? fs/hugetlbfs/inode.c | 7 ++++++- mm/gup.c | 18 ++++++++++++++++++ mm/huge_memory.c | 2 +- mm/hugetlb.c | 9 ++++++++- 4 files changed, 33 insertions(+), 3 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 62408047e8d7..993bb7227a2f 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -650,6 +650,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, loff_t hpage_size = huge_page_size(h); unsigned long hpage_shift = huge_page_shift(h); pgoff_t start, index, end; + bool hint_non_caching; int error; u32 hash; @@ -667,6 +668,9 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, start = offset >> hpage_shift; end = (offset + len + hpage_size - 1) >> hpage_shift; + /* Don't pollute the cache if we are fallocte'ing a large region. */ + hint_non_caching = clear_page_prefer_non_caching((end - start) << hpage_shift); + inode_lock(inode); /* We need to check rlimit even when FALLOC_FL_KEEP_SIZE */ @@ -745,7 +749,8 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, error = PTR_ERR(page); goto out; } - clear_huge_page(page, addr, pages_per_huge_page(h)); + clear_huge_page(page, addr, pages_per_huge_page(h), + hint_non_caching); __SetPageUptodate(page); error = huge_add_to_page_cache(page, mapping, index); if (unlikely(error)) { diff --git a/mm/gup.c b/mm/gup.c index 551264407624..bceb6ff64687 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -944,6 +944,13 @@ static int faultin_page(struct vm_area_struct *vma, */ fault_flags |= FAULT_FLAG_TRIED; } + if (*flags & FOLL_HINT_BULK) { + /* + * This page is part of a large region being faulted-in + * so attempt to minimize cache-pollution. + */ + fault_flags |= FAULT_FLAG_NON_CACHING; + } if (unshare) { fault_flags |= FAULT_FLAG_UNSHARE; /* FAULT_FLAG_WRITE and FAULT_FLAG_UNSHARE are incompatible */ @@ -1116,6 +1123,17 @@ static long __get_user_pages(struct mm_struct *mm, if (!(gup_flags & FOLL_FORCE)) gup_flags |= FOLL_NUMA; + /* + * Non-cached page clearing is generally faster when clearing regions + * larger than O(LLC-size). So hint the non-caching path based on + * clear_page_prefer_non_caching(). + * + * Note, however this check is optimistic -- nr_pages is the upper + * limit and we might be clearing less than that. + */ + if (clear_page_prefer_non_caching(nr_pages * PAGE_SIZE)) + gup_flags |= FOLL_HINT_BULK; + do { struct page *page; unsigned int foll_flags = gup_flags; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 73654db77a1c..c7294cffc384 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -594,7 +594,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, pgtable_t pgtable; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; vm_fault_t ret = 0; - bool non_cached = false; + bool non_cached = vmf->flags & FAULT_FLAG_NON_CACHING; VM_BUG_ON_PAGE(!PageCompound(page), page); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0c4a31b5c1e9..d906c6558b15 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5481,7 +5481,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spinlock_t *ptl; unsigned long haddr = address & huge_page_mask(h); bool new_page, new_pagecache_page = false; - bool non_cached = false; + bool non_cached = flags & FAULT_FLAG_NON_CACHING; /* * Currently, we are forced to kill the process in the event the @@ -6182,6 +6182,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, */ fault_flags |= FAULT_FLAG_TRIED; } + if (flags & FOLL_HINT_BULK) { + /* + * From the user hint, we might be faulting-in + * a large region so minimize cache-pollution. + */ + fault_flags |= FAULT_FLAG_NON_CACHING; + } ret = hugetlb_fault(mm, vma, vaddr, fault_flags); if (ret & VM_FAULT_ERROR) { err = vm_fault_to_errno(ret, flags); From patchwork Mon Jun 6 20:37:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF92CCA473 for ; Mon, 6 Jun 2022 20:39:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44E158D0005; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FDED8D0001; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2501C8D0005; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 177DC8D0001 for ; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ECFA03338D for ; Mon, 6 Jun 2022 20:39:33 +0000 (UTC) X-FDA: 79548976626.07.30E017B Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf24.hostedemail.com (Postfix) with ESMTP id 5D1D3180038 for ; Mon, 6 Jun 2022 20:39:16 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256HJ0oK019941; Mon, 6 Jun 2022 20:39:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=PkuCAOvPHyodxuo0lxQ/4fnpf2Sh3u+TVEsU5WG4ZLs=; b=LhFr3xc+QIi15yT9Oae9jpSpHxcWNa7awPkHbOFwS/eXR10XrE3j65CBCqCofBrBHHbt piCd1h4vtKRJp9exHz/FYOV/Uml6HYiMNbOxW5G5FIHNeNNr6UIfRI2P+k8OFaXCjjDO VFqumIizmDokNx+2C+UZL3qmWuB/RFrD6sxLmkz8NUws7RiiK1tlMUaVgPJA2Amrngxo pakBkPK6/sYCOksrZPqKdZ9mP+ts1FK5yNqZsykWDNwA3dtS7PBEHrHAmC1j93hooJYR n7yf+a13unHl8fySdckNM5s3ZPI0j2nfsex/ExhCIoG41xJhl/RQaPnPn3h4rDf26eYh 5A== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtg0m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:17 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFsx6016233; Mon, 6 Jun 2022 20:39:16 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2105.outbound.protection.outlook.com [104.47.55.105]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu80ags-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DnDBrCtK6LKuHOl42KCS7T6PGO7zifUYgzXNNXK5LWziScwsq2T5hqpEmfyhgHATvGN4Aml1uL9wC5ECYgem5uIzQoqW0oRQAw9VfAhmyYDehmpmq88mDAkHKAs/u39uKCe4m4AT56TDSGDkgqgG3MlNM5zhcdl56IR9uXkkWNqyXKUksDE2RPkasjCyi3XNT3/iXk+pll/ja2lMNqqsK14SqiRkoYrkzCccrSE+UXKBZAY/hcW0KD5nsGM115sU5PQq46sJft5vpABHsdbV8Rn8q+9uRJnaOdjYcD5DaJ/qWiGUw+Nk7miKCRdeVBAzlNiEqdECK4AzJa6acpFU7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PkuCAOvPHyodxuo0lxQ/4fnpf2Sh3u+TVEsU5WG4ZLs=; b=eF90TXRyDnmFWdtu4ilRqMuwroGER0z3OeQVbtuoqjPjDiC8vEA6R6OpkhmacYorfzl2u14p7MD0MizXEp2f48e1LTuDlXVqMGhTzS39JtGKmRV1MQgfc81EJLCwCiIZvnm2tuPGe5H3qExnXa7jDyPAw5JAvUO4vtX+cZoe8cyiBGQw9AO+Bxp+oYtQFh+QK21ihiJLzdh4xY3a0N/5FRvakO5N8WON/H4C414wqwINYryYtjkaSce5jA7Jp9V/50oZOQ+d9nWlYLy7fmbchBy82uA7o1KrobHmSugws9oJL/iGxnMns2ZV17Ho7EUmxCaaasrrhcsaGoU70Gcabg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PkuCAOvPHyodxuo0lxQ/4fnpf2Sh3u+TVEsU5WG4ZLs=; b=T6qXXSaCbuPiXtpDyT/YpbXBkKsfpZZ3L0ZbMJVlW6glRAHk7GQVOp8IkDoVY0TsTsO6N1JeK1/vxv8/ULMw3C9fVf7+OMpc5vgiXK/sOU4P9RCnO/yO9mZgcixGxAwEYtna9Vgo7Z8I7HjEr1n3rpiV0h+D0/wZVJTU0l5RQSc= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:39:14 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:39:14 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com, alex.williamson@redhat.com Subject: [PATCH v3 20/21] vfio_iommu_type1: specify FOLL_HINT_BULK to pin_user_pages() Date: Mon, 6 Jun 2022 20:37:24 +0000 Message-Id: <20220606203725.1313715-16-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SA9PR13CA0176.namprd13.prod.outlook.com (2603:10b6:806:28::31) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5685fea1-4242-4edc-19c8-08da47fc9e54 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5yhIKur8HwoTUn0EmJLNAfSLUBvLlaN5yZkCuQrFQO5s+xKSTGoZqUAuz0wIfdDJ2QHXpzaVW0z92r2SHbauzLIxHVtj7WrDOFEgVgMFZt1UG04vPiq+L9/Hf9wdlInNrt3ImaSxrqBRx7hEWB0qk40jkNi3BlNtTc6vVt82Uwm9f9247JENLOFV+JYcdWFA7FSHGhdixfUhVdjjrR2QVSgfN9NuJsoomHISy4E6FJ5t3xuKX0D0eG00lGcBx67lLCnm8ZR4IB8eCgxhcWXNDgXA1gHsRvIaJ4uMYwVv3V3dNMhsp613a3cxf9IdJi5hkSfmk+A4oXsg0OCuuBVAQlow1T3P7eeGn6qWTyv5aFApvzZrieKe/khjcw6BmynDhzOzqmKliJu7x4wfz9RbRMUSC9EQsrriDOZKeEA/YptnWC/BAZ0dutkBqJcgXejtJhMWUP19NxpNXvYZtgZUZC8kXk3+49bJDdFV8igjcSduZlujUZAuH6aNjwrSZbgmZBa+M0skzR1H5KOxh9MwVde475LikmEPMoJ4eZevGRsQO1tavvmJEkNBSpm1AJr+c070h7Tfm9CC4P0Tvablu2xHy4sg7ty/Vy81Pog1WIy5UDsEXAhyuSqCVXz9Xb5LS0+6+NsQt1vtAogC1uaQUOucFvFV2u0fL8wRtiSexndKXJZgkIYUGbfPK5xjqVP9tzh8DaFOre/Ok+QfParocg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(4744005)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8num74lPgp/AFnZ626oETUsU2MYjBLPZ4LX7fGBD6XUw0AFgPbXJPDJcYrogO28tmTtKg2CRqR9m9z+g3pMqEkAHkTM4lMeLhQ7d/GNFebqPueJhLk21HCsreQCHP9h3IQW5A4Hj/QxL9lf1F+cvzWJn44c8DYl7roBI9AWgykQdS6pXTG8jXXM5zu7YUgmtQSE9jOFP6Z2OejW8jijutSE3cVbePZdYY2CfX6cthjnbWnfK9QAd8mi0poOcDyCZnGtfR2lQZ/lCAmTJ5j3gwQQ0so1OcQbLDTT5kWqiM7dW+ATxtVDSa2V1GHgcbHqc5SAAFpwhnTHOWILA2RPQv5IUfOP+pZ0Ow0L64Q7y4d+QM7/TrH4S+PcZyLrtsVxT7KChKNDzQhNe6A02eLzvzhonx3OFE5ju82mHP1FOQYF4tgMQYfqb3OWJjjfT/nepYMaqDFZaU3wHqFhRxfR0GzgtnaurCKtjkqeef1LuzUdk9lLgMYqf7/tzY5R1EKTO3uMRiIaEx2o0vTPEVqTRKm4QJFjo6WCVYajeF83P+7+S2HB255dxVLDSHMQWH/ohDtIBUM56L3Qjxg2ZHswaXexQVmR/4HrOZPov/DaiYfZ1boxfg5jK2j5kphxqyV4F6Zl5safUEpZ7y/5EO3dTFqB2QfrTAB+m64Qi8oJarNV51z197JTPh4z3N+C5tC4/ncRqRTivq7ssxcRxTIzGICe5Tb819PBHvrWpCFGeb3cxvrxv1TC/TuLoUwlWcB6rOspkO7xv0byDYnHXo+/iIrICxkHJ2SdwvDEOi2ALOqNHmZ/9hNL/x4gGLIN4BTD9ydMHUrbsD8IzlLqSpg7FpdtcjUmRcH2A1qCOM3HwGv25l2TiT+vEX3dgxlsuqlg6emv1SIc73wXR044ru8vs+EGKnG4IcrvdpRFGwdwCYEr17nT+ESqffz1k82BJ7W8QOBIdcZdVJYXNhpfQ0WSux6tN0apZfXVigw98KHXC8QCQ2NG5X1RYgLnYk87OYONroo3mdCwGI7QTKz9WiTNUFyTPdoSTE4mw4io7HQxPl7BybsQCl6PMLaoEIKeyt9MX74DB7C343VSSgN/nZWRKDpL1ogdSj+QJTNbPEowc0eH808gacpWaLgZjorHXK1aUJQQVwKONWZylcAlho8+f+0gd4WrLbYPxZMS0VHHZMwhUlJTPAFsPrFX7jV7oYaMEfjDyCp6YRjoe68+eEi3xyw270DXVRaBTT1i/C5uevY9HI00XkwK4Ptt1LQWQKjW7kfye1EdIs4WZ8IXeyqwvjzykqKwQrQHazKJyQPqac/CuFs9h47g+bdhmIR+Wb/aBjxHiS4fV+8lC0O/21A99ZQ9IDttL25uw57jrQswCkNdPyBaIvVDpZ0keSYnFnxgBJtEz14wC9f4m+xcknALsGVx1TD9cVx77/ND8BfeDEtpxV2WQQqDYkJwU86f3PDrq2u3mq0fB/z0P4kbMgJI/GfS3bXUR6efbDXqzxb6HqCffrUpNdXiJ+aTztb/QyFqhkK/cac/C6+MJZh0po6I5YCmRGgCaGyi0qfydZUYFuPBZfWS3/PvkadJC1gB2XiY3TZHrtIe70CLCig/HG5C3cAWfXk2+K2+307RYDyY3xh7KPthkazmpXIZ2WsETSSIRTc6fD17FbaKjtLcz2aljrLCF/7OSYiS6gWhDycMy9+6kuPYfo53hWOBzc/pStNGSIrWWQlZiNIA04F6pGiHJoaE2IMi7TgvE2MCRrudOljU= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5685fea1-4242-4edc-19c8-08da47fc9e54 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:39:14.1707 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gJIi+75btb6esrWQJjL+WL4rRszu3Ygc+KqmeGwcM+K1W+z5jV+mP6Y6SetwgKGEu7jwaebwv+2HYTrm799+LHrIO2/BDbiP2uiiewlejg8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: BJIfJo3FwDi-f3Nj_S944Ik_UlToqORL X-Proofpoint-ORIG-GUID: BJIfJo3FwDi-f3Nj_S944Ik_UlToqORL X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5D1D3180038 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=LhFr3xc+; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=T6qXXSaC; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf24.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Stat-Signature: s6xczpbmdusknjfxd39fq8h7jny9gfzj X-Rspam-User: X-HE-Tag: 1654547956-776053 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Specify FOLL_HINT_BULK to pin_user_pages_remote() so it is aware that this pin is part of a larger region being pinned, and can optimize based on that expectation. Cc: alex.williamson@redhat.com Signed-off-by: Ankur Arora --- drivers/vfio/vfio_iommu_type1.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 9394aa9444c1..138b23769793 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -553,6 +553,9 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, if (prot & IOMMU_WRITE) flags |= FOLL_WRITE; + /* Tell gup that this pin iteration is part of a larger set of pins. */ + flags |= FOLL_HINT_BULK; + mmap_read_lock(mm); ret = pin_user_pages_remote(mm, vaddr, npages, flags | FOLL_LONGTERM, pages, NULL, NULL); From patchwork Mon Jun 6 20:37:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870892 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E687C43334 for ; Mon, 6 Jun 2022 20:39:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18F4D8D0006; Mon, 6 Jun 2022 16:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 144708D0001; Mon, 6 Jun 2022 16:39:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E44658D0006; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C6F178D0001 for ; Mon, 6 Jun 2022 16:39:34 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id A23CC120AB9 for ; Mon, 6 Jun 2022 20:39:34 +0000 (UTC) X-FDA: 79548976668.01.7577E2D Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf09.hostedemail.com (Postfix) with ESMTP id 18FD814001A for ; Mon, 6 Jun 2022 20:39:16 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256Ivt9C017195; Mon, 6 Jun 2022 20:39:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=VxXLKjeuUfXGFnmu/XOWoPvkEANkl+2Tw+Bidk8Tj9Y=; b=Xe7c8DqePAoVnBDcNlMSpEmF7VrLvL8V2NqtgEEyYKJ1/RhYYzoZ9jBPL1TMUPgvtLcU ya+zd1V+gJuS3IALo9rx3Chb+NFH0Cie85cX7fjgwvleMT2Y7iBtgwjkUrAIzxAQDiqR jxEOsu8FDQDeLeCKDK8qaCTNTab8QG/s1T6LgThgCGAIZ354cR0/jIgkGwxvroob9M9o PuKd9Ql5FUvyMlzIECGwHSp3HUuTGrpT5XXeY8oCYENmiSTBwoekfXREUaYv8/E6yJXl XetVxNhwT7kR3xQyogIFbOstMJSZ3k8vZc6O6yFEKVOf+HUmmxVEDINTiiaXbAo37sSr uQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ggvxmtg0r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:18 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KF7HU001111; Mon, 6 Jun 2022 20:39:18 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2100.outbound.protection.outlook.com [104.47.58.100]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu91d57-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:39:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CXDOTteAVzsx6S0zE5kqtXJH2JtmH9ak1Utrxw57MXuKrH4aad3wf1zMV0Jj+WyNzlbwC09QtdVRuLGVDG/xCI56+srs09w/WmulAXIcDGNV3mPhPY4mEfuQggvTKJ/CdIRF8uj/fpiUjU4O7bZ9splJfrSi7GasDTpZ5NphOPL0HYwRE5tnf2NFresR10wlcetfYdS1WTqj8WRSdauMMFGf9y5fVQc0Q73IjWL++MD1KaKpqkDuaqMjDrlaGSl5QA9VWLyMWbCq3IxvNXX2M3571EwOCUYr6eJRRD2GFsEm8pULbJDlvJkID7JNibi0zoL3KO+0YbI5lscFWkuJkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VxXLKjeuUfXGFnmu/XOWoPvkEANkl+2Tw+Bidk8Tj9Y=; b=mIQavfGx2027J9PwMrW7fn+b17oTif26ldTcyw19w2Kb9LeaByC6c/TqvxpihTbydYA91+SeG66mp6wLzFu2ifWBVUG/6q2RE9UA7v1Fmh3ZanNwQs/EN7y+YzKHggZ95kE37NJbCd7JA1aWJAZ1fqMPyMcImH/IrG9tKLrNFdut1neQ3NukTd2/5DMhfqrhUnXJWn+tWPDfDHDlZQ0rzvBHvBFI1xbgTD/MXKsy+WyPTsCbbvBPuGt2lRv7SK8j0h/Ds0VjT3IwTc/uqpjET+jrwGIOAK4rOkyXEVVRpZH28IdJ46LZ5BEq+giU921cXWHYGsBgeFk8OiwvUeOliA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VxXLKjeuUfXGFnmu/XOWoPvkEANkl+2Tw+Bidk8Tj9Y=; b=wpoQbO3DfTyVuMfbz1XC1HF7+2AlWmIAjotous26vGD5U5z6knl+w+h7iBoc5q/byX3Jez3E0jTr1rui7CCaToW8iIWz4wyE3Hv5/HZzpSFsudgm0h+UMWdxpYw7wJrkz1gMObKRwluk9r4eXozgno4IkXFrj6837oVDdGrPhto= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by DM6PR10MB3691.namprd10.prod.outlook.com (2603:10b6:5:157::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 20:39:15 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:39:15 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 21/21] x86/cpu/intel: set X86_FEATURE_MOVNT_SLOW for Skylake Date: Mon, 6 Jun 2022 20:37:25 +0000 Message-Id: <20220606203725.1313715-17-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: SA9PR13CA0176.namprd13.prod.outlook.com (2603:10b6:806:28::31) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 60fac332-1480-4ec5-aa42-08da47fc9f42 X-MS-TrafficTypeDiagnostic: DM6PR10MB3691:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vAm0C1H42b1Ohuv+nCsbadx4fH3djyClnbZtSDdU/S8LQZ6gIMIy0u+RxBce1sxHeDknBDM5ieMOX2e/+3RmuR8kE3E12zrZ/m9h8J90VSU8tzsbfxOlz4VfcbK7xJRRn/gFPGpP6qCliZVRLlQRPmtUlwk+MzMAbBsmnd1HSYhsQtrxkSpI/0nAywufLwLxX7mQNcNiMifmx3ZHGfXks1srY9FeAxwB8YOrVq7rhi6HO2avDQFpGs0HziAi2ho69vJ4PFDvykRDEW8g3YaD5XGX1rqWuwxdeg2AxPMIH3TT3LMxFc58UvLE1Dwbo04gRlqMOwvNkDxJOPm5kkRAcczpNWW9ZAyjkHF2zO548A1215MIDKJgVVOpcnQRYRo1Tj3L0X2+MUxZlSCiQn7ugUTJBxuq0bfO0qxWX9G17unhGFqI4ZTm42Nu7KhKVIijhEE2WKA8WinYRHGhkYLXYjivqqXPJDFOvZVtMBFrIPHh/GJ6q1r4NDT7u92cVLRTQtrZ4M9dWNTXBz2NSBHaufh+JB/nHhM72FYY87aTgt90E1larNqDYOER1jpnW4KhnF8AYmr5bUvVwwjGYMYNvqz0wiRZd7h8CD8P0g8KQA0FT5NtlYAoGYgHNsGM6zioS2JC++EiVwb4u8WTimuyw0JnWz/jOnLBjc0o+05lsMl5QhaylxQL3zy4SODVmzL87pzcg417cMHLQrPIQ785eQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(2906002)(5660300002)(2616005)(508600001)(8676002)(66556008)(7416002)(36756003)(107886003)(8936002)(186003)(26005)(316002)(1076003)(83380400001)(6512007)(66476007)(38100700002)(38350700002)(6506007)(66946007)(52116002)(6486002)(86362001)(4326008)(6666004)(103116003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2vYNDn78uWcO+ivRCCEFMFf9ZMV8f12zzpfHRko3VmsN6D+YEUaB4AWnp3gW3iVJ4v3Pf9f+VM7EY5N1x10Lpm/RnnjmSQXZQd1Fbe2jKhkDXVC9cuuZ4be+wnQUkPNrXg2/J6ZZy5+ibyOlfWcP4Bkf/XJVvf+5ZMUCxG8OCi9SjO6l+opkQbY4+1Q1tijbiCIMlEkT8vTXrBVWNvKpiaCIwy4XnbozTf3Mrivv+YdtASdZRQ+FpiMX2TTaglg4tIkVul4BCdxt4ohYH1WjAQYyf1EMQaUOUgZVj4hUY+iLH8/Lq85E3No1qQHRftTU9+rWfNbWDqIhym3pjknVrXxZ30E1IxyG7cv1mrRi1cM9UpZMbJB1qoX7i+yTMfh8aHtX0cHzbXFNoyzYKPmfPqIFWHt+j3IplMpZ+uOUEiaMdJYhikY7BHXlD13YwGfjHb5L+xJX9GDQJ84R/9OGmI3jJ9ySNmG9HmF2xZoeWkisJrg+pr5v2GXzFtdJ6f293xtEf7kLEXh4SaabDvpaxO/feIcNv+hMa21hbUawXBF8dho7uD+8k6+fLzbtZMNWmGJER0Jakb2Q0u+jwn6GLXQI+d+EDMOkjP9j/4dtvpULO3SpvdaiDxmBO1R3kSNqkUE+WxEjgETBiVjcRssuxfvRtcngJAy5HiYNV7lq4EuBj5GESiHwI0jUxbmqILNKjFSkzrucBKCcCgrBQDGu4yphI2ehUKgVRkL9aaY9D46rhCTrFSYj2VvTPTXwJIkp/b4cvtRSz5PF6rDUEd4fUuQ8gMxOTAvwnIl+q+DuFBaEma6JJxQqeCf/FLroT4F0qhauCCZ2PHgoJ2sm2kowNDh/ZW1yCJEHvfI9pK1sthPaPZWb4Tc2zBKjQzGQsvdVVeD3+hJcniZ0Shtuq6glh0ShlM3sVsifRPX7xyv3pbqqiLUCWanz7/dcjrw2WJ2Bu8XfQwdT+QVoUyL6P3703f4yjUd0d3KghUinyouaDPoWTyAbZeffVypd9FuRtere6A7tsV+h6dCbLPLHlbRwUUHHKkdcsblvOQ4UyeaCSq3oBqXVAPTg0B8WYF4NMxRdOYis+3PDmKZw8W28WEx6n1LWJ3Am2/mT0QBdZjskmzwwsMGzyS0pAvWukaYGotiAC/L3QtWEGS6m1LdjVd07BewkwxU2sqEvTOUFoFsZFNbOLH2uu4eqPAf9GJPEoNwZMIPWBk6PAiEsHOWxfJuYAVbd6I46aPdQzORPeD4em/X4KqMdpKLmm0AhElpb8gX+7vSy/yNFUx/JMpWFdYVGW/w4QFD28UULcFTVpQhHFrz2jefGhKLuxtZw2SEbBmckD1hbc9tARoGKWLWoynudNbJTKkRNiA+lvtLSeyRMM2dWUbqUgu24gi3jHUTNhJqv0BXK+rhCGPLEnNdd1yqXFr/WwSnWtKzC1LVJ4znO+pj1PF6/dwGZkC7sq1fa/fAVum10dccGqKzfcsEOVI+hJb76K09araq++KRejELced1KlQA93L7X5GsUf9f7SM320Lrd3M3Aqu6tqWeBE7Hm0/N+fDe1IvHyuAXYyTVe5GYjbsbz6pXcMXcrUvPoNpMsezdBDztWGUdiBIP8aAo6KQdgRwQh4coAhUVktcIIy+SaQHnv9lVtGdkyX8JON0qKewoV59s9LUmJOM1itOXOawWSqQWqIb5KP4AssMqqAICWU4ozfG1P870aXynlrlHUYIY1DpDRGQt9oLNdCVsAbWH/mTneGdon9ZXOHWXHX6o= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 60fac332-1480-4ec5-aa42-08da47fc9f42 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:39:15.6560 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UjWT5pp78c7/X4sDWovwmWoOV2luxq0g/ZWiKFypxSv4xiXOp3yV81YTXkUOVASPV4BJPCM0nYYeNRS6ytwSBDeIMOKjn1SXwaMjNJllDQU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3691 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: KLPhswKR-ToVuUfvVO86phz6Mj6nJJeR X-Proofpoint-ORIG-GUID: KLPhswKR-ToVuUfvVO86phz6Mj6nJJeR X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 18FD814001A X-Stat-Signature: fnxhjx6abw9tk5jy889ezojfwpp5zzmm X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=Xe7c8Dqe; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=wpoQbO3D; spf=none (imf09.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-HE-Tag: 1654547956-961892 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: System: Oracle X8-2 (2 nodes * 26 cores/node * 2 threads/core) Processor: Intel Xeon Platinum 8270CL (Skylakex, 6:85:7) Memory: 3TB evenly split between nodes Microcode: 0x5002f01 scaling_governor: performance LLC size: 36MB for each node intel_pstate/no_turbo: 1 $ for i in 2 8 32 128 512; do perf bench mem memset -f x86-64-movnt -s ${i}MB done # Running 'mem/memset' benchmark: # function 'x86-64-movnt' (movnt-based memset() in arch/x86/lib/memset_64.S) # Copying 2MB bytes ... 6.361971 GB/sec # Copying 8MB bytes ... 6.300403 GB/sec # Copying 32MB bytes ... 6.288992 GB/sec # Copying 128MB bytes ... 6.328793 GB/sec # Copying 512MB bytes ... 6.324471 GB/sec # Performance comparison of 'perf bench mem memset -l 1' for x86-64-stosb # (X86_FEATURE_ERMS) and x86-64-movnt: x86-64-stosb (5 runs) x86-64-movnt (5 runs) speedup ----------------------- ----------------------- ------- size BW ( pstdev) BW ( pstdev) 16MB 20.38 GB/s ( +- 2.58%) 6.25 GB/s ( +- 0.41%) -69.28% 128MB 6.52 GB/s ( +- 0.14%) 6.31 GB/s ( +- 0.47%) -3.22% 1024MB 6.48 GB/s ( +- 0.31%) 6.24 GB/s ( +- 0.00%) -3.70% 4096MB 6.51 GB/s ( +- 0.01%) 6.27 GB/s ( +- 0.42%) -3.68% Comparing perf stats for size=4096MB: $ perf stat -r 5 --all-user -e ... perf bench mem memset -l 1 -s 4096MB -f x86-64-stosb # Running 'mem/memset' benchmark: # function 'x86-64-stosb' (movsb-based memset() in arch/x86/lib/memset_64.S) # Copying 4096MB bytes ... 6.516972 GB/sec (+- 0.01%) Performance counter stats for 'perf bench mem memset -l 1 -s 4096MB -f x86-64-stosb' (5 runs): 3,357,373,317 cpu-cycles # 1.133 GHz ( +- 0.01% ) (29.38%) 165,063,710 instructions # 0.05 insn per cycle ( +- 1.54% ) (35.29%) 358,997 cache-references # 0.121 M/sec ( +- 0.89% ) (35.32%) 205,420 cache-misses # 57.221 % of all cache refs ( +- 3.61% ) (35.36%) 6,117,673 branch-instructions # 2.065 M/sec ( +- 1.48% ) (35.38%) 58,309 branch-misses # 0.95% of all branches ( +- 1.30% ) (35.39%) 31,329,466 bus-cycles # 10.575 M/sec ( +- 0.03% ) (23.56%) 68,543,766 L1-dcache-load-misses # 157.03% of all L1-dcache accesses ( +- 0.02% ) (23.53%) 43,648,909 L1-dcache-loads # 14.734 M/sec ( +- 0.50% ) (23.50%) 137,498 LLC-loads # 0.046 M/sec ( +- 0.21% ) (23.49%) 12,308 LLC-load-misses # 8.95% of all LL-cache accesses ( +- 2.52% ) (23.49%) 26,335 LLC-stores # 0.009 M/sec ( +- 5.65% ) (11.75%) 25,008 LLC-store-misses # 0.008 M/sec ( +- 3.42% ) (11.75%) 2.962842 +- 0.000162 seconds time elapsed ( +- 0.01% ) $ perf stat -r 5 --all-user -e ... perf bench mem memset -l 1 -s 4096MB -f x86-64-movnt # Running 'mem/memset' benchmark: # function 'x86-64-movnt' (movnt-based memset() in arch/x86/lib/memset_64.S) # Copying 4096MB bytes ... 6.283420 GB/sec (+- 0.01%) Performance counter stats for 'perf bench mem memset -l 1 -s 4096MB -f x86-64-movnt' (5 runs): 4,462,272,094 cpu-cycles # 1.322 GHz ( +- 0.30% ) (29.38%) 1,633,675,881 instructions # 0.37 insn per cycle ( +- 0.21% ) (35.28%) 283,627 cache-references # 0.084 M/sec ( +- 0.58% ) (35.31%) 28,824 cache-misses # 10.163 % of all cache refs ( +- 20.67% ) (35.34%) 139,719,697 branch-instructions # 41.407 M/sec ( +- 0.16% ) (35.35%) 58,062 branch-misses # 0.04% of all branches ( +- 1.49% ) (35.36%) 41,760,350 bus-cycles # 12.376 M/sec ( +- 0.05% ) (23.55%) 303,300 L1-dcache-load-misses # 0.69% of all L1-dcache accesses ( +- 2.08% ) (23.53%) 43,769,498 L1-dcache-loads # 12.972 M/sec ( +- 0.54% ) (23.52%) 99,570 LLC-loads # 0.030 M/sec ( +- 1.06% ) (23.52%) 1,966 LLC-load-misses # 1.97% of all LL-cache accesses ( +- 6.17% ) (23.52%) 129 LLC-stores # 0.038 K/sec ( +- 27.85% ) (11.75%) 7 LLC-store-misses # 0.002 K/sec ( +- 47.82% ) (11.75%) 3.37465 +- 0.00474 seconds time elapsed ( +- 0.14% ) It's unclear if using MOVNT is a net negative on Skylake. For bulk stores MOVNT is slightly slower than REP;STOSB, but from the L1-dcache-load-misses stats (L1D.REPLACEMENT), it does elide the write-allocate and thus helps with cache efficiency. However, we err on the side of caution and set X86_FEATURE_MOVNT_SLOW on Skylake. Signed-off-by: Ankur Arora --- arch/x86/kernel/cpu/bugs.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 16e293654d34..ee7206f03d15 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -97,7 +97,21 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); void check_movnt_quirks(struct cpuinfo_x86 *c) { #ifdef CONFIG_X86_64 - + if (c->x86_vendor == X86_VENDOR_INTEL) { + if (c->x86 == 6) { + switch (c->x86_model) { + case INTEL_FAM6_SKYLAKE_L: + fallthrough; + case INTEL_FAM6_SKYLAKE: + fallthrough; + case INTEL_FAM6_SKYLAKE_X: + set_cpu_cap(c, X86_FEATURE_MOVNT_SLOW); + break; + default: + break; + } + } + } #endif }