From patchwork Mon Jun 6 20:37:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 12870887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C31FC433EF for ; Mon, 6 Jun 2022 20:39:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C7A68D0009; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 275CA8D0002; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F04C8D0009; Mon, 6 Jun 2022 16:39:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EE8818D0002 for ; Mon, 6 Jun 2022 16:39:03 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id CF9C560A50 for ; Mon, 6 Jun 2022 20:39:03 +0000 (UTC) X-FDA: 79548975366.22.0DC6941 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf27.hostedemail.com (Postfix) with ESMTP id 0008F4005C for ; Mon, 6 Jun 2022 20:38:58 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 256KC7oI032448; Mon, 6 Jun 2022 20:38:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=sf8L2jPN0S+qcGllsHo55AVeANBqjQC5Q48G9fkQtbQI2IR9KyFH0GcYu654peq00Ctd mUMvzXYX25e7onDYvk2hIwfMMcFKYQXMaLkesHmuL6kNo88vPSO9qkhwVo4lEI0E2ZQi 5RraKegRbBGwna20VjoKLGnO/yC+yNh8/LCkZg9PXcxmGO7KfeEHbgu4OIkZY4rncWYr Qu17QDVFGULEo73tYoC7gMdfy7Nx9ImzGzHSf1EnQdepiiWl5bIO7UiD8RIVRMnL7lgs 1CdRqyh0cZIVKq64exhifKGWQxiucNQjJyu8XJf+2wUhNV6ReHCRZFwsBiRhEkoR5TbG nw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ghexe9gfq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:45 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 256KFBbL001327; Mon, 6 Jun 2022 20:38:44 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2104.outbound.protection.outlook.com [104.47.55.104]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gfwu91cv3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Jun 2022 20:38:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cD3IRHRoI4wKMmegAhx/yZE0QjkKLxSzGk7dUJ+5Ml8FgsboM5aW7SSPD6AO/AIndyVL4twTFOq5yGBv3nMNY5hIvSBSTYQrhMwgrQzCVfZupbEp+oIut0hClntgBkF9yq8ylihNBVs4vwY3SnjIiv+io/WDGV4hmn0M3jEzJQeFh3v6kJxfXMgfJ/ilF/1SRY+deeDlUoJUqxfI24swgOhwy9wVkofbx8Rxp13kT6vZp8ghlbV9pTMvBQEW4QCVH8OBxKD89Dt84agyQULF71rT3YUijgQ086WYIBQXA3RZlYgc3LDj4t8v/+NwoQtuHXBH5jxUPjlrsNXn0nDffw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=bEGY+LLjovOnM/8sIt/pas3QWdarhy/EyZ+K5unMz3ehaJrjoGA9TlSXIGPYeWS1441wBeut/5PrUBZ9awFri2vTxDUQb2/AfjEfOnUOh/LOadgNm8LjaffV3l4/FPqJrIWnWoHQ96CiIp/rwY4ZJa2Ek+1EgvzktyOsXPjBAW8Dff+3cjDPcgF+dhn10mQuOl2QyIiLJ2eeb9RqLzn87NWl2q9yQVkZHUqTCCgb6Dic2bHwgy+SnkuPcrCqz3Kgg5N+jXG0+AvyepLNQPL/aX7BY5ZePAlZeJCm0ESRdcA3EuwOiTh7oqedo/PEMGe5UW98Wj/whEEiUUPe1OuyqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TEZNavbdfxj4wE2bNU4KqeGyaSH1Z903EFnUKJBx+B4=; b=a45EdSDh0pZXgt5jaG5yKucGm2VLPTjFRWC1i6UZo+RdDZC8BUQIkvoVnkCsYEuWj2cq8n7tMddVweAi0gn5oouuTObPG2YEn9RAV02GlkhSZP10SSyh27SQ0G17tyeC7Sks167tGqFFvISV7+Wg1C1VaVwU264JOMe5bCk0IYc= Received: from DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) by SN6PR10MB2734.namprd10.prod.outlook.com (2603:10b6:805:41::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 20:38:42 +0000 Received: from DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08]) by DM8PR10MB5416.namprd10.prod.outlook.com ([fe80::49dc:6a95:a2d5:fc08%5]) with mapi id 15.20.5314.018; Mon, 6 Jun 2022 20:38:42 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org, tglx@linutronix.de, bp@alien8.de, peterz@infradead.org, ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com, jon.grimm@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, joao.m.martins@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 15/21] mm/clear_page: add clear_page_non_caching_threshold() Date: Mon, 6 Jun 2022 20:37:19 +0000 Message-Id: <20220606203725.1313715-11-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com> References: <20220606202109.1306034-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: BYAPR01CA0016.prod.exchangelabs.com (2603:10b6:a02:80::29) To DM8PR10MB5416.namprd10.prod.outlook.com (2603:10b6:8:3f::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4d5116e2-1d14-4430-1031-08da47fc8b80 X-MS-TrafficTypeDiagnostic: SN6PR10MB2734:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YzsBHFeVXkvQRRmznNMBorsNgAIPb6kSSJQGxZqUkmRXY26Dqni/YK+QLIxfZXc9sclvkEuMRdd9hX8/8ddVM3DnZLyQVl/NjCu6HMZmQ/gavPdpgtnXeN3ZF8vQq29nxn1X2eOIsiFdVaQm7VWArXv5BbiVf85++t4Q5MfT6sZev47z7BpBgjhcHLC7oqc09nIw5M5UJGvSuzNxvtVImbptyOS/BJOJzIk8vh/TF1t5S9IKL8ub8a9yaBwwCg51bOALsCecth9LHT/SuSEfow1XATz/rd7BGVv6qLFPeVqVG/WJRc8D75GSXvcIBSgYwnEF2FW318nbfNYRqU5NqPHhdF/LNR5DOuW+OnTJP/s2ahTmIbDAbtN4S09kF6iXNovV3DpBZkG6ILnU8eeqmgAw6Xa34wxY1PnRHn6SYkrXCetcacwAySwz77/uGUqUWqGa4/0tE/PIl+J1wCl4iS1GHCRw0WvH+dxQlAbaz2RAHhhte1k9DgZuSeLIRTyFooSH/UQoaNBp0XKIwjBZBXz7HfFgkRXwGRJ5gPgU5X0cUJr+T6f+gDm4/+x9p9XOm161TIMnSnnBCnrw9ZzdaFSTemPMBdj/wiVIaMb1yRFGPKMIG58uYZbTV/Hoig6Kap6pcpY4Jzwtkj9Zjh2hWu7F55Az2axJdyakXgvOJiGmry0byjdqK7GuT6wulw3mdPYJ7gTKXSxmaB26ThD2nw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR10MB5416.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(8936002)(83380400001)(52116002)(4326008)(2906002)(6506007)(1076003)(8676002)(186003)(508600001)(38100700002)(38350700002)(86362001)(6486002)(2616005)(103116003)(6666004)(66556008)(36756003)(107886003)(7416002)(6512007)(26005)(316002)(5660300002)(66946007)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QoJeFBYdwr+6StVXjjP+05MLL/ah1YBFGVZtJ+d1l38tRYYCb4myENnmq/EQSlizsCs9m1ulgB/4ulAKKrHHgA40jTqZQXamdJC/LWEu7bKzhnAtH+cz5+8bDqxwOinKqesde9jycwFgfIPE7q87WQpBbsHFP8U1RRHqbTjIXhvN1k08UEXdQD7oY32YBqbgLiqGM4kwdB4LmCQaHqi2qRuvniM2govKNmSqYABHK9Y6Pqa9baTut4oT8swe0Dt1+pOxwWIVP0MeXr5QXiQP1BhY6uqF5F3vzJbOKFa2PLYx/QFsZc3E8+MCGuhUGhKRl66OP7Fh0sg+LRsPGsqLyobj7XO32XOoJ01xzCQF1VFoKp5wOqDDKTf6uWAFG1VHN3Rj2dC1zF/1eoc3zTeqtWwD8ZLF6wyL9BKTlFEkZbTyHmY6lJ7K5/vPtMND0dSRPtYTB8+etS+se2fiSKZNbcJSpHn9V/SepC1olRK6q9KZ8GC2PHDYC/h3dAAApUCjxH1HOYDvm9ZNSO4N4mlYKHdowI3S1k6bUJpYmbDDeCzk8EwcVccqQ1iEbDI7yhqvUkagrWeb9q5KxftQraJbyrRbMqGXZ+VuLV2PxdmVkphgrknVG6+8r2KF51nbaRj/iIaRrqJYsoJt/YdRoVYuA8058oDTMg06R4VKFhuoTM2jPhBaP0Ma2iNtMoymBGM0pmAUzOTsZz5CW2+RL9rt13U1R66p9ok94YIydeK839OIhX7o1ADaLodv/JuJ8lcoLW8mL39x7glJDlU9+925vBd3IBlljXfchpFs7fDA+9xaRdR2hPITjVFDiCtm/NX4vezXrSJLZdNm6mJiRsSy8kYsXa4A0W4mEgt1S1/XXDS9E4sAk5lDAF16eG0xbxKsjyJHMkP7zPdfHS9CqLHCwY52pxfRfQl6C6JFZm7Vq21FvJx7/KAhcvRw7KNj4eE50xoFIvI+Gxr8kd2Pp9gAGsm2/3KSy2+1vIZe/m0o5EywWODaONMDUaA7bbfNV89P/IpQXmloYAj8eR3+b1kvNcuvnK+VYK9TqFCkakqxqyMZVSX6N9UrnP9sta7boCaZQzD2D0wloDoB703Qw+Q0+Q7EgQcM+XnbUBfbyr/8ujVOFqxi/x1dVnQTrPli3kp4eNvWjgrLo3yu80nANVsB9tO9mY1vOBxqF+b8uWix8F7mroIGNBkTz/T4amoF0Dv155pXVn5YEfzg1+gMUu4Auedn4WVe+l5umbvqr0k+pcgvDHWD7zic68MHnnRPH7EDQ7n7w7QtExZUrO/3tiY5TzR0TwhqOOYtBPxbEXus3l1kYNYkoiyuHn8YRBwE43xArcQ2RE48TBZwTDEIAAuefeXWr/JaL1+r2nCKsuRQpncsmDarCm8I7T3CQCHiISE8ofqGCY4r5VANANxaLX9MdmvhqGLxiPeDr/U3XRpOy/gRL68Scyu0bcSw1ccL5Y0MOWHK9jfvsRp6r+efRdikgOVehxavKqFJ3skL0zfEHnBYnC/kyM/qWYGBiTFZUQie/ugtuzZI1ZgWSvALqslDitFPKN/CiKhkX+uZD9aDRsurmJVe1TGiLlPOO5+TY8ukNGbf3oxBjc5gkqx7MJ+fmtLOcQO/qiqLZ4QlEZA+hb/n/iuEjdrnl/9Uti3pMaLlITx1HWJHdVoAkUYglxJzdOQ40HccRJir8qtobqTNeDD29PEd63bytlWadYoZl9akSh93p98TqiriV4mYMOBb4R9m66G6wDqSZBNzwIyX5KA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d5116e2-1d14-4430-1031-08da47fc8b80 X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:42.5352 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FWb+RWeYbgDQnonL6o4cNdq8JO5cYxgxpyQpwNKHe+ezdEHMHGr0dakaDrBYU6iymu/wa/QD40YllAjPs5S5fzhQYmsdfU7l3/TSuiVeOu4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2734 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206060081 X-Proofpoint-GUID: KGXfBLqyjjy9W3PpTADtgg0MsBKYboGE X-Proofpoint-ORIG-GUID: KGXfBLqyjjy9W3PpTADtgg0MsBKYboGE X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=sf8L2jPN; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=a45EdSDh; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf27.hostedemail.com: domain of ankur.a.arora@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=ankur.a.arora@oracle.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 0008F4005C X-Stat-Signature: o3sfnqxuistmxwfmq7tyz3kq5u9h89wr X-HE-Tag: 1654547938-587765 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce clear_page_non_caching_threshold_pages which specifies the threshold above which clear_page_incoherent() is used. The ideal threshold value depends on the CPU uarch and where the performance curves for cached and non-cached stores intersect. Typically this would depend on microarchitectural details and the LLC-size. Here, we arbitrarily choose a default value of 8MB (CLEAR_PAGE_NON_CACHING_THRESHOLD), a reasonably large LLC. Also define clear_page_prefer_incoherent() which provides the interface for querying this. Signed-off-by: Ankur Arora --- include/asm-generic/clear_page.h | 4 ++++ include/linux/mm.h | 6 ++++++ mm/memory.c | 25 +++++++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/include/asm-generic/clear_page.h b/include/asm-generic/clear_page.h index 0ebff70a60a9..b790000661ce 100644 --- a/include/asm-generic/clear_page.h +++ b/include/asm-generic/clear_page.h @@ -62,4 +62,8 @@ static inline void clear_page_make_coherent(void) { } #endif /* __ASSEMBLY__ */ #endif /* __HAVE_ARCH_CLEAR_USER_PAGES_INCOHERENT */ +#ifndef __ASSEMBLY__ +extern unsigned long __init arch_clear_page_non_caching_threshold(void); +#endif + #endif /* __ASM_GENERIC_CLEAR_PAGE_H */ diff --git a/include/linux/mm.h b/include/linux/mm.h index bc8f326be0ce..5084571b2fb6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3328,6 +3328,12 @@ static inline bool vma_is_special_huge(const struct vm_area_struct *vma) (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))); } +extern bool clear_page_prefer_non_caching(unsigned long extent); +#else /* !(CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS) */ +static inline bool clear_page_prefer_non_caching(unsigned long extent) +{ + return false; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ #ifdef CONFIG_DEBUG_PAGEALLOC diff --git a/mm/memory.c b/mm/memory.c index 04c6bb5d75f6..b78b32a3e915 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5563,10 +5563,28 @@ EXPORT_SYMBOL(__might_fault); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +/* + * Default size beyond which huge page clearing uses the non-caching + * path. Size it for a reasonable sized LLC. + */ +#define CLEAR_PAGE_NON_CACHING_THRESHOLD (8 << 20) static unsigned int __ro_after_init clear_page_unit = 1; + +static unsigned long __read_mostly clear_page_non_caching_threshold_pages = + CLEAR_PAGE_NON_CACHING_THRESHOLD / PAGE_SIZE; + +/* Arch code can override for a machine specific value. */ +unsigned long __weak __init arch_clear_page_non_caching_threshold(void) +{ + return CLEAR_PAGE_NON_CACHING_THRESHOLD; +} + static int __init setup_clear_page_params(void) { clear_page_unit = 1 << min(MAX_ORDER - 1, ARCH_MAX_CLEAR_PAGES_ORDER); + + clear_page_non_caching_threshold_pages = + arch_clear_page_non_caching_threshold() / PAGE_SIZE; return 0; } @@ -5576,6 +5594,13 @@ static int __init setup_clear_page_params(void) */ late_initcall(setup_clear_page_params); +bool clear_page_prefer_non_caching(unsigned long extent) +{ + unsigned long pages = extent / PAGE_SIZE; + + return pages >= clear_page_non_caching_threshold_pages; +} + /* * Clear a page extent. *