From patchwork Mon Mar 25 04:55:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 13601191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C203BC54E64 for ; Mon, 25 Mar 2024 04:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3801C6B008A; Mon, 25 Mar 2024 00:56:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32EFE6B008C; Mon, 25 Mar 2024 00:56:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F6616B0092; Mon, 25 Mar 2024 00:56:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0F2D06B008A for ; Mon, 25 Mar 2024 00:56:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C499B12090B for ; Mon, 25 Mar 2024 04:56:46 +0000 (UTC) X-FDA: 81934351212.15.E4F99A5 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2061.outbound.protection.outlook.com [40.107.93.61]) by imf29.hostedemail.com (Postfix) with ESMTP id DE8CC12000B for ; Mon, 25 Mar 2024 04:56:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=eu7jsqMk; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.93.61 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711342604; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=fyBLUPD/mCdiBYehrBqpjV0AQR8JIeoBZz38Rg0Wi+k=; b=NGYwdfHV/XXxH0bLXE7vMUCP4My/3MEmcLF8TiB20gCK1+nryRYYyp+xCvM8AKDByUWPet C3cWqsglOK7eFl5DGbTMbhl+acDKSexgSl0QQh2p3EWSJM9Iw+VTmQ54oADzHHPONBNzV2 B6fiPauNbQJYE9uiRz0vRiTmJo3kby0= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=eu7jsqMk; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.93.61 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1711342604; a=rsa-sha256; cv=pass; b=lcMV3fKQPuplvh2S6uBbun5QIOtmUyTWnQcqV78HML5FC9qdDdp/EDYNq6IMJlX0ksP7Zn ZJ20Z44I2taXIukvJrHS3ATY7V7ZkkBuBssyIGZoj950HBYLZj58V2M935FuTBLEYC8Ab5 m6kqT9XNRgfszRC642Aa3Oa/HNBr0/E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QS7/+W+2S8gsvhw3TLndSCrd7hNWDuS3xDrlbBcpUXoH1BqGg36Ek3dpPAXGgnCAZq4cxNQ+xUFo5PN+KBaScLpwUHr52ytBqRLRbe5zevjhVCstKzGUvHDeNKJ/B0J5VtenofjhmxpK+awQRZ33WAk/bcAEUI2Hlc0ZSV8AwVlrO3HrtkAN3qm/1EVrZPRaGr316e5+U5ZKwjYIR+F+X1Co1TY3RZr0Exb/kthSgM5aDO9Y7BZS3vW1DoburUMikpkRuhZ8vqzJkFtTgKsDXDOeqshscWzlbo4yHPLu5t4fn6eh6+1cwKP4XhF4Pj30YdyJBcCeS6HIK+iB0bTOJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fyBLUPD/mCdiBYehrBqpjV0AQR8JIeoBZz38Rg0Wi+k=; b=W19P5BjfaWvzKg09Qah8oLYmICk9snlXdzCI4yyYyueopxlfadOuq94B4GtxfkiaBX8nGV3XMlywMhmAg30KXg5TjS9Abc9ChAM8O+th+W0jP2HxkUMI0Nu3xXbd6JyfNKu1hk7ReZ6Da1klcygbcQHCudMI+p+zjfdtdbYH/6xEHOerJKuoFsL3/tyAfK5RLFW0VukJPBQZrz5foSv+J8TMYl/pyKDk5aSr9Sle6hEAbU7hgPfXiCWOJMi761gOVQu/xbIp2btUplmWPUgehcAlRX9yuh4Pxi1TZk1vu27QqjWBzJMKDDL/mQ210xLZRMVGXpCKNKAkShzDE1D96A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fyBLUPD/mCdiBYehrBqpjV0AQR8JIeoBZz38Rg0Wi+k=; b=eu7jsqMkrpxfeI4Nr0GoAbFqk+sPH6mEDTFIInLWwzn8/008b5KqjEsT5Di7+PQfaQeIeuayf0HjSziUNizlqTLp8GUCA+pJ4N8sIw9J1dtjF06pqVlghk0XNDPIoSFHzDVRHFi54Ttki/X6GkoH2qF79K1h+bWPBNR85I6Waey/65vMBrYCnylk6ZSgXfVYVv3RNqweDZrpt9WWGFB6sjt1lP6V6/iVEH6IxMSOux/97khGmxqDKONSh0+wN3WFMVJz9ox2qnjuItTdmWIVg0LiCTouKXBdrH0uyz2rDtm6dyTtj0s8chKNMMmnjFWxusM0jHCsv2lncJ93mTdxkg== Received: from CH0PR03CA0335.namprd03.prod.outlook.com (2603:10b6:610:11a::31) by MN2PR12MB4174.namprd12.prod.outlook.com (2603:10b6:208:15f::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.31; Mon, 25 Mar 2024 04:56:40 +0000 Received: from DS2PEPF00003442.namprd04.prod.outlook.com (2603:10b6:610:11a:cafe::a) by CH0PR03CA0335.outlook.office365.com (2603:10b6:610:11a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.13 via Frontend Transport; Mon, 25 Mar 2024 04:56:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by DS2PEPF00003442.mail.protection.outlook.com (10.167.17.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.10 via Frontend Transport; Mon, 25 Mar 2024 04:56:40 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 24 Mar 2024 21:56:33 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Sun, 24 Mar 2024 21:56:32 -0700 Received: from sandstorm.attlocal.net (10.127.8.12) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Sun, 24 Mar 2024 21:56:32 -0700 From: John Hubbard To: Andrew Morton CC: LKML , , John Hubbard , David Hildenbrand , Matthew Wilcox , Mike Rapoport , Theodore Ts'o , Vishal Moola , Peter Collingbourne Subject: [RFC] mm: page-flags.h: remove the bias against tail pages Date: Sun, 24 Mar 2024 21:55:19 -0700 Message-ID: <20240325045519.222458-1-jhubbard@nvidia.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-NVConfidentiality: public X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003442:EE_|MN2PR12MB4174:EE_ X-MS-Office365-Filtering-Correlation-Id: a6191545-2444-4299-793f-08dc4c87f5b2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MeAjrvrEnbazyAMDOPEBU3bX8PbhNCZkgDoxWTpAqWkC83/EewDiIxscqwd47B30FxmQIhxP2QyHMnqhTj8Yej0AL0hFMD8Re/vxn9yLOEOeJUNwmapnyEr4krETJYXtUUe9fgHRp/40jTA8pq5bst8Y6NJVJi0AbbcyeX9AK2LLIyk0FOcRNW6omP+7tEFkoi/NVcpFXzCRgFRcbO38lg33ajByjRUnQWsBeg7lBb3hoCL6APssmPRdCCgsyQTbosTkD2WkLSf+kxlpifKxVVTWigL0Su7dOcSih1+QZgjw3YnweC3BqKljIQpJWkobwdbsigubSucpNw1Uy8RXv0G8soeKSmIYYFwJnbDb9C4ECdk8cJ8FxuX3TCa3Sf2Lynb9M1M4v4Kk5IbUOfGqcn8sul1IFYUg9sQrxKkd/svMEQzD/zd3SB6XBYj/SXQdN6J8A8O/55hz1tiF/lNQxpC4hplfbwBOJxxfneTjFvMCjoCA+opJ7XeO5w7q3WKXZo2WHjBIljjBHDHETMyAhJuT3uHAgPAPAsiuVAw1NYlGGOld8JfK7N4NGD/9qrwqSZLT8JaXgQAiVPkRAMN1uqoMOP3TMMcVhNgiwWJg+b+DvebBHM+R9UtmhfcoGDwxujQDW2X4ekIs20Ln1CbEa2eLauMn2HT6kRimAA/grbERlcKnnNTDBFMBCtW3txmarA4QrVmsSSY7wzRwQCHRB7qj+yMPctgku9XrTYRMg+Wo++Ho+S7IXOGrUxBwY8Tz X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230031)(36860700004)(376005)(82310400014)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2024 04:56:40.4771 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a6191545-2444-4299-793f-08dc4c87f5b2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003442.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4174 X-Rspamd-Queue-Id: DE8CC12000B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: bmsmh5yk8qscnhphrj9ndppyk8npthdr X-HE-Tag: 1711342603-975640 X-HE-Meta: U2FsdGVkX1/cd2x6R0wkb9V6PO1Nc7bCVb6NxdzD3zJYkVCnaU4a+RA98+QxAAsHCDl0X/640fh5YJxYVc2tPm4pVXxvtphcAqEXkHPLrtcUICBCN2KTmjIQjnT1oIFOSVwgxons7OllDq7jPkBQQ1xYjtfnNxlzYfq78iqoH5ClPd72XqNKcQz31kFL1gm50mpReAl7O67O/0+iUMy6fhG5rMcuLa5L9pAC751VIM4/iAI/lfAHg3DLLUwTEDLO7T5V1ssEJs2x6iqHDacB10YYmT9iUBf7jbeK/SGnUKyKyq9q3wEcQ+J6j98QId21+1ISrayOdDj9NMRZrq1pI86e5Wq9ZO+H3Y6nRcHoIcS1qwUyC4IXWVI7pMrRCsEZDgLswLT5+qvUtfbmS2xwLRrDQQdil1fTNWHYJ6t64pAqQ1mNVO85pY0VidIs/DmqdBze+i0ZNHzTps4exl5YTZvQOds5SyG4NZrbeXeBhZz+qFOwgHfIszG/jf6lP8Cw8BPuacGwQdXcvZEkywzZfAynjllhZel5AomxJXTj+iN8lNJxWsSN9WnPDO0j87TXZBzsoS+rOWzBJyb3RTMWdBblIt2GIF/a34E+/FiWUMMft4D/OjE/yLG1rIVWllnXeA1Ib+h+NS3Ad2jtCILozEcJfft0FhSnHpcjPyBFHKVNkrHWcjFDzIT7c7/kHp8N6J69DK81J28gqVFh/f02YvUFrS8cOfTWvFlolO9ad0ryC33C9m8Na1F7ZxXEQgpj1UV0SbaqtGptWXcmklHMFYBmonQ1Wvaew9EKd+hBNORJn9uBk6gxpsnYxn9/PbxIZv98FWM0IUKQm0Ue9ShrcXGsLwXYkYipmqVoIA0i95ofTgKiP1520c/Hkf3nBVBPMgyjnf2xR8VJkANYyBi7iBBJlKCug92zfdOh0fTTGO3b7rLp7sAJkd2mN1oS4LPyBKN267qQYTnUK3aIyKu 0dp8OePD x/uRphMGfsFrMGqG1qNnYztyyzJ73QbaNCEoyLD+kFo4PtKG4XWZRGCyuykFsqnlAj6q7JrbxaEhAbKqyLy8sd/pTg2hb638wwf6vaY2KIOjEVv/CNVMFuIzlD9H4BomKikXsVFX4TTxFaN64AWxotoLzvP6P8g/C30xDrxsjPhsBwWNHElJGSZmoYjy2bkV84E5xau3tZ6EHIa+QkvuuvK6A49FCE3j1TW7Z6kHq/EzjGmnsqe2V8O52no+tfMUkuULbjF9eJFZQJP6lOZ9emjwUWxb6AXNgoi9NINAX93J4WT10K751tiUQWtRCnpzodRZPCy5a6N1d97h9DxIMuAl1DESB5GKlh9FpIRvTYdbtZynqqKpo77+yntQpKJeOvBgubtpRxu+0UktxSo19g2RdwDVF0YTu3Pexc488eGRLhvi06txbhogLZL1NDeZTXbki82DKvx8M8pqtZ1QzHhatTInE2afAom9HmyV9SmvmwIMzjaQ/crs+4VzEiWBtglappFD4+JrnWlo9AQKFg3YHtg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: commit 1d798ca3f1643 ("mm: make compound_head() robust") added page->compound_head and the associated "unlikely" check for a tail page in compound_head(): if (unlikely(head & 1)) return (struct page *) (head - 1); return page; That worked nicely in 2015. However, in the 8.5 years since then, things have changed: folios and huge pages are heavily used, with more uses coming. See for example the various THP enhancements being proposed. And hugetlbfs remains alive and well. And large folios are being plumbed into everything. With that in mind, remove the "unlikely" attribute when checking for a tail page in compound_head(), and let normal CPU branch prediction do what it may. Cc: David Hildenbrand Cc: Matthew Wilcox Cc: Mike Rapoport (IBM) Cc: Theodore Ts'o Cc: Vishal Moola (Oracle) Cc: Peter Collingbourne Signed-off-by: John Hubbard --- Hi, Is this reasonable? I haven't gone out and gathered test data, because the original patch to create this just assumed that compound pages were uncommon, and so now it's time to stop making that assumption. I think that's sufficient reasoning here to leave out the compiler hint, right? thanks, John Hubbard NVIDIA include/linux/page-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 652d77805e99..ae9509c6736c 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -246,7 +246,7 @@ static inline unsigned long _compound_head(const struct page *page) { unsigned long head = READ_ONCE(page->compound_head); - if (unlikely(head & 1)) + if (head & 1) return head - 1; return (unsigned long)page_fixed_fake_head(page); }