From patchwork Tue Dec 3 02:31:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zuoze X-Patchwork-Id: 13891650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3E56E6C5EA for ; Tue, 3 Dec 2024 02:32:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A1206B0083; Mon, 2 Dec 2024 21:32:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5512D6B0085; Mon, 2 Dec 2024 21:32:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 418AE6B0088; Mon, 2 Dec 2024 21:32:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 25A556B0083 for ; Mon, 2 Dec 2024 21:32:27 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A578D80497 for ; Tue, 3 Dec 2024 02:32:26 +0000 (UTC) X-FDA: 82852073640.11.24176A1 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf16.hostedemail.com (Postfix) with ESMTP id 0D8E718000D for ; Tue, 3 Dec 2024 02:32:10 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf16.hostedemail.com: domain of zuoze1@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zuoze1@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733193139; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=Hgs/2q6eN5eaWB9YFzKEJzfRsp7At5i5+uThvijutSs=; b=DhDs6sxfWUjoe+slX+mKYxw+TYp7P+Xdu+eNMmr2v8no/wp412y7Pszopbuk0FsHLeqJxb v1ztyxj8LZWjjU15PRIh955EVhpF6HJ2fyWt0eKA7PgwhCLznDfeb1pBvipsDT9ju3GySV ldS+SKpVkEiudICW7nFIT7kAhJjV/rI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf16.hostedemail.com: domain of zuoze1@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=zuoze1@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733193139; a=rsa-sha256; cv=none; b=uWcwDBeLJz0EodSDX12SLzzHA96PFT050mA01esClYFO8kVjAZFIno9L3iNLObLXqksuqG bzKHMlc6dJatmJo/lEtsX3lu7rG0zqrJ8d0/a9DO8xtkRWbTu5PLuDI/c4gevNjy0amUgs 3QE/TZbg/Bw5ZoUDe8JGdYZjzupXta8= Received: from mail.maildlp.com (unknown [172.19.88.105]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Y2Pkg5qgPz11PYg; Tue, 3 Dec 2024 10:30:03 +0800 (CST) Received: from kwepemg500008.china.huawei.com (unknown [7.202.181.45]) by mail.maildlp.com (Postfix) with ESMTPS id 6318E1400D4; Tue, 3 Dec 2024 10:32:19 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by kwepemg500008.china.huawei.com (7.202.181.45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 3 Dec 2024 10:32:18 +0800 From: Ze Zuo To: , CC: , , , , , , Subject: [PATCH -next] mm: usercopy: add a debugfs interface to bypass the vmalloc check. Date: Tue, 3 Dec 2024 10:31:59 +0800 Message-ID: <20241203023159.219355-1-zuoze1@huawei.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemg500008.china.huawei.com (7.202.181.45) X-Rspamd-Queue-Id: 0D8E718000D X-Stat-Signature: u9yzatmm7h5oxz9zjrub3mng88878dr4 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733193130-585129 X-HE-Meta: U2FsdGVkX19AW9Dnf2d7XYuGIBI07xkgRKFHOVQGaStbqsWIt0n+kR2nkvFX1BTp5Njwi8iWILTWjd/pfuWpMu6lDB9UtRjTxe4eCh7VbOdLQINgkg4ZVBohO8juaDOyVvl44Z3Fe/krDk1dtG4XVFwTGBfyrUyerl+Q6grr+ihZnouEMYelZnSa9soUgE88f0i66Vfwu7D140IMGRg4I8nJSqzn1abjCSlDYnun6uU64PUH4zeyhJyHNA7hhy89FnY2WRgoEwkI9sOD1PCXKhwFcTmfiag155YXx5ItLJf6E37H6fvnQO+tNXToijwm93ys7tQsk0pMa0gIhoUIeLC/FwDe34yFQEqpZpFbr2ZFcHAQmSuAa1UFfn+GIK2tssZUYZed43q8NYHQsUNhhFaXTB/lNbMmLn63o3mn1YpE+dg49Fu2tNwIrXVj6N3GcMpsXRJJNEFdtkgEqR81CvbSshgXDXWEKzkHXrfs+d/sh0jmP/ZFj2RPcv2shVtG3rGCERsne72aBjYL3+c9owal3pGgs8xEQ/mWzHADQ66XYfvfyuGJW3s8+y/gEkeFzy3q2fMRC62VOp/3J56g+St1/uMCfK/RytRhNFDWFrGhpg3pvLvYK7RCidszX76Ymjv427SMnghff7iamHTDDAN5U/d1ADy+AZkvA9IpQ1q4DwvgdYpmpCOJ0Be+eWIxNwHNOggwKlytEvj701AU6Xe3oWuHa/ZAjrCl8G7O1xWU9kc18UNb3g8hjf9a/9AaG+aTFp6+UEmuik6tyFi6WdCoJUzvV5MU07WcjvjrzQttOE1Oho6Mc7JloDTluIP6ENT6kRlSLgDyqHHm/burDXw5jtLZuBfk2WEhFDpndaLLJ+MX+aNyNcgySWy5tWwfqvVYBAEInJ+xoBz1qEAgjbZpnUWxdPVtYMz7mYhNyHxT4I+I4ENZOpKRN653UjSdOYQojyKNuwtTN/5LVQX DNSzcxdW AZSpDmKQiNe8SZ78QF3zmSI75ixlxKvXLV0TpjEWSLM3mXroWNB9hLMIFz+DsSgZAIu3GfHTK4e5aYOW+Q3MOzq9Tj5n9ltLVBqGMzIb87SBEoEwWDMYcjnDp6Ati45m0rTDcFHFtXvCt4ZgXrqJj1OMALg59z4AN2qVFsfb6hwFRPFYY2Pj9iGDBtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The commit 0aef499f3172 ("mm/usercopy: Detect vmalloc overruns") introduced vmalloc check for usercopy. However, in subsystems like networking, when memory allocated using vmalloc or vmap is subsequently copied using functions like copy_to_iter/copy_from_iter, the check is triggered. This adds overhead in the copy path, such as the cost of searching the red-black tree, which increases the performance burden. We found that after merging this patch, network bandwidth performance in the XDP scenario significantly dropped from 25 Gbits/sec to 8 Gbits/sec, the hardened_usercopy is enabled by default. To address this, we introduced a debugfs interface that allows selectively enabling or disabling the vmalloc check based on the use case, optimizing performance. By default, vmalloc check for usercopy is enabled. To disable the vmalloc check: echo Y > /sys/kernel/debug/bypass_usercopy_vmalloc_check After executing the above command, the XDP performance returns to 25 Gbits/sec. Signed-off-by: Ze Zuo --- mm/usercopy.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/mm/usercopy.c b/mm/usercopy.c index 83c164aba6e0..ef1eb23b2273 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include "slab.h" @@ -159,6 +160,8 @@ static inline void check_bogus_address(const unsigned long ptr, unsigned long n, usercopy_abort("null address", NULL, to_user, ptr, n); } +static bool bypass_vmalloc_check __read_mostly; + static inline void check_heap_object(const void *ptr, unsigned long n, bool to_user) { @@ -174,8 +177,13 @@ static inline void check_heap_object(const void *ptr, unsigned long n, } if (is_vmalloc_addr(ptr) && !pagefault_disabled()) { - struct vmap_area *area = find_vmap_area(addr); + struct vmap_area *area; + + /* Bypass it since searching the kernel VM area is slow */ + if (bypass_vmalloc_check) + return; + area = find_vmap_area(addr); if (!area) usercopy_abort("vmalloc", "no area", to_user, 0, n); @@ -271,6 +279,9 @@ static int __init set_hardened_usercopy(void) { if (enable_checks == false) static_branch_enable(&bypass_usercopy_checks); + else + debugfs_create_bool("bypass_usercopy_vmalloc_check", 0600, + NULL, &bypass_vmalloc_check); return 1; }