From patchwork Wed Apr 3 08:37:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1B61CD1288 for ; Wed, 3 Apr 2024 08:38:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 224116B0087; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AFAE6B0089; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF2AC6B008C; Wed, 3 Apr 2024 04:38:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C1D796B0087 for ; Wed, 3 Apr 2024 04:38:30 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EA8B7C0DA0 for ; Wed, 3 Apr 2024 08:38:29 +0000 (UTC) X-FDA: 81967569138.08.53EB47B Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf22.hostedemail.com (Postfix) with ESMTP id 5361FC000A for ; Wed, 3 Apr 2024 08:38:26 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QRwkJuD+wwWKi9hQAlECfpj4bZ/nFN+mVshnbbGJMKQ=; b=rmwH+g7updDreBHf+2bTBDUp5muCn4tToMiQFxz4R3v/JlPQrujiFv3pV5Y/ggu7HFUaDu b+bbLJaP8JOr1OpJUIsCnecwS40DxhC9Byn8olRljP4jvyatbmigOSK0JapCfQ6T3ZzO+1 qWImdQRJX+CtGBlyS+03g/yR2LTn6CE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133508; a=rsa-sha256; cv=none; b=aMr/QZFFgWX4Kz+Zn/RPIWAfX7p6VYs86L1MtNXHOENsCi+Te2OafUdB/BaTK1zeKCA7be noSDN+cqFb+Perq9MC4eDhIWXRJZrY9QQcqNBGGjE9OILS/GrSwm7HqWf+l3OjogwT+piN QGfTl0FLlZ1Nf0aFy/maKwMXmAYyASk= Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dP626VPz29lXb; Wed, 3 Apr 2024 16:35:38 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id EC3921A0172; Wed, 3 Apr 2024 16:38:23 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:22 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 1/7] arm64: mm: cleanup __do_page_fault() Date: Wed, 3 Apr 2024 16:37:59 +0800 Message-ID: <20240403083805.1818160-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Queue-Id: 5361FC000A X-Rspam-User: X-Stat-Signature: gfj4x3e1uw8af5tpd8jjqam1qpas5yzn X-Rspamd-Server: rspam01 X-HE-Tag: 1712133506-848064 X-HE-Meta: U2FsdGVkX185c2EkHxPf5gqFZG7d/lnKLSSCmqGZBSfQ72V1uYp2i90dV1/pqmnUW7KZ8cAJQfj9SIsFLDNtqTKFR8GWV/KCV1MU5kX8oWe7eOw7ueKCNCgwG9BYF9wQ1iaQ9JNWb2NmUuKZQ27OwoxBEcqGHx5GQl1FjdJysb8xj+4ipHk9QjW7i2HyHy9SNkCyhVSzhvl0KLV9CizTa6NLQJ1Bn8y3YQlQeck1qE9BmxiGCIxd6oWG4jAaOYUbp0v6CYCCgce/40Ka8pNDqSZq47VimgYqkcodec+Vf/KTjLw38XymhTLYPdOA1OOjIWr4WbETb/d/HKYPfByd+L8q7tNWw6DFz7v7H00eHR6hIeCxzEK9UiIYUqs2NxlH9pbYD7DAR+ngx3WPQB1dzzkRtVRZEjl3so9eqHvTPvZ77TXRVdygdpBl04FsL5JJELOkMuXyvDYXQ/MGDHidQzn/wRE7kTBYe4fmGHQqy/7x8hChpos0bSaOqAnYrdS+8CEVEOiegP6z3VJXoBVi50ZXx/+wITMu1XdEtCuOh8AhcYr6MuERvJ2E2Shy3E9wPAU/meaKOqXnClRP/4hZgfRiITsi8JCCAH4wClNZblw7uue1XJNczDuRzSCoGkSJYBfS6K7JDJeQuSLZDb+ke8dwg1JT7ItxYBrdvnE1CalAQA1OGHvdnowODeLyWOVzuovI+AxQ1AzM9X0A3IhDr1TJKkgRPWT1rOdxKLz7Z6++TUgWba8DogK8xPL9QsdNI39nI9q4FUGby0AJucnd6IH8BRgdKzSlSndxBTt5Cmv9s4zMfvbnWfN9iF7adizLPRh3JB0Q7S5d6SUW+yYDzcmpFi3T1UF3ZXflhSsxnZ1slnW50uf5E9O2ir+68i8n30gFvoubB6x/KPy62t5kTBvQsdhL0QPGl8YKM+G/AGo6MA5P6c9dp/1ElTMiifeb7f/aLDqETjxyqxmMFNV GkhdAlMn 5XuB7SfSsqXwaM7mRof3pwoqidTDnVsl9d+zvuQT7jO1BtGy+GzuQGQACZtp7AOJm21fRL86mQTf3nfXdfaSp6J5TW+/cdHo8azo+t86P+H0SA1JZie079vhXB4T6N9Co+OIydVueMAFcg8IUENbK3GwuJpNVNDkYE23lEwJQeborzvCEb1zgVhKJc0lrGUcMbCdP3pA0m4K5nUV87AaE60HoyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __do_page_fault() only calls handle_mm_fault() after vm_flags checked, and it is only called by do_page_fault(), let's squash it into do_page_fault() to cleanup code. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Catalin Marinas --- arch/arm64/mm/fault.c | 27 +++++++-------------------- 1 file changed, 7 insertions(+), 20 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 8251e2fea9c7..9bb9f395351a 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -486,25 +486,6 @@ static void do_bad_area(unsigned long far, unsigned long esr, } } -#define VM_FAULT_BADMAP ((__force vm_fault_t)0x010000) -#define VM_FAULT_BADACCESS ((__force vm_fault_t)0x020000) - -static vm_fault_t __do_page_fault(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr, - unsigned int mm_flags, unsigned long vm_flags, - struct pt_regs *regs) -{ - /* - * Ok, we have a good vm_area for this memory access, so we can handle - * it. - * Check that the permissions on the VMA allow for the fault which - * occurred. - */ - if (!(vma->vm_flags & vm_flags)) - return VM_FAULT_BADACCESS; - return handle_mm_fault(vma, addr, mm_flags, regs); -} - static bool is_el0_instruction_abort(unsigned long esr) { return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; @@ -519,6 +500,9 @@ static bool is_write_abort(unsigned long esr) return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); } +#define VM_FAULT_BADMAP ((__force vm_fault_t)0x010000) +#define VM_FAULT_BADACCESS ((__force vm_fault_t)0x020000) + static int __kprobes do_page_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { @@ -617,7 +601,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, goto done; } - fault = __do_page_fault(mm, vma, addr, mm_flags, vm_flags, regs); + if (!(vma->vm_flags & vm_flags)) + fault = VM_FAULT_BADACCESS; + else + fault = handle_mm_fault(vma, addr, mm_flags, regs); /* Quick path to respond to signals */ if (fault_signal_pending(fault, regs)) { From patchwork Wed Apr 3 08:38:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2EA8CD1288 for ; Wed, 3 Apr 2024 08:38:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7537C6B0089; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DCA86B008A; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57C926B008C; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1EEE56B008A for ; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D2A91A0D6D for ; Wed, 3 Apr 2024 08:38:30 +0000 (UTC) X-FDA: 81967569180.06.2FA1834 Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by imf09.hostedemail.com (Postfix) with ESMTP id AFEB814001C for ; Wed, 3 Apr 2024 08:38:28 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf09.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dnCIzWeY1qqTgbyWio8O1U8KrwKcOC13DqOIgOib+lg=; b=EpM/uE9yH7ReM9NgibaeYjW/trGYWBWJB9vOQWu3VkN4SsC7WSLKEcF6AlaesfH24ZPYjV KTFCk/jiSlGeKAe6tHTBaePVPqIswvsLGZRxWTdbE+qHIzCWwoVQWsSPsXKu4ijGN06twJ pqvTbKfZSWseUI1RMHzhVh25k1c/9e8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf09.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133509; a=rsa-sha256; cv=none; b=qLPucekvanikOM36QM8SuU8eKqnq8XC9eeG3eywLg7RkvegfIUF2PiLX1fcLk7LzlZQLBp sHgb5wgCppskkHy9iw4v32ejhFt+s/C9m4kfTgMejJo8HnEpEVMZXweBWW18NVctuisyeK /STXVs4m6OlU4Xt8GA4XTMlStymSnIM= Received: from mail.maildlp.com (unknown [172.19.88.163]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4V8dP72twYz1h5XQ; Wed, 3 Apr 2024 16:35:39 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id D79AA18001A; Wed, 3 Apr 2024 16:38:24 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:23 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 2/7] arm64: mm: accelerate pagefault when VM_FAULT_BADACCESS Date: Wed, 3 Apr 2024 16:38:00 +0800 Message-ID: <20240403083805.1818160-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Stat-Signature: z64kuww8buijym596xhkazt6o5br4s39 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AFEB814001C X-HE-Tag: 1712133508-813244 X-HE-Meta: U2FsdGVkX1/ihPxq7gbdmE+tkXA1esyrddlOJ0R1Nm/VNRACPXUNXG7KRvg1qQ2sx3OT4aWUukvpxs/b9BZLigN2qzKqIpdbFunbDtN8NZrhw7ldMbUPLgEIlLgwTYlHCdyMX3gbMbYvbfeEEkLI3JAFyDeOoCio9oMB4/PdsTbA0V1eUnK3k2mjLQ5joNwzTyk4Ai9ZsusOTGDEzYo7VFb7rYX7OZMcVMvdjuGt+yJ4NpPyzJD8oenioUpa/ThAN7eNu2Puf7W7QV6Y0bdTd3WPNpXvr3b4ztIfMhpYJEgNv2acjeil9M9fy5Et2D/UOmzoyMgF4r8FNRsgtlzIvJiEPWEF42NU22g342zFO121r8MEhFFf0nXgWjVG875cm9VlNLCI99poSvXQiYzaK/TrSdUsjbATWk4s5hS1WtcrWrPIy0xgn/q9xeHJP3I4JrjlWbW3Zpqksp6QdJo4Qh5FVblNVLGrTbnnl9hSmcn5VG/RAp/H7iogONN4SESrbhipiOvpPLTa7OI3q/3xQG6v3zgwJqpuq0xq3eLZscsYIDpblrsC6x6TC51F54wV69gnhdR3INYRYdgWTGYYdsMdewwUeJ+8ef2PFJ0f2vWlYmI30J8ll59oUCyJajMP75ZxXx7KYatscAjyBCa0BMMb26ZewmUdKUBb84x1/eyDd+NgzDAupPOwqpEQpDlQbcKzRJ8lZDJSqAW5o42HcD209abUmTaxYqhYoxbXLo8CsOJ57/b/EXtCZbqIEAM8QvXbmpGZvxGhYtKS+B4rLAvSIOWFQOmn7Mak9aW74HXV2Ble7jeMaia9obUPkwYHu8G/isgn/OS4psymrdiyvu1pp6R2QWuOPSyOyh0lPgLcEHeL+py5ciIuureKIRJa9eSgi8zkvwT4GW08fnM02TCqsvxJ49EQHzQG5FRXjC9wBu2YTkrZJAQc0n623yu7FxCkoE4uD8hgXX/lyB2 NtEJQQlk APDiTm1dEPY6/Ld4H/0cUXQQTObf1TJC3rW/jCBeQBQ7ebtfRVQNCxiAPkyFO0ZPI+crPEn88LOYBZPxS7H0a6UvOCmQlnoC/J4vdfAWPurPPw9aY7uEt5Tu+/Xs0dKId216s+62gLcEEivj50voD2hdRbSmXq7Gb7pQi3aIteY6S5m8M02nM8klaE0btWYi3/DscCKJZQGz1Ij3a5KJ+yPkR9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly set fault to VM_FAULT_BADACCESS and handle error, no need to retry with mmap_lock again, the latency time reduces 34% in 'lat_sig -P 1 prot lat_sig' from lmbench testcase. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Catalin Marinas --- arch/arm64/mm/fault.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 9bb9f395351a..405f9aa831bd 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -572,7 +572,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); - goto lock_mmap; + fault = VM_FAULT_BADACCESS; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; } fault = handle_mm_fault(vma, addr, mm_flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615370 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33A9FC6FD1F for ; Wed, 3 Apr 2024 08:38:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D49216B008A; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C81A46B008C; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5F066B0092; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 701EC6B008C for ; Wed, 3 Apr 2024 04:38:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 205C51C0F95 for ; Wed, 3 Apr 2024 08:38:31 +0000 (UTC) X-FDA: 81967569222.29.5F35AA9 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by imf03.hostedemail.com (Postfix) with ESMTP id DE4A92000E for ; Wed, 3 Apr 2024 08:38:28 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H0htoAApM/tV8/kO7sKeBTt3Gf8XMgCv7CkUu0xBLQ8=; b=b7NroV9gORX7SaOuM8XhiDJKnG7QtsSR5cYFkOFf0s8IIWDjTefojQAh5nnAzByutLwDVS uTniqD3uKzZqM19UAcJDCeBN+BuFYR06sIwcAZZpX0/lc4zHSErAtnaOBZguXzAgn6UpPY XocRIGi0d9KyWINoYTUYGVk2RINGuMg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf03.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133509; a=rsa-sha256; cv=none; b=ZbF73vUFV6nOMAq6BvR16X1uYcSFx3SfnGlt45NRHKlK0Yj70bhrcYhCMFW9aAhGI0nq6w 0AGxNOSHqpv2NJJRG6aTFWfrRIToXiVNiTEhUHj+LqWAKFqk1IPsvEgQGbBs8tF9G5nJ/w Omz/KX+ygCXO+uleM7VAi1U76Z7IysQ= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4V8dPs2hLCz1JB4D; Wed, 3 Apr 2024 16:36:17 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id C3C161402C7; Wed, 3 Apr 2024 16:38:25 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:24 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 3/7] arm: mm: accelerate pagefault when VM_FAULT_BADACCESS Date: Wed, 3 Apr 2024 16:38:01 +0800 Message-ID: <20240403083805.1818160-4-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DE4A92000E X-Stat-Signature: 9c6crxzebb6tqp8han4ti8wfyean1n3d X-Rspam-User: X-HE-Tag: 1712133508-581165 X-HE-Meta: U2FsdGVkX1/tEEj0pPIgZultoCauLj765FrRGhY5THgI7pUGWTsS1unsWSyq6nzDoFb9sBy7n3fEbwvm7nOkUoVlPpw54c+Olfmz0B0+m6svgjzTdNA2UH2WcXC8pKFTTaJz28e1sP5l0PVCcFLVoxJg1ShCfpSy86HbNMwVGILAiKzcM+mPa0govl1aoJWLv1HMLK2AFAYF2zEgfX2esRmQQHlJPwec2xjB1tBmnaM1qZeA5X7RJPjjtMwZb1oxWJsVRfobrw38ZqFxWx+jz/9paIGzStX+tQGXIspb9/0torN+eJ/zr+UiHePioU3UnwM1yOJQnQzUhSFDCwJgbwzlUhFmXvZz9atU7kTLPh9K8Vzf6wBFSTVPBneVAqjVC4T29egkplm1wblOtL1+7DETXpge64oidU4XkR16neHOZTk/j0Bxf5VVUHuykGAJ6x5TvJocDp9LNuzp7Q0U6XQeMNdhSyFO1jNIVwjusjowjCp1GAspUr/pJo31AbTzRF12zOJ5gh//GLU5XAv2In69HJ8wLJIURZdJlgCoyOmuSJ8sLGszvFMcl/OMhStUgkTjMvqUSv4CC5Re0c3R6rIBzvmpTCVx7YFWj4r32MZJUwsIhd8bhENQVi1CmY64qoJL6bzp3Z3RetnMk1rBc7N1g304vvy4o906WCWkm7EMGS48X/CnGBqBpwJk7JIE5rAYktFIFCayXJXUiS491pgL1Hhj6zdPPUw1lXpWhFiZ0DeVL0P4qYt6jtKtxsRHjfWn5VMZw+aBTA5GkzR8/+Ky7oqBPE6UtjMMKX8/tKNiK89Ra78jPl9FMrfL4W7a7AtJYBSO+rIeQCUm2e0R0SfX4MVZCBUxUwO+HeqooqhTbCVITCJ00xx5aDMQNiOKzT5BdXVeE8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly set fault to VM_FAULT_BADACCESS and handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang --- arch/arm/mm/fault.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 439dc6a26bb9..5c4b417e24f9 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -294,7 +294,9 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + fault = VM_FAULT_BADACCESS; + goto bad_area; } fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81B91CD128D for ; Wed, 3 Apr 2024 08:38:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C7676B008C; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECD6E6B0092; Wed, 3 Apr 2024 04:38:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFAE26B0093; Wed, 3 Apr 2024 04:38:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A19486B008C for ; Wed, 3 Apr 2024 04:38:32 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6C0FC80CC6 for ; Wed, 3 Apr 2024 08:38:32 +0000 (UTC) X-FDA: 81967569264.13.A300A41 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf26.hostedemail.com (Postfix) with ESMTP id 0755F14000A for ; Wed, 3 Apr 2024 08:38:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vhZLrHVELrtbu5DW0tOonOUKFnnhSmrXd1Sgos/yGkA=; b=qrmAUalkvfWui39hqHzv1UvQtcFq0GSNMZxzFx+HQmG65EgsYirx8Nhl52aY2P8kEktlyJ eQQPqv55ImPw5oSImmsL+pOjNwh3L69hXT8Z62vQYsHvk2WPJXps1HcSGBpWs9PeI86LLc vAA34tOdGr5FOY9Y0TfcJu24Q8/bNmw= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133510; a=rsa-sha256; cv=none; b=xPCEQulRBcXEqy6EvBcQYkCIg5nhHq6LcoPwGIikNK+quNT5kkwYmBHTLUUh5RIMdKJwCR D/CbzbtDGJ0kxXqZpXUZzZbQBhxQ35zpLlPcOQFtthDlYREMXV/H/fMY9M9AllD1yuNKd7 7ULKMUy3wX/1x+ZJkVjaKPPz3ad98kI= Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dP90NXPz29lXd; Wed, 3 Apr 2024 16:35:41 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id B1225140410; Wed, 3 Apr 2024 16:38:26 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:25 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 4/7] powerpc: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:02 +0800 Message-ID: <20240403083805.1818160-5-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Queue-Id: 0755F14000A X-Rspam-User: X-Stat-Signature: k6j6c14wk3efx3iozatm81ewq5ehzqpt X-Rspamd-Server: rspam01 X-HE-Tag: 1712133509-986337 X-HE-Meta: U2FsdGVkX1/7S76NOUpjVPItyc3SJiPHjhZvnPB1/6ktNFWJQf8cIRV4Zcw2mTvSU/Qot8gw7vuot6nteMrDHnsdxvFRob/qFDmMCJuX4gYku6wn22IzbTjE/lKntwX5OdVARGGdyY1LdXDCyiJ6Vbyuo/ubxxTCX88pQXGHt3+YiCBusbzA+OXX+MZrYN21sjuI68UgUIBM+t1e8RvPUM++s6PDcPz2N/rAEYt6HbyjCTpEAmK+Z8jfg60h5m1vowYS9Pfd6rIbN/2Yw6Z4jJGAKB6CW7Q7K0J52alZwKTvsVhnsS13xp2vYRuFvMu7UDMgWScs4BDyI8B61jmw9wm4O8Z/j2t34tBP7TVsx/c3xDBcaR0TjBh6Y9yHBKOGwUfi/3wiop/PPh58ICyFarzaI6QQfvyzy/lEFhwQSusiF+LxL5ovC7FlKCE6prjum53pVdsy/Myfm2hd/K+fVGz4hldFLloVNeF5FpTrjkqtNVGOWuDFpg/wi+rUuYRFVO0n9GcptoBBw3wJ74h8SIjFfRlo/+mx7ukwTxq1N7vXaBsNqGYcgIq/QCuvD0QGDnJG62GCtUGcNWhe31PrJHER3Cma7Lgb/v5F9OCMo2+bnz0rV83/tu0hvNugXQ22HMxzn7Gy3dhEKOfqWQfCtAFIZdIbBEnqhVxC5uOmbLcwp+hEUHMxaK3G4a65n4SnHsuH9G3TwhOlPwe1uYK0kYAnT5hv/jjIotK/AUuKgKTEnBobKxfFd1t4lhjfYBqAV6MYt5HxJYckVtQ+bQtseTLNZgbkYWMeJcgNX5HlliNO7Ad7yPWXAxCmPjd6zlyhVjE07FEj1HKml2W2Jjy12ngDW2R8AX1xritXbwk+44EsgJ7Agizjdf+in42745Q/A5GupfhjvqLA2qoo1ygoa8RJBiTX6mNqwypDzbvF2t1xSUM5pURDU01rXd6dhtx2P+0ReBjuN1GFrqNNT2H 93Ntg+B3 ra46ymwTJjE8acO86POT5yH3TjzCBP2XTaWioOlv7R2SToGo/Mzigua5lEXW8tJerRNnSbSQxUaAlwFUo49CofFSNYtdOqszNov1m8GncTIOuSDl4yYK0NsxwkXZS2QurM/BC/iSCI4tUgPGfZRPPUoXThA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The access_[pkey]_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. In order to release the correct lock, pass the mm_struct into bad_access_pkey()/bad_access(), if mm is NULL, release vma lock, or release mmap_lock. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Signed-off-by: Kefeng Wang Acked-by: Michael Ellerman (powerpc) --- arch/powerpc/mm/fault.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 53335ae21a40..215690452495 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -71,23 +71,26 @@ static noinline int bad_area_nosemaphore(struct pt_regs *regs, unsigned long add return __bad_area_nosemaphore(regs, address, SEGV_MAPERR); } -static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code) +static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code, + struct mm_struct *mm, struct vm_area_struct *vma) { - struct mm_struct *mm = current->mm; /* * Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); return __bad_area_nosemaphore(regs, address, si_code); } static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, + struct mm_struct *mm, struct vm_area_struct *vma) { - struct mm_struct *mm = current->mm; int pkey; /* @@ -109,7 +112,10 @@ static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, */ pkey = vma_pkey(vma); - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); /* * If we are in kernel mode, bail out with a SEGV, this will @@ -124,9 +130,10 @@ static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, return 0; } -static noinline int bad_access(struct pt_regs *regs, unsigned long address) +static noinline int bad_access(struct pt_regs *regs, unsigned long address, + struct mm_struct *mm, struct vm_area_struct *vma) { - return __bad_area(regs, address, SEGV_ACCERR); + return __bad_area(regs, address, SEGV_ACCERR, mm, vma); } static int do_sigbus(struct pt_regs *regs, unsigned long address, @@ -479,13 +486,13 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (unlikely(access_pkey_error(is_write, is_exec, (error_code & DSISR_KEYFAULT), vma))) { - vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return bad_access_pkey(regs, address, NULL, vma); } if (unlikely(access_error(is_write, is_exec, vma))) { - vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return bad_access(regs, address, NULL, vma); } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); @@ -521,10 +528,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (unlikely(access_pkey_error(is_write, is_exec, (error_code & DSISR_KEYFAULT), vma))) - return bad_access_pkey(regs, address, vma); + return bad_access_pkey(regs, address, mm, vma); if (unlikely(access_error(is_write, is_exec, vma))) - return bad_access(regs, address); + return bad_access(regs, address, mm, vma); /* * If for any reason at all we couldn't handle the fault, From patchwork Wed Apr 3 08:38:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615372 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B13DACD1288 for ; Wed, 3 Apr 2024 08:38:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D72496B0092; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD1FD6B0093; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EBAD6B0095; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7BF3C6B0092 for ; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4E4F7A1269 for ; Wed, 3 Apr 2024 08:38:33 +0000 (UTC) X-FDA: 81967569306.06.979FEEE Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf09.hostedemail.com (Postfix) with ESMTP id 1887C14000F for ; Wed, 3 Apr 2024 08:38:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Myv/h7kbb21tl1+xfuWM9tOyrAQENAdc7X/Z/48zHSM=; b=Ej2Pyvm2GQrZs1wSAQfghNn+I1hH7pxnFBgpALVeDagrbdY9oFyspViy+zj/XTOhkBRPoO owJAVMZl7OIZfg7hE8VLf8zia5gp6LO1THzhIGtGb2ESqVoWBF78/mlfHL6zaKmmXXAKhn ThSBtec56DKL/fJe+PXWmC+PBchiml8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133511; a=rsa-sha256; cv=none; b=oZXQEnBT5BhwEqzYCnfSF6/pSELeQBFWCPLXD19KslBYNvOKrXWZfXPxJgxJEB9B+5mM3M XCfPHF3HfI390kZNpW7JfEq5L/nT/MRkGd2IhgKjnB4ji5NtPT7/+f8VOMb1R42Abz1dbF qHgcEKnZ/T5E8ksjbGftDpFL6MngREU= Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4V8dRJ4hz5zbf6r; Wed, 3 Apr 2024 16:37:32 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 97DF218007C; Wed, 3 Apr 2024 16:38:27 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:26 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 5/7] riscv: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:03 +0800 Message-ID: <20240403083805.1818160-6-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Queue-Id: 1887C14000F X-Rspam-User: X-Stat-Signature: mtakfqxxqihkm61nwxqctrt7qiz1dcks X-Rspamd-Server: rspam01 X-HE-Tag: 1712133510-722024 X-HE-Meta: U2FsdGVkX19xyk5sbmEVSrdNBWFmzJLU8+96T6fYd48IJHSVK2AbxJBqMb7uXV+th5KPopiGkiDFe0Thauog8u06vzYRWVGPjDTm84xeP+CZ8lsohki/f69K+m6HuU9GBMqXt3/2NAyDh2glRW5Q7pmjs0SSOcpUNiL48UZ1Lo5lOsjs20ofTZIbjnumOTP6T+Xhb+lD1R/bDDx0e5V6zQkgbEfYJ+bAHKOA8nSh/64Ndtf4UPA3cRghXbBbUPJ8GHadalOwDKr1Z4aeHbk/SexlAlHmCRGQsUd/8XYZjJAnMKYkzflUtTd35YSz2FYpSxfN9eJehTFh/47Uvz2iZiETegg6f6GEQR1emgrd8w5tNvup1qTj5nRW1Vp1zPP6tmBfXYFJIw3N0IoTdDmWeD7XzorV0QxrVVsPnRCcVJuyRV1RZ3/Qv+HVNp/ERDc4OO7ckF2gLnUqPbqlHq7JlN3T6m8xLiFsPsDJ1ghPJLA5EVtAicLD9gdqb+G3LYZsRmDRvwPAhkvG8PWzvCOMuRYcAdM6QUrHBQuXp/MQaRSIAEONGzADSLQLt3nUK+FA+mKbZ8bY78SjddJqNGi7iKtDrbmdK62HsZD2b1o/tEFur+cQLmDWv6sjLLnjvpQY4ri531L2EKxhDt0XhwxVEul2NsP99XmdZ/l8kZh8/kW9MAFZ8gvmB6YoGThGv8ygPdU4USg9unK32IWQm+H4c6kteYF3zO+1iybLRLoZOfMaJEMKbt3/KUHZRb8ma/SVZxRl/ZrwTsnm1HCsZvqD0nBas+GxWSxdKXkNOBydp4cMGBmCcp7Cdb+Hx241JE5WW4J0FXCQ6tj12Erq6O/ppDcTdXjAerYV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The access_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Alexandre Ghiti Tested-by: Alexandre Ghiti --- arch/riscv/mm/fault.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 3ba1d4dde5dd..b3fcf7d67efb 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -292,7 +292,10 @@ void handle_page_fault(struct pt_regs *regs) if (unlikely(access_error(cause, vma))) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + tsk->thread.bad_cause = SEGV_ACCERR; + bad_area_nosemaphore(regs, code, addr); + return; } fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs); From patchwork Wed Apr 3 08:38:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14A2FCD1294 for ; Wed, 3 Apr 2024 08:38:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 494BA6B0093; Wed, 3 Apr 2024 04:38:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F7FA6B0095; Wed, 3 Apr 2024 04:38:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29A356B0098; Wed, 3 Apr 2024 04:38:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 003AD6B0095 for ; Wed, 3 Apr 2024 04:38:33 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 86D72120DA8 for ; Wed, 3 Apr 2024 08:38:33 +0000 (UTC) X-FDA: 81967569306.08.802809A Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf16.hostedemail.com (Postfix) with ESMTP id 5BCDF18001B for ; Wed, 3 Apr 2024 08:38:30 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133511; a=rsa-sha256; cv=none; b=Tl8qn3u5H5cL6V3n7ntDUrNos0mwxofFJoggUs14D1HIItk1/imHYr1hvHeIxIFo5KcNyF lcpV+benSgKKpdcS9CScoifFG2USu8Gg0dJ6eG33SUR7gINjt1lqolbDjw1pBXZRPjMpOS jKjsjHyLB7kFeqI6EOmFVBaqOiEdNV4= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7HHcWzzE9AwVcj9rXnePyGty9Nb+tULGuXwsa1lDWyA=; b=vHA5GeWvkyN+RT/2TKIRpuQfPw6OgBTuA5HqUHeCJz1uczn+GWddWrGuUu3h/9nDOcmGYv pDRFfDmYa+uIBydNKYm6Ke2QDjzpJBqZhvthBuXzpJXphfGLPuG0v4O5C2vUrX2loSZTvc H3AJaixnByo2t3wtGLEQpmgw2L4GM8A= Received: from mail.maildlp.com (unknown [172.19.88.163]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dPB65yQz29lXY; Wed, 3 Apr 2024 16:35:42 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 7BC4E180063; Wed, 3 Apr 2024 16:38:28 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:27 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 6/7] s390: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:04 +0800 Message-ID: <20240403083805.1818160-7-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5BCDF18001B X-Stat-Signature: u7n4pczymobq78ht3oocuensg749xa6y X-HE-Tag: 1712133510-326071 X-HE-Meta: U2FsdGVkX1+D7AycTcBrg8elZkQQuCNJ18VsmQa6MDT+0MOBDPIcv4OW7sGZ3+VrOkJOiq3fKiiTgWiyMuOKl0cXPCmcLNrhZ6mrjVMxVtGkHr6tonJTsGVtdPsi8jXdVDt6f51Fb/doh2WBP6mFIOAidFosehbyBRXna0dPd7cLHjlfioJMWqHKQrxF/x8gYlka/mF30Prb5pJyXrxLpOrCyeif8iM4Rew9yGQFHDm9zDhwtfdkceh+C1w8zgyiua99Laeqt8r8xnlF7C33SP2HA4DCH7nuD0t6hKvytYwQqZWF13CfEUYtopMZcwb9pjHTvdbHwXg46pr9pxDk2jAgJrHpsJfFuY7Zm+WlvlekvZ/H6lhj85oYp6xwIWm7jiwEfv1AGdYnWp7mkBXSx0kEQddb8pBoo03huAattQGv+1IVCxFTlptxwH2IggA4lvGIei+RwTRp1t8C+3ZbeUifJUyNZthw5Pirv4lMctGphNEAFPSgVU+avIZyGAT8weJqdzrjmF6eCoGaea333sOShqu3k6vazZO9VTyUpMrYEtyaMc2hh/jQ5arU6pX6RoMcXKm6ZVSIDMs+6WZMvMuxOwpS38+9wF4X57aQFyCE856JrTOtiohBRzFkor+ZxFB08kuHGu7mqtNHc2TnL9Nc/DPt+O0rvMzi0/9fXaQw2mXxgJQLX0bSBPJ38KU29nIxxiahGfUvBK2OcsnjSfE60dtWr0AmNwsrX6i5/Dc2yVttli7tA72bhKhe0B27FGjr6PG0he9UglxHgByeMDYVzP3sQg/ix25fncXdeCuTMVK1XZt0BvnyQtHN/Pi74heCNjYuzsKEtA2L+vqd9wQi/P25w8Xhp/eyteZjXg8oNDeyd2DeoixPNPLNt3m1PWImB/1h2BcaLgBVPU4REDeeyYaGIy/3hgceVsbGSiyW4ESM+kK7jR+8CbumE/TK+LArnV++3PfM5UBVTzz W9O9+I9r SAYx+zZKiZwVkKAHWVGCpJdUXFdr2sHlXrDheWuLAa6A1rEtFW4oA5OcIkDTyUkNzzwsuykjWz/S878ryHxpNatODZcTeJSL7ssASX8H5QXg5vCvmF6GG0rNCwtIEkB65oNFeIvaT5J70zwT9Ir8lCExTo2wCBcoxeyWHNm/IA2CliGI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Signed-off-by: Kefeng Wang Reviewed-by: Heiko Carstens --- arch/s390/mm/fault.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index c421dd44ffbe..162ca2576fd4 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -325,7 +325,8 @@ static void do_exception(struct pt_regs *regs, int access) goto lock_mmap; if (!(vma->vm_flags & access)) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return handle_fault_error_nolock(regs, SEGV_ACCERR); } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 702E2C6FD1F for ; Wed, 3 Apr 2024 08:38:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EBAF6B0095; Wed, 3 Apr 2024 04:38:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94F2D6B0098; Wed, 3 Apr 2024 04:38:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72C566B0099; Wed, 3 Apr 2024 04:38:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4360B6B0095 for ; Wed, 3 Apr 2024 04:38:35 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0028880D0C for ; Wed, 3 Apr 2024 08:38:34 +0000 (UTC) X-FDA: 81967569348.08.8E7B765 Received: from szxga06-in.huawei.com (szxga06-in.huawei.com [45.249.212.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 737201C001E for ; Wed, 3 Apr 2024 08:38:32 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf20.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.32 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lmVlreGnPFbXRLejlXB9QDif0bxo2zNzlVV4g4MDH8w=; b=Jwja55E3UFPoTafZtDVmNs8VjMvfFdnrHC1YwoIzWXebtgHWwjaJwsdAfzZFfOjrmTM5N4 Wc84oue0WBOUQ0NmgtxMfw6m156dvk5xIDHtQ/UigLHFYYka4y2M3o24qWxNE+yRqExRU+ prWFxg8CggosTj2pn2FB8ory6pB8WL0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf20.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.32 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133513; a=rsa-sha256; cv=none; b=SGi43ZVnBEseCp+TcyH+g9iwSo03XV4oodjOetfmtjFqTZfcwDputrxlcxflTlmKiUbCfP FIds3jj0hlq/dlwFF+pnF99/OIpAeAY3VZqmkAWNnBqSHTkhaw8QwhWy7xbDGfY+QYUbWX YTebTTYRI/Y1C4+Ehh49yN83bW0iJIo= Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4V8dRP0T1xz17MxK; Wed, 3 Apr 2024 16:37:37 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 67C731A016C; Wed, 3 Apr 2024 16:38:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:28 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 7/7] x86: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:05 +0800 Message-ID: <20240403083805.1818160-8-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-Rspamd-Queue-Id: 737201C001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 8t3hcb7pewa69bhack7k9qroin9e7pop X-HE-Tag: 1712133512-892161 X-HE-Meta: U2FsdGVkX1819Fc9Njmvfdr1081TVsvBTQlHMQyrUDLjkYj5bJsSNdoxH6nRCseEmJwSRgu+orET/0neNporPXKihjBe6HvNsBe/ALIIXeqmttjkUWpZ+H3qGfT0a61x7zRa8Bu0vd+fwTg3pG5G0bs2BWDenhI2DFCllFoYc78u2C31HOtEZMpv0emqJGPnGJZV9GMSbmf5XBgUWawGs5Fb0sone7WUZ3vSvPQVeDF3EFkLhsr5+quFB1BpDwqkP8/DZ6KOtwK+n3ILu5zw65QeWYa3qyu1wPdM9JCDBOziEL8jLQEsCzTir+jXZsLTHXFKcQQUgAWXRvAZm5CJ6vZmH0DWNy6osRE4y9M/b7QXBz9sX+hfX1981e7QUfIQFTTrWE8UBl1dkLRG4QrQcn5WVnILs7lf0loYzk85tE4vq3md3zSFVzi6jd6ZTUwe9M+x6mgvbxrg1bpMWOUvranel1p4vjZMwtIneo/AkavHB6CkW1Efyg4Dlz2Jbd0XAAT4ip+kBqfTHmEoh6eyAt91HxDgC7bUkowni+bgKcyhwP7peIbCs9n58NdQ/Lz2tvwfq/x0K+DQuHo3McNgeWWfW1GXTGjT0SMFiutzu2VDSqhRmT47hElOT5TIUhzICNuX/9U35lWTGu4jzKCc4o2/4SOalwU9/i7LTUaBxFSMp5U2WSLWh8DYpS8cTGqyVGGvPFJ8h3wUL/VIRqGKAcEaF+5TavPhis8zbP42DsI5vYbtx5wPLEsdviMLzT0S5/iTsSw3IlIYXFw5ZI3yZwq/cwV1+joIotwH/XxhzFHt5TIuCG6f9e7IK+KpIk+Iu+tukJBTFG/vhUWTUz4Qs1+b1zPRDLlEIyYjTMuANo4F/NLxVOSDMeiMVZ4SGpyCQ/vFQ8/HfNJzHv05NOiMnz5P5XHp8WnRsgFUqjuFEBmPwux8pveOuGgsmmSbxX6qx06gLB5CHYCbYRovhYf sQHBS1K8 kCXinhBrOFi1eP9391wB+VJCbX2PkyvG1a3pIu24yiyhIciaEb9Yr0znNRikzFvlGEpLo6tamw1hfi2leVo3VJMIw3YueT+cfnHgiVBeACG5AcWC+jnXFXZt69mgMDEf2s//0n7FH9fHMizUI3lQ9akKW6u0mxWOAlP54 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The access_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. In order to release the correct lock, pass the mm_struct into bad_area_access_error(), if mm is NULL, release vma lock, or release mmap_lock. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang --- arch/x86/mm/fault.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a4cc20d0036d..67b18adc75dd 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -866,14 +866,17 @@ bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, static void __bad_area(struct pt_regs *regs, unsigned long error_code, - unsigned long address, u32 pkey, int si_code) + unsigned long address, struct mm_struct *mm, + struct vm_area_struct *vma, u32 pkey, int si_code) { - struct mm_struct *mm = current->mm; /* * Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); __bad_area_nosemaphore(regs, error_code, address, pkey, si_code); } @@ -897,7 +900,8 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, static noinline void bad_area_access_error(struct pt_regs *regs, unsigned long error_code, - unsigned long address, struct vm_area_struct *vma) + unsigned long address, struct mm_struct *mm, + struct vm_area_struct *vma) { /* * This OSPKE check is not strictly necessary at runtime. @@ -927,9 +931,9 @@ bad_area_access_error(struct pt_regs *regs, unsigned long error_code, */ u32 pkey = vma_pkey(vma); - __bad_area(regs, error_code, address, pkey, SEGV_PKUERR); + __bad_area(regs, error_code, address, mm, vma, pkey, SEGV_PKUERR); } else { - __bad_area(regs, error_code, address, 0, SEGV_ACCERR); + __bad_area(regs, error_code, address, mm, vma, 0, SEGV_ACCERR); } } @@ -1357,8 +1361,9 @@ void do_user_addr_fault(struct pt_regs *regs, goto lock_mmap; if (unlikely(access_error(error_code, vma))) { - vma_end_read(vma); - goto lock_mmap; + bad_area_access_error(regs, error_code, address, NULL, vma); + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return; } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) @@ -1394,7 +1399,7 @@ void do_user_addr_fault(struct pt_regs *regs, * we can handle it.. */ if (unlikely(access_error(error_code, vma))) { - bad_area_access_error(regs, error_code, address, vma); + bad_area_access_error(regs, error_code, address, mm, vma); return; }