From patchwork Fri Mar 10 04:29:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 13168727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D62BAC6FA99 for ; Fri, 10 Mar 2023 04:30:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C1856B0074; Thu, 9 Mar 2023 23:30:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 56C166B0075; Thu, 9 Mar 2023 23:30:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4344B6B0078; Thu, 9 Mar 2023 23:30:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3389B6B0074 for ; Thu, 9 Mar 2023 23:30:03 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ED9C41A1492 for ; Fri, 10 Mar 2023 04:30:02 +0000 (UTC) X-FDA: 80551711044.07.F8BDE77 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 2B19E10000A for ; Fri, 10 Mar 2023 04:29:59 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IUwzYC9Q; spf=pass (imf05.hostedemail.com: domain of 3R7IKZAMKCPwtggksskpi.gsqpmry1-qqozego.svk@flex--pcc.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3R7IKZAMKCPwtggksskpi.gsqpmry1-qqozego.svk@flex--pcc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678422600; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QkRgpCrNH+soo+fiEkY3prqCJENxuAn2pQWgX5GFHYo=; b=e4ANEAARNCWfJyftwxu18jpxze20JVCYO312rpZW4Ej2q1uTvrPZzJ8/H0eHoYDL+WozZW VErujdiamLAYCstQ5oI5wNk0mGyZoMJXndJe7GsyU0hBrioSEDEdBKM46gO0dzrmCZiNLd 0JEbZJo1cm0vVhAxLSchFdRFUCKyULs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IUwzYC9Q; spf=pass (imf05.hostedemail.com: domain of 3R7IKZAMKCPwtggksskpi.gsqpmry1-qqozego.svk@flex--pcc.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3R7IKZAMKCPwtggksskpi.gsqpmry1-qqozego.svk@flex--pcc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678422600; a=rsa-sha256; cv=none; b=N03pyavrQUhcLfGGkUH4gV0tV2J2BeFm45sOd8v1zkxtZfuyGT3veewE93FgzaG9f+3gxN xA7KU6LleSrd4Qz92i3RVbHGVf2xhhceIPPsrEdn6oWZV1qcdsxvXADu7YXg/fE9pYiWzw k9suaOQcDuPkJavITIXBYj0+Fc4EHWA= Received: by mail-yb1-f202.google.com with SMTP id 204-20020a2514d5000000b00a3637aea9e1so4660243ybu.17 for ; Thu, 09 Mar 2023 20:29:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1678422599; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QkRgpCrNH+soo+fiEkY3prqCJENxuAn2pQWgX5GFHYo=; b=IUwzYC9QwdzKEGQekhpa3YY0njrREkSmoVgmtl99u7nOXC6fV8I6Qj8xuVm2JdNjLn ubRsEea2cY/79WDUcqLNBHNxYbZC0yowAuL8uRs9dKMJlfl6FryHgcWIuU1DGLMuyXBt PEjg/9aOEgNcJYJvM7lq/BUNAYKdWmpfQbreuO0+RIIH2FkduGalkt7xW4w8KqiPD21w 548hOgBBPCkxL0InIdqn8serccLJf2BAmAuuJF5wGZH6yih+2Uf1yosBEuzcLLl+tdHt jMdq1pIgADFODU1v+bGWjfsUjbguZJAQaVh0EbM8oOlgQP5bG02Ujj2O//AQw742BkZw avyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678422599; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QkRgpCrNH+soo+fiEkY3prqCJENxuAn2pQWgX5GFHYo=; b=14Nd/v/1Uk01P2pwYjZwk07cVsf7oeGm8nRDY0EAH9sV0L/p/TENoqTfoGQA6PVv/9 cuB8sVHU3VUm72Wgf9zPr5Gn6RHDYM8p1lbQbTeTAMY6lXtMIsE6f91buHK8z6o7ukG6 xkI4wHtplxB5T9vx1jzoGAo3C8Js7H0Ngv2hOidexqBV7ZP1VilC5u/pJOIFDoJvgirN q9G8edtW0cfuFSuvLBWMlv7LLPUVRUhljw0xb+G85MNC+hgeqd7/5CvKxK5eSuzNq/PK 4TjTr1xE6bt/GRsmFtD4Aieq7YDMJ3s+3tWYUQxTDZ8MQt9U214zkVmZH9AHSjOPOBIm AE2Q== X-Gm-Message-State: AO0yUKV6RXphRK9Slg35Ar2Xec+fWgf9y6CkgaMntQipmpyeADZ/+HSg P+tmPeSM5BEInK82QX5FlVeBPGE= X-Google-Smtp-Source: AK7set/EOxMMKghq1Vn5wpYfyDCvnNJSf9PCauOlQqG2n14zxfpAuyizvUnSCrDxjZGqveuY2rprJzc= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2d3:205:4760:7b08:a3d0:bc10]) (user=pcc job=sendgmr) by 2002:a81:b289:0:b0:53c:7095:595a with SMTP id q131-20020a81b289000000b0053c7095595amr15950888ywh.7.1678422599352; Thu, 09 Mar 2023 20:29:59 -0800 (PST) Date: Thu, 9 Mar 2023 20:29:13 -0800 In-Reply-To: <20230310042914.3805818-1-pcc@google.com> Message-Id: <20230310042914.3805818-2-pcc@google.com> Mime-Version: 1.0 References: <20230310042914.3805818-1-pcc@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Subject: [PATCH v4 1/2] Revert "kasan: drop skip_kasan_poison variable in free_pages_prepare" From: Peter Collingbourne To: catalin.marinas@arm.com, andreyknvl@gmail.com Cc: Peter Collingbourne , linux-mm@kvack.org, kasan-dev@googlegroups.com, ryabinin.a.a@gmail.com, linux-arm-kernel@lists.infradead.org, vincenzo.frascino@arm.com, will@kernel.org, eugenis@google.com, stable@vger.kernel.org X-Rspamd-Queue-Id: 2B19E10000A X-Stat-Signature: goqbohnfh79yuyamhxoehmrse49pjz5u X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678422599-766214 X-HE-Meta: U2FsdGVkX1/1c/bMxnIbTHX+HEq7PggA9i0fJ8dxxIks8vhvQ4qpnQg06gpBn6ITdeEIFb0RjmT/ueUfvNDWmhbwWuP1PLh15sOmeA+gMPr96NsUUZYNsV+KmdtDHDvdzkAdWCigulfPf8o89q9lAIYRhSy9pOC0PIBi3iNm23jhtCdmaGNAMmkXx+nKyixDwXS+/lzJTLElMUC8wu8Iapi6/V993qF3ygasa8YbU/9/BOw62/gwXkdxJPMTYmbVP2nhYTbcmahtueCoaQG0FwqeWCelRq/PPEXapCjpyU5z0mD78JAGK05d7QJbzPhSE+o4DVyb9xGOnSQG38dkK3AMNVOTi2bHI8WvyAjI3hPRamQRpYwAe8Kz+3El8hTzgC3lRrFEKSb91nmuoK5W3FjiwrOJBHlPQguZ0DTDNoLJLe6+E2DvvmTWb9qpuKRkkIeTYSmVeKcULSxbU2yJ9Xt+lnVmslMNcQzdv5UzYZynVwCmkLJ5kmaD9n4U6fBMN0ANbLk7xNdvk/MCA0pcsO0VN9fchNP35MCHIUnjxWEznUjx/c0JgNBdeFatZdEZG/RPxT8t8FrcQADNViF2dCEQJ9EtCHI9x8O6lye853jEzQ6+rS9Z7xFJw4HsH/uRgaOBwgM8NXtM3Z6FKV6trLNVe3MhrdgGGfmjwhSSS+KfNOW/NnIEWhVFoo7q8KNMEu0DquzTHaszZPtU6SywbNXvL2ehMWJRyJSCtOlcSarMBw7NqDHI1NbNFYv/My1/k4xzu60aBqgvVP26RY8v/II4l5FDDwiHJGR4bVfGTo0YFvu/feuqPnrfTFQlqSuxp8tC5n5u6RRc6euz/YiZx0wnFRHou52eUUVBk2rM6ecNU3vczh+Ys5+Gvv8cMLrIV6C6/uMVqulx7UeNNqK5P08YgzCwT10b7TDSNWoNZNGuyKt4ijw6w8zkkq/44XjL62Y/34GzMktT2RVvZWG N6KwEwLl VmkWNRFyWjMZHwO0hGvPg7v0l0ioVZxNCbgcmK2oG3yItpNGrF2Drhj/zFC2D8jVKXmngi46ALGO1YDjxA/BGzEdOqTREojrAzGunWMRw5q0bnAXA+Zy01qiImRdYHdOGMHonYj2UyTNHJDZKtzK5VeZW9B5l5kv4w6h8ScoUum1xC3uZmh4G+WO2qtdPD2pKzFxOtwujzofFxl2Nzau+oOI0hd9huYlobA3UklhCLzYzLUHTzZ0Brflf/IJUOynQg8mafMSbDF4LreLFC/Fec77J5Gl3bdwU9PaDhjkrCpLJQ9HKNBOXeyy9Y6CZWvfv1kI6mEYCFcOh6Te2W2CIMz7k7pwd8PHl6fI0h/dQM82lRXe3DE31Ov6r14D92HqWvaAQjQIjXgvnPTKTcP4bmx693nIbB1Zc1uecPIFWHKsUcwEy8wgyI7U4Hfu3gNMpRLDrY8Xfqna3stm2RkJD6IvbfyVEyxlZ3DeEju1pTbEqla46FhUcJwlyU7kPpCeKkMACvyloVhzLmO4sAxExMzwlYdPJM3MK7JBE/yEFikh//0sB3dg/5oh4tprUArE4Rj0npgJaV7qxyHuwRAblb61jQzYGMm7P2LILSApnjsUdQXnzRT13NxBy4Jzrv3spIxt5E0FG4wd2hNSYSuhjJ+2EUZIpEOgbHondwc/taeFR54Was5gJ/cc3C/M/RtntsURqxxbjLSnw9AYnayqzBP3rNFhcE0jfw+48/dFu7UnSZgBPnGSTJu4N6Q59B7pv2C2IB3dqCDdE1lgQIE8QSLlMpkc2gtPBVf5cMUyZUjhUCaSfecVYYmNxORMQJ7et5muJViBiHMyAQOsSFpaJG/Tj1i5Cp8P6PmEsZ0a8R3QfCQ8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This reverts commit 487a32ec24be819e747af8c2ab0d5c515508086a. The should_skip_kasan_poison() function reads the PG_skip_kasan_poison flag from page->flags. However, this line of code in free_pages_prepare(): page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; clears most of page->flags, including PG_skip_kasan_poison, before calling should_skip_kasan_poison(), which meant that it would never return true as a result of the page flag being set. Therefore, fix the code to call should_skip_kasan_poison() before clearing the flags, as we were doing before the reverted patch. This fixes a measurable performance regression introduced in the reverted commit, where munmap() takes longer than intended if HW tags KASAN is supported and enabled at runtime. Without this patch, we see a single-digit percentage performance regression in a particular mmap()-heavy benchmark when enabling HW tags KASAN, and with the patch, there is no statistically significant performance impact when enabling HW tags KASAN. Signed-off-by: Peter Collingbourne Fixes: 487a32ec24be ("kasan: drop skip_kasan_poison variable in free_pages_prepare") Cc: # 6.1 Link: https://linux-review.googlesource.com/id/Ic4f13affeebd20548758438bb9ed9ca40e312b79 Reviewed-by: Andrey Konovalov --- mm/page_alloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1c54790c2d17..c58ebf21ce63 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1413,6 +1413,7 @@ static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, fpi_t fpi_flags) { int bad = 0; + bool skip_kasan_poison = should_skip_kasan_poison(page, fpi_flags); bool init = want_init_on_free(); VM_BUG_ON_PAGE(PageTail(page), page); @@ -1489,7 +1490,7 @@ static __always_inline bool free_pages_prepare(struct page *page, * With hardware tag-based KASAN, memory tags must be set before the * page becomes unavailable via debug_pagealloc or arch_free_page. */ - if (!should_skip_kasan_poison(page, fpi_flags)) { + if (!skip_kasan_poison) { kasan_poison_pages(page, order, init); /* Memory is already initialized if KASAN did it internally. */ From patchwork Fri Mar 10 04:29:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 13168728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29B6DC6FD1F for ; Fri, 10 Mar 2023 04:30:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C586B0075; Thu, 9 Mar 2023 23:30:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67E1A6B0078; Thu, 9 Mar 2023 23:30:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 398646B007B; Thu, 9 Mar 2023 23:30:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 232D06B0075 for ; Thu, 9 Mar 2023 23:30:04 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EF7D9A11EE for ; Fri, 10 Mar 2023 04:30:03 +0000 (UTC) X-FDA: 80551711086.30.42060F6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 38ACA100004 for ; Fri, 10 Mar 2023 04:30:02 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q2JEHoLx; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3SbIKZAMKCP4viimuumrk.iusrot03-ssq1giq.uxm@flex--pcc.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SbIKZAMKCP4viimuumrk.iusrot03-ssq1giq.uxm@flex--pcc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678422602; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hPpibPG6fh6YQoXzVokXBbAJ/B5/ZQPeLynzU+wCe6c=; b=apwT9cMXJDBNfYLuZagHoJgDHh2J+4DwJ4I64jR1TExxPRxf2SVcmF3ZCHIgxvCkQq7pBb tzcMu/oTdrLY85ixsJW9NrzHKiA7Ogt7ddJG57oZGFIfD1gh1mR6qCM8SEzwgqsP+yO0oR lXMy70t2+A219mhObb1NRY2fkZxLUX8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q2JEHoLx; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3SbIKZAMKCP4viimuumrk.iusrot03-ssq1giq.uxm@flex--pcc.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SbIKZAMKCP4viimuumrk.iusrot03-ssq1giq.uxm@flex--pcc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678422602; a=rsa-sha256; cv=none; b=Smqot2hOg0ghcwpNsU5tL4HZ0gH9aGt0TUGpi/MTPfJaEbKFwl2zTAwoGZdOVhav5iE8gx iTC+bMQDnQHSpbX+N2xIpBNfHHs2e1cA8qSJSftH+0TrpGUDaqPXM6oXJ9VQlF3248Ew6r WCb+aedOLAxN8U18NW4n3WsE5MTn+1s= Received: by mail-yb1-f201.google.com with SMTP id n139-20020a25d691000000b00b1f2dad8a21so4696523ybg.10 for ; Thu, 09 Mar 2023 20:30:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1678422601; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hPpibPG6fh6YQoXzVokXBbAJ/B5/ZQPeLynzU+wCe6c=; b=q2JEHoLxDKgc8uS3oejRmJaIA+dgT3DW2ASDjoDksocmCz28T8UmWZ+pzxgikldXIX reh64QIsDUBWEC2/1u4XmdUd1jy4lKDgdM+YAGJd85ctXFurByB2KmpgltL8JM4vXMaa lsGSvFwlzZruRBQ+ftdanrD/6+Re4n68j9mL6Nlgen2SPmdVseHBHf5yLbEv6GZUW7nq GacnDJQPQSwiPxOszvi7eIishEyEIV9xc3DRJLIDGrxlu3eFYIRzEOEgywLcduq2Ibs0 N61+4uasixJVbA+7vkDqbxVinEmMlxhCFo22aOBDFXgC4JEKjKIWbbUUmVoq34YIxCBa 1tzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678422601; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hPpibPG6fh6YQoXzVokXBbAJ/B5/ZQPeLynzU+wCe6c=; b=Mpp9htyZIZ/5afak4Y8WaqNw6w5Vz1hRGitUz+MnknnW+iuKMSGKe/SLOQi+lpQ1Vd mbtTjcAma/MgjYgxPYcWU6+jxKen46QFylqTmxnX8kRjABWK6v0We1zRHIZCPChMCnvB qKA03cSIrb/UZpt9UupjTtzJCnmbr+PT8Lxegrt68ojAijkji1dG7xrhNrwtzoB/Fl0b tnRjwXmGVnq8PkuMZwadeiYY/hJs7YuGQIajrCC7m4dkxuLseSp8aCxiqVdomj34Nc7q OY4u1p80DylzkI14sbMeEmK6URIHll6vsX4qu4xHTn2HzKi5f2/W5SG/VlG52wKjWQcg 0OxA== X-Gm-Message-State: AO0yUKVa8QM5M+MPlXW3/Q3LuBXW4WOJq8vGrpsS1RH2rR8AycG/8vCk MtE8rKY+6L6pKTS2O5LTQ83bw48= X-Google-Smtp-Source: AK7set87jur81KzCMPz8Y0+5pvv5t8Mj23viiWNXddTvCqZd7E8Ju9et2DF7mTppNXEOC0lDqKeb0Mc= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2d3:205:4760:7b08:a3d0:bc10]) (user=pcc job=sendgmr) by 2002:a81:af4b:0:b0:533:91d2:9d94 with SMTP id x11-20020a81af4b000000b0053391d29d94mr15728006ywj.5.1678422601498; Thu, 09 Mar 2023 20:30:01 -0800 (PST) Date: Thu, 9 Mar 2023 20:29:14 -0800 In-Reply-To: <20230310042914.3805818-1-pcc@google.com> Message-Id: <20230310042914.3805818-3-pcc@google.com> Mime-Version: 1.0 References: <20230310042914.3805818-1-pcc@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Subject: [PATCH v4 2/2] kasan: remove PG_skip_kasan_poison flag From: Peter Collingbourne To: catalin.marinas@arm.com, andreyknvl@gmail.com Cc: Peter Collingbourne , linux-mm@kvack.org, kasan-dev@googlegroups.com, ryabinin.a.a@gmail.com, linux-arm-kernel@lists.infradead.org, vincenzo.frascino@arm.com, will@kernel.org, eugenis@google.com X-Rspamd-Queue-Id: 38ACA100004 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 6x7rdbwaoorykaeasttod446fwfe34pc X-HE-Tag: 1678422602-981095 X-HE-Meta: U2FsdGVkX18/ccKkLHKcqBlI9Iqp0sQgOpIWLe87BGakven9ig+ET6i/VGASYxaJ9IX4qf453kvaZp00PrmYhUOhPVLjL35ehPtGpSqXFfZlGzHco4SRfLvXgq8zPbmx2crqcrNvZhAFDzdskJKleOE2BjXhxVEF3qM1O32xnaV/6g49JC9ePOy/yCLvhiBSMVG+Of/oZGqeS/TkLyDW+PrtnUPY9/DFg2SEUL0E5e/oeSJ6WHCTs7EChfQosrQQYuksfwnbfKG5KJGKEo7lQekpNqM83LKnlsmwAcdhj/7TCy6/zrE+OSAgAVCDnE86E6nm1wZOnQ3ADl90Lv78Xdr0XfPIbv9lnr7LfIukiVK6fHUVHKA9+ck8FiXo3+OpKguEyfWngtz+VhXetUYm4wnVfKZUVTxVI/KZ5tRGW7dPc2CH7yZAFjWelsSg1P5Jw9ATT9mxEgKljAhpzIlBLWrU9O0H46Hu6M/5VsNZCP3vCuKW/VN1FhwT0eVwbWC1J/ZACMRG3nP69qSBy4Bo+Z9sXIeGN+7o2xqzmu5iLYaG2kcJzwmqJUQRUb9y+CtPVoeMxYuld1/tBQs9T03/V50VBH98mlmd4RR0FTTs8whiQxnDL8GrSzIGXFktsWkt15PyY/WkPsTsY1OLaC/mWJhAqbMONn1Qww5S9+qBlcKoj15x6P7TXpIcETZWwgy3WE5rev8mqgSHKtX7JXr8OTpCyFuOxmJE5DNq+GTO2gzBhLXMTdt94hnfNOKhDl6Eeyk7FFKlh4eXP7/+qfONyf6N37B8+OAQc8+Zt27EFED7Sh2/P8swDdFmVGu9xJaiCVY3QpBwD5UI3j0ekRUjtyXcWPDWC3EfAyFCJO4QzHW1fycqdyQQYM+6jewe8Xsqf6BJAoEIMnsRSGrloxjyoL4/UtYdC5EIIIGmW00TeEU+BLdAe827lqhIVchQ31OdvwuKdLhUvsPqpZJ+GWa WHTxTQ98 DvJbryfVq1EJeZok+hPP0Hf1FWNbljARX594RBL5BiCm++u91qTi5Z3Hte8cPEpNAj/Y56C2mcW5IlOIsTPitBzH4JELxq7r9UJSZvUPAkAGDdrj5no7U/yxlgAOlwwVsB3XdK63Exh0yPIUZdz3RF6gHXNvbWptVJWNml2nSEL6vPJc72BP15DR8ZaS91IVxwMPm4Ps+TePjnjBSm68QN2vK0BEBu7Da6UgrmYy5tC2PCQUJFvxZjg0bJHmVTvo43F6QXWrwPA3c5vtVFDKGQPHgXOiBG3yCwIQ5195mGmp68agLUOr/9YDSs7fx2cod+pkQM4nTR+69IgFN5k6XIIRRpEM/1reL3VnWeegsg3RcdvqX067jDzYuTQvG0D/VKI1Oz7U8mKj33077mQ7XOj9PQAdO8UXJbyyKezxEvMuKPEIO+k63I3l9Dm4ZubTunVVKgCruHjBmlIyheZ17zeyjn1pu8lUSCL9B101N0n9Y6GB3492tYUcrbQQcSb5XBX+BngOmxBACHqS0PCADbuOwtFUSR57I3ngsG6iZqnCwfU8Dp6P59YSv5qF347mX5DZQagEam0CcNM6hb88gQFXd8SPjo8htfmimZXlMvWq1g18+TmLcvdWbNy6bXoM1OcH22Y/PgOjeU6z+zsaVRUyYvslfk9yMMbLjcRsSOhlf1xFwkSQnFeYluiA/q3GYVWA1AsuWziHgGdWtDknqwTckRln8+ZYAn1T39OrPkfhUOOxTuhkvAyYEmUhqxDmUYups7aO6PhuxC+Vc0hmjP0Ew+KzDckDBrGMfVsBCG0E8cWo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Code inspection reveals that PG_skip_kasan_poison is redundant with kasantag, because the former is intended to be set iff the latter is the match-all tag. It can also be observed that it's basically pointless to poison pages which have kasantag=0, because any pages with this tag would have been pointed to by pointers with match-all tags, so poisoning the pages would have little to no effect in terms of bug detection. Therefore, change the condition in should_skip_kasan_poison() to check kasantag instead, and remove PG_skip_kasan_poison and associated flags. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I57f825f2eaeaf7e8389d6cf4597c8a5821359838 Reviewed-by: Andrey Konovalov --- v4: - rebased to linux-next v3: - update comments v2: - also remove GFP_SKIP_KASAN_POISON and FPI_SKIP_KASAN_POISON - rename GFP_SKIP_KASAN_UNPOISON to GFP_SKIP_KASAN - update comments - simplify control flow by removing reset_tags include/linux/gfp_types.h | 30 ++++++------- include/linux/page-flags.h | 9 ---- include/trace/events/mmflags.h | 13 +----- mm/kasan/hw_tags.c | 2 +- mm/page_alloc.c | 81 +++++++++++++--------------------- mm/vmalloc.c | 2 +- 6 files changed, 47 insertions(+), 90 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 5088637fe5c2..6583a58670c5 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -47,16 +47,14 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_ACCOUNT 0x400000u #define ___GFP_ZEROTAGS 0x800000u #ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN_UNPOISON 0x2000000u -#define ___GFP_SKIP_KASAN_POISON 0x4000000u +#define ___GFP_SKIP_ZERO 0x1000000u +#define ___GFP_SKIP_KASAN 0x2000000u #else -#define ___GFP_SKIP_ZERO 0 -#define ___GFP_SKIP_KASAN_UNPOISON 0 -#define ___GFP_SKIP_KASAN_POISON 0 +#define ___GFP_SKIP_ZERO 0 +#define ___GFP_SKIP_KASAN 0 #endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x8000000u +#define ___GFP_NOLOCKDEP 0x4000000u #else #define ___GFP_NOLOCKDEP 0 #endif @@ -234,25 +232,24 @@ typedef unsigned int __bitwise gfp_t; * memory tags at the same time as zeroing memory has minimal additional * performace impact. * - * %__GFP_SKIP_KASAN_UNPOISON makes KASAN skip unpoisoning on page allocation. - * Only effective in HW_TAGS mode. - * - * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation. - * Typically, used for userspace pages. Only effective in HW_TAGS mode. + * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation. + * Used for userspace and vmalloc pages; the latter are unpoisoned by + * kasan_unpoison_vmalloc instead. For userspace pages, results in + * poisoning being skipped as well, see should_skip_kasan_poison for + * details. Only effective in HW_TAGS mode. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) #define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS) #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) -#define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON) -#define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON) +#define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** @@ -335,8 +332,7 @@ typedef unsigned int __bitwise gfp_t; #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 #define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM) -#define GFP_HIGHUSER_MOVABLE (GFP_HIGHUSER | __GFP_MOVABLE | \ - __GFP_SKIP_KASAN_POISON | __GFP_SKIP_KASAN_UNPOISON) +#define GFP_HIGHUSER_MOVABLE (GFP_HIGHUSER | __GFP_MOVABLE | __GFP_SKIP_KASAN) #define GFP_TRANSHUGE_LIGHT ((GFP_HIGHUSER_MOVABLE | __GFP_COMP | \ __GFP_NOMEMALLOC | __GFP_NOWARN) & ~__GFP_RECLAIM) #define GFP_TRANSHUGE (GFP_TRANSHUGE_LIGHT | __GFP_DIRECT_RECLAIM) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 57287102c5bd..dcda20c47b8f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -135,9 +135,6 @@ enum pageflags { #ifdef CONFIG_ARCH_USES_PG_ARCH_X PG_arch_2, PG_arch_3, -#endif -#ifdef CONFIG_KASAN_HW_TAGS - PG_skip_kasan_poison, #endif __NR_PAGEFLAGS, @@ -594,12 +591,6 @@ TESTCLEARFLAG(Young, young, PF_ANY) PAGEFLAG(Idle, idle, PF_ANY) #endif -#ifdef CONFIG_KASAN_HW_TAGS -PAGEFLAG(SkipKASanPoison, skip_kasan_poison, PF_HEAD) -#else -PAGEFLAG_FALSE(SkipKASanPoison, skip_kasan_poison) -#endif - /* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index b28218b7998e..b63e7c0fbbe5 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -55,8 +55,7 @@ #ifdef CONFIG_KASAN_HW_TAGS #define __def_gfpflag_names_kasan , \ gfpflag_string(__GFP_SKIP_ZERO), \ - gfpflag_string(__GFP_SKIP_KASAN_POISON), \ - gfpflag_string(__GFP_SKIP_KASAN_UNPOISON) + gfpflag_string(__GFP_SKIP_KASAN) #else #define __def_gfpflag_names_kasan #endif @@ -96,13 +95,6 @@ #define IF_HAVE_PG_ARCH_X(_name) #endif -#ifdef CONFIG_KASAN_HW_TAGS -#define IF_HAVE_PG_SKIP_KASAN_POISON(_name) \ - ,{1UL << PG_##_name, __stringify(_name)} -#else -#define IF_HAVE_PG_SKIP_KASAN_POISON(_name) -#endif - #define DEF_PAGEFLAG_NAME(_name) { 1UL << PG_##_name, __stringify(_name) } #define __def_pageflag_names \ @@ -133,8 +125,7 @@ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ IF_HAVE_PG_IDLE(young) \ IF_HAVE_PG_ARCH_X(arch_2) \ -IF_HAVE_PG_ARCH_X(arch_3) \ -IF_HAVE_PG_SKIP_KASAN_POISON(skip_kasan_poison) +IF_HAVE_PG_ARCH_X(arch_3) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index d1bcb0205327..bb4f56e5bdec 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -318,7 +318,7 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, * Thus, for VM_ALLOC mappings, hardware tag-based KASAN only tags * the first virtual mapping, which is created by vmalloc(). * Tagging the page_alloc memory backing that vmalloc() allocation is - * skipped, see ___GFP_SKIP_KASAN_UNPOISON. + * skipped, see ___GFP_SKIP_KASAN. * * For non-VM_ALLOC allocations, page_alloc memory is tagged as usual. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c58ebf21ce63..680a4d76460e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -112,17 +112,6 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) -/* - * Don't poison memory with KASAN (only for the tag-based modes). - * During boot, all non-reserved memblock memory is exposed to page_alloc. - * Poisoning all that memory lengthens boot time, especially on systems with - * large amount of RAM. This flag is used to skip that poisoning. - * This is only done for the tag-based KASAN modes, as those are able to - * detect memory corruptions with the memory tags assigned by default. - * All memory allocated normally after boot gets poisoned as usual. - */ -#define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) - /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1370,13 +1359,19 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) /* * Skip KASAN memory poisoning when either: * - * 1. Deferred memory initialization has not yet completed, - * see the explanation below. - * 2. Skipping poisoning is requested via FPI_SKIP_KASAN_POISON, - * see the comment next to it. - * 3. Skipping poisoning is requested via __GFP_SKIP_KASAN_POISON, - * see the comment next to it. - * 4. The allocation is excluded from being checked due to sampling, + * 1. For generic KASAN: deferred memory initialization has not yet completed. + * Tag-based KASAN modes skip pages freed via deferred memory initialization + * using page tags instead (see below). + * 2. For tag-based KASAN modes: the page has a match-all KASAN tag, indicating + * that error detection is disabled for accesses via the page address. + * + * Pages will have match-all tags in the following circumstances: + * + * 1. Pages are being initialized for the first time, including during deferred + * memory init; see the call to page_kasan_tag_reset in __init_single_page. + * 2. The allocation was not unpoisoned due to __GFP_SKIP_KASAN, with the + * exception of pages unpoisoned by kasan_unpoison_vmalloc. + * 3. The allocation was excluded from being checked due to sampling, * see the call to kasan_unpoison_pages. * * Poisoning pages during deferred memory init will greatly lengthen the @@ -1392,10 +1387,10 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) */ static inline bool should_skip_kasan_poison(struct page *page, fpi_t fpi_flags) { - return deferred_pages_enabled() || - (!IS_ENABLED(CONFIG_KASAN_GENERIC) && - (fpi_flags & FPI_SKIP_KASAN_POISON)) || - PageSkipKASanPoison(page); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + return deferred_pages_enabled(); + + return page_kasan_tag(page) == 0xff; } static void kernel_init_pages(struct page *page, int numpages) @@ -1730,7 +1725,7 @@ void __free_pages_core(struct page *page, unsigned int order) * Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. */ - __free_pages_ok(page, order, FPI_TO_TAIL | FPI_SKIP_KASAN_POISON); + __free_pages_ok(page, order, FPI_TO_TAIL); } #ifdef CONFIG_NUMA @@ -2396,9 +2391,9 @@ static inline bool should_skip_kasan_unpoison(gfp_t flags) /* * With hardware tag-based KASAN enabled, skip if this has been - * requested via __GFP_SKIP_KASAN_UNPOISON. + * requested via __GFP_SKIP_KASAN. */ - return flags & __GFP_SKIP_KASAN_UNPOISON; + return flags & __GFP_SKIP_KASAN; } static inline bool should_skip_init(gfp_t flags) @@ -2417,7 +2412,6 @@ inline void post_alloc_hook(struct page *page, unsigned int order, bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && !should_skip_init(gfp_flags); bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); - bool reset_tags = true; int i; set_page_private(page, 0); @@ -2451,37 +2445,22 @@ inline void post_alloc_hook(struct page *page, unsigned int order, /* Take note that memory was initialized by the loop above. */ init = false; } - if (!should_skip_kasan_unpoison(gfp_flags)) { - /* Try unpoisoning (or setting tags) and initializing memory. */ - if (kasan_unpoison_pages(page, order, init)) { - /* Take note that memory was initialized by KASAN. */ - if (kasan_has_integrated_init()) - init = false; - /* Take note that memory tags were set by KASAN. */ - reset_tags = false; - } else { - /* - * KASAN decided to exclude this allocation from being - * (un)poisoned due to sampling. Make KASAN skip - * poisoning when the allocation is freed. - */ - SetPageSkipKASanPoison(page); - } - } - /* - * If memory tags have not been set by KASAN, reset the page tags to - * ensure page_address() dereferencing does not fault. - */ - if (reset_tags) { + if (!should_skip_kasan_unpoison(gfp_flags) && + kasan_unpoison_pages(page, order, init)) { + /* Take note that memory was initialized by KASAN. */ + if (kasan_has_integrated_init()) + init = false; + } else { + /* + * If memory tags have not been set by KASAN, reset the page + * tags to ensure page_address() dereferencing does not fault. + */ for (i = 0; i != 1 << order; ++i) page_kasan_tag_reset(page + i); } /* If memory is still not initialized, initialize it now. */ if (init) kernel_init_pages(page, 1 << order); - /* Propagate __GFP_SKIP_KASAN_POISON to page flags. */ - if (kasan_hw_tags_enabled() && (gfp_flags & __GFP_SKIP_KASAN_POISON)) - SetPageSkipKASanPoison(page); set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index bef6cf2b4d46..5e60e9792cbf 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3188,7 +3188,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * pages backing VM_ALLOC mapping. Memory is instead * poisoned and zeroed by kasan_unpoison_vmalloc(). */ - gfp_mask |= __GFP_SKIP_KASAN_UNPOISON | __GFP_SKIP_ZERO; + gfp_mask |= __GFP_SKIP_KASAN | __GFP_SKIP_ZERO; } /* Take note that the mapping is PAGE_KERNEL. */