From patchwork Wed Sep 20 03:53:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13392066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21C15CE79AC for ; Wed, 20 Sep 2023 03:53:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 464816B0101; Tue, 19 Sep 2023 23:53:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EBF56B0102; Tue, 19 Sep 2023 23:53:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D9276B0103; Tue, 19 Sep 2023 23:53:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1AB7F6B0101 for ; Tue, 19 Sep 2023 23:53:45 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DA6D2160201 for ; Wed, 20 Sep 2023 03:53:44 +0000 (UTC) X-FDA: 81255606768.22.70FE44D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 6774B1C0006 for ; Wed, 20 Sep 2023 03:53:42 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Kxm0OxXW; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695182022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=q9PrX4Gu5y2UUk0yElrgd6wrH7FBGev7q0nCeS79mig=; b=uLpvYCYuU4uXl1DeloJtg/4QbU/xhQY2KsrXARSm0LLCbSNFUJyjHFlR5ZGzhP4p6bzPkV cZdvtQ36QPLaQkjhjN14JbBGFi4aYEVKKgpuQvvuJZ0pe0V1t5Ax5RXoZoxvcUK5iyLldE GpxjfPLMyHxgeswncT54FeADZQUopJI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Kxm0OxXW; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695182022; a=rsa-sha256; cv=none; b=lbleSNE0JMhAYHvJ6sy6sUHMuh1jbqNNpM1UUw2CH+CKW3LjccZunDbprTXaenr64y7ucF ZZbVTJqJJMFVknvwCudF4rvL3Ux5vDNRuwfdCb82vVPNTk41v/hIeBa8hJfjmEoL2VIRAN Kz9MQVZzgqye56e3pgvjQaNdNnKSmvI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:In-Reply-To:References; bh=q9PrX4Gu5y2UUk0yElrgd6wrH7FBGev7q0nCeS79mig=; b=Kxm0OxXWxT/9+KCQ281rJEi++u JySZTQMmZFWpE91zuLvEN1pkggWcii/tyN4kCN338z46XH3JDy9bN9wZQnrwuF/D23S8svea4tHch Od/Rl+WCdvwAZVSfcTsM95mfeWgKfecNtRees115/pChFXn7gcGr+OZXx0zabQQxDWmQOg2nkUPzp yM1voemEFFXMxGdOSW9uNiNZhwzHNW623KikFdbZr8XwDwRziHQx8oPXvMv3TPlfOnX4KVAfCJfRu HpbBFPu59qEIHPOoTiVDCVgtpdm5HQeHzADCqaKgdcyj6UwydMHUL1OPBqC+DhXcVskavqIw6PHxn OE0B2brA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qioHS-003aDl-CY; Wed, 20 Sep 2023 03:53:38 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Yin Fengwei , Dave Hansen , David Hildenbrand , Thomas Gleixner Subject: [PATCH 1/2] mm: Report success more often from filemap_map_folio_range() Date: Wed, 20 Sep 2023 04:53:35 +0100 Message-Id: <20230920035336.854212-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6774B1C0006 X-Stat-Signature: 3rh68hd98yr87mtjti4z6kt61qybuq1n X-HE-Tag: 1695182022-168500 X-HE-Meta: U2FsdGVkX1/rLTxrNn2rBRyRNCUMs0PgNed25Jo/MRiNYUtYggjq6c/fv+GyAjX6xNmG5niZHSQHgv8NJRW+lIE8ETByh50Ih1Ky0kViJL/0+of8c8NknhU9p6TWYUNRrdAgltMPKVzb5ezzTvxjJAkWjfZ/mkUlvBb0XsxJvVRvDwGo4a0MPlfLcuhi+Iccc1uhvmJ8QyyJV8tTiSZKjXyyJ4kT1VV6h0vW8Lz4ZJDr9P/dsy1l+h45W+6ZIYOr4/TXo45JZJAwev/+MZQZvC11BFTa9Rd+qdKAFC+X/CzLKVAkzQdM988TZv0IJuSuL0XJuyF+7ECFSo0nL0atkvnaTl9HHhIVZjovnE9+CVd6XcBlPIWx/VRcl/fKSeU/LoemKi4qOQC9S+WeXhS5T9ni1m80Oz1gL5m1RF4pCXCZrXijCP+QQjsggpbfLRqkGmJeDxzmLLJOdsbLOHpouj2E8qSDYJcjHT5NbV65sO6N7jpdbQjEFYiCpf13kXqWR6s23U5XjW9tEtqC9GBFG6Dr7kVFmTeLLkNUZWOfQNaIQzUodtRoPvrB2SpbloFIN7MrU6TEaAI+K+nsOyb/s6viDUrLHDpEfGWcWbW4IMF5CQ1fpSpQuOEJ50RcIKOhIMElr75IOgW8Oh+EEfrOBg2XB0iqZGoPKiXuVbj8rWR5v7rJvny84spMWF4E5mNVfla/ycyTo//UjQmG9QzMnJES5eNFOUq965ji7daLipbsRaTgkLMyInBTZoz2WeTkF2CDlz490TYUaUEuSx3JVfC99N+JXIQb8V2BHBJxL/iyLuUXwjzQ8ld4/4z80Nl6EO1yVoqPjlcUqebk5Bnzv3KvYw4qjrJaRuNDQOKrqIa/8gmWKcThsLGmbj3z4qle02iFbJTxvMhjOst4ewXbcklVfsVfstUcKS/CvdiBZME/QQTMSYPdOoyiEbiajwBV2MreMS1XxqFC3piROzx xumkI590 bYxXNE5hXgvyeuLooXyk0jLHtMzbgmiHFmGXK4eft56Bt3WmPx1qA1uYa/kidqpTJEgMnZC+ByhyD76iLlly95Eol2eg5KG9aK0O974iSasoYghmdLNIuO7py5SCSzD7GCjlaic5va8adQZ/pnzOgk2Bt/8HBp9AxMmyPfJDBxr/7DxFbvxU8xnqufKOxXjW21DddLFwb/v05gahkd3+TKaaB0qXmvKHpgwLz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though we had successfully mapped the relevant page, we would rarely return success from filemap_map_folio_range(). That leads to falling back from the VMA lock path to the mmap_lock path, which is a speed & scalability issue. Found by inspection. Signed-off-by: Matthew Wilcox (Oracle) Fixes: 617c28ecab22 ("filemap: batch PTE mappings") Reviewed-by: Yin Fengwei --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 582f5317ff71..580d0b2b1a7c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3506,7 +3506,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { set_pte_range(vmf, folio, page, count, addr); folio_ref_add(folio, count); - if (in_range(vmf->address, addr, count)) + if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } @@ -3520,7 +3520,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, if (count) { set_pte_range(vmf, folio, page, count, addr); folio_ref_add(folio, count); - if (in_range(vmf->address, addr, count)) + if (in_range(vmf->address, addr, count * PAGE_SIZE)) ret = VM_FAULT_NOPAGE; } From patchwork Wed Sep 20 03:53:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13392067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AE7ECE79AD for ; Wed, 20 Sep 2023 03:53:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA19A6B0102; Tue, 19 Sep 2023 23:53:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C28CD6B0103; Tue, 19 Sep 2023 23:53:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A545F6B0104; Tue, 19 Sep 2023 23:53:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 920286B0102 for ; Tue, 19 Sep 2023 23:53:46 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5E0F5120EC1 for ; Wed, 20 Sep 2023 03:53:46 +0000 (UTC) X-FDA: 81255606852.15.312F119 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id B139B1C0006 for ; Wed, 20 Sep 2023 03:53:44 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FO6kU4Tj; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695182024; a=rsa-sha256; cv=none; b=uUxMmx9AWYjFpTW/PFKPyoT79rF3UxruwHdyuXqcJfZJrKj1LOsGnO73/EADWl9x4GXtQH RnNOVXHp10uctmziNSC5yYFH9B1rJX3Kxm1Iqu3UYc7mFQLKIKOe64kQCOB5zjhTjYZPRa WZS/Eq2clUiun5ciHXZagS2tkgtGcrI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FO6kU4Tj; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695182024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=17Ab6P2j5OVF8Gr3Yg5Qn24MvwgRH/J1nkWqaHudOc0=; b=DvjNoOiTlZzQGSqQ66/ZL8SjFn7jtofLaRaFnYa8dnyvcnHG6kkqHK0RxHTjlR16LPnsAq lckDImg9sBmN8mgFuZJsiaVnllJt4ZRGHzOD5rb1Jku4hN4Z+sotyBhyXeHwSHQOJoHeX/ xgLTThIMI1C9eOJ1RRQq3eV3pzKOTng= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=17Ab6P2j5OVF8Gr3Yg5Qn24MvwgRH/J1nkWqaHudOc0=; b=FO6kU4TjQRsfkE7Go9HFs/q+JM x69oKhqVSUCygDweJvQjxbYJO6KdaGYKozVUy/YUgqW1/EPfX9jjsC+ObO0o5GSyOWTMw6Lz9QSAU sopjRdId2JNgSTWycz8MxVexI5ptju0l94H50ooMxsHjSlO/CrpnE+RukfWHQ47Ped+RYPxQYIKhu GpkeK8oExTOG2HjvjFleOgeUXQpcYiffQ4Jt8HAP5KiHskraGF7wFKom1+j/WtEe599nbGfDS2DK+ JGcgty/joPOfR7vLD7YQyb3n+9RfH4fxye15dQWSGkGJqMXA/2wpHtY3JAAvobsHHkJHzqE5eRBGN dnGNjKtw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qioHS-003aDn-F8; Wed, 20 Sep 2023 03:53:38 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Yin Fengwei , Dave Hansen , David Hildenbrand , Thomas Gleixner , syzbot+55cc72f8cc3a549119df@syzkaller.appspotmail.com Subject: [PATCH 2/2] mm: Abstract moving to the next PFN Date: Wed, 20 Sep 2023 04:53:36 +0100 Message-Id: <20230920035336.854212-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230920035336.854212-1-willy@infradead.org> References: <20230920035336.854212-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B139B1C0006 X-Stat-Signature: q3zpywjtddw7enczd36jf1g1wtcbimhs X-Rspam-User: X-HE-Tag: 1695182024-552750 X-HE-Meta: U2FsdGVkX1+PHoeDrpVq204dbZky6bSFYX4s9EJ9PhtHfQU528NWiohLhvsvBiztEqQJhJrgYfjrvcPCSMz8YJIQrgmN6MJJYojKh5NQPyTBk9Rem8NmFQH/rYqFaOzRxB0cjKhggWuwJjD8NPuv2CaRlP8ZKKWnxOeuoTWefr/QiqjxVnROqCaOico1Kbk0/pgV/JeByStAkLIBjI4IuW6EBWTiA/1bk72kPDEYc6g5dptvZJPIOmB45N8SqUmMrhhujqW8AXVuZqTA3KB6CalppHJZnNwCURwGOrOrqY0N44kOi6yPmap07U70bN0SUbbbYdyvBZjGz51mNt0T2VFC1c9gVk45crGXZEtUCD6rzxLKihyI2+mA3hfDyaHqhAxD7zbzIZqMefEM5SlMFiCwjmQG+AqG5wjbrwfBY8J4sjMXpAmxmd3PXfQnDbmsk1oS07v24GCnEJarTp0B3Vzbw9tfKjakRkvDB2YZCDc41avJ+IZXBlC0XpvZPOU4ywz8hvjs9zrGnakcDverBCsjwNnBd4B1/TABNlLgtzGFn8hTR4hyQAYZ8BKhxHXv7EM7e/g5g7J1Agfz+n++aVM3dnXWQKMBvBWF4yHSNz6Ma5Q6gpRWssHBeULawnKlJQhvuTheTKxAFDH/A+FItWQPQkCGnll1oifv1VqkXuWWgSgUHJMYAKu6otd39ClY5ei8sYTaeEirm98d781fN+WmEIhewqImb8cgTjJe8uCDtplCjGcdEcbg0kZNyNUTnqZh2LCHV9OjyvSn65zpwLJy+wAA31QsR+/zK2i+jC8M+xbLXBnHdhyJTXcdbw4yevPutua305QTTgab3b5d2efAKNh7VzJK0vzAUERRgT1SFKdYnRQA+1l79yxSqrejp7iqth+XymyKnWBaawukf2i4JS8pOP9tEnFsCVPUes3TEYHmz6SH6FKv52OW/dK+w8nSTWCQZ9eb0NdRJgn enCQRuSG a6dlXA/Yc+7wtjbst8Tgs3w565wRxksbTQIMwzgj0He+U9QKGnbtq8qG0DdcgPb63L5vHQEyNzME5zO/8b1PGxLX5oMXOk3bPLe4qTo5kNcAfgwlrUCPM1mhhE9IjYq65JyNGvgVIRe/DAQXKnMnnFe1aJOifBN0cAI/ZUURcdLD1LR76f1Ul+BQN8AR/++Ox1h8mg3Qn/EnyobrvgWz2fuPf89bIK1GXoVMjeb5S7oQcsYUUdeIsW03mAjEsCzY4SNW7G/p0nSsL+rWRZuoCxyXTi7P+v6Hy9aiyvqWgNaTmeylypPR2GWkz3HpxzSJM0eTwy+y1T70AsuYgKjg8R22KaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to fix the L1TF vulnerability, x86 can invert the PTE bits for PROT_NONE VMAs, which means we cannot move from one PTE to the next by adding 1 to the PFN field of the PTE. Abstract advancing the PTE to the next PFN through a pte_next_pfn() function/macro. Signed-off-by: Matthew Wilcox (Oracle) Fixes: bcc6cc832573 ("mm: add default definition of set_ptes()") Reported-by: syzbot+55cc72f8cc3a549119df@syzkaller.appspotmail.com --- arch/x86/include/asm/pgtable.h | 8 ++++++++ include/linux/pgtable.h | 6 +++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d6ad98ca1288..e02b179ec659 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -955,6 +955,14 @@ static inline int pte_same(pte_t a, pte_t b) return a.pte == b.pte; } +static inline pte_t pte_next_pfn(pte_t pte) +{ + if (__pte_needs_invert(pte_val(pte))) + return __pte(pte_val(pte) - (1UL << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); +} +#define pte_next_pfn pte_next_pfn + static inline int pte_present(pte_t a) { return pte_flags(a) & (_PAGE_PRESENT | _PAGE_PROTNONE); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1fba072b3dac..5bdf78aa7852 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -205,6 +205,10 @@ static inline int pmd_young(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif +#ifndef pte_next_pfn +#define pte_next_pfn(pte) ((pte) + (1UL << PFN_PTE_SHIFT)) +#endif + #ifndef set_ptes /** * set_ptes - Map consecutive pages to a contiguous range of addresses. @@ -231,7 +235,7 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pte = __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + pte = pte_next_pfn(pte); } arch_leave_lazy_mmu_mode(); }