From patchwork Mon Aug 5 12:55:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13753595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACEE0C3DA4A for ; Mon, 5 Aug 2024 13:03:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29A6C6B0089; Mon, 5 Aug 2024 09:03:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24B066B009D; Mon, 5 Aug 2024 09:03:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EB696B00A2; Mon, 5 Aug 2024 09:03:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E2F2C6B0089 for ; Mon, 5 Aug 2024 09:03:21 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 713F71402CE for ; Mon, 5 Aug 2024 13:03:21 +0000 (UTC) X-FDA: 82418207802.19.E4FA8FD Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf09.hostedemail.com (Postfix) with ESMTP id 7F6AF14001B for ; Mon, 5 Aug 2024 13:03:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Up3tApbB; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722862938; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XC0Rcbjdh8ylucXC7ADaLu0MmCwUQ8ESfO/B+g+YjaQ=; b=yD0stz5YIhJ+lIidNe3fMPrj39pM43f++JR20gu4XrBt3SbZhbkWGhMEn5qZNKYRMMdzbY 7qfcG+flIj/097TJnxwkwcqwzpebT4uwrxKgvAZWJl0UN7ScktaP6JaJbHWLt+yjSVm614 /ncHgHk2ur7Mz7TcWmGttWuqO1pI/SY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722862938; a=rsa-sha256; cv=none; b=viLpK9xMTK6qKv0uf1tcCJ4+3IKsuxv2VBJ9JKqqT++nwP9OO4SdhV9bfJH34vFZeL0hxy F5Rg0Jc5FaqGTOeTVeCJGdyCWRTpsXBjDQXZ3AZEfQEtrali8Fto3NJOiVYhgV3sr1VoSt QsZe1MH8Q5B6phe4vgXZBgSo+w7FArs= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Up3tApbB; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7a201bcf379so80568085a.3 for ; Mon, 05 Aug 2024 06:03:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1722862998; x=1723467798; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XC0Rcbjdh8ylucXC7ADaLu0MmCwUQ8ESfO/B+g+YjaQ=; b=Up3tApbBkS257jK8LBsoHpof7uk2VzGnnLcz9XUp0/C37RtpdkYmYyO7AWPZKZh4m0 EBnqsjjz+YnzZMTtMAgGr6VSW7+tUQthv0+XAA0KQwB5jV1y3Hi0eBetIy3uRfVLe1xj 0Ey7sc5AN2ZvCF1N3Gn0Mq9YQ+J+n8pg3xrL+dHz+5bAB9qYzs8yPy4wDYghQ4S+BTtu iyOg8KWncCR8RyguI0cw5uBJpgiR8sWwfVwzPdYame7RAMH/5J5turYbV48+GEmjSpzS ijf5jMxVu+1/cN8MOwPtETjmWi7lsGuAdvbl4T51XFxV1HRORF2HpMwM53M2YdtSCY5M UmpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722862998; x=1723467798; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XC0Rcbjdh8ylucXC7ADaLu0MmCwUQ8ESfO/B+g+YjaQ=; b=c2OjgY3WGIHHTnLgLmpP2aibVMqwL6Eckjw2ePk7SArlialxz5Su2ZLdaq8yn1RAXG Mx3P8u/Udt5+OtKhw0YsPqLAYSIRRgOhJB8EHZgzaFe24ytYOJJdKWaeP09qWEYN+MVi yZiRvwUCPClbmh+7Do+LJtsP6prxp8MXsEwbvrka+VaLrqeYYFiwP2BNcxx5M/QQKxUc RH9YmCSGC4ogBN3aXFMURMfYZoZPO6AmilsRAPGZfOaZ0sHHxoJmkSPlfmohiv00OPRA 4JwP2St48g5/6lxZIcdGNzgdUqS2O58ploFIBncDmjDdq4teNyQhynOo7Y5t1VSmHEuC KXbA== X-Gm-Message-State: AOJu0Yz7lS+q8MrlvvbfH5cvUS4dn1aqMNL8GZqm91EsgfAZgPZdisvm 78pReEaHwctFi9efQxVTmr9dPTsOraPTfwfswIyzkCOh0H4LlopVEOvwjTEoKjrml/Mqbh3K9m3 U X-Google-Smtp-Source: AGHT+IF/l8kL7sdJ/NhN/cq8F6a1ZTnpgYnj4ROM1RQNcS/ED4bxJRGoEHg/fBeOyRXw7ShGlIlTUA== X-Received: by 2002:a05:6a00:3a20:b0:706:aadc:b0a7 with SMTP id d2e1a72fcca58-7106cf94aa4mr8039553b3a.1.1722862575131; Mon, 05 Aug 2024 05:56:15 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.232]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7106ecfaf1asm5503030b3a.142.2024.08.05.05.56.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Aug 2024 05:56:14 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [RFC PATCH v2 6/7] x86: mm: define arch_flush_tlb_before_set_huge_page Date: Mon, 5 Aug 2024 20:55:10 +0800 Message-Id: <1c8bee0c868c1e67ea02a6fa49225b00503b5436.1722861064.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7F6AF14001B X-Stat-Signature: 414zq7jtwnbw7rbzbizmubmr1huupds8 X-HE-Tag: 1722862999-164335 X-HE-Meta: U2FsdGVkX18XZ7Qo54zbxJM/XNRGCTJeuyyLaxYYmXBtbvKQ20R6yrwGFX+Xu0OCuIdbkqbNImZcGVn3mvD4AX+008LFjjc19rzFhBd6wJhJ9RdmA4Nz06SGTIMTp9o6SsShCj9qDIKigwTb3vDmmztHi0Tnq8R4cKFjXhDlfJgSbFr2LZT0jjOr/RpGV8zL0qusimzFJI7XGZ3XmhdOBfMgzkX3tDcdCpSHKT4QEOjGW/0orZWgz0jTTY40gfZi78XRTbrbONFofpm0TsuazfRMGxL21GBmKxGyEA37wDMt5d6y6xaQJvTtCGnRvCzolkPQzgjHlXpq9XG/N1rxwQTxYoBDuYiXUlaAdofJFuaFZYJSazB4PoMqpRlC8W3I9WpRpcPxpgqs2qEYFiSAgLnTk5FSQSdv++wstOJgtiQQEXlJOYLrDWUnbJNjnEsLIBwvk5Okx+iFG4ELty05nufw2072q5kp1LgTsknBl1/bRSDnBoRf80CyMNz0E/vAlvm2lUNm+4u/HMzZ7wro4xd/Q9UlOFZ6bwIjIg7Emt8bIEB4dxpmRJ84MC1ddueun0FybZZpQMdkWCm8nrlhQHahRfT0QuAU5Omf+eyJCFOTsl6TTxRHhMy9950Zk+wYdQ0yA1aj/O05pYHCCgKLEw8GOaUBC4+E/gMg7hglMYTA6DgtUddr7iqvjmF6qg238hl/hYqTwE+Wy3hkx4XMVxGDlbz1YG8JdTZWdoFZRxbTrxgBIdEYQxvNMX4miv4+lLqtadYaFVGOp4bik/1jz7JOFxCGQY9SSZOLevqSAhAnhV8LnuxkNmj6iEKryP4Wgg42c/+gNo6BhmBX0LJXJ/Fvj5JBmkj5ymeijCRN4GbVnHkp7qpXqUAR4ZP00DS+IoNK6m5SwR0i28oZinLz7xiPo5HdpzAiSxZcSCH8QjPt1AljzGLXSnPyJi7d8eYWygaqDzorqot1kNfmEHU u8gNZgZi oZRa8OMALHYFo5t8vfxQRsZe4C5RBWIMy7/RC5BRU62CjxWkybfxY91cvtoPdqUhx+gptSIbV753EMfBN8KIJ2ITlhIphk47K75FCUnWAAXoRKzY7Vo6U1jkpp0CLJIiS6sH95mLwV+A/zQO21KbkURWwQcBtlGz3S7pu98Agi5sS0cZmPG36to56fz0tpioyeRDmHUDIeDo34SaUJjfMbmfZ61MR0W2riC2XlosFAMtkV+kNNtC9AZmlMqLFYT90kr4/7Rzk7GmZ5wXi8rQDBka8Wy8J3x5tVGOG/uTI19NkUY2m0pCOzzxWbC01DeTxgxAc1nbDoe44udVlFix6G3uDnZkc3Hx7EMGKdR5AD0/pT2t/jVmD+vMESA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When we use mmu_gather to batch flush tlb and free PTE pages, the TLB is not flushed before pmd lock is unlocked. This may result in the following two situations: 1) Userland can trigger page fault and fill a huge page, which will cause the existence of small size TLB and huge TLB for the same address. 2) Userland can also trigger page fault and fill a PTE page, which will cause the existence of two small size TLBs, but the PTE page they map are different. According to Intel's TLB Application note (317080), some CPUs of x86 do not allow the 1) case, so define arch_flush_tlb_before_set_huge_page to detect and fix this issue. Signed-off-by: Qi Zheng --- arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 13 +++++++++++++ 2 files changed, 19 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e39311a89bf47..f93d964ab6a3e 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1668,6 +1668,12 @@ void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); #define arch_check_zapped_pmd arch_check_zapped_pmd void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); +#ifdef CONFIG_PT_RECLAIM +#define arch_flush_tlb_before_set_huge_page arch_flush_tlb_before_set_huge_page +void arch_flush_tlb_before_set_huge_page(struct mm_struct *mm, + unsigned long addr); +#endif + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ea8522289c93d..7e14cae819edd 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -934,3 +934,16 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pmd_shstk(pmd)); } + +#ifdef CONFIG_PT_RECLAIM +void arch_flush_tlb_before_set_huge_page(struct mm_struct *mm, + unsigned long addr) +{ + if (atomic_read(&mm->tlb_flush_pending)) { + unsigned long start = ALIGN_DOWN(addr, PMD_SIZE); + unsigned long end = start + PMD_SIZE; + + flush_tlb_mm_range(mm, start, end, PAGE_SHIFT, false); + } +} +#endif /* CONFIG_PT_RECLAIM */