From patchwork Mon Jan 29 14:32:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15D55C47422 for ; Mon, 29 Jan 2024 14:32:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A002B6B0072; Mon, 29 Jan 2024 09:32:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AF9B6B0085; Mon, 29 Jan 2024 09:32:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80BB26B0087; Mon, 29 Jan 2024 09:32:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6C6D06B0072 for ; Mon, 29 Jan 2024 09:32:41 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0F8DA160A21 for ; Mon, 29 Jan 2024 14:32:41 +0000 (UTC) X-FDA: 81732589722.16.DE6E82A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id EF36AA0007 for ; Mon, 29 Jan 2024 14:32:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjHv3cwS; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5k519pv5yC0dphpRGDmBdAjVLNwRA0D+0XMigEKB7uE=; b=ZVlGnhZmxEgf5r5IqE/e23LDbeOB5SNlyyCd+rZgIh04Zm7A8Yp5tdG0MBerbSE4lKOnR8 dT8yEaUgzAtSAk9fXy0wp6Ssie5KhedwWEd9BaL85JbU8cxjym4uTnx7r86/k/MhesxuKy 63NF26zWy0PnkF9oYWgh8DKUZiXjZlA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538758; a=rsa-sha256; cv=none; b=lWBfEFDnJn7zXW+wQcdGNOchDwf59MUQlMQb2sL//teYvUamQ8EKZgOzVXWXYkivBkXq4e 4VYwgFNL8c5FW270R6kQXWJ2d6swA/Z7CTosPmO0+uOb4yDIePE9jSFlqxMbph/u1NShrj B89kT2OVIey6L/8WfZ9u3cYmXBTjNmk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjHv3cwS; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5k519pv5yC0dphpRGDmBdAjVLNwRA0D+0XMigEKB7uE=; b=MjHv3cwSmKkN/sECJM1MXpDRLZhjHNrUnrFsIt1+qBX/2Svp+dw3Oqxrb63tbOuT56VkYR ns8C5RbWni8uMxAtXSr6g38J+55I/NCSqIBRZTllicTZ7ixb/aAwzA03wSzaYBZ/QbyaIz 91oWkeygLtg6khi5FUWbZ8gGgc7Kea0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-502-hf9kkbPYM4OwjicnjD23xg-1; Mon, 29 Jan 2024 09:32:33 -0500 X-MC-Unique: hf9kkbPYM4OwjicnjD23xg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B4A5D1C05137; Mon, 29 Jan 2024 14:32:32 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id 02540AD1; Mon, 29 Jan 2024 14:32:27 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 1/9] mm/memory: factor out zapping of present pte into zap_present_pte() Date: Mon, 29 Jan 2024 15:32:13 +0100 Message-ID: <20240129143221.263763-2-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Stat-Signature: uhrqn6p5grs7q19i76361f3ar5aifr7j X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EF36AA0007 X-Rspam-User: X-HE-Tag: 1706538757-131194 X-HE-Meta: U2FsdGVkX1/yq63892YAWRdit7i56iqDhDDlab2aM0V8Vw2UNJ1gwXV0Ewrqc4xSvSP4LG7NccPFFt4feGHGk+xM9hhUMpmDt2xz5QQsaoeDBKN98ppCMSnEVkRqyPM46sJ7d5aEdHbUe8Q482xqbvem+/zdlSrEzCBBOSCrO7jIeiD+LsPwN4InaWgN6pL8HnIk04ifA34QOQYfKc+jqIUdI2GnuWn7c6LfJoDAUfO93wtH/HO3TflPc6NFiQpt+9PqfQb+e2I1AA+G4vZXEdjKDFEcTiR4dfk0E41l8lhytg+zkQzrpX/TsvdvmkGrrSp3WPN9Qby5jj6XQs8A0r4LynPKBvJNt9sPCJw9D3+4fdeKnVvHc54SSi9liMSZXOpGGaE2AAB/d+ItUghj0mSUtoSS0X59IIRFHzmNr7wt3uF9dmFGVe15tpJi3/I2zkB58m4xCW66o88sk4z6nJacGjWU2X47Zw2K8CEDwz6xPX8lh1je25fkR0bZbT0vA395g+CNo4AW/TKZAJ3OdPOO7v67RgL9ARzHPy9gpjsGbYE7YHPu86r0lh9y2aVvelEDPPeq6s/DRU7H/8B+Gv4rahiUK6hVeSjyqaeE1XKtky1Z0lBdhiFcpU/ccy54KmAwF4YhZSj2xDUOqqK+Vv6hMp6Wg29iUHWa+lTE8NFBrUAfvoaN9JZ+X6PiMpipahx2Pub9gstVgeUSfD29eTwBJB8eduGvX7bsUN+LefZsBoXpwb8EPmWvk90fP1ygkSZgntATEeF1oUTysjsgMHdSjfn0XPx8/HT4Nh4kXig/Xyf9FOOSp732Fmjrg/7NItCfbozBSZaw/zyZL9xSkJAI290inh4PNJnveP0KOS7DBvry2WUGGUvcSxqPHkxWVAeb7hZ+DhU2/Eb2AeDN0ZJoeKpRBreTgt/rKvYyKybTy9etMM4ijifJhoFS8/iwwFDlRwutza34/t+ZfcN ya6vBtKL giD+gVXLwr7+505UK4p4ZPbVxcGf3V39NJFGWtCiM4Iy3s/8vqCKoTUnwkRybRgKgg58Ot58Ny4LehIy6MnLnMmeBjRF6EYM0C4wI8qG4iACWAVbO0N5rkRzlUH7p4VwhdVsmZVgj76SYAzsicIBIO5y1rMx/YOKY6fClno/SH4mHOYKhR3D052FXI20pBxu7tyXbgbh/U8iW0eGL6U8FKLU5oVDV2DrTARus X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's prepare for further changes by factoring out processing of present PTEs. Signed-off-by: David Hildenbrand --- mm/memory.c | 92 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 40 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index b05fd28dbce1..50a6c79c78fc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1532,13 +1532,61 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); } +static inline void zap_present_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned long addr, struct zap_details *details, + int *rss, bool *force_flush, bool *force_break) +{ + struct mm_struct *mm = tlb->mm; + bool delay_rmap = false; + struct folio *folio; + struct page *page; + + page = vm_normal_page(vma, addr, ptent); + if (page) + folio = page_folio(page); + + if (unlikely(!should_zap_folio(details, folio))) + return; + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (unlikely(!page)) { + ksm_might_unmap_zero_page(mm, ptent); + return; + } + + if (!folio_test_anon(folio)) { + if (pte_dirty(ptent)) { + folio_mark_dirty(folio); + if (tlb_delay_rmap(tlb)) { + delay_rmap = true; + *force_flush = true; + } + } + if (pte_young(ptent) && likely(vma_has_recency(vma))) + folio_mark_accessed(folio); + } + rss[mm_counter(folio)]--; + if (!delay_rmap) { + folio_remove_rmap_pte(folio, page, vma); + if (unlikely(page_mapcount(page) < 0)) + print_bad_pte(vma, addr, ptent, page); + } + if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { + *force_flush = true; + *force_break = true; + } +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, struct zap_details *details) { + bool force_flush = false, force_break = false; struct mm_struct *mm = tlb->mm; - int force_flush = 0; int rss[NR_MM_COUNTERS]; spinlock_t *ptl; pte_t *start_pte; @@ -1565,45 +1613,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - unsigned int delay_rmap; - - page = vm_normal_page(vma, addr, ptent); - if (page) - folio = page_folio(page); - - if (unlikely(!should_zap_folio(details, folio))) - continue; - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, - ptent); - if (unlikely(!page)) { - ksm_might_unmap_zero_page(mm, ptent); - continue; - } - - delay_rmap = 0; - if (!folio_test_anon(folio)) { - if (pte_dirty(ptent)) { - folio_mark_dirty(folio); - if (tlb_delay_rmap(tlb)) { - delay_rmap = 1; - force_flush = 1; - } - } - if (pte_young(ptent) && likely(vma_has_recency(vma))) - folio_mark_accessed(folio); - } - rss[mm_counter(folio)]--; - if (!delay_rmap) { - folio_remove_rmap_pte(folio, page, vma); - if (unlikely(page_mapcount(page) < 0)) - print_bad_pte(vma, addr, ptent, page); - } - if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { - force_flush = 1; + zap_present_pte(tlb, vma, pte, ptent, addr, details, + rss, &force_flush, &force_break); + if (unlikely(force_break)) { addr += PAGE_SIZE; break; } From patchwork Mon Jan 29 14:32:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 549F8C47DDB for ; Mon, 29 Jan 2024 14:32:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 852606B0089; Mon, 29 Jan 2024 09:32:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DBDA6B008A; Mon, 29 Jan 2024 09:32:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 630AA6B008C; Mon, 29 Jan 2024 09:32:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 47ED96B0089 for ; Mon, 29 Jan 2024 09:32:51 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 13D9F160A61 for ; Mon, 29 Jan 2024 14:32:51 +0000 (UTC) X-FDA: 81732590142.23.4A77911 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 177B58002B for ; Mon, 29 Jan 2024 14:32:48 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dlZY9/hn"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538769; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FNlnKqTU8a/1Fu37QrFx8MEnv7tH60nNAECNkKLkAAs=; b=433fIL774sP7EkoAmkdN+zZGxcWgqQ3nNcazEN/45JSQSVnrewJ9QdRAtIFzX4h2fKPW6Q GtcWbolh1jrTuXhGW6f2clHJixoMSTCsreb6ObKix7z4zk8Nmg2Whpx475UHk/CeTqZAH5 bwbyLShCojYV32aZIeWdxY6obEvaMWc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dlZY9/hn"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538769; a=rsa-sha256; cv=none; b=L2pUHF79WAYrPmI6lNlmxQNX/r/8Zk7y7mAVji5STTOmnrk97UXvOfVQuODUOuX5iml8d6 1ovMBVS7ZGJ8uqz0guGjfgDYDhvmiDI06hKr/FG5AzbRyZ3aRpF/zjhQ6WvaEJpmyOsGIb 2lnMhYjQcg9Ze/4GkcKC6swJaNonP5U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNlnKqTU8a/1Fu37QrFx8MEnv7tH60nNAECNkKLkAAs=; b=dlZY9/hnjnZceAb2FjAWaLhBT4yWwN+Wz8A0qXGMWmqrggv9tA11IJBDCMh9KI/3ggtaST +21nqDue6snZJA0b4cMG0NQOR7UADOoPtZmDIFNvnAkYf9CqdM+cXGSPWr+ZGJsHM4lOP6 q7f8FN53sj4/2EfVpvqFC4ujRUkCGWk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-151-unK-9W-5NjSJSLj6qcQ8jg-1; Mon, 29 Jan 2024 09:32:41 -0500 X-MC-Unique: unK-9W-5NjSJSLj6qcQ8jg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 902141C05137; Mon, 29 Jan 2024 14:32:37 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id 20796AD1; Mon, 29 Jan 2024 14:32:32 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 2/9] mm/memory: handle !page case in zap_present_pte() separately Date: Mon, 29 Jan 2024 15:32:14 +0100 Message-ID: <20240129143221.263763-3-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspamd-Queue-Id: 177B58002B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gxbx1zku8xwy1dk6dx7hfj8418q419cj X-HE-Tag: 1706538768-908567 X-HE-Meta: U2FsdGVkX18fKRoEUMivODvs5U+Kf7OL7TVWz/orhOlMb590cuz/Cw23IAkMA3PlMRfqUM6KO6qw/N/DN7cU7wlEWAw4yCDGMeRk3Wd29NFEjbucBqEvw4ZPHPy9fM4QfKvJqek31tMHvs2WtHkB5q2IGsKtEkXcbIANT38fFT583cyE6Ra+LyVk0p1m+POsL2+yhP8MMotsKbef0uUcf42ibJI1qu1SnqcWx1RSXqPU9LwOrA4l2I+/DV2Yg03F5rle+DTXN9v0xtzcKmaWUjmxVjJaoKuKAbHbwibIRjQ01+GgTybqQS1UZGj9NkSPYTyH1W83TxyvNIxnQBtVmDsAMlY9GBHF36Lr1dYtR4V6q5mWxdUS91f/Bf/MONWUoK2EV4TgHrGcFsUzHFtYS5lBcp+rx9PRmZRD7b3wyU0gKIb3HlOeoeGJLzFellsFkp3PW0FbuVfeGaUtUKmz3n0SnMsc+tNu3EXqfyTV8qpFtfnNiRMXhJau963S1g6iJA70mOpHHfWN9iVTPrQLBVTb97r0Ca2wVK31AahAgKkIu+GBaQeOXbuFoqka6mfP326FWkkBiwENrbKjsD1fbqii71OY2PBFmOr1ojkutewCAjt36ZNKyqVg9oJz4BqnDZOiu2P2btgzDG0a+zZgLAFSwgtbsKe2xqrxSA0dkx0lzYv5/I2DKGFluN2Ubn/ixfCPkn+LR1e66vPv7KtaLxJ8GDXLx33EQR7Rvt6+HEQFSK2wARc/Gb6k7n7CZniR8TKrv+u3cwd7/zsspFMGW5HMU+ZOTZhbyLOuWRB3vAwWE7sOjH2t2xy+gonB6E0uOfapqDnOnSDRZSa45ITZfXhKUhqJKrXDp44LBcldnDJen+EV/ukkD/NOv0AVVsjUbnSv2CXxEeHl+OAgFlhJDlqz7RerqIUhbvlsPK1Fp7cRqlEJpJx9plkZuiCqLrn9RPT63PR03uqLVxdeHkV z7yOxCK4 uBX7ZaPcr5wkdizi8pam0SRgiT8/pDVWjcvRCaPqSRI5CFYiJYSdDpfYR3pRUsaddG6gv5w1hlEXqOr9trSwExl3jFytiCBxLEA8D4EgB1z7u9NYWdvWV8+OQ6GdGu5cUwz3a8pbV2R2D/6X+AFMP+4MlxDUEd+CBZ4wtv1pV6e4qVuJOtsJ47oiJeZ4DpG5lmZd8vuRCmszGcChi2Mh6z+mAtRqFqxnJoIZmFvtpqH8/1d1Kl3D7bnF1Ay+9wZhmzv6ll0r5PdU/8YI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We don't need uptodate accessed/dirty bits, so in theory we could replace ptep_get_and_clear_full() by an optimized ptep_clear_full() function. Let's rely on the provided pte. Further, there is no scenario where we would have to insert uffd-wp markers when zapping something that is not a normal page (i.e., zeropage). Add a sanity check to make sure this remains true. should_zap_folio() no longer has to handle NULL pointers. This change replaces 2/3 "!page/!folio" checks by a single "!page" one. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 50a6c79c78fc..69502cdc0a7d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1497,10 +1497,6 @@ static inline bool should_zap_folio(struct zap_details *details, if (should_zap_cows(details)) return true; - /* E.g. the caller passes NULL for the case of a zero folio */ - if (!folio) - return true; - /* Otherwise we should only zap non-anon folios */ return !folio_test_anon(folio); } @@ -1543,19 +1539,23 @@ static inline void zap_present_pte(struct mmu_gather *tlb, struct page *page; page = vm_normal_page(vma, addr, ptent); - if (page) - folio = page_folio(page); + if (!page) { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + VM_WARN_ON_ONCE(userfaultfd_wp(vma)); + ksm_might_unmap_zero_page(mm, ptent); + return; + } + folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) return; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); - if (unlikely(!page)) { - ksm_might_unmap_zero_page(mm, ptent); - return; - } if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { From patchwork Mon Jan 29 14:32:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC146C47DB3 for ; Mon, 29 Jan 2024 14:32:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 347C06B0088; Mon, 29 Jan 2024 09:32:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F7336B0089; Mon, 29 Jan 2024 09:32:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 198266B008A; Mon, 29 Jan 2024 09:32:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 060B56B0088 for ; Mon, 29 Jan 2024 09:32:50 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C485C80A7B for ; Mon, 29 Jan 2024 14:32:49 +0000 (UTC) X-FDA: 81732590058.11.95CDFC7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id D07ACA0007 for ; Mon, 29 Jan 2024 14:32:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OJh2Shmx; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538767; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OkSqLVnz5/2y8MLWUf6UFgTHMoYCjTvN5uvEMNfOCyQ=; b=0XV210mhR8nrlALjiXCyvcASnCftWfiE9/tb9A0dyZeailVCCGT/DeP8Loii2yTLU7CDhe cwR2j/4yT4mR3xqPmeDbXs1opo+Ax8NLSZmiMJ46PmvW89HRQvSKxxsxJQIhNbBW3ZJMdL nDz1rkhVUxexQZoUXfbIiT+4j8znPHU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538767; a=rsa-sha256; cv=none; b=7efoRmMLO1D2FpLyhRRXiZJFYjlHjV21qoyPK50HaV68e7UdU/dOdUSU2/MxHQmP6594gd uXHHFcCkZjO/5crbv7qieTb4wusuyVn7i/ETJjgvcwXLKNMP/gP7bFajAYJXOL0BpFSwc4 opiB1iA/ct63e+mdaj/IY2Nsn+yNE6Q= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OJh2Shmx; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538767; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OkSqLVnz5/2y8MLWUf6UFgTHMoYCjTvN5uvEMNfOCyQ=; b=OJh2Shmx2U8Ejx2Oqr5Eihdzs/VG8dln4YzNP7+gzND90vUObDov0FJzfZ9A2AhE+l3UJ7 WHG+g4xow3Zmw3txXhCXrvkH5KamXCyjjjbnQhdY9p4WXM5CtI4HSzMaIqGE4boYu7pl9b HfSQ/DahZCiPklyWTH9v9vtMi9h5qZM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-332-tQCqrTzqPz-8PYoPDfxKqg-1; Mon, 29 Jan 2024 09:32:44 -0500 X-MC-Unique: tQCqrTzqPz-8PYoPDfxKqg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8BAC885A599; Mon, 29 Jan 2024 14:32:42 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id CCEB7AD1; Mon, 29 Jan 2024 14:32:37 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 3/9] mm/memory: further separate anon and pagecache folio handling in zap_present_pte() Date: Mon, 29 Jan 2024 15:32:15 +0100 Message-ID: <20240129143221.263763-4-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspamd-Queue-Id: D07ACA0007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kanmfge65unoihmaig7f6rpy8zfyykya X-HE-Tag: 1706538767-301588 X-HE-Meta: U2FsdGVkX187WE5eD77MuKWt0Rg6LDjpM2GijJSJE8t9nYWylUSuYPaZcB7CyMFZDeX7gJV8vhxUVwPY+tcE71INYc8/k/v/silSr/X0aEXKq4yFallwgxeDhhnW1LXZIYAKJwD4Qwt5QPVVEIqUqSNFeofxtmlFNqD03JI8NxW/A55RaI+XkEXMPXq/RdOf0aoTGKUT/8Pq55SEzH3QM/t5/y6ScOT6TbRIfKRfOJ1IYyNdCtyB/cem8VW6AA6XHt4PCZ1P5qei4YYXbWhrp4xqGxejdexe+ED3nPS4h9aF32KXisVibrAA+Luj7gX6CxWNdw50FbfATqFjReZJPpZeq2kui1fnl30tFpSMN1JEHDQVoKQ4Oqe4eLGnL0wjpFa3/HeuvWX7C1+nFYsWq70oiVUosWAxY7e5EPXCQTlGxHtWveV9gK1wkkq3lK5z6dA61np6CpX2cENNY0TrvACniMvjJfY7X/3BH+JMcLvJ69eu/ulq8eXIqCUzkpVMui8BdTI1kgvHdPdwaP4enJAC37s1U01azan4HnqKIYhtT3myvo/lqc9a+WIB99fn/34OC8sPo6NTHinodXsOsajmNCpLIMmXXAkpPj9wMAPlmsYpLV1PwdnwpqOfcbKJkkKzxDXwPOPI97J9fBHwA1JLxk/9h10C3zUVp/hKDr6pO9DI2rxJefJtBw34iiw84xQOdIgM9O9321dOMZHPfBQ9Z4CwNEq8W76Si5aGtk848zGhtgz2KHrouWSNtFcLbf4syfmO7ZOyvEQH/B1gPO5kFt1iw6nkH30GnpYp1blEIas/Rc18c8ph/AQRb+bGgi3yCK8H7uDr1AXqROgzPbUd38zXunkDnNF30vqQ+gU4C7B4s/ljp8JvAsOjfGQke79Ql1Qif/MdYu2wVElu0nyVOo3xJDYNJ8411HXiU8oyLPMUjniFgE8Z8zzF9DshEaZpHiyQW2KrDW4FL5i PHJPCyrv MPpE7dgIMTM6MZ5xxOaHRfyjUG93Dr/nk9w99aTK/UyRMHMikIXgc9e63p27r6lt4kuzq55zbelp0mftyS96h7r/6rpKdNrXUclTc4o6DjH6IbQRFNV61/tvCUP05fVMsrlSBEkjARoH2g7Vr+gvYS4Q8NG8aoOkmxCb3Q/oeTCXrG6SHSfBvRYg+kAWANghpCAEYi9tT7u4vWqetw7xgtq/brhbYW0Zyejq5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We don't need up-to-date accessed-dirty information for anon folios and can simply work with the ptent we already have. Also, we know the RSS counter we want to update. We can safely move arch_check_zapped_pte() + tlb_remove_tlb_entry() + zap_install_uffd_wp_if_needed() after updating the folio and RSS. While at it, only call zap_install_uffd_wp_if_needed() if there is even any chance that pte_install_uffd_wp_if_needed() would do *something*. That is, just don't bother if uffd-wp does not apply. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 69502cdc0a7d..20bc13ab8db2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1552,12 +1552,9 @@ static inline void zap_present_pte(struct mmu_gather *tlb, folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) return; - ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); if (!folio_test_anon(folio)) { + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); if (pte_dirty(ptent)) { folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { @@ -1567,8 +1564,17 @@ static inline void zap_present_pte(struct mmu_gather *tlb, } if (pte_young(ptent) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); + rss[mm_counter(folio)]--; + } else { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + rss[MM_ANONPAGES]--; } - rss[mm_counter(folio)]--; + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + if (unlikely(userfaultfd_pte_wp(vma, ptent))) + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (!delay_rmap) { folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) From patchwork Mon Jan 29 14:32:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5DCFC47DB3 for ; Mon, 29 Jan 2024 14:32:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CC8D6B008A; Mon, 29 Jan 2024 09:32:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 62C656B008C; Mon, 29 Jan 2024 09:32:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47FF96B0092; Mon, 29 Jan 2024 09:32:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2B6B66B008A for ; Mon, 29 Jan 2024 09:32:53 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0CC8B40A37 for ; Mon, 29 Jan 2024 14:32:53 +0000 (UTC) X-FDA: 81732590226.27.DFA55E6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 38671140012 for ; Mon, 29 Jan 2024 14:32:50 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QNIU1gJX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538771; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L2zNeb5SO5KkYO9vfWjJtlZWbswbqsHyyrv464V13y4=; b=VUvfpaGNZa2hUsjHZRWQLfzMoMdBW/COsqx+ncSOliuQ4aIGOBcDlBdObpmVkE6rjy+gn1 +gcVwIIq/fYXzRpui9Z5WkdDPQuwpgQwJYi6zlhuKunDrcv0DfS+H9xvBXRRtwdz+wn2nY S9yH920g68aP4e5F9DfIrCYJPVnMxzI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QNIU1gJX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538771; a=rsa-sha256; cv=none; b=Y1pxmnRkdrXA/YMfTEfFeeh/+jGBn9B5dfDPqfh4SLtgHowALIw7jjp3zwi2xIJY/dGXii ON7GE70EFTloaxP+B8UVZcl5Y0ob7KjrsWWAW8o0zNYLADSePDv2a7rIQivacWsXKs8yYn 9TH6/Nu3/Pon91zrn9c5GthyCcmGykw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538770; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L2zNeb5SO5KkYO9vfWjJtlZWbswbqsHyyrv464V13y4=; b=QNIU1gJXB9wIHa6WsKmpy7UZxXt3WSJD5ezjENymdJ3GPs2wOltEEItlbHvWf41I+bCF1V zey8WBeeRNKLn6CDbHq2hanQVl+tE+jYnWC/gsEvVY24cCt67qrQw4IrIhQ6Y5L/4Y+Vi5 kdkIKb+zILezFEkYIpyZWD6ixE9jBLY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-VwTjaewtMEGP-PpguyfRyw-1; Mon, 29 Jan 2024 09:32:48 -0500 X-MC-Unique: VwTjaewtMEGP-PpguyfRyw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 796401013767; Mon, 29 Jan 2024 14:32:47 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id D15AFAD1; Mon, 29 Jan 2024 14:32:42 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 4/9] mm/memory: factor out zapping folio pte into zap_present_folio_pte() Date: Mon, 29 Jan 2024 15:32:16 +0100 Message-ID: <20240129143221.263763-5-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspamd-Queue-Id: 38671140012 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: usrsp9m7efihadto6dawpub4sk7xpcqm X-HE-Tag: 1706538770-912899 X-HE-Meta: U2FsdGVkX182jQ8aIVoghbo9HsyP0Mk4ficQ4BVcRxEaBtOXKqk2KD9LGugoT5zZubYL5Mc1ezTH+z4/2mytGZxxrWuZHqGn6I4ObaN8ttYIapNoUAzvFQEwCBl1Mz8WxF3Yb2t++24McqAPSmBb/d8HxgrBzBLo+YFGtG7P0DBve7vd/GUzlgGGrybnPkfjH5jIwi8TqkY2wM7DBKVbnn2TFqQiKO9CzPXwhlSH6Gw6e0e497q5kzbdn+cskYM6sPHwIeDMNlUt3otHklTd7UyHCAi2kpMYKA3wjib+ebVyUvGrL1H7LaG1qjsREMAuNu5FhpDZEf/Qdhizrn13dWzz3IL6OAJSn+Q7/gkUuHnnO2WNg0WyI5LbXz6k0vxfU++N/831OYx45vt+A4EW31TuQ5m2vzoHtXpm3lWqstf8iTGarA32JliMkKKXqGW4l+xJHtl1ewotpUWyvgGTwwA56NClsUpaNiZ0S8kaZ8suDAVUiLPX60kcr+ov8T10mIFCUypGj2uRqLVZsu4oBeibQnYdLUgc7NNoOU9+yd89T3Uol1jPJAVDpx2Y9vDZQ4LPumtYDVSJuIfH+J6SAEQzLGCXN5+XsdKAdi/uDxezEi3k9zPh3fmG/xIu8CnXW2g5Xm068xcQD1GxQHdslTlqtXw9myVF+Xz7RoEmDaJh489dxZhvqCE/b/fMy42QDKCTBQ+VZXP2LIrJykE0MGhI7cOQpVeaR1ey5vob2P/UXtPIzk8epPWGMoR9++6HITf3E1yrflFNenVg1Vnuuk0v4rpiomLSM9PgLjrioDZZX3wXChPAI4l242YnkiSQPIScA7MasHl1r7b4fl9cZsOJaAop88yaAiEXSSxqlFbqdApaWEINA7zMoAAdeZtWkStvddpAEPaItLILYWl4uP7CnWtk96wOCKWEzwrBwOpRzyCdzNo6K6KrXTevLw1U2JX/VGAnmaPf9vjzKvj JirAEHY8 ymPa6N8/fsZpc2NFfdMgqf1Eyykhu2vLPoYidupSLDEMnaxJyQLR7vTrZmloQReLUZwyT9fV5zCfyPtGmMf7bH29ap8aSxF+FE2cVbOcE2xIVCrFd9yf356PM7ftbuWZsg24Zxqq0UgqRTPao1V8bZYi7yeg37lDTZrdp5+JgMgsBVUJyBwgFiffEws+ksqv7+21Y+zBxiRpaf8RyC6bt76tFxBHNbww/Pf4T X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's prepare for further changes by factoring it out into a separate function. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 53 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 20bc13ab8db2..a2190d7cfa74 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1528,30 +1528,14 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); } -static inline void zap_present_pte(struct mmu_gather *tlb, - struct vm_area_struct *vma, pte_t *pte, pte_t ptent, - unsigned long addr, struct zap_details *details, - int *rss, bool *force_flush, bool *force_break) +static inline void zap_present_folio_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, struct folio *folio, + struct page *page, pte_t *pte, pte_t ptent, unsigned long addr, + struct zap_details *details, int *rss, bool *force_flush, + bool *force_break) { struct mm_struct *mm = tlb->mm; bool delay_rmap = false; - struct folio *folio; - struct page *page; - - page = vm_normal_page(vma, addr, ptent); - if (!page) { - /* We don't need up-to-date accessed/dirty bits. */ - ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - VM_WARN_ON_ONCE(userfaultfd_wp(vma)); - ksm_might_unmap_zero_page(mm, ptent); - return; - } - - folio = page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) - return; if (!folio_test_anon(folio)) { ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1586,6 +1570,33 @@ static inline void zap_present_pte(struct mmu_gather *tlb, } } +static inline void zap_present_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned long addr, struct zap_details *details, + int *rss, bool *force_flush, bool *force_break) +{ + struct mm_struct *mm = tlb->mm; + struct folio *folio; + struct page *page; + + page = vm_normal_page(vma, addr, ptent); + if (!page) { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + VM_WARN_ON_ONCE(userfaultfd_wp(vma)); + ksm_might_unmap_zero_page(mm, ptent); + return; + } + + folio = page_folio(page); + if (unlikely(!should_zap_folio(details, folio))) + return; + zap_present_folio_pte(tlb, vma, folio, page, pte, ptent, addr, details, + rss, force_flush, force_break); +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, From patchwork Mon Jan 29 14:32:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB5B1C47422 for ; Mon, 29 Jan 2024 14:33:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A3DD6B0095; Mon, 29 Jan 2024 09:33:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 653306B0096; Mon, 29 Jan 2024 09:33:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A5CA6B0098; Mon, 29 Jan 2024 09:33:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3C2096B0095 for ; Mon, 29 Jan 2024 09:33:02 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 05A6D12038F for ; Mon, 29 Jan 2024 14:33:01 +0000 (UTC) X-FDA: 81732590604.30.FF38178 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 3F489C002E for ; Mon, 29 Jan 2024 14:33:00 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="POJj4/lN"; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538780; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hRXeU82IBxy/RcnY5wfdvSjzmzqpCs6vJAdTnhTQH90=; b=Hzw9XU0DcXZMunE8AH7jzfgduAfGkJIzlnSPqpAWO09L74ryhWI0xMJ9cn7c6SKnEF+UNo NG21jcxwS4JG1lPMm5FP3WoxTLPQO2iDyYtwWV8dUKiUAAHWR/TQ6GzcaXtXg06rHjX+Jv HNf1ONOQ5WKO43nk/5QsTC1UVzqwTAM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538780; a=rsa-sha256; cv=none; b=uGJpQbcTKcpx5+caBJwNDzhLvk0eOIjdIF8OqkOdrcIcH/e7jUIuwkDKmuNX6bOIC+aZ00 NwpVaDKjQF9gS8WVTflzcCaZUX1XVi1d9OOexHmkh91jGS3fsJi17QfwjYhxBtJ47vz7GU NEbqDFhNm6aVgf3LzabB5W3v0XjFA8Y= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="POJj4/lN"; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hRXeU82IBxy/RcnY5wfdvSjzmzqpCs6vJAdTnhTQH90=; b=POJj4/lNGKrPMzIG/2zii53FhYQWi8vNmGtPCJ/aJD3+d27Ag3c1UhRujedqdxTGdfVUE0 m6MkFsFGP2ZDVfS3onBCiMQI91m7bdTnK/e4JPmmbSRQhZxww6/7Lf1zIuzUHF9HarUfUO isC4DgdJrKtyqJ4BHSUukJf+xtLJTVs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-654-RcYHT5HwMdeLqdQ1EEIoXA-1; Mon, 29 Jan 2024 09:32:53 -0500 X-MC-Unique: RcYHT5HwMdeLqdQ1EEIoXA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 61CE93C13AA2; Mon, 29 Jan 2024 14:32:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id D5AF0AD1; Mon, 29 Jan 2024 14:32:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 5/9] mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size() Date: Mon, 29 Jan 2024 15:32:17 +0100 Message-ID: <20240129143221.263763-6-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspamd-Queue-Id: 3F489C002E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: q4qjaphcxu8yhhnpxsq7tc3kfu8jnizf X-HE-Tag: 1706538780-547459 X-HE-Meta: U2FsdGVkX18yioCJwhpr9h3ACZGwY1Q9MD/4PExA6LSRU6YXdvotSqebi0OBYDQ3xJMpdLj58WIkG4R7JD9IdgjCJqZdRmld+N6HQOjU2as+AxpiSsT8xHUCxSTOtDuidAr4Mt25ovxXZy1lyaXE7Y+cBtdhUmCdslCe2A7szOgJB8afuzY3cNgay9dhyvWREVUUnpV4YyyDBNtoHB4CWp6FiSGrbTFE6+GE4Wn4Bn6s5Lg1a+UdMKmGCXUgKoQeSYq4GMndshYBG30vpKsrc9w40n6tWNBcUKDnPNjkT60tXUOzvLGA9dxCYvvaU9k9hB3mYlgi2l8hX1L2WX5IBxgLad1sPFkR907Pd6oA5zjJcoh90SYqyOAdkz/Iyagi8f7v3NL7C/z9t1bdqe1/ojwkY6z+FwBrWor3Hoiq79/FKFCeGT583vqgEU66KnfYCTZj8UbIQmWpZs6RJa7ebp8OLcvvCEJCNVlI3xIGCf+XaCFbdJ5t4bfIuN6+krraWgZD3D8IQfrkqMoASF7DSy0TPpGeEx7B2uejQSnOom7BOorFWi7shlvSKuBV7LfnM/JvD3AsBVVbP5JXTxNYikzYczP+4S2/cuHah11iAyZkod3EGQ59i1BrSqURh295Q1ypcRyVDrZik3T3THS0Wf3wzdXVQDpfBqYSmjDQCai4WqKyiTUKXCcd1Y64/HhV1vxaHtfsmOogzE9in2MkgIP1j9vkv0mwco2XPQAmTU0Kd+y6HHbtpCXsbBvmd+brcpBD1JVRRbOvKeQmQyhIFnsBTM4yaYAEoxlFpSzzEJfQCQ4L8WUgW/FvaJVPvFR5orlHRIIZ+m4dY2pv+fune1r0fxcndIlOnUvAEPiS4WqL4EGOt6M+nqmE0XKO+jW6YT4lFPjmwEo1KfTcEX5b1xToQF8QnGp2OX26ujt9pR8xcStZW47DWF+16FW+uGzhvgSeWR2Bl1vbRs+t1lU YuY9Vc+j ofFbePDUWq72KECeNVVBJ5HORS+hiZz5qzXPa0hPLS0VMT+arCTuOE8TUg4+l64StDaw0g19k52w2Yvzmy4dFHSdLu1RWW80ysTGx+dy7tyzPd+RLCUomJ6ydTDyOT3kD80lrvE4NQLiBfapMJrCuJcSX7hTkLjpUYByQftEOb69+rwp3JNfUGPeXq4Jys71dfxL7oYgwFx3ol2eKdVzd9rrs2oQFJ703XGeh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We have two bits available in the encoded page pointer to store additional information. Currently, we use one bit to request delay of the rmap removal until after a TLB flush. We want to make use of the remaining bit internally for batching of multiple pages of the same folio, specifying that the next encoded page pointer in an array is actually "nr_pages". So pass page + delay_rmap flag instead of an encoded page, to handle the encoding internally. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- arch/s390/include/asm/tlb.h | 13 ++++++------- include/asm-generic/tlb.h | 12 ++++++------ mm/mmu_gather.c | 7 ++++--- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index d1455a601adc..48df896d5b79 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -25,8 +25,7 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); + struct page *page, bool delay_rmap, int page_size); #define tlb_flush tlb_flush #define pte_free_tlb pte_free_tlb @@ -42,14 +41,14 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page * has already been freed, so just do free_page_and_swap_cache. * - * s390 doesn't delay rmap removal, so there is nothing encoded in - * the page pointer. + * s390 doesn't delay rmap removal. */ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size) + struct page *page, bool delay_rmap, int page_size) { - free_page_and_swap_cache(encoded_page_ptr(page)); + VM_WARN_ON_ONCE(delay_rmap); + + free_page_and_swap_cache(page); return false; } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..2eb7b0d4f5d2 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -260,9 +260,8 @@ struct mmu_gather_batch { */ #define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH) -extern bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); +extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size); #ifdef CONFIG_SMP /* @@ -462,13 +461,14 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { - if (__tlb_remove_page_size(tlb, encode_page(page, 0), page_size)) + if (__tlb_remove_page_size(tlb, page, false, page_size)) tlb_flush_mmu(tlb); } -static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page, unsigned int flags) +static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, + struct page *page, bool delay_rmap) { - return __tlb_remove_page_size(tlb, encode_page(page, flags), PAGE_SIZE); + return __tlb_remove_page_size(tlb, page, delay_rmap, PAGE_SIZE); } /* tlb_remove_page diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 604ddf08affe..ac733d81b112 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -116,7 +116,8 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, int page_size) +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size) { struct mmu_gather_batch *batch; @@ -131,13 +132,13 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, i * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = page; + batch->encoded_pages[batch->nr++] = encode_page(page, delay_rmap); if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, encoded_page_ptr(page)); + VM_BUG_ON_PAGE(batch->nr > batch->max, page); return false; } From patchwork Mon Jan 29 14:32:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8843C47422 for ; Mon, 29 Jan 2024 14:33:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BD3B6B0096; Mon, 29 Jan 2024 09:33:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 36D2A6B0098; Mon, 29 Jan 2024 09:33:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E7456B0099; Mon, 29 Jan 2024 09:33:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0C03B6B0096 for ; Mon, 29 Jan 2024 09:33:06 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DD6CDC0A2A for ; Mon, 29 Jan 2024 14:33:05 +0000 (UTC) X-FDA: 81732590730.14.BB3D592 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 0C316C0024 for ; Mon, 29 Jan 2024 14:33:03 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DM6pZeq4; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mNf9J4fpCfYM4JsYJaH2xcr0nS1jv7jO8GYZWdUDb4k=; b=Z+cLDuVhvL3g947OD7vijMw+okK3U5eWfeps4govr/c4mg7hcvoCuqvRaaAedJEIfBLqji 0pmcPqcrPwwyQYIyYkEv2KpVAEOp4ZJdho/Du1CRH077zxmIFecf6ONTBoU+7ja02tzbtx XviFukSji36xy8ufpHJkCaKBwd8C8qw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538784; a=rsa-sha256; cv=none; b=k3oWCyCqfNBmQcglBitXzT+2L6k5lB8SDl/5zWcfQa95AlRmGhuf2FymeXAZK8jje0gsyI bvKQUJ8pxZX1P2HvYmYnRps5i9se8NkT1XdQ90fXIdJOYnCQfjlVrNoYCzalfMB/4t74oj RmxtNPPqK4Na9PTkCkW9/kAznYx3a1g= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DM6pZeq4; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mNf9J4fpCfYM4JsYJaH2xcr0nS1jv7jO8GYZWdUDb4k=; b=DM6pZeq4ge7oQvggcpLdBJIIO6GHCDJCdbCGf4H5sByIka12OW2JXgDZJ6IOx10amgTC94 aUGfJSrcMMmGI1OOidDg7JJFv4p+VZ+UxKJ4GO1NEyUHFeBUi8y6UfqaBIAQQqtZqiMeo/ 7ykXnEQV4k1Oa1zpkoW95wUPt2qFEMU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-136-7zFZXTSTNpira0N1ASkqtg-1; Mon, 29 Jan 2024 09:32:58 -0500 X-MC-Unique: 7zFZXTSTNpira0N1ASkqtg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 55A0B185A781; Mon, 29 Jan 2024 14:32:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id C0D15AD1; Mon, 29 Jan 2024 14:32:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 6/9] mm/mmu_gather: define ENCODED_PAGE_FLAG_DELAY_RMAP Date: Mon, 29 Jan 2024 15:32:18 +0100 Message-ID: <20240129143221.263763-7-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspamd-Queue-Id: 0C316C0024 X-Rspam-User: X-Stat-Signature: 3zrbzud9iacpfms797arewh33erkybdy X-Rspamd-Server: rspam03 X-HE-Tag: 1706538783-136308 X-HE-Meta: U2FsdGVkX1+BWWR/7Hmf6ZV4kIdOAVbrI83GDYiM2hBnIL4zAyqfs3e3W1rnX90LIJOYJvvezVOYcDE6Ok9KAeJYshRqc0Od70Y9iP/Kq65mhfHPRqszanO7VM3orb4ZviJjtlTqd6K0p1Kr4p31OuGqlu6xPLRZZXbw/IoTi8F7FJcacWGRdtS3itDiVIEJ3585WSIcWHThNb1GvGsw/Rvf8tRTWVbAdBJWWL6PVz+pzVNrtgCy2efC0eGdJbSNnmYyYSBJzgrcHPOfD38Iv+7oHZAsckjgIcDhdpSVisaxY7jrJ12AQCixunwY61JlEwsQHoctdOSq3wpuMGshD6H9BYdJPQRtZ1TjWUI08YTH3MUtIe/1cNVD7oouJuJwgRCE9JfNjMbf9tgUt1dBaSzrzr9TIP/aaulI/6ptI/n0zS3C5cP4aaHXpz5TP4yT2GOt5QaQvXkwg7kuEis6WG7ekpJBSIquXE4NW2s89695BekLAitjmvuuymyoAmGqIWyotj0auRfViGLilBCxGFgSpW779tXZ9p69KVvNW06iAK9hWaFApH9vTmcwkI9Qo72PIUaLF261RruT00vaAzUAMQeZb3R6f2pQCGrin4sQCyDrIe9m6JJqBW/gkKwTH0nSovRPAlXVm3d9eFA4lvs0zCpQ7wERDRHMDxKoKPrEmfc5SB0ThlVPgRrnyxxZK6rFHwynUh3s8xrFsF6W4lReFxAXdqTyBJ0Lnza7YwBjco2PWvDuau8SwmEhR/aRSy7yHEw1NySMX3FAqTfEBiP6fi/aMJIHNnI03LMZHSfb3hz436s+Ny+OtwQoJvhLnqEkSponcKsXWN019N7IsBlnjTJ1LzYLvDYPpT0EkmoJmvGx1xoTaMoJdIVv6EKdFDiRM+TckTl4Mk7yoTkToVqtbU3lH/Qmcpgx/VybSgYZFVzmPOdsnSmcePcseso0kj4ZoVkFTC0GXpIfkZ+ AaMwuCNH M7jYTSqSM24WtY3cWkaQMkGZEUPFrViskWwhvEE0XkwD9pbh27Qqcd6eaZNXtZUi3fmhPTyVyVvJyy4R8PfGR+MRb62diV2wXbl8bS3oFzw1yJpQb9kpY4TGQaJCIBP8GYOK0cJTcjvzbAwScVy7gKZv1U7L3aCjmuPAlYrXlX+q1Wkg3IYZa+OkMfWJ6lLgRzm6tVEjyTGz/b3LJTpH+G7dhmT7kWSYW/4Y3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Nowadays, encoded pages are only used in mmu_gather handling. Let's update the documentation, and define ENCODED_PAGE_BIT_DELAY_RMAP. While at it, rename ENCODE_PAGE_BITS to ENCODED_PAGE_BITS. If encoded page pointers would ever be used in other context again, we'd likely want to change the defines to reflect their context (e.g., ENCODED_PAGE_FLAG_MMU_GATHER_DELAY_RMAP). For now, let's keep it simple. This is a preparation for using the remaining spare bit to indicate that the next item in an array of encoded pages is a "nr_pages" argument and not an encoded page. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/mm_types.h | 17 +++++++++++------ mm/mmu_gather.c | 5 +++-- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..1b89eec0d6df 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -210,8 +210,8 @@ struct page { * * An 'encoded_page' pointer is a pointer to a regular 'struct page', but * with the low bits of the pointer indicating extra context-dependent - * information. Not super-common, but happens in mmu_gather and mlock - * handling, and this acts as a type system check on that use. + * information. Only used in mmu_gather handling, and this acts as a type + * system check on that use. * * We only really have two guaranteed bits in general, although you could * play with 'struct page' alignment (see CONFIG_HAVE_ALIGNED_STRUCT_PAGE) @@ -220,21 +220,26 @@ struct page { * Use the supplied helper functions to endcode/decode the pointer and bits. */ struct encoded_page; -#define ENCODE_PAGE_BITS 3ul + +#define ENCODED_PAGE_BITS 3ul + +/* Perform rmap removal after we have flushed the TLB. */ +#define ENCODED_PAGE_BIT_DELAY_RMAP 1ul + static __always_inline struct encoded_page *encode_page(struct page *page, unsigned long flags) { - BUILD_BUG_ON(flags > ENCODE_PAGE_BITS); + BUILD_BUG_ON(flags > ENCODED_PAGE_BITS); return (struct encoded_page *)(flags | (unsigned long)page); } static inline unsigned long encoded_page_flags(struct encoded_page *page) { - return ENCODE_PAGE_BITS & (unsigned long)page; + return ENCODED_PAGE_BITS & (unsigned long)page; } static inline struct page *encoded_page_ptr(struct encoded_page *page) { - return (struct page *)(~ENCODE_PAGE_BITS & (unsigned long)page); + return (struct page *)(~ENCODED_PAGE_BITS & (unsigned long)page); } /* diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index ac733d81b112..6540c99c6758 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -53,7 +53,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ for (int i = 0; i < batch->nr; i++) { struct encoded_page *enc = batch->encoded_pages[i]; - if (encoded_page_flags(enc)) { + if (encoded_page_flags(enc) & ENCODED_PAGE_BIT_DELAY_RMAP) { struct page *page = encoded_page_ptr(enc); folio_remove_rmap_pte(page_folio(page), page, vma); } @@ -119,6 +119,7 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size) { + int flags = delay_rmap ? ENCODED_PAGE_BIT_DELAY_RMAP : 0; struct mmu_gather_batch *batch; VM_BUG_ON(!tlb->end); @@ -132,7 +133,7 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = encode_page(page, delay_rmap); + batch->encoded_pages[batch->nr++] = encode_page(page, flags); if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; From patchwork Mon Jan 29 14:32:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEEDBC47DB3 for ; Mon, 29 Jan 2024 14:33:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 459386B009A; Mon, 29 Jan 2024 09:33:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 40A2C6B009B; Mon, 29 Jan 2024 09:33:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 283F06B009C; Mon, 29 Jan 2024 09:33:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 15B576B009A for ; Mon, 29 Jan 2024 09:33:15 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D6AD71C1266 for ; Mon, 29 Jan 2024 14:33:14 +0000 (UTC) X-FDA: 81732591108.02.070EC94 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 09923140012 for ; Mon, 29 Jan 2024 14:33:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ad1FhxUp; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538793; a=rsa-sha256; cv=none; b=TgF0HVMq6YRIADsD5pxfg3Al0dK9zvBWamzkQt9AGEJ7lIcv2Vmcqm/EYfsUvJxfMzZhZZ W++dDF1Z483GJ6Madr6fq38V2KYYGrvtDN9mrC6dRcUVXLR4rOeutnn8JIGS+yNSUh0LzQ RTnRA4EOIg6w50IZCVga7dW7e5NVmso= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ad1FhxUp; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+kOcFg6PGNbb+ifN7/MrtgmE3FIL277TBSGnt13KTXI=; b=YkU+Lh6NZ6ZQ0Ih6j3O+/MJU+rhAufntPuaHGhF9O0E/JoW73R21oj9kz+O6xfbbxvEmg3 1xEso0ibueMx147y/WKl5lx0LAGpquAlGNyBsjcs8rwZBJrwHSDDyDxmb/C2AEgL1jLsbr UfeVgkThcPUEJH/aUHrhx5VTGRl/CjU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+kOcFg6PGNbb+ifN7/MrtgmE3FIL277TBSGnt13KTXI=; b=ad1FhxUp5TAvb7lcNA/r2BXiM3Ye7HRNvEDJKnsD0uCrTiQZjctcjLV+Eu84n8JhVT/K1n iwvqKUm3AdZmUueoeLJ9OAPziRcd0m+IMw8PrADtzJRCg3Xh0HfqAd8rQgN1Ur7TgnvkD0 Ca0hBcW1DoVqcc/dzC1r+MGMiDJbSg8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-266-skzeZ3g1NrqHlItJu1KF5Q-1; Mon, 29 Jan 2024 09:33:03 -0500 X-MC-Unique: skzeZ3g1NrqHlItJu1KF5Q-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 518D43C025AD; Mon, 29 Jan 2024 14:33:02 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id AC506C3F; Mon, 29 Jan 2024 14:32:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 7/9] mm/mmu_gather: add __tlb_remove_folio_pages() Date: Mon, 29 Jan 2024 15:32:19 +0100 Message-ID: <20240129143221.263763-8-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 09923140012 X-Stat-Signature: rwe4bucswbqaf19sndwan9ip4rg9nri3 X-HE-Tag: 1706538792-87150 X-HE-Meta: U2FsdGVkX186c1SkjJB5ynR3DYvagWDj9OJd55EjuHT28gFJIkfoQSlPq2lXn/z/RHWWqx8F/JpeBYVYGa+OoVRSbkN2wedf/vvdNxo3VUwlCbr/zuEsOYYDMEVGbpkcFiAhnXfW0U2QliLkEgX/qPOHv4CHZrKrCJiMs2b6+Z372I/3FiRUZJ7DEQFVCo1lctBaMMnkbeiZuethbuZrhSBlVP3Z5O4h4B+lmc02/rn/eurTAFIbZ8XQwKgNOCmHyRYk9ZOtDgVoolIPtX01G6yZA0V75ds3R5rlyqfAxomEMWrv8u+ewjflua40b+bf/koE1Fw0cYCHNJPhOwQv32HEEB9oISSsKY778z0dd5mgFa9Icmdu5XISORFGxDBPuJruWqGBle7tGsL0wryYnloV9e9ADdHebXyBSVCX8+3OOx9Djk0Peg5L40nd/YfNbi4/iZtWzp+ZErpK+ramRII8OXIEC1ny938O7k/OOKHUqgKDo9PKurM0Bi5HpoF7JCHvguLND62JlVdnexOFHYCGYpQe9okFsiiRh79Qmil8D5hFZ1nBBqlH49k3NxulT8GL6ba0Mww4DtPvStnIWVCr/0S3vSi81kBoAdv8B+CLyLxlqmmJDy9CiIe8d+lJnqryH6IDG5ND5tab6ClWRTeFIK3fRw9VyY4+uK+OPCqFqvxjQVOb/cX/IYBG0jlOJE43qGOi1lQAj9PXznE8lOGytAfmg62ugfBabK6KYn0DgCt2MElPTR4shDn+RAV+aE62AMRw9DQAN4RdGkKU+FKONo9tlTPCk1D7i7jpPJJgGEaDHUeJ/tobaaQrgopIKndioRa5e2UcNpuRkXWiA2Ep9BU6mts/Tk+u4tOuvIwJr0bx+Pby/IaqhEU0OASVisKYsfq1Xp6RZC9NfAWQrqnWy6OKfKxESej5bJ+FZAnI1lJxrQ6j8t4efWwtzpCdtTWmQ4bB2P00lP/yKJ9 xUqEQsTf 70yJEyV7Rojsj5RmhAdkJY6BGRtbgFCr8TDDZJlqZY/PyfIROXB8T8L+DJITEXA4qd72C4t/28hmIxJqXkxliDEsjNqiolkCVbYcghExI8R9EIgFu3gBHr65pTmqjUyfEyzixjQ1sHE1X+65WUm6MElJmXsaz9VdB0furhm1C/JEvS1P8KfMOsMzE43PZN/ivSTJ9Dve2Mlp9Rk/TAj38QG3wTpm9Aapwt5Lm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add __tlb_remove_folio_pages(), which will remove multiple consecutive pages that belong to the same large folio, instead of only a single page. We'll be using this function when optimizing unmapping/zapping of large folios that are mapped by PTEs. We're using the remaining spare bit in an encoded_page to indicate that the next enoced page in an array contains actually shifted "nr_pages". Teach swap/freeing code about putting multiple folio references, and delayed rmap handling to remove page ranges of a folio. This extension allows for still gathering almost as many small folios as we used to (-1, because we have to prepare for a possibly bigger next entry), but still allows for gathering consecutive pages that belong to the same large folio. Note that we don't pass the folio pointer, because it is not required for now. Further, we don't support page_size != PAGE_SIZE, it won't be required for simple PTE batching. We have to provide a separate s390 implementation, but it's fairly straight forward. Another, more invasive and likely more expensive, approach would be to use folio+range or a PFN range instead of page+nr_pages. But, we should do that consistently for the whole mmu_gather. For now, let's keep it simple and add "nr_pages" only. Signed-off-by: David Hildenbrand --- arch/s390/include/asm/tlb.h | 17 +++++++++++ include/asm-generic/tlb.h | 8 +++++ include/linux/mm_types.h | 20 ++++++++++++ mm/mmu_gather.c | 61 +++++++++++++++++++++++++++++++------ mm/swap.c | 12 ++++++-- mm/swap_state.c | 12 ++++++-- 6 files changed, 116 insertions(+), 14 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 48df896d5b79..abfd2bf29e9e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -26,6 +26,8 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); +static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap); #define tlb_flush tlb_flush #define pte_free_tlb pte_free_tlb @@ -52,6 +54,21 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, return false; } +static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap) +{ + struct encoded_page *encoded_pages[] = { + encode_page(page, ENCODED_PAGE_BIT_NR_PAGES), + encode_nr_pages(nr_pages), + }; + + VM_WARN_ON_ONCE(delay_rmap); + VM_WARN_ON_ONCE(page_folio(page) != page_folio(page + nr_pages - 1)); + + free_pages_and_swap_cache(encoded_pages, ARRAY_SIZE(encoded_pages)); + return false; +} + static inline void tlb_flush(struct mmu_gather *tlb) { __tlb_flush_mm_lazy(tlb->mm); diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 2eb7b0d4f5d2..428c3f93addc 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -69,6 +69,7 @@ * * - tlb_remove_page() / __tlb_remove_page() * - tlb_remove_page_size() / __tlb_remove_page_size() + * - __tlb_remove_folio_pages() * * __tlb_remove_page_size() is the basic primitive that queues a page for * freeing. __tlb_remove_page() assumes PAGE_SIZE. Both will return a @@ -78,6 +79,11 @@ * tlb_remove_page() and tlb_remove_page_size() imply the call to * tlb_flush_mmu() when required and has no return value. * + * __tlb_remove_folio_pages() is similar to __tlb_remove_page(), however, + * instead of removing a single page, remove the given number of consecutive + * pages that are all part of the same (large) folio: just like calling + * __tlb_remove_page() on each page individually. + * * - tlb_change_page_size() * * call before __tlb_remove_page*() to set the current page-size; implies a @@ -262,6 +268,8 @@ struct mmu_gather_batch { extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); +bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, + unsigned int nr_pages, bool delay_rmap); #ifdef CONFIG_SMP /* diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1b89eec0d6df..198662b7a39a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -226,6 +226,15 @@ struct encoded_page; /* Perform rmap removal after we have flushed the TLB. */ #define ENCODED_PAGE_BIT_DELAY_RMAP 1ul +/* + * The next item in an encoded_page array is the "nr_pages" argument, specifying + * the number of consecutive pages starting from this page, that all belong to + * the same folio. For example, "nr_pages" corresponds to the number of folio + * references that must be dropped. If this bit is not set, "nr_pages" is + * implicitly 1. + */ +#define ENCODED_PAGE_BIT_NR_PAGES 2ul + static __always_inline struct encoded_page *encode_page(struct page *page, unsigned long flags) { BUILD_BUG_ON(flags > ENCODED_PAGE_BITS); @@ -242,6 +251,17 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) return (struct page *)(~ENCODED_PAGE_BITS & (unsigned long)page); } +static __always_inline struct encoded_page *encode_nr_pages(unsigned long nr) +{ + VM_WARN_ON_ONCE((nr << 2) >> 2 != nr); + return (struct encoded_page *)(nr << 2); +} + +static __always_inline unsigned long encoded_nr_pages(struct encoded_page *page) +{ + return ((unsigned long)page) >> 2; +} + /* * A swap entry has to fit into a "unsigned long", as the entry is hidden * in the "index" field of the swapper address space. diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 6540c99c6758..dba1973dfe25 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -50,12 +50,21 @@ static bool tlb_next_batch(struct mmu_gather *tlb) #ifdef CONFIG_SMP static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_struct *vma) { + struct encoded_page **pages = batch->encoded_pages; + for (int i = 0; i < batch->nr; i++) { - struct encoded_page *enc = batch->encoded_pages[i]; + struct encoded_page *enc = pages[i]; if (encoded_page_flags(enc) & ENCODED_PAGE_BIT_DELAY_RMAP) { struct page *page = encoded_page_ptr(enc); - folio_remove_rmap_pte(page_folio(page), page, vma); + unsigned int nr_pages = 1; + + if (unlikely(encoded_page_flags(enc) & + ENCODED_PAGE_BIT_NR_PAGES)) + nr_pages = encoded_nr_pages(pages[++i]); + + folio_remove_rmap_ptes(page_folio(page), page, nr_pages, + vma); } } } @@ -89,18 +98,26 @@ static void tlb_batch_pages_flush(struct mmu_gather *tlb) for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { struct encoded_page **pages = batch->encoded_pages; - do { + while (batch->nr) { /* * limit free batch count when PAGE_SIZE > 4K */ unsigned int nr = min(512U, batch->nr); + /* + * Make sure we cover page + nr_pages, and don't leave + * nr_pages behind when capping the number of entries. + */ + if (unlikely(encoded_page_flags(pages[nr - 1]) & + ENCODED_PAGE_BIT_NR_PAGES)) + nr++; + free_pages_and_swap_cache(pages, nr); pages += nr; batch->nr -= nr; cond_resched(); - } while (batch->nr); + } } tlb->active = &tlb->local; } @@ -116,8 +133,9 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, - bool delay_rmap, int page_size) +static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap, + int page_size) { int flags = delay_rmap ? ENCODED_PAGE_BIT_DELAY_RMAP : 0; struct mmu_gather_batch *batch; @@ -126,6 +144,8 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, #ifdef CONFIG_MMU_GATHER_PAGE_SIZE VM_WARN_ON(tlb->page_size != page_size); + VM_WARN_ON_ONCE(nr_pages != 1 && page_size != PAGE_SIZE); + VM_WARN_ON_ONCE(page_folio(page) != page_folio(page + nr_pages - 1)); #endif batch = tlb->active; @@ -133,17 +153,40 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = encode_page(page, flags); - if (batch->nr == batch->max) { + if (likely(nr_pages == 1)) { + batch->encoded_pages[batch->nr++] = encode_page(page, flags); + } else { + flags |= ENCODED_PAGE_BIT_NR_PAGES; + batch->encoded_pages[batch->nr++] = encode_page(page, flags); + batch->encoded_pages[batch->nr++] = encode_nr_pages(nr_pages); + } + /* + * Make sure that we can always add another "page" + "nr_pages", + * requiring two entries instead of only a single one. + */ + if (batch->nr >= batch->max - 1) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, page); + VM_BUG_ON_PAGE(batch->nr > batch->max - 1, page); return false; } +bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, + unsigned int nr_pages, bool delay_rmap) +{ + return __tlb_remove_folio_pages_size(tlb, page, nr_pages, delay_rmap, + PAGE_SIZE); +} + +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size) +{ + return __tlb_remove_folio_pages_size(tlb, page, 1, delay_rmap, page_size); +} + #endif /* MMU_GATHER_NO_GATHER */ #ifdef CONFIG_MMU_GATHER_TABLE_FREE diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..2a217520b80b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -967,11 +967,17 @@ void release_pages(release_pages_arg arg, int nr) unsigned int lock_batch; for (i = 0; i < nr; i++) { + unsigned int nr_refs = 1; struct folio *folio; /* Turn any of the argument types into a folio */ folio = page_folio(encoded_page_ptr(encoded[i])); + /* Is our next entry actually "nr_pages" -> "nr_refs" ? */ + if (unlikely(encoded_page_flags(encoded[i]) & + ENCODED_PAGE_BIT_NR_PAGES)) + nr_refs = encoded_nr_pages(encoded[++i]); + /* * Make sure the IRQ-safe lock-holding time does not get * excessive with a continuous string of pages from the @@ -990,14 +996,14 @@ void release_pages(release_pages_arg arg, int nr) unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - if (put_devmap_managed_page(&folio->page)) + if (put_devmap_managed_page_refs(&folio->page, nr_refs)) continue; - if (folio_put_testzero(folio)) + if (folio_ref_sub_and_test(folio, nr_refs)) free_zone_device_page(&folio->page); continue; } - if (!folio_put_testzero(folio)) + if (!folio_ref_sub_and_test(folio, nr_refs)) continue; if (folio_test_large(folio)) { diff --git a/mm/swap_state.c b/mm/swap_state.c index e671266ad772..ae0c0f1f51bd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -311,8 +311,16 @@ void free_page_and_swap_cache(struct page *page) void free_pages_and_swap_cache(struct encoded_page **pages, int nr) { lru_add_drain(); - for (int i = 0; i < nr; i++) - free_swap_cache(encoded_page_ptr(pages[i])); + for (int i = 0; i < nr; i++) { + struct page *page = encoded_page_ptr(pages[i]); + + /* Skip over "nr_pages". Only call it once for the folio. */ + if (unlikely(encoded_page_flags(pages[i]) & + ENCODED_PAGE_BIT_NR_PAGES)) + i++; + + free_swap_cache(page); + } release_pages(pages, nr); } From patchwork Mon Jan 29 14:32:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AADFC47422 for ; Mon, 29 Jan 2024 14:33:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBF446B009B; Mon, 29 Jan 2024 09:33:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6EAE6B009C; Mon, 29 Jan 2024 09:33:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0DC26B009D; Mon, 29 Jan 2024 09:33:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D3DA6B009B for ; Mon, 29 Jan 2024 09:33:15 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6FB1B160619 for ; Mon, 29 Jan 2024 14:33:15 +0000 (UTC) X-FDA: 81732591150.25.EBCBF83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 86028C0022 for ; Mon, 29 Jan 2024 14:33:13 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e5oe6GhB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hD+wJlVoYoeun3CtWSKYM8LYSFldaYp0TLUcFe85aZ0=; b=RgEW/KqzzPXhU+1h00uY5L3XN8ip5/wnxIfbn3/PS9NWEOKVj1MSK0ZW/5LSq4BdMUd39l iaC7QMS/r+oiDfuVZkyJSp6+LfkVGZi8K+PCGjQMOo150rlsRj9v2bNaARbWJulTtBIe8V 41hk9km27f4yJJlVnKwplnSLjgkQ94I= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e5oe6GhB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538793; a=rsa-sha256; cv=none; b=H2vdDKlzRXO48ordDU9+p/slZoPzDlJR+QHmq7jzUk6//hy4PTPUWWA8HCzJSU07OCC+FC tjnzkV6c+Hsyaqiey082cqbDI7ewbqSyfepXwr6PAYn9aT44DacPpG5Lz6qWx/AdU/onUq +LKzjrFIL+QP/HIRX0XPwt4KV01RBDI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hD+wJlVoYoeun3CtWSKYM8LYSFldaYp0TLUcFe85aZ0=; b=e5oe6GhB2IFgCCUYpTH5h6KOce1lhOz+S00vu2pc8f2g2PP9J8oS+9ZetT9cVRtxSQEhSR UVDQFk44wGln3iaGUin/SGjkmPZ7NtQEU8jZ8t45MxfbWOEE1mdzn+Y5yeDGvQAqlDNlsc mw5kE5vyCt8/Gswez8T1Hn1c6octsAg= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-64-0ZkSAIR7N62aRzlXnFbltw-1; Mon, 29 Jan 2024 09:33:09 -0500 X-MC-Unique: 0ZkSAIR7N62aRzlXnFbltw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 88CF43C13A9D; Mon, 29 Jan 2024 14:33:07 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1D40AD1; Mon, 29 Jan 2024 14:33:02 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 8/9] mm/mmu_gather: add tlb_remove_tlb_entries() Date: Mon, 29 Jan 2024 15:32:20 +0100 Message-ID: <20240129143221.263763-9-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: p7gw9yr4ifc6q64zunk4wmqx71r5469m X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 86028C0022 X-HE-Tag: 1706538793-626130 X-HE-Meta: U2FsdGVkX1+OSm3Rm6dDXkFwba4jJP/Dr9oGRUSDBDZD6ND9FQ/Hl9L3Y575Zml7jD0Sd7H+ZglC2JhnY/agXkwdWF/TUybFvAtXPfSG6gc9Tmy7M5TsGBgtw0cXUq3Sex3cKffoq/EM08FHO2xP16pjsw+8DF3kZfwxyZn7A+Q8Beyjq+0pRcnWZvp0pGw2S8FiGOWYNgFdSXGKPKW9p/CDQGAT4jBWHVzY7QH2gVAweM8MXPBbTbBXJOBhVob8YawXEC4U4DAkwWW016k+tqj/IccQ5hvLBtDuH1Hgi88HCR9R6ea9yJJjFSNDnEYiPe+DU4KLkdNPIIpEDts/owSA6kxWjFdMrWBTOu86xyLxhtODhYp1UWpaW2vmTYugH/AzBBirPg1mt5pWow/fXUsJnvYKSzMUhZU/wKzJsI3VY1iKwwnflPJGaJ1Q5WAWH8viUXQDm280GhJD2AneGwlNskwbSaMHt6bBTr88Ff4S80UZWO+pG8h+moA8HY+pB8ZwNExthzHDo3N/dIzWhFwnU9kAuvcsPyDhdFQ2M+fUcwy0p/LQEVToVZibTrrZ8oCa53Qo6ff562HBIiuoiHWCJomu13NKZRejTN96L5ArJTX7LroAcoJjMCqA4M+J9J5JBhyrJI0PhQv3CoTBwjvEPHYRsa1VI3Aup0VBa+iLhiDGv4YDjNsf7US4oXVPWh0LCNObZObunh9f3HRiBYJ89QvycU1Mun9Z056QBOEIiHOk1cW9Tg1mxcoaeANFB6sgeiKc8WQD5Na/ZKs5Zqt+qL1elUYp9HdGJrndQkgDRulKNSV0Zi/Mdu4N/tdr1PfBkQduu/EkkaYP7IE+cLDzalsXOuwrHhXsBuz1E0EEZUF6/I1aze2dmrs9SZMxXOtBIxqsiKEnUHkQtl14oA1gvPrO5iesqL1gtWEs1a17zGA4PICI81A7bnDY5qXwu9GzSfyZHLHRJ9vb+b9 /NFmTAQz nMGn5nOu6E+H2/UYUxxZObVsRE9ze8Y5gy1+4VWLxbeLJkITm3YBOYOp0hqYtFXPufhQUuJivnz77hjr1p2tfuneYXN3X2SZHanfQvsCyvPUP2B0dCXBjxfcdcIfcRQ6+GBiNxVg3lzs29tMh3Nnt1SAHRslzAxWPbb8UrMvm7kp4R7s5taWQyCCTkhWXDa2bBfYYH9sEdi3KQimMG6jHJPuTH6H329KRDE0o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's add a helper that lets us batch-process multiple consecutive PTEs. Note that the loop will get optimized out on all architectures except on powerpc. We have to add an early define of __tlb_remove_tlb_entry() on ppc to make the compiler happy (and avoid making tlb_remove_tlb_entries() a macro). Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- arch/powerpc/include/asm/tlb.h | 2 ++ include/asm-generic/tlb.h | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index b3de6102a907..1ca7d4c4b90d 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -19,6 +19,8 @@ #include +static inline void __tlb_remove_tlb_entry(struct mmu_gather *tlb, pte_t *ptep, + unsigned long address); #define __tlb_remove_tlb_entry __tlb_remove_tlb_entry #define tlb_flush tlb_flush diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 428c3f93addc..bd00dd238b79 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -616,6 +616,26 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) +/** + * tlb_remove_tlb_entries - remember unmapping of multiple consecutive ptes for + * later tlb invalidation. + * + * Similar to tlb_remove_tlb_entry(), but remember unmapping of multiple + * consecutive ptes instead of only a single one. + */ +static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, + pte_t *ptep, unsigned int nr, unsigned long address) +{ + tlb_flush_pte_range(tlb, address, PAGE_SIZE * nr); + for (;;) { + __tlb_remove_tlb_entry(tlb, ptep, address); + if (--nr == 0) + break; + ptep++; + address += PAGE_SIZE; + } +} + #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ unsigned long _sz = huge_page_size(h); \ From patchwork Mon Jan 29 14:32:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07760C47DDF for ; Mon, 29 Jan 2024 14:33:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8791A6B009E; Mon, 29 Jan 2024 09:33:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 829AC6B009F; Mon, 29 Jan 2024 09:33:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67C056B00A0; Mon, 29 Jan 2024 09:33:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 541D76B009E for ; Mon, 29 Jan 2024 09:33:21 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 373CFC0358 for ; Mon, 29 Jan 2024 14:33:21 +0000 (UTC) X-FDA: 81732591402.09.D1270D7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 62BB614002A for ; Mon, 29 Jan 2024 14:33:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jde4vgNt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706538799; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zfDWyrOXB1mD+zgjXEawbz7aquFQMadFUbUPJ1Y1hbE=; b=Rs/sm6zFgSXJ4LjpgW6uj0MpB4dwjYvsDlMdpdOVakwESySdAfV/tuZhoYBPQ373sMW1Ax T4+zQ8lA3W9YEbwRhGc3ktd9HE5torykoctZ0QnY5XFO1vvdjPqgCr+PUbyRKV1j6Cbv/e 9EDkxB5VPUKe7zxqUPnsBchh8H+osJk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jde4vgNt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706538799; a=rsa-sha256; cv=none; b=y2j3L2PyI/SevmEdNyrgTi6Ft5rxeffngDfvIyC74EO11FkvcKUYH5N8kYdusv9ywsBMPL eCXVnytw3Ka3Sl0JjyrkdYD1wczz4qEXPlwTa/BgadG2yLBsM37N3GZTFOyInKqLQ++ZIw HhgiQ5KRDvm4il36ceXmG3jmwlHTDyY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706538798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zfDWyrOXB1mD+zgjXEawbz7aquFQMadFUbUPJ1Y1hbE=; b=Jde4vgNt9FLcTh04SK6lPnB/a1lN3TqH3VvDst+prnSTm+Lx2wZCYBUet5NH9jfcpmNsGV 4dz024++StKb1BRiSv2vvazWlWa5NKfpnMqeo3WfmcHO9HNsXNYHkqeC+9ZGuh2rB8sBdu libY0GiUIVG8xxVSS+nCk0ZzuwHKq6c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-434-G5xnxcuDPh6_dHlX6DUXrg-1; Mon, 29 Jan 2024 09:33:13 -0500 X-MC-Unique: G5xnxcuDPh6_dHlX6DUXrg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D140685A597; Mon, 29 Jan 2024 14:33:11 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id E6EB5AD1; Mon, 29 Jan 2024 14:33:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v1 9/9] mm/memory: optimize unmap/zap with PTE-mapped THP Date: Mon, 29 Jan 2024 15:32:21 +0100 Message-ID: <20240129143221.263763-10-david@redhat.com> In-Reply-To: <20240129143221.263763-1-david@redhat.com> References: <20240129143221.263763-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 62BB614002A X-Stat-Signature: uu6ck3w5ao7ntu3wumgxsgfwxgoaby5z X-HE-Tag: 1706538799-352849 X-HE-Meta: U2FsdGVkX1+YU5HqP6J67Q/YK3eAo6mk4gUvSMBBBJAZx0gHGpAzRBOUYLbuWU7ok1rQ6Oo5jTko6b7AviCPpqbzzgJMbjdQ8KI86fJSZKPMuVWQLG1OaDvCiL56uwErEwDhGDEU83NVUlNKqPVoxL5msdAjorYULH9oIi1yX77WdbdNDVEK6BfIaXQAJFC7kBDGeUbEyo35mLXAsLqhUznIJtjpIozo92UlXK1Gt0rX9IMqL/c7GCfblFIOokS+CGA+5iH4KCPlyJKRR6AqV+j2e5wIyxXVBWJtJb9aKgNGZdMWO3I8pJP2y+xXQ6UdsHLc8awRH0BoA26usBMJ+g6rbv7AVaoAmciIASJhCNgPJyEuuva9Zx2/UCgIRjGveXp6vBmucumabbLN2MojHjaR70o4zalqAn4se/qR+xDtxYIf7kAj7aPiT0Duh93BPbVI3qPM3PMRvQAHgW7W32b+w6Jwtaa40z6Aj4q+bZfK+/agGNBeS+RNxpNfH6yOBEbbbWrUKb2Y1E55//H+3aHS5orNMM05VYN18t8Cjdrfn/DYqVm0NNqL5B2wdpB9883L1TkzHG36TrQDFZdLZ44ngvxMg7oRIhDLQEtmhvcx0LAHjtx2DpHoXZAXrrHyBuLEBvGoeOyMyupdwpnbOX9BWgUyJvrIFNi5UtZM91SlSIVM057Dc/sArnCWaUufO4+BN95ueB1/HlYiYdyd5je0uxXF9kIDXmns/jOK/XvLgg4n4pj+46R5Ta4Kdxx1k1/xWmYfXCsIIOWfBwwBX/y4FLCw/NMoKKMBXPJRocyZ1tMpx5wNyi5y+EM/KWeyPZg34vR9EMQ4H7ojbno/74Y2Hir7inw7cSU4QFJSuts4fPIISo/RhG3iDxMttyHz2wvgx+0AKP6z1NdU76obbkPnlgwg2zinATRqu+y7NAcrqKnUwjJ0vlEci6Vry0YvLjhSx80IX8FiD/4PGrs KF9CA8m1 HEUrRMEbnOozQOwJY5uBCChOWZxWBjK8eXJsfnAiBTKOiKownLBa8W0fChTOtwRI3dp/DiVf1NWcBgZUtad5cOnO19KSUA10ZF26gZFuh/QeDXgsz32Ebge2Zg7W6bJWbFQtu88yTmRCh9KTp2/FZYSvYiuBU7lmmZBP1mbYwzE16EEE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to how we optimized fork(), let's implement PTE batching when consecutive (present) PTEs map consecutive pages of the same large folio. Most infrastructure we need for batching (mmu gather, rmap) is already there. We only have to add get_and_clear_full_ptes() and clear_full_ptes(). Similarly, extend zap_install_uffd_wp_if_needed() to process a PTE range. We won't bother sanity-checking the mapcount of all subpages, but only check the mapcount of the first subpage we process. To keep small folios as fast as possible force inlining of a specialized variant using __always_inline with nr=1. Signed-off-by: David Hildenbrand --- include/linux/pgtable.h | 66 +++++++++++++++++++++++++++++ mm/memory.c | 92 +++++++++++++++++++++++++++++------------ 2 files changed, 132 insertions(+), 26 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index aab227e12493..f0feae7f89fb 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -580,6 +580,72 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, } #endif +#ifndef get_and_clear_full_ptes +/** + * get_and_clear_full_ptes - Clear PTEs that map consecutive pages of the same + * folio, collecting dirty/accessed bits. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to clear. + * @full: Whether we are clearing a full mm. + * + * May be overridden by the architecture; otherwise, implemented as a simple + * loop over ptep_get_and_clear_full(), merging dirty/accessed bits into + * returned PTE. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = ptep_get_and_clear_full(mm, addr, ptep, full); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_get_and_clear_full(mm, addr, ptep, full); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +#ifndef clear_full_ptes +/** + * clear_full_ptes - Clear PTEs that map consecutive pages of the same folio. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to clear. + * @full: Whether we are clearing a full mm. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + ptep_get_and_clear_full(mm, addr, ptep, full); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} +#endif /* * If two threads concurrently fault at the same page, the thread that diff --git a/mm/memory.c b/mm/memory.c index a2190d7cfa74..38a010c4d04d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1515,7 +1515,7 @@ static inline bool zap_drop_file_uffd_wp(struct zap_details *details) */ static inline void zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, - unsigned long addr, pte_t *pte, + unsigned long addr, pte_t *pte, int nr, struct zap_details *details, pte_t pteval) { /* Zap on anonymous always means dropping everything */ @@ -1525,20 +1525,27 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, if (zap_drop_file_uffd_wp(details)) return; - pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + for (;;) { + /* the PFN in the PTE is irrelevant. */ + pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + if (--nr == 0) + break; + pte++; + addr += PAGE_SIZE; + } } -static inline void zap_present_folio_pte(struct mmu_gather *tlb, +static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, struct vm_area_struct *vma, struct folio *folio, - struct page *page, pte_t *pte, pte_t ptent, unsigned long addr, - struct zap_details *details, int *rss, bool *force_flush, - bool *force_break) + struct page *page, pte_t *pte, pte_t ptent, unsigned int nr, + unsigned long addr, struct zap_details *details, int *rss, + bool *force_flush, bool *force_break) { struct mm_struct *mm = tlb->mm; bool delay_rmap = false; if (!folio_test_anon(folio)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + ptent = get_and_clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); if (pte_dirty(ptent)) { folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { @@ -1548,36 +1555,49 @@ static inline void zap_present_folio_pte(struct mmu_gather *tlb, } if (pte_young(ptent) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); - rss[mm_counter(folio)]--; + rss[mm_counter(folio)] -= nr; } else { /* We don't need up-to-date accessed/dirty bits. */ - ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - rss[MM_ANONPAGES]--; + clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); + rss[MM_ANONPAGES] -= nr; } + /* Checking a single PTE in a batch is sufficient. */ arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + tlb_remove_tlb_entries(tlb, pte, nr, addr); if (unlikely(userfaultfd_pte_wp(vma, ptent))) - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, + ptent); if (!delay_rmap) { - folio_remove_rmap_pte(folio, page, vma); + folio_remove_rmap_ptes(folio, page, nr, vma); + + /* Only sanity-check the first page in a batch. */ if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } - if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { + if (unlikely(__tlb_remove_folio_pages(tlb, page, nr, delay_rmap))) { *force_flush = true; *force_break = true; } } -static inline void zap_present_pte(struct mmu_gather *tlb, +/* + * Zap or skip one present PTE, trying to batch-process subsequent PTEs that map + * consecutive pages of the same folio. + * + * Returns the number of processed (skipped or zapped) PTEs (at least 1). + */ +static inline int zap_present_ptes(struct mmu_gather *tlb, struct vm_area_struct *vma, pte_t *pte, pte_t ptent, - unsigned long addr, struct zap_details *details, - int *rss, bool *force_flush, bool *force_break) + unsigned int max_nr, unsigned long addr, + struct zap_details *details, int *rss, bool *force_flush, + bool *force_break) { + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; struct mm_struct *mm = tlb->mm; struct folio *folio; struct page *page; + int nr; page = vm_normal_page(vma, addr, ptent); if (!page) { @@ -1587,14 +1607,29 @@ static inline void zap_present_pte(struct mmu_gather *tlb, tlb_remove_tlb_entry(tlb, pte, addr); VM_WARN_ON_ONCE(userfaultfd_wp(vma)); ksm_might_unmap_zero_page(mm, ptent); - return; + return 1; } folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) - return; - zap_present_folio_pte(tlb, vma, folio, page, pte, ptent, addr, details, - rss, force_flush, force_break); + return 1; + + /* + * Make sure that the common "small folio" case is as fast as possible + * by keeping the batching logic separate. + */ + if (unlikely(folio_test_large(folio) && max_nr != 1)) { + nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, + NULL); + + zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, + addr, details, rss, force_flush, + force_break); + return nr; + } + zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, 1, addr, + details, rss, force_flush, force_break); + return 1; } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1609,6 +1644,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t *start_pte; pte_t *pte; swp_entry_t entry; + int nr; tlb_change_page_size(tlb, PAGE_SIZE); init_rss_vec(rss); @@ -1622,7 +1658,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t ptent = ptep_get(pte); struct folio *folio = NULL; struct page *page; + int max_nr; + nr = 1; if (pte_none(ptent)) continue; @@ -1630,10 +1668,12 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - zap_present_pte(tlb, vma, pte, ptent, addr, details, - rss, &force_flush, &force_break); + max_nr = (end - addr) / PAGE_SIZE; + nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr, + addr, details, rss, &force_flush, + &force_break); if (unlikely(force_break)) { - addr += PAGE_SIZE; + addr += nr * PAGE_SIZE; break; } continue; @@ -1687,8 +1727,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(1); } pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); - } while (pte++, addr += PAGE_SIZE, addr != end); + zap_install_uffd_wp_if_needed(vma, addr, pte, 1, details, ptent); + } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss); arch_leave_lazy_mmu_mode();