From patchwork Sat Oct 28 23:12:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A61D6C4167D for ; Sat, 28 Oct 2023 23:13:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F25CF6B02CA; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAD616B02CD; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D29E96B02CF; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B38DC6B02CA for ; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 941811CB445 for ; Sat, 28 Oct 2023 23:13:47 +0000 (UTC) X-FDA: 81396424494.03.F679421 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf17.hostedemail.com (Postfix) with ESMTP id C73CB40004 for ; Sat, 28 Oct 2023 23:13:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=jmZ0tQWK; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=kguaAV6mm40UKTtYsL9Xv6tvyel4SsmmI5uYo5XPeQ++reaOYBXU6Gf8ADuww5gqeUMQeo RHfpLwsiqGZQn5SKSCcakA1uVXW160sxc2C6IUGlWwNSM2Tsfi5+s8kpwJ3GDclX+brn17 0+fgmrW5xNeYii7omf/FgOHI6nk1QQk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=jmZ0tQWK; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534825; a=rsa-sha256; cv=none; b=HTpKqRUwRmh1WZ4m/465ETms21OKRTKkEjh7h9hGMOuWasuL6F4r9meZa1elRd72Z4SYDj weY0/12vxo6J/KWFMzOsyJok9CTT2fTs0+lMoIC+H6K1rdEASPN0ZG+u91oIT1LXEtRvs5 RRVG4/celI6i2KMt97fsvDcoOhl2XKk= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-692c02adeefso2868166b3a.3 for ; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534824; x=1699139624; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=jmZ0tQWKIdIkV6pbj6H+mmbvBqe+7No+YEdehSaS2aCUV1aHU9mN+B2lMgyBnDchz2 l5Sw8GveNB4PPUecHVAjo/7AO+qtsOrG5gaP9j+itcgBm0glpSSvkX+LtnJkiqQMdlQ4 BcvMKTYZxjDRQNHQ8Jx6mSfkEd9Czmik8A/QZ4CKBfytH2S1bLl+EN82BcEuuE108kb2 wGtndh3cSSANbnsyRSSK46kkI1qquFxiXSHaT0rkrpeaCbqPLrjDjYw2o6MF72Uxw7V5 jOjaX4HGIMU/8DM1AO3fcdRbQUUFEWesEUvta//K2qAA2XsJHRk7/sc32+PtqYzf1y2y hLYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534824; x=1699139624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=E+s4QGdkqbRcsWFH0dLPuKslJxgidJcrOwOMwrYOj8CRRPz9+3yPRX9sGEg9McV0fl dev73iZuVTUArzcV8gfEq/BAxuCUxin/G+mw1Mm+GSG0NX9e1mOmNQoC69V3xPlGFR0n QsEnZFa/wmitBy/bjHRf4aFhluznEKryZna8DpEftYnrxBpU9JLDnjWnwWsb3xA7UdQr nmAIQIeQ5vzJKmWDamk5jNgFpKqaqr1p8NvUm+Hi6joLop6AjdzQTpjhjvWu638kOJIN E1hg1kGZ+FUDY0xsd6h5vTQkFBraKMF3U0AjXcop3zhP0b9srZl/h+gkNaQ/LLuhATuZ u1tg== X-Gm-Message-State: AOJu0Yxp1i11T0NJqynWX4gNfhrC/U+WwWYPicrLPSPZbP2mHogkyxid TcXqyW+FlMJZOjpvDgfhdPLPxw== X-Google-Smtp-Source: AGHT+IHFtwqkvMLL8vTXw7lsMTaEWiNW/nYYuBPFSSXqkCgtp1/aoWCoS2LcxRJlFYSU2eWn0N2URQ== X-Received: by 2002:a05:6a21:3b45:b0:16b:a5fb:eee5 with SMTP id zy5-20020a056a213b4500b0016ba5fbeee5mr5984191pzb.28.1698534824609; Sat, 28 Oct 2023 16:13:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 02/11] riscv: Improve flush_tlb_range() for hugetlb pages Date: Sat, 28 Oct 2023 16:12:00 -0700 Message-ID: <20231028231339.3116618-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: bpp59cqwnjcr69ji94bokyyydgcs9use X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C73CB40004 X-HE-Tag: 1698534825-842275 X-HE-Meta: U2FsdGVkX18+QtnlX2o8oa6OT844G0Clulk0rYzMy4rwrqgEjlJ7SYXLsQwukrNoyT/aefA+S4xwXT0kJRJjODWs+DaMKgrZL8M/jgf5edAg8CKVVDk9FL83BcLQr1VypLMsnQk7wFeNLvGveWLEVT+ZB2bqea4TrGgSH97RamTHCCmmmj7fUuPHiYGfrsD+iEdgyLcYbBrVktcvkDgCvv9ysgNHwoXhPT728Nk/Cd29vtovPPtaF7rdHTVKSDnHeQlClGWQxeRXllWpPd8P5bA9y96NNVWAx6z3feK2Qdb/WtxYwH33BdawlquHiUWZUWR6jOtY+XXdEkzSw2gtb+SmUqCkHZrhzJSh8MUj9E28omHsWAibuva6gx0B2SFG7ngC2s3JIQQ57o+U+yJOM3KgPNWPoSCOCr6prj/YlvH+G7NPbxOJhRv6nRwIgOSiGjdHVfnDdyZRTlO6v2QyIyua8Bz+Rth2MSRy+tKhXE3zXzpEulxamKdqzJV6TtZXAZ4e+B/RNBggWM7AR5adzJCpDJKC3XKmslviPSFvAv2Qo8Z8knYYAJgs7QWJaI4V/2c4IcYI3ESYs+s8sgd3EMl3Q0t2/6UX7H62osDFttQRJey1FY3lAexrQa9DbaKCTSqQ8XxTnVG5SxLZq+ke+Gito1tSeFzyv+jSEJnwVEL7XMKFjNm4rrWN0EV2TahJGJo7hgBfjEgO+2jE7r5qEA5RROkQTkhCAZtumHvp6cGNGfiOtY1b0Vpp7f0cir3jsYOaVWLsgLN09yz8nDXNzfk23a0eXysYAkSLAwH1M04BXFVHKEYTS7kaNAKNuZrcgXzQ3db3fbrIaAVOYXNOsmPk6Qjy7vPDbB1nWaia+ey1G0IuJ/WhGBxaAKuYP9l5ONEOHTbbYRxtBfWPp2dlishl80OfmjoPVM0Mvrrmu6G2nI6jv832F4SczUDSYrMwSGTS7xhxp9SQLP6jbgw Lgpi4jL5 UKKlvioGdOdHjHYhhSvLgCXPbWb/WRYez0utlQnaoK9vswkKBEmoV+rABZSlCmNss8fijOdxERbYptkIjgZS1q9KuJ1g9NdYMYNAxrwYR/NvUvFJigJO5EoyewbT5mTFFOoPMbU4zbr5YXvhZV0vSKYWoAs15SsGZuqmM0iXi66j6yvik8TPL8ao7JEBuKkf6EcHY0P+sW2JK2aUz9/RBCUrpcif+aJy9b4XQPm092l75bwVA0LsFAo/7vBO67Gb67Mx7K7yiuIilWcGjRk1hkSBstWgIF57qhuDps8sQNJnXaVixQ4slhQpnGWOYakpsR38lsHPP5YQSrBI2c/5fEKLj+ojeuHQKE39/6zsnXXj8JI3KvC5xT6i9DyhUyfCj6T7pY4PB+Zzsj36y3OkYEigwFVo2sEYa+dYDyB66hH2elpRi9OvBVg2jc+7UJzYFIkIPJ2UWSGYmt4QzDB9tfb9yKpqCRhKJQS9NVCIfMAK6f1VD17Prn9dPdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexandre Ghiti flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. However, if the hugepage is a NAPOT region, all PTEs that constitute this mapping must be invalidated, so the stride size must actually be the size of the PTE. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti [Samuel: Removed CONFIG_RISCV_ISA_SVNAPOT check] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/mm/tlbflush.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..b6d712a82306 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -147,7 +148,33 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_size; + + if (!is_vm_hugetlb_page(vma)) { + stride_size = PAGE_SIZE; + } else { + stride_size = huge_page_size(hstate_vma(vma)); + + /* + * As stated in the privileged specification, every PTE in a + * NAPOT region must be invalidated, so reset the stride in that + * case. + */ + if (has_svnapot()) { + if (stride_size >= PGDIR_SIZE) + stride_size = PGDIR_SIZE; + else if (stride_size >= P4D_SIZE) + stride_size = P4D_SIZE; + else if (stride_size >= PUD_SIZE) + stride_size = PUD_SIZE; + else if (stride_size >= PMD_SIZE) + stride_size = PMD_SIZE; + else + stride_size = PAGE_SIZE; + } + } + + __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,