From patchwork Mon Oct 30 13:30:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13440585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60496C4332F for ; Mon, 30 Oct 2023 13:31:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF3BB6B01F5; Mon, 30 Oct 2023 09:31:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7C186B01F6; Mon, 30 Oct 2023 09:31:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1DAE6B01F7; Mon, 30 Oct 2023 09:31:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AEB166B01F5 for ; Mon, 30 Oct 2023 09:31:06 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 160F1B5A9E for ; Mon, 30 Oct 2023 13:31:06 +0000 (UTC) X-FDA: 81402213732.29.0FD2BBD Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf11.hostedemail.com (Postfix) with ESMTP id D90744002D for ; Mon, 30 Oct 2023 13:31:03 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=Q03h2gYW; spf=pass (imf11.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698672664; a=rsa-sha256; cv=none; b=5cYNHubKzgHdX8GnAfhD3zAQFZs83nh6ZoHw2FV8sXojUbON6pR/CH8Uy4VAWNPpPTFZkR 5iTL6Ra9nDrWYHddevLCRlvHyjrq39hCN2RhH7MrnbvfiHWoFdWle4bTMdAr5KuQSbWfsb bKZVPraH6Te2c/gGgt6UFLu8x95qt1E= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=Q03h2gYW; spf=pass (imf11.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698672664; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=cFAyygTf1wEzmKUuixPjgxrlB1EH/ZEG0d5M9D9G9l4=; b=IdOoAvN7pSYcj7//pdarKIaaycGp5GCgPCtrPGWaUmsk4GbhvzXORDB/f6iMJz46TyDYVx wEu3UKc+vJf1GPHrF1GEN0pQEYfzB5ru9vsWA8AZebhvLo8d9WI/o1qRLM2DJYAQp8Kd/2 Nl8Z16YdrQf1sxkFDQUllelcowLeK3Y= Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2c5629fdbf8so61865331fa.0 for ; Mon, 30 Oct 2023 06:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1698672662; x=1699277462; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=cFAyygTf1wEzmKUuixPjgxrlB1EH/ZEG0d5M9D9G9l4=; b=Q03h2gYWriLwv2/Ul0E02yJ05dILRPSfRD0B0oKJWt1/avLFvoiH0OhlsrTbPcBPqc mdcbNiLQJkX4mJu6STeo5qPsQBKK2l3o9yF4vcZ3N50JdVpVfFbKqMuhbHjZhowO7Xol G6b1UIX6mYB5y04rmmB102CfrhCEylcWspuCL9oQZbI8LkwmPPahSiI2qLMncB2x2wqM /KvIsciv1O5sapVk2cXxxQ9c2FRH1aDTGafaCQL7Pq4blKQNjJ3a/HhN/GuxD6duOTjX 0OaDFKrcl9RBuxIZfr2/Ya6AUAbq3XZAMWSh9VaYGk71428y6ItMuDYN/kcGWxCWfJuC F80w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698672662; x=1699277462; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cFAyygTf1wEzmKUuixPjgxrlB1EH/ZEG0d5M9D9G9l4=; b=EqMhMfLJVDte1f60UCWdYxKhq2n0us8uZvkc9Sxb0SfgPOoCqZYaJmJKmgMQjRvax6 ki1Qcmc6xICU3O46V58WgurnRTEskulC1QYoarTtcFFQpGoW+anyasRUBugFJfm35GZW eiC/bVRVc2RFF3nQFny/IpAaz8Y2yVdhq37CiHYCwl0dBZywFZ2ITLJvj49AP11KzWll pZXfAJzWWLJgNyXZ37ZczioFib9ZWxx0u+QOAb9EMCOsI58Dqp1bQvuPft0MOR2ZvXNy Bv2AmJZyT7PP/ONF+41RrkkkGwDhw/uvFqNy4fNutt3I+2ir4rXd3TTE6kZiD/pThbG8 GwxQ== X-Gm-Message-State: AOJu0YwzLbxS5+c9SbCYz88fiLqGyBAiV1SWILes9kBRI13jWnyRai1F m5uzrilDDm5ujn3wa/tL6OYOMw== X-Google-Smtp-Source: AGHT+IGp3Flp6e8u4wQXKFFe94N4J3LW0CdDXYSPeu5D/05PAkg9EMs3NFgFQ+aCgOk5cp4xvQIWmA== X-Received: by 2002:a2e:a48d:0:b0:2c5:2357:c659 with SMTP id h13-20020a2ea48d000000b002c52357c659mr7665145lji.38.1698672661824; Mon, 30 Oct 2023 06:31:01 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id iv8-20020a05600c548800b003fefaf299b6sm9383887wmb.38.2023.10.30.06.31.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 06:31:01 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Samuel Holland , Lad Prabhakar Cc: Alexandre Ghiti Subject: [PATCH v6 0/4] riscv: tlb flush improvements Date: Mon, 30 Oct 2023 14:30:24 +0100 Message-Id: <20231030133027.19542-1-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D90744002D X-Stat-Signature: 5u1o8iueprn1py4ynb4j6h1sswdxxj7g X-Rspam-User: X-HE-Tag: 1698672663-969317 X-HE-Meta: U2FsdGVkX18cwnkN8cOJrWUy0rwUsSl+0COYrr9IBCMcTi48lJWW1oSwagNaN3gtMC+BTpfPrSSKf204FjQo/j5TxPpfxt5JhnFPGFtKGXwBlTWUxvk5O17f2UYf9Y+IQuQjB9Kt/zTJuWTanaO0bRcar5FNzwO/6unJTjkNAd9HbwbRLC8SbJ/iPQmYES/HbQiBF+OeNOmrSvwiAISdIMMLffnyWgzRb5ZDV726sWp6sHCmTeeJ4vPw87u/tWb56NPGCPuL8CKFsrtcfIEBvCU1blFq9ErWsgKc4FBxKA9S05w++QGnhEAsVNjkRFIY0xSoHsbURJF1/c0a4G7A+zu9UVmjpPdWBtKrRo0uVhKHqHK+g4OEKaL0VG1QZQ/xz4ISxJwKxhzmMzYqPCLBx4tkg67MrXUr4H7s09DcJ2/CVTtc5Tw5b0NIGr+YlegpMkzOGSZCzWuWhjbPlPYR3gXoK5Rj6W0DxWXOjym77GOQhsEHY9B+BhYEpL5pYu56MM0D87BTQTL1QeJ3DUCVkWUDD+zwatGf/S/tem2eFEkGCQnFGNkCsDQG7rM2/E3NZnhoNs+vuO2dMLofIppCRWJ+pVcQMsRQLsHb2SW3sivC4OUPh0/GtNuGx+RzW+sWrlu3oAT/7D1O6ZsYNRXFsA4rxLYwszeaCsTg5hC25qAC5uiOuT0MxlRcGQChOu5WintQGim//Wlp0Fzl8A8klUijTbLXjlmtwqTyyTFAks4RX13o5pE3urV2O8Juo1tRZjVSWOebOXbhmphzT1egDo+qzhBtLj0eQFBK/9nJ/LBXLgVU+LoiiWd3kP7SZqOcKQkh5RaizfdmR+YSNcL4YaqdEn9/UcMwSXR5nxPycO2pi4ugOYCvrHs11Syui7hybmRJ16m0GXopHwqx6IO2i0LVW3NZ96pLK4Z67dSq2fHDg+WGA6CkbvfvOJBTs7JL95Y8J0aInK9eMob+d85 JS7VKkAG IgN/rODdvfavZuUm03w9XPjdvc6irc7Qr5e1BsRi3JHYxXRltOXFu5LWJYS0B51Ufpp6YrJ02h52dEXIH0u2ZTvk+fIuJY1e4c7aTRmDlhE543dnsl+eavV6XNOkiVhZKfY5PZ1PN5GGQHLan9oYOyINbh1ArAgwBp1iHj9BHlAPnmKz8dtJ7EyNDiMKBJySppZRneRf7RBCPRGhvp3AFbMx/ot+QJ322HX8szOukwFXxZtzHRSqVa8m77f8q/BgZNdkr2RlrYloc4+O3ZG2/kLKZ0hXQLJ5KG3a2x4Cu+AlI6bTXV63SCh7EFuwmtl58XAHhwxh0Sh125aRMj9oszBFnt8JK1TH1oShqntgZe9yAix5sP85k07UUMTmMdJs9a+lzr+kYIGZzKJAwyU6nrXZtzsQO+pPZZPkQVObReUB+ghUy+b67iNgqllJgYgg37y3+g9oOPqR/K0pT0AdVCwk5m4YRm4j9pjaC4+Osu7hvE6SLw5vSROzsztMkF6qtv2Hkh0jjYev1RShhZnzzogoYlX2jcbo8nhv9NNLNSSU7kzUj+mtXbPagMuFVV6m2nd8uma7KEV7QxqI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This series optimizes the tlb flushes on riscv which used to simply flush the whole tlb whatever the size of the range to flush or the size of the stride. Patch 3 introduces a threshold that is microarchitecture specific and will very likely be modified by vendors, not sure though which mechanism we'll use to do that (dt? alternatives? vendor initialization code?). Next steps would be to implement: - svinval extension as Mayuresh did here [1] - BATCHED_UNMAP_TLB_FLUSH (I'll wait for arm64 patchset to land) - MMU_GATHER_RCU_TABLE_FREE - MMU_GATHER_MERGE_VMAS Any other idea welcome. [1] https://lore.kernel.org/linux-riscv/20230623123849.1425805-1-mchitale@ventanamicro.com/ Changes in v6: - Remove ifdef SVNAPOT, as suggested by Samuel - Fix usage of u16 which could overflow, as noted by Samuel - Use cpu_online_mask, as suggested by Samuel - Move static_branch_unlikely(&use_asid_allocator) test, as suggested by Samuel - Add TB/RB from Prabhakar and Samuel, thanks guys! Changes in v5: - Fix commit message s/flush_tlb/tlb_flush thanks to Samuel - Simplify NAPOT mapping stride size handling, as suggested by Samuel - Add TB from Prabhakar - Add RB from Samuel - Remove TB/RB from patch 2 as it changed enough Changes in v4: - Correctly handle the stride size for a NAPOT hugepage, thanks to Aaron Durbin! - Fix flush_tlb_kernel_range() which passed a wrong argument to __flush_tlb_range() - Factorize code to handle asid/no asid flushes - Fix kernel flush bug where I used to pass 0 instead of x0, big thanks to Samuel for finding that! Changes in v3: - Add RB from Andrew, thanks! - Unwrap a few lines, as suggested by Andrew - Introduce defines for -1 constants used in tlbflush.c, as suggested by Andrew and Conor - Use huge_page_size() directly instead of using the shift, as suggested by Andrew - Remove misleading comments as suggested by Conor Changes in v2: - Make static tlb_flush_all_threshold, we'll figure out later how to override this value on a vendor basis, as suggested by Conor and Palmer - Fix nommu build, as reported by Conor Alexandre Ghiti (4): riscv: Improve tlb_flush() riscv: Improve flush_tlb_range() for hugetlb pages riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb riscv: Improve flush_tlb_kernel_range() arch/riscv/include/asm/sbi.h | 3 - arch/riscv/include/asm/tlb.h | 8 +- arch/riscv/include/asm/tlbflush.h | 15 ++- arch/riscv/kernel/sbi.c | 32 ++---- arch/riscv/mm/tlbflush.c | 181 +++++++++++++++++++----------- 5 files changed, 144 insertions(+), 95 deletions(-)