From patchwork Mon Oct 2 15:10:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13406331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C307E784BC for ; Mon, 2 Oct 2023 15:11:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0F1F8D0029; Mon, 2 Oct 2023 11:11:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBEAA8D000E; Mon, 2 Oct 2023 11:11:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5FDE8D0029; Mon, 2 Oct 2023 11:11:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B2FB98D000E for ; Mon, 2 Oct 2023 11:11:40 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 97113160322 for ; Mon, 2 Oct 2023 15:11:40 +0000 (UTC) X-FDA: 81300860760.17.303995F Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf01.hostedemail.com (Postfix) with ESMTP id BF6FA40014 for ; Mon, 2 Oct 2023 15:11:38 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=GnYs8dyE; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696259498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CH5x2RpAAFVf+XEG+0f6TamSE9fPT169LwjSjxAPY4U=; b=IPVquCvr2MjgAtVrcrZrDDhuqU7QeNmsFkDqlNKo/isfd/eOpJgrqHX94VQYVIwq8Pi+2D 5NJk1OC+B062IcrAbzd9nG47LVqGmLNrUXwDH0sidPlcsTEWVrt9uvYAoP0jOpOB7MEOfV b2HBvglvYWTj/QRvC9RzBQ6XWTSHh/0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696259498; a=rsa-sha256; cv=none; b=KCkvILeI22KAd77f6c9daxhVa6CJNsJ0q4mHsncxnc3cTiU408PmnPe3s+D6lQxrqF20C+ T2yMyDHepqa3iVYaMApzM0r4fBVc3tO9SxhVVqp2CCEDabNR2xQvwYmauv0VIBiUnNVWRS AipePqo6Om20dq2sqjj700dI5Sv8+fg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=GnYs8dyE; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4060b623e64so19752985e9.0 for ; Mon, 02 Oct 2023 08:11:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1696259497; x=1696864297; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CH5x2RpAAFVf+XEG+0f6TamSE9fPT169LwjSjxAPY4U=; b=GnYs8dyECQNxaLgFbts5gpWnEuinONDK/6qoAXiADg4EgnCtyjmRjG1k87fRpQVmyu 38QKl5Az65ArDw7sTJ3StU+xjJ3OfjWYtFAhqj9MfAavDNGkff7qW+1TzNrYpLaziBGA lOj4QygrJaMhmnD26CvNr3sDX8s8JcUYD6ASwquiP+bynxmRDrvmeGQKf3VeSc/vzJ4p j2RqxJ/ZAOo6fTOtNNq+FnevmKnQT5zuq75ISNjydNpvO55z6Nl+V20X36qMtVLAAnVM Dhhg2r4RdfX81rGFgk+ufySz4uxOYnoiEF+KD6eUNBLCNgb6qqnqqQpLPRTzKMrmuSdS b2OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696259497; x=1696864297; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CH5x2RpAAFVf+XEG+0f6TamSE9fPT169LwjSjxAPY4U=; b=Asz0GfO5fiMIenqiCg+rmxi39u5h8OXvGAsWEZXLqckr7Off7KA9imvCoWDnJC213P 2oqB16eHJVxAKO7/MPPByNEMERk7Rbf0we2Apb/u9FgIIbGN54WApgIAksq4iyeFu3/Q SxBlbAxlQ9F97xmnXsMVAEv5ewMh0NSJSjODEn0CHfBq3x4VR3uRY/oOSzmqPLeAL+j6 Zjx4sygev65Qfax1U3e2hKlR1PriK9laE+yl/KB0i43gD1/31BpdAxpuVA9JiYKhGWW3 +cNaa9EEPolpwTBEormFXMF/974N0Zu3PtY2tuYizSrd7NgEhWjNaVxVEa8njfjuwneX zjvQ== X-Gm-Message-State: AOJu0YzoMIpBQq8TiHfo0IYois8Ud9aVfHTb/eIl0QTzZw1r630nZJfg zlHDE6f0Py2PP12zpkwlNO7wuQ== X-Google-Smtp-Source: AGHT+IE3cvPQyM2FMWspPhPjMKub5vjeWt6/Vb1fFmHzwxugTPglcumTh9TgVZ+QYgvakVdrqaJiiQ== X-Received: by 2002:a05:600c:1c9d:b0:401:609f:7f9a with SMTP id k29-20020a05600c1c9d00b00401609f7f9amr10681875wms.8.1696259497220; Mon, 02 Oct 2023 08:11:37 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id o14-20020a05600c4fce00b004065d67c3c9sm7473193wmq.8.2023.10.02.08.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 08:11:36 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Anup Patel , Atish Patra , Ard Biesheuvel , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 1/5] riscv: Use WRITE_ONCE() when setting page table entries Date: Mon, 2 Oct 2023 17:10:27 +0200 Message-Id: <20231002151031.110551-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231002151031.110551-1-alexghiti@rivosinc.com> References: <20231002151031.110551-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BF6FA40014 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: oiz9j7dektbffpytifb5gotbbq18we6g X-HE-Tag: 1696259498-219804 X-HE-Meta: U2FsdGVkX19FCU04KFKndL6zwdxerAtkPYcA6XyesMShizDOuAjHGBRPkjO+ZLghrQew7nqOItfokAJNnZCDnaeKKC5l33JurFX1YX3/yz8AIF0+uZ3wQ2IF92LTjbtjPJae5XLCnFdQHzGdUI623CY6Hh3gxozlawlxE/hJHwpy4+SfF3y4WUBbQTzUWF9gWMCStLk5sHx2Z3FvL0xVNKeJ/g+y3C1L19tbyQWEH3t7YMmYp1mnMVr/w70hUHOogrs0Evzwnomhbi7ceBG/KLKWNyv8yu8sPM3TeR8AObobp7B0EXVsGtI1OL+ngJRYAK6UeW3w5Z9KQ0XL9C4zeMxHTlYi27aAbSbQFeazbNu2HDAfn/RvyUabGZ1USvbrmE/T//RFYgPOySYKKoUseUn5jxrYMy7MraLLcWPHS8t95/wnMURdSLib5SvVpbX8EuDz746iDBRUPg1EIVs6i1MzFw7163wU8Oqmm5ecJ0blpchUbUpst3sZtgKZ20iYZAMBMoNQsvtJ+BS6UNAxtX/E/xCWZJ8o/LWL7eV8/Pt3JHm+DTpukcWpSDkX4CnmO+wtcurvwQx3nGp0AcOqw2ns6QiF+HMMOu/XFW4H0b/qwgWUT9he11dRcLIEv6H63mBIHkQCykglZud80og53nChSZ2SsXMLrm9cxClJOMQg9tJqrZJg/g1TlNyoeNB+VFTqd8DxUw2CpMrWkmb770zUg5xUyJ5eS8Ohf9boOTeKky7ARsN0Aw2XspiidHjMrANdL/e4ZovXlGUauJhtEAi3snlj0llasBU66Bs5vm9icn5WX/pp8S7ng2SUw1GILbCN1nMkd5AVSOmEshtXtyBNsz/Or5GnBj0GYBZdfEX2/bD/03xcVQuGVWaz7oRVXb6Gi6mGNT7fHixJsk62ZjKJ2RRtl48KGDq4B/AZcsFnhY9lBMaHSecYKSu5Om1YUjgdmWJlVKs3QVAX0dB xqB7GtRU egCY8tFYuNi5YCfTJnav5C1mPdCU94E1bQcONC8Bm3GpRIQCFKW1ZSR4ISLzUYfj2ll+npnu70rPPmHA2tk9CgdxbE7FAefaj2rC4iLQ27UGxJV7cpABHPdtu51T7ZfykwO/d/ODgbfUG1NQXLor4HMaw8KxA26i/hFDDHNVdowRuvMArVbtp5IlwSC8Jr5uADf4iCIjdc/6WftZyZF8Cavb0b5jnqJOmLDSelsFpMCIbw7oKVJJTJ7abXIkiYqz3pi3iDhK+PeQuJNMWjJcstETmJaBCykekCOMGybIr0WEQTPF3vjAzbd3HDoC097kiEgUwd504nEBcIcCWBmgxaya7/CXoMiRlW/ENMz8GIJcIaSYjGtrrpYaEcnK3uHC044vbyi0WbZJzGCgh67ZY4TCAIjIGdYivu9p/PgsPqCwb852ffKtT/TApu8jgBRgzyEuaueCF0ujHzGuwOOUmSzVm08rbJIzurcIbBt3rHn7/dMrmOoRERhq8cN0BsGa4o6VL/VesToIMOA5UooHjP0mRkwWdB4VS/yD7vKpcLB56txzjN6XessjN9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To avoid any compiler "weirdness" when accessing page table entries which are concurrently modified by the HW, let's use WRITE_ONCE() macro (commit 20a004e7b017 ("arm64: mm: Use READ_ONCE/WRITE_ONCE when accessing page tables") gives a great explanation with more details). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable-64.h | 6 +++--- arch/riscv/include/asm/pgtable.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 7a5097202e15..a65a352dcfbf 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -198,7 +198,7 @@ static inline int pud_user(pud_t pud) static inline void set_pud(pud_t *pudp, pud_t pud) { - *pudp = pud; + WRITE_ONCE(*pudp, pud); } static inline void pud_clear(pud_t *pudp) @@ -274,7 +274,7 @@ static inline unsigned long _pmd_pfn(pmd_t pmd) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { if (pgtable_l4_enabled) - *p4dp = p4d; + WRITE_ONCE(*p4dp, p4d); else set_pud((pud_t *)p4dp, (pud_t){ p4d_val(p4d) }); } @@ -347,7 +347,7 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) { if (pgtable_l5_enabled) - *pgdp = pgd; + WRITE_ONCE(*pgdp, pgd); else set_p4d((p4d_t *)pgdp, (p4d_t){ pgd_val(pgd) }); } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b2ba3f79cfe9..b820775f4973 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -248,7 +248,7 @@ static inline int pmd_leaf(pmd_t pmd) static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { - *pmdp = pmd; + WRITE_ONCE(*pmdp, pmd); } static inline void pmd_clear(pmd_t *pmdp) @@ -509,7 +509,7 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) */ static inline void set_pte(pte_t *ptep, pte_t pteval) { - *ptep = pteval; + WRITE_ONCE(*ptep, pteval); } void flush_icache_pte(pte_t pte); From patchwork Mon Oct 2 15:10:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13406332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67F32E784BB for ; Mon, 2 Oct 2023 15:12:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02C788D002A; Mon, 2 Oct 2023 11:12:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F1EED8D000E; Mon, 2 Oct 2023 11:12:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC0478D002A; Mon, 2 Oct 2023 11:12:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CAF7B8D000E for ; Mon, 2 Oct 2023 11:12:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 952F5802C7 for ; Mon, 2 Oct 2023 15:12:43 +0000 (UTC) X-FDA: 81300863406.08.E5C0D8E Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf02.hostedemail.com (Postfix) with ESMTP id 96FA280026 for ; Mon, 2 Oct 2023 15:12:41 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=Kg5TuK7f; dmarc=none; spf=pass (imf02.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696259561; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WAROWeuegt9axYlR2gFhUBC1uOuBg0VWiNfGep/WqRA=; b=pwmRke6dD9OODv7mNfPI+wgOcKUck3A306P/y85YIxEBpqkioLc7dKjpuEjahLPpsXf1yV VU+Djzv9N2FThFbJPOYeG9h7Vamv4Ek8NEMC59bHj4WFSDYIuCF4uPOvgEn+LmLxWgaf9/ 0NPFM3Qo8VbD+pM7dmE+xC5RIX19Kro= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=Kg5TuK7f; dmarc=none; spf=pass (imf02.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696259561; a=rsa-sha256; cv=none; b=0C/eG7TVWJKDzhN86S/Ft+U3AuNQUl+6jEnh4zF8cols62fq2AGLexXhDBXtVEf7GQ+Rtj Xke9XJ/Uo9PolOFEpZLE37gGpaBh6LMLy/tEzoCsOHoy6UbYLJEXgSQU6ELO7pA8tXl29Y CbeqjGgf+hcO4ZfyZvTcJkXjTAaXmC0= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-40675f06f1fso5112105e9.1 for ; Mon, 02 Oct 2023 08:12:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1696259560; x=1696864360; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WAROWeuegt9axYlR2gFhUBC1uOuBg0VWiNfGep/WqRA=; b=Kg5TuK7fa63jvndLucImYBdCOUd9e+AzCW5JwqJuRe85WJxHRn9Frg0Bozuv2zeOv0 +7duWU0LnKSx2eh55rvw7N3+Z87UhHCCAaNvIBZD2O3JkxIR0bXssQQu1UkRLnOWh/cr b66teAkKNe9r4cR/oqAR6fKuiMf5D8LUygo3Eo8suiaRVTSQw1L6GUSGq9R8ZPKNQXDJ HJW72x4l1RVuQWD52KoFTZA0crbYvR0ZSJa3EEFjeEZpQQv2ipGvQBAaCsPWhN7z51la Wyzr5pPN/Q8fMPc3ii9cQJzF+lk4+QQjQ7wnCqz3QIydgmwXekCllqvnuIM8uMcemDAz 6N8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696259560; x=1696864360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WAROWeuegt9axYlR2gFhUBC1uOuBg0VWiNfGep/WqRA=; b=eTlHt2AOt5i60TNA8yO2zHcbj0U/fuYQLy+A1ZNlwKpXn/sJGugaksbLzJZekXx9/y toUzG7C/h0fvYY5dggDSWnIiqEGqGNXyA51+nrFrz+eHq2QE0PyV6mveYbZBQy7i6Tua xP0FVnKllMyk9e2rfFKRRaICi9NQC4Q3MsAopEBbMgTQMHQKyUAOa1Ve3cbn6b8vGPLX HNkEhUu4wtMUbvrpzqTrzuahuooTdEUweaw9s1zbXxrZMaGpLV+yUVUAAwqlXzCZKWd5 zyiZXP5o5iU97arWxoLzWcZcopzlcXxEhCascR74F7/E7C3Y0KcHilFEFil16aG+TbZS AOzg== X-Gm-Message-State: AOJu0YxJG6AeXjNXPGgJ8WKOY0dFwwzSMmKV43Asb+K3pjEoXS8xGLJY MJm1R6f4Qrijdjwow2qagQqbpg== X-Google-Smtp-Source: AGHT+IEp4TyiMTncEFWIJawksGoamhltIe4ZJc8GCSTvwz3nHyivxU99XVIdB916hW0jM9qw6ka5vg== X-Received: by 2002:a05:600c:5022:b0:405:3f06:d2ef with SMTP id n34-20020a05600c502200b004053f06d2efmr9800701wmr.4.1696259560252; Mon, 02 Oct 2023 08:12:40 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id l5-20020a7bc445000000b003fbe791a0e8sm7507939wmi.0.2023.10.02.08.12.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 08:12:39 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Anup Patel , Atish Patra , Ard Biesheuvel , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 2/5] mm: Introduce pudp/p4dp/pgdp_get() functions Date: Mon, 2 Oct 2023 17:10:28 +0200 Message-Id: <20231002151031.110551-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231002151031.110551-1-alexghiti@rivosinc.com> References: <20231002151031.110551-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 96FA280026 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 5y174rnitwa4x4mzxisw84tptra4bkjb X-HE-Tag: 1696259561-599988 X-HE-Meta: U2FsdGVkX1+M4KJLqoRdnYQu8uuV50CEvHg2qcnCcxUcLB5hZURpRCCKl+G8hpGeJ8Ki/RPNfUlfjRQI35QUB27uk6oIAkufkr1LnMADMhWNxwnODZgqVC/2bib4kt1aADXlyPM+VlorNTfKlE0SZxpoQNnKN28lD7p4R8Pi5C52XxJf+f49e67ApPGzZPMJU8mig0UCMhrziJya0KlyVioEQsLyXwLq/W35d/dXdTvd/d/mn9KeXJ0a/fyIzMKbHejm/LLEII9R2KJZyB6p8wzUvstQqL53IVPGfjVhEk3dvRoylyzDGBxDDUpHeOXF7CMojz1Kq/AHbo3l9zn6D6vIOIA7zIiQmZ2drgVdCsGwt9fyAZGqB8V0EKQDfLzQV2eKZPzaMJ4LnER9VNRkFuE8jbP81KQG19tavi+ntPAfZp75vjunzNKV4mlvrzanYDb/stgb/Pg9kFAwAHmwSnSTsUSI5RcAz78U5SiZl4E2FbRTY7ljYWR04n6HcoZG/OrdQWJFGAz0A5zzgonaMdt+bb5DX/PhlTR7RB9z31Z8Xviai0tjiyVypIGR7CvrUDwaD1VOO9kAOvO151WH+eoU8iAztEBy/lMc1qW+xMJH1stPgbd0pHYjALnBm4UlSuTeS3Cu9ryPwq3gAqGEN6L5mllMHE5jcDeFXgxdPF6apJ1kuZi9zafRx/VDXowj17T85w9L94HZzflNKrCczCo8zGFlQGNqtv2p/p+e/AyFJSF3C15FwzegL7oQqrEqh3wl9/pi7vq8hSloJRWpEDWBY3M8kLy7FjWLYzBBdN1fQQ+UPlymmUDtb5LFND6hCd2l5OFfhVJ2ZX2kK90Reqze4LwNm4KGCQh5FfRc5jVAAZlhXEh6rFW57gEVmDBhJLiYuV9Q0/5siTRxsX6lPK8aL3PV077PzL9PzbsAnAeGbBQao08oZ6kmSA2zBkfuC+AmCX1fDZ2gkFzI5oh KcbyE+Lj StE8nJHyiOVancWyosWqPQtO9LzNe8ZCIqH6c0mpAfYJrVfy2gZFhgcusF7uW3Wxgsf8O7/HQL/D4w1DPsiIVEAl5pOiAPYTLwDYzwhNMVlqaYbbjDq5ux2aJ2FjCXMcCun6TgR0nemOAa3lc1grmeccB2lFAwbD79QDn13OMEqPkzR5cO7ke+qoZt1fk5CQ2ucaL+pUY26HKfy1s3Kl88xxq7a1EGPofulTVkPzEbDO20hoPELFFdcVCY54Y/eDd/DaPtkkOBH6PzxHLSd6JAPF8Esd1xUrQb08RBgBnmBwKlrSmNM+/NqIWl9GHa+a1avrQ3PNuH9SN5aD0GwQitEA7pplOQcEDos59MdbXDHpZzGpxLWrOaYC+FZG22wxekAPfBF6R/TrTNBje2Ptdu/GIbdvt0/RwgYOQdPfSJ5wuftJWPbiHwTmlW67sUS7YgizBOSNZC6ElnsyiUNb/9Bpgk8bTosN+JnsB+MFu28Efix0H8GPillL9oA1KXpndljAKd627K8tlPfHWuGOIRjQLMY3dkrD7vrXOY3ATim3QkZOktccK73VlgQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of directly dereferencing page tables entries, which can cause issues (see commit 20a004e7b017 ("arm64: mm: Use READ_ONCE/WRITE_ONCE when accessing page tables"), let's introduce new functions to get the pud/p4d/pgd entries (the pte and pmd versions already exist). Those new functions will be used in subsequent commits by the riscv architecture. Signed-off-by: Alexandre Ghiti --- include/linux/pgtable.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1fba072b3dac..4ce68bcc201d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -284,6 +284,27 @@ static inline pmd_t pmdp_get(pmd_t *pmdp) } #endif +#ifndef pudp_get +static inline pud_t pudp_get(pud_t *pudp) +{ + return READ_ONCE(*pudp); +} +#endif + +#ifndef p4dp_get +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + return READ_ONCE(*p4dp); +} +#endif + +#ifndef pgdp_get +static inline pgd_t pgdp_get(pgd_t *pgdp) +{ + return READ_ONCE(*pgdp); +} +#endif + #ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, From patchwork Mon Oct 2 15:10:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13406333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB1F0E784BC for ; Mon, 2 Oct 2023 15:13:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 472C78D002B; Mon, 2 Oct 2023 11:13:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 449FD8D000E; Mon, 2 Oct 2023 11:13:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ED988D002B; Mon, 2 Oct 2023 11:13:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1CF0A8D000E for ; Mon, 2 Oct 2023 11:13:45 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E7FB812032B for ; Mon, 2 Oct 2023 15:13:44 +0000 (UTC) X-FDA: 81300865968.24.B105E6C Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf19.hostedemail.com (Postfix) with ESMTP id 13A9A1A002E for ; Mon, 2 Oct 2023 15:13:42 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=LXWiwC6k; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696259623; a=rsa-sha256; cv=none; b=yf47GgdcVVNRMSbeTDlpZbQJkZkBtFPS420mVtK/NieJ5AmCeqAmRA2uthXw5vNSZYwOME TUPOf+rupFITeEG/xEoxeZDvQqGqW0K5Xji411Pw1QRlFadT6ajLzXO85WAcp7hd1/g8vl owdb4vzohJyN562SEegvX7un1mjxQrU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=LXWiwC6k; spf=pass (imf19.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696259623; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PKlKP5/9RI4UDE4Mv9BGz18R+1865OFX4ZcGb0PXpPY=; b=8H2iAh37LfteyOfNKXCwWxYyflpp2/gWDQGZmqk2nAFtxAxI70hSmnXyjjIBAVhO8VmwvJ gMSad3GBB9pDkOKPaOgbeZ68y5ZEK5vXOXAoFSZSOGEJU8/0SDGOj60v0oOZYTB2ti0F8n DleG089ACentT/U8rgPnL1KS63USzDE= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-40651a72807so43785075e9.1 for ; Mon, 02 Oct 2023 08:13:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1696259622; x=1696864422; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PKlKP5/9RI4UDE4Mv9BGz18R+1865OFX4ZcGb0PXpPY=; b=LXWiwC6kSBUHLMxL76XhUa2kuNYJMe6DVtbmvCCtEWHDQV5TH7UqerY/gFQn1sDgmm kneSPjAZ3O9v4bFDCffs3PRxhlwM13FjZr7yrTnL4odkSZKXo95Nr8iAvVmS3o/geeW/ KqZmshvRRVDdUyR7en8WJSj//XSHiUHxoBMPyAA3a0aouPikMuVE8IDkn6SRr4P+D7+T xwl48q4n0ZynUqzIebn9oCOXe92Be98yJO3CqFd2wz/9c1KkdP1xCDFdgcGHm1i7KdAc X/fqK3RiW2DvAulFuHrd7Yuh3RXDYVWj5d9+Di/3QcOckC5pmko9qOq1tOHv41soer7j EUTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696259622; x=1696864422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PKlKP5/9RI4UDE4Mv9BGz18R+1865OFX4ZcGb0PXpPY=; b=uNp/CmryLIeemHtPuDUol0SjotptBiD5S62Q4niA7RmPuOBT8Rk0RsPDySZX4ef+dp pJudJmDmUWh2YBV2UPSjY+DKoLCrQEp4VZRU5edDDucA31Yj5mkalY/jhvkwUmtdXI3e EI56tsbCL7q3sBkw70/YhI0jdqm4XOaq1oOEu37k6GrYtnELRYopxNut8El4/+EF0pMR dvaytX+by+6UM2HQALPCMZMtBJeoQWrcaG2UV4XHlyCi61BS5gS/iT7hfhH3XmOUyT3j PzVKsFm25zf80oEs6fFvSHL6SOMzqICcbca+AZ7cqWGkXU5EiF8C7ZzKn+u86rp/TnYU tTkg== X-Gm-Message-State: AOJu0Yytjtn/13raREgRKgHI/1yp4EouC3FlL4uKC2inkxk/HQr/kj1L 6+nen/mUyHTBqAh6K7t9xSgRhg== X-Google-Smtp-Source: AGHT+IHO+YzsPhqZFo4O8yw5o3ilb0oSA+vYnzJ8XqlEcYHitO4Hyb9F4EkPU7aucBS8UpRf+rPooA== X-Received: by 2002:a7b:c8d6:0:b0:406:5190:7d07 with SMTP id f22-20020a7bc8d6000000b0040651907d07mr11349388wml.17.1696259621727; Mon, 02 Oct 2023 08:13:41 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id l5-20020a7bc445000000b003fbe791a0e8sm7509731wmi.0.2023.10.02.08.13.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 08:13:41 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Anup Patel , Atish Patra , Ard Biesheuvel , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 3/5] riscv: mm: Only compile pgtable.c if MMU Date: Mon, 2 Oct 2023 17:10:29 +0200 Message-Id: <20231002151031.110551-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231002151031.110551-1-alexghiti@rivosinc.com> References: <20231002151031.110551-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 13A9A1A002E X-Stat-Signature: mhckz9djrn6keg9enyozrmbeotfhi5kt X-Rspam-User: X-HE-Tag: 1696259622-97926 X-HE-Meta: U2FsdGVkX19hyRBlO9nTkU9XkKqH1E6rNrSJl96Gugz8+DMIPy4tqYjwipLUEamYgLzZ9cTsKGk551E2PMKFK32qQlMQL9zuabFiFAlsm/DG7o5SVosZfHvyACvoURKkra57Xg1/o1U7tHIP3yCDeljdiuXQSClKUcZbT2BNfy1CgBVFzrIp1ESNf9FBADqhaMwc/IpbanDFfWY0tWhBbCYboFCz/BWlAbyKmw1LHO5zqMjZIh/fG2P//QYRbhClUgCqOhijofCVKkAoKRjnX/olPxtz5PfJ8F3nYhBShRvR6w+Cq8ib+VqmSrv57Ammipmt0cPutAK41oPLbKn574fNWGlf8mw8NA+tV1eBTVBsxV/aBMzP+Df7L9KRmWIs8j4gulGDNQ5fbBz0x0TpL+ekbDjf7HILynDKgR6KehfJUJQZsPzmqi2YzC5sk+IOP/L1dj4rZcZPt5fMgTCrYZrBBUwc2pgE1TBCnkkGfEhxA8PhNsdKEP+8hIRqMRXOdvBswmjcg36L6bmu+UyLFxZS6+kMWt3qGI1bTA2lC7hAv0Tj8GbS+UcNaPP6NAizrkX7tJMXh04S4A6tRVGSfHrO5Mn+/v+kgkyQBdWmQ2pN8tbPKFy05rmbktOPS5R+woO82FVgpbSEunzhcHqd0NdUOT3lDVvc6I9fKfJA0FAqVOaQi80i+cHah+wbIfcAw2MhnWT48YaM1UC49Fgj7sx7rBpzPdP1iWOYznokZiZVGchm712tpoIdGlMJ2dEHX1+C0gUCEGnOl/rWb3eHw+qU7844P1Lml6p8BjIepbh2o0WU422Bi9Ym+nwFCaJ7A0A4td4ZHtbPTD1t4wdhpq8EYk7eyOWgbblBh2yZYliHk+DcNdIJCJzC4xapQZJSmpy18dSqeJNYrpkgpXgjqpzkgEnXAhfM99IIZebgW0cRotgUSKHmEdtUT5SfUEREpUEzuq7IMpI1+Lplluw UTwOGLAL /OvA06sYQvbbVJfygJcJQyQ8sBSTW8XRchzWI0+PbiThWPZmb2+xdoGKu9WkzCtFRjiQ5uBsiOQORcpSR3KCG5pkMAwIjAFMWMZtMHufksMhvQedAeMzCyEWQIjEYAT8Qd8nG33FNMMzoRB7vcCrHohgY/a0IWmxV/tKW+yEV9aq5NM5bIq3GyzajJ5fYWTnlnLYQ+1ELiV78cMARrguJBvXaoQPaoFovxxhFiuc1VpPMANXrPuFc+pqBwknWZzoW1ZwOW93woiEBX0x2q2gzEK+tqIDvOwK6WMErQDjNox8Pj7PfVMt/ku6rcVjo4lHt8cGZFumXuAz2pUb+b6eZyV2doQPZm37S76qsYQqpr+aUWa4GVm3Eac/01LPKUpJ7skncnSeZvR2NgcKPsSlIaW3IDuHMplbaDDe1XPFdn1t/XgoOJ81LkxJFiCcpanH393owq4F6NRw7r52Fpa2WMGqtViLWv91a7VuwyTT/nJeBUqCGRH6+JBRZ8Onb78xvBw7ySAHeaJjUGVI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All functions defined in there depend on MMU, so no need to compile it for !MMU configs. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/Makefile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 9c454f90fd3d..c71d4253a171 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,10 +13,9 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o pgtable.o obj-y += cacheflush.o obj-y += context.o -obj-y += pgtable.o obj-y += pmem.o ifeq ($(CONFIG_MMU),y) From patchwork Mon Oct 2 15:10:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13406363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9881E784BF for ; Mon, 2 Oct 2023 15:14:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29F898D002D; Mon, 2 Oct 2023 11:14:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24E468D000E; Mon, 2 Oct 2023 11:14:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A2248D002D; Mon, 2 Oct 2023 11:14:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E9F718D000E for ; Mon, 2 Oct 2023 11:14:46 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C32EEB3BB3 for ; Mon, 2 Oct 2023 15:14:46 +0000 (UTC) X-FDA: 81300868572.07.8B7CB70 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by imf29.hostedemail.com (Postfix) with ESMTP id A8FEE120016 for ; Mon, 2 Oct 2023 15:14:44 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=PlOBEwyC; spf=pass (imf29.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696259684; a=rsa-sha256; cv=none; b=ub/POtuwkoi1wy8a/X/2mnHsO/aJQQ8i8Ts3MU7Lh+tF4oUAYPs9aCF9lFjF69E9gE2jxx RV2jIIy+GFi+qh3hdoyOCa6F/19QrkgqF4vurPueJ1pZVZ2zL355m51kIBw/5VpmJH1Aw5 e1XmZOk7hT0w9rhwi2jniIXdv7Bwp2E= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=PlOBEwyC; spf=pass (imf29.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696259684; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xIMzmr/PY70Z2D2FCaui5pQoBeqQRhmKx80FxNdR6+A=; b=5iQVm/EA1XmIeLgSjuBZp5brtccX6iXIm8q8c2y2fZQBQu1YN7fhC9HnZQh1Fbhl5cj9kp QK+Z7JOnq6jp4zSWL1Qixqofz8z3mRHwzEgi0YR6SC+eRReyGRAvPie9nCLqebOxcxlai+ o3Dg2/IXDLlZDoRy5q+HOgMFCENFvO4= Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-405497850dbso147199715e9.0 for ; Mon, 02 Oct 2023 08:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1696259683; x=1696864483; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xIMzmr/PY70Z2D2FCaui5pQoBeqQRhmKx80FxNdR6+A=; b=PlOBEwyC/D0p4l+N9T4No5y9gt2nvWKUUQRTnxs8jKVBc6cftowUjKjgaJyJq8Cy5h in6Hbur7G6+lSVS81Z0nWR1kS6MOquv8SdKZ2tcXUZbUMczFRWLSJooZ6ThjQ0VTZvL5 ltqvkrpmYAoOhvJB6gijgUblzwV6koFp32etY225jjR+CkSa+dwpXctimuMoyc8ejSSp yqvzmB6dWYGLXDMp9lEa6O3Ckp4yZJzmFW1dPuXeoOh5wNNG4M9UpglziLUgMHPEqGys Q6v7S+kDPIJC5Xe4RExkhxajq/qrswfrtEMvwwOxrQ5xoxdYOD81QjWz2ZTT2lSrtn9d pzAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696259683; x=1696864483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xIMzmr/PY70Z2D2FCaui5pQoBeqQRhmKx80FxNdR6+A=; b=ii/1HN466iP/+zpmFosMoWfTQCDyp/ab4ERBD+k1FwG94FjEZy427ibAZfPFUvoQ9O VKM63qNsgjQ1ohwf+FSwLOwhfNMolzy1JTTlcKpnku0+7PQNRixi+r3ZM7IlF0FZCu7W 680qTPR0sA7xrZih/u1Q95VPIrvBlKPBwAVJ/hz09Ox/Mbbo2sQsUvf3wdz8+dQ+EeSk lJBbKzpmrBrqAjNKRBdOCFDIZ8wY1+E3z7kMXJj1qkvc3JaPb/Hp6WDi83IEXenzx2ng OurTxLckuTjpiaLeYWQjSMPxKMc8cYIE/vtnOmREXMKJ18Lx+j/3IcG5haRj8L8vKp8g 3LbA== X-Gm-Message-State: AOJu0YzvTUOxvUDcTmf27CCrDMQnKaD6LfOizo4mzQIpZ1uzgmipnySx Ci0CMRPDsJsdIAUR2fGtcUz8gw== X-Google-Smtp-Source: AGHT+IG214vp+ZTT5MpbSlfQSK0LrUZVQ/xG9fCJWALrjsfwrlLNz7Oxp25mnoTTeHy0voW9+EtBtw== X-Received: by 2002:a7b:c8d1:0:b0:405:3b92:2fed with SMTP id f17-20020a7bc8d1000000b004053b922fedmr9764385wml.26.1696259682974; Mon, 02 Oct 2023 08:14:42 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 9-20020a05600c234900b00401d8181f8bsm7491028wmq.25.2023.10.02.08.14.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 08:14:42 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Anup Patel , Atish Patra , Ard Biesheuvel , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 4/5] riscv: Suffix all page table entry pointers with 'p' Date: Mon, 2 Oct 2023 17:10:30 +0200 Message-Id: <20231002151031.110551-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231002151031.110551-1-alexghiti@rivosinc.com> References: <20231002151031.110551-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A8FEE120016 X-Stat-Signature: a739z6xzdcgojgzyak7sjyxy3cu3f4gh X-Rspam-User: X-HE-Tag: 1696259684-832558 X-HE-Meta: U2FsdGVkX18Q7i8gyJfql+mSB8KiW69VJAdH2duhPbkCa7WQKbSkYUFr57f6y25QnnrwdV7SZOtkEbdGIutEqqmCRO8pU572RToiGBIqWtIYN7qLOACkLjRde9lBFtuVfArIWChkYLMtg8OMpQtVsqT16AuJPNzaOmX5pRMawipguXoLIZd8zTbcwzrAJs2DSmGviEjTn2lyz7lN0SaNpOOWJDWILUdxk8muGxCJGFI+gOqFaEcfO3Jcb9R3r7sBQh+2DtfFgu7ICmdcp3zGxyk2d3U3q7FeHAED3me7XwjC/BlpqR+Eu9iohdr8Sc9Ws5+vQSkm3gN0ghfImwyzEK2BQhUlDX+cE+yC5feu6LLRh8Dak3KZfFt9eEljJxtntKsvk43XAXuZGs7b6eVSeYQEO5tUPFSswj9vWZ/gT8NaNdYuHa/xY+QJS6mYC3IzZMWQFrkGE+dJha+Lg2+JNXIAA8QNGjlzyxtxlfZBvKhKedSuwUQdjYkh1XRR7mCJStXXldiu0DUG0sWZDaDaYa3jxvPbUBSIXhPjYAgZs7uQmhZYAadOGb72RLlty1bdkqgJtm2vDesOSaaNZzhu8Fb/siN9Xs+3qH6mIVXEYiPR/QfgbcLM3YAX6NxQqWczFI1BiPwYbF78wPLyA1ZdBgZE9i7PQn9aahxZ0E/TCRBEheeMNRoWnsEV2fxs8VmFDn6UB9eSQnrW95bzPjtLeMCYhVuU7yw/aU6NyZpsFSCV4K/SeTRA3hEyOEYU43oHEBQnl7/dipoLfnT/OFGVE17lbMuooygX8lpzY9GSznnbKraKNsP2dmjjf5eKBEEKq7slE2lSUrN2xRPz0RAjLbugfTjTc0+0YK54gBAgWO7CFLD+xPY57hGj6EL4LI/tUL7/2R7hw7IUeTk9BfFJEFDbZ0DtB7E9lKQBT9fCnsu8bGDKcGe6gmZHMluUC734cHD+cPTwmyH3w+UX0pC +V2NKNSv S46aDIKbjtoMEXKaG5sss5EkyHIb3pjE4ZtpXwmHPpJqmVQ/Wg/afKYnbO7oC1YCGUOZ7pmiY0AKtH2EFeNnekyV0vf67cXAy1eQdMNQfR2EXszuIXhF3a+0/ky4MOVYL7R1nkBArzbtV4k2kqOp9FQ+F9+pkAnLveD8r7Owlz+vhxR+Y8DpgG7KnonQHzKXZ+E2sOeUovfOs/7UzmcQAMDiKn2z1jxJgNhxLEYrGLAub5NWB9sfp+FNcHqvd1Em1zdrZhH2i3L3kSGqYXqQls9xZ15wGEBspBAQ/dH+p6vZRwCgCkWyrgY4+zMlLUQVCcDuld+6Iuq9bRkOHQ+GaqIiyi2GQlWFFc/SkKG5Ar16SeWG6j8KgLOj/hyWp1Cr042Ye7JvyhEvByIcqusQBISIinATAnQxXHdeQEglkocOiyDxw7iJd0fLosTUC9xwtmGk4Z0J8FtWjjkYD5WX1ijEAjohT93CmKFD3i35lsNNRXoJMp33tIsdj3/6Oh4VaKGYUCHwcyi416nk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: That makes it more clear what the underlying type is, no functional changes intended. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/kfence.h | 6 +- arch/riscv/include/asm/kvm_host.h | 2 +- arch/riscv/include/asm/pgalloc.h | 86 +++++++++++++------------- arch/riscv/include/asm/pgtable-64.h | 20 +++--- arch/riscv/kvm/mmu.c | 22 +++---- arch/riscv/mm/fault.c | 38 ++++++------ arch/riscv/mm/hugetlbpage.c | 78 +++++++++++------------ arch/riscv/mm/init.c | 30 ++++----- arch/riscv/mm/kasan_init.c | 96 ++++++++++++++--------------- arch/riscv/mm/pageattr.c | 74 +++++++++++----------- arch/riscv/mm/pgtable.c | 46 +++++++------- 11 files changed, 251 insertions(+), 247 deletions(-) diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index 0bbffd528096..3b482d0a4633 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -15,12 +15,12 @@ static inline bool arch_kfence_init_pool(void) static inline bool kfence_protect_page(unsigned long addr, bool protect) { - pte_t *pte = virt_to_kpte(addr); + pte_t *ptep = virt_to_kpte(addr); if (protect) - set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + set_pte(ptep, __pte(pte_val(*ptep) & ~_PAGE_PRESENT)); else - set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + set_pte(ptep, __pte(pte_val(*ptep) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h index 1ebf20dfbaa6..510045c05ee8 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -93,7 +93,7 @@ struct kvm_arch { struct kvm_vmid vmid; /* G-stage page table */ - pgd_t *pgd; + pgd_t *pgdp; phys_addr_t pgd_phys; /* Guest Timer */ diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index d169a4f41a2e..1bb3c9fcbe04 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -16,65 +16,65 @@ #include static inline void pmd_populate_kernel(struct mm_struct *mm, - pmd_t *pmd, pte_t *pte) + pmd_t *pmdp, pte_t *ptep) { - unsigned long pfn = virt_to_pfn(pte); + unsigned long pfn = virt_to_pfn(ptep); - set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pmd(pmdp, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } static inline void pmd_populate(struct mm_struct *mm, - pmd_t *pmd, pgtable_t pte) + pmd_t *pmdp, pgtable_t pte) { unsigned long pfn = virt_to_pfn(page_address(pte)); - set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pmd(pmdp, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } #ifndef __PAGETABLE_PMD_FOLDED -static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) +static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) { - unsigned long pfn = virt_to_pfn(pmd); + unsigned long pfn = virt_to_pfn(pmdp); - set_pud(pud, __pud((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pud(pudp, __pud((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } -static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) +static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) { if (pgtable_l4_enabled) { - unsigned long pfn = virt_to_pfn(pud); + unsigned long pfn = virt_to_pfn(pudp); - set_p4d(p4d, __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_p4d(p4dp, __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } -static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d, - pud_t *pud) +static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4dp, + pud_t *pudp) { if (pgtable_l4_enabled) { - unsigned long pfn = virt_to_pfn(pud); + unsigned long pfn = virt_to_pfn(pudp); - set_p4d_safe(p4d, + set_p4d_safe(p4dp, __p4d((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } -static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) +static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) { if (pgtable_l5_enabled) { - unsigned long pfn = virt_to_pfn(p4d); + unsigned long pfn = virt_to_pfn(p4dp); - set_pgd(pgd, __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + set_pgd(pgdp, __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } -static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, - p4d_t *p4d) +static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgdp, + p4d_t *p4dp) { if (pgtable_l5_enabled) { - unsigned long pfn = virt_to_pfn(p4d); + unsigned long pfn = virt_to_pfn(p4dp); - set_pgd_safe(pgd, + set_pgd_safe(pgdp, __pgd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } } @@ -89,13 +89,13 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) } #define pud_free pud_free -static inline void pud_free(struct mm_struct *mm, pud_t *pud) +static inline void pud_free(struct mm_struct *mm, pud_t *pudp) { if (pgtable_l4_enabled) - __pud_free(mm, pud); + __pud_free(mm, pudp); } -#define __pud_free_tlb(tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb(tlb, pudp, addr) pud_free((tlb)->mm, pudp) #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) @@ -111,52 +111,52 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) return NULL; } -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4dp) { - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + BUG_ON((unsigned long)p4dp & (PAGE_SIZE-1)); + free_page((unsigned long)p4dp); } #define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4dp) { if (pgtable_l5_enabled) - __p4d_free(mm, p4d); + __p4d_free(mm, p4dp); } #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #endif /* __PAGETABLE_PMD_FOLDED */ -static inline void sync_kernel_mappings(pgd_t *pgd) +static inline void sync_kernel_mappings(pgd_t *pgdp) { - memcpy(pgd + USER_PTRS_PER_PGD, + memcpy(pgdp + USER_PTRS_PER_PGD, init_mm.pgd + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); } static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *pgd; + pgd_t *pgdp; - pgd = (pgd_t *)__get_free_page(GFP_KERNEL); - if (likely(pgd != NULL)) { - memset(pgd, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); + pgdp = (pgd_t *)__get_free_page(GFP_KERNEL); + if (likely(pgdp != NULL)) { + memset(pgdp, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); /* Copy kernel mappings */ - sync_kernel_mappings(pgd); + sync_kernel_mappings(pgdp); } - return pgd; + return pgdp; } #ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb(tlb, pmdp, addr) pmd_free((tlb)->mm, pmdp) #endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb(tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ +#define __pte_free_tlb(tlb, ptep, buf) \ +do { \ + pagetable_pte_dtor(page_ptdesc(ptep)); \ + tlb_remove_page_ptdesc((tlb), page_ptdesc(ptep)); \ } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index a65a352dcfbf..c5f608fe9980 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -303,10 +303,10 @@ static inline int p4d_bad(p4d_t p4d) return 0; } -static inline void p4d_clear(p4d_t *p4d) +static inline void p4d_clear(p4d_t *p4dp) { if (pgtable_l4_enabled) - set_p4d(p4d, __p4d(0)); + set_p4d(p4dp, __p4d(0)); } static inline p4d_t pfn_p4d(unsigned long pfn, pgprot_t prot) @@ -336,12 +336,12 @@ static inline struct page *p4d_page(p4d_t p4d) #define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) #define pud_offset pud_offset -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) { if (pgtable_l4_enabled) - return p4d_pgtable(*p4d) + pud_index(address); + return p4d_pgtable(*p4dp) + pud_index(address); - return (pud_t *)p4d; + return (pud_t *)p4dp; } static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) @@ -376,10 +376,10 @@ static inline int pgd_bad(pgd_t pgd) return 0; } -static inline void pgd_clear(pgd_t *pgd) +static inline void pgd_clear(pgd_t *pgdp) { if (pgtable_l5_enabled) - set_pgd(pgd, __pgd(0)); + set_pgd(pgdp, __pgd(0)); } static inline p4d_t *pgd_pgtable(pgd_t pgd) @@ -400,12 +400,12 @@ static inline struct page *pgd_page(pgd_t pgd) #define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1)) #define p4d_offset p4d_offset -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { if (pgtable_l5_enabled) - return pgd_pgtable(*pgd) + p4d_index(address); + return pgd_pgtable(*pgdp) + p4d_index(address); - return (p4d_t *)pgd; + return (p4d_t *)pgdp; } #endif /* _ASM_RISCV_PGTABLE_64_H */ diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 068c74593871..35db276bf0c2 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -101,7 +101,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, u32 current_level = gstage_pgd_levels - 1; *ptep_level = current_level; - ptep = (pte_t *)kvm->arch.pgd; + ptep = (pte_t *)kvm->arch.pgdp; ptep = &ptep[gstage_pte_index(addr, current_level)]; while (ptep && pte_val(*ptep)) { if (gstage_pte_leaf(ptep)) { @@ -139,7 +139,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, gpa_t addr, const pte_t *new_pte) { u32 current_level = gstage_pgd_levels - 1; - pte_t *next_ptep = (pte_t *)kvm->arch.pgd; + pte_t *next_ptep = (pte_t *)kvm->arch.pgdp; pte_t *ptep = &next_ptep[gstage_pte_index(addr, current_level)]; if (current_level < level) @@ -541,7 +541,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { - if (!kvm->arch.pgd) + if (!kvm->arch.pgdp) return false; gstage_unmap_range(kvm, range->start << PAGE_SHIFT, @@ -555,7 +555,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) int ret; kvm_pfn_t pfn = pte_pfn(range->arg.pte); - if (!kvm->arch.pgd) + if (!kvm->arch.pgdp) return false; WARN_ON(range->end - range->start != 1); @@ -576,7 +576,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) u32 ptep_level = 0; u64 size = (range->end - range->start) << PAGE_SHIFT; - if (!kvm->arch.pgd) + if (!kvm->arch.pgdp) return false; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE); @@ -594,7 +594,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) u32 ptep_level = 0; u64 size = (range->end - range->start) << PAGE_SHIFT; - if (!kvm->arch.pgd) + if (!kvm->arch.pgdp) return false; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE); @@ -712,7 +712,7 @@ int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm) { struct page *pgd_page; - if (kvm->arch.pgd != NULL) { + if (kvm->arch.pgdp != NULL) { kvm_err("kvm_arch already initialized?\n"); return -EINVAL; } @@ -721,7 +721,7 @@ int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm) get_order(gstage_pgd_size)); if (!pgd_page) return -ENOMEM; - kvm->arch.pgd = page_to_virt(pgd_page); + kvm->arch.pgdp = page_to_virt(pgd_page); kvm->arch.pgd_phys = page_to_phys(pgd_page); return 0; @@ -732,10 +732,10 @@ void kvm_riscv_gstage_free_pgd(struct kvm *kvm) void *pgd = NULL; spin_lock(&kvm->mmu_lock); - if (kvm->arch.pgd) { + if (kvm->arch.pgdp) { gstage_unmap_range(kvm, 0UL, gstage_gpa_size, false); - pgd = READ_ONCE(kvm->arch.pgd); - kvm->arch.pgd = NULL; + pgd = READ_ONCE(kvm->arch.pgdp); + kvm->arch.pgdp = NULL; kvm->arch.pgd_phys = 0; } spin_unlock(&kvm->mmu_lock); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 6115d7514972..6284ef4b644a 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -111,11 +111,11 @@ bad_area(struct pt_regs *regs, struct mm_struct *mm, int code, static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr) { - pgd_t *pgd, *pgd_k; - pud_t *pud_k; - p4d_t *p4d_k; - pmd_t *pmd_k; - pte_t *pte_k; + pgd_t *pgdp, *pgdp_k; + pud_t *pudp_k; + p4d_t *p4dp_k; + pmd_t *pmdp_k; + pte_t *ptep_k; int index; unsigned long pfn; @@ -133,39 +133,39 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a */ index = pgd_index(addr); pfn = csr_read(CSR_SATP) & SATP_PPN; - pgd = (pgd_t *)pfn_to_virt(pfn) + index; - pgd_k = init_mm.pgd + index; + pgdp = (pgd_t *)pfn_to_virt(pfn) + index; + pgdp_k = init_mm.pgd + index; - if (!pgd_present(*pgd_k)) { + if (!pgd_present(*pgdp_k)) { no_context(regs, addr); return; } - set_pgd(pgd, *pgd_k); + set_pgd(pgdp, *pgdp_k); - p4d_k = p4d_offset(pgd_k, addr); - if (!p4d_present(*p4d_k)) { + p4dp_k = p4d_offset(pgdp_k, addr); + if (!p4d_present(*p4dp_k)) { no_context(regs, addr); return; } - pud_k = pud_offset(p4d_k, addr); - if (!pud_present(*pud_k)) { + pudp_k = pud_offset(p4dp_k, addr); + if (!pud_present(*pudp_k)) { no_context(regs, addr); return; } - if (pud_leaf(*pud_k)) + if (pud_leaf(*pudp_k)) goto flush_tlb; /* * Since the vmalloc area is global, it is unnecessary * to copy individual PTEs */ - pmd_k = pmd_offset(pud_k, addr); - if (!pmd_present(*pmd_k)) { + pmdp_k = pmd_offset(pudp_k, addr); + if (!pmd_present(*pmdp_k)) { no_context(regs, addr); return; } - if (pmd_leaf(*pmd_k)) + if (pmd_leaf(*pmdp_k)) goto flush_tlb; /* @@ -174,8 +174,8 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * addresses. If we don't do this, this will just * silently loop forever. */ - pte_k = pte_offset_kernel(pmd_k, addr); - if (!pte_present(*pte_k)) { + ptep_k = pte_offset_kernel(pmdp_k, addr); + if (!pte_present(*ptep_k)) { no_context(regs, addr); return; } diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 96225a8533ad..7781e83b2f29 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -33,52 +33,52 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long sz) { unsigned long order; - pte_t *pte = NULL; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - - pgd = pgd_offset(mm, addr); - p4d = p4d_alloc(mm, pgd, addr); - if (!p4d) + pte_t *ptep = NULL; + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp; + pmd_t *pmdp; + + pgdp = pgd_offset(mm, addr); + p4dp = p4d_alloc(mm, pgdp, addr); + if (!p4dp) return NULL; - pud = pud_alloc(mm, p4d, addr); - if (!pud) + pudp = pud_alloc(mm, p4dp, addr); + if (!pudp) return NULL; if (sz == PUD_SIZE) { - pte = (pte_t *)pud; + ptep = (pte_t *)pudp; goto out; } if (sz == PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(*pud)) - pte = huge_pmd_share(mm, vma, addr, pud); + if (want_pmd_share(vma, addr) && pud_none(*pudp)) + ptep = huge_pmd_share(mm, vma, addr, pudp); else - pte = (pte_t *)pmd_alloc(mm, pud, addr); + ptep = (pte_t *)pmd_alloc(mm, pudp, addr); goto out; } - pmd = pmd_alloc(mm, pud, addr); - if (!pmd) + pmdp = pmd_alloc(mm, pudp, addr); + if (!pmdp) return NULL; for_each_napot_order(order) { if (napot_cont_size(order) == sz) { - pte = pte_alloc_huge(mm, pmd, addr & napot_cont_mask(order)); + ptep = pte_alloc_huge(mm, pmdp, addr & napot_cont_mask(order)); break; } } out: - if (pte) { - pte_t pteval = ptep_get_lockless(pte); + if (ptep) { + pte_t pteval = ptep_get_lockless(ptep); WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval)); } - return pte; + return ptep; } pte_t *huge_pte_offset(struct mm_struct *mm, @@ -86,43 +86,43 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long sz) { unsigned long order; - pte_t *pte = NULL; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - - pgd = pgd_offset(mm, addr); - if (!pgd_present(*pgd)) + pte_t *ptep = NULL; + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp; + pmd_t *pmdp; + + pgdp = pgd_offset(mm, addr); + if (!pgd_present(*pgdp)) return NULL; - p4d = p4d_offset(pgd, addr); - if (!p4d_present(*p4d)) + p4dp = p4d_offset(pgdp, addr); + if (!p4d_present(*p4dp)) return NULL; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4dp, addr); if (sz == PUD_SIZE) /* must be pud huge, non-present or none */ - return (pte_t *)pud; + return (pte_t *)pudp; - if (!pud_present(*pud)) + if (!pud_present(*pudp)) return NULL; - pmd = pmd_offset(pud, addr); + pmdp = pmd_offset(pudp, addr); if (sz == PMD_SIZE) /* must be pmd huge, non-present or none */ - return (pte_t *)pmd; + return (pte_t *)pmdp; - if (!pmd_present(*pmd)) + if (!pmd_present(*pmdp)) return NULL; for_each_napot_order(order) { if (napot_cont_size(order) == sz) { - pte = pte_offset_huge(pmd, addr & napot_cont_mask(order)); + ptep = pte_offset_huge(pmdp, addr & napot_cont_mask(order)); break; } } - return pte; + return ptep; } static pte_t get_clear_contig(struct mm_struct *mm, diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 0798bd861dcb..2afd13fd6e18 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -883,7 +883,7 @@ static void __init relocate_kernel(void) #endif /* CONFIG_RELOCATABLE */ #ifdef CONFIG_XIP_KERNEL -static void __init create_kernel_page_table(pgd_t *pgdir, +static void __init create_kernel_page_table(pgd_t *pgdp, __always_unused bool early) { uintptr_t va, end_va; @@ -891,25 +891,25 @@ static void __init create_kernel_page_table(pgd_t *pgdir, /* Map the flash resident part */ end_va = kernel_map.virt_addr + kernel_map.xiprom_sz; for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) - create_pgd_mapping(pgdir, va, + create_pgd_mapping(pgdp, va, kernel_map.xiprom + (va - kernel_map.virt_addr), PMD_SIZE, PAGE_KERNEL_EXEC); /* Map the data in RAM */ end_va = kernel_map.virt_addr + XIP_OFFSET + kernel_map.size; for (va = kernel_map.virt_addr + XIP_OFFSET; va < end_va; va += PMD_SIZE) - create_pgd_mapping(pgdir, va, + create_pgd_mapping(pgdp, va, kernel_map.phys_addr + (va - (kernel_map.virt_addr + XIP_OFFSET)), PMD_SIZE, PAGE_KERNEL); } #else -static void __init create_kernel_page_table(pgd_t *pgdir, bool early) +static void __init create_kernel_page_table(pgd_t *pgdp, bool early) { uintptr_t va, end_va; end_va = kernel_map.virt_addr + kernel_map.size; for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) - create_pgd_mapping(pgdir, va, + create_pgd_mapping(pgdp, va, kernel_map.phys_addr + (va - kernel_map.virt_addr), PMD_SIZE, early ? @@ -1521,30 +1521,30 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon const char *lvl; for (addr = start; addr < end && addr >= start; addr = ALIGN(addr + 1, PGDIR_SIZE)) { - pgd_t *pgd = pgd_offset_k(addr); - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; + pgd_t *pgdp = pgd_offset_k(addr); + p4d_t *p4dp; + pud_t *pudp; + pmd_t *pmdp; lvl = "p4d"; - p4d = p4d_alloc(&init_mm, pgd, addr); - if (!p4d) + p4dp = p4d_alloc(&init_mm, pgdp, addr); + if (!p4dp) goto failed; if (pgtable_l5_enabled) continue; lvl = "pud"; - pud = pud_alloc(&init_mm, p4d, addr); - if (!pud) + pudp = pud_alloc(&init_mm, p4dp, addr); + if (!pudp) goto failed; if (pgtable_l4_enabled) continue; lvl = "pmd"; - pmd = pmd_alloc(&init_mm, pud, addr); - if (!pmd) + pmdp = pmd_alloc(&init_mm, pudp, addr); + if (!pmdp) goto failed; } return; diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 5e39dcf23fdb..ce0cb8e51d0a 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -26,17 +26,17 @@ static pgd_t tmp_pg_dir[PTRS_PER_PGD] __page_aligned_bss; static p4d_t tmp_p4d[PTRS_PER_P4D] __page_aligned_bss; static pud_t tmp_pud[PTRS_PER_PUD] __page_aligned_bss; -static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned long end) +static void __init kasan_populate_pte(pmd_t *pmdp, unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; pte_t *ptep, *p; - if (pmd_none(*pmd)) { + if (pmd_none(*pmdp)) { p = memblock_alloc(PTRS_PER_PTE * sizeof(pte_t), PAGE_SIZE); - set_pmd(pmd, pfn_pmd(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_pmd(pmdp, pfn_pmd(PFN_DOWN(__pa(p)), PAGE_TABLE)); } - ptep = pte_offset_kernel(pmd, vaddr); + ptep = pte_offset_kernel(pmdp, vaddr); do { if (pte_none(*ptep)) { @@ -47,18 +47,18 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned } while (ptep++, vaddr += PAGE_SIZE, vaddr != end); } -static void __init kasan_populate_pmd(pud_t *pud, unsigned long vaddr, unsigned long end) +static void __init kasan_populate_pmd(pud_t *pudp, unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; pmd_t *pmdp, *p; unsigned long next; - if (pud_none(*pud)) { + if (pud_none(*pudp)) { p = memblock_alloc(PTRS_PER_PMD * sizeof(pmd_t), PAGE_SIZE); - set_pud(pud, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_pud(pudp, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); } - pmdp = pmd_offset(pud, vaddr); + pmdp = pmd_offset(pudp, vaddr); do { next = pmd_addr_end(vaddr, end); @@ -76,19 +76,19 @@ static void __init kasan_populate_pmd(pud_t *pud, unsigned long vaddr, unsigned } while (pmdp++, vaddr = next, vaddr != end); } -static void __init kasan_populate_pud(p4d_t *p4d, +static void __init kasan_populate_pud(p4d_t *p4dp, unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; pud_t *pudp, *p; unsigned long next; - if (p4d_none(*p4d)) { + if (p4d_none(*p4dp)) { p = memblock_alloc(PTRS_PER_PUD * sizeof(pud_t), PAGE_SIZE); - set_p4d(p4d, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_p4d(p4dp, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); } - pudp = pud_offset(p4d, vaddr); + pudp = pud_offset(p4dp, vaddr); do { next = pud_addr_end(vaddr, end); @@ -106,19 +106,19 @@ static void __init kasan_populate_pud(p4d_t *p4d, } while (pudp++, vaddr = next, vaddr != end); } -static void __init kasan_populate_p4d(pgd_t *pgd, +static void __init kasan_populate_p4d(pgd_t *pgdp, unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; p4d_t *p4dp, *p; unsigned long next; - if (pgd_none(*pgd)) { + if (pgd_none(*pgdp)) { p = memblock_alloc(PTRS_PER_P4D * sizeof(p4d_t), PAGE_SIZE); - set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_pgd(pgdp, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); } - p4dp = p4d_offset(pgd, vaddr); + p4dp = p4d_offset(pgdp, vaddr); do { next = p4d_addr_end(vaddr, end); @@ -162,14 +162,14 @@ static void __init kasan_populate_pgd(pgd_t *pgdp, static void __init kasan_early_clear_pud(p4d_t *p4dp, unsigned long vaddr, unsigned long end) { - pud_t *pudp, *base_pud; + pud_t *pudp, *base_pudp; unsigned long next; if (!pgtable_l4_enabled) { pudp = (pud_t *)p4dp; } else { - base_pud = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); - pudp = base_pud + pud_index(vaddr); + base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); + pudp = base_pudp + pud_index(vaddr); } do { @@ -187,14 +187,14 @@ static void __init kasan_early_clear_pud(p4d_t *p4dp, static void __init kasan_early_clear_p4d(pgd_t *pgdp, unsigned long vaddr, unsigned long end) { - p4d_t *p4dp, *base_p4d; + p4d_t *p4dp, *base_p4dp; unsigned long next; if (!pgtable_l5_enabled) { p4dp = (p4d_t *)pgdp; } else { - base_p4d = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); - p4dp = base_p4d + p4d_index(vaddr); + base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); + p4dp = base_p4dp + p4d_index(vaddr); } do { @@ -232,15 +232,15 @@ static void __init kasan_early_populate_pud(p4d_t *p4dp, unsigned long vaddr, unsigned long end) { - pud_t *pudp, *base_pud; + pud_t *pudp, *base_pudp; phys_addr_t phys_addr; unsigned long next; if (!pgtable_l4_enabled) { pudp = (pud_t *)p4dp; } else { - base_pud = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); - pudp = base_pud + pud_index(vaddr); + base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); + pudp = base_pudp + pud_index(vaddr); } do { @@ -261,7 +261,7 @@ static void __init kasan_early_populate_p4d(pgd_t *pgdp, unsigned long vaddr, unsigned long end) { - p4d_t *p4dp, *base_p4d; + p4d_t *p4dp, *base_p4dp; phys_addr_t phys_addr; unsigned long next; @@ -277,8 +277,8 @@ static void __init kasan_early_populate_p4d(pgd_t *pgdp, if (!pgtable_l5_enabled) { p4dp = (p4d_t *)pgdp; } else { - base_p4d = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); - p4dp = base_p4d + p4d_index(vaddr); + base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); + p4dp = base_p4dp + p4d_index(vaddr); } do { @@ -371,63 +371,63 @@ static void __init kasan_populate(void *start, void *end) kasan_populate_pgd(pgd_offset_k(vaddr), vaddr, vend); } -static void __init kasan_shallow_populate_pud(p4d_t *p4d, +static void __init kasan_shallow_populate_pud(p4d_t *p4dp, unsigned long vaddr, unsigned long end) { unsigned long next; void *p; - pud_t *pud_k = pud_offset(p4d, vaddr); + pud_t *pudp_k = pud_offset(p4dp, vaddr); do { next = pud_addr_end(vaddr, end); - if (pud_none(*pud_k)) { + if (pud_none(*pudp_k)) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - set_pud(pud_k, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_pud(pudp_k, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; } BUG(); - } while (pud_k++, vaddr = next, vaddr != end); + } while (pudp_k++, vaddr = next, vaddr != end); } -static void __init kasan_shallow_populate_p4d(pgd_t *pgd, +static void __init kasan_shallow_populate_p4d(pgd_t *pgdp, unsigned long vaddr, unsigned long end) { unsigned long next; void *p; - p4d_t *p4d_k = p4d_offset(pgd, vaddr); + p4d_t *p4dp_k = p4d_offset(pgdp, vaddr); do { next = p4d_addr_end(vaddr, end); - if (p4d_none(*p4d_k)) { + if (p4d_none(*p4dp_k)) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - set_p4d(p4d_k, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_p4d(p4dp_k, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; } - kasan_shallow_populate_pud(p4d_k, vaddr, end); - } while (p4d_k++, vaddr = next, vaddr != end); + kasan_shallow_populate_pud(p4dp_k, vaddr, end); + } while (p4dp_k++, vaddr = next, vaddr != end); } static void __init kasan_shallow_populate_pgd(unsigned long vaddr, unsigned long end) { unsigned long next; void *p; - pgd_t *pgd_k = pgd_offset_k(vaddr); + pgd_t *pgdp_k = pgd_offset_k(vaddr); do { next = pgd_addr_end(vaddr, end); - if (pgd_none(*pgd_k)) { + if (pgd_none(*pgdp_k)) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - set_pgd(pgd_k, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); + set_pgd(pgdp_k, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; } - kasan_shallow_populate_p4d(pgd_k, vaddr, next); - } while (pgd_k++, vaddr = next, vaddr != end); + kasan_shallow_populate_p4d(pgdp_k, vaddr, next); + } while (pgdp_k++, vaddr = next, vaddr != end); } static void __init kasan_shallow_populate(void *start, void *end) @@ -441,7 +441,7 @@ static void __init kasan_shallow_populate(void *start, void *end) static void __init create_tmp_mapping(void) { void *ptr; - p4d_t *base_p4d; + p4d_t *base_p4dp; /* * We need to clean the early mapping: this is hard to achieve "in-place", @@ -455,16 +455,16 @@ static void __init create_tmp_mapping(void) memcpy(tmp_p4d, ptr, sizeof(p4d_t) * PTRS_PER_P4D); set_pgd(&tmp_pg_dir[pgd_index(KASAN_SHADOW_END)], pfn_pgd(PFN_DOWN(__pa(tmp_p4d)), PAGE_TABLE)); - base_p4d = tmp_p4d; + base_p4dp = tmp_p4d; } else { - base_p4d = (p4d_t *)tmp_pg_dir; + base_p4dp = (p4d_t *)tmp_pg_dir; } /* Copy the last pud since it is shared with the kernel mapping. */ if (pgtable_l4_enabled) { ptr = (pud_t *)p4d_page_vaddr(*(base_p4d + p4d_index(KASAN_SHADOW_END))); memcpy(tmp_pud, ptr, sizeof(pud_t) * PTRS_PER_PUD); - set_p4d(&base_p4d[p4d_index(KASAN_SHADOW_END)], + set_p4d(&base_p4dp[p4d_index(KASAN_SHADOW_END)], pfn_p4d(PFN_DOWN(__pa(tmp_pud)), PAGE_TABLE)); } } diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 161d0b34c2cb..ffca6f19dd9c 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -25,65 +25,65 @@ static unsigned long set_pageattr_masks(unsigned long val, struct mm_walk *walk) return new_val; } -static int pageattr_pgd_entry(pgd_t *pgd, unsigned long addr, +static int pageattr_pgd_entry(pgd_t *pgdp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pgd_t val = READ_ONCE(*pgd); + pgd_t val = READ_ONCE(*pgdp); if (pgd_leaf(val)) { val = __pgd(set_pageattr_masks(pgd_val(val), walk)); - set_pgd(pgd, val); + set_pgd(pgdp, val); } return 0; } -static int pageattr_p4d_entry(p4d_t *p4d, unsigned long addr, +static int pageattr_p4d_entry(p4d_t *p4dp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - p4d_t val = READ_ONCE(*p4d); + p4d_t val = READ_ONCE(*p4dp); if (p4d_leaf(val)) { val = __p4d(set_pageattr_masks(p4d_val(val), walk)); - set_p4d(p4d, val); + set_p4d(p4dp, val); } return 0; } -static int pageattr_pud_entry(pud_t *pud, unsigned long addr, +static int pageattr_pud_entry(pud_t *pudp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pud_t val = READ_ONCE(*pud); + pud_t val = READ_ONCE(*pudp); if (pud_leaf(val)) { val = __pud(set_pageattr_masks(pud_val(val), walk)); - set_pud(pud, val); + set_pud(pudp, val); } return 0; } -static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr, +static int pageattr_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pmd_t val = READ_ONCE(*pmd); + pmd_t val = READ_ONCE(*pmdp); if (pmd_leaf(val)) { val = __pmd(set_pageattr_masks(pmd_val(val), walk)); - set_pmd(pmd, val); + set_pmd(pmdp, val); } return 0; } -static int pageattr_pte_entry(pte_t *pte, unsigned long addr, +static int pageattr_pte_entry(pte_t *ptep, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val = READ_ONCE(*pte); + pte_t val = READ_ONCE(*ptep); val = __pte(set_pageattr_masks(pte_val(val), walk)); - set_pte(pte, val); + set_pte(ptep, val); return 0; } @@ -209,36 +209,40 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) bool kernel_page_present(struct page *page) { unsigned long addr = (unsigned long)page_address(page); - pgd_t *pgd; - pud_t *pud; - p4d_t *p4d; - pmd_t *pmd; - pte_t *pte; - - pgd = pgd_offset_k(addr); - if (!pgd_present(*pgd)) + pgd_t *pgdp, pgd; + pud_t *pudp, pud; + p4d_t *p4dp, p4d; + pmd_t *pmdp, pmd; + pte_t *ptep; + + pgdp = pgd_offset_k(addr); + pgd = *pgdp; + if (!pgd_present(pgd)) return false; - if (pgd_leaf(*pgd)) + if (pgd_leaf(pgd)) return true; - p4d = p4d_offset(pgd, addr); - if (!p4d_present(*p4d)) + p4dp = p4d_offset(pgdp, addr); + p4d = *p4dp; + if (!p4d_present(p4d)) return false; - if (p4d_leaf(*p4d)) + if (p4d_leaf(p4d)) return true; - pud = pud_offset(p4d, addr); - if (!pud_present(*pud)) + pudp = pud_offset(p4dp, addr); + pud = *pudp; + if (!pud_present(pud)) return false; - if (pud_leaf(*pud)) + if (pud_leaf(pud)) return true; - pmd = pmd_offset(pud, addr); - if (!pmd_present(*pmd)) + pmdp = pmd_offset(pudp, addr); + pmd = *pmdp; + if (!pmd_present(pmd)) return false; - if (pmd_leaf(*pmd)) + if (pmd_leaf(pmd)) return true; - pte = pte_offset_kernel(pmd, addr); - return pte_present(*pte); + ptep = pte_offset_kernel(pmdp, addr); + return pte_present(*ptep); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index fef4e7328e49..9c93f24d0829 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -6,77 +6,77 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot) +int p4d_set_huge(p4d_t *p4dp, phys_addr_t addr, pgprot_t prot) { return 0; } -void p4d_clear_huge(p4d_t *p4d) +void p4d_clear_huge(p4d_t *p4dp) { } -int pud_set_huge(pud_t *pud, phys_addr_t phys, pgprot_t prot) +int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { pud_t new_pud = pfn_pud(__phys_to_pfn(phys), prot); - set_pud(pud, new_pud); + set_pud(pudp, new_pud); return 1; } -int pud_clear_huge(pud_t *pud) +int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pud))) + if (!pud_leaf(READ_ONCE(*pudp))) return 0; - pud_clear(pud); + pud_clear(pudp); return 1; } -int pud_free_pmd_page(pud_t *pud, unsigned long addr) +int pud_free_pmd_page(pud_t *pudp, unsigned long addr) { - pmd_t *pmd = pud_pgtable(*pud); + pmd_t *pmdp = pud_pgtable(*pudp); int i; - pud_clear(pud); + pud_clear(pudp); flush_tlb_kernel_range(addr, addr + PUD_SIZE); for (i = 0; i < PTRS_PER_PMD; i++) { - if (!pmd_none(pmd[i])) { - pte_t *pte = (pte_t *)pmd_page_vaddr(pmd[i]); + if (!pmd_none(pmdp[i])) { + pte_t *ptep = (pte_t *)pmd_page_vaddr(pmdp[i]); - pte_free_kernel(NULL, pte); + pte_free_kernel(NULL, ptep); } } - pmd_free(NULL, pmd); + pmd_free(NULL, pmdp); return 1; } -int pmd_set_huge(pmd_t *pmd, phys_addr_t phys, pgprot_t prot) +int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) { pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), prot); - set_pmd(pmd, new_pmd); + set_pmd(pmdp, new_pmd); return 1; } -int pmd_clear_huge(pmd_t *pmd) +int pmd_clear_huge(pmd_t *pmdp) { - if (!pmd_leaf(READ_ONCE(*pmd))) + if (!pmd_leaf(READ_ONCE(*pmdp))) return 0; - pmd_clear(pmd); + pmd_clear(pmdp); return 1; } -int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) +int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { - pte_t *pte = (pte_t *)pmd_page_vaddr(*pmd); + pte_t *ptep = (pte_t *)pmd_page_vaddr(*pmdp); - pmd_clear(pmd); + pmd_clear(pmdp); flush_tlb_kernel_range(addr, addr + PMD_SIZE); - pte_free_kernel(NULL, pte); + pte_free_kernel(NULL, ptep); return 1; } From patchwork Mon Oct 2 15:10:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13406364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E81CE784BD for ; Mon, 2 Oct 2023 15:15:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1407C8D002E; Mon, 2 Oct 2023 11:15:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F1378D000E; Mon, 2 Oct 2023 11:15:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAC2B8D002E; Mon, 2 Oct 2023 11:15:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DA0048D000E for ; Mon, 2 Oct 2023 11:15:48 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A6AE34031F for ; Mon, 2 Oct 2023 15:15:48 +0000 (UTC) X-FDA: 81300871176.29.6CC5195 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by imf28.hostedemail.com (Postfix) with ESMTP id 75D73C001C for ; Mon, 2 Oct 2023 15:15:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jac2kfiE; dmarc=none; spf=pass (imf28.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696259746; a=rsa-sha256; cv=none; b=vi1gc0xI0fgaSibzWmh/040++X89xKaRsK9em5aAxHF9YXfqpabfbMFL85Bt6DgD558667 eQCjOvKNcuPXinTiyivYKuyyhh/UJxFxDg1umOD0OdK4uzVuP8c+rFgKv4TqKK/Vxz38qQ jOALPSKKqtGym+i/QyJ2FSVGChdV3os= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=jac2kfiE; dmarc=none; spf=pass (imf28.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696259746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OEEG+xtUAsULDYONgax1Tf3ExxOI3c/UAKYhQT00Brk=; b=FUNOhxyzb/RM+bOGjbV6u8kXDZX8E6tk8cRM9hlHGGRR6SbrWsY6egbYHD6PukaU81nvYL 1TjfH5c5/vf6iwY0pXf4h89reEAx+mKwI508/erPzm7mLL9exv7bni3AHYPJQeXdjfhUaP sxuWjaUBsG+GE6CQbtawFUsIB5gJtFY= Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-40652e570d9so44792175e9.1 for ; Mon, 02 Oct 2023 08:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1696259745; x=1696864545; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OEEG+xtUAsULDYONgax1Tf3ExxOI3c/UAKYhQT00Brk=; b=jac2kfiErEc0qp+brPqsDxdHwxUxHtmbsMfcsS+lszALlnokCUBEaGo2vxN2LvKqiN pRXuUXv47qKBUqbpWjkur4xoomd4wBG2caWl53/E0GDEsLylqHH+gKH4WEmW1QHdwXSg kKtBVb5XxYQ41veJBXij5JvevDgHOk02ZbYvwX3cYq5GHp8xWBJs2p92tQhTrkbXj/Nz iUP5WzEDBXRYgFSupdnu1MBKgfKDGXQIn4HRbkZ/kOtDG2LuXcxNElcGUO/r92qoxlJA 6feu438m/ggIfG3/r8rcI8KL5QzwtWzKBV5iAUNBuzIEaamRGI45Ua/NgsEsPmExUqvj 13AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696259745; x=1696864545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OEEG+xtUAsULDYONgax1Tf3ExxOI3c/UAKYhQT00Brk=; b=p9FCCB2nNfCV59QWGE2hYflR/hhIJQFYMF51EpfoLF7NsWn5Of3RMASqPn+6iGOuEW z1y8+WdPkv6Vk24P8Id5A7uaEm4nnsJ0aN3ptGizP0rf6Bk2fM+cw9KWNol236sqLBnP 4ZTIidMw8JQHqRvbXnqBt383DE6J4m6oGyInCcP7GCfLWwwHYNBqR4U0FQxTMZ6rOgew SMYDUTMeRG40iR3vj7lxSjfr/eW+SwE9O/ken3lwut9xYsg1xzi5w9PB2whyivnF+uBc RGpHlNU8MVaznKL7/0842l4EDF3lLqSusCJYPEWY3U4Z62TdTRiIiR+KRsifuXN+P0ms Yx5g== X-Gm-Message-State: AOJu0YxpwHlV6cs9GFQk45aosTSSF7PICGQYpWBjYrUqaD75KOYyvdUi tFiB3TRWrhmuSE+eMJdPpUS33w== X-Google-Smtp-Source: AGHT+IFSmhkeyv71Y8ZW6SZ/8AIDQxUouTsHv+LksJFX3OOVwpF8GS+7Ljl1N8uJ0lmVIkcHQqystA== X-Received: by 2002:a05:600c:470e:b0:406:7029:c4f2 with SMTP id v14-20020a05600c470e00b004067029c4f2mr3333134wmo.26.1696259744708; Mon, 02 Oct 2023 08:15:44 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id q9-20020a7bce89000000b003fefcbe7fa8sm7451252wmj.28.2023.10.02.08.15.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Oct 2023 08:15:44 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Anup Patel , Atish Patra , Ard Biesheuvel , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 5/5] riscv: Use accessors to page table entries instead of direct dereference Date: Mon, 2 Oct 2023 17:10:31 +0200 Message-Id: <20231002151031.110551-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231002151031.110551-1-alexghiti@rivosinc.com> References: <20231002151031.110551-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 75D73C001C X-Stat-Signature: oqj96fobqmbcq314mooq44hcbrzs5ie7 X-HE-Tag: 1696259746-337223 X-HE-Meta: U2FsdGVkX18urnF+IOW2gH2qovcH7sgPCWQbIcNfzb0k5XaLKzLpJqvSVeJbMgEZbDaOC9s1TC/FL3N+/63MHSWfWTTlW5D93/+RtjpmyXOu8aofHy7Yh0bXHAS5cqgBcYvUjTaWyu3xNm651GdF4bZkzVFi7R+U2DU5XjBlJwXRpm2oawYNzL/hzTDltFPg1IJEM0JkwY0sgOwzBEJZQwBCdct8eDAOMqKr2YyHfvPI7pFc5sYyU0MSdurB6jE0LL6IUzY185szcamdv+0xpTKYQtVGdt4uJHwMp9U4OWsBjEZ2x++NWx4TiCVF9DLdVdxOdGVs6nqJDcav55j/RbKJ999Rw7jMk/DPC4hDU9sm86YdvuwYRGDOKllJv3/kYF6jfZGT0AYR0CqKYnw3R1fTdNeunTa1oz2we+FdV5u6xew9FZ5JjSCCdlOlGIp27HXaMMsPhysR5ZLwH9k1nHkBlDjDiEjKOeeiGONV4BCXDNyHWn5goswlQqM+hV/ApKRgO38ijqpUJlmrn0h1EmZjvYJVR6rIuPgw1hg3XMtAfO2MRD9AJ7/IgVXe8KwkmjVUhV9kHvQpqFiprj3JqsahFb44tfFOP6K6ybRtFEEk+izkqiiPCuhMj87SkLSuiBeIpF/ICEqZoJ13FbgkMNcXIR4K+rM3+VlrheKWzOJmffCXorMkHkQH+XZCgoPbI5KQKm/fR9X114DJA7G399KwLoSF+/7tymXuq1zWRrJv2XiBXAe8ieS6nFDE8haDsuVJYHEJmS6VRMHr/wGWIFMtbGAZhkr/ShUPeeceN114Giwxrg9OT1NRetAbP5XSFPVW2aUHxRSVFAU1eUs/ICG1q43Li16Jyc3Yai1u71+nSbkuiSXlEZdrjmxEaYldiSEPdNlxABMfSzdnhlokxhM8u8jpnHtzytl7TmPFOJrjr/aFK1pqrXJeD+Dwuk56kadhIVb1Zi5H2FjPb2Y MbeIbN0r 3XTZr/RKTehhtq3kFTi8ZTRJ+Qj+49ssFVAP9XqAmBUb6KW9FGQT9WN58fZEcJDcrf2UqWcijtw/J9KsdnMPuz73z9gX5X3WklfhcukayZxt6bpbN/b8Q5SDqQ6NZqvYEThCN6OyvmT2+RhKkzEUt8mki2p15FtyHt7a/HGGh/bOS49mXRZQ8w/fATr1Ss6PJW6zQh7DAGjjepAA1WupNIvC5eJE5Vt5P+u1uxnl0njQrF+EdVaq5eN59wmFVRrePQEHcH2JDt2iaLMYB/uI+7bFx/zZJgFqUFGckqr4QmJM/xrF0Tkpj7W2odlUvzPbEs/6MXVrJ+UP+B+3MR+Uh3/70vVT/QyyoTeVC3O8nIuXFQS/TztJrfQnzrfJD0CZc7CFxS0xyE20nIqUb7B8gkZOxztlLXIeOqBbBtpr5ICWleFqJenEvMkW7dxgZKu1r5f0LuMlNLFli5VzOrgjIjtA57h8TZ+eKzSTi7y4QjckgTtg9SdoNE2M2eOYFtfMYUfNf/LovV18Ku10= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As very well explained in commit 20a004e7b017 ("arm64: mm: Use READ_ONCE/WRITE_ONCE when accessing page tables"), an architecture whose page table walker can modify the PTE in parallel must use READ_ONCE()/WRITE_ONCE() macro to avoid any compiler transformation. So apply that to riscv which is such architecture. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/kfence.h | 4 +-- arch/riscv/include/asm/pgtable.h | 29 +++++--------------- arch/riscv/kernel/efi.c | 2 +- arch/riscv/kvm/mmu.c | 22 ++++++++-------- arch/riscv/mm/fault.c | 16 ++++++------ arch/riscv/mm/hugetlbpage.c | 12 ++++----- arch/riscv/mm/kasan_init.c | 45 +++++++++++++++++--------------- arch/riscv/mm/pageattr.c | 20 +++++++------- arch/riscv/mm/pgtable.c | 33 +++++++++++++++++++---- 9 files changed, 96 insertions(+), 87 deletions(-) diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index 3b482d0a4633..1ed91e379723 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -18,9 +18,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) pte_t *ptep = virt_to_kpte(addr); if (protect) - set_pte(ptep, __pte(pte_val(*ptep) & ~_PAGE_PRESENT)); + set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_PRESENT)); else - set_pte(ptep, __pte(pte_val(*ptep) | _PAGE_PRESENT)); + set_pte(ptep, __pte(pte_val(ptep_get(ptep)) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b820775f4973..8d2370ca9909 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -543,19 +543,12 @@ static inline void pte_clear(struct mm_struct *mm, __set_pte_at(ptep, __pte(0)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -static inline int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) -{ - if (!pte_same(*ptep, entry)) - __set_pte_at(ptep, entry); - /* - * update_mmu_cache will unconditionally execute, handling both - * the case that the PTE changed and the spurious fault case. - */ - return true; -} +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */ +extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep, pte_t entry, int dirty); +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ +extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep); #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, @@ -568,16 +561,6 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, return pte; } -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) -{ - if (!pte_young(*ptep)) - return 0; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); -} - #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index aa6209a74c83..b64bf1624a05 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md) static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { efi_memory_desc_t *md = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); unsigned long val; if (md->attribute & EFI_MEMORY_RO) { diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 35db276bf0c2..8d249877f9d8 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -103,7 +103,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, *ptep_level = current_level; ptep = (pte_t *)kvm->arch.pgdp; ptep = &ptep[gstage_pte_index(addr, current_level)]; - while (ptep && pte_val(*ptep)) { + while (ptep && pte_val(ptep_get(ptep))) { if (gstage_pte_leaf(ptep)) { *ptep_level = current_level; *ptepp = ptep; @@ -113,7 +113,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, if (current_level) { current_level--; *ptep_level = current_level; - ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); + ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); ptep = &ptep[gstage_pte_index(addr, current_level)]; } else { ptep = NULL; @@ -149,25 +149,25 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, if (gstage_pte_leaf(ptep)) return -EEXIST; - if (!pte_val(*ptep)) { + if (!pte_val(ptep_get(ptep))) { if (!pcache) return -ENOMEM; next_ptep = kvm_mmu_memory_cache_alloc(pcache); if (!next_ptep) return -ENOMEM; - *ptep = pfn_pte(PFN_DOWN(__pa(next_ptep)), - __pgprot(_PAGE_TABLE)); + set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), + __pgprot(_PAGE_TABLE))); } else { if (gstage_pte_leaf(ptep)) return -EEXIST; - next_ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); + next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); } current_level--; ptep = &next_ptep[gstage_pte_index(addr, current_level)]; } - *ptep = *new_pte; + set_pte(ptep, *new_pte); if (gstage_pte_leaf(ptep)) gstage_remote_tlb_flush(kvm, current_level, addr); @@ -239,11 +239,11 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, BUG_ON(addr & (page_size - 1)); - if (!pte_val(*ptep)) + if (!pte_val(ptep_get(ptep))) return; if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep = (pte_t *)gstage_pte_page_vaddr(*ptep); + next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(next_ptep_level, &next_page_size); @@ -261,7 +261,7 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(*ptep) & ~_PAGE_WRITE)); + set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); gstage_remote_tlb_flush(kvm, ptep_level, addr); } } @@ -603,7 +603,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return pte_young(*ptep); + return pte_young(ptep_get(ptep)); } int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 6284ef4b644a..24c826746322 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -136,24 +136,24 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a pgdp = (pgd_t *)pfn_to_virt(pfn) + index; pgdp_k = init_mm.pgd + index; - if (!pgd_present(*pgdp_k)) { + if (!pgd_present(pgdp_get(pgdp_k))) { no_context(regs, addr); return; } - set_pgd(pgdp, *pgdp_k); + set_pgd(pgdp, pgdp_get(pgdp_k)); p4dp_k = p4d_offset(pgdp_k, addr); - if (!p4d_present(*p4dp_k)) { + if (!p4d_present(p4dp_get(p4dp_k))) { no_context(regs, addr); return; } pudp_k = pud_offset(p4dp_k, addr); - if (!pud_present(*pudp_k)) { + if (!pud_present(pudp_get(pudp_k))) { no_context(regs, addr); return; } - if (pud_leaf(*pudp_k)) + if (pud_leaf(pudp_get(pudp_k))) goto flush_tlb; /* @@ -161,11 +161,11 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * to copy individual PTEs */ pmdp_k = pmd_offset(pudp_k, addr); - if (!pmd_present(*pmdp_k)) { + if (!pmd_present(pmdp_get(pmdp_k))) { no_context(regs, addr); return; } - if (pmd_leaf(*pmdp_k)) + if (pmd_leaf(pmdp_get(pmdp_k))) goto flush_tlb; /* @@ -175,7 +175,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * silently loop forever. */ ptep_k = pte_offset_kernel(pmdp_k, addr); - if (!pte_present(*ptep_k)) { + if (!pte_present(ptep_get(ptep_k))) { no_context(regs, addr); return; } diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 7781e83b2f29..a9d8e5bbac27 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -54,7 +54,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, } if (sz == PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(*pudp)) + if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp))) ptep = huge_pmd_share(mm, vma, addr, pudp); else ptep = (pte_t *)pmd_alloc(mm, pudp, addr); @@ -93,11 +93,11 @@ pte_t *huge_pte_offset(struct mm_struct *mm, pmd_t *pmdp; pgdp = pgd_offset(mm, addr); - if (!pgd_present(*pgdp)) + if (!pgd_present(pgdp_get(pgdp))) return NULL; p4dp = p4d_offset(pgdp, addr); - if (!p4d_present(*p4dp)) + if (!p4d_present(p4dp_get(p4dp))) return NULL; pudp = pud_offset(p4dp, addr); @@ -105,7 +105,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, /* must be pud huge, non-present or none */ return (pte_t *)pudp; - if (!pud_present(*pudp)) + if (!pud_present(pudp_get(pudp))) return NULL; pmdp = pmd_offset(pudp, addr); @@ -113,7 +113,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, /* must be pmd huge, non-present or none */ return (pte_t *)pmdp; - if (!pmd_present(*pmdp)) + if (!pmd_present(pmdp_get(pmdp))) return NULL; for_each_napot_order(order) { @@ -285,7 +285,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); int i, pte_num; if (!pte_napot(pte)) { diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index ce0cb8e51d0a..aaadb7385e9f 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -31,7 +31,7 @@ static void __init kasan_populate_pte(pmd_t *pmdp, unsigned long vaddr, unsigned phys_addr_t phys_addr; pte_t *ptep, *p; - if (pmd_none(*pmdp)) { + if (pmd_none(pmdp_get(pmdp))) { p = memblock_alloc(PTRS_PER_PTE * sizeof(pte_t), PAGE_SIZE); set_pmd(pmdp, pfn_pmd(PFN_DOWN(__pa(p)), PAGE_TABLE)); } @@ -39,7 +39,7 @@ static void __init kasan_populate_pte(pmd_t *pmdp, unsigned long vaddr, unsigned ptep = pte_offset_kernel(pmdp, vaddr); do { - if (pte_none(*ptep)) { + if (pte_none(ptep_get(ptep))) { phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); @@ -53,7 +53,7 @@ static void __init kasan_populate_pmd(pud_t *pudp, unsigned long vaddr, unsigned pmd_t *pmdp, *p; unsigned long next; - if (pud_none(*pudp)) { + if (pud_none(pudp_get(pudp))) { p = memblock_alloc(PTRS_PER_PMD * sizeof(pmd_t), PAGE_SIZE); set_pud(pudp, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); } @@ -63,7 +63,8 @@ static void __init kasan_populate_pmd(pud_t *pudp, unsigned long vaddr, unsigned do { next = pmd_addr_end(vaddr, end); - if (pmd_none(*pmdp) && IS_ALIGNED(vaddr, PMD_SIZE) && (next - vaddr) >= PMD_SIZE) { + if (pmd_none(pmdp_get(pmdp)) && IS_ALIGNED(vaddr, PMD_SIZE) && + (next - vaddr) >= PMD_SIZE) { phys_addr = memblock_phys_alloc(PMD_SIZE, PMD_SIZE); if (phys_addr) { set_pmd(pmdp, pfn_pmd(PFN_DOWN(phys_addr), PAGE_KERNEL)); @@ -83,7 +84,7 @@ static void __init kasan_populate_pud(p4d_t *p4dp, pud_t *pudp, *p; unsigned long next; - if (p4d_none(*p4dp)) { + if (p4d_none(p4dp_get(p4dp))) { p = memblock_alloc(PTRS_PER_PUD * sizeof(pud_t), PAGE_SIZE); set_p4d(p4dp, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); } @@ -93,7 +94,8 @@ static void __init kasan_populate_pud(p4d_t *p4dp, do { next = pud_addr_end(vaddr, end); - if (pud_none(*pudp) && IS_ALIGNED(vaddr, PUD_SIZE) && (next - vaddr) >= PUD_SIZE) { + if (pud_none(pudp_get(pudp)) && IS_ALIGNED(vaddr, PUD_SIZE) && + (next - vaddr) >= PUD_SIZE) { phys_addr = memblock_phys_alloc(PUD_SIZE, PUD_SIZE); if (phys_addr) { set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_KERNEL)); @@ -113,7 +115,7 @@ static void __init kasan_populate_p4d(pgd_t *pgdp, p4d_t *p4dp, *p; unsigned long next; - if (pgd_none(*pgdp)) { + if (pgd_none(pgdp_get(pgdp))) { p = memblock_alloc(PTRS_PER_P4D * sizeof(p4d_t), PAGE_SIZE); set_pgd(pgdp, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); } @@ -123,7 +125,8 @@ static void __init kasan_populate_p4d(pgd_t *pgdp, do { next = p4d_addr_end(vaddr, end); - if (p4d_none(*p4dp) && IS_ALIGNED(vaddr, P4D_SIZE) && (next - vaddr) >= P4D_SIZE) { + if (p4d_none(p4dp_get(p4dp)) && IS_ALIGNED(vaddr, P4D_SIZE) && + (next - vaddr) >= P4D_SIZE) { phys_addr = memblock_phys_alloc(P4D_SIZE, P4D_SIZE); if (phys_addr) { set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_KERNEL)); @@ -145,7 +148,7 @@ static void __init kasan_populate_pgd(pgd_t *pgdp, do { next = pgd_addr_end(vaddr, end); - if (pgd_none(*pgdp) && IS_ALIGNED(vaddr, PGDIR_SIZE) && + if (pgd_none(pgdp_get(pgdp)) && IS_ALIGNED(vaddr, PGDIR_SIZE) && (next - vaddr) >= PGDIR_SIZE) { phys_addr = memblock_phys_alloc(PGDIR_SIZE, PGDIR_SIZE); if (phys_addr) { @@ -168,7 +171,7 @@ static void __init kasan_early_clear_pud(p4d_t *p4dp, if (!pgtable_l4_enabled) { pudp = (pud_t *)p4dp; } else { - base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); + base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(p4dp_get(p4dp)))); pudp = base_pudp + pud_index(vaddr); } @@ -193,7 +196,7 @@ static void __init kasan_early_clear_p4d(pgd_t *pgdp, if (!pgtable_l5_enabled) { p4dp = (p4d_t *)pgdp; } else { - base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); + base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(pgdp_get(pgdp)))); p4dp = base_p4dp + p4d_index(vaddr); } @@ -239,14 +242,14 @@ static void __init kasan_early_populate_pud(p4d_t *p4dp, if (!pgtable_l4_enabled) { pudp = (pud_t *)p4dp; } else { - base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); + base_pudp = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(p4dp_get(p4dp)))); pudp = base_pudp + pud_index(vaddr); } do { next = pud_addr_end(vaddr, end); - if (pud_none(*pudp) && IS_ALIGNED(vaddr, PUD_SIZE) && + if (pud_none(pudp_get(pudp)) && IS_ALIGNED(vaddr, PUD_SIZE) && (next - vaddr) >= PUD_SIZE) { phys_addr = __pa((uintptr_t)kasan_early_shadow_pmd); set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_TABLE)); @@ -277,14 +280,14 @@ static void __init kasan_early_populate_p4d(pgd_t *pgdp, if (!pgtable_l5_enabled) { p4dp = (p4d_t *)pgdp; } else { - base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); + base_p4dp = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(pgdp_get(pgdp)))); p4dp = base_p4dp + p4d_index(vaddr); } do { next = p4d_addr_end(vaddr, end); - if (p4d_none(*p4dp) && IS_ALIGNED(vaddr, P4D_SIZE) && + if (p4d_none(p4dp_get(p4dp)) && IS_ALIGNED(vaddr, P4D_SIZE) && (next - vaddr) >= P4D_SIZE) { phys_addr = __pa((uintptr_t)kasan_early_shadow_pud); set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_TABLE)); @@ -305,7 +308,7 @@ static void __init kasan_early_populate_pgd(pgd_t *pgdp, do { next = pgd_addr_end(vaddr, end); - if (pgd_none(*pgdp) && IS_ALIGNED(vaddr, PGDIR_SIZE) && + if (pgd_none(pgdp_get(pgdp)) && IS_ALIGNED(vaddr, PGDIR_SIZE) && (next - vaddr) >= PGDIR_SIZE) { phys_addr = __pa((uintptr_t)kasan_early_shadow_p4d); set_pgd(pgdp, pfn_pgd(PFN_DOWN(phys_addr), PAGE_TABLE)); @@ -381,7 +384,7 @@ static void __init kasan_shallow_populate_pud(p4d_t *p4dp, do { next = pud_addr_end(vaddr, end); - if (pud_none(*pudp_k)) { + if (pud_none(pudp_get(pudp_k))) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); set_pud(pudp_k, pfn_pud(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; @@ -401,7 +404,7 @@ static void __init kasan_shallow_populate_p4d(pgd_t *pgdp, do { next = p4d_addr_end(vaddr, end); - if (p4d_none(*p4dp_k)) { + if (p4d_none(p4dp_get(p4dp_k))) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); set_p4d(p4dp_k, pfn_p4d(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; @@ -420,7 +423,7 @@ static void __init kasan_shallow_populate_pgd(unsigned long vaddr, unsigned long do { next = pgd_addr_end(vaddr, end); - if (pgd_none(*pgdp_k)) { + if (pgd_none(pgdp_get(pgdp_k))) { p = memblock_alloc(PAGE_SIZE, PAGE_SIZE); set_pgd(pgdp_k, pfn_pgd(PFN_DOWN(__pa(p)), PAGE_TABLE)); continue; @@ -451,7 +454,7 @@ static void __init create_tmp_mapping(void) /* Copy the last p4d since it is shared with the kernel mapping. */ if (pgtable_l5_enabled) { - ptr = (p4d_t *)pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_END)); + ptr = (p4d_t *)pgd_page_vaddr(pgdp_get(pgd_offset_k(KASAN_SHADOW_END))); memcpy(tmp_p4d, ptr, sizeof(p4d_t) * PTRS_PER_P4D); set_pgd(&tmp_pg_dir[pgd_index(KASAN_SHADOW_END)], pfn_pgd(PFN_DOWN(__pa(tmp_p4d)), PAGE_TABLE)); @@ -462,7 +465,7 @@ static void __init create_tmp_mapping(void) /* Copy the last pud since it is shared with the kernel mapping. */ if (pgtable_l4_enabled) { - ptr = (pud_t *)p4d_page_vaddr(*(base_p4d + p4d_index(KASAN_SHADOW_END))); + ptr = (pud_t *)p4d_page_vaddr(p4dp_get(base_p4dp + p4d_index(KASAN_SHADOW_END))); memcpy(tmp_pud, ptr, sizeof(pud_t) * PTRS_PER_PUD); set_p4d(&base_p4dp[p4d_index(KASAN_SHADOW_END)], pfn_p4d(PFN_DOWN(__pa(tmp_pud)), PAGE_TABLE)); diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index ffca6f19dd9c..2198a8810811 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -28,7 +28,7 @@ static unsigned long set_pageattr_masks(unsigned long val, struct mm_walk *walk) static int pageattr_pgd_entry(pgd_t *pgdp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pgd_t val = READ_ONCE(*pgdp); + pgd_t val = pgdp_get(pgdp); if (pgd_leaf(val)) { val = __pgd(set_pageattr_masks(pgd_val(val), walk)); @@ -41,7 +41,7 @@ static int pageattr_pgd_entry(pgd_t *pgdp, unsigned long addr, static int pageattr_p4d_entry(p4d_t *p4dp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - p4d_t val = READ_ONCE(*p4dp); + p4d_t val = p4dp_get(p4dp); if (p4d_leaf(val)) { val = __p4d(set_pageattr_masks(p4d_val(val), walk)); @@ -54,7 +54,7 @@ static int pageattr_p4d_entry(p4d_t *p4dp, unsigned long addr, static int pageattr_pud_entry(pud_t *pudp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pud_t val = READ_ONCE(*pudp); + pud_t val = pudp_get(pudp); if (pud_leaf(val)) { val = __pud(set_pageattr_masks(pud_val(val), walk)); @@ -67,7 +67,7 @@ static int pageattr_pud_entry(pud_t *pudp, unsigned long addr, static int pageattr_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pmd_t val = READ_ONCE(*pmdp); + pmd_t val = pmdp_get(pmdp); if (pmd_leaf(val)) { val = __pmd(set_pageattr_masks(pmd_val(val), walk)); @@ -80,7 +80,7 @@ static int pageattr_pmd_entry(pmd_t *pmdp, unsigned long addr, static int pageattr_pte_entry(pte_t *ptep, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val = READ_ONCE(*ptep); + pte_t val = ptep_get(ptep); val = __pte(set_pageattr_masks(pte_val(val), walk)); set_pte(ptep, val); @@ -216,33 +216,33 @@ bool kernel_page_present(struct page *page) pte_t *ptep; pgdp = pgd_offset_k(addr); - pgd = *pgdp; + pgd = pgdp_get(pgdp); if (!pgd_present(pgd)) return false; if (pgd_leaf(pgd)) return true; p4dp = p4d_offset(pgdp, addr); - p4d = *p4dp; + p4d = p4dp_get(p4dp); if (!p4d_present(p4d)) return false; if (p4d_leaf(p4d)) return true; pudp = pud_offset(p4dp, addr); - pud = *pudp; + pud = pudp_get(pudp); if (!pud_present(pud)) return false; if (pud_leaf(pud)) return true; pmdp = pmd_offset(pudp, addr); - pmd = *pmdp; + pmd = pmdp_get(pmdp); if (!pmd_present(pmd)) return false; if (pmd_leaf(pmd)) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_present(*ptep); + return pte_present(ptep_get(ptep)); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 9c93f24d0829..777fcb116bb4 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,6 +5,29 @@ #include #include +int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) +{ + if (!pte_same(ptep_get(ptep), entry)) + __set_pte_at(ptep, entry); + /* + * update_mmu_cache will unconditionally execute, handling both + * the case that the PTE changed and the spurious fault case. + */ + return true; +} + +int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) +{ + if (!pte_young(ptep_get(ptep))) + return 0; + return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); +} +EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); + #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP int p4d_set_huge(p4d_t *p4dp, phys_addr_t addr, pgprot_t prot) { @@ -25,7 +48,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pudp))) + if (!pud_leaf(pudp_get(pudp))) return 0; pud_clear(pudp); return 1; @@ -33,7 +56,7 @@ int pud_clear_huge(pud_t *pudp) int pud_free_pmd_page(pud_t *pudp, unsigned long addr) { - pmd_t *pmdp = pud_pgtable(*pudp); + pmd_t *pmdp = pud_pgtable(pudp_get(pudp)); int i; pud_clear(pudp); @@ -63,7 +86,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) int pmd_clear_huge(pmd_t *pmdp) { - if (!pmd_leaf(READ_ONCE(*pmdp))) + if (!pmd_leaf(pmdp_get(pmdp))) return 0; pmd_clear(pmdp); return 1; @@ -71,7 +94,7 @@ int pmd_clear_huge(pmd_t *pmdp) int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { - pte_t *ptep = (pte_t *)pmd_page_vaddr(*pmdp); + pte_t *ptep = (pte_t *)pmd_page_vaddr(pmdp_get(pmdp)); pmd_clear(pmdp); @@ -88,7 +111,7 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp); VM_BUG_ON(address & ~HPAGE_PMD_MASK); - VM_BUG_ON(pmd_trans_huge(*pmdp)); + VM_BUG_ON(pmd_trans_huge(pmdp_get(pmdp))); /* * When leaf PTE entries (regular pages) are collapsed into a leaf * PMD entry (huge page), a valid non-leaf PTE is converted into a