From patchwork Tue Mar 18 03:59:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 14020271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86D74C282EC for ; Tue, 18 Mar 2025 03:59:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DBDC280004; Mon, 17 Mar 2025 23:59:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1892E280001; Mon, 17 Mar 2025 23:59:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02927280004; Mon, 17 Mar 2025 23:59:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D7AC2280001 for ; Mon, 17 Mar 2025 23:59:52 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7BD40C0164 for ; Tue, 18 Mar 2025 03:59:53 +0000 (UTC) X-FDA: 83233318266.09.577C5A6 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf28.hostedemail.com (Postfix) with ESMTP id 9B7ABC000E for ; Tue, 18 Mar 2025 03:59:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ctLLxDqy; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742270391; a=rsa-sha256; cv=none; b=IY9OSmbrpuotnRbP2ajwko7wFEm3RBhPNjSPq4OydZvJJGZewf5uxixAlBZg3DobYu8553 uu/qx5BJe67oIExqVz7g8dfpFOPkmLnQBPETnHCXhkNASP62yth6Gvni2EH+RwJnCftUlJ qzP7zuI7uVKxdo3J6vUbyw6VZ1JCEcc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ctLLxDqy; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742270391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4RzuTPLpNm7preMNxFl4olDUAY05FyWZH15sy7wmcR0=; b=UOsUz8RJxuCLK71KZBTMNel9Lds6j6rJJCUNm4FGlqbm1pbtZ2kZFkwKabAV2Nwrn8I2NK 8yfE2o/YPxIyx9iz+KcIi1mA/f9SDNbJ62OqMWJn4gTDpKVZ4vbE+820nvqX7fxY4+HiTF Q2oPzD7hD/OmjmvoZwD00CcMZ9iT4f0= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-22435603572so81710255ad.1 for ; Mon, 17 Mar 2025 20:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1742270390; x=1742875190; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4RzuTPLpNm7preMNxFl4olDUAY05FyWZH15sy7wmcR0=; b=ctLLxDqypUlVU0oUxl+iGQC58NTLMmJxKetw+BYw+JUGbhYJb8Z+PPJTakdotRRy9p DxmWJxEYq4k6VzEd+y9McEWt7lRzHES9abxSiR8SR17E36+bJFNn45YM8sXylCzCPCsS EuzgDCIvb7baDzXGpudnTyVW7OLeeLApsC0yQgAihKoscoEeeRe9yA7uUb1wU4xHQ9HG 3hZ1k9PfRXQZaCxVArQGDMQLNtrOVFktZrpRmadOgPeB9BpFLNWeD/O6Ab7TsB/ZeOjd NjvF3s3nHZLMfXqJarAa1+C19PVLnjZOaOstMi6cc08SfiCfjdMMfd4FUA0Y5i2beFBl Qrgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742270390; x=1742875190; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4RzuTPLpNm7preMNxFl4olDUAY05FyWZH15sy7wmcR0=; b=UYWHb4tUzscv16dggtVEj6Y0O70Bqv1lILmiAKNYJhGuMoJBbzwHizNGAfJm55XH7N Fd5r9yeYs4uoHCmsLCRXP5PAZPv8aUxoeZuWhgtymWVGTqDFeVKGUh8mWnan/uqqS+dZ e5f9syHakmGM/5/nE5W1NIbmXBC6U4u47xpDE+MjpaYzmv/95YCwGlT23gKKjIfy8Akg sZ3IHwVx/pq2hCIPFE24C7+hgNrylFvCiokvukdQRkL2LCmNoNPg4+VGqmO0Ih8S49ES r9w2LzBu5pB50rfJ59fxwt/siTtoMLAL/cMCRbUD/kLOEBL+DY1NmR9zT+vz7iRJNGr5 GR3Q== X-Forwarded-Encrypted: i=1; AJvYcCX5gGtz+xKZ2fhm2elh3AAhe6yzkzvnmP9JSAA8Yd3b2IhDYFfjN1mqYQEl+LIauGvoDi95clhXzw==@kvack.org X-Gm-Message-State: AOJu0Yyo0Q93TkjYRea6ZIzFMrAglQpI4SGKuvgkeS5oT2iaV1K9Asl6 wGwV37sNt/h9Llvb/he7wLN5PV217qPaOEiyEVrOnZelpHHTCVaPWq8snSrUqBs= X-Gm-Gg: ASbGncsjnIik4acR1YPtgACdbun34fEjRkZwOH7cuRIOBETWfWYE9zYLg3aGdrV1Ep7 3rTJTxioF/WfGAv85uozBS1WPaFp8ss7Qql6lIlS7kv+ot5JBxmz+fvb0Nzi3dUywlM8pPo1PIb e8OdCoNDESUpv2yj6RV6d2RHevFxcDpYPMNG/Xdch8IolPjVV4pHUS/9snmCLPUxl4jWBpHe6u5 PvWig9ITI/dmX0zKhWY7G1Tef0rWX94cHc8FTDCXEXzWoHdLTNCxnQ42Y2ORD4wsjMNQFFDHsWy +opmy6F2G0Y3bDh3u+9z/5r83itC+b4Vw1U0H/dan0pBXLouin8FSPxI4+pS/XgUEArLQVB3zam n0HRKHWoS01Sk2FGf9HYgxpqb5DQ= X-Google-Smtp-Source: AGHT+IFE368rAoYFIO0lZVoCVYA7ODibd0bQTYRx4G4UreI6PSfoC7ExcjehLoXI4QO5c2ehzPanCg== X-Received: by 2002:a17:902:e886:b0:211:e812:3948 with SMTP id d9443c01a7336-225e08597f2mr204021375ad.0.1742270390531; Mon, 17 Mar 2025 20:59:50 -0700 (PDT) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.55]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-225c6bd4b30sm83720135ad.235.2025.03.17.20.59.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 17 Mar 2025 20:59:49 -0700 (PDT) From: Xu Lu To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill.shutemov@linux.intel.com, tjeznach@rivosinc.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com Cc: lihangjing@bytedance.com, xieyongji@bytedance.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [PATCH RESEND v2 2/4] iommu/riscv: Use pte_t to represent page table entry Date: Tue, 18 Mar 2025 11:59:28 +0800 Message-Id: <20250318035930.11855-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250318035930.11855-1-luxu.kernel@bytedance.com> References: <20250318035930.11855-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9B7ABC000E X-Stat-Signature: y3etc8bn8d3b64596ygf8qkjkuuin617 X-Rspamd-Server: rspam06 X-HE-Tag: 1742270391-428783 X-HE-Meta: U2FsdGVkX1/MY7tHFNjYKI2mLPthbgjWjf1EbdSLRrVK2ZLxrHAm7KjlUIVyCae6M7aOxcQd4HejG4jQui3ObqAnFKXwVuqrha2VcKYwbZZ3OJlKWofco2DZy7o2trby53Tw8qiHVYl5Aac8kRcltZoHzRkHxKH4lP+0iElONs8P1+CT6p/JmW/uHUWz22WZi+IJxlH+SxFepi2FYfp8YvYVEINYe+Q3ce1ERkC8k2S4V9FAYnAkvYaJ+bEuQHCIXEDpki5aA6VgYdakEaweA7lnBS4UbtonOaq5DEgJcgdqBvLHAyaJ1fwR7Ijc1M66bKgrLRA7fEZzMBJbLp9bGSjZMvb4hZoIgiG/NLrfLCQ2ajT2VC5Dd+da0GEpmetosMgoC8NrD76D8GTnfujnnDsMcnxMywQtwuMiq/inYVgoFgHS9eQsOF0o7lJb0GLheahcYx1JuFJOKq+BSF8MjKgn6EW15hUSnnE1CzVu14x+270nbPRYdQjNqM2Y71Qx06BxSJLpkFu5rWPsLOMTVfyhqW2ucdq11XIlDxbnMTsGDeunRB/2VnOlMpGJOp3tsXqOoEhWS/BH4Z7wF2+DF1vlV6y1EwqLr1iqL2+4jX8iNjPVNu0GGuJro4O02u1tREpzAQmBfIYN/nzDF1GfTjn0mNu0N1MFf5drNZyvf0y6aFqvLcnby3WtBtc1F3blqzG8WgUtM10PWe2/TwIhhzCYt+CPuyuGWmCxSg1e9r1CvSUhwtvbMF0sh1aNj2kek0VOV6LI1SeTKy7iInkNNotLFWJO03SoTLKSRakba/zvxeg9qujEVbBE5QXsBIjrR78kyS4oBYwM7Cik+x39nLHU6F/3gqMGqDA7/eyEAzBDXuSDEwFvfxAVhNzkIBulBiEx8PkQxHtgu88W8dRcA77+YSWMgNdq8hHkzFt4yJQrGzLtVOwXbzhmGt5VecS/dmS+QAX/ZNiy5UbKfTB SHVKiqoI xfwdrgZQA8qgU3cIBo4aNyv1Wh7dsjYJNfFNAeiXTiJ0IiPdIvh3WRYsL5xQ0PzSiPkWfM/v3gXVKYQ4/plqBE0Peeeb3tO4WsJS+zA43jRN4aCcW6jR3NruWc8pp6iNcr7UVWrAtG+q1VDDgC6P8toRzB7axbyey8PiTgL6HmoQSR0JF8b5iMHCsfhCkz2JC5v1qnB4qKOYQmq5lBSzmpVdJP41dcYcLqxSr7XLoYtxtoxWR2em2oVvtwEIgQMHn1b3w2YxwHApxnlE+xcnE1g6WpFuXoJtJVp8GX/8gnG52r24mdzGZbB3IEQn2aS1gvMa+0AnGckqYEvck/ocuyh8ZYMQ2kaPa4JAJ9e4gj8YysZw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since RISC-V IOMMU has the same pte format and translation process with MMU as is specified in RISC-V Privileged specification, we use pte_t to represent IOMMU pte too to reuse existing pte operation functions. Signed-off-by: Xu Lu --- drivers/iommu/riscv/iommu.c | 79 ++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 40 deletions(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 8f049d4a0e2cb..3b0c934decd08 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -812,7 +812,7 @@ struct riscv_iommu_domain { bool amo_enabled; int numa_node; unsigned int pgd_mode; - unsigned long *pgd_root; + pte_t *pgd_root; }; #define iommu_domain_to_riscv(iommu_domain) \ @@ -1081,27 +1081,29 @@ static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain, #define PT_SHIFT (PAGE_SHIFT - ilog2(sizeof(pte_t))) -#define _io_pte_present(pte) ((pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) -#define _io_pte_leaf(pte) ((pte) & _PAGE_LEAF) -#define _io_pte_none(pte) ((pte) == 0) -#define _io_pte_entry(pn, prot) ((_PAGE_PFN_MASK & ((pn) << _PAGE_PFN_SHIFT)) | (prot)) +#define _io_pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) +#define _io_pte_leaf(pte) (pte_val(pte) & _PAGE_LEAF) +#define _io_pte_none(pte) (pte_val(pte) == 0) +#define _io_pte_entry(pn, prot) (__pte((_PAGE_PFN_MASK & ((pn) << _PAGE_PFN_SHIFT)) | (prot))) static void riscv_iommu_pte_free(struct riscv_iommu_domain *domain, - unsigned long pte, struct list_head *freelist) + pte_t pte, struct list_head *freelist) { - unsigned long *ptr; + pte_t *ptr; int i; if (!_io_pte_present(pte) || _io_pte_leaf(pte)) return; - ptr = (unsigned long *)pfn_to_virt(__page_val_to_pfn(pte)); + ptr = (pte_t *)pfn_to_virt(pte_pfn(pte)); /* Recursively free all sub page table pages */ for (i = 0; i < PTRS_PER_PTE; i++) { - pte = READ_ONCE(ptr[i]); - if (!_io_pte_none(pte) && cmpxchg_relaxed(ptr + i, pte, 0) == pte) + pte = ptr[i]; + if (!_io_pte_none(pte)) { + ptr[i] = __pte(0); riscv_iommu_pte_free(domain, pte, freelist); + } } if (freelist) @@ -1110,12 +1112,12 @@ static void riscv_iommu_pte_free(struct riscv_iommu_domain *domain, iommu_free_page(ptr); } -static unsigned long *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, - unsigned long iova, size_t pgsize, - gfp_t gfp) +static pte_t *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, + unsigned long iova, size_t pgsize, + gfp_t gfp) { - unsigned long *ptr = domain->pgd_root; - unsigned long pte, old; + pte_t *ptr = domain->pgd_root; + pte_t pte, old; int level = domain->pgd_mode - RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 + 2; void *addr; @@ -1131,7 +1133,7 @@ static unsigned long *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, if (((size_t)1 << shift) == pgsize) return ptr; pte_retry: - pte = READ_ONCE(*ptr); + pte = ptep_get(ptr); /* * This is very likely incorrect as we should not be adding * new mapping with smaller granularity on top @@ -1147,38 +1149,37 @@ static unsigned long *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, addr = iommu_alloc_page_node(domain->numa_node, gfp); if (!addr) return NULL; - old = pte; - pte = _io_pte_entry(virt_to_pfn(addr), _PAGE_TABLE); - if (cmpxchg_relaxed(ptr, old, pte) != old) { - iommu_free_page(addr); + old = ptep_get(ptr); + if (!_io_pte_none(old)) goto pte_retry; - } + pte = _io_pte_entry(virt_to_pfn(addr), _PAGE_TABLE); + set_pte(ptr, pte); } - ptr = (unsigned long *)pfn_to_virt(__page_val_to_pfn(pte)); + ptr = (pte_t *)pfn_to_virt(pte_pfn(pte)); } while (level-- > 0); return NULL; } -static unsigned long *riscv_iommu_pte_fetch(struct riscv_iommu_domain *domain, - unsigned long iova, size_t *pte_pgsize) +static pte_t *riscv_iommu_pte_fetch(struct riscv_iommu_domain *domain, + unsigned long iova, size_t *pte_pgsize) { - unsigned long *ptr = domain->pgd_root; - unsigned long pte; + pte_t *ptr = domain->pgd_root; + pte_t pte; int level = domain->pgd_mode - RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 + 2; do { const int shift = PAGE_SHIFT + PT_SHIFT * level; ptr += ((iova >> shift) & (PTRS_PER_PTE - 1)); - pte = READ_ONCE(*ptr); + pte = ptep_get(ptr); if (_io_pte_present(pte) && _io_pte_leaf(pte)) { *pte_pgsize = (size_t)1 << shift; return ptr; } if (_io_pte_none(pte)) return NULL; - ptr = (unsigned long *)pfn_to_virt(__page_val_to_pfn(pte)); + ptr = (pte_t *)pfn_to_virt(pte_pfn(pte)); } while (level-- > 0); return NULL; @@ -1191,8 +1192,9 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); size_t size = 0; - unsigned long *ptr; - unsigned long pte, old, pte_prot; + pte_t *ptr; + pte_t pte, old; + unsigned long pte_prot; int rc = 0; LIST_HEAD(freelist); @@ -1210,10 +1212,9 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, break; } - old = READ_ONCE(*ptr); + old = ptep_get(ptr); pte = _io_pte_entry(phys_to_pfn(phys), pte_prot); - if (cmpxchg_relaxed(ptr, old, pte) != old) - continue; + set_pte(ptr, pte); riscv_iommu_pte_free(domain, old, &freelist); @@ -1247,7 +1248,7 @@ static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); size_t size = pgcount << __ffs(pgsize); - unsigned long *ptr, old; + pte_t *ptr; size_t unmapped = 0; size_t pte_size; @@ -1260,9 +1261,7 @@ static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, if (iova & (pte_size - 1)) return unmapped; - old = READ_ONCE(*ptr); - if (cmpxchg_relaxed(ptr, old, 0) != old) - continue; + set_pte(ptr, __pte(0)); iommu_iotlb_gather_add_page(&domain->domain, gather, iova, pte_size); @@ -1279,13 +1278,13 @@ static phys_addr_t riscv_iommu_iova_to_phys(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); size_t pte_size; - unsigned long *ptr; + pte_t *ptr; ptr = riscv_iommu_pte_fetch(domain, iova, &pte_size); - if (_io_pte_none(*ptr) || !_io_pte_present(*ptr)) + if (_io_pte_none(ptep_get(ptr)) || !_io_pte_present(ptep_get(ptr))) return 0; - return pfn_to_phys(__page_val_to_pfn(*ptr)) | (iova & (pte_size - 1)); + return pfn_to_phys(pte_pfn(ptep_get(ptr))) | (iova & (pte_size - 1)); } static void riscv_iommu_free_paging_domain(struct iommu_domain *iommu_domain)