From patchwork Tue Mar 18 03:59:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 14020273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07F39C35FF3 for ; Tue, 18 Mar 2025 04:00:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AC47280006; Tue, 18 Mar 2025 00:00:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75D0B280001; Tue, 18 Mar 2025 00:00:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FB5C280006; Tue, 18 Mar 2025 00:00:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3FC69280001 for ; Tue, 18 Mar 2025 00:00:03 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CAE0B56171 for ; Tue, 18 Mar 2025 04:00:03 +0000 (UTC) X-FDA: 83233318686.10.0451C9D Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf11.hostedemail.com (Postfix) with ESMTP id 1920340006 for ; Tue, 18 Mar 2025 04:00:01 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YvAWIJL2; spf=pass (imf11.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742270402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=25EEGwiFn3VnbkwivdTLYPKqZQgF1BeYiMnWI1gQipU=; b=CGGL1f00FUwtgodTWesFwjOXVbTOpe1X30I1n892R4onafe4jxkxkV9u1NeGQBtCVnTYrJ KEmJEwpLVNMG+KGkgIilplERU6zpVBbuv3jYYkKRIyypdNc/WAYR9H6Zym4V8VYCTXQENc fg29LbwjqiA/T6t9R9RdbQ/+YBROUZw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742270402; a=rsa-sha256; cv=none; b=gnHz3c/2tyfwuG41eGy+vfFP1PO3IkJJsItmhrR2IvCpU0+RbLzFyfpK/j7qZQVD/SGSQU FSL9hhw3uDgiVtc1IBLkhncIIJJqT91uoZiZ2Ky+SmKuzi5KNb7QbZL6+DRdM13bADmonc EP+KNXPAPPirBNQk5uyQcyP7nJbUWTU= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YvAWIJL2; spf=pass (imf11.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-22403cbb47fso93737845ad.0 for ; Mon, 17 Mar 2025 21:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1742270401; x=1742875201; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=25EEGwiFn3VnbkwivdTLYPKqZQgF1BeYiMnWI1gQipU=; b=YvAWIJL2ZsEYwb81p7yeE5GNz/SbGNY90K4ePPh6PZkznETckXJgZlACx4JPe3LlQN wz8+sHlVmX7qt0SbyWRe/L/LHe23UYQy5KqGPfKuJ67AXNc4rehANlHlr/QFZIVJHzNo cwDv9u9D2UqJoIkaGyh/+LaF9k+HHq7Mt8vcNXei11nVb4+Dl8JQwoSX62ICTrF6CWoI NP/yPSyHax/+yROl4TEsCjEe8ZCcyR41VCqx/VWv9MxJwint/aBtE+PKgxi5KdbO8IaY zAjPaQeFx9ug8TZnKv5cBb/aCX/7NUSNPSVoej7S7tr6rJ3n4N10tdQCnX1ZnN/GypB7 YXPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742270401; x=1742875201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=25EEGwiFn3VnbkwivdTLYPKqZQgF1BeYiMnWI1gQipU=; b=HgfYN73tZlrW4NXMgOECSPTPM0FD8QBmLtN1X2iwIxIgAZ9OflP6BhmOovqh9dWxmB usIBLc5e11dKQSVtAggW/mCTMpO56alxJ90xLIfBOhKNQ0TvAQNgBKlltRthxXxjIlmv m3bKpaLTvAKPoDQK4cTvYyblO/JlAZ7jD3lj+n0hOXlmAsrwgOGyecJV92GuFliKAi75 vocvBM/+QVQkWMn+khyPpcyscksb2hdMAuc4o6tTySvUdJortdAFuXyF3dQRfXW+Xpts h3HU4bYM0EZFNEGCMjqxUs+wNgOmaJHJBGEr1PyJpRoyZ7hJI+E2rVSydWEU9j0XIMD7 Jgfg== X-Forwarded-Encrypted: i=1; AJvYcCXulWf9xu5k5w2vw+a5W7+9us4CZhXK90a3K7KIK7TPwkq92oWapNuzRW732no9/j43b+6w5TS1Ow==@kvack.org X-Gm-Message-State: AOJu0YzXRE+U5M3C0gfNmeZQFk4uSgUZUAg5YDw21+/LFKvmbpNyif8/ /YE3iPAFFeO1RmTP4L8mnbSHBJjPTsKnDN8duWyGHbKqBYUdSKek/fh68t/ou9cx/DbgZc1EH0I GtPA= X-Gm-Gg: ASbGncvq1hl7d/qxBe/1zQylGmrj5TN/rHcnwnMuotijOg9limdegqrpTRgRnLkicPR TjfcLaGeO9K8SOdEcW+Fs7O0qL3AvHANCS9pkpmMzkZvFVVibTH/vDm7PpfvGVPZD4uZ+B7GFoR K9nqxt3GocgO8e8l/ervUl7R3wFtINI20/MRX3qHOw7TYNtarpd7geds0bwMs1r0mRBk6d9UZvz 5Ed1TvLI+RmE6Qb2j6XalFkGmvUn6sXAVr6wY1Loy7f/gAjuur/8tmTSbgD4cmm/x1j4Vk4x9ze uHVLvWi9RJuud6zog6Ce4xMmkQwirbCINWOqdak0OCTxcCrOsrTgp6qBFGaMUheW9gNHD6M7Lck CBEJ+gLfrYA7ZfvPIU3yvq0Ty1qHQCN6z4JmKmA== X-Google-Smtp-Source: AGHT+IFBc6TvuWKKSrWPTLSErVQ1GkoHiMOoKLD8NwCVFkbU9sjHFxonJru7yaMaEU5aIspYftiubw== X-Received: by 2002:a17:903:230f:b0:223:5e54:c521 with SMTP id d9443c01a7336-225e0859fafmr199239715ad.0.1742270400974; Mon, 17 Mar 2025 21:00:00 -0700 (PDT) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.55]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-225c6bd4b30sm83720135ad.235.2025.03.17.20.59.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 17 Mar 2025 20:59:59 -0700 (PDT) From: Xu Lu To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill.shutemov@linux.intel.com, tjeznach@rivosinc.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com Cc: lihangjing@bytedance.com, xieyongji@bytedance.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [PATCH RESEND v2 4/4] iommu/riscv: Add support for Svnapot Date: Tue, 18 Mar 2025 11:59:30 +0800 Message-Id: <20250318035930.11855-5-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250318035930.11855-1-luxu.kernel@bytedance.com> References: <20250318035930.11855-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 1920340006 X-Stat-Signature: f6coitboanxgky4inp94buzjf6h98yki X-HE-Tag: 1742270401-352659 X-HE-Meta: U2FsdGVkX19AHOdJGQ4ZUtcjPGfCmfVT3jxgDWaYTeK9j6gnUkLLPOmsED8xwJhdcsvYv+p4XjKilrbD24x+nmf1neQJUabfFJz/IuWEJIh/QTZb9BEQCDiLWFxVB/E12DmuJUskjejGpDF9HjN+OZ/H7wHimtrs0ncaJmiSRqgAirdzB/Ca2w4G0WQS/yW1b6AHDbSnfSVhjuNUfe/5yMJlvTetWUyU656fav2dwpNaSHvvymPi7Qj/G0aWEjQU42CNMHBKlgkBFdef7y+CjXubNTABRPvcmQvYuZYcbLDoOirSMNQghXmj8teHx0l8lqthGoK0aky+f+Q0FTCJQC+T96/SWPoN849b1ciaFIX29AQ4o8fUeLxQe57kMIIzhgXdaGv5tfKS7t9V4ct7aYkKZWvo3Ye8K0/0FfUexbGuBlGkVlCW4OvUpa770poAIqfst449a6NeMT8MWo0Zc6KIzCeTmZUkOt7MyHL41V9c1wWL8Z/5hxg0VIJRuLMJJ7jjRerPuOVHQQ9kZugt1ia27FWZ8fGRw/a9QMUSWCgKya88pa2IDNskXAtOi1rrCfv/3WGKtegPCiNFjoeYnY1tVTVolWq70R8tAKeP5OofOUIhbuFdaOIYFcOw8+XiU6SwI/RlHflyAFAhp+fzUmQ5Eiq+6626VOX8Tzr0w3vuOgcPFw2GIshiSDzS/xEATaWU4BZiqSpFHDRGX4r5ZgKCxmFcRb/fZGtj44f4mFYTSgX8iIQ7TWf0n7YbWEUAfjnEcfOKgC4lkYN58wXb3pZOB0qIdpZPgcQWIs4C38Tc8jsmdwHJjhwjotAjcr7LEvnwGTbdET3o44i/iHdkYbiCXZ82uE/Z/GF2H1nVE/CP+JhL/WqF+QpbP3f24AwujcR3Mwr3ULLtzCPePMIyG1XVGAEgSDH0+AFMjzhWCySgeESP+mFmUk3eNkOtmA3UM+Pcvdki2j3Wvy51Sku E08nr4uk Q3+ZxQsU0HhQHlO8y5896xp6soq8thdDogLpN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add Svnapot size as supported page size and apply Svnapot when it is possible. Signed-off-by: Xu Lu --- drivers/iommu/riscv/iommu.c | 86 +++++++++++++++++++++++++++++++++---- 1 file changed, 77 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index ce4cf6569ffb4..7cc736abd2a61 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -1158,6 +1158,26 @@ static int pgsize_to_level(size_t pgsize) return level; } +static unsigned long napot_size_to_order(unsigned long size) +{ + unsigned long order; + + if (!has_svnapot()) + return 0; + + for_each_napot_order(order) { + if (size == napot_cont_size(order)) + return order; + } + + return 0; +} + +static bool is_napot_size(unsigned long size) +{ + return napot_size_to_order(size) != 0; +} + static void riscv_iommu_pte_free(struct riscv_iommu_domain *domain, pte_t pte, int level, struct list_head *freelist) @@ -1205,7 +1225,8 @@ static pte_t *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, * existing mapping with smaller granularity. Up to the caller * to replace and invalidate. */ - if (((size_t)1 << shift) == pgsize) + if ((((size_t)1 << shift) == pgsize) || + (is_napot_size(pgsize) && pgsize_to_level(pgsize) == level)) return ptr; pte_retry: pte = ptep_get(ptr); @@ -1256,7 +1277,10 @@ static pte_t *riscv_iommu_pte_fetch(struct riscv_iommu_domain *domain, ptr += ((iova >> shift) & (PTRS_PER_PTE - 1)); pte = ptep_get(ptr); if (_io_pte_present(pte) && _io_pte_leaf(pte)) { - *pte_pgsize = (size_t)1 << shift; + if (pte_napot(pte)) + *pte_pgsize = napot_cont_size(napot_cont_order(pte)); + else + *pte_pgsize = (size_t)1 << shift; return ptr; } if (_io_pte_none(pte)) @@ -1274,13 +1298,18 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); size_t size = 0; - pte_t *ptr; - pte_t pte; - unsigned long pte_prot; - int rc = 0, level; + pte_t *ptr, old, pte; + unsigned long pte_prot, order = 0; + int rc = 0, level, i; spinlock_t *ptl; /* page table page lock */ LIST_HEAD(freelist); + if (iova & (pgsize - 1)) + return -EINVAL; + + if (is_napot_size(pgsize)) + order = napot_size_to_order(pgsize); + if (!(prot & IOMMU_WRITE)) pte_prot = _PAGE_BASE | _PAGE_READ; else if (domain->amo_enabled) @@ -1297,9 +1326,27 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, level = pgsize_to_level(pgsize); ptl = riscv_iommu_ptlock(domain, ptr, level); - riscv_iommu_pte_free(domain, ptep_get(ptr), level, &freelist); + + old = ptep_get(ptr); + if (pte_napot(old) && napot_cont_size(napot_cont_order(old)) > pgsize) { + spin_unlock(ptl); + rc = -EFAULT; + break; + } + pte = _io_pte_entry(phys_to_pfn(phys), pte_prot); - set_pte(ptr, pte); + if (order) { + pte = pte_mknapot(pte, order); + for (i = 0; i < napot_pte_num(order); i++, ptr++) { + old = ptep_get(ptr); + riscv_iommu_pte_free(domain, old, level, &freelist); + set_pte(ptr, pte); + } + } else { + riscv_iommu_pte_free(domain, old, level, &freelist); + set_pte(ptr, pte); + } + spin_unlock(ptl); size += pgsize; @@ -1336,6 +1383,9 @@ static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, size_t unmapped = 0; size_t pte_size; spinlock_t *ptl; /* page table page lock */ + unsigned long pte_num; + pte_t pte; + int i; while (unmapped < size) { ptr = riscv_iommu_pte_fetch(domain, iova, &pte_size); @@ -1347,7 +1397,21 @@ static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, return unmapped; ptl = riscv_iommu_ptlock(domain, ptr, pgsize_to_level(pte_size)); - set_pte(ptr, __pte(0)); + if (is_napot_size(pte_size)) { + pte = ptep_get(ptr); + + if (!pte_napot(pte) || + napot_cont_size(napot_cont_order(pte)) != pte_size) { + spin_unlock(ptl); + return unmapped; + } + + pte_num = napot_pte_num(napot_cont_order(pte)); + for (i = 0; i < pte_num; i++, ptr++) + set_pte(ptr, __pte(0)); + } else { + set_pte(ptr, __pte(0)); + } spin_unlock(ptl); iommu_iotlb_gather_add_page(&domain->domain, gather, iova, @@ -1447,6 +1511,7 @@ static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev) unsigned int pgd_mode; dma_addr_t va_mask; int va_bits, level; + size_t order; iommu = dev_to_iommu(dev); if (iommu->caps & RISCV_IOMMU_CAPABILITIES_SV57) { @@ -1506,6 +1571,9 @@ static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev) domain->domain.geometry.aperture_end = va_mask; domain->domain.geometry.force_aperture = true; domain->domain.pgsize_bitmap = va_mask & (SZ_4K | SZ_2M | SZ_1G | SZ_512G); + if (has_svnapot()) + for_each_napot_order(order) + domain->domain.pgsize_bitmap |= napot_cont_size(order) & va_mask; domain->domain.ops = &riscv_iommu_paging_domain_ops;