From patchwork Thu Jul 23 14:57:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11681179 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5C18E138A for ; Thu, 23 Jul 2020 15:09:12 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2EFCC20771 for ; Thu, 23 Jul 2020 15:09:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="kbLKt005"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="aCMH28yf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2EFCC20771 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=H8fqcH90WCz8hie8vRgv8EgxIY7hrvjOY7csvrN8k/Q=; b=kbLKt0059r7p8j5cliGIaYbul fOUdVzbar7t13HIIgeMFl3Q3kbMeP32uMSFpDob053OAhjqjsKF4WMUcSXqvqCXEaDyDDqv7wVigq ulx1uF+iSjcv/OKpL7h3oq0HF0jJEuNS+9Y4phOOvrCQdZiw7l/nAkNB21azfUmKaIfnhcbm4rj6O CAwVCOGztpd0ybLJp48jY4Rn+XbvOZx/6v5P9XefLHw4mSN8HvztH775QNwdibv3zFpKapPZeNLX4 bau87nGt41kJoqxD5/l80pfezjQwgn2S54drlK9FREeWC5UFogHdkoIjH9dgyKGFNMYlwzb457qyt OTLr2BC1g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jycoh-0002vX-0w; Thu, 23 Jul 2020 15:07:27 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jycnF-0002PV-FP for linux-arm-kernel@lists.infradead.org; Thu, 23 Jul 2020 15:06:09 +0000 Received: by mail-ej1-x641.google.com with SMTP id w9so6730991ejc.8 for ; Thu, 23 Jul 2020 08:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wskuJloDXQffCZslMedcsZLhDNM/RTcftR4W5M+lmWM=; b=aCMH28yfVT0dvw1jNUUZbraQg0/4QDSbSz+ipcvfIDhdbz2vRFp3gQyyZcGEeOpIOf Rk6ZnzcSzfP5U5s7YRqSVsQpUjLMeafohbmXpBJ7W3cP6eonBGltklz41+NaxrRh6KXR 8nNwD3m8wK5KJI5I9RTCs8hDN+qtGUb0sL89irJmOnt7TQswOSqjyvx+Wz0jNW/5DKde fBEmAZK5nmCq5IqFkb7+f6r0CSP5mi6nGaFF2fC/yjVrGHFemFsd/NxvuU7UB/IEY2QF 6M6jVyc3K+eUpLxJwuaNq2xpP+hdAOSlWiiFig33xtIJIKR1oyfLwAn2X87VVQSoEQqK xLlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wskuJloDXQffCZslMedcsZLhDNM/RTcftR4W5M+lmWM=; b=h99eNUuu2/idTrUcet1r4u/K1MqUNVDS+XufxbZ8HqQqlRxfAvnTUkPSy5aidWJPNH hOhX+jNa5BN4f2jm/R7AXs0+5tTvF+8d/Cxb4CYEIxzMU6GTnGlgBeLlCOk8xq5SkIYk ac+7vX4mIIfAGDrRxl+Trrrr3HHx+/002r0pL5IGACuNJf6eJB02JFBwzoXnTp5xakwK 3q1K4/VJLhHCye0tVhGVND2CijtXGGACWB8C7nIe+8wWXF3AVeJbJeIXT2RxIR0PUkdQ 6N5AMmYadDkOc+gqwFtFu1QJptmMAZecAxcWJIP27BXpwvz0F7sfaNNNGGyHcP1ZzLRU sytA== X-Gm-Message-State: AOAM533YmerJknl3gUKuAYNTGRs6zKJYA9r1YXFOsWyNNC9wBvcYfqgr 7DQX3tcxS7lD7BiXIys+RQpmVA== X-Google-Smtp-Source: ABdhPJyB1Vkh5KERl8YmjBS8VfddZR0PFejHbSzUL2vYjqtEO7CwYUzdDRG0iR2rh/YcuHmXTAF++A== X-Received: by 2002:a17:906:1747:: with SMTP id d7mr4752761eje.39.1595516756567; Thu, 23 Jul 2020 08:05:56 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id k15sm2145952eji.49.2020.07.23.08.05.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2020 08:05:56 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v9 09/13] iommu/arm-smmu-v3: Seize private ASID Date: Thu, 23 Jul 2020 16:57:21 +0200 Message-Id: <20200723145724.3014766-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200723145724.3014766-1-jean-philippe@linaro.org> References: <20200723145724.3014766-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200723_110557_606291_7B2F8F31 X-CRM114-Status: GOOD ( 23.23 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.h | 3 +++ drivers/iommu/arm-smmu-v3-sva.c | 36 +++++++++++++++++++++++++++++++-- drivers/iommu/arm-smmu-v3.c | 34 +++++++++++++++++++------------ 3 files changed, 58 insertions(+), 15 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.h b/drivers/iommu/arm-smmu-v3.h index 6b06a6f19604..90c08f156b43 100644 --- a/drivers/iommu/arm-smmu-v3.h +++ b/drivers/iommu/arm-smmu-v3.h @@ -678,6 +678,9 @@ struct arm_smmu_domain { extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, + struct arm_smmu_ctx_desc *cd); +void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/arm-smmu-v3-sva.c b/drivers/iommu/arm-smmu-v3-sva.c index 7c1541864688..d590c864bdf3 100644 --- a/drivers/iommu/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm-smmu-v3-sva.c @@ -10,10 +10,19 @@ #include "arm-smmu-v3.h" #include "io-pgtable-arm.h" +/* + * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from + * the private entry. Careful here, we may be modifying the context tables of + * another SMMU! + */ static struct arm_smmu_ctx_desc * arm_smmu_share_asid(struct mm_struct *mm, u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&arm_smmu_asid_xa, asid); if (!cd) @@ -27,8 +36,31 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return cd; } - /* Ouch, ASID is already in use for a private cd. */ - return ERR_PTR(-EBUSY); + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, + XA_LIMIT(1, 1 << smmu->asid_bits), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + /* + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. + */ + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + xa_erase(&arm_smmu_asid_xa, asid); + return NULL; } __maybe_unused diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 06f148a9ed82..4a47b977ed01 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -873,6 +873,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -953,8 +964,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; } -static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, - int ssid, struct arm_smmu_ctx_desc *cd) +int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, + struct arm_smmu_ctx_desc *cd) { /* * This function handles the following cases: @@ -1610,15 +1621,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -1626,8 +1628,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); }