From patchwork Fri Sep 18 10:18:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11784575 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43AE4618 for ; Fri, 18 Sep 2020 10:20:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E3F60208C3 for ; Fri, 18 Sep 2020 10:20:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="djqtBFcG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3F60208C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 518996B0072; Fri, 18 Sep 2020 06:20:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4C65D6B0073; Fri, 18 Sep 2020 06:20:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3658F8E0001; Fri, 18 Sep 2020 06:20:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1DC866B0072 for ; Fri, 18 Sep 2020 06:20:07 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E26B21EE6 for ; Fri, 18 Sep 2020 10:20:06 +0000 (UTC) X-FDA: 77275786812.12.field11_46018d82712a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id BC864180054E2 for ; Fri, 18 Sep 2020 10:20:06 +0000 (UTC) X-Spam-Summary: 1,0,0,4faae038fbc3f0e1,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:2:41:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2196:2199:2376:2393:2553:2559:2562:2693:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4385:4605:5007:6119:6261:6653:6742:7874:7903:9592:10004:11026:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14096:14394:21080:21433:21444:21451:21627:21990:30054:30070:30090,0,RBL:209.85.218.68:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201;04yfbgeobkxzgmyhpipkgrgbddatmoc7txqnb1xc734m1jn3xihmfki13sipdfi.jo68zw9homrihhd1w56e41ixdq65nt991y638kss4sjrnwesqu748zx5ztoignf.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: field11_46018d82712a X-Filterd-Recvd-Size: 9280 Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Fri, 18 Sep 2020 10:20:06 +0000 (UTC) Received: by mail-ej1-f68.google.com with SMTP id lo4so7355118ejb.8 for ; Fri, 18 Sep 2020 03:20:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yPgjhGonSvJl6H0PgY9R1AwxtPIq+u+Zb+CPVqm2BwY=; b=djqtBFcGU7t/2T3BM/9Z7k5g1M3+HR9InenhzlIAPeoMgaFfOtNhdr3XIgo9HHMQAB pXJASq5jMaLE/Xvt7CnkZZGjgDnzO+xuvNsk0daMVw+QztKJSeCbBq/Y18pIKTivGOLu u5K4WGHHscl3Hq8y1jYOUGpBUm8n9y5gvNZ56y1dxuirSBVS/1SvAxVH2i9CXh+UmDdJ GlUuf5SeCbjDvXwHn7pw4INxTvqTzZbl2eYXT0R5e8VgsLLHCuV73DLvASaTBWhbNq+q w5zCgDezE7tC0skBn9bPN9MvpMKmXsYNH81JV2CEk3DEbJ/uJjYB8kvbSqAISMg/eIrC gbJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yPgjhGonSvJl6H0PgY9R1AwxtPIq+u+Zb+CPVqm2BwY=; b=LE/Zk1DZaomgUKCMl7nCSkU5joxc9Gyn93mncFYCQe1wUTKBc5gQbof3TCb6zEFpG3 Ont56Ljxcl7XUYXIYCqcSMKBzSCPZBbts7AVLbl+jQ/uYR521/LOxKuzLBLRtpdEDGq6 dR6xsASVtaH2iGVKOrZCqau5FbXoCSLlADDcA0sX0TXKWrZtXA4lGBVzg9NOwQOuTGwo IxIs/x3Xv6Ul1gbhgwnrnCZ2s+LbUDl+QtDl7+sVQLxNA/ldmgFMxJbBG7nVXQ74WF+b 1Pxb2q6g+pyL+42/e4uP7PmgI4Hc+L4vQFAVwum6kvB0Q7RglQ6kIMm8BUgJIrj3Bq8P 6Xzw== X-Gm-Message-State: AOAM532aWMwgxI3wgTfbHNDaJodmjfd8tYMeoJ8O/uPwVOR0vS9H+IJG au352YsTrkdgfGwPOO98JC5D7A== X-Google-Smtp-Source: ABdhPJwK/QsZG8pvRHZ3j6TnIwleGLS8UjaXCak0d3fWwKFgKH0cN5vopwZ4qM7FIG2u1xQEmxIxqw== X-Received: by 2002:a17:906:8258:: with SMTP id f24mr34077841ejx.551.1600424405116; Fri, 18 Sep 2020 03:20:05 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id r16sm1940674edc.57.2020.09.18.03.20.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2020 03:20:04 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, fenghua.yu@intel.com, eric.auger@redhat.com, Jean-Philippe Brucker Subject: [PATCH v10 09/13] iommu/arm-smmu-v3: Seize private ASID Date: Fri, 18 Sep 2020 12:18:49 +0200 Message-Id: <20200918101852.582559-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200918101852.582559-1-jean-philippe@linaro.org> References: <20200918101852.582559-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Signed-off-by: Jean-Philippe Brucker --- v10: fix ASID limit, small comment update --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 ++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 35 +++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++------- 3 files changed, 57 insertions(+), 15 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 6b06a6f19604..90c08f156b43 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -678,6 +678,9 @@ struct arm_smmu_domain { extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, + struct arm_smmu_ctx_desc *cd); +void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 6c1113059632..ef3fcfa72187 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -10,10 +10,18 @@ #include "arm-smmu-v3.h" #include "../../io-pgtable-arm.h" +/* + * Check if the CPU ASID is available on the SMMU side. If a private context + * descriptor is using it, try to replace it. + */ static struct arm_smmu_ctx_desc * arm_smmu_share_asid(struct mm_struct *mm, u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&arm_smmu_asid_xa, asid); if (!cd) @@ -27,8 +35,31 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return cd; } - /* Ouch, ASID is already in use for a private cd. */ - return ERR_PTR(-EBUSY); + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + /* + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. + */ + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + xa_erase(&arm_smmu_asid_xa, asid); + return NULL; } __maybe_unused diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 19af27fd183b..e99ebdd4c841 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -872,6 +872,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -952,8 +963,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; } -static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, - int ssid, struct arm_smmu_ctx_desc *cd) +int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, + struct arm_smmu_ctx_desc *cd) { /* * This function handles the following cases: @@ -1609,15 +1620,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -1625,8 +1627,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); }