From patchwork Mon Aug 17 17:15:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11718921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D19D713A4 for ; Mon, 17 Aug 2020 17:22:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 978DB206FA for ; Mon, 17 Aug 2020 17:22:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TxgpckBr"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HfxhdiAy"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="uyIRteLM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 978DB206FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=85uvwjcdveyXRu/GQGjyU6HOrg9Y6ze4nG41JvLBBG4=; b=TxgpckBr+PcjtjfamjaPvvHKU CkhB6BhhP/PZga1PZ4YzBB+CKNXzXGsO5lQ5FI8gqYw2C2keKPGz4UkPEB5lOl8rDgBhK8ElMi1vw 1LpHrfOFNc1RONVnkYgV0k+f7A1DmV8+C1WG4uv8Ra7MPD+crAN3IlCBDKs4p0e+/hKaYQpHxU0oT u53stiJXEAcqpPJTPUduEoq/gTl+fupEjDdNuwHJiS6ekUqvjBUKZ542LkrBCg2fTU7kwcfvEcjN5 ut+l9dnlhO9FtTYD46GZtOc+HNA+UaDJta6zWQem2QQTwqC3pZ1ZAzVKISYSWHvKcZQKFpPtkLoD0 2lehrcoqw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7ioJ-0003AN-8g; Mon, 17 Aug 2020 17:20:39 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7iml-0002Wx-CX for linux-arm-kernel@merlin.infradead.org; Mon, 17 Aug 2020 17:19:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3aWOpCZA/Y+YQtAVagpaBA4fiMS5nd1SZGxvlnabr5U=; b=HfxhdiAytunqsyDXbtdXgsRZrx eBQIa+9KGelECcivPEnwDmfK6bbExYpcUtHOfVIX7tXE/ZG4H/2xMv0WlAUKntKbr1D/ldTXVpnXC aBe1FVuQjEIKrM88XKE2fofU6YYWszcKhMlk7bZQu1mJeQwjF5ZTr4oXQIFAEQSQdX0EHrUW1KuZo c6Anwlg1gnuAu8L3MdOsUBs6kkWbFJ4KciAuVUfZJkwiwdqTDjcnNQrMN5tue/evIT0FGab6pnvUj TQZOW2OBkXL/RqBlJ35/DwbOa34FSWWDvxIk3FPrMiKapVtk+ttNc0XLq/1Nwemwfv9aVhXI+NcZW Mfg+3loQ==; Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k7imS-0008GV-7C for linux-arm-kernel@lists.infradead.org; Mon, 17 Aug 2020 17:19:02 +0000 Received: by mail-ej1-x641.google.com with SMTP id kq25so18658217ejb.3 for ; Mon, 17 Aug 2020 10:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3aWOpCZA/Y+YQtAVagpaBA4fiMS5nd1SZGxvlnabr5U=; b=uyIRteLMq0Wrqq71EnbDuPuH50m5MibwIwVKId5tn73lXo1zgQA6RXKAfOAkHz/j0v qQfZgYci1ylDYQQ6BR3jbas4YIn2FSjoO3blTI3touLapIksEq17f2rDKblWi4XxhwnI vS5NzTbfrZOH748ujjuRWo/BGBEl0TRtWskCK/3blzcpd5wHmAzerg3criTz2YxbcXeb EdSGrnyZodZucSqmXSwFkET9X/pgGUK0APXIs13NAZiswsVZZzzTLQVcWfQVSHmL28rW Ge/t0qj+Qe16/WbHyDlzBCkkAJfZ/+CeiCUz+DZwycZdgD9vvionS3dDuNRVehmmVR68 ptew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3aWOpCZA/Y+YQtAVagpaBA4fiMS5nd1SZGxvlnabr5U=; b=NYTYmJlljHMNdK9hSmLMO9stblD36ajnXDl6M5/dQak9tCjer++8zgueCAQo97Lt7L ueif48zdpWngsE0JYgExrbbnVhagyHn0bQCfTSa1bKkh4dSSXF1dMZk4FKOqphrBn+98 oGdv7u/v27tkYZIDIdEbw4m1JfOUXjDmmRtpbIeggmhGZL/DH1Oo7vTBZefr34qUrVgr HXtBcorNbQV5qGGVMiApvTUt8NdQZ6eYx8HBuwF7p7nj+io6Om57dU4gk0kN3uK420fV 30E2M566AlS+Sk/F9Eub5o7hOcRcyvQKMmTV5tHRjZLTZbguTsOFYzcYH7PiK/UaiFcV JU1g== X-Gm-Message-State: AOAM533Qq73V/RP8gmVaklZ/m2Swn5qXcM3gK91HEqV02BHbsCYY6p4w MyOfgOmNI4XaUlNIgn6Q1SwtnA== X-Google-Smtp-Source: ABdhPJwq8yudrMb+Gat31EQu6+nRCgs3dpMMdBJdTBkVTiTF3hu+bXLOJvsORD7OrV8S1wItkGz/Gw== X-Received: by 2002:a17:906:c2c8:: with SMTP id ch8mr15507619ejb.161.1597684722697; Mon, 17 Aug 2020 10:18:42 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id lc18sm14672502ejb.29.2020.08.17.10.18.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2020 10:18:42 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH RESEND v9 12/13] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Date: Mon, 17 Aug 2020 19:15:58 +0200 Message-Id: <20200817171558.325917-13-jean-philippe@linaro.org> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200817171558.325917-1-jean-philippe@linaro.org> References: <20200817171558.325917-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200817_181848_661976_531C4D66 X-CRM114-Status: GOOD ( 27.56 ) X-Spam-Score: -2.1 (--) X-Spam-Report: SpamAssassin version 3.4.4 on casper.infradead.org summary: Content analysis details: (-2.1 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The sva_bind() function allows devices to access process address spaces using a PASID (aka SSID). (1) bind() allocates or gets an existing MMU notifier tied to the (domain, mm) pair. Each mm gets one PASID. (2) Any change to the address space calls invalidate_range() which sends ATC invalidations (in a subsequent patch). (3) When the process address space dies, the release() notifier disables the CD to allow reclaiming the page tables. Since release() has to be light we do not instruct device drivers to stop DMA here, we just ignore incoming page faults from this point onwards. To avoid any event 0x0a print (C_BAD_CD) we disable translation without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. Don't clear the R bit because the S bit can't be cleared when STALL_MODEL==0b10 (forced), and clearing R without clearing S is useless. Faulting transactions will stall and will be aborted by the IOPF handler. (4) After stopping DMA, the device driver releases the bond by calling unbind(). We release the MMU notifier, free the PASID and the bond. Three structures keep track of bonds: * arm_smmu_bond: one per {device, mm} pair, the handle returned to the device driver for a bind() request. * arm_smmu_mmu_notifier: one per {domain, mm} pair, deals with ATS/TLB invalidations and clearing the context descriptor on mm exit. * arm_smmu_ctx_desc: one per mm, holds the pinned ASID and pgd. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/Kconfig | 2 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 28 +++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 230 +++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 32 ++- 4 files changed, 282 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index b1d592cd9984..a8eb8b7f35f7 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -316,6 +316,8 @@ config ARM_SMMU_V3 config ARM_SMMU_V3_SVA bool "Shared Virtual Addressing support for the ARM SMMUv3" depends on ARM_SMMU_V3 + select IOMMU_SVA_LIB + select MMU_NOTIFIER help Support for sharing process address spaces with devices using the SMMUv3. diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index ba34914813ff..6365c81a4614 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -677,10 +677,18 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; + + struct list_head mmu_notifiers; }; +static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) +{ + return container_of(dom, struct arm_smmu_domain, domain); +} + extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +extern struct arm_smmu_ctx_desc quiet_cd; int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, struct arm_smmu_ctx_desc *cd); @@ -693,6 +701,11 @@ bool arm_smmu_master_sva_supported(struct arm_smmu_master *master); bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master); int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); +struct iommu_sva *arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, + void *drvdata); +void arm_smmu_sva_unbind(struct iommu_sva *handle); +int arm_smmu_sva_get_pasid(struct iommu_sva *handle); +void arm_smmu_sva_notifier_synchronize(void); #else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { @@ -718,5 +731,20 @@ static inline int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { return -ENODEV; } + +static inline struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + return ERR_PTR(-ENODEV); +} + +static inline void arm_smmu_sva_unbind(struct iommu_sva *handle) {} + +static inline int arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + return IOMMU_PASID_INVALID; +} + +static inline void arm_smmu_sva_notifier_synchronize(void) {} #endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 28027620cf2e..fb9bfda2d192 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -5,11 +5,35 @@ #include #include +#include #include #include "arm-smmu-v3.h" +#include "../../iommu-sva-lib.h" #include "../../io-pgtable-arm.h" +struct arm_smmu_mmu_notifier { + struct mmu_notifier mn; + struct arm_smmu_ctx_desc *cd; + bool cleared; + refcount_t refs; + struct list_head list; + struct arm_smmu_domain *domain; +}; + +#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) + +struct arm_smmu_bond { + struct iommu_sva sva; + struct mm_struct *mm; + struct arm_smmu_mmu_notifier *smmu_mn; + struct list_head list; + refcount_t refs; +}; + +#define sva_to_bond(handle) \ + container_of(handle, struct arm_smmu_bond, sva) + static DEFINE_MUTEX(sva_lock); /* @@ -65,7 +89,6 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return NULL; } -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; @@ -146,7 +169,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return err < 0 ? ERR_PTR(err) : ret; } -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { if (arm_smmu_free_asid(cd)) { @@ -156,6 +178,201 @@ static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) } } +static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + mutex_lock(&sva_lock); + if (smmu_mn->cleared) { + mutex_unlock(&sva_lock); + return; + } + + /* + * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, + * but disable translation. + */ + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &quiet_cd); + + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + + smmu_mn->cleared = true; + mutex_unlock(&sva_lock); +} + +static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) +{ + kfree(mn_to_smmu(mn)); +} + +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, +}; + +/* Allocate or get existing MMU notifier for this {domain, mm} pair */ +static struct arm_smmu_mmu_notifier * +arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + int ret; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_mmu_notifier *smmu_mn; + + list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { + if (smmu_mn->mn.mm == mm) { + refcount_inc(&smmu_mn->refs); + return smmu_mn; + } + } + + cd = arm_smmu_alloc_shared_cd(mm); + if (IS_ERR(cd)) + return ERR_CAST(cd); + + smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); + if (!smmu_mn) { + ret = -ENOMEM; + goto err_free_cd; + } + + refcount_set(&smmu_mn->refs, 1); + smmu_mn->cd = cd; + smmu_mn->domain = smmu_domain; + smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; + + ret = mmu_notifier_register(&smmu_mn->mn, mm); + if (ret) { + kfree(smmu_mn); + goto err_free_cd; + } + + ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + if (ret) + goto err_put_notifier; + + list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); + return smmu_mn; + +err_put_notifier: + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); +err_free_cd: + arm_smmu_free_shared_cd(cd); + return ERR_PTR(ret); +} + +static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +{ + struct mm_struct *mm = smmu_mn->mn.mm; + struct arm_smmu_ctx_desc *cd = smmu_mn->cd; + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + if (!refcount_dec_and_test(&smmu_mn->refs)) + return; + + list_del(&smmu_mn->list); + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + + /* + * If we went through clear(), we've already invalidated, and no + * new TLB entry can have been formed. + */ + if (!smmu_mn->cleared) + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); + arm_smmu_free_shared_cd(cd); +} + +static struct iommu_sva * +__arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +{ + int ret; + struct arm_smmu_bond *bond; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (!master || !master->sva_enabled) + return ERR_PTR(-ENODEV); + + /* If bind() was already called for this {dev, mm} pair, reuse it. */ + list_for_each_entry(bond, &master->bonds, list) { + if (bond->mm == mm) { + refcount_inc(&bond->refs); + return &bond->sva; + } + } + + bond = kzalloc(sizeof(*bond), GFP_KERNEL); + if (!bond) + return ERR_PTR(-ENOMEM); + + /* Allocate a PASID for this mm if necessary */ + ret = iommu_sva_alloc_pasid(mm, 1, (1U << master->ssid_bits) - 1); + if (ret) + goto err_free_bond; + + bond->mm = mm; + bond->sva.dev = dev; + refcount_set(&bond->refs, 1); + + bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + if (IS_ERR(bond->smmu_mn)) { + ret = PTR_ERR(bond->smmu_mn); + goto err_free_pasid; + } + + list_add(&bond->list, &master->bonds); + return &bond->sva; + +err_free_pasid: + iommu_sva_free_pasid(mm); +err_free_bond: + kfree(bond); + return ERR_PTR(ret); +} + +struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + struct iommu_sva *handle; + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + mutex_lock(&sva_lock); + handle = __arm_smmu_sva_bind(dev, mm); + mutex_unlock(&sva_lock); + return handle; +} + +void arm_smmu_sva_unbind(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + mutex_lock(&sva_lock); + if (refcount_dec_and_test(&bond->refs)) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + iommu_sva_free_pasid(bond->mm); + kfree(bond); + } + mutex_unlock(&sva_lock); +} + +int arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + return bond->mm->pasid; +} + bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { unsigned long reg, fld; @@ -245,3 +462,12 @@ int arm_smmu_master_disable_sva(struct arm_smmu_master *master) return 0; } + +void arm_smmu_sva_notifier_synchronize(void) +{ + /* + * Some MMU notifiers may still be waiting to be freed, using + * arm_smmu_mmu_notifier_free(). Wait for them. + */ + mmu_notifier_synchronize(); +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5ed5bb42298f..ae93cdd09eff 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -71,6 +71,12 @@ struct arm_smmu_option_prop { DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); DEFINE_MUTEX(arm_smmu_asid_lock); +/* + * Special value used by SVA when a process dies, to quiesce a CD without + * disabling it. + */ +struct arm_smmu_ctx_desc quiet_cd = { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -86,11 +92,6 @@ static inline void __iomem *arm_smmu_page1_fixup(unsigned long offset, return smmu->base + offset; } -static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) -{ - return container_of(dom, struct arm_smmu_domain, domain); -} - static void parse_driver_options(struct arm_smmu_device *smmu) { int i = 0; @@ -974,7 +975,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -991,8 +994,10 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val = 0; + } else if (cd == &quiet_cd) { /* (4) */ + val |= CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1785,6 +1790,7 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; } @@ -2576,6 +2582,9 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, + .sva_bind = arm_smmu_sva_bind, + .sva_unbind = arm_smmu_sva_unbind, + .sva_get_pasid = arm_smmu_sva_get_pasid, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -3595,6 +3604,12 @@ static const struct of_device_id arm_smmu_of_match[] = { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + arm_smmu_sva_notifier_synchronize(); + platform_driver_unregister(drv); +} + static struct platform_driver arm_smmu_driver = { .driver = { .name = "arm-smmu-v3", @@ -3605,7 +3620,8 @@ static struct platform_driver arm_smmu_driver = { .remove = arm_smmu_device_remove, .shutdown = arm_smmu_device_shutdown, }; -module_platform_driver(arm_smmu_driver); +module_driver(arm_smmu_driver, platform_driver_register, + arm_smmu_driver_unregister); MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon ");