From patchwork Fri Nov 6 15:50:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11887305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32D5DC2D0A3 for ; Fri, 6 Nov 2020 15:54:02 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B036622203 for ; Fri, 6 Nov 2020 15:54:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="sa07d40l"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="wDH2bv+2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B036622203 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=I3FMF2AWEm9ObmkZbjw+S2LEDMakbsyhBbOhD/4eGFQ=; b=sa07d40lMYycQEeFMGswsXk3o tV+bQG+tPOrzsypKQu6IBBzOESuMMDR71uFQWrKTM3J5O4lISZVCzf9mn9/Pdf5RkT6PrwHsfpCG0 Sp0nw0f/919URlKxHHgmZbbo9PPScZr3Nqc+1Pe5TU4f9CVUhR+Tt6IQTrwzEHISVO9vEBQ1twKu/ hmGUThmYdfC8HA0LrbcFjvXPVMTgE3Rp6JFxriCx5iqEo1OXvZGrNtlrDaSHD91w+oSkawpmeYdLt on2j3ILAxPOGfBf3Q9qUCnHNaZ5TKPazQ1DzpsMVEY5m8PH485jpH4kSUpC6rGnI+VkI7qtDbzliP Oq++r+xwg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43I-00071x-0W; Fri, 06 Nov 2020 15:53:24 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43B-0006zO-F5 for linux-arm-kernel@lists.infradead.org; Fri, 06 Nov 2020 15:53:18 +0000 Received: by mail-ej1-x641.google.com with SMTP id o9so2603781ejg.1 for ; Fri, 06 Nov 2020 07:53:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3FJUppGtFWGOx0FMlu9BmhAJNre7egam/1ZLuXa6iwY=; b=wDH2bv+2aNvcIs+dcbitD6EJlLk+LPCIealwWV0NZCUQaHyBwE3dTdrE9HUEx8FXEF pIFUaI8G9zA5yVwtPxMen7nF6o2nfrOZU5WO9Xu2rXWB3ZsjCI2CIFPGSquZaSQzoZ5X Ftc1ji7Aob2IfRv4IPyBnampxe3A1XJfhCbakQNK/Yn64bPi9JUtOZiLDShE+f5xC47D aAjrwqLVIEM1zrVlXrZSVYWqfNFnUUtThhx/xutxu4UAHoFyMomdBMAtrLAy7nk3yc8C 8gThSz32E+e2chZI0gDT8OAkXthcS2AqwvGUtm4fGvmJraKgj8cw5ihOL92Vg/y9uTM2 +Q4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3FJUppGtFWGOx0FMlu9BmhAJNre7egam/1ZLuXa6iwY=; b=F2u1fu/t/Abj3460o/OdMtD5lmqZJLWTvAPVAb58JV0GnBwec5987QAqh+0u0nbMGh kUOXhcqEypbYwKxtR5F/W+kr8lHGbsVhpjWJWZDn++gt9MIrUmnBJJeYWrejc07WJ4BA MzVq2T1TxOCPwJZ1/bu5NPqo34mg4sXh11bkZz5QQu8ue12K0KNqP35Ip9sxRrop2P4x zNPu3XdbLGjNpBKRfYfe9/KHSW0uAg1zfFrHDjWiwbIUjuJBd2du4BakD63PLdRn4cPl wE3WOPzJpFIlnw1j39I5TOPor1OESTA4rp9I2Cl4lY7S/PQ3GsExaQwJjJfBqtTZNHNw dQMQ== X-Gm-Message-State: AOAM53226bIPalYbKlASdhavwHMWhlsJjFukmJlFZlzXfz5OfAYaFYr9 EFQxDgLg5dYorVy4ye8X2xAA8w== X-Google-Smtp-Source: ABdhPJxRZGeYraDKkqG8j/MALnwJuWA6lsht8i2c1YOj7Nt7o1eeGpD1tNKvLYoIGg6q25qWP3wm+g== X-Received: by 2002:a17:906:2581:: with SMTP id m1mr2631661ejb.28.1604677994309; Fri, 06 Nov 2020 07:53:14 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id lz27sm1249262ejb.39.2020.11.06.07.53.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 07:53:13 -0800 (PST) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH RESEND v10 1/4] iommu/ioasid: Add ioasid references Date: Fri, 6 Nov 2020 16:50:47 +0100 Message-Id: <20201106155048.997886-2-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201106155048.997886-1-jean-philippe@linaro.org> References: <20201106155048.997886-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201106_105317_557024_A743C0F1 X-CRM114-Status: GOOD ( 21.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jean-Philippe Brucker , jacob.jun.pan@linux.intel.com, will@kernel.org, joro@8bytes.org, eric.auger@redhat.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, robin.murphy@arm.com, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Let IOASID users take references to existing ioasids with ioasid_get(). ioasid_put() drops a reference and only frees the ioasid when its reference number is zero. It returns true if the ioasid was freed. For drivers that don't call ioasid_get(), ioasid_put() is the same as ioasid_free(). Reviewed-by: Eric Auger Reviewed-by: Lu Baolu Signed-off-by: Jean-Philippe Brucker --- include/linux/ioasid.h | 10 ++++++++-- drivers/iommu/intel/iommu.c | 4 ++-- drivers/iommu/intel/svm.c | 6 +++--- drivers/iommu/ioasid.c | 38 +++++++++++++++++++++++++++++++++---- 4 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index 6f000d7a0ddc..e9dacd4b9f6b 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -34,7 +34,8 @@ struct ioasid_allocator_ops { #if IS_ENABLED(CONFIG_IOASID) ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, void *private); -void ioasid_free(ioasid_t ioasid); +void ioasid_get(ioasid_t ioasid); +bool ioasid_put(ioasid_t ioasid); void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)); int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); @@ -48,10 +49,15 @@ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, return INVALID_IOASID; } -static inline void ioasid_free(ioasid_t ioasid) +static inline void ioasid_get(ioasid_t ioasid) { } +static inline bool ioasid_put(ioasid_t ioasid) +{ + return false; +} + static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)) { diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 8651f6d4dfa0..243d642142f1 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -5195,7 +5195,7 @@ static void auxiliary_unlink_device(struct dmar_domain *domain, domain->auxd_refcnt--; if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); } static int aux_domain_add_dev(struct dmar_domain *domain, @@ -5256,7 +5256,7 @@ static int aux_domain_add_dev(struct dmar_domain *domain, spin_unlock(&iommu->lock); spin_unlock_irqrestore(&device_domain_lock, flags); if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); return ret; } diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index f1861fa3d0e4..34f1914050f4 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -592,7 +592,7 @@ intel_svm_bind_mm(struct device *dev, unsigned int flags, if (mm) { ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -610,7 +610,7 @@ intel_svm_bind_mm(struct device *dev, unsigned int flags, if (ret) { if (mm) mmu_notifier_unregister(&svm->notifier, mm); - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -683,7 +683,7 @@ static int intel_svm_unbind_mm(struct device *dev, u32 pasid) kfree_rcu(sdev, rcu); if (list_empty(&svm->devs)) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); if (svm->mm) { mmu_notifier_unregister(&svm->notifier, svm->mm); /* Clear mm's pasid. */ diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c index 0f8dd377aada..50ee27bbd04e 100644 --- a/drivers/iommu/ioasid.c +++ b/drivers/iommu/ioasid.c @@ -2,7 +2,7 @@ /* * I/O Address Space ID allocator. There is one global IOASID space, split into * subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and - * free IOASIDs with ioasid_alloc and ioasid_free. + * free IOASIDs with ioasid_alloc and ioasid_put. */ #include #include @@ -15,6 +15,7 @@ struct ioasid_data { struct ioasid_set *set; void *private; struct rcu_head rcu; + refcount_t refs; }; /* @@ -314,6 +315,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; + refcount_set(&data->refs, 1); /* * Custom allocator needs allocator data to perform platform specific @@ -346,11 +348,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, EXPORT_SYMBOL_GPL(ioasid_alloc); /** - * ioasid_free - Free an IOASID + * ioasid_get - obtain a reference to the IOASID + */ +void ioasid_get(ioasid_t ioasid) +{ + struct ioasid_data *ioasid_data; + + spin_lock(&ioasid_allocator_lock); + ioasid_data = xa_load(&active_allocator->xa, ioasid); + if (ioasid_data) + refcount_inc(&ioasid_data->refs); + else + WARN_ON(1); + spin_unlock(&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL(ioasid_get); + +/** + * ioasid_put - Release a reference to an ioasid * @ioasid: the ID to remove + * + * Put a reference to the IOASID, free it when the number of references drops to + * zero. + * + * Return: %true if the IOASID was freed, %false otherwise. */ -void ioasid_free(ioasid_t ioasid) +bool ioasid_put(ioasid_t ioasid) { + bool free = false; struct ioasid_data *ioasid_data; spin_lock(&ioasid_allocator_lock); @@ -360,6 +385,10 @@ void ioasid_free(ioasid_t ioasid) goto exit_unlock; } + free = refcount_dec_and_test(&ioasid_data->refs); + if (!free) + goto exit_unlock; + active_allocator->ops->free(ioasid, active_allocator->ops->pdata); /* Custom allocator needs additional steps to free the xa element */ if (active_allocator->flags & IOASID_ALLOCATOR_CUSTOM) { @@ -369,8 +398,9 @@ void ioasid_free(ioasid_t ioasid) exit_unlock: spin_unlock(&ioasid_allocator_lock); + return free; } -EXPORT_SYMBOL_GPL(ioasid_free); +EXPORT_SYMBOL_GPL(ioasid_put); /** * ioasid_find - Find IOASID data From patchwork Fri Nov 6 15:50:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11887317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30C55C4741F for ; Fri, 6 Nov 2020 15:55:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AA8FF22202 for ; Fri, 6 Nov 2020 15:55:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="vgQzQpFQ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="sk2Sdvuz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA8FF22202 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SgaEQKWmxqqNjx4ZR4Fy5X0pKxHB25NsWBqLkedEA3Q=; b=vgQzQpFQZPfbc88xKUEKk+owh +YcgZu3041ODb26RLAr0qaIwHpi0KfJHdkjJ1P6M0FasJIqdNWpVY92BBzKL4T0vekWdsuSpPE1zi 9WbDMbO3eKbzbm+Wa1Wat4n5KuFl3A12vuQzxWpiSqoLz7JAwAXBSCmPILKxGbfznfbiOFnaNvQYF uMRbtk+jZVNdTKmx0QNFx16eagFB+NoyWSYMO6mEtu/9JBNHsLhUMoUUqY9rJPtfSTwCT10QxE4ux hn2qwEm93GVCPjr6Gghpi8FXIPo6NDv8JKrubU/V3AcfIiDsJ9NOAWsX5hjcD+YXRaQcK9FB9P789 7SseOCRHA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43G-00071R-34; Fri, 06 Nov 2020 15:53:22 +0000 Received: from mail-ed1-x544.google.com ([2a00:1450:4864:20::544]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43A-0006zP-UO for linux-arm-kernel@lists.infradead.org; Fri, 06 Nov 2020 15:53:18 +0000 Received: by mail-ed1-x544.google.com with SMTP id a15so1767415edy.1 for ; Fri, 06 Nov 2020 07:53:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ELWmKXkijbiS5Ma0VBZ5sbWqTj8BZEn1XnX8tqwoCMo=; b=sk2SdvuzZa6QsSThecsbEOQSwIRiwBZu8rCZ57uy+sc625zUUvlIrfdmylODn0+eRy Xn8SAts9FcUuBX5tZ26Kzdp14zfwBdoGxn3SUrt6EslYqhqa+28grJ/hladi3A8pzZaB Fw0k5txNF5hBHiodVUIbPyizsUqild3DkpBEGvRBpJGTAlXHfTkCryAXfJiJHvOvp3MC EAio6UhpxORgcqHATrOZ+rMA2Oj1+2pj3Rlo3PV7T8DRc8tSwNkbbcSuupyaWutAfWce N+mN4FcSwKThrI79enpZnEqAk7kWkt0hzjwTB4Ghk5cnmn0EjqX0KTCt2jv6XgQ5nbFX 7QeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ELWmKXkijbiS5Ma0VBZ5sbWqTj8BZEn1XnX8tqwoCMo=; b=k9T/TWTKCEfxvAMk2ek6FW7Y028ygBvn95m4eFi+BMHNzn+g8v7mjbw+8bvPs7eNnB Q3IDcwVHQdp++FGyMKmMN52jopcgz3Os8/JvYPv0Ru3nPI+Camzq6Ny+syPhRf39lVCO byE8c+CLq+j+jpAEHXR1RtcghbUEExugK2O7DjMNeUVvn9KcSjs8LfYr/15JOLEKLfuE q7j1Eh0HERZv+9ZvSNDrW9WSH/cwzrp5WPZtJPdolgrkV2kwN4S6ALuRR9Nt4hlAwoZr xaJlf86yplk2j54Av4P/hH1L2T5d09f93Hd+5z5RMdoyx3ohIhZKaZJ9+KZ4XA5dTuS9 s+9Q== X-Gm-Message-State: AOAM5300kwElBHq5pLtQWoUUo2+FwseYroEz9XZHGKGIL+KewMUr5ZBU SLOBellO3uQjLG6wxRvmqkwOsQ== X-Google-Smtp-Source: ABdhPJzbDbPDUq6G5kXrXalDD59F6PsWMh3RakrBnB5gfl2A1mhPmFwKosYVkySpHEED3Qr+PBrTnA== X-Received: by 2002:a50:fd19:: with SMTP id i25mr2576697eds.360.1604677995455; Fri, 06 Nov 2020 07:53:15 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id lz27sm1249262ejb.39.2020.11.06.07.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 07:53:14 -0800 (PST) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH RESEND v10 2/4] iommu/sva: Add PASID helpers Date: Fri, 6 Nov 2020 16:50:48 +0100 Message-Id: <20201106155048.997886-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201106155048.997886-1-jean-philippe@linaro.org> References: <20201106155048.997886-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201106_105317_202837_1B6875CF X-CRM114-Status: GOOD ( 23.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jean-Philippe Brucker , jacob.jun.pan@linux.intel.com, will@kernel.org, joro@8bytes.org, eric.auger@redhat.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, robin.murphy@arm.com, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Let IOMMU drivers allocate a single PASID per mm. Store the mm in the IOASID set to allow refcounting and searching mm by PASID, when handling an I/O page fault. Reviewed-by: Lu Baolu Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/Kconfig | 5 ++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-sva-lib.h | 15 ++++++ drivers/iommu/iommu-sva-lib.c | 86 +++++++++++++++++++++++++++++++++++ 4 files changed, 107 insertions(+) create mode 100644 drivers/iommu/iommu-sva-lib.h create mode 100644 drivers/iommu/iommu-sva-lib.c diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 04878caf6da4..f5ebbdfbf636 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -103,6 +103,11 @@ config IOMMU_DMA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH +# Shared Virtual Addressing library +config IOMMU_SVA_LIB + bool + select IOASID + config FSL_PAMU bool "Freescale IOMMU support" depends on PCI diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 11f1771104f3..61bd30cd8369 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -27,3 +27,4 @@ obj-$(CONFIG_FSL_PAMU) += fsl_pamu.o fsl_pamu_domain.o obj-$(CONFIG_S390_IOMMU) += s390-iommu.o obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o +obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o diff --git a/drivers/iommu/iommu-sva-lib.h b/drivers/iommu/iommu-sva-lib.h new file mode 100644 index 000000000000..b40990aef3fd --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * SVA library for IOMMU drivers + */ +#ifndef _IOMMU_SVA_LIB_H +#define _IOMMU_SVA_LIB_H + +#include +#include + +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max); +void iommu_sva_free_pasid(struct mm_struct *mm); +struct mm_struct *iommu_sva_find(ioasid_t pasid); + +#endif /* _IOMMU_SVA_LIB_H */ diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c new file mode 100644 index 000000000000..bd41405d34e9 --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.c @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Helpers for IOMMU drivers implementing SVA + */ +#include +#include + +#include "iommu-sva-lib.h" + +static DEFINE_MUTEX(iommu_sva_lock); +static DECLARE_IOASID_SET(iommu_sva_pasid); + +/** + * iommu_sva_alloc_pasid - Allocate a PASID for the mm + * @mm: the mm + * @min: minimum PASID value (inclusive) + * @max: maximum PASID value (inclusive) + * + * Try to allocate a PASID for this mm, or take a reference to the existing one + * provided it fits within the [@min, @max] range. On success the PASID is + * available in mm->pasid, and must be released with iommu_sva_free_pasid(). + * @min must be greater than 0, because 0 indicates an unused mm->pasid. + * + * Returns 0 on success and < 0 on error. + */ +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) +{ + int ret = 0; + ioasid_t pasid; + + if (min == INVALID_IOASID || max == INVALID_IOASID || + min == 0 || max < min) + return -EINVAL; + + mutex_lock(&iommu_sva_lock); + if (mm->pasid) { + if (mm->pasid >= min && mm->pasid <= max) + ioasid_get(mm->pasid); + else + ret = -EOVERFLOW; + } else { + pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); + if (pasid == INVALID_IOASID) + ret = -ENOMEM; + else + mm->pasid = pasid; + } + mutex_unlock(&iommu_sva_lock); + return ret; +} +EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid); + +/** + * iommu_sva_free_pasid - Release the mm's PASID + * @mm: the mm + * + * Drop one reference to a PASID allocated with iommu_sva_alloc_pasid() + */ +void iommu_sva_free_pasid(struct mm_struct *mm) +{ + mutex_lock(&iommu_sva_lock); + if (ioasid_put(mm->pasid)) + mm->pasid = 0; + mutex_unlock(&iommu_sva_lock); +} +EXPORT_SYMBOL_GPL(iommu_sva_free_pasid); + +/* ioasid_find getter() requires a void * argument */ +static bool __mmget_not_zero(void *mm) +{ + return mmget_not_zero(mm); +} + +/** + * iommu_sva_find() - Find mm associated to the given PASID + * @pasid: Process Address Space ID assigned to the mm + * + * On success a reference to the mm is taken, and must be released with mmput(). + * + * Returns the mm corresponding to this PASID, or an error if not found. + */ +struct mm_struct *iommu_sva_find(ioasid_t pasid) +{ + return ioasid_find(&iommu_sva_pasid, pasid, __mmget_not_zero); +} +EXPORT_SYMBOL_GPL(iommu_sva_find); From patchwork Fri Nov 6 15:50:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11887307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20F84C4741F for ; Fri, 6 Nov 2020 15:54:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8925A22202 for ; Fri, 6 Nov 2020 15:54:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hE35aKm5"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="t3aevIQX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8925A22202 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tyatIPJ3LrSo4kBXfNsnMhXxLEexHkZmvq2hYYeR5xk=; b=hE35aKm55jGHqEvF8kdPXzj8J BtPnawB/ywha6PUHiLxf7vKOsnwtkkhZXZf9WNGenekUdqIH/czM8tjCGet5KP2xhlrmrvAbIq6pr MksLsDDhsjovTVGGCNIlZSLLqjczS7iBmYMKtHzjvl180JCgIrcTuz/8xe5BiYq42998gC92FFIro dKM6It7NztsTjZFh2oQ06LrrfwWtOh7rfQwdmUvfy5+ooE3h7m2SEV2ry3rMxWrG+QHMceJyom5vN ECuQgp/jG0WZpNP6wHCqqeuuDu612a0wdbBbjvPg/wyEARi1KfgWKJ3Ez2bDMtkD5VpQiJgihvcq8 qIM4Y4MDg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43M-00073I-4l; Fri, 06 Nov 2020 15:53:28 +0000 Received: from mail-ej1-x642.google.com ([2a00:1450:4864:20::642]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43C-0006zn-7t for linux-arm-kernel@lists.infradead.org; Fri, 06 Nov 2020 15:53:19 +0000 Received: by mail-ej1-x642.google.com with SMTP id o23so2565327ejn.11 for ; Fri, 06 Nov 2020 07:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R4pmtVTCQAORkBrrk+wQN2T/YnLvtVms9fBQEGuovCw=; b=t3aevIQXhZEwd8a/6+Aa/hV9BJ6AtwH1K9plFNios6sSY2ZV5kl6nOFA5Q/djtm6tu aDW0D417b47ie7f/vHCHuu2Zj9NENwBwyIMGD2fl63TQeJPTpnqlkXwX9oVy9asvX6Vl ruJVRmtQ3vvnIUCt38qbUpl+StZTWB/1eFEDhw5IAkDBTOi/HDt5aGtDucv+TGKg6Oem jaa6dGlqnl1eXTWFOSHRtwafMiH51siqpsAND6Uq/icOVaVcAmlS6An/5tRUcMMCCmrr L+iKwSYDdVBLpOIiDEdualDSMJqDtnRRTsnYH8mb22LOBNyoP5Ukum7NVN2kTwDJb7+r SvNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R4pmtVTCQAORkBrrk+wQN2T/YnLvtVms9fBQEGuovCw=; b=mqAOslZlXelCrALTb4RoBDNXE54XHQpQJnH8pC1dmrzQBowVsF6nfDybEGlpC1EdgT 3r23CqDTz3vfhoQdFoFW/mnBY/VXoM6EPFgMSbnkXW85AeDwOdnfzaUW013eZnRe6GdO fWjQ77pK+uZtihkV9c4+MX8fzf5Dt90aKzV7rDURMD6QeP5l3ZYj/BrTNF3oYMBc8c0s 9SGZxfssOS9kgHBAxJ7+/4DHIexjrWudCDVLu6cmkgpfwtbll7PRYRdRbK4qBaOI3GJl luY7b45LK2WObeLy6iSODYEjrlDDBF+d3WC35yZ5m5CKrAS4NkRjiAK2paPWbJqCZU80 7DNA== X-Gm-Message-State: AOAM533vo4DMvNLCr2vblkJHVF5Pej1//eA2pVkfKdEnmE402019wZCr hQVBlRkNg9sg/cGgFbdWUQtegg== X-Google-Smtp-Source: ABdhPJzE9CLY80Dr1z6puxAH1j1LM/ZkbI4PDC5J2kAW1yH+uiQx3vkauIErW4HmId9lra2UtuP6gQ== X-Received: by 2002:a17:906:1381:: with SMTP id f1mr2570511ejc.87.1604677996742; Fri, 06 Nov 2020 07:53:16 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id lz27sm1249262ejb.39.2020.11.06.07.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 07:53:16 -0800 (PST) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH RESEND v10 3/4] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Date: Fri, 6 Nov 2020 16:50:49 +0100 Message-Id: <20201106155048.997886-4-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201106155048.997886-1-jean-philippe@linaro.org> References: <20201106155048.997886-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201106_105318_328071_6CA0D7AB X-CRM114-Status: GOOD ( 31.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jean-Philippe Brucker , jacob.jun.pan@linux.intel.com, will@kernel.org, joro@8bytes.org, eric.auger@redhat.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, robin.murphy@arm.com, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The sva_bind() function allows devices to access process address spaces using a PASID (aka SSID). (1) bind() allocates or gets an existing MMU notifier tied to the (domain, mm) pair. Each mm gets one PASID. (2) Any change to the address space calls invalidate_range() which sends ATC invalidations (in a subsequent patch). (3) When the process address space dies, the release() notifier disables the CD to allow reclaiming the page tables. Since release() has to be light we do not instruct device drivers to stop DMA here, we just ignore incoming page faults from this point onwards. To avoid any event 0x0a print (C_BAD_CD) we disable translation without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. Don't clear the R bit because the S bit can't be cleared when STALL_MODEL==0b10 (forced), and clearing R without clearing S is useless. Faulting transactions will stall and will be aborted by the IOPF handler. (4) After stopping DMA, the device driver releases the bond by calling unbind(). We release the MMU notifier, free the PASID and the bond. Three structures keep track of bonds: * arm_smmu_bond: one per {device, mm} pair, the handle returned to the device driver for a bind() request. * arm_smmu_mmu_notifier: one per {domain, mm} pair, deals with ATS/TLB invalidations and clearing the context descriptor on mm exit. * arm_smmu_ctx_desc: one per mm, holds the pinned ASID and pgd. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/Kconfig | 2 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 28 +++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 230 +++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 32 ++- 4 files changed, 282 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index f5ebbdfbf636..192ef8f61310 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -316,6 +316,8 @@ config ARM_SMMU_V3 config ARM_SMMU_V3_SVA bool "Shared Virtual Addressing support for the ARM SMMUv3" depends on ARM_SMMU_V3 + select IOMMU_SVA_LIB + select MMU_NOTIFIER help Support for sharing process address spaces with devices using the SMMUv3. diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index d4b7f40ccb02..e03ca01e0908 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -678,10 +678,18 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; + + struct list_head mmu_notifiers; }; +static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) +{ + return container_of(dom, struct arm_smmu_domain, domain); +} + extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +extern struct arm_smmu_ctx_desc quiet_cd; int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, struct arm_smmu_ctx_desc *cd); @@ -694,6 +702,11 @@ bool arm_smmu_master_sva_supported(struct arm_smmu_master *master); bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master); int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); +struct iommu_sva *arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, + void *drvdata); +void arm_smmu_sva_unbind(struct iommu_sva *handle); +u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle); +void arm_smmu_sva_notifier_synchronize(void); #else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { @@ -719,5 +732,20 @@ static inline int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { return -ENODEV; } + +static inline struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + return ERR_PTR(-ENODEV); +} + +static inline void arm_smmu_sva_unbind(struct iommu_sva *handle) {} + +static inline u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + return IOMMU_PASID_INVALID; +} + +static inline void arm_smmu_sva_notifier_synchronize(void) {} #endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 9255c9600fb8..4d03481289ff 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -5,11 +5,35 @@ #include #include +#include #include #include "arm-smmu-v3.h" +#include "../../iommu-sva-lib.h" #include "../../io-pgtable-arm.h" +struct arm_smmu_mmu_notifier { + struct mmu_notifier mn; + struct arm_smmu_ctx_desc *cd; + bool cleared; + refcount_t refs; + struct list_head list; + struct arm_smmu_domain *domain; +}; + +#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) + +struct arm_smmu_bond { + struct iommu_sva sva; + struct mm_struct *mm; + struct arm_smmu_mmu_notifier *smmu_mn; + struct list_head list; + refcount_t refs; +}; + +#define sva_to_bond(handle) \ + container_of(handle, struct arm_smmu_bond, sva) + static DEFINE_MUTEX(sva_lock); /* @@ -64,7 +88,6 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return NULL; } -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; @@ -145,7 +168,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return err < 0 ? ERR_PTR(err) : ret; } -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { if (arm_smmu_free_asid(cd)) { @@ -155,6 +177,201 @@ static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) } } +static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + mutex_lock(&sva_lock); + if (smmu_mn->cleared) { + mutex_unlock(&sva_lock); + return; + } + + /* + * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, + * but disable translation. + */ + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &quiet_cd); + + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + + smmu_mn->cleared = true; + mutex_unlock(&sva_lock); +} + +static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) +{ + kfree(mn_to_smmu(mn)); +} + +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, +}; + +/* Allocate or get existing MMU notifier for this {domain, mm} pair */ +static struct arm_smmu_mmu_notifier * +arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + int ret; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_mmu_notifier *smmu_mn; + + list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { + if (smmu_mn->mn.mm == mm) { + refcount_inc(&smmu_mn->refs); + return smmu_mn; + } + } + + cd = arm_smmu_alloc_shared_cd(mm); + if (IS_ERR(cd)) + return ERR_CAST(cd); + + smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); + if (!smmu_mn) { + ret = -ENOMEM; + goto err_free_cd; + } + + refcount_set(&smmu_mn->refs, 1); + smmu_mn->cd = cd; + smmu_mn->domain = smmu_domain; + smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; + + ret = mmu_notifier_register(&smmu_mn->mn, mm); + if (ret) { + kfree(smmu_mn); + goto err_free_cd; + } + + ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + if (ret) + goto err_put_notifier; + + list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); + return smmu_mn; + +err_put_notifier: + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); +err_free_cd: + arm_smmu_free_shared_cd(cd); + return ERR_PTR(ret); +} + +static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +{ + struct mm_struct *mm = smmu_mn->mn.mm; + struct arm_smmu_ctx_desc *cd = smmu_mn->cd; + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + if (!refcount_dec_and_test(&smmu_mn->refs)) + return; + + list_del(&smmu_mn->list); + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + + /* + * If we went through clear(), we've already invalidated, and no + * new TLB entry can have been formed. + */ + if (!smmu_mn->cleared) + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); + arm_smmu_free_shared_cd(cd); +} + +static struct iommu_sva * +__arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +{ + int ret; + struct arm_smmu_bond *bond; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (!master || !master->sva_enabled) + return ERR_PTR(-ENODEV); + + /* If bind() was already called for this {dev, mm} pair, reuse it. */ + list_for_each_entry(bond, &master->bonds, list) { + if (bond->mm == mm) { + refcount_inc(&bond->refs); + return &bond->sva; + } + } + + bond = kzalloc(sizeof(*bond), GFP_KERNEL); + if (!bond) + return ERR_PTR(-ENOMEM); + + /* Allocate a PASID for this mm if necessary */ + ret = iommu_sva_alloc_pasid(mm, 1, (1U << master->ssid_bits) - 1); + if (ret) + goto err_free_bond; + + bond->mm = mm; + bond->sva.dev = dev; + refcount_set(&bond->refs, 1); + + bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + if (IS_ERR(bond->smmu_mn)) { + ret = PTR_ERR(bond->smmu_mn); + goto err_free_pasid; + } + + list_add(&bond->list, &master->bonds); + return &bond->sva; + +err_free_pasid: + iommu_sva_free_pasid(mm); +err_free_bond: + kfree(bond); + return ERR_PTR(ret); +} + +struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + struct iommu_sva *handle; + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + mutex_lock(&sva_lock); + handle = __arm_smmu_sva_bind(dev, mm); + mutex_unlock(&sva_lock); + return handle; +} + +void arm_smmu_sva_unbind(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + mutex_lock(&sva_lock); + if (refcount_dec_and_test(&bond->refs)) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + iommu_sva_free_pasid(bond->mm); + kfree(bond); + } + mutex_unlock(&sva_lock); +} + +u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + return bond->mm->pasid; +} + bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { unsigned long reg, fld; @@ -246,3 +463,12 @@ int arm_smmu_master_disable_sva(struct arm_smmu_master *master) return 0; } + +void arm_smmu_sva_notifier_synchronize(void) +{ + /* + * Some MMU notifiers may still be waiting to be freed, using + * arm_smmu_mmu_notifier_free(). Wait for them. + */ + mmu_notifier_synchronize(); +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 61002f305a72..769845c34503 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -76,6 +76,12 @@ struct arm_smmu_option_prop { DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); DEFINE_MUTEX(arm_smmu_asid_lock); +/* + * Special value used by SVA when a process dies, to quiesce a CD without + * disabling it. + */ +struct arm_smmu_ctx_desc quiet_cd = { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -91,11 +97,6 @@ static inline void __iomem *arm_smmu_page1_fixup(unsigned long offset, return smmu->base + offset; } -static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) -{ - return container_of(dom, struct arm_smmu_domain, domain); -} - static void parse_driver_options(struct arm_smmu_device *smmu) { int i = 0; @@ -983,7 +984,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -1000,8 +1003,10 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val = 0; + } else if (cd == &quiet_cd) { /* (4) */ + val |= CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1794,6 +1799,7 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; } @@ -2585,6 +2591,9 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, + .sva_bind = arm_smmu_sva_bind, + .sva_unbind = arm_smmu_sva_unbind, + .sva_get_pasid = arm_smmu_sva_get_pasid, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -3607,6 +3616,12 @@ static const struct of_device_id arm_smmu_of_match[] = { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + arm_smmu_sva_notifier_synchronize(); + platform_driver_unregister(drv); +} + static struct platform_driver arm_smmu_driver = { .driver = { .name = "arm-smmu-v3", @@ -3617,7 +3632,8 @@ static struct platform_driver arm_smmu_driver = { .remove = arm_smmu_device_remove, .shutdown = arm_smmu_device_shutdown, }; -module_platform_driver(arm_smmu_driver); +module_driver(arm_smmu_driver, platform_driver_register, + arm_smmu_driver_unregister); MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); From patchwork Fri Nov 6 15:50:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11887319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42289C2D0A3 for ; Fri, 6 Nov 2020 15:54:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B2E7722202 for ; Fri, 6 Nov 2020 15:54:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="VQRtkVs7"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="hjfmsyRe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2E7722202 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SZb8Ma1t7zG/8FAb5WrpZVahd6ho3fV5RKKv5VB/joI=; b=VQRtkVs7G0eym44HIOwkPOjZ6 PF0an7xrf32fXJB5VEecvMlDH3npH20cldpwLfMFqEOH8uq3UzL41Y2yMs6inTCSDkmRXmK7sarm6 J8QZxK1kTz9/7DIzg98oNY21SAIUPNr14mf6DQuUXB0xp4EY8buod5oryqgE9+xYejZ0Ko+GqoRwF hjIsHrEPs0zyVGGTbypR0dcFgsD8BWIgj7of2m2PfSSczzQct6qmmVs6lS9c1ksAoyFml7s+xOpnl 1Wmslg7csItHmKGjSN4xRk/w9q7Cqhvi/gzfSyfHe3uWyeqNa0VQ9OSxVYFWADrXHoDViwHLsLZSN MaWRX5IDg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43O-00073g-HR; Fri, 06 Nov 2020 15:53:30 +0000 Received: from mail-ej1-x643.google.com ([2a00:1450:4864:20::643]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kb43C-00070U-UZ for linux-arm-kernel@lists.infradead.org; Fri, 06 Nov 2020 15:53:19 +0000 Received: by mail-ej1-x643.google.com with SMTP id dk16so2550400ejb.12 for ; Fri, 06 Nov 2020 07:53:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aCmYESCWdUDIsMdymeogP1/5qm+ZjvKp4ksn7abIdX8=; b=hjfmsyReTdoMJGEHLf0pSfWCp8PwYnoLg7ik/pRJYOYwIVgvWMJto442xC9XSAJanx TrzMUXvkjDhGlKWu/tmstaA8zIJrGMAXTeFfcAK21y1+8IX5J2vclrn5TXz3/OrISp0g b7Ww68l4WaACGkzbPx49w0aTdKTOTqUPfe7aXht5dbf0nqfeGnr/n5nXJsuV2WCgq0YD kLvl4E43hdc1FYQ7L7GdeeaohJNO8VMnkhvsHxEiFztHEybT+jObRYuU3n9UIUFXK2tZ zUrYWqvXe5Rz0ge5w62GTXs/63Kfl4cc33MHw0mrn8otgsmYDXw4NR2Tlh9EEedZ5VGC CAPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aCmYESCWdUDIsMdymeogP1/5qm+ZjvKp4ksn7abIdX8=; b=c6fEH8qJwBEThOkV244VDFfBdfhUKW1U6Gtl4oQcyiQzJdcwAXIn/AzaWw/1fMH4xd N8StM+9eg58h+e8h3MjaJWD2MyXT6EJEkbPaKxEtDUlwCC0t/NrnNwvEuPf8Umk+E/2C fhd0EvbexSUsn3wyNhPrB5x3rAFYfBG3Bj2LCpC0VDDdMnlZXDF397BZ3oIVOVll8khG /aqLnNfQkxqR5HlAalSb5xjx35OG6fp1pyY7rSIdu8ninMbCYHBCDfbE0UXxPoxcXIv3 fK2kKZrnKIdtxWMYBYWGh8H6ecb6CBa3PuLWQ7AG+4pqejlzPqrNQkeZqrlCEHxlN4PO FPkw== X-Gm-Message-State: AOAM530m1qMO8zqfd7pxG4BsrqE+zwFmBHdDlmnr66ewriExTcUxE1HQ 7Td4xG5UIJS4DWk3T9d2ICN0+A== X-Google-Smtp-Source: ABdhPJz44VKBog74/PF6QWsQ+kOgaAhJKxdTsStVxT++f9UpVxqRwTSl4aGcKSQ7CQw1z/o9KcemvQ== X-Received: by 2002:a17:906:6545:: with SMTP id u5mr2631563ejn.346.1604677998053; Fri, 06 Nov 2020 07:53:18 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id lz27sm1249262ejb.39.2020.11.06.07.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 07:53:17 -0800 (PST) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH RESEND v10 4/4] iommu/arm-smmu-v3: Hook up ATC invalidation to mm ops Date: Fri, 6 Nov 2020 16:50:50 +0100 Message-Id: <20201106155048.997886-5-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201106155048.997886-1-jean-philippe@linaro.org> References: <20201106155048.997886-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201106_105319_056940_DD0EBA0B X-CRM114-Status: GOOD ( 17.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jean-Philippe Brucker , jacob.jun.pan@linux.intel.com, will@kernel.org, joro@8bytes.org, eric.auger@redhat.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, robin.murphy@arm.com, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The invalidate_range() notifier is called for any change to the address space. Perform the required ATC invalidations. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 16 +++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 18 ++++++++++++++++-- 3 files changed, 33 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index e03ca01e0908..96c2e9565e00 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -695,6 +695,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, + unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 4d03481289ff..e13b092e6004 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -177,6 +177,16 @@ static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) } } +static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + + arm_smmu_atc_inv_domain(smmu_mn->domain, mm->pasid, start, + end - start + 1); +} + static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); @@ -195,6 +205,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &quiet_cd); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -206,6 +217,7 @@ static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) } static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .invalidate_range = arm_smmu_mm_invalidate_range, .release = arm_smmu_mm_release, .free_notifier = arm_smmu_mmu_notifier_free, }; @@ -278,8 +290,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) * If we went through clear(), we've already invalidated, and no * new TLB entry can have been formed. */ - if (!smmu_mn->cleared) + if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + } /* Frees smmu_mn */ mmu_notifier_put(&smmu_mn->mn); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 769845c34503..73c26e68e12e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1524,6 +1524,20 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, size_t inval_grain_shift = 12; unsigned long page_start, page_end; + /* + * ATS and PASID: + * + * If substream_valid is clear, the PCIe TLP is sent without a PASID + * prefix. In that case all ATC entries within the address range are + * invalidated, including those that were requested with a PASID! There + * is no way to invalidate only entries without PASID. + * + * When using STRTAB_STE_1_S1DSS_SSID0 (reserving CD 0 for non-PASID + * traffic), translation requests without PASID create ATC entries + * without PASID, which must be invalidated with substream_valid clear. + * This has the unpleasant side-effect of invalidating all PASID-tagged + * ATC entries within the address range. + */ *cmd = (struct arm_smmu_cmdq_ent) { .opcode = CMDQ_OP_ATC_INV, .substream_valid = !!ssid, @@ -1582,8 +1596,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) return arm_smmu_cmdq_issue_sync(master->smmu); } -static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - int ssid, unsigned long iova, size_t size) +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, + unsigned long iova, size_t size) { int i; unsigned long flags;