From patchwork Thu Jun 18 15:51:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612701 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 782B514E3 for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 44982208DB for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="JW/LxFcI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44982208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6AA9B8D002F; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 65E7F8D0018; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C5158D002F; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id 207418D0018 for ; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D0945B286 for ; Thu, 18 Jun 2020 15:52:41 +0000 (UTC) X-FDA: 76942775322.11.uncle97_3a1143426e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 8B0D2180F8B86 for ; Thu, 18 Jun 2020 15:52:41 +0000 (UTC) X-Spam-Summary: 2,0,0,6890920ee416025a,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:355:379:541:800:960:967:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3351:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4043:4250:4321:5007:6261:6653:6742:7576:7903:9025:9391:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13069:13311:13357:13845:13894:14096:14181:14384:14394:14721:14777:21080:21433:21444:21451:21627:21740:21749:21819:30054:30064:30070,0,RBL:209.85.208.67:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: uncle97_3a1143426e11 X-Filterd-Recvd-Size: 4221 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:40 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id w7so5292771edt.1 for ; Thu, 18 Jun 2020 08:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yTcZ/yCb5ZAy6PQyVCfw2AueTEndBOUdt4lrv7Qejz4=; b=JW/LxFcIwVMI0FUMp2Hp5J9IV4CbwtAZdbhqoRMEX3idf6gsNu+wnpKU+K3PCkEzT8 Z3hEEF3Lg+SzSDaBz8tcAqxQjhy71Hu/Nh7PPuiPBeASjeLBJqrfF5i7bXMkwYa+eigH 8+Y0YMlZi4ZiOEFSBA3y9zy+M2LQWPS8nlsoOCRjNS4B97KfnIBhcC8uZYIvrZkmkCXZ fy4O/KVuDHT8KFgHZkRQkJ1/cOE6bXgfZT9ug04N9mAHyTmklDntEvl1BaNCOX6gnZO9 laYmc4vN5AFc+ATfGfVf5zHp5iPGL+9By9clhNoZNcyQO2BWBFKRiJtia49h75bNu6W8 +pAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yTcZ/yCb5ZAy6PQyVCfw2AueTEndBOUdt4lrv7Qejz4=; b=AH5GIDDZ1x0BGw7KnB7oTfxFNKAHGPwCP2OCORKun1amUVg11BlVCu0ndDrrXln9QQ aDLfJtgF7GY7l5QTlDxq+PQCia/+l4KEr/xNnN9LoyA+l91G+ydUN0BJIQoJ3hYOun7j qs1kVjasl3HAVmPF9NLkyGnc9ZjcpCXbIdJg3AM59Po5N2SCfpUAKf72963IIEO1V56v cQZuhvD41tAZU6lb4e5233WtTO96VL3ZndmPDNkRKACDMMxNKV0WE0qG8pUqMnC2RehV i13/L0U2Vs58XBdx4sxNn7uZlAo0hYaqn+NPjQCLozN8BMO8KVEWmrpjd5nFqdhj0Kn/ iZCA== X-Gm-Message-State: AOAM533Osn4FVRMj146vETNVjBpBa0eJckAr1J+6HPRhlLz681m1Tiwh 4+eImk/43AebpFK1003rp5QytQ== X-Google-Smtp-Source: ABdhPJyr+YsmKXyyVo8ML8+oCnbztzhK2ZSjk0S597/jP9A5VGpW8hFIgvZu6EuzmCv59bIsyCybEw== X-Received: by 2002:aa7:c908:: with SMTP id b8mr4725695edt.76.1592495559755; Thu, 18 Jun 2020 08:52:39 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:39 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Christoph Hellwig , Tony Luck Subject: [PATCH v8 01/12] mm: Define pasid in mm Date: Thu, 18 Jun 2020 17:51:14 +0200 Message-Id: <20200618155125.1548969-2-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8B0D2180F8B86 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Fenghua Yu PASID is shared by all threads in a process. So the logical place to keep track of it is in the "mm". Both ARM and X86 need to use the PASID in the "mm". Suggested-by: Christoph Hellwig Signed-off-by: Fenghua Yu Reviewed-by: Tony Luck --- https://lore.kernel.org/linux-iommu/1592418233-17762-9-git-send-email-fenghua.yu@intel.com/ --- include/linux/mm_types.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 64ede5f150dc5..1ad0e54ebbbaa 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -538,6 +538,10 @@ struct mm_struct { atomic_long_t hugetlb_usage; #endif struct work_struct async_put_work; + +#ifdef CONFIG_IOMMU_SUPPORT + unsigned int pasid; +#endif } __randomize_layout; /* From patchwork Thu Jun 18 15:51:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612703 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27A5214E3 for ; Thu, 18 Jun 2020 15:52:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB1E6208B3 for ; Thu, 18 Jun 2020 15:52:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Eds4e2o9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB1E6208B3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 090678D0030; Thu, 18 Jun 2020 11:52:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 068858D0018; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E73A48D0030; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id D1B968D0018 for ; Thu, 18 Jun 2020 11:52:42 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 933281272BC for ; Thu, 18 Jun 2020 15:52:42 +0000 (UTC) X-FDA: 76942775364.02.face86_0a0ae6926e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 6AB9C100B38 for ; Thu, 18 Jun 2020 15:52:42 +0000 (UTC) X-Spam-Summary: 2,0,0,fad5f8b54845fa7d,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:2:41:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3872:3874:4049:4120:4250:4321:4385:4605:5007:6261:6653:6742:7875:8660:9010:9167:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13148:13230:13894:13972:14096:14394:21080:21324:21444:21451:21627:21939:21987:21990:30003:30034:30054:30055:30056:30080,0,RBL:209.85.218.65:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: face86_0a0ae6926e11 X-Filterd-Recvd-Size: 9454 Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:41 +0000 (UTC) Received: by mail-ej1-f65.google.com with SMTP id n24so6972492ejd.0 for ; Thu, 18 Jun 2020 08:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gANJ9nXJh01yt9NJy6mJRUbndSE80vIrxk0QteKIq3c=; b=Eds4e2o9/aTlytifgQchMEkJmVJnHukU0ovaep4b99cT1h6zth+zUqp366TT6K7ePX HiZEqi5WIama9YvlIAens9d1oSuRcu30EdLiupH19lT8Hmm2muD9gKIAMgXLTT8oZzjY MoyoHHPnrcr8cOojysoVY47yfdgPaVRYmZmo+khdV4Ej00BOefC8o/w6f/PYTJyS6JMS IE97PCTm2L9rvU65hRt0iESOfgfc1BzLFCJZHQEDWvQTL3BUfPf06ylKZClOTGVvHCHR HxJDIw/+s/Qhsbwm5tathG8DD43vy8bx/z/kb1lxFBK35YaCkkejan7zgAl8K15t3TWZ HQ7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gANJ9nXJh01yt9NJy6mJRUbndSE80vIrxk0QteKIq3c=; b=q5rdE4Nkhn4Jmv7Y1gPHq3IHS5IR3/Mdm/Celxk2/ZF/bPX3gGVXFjgTjFU7RlDHpQ 1SbjfEFvCb/mbm3syUhNfF+S7o0pPnJaKlIvsty7dVRdhl9F1HPFWWSm2WQPHHNyjq9q IuokpIJQqsp8wuHBhaGJezbCCTUmTn7EfmwOcHtWBNb+BPKUH52HBzhPXUmnr+NAhdrM hu9Tn00gZtLSI6mtnA8tJH1Kh9TUqVuNsMMwHQ16sE+7SwN43twkDl3QbJ5m4+NnlI7W qtCxvCI8+uJlFFZAG8KlKrm1O0X85mqyguIwpNKb2Ep4aZe/7iP0u1DpyQk7pP/uhuiK j1KA== X-Gm-Message-State: AOAM532p9TUqZh8LgJIktFoPSNUpVBVDas6p3nRBMAZC9ddBY+oJeIBd opafJh5lRqiOVPrRk3XJqGVlEA== X-Google-Smtp-Source: ABdhPJwyzOwFmlJDohdbHbSJOb408dno2oi+63HZld2Ti01+pRXVJHbN4UECF6xp5+TKBsOTjUOPvw== X-Received: by 2002:a17:906:68c5:: with SMTP id y5mr4450793ejr.436.1592495560894; Thu, 18 Jun 2020 08:52:40 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:40 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 02/12] iommu/ioasid: Add ioasid references Date: Thu, 18 Jun 2020 17:51:15 +0200 Message-Id: <20200618155125.1548969-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6AB9C100B38 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let IOASID users take references to existing ioasids with ioasid_get(). ioasid_put() drops a reference and only frees the ioasid when its reference number is zero. It returns true if the ioasid was freed. For drivers that don't call ioasid_get(), ioasid_put() is the same as ioasid_free(). Reviewed-by: Lu Baolu Signed-off-by: Jean-Philippe Brucker --- include/linux/ioasid.h | 10 ++++++++-- drivers/iommu/intel/iommu.c | 4 ++-- drivers/iommu/intel/svm.c | 6 +++--- drivers/iommu/ioasid.c | 38 +++++++++++++++++++++++++++++++++---- 4 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index 6f000d7a0ddcd..e9dacd4b9f6bb 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -34,7 +34,8 @@ struct ioasid_allocator_ops { #if IS_ENABLED(CONFIG_IOASID) ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, void *private); -void ioasid_free(ioasid_t ioasid); +void ioasid_get(ioasid_t ioasid); +bool ioasid_put(ioasid_t ioasid); void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)); int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); @@ -48,10 +49,15 @@ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, return INVALID_IOASID; } -static inline void ioasid_free(ioasid_t ioasid) +static inline void ioasid_get(ioasid_t ioasid) { } +static inline bool ioasid_put(ioasid_t ioasid) +{ + return false; +} + static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)) { diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 9129663a7406b..a5840b8ef14b7 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -5131,7 +5131,7 @@ static void auxiliary_unlink_device(struct dmar_domain *domain, domain->auxd_refcnt--; if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); } static int aux_domain_add_dev(struct dmar_domain *domain, @@ -5193,7 +5193,7 @@ static int aux_domain_add_dev(struct dmar_domain *domain, spin_unlock(&iommu->lock); spin_unlock_irqrestore(&device_domain_lock, flags); if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); return ret; } diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 6c87c807a0abb..b078a697c42e8 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -546,7 +546,7 @@ intel_svm_bind_mm(struct device *dev, int flags, struct svm_dev_ops *ops, if (mm) { ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -564,7 +564,7 @@ intel_svm_bind_mm(struct device *dev, int flags, struct svm_dev_ops *ops, if (ret) { if (mm) mmu_notifier_unregister(&svm->notifier, mm); - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -639,7 +639,7 @@ static int intel_svm_unbind_mm(struct device *dev, int pasid) kfree_rcu(sdev, rcu); if (list_empty(&svm->devs)) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); list_del(&svm->list); diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c index 0f8dd377aada3..50ee27bbd04ec 100644 --- a/drivers/iommu/ioasid.c +++ b/drivers/iommu/ioasid.c @@ -2,7 +2,7 @@ /* * I/O Address Space ID allocator. There is one global IOASID space, split into * subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and - * free IOASIDs with ioasid_alloc and ioasid_free. + * free IOASIDs with ioasid_alloc and ioasid_put. */ #include #include @@ -15,6 +15,7 @@ struct ioasid_data { struct ioasid_set *set; void *private; struct rcu_head rcu; + refcount_t refs; }; /* @@ -314,6 +315,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; + refcount_set(&data->refs, 1); /* * Custom allocator needs allocator data to perform platform specific @@ -346,11 +348,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, EXPORT_SYMBOL_GPL(ioasid_alloc); /** - * ioasid_free - Free an IOASID + * ioasid_get - obtain a reference to the IOASID + */ +void ioasid_get(ioasid_t ioasid) +{ + struct ioasid_data *ioasid_data; + + spin_lock(&ioasid_allocator_lock); + ioasid_data = xa_load(&active_allocator->xa, ioasid); + if (ioasid_data) + refcount_inc(&ioasid_data->refs); + else + WARN_ON(1); + spin_unlock(&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL(ioasid_get); + +/** + * ioasid_put - Release a reference to an ioasid * @ioasid: the ID to remove + * + * Put a reference to the IOASID, free it when the number of references drops to + * zero. + * + * Return: %true if the IOASID was freed, %false otherwise. */ -void ioasid_free(ioasid_t ioasid) +bool ioasid_put(ioasid_t ioasid) { + bool free = false; struct ioasid_data *ioasid_data; spin_lock(&ioasid_allocator_lock); @@ -360,6 +385,10 @@ void ioasid_free(ioasid_t ioasid) goto exit_unlock; } + free = refcount_dec_and_test(&ioasid_data->refs); + if (!free) + goto exit_unlock; + active_allocator->ops->free(ioasid, active_allocator->ops->pdata); /* Custom allocator needs additional steps to free the xa element */ if (active_allocator->flags & IOASID_ALLOCATOR_CUSTOM) { @@ -369,8 +398,9 @@ void ioasid_free(ioasid_t ioasid) exit_unlock: spin_unlock(&ioasid_allocator_lock); + return free; } -EXPORT_SYMBOL_GPL(ioasid_free); +EXPORT_SYMBOL_GPL(ioasid_put); /** * ioasid_find - Find IOASID data From patchwork Thu Jun 18 15:51:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612705 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C58D92A for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5C21C20732 for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="sROLygHw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C21C20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E299E8D0031; Thu, 18 Jun 2020 11:52:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D8B138D0018; Thu, 18 Jun 2020 11:52:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C522B8D0031; Thu, 18 Jun 2020 11:52:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id A027C8D0018 for ; Thu, 18 Jun 2020 11:52:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 66B9A180AD801 for ; Thu, 18 Jun 2020 15:52:44 +0000 (UTC) X-FDA: 76942775448.07.men88_59136e626e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 090291803F9AA for ; Thu, 18 Jun 2020 15:52:43 +0000 (UTC) X-Spam-Summary: 2,0,0,5208171c37ef6fa9,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3865:3867:3868:3870:3871:3872:4118:4250:4321:4385:4605:5007:6119:6261:6653:6742:7875:7903:10004:10903:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13894:14096:14181:14394:14721:14819:21080:21324:21433:21444:21451:21611:21627:21987:21990:30003:30054:30056:30080,0,RBL:209.85.218.65:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: men88_59136e626e11 X-Filterd-Recvd-Size: 7918 Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:43 +0000 (UTC) Received: by mail-ej1-f65.google.com with SMTP id f7so6943818ejq.6 for ; Thu, 18 Jun 2020 08:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RIXfyWsdCSmhkCPoxdMwgb5HoQjrgnKhMBMykTzupqQ=; b=sROLygHw732kSPKgqm1mAJj8riM2WWL2GR/3RO14UTzqzDo+ON4NJHm/mIDeC2LCl4 +RP9frVhkxIRCUhwMIcnsdOC7S0WnA2G6KfJuTe/Je028x/UMJrQov9nesMCYthhJ/lP pJ1rzJVPXLe1HLn7VCsYhkZNAAQANsfO/2H3NkbXt6L2YJO5UoWuzfUlEG0dlXco6I1Q 0c6m2NjRQFUEJEYSQQaKlU23EX57kMW88YOG6S0iyWbgDwJGGwXBOOO7jnrsWYGC/4O9 KIfikh5Y7YxEg+ObMIpjzN59DZNakYHrfmJokgjdphmsjh21ggon+8JPNKuNaBofykdO FZOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RIXfyWsdCSmhkCPoxdMwgb5HoQjrgnKhMBMykTzupqQ=; b=PWKaU8Qj2mkt3Q1wUSNQIaS4NUpRF9jaJKITN8V8e5CpTZjU/KMnUwZdOF75IHbdys AiVKpmnxH1jegB05ygDb3GG3ogSpYkrpia7D0R8ONfkOCjw4VDMWP6anomnel5H9Hfms Saam9NZL0YtFWdF48Ru/Rdc1PQGsoc8VfgzO/nM/NQLdx+3rfDM5J4L7NWPfIXC5lgGJ jFajHNUbMYWNsdC/va4pN/4FG6INWRkgHeyOPsmkbNUjMJsyXVGMqi2IQA4ixDhFDDia GBRm8yGcVPg9/+hsNiP52DwoFGDlJH0sD3dzxSfdjvxR9nc3JS3RG6cMHFaEtOELocKq ic0w== X-Gm-Message-State: AOAM533DRZLC4HQWkRDidxBAeM01MduO+xKudrikTi1MwqZgQb+ZCdyw fsXw1UtYUobKiq0fHRho/g97hA== X-Google-Smtp-Source: ABdhPJxBNDhV30ThfSe9ibwBo1JFl7QA4iTqkJCsARWmrputOq3YDJfzAm2LRPiQdaBFKF1afEsQww== X-Received: by 2002:a17:906:1f4f:: with SMTP id d15mr4481967ejk.206.1592495562342; Thu, 18 Jun 2020 08:52:42 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:41 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 03/12] iommu/sva: Add PASID helpers Date: Thu, 18 Jun 2020 17:51:16 +0200 Message-Id: <20200618155125.1548969-4-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 090291803F9AA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let IOMMU drivers allocate a single PASID per mm. Store the mm in the IOASID set to allow refcounting and searching mm by PASID, when handling an I/O page fault. Signed-off-by: Jean-Philippe Brucker Reviewed-by: Lu Baolu --- v7->v8: rename to IOMMU_SVA_LIB (Lu Baolu) --- drivers/iommu/Kconfig | 5 +++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-sva-lib.h | 15 +++++++ drivers/iommu/iommu-sva-lib.c | 85 +++++++++++++++++++++++++++++++++++ 4 files changed, 106 insertions(+) create mode 100644 drivers/iommu/iommu-sva-lib.h create mode 100644 drivers/iommu/iommu-sva-lib.c diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index b510f67dfa499..74a10e7a8d082 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -102,6 +102,11 @@ config IOMMU_DMA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH +# Shared Virtual Addressing library +config IOMMU_SVA_LIB + bool + select IOASID + config FSL_PAMU bool "Freescale IOMMU support" depends on PCI diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 342190196dfb0..0fe5a7f9bc69c 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -38,3 +38,4 @@ obj-$(CONFIG_S390_IOMMU) += s390-iommu.o obj-$(CONFIG_QCOM_IOMMU) += qcom_iommu.o obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o +obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o diff --git a/drivers/iommu/iommu-sva-lib.h b/drivers/iommu/iommu-sva-lib.h new file mode 100644 index 0000000000000..b40990aef3fde --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * SVA library for IOMMU drivers + */ +#ifndef _IOMMU_SVA_LIB_H +#define _IOMMU_SVA_LIB_H + +#include +#include + +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max); +void iommu_sva_free_pasid(struct mm_struct *mm); +struct mm_struct *iommu_sva_find(ioasid_t pasid); + +#endif /* _IOMMU_SVA_LIB_H */ diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c new file mode 100644 index 0000000000000..db7e6c104d6b0 --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Helpers for IOMMU drivers implementing SVA + */ +#include +#include + +#include "iommu-sva-lib.h" + +static DEFINE_MUTEX(iommu_sva_lock); +static DECLARE_IOASID_SET(iommu_sva_pasid); + +/** + * iommu_sva_alloc_pasid - Allocate a PASID for the mm + * @mm: the mm + * @min: minimum PASID value (inclusive) + * @max: maximum PASID value (inclusive) + * + * Try to allocate a PASID for this mm, or take a reference to the existing one + * provided it fits within the [min, max] range. On success the PASID is + * available in mm->pasid, and must be released with iommu_sva_free_pasid(). + * + * Returns 0 on success and < 0 on error. + */ +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) +{ + int ret = 0; + ioasid_t pasid; + + if (min == INVALID_IOASID || max == INVALID_IOASID || + min == 0 || max < min) + return -EINVAL; + + mutex_lock(&iommu_sva_lock); + if (mm->pasid) { + if (mm->pasid >= min && mm->pasid <= max) + ioasid_get(mm->pasid); + else + ret = -EOVERFLOW; + } else { + pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); + if (pasid == INVALID_IOASID) + ret = -ENOMEM; + else + mm->pasid = pasid; + } + mutex_unlock(&iommu_sva_lock); + return ret; +} +EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid); + +/** + * iommu_sva_free_pasid - Release the mm's PASID + * @mm: the mm. + * + * Drop one reference to a PASID allocated with iommu_sva_alloc_pasid() + */ +void iommu_sva_free_pasid(struct mm_struct *mm) +{ + mutex_lock(&iommu_sva_lock); + if (ioasid_put(mm->pasid)) + mm->pasid = 0; + mutex_unlock(&iommu_sva_lock); +} +EXPORT_SYMBOL_GPL(iommu_sva_free_pasid); + +/* ioasid wants a void * argument */ +static bool __mmget_not_zero(void *mm) +{ + return mmget_not_zero(mm); +} + +/** + * iommu_sva_find() - Find mm associated to the given PASID + * @pasid: Process Address Space ID assigned to the mm + * + * On success a reference to the mm is taken, and must be released with mmput(). + * + * Returns the mm corresponding to this PASID, or an error if not found. + */ +struct mm_struct *iommu_sva_find(ioasid_t pasid) +{ + return ioasid_find(&iommu_sva_pasid, pasid, __mmget_not_zero); +} +EXPORT_SYMBOL_GPL(iommu_sva_find); From patchwork Thu Jun 18 15:51:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B82592A for ; Thu, 18 Jun 2020 15:52:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4AC04208B3 for ; Thu, 18 Jun 2020 15:52:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="qKis8+0z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4AC04208B3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 96DC38D0032; Thu, 18 Jun 2020 11:52:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8A7A68D0018; Thu, 18 Jun 2020 11:52:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 747FB8D0032; Thu, 18 Jun 2020 11:52:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 5D3558D0018 for ; Thu, 18 Jun 2020 11:52:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1BF05181ABE84 for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) X-FDA: 76942775490.05.cub34_311024626e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id EDAED18001C1E for ; Thu, 18 Jun 2020 15:52:44 +0000 (UTC) X-Spam-Summary: 2,0,0,9eb1f1a815d036a7,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:1:2:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2194:2199:2393:2559:2562:2689:2693:2739:2895:2903:3138:3139:3140:3141:3142:3165:3865:3866:3867:3868:3870:3871:3872:3874:4050:4250:4321:4605:5007:6119:6261:6653:6742:7901:7903:7904:8660:10004:11026:11473:11657:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12895:12986:13053:13148:13230:13894:14394:21080:21433:21444:21451:21611:21627:21740:21795:21939:21990:30003:30034:30051:30054:30056:30070:30080:30091,0,RBL:209.85.218.68:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: cub34_311024626e11 X-Filterd-Recvd-Size: 11219 Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:44 +0000 (UTC) Received: by mail-ej1-f68.google.com with SMTP id l27so6978091ejc.1 for ; Thu, 18 Jun 2020 08:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tNsKKlZrMS7qavcUPxigtHJpxuII47BMB2PD7LPqYhM=; b=qKis8+0zz9FpJJzO2PcVBzxN7h/KVSGahqjTo3rC/34bBhNsoWpi6DqOiFkJFrgcXw PWN4tPDht0f3kfRBr6KB4nLmSlDlX+REPkX1b57yDdX2jzIFFYyGSZK2/cg3NmCt7Cr2 yPL7/xQjFWTj9g2yUHgLHO3sPtQu6A2NoTG9flk5EjnjRekGXMA4Xp8C3ClPIkQoKt3a 8eSPbkQPV1DxwCylidi3u8x7X+PSNNwhd2TVUoCUhvNrLmS4kwWv70lYMXihoOmXlX+6 erobYWDFdNqg3HdBh17QtQaswTrMhad1LxNT5/v8Uph4G7Olvb71xPiHyDHjvF68bx2X jvKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tNsKKlZrMS7qavcUPxigtHJpxuII47BMB2PD7LPqYhM=; b=VZ8rXsMNEX1Iri9JqgjH3eCyQrONs7t+BMOPB+4pwmBi7+uO0dPSDRLr4lgZ5Gh9GR 1nGf4g6AkCVG1UKsAJNkaFwr/cZuw++3xlKC/Tw2ZIYo+43q5HGfB68XrOtF1et2ws+z PM8eb6ICApoA75BxUkmrByTmFiGWZ75UD1sJ7scNcZbR12LzoOoTjLXHDlyu4Wejylpx +thDZOXV1/4UPrwqA66eXJ+0borc70rhI2bVay9hX/NhJYK/rWEeltTM/ah1KZbZ/gr0 ugb/kawWr/NUP6GJqYtUevs80UgTGPodL1i8Ustts2Xmmv834goSVwklsK977Urde84u GiTg== X-Gm-Message-State: AOAM532f0xrQ485L3+rgwG/DK9655/qlUxIH/dY8UIybq/OKq8oOVOxG 4Em83tDtBcJE3elCLKziTKWozg== X-Google-Smtp-Source: ABdhPJwMTz6VrTL+YpR+VUTsW2WpdHUrvHPd0nxeIDXU3aORNj7p14GXmHBByZRwRDr/GjFSjYh2ZQ== X-Received: by 2002:a17:906:4a03:: with SMTP id w3mr4472232eju.154.1592495563380; Thu, 18 Jun 2020 08:52:43 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:42 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 04/12] arm64: mm: Pin down ASIDs for sharing mm with devices Date: Thu, 18 Jun 2020 17:51:17 +0200 Message-Id: <20200618155125.1548969-5-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EDAED18001C1E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To enable address space sharing with the IOMMU, introduce mm_context_get() and mm_context_put(), that pin down a context and ensure that it will keep its ASID after a rollover. Export the symbols to let the modular SMMUv3 driver use them. Pinning is necessary because a device constantly needs a valid ASID, unlike tasks that only require one when running. Without pinning, we would need to notify the IOMMU when we're about to use a new ASID for a task, and it would get complicated when a new task is assigned a shared ASID. Consider the following scenario with no ASID pinned: 1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1) 2. Task t2 is scheduled on CPUx, gets ASID (1, 2) 3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1) We would now have to immediately generate a new ASID for t1, notify the IOMMU, and finally enable task tn. We are holding the lock during all that time, since we can't afford having another CPU trigger a rollover. The IOMMU issues invalidation commands that can take tens of milliseconds. It gets needlessly complicated. All we wanted to do was schedule task tn, that has no business with the IOMMU. By letting the IOMMU pin tasks when needed, we avoid stalling the slow path, and let the pinning fail when we're out of shareable ASIDs. After a rollover, the allocator expects at least one ASID to be available in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS - 1) is the maximum number of ASIDs that can be shared with the IOMMU. Signed-off-by: Jean-Philippe Brucker --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/mmu_context.h | 11 +++- arch/arm64/mm/context.c | 95 +++++++++++++++++++++++++++- 3 files changed, 104 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 68140fdd89d6b..bbdd291e31d59 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -19,6 +19,7 @@ typedef struct { atomic64_t id; + unsigned long pinned; void *vdso; unsigned long flags; } mm_context_t; diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index b0bd9b55594c5..7b0e0f6cb7e87 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -177,7 +177,13 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) #define destroy_context(mm) do { } while(0) void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); -#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ + atomic64_set(&mm->context.id, 0); + mm->context.pinned = 0; + return 0; +} #ifdef CONFIG_ARM64_SW_TTBR0_PAN static inline void update_saved_ttbr0(struct task_struct *tsk, @@ -250,6 +256,9 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, void verify_cpu_asid_bits(void); void post_ttbr_update_workaround(void); +unsigned long mm_context_get(struct mm_struct *mm); +void mm_context_put(struct mm_struct *mm); + #endif /* !__ASSEMBLY__ */ #endif /* !__ASM_MMU_CONTEXT_H */ diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d702d60e64dab..d0ddd413f5645 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -27,6 +27,10 @@ static DEFINE_PER_CPU(atomic64_t, active_asids); static DEFINE_PER_CPU(u64, reserved_asids); static cpumask_t tlb_flush_pending; +static unsigned long max_pinned_asids; +static unsigned long nr_pinned_asids; +static unsigned long *pinned_asid_map; + #define ASID_MASK (~GENMASK(asid_bits - 1, 0)) #define ASID_FIRST_VERSION (1UL << asid_bits) @@ -74,6 +78,9 @@ void verify_cpu_asid_bits(void) static void set_kpti_asid_bits(void) { + unsigned int k; + u8 *dst = (u8 *)asid_map; + u8 *src = (u8 *)pinned_asid_map; unsigned int len = BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(unsigned long); /* * In case of KPTI kernel/user ASIDs are allocated in @@ -81,7 +88,8 @@ static void set_kpti_asid_bits(void) * is set, then the ASID will map only userspace. Thus * mark even as reserved for kernel. */ - memset(asid_map, 0xaa, len); + for (k = 0; k < len; k++) + dst[k] = src[k] | 0xaa; } static void set_reserved_asid_bits(void) @@ -89,7 +97,7 @@ static void set_reserved_asid_bits(void) if (arm64_kernel_unmapped_at_el0()) set_kpti_asid_bits(); else - bitmap_clear(asid_map, 0, NUM_USER_ASIDS); + bitmap_copy(asid_map, pinned_asid_map, NUM_USER_ASIDS); } #define asid_gen_match(asid) \ @@ -165,6 +173,14 @@ static u64 new_context(struct mm_struct *mm) if (check_update_reserved_asid(asid, newasid)) return newasid; + /* + * If it is pinned, we can keep using it. Note that reserved + * takes priority, because even if it is also pinned, we need to + * update the generation into the reserved_asids. + */ + if (mm->context.pinned) + return newasid; + /* * We had a valid ASID in a previous life, so try to re-use * it if possible. @@ -254,6 +270,68 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +unsigned long mm_context_get(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + asid = atomic64_read(&mm->context.id); + + if (mm->context.pinned) { + mm->context.pinned++; + asid &= ~ASID_MASK; + goto out_unlock; + } + + if (nr_pinned_asids >= max_pinned_asids) { + asid = 0; + goto out_unlock; + } + + if (!asid_gen_match(asid)) { + /* + * We went through one or more rollover since that ASID was + * used. Ensure that it is still valid, or generate a new one. + */ + asid = new_context(mm); + atomic64_set(&mm->context.id, asid); + } + + asid &= ~ASID_MASK; + + nr_pinned_asids++; + __set_bit(asid2idx(asid), pinned_asid_map); + mm->context.pinned++; + +out_unlock: + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); + + /* Set the equivalent of USER_ASID_BIT */ + if (asid && IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) + asid |= 1; + + return asid; +} +EXPORT_SYMBOL_GPL(mm_context_get); + +void mm_context_put(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid = atomic64_read(&mm->context.id) & ~ASID_MASK; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + if (--mm->context.pinned == 0) { + __clear_bit(asid2idx(asid), pinned_asid_map); + nr_pinned_asids--; + } + + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); +} +EXPORT_SYMBOL_GPL(mm_context_put); + /* Errata workaround post TTBRx_EL1 update. */ asmlinkage void post_ttbr_update_workaround(void) { @@ -303,6 +381,13 @@ static int asids_update_limit(void) WARN_ON(num_available_asids - 1 <= num_possible_cpus()); pr_info("ASID allocator initialised with %lu entries\n", num_available_asids); + + /* + * We assume that an ASID is always available after a rollover. This + * means that even if all CPUs have a reserved ASID, there still is at + * least one slot available in the asid map. + */ + max_pinned_asids = num_available_asids - num_possible_cpus() - 2; return 0; } arch_initcall(asids_update_limit); @@ -317,6 +402,12 @@ static int asids_init(void) panic("Failed to allocate bitmap for %lu ASIDs\n", NUM_USER_ASIDS); + pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS), + sizeof(*pinned_asid_map), GFP_KERNEL); + if (!pinned_asid_map) + panic("Failed to allocate pinned ASID bitmap\n"); + nr_pinned_asids = 0; + /* * We cannot call set_reserved_asid_bits() here because CPU * caps are not finalized yet, so it is safer to assume KPTI From patchwork Thu Jun 18 15:51:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612709 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3366514E3 for ; Thu, 18 Jun 2020 15:52:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F3BC620732 for ; Thu, 18 Jun 2020 15:52:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="h0r3TSsb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3BC620732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 63B558D0033; Thu, 18 Jun 2020 11:52:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5ECEB8D0018; Thu, 18 Jun 2020 11:52:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D6D48D0033; Thu, 18 Jun 2020 11:52:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 367358D0018 for ; Thu, 18 Jun 2020 11:52:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E272E181ABE84 for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) X-FDA: 76942775490.19.toes86_490d4ca26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id BA3821AD1B4 for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) X-Spam-Summary: 2,0,0,48694875941cf8eb,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:1801:1981:2194:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3354:3865:3866:4117:4250:4321:4605:5007:6119:6261:6653:6742:9592:10004:11026:11473:11658:11914:12043:12291:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:13972:14096:14110:14181:14394:14721:21080:21444:21451:21611:21627:30012:30054,0,RBL:209.85.218.67:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: toes86_490d4ca26e11 X-Filterd-Recvd-Size: 6968 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:45 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id n24so6972665ejd.0 for ; Thu, 18 Jun 2020 08:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FY0aPXr4T1XKvh6YqCT/xGUuNfXzsIgkFE1SNeWNDTU=; b=h0r3TSsbVyAd/GgpK2IGqK86/znVI1rJ0Spx39sObm1jycjjG19VLycMXWz+PqwAN1 oJI824ldExIo5cDBCcj0Ykma1Yj12YE9UxnQUpmC2qwfvxAhm44lkQ/X5pRfLXPtH9Xn 58s1XcNZnylhoa5mj1clF3V/tN61SIwOU5PuOJz7lf44dVjhinD4zYC005kKGHt8NHyB pOmYn3TiDbLE9pcftz7/UUy3qwXjdz10rktsWo+MzTpkvFAVrA3BQxS77yG/SRKHV3bN eE4pDvlRXb9dT7bdEDJdIHi9I5KBg2fxj0Ez+BMUb+E08Ehg0Oi7tTdLqUqvYvMJap3t WJPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FY0aPXr4T1XKvh6YqCT/xGUuNfXzsIgkFE1SNeWNDTU=; b=RXotjOTYe892bmE6/fy8fjPZEzGoOQTpxhkJDj/r7C3PRTIBVQ8nW45uQVNHFI3HfX RdYrbAvFzy5OogakVuLehh1lwhrDvHqqJrPwgpGBAkftzxz9K1qDFZF6/nz8M3twxS5f NSzkLw8oavYSSeTrvTQfIyNYiNP8HBjE2Z7LjSVdfkhg5oExDGUdR4UXXV0NKhF8n7WD DSsll/p7Cbd1q4RrjNp6j9mAMu1Y9+uhXOYF4xcCE+ir8UX6dc0d8zmrwfokSB19L4vf 3QEWBxOmiM446G+5o3fX9olCqKIovO5ekcudyVom9Hz+gRKR4yDUzK81ZFy8L7v7Kuq3 krhw== X-Gm-Message-State: AOAM5322n3W3ku9C6jQbvrrH/YszuNNxTahFRHX5cogalwhZ0y/KTg/7 buKwcM0/YCDMsBpjKuRnHHcO/g== X-Google-Smtp-Source: ABdhPJxPqR2tMmpRqJZE3NxKQ3H6rUsjM+EqJN0+OqhoA1wLKcbML/g//NRUZvq8y8ascWv+p439Rg== X-Received: by 2002:a17:906:3e0c:: with SMTP id k12mr4362788eji.441.1592495564354; Thu, 18 Jun 2020 08:52:44 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:43 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 05/12] iommu/io-pgtable-arm: Move some definitions to a header Date: Thu, 18 Jun 2020 17:51:18 +0200 Message-Id: <20200618155125.1548969-6-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BA3821AD1B4 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extract some of the most generic TCR defines, so they can be reused by the page table sharing code. Acked-by: Will Deacon Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/io-pgtable-arm.h | 30 ++++++++++++++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 27 ++------------------------- MAINTAINERS | 3 +-- 3 files changed, 33 insertions(+), 27 deletions(-) create mode 100644 drivers/iommu/io-pgtable-arm.h diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h new file mode 100644 index 0000000000000..ba7cfdf7afa03 --- /dev/null +++ b/drivers/iommu/io-pgtable-arm.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef IO_PGTABLE_ARM_H_ +#define IO_PGTABLE_ARM_H_ + +#define ARM_LPAE_TCR_TG0_4K 0 +#define ARM_LPAE_TCR_TG0_64K 1 +#define ARM_LPAE_TCR_TG0_16K 2 + +#define ARM_LPAE_TCR_TG1_16K 1 +#define ARM_LPAE_TCR_TG1_4K 2 +#define ARM_LPAE_TCR_TG1_64K 3 + +#define ARM_LPAE_TCR_SH_NS 0 +#define ARM_LPAE_TCR_SH_OS 2 +#define ARM_LPAE_TCR_SH_IS 3 + +#define ARM_LPAE_TCR_RGN_NC 0 +#define ARM_LPAE_TCR_RGN_WBWA 1 +#define ARM_LPAE_TCR_RGN_WT 2 +#define ARM_LPAE_TCR_RGN_WB 3 + +#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL +#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL +#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL +#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL +#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL +#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL +#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL + +#endif /* IO_PGTABLE_ARM_H_ */ diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 04fbd4bf0ff9f..f71a2eade04ab 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -20,6 +20,8 @@ #include +#include "io-pgtable-arm.h" + #define ARM_LPAE_MAX_ADDR_BITS 52 #define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 #define ARM_LPAE_MAX_LEVELS 4 @@ -100,23 +102,6 @@ #define ARM_LPAE_PTE_MEMATTR_DEV (((arm_lpae_iopte)0x1) << 2) /* Register bits */ -#define ARM_LPAE_TCR_TG0_4K 0 -#define ARM_LPAE_TCR_TG0_64K 1 -#define ARM_LPAE_TCR_TG0_16K 2 - -#define ARM_LPAE_TCR_TG1_16K 1 -#define ARM_LPAE_TCR_TG1_4K 2 -#define ARM_LPAE_TCR_TG1_64K 3 - -#define ARM_LPAE_TCR_SH_NS 0 -#define ARM_LPAE_TCR_SH_OS 2 -#define ARM_LPAE_TCR_SH_IS 3 - -#define ARM_LPAE_TCR_RGN_NC 0 -#define ARM_LPAE_TCR_RGN_WBWA 1 -#define ARM_LPAE_TCR_RGN_WT 2 -#define ARM_LPAE_TCR_RGN_WB 3 - #define ARM_LPAE_VTCR_SL0_MASK 0x3 #define ARM_LPAE_TCR_T0SZ_SHIFT 0 @@ -124,14 +109,6 @@ #define ARM_LPAE_VTCR_PS_SHIFT 16 #define ARM_LPAE_VTCR_PS_MASK 0x7 -#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL -#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL -#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL -#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL -#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL -#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL -#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL - #define ARM_LPAE_MAIR_ATTR_SHIFT(n) ((n) << 3) #define ARM_LPAE_MAIR_ATTR_MASK 0xff #define ARM_LPAE_MAIR_ATTR_DEVICE 0x04 diff --git a/MAINTAINERS b/MAINTAINERS index 68f21d46614c4..2e3220273e4d5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1499,8 +1499,7 @@ L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/iommu/arm,smmu* F: drivers/iommu/arm-smmu* -F: drivers/iommu/io-pgtable-arm-v7s.c -F: drivers/iommu/io-pgtable-arm.c +F: drivers/iommu/io-pgtable-arm* ARM SUB-ARCHITECTURES L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) From patchwork Thu Jun 18 15:51:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612711 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADE3D92A for ; Thu, 18 Jun 2020 15:52:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7A74B20732 for ; Thu, 18 Jun 2020 15:52:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="CXa/IxLq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A74B20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 748BD8D0034; Thu, 18 Jun 2020 11:52:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6FAF38D0018; Thu, 18 Jun 2020 11:52:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60F4B8D0034; Thu, 18 Jun 2020 11:52:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 4A5628D0018 for ; Thu, 18 Jun 2020 11:52:47 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1101A6F24D for ; Thu, 18 Jun 2020 15:52:47 +0000 (UTC) X-FDA: 76942775574.22.sheep88_361695f26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id B27D61803A254 for ; Thu, 18 Jun 2020 15:52:46 +0000 (UTC) X-Spam-Summary: 2,0,0,762f1732a3677727,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:334:355:368:369:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3350:3865:3867:3870:4250:4321:5007:6261:6653:6742:7903:7904:10004:11026:11473:11658:11914:12043:12114:12297:12438:12517:12519:12555:12895:12986:13069:13138:13161:13229:13231:13311:13357:13894:14181:14384:14394:14721:21080:21222:21444:21451:21627:21990:30054:30080:30089,0,RBL:209.85.218.67:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: sheep88_361695f26e11 X-Filterd-Recvd-Size: 4188 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:46 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id p20so6897085ejd.13 for ; Thu, 18 Jun 2020 08:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OszwHFuvAODAZ4JcDGk8vTdgEsm3AXU3gmIMXDwpr1E=; b=CXa/IxLq2sAhhWXEwadxVdBTNRXCjnNRJ86QHDU2Mv2XE4055xNa1KydScB6wK21ZB cXy76zowZ7DH6oYph6Pl7OuFVuh4YQj0mvtW0MxteQ4UcjzMXY+xljI/4UnbdNVn8beh plIgdS+Z0wk8kS8Nt5E8eD6lwtdFgeDqCh4xStJzlWi7O9hNV7KqxuDuPglptEYvakjg 4gRgbFs6be6neKEhVJpZF7i5q6uxLLCAUoZE3762HcKHIu0WyhDcJaTICqaeIgkR/5so ELvNZAlx+cNLKnbAJ14SIMrBO5wUtNh1LjFnDsET/fpJA6CuAxpepH0+ZH331HeSTxGC PuMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OszwHFuvAODAZ4JcDGk8vTdgEsm3AXU3gmIMXDwpr1E=; b=EThGjH/DHXvx+ezt7zr/8fFkcJHkfI5ntP+vC3XD5JUW3bNd3N6kFndi8paG0Dt588 yTlSgVHzYBI5xclKarQA1aynBgwDKpLahsaOQFNV7S+Fw1+bAwL8ZGy+m85/bsWwE0vA NMEGA4Zn/z7WoNyzJvyK1pG66whnJo7nob67yl08UJCHs/sLcFOdhkQTnSDXcrAfOXPe ph3RQPh9LhWf4e+p36WuBxwKuTz0IJ2DUUyw/nBIh999et11sqZkzxBoasgaI9of+dO7 y5Ucie+7q7v5AimltekIeDHZmIGfQJBPv2Qii+KnCY+M8xQreB1/kiOfBs4lJvzAnunV tAHw== X-Gm-Message-State: AOAM533idFA28MVSHoQ4GT9WhUB9lH2t+XPDrrdMLRGDrLVRveKgddCR eg/Kr3HVXtWkJw6rmJ0qVAnPow== X-Google-Smtp-Source: ABdhPJyc8Mnmi4a1Aktae1QZY9O3zjJrhf37UfpwnWXbqDPMbp6c+LvxNDW5UP6lbOX5cvWPvekjsQ== X-Received: by 2002:a17:906:9257:: with SMTP id c23mr4487938ejx.86.1592495565343; Thu, 18 Jun 2020 08:52:45 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:44 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Suzuki K Poulose Subject: [PATCH v8 06/12] arm64: cpufeature: Export symbol read_sanitised_ftr_reg() Date: Thu, 18 Jun 2020 17:51:19 +0200 Message-Id: <20200618155125.1548969-7-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B27D61803A254 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SMMUv3 driver would like to read the MMFR0 PARANGE field in order to share CPU page tables with devices. Allow the driver to be built as module by exporting the read_sanitized_ftr_reg() cpufeature symbol. Acked-by: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- arch/arm64/kernel/cpufeature.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 4ae41670c2e6b..ba1e17ad17447 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1068,6 +1068,7 @@ u64 read_sanitised_ftr_reg(u32 id) return 0; return regp->sys_val; } +EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); #define read_sysreg_case(r) \ case r: return read_sysreg_s(r) From patchwork Thu Jun 18 15:51:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612713 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD03492A for ; Thu, 18 Jun 2020 15:53:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8C6E4208C7 for ; Thu, 18 Jun 2020 15:53:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="nbPHVqQ4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C6E4208C7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8CF328D0035; Thu, 18 Jun 2020 11:52:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8801F8D0018; Thu, 18 Jun 2020 11:52:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F9EB8D0035; Thu, 18 Jun 2020 11:52:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 551908D0018 for ; Thu, 18 Jun 2020 11:52:48 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1A299824805A for ; Thu, 18 Jun 2020 15:52:48 +0000 (UTC) X-FDA: 76942775616.28.tank62_1f0cfcf26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id E69426F230 for ; Thu, 18 Jun 2020 15:52:47 +0000 (UTC) X-Spam-Summary: 2,0,0,c7fe442984e26c61,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:2:41:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4385:4605:5007:6117:6119:6261:6653:6742:7903:7904:8784:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13894:13972:14096:14394:21080:21433:21444:21451:21627:21990:30054:30070,0,RBL:209.85.218.66:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: tank62_1f0cfcf26e11 X-Filterd-Recvd-Size: 9697 Received: from mail-ej1-f66.google.com (mail-ej1-f66.google.com [209.85.218.66]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:47 +0000 (UTC) Received: by mail-ej1-f66.google.com with SMTP id q19so6941797eja.7 for ; Thu, 18 Jun 2020 08:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WJ6cCCV8E6DMFD7HqvFq9WRcUvuKrkpNdYaDLNUVviA=; b=nbPHVqQ4T+kuk9t8qrzVLGO2UEs5ToyO9mWMZ41PrASGRYC47YvXea1KDC9gDv25Md jtIzi3L7g542lm5vl1vNYWnl3LW7kdNQbBQKQwC2PkP5LFi6I9SJt1E2rpG/3mGzYBB2 htIOMn3gzhav+OS8Em4uEM8SHSczva7uyIuO04C1EYODgPhoZxpLfS8AM5ED8QgXim4R rY/YaGnn+bWlzEiIY8ceUIF6biDZiVmVaE6Kmg/zKCqHph9n17pkcS7fCpstKVVP8VuY l4nA+Ux69hXavZ4QiDwUcRa3y8V/VIWa3AJEl/xf41kb1fLuFalWazAHILTp/PWsqczP HkPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WJ6cCCV8E6DMFD7HqvFq9WRcUvuKrkpNdYaDLNUVviA=; b=Qs0f6z/YQMsA2nkNzvvLohW0u9Asv78UMBa9iTUMiMoEiwg11o16vNalPOrb2qjVxj kMYtCVT7YRb0zEKuXfOju96Q1ZfpKitYpSz+LhypvGxA6/dIh2LV7zHgWs3tPRIv6/vR IF4sAbybgXjocrSWlSSf6A7JnMfoxWSlMuOTNC3KJjL7N55LZBJPKPk88vNJzjvPmNq1 xb8Y938QJdM/qIxXk63nzFFhBhtyE9JYVGScAkiEKGApZAOpq4xmr4pdM6dLoAM8kWZH MuMjRYOlsEsGX+Gv5tsydoxjL+kG/O0ife/Y0sDYBzXezMW4YPQTfKQ2K06l1ZrkHvt5 LbbA== X-Gm-Message-State: AOAM5300xBUd86LO0pM47NmIjyiH6c2AzH78jORE4fvYYCw+P+wuVsfV 1hhVY/I94qJ3BwnroEJntbu+Qw== X-Google-Smtp-Source: ABdhPJwT/k720uyCWIrCGDDjhYeelrlNaUkLfPLh0LBU3DqPpBSXfORheDk7AjOINMKzueIk2y95Ug== X-Received: by 2002:a17:907:2052:: with SMTP id pg18mr4376057ejb.513.1592495566301; Thu, 18 Jun 2020 08:52:46 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:45 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Suzuki K Poulose Subject: [PATCH v8 07/12] iommu/arm-smmu-v3: Share process page tables Date: Thu, 18 Jun 2020 17:51:20 +0200 Message-Id: <20200618155125.1548969-8-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E69426F230 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With Shared Virtual Addressing (SVA), we need to mirror CPU TTBR, TCR, MAIR and ASIDs in SMMU contexts. Each SMMU has a single ASID space split into two sets, shared and private. Shared ASIDs correspond to those obtained from the arch ASID allocator, and private ASIDs are used for "classic" map/unmap DMA. A possible conflict happens when trying to use a shared ASID that has already been allocated for private use by the SMMU driver. This will be addressed in a later patch by replacing the private ASID. At the moment we return -EBUSY. Each mm_struct shared with the SMMU will have a single context descriptor. Add a refcount to keep track of this. It will be protected by the global SVA lock. Acked-by: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 150 +++++++++++++++++++++++++++++++++++- 1 file changed, 146 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 937aa1af428d5..cabd942e4cbf3 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -33,6 +34,8 @@ #include +#include "io-pgtable-arm.h" + /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 #define IDR0_ST_LVL GENMASK(28, 27) @@ -589,6 +592,9 @@ struct arm_smmu_ctx_desc { u64 ttbr; u64 tcr; u64 mair; + + refcount_t refs; + struct mm_struct *mm; }; struct arm_smmu_l1_ctx_desc { @@ -727,6 +733,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_MUTEX(sva_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, @@ -1662,7 +1669,8 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif - CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | + CTXDESC_CD_0_R | CTXDESC_CD_0_A | + (cd->mm ? 0 : CTXDESC_CD_0_ASET) | CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | CTXDESC_CD_0_V; @@ -1766,12 +1774,144 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) cdcfg->cdtab = NULL; } -static void arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) +static void arm_smmu_init_cd(struct arm_smmu_ctx_desc *cd) { + refcount_set(&cd->refs, 1); +} + +static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) +{ + bool free; + struct arm_smmu_ctx_desc *old_cd; + if (!cd->asid) - return; + return false; + + free = refcount_dec_and_test(&cd->refs); + if (free) { + old_cd = xa_erase(&asid_xa, cd->asid); + WARN_ON(old_cd != cd); + } + return free; +} + +static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) +{ + struct arm_smmu_ctx_desc *cd; - xa_erase(&asid_xa, cd->asid); + cd = xa_load(&asid_xa, asid); + if (!cd) + return NULL; + + if (cd->mm) { + /* All devices bound to this mm use the same cd struct. */ + refcount_inc(&cd->refs); + return cd; + } + + /* Ouch, ASID is already in use for a private cd. */ + return ERR_PTR(-EBUSY); +} + +__maybe_unused +static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) +{ + u16 asid; + int ret = 0; + u64 tcr, par, reg; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_ctx_desc *old_cd = NULL; + + lockdep_assert_held(&sva_lock); + + asid = mm_context_get(mm); + if (!asid) + return ERR_PTR(-ESRCH); + + cd = kzalloc(sizeof(*cd), GFP_KERNEL); + if (!cd) { + ret = -ENOMEM; + goto err_put_context; + } + + arm_smmu_init_cd(cd); + + old_cd = arm_smmu_share_asid(asid); + if (IS_ERR(old_cd)) { + ret = PTR_ERR(old_cd); + goto err_free_cd; + } else if (old_cd) { + if (WARN_ON(old_cd->mm != mm)) { + ret = -EINVAL; + goto err_free_cd; + } + kfree(cd); + mm_context_put(mm); + return old_cd; + } + + /* Fails if a private ASID has been allocated since we last checked */ + ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); + if (ret) + goto err_free_cd; + + tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | + CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; + + switch (PAGE_SIZE) { + case SZ_4K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); + break; + case SZ_16K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); + break; + case SZ_64K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); + break; + default: + WARN_ON(1); + ret = -EINVAL; + goto err_free_asid; + } + + reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); + + cd->ttbr = virt_to_phys(mm->pgd); + cd->tcr = tcr; + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + cd->mair = read_sysreg(mair_el1); + cd->asid = asid; + cd->mm = mm; + + return cd; + +err_free_asid: + arm_smmu_free_asid(cd); +err_free_cd: + kfree(cd); +err_put_context: + mm_context_put(mm); + return ERR_PTR(ret); +} + +__maybe_unused +static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) +{ + lockdep_assert_held(&sva_lock); + + if (arm_smmu_free_asid(cd)) { + /* Unpin ASID */ + mm_context_put(cd->mm); + kfree(cd); + } } /* Stream table manipulation functions */ @@ -2481,6 +2621,8 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; + arm_smmu_init_cd(&cfg->cd); + ret = xa_alloc(&asid_xa, &asid, &cfg->cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) From patchwork Thu Jun 18 15:51:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612715 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24B7A92A for ; Thu, 18 Jun 2020 15:53:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D79702070A for ; Thu, 18 Jun 2020 15:53:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="M9XzF/zD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D79702070A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E005A8D0036; Thu, 18 Jun 2020 11:52:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D8A4D8D0018; Thu, 18 Jun 2020 11:52:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2A528D0036; Thu, 18 Jun 2020 11:52:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0005.hostedemail.com [216.40.44.5]) by kanga.kvack.org (Postfix) with ESMTP id AC6918D0018 for ; Thu, 18 Jun 2020 11:52:49 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 42EC1824805A for ; Thu, 18 Jun 2020 15:52:49 +0000 (UTC) X-FDA: 76942775658.24.twig45_03111de26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 14C516F220 for ; Thu, 18 Jun 2020 15:52:49 +0000 (UTC) X-Spam-Summary: 2,0,0,414bbb0619888476,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:1:2:41:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2198:2199:2200:2376:2393:2553:2559:2562:2693:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4050:4250:4321:4385:4605:5007:6119:6261:6653:6742:7874:7903:9040:9592:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14096:14394:21080:21433:21444:21627:21990:30054:30070:30090,0,RBL:209.85.218.68:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: twig45_03111de26e11 X-Filterd-Recvd-Size: 10446 Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:48 +0000 (UTC) Received: by mail-ej1-f68.google.com with SMTP id l12so6929175ejn.10 for ; Thu, 18 Jun 2020 08:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gl8/KHztk13IMZalRx1XfsgApopMYKofIndN5FPkOyc=; b=M9XzF/zDD9NAd1p8Tb01DcQs5VrpQgVrIqRMkSqX0CuDMCDkNlus8SVz7PXa0p7aG4 Ux3pLWztH/B/qiGFwSaVD7dNV3BgOvBi+WcVqdduC7tTgvezsmSgG2C5h8MrR9Rw9KXy tp7tE/DV/Het50vu5ncJZRtXgvPk4c8b9h2I5QA+pWaZxHvM3tO1ra5c0vRF4AuzPs9N +V2vQGJy5/RB7B57uhi0SPJspVMaAgH9v64B7DO0NS/BN461NbV/KNlr9LoVEY30joYc ftiIDufy0ptnDHRQNB3nm7Q9MXl+6OINX0Cp37Eht4FS9eHqKpVmu9x/NM7Q9Yx4HngK UKaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gl8/KHztk13IMZalRx1XfsgApopMYKofIndN5FPkOyc=; b=j/SebGSsbDDpcDJ6+vdBBlpGwyGFYsLdSDaQHhzXAlNOb/10CmvRVrQAOBN276En5k hrUJR2oJvSISTmoH4HBMvbkmOMvZ2KEbc23/gSjnUU8QVB0Q8OxT1Q4YctmeuS2te2NC 8a7O1RaMul14gdJ5fpKgqxypuyZ3D/ATNUAO67mrmyuPM8wicl70Hpy5RDrQbpJpE8z0 BIh4ZY+yJwMGQQ95zs241xROCG/kSwxmZsbjtqcyJcKxI6DGZX+7V4CGBLDvAB9Cx3uH zDGkRm195Yuxkrg4Uh6D7nfdQQPffAYByc7MWgQuacCwynBMBAIOMI8sJWgwVqlBc+Xp HRZw== X-Gm-Message-State: AOAM530COpAfTMYW88GoGcJ8ac7j03KP9nZi3BExJOlipA6wO0pCILXq gwYbRLPjpTzC8+eY3sNzMmQMPw== X-Google-Smtp-Source: ABdhPJyO56ngTMw3kmdlxk17L2Xh8uXctk6ktH/xwwqapMb0kUPyRN+SHVC6d7bMVVWlr+RSxddU0w== X-Received: by 2002:a17:906:6890:: with SMTP id n16mr4383830ejr.553.1592495567422; Thu, 18 Jun 2020 08:52:47 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:47 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 08/12] iommu/arm-smmu-v3: Seize private ASID Date: Thu, 18 Jun 2020 17:51:21 +0200 Message-Id: <20200618155125.1548969-9-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 14C516F220 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 101 +++++++++++++++++++++++++++++------- 1 file changed, 82 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index cabd942e4cbf3..5506add42c9c8 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -733,6 +733,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { @@ -1537,6 +1538,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -1795,9 +1807,18 @@ static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) return free; } +/* + * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from + * the private entry. Careful here, we may be modifying the context tables of + * another SMMU! + */ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&asid_xa, asid); if (!cd) @@ -1809,8 +1830,31 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return cd; } - /* Ouch, ASID is already in use for a private cd. */ - return ERR_PTR(-EBUSY); + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + ret = xa_alloc(&asid_xa, &new_asid, cd, + XA_LIMIT(1, 1 << smmu->asid_bits), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + + /* + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. + */ + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + return NULL; } __maybe_unused @@ -1836,7 +1880,20 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) arm_smmu_init_cd(cd); + /* + * Serialize against arm_smmu_domain_finalise_s1() and + * arm_smmu_domain_free() as we might need to replace the private ASID + * from an existing CD. + */ + mutex_lock(&asid_lock); old_cd = arm_smmu_share_asid(asid); + if (!old_cd) { + ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); + if (ret) + old_cd = ERR_PTR(ret); + } + mutex_unlock(&asid_lock); + if (IS_ERR(old_cd)) { ret = PTR_ERR(old_cd); goto err_free_cd; @@ -1850,11 +1907,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return old_cd; } - /* Fails if a private ASID has been allocated since we last checked */ - ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); - if (ret) - goto err_free_cd; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | @@ -2398,15 +2450,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -2414,8 +2457,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); } @@ -2599,9 +2648,15 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + /* + * Prevent arm_smmu_share_asid() from rewriting CD#0 while we're + * freeing it. + */ + mutex_lock(&asid_lock); if (cfg->cdcfg.cdtab) arm_smmu_free_cd_tables(smmu_domain); arm_smmu_free_asid(&cfg->cd); + mutex_unlock(&asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; if (cfg->vmid) @@ -2623,10 +2678,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, arm_smmu_init_cd(&cfg->cd); + /* + * Prevent arm_smmu_share_asid() from seizing the private ASID we're + * allocating here until it is written to the CD. + */ + mutex_lock(&asid_lock); ret = xa_alloc(&asid_xa, &asid, &cfg->cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) - return ret; + goto out_unlock; cfg->s1cdmax = master->ssid_bits; @@ -2654,12 +2714,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, if (ret) goto out_free_cd_tables; + mutex_unlock(&asid_lock); return 0; out_free_cd_tables: arm_smmu_free_cd_tables(smmu_domain); out_free_asid: arm_smmu_free_asid(&cfg->cd); +out_unlock: + mutex_unlock(&asid_lock); return ret; } From patchwork Thu Jun 18 15:51:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612717 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D3C0592A for ; Thu, 18 Jun 2020 15:53:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F0F32070A for ; Thu, 18 Jun 2020 15:53:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="fDbtc15F" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F0F32070A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C7B6D8D0037; Thu, 18 Jun 2020 11:52:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BDC098D0018; Thu, 18 Jun 2020 11:52:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A552B8D0037; Thu, 18 Jun 2020 11:52:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0239.hostedemail.com [216.40.44.239]) by kanga.kvack.org (Postfix) with ESMTP id 8E7288D0018 for ; Thu, 18 Jun 2020 11:52:50 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 497056FE41 for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) X-FDA: 76942775700.14.limit07_241731926e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 1E350181DF182 for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) X-Spam-Summary: 2,0,0,db572087b522c843,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:4117:4250:4321:5007:6119:6261:6653:6742:7903:7904:8603:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13894:14181:14394:14721:21080:21433:21444:21451:21627:21990:30034:30054:30070,0,RBL:209.85.218.68:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: limit07_241731926e11 X-Filterd-Recvd-Size: 6536 Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:49 +0000 (UTC) Received: by mail-ej1-f68.google.com with SMTP id dp18so6926516ejc.8 for ; Thu, 18 Jun 2020 08:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MNKD6Dmbg1TaE6fvYxzmdVGqFWuaOjJYtMjLhWmnIvo=; b=fDbtc15FP1PMKFiVJC1xPK+dnPneh4gHR5Z5pyZz6iDbXrnrg7Gai4MJxSBwm5mM9v NGRaKISThUuuwgRt+TMw/RfEMNNwRvvCO988/Cysnqn3CIqtZE78iqHHATKrUvig9huz 5bDTNN+8IoMVQimN2851pvQMREQWtJ9Y/CFGobZqLvRXU5qzw9C9YZmBwdZ6YbFi5CLN 2ELl6LeunpXJFw9N3rafo00si6+tSLlbAoJPAInGGLozdmWbUReNSH6gjn+rshCdWJcQ PeYvdxW62Le2lQd7JDTWKinVvZq+ztbeM4jyLNX8bNHJB3Q3vj6RoJT4QVMe457y1ZGE erhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MNKD6Dmbg1TaE6fvYxzmdVGqFWuaOjJYtMjLhWmnIvo=; b=FLQqq8g0ro7ajYzFKWXixghMjQJIG8BXUcNIXPL+ZfH4VgSnEZ+Qe94C/c0dCziIG6 XWoz8S3LWiTj8AcokYZrBuP47iUfu+pvb/qHes+aB8gePUdOzKaR0swvYFOifxCJt//C BAyligq9vAmiuH4cX1PgFmbeBntHtXiJYUsWEZz4fZH2twgBuYcrVJsRwEPCdlXeeglF 8EgflZjbS3z1qkrh3r9nvV13RXFQNr38agvfoQ4hPhQGwKxL/PftmBpZ3/iJ9z7+fTmd JSuJVKOK1I11st8STBxA73p0htXZDANKGzijh1XHV3X/FjvhjHwJVNNJV/4czxho2wZT Pz/g== X-Gm-Message-State: AOAM531tBRpzPxhY3sbHaZON7ykRG8OR2Y6nasWd4bkC82o167QZ7Tp9 V31nQegI9vN5D6I7SfpDitllrQ== X-Google-Smtp-Source: ABdhPJztByorObtVQlxC6WShKEAqlvU9oXK33GgelOz6P5sV8o7rZktu08Gnv9xaL32FeH+u69sT9A== X-Received: by 2002:a17:906:4e59:: with SMTP id g25mr4478633ejw.60.1592495568467; Thu, 18 Jun 2020 08:52:48 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:48 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Suzuki K Poulose Subject: [PATCH v8 09/12] iommu/arm-smmu-v3: Check for SVA features Date: Thu, 18 Jun 2020 17:51:22 +0200 Message-Id: <20200618155125.1548969-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E350181DF182 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Aggregate all sanity-checks for sharing CPU page tables with the SMMU under a single ARM_SMMU_FEAT_SVA bit. For PCIe SVA, users also need to check FEAT_ATS and FEAT_PRI. For platform SVA, they will have to check FEAT_STALLS. Introduce ARM_SMMU_FEAT_BTM (Broadcast TLB Maintenance), but don't enable it at the moment. Since the entire VMID space is shared with the CPU, enabling DVM (by clearing SMMU_CR2.PTM) could result in over-invalidation and affect performance of stage-2 mappings. Cc: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- v7->v8: Use id_aa64mmfr0_parange_to_phys_shift() --- drivers/iommu/arm-smmu-v3.c | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 5506add42c9c8..e2d5171bfb7b9 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -654,6 +654,8 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_STALL_FORCE (1 << 13) #define ARM_SMMU_FEAT_VAX (1 << 14) #define ARM_SMMU_FEAT_RANGE_INV (1 << 15) +#define ARM_SMMU_FEAT_BTM (1 << 16) +#define ARM_SMMU_FEAT_SVA (1 << 17) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) @@ -3894,6 +3896,49 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) return 0; } +static bool arm_smmu_supports_sva(struct arm_smmu_device *smmu) +{ + unsigned long reg, fld; + unsigned long oas; + unsigned long asid_bits; + + u32 feat_mask = ARM_SMMU_FEAT_BTM | ARM_SMMU_FEAT_COHERENCY; + + if ((smmu->features & feat_mask) != feat_mask) + return false; + + if (!(smmu->pgsize_bitmap & PAGE_SIZE)) + return false; + + /* + * Get the smallest PA size of all CPUs (sanitized by cpufeature). We're + * not even pretending to support AArch32 here. Abort if the MMU outputs + * addresses larger than what we support. + */ + reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); + oas = id_aa64mmfr0_parange_to_phys_shift(fld); + if (smmu->oas < oas) + return false; + + /* We can support bigger ASIDs than the CPU, but not smaller */ + fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_ASID_SHIFT); + asid_bits = fld ? 16 : 8; + if (smmu->asid_bits < asid_bits) + return false; + + /* + * See max_pinned_asids in arch/arm64/mm/context.c. The following is + * generally the maximum number of bindable processes. + */ + if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) + asid_bits--; + dev_dbg(smmu->dev, "%d shared contexts\n", (1 << asid_bits) - + num_possible_cpus() - 2); + + return true; +} + static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) { u32 reg; @@ -4093,6 +4138,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) smmu->ias = max(smmu->ias, smmu->oas); + if (arm_smmu_supports_sva(smmu)) + smmu->features |= ARM_SMMU_FEAT_SVA; + dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n", smmu->ias, smmu->oas, smmu->features); return 0; From patchwork Thu Jun 18 15:51:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612719 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F286792A for ; Thu, 18 Jun 2020 15:53:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B0F2A2070A for ; Thu, 18 Jun 2020 15:53:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="upmKPCGB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0F2A2070A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A7D608D0038; Thu, 18 Jun 2020 11:52:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A2D968D0018; Thu, 18 Jun 2020 11:52:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7C48D0038; Thu, 18 Jun 2020 11:52:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0206.hostedemail.com [216.40.44.206]) by kanga.kvack.org (Postfix) with ESMTP id 70AE58D0018 for ; Thu, 18 Jun 2020 11:52:51 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 31C11180AD806 for ; Thu, 18 Jun 2020 15:52:51 +0000 (UTC) X-FDA: 76942775742.15.rock56_3f08ce226e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id F32791814B0C7 for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) X-Spam-Summary: 2,0,0,c53982d790946304,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:2:41:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4119:4250:4385:4605:5007:6261:6653:6742:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12663:12683:12895:13894:14096:14110:14394:21080:21444:21611:21627:21789:21966:21987:21990:30054:30070,0,RBL:209.85.218.67:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: rock56_3f08ce226e11 X-Filterd-Recvd-Size: 8348 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:50 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id dr13so6951547ejc.3 for ; Thu, 18 Jun 2020 08:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fvgnML7PsW3yuBKl7xZjxhkiu8/hUKUQrLt8sPfDx2o=; b=upmKPCGBBIWHElLSb36en66bExWNyxKTU8+A6gjfnLDCbV4TrPCbN/fl0wuPvVmkGr d8jLr65IH2jaDOYHVDrozzBMWtqseimc1dPo+ZDyg2SFM8cQgdHPycaXPFy5e2K/WwOD jwbVFdqYWQeju2yJe6gVJE2fEZ13tW4N0uLNM5zfiLAON/7LJ+VHAMXiXbr/XzEvC+4e 9dzkdOSj32WiAZUYacq0/NJtrKIAQK5MWosVVifejiJz3X9Zox6CZ3aBbn4584SnaRbh 2iy7wI/GO2l0PzURfeGYBqufuvU14VMyND1O1Yle9TjmVSoLHc1QIYduERuKv4e1yaJx DFgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fvgnML7PsW3yuBKl7xZjxhkiu8/hUKUQrLt8sPfDx2o=; b=jLP6EgHUeqLCRnP3MhyIYae+lG4ji3m/WHuAoclSs/GmxCPqsiKy3WPEpk/2FcWArZ J6WP4aPYqXj2QkUcHAEpp7uz+BJ+t4JX7MXx5VYfDlb0kDtdyLTLeWGAAyhZKM02Pnjc 3leO6GgKdyHlpCpPvBYtuXhTyAlziPvGJNGP/2ZfNXhnusxDW75pps2AriKBixneJxuE dkAPUgzuz0c5///uZ18e5Za1l2tQK5SOnyBt1nwVMbCLqi/ClVagTi/W0038uW9VLo8n b7DKySyTtdaK9A6VucX5J99wjSGbeNAAZ6iVTE2tw8Hv8mgbm59kl8CzGA09zB9IVxxH XdZw== X-Gm-Message-State: AOAM531xSL+WX/KDU5OWmORZWa0U7LBRjmhw/NyF7vcjcyatqCcd1bDu M5pYA+7PIYMi4TJFSRo+rqJ1CA== X-Google-Smtp-Source: ABdhPJwfoWVDnLWg9TdIqXpYSSCtIY2YsTCrcdjU9F3xU3f5D07bEev+3xV58gc2YYHmLVPFIbJD1g== X-Received: by 2002:a17:906:1d5b:: with SMTP id o27mr4460428ejh.344.1592495569497; Thu, 18 Jun 2020 08:52:49 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:49 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 10/12] iommu/arm-smmu-v3: Add SVA device feature Date: Thu, 18 Jun 2020 17:51:23 +0200 Message-Id: <20200618155125.1548969-11-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: F32791814B0C7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement the IOMMU device feature callbacks to support the SVA feature. At the moment dev_has_feat() returns false since I/O Page Faults isn't yet implemented. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 124 ++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e2d5171bfb7b9..d357c9f77bf7f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -698,6 +698,8 @@ struct arm_smmu_master { u32 *sids; unsigned int num_sids; bool ats_enabled; + bool sva_enabled; + struct list_head bonds; unsigned int ssid_bits; }; @@ -3000,6 +3002,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; + /* + * Checking that SVA is disabled ensures that this device isn't bound to + * any mm, and can be safely detached from its old domain. Bonds cannot + * be removed concurrently since we're holding the group mutex. + */ + mutex_lock(&sva_lock); + if (master->sva_enabled) { + mutex_unlock(&sva_lock); + dev_err(dev, "cannot attach - SVA enabled\n"); + return -EBUSY; + } + mutex_unlock(&sva_lock); + arm_smmu_detach_dev(master); mutex_lock(&smmu_domain->init_mutex); @@ -3147,6 +3162,7 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->smmu = smmu; master->sids = fwspec->ids; master->num_sids = fwspec->num_ids; + INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); /* Check the SIDs are in range of the SMMU and our stream table */ @@ -3199,6 +3215,7 @@ static void arm_smmu_release_device(struct device *dev) return; master = dev_iommu_priv_get(dev); + WARN_ON(master->sva_enabled); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); kfree(master); @@ -3316,6 +3333,109 @@ static void arm_smmu_get_resv_regions(struct device *dev, iommu_dma_get_resv_regions(dev, head); } +static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) +{ + return false; +} + +static bool arm_smmu_dev_has_feature(struct device *dev, + enum iommu_dev_features feat) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) + return false; + + /* SSID and IOPF support are mandatory for the moment */ + return master->ssid_bits && arm_smmu_iopf_supported(master); + default: + return false; + } +} + +static bool arm_smmu_dev_feature_enabled(struct device *dev, + enum iommu_dev_features feat) +{ + bool enabled = false; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + mutex_lock(&sva_lock); + enabled = master->sva_enabled; + mutex_unlock(&sva_lock); + return enabled; + default: + return false; + } +} + +static int arm_smmu_dev_enable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + master->sva_enabled = true; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_disable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + if (!list_empty(&master->bonds)) { + dev_err(dev, "cannot disable SVA, device is bound\n"); + mutex_unlock(&sva_lock); + return -EBUSY; + } + master->sva_enabled = false; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_enable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_has_feature(dev, feat)) + return -ENODEV; + + if (arm_smmu_dev_feature_enabled(dev, feat)) + return -EBUSY; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_enable_sva(dev); + default: + return -EINVAL; + } +} + +static int arm_smmu_dev_disable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_feature_enabled(dev, feat)) + return -EINVAL; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_disable_sva(dev); + default: + return -EINVAL; + } +} + static struct iommu_ops arm_smmu_ops = { .capable = arm_smmu_capable, .domain_alloc = arm_smmu_domain_alloc, @@ -3334,6 +3454,10 @@ static struct iommu_ops arm_smmu_ops = { .of_xlate = arm_smmu_of_xlate, .get_resv_regions = arm_smmu_get_resv_regions, .put_resv_regions = generic_iommu_put_resv_regions, + .dev_has_feat = arm_smmu_dev_has_feature, + .dev_feat_enabled = arm_smmu_dev_feature_enabled, + .dev_enable_feat = arm_smmu_dev_enable_feature, + .dev_disable_feat = arm_smmu_dev_disable_feature, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; From patchwork Thu Jun 18 15:51:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DD2592A for ; Thu, 18 Jun 2020 15:53:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C5D9D2070A for ; Thu, 18 Jun 2020 15:53:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="F3rq2hPx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5D9D2070A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F3AB18D003B; Thu, 18 Jun 2020 11:52:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EC4448D0018; Thu, 18 Jun 2020 11:52:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8ADD8D003B; Thu, 18 Jun 2020 11:52:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id BD1338D0018 for ; Thu, 18 Jun 2020 11:52:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8075F824556B for ; Thu, 18 Jun 2020 15:52:52 +0000 (UTC) X-FDA: 76942775784.23.wren60_4112b3726e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 480AE37609 for ; Thu, 18 Jun 2020 15:52:52 +0000 (UTC) X-Spam-Summary: 2,0,0,f45b5f5b69e50a3c,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:4:41:355:379:541:800:960:966:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1500:1515:1605:1730:1747:1777:1792:2194:2196:2199:2200:2393:2553:2559:2562:2639:2693:2736:2895:2901:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4250:4321:4385:4605:5007:6119:6261:6653:6742:7874:7903:7904:9010:10004:11026:11473:11658:11914:12043:12114:12291:12294:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:13972:14096:14394:21080:21324:21433:21444:21451:21627:21789:21987:21990:30003:30029:30054:30070:30090,0,RBL:209.85.218.67:@linaro.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:4,LUA_SUMMARY:none X-HE-Tag: wren60_4112b3726e11 X-Filterd-Recvd-Size: 16349 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:51 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id dr13so6951608ejc.3 for ; Thu, 18 Jun 2020 08:52:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZuZSIKZP3e6lWCHyBpmCAdFcfA2fkGVm1e+K1v+Ant0=; b=F3rq2hPxOLLhcTkt0qXcmgkdti/SxA0Grlpb18yg1aqay3xp7saO7yaI5cJDrwNGU9 7YzJSsMWNHBgA8tjL/TKszprqUhJZ2rxejsrejBhQXpziLn58+b6UTqLuiAYehEKmkU9 i/Z3FzVcQRpWCmkkDlz0h0M9NhLsCLrMbDJCcyugK58ZkQzG7PYygeDXpUxg6AMOOuAT ElVKAq+V4tbKwK8OAuGOaK4LkH+lYtozfz0PzAt+DxmXYWvkE413Uj+2gdjl2jJTdKDJ YnUCvfWN1cC5XWokXC50laaYfF7W2292OEVTTaBN0HEHVpHaZErrJIg1q4mhOAiF9pHg BKIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZuZSIKZP3e6lWCHyBpmCAdFcfA2fkGVm1e+K1v+Ant0=; b=Pt5YB4zkKds4OEP3r5uqRkLHZyufWrNAuejTCuYBGFmQ4mzHxSPJ5QP7yQ5vVKm/5f Oq4nyyyCfsMnLSmm3ZJgAS7VGFpflh/MpNGNikqxAbkrlKD6fUWebul7U+gNNf4jCn2e xZE+RPwxCVa7MlmFLp2ikN9GgZkWI4s/OsDvQVM3/J6Cr0jibItg2OeXklxe2D6B2nHg yy0WtzvGtxZkrCMVHh95MykrmCVa8woy7J3zGHvZg3fhCsH8aKENNv6GmMMtTfMHquWT a0TvVcL+Y6NM7BTX2mxfqGjnWV+E7VLcHYZeDQasMgCGjW4oGEBuvZ6Dz3eFu5C+zDsy yCpw== X-Gm-Message-State: AOAM532IXht6NwGu00oexXlauqoPAV2ymc3cjUjhOhA7cCLaWusCWYWo 5JnWBEcOSDa+o2lchK4uJu/2rA== X-Google-Smtp-Source: ABdhPJxCAIkVq2rLjUObspQLQB8nCyvTQOzGP/GfT0DoAcdczzKtcZexhFXO/NIEgpFwmTF2mlsfag== X-Received: by 2002:a17:906:241b:: with SMTP id z27mr4457689eja.267.1592495570579; Thu, 18 Jun 2020 08:52:50 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:50 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 11/12] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Date: Thu, 18 Jun 2020 17:51:24 +0200 Message-Id: <20200618155125.1548969-12-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 480AE37609 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The sva_bind() function allows devices to access process address spaces using a PASID (aka SSID). (1) bind() allocates or gets an existing MMU notifier tied to the (domain, mm) pair. Each mm gets one PASID. (2) Any change to the address space calls invalidate_range() which sends ATC invalidations (in a subsequent patch). (3) When the process address space dies, the release() notifier disables the CD to allow reclaiming the page tables. Since release() has to be light we do not instruct device drivers to stop DMA here, we just ignore incoming page faults from this point onwards. To avoid any event 0x0a print (C_BAD_CD) we disable translation without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. Don't clear the R bit because the S bit can't be cleared when STALL_MODEL==0b10 (forced), and clearing R without clearing S is useless. Faulting transactions will stall and will be aborted by the IOPF handler. (4) After stopping DMA, the device driver releases the bond by calling unbind(). We release the MMU notifier, free the PASID and the bond. Three structures keep track of bonds: * arm_smmu_bond: one per {device, mm} pair, the handle returned to the device driver for a bind() request. * arm_smmu_mmu_notifier: one per {domain, mm} pair, deals with ATS/TLB invalidations and clearing the context descriptor on mm exit. * arm_smmu_ctx_desc: one per mm, holds the pinned ASID and pgd. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/Kconfig | 2 + drivers/iommu/arm-smmu-v3.c | 263 +++++++++++++++++++++++++++++++++++- 2 files changed, 260 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 74a10e7a8d082..d1ad6f63d2d42 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -437,8 +437,10 @@ config ARM_SMMU_V3 tristate "ARM Ltd. System MMU Version 3 (SMMUv3) Support" depends on ARM64 select IOMMU_API + select IOMMU_SVA_LIB select IOMMU_IO_PGTABLE_LPAE select GENERIC_MSI_IRQ_DOMAIN + select MMU_NOTIFIER help Support for implementations of the ARM System MMU architecture version 3 providing translation support to a PCIe root complex. diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index d357c9f77bf7f..af551f3c78a78 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -35,6 +36,7 @@ #include #include "io-pgtable-arm.h" +#include "iommu-sva-lib.h" /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 @@ -729,8 +731,32 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; + + struct list_head mmu_notifiers; +}; + +struct arm_smmu_mmu_notifier { + struct mmu_notifier mn; + struct arm_smmu_ctx_desc *cd; + bool cleared; + refcount_t refs; + struct list_head list; + struct arm_smmu_domain *domain; }; +#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) + +struct arm_smmu_bond { + struct iommu_sva sva; + struct mm_struct *mm; + struct arm_smmu_mmu_notifier *smmu_mn; + struct list_head list; + refcount_t refs; +}; + +#define sva_to_bond(handle) \ + container_of(handle, struct arm_smmu_bond, sva) + struct arm_smmu_option_prop { u32 opt; const char *prop; @@ -740,6 +766,13 @@ static DEFINE_XARRAY_ALLOC1(asid_xa); static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); +/* + * When a process dies, DMA is still running but we need to clear the pgd. If we + * simply cleared the valid bit from the context descriptor, we'd get event 0x0a + * which are not recoverable. + */ +static struct arm_smmu_ctx_desc invalid_cd = { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1643,7 +1676,9 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -1660,8 +1695,10 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val = 0; + } else if (cd == &invalid_cd) { /* (4) */ + val |= CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1861,7 +1898,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return NULL; } -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; @@ -1958,7 +1994,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return ERR_PTR(ret); } -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { lockdep_assert_held(&sva_lock); @@ -2591,6 +2626,8 @@ static bool arm_smmu_capable(enum iommu_cap cap) } } +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops; + static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) { struct arm_smmu_domain *smmu_domain; @@ -2618,6 +2655,7 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; } @@ -3114,6 +3152,207 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) return ops->iova_to_phys(ops, iova); } +static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + mutex_lock(&sva_lock); + if (smmu_mn->cleared) { + mutex_unlock(&sva_lock); + return; + } + + /* + * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, + * but disable translation. + */ + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &invalid_cd); + + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + + smmu_mn->cleared = true; + mutex_unlock(&sva_lock); +} + +static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) +{ + kfree(mn_to_smmu(mn)); +} + +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, +}; + +/* Allocate or get existing MMU notifier for this {domain, mm} pair */ +static struct arm_smmu_mmu_notifier * +arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + int ret; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_mmu_notifier *smmu_mn; + + lockdep_assert_held(&sva_lock); + + list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { + if (smmu_mn->mn.mm == mm) { + refcount_inc(&smmu_mn->refs); + return smmu_mn; + } + } + + cd = arm_smmu_alloc_shared_cd(mm); + if (IS_ERR(cd)) + return ERR_CAST(cd); + + smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); + if (!smmu_mn) { + ret = -ENOMEM; + goto err_free_cd; + } + + refcount_set(&smmu_mn->refs, 1); + smmu_mn->cd = cd; + smmu_mn->domain = smmu_domain; + smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; + + ret = mmu_notifier_register(&smmu_mn->mn, mm); + if (ret) { + kfree(smmu_mn); + goto err_free_cd; + } + + ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + if (ret) + goto err_put_notifier; + + list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); + return smmu_mn; + +err_put_notifier: + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); +err_free_cd: + arm_smmu_free_shared_cd(cd); + return ERR_PTR(ret); +} + +static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +{ + struct mm_struct *mm = smmu_mn->mn.mm; + struct arm_smmu_ctx_desc *cd = smmu_mn->cd; + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + lockdep_assert_held(&sva_lock); + + if (!refcount_dec_and_test(&smmu_mn->refs)) + return; + + list_del(&smmu_mn->list); + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + + /* + * If we went through clear(), we've already invalidated, and no + * new TLB entry can have been formed. + */ + if (!smmu_mn->cleared) + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); + arm_smmu_free_shared_cd(cd); +} + +static struct iommu_sva * +__arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +{ + int ret; + struct arm_smmu_bond *bond; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + lockdep_assert_held(&sva_lock); + + if (!master || !master->sva_enabled) + return ERR_PTR(-ENODEV); + + /* If bind() was already called for this {dev, mm} pair, reuse it. */ + list_for_each_entry(bond, &master->bonds, list) { + if (bond->mm == mm) { + refcount_inc(&bond->refs); + return &bond->sva; + } + } + + bond = kzalloc(sizeof(*bond), GFP_KERNEL); + if (!bond) + return ERR_PTR(-ENOMEM); + + /* Allocate a PASID for this mm if necessary */ + ret = iommu_sva_alloc_pasid(mm, 1, (1U << master->ssid_bits) - 1); + if (ret) + goto err_free_bond; + + bond->mm = mm; + bond->sva.dev = dev; + refcount_set(&bond->refs, 1); + + bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + if (IS_ERR(bond->smmu_mn)) { + ret = PTR_ERR(bond->smmu_mn); + goto err_free_pasid; + } + + list_add(&bond->list, &master->bonds); + return &bond->sva; + +err_free_pasid: + iommu_sva_free_pasid(mm); +err_free_bond: + kfree(bond); + return ERR_PTR(ret); +} + +static struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + struct iommu_sva *handle; + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + mutex_lock(&sva_lock); + handle = __arm_smmu_sva_bind(dev, mm); + mutex_unlock(&sva_lock); + return handle; +} + +static void arm_smmu_sva_unbind(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + mutex_lock(&sva_lock); + if (refcount_dec_and_test(&bond->refs)) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + iommu_sva_free_pasid(bond->mm); + kfree(bond); + } + mutex_unlock(&sva_lock); +} + +static int arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + return bond->mm->pasid; +} + static struct platform_driver arm_smmu_driver; static @@ -3458,6 +3697,9 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, + .sva_bind = arm_smmu_sva_bind, + .sva_unbind = arm_smmu_sva_unbind, + .sva_get_pasid = arm_smmu_sva_get_pasid, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -4520,6 +4762,16 @@ static const struct of_device_id arm_smmu_of_match[] = { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + /* + * Wait for all notifiers free() RCU callbacks, since they are still + * using the arm_smmu_mmu_notifier_ops. + */ + mmu_notifier_synchronize(); + platform_driver_unregister(drv); +} + static struct platform_driver arm_smmu_driver = { .driver = { .name = "arm-smmu-v3", @@ -4530,7 +4782,8 @@ static struct platform_driver arm_smmu_driver = { .remove = arm_smmu_device_remove, .shutdown = arm_smmu_device_shutdown, }; -module_platform_driver(arm_smmu_driver); +module_driver(arm_smmu_driver, platform_driver_register, + arm_smmu_driver_unregister); MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); From patchwork Thu Jun 18 15:51:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612723 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE4B092A for ; Thu, 18 Jun 2020 15:53:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9959620732 for ; Thu, 18 Jun 2020 15:53:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="DYibauvJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9959620732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DD9648D0040; Thu, 18 Jun 2020 11:52:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D87F38D0018; Thu, 18 Jun 2020 11:52:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9D9F8D0040; Thu, 18 Jun 2020 11:52:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0061.hostedemail.com [216.40.44.61]) by kanga.kvack.org (Postfix) with ESMTP id B32938D0018 for ; Thu, 18 Jun 2020 11:52:53 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7C562180AD807 for ; Thu, 18 Jun 2020 15:52:53 +0000 (UTC) X-FDA: 76942775826.28.death61_250f88826e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 5B56F6124 for ; Thu, 18 Jun 2020 15:52:53 +0000 (UTC) X-Spam-Summary: 2,0,0,ec470cba03c41983,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3874:4118:4250:4321:4385:5007:6119:6261:6653:6742:7862:7874:7875:7901:8603:9040:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13161:13229:13230:13894:14096:14110:14181:14394:14721:21080:21211:21444:21627:21987:21990:30054:30070,0,RBL:209.85.218.65:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: death61_250f88826e11 X-Filterd-Recvd-Size: 7207 Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 15:52:52 +0000 (UTC) Received: by mail-ej1-f65.google.com with SMTP id dr13so6951661ejc.3 for ; Thu, 18 Jun 2020 08:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K0sD9muywK0nbycZHdsqSBZ4T+dLVPNn3EX5mSicmkA=; b=DYibauvJ2nimmPQCzKZsqFWxsw1cJOa83Q8lBOizOX8qC2sph7xBeWfJVDLqHcPsZT jUpy1d/5G5neFk7+aPYKN3NiWZz2OD8ek+65tg6OyKTdhesLhyiouRddaM9PTbG2zSM0 vNbXEBv/Zfu7Xpa9afY1wlNWbf7NYURm9ezvJnPuhwmG6QXxlemJYFcE9cyW4NEcWZin j2SQ+U1tQkwf6J+OWfDQA3p4lIuVm/Opa2zG6+W+u6svNiCqSklKXaS1QWT1/RMzMNp0 SPJ6ObCtmIf61+7zubUKVxAmdcChrBc908ruLceXol1cTPgQeBcB+yXy6tuW8+XaC7gk FL3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K0sD9muywK0nbycZHdsqSBZ4T+dLVPNn3EX5mSicmkA=; b=evZeCeR4jrw/x9kiShztw9TtlE70IIuvGyqrcBDbI75bLh1FgpPxGvf4tcYCKyVvy8 o+aHFP1stD/JBA8CKP7kX2KO8Y3GkZxW0kZylwc8wgqvrB/O5iW9pBNxYVhllS5WAQHs tBmlnbZmiiGf7kyK/Qeh8qUvUXIX7PXVgWIZFvKMwGSl32qclROB0oD4O//nKyf/fT8E fq2y+xm3xRSQat7bILICqwjotqd8jlqfIb4ST6zOYXT0QOSB8RX8h/Gut27vMUeMbAN5 9czVrHWoXiuug3+66vohCLTRAurx/4zJFqMZNKmyc2vRyWyB9aI4of0DQxzLTIK/9yyZ 54kg== X-Gm-Message-State: AOAM532SKr0MxnOTw2/p+cXNwgXprb8r8q2DFsfUaB7y3FbH/TWAHQqR W+O+rG57Czss2emuvBImjD+CLw== X-Google-Smtp-Source: ABdhPJyPJTA3MoAcCwZxTHDutpdNm45kp002C6I1TJUitRaZbiXuqClT8u8iuWaKBu95Oo1397Yh/g== X-Received: by 2002:a17:906:ae85:: with SMTP id md5mr4476555ejb.213.1592495571768; Thu, 18 Jun 2020 08:52:51 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:51 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, zhangfei.gao@linaro.org, xuzaibo@huawei.com, zhengxiang9@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v8 12/12] iommu/arm-smmu-v3: Hook up ATC invalidation to mm ops Date: Thu, 18 Jun 2020 17:51:25 +0200 Message-Id: <20200618155125.1548969-13-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5B56F6124 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The invalidate_range() notifier is called for any change to the address space. Perform the required ATC invalidations. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index af551f3c78a78..972c061399fc9 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -2378,6 +2378,20 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, size_t inval_grain_shift = 12; unsigned long page_start, page_end; + /* + * ATS and PASID: + * + * If substream_valid is clear, the PCIe TLP is sent without a PASID + * prefix. In that case all ATC entries within the address range are + * invalidated, including those that were requested with a PASID! There + * is no way to invalidate only entries without PASID. + * + * When using STRTAB_STE_1_S1DSS_SSID0 (reserving CD 0 for non-PASID + * traffic), translation requests without PASID create ATC entries + * without PASID, which must be invalidated with substream_valid clear. + * This has the unpleasant side-effect of invalidating all PASID-tagged + * ATC entries within the address range. + */ *cmd = (struct arm_smmu_cmdq_ent) { .opcode = CMDQ_OP_ATC_INV, .substream_valid = !!ssid, @@ -2421,12 +2435,12 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, cmd->atc.size = log2_span; } -static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) +static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, int ssid) { int i; struct arm_smmu_cmdq_ent cmd; - arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); + arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd); for (i = 0; i < master->num_sids; i++) { cmd.atc.sid = master->sids[i]; @@ -2955,7 +2969,7 @@ static void arm_smmu_disable_ats(struct arm_smmu_master *master) * ATC invalidation via the SMMU. */ wmb(); - arm_smmu_atc_inv_master(master); + arm_smmu_atc_inv_master(master, 0); atomic_dec(&smmu_domain->nr_ats_masters); } @@ -3152,6 +3166,16 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) return ops->iova_to_phys(ops, iova); } +static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + + arm_smmu_atc_inv_domain(smmu_mn->domain, mm->pasid, start, + end - start + 1); +} + static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); @@ -3170,6 +3194,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &invalid_cd); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -3181,6 +3206,7 @@ static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) } static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .invalidate_range = arm_smmu_mm_invalidate_range, .release = arm_smmu_mm_release, .free_notifier = arm_smmu_mmu_notifier_free, }; @@ -3257,8 +3283,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) * If we went through clear(), we've already invalidated, and no * new TLB entry can have been formed. */ - if (!smmu_mn->cleared) + if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + } /* Frees smmu_mn */ mmu_notifier_put(&smmu_mn->mn);