From patchwork Wed Nov 16 03:16:12 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 9430851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 40DF260469 for ; Wed, 16 Nov 2016 03:16:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2C25328DB9 for ; Wed, 16 Nov 2016 03:16:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 210E028DBC; Wed, 16 Nov 2016 03:16:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D4F828DB9 for ; Wed, 16 Nov 2016 03:16:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753547AbcKPDQR (ORCPT ); Tue, 15 Nov 2016 22:16:17 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39378 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751982AbcKPDQP (ORCPT ); Tue, 15 Nov 2016 22:16:15 -0500 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 08DF62BA0EF; Wed, 16 Nov 2016 03:16:14 +0000 (UTC) Received: from t450s.home (ovpn03.gateway.prod.ext.phx2.redhat.com [10.5.9.3]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id uAG3GDtj025899; Tue, 15 Nov 2016 22:16:13 -0500 Date: Tue, 15 Nov 2016 20:16:12 -0700 From: Alex Williamson To: Kirti Wankhede Cc: , , , , , , , , Subject: Re: [PATCH v13 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Message-ID: <20161115201612.103893d7@t450s.home> In-Reply-To: References: <1479223805-22895-1-git-send-email-kwankhede@nvidia.com> <1479223805-22895-12-git-send-email-kwankhede@nvidia.com> <20161115151950.1e8ab7d6@t450s.home> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 16 Nov 2016 03:16:14 +0000 (UTC) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, 16 Nov 2016 08:16:15 +0530 Kirti Wankhede wrote: > On 11/16/2016 3:49 AM, Alex Williamson wrote: > > On Tue, 15 Nov 2016 20:59:54 +0530 > > Kirti Wankhede wrote: > > > ... > > >> @@ -854,7 +857,28 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, > >> */ > >> if (dma->task->mm != current->mm) > >> break; > >> + > >> unmapped += dma->size; > >> + > >> + if (iommu->external_domain && !RB_EMPTY_ROOT(&dma->pfn_list)) { > >> + struct vfio_iommu_type1_dma_unmap nb_unmap; > >> + > >> + nb_unmap.iova = dma->iova; > >> + nb_unmap.size = dma->size; > >> + > >> + /* > >> + * Notifier callback would call vfio_unpin_pages() which > >> + * would acquire iommu->lock. Release lock here and > >> + * reacquire it again. > >> + */ > >> + mutex_unlock(&iommu->lock); > >> + blocking_notifier_call_chain(&iommu->notifier, > >> + VFIO_IOMMU_NOTIFY_DMA_UNMAP, > >> + &nb_unmap); > >> + mutex_lock(&iommu->lock); > >> + if (WARN_ON(!RB_EMPTY_ROOT(&dma->pfn_list))) > >> + break; > >> + } > > > > > > Why exactly do we need to notify per vfio_dma rather than per unmap > > request? If we do the latter we can send the notify first, limiting us > > to races where a page is pinned between the notify and the locking, > > whereas here, even our dma pointer is suspect once we re-acquire the > > lock, we don't technically know if another unmap could have removed > > that already. Perhaps something like this (untested): > > > > There are checks to validate unmap request, like v2 check and who is > calling unmap and is it allowed for that task to unmap. Before these > checks its not sure that unmap region range which asked for would be > unmapped all. Notify call should be at the place where its sure that the > range provided to notify call is definitely going to be removed. My > change do that. Ok, but that does solve the problem. What about this (untested): --- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index ee9a680..50cafdf 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -782,9 +782,9 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, struct vfio_iommu_type1_dma_unmap *unmap) { uint64_t mask; - struct vfio_dma *dma; + struct vfio_dma *dma, *dma_last = NULL; size_t unmapped = 0; - int ret = 0; + int ret = 0, retries; mask = ((uint64_t)1 << __ffs(vfio_pgsize_bitmap(iommu))) - 1; @@ -794,7 +794,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, return -EINVAL; WARN_ON(mask & PAGE_MASK); - +again: mutex_lock(&iommu->lock); /* @@ -851,11 +851,16 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, if (dma->task->mm != current->mm) break; - unmapped += dma->size; - - if (iommu->external_domain && !RB_EMPTY_ROOT(&dma->pfn_list)) { + if (!RB_EMPTY_ROOT(&dma->pfn_list)) { struct vfio_iommu_type1_dma_unmap nb_unmap; + if (dma_last == dma) { + BUG_ON(++retries > 10); + } else { + dma_last = dma; + retries = 0; + } + nb_unmap.iova = dma->iova; nb_unmap.size = dma->size; @@ -868,11 +873,11 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, blocking_notifier_call_chain(&iommu->notifier, VFIO_IOMMU_NOTIFY_DMA_UNMAP, &nb_unmap); - mutex_lock(&iommu->lock); - if (WARN_ON(!RB_EMPTY_ROOT(&dma->pfn_list))) - break; + goto again: } + unmapped += dma->size; vfio_remove_dma(iommu, dma); + } unlock: