From patchwork Tue Feb 18 22:22:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980973 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60D7F17A309 for ; Tue, 18 Feb 2025 22:22:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917350; cv=none; b=Fc/+BzAuwYJqs/6XUKg02znz3sUBW8V+l+6OMjM0N04PkFhJyCEnFZBx+nHq2UnxjsZKyns0x0mRXQ8b2D2+fDcBM3YeWDscGziLgGRqeoamdKooP6kVYGorZwH37nUlax0jLHcD1IxxnDh3iFtBz1UoRLUfsKzoTebmk5qjzN0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917350; c=relaxed/simple; bh=OFxtNMO544u4QXflyKyJc+PEeJaBK8eyu4Fj+6JaLc8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c/0AKVV87imHqixs2jULPw4nmc/qWqxPbIsw5X0LfG0cD2q40KiYOl/1prURjhYalOzrPj79ORUI1ZJnrhTeVI6IoH6ee384dZJAo/wycaLTR5+D+wk7w5GgxLcC+9eyi/BZ8PNpHo+bl2qD/Ori4lj0gGMl/qwzJ5+qI85i6i8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ANYQoFGG; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ANYQoFGG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z8/iHQhrJ/aZGVyYP8XcAahgAPdzcbyxdw/jYg/Ap0s=; b=ANYQoFGG6jwphE+gCsiQAQlVC2GprNd4qAXo6VnENCKyjZrhfNNML8d5sKdl9F6xXT4HWy YJ8P+2YhXXhuo4Lr0TfRzwd2eylTwFnTn86IZsM6iOcjd7UxmzXzoJOvL3SA4vOSXUv/au m0RbrTvWXmGoWuNOjG3cMylFMf975PA= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-641-keIo2oqgMQmrNKfT2a6hWQ-1; Tue, 18 Feb 2025 17:22:23 -0500 X-MC-Unique: keIo2oqgMQmrNKfT2a6hWQ-1 X-Mimecast-MFC-AGG-ID: keIo2oqgMQmrNKfT2a6hWQ_1739917342 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A6950180087A; Tue, 18 Feb 2025 22:22:22 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3A65D30001A5; Tue, 18 Feb 2025 22:22:20 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com Subject: [PATCH v2 1/6] vfio/type1: Catch zero from pin_user_pages_remote() Date: Tue, 18 Feb 2025 15:22:01 -0700 Message-ID: <20250218222209.1382449-2-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 pin_user_pages_remote() can currently return zero for invalid args or zero nr_pages, neither of which should ever happen. However vaddr_get_pfns() indicates it should only ever return a positive value or -errno and there's a theoretical case where this can slip through and be unhandled by callers. Therefore convert zero to -EFAULT. Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" Reviewed-by: Jason Gunthorpe Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 50ebc9593c9d..119cf886d8c0 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -564,6 +564,8 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, if (ret > 0) { *pfn = page_to_pfn(pages[0]); goto done; + } else if (!ret) { + ret = -EFAULT; } vaddr = untagged_addr_remote(mm, vaddr); From patchwork Tue Feb 18 22:22:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980974 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67D9D28629B for ; Tue, 18 Feb 2025 22:22:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917351; cv=none; b=V7Rp3+i/B6RPh5fZ2pmwKZckz0Yf2oT4XEvk3HckzWytdi1mweUIZ7RA31PKhV6/Ho6wIWRRf/VgXHVIBZhgSvNadOshgbpFkfQlUtKVHrFsN5DSFxuyN6YzBMqYt6gpHYtB9Iy7F+5UEUSHq7AnGkqt4o2iHep56WDNHpjEuFY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917351; c=relaxed/simple; bh=wz3yzYejx/8v4irersUarogzfJY+Deu6ePxVH2PlB2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qrYqi/a8gWdAo87m64WCvO0d9h+Eytv/Xu3FAHySGhcRR3N55Mriwdz1zRir9+Z6Mpjv+mc2rONfxcQezF34BhcCQdFVzPaDP1aFl0agGWLCJZu36af6NB30M7eiyZU2LSUdT+Kia0c1OoeV7r6HTfM3u6wjU9tdtKdlcmVXjqw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=W/HPxvC3; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="W/HPxvC3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PDNwQ6cJzqO/P0dW5wK2COewyumLqAUhMRnX8cwH+GU=; b=W/HPxvC38MpERWzipgGLL7BNWOt4sA5DniXMG6WK3Gm+t3av1EYIwwuEIJtA1QlWYb9yTw FJAt/u8lR82TVhav17TFfxPqORyIDlkMSY3KhCMvdlEbI6kgTyJuzEM1BCmrcILL+ytf6R 20I9VFXty8+sOFHliNiac81EpFoK2Co= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-665-cTjyOuWhM3StQl89hVlG0w-1; Tue, 18 Feb 2025 17:22:26 -0500 X-MC-Unique: cTjyOuWhM3StQl89hVlG0w-1 X-Mimecast-MFC-AGG-ID: cTjyOuWhM3StQl89hVlG0w_1739917345 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F397B1800874; Tue, 18 Feb 2025 22:22:24 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 08A9830001A5; Tue, 18 Feb 2025 22:22:22 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com Subject: [PATCH v2 2/6] vfio/type1: Convert all vaddr_get_pfns() callers to use vfio_batch Date: Tue, 18 Feb 2025 15:22:02 -0700 Message-ID: <20250218222209.1382449-3-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 This is a step towards passing the structure to vaddr_get_pfns() directly in order to provide greater distinction between page backed pfns and pfnmaps. Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 119cf886d8c0..2e95f5f4d881 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -471,12 +471,12 @@ static int put_pfn(unsigned long pfn, int prot) #define VFIO_BATCH_MAX_CAPACITY (PAGE_SIZE / sizeof(struct page *)) -static void vfio_batch_init(struct vfio_batch *batch) +static void __vfio_batch_init(struct vfio_batch *batch, bool single) { batch->size = 0; batch->offset = 0; - if (unlikely(disable_hugepages)) + if (single || unlikely(disable_hugepages)) goto fallback; batch->pages = (struct page **) __get_free_page(GFP_KERNEL); @@ -491,6 +491,16 @@ static void vfio_batch_init(struct vfio_batch *batch) batch->capacity = 1; } +static void vfio_batch_init(struct vfio_batch *batch) +{ + __vfio_batch_init(batch, false); +} + +static void vfio_batch_init_single(struct vfio_batch *batch) +{ + __vfio_batch_init(batch, true); +} + static void vfio_batch_unpin(struct vfio_batch *batch, struct vfio_dma *dma) { while (batch->size) { @@ -730,7 +740,7 @@ static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr, unsigned long *pfn_base, bool do_accounting) { - struct page *pages[1]; + struct vfio_batch batch; struct mm_struct *mm; int ret; @@ -738,7 +748,9 @@ static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr, if (!mmget_not_zero(mm)) return -ENODEV; - ret = vaddr_get_pfns(mm, vaddr, 1, dma->prot, pfn_base, pages); + vfio_batch_init_single(&batch); + + ret = vaddr_get_pfns(mm, vaddr, 1, dma->prot, pfn_base, batch.pages); if (ret != 1) goto out; @@ -757,6 +769,7 @@ static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr, } out: + vfio_batch_fini(&batch); mmput(mm); return ret; } From patchwork Tue Feb 18 22:22:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980975 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B1B61D86D6 for ; Tue, 18 Feb 2025 22:22:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917353; cv=none; b=te5D/daqZ4ycYETnX0qrBZUe3w2QCKGvbhyVsxs41EIwmcZLPUa6J2yHZjrqFzLISI6eX7xZGreVa33qxbyjT2NUXchcDqvB/ai4F8ezFc2nNt6MHzkJFKgg2jcHzLgmatmAzJ2w+jdPO33jjzbz5bcLMM2jyrgucYnjP6ScCR8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917353; c=relaxed/simple; bh=uf8vkAxRv8FmnfC2BO2dkQyzqST1cQSOXl1P4++13Xo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W2Pfo3mBiAJP2T9avBhAFMcZIZ9X8xGtYormWUDYznJuRx6f173EyGocnYbMuQBhapL4Y08AOhwjizUB01Vq9YVhHxtFcrsa4M7COX62pWBboddVHrKDsHrwb3M45Bp2gyoGHlsh0ERIbmvgKI0H7O/g7fEi+b6GpvC/RK6NFOw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ixlMTSJe; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ixlMTSJe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tc3MDK0ssjIqR64/Cyp3saXm6VeBQzlC8zqjX2lM+Zo=; b=ixlMTSJelYn300LK8UI1FpXTQNVknn4nby1c6MvMydZCk4pTk4Y6dbQp5xcscSgk3Z4l8Z +tSGCQLKMjoSkK9pxaL9I7bCgj2zJHdQIGjJuZeUcE3chXLWVIiPbGITyxk7jpB+dszHWC yA9GwS3YMa+l8ih2H9jUs7OOhvMCqfo= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-679-yWn53nA9NteBtkAH18firQ-1; Tue, 18 Feb 2025 17:22:28 -0500 X-MC-Unique: yWn53nA9NteBtkAH18firQ-1 X-Mimecast-MFC-AGG-ID: yWn53nA9NteBtkAH18firQ_1739917347 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B5CBD180036F; Tue, 18 Feb 2025 22:22:27 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 440B4300019F; Tue, 18 Feb 2025 22:22:25 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com Subject: [PATCH v2 3/6] vfio/type1: Use vfio_batch for vaddr_get_pfns() Date: Tue, 18 Feb 2025 15:22:03 -0700 Message-ID: <20250218222209.1382449-4-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Passing the vfio_batch to vaddr_get_pfns() allows for greater distinction between page backed pfns and pfnmaps. In the case of page backed pfns, vfio_batch.size is set to a positive value matching the number of pages filled in vfio_batch.pages. For a pfnmap, vfio_batch.size remains zero as vfio_batch.pages are not used. In both cases the return value continues to indicate the number of pfns and the provided pfn arg is set to the initial pfn value. This allows us to shortcut the pfnmap case, which is detected by the zero vfio_batch.size. pfnmaps do not contribute to locked memory accounting, therefore we can update counters and continue directly, which also enables a future where vaddr_get_pfns() can return a value greater than one for consecutive pfnmaps. NB. Now that we're not guessing whether the initial pfn is page backed or pfnmap, we no longer need to special case the put_pfn() and batch size reset. It's safe for vfio_batch_unpin() to handle this case. Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 63 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 2e95f5f4d881..fafd8af125c7 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -555,12 +555,16 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, /* * Returns the positive number of pfns successfully obtained or a negative - * error code. + * error code. The initial pfn is stored in the pfn arg. For page-backed + * pfns, the provided batch is also updated to indicate the filled pages and + * initial offset. For VM_PFNMAP pfns, only the returned number of pfns and + * returned initial pfn are provided; subsequent pfns are contiguous. */ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, long npages, int prot, unsigned long *pfn, - struct page **pages) + struct vfio_batch *batch) { + long pin_pages = min_t(long, npages, batch->capacity); struct vm_area_struct *vma; unsigned int flags = 0; int ret; @@ -569,10 +573,12 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, flags |= FOLL_WRITE; mmap_read_lock(mm); - ret = pin_user_pages_remote(mm, vaddr, npages, flags | FOLL_LONGTERM, - pages, NULL); + ret = pin_user_pages_remote(mm, vaddr, pin_pages, flags | FOLL_LONGTERM, + batch->pages, NULL); if (ret > 0) { - *pfn = page_to_pfn(pages[0]); + *pfn = page_to_pfn(batch->pages[0]); + batch->size = ret; + batch->offset = 0; goto done; } else if (!ret) { ret = -EFAULT; @@ -628,32 +634,42 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, *pfn_base = 0; } + if (unlikely(disable_hugepages)) + npage = 1; + while (npage) { if (!batch->size) { /* Empty batch, so refill it. */ - long req_pages = min_t(long, npage, batch->capacity); - - ret = vaddr_get_pfns(mm, vaddr, req_pages, dma->prot, - &pfn, batch->pages); + ret = vaddr_get_pfns(mm, vaddr, npage, dma->prot, + &pfn, batch); if (ret < 0) goto unpin_out; - batch->size = ret; - batch->offset = 0; - if (!*pfn_base) { *pfn_base = pfn; rsvd = is_invalid_reserved_pfn(*pfn_base); } + + /* Handle pfnmap */ + if (!batch->size) { + if (pfn != *pfn_base + pinned || !rsvd) + goto out; + + pinned += ret; + npage -= ret; + vaddr += (PAGE_SIZE * ret); + iova += (PAGE_SIZE * ret); + continue; + } } /* - * pfn is preset for the first iteration of this inner loop and - * updated at the end to handle a VM_PFNMAP pfn. In that case, - * batch->pages isn't valid (there's no struct page), so allow - * batch->pages to be touched only when there's more than one - * pfn to check, which guarantees the pfns are from a - * !VM_PFNMAP vma. + * pfn is preset for the first iteration of this inner loop + * due to the fact that vaddr_get_pfns() needs to provide the + * initial pfn for pfnmaps. Therefore to reduce redundancy, + * the next pfn is fetched at the end of the loop. + * A PageReserved() page could still qualify as page backed + * and rsvd here, and therefore continues to use the batch. */ while (true) { if (pfn != *pfn_base + pinned || @@ -688,21 +704,12 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, pfn = page_to_pfn(batch->pages[batch->offset]); } - - if (unlikely(disable_hugepages)) - break; } out: ret = vfio_lock_acct(dma, lock_acct, false); unpin_out: - if (batch->size == 1 && !batch->offset) { - /* May be a VM_PFNMAP pfn, which the batch can't remember. */ - put_pfn(pfn, dma->prot); - batch->size = 0; - } - if (ret < 0) { if (pinned && !rsvd) { for (pfn = *pfn_base ; pinned ; pfn++, pinned--) @@ -750,7 +757,7 @@ static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr, vfio_batch_init_single(&batch); - ret = vaddr_get_pfns(mm, vaddr, 1, dma->prot, pfn_base, batch.pages); + ret = vaddr_get_pfns(mm, vaddr, 1, dma->prot, pfn_base, &batch); if (ret != 1) goto out; From patchwork Tue Feb 18 22:22:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980978 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D3911DE3AC for ; Tue, 18 Feb 2025 22:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917358; cv=none; b=bz3kJbTcKGJtFfMAq+Yta5V1V5amSUeb1mvLk9LZKJju0GaTIL+nZGmo1RTlZZ3oHYtfDKuKO/2VR8YIu8h2FoTWE/G9JwWPIir0nE8tM4xepX7ome6jFkfWsKWo3Y0pAJixoSu3nEfp5r2ykC678rPKfypw2WZ34BXxdVVk2DI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917358; c=relaxed/simple; bh=smR5j7mr7NCBKqFW+ijihL1Yythw0728BzIgi2gtTWo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z7C7/R7+Sd51bcsUpzrKyqghbFdkHW0M2zr4g2pp/b+2ywNYgQDZM9T98PWlycB2LtUciPqS8NQR9T8PYl7As8lhV8Ah31yjCL8SUDgAcuigQXqkbmAilH/QoIkmkI+tj1wE46yqUZX/OIoeNcvx3FebIG/g+8B6IZ3F63cKqGY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=b89IZLI1; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="b89IZLI1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y50YBJiD0QUWKJiH9oMv1Fk/Dv9+zovo8IalJtkvn1w=; b=b89IZLI1Tus3OcF3xBzJ/nlbvnLgMJXrycneZafHH3Gby3PDV4smOEm1Q/Z7suTUMnFCxV GvMUffJG2Dwe4j8GntwBb6ylrRtNa2s1o4nVhDIGmR2FBhhZQt70EHOcaVorQDrrn8VclY xrbZm5jIFfQzvmPRsAMJx8Egx+Il4B4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-37-g9vmJ1x2MKCzjrR-ygYzqQ-1; Tue, 18 Feb 2025 17:22:31 -0500 X-MC-Unique: g9vmJ1x2MKCzjrR-ygYzqQ-1 X-Mimecast-MFC-AGG-ID: g9vmJ1x2MKCzjrR-ygYzqQ_1739917350 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7B2D0180056F; Tue, 18 Feb 2025 22:22:30 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 18A2930001A5; Tue, 18 Feb 2025 22:22:27 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com Subject: [PATCH v2 4/6] vfio/type1: Use consistent types for page counts Date: Tue, 18 Feb 2025 15:22:04 -0700 Message-ID: <20250218222209.1382449-5-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Page count should more consistently be an unsigned long when passed as an argument while functions returning a number of pages should use a signed long to allow for -errno. vaddr_get_pfns() can therefore be upgraded to return long, though in practice it's currently limited by the batch capacity. In fact, the batch indexes are noted to never hold negative values, so while it doesn't make sense to bloat the structure with unsigned longs in this case, it does make sense to specify these as unsigned. No change in behavior expected. Signed-off-by: Alex Williamson Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" --- drivers/vfio/vfio_iommu_type1.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index fafd8af125c7..ce661f03f139 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -103,9 +103,9 @@ struct vfio_dma { struct vfio_batch { struct page **pages; /* for pin_user_pages_remote */ struct page *fallback_page; /* if pages alloc fails */ - int capacity; /* length of pages array */ - int size; /* of batch currently */ - int offset; /* of next entry in pages */ + unsigned int capacity; /* length of pages array */ + unsigned int size; /* of batch currently */ + unsigned int offset; /* of next entry in pages */ }; struct vfio_iommu_group { @@ -560,14 +560,14 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, * initial offset. For VM_PFNMAP pfns, only the returned number of pfns and * returned initial pfn are provided; subsequent pfns are contiguous. */ -static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, - long npages, int prot, unsigned long *pfn, - struct vfio_batch *batch) +static long vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, + unsigned long npages, int prot, unsigned long *pfn, + struct vfio_batch *batch) { - long pin_pages = min_t(long, npages, batch->capacity); + unsigned long pin_pages = min_t(unsigned long, npages, batch->capacity); struct vm_area_struct *vma; unsigned int flags = 0; - int ret; + long ret; if (prot & IOMMU_WRITE) flags |= FOLL_WRITE; @@ -612,7 +612,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, * first page and all consecutive pages with the same locking. */ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, - long npage, unsigned long *pfn_base, + unsigned long npage, unsigned long *pfn_base, unsigned long limit, struct vfio_batch *batch) { unsigned long pfn; @@ -724,7 +724,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, } static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, - unsigned long pfn, long npage, + unsigned long pfn, unsigned long npage, bool do_accounting) { long unlocked = 0, locked = 0; From patchwork Tue Feb 18 22:22:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980977 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C15A11DE3A8 for ; Tue, 18 Feb 2025 22:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917358; cv=none; b=gbHOv2546J2oMZxPUuhXDC10g8rpSUdKEgoWYuCKtr7TwpnDIlTQVeRjlj2lD+sZXeFyprnIKFMHMpOtALtcQ/vhvKvgQGUnlOgqCFfmYU5HIumK/JZ5YMzPuJr19aPLPWZmRWSwm8BREtPJT+banLI1qzyOEJrYKlyMm1ZgS7Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917358; c=relaxed/simple; bh=gHCBDboqgHp/mDOI6vbjXgBSoA+DagFAsYJPdvdDxPs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UKmxQl37EEtOLxOgI+ylDUxowxEVH91n52Onr1xqiwNaSu/9YoqL0IQeLC3Pa3z6+9aNdZnfWwQ9XjnDFRITUiQehXFmHzGLz6YbJHPG8p0F93/tQCj9lW86HYk3+ZNZ0g4CW5Ea+td/G//a6ZHY6UuMhgJn2V3/YqvyOpP9/AA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NaAfwJnQ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NaAfwJnQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UlUtITEYQJv1hpgGQ8C/UoW2nF0g3ARQ77wXP5+0OfU=; b=NaAfwJnQOVdpmvy8ML667l34bvPZQ84/cqifavRrQBjg6ExFl17saAvvyOudAcWJnbMAQK ub6MWZ6eoNPa9LOrg3r0i5V1V49c0pCg9QB1Qs61CwTNObQF8HfKIb/VyRiaTk0pwJBudU x6fsG7wyK3BEkaf6qzx6FHIC8Rr2urM= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-637-bV8OMXniNsCt5H9n6wIQ_A-1; Tue, 18 Feb 2025 17:22:34 -0500 X-MC-Unique: bV8OMXniNsCt5H9n6wIQ_A-1 X-Mimecast-MFC-AGG-ID: bV8OMXniNsCt5H9n6wIQ_A_1739917353 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8C000180087B; Tue, 18 Feb 2025 22:22:33 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CD75E300019F; Tue, 18 Feb 2025 22:22:30 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com, akpm@linux-foundation.org, linux-mm@kvack.org, david@redhat.com Subject: [PATCH v2 5/6] mm: Provide address mask in struct follow_pfnmap_args Date: Tue, 18 Feb 2025 15:22:05 -0700 Message-ID: <20250218222209.1382449-6-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 follow_pfnmap_start() walks the page table for a given address and fills out the struct follow_pfnmap_args in pfnmap_args_setup(). The address mask of the page table level is already provided to this latter function for calculating the pfn. This address mask can also be useful for the caller to determine the extent of the contiguous mapping. For example, vfio-pci now supports huge_fault for pfnmaps and is able to insert pud and pmd mappings. When we DMA map these pfnmaps, ex. PCI MMIO BARs, we iterate follow_pfnmap_start() to get each pfn to test for a contiguous pfn range. Providing the mapping address mask allows us to skip the extent of the mapping level. Assuming a 1GB pud level and 4KB page size, iterations are reduced by a factor of 256K. In wall clock time, mapping a 32GB PCI BAR is reduced from ~1s to <1ms. Cc: Andrew Morton Cc: David Hildenbrand Cc: linux-mm@kvack.org Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" Reviewed-by: Jason Gunthorpe Signed-off-by: Alex Williamson Acked-by: David Hildenbrand --- include/linux/mm.h | 2 ++ mm/memory.c | 1 + 2 files changed, 3 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b1068ddcbb7..92b30dba7e38 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2417,11 +2417,13 @@ struct follow_pfnmap_args { * Outputs: * * @pfn: the PFN of the address + * @addr_mask: address mask covering pfn * @pgprot: the pgprot_t of the mapping * @writable: whether the mapping is writable * @special: whether the mapping is a special mapping (real PFN maps) */ unsigned long pfn; + unsigned long addr_mask; pgprot_t pgprot; bool writable; bool special; diff --git a/mm/memory.c b/mm/memory.c index 539c0f7c6d54..8f0969f132fe 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6477,6 +6477,7 @@ static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, args->lock = lock; args->ptep = ptep; args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->addr_mask = addr_mask; args->pgprot = pgprot; args->writable = writable; args->special = special; From patchwork Tue Feb 18 22:22:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13980979 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E39D31DE880 for ; Tue, 18 Feb 2025 22:22:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917364; cv=none; b=NYt87Y9x1nTY6p/O/Dls+dkmYAzD6RcBBbjfOzUtzgk4zHIQFVyEaGhrY+KvOuXQ7Ppj7CiIyd5G5rdxdTQ7kC6z9WWGp/873Mx9cMufDFQolMaYJImO2muepPA6MEUmUx967hDZk+M9JD9yhwi0t1X1te7rOFs67wDd8iH2h6g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739917364; c=relaxed/simple; bh=I0ZdTj8BmgqXnptu5DqBvvj81DWasr9opqmNnG45eSQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NGkZWX59ZOjeQdErNacAc9H8cCW+5phScCK4NLOBQUkIACM9Ug37Fi0d/2W3ePqXM+iDP+8Otf2lMSYznfTxomPkm5eoljlMbBqjQXqs4P9xM9KNKvdf8m83tyVNWWm6zi9QGCcgHvN5pVmNb191Q4OTrC2cGJ9a4o1a85pHECA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fpkB22U9; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fpkB22U9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739917360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jnx3EEunpJZjox9YHzpXUDwmXWPi4uIum6tx1nb7cx0=; b=fpkB22U9LHdm2TreR3Bo/vOrEax0x1Y9Bm66le9G0X3ZEI8uLOoBFsquax6wB/Xa9ISVol agDEHVjnDLAneyWvcwoj33k5gwFale7OTm03RJYpiUzR4VHvqSykBP6Xs7SE+bEaCDFRry d7qgFmA90XYi0LeewOuwiUy48bM7t4M= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-681-y8fXHio6OM-KxVoJCnyMDA-1; Tue, 18 Feb 2025 17:22:37 -0500 X-MC-Unique: y8fXHio6OM-KxVoJCnyMDA-1 X-Mimecast-MFC-AGG-ID: y8fXHio6OM-KxVoJCnyMDA_1739917356 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 72AA21800877; Tue, 18 Feb 2025 22:22:36 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.88.77]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 280D4300019F; Tue, 18 Feb 2025 22:22:33 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, jgg@nvidia.com, willy@infradead.org Subject: [PATCH v2 6/6] vfio/type1: Use mapping page mask for pfnmaps Date: Tue, 18 Feb 2025 15:22:06 -0700 Message-ID: <20250218222209.1382449-7-alex.williamson@redhat.com> In-Reply-To: <20250218222209.1382449-1-alex.williamson@redhat.com> References: <20250218222209.1382449-1-alex.williamson@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and pmd mappings for well aligned mappings. follow_pfnmap_start() walks the page table and therefore knows the page mask of the level where the address is found and returns this through follow_pfnmap_args.pgmask. Subsequent pfns from this address until the end of the mapping page are necessarily consecutive. Use this information to retrieve a range of pfnmap pfns in a single pass. With optimal mappings and alignment on systems with 1GB pud and 4KB page size, this reduces iterations for DMA mapping PCI BARs by a factor of 256K. In real world testing, the overhead of iterating pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to sub-millisecond overhead. Signed-off-by: Alex Williamson Reviewed-by: Peter Xu Reviewed-by: "Mitchell Augustin" Tested-by: "Mitchell Augustin" --- drivers/vfio/vfio_iommu_type1.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index ce661f03f139..0ac56072af9f 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -520,7 +520,7 @@ static void vfio_batch_fini(struct vfio_batch *batch) static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, - bool write_fault) + unsigned long *addr_mask, bool write_fault) { struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; @@ -544,10 +544,12 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, return ret; } - if (write_fault && !args.writable) + if (write_fault && !args.writable) { ret = -EFAULT; - else + } else { *pfn = args.pfn; + *addr_mask = args.addr_mask; + } follow_pfnmap_end(&args); return ret; @@ -590,15 +592,22 @@ static long vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, vma = vma_lookup(mm, vaddr); if (vma && vma->vm_flags & VM_PFNMAP) { - ret = follow_fault_pfn(vma, mm, vaddr, pfn, prot & IOMMU_WRITE); + unsigned long addr_mask; + + ret = follow_fault_pfn(vma, mm, vaddr, pfn, &addr_mask, + prot & IOMMU_WRITE); if (ret == -EAGAIN) goto retry; if (!ret) { - if (is_invalid_reserved_pfn(*pfn)) - ret = 1; - else + if (is_invalid_reserved_pfn(*pfn)) { + unsigned long epfn; + + epfn = (*pfn | (~addr_mask >> PAGE_SHIFT)) + 1; + ret = min_t(long, npages, epfn - *pfn); + } else { ret = -EFAULT; + } } } done: