From patchwork Wed Aug 28 02:48:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EEB13C54751 for ; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E71E710E444; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="LPqhhLTW"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8095D10E43F; Wed, 28 Aug 2024 02:48:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=X4NcHhht7R85hRcl43GBO0i1KlziGSyVufpAc1OriAU=; b=LPqhhLTWAp6JMJOOM7E5FE8UEkh3UYjP83Y98zdbGALzHcH6Ds5uMNOo gfMr4IsnTm94W+e19Sq5AOJUM/qgvacXKFNow/QQxvxeGaqjPxhzzGUAR otoLYjzxCD6LayYiOTrG9o9FJIE0DL+j1D71kA2mmWljhKqSDU/EfoDf7 gET+xpR5W5NbNIWcBLflwDKaEMlB9SQ2kefHBGXn8rpoSCgaQml6891p8 kycP849wwyPBTVy1iZYOzGLLdCuPR0zesngPFh1GUMqX2mqjZwT/juhDx QE3oXG/8TZiy8hjpNOuZE+utTIb3Cadk/PbITNfMBoq8J85vN13Hs+nSa g==; X-CSE-ConnectionGUID: R4ptwx9jTGOyYp+blpkqiA== X-CSE-MsgGUID: VxWdaxNWTD+v7auPvufuZw== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251846" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251846" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 X-CSE-ConnectionGUID: OjqwvTI+QxGQ623af+GINg== X-CSE-MsgGUID: QG7qk4vRQjmjBmsL1Lj36Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224584" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:06 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 01/28] dma-buf: Split out dma fence array create into alloc and arm functions Date: Tue, 27 Aug 2024 19:48:34 -0700 Message-Id: <20240828024901.2582335-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Useful to preallocate dma fence array and then arm in path of reclaim or a dma fence. v2: - s/arm/init (Christian) - Drop !array warn (Christian) Cc: Sumit Semwal Cc: Christian König Signed-off-by: Matthew Brost --- drivers/dma-buf/dma-fence-array.c | 78 ++++++++++++++++++++++--------- include/linux/dma-fence-array.h | 6 +++ 2 files changed, 63 insertions(+), 21 deletions(-) diff --git a/drivers/dma-buf/dma-fence-array.c b/drivers/dma-buf/dma-fence-array.c index c74ac197d5fe..0659e6b29b3c 100644 --- a/drivers/dma-buf/dma-fence-array.c +++ b/drivers/dma-buf/dma-fence-array.c @@ -144,37 +144,38 @@ const struct dma_fence_ops dma_fence_array_ops = { EXPORT_SYMBOL(dma_fence_array_ops); /** - * dma_fence_array_create - Create a custom fence array + * dma_fence_array_alloc - Allocate a custom fence array + * @num_fences: [in] number of fences to add in the array + * + * Return dma fence array on success, NULL on failure + */ +struct dma_fence_array *dma_fence_array_alloc(int num_fences) +{ + struct dma_fence_array *array; + + return kzalloc(struct_size(array, callbacks, num_fences), GFP_KERNEL); +} +EXPORT_SYMBOL(dma_fence_array_alloc); + +/** + * dma_fence_array_init - Arm a custom fence array + * @array: [in] dma fence array to arm * @num_fences: [in] number of fences to add in the array * @fences: [in] array containing the fences * @context: [in] fence context to use * @seqno: [in] sequence number to use * @signal_on_any: [in] signal on any fence in the array * - * Allocate a dma_fence_array object and initialize the base fence with - * dma_fence_init(). - * In case of error it returns NULL. - * - * The caller should allocate the fences array with num_fences size - * and fill it with the fences it wants to add to the object. Ownership of this - * array is taken and dma_fence_put() is used on each fence on release. - * - * If @signal_on_any is true the fence array signals if any fence in the array - * signals, otherwise it signals when all fences in the array signal. + * Implementation of @dma_fence_array_create without allocation. Useful to arm a + * preallocated dma fence fence in the path of reclaim or dma fence signaling. */ -struct dma_fence_array *dma_fence_array_create(int num_fences, - struct dma_fence **fences, - u64 context, unsigned seqno, - bool signal_on_any) +void dma_fence_array_init(struct dma_fence_array *array, + int num_fences, struct dma_fence **fences, + u64 context, unsigned seqno, + bool signal_on_any) { - struct dma_fence_array *array; - WARN_ON(!num_fences || !fences); - array = kzalloc(struct_size(array, callbacks, num_fences), GFP_KERNEL); - if (!array) - return NULL; - array->num_fences = num_fences; spin_lock_init(&array->lock); @@ -200,6 +201,41 @@ struct dma_fence_array *dma_fence_array_create(int num_fences, */ while (num_fences--) WARN_ON(dma_fence_is_container(fences[num_fences])); +} +EXPORT_SYMBOL(dma_fence_array_init); + +/** + * dma_fence_array_create - Create a custom fence array + * @num_fences: [in] number of fences to add in the array + * @fences: [in] array containing the fences + * @context: [in] fence context to use + * @seqno: [in] sequence number to use + * @signal_on_any: [in] signal on any fence in the array + * + * Allocate a dma_fence_array object and initialize the base fence with + * dma_fence_init(). + * In case of error it returns NULL. + * + * The caller should allocate the fences array with num_fences size + * and fill it with the fences it wants to add to the object. Ownership of this + * array is taken and dma_fence_put() is used on each fence on release. + * + * If @signal_on_any is true the fence array signals if any fence in the array + * signals, otherwise it signals when all fences in the array signal. + */ +struct dma_fence_array *dma_fence_array_create(int num_fences, + struct dma_fence **fences, + u64 context, unsigned seqno, + bool signal_on_any) +{ + struct dma_fence_array *array; + + array = dma_fence_array_alloc(num_fences); + if (!array) + return NULL; + + dma_fence_array_init(array, num_fences, fences, + context, seqno, signal_on_any); return array; } diff --git a/include/linux/dma-fence-array.h b/include/linux/dma-fence-array.h index 29c5650c1038..079b3dec0a16 100644 --- a/include/linux/dma-fence-array.h +++ b/include/linux/dma-fence-array.h @@ -79,6 +79,12 @@ to_dma_fence_array(struct dma_fence *fence) for (index = 0, fence = dma_fence_array_first(head); fence; \ ++(index), fence = dma_fence_array_next(head, index)) +struct dma_fence_array *dma_fence_array_alloc(int num_fences); +void dma_fence_array_init(struct dma_fence_array *array, + int num_fences, struct dma_fence **fences, + u64 context, unsigned seqno, + bool signal_on_any); + struct dma_fence_array *dma_fence_array_create(int num_fences, struct dma_fence **fences, u64 context, unsigned seqno, From patchwork Wed Aug 28 02:48:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AFC60C5474B for ; Wed, 28 Aug 2024 02:48:15 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6D94310E449; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="HINPOc6Z"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id D733910E440; Wed, 28 Aug 2024 02:48:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=51K5wzXb0gghUiwWQs7D+UNeNaRsXn/Mb6dnZ9Xz5fw=; b=HINPOc6Z4fzcZj+tkn93eTFeu0fZbu+qZts+ojMtdTPcYLUs00dmoPmH wWAdAZSZKyrZBQ2yJqG9vHiqKtBvuxiYhTuzQfzxFqrCesf/G6GAZMRNX 9NUQoySjdgW+8KoSCo16zxKN1eMqQx5yE22FpMP1k6jBMrcpMyE5O3OqU FQC8fY1MWEX74H4OLC812q5RNQIWP2Mird3cFfy0xcEcTVbRUuJEwl2tO cYPRxMx7v9DUUPTmcJHKGhgKJkCINihexTPBirFfKPkiylDrRY69cpwkc cY9muZ4zFDbm9e7aBaLX5N6tB07XYxNkdtICUWAzBCgFnAQDCgIaW+f4F g==; X-CSE-ConnectionGUID: kq2TV9UjTBqRfaa5PujBUw== X-CSE-MsgGUID: GPvLeM+PQ0u4PQ60n3sC4A== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251852" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251852" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 X-CSE-ConnectionGUID: 7ViM99rpRmOdiTXn9D5PSg== X-CSE-MsgGUID: gf3f1i0iTtaoo/ET0ZuePA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224587" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:06 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 02/28] drm/xe: Invalidate media_gt TLBs in PT code Date: Tue, 27 Aug 2024 19:48:35 -0700 Message-Id: <20240828024901.2582335-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Testing on LNL has shown media GT's TLBs need to be invalidated via the GuC, update PT code appropriately. v2: - Do dma_fence_get before first call of invalidation_fence_init (Himal) - No need to check for valid chain fence (Himal) v3: - Use dma-fence-array Fixes: 3330361543fc ("drm/xe/lnl: Add LNL platform definition") Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 117 ++++++++++++++++++++++++++++++------- 1 file changed, 96 insertions(+), 21 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index 579ed31b46db..d6353e8969f0 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -3,6 +3,8 @@ * Copyright © 2022 Intel Corporation */ +#include + #include "xe_pt.h" #include "regs/xe_gtt_defs.h" @@ -1627,9 +1629,11 @@ xe_pt_update_ops_rfence_interval(struct xe_vm_pgtable_update_ops *pt_update_ops, static int vma_reserve_fences(struct xe_device *xe, struct xe_vma *vma) { + int shift = xe_device_get_root_tile(xe)->media_gt ? 1 : 0; + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) return dma_resv_reserve_fences(xe_vma_bo(vma)->ttm.base.resv, - xe->info.tile_count); + xe->info.tile_count << shift); return 0; } @@ -1816,6 +1820,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops) struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[tile->id]; struct xe_vma_op *op; + int shift = tile->media_gt ? 1 : 0; int err; lockdep_assert_held(&vops->vm->lock); @@ -1824,7 +1829,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops) xe_pt_update_ops_init(pt_update_ops); err = dma_resv_reserve_fences(xe_vm_resv(vops->vm), - tile_to_xe(tile)->info.tile_count); + tile_to_xe(tile)->info.tile_count << shift); if (err) return err; @@ -1849,13 +1854,20 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops) static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile, struct xe_vm_pgtable_update_ops *pt_update_ops, - struct xe_vma *vma, struct dma_fence *fence) + struct xe_vma *vma, struct dma_fence *fence, + struct dma_fence *fence2) { - if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) { dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ? DMA_RESV_USAGE_KERNEL : DMA_RESV_USAGE_BOOKKEEP); + if (fence2) + dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence2, + pt_update_ops->wait_vm_bookkeep ? + DMA_RESV_USAGE_KERNEL : + DMA_RESV_USAGE_BOOKKEEP); + } vma->tile_present |= BIT(tile->id); vma->tile_staged &= ~BIT(tile->id); if (xe_vma_is_userptr(vma)) { @@ -1875,13 +1887,20 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile, static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile, struct xe_vm_pgtable_update_ops *pt_update_ops, - struct xe_vma *vma, struct dma_fence *fence) + struct xe_vma *vma, struct dma_fence *fence, + struct dma_fence *fence2) { - if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) { dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ? DMA_RESV_USAGE_KERNEL : DMA_RESV_USAGE_BOOKKEEP); + if (fence2) + dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence2, + pt_update_ops->wait_vm_bookkeep ? + DMA_RESV_USAGE_KERNEL : + DMA_RESV_USAGE_BOOKKEEP); + } vma->tile_present &= ~BIT(tile->id); if (!vma->tile_present) { list_del_init(&vma->combined_links.rebind); @@ -1898,7 +1917,8 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile, static void op_commit(struct xe_vm *vm, struct xe_tile *tile, struct xe_vm_pgtable_update_ops *pt_update_ops, - struct xe_vma_op *op, struct dma_fence *fence) + struct xe_vma_op *op, struct dma_fence *fence, + struct dma_fence *fence2) { xe_vm_assert_held(vm); @@ -1907,26 +1927,28 @@ static void op_commit(struct xe_vm *vm, if (!op->map.immediate && xe_vm_in_fault_mode(vm)) break; - bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence); + bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence, + fence2); break; case DRM_GPUVA_OP_REMAP: unbind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.remap.unmap->va), fence); + gpuva_to_vma(op->base.remap.unmap->va), fence, + fence2); if (op->remap.prev) bind_op_commit(vm, tile, pt_update_ops, op->remap.prev, - fence); + fence, fence2); if (op->remap.next) bind_op_commit(vm, tile, pt_update_ops, op->remap.next, - fence); + fence, fence2); break; case DRM_GPUVA_OP_UNMAP: unbind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.unmap.va), fence); + gpuva_to_vma(op->base.unmap.va), fence, fence2); break; case DRM_GPUVA_OP_PREFETCH: bind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.prefetch.va), fence); + gpuva_to_vma(op->base.prefetch.va), fence, fence2); break; default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); @@ -1963,7 +1985,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[tile->id]; struct dma_fence *fence; - struct invalidation_fence *ifence = NULL; + struct invalidation_fence *ifence = NULL, *mfence = NULL; + struct dma_fence **fences = NULL; + struct dma_fence_array *cf = NULL; struct xe_range_fence *rfence; struct xe_vma_op *op; int err = 0, i; @@ -1996,6 +2020,23 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) err = -ENOMEM; goto kill_vm_tile1; } + if (tile->media_gt) { + mfence = kzalloc(sizeof(*ifence), GFP_KERNEL); + if (!mfence) { + err = -ENOMEM; + goto free_ifence; + } + fences = kmalloc_array(2, sizeof(*fences), GFP_KERNEL); + if (!fences) { + err = -ENOMEM; + goto free_ifence; + } + cf = dma_fence_array_alloc(2); + if (!cf) { + err = -ENOMEM; + goto free_ifence; + } + } } rfence = kzalloc(sizeof(*rfence), GFP_KERNEL); @@ -2027,19 +2068,50 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) /* tlb invalidation must be done before signaling rebind */ if (ifence) { + if (mfence) + dma_fence_get(fence); invalidation_fence_init(tile->primary_gt, ifence, fence, pt_update_ops->start, pt_update_ops->last, vm->usm.asid); - fence = &ifence->base.base; + if (mfence) { + invalidation_fence_init(tile->media_gt, mfence, fence, + pt_update_ops->start, + pt_update_ops->last, vm->usm.asid); + fences[0] = &ifence->base.base; + fences[1] = &mfence->base.base; + dma_fence_array_init(cf, 2, fences, + vm->composite_fence_ctx, + vm->composite_fence_seqno++, + false); + fence = &cf->base; + } else { + fence = &ifence->base.base; + } } - dma_resv_add_fence(xe_vm_resv(vm), fence, - pt_update_ops->wait_vm_bookkeep ? - DMA_RESV_USAGE_KERNEL : - DMA_RESV_USAGE_BOOKKEEP); + if (!mfence) { + dma_resv_add_fence(xe_vm_resv(vm), fence, + pt_update_ops->wait_vm_bookkeep ? + DMA_RESV_USAGE_KERNEL : + DMA_RESV_USAGE_BOOKKEEP); - list_for_each_entry(op, &vops->list, link) - op_commit(vops->vm, tile, pt_update_ops, op, fence); + list_for_each_entry(op, &vops->list, link) + op_commit(vops->vm, tile, pt_update_ops, op, fence, NULL); + } else { + dma_resv_add_fence(xe_vm_resv(vm), &ifence->base.base, + pt_update_ops->wait_vm_bookkeep ? + DMA_RESV_USAGE_KERNEL : + DMA_RESV_USAGE_BOOKKEEP); + + dma_resv_add_fence(xe_vm_resv(vm), &mfence->base.base, + pt_update_ops->wait_vm_bookkeep ? + DMA_RESV_USAGE_KERNEL : + DMA_RESV_USAGE_BOOKKEEP); + + list_for_each_entry(op, &vops->list, link) + op_commit(vops->vm, tile, pt_update_ops, op, + &ifence->base.base, &mfence->base.base); + } if (pt_update_ops->needs_userptr_lock) up_read(&vm->userptr.notifier_lock); @@ -2049,6 +2121,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) free_rfence: kfree(rfence); free_ifence: + kfree(cf); + kfree(fences); + kfree(mfence); kfree(ifence); kill_vm_tile1: if (err != -EAGAIN && tile->id) From patchwork Wed Aug 28 02:48:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C3A1C5474D for ; Wed, 28 Aug 2024 02:48:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B826010E44D; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="eDNSH5Zu"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1BF2610E43F; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jMM9UwLR2Y/NKoUEZ4P8mdamrRObjrVRCDawmMDsk1w=; b=eDNSH5ZuIh/M5oLOqP4Vw0JMiYCrFnsqcLsOz1c4UXOeqZF4oxVQy0fC u+zX+pcyq362eZhZRkOWiElYrE//MAxHFExq97t//CQt7pMotTd3lDUu6 47D+uD8f1Qaf+iwQFb52KAxGxuUcW47l+LpRfh+d05lYEkFhFHqGG0m8P ACMKyxE7dgaEyCEyjStcixQB9qa2ds1r7Lz64znd9TPFlx5JoP/utJJio RdrAnG/qwkQmccqRQGlTQNSVBWWF2f2Bmz/SDz27TTykRg9N8iW+JTFky FxNlToueS5qHzPcLp5IQ73VBB7mOSZIjcIXo84SCU25Vbvn/AoObSQeRI A==; X-CSE-ConnectionGUID: hUzx0c+nTw6LtsQuvqKLYw== X-CSE-MsgGUID: OrEep0aCS8aF9c5ik4e1GA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251856" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251856" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 X-CSE-ConnectionGUID: 7idKd+tkScOc9tJBQM/o5g== X-CSE-MsgGUID: qunkH3tYTfu2dCHSpL1gqw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224592" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 03/28] drm/xe: Retry BO allocation Date: Tue, 27 Aug 2024 19:48:36 -0700 Message-Id: <20240828024901.2582335-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_bo.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index cbe7bf098970..b6c6a4a3b4d4 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -1977,6 +1977,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, struct xe_file *xef = to_xe_file(file); struct drm_xe_gem_create *args = data; struct xe_vm *vm = NULL; + ktime_t end = 0; struct xe_bo *bo; unsigned int bo_flags; u32 handle; @@ -2042,11 +2043,14 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, vm = xe_vm_lookup(xef, args->vm_id); if (XE_IOCTL_DBG(xe, !vm)) return -ENOENT; + } + +retry: + if (vm) { err = xe_vm_lock(vm, true); if (err) goto out_vm; } - bo = xe_bo_create_user(xe, NULL, vm, args->size, args->cpu_caching, bo_flags); @@ -2055,6 +2059,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, if (IS_ERR(bo)) { err = PTR_ERR(bo); + if (xe_vm_validate_should_retry(NULL, err, &end)) + goto retry; goto out_vm; } From patchwork Wed Aug 28 02:48:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ECD43C54749 for ; Wed, 28 Aug 2024 02:48:13 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1690610E446; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="nbfrk9R1"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 32A5010E440; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Fp+nlxf77E8ObSScKrGPzO5OLjkgp3O8UmyRwhsVGlI=; b=nbfrk9R1vaMYXSqelVocMUD7wTZo6buyGA7YHC7VCr6nYfpcjVv0Tuwv QIuWkZSDWlEjOt1XsujLNMRWCBxwUGxclvr8gSer/nJLARa3+aiPcGAAv Dff0HlSS4KoR7QdHk0KfCnLQAg9x/cGfvs8fgUTgbsAMnK+fIENCeyllk n12RWVkfSvuQflICWzEV2cKicK9gOW3gauM8HFn6XSi68JqitJnuT7QEf 1kymW/mkMF2QxOS0cTJHnp/o7Vg8bqua8H57uTtUujn4FFsFdi1HKpFdj FP+xGVms4kH1i6xfXrly5hLVK7iqZDEO67uDtVG0juaSNhx4oPGCGHB/G g==; X-CSE-ConnectionGUID: 0F99W7N4RxWkYzUqXGjZow== X-CSE-MsgGUID: Q5jNhwF+RJevdTuOyTWFig== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251861" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251861" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 X-CSE-ConnectionGUID: +EMLZZrWTKGENg8O4+lX3w== X-CSE-MsgGUID: E/sDT5oqTsan1eAWte+KLg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224595" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 04/28] mm/migrate: Add migrate_device_vma_range Date: Tue, 27 Aug 2024 19:48:37 -0700 Message-Id: <20240828024901.2582335-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add migrate_device_vma_range which prepares an array of pre-populated device pages for migration and issues a MMU invalidation. Cc: Andrew Morton Signed-off-by: Matthew Brost --- include/linux/migrate.h | 3 +++ mm/migrate_device.c | 53 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 644be30b69c8..e8cce05bf9c2 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -226,6 +226,9 @@ void migrate_vma_pages(struct migrate_vma *migrate); void migrate_vma_finalize(struct migrate_vma *migrate); int migrate_device_range(unsigned long *src_pfns, unsigned long start, unsigned long npages); +int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner, + unsigned long *src_pfns, unsigned long npages, + unsigned long start); void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns, unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns, diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6d66dc1c6ffa..e25f12a132e8 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -920,6 +920,59 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start, } EXPORT_SYMBOL(migrate_device_range); +/** + * migrate_device_vma_range() - migrate device private pfns to normal memory and + * trigger MMU invalidation. + * @mm: struct mm of device pages. + * @src_pfns: pre-popluated array of source device private pfns to migrate. + * @pgmap_owner: page group map owner of device pages. + * @npages: number of pages to migrate. + * @start: VMA start of device pages. + * + * Similar to migrate_device_range() but supports non-contiguous pre-popluated + * array of device pages to migrate. Also triggers MMU invalidation. Useful in + * device memory eviction paths where lock is held protecting the device pages + * but where the mmap lock cannot be taken to due to a locking inversion (e.g. + * DRM drivers). Since the mmap lock is not required to be held, the MMU + * invalidation can race with with VMA start being repurposed, worst case this + * would result in an unecessary invalidation. + */ +int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner, + unsigned long *src_pfns, unsigned long npages, + unsigned long start) +{ + struct mmu_notifier_range range; + unsigned long i; + + mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0, + mm, start, start + npages * PAGE_SIZE, + pgmap_owner); + mmu_notifier_invalidate_range_start(&range); + + for (i = 0; i < npages; i++) { + struct page *page = pfn_to_page(src_pfns[i]); + + if (!get_page_unless_zero(page)) { + src_pfns[i] = 0; + continue; + } + + if (!trylock_page(page)) { + src_pfns[i] = 0; + put_page(page); + continue; + } + + src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE; + } + + migrate_device_unmap(src_pfns, npages, NULL); + mmu_notifier_invalidate_range_end(&range); + + return 0; +} +EXPORT_SYMBOL(migrate_device_vma_range); + /* * Migrate a device coherent page back to normal memory. The caller should have * a reference on page which will be copied to the new page if migration is From patchwork Wed Aug 28 02:48:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F09C0C54751 for ; Wed, 28 Aug 2024 02:48:24 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2472010E45D; Wed, 28 Aug 2024 02:48:13 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="In3jek79"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 53C3310E441; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zjmiHYc8irRKSQ11PgT9qhXcNpHi/CB4knZ1R50LGvI=; b=In3jek79Se6+TBo1JpZSJCcLrNMEIixH9Seay8iClR4cUE5RWT7qdm8v KxVafsp46hMpmSzJBDxl/rwYFn3IdRo1F6TxkPDdOVIX9t/k08YP0QJiR hC8PDMjcRTnwuZGClwBQse3DHiDqh3OoP76ptAf5Rlj/GzrbQclLJOGdE tjuoO8yg6mIQlWco4iR7W6ky5I6XP4zKLKI/IVFZ/EQ63TOECY5/Ty3w+ sm/vL43RiXMsykvlk+tCP/ZqhfB+YieO7Hx73b8rCz8E5olBQsFakre0T MX6VK+eTR04UyO8c+6o4X1P1wTizQ8Ymkbi7A0nKXKQ27XA6nkmpGgcUS A==; X-CSE-ConnectionGUID: d+LwWJVTSR+JooEwKYP9Aw== X-CSE-MsgGUID: rb1Ot7IuTZmkDDo0KtDK+A== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251867" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251867" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 X-CSE-ConnectionGUID: JK8QIxwZTnmBPp2OgUekFg== X-CSE-MsgGUID: fg1/PFrbT8W5FWygsFWt8A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224598" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory Date: Tue, 27 Aug 2024 19:48:38 -0700 Message-Id: <20240828024901.2582335-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks. The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM. This mid-layer is largely inspired by GPUVM. Cc: Dave Airlie Cc: Thomas Hellström Cc: Christian König Cc: Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/Makefile | 3 +- drivers/gpu/drm/xe/drm_gpusvm.c | 2174 +++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/drm_gpusvm.h | 415 ++++++ 3 files changed, 2591 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile index b9670ae09a9e..b8fc2ee58f1a 100644 --- a/drivers/gpu/drm/xe/Makefile +++ b/drivers/gpu/drm/xe/Makefile @@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \ # core driver code -xe-y += xe_bb.o \ +xe-y += drm_gpusvm.o \ + xe_bb.o \ xe_bo.o \ xe_bo_evict.o \ xe_devcoredump.o \ diff --git a/drivers/gpu/drm/xe/drm_gpusvm.c b/drivers/gpu/drm/xe/drm_gpusvm.c new file mode 100644 index 000000000000..fc1e44e6ae72 --- /dev/null +++ b/drivers/gpu/drm/xe/drm_gpusvm.c @@ -0,0 +1,2174 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2024 Intel Corporation + * + * Authors: + * Matthew Brost + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "drm_gpusvm.h" + +/** + * DOC: Overview + * + * GPU Shared Virtual Memory (GPU SVM) layer for the Direct Rendering Manager (DRM) + * + * The GPU SVM layer is a component of the DRM framework designed to manage shared + * virtual memory between the CPU and GPU. It enables efficient data exchange and + * processing for GPU-accelerated applications by allowing memory sharing and + * synchronization between the CPU's and GPU's virtual address spaces. + * + * Key GPU SVM Components: + * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the + * GPU of changes, notifiers are sized based on a GPU SVM + * initialization parameter, with a recommendation of 512M or + * larger. They maintain a Red-BlacK tree and a list of ranges that + * fall within the notifier interval. Notifiers are tracked within + * a GPU SVM Red-BlacK tree and list and are dynamically inserted + * or removed as ranges within the interval are created or + * destroyed. + * - Ranges: Represent memory ranges mapped in a DRM device and managed + * by GPU SVM. They are sized based on an array of chunk sizes, which + * is a GPU SVM initialization parameter, and the CPU address space. + * Upon GPU fault, the largest aligned chunk that fits within the + * faulting CPU address space is chosen for the range size. Ranges are + * expected to be dynamically allocated on GPU fault and removed on an + * MMU notifier UNMAP event. As mentioned above, ranges are tracked in + * a notifier's Red-Black tree. + * - Operations: Define the interface for driver-specific SVM operations such as + * allocation, page collection, migration, invalidations, and VRAM + * release. + * + * This layer provides interfaces for allocating, mapping, migrating, and + * releasing memory ranges between the CPU and GPU. It handles all core memory + * management interactions (DMA mapping, HMM, and migration) and provides + * driver-specific virtual functions (vfuncs). This infrastructure is sufficient + * to build the expected driver components for an SVM implementation as detailed + * below. + * + * Expected Driver Components: + * - GPU page fault handler: Used to create ranges and notifiers based on the + * fault address, optionally migrate the range to + * VRAM, and create GPU bindings. + * - Garbage collector: Used to destroy GPU bindings for ranges. Ranges are + * expected to be added to the garbage collector upon + * MMU_NOTIFY_UNMAP event. + */ + +/** + * DOC: Locking + * + * GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the + * mmap lock as needed. Alternatively, if the driver prefers to handle the mmap + * lock itself, a 'locked' argument is provided to the functions that require + * the mmap lock. This option may be useful for drivers that need to call into + * GPU SVM while also holding a dma-resv lock, thus preventing locking + * inversions between the mmap and dma-resv locks. + * + * GPU SVM introduces a global notifier lock, which safeguards the notifier's + * range RB tree and list, as well as the range's DMA mappings and sequence + * number. GPU SVM manages all necessary locking and unlocking operations, + * except for the recheck of the range's sequence number + * (mmu_interval_read_retry) when the driver is committing GPU bindings. This + * lock corresponds to the 'driver->update' lock mentioned in the HMM + * documentation (TODO: Link). Future revisions may transition from a GPU SVM + * global lock to a per-notifier lock if finer-grained locking is deemed + * necessary. + * + * In addition to the locking mentioned above, the driver should implement a + * lock to safeguard core GPU SVM function calls that modify state, such as + * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. Alternatively, + * these core functions can be called within a single kernel thread, for + * instance, using an ordered work queue. This lock is denoted as + * 'driver_svm_lock' in code examples. + */ + +/** + * DOC: Migrataion + * + * The migration support is quite simple, allowing migration between SRAM and + * VRAM at the range granularity. For example, GPU SVM currently does not + * support mixing SRAM and VRAM pages within a range. This means that upon GPU + * fault, the entire range can be migrated to VRAM, and upon CPU fault, the + * entire range is migrated to SRAM. + * + * The reasoning for only supporting range granularity is as follows: it + * simplifies the implementation, and range sizes are driver-defined and should + * be relatively small. + */ + +/** + * DOC: Partial Unmapping of Ranges + * + * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by CPU resulting + * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one + * being that a subset of the range still has CPU and GPU mappings. If the + * backing store for the range is in VRAM, a subset of the backing store has + * references. One option would be to split the range and VRAM backing store, + * but the implementation for this would be quite complicated. Given that + * partial unmappings are rare and driver-defined range sizes are relatively + * small, GPU SVM does not support splitting of ranges. + * + * With no support for range splitting, upon partial unmapping of a range, the + * driver is expected to invalidate and destroy the entire range. If the range + * has VRAM as its backing, the driver is also expected to migrate any remaining + * pages back to SRAM. + */ + +/** + * DOC: Examples + * + * This section provides two examples of how to build the expected driver + * components: the GPU page fault handler and the garbage collector. A third + * example demonstrates a sample invalidation driver vfunc. + * + * The generic code provided does not include logic for complex migration + * policies, optimized invalidations, or other potentially required driver + * locking (e.g., DMA-resv locks). + * + * 1) GPU page fault handler + * + * int driver_bind_range(struct drm_gpusvm *gpusvm, struct drm_gpusvm_range *range) + * { + * int err = 0; + * + * driver_alloc_and_setup_memory_for_bind(gpusvm, range); + * + * drm_gpusvm_notifier_lock(gpusvm); + * if (drm_gpusvm_range_pages_valid(range)) + * driver_commit_bind(gpusvm, range); + * else + * err = -EAGAIN; + * drm_gpusvm_notifier_unlock(gpusvm); + * + * return err; + * } + * + * int driver_gpu_fault(struct drm_gpusvm *gpusvm, u64 fault_addr, + * u64 gpuva_start, u64 gpuva_end) + * { + * struct drm_gpusvm_ctx ctx = {}; + * int err; + * + * driver_svm_lock(); + * retry: + * // Always process UNMAPs first so view of GPU SVM ranges is current + * driver_garbage_collector(gpusvm); + * + * range = drm_gpusvm_range_find_or_insert(gpusvm, fault_addr, + * gpuva_start, gpuva_end, + * &ctx); + * if (IS_ERR(range)) { + * err = PTR_ERR(range); + * goto unlock; + * } + * + * if (driver_migration_policy(range)) { + * bo = driver_alloc_bo(); + * err = drm_gpusvm_migrate_to_vram(gpusvm, range, bo, &ctx); + * if (err) // CPU mappings may have changed + * goto retry; + * } + * + * err = drm_gpusvm_range_get_pages(gpusvm, range, &ctx); + * if (err == -EFAULT || err == -EPERM) // CPU mappings changed + * goto retry; + * else if (err) + * goto unlock; + * + * err = driver_bind_range(gpusvm, range); + * if (err == -EAGAIN) // CPU mappings changed + * goto retry + * + * unlock: + * driver_svm_unlock(); + * return err; + * } + * + * 2) Garbage Collector. + * + * void __driver_garbage_collector(struct drm_gpusvm *gpusvm, + * struct drm_gpusvm_range *range) + * { + * struct drm_gpusvm_ctx ctx = {}; + * + * assert_driver_svm_locked(gpusvm); + * + * // Partial unmap, migrate any remaining VRAM pages back to SRAM + * if (range->flags.partial_unmap) + * drm_gpusvm_migrate_to_sram(gpusvm, range, &ctx); + * + * driver_unbind_range(range); + * drm_gpusvm_range_remove(gpusvm, range); + * } + * + * void driver_garbage_collector(struct drm_gpusvm *gpusvm) + * { + * assert_driver_svm_locked(gpusvm); + * + * for_each_range_in_garbage_collector(gpusvm, range) + * __driver_garbage_collector(gpusvm, range); + * } + * + * 3) Invalidation driver vfunc. + * + * void driver_invalidation(struct drm_gpusvm *gpusvm, + * struct drm_gpusvm_notifier *notifier, + * const struct mmu_notifier_range *mmu_range) + * { + * struct drm_gpusvm_ctx ctx = { .in_notifier = true, }; + * struct drm_gpusvm_range *range = NULL; + * + * driver_invalidate_device_tlb(gpusvm, mmu_range->start, mmu_range->end); + * + * drm_gpusvm_for_each_range(range, notifier, mmu_range->start, + * mmu_range->end) { + * drm_gpusvm_range_unmap_pages(gpusvm, range, &ctx); + * + * if (mmu_range->event != MMU_NOTIFY_UNMAP) + * continue; + * + * drm_gpusvm_range_set_unmapped(range, mmu_range); + * driver_garbage_collector_add(gpusvm, range); + * } + * } + */ + +#define DRM_GPUSVM_RANGE_START(_range) ((_range)->va.start) +#define DRM_GPUSVM_RANGE_END(_range) ((_range)->va.end - 1) +INTERVAL_TREE_DEFINE(struct drm_gpusvm_range, rb.node, u64, rb.__subtree_last, + DRM_GPUSVM_RANGE_START, DRM_GPUSVM_RANGE_END, + static __maybe_unused, range); + +#define DRM_GPUSVM_NOTIFIER_START(_notifier) ((_notifier)->interval.start) +#define DRM_GPUSVM_NOTIFIER_END(_notifier) ((_notifier)->interval.end - 1) +INTERVAL_TREE_DEFINE(struct drm_gpusvm_notifier, rb.node, u64, + rb.__subtree_last, DRM_GPUSVM_NOTIFIER_START, + DRM_GPUSVM_NOTIFIER_END, static __maybe_unused, notifier); + +/** + * npages_in_range() - Calculate the number of pages in a given range + * @start__: The start address of the range + * @end__: The end address of the range + * + * This macro calculates the number of pages in a given memory range, + * specified by the start and end addresses. It divides the difference + * between the end and start addresses by the page size (PAGE_SIZE) to + * determine the number of pages in the range. + * + * Return: The number of pages in the specified range. + */ +#define npages_in_range(start__, end__) \ + (((end__) - (start__)) >> PAGE_SHIFT) + +/** + * struct drm_gpusvm_zdd - GPU SVM zone device data + * + * @refcount: Reference count for the zdd + * @destroy_work: Work structure for asynchronous zdd destruction + * @range: Pointer to the GPU SVM range + * @vram_allocation: Driver-private pointer to the VRAM allocation + * + * This structure serves as a generic wrapper installed in + * page->zone_device_data. It provides infrastructure for looking up a range + * upon CPU page fault and asynchronously releasing VRAM once the CPU has no + * page references. Asynchronous release is useful because CPU page references + * can be dropped in IRQ contexts, while releasing VRAM likely requires sleeping + * locks. + */ +struct drm_gpusvm_zdd { + struct kref refcount; + struct work_struct destroy_work; + struct drm_gpusvm_range *range; + void *vram_allocation; +}; + +/** + * drm_gpusvm_zdd_destroy_work_func - Work function for destroying a zdd + * @w: Pointer to the work_struct + * + * This function releases VRAM, puts GPU SVM range, and frees zdd. + */ +static void drm_gpusvm_zdd_destroy_work_func(struct work_struct *w) +{ + struct drm_gpusvm_zdd *zdd = + container_of(w, struct drm_gpusvm_zdd, destroy_work); + struct drm_gpusvm_range *range = zdd->range; + struct drm_gpusvm *gpusvm = range->gpusvm; + + if (gpusvm->ops->vram_release && zdd->vram_allocation) + gpusvm->ops->vram_release(zdd->vram_allocation); + drm_gpusvm_range_put(range); + kfree(zdd); +} + +/** + * drm_gpusvm_zdd_alloc - Allocate a zdd structure. + * @range: Pointer to the GPU SVM range. + * + * This function allocates and initializes a new zdd structure. It sets up the + * reference count, initializes the destroy work, and links the provided GPU SVM + * range. + * + * Returns: + * Pointer to the allocated zdd on success, ERR_PTR() on failure. + */ +static struct drm_gpusvm_zdd * +drm_gpusvm_zdd_alloc(struct drm_gpusvm_range *range) +{ + struct drm_gpusvm_zdd *zdd; + + zdd = kmalloc(sizeof(*zdd), GFP_KERNEL); + if (!zdd) + return NULL; + + kref_init(&zdd->refcount); + INIT_WORK(&zdd->destroy_work, drm_gpusvm_zdd_destroy_work_func); + zdd->range = drm_gpusvm_range_get(range); + zdd->vram_allocation = NULL; + + return zdd; +} + +/** + * drm_gpusvm_zdd_get - Get a reference to a zdd structure. + * @zdd: Pointer to the zdd structure. + * + * This function increments the reference count of the provided zdd structure. + * + * Returns: Pointer to the zdd structure. + */ +static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd) +{ + kref_get(&zdd->refcount); + return zdd; +} + +/** + * drm_gpusvm_zdd_destroy - Destroy a zdd structure. + * @ref: Pointer to the reference count structure. + * + * This function queues the destroy_work of the zdd for asynchronous destruction. + */ +static void drm_gpusvm_zdd_destroy(struct kref *ref) +{ + struct drm_gpusvm_zdd *zdd = + container_of(ref, struct drm_gpusvm_zdd, refcount); + struct drm_gpusvm *gpusvm = zdd->range->gpusvm; + + queue_work(gpusvm->zdd_wq, &zdd->destroy_work); +} + +/** + * drm_gpusvm_zdd_put - Put a zdd reference. + * @zdd: Pointer to the zdd structure. + * + * This function decrements the reference count of the provided zdd structure + * and schedules its destruction if the count drops to zero. + */ +static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd) +{ + kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy); +} + +/** + * drm_gpusvm_range_find - Find GPU SVM range from GPU SVM notifier + * @notifier: Pointer to the GPU SVM notifier structure. + * @start: Start address of the range + * @end: End address of the range + * + * Return: A pointer to the drm_gpusvm_range if found or NULL + */ +struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end) +{ + return range_iter_first(¬ifier->root, start, end - 1); +} + +/** + * drm_gpusvm_for_each_range_safe - Safely iterate over GPU SVM ranges in a notifier + * @range__: Iterator variable for the ranges + * @next__: Iterator variable for the ranges temporay storage + * @notifier__: Pointer to the GPU SVM notifier + * @start__: Start address of the range + * @end__: End address of the range + * + * This macro is used to iterate over GPU SVM ranges in a notifier while + * removing ranges from it. + */ +#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__, start__, end__) \ + for ((range__) = drm_gpusvm_range_find((notifier__), (start__), (end__)), \ + (next__) = __drm_gpusvm_range_next(range__); \ + (range__) && (range__->va.start < (end__)); \ + (range__) = (next__), (next__) = __drm_gpusvm_range_next(range__)) + +/** + * __drm_gpusvm_notifier_next - get the next drm_gpusvm_notifier in the list + * @notifier: a pointer to the current drm_gpusvm_notifier + * + * Return: A pointer to the next drm_gpusvm_notifier if available, or NULL if + * the current notifier is the last one or if the input notifier is + * NULL. + */ +static struct drm_gpusvm_notifier * +__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier) +{ + if (notifier && !list_is_last(¬ifier->rb.entry, + ¬ifier->gpusvm->notifier_list)) + return list_next_entry(notifier, rb.entry); + + return NULL; +} + +/** + * drm_gpusvm_for_each_notifier - Iterate over GPU SVM notifiers in a gpusvm + * @notifier__: Iterator variable for the notifiers + * @notifier__: Pointer to the GPU SVM notifier + * @start__: Start address of the notifier + * @end__: End address of the notifier + * + * This macro is used to iterate over GPU SVM notifiers in a gpusvm. + */ +#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__, end__) \ + for ((notifier__) = notifier_iter_first(&(gpusvm__)->root, (start__), (end__) - 1); \ + (notifier__) && (notifier__->interval.start < (end__)); \ + (notifier__) = __drm_gpusvm_notifier_next(notifier__)) + +/** + * drm_gpusvm_for_each_notifier_safe - Safely iterate over GPU SVM notifiers in a gpusvm + * @notifier__: Iterator variable for the notifiers + * @next__: Iterator variable for the notifiers temporay storage + * @notifier__: Pointer to the GPU SVM notifier + * @start__: Start address of the notifier + * @end__: End address of the notifier + * + * This macro is used to iterate over GPU SVM notifiers in a gpusvm while + * removing notifiers from it. + */ +#define drm_gpusvm_for_each_notifier_safe(notifier__, next__, gpusvm__, start__, end__) \ + for ((notifier__) = notifier_iter_first(&(gpusvm__)->root, (start__), (end__) - 1), \ + (next__) = __drm_gpusvm_notifier_next(notifier__); \ + (notifier__) && (notifier__->interval.start < (end__)); \ + (notifier__) = (next__), (next__) = __drm_gpusvm_notifier_next(notifier__)) + +/** + * drm_gpusvm_notifier_invalidate - Invalidate a GPU SVM notifier. + * @mni: Pointer to the mmu_interval_notifier structure. + * @mmu_range: Pointer to the mmu_notifier_range structure. + * @cur_seq: Current sequence number. + * + * This function serves as a generic MMU notifier for GPU SVM. It sets the MMU + * notifier sequence number and calls the driver invalidate vfunc under + * gpusvm->notifier_lock. + * + * Returns: + * true if the operation succeeds, false otherwise. + */ +static bool +drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *mmu_range, + unsigned long cur_seq) +{ + struct drm_gpusvm_notifier *notifier = + container_of(mni, typeof(*notifier), notifier); + struct drm_gpusvm *gpusvm = notifier->gpusvm; + + if (!mmu_notifier_range_blockable(mmu_range)) + return false; + + down_write(&gpusvm->notifier_lock); + mmu_interval_set_seq(mni, cur_seq); + gpusvm->ops->invalidate(gpusvm, notifier, mmu_range); + up_write(&gpusvm->notifier_lock); + + return true; +} + +/** + * drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM + */ +static const struct mmu_interval_notifier_ops drm_gpusvm_notifier_ops = { + .invalidate = drm_gpusvm_notifier_invalidate, +}; + +/** + * drm_gpusvm_init - Initialize the GPU SVM. + * @gpusvm: Pointer to the GPU SVM structure. + * @name: Name of the GPU SVM. + * @drm: Pointer to the DRM device structure. + * @mm: Pointer to the mm_struct for the address space. + * @device_private_page_owner: Device private pages owner. + * @mm_start: Start address of GPU SVM. + * @mm_range: Range of the GPU SVM. + * @notifier_size: Size of individual notifiers. + * @ops: Pointer to the operations structure for GPU SVM. + * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation. + * Entries should be powers of 2 in descending order with last + * entry being SZ_4K. + * @num_chunks: Number of chunks. + * + * This function initializes the GPU SVM. + * + * Returns: + * 0 on success, a negative error code on failure. + */ +int drm_gpusvm_init(struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void *device_private_page_owner, + u64 mm_start, u64 mm_range, u64 notifier_size, + const struct drm_gpusvm_ops *ops, + const u64 *chunk_sizes, int num_chunks) +{ + if (!ops->invalidate || !num_chunks) + return -EINVAL; + + gpusvm->name = name; + gpusvm->drm = drm; + gpusvm->mm = mm; + gpusvm->device_private_page_owner = device_private_page_owner; + gpusvm->mm_start = mm_start; + gpusvm->mm_range = mm_range; + gpusvm->notifier_size = notifier_size; + gpusvm->ops = ops; + gpusvm->chunk_sizes = chunk_sizes; + gpusvm->num_chunks = num_chunks; + gpusvm->zdd_wq = system_wq; + + mmgrab(mm); + gpusvm->root = RB_ROOT_CACHED; + INIT_LIST_HEAD(&gpusvm->notifier_list); + + init_rwsem(&gpusvm->notifier_lock); + + fs_reclaim_acquire(GFP_KERNEL); + might_lock(&gpusvm->notifier_lock); + fs_reclaim_release(GFP_KERNEL); + + return 0; +} + +/** + * drm_gpusvm_notifier_find - Find GPU SVM notifier + * @gpusvm__: Pointer to the GPU SVM structure + * @fault_addr__: Fault address + * + * This macro finds the GPU SVM notifier associated with the fault address. + * + * Returns: + * Pointer to the GPU SVM notifier on success, NULL otherwise. + */ +#define drm_gpusvm_notifier_find(gpusvm__, fault_addr__) \ + notifier_iter_first(&(gpusvm__)->root, (fault_addr__), \ + (fault_addr__ + 1)) + +/** + * to_drm_gpusvm_notifier - retrieve the container struct for a given rbtree node + * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct + * + * Return: A pointer to the containing drm_gpusvm_notifier structure. + */ +#define to_drm_gpusvm_notifier(__node) \ + container_of((__node), struct drm_gpusvm_notifier, rb.node) + +/** + * drm_gpusvm_notifier_insert - Insert GPU SVM notifier + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier structure + * + * This function inserts the GPU SVM notifier into the GPU SVM RB tree and list. + */ +static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier) +{ + struct rb_node *node; + struct list_head *head; + + notifier_insert(notifier, &gpusvm->root); + + node = rb_prev(¬ifier->rb.node); + if (node) + head = &(to_drm_gpusvm_notifier(node))->rb.entry; + else + head = &gpusvm->notifier_list; + + list_add(¬ifier->rb.entry, head); +} + +/** + * drm_gpusvm_notifier_remove - Remove GPU SVM notifier + * @gpusvm__: Pointer to the GPU SVM tructure + * @notifier__: Pointer to the GPU SVM notifier structure + * + * This macro removes the GPU SVM notifier from the GPU SVM RB tree and list. + */ +#define drm_gpusvm_notifier_remove(gpusvm__, notifier__) \ + notifier_remove((notifier__), &(gpusvm__)->root); \ + list_del(&(notifier__)->rb.entry) + +/** + * drm_gpusvm_fini - Finalize the GPU SVM. + * @gpusvm: Pointer to the GPU SVM structure. + * + * This function finalizes the GPU SVM by cleaning up any remaining ranges and + * notifiers, and dropping a reference to struct MM. + */ +void drm_gpusvm_fini(struct drm_gpusvm *gpusvm) +{ + struct drm_gpusvm_notifier *notifier, *next; + + drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0, LONG_MAX) { + struct drm_gpusvm_range *range, *__next; + + /* + * Remove notifier first to avoid racing with any invalidation + */ + mmu_interval_notifier_remove(¬ifier->notifier); + notifier->flags.removed = true; + + drm_gpusvm_for_each_range_safe(range, __next, notifier, 0, + LONG_MAX) + drm_gpusvm_range_remove(gpusvm, range); + } + + mmdrop(gpusvm->mm); + WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root)); +} + +/** + * drm_gpusvm_notifier_alloc - Allocate GPU SVM notifier + * @gpusvm: Pointer to the GPU SVM structure + * @fault_addr: Fault address + * + * This function allocates and initializes the GPU SVM notifier structure. + * + * Returns: + * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure. + */ +static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, u64 fault_addr) +{ + struct drm_gpusvm_notifier *notifier; + + if (gpusvm->ops->notifier_alloc) + notifier = gpusvm->ops->notifier_alloc(); + else + notifier = kzalloc(sizeof(*notifier), GFP_KERNEL); + + if (!notifier) + return ERR_PTR(-ENOMEM); + + notifier->gpusvm = gpusvm; + notifier->interval.start = ALIGN_DOWN(fault_addr, gpusvm->notifier_size); + notifier->interval.end = ALIGN(fault_addr + 1, gpusvm->notifier_size); + INIT_LIST_HEAD(¬ifier->rb.entry); + notifier->root = RB_ROOT_CACHED; + INIT_LIST_HEAD(¬ifier->range_list); + + return notifier; +} + +/** + * drm_gpusvm_notifier_free - Free GPU SVM notifier + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier structure + * + * This function frees the GPU SVM notifier structure. + */ +static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier) +{ + WARN_ON(!RB_EMPTY_ROOT(¬ifier->root.rb_root)); + + if (gpusvm->ops->notifier_free) + gpusvm->ops->notifier_free(notifier); + else + kfree(notifier); +} + +/** + * to_drm_gpusvm_range - retrieve the container struct for a given rbtree node + * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_range struct + * + * Return: A pointer to the containing drm_gpusvm_range structure. + */ +#define to_drm_gpusvm_range(node__) \ + container_of((node__), struct drm_gpusvm_range, rb.node) + +/** + * drm_gpusvm_range_insert - Insert GPU SVM range + * @notifier: Pointer to the GPU SVM notifier structure + * @range: Pointer to the GPU SVM range structure + * + * This function inserts the GPU SVM range into the notifier RB tree and list. + */ +static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier, + struct drm_gpusvm_range *range) +{ + struct rb_node *node; + struct list_head *head; + + drm_gpusvm_notifier_lock(notifier->gpusvm); + range_insert(range, ¬ifier->root); + + node = rb_prev(&range->rb.node); + if (node) + head = &(to_drm_gpusvm_range(node))->rb.entry; + else + head = ¬ifier->range_list; + + list_add(&range->rb.entry, head); + drm_gpusvm_notifier_unlock(notifier->gpusvm); +} + +/** + * __drm_gpusvm_range_remove - Remove GPU SVM range + * @notifier__: Pointer to the GPU SVM notifier structure + * @range__: Pointer to the GPU SVM range structure + * + * This macro removes the GPU SVM range from the notifier RB tree and list. + */ +#define __drm_gpusvm_range_remove(notifier__, range__) \ + range_remove((range__), &(notifier__)->root); \ + list_del(&(range__)->rb.entry) + +/** + * drm_gpusvm_range_alloc - Allocate GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier structure + * @fault_addr: Fault address + * @chunk_size: Chunk size + * @migrate_vram: Flag indicating whether to migrate VRAM + * + * This function allocates and initializes the GPU SVM range structure. + * + * Returns: + * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure. + */ +static struct drm_gpusvm_range * +drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + u64 fault_addr, u64 chunk_size, bool migrate_vram) +{ + struct drm_gpusvm_range *range; + + if (gpusvm->ops->range_alloc) + range = gpusvm->ops->range_alloc(gpusvm); + else + range = kzalloc(sizeof(*range), GFP_KERNEL); + + if (!range) + return ERR_PTR(-ENOMEM); + + kref_init(&range->refcount); + range->gpusvm = gpusvm; + range->notifier = notifier; + range->va.start = ALIGN_DOWN(fault_addr, chunk_size); + range->va.end = ALIGN(fault_addr + 1, chunk_size); + INIT_LIST_HEAD(&range->rb.entry); + range->notifier_seq = LONG_MAX; + range->flags.migrate_vram = migrate_vram ? 1 : 0; + + return range; +} + +/** + * drm_gpusvm_check_pages - Check pages + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier structure + * @start: Start address + * @end: End address + * + * Check if pages between start and end have been faulted in on the CPU. Use to + * prevent migration of pages without CPU backing store. + * + * Returns: + * True if pages have been faulted into CPU, False otherwise + */ +static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + u64 start, u64 end) +{ + struct hmm_range hmm_range = { + .default_flags = 0, + .notifier = ¬ifier->notifier, + .start = start, + .end = end, + .dev_private_owner = gpusvm->device_private_page_owner, + }; + unsigned long timeout = + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + unsigned long *pfns; + unsigned long npages = npages_in_range(start, end); + int err, i; + + mmap_assert_locked(gpusvm->mm); + + pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL); + if (!pfns) + return false; + + hmm_range.notifier_seq = mmu_interval_read_begin(¬ifier->notifier); + hmm_range.hmm_pfns = pfns; + + while (true) { + err = hmm_range_fault(&hmm_range); + if (err == -EBUSY) { + if (time_after(jiffies, timeout)) + break; + + hmm_range.notifier_seq = mmu_interval_read_begin(¬ifier->notifier); + continue; + } + break; + } + if (err) + goto err_free; + + for (i = 0; i < npages; ++i) { + if (!(pfns[i] & HMM_PFN_VALID)) { + err = -EFAULT; + goto err_free; + } + } + +err_free: + kvfree(pfns); + return err ? false : true; +} + +/** + * drm_gpusvm_range_chunk_size - Determine chunk size for GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier structure + * @vas: Pointer to the virtual memory area structure + * @fault_addr: Fault address + * @gpuva_start: Start address of GPUVA which mirrors CPU + * @gpuva_end: End address of GPUVA which mirrors CPU + * @check_pages: Flag indicating whether to check pages + * + * This function determines the chunk size for the GPU SVM range based on the + * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual + * memory area boundaries. + * + * Returns: + * Chunk size on success, LONG_MAX on failure. + */ +static u64 drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + struct vm_area_struct *vas, + u64 fault_addr, u64 gpuva_start, + u64 gpuva_end, bool check_pages) +{ + u64 start, end; + int i = 0; + +retry: + for (; i < gpusvm->num_chunks; ++i) { + start = ALIGN_DOWN(fault_addr, gpusvm->chunk_sizes[i]); + end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]); + + if (start >= vas->vm_start && end <= vas->vm_end && + start >= notifier->interval.start && + end <= notifier->interval.end && + start >= gpuva_start && end <= gpuva_end) + break; + } + + if (i == gpusvm->num_chunks) + return LONG_MAX; + + /* + * If allocation more than page, ensure not to overlap with existing + * ranges. + */ + if (end - start != SZ_4K) { + struct drm_gpusvm_range *range; + + range = drm_gpusvm_range_find(notifier, start, end); + if (range) { + ++i; + goto retry; + } + + /* + * XXX: Only create range on pages CPU has faulted in. Without + * this check, or prefault, on BMG 'xe_exec_system_allocator --r + * process-many-malloc' fails. In the failure case, each process + * mallocs 16k but the CPU VMA is ~128k which results in 64k SVM + * ranges. When migrating the SVM ranges, some processes fail in + * drm_gpusvm_migrate_to_vram with 'migrate.cpages != npages' + * and then upon drm_gpusvm_range_get_pages device pages from + * other processes are collected + faulted in which creates all + * sorts of problems. Unsure exactly how this happening, also + * problem goes away if 'xe_exec_system_allocator --r + * process-many-malloc' mallocs at least 64k at a time. + */ + if (check_pages && + !drm_gpusvm_check_pages(gpusvm, notifier, start, end)) { + ++i; + goto retry; + } + } + + return end - start; +} + +/** + * drm_gpusvm_range_find_or_insert - Find or insert GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @fault_addr: Fault address + * @gpuva_start: Start address of GPUVA which mirrors CPU + * @gpuva_end: End address of GPUVA which mirrors CPU + * @ctx: GPU SVM context + * + * This function finds or inserts a newly allocated a GPU SVM range based on the + * fault address. Caller must hold a lock to protect range lookup and insertion. + * + * Returns: + * Pointer to the GPU SVM range on success, ERR_PTR() on failure. + */ +struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr, + u64 gpuva_start, u64 gpuva_end, + const struct drm_gpusvm_ctx *ctx) +{ + struct drm_gpusvm_notifier *notifier; + struct drm_gpusvm_range *range; + struct mm_struct *mm = gpusvm->mm; + struct vm_area_struct *vas; + bool notifier_alloc = false; + u64 chunk_size; + int err; + bool migrate_vram; + + if (fault_addr < gpusvm->mm_start || + fault_addr > gpusvm->mm_start + gpusvm->mm_range) { + err = -EINVAL; + goto err_out; + } + + if (!ctx->mmap_locked) { + if (!mmget_not_zero(mm)) { + err = -EFAULT; + goto err_out; + } + mmap_write_lock(mm); + } + + mmap_assert_write_locked(mm); + + notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr); + if (!notifier) { + notifier = drm_gpusvm_notifier_alloc(gpusvm, fault_addr); + if (IS_ERR(notifier)) { + err = PTR_ERR(notifier); + goto err_mmunlock; + } + notifier_alloc = true; + err = mmu_interval_notifier_insert_locked(¬ifier->notifier, + mm, notifier->interval.start, + notifier->interval.end - + notifier->interval.start, + &drm_gpusvm_notifier_ops); + if (err) + goto err_notifier; + } + + vas = vma_lookup(mm, fault_addr); + if (!vas) { + err = -ENOENT; + goto err_notifier_remove; + } + + if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) { + err = -EPERM; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_find(notifier, fault_addr, fault_addr + 1); + if (range) + goto out_mmunlock; + /* + * XXX: Short-circuiting migration based on migrate_vma_* current + * limitations. If/when migrate_vma_* add more support, this logic will + * have to change. + */ + migrate_vram = ctx->vram_possible && + vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas); + + chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier, vas, + fault_addr, gpuva_start, + gpuva_end, migrate_vram && + !ctx->prefault); + if (chunk_size == LONG_MAX) { + err = -EINVAL; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr, chunk_size, + migrate_vram); + if (IS_ERR(range)) { + err = PTR_ERR(range); + goto err_notifier_remove; + } + + drm_gpusvm_range_insert(notifier, range); + if (notifier_alloc) + drm_gpusvm_notifier_insert(gpusvm, notifier); + + if (ctx->prefault) { + struct drm_gpusvm_ctx __ctx = *ctx; + + __ctx.mmap_locked = true; + err = drm_gpusvm_range_get_pages(gpusvm, range, &__ctx); + if (err) + goto err_range_remove; + } + +out_mmunlock: + if (!ctx->mmap_locked) { + mmap_write_unlock(mm); + mmput(mm); + } + + return range; + +err_range_remove: + __drm_gpusvm_range_remove(notifier, range); +err_notifier_remove: + if (notifier_alloc) + mmu_interval_notifier_remove(¬ifier->notifier); +err_notifier: + if (notifier_alloc) + drm_gpusvm_notifier_free(gpusvm, notifier); +err_mmunlock: + if (!ctx->mmap_locked) { + mmap_write_unlock(mm); + mmput(mm); + } +err_out: + return ERR_PTR(err); +} + +/** + * for_each_dma_page - iterate over pages in a DMA regio`n + * @i__: the current page index in the iteration + * @j__: the current page index, log order, in the iteration + * @npages__: the total number of pages in the DMA region + * @order__: the order of the pages in the DMA region + * + * This macro iterates over each page in a DMA region. The DMA region + * is assumed to be composed of 2^@order__ pages, and the macro will + * step through the region one block of 2^@order__ pages at a time. + */ +#define for_each_dma_page(i__, j__, npages__, order__) \ + for ((i__) = 0, (j__) = 0; (i__) < (npages__); \ + (j__)++, (i__) += 0x1 << (order__)) + +/** + * __drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range (internal) + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * + * This function unmap pages associated with a GPU SVM range. Assumes and + * asserts correct locking is in place when called. + */ +static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + lockdep_assert_held(&gpusvm->notifier_lock); + + if (range->pages) { + unsigned long i, j, npages = npages_in_range(range->va.start, + range->va.end); + + if (range->flags.has_dma_mapping) { + for_each_dma_page(i, j, npages, range->order) + dma_unmap_page(gpusvm->drm->dev, + range->dma_addr[j], + PAGE_SIZE << range->order, + DMA_BIDIRECTIONAL); + } + + range->flags.has_vram_pages = false; + range->flags.has_dma_mapping = false; + } +} + +/** + * drm_gpusvm_range_free_pages - Free pages associated with a GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * + * This function free pages associated with a GPU SVM range. + */ +static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + lockdep_assert_held(&gpusvm->notifier_lock); + + if (range->pages) { + if (range->flags.kfree_mapping) { + kfree(range->dma_addr); + range->flags.kfree_mapping = false; + range->pages = NULL; + } else { + kvfree(range->pages); + range->pages = NULL; + } + } +} + +/** + * drm_gpusvm_range_remove - Remove GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range to be removed + * + * This function removes the specified GPU SVM range and also removes the parent + * GPU SVM notifier if no more ranges remain in the notifier. The caller must + * hold a lock to protect range and notifier removal. + */ +void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + struct drm_gpusvm_notifier *notifier; + + notifier = drm_gpusvm_notifier_find(gpusvm, range->va.start); + if (WARN_ON_ONCE(!notifier)) + return; + + drm_gpusvm_notifier_lock(gpusvm); + __drm_gpusvm_range_unmap_pages(gpusvm, range); + drm_gpusvm_range_free_pages(gpusvm, range); + __drm_gpusvm_range_remove(notifier, range); + drm_gpusvm_notifier_unlock(gpusvm); + + drm_gpusvm_range_put(range); + + if (RB_EMPTY_ROOT(¬ifier->root.rb_root)) { + if (!notifier->flags.removed) + mmu_interval_notifier_remove(¬ifier->notifier); + drm_gpusvm_notifier_remove(gpusvm, notifier); + drm_gpusvm_notifier_free(gpusvm, notifier); + } +} + +/** + * drm_gpusvm_range_get - Get a reference to GPU SVM range + * @range: Pointer to the GPU SVM range + * + * This function increments the reference count of the specified GPU SVM range. + * + * Returns: + * Pointer to the GPU SVM range. + */ +struct drm_gpusvm_range * +drm_gpusvm_range_get(struct drm_gpusvm_range *range) +{ + kref_get(&range->refcount); + + return range; +} + +/** + * drm_gpusvm_range_destroy - Destroy GPU SVM range + * @refcount: Pointer to the reference counter embedded in the GPU SVM range + * + * This function destroys the specified GPU SVM range when its reference count + * reaches zero. If a custom range-free function is provided, it is invoked to + * free the range; otherwise, the range is deallocated using kfree(). + */ +static void drm_gpusvm_range_destroy(struct kref *refcount) +{ + struct drm_gpusvm_range *range = + container_of(refcount, struct drm_gpusvm_range, refcount); + struct drm_gpusvm *gpusvm = range->gpusvm; + + if (gpusvm->ops->range_free) + gpusvm->ops->range_free(range); + else + kfree(range); +} + +/** + * drm_gpusvm_range_put - Put a reference to GPU SVM range + * @range: Pointer to the GPU SVM range + * + * This function decrements the reference count of the specified GPU SVM range + * and frees it when the count reaches zero. + */ +void drm_gpusvm_range_put(struct drm_gpusvm_range *range) +{ + kref_put(&range->refcount, drm_gpusvm_range_destroy); +} + +/** + * drm_gpusvm_range_pages_valid - GPU SVM range pages valid + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * + * This function determines if a GPU SVM range pages are valid. Expected be + * called holding gpusvm->notifier_lock and as the last step before commiting a + * GPU binding. + * + * Returns: + * True if GPU SVM range has valid pages, False otherwise + */ +bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + lockdep_assert_held(&gpusvm->notifier_lock); + + return range->flags.has_vram_pages || range->flags.has_dma_mapping; +} + +/** + * drm_gpusvm_range_pages_valid_unlocked - GPU SVM range pages valid unlocked + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * + * This function determines if a GPU SVM range pages are valid. Expected be + * called without holding gpusvm->notifier_lock. + * + * Returns: + * True if GPU SVM range has valid pages, False otherwise + */ +static bool +drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + bool pages_valid; + + if (!range->pages) + return false; + + drm_gpusvm_notifier_lock(gpusvm); + pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range); + if (!pages_valid && range->flags.kfree_mapping) { + kfree(range->dma_addr); + range->flags.kfree_mapping = false; + range->pages = NULL; + } + drm_gpusvm_notifier_unlock(gpusvm); + + return pages_valid; +} + +/** + * drm_gpusvm_range_get_pages - Get pages for a GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * @ctx: GPU SVM context + * + * This function gets pages for a GPU SVM range and ensures they are mapped for + * DMA access. + * + * Returns: + * 0 on success, negative error code on failure. + */ +int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct mmu_interval_notifier *notifier = &range->notifier->notifier; + struct hmm_range hmm_range = { + .default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only ? 0 : + HMM_PFN_REQ_WRITE), + .notifier = notifier, + .start = range->va.start, + .end = range->va.end, + .dev_private_owner = gpusvm->device_private_page_owner, + }; + struct mm_struct *mm = gpusvm->mm; + unsigned long timeout = + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + unsigned long i, j; + unsigned long npages = npages_in_range(range->va.start, range->va.end); + unsigned int order = 0; + unsigned long *pfns; + struct page **pages; + int err = 0; + bool vram_pages = !!range->flags.migrate_vram; + bool alloc_pfns = false, kfree_mapping; + +retry: + kfree_mapping = false; + hmm_range.notifier_seq = mmu_interval_read_begin(notifier); + if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range)) + return 0; + + if (range->notifier_seq == hmm_range.notifier_seq && range->pages) { + if (ctx->prefault) + return 0; + + pfns = (unsigned long *)range->pages; + pages = range->pages; + goto map_pages; + } + + if (!range->pages) { + pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL); + if (!pfns) + return -ENOMEM; + alloc_pfns = true; + } else { + pfns = (unsigned long *)range->pages; + } + + if (!ctx->mmap_locked) { + if (!mmget_not_zero(mm)) { + err = -EFAULT; + goto err_out; + } + } + + hmm_range.hmm_pfns = pfns; + while (true) { + /* Must be checked after mmu_interval_read_begin */ + if (range->flags.unmapped) { + err = -EFAULT; + break; + } + + if (!ctx->mmap_locked) { + /* + * XXX: HMM locking document indicates only a read-lock + * is required but there apears to be a window between + * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault + * via migrate_vma_setup and the pages actually moving + * in migrate_vma_finalize in which this code can grab + * garbage pages. Grabbing the write-lock if the range + * is attached to vram appears to protect against this + * race. + */ + if (vram_pages) + mmap_write_lock(mm); + else + mmap_read_lock(mm); + } + err = hmm_range_fault(&hmm_range); + if (!ctx->mmap_locked) { + if (vram_pages) + mmap_write_unlock(mm); + else + mmap_read_unlock(mm); + } + + if (err == -EBUSY) { + if (time_after(jiffies, timeout)) + break; + + hmm_range.notifier_seq = mmu_interval_read_begin(notifier); + continue; + } + break; + } + if (!ctx->mmap_locked) + mmput(mm); + if (err) + goto err_free; + + pages = (struct page **)pfns; + + if (ctx->prefault) { + range->pages = pages; + goto set_seqno; + } + +map_pages: + if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) { + WARN_ON_ONCE(!range->vram_allocation); + + for (i = 0; i < npages; ++i) { + pages[i] = hmm_pfn_to_page(pfns[i]); + + if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) { + err = -EOPNOTSUPP; + goto err_free; + } + } + + /* Do not race with notifier unmapping pages */ + drm_gpusvm_notifier_lock(gpusvm); + range->flags.has_vram_pages = true; + range->pages = pages; + if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) { + err = -EAGAIN; + __drm_gpusvm_range_unmap_pages(gpusvm, range); + } + drm_gpusvm_notifier_unlock(gpusvm); + } else { + dma_addr_t *dma_addr = (dma_addr_t *)pfns; + + for_each_dma_page(i, j, npages, order) { + if (WARN_ON_ONCE(i && order != + hmm_pfn_to_map_order(pfns[i]))) { + err = -EOPNOTSUPP; + npages = i; + goto err_unmap; + } + order = hmm_pfn_to_map_order(pfns[i]); + + pages[j] = hmm_pfn_to_page(pfns[i]); + if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) { + err = -EOPNOTSUPP; + npages = i; + goto err_unmap; + } + + set_page_dirty_lock(pages[j]); + mark_page_accessed(pages[j]); + + dma_addr[j] = dma_map_page(gpusvm->drm->dev, + pages[j], 0, + PAGE_SIZE << order, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) { + err = -EFAULT; + npages = i; + goto err_unmap; + } + } + + /* Huge pages, reduce memory footprint */ + if (order) { + dma_addr = kmalloc_array(j, sizeof(*dma_addr), + GFP_KERNEL); + if (dma_addr) { + for (i = 0; i < j; ++i) + dma_addr[i] = (dma_addr_t)pfns[i]; + kvfree(pfns); + kfree_mapping = true; + } else { + dma_addr = (dma_addr_t *)pfns; + } + } + + /* Do not race with notifier unmapping pages */ + drm_gpusvm_notifier_lock(gpusvm); + range->order = order; + range->flags.kfree_mapping = kfree_mapping; + range->flags.has_dma_mapping = true; + range->dma_addr = dma_addr; + range->vram_allocation = NULL; + if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) { + err = -EAGAIN; + __drm_gpusvm_range_unmap_pages(gpusvm, range); + } + drm_gpusvm_notifier_unlock(gpusvm); + } + + if (err == -EAGAIN) + goto retry; +set_seqno: + range->notifier_seq = hmm_range.notifier_seq; + + return 0; + +err_unmap: + for_each_dma_page(i, j, npages, order) + dma_unmap_page(gpusvm->drm->dev, + (dma_addr_t)pfns[j], + PAGE_SIZE << order, DMA_BIDIRECTIONAL); +err_free: + if (alloc_pfns) + kvfree(pfns); +err_out: + return err; +} + +/** + * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * @ctx: GPU SVM context + * + * This function unmaps pages associated with a GPU SVM range. If @in_notifier + * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it + * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on + * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU + * security model. + */ +void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + if (ctx->in_notifier) + lockdep_assert_held_write(&gpusvm->notifier_lock); + else + drm_gpusvm_notifier_lock(gpusvm); + + __drm_gpusvm_range_unmap_pages(gpusvm, range); + + if (!ctx->in_notifier) + drm_gpusvm_notifier_unlock(gpusvm); +} + +/** + * drm_gpusvm_migration_put_page - Put a migration page + * @page: Pointer to the page to put + * + * This function unlocks and puts a page. + */ +static void drm_gpusvm_migration_put_page(struct page *page) +{ + unlock_page(page); + put_page(page); +} + +/** + * drm_gpusvm_migration_put_pages - Put migration pages + * @npages: Number of pages + * @migrate_pfn: Array of migrate page frame numbers + * + * This function puts an array of pages. + */ +static void drm_gpusvm_migration_put_pages(unsigned long npages, + unsigned long *migrate_pfn) +{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!migrate_pfn[i]) + continue; + + drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i])); + migrate_pfn[i] = 0; + } +} + +/** + * drm_gpusvm_get_vram_page - Get a reference to a VRAM page + * @page: Pointer to the page + * @zdd: Pointer to the GPU SVM zone device data + * + * This function associates the given page with the specified GPU SVM zone + * device data and initializes it for zone device usage. + */ +static void drm_gpusvm_get_vram_page(struct page *page, + struct drm_gpusvm_zdd *zdd) +{ + page->zone_device_data = drm_gpusvm_zdd_get(zdd); + zone_device_page_init(page); +} + +/** + * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration + * @dev: The device for which the pages are being mapped + * @dma_addr: Array to store DMA addresses corresponding to mapped pages + * @migrate_pfn: Array of migrate page frame numbers to map + * @npages: Number of pages to map + * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL) + * + * This function maps pages of memory for migration usage in GPU SVM. It + * iterates over each page frame number provided in @migrate_pfn, maps the + * corresponding page, and stores the DMA address in the provided @dma_addr + * array. + * + * Return: 0 on success, -EFAULT if an error occurs during mapping. + */ +static int drm_gpusvm_migrate_map_pages(struct device *dev, + dma_addr_t *dma_addr, + long unsigned int *migrate_pfn, + unsigned long npages, + enum dma_data_direction dir) +{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + struct page *page = migrate_pfn_to_page(migrate_pfn[i]); + + if (!page) + continue; + + if (WARN_ON_ONCE(is_zone_device_page(page))) + return -EFAULT; + + dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir); + if (dma_mapping_error(dev, dma_addr[i])) + return -EFAULT; + } + + return 0; +} + +/** + * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration + * @dev: The device for which the pages were mapped + * @dma_addr: Array of DMA addresses corresponding to mapped pages + * @npages: Number of pages to unmap + * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL) + * + * This function unmaps previously mapped pages of memory for GPU Shared Virtual + * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks + * if it's valid and not already unmapped, and unmaps the corresponding page. + */ +static void drm_gpusvm_migrate_unmap_pages(struct device *dev, + dma_addr_t *dma_addr, + unsigned long npages, + enum dma_data_direction dir) +{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i])) + continue; + + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); + } +} + +/** + * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * failure of this function. + * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller + * should hold a reference to the VRAM allocation, which + * should be dropped via ops->vram_allocation or upon the + * failure of this function. + * @ctx: GPU SVM context + * + * This function migrates the specified GPU SVM range to VRAM. It performs the + * necessary setup and invokes the driver-specific operations for migration to + * VRAM. Upon successful return, @vram_allocation can safely reference @range + * until ops->vram_release is called which only upon successful return. + * + * Returns: + * 0 on success, negative error code on failure. + */ +int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + void *vram_allocation, + const struct drm_gpusvm_ctx *ctx) +{ + u64 start = range->va.start, end = range->va.end; + struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = gpusvm->device_private_page_owner, + .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; + struct mm_struct *mm = gpusvm->mm; + unsigned long i, npages = npages_in_range(start, end); + struct vm_area_struct *vas; + struct drm_gpusvm_zdd *zdd = NULL; + struct page **pages; + dma_addr_t *dma_addr; + void *buf; + int err; + + if (!range->flags.migrate_vram) + return -EINVAL; + + if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram || + !gpusvm->ops->copy_to_sram) + return -EOPNOTSUPP; + + if (!ctx->mmap_locked) { + if (!mmget_not_zero(mm)) { + err = -EFAULT; + goto err_out; + } + mmap_write_lock(mm); + } + + mmap_assert_locked(mm); + + vas = vma_lookup(mm, start); + if (!vas) { + err = -ENOENT; + goto err_mmunlock; + } + + if (end > vas->vm_end || start < vas->vm_start) { + err = -EINVAL; + goto err_mmunlock; + } + + if (!vma_is_anonymous(vas)) { + err = -EBUSY; + goto err_mmunlock; + } + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); + if (!buf) { + err = -ENOMEM; + goto err_mmunlock; + } + dma_addr = buf + (2 * sizeof(*migrate.src) * npages); + pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages; + + zdd = drm_gpusvm_zdd_alloc(range); + if (!zdd) { + err = -ENOMEM; + goto err_free; + } + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err = migrate_vma_setup(&migrate); + if (err) + goto err_free; + + /* + * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not + * always an error. Need to revisit possible cases and how to handle. We + * could prefault on migrate.cpages != npages via hmm_range_fault. + */ + + if (!migrate.cpages) { + err = -EFAULT; + goto err_free; + } + + if (migrate.cpages != npages) { + err = -EBUSY; + goto err_finalize; + } + + err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages, + migrate.dst); + if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr, + migrate.src, npages, DMA_TO_DEVICE); + if (err) + goto err_finalize; + + for (i = 0; i < npages; ++i) { + struct page *page = pfn_to_page(migrate.dst[i]); + + pages[i] = page; + migrate.dst[i] = migrate_pfn(migrate.dst[i]); + drm_gpusvm_get_vram_page(page, zdd); + } + + err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages); + if (err) + goto err_finalize; + + /* Upon success bind vram allocation to range and zdd */ + range->vram_allocation = vram_allocation; + WRITE_ONCE(zdd->vram_allocation, vram_allocation); /* Owns ref */ + +err_finalize: + if (err) + drm_gpusvm_migration_put_pages(npages, migrate.dst); + migrate_vma_pages(&migrate); + migrate_vma_finalize(&migrate); + drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages, + DMA_TO_DEVICE); +err_free: + if (zdd) + drm_gpusvm_zdd_put(zdd); + kvfree(buf); +err_mmunlock: + if (!ctx->mmap_locked) { + mmap_write_unlock(mm); + mmput(mm); + } +err_out: + return err; +} + +/** + * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area + * @vas: Pointer to the VM area structure, can be NULL + * @npages: Number of pages to populate + * @src_mpfn: Source array of migrate PFNs + * @mpfn: Array of migrate PFNs to populate + * @addr: Start address for PFN allocation + * + * This function populates the SRAM migrate page frame numbers (PFNs) for the + * specified VM area structure. It allocates and locks pages in the VM area for + * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use + * alloc_page for allocation. + * + * Returns: + * 0 on success, negative error code on failure. + */ +static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas, + unsigned long npages, + unsigned long *src_mpfn, + unsigned long *mpfn, u64 addr) +{ + unsigned long i; + + for (i = 0; i < npages; ++i, addr += PAGE_SIZE) { + struct page *page; + + if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE)) + continue; + + if (vas) + page = alloc_page_vma(GFP_HIGHUSER, vas, addr); + else + page = alloc_page(GFP_HIGHUSER); + + if (!page) + return -ENOMEM; + + lock_page(page); + mpfn[i] = migrate_pfn(page_to_pfn(page)); + } + + return 0; +} + +/** + * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * + * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and + * migration done via migrate_device_* functions. Fallback path as it is + * preferred to issue migrations with mmap lock. + * + * Returns: + * 0 on success, negative error code on failure. + */ +static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + unsigned long npages; + struct page **pages; + unsigned long *src, *dst; + dma_addr_t *dma_addr; + void *buf; + int i, err = 0; + + npages = npages_in_range(range->va.start, range->va.end); + + buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); + if (!buf) { + err = -ENOMEM; + goto err_out; + } + src = buf; + dst = buf + (sizeof(*src) * npages); + dma_addr = buf + (2 * sizeof(*src) * npages); + pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages; + + err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation, + npages, src); + if (err) + goto err_free; + + err = migrate_device_vma_range(gpusvm->mm, + gpusvm->device_private_page_owner, src, + npages, range->va.start); + if (err) + goto err_free; + + err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0); + if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr, + dst, npages, DMA_BIDIRECTIONAL); + if (err) + goto err_finalize; + + for (i = 0; i < npages; ++i) + pages[i] = migrate_pfn_to_page(src[i]); + + err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages); + if (err) + goto err_finalize; + +err_finalize: + if (err) + drm_gpusvm_migration_put_pages(npages, dst); + migrate_device_pages(src, dst, npages); + migrate_device_finalize(src, dst, npages); + drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages, + DMA_BIDIRECTIONAL); +err_free: + kvfree(buf); +err_out: + + return err; +} + +/** + * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal) + * @gpusvm: Pointer to the GPU SVM structure + * @vas: Pointer to the VM area structure + * @page: Pointer to the page for fault handling (can be NULL) + * @start: Start address of the migration range + * @end: End address of the migration range + * + * This internal function performs the migration of the specified GPU SVM range + * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and + * invokes the driver-specific operations for migration to SRAM. + * + * Returns: + * 0 on success, negative error code on failure. + */ +static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm, + struct vm_area_struct *vas, + struct page *page, + u64 start, u64 end) +{ + struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = gpusvm->device_private_page_owner, + .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE, + .fault_page = page, + }; + unsigned long npages; + struct page **pages; + dma_addr_t *dma_addr; + void *buf; + int i, err = 0; + + mmap_assert_locked(gpusvm->mm); + + /* Corner where VMA area struct has been partially unmapped */ + if (start < vas->vm_start) + start = vas->vm_start; + if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages = npages_in_range(start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); + if (!buf) { + err = -ENOMEM; + goto err_out; + } + dma_addr = buf + (2 * sizeof(*migrate.src) * npages); + pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages; + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err = migrate_vma_setup(&migrate); + if (err) + goto err_free; + + /* Raced with another CPU fault, nothing to do */ + if (!migrate.cpages) + goto err_free; + + err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages, + migrate.src, migrate.dst, + start); + if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr, + migrate.dst, npages, + DMA_BIDIRECTIONAL); + if (err) + goto err_finalize; + + for (i = 0; i < npages; ++i) + pages[i] = migrate_pfn_to_page(migrate.src[i]); + + err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages); + if (err) + goto err_finalize; + +err_finalize: + if (err) + drm_gpusvm_migration_put_pages(npages, migrate.dst); + migrate_vma_pages(&migrate); + migrate_vma_finalize(&migrate); + drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages, + DMA_BIDIRECTIONAL); +err_free: + kvfree(buf); +err_out: + mmap_assert_locked(gpusvm->mm); + + return err; +} + +/** + * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM + * @gpusvm: Pointer to the GPU SVM structure + * @range: Pointer to the GPU SVM range structure + * @ctx: GPU SVM context + * + * This function initiates the migration of the specified GPU SVM range to + * SRAM. It performs necessary checks and invokes the internal migration + * function for actual migration. + * + * Returns: + * 0 on success, negative error code on failure. + */ +int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + u64 start = range->va.start, end = range->va.end; + struct mm_struct *mm = gpusvm->mm; + struct vm_area_struct *vas; + int err; + bool retry = false; + + if (!ctx->mmap_locked) { + if (!mmget_not_zero(mm)) { + err = -EFAULT; + goto err_out; + } + if (ctx->trylock_mmap) { + if (!mmap_read_trylock(mm)) { + err = drm_gpusvm_evict_to_sram(gpusvm, range); + goto err_mmput; + } + } else { + mmap_read_lock(mm); + } + } + + mmap_assert_locked(mm); + + /* + * Loop required to find all VMA area structs for the corner case when + * VRAM backing has been partially unmapped from MM's address space. + */ +again: + vas = find_vma(mm, start); + if (!vas) { + if (!retry) + err = -ENOENT; + goto err_mmunlock; + } + + if (end <= vas->vm_start || start >= vas->vm_end) { + if (!retry) + err = -EINVAL; + goto err_mmunlock; + } + + err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end); + if (err) + goto err_mmunlock; + + if (vas->vm_end < end) { + retry = true; + start = vas->vm_end; + goto again; + } + + if (!ctx->mmap_locked) { + mmap_read_unlock(mm); + /* + * Using mmput_async as this function can be called while + * holding a dma-resv lock, and a final put can grab the mmap + * lock, causing a lock inversion. + */ + mmput_async(mm); + } + + return 0; + +err_mmunlock: + if (!ctx->mmap_locked) + mmap_read_unlock(mm); +err_mmput: + if (!ctx->mmap_locked) + mmput_async(mm); +err_out: + return err; +} + +/** + * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page + * @page: Pointer to the page + * + * This function is a callback used to put the GPU SVM zone device data + * associated with a page when it is being released. + */ +static void drm_gpusvm_page_free(struct page *page) +{ + drm_gpusvm_zdd_put(page->zone_device_data); +} + +/** + * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler) + * @vmf: Pointer to the fault information structure + * + * This function is a page fault handler used to migrate a GPU SVM range to RAM. + * It retrieves the GPU SVM range information from the faulting page and invokes + * the internal migration function to migrate the range back to RAM. + * + * Returns: + * VM_FAULT_SIGBUS on failure, 0 on success. + */ +static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf) +{ + struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data; + int err; + + err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm, + vmf->vma, vmf->page, + zdd->range->va.start, + zdd->range->va.end); + + return err ? VM_FAULT_SIGBUS : 0; +} + +/** + * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM + */ +static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = { + .page_free = drm_gpusvm_page_free, + .migrate_to_ram = drm_gpusvm_migrate_to_ram, +}; + +/** + * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations + * + * Returns: + * Pointer to the GPU SVM device page map operations structure. + */ +const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void) +{ + return &drm_gpusvm_pagemap_ops; +} + +/** + * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range + * @gpusvm: Pointer to the GPU SVM structure. + * @start: Start address + * @end: End address + * + * Returns: + * True if GPU SVM has mapping, False otherwise + */ +bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end) +{ + struct drm_gpusvm_notifier *notifier; + + drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) { + struct drm_gpusvm_range *range = NULL; + + drm_gpusvm_for_each_range(range, notifier, start, end) + return true; + } + + return false; +} diff --git a/drivers/gpu/drm/xe/drm_gpusvm.h b/drivers/gpu/drm/xe/drm_gpusvm.h new file mode 100644 index 000000000000..0ea70f8534a8 --- /dev/null +++ b/drivers/gpu/drm/xe/drm_gpusvm.h @@ -0,0 +1,415 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 Intel Corporation + */ + +#ifndef __DRM_GPUSVM_H__ +#define __DRM_GPUSVM_H__ + +#include +#include +#include + +struct dev_pagemap_ops; +struct drm_device; +struct drm_gpusvm; +struct drm_gpusvm_notifier; +struct drm_gpusvm_ops; +struct drm_gpusvm_range; + +/** + * struct drm_gpusvm_ops - Operations structure for GPU SVM + * + * This structure defines the operations for GPU Shared Virtual Memory (SVM). + * These operations are provided by the GPU driver to manage SVM ranges and + * perform operations such as migration between VRAM and system RAM. + */ +struct drm_gpusvm_ops { + /** + * @notifier_alloc: Allocate a GPU SVM notifier (optional) + * + * This function shall allocate a GPU SVM notifier. + * + * Returns: + * Pointer to the allocated GPU SVM notifier on success, NULL on failure. + */ + struct drm_gpusvm_notifier *(*notifier_alloc)(void); + + /** + * @notifier_free: Free a GPU SVM notifier (optional) + * @notifier: Pointer to the GPU SVM notifier to be freed + * + * This function shall free a GPU SVM notifier. + */ + void (*notifier_free)(struct drm_gpusvm_notifier *notifier); + + /** + * @range_alloc: Allocate a GPU SVM range (optional) + * @gpusvm: Pointer to the GPU SVM + * + * This function shall allocate a GPU SVM range. + * + * Returns: + * Pointer to the allocated GPU SVM range on success, NULL on failure. + */ + struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm); + + /** + * @range_free: Free a GPU SVM range (optional) + * @range: Pointer to the GPU SVM range to be freed + * + * This function shall free a GPU SVM range. + */ + void (*range_free)(struct drm_gpusvm_range *range); + + /** + * @vram_release: Release VRAM allocation (optional) + * @vram_allocation: Driver-private pointer to the VRAM allocation + * + * This function shall release VRAM allocation and expects to drop a + * reference to VRAM allocation. + */ + void (*vram_release)(void *vram_allocation); + + /** + * @populate_vram_pfn: Populate VRAM PFN (required for migration) + * @gpusvm: Pointer to the GPU SVM + * @vram_allocation: Driver-private pointer to the VRAM allocation + * @npages: Number of pages to populate + * @pfn: Array of page frame numbers to populate + * + * This function shall populate VRAM page frame numbers (PFN). + * + * Returns: + * 0 on success, a negative error code on failure. + */ + int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm, + void *vram_allocation, + unsigned long npages, + unsigned long *pfn); + + /** + * @copy_to_vram: Copy to VRAM (required for migration) + * @gpusvm: Pointer to the GPU SVM + * @pages: Pointer to array of VRAM pages (destination) + * @dma_addr: Pointer to array of DMA addresses (source) + * @npages: Number of pages to copy + * + * This function shall copy pages to VRAM. + * + * Returns: + * 0 on success, a negative error code on failure. + */ + int (*copy_to_vram)(struct drm_gpusvm *gpusvm, + struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + + /** + * @copy_to_sram: Copy to system RAM (required for migration) + * @gpusvm: Pointer to the GPU SVM + * @pages: Pointer to array of VRAM pages (source) + * @dma_addr: Pointer to array of DMA addresses (destination) + * @npages: Number of pages to copy + * + * This function shall copy pages to system RAM. + * + * Returns: + * 0 on success, a negative error code on failure. + */ + int (*copy_to_sram)(struct drm_gpusvm *gpusvm, + struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + + /** + * @invalidate: Invalidate GPU SVM notifier (required) + * @gpusvm: Pointer to the GPU SVM + * @notifier: Pointer to the GPU SVM notifier + * @mmu_range: Pointer to the mmu_notifier_range structure + * + * This function shall invalidate the GPU page tables. It can safely + * walk the notifier range RB tree/list in this function. Called while + * holding the notifier lock. + */ + void (*invalidate)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range *mmu_range); +}; + +/** + * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier + * + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: MMU interval notifier + * @interval: Interval for the notifier + * @rb: Red-black tree node for the parent GPU SVM structure notifier tree + * @root: Cached root node of the RB tree containing ranges + * @range_list: List head containing of ranges in the same order they appear in + * interval tree. This is useful to keep iterating ranges while + * doing modifications to RB tree. + * @flags.removed: Flag indicating whether the MMU interval notifier has been + * removed + * + * This structure represents a GPU SVM notifier. + */ +struct drm_gpusvm_notifier { + struct drm_gpusvm *gpusvm; + struct mmu_interval_notifier notifier; + struct { + u64 start; + u64 end; + } interval; + struct { + struct rb_node node; + struct list_head entry; + u64 __subtree_last; + } rb; + struct rb_root_cached root; + struct list_head range_list; + struct { + u32 removed : 1; + } flags; +}; + +/** + * struct drm_gpusvm_range - Structure representing a GPU SVM range + * + * @gpusvm: Pointer to the GPU SVM structure + * @notifier: Pointer to the GPU SVM notifier + * @refcount: Reference count for the range + * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree + * @va: Virtual address range + * @notifier_seq: Notifier sequence number of the range's pages + * @pages: Pointer to the array of pages (if backing store is in VRAM) + * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped) + * @vram_allocation: Driver-private pointer to the VRAM allocation + * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size + * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM + * @flags.unmapped: Flag indicating if the range has been unmapped + * @flags.partial_unmap: Flag indicating if the range has been partially unmapped + * @flags.has_vram_pages: Flag indicating if the range has vram pages + * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping + * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based + * on @order which releases via kfree + * + * This structure represents a GPU SVM range used for tracking memory ranges + * mapped in a DRM device. + */ +struct drm_gpusvm_range { + struct drm_gpusvm *gpusvm; + struct drm_gpusvm_notifier *notifier; + struct kref refcount; + struct { + struct rb_node node; + struct list_head entry; + u64 __subtree_last; + } rb; + struct { + u64 start; + u64 end; + } va; + unsigned long notifier_seq; + union { + struct page **pages; + dma_addr_t *dma_addr; + }; + void *vram_allocation; + u16 order; + struct { + /* All flags below must be set upon creation */ + u16 migrate_vram : 1; + /* All flags below must be set / cleared under notifier lock */ + u16 unmapped : 1; + u16 partial_unmap : 1; + u16 has_vram_pages : 1; + u16 has_dma_mapping : 1; + u16 kfree_mapping : 1; + } flags; +}; + +/** + * struct drm_gpusvm - GPU SVM structure + * + * @name: Name of the GPU SVM + * @drm: Pointer to the DRM device structure + * @mm: Pointer to the mm_struct for the address space + * @device_private_page_owner: Device private pages owner + * @mm_start: Start address of GPU SVM + * @mm_range: Range of the GPU SVM + * @notifier_size: Size of individual notifiers + * @ops: Pointer to the operations structure for GPU SVM + * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation. + * Entries should be powers of 2 in descending order. + * @num_chunks: Number of chunks + * @notifier_lock: Read-write semaphore for protecting notifier operations + * @zdd_wq: Workqueue for deferred work on zdd destruction + * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers + * @notifier_list: list head containing of notifiers in the same order they + * appear in interval tree. This is useful to keep iterating + * notifiers while doing modifications to RB tree. + * + * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking + * memory ranges mapped in a DRM (Direct Rendering Manager) device. + * + * No reference counting is provided, as this is expected to be embedded in the + * driver VM structure along with the struct drm_gpuvm, which handles reference + * counting. + */ +struct drm_gpusvm { + const char *name; + struct drm_device *drm; + struct mm_struct *mm; + void *device_private_page_owner; + u64 mm_start; + u64 mm_range; + u64 notifier_size; + const struct drm_gpusvm_ops *ops; + const u64 *chunk_sizes; + int num_chunks; + struct rw_semaphore notifier_lock; + struct workqueue_struct *zdd_wq; + struct rb_root_cached root; + struct list_head notifier_list; +}; + +/** + * struct drm_gpusvm_ctx - DRM GPU SVM context + * + * @mmap_locked: mmap lock is locked + * @trylock_mmap: trylock mmap lock, used to avoid locking inversions + * (e.g.dma-revs -> mmap lock) + * @in_notifier: entering from a MMU notifier + * @read_only: operating on read-only memory + * @vram_possible: possible to use VRAM + * @prefault: prefault pages + * + * Context that is DRM GPUSVM is operating in (i.e. user arguments). + */ +struct drm_gpusvm_ctx { + u32 mmap_locked :1; + u32 trylock_mmap :1; + u32 in_notifier :1; + u32 read_only :1; + u32 vram_possible :1; + u32 prefault :1; +}; + +int drm_gpusvm_init(struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void *device_private_page_owner, + u64 mm_start, u64 mm_range, u64 notifier_size, + const struct drm_gpusvm_ops *ops, + const u64 *chunk_sizes, int num_chunks); +void drm_gpusvm_fini(struct drm_gpusvm *gpusvm); +void drm_gpusvm_free(struct drm_gpusvm *gpusvm); + +struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr, + u64 gpuva_start, u64 gpuva_end, + const struct drm_gpusvm_ctx *ctx); +void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +struct drm_gpusvm_range * +drm_gpusvm_range_get(struct drm_gpusvm_range *range); +void drm_gpusvm_range_put(struct drm_gpusvm_range *range); + +bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); +void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + void *vram_allocation, + const struct drm_gpusvm_ctx *ctx); +int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void); + +bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end); + +struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end); + +/** + * drm_gpusvm_notifier_lock - Lock GPU SVM notifier + * @gpusvm__: Pointer to the GPU SVM structure. + * + * Abstract client usage GPU SVM notifier lock, take lock + */ +#define drm_gpusvm_notifier_lock(gpusvm__) \ + down_read(&(gpusvm__)->notifier_lock) + +/** + * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier + * @gpusvm__: Pointer to the GPU SVM structure. + * + * Abstract client usage GPU SVM notifier lock, drop lock + */ +#define drm_gpusvm_notifier_unlock(gpusvm__) \ + up_read(&(gpusvm__)->notifier_lock) + +/** + * __drm_gpusvm_range_next - Get the next GPU SVM range in the list + * @range: a pointer to the current GPU SVM range + * + * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the + * current range is the last one or if the input range is NULL. + */ +static inline struct drm_gpusvm_range * +__drm_gpusvm_range_next(struct drm_gpusvm_range *range) +{ + if (range && !list_is_last(&range->rb.entry, + &range->notifier->range_list)) + return list_next_entry(range, rb.entry); + + return NULL; +} + +/** + * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier + * @range__: Iterator variable for the ranges. If set, it indicates the start of + * the iterator. If NULL, call drm_gpusvm_range_find() to get the range. + * @notifier__: Pointer to the GPU SVM notifier + * @start__: Start address of the range + * @end__: End address of the range + * + * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe + * to use while holding the driver SVM lock or the notifier lock. + */ +#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__) \ + for ((range__) = (range__) ?: \ + drm_gpusvm_range_find((notifier__), (start__), (end__)); \ + (range__) && (range__->va.start < (end__)); \ + (range__) = __drm_gpusvm_range_next(range__)) + +/** + * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped + * @range: Pointer to the GPU SVM range structure. + * @mmu_range: Pointer to the MMU notifier range structure. + * + * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag + * if the range partially falls within the provided MMU notifier range. + */ +static inline void +drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range, + const struct mmu_notifier_range *mmu_range) +{ + lockdep_assert_held_write(&range->gpusvm->notifier_lock); + + range->flags.unmapped = true; + if (range->va.start < mmu_range->start || + range->va.end > mmu_range->end) + range->flags.partial_unmap = true; +} + +#endif /* __DRM_GPUSVM_H__ */ From patchwork Wed Aug 28 02:48:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19ABDC54755 for ; Wed, 28 Aug 2024 02:48:24 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DEA1110E45C; Wed, 28 Aug 2024 02:48:12 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="chcGZsmc"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8ED4F10E442; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tjOgg2Li/uaVxZ7/r4oELUZUwCKuKabi8wWQZFYGD5I=; b=chcGZsmcoNcv9CY6QoHLZb5kKWI2QgRWTgNtAVYSMX1DfBHLMJIKCunE fPUe/eciuobFT40yxISs6b7eUlfFXYzNh1IU4pNFVMYHAsiNHSYq+want RTSfoaqh4rlGDBVT4VS1ENFGqu2nJDPJ68CMWQAG7uH0MNrqCJILMnDJA I6Mr13KSNkXu+azknfCF4lAAFcxtR0ykeScLIJ8YpNgkLtrrATJm3KdrP 92vFKxkGLWyvt/gB0+gL7a1fpQ2iUNYtdQZTC2f1Rt5S7kVuHQyU+AM3e 88clvbgYcX4nhfbbjLIEDvxZjG1w7Tb4Xc1R9WUF+wTABJt63cCKHFQIC w==; X-CSE-ConnectionGUID: 9+ixeUAKRkSk073LwLfj1Q== X-CSE-MsgGUID: XqFqMusQTT2UiHjLGmnwcQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251871" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251871" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: dw7IM5JnR1CHu5vOZRyocA== X-CSE-MsgGUID: Zjh+Qz4/QF+RDQ0vJ72+zg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224601" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 06/28] drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATON flag Date: Tue, 27 Aug 2024 19:48:39 -0700 Message-Id: <20240828024901.2582335-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add the DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag, which is used to create unpopulated virtual memory areas (VMAs) without memory backing or GPU page tables. These VMAs are referred to as system allocator VMAs. The idea is that upon a page fault or prefetch, the memory backing and GPU page tables will be populated. System allocator VMAs only update GPUVM state; they do not have an internal page table (PT) state, nor do they have GPU mappings. It is expected that system allocator VMAs will be mixed with buffer object (BO) VMAs within a single VM. In other words, system allocations and runtime allocations can be mixed within a single user-mode driver (UMD) program. Expected usage: - Bind the entire virtual address (VA) space upon program load using the DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag. - If a buffer object (BO) requires GPU mapping, allocate an address using malloc, and bind the BO to the malloc'd address using existing bind IOCTLs (runtime allocation). - If a BO no longer requires GPU mapping, bind the mapping address with the DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag. - Any malloc'd address accessed by the GPU will be faulted in via the SVM implementation (system allocation). - Upon freeing any malloc'd data, the SVM implementation will remove GPU mappings. Only supporting 1 to 1 mapping between user address space and GPU address space at the moment as that is the expected use case. uAPI defines interface for non 1 to 1 but enforces 1 to 1, this restriction can be lifted if use cases arrise for non 1 to 1 mappings. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 76 +++++++++++++++++----- drivers/gpu/drm/xe/xe_vm.c | 107 ++++++++++++++++++++----------- drivers/gpu/drm/xe/xe_vm.h | 8 ++- drivers/gpu/drm/xe/xe_vm_types.h | 3 + include/uapi/drm/xe_drm.h | 19 +++++- 5 files changed, 157 insertions(+), 56 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index d6353e8969f0..d21e45efeaab 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -1068,6 +1068,11 @@ static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op, { int err = 0; + /* + * No need to check for is_system_allocator here as vma_add_deps is a + * NOP if VMA is_system_allocator + */ + switch (op->base.op) { case DRM_GPUVA_OP_MAP: if (!op->map.immediate && xe_vm_in_fault_mode(vm)) @@ -1646,6 +1651,7 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile, struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; int err; + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); xe_bo_assert_held(xe_vma_bo(vma)); vm_dbg(&xe_vma_vm(vma)->xe->drm, @@ -1713,6 +1719,7 @@ static int unbind_op_prepare(struct xe_tile *tile, if (!((vma->tile_present | vma->tile_staged) & BIT(tile->id))) return 0; + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); xe_bo_assert_held(xe_vma_bo(vma)); vm_dbg(&xe_vma_vm(vma)->xe->drm, @@ -1759,15 +1766,21 @@ static int op_prepare(struct xe_vm *vm, switch (op->base.op) { case DRM_GPUVA_OP_MAP: - if (!op->map.immediate && xe_vm_in_fault_mode(vm)) + if ((!op->map.immediate && xe_vm_in_fault_mode(vm)) || + op->map.is_system_allocator) break; err = bind_op_prepare(vm, tile, pt_update_ops, op->map.vma); pt_update_ops->wait_vm_kernel = true; break; case DRM_GPUVA_OP_REMAP: - err = unbind_op_prepare(tile, pt_update_ops, - gpuva_to_vma(op->base.remap.unmap->va)); + { + struct xe_vma *old = gpuva_to_vma(op->base.remap.unmap->va); + + if (xe_vma_is_system_allocator(old)) + break; + + err = unbind_op_prepare(tile, pt_update_ops, old); if (!err && op->remap.prev) { err = bind_op_prepare(vm, tile, pt_update_ops, @@ -1780,15 +1793,28 @@ static int op_prepare(struct xe_vm *vm, pt_update_ops->wait_vm_bookkeep = true; } break; + } case DRM_GPUVA_OP_UNMAP: - err = unbind_op_prepare(tile, pt_update_ops, - gpuva_to_vma(op->base.unmap.va)); + { + struct xe_vma *vma = gpuva_to_vma(op->base.unmap.va); + + if (xe_vma_is_system_allocator(vma)) + break; + + err = unbind_op_prepare(tile, pt_update_ops, vma); break; + } case DRM_GPUVA_OP_PREFETCH: - err = bind_op_prepare(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.prefetch.va)); + { + struct xe_vma *vma = gpuva_to_vma(op->base.prefetch.va); + + if (xe_vma_is_system_allocator(vma)) + break; + + err = bind_op_prepare(vm, tile, pt_update_ops, vma); pt_update_ops->wait_vm_kernel = true; break; + } default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } @@ -1857,6 +1883,8 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile, struct xe_vma *vma, struct dma_fence *fence, struct dma_fence *fence2) { + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) { dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ? @@ -1890,6 +1918,8 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile, struct xe_vma *vma, struct dma_fence *fence, struct dma_fence *fence2) { + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) { dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ? @@ -1924,16 +1954,21 @@ static void op_commit(struct xe_vm *vm, switch (op->base.op) { case DRM_GPUVA_OP_MAP: - if (!op->map.immediate && xe_vm_in_fault_mode(vm)) + if ((!op->map.immediate && xe_vm_in_fault_mode(vm)) || + op->map.is_system_allocator) break; bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence, fence2); break; case DRM_GPUVA_OP_REMAP: - unbind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.remap.unmap->va), fence, - fence2); + { + struct xe_vma *old = gpuva_to_vma(op->base.remap.unmap->va); + + if (xe_vma_is_system_allocator(old)) + break; + + unbind_op_commit(vm, tile, pt_update_ops, old, fence, fence2); if (op->remap.prev) bind_op_commit(vm, tile, pt_update_ops, op->remap.prev, @@ -1942,14 +1977,25 @@ static void op_commit(struct xe_vm *vm, bind_op_commit(vm, tile, pt_update_ops, op->remap.next, fence, fence2); break; + } case DRM_GPUVA_OP_UNMAP: - unbind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.unmap.va), fence, fence2); + { + struct xe_vma *vma = gpuva_to_vma(op->base.unmap.va); + + if (!xe_vma_is_system_allocator(vma)) + unbind_op_commit(vm, tile, pt_update_ops, vma, fence, + fence2); break; + } case DRM_GPUVA_OP_PREFETCH: - bind_op_commit(vm, tile, pt_update_ops, - gpuva_to_vma(op->base.prefetch.va), fence, fence2); + { + struct xe_vma *vma = gpuva_to_vma(op->base.prefetch.va); + + if (!xe_vma_is_system_allocator(vma)) + bind_op_commit(vm, tile, pt_update_ops, vma, fence, + fence2); break; + } default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 4cc13eddb6b3..5ec160561662 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -901,9 +901,10 @@ static void xe_vma_free(struct xe_vma *vma) kfree(vma); } -#define VMA_CREATE_FLAG_READ_ONLY BIT(0) -#define VMA_CREATE_FLAG_IS_NULL BIT(1) -#define VMA_CREATE_FLAG_DUMPABLE BIT(2) +#define VMA_CREATE_FLAG_READ_ONLY BIT(0) +#define VMA_CREATE_FLAG_IS_NULL BIT(1) +#define VMA_CREATE_FLAG_DUMPABLE BIT(2) +#define VMA_CREATE_FLAG_IS_SYSTEM_ALLOCATOR BIT(3) static struct xe_vma *xe_vma_create(struct xe_vm *vm, struct xe_bo *bo, @@ -917,6 +918,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, bool read_only = (flags & VMA_CREATE_FLAG_READ_ONLY); bool is_null = (flags & VMA_CREATE_FLAG_IS_NULL); bool dumpable = (flags & VMA_CREATE_FLAG_DUMPABLE); + bool is_system_allocator = + (flags & VMA_CREATE_FLAG_IS_SYSTEM_ALLOCATOR); xe_assert(vm->xe, start < end); xe_assert(vm->xe, end < vm->size); @@ -925,7 +928,7 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, * Allocate and ensure that the xe_vma_is_userptr() return * matches what was allocated. */ - if (!bo && !is_null) { + if (!bo && !is_null && !is_system_allocator) { struct xe_userptr_vma *uvma = kzalloc(sizeof(*uvma), GFP_KERNEL); if (!uvma) @@ -937,6 +940,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, if (!vma) return ERR_PTR(-ENOMEM); + if (is_system_allocator) + vma->gpuva.flags |= XE_VMA_SYSTEM_ALLOCATOR; if (is_null) vma->gpuva.flags |= DRM_GPUVA_SPARSE; if (bo) @@ -979,7 +984,7 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, drm_gpuva_link(&vma->gpuva, vm_bo); drm_gpuvm_bo_put(vm_bo); } else /* userptr or null */ { - if (!is_null) { + if (!is_null && !is_system_allocator) { struct xe_userptr *userptr = &to_userptr_vma(vma)->userptr; u64 size = end - start + 1; int err; @@ -1029,7 +1034,7 @@ static void xe_vma_destroy_late(struct xe_vma *vma) */ mmu_interval_notifier_remove(&userptr->notifier); xe_vm_put(vm); - } else if (xe_vma_is_null(vma)) { + } else if (xe_vma_is_null(vma) || xe_vma_is_system_allocator(vma)) { xe_vm_put(vm); } else { xe_bo_put(xe_vma_bo(vma)); @@ -1068,7 +1073,7 @@ static void xe_vma_destroy(struct xe_vma *vma, struct dma_fence *fence) spin_lock(&vm->userptr.invalidated_lock); list_del(&to_userptr_vma(vma)->userptr.invalidate_link); spin_unlock(&vm->userptr.invalidated_lock); - } else if (!xe_vma_is_null(vma)) { + } else if (!xe_vma_is_null(vma) && !xe_vma_is_system_allocator(vma)) { xe_bo_assert_held(xe_vma_bo(vma)); drm_gpuva_unlink(&vma->gpuva); @@ -1971,6 +1976,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, op->map.read_only = flags & DRM_XE_VM_BIND_FLAG_READONLY; op->map.is_null = flags & DRM_XE_VM_BIND_FLAG_NULL; + op->map.is_system_allocator = flags & + DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR; op->map.dumpable = flags & DRM_XE_VM_BIND_FLAG_DUMPABLE; op->map.pat_index = pat_index; } else if (__op->op == DRM_GPUVA_OP_PREFETCH) { @@ -2162,6 +2169,8 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, VMA_CREATE_FLAG_IS_NULL : 0; flags |= op->map.dumpable ? VMA_CREATE_FLAG_DUMPABLE : 0; + flags |= op->map.is_system_allocator ? + VMA_CREATE_FLAG_IS_SYSTEM_ALLOCATOR : 0; vma = new_vma(vm, &op->base.map, op->map.pat_index, flags); @@ -2169,7 +2178,8 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, return PTR_ERR(vma); op->map.vma = vma; - if (op->map.immediate || !xe_vm_in_fault_mode(vm)) + if ((op->map.immediate || !xe_vm_in_fault_mode(vm)) && + !op->map.is_system_allocator) xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); break; @@ -2178,21 +2188,24 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, { struct xe_vma *old = gpuva_to_vma(op->base.remap.unmap->va); + bool skip = xe_vma_is_system_allocator(old); op->remap.start = xe_vma_start(old); op->remap.range = xe_vma_size(old); - if (op->base.remap.prev) { - flags |= op->base.remap.unmap->va->flags & - XE_VMA_READ_ONLY ? - VMA_CREATE_FLAG_READ_ONLY : 0; - flags |= op->base.remap.unmap->va->flags & - DRM_GPUVA_SPARSE ? - VMA_CREATE_FLAG_IS_NULL : 0; - flags |= op->base.remap.unmap->va->flags & - XE_VMA_DUMPABLE ? - VMA_CREATE_FLAG_DUMPABLE : 0; + flags |= op->base.remap.unmap->va->flags & + XE_VMA_READ_ONLY ? + VMA_CREATE_FLAG_READ_ONLY : 0; + flags |= op->base.remap.unmap->va->flags & + DRM_GPUVA_SPARSE ? + VMA_CREATE_FLAG_IS_NULL : 0; + flags |= op->base.remap.unmap->va->flags & + XE_VMA_DUMPABLE ? + VMA_CREATE_FLAG_DUMPABLE : 0; + flags |= xe_vma_is_system_allocator(old) ? + VMA_CREATE_FLAG_IS_SYSTEM_ALLOCATOR : 0; + if (op->base.remap.prev) { vma = new_vma(vm, op->base.remap.prev, old->pat_index, flags); if (IS_ERR(vma)) @@ -2204,9 +2217,10 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, * Userptr creates a new SG mapping so * we must also rebind. */ - op->remap.skip_prev = !xe_vma_is_userptr(old) && + op->remap.skip_prev = skip || + (!xe_vma_is_userptr(old) && IS_ALIGNED(xe_vma_end(vma), - xe_vma_max_pte_size(old)); + xe_vma_max_pte_size(old))); if (op->remap.skip_prev) { xe_vma_set_pte_size(vma, xe_vma_max_pte_size(old)); op->remap.range -= @@ -2222,16 +2236,6 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, } if (op->base.remap.next) { - flags |= op->base.remap.unmap->va->flags & - XE_VMA_READ_ONLY ? - VMA_CREATE_FLAG_READ_ONLY : 0; - flags |= op->base.remap.unmap->va->flags & - DRM_GPUVA_SPARSE ? - VMA_CREATE_FLAG_IS_NULL : 0; - flags |= op->base.remap.unmap->va->flags & - XE_VMA_DUMPABLE ? - VMA_CREATE_FLAG_DUMPABLE : 0; - vma = new_vma(vm, op->base.remap.next, old->pat_index, flags); if (IS_ERR(vma)) @@ -2243,9 +2247,10 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, * Userptr creates a new SG mapping so * we must also rebind. */ - op->remap.skip_next = !xe_vma_is_userptr(old) && + op->remap.skip_next = skip || + (!xe_vma_is_userptr(old) && IS_ALIGNED(xe_vma_start(vma), - xe_vma_max_pte_size(old)); + xe_vma_max_pte_size(old))); if (op->remap.skip_next) { xe_vma_set_pte_size(vma, xe_vma_max_pte_size(old)); op->remap.range -= @@ -2258,14 +2263,27 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); } } - xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); + if (!skip) + xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); break; } case DRM_GPUVA_OP_UNMAP: + { + struct xe_vma *vma = gpuva_to_vma(op->base.unmap.va); + + if (!xe_vma_is_system_allocator(vma)) + xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); + break; + } case DRM_GPUVA_OP_PREFETCH: + { + struct xe_vma *vma = gpuva_to_vma(op->base.prefetch.va); + /* FIXME: Need to skip some prefetch ops */ - xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); + if (!xe_vma_is_system_allocator(vma)) + xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); break; + } default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } @@ -2706,7 +2724,8 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm, (DRM_XE_VM_BIND_FLAG_READONLY | \ DRM_XE_VM_BIND_FLAG_IMMEDIATE | \ DRM_XE_VM_BIND_FLAG_NULL | \ - DRM_XE_VM_BIND_FLAG_DUMPABLE) + DRM_XE_VM_BIND_FLAG_DUMPABLE | \ + DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR) #ifdef TEST_VM_OPS_ERROR #define SUPPORTED_FLAGS (SUPPORTED_FLAGS_STUB | FORCE_OP_ERROR) @@ -2761,9 +2780,17 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe, u64 obj_offset = (*bind_ops)[i].obj_offset; u32 prefetch_region = (*bind_ops)[i].prefetch_mem_region_instance; bool is_null = flags & DRM_XE_VM_BIND_FLAG_NULL; + bool is_system_allocator = flags & + DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR; u16 pat_index = (*bind_ops)[i].pat_index; u16 coh_mode; + /* FIXME: Disabling system allocator for now */ + if (XE_IOCTL_DBG(xe, is_system_allocator)) { + err = -EOPNOTSUPP; + goto free_bind_ops; + } + if (XE_IOCTL_DBG(xe, pat_index >= xe->pat.n_entries)) { err = -EINVAL; goto free_bind_ops; @@ -2784,13 +2811,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe, if (XE_IOCTL_DBG(xe, op > DRM_XE_VM_BIND_OP_PREFETCH) || XE_IOCTL_DBG(xe, flags & ~SUPPORTED_FLAGS) || - XE_IOCTL_DBG(xe, obj && is_null) || - XE_IOCTL_DBG(xe, obj_offset && is_null) || + XE_IOCTL_DBG(xe, obj && (is_null || is_system_allocator)) || + XE_IOCTL_DBG(xe, obj_offset && (is_null || + is_system_allocator)) || XE_IOCTL_DBG(xe, op != DRM_XE_VM_BIND_OP_MAP && - is_null) || + (is_null || is_system_allocator)) || XE_IOCTL_DBG(xe, !obj && op == DRM_XE_VM_BIND_OP_MAP && - !is_null) || + !is_null && !is_system_allocator) || XE_IOCTL_DBG(xe, !obj && op == DRM_XE_VM_BIND_OP_UNMAP_ALL) || XE_IOCTL_DBG(xe, addr && @@ -3165,6 +3193,7 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) int ret = 0; xe_assert(xe, !xe_vma_is_null(vma)); + xe_assert(xe, !xe_vma_is_system_allocator(vma)); trace_xe_vma_invalidate(vma); vm_dbg(&xe_vma_vm(vma)->xe->drm, diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index c864dba35e1d..1a5aed678214 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -151,6 +151,11 @@ static inline bool xe_vma_is_null(struct xe_vma *vma) return vma->gpuva.flags & DRM_GPUVA_SPARSE; } +static inline bool xe_vma_is_system_allocator(struct xe_vma *vma) +{ + return vma->gpuva.flags & XE_VMA_SYSTEM_ALLOCATOR; +} + static inline bool xe_vma_has_no_bo(struct xe_vma *vma) { return !xe_vma_bo(vma); @@ -158,7 +163,8 @@ static inline bool xe_vma_has_no_bo(struct xe_vma *vma) static inline bool xe_vma_is_userptr(struct xe_vma *vma) { - return xe_vma_has_no_bo(vma) && !xe_vma_is_null(vma); + return xe_vma_has_no_bo(vma) && !xe_vma_is_null(vma) && + !xe_vma_is_system_allocator(vma); } /** diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index 7f9a303e51d8..1764781c376b 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -42,6 +42,7 @@ struct xe_vm_pgtable_update_op; #define XE_VMA_PTE_64K (DRM_GPUVA_USERBITS << 6) #define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 7) #define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 8) +#define XE_VMA_SYSTEM_ALLOCATOR (DRM_GPUVA_USERBITS << 9) /** struct xe_userptr - User pointer */ struct xe_userptr { @@ -294,6 +295,8 @@ struct xe_vma_op_map { bool read_only; /** @is_null: is NULL binding */ bool is_null; + /** @is_system_allocator: is system allocator binding */ + bool is_system_allocator; /** @dumpable: whether BO is dumped on GPU hang */ bool dumpable; /** @pat_index: The pat index to use for this operation. */ diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h index b6fbe4988f2e..27003777cb62 100644 --- a/include/uapi/drm/xe_drm.h +++ b/include/uapi/drm/xe_drm.h @@ -904,6 +904,12 @@ struct drm_xe_vm_destroy { * will only be valid for DRM_XE_VM_BIND_OP_MAP operations, the BO * handle MBZ, and the BO offset MBZ. This flag is intended to * implement VK sparse bindings. + * - %DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR - When the system allocator flag is + * set, no mappings are created rather the range is reserved for system + * allocations which will be populated on GPU page faults. Only valid on VMs + * with DRM_XE_VM_CREATE_FLAG_FAULT_MODE set. The system allocator flag are + * only valid for DRM_XE_VM_BIND_OP_MAP operations, the BO handle MBZ, and + * the BO offset MBZ. */ struct drm_xe_vm_bind_op { /** @extensions: Pointer to the first extension struct, if any */ @@ -956,7 +962,9 @@ struct drm_xe_vm_bind_op { * on the @pat_index. For such mappings there is no actual memory being * mapped (the address in the PTE is invalid), so the various PAT memory * attributes likely do not apply. Simply leaving as zero is one - * option (still a valid pat_index). + * option (still a valid pat_index). Same applies to + * DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR bindings as for such mapping + * there is no actual memory being mapped. */ __u16 pat_index; @@ -972,6 +980,14 @@ struct drm_xe_vm_bind_op { /** @userptr: user pointer to bind on */ __u64 userptr; + + /** + * @system_allocator_offset: Offset from GPU @addr to create + * system allocator mappings. MBZ with current level of support + * (e.g. 1 to 1 mapping between GPU and CPU mappings only + * supported). + */ + __s64 system_allocator_offset; }; /** @@ -994,6 +1010,7 @@ struct drm_xe_vm_bind_op { #define DRM_XE_VM_BIND_FLAG_IMMEDIATE (1 << 1) #define DRM_XE_VM_BIND_FLAG_NULL (1 << 2) #define DRM_XE_VM_BIND_FLAG_DUMPABLE (1 << 3) +#define DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR (1 << 4) /** @flags: Bind flags */ __u32 flags; From patchwork Wed Aug 28 02:48:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7798C5474E for ; Wed, 28 Aug 2024 02:48:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B97C110E465; Wed, 28 Aug 2024 02:48:13 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Rv7Pa/qu"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 62AA110E43F; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813288; x=1756349288; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lRe+mxjUJ3daY2t9QkGYxv3ZPXLMMbSxj4sVAJUhbOQ=; b=Rv7Pa/qu408wCsvttLwBaHlizmush3aPa4t8Ce2HV6HrUGT9+yaFyZa/ HCER6PvYx5I7V6tLaP8f7OeWDe5mYOy2ze+9W8O4/s29CJJVsV+VyILVx 8nc23W20nyd5XJp+35VL2CSGsgy6QbcesDSgLQtKYbd4poCwNQ3OcYTzJ Eql0LCmiocsEKJlDMdjXtnt6cjaDBmQaNsgxjgSHpOwSf41AVvRi5kYcU PGeE2Iym+IwNl4+DtqrwFUd3uQXRaVGq+obFKTexObBimcahqJBitBcvS gzNZwGrsNOFLHjbA8AbS1bwB86gqS2VhEHnRITKjNnEnyuSd3dcc8eOfx A==; X-CSE-ConnectionGUID: qTgV7ezVRdKwstFR2VPVig== X-CSE-MsgGUID: rBO25tboSPurXL6K7i/nGw== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251876" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251876" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: ER9EcjSNRdSpxdYOqv/CYA== X-CSE-MsgGUID: 7nmdmOKPTyW9MKSRXuWkpQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224605" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:07 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 07/28] drm/xe: Add SVM init / fini to faulting VMs Date: Tue, 27 Aug 2024 19:48:40 -0700 Message-Id: <20240828024901.2582335-8-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add SVM init / fini to faulting VMs. Minimual implementation. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.c | 40 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 14 +++++++++++ drivers/gpu/drm/xe/xe_vm.c | 10 ++++++++ drivers/gpu/drm/xe/xe_vm_types.h | 7 ++++++ 5 files changed, 72 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.h diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile index b8fc2ee58f1a..17bd7cfc9a62 100644 --- a/drivers/gpu/drm/xe/Makefile +++ b/drivers/gpu/drm/xe/Makefile @@ -94,6 +94,7 @@ xe-y += drm_gpusvm.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm.o \ xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \ diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c new file mode 100644 index 000000000000..7166100e3298 --- /dev/null +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2024 Intel Corporation + */ + +#include "drm_gpusvm.h" + +#include "xe_svm.h" +#include "xe_vm.h" +#include "xe_vm_types.h" + +static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range *mmu_range) +{ + /* TODO: Implement */ +} + +static const struct drm_gpusvm_ops gpusvm_ops = { + .invalidate = xe_svm_invalidate, +}; + +static const u64 fault_chunk_sizes[] = { + SZ_2M, + SZ_64K, + SZ_4K, +}; + +int xe_svm_init(struct xe_vm *vm) +{ + return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM", &vm->xe->drm, + current->mm, NULL, 0, vm->size, + SZ_512M, &gpusvm_ops, fault_chunk_sizes, + ARRAY_SIZE(fault_chunk_sizes)); +} + +void xe_svm_fini(struct xe_vm *vm) +{ + drm_gpusvm_fini(&vm->svm.gpusvm); +} diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h new file mode 100644 index 000000000000..4982d9168095 --- /dev/null +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 Intel Corporation + */ + +#ifndef _XE_SVM_H_ +#define _XE_SVM_H_ + +struct xe_vm; + +int xe_svm_init(struct xe_vm *vm); +void xe_svm_fini(struct xe_vm *vm); + +#endif diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 5ec160561662..17ad6a533b2f 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -35,6 +35,7 @@ #include "xe_preempt_fence.h" #include "xe_pt.h" #include "xe_res_cursor.h" +#include "xe_svm.h" #include "xe_sync.h" #include "xe_trace_bo.h" #include "xe_wa.h" @@ -1503,6 +1504,12 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags) } } + if (flags & XE_VM_FLAG_FAULT_MODE) { + err = xe_svm_init(vm); + if (err) + goto err_close; + } + if (number_tiles > 1) vm->composite_fence_ctx = dma_fence_context_alloc(1); @@ -1616,6 +1623,9 @@ void xe_vm_close_and_put(struct xe_vm *vm) xe_vma_destroy_unlocked(vma); } + if (xe_vm_in_fault_mode(vm)) + xe_svm_fini(vm); + up_write(&vm->lock); mutex_lock(&xe->usm.lock); diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index 1764781c376b..bd1c0e368238 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -6,6 +6,7 @@ #ifndef _XE_VM_TYPES_H_ #define _XE_VM_TYPES_H_ +#include "drm_gpusvm.h" #include #include @@ -140,6 +141,12 @@ struct xe_vm { /** @gpuvm: base GPUVM used to track VMAs */ struct drm_gpuvm gpuvm; + /** @svm: Shared virtual memory state */ + struct { + /** @svm.gpusvm: base GPUSVM used to track fault allocations */ + struct drm_gpusvm gpusvm; + } svm; + struct xe_device *xe; /* exec queue used for (un)binding vma's */ From patchwork Wed Aug 28 02:48:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 464BCC54751 for ; Wed, 28 Aug 2024 02:48:18 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B656610E450; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="VEVscTMG"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8E64C10E440; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6srlCMSGxAXdO4Bj6bo4VXVuLaCivCvijEQZeZRRPp8=; b=VEVscTMGpGKMuGUqLBReFtG7XX4XI6Kmj31Td2LKNtVs2kh+60EsPQoK Q5QsQ0Q3vSVhMpQ1PSBrCC6Ezqo9crtmz2QHcuiHKQJw1srzWEeV2Ek81 2bP32AqjHa88XyK26o4YfV7+1ep65+rp7wgzxtBe7hgwa0Jv9Q8DMk9zi 8qnOkLaMeaIujR0h2IW4zVtxI04ZCQLzPNGKXfv2X3AJ3233PoKZfuYLe 0SXmX6pypcPbYM7DlMpt2PWXsf27ZH7au0u5hfsWKxCDGrYgvIXBiEj+q 0t+SerASsjShVMv0KLGnDXQiaaHEJt6fZP6v1+a4gJhQWDPcrQ3oD7BzD w==; X-CSE-ConnectionGUID: z91zPE0fQquVocjAcnr2/w== X-CSE-MsgGUID: dz3r/koaRmSY2sDjAx6wuQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251880" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251880" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: ZJ7reUp2RjGYp+zcpdiWWw== X-CSE-MsgGUID: OhfE2StHSn+hj6OR4fEYNA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224608" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 08/28] drm/xe: Add dma_addr res cursor Date: Tue, 27 Aug 2024 19:48:41 -0700 Message-Id: <20240828024901.2582335-9-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Useful for SVM ranges in SRAM and programing page tables. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_res_cursor.h | 50 +++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h index dca374b6521c..3df630af9252 100644 --- a/drivers/gpu/drm/xe/xe_res_cursor.h +++ b/drivers/gpu/drm/xe/xe_res_cursor.h @@ -43,7 +43,9 @@ struct xe_res_cursor { u64 remaining; void *node; u32 mem_type; + unsigned int order; struct scatterlist *sgl; + const dma_addr_t *dma_addr; struct drm_buddy *mm; }; @@ -70,6 +72,7 @@ static inline void xe_res_first(struct ttm_resource *res, struct xe_res_cursor *cur) { cur->sgl = NULL; + cur->dma_addr = NULL; if (!res) goto fallback; @@ -160,11 +163,43 @@ static inline void xe_res_first_sg(const struct sg_table *sg, cur->start = start; cur->remaining = size; cur->size = 0; + cur->dma_addr = NULL; cur->sgl = sg->sgl; cur->mem_type = XE_PL_TT; __xe_res_sg_next(cur); } +/** + * xe_res_first_dma - initialize a xe_res_cursor with dma_addr array + * + * @dma_addr: dma_addr array to walk + * @start: Start of the range + * @size: Size of the range + * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size + * @cur: cursor object to initialize + * + * Start walking over the range of allocations between @start and @size. + */ +static inline void xe_res_first_dma(const dma_addr_t *dma_addr, + u64 start, u64 size, + unsigned int order, + struct xe_res_cursor *cur) +{ + XE_WARN_ON(start); + XE_WARN_ON(!dma_addr); + XE_WARN_ON(!IS_ALIGNED(start, PAGE_SIZE) || + !IS_ALIGNED(size, PAGE_SIZE)); + + cur->node = NULL; + cur->start = start; + cur->remaining = size; + cur->size = PAGE_SIZE << order; + cur->dma_addr = dma_addr; + cur->order = order; + cur->sgl = NULL; + cur->mem_type = XE_PL_TT; +} + /** * xe_res_next - advance the cursor * @@ -191,6 +226,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size) return; } + if (cur->dma_addr) { + cur->size = (PAGE_SIZE << cur->order) - + (size - cur->size); + cur->start += size; + return; + } + if (cur->sgl) { cur->start += size; __xe_res_sg_next(cur); @@ -232,6 +274,12 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size) */ static inline u64 xe_res_dma(const struct xe_res_cursor *cur) { - return cur->sgl ? sg_dma_address(cur->sgl) + cur->start : cur->start; + if (cur->dma_addr) + return cur->dma_addr[cur->start >> (PAGE_SHIFT + cur->order)] + + (cur->start & ((PAGE_SIZE << cur->order) - 1)); + else if (cur->sgl) + return sg_dma_address(cur->sgl) + cur->start; + else + return cur->start; } #endif From patchwork Wed Aug 28 02:48:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A52E7C5474C for ; Wed, 28 Aug 2024 02:48:29 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1C0D610E466; Wed, 28 Aug 2024 02:48:14 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="a1WwF0xg"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id B8F0310E43F; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q8fOjpTfHZ6g2EmIgk1IyKTPeb6OEjoIC8MZfXwxOwk=; b=a1WwF0xgdg8oTFgZz1ZwdnbSdav+t1w1fcFbZDYSG0BfduWoQguQ30Qq fEvrc/mw7IOjgHj7mv7Km9rwwjGfzO6xIXII7TpNj+d+njRE261JtWYqC 5GAjDWTGa4N4653WIX+FbU8O9hTnWH1xvww9kht6wTKD67FTWQ1P/OHsg +GElZ1PZ6uT5JojWp40VjNJa4FOlexvPRo15T5TvXgGnp0hjUk3rEcT6k bM+JJbNLJ2i5YmHaJUqXwDxEtoBt5bsvluYNtTWy24+FftmJWF5CACjDP fo2EtNBmGl9JvdC2H7L7hRxBkhlXsfoPDQcDzaI6ZG6G5KD0WYSpGMg+X g==; X-CSE-ConnectionGUID: U1sugrWCQlyOynVWeYqTSA== X-CSE-MsgGUID: WyXd12VZQAabA2nRrzV4BQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251884" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251884" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: Yso5XPJrSc6vqHZGD6aLWw== X-CSE-MsgGUID: PO/hx82wQd+QT4/iu0uioA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224611" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 09/28] drm/xe: Add SVM range invalidation Date: Tue, 27 Aug 2024 19:48:42 -0700 Message-Id: <20240828024901.2582335-10-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add SVM range invalidation vfunc. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_gt_pagefault.c | 17 ++- drivers/gpu/drm/xe/xe_pt.c | 24 ++++ drivers/gpu/drm/xe/xe_pt.h | 3 + drivers/gpu/drm/xe/xe_svm.c | 201 ++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_svm.h | 14 ++ 5 files changed, 253 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_gt_pagefault.c b/drivers/gpu/drm/xe/xe_gt_pagefault.c index 0be4687bfc20..e1f32d782f65 100644 --- a/drivers/gpu/drm/xe/xe_gt_pagefault.c +++ b/drivers/gpu/drm/xe/xe_gt_pagefault.c @@ -19,6 +19,7 @@ #include "xe_guc.h" #include "xe_guc_ct.h" #include "xe_migrate.h" +#include "xe_svm.h" #include "xe_trace_bo.h" #include "xe_vm.h" @@ -125,18 +126,17 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma, return 0; } -static int handle_vma_pagefault(struct xe_tile *tile, struct pagefault *pf, - struct xe_vma *vma) +static int handle_vma_pagefault(struct xe_tile *tile, struct xe_vma *vma, + bool atomic) { struct xe_vm *vm = xe_vma_vm(vma); struct drm_exec exec; struct dma_fence *fence; ktime_t end = 0; int err; - bool atomic; + lockdep_assert_held_write(&vm->lock); trace_xe_vma_pagefault(vma); - atomic = access_is_atomic(pf->access_type); /* Check if VMA is valid */ if (vma_is_valid(tile, vma) && !atomic) @@ -192,6 +192,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) struct xe_vm *vm; struct xe_vma *vma = NULL; int err; + bool atomic; /* SW isn't expected to handle TRTT faults */ if (pf->trva_fault) @@ -218,7 +219,13 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) goto unlock_vm; } - err = handle_vma_pagefault(tile, pf, vma); + atomic = access_is_atomic(pf->access_type); + + if (xe_vma_is_system_allocator(vma)) + err = xe_svm_handle_pagefault(vm, vma, tile, + pf->page_addr, atomic); + else + err = handle_vma_pagefault(tile, vma, atomic); unlock_vm: if (!err) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index d21e45efeaab..b2db79251825 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -20,6 +20,7 @@ #include "xe_res_cursor.h" #include "xe_sched_job.h" #include "xe_sync.h" +#include "xe_svm.h" #include "xe_trace.h" #include "xe_ttm_stolen_mgr.h" #include "xe_vm.h" @@ -829,6 +830,29 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma) return xe_walk.needs_invalidate; } +bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm, + struct xe_svm_range *range) +{ + struct xe_pt_zap_ptes_walk xe_walk = { + .base = { + .ops = &xe_pt_zap_ptes_ops, + .shifts = xe_normal_pt_shifts, + .max_level = XE_PT_HIGHEST_LEVEL, + }, + .tile = tile, + }; + struct xe_pt *pt = vm->pt_root[tile->id]; + u8 pt_mask = (range->tile_present & ~range->tile_invalidated); + + if (!(pt_mask & BIT(tile->id))) + return false; + + (void)xe_pt_walk_shared(&pt->base, pt->level, range->base.va.start, + range->base.va.end, &xe_walk.base); + + return xe_walk.needs_invalidate; +} + static void xe_vm_populate_pgtable(struct xe_migrate_pt_update *pt_update, struct xe_tile *tile, struct iosys_map *map, void *data, diff --git a/drivers/gpu/drm/xe/xe_pt.h b/drivers/gpu/drm/xe/xe_pt.h index 9ab386431cad..5f333eeedf5c 100644 --- a/drivers/gpu/drm/xe/xe_pt.h +++ b/drivers/gpu/drm/xe/xe_pt.h @@ -13,6 +13,7 @@ struct dma_fence; struct xe_bo; struct xe_device; struct xe_exec_queue; +struct xe_svm_range; struct xe_sync_entry; struct xe_tile; struct xe_vm; @@ -42,5 +43,7 @@ void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops); void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops); bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma); +bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm, + struct xe_svm_range *range); #endif diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 7166100e3298..3ac84f9615e2 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -5,18 +5,189 @@ #include "drm_gpusvm.h" +#include "xe_gt_tlb_invalidation.h" +#include "xe_pt.h" #include "xe_svm.h" #include "xe_vm.h" #include "xe_vm_types.h" +static struct xe_vm *gpusvm_to_vm(struct drm_gpusvm *gpusvm) + { + return container_of(gpusvm, struct xe_vm, svm.gpusvm); +} + +static struct xe_vm *range_to_vm(struct drm_gpusvm_range *r) +{ + return gpusvm_to_vm(r->gpusvm); +} + +static struct drm_gpusvm_range * +xe_svm_range_alloc(struct drm_gpusvm *gpusvm) +{ + struct xe_svm_range *range; + + range = kzalloc(sizeof(*range), GFP_KERNEL); + if (!range) + return ERR_PTR(-ENOMEM); + + xe_vm_get(gpusvm_to_vm(gpusvm)); + + return &range->base; +} + +static void xe_svm_range_free(struct drm_gpusvm_range *range) +{ + xe_vm_put(range_to_vm(range)); + kfree(range); +} + +static struct xe_svm_range *to_xe_range(struct drm_gpusvm_range *r) +{ + return container_of(r, struct xe_svm_range, base); +} + +static u8 +xe_svm_range_notifier_event_begin(struct xe_vm *vm, struct drm_gpusvm_range *r, + const struct mmu_notifier_range *mmu_range, + u64 *adj_start, u64 *adj_end) +{ + struct xe_svm_range *range = to_xe_range(r); + struct xe_device *xe = vm->xe; + struct xe_tile *tile; + u8 tile_mask = 0; + u8 id; + + /* Skip if already unmapped or if no binding exist */ + if (range->base.flags.unmapped || !range->tile_present) + return 0; + + /* Adjust invalidation to range boundaries */ + if (range->base.va.start < mmu_range->start) + *adj_start = range->base.va.start; + if (range->base.va.end > mmu_range->end) + *adj_end = range->base.va.end; + + /* + * XXX: Ideally would zap PTEs in one shot in xe_svm_invalidate but the + * invalidation code can't correctly cope with sparse ranges or + * invalidations spanning multiple ranges. + */ + for_each_tile(tile, xe, id) + if (xe_pt_zap_ptes_range(tile, vm, range)) { + tile_mask |= BIT(id); + range->tile_invalidated |= BIT(id); + } + + return tile_mask; +} + +static void +xe_svm_range_notifier_event_end(struct xe_vm *vm, struct drm_gpusvm_range *r, + const struct mmu_notifier_range *mmu_range) +{ + struct drm_gpusvm_ctx ctx = { .in_notifier = true, }; + + drm_gpusvm_range_unmap_pages(&vm->svm.gpusvm, r, &ctx); + /* TODO: Add range to garbage collector */ +} + static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, struct drm_gpusvm_notifier *notifier, const struct mmu_notifier_range *mmu_range) { - /* TODO: Implement */ + struct xe_vm *vm = gpusvm_to_vm(gpusvm); + struct xe_device *xe = vm->xe; + struct xe_tile *tile; + struct drm_gpusvm_range *r, *first; + struct xe_gt_tlb_invalidation_fence + fence[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE]; + u64 adj_start = mmu_range->start, adj_end = mmu_range->end; + u8 tile_mask = 0; + u8 id; + u32 fence_id = 0; + long err; + + /* Adjust invalidation to notifier boundaries */ + if (adj_start < notifier->interval.start) + adj_start = notifier->interval.start; + if (adj_end > notifier->interval.end) + adj_end = notifier->interval.end; + + first = drm_gpusvm_range_find(notifier, adj_start, adj_end); + if (!first) + return; + + /* + * XXX: Less than ideal to always wait on VM's resv slots if an + * invalidation is not required. Could walk range list twice to figure + * out if an invalidations is need, but also not ideal. Maybe a counter + * within the notifier, seems like that could work. + */ + err = dma_resv_wait_timeout(xe_vm_resv(vm), + DMA_RESV_USAGE_BOOKKEEP, + false, MAX_SCHEDULE_TIMEOUT); + XE_WARN_ON(err <= 0); + + r = first; + drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end) + tile_mask |= xe_svm_range_notifier_event_begin(vm, r, mmu_range, + &adj_start, + &adj_end); + if (!tile_mask) + goto range_notifier_event_end; + + xe_device_wmb(xe); + + for_each_tile(tile, xe, id) { + if (tile_mask & BIT(id)) { + int err; + + xe_gt_tlb_invalidation_fence_init(tile->primary_gt, + &fence[fence_id], true); + + err = xe_gt_tlb_invalidation_range(tile->primary_gt, + &fence[fence_id], + adj_start, + adj_end, + vm->usm.asid); + if (WARN_ON_ONCE(err < 0)) { + xe_gt_tlb_invalidation_fence_fini(&fence[fence_id]); + goto wait; + } + ++fence_id; + + if (!tile->media_gt) + continue; + + xe_gt_tlb_invalidation_fence_init(tile->media_gt, + &fence[fence_id], true); + + err = xe_gt_tlb_invalidation_range(tile->media_gt, + &fence[fence_id], + adj_start, + adj_end, + vm->usm.asid); + if (WARN_ON_ONCE(err < 0)) { + xe_gt_tlb_invalidation_fence_fini(&fence[fence_id]); + goto wait; + } + ++fence_id; + } + } + +wait: + for (id = 0; id < fence_id; ++id) + xe_gt_tlb_invalidation_fence_wait(&fence[id]); + +range_notifier_event_end: + r = first; + drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end) + xe_svm_range_notifier_event_end(vm, r, mmu_range); } static const struct drm_gpusvm_ops gpusvm_ops = { + .range_alloc = xe_svm_range_alloc, + .range_free = xe_svm_range_free, .invalidate = xe_svm_invalidate, }; @@ -38,3 +209,31 @@ void xe_svm_fini(struct xe_vm *vm) { drm_gpusvm_fini(&vm->svm.gpusvm); } + +int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, + struct xe_tile *tile, u64 fault_addr, + bool atomic) +{ + struct drm_gpusvm_ctx ctx = { .read_only = xe_vma_read_only(vma), }; + struct drm_gpusvm_range *r; + int err; + + lockdep_assert_held_write(&vm->lock); + +retry: + /* TODO: Run garbage collector */ + + r = drm_gpusvm_range_find_or_insert(&vm->svm.gpusvm, fault_addr, + xe_vma_start(vma), xe_vma_end(vma), + &ctx); + if (IS_ERR(r)) + return PTR_ERR(r); + + err = drm_gpusvm_range_get_pages(&vm->svm.gpusvm, r, false); + if (err == -EFAULT || err == -EPERM) /* Corner where CPU mappings have change */ + goto retry; + + /* TODO: Issue bind */ + + return err; +} diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 4982d9168095..b053b11692f0 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -6,9 +6,23 @@ #ifndef _XE_SVM_H_ #define _XE_SVM_H_ +#include "drm_gpusvm.h" + +struct xe_tile; struct xe_vm; +struct xe_vma; + +struct xe_svm_range { + struct drm_gpusvm_range base; + u8 tile_present; + u8 tile_invalidated; +}; int xe_svm_init(struct xe_vm *vm); void xe_svm_fini(struct xe_vm *vm); +int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, + struct xe_tile *tile, u64 fault_addr, + bool atomic); + #endif From patchwork Wed Aug 28 02:48:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05D93C54756 for ; Wed, 28 Aug 2024 02:48:26 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 828CD10E464; Wed, 28 Aug 2024 02:48:13 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="lTlCo86J"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id B961F10E441; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Vh+BnVKxl1twpfwV0qWGQ7BR4pA5oXHUShwq1A98TeI=; b=lTlCo86J1LkIZZr15nsVH0jXAv6hYqJAXl0SwCbupvDMR+oWc289oL/i ENEeQ9ZVxqssaLvgtMvD958bVJUyO5g1R+jGtcGjUtgS9JeqK0/JnCYVw /3MU3D98rMeZBKE+YyCH6jWToo+/YWVxJJHVBIkgHPQf1uXrI9zfYI88V k96dckWTLJWDjS3V21TM8DwGcgBoLopIE7chvxyGJWXSdxFD/CV24Y3qr F+2QJAEcKm4Ap5QfhBEo+ZDMZBuxS6geuSzZUXBPv+SENTkqVn3iqurAa //DUVl3JeaPIJCTFK5D+pvJUjeg7X2R2YnOp5D6Fl7LV1Oqc44hMN92+T w==; X-CSE-ConnectionGUID: VOOcArkHQAC6P6rZ4ACaXA== X-CSE-MsgGUID: FTSh9FP4TbyN++6h/jeeug== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251888" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251888" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: caqOARzOTwejXeHBZJuytw== X-CSE-MsgGUID: kvVXUk99Tie4s/oIuHYvCQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224614" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 10/28] drm/gpuvm: Add DRM_GPUVA_OP_USER Date: Tue, 27 Aug 2024 19:48:43 -0700 Message-Id: <20240828024901.2582335-11-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add DRM_GPUVA_OP_USER which allows driver to define their own gpuvm ops. Cc: Danilo Krummrich Signed-off-by: Matthew Brost --- include/drm/drm_gpuvm.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/drm/drm_gpuvm.h b/include/drm/drm_gpuvm.h index 00d4e43b76b6..cc3f8ed5113b 100644 --- a/include/drm/drm_gpuvm.h +++ b/include/drm/drm_gpuvm.h @@ -812,6 +812,11 @@ enum drm_gpuva_op_type { * @DRM_GPUVA_OP_PREFETCH: the prefetch op type */ DRM_GPUVA_OP_PREFETCH, + + /** + * @DRM_GPUVA_OP_USER: the user defined op type + */ + DRM_GPUVA_OP_USER, }; /** From patchwork Wed Aug 28 02:48:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BFA56C5474E for ; Wed, 28 Aug 2024 02:48:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3469F10E476; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="dP3uwT8B"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id DCF6110E442; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8Ax+Nyx+XQkurLCCVYv9e87Nj8kTFc57NDwddBspFj8=; b=dP3uwT8Br/0F1/G4Hm8kH2M2fmaWm1MrgxtDqIBtVfQn1he2UB0jCUZb RBUnESI0rnj6XT+q7uebYWCY/i59h9fX14bIGo2yxK4OTdPq8XmKdzZ1F 2f534pPIQ+JRiqd4XNu2BdrLYM3FV8QtL8DvQn9Lbc6pMfWrfs3vXrMsL 2Zvw0VdUUXOfK5X0grqYrGviV7+oc5f8muYlkCC78tjbXksPMQ08zmrTd w9+cD8HGmnCiYxFJVcfIdvgJV47gt02aTfxvyM0Ut8ax4VjdHNMZKEKP+ 74fvdriU6kHrb8FGx8XJIemNFmOaz9MW998F2sLS46NWtmLgcZccyjIy2 g==; X-CSE-ConnectionGUID: Lh6YIQKkShqwHEK8wk3jfg== X-CSE-MsgGUID: raVKfNvJTv6SA+ZFL0zjgw== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251892" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251892" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 X-CSE-ConnectionGUID: dziF1TnyScOT16b6rgRfcg== X-CSE-MsgGUID: jWDYsw4HRLKdfyLvRqwNJg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224618" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 11/28] drm/xe: Add (re)bind to SVM page fault handler Date: Tue, 27 Aug 2024 19:48:44 -0700 Message-Id: <20240828024901.2582335-12-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add (re)bind to SVM page fault handler. To facilitate add support function to VM layer which (re)binds a SVM range. Also teach PT layer to understand (re)binds of SVM ranges. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 140 +++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_pt_types.h | 2 + drivers/gpu/drm/xe/xe_svm.c | 45 +++++++++- drivers/gpu/drm/xe/xe_svm.h | 11 +++ drivers/gpu/drm/xe/xe_vm.c | 80 ++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 5 ++ drivers/gpu/drm/xe/xe_vm_types.h | 19 +++++ 7 files changed, 286 insertions(+), 16 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index b2db79251825..d5e444af7e02 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -587,6 +587,7 @@ static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { * range. * @tile: The tile we're building for. * @vma: The vma indicating the address range. + * @range: The range indicating the address range. * @entries: Storage for the update entries used for connecting the tree to * the main tree at commit time. * @num_entries: On output contains the number of @entries used. @@ -602,6 +603,7 @@ static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { */ static int xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, + struct xe_svm_range *range, struct xe_vm_pgtable_update *entries, u32 *num_entries) { struct xe_device *xe = tile_to_xe(tile); @@ -618,7 +620,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, .vm = xe_vma_vm(vma), .tile = tile, .curs = &curs, - .va_curs_start = xe_vma_start(vma), + .va_curs_start = range ? range->base.va.start : + xe_vma_start(vma), .vma = vma, .wupd.entries = entries, .needs_64K = (xe_vma_vm(vma)->flags & XE_VM_FLAG_64K) && is_devmem, @@ -671,7 +674,11 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, xe_bo_assert_held(bo); - if (!xe_vma_is_null(vma)) { + if (range) { + xe_res_first_dma(range->base.dma_addr, 0, + range->base.va.end - range->base.va.start, + range->base.order, &curs); + } else if (!xe_vma_is_null(vma)) { if (xe_vma_is_userptr(vma)) xe_res_first_sg(to_userptr_vma(vma)->userptr.sg, 0, xe_vma_size(vma), &curs); @@ -685,8 +692,10 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, curs.size = xe_vma_size(vma); } - ret = xe_pt_walk_range(&pt->base, pt->level, xe_vma_start(vma), - xe_vma_end(vma), &xe_walk.base); + ret = xe_pt_walk_range(&pt->base, pt->level, + range ? range->base.va.start : xe_vma_start(vma), + range ? range->base.va.end : xe_vma_end(vma), + &xe_walk.base); *num_entries = xe_walk.wupd.num_used_entries; return ret; @@ -902,7 +911,7 @@ static void xe_pt_commit_locks_assert(struct xe_vma *vma) lockdep_assert_held(&vm->lock); - if (!xe_vma_is_userptr(vma) && !xe_vma_is_null(vma)) + if (!xe_vma_has_no_bo(vma)) dma_resv_assert_held(xe_vma_bo(vma)->ttm.base.resv); xe_vm_assert_held(vm); @@ -1004,12 +1013,13 @@ static void xe_pt_free_bind(struct xe_vm_pgtable_update *entries, static int xe_pt_prepare_bind(struct xe_tile *tile, struct xe_vma *vma, + struct xe_svm_range *range, struct xe_vm_pgtable_update *entries, u32 *num_entries) { int err; *num_entries = 0; - err = xe_pt_stage_bind(tile, vma, entries, num_entries); + err = xe_pt_stage_bind(tile, vma, range, entries, num_entries); if (!err) xe_tile_assert(tile, *num_entries); @@ -1115,6 +1125,8 @@ static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op, case DRM_GPUVA_OP_PREFETCH: err = vma_add_deps(gpuva_to_vma(op->base.prefetch.va), job); break; + case DRM_GPUVA_OP_USER: + break; default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } @@ -1339,6 +1351,34 @@ static int xe_pt_userptr_pre_commit(struct xe_migrate_pt_update *pt_update) return err; } +static int xe_pt_svm_pre_commit(struct xe_migrate_pt_update *pt_update) +{ + struct xe_vm *vm = pt_update->vops->vm; + struct xe_vma_ops *vops = pt_update->vops; + struct xe_vma_op *op; + int err; + + err = xe_pt_pre_commit(pt_update); + if (err) + return err; + + xe_svm_notifier_lock(vm); + + list_for_each_entry(op, &vops->list, link) { + struct xe_svm_range *range = op->map_range.range; + + xe_assert(vm->xe, xe_vma_is_system_allocator(op->map_range.vma)); + xe_assert(vm->xe, op->subop == XE_VMA_SUBOP_MAP_RANGE); + + if (!xe_svm_range_pages_valid(range)) { + xe_svm_notifier_unlock(vm); + return -EAGAIN; + } + } + + return 0; +} + struct invalidation_fence { struct xe_gt_tlb_invalidation_fence base; struct xe_gt *gt; @@ -1632,12 +1672,12 @@ xe_pt_commit_prepare_unbind(struct xe_vma *vma, static void xe_pt_update_ops_rfence_interval(struct xe_vm_pgtable_update_ops *pt_update_ops, - struct xe_vma *vma) + u64 start, u64 end) { + u64 last; u32 current_op = pt_update_ops->current_op; struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; int i, level = 0; - u64 start, last; for (i = 0; i < pt_op->num_entries; i++) { const struct xe_vm_pgtable_update *entry = &pt_op->entries[i]; @@ -1647,8 +1687,8 @@ xe_pt_update_ops_rfence_interval(struct xe_vm_pgtable_update_ops *pt_update_ops, } /* Greedy (non-optimal) calculation but simple */ - start = ALIGN_DOWN(xe_vma_start(vma), 0x1ull << xe_pt_shift(level)); - last = ALIGN(xe_vma_end(vma), 0x1ull << xe_pt_shift(level)) - 1; + start = ALIGN_DOWN(start, 0x1ull << xe_pt_shift(level)); + last = ALIGN(end, 0x1ull << xe_pt_shift(level)) - 1; if (start < pt_update_ops->start) pt_update_ops->start = start; @@ -1690,7 +1730,7 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile, if (err) return err; - err = xe_pt_prepare_bind(tile, vma, pt_op->entries, + err = xe_pt_prepare_bind(tile, vma, NULL, pt_op->entries, &pt_op->num_entries); if (!err) { xe_tile_assert(tile, pt_op->num_entries <= @@ -1698,7 +1738,9 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile, xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries, pt_op->num_entries, true); - xe_pt_update_ops_rfence_interval(pt_update_ops, vma); + xe_pt_update_ops_rfence_interval(pt_update_ops, + xe_vma_start(vma), + xe_vma_end(vma)); ++pt_update_ops->current_op; pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma); @@ -1732,6 +1774,48 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile, return err; } +static int bind_range_prepare(struct xe_vm *vm, struct xe_tile *tile, + struct xe_vm_pgtable_update_ops *pt_update_ops, + struct xe_vma *vma, struct xe_svm_range *range) +{ + u32 current_op = pt_update_ops->current_op; + struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; + int err; + + xe_tile_assert(tile, xe_vma_is_system_allocator(vma)); + + vm_dbg(&xe_vma_vm(vma)->xe->drm, + "Preparing bind, with range [%llx...%llx)\n", + range->base.va.start, range->base.va.end - 1); + + pt_op->vma = NULL; + pt_op->bind = true; + pt_op->rebind = BIT(tile->id) & range->tile_present; + + err = xe_pt_prepare_bind(tile, vma, range, pt_op->entries, + &pt_op->num_entries); + if (!err) { + xe_tile_assert(tile, pt_op->num_entries <= + ARRAY_SIZE(pt_op->entries)); + xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries, + pt_op->num_entries, true); + + xe_pt_update_ops_rfence_interval(pt_update_ops, + range->base.va.start, + range->base.va.end); + ++pt_update_ops->current_op; + pt_update_ops->needs_svm_lock = true; + + pt_op->vma = vma; + xe_pt_commit_prepare_bind(vma, pt_op->entries, + pt_op->num_entries, pt_op->rebind); + } else { + xe_pt_cancel_bind(vma, pt_op->entries, pt_op->num_entries); + } + + return err; +} + static int unbind_op_prepare(struct xe_tile *tile, struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_vma *vma) @@ -1769,7 +1853,8 @@ static int unbind_op_prepare(struct xe_tile *tile, xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries, pt_op->num_entries, false); - xe_pt_update_ops_rfence_interval(pt_update_ops, vma); + xe_pt_update_ops_rfence_interval(pt_update_ops, xe_vma_start(vma), + xe_vma_end(vma)); ++pt_update_ops->current_op; pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma); pt_update_ops->needs_invalidation = true; @@ -1839,6 +1924,15 @@ static int op_prepare(struct xe_vm *vm, pt_update_ops->wait_vm_kernel = true; break; } + case DRM_GPUVA_OP_USER: + if (op->subop == XE_VMA_SUBOP_MAP_RANGE) { + xe_assert(vm->xe, xe_vma_is_system_allocator(op->map_range.vma)); + + err = bind_range_prepare(vm, tile, pt_update_ops, + op->map_range.vma, + op->map_range.range); + } + break; default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } @@ -2020,6 +2114,14 @@ static void op_commit(struct xe_vm *vm, fence2); break; } + case DRM_GPUVA_OP_USER: + { + if (op->subop == XE_VMA_SUBOP_MAP_RANGE) { + op->map_range.range->tile_present |= BIT(tile->id); + op->map_range.range->tile_invalidated &= ~BIT(tile->id); + } + break; + } default: drm_warn(&vm->xe->drm, "NOT POSSIBLE"); } @@ -2037,6 +2139,12 @@ static const struct xe_migrate_pt_update_ops userptr_migrate_ops = { .pre_commit = xe_pt_userptr_pre_commit, }; +static const struct xe_migrate_pt_update_ops svm_migrate_ops = { + .populate = xe_vm_populate_pgtable, + .clear = xe_migrate_clear_pgtable_callback, + .pre_commit = xe_pt_svm_pre_commit, +}; + /** * xe_pt_update_ops_run() - Run PT update operations * @tile: Tile of PT update operations @@ -2062,7 +2170,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) struct xe_vma_op *op; int err = 0, i; struct xe_migrate_pt_update update = { - .ops = pt_update_ops->needs_userptr_lock ? + .ops = pt_update_ops->needs_svm_lock ? + &svm_migrate_ops : + pt_update_ops->needs_userptr_lock ? &userptr_migrate_ops : &migrate_ops, .vops = vops, @@ -2183,6 +2293,8 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) &ifence->base.base, &mfence->base.base); } + if (pt_update_ops->needs_svm_lock) + xe_svm_notifier_unlock(vm); if (pt_update_ops->needs_userptr_lock) up_read(&vm->userptr.notifier_lock); diff --git a/drivers/gpu/drm/xe/xe_pt_types.h b/drivers/gpu/drm/xe/xe_pt_types.h index 384cc04de719..69eab6f37cfe 100644 --- a/drivers/gpu/drm/xe/xe_pt_types.h +++ b/drivers/gpu/drm/xe/xe_pt_types.h @@ -104,6 +104,8 @@ struct xe_vm_pgtable_update_ops { u32 num_ops; /** @current_op: current operations */ u32 current_op; + /** @needs_svm_lock: Needs SVM lock */ + bool needs_svm_lock; /** @needs_userptr_lock: Needs userptr lock */ bool needs_userptr_lock; /** @needs_invalidation: Needs invalidation */ diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 3ac84f9615e2..28d139b3dbb7 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -210,12 +210,22 @@ void xe_svm_fini(struct xe_vm *vm) drm_gpusvm_fini(&vm->svm.gpusvm); } +static bool xe_svm_range_is_valid(struct xe_svm_range *range, + struct xe_tile *tile) +{ + return (range->tile_present & ~range->tile_invalidated) & BIT(tile->id); +} + int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, struct xe_tile *tile, u64 fault_addr, bool atomic) { struct drm_gpusvm_ctx ctx = { .read_only = xe_vma_read_only(vma), }; + struct xe_svm_range *range; struct drm_gpusvm_range *r; + struct drm_exec exec; + struct dma_fence *fence; + ktime_t end = 0; int err; lockdep_assert_held_write(&vm->lock); @@ -229,11 +239,42 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, if (IS_ERR(r)) return PTR_ERR(r); - err = drm_gpusvm_range_get_pages(&vm->svm.gpusvm, r, false); + range = to_xe_range(r); + if (xe_svm_range_is_valid(range, tile)) + return 0; + + err = drm_gpusvm_range_get_pages(&vm->svm.gpusvm, r, &ctx); if (err == -EFAULT || err == -EPERM) /* Corner where CPU mappings have change */ goto retry; + if (err) + goto err_out; + +retry_bind: + drm_exec_init(&exec, 0, 0); + drm_exec_until_all_locked(&exec) { + err = drm_exec_lock_obj(&exec, vm->gpuvm.r_obj); + drm_exec_retry_on_contention(&exec); + if (err) { + drm_exec_fini(&exec); + goto err_out; + } + + fence = xe_vm_range_rebind(vm, vma, range, BIT(tile->id)); + if (IS_ERR(fence)) { + drm_exec_fini(&exec); + err = PTR_ERR(fence); + if (err == -EAGAIN) + goto retry; + if (xe_vm_validate_should_retry(&exec, err, &end)) + goto retry_bind; + goto err_out; + } + } + drm_exec_fini(&exec); - /* TODO: Issue bind */ + dma_fence_wait(fence, false); + dma_fence_put(fence); +err_out: return err; } diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index b053b11692f0..7dabffaf4c65 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -25,4 +25,15 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, struct xe_tile *tile, u64 fault_addr, bool atomic); +static inline bool xe_svm_range_pages_valid(struct xe_svm_range *range) +{ + return drm_gpusvm_range_pages_valid(range->base.gpusvm, &range->base); +} + +#define xe_svm_notifier_lock(vm__) \ + drm_gpusvm_notifier_lock(&(vm__)->svm.gpusvm) + +#define xe_svm_notifier_unlock(vm__) \ + drm_gpusvm_notifier_unlock(&(vm__)->svm.gpusvm) + #endif diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 17ad6a533b2f..6261a4cb2e1d 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -894,6 +894,84 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_ma return fence; } +static void xe_vm_populate_range_rebind(struct xe_vma_op *op, + struct xe_vma *vma, + struct xe_svm_range *range, + u8 tile_mask) +{ + INIT_LIST_HEAD(&op->link); + op->tile_mask = tile_mask; + op->base.op = DRM_GPUVA_OP_USER; + op->subop = XE_VMA_SUBOP_MAP_RANGE; + op->map_range.vma = vma; + op->map_range.range = range; +} + +static int +xe_vm_ops_add_range_rebind(struct xe_vma_ops *vops, + struct xe_vma *vma, + struct xe_svm_range *range, + u8 tile_mask) +{ + struct xe_vma_op *op; + + op = kzalloc(sizeof(*op), GFP_KERNEL); + if (!op) + return -ENOMEM; + + xe_vm_populate_range_rebind(op, vma, range, tile_mask); + list_add_tail(&op->link, &vops->list); + xe_vma_ops_incr_pt_update_ops(vops, tile_mask); + + return 0; +} + +struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, + struct xe_vma *vma, + struct xe_svm_range *range, + u8 tile_mask) +{ + struct dma_fence *fence = NULL; + struct xe_vma_ops vops; + struct xe_vma_op *op, *next_op; + struct xe_tile *tile; + u8 id; + int err; + + lockdep_assert_held(&vm->lock); + xe_vm_assert_held(vm); + xe_assert(vm->xe, xe_vm_in_fault_mode(vm)); + xe_assert(vm->xe, xe_vma_is_system_allocator(vma)); + + xe_vma_ops_init(&vops, vm, NULL, NULL, 0); + for_each_tile(tile, vm->xe, id) { + vops.pt_update_ops[id].wait_vm_bookkeep = true; + vops.pt_update_ops[tile->id].q = + xe_tile_migrate_exec_queue(tile); + } + + err = xe_vm_ops_add_range_rebind(&vops, vma, range, tile_mask); + if (err) + return ERR_PTR(err); + + err = xe_vma_ops_alloc(&vops, false); + if (err) { + fence = ERR_PTR(err); + goto free_ops; + } + + fence = ops_execute(vm, &vops); + +free_ops: + list_for_each_entry_safe(op, next_op, &vops.list, link) { + list_del(&op->link); + kfree(op); + } + xe_vma_ops_fini(&vops); + + return fence; +} + static void xe_vma_free(struct xe_vma *vma) { if (xe_vma_is_userptr(vma)) @@ -2516,6 +2594,8 @@ static void op_trace(struct xe_vma_op *op) case DRM_GPUVA_OP_PREFETCH: trace_xe_vma_bind(gpuva_to_vma(op->base.prefetch.va)); break; + case DRM_GPUVA_OP_USER: + break; default: XE_WARN_ON("NOT POSSIBLE"); } diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index 1a5aed678214..8bd921b33090 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -22,6 +22,7 @@ struct ttm_validate_buffer; struct xe_exec_queue; struct xe_file; struct xe_sync_entry; +struct xe_svm_range; struct drm_exec; struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags); @@ -217,6 +218,10 @@ int xe_vm_userptr_check_repin(struct xe_vm *vm); int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker); struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_mask); +struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, + struct xe_vma *vma, + struct xe_svm_range *range, + u8 tile_mask); int xe_vm_invalidate_vma(struct xe_vma *vma); diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index bd1c0e368238..b736e53779d2 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -19,6 +19,7 @@ #include "xe_range_fence.h" struct xe_bo; +struct xe_svm_range; struct xe_sync_entry; struct xe_user_fence; struct xe_vm; @@ -334,6 +335,14 @@ struct xe_vma_op_prefetch { u32 region; }; +/** struct xe_vma_op_map_range - VMA map range operation */ +struct xe_vma_op_map_range { + /** @vma: VMA to map (system allocator VMA) */ + struct xe_vma *vma; + /** @range: SVM range to map */ + struct xe_svm_range *range; +}; + /** enum xe_vma_op_flags - flags for VMA operation */ enum xe_vma_op_flags { /** @XE_VMA_OP_COMMITTED: VMA operation committed */ @@ -344,6 +353,12 @@ enum xe_vma_op_flags { XE_VMA_OP_NEXT_COMMITTED = BIT(2), }; +/** enum xe_vma_subop - VMA sub-operation */ +enum xe_vma_subop { + /** @XE_VMA_SUBOP_MAP_RANGE: Map range */ + XE_VMA_SUBOP_MAP_RANGE, +}; + /** struct xe_vma_op - VMA operation */ struct xe_vma_op { /** @base: GPUVA base operation */ @@ -352,6 +367,8 @@ struct xe_vma_op { struct list_head link; /** @flags: operation flags */ enum xe_vma_op_flags flags; + /** @subop: user defined sub-operation */ + enum xe_vma_subop subop; /** @tile_mask: Tile mask for operation */ u8 tile_mask; @@ -362,6 +379,8 @@ struct xe_vma_op { struct xe_vma_op_remap remap; /** @prefetch: VMA prefetch operation specific data */ struct xe_vma_op_prefetch prefetch; + /** @map: VMA map range operation specific data */ + struct xe_vma_op_map_range map_range; }; }; From patchwork Wed Aug 28 02:48:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4401EC5474A for ; Wed, 28 Aug 2024 02:48:31 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F345910E45E; Wed, 28 Aug 2024 02:48:18 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Kb+Kprj2"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id E363910E443; Wed, 28 Aug 2024 02:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QQOoj7usOfsVkGJhx1Uhtdq3rwl6xhH5Dkkdh4E17wA=; b=Kb+Kprj2Yo54BFF94z6Ew06XCtkD0CkhgnAlYZ/fSmGPzjHdWBzEJhez N4PZOLrd1d8dJVD2AQ8I/GWF/I+xE4zR6VnIV5wEfRHm/Ms8JOFWI69og 9sZegX0iCJN6OyjI+N6JnkZC72yv0HloVdaVLLYg4S6+Io4T5bGphsfgc ZG15k0QikgZ2YBe7qsqYtTA3QDlKdKnILeqUw4W/upotWLnb9xkVmutCZ tD0d4XknS6XT5R2mTabEgnkJJlXzHz0MY00QA50RocKpy2uPafiOPQQ46 hvPB/hFMsXp5agVs1eQ2ECMRrs+t/a4T4TzAovyHh10DxsWxr2jP6TriJ w==; X-CSE-ConnectionGUID: bY+Oa2osQ6qTIlukJ4BgOw== X-CSE-MsgGUID: ANGL60zpQU661H8WkQoVbQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251896" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251896" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 X-CSE-ConnectionGUID: BlVk3JLbQrKPBQIXROAhdw== X-CSE-MsgGUID: aXztgtR+TNis8GVuO6pSMw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224621" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 12/28] drm/xe: Add SVM garbage collector Date: Tue, 27 Aug 2024 19:48:45 -0700 Message-Id: <20240828024901.2582335-13-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add basic SVM garbage collector which can destroy an SVM range upon an MMU UNMAP event. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 85 +++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_svm.h | 1 + drivers/gpu/drm/xe/xe_vm.c | 6 +++ drivers/gpu/drm/xe/xe_vm_types.h | 5 ++ 4 files changed, 95 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 28d139b3dbb7..20010c09e125 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -30,6 +30,7 @@ xe_svm_range_alloc(struct drm_gpusvm *gpusvm) if (!range) return ERR_PTR(-ENOMEM); + INIT_LIST_HEAD(&range->garbage_collector_link); xe_vm_get(gpusvm_to_vm(gpusvm)); return &range->base; @@ -46,6 +47,24 @@ static struct xe_svm_range *to_xe_range(struct drm_gpusvm_range *r) return container_of(r, struct xe_svm_range, base); } +static void +xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, + const struct mmu_notifier_range *mmu_range) +{ + struct xe_device *xe = vm->xe; + + drm_gpusvm_range_set_unmapped(&range->base, mmu_range); + + spin_lock(&vm->svm.garbage_collector.lock); + if (list_empty(&range->garbage_collector_link)) + list_add_tail(&range->garbage_collector_link, + &vm->svm.garbage_collector.range_list); + spin_unlock(&vm->svm.garbage_collector.lock); + + queue_work(xe_device_get_root_tile(xe)->primary_gt->usm.pf_wq, + &vm->svm.garbage_collector.work); +} + static u8 xe_svm_range_notifier_event_begin(struct xe_vm *vm, struct drm_gpusvm_range *r, const struct mmu_notifier_range *mmu_range, @@ -88,7 +107,9 @@ xe_svm_range_notifier_event_end(struct xe_vm *vm, struct drm_gpusvm_range *r, struct drm_gpusvm_ctx ctx = { .in_notifier = true, }; drm_gpusvm_range_unmap_pages(&vm->svm.gpusvm, r, &ctx); - /* TODO: Add range to garbage collector */ + if (mmu_range->event == MMU_NOTIFY_UNMAP) + xe_svm_garbage_collector_add_range(vm, to_xe_range(r), + mmu_range); } static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, @@ -185,6 +206,58 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, xe_svm_range_notifier_event_end(vm, r, mmu_range); } +static int __xe_svm_garbage_collector(struct xe_vm *vm, + struct xe_svm_range *range) +{ + /* TODO: Do unbind */ + + drm_gpusvm_range_remove(&vm->svm.gpusvm, &range->base); + + return 0; +} + +static int xe_svm_garbage_collector(struct xe_vm *vm) +{ + struct xe_svm_range *range, *next; + int err; + + lockdep_assert_held_write(&vm->lock); + + if (xe_vm_is_closed_or_banned(vm)) + return -ENOENT; + + spin_lock(&vm->svm.garbage_collector.lock); + list_for_each_entry_safe(range, next, + &vm->svm.garbage_collector.range_list, + garbage_collector_link) { + list_del(&range->garbage_collector_link); + spin_unlock(&vm->svm.garbage_collector.lock); + + err = __xe_svm_garbage_collector(vm, range); + if (err) { + drm_warn(&vm->xe->drm, + "Garbage collection failed: %d\n", err); + xe_vm_kill(vm, true); + return err; + } + + spin_lock(&vm->svm.garbage_collector.lock); + } + spin_unlock(&vm->svm.garbage_collector.lock); + + return 0; +} + +static void xe_svm_garbage_collector_work_func(struct work_struct *w) +{ + struct xe_vm *vm = container_of(w, struct xe_vm, + svm.garbage_collector.work); + + down_write(&vm->lock); + xe_svm_garbage_collector(vm); + up_write(&vm->lock); +} + static const struct drm_gpusvm_ops gpusvm_ops = { .range_alloc = xe_svm_range_alloc, .range_free = xe_svm_range_free, @@ -199,6 +272,11 @@ static const u64 fault_chunk_sizes[] = { int xe_svm_init(struct xe_vm *vm) { + spin_lock_init(&vm->svm.garbage_collector.lock); + INIT_LIST_HEAD(&vm->svm.garbage_collector.range_list); + INIT_WORK(&vm->svm.garbage_collector.work, + xe_svm_garbage_collector_work_func); + return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM", &vm->xe->drm, current->mm, NULL, 0, vm->size, SZ_512M, &gpusvm_ops, fault_chunk_sizes, @@ -231,7 +309,10 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, lockdep_assert_held_write(&vm->lock); retry: - /* TODO: Run garbage collector */ + /* Always process UNMAPs first so view SVM ranges is current */ + err = xe_svm_garbage_collector(vm); + if (err) + return err; r = drm_gpusvm_range_find_or_insert(&vm->svm.gpusvm, fault_addr, xe_vma_start(vma), xe_vma_end(vma), diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 7dabffaf4c65..84fd0d8c3380 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -14,6 +14,7 @@ struct xe_vma; struct xe_svm_range { struct drm_gpusvm_range base; + struct list_head garbage_collector_link; u8 tile_present; u8 tile_invalidated; }; diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 6261a4cb2e1d..1fd2c99245f2 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -1633,6 +1633,8 @@ void xe_vm_close_and_put(struct xe_vm *vm) xe_vm_close(vm); if (xe_vm_in_preempt_fence_mode(vm)) flush_work(&vm->preempt.rebind_work); + if (xe_vm_in_fault_mode(vm)) + flush_work(&vm->svm.garbage_collector.work); down_write(&vm->lock); for_each_tile(tile, xe, id) { @@ -3064,6 +3066,10 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_exec_queue; } + /* Ensure all UNMAPs visable */ + if (xe_vm_in_fault_mode(vm)) + flush_work(&vm->svm.garbage_collector.work); + err = down_write_killable(&vm->lock); if (err) goto put_vm; diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index b736e53779d2..2eae3575c409 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -146,6 +146,11 @@ struct xe_vm { struct { /** @svm.gpusvm: base GPUSVM used to track fault allocations */ struct drm_gpusvm gpusvm; + struct { + spinlock_t lock; + struct list_head range_list; + struct work_struct work; + } garbage_collector; } svm; struct xe_device *xe; From patchwork Wed Aug 28 02:48:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C454C5475C for ; Wed, 28 Aug 2024 02:48:23 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A100A10E45B; Wed, 28 Aug 2024 02:48:12 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="dvXfEvxq"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2811210E448; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813289; x=1756349289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gqSgQBX2UKF5ScoIm7eLAT82+JupHu63duFHc9kcuTA=; b=dvXfEvxqYeGWlJ6b2KQwmqsfRjM22+g7WK0NJkFNiMMzc/iT3JrveCym NU/96Or+144l6ke4Ggc3ftGkXlUSIqfV/JNrk8hwnfQJBPe570BDTnpG9 HBINT688hjCU3l0XnFdn6DYp/7gKacVdRDuX0kBIBM2S9roFTyAcz8Dgw dhyn6ZzWvq2SeqlwS3/EmkBqEzNgphzkdm//ErJ6x6BRuHjlTD8uYoIVk HYmdBezfIqzy0N/rkXo9Fr5ETgEaBnQij0F9Bk698XGgn0x2pJEEM48gF uBsgpsW//FGDENsM8OIzTCPOQ0N0UWSzyqMIiCHgPQKtzOagaw0wWkc18 g==; X-CSE-ConnectionGUID: EfM2XtujQK2NhgSrPFv5uQ== X-CSE-MsgGUID: OoCo1sWOQOakJMGMqhNoIg== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251900" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251900" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 X-CSE-ConnectionGUID: sDIJeYWaShC3XPfGcgOu/Q== X-CSE-MsgGUID: J3QgokuwT7O/prAyFv/NZw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224624" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:08 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 13/28] drm/xe: Add unbind to SVM garbage collector Date: Tue, 27 Aug 2024 19:48:46 -0700 Message-Id: <20240828024901.2582335-14-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add unbind to SVM garbage collector. To facilitate add unbind support function to VM layer which unbinds a SVM range. Also teach PY layer to understand unbinds of SVM ranges. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 84 ++++++++++++++++++++++++++------ drivers/gpu/drm/xe/xe_svm.c | 9 +++- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 12 ++++- 5 files changed, 162 insertions(+), 18 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index d5e444af7e02..fc86adf9f0a6 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -905,10 +905,16 @@ static void xe_pt_cancel_bind(struct xe_vma *vma, } } +#define INVALID_VMA (struct xe_vma*)(0xdeaddeadull) + static void xe_pt_commit_locks_assert(struct xe_vma *vma) { - struct xe_vm *vm = xe_vma_vm(vma); + struct xe_vm *vm; + if (vma == INVALID_VMA) + return; + + vm = xe_vma_vm(vma); lockdep_assert_held(&vm->lock); if (!xe_vma_has_no_bo(vma)) @@ -934,7 +940,8 @@ static void xe_pt_commit(struct xe_vma *vma, for (j = 0; j < entries[i].qwords; j++) { struct xe_pt *oldpte = entries[i].pt_entries[j].pt; - xe_pt_destroy(oldpte, xe_vma_vm(vma)->flags, deferred); + xe_pt_destroy(oldpte, (vma == INVALID_VMA) ? 0 : + xe_vma_vm(vma)->flags, deferred); } } } @@ -1367,6 +1374,9 @@ static int xe_pt_svm_pre_commit(struct xe_migrate_pt_update *pt_update) list_for_each_entry(op, &vops->list, link) { struct xe_svm_range *range = op->map_range.range; + if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE) + continue; + xe_assert(vm->xe, xe_vma_is_system_allocator(op->map_range.vma)); xe_assert(vm->xe, op->subop == XE_VMA_SUBOP_MAP_RANGE); @@ -1565,7 +1575,9 @@ static const struct xe_pt_walk_ops xe_pt_stage_unbind_ops = { * xe_pt_stage_unbind() - Build page-table update structures for an unbind * operation * @tile: The tile we're unbinding for. + * @vm: The vm * @vma: The vma we're unbinding. + * @range: The range we're unbinding. * @entries: Caller-provided storage for the update structures. * * Builds page-table update structures for an unbind operation. The function @@ -1575,9 +1587,14 @@ static const struct xe_pt_walk_ops xe_pt_stage_unbind_ops = { * * Return: The number of entries used. */ -static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, struct xe_vma *vma, +static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, + struct xe_vm *vm, + struct xe_vma *vma, + struct xe_svm_range *range, struct xe_vm_pgtable_update *entries) { + u64 start = range ? range->base.va.start : xe_vma_start(vma); + u64 end = range ? range->base.va.end : xe_vma_end(vma); struct xe_pt_stage_unbind_walk xe_walk = { .base = { .ops = &xe_pt_stage_unbind_ops, @@ -1585,14 +1602,14 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, struct xe_vma *vma, .max_level = XE_PT_HIGHEST_LEVEL, }, .tile = tile, - .modified_start = xe_vma_start(vma), - .modified_end = xe_vma_end(vma), + .modified_start = start, + .modified_end = end, .wupd.entries = entries, }; - struct xe_pt *pt = xe_vma_vm(vma)->pt_root[tile->id]; + struct xe_pt *pt = vm->pt_root[tile->id]; - (void)xe_pt_walk_shared(&pt->base, pt->level, xe_vma_start(vma), - xe_vma_end(vma), &xe_walk.base); + (void)xe_pt_walk_shared(&pt->base, pt->level, start, end, + &xe_walk.base); return xe_walk.wupd.num_used_entries; } @@ -1834,13 +1851,6 @@ static int unbind_op_prepare(struct xe_tile *tile, "Preparing unbind, with range [%llx...%llx)\n", xe_vma_start(vma), xe_vma_end(vma) - 1); - /* - * Wait for invalidation to complete. Can corrupt internal page table - * state if an invalidation is running while preparing an unbind. - */ - if (xe_vma_is_userptr(vma) && xe_vm_in_fault_mode(xe_vma_vm(vma))) - mmu_interval_read_begin(&to_userptr_vma(vma)->userptr.notifier); - pt_op->vma = vma; pt_op->bind = false; pt_op->rebind = false; @@ -1849,7 +1859,8 @@ static int unbind_op_prepare(struct xe_tile *tile, if (err) return err; - pt_op->num_entries = xe_pt_stage_unbind(tile, vma, pt_op->entries); + pt_op->num_entries = xe_pt_stage_unbind(tile, xe_vma_vm(vma), + vma, NULL, pt_op->entries); xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries, pt_op->num_entries, false); @@ -1864,6 +1875,42 @@ static int unbind_op_prepare(struct xe_tile *tile, return 0; } +static int unbind_range_prepare(struct xe_vm *vm, + struct xe_tile *tile, + struct xe_vm_pgtable_update_ops *pt_update_ops, + struct xe_svm_range *range) +{ + u32 current_op = pt_update_ops->current_op; + struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; + + if (!(range->tile_present & BIT(tile->id))) + return 0; + + vm_dbg(&vm->xe->drm, + "Preparing unbind, with range [%llx...%llx)\n", + range->base.va.start, range->base.va.end - 1); + + pt_op->vma = INVALID_VMA; + pt_op->bind = false; + pt_op->rebind = false; + + pt_op->num_entries = xe_pt_stage_unbind(tile, vm, NULL, range, + pt_op->entries); + + xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries, + pt_op->num_entries, false); + xe_pt_update_ops_rfence_interval(pt_update_ops, range->base.va.start, + range->base.va.end); + ++pt_update_ops->current_op; + pt_update_ops->needs_svm_lock = true; + pt_update_ops->needs_invalidation = true; + + xe_pt_commit_prepare_unbind(INVALID_VMA, pt_op->entries, + pt_op->num_entries); + + return 0; +} + static int op_prepare(struct xe_vm *vm, struct xe_tile *tile, struct xe_vm_pgtable_update_ops *pt_update_ops, @@ -1931,6 +1978,9 @@ static int op_prepare(struct xe_vm *vm, err = bind_range_prepare(vm, tile, pt_update_ops, op->map_range.vma, op->map_range.range); + } else if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE) { + err = unbind_range_prepare(vm, tile, pt_update_ops, + op->unmap_range.range); } break; default: @@ -2119,6 +2169,8 @@ static void op_commit(struct xe_vm *vm, if (op->subop == XE_VMA_SUBOP_MAP_RANGE) { op->map_range.range->tile_present |= BIT(tile->id); op->map_range.range->tile_invalidated &= ~BIT(tile->id); + } else if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE) { + op->unmap_range.range->tile_present &= ~BIT(tile->id); } break; } diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 20010c09e125..7188aa590fa5 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -209,7 +209,14 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, static int __xe_svm_garbage_collector(struct xe_vm *vm, struct xe_svm_range *range) { - /* TODO: Do unbind */ + struct dma_fence *fence; + + xe_vm_lock(vm, false); + fence = xe_vm_range_unbind(vm, range); + xe_vm_unlock(vm); + if (IS_ERR(fence)) + return PTR_ERR(fence); + dma_fence_put(fence); drm_gpusvm_range_remove(&vm->svm.gpusvm, &range->base); diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 1fd2c99245f2..6916cdfe4be3 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -972,6 +972,79 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, return fence; } +static void xe_vm_populate_range_unbind(struct xe_vma_op *op, + struct xe_svm_range *range) +{ + INIT_LIST_HEAD(&op->link); + op->tile_mask = range->tile_present; + op->base.op = DRM_GPUVA_OP_USER; + op->subop = XE_VMA_SUBOP_UNMAP_RANGE; + op->unmap_range.range = range; +} + +static int +xe_vm_ops_add_range_unbind(struct xe_vma_ops *vops, + struct xe_svm_range *range) +{ + struct xe_vma_op *op; + + op = kzalloc(sizeof(*op), GFP_KERNEL); + if (!op) + return -ENOMEM; + + xe_vm_populate_range_unbind(op, range); + list_add_tail(&op->link, &vops->list); + xe_vma_ops_incr_pt_update_ops(vops, range->tile_present); + + return 0; +} + +struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, + struct xe_svm_range *range) +{ + struct dma_fence *fence = NULL; + struct xe_vma_ops vops; + struct xe_vma_op *op, *next_op; + struct xe_tile *tile; + u8 id; + int err; + + lockdep_assert_held(&vm->lock); + xe_vm_assert_held(vm); + xe_assert(vm->xe, xe_vm_in_fault_mode(vm)); + + if (!range->tile_present) + return dma_fence_get_stub(); + + xe_vma_ops_init(&vops, vm, NULL, NULL, 0); + for_each_tile(tile, vm->xe, id) { + vops.pt_update_ops[id].wait_vm_bookkeep = true; + vops.pt_update_ops[tile->id].q = + xe_tile_migrate_exec_queue(tile); + } + + err = xe_vm_ops_add_range_unbind(&vops, range); + if (err) + return ERR_PTR(err); + + err = xe_vma_ops_alloc(&vops, false); + if (err) { + fence = ERR_PTR(err); + goto free_ops; + } + + fence = ops_execute(vm, &vops); + +free_ops: + list_for_each_entry_safe(op, next_op, &vops.list, link) { + list_del(&op->link); + kfree(op); + } + xe_vma_ops_fini(&vops); + + return fence; +} + static void xe_vma_free(struct xe_vma *vma) { if (xe_vma_is_userptr(vma)) diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index 8bd921b33090..d577ca9e3d65 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -222,6 +222,8 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, struct xe_vma *vma, struct xe_svm_range *range, u8 tile_mask); +struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, + struct xe_svm_range *range); int xe_vm_invalidate_vma(struct xe_vma *vma); diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index 2eae3575c409..d38cf7558f62 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -348,6 +348,12 @@ struct xe_vma_op_map_range { struct xe_svm_range *range; }; +/** struct xe_vma_op_unmap_range - VMA unmap range operation */ +struct xe_vma_op_unmap_range { + /** @range: SVM range to unmap */ + struct xe_svm_range *range; +}; + /** enum xe_vma_op_flags - flags for VMA operation */ enum xe_vma_op_flags { /** @XE_VMA_OP_COMMITTED: VMA operation committed */ @@ -362,6 +368,8 @@ enum xe_vma_op_flags { enum xe_vma_subop { /** @XE_VMA_SUBOP_MAP_RANGE: Map range */ XE_VMA_SUBOP_MAP_RANGE, + /** @XE_VMA_SUBOP_UNMAP_RANGE: Unmap range */ + XE_VMA_SUBOP_UNMAP_RANGE, }; /** struct xe_vma_op - VMA operation */ @@ -384,8 +392,10 @@ struct xe_vma_op { struct xe_vma_op_remap remap; /** @prefetch: VMA prefetch operation specific data */ struct xe_vma_op_prefetch prefetch; - /** @map: VMA map range operation specific data */ + /** @map_range: VMA map range operation specific data */ struct xe_vma_op_map_range map_range; + /** @unmap_range: VMA unmap range operation specific data */ + struct xe_vma_op_map_range unmap_range; }; }; From patchwork Wed Aug 28 02:48:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61EBFC5474A for ; Wed, 28 Aug 2024 02:48:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 55FA410E47C; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="WeBwd2xp"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 67D3110E443; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mDtZfjyBpF1EM1TamxTeoVGjdwIM8ZdiqTK97tr6KrA=; b=WeBwd2xpVDsKzY3Wwl/WA6vAoymPoNk3DtnxJ4tWdWOGI/Vn4IQlsAww ADb5BynfTDGOIwyej2MBExOyvE5edz/LCQmmfcg1E7w7X67N22+04QpLW zNHxSs5gBV2g9TVkkvv9M0q8g25dI8mEFU8FD+PQ6nkH3MYg8XYV3jKqm VYfqvcM7cZ73ifV9YUO2JNAiJYdD+54QjGaFxkWQ8j1DH4T0AzMgCCW2n IuI+9zK9cg4Suv9OmOvLpj7lFIwwcBAlv2BZuhUV8xN3ky31OPDIumZDj sqDUoaS6L7aK4DDxkwG0ge8TYYdNpPJ/z4dqsBD6TPnpKq7TZwOun5Asr w==; X-CSE-ConnectionGUID: JL2uRob2Q7SrX+awrflr3A== X-CSE-MsgGUID: ni3a4stWSmif2IJs2ZgTsw== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251904" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251904" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 X-CSE-ConnectionGUID: wHHVLFrkRxaT/nFizO0hZQ== X-CSE-MsgGUID: s05ZAlXcTfSTbu3gV/7JnA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224628" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 14/28] drm/xe: Do not allow system allocator VMA unbind if the GPU has bindings Date: Tue, 27 Aug 2024 19:48:47 -0700 Message-Id: <20240828024901.2582335-15-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" uAPI is designed with the the use case that only mapping a BO to a malloc'd address will unbind a system allocator VMA. Thus it doesn't make tons of sense to allow a system allocator VMA unbind if the GPU has bindings in the range being unbound. Do not support this as it simplifies the code. Can always be revisited if a use case for this arrises. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 5 +++++ drivers/gpu/drm/xe/xe_svm.h | 1 + drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++ 3 files changed, 22 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 7188aa590fa5..2339359a1d91 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -366,3 +366,8 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, err_out: return err; } + +bool xe_svm_has_mapping(struct xe_vm *vm, u64 start, u64 end) +{ + return drm_gpusvm_has_mapping(&vm->svm.gpusvm, start, end); +} diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 84fd0d8c3380..a4f764bcd835 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -25,6 +25,7 @@ void xe_svm_fini(struct xe_vm *vm); int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, struct xe_tile *tile, u64 fault_addr, bool atomic); +bool xe_svm_has_mapping(struct xe_vm *vm, u64 start, u64 end); static inline bool xe_svm_range_pages_valid(struct xe_svm_range *range) { diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 6916cdfe4be3..d9bff07ef8d1 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -2352,6 +2352,17 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, struct xe_vma *old = gpuva_to_vma(op->base.remap.unmap->va); bool skip = xe_vma_is_system_allocator(old); + u64 start = xe_vma_start(old), end = xe_vma_end(old); + + if (op->base.remap.prev) + start = op->base.remap.prev->va.addr + + op->base.remap.prev->va.range; + if (op->base.remap.next) + end = op->base.remap.next->va.addr; + + if (xe_vma_is_system_allocator(old) && + xe_svm_has_mapping(vm, start, end)) + return -EBUSY; op->remap.start = xe_vma_start(old); op->remap.range = xe_vma_size(old); @@ -2434,6 +2445,11 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, { struct xe_vma *vma = gpuva_to_vma(op->base.unmap.va); + if (xe_vma_is_system_allocator(vma) && + xe_svm_has_mapping(vm, xe_vma_start(vma), + xe_vma_end(vma))) + return -EBUSY; + if (!xe_vma_is_system_allocator(vma)) xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask); break; From patchwork Wed Aug 28 02:48:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDFF7C5474B for ; Wed, 28 Aug 2024 02:48:28 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2606010E467; Wed, 28 Aug 2024 02:48:14 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="H293QB3u"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id AA2C910E44B; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mHgoc+QGoEPC26LQktqCllqoj2jzpaO6eQsC9ossdvc=; b=H293QB3unntHYVB5IepoCCXr/nLYQgHPdSiTxQnYqQ6HtmiNifJXmAtv JlTywkQd9huoM99ejwhzTdovKU6gDU6OZS+vzo9NDvQL63ZkT9LF7pbmC +EVW3Zg5LDEpzDnto9/C9nptIeFnMSTSygNscPhTeZYC4z0uPOdz5mXnf 3ccZs3OrM254NKBdUGxuWknayshGopBIVo2A3L91Qd41WxWwQVjs5GXdP AQjrv0lLF1Bo7TiRRwaddZw1dz8D0Belaq9SF2UX8XldOHlfai/0PHvNS nGu3T5Uo+u4nVOHZNxb4zeM4IGsxEaIYIFbzWrjMmT5BytIxEmCmSrlUo w==; X-CSE-ConnectionGUID: S7CLeUONQb2TNEeVgKR5ZA== X-CSE-MsgGUID: 0JdBqmk+R6Cv7rmxyar3mA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251910" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251910" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 X-CSE-ConnectionGUID: g6wxsU3KQXqGk5rqU2+w5Q== X-CSE-MsgGUID: b777olezQEitpx0RsgugTg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224631" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 15/28] drm/xe: Enable system allocator uAPI Date: Tue, 27 Aug 2024 19:48:48 -0700 Message-Id: <20240828024901.2582335-16-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Support for system allocator bindings in SRAM fully in place, enable the implementation. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_vm.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index d9bff07ef8d1..f7dc681a8b0e 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -2966,12 +2966,6 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe, u16 pat_index = (*bind_ops)[i].pat_index; u16 coh_mode; - /* FIXME: Disabling system allocator for now */ - if (XE_IOCTL_DBG(xe, is_system_allocator)) { - err = -EOPNOTSUPP; - goto free_bind_ops; - } - if (XE_IOCTL_DBG(xe, pat_index >= xe->pat.n_entries)) { err = -EINVAL; goto free_bind_ops; From patchwork Wed Aug 28 02:48:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2280C54754 for ; Wed, 28 Aug 2024 02:48:25 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5DFDA10E463; Wed, 28 Aug 2024 02:48:13 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="j8AaiZ+0"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id CE4D810E44E; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J+1HxTqRJ6rd3/J2jVcEtVEuqmQdZM6MGUO9fhdvQ+A=; b=j8AaiZ+0sQ8VzI8CfTdGBMSD/olAvsQWae+FXh4WxHKiXDtrXvnAlGp2 Nu/wxUw0nxlFb85ts39v1uQVI8a9PVHv8XCqrJhtHCfwQa5XDzGaeJCSH ansSXe0wiRnj+lHndn4KC1Z8A7Su0D+3nzMhnbhCIaDyui9fLCB+KtDS+ oMvMdCkzE8BKT30KcAjSgOO9VPPb93SJ0UEt+GVTXINr5D6jIAijy4rFu 7J5QZe9GWasZGJ1B4oZIisLoZLniOWC0hZ5seaFxRqwozp9VNxnbbtNGr wmK/lI5WW0owJWafFQuBypWRpU2vAHLMlE3LkkOrEp4ob3bJomriQvcsW Q==; X-CSE-ConnectionGUID: +PPgEAEqRdOznInfaToPfw== X-CSE-MsgGUID: Nrzqz45fT6il/iGw43vMOw== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251914" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251914" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 X-CSE-ConnectionGUID: 3PtvXro5Tty4e1e4QbSxmQ== X-CSE-MsgGUID: mg+/UnP+RRmnlwAkzS8SGg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224636" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 16/28] drm/xe: Add migrate layer functions for SVM support Date: Tue, 27 Aug 2024 19:48:49 -0700 Message-Id: <20240828024901.2582335-17-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add functions which migrate to / from VRAM accepting a single DPA argument (VRAM) and array of dma addresses (SRAM). Signed-off-by: Oak Zeng Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_migrate.c | 150 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_migrate.h | 10 +++ 2 files changed, 160 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c index cbf54be224c9..ec033f354e1c 100644 --- a/drivers/gpu/drm/xe/xe_migrate.c +++ b/drivers/gpu/drm/xe/xe_migrate.c @@ -1542,6 +1542,156 @@ void xe_migrate_wait(struct xe_migrate *m) dma_fence_wait(m->fence, false); } +static u32 pte_update_cmd_size(u64 size) +{ + u32 dword; + u64 entries = DIV_ROUND_UP(size, XE_PAGE_SIZE); + + XE_WARN_ON(size > MAX_PREEMPTDISABLE_TRANSFER); + /* + * MI_STORE_DATA_IMM command is used to update page table. Each + * instruction can update maximumly 0x1ff pte entries. To update + * n (n <= 0x1ff) pte entries, we need: + * 1 dword for the MI_STORE_DATA_IMM command header (opcode etc) + * 2 dword for the page table's physical location + * 2*n dword for value of pte to fill (each pte entry is 2 dwords) + */ + dword = (1 + 2) * DIV_ROUND_UP(entries, 0x1ff); + dword += entries * 2; + + return dword; +} + +static void build_pt_update_batch_sram(struct xe_migrate *m, + struct xe_bb *bb, u32 pt_offset, + dma_addr_t *sram_addr, u32 size) +{ + u16 pat_index = tile_to_xe(m->tile)->pat.idx[XE_CACHE_WB]; + u32 ptes; + int i = 0; + + ptes = DIV_ROUND_UP(size, XE_PAGE_SIZE); + while (ptes) { + u32 chunk = min(0x1ffU, ptes); + + bb->cs[bb->len++] = MI_STORE_DATA_IMM | MI_SDI_NUM_QW(chunk); + bb->cs[bb->len++] = pt_offset; + bb->cs[bb->len++] = 0; + + pt_offset += chunk * 8; + ptes -= chunk; + + while (chunk--) { + u64 addr = sram_addr[i++] & PAGE_MASK; + + xe_tile_assert(m->tile, addr); + addr = m->q->vm->pt_ops->pte_encode_addr(m->tile->xe, + addr, pat_index, + 0, false, 0); + bb->cs[bb->len++] = lower_32_bits(addr); + bb->cs[bb->len++] = upper_32_bits(addr); + } + } +} + +enum xe_migrate_copy_dir { + XE_MIGRATE_COPY_TO_VRAM, + XE_MIGRATE_COPY_TO_SRAM, +}; + +static struct dma_fence *xe_migrate_vram(struct xe_migrate *m, + unsigned long npages, + dma_addr_t *sram_addr, u64 vram_addr, + const enum xe_migrate_copy_dir dir) +{ + struct xe_gt *gt = m->tile->primary_gt; + struct xe_device *xe = gt_to_xe(gt); + struct dma_fence *fence = NULL; + u32 batch_size = 2; + u64 src_L0_ofs, dst_L0_ofs; + u64 round_update_size; + struct xe_sched_job *job; + struct xe_bb *bb; + u32 update_idx, pt_slot = 0; + int err; + + round_update_size = min_t(u64, npages * PAGE_SIZE, + MAX_PREEMPTDISABLE_TRANSFER); + batch_size += pte_update_cmd_size(round_update_size); + batch_size += EMIT_COPY_DW; + + bb = xe_bb_new(gt, batch_size, true); + if (IS_ERR(bb)) { + err = PTR_ERR(bb); + return ERR_PTR(err); + } + + build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE, + sram_addr, round_update_size); + + if (dir == XE_MIGRATE_COPY_TO_VRAM) { + src_L0_ofs = xe_migrate_vm_addr(pt_slot, 0); + dst_L0_ofs = xe_migrate_vram_ofs(xe, vram_addr, false); + + } else { + src_L0_ofs = xe_migrate_vram_ofs(xe, vram_addr, false); + dst_L0_ofs = xe_migrate_vm_addr(pt_slot, 0); + } + + bb->cs[bb->len++] = MI_BATCH_BUFFER_END; + update_idx = bb->len; + + emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, round_update_size, + XE_PAGE_SIZE); + + job = xe_bb_create_migration_job(m->q, bb, + xe_migrate_batch_base(m, true), + update_idx); + if (IS_ERR(job)) { + err = PTR_ERR(job); + goto err; + } + + xe_sched_job_add_migrate_flush(job, 0); + + mutex_lock(&m->job_mutex); + xe_sched_job_arm(job); + fence = dma_fence_get(&job->drm.s_fence->finished); + xe_sched_job_push(job); + + dma_fence_put(m->fence); + m->fence = dma_fence_get(fence); + mutex_unlock(&m->job_mutex); + + xe_bb_free(bb, fence); + + return fence; + +err: + mutex_unlock(&m->job_mutex); + xe_bb_free(bb, NULL); + + return ERR_PTR(err); +} + +struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, + unsigned long npages, + dma_addr_t *src_addr, + u64 dst_addr) +{ + return xe_migrate_vram(m, npages, src_addr, dst_addr, + XE_MIGRATE_COPY_TO_VRAM); +} + +struct dma_fence *xe_migrate_from_vram(struct xe_migrate *m, + unsigned long npages, + u64 src_addr, + dma_addr_t *dst_addr) +{ + return xe_migrate_vram(m, npages, dst_addr, src_addr, + XE_MIGRATE_COPY_TO_SRAM); +} + #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST) #include "tests/xe_migrate.c" #endif diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h index 0109866e398a..6ff9a963425c 100644 --- a/drivers/gpu/drm/xe/xe_migrate.h +++ b/drivers/gpu/drm/xe/xe_migrate.h @@ -95,6 +95,16 @@ struct xe_migrate_pt_update { struct xe_migrate *xe_migrate_init(struct xe_tile *tile); +struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, + unsigned long npages, + dma_addr_t *src_addr, + u64 dst_addr); + +struct dma_fence *xe_migrate_from_vram(struct xe_migrate *m, + unsigned long npages, + u64 src_addr, + dma_addr_t *dst_addr); + struct dma_fence *xe_migrate_copy(struct xe_migrate *m, struct xe_bo *src_bo, struct xe_bo *dst_bo, From patchwork Wed Aug 28 02:48:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 493DCC5474E for ; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id ED62010E45A; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="N5PgOSOv"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0D61510E443; Wed, 28 Aug 2024 02:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kxdKfB/eKyRFkT0+jlY6i1Ejdz0dlNPzmMI6dL9xcyg=; b=N5PgOSOvURErKFV4Rx/nH+LENy5tqAoykLjpB2j5PQbpriuLoX7ipqSB v3K4fZBj4SEnQlOUJ8UWwLUC/v9iiZCXGotu7JuzmI2hO0Vzj8c9zfRfT vstID/yQXK1m6ib/fk95wIKK6fWv3VygzEECFNRzxA/RrZAdqUz/Z0+LM ucoU8qFafki85PXLT8uM7lZzXZ+NfWXQV1IQXX3lWDh+SA0B4QioEWz2U yu95SxgC0Xei5DQbxlK/kBHdh2FUiBLHd4qFfgtLypBkRJQimX0rNULM+ FzzC2zC6tLVNVeKkS6Qj/ZbU3ijUU4CphBcupQJxYrKMqLLBtrN/obejG Q==; X-CSE-ConnectionGUID: NciJ6zDAT2O5635UbMguRg== X-CSE-MsgGUID: iK3OcEpkSjKqbItIr9UuGA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251918" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251918" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: hATsDLD0SGi+neaxWYgx/Q== X-CSE-MsgGUID: 0MrZ0eJkTayOzm40wmX56A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224639" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 17/28] drm/xe: Add SVM device memory mirroring Date: Tue, 27 Aug 2024 19:48:50 -0700 Message-Id: <20240828024901.2582335-18-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add SVM device memory mirroring which enables device pages for migration. TODO: Hide this behind Kconfig Signed-off-by: Niranjana Vishwanathapura Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_device_types.h | 8 ++++ drivers/gpu/drm/xe/xe_svm.c | 56 +++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_svm.h | 3 ++ drivers/gpu/drm/xe/xe_tile.c | 5 +++ 4 files changed, 70 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h index 4ecd620921a3..b4367efae55b 100644 --- a/drivers/gpu/drm/xe/xe_device_types.h +++ b/drivers/gpu/drm/xe/xe_device_types.h @@ -105,6 +105,14 @@ struct xe_mem_region { resource_size_t actual_physical_size; /** @mapping: pointer to VRAM mappable space */ void __iomem *mapping; + /** @pagemap: Used to remap device memory as ZONE_DEVICE */ + struct dev_pagemap pagemap; + /** + * @hpa_base: base host physical address + * + * This is generated when remap device memory as ZONE_DEVICE + */ + resource_size_t hpa_base; }; /** diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 2339359a1d91..258a94e83e57 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -21,6 +21,11 @@ static struct xe_vm *range_to_vm(struct drm_gpusvm_range *r) return gpusvm_to_vm(r->gpusvm); } +static void *xe_svm_devm_owner(struct xe_device *xe) +{ + return xe; +} + static struct drm_gpusvm_range * xe_svm_range_alloc(struct drm_gpusvm *gpusvm) { @@ -285,8 +290,9 @@ int xe_svm_init(struct xe_vm *vm) xe_svm_garbage_collector_work_func); return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM", &vm->xe->drm, - current->mm, NULL, 0, vm->size, - SZ_512M, &gpusvm_ops, fault_chunk_sizes, + current->mm, xe_svm_devm_owner(vm->xe), 0, + vm->size, SZ_512M, &gpusvm_ops, + fault_chunk_sizes, ARRAY_SIZE(fault_chunk_sizes)); } @@ -371,3 +377,49 @@ bool xe_svm_has_mapping(struct xe_vm *vm, u64 start, u64 end) { return drm_gpusvm_has_mapping(&vm->svm.gpusvm, start, end); } + +/** + * xe_devm_add: Remap and provide memmap backing for device memory + * @tile: tile that the memory region belongs to + * @mr: memory region to remap + * + * This remap device memory to host physical address space and create + * struct page to back device memory + * + * Return: 0 on success standard error code otherwise + */ +int xe_devm_add(struct xe_tile *tile, struct xe_mem_region *mr) +{ + struct xe_device *xe = tile_to_xe(tile); + struct device *dev = &to_pci_dev(xe->drm.dev)->dev; + struct resource *res; + void *addr; + int ret; + + res = devm_request_free_mem_region(dev, &iomem_resource, + mr->usable_size); + if (IS_ERR(res)) { + ret = PTR_ERR(res); + return ret; + } + + mr->pagemap.type = MEMORY_DEVICE_PRIVATE; + mr->pagemap.range.start = res->start; + mr->pagemap.range.end = res->end; + mr->pagemap.nr_range = 1; + mr->pagemap.ops = drm_gpusvm_pagemap_ops_get(); + mr->pagemap.owner = xe_svm_devm_owner(xe); + addr = devm_memremap_pages(dev, &mr->pagemap); + if (IS_ERR(addr)) { + devm_release_mem_region(dev, res->start, resource_size(res)); + ret = PTR_ERR(addr); + drm_err(&xe->drm, "Failed to remap tile %d memory, errno %d\n", + tile->id, ret); + return ret; + } + mr->hpa_base = res->start; + + drm_info(&xe->drm, "Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n", + tile->id, mr->io_start, mr->io_start + mr->usable_size, res); + return 0; +} diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index a4f764bcd835..f15df5c813f1 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -8,6 +8,7 @@ #include "drm_gpusvm.h" +struct xe_mem_region; struct xe_tile; struct xe_vm; struct xe_vma; @@ -19,6 +20,8 @@ struct xe_svm_range { u8 tile_invalidated; }; +int xe_devm_add(struct xe_tile *tile, struct xe_mem_region *mr); + int xe_svm_init(struct xe_vm *vm); void xe_svm_fini(struct xe_vm *vm); diff --git a/drivers/gpu/drm/xe/xe_tile.c b/drivers/gpu/drm/xe/xe_tile.c index 15ea0a942f67..1c1b3d406f1e 100644 --- a/drivers/gpu/drm/xe/xe_tile.c +++ b/drivers/gpu/drm/xe/xe_tile.c @@ -10,6 +10,7 @@ #include "xe_gt.h" #include "xe_migrate.h" #include "xe_sa.h" +#include "xe_svm.h" #include "xe_tile.h" #include "xe_tile_sysfs.h" #include "xe_ttm_vram_mgr.h" @@ -158,6 +159,7 @@ static int tile_ttm_mgr_init(struct xe_tile *tile) */ int xe_tile_init_noalloc(struct xe_tile *tile) { + struct xe_device *xe = tile_to_xe(tile); int err; err = tile_ttm_mgr_init(tile); @@ -170,6 +172,9 @@ int xe_tile_init_noalloc(struct xe_tile *tile) xe_wa_apply_tile_workarounds(tile); + if (xe->info.has_usm && IS_DGFX(xe)) + xe_devm_add(tile, &tile->mem.vram); + err = xe_tile_sysfs_init(tile); return 0; From patchwork Wed Aug 28 02:48:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F3FAC54754 for ; Wed, 28 Aug 2024 02:48:19 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CF3C710E458; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="SN+UjAM1"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2FBC710E44B; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nEIA6EZhqfbNpqJqrcdWXkhSdB3sakd8rSbMVxuyzLk=; b=SN+UjAM1fvC5VJBO4FZLGpuZT9Nt6Y8X2FkiOUiJ0G6PMOyc1QL8r8kr nEBgX2HLOYdtRgOZWME9vJkwoxewPGc9vyYZzuBvls03/uWyyhjiX4rw5 GAGO4/2uuGbm2fljsgJ+1utZwleGlv7q3JeCF9KZMgH6yXTrU+W+6wGCe clBLWt9x1XzXxct631ncQq/18M4/ANNHOwTbCsSJt3s+RmPsJTzxI2amH ryiSzHUjjI28KBVhpgqdLjfrfw6LdT6OBgND6TRn+49YgjVgeaHT5O3jE zeL8Zw+j5C+lCx26a+CEuuI7Eaq9J6NXkuJ5ppWEsfPyH7gH4VuofxSna w==; X-CSE-ConnectionGUID: hPOTDWDiQAOEBGqEgeqzzQ== X-CSE-MsgGUID: akWFcJmxRlyYz9JnQxAAZA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251922" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251922" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: tb74yusGTwy+HCubENekKg== X-CSE-MsgGUID: mOjcp6eJRQGhzKhOq8XXOw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224643" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:09 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 18/28] drm/xe: Add GPUSVM copy SRAM / VRAM vfunc functions Date: Tue, 27 Aug 2024 19:48:51 -0700 Message-Id: <20240828024901.2582335-19-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add GPUSVM copy SRAM / VRAM vfunc functions and connect to migration layer. Signed-off-by: Matthew Brost Me: Fix vram_addr == 0 case --- drivers/gpu/drm/xe/xe_svm.c | 153 ++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 258a94e83e57..6c690ba827e7 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -6,6 +6,7 @@ #include "drm_gpusvm.h" #include "xe_gt_tlb_invalidation.h" +#include "xe_migrate.h" #include "xe_pt.h" #include "xe_svm.h" #include "xe_vm.h" @@ -270,9 +271,161 @@ static void xe_svm_garbage_collector_work_func(struct work_struct *w) up_write(&vm->lock); } +static struct xe_mem_region *page_to_mr(struct page *page) +{ + return container_of(page->pgmap, struct xe_mem_region, pagemap); +} + +static struct xe_tile *mr_to_tile(struct xe_mem_region *mr) +{ + return container_of(mr, struct xe_tile, mem.vram); +} + +static u64 xe_mem_region_page_to_dpa(struct xe_mem_region *mr, + struct page *page) +{ + u64 dpa; + struct xe_tile *tile = mr_to_tile(mr); + u64 pfn = page_to_pfn(page); + u64 offset; + + xe_tile_assert(tile, is_device_private_page(page)); + xe_tile_assert(tile, (pfn << PAGE_SHIFT) >= mr->hpa_base); + + offset = (pfn << PAGE_SHIFT) - mr->hpa_base; + dpa = mr->dpa_base + offset; + + return dpa; +} + +enum xe_svm_copy_dir { + XE_SVM_COPY_TO_VRAM, + XE_SVM_COPY_TO_SRAM, +}; + +static int xe_svm_copy(struct drm_gpusvm *gpusvm, struct page **pages, + dma_addr_t *dma_addr, unsigned long npages, + const enum xe_svm_copy_dir dir) +{ + struct xe_vm *vm = gpusvm_to_vm(gpusvm); + struct xe_mem_region *mr = NULL; + struct xe_tile *tile; + struct dma_fence *fence = NULL; + unsigned long i; +#define VRAM_ADDR_INVALID ~0x0ull + u64 vram_addr = VRAM_ADDR_INVALID; + int err = 0, pos = 0; + bool sram = dir == XE_SVM_COPY_TO_SRAM; + + for (i = 0; i < npages; ++i) { + struct page *spage = pages[i]; + struct dma_fence *__fence; + u64 __vram_addr; + bool match = false, chunk, last; + + chunk = (i - pos) == (SZ_2M / PAGE_SIZE); + last = (i + 1) == npages; + + if (!dma_addr[i] && vram_addr == VRAM_ADDR_INVALID) + continue; + + if (!mr) { + mr = page_to_mr(spage); + tile = mr_to_tile(mr); + } + + if (dma_addr[i]) { + __vram_addr = xe_mem_region_page_to_dpa(mr, spage); + if (vram_addr == VRAM_ADDR_INVALID) { + vram_addr = __vram_addr; + pos = i; + } + + xe_assert(vm->xe, __vram_addr != VRAM_ADDR_INVALID); + xe_assert(vm->xe, vram_addr != VRAM_ADDR_INVALID); + + match = vram_addr + PAGE_SIZE * (i - pos) == __vram_addr; + } + + if (!match || chunk || last) { + int incr = (match && last) ? 1 : 0; + + if (vram_addr != VRAM_ADDR_INVALID) { + if (sram) + __fence = xe_migrate_from_vram(tile->migrate, + i - pos + incr, + vram_addr, + dma_addr + pos); + else + __fence = xe_migrate_to_vram(tile->migrate, + i - pos + incr, + dma_addr + pos, + vram_addr); + if (IS_ERR(__fence)) { + err = PTR_ERR(__fence); + goto err_out; + } + + dma_fence_put(fence); + fence = __fence; + } + + if (dma_addr[i]) { + vram_addr = __vram_addr; + pos = i; + } else { + vram_addr = VRAM_ADDR_INVALID; + } + + if (!match && last && dma_addr[i]) { + if (sram) + __fence = xe_migrate_from_vram(tile->migrate, 1, + vram_addr, + dma_addr + pos); + else + __fence = xe_migrate_to_vram(tile->migrate, 1, + dma_addr + pos, + vram_addr); + if (IS_ERR(__fence)) { + err = PTR_ERR(__fence); + goto err_out; + } + + dma_fence_put(fence); + fence = __fence; + } + } + } + +err_out: + if (fence) { + dma_fence_wait(fence, false); + dma_fence_put(fence); + } + + return err; +#undef VRAM_ADDR_INVALID +} + +static int xe_svm_copy_to_vram(struct drm_gpusvm *gpusvm, struct page **pages, + dma_addr_t *dma_addr, unsigned long npages) +{ + return xe_svm_copy(gpusvm, pages, dma_addr, npages, + XE_SVM_COPY_TO_VRAM); +} + +static int xe_svm_copy_to_sram(struct drm_gpusvm *gpusvm, struct page **pages, + dma_addr_t *dma_addr, unsigned long npages) +{ + return xe_svm_copy(gpusvm, pages, dma_addr, npages, + XE_SVM_COPY_TO_SRAM); +} + static const struct drm_gpusvm_ops gpusvm_ops = { .range_alloc = xe_svm_range_alloc, .range_free = xe_svm_range_free, + .copy_to_vram = xe_svm_copy_to_vram, + .copy_to_sram = xe_svm_copy_to_sram, .invalidate = xe_svm_invalidate, }; From patchwork Wed Aug 28 02:48:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E17AC5474D for ; Wed, 28 Aug 2024 02:48:30 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B051B10E44C; Wed, 28 Aug 2024 02:48:14 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="cuB+3uKa"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3DF5D10E44E; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JkbPLNkKBaSRiE5EGFBuRjRblyt1zA7OOQaLpNyGVeM=; b=cuB+3uKa1EQx9CqhAQdTXOF5exScpN4HbqUMEpFgO/wZJIhsWwUQ4gsb UafVBJKgE9C34GmWpyuorPeQydGocIT9dBzyBmmF7QwL6TM4kxNb1yp+P hdu7QeORbD3SSQC+Zj3yrqgSnnLHEnDWUt7A1JfPFn22DYYaGU9bdAM7Q u2J//zj4AkzIdoay9r0VGEV9ncpNoSJm/zj7itv+yeqQTP+Lh27hFzBEn ho4RYW3sqelNZ8G8W9xxOVK+4xJwkLtqjN7figX/tu7q56dkuLudGsRH3 J68NtfUNThz5mfR37aCVn068of5cZpKav3jWS5lchmeVTzblR3eTBHoRR w==; X-CSE-ConnectionGUID: PPAvnqQtSguj0rCJ6fPP4w== X-CSE-MsgGUID: SpvnCP/qRcOorYpke97moQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251926" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251926" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: YcSSrhn4TX+N3vaaddauWQ== X-CSE-MsgGUID: uMDBUtBySI+w5lNOX1iVXA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224646" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 19/28] drm/xe: Update PT layer to understand ranges in VRAM Date: Tue, 27 Aug 2024 19:48:52 -0700 Message-Id: <20240828024901.2582335-20-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Kinda cheating here using BO directly rather than VRAM pages. Same at the moment as mixed mappings are not supported. If this changes, then the arary of pages / dma addresses will need a cursor. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 22 ++++++++++++++++------ drivers/gpu/drm/xe/xe_svm.h | 10 ++++++++++ 2 files changed, 26 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index fc86adf9f0a6..e9195029ea60 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -607,9 +607,12 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, struct xe_vm_pgtable_update *entries, u32 *num_entries) { struct xe_device *xe = tile_to_xe(tile); - struct xe_bo *bo = xe_vma_bo(vma); - bool is_devmem = !xe_vma_is_userptr(vma) && bo && - (xe_bo_is_vram(bo) || xe_bo_is_stolen_devmem(bo)); + bool range_devmem = range && xe_svm_range_in_vram(range); + struct xe_bo *bo = range_devmem ? range->base.vram_allocation : + xe_vma_bo(vma); + bool is_devmem = range_devmem || + (!xe_vma_is_userptr(vma) && bo && + (xe_bo_is_vram(bo) || xe_bo_is_stolen_devmem(bo))); struct xe_res_cursor curs; struct xe_pt_stage_bind_walk xe_walk = { .base = { @@ -675,9 +678,16 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, xe_bo_assert_held(bo); if (range) { - xe_res_first_dma(range->base.dma_addr, 0, - range->base.va.end - range->base.va.start, - range->base.order, &curs); + if (is_devmem) + xe_res_first(bo->ttm.resource, 0, + range->base.va.end - range->base.va.start, + &curs); + else if (xe_svm_range_has_dma_mapping(range)) + xe_res_first_dma(range->base.dma_addr, 0, + range->base.va.end - range->base.va.start, + range->base.order, &curs); + else + return -EAGAIN; /* Invalidation corner case */ } else if (!xe_vma_is_null(vma)) { if (xe_vma_is_userptr(vma)) xe_res_first_sg(to_userptr_vma(vma)->userptr.sg, 0, diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index f15df5c813f1..8b72e91cc37d 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -35,6 +35,16 @@ static inline bool xe_svm_range_pages_valid(struct xe_svm_range *range) return drm_gpusvm_range_pages_valid(range->base.gpusvm, &range->base); } +static inline bool xe_svm_range_in_vram(struct xe_svm_range *range) +{ + return range->base.flags.has_vram_pages; +} + +static inline bool xe_svm_range_has_dma_mapping(struct xe_svm_range *range) +{ + return range->base.flags.has_dma_mapping; +} + #define xe_svm_notifier_lock(vm__) \ drm_gpusvm_notifier_lock(&(vm__)->svm.gpusvm) From patchwork Wed Aug 28 02:48:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38E57C54799 for ; Wed, 28 Aug 2024 02:48:22 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 61C9210E457; Wed, 28 Aug 2024 02:48:12 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="ggG2JsrS"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5997110E443; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dffn3G8V5DqXAkynazLT52niPnWTv1oq1p1BNlYdnXE=; b=ggG2JsrSpzecF3NLa4xV9H5n+HSyTEz+NmLL57FzPMKkGv97MJDdHz66 uN5O+/sL81H21Ik5d3iKMc4UImsRMH9PkQOHE8au31UBxJB26+7xBgpY3 1p2gRRh2cr13M4DD7UO9sl197VZuOC8q9SebI1NSXRNK8C9l2vtQt+PTi zrnBU7tuYS+WQBu1IOd3CGxDAHZ02LCj6dUQLWcMXs6uE/z9HdzurcwfB 5nY1EkVuyLesfEgoZKm8JYKwj5J379+8deUOdm2wAmyQB14oNrLg2oxQe OasTZnuhbcQwRYhg1Nb/RcrxHDpew6sOG2Que8GDV+dGlLhmHxRFbYtXV g==; X-CSE-ConnectionGUID: +ESlr6mhRKKGUfQygw0+tA== X-CSE-MsgGUID: T/5kN/vJTSikVBe2bMw0sQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251930" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251930" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: XAjmRqzMSk+piUc+Wwezdw== X-CSE-MsgGUID: 3z2oBxoKT6mprXwTnkZ0ww== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224649" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 20/28] drm/xe: Add Xe SVM populate_vram_pfn vfunc Date: Tue, 27 Aug 2024 19:48:53 -0700 Message-Id: <20240828024901.2582335-21-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Signed-off-by: Niranjana Vishwanathapura Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 6c690ba827e7..82cb5a260c87 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -9,6 +9,7 @@ #include "xe_migrate.h" #include "xe_pt.h" #include "xe_svm.h" +#include "xe_ttm_vram_mgr.h" #include "xe_vm.h" #include "xe_vm_types.h" @@ -421,9 +422,45 @@ static int xe_svm_copy_to_sram(struct drm_gpusvm *gpusvm, struct page **pages, XE_SVM_COPY_TO_SRAM); } +static u64 block_offset_to_pfn(struct xe_mem_region *mr, u64 offset) +{ + return PHYS_PFN(offset + mr->hpa_base); +} + +static struct drm_buddy *tile_to_buddy(struct xe_tile *tile) +{ + return &tile->mem.vram_mgr->mm; +} + +static int xe_svm_populate_vram_pfn(struct drm_gpusvm *gpusvm, + void *vram_allocation, + unsigned long npages, + unsigned long *pfn) +{ + struct xe_bo *bo = vram_allocation; + struct ttm_resource *res = bo->ttm.resource; + struct list_head *blocks = &to_xe_ttm_vram_mgr_resource(res)->blocks; + struct drm_buddy_block *block; + int j =0; + + list_for_each_entry(block, blocks, link) { + struct xe_mem_region *mr = block->private; + struct xe_tile *tile = mr_to_tile(mr); + struct drm_buddy *buddy = tile_to_buddy(tile); + u64 block_pfn = block_offset_to_pfn(mr, drm_buddy_block_offset(block)); + int i; + + for(i = 0; i < drm_buddy_block_size(buddy, block) >> PAGE_SHIFT; ++i) + pfn[j++] = block_pfn + i; + } + + return 0; +} + static const struct drm_gpusvm_ops gpusvm_ops = { .range_alloc = xe_svm_range_alloc, .range_free = xe_svm_range_free, + .populate_vram_pfn = xe_svm_populate_vram_pfn, .copy_to_vram = xe_svm_copy_to_vram, .copy_to_sram = xe_svm_copy_to_sram, .invalidate = xe_svm_invalidate, From patchwork Wed Aug 28 02:48:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3557C5474D for ; Wed, 28 Aug 2024 02:48:21 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6168C10E456; Wed, 28 Aug 2024 02:48:12 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="FY0cVVMw"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7460D10E44B; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813290; x=1756349290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SM8XB4VrHy7O7uQVm5RJ4pT4QRxR7TsVKZl2nXaN9yM=; b=FY0cVVMw/hL8PMgrEr2UWlwRhId6AlMsjqFF4/ANR7P25ZCXJzS99b/X YZQoyRrw5DW1KWa9igcpHpGSGRuxK+af0bvR6JVt9HPEqBFmbTnIRqRLI Zw/tVeBqVmuB+xuSaTZrkoTOd6Zoer+7yipLvXYDahQn5tvVoqd+Ny0tY X5i7OFJcI+4hrRvVGyfSSYERpUUaoKGdRd3j4Hxcs44a+kiLt5PvI/OcD vP545K27Ne8j7o7+MbDR42i5IQGaX7jveKA9E+smllki2p8NZoGwo/0cS TU6gd+qRyBSyDJTDicsVlUMXmSvOH2H5mVnG+d8j1fEHDiWHeNGvBUoMa A==; X-CSE-ConnectionGUID: 56hjmnEyQdOziFHjQl8EXA== X-CSE-MsgGUID: Q2kU9nGGRE2/3CDfJHan5A== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251934" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251934" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: QasRnbUpRJCAI7sYOwu99g== X-CSE-MsgGUID: i7OB8gWiRxqUSlDs9ks4Hg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224654" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 21/28] drm/xe: Add Xe SVM vram_release vfunc Date: Tue, 27 Aug 2024 19:48:54 -0700 Message-Id: <20240828024901.2582335-22-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Implement with a simple BO put. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_svm.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 82cb5a260c87..4372c02a341f 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -5,6 +5,7 @@ #include "drm_gpusvm.h" +#include "xe_bo.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" #include "xe_pt.h" @@ -422,6 +423,11 @@ static int xe_svm_copy_to_sram(struct drm_gpusvm *gpusvm, struct page **pages, XE_SVM_COPY_TO_SRAM); } +static void xe_svm_vram_release(void *vram_allocation) +{ + xe_bo_put(vram_allocation); +} + static u64 block_offset_to_pfn(struct xe_mem_region *mr, u64 offset) { return PHYS_PFN(offset + mr->hpa_base); @@ -460,6 +466,7 @@ static int xe_svm_populate_vram_pfn(struct drm_gpusvm *gpusvm, static const struct drm_gpusvm_ops gpusvm_ops = { .range_alloc = xe_svm_range_alloc, .range_free = xe_svm_range_free, + .vram_release = xe_svm_vram_release, .populate_vram_pfn = xe_svm_populate_vram_pfn, .copy_to_vram = xe_svm_copy_to_vram, .copy_to_sram = xe_svm_copy_to_sram, From patchwork Wed Aug 28 02:48:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EDC96C5474C for ; Wed, 28 Aug 2024 02:48:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 24B6410E484; Wed, 28 Aug 2024 02:48:26 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="bV6FLgwS"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 97EE310E44E; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813291; x=1756349291; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sMk/7YX2p/2MK0vK2YrTbxN0/jQNK9Tnxet3NvVWjS0=; b=bV6FLgwS55VuKmmNLWUR5f89eGYTEDROLpvTBcTonx4edxxBkE3OEw8C p17WRYKYP3aUQNthKcy1Peo78hlVRClFUqjbBX0DW4HJxHS/OW6p3tKeK 6l7r0JZgSm0P8QUVWszVPz2BbkKemXNLSdyeSJ307TZMaeWYIB2ppFnqG c+MCbvoMSwumh+clQQqhPSfshAoOhYsvpz3gx8rrvBDorKqHm0YhRugdw yCH5b7m1F5IKmzRhiwjTtl2XQ9cdiXsmM7P/WgScUk4q9mJMAudfl7wVc rtCpgXBnQJ0EdPbw7eaZ+3Ffbbg/WPHuuc78AFHeIENAefmNrD+1xpTKV g==; X-CSE-ConnectionGUID: kLMyFBrRTzOxdf6vpktZlQ== X-CSE-MsgGUID: 0fi5XLrWQfCCIC0xG9xp8A== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251938" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251938" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 X-CSE-ConnectionGUID: yeKZ6j5/RlGNe0G+ovNGyw== X-CSE-MsgGUID: c8/3JMfkS3G1gTVyeQ3cXQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224658" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 22/28] drm/xe: Add BO flags required for SVM Date: Tue, 27 Aug 2024 19:48:55 -0700 Message-Id: <20240828024901.2582335-23-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add XE_BO_FLAG_SYSTEM_ALLOC to indicate BO is tied to SVM range. Add XE_BO_FLAG_SKIP_CLEAR to indicate BO does not need to cleared. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_bo.c | 11 +++++++---- drivers/gpu/drm/xe/xe_bo.h | 2 ++ 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index b6c6a4a3b4d4..ad804b6f9e84 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -704,9 +704,10 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, (!mem_type_is_vram(old_mem_type) && !tt_has_data); clear_system_pages = ttm && (ttm->page_flags & TTM_TT_FLAG_CLEARED_ON_FREE); - needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) || + needs_clear = !(bo->flags & XE_BO_FLAG_SKIP_CLEAR) && + ((ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) || (!ttm && ttm_bo->type == ttm_bo_type_device) || - clear_system_pages; + clear_system_pages); if (new_mem->mem_type == XE_PL_TT) { ret = xe_tt_map_sg(ttm); @@ -1284,7 +1285,8 @@ struct xe_bo *___xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo, int err; /* Only kernel objects should set GT */ - xe_assert(xe, !tile || type == ttm_bo_type_kernel); + xe_assert(xe, !tile || type == ttm_bo_type_kernel || + flags & XE_BO_FLAG_SYSTEM_ALLOC); if (XE_WARN_ON(!size)) { xe_bo_free(bo); @@ -2292,7 +2294,8 @@ bool xe_bo_needs_ccs_pages(struct xe_bo *bo) * can't be used since there's no CCS storage associated with * non-VRAM addresses. */ - if (IS_DGFX(xe) && (bo->flags & XE_BO_FLAG_SYSTEM)) + if (IS_DGFX(xe) && ((bo->flags & XE_BO_FLAG_SYSTEM) || + (bo->flags & XE_BO_FLAG_SYSTEM_ALLOC))) return false; return true; diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h index dbfb3209615d..fe2ce641b256 100644 --- a/drivers/gpu/drm/xe/xe_bo.h +++ b/drivers/gpu/drm/xe/xe_bo.h @@ -39,6 +39,8 @@ #define XE_BO_FLAG_NEEDS_64K BIT(15) #define XE_BO_FLAG_NEEDS_2M BIT(16) #define XE_BO_FLAG_GGTT_INVALIDATE BIT(17) +#define XE_BO_FLAG_SYSTEM_ALLOC BIT(18) +#define XE_BO_FLAG_SKIP_CLEAR BIT(19) /* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30) #define XE_BO_FLAG_INTERNAL_64K BIT(31) From patchwork Wed Aug 28 02:48:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C94C9C5474A for ; Wed, 28 Aug 2024 02:48:38 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4915210E487; Wed, 28 Aug 2024 02:48:26 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="atw40QJj"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id F080B10E44B; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813291; x=1756349291; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gypVPkmhZXEdTDatyyfXN3j/p9WiXV7EnFZRvn9huV4=; b=atw40QJjTg/u++A7JtADK6AsdrQc7K9r7+06BJC7e+/KPLy/FqdPHHcB SIuVHGcF/DWFcZVn3SGP4b5EcVS3fs7TD+hD7ZuU4n+VfMNOeA9ZnomvV vIjkYUp4Qzt7vcErihdxSqEC34TtfKoYXPYcPqFp9Vezg3TWLskJWngl7 WZdEweOCXnkBQrXOjX/XhdlNf4UGdI5hur0mVbYRqRva5uvchbGySUZcp Q6JByBZbPh57MXOgAVqNSdXIhJchJxqaOq+HttdNtjpg0Y5onaTm34NsL yfVqBM8KeyrJIM3nVjW1hgVGL0xar8Cr0jJnANcnXOFqCUBt4W1WVX3Kp w==; X-CSE-ConnectionGUID: hIKASkQ5R4GbId2yW9/TQw== X-CSE-MsgGUID: e4DcgTmcS3Gboc3GRVVQ8w== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251942" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251942" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 X-CSE-ConnectionGUID: PDJKy8m2QPCaXFtLJjJP3w== X-CSE-MsgGUID: 6LbZLG4qQVWMeAiN5H/R6g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224661" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 23/28] drm/xe: Add SVM VRAM migration Date: Tue, 27 Aug 2024 19:48:56 -0700 Message-Id: <20240828024901.2582335-24-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Migration is implemented with range granularity, with VRAM backing being a VM private TTM BO (i.e., shares dma-resv with VM). The lifetime of the TTM BO is limited to when the SVM range is in VRAM (i.e., when a VRAM SVM range is migrated to SRAM, the TTM BO is destroyed). The design choice for using TTM BO for VRAM backing store, as opposed to direct buddy allocation, is as follows: - DRM buddy allocations are not at page granularity, offering no advantage over a BO. - DRM buddy allocations do not solve locking inversion problems between mmap lock and dma-resv locks. - Unified eviction is required (SVM VRAM and TTM BOs need to be able to evict each other). - For exhaustive eviction [1], SVM VRAM allocations will almost certainly require a dma-resv. - Likely allocation size is 2M which makes of size of BO (872) acceptable per allocation (872 / 2M == .0004158). With this, using TTM BO for VRAM backing store seems to be an obvious choice as it allows leveraging of the TTM eviction code. Current migration policy is migrate any SVM range greater than or equal to 64k once. [1] https://patchwork.freedesktop.org/series/133643/ Signed-off-by: Matthew Brost matthew.brost@intel.com --- drivers/gpu/drm/xe/xe_svm.c | 81 ++++++++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_svm.h | 1 + 2 files changed, 81 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 4372c02a341f..fd8987e0a506 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -217,8 +217,13 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, static int __xe_svm_garbage_collector(struct xe_vm *vm, struct xe_svm_range *range) { + struct drm_gpusvm_ctx ctx = {}; struct dma_fence *fence; + /* Evict any pages holding references to vram allocation */ + if (range->base.flags.partial_unmap && IS_DGFX(vm->xe)) + drm_gpusvm_migrate_to_sram(&vm->svm.gpusvm, &range->base, &ctx); + xe_vm_lock(vm, false); fence = xe_vm_range_unbind(vm, range); xe_vm_unlock(vm); @@ -504,21 +509,77 @@ static bool xe_svm_range_is_valid(struct xe_svm_range *range, return (range->tile_present & ~range->tile_invalidated) & BIT(tile->id); } +static struct xe_mem_region *tile_to_mr(struct xe_tile *tile) +{ + return &tile->mem.vram; +} + +static struct xe_bo *xe_svm_alloc_vram(struct xe_vm *vm, struct xe_tile *tile, + struct xe_svm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct xe_mem_region *mr = tile_to_mr(tile); + struct drm_buddy_block *block; + struct list_head *blocks; + struct xe_bo *bo; + ktime_t end = 0; + int err; + +retry: + xe_vm_lock(vm, false); + bo = xe_bo_create(tile_to_xe(tile), tile, vm, range->base.va.end - + range->base.va.start, ttm_bo_type_device, + XE_BO_FLAG_VRAM_IF_DGFX(tile) | + XE_BO_FLAG_SYSTEM_ALLOC | XE_BO_FLAG_SKIP_CLEAR); + xe_vm_unlock(vm); + if (IS_ERR(bo)) { + err = PTR_ERR(bo); + if (xe_vm_validate_should_retry(NULL, err, &end)) + goto retry; + return bo; + } + + blocks = &to_xe_ttm_vram_mgr_resource(bo->ttm.resource)->blocks; + list_for_each_entry(block, blocks, link) + block->private = mr; + + /* + * Take ref because as soon as drm_gpusvm_migrate_to_vram succeeds the + * creation ref can be dropped upon CPU fault or unmap. + */ + xe_bo_get(bo); + + err = drm_gpusvm_migrate_to_vram(&vm->svm.gpusvm, &range->base, + bo, ctx); + if (err) { + xe_bo_put(bo); /* Local ref */ + xe_bo_put(bo); /* Creation ref */ + return ERR_PTR(err); + } + + return bo; +} + int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, struct xe_tile *tile, u64 fault_addr, bool atomic) { - struct drm_gpusvm_ctx ctx = { .read_only = xe_vma_read_only(vma), }; + struct drm_gpusvm_ctx ctx = { .read_only = xe_vma_read_only(vma), + .vram_possible = IS_DGFX(vm->xe), }; struct xe_svm_range *range; struct drm_gpusvm_range *r; struct drm_exec exec; struct dma_fence *fence; + struct xe_bo *bo = NULL; ktime_t end = 0; int err; lockdep_assert_held_write(&vm->lock); retry: + xe_bo_put(bo); + bo = NULL; + /* Always process UNMAPs first so view SVM ranges is current */ err = xe_svm_garbage_collector(vm); if (err) @@ -534,6 +595,22 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, if (xe_svm_range_is_valid(range, tile)) return 0; + /* XXX: Add migration policy, for now migrate range once */ + if (IS_DGFX(vm->xe) && !range->migrated && + range->base.flags.migrate_vram && + (range->base.va.end - range->base.va.start) >= SZ_64K) { + range->migrated = true; + + bo = xe_svm_alloc_vram(vm, tile, range, &ctx); + if (IS_ERR(bo)) { + drm_info(&vm->xe->drm, + "VRAM allocation failed, falling back to retrying, asid=%u, errno %ld\n", + vm->usm.asid, PTR_ERR(bo)); + bo = NULL; + goto retry; + } + } + err = drm_gpusvm_range_get_pages(&vm->svm.gpusvm, r, &ctx); if (err == -EFAULT || err == -EPERM) /* Corner where CPU mappings have change */ goto retry; @@ -567,6 +644,8 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, dma_fence_put(fence); err_out: + xe_bo_put(bo); + return err; } diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 8b72e91cc37d..3f432483a230 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -18,6 +18,7 @@ struct xe_svm_range { struct list_head garbage_collector_link; u8 tile_present; u8 tile_invalidated; + u8 migrated :1; }; int xe_devm_add(struct xe_tile *tile, struct xe_mem_region *mr); From patchwork Wed Aug 28 02:48:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DD24C54750 for ; Wed, 28 Aug 2024 02:48:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0873A10E474; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="i9wYjkcv"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0164A10E450; Wed, 28 Aug 2024 02:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813291; x=1756349291; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tZD8mug3jPxdp5TPQkl6IEM3eKj6lxJMgG/IzZW/DJM=; b=i9wYjkcvCCPfDM45yrsXOly925+aLvam3gVItSMGJHK13uJSPQhbATnd /w68si8SCDBRZyVQ7J5sU+ytfxZBQQYIBXwX86hMZG+4JcoM8L5X4NFYH b7Ld9t2xNEKmkj8ItgJ69ODF+YS7wEEBPvYWWckJMOacL3daNukHoVK4L 3K0uqeAhXXsob8AcfPyKA8DP2Vi/0vRelCmoq/F8QzJmyl1HYtUzaQhkP /T3YoIC26m1pOnXQ9wZ9jalQajDGoaORofgcA8wGKhZA+/qZ8aORgyUWo Gwap+jUL1wUIFoKzSZ2jamx+oZwl14JpD3aYwFsrS/LXNxCZTxi8YoyKV Q==; X-CSE-ConnectionGUID: o81DBNUhTkqb/FDkGStTaA== X-CSE-MsgGUID: 2ccikKw3RjKdj9wyHxVbAQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251946" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251946" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 X-CSE-ConnectionGUID: ORECK4K3Te+deQS540uMDA== X-CSE-MsgGUID: zQykWTUIT2y71MkGjKXBCQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224667" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:10 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 24/28] drm/xe: Basic SVM BO eviction Date: Tue, 27 Aug 2024 19:48:57 -0700 Message-Id: <20240828024901.2582335-25-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Wire xe_bo_move to GPUSVM migration to SRAM with trylocking of mmap lock. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_bo.c | 35 +++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_bo_types.h | 3 +++ drivers/gpu/drm/xe/xe_svm.c | 2 ++ drivers/gpu/drm/xe/xe_svm.h | 13 ++++++++++++ 4 files changed, 52 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index ad804b6f9e84..ae71fcbe5380 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -25,6 +25,7 @@ #include "xe_pm.h" #include "xe_preempt_fence.h" #include "xe_res_cursor.h" +#include "xe_svm.h" #include "xe_trace_bo.h" #include "xe_ttm_stolen_mgr.h" #include "xe_vm.h" @@ -250,6 +251,8 @@ int xe_bo_placement_for_flags(struct xe_device *xe, struct xe_bo *bo, static void xe_evict_flags(struct ttm_buffer_object *tbo, struct ttm_placement *placement) { + struct xe_bo *bo; + if (!xe_bo_is_xe_bo(tbo)) { /* Don't handle scatter gather BOs */ if (tbo->type == ttm_bo_type_sg) { @@ -261,6 +264,12 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo, return; } + bo = ttm_to_xe_bo(tbo); + if (bo->flags & XE_BO_FLAG_SYSTEM_ALLOC) { + *placement = sys_placement; + return; + } + /* * For xe, sg bos that are evicted to system just triggers a * rebind of the sg list upon subsequent validation to XE_PL_TT. @@ -758,6 +767,17 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, } } + if (!move_lacks_source && (bo->flags & XE_BO_FLAG_SYSTEM_ALLOC) && + new_mem->mem_type == XE_PL_SYSTEM) { + ret = xe_svm_range_evict(bo->range); + if (!ret) { + drm_dbg(&xe->drm, "Evict system allocator BO success\n"); + ttm_bo_move_null(ttm_bo, new_mem); + } + + goto out; + } + if (!move_lacks_source && ((old_mem_type == XE_PL_SYSTEM && resource_is_vram(new_mem)) || (mem_type_is_vram(old_mem_type) && @@ -1096,6 +1116,19 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } +static bool xe_bo_eviction_valuable(struct ttm_buffer_object *ttm_bo, + const struct ttm_place *place) +{ + struct xe_bo *bo = ttm_to_xe_bo(ttm_bo); + + /* Do not evict SVMs before having a binding */ + if (bo->flags & XE_BO_FLAG_SYSTEM_ALLOC && + !xe_svm_range_has_vram_binding(bo->range)) + return false; + + return ttm_bo_eviction_valuable(ttm_bo, place); +} + const struct ttm_device_funcs xe_ttm_funcs = { .ttm_tt_create = xe_ttm_tt_create, .ttm_tt_populate = xe_ttm_tt_populate, @@ -1106,7 +1139,7 @@ const struct ttm_device_funcs xe_ttm_funcs = { .io_mem_reserve = xe_ttm_io_mem_reserve, .io_mem_pfn = xe_ttm_io_mem_pfn, .release_notify = xe_ttm_bo_release_notify, - .eviction_valuable = ttm_bo_eviction_valuable, + .eviction_valuable = xe_bo_eviction_valuable, .delete_mem_notify = xe_ttm_bo_delete_mem_notify, }; diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h index 2ed558ac2264..4523b033417c 100644 --- a/drivers/gpu/drm/xe/xe_bo_types.h +++ b/drivers/gpu/drm/xe/xe_bo_types.h @@ -16,6 +16,7 @@ #include "xe_ggtt_types.h" struct xe_device; +struct xe_svm_range; struct xe_vm; #define XE_BO_MAX_PLACEMENTS 3 @@ -47,6 +48,8 @@ struct xe_bo { struct ttm_bo_kmap_obj kmap; /** @pinned_link: link to present / evicted list of pinned BO */ struct list_head pinned_link; + /** @range: SVM range for BO */ + struct xe_svm_range *range; #ifdef CONFIG_PROC_FS /** * @client: @xe_drm_client which created the bo diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index fd8987e0a506..dc9810828c0a 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -531,6 +531,8 @@ static struct xe_bo *xe_svm_alloc_vram(struct xe_vm *vm, struct xe_tile *tile, range->base.va.start, ttm_bo_type_device, XE_BO_FLAG_VRAM_IF_DGFX(tile) | XE_BO_FLAG_SYSTEM_ALLOC | XE_BO_FLAG_SKIP_CLEAR); + if (!IS_ERR(bo)) + bo->range = range; xe_vm_unlock(vm); if (IS_ERR(bo)) { err = PTR_ERR(bo); diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 3f432483a230..b9cf0e2500da 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -46,6 +46,19 @@ static inline bool xe_svm_range_has_dma_mapping(struct xe_svm_range *range) return range->base.flags.has_dma_mapping; } +static inline bool xe_svm_range_has_vram_binding(struct xe_svm_range *range) +{ + return xe_svm_range_in_vram(range) && range->tile_present; +} + +static inline int xe_svm_range_evict(struct xe_svm_range *range) +{ + struct drm_gpusvm_ctx ctx = { .trylock_mmap = true, }; + + return drm_gpusvm_migrate_to_sram(range->base.gpusvm, &range->base, + &ctx); +} + #define xe_svm_notifier_lock(vm__) \ drm_gpusvm_notifier_lock(&(vm__)->svm.gpusvm) From patchwork Wed Aug 28 02:48:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AAE5EC5474D for ; Wed, 28 Aug 2024 02:48:34 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 35A7010E479; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="n3Mu/Y70"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5A12010E450; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813291; x=1756349291; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iJEAfp1tAgOY2YyFFIOULX7IqCWYUTyYRhmsLRAPbpA=; b=n3Mu/Y70nrdfZLSZEDsSZ/t2jRqfmVNpmS6EImTnR6t1ZbvdjmsE8pvn 9o5UarqQhEwrqkFtZXoabg8QiucKenlLFMViuq34ayqZOBIahgIMh0X9G fndzRJW2Loi1SjDEXtE/Bjr3a2+xaIDzSXNk3SCUVJYYomQL0Zx2AfSYA pwrItubalukAw7NVZygm0t5oxpLji6fck4RaI380k+bD2qAM7CFL6C7cX Vn9f9uktPiaSa5+3Dszez8qJrzPmuoEwqwYmzOxjda4HE5JGLyhEcal0O HRIOdPne2cJgwNF8Lrmm2ieaTHsq0of5k5LgA49NBo0SrPEy9EDLe7aEV A==; X-CSE-ConnectionGUID: q9sTCFGNQE+PbBV2OHewyg== X-CSE-MsgGUID: JI7mXSB/QPOo7nkawVSzoA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251950" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251950" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 X-CSE-ConnectionGUID: M6MbO7anTYSV9n87pELQWg== X-CSE-MsgGUID: lFFscOVAQ7SSZcaCASAqFA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224671" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 25/28] drm/xe: Add SVM debug Date: Tue, 27 Aug 2024 19:48:58 -0700 Message-Id: <20240828024901.2582335-26-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add some useful SVM debug logging. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_pt.c | 13 ++++-- drivers/gpu/drm/xe/xe_svm.c | 93 ++++++++++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_svm.h | 2 + 3 files changed, 93 insertions(+), 15 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index e9195029ea60..e31af84ceb32 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -678,16 +678,20 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, xe_bo_assert_held(bo); if (range) { - if (is_devmem) + if (is_devmem) { + xe_svm_range_debug(range, "BIND PREPARE - VRAM"); xe_res_first(bo->ttm.resource, 0, range->base.va.end - range->base.va.start, &curs); - else if (xe_svm_range_has_dma_mapping(range)) + } else if (xe_svm_range_has_dma_mapping(range)) { + xe_svm_range_debug(range, "BIND PREPARE - DMA"); xe_res_first_dma(range->base.dma_addr, 0, range->base.va.end - range->base.va.start, range->base.order, &curs); - else + } else { + xe_svm_range_debug(range, "BIND PREPARE - RETRY"); return -EAGAIN; /* Invalidation corner case */ + } } else if (!xe_vma_is_null(vma)) { if (xe_vma_is_userptr(vma)) xe_res_first_sg(to_userptr_vma(vma)->userptr.sg, 0, @@ -1387,10 +1391,13 @@ static int xe_pt_svm_pre_commit(struct xe_migrate_pt_update *pt_update) if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE) continue; + xe_svm_range_debug(range, "PRE-COMMIT"); + xe_assert(vm->xe, xe_vma_is_system_allocator(op->map_range.vma)); xe_assert(vm->xe, op->subop == XE_VMA_SUBOP_MAP_RANGE); if (!xe_svm_range_pages_valid(range)) { + xe_svm_range_debug(range, "PRE-COMMIT - RETRY"); xe_svm_notifier_unlock(vm); return -EAGAIN; } diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index dc9810828c0a..f9c2bffd1783 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -24,6 +24,23 @@ static struct xe_vm *range_to_vm(struct drm_gpusvm_range *r) return gpusvm_to_vm(r->gpusvm); } +#define range_debug(r__, operaton__) \ + vm_dbg(&range_to_vm(&(r__)->base)->xe->drm, \ + "%s: asid=%u, gpusvm=0x%016llx, vram=%d,%d,%d, seqno=%lu, order=%u, start=0x%014llx, end=0x%014llx, size=%llu", \ + (operaton__), range_to_vm(&(r__)->base)->usm.asid, \ + (u64)(r__)->base.gpusvm, \ + (r__)->base.vram_allocation ? 1 : 0, \ + xe_svm_range_in_vram((r__)) ? 1 : 0, \ + xe_svm_range_has_vram_binding((r__)) ? 1 : 0, \ + (r__)->base.notifier_seq, (r__)->base.order, \ + (r__)->base.va.start, (r__)->base.va.end, \ + (r__)->base.va.end - (r__)->base.va.start) + +void xe_svm_range_debug(struct xe_svm_range *range, const char *operation) +{ + range_debug(range, operation); +} + static void *xe_svm_devm_owner(struct xe_device *xe) { return xe; @@ -61,6 +78,8 @@ xe_svm_garbage_collector_add_range(struct xe_vm *vm, struct xe_svm_range *range, { struct xe_device *xe = vm->xe; + range_debug(range, "GARBAGE COLLECTOR ADD"); + drm_gpusvm_range_set_unmapped(&range->base, mmu_range); spin_lock(&vm->svm.garbage_collector.lock); @@ -84,10 +103,14 @@ xe_svm_range_notifier_event_begin(struct xe_vm *vm, struct drm_gpusvm_range *r, u8 tile_mask = 0; u8 id; + range_debug(range, "NOTIFIER"); + /* Skip if already unmapped or if no binding exist */ if (range->base.flags.unmapped || !range->tile_present) return 0; + range_debug(range, "NOTIFIER - EXECUTE"); + /* Adjust invalidation to range boundaries */ if (range->base.va.start < mmu_range->start) *adj_start = range->base.va.start; @@ -136,6 +159,11 @@ static void xe_svm_invalidate(struct drm_gpusvm *gpusvm, u32 fence_id = 0; long err; + vm_dbg(&gpusvm_to_vm(gpusvm)->xe->drm, + "INVALIDATE: asid=%u, gpusvm=0x%016llx, seqno=%lu, start=0x%016lx, end=0x%016lx, event=%d", + vm->usm.asid, (u64)gpusvm, notifier->notifier.invalidate_seq, + mmu_range->start, mmu_range->end, mmu_range->event); + /* Adjust invalidation to notifier boundaries */ if (adj_start < notifier->interval.start) adj_start = notifier->interval.start; @@ -220,9 +248,13 @@ static int __xe_svm_garbage_collector(struct xe_vm *vm, struct drm_gpusvm_ctx ctx = {}; struct dma_fence *fence; + range_debug(range, "GARBAGE COLLECTOR"); + /* Evict any pages holding references to vram allocation */ - if (range->base.flags.partial_unmap && IS_DGFX(vm->xe)) + if (range->base.flags.partial_unmap && IS_DGFX(vm->xe)) { + range_debug(range, "GARBAGE COLLECTOR - EVICT"); drm_gpusvm_migrate_to_sram(&vm->svm.gpusvm, &range->base, &ctx); + } xe_vm_lock(vm, false); fence = xe_vm_range_unbind(vm, range); @@ -358,16 +390,25 @@ static int xe_svm_copy(struct drm_gpusvm *gpusvm, struct page **pages, int incr = (match && last) ? 1 : 0; if (vram_addr != VRAM_ADDR_INVALID) { - if (sram) + if (sram) { + vm_dbg(&gpusvm_to_vm(gpusvm)->xe->drm, + "COPY TO SRAM - 0x%016llx -> 0x%016llx, NPAGES=%ld, asid=%u, gpusvm=0x%016llx", + vram_addr, dma_addr[pos], i - pos + incr, + vm->usm.asid, (u64)gpusvm); __fence = xe_migrate_from_vram(tile->migrate, i - pos + incr, vram_addr, dma_addr + pos); - else + } else { + vm_dbg(&gpusvm_to_vm(gpusvm)->xe->drm, + "COPY TO VRAM - 0x%016llx -> 0x%016llx, NPAGES=%ld, asid=%u, gpusvm=0x%016llx", + dma_addr[pos], vram_addr, i - pos + incr, + vm->usm.asid, (u64)gpusvm); __fence = xe_migrate_to_vram(tile->migrate, i - pos + incr, dma_addr + pos, vram_addr); + } if (IS_ERR(__fence)) { err = PTR_ERR(__fence); goto err_out; @@ -385,14 +426,23 @@ static int xe_svm_copy(struct drm_gpusvm *gpusvm, struct page **pages, } if (!match && last && dma_addr[i]) { - if (sram) + if (sram) { + vm_dbg(&gpusvm_to_vm(gpusvm)->xe->drm, + "COPY TO SRAM - 0x%016llx -> 0x%016llx, NPAGES=%d, asid=%u, gpusvm=0x%016llx", + vram_addr, dma_addr[pos], 1, + vm->usm.asid, (u64)gpusvm); __fence = xe_migrate_from_vram(tile->migrate, 1, vram_addr, dma_addr + pos); - else + } else { + vm_dbg(&gpusvm_to_vm(gpusvm)->xe->drm, + "COPY TO VRAM - 0x%016llx -> 0x%016llx, NPAGES=%d, asid=%u, gpusvm=0x%016llx", + dma_addr[pos], vram_addr, 1, + vm->usm.asid, (u64)gpusvm); __fence = xe_migrate_to_vram(tile->migrate, 1, dma_addr + pos, vram_addr); + } if (IS_ERR(__fence)) { err = PTR_ERR(__fence); goto err_out; @@ -519,12 +569,14 @@ static struct xe_bo *xe_svm_alloc_vram(struct xe_vm *vm, struct xe_tile *tile, const struct drm_gpusvm_ctx *ctx) { struct xe_mem_region *mr = tile_to_mr(tile); + struct drm_buddy *buddy = tile_to_buddy(tile); struct drm_buddy_block *block; struct list_head *blocks; struct xe_bo *bo; ktime_t end = 0; int err; + range_debug(range, "ALLOCATE VRAM"); retry: xe_vm_lock(vm, false); bo = xe_bo_create(tile_to_xe(tile), tile, vm, range->base.va.end - @@ -542,8 +594,13 @@ static struct xe_bo *xe_svm_alloc_vram(struct xe_vm *vm, struct xe_tile *tile, } blocks = &to_xe_ttm_vram_mgr_resource(bo->ttm.resource)->blocks; - list_for_each_entry(block, blocks, link) + list_for_each_entry(block, blocks, link) { + vm_dbg(&vm->xe->drm, "ALLOC VRAM: asid=%u, gpusvm=0x%016llx, pfn=%llu, npages=%llu", + vm->usm.asid, (u64)&vm->svm.gpusvm, + block_offset_to_pfn(mr, drm_buddy_block_offset(block)), + drm_buddy_block_size(buddy, block) >> PAGE_SHIFT); block->private = mr; + } /* * Take ref because as soon as drm_gpusvm_migrate_to_vram succeeds the @@ -597,6 +654,8 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, if (xe_svm_range_is_valid(range, tile)) return 0; + range_debug(range, "PAGE FAULT"); + /* XXX: Add migration policy, for now migrate range once */ if (IS_DGFX(vm->xe) && !range->migrated && range->base.flags.migrate_vram && @@ -606,18 +665,26 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, bo = xe_svm_alloc_vram(vm, tile, range, &ctx); if (IS_ERR(bo)) { drm_info(&vm->xe->drm, - "VRAM allocation failed, falling back to retrying, asid=%u, errno %ld\n", - vm->usm.asid, PTR_ERR(bo)); + "VRAM allocation failed, falling back to retrying, asid=%u, gpusvm=0x%016llx, errno %ld\n", + vm->usm.asid, (u64)&vm->svm.gpusvm, + PTR_ERR(bo)); bo = NULL; goto retry; } } + range_debug(range, "GET PAGES"); err = drm_gpusvm_range_get_pages(&vm->svm.gpusvm, r, &ctx); - if (err == -EFAULT || err == -EPERM) /* Corner where CPU mappings have change */ - goto retry; - if (err) + if (err == -EFAULT || err == -EPERM) { /* Corner where CPU mappings have change */ + range_debug(range, "PAGE FAULT - RETRY PAGES"); + goto retry; + } + if (err) { + range_debug(range, "PAGE FAULT - FAIL PAGE COLLECT"); goto err_out; + } + + range_debug(range, "PAGE FAULT - BIND"); retry_bind: drm_exec_init(&exec, 0, 0); @@ -633,8 +700,10 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, if (IS_ERR(fence)) { drm_exec_fini(&exec); err = PTR_ERR(fence); - if (err == -EAGAIN) + if (err == -EAGAIN) { + range_debug(range, "PAGE FAULT - RETRY BIND"); goto retry; + } if (xe_vm_validate_should_retry(&exec, err, &end)) goto retry_bind; goto err_out; diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index b9cf0e2500da..1ea5d29a6868 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -31,6 +31,8 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, bool atomic); bool xe_svm_has_mapping(struct xe_vm *vm, u64 start, u64 end); +void xe_svm_range_debug(struct xe_svm_range *range, const char *operation); + static inline bool xe_svm_range_pages_valid(struct xe_svm_range *range) { return drm_gpusvm_range_pages_valid(range->base.gpusvm, &range->base); From patchwork Wed Aug 28 02:48:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D11DC5474B for ; Wed, 28 Aug 2024 02:48:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A990810E46C; Wed, 28 Aug 2024 02:48:20 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Nq4C9X5X"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9EA4310E456; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813292; x=1756349292; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=S2usf0NmcJM5c8jN/Hogz1fEnGMWjfMR8DKBwZqNbx4=; b=Nq4C9X5XBY/2/noHGQosm+1MPX3VROfAmdGv+vPjbVz5SpgXkLA5YX5h tXUInxW+vEeJ/YPAfBt9KjB1J0nCHak7SsBoEs5bMcrqOQzrO0wYB3TOm 29y929DvVbYGq+RLuBbeGHuUDwCAvGun41GWZY3uHfLcBkxyw8xDMtdw4 x7UvYOxYzt1PN/qYro5J4TDqjAU71p0WUn91tokwcl5RAPqPy+vsjVHcU QlOIxb2W5kcILW/4LL0UpBHu6q4lX1sFaQYAHYsIF/G16sS42L3uNOd5q /GmryhjYnRPqoLYvvlrFGesrnZLojNmTd4OzX+gi6w1fTbzUdm2nhJirf g==; X-CSE-ConnectionGUID: RGeKOj7aRFmEtEWhqPv9Aw== X-CSE-MsgGUID: NLzdC5UTRIuneOM3AsF7EA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251954" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251954" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 X-CSE-ConnectionGUID: Nx8P1ThZQmqTATxQoGjOSA== X-CSE-MsgGUID: P5G3nnI1SvGjA2nt777iCA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224674" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 26/28] drm/xe: Add modparam for SVM notifier size Date: Tue, 27 Aug 2024 19:48:59 -0700 Message-Id: <20240828024901.2582335-27-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Useful to experiment with notifier size and how it affects performance. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_module.c | 4 ++++ drivers/gpu/drm/xe/xe_module.h | 1 + drivers/gpu/drm/xe/xe_svm.c | 5 +++-- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_module.c b/drivers/gpu/drm/xe/xe_module.c index 923460119cec..30cfb76344a1 100644 --- a/drivers/gpu/drm/xe/xe_module.c +++ b/drivers/gpu/drm/xe/xe_module.c @@ -22,9 +22,13 @@ struct xe_modparam xe_modparam = { .max_vfs = IS_ENABLED(CONFIG_DRM_XE_DEBUG) ? ~0 : 0, #endif .wedged_mode = 1, + .svm_notifier_size = 512, /* the rest are 0 by default */ }; +module_param_named(svm_notifier_size, xe_modparam.svm_notifier_size, uint, 0600); +MODULE_PARM_DESC(svm_notifier_size, "Set the svm notifier size(in MiB), must be pow2"); + module_param_named_unsafe(force_execlist, xe_modparam.force_execlist, bool, 0444); MODULE_PARM_DESC(force_execlist, "Force Execlist submission"); diff --git a/drivers/gpu/drm/xe/xe_module.h b/drivers/gpu/drm/xe/xe_module.h index 161a5e6f717f..5a3bfea8b7b4 100644 --- a/drivers/gpu/drm/xe/xe_module.h +++ b/drivers/gpu/drm/xe/xe_module.h @@ -22,6 +22,7 @@ struct xe_modparam { unsigned int max_vfs; #endif int wedged_mode; + u32 svm_notifier_size; }; extern struct xe_modparam xe_modparam; diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index f9c2bffd1783..5e2ec25c3cb2 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -8,6 +8,7 @@ #include "xe_bo.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_module.h" #include "xe_pt.h" #include "xe_svm.h" #include "xe_ttm_vram_mgr.h" @@ -543,8 +544,8 @@ int xe_svm_init(struct xe_vm *vm) return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM", &vm->xe->drm, current->mm, xe_svm_devm_owner(vm->xe), 0, - vm->size, SZ_512M, &gpusvm_ops, - fault_chunk_sizes, + vm->size, xe_modparam.svm_notifier_size * SZ_1M, + &gpusvm_ops, fault_chunk_sizes, ARRAY_SIZE(fault_chunk_sizes)); } From patchwork Wed Aug 28 02:49:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DB77C54749 for ; Wed, 28 Aug 2024 02:48:35 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E482710E473; Wed, 28 Aug 2024 02:48:19 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="KMjtA/HJ"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id C288010E457; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813292; x=1756349292; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Vx+8VWEdGezuA95bABYtF6TDAgHg9iYCZ0igRn4uPGI=; b=KMjtA/HJD37cYXucxboGaroOFzUE//oKc36E/QjrS7PUtxmHGMRLondL KFon94iclv1w4no+zMekxBFx8axwQZZZ/GYvMHqbNAZZh6odN8nm+73W5 okhELzcaNaoR8xX05F/pUP9xYwchiTa4PTNH5RSd8ultpeu/NBxeyB7U/ qbmcdmSiPcBvilK7WzZ9L73GPtUHbT2Dy2xzC/8yIxzh+H76GoZ8kIAiZ eLdtOpXpSvVpvQZ/mqPnzKN0e2DNq7Ww7INOvsAVycmTepDB54Ttxpwzj MeOey7NRKSkw0/T/O0d+OOz8HJ/kEFORcG3xLcUdXtnZ0zHE144ZjNs2f Q==; X-CSE-ConnectionGUID: Nq2BaDVGQwKUqW8n/ZKs7Q== X-CSE-MsgGUID: 3eV5EQc+Tn2JXhEeT5rXOA== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251959" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251959" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 X-CSE-ConnectionGUID: FL1tGSn/SEya0OVl8qN9cg== X-CSE-MsgGUID: 55MfEsFMSaa1naPUH44qBQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224677" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 27/28] drm/xe: Add modparam for SVM prefault Date: Tue, 27 Aug 2024 19:49:00 -0700 Message-Id: <20240828024901.2582335-28-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Useful to experiment with SVM prefault and how it affects performance. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_module.c | 3 +++ drivers/gpu/drm/xe/xe_module.h | 1 + drivers/gpu/drm/xe/xe_svm.c | 2 ++ 3 files changed, 6 insertions(+) diff --git a/drivers/gpu/drm/xe/xe_module.c b/drivers/gpu/drm/xe/xe_module.c index 30cfb76344a1..edda9898f3cf 100644 --- a/drivers/gpu/drm/xe/xe_module.c +++ b/drivers/gpu/drm/xe/xe_module.c @@ -29,6 +29,9 @@ struct xe_modparam xe_modparam = { module_param_named(svm_notifier_size, xe_modparam.svm_notifier_size, uint, 0600); MODULE_PARM_DESC(svm_notifier_size, "Set the svm notifier size(in MiB), must be pow2"); +module_param_named(svm_prefault, xe_modparam.svm_prefault, bool, 0444); +MODULE_PARM_DESC(svm_prefault, "SVM prefault CPU pages upon range allocation"); + module_param_named_unsafe(force_execlist, xe_modparam.force_execlist, bool, 0444); MODULE_PARM_DESC(force_execlist, "Force Execlist submission"); diff --git a/drivers/gpu/drm/xe/xe_module.h b/drivers/gpu/drm/xe/xe_module.h index 5a3bfea8b7b4..c1571cd8f9fe 100644 --- a/drivers/gpu/drm/xe/xe_module.h +++ b/drivers/gpu/drm/xe/xe_module.h @@ -12,6 +12,7 @@ struct xe_modparam { bool force_execlist; bool probe_display; + bool svm_prefault; u32 force_vram_bar_size; int guc_log_level; char *guc_firmware_path; diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index 5e2ec25c3cb2..8e80e8704534 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -645,9 +645,11 @@ int xe_svm_handle_pagefault(struct xe_vm *vm, struct xe_vma *vma, if (err) return err; + ctx.prefault = xe_modparam.svm_prefault; r = drm_gpusvm_range_find_or_insert(&vm->svm.gpusvm, fault_addr, xe_vma_start(vma), xe_vma_end(vma), &ctx); + ctx.prefault = false; if (IS_ERR(r)) return PTR_ERR(r); From patchwork Wed Aug 28 02:49:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13780328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B5FBC5474F for ; Wed, 28 Aug 2024 02:48:32 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 91F6E10E472; Wed, 28 Aug 2024 02:48:19 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="hS5SSAER"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id EC8A410E456; Wed, 28 Aug 2024 02:48:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724813292; x=1756349292; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oDvqa61JpMnQYg9Ts2SQo7j70AwY6QthuLbaa4R1yJA=; b=hS5SSAERhY+dEasXZWXLMqw6zEqqYPp9hcKU+zp/j/Ye7jrdU4RONqQj LDeyZFfpq91vA+hRbFyhOcUKSraOxbdZ2lXUVIQ+XkDQLI/OfY5D/cvlA aTFf/8QeinIMa9qErZXF1JhsqIu3vIYz/XFoFjKXqodVh+3bxKqq4buIG fgxgJex+BRlmbXKz+cvs9/9yIfardegVe3y1rwqdrPB4AOFmrJfVjYVwM /7MdG6AM5ZnhHyiY5/LUEvq6b+nhpssV2WOIokdHWG8A3f0CJ3QeEnVfk AMINqycrlyj6ruudsuH6RY7SmsylOhqP507ZHVHPunp8CLwwa5v7A+SEy A==; X-CSE-ConnectionGUID: QjlQWw35S0W4cSk1ZVpbNg== X-CSE-MsgGUID: 3TpFzGnSTDmd77bxTRM6EQ== X-IronPort-AV: E=McAfee;i="6700,10204,11177"; a="13251963" X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="13251963" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:12 -0700 X-CSE-ConnectionGUID: BT1Yt9icTDST+Q6BS0qKRQ== X-CSE-MsgGUID: tP1BpzN9TM+Tn9Gt27335g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,181,1719903600"; d="scan'208";a="67224680" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2024 19:48:11 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: airlied@gmail.com, christian.koenig@amd.com, thomas.hellstrom@linux.intel.com, matthew.auld@intel.com, daniel@ffwll.ch Subject: [RFC PATCH 28/28] drm/gpusvm: Ensure all pages migrated upon eviction Date: Tue, 27 Aug 2024 19:49:01 -0700 Message-Id: <20240828024901.2582335-29-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240828024901.2582335-1-matthew.brost@intel.com> References: <20240828024901.2582335-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Let's make sure we know what we are doing and check to ensure all pages are migrated upon eviction. Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/drm_gpusvm.c | 39 +++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/drivers/gpu/drm/xe/drm_gpusvm.c b/drivers/gpu/drm/xe/drm_gpusvm.c index fc1e44e6ae72..6df3580cf4ca 100644 --- a/drivers/gpu/drm/xe/drm_gpusvm.c +++ b/drivers/gpu/drm/xe/drm_gpusvm.c @@ -1830,6 +1830,40 @@ static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas, return 0; } +#define DRM_GPUSVM_DEBUG /* TODO: Connect to Kconfig */ + +#ifdef DRM_GPUSVM_DEBUG +/** + * drm_gpusvm_pages_migrated - count the number of pages migrated + * @src_pfns: source migration pfns + * @npages the total number of pages in src_pfns + * + * Examine the MIGRATE_PFN_MIGRATE bit of each sfn_pfn to get a count of the + * number of pages migrated. + * + * Returns: + * Number of pages migrated + */ +static unsigned long +drm_gpusvm_pages_migrated(unsigned long *src_pfns, unsigned long npages) +{ + int pages_migrated = 0; + unsigned long i; + + for (i = 0; i < npages; ++i) + if (src_pfns[i] && src_pfns[i] & MIGRATE_PFN_MIGRATE) + ++pages_migrated; + + return pages_migrated; +} +#else +static unsigned long +drm_gpusvm_pages_migrated(unsigned long *src_pfns, unsigned long npages) +{ + return npages; +} +#endif + /** * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM * @gpusvm: Pointer to the GPU SVM structure @@ -1896,6 +1930,8 @@ static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm, if (err) drm_gpusvm_migration_put_pages(npages, dst); migrate_device_pages(src, dst, npages); + if (!err) + WARN_ON(npages > drm_gpusvm_pages_migrated(src, npages)); migrate_device_finalize(src, dst, npages); drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages, DMA_BIDIRECTIONAL); @@ -1994,6 +2030,9 @@ static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm, if (err) drm_gpusvm_migration_put_pages(npages, migrate.dst); migrate_vma_pages(&migrate); + if (!err && !page) /* Only check on eviction */ + WARN_ON(migrate.cpages > + drm_gpusvm_pages_migrated(migrate.src, npages)); migrate_vma_finalize(&migrate); drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages, DMA_BIDIRECTIONAL);