From patchwork Thu Nov 2 04:32:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C93D0C4332F for ; Thu, 2 Nov 2023 04:24:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 25C2210E7FD; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 18DF510E7F9; Thu, 2 Nov 2023 04:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899071; x=1730435071; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=633an+sC7R5RcLrkrKMH9N1hMZcmLC3khbxtaiNAfjY=; b=eNGUnQXuieCftfuy8xWv4BX82cE2JnMDTXK0KY7YRrN7ccsNzW8sR++j M9wLPHCGnV3UqBI4VmbFq36PjVQWlx4Omm4VEwMNlCUwtjxM9wOkREQoO s8r3K6jW7F5pgkMgUWQ8pRANPvRveTwZNiaDQjDV+DaJv4+08reZTuOZh FJaivSe9WPn1diApJFH5u04+0JQFMOdZDdF2od60RY7La0zwslcAFHUoL xhC6xE7ZOfBfCV4Lc2JRUoqFBnb3RGzwGugrt2/pVZTjD91xcNm2gfbpn 7MtNTabL1RKhaYNZBQkv4uPwfouEh5Wz5l5MUKAWVoE9AZuyokzLEmVA4 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930901" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930901" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762613" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762613" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 01/11] drm/ttm: re-parameter ttm_device_init Date: Thu, 2 Nov 2023 00:32:56 -0400 Message-Id: <20231102043306.2931989-2-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Change the 3rd parameter of ttm_device_init from struct device * to struct drm_device *. This is a prepare work for moving lru_lock from ttm_device to drm_device. Signed-off-by: Oak Zeng --- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/drm_gem_vram_helper.c | 2 +- drivers/gpu/drm/i915/intel_region_ttm.c | 2 +- drivers/gpu/drm/loongson/lsdc_ttm.c | 2 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 2 +- drivers/gpu/drm/radeon/radeon_ttm.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 7 ++++--- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 2 +- drivers/gpu/drm/xe/xe_device.c | 2 +- include/drm/ttm/ttm_device.h | 3 ++- 11 files changed, 15 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 4e51dce3aab5..5cdbc901cbe2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -1817,7 +1817,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) mutex_init(&adev->mman.gtt_window_lock); /* No others user of address space so set it to 0 */ - r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev->dev, + r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev_to_drm(adev), adev_to_drm(adev)->anon_inode->i_mapping, adev_to_drm(adev)->vma_offset_manager, adev->need_swiotlb, diff --git a/drivers/gpu/drm/drm_gem_vram_helper.c b/drivers/gpu/drm/drm_gem_vram_helper.c index b67eafa55715..56749e40459f 100644 --- a/drivers/gpu/drm/drm_gem_vram_helper.c +++ b/drivers/gpu/drm/drm_gem_vram_helper.c @@ -1002,7 +1002,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev, vmm->vram_base = vram_base; vmm->vram_size = vram_size; - ret = ttm_device_init(&vmm->bdev, &bo_driver, dev->dev, + ret = ttm_device_init(&vmm->bdev, &bo_driver, dev, dev->anon_inode->i_mapping, dev->vma_offset_manager, false, true); diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c index bf6097e7433d..b845782c9859 100644 --- a/drivers/gpu/drm/i915/intel_region_ttm.c +++ b/drivers/gpu/drm/i915/intel_region_ttm.c @@ -33,7 +33,7 @@ int intel_region_ttm_device_init(struct drm_i915_private *dev_priv) struct drm_device *drm = &dev_priv->drm; return ttm_device_init(&dev_priv->bdev, i915_ttm_driver(), - drm->dev, drm->anon_inode->i_mapping, + drm, drm->anon_inode->i_mapping, drm->vma_offset_manager, false, false); } diff --git a/drivers/gpu/drm/loongson/lsdc_ttm.c b/drivers/gpu/drm/loongson/lsdc_ttm.c index bf79dc55afa4..bd68cb9366b5 100644 --- a/drivers/gpu/drm/loongson/lsdc_ttm.c +++ b/drivers/gpu/drm/loongson/lsdc_ttm.c @@ -548,7 +548,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev) unsigned long num_gtt_pages; int ret; - ret = ttm_device_init(&ldev->bdev, &lsdc_bo_driver, ddev->dev, + ret = ttm_device_init(&ldev->bdev, &lsdc_bo_driver, ddev, ddev->anon_inode->i_mapping, ddev->vma_offset_manager, false, true); if (ret) diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c index 486f39f31a38..831918437850 100644 --- a/drivers/gpu/drm/nouveau/nouveau_ttm.c +++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c @@ -299,7 +299,7 @@ nouveau_ttm_init(struct nouveau_drm *drm) drm->agp.cma = pci->agp.cma; } - ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, drm->dev->dev, + ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, dev, dev->anon_inode->i_mapping, dev->vma_offset_manager, drm_need_swiotlb(drm->client.mmu.dmabits), diff --git a/drivers/gpu/drm/radeon/radeon_ttm.c b/drivers/gpu/drm/radeon/radeon_ttm.c index 4eb83ccc4906..77ca50187162 100644 --- a/drivers/gpu/drm/radeon/radeon_ttm.c +++ b/drivers/gpu/drm/radeon/radeon_ttm.c @@ -688,7 +688,7 @@ int radeon_ttm_init(struct radeon_device *rdev) int r; /* No others user of address space so set it to 0 */ - r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->dev, + r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->ddev, rdev->ddev->anon_inode->i_mapping, rdev->ddev->vma_offset_manager, rdev->need_swiotlb, diff --git a/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c b/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c index 81661d8827aa..63eb6fdc3460 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c +++ b/drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c @@ -16,7 +16,7 @@ int ttm_device_kunit_init(struct ttm_test_devices *priv, struct drm_device *drm = priv->drm; int err; - err = ttm_device_init(ttm, &ttm_dev_funcs, drm->dev, + err = ttm_device_init(ttm, &ttm_dev_funcs, drm, drm->anon_inode->i_mapping, drm->vma_offset_manager, use_dma_alloc, use_dma32); diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 7726a72befc5..12014788b595 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -179,7 +179,7 @@ EXPORT_SYMBOL(ttm_device_swapout); * * @bdev: A pointer to a struct ttm_device to initialize. * @funcs: Function table for the device. - * @dev: The core kernel device pointer for DMA mappings and allocations. + * @drm: drm_device pointer * @mapping: The address space to use for this bo. * @vma_manager: A pointer to a vma manager. * @use_dma_alloc: If coherent DMA allocation API should be used. @@ -190,7 +190,7 @@ EXPORT_SYMBOL(ttm_device_swapout); * !0: Failure. */ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *funcs, - struct device *dev, struct address_space *mapping, + struct drm_device *drm, struct address_space *mapping, struct drm_vma_offset_manager *vma_manager, bool use_dma_alloc, bool use_dma32) { @@ -213,7 +213,8 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func bdev->funcs = funcs; ttm_sys_man_init(bdev); - ttm_pool_init(&bdev->pool, dev, NUMA_NO_NODE, use_dma_alloc, use_dma32); + ttm_pool_init(&bdev->pool, drm?drm->dev:NULL, NUMA_NO_NODE, + use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager; spin_lock_init(&bdev->lru_lock); diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index 8b24ecf60e3e..cf1c1f16102a 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -1047,7 +1047,7 @@ static int vmw_driver_load(struct vmw_private *dev_priv, u32 pci_id) } ret = ttm_device_init(&dev_priv->bdev, &vmw_bo_driver, - dev_priv->drm.dev, + &dev_priv->drm, dev_priv->drm.anon_inode->i_mapping, dev_priv->drm.vma_offset_manager, dev_priv->map_mode == vmw_dma_alloc_coherent, diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index a964dc205da9..3377bfd5b1a1 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -191,7 +191,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev, if (IS_ERR(xe)) return xe; - err = ttm_device_init(&xe->ttm, &xe_ttm_funcs, xe->drm.dev, + err = ttm_device_init(&xe->ttm, &xe_ttm_funcs, &xe->drm, xe->drm.anon_inode->i_mapping, xe->drm.vma_offset_manager, false, false); if (WARN_ON(err)) diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index c22f30535c84..bab868d55383 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -29,6 +29,7 @@ #include #include #include +#include struct ttm_device; struct ttm_placement; @@ -288,7 +289,7 @@ static inline void ttm_set_driver_manager(struct ttm_device *bdev, int type, } int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *funcs, - struct device *dev, struct address_space *mapping, + struct drm_device *drm, struct address_space *mapping, struct drm_vma_offset_manager *vma_manager, bool use_dma_alloc, bool use_dma32); void ttm_device_fini(struct ttm_device *bdev); From patchwork Thu Nov 2 04:32:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37948C4332F for ; Thu, 2 Nov 2023 04:24:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 13C4510E7FF; Thu, 2 Nov 2023 04:24:34 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5BEE010E7FA; Thu, 2 Nov 2023 04:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899071; x=1730435071; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HDjBY6VY37svVHJzhhAHXoSsugcD4N0Z6EUBPnYr33o=; b=VebJ5+1eOewBZo5IOtv+xe96u9b3dCtQcC2Cjz3r5pMPQI11jPyT760Q ivTVY3HqrksPa5WhGu5Juoi3OBctwCMnSOsiA8oUyo/3OmbNd7Akz6MiF NRez11bC8BBPwnoeJujsk+7O7Hd4qvPrUeC9zgcxWvocpy7hY5SsFc9hs 7kuemZRZ7ldxlLcx6LKBpC884zx11btrpy8PgZJiVzxxpYi10M47ZE7Kj QJYZjAcuLvwu18RSxQXCnSySygr6FA7MqPAOw34lynhPbx3R+key3aODA 2VdzfhOOu6pMPASWYA86jv5NrNkVnY7GOTo0xU+9qCjI5Zv+32MzL6TxQ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930904" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930904" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762616" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762616" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device Date: Thu, 2 Nov 2023 00:32:57 -0400 Message-Id: <20231102043306.2931989-3-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In the coming patches, we will share the lru list b/t ttm bo based memory allocator and hmm/svm based memory allocator. Thus lru_lock (which is used mainly to protect the lru list) is moved from struct ttm_device to struct drm_device, so this lock can be shared b/t those two memory allocators. To minimize code change, struct ttm_device still hold a weak reference of lru_lock, so ttm layer can still reference to this lock easily. Signed-off-by: Oak Zeng --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +- drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++---------- drivers/gpu/drm/ttm/ttm_device.c | 18 ++++----- drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++---------- drivers/gpu/drm/xe/xe_bo.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 4 +- drivers/gpu/drm/xe/xe_vm.c | 4 +- include/drm/drm_device.h | 5 +++ include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_device.h | 4 +- 13 files changed, 72 insertions(+), 66 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index f5daadcec865..747bcad86d5d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec, void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev, struct amdgpu_vm *vm) { - spin_lock(&adev->mman.bdev.lru_lock); + spin_lock(adev->mman.bdev.lru_lock); ttm_lru_bulk_move_tail(&vm->lru_bulk_move); - spin_unlock(&adev->mman.bdev.lru_lock); + spin_unlock(adev->mman.bdev.lru_lock); } /* Create scheduler entities for page table updates */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c index c7085a747b03..b83e1741905e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c @@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct ttm_resource_manager *man) vis_usage = amdgpu_vram_mgr_vis_size(adev, block); atomic64_add(vis_usage, &mgr->vis_usage); - spin_lock(&man->bdev->lru_lock); + spin_lock(man->bdev->lru_lock); man->usage += rsv->size; - spin_unlock(&man->bdev->lru_lock); + spin_unlock(man->bdev->lru_lock); list_move(&rsv->blocks, &mgr->reserved_pages); } } diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 3eda026ffac6..1943c38815aa 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev, INIT_LIST_HEAD(&dev->managed.resources); spin_lock_init(&dev->managed.lock); + spin_lock_init(&dev->lru_lock); /* no per-device feature limits by default */ dev->driver_features = ~0u; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c index 9227f8146a58..c46f54f83f54 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c @@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj) /* * Put on the correct LRU list depending on the MADV status */ - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); if (shrinkable) { /* Try to keep shmem_tt from being considered for shrinking. */ bo->priority = TTM_MAX_BO_PRIORITY - 1; @@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj) } ttm_bo_move_to_lru_tail(bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } /* diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index e58b7e249816..26e0555bad0c 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct ttm_buffer_object *bo, * @bo: The buffer object. * * Move this BO to the tail of all lru lists used to lookup and reserve an - * object. This function must be called with struct ttm_global::lru_lock + * object. This function must be called with struct drm_device::lru_lock * held, and is used to make a BO less likely to be considered for eviction. */ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo) @@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo, if (bo->bulk_move == bulk) return; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); if (bo->resource) ttm_resource_del_bulk_move(bo->resource, bo); bo->bulk_move = bulk; if (bo->resource) ttm_resource_add_bulk_move(bo->resource, bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } EXPORT_SYMBOL(ttm_bo_set_bulk_move); @@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo) * reference it any more. The only tricky case is the trylock on * the resv object while holding the lru_lock. */ - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); bo->base.resv = &bo->base._resv; - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } return r; @@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, if (unlock_resv) dma_resv_unlock(bo->base.resv); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP, interruptible, @@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, else if (lret == 0) return -EBUSY; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); if (unlock_resv && !dma_resv_trylock(bo->base.resv)) { /* * We raced, and lost, someone else holds the reservation now, @@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, * delayed destruction would succeed, so just return success * here. */ - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); return 0; } ret = 0; @@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, if (ret) { if (unlock_resv) dma_resv_unlock(bo->base.resv); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); return ret; } - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); ttm_bo_cleanup_memtype_use(bo); if (unlock_resv) @@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref) ttm_bo_flush_all_fences(bo); bo->deleted = true; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); /* * Make pinned bos immediately available to @@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref) } kref_init(&bo->kref); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); INIT_WORK(&bo->delayed_delete, ttm_bo_delayed_delete); queue_work(bdev->wq, &bo->delayed_delete); @@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, bool locked = false; int ret; - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); ttm_resource_manager_for_each_res(man, &cursor, res) { bool busy; @@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, if (!bo) { if (busy_bo && !ttm_bo_get_unless_zero(busy_bo)) busy_bo = NULL; - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket); if (busy_bo) ttm_bo_put(busy_bo); @@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, return ret; } - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); ret = ttm_bo_evict(bo, ctx); if (locked) @@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo) { dma_resv_assert_held(bo->base.resv); WARN_ON_ONCE(!kref_read(&bo->kref)); - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); if (bo->resource) ttm_resource_del_bulk_move(bo->resource, bo); ++bo->pin_count; - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } EXPORT_SYMBOL(ttm_bo_pin); @@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo) if (WARN_ON_ONCE(!bo->pin_count)) return; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); --bo->pin_count; if (bo->resource) ttm_resource_add_bulk_move(bo->resource, bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } EXPORT_SYMBOL(ttm_bo_unpin); @@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, } /* TODO: Cleanup the locking */ - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); /* * Move to system cached diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 12014788b595..d18eca86ebd6 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, unsigned i; int ret; - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) { man = ttm_manager_type(bdev, i); if (!man || !man->use_tt) @@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, return ret; } } - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); return 0; } EXPORT_SYMBOL(ttm_device_swapout); @@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager; - spin_lock_init(&bdev->lru_lock); + bdev->lru_lock = &drm->lru_lock; INIT_LIST_HEAD(&bdev->pinned); bdev->dev_mapping = mapping; mutex_lock(&ttm_global_mutex); @@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev) drain_workqueue(bdev->wq); destroy_workqueue(bdev->wq); - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) if (list_empty(&man->lru[0])) pr_debug("Swap list %d was clean\n", i); - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); ttm_pool_fini(&bdev->pool); ttm_global_release(); @@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, { struct ttm_resource *res; - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); while ((res = list_first_entry_or_null(list, typeof(*res), lru))) { struct ttm_buffer_object *bo = res->bo; @@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, continue; list_del_init(&res->lru); - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); if (bo->ttm) ttm_tt_unpopulate(bo->bdev, bo->ttm); ttm_bo_put(bo); - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); } - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); } void ttm_device_clear_dma_mappings(struct ttm_device *bdev) diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 46ff9c75bb12..6ada77f51fba 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init); * @bulk: bulk move structure * * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that - * resource order never changes. Should be called with &ttm_device.lru_lock held. + * resource order never changes. Should be called with &drm_device.lru_lock held. */ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk) { @@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk) if (!pos->first) continue; - lockdep_assert_held(&pos->first->bo->bdev->lru_lock); + lockdep_assert_held(pos->first->bo->bdev->lru_lock); dma_resv_assert_held(pos->first->bo->base.resv); dma_resv_assert_held(pos->last->bo->base.resv); @@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res) struct ttm_buffer_object *bo = res->bo; struct ttm_device *bdev = bo->bdev; - lockdep_assert_held(&bo->bdev->lru_lock); + lockdep_assert_held(bo->bdev->lru_lock); if (bo->pin_count) { list_move_tail(&res->lru, &bdev->pinned); @@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo, res->bo = bo; man = ttm_manager_type(bo->bdev, place->mem_type); - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); if (bo->pin_count) list_add_tail(&res->lru, &bo->bdev->pinned); else list_add_tail(&res->lru, &man->lru[bo->priority]); man->usage += res->size; - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } EXPORT_SYMBOL(ttm_resource_init); @@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager *man, { struct ttm_device *bdev = man->bdev; - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); list_del_init(&res->lru); man->usage -= res->size; - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); } EXPORT_SYMBOL(ttm_resource_fini); @@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo, if (ret) return ret; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); ttm_resource_add_bulk_move(*res_ptr, bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); return 0; } @@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res) if (!*res) return; - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); ttm_resource_del_bulk_move(*res, bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); man = ttm_manager_type(bo->bdev, (*res)->mem_type); man->func->free(man, *res); *res = NULL; @@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res, void ttm_resource_set_bo(struct ttm_resource *res, struct ttm_buffer_object *bo) { - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); res->bo = bo; - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } /** @@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, * Can't use standard list traversal since we're unlocking. */ - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { while (!list_empty(&man->lru[i])) { - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx, NULL); if (ret) return ret; - spin_lock(&bdev->lru_lock); + spin_lock(bdev->lru_lock); } } - spin_unlock(&bdev->lru_lock); + spin_unlock(bdev->lru_lock); spin_lock(&man->move_lock); fence = dma_fence_get(man->move); @@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man) { uint64_t usage; - spin_lock(&man->bdev->lru_lock); + spin_lock(man->bdev->lru_lock); usage = man->usage; - spin_unlock(&man->bdev->lru_lock); + spin_unlock(man->bdev->lru_lock); return usage; } EXPORT_SYMBOL(ttm_resource_manager_usage); @@ -502,7 +502,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man, { struct ttm_resource *res; - lockdep_assert_held(&man->bdev->lru_lock); + lockdep_assert_held(man->bdev->lru_lock); for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY; ++cursor->priority) @@ -526,7 +526,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor, struct ttm_resource *res) { - lockdep_assert_held(&man->bdev->lru_lock); + lockdep_assert_held(man->bdev->lru_lock); list_for_each_entry_continue(res, &man->lru[cursor->priority], lru) return res; diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index 25fdc04627ca..827f798cccc0 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct ttm_buffer_object *ttm_bo) * the ttm_bo refcount is zero at this point. So trylocking *should* * always succeed here, as long as we hold the lru lock. */ - spin_lock(&ttm_bo->bdev->lru_lock); + spin_lock(ttm_bo->bdev->lru_lock); locked = dma_resv_trylock(ttm_bo->base.resv); - spin_unlock(&ttm_bo->bdev->lru_lock); + spin_unlock(ttm_bo->bdev->lru_lock); XE_WARN_ON(!locked); return locked; diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c index 890fadb0a93e..dafebdfb2368 100644 --- a/drivers/gpu/drm/xe/xe_exec.c +++ b/drivers/gpu/drm/xe/xe_exec.c @@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file) xe_vm_reactivate_rebind(vm); if (!err && !xe_vm_no_dma_fences(vm)) { - spin_lock(&xe->ttm.lru_lock); + spin_lock(xe->ttm.lru_lock); ttm_lru_bulk_move_tail(&vm->lru_bulk_move); - spin_unlock(&xe->ttm.lru_lock); + spin_unlock(xe->ttm.lru_lock); } err_repin: diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index a6a0f17fec1d..44e038276d41 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct *w) #undef retry_required - spin_lock(&vm->xe->ttm.lru_lock); + spin_lock(vm->xe->ttm.lru_lock); ttm_lru_bulk_move_tail(&vm->lru_bulk_move); - spin_unlock(&vm->xe->ttm.lru_lock); + spin_unlock(vm->xe->ttm.lru_lock); /* Point of no return. */ arm_preempt_fences(vm, &preempt_fences); diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h index 7cf4afae2e79..d0b5f42786be 100644 --- a/include/drm/drm_device.h +++ b/include/drm/drm_device.h @@ -326,6 +326,11 @@ struct drm_device { */ struct list_head debugfs_list; + /** + * @lru_lock: Protection for the per manager LRU and destroy lists. + */ + spinlock_t lru_lock; + /* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED(CONFIG_DRM_LEGACY) diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h index 0223a41a64b2..49f32df32204 100644 --- a/include/drm/ttm/ttm_bo.h +++ b/include/drm/ttm/ttm_bo.h @@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo); static inline void ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo) { - spin_lock(&bo->bdev->lru_lock); + spin_lock(bo->bdev->lru_lock); ttm_bo_move_to_lru_tail(bo); - spin_unlock(&bo->bdev->lru_lock); + spin_unlock(bo->bdev->lru_lock); } static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo, diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index bab868d55383..4d29e96bd892 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -248,9 +248,9 @@ struct ttm_device { struct ttm_pool pool; /** - * @lru_lock: Protection for the per manager LRU and ddestroy lists. + * @lru_lock: Weak reference to drm_device::lru_lock. */ - spinlock_t lru_lock; + spinlock_t *lru_lock; /** * @pinned: Buffer objects which are pinned and so not on any LRU list. From patchwork Thu Nov 2 04:32:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 551D3C4167B for ; Thu, 2 Nov 2023 04:24:43 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 96C3210E7FB; Thu, 2 Nov 2023 04:24:33 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id CD3D410E7FA; Thu, 2 Nov 2023 04:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899071; x=1730435071; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HgTUWuzDIh3D+iOw2MX6Zq7vQjeVu0sfP+jd16v7JBI=; b=BY7T0Ydxv2E4bVI7Kyn7mIwbP63gAXD5/WMAMyleciV42f5bje0M6iav cOuGqcNHL0SFQvlBoM32bGt688uZ2/EcarT8rSK5QJRVElwlpDgj7X/1p bYV0aQ5TeWuAO+Dy5cgiObRKhjnDKUlL6CgaSuVAAdcSgF6jWezatyrNf ooMWzR41NicthbEZG8XyR3yg27xa91dQ00A1TAG799xPPz3Wk/i+lyvxy MsXok1iAORNHu36f3bYmqf9jCbH6LnBC59rY06MOEB19uWwFKqjIEkM2c Pug8vhjCFJZSJOSWDluEGsAe8D6rpF4+R9vDuL8ef7tsUcJWaXYsRFrJw w==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930908" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930908" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762619" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762619" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 03/11] drm: introduce drm evictable LRU Date: Thu, 2 Nov 2023 00:32:58 -0400 Message-Id: <20231102043306.2931989-4-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drm LRU manager is introuced for resource eviction purpose. It maintains a LRU list per resource type. It provides functions to add or remove resource to or from the list. It also provides function to retrieve the first entity on the LRU list. drm LRU manager also provides functions for bulk moving resources on the LRU lists. drm LRU manager also does very basic memory accounting function, i.e., LRU manager keeps a size of this resource type and a usage member for how much of resource has been added to this LRU manager's LRU list. TTM resource manager memory accounting functoins such as struct ttm_resource_manager::size and struct ttm_resource_manger::usage are still kept. In the future, when SVM codes are in the picture, those memory accounting functions need some rework to consider the memory used by both TTM and SVM. For one device, a global drm LRU manager per resource type should be created/initialized at device initialization time. Drm LRU manager instances are embedded in struct drm_device. It is pretty much moving some of the ttm resource manager functions to the drm layer. The reason of this code refactory is, we want to create a single LRU list for memory allocated from BO(buffer object) based driver and hmm/svm(shared virtual memory) based driver, thus BO driver and svm driver can evict memory from each other. Previously the LRU list in TTM resource manager (lru field in struct ttm_reource_manager) is coupled with ttm_buffer_object concept, i.e., each ttm resource is backed by a ttm_buffer_object and the LRU list is essentially a list of ttm_buffer_object. Due to this behavior, the TTM resource manager can't be used by hmm/svm driver as we don't plan to have the BO concept for the hmm/svm implemenation. So we decouple the evictable LRU list from the BO concept in this series. The design goal of drm lru manager is to make it as lean as possible. So each lru entity only has a list node member used to link this entity to the evictable LRU list, and the basic resource size/type/priority of this entity. It doesn't have any driver specify information. A lru entity also has a function pointer of evict function. This is used to implement ttm or svm specific eviction function. A lru entity is supposed to be embedded in a driver specific structure such as struct ttm_resource, see the usage in the next patch of this series. The ttm resource manager, and some of the ttm_bo functions such as ttm_mem_evict_first will be rewriten using the new drm lru manager library, see the next patch in this series. The future hmm/svm implemenation will call lru manager function to add hmm/svm allocations to the shared evictable lru list. Lock design: previously ttm_resource LRU list is protected by a device global ttm_device::lru_lock (bdev->lru_lock in codes). This lock also protects ttm_buffer_object::pin_count, ttm_resource_manager::usage, ttm_resource::bo, ttm_device::pinned list etc. With this refactory, lru_lock is moved out of ttm_device and is added to struct drm_deive, so it can be shared b/t ttm code and svm code. Signed-off-by: Oak Zeng --- drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++ include/drm/drm_device.h | 7 + include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++ 4 files changed, 428 insertions(+) create mode 100644 drivers/gpu/drm/drm_evictable_lru.c create mode 100644 include/drm/drm_evictable_lru.h diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile index 1ad88efb1752..13953b0d271b 100644 --- a/drivers/gpu/drm/Makefile +++ b/drivers/gpu/drm/Makefile @@ -46,6 +46,7 @@ drm-y := \ drm_vblank_work.o \ drm_vma_manager.o \ drm_gpuva_mgr.o \ + drm_evictable_lru.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \ diff --git a/drivers/gpu/drm/drm_evictable_lru.c b/drivers/gpu/drm/drm_evictable_lru.c new file mode 100644 index 000000000000..2ba9105cca03 --- /dev/null +++ b/drivers/gpu/drm/drm_evictable_lru.c @@ -0,0 +1,232 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2023 Intel Corporation + */ + +#include +#include +#include +#include + +static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity) +{ + struct drm_lru_mgr *mgr; + + mgr = &entity->drm->lru_mgr[entity->mem_type]; + BUG_ON(!mgr->used); + + return mgr; +} + +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm, + uint32_t mem_type, uint64_t size, uint32_t priority) +{ + entity->drm = drm; + entity->mem_type = mem_type; + entity->size = size; + entity->priority = priority; + INIT_LIST_HEAD(&entity->lru); +} + +/** + * drm_lru_mgr_init + * + * @mgr: drm lru manager to init + * @size: size of the resource managed by this manager + * @lock: pointer of the global lru_lock + * + * Initialize a drm lru manager + */ +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size, spinlock_t *lock) +{ + unsigned j; + + mgr->used = true; + mgr->size = size; + mgr->usage = 0; + mgr->lru_lock = lock; + + for(j = 0; j < DRM_MAX_LRU_PRIORITY; j++) + INIT_LIST_HEAD(&mgr->lru[j]); +} + +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move) +{ + memset(bulk_move, 0, sizeof(*bulk_move)); +} + +/** + * drm_lru_first + * + * @mgr: drm lru manager to iterate over + * @cursor: cursor of the current position + * + * Returns the first entity in drm lru manager + */ +struct drm_lru_entity * +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor) +{ + struct drm_lru_entity *entity; + + lockdep_assert_held(mgr->lru_lock); + + for(cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority) + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru) + return entity; + + return NULL; +} + +/** + * drm_lru_next + * + * @mgr: drm lru manager to iterate over + * @cursor: cursor of the current position + * @entity: the current lru entity pointer + * + * Returns the next entity from drm lru manager + */ +struct drm_lru_entity * +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor, + struct drm_lru_entity *entity) +{ + lockdep_assert_held(mgr->lru_lock); + + list_for_each_entry_continue(entity, &mgr->lru[cursor->priority], lru) + return entity; + + for(++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority) + list_for_each_entry(entity, &mgr->lru[cursor->priority], lru) + return entity; + + return NULL; +} + +/** + * drm_lru_move_to_tail + * + * @entity: the lru entity to move to lru tail + * + * Move a lru entity to lru tail + */ +void drm_lru_move_to_tail(struct drm_lru_entity * entity) +{ + struct list_head *lru; + struct drm_lru_mgr *mgr; + + mgr = entity_to_mgr(entity); + lockdep_assert_held(mgr->lru_lock); + lru = &mgr->lru[entity->priority]; + list_move_tail(&entity->lru, lru); +} + +/** + * drm_lru_bulk_move_range_tail + * + * @range: bulk move range + * @entity: lru_entity to move + * + * Move a lru_entity to the tail of a bulk move range + */ +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range, + struct drm_lru_entity *entity) +{ + if (entity == range->last) + return; + + if (entity == range->first) + range->first = container_of(entity->lru.next, struct drm_lru_entity, lru); + + if (range->last) + list_move(&entity->lru, &range->last->lru); + + range->last = entity; +} +EXPORT_SYMBOL(drm_lru_bulk_move_range_tail); + +/** + * drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail. + * + * @bulk: bulk_move structure + * + * Bulk move entities to the LRU tail, only valid to use when driver makes sure that + * resource order never changes. + */ +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk) +{ + + unsigned i, j; + + for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) { + for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) { + struct drm_lru_bulk_move_range *range = &bulk->range[i][j]; + struct drm_lru_mgr *mgr; + + if (!range->first) + continue; + + mgr = entity_to_mgr(range->first); + lockdep_assert_held(mgr->lru_lock); + list_bulk_move_tail(&mgr->lru[range->first->priority], &range->first->lru, + &range->last->lru); + } + } +} +EXPORT_SYMBOL(drm_lru_bulk_move_tail); + +/** + * drm_lru_add_bulk_move + * + * @entity: the lru entity to add to the bulk move range + * @bulk_move: the bulk move ranges to add the entity + * + * Add a lru entity to the tail of a bulk move range + */ +void drm_lru_add_bulk_move(struct drm_lru_entity *entity, + struct drm_lru_bulk_move *bulk_move) +{ + struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + + if (!range->first) { + range->first = entity; + range->last = entity; + return; + } + + drm_lru_bulk_move_range_tail(range, entity); +} + +EXPORT_SYMBOL(drm_lru_add_bulk_move); +/** + * drm_lru_del_bulk_move + * + * @entity: the lru entity to move from the bulk move range + * @bulk_move: the bulk move ranges to move the entity out of + * + * Move a lru entity out of bulk move range. This doesn't + * delete entity from lru manager's lru list. + */ +void drm_lru_del_bulk_move(struct drm_lru_entity *entity, + struct drm_lru_bulk_move *bulk_move) +{ + struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + + if (unlikely(WARN_ON(!range->first || !range->last) || + (range->first == entity && range->last == entity))) { + range->first = NULL; + range->last = NULL; + } else if (range->first == entity) { + range->first = container_of(entity->lru.next, + struct drm_lru_entity, lru); + } else if (range->last == entity) { + range->last = container_of(entity->lru.prev, + struct drm_lru_entity, lru); + } else { + list_move(&entity->lru, &range->last->lru); + } +} +EXPORT_SYMBOL(drm_lru_del_bulk_move); diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h index d0b5f42786be..1bdcd34d3f6b 100644 --- a/include/drm/drm_device.h +++ b/include/drm/drm_device.h @@ -8,6 +8,7 @@ #include #include +#include struct drm_driver; struct drm_minor; @@ -331,6 +332,12 @@ struct drm_device { */ spinlock_t lru_lock; + /** + * @lru_mgr: Device global lru managers per memory type or memory + * region. Each lru manager manages a lru list of this memory type. + */ + struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES]; + /* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED(CONFIG_DRM_LEGACY) diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h new file mode 100644 index 000000000000..3fd6bd2475d9 --- /dev/null +++ b/include/drm/drm_evictable_lru.h @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2023 Intel Corporation + */ + +#ifndef _DRM_EVICTABLE_LRU_H_ +#define _DRM_EVICTABLE_LRU_H_ + +#include +#include +#include + +struct drm_device; + +#define DRM_MAX_LRU_PRIORITY 4 +#define DRM_NUM_MEM_TYPES 8 + +/** + * struct drm_lru_entity + * + * @drm: drm device that this entity belongs to + * @mem_type: The memory type that this entity belongs to + * @size: resource size of this entity + * @priority: The priority of this entity + * @lru: least recent used list node, see &drm_lru_mgr.lru + * + * This structure represents an entity in drm_lru_mgr's + * list. This structure is supposed to be embedded in + * user's data structure. + */ +struct drm_lru_entity { + struct drm_device *drm; + uint32_t mem_type; + uint64_t size; + uint32_t priority; + struct list_head lru; +}; + +/** + * struct drm_lru_mgr + * + * @used: whether this lru manager is used or not + * @size: size of the resource + * @usage: how much resource has been used + * @lru_lock: a weak reference to the global lru_lock + * @lru: least recent used list, per priority + * + * This structure maintains all the buffer allocations + * in a least recent used list, so a victim for eviction + * can be easily found. + */ +struct drm_lru_mgr { + bool used; + uint64_t size; + uint64_t usage; + spinlock_t *lru_lock; + struct list_head lru[DRM_MAX_LRU_PRIORITY]; +}; + +/** + * struct drm_lru_cursor + * + * @priority: the current priority + * + * Cursor to iterate over all entities in lru manager. + */ +struct drm_lru_cursor { + unsigned priority; +}; + +/** + * struct drm_lru_bulk_move_range + * + * @first: the first entity in the range + * @last: the last entity in the range + * + * Range of entities on a lru list. + */ +struct drm_lru_bulk_move_range +{ + struct drm_lru_entity *first; + struct drm_lru_entity *last; +}; + +/** + * struct drm_lru_bulk_move + * + * @range: An array of bulk move range, each corelates to the drm_lru_mgr's + * lru list of the same memory type and same priority. + * + * A collection of bulk_move range which can be used to move drm_lru_entity + * on the lru list in a bulk way. It should be initialized through + * drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call + * drm_lru_add_bulk_move/drm_lru_del_bulk_move. + */ +struct drm_lru_bulk_move { + struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY]; +}; + + + +/** + * drm_lru_add_entity + * + * @entity: the lru entity to add + * @mgr: the drm lru manager + * @priority: specify which priority list to add + * + * Add an entity to lru list + */ +static inline void drm_lru_add_entity(struct drm_lru_entity *entity, + struct drm_lru_mgr *mgr, unsigned priority) +{ + lockdep_assert_held(mgr->lru_lock); + list_add_tail(&entity->lru, &mgr->lru[priority]); + mgr->usage += entity->size; +} + +/** + * drm_lru_remove_entity + * + * @entity: the lru entity to remove + * @mgr: the drm lru manager + * + * Remove an entity from lru list + */ +static inline void drm_lru_remove_entity(struct drm_lru_entity *entity, + struct drm_lru_mgr *mgr) +{ + lockdep_assert_held(mgr->lru_lock); + list_del_init(&entity->lru); + mgr->usage -= entity->size; +} + +/** + * drm_lru_mgr_fini + * + * @mgr: the drm lru manager + * + * de-initialize a lru manager + */ +static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr) +{ + mgr->used = false; +} + +void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm, + uint32_t mem_type, uint64_t size, uint32_t priority); + +struct drm_lru_entity * +drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor); + +struct drm_lru_entity * +drm_lru_next(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor, + struct drm_lru_entity *entity); + +void drm_lru_mgr_init(struct drm_lru_mgr *mgr, uint64_t size, + spinlock_t *lru_lock); + +void drm_lru_move_to_tail(struct drm_lru_entity * entity); + +void drm_lru_bulk_move_init(struct drm_lru_bulk_move *bulk_move); + + +void drm_lru_bulk_move_tail(struct drm_lru_bulk_move *bulk); + +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range, + struct drm_lru_entity *entity); + +void drm_lru_add_bulk_move(struct drm_lru_entity *entity, + struct drm_lru_bulk_move *bulk_move); + +void drm_lru_del_bulk_move(struct drm_lru_entity *entity, + struct drm_lru_bulk_move *bulk_move); +/** + * drm_lru_for_each_entity + * + * @mgr: the drm lru manager + * @cursor: cursor for the current position + * @entity: the current drm_lru_entity + * + * Iterate over all entities in drm lru manager + */ +#define drm_lru_for_each_entity(mgr, cursor, entity) \ + for (entity = drm_lru_first(mgr, cursor); entity; \ + entity = drm_lru_next(mgr, cursor, entity)) + +#endif From patchwork Thu Nov 2 04:32:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 077F5C4332F for ; Thu, 2 Nov 2023 04:25:03 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4973910E804; Thu, 2 Nov 2023 04:24:40 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id F1BC410E7F9; Thu, 2 Nov 2023 04:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899071; x=1730435071; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jh8YduMlkBO3fPFoyyIlELlsSlB5g9xW1LHQvAUwav0=; b=QsUjQlnUiocY2QcVHJtCFGv9IKmX1/rvb1eplx52LM+VvwJEBxWBzb+R OEI9uuu6ViMzd0jvFLcHUqJSjoA8iJyAX3M2OoptxhzTaSmcOcHwcJ9il 2LTMYhjCM3+IdkHk12osVJrK33VyXUo/vWBKESuLTC2O4XL8fhHAyetlG 7RFZZCofVWUVyvmEpB7t41fFAshgTO9HZ25jHhqp6jRP2RM1PlRe2kHkt XrMESOiHY/S2X4U3SZJv8D0UG1B3dGtRq4fSJc0Cjm1FUqePKIfJRsFi2 Zxrg1mAI8C0inA9fES9EvrZ4leWZgAffWdkNfAtQwW5s9FLv3gOzPaM1H g==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930913" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930913" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762622" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762622" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 04/11] drm: Add evict function pointer to drm lru entity Date: Thu, 2 Nov 2023 00:32:59 -0400 Message-Id: <20231102043306.2931989-5-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Drm lru manager provides generic functions to manage lru list, and function to evict a lru entity. But how to evict an entity is implemented in an entity's sub-class. This patch introduces a few function pointers to drm lru entity for this purpose. Those functions are abstracted from the current ttm resource eviction process. They need to be tunned in the future when svm code comes into the picture. Also implemented a drm_lru_evict_first function to evict the first lru entity from lru manager. Both ttm and svm codes are supposed to call this function to evict the first resource from lru list. This way ttm and svm codes can mutually evict resources from each other. Signed-off-by: Oak Zeng --- drivers/gpu/drm/drm_evictable_lru.c | 36 +++++++++++++- include/drm/drm_evictable_lru.h | 74 ++++++++++++++++++++++++++++- 2 files changed, 108 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/drm_evictable_lru.c b/drivers/gpu/drm/drm_evictable_lru.c index 2ba9105cca03..7b62cae2dfea 100644 --- a/drivers/gpu/drm/drm_evictable_lru.c +++ b/drivers/gpu/drm/drm_evictable_lru.c @@ -19,13 +19,15 @@ static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity) } void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm, - uint32_t mem_type, uint64_t size, uint32_t priority) + uint32_t mem_type, uint64_t size, uint32_t priority, + struct drm_lru_evict_func *evict_func) { entity->drm = drm; entity->mem_type = mem_type; entity->size = size; entity->priority = priority; INIT_LIST_HEAD(&entity->lru); + entity->evict_func = evict_func; } /** @@ -230,3 +232,35 @@ void drm_lru_del_bulk_move(struct drm_lru_entity *entity, } } EXPORT_SYMBOL(drm_lru_del_bulk_move); + +int drm_lru_evict_first(struct drm_lru_mgr *mgr, + const struct drm_lru_evict_ctx *evict_ctx) +{ + struct drm_lru_entity *entity, *busy_entity = NULL; + struct drm_lru_cursor cursor; + bool locked = false, busy = false, found = false; + + spin_lock(mgr->lru_lock); + + /* First find a victim to evict*/ + drm_lru_for_each_entity(mgr, &cursor, entity) { + if (!entity->evict_func->evict_allowable(entity, + evict_ctx, &busy, &locked)) { + if (!busy_entity && busy) + busy_entity = entity; + continue; + } + found = true; + break; + } + + /* We didn't find a victim, but we found a busy entity, i.e., + * other clients hold a reservation lock of this entity. Try + * to wait and evict this busy entity. + */ + if (!found && busy_entity) + return busy_entity->evict_func->evict_busy_entity(busy_entity, evict_ctx); + + /* If here, we found a victim to evict*/ + return entity->evict_func->evict_entity(entity, evict_ctx, locked); +} diff --git a/include/drm/drm_evictable_lru.h b/include/drm/drm_evictable_lru.h index 3fd6bd2475d9..7f49964f2f9b 100644 --- a/include/drm/drm_evictable_lru.h +++ b/include/drm/drm_evictable_lru.h @@ -15,6 +15,12 @@ struct drm_device; #define DRM_MAX_LRU_PRIORITY 4 #define DRM_NUM_MEM_TYPES 8 +struct drm_lru_evict_ctx { + void *data1; + void *data2; + void *data3; +}; + /** * struct drm_lru_entity * @@ -23,6 +29,7 @@ struct drm_device; * @size: resource size of this entity * @priority: The priority of this entity * @lru: least recent used list node, see &drm_lru_mgr.lru + * @evict_func: functions to evict this entity * * This structure represents an entity in drm_lru_mgr's * list. This structure is supposed to be embedded in @@ -34,6 +41,7 @@ struct drm_lru_entity { uint64_t size; uint32_t priority; struct list_head lru; + struct drm_lru_evict_func *evict_func; }; /** @@ -97,7 +105,67 @@ struct drm_lru_bulk_move { struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY]; }; +struct drm_lru_evict_func { + /** + * evict_allowable + * + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is + * passed to drm lru layer through drm_lru_evict_first function and drm lru + * layer just pass it back to ttm or svm code by calling some ttm or svm + * callback functions. + * @busy: used to return whether the current resource is busy (i.e., locked + * by other clients) + * @locked: used to return whether this resource is locked during this check, + * i.e., successfully trylocked bo's dma reservation object + * + * Check whether we are allowed to evict a memory resource. Return true if we + * are allowed to evict resource; otherwise false. + * + * When this function returns true, a resource reference counter is hold. This + * reference counter need to be released after evict operation later on. + * + * This function should be called with lru_lock hold. + */ + bool (*evict_allowable)(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *evict_ctx, + bool *busy, bool *locked); + /** + * evict_busy_entity + * + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is + * passed to drm lru layer through drm_lru_evict_first function and drm lru + * layer just pass it back to ttm or svm code by calling some ttm or svm + * callback functions. + * + * Evict a busy memory resource. + * This function should be called with lru_lock hold. + */ + int (*evict_busy_entity)(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *evict_ctx); + + /** + * evict_entity + * + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @evict_ctx: eviction context. This is opaque data to drm lru layer. It is + * passed to drm lru layer through drm_lru_evict_first function and drm lru + * layer just pass it back to ttm or svm code by calling some ttm or svm + * callback functions. + * @locked: whether this resource is dma-reserved (if reserved, we need to + * unreserve it in this function) + * + * Evict a memory resource corresponding to a lru_entity. This should be + * called holding lru_lock + */ + int (*evict_entity)(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *evict_ctx, bool locked); +}; /** * drm_lru_add_entity @@ -145,7 +213,8 @@ static inline void drm_lru_mgr_fini(struct drm_lru_mgr *mgr) } void drm_lru_entity_init(struct drm_lru_entity *entity, struct drm_device *drm, - uint32_t mem_type, uint64_t size, uint32_t priority); + uint32_t mem_type, uint64_t size, uint32_t priority, + struct drm_lru_evict_func *evict_func); struct drm_lru_entity * drm_lru_first(struct drm_lru_mgr *mgr, struct drm_lru_cursor *cursor); @@ -172,6 +241,9 @@ void drm_lru_add_bulk_move(struct drm_lru_entity *entity, void drm_lru_del_bulk_move(struct drm_lru_entity *entity, struct drm_lru_bulk_move *bulk_move); + +int drm_lru_evict_first(struct drm_lru_mgr *mgr, + const struct drm_lru_evict_ctx *evict_ctx); /** * drm_lru_for_each_entity * From patchwork Thu Nov 2 04:33:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0462AC4332F for ; Thu, 2 Nov 2023 04:24:55 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C8C4A10E80A; Thu, 2 Nov 2023 04:24:36 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 15A7F10E7FB; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QhHvWEZ31Nrqe6D9olvuRmqbl4OUEcTS9Em2K9q5xkY=; b=kTP0T/dC9OXfTFuzxANOq5wTxx2D5QommHunu8dJHOnDhJ6uVVKINNLn xAFVkucutZSKkfaoSRvpC/5W0qlkuTmGT3K/Grw6qMeOzz3xpYA9cMSYq sw/ogT+sMnJaOcQSkobAUfpbSZ2xFJ5+ZpnfTmiGI2H7f7CvMob6IjZBe f+5r88E0qYCQkuRlEvSbQ2ZgdCNAq9DbFPBLKwkmKZF/XSATNwdzyaH4M wIrk4Qnjv+QYM8824T5n5pVI1+zRthowpPrHjYsCeyLxNoO59qm9Y3DLH A/Z0bWFMxZx+BAZwcwqBvS6LAm1mTbFUbvSsQ8UfQ45LrBPuEEivgL+Sg g==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930916" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930916" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762625" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762625" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 05/11] drm: Replace ttm macros with drm macros Date: Thu, 2 Nov 2023 00:33:00 -0400 Message-Id: <20231102043306.2931989-6-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" TTM_MAX_BO_PRIORITY and TTM_NUM_MEM_TYPES are move from ttm to drm, so: s/TTM_MAX_BO_PRIORITY/DRM_MAX_LRU_PRIORITY s/TTM_NUM_MEM_TYPES/DRM_NUM_MEM_TYPES Signed-off-by: Oak Zeng --- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 2 +- drivers/gpu/drm/i915/intel_region_ttm.c | 2 +- drivers/gpu/drm/i915/selftests/mock_region.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 8 ++++---- drivers/gpu/drm/ttm/ttm_resource.c | 12 ++++++------ drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +- drivers/gpu/drm/xe/xe_bo.h | 2 +- include/drm/ttm/ttm_device.h | 6 +++--- include/drm/ttm/ttm_range_manager.h | 4 ++-- include/drm/ttm/ttm_resource.h | 10 ++++------ 11 files changed, 25 insertions(+), 27 deletions(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c index c46f54f83f54..228dbea60949 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c @@ -987,7 +987,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj) spin_lock(bo->bdev->lru_lock); if (shrinkable) { /* Try to keep shmem_tt from being considered for shrinking. */ - bo->priority = TTM_MAX_BO_PRIORITY - 1; + bo->priority = DRM_MAX_LRU_PRIORITY - 1; } else if (obj->mm.madv != I915_MADV_WILLNEED) { bo->priority = I915_TTM_PRIO_PURGE; } else if (!i915_gem_object_has_pages(obj)) { diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c index b845782c9859..f75520c2ba59 100644 --- a/drivers/gpu/drm/i915/intel_region_ttm.c +++ b/drivers/gpu/drm/i915/intel_region_ttm.c @@ -63,7 +63,7 @@ int intel_region_to_ttm_type(const struct intel_memory_region *mem) return TTM_PL_SYSTEM; type = mem->instance + TTM_PL_PRIV; - GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES); + GEM_BUG_ON(type >= DRM_NUM_MEM_TYPES); return type; } diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c index 6324eb32d4dd..6ea0e6bec812 100644 --- a/drivers/gpu/drm/i915/selftests/mock_region.c +++ b/drivers/gpu/drm/i915/selftests/mock_region.c @@ -111,7 +111,7 @@ mock_region_create(struct drm_i915_private *i915, resource_size_t io_size) { int instance = ida_alloc_max(&i915->selftest.mock_region_instances, - TTM_NUM_MEM_TYPES - TTM_PL_PRIV - 1, + DRM_NUM_MEM_TYPES - TTM_PL_PRIV - 1, GFP_KERNEL); if (instance < 0) diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index d18eca86ebd6..e8c8006ba748 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -148,7 +148,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, int ret; spin_lock(bdev->lru_lock); - for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) { + for (i = TTM_PL_SYSTEM; i < DRM_NUM_MEM_TYPES; ++i) { man = ttm_manager_type(bdev, i); if (!man || !man->use_tt) continue; @@ -245,7 +245,7 @@ void ttm_device_fini(struct ttm_device *bdev) destroy_workqueue(bdev->wq); spin_lock(bdev->lru_lock); - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) + for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) if (list_empty(&man->lru[0])) pr_debug("Swap list %d was clean\n", i); spin_unlock(bdev->lru_lock); @@ -287,12 +287,12 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev) ttm_device_clear_lru_dma_mappings(bdev, &bdev->pinned); - for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) { + for (i = TTM_PL_SYSTEM; i < DRM_NUM_MEM_TYPES; ++i) { man = ttm_manager_type(bdev, i); if (!man || !man->use_tt) continue; - for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) + for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) ttm_device_clear_lru_dma_mappings(bdev, &man->lru[j]); } } diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 6ada77f51fba..05eef866065e 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -54,8 +54,8 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk) { unsigned i, j; - for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) { - for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) { + for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) { + for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) { struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j]; struct ttm_resource_manager *man; @@ -393,7 +393,7 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man, man->size = size; man->usage = 0; - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) + for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) INIT_LIST_HEAD(&man->lru[i]); man->move = NULL; } @@ -425,7 +425,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, */ spin_lock(bdev->lru_lock); - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { + for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) { while (!list_empty(&man->lru[i])) { spin_unlock(bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx, @@ -504,7 +504,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man, lockdep_assert_held(man->bdev->lru_lock); - for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY; + for (cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority) list_for_each_entry(res, &man->lru[cursor->priority], lru) return res; @@ -531,7 +531,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man, list_for_each_entry_continue(res, &man->lru[cursor->priority], lru) return res; - for (++cursor->priority; cursor->priority < TTM_MAX_BO_PRIORITY; + for (++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY; ++cursor->priority) list_for_each_entry(res, &man->lru[cursor->priority], lru) return res; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c index c43853597776..9efde9ba9fe2 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.c @@ -383,7 +383,7 @@ static int vmw_bo_init(struct vmw_private *dev_priv, memset(vmw_bo, 0, sizeof(*vmw_bo)); - BUILD_BUG_ON(TTM_MAX_BO_PRIORITY <= 3); + BUILD_BUG_ON(DRM_MAX_LRU_PRIORITY <= 3); vmw_bo->tbo.priority = 3; vmw_bo->res_tree = RB_ROOT; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h index 1d433fceed3d..c43a82c8afb9 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_bo.h @@ -82,7 +82,7 @@ struct vmw_bo { struct ttm_bo_kmap_obj map; struct rb_root res_tree; - u32 res_prios[TTM_MAX_BO_PRIORITY]; + u32 res_prios[DRM_MAX_LRU_PRIORITY]; atomic_t cpu_writers; /* Not ref-counted. Protected by binding_mutex */ diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h index 9097bcc13209..9918b2d630e1 100644 --- a/drivers/gpu/drm/xe/xe_bo.h +++ b/drivers/gpu/drm/xe/xe_bo.h @@ -72,7 +72,7 @@ #define XE_PL_TT TTM_PL_TT #define XE_PL_VRAM0 TTM_PL_VRAM #define XE_PL_VRAM1 (XE_PL_VRAM0 + 1) -#define XE_PL_STOLEN (TTM_NUM_MEM_TYPES - 1) +#define XE_PL_STOLEN (DRM_NUM_MEM_TYPES - 1) #define XE_BO_PROPS_INVALID (-1) diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index 4d29e96bd892..a75e820dc671 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -235,7 +235,7 @@ struct ttm_device { /** * @man_drv: An array of resource_managers, one per resource type. */ - struct ttm_resource_manager *man_drv[TTM_NUM_MEM_TYPES]; + struct ttm_resource_manager *man_drv[DRM_NUM_MEM_TYPES]; /** * @vma_manager: Address space manager for finding BOs to mmap. @@ -277,14 +277,14 @@ static inline struct ttm_resource_manager * ttm_manager_type(struct ttm_device *bdev, int mem_type) { BUILD_BUG_ON(__builtin_constant_p(mem_type) - && mem_type >= TTM_NUM_MEM_TYPES); + && mem_type >= DRM_NUM_MEM_TYPES); return bdev->man_drv[mem_type]; } static inline void ttm_set_driver_manager(struct ttm_device *bdev, int type, struct ttm_resource_manager *manager) { - BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES); + BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES); bdev->man_drv[type] = manager; } diff --git a/include/drm/ttm/ttm_range_manager.h b/include/drm/ttm/ttm_range_manager.h index 7963b957e9ef..becdb88c4d84 100644 --- a/include/drm/ttm/ttm_range_manager.h +++ b/include/drm/ttm/ttm_range_manager.h @@ -43,14 +43,14 @@ static __always_inline int ttm_range_man_init(struct ttm_device *bdev, unsigned int type, bool use_tt, unsigned long p_size) { - BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES); + BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES); return ttm_range_man_init_nocheck(bdev, type, use_tt, p_size); } static __always_inline int ttm_range_man_fini(struct ttm_device *bdev, unsigned int type) { - BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES); + BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES); return ttm_range_man_fini_nocheck(bdev, type); } #endif diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index 78a226eba953..92241c2374fa 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -34,9 +34,7 @@ #include #include #include - -#define TTM_MAX_BO_PRIORITY 4U -#define TTM_NUM_MEM_TYPES 8 +#include struct ttm_device; struct ttm_resource_manager; @@ -167,7 +165,7 @@ struct ttm_resource_manager { /* * Protected by the bdev->lru_lock. */ - struct list_head lru[TTM_MAX_BO_PRIORITY]; + struct list_head lru[DRM_MAX_LRU_PRIORITY]; /** * @usage: How much of the resources are used, protected by the @@ -253,7 +251,7 @@ struct ttm_lru_bulk_move_pos { * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move(). */ struct ttm_lru_bulk_move { - struct ttm_lru_bulk_move_pos pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY]; + struct ttm_lru_bulk_move_pos pos[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY]; }; /** @@ -309,7 +307,7 @@ ttm_resource_manager_set_used(struct ttm_resource_manager *man, bool used) { int i; - for (i = 0; i < TTM_MAX_BO_PRIORITY; i++) + for (i = 0; i < DRM_MAX_LRU_PRIORITY; i++) WARN_ON(!list_empty(&man->lru[i])); man->use_type = used; } From patchwork Thu Nov 2 04:33:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B5E0C4167D for ; Thu, 2 Nov 2023 04:24:59 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6418810E80E; Thu, 2 Nov 2023 04:24:37 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 40AF610E7FE; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=998btFXeKIHao/vtnWrxBwSg/rS8lCXIJvNCwrK1FKc=; b=DTBqIRjuHqz5gDDqF+oCQ+zUpC+kJTgL8Qall/eNUx6ZN9VjRi/4ld+x qsRl/9HX6y3lC0v/rufA561HQHW1Jnv8KeNZF3onLW985jviZ8Vjt/Ygh lsQjaelK0oMuCeNUKu664GuRNDd7mNBn2oi4LirhzSPSe3bDJa4jCn5TW yeL6rvRk9dUbtmDXZ9vTFdD/xKLW3oD+2DTQ+dNrBMGlg2cWcQjBc4VWk uwJmxY34dJ4ku8xH12L7gimOBRxCEB1qwtSAftyhiFUmFVbjriLX7P+zS kN4X1SSZfBv2KQroum8sX4Jzq2KFptAFFqCmDOWuGJ2UY5NM/VNcZHUzl Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930920" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930920" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762628" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762628" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 06/11] drm/ttm: Set lru manager to ttm resource manager Date: Thu, 2 Nov 2023 00:33:01 -0400 Message-Id: <20231102043306.2931989-7-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add a weak reference of lru manager to ttm resource manager, and add a function to set lru manager for ttm resource manager. Signed-off-by: Oak Zeng --- include/drm/ttm/ttm_resource.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index 92241c2374fa..e4fc1ada5236 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -46,6 +46,7 @@ struct iosys_map; struct io_mapping; struct sg_table; struct scatterlist; +struct drm_lru_mgr; struct ttm_resource_manager_func { /** @@ -172,6 +173,12 @@ struct ttm_resource_manager { * bdev->lru_lock. */ uint64_t usage; + + /** + * @lru_mgr: weak reference of the lru manager that manages lru + * list for this ttm resource manager. + */ + struct drm_lru_mgr *lru_mgr; }; /** @@ -326,6 +333,18 @@ static inline bool ttm_resource_manager_used(struct ttm_resource_manager *man) return man->use_type; } +/** + * ttm_resource_manager_set_lru_mgr + * + * @man: ttm resource manager + * @mgr: pointing to lru manager + */ +static inline void +ttm_resource_manager_set_lru_mgr(struct ttm_resource_manager *man, struct drm_lru_mgr *mgr) +{ + man->lru_mgr = mgr; +} + /** * ttm_resource_manager_cleanup * From patchwork Thu Nov 2 04:33:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17BD2C001DB for ; Thu, 2 Nov 2023 04:24:52 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F10EF10E808; Thu, 2 Nov 2023 04:24:35 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5D8F310E7FF; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F3OFjU24sUWWNI1HDbZ5mOYwa2qg9nmKmjhkLAPsc9Y=; b=Grwg1e1i6vPKj0fZYZBZ27KYdg7nF0dVtZUktY+BEBVlRJo5ac6bmuIh FLRok3nrmy4k2CkWTlfKI1c7cl+i0/1GFKfXdoyUiHg9k2WnwnsfQyaHQ fdliNMQdylD6DW8JCfZZ0JZ5nrcfGabC6oI3hlIMFbx0p8WaB+EeXpXcN CmZH9moaGRR+O72pWsJz07PbGz3TcLAh/GJG1QtsnG0L1pGbiDnN03nFN aaau3AcvbY+gYjcw7/TUU2R348Ef0mEyjPnueC2B/y6oP/VDFWTA2H1MS YggFf+20RM9Yuk2m3WLj83ikif4AMVCXWqkOIl7OxEWzQejCG3J8VG5iB A==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930923" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930923" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762632" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762632" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 07/11] drm/ttm: re-parameterize a few ttm functions Date: Thu, 2 Nov 2023 00:33:02 -0400 Message-Id: <20231102043306.2931989-8-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add a struct drm_device *drm parameter to function ttm_range_man_init, ttm_range_man_fini, ttm_sys_man_init, and ttm_sys_man_free. This drm parameter will be used in the coming patches to retrieve and initialize drm lru manager. Signed-off-by: Oak Zeng --- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 ++++---- drivers/gpu/drm/drm_gem_vram_helper.c | 8 ++++---- drivers/gpu/drm/loongson/lsdc_ttm.c | 8 ++++---- drivers/gpu/drm/nouveau/nouveau_ttm.c | 8 ++++---- drivers/gpu/drm/qxl/qxl_ttm.c | 6 +++--- drivers/gpu/drm/radeon/radeon_ttm.c | 8 ++++---- drivers/gpu/drm/ttm/ttm_device.c | 2 +- drivers/gpu/drm/ttm/ttm_module.h | 3 ++- drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_sys_manager.c | 6 +++++- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 4 ++-- include/drm/ttm/ttm_range_manager.h | 13 +++++++------ 12 files changed, 44 insertions(+), 36 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 5cdbc901cbe2..cc0736f82a80 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -75,7 +75,7 @@ static int amdgpu_ttm_init_on_chip(struct amdgpu_device *adev, unsigned int type, uint64_t size_in_page) { - return ttm_range_man_init(&adev->mman.bdev, type, + return ttm_range_man_init(adev_to_drm(adev), &adev->mman.bdev, type, false, size_in_page); } @@ -2026,9 +2026,9 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev) amdgpu_vram_mgr_fini(adev); amdgpu_gtt_mgr_fini(adev); amdgpu_preempt_mgr_fini(adev); - ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_GDS); - ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_GWS); - ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_OA); + ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_GDS); + ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_GWS); + ttm_range_man_fini(adev_to_drm(adev), &adev->mman.bdev, AMDGPU_PL_OA); ttm_device_fini(&adev->mman.bdev); adev->mman.initialized = false; DRM_INFO("amdgpu: ttm finalized\n"); diff --git a/drivers/gpu/drm/drm_gem_vram_helper.c b/drivers/gpu/drm/drm_gem_vram_helper.c index 56749e40459f..5b18db72cc96 100644 --- a/drivers/gpu/drm/drm_gem_vram_helper.c +++ b/drivers/gpu/drm/drm_gem_vram_helper.c @@ -1009,7 +1009,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev, if (ret) return ret; - ret = ttm_range_man_init(&vmm->bdev, TTM_PL_VRAM, + ret = ttm_range_man_init(dev, &vmm->bdev, TTM_PL_VRAM, false, vram_size >> PAGE_SHIFT); if (ret) return ret; @@ -1017,9 +1017,9 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev, return 0; } -static void drm_vram_mm_cleanup(struct drm_vram_mm *vmm) +static void drm_vram_mm_cleanup(struct drm_device *drm, struct drm_vram_mm *vmm) { - ttm_range_man_fini(&vmm->bdev, TTM_PL_VRAM); + ttm_range_man_fini(drm, &vmm->bdev, TTM_PL_VRAM); ttm_device_fini(&vmm->bdev); } @@ -1056,7 +1056,7 @@ static void drm_vram_helper_release_mm(struct drm_device *dev) if (!dev->vram_mm) return; - drm_vram_mm_cleanup(dev->vram_mm); + drm_vram_mm_cleanup(dev, dev->vram_mm); kfree(dev->vram_mm); dev->vram_mm = NULL; } diff --git a/drivers/gpu/drm/loongson/lsdc_ttm.c b/drivers/gpu/drm/loongson/lsdc_ttm.c index bd68cb9366b5..f7f226314a09 100644 --- a/drivers/gpu/drm/loongson/lsdc_ttm.c +++ b/drivers/gpu/drm/loongson/lsdc_ttm.c @@ -533,8 +533,8 @@ static void lsdc_ttm_fini(struct drm_device *ddev, void *data) { struct lsdc_device *ldev = (struct lsdc_device *)data; - ttm_range_man_fini(&ldev->bdev, TTM_PL_VRAM); - ttm_range_man_fini(&ldev->bdev, TTM_PL_TT); + ttm_range_man_fini(ddev, &ldev->bdev, TTM_PL_VRAM); + ttm_range_man_fini(ddev, &ldev->bdev, TTM_PL_TT); ttm_device_fini(&ldev->bdev); @@ -556,7 +556,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev) num_vram_pages = ldev->vram_size >> PAGE_SHIFT; - ret = ttm_range_man_init(&ldev->bdev, TTM_PL_VRAM, false, num_vram_pages); + ret = ttm_range_man_init(&ldev->base, &ldev->bdev, TTM_PL_VRAM, false, num_vram_pages); if (unlikely(ret)) return ret; @@ -567,7 +567,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev) num_gtt_pages = ldev->gtt_size >> PAGE_SHIFT; - ret = ttm_range_man_init(&ldev->bdev, TTM_PL_TT, true, num_gtt_pages); + ret = ttm_range_man_init(&ldev->base, &ldev->bdev, TTM_PL_TT, true, num_gtt_pages); if (unlikely(ret)) return ret; diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c index 831918437850..1898f27f0510 100644 --- a/drivers/gpu/drm/nouveau/nouveau_ttm.c +++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c @@ -194,7 +194,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm) ttm_resource_manager_set_used(man, true); return 0; } else { - return ttm_range_man_init(&drm->ttm.bdev, TTM_PL_VRAM, false, + return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM, false, drm->gem.vram_available >> PAGE_SHIFT); } } @@ -211,7 +211,7 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm) ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, NULL); kfree(man); } else - ttm_range_man_fini(&drm->ttm.bdev, TTM_PL_VRAM); + ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM); } static int @@ -226,7 +226,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm) else if (!drm->agp.bridge) func = &nv04_gart_manager; else - return ttm_range_man_init(&drm->ttm.bdev, TTM_PL_TT, true, + return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_TT, true, size_pages); man = kzalloc(sizeof(*man), GFP_KERNEL); @@ -248,7 +248,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm) if (drm->client.device.info.family < NV_DEVICE_INFO_V0_TESLA && drm->agp.bridge) - ttm_range_man_fini(&drm->ttm.bdev, TTM_PL_TT); + ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_TT); else { ttm_resource_manager_set_used(man, false); ttm_resource_manager_evict_all(&drm->ttm.bdev, man); diff --git a/drivers/gpu/drm/qxl/qxl_ttm.c b/drivers/gpu/drm/qxl/qxl_ttm.c index 1a82629bce3f..9f984afca831 100644 --- a/drivers/gpu/drm/qxl/qxl_ttm.c +++ b/drivers/gpu/drm/qxl/qxl_ttm.c @@ -186,7 +186,7 @@ static int qxl_ttm_init_mem_type(struct qxl_device *qdev, unsigned int type, uint64_t size) { - return ttm_range_man_init(&qdev->mman.bdev, type, false, size); + return ttm_range_man_init(&qdev->ddev, &qdev->mman.bdev, type, false, size); } int qxl_ttm_init(struct qxl_device *qdev) @@ -227,8 +227,8 @@ int qxl_ttm_init(struct qxl_device *qdev) void qxl_ttm_fini(struct qxl_device *qdev) { - ttm_range_man_fini(&qdev->mman.bdev, TTM_PL_VRAM); - ttm_range_man_fini(&qdev->mman.bdev, TTM_PL_PRIV); + ttm_range_man_fini(&qdev->ddev, &qdev->mman.bdev, TTM_PL_VRAM); + ttm_range_man_fini(&qdev->ddev, &qdev->mman.bdev, TTM_PL_PRIV); ttm_device_fini(&qdev->mman.bdev); DRM_INFO("qxl: ttm finalized\n"); } diff --git a/drivers/gpu/drm/radeon/radeon_ttm.c b/drivers/gpu/drm/radeon/radeon_ttm.c index 77ca50187162..5ab3f229082e 100644 --- a/drivers/gpu/drm/radeon/radeon_ttm.c +++ b/drivers/gpu/drm/radeon/radeon_ttm.c @@ -68,13 +68,13 @@ struct radeon_device *radeon_get_rdev(struct ttm_device *bdev) static int radeon_ttm_init_vram(struct radeon_device *rdev) { - return ttm_range_man_init(&rdev->mman.bdev, TTM_PL_VRAM, + return ttm_range_man_init(rdev->ddev, &rdev->mman.bdev, TTM_PL_VRAM, false, rdev->mc.real_vram_size >> PAGE_SHIFT); } static int radeon_ttm_init_gtt(struct radeon_device *rdev) { - return ttm_range_man_init(&rdev->mman.bdev, TTM_PL_TT, + return ttm_range_man_init(rdev->ddev, &rdev->mman.bdev, TTM_PL_TT, true, rdev->mc.gtt_size >> PAGE_SHIFT); } @@ -753,8 +753,8 @@ void radeon_ttm_fini(struct radeon_device *rdev) } radeon_bo_unref(&rdev->stolen_vga_memory); } - ttm_range_man_fini(&rdev->mman.bdev, TTM_PL_VRAM); - ttm_range_man_fini(&rdev->mman.bdev, TTM_PL_TT); + ttm_range_man_fini(rdev->ddev, &rdev->mman.bdev, TTM_PL_VRAM); + ttm_range_man_fini(rdev->ddev, &rdev->mman.bdev, TTM_PL_TT); ttm_device_fini(&rdev->mman.bdev); radeon_gart_fini(rdev); rdev->mman.initialized = false; diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index e8c8006ba748..393c3e27016e 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -212,7 +212,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func bdev->funcs = funcs; - ttm_sys_man_init(bdev); + ttm_sys_man_init(drm, bdev); ttm_pool_init(&bdev->pool, drm?drm->dev:NULL, NUMA_NO_NODE, use_dma_alloc, use_dma32); diff --git a/drivers/gpu/drm/ttm/ttm_module.h b/drivers/gpu/drm/ttm/ttm_module.h index 767fe22aed48..6c1443704a35 100644 --- a/drivers/gpu/drm/ttm/ttm_module.h +++ b/drivers/gpu/drm/ttm/ttm_module.h @@ -35,9 +35,10 @@ struct dentry; struct ttm_device; +struct drm_device; extern struct dentry *ttm_debugfs_root; -void ttm_sys_man_init(struct ttm_device *bdev); +void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev); #endif /* _TTM_MODULE_H_ */ diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c index ae11d07eb63a..db1ae370580d 100644 --- a/drivers/gpu/drm/ttm/ttm_range_manager.c +++ b/drivers/gpu/drm/ttm/ttm_range_manager.c @@ -166,6 +166,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = { * ttm_range_man_init_nocheck - Initialise a generic range manager for the * selected memory type. * + * @drm: drm device * @bdev: ttm device * @type: memory manager type * @use_tt: if the memory manager uses tt @@ -175,7 +176,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = { * * Return: %0 on success or a negative error code on failure */ -int ttm_range_man_init_nocheck(struct ttm_device *bdev, +int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev, unsigned type, bool use_tt, unsigned long p_size) { @@ -206,12 +207,13 @@ EXPORT_SYMBOL(ttm_range_man_init_nocheck); * ttm_range_man_fini_nocheck - Remove the generic range manager from a slot * and tear it down. * + * @drm: drm device * @bdev: ttm device * @type: memory manager type * * Return: %0 on success or a negative error code on failure */ -int ttm_range_man_fini_nocheck(struct ttm_device *bdev, +int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev, unsigned type) { struct ttm_resource_manager *man = ttm_manager_type(bdev, type); diff --git a/drivers/gpu/drm/ttm/ttm_sys_manager.c b/drivers/gpu/drm/ttm/ttm_sys_manager.c index 2ced169513cb..f0f026d40a69 100644 --- a/drivers/gpu/drm/ttm/ttm_sys_manager.c +++ b/drivers/gpu/drm/ttm/ttm_sys_manager.c @@ -20,6 +20,10 @@ static int ttm_sys_man_alloc(struct ttm_resource_manager *man, return 0; } +/* FIXME: Need to call drm_lru_mgr_fini. This requires a + * struct drm_device *drm parameter which need the change + * the definition of struct ttm_resource_manager_func. A + * very intrusive change. Leave it for now */ static void ttm_sys_man_free(struct ttm_resource_manager *man, struct ttm_resource *res) { @@ -32,7 +36,7 @@ static const struct ttm_resource_manager_func ttm_sys_manager_func = { .free = ttm_sys_man_free, }; -void ttm_sys_man_init(struct ttm_device *bdev) +void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev) { struct ttm_resource_manager *man = &bdev->sysman; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index cf1c1f16102a..1a0c161e9977 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -708,7 +708,7 @@ static int vmw_dma_masks(struct vmw_private *dev_priv) static int vmw_vram_manager_init(struct vmw_private *dev_priv) { int ret; - ret = ttm_range_man_init(&dev_priv->bdev, TTM_PL_VRAM, false, + ret = ttm_range_man_init(&dev_priv->drm, &dev_priv->bdev, TTM_PL_VRAM, false, dev_priv->vram_size >> PAGE_SHIFT); ttm_resource_manager_set_used(ttm_manager_type(&dev_priv->bdev, TTM_PL_VRAM), false); return ret; @@ -716,7 +716,7 @@ static int vmw_vram_manager_init(struct vmw_private *dev_priv) static void vmw_vram_manager_fini(struct vmw_private *dev_priv) { - ttm_range_man_fini(&dev_priv->bdev, TTM_PL_VRAM); + ttm_range_man_fini(&dev_priv->drm, &dev_priv->bdev, TTM_PL_VRAM); } static int vmw_setup_pci_resources(struct vmw_private *dev, diff --git a/include/drm/ttm/ttm_range_manager.h b/include/drm/ttm/ttm_range_manager.h index becdb88c4d84..33cb5016bde6 100644 --- a/include/drm/ttm/ttm_range_manager.h +++ b/include/drm/ttm/ttm_range_manager.h @@ -34,23 +34,24 @@ to_ttm_range_mgr_node(struct ttm_resource *res) return container_of(res, struct ttm_range_mgr_node, base); } -int ttm_range_man_init_nocheck(struct ttm_device *bdev, +int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev, unsigned type, bool use_tt, unsigned long p_size); -int ttm_range_man_fini_nocheck(struct ttm_device *bdev, +int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev, unsigned type); -static __always_inline int ttm_range_man_init(struct ttm_device *bdev, +static __always_inline int ttm_range_man_init(struct drm_device *drm, + struct ttm_device *bdev, unsigned int type, bool use_tt, unsigned long p_size) { BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES); - return ttm_range_man_init_nocheck(bdev, type, use_tt, p_size); + return ttm_range_man_init_nocheck(drm, bdev, type, use_tt, p_size); } -static __always_inline int ttm_range_man_fini(struct ttm_device *bdev, +static __always_inline int ttm_range_man_fini(struct drm_device *drm, struct ttm_device *bdev, unsigned int type) { BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES); - return ttm_range_man_fini_nocheck(bdev, type); + return ttm_range_man_fini_nocheck(drm, bdev, type); } #endif From patchwork Thu Nov 2 04:33:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 238F8C001B2 for ; Thu, 2 Nov 2023 04:24:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1C9FE10E80D; Thu, 2 Nov 2023 04:24:37 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7349E10E7FA; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AacnrrassvtP4cpr0jl++LRZndhOkoKuF+0l1IzrsrU=; b=WxLZo4QJeD2Do5qoyNLaFqgrXD3/73IdMd0Mf7NQpnXD0a7LP4pn9d8M YMXgXCs7lIN9od4yb6QRqNQBWjji9xeyAvKkE9Xcdichh2DUBSBwIL0FJ RWILE3XOA0jfWosvzKVQV+gTkww24tkwbFedpVIYlvAkSSEYcxRKL0ahr O19gbYVTqTeIyYaTFQhoGZTbP+3bZ+o2A2EGHJa8FPqkbuw1syVU1tS36 Yd30oTZA4E13XkXYA+3wkYaRSzH2MrKf/OBIMFibM0qDySbOcQHHC0uYi JZ6zqb5FnhR556g1t1TmQmbmVBfwu58YlabzJtoH8N9ho18892IGxDTVZ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930926" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930926" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762636" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762636" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 08/11] drm: Initialize drm lru manager Date: Thu, 2 Nov 2023 00:33:03 -0400 Message-Id: <20231102043306.2931989-9-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Initialize lru_mgr for each memory type or memory region. Also set ttm_resource_manager's weak reference to drm lru manager. Signed-off-by: Oak Zeng --- drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 ++++++ drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 ++++++++++ drivers/gpu/drm/nouveau/nouveau_ttm.c | 12 ++++++++++++ drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++++ drivers/gpu/drm/ttm/ttm_sys_manager.c | 2 ++ drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 ++++++ drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 6 ++++++ drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 6 ++++++ 10 files changed, 66 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c index 44367f03316f..57e8b1688977 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c @@ -278,6 +278,7 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size) { struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr; struct ttm_resource_manager *man = &mgr->manager; + struct drm_device *drm = adev_to_drm(adev); uint64_t start, size; man->use_tt = true; @@ -292,6 +293,9 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size) ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_TT, &mgr->manager); ttm_resource_manager_set_used(man, true); + + drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_TT], gtt_size, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_TT]); return 0; } @@ -307,6 +311,7 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev) { struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr; struct ttm_resource_manager *man = &mgr->manager; + struct drm_device *drm = adev_to_drm(adev); int ret; ttm_resource_manager_set_used(man, false); @@ -321,4 +326,5 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev) ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_TT, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[TTM_PL_TT]); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c index e8adfd0a570a..f989aca2bfc4 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c @@ -100,6 +100,7 @@ static const struct ttm_resource_manager_func amdgpu_preempt_mgr_func = { int amdgpu_preempt_mgr_init(struct amdgpu_device *adev) { struct ttm_resource_manager *man = &adev->mman.preempt_mgr; + struct drm_device *drm = adev_to_drm(adev); int ret; man->use_tt = true; @@ -115,6 +116,9 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev) ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, man); ttm_resource_manager_set_used(man, true); + + drm_lru_mgr_init(&drm->lru_mgr[AMDGPU_PL_PREEMPT], (1 << 30), &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[VMW_PL_SYSTEM]); return 0; } @@ -129,6 +133,7 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev) void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev) { struct ttm_resource_manager *man = &adev->mman.preempt_mgr; + struct drm_device *drm = adev_to_drm(adev); int ret; ttm_resource_manager_set_used(man, false); @@ -141,4 +146,5 @@ void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev) ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[AMDGPU_PL_PREEMPT]); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c index b83e1741905e..0792d22508c9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c @@ -884,6 +884,7 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev) { struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr; struct ttm_resource_manager *man = &mgr->manager; + struct drm_device *drm = adev_to_drm(adev); int err; ttm_resource_manager_init(man, &adev->mman.bdev, @@ -907,6 +908,9 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev) ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, &mgr->manager); ttm_resource_manager_set_used(man, true); + drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_VRAM], adev->gmc.real_vram_size, + &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_VRAM]); return 0; } @@ -922,6 +926,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev) { struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr; struct ttm_resource_manager *man = &mgr->manager; + struct drm_device *drm = adev_to_drm(adev); int ret; struct amdgpu_vram_reservation *rsv, *temp; @@ -945,4 +950,5 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev) ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&adev->mman.bdev, TTM_PL_VRAM, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[TTM_PL_VRAM]); } diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c index a1bc804cfa15..968f9c01152e 100644 --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c @@ -304,6 +304,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev, { struct ttm_resource_manager *man; struct i915_ttm_buddy_manager *bman; + struct drm_i915_private *i915 = container_of(bdev, + struct drm_i915_private, bdev); + struct drm_device *drm = &i915->drm; int err; bman = kzalloc(sizeof(*bman), GFP_KERNEL); @@ -329,6 +332,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev, ttm_resource_manager_set_used(man, true); ttm_set_driver_manager(bdev, type, man); + drm_lru_mgr_init(&drm->lru_mgr[type], bman->mm.size >> PAGE_SHIFT, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[type]); + return 0; err_free_bman: @@ -350,6 +356,9 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type) { struct ttm_resource_manager *man = ttm_manager_type(bdev, type); struct i915_ttm_buddy_manager *bman = to_buddy_manager(man); + struct drm_i915_private *i915 = container_of(bdev, + struct drm_i915_private, bdev); + struct drm_device *drm = &i915->drm; struct drm_buddy *mm = &bman->mm; int ret; @@ -369,6 +378,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type) mutex_unlock(&bman->lock); ttm_resource_manager_cleanup(man); + drm_lru_mgr_fini(&drm->lru_mgr[type]); kfree(bman); return 0; diff --git a/drivers/gpu/drm/nouveau/nouveau_ttm.c b/drivers/gpu/drm/nouveau/nouveau_ttm.c index 1898f27f0510..b0936c235ff6 100644 --- a/drivers/gpu/drm/nouveau/nouveau_ttm.c +++ b/drivers/gpu/drm/nouveau/nouveau_ttm.c @@ -182,6 +182,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm) { if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA) { struct ttm_resource_manager *man = kzalloc(sizeof(*man), GFP_KERNEL); + struct drm_device *drm_dev = drm->dev; if (!man) return -ENOMEM; @@ -192,6 +193,9 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm) drm->gem.vram_available >> PAGE_SHIFT); ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, man); ttm_resource_manager_set_used(man, true); + drm_lru_mgr_init(&drm_dev->lru_mgr[TTM_PL_VRAM], + drm->gem.vram_available >> PAGE_SHIFT, &drm_dev->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm_dev->lru_mgr[TTM_PL_VRAM]); return 0; } else { return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM, false, @@ -205,10 +209,13 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm) struct ttm_resource_manager *man = ttm_manager_type(&drm->ttm.bdev, TTM_PL_VRAM); if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA) { + struct drm_device *drm_dev = drm->dev; + ttm_resource_manager_set_used(man, false); ttm_resource_manager_evict_all(&drm->ttm.bdev, man); ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_VRAM, NULL); + drm_lru_mgr_fini(&drm_dev->lru_mgr[TTM_PL_VRAM]); kfree(man); } else ttm_range_man_fini(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM); @@ -220,6 +227,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm) struct ttm_resource_manager *man; unsigned long size_pages = drm->gem.gart_available >> PAGE_SHIFT; const struct ttm_resource_manager_func *func = NULL; + struct drm_device *drm_dev = drm->dev; if (drm->client.device.info.family >= NV_DEVICE_INFO_V0_TESLA) func = &nouveau_gart_manager; @@ -238,6 +246,8 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm) ttm_resource_manager_init(man, &drm->ttm.bdev, size_pages); ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_TT, man); ttm_resource_manager_set_used(man, true); + drm_lru_mgr_init(&drm_dev->lru_mgr[TTM_PL_TT], size_pages, &drm_dev->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm_dev->lru_mgr[TTM_PL_TT]); return 0; } @@ -245,6 +255,7 @@ static void nouveau_ttm_fini_gtt(struct nouveau_drm *drm) { struct ttm_resource_manager *man = ttm_manager_type(&drm->ttm.bdev, TTM_PL_TT); + struct drm_device *drm_dev = drm->dev; if (drm->client.device.info.family < NV_DEVICE_INFO_V0_TESLA && drm->agp.bridge) @@ -254,6 +265,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm) ttm_resource_manager_evict_all(&drm->ttm.bdev, man); ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&drm->ttm.bdev, TTM_PL_TT, NULL); + drm_lru_mgr_fini(&drm_dev->lru_mgr[TTM_PL_TT]); kfree(man); } } diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c index db1ae370580d..898ede7d25c4 100644 --- a/drivers/gpu/drm/ttm/ttm_range_manager.c +++ b/drivers/gpu/drm/ttm/ttm_range_manager.c @@ -36,6 +36,7 @@ #include #include #include +#include /* * Currently we use a spinlock for the lock, but a mutex *may* be @@ -199,6 +200,10 @@ int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev, ttm_set_driver_manager(bdev, type, &rman->manager); ttm_resource_manager_set_used(man, true); + + drm_lru_mgr_init(&drm->lru_mgr[type], p_size, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[type]); + return 0; } EXPORT_SYMBOL(ttm_range_man_init_nocheck); @@ -236,6 +241,7 @@ int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev, ttm_resource_manager_cleanup(man); ttm_set_driver_manager(bdev, type, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[type]); kfree(rman); return 0; } diff --git a/drivers/gpu/drm/ttm/ttm_sys_manager.c b/drivers/gpu/drm/ttm/ttm_sys_manager.c index f0f026d40a69..db410c7f73fe 100644 --- a/drivers/gpu/drm/ttm/ttm_sys_manager.c +++ b/drivers/gpu/drm/ttm/ttm_sys_manager.c @@ -50,4 +50,6 @@ void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev) ttm_resource_manager_init(man, bdev, 0); ttm_set_driver_manager(bdev, TTM_PL_SYSTEM, man); ttm_resource_manager_set_used(man, true); + drm_lru_mgr_init(&drm->lru_mgr[TTM_PL_SYSTEM], 0, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[TTM_PL_SYSTEM]); } diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c b/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c index ee7964cbdaca..102296399e00 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c @@ -62,6 +62,7 @@ int vmw_sys_man_init(struct vmw_private *dev_priv) struct ttm_device *bdev = &dev_priv->bdev; struct ttm_resource_manager *man = kzalloc(sizeof(*man), GFP_KERNEL); + struct drm_device *drm = &dev_priv->drm; if (!man) return -ENOMEM; @@ -72,6 +73,9 @@ int vmw_sys_man_init(struct vmw_private *dev_priv) ttm_resource_manager_init(man, bdev, 0); ttm_set_driver_manager(bdev, VMW_PL_SYSTEM, man); ttm_resource_manager_set_used(man, true); + + drm_lru_mgr_init(&drm->lru_mgr[WMW_PL_SYSTEM], 0, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[VMW_PL_SYSTEM]); return 0; } @@ -79,6 +83,7 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv) { struct ttm_resource_manager *man = ttm_manager_type(&dev_priv->bdev, VMW_PL_SYSTEM); + struct drm_device *drm = &dev_priv->drm; ttm_resource_manager_evict_all(&dev_priv->bdev, man); @@ -86,5 +91,6 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv) ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&dev_priv->bdev, VMW_PL_SYSTEM, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[VMW_PL_SYSTEM]); kfree(man); } diff --git a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c index 3e1fa0c832ca..ace42852a419 100644 --- a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +++ b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c @@ -96,11 +96,13 @@ static void ttm_sys_mgr_fini(struct drm_device *drm, void *arg) ttm_resource_manager_cleanup(man); ttm_set_driver_manager(&xe->ttm, XE_PL_TT, NULL); + drm_lru_mgr_fini(&drm->lru_mgr[XE_PL_TT]); } int xe_ttm_sys_mgr_init(struct xe_device *xe) { struct ttm_resource_manager *man = &xe->mem.sys_mgr; + struct drm_device *drm = &xe->drm; struct sysinfo si; u64 gtt_size; @@ -114,5 +116,9 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe) ttm_resource_manager_init(man, &xe->ttm, gtt_size >> PAGE_SHIFT); ttm_set_driver_manager(&xe->ttm, XE_PL_TT, man); ttm_resource_manager_set_used(man, true); + + drm_lru_mgr_init(&drm->lru_mgr[XE_PL_TT], gtt_size >> PAGE_SHIFT, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[XE_PL_TT]); + return drmm_add_action_or_reset(&xe->drm, ttm_sys_mgr_fini, xe); } diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c index 06a54c8bd46f..a3c1bf555c06 100644 --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c @@ -328,6 +328,8 @@ static void ttm_vram_mgr_fini(struct drm_device *dev, void *arg) ttm_resource_manager_cleanup(&mgr->manager); ttm_set_driver_manager(&xe->ttm, mgr->mem_type, NULL); + + drm_lru_mgr_fini(&dev->lru_mgr[mgr->mem_type]); } int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr, @@ -335,6 +337,7 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr, u64 default_page_size) { struct ttm_resource_manager *man = &mgr->manager; + struct drm_device *drm = &xe->drm; int err; man->func = &xe_ttm_vram_mgr_func; @@ -350,6 +353,9 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr, ttm_set_driver_manager(&xe->ttm, mem_type, &mgr->manager); ttm_resource_manager_set_used(&mgr->manager, true); + drm_lru_mgr_init(&drm->lru_mgr[mem_type], size, &drm->lru_lock); + ttm_resource_manager_set_lru_mgr(man, &drm->lru_mgr[mem_type]); + return drmm_add_action_or_reset(&xe->drm, ttm_vram_mgr_fini, mgr); } From patchwork Thu Nov 2 04:33:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD050C4167B for ; Thu, 2 Nov 2023 04:24:57 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0389310E80C; Thu, 2 Nov 2023 04:24:37 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8F5D110E800; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bxC2nAobvtmH45pcWnnEyOv5q+f4BZEE/xT/CeWnvo8=; b=DQtDAdtHi2OdTsWT0MeDfiHrR1pYM8fLYFQu2O/8SaSaCmB1MuQglJLA mWiQXCV/Wru6dwYcvFCBOKes8dY5vt55r2lKXuj+1BfA5olak4hTpsiOu JCTzLIhLRQtpajOoK1r+vvh4FTHJUymRwwro8K6HGCHGRThctp1vKNLo1 TwuvUVG8/FAWKWqzYMO7LHtPgodLtfv8XLmaiIOJMFI1i0vQNeMBoT+ap le3aFgh8I1msh/PEMfiqnYjg/Pj9DeTdK7v6hsuswX1TrQBkxzLqPYY02 JnZtOx4LMxB31BlLFbnEoHkLMGk3BX8nmuc3mgw5GepD7YQ5jOuLQPLk0 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930930" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930930" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762639" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762639" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 09/11] drm/ttm: Use drm LRU manager iterator Date: Thu, 2 Nov 2023 00:33:04 -0400 Message-Id: <20231102043306.2931989-10-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since TTM resource LRU list is moved to drm LRU manager layer, use drm lru manager iterator instead of TTM resource manager iterator. TTM resource manager iterator is deleted. No function change. Signed-off-by: Oak Zeng --- drivers/gpu/drm/ttm/ttm_bo.c | 7 ++-- drivers/gpu/drm/ttm/ttm_device.c | 10 ++++-- drivers/gpu/drm/ttm/ttm_resource.c | 51 ------------------------------ include/drm/ttm/ttm_resource.h | 33 ++----------------- 4 files changed, 14 insertions(+), 87 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index 26e0555bad0c..4a5ffa920665 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -43,6 +43,7 @@ #include #include #include +#include #include "ttm_module.h" @@ -593,15 +594,17 @@ int ttm_mem_evict_first(struct ttm_device *bdev, struct ww_acquire_ctx *ticket) { struct ttm_buffer_object *bo = NULL, *busy_bo = NULL; - struct ttm_resource_cursor cursor; + struct drm_lru_cursor cursor; struct ttm_resource *res; + struct drm_lru_entity *entity; bool locked = false; int ret; spin_lock(bdev->lru_lock); - ttm_resource_manager_for_each_res(man, &cursor, res) { + drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) { bool busy; + res = container_of(entity, struct ttm_resource, lru_entity); if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place, &locked, &busy)) { if (busy && !busy_bo && ticket != diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 393c3e27016e..881662d69aba 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "ttm_module.h" @@ -141,7 +142,8 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, gfp_t gfp_flags) { - struct ttm_resource_cursor cursor; + struct drm_lru_cursor cursor; + struct drm_lru_entity *entity; struct ttm_resource_manager *man; struct ttm_resource *res; unsigned i; @@ -153,10 +155,12 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, if (!man || !man->use_tt) continue; - ttm_resource_manager_for_each_res(man, &cursor, res) { - struct ttm_buffer_object *bo = res->bo; + drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) { + struct ttm_buffer_object *bo; uint32_t num_pages; + res = container_of(entity, struct ttm_resource, lru_entity); + bo = res->bo; if (!bo || bo->resource != res) continue; diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 05eef866065e..0c6e0dbeff07 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -488,57 +488,6 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man, } EXPORT_SYMBOL(ttm_resource_manager_debug); -/** - * ttm_resource_manager_first - * - * @man: resource manager to iterate over - * @cursor: cursor to record the position - * - * Returns the first resource from the resource manager. - */ -struct ttm_resource * -ttm_resource_manager_first(struct ttm_resource_manager *man, - struct ttm_resource_cursor *cursor) -{ - struct ttm_resource *res; - - lockdep_assert_held(man->bdev->lru_lock); - - for (cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY; - ++cursor->priority) - list_for_each_entry(res, &man->lru[cursor->priority], lru) - return res; - - return NULL; -} - -/** - * ttm_resource_manager_next - * - * @man: resource manager to iterate over - * @cursor: cursor to record the position - * @res: the current resource pointer - * - * Returns the next resource from the resource manager. - */ -struct ttm_resource * -ttm_resource_manager_next(struct ttm_resource_manager *man, - struct ttm_resource_cursor *cursor, - struct ttm_resource *res) -{ - lockdep_assert_held(man->bdev->lru_lock); - - list_for_each_entry_continue(res, &man->lru[cursor->priority], lru) - return res; - - for (++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY; - ++cursor->priority) - list_for_each_entry(res, &man->lru[cursor->priority], lru) - return res; - - return NULL; -} - static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter, struct iosys_map *dmap, pgoff_t i) diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index e4fc1ada5236..c2528cec12e6 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -207,6 +207,7 @@ struct ttm_bus_placement { * @placement: Placement flags. * @bus: Placement on io bus accessible to the CPU * @bo: weak reference to the BO, protected by ttm_device::lru_lock + * @lru_entity: lru entity for this ttm resource. * * Structure indicating the placement and space resources used by a * buffer object. @@ -223,17 +224,7 @@ struct ttm_resource { * @lru: Least recently used list, see &ttm_resource_manager.lru */ struct list_head lru; -}; - -/** - * struct ttm_resource_cursor - * - * @priority: the current priority - * - * Cursor to iterate over the resources in a manager. - */ -struct ttm_resource_cursor { - unsigned int priority; + struct drm_lru_entity lru_entity; }; /** @@ -402,26 +393,6 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man); void ttm_resource_manager_debug(struct ttm_resource_manager *man, struct drm_printer *p); -struct ttm_resource * -ttm_resource_manager_first(struct ttm_resource_manager *man, - struct ttm_resource_cursor *cursor); -struct ttm_resource * -ttm_resource_manager_next(struct ttm_resource_manager *man, - struct ttm_resource_cursor *cursor, - struct ttm_resource *res); - -/** - * ttm_resource_manager_for_each_res - iterate over all resources - * @man: the resource manager - * @cursor: struct ttm_resource_cursor for the current position - * @res: the current resource - * - * Iterate over all the evictable resources in a resource manager. - */ -#define ttm_resource_manager_for_each_res(man, cursor, res) \ - for (res = ttm_resource_manager_first(man, cursor); res; \ - res = ttm_resource_manager_next(man, cursor, res)) - struct ttm_kmap_iter * ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io, struct io_mapping *iomap, From patchwork Thu Nov 2 04:33:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F53CC0018C for ; Thu, 2 Nov 2023 04:25:01 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4D19C10E815; Thu, 2 Nov 2023 04:24:39 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id A388E10E801; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nh/H1JzDMpGQr1G+PCKAigkrHf6LP1Kp/ev0gnoHg6E=; b=nxzxZmdpb3aK0TbzXTriTjW3Tzwa3w9Jhq14w0zYvufwMtwyzk6PC0pj VSKPZG3wJ00PHgs+fC0HyLwILF9Zs6beCmhQ30iyHxan2Yd3Vv9MUZH40 NLdKOVQmjxsXRQ8geMRgaf1lWoOAfqKluTA4DHrhBh72+nDbtYjKxdvHu J0FxmUASjbo8/isvCUKmh6ntXVI5LAU8m4BpyFa5+kYsFpo3yFRxtW7gt BjaGecEEoEgXb20X2dC7Nr3YL8vZvEM1wLxGnl3BUAQQPnY3UDL2JE6Wv 6RvMQHJB89QaLKxB4Dn5boUsVQ6OLliGemAx5Ke++ZscrDBEPqPsU2bIV g==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930933" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930933" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762641" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762641" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 10/11] drm/ttm: Implement ttm memory evict functions Date: Thu, 2 Nov 2023 00:33:05 -0400 Message-Id: <20231102043306.2931989-11-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Implement ttm_mem_evict_valuable, ttm_mem_evict_entity and ttm_mem_evict_busy_entity. Those are callback functions from drm lru manager. Register those functions during drm lru entity initialization. Those 3 functions are splitted from original ttm_mem_evict_first function. Reimplemented ttm_mem_evict_first function using drm_lru_evict_first function. For now, drm_lru_evict_first just calls back to above 3 functions which are splitted from ttm_mem_evict_first function, so there is no function change. In the future, when SVM code is added, drm_lru_evict_first function can also calls into SVM resource eviction functions, thus TTM and SVM can mutually evict resources from each other. Signed-off-by: Oak Zeng --- drivers/gpu/drm/ttm/ttm_bo.c | 192 ++++++++++++++++++++++++++++------- include/drm/ttm/ttm_bo.h | 2 + 2 files changed, 158 insertions(+), 36 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index 4a5ffa920665..9ec7a246e2ad 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -587,50 +587,148 @@ static int ttm_mem_evict_wait_busy(struct ttm_buffer_object *busy_bo, return r == -EDEADLK ? -EBUSY : r; } -int ttm_mem_evict_first(struct ttm_device *bdev, - struct ttm_resource_manager *man, - const struct ttm_place *place, - struct ttm_operation_ctx *ctx, - struct ww_acquire_ctx *ticket) +struct ttm_mem_evict_ctx { + const struct ttm_place *place; + struct ttm_operation_ctx *ctx; + struct ww_acquire_ctx *ticket; +}; + +/** + * ttm_mem_evict_allowable + * + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @place: The preferred ttm placement where we want to evict memory for + * more memory space. If the current ttm_resource doesn't match the preferred + * placement, then there is no need to evict the current resource. + * @ctx: ttm operation context + * @ticket: dma reservation's context used to lock resource + * @busy: used to return whether the current resource is busy (i.e., locked + * by other clients) + * @locked: used to return whether this resource is locked during this check, + * i.e., successfully trylocked bo's dma reservation object + * + * Check whether we are allowed to evict a memory resource. Return true if we + * are allowed to evict resource; otherwise false. + * + * When this function returns true, a resource reference counter (bo's reference) + * is hold. This reference counter need to be released after evict operation later + * on. + * + * This function should be called with lru_lock hold. + */ +bool ttm_mem_evict_allowable(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *lru_evict_ctx, + bool *busy, bool *locked) { - struct ttm_buffer_object *bo = NULL, *busy_bo = NULL; - struct drm_lru_cursor cursor; struct ttm_resource *res; - struct drm_lru_entity *entity; - bool locked = false; - int ret; + struct ttm_buffer_object *bo = NULL; + struct ttm_device *bdev; + const struct ttm_place *place; + struct ttm_operation_ctx *ctx; + struct ww_acquire_ctx *ticket; + struct ttm_mem_evict_ctx *evict_ctx; - spin_lock(bdev->lru_lock); - drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) { - bool busy; + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + place = evict_ctx->place; + ctx = evict_ctx->ctx; + ticket = evict_ctx->ticket; - res = container_of(entity, struct ttm_resource, lru_entity); - if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place, - &locked, &busy)) { - if (busy && !busy_bo && ticket != - dma_resv_locking_ctx(res->bo->base.resv)) - busy_bo = res->bo; - continue; - } + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; - if (ttm_bo_get_unless_zero(res->bo)) { - bo = res->bo; - break; - } - if (locked) - dma_resv_unlock(res->bo->base.resv); - } + if (!ttm_bo_evict_swapout_allowable(bo, ctx, place, locked, busy)) { + if (busy && ticket != dma_resv_locking_ctx(bo->base.resv)) + *busy = true; - if (!bo) { - if (busy_bo && !ttm_bo_get_unless_zero(busy_bo)) - busy_bo = NULL; - spin_unlock(bdev->lru_lock); - ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket); - if (busy_bo) - ttm_bo_put(busy_bo); - return ret; + return false; } + if (ttm_bo_get_unless_zero(bo)) + return true; + + if (locked) + dma_resv_unlock(bo->base.resv); + + return false; +} + +/** + * ttm_mem_evict_busy_entity + * + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @ctx: ttm operation context + * @ticket: dma reservation's context used to lock resource + * + * Evict a busy memory resource. + * This function should be called with lru_lock hold. + */ +int ttm_mem_evict_busy_entity(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *lru_evict_ctx) +{ + struct ttm_resource *res; + struct ttm_buffer_object *bo = NULL; + struct ttm_device *bdev; + int ret; + struct ttm_operation_ctx *ctx; + struct ww_acquire_ctx *ticket; + struct ttm_mem_evict_ctx *evict_ctx; + + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + ctx = evict_ctx->ctx; + ticket = evict_ctx->ticket; + + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; + + if (bo && !ttm_bo_get_unless_zero(bo)) + bo = NULL; + spin_unlock(bdev->lru_lock); + ret = ttm_mem_evict_wait_busy(bo, ctx, ticket); + /* FIXME: this is code copied originally from ttm_mem_evict_first. + * 1) Shouldn't we ttm_bo_evict this bo also? Otherwise how can we + * make any memory space? + * 2) We also need to check whether this busy entity is in the same + * ttm_place as specified in lru_evict_ctx::place; if not, there is + * no help to wait this busy entity. + */ + if (bo) + ttm_bo_put(bo); + + return ret; +} + +/** + * @lru_entity: The struct ttm_resource::lru_entity when this resource is + * added to drm lru list. + * @ctx: ttm operation context + * @locked: whether this resource is dma-reserved (if reserved, we need to + * unreserve it in this function) + * + * Evict a memory resource corresponding to a lru_entity. This should be + * called holding lru_lock + * + */ +int ttm_mem_evict_entity(struct drm_lru_entity *lru_entity, + const struct drm_lru_evict_ctx *lru_evict_ctx, bool locked) +{ + struct ttm_resource *res; + struct ttm_buffer_object *bo = NULL; + struct ttm_device *bdev; + int ret; + struct ttm_operation_ctx *ctx; + struct ttm_mem_evict_ctx *evict_ctx; + + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + ctx = evict_ctx->ctx; + + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; + if (bo->deleted) { ret = ttm_bo_cleanup_refs(bo, ctx->interruptible, ctx->no_wait_gpu, locked); @@ -650,6 +748,28 @@ int ttm_mem_evict_first(struct ttm_device *bdev, return ret; } +struct drm_lru_evict_func ttm_evict_func = { + .evict_allowable = ttm_mem_evict_allowable, + .evict_busy_entity = ttm_mem_evict_busy_entity, + .evict_entity = ttm_mem_evict_entity +}; +EXPORT_SYMBOL(ttm_evict_func); + +int ttm_mem_evict_first(struct ttm_device *bdev, + struct ttm_resource_manager *man, + const struct ttm_place *place, + struct ttm_operation_ctx *ctx, + struct ww_acquire_ctx *ticket) +{ + struct drm_lru_evict_ctx evict_ctx = { + .data1 = place, + .data2 = ctx, + .data3 = ticket + }; + + return drm_lru_evict_first(man->lru_mgr, &evict_ctx); +} + /** * ttm_bo_pin - Pin the buffer object. * @bo: The buffer object to pin diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h index 49f32df32204..223b198fe371 100644 --- a/include/drm/ttm/ttm_bo.h +++ b/include/drm/ttm/ttm_bo.h @@ -50,6 +50,7 @@ struct ttm_place; struct ttm_resource; struct ttm_resource_manager; struct ttm_tt; +struct drm_lru_evict_func; /** * enum ttm_bo_type @@ -424,4 +425,5 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp); void ttm_bo_tt_destroy(struct ttm_buffer_object *bo); +extern struct drm_lru_evict_func ttm_evict_func; #endif From patchwork Thu Nov 2 04:33:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zeng, Oak" X-Patchwork-Id: 13443399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62008C0018A for ; Thu, 2 Nov 2023 04:25:00 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F315910E812; Thu, 2 Nov 2023 04:24:38 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id BF36C10E7F9; Thu, 2 Nov 2023 04:24:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698899072; x=1730435072; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4PqUq6kMXcSWQgJfyfvlfN+oe1xQjTtJBgXUXi0IZtw=; b=kiWLSm4ZRft6IcF5DWDPMvLOQv8yI0LaJM1D+qGSWMoGdBX8YO9Z6Sbj w7LCRgotEf4hYOde0E3b0WS/FBi/n4ss437Ci4JUGncdYqLa62jm90N9c e9otI4WxIg+UDcHb1lcNEv3eZLY9IDty/1oRmOEDh9ryFSSHKpqqwS/PB WRnOLlx7fVeTHD7RssmZhK4+qj978/ZMegjIr1Xhzpudw9+ayGTqWCHuh Dz8077ZFrd1ewHFonKkZKs7dlZz7adPRAOgxwR8l/I+b8jBxQLBXpPGRt 68zBmW5yhruVaBZ3KUFqyhyCC3xEG9OgOx9DB42fnvoIqu/eE9XPB88W3 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="452930935" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="452930935" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10881"; a="884762644" X-IronPort-AV: E=Sophos;i="6.03,270,1694761200"; d="scan'208";a="884762644" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2023 21:24:30 -0700 From: Oak Zeng To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC 11/11] drm/ttm: Write ttm functions using drm lru manager functions Date: Thu, 2 Nov 2023 00:33:06 -0400 Message-Id: <20231102043306.2931989-12-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20231102043306.2931989-1-oak.zeng@intel.com> References: <20231102043306.2931989-1-oak.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas.Hellstrom@linux.intel.com, felix.kuehling@amd.com, brian.welty@intel.com, christian.koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Replace struct ttm_resource::lru with drm lru entity. Replace struct ttm_resource_manager::lru[] with drm lru manager. Remove ttm_lru_bulk_move functions and definitions as those are moved to drm lru manager. Some of ttm resource, ttm bo and ttm device functions are re-written using drm lru manager functions. Signed-off-by: Oak Zeng --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 20 +-- drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +-- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 10 +- drivers/gpu/drm/ttm/ttm_range_manager.c | 2 +- drivers/gpu/drm/ttm/ttm_resource.c | 155 ++++---------------- drivers/gpu/drm/xe/xe_bo.c | 44 +++--- drivers/gpu/drm/xe/xe_bo.h | 3 +- drivers/gpu/drm/xe/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 6 +- drivers/gpu/drm/xe/xe_res_cursor.h | 10 +- drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 2 +- drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 12 +- drivers/gpu/drm/xe/xe_vm.c | 2 +- drivers/gpu/drm/xe/xe_vm_types.h | 2 +- include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_resource.h | 59 ++------ 21 files changed, 112 insertions(+), 253 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 747bcad86d5d..c977c00e986a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -369,7 +369,7 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev, struct amdgpu_vm *vm) { spin_lock(adev->mman.bdev.lru_lock); - ttm_lru_bulk_move_tail(&vm->lru_bulk_move); + drm_lru_bulk_move_tail(&vm->lru_bulk_move); spin_unlock(adev->mman.bdev.lru_lock); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 204ab13184ed..fec545b5d154 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -337,7 +337,7 @@ struct amdgpu_vm { struct amdgpu_task_info task_info; /* Store positions of group of BOs */ - struct ttm_lru_bulk_move lru_bulk_move; + struct drm_lru_bulk_move lru_bulk_move; /* Flag to indicate if VM is used for compute */ bool is_compute_context; diff --git a/drivers/gpu/drm/ttm/tests/ttm_device_test.c b/drivers/gpu/drm/ttm/tests/ttm_device_test.c index b1b423b68cdf..a62ca31b55df 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_device_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_device_test.c @@ -90,7 +90,7 @@ static void ttm_device_fini_basic(struct kunit *test) ttm_device_fini(ttm_dev); KUNIT_ASSERT_FALSE(test, man->use_type); - KUNIT_ASSERT_TRUE(test, list_empty(&man->lru[0])); + KUNIT_ASSERT_TRUE(test, list_empty(&man->lru_mgr->lru[0])); KUNIT_ASSERT_NULL(test, ttm_dev->man_drv[TTM_PL_SYSTEM]); } diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index 9ec7a246e2ad..d44ca5e51dff 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -92,11 +92,11 @@ EXPORT_SYMBOL(ttm_bo_move_to_lru_tail); * resulting in much less overhead of maintaining the LRU. * The only requirement is that the resources stay together on the LRU and are * never separated. This is enforces by setting the bulk_move structure on a BO. - * ttm_lru_bulk_move_tail() should be used to move all resources to the tail of + * drm_lru_bulk_move_tail() should be used to move all resources to the tail of * their LRU list. */ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo, - struct ttm_lru_bulk_move *bulk) + struct drm_lru_bulk_move *bulk) { dma_resv_assert_held(bo->base.resv); @@ -122,8 +122,8 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, bool old_use_tt, new_use_tt; int ret; - old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt; - new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt; + old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->lru_entity.mem_type)->use_tt; + new_use_tt = ttm_manager_type(bdev, mem->lru_entity.mem_type)->use_tt; ttm_bo_unmap_virtual(bo); @@ -139,7 +139,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, if (ret) goto out_err; - if (mem->mem_type != TTM_PL_SYSTEM) { + if (mem->lru_entity.mem_type != TTM_PL_SYSTEM) { ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx); if (ret) goto out_err; @@ -492,7 +492,7 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo, struct ttm_device *bdev = bo->bdev; dma_resv_assert_held(bo->base.resv); - if (bo->resource->mem_type == TTM_PL_SYSTEM) + if (bo->resource->lru_entity.mem_type == TTM_PL_SYSTEM) return true; /* Don't evict this BO if it's outside of the @@ -540,7 +540,7 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo, *busy = !ret; } - if (ret && place && (bo->resource->mem_type != place->mem_type || + if (ret && place && (bo->resource->lru_entity.mem_type != place->mem_type || !bo->bdev->funcs->eviction_valuable(bo, place))) { ret = false; if (*locked) { @@ -1039,7 +1039,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo, /* * We might need to add a TTM. */ - if (!bo->resource || bo->resource->mem_type == TTM_PL_SYSTEM) { + if (!bo->resource || bo->resource->lru_entity.mem_type == TTM_PL_SYSTEM) { ret = ttm_tt_create(bo, true); if (ret) return ret; @@ -1259,7 +1259,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, * as an indication that we're about to swap out. */ memset(&place, 0, sizeof(place)); - place.mem_type = bo->resource->mem_type; + place.mem_type = bo->resource->lru_entity.mem_type; if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL)) return -EBUSY; @@ -1284,7 +1284,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, /* * Move to system cached */ - if (bo->resource->mem_type != TTM_PL_SYSTEM) { + if (bo->resource->lru_entity.mem_type != TTM_PL_SYSTEM) { struct ttm_resource *evict_mem; struct ttm_place hop; diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c index fd9fd3d15101..7176fbfca5eb 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_util.c +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c @@ -145,7 +145,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, { struct ttm_device *bdev = bo->bdev; struct ttm_resource_manager *dst_man = - ttm_manager_type(bo->bdev, dst_mem->mem_type); + ttm_manager_type(bo->bdev, dst_mem->lru_entity.mem_type); struct ttm_tt *ttm = bo->ttm; struct ttm_resource *src_mem = bo->resource; struct ttm_resource_manager *src_man; @@ -160,7 +160,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, if (WARN_ON(!src_mem)) return -EINVAL; - src_man = ttm_manager_type(bdev, src_mem->mem_type); + src_man = ttm_manager_type(bdev, src_mem->lru_entity.mem_type); if (ttm && ((ttm->page_flags & TTM_TT_FLAG_SWAPPED) || dst_man->use_tt)) { ret = ttm_tt_populate(bdev, ttm, ctx); @@ -184,7 +184,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, clear = src_iter->ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm)); if (!(clear && ttm && !(ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC))) - ttm_move_memcpy(clear, PFN_UP(dst_mem->size), dst_iter, src_iter); + ttm_move_memcpy(clear, PFN_UP(dst_mem->lru_entity.size), dst_iter, src_iter); if (!src_iter->ops->maps_tt) ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, src_mem); @@ -293,7 +293,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, struct ttm_resource_manager *man; enum ttm_caching caching; - man = ttm_manager_type(bo->bdev, res->mem_type); + man = ttm_manager_type(bo->bdev, res->lru_entity.mem_type); caching = man->use_tt ? bo->ttm->caching : res->bus.caching; return ttm_prot_from_caching(caching, tmp); @@ -393,9 +393,9 @@ int ttm_bo_kmap(struct ttm_buffer_object *bo, map->virtual = NULL; map->bo = bo; - if (num_pages > PFN_UP(bo->resource->size)) + if (num_pages > PFN_UP(bo->resource->lru_entity.size)) return -EINVAL; - if ((start_page + num_pages) > PFN_UP(bo->resource->size)) + if ((start_page + num_pages) > PFN_UP(bo->resource->lru_entity.size)) return -EINVAL; ret = ttm_mem_io_reserve(bo->bdev, bo->resource); @@ -607,7 +607,7 @@ static void ttm_bo_move_pipeline_evict(struct ttm_buffer_object *bo, struct ttm_device *bdev = bo->bdev; struct ttm_resource_manager *from; - from = ttm_manager_type(bdev, bo->resource->mem_type); + from = ttm_manager_type(bdev, bo->resource->lru_entity.mem_type); /** * BO doesn't have a TTM we need to bind/unbind. Just remember @@ -646,8 +646,8 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo, struct ttm_resource *new_mem) { struct ttm_device *bdev = bo->bdev; - struct ttm_resource_manager *from = ttm_manager_type(bdev, bo->resource->mem_type); - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type); + struct ttm_resource_manager *from = ttm_manager_type(bdev, bo->resource->lru_entity.mem_type); + struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->lru_entity.mem_type); int ret = 0; dma_resv_add_fence(bo->base.resv, fence, DMA_RESV_USAGE_KERNEL); @@ -680,7 +680,7 @@ void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo, struct ttm_resource *new_mem) { struct ttm_device *bdev = bo->bdev; - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type); + struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->lru_entity.mem_type); int ret; ret = ttm_bo_wait_free_node(bo, man->use_tt); diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 4212b8c91dd4..0e550430fa85 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -421,7 +421,7 @@ int ttm_bo_vm_access(struct vm_area_struct *vma, unsigned long addr, if (ret) return ret; - switch (bo->resource->mem_type) { + switch (bo->resource->lru_entity.mem_type) { case TTM_PL_SYSTEM: fallthrough; case TTM_PL_TT: diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 881662d69aba..7e5bfdffc08d 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -250,7 +250,7 @@ void ttm_device_fini(struct ttm_device *bdev) spin_lock(bdev->lru_lock); for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) - if (list_empty(&man->lru[0])) + if (list_empty(&man->lru_mgr->lru[0])) pr_debug("Swap list %d was clean\n", i); spin_unlock(bdev->lru_lock); @@ -263,16 +263,18 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, struct list_head *list) { struct ttm_resource *res; + struct drm_lru_entity *entity; spin_lock(bdev->lru_lock); - while ((res = list_first_entry_or_null(list, typeof(*res), lru))) { + while ((entity = list_first_entry_or_null(list, typeof(*entity), lru))){ + res = container_of(entity, struct ttm_resource, lru_entity); struct ttm_buffer_object *bo = res->bo; /* Take ref against racing releases once lru_lock is unlocked */ if (!ttm_bo_get_unless_zero(bo)) continue; - list_del_init(&res->lru); + list_del_init(&entity->lru); spin_unlock(bdev->lru_lock); if (bo->ttm) @@ -297,7 +299,7 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev) continue; for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) - ttm_device_clear_lru_dma_mappings(bdev, &man->lru[j]); + ttm_device_clear_lru_dma_mappings(bdev, &man->lru_mgr->lru[j]); } } EXPORT_SYMBOL(ttm_device_clear_dma_mappings); diff --git a/drivers/gpu/drm/ttm/ttm_range_manager.c b/drivers/gpu/drm/ttm/ttm_range_manager.c index 898ede7d25c4..afdea64ffc10 100644 --- a/drivers/gpu/drm/ttm/ttm_range_manager.c +++ b/drivers/gpu/drm/ttm/ttm_range_manager.c @@ -84,7 +84,7 @@ static int ttm_range_man_alloc(struct ttm_resource_manager *man, spin_lock(&rman->lock); ret = drm_mm_insert_node_in_range(mm, &node->mm_nodes[0], - PFN_UP(node->base.size), + PFN_UP(node->base.lru_entity.size), bo->page_alignment, 0, place->fpfn, lpfn, mode); spin_unlock(&rman->lock); diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 0c6e0dbeff07..b6ff3b9e0614 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -30,108 +30,12 @@ #include #include -/** - * ttm_lru_bulk_move_init - initialize a bulk move structure - * @bulk: the structure to init - * - * For now just memset the structure to zero. - */ -void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk) -{ - memset(bulk, 0, sizeof(*bulk)); -} -EXPORT_SYMBOL(ttm_lru_bulk_move_init); - -/** - * ttm_lru_bulk_move_tail - bulk move range of resources to the LRU tail. - * - * @bulk: bulk move structure - * - * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that - * resource order never changes. Should be called with &drm_device.lru_lock held. - */ -void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk) -{ - unsigned i, j; - - for (i = 0; i < DRM_NUM_MEM_TYPES; ++i) { - for (j = 0; j < DRM_MAX_LRU_PRIORITY; ++j) { - struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j]; - struct ttm_resource_manager *man; - - if (!pos->first) - continue; - - lockdep_assert_held(pos->first->bo->bdev->lru_lock); - dma_resv_assert_held(pos->first->bo->base.resv); - dma_resv_assert_held(pos->last->bo->base.resv); - - man = ttm_manager_type(pos->first->bo->bdev, i); - list_bulk_move_tail(&man->lru[j], &pos->first->lru, - &pos->last->lru); - } - } -} -EXPORT_SYMBOL(ttm_lru_bulk_move_tail); - -/* Return the bulk move pos object for this resource */ -static struct ttm_lru_bulk_move_pos * -ttm_lru_bulk_move_pos(struct ttm_lru_bulk_move *bulk, struct ttm_resource *res) -{ - return &bulk->pos[res->mem_type][res->bo->priority]; -} - -/* Move the resource to the tail of the bulk move range */ -static void ttm_lru_bulk_move_pos_tail(struct ttm_lru_bulk_move_pos *pos, - struct ttm_resource *res) -{ - if (pos->last != res) { - if (pos->first == res) - pos->first = list_next_entry(res, lru); - list_move(&res->lru, &pos->last->lru); - pos->last = res; - } -} - -/* Add the resource to a bulk_move cursor */ -static void ttm_lru_bulk_move_add(struct ttm_lru_bulk_move *bulk, - struct ttm_resource *res) -{ - struct ttm_lru_bulk_move_pos *pos = ttm_lru_bulk_move_pos(bulk, res); - - if (!pos->first) { - pos->first = res; - pos->last = res; - } else { - ttm_lru_bulk_move_pos_tail(pos, res); - } -} - -/* Remove the resource from a bulk_move range */ -static void ttm_lru_bulk_move_del(struct ttm_lru_bulk_move *bulk, - struct ttm_resource *res) -{ - struct ttm_lru_bulk_move_pos *pos = ttm_lru_bulk_move_pos(bulk, res); - - if (unlikely(WARN_ON(!pos->first || !pos->last) || - (pos->first == res && pos->last == res))) { - pos->first = NULL; - pos->last = NULL; - } else if (pos->first == res) { - pos->first = list_next_entry(res, lru); - } else if (pos->last == res) { - pos->last = list_prev_entry(res, lru); - } else { - list_move(&res->lru, &pos->last->lru); - } -} - /* Add the resource to a bulk move if the BO is configured for it */ void ttm_resource_add_bulk_move(struct ttm_resource *res, struct ttm_buffer_object *bo) { if (bo->bulk_move && !bo->pin_count) - ttm_lru_bulk_move_add(bo->bulk_move, res); + drm_lru_add_bulk_move(&res->lru_entity, bo->bulk_move); } /* Remove the resource from a bulk move if the BO is configured for it */ @@ -139,7 +43,7 @@ void ttm_resource_del_bulk_move(struct ttm_resource *res, struct ttm_buffer_object *bo) { if (bo->bulk_move && !bo->pin_count) - ttm_lru_bulk_move_del(bo->bulk_move, res); + drm_lru_del_bulk_move(&res->lru_entity, bo->bulk_move); } /* Move a resource to the LRU or bulk tail */ @@ -150,20 +54,16 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res) lockdep_assert_held(bo->bdev->lru_lock); - if (bo->pin_count) { - list_move_tail(&res->lru, &bdev->pinned); + if (bo->pin_count) + list_move_tail(&res->lru_entity.lru, &bdev->pinned); - } else if (bo->bulk_move) { - struct ttm_lru_bulk_move_pos *pos = - ttm_lru_bulk_move_pos(bo->bulk_move, res); + else if (bo->bulk_move) { + struct drm_lru_bulk_move_range *range = + &bo->bulk_move->range[res->lru_entity.mem_type][res->lru_entity.priority]; - ttm_lru_bulk_move_pos_tail(pos, res); - } else { - struct ttm_resource_manager *man; - - man = ttm_manager_type(bdev, res->mem_type); - list_move_tail(&res->lru, &man->lru[bo->priority]); - } + drm_lru_bulk_move_range_tail(range, &res->lru_entity); + } else + drm_lru_move_to_tail(&res->lru_entity); } /** @@ -181,22 +81,22 @@ void ttm_resource_init(struct ttm_buffer_object *bo, struct ttm_resource_manager *man; res->start = 0; - res->size = bo->base.size; - res->mem_type = place->mem_type; res->placement = place->flags; res->bus.addr = NULL; res->bus.offset = 0; res->bus.is_iomem = false; res->bus.caching = ttm_cached; res->bo = bo; + drm_lru_entity_init(&res->lru_entity, bo->base.dev, place->mem_type, + bo->base.size, bo->priority, &ttm_evict_func); man = ttm_manager_type(bo->bdev, place->mem_type); spin_lock(bo->bdev->lru_lock); if (bo->pin_count) - list_add_tail(&res->lru, &bo->bdev->pinned); + list_add_tail(&res->lru_entity.lru, &bo->bdev->pinned); else - list_add_tail(&res->lru, &man->lru[bo->priority]); - man->usage += res->size; + drm_lru_add_entity(&res->lru_entity, man->lru_mgr, bo->priority); + man->usage += res->lru_entity.size; spin_unlock(bo->bdev->lru_lock); } EXPORT_SYMBOL(ttm_resource_init); @@ -217,8 +117,8 @@ void ttm_resource_fini(struct ttm_resource_manager *man, struct ttm_device *bdev = man->bdev; spin_lock(bdev->lru_lock); - list_del_init(&res->lru); - man->usage -= res->size; + drm_lru_remove_entity(&res->lru_entity, man->lru_mgr); + man->usage -= res->lru_entity.size; spin_unlock(bdev->lru_lock); } EXPORT_SYMBOL(ttm_resource_fini); @@ -251,7 +151,7 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res) spin_lock(bo->bdev->lru_lock); ttm_resource_del_bulk_move(*res, bo); spin_unlock(bo->bdev->lru_lock); - man = ttm_manager_type(bo->bdev, (*res)->mem_type); + man = ttm_manager_type(bo->bdev, (*res)->lru_entity.mem_type); man->func->free(man, *res); *res = NULL; } @@ -280,7 +180,7 @@ bool ttm_resource_intersects(struct ttm_device *bdev, if (!res) return false; - man = ttm_manager_type(bdev, res->mem_type); + man = ttm_manager_type(bdev, res->lru_entity.mem_type); if (!place || !man->func->intersects) return true; @@ -309,7 +209,7 @@ bool ttm_resource_compatible(struct ttm_device *bdev, if (!res || !place) return false; - man = ttm_manager_type(bdev, res->mem_type); + man = ttm_manager_type(bdev, res->lru_entity.mem_type); if (!man->func->compatible) return true; @@ -333,7 +233,7 @@ static bool ttm_resource_places_compat(struct ttm_resource *res, if (!ttm_resource_compatible(bdev, res, heap, bo->base.size)) continue; - if ((res->mem_type == heap->mem_type) && + if ((res->lru_entity.mem_type == heap->mem_type) && (!(heap->flags & TTM_PL_FLAG_CONTIGUOUS) || (res->placement & TTM_PL_FLAG_CONTIGUOUS))) return true; @@ -386,15 +286,10 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man, struct ttm_device *bdev, uint64_t size) { - unsigned i; - spin_lock_init(&man->move_lock); man->bdev = bdev; man->size = size; man->usage = 0; - - for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) - INIT_LIST_HEAD(&man->lru[i]); man->move = NULL; } EXPORT_SYMBOL(ttm_resource_manager_init); @@ -426,7 +321,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, spin_lock(bdev->lru_lock); for (i = 0; i < DRM_MAX_LRU_PRIORITY; ++i) { - while (!list_empty(&man->lru[i])) { + while (!list_empty(&man->lru_mgr->lru[i])) { spin_unlock(bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx, NULL); @@ -622,10 +517,10 @@ ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io, if (mem->bus.caching == ttm_write_combined) iosys_map_set_vaddr_iomem(&iter_io->dmap, ioremap_wc(mem->bus.offset, - mem->size)); + mem->lru_entity.size)); else if (mem->bus.caching == ttm_cached) iosys_map_set_vaddr(&iter_io->dmap, - memremap(mem->bus.offset, mem->size, + memremap(mem->bus.offset, mem->lru_entity.size, MEMREMAP_WB | MEMREMAP_WT | MEMREMAP_WC)); @@ -634,7 +529,7 @@ ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io, if (iosys_map_is_null(&iter_io->dmap)) iosys_map_set_vaddr_iomem(&iter_io->dmap, ioremap(mem->bus.offset, - mem->size)); + mem->lru_entity.size)); if (iosys_map_is_null(&iter_io->dmap)) { ret = -ENOMEM; diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index 827f798cccc0..1cae43532eac 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -61,12 +61,12 @@ bool mem_type_is_vram(u32 mem_type) static bool resource_is_stolen_vram(struct xe_device *xe, struct ttm_resource *res) { - return res->mem_type == XE_PL_STOLEN && IS_DGFX(xe); + return res->lru_entity.mem_type == XE_PL_STOLEN && IS_DGFX(xe); } static bool resource_is_vram(struct ttm_resource *res) { - return mem_type_is_vram(res->mem_type); + return mem_type_is_vram(res->lru_entity.mem_type); } bool xe_bo_is_vram(struct xe_bo *bo) @@ -77,7 +77,7 @@ bool xe_bo_is_vram(struct xe_bo *bo) bool xe_bo_is_stolen(struct xe_bo *bo) { - return bo->ttm.resource->mem_type == XE_PL_STOLEN; + return bo->ttm.resource->lru_entity.mem_type == XE_PL_STOLEN; } /** @@ -118,7 +118,7 @@ mem_type_to_tile(struct xe_device *xe, u32 mem_type) */ struct xe_tile *xe_bo_to_tile(struct xe_bo *bo) { - return mem_type_to_tile(xe_bo_device(bo), bo->ttm.resource->mem_type); + return mem_type_to_tile(xe_bo_device(bo), bo->ttm.resource->lru_entity.mem_type); } static void try_add_system(struct xe_bo *bo, struct ttm_place *places, @@ -259,7 +259,7 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo, */ bo = ttm_to_xe_bo(tbo); - switch (tbo->resource->mem_type) { + switch (tbo->resource->lru_entity.mem_type) { case XE_PL_VRAM0: case XE_PL_VRAM1: case XE_PL_STOLEN: @@ -410,17 +410,17 @@ static int xe_ttm_io_mem_reserve(struct ttm_device *bdev, { struct xe_device *xe = ttm_to_xe_device(bdev); - switch (mem->mem_type) { + switch (mem->lru_entity.mem_type) { case XE_PL_SYSTEM: case XE_PL_TT: return 0; case XE_PL_VRAM0: case XE_PL_VRAM1: { - struct xe_tile *tile = mem_type_to_tile(xe, mem->mem_type); + struct xe_tile *tile = mem_type_to_tile(xe, mem->lru_entity.mem_type); struct xe_ttm_vram_mgr_resource *vres = to_xe_ttm_vram_mgr_resource(mem); - if (vres->used_visible_size < mem->size) + if (vres->used_visible_size < mem->lru_entity.size) return -EINVAL; mem->bus.offset = mem->start << PAGE_SHIFT; @@ -546,7 +546,7 @@ static int xe_bo_move_dmabuf(struct ttm_buffer_object *ttm_bo, XE_WARN_ON(!attach); XE_WARN_ON(!ttm_bo->ttm); - if (new_res->mem_type == XE_PL_SYSTEM) + if (new_res->lru_entity.mem_type == XE_PL_SYSTEM) goto out; if (ttm_bo->sg) { @@ -620,7 +620,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct xe_device *xe = ttm_to_xe_device(ttm_bo->bdev); struct xe_bo *bo = ttm_to_xe_bo(ttm_bo); struct ttm_resource *old_mem = ttm_bo->resource; - u32 old_mem_type = old_mem ? old_mem->mem_type : XE_PL_SYSTEM; + u32 old_mem_type = old_mem ? old_mem->lru_entity.mem_type : XE_PL_SYSTEM; struct ttm_tt *ttm = ttm_bo->ttm; struct xe_tile *tile = NULL; struct dma_fence *fence; @@ -652,7 +652,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, if ((move_lacks_source && !needs_clear) || (old_mem_type == XE_PL_SYSTEM && - new_mem->mem_type == XE_PL_TT)) { + new_mem->lru_entity.mem_type == XE_PL_TT)) { ttm_bo_move_null(ttm_bo, new_mem); goto out; } @@ -662,7 +662,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, * TTM_PL_FLAG_TEMPORARY, should just be a dummy move. */ if (old_mem_type == XE_PL_TT && - new_mem->mem_type == XE_PL_TT) { + new_mem->lru_entity.mem_type == XE_PL_TT) { ttm_bo_move_null(ttm_bo, new_mem); goto out; } @@ -674,7 +674,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, } if (old_mem_type == XE_PL_TT && - new_mem->mem_type == XE_PL_SYSTEM) { + new_mem->lru_entity.mem_type == XE_PL_SYSTEM) { long timeout = dma_resv_wait_timeout(ttm_bo->base.resv, DMA_RESV_USAGE_BOOKKEEP, true, @@ -690,7 +690,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, if (!move_lacks_source && ((old_mem_type == XE_PL_SYSTEM && resource_is_vram(new_mem)) || (mem_type_is_vram(old_mem_type) && - new_mem->mem_type == XE_PL_SYSTEM))) { + new_mem->lru_entity.mem_type == XE_PL_SYSTEM))) { hop->fpfn = 0; hop->lpfn = 0; hop->mem_type = XE_PL_TT; @@ -702,7 +702,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, if (bo->tile) tile = bo->tile; else if (resource_is_vram(new_mem)) - tile = mem_type_to_tile(xe, new_mem->mem_type); + tile = mem_type_to_tile(xe, new_mem->lru_entity.mem_type); else if (mem_type_is_vram(old_mem_type)) tile = mem_type_to_tile(xe, old_mem_type); @@ -777,7 +777,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, } xe_device_mem_access_put(xe); - trace_printk("new_mem->mem_type=%d\n", new_mem->mem_type); + trace_printk("new_mem->lru_entity.mem_type=%d\n", new_mem->lru_entity.mem_type); out: return ret; @@ -918,10 +918,10 @@ static unsigned long xe_ttm_io_mem_pfn(struct ttm_buffer_object *ttm_bo, { struct xe_device *xe = ttm_to_xe_device(ttm_bo->bdev); struct xe_bo *bo = ttm_to_xe_bo(ttm_bo); - struct xe_tile *tile = mem_type_to_tile(xe, ttm_bo->resource->mem_type); + struct xe_tile *tile = mem_type_to_tile(xe, ttm_bo->resource->lru_entity.mem_type); struct xe_res_cursor cursor; - if (ttm_bo->resource->mem_type == XE_PL_STOLEN) + if (ttm_bo->resource->lru_entity.mem_type == XE_PL_STOLEN) return xe_ttm_stolen_io_offset(bo, page_offset << PAGE_SHIFT) >> PAGE_SHIFT; xe_res_first(ttm_bo->resource, (u64)page_offset << PAGE_SHIFT, 0, &cursor); @@ -1183,7 +1183,7 @@ void xe_bo_free(struct xe_bo *bo) struct xe_bo *__xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo, struct xe_tile *tile, struct dma_resv *resv, - struct ttm_lru_bulk_move *bulk, size_t size, + struct drm_lru_bulk_move *bulk, size_t size, enum ttm_bo_type type, u32 flags) { struct ttm_operation_ctx ctx = { @@ -1452,9 +1452,9 @@ struct xe_bo *xe_bo_create_from_data(struct xe_device *xe, struct xe_tile *tile, uint64_t vram_region_gpu_offset(struct ttm_resource *res) { struct xe_device *xe = ttm_to_xe_device(res->bo->bdev); - struct xe_tile *tile = mem_type_to_tile(xe, res->mem_type); + struct xe_tile *tile = mem_type_to_tile(xe, res->lru_entity.mem_type); - if (res->mem_type == XE_PL_STOLEN) + if (res->lru_entity.mem_type == XE_PL_STOLEN) return xe_ttm_stolen_gpu_offset(xe); return tile->mem.vram.dpa_base; @@ -1960,7 +1960,7 @@ int xe_bo_migrate(struct xe_bo *bo, u32 mem_type) xe_bo_assert_held(bo); - if (bo->ttm.resource->mem_type == mem_type) + if (bo->ttm.resource->lru_entity.mem_type == mem_type) return 0; if (xe_bo_is_pinned(bo)) diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h index 9918b2d630e1..3ab17c81fe6e 100644 --- a/drivers/gpu/drm/xe/xe_bo.h +++ b/drivers/gpu/drm/xe/xe_bo.h @@ -9,6 +9,7 @@ #include "xe_bo_types.h" #include "xe_macros.h" #include "xe_vm_types.h" +#include #define XE_DEFAULT_GTT_SIZE_MB 3072ULL /* 3GB by default */ @@ -83,7 +84,7 @@ void xe_bo_free(struct xe_bo *bo); struct xe_bo *__xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo, struct xe_tile *tile, struct dma_resv *resv, - struct ttm_lru_bulk_move *bulk, size_t size, + struct drm_lru_bulk_move *bulk, size_t size, enum ttm_bo_type type, u32 flags); struct xe_bo * xe_bo_create_locked_range(struct xe_device *xe, diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c index 09343b8b3e96..f22a4e0388d4 100644 --- a/drivers/gpu/drm/xe/xe_dma_buf.c +++ b/drivers/gpu/drm/xe/xe_dma_buf.c @@ -82,7 +82,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach, if (!xe_bo_is_pinned(bo)) { if (!attach->peer2peer || - bo->ttm.resource->mem_type == XE_PL_SYSTEM) { + bo->ttm.resource->lru_entity.mem_type == XE_PL_SYSTEM) { if (xe_bo_can_migrate(bo, XE_PL_TT)) r = xe_bo_migrate(bo, XE_PL_TT); else @@ -92,7 +92,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach, return ERR_PTR(r); } - switch (bo->ttm.resource->mem_type) { + switch (bo->ttm.resource->lru_entity.mem_type) { case XE_PL_TT: sgt = drm_prime_pages_to_sg(obj->dev, bo->ttm.ttm->pages, diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c index dafebdfb2368..100385e71a94 100644 --- a/drivers/gpu/drm/xe/xe_exec.c +++ b/drivers/gpu/drm/xe/xe_exec.c @@ -371,7 +371,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file) if (!err && !xe_vm_no_dma_fences(vm)) { spin_lock(xe->ttm.lru_lock); - ttm_lru_bulk_move_tail(&vm->lru_bulk_move); + drm_lru_bulk_move_tail(&vm->lru_bulk_move); spin_unlock(xe->ttm.lru_lock); } diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c index ee8bc5f3ba3d..fdf2a448457d 100644 --- a/drivers/gpu/drm/xe/xe_migrate.c +++ b/drivers/gpu/drm/xe/xe_migrate.c @@ -643,8 +643,8 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, u64 src_L0, dst_L0; int pass = 0; int err; - bool src_is_vram = mem_type_is_vram(src->mem_type); - bool dst_is_vram = mem_type_is_vram(dst->mem_type); + bool src_is_vram = mem_type_is_vram(src->lru_entity.mem_type); + bool dst_is_vram = mem_type_is_vram(dst->lru_entity.mem_type); bool copy_ccs = xe_device_has_flat_ccs(xe) && xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo); bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram); @@ -895,7 +895,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m, struct xe_bo *bo, struct ttm_resource *dst) { - bool clear_vram = mem_type_is_vram(dst->mem_type); + bool clear_vram = mem_type_is_vram(dst->lru_entity.mem_type); struct xe_gt *gt = m->tile->primary_gt; struct xe_device *xe = gt_to_xe(gt); struct dma_fence *fence = NULL; diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h index 5cb4b66a5d74..64c5549f4d44 100644 --- a/drivers/gpu/drm/xe/xe_res_cursor.h +++ b/drivers/gpu/drm/xe/xe_res_cursor.h @@ -53,8 +53,8 @@ static struct drm_buddy *xe_res_get_buddy(struct ttm_resource *res) struct xe_device *xe = ttm_to_xe_device(res->bo->bdev); struct ttm_resource_manager *mgr; - if (res->mem_type != XE_PL_STOLEN) - return &xe->tiles[res->mem_type - XE_PL_VRAM0].mem.vram_mgr->mm; + if (res->lru_entity.mem_type != XE_PL_STOLEN) + return &xe->tiles[res->lru_entity.mem_type - XE_PL_VRAM0].mem.vram_mgr->mm; mgr = ttm_manager_type(&xe->ttm, XE_PL_STOLEN); @@ -79,9 +79,9 @@ static inline void xe_res_first(struct ttm_resource *res, if (!res) goto fallback; - XE_WARN_ON(start + size > res->size); + XE_WARN_ON(start + size > res->lru_entity.size); - cur->mem_type = res->mem_type; + cur->mem_type = res->lru_entity.mem_type; switch (cur->mem_type) { case XE_PL_STOLEN: @@ -128,7 +128,7 @@ static inline void xe_res_first(struct ttm_resource *res, cur->remaining = size; cur->node = NULL; cur->mem_type = XE_PL_TT; - XE_WARN_ON(res && start + size > res->size); + XE_WARN_ON(res && start + size > res->lru_entity.size); } static inline void __xe_res_sg_next(struct xe_res_cursor *cur) diff --git a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c index ace42852a419..7aa179e73d50 100644 --- a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +++ b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c @@ -48,7 +48,7 @@ static int xe_ttm_sys_mgr_new(struct ttm_resource_manager *man, } node->base.mm_nodes[0].start = 0; - node->base.mm_nodes[0].size = PFN_UP(node->base.base.size); + node->base.mm_nodes[0].size = PFN_UP(node->base.base.lru_entity.size); node->base.base.start = XE_BO_INVALID_OFFSET; *res = &node->base.base; diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c index a3c1bf555c06..06a469cc8490 100644 --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c @@ -83,11 +83,11 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man, if (place->fpfn || lpfn != man->size >> PAGE_SHIFT) vres->flags |= DRM_BUDDY_RANGE_ALLOCATION; - if (WARN_ON(!vres->base.size)) { + if (WARN_ON(!vres->base.lru_entity.size)) { err = -EINVAL; goto error_fini; } - size = vres->base.size; + size = vres->base.lru_entity.size; min_page_size = mgr->default_page_size; if (tbo->page_alignment) @@ -150,8 +150,8 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man, } while (remaining_size); if (place->flags & TTM_PL_FLAG_CONTIGUOUS) { - if (!drm_buddy_block_trim(mm, vres->base.size, &vres->blocks)) - size = vres->base.size; + if (!drm_buddy_block_trim(mm, vres->base.lru_entity.size, &vres->blocks)) + size = vres->base.lru_entity.size; } if (lpfn <= mgr->visible_size >> PAGE_SHIFT) { @@ -378,14 +378,14 @@ int xe_ttm_vram_mgr_alloc_sgt(struct xe_device *xe, enum dma_data_direction dir, struct sg_table **sgt) { - struct xe_tile *tile = &xe->tiles[res->mem_type - XE_PL_VRAM0]; + struct xe_tile *tile = &xe->tiles[res->lru_entity.mem_type - XE_PL_VRAM0]; struct xe_ttm_vram_mgr_resource *vres = to_xe_ttm_vram_mgr_resource(res); struct xe_res_cursor cursor; struct scatterlist *sg; int num_entries = 0; int i, r; - if (vres->used_visible_size < res->size) + if (vres->used_visible_size < res->lru_entity.size) return -EOPNOTSUPP; *sgt = kmalloc(sizeof(**sgt), GFP_KERNEL); diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 44e038276d41..f96009b02800 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -652,7 +652,7 @@ static void preempt_rebind_work_func(struct work_struct *w) #undef retry_required spin_lock(vm->xe->ttm.lru_lock); - ttm_lru_bulk_move_tail(&vm->lru_bulk_move); + drm_lru_bulk_move_tail(&vm->lru_bulk_move); spin_unlock(vm->xe->ttm.lru_lock); /* Point of no return. */ diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index 52e5eaed91c3..784b07660fff 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -149,7 +149,7 @@ struct xe_vm { struct dma_resv resv; /** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ - struct ttm_lru_bulk_move lru_bulk_move; + struct drm_lru_bulk_move lru_bulk_move; u64 size; diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h index 223b198fe371..f4d939ee174c 100644 --- a/include/drm/ttm/ttm_bo.h +++ b/include/drm/ttm/ttm_bo.h @@ -118,7 +118,7 @@ struct ttm_buffer_object { struct ttm_resource *resource; struct ttm_tt *ttm; bool deleted; - struct ttm_lru_bulk_move *bulk_move; + struct drm_lru_bulk_move *bulk_move; unsigned priority; unsigned pin_count; @@ -355,7 +355,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx); void ttm_bo_put(struct ttm_buffer_object *bo); void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo, - struct ttm_lru_bulk_move *bulk); + struct drm_lru_bulk_move *bulk); bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo, const struct ttm_place *place); int ttm_bo_init_reserved(struct ttm_device *bdev, struct ttm_buffer_object *bo, diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index c2528cec12e6..2401a7510ef6 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -47,6 +47,7 @@ struct io_mapping; struct sg_table; struct scatterlist; struct drm_lru_mgr; +struct drm_lru_entity; struct ttm_resource_manager_func { /** @@ -143,7 +144,7 @@ struct ttm_resource_manager_func { * @func: structure pointer implementing the range manager. See above * @move_lock: lock for move fence * @move: The fence of the last pipelined move operation. - * @lru: The lru list for this memory type. + * @lru_mgr: The lru manager for this ttm_resource_manager * * This structure is used to identify and manage memory types for a device. */ @@ -163,14 +164,9 @@ struct ttm_resource_manager { */ struct dma_fence *move; - /* - * Protected by the bdev->lru_lock. - */ - struct list_head lru[DRM_MAX_LRU_PRIORITY]; - /** * @usage: How much of the resources are used, protected by the - * bdev->lru_lock. + * drm_device::lru_lock. */ uint64_t usage; @@ -202,8 +198,6 @@ struct ttm_bus_placement { * struct ttm_resource * * @start: Start of the allocation. - * @size: Actual size of resource in bytes. - * @mem_type: Resource type of the allocation. * @placement: Placement flags. * @bus: Placement on io bus accessible to the CPU * @bo: weak reference to the BO, protected by ttm_device::lru_lock @@ -214,44 +208,12 @@ struct ttm_bus_placement { */ struct ttm_resource { unsigned long start; - size_t size; - uint32_t mem_type; uint32_t placement; struct ttm_bus_placement bus; struct ttm_buffer_object *bo; - - /** - * @lru: Least recently used list, see &ttm_resource_manager.lru - */ - struct list_head lru; struct drm_lru_entity lru_entity; }; -/** - * struct ttm_lru_bulk_move_pos - * - * @first: first res in the bulk move range - * @last: last res in the bulk move range - * - * Range of resources for a lru bulk move. - */ -struct ttm_lru_bulk_move_pos { - struct ttm_resource *first; - struct ttm_resource *last; -}; - -/** - * struct ttm_lru_bulk_move - * - * @pos: first/last lru entry for resources in the each domain/priority - * - * Container for the current bulk move state. Should be used with - * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move(). - */ -struct ttm_lru_bulk_move { - struct ttm_lru_bulk_move_pos pos[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY]; -}; - /** * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping + * struct sg_table backed struct ttm_resource. @@ -306,7 +268,7 @@ ttm_resource_manager_set_used(struct ttm_resource_manager *man, bool used) int i; for (i = 0; i < DRM_MAX_LRU_PRIORITY; i++) - WARN_ON(!list_empty(&man->lru[i])); + WARN_ON(!list_empty(&man->lru_mgr->lru[i])); man->use_type = used; } @@ -350,13 +312,6 @@ ttm_resource_manager_cleanup(struct ttm_resource_manager *man) man->move = NULL; } -void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk); -void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk); - -void ttm_resource_add_bulk_move(struct ttm_resource *res, - struct ttm_buffer_object *bo); -void ttm_resource_del_bulk_move(struct ttm_resource *res, - struct ttm_buffer_object *bo); void ttm_resource_move_to_lru_tail(struct ttm_resource *res); void ttm_resource_init(struct ttm_buffer_object *bo, @@ -382,6 +337,12 @@ bool ttm_resource_compat(struct ttm_resource *res, void ttm_resource_set_bo(struct ttm_resource *res, struct ttm_buffer_object *bo); +void ttm_resource_add_bulk_move(struct ttm_resource *res, + struct ttm_buffer_object *bo); + +void ttm_resource_del_bulk_move(struct ttm_resource *res, + struct ttm_buffer_object *bo); + void ttm_resource_manager_init(struct ttm_resource_manager *man, struct ttm_device *bdev, uint64_t size);