From patchwork Wed Feb 15 16:13:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Thomas Hellstrom X-Patchwork-Id: 13141850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B47B7C636CC for ; Wed, 15 Feb 2023 16:15:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F2576B0083; Wed, 15 Feb 2023 11:15:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47A9E6B0085; Wed, 15 Feb 2023 11:15:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F4CB6B0087; Wed, 15 Feb 2023 11:15:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1FDEF6B0083 for ; Wed, 15 Feb 2023 11:15:15 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DF9B01C68E0 for ; Wed, 15 Feb 2023 16:15:14 +0000 (UTC) X-FDA: 80470025748.25.47B5A22 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf09.hostedemail.com (Postfix) with ESMTP id 42B29140021 for ; Wed, 15 Feb 2023 16:15:11 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QwX0o0xa; spf=none (imf09.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676477712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KxTfDj7jbAB9F6B1kdSBhtSWJt6HRd7nbb2H7M59kd8=; b=ZCvEG3E3ZfVjhWBbpG+KEkmmFJP+4MkfvTjrQryT4JhY+AZ5nfDKxkRasNDUzrBoH/ssgG iav8AmKQ/jhxz841867dfHJeGvBCQTrYl9V7iNEXM4WFmbj7Nckx6/y3AoC/R9R7VLw2Kf HSO265C25I5HgRkgtP3D9sW53wWODUY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QwX0o0xa; spf=none (imf09.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676477712; a=rsa-sha256; cv=none; b=mo7KaqxuXTzRP9mzIrNS12/KbEUh0kOHGQ7h374qgd2nG/YtITbZFZWu4BHsbAdLLDp9LC RSMhkqMEKc0UfzBkrwaVBbkaVlx46Mo85QWdaoGEgkCtvO0rE84/h9D0TLgsKf6kwHiNnY co0xqMGUrCEhA1zosAifP9mRMTlRwZk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676477712; x=1708013712; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=n4ovsEqlT6l3tqJByLmbZrFyXkrQBTKpHU8ibleASdU=; b=QwX0o0xa5h930OFKF6BmWF0uNc+EHDRASzlTBRvQLvmimmCr5CcdRJ5x RgbnN27FBD5IienO8nVDVEv9TGqPHMnglT9GXRUDX3m3JMccgeAHQDk36 +R3YDa+eYvaXEmKcKcTJ5vJtMMz/zK6CH+Lelcb5yjutT0o5hg+HtWKYs pl3w79GV7KE9K5/H4QIf32eWez4hlQne0elczF7AzdXnss9JfRPuOIiYN rXrz5YJEUwTUttuHO/t4w342kOprzIChYIQnyc134WdiP4rLoixaBcgrv r/AVUtC7Apnp7AiurA1pGIcL1iEL6B1cRG8zCvBpl5c1k/1mxlK4whie6 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10622"; a="393870931" X-IronPort-AV: E=Sophos;i="5.97,300,1669104000"; d="scan'208";a="393870931" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 08:15:10 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10622"; a="758472471" X-IronPort-AV: E=Sophos;i="5.97,300,1669104000"; d="scan'208";a="758472471" Received: from auliel-mobl1.ger.corp.intel.com (HELO thellstr-mobl1.intel.com) ([10.249.254.14]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 08:15:06 -0800 From: =?utf-8?q?Thomas_Hellstr=C3=B6m?= To: dri-devel@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Andrew Morton , "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Daniel Vetter , Christian Koenig , Dave Airlie , Dave Hansen , Matthew Auld , linux-graphics-maintainer@vmware.com, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org Subject: [RFC PATCH 08/16] drm/ttm: Add a shrinker and shrinker accounting Date: Wed, 15 Feb 2023 17:13:57 +0100 Message-Id: <20230215161405.187368-9-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230215161405.187368-1-thomas.hellstrom@linux.intel.com> References: <20230215161405.187368-1-thomas.hellstrom@linux.intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 42B29140021 X-Stat-Signature: trqmzzn9ir5ehugfz31zoqaco4zwsuw3 X-HE-Tag: 1676477711-605815 X-HE-Meta: U2FsdGVkX19Cso9oT2P7iAxVCVa9RjZ8n4hnrrBwHproK/x4DxejMdrdqUh5BzXSRvb0f/yx1efpTbMBrtl9AJQreHvexVeEJJwxM3vkZD4AI41wJ20FAwpRNalvBvQYsfn+ntEeXYLmU1V/Pu1KBRHprZAJQ/m/bh1tdqhJNIOCpAtRrUSL93ErL5mSflphjQEFQV6tfkWALqoRHbalG3ASbs7M6D7fLsW1rSkdTAe42HNFPFKu1pt4nL2wfCztnaaM/E042UUFmF90SqhsVWCxtkIa3a9C5FdDOwDQ80drAHcV8SjnAWjYQ4PQ3/5rwV0lvxoa7sQj8kYMIIICH5GodU15o9rDqsA/1SWwPpoF7n4xigNLnKwzEUfokMd8m0rZtsesBSH9MDQ92RaD2mrS2svCMpJTVI6GAv/EqAClIp2ty6yGs7DX7qJz8VbIpnHUwRQ9ox2s9zJIEArSUAu6V7MZyaL6l9/u/vQd3zb7c1Np4UM65mHZ/RnOuIggFL/UOTTdunbrf/Iu3TQjVJsMybiVt2EK+hIsNk1MNI0G05ZeHOE3a0Q4wKa+AI9+uRyExy6t9nm1XKXmqkFG+NGZEzFexp+ClLcUE5Zq/y2x51gg4urX9ZfsPiNK+jly9X81udLCbb1l/rEIzSZcjIOQ+4+S9etFAYPhtxmSYDP4b2j5cTd+cEz3JwTXte2Acjb28HADx/iQw4ViuH2n+dwCPu3ZZZJyfLZvI1jVwKqqgfOWs+7iICtL6dzlxNH4SEMPvmTH3qI5564I8se07Ym5tI/Nxmpn/UR6ZNyzhrZlao6nv8OXzlv/YOFEqEUx3XzkNloLflbUj+iEFmsZaUBx8JvHn2zggaJDVJSxxVmaLT+kKKdKHgu8S/tiKQpKdskEr4Eshx/3QpgoIYOyqigSEGvISwqOl9Js8eGdJ7A2r/G3RYaxWHDUzyJ3WzwG5VuvO/Db9gXCZMN2PLk xHF32dOC jdI6F/vndYBlZtoC5PRjjey7oZMWkjRaQqQr36Xg2o/VoieFRNz2XpZfXZi9U0CegOdS10VxLYSxaz0yxojDfA0X8zpLgAfOxx+qDQ1vglG78KPJEadKrxJvzQAxRrwOrMum7GNN+0eIKYw+FTC6U7ETAW0z5EKdjGuFeQtNi8mM6YCIdjH3TbhYYC1TXsNA7hLTC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Register a TTM system memory-backed object shrinker and add accounting for shrinkable and purgeable pages. For the shrinker to work, the driver needs to register the bo_shrink callback which is responsible for unbinding from GPU and the dma layer if needed. Helpers for that callback to actually perform shrinking will be introduced in upcoming patches. Note that we can't lock the ttm_global_mutex from within the shrinker scan() function as that might cause a deadlock issue. To fix that, add and use a mutex which is used for global device list manipulation only and make sure it isn't held when registering the shrinker. Signed-off-by: Thomas Hellström --- drivers/gpu/drm/ttm/ttm_device.c | 26 ++++--- drivers/gpu/drm/ttm/ttm_tt.c | 112 +++++++++++++++++++++++++++++-- include/drm/ttm/ttm_tt.h | 2 + 3 files changed, 125 insertions(+), 15 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index e0a2be3ed13d..ce98752d2d32 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -36,10 +36,10 @@ #include "ttm_module.h" -/* - * ttm_global_mutex - protecting the global state - */ +/* ttm_global_mutex - protects the global state init and fini. */ static DEFINE_MUTEX(ttm_global_mutex); +/* ttm_global_list_mutex - protects the device list. */ +static DEFINE_MUTEX(ttm_global_list_mutex); static unsigned ttm_glob_use_count; struct ttm_global ttm_glob; EXPORT_SYMBOL(ttm_glob); @@ -54,6 +54,7 @@ static void ttm_global_release(void) if (--ttm_glob_use_count > 0) goto out; + ttm_tt_mgr_fini(); ttm_pool_mgr_fini(); debugfs_remove(ttm_debugfs_root); @@ -102,7 +103,10 @@ static int ttm_global_init(void) goto out; } + mutex_lock(&ttm_global_list_mutex); INIT_LIST_HEAD(&glob->device_list); + mutex_unlock(&ttm_global_list_mutex); + atomic_set(&glob->bo_count, 0); debugfs_create_atomic_t("buffer_objects", 0444, ttm_debugfs_root, @@ -135,7 +139,7 @@ long ttm_global_swapout(struct ttm_operation_ctx *ctx, struct ttm_device *bdev; long ret = 0; - mutex_lock(&ttm_global_mutex); + mutex_lock(&ttm_global_list_mutex); list_for_each_entry(bdev, &glob->device_list, device_list) { ret = ttm_device_swapout(bdev, ctx, reason); if (ret > 0) { @@ -143,7 +147,7 @@ long ttm_global_swapout(struct ttm_operation_ctx *ctx, break; } } - mutex_unlock(&ttm_global_mutex); + mutex_unlock(&ttm_global_list_mutex); return ret; } @@ -247,9 +251,9 @@ int ttm_device_init(struct ttm_device *bdev, struct ttm_device_funcs *funcs, spin_lock_init(&bdev->lru_lock); INIT_LIST_HEAD(&bdev->pinned); bdev->dev_mapping = mapping; - mutex_lock(&ttm_global_mutex); + mutex_lock(&ttm_global_list_mutex); list_add_tail(&bdev->device_list, &glob->device_list); - mutex_unlock(&ttm_global_mutex); + mutex_unlock(&ttm_global_list_mutex); return 0; } @@ -260,14 +264,14 @@ void ttm_device_fini(struct ttm_device *bdev) struct ttm_resource_manager *man; unsigned i; + mutex_lock(&ttm_global_list_mutex); + list_del(&bdev->device_list); + mutex_unlock(&ttm_global_list_mutex); + man = ttm_manager_type(bdev, TTM_PL_SYSTEM); ttm_resource_manager_set_used(man, false); ttm_set_driver_manager(bdev, TTM_PL_SYSTEM, NULL); - mutex_lock(&ttm_global_mutex); - list_del(&bdev->device_list); - mutex_unlock(&ttm_global_mutex); - drain_workqueue(bdev->wq); destroy_workqueue(bdev->wq); diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c index 771e5f3c2fee..5a57117c21ec 100644 --- a/drivers/gpu/drm/ttm/ttm_tt.c +++ b/drivers/gpu/drm/ttm/ttm_tt.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include "ttm_module.h" @@ -54,6 +55,11 @@ module_param_named(dma32_pages_limit, ttm_dma32_pages_limit, ulong, 0644); static atomic_long_t ttm_pages_allocated; static atomic_long_t ttm_dma32_pages_allocated; +static long shrinkable_pages; +static long purgeable_pages; +static DEFINE_RWLOCK(shrinkable_lock); +static struct shrinker mm_shrinker; + static bool ttm_tt_shrinkable(const struct ttm_device *bdev, const struct ttm_tt *tt) { @@ -69,6 +75,14 @@ static void ttm_tt_mod_allocated(bool dma32, long value) atomic_long_add(value, &ttm_dma32_pages_allocated); } +static void ttm_tt_mod_shrinkable_pages(long shrinkable, long purgeable) +{ + write_lock(&shrinkable_lock); + shrinkable_pages += shrinkable; + purgeable_pages += purgeable; + write_unlock(&shrinkable_lock); +} + /* * Allocates a ttm structure for the given BO. */ @@ -352,6 +366,9 @@ int ttm_tt_populate(struct ttm_device *bdev, } } + if (ttm_tt_shrinkable(bdev, ttm)) + ttm_tt_mod_shrinkable_pages(ttm->num_pages, 0); + return 0; error: @@ -368,6 +385,13 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) if (!ttm_tt_is_populated(ttm)) return; + if (ttm_tt_shrinkable(bdev, ttm)) { + if (ttm_tt_purgeable(ttm)) + ttm_tt_mod_shrinkable_pages(0, -(long)ttm->num_pages); + else + ttm_tt_mod_shrinkable_pages(-(long)ttm->num_pages, 0); + } + if (bdev->funcs->ttm_tt_unpopulate) bdev->funcs->ttm_tt_unpopulate(bdev, ttm); else @@ -394,11 +418,86 @@ DEFINE_SHOW_ATTRIBUTE(ttm_tt_debugfs_shrink); #endif +static unsigned long ttm_tt_shrinker_count(struct shrinker *shrink, + struct shrink_control *sc) +{ + unsigned long num_pages; -/* - * ttm_tt_mgr_init - register with the MM shrinker - * - * Register with the MM shrinker for swapping out BOs. + num_pages = get_nr_swap_pages(); + read_lock(&shrinkable_lock); + num_pages = min_t(unsigned long, num_pages, shrinkable_pages); + num_pages += purgeable_pages; + read_unlock(&shrinkable_lock); + + return num_pages ? num_pages : SHRINK_EMPTY; +} + +static unsigned long ttm_tt_shrinker_scan(struct shrinker *shrink, + struct shrink_control *sc) +{ + bool is_kswapd = current_is_kswapd(); + struct ttm_operation_ctx ctx = { + .interruptible = false, + .no_wait_gpu = !is_kswapd, + }; + unsigned long nr_to_scan, freed = 0; + long ret; + + sc->nr_scanned = 0; + nr_to_scan = sc->nr_to_scan; + + while (freed < nr_to_scan) { + ret = ttm_global_swapout(&ctx, TTM_SHRINK_PURGE); + if (ret <= 0) + break; + + freed += ret; + } + + sc->nr_scanned = freed; + if (freed < nr_to_scan) + nr_to_scan -= freed; + else + nr_to_scan = 0; + if (!nr_to_scan) + return freed ? freed : SHRINK_STOP; + + while (freed < nr_to_scan) { + ret = ttm_global_swapout(&ctx, TTM_SHRINK_SWAP); + if (ret <= 0) + break; + + freed += ret; + } + + sc->nr_scanned = freed; + + return freed ? freed : SHRINK_STOP; +} + +/** + * ttm_tt_mgr_fini() - Check shrinkable accounting consistensy and remove + * the shrinker. + */ +void ttm_tt_mgr_fini(void) +{ + if (WARN_ON_ONCE(atomic_long_read(&ttm_pages_allocated) || + atomic_long_read(&ttm_dma32_pages_allocated) || + shrinkable_pages || purgeable_pages)) { + pr_warn("Inconsistent ttm_tt accounting:\n"); + pr_warn("pages %ld dma32 %ld shrinkable %ld purgeable %ld\n", + atomic_long_read(&ttm_pages_allocated), + atomic_long_read(&ttm_dma32_pages_allocated), + shrinkable_pages, purgeable_pages); + } + + unregister_shrinker(&mm_shrinker); +} + +/** + * ttm_tt_mgr_init() - Provide watermark limits and register the shrinker. + * @num_pages - Number of pages TTM is allowed to pin. + * @num_dma32_pages - Number of dma32 pages TTM is allowed to pin. */ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages) { @@ -412,6 +511,11 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages) if (!ttm_dma32_pages_limit) ttm_dma32_pages_limit = num_dma32_pages; + + mm_shrinker.count_objects = ttm_tt_shrinker_count; + mm_shrinker.scan_objects = ttm_tt_shrinker_scan; + mm_shrinker.seeks = DEFAULT_SEEKS; + (void)register_shrinker(&mm_shrinker, "ttm-objects"); } static void ttm_kmap_iter_tt_map_local(struct ttm_kmap_iter *iter, diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h index 627168eba8f6..3f99787e2b93 100644 --- a/include/drm/ttm/ttm_tt.h +++ b/include/drm/ttm/ttm_tt.h @@ -221,6 +221,8 @@ static inline void ttm_tt_mark_for_clear(struct ttm_tt *ttm) void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages); +void ttm_tt_mgr_fini(void); + struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt, struct ttm_tt *tt);