From patchwork Thu Jun 27 15:47:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maarten Lankhorst X-Patchwork-Id: 13714716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41E09C30653 for ; Thu, 27 Jun 2024 15:48:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53D406B0095; Thu, 27 Jun 2024 11:48:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C5CA6B0096; Thu, 27 Jun 2024 11:48:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C9C56B0099; Thu, 27 Jun 2024 11:48:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 011F06B0095 for ; Thu, 27 Jun 2024 11:48:09 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 839FC1A0AE6 for ; Thu, 27 Jun 2024 15:48:09 +0000 (UTC) X-FDA: 82277099898.18.0FA6609 Received: from mblankhorst.nl (lankhorst.se [141.105.120.124]) by imf16.hostedemail.com (Postfix) with ESMTP id CBCC7180021 for ; Thu, 27 Jun 2024 15:48:07 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=none (imf16.hostedemail.com: domain of mlankhorst@mblankhorst.nl has no SPF policy when checking 141.105.120.124) smtp.mailfrom=mlankhorst@mblankhorst.nl; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=intel.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719503270; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MYCeKgjWukJyBeJ8r5qHndxiIkxkeOpXrRB+1H8rRJw=; b=c1knIy9wvzlYxZKxRCW7PkfoMxFHP1N25bMaWTEZ3tD2r0FXkF+256D9AdaH++29O4C8u1 XU1JCOom3cLLdgpL9dqV/+yjwP8MHYpkGHguaOKmnsmrv+naA1LUJXsRPJ4K8g5UoLtBcI 5Aqy+Q8kOCHNnhXKNgm+U31e80buVHM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719503270; a=rsa-sha256; cv=none; b=TNtUH2r1tEZEH1FFv3wKmGqVHnhsneINV1NV7yHJ4e0mc0B2UMOC5hqyJcHGwUNOlE4Whw Lca9/ScnOvFUS/THDAYHOLID5WIjz00WUwgqlS1SrdelqgQ2zJo5l26WKG/VDaxFb6iNRJ XH3ZjGvdt7qcgNaSBiNKPtBEbLgG638= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=none (imf16.hostedemail.com: domain of mlankhorst@mblankhorst.nl has no SPF policy when checking 141.105.120.124) smtp.mailfrom=mlankhorst@mblankhorst.nl; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=intel.com (policy=none) From: Maarten Lankhorst To: intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Tejun Heo , Zefan Li , Johannes Weiner , Andrew Morton , Christian Koenig , Huang Rui , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter Cc: Friedrich Vock , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 3/6] drm/ttm: Handle cgroup based eviction in TTM Date: Thu, 27 Jun 2024 17:47:22 +0200 Message-ID: <20240627154754.74828-4-maarten.lankhorst@linux.intel.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240627154754.74828-1-maarten.lankhorst@linux.intel.com> References: <20240627154754.74828-1-maarten.lankhorst@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: bkbjzxbcwhc1nq67py8nao9hp7ueotdr X-Rspamd-Queue-Id: CBCC7180021 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1719503287-464576 X-HE-Meta: U2FsdGVkX19bll0jQfSpUcbL0sfd6V+rV9qOo/5TxQLUUdMRKnD9BSFdLwTZEV5UkJTvAPNoWcodcFeLKPuixEA1qD3hakw+4U5LXNTJyzSOAzdR8S4nfmxecO+/Mny357ZlIGrJUAhuOrzU/VdGcDMY/6fhks1hra8aGf4Im1qg93tSzIl9SmC5CZxB84Xic9WEuepK3MaH4KzHtKon5lrxE5Aoh2L+rIOMnxZNHtVxgqdkRQXdI9CnmZv2nt0vMZQn0amQ3BJIPnzuSCKSMjXLsytb+biqISkiZhwtzOQZnAvPFg7Tlbko0/8gbgspOrB18yWzxe028MyciGcwwC90/dH2X6a3s0NBkGTPwii6NUbULaKkKzW4Y54X2W0fG4bS7bhUZB1eYknV+wwQe3WUFiuQ0EcT2YDathXbcPY+Lih8hItHnOU7OpeFoLR/0nSRMd1VbW/emfQjYN07JY4XrAGNmsCuYNXZzPfCcT8AAy9YQIbZ/jiY02BQ7VLJx4qgrlqH7wAcqrQkvDs/jQD190kMSudGfF/4btwKzxAXwILffy0vyaLKo3/OFdIhSOIfxwjckFhVQEY6+nnby4+xIEE6AcPaOTwVe4dfQc19WNyijY9AgpZWxWK7hlvR5T2ELKgKwhDv33b91+9G52zOl5kTlo2fnq9VDac9YFGdjPRaTYB0nVfisuDYCmPZKypZ0ruoPoTj01dqVa3e18QCnNCdaTDvZHRztRwvsaqhh8lGk7DiJFfeZnOwxnzeilVY2MuodgMtUrGxAgVjfuZ7CsxQiKqzn+hEBcyFX5kbZ23P6GXBEak/OX+lwf+Ja9OC1GOcA8WYrTO7W+i6mSlh1tTFX8+L+AX/VKAtH5HJnHTSQ75N9kiECRpchW9O/4W4ZAmU89w+ocT5Qnf5lYAD/82qDcmbylGN2zyR5hd57snT+MkOYstrMbaIRsrU52HRAz3HHymu4vjVfIk tZfYO2Kk KO0pF+aCcj9Gk+flLj4VEO/qoXDLc8DZOviztl9fRxNIvQTBRDeEL64yevEcI4iknkrKK5mrlQ2q+FW3Ezom8XdZjg2e0Z/HWZ19PCqDHB0QB71N63gFOFWY+kmYsMv7UJNNX2RCTxcVXyOrG3OUg2BJk5TZQDfXLTPLqavaWEka6Pvb1ss1TODx294gReKlI5G6XseIpB7vDWEu4oZf8ioEzUJC2rl7gxPicKt1oc6u+ytk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: cgroup resource allocation has to be handled in TTM, so -EAGAIN from cgroups can be converted into -ENOSPC, and the limitcg can be properly evicted in ttm code. When hitting a resource limit through -EAGAIN, the cgroup for which the limit is hit is also returned. This allows eviction to delete only from cgroups which are a subgroup of the current cgroup. The returned CSS is used to determine if eviction is valuable for a given resource, and allows TTM to only target specific resources to lower memory usage. Signed-off-by: Maarten Lankhorst Co-developed-by: Friedrich Vock --- drivers/gpu/drm/ttm/tests/ttm_bo_test.c | 18 ++++----- drivers/gpu/drm/ttm/tests/ttm_resource_test.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 38 ++++++++++++++++--- drivers/gpu/drm/ttm/ttm_resource.c | 28 ++++++++++++-- include/drm/ttm/ttm_bo.h | 3 +- include/drm/ttm/ttm_resource.h | 16 +++++++- 6 files changed, 84 insertions(+), 21 deletions(-) diff --git a/drivers/gpu/drm/ttm/tests/ttm_bo_test.c b/drivers/gpu/drm/ttm/tests/ttm_bo_test.c index 1f8a4f8adc92..e2adc336dda8 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_bo_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_bo_test.c @@ -252,13 +252,13 @@ static void ttm_bo_unreserve_basic(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE); bo->priority = bo_prio; - err = ttm_resource_alloc(bo, place, &res1); + err = ttm_resource_alloc(bo, place, &res1, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res1; /* Add a dummy resource to populate LRU */ - ttm_resource_alloc(bo, place, &res2); + ttm_resource_alloc(bo, place, &res2, NULL); dma_resv_lock(bo->base.resv, NULL); ttm_bo_unreserve(bo); @@ -294,12 +294,12 @@ static void ttm_bo_unreserve_pinned(struct kunit *test) dma_resv_lock(bo->base.resv, NULL); ttm_bo_pin(bo); - err = ttm_resource_alloc(bo, place, &res1); + err = ttm_resource_alloc(bo, place, &res1, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res1; /* Add a dummy resource to the pinned list */ - err = ttm_resource_alloc(bo, place, &res2); + err = ttm_resource_alloc(bo, place, &res2, NULL); KUNIT_ASSERT_EQ(test, err, 0); KUNIT_ASSERT_EQ(test, list_is_last(&res2->lru, &priv->ttm_dev->pinned), 1); @@ -343,7 +343,7 @@ static void ttm_bo_unreserve_bulk(struct kunit *test) ttm_bo_set_bulk_move(bo1, &lru_bulk_move); dma_resv_unlock(bo1->base.resv); - err = ttm_resource_alloc(bo1, place, &res1); + err = ttm_resource_alloc(bo1, place, &res1, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo1->resource = res1; @@ -351,7 +351,7 @@ static void ttm_bo_unreserve_bulk(struct kunit *test) ttm_bo_set_bulk_move(bo2, &lru_bulk_move); dma_resv_unlock(bo2->base.resv); - err = ttm_resource_alloc(bo2, place, &res2); + err = ttm_resource_alloc(bo2, place, &res2, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo2->resource = res2; @@ -387,7 +387,7 @@ static void ttm_bo_put_basic(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE); bo->type = ttm_bo_type_device; - err = ttm_resource_alloc(bo, place, &res); + err = ttm_resource_alloc(bo, place, &res, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; @@ -504,7 +504,7 @@ static void ttm_bo_pin_unpin_resource(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE); - err = ttm_resource_alloc(bo, place, &res); + err = ttm_resource_alloc(bo, place, &res, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; @@ -555,7 +555,7 @@ static void ttm_bo_multiple_pin_one_unpin(struct kunit *test) bo = ttm_bo_kunit_init(test, test->priv, BO_SIZE); - err = ttm_resource_alloc(bo, place, &res); + err = ttm_resource_alloc(bo, place, &res, NULL); KUNIT_ASSERT_EQ(test, err, 0); bo->resource = res; diff --git a/drivers/gpu/drm/ttm/tests/ttm_resource_test.c b/drivers/gpu/drm/ttm/tests/ttm_resource_test.c index 029e1f094bb0..c7d3d86ff98b 100644 --- a/drivers/gpu/drm/ttm/tests/ttm_resource_test.c +++ b/drivers/gpu/drm/ttm/tests/ttm_resource_test.c @@ -302,7 +302,7 @@ static void ttm_sys_man_free_basic(struct kunit *test) res = kunit_kzalloc(test, sizeof(*res), GFP_KERNEL); KUNIT_ASSERT_NOT_NULL(test, res); - ttm_resource_alloc(bo, place, &res); + ttm_resource_alloc(bo, place, &res, NULL, NULL); man = ttm_manager_type(priv->devs->ttm_dev, mem_type); man->func->free(man, res); diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index 6396dece0db1..6ca92b64f2fe 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include "ttm_module.h" @@ -594,18 +595,24 @@ int ttm_mem_evict_first(struct ttm_device *bdev, struct ttm_resource_manager *man, const struct ttm_place *place, struct ttm_operation_ctx *ctx, - struct ww_acquire_ctx *ticket) + struct ww_acquire_ctx *ticket, + struct drmcgroup_pool_state *limitcss) { struct ttm_buffer_object *bo = NULL, *busy_bo = NULL; struct ttm_resource_cursor cursor; struct ttm_resource *res; bool locked = false; int ret; + bool try_low = false, hit_low = false; spin_lock(&bdev->lru_lock); +retry: ttm_resource_manager_for_each_res(man, &cursor, res) { bool busy; + if (!drmcs_evict_valuable(limitcss, man->cgdev, man->cgidx, res->css, try_low, &hit_low)) + continue; + if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place, &locked, &busy)) { if (busy && !busy_bo && ticket != @@ -623,13 +630,25 @@ int ttm_mem_evict_first(struct ttm_device *bdev, } if (!bo) { + if (!ticket && !try_low && hit_low) + goto hit_low; + if (busy_bo && !ttm_bo_get_unless_zero(busy_bo)) busy_bo = NULL; + + if (!busy_bo && !try_low && hit_low) + goto hit_low; + spin_unlock(&bdev->lru_lock); ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket); if (busy_bo) ttm_bo_put(busy_bo); return ret; + +hit_low: + busy_bo = NULL; + try_low = true; + goto retry; } if (bo->deleted) { @@ -769,14 +788,19 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo, continue; do { - ret = ttm_resource_alloc(bo, place, res); - if (unlikely(ret && ret != -ENOSPC)) + struct drmcgroup_pool_state *limitcss = NULL; + + ret = ttm_resource_alloc(bo, place, res, force_space ? &limitcss : NULL); + if (unlikely(ret && ret != -ENOSPC && ret != -EAGAIN)) { + drmcs_pool_put(limitcss); return ret; + } if (likely(!ret) || !force_space) break; ret = ttm_mem_evict_first(bdev, man, place, ctx, - ticket); + ticket, limitcss); + drmcs_pool_put(limitcss); if (unlikely(ret == -EBUSY)) break; if (unlikely(ret)) @@ -1162,7 +1186,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, memset(&hop, 0, sizeof(hop)); place.mem_type = TTM_PL_SYSTEM; - ret = ttm_resource_alloc(bo, &place, &evict_mem); + ret = ttm_resource_alloc(bo, &place, &evict_mem, NULL); if (unlikely(ret)) goto out; @@ -1201,7 +1225,9 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, if (locked) dma_resv_unlock(bo->base.resv); ttm_bo_put(bo); - return ret == -EBUSY ? -ENOSPC : ret; + if (ret == -EAGAIN || ret == -EBUSY) + return -ENOSPC; + return ret; } void ttm_bo_tt_destroy(struct ttm_buffer_object *bo) diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index 4a66b851b67d..1a8312afe247 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -229,15 +230,31 @@ EXPORT_SYMBOL(ttm_resource_fini); int ttm_resource_alloc(struct ttm_buffer_object *bo, const struct ttm_place *place, - struct ttm_resource **res_ptr) + struct ttm_resource **res_ptr, + struct drmcgroup_pool_state **limitcs) { struct ttm_resource_manager *man = ttm_manager_type(bo->bdev, place->mem_type); + struct drmcgroup_pool_state *drmcs = NULL; int ret; + if (man->cgdev) { + ret = drmcg_try_charge(&drmcs, limitcs, + man->cgdev, man->cgidx, + bo->base.size); + if (ret) + return ret; + } + ret = man->func->alloc(man, bo, place, res_ptr); - if (ret) + if (ret) { + if (drmcs) + drmcg_uncharge(drmcs, man->cgdev, man->cgidx, + bo->base.size); return ret; + } + + (*res_ptr)->css = drmcs; spin_lock(&bo->bdev->lru_lock); ttm_resource_add_bulk_move(*res_ptr, bo); @@ -249,6 +266,7 @@ EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_resource_alloc); void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res) { struct ttm_resource_manager *man; + struct drmcgroup_pool_state *css; if (!*res) return; @@ -256,9 +274,13 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res) spin_lock(&bo->bdev->lru_lock); ttm_resource_del_bulk_move(*res, bo); spin_unlock(&bo->bdev->lru_lock); + + css = (*res)->css; man = ttm_manager_type(bo->bdev, (*res)->mem_type); man->func->free(man, *res); *res = NULL; + if (man->cgdev) + drmcg_uncharge(css, man->cgdev, man->cgidx, bo->base.size); } EXPORT_SYMBOL(ttm_resource_free); @@ -401,7 +423,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, while (!list_empty(&man->lru[i])) { spin_unlock(&bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx, - NULL); + NULL, NULL); if (ret) return ret; spin_lock(&bdev->lru_lock); diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h index 6ccf96c91f3a..709a30b6bfe6 100644 --- a/include/drm/ttm/ttm_bo.h +++ b/include/drm/ttm/ttm_bo.h @@ -386,7 +386,8 @@ int ttm_mem_evict_first(struct ttm_device *bdev, struct ttm_resource_manager *man, const struct ttm_place *place, struct ttm_operation_ctx *ctx, - struct ww_acquire_ctx *ticket); + struct ww_acquire_ctx *ticket, + struct drmcgroup_pool_state *limitcg); vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo, struct vm_fault *vmf); vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h index 69769355139f..0574ebb226cb 100644 --- a/include/drm/ttm/ttm_resource.h +++ b/include/drm/ttm/ttm_resource.h @@ -38,6 +38,7 @@ #define TTM_MAX_BO_PRIORITY 4U #define TTM_NUM_MEM_TYPES 8 +struct drmcgroup_device; struct ttm_device; struct ttm_resource_manager; struct ttm_resource; @@ -174,6 +175,15 @@ struct ttm_resource_manager { * bdev->lru_lock. */ uint64_t usage; + + /** + * @cgdev: drmcgroup_device used for memory accounting, if not NULL. + */ + struct drmcgroup_device *cgdev; + /** + * @cgidx: Resource index used by this resource manager for cgroup accounting + */ + u32 cgidx; }; /** @@ -202,6 +212,7 @@ struct ttm_bus_placement { * @placement: Placement flags. * @bus: Placement on io bus accessible to the CPU * @bo: weak reference to the BO, protected by ttm_device::lru_lock + * @css: cgroup state this resource is charged to * * Structure indicating the placement and space resources used by a * buffer object. @@ -214,6 +225,8 @@ struct ttm_resource { struct ttm_bus_placement bus; struct ttm_buffer_object *bo; + struct drmcgroup_pool_state *css; + /** * @lru: Least recently used list, see &ttm_resource_manager.lru */ @@ -362,7 +375,8 @@ void ttm_resource_fini(struct ttm_resource_manager *man, int ttm_resource_alloc(struct ttm_buffer_object *bo, const struct ttm_place *place, - struct ttm_resource **res); + struct ttm_resource **res, + struct drmcgroup_pool_state **limitcs); void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res); bool ttm_resource_intersects(struct ttm_device *bdev, struct ttm_resource *res,