From patchwork Thu Apr 15 11:56:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12205239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8353C433B4 for ; Thu, 15 Apr 2021 11:56:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5797C613B0 for ; Thu, 15 Apr 2021 11:56:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5797C613B0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B24246B0036; Thu, 15 Apr 2021 07:56:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AFA3D6B006C; Thu, 15 Apr 2021 07:56:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 974126B0070; Thu, 15 Apr 2021 07:56:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 793736B0036 for ; Thu, 15 Apr 2021 07:56:27 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 34424349B for ; Thu, 15 Apr 2021 11:56:27 +0000 (UTC) X-FDA: 78034448814.18.AAF52A8 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by imf08.hostedemail.com (Postfix) with ESMTP id 964B580192F1 for ; Thu, 15 Apr 2021 11:56:12 +0000 (UTC) Received: by mail-ej1-f51.google.com with SMTP id n2so36411524ejy.7 for ; Thu, 15 Apr 2021 04:56:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MAgKFkUbajfTwkiBbjHCwLbY8LqpQ8PBmh7VA/cwU0E=; b=m1wDMNbKQex77GuTlU5+skb5l8cZdzF7kNI3HPNE8cQmGFBDeoAAfO3z62VHp2W3S6 g8hW0/7uNFuyEGt2cBpWlxY7pcdErauqg3dyEdRwrM0YzVCRm1joAJpAAV5NNdVsHQoF hahSKm6dCyKXf1Qc/eKCUHyZ69K7r1luODPcgcG5j0yP0ohshKObaq7m0ImYa5GWbCry 9Al1nVGzFju2O2/uydbJISxb5xBQhjAZqCnMhAz9VcnbR6pq5rT9YQjA3ZD+kQvHIDEE 6sfiAKFL4nt2J7KcscGOnWoI8wLfSRKUNDg4gXyWvv6+zlHwL+czM74h6/wzk/VJYSXm F+NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MAgKFkUbajfTwkiBbjHCwLbY8LqpQ8PBmh7VA/cwU0E=; b=jPldzIfvpVDCamC2vukIy+kZmYc8Ei1aVkSrDBKWcsM4fW0SURPzquTyLaZrE3vf+J POWoZoBTRbXO2wUioWYSP5pFnLaphvdke/cstxdoHm4EiCfEF2QI3QgPWI6lUbqsDR4t vHX4JwIrWHeboSoz527gJljDbF+IgulnL5zy+WlYBo9fq6/H4w8TYN+dGhp5F2+C5pT+ 1yUakB3An5qsJy1kPWzwvGDwINcE6fzYJaGgOaceVgIwxkgrIRh/fXoXMsCqcZry5M9o DnWOHAQYHOIefgleL9WmK5NLnCXSsa2axFzgqfvYRypjaAyiCvtTvO2cW09gYBZDXdgo LAyQ== X-Gm-Message-State: AOAM533H/6v9F6lF2uMkjMI8ScjH5QWZS6Rpsep6w+ZAj+L4QXaxTYJE swkYnUAxYtumOTYYxKGnTSSnUnK6a7U= X-Google-Smtp-Source: ABdhPJwgvhGuyBEvsRLBj8pJVlv1MD8sM4alBQQIq3AaHw/fvXpcPyQjXZnukRoS19wBBYBX8H5fjg== X-Received: by 2002:a17:906:524f:: with SMTP id y15mr2973184ejm.65.1618487785620; Thu, 15 Apr 2021 04:56:25 -0700 (PDT) Received: from abel.fritz.box ([2a02:908:1252:fb60:3ba1:ccd9:cd93:a8ed]) by smtp.gmail.com with ESMTPSA id s9sm2182461edd.16.2021.04.15.04.56.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Apr 2021 04:56:25 -0700 (PDT) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: vbabka@suse.cz, daniel@ffwll.ch, ray.huang@amd.com, akpm@linux-foundation.org Subject: [PATCH 1/2] mm/vmscan: add sync_shrinkers function Date: Thu, 15 Apr 2021 13:56:23 +0200 Message-Id: <20210415115624.2904-1-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Stat-Signature: d4mkuuro7xbqihxwp85i6qx5bnibjfk3 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 964B580192F1 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-ej1-f51.google.com; client-ip=209.85.218.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618487772-577202 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To be able to switch to a spinlock and reduce lock contention in the TTM shrinker we don't want to hold a mutex while unmapping and freeing pages from the pool. But then we somehow need to prevent a race between (for example) the shrinker trying to free pages and hotplug trying to remove the device which those pages belong to. Taking and releasing the shrinker semaphore on the write side after unmapping and freeing all pages should make sure that no shrinker is running in paralell any more. Signed-off-by: Christian König Acked-by: Daniel Vetter --- include/linux/shrinker.h | 1 + mm/vmscan.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..6b75dc372fce 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -92,4 +92,5 @@ extern void register_shrinker_prepared(struct shrinker *shrinker); extern int register_shrinker(struct shrinker *shrinker); extern void unregister_shrinker(struct shrinker *shrinker); extern void free_prealloced_shrinker(struct shrinker *shrinker); +extern void sync_shrinkers(void); #endif diff --git a/mm/vmscan.c b/mm/vmscan.c index 562e87cbd7a1..46cd9c215d73 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -408,6 +408,16 @@ void unregister_shrinker(struct shrinker *shrinker) } EXPORT_SYMBOL(unregister_shrinker); +/** + * sync_shrinker - Wait for all running shrinkers to complete. + */ +void sync_shrinkers(void) +{ + down_write(&shrinker_rwsem); + up_write(&shrinker_rwsem); +} +EXPORT_SYMBOL(sync_shrinkers); + #define SHRINK_BATCH 128 static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, From patchwork Thu Apr 15 11:56:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12205241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1215C43461 for ; Thu, 15 Apr 2021 11:56:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 65C34610FA for ; Thu, 15 Apr 2021 11:56:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65C34610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5E20F6B006C; Thu, 15 Apr 2021 07:56:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A4266B0070; Thu, 15 Apr 2021 07:56:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32DC46B0071; Thu, 15 Apr 2021 07:56:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 1462C6B006C for ; Thu, 15 Apr 2021 07:56:28 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C926B8248047 for ; Thu, 15 Apr 2021 11:56:27 +0000 (UTC) X-FDA: 78034448814.34.2F6F8EB Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf22.hostedemail.com (Postfix) with ESMTP id 53820C0007C2 for ; Thu, 15 Apr 2021 11:56:23 +0000 (UTC) Received: by mail-ej1-f44.google.com with SMTP id mh2so14882914ejb.8 for ; Thu, 15 Apr 2021 04:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Mmzr1BXITN8/WsQtuIlDuVTV5ipAk7FG2VM5j+Bi1So=; b=QvQMXegx4Q6FfPIpt3BYD7d/rIzGJpkhRMRMpxNI2Jc9ovzo2ea9Po/pjK9xcot10J qfrgGeGvDxK0UtoQelPc859qvv3lCo3V+I61eOMYlY8VoruRr/1VzBriz8TYVT38gXGi bKsI+0K3iURKjYw5ibQslYc04LzCB2HkYJNOWtQdeJtmREQywEIO3U6qpcFFSzp1UUSr dtudZ7kiiIBbHYxn6lznMpdvme+RuMnAyO8YBP/JdG6KqY3gxKnonMJ2xQr0SQ3k8UEY 6eAqAWXS/841WW5UHmuMlxOhssYyTS1s5DnIKzXWT6WFbvJt2mBq4t3ivxQ5Rtr6d49m ObjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Mmzr1BXITN8/WsQtuIlDuVTV5ipAk7FG2VM5j+Bi1So=; b=M9V/18IwfL7dNpsHuXDZ5wxsUX08/P7VoCAc6t7JjjzULvme1whu+2+yqdiBA+1GXa s91eECmchCnnsU/TvWoWWayqRLpTPKWFBS5rBduceu6cPQfCoXrWTIKFYudKXy3/3MQ9 KQYWxUtCpKidTwqeXodwIg4K5ICHg3OXKpZubL8YHqvU5m5huAWK449LGWrIIgT8g4Kb qfYxELSJ01usGelhil9hFLVva95o7Zr9AGGKvv3rIYJqPpqIkq4mT/6C0N6gb6ihOiGO iNcSIFWakvGGEN5jWVgUbkVqd3xRWsi+KMEaYKtlloseYayt+UCQAaq5IvwG9MyVe8/Z BXaw== X-Gm-Message-State: AOAM531mRZsJ6R+gayRzlqNT9ba5XDCYMDxZbCc5IrNY2PkG4/j1ehr/ KoBXh308sWv4sYgOP6RslO/7VfgiWHE= X-Google-Smtp-Source: ABdhPJyRvFr8GCcyp6cP4pUhEHuqjgAxms+Re3dD8oeMsGnJsnFn3mLckPuF7TSkfuSpiaA2RZVQPA== X-Received: by 2002:a17:906:278e:: with SMTP id j14mr3012533ejc.224.1618487786343; Thu, 15 Apr 2021 04:56:26 -0700 (PDT) Received: from abel.fritz.box ([2a02:908:1252:fb60:3ba1:ccd9:cd93:a8ed]) by smtp.gmail.com with ESMTPSA id s9sm2182461edd.16.2021.04.15.04.56.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Apr 2021 04:56:26 -0700 (PDT) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: vbabka@suse.cz, daniel@ffwll.ch, ray.huang@amd.com, akpm@linux-foundation.org Subject: [PATCH 2/2] drm/ttm: optimize the pool shrinker a bit v2 Date: Thu, 15 Apr 2021 13:56:24 +0200 Message-Id: <20210415115624.2904-2-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210415115624.2904-1-christian.koenig@amd.com> References: <20210415115624.2904-1-christian.koenig@amd.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 53820C0007C2 X-Stat-Signature: gng6yjnfhp77ok1mxrrup7npgfycx8ef Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-ej1-f44.google.com; client-ip=209.85.218.44 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618487783-15962 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Switch back to using a spinlock again by moving the IOMMU unmap outside of the locked region. v2: Add a comment explaining why we need sync_shrinkers(). Signed-off-by: Christian König --- drivers/gpu/drm/ttm/ttm_pool.c | 44 +++++++++++++++++----------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c index cb38b1a17b09..955836d569cc 100644 --- a/drivers/gpu/drm/ttm/ttm_pool.c +++ b/drivers/gpu/drm/ttm/ttm_pool.c @@ -70,7 +70,7 @@ static struct ttm_pool_type global_uncached[MAX_ORDER]; static struct ttm_pool_type global_dma32_write_combined[MAX_ORDER]; static struct ttm_pool_type global_dma32_uncached[MAX_ORDER]; -static struct mutex shrinker_lock; +static spinlock_t shrinker_lock; static struct list_head shrinker_list; static struct shrinker mm_shrinker; @@ -263,9 +263,9 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, spin_lock_init(&pt->lock); INIT_LIST_HEAD(&pt->pages); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); list_add_tail(&pt->shrinker_list, &shrinker_list); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); } /* Remove a pool_type from the global shrinker list and free all pages */ @@ -273,9 +273,9 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt) { struct page *p; - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); list_del(&pt->shrinker_list); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); while ((p = ttm_pool_type_take(pt))) ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); @@ -313,24 +313,19 @@ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, static unsigned int ttm_pool_shrink(void) { struct ttm_pool_type *pt; - unsigned int num_freed; struct page *p; - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list); + list_move_tail(&pt->shrinker_list, &shrinker_list); + spin_unlock(&shrinker_lock); p = ttm_pool_type_take(pt); - if (p) { - ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); - num_freed = 1 << pt->order; - } else { - num_freed = 0; - } - - list_move_tail(&pt->shrinker_list, &shrinker_list); - mutex_unlock(&shrinker_lock); + if (!p) + return 0; - return num_freed; + ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); + return 1 << pt->order; } /* Return the allocation order based for a page */ @@ -530,6 +525,11 @@ void ttm_pool_fini(struct ttm_pool *pool) for (j = 0; j < MAX_ORDER; ++j) ttm_pool_type_fini(&pool->caching[i].orders[j]); } + + /* We removed the pool types from the LRU, but we need to also make sure + * that no shrinker is concurrently freeing pages from the pool. + */ + sync_shrinkers(); } /* As long as pages are available make sure to release at least one */ @@ -604,7 +604,7 @@ static int ttm_pool_debugfs_globals_show(struct seq_file *m, void *data) { ttm_pool_debugfs_header(m); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); seq_puts(m, "wc\t:"); ttm_pool_debugfs_orders(global_write_combined, m); seq_puts(m, "uc\t:"); @@ -613,7 +613,7 @@ static int ttm_pool_debugfs_globals_show(struct seq_file *m, void *data) ttm_pool_debugfs_orders(global_dma32_write_combined, m); seq_puts(m, "uc 32\t:"); ttm_pool_debugfs_orders(global_dma32_uncached, m); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); ttm_pool_debugfs_footer(m); @@ -640,7 +640,7 @@ int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m) ttm_pool_debugfs_header(m); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); for (i = 0; i < TTM_NUM_CACHING_TYPES; ++i) { seq_puts(m, "DMA "); switch (i) { @@ -656,7 +656,7 @@ int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m) } ttm_pool_debugfs_orders(pool->caching[i].orders, m); } - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); ttm_pool_debugfs_footer(m); return 0; @@ -693,7 +693,7 @@ int ttm_pool_mgr_init(unsigned long num_pages) if (!page_pool_size) page_pool_size = num_pages; - mutex_init(&shrinker_lock); + spin_lock_init(&shrinker_lock); INIT_LIST_HEAD(&shrinker_list); for (i = 0; i < MAX_ORDER; ++i) {