From patchwork Mon Nov 5 11:13:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuo-Hsin Yang X-Patchwork-Id: 10667789 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 916E315A6 for ; Mon, 5 Nov 2018 11:14:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CF5E2919E for ; Mon, 5 Nov 2018 11:14:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7069A29556; Mon, 5 Nov 2018 11:14:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3B4D2919E for ; Mon, 5 Nov 2018 11:14:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32FBD6B0003; Mon, 5 Nov 2018 06:14:00 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2DE516B0006; Mon, 5 Nov 2018 06:14:00 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17EE86B0007; Mon, 5 Nov 2018 06:14:00 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id CA6B36B0003 for ; Mon, 5 Nov 2018 06:13:59 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id n5-v6so8232349pgv.6 for ; Mon, 05 Nov 2018 03:13:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:mime-version:content-transfer-encoding; bh=abV7ObZZNRBHakE/HnnauCZ6ct0p4d6mTWV3qMkw7j0=; b=tp6VbT76g7vxG5z3yqjYC0v4Zgndyfr7ePd43wCasZgSzoJoDju0YbV89CoYY0U9Mv MQUPyULP5+3twXKyJp/dWrzkarMv/izyivJvGbERBt6ATE3qG6059QU6TkZuxKbqh4vK /k9E2brihl5JTHVvVGldEDecVzzao1Gavz6ixfKGyxe/nFjSl3SzN4jxVpmGRKO2peXD glUScyT/PqKmvv5MWGb4h8aUx07ADMWpCNTlyKjgciLKH0AK/pB4Ihk5D/m+oyPPUpud q8i5+Ku2FnW66I27YpCfNKbaKQMjp9DOs3lwyEZmTaeOQc8BoNWXWtykU2TPy2KgpzCX QWyw== X-Gm-Message-State: AGRZ1gI2YJJjrVStfb4WbwFNDjejGVcMy3kh/+OtEmeUGGCVSqpPgCUE cUJ4svZW2vNQD7I6VcOekEh4Jv357i15Fv/NrJPwK7P/dYazbJHQbNG8EDWdgwnht88uVWtfZD2 9V93LtYU2SWOx5qUfs3OTuoLao0fkmPRSb4PT6gWouex57mKpK+a37Jq8bf3UlwXouxegQ0T95M +Tdds0E9+5tasaxHHKcxWvoU4TriBpNk59qSq6+uU5fmQ2c2AQ0+a4ZejaGdJgWFPou/Ab6VVY5 DqdhrTmZSXjoQ/3LsUHHk8WtPMUjqx3mXI8ZtxQs5jEOb3JQ1ndlcDysyaWvONV7hLhQ1pK+45J X+eMjzNOC1cufM9/xuLMi2rdn5k/nHMI8XTrgCFEzk7WZwhxHltV2gRYCYCz7sNXHYFZWQtIf7K l X-Received: by 2002:a62:cac4:: with SMTP id y65-v6mr21551993pfk.27.1541416439464; Mon, 05 Nov 2018 03:13:59 -0800 (PST) X-Received: by 2002:a62:cac4:: with SMTP id y65-v6mr21551939pfk.27.1541416438349; Mon, 05 Nov 2018 03:13:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541416438; cv=none; d=google.com; s=arc-20160816; b=z+XArdHD5NXNQi8fZiyYbmxjznYFI7UV5lup0v6Xw3ziSaKJ/21xIzY5BMdgmdd3V8 0NwoivoMqtIrTZJhVKtvA4AcXvgAdGpR0vuEbXoLK+5SDK/HT4HybhxqijfkW/OLy2Eh Sz0lwZdcPaGGJEvxAZFIduOPom7qrtQveCJdQcutop+PQU+EehPime/rmdxrZjC+yJYH U20uMHf8egmXzy75hZuqOXtZPB31rMx6wsoYuFWYGzFJzhGC9Y41QtMs1MtBFF2TqvkG /gw0pDur25RzixVQUHCB8YudUgA0BzHfsCqbpMzlWLqM1p75gYcp09rrDccyeTTTPPGh am+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=abV7ObZZNRBHakE/HnnauCZ6ct0p4d6mTWV3qMkw7j0=; b=d/Fe1jUARpTcOj15WnY25+Tr/kqkPPnRmdmYm0lv1xWSIiQrgUmjykXTGQ1FN61srn VR28r+3fe+Ikc8ksjZPIYDXEKO9SWcDPFmvOUBAhumzIAJccK+NG9kIalkszwKk7sQyr DtITY5CPBQovvPG5rqzjz55Fm1Um8ONwv8IGqJpw4gkxJwgiBDjBGDsNlFN1id9c/LKD MdAFviBV5ohb3lu0zdixeoIsC0NfxGVLflOOvD1tQz8Zg8+rLlSRElZUmvfPabRYh7Xs /6YHu4driy2H03L2QDf3j0kYIcLEJwcAIFxizc8WbUZ6yo35/uQJM0Wp6CJKnV3k2bk0 LmYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=OOYpW0HS; spf=pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=vovoy@chromium.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s17-v6sor43289423pfi.2.2018.11.05.03.13.58 for (Google Transport Security); Mon, 05 Nov 2018 03:13:58 -0800 (PST) Received-SPF: pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=OOYpW0HS; spf=pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=vovoy@chromium.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=abV7ObZZNRBHakE/HnnauCZ6ct0p4d6mTWV3qMkw7j0=; b=OOYpW0HSCeUaR7Kuw0j9tbMZ2QdR5xrjCMgv2gfCleUFlk+4YeIOWJ1UdwQpj+TPyy mM46OWlwbGImPgIVAB7jEsOSCAkKssye9+QY4aY5+zZ9iLcJUzgv0CCnq/JJJt6YShaI sk3TePgyxBVYlzJMOQZPwAUpVdE4wnb4rkWJk= X-Google-Smtp-Source: AJdET5e3OdUhEZQV+dA1Umy8XyycOtq6naZlJBcd4a889AMmcXUPosDVO8MEybuRkCxxfAsHubmZLQ== X-Received: by 2002:a62:647:: with SMTP id 68-v6mr12181493pfg.42.1541416437676; Mon, 05 Nov 2018 03:13:57 -0800 (PST) Received: from vovoy-z840.tpe.corp.google.com ([2401:fa00:1:b:c11e:9571:ae83:d95b]) by smtp.gmail.com with ESMTPSA id r81-v6sm81531479pfa.110.2018.11.05.03.13.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 03:13:57 -0800 (PST) From: Kuo-Hsin Yang To: linux-kernel@vger.kernel.org, intel-gfx@lists.freedesktop.org, linux-mm@kvack.org Cc: Kuo-Hsin Yang , Chris Wilson , Michal Hocko , Joonas Lahtinen , Peter Zijlstra , Andrew Morton , Dave Hansen Subject: [PATCH v4] mm, drm/i915: mark pinned shmemfs pages as unevictable Date: Mon, 5 Nov 2018 19:13:48 +0800 Message-Id: <20181105111348.182492-1-vovoy@chromium.org> X-Mailer: git-send-email 2.19.1.930.g4563a0d9d0-goog MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The i915 driver uses shmemfs to allocate backing storage for gem objects. These shmemfs pages can be pinned (increased ref count) by shmem_read_mapping_page_gfp(). When a lot of pages are pinned, vmscan wastes a lot of time scanning these pinned pages. In some extreme case, all pages in the inactive anon lru are pinned, and only the inactive anon lru is scanned due to inactive_ratio, the system cannot swap and invokes the oom-killer. Mark these pinned pages as unevictable to speed up vmscan. Export pagevec API check_move_unevictable_pages(). This patch was inspired by Chris Wilson's change [1]. [1]: https://patchwork.kernel.org/patch/9768741/ Cc: Chris Wilson Cc: Michal Hocko Cc: Joonas Lahtinen Cc: Peter Zijlstra Cc: Andrew Morton Cc: Dave Hansen Signed-off-by: Kuo-Hsin Yang Acked-by: Michal Hocko --- Changes for v4: Export pagevec API check_move_unevictable_pages(). Changes for v3: Use check_move_lru_page instead of shmem_unlock_mapping to move pages to appropriate lru lists. Changes for v2: Squashed the two patches. Documentation/vm/unevictable-lru.rst | 4 +++- drivers/gpu/drm/i915/i915_gem.c | 25 +++++++++++++++++++++++-- include/linux/swap.h | 4 +++- mm/shmem.c | 2 +- mm/vmscan.c | 18 +++++++++--------- 5 files changed, 39 insertions(+), 14 deletions(-) diff --git a/Documentation/vm/unevictable-lru.rst b/Documentation/vm/unevictable-lru.rst index fdd84cb8d511..a812fb55136d 100644 --- a/Documentation/vm/unevictable-lru.rst +++ b/Documentation/vm/unevictable-lru.rst @@ -143,7 +143,7 @@ using a number of wrapper functions: Query the address space, and return true if it is completely unevictable. -These are currently used in two places in the kernel: +These are currently used in three places in the kernel: (1) By ramfs to mark the address spaces of its inodes when they are created, and this mark remains for the life of the inode. @@ -154,6 +154,8 @@ These are currently used in two places in the kernel: swapped out; the application must touch the pages manually if it wants to ensure they're in memory. + (3) By the i915 driver to mark pinned address space until it's unpinned. + Detecting Unevictable Pages --------------------------- diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 0c8aa57ce83b..7972eeb2e921 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -2381,12 +2381,22 @@ void __i915_gem_object_invalidate(struct drm_i915_gem_object *obj) invalidate_mapping_pages(mapping, 0, (loff_t)-1); } +/* Move pages to appropriate lru and release the pagevec */ +static inline void check_release_pagevec(struct pagevec *pvec) +{ + if (pagevec_count(pvec)) { + check_move_unevictable_pages(pvec); + __pagevec_release(pvec); + } +} + static void i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, struct sg_table *pages) { struct sgt_iter sgt_iter; struct page *page; + struct pagevec pvec; __i915_gem_object_release_shmem(obj, pages, true); @@ -2395,6 +2405,9 @@ i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, if (i915_gem_object_needs_bit17_swizzle(obj)) i915_gem_object_save_bit_17_swizzle(obj, pages); + mapping_clear_unevictable(file_inode(obj->base.filp)->i_mapping); + + pagevec_init(&pvec); for_each_sgt_page(page, sgt_iter, pages) { if (obj->mm.dirty) set_page_dirty(page); @@ -2402,8 +2415,10 @@ i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, if (obj->mm.madv == I915_MADV_WILLNEED) mark_page_accessed(page); - put_page(page); + if (!pagevec_add(&pvec, page)) + check_release_pagevec(&pvec); } + check_release_pagevec(&pvec); obj->mm.dirty = false; sg_free_table(pages); @@ -2526,6 +2541,7 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) unsigned int sg_page_sizes; gfp_t noreclaim; int ret; + struct pagevec pvec; /* * Assert that the object is not currently in any GPU domain. As it @@ -2559,6 +2575,7 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) * Fail silently without starting the shrinker */ mapping = obj->base.filp->f_mapping; + mapping_set_unevictable(mapping); noreclaim = mapping_gfp_constraint(mapping, ~__GFP_RECLAIM); noreclaim |= __GFP_NORETRY | __GFP_NOWARN; @@ -2673,8 +2690,12 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) err_sg: sg_mark_end(sg); err_pages: + mapping_clear_unevictable(mapping); + pagevec_init(&pvec); for_each_sgt_page(page, sgt_iter, st) - put_page(page); + if (!pagevec_add(&pvec, page)) + check_release_pagevec(&pvec); + check_release_pagevec(&pvec); sg_free_table(st); kfree(st); diff --git a/include/linux/swap.h b/include/linux/swap.h index d8a07a4f171d..a8f6d5d89524 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -18,6 +18,8 @@ struct notifier_block; struct bio; +struct pagevec; + #define SWAP_FLAG_PREFER 0x8000 /* set if swap priority specified */ #define SWAP_FLAG_PRIO_MASK 0x7fff #define SWAP_FLAG_PRIO_SHIFT 0 @@ -369,7 +371,7 @@ static inline int node_reclaim(struct pglist_data *pgdat, gfp_t mask, #endif extern int page_evictable(struct page *page); -extern void check_move_unevictable_pages(struct page **, int nr_pages); +extern void check_move_unevictable_pages(struct pagevec *pvec); extern int kswapd_run(int nid); extern void kswapd_stop(int nid); diff --git a/mm/shmem.c b/mm/shmem.c index 56bf122e0bb4..6ada7c97264d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -756,7 +756,7 @@ void shmem_unlock_mapping(struct address_space *mapping) break; index = indices[pvec.nr - 1] + 1; pagevec_remove_exceptionals(&pvec); - check_move_unevictable_pages(pvec.pages, pvec.nr); + check_move_unevictable_pages(&pvec); pagevec_release(&pvec); cond_resched(); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 62ac0c488624..6d0d08a33144 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -50,6 +50,7 @@ #include #include #include +#include #include #include @@ -4184,15 +4185,13 @@ int page_evictable(struct page *page) #ifdef CONFIG_SHMEM /** - * check_move_unevictable_pages - check pages for evictability and move to appropriate zone lru list - * @pages: array of pages to check - * @nr_pages: number of pages to check + * check_move_unevictable_pages - move evictable pages to appropriate evictable + * lru lists + * @pvec: pagevec with pages to check * - * Checks pages for evictability and moves them to the appropriate lru list. - * - * This function is only used for SysV IPC SHM_UNLOCK. + * This function is only used to move shmem pages. */ -void check_move_unevictable_pages(struct page **pages, int nr_pages) +void check_move_unevictable_pages(struct pagevec *pvec) { struct lruvec *lruvec; struct pglist_data *pgdat = NULL; @@ -4200,8 +4199,8 @@ void check_move_unevictable_pages(struct page **pages, int nr_pages) int pgrescued = 0; int i; - for (i = 0; i < nr_pages; i++) { - struct page *page = pages[i]; + for (i = 0; i < pvec->nr; i++) { + struct page *page = pvec->pages[i]; struct pglist_data *pagepgdat = page_pgdat(page); pgscanned++; @@ -4233,4 +4232,5 @@ void check_move_unevictable_pages(struct page **pages, int nr_pages) spin_unlock_irq(&pgdat->lru_lock); } } +EXPORT_SYMBOL(check_move_unevictable_pages); #endif /* CONFIG_SHMEM */