From patchwork Fri Jul 6 19:32:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10512455 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D623C60325 for ; Fri, 6 Jul 2018 19:34:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4FEE287B4 for ; Fri, 6 Jul 2018 19:34:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B91E9287B9; Fri, 6 Jul 2018 19:34:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16447287B4 for ; Fri, 6 Jul 2018 19:34:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C3A96B026D; Fri, 6 Jul 2018 15:34:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 451576B0271; Fri, 6 Jul 2018 15:34:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D3656B026D; Fri, 6 Jul 2018 15:34:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f199.google.com (mail-qk0-f199.google.com [209.85.220.199]) by kanga.kvack.org (Postfix) with ESMTP id CB1A06B0273 for ; Fri, 6 Jul 2018 15:34:31 -0400 (EDT) Received: by mail-qk0-f199.google.com with SMTP id w126-v6so4895843qka.11 for ; Fri, 06 Jul 2018 12:34:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=NZsEjb3hm6vtBy2+vEJjj0jCz2k2OXZ1oGJN7Xuw6g0=; b=DKjxkkdGEXj5T34a1QfJxa45E65L8MdPmkLoTI+F7TGM69Hb4CpjoGk79kEmgs7mJV aLm8xnfqZMEt5qMbtoNPNPjssEXNfdB5foX21+N+IPsEMcJfLFx/7S6yloXg5m8YbTxQ xEEloBdUtIXHCyFYVWEyJqO63al3C3DlIDIi11J8I7jM8eYQXI7SpjVs45lOH3bE1D4A Bn8avNSR+7yCeSTRnwgRDkDazP3q1MvpuyH4Voly8ZgBXRsu9PmVzV00B5JppKDyXUkM uzv/9xP23DbNkbM9HvBN5VN74/5ZC/ODxV5aIOP2sKy39MqLdXeADNBHhB9N5vnMhDgz 0+5w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APt69E3RQMkMFdoVYNkFsH07asGPMQheHT1GafKyBMPv055bbOUvH0tP dm0CgWe5PEPwi1/8mcAZNnqQ12EAyS7ea7GYbhse5EdId/HWAPDOeb5At9SjbuZCAOBvmrWUA3A eRTnNagopTfFEf9hQizWXpX9Ph+FnO1PxOM4m+CFzn7iUNTZUg6I0ywJbmZw+NCQYLQ== X-Received: by 2002:ac8:25c6:: with SMTP id f6-v6mr10871183qtf.270.1530905671605; Fri, 06 Jul 2018 12:34:31 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeiF/7QP+21v/XBOGvPcNNqNXLr7R+mFWuu16U2G0qDms1t1LS8zK9h4AzLxjD8V/o3pd/i X-Received: by 2002:ac8:25c6:: with SMTP id f6-v6mr10871139qtf.270.1530905670819; Fri, 06 Jul 2018 12:34:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530905670; cv=none; d=google.com; s=arc-20160816; b=Tk4hdNOv1lS+DafSUIBdHLEx2RZLzKXcQb13CBoivoDpp/O6HCgrfwS291m9WCfnhI PqTt5R05mosaR9ixTlWyu+g+c0b3WAuuEg8tfhSqdVNqptLB7GAaR72mWBBy5O2B+OvU eIaWyDevcy1RwQovVKZGUYAmrcy18t9FNizf7az8gZC80xjcZSaRjwRSIzkVnBHtH/+J 0YWj7lQjWyGjYESIblo1qVubA4lSd4q/j2QxXmFle6d6S3WkXS8RcxGCL3UBta3LtWSn gFDaD5HpmY3W5U+uXFkOmRaQLK903hWl6PS/HlhvAv//gz33VYVV4jL7SrTDWjS6RMzw q2+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=NZsEjb3hm6vtBy2+vEJjj0jCz2k2OXZ1oGJN7Xuw6g0=; b=fWNx7U7YtAT6GCrrtBlKHUr32VdwYjQzohthPjIb8Na4dVmrPcSOaU1JLCp6tRu+wS J8Vb5dgg8gL1IldPtiyQwO6TUEVS/zsiAlc/w1yrPM6N5m1rEjxrJJV/syTmK0VL4fI5 h4qk2vKcUCSPKDqbw3+hyIGHqK6OgbSMXa0G8N5jCc7jfckwmli78pyLEnpJZY4EWZxx 1azIgDYbxhaxWe3P4uY8GYma02NMz+qq2hRIaI2zSnRX+ee99HjYt7rdaUJHMie6ytj/ jhM9WoNzZ19JajOcL1kFaaBD3ZjqFTLcfSbS9gthiDRVd/0r9Z3mRKyGwmSLFf7hBQ0J 7P0w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id v50-v6si3930680qtv.73.2018.07.06.12.34.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jul 2018 12:34:30 -0700 (PDT) Received-SPF: pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) client-ip=66.187.233.73; Authentication-Results: mx.google.com; spf=pass (google.com: domain of longman@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 68091401DE71; Fri, 6 Jul 2018 19:34:30 +0000 (UTC) Received: from llong.com (dhcp-17-175.bos.redhat.com [10.18.17.175]) by smtp.corp.redhat.com (Postfix) with ESMTP id EABC97C2F; Fri, 6 Jul 2018 19:34:29 +0000 (UTC) From: Waiman Long To: Alexander Viro , Jonathan Corbet , "Luis R. Rodriguez" , Kees Cook Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, Linus Torvalds , Jan Kara , "Paul E. McKenney" , Andrew Morton , Ingo Molnar , Miklos Szeredi , Matthew Wilcox , Larry Woodman , James Bottomley , "Wangkai (Kevin C)" , Waiman Long Subject: [PATCH v6 5/7] fs/dcache: Add negative dentries to LRU head initially Date: Fri, 6 Jul 2018 15:32:50 -0400 Message-Id: <1530905572-817-6-git-send-email-longman@redhat.com> In-Reply-To: <1530905572-817-1-git-send-email-longman@redhat.com> References: <1530905572-817-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Fri, 06 Jul 2018 19:34:30 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Fri, 06 Jul 2018 19:34:30 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'longman@redhat.com' RCPT:'' X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP For negative dentries that are accessed once and never reused again, there is not much value in putting the dentries at the tail of the LRU list and keep it for a long time. So a new DCACHE_LRU_HEAD flag is added to a negative dentry when it is initially created. When such a dentry is added to the LRU, it will be added to the head so that it will be the first to go when a shrinker is running. The flag is then cleared after the LRU list addition. So if that dentry is accessed again, it will be put back to the tail like the rest of the dentries. By running a negative dentry generator for a certain period of time and let the automatic pruning process to run through its course, the number of negative and positive dentries discarded were: 681 iterations, 43503/60 neg/pos dentries freed. 45115 iterations, 2884992/64 neg/pos dentries freed. So the number of positive dentries discarded is only 124. Without this patch, the number of negative and positive dentries discarded would be: 20 iterations, 598/483 neg/pos dentries freed. 60 iterations, 2977/517 neg/pos dentries freed. 31 iterations, 1060/599 neg/pos dentries freed. 11 iterations, 447/103 neg/pos dentries freed. 17 iterations, 682/304 neg/pos dentries freed. 17 iterations, 555/196 neg/pos dentries freed. 33008 iterations, 2094860/7624 neg/pos dentries freed. It can be seen that a lot more positive dentries would have been lost as collateral damage in this case. Suggested-by: Larry Woodman Signed-off-by: Waiman Long --- fs/dcache.c | 18 +++++++++++++++++- include/linux/dcache.h | 1 + include/linux/list_lru.h | 17 +++++++++++++++++ mm/list_lru.c | 19 +++++++++++++++++-- 4 files changed, 52 insertions(+), 3 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 3be9246..ec007ac 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -615,10 +615,23 @@ static void dentry_unlink_inode(struct dentry * dentry) #define D_FLAG_VERIFY(dentry,x) WARN_ON_ONCE(((dentry)->d_flags & (DCACHE_LRU_LIST | DCACHE_SHRINK_LIST)) != (x)) static void d_lru_add(struct dentry *dentry) { + int ret; + D_FLAG_VERIFY(dentry, 0); dentry->d_flags |= DCACHE_LRU_LIST; this_cpu_inc(nr_dentry_unused); - WARN_ON_ONCE(!list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); + if (unlikely(dentry->d_flags & DCACHE_LRU_HEAD)) { + /* + * Add to the head once, it will be added to the tail + * next time. + */ + ret = list_lru_add_head(&dentry->d_sb->s_dentry_lru, + &dentry->d_lru); + dentry->d_flags &= ~DCACHE_LRU_HEAD; + } else { + ret = list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru); + } + WARN_ON_ONCE(!ret); neg_dentry_inc(dentry); } @@ -2988,6 +3001,9 @@ static inline void __d_add(struct dentry *dentry, struct inode *inode) __d_set_inode_and_type(dentry, inode, add_flags); raw_write_seqcount_end(&dentry->d_seq); fsnotify_update_flags(dentry); + } else { + /* It is a negative dentry, add it to LRU head initially. */ + dentry->d_flags |= DCACHE_LRU_HEAD; } __d_rehash(dentry); if (dir) diff --git a/include/linux/dcache.h b/include/linux/dcache.h index 44e19d9..317e040 100644 --- a/include/linux/dcache.h +++ b/include/linux/dcache.h @@ -215,6 +215,7 @@ struct dentry_operations { #define DCACHE_FALLTHRU 0x01000000 /* Fall through to lower layer */ #define DCACHE_ENCRYPTED_WITH_KEY 0x02000000 /* dir is encrypted with a valid key */ #define DCACHE_OP_REAL 0x04000000 +#define DCACHE_LRU_HEAD 0x08000000 /* Add to LRU head initially */ #define DCACHE_PAR_LOOKUP 0x10000000 /* being looked up (with parent locked shared) */ #define DCACHE_DENTRY_CURSOR 0x20000000 diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index a9598a0..7856435 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -87,6 +87,23 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, bool list_lru_add(struct list_lru *lru, struct list_head *item); /** + * list_lru_add_head: add an element to the lru list's head + * @list_lru: the lru pointer + * @item: the item to be added. + * + * This is similar to list_lru_add(). The only difference is the location + * where the new item will be added. The list_lru_add() function will add + * the new item to the tail as it is the most recently used one. The + * list_lru_add_head() will add the new item into the head so that it + * will the first to go if a shrinker is running. So this function should + * only be used for less important item that can be the first to go if + * the system is under memory pressure. + * + * Return value: true if the list was updated, false otherwise + */ +bool list_lru_add_head(struct list_lru *lru, struct list_head *item); + +/** * list_lru_del: delete an element to the lru list * @list_lru: the lru pointer * @item: the item to be deleted. diff --git a/mm/list_lru.c b/mm/list_lru.c index 2ee5d3a..4ea3c1e 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -107,7 +107,8 @@ static inline bool list_lru_memcg_aware(struct list_lru *lru) } #endif /* CONFIG_MEMCG && !CONFIG_SLOB */ -bool list_lru_add(struct list_lru *lru, struct list_head *item) +static inline bool __list_lru_add(struct list_lru *lru, struct list_head *item, + const bool add_tail) { int nid = page_to_nid(virt_to_page(item)); struct list_lru_node *nlru = &lru->node[nid]; @@ -116,7 +117,10 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (list_empty(item)) { l = list_lru_from_kmem(nlru, item); - list_add_tail(item, &l->list); + if (add_tail) + list_add_tail(item, &l->list); + else + list_add(item, &l->list); l->nr_items++; nlru->nr_items++; spin_unlock(&nlru->lock); @@ -125,8 +129,19 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) spin_unlock(&nlru->lock); return false; } + +bool list_lru_add(struct list_lru *lru, struct list_head *item) +{ + return __list_lru_add(lru, item, true); +} EXPORT_SYMBOL_GPL(list_lru_add); +bool list_lru_add_head(struct list_lru *lru, struct list_head *item) +{ + return __list_lru_add(lru, item, false); +} +EXPORT_SYMBOL_GPL(list_lru_add_head); + bool list_lru_del(struct list_lru *lru, struct list_head *item) { int nid = page_to_nid(virt_to_page(item));