From patchwork Mon Jul 10 07:17:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 9832385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7EEDB60318 for ; Mon, 10 Jul 2017 07:18:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 048C326E96 for ; Mon, 10 Jul 2017 07:18:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EAAAF26256; Mon, 10 Jul 2017 07:18:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B734B26256 for ; Mon, 10 Jul 2017 07:18:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752454AbdGJHR6 (ORCPT ); Mon, 10 Jul 2017 03:17:58 -0400 Received: from mx2.suse.de ([195.135.220.15]:49287 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752317AbdGJHR5 (ORCPT ); Mon, 10 Jul 2017 03:17:57 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 33297AAC5; Mon, 10 Jul 2017 07:17:37 +0000 (UTC) From: NeilBrown To: Phil Kauffman , linux-nfs@vger.kernel.org Date: Mon, 10 Jul 2017 17:17:27 +1000 Subject: Re: /etc/mtab read ~900 times by rpc.mountd In-Reply-To: <595F1A3A.7070405@cs.uchicago.edu> References: <8737a9x9ky.fsf@notabene.neil.brown.name> <595F1A3A.7070405@cs.uchicago.edu> Message-ID: <87efto69rs.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Jul 07 2017, Phil Kauffman wrote: >> I can imagine /etc/mtab being read once for every line in /etc/exports, >> but unless your /etc/exports is very big, I can't easily see why it >> would be read 900 times. >> Maybe lots of different mount points are being accessed by something and >> each one triggers a few reads... >> >> Can you show your /etc/exports file? > > These get generated via puppet and dropped into /etc/exports.d/ ... > > Here we can see that there are 960 files in /etc/exports.d > root@storage1:~# find /etc/exports.d/ -type f | wc > 960 960 88701 Ahhh.. 960 exports. That could do it. $ grep -c crossmnt all_etc_exports.d_in_one_file.txt 957 I get 957 - much the same number. $ grep open redacted_strace.txt | uniq -c 2 open("/var/lib/nfs/etab", O_RDONLY) = 6 942 open("/etc/mtab", O_RDONLY|O_CLOEXEC) = 6 1 open("/proc/net/rpc/nfsd.export/channel", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 6 /etc/mtab was ready 942 for a single access. I would have expected 957, but maybe the system is dynamic and something changed between the two samples. This makes it fairly clear what is happening. Now we just need to fix it. One option would be to cache some of the details extracted from /etc/mtab, but that could get messy. Another is to move the code around. In your case there are really just 3 exports to each of 300+ clients (I assume "client.cs.uchicago.edu" in the combined exports file is really different in different files). So any one client only needs to consider 3 exports, not 300. There is room to optimize this code further than the below, but let's start simple. Could you test with this patch applied and see what difference it makes? Thanks, NeilBrown diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c index ca6c84f4d93d..e712cc166157 100644 --- a/utils/mountd/cache.c +++ b/utils/mountd/cache.c @@ -727,6 +727,17 @@ static void nfsd_fh(int f) for (exp = exportlist[i].p_head; exp; exp = next_exp) { char *path; + if (!is_ipaddr_client(dom) + && !namelist_client_matches(exp, dom)) { + next_exp = exp->m_next; + continue; + } + if (is_ipaddr_client(dom) + && !ipaddr_client_matches(exp, ai)) { + next_exp = exp->m_next; + continue; + } + if (exp->m_export.e_flags & NFSEXP_CROSSMOUNT) { static nfs_export *prev = NULL; static void *mnt = NULL; @@ -751,9 +762,6 @@ static void nfsd_fh(int f) next_exp = exp->m_next; } - if (!is_ipaddr_client(dom) - && !namelist_client_matches(exp, dom)) - continue; if (exp->m_export.e_mountpoint && !is_mountpoint(exp->m_export.e_mountpoint[0]? exp->m_export.e_mountpoint: @@ -762,9 +770,6 @@ static void nfsd_fh(int f) if (!match_fsid(&parsed, exp, path)) continue; - if (is_ipaddr_client(dom) - && !ipaddr_client_matches(exp, ai)) - continue; if (!found || subexport(&exp->m_export, found)) { found = &exp->m_export; free(found_path);