From patchwork Fri Jul 14 06:51:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 9840033 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0EDD7602BD for ; Fri, 14 Jul 2017 06:51:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F12E428750 for ; Fri, 14 Jul 2017 06:51:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E57AD28779; Fri, 14 Jul 2017 06:51:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1416E28750 for ; Fri, 14 Jul 2017 06:51:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751403AbdGNGvs (ORCPT ); Fri, 14 Jul 2017 02:51:48 -0400 Received: from mx2.suse.de ([195.135.220.15]:40512 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751166AbdGNGvr (ORCPT ); Fri, 14 Jul 2017 02:51:47 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id DA339AE61; Fri, 14 Jul 2017 06:51:45 +0000 (UTC) From: NeilBrown To: Phil Kauffman , linux-nfs@vger.kernel.org Date: Fri, 14 Jul 2017 16:51:36 +1000 Subject: Re: /etc/mtab read ~900 times by rpc.mountd In-Reply-To: <9e16d6c3-a675-b53e-c6f3-dfa9cdf1d5c9@cs.uchicago.edu> References: <8737a9x9ky.fsf@notabene.neil.brown.name> <595F1A3A.7070405@cs.uchicago.edu> <87efto69rs.fsf@notabene.neil.brown.name> <4ec2a8fc-3ca5-d26b-7742-be4e2f749c21@cs.uchicago.edu> <87y3rv4zrb.fsf@notabene.neil.brown.name> <1740081e-6180-1c88-0a0c-8747a92c65a1@cs.uchicago.edu> <87bmoq4h41.fsf@notabene.neil.brown.name> <9e16d6c3-a675-b53e-c6f3-dfa9cdf1d5c9@cs.uchicago.edu> Message-ID: <871spj1pfr.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, Jul 12 2017, Phil Kauffman wrote: > On 07/11/2017 07:46 PM, NeilBrown wrote: >> So the new data shows about 7 seconds for a login, which is probably >> a little longer than you would like, but might be acceptable? > Unfortunately, the delay is not acceptable. > > The ideal would be to achieve performance parity with when one is not > forced to use the 'crossmnt' option. > > My current setup (home directory server) does not require 'crossmnt' and > does not incur a delay. It is a standard nfs server using mdadm, lvm, > and xfs. > > While my current setup is probably "normal" and using nested datasets > with the 'crossmnt' option might be "weird" now; nested mounts will > probably only become more common as BTRFS, ZFS, and other filesystems > with similar features gain traction on linux. > > >> That is more than a 2-line patch. I might have a go later this week. > That would be super! Thanks for taking a look. Please try this (against a clean nfs-utils. i.e. remove the previous patch). It is a hack and would need to be totally re-written, but hopely the measurements you make and strace that you report could be useful. Also, for the strace, please use "-ttt" rather than "-tt" like I asked before. It is easier to find the difference between two times with -ttt. And add -T as well. Thanks, NeilBrown From f15571d747bb6272ec539e46b6b3aee3ccc53b30 Mon Sep 17 00:00:00 2001 From: NeilBrown Date: Fri, 14 Jul 2017 16:43:37 +1000 Subject: [PATCH] testing Signed-off-by: NeilBrown --- utils/mountd/cache.c | 105 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 81 insertions(+), 24 deletions(-) diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c index ca6c84f4d93d..db89a4feb7ae 100644 --- a/utils/mountd/cache.c +++ b/utils/mountd/cache.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -299,7 +300,7 @@ static const long int nonblkid_filesystems[] = { 0 /* last */ }; -static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid) +static int uuid_by_path(char *path, struct statfs64 *stp, int type, size_t uuidlen, char *uuid) { /* get a uuid for the filesystem found at 'path'. * There are several possible ways of generating the @@ -334,12 +335,17 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid) const char *val; int rc; - rc = statfs64(path, &st); + if (stp) + rc = 0; + else { + stp = &st; + rc= statfs64(path, stp); + } if (type == 0 && rc == 0) { const long int *bad; for (bad = nonblkid_filesystems; *bad; bad++) { - if (*bad == st.f_type) + if (*bad == stp->f_type) break; } if (*bad == 0) @@ -347,9 +353,9 @@ static int uuid_by_path(char *path, int type, size_t uuidlen, char *uuid) } if (rc == 0 && - (st.f_fsid.__val[0] || st.f_fsid.__val[1])) + (stp->f_fsid.__val[0] || stp->f_fsid.__val[1])) snprintf(fsid_val, 17, "%08x%08x", - st.f_fsid.__val[0], st.f_fsid.__val[1]); + stp->f_fsid.__val[0], stp->f_fsid.__val[1]); else fsid_val[0] = 0; @@ -603,25 +609,64 @@ static int parse_fsid(int fsidtype, int fsidlen, char *fsid, return 0; } +#define entry FOO +#include +struct pinfo { + unsigned int stb:1, mntp:1, stfs:1; + unsigned int mountpoint:1; + time_t valid; + struct stat statbuf; + struct statfs64 statfsbuf; +}; +static struct hsearch_data hdata; +static int hdata_init = 0; + + static bool match_fsid(struct parsed_fsid *parsed, nfs_export *exp, char *path) { - struct stat stb; + struct stat *stb; int type; char u[16]; + struct pinfo *pi; + ENTRY ent, *ret; - if (stat(path, &stb) != 0) - return false; - if (!S_ISDIR(stb.st_mode) && !S_ISREG(stb.st_mode)) + if (!hdata_init) { + hcreate_r(1000, &hdata); + hdata_init = 1; + } + ent.key = path; + if (hsearch_r(ent, FIND, &ret, &hdata) == 0) { + ent.key = strdup(path); + pi = xmalloc(sizeof(*pi)); + pi->stb = pi->mntp = pi->stfs = 0; + pi->valid = 0; + ent.data = pi; + hsearch_r(ent, ENTER, &ret, &hdata); + } else + pi = ret->data; + + if (pi->valid < time(NULL)+120) { + pi->stb = pi->mntp = pi->stfs = 0; + pi->valid = time(NULL); + } + + stb = &pi->statbuf; + if (!pi->stb) { + if (stat(path, stb) != 0) + return false; + pi->stb = 1; + } + if (!S_ISDIR(stb->st_mode) && !S_ISREG(stb->st_mode)) return false; switch (parsed->fsidtype) { case FSID_DEV: case FSID_MAJOR_MINOR: case FSID_ENCODE_DEV: - if (stb.st_ino != parsed->inode) + if (stb->st_ino != parsed->inode) return false; - if (parsed->major != major(stb.st_dev) || - parsed->minor != minor(stb.st_dev)) + if (parsed->major != major(stb->st_dev) || + parsed->minor != minor(stb->st_dev)) return false; return true; case FSID_NUM: @@ -631,12 +676,16 @@ static bool match_fsid(struct parsed_fsid *parsed, nfs_export *exp, char *path) return true; case FSID_UUID4_INUM: case FSID_UUID16_INUM: - if (stb.st_ino != parsed->inode) + if (stb->st_ino != parsed->inode) return false; goto check_uuid; case FSID_UUID8: case FSID_UUID16: - if (!is_mountpoint(path)) + if (!pi->mntp) { + pi->mountpoint = is_mountpoint(path); + pi->mntp = 1; + } + if (!pi->mountpoint) return false; check_uuid: if (exp->m_export.e_uuid) { @@ -644,12 +693,18 @@ static bool match_fsid(struct parsed_fsid *parsed, nfs_export *exp, char *path) if (memcmp(u, parsed->fhuuid, parsed->uuidlen) == 0) return true; } - else + else { + if (!pi->stfs) { + if (statfs64(path, &pi->statfsbuf) != 0) + return false; + pi->stfs = 1; + } for (type = 0; - uuid_by_path(path, type, parsed->uuidlen, u); + uuid_by_path(path, &pi->statfsbuf, type, parsed->uuidlen, u); type++) if (memcmp(u, parsed->fhuuid, parsed->uuidlen) == 0) return true; + } } /* Well, unreachable, actually: */ return false; @@ -727,10 +782,18 @@ static void nfsd_fh(int f) for (exp = exportlist[i].p_head; exp; exp = next_exp) { char *path; + if (!is_ipaddr_client(dom)) { + if (!namelist_client_matches(exp, dom)) + continue; + } else { + if (!ipaddr_client_matches(exp, ai)) + continue; + } + if (exp->m_export.e_flags & NFSEXP_CROSSMOUNT) { static nfs_export *prev = NULL; static void *mnt = NULL; - + if (prev == exp) { /* try a submount */ path = next_mnt(&mnt, exp->m_export.e_path); @@ -751,9 +814,6 @@ static void nfsd_fh(int f) next_exp = exp->m_next; } - if (!is_ipaddr_client(dom) - && !namelist_client_matches(exp, dom)) - continue; if (exp->m_export.e_mountpoint && !is_mountpoint(exp->m_export.e_mountpoint[0]? exp->m_export.e_mountpoint: @@ -762,9 +822,6 @@ static void nfsd_fh(int f) if (!match_fsid(&parsed, exp, path)) continue; - if (is_ipaddr_client(dom) - && !ipaddr_client_matches(exp, ai)) - continue; if (!found || subexport(&exp->m_export, found)) { found = &exp->m_export; free(found_path); @@ -906,7 +963,7 @@ static int dump_to_cache(int f, char *buf, int buflen, char *domain, write_secinfo(&bp, &blen, exp, flag_mask); if (exp->e_uuid == NULL || different_fs) { char u[16]; - if (uuid_by_path(path, 0, 16, u)) { + if (uuid_by_path(path, NULL, 0, 16, u)) { qword_add(&bp, &blen, "uuid"); qword_addhex(&bp, &blen, u, 16); }