From patchwork Wed Feb 19 03:38:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11390287 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 23B1F159A for ; Wed, 19 Feb 2020 03:40:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EFB13222D9 for ; Wed, 19 Feb 2020 03:40:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="d7VOsKdv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726556AbgBSDkF (ORCPT ); Tue, 18 Feb 2020 22:40:05 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:53420 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726403AbgBSDkF (ORCPT ); Tue, 18 Feb 2020 22:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582083603; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h99A+kyn3626Zby0qGOLCcHNxS9Zim//VIxkE/lqlCk=; b=d7VOsKdvL0PvWQpO2NG+Oa2g6K7hSE9eX/jtZL3SXtAsrTgtkWu5fORw+BPUJxHAET0FT3 RmL61P47W6GWUeyg8FRvckGqwLwj4rug03GZ93/B2WGgG8b4tpgmR3TFGHM1tkz2YOrMzJ OdqNEXwV5NHMlYGOEdcuPAnULvR/bFI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-305-LZUeNhQ3PLOGr3mYw4KsjA-1; Tue, 18 Feb 2020 22:39:56 -0500 X-MC-Unique: LZUeNhQ3PLOGr3mYw4KsjA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B4EAF801E5C; Wed, 19 Feb 2020 03:39:55 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-94.pek2.redhat.com [10.72.12.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id D39E560C81; Wed, 19 Feb 2020 03:39:52 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH v7 1/5] ceph: add global dentry lease metric support Date: Tue, 18 Feb 2020 22:38:47 -0500 Message-Id: <20200219033851.6548-2-xiubli@redhat.com> In-Reply-To: <20200219033851.6548-1-xiubli@redhat.com> References: <20200219033851.6548-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li For the dentry lease we will only count the hit/miss info triggered from the vfs calls, for the cases like request reply handling and perodically ceph_trim_dentries() we will ignore them. Currently only the debugfs is support: The output will be: item total miss hit ------------------------------------------------- d_lease 11 7 141 URL: https://tracker.ceph.com/issues/43215 Signed-off-by: Xiubo Li --- fs/ceph/debugfs.c | 32 ++++++++++++++++++++++++++++---- fs/ceph/dir.c | 16 ++++++++++++++-- fs/ceph/mds_client.c | 37 +++++++++++++++++++++++++++++++++++-- fs/ceph/mds_client.h | 4 ++++ fs/ceph/metric.h | 11 +++++++++++ fs/ceph/super.h | 1 + 6 files changed, 93 insertions(+), 8 deletions(-) create mode 100644 fs/ceph/metric.h diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index 481ac97b4d25..15975ba95d9a 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -124,6 +124,22 @@ static int mdsc_show(struct seq_file *s, void *p) return 0; } +static int metric_show(struct seq_file *s, void *p) +{ + struct ceph_fs_client *fsc = s->private; + struct ceph_mds_client *mdsc = fsc->mdsc; + + seq_printf(s, "item total miss hit\n"); + seq_printf(s, "-------------------------------------------------\n"); + + seq_printf(s, "%-14s%-16lld%-16lld%lld\n", "d_lease", + atomic64_read(&mdsc->metric.total_dentries), + percpu_counter_sum(&mdsc->metric.d_lease_mis), + percpu_counter_sum(&mdsc->metric.d_lease_hit)); + + return 0; +} + static int caps_show_cb(struct inode *inode, struct ceph_cap *cap, void *p) { struct seq_file *s = p; @@ -222,6 +238,7 @@ DEFINE_SHOW_ATTRIBUTE(mdsmap); DEFINE_SHOW_ATTRIBUTE(mdsc); DEFINE_SHOW_ATTRIBUTE(caps); DEFINE_SHOW_ATTRIBUTE(mds_sessions); +DEFINE_SHOW_ATTRIBUTE(metric); /* @@ -255,6 +272,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) debugfs_remove(fsc->debugfs_mdsmap); debugfs_remove(fsc->debugfs_mds_sessions); debugfs_remove(fsc->debugfs_caps); + debugfs_remove(fsc->debugfs_metric); debugfs_remove(fsc->debugfs_mdsc); } @@ -295,11 +313,17 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc) fsc, &mdsc_fops); + fsc->debugfs_metric = debugfs_create_file("metrics", + 0400, + fsc->client->debugfs_dir, + fsc, + &metric_fops); + fsc->debugfs_caps = debugfs_create_file("caps", - 0400, - fsc->client->debugfs_dir, - fsc, - &caps_fops); + 0400, + fsc->client->debugfs_dir, + fsc, + &caps_fops); } diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index 80bd3be4715a..f2a477fdfffb 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -38,6 +38,8 @@ static int __dir_lease_try_check(const struct dentry *dentry); static int ceph_d_init(struct dentry *dentry) { struct ceph_dentry_info *di; + struct ceph_fs_client *fsc = ceph_sb_to_client(dentry->d_sb); + struct ceph_mds_client *mdsc = fsc->mdsc; di = kmem_cache_zalloc(ceph_dentry_cachep, GFP_KERNEL); if (!di) @@ -48,6 +50,9 @@ static int ceph_d_init(struct dentry *dentry) di->time = jiffies; dentry->d_fsdata = di; INIT_LIST_HEAD(&di->lease_list); + + atomic64_inc(&mdsc->metric.total_dentries); + return 0; } @@ -1551,6 +1556,7 @@ static int dir_lease_is_valid(struct inode *dir, struct dentry *dentry) */ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) { + struct ceph_mds_client *mdsc; int valid = 0; struct dentry *parent; struct inode *dir, *inode; @@ -1589,9 +1595,8 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) } } + mdsc = ceph_sb_to_client(dir->i_sb)->mdsc; if (!valid) { - struct ceph_mds_client *mdsc = - ceph_sb_to_client(dir->i_sb)->mdsc; struct ceph_mds_request *req; int op, err; u32 mask; @@ -1599,6 +1604,8 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) if (flags & LOOKUP_RCU) return -ECHILD; + percpu_counter_inc(&mdsc->metric.d_lease_mis); + op = ceph_snap(dir) == CEPH_SNAPDIR ? CEPH_MDS_OP_LOOKUPSNAP : CEPH_MDS_OP_LOOKUP; req = ceph_mdsc_create_request(mdsc, op, USE_ANY_MDS); @@ -1630,6 +1637,8 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) dout("d_revalidate %p lookup result=%d\n", dentry, err); } + } else { + percpu_counter_inc(&mdsc->metric.d_lease_hit); } dout("d_revalidate %p %s\n", dentry, valid ? "valid" : "invalid"); @@ -1672,9 +1681,12 @@ static int ceph_d_delete(const struct dentry *dentry) static void ceph_d_release(struct dentry *dentry) { struct ceph_dentry_info *di = ceph_dentry(dentry); + struct ceph_fs_client *fsc = ceph_sb_to_client(dentry->d_sb); dout("d_release %p\n", dentry); + atomic64_dec(&fsc->mdsc->metric.total_dentries); + spin_lock(&dentry->d_lock); __dentry_lease_unlist(di); dentry->d_fsdata = NULL; diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 46fd5c77a6e4..511b6c0a738d 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4160,10 +4160,31 @@ static void delayed_work(struct work_struct *work) schedule_delayed(mdsc); } +static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) +{ + int ret; + + if (!metric) + return -EINVAL; + + atomic64_set(&metric->total_dentries, 0); + ret = percpu_counter_init(&metric->d_lease_hit, 0, GFP_KERNEL); + if (ret) + return ret; + ret = percpu_counter_init(&metric->d_lease_mis, 0, GFP_KERNEL); + if (ret) { + percpu_counter_destroy(&metric->d_lease_hit); + return ret; + } + + return 0; +} + int ceph_mdsc_init(struct ceph_fs_client *fsc) { struct ceph_mds_client *mdsc; + int err; mdsc = kzalloc(sizeof(struct ceph_mds_client), GFP_NOFS); if (!mdsc) @@ -4172,8 +4193,8 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) mutex_init(&mdsc->mutex); mdsc->mdsmap = kzalloc(sizeof(*mdsc->mdsmap), GFP_NOFS); if (!mdsc->mdsmap) { - kfree(mdsc); - return -ENOMEM; + err = -ENOMEM; + goto err_mdsc; } fsc->mdsc = mdsc; @@ -4212,6 +4233,9 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) init_waitqueue_head(&mdsc->cap_flushing_wq); INIT_WORK(&mdsc->cap_reclaim_work, ceph_cap_reclaim_work); atomic_set(&mdsc->cap_reclaim_pending, 0); + err = ceph_mdsc_metric_init(&mdsc->metric); + if (err) + goto err_mdsmap; spin_lock_init(&mdsc->dentry_list_lock); INIT_LIST_HEAD(&mdsc->dentry_leases); @@ -4230,6 +4254,12 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) strscpy(mdsc->nodename, utsname()->nodename, sizeof(mdsc->nodename)); return 0; + +err_mdsmap: + kfree(mdsc->mdsmap); +err_mdsc: + kfree(mdsc); + return err; } /* @@ -4487,6 +4517,9 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.d_lease_mis); + percpu_counter_destroy(&mdsc->metric.d_lease_hit); + fsc->mdsc = NULL; kfree(mdsc); dout("mdsc_destroy %p done\n", mdsc); diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 27a7446e10d3..674fc7725913 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -16,6 +16,8 @@ #include #include +#include "metric.h" + /* The first 8 bits are reserved for old ceph releases */ enum ceph_feature_type { CEPHFS_FEATURE_MIMIC = 8, @@ -446,6 +448,8 @@ struct ceph_mds_client { struct list_head dentry_leases; /* fifo list */ struct list_head dentry_dir_leases; /* lru list */ + struct ceph_client_metric metric; + spinlock_t snapid_map_lock; struct rb_root snapid_map_tree; struct list_head snapid_map_lru; diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h new file mode 100644 index 000000000000..998fe2a643cf --- /dev/null +++ b/fs/ceph/metric.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _FS_CEPH_MDS_METRIC_H +#define _FS_CEPH_MDS_METRIC_H + +/* This is the global metrics */ +struct ceph_client_metric { + atomic64_t total_dentries; + struct percpu_counter d_lease_hit; + struct percpu_counter d_lease_mis; +}; +#endif /* _FS_CEPH_MDS_METRIC_H */ diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 37dc1ac8f6c3..ebcf7612eac9 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -125,6 +125,7 @@ struct ceph_fs_client { struct dentry *debugfs_congestion_kb; struct dentry *debugfs_bdi; struct dentry *debugfs_mdsc, *debugfs_mdsmap; + struct dentry *debugfs_metric; struct dentry *debugfs_mds_sessions; #endif From patchwork Wed Feb 19 03:38:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11390289 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D947138D for ; Wed, 19 Feb 2020 03:40:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D73A2222D9 for ; Wed, 19 Feb 2020 03:40:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HG9wzHij" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726515AbgBSDkF (ORCPT ); Tue, 18 Feb 2020 22:40:05 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:28119 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726508AbgBSDkF (ORCPT ); Tue, 18 Feb 2020 22:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582083603; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pea00b3UElR/NzQj0SuI+kCNwz6eHFDPcLwUWlMtz0s=; b=HG9wzHijqpi26nyuH7aw7qrKdUA6YyNvAvkJW1LcpKp+VFF3r/QJyHBWR5YB9veLyOJiks j0WpQEzyxnIZLc11FBGvZB5PqETcqZi32gOaVB/FWwI7+j/oQzUwnoanCnleKZNlXaI+qg Y9PmYpaH8/qjRpDwsVg2i7aaiLlcyYY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-378-jFZtMBSjPb-afQ-LC7g59Q-1; Tue, 18 Feb 2020 22:40:01 -0500 X-MC-Unique: jFZtMBSjPb-afQ-LC7g59Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5ABCD1857346; Wed, 19 Feb 2020 03:40:00 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-94.pek2.redhat.com [10.72.12.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id 561CC60C81; Wed, 19 Feb 2020 03:39:56 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH v7 2/5] ceph: add caps perf metric for each session Date: Tue, 18 Feb 2020 22:38:48 -0500 Message-Id: <20200219033851.6548-3-xiubli@redhat.com> In-Reply-To: <20200219033851.6548-1-xiubli@redhat.com> References: <20200219033851.6548-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li This will fulfill the cap hit/mis metric stuff per-superblock, it will count the hit/mis counters based each inode, and if one inode's 'issued & ~revoking == mask' will mean a hit, or a miss. item total miss hit ------------------------------------------------- caps 295 107 4119 URL: https://tracker.ceph.com/issues/43215 Signed-off-by: Xiubo Li --- fs/ceph/acl.c | 2 ++ fs/ceph/caps.c | 31 +++++++++++++++++++++++++++++++ fs/ceph/debugfs.c | 16 ++++++++++++++++ fs/ceph/dir.c | 9 +++++++-- fs/ceph/file.c | 2 ++ fs/ceph/mds_client.c | 26 ++++++++++++++++++++++---- fs/ceph/metric.h | 3 +++ fs/ceph/quota.c | 9 +++++++-- fs/ceph/super.h | 9 +++++++++ fs/ceph/xattr.c | 17 ++++++++++++++--- 10 files changed, 113 insertions(+), 11 deletions(-) diff --git a/fs/ceph/acl.c b/fs/ceph/acl.c index 26be6520d3fb..58e119e3519f 100644 --- a/fs/ceph/acl.c +++ b/fs/ceph/acl.c @@ -22,6 +22,8 @@ static inline void ceph_set_cached_acl(struct inode *inode, struct ceph_inode_info *ci = ceph_inode(inode); spin_lock(&ci->i_ceph_lock); + __ceph_caps_metric(ci, CEPH_CAP_XATTR_SHARED); + if (__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 0)) set_cached_acl(inode, type, acl); else diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c index d05717397c2a..bf7d96125e3a 100644 --- a/fs/ceph/caps.c +++ b/fs/ceph/caps.c @@ -818,6 +818,32 @@ int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented) return have; } +/* + * Counts the cap metric. + * + * This will try to traverse all the ci->i_caps, if we can + * get all the cap 'mask' it will count the hit, or the mis. + */ +void __ceph_caps_metric(struct ceph_inode_info *ci, int mask) +{ + struct ceph_mds_client *mdsc = + ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc; + struct ceph_client_metric *metric = &mdsc->metric; + int issued; + + lockdep_assert_held(&ci->i_ceph_lock); + + if (mask <= 0) + return; + + issued = __ceph_caps_issued(ci, NULL); + + if ((mask & issued) == mask) + percpu_counter_inc(&metric->i_caps_hit); + else + percpu_counter_inc(&metric->i_caps_mis); +} + /* * Get cap bits issued by caps other than @ocap */ @@ -2744,8 +2770,11 @@ int ceph_try_get_caps(struct inode *inode, int need, int want, if (ret < 0) return ret; + ceph_caps_metric(ceph_inode(inode), need | want); + ret = try_get_cap_refs(inode, need, want, 0, (nonblock ? NON_BLOCKING : 0), got); + return ret == -EAGAIN ? 0 : ret; } @@ -2771,6 +2800,8 @@ int ceph_get_caps(struct file *filp, int need, int want, fi->filp_gen != READ_ONCE(fsc->filp_gen)) return -EBADF; + ceph_caps_metric(ci, need | want); + while (true) { if (endoff > 0) check_max_size(inode, endoff); diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index 15975ba95d9a..c83e52bd9961 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -128,6 +128,7 @@ static int metric_show(struct seq_file *s, void *p) { struct ceph_fs_client *fsc = s->private; struct ceph_mds_client *mdsc = fsc->mdsc; + int i, nr_caps = 0; seq_printf(s, "item total miss hit\n"); seq_printf(s, "-------------------------------------------------\n"); @@ -137,6 +138,21 @@ static int metric_show(struct seq_file *s, void *p) percpu_counter_sum(&mdsc->metric.d_lease_mis), percpu_counter_sum(&mdsc->metric.d_lease_hit)); + mutex_lock(&mdsc->mutex); + for (i = 0; i < mdsc->max_sessions; i++) { + struct ceph_mds_session *s; + + s = __ceph_lookup_mds_session(mdsc, i); + if (!s) + continue; + nr_caps += s->s_nr_caps; + ceph_put_mds_session(s); + } + mutex_unlock(&mdsc->mutex); + seq_printf(s, "%-14s%-16d%-16lld%lld\n", "caps", nr_caps, + percpu_counter_sum(&mdsc->metric.i_caps_mis), + percpu_counter_sum(&mdsc->metric.i_caps_hit)); + return 0; } diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index f2a477fdfffb..f269b929836d 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -313,7 +313,7 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) struct ceph_fs_client *fsc = ceph_inode_to_client(inode); struct ceph_mds_client *mdsc = fsc->mdsc; int i; - int err; + int err, ret = -1; unsigned frag = -1; struct ceph_mds_reply_info_parsed *rinfo; @@ -346,13 +346,16 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) !ceph_test_mount_opt(fsc, NOASYNCREADDIR) && ceph_snap(inode) != CEPH_SNAPDIR && __ceph_dir_is_complete_ordered(ci) && - __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1)) { + (ret = __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1))) { int shared_gen = atomic_read(&ci->i_shared_gen); + __ceph_caps_metric(ci, CEPH_CAP_FILE_SHARED); spin_unlock(&ci->i_ceph_lock); err = __dcache_readdir(file, ctx, shared_gen); if (err != -EAGAIN) return err; } else { + if (ret != -1) + __ceph_caps_metric(ci, CEPH_CAP_FILE_SHARED); spin_unlock(&ci->i_ceph_lock); } @@ -757,6 +760,8 @@ static struct dentry *ceph_lookup(struct inode *dir, struct dentry *dentry, struct ceph_dentry_info *di = ceph_dentry(dentry); spin_lock(&ci->i_ceph_lock); + __ceph_caps_metric(ci, CEPH_CAP_FILE_SHARED); + dout(" dir %p flags are %d\n", dir, ci->i_ceph_flags); if (strncmp(dentry->d_name.name, fsc->mount_options->snapdir_name, diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 7e0190b1f821..b1b5aa35d25f 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -384,6 +384,8 @@ int ceph_open(struct inode *inode, struct file *file) * asynchronously. */ spin_lock(&ci->i_ceph_lock); + __ceph_caps_metric(ci, wanted); + if (__ceph_is_any_real_caps(ci) && (((fmode & CEPH_FILE_MODE_WR) == 0) || ci->i_auth_cap)) { int mds_wanted = __ceph_caps_mds_wanted(ci, true); diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 511b6c0a738d..4993ccaceefe 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4171,13 +4171,29 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) ret = percpu_counter_init(&metric->d_lease_hit, 0, GFP_KERNEL); if (ret) return ret; + ret = percpu_counter_init(&metric->d_lease_mis, 0, GFP_KERNEL); - if (ret) { - percpu_counter_destroy(&metric->d_lease_hit); - return ret; - } + if (ret) + goto err_d_lease_mis; + + ret = percpu_counter_init(&metric->i_caps_hit, 0, GFP_KERNEL); + if (ret) + goto err_i_caps_hit; + + ret = percpu_counter_init(&metric->i_caps_mis, 0, GFP_KERNEL); + if (ret) + goto err_i_caps_mis; return 0; + +err_i_caps_mis: + percpu_counter_destroy(&metric->i_caps_hit); +err_i_caps_hit: + percpu_counter_destroy(&metric->d_lease_mis); +err_d_lease_mis: + percpu_counter_destroy(&metric->d_lease_hit); + + return ret; } int ceph_mdsc_init(struct ceph_fs_client *fsc) @@ -4517,6 +4533,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.i_caps_mis); + percpu_counter_destroy(&mdsc->metric.i_caps_hit); percpu_counter_destroy(&mdsc->metric.d_lease_mis); percpu_counter_destroy(&mdsc->metric.d_lease_hit); diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h index 998fe2a643cf..e2fceb38a924 100644 --- a/fs/ceph/metric.h +++ b/fs/ceph/metric.h @@ -7,5 +7,8 @@ struct ceph_client_metric { atomic64_t total_dentries; struct percpu_counter d_lease_hit; struct percpu_counter d_lease_mis; + + struct percpu_counter i_caps_hit; + struct percpu_counter i_caps_mis; }; #endif /* _FS_CEPH_MDS_METRIC_H */ diff --git a/fs/ceph/quota.c b/fs/ceph/quota.c index de56dee60540..4ce2f658e63d 100644 --- a/fs/ceph/quota.c +++ b/fs/ceph/quota.c @@ -147,9 +147,14 @@ static struct inode *lookup_quotarealm_inode(struct ceph_mds_client *mdsc, return NULL; } if (qri->inode) { + struct ceph_inode_info *ci = ceph_inode(qri->inode); + int ret; + + ceph_caps_metric(ci, CEPH_STAT_CAP_INODE); + /* get caps */ - int ret = __ceph_do_getattr(qri->inode, NULL, - CEPH_STAT_CAP_INODE, true); + ret = __ceph_do_getattr(qri->inode, NULL, + CEPH_STAT_CAP_INODE, true); if (ret >= 0) in = qri->inode; else diff --git a/fs/ceph/super.h b/fs/ceph/super.h index ebcf7612eac9..67e2952965a8 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -637,6 +637,14 @@ static inline bool __ceph_is_any_real_caps(struct ceph_inode_info *ci) return !RB_EMPTY_ROOT(&ci->i_caps); } +extern void __ceph_caps_metric(struct ceph_inode_info *ci, int mask); +static inline void ceph_caps_metric(struct ceph_inode_info *ci, int mask) +{ + spin_lock(&ci->i_ceph_lock); + __ceph_caps_metric(ci, mask); + spin_unlock(&ci->i_ceph_lock); +} + extern int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented); extern int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int t); extern int __ceph_caps_issued_other(struct ceph_inode_info *ci, @@ -923,6 +931,7 @@ extern int __ceph_do_getattr(struct inode *inode, struct page *locked_page, int mask, bool force); static inline int ceph_do_getattr(struct inode *inode, int mask, bool force) { + ceph_caps_metric(ceph_inode(inode), mask); return __ceph_do_getattr(inode, NULL, mask, force); } extern int ceph_permission(struct inode *inode, int mask); diff --git a/fs/ceph/xattr.c b/fs/ceph/xattr.c index 7b8a070a782d..9b28e87b6719 100644 --- a/fs/ceph/xattr.c +++ b/fs/ceph/xattr.c @@ -829,6 +829,7 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value, struct ceph_vxattr *vxattr = NULL; int req_mask; ssize_t err; + int ret = -1; /* let's see if a virtual xattr was requested */ vxattr = ceph_match_vxattr(inode, name); @@ -856,7 +857,9 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value, if (ci->i_xattrs.version == 0 || !((req_mask & CEPH_CAP_XATTR_SHARED) || - __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1))) { + (ret = __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1)))) { + if (ret != -1) + __ceph_caps_metric(ci, CEPH_CAP_XATTR_SHARED); spin_unlock(&ci->i_ceph_lock); /* security module gets xattr while filling trace */ @@ -871,6 +874,9 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value, if (err) return err; spin_lock(&ci->i_ceph_lock); + } else { + if (ret != -1) + __ceph_caps_metric(ci, CEPH_CAP_XATTR_SHARED); } err = __build_xattrs(inode); @@ -907,19 +913,24 @@ ssize_t ceph_listxattr(struct dentry *dentry, char *names, size_t size) struct ceph_inode_info *ci = ceph_inode(inode); bool len_only = (size == 0); u32 namelen; - int err; + int err, ret = -1; spin_lock(&ci->i_ceph_lock); dout("listxattr %p ver=%lld index_ver=%lld\n", inode, ci->i_xattrs.version, ci->i_xattrs.index_version); if (ci->i_xattrs.version == 0 || - !__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1)) { + !(ret = __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1))) { + if (ret != -1) + __ceph_caps_metric(ci, CEPH_CAP_XATTR_SHARED); spin_unlock(&ci->i_ceph_lock); err = ceph_do_getattr(inode, CEPH_STAT_CAP_XATTR, true); if (err) return err; spin_lock(&ci->i_ceph_lock); + } else { + if (ret != -1) + __ceph_caps_metric(ci, CEPH_CAP_XATTR_SHARED); } err = __build_xattrs(inode); From patchwork Wed Feb 19 03:38:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11390291 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 561BA138D for ; Wed, 19 Feb 2020 03:40:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2E534222D9 for ; Wed, 19 Feb 2020 03:40:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L/Gc0MeQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726569AbgBSDkY (ORCPT ); Tue, 18 Feb 2020 22:40:24 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:24831 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726403AbgBSDkY (ORCPT ); Tue, 18 Feb 2020 22:40:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582083622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cFimMWpKRx2Wd1BRxTKZwehTgeOsohdPlflif0gme2o=; b=L/Gc0MeQymQCU9bfXmTNICTgCJfWvUwtWtvjJRUCzETp6FAz7EalCA1mREPMtxZhlPhRZT ad3taj77kjRQFb43bjozJM9ZlaNZJrfmgKWEOhVF0GMhjy6+8KMAO0/USuORjzCmgHuaMR zkisEKBVD/4gHeLMgLGHo9f8MmptkDY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-436-L0N35ulrNQ2PqpC2akV-cQ-1; Tue, 18 Feb 2020 22:40:15 -0500 X-MC-Unique: L0N35ulrNQ2PqpC2akV-cQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 896FA800D48; Wed, 19 Feb 2020 03:40:14 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-94.pek2.redhat.com [10.72.12.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3C2360C81; Wed, 19 Feb 2020 03:40:00 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH v7 3/5] ceph: add global read latency metric support Date: Tue, 18 Feb 2020 22:38:49 -0500 Message-Id: <20200219033851.6548-4-xiubli@redhat.com> In-Reply-To: <20200219033851.6548-1-xiubli@redhat.com> References: <20200219033851.6548-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li It will calculate the latency for the read osd requests, which only include the time cousumed by network and the ceph osd. item total sum_lat(us) avg_lat(us) ----------------------------------------------------- read 1036 848000 818 URL: https://tracker.ceph.com/issues/43215 Signed-off-by: Xiubo Li --- fs/ceph/addr.c | 6 ++++++ fs/ceph/debugfs.c | 11 +++++++++++ fs/ceph/file.c | 13 +++++++++++++ fs/ceph/mds_client.c | 14 ++++++++++++++ fs/ceph/metric.h | 20 ++++++++++++++++++++ include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 2 ++ 7 files changed, 67 insertions(+) diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 6f4678d98df7..16573a13ffee 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -216,6 +216,8 @@ static int ceph_sync_readpages(struct ceph_fs_client *fsc, if (!rc) rc = ceph_osdc_wait_request(osdc, req); + ceph_update_read_latency(&fsc->mdsc->metric, req, rc); + ceph_osdc_put_request(req); dout("readpages result %d\n", rc); return rc; @@ -299,6 +301,7 @@ static int ceph_readpage(struct file *filp, struct page *page) static void finish_read(struct ceph_osd_request *req) { struct inode *inode = req->r_inode; + struct ceph_fs_client *fsc = ceph_inode_to_client(inode); struct ceph_osd_data *osd_data; int rc = req->r_result <= 0 ? req->r_result : 0; int bytes = req->r_result >= 0 ? req->r_result : 0; @@ -336,6 +339,9 @@ static void finish_read(struct ceph_osd_request *req) put_page(page); bytes -= PAGE_SIZE; } + + ceph_update_read_latency(&fsc->mdsc->metric, req, rc); + kfree(osd_data->pages); } diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index c83e52bd9961..d814a3a27611 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -129,7 +129,18 @@ static int metric_show(struct seq_file *s, void *p) struct ceph_fs_client *fsc = s->private; struct ceph_mds_client *mdsc = fsc->mdsc; int i, nr_caps = 0; + s64 total, sum, avg = 0; + seq_printf(s, "item total sum_lat(us) avg_lat(us)\n"); + seq_printf(s, "-----------------------------------------------------\n"); + + total = percpu_counter_sum(&mdsc->metric.total_reads); + sum = percpu_counter_sum(&mdsc->metric.read_latency_sum); + sum = jiffies_to_usecs(sum); + avg = total ? sum / total : 0; + seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "read", total, sum, avg); + + seq_printf(s, "\n"); seq_printf(s, "item total miss hit\n"); seq_printf(s, "-------------------------------------------------\n"); diff --git a/fs/ceph/file.c b/fs/ceph/file.c index b1b5aa35d25f..96e35935b764 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -660,6 +660,9 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, ret = ceph_osdc_start_request(osdc, req, false); if (!ret) ret = ceph_osdc_wait_request(osdc, req); + + ceph_update_read_latency(&fsc->mdsc->metric, req, ret); + ceph_osdc_put_request(req); i_size = i_size_read(inode); @@ -798,6 +801,8 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) struct inode *inode = req->r_inode; struct ceph_aio_request *aio_req = req->r_priv; struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0); + struct ceph_fs_client *fsc = ceph_inode_to_client(inode); + struct ceph_client_metric *metric = &fsc->mdsc->metric; BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS); BUG_ON(!osd_data->num_bvecs); @@ -805,6 +810,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, osd_data->bvec_pos.iter.bi_size); + /* r_start_stamp == 0 means the request was not submitted */ + if (req->r_start_stamp && !aio_req->write) + ceph_update_read_latency(metric, req, rc); + if (rc == -EOLDSNAPC) { struct ceph_aio_work *aio_work; BUG_ON(!aio_req->write); @@ -933,6 +942,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, struct inode *inode = file_inode(file); struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_fs_client *fsc = ceph_inode_to_client(inode); + struct ceph_client_metric *metric = &fsc->mdsc->metric; struct ceph_vino vino; struct ceph_osd_request *req; struct bio_vec *bvecs; @@ -1049,6 +1059,9 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, if (!ret) ret = ceph_osdc_wait_request(&fsc->client->osdc, req); + if (!write) + ceph_update_read_latency(metric, req, ret); + size = i_size_read(inode); if (!write) { if (ret == -ENOENT) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 4993ccaceefe..b7ada9cde4f8 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4184,8 +4184,20 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) if (ret) goto err_i_caps_mis; + ret = percpu_counter_init(&metric->total_reads, 0, GFP_KERNEL); + if (ret) + goto err_total_reads; + + ret = percpu_counter_init(&metric->read_latency_sum, 0, GFP_KERNEL); + if (ret) + goto err_read_latency_sum; + return 0; +err_read_latency_sum: + percpu_counter_destroy(&metric->total_reads); +err_total_reads: + percpu_counter_destroy(&metric->i_caps_mis); err_i_caps_mis: percpu_counter_destroy(&metric->i_caps_hit); err_i_caps_hit: @@ -4533,6 +4545,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.read_latency_sum); + percpu_counter_destroy(&mdsc->metric.total_reads); percpu_counter_destroy(&mdsc->metric.i_caps_mis); percpu_counter_destroy(&mdsc->metric.i_caps_hit); percpu_counter_destroy(&mdsc->metric.d_lease_mis); diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h index e2fceb38a924..afea44a3794b 100644 --- a/fs/ceph/metric.h +++ b/fs/ceph/metric.h @@ -2,6 +2,8 @@ #ifndef _FS_CEPH_MDS_METRIC_H #define _FS_CEPH_MDS_METRIC_H +#include + /* This is the global metrics */ struct ceph_client_metric { atomic64_t total_dentries; @@ -10,5 +12,23 @@ struct ceph_client_metric { struct percpu_counter i_caps_hit; struct percpu_counter i_caps_mis; + + struct percpu_counter total_reads; + struct percpu_counter read_latency_sum; }; + +static inline void ceph_update_read_latency(struct ceph_client_metric *m, + struct ceph_osd_request *req, + int rc) +{ + if (!m || !req) + return; + + if (rc >= 0 || rc == -ENOENT || rc == -ETIMEDOUT) { + s64 latency = req->r_end_stamp - req->r_start_stamp; + + percpu_counter_inc(&m->total_reads); + percpu_counter_add(&m->read_latency_sum, latency); + } +} #endif /* _FS_CEPH_MDS_METRIC_H */ diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h index 9d9f745b98a1..02ff3a302d26 100644 --- a/include/linux/ceph/osd_client.h +++ b/include/linux/ceph/osd_client.h @@ -213,6 +213,7 @@ struct ceph_osd_request { /* internal */ unsigned long r_stamp; /* jiffies, send or check time */ unsigned long r_start_stamp; /* jiffies */ + unsigned long r_end_stamp; /* jiffies */ int r_attempts; u32 r_map_dne_bound; diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index 8ff2856e2d52..108c9457d629 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -2389,6 +2389,8 @@ static void finish_request(struct ceph_osd_request *req) WARN_ON(lookup_request_mc(&osdc->map_checks, req->r_tid)); dout("%s req %p tid %llu\n", __func__, req, req->r_tid); + req->r_end_stamp = jiffies; + if (req->r_osd) unlink_request(req->r_osd, req); atomic_dec(&osdc->num_requests); From patchwork Wed Feb 19 03:38:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11390293 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B57E0138D for ; Wed, 19 Feb 2020 03:40:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 96C46222D9 for ; Wed, 19 Feb 2020 03:40:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YfdCUEJS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726609AbgBSDk0 (ORCPT ); Tue, 18 Feb 2020 22:40:26 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:52231 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726403AbgBSDk0 (ORCPT ); Tue, 18 Feb 2020 22:40:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582083625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ef6shB1dA4mdf7G9MYmF6EUk0XivhOc57UIOtJvjxes=; b=YfdCUEJSoq4XRwPwMO27NrIqH6fT9jj3Xr0K8feiHwIvJdvDuDX1KQX1IpFcIfw09SUI4D MWTSedjKrzBAkqE+8M8HIUbwJ0a06kq8jwTFs+4dCeBkq6JMd0i2Cm41gk1jdnKEf3WUAV OD6X7E8qhRJ9EUfRfVTNLr6XRUV4f90= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-258-lQmrUnsoPDKR86s5hYuWDQ-1; Tue, 18 Feb 2020 22:40:23 -0500 X-MC-Unique: lQmrUnsoPDKR86s5hYuWDQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 955D5801E5C; Wed, 19 Feb 2020 03:40:22 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-94.pek2.redhat.com [10.72.12.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id 95C9460C81; Wed, 19 Feb 2020 03:40:15 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH v7 4/5] ceph: add global write latency metric support Date: Tue, 18 Feb 2020 22:38:50 -0500 Message-Id: <20200219033851.6548-5-xiubli@redhat.com> In-Reply-To: <20200219033851.6548-1-xiubli@redhat.com> References: <20200219033851.6548-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li It will calculate the latency for the write osd requests, which only include the time cousumed by network and the ceph osd. item total sum_lat(us) avg_lat(us) ----------------------------------------------------- write 1048 8778000 8375 URL: https://tracker.ceph.com/issues/43215 Signed-off-by: Xiubo Li --- fs/ceph/addr.c | 7 +++++++ fs/ceph/debugfs.c | 6 ++++++ fs/ceph/file.c | 13 ++++++++++--- fs/ceph/mds_client.c | 14 ++++++++++++++ fs/ceph/metric.h | 18 ++++++++++++++++++ 5 files changed, 55 insertions(+), 3 deletions(-) diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 16573a13ffee..aca2ca592e53 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -649,6 +649,8 @@ static int ceph_sync_writepages(struct ceph_fs_client *fsc, if (!rc) rc = ceph_osdc_wait_request(osdc, req); + ceph_update_write_latency(&fsc->mdsc->metric, req, rc); + ceph_osdc_put_request(req); if (rc == 0) rc = len; @@ -800,6 +802,8 @@ static void writepages_finish(struct ceph_osd_request *req) ceph_clear_error_write(ci); } + ceph_update_write_latency(&fsc->mdsc->metric, req, rc); + /* * We lost the cache cap, need to truncate the page before * it is unlocked, otherwise we'd truncate it later in the @@ -1858,6 +1862,9 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) err = ceph_osdc_start_request(&fsc->client->osdc, req, false); if (!err) err = ceph_osdc_wait_request(&fsc->client->osdc, req); + + ceph_update_write_latency(&fsc->mdsc->metric, req, err); + out_put: ceph_osdc_put_request(req); if (err == -ECANCELED) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index d814a3a27611..464bfbdb970d 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -140,6 +140,12 @@ static int metric_show(struct seq_file *s, void *p) avg = total ? sum / total : 0; seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "read", total, sum, avg); + total = percpu_counter_sum(&mdsc->metric.total_writes); + sum = percpu_counter_sum(&mdsc->metric.write_latency_sum); + sum = jiffies_to_usecs(sum); + avg = total ? sum / total : 0; + seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "write", total, sum, avg); + seq_printf(s, "\n"); seq_printf(s, "item total miss hit\n"); seq_printf(s, "-------------------------------------------------\n"); diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 96e35935b764..0a25dc7e3a52 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -811,8 +811,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) inode, rc, osd_data->bvec_pos.iter.bi_size); /* r_start_stamp == 0 means the request was not submitted */ - if (req->r_start_stamp && !aio_req->write) - ceph_update_read_latency(metric, req, rc); + if (req->r_start_stamp) { + if (aio_req->write) + ceph_update_write_latency(metric, req, rc); + else + ceph_update_read_latency(metric, req, rc); + } if (rc == -EOLDSNAPC) { struct ceph_aio_work *aio_work; @@ -1059,7 +1063,9 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, if (!ret) ret = ceph_osdc_wait_request(&fsc->client->osdc, req); - if (!write) + if (write) + ceph_update_write_latency(metric, req, ret); + else ceph_update_read_latency(metric, req, ret); size = i_size_read(inode); @@ -1233,6 +1239,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, if (!ret) ret = ceph_osdc_wait_request(&fsc->client->osdc, req); + ceph_update_write_latency(&fsc->mdsc->metric, req, ret); out: ceph_osdc_put_request(req); if (ret != 0) { diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index b7ada9cde4f8..58e97ac004d6 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4192,8 +4192,20 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) if (ret) goto err_read_latency_sum; + ret = percpu_counter_init(&metric->total_writes, 0, GFP_KERNEL); + if (ret) + goto err_total_writes; + + ret = percpu_counter_init(&metric->write_latency_sum, 0, GFP_KERNEL); + if (ret) + goto err_write_latency_sum; + return 0; +err_write_latency_sum: + percpu_counter_destroy(&metric->total_writes); +err_total_writes: + percpu_counter_destroy(&metric->read_latency_sum); err_read_latency_sum: percpu_counter_destroy(&metric->total_reads); err_total_reads: @@ -4545,6 +4557,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.write_latency_sum); + percpu_counter_destroy(&mdsc->metric.total_writes); percpu_counter_destroy(&mdsc->metric.read_latency_sum); percpu_counter_destroy(&mdsc->metric.total_reads); percpu_counter_destroy(&mdsc->metric.i_caps_mis); diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h index afea44a3794b..a87197f3e915 100644 --- a/fs/ceph/metric.h +++ b/fs/ceph/metric.h @@ -15,6 +15,9 @@ struct ceph_client_metric { struct percpu_counter total_reads; struct percpu_counter read_latency_sum; + + struct percpu_counter total_writes; + struct percpu_counter write_latency_sum; }; static inline void ceph_update_read_latency(struct ceph_client_metric *m, @@ -31,4 +34,19 @@ static inline void ceph_update_read_latency(struct ceph_client_metric *m, percpu_counter_add(&m->read_latency_sum, latency); } } + +static inline void ceph_update_write_latency(struct ceph_client_metric *m, + struct ceph_osd_request *req, + int rc) +{ + if (!m || !req) + return; + + if (!rc || rc == -ETIMEDOUT) { + s64 latency = req->r_end_stamp - req->r_start_stamp; + + percpu_counter_inc(&m->total_writes); + percpu_counter_add(&m->write_latency_sum, latency); + } +} #endif /* _FS_CEPH_MDS_METRIC_H */ From patchwork Wed Feb 19 03:38:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11390295 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A2B1138D for ; Wed, 19 Feb 2020 03:40:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1B01F222D9 for ; Wed, 19 Feb 2020 03:40:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="f7x1djXT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726617AbgBSDkd (ORCPT ); Tue, 18 Feb 2020 22:40:33 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:20094 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726403AbgBSDkd (ORCPT ); Tue, 18 Feb 2020 22:40:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582083632; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FGwh+mJVd3aU9xgQWNmizDxEhuq6F1hS4iFZ3LUflG8=; b=f7x1djXTGBlIAX77a99yzLF+IF2GCYwJO0u0A+XFXxYzns5M2mFa1Vv/yZ0uw1yuuH6azq VAWKE0CKo3xPpqA1FHJ94OG1tlmJ89hWaRR5AYVvl3TKmJD9j5C00iuo1zikoVeATkJVuZ 8Q9etEz0/8/8IGPsvkU0KCzYmz3+8HA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-330-Hk-URr4wMgi8d831l99HTg-1; Tue, 18 Feb 2020 22:40:30 -0500 X-MC-Unique: Hk-URr4wMgi8d831l99HTg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0BC6B13E4; Wed, 19 Feb 2020 03:40:29 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-94.pek2.redhat.com [10.72.12.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9EE8060C81; Wed, 19 Feb 2020 03:40:23 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH v7 5/5] ceph: add global metadata perf metric support Date: Tue, 18 Feb 2020 22:38:51 -0500 Message-Id: <20200219033851.6548-6-xiubli@redhat.com> In-Reply-To: <20200219033851.6548-1-xiubli@redhat.com> References: <20200219033851.6548-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li It will calculate the latency for the metedata requests, which only include the time cousumed by network and the ceph. item total sum_lat(us) avg_lat(us) ----------------------------------------------------- metadata 113 220000 1946 URL: https://tracker.ceph.com/issues/43215 Signed-off-by: Xiubo Li --- fs/ceph/debugfs.c | 6 ++++++ fs/ceph/mds_client.c | 20 ++++++++++++++++++++ fs/ceph/metric.h | 13 +++++++++++++ 3 files changed, 39 insertions(+) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index 464bfbdb970d..60f3e307fca1 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -146,6 +146,12 @@ static int metric_show(struct seq_file *s, void *p) avg = total ? sum / total : 0; seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "write", total, sum, avg); + total = percpu_counter_sum(&mdsc->metric.total_metadatas); + sum = percpu_counter_sum(&mdsc->metric.metadata_latency_sum); + sum = jiffies_to_usecs(sum); + avg = total ? sum / total : 0; + seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "metadata", total, sum, avg); + seq_printf(s, "\n"); seq_printf(s, "item total miss hit\n"); seq_printf(s, "-------------------------------------------------\n"); diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 58e97ac004d6..eb494f1aa137 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -3021,6 +3021,12 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg) /* kick calling process */ complete_request(mdsc, req); + + if (!result || result == -ENOENT) { + s64 latency = jiffies - req->r_started; + + ceph_update_metadata_latency(&mdsc->metric, latency); + } out: ceph_mdsc_put_request(req); return; @@ -4200,8 +4206,20 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) if (ret) goto err_write_latency_sum; + ret = percpu_counter_init(&metric->total_metadatas, 0, GFP_KERNEL); + if (ret) + goto err_total_metadatas; + + ret = percpu_counter_init(&metric->metadata_latency_sum, 0, GFP_KERNEL); + if (ret) + goto err_metadata_latency_sum; + return 0; +err_metadata_latency_sum: + percpu_counter_destroy(&metric->total_metadatas); +err_total_metadatas: + percpu_counter_destroy(&metric->write_latency_sum); err_write_latency_sum: percpu_counter_destroy(&metric->total_writes); err_total_writes: @@ -4557,6 +4575,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.metadata_latency_sum); + percpu_counter_destroy(&mdsc->metric.total_metadatas); percpu_counter_destroy(&mdsc->metric.write_latency_sum); percpu_counter_destroy(&mdsc->metric.total_writes); percpu_counter_destroy(&mdsc->metric.read_latency_sum); diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h index a87197f3e915..9de8beb436c7 100644 --- a/fs/ceph/metric.h +++ b/fs/ceph/metric.h @@ -18,6 +18,9 @@ struct ceph_client_metric { struct percpu_counter total_writes; struct percpu_counter write_latency_sum; + + struct percpu_counter total_metadatas; + struct percpu_counter metadata_latency_sum; }; static inline void ceph_update_read_latency(struct ceph_client_metric *m, @@ -49,4 +52,14 @@ static inline void ceph_update_write_latency(struct ceph_client_metric *m, percpu_counter_add(&m->write_latency_sum, latency); } } + +static inline void ceph_update_metadata_latency(struct ceph_client_metric *m, + s64 latency) +{ + if (!m) + return; + + percpu_counter_inc(&m->total_metadatas); + percpu_counter_add(&m->metadata_latency_sum, latency); +} #endif /* _FS_CEPH_MDS_METRIC_H */