From patchwork Tue Dec 24 04:05:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11309029 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA6E1109A for ; Tue, 24 Dec 2019 04:05:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A511F206D3 for ; Tue, 24 Dec 2019 04:05:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="asaR/rgJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726924AbfLXEFd (ORCPT ); Mon, 23 Dec 2019 23:05:33 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:54747 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726747AbfLXEFd (ORCPT ); Mon, 23 Dec 2019 23:05:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1577160331; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0E7kWaQVA4ZcQxO8byvdRefn8POI7zlJJR2ZiVAdkRI=; b=asaR/rgJ430nBuieTEwnN3tUOz4vBVCAjho2tRVuHv4Ni3C51vOqDdfn4wmGOAZLa2Y4S9 Au58t6e5S1Z3XM1JoRz+5E4WTKgIDEZbTM4nH/XCdHC3YwnbyARgNiAsD/m6WqvS8FYNHi 0iOn+FH/KZOhnxtlJ1P9dEV7RX2r5O4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-316-Sa7vOOinNXy7z08ZMs1wsw-1; Mon, 23 Dec 2019 23:05:29 -0500 X-MC-Unique: Sa7vOOinNXy7z08ZMs1wsw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 17624184B44C; Tue, 24 Dec 2019 04:05:28 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-64.pek2.redhat.com [10.72.12.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 661811000322; Tue, 24 Dec 2019 04:05:25 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 1/4] ceph: add global dentry lease metric support Date: Mon, 23 Dec 2019 23:05:11 -0500 Message-Id: <20191224040514.26144-2-xiubli@redhat.com> In-Reply-To: <20191224040514.26144-1-xiubli@redhat.com> References: <20191224040514.26144-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li For the dentry lease we will only count the hit/miss info triggered from the vfs calls, for the cases like request reply handling and perodically ceph_trim_dentries() we will ignore them. Currently only the debugfs is support: The output will be: item total miss hit ------------------------------------------------- d_lease 11 7 141 Signed-off-by: Xiubo Li --- fs/ceph/debugfs.c | 32 ++++++++++++++++++++++++++++---- fs/ceph/dir.c | 18 ++++++++++++++++-- fs/ceph/mds_client.c | 39 +++++++++++++++++++++++++++++++++++++-- fs/ceph/mds_client.h | 9 +++++++++ fs/ceph/super.h | 1 + 5 files changed, 91 insertions(+), 8 deletions(-) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index fb7cabd98e7b..40a22da0214a 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -124,6 +124,22 @@ static int mdsc_show(struct seq_file *s, void *p) return 0; } +static int metric_show(struct seq_file *s, void *p) +{ + struct ceph_fs_client *fsc = s->private; + struct ceph_mds_client *mdsc = fsc->mdsc; + + seq_printf(s, "item total miss hit\n"); + seq_printf(s, "-------------------------------------------------\n"); + + seq_printf(s, "%-14s%-16lld%-16lld%lld\n", "d_lease", + atomic64_read(&mdsc->metric.total_dentries), + percpu_counter_sum(&mdsc->metric.d_lease_mis), + percpu_counter_sum(&mdsc->metric.d_lease_hit)); + + return 0; +} + static int caps_show_cb(struct inode *inode, struct ceph_cap *cap, void *p) { struct seq_file *s = p; @@ -220,6 +236,7 @@ static int mds_sessions_show(struct seq_file *s, void *ptr) CEPH_DEFINE_SHOW_FUNC(mdsmap_show) CEPH_DEFINE_SHOW_FUNC(mdsc_show) +CEPH_DEFINE_SHOW_FUNC(metric_show) CEPH_DEFINE_SHOW_FUNC(caps_show) CEPH_DEFINE_SHOW_FUNC(mds_sessions_show) @@ -255,6 +272,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) debugfs_remove(fsc->debugfs_mdsmap); debugfs_remove(fsc->debugfs_mds_sessions); debugfs_remove(fsc->debugfs_caps); + debugfs_remove(fsc->debugfs_metric); debugfs_remove(fsc->debugfs_mdsc); } @@ -295,11 +313,17 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc) fsc, &mdsc_show_fops); + fsc->debugfs_metric = debugfs_create_file("metrics", + 0400, + fsc->client->debugfs_dir, + fsc, + &metric_show_fops); + fsc->debugfs_caps = debugfs_create_file("caps", - 0400, - fsc->client->debugfs_dir, - fsc, - &caps_show_fops); + 0400, + fsc->client->debugfs_dir, + fsc, + &caps_show_fops); } diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index d0cd0aba5843..382beb04bacb 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -38,6 +38,8 @@ static int __dir_lease_try_check(const struct dentry *dentry); static int ceph_d_init(struct dentry *dentry) { struct ceph_dentry_info *di; + struct ceph_fs_client *fsc = ceph_sb_to_client(dentry->d_sb); + struct ceph_mds_client *mdsc = fsc->mdsc; di = kmem_cache_zalloc(ceph_dentry_cachep, GFP_KERNEL); if (!di) @@ -48,6 +50,9 @@ static int ceph_d_init(struct dentry *dentry) di->time = jiffies; dentry->d_fsdata = di; INIT_LIST_HEAD(&di->lease_list); + + atomic64_inc(&mdsc->metric.total_dentries); + return 0; } @@ -1551,6 +1556,7 @@ static int dir_lease_is_valid(struct inode *dir, struct dentry *dentry) */ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) { + struct ceph_mds_client *mdsc; int valid = 0; struct dentry *parent; struct inode *dir, *inode; @@ -1589,13 +1595,14 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) } } + mdsc = ceph_sb_to_client(dir->i_sb)->mdsc; if (!valid) { - struct ceph_mds_client *mdsc = - ceph_sb_to_client(dir->i_sb)->mdsc; struct ceph_mds_request *req; int op, err; u32 mask; + percpu_counter_inc(&mdsc->metric.d_lease_mis); + if (flags & LOOKUP_RCU) return -ECHILD; @@ -1630,6 +1637,8 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) dout("d_revalidate %p lookup result=%d\n", dentry, err); } + } else { + percpu_counter_inc(&mdsc->metric.d_lease_hit); } dout("d_revalidate %p %s\n", dentry, valid ? "valid" : "invalid"); @@ -1638,6 +1647,7 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) if (!(flags & LOOKUP_RCU)) dput(parent); + return valid; } @@ -1672,9 +1682,13 @@ static int ceph_d_delete(const struct dentry *dentry) static void ceph_d_release(struct dentry *dentry) { struct ceph_dentry_info *di = ceph_dentry(dentry); + struct ceph_fs_client *fsc = ceph_sb_to_client(dentry->d_sb); + struct ceph_mds_client *mdsc = fsc->mdsc; dout("d_release %p\n", dentry); + atomic64_dec(&mdsc->metric.total_dentries); + spin_lock(&dentry->d_lock); __dentry_lease_unlist(di); dentry->d_fsdata = NULL; diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index d6c3d8d854e0..b1b4d1a9bd2f 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4151,10 +4151,33 @@ static void delayed_work(struct work_struct *work) schedule_delayed(mdsc); } +static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) +{ + int ret; + + if (!metric) + return -EINVAL; + + ret = percpu_counter_init(&metric->d_lease_hit, 0, GFP_KERNEL); + if (ret) + goto err; + ret = percpu_counter_init(&metric->d_lease_mis, 0, GFP_KERNEL); + if (ret) + goto err; + + atomic64_set(&metric->total_dentries, 0); + return 0; + +err: + percpu_counter_destroy(&metric->d_lease_hit); + return ret; +} + int ceph_mdsc_init(struct ceph_fs_client *fsc) { struct ceph_mds_client *mdsc; + int err; mdsc = kzalloc(sizeof(struct ceph_mds_client), GFP_NOFS); if (!mdsc) @@ -4163,8 +4186,8 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) mutex_init(&mdsc->mutex); mdsc->mdsmap = kzalloc(sizeof(*mdsc->mdsmap), GFP_NOFS); if (!mdsc->mdsmap) { - kfree(mdsc); - return -ENOMEM; + err = -ENOMEM; + goto err_mdsc; } fsc->mdsc = mdsc; @@ -4203,6 +4226,9 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) init_waitqueue_head(&mdsc->cap_flushing_wq); INIT_WORK(&mdsc->cap_reclaim_work, ceph_cap_reclaim_work); atomic_set(&mdsc->cap_reclaim_pending, 0); + err = ceph_mdsc_metric_init(&mdsc->metric); + if (err) + goto err_mdsmap; spin_lock_init(&mdsc->dentry_list_lock); INIT_LIST_HEAD(&mdsc->dentry_leases); @@ -4221,6 +4247,12 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) strscpy(mdsc->nodename, utsname()->nodename, sizeof(mdsc->nodename)); return 0; + +err_mdsmap: + kfree(mdsc->mdsmap); +err_mdsc: + kfree(mdsc); + return err; } /* @@ -4478,6 +4510,9 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc) ceph_mdsc_stop(mdsc); + percpu_counter_destroy(&mdsc->metric.d_lease_mis); + percpu_counter_destroy(&mdsc->metric.d_lease_hit); + fsc->mdsc = NULL; kfree(mdsc); dout("mdsc_destroy %p done\n", mdsc); diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index c021df5f50ce..55249f667a98 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -349,6 +349,13 @@ struct cap_wait { int want; }; +/* This is the global metrics */ +struct ceph_client_metric { + struct percpu_counter d_lease_hit; + struct percpu_counter d_lease_mis; + atomic64_t total_dentries; +}; + /* * mds client state */ @@ -437,6 +444,8 @@ struct ceph_mds_client { struct list_head dentry_leases; /* fifo list */ struct list_head dentry_dir_leases; /* lru list */ + struct ceph_client_metric metric; + spinlock_t snapid_map_lock; struct rb_root snapid_map_tree; struct list_head snapid_map_lru; diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 3bf1a01cd536..40703588b889 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -123,6 +123,7 @@ struct ceph_fs_client { struct dentry *debugfs_congestion_kb; struct dentry *debugfs_bdi; struct dentry *debugfs_mdsc, *debugfs_mdsmap; + struct dentry *debugfs_metric; struct dentry *debugfs_mds_sessions; #endif From patchwork Tue Dec 24 04:05:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11309031 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5D35B109A for ; Tue, 24 Dec 2019 04:05:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1D06E206D3 for ; Tue, 24 Dec 2019 04:05:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VRCGrQgC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726928AbfLXEFg (ORCPT ); Mon, 23 Dec 2019 23:05:36 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:26879 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726747AbfLXEFg (ORCPT ); Mon, 23 Dec 2019 23:05:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1577160334; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b0wkM2zcCzCbHnoSYwrbUTRypTa0sIEfxJyNAaQj8N4=; b=VRCGrQgCr1wG8HShf7N2LnGY2ivmBlKB1pxHxPksua0x86mP1O2nDtcgB4dsiLz9OjMiNn xTv7oUcb4mMUMVqoDYQ8C8z9Cfv9cMrVTH/8Rtjgrtv72VB/tTIrgms2+a2PG4w13FwOHo svnQfIFIzMD1ka3uYzq0T/a8aGDsZbk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-81-7YSFNyZXPruFwNAV5PEEZA-1; Mon, 23 Dec 2019 23:05:32 -0500 X-MC-Unique: 7YSFNyZXPruFwNAV5PEEZA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 66831800D48; Tue, 24 Dec 2019 04:05:31 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-64.pek2.redhat.com [10.72.12.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id A8B351000322; Tue, 24 Dec 2019 04:05:28 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 2/4] ceph: add caps perf metric for each session Date: Mon, 23 Dec 2019 23:05:12 -0500 Message-Id: <20191224040514.26144-3-xiubli@redhat.com> In-Reply-To: <20191224040514.26144-1-xiubli@redhat.com> References: <20191224040514.26144-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li This will fulfill the caps hit/miss metric for each session. When checking the "need" mask and if one cap has the subset of the "need" mask it means hit, or missed. item total miss hit ------------------------------------------------- d_lease 295 0 993 session caps miss hit ------------------------------------------------- 0 295 107 4119 1 1 107 9 Signed-off-by: Xiubo Li --- fs/ceph/acl.c | 2 +- fs/ceph/caps.c | 63 +++++++++++++++++++++++++++++++++++--------- fs/ceph/debugfs.c | 20 ++++++++++++++ fs/ceph/dir.c | 20 +++++++------- fs/ceph/file.c | 6 ++--- fs/ceph/inode.c | 8 +++--- fs/ceph/mds_client.c | 16 ++++++++++- fs/ceph/mds_client.h | 3 +++ fs/ceph/snap.c | 2 +- fs/ceph/super.h | 12 +++++---- fs/ceph/xattr.c | 8 +++--- 11 files changed, 120 insertions(+), 40 deletions(-) diff --git a/fs/ceph/acl.c b/fs/ceph/acl.c index 26be6520d3fb..fca6ff231020 100644 --- a/fs/ceph/acl.c +++ b/fs/ceph/acl.c @@ -22,7 +22,7 @@ static inline void ceph_set_cached_acl(struct inode *inode, struct ceph_inode_info *ci = ceph_inode(inode); spin_lock(&ci->i_ceph_lock); - if (__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 0)) + if (__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 0, true)) set_cached_acl(inode, type, acl); else forget_cached_acl(inode, type); diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c index 28ae0c134700..6ab02aab7d9c 100644 --- a/fs/ceph/caps.c +++ b/fs/ceph/caps.c @@ -567,7 +567,7 @@ static void __cap_delay_cancel(struct ceph_mds_client *mdsc, static void __check_cap_issue(struct ceph_inode_info *ci, struct ceph_cap *cap, unsigned issued) { - unsigned had = __ceph_caps_issued(ci, NULL); + unsigned int had = __ceph_caps_issued(ci, NULL, -1); /* * Each time we receive FILE_CACHE anew, we increment @@ -787,20 +787,43 @@ static int __cap_is_valid(struct ceph_cap *cap) * out, and may be invalidated in bulk if the client session times out * and session->s_cap_gen is bumped. */ -int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented) +int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented, int mask) { int have = ci->i_snap_caps; struct ceph_cap *cap; struct rb_node *p; + int revoking = 0; + + if (ci->i_auth_cap) { + cap = ci->i_auth_cap; + revoking = cap->implemented & ~cap->issued; + } if (implemented) *implemented = 0; for (p = rb_first(&ci->i_caps); p; p = rb_next(p)) { + struct ceph_mds_session *s; + int r = 0; + cap = rb_entry(p, struct ceph_cap, ci_node); if (!__cap_is_valid(cap)) continue; dout("__ceph_caps_issued %p cap %p issued %s\n", &ci->vfs_inode, cap, ceph_cap_string(cap->issued)); + + if (mask >= 0) { + s = ceph_get_mds_session(cap->session); + if (cap == ci->i_auth_cap) + r = revoking; + if (s) { + if (mask & (cap->issued & ~r)) + percpu_counter_inc(&s->i_caps_hit); + else + percpu_counter_inc(&s->i_caps_mis); + ceph_put_mds_session(s); + } + } + have |= cap->issued; if (implemented) *implemented |= cap->implemented; @@ -810,10 +833,8 @@ int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented) * by the auth MDS. The non-auth MDS should be revoking/exporting * these caps, but the message is delayed. */ - if (ci->i_auth_cap) { - cap = ci->i_auth_cap; - have &= ~cap->implemented | cap->issued; - } + have &= ~revoking; + return have; } @@ -862,7 +883,8 @@ static void __touch_cap(struct ceph_cap *cap) * front of their respective LRUs. (This is the preferred way for * callers to check for caps they want.) */ -int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int touch) +int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int touch, + bool metric) { struct ceph_cap *cap; struct rb_node *p; @@ -877,9 +899,13 @@ int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int touch) } for (p = rb_first(&ci->i_caps); p; p = rb_next(p)) { + struct ceph_mds_session *s; + cap = rb_entry(p, struct ceph_cap, ci_node); if (!__cap_is_valid(cap)) continue; + + s = ceph_get_mds_session(cap->session); if ((cap->issued & mask) == mask) { dout("__ceph_caps_issued_mask ino 0x%lx cap %p issued %s" " (mask %s)\n", ci->vfs_inode.i_ino, cap, @@ -887,9 +913,22 @@ int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int touch) ceph_cap_string(mask)); if (touch) __touch_cap(cap); + if (s) { + if (metric) + percpu_counter_inc(&s->i_caps_hit); + ceph_put_mds_session(s); + } return 1; } + if (s) { + if (cap->issued & mask) + percpu_counter_inc(&s->i_caps_hit); + else + percpu_counter_inc(&s->i_caps_mis); + ceph_put_mds_session(s); + } + /* does a combination of caps satisfy mask? */ have |= cap->issued; if ((have & mask) == mask) { @@ -1849,7 +1888,7 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags, retry_locked: file_wanted = __ceph_caps_file_wanted(ci); used = __ceph_caps_used(ci); - issued = __ceph_caps_issued(ci, &implemented); + issued = __ceph_caps_issued(ci, &implemented, -1); revoking = implemented & ~issued; want = file_wanted; @@ -2577,7 +2616,7 @@ static int try_get_cap_refs(struct inode *inode, int need, int want, spin_lock(&ci->i_ceph_lock); } - have = __ceph_caps_issued(ci, &implemented); + have = __ceph_caps_issued(ci, &implemented, need); if (have & need & CEPH_CAP_FILE_WR) { if (endoff >= 0 && endoff > (loff_t)ci->i_max_size) { @@ -3563,7 +3602,7 @@ static void handle_cap_trunc(struct inode *inode, u64 size = le64_to_cpu(trunc->size); int implemented = 0; int dirty = __ceph_caps_dirty(ci); - int issued = __ceph_caps_issued(ceph_inode(inode), &implemented); + int issued = __ceph_caps_issued(ceph_inode(inode), &implemented, -1); int queue_trunc = 0; issued |= implemented | dirty; @@ -3770,7 +3809,7 @@ static void handle_cap_import(struct ceph_mds_client *mdsc, } } - __ceph_caps_issued(ci, &issued); + __ceph_caps_issued(ci, &issued, -1); issued |= __ceph_caps_dirty(ci); ceph_add_cap(inode, session, cap_id, -1, caps, wanted, seq, mseq, @@ -3996,7 +4035,7 @@ void ceph_handle_caps(struct ceph_mds_session *session, switch (op) { case CEPH_CAP_OP_REVOKE: case CEPH_CAP_OP_GRANT: - __ceph_caps_issued(ci, &extra_info.issued); + __ceph_caps_issued(ci, &extra_info.issued, -1); extra_info.issued |= __ceph_caps_dirty(ci); handle_cap_grant(inode, session, cap, h, msg->middle, &extra_info); diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index 40a22da0214a..c132fdb40d53 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -128,6 +128,7 @@ static int metric_show(struct seq_file *s, void *p) { struct ceph_fs_client *fsc = s->private; struct ceph_mds_client *mdsc = fsc->mdsc; + int i; seq_printf(s, "item total miss hit\n"); seq_printf(s, "-------------------------------------------------\n"); @@ -137,6 +138,25 @@ static int metric_show(struct seq_file *s, void *p) percpu_counter_sum(&mdsc->metric.d_lease_mis), percpu_counter_sum(&mdsc->metric.d_lease_hit)); + seq_printf(s, "\n"); + seq_printf(s, "session caps miss hit\n"); + seq_printf(s, "-------------------------------------------------\n"); + + mutex_lock(&mdsc->mutex); + for (i = 0; i < mdsc->max_sessions; i++) { + struct ceph_mds_session *session; + + session = __ceph_lookup_mds_session(mdsc, i); + if (!session) + continue; + seq_printf(s, "%-14d%-16d%-16lld%lld\n", i, + session->s_nr_caps, + percpu_counter_sum(&session->i_caps_mis), + percpu_counter_sum(&session->i_caps_hit)); + ceph_put_mds_session(session); + } + mutex_unlock(&mdsc->mutex); + return 0; } diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index 382beb04bacb..1e1ccae8953d 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -30,7 +30,7 @@ const struct dentry_operations ceph_dentry_ops; static bool __dentry_lease_is_valid(struct ceph_dentry_info *di); -static int __dir_lease_try_check(const struct dentry *dentry); +static int __dir_lease_try_check(const struct dentry *dentry, bool metric); /* * Initialize ceph dentry state. @@ -346,7 +346,7 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) !ceph_test_mount_opt(fsc, NOASYNCREADDIR) && ceph_snap(inode) != CEPH_SNAPDIR && __ceph_dir_is_complete_ordered(ci) && - __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1)) { + __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1, true)) { int shared_gen = atomic_read(&ci->i_shared_gen); spin_unlock(&ci->i_ceph_lock); err = __dcache_readdir(file, ctx, shared_gen); @@ -764,7 +764,8 @@ static struct dentry *ceph_lookup(struct inode *dir, struct dentry *dentry, !is_root_ceph_dentry(dir, dentry) && ceph_test_mount_opt(fsc, DCACHE) && __ceph_dir_is_complete(ci) && - (__ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1))) { + (__ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1, + true))) { spin_unlock(&ci->i_ceph_lock); dout(" dir %p complete, -ENOENT\n", dir); d_add(dentry, NULL); @@ -1341,7 +1342,7 @@ static int __dentry_lease_check(struct dentry *dentry, void *arg) if (__dentry_lease_is_valid(di)) return STOP; - ret = __dir_lease_try_check(dentry); + ret = __dir_lease_try_check(dentry, false); if (ret == -EBUSY) return KEEP; if (ret > 0) @@ -1354,7 +1355,7 @@ static int __dir_lease_check(struct dentry *dentry, void *arg) struct ceph_lease_walk_control *lwc = arg; struct ceph_dentry_info *di = ceph_dentry(dentry); - int ret = __dir_lease_try_check(dentry); + int ret = __dir_lease_try_check(dentry, false); if (ret == -EBUSY) return KEEP; if (ret > 0) { @@ -1493,7 +1494,7 @@ static int dentry_lease_is_valid(struct dentry *dentry, unsigned int flags) /* * Called under dentry->d_lock. */ -static int __dir_lease_try_check(const struct dentry *dentry) +static int __dir_lease_try_check(const struct dentry *dentry, bool metric) { struct ceph_dentry_info *di = ceph_dentry(dentry); struct inode *dir; @@ -1510,7 +1511,8 @@ static int __dir_lease_try_check(const struct dentry *dentry) if (spin_trylock(&ci->i_ceph_lock)) { if (atomic_read(&ci->i_shared_gen) == di->lease_shared_gen && - __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 0)) + __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 0, + metric)) valid = 1; spin_unlock(&ci->i_ceph_lock); } else { @@ -1532,7 +1534,7 @@ static int dir_lease_is_valid(struct inode *dir, struct dentry *dentry) int shared_gen; spin_lock(&ci->i_ceph_lock); - valid = __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1); + valid = __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1, true); shared_gen = atomic_read(&ci->i_shared_gen); spin_unlock(&ci->i_ceph_lock); if (valid) { @@ -1670,7 +1672,7 @@ static int ceph_d_delete(const struct dentry *dentry) if (di) { if (__dentry_lease_is_valid(di)) return 0; - if (__dir_lease_try_check(dentry)) + if (__dir_lease_try_check(dentry, true)) return 0; } return 1; diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 11929d2bb594..418c7b30c6db 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -296,7 +296,7 @@ int ceph_renew_caps(struct inode *inode) wanted = __ceph_caps_file_wanted(ci); if (__ceph_is_any_real_caps(ci) && (!(wanted & CEPH_CAP_ANY_WR) || ci->i_auth_cap)) { - int issued = __ceph_caps_issued(ci, NULL); + int issued = __ceph_caps_issued(ci, NULL, -1); spin_unlock(&ci->i_ceph_lock); dout("renew caps %p want %s issued %s updating mds_wanted\n", inode, ceph_cap_string(wanted), ceph_cap_string(issued)); @@ -387,7 +387,7 @@ int ceph_open(struct inode *inode, struct file *file) if (__ceph_is_any_real_caps(ci) && (((fmode & CEPH_FILE_MODE_WR) == 0) || ci->i_auth_cap)) { int mds_wanted = __ceph_caps_mds_wanted(ci, true); - int issued = __ceph_caps_issued(ci, NULL); + int issued = __ceph_caps_issued(ci, NULL, fmode); dout("open %p fmode %d want %s issued %s using existing\n", inode, fmode, ceph_cap_string(wanted), @@ -403,7 +403,7 @@ int ceph_open(struct inode *inode, struct file *file) return ceph_init_file(inode, file, fmode); } else if (ceph_snap(inode) != CEPH_NOSNAP && - (ci->i_snap_caps & wanted) == wanted) { + (ci->i_snap_caps & wanted) == wanted) { __ceph_get_fmode(ci, fmode); spin_unlock(&ci->i_ceph_lock); return ceph_init_file(inode, file, fmode); diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index 64634c5af403..c0108b8582c3 100644 --- a/fs/ceph/inode.c +++ b/fs/ceph/inode.c @@ -798,7 +798,7 @@ static int fill_inode(struct inode *inode, struct page *locked_page, /* Update change_attribute */ inode_set_max_iversion_raw(inode, iinfo->change_attr); - __ceph_caps_issued(ci, &issued); + __ceph_caps_issued(ci, &issued, -1); issued |= __ceph_caps_dirty(ci); new_issued = ~issued & info_caps; @@ -2029,7 +2029,7 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) } spin_lock(&ci->i_ceph_lock); - issued = __ceph_caps_issued(ci, NULL); + issued = __ceph_caps_issued(ci, NULL, -1); if (!ci->i_head_snapc && (issued & (CEPH_CAP_ANY_EXCL | CEPH_CAP_FILE_WR))) { @@ -2038,7 +2038,7 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) spin_unlock(&ci->i_ceph_lock); down_read(&mdsc->snap_rwsem); spin_lock(&ci->i_ceph_lock); - issued = __ceph_caps_issued(ci, NULL); + issued = __ceph_caps_issued(ci, NULL, -1); } } @@ -2269,7 +2269,7 @@ int __ceph_do_getattr(struct inode *inode, struct page *locked_page, dout("do_getattr inode %p mask %s mode 0%o\n", inode, ceph_cap_string(mask), inode->i_mode); - if (!force && ceph_caps_issued_mask(ceph_inode(inode), mask, 1)) + if (!force && ceph_caps_issued_mask(ceph_inode(inode), mask, 1, true)) return 0; mode = (mask & CEPH_STAT_RSTAT) ? USE_AUTH_MDS : USE_ANY_MDS; diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index b1b4d1a9bd2f..f58b74b2d1ec 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -557,6 +557,8 @@ void ceph_put_mds_session(struct ceph_mds_session *s) if (refcount_dec_and_test(&s->s_ref)) { if (s->s_auth.authorizer) ceph_auth_destroy_authorizer(s->s_auth.authorizer); + percpu_counter_destroy(&s->i_caps_hit); + percpu_counter_destroy(&s->i_caps_mis); kfree(s); } } @@ -597,6 +599,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, int mds) { struct ceph_mds_session *s; + int err; if (mds >= mdsc->mdsmap->possible_max_rank) return ERR_PTR(-EINVAL); @@ -611,8 +614,10 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, dout("%s: realloc to %d\n", __func__, newmax); sa = kcalloc(newmax, sizeof(void *), GFP_NOFS); - if (!sa) + if (!sa) { + err = -ENOMEM; goto fail_realloc; + } if (mdsc->sessions) { memcpy(sa, mdsc->sessions, mdsc->max_sessions * sizeof(void *)); @@ -652,6 +657,13 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, INIT_LIST_HEAD(&s->s_cap_flushing); + err = percpu_counter_init(&s->i_caps_hit, 0, GFP_NOFS); + if (err) + goto fail_realloc; + err = percpu_counter_init(&s->i_caps_mis, 0, GFP_NOFS); + if (err) + goto fail_init; + mdsc->sessions[mds] = s; atomic_inc(&mdsc->num_sessions); refcount_inc(&s->s_ref); /* one ref to sessions[], one to caller */ @@ -661,6 +673,8 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, return s; +fail_init: + percpu_counter_destroy(&s->i_caps_hit); fail_realloc: kfree(s); return ERR_PTR(-ENOMEM); diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 55249f667a98..6b91f20132c0 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -192,6 +192,9 @@ struct ceph_mds_session { refcount_t s_ref; struct list_head s_waiting; /* waiting requests */ struct list_head s_unsafe; /* unsafe requests */ + + struct percpu_counter i_caps_hit; + struct percpu_counter i_caps_mis; }; /* diff --git a/fs/ceph/snap.c b/fs/ceph/snap.c index ccfcc66aaf44..3d34af145f81 100644 --- a/fs/ceph/snap.c +++ b/fs/ceph/snap.c @@ -534,7 +534,7 @@ void ceph_queue_cap_snap(struct ceph_inode_info *ci) INIT_LIST_HEAD(&capsnap->ci_item); capsnap->follows = old_snapc->seq; - capsnap->issued = __ceph_caps_issued(ci, NULL); + capsnap->issued = __ceph_caps_issued(ci, NULL, -1); capsnap->dirty = dirty; capsnap->mode = inode->i_mode; diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 40703588b889..88da9e21af75 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -635,8 +635,10 @@ static inline bool __ceph_is_any_real_caps(struct ceph_inode_info *ci) return !RB_EMPTY_ROOT(&ci->i_caps); } -extern int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented); -extern int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int t); +extern int __ceph_caps_issued(struct ceph_inode_info *ci, int *implemented, + int mask); +extern int __ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, int t, + bool metric); extern int __ceph_caps_issued_other(struct ceph_inode_info *ci, struct ceph_cap *cap); @@ -644,17 +646,17 @@ static inline int ceph_caps_issued(struct ceph_inode_info *ci) { int issued; spin_lock(&ci->i_ceph_lock); - issued = __ceph_caps_issued(ci, NULL); + issued = __ceph_caps_issued(ci, NULL, -1); spin_unlock(&ci->i_ceph_lock); return issued; } static inline int ceph_caps_issued_mask(struct ceph_inode_info *ci, int mask, - int touch) + int touch, bool metric) { int r; spin_lock(&ci->i_ceph_lock); - r = __ceph_caps_issued_mask(ci, mask, touch); + r = __ceph_caps_issued_mask(ci, mask, touch, metric); spin_unlock(&ci->i_ceph_lock); return r; } diff --git a/fs/ceph/xattr.c b/fs/ceph/xattr.c index 98a9a3101cda..aa9ee2c2d8f3 100644 --- a/fs/ceph/xattr.c +++ b/fs/ceph/xattr.c @@ -856,7 +856,7 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value, if (ci->i_xattrs.version == 0 || !((req_mask & CEPH_CAP_XATTR_SHARED) || - __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1))) { + __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1, true))) { spin_unlock(&ci->i_ceph_lock); /* security module gets xattr while filling trace */ @@ -914,7 +914,7 @@ ssize_t ceph_listxattr(struct dentry *dentry, char *names, size_t size) ci->i_xattrs.version, ci->i_xattrs.index_version); if (ci->i_xattrs.version == 0 || - !__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1)) { + !__ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 1, true)) { spin_unlock(&ci->i_ceph_lock); err = ceph_do_getattr(inode, CEPH_STAT_CAP_XATTR, true); if (err) @@ -1064,7 +1064,7 @@ int __ceph_setxattr(struct inode *inode, const char *name, spin_lock(&ci->i_ceph_lock); retry: - issued = __ceph_caps_issued(ci, NULL); + issued = __ceph_caps_issued(ci, NULL, -1); if (ci->i_xattrs.version == 0 || !(issued & CEPH_CAP_XATTR_EXCL)) goto do_sync; @@ -1192,7 +1192,7 @@ bool ceph_security_xattr_deadlock(struct inode *in) spin_lock(&ci->i_ceph_lock); ret = !(ci->i_ceph_flags & CEPH_I_SEC_INITED) && !(ci->i_xattrs.version > 0 && - __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 0)); + __ceph_caps_issued_mask(ci, CEPH_CAP_XATTR_SHARED, 0, true)); spin_unlock(&ci->i_ceph_lock); return ret; } From patchwork Tue Dec 24 04:05:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11309033 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC41E109A for ; Tue, 24 Dec 2019 04:05:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 818E9206D3 for ; Tue, 24 Dec 2019 04:05:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="INnajvwU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726936AbfLXEFi (ORCPT ); Mon, 23 Dec 2019 23:05:38 -0500 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:48980 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726747AbfLXEFi (ORCPT ); Mon, 23 Dec 2019 23:05:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1577160337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bvsOh3QEqoexaAJgVfTGKMtv2Y02aNfCOFM5VOGed7I=; b=INnajvwUB7qyzCGp1ecMOi4ISG3lRStIejEdSIFzGAKVm0CEukfUOiCrWTmd++2tmXmyID dy2PmPyMHhKLy5lAGUvNbs72UAmpE3XgEHHpbSFe6NYibOB9d0YGrfRLfO8nT/oGpfsfaB 0LVFWnWUWSkXqd/kzl+Fy9eagS6a/eM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-234-pW7L4-RDPEO_ATbyclH5qA-1; Mon, 23 Dec 2019 23:05:35 -0500 X-MC-Unique: pW7L4-RDPEO_ATbyclH5qA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A7A1E184B44F; Tue, 24 Dec 2019 04:05:34 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-64.pek2.redhat.com [10.72.12.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 021931000322; Tue, 24 Dec 2019 04:05:31 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 3/4] ceph: periodically send cap and dentry lease perf metrics to MDS Date: Mon, 23 Dec 2019 23:05:13 -0500 Message-Id: <20191224040514.26144-4-xiubli@redhat.com> In-Reply-To: <20191224040514.26144-1-xiubli@redhat.com> References: <20191224040514.26144-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li Currently only the cap and dentry lease perf metrics are support, and will send the metrics per 5 seconds. Signed-off-by: Xiubo Li --- fs/ceph/mds_client.c | 79 ++++++++++++++++++++++++++++++++++++ include/linux/ceph/ceph_fs.h | 39 ++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index f58b74b2d1ec..5b74202ed68f 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4086,6 +4086,79 @@ static void maybe_recover_session(struct ceph_mds_client *mdsc) ceph_force_reconnect(fsc->sb); } +/* + * called under s_mutex + */ +static bool ceph_mdsc_send_metrics(struct ceph_mds_client *mdsc, + struct ceph_mds_session *s, + bool skip_global) +{ + struct ceph_metric_head *head; + struct ceph_metric_cap *cap; + struct ceph_metric_dentry_lease *lease; + struct ceph_msg *msg; + s32 len = sizeof(*head) + sizeof(*cap); + s32 items = 0; + + if (!mdsc || !s) + return false; + + if (!skip_global) + len += sizeof(*lease); + + msg = ceph_msg_new(CEPH_MSG_CLIENT_METRICS, len, GFP_NOFS, true); + if (!msg) { + pr_err("send metrics to mds%d, failed to allocate message\n", + s->s_mds); + return false; + } + + head = msg->front.iov_base; + + /* encode the cap metric */ + cap = (struct ceph_metric_cap *)(head + 1); + cap->type = cpu_to_le32(CLIENT_METRIC_TYPE_CAP_INFO); + cap->ver = 1; + cap->campat = 1; + cap->data_len = cpu_to_le32(sizeof(*cap) - 6); + cap->hit = cpu_to_le64(percpu_counter_sum(&s->i_caps_hit)); + cap->mis = cpu_to_le64(percpu_counter_sum(&s->i_caps_mis)); + cap->total = cpu_to_le64(s->s_nr_caps); + items++; + + dout("cap metric type %d, hit %lld, mis %lld, total %lld", + cap->type, cap->hit, cap->mis, cap->total); + + /* only send the global once */ + if (skip_global) + goto skip_global; + + /* encode the dentry lease metric */ + lease = (struct ceph_metric_dentry_lease *)(cap + 1); + lease->type = cpu_to_le32(CLIENT_METRIC_TYPE_DENTRY_LEASE); + lease->ver = 1; + lease->campat = 1; + lease->data_len = cpu_to_le32(sizeof(*cap) - 6); + lease->hit = cpu_to_le64(percpu_counter_sum(&mdsc->metric.d_lease_hit)); + lease->mis = cpu_to_le64(percpu_counter_sum(&mdsc->metric.d_lease_mis)); + lease->total = cpu_to_le64(atomic64_read(&mdsc->metric.total_dentries)); + items++; + + dout("dentry lease metric type %d, hit %lld, mis %lld, total %lld", + lease->type, lease->hit, lease->mis, lease->total); + +skip_global: + put_unaligned_le32(items, &head->num); + msg->front.iov_len = cpu_to_le32(len); + msg->hdr.version = cpu_to_le16(1); + msg->hdr.compat_version = cpu_to_le16(1); + msg->hdr.front_len = cpu_to_le32(msg->front.iov_len); + dout("send metrics to mds%d %p\n", s->s_mds, msg); + ceph_con_send(&s->s_con, msg); + + return true; +} + /* * delayed work -- periodically trim expired leases, renew caps with mds */ @@ -4115,6 +4188,8 @@ static void delayed_work(struct work_struct *work) for (i = 0; i < mdsc->max_sessions; i++) { struct ceph_mds_session *s = __ceph_lookup_mds_session(mdsc, i); + bool g_skip = false; + if (!s) continue; if (s->s_state == CEPH_MDS_SESSION_CLOSING) { @@ -4140,6 +4215,9 @@ static void delayed_work(struct work_struct *work) mutex_unlock(&mdsc->mutex); mutex_lock(&s->s_mutex); + + g_skip = ceph_mdsc_send_metrics(mdsc, s, g_skip); + if (renew_caps) send_renew_caps(mdsc, s); else @@ -4147,6 +4225,7 @@ static void delayed_work(struct work_struct *work) if (s->s_state == CEPH_MDS_SESSION_OPEN || s->s_state == CEPH_MDS_SESSION_HUNG) ceph_send_cap_releases(mdsc, s); + mutex_unlock(&s->s_mutex); ceph_put_mds_session(s); diff --git a/include/linux/ceph/ceph_fs.h b/include/linux/ceph/ceph_fs.h index cb21c5cf12c3..32758f9a2f1d 100644 --- a/include/linux/ceph/ceph_fs.h +++ b/include/linux/ceph/ceph_fs.h @@ -130,6 +130,7 @@ struct ceph_dir_layout { #define CEPH_MSG_CLIENT_REQUEST 24 #define CEPH_MSG_CLIENT_REQUEST_FORWARD 25 #define CEPH_MSG_CLIENT_REPLY 26 +#define CEPH_MSG_CLIENT_METRICS 29 #define CEPH_MSG_CLIENT_CAPS 0x310 #define CEPH_MSG_CLIENT_LEASE 0x311 #define CEPH_MSG_CLIENT_SNAP 0x312 @@ -752,6 +753,44 @@ struct ceph_mds_lease { } __attribute__ ((packed)); /* followed by a __le32+string for dname */ +enum ceph_metric_type { + CLIENT_METRIC_TYPE_CAP_INFO, + CLIENT_METRIC_TYPE_READ_LATENCY, + CLIENT_METRIC_TYPE_WRITE_LATENCY, + CLIENT_METRIC_TYPE_METADATA_LATENCY, + CLIENT_METRIC_TYPE_DENTRY_LEASE, +}; + +/* metric caps header */ +struct ceph_metric_cap { + __le32 type; /* ceph metric type */ + + __u8 ver; + __u8 campat; + + __le32 data_len; /* length of sizeof(hit + mis + total) */ + __le64 hit; + __le64 mis; + __le64 total; +} __attribute__ ((packed)); + +/* metric caps header */ +struct ceph_metric_dentry_lease { + __le32 type; /* ceph metric type */ + + __u8 ver; + __u8 campat; + + __le32 data_len; /* length of sizeof(hit + mis + total) */ + __le64 hit; + __le64 mis; + __le64 total; +} __attribute__ ((packed)); + +struct ceph_metric_head { + __le32 num; /* the number of metrics will be sent */ +} __attribute__ ((packed)); + /* client reconnect */ struct ceph_mds_cap_reconnect { __le64 cap_id; From patchwork Tue Dec 24 04:05:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11309035 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9838C109A for ; Tue, 24 Dec 2019 04:05:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6D497206D3 for ; Tue, 24 Dec 2019 04:05:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DMfPnSp9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726897AbfLXEFq (ORCPT ); Mon, 23 Dec 2019 23:05:46 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:43356 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726747AbfLXEFq (ORCPT ); Mon, 23 Dec 2019 23:05:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1577160344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x/4wmeMUX8q5zvdacEsm6lixhzvKw+ZQRDwYhgkBqmg=; b=DMfPnSp9julzUaUroTuKMABlyqCVNIP9DEvOL8wDZsinYYBAVMejlPxlkGU6dTSK4k6z+B lEUL/Qt9ubiss+HNV6q3HSKVDPXVztWWo5BLC71TF4/CFk/7JZ3fDAQirAGoghMyPgYVtx nY4newIK5zBw2Uz0Vp+BCGL3wbY1wO0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-241-4gqGbh5kM3i5FwHNh4Kr2Q-1; Mon, 23 Dec 2019 23:05:39 -0500 X-MC-Unique: 4gqGbh5kM3i5FwHNh4Kr2Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DE6A5184B44C; Tue, 24 Dec 2019 04:05:37 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-64.pek2.redhat.com [10.72.12.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3B7911081331; Tue, 24 Dec 2019 04:05:34 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org, idryomov@gmail.com Cc: sage@redhat.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 4/4] ceph: add enable/disable sending metrics to MDS debugfs support Date: Mon, 23 Dec 2019 23:05:14 -0500 Message-Id: <20191224040514.26144-5-xiubli@redhat.com> In-Reply-To: <20191224040514.26144-1-xiubli@redhat.com> References: <20191224040514.26144-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li Disabled as default, if it's enabled the kclient will send metrics every second. Signed-off-by: Xiubo Li --- fs/ceph/debugfs.c | 44 ++++++++++++++++++++++++++++++-- fs/ceph/mds_client.c | 60 +++++++++++++++++++++++++++++++------------- fs/ceph/mds_client.h | 3 +++ fs/ceph/super.h | 1 + 4 files changed, 89 insertions(+), 19 deletions(-) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index c132fdb40d53..a26e559473fd 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -124,6 +124,40 @@ static int mdsc_show(struct seq_file *s, void *p) return 0; } +/* + * metrics debugfs + */ +static int sending_metrics_set(void *data, u64 val) +{ + struct ceph_fs_client *fsc = (struct ceph_fs_client *)data; + struct ceph_mds_client *mdsc = fsc->mdsc; + + if (val > 1) { + pr_err("Invalid sending metrics set value %llu\n", val); + return -EINVAL; + } + + mutex_lock(&mdsc->mutex); + mdsc->sending_metrics = (unsigned int)val; + mutex_unlock(&mdsc->mutex); + + return 0; +} + +static int sending_metrics_get(void *data, u64 *val) +{ + struct ceph_fs_client *fsc = (struct ceph_fs_client *)data; + struct ceph_mds_client *mdsc = fsc->mdsc; + + mutex_lock(&mdsc->mutex); + *val = (u64)mdsc->sending_metrics; + mutex_unlock(&mdsc->mutex); + + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(sending_metrics_fops, sending_metrics_get, + sending_metrics_set, "%llu\n"); + static int metric_show(struct seq_file *s, void *p) { struct ceph_fs_client *fsc = s->private; @@ -279,11 +313,9 @@ static int congestion_kb_get(void *data, u64 *val) *val = (u64)fsc->mount_options->congestion_kb; return 0; } - DEFINE_SIMPLE_ATTRIBUTE(congestion_kb_fops, congestion_kb_get, congestion_kb_set, "%llu\n"); - void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) { dout("ceph_fs_debugfs_cleanup\n"); @@ -293,6 +325,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) debugfs_remove(fsc->debugfs_mds_sessions); debugfs_remove(fsc->debugfs_caps); debugfs_remove(fsc->debugfs_metric); + debugfs_remove(fsc->debugfs_sending_metrics); debugfs_remove(fsc->debugfs_mdsc); } @@ -333,6 +366,13 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc) fsc, &mdsc_show_fops); + fsc->debugfs_sending_metrics = + debugfs_create_file_unsafe("sending_metrics", + 0600, + fsc->client->debugfs_dir, + fsc, + &sending_metrics_fops); + fsc->debugfs_metric = debugfs_create_file("metrics", 0400, fsc->client->debugfs_dir, diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 5b74202ed68f..d31612bdc1e3 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -4162,10 +4162,18 @@ static bool ceph_mdsc_send_metrics(struct ceph_mds_client *mdsc, /* * delayed work -- periodically trim expired leases, renew caps with mds */ +#define CEPH_WORK_DELAY_DEF 5 static void schedule_delayed(struct ceph_mds_client *mdsc) { - int delay = 5; - unsigned hz = round_jiffies_relative(HZ * delay); + unsigned int hz; + int delay = CEPH_WORK_DELAY_DEF; + + mutex_lock(&mdsc->mutex); + if (mdsc->sending_metrics) + delay = 1; + mutex_unlock(&mdsc->mutex); + + hz = round_jiffies_relative(HZ * delay); schedule_delayed_work(&mdsc->delayed_work, hz); } @@ -4176,15 +4184,23 @@ static void delayed_work(struct work_struct *work) container_of(work, struct ceph_mds_client, delayed_work.work); int renew_interval; int renew_caps; + bool metric_only; + bool sending_metrics; dout("mdsc delayed_work\n"); mutex_lock(&mdsc->mutex); - renew_interval = mdsc->mdsmap->m_session_timeout >> 2; - renew_caps = time_after_eq(jiffies, HZ*renew_interval + - mdsc->last_renew_caps); - if (renew_caps) - mdsc->last_renew_caps = jiffies; + sending_metrics = !!mdsc->sending_metrics; + metric_only = mdsc->sending_metrics && + (mdsc->ticks++ % CEPH_WORK_DELAY_DEF); + + if (!metric_only) { + renew_interval = mdsc->mdsmap->m_session_timeout >> 2; + renew_caps = time_after_eq(jiffies, HZ*renew_interval + + mdsc->last_renew_caps); + if (renew_caps) + mdsc->last_renew_caps = jiffies; + } for (i = 0; i < mdsc->max_sessions; i++) { struct ceph_mds_session *s = __ceph_lookup_mds_session(mdsc, i); @@ -4216,15 +4232,18 @@ static void delayed_work(struct work_struct *work) mutex_lock(&s->s_mutex); - g_skip = ceph_mdsc_send_metrics(mdsc, s, g_skip); + if (sending_metrics) + g_skip = ceph_mdsc_send_metrics(mdsc, s, g_skip); - if (renew_caps) - send_renew_caps(mdsc, s); - else - ceph_con_keepalive(&s->s_con); - if (s->s_state == CEPH_MDS_SESSION_OPEN || - s->s_state == CEPH_MDS_SESSION_HUNG) - ceph_send_cap_releases(mdsc, s); + if (!metric_only) { + if (renew_caps) + send_renew_caps(mdsc, s); + else + ceph_con_keepalive(&s->s_con); + if (s->s_state == CEPH_MDS_SESSION_OPEN || + s->s_state == CEPH_MDS_SESSION_HUNG) + ceph_send_cap_releases(mdsc, s); + } mutex_unlock(&s->s_mutex); ceph_put_mds_session(s); @@ -4233,6 +4252,9 @@ static void delayed_work(struct work_struct *work) } mutex_unlock(&mdsc->mutex); + if (metric_only) + goto delay_work; + ceph_check_delayed_caps(mdsc); ceph_queue_cap_reclaim_work(mdsc); @@ -4241,11 +4263,13 @@ static void delayed_work(struct work_struct *work) maybe_recover_session(mdsc); +delay_work: schedule_delayed(mdsc); } -static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) +static int ceph_mdsc_metric_init(struct ceph_mds_client *mdsc) { + struct ceph_client_metric *metric = &mdsc->metric; int ret; if (!metric) @@ -4259,6 +4283,8 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric) goto err; atomic64_set(&metric->total_dentries, 0); + mdsc->sending_metrics = 0; + mdsc->ticks = 0; return 0; err: @@ -4319,7 +4345,7 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) init_waitqueue_head(&mdsc->cap_flushing_wq); INIT_WORK(&mdsc->cap_reclaim_work, ceph_cap_reclaim_work); atomic_set(&mdsc->cap_reclaim_pending, 0); - err = ceph_mdsc_metric_init(&mdsc->metric); + err = ceph_mdsc_metric_init(mdsc); if (err) goto err_mdsmap; diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 6b91f20132c0..c926256fc76b 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -447,6 +447,9 @@ struct ceph_mds_client { struct list_head dentry_leases; /* fifo list */ struct list_head dentry_dir_leases; /* lru list */ + /* metrics */ + unsigned int sending_metrics; + unsigned int ticks; struct ceph_client_metric metric; spinlock_t snapid_map_lock; diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 88da9e21af75..9d2a5f1ce418 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -123,6 +123,7 @@ struct ceph_fs_client { struct dentry *debugfs_congestion_kb; struct dentry *debugfs_bdi; struct dentry *debugfs_mdsc, *debugfs_mdsmap; + struct dentry *debugfs_sending_metrics; struct dentry *debugfs_metric; struct dentry *debugfs_mds_sessions; #endif