From patchwork Fri May 31 12:28:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yan, Zheng" X-Patchwork-Id: 10970107 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9060C912 for ; Fri, 31 May 2019 12:28:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8041D28C16 for ; Fri, 31 May 2019 12:28:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 745E828CC3; Fri, 31 May 2019 12:28:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F269528C16 for ; Fri, 31 May 2019 12:28:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727254AbfEaM2R (ORCPT ); Fri, 31 May 2019 08:28:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38596 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726403AbfEaM2Q (ORCPT ); Fri, 31 May 2019 08:28:16 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 28031C0AEE59; Fri, 31 May 2019 12:28:10 +0000 (UTC) Received: from zhyan-laptop.redhat.com (ovpn-12-56.pek2.redhat.com [10.72.12.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 970EF1758B; Fri, 31 May 2019 12:28:04 +0000 (UTC) From: "Yan, Zheng" To: ceph-devel@vger.kernel.org Cc: idryomov@redhat.com, jlayton@redhat.com, lhenriques@suse.com, "Yan, Zheng" Subject: [PATCH 1/3] libceph: add function that reset client's entity addr Date: Fri, 31 May 2019 20:28:00 +0800 Message-Id: <20190531122802.12814-1-zyan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 31 May 2019 12:28:16 +0000 (UTC) Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: "Yan, Zheng" --- include/linux/ceph/libceph.h | 1 + include/linux/ceph/messenger.h | 1 + include/linux/ceph/mon_client.h | 1 + include/linux/ceph/osd_client.h | 1 + net/ceph/ceph_common.c | 8 ++++++++ net/ceph/messenger.c | 5 +++++ net/ceph/mon_client.c | 7 +++++++ net/ceph/osd_client.c | 16 ++++++++++++++++ 8 files changed, 40 insertions(+) diff --git a/include/linux/ceph/libceph.h b/include/linux/ceph/libceph.h index a3cddf5f0e60..f29959eed025 100644 --- a/include/linux/ceph/libceph.h +++ b/include/linux/ceph/libceph.h @@ -291,6 +291,7 @@ struct ceph_client *ceph_create_client(struct ceph_options *opt, void *private); struct ceph_entity_addr *ceph_client_addr(struct ceph_client *client); u64 ceph_client_gid(struct ceph_client *client); extern void ceph_destroy_client(struct ceph_client *client); +extern void ceph_reset_client_addr(struct ceph_client *client); extern int __ceph_open_session(struct ceph_client *client, unsigned long started); extern int ceph_open_session(struct ceph_client *client); diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h index 23895d178149..c4458dc6a757 100644 --- a/include/linux/ceph/messenger.h +++ b/include/linux/ceph/messenger.h @@ -337,6 +337,7 @@ extern void ceph_msgr_flush(void); extern void ceph_messenger_init(struct ceph_messenger *msgr, struct ceph_entity_addr *myaddr); extern void ceph_messenger_fini(struct ceph_messenger *msgr); +extern void ceph_messenger_reset_nonce(struct ceph_messenger *msgr); extern void ceph_con_init(struct ceph_connection *con, void *private, const struct ceph_connection_operations *ops, diff --git a/include/linux/ceph/mon_client.h b/include/linux/ceph/mon_client.h index 3a4688af7455..0d8d890c6759 100644 --- a/include/linux/ceph/mon_client.h +++ b/include/linux/ceph/mon_client.h @@ -110,6 +110,7 @@ extern int ceph_monmap_contains(struct ceph_monmap *m, extern int ceph_monc_init(struct ceph_mon_client *monc, struct ceph_client *cl); extern void ceph_monc_stop(struct ceph_mon_client *monc); +extern void ceph_monc_reopen_session(struct ceph_mon_client *monc); enum { CEPH_SUB_MONMAP = 0, diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h index 2294f963dab7..a12b7fc9cfd6 100644 --- a/include/linux/ceph/osd_client.h +++ b/include/linux/ceph/osd_client.h @@ -381,6 +381,7 @@ extern void ceph_osdc_cleanup(void); extern int ceph_osdc_init(struct ceph_osd_client *osdc, struct ceph_client *client); extern void ceph_osdc_stop(struct ceph_osd_client *osdc); +extern void ceph_osdc_reopen_osds(struct ceph_osd_client *osdc); extern void ceph_osdc_handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg); diff --git a/net/ceph/ceph_common.c b/net/ceph/ceph_common.c index 79eac465ec65..55210823d1cc 100644 --- a/net/ceph/ceph_common.c +++ b/net/ceph/ceph_common.c @@ -693,6 +693,14 @@ void ceph_destroy_client(struct ceph_client *client) } EXPORT_SYMBOL(ceph_destroy_client); +void ceph_reset_client_addr(struct ceph_client *client) +{ + ceph_messenger_reset_nonce(&client->msgr); + ceph_monc_reopen_session(&client->monc); + ceph_osdc_reopen_osds(&client->osdc); +} +EXPORT_SYMBOL(ceph_reset_client_addr); + /* * true if we have the mon map (and have thus joined the cluster) */ diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c index 3ee380758ddd..cd03a1cba849 100644 --- a/net/ceph/messenger.c +++ b/net/ceph/messenger.c @@ -3028,6 +3028,11 @@ static void con_fault(struct ceph_connection *con) } +void ceph_messenger_reset_nonce(struct ceph_messenger *msgr) +{ + msgr->inst.addr.nonce += 1000000; + encode_my_addr(msgr); +} /* * initialize a new messenger instance diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c index 895679d3529b..6dab6a94e9cc 100644 --- a/net/ceph/mon_client.c +++ b/net/ceph/mon_client.c @@ -209,6 +209,13 @@ static void reopen_session(struct ceph_mon_client *monc) __open_session(monc); } +void ceph_monc_reopen_session(struct ceph_mon_client *monc) +{ + mutex_lock(&monc->mutex); + reopen_session(monc); + mutex_unlock(&monc->mutex); +} + static void un_backoff(struct ceph_mon_client *monc) { monc->hunt_mult /= 2; /* reduce by 50% */ diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index e6d31e0f0289..67e9466f27fd 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -5089,6 +5089,22 @@ int ceph_osdc_call(struct ceph_osd_client *osdc, } EXPORT_SYMBOL(ceph_osdc_call); +/* + * reset all osd connections + */ +void ceph_osdc_reopen_osds(struct ceph_osd_client *osdc) +{ + struct rb_node *n; + down_write(&osdc->lock); + for (n = rb_first(&osdc->osds); n; ) { + struct ceph_osd *osd = rb_entry(n, struct ceph_osd, o_node); + n = rb_next(n); + if (!reopen_osd(osd)) + kick_osd_requests(osd); + } + up_write(&osdc->lock); +} + /* * init, shutdown */ From patchwork Fri May 31 12:28:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yan, Zheng" X-Patchwork-Id: 10970105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7E06912 for ; Fri, 31 May 2019 12:28:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B72C228C16 for ; Fri, 31 May 2019 12:28:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A931B28CC3; Fri, 31 May 2019 12:28:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34D1328C16 for ; Fri, 31 May 2019 12:28:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727296AbfEaM2R (ORCPT ); Fri, 31 May 2019 08:28:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50702 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726415AbfEaM2R (ORCPT ); Fri, 31 May 2019 08:28:17 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 808DBC0578FA; Fri, 31 May 2019 12:28:16 +0000 (UTC) Received: from zhyan-laptop.redhat.com (ovpn-12-56.pek2.redhat.com [10.72.12.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC75A17598; Fri, 31 May 2019 12:28:10 +0000 (UTC) From: "Yan, Zheng" To: ceph-devel@vger.kernel.org Cc: idryomov@redhat.com, jlayton@redhat.com, lhenriques@suse.com, "Yan, Zheng" Subject: [PATCH 2/3] ceph: add method that forces client to reconnect using new entity addr Date: Fri, 31 May 2019 20:28:01 +0800 Message-Id: <20190531122802.12814-2-zyan@redhat.com> In-Reply-To: <20190531122802.12814-1-zyan@redhat.com> References: <20190531122802.12814-1-zyan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Fri, 31 May 2019 12:28:16 +0000 (UTC) Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP echo force_reconnect > /sys/kernel/debug/ceph/xxx/control Signed-off-by: "Yan, Zheng" --- fs/ceph/debugfs.c | 34 +++++++++++++++++++++++++++++++++- fs/ceph/mds_client.c | 41 ++++++++++++++++++++++++++++++++++++++++- fs/ceph/mds_client.h | 1 + fs/ceph/super.h | 1 + 4 files changed, 75 insertions(+), 2 deletions(-) diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index a14d64664878..d65da57406bd 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -210,6 +210,31 @@ CEPH_DEFINE_SHOW_FUNC(mdsc_show) CEPH_DEFINE_SHOW_FUNC(caps_show) CEPH_DEFINE_SHOW_FUNC(mds_sessions_show) +static ssize_t control_file_write(struct file *file, + const char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct ceph_fs_client *fsc = file_inode(file)->i_private; + char buf[16]; + ssize_t len; + + len = min(count, sizeof(buf) - 1); + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + + buf[len] = '\0'; + if (!strcmp(buf, "force_reconnect")) { + ceph_mdsc_force_reconnect(fsc->mdsc); + } else { + return -EINVAL; + } + + return count; +} + +static const struct file_operations control_file_fops = { + .write = control_file_write, +}; /* * debugfs @@ -233,7 +258,6 @@ static int congestion_kb_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(congestion_kb_fops, congestion_kb_get, congestion_kb_set, "%llu\n"); - void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) { dout("ceph_fs_debugfs_cleanup\n"); @@ -243,6 +267,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc) debugfs_remove(fsc->debugfs_mds_sessions); debugfs_remove(fsc->debugfs_caps); debugfs_remove(fsc->debugfs_mdsc); + debugfs_remove(fsc->debugfs_control); } int ceph_fs_debugfs_init(struct ceph_fs_client *fsc) @@ -302,6 +327,13 @@ int ceph_fs_debugfs_init(struct ceph_fs_client *fsc) if (!fsc->debugfs_caps) goto out; + fsc->debugfs_control = debugfs_create_file("control", + 0200, + fsc->client->debugfs_dir, + fsc, + &control_file_fops); + if (!fsc->debugfs_control) + goto out; return 0; out: diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index f5c3499fdec6..95ee893205c5 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -2631,7 +2631,7 @@ static void kick_requests(struct ceph_mds_client *mdsc, int mds) if (req->r_attempts > 0) continue; /* only new requests */ if (req->r_session && - req->r_session->s_mds == mds) { + (mds == -1 || req->r_session->s_mds == mds)) { dout(" kicking tid %llu\n", req->r_tid); list_del_init(&req->r_wait); __do_request(mdsc, req); @@ -4371,6 +4371,45 @@ void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc) mutex_unlock(&mdsc->mutex); } +void ceph_mdsc_force_reconnect(struct ceph_mds_client *mdsc) +{ + struct ceph_mds_session *session; + int mds; + LIST_HEAD(to_wake); + + pr_info("force reconnect\n"); + + /* this also reset add mon/osd conntions */ + ceph_reset_client_addr(mdsc->fsc->client); + + mutex_lock(&mdsc->mutex); + + /* reset mds connections */ + for (mds = 0; mds < mdsc->max_sessions; mds++) { + session = __ceph_lookup_mds_session(mdsc, mds); + if (!session) + continue; + + __unregister_session(mdsc, session); + list_splice_init(&session->s_waiting, &to_wake); + mutex_unlock(&mdsc->mutex); + + mutex_lock(&session->s_mutex); + cleanup_session_requests(mdsc, session); + remove_session_caps(session); + mutex_unlock(&session->s_mutex); + + ceph_put_mds_session(session); + mutex_lock(&mdsc->mutex); + } + + list_splice_init(&mdsc->waiting_for_map, &to_wake); + __wake_requests(mdsc, &to_wake); + kick_requests(mdsc, -1); + + mutex_unlock(&mdsc->mutex); +} + static void ceph_mdsc_stop(struct ceph_mds_client *mdsc) { dout("stop\n"); diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 330769ecb601..125e26895f14 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -457,6 +457,7 @@ extern int ceph_send_msg_mds(struct ceph_mds_client *mdsc, extern int ceph_mdsc_init(struct ceph_fs_client *fsc); extern void ceph_mdsc_close_sessions(struct ceph_mds_client *mdsc); extern void ceph_mdsc_force_umount(struct ceph_mds_client *mdsc); +extern void ceph_mdsc_force_reconnect(struct ceph_mds_client *mdsc); extern void ceph_mdsc_destroy(struct ceph_fs_client *fsc); extern void ceph_mdsc_sync(struct ceph_mds_client *mdsc); diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 9c82d213a5ab..9ccb6e031988 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -118,6 +118,7 @@ struct ceph_fs_client { struct dentry *debugfs_bdi; struct dentry *debugfs_mdsc, *debugfs_mdsmap; struct dentry *debugfs_mds_sessions; + struct dentry *debugfs_control; #endif #ifdef CONFIG_CEPH_FSCACHE From patchwork Fri May 31 12:28:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yan, Zheng" X-Patchwork-Id: 10970109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C5DE912 for ; Fri, 31 May 2019 12:28:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CEB728C16 for ; Fri, 31 May 2019 12:28:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 60A2028CC3; Fri, 31 May 2019 12:28:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00BB628C16 for ; Fri, 31 May 2019 12:28:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727310AbfEaM2Y (ORCPT ); Fri, 31 May 2019 08:28:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55102 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726415AbfEaM2X (ORCPT ); Fri, 31 May 2019 08:28:23 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F3F853179B49; Fri, 31 May 2019 12:28:19 +0000 (UTC) Received: from zhyan-laptop.redhat.com (ovpn-12-56.pek2.redhat.com [10.72.12.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3025D1758B; Fri, 31 May 2019 12:28:16 +0000 (UTC) From: "Yan, Zheng" To: ceph-devel@vger.kernel.org Cc: idryomov@redhat.com, jlayton@redhat.com, lhenriques@suse.com, "Yan, Zheng" Subject: [PATCH 3/3] ceph: fix infinite loop in get_quota_realm() Date: Fri, 31 May 2019 20:28:02 +0800 Message-Id: <20190531122802.12814-3-zyan@redhat.com> In-Reply-To: <20190531122802.12814-1-zyan@redhat.com> References: <20190531122802.12814-1-zyan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Fri, 31 May 2019 12:28:23 +0000 (UTC) Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP get_quota_realm() enters infinite loop if quota inode has no caps. This can happen after client gets evicted. Signed-off-by: "Yan, Zheng" Reviewed-by: Luis Henriques --- fs/ceph/quota.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/fs/ceph/quota.c b/fs/ceph/quota.c index d629fc857450..de56dee60540 100644 --- a/fs/ceph/quota.c +++ b/fs/ceph/quota.c @@ -135,7 +135,7 @@ static struct inode *lookup_quotarealm_inode(struct ceph_mds_client *mdsc, return NULL; mutex_lock(&qri->mutex); - if (qri->inode) { + if (qri->inode && ceph_is_any_caps(qri->inode)) { /* A request has already returned the inode */ mutex_unlock(&qri->mutex); return qri->inode; @@ -146,7 +146,18 @@ static struct inode *lookup_quotarealm_inode(struct ceph_mds_client *mdsc, mutex_unlock(&qri->mutex); return NULL; } - in = ceph_lookup_inode(sb, realm->ino); + if (qri->inode) { + /* get caps */ + int ret = __ceph_do_getattr(qri->inode, NULL, + CEPH_STAT_CAP_INODE, true); + if (ret >= 0) + in = qri->inode; + else + in = ERR_PTR(ret); + } else { + in = ceph_lookup_inode(sb, realm->ino); + } + if (IS_ERR(in)) { pr_warn("Can't lookup inode %llx (err: %ld)\n", realm->ino, PTR_ERR(in));