From patchwork Thu May 24 03:27:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dongsheng Yang X-Patchwork-Id: 10422661 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 48EC8602D8 for ; Thu, 24 May 2018 03:27:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22C3329301 for ; Thu, 24 May 2018 03:27:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1257B29303; Thu, 24 May 2018 03:27:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FC5D29301 for ; Thu, 24 May 2018 03:27:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935290AbeEXD1S (ORCPT ); Wed, 23 May 2018 23:27:18 -0400 Received: from m97138.mail.qiye.163.com ([220.181.97.138]:11132 "EHLO m97138.mail.qiye.163.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935233AbeEXD1R (ORCPT ); Wed, 23 May 2018 23:27:17 -0400 Received: from atest-guest.localdomain (unknown [218.94.118.90]) by smtp9 (Coremail) with SMTP id u+CowADH_JESMQZbzioEAA--.89S2; Thu, 24 May 2018 11:27:14 +0800 (CST) From: Dongsheng Yang To: ceph-devel@vger.kernel.org, idryomov@gmail.com Cc: Dongsheng Yang Subject: [PATCH] ceph: make osd_request_timeout changable online in debugfs Date: Wed, 23 May 2018 23:27:00 -0400 Message-Id: <1527132420-10740-1-git-send-email-dongsheng.yang@easystack.cn> X-Mailer: git-send-email 1.8.3.1 X-CM-TRANSID: u+CowADH_JESMQZbzioEAA--.89S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxXr45AF1fGFyxXr15CFy5CFg_yoWrGFWrpa 1qkas5AF1kJF13GrZa9a4DAr1S9rs7Zr47XryDuw45C3Z5JrnYqF10k3sYqry7uFW8J3Wq gF1FgFWqgr1jvwUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0Jb6dgXUUUUU= X-Originating-IP: [218.94.118.90] X-CM-SenderInfo: 5grqw2pkhqwhp1dqwq5hdv52pwdfyhdfq/1tbiexJQelrpMULvoQAAsN Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Default value of osd_request_timeout is 0 means never timeout, and we can set this value in rbd mapping with -o "osd_request_timeout=XX". But we can't change this value online. [Question 1]: Why we need to set osd_request_timeout? When we are going to reboot a node which have krbd devices mapped, even with the rbdmap.service enabled, we will be blocked in shuting down if the ceph cluster is not working. Especially, If we have three controller nodes which is running as ceph mon, but at the same time, there are some k8s pod with krbd devices on this nodes, then we can't shut down the last controller node when we want to shutdown all nodes, because when we are going to shutdown last controller node, the ceph cluster is not reachable. [Question 2]: Why don't we use rbd map -o "osd_request_timeout=XX"? We don't want to set the osd_request_timeout in rbd device whole lifecycle, there would be some problem in networking or cluster recovery to make the request timeout. This would make the fs readonly and application down. [Question 3]: How can this patch solve this problems? With this patch, we can map rbd device with default value of osd_reques_timeout, means never timeout, then we can solve the problem mentioned Question 2. At the same time we can set the osd_request_timeout to what we need, in system shuting down, for example, we can do this in rbdmap.service. then we can make sure we can shutdown or reboot host normally no matter ceph cluster is working well or not. This can solve the problem mentioned in Question 1. Signed-off-by: Dongsheng Yang --- include/linux/ceph/libceph.h | 1 + net/ceph/debugfs.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/ceph/libceph.h b/include/linux/ceph/libceph.h index 9ce689a..8fdea50 100644 --- a/include/linux/ceph/libceph.h +++ b/include/linux/ceph/libceph.h @@ -133,6 +133,7 @@ struct ceph_client { struct dentry *debugfs_monmap; struct dentry *debugfs_osdmap; struct dentry *debugfs_options; + struct dentry *debugfs_osd_req_timeout; #endif }; diff --git a/net/ceph/debugfs.c b/net/ceph/debugfs.c index 46f6570..2b6cae3 100644 --- a/net/ceph/debugfs.c +++ b/net/ceph/debugfs.c @@ -389,6 +389,28 @@ static int client_options_show(struct seq_file *s, void *p) CEPH_DEFINE_SHOW_FUNC(osdc_show) CEPH_DEFINE_SHOW_FUNC(client_options_show) +static int osd_request_timeout_get(void *data, u64 *val) +{ + struct ceph_client *client = data; + + *val = client->options->osd_request_timeout; + return 0; +} + +static int osd_request_timeout_set(void *data, u64 val) +{ + struct ceph_client *client = data; + + client->options->osd_request_timeout = val; + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(client_osd_req_timeout_fops, + osd_request_timeout_get, + osd_request_timeout_set, + "%lld\n"); + + int __init ceph_debugfs_init(void) { ceph_debugfs_dir = debugfs_create_dir("ceph", NULL); @@ -457,6 +479,14 @@ int ceph_debugfs_client_init(struct ceph_client *client) if (!client->debugfs_options) goto out; + client->debugfs_osd_req_timeout = debugfs_create_file("osd_request_timeout", + 0600, + client->debugfs_dir, + client, + &client_osd_req_timeout_fops); + if (!client->debugfs_osd_req_timeout) + goto out; + return 0; out: @@ -467,6 +497,7 @@ int ceph_debugfs_client_init(struct ceph_client *client) void ceph_debugfs_client_cleanup(struct ceph_client *client) { dout("ceph_debugfs_client_cleanup %p\n", client); + debugfs_remove(client->debugfs_osd_req_timeout); debugfs_remove(client->debugfs_options); debugfs_remove(client->debugfs_osdmap); debugfs_remove(client->debugfs_monmap);