From patchwork Thu Jun 3 22:59:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Olga Kornievskaia X-Patchwork-Id: 12298371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F63C47082 for ; Thu, 3 Jun 2021 22:59:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 04A4F6109F for ; Thu, 3 Jun 2021 22:59:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229721AbhFCXA6 (ORCPT ); Thu, 3 Jun 2021 19:00:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229610AbhFCXA6 (ORCPT ); Thu, 3 Jun 2021 19:00:58 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21130C06174A for ; Thu, 3 Jun 2021 15:59:13 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id m6so4089048qvg.1 for ; Thu, 03 Jun 2021 15:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UlQsP7v7vMM8cBzMOBEgavYrD1ix6NnsgDsDx9brgZ0=; b=KDmjBNE/tjG1d7egD0GKdOAlvqxfmV4y5qmwAeLUeKNy32rhfPfXykYM8zyScd9RyI gsIln/0ZgPjOYG45Kw8KXvHZ9CZJct+TGXSa17K71y9/y3/9qeKadCnmT1GoT2YoLQCS Heo/odmb9h3qXi5lzW8jKm6O4AdnzDLLXkEQE6onjYCZKXPfNbUVhqEnHLVDZAbZigiQ QzHcuh8rHAVJsMx2kHQ/YFt62nVcBrdAWQDhoxE6/V28cqXNcjFLOLtEkqHNraCzlvt9 EvUoKw0Zd9mfdLety4IZNRJA7/Rk47xCv8NmVnkh6UZ5pekH1YX1ip+YNmHF8tokZ/iL oiZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UlQsP7v7vMM8cBzMOBEgavYrD1ix6NnsgDsDx9brgZ0=; b=QNGa0KOLD1rjGY7lvboglnTLV2BspgpS7Vk2QD1y+JNbpIGNmq6bcThglTIY6nt/TX Tjrq6u6cqOTQVyIOVqLJpKdOjrveBB7Zh12jfviXpGyVGfczd7tTYfp9vW046HeHDluZ 3Ep5KiVOvpCM2v73YRZghZAv1FeJbwp6qUs7oUIP5wU9YVamd1G87TA3TBUBwuN9b80D cFFaTRBjjO7Gq+Vp/kJZhjC4artWTDl45wS45XWqrjY9uQw1oAk2GWLUPgsNL1J4Xfau Cgc/4R0qTCFJwlGFQHRe1N6VGwWeeybplDP6b6TiOIO3tlIoP52cjQR2sHxMfC9lZ79u ChqQ== X-Gm-Message-State: AOAM533Y6MuQRm5tytILfTqbzUAYnqndtbWQL1ezJ9ghU7XPQBOT5aYx VQEZL+eXCWmm+EL6rTDbQsQgz6rrAF0= X-Google-Smtp-Source: ABdhPJwfBngg2oVDG4OdjS+Sa8NffJhgAdQ+fbbUB3x8zauv2gkNUSOIUXDeCVpBmuoLyjUc6VGR6w== X-Received: by 2002:a05:6214:c2d:: with SMTP id a13mr1970952qvd.37.1622761151260; Thu, 03 Jun 2021 15:59:11 -0700 (PDT) Received: from kolga-mac-1.vpn.netapp.com (nat-216-240-30-23.netapp.com. [216.240.30.23]) by smtp.gmail.com with ESMTPSA id h19sm1479497qtq.5.2021.06.03.15.59.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jun 2021 15:59:10 -0700 (PDT) From: Olga Kornievskaia To: trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH v1 1/3] sunrpc: take a xprt offline using sysfs Date: Thu, 3 Jun 2021 18:59:05 -0400 Message-Id: <20210603225907.19981-2-olga.kornievskaia@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603225907.19981-1-olga.kornievskaia@gmail.com> References: <20210603225907.19981-1-olga.kornievskaia@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Olga Kornievskaia Using sysfs's xprt_state attribute, mark a particular transport offline. It will not be picked during the round-robin selection. It's not allowed to take the main (1st created transport associated with the rpc_client) offline. Signed-off-by: Olga Kornievskaia --- include/linux/sunrpc/xprt.h | 2 ++ net/sunrpc/clnt.c | 1 + net/sunrpc/sysfs.c | 42 +++++++++++++++++++++++++++++++++---- net/sunrpc/xprtmultipath.c | 3 ++- 4 files changed, 43 insertions(+), 5 deletions(-) diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index 13a4eaf385cf..72a858f032c7 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -293,6 +293,7 @@ struct rpc_xprt { struct rcu_head rcu; const struct xprt_class *xprt_class; struct rpc_sysfs_xprt *xprt_sysfs; + bool main; /* marked if it's the 1st transport */ }; #if defined(CONFIG_SUNRPC_BACKCHANNEL) @@ -426,6 +427,7 @@ void xprt_release_write(struct rpc_xprt *, struct rpc_task *); #define XPRT_BOUND (4) #define XPRT_BINDING (5) #define XPRT_CLOSING (6) +#define XPRT_OFFLINE (7) #define XPRT_CONGESTED (9) #define XPRT_CWND_WAIT (10) #define XPRT_WRITE_SPACE (11) diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index 9bf820bad84c..408618765aa5 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -412,6 +412,7 @@ static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args, } rpc_clnt_set_transport(clnt, xprt, timeout); + xprt->main = true; xprt_iter_init(&clnt->cl_xpi, xps); xprt_switch_put(xps); diff --git a/net/sunrpc/sysfs.c b/net/sunrpc/sysfs.c index ec06c9257c07..02c918c5061b 100644 --- a/net/sunrpc/sysfs.c +++ b/net/sunrpc/sysfs.c @@ -118,7 +118,7 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, struct rpc_xprt *xprt = rpc_sysfs_xprt_kobj_get_xprt(kobj); ssize_t ret; int locked, connected, connecting, close_wait, bound, binding, - closing, congested, cwnd_wait, write_space; + closing, congested, cwnd_wait, write_space, offline; if (!xprt) return 0; @@ -136,8 +136,9 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, congested = test_bit(XPRT_CONGESTED, &xprt->state); cwnd_wait = test_bit(XPRT_CWND_WAIT, &xprt->state); write_space = test_bit(XPRT_WRITE_SPACE, &xprt->state); + offline = test_bit(XPRT_OFFLINE, &xprt->state); - ret = sprintf(buf, "state=%s %s %s %s %s %s %s %s %s %s\n", + ret = sprintf(buf, "state=%s %s %s %s %s %s %s %s %s %s %s\n", locked ? "LOCKED" : "", connected ? "CONNECTED" : "", connecting ? "CONNECTING" : "", @@ -147,7 +148,8 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, closing ? "CLOSING" : "", congested ? "CONGESTED" : "", cwnd_wait ? "CWND_WAIT" : "", - write_space ? "WRITE_SPACE" : ""); + write_space ? "WRITE_SPACE" : "", + offline ? "OFFLINE" : ""); } xprt_put(xprt); @@ -223,6 +225,38 @@ static ssize_t rpc_sysfs_xprt_dstaddr_store(struct kobject *kobj, goto out; } +static ssize_t rpc_sysfs_xprt_state_change(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct rpc_xprt *xprt = rpc_sysfs_xprt_kobj_get_xprt(kobj); + int offline = 0; + + if (!xprt) + return 0; + + if (!strncmp(buf, "offline", 7)) + offline = 1; + else + return -EINVAL; + + if (wait_on_bit_lock(&xprt->state, XPRT_LOCKED, TASK_KILLABLE)) { + count = -EINTR; + goto out_put; + } + if (offline) { + if (xprt->main) + count = -EINVAL; + else + set_bit(XPRT_OFFLINE, &xprt->state); + } + + xprt_release_write(xprt, NULL); +out_put: + xprt_put(xprt); + return count; +} + int rpc_sysfs_init(void) { rpc_sunrpc_kset = kset_create_and_add("sunrpc", NULL, kernel_kobj); @@ -293,7 +327,7 @@ static struct kobj_attribute rpc_sysfs_xprt_info = __ATTR(xprt_info, 0444, rpc_sysfs_xprt_info_show, NULL); static struct kobj_attribute rpc_sysfs_xprt_change_state = __ATTR(xprt_state, - 0644, rpc_sysfs_xprt_state_show, NULL); + 0644, rpc_sysfs_xprt_state_show, rpc_sysfs_xprt_state_change); static struct attribute *rpc_sysfs_xprt_attrs[] = { &rpc_sysfs_xprt_dstaddr.attr, diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c index 07e76ae1028a..39551b794b80 100644 --- a/net/sunrpc/xprtmultipath.c +++ b/net/sunrpc/xprtmultipath.c @@ -230,7 +230,8 @@ void xprt_iter_default_rewind(struct rpc_xprt_iter *xpi) static bool xprt_is_active(const struct rpc_xprt *xprt) { - return kref_read(&xprt->kref) != 0; + return (kref_read(&xprt->kref) != 0 && + !test_bit(XPRT_OFFLINE, &xprt->state)); } static From patchwork Thu Jun 3 22:59:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Olga Kornievskaia X-Patchwork-Id: 12298375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97CF6C47096 for ; Thu, 3 Jun 2021 23:00:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7582E6109F for ; Thu, 3 Jun 2021 23:00:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229610AbhFCXCI (ORCPT ); Thu, 3 Jun 2021 19:02:08 -0400 Received: from mail-qt1-f172.google.com ([209.85.160.172]:44755 "EHLO mail-qt1-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbhFCXCH (ORCPT ); Thu, 3 Jun 2021 19:02:07 -0400 Received: by mail-qt1-f172.google.com with SMTP id t17so5627056qta.11 for ; Thu, 03 Jun 2021 16:00:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n58lc8G0mdEq7LF3zlLfa7Uh+6yQYWsYuOuqcdnDFK4=; b=lSWw7xzZeRqANBG8uNKJ2F/rWADktusI6FlTydtxOOH9zfp2wRCAAvZQ1cmL6RUuuy 7bw72Z8ZJpGR6zepvxnlRRS3Adf0hxaDUkyNsRbNysihhjfxyLfA07B98hc9+Hq8mHzs 8+56lUfuQCuCPM/Gpll72pg7Wa/H/kCndv48I+gLp12ittjMpbmRj4JzaR6alEjJZk5X dgeEi1plt/6Taas8ZnmzSbxKxxCSVMxFTmMIBM/kjbK4hV2uzlpa4qa0gxNXZqyZPTn6 3xUDapmKcPRVWDQC++NPuZRRglXUI8EGqtuSW+1pCBd7KMFjZBCbE27rEhIoV4lZlcKZ dm/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n58lc8G0mdEq7LF3zlLfa7Uh+6yQYWsYuOuqcdnDFK4=; b=S1MtS4oDxqkhxTRA6lq9iwacSTZzfoYQ27wLiK+2R9EJNAwU5oPx1WOxk9Qsj5V7iW 1BaRsj1q1ayITsoi6wxSqopbfbOTnEcSXVTX3m/L+ebbZ5fmIYu/EhhKAWrwMmfX7gzx 9QqIhvT1RIUx0KV503Nl/Q2R5LAYGUA4bRxclqvTnrRZWgw7DohqFPGUsCxc0omKS/xa 5OqalhA8Ddpm8YKHgvrowWJglljAgKwF+58fs7NM5bQ2QbgPDmE5xh5HFPDjp/2q63n+ FHrrxVxcbDPagla2dsDQJ1vI6W2U04dkmhnmA4HSyENXgUIBK5Q1+u7XmfvyYEtPSH3c OXIw== X-Gm-Message-State: AOAM531cSpOun9Ys3cV8Kw+wdKfKExcQkyRL54cX2pQ/T2Iu7J//yap5 myaP9L6Neongz3pyJ2vm6k0= X-Google-Smtp-Source: ABdhPJwoCkxDItECau2UvYGKkXki2rVisvNKjCJI0b0nt+Wku++TCNO9NkghXhkJbTwu1ke0LP91+w== X-Received: by 2002:ac8:70cf:: with SMTP id g15mr1868462qtp.360.1622761152449; Thu, 03 Jun 2021 15:59:12 -0700 (PDT) Received: from kolga-mac-1.vpn.netapp.com (nat-216-240-30-23.netapp.com. [216.240.30.23]) by smtp.gmail.com with ESMTPSA id h19sm1479497qtq.5.2021.06.03.15.59.11 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jun 2021 15:59:12 -0700 (PDT) From: Olga Kornievskaia To: trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH v1 2/3] NFSv4.1 identify and mark RPC tasks that can move between transports Date: Thu, 3 Jun 2021 18:59:06 -0400 Message-Id: <20210603225907.19981-3-olga.kornievskaia@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603225907.19981-1-olga.kornievskaia@gmail.com> References: <20210603225907.19981-1-olga.kornievskaia@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Olga Kornievskaia In preparation for when we can re-try a task on a different transport, identify and mark such RPC tasks as moveable. Only 4.1+ operarations can be re-tried on a different transport. Signed-off-by: Olga Kornievskaia --- fs/nfs/nfs4proc.c | 38 +++++++++++++++++++++++++++++++----- fs/nfs/pagelist.c | 8 ++++++-- fs/nfs/write.c | 6 +++++- include/linux/sunrpc/sched.h | 2 ++ 4 files changed, 46 insertions(+), 8 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index e25c16257545..40e8d36cbfa8 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -1155,7 +1155,11 @@ static int nfs4_call_sync_sequence(struct rpc_clnt *clnt, struct nfs4_sequence_args *args, struct nfs4_sequence_res *res) { - return nfs4_do_call_sync(clnt, server, msg, args, res, 0); + unsigned short task_flags = 0; + + if (server->nfs_client->cl_minorversion) + task_flags = RPC_TASK_MOVEABLE; + return nfs4_do_call_sync(clnt, server, msg, args, res, task_flags); } @@ -2569,6 +2573,9 @@ static int nfs4_run_open_task(struct nfs4_opendata *data, }; int status; + if (server->nfs_client->cl_minorversion) + task_setup_data.flags |= RPC_TASK_MOVEABLE; + kref_get(&data->kref); data->rpc_done = false; data->rpc_status = 0; @@ -3749,6 +3756,9 @@ int nfs4_do_close(struct nfs4_state *state, gfp_t gfp_mask, int wait) }; int status = -ENOMEM; + if (server->nfs_client->cl_minorversion) + task_setup_data.flags |= RPC_TASK_MOVEABLE; + nfs4_state_protect(server->nfs_client, NFS_SP4_MACH_CRED_CLEANUP, &task_setup_data.rpc_client, &msg); @@ -4188,6 +4198,9 @@ static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle, }; unsigned short task_flags = 0; + if (server->nfs_client->cl_minorversion) + task_flags = RPC_TASK_MOVEABLE; + /* Is this is an attribute revalidation, subject to softreval? */ if (inode && (server->flags & NFS_MOUNT_SOFTREVAL)) task_flags |= RPC_TASK_TIMEOUT; @@ -4307,6 +4320,9 @@ static int _nfs4_proc_lookup(struct rpc_clnt *clnt, struct inode *dir, }; unsigned short task_flags = 0; + if (server->nfs_client->cl_minorversion) + task_flags = RPC_TASK_MOVEABLE; + /* Is this is an attribute revalidation, subject to softreval? */ if (nfs_lookup_is_soft_revalidate(dentry)) task_flags |= RPC_TASK_TIMEOUT; @@ -6538,7 +6554,7 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred, .rpc_client = server->client, .rpc_message = &msg, .callback_ops = &nfs4_delegreturn_ops, - .flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT, + .flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT | RPC_TASK_MOVEABLE, }; int status = 0; @@ -6856,6 +6872,11 @@ static struct rpc_task *nfs4_do_unlck(struct file_lock *fl, .workqueue = nfsiod_workqueue, .flags = RPC_TASK_ASYNC, }; + struct nfs_client *client = + NFS_SERVER(lsp->ls_state->inode)->nfs_client; + + if (client->cl_minorversion) + task_setup_data.flags |= RPC_TASK_MOVEABLE; nfs4_state_protect(NFS_SERVER(lsp->ls_state->inode)->nfs_client, NFS_SP4_MACH_CRED_CLEANUP, &task_setup_data.rpc_client, &msg); @@ -7130,6 +7151,10 @@ static int _nfs4_do_setlk(struct nfs4_state *state, int cmd, struct file_lock *f .flags = RPC_TASK_ASYNC | RPC_TASK_CRED_NOREF, }; int ret; + struct nfs_client *client = NFS_SERVER(state->inode)->nfs_client; + + if (client->cl_minorversion) + task_setup_data.flags |= RPC_TASK_MOVEABLE; dprintk("%s: begin!\n", __func__); data = nfs4_alloc_lockdata(fl, nfs_file_open_context(fl->fl_file), @@ -9186,7 +9211,7 @@ static struct rpc_task *_nfs41_proc_sequence(struct nfs_client *clp, .rpc_client = clp->cl_rpcclient, .rpc_message = &msg, .callback_ops = &nfs41_sequence_ops, - .flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT, + .flags = RPC_TASK_ASYNC | RPC_TASK_TIMEOUT | RPC_TASK_MOVEABLE, }; struct rpc_task *ret; @@ -9509,7 +9534,8 @@ nfs4_proc_layoutget(struct nfs4_layoutget *lgp, long *timeout) .rpc_message = &msg, .callback_ops = &nfs4_layoutget_call_ops, .callback_data = lgp, - .flags = RPC_TASK_ASYNC | RPC_TASK_CRED_NOREF, + .flags = RPC_TASK_ASYNC | RPC_TASK_CRED_NOREF | + RPC_TASK_MOVEABLE, }; struct pnfs_layout_segment *lseg = NULL; struct nfs4_exception exception = { @@ -9650,6 +9676,7 @@ int nfs4_proc_layoutreturn(struct nfs4_layoutreturn *lrp, bool sync) .rpc_message = &msg, .callback_ops = &nfs4_layoutreturn_call_ops, .callback_data = lrp, + .flags = RPC_TASK_MOVEABLE, }; int status = 0; @@ -9804,6 +9831,7 @@ nfs4_proc_layoutcommit(struct nfs4_layoutcommit_data *data, bool sync) .rpc_message = &msg, .callback_ops = &nfs4_layoutcommit_ops, .callback_data = data, + .flags = RPC_TASK_MOVEABLE, }; struct rpc_task *task; int status = 0; @@ -10131,7 +10159,7 @@ static int nfs41_free_stateid(struct nfs_server *server, .rpc_client = server->client, .rpc_message = &msg, .callback_ops = &nfs41_free_stateid_ops, - .flags = RPC_TASK_ASYNC, + .flags = RPC_TASK_ASYNC | RPC_TASK_MOVEABLE, }; struct nfs_free_stateid_data *data; struct rpc_task *task; diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c index cf9cc62ec48e..cc232d1f16f2 100644 --- a/fs/nfs/pagelist.c +++ b/fs/nfs/pagelist.c @@ -954,6 +954,7 @@ static int nfs_generic_pg_pgios(struct nfs_pageio_descriptor *desc) { struct nfs_pgio_header *hdr; int ret; + unsigned short task_flags = 0; hdr = nfs_pgio_header_alloc(desc->pg_rw_ops); if (!hdr) { @@ -962,14 +963,17 @@ static int nfs_generic_pg_pgios(struct nfs_pageio_descriptor *desc) } nfs_pgheader_init(desc, hdr, nfs_pgio_header_free); ret = nfs_generic_pgio(desc, hdr); - if (ret == 0) + if (ret == 0) { + if (NFS_SERVER(hdr->inode)->nfs_client->cl_minorversion) + task_flags = RPC_TASK_MOVEABLE; ret = nfs_initiate_pgio(NFS_CLIENT(hdr->inode), hdr, hdr->cred, NFS_PROTO(hdr->inode), desc->pg_rpc_callops, desc->pg_ioflags, - RPC_TASK_CRED_NOREF); + RPC_TASK_CRED_NOREF | task_flags); + } return ret; } diff --git a/fs/nfs/write.c b/fs/nfs/write.c index 3bf82178166a..eae9bf114041 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -1810,6 +1810,7 @@ nfs_commit_list(struct inode *inode, struct list_head *head, int how, struct nfs_commit_info *cinfo) { struct nfs_commit_data *data; + unsigned short task_flags = 0; /* another commit raced with us */ if (list_empty(head)) @@ -1820,8 +1821,11 @@ nfs_commit_list(struct inode *inode, struct list_head *head, int how, /* Set up the argument struct */ nfs_init_commit(data, head, NULL, cinfo); atomic_inc(&cinfo->mds->rpcs_out); + if (NFS_SERVER(inode)->nfs_client->cl_minorversion) + task_flags = RPC_TASK_MOVEABLE; return nfs_initiate_commit(NFS_CLIENT(inode), data, NFS_PROTO(inode), - data->mds_ops, how, RPC_TASK_CRED_NOREF); + data->mds_ops, how, + RPC_TASK_CRED_NOREF | task_flags); } /* diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h index df696efdd675..a237b8dbf608 100644 --- a/include/linux/sunrpc/sched.h +++ b/include/linux/sunrpc/sched.h @@ -121,6 +121,7 @@ struct rpc_task_setup { */ #define RPC_TASK_ASYNC 0x0001 /* is an async task */ #define RPC_TASK_SWAPPER 0x0002 /* is swapping in/out */ +#define RPC_TASK_MOVEABLE 0x0004 /* nfs4.1+ rpc tasks */ #define RPC_TASK_NULLCREDS 0x0010 /* Use AUTH_NULL credential */ #define RPC_CALL_MAJORSEEN 0x0020 /* major timeout seen */ #define RPC_TASK_ROOTCREDS 0x0040 /* force root creds */ @@ -139,6 +140,7 @@ struct rpc_task_setup { #define RPC_IS_SOFT(t) ((t)->tk_flags & (RPC_TASK_SOFT|RPC_TASK_TIMEOUT)) #define RPC_IS_SOFTCONN(t) ((t)->tk_flags & RPC_TASK_SOFTCONN) #define RPC_WAS_SENT(t) ((t)->tk_flags & RPC_TASK_SENT) +#define RPC_IS_MOVEABLE(t) ((t)->tk_flags & RPC_TASK_MOVEABLE) #define RPC_TASK_RUNNING 0 #define RPC_TASK_QUEUED 1 From patchwork Thu Jun 3 22:59:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Olga Kornievskaia X-Patchwork-Id: 12298373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61717C47096 for ; Thu, 3 Jun 2021 23:00:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 356786109F for ; Thu, 3 Jun 2021 23:00:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229675AbhFCXB7 (ORCPT ); Thu, 3 Jun 2021 19:01:59 -0400 Received: from mail-qt1-f180.google.com ([209.85.160.180]:37491 "EHLO mail-qt1-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229610AbhFCXB7 (ORCPT ); Thu, 3 Jun 2021 19:01:59 -0400 Received: by mail-qt1-f180.google.com with SMTP id z4so2609547qts.4 for ; Thu, 03 Jun 2021 16:00:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f9/bK+BrZI+CvpPuTxzlZKcrMOYWFH6TlB5Awi0+0VI=; b=VrwvoBxZtI1n5fYafcK2MWgadthXtzHCnbDMXsDog704vjWW4ydK9ec3cli7YsfQRo esN3jUz/d5rt6JhQs09UYlooeRokGYYTO4uMVNMolB3VrxxKw5W2o6cv/XPwg6DHKpZz 3vzqPF73Ho8htMvgIC5CVxb2/WguhRaB9c67CkqpqB0tyQIQnUMixdvOjVSnumiijcZT Nd1ov7M2sVgXj+F6eYO+RU0vKM4tPWDm3fajNNVcdo18V80XGaQ3TwdTFsRSH7Mwrp+Q lq7tFTimbZCPW63hs7SVQus1n6iXH210Fk3JSHOfVwCUwjnKdKL5qLHR3jJfSTAvJdvi mTqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f9/bK+BrZI+CvpPuTxzlZKcrMOYWFH6TlB5Awi0+0VI=; b=GfS5LefX0WtkahEMPSwI78T1QoUvQmJYWyEzt3HTunCrZ8M8D3DSVbpNQXRREGN+py 8WVxBnM8AG3mIQPpqx3f75ZfhDoJeqDKs2SVM5C+HVmKIrZl6Mh276aJztXj3gtXzZM7 Ldo5SbIds491KvqNgInYR3Z6ssLAlv1fNYIVYxdY2ffWGlGwgRToE3EnpKbPiQMUEWNW ls+3QyNCGuhSjuaWC0O8VQ8xwI2zqumaXrwlTpcXaX9+wWz2YEOqLLx+ghRAa9Hi7jh7 Ilxc5t1U9VoLSGqMKEqOYA2OIQ2b3j/yEIn6ejLr1PHCQ/PeQ0lkjaSEfY+2/n3j4rEN bsMA== X-Gm-Message-State: AOAM532kYTw5nAp7kguD+PKKddovSDvZXsKn+IbvMkq+oUoCUG7OC2eS EKS07YG3X9V8rizhoyAKC3aepaojnv0= X-Google-Smtp-Source: ABdhPJxO+b7TeKGCbKTUIgxK+Se3ffVISCipesrFKl52FGXazWEeQGF90/GUq6LyAaALa4rDjgu0sQ== X-Received: by 2002:ac8:5c8e:: with SMTP id r14mr1896687qta.248.1622761153575; Thu, 03 Jun 2021 15:59:13 -0700 (PDT) Received: from kolga-mac-1.vpn.netapp.com (nat-216-240-30-23.netapp.com. [216.240.30.23]) by smtp.gmail.com with ESMTPSA id h19sm1479497qtq.5.2021.06.03.15.59.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jun 2021 15:59:13 -0700 (PDT) From: Olga Kornievskaia To: trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH v1 3/3] sunrpc: remove an offlined xprt using sysfs Date: Thu, 3 Jun 2021 18:59:07 -0400 Message-Id: <20210603225907.19981-4-olga.kornievskaia@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210603225907.19981-1-olga.kornievskaia@gmail.com> References: <20210603225907.19981-1-olga.kornievskaia@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Olga Kornievskaia Once a transport has been put offline, this transport can be also removed from the list of transports. Any tasks that have been stuck on this transport would find the next available active transport and be re-tried. This transport would be removed from the xprt_switch list and freed. Signed-off-by: Olga Kornievskaia --- include/linux/sunrpc/xprt.h | 1 + net/sunrpc/clnt.c | 20 ++++++++++++++++++++ net/sunrpc/sysfs.c | 18 ++++++++++++++---- 3 files changed, 35 insertions(+), 4 deletions(-) diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index 72a858f032c7..c2ecf5cc3802 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -428,6 +428,7 @@ void xprt_release_write(struct rpc_xprt *, struct rpc_task *); #define XPRT_BINDING (5) #define XPRT_CLOSING (6) #define XPRT_OFFLINE (7) +#define XPRT_REMOVE (8) #define XPRT_CONGESTED (9) #define XPRT_CWND_WAIT (10) #define XPRT_WRITE_SPACE (11) diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index 408618765aa5..36040ec53404 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -2106,6 +2106,26 @@ call_connect_status(struct rpc_task *task) case -ENOTCONN: case -EAGAIN: case -ETIMEDOUT: + if (!(task->tk_flags & RPC_TASK_NO_ROUND_ROBIN) && + (task->tk_flags & RPC_TASK_MOVEABLE) && + test_bit(XPRT_REMOVE, &xprt->state)) { + struct rpc_xprt *saved = task->tk_xprt; + struct rpc_xprt_switch *xps; + + rcu_read_lock(); + xps = xprt_switch_get(rcu_dereference(clnt->cl_xpi.xpi_xpswitch)); + rcu_read_unlock(); + if (xps->xps_nactive > 1) { + xprt_release(task); + xprt_put(saved); + rpc_xprt_switch_remove_xprt(xps, saved); + task->tk_xprt = NULL; + task->tk_action = call_start; + } + xprt_switch_put(xps); + if (!task->tk_xprt) + return; + } goto out_retry; case -ENOBUFS: rpc_delay(task, HZ >> 2); diff --git a/net/sunrpc/sysfs.c b/net/sunrpc/sysfs.c index 02c918c5061b..44a69638c55f 100644 --- a/net/sunrpc/sysfs.c +++ b/net/sunrpc/sysfs.c @@ -118,7 +118,7 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, struct rpc_xprt *xprt = rpc_sysfs_xprt_kobj_get_xprt(kobj); ssize_t ret; int locked, connected, connecting, close_wait, bound, binding, - closing, congested, cwnd_wait, write_space, offline; + closing, congested, cwnd_wait, write_space, offline, remove; if (!xprt) return 0; @@ -137,8 +137,9 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, cwnd_wait = test_bit(XPRT_CWND_WAIT, &xprt->state); write_space = test_bit(XPRT_WRITE_SPACE, &xprt->state); offline = test_bit(XPRT_OFFLINE, &xprt->state); + remove = test_bit(XPRT_REMOVE, &xprt->state); - ret = sprintf(buf, "state=%s %s %s %s %s %s %s %s %s %s %s\n", + ret = sprintf(buf, "state=%s %s %s %s %s %s %s %s %s %s %s %s\n", locked ? "LOCKED" : "", connected ? "CONNECTED" : "", connecting ? "CONNECTING" : "", @@ -149,7 +150,8 @@ static ssize_t rpc_sysfs_xprt_state_show(struct kobject *kobj, congested ? "CONGESTED" : "", cwnd_wait ? "CWND_WAIT" : "", write_space ? "WRITE_SPACE" : "", - offline ? "OFFLINE" : ""); + offline ? "OFFLINE" : "", + remove ? "REMOVE" : ""); } xprt_put(xprt); @@ -230,13 +232,15 @@ static ssize_t rpc_sysfs_xprt_state_change(struct kobject *kobj, const char *buf, size_t count) { struct rpc_xprt *xprt = rpc_sysfs_xprt_kobj_get_xprt(kobj); - int offline = 0; + int offline = 0, remove = 0; if (!xprt) return 0; if (!strncmp(buf, "offline", 7)) offline = 1; + else if (!strncmp(buf, "remove", 6)) + remove = 1; else return -EINVAL; @@ -249,6 +253,12 @@ static ssize_t rpc_sysfs_xprt_state_change(struct kobject *kobj, count = -EINVAL; else set_bit(XPRT_OFFLINE, &xprt->state); + } else if (remove) { + if (!test_bit(XPRT_CONNECTED, &xprt->state) && + test_bit(XPRT_OFFLINE, &xprt->state)) + set_bit(XPRT_REMOVE, &xprt->state); + else + count = -EINVAL; } xprt_release_write(xprt, NULL);