From patchwork Mon Sep 11 18:50:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trond Myklebust X-Patchwork-Id: 13380175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AFB3CA0EEA for ; Mon, 11 Sep 2023 22:00:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242078AbjIKV7H (ORCPT ); Mon, 11 Sep 2023 17:59:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244048AbjIKS5E (ORCPT ); Mon, 11 Sep 2023 14:57:04 -0400 Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7FF71B8 for ; Mon, 11 Sep 2023 11:56:58 -0700 (PDT) Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-575f45e255dso2977897eaf.2 for ; Mon, 11 Sep 2023 11:56:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694458618; x=1695063418; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=t9FA10xdzo3rI/p2Oc+v/6tnWHhSYIWVUzHlU2v7eYk=; b=VvAHsN+geQ+fxlpSMummDCxFn2YmSv3GDJOp9KZp/fgoSO3ukIcwjMj6H4LXfZnRNZ 7gsvlYuWYYgeT1PO64UP/MmyWuEU+4ybr5sHpUZoZbZJzEEPQHz8/+D1WVCe6paDK6jZ 2STCCx2R7Elbncg3p2BGXencKAPqKrAgt+vwWLp8BXvgmEpeWvSYshPJh6BQMqInvdUM b9wNdWf+mNrltbQlGTWYCWOyLTY7NYWa1k8OZbDk/GIHfL0SX0uvbdh1DsJe/LXqqtcy Kk5gQ7jAbsTbgzH6nH24gWp9sPNTSf8Or+A3EBjkMHeJS7YzSK7vsz/sjGxouQatJatr KgVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694458618; x=1695063418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t9FA10xdzo3rI/p2Oc+v/6tnWHhSYIWVUzHlU2v7eYk=; b=PUsv5Y0NZcp2EyC014F7fyGD9MVFKEeD9Pi8LwcRByDOn+tnYojEgibMqwTj0dYYPM nY2caOFbjoECXtPIrQMGdm/dEWN2elpXR4+bSWwIOn3Cg2tro7GReycOmOmGYaLAroOw h3DdpbVNovByVCTERc3YTqjQdaDPILNwouvXFN/+hfI42GADLijjhvdpc6dMINtZK6al ray8ZWkSZkRUfGdRCqlEJGwlUwCgyAs7FMWS/AaVo456OStxoMVtfWDPO+DIagp4SbW6 7yt4RrIY+V9BbIKSwNOUSMbn591cAONAziv2XavqON9C8r2JJ3G6WJbW7H3FaeaoW5os dFIA== X-Gm-Message-State: AOJu0Yw1fJqX53z8KavcNkjdIeJiZCT0y1y8MTlwzRzsdmLpXJwmHPyU Uck5OKujreUBRjlWbBfNpHFXe/LvzQ== X-Google-Smtp-Source: AGHT+IEmMe45P4eyqFhQgjrxzGmP8x65sHDhR8/H2Z6ptaGjq0UvSODK2ChEZv/idB86WvhG0Tx2qg== X-Received: by 2002:a05:6358:729e:b0:140:f08c:2b50 with SMTP id w30-20020a056358729e00b00140f08c2b50mr7138047rwf.6.1694458617701; Mon, 11 Sep 2023 11:56:57 -0700 (PDT) Received: from localhost.localdomain (c-68-32-72-208.hsd1.mi.comcast.net. [68.32.72.208]) by smtp.gmail.com with ESMTPSA id e15-20020a0caa4f000000b006263a9e7c63sm3106068qvb.104.2023.09.11.11.56.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 11:56:57 -0700 (PDT) From: trondmy@gmail.com X-Google-Original-From: trond.myklebust@hammerspace.com To: linux-nfs@vger.kernel.org Subject: [PATCH 1/2] NFSv4: Add a parameter to limit the number of retries after NFS4ERR_DELAY Date: Mon, 11 Sep 2023 14:50:30 -0400 Message-ID: <20230911185031.11903-2-trond.myklebust@hammerspace.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230911185031.11903-1-trond.myklebust@hammerspace.com> References: <20230911185031.11903-1-trond.myklebust@hammerspace.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Trond Myklebust When using a 'softerr' mount, the NFSv4 client can get stuck waiting forever while the server just returns NFS4ERR_DELAY. Among other things, this causes the knfsd server threads to busy wait. Add a parameter that tells the NFSv4 client how many times to retry before giving up. Signed-off-by: Trond Myklebust --- .../admin-guide/kernel-parameters.txt | 7 ++++++ fs/nfs/nfs4_fs.h | 2 ++ fs/nfs/nfs4proc.c | 25 +++++++++++++++++++ fs/nfs/super.c | 8 +++++- 4 files changed, 41 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 722b6eca2e93..62af591ee2ad 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3517,6 +3517,13 @@ [NFS] set the TCP port on which the NFSv4 callback channel should listen. + nfs.delay_retrans= + [NFS] specifies the number of times the NFSv4 client + retries the request before returning an EAGAIN error, + after a reply of NFS4ERR_DELAY from the server. + Only applies if the softerr mount option is enabled, + and the specified value is >= 0. + nfs.enable_ino64= [NFS] enable 64-bit inode numbers. If zero, the NFS client will fake up a 32-bit inode diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h index 4c9f8bd866ab..29b91482e5f4 100644 --- a/fs/nfs/nfs4_fs.h +++ b/fs/nfs/nfs4_fs.h @@ -209,6 +209,7 @@ struct nfs4_exception { struct inode *inode; nfs4_stateid *stateid; long timeout; + unsigned short retrans; unsigned char task_is_privileged : 1; unsigned char delay : 1, recovering : 1, @@ -546,6 +547,7 @@ extern unsigned short max_session_slots; extern unsigned short max_session_cb_slots; extern unsigned short send_implementation_id; extern bool recover_lost_locks; +extern short nfs_delay_retrans; #define NFS4_CLIENT_ID_UNIQ_LEN (64) extern char nfs4_client_id_uniquifier[NFS4_CLIENT_ID_UNIQ_LEN]; diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 832fa226b8f2..99c054a6c7b5 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -585,6 +585,21 @@ static int nfs4_do_handle_exception(struct nfs_server *server, return 0; } +/* + * Track the number of NFS4ERR_DELAY related retransmissions and return + * EAGAIN if the 'softerr' mount option is set, and we've exceeded the limit + * set by 'nfs_delay_retrans'. + */ +static int nfs4_exception_should_retrans(const struct nfs_server *server, + struct nfs4_exception *exception) +{ + if (server->flags & NFS_MOUNT_SOFTERR && nfs_delay_retrans >= 0) { + if (exception->retrans++ >= (unsigned short)nfs_delay_retrans) + return -EAGAIN; + } + return 0; +} + /* This is the error handling routine for processes that are allowed * to sleep. */ @@ -595,6 +610,11 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_ ret = nfs4_do_handle_exception(server, errorcode, exception); if (exception->delay) { + int ret2 = nfs4_exception_should_retrans(server, exception); + if (ret2 < 0) { + exception->retry = 0; + return ret2; + } ret = nfs4_delay(&exception->timeout, exception->interruptible); goto out_retry; @@ -623,6 +643,11 @@ nfs4_async_handle_exception(struct rpc_task *task, struct nfs_server *server, ret = nfs4_do_handle_exception(server, errorcode, exception); if (exception->delay) { + int ret2 = nfs4_exception_should_retrans(server, exception); + if (ret2 < 0) { + exception->retry = 0; + return ret2; + } rpc_delay(task, nfs4_update_delay(&exception->timeout)); goto out_retry; } diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 2284f749d892..929e122ff34b 100644 --- a/fs/nfs/super.c +++ b/fs/nfs/super.c @@ -1368,6 +1368,7 @@ unsigned short max_session_cb_slots = NFS4_DEF_CB_SLOT_TABLE_SIZE; unsigned short send_implementation_id = 1; char nfs4_client_id_uniquifier[NFS4_CLIENT_ID_UNIQ_LEN] = ""; bool recover_lost_locks = false; +short nfs_delay_retrans = -1; EXPORT_SYMBOL_GPL(nfs_callback_nr_threads); EXPORT_SYMBOL_GPL(nfs_callback_set_tcpport); @@ -1378,6 +1379,7 @@ EXPORT_SYMBOL_GPL(max_session_cb_slots); EXPORT_SYMBOL_GPL(send_implementation_id); EXPORT_SYMBOL_GPL(nfs4_client_id_uniquifier); EXPORT_SYMBOL_GPL(recover_lost_locks); +EXPORT_SYMBOL_GPL(nfs_delay_retrans); #define NFS_CALLBACK_MAXPORTNR (65535U) @@ -1426,5 +1428,9 @@ MODULE_PARM_DESC(recover_lost_locks, "If the server reports that a lock might be lost, " "try to recover it risking data corruption."); - +module_param_named(delay_retrans, nfs_delay_retrans, short, 0644); +MODULE_PARM_DESC(delay_retrans, + "Unless negative, specifies the number of times the NFSv4 " + "client retries a request before returning an EAGAIN error, " + "after a reply of NFS4ERR_DELAY from the server."); #endif /* CONFIG_NFS_V4 */ From patchwork Mon Sep 11 18:50:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trond Myklebust X-Patchwork-Id: 13380167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE854CA0EC7 for ; Mon, 11 Sep 2023 21:59:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235513AbjIKV6M (ORCPT ); Mon, 11 Sep 2023 17:58:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244049AbjIKS5F (ORCPT ); Mon, 11 Sep 2023 14:57:05 -0400 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED2141B6 for ; Mon, 11 Sep 2023 11:56:59 -0700 (PDT) Received: by mail-qv1-xf2c.google.com with SMTP id 6a1803df08f44-64f4ac604c2so25874286d6.1 for ; Mon, 11 Sep 2023 11:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694458619; x=1695063419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=lJFAHGplLbWURCk//AHrAxo6w0nka4O2qa2RE6264FE=; b=h1/qfLbHwt83rFbuuzdp7VgKC7Q/z2X2rW7+bMRflx6K0W1k6Z90LKWhO1ZmKjhQ8t GiQEAy9PtKrYlayf5OEswwVX0ObIYwP/Gxm1i173Xewhvwsu2/PoJFh1WMu0eT1vU8OA 8jiY9jCN80iZbvKOHN5h83uaeFqzigJRWk+HUC5pZzIoC8RZGNKtsce4jsw/U9dx7Cs2 xyfzbDqz8zOtuUuD+FYjeTHPjk3Rs6K2Y2s1l0K1Uip+tu0fCSR8ikQR4QgkeZtDFAeu 2xs0mUiIfXQ9NFDen2c8WExajmZKH4cNSHc6mwcOf3AxfA+4kJkk7yqMhsiAcAT5fXI6 Sriw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694458619; x=1695063419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lJFAHGplLbWURCk//AHrAxo6w0nka4O2qa2RE6264FE=; b=NcC7DraPb7w8X9s0PAq4z8hRZcX+KWjB1wyCIxqb5SM+5oO+y1fflTEz/bV70d9p5A syhecF5IGzxpWFfeZ1Jj9kJM+anFnD0GDU8TgFDP6UrapE6xsVBGcPgLYTl9NQBfMtkh PWNQ/9pSAqh75goW1YG32UBJdWIoOyzuGYIerdvVNfu/3PC+Ujoy6+UEj9lDDPQZRRZU 9pjsO4ZcOJbfZV09zfvDr4/Hk8nJDVgUK0UgWVyf01vuOwia926ZEhCx2hBMz8isAU+2 mHkGAfzrDAf6ncRoDAwXRScKBMUfJ8W/rvw2WEQFk9ZOkQ4RwnYzfeDpqhHIWHeMTNOy fpAg== X-Gm-Message-State: AOJu0YwW8YWjOoIrEvcGzwHk01BDYLsNtAJqANaSoJRIgL2c+HlVVVpM lBzsa5uRRRWhVFNf3WhrXOtzl8swuQ== X-Google-Smtp-Source: AGHT+IH5zNkAnVvOUQ5OewtqtmwgWYKBQS8YcBCvYSE8kD5QnDMvpe2aUSifGriciAYT/a2Ws4kcVw== X-Received: by 2002:a0c:f189:0:b0:64f:3795:c10 with SMTP id m9-20020a0cf189000000b0064f37950c10mr9114264qvl.10.1694458618796; Mon, 11 Sep 2023 11:56:58 -0700 (PDT) Received: from localhost.localdomain (c-68-32-72-208.hsd1.mi.comcast.net. [68.32.72.208]) by smtp.gmail.com with ESMTPSA id e15-20020a0caa4f000000b006263a9e7c63sm3106068qvb.104.2023.09.11.11.56.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 11:56:58 -0700 (PDT) From: trondmy@gmail.com X-Google-Original-From: trond.myklebust@hammerspace.com To: linux-nfs@vger.kernel.org Subject: [PATCH 2/2] NFSv4/pnfs: Allow layoutget to return EAGAIN for softerr mounts Date: Mon, 11 Sep 2023 14:50:31 -0400 Message-ID: <20230911185031.11903-3-trond.myklebust@hammerspace.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230911185031.11903-2-trond.myklebust@hammerspace.com> References: <20230911185031.11903-1-trond.myklebust@hammerspace.com> <20230911185031.11903-2-trond.myklebust@hammerspace.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Trond Myklebust If we're using the 'softerr' mount option, we may want to allow layoutget to return EAGAIN to allow knfsd server threads to return a JUKEBOX/DELAY error to the client instead of busy waiting. Signed-off-by: Trond Myklebust --- fs/nfs/nfs4proc.c | 18 ++++++++++-------- fs/nfs/pnfs.c | 8 ++++++-- fs/nfs/pnfs.h | 5 ++++- fs/nfs/write.c | 2 ++ 4 files changed, 22 insertions(+), 11 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 99c054a6c7b5..5deeaea8026e 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -9651,6 +9651,9 @@ nfs4_layoutget_handle_exception(struct rpc_task *task, nfs4_sequence_free_slot(&lgp->res.seq_res); + exception->state = NULL; + exception->stateid = NULL; + switch (nfs4err) { case 0: goto out; @@ -9746,7 +9749,8 @@ static const struct rpc_call_ops nfs4_layoutget_call_ops = { }; struct pnfs_layout_segment * -nfs4_proc_layoutget(struct nfs4_layoutget *lgp, long *timeout) +nfs4_proc_layoutget(struct nfs4_layoutget *lgp, + struct nfs4_exception *exception) { struct inode *inode = lgp->args.inode; struct nfs_server *server = NFS_SERVER(inode); @@ -9766,13 +9770,10 @@ nfs4_proc_layoutget(struct nfs4_layoutget *lgp, long *timeout) RPC_TASK_MOVEABLE, }; struct pnfs_layout_segment *lseg = NULL; - struct nfs4_exception exception = { - .inode = inode, - .timeout = *timeout, - }; int status = 0; nfs4_init_sequence(&lgp->args.seq_args, &lgp->res.seq_res, 0, 0); + exception->retry = 0; task = rpc_run_task(&task_setup_data); if (IS_ERR(task)) @@ -9783,11 +9784,12 @@ nfs4_proc_layoutget(struct nfs4_layoutget *lgp, long *timeout) goto out; if (task->tk_status < 0) { - status = nfs4_layoutget_handle_exception(task, lgp, &exception); - *timeout = exception.timeout; + exception->retry = 1; + status = nfs4_layoutget_handle_exception(task, lgp, exception); } else if (lgp->res.layoutp->len == 0) { + exception->retry = 1; status = -EAGAIN; - *timeout = nfs4_update_delay(&exception.timeout); + nfs4_update_delay(&exception->timeout); } else lseg = pnfs_layout_process(lgp); out: diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 306cba0b9e69..63904a372b2f 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -1980,7 +1980,9 @@ pnfs_update_layout(struct inode *ino, struct pnfs_layout_segment *lseg = NULL; struct nfs4_layoutget *lgp; nfs4_stateid stateid; - long timeout = 0; + struct nfs4_exception exception = { + .inode = ino, + }; unsigned long giveup = jiffies + (clp->cl_lease_time << 1); bool first; @@ -2144,7 +2146,7 @@ pnfs_update_layout(struct inode *ino, lgp->lo = lo; pnfs_get_layout_hdr(lo); - lseg = nfs4_proc_layoutget(lgp, &timeout); + lseg = nfs4_proc_layoutget(lgp, &exception); trace_pnfs_update_layout(ino, pos, count, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_SEND_LAYOUTGET); nfs_layoutget_end(lo); @@ -2171,6 +2173,8 @@ pnfs_update_layout(struct inode *ino, goto out_put_layout_hdr; } if (lseg) { + if (!exception.retry) + goto out_put_layout_hdr; if (first) pnfs_clear_first_layoutget(lo); trace_pnfs_update_layout(ino, pos, count, diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index d886c8226d8f..db57a85500ee 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -35,6 +35,7 @@ #include #include +struct nfs4_exception; struct nfs4_opendata; enum { @@ -245,7 +246,9 @@ extern size_t max_response_pages(struct nfs_server *server); extern int nfs4_proc_getdeviceinfo(struct nfs_server *server, struct pnfs_device *dev, const struct cred *cred); -extern struct pnfs_layout_segment* nfs4_proc_layoutget(struct nfs4_layoutget *lgp, long *timeout); +extern struct pnfs_layout_segment * +nfs4_proc_layoutget(struct nfs4_layoutget *lgp, + struct nfs4_exception *exception); extern int nfs4_proc_layoutreturn(struct nfs4_layoutreturn *lrp, bool sync); /* pnfs.c */ diff --git a/fs/nfs/write.c b/fs/nfs/write.c index 8c1ee1a1a28f..59478a32e19f 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -739,6 +739,8 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc) &pgio); pgio.pg_error = 0; nfs_pageio_complete(&pgio); + if (err == -EAGAIN && mntflags & NFS_MOUNT_SOFTERR) + break; } while (err < 0 && !nfs_error_is_fatal(err)); nfs_io_completion_put(ioc);