From patchwork Sat Apr 15 00:17:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13212221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DDFEC77B71 for ; Sat, 15 Apr 2023 00:18:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229643AbjDOASB (ORCPT ); Fri, 14 Apr 2023 20:18:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbjDOASA (ORCPT ); Fri, 14 Apr 2023 20:18:00 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA1DC40FB for ; Fri, 14 Apr 2023 17:17:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4470F63856 for ; Sat, 15 Apr 2023 00:17:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A180C433A0; Sat, 15 Apr 2023 00:17:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681517877; bh=w0AGIKncJ+aMWbYsTzdaumoT4rTj66CdiASkX6lZ4UU=; h=Subject:From:To:Date:In-Reply-To:References:From; b=hXfE/aAz5R89eQvCA4gEJ1LT2dArTmazeb+qhr8Mtd66M5OTM+PGTvU9rdnSeR6Vg 074TTlq2rc4bDFuQK7AqzwnV8Zl5ifmGeJe/TPUUWrotsP4PD2QuLYXTI9saUYiYjt wSjljvZ3yk5NpYqS1z4iRE7Ks4rnxK0Myvax2Zh380qVjPqBqyQTxOKU7xP8Ur2zrs IWBmKDIXIWAZJAqA3pmiSo13JBhPGxHGHZoCnjGeg2uV4skBFbMuw6BLbB13t1iNlV ja9fRCsIDDXwBVt+1usSOEjYbiY9VX328hIVzOHLo7/PIxsJYooPL/nGq1s8tpQkr0 ip9uTDQcWasOA== Subject: [PATCH v1 1/4] SUNRPC: Relocate svc_free_res_pages() From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-mm@kvack.org Date: Fri, 14 Apr 2023 20:17:56 -0400 Message-ID: <168151787655.1588.13024575588334839763.stgit@klimt.1015granger.net> In-Reply-To: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> References: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Chuck Lever Clean-up: There doesn't seem to be a reason why this function is stuck in a header. One thing it prevents is the convenient addition of tracing. Moving it to a source file also makes the rq_respages clean-up logic easier to find. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc.h | 12 +----------- net/sunrpc/svc.c | 19 +++++++++++++++++++ net/sunrpc/svc_xprt.c | 2 +- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 2d31121fc2e6..762d7231e574 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -309,17 +309,6 @@ static inline struct sockaddr *svc_daddr(const struct svc_rqst *rqst) return (struct sockaddr *) &rqst->rq_daddr; } -static inline void svc_free_res_pages(struct svc_rqst *rqstp) -{ - while (rqstp->rq_next_page != rqstp->rq_respages) { - struct page **pp = --rqstp->rq_next_page; - if (*pp) { - put_page(*pp); - *pp = NULL; - } - } -} - struct svc_deferred_req { u32 prot; /* protocol (UDP or TCP) */ struct svc_xprt *xprt; @@ -424,6 +413,7 @@ struct svc_rqst *svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node); bool svc_rqst_replace_page(struct svc_rqst *rqstp, struct page *page); +void svc_rqst_release_pages(struct svc_rqst *rqstp); void svc_rqst_free(struct svc_rqst *); void svc_exit_thread(struct svc_rqst *); struct svc_serv * svc_create_pooled(struct svc_program *, unsigned int, diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 0aa8892fad63..0fc70cc405b2 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -869,6 +869,25 @@ bool svc_rqst_replace_page(struct svc_rqst *rqstp, struct page *page) } EXPORT_SYMBOL_GPL(svc_rqst_replace_page); +/** + * svc_rqst_release_pages - Release Reply buffer pages + * @rqstp: RPC transaction context + * + * Release response pages that might still be in flight after + * svc_send, and any spliced filesystem-owned pages. + */ +void svc_rqst_release_pages(struct svc_rqst *rqstp) +{ + while (rqstp->rq_next_page != rqstp->rq_respages) { + struct page **pp = --rqstp->rq_next_page; + + if (*pp) { + put_page(*pp); + *pp = NULL; + } + } +} + /* * Called from a server thread as it's exiting. Caller must hold the "service * mutex" for the service. diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 36c79b718323..533e08c4f319 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -542,7 +542,7 @@ static void svc_xprt_release(struct svc_rqst *rqstp) rqstp->rq_deferred = NULL; pagevec_release(&rqstp->rq_pvec); - svc_free_res_pages(rqstp); + svc_rqst_release_pages(rqstp); rqstp->rq_res.page_len = 0; rqstp->rq_res.page_base = 0; From patchwork Sat Apr 15 00:18:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13212222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA739C77B76 for ; Sat, 15 Apr 2023 00:18:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229760AbjDOASG (ORCPT ); Fri, 14 Apr 2023 20:18:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbjDOASF (ORCPT ); Fri, 14 Apr 2023 20:18:05 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E997540FB for ; Fri, 14 Apr 2023 17:18:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 84BDC64632 for ; Sat, 15 Apr 2023 00:18:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C42B6C4339E; Sat, 15 Apr 2023 00:18:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681517883; bh=RshTAt8N1txg6TF0maDP8tdDMDzPklBfrU/3+g+KeB8=; h=Subject:From:To:Date:In-Reply-To:References:From; b=jYSxYefrqfWu1kEKIKaipUqoEVNeS1w6Bew7vDGwis67wzwDtoDRpSup7gSR+lyhS 3o24x6abUfaF6oJH9e9zRDr/EucGPFxwfFnj6xwyJFgbNDDpCU5Y8S1hnbiVph2FDu k/iVHF6mKtq4B2eJdfBOZJomYCHedALLd0xFWnwyF9Z7VTN9YR9iy2DyMTSojnT7Gm mluQbC38LoNAC+bN8BgR53VzYVoDCbVzD2Yh23wE11YLdcK4pwjN0P/5sbT5OnFaaM a25PnOzBTCKvasRQP1BOitBiHL0JfByCoxIfCSfI14P4xAkpNJ2WU4QRySiSZWYrkr xhcHRqTR5aaDQ== Subject: [PATCH v1 2/4] SUNRPC: Convert svc_xprt_release() to the release_pages() API From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-mm@kvack.org Date: Fri, 14 Apr 2023 20:18:02 -0400 Message-ID: <168151788284.1588.1198570306681230834.stgit@klimt.1015granger.net> In-Reply-To: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> References: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Chuck Lever Instead of invoking put_page() one-at-a-time, pass the "response" portion of rq_pages directly to release_pages() to reduce the number of times each nfsd thread invokes a page allocator API. Since svc_xprt_release() is not invoked while a client is waiting for an RPC Reply, this is not expected to directly impact mean request latencies on a lightly or moderately loaded server. However as workload intensity increases, I expect somewhat better scalability: the same number of server threads should be able to handle more work. Signed-off-by: Chuck Lever --- net/sunrpc/svc.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 0fc70cc405b2..b982f802f2a0 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -878,13 +878,12 @@ EXPORT_SYMBOL_GPL(svc_rqst_replace_page); */ void svc_rqst_release_pages(struct svc_rqst *rqstp) { - while (rqstp->rq_next_page != rqstp->rq_respages) { - struct page **pp = --rqstp->rq_next_page; + int i, count = rqstp->rq_next_page - rqstp->rq_respages; - if (*pp) { - put_page(*pp); - *pp = NULL; - } + if (count) { + release_pages(rqstp->rq_respages, count); + for (i = 0; i < count; i++) + rqstp->rq_respages[i] = NULL; } } From patchwork Sat Apr 15 00:18:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13212223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E049C77B71 for ; Sat, 15 Apr 2023 00:18:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229822AbjDOASM (ORCPT ); Fri, 14 Apr 2023 20:18:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbjDOASL (ORCPT ); Fri, 14 Apr 2023 20:18:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3614E40FB for ; Fri, 14 Apr 2023 17:18:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C9E5A645C6 for ; Sat, 15 Apr 2023 00:18:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 18C2EC433D2; Sat, 15 Apr 2023 00:18:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681517890; bh=sTDSRZds3I5VoEZES7MzTU1WmuMqkd/9J180VGx4J2Y=; h=Subject:From:To:Date:In-Reply-To:References:From; b=AJ6awgWMSm3B3TfX59IIKRBUQuR6yBd3AA4wftjj7YJ+BvQMnJ0ebiRpshGEI4TnY RSUWdgYCu156/OJ1DjxyI11+KFeDIgxUPn2eC2s7bxqBHyla3bWJVJnkcRzWwy2zNc Ok0LfTV38XgpG/5m6JrR1SXv3emuat0CRFGdGkV2wxM/uIco8zBy+QIyuZBcN8HG2P 3InJk3jD8BqyeUP0+Qjo+z5hSOksJ86V+31kERTdQw8brsCHHmSccZKdyHJcP0m94i DxpFmg+fAMe3tn4H5Knu9gj6TXpJco+RSV1CSlD/V7aSnGckIt083n3wSk3uo4Mv3f P70g1Rjnh911w== Subject: [PATCH v1 3/4] SUNRPC: Convert svc_tcp_restore_pages() to the release_pages() API From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-mm@kvack.org Date: Fri, 14 Apr 2023 20:18:09 -0400 Message-ID: <168151788914.1588.17896994938586012232.stgit@klimt.1015granger.net> In-Reply-To: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> References: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Chuck Lever Instead of invoking put_page() one-at-a-time, pass the portion of rq_pages to be released directly to release_pages() to reduce the number of times each server thread invokes a page allocator API. Signed-off-by: Chuck Lever --- net/sunrpc/svcsock.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 302a14dd7882..44f72b558a1c 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -812,12 +812,12 @@ static size_t svc_tcp_restore_pages(struct svc_sock *svsk, if (!len) return 0; npages = (len + PAGE_SIZE - 1) >> PAGE_SHIFT; - for (i = 0; i < npages; i++) { - if (rqstp->rq_pages[i] != NULL) - put_page(rqstp->rq_pages[i]); - BUG_ON(svsk->sk_pages[i] == NULL); - rqstp->rq_pages[i] = svsk->sk_pages[i]; - svsk->sk_pages[i] = NULL; + if (npages) { + release_pages(rqstp->rq_pages, npages); + for (i = 0; i < npages; i++) { + rqstp->rq_pages[i] = svsk->sk_pages[i]; + svsk->sk_pages[i] = NULL; + } } rqstp->rq_arg.head[0].iov_base = page_address(rqstp->rq_pages[0]); return len; From patchwork Sat Apr 15 00:18:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13212224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BF58C77B76 for ; Sat, 15 Apr 2023 00:18:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229841AbjDOASU (ORCPT ); Fri, 14 Apr 2023 20:18:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbjDOASS (ORCPT ); Fri, 14 Apr 2023 20:18:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C33340F8 for ; Fri, 14 Apr 2023 17:18:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 18E1063856 for ; Sat, 15 Apr 2023 00:18:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5AFF5C433EF; Sat, 15 Apr 2023 00:18:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681517896; bh=gwoDKthJAF1WHfbGuiCRuLpxNhrgjIi9oMkJ9fM63b0=; h=Subject:From:To:Date:In-Reply-To:References:From; b=q30o4mYFhbelsQ8SGE1YZKn7WuVdtBSkF/hM8jmtBXkBFL3iTHEyl/yOTRJojvgFL HtRjb1M2zuyVP9HdYzisBRXDr7Nudgn6kl8Sfkd6Wk5RiNo0Awfxy9tR01XCJdY+vN ztCNBwS4uIFBTzJzqXnDdlzU7Z+Pbk+zEUIZH+dmsfxnAMXQpeOg8MMG+e9ilf85Lq MK0S+fv153+w9uV3dASWZ/ekqK5L93DeDpcoPINWbGQmkp5OkG2brxO0k+oH/22SMe +XlBeo3arby3UnHwREkIDVlkbDJfJ+dOYot8uc+M3lMHXMV6OqcBkP5MVFIGHjG+1V EAIAoVm5G/SeQ== Subject: [PATCH v1 4/4] SUNRPC: Be even lazier about releasing pages From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-mm@kvack.org Date: Fri, 14 Apr 2023 20:18:15 -0400 Message-ID: <168151789543.1588.6591139655130358318.stgit@klimt.1015granger.net> In-Reply-To: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> References: <168151777579.1588.7882383278745556830.stgit@klimt.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Chuck Lever A single RPC transaction that touches only a couple of pages means rq_pvec will not be even close to full in svc_xpt_release(). This is a common case. Instead, just leave the pages in rq_pvec until it is completely full. This improves the efficiency of the batch release mechanism on workloads that involve small RPC messages. The rq_pvec is also fully emptied just before thread exit. Signed-off-by: Chuck Lever --- net/sunrpc/svc.c | 3 +++ net/sunrpc/svc_xprt.c | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index b982f802f2a0..26367cf4c17a 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -649,6 +649,8 @@ svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node) if (!rqstp) return rqstp; + pagevec_init(&rqstp->rq_pvec); + __set_bit(RQ_BUSY, &rqstp->rq_flags); rqstp->rq_server = serv; rqstp->rq_pool = pool; @@ -894,6 +896,7 @@ void svc_rqst_release_pages(struct svc_rqst *rqstp) void svc_rqst_free(struct svc_rqst *rqstp) { + pagevec_release(&rqstp->rq_pvec); svc_release_buffer(rqstp); if (rqstp->rq_scratch_page) put_page(rqstp->rq_scratch_page); diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 533e08c4f319..e3952b690f54 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -541,7 +541,6 @@ static void svc_xprt_release(struct svc_rqst *rqstp) kfree(rqstp->rq_deferred); rqstp->rq_deferred = NULL; - pagevec_release(&rqstp->rq_pvec); svc_rqst_release_pages(rqstp); rqstp->rq_res.page_len = 0; rqstp->rq_res.page_base = 0; @@ -667,8 +666,6 @@ static int svc_alloc_arg(struct svc_rqst *rqstp) struct xdr_buf *arg = &rqstp->rq_arg; unsigned long pages, filled, ret; - pagevec_init(&rqstp->rq_pvec); - pages = (serv->sv_max_mesg + 2 * PAGE_SIZE) >> PAGE_SHIFT; if (pages > RPCSVC_MAXPAGES) { pr_warn_once("svc: warning: pages=%lu > RPCSVC_MAXPAGES=%lu\n",