From patchwork Tue Jun 7 20:47:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12872754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9A87C433EF for ; Wed, 8 Jun 2022 02:55:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229653AbiFHCzi (ORCPT ); Tue, 7 Jun 2022 22:55:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1448133AbiFHCxn (ORCPT ); Tue, 7 Jun 2022 22:53:43 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA7AE1F66D1 for ; Tue, 7 Jun 2022 13:47:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 32EB361645 for ; Tue, 7 Jun 2022 20:47:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6848BC385A5; Tue, 7 Jun 2022 20:47:53 +0000 (UTC) Subject: [PATCH v2 1/5] SUNRPC: Fix the calculation of xdr->end in xdr_get_next_encode_buffer() From: Chuck Lever To: linux-nfs@vger.kernel.org Cc: trondmy@hammerspace.com, anna.schumaker@netapp.com Date: Tue, 07 Jun 2022 16:47:52 -0400 Message-ID: <165463487244.38298.10564474394703118949.stgit@bazille.1015granger.net> In-Reply-To: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> References: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org I found that NFSD's new NFSv3 READDIRPLUS XDR encoder was screwing up right at the end of the page array. xdr_get_next_encode_buffer() does not compute the value of xdr->end correctly: * The check to see if we're on the final available page in xdr->buf needs to account for the space consumed by @nbytes. * The new xdr->end value needs to account for the portion of @nbytes that is to be encoded into the previous buffer. Fixes: 2825a7f90753 ("nfsd4: allow encoding across page boundaries") Signed-off-by: Chuck Lever Reviewed-by: NeilBrown --- net/sunrpc/xdr.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index df194cc07035..b57cf9df4de8 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -979,7 +979,11 @@ static __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, */ xdr->p = (void *)p + frag2bytes; space_left = xdr->buf->buflen - xdr->buf->len; - xdr->end = (void *)p + min_t(int, space_left, PAGE_SIZE); + if (space_left - nbytes >= PAGE_SIZE) + xdr->end = (void *)p + PAGE_SIZE; + else + xdr->end = (void *)p + space_left - frag1bytes; + xdr->buf->page_len += frag2bytes; xdr->buf->len += nbytes; return p; From patchwork Tue Jun 7 20:47:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12872646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDC85CCA47C for ; Wed, 8 Jun 2022 00:55:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236149AbiFHAse (ORCPT ); Tue, 7 Jun 2022 20:48:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1450007AbiFGXKo (ORCPT ); Tue, 7 Jun 2022 19:10:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FDDA38EB8A for ; Tue, 7 Jun 2022 13:48:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 86C9A6166B for ; Tue, 7 Jun 2022 20:48:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BFFEDC3411C; Tue, 7 Jun 2022 20:47:59 +0000 (UTC) Subject: [PATCH v2 2/5] SUNRPC: Optimize xdr_reserve_space() From: Chuck Lever To: linux-nfs@vger.kernel.org Cc: trondmy@hammerspace.com, anna.schumaker@netapp.com Date: Tue, 07 Jun 2022 16:47:58 -0400 Message-ID: <165463487881.38298.620791710948316864.stgit@bazille.1015granger.net> In-Reply-To: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> References: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Transitioning between encode buffers is quite infrequent. It happens about 1 time in 400 calls to xdr_reserve_space(), measured on NFSD with a typical build/test workload. Force the compiler to remove that code from xdr_reserve_space(), which is a hot path on both the server and the client. This change reduces the size of xdr_reserve_space() from 10 cache lines to 2 when compiled with -Os. Signed-off-by: Chuck Lever --- include/linux/sunrpc/xdr.h | 16 +++++++++++++++- net/sunrpc/xdr.c | 17 ++++++++++------- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h index 4417f667c757..5860f32e3958 100644 --- a/include/linux/sunrpc/xdr.h +++ b/include/linux/sunrpc/xdr.h @@ -243,7 +243,7 @@ extern void xdr_init_encode(struct xdr_stream *xdr, struct xdr_buf *buf, extern __be32 *xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes); extern int xdr_reserve_space_vec(struct xdr_stream *xdr, struct kvec *vec, size_t nbytes); -extern void xdr_commit_encode(struct xdr_stream *xdr); +extern void __xdr_commit_encode(struct xdr_stream *xdr); extern void xdr_truncate_encode(struct xdr_stream *xdr, size_t len); extern int xdr_restrict_buflen(struct xdr_stream *xdr, int newbuflen); extern void xdr_write_pages(struct xdr_stream *xdr, struct page **pages, @@ -306,6 +306,20 @@ xdr_reset_scratch_buffer(struct xdr_stream *xdr) xdr_set_scratch_buffer(xdr, NULL, 0); } +/** + * xdr_commit_encode - Ensure all data is written to xdr->buf + * @xdr: pointer to xdr_stream + * + * Handle encoding across page boundaries by giving the caller a + * temporary location to write to, then later copying the data into + * place. __xdr_commit_encode() does that copying. + */ +static inline void xdr_commit_encode(struct xdr_stream *xdr) +{ + if (unlikely(xdr->scratch.iov_len)) + __xdr_commit_encode(xdr); +} + /** * xdr_stream_remaining - Return the number of bytes remaining in the stream * @xdr: pointer to struct xdr_stream diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index b57cf9df4de8..1ad8b4ef14de 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -919,7 +919,7 @@ void xdr_init_encode(struct xdr_stream *xdr, struct xdr_buf *buf, __be32 *p, EXPORT_SYMBOL_GPL(xdr_init_encode); /** - * xdr_commit_encode - Ensure all data is written to buffer + * __xdr_commit_encode - Ensure all data is written to buffer * @xdr: pointer to xdr_stream * * We handle encoding across page boundaries by giving the caller a @@ -931,22 +931,25 @@ EXPORT_SYMBOL_GPL(xdr_init_encode); * required at the end of encoding, or any other time when the xdr_buf * data might be read. */ -inline void xdr_commit_encode(struct xdr_stream *xdr) +void __xdr_commit_encode(struct xdr_stream *xdr) { int shift = xdr->scratch.iov_len; void *page; - if (shift == 0) - return; page = page_address(*xdr->page_ptr); memcpy(xdr->scratch.iov_base, page, shift); memmove(page, page + shift, (void *)xdr->p - page); xdr_reset_scratch_buffer(xdr); } -EXPORT_SYMBOL_GPL(xdr_commit_encode); +EXPORT_SYMBOL_GPL(__xdr_commit_encode); -static __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, - size_t nbytes) +/* + * The buffer space to be reserved crosses the boundary between + * xdr->buf->head and xdr->buf->pages, or between two pages + * in xdr->buf->pages. + */ +static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, + size_t nbytes) { __be32 *p; int space_left; From patchwork Tue Jun 7 20:48:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12872645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE861CCA482 for ; Wed, 8 Jun 2022 00:55:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233773AbiFHAso (ORCPT ); Tue, 7 Jun 2022 20:48:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1450005AbiFGXKo (ORCPT ); Tue, 7 Jun 2022 19:10:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8691C38EBBA for ; Tue, 7 Jun 2022 13:48:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F19A26166B for ; Tue, 7 Jun 2022 20:48:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33FFAC385A2; Tue, 7 Jun 2022 20:48:06 +0000 (UTC) Subject: [PATCH v2 3/5] SUNRPC: Clean up xdr_commit_encode() From: Chuck Lever To: linux-nfs@vger.kernel.org Cc: trondmy@hammerspace.com, anna.schumaker@netapp.com Date: Tue, 07 Jun 2022 16:48:05 -0400 Message-ID: <165463488517.38298.1636534035418213053.stgit@bazille.1015granger.net> In-Reply-To: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> References: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Both the kvec::iov_len field and the third parameter of memcpy() and memmove() are size_t. There's no reason for the implicit conversion from size_t to int and back. Change the type of @shift to make the code easier to read and understand. Signed-off-by: Chuck Lever Reviewed-by: NeilBrown --- net/sunrpc/xdr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 1ad8b4ef14de..3c182041e790 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -933,7 +933,7 @@ EXPORT_SYMBOL_GPL(xdr_init_encode); */ void __xdr_commit_encode(struct xdr_stream *xdr) { - int shift = xdr->scratch.iov_len; + size_t shift = xdr->scratch.iov_len; void *page; page = page_address(*xdr->page_ptr); From patchwork Tue Jun 7 20:48:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12872647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11FC2CCA488 for ; Wed, 8 Jun 2022 00:55:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235933AbiFHAss (ORCPT ); Tue, 7 Jun 2022 20:48:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1450062AbiFGXKs (ORCPT ); Tue, 7 Jun 2022 19:10:48 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C142718492A for ; Tue, 7 Jun 2022 13:48:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 27E77B823EC for ; Tue, 7 Jun 2022 20:48:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A3190C385A2; Tue, 7 Jun 2022 20:48:12 +0000 (UTC) Subject: [PATCH v2 4/5] SUNRPC: Clean up xdr_get_next_encode_buffer() From: Chuck Lever To: linux-nfs@vger.kernel.org Cc: trondmy@hammerspace.com, anna.schumaker@netapp.com Date: Tue, 07 Jun 2022 16:48:11 -0400 Message-ID: <165463489159.38298.13323606688028914163.stgit@bazille.1015granger.net> In-Reply-To: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> References: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org The value of @p is not used until the "location of the next item" is computed. Help human readers by moving its initial assignment to the paragraph where that value is used and by clarifying the antecedents in the documenting comment. Signed-off-by: Chuck Lever Reviewed-by: NeilBrown --- net/sunrpc/xdr.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 3c182041e790..eca02d122476 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -967,6 +967,7 @@ static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, xdr->buf->page_len += frag1bytes; xdr->page_ptr++; xdr->iov = NULL; + /* * If the last encode didn't end exactly on a page boundary, the * next one will straddle boundaries. Encode into the next @@ -975,11 +976,12 @@ static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, * space at the end of the previous buffer: */ xdr_set_scratch_buffer(xdr, xdr->p, frag1bytes); - p = page_address(*xdr->page_ptr); + /* - * Note this is where the next encode will start after we've - * shifted this one back: + * xdr->p is where the next encode will start after + * xdr_commit_encode() has shifted this one back: */ + p = page_address(*xdr->page_ptr); xdr->p = (void *)p + frag2bytes; space_left = xdr->buf->buflen - xdr->buf->len; if (space_left - nbytes >= PAGE_SIZE) From patchwork Tue Jun 7 20:48:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12872755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 595B0C433EF for ; Wed, 8 Jun 2022 02:55:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231810AbiFHCzn (ORCPT ); Tue, 7 Jun 2022 22:55:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1448134AbiFHCxn (ORCPT ); Tue, 7 Jun 2022 22:53:43 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E35F238FB85 for ; Tue, 7 Jun 2022 13:48:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 81CDAB823EA for ; Tue, 7 Jun 2022 20:48:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E1F3C385A5; Tue, 7 Jun 2022 20:48:19 +0000 (UTC) Subject: [PATCH v2 5/5] SUNRPC: Remove pointer type casts from xdr_get_next_encode_buffer() From: Chuck Lever To: linux-nfs@vger.kernel.org Cc: trondmy@hammerspace.com, anna.schumaker@netapp.com Date: Tue, 07 Jun 2022 16:48:18 -0400 Message-ID: <165463489811.38298.8036154060998518968.stgit@bazille.1015granger.net> In-Reply-To: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> References: <165463444560.38298.18296069287423675496.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org To make the code easier to read, remove visual clutter by changing the declared type of @p. Signed-off-by: Chuck Lever Reviewed-by: NeilBrown --- net/sunrpc/xdr.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index eca02d122476..f87a2d8f23a7 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -951,9 +951,9 @@ EXPORT_SYMBOL_GPL(__xdr_commit_encode); static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, size_t nbytes) { - __be32 *p; int space_left; int frag1bytes, frag2bytes; + void *p; if (nbytes > PAGE_SIZE) goto out_overflow; /* Bigger buffers require special handling */ @@ -982,12 +982,12 @@ static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, * xdr_commit_encode() has shifted this one back: */ p = page_address(*xdr->page_ptr); - xdr->p = (void *)p + frag2bytes; + xdr->p = p + frag2bytes; space_left = xdr->buf->buflen - xdr->buf->len; if (space_left - nbytes >= PAGE_SIZE) - xdr->end = (void *)p + PAGE_SIZE; + xdr->end = p + PAGE_SIZE; else - xdr->end = (void *)p + space_left - frag1bytes; + xdr->end = p + space_left - frag1bytes; xdr->buf->page_len += frag2bytes; xdr->buf->len += nbytes;