From patchwork Fri Mar 31 16:08:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74EE1C761A6 for ; Fri, 31 Mar 2023 16:09:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E08FE6B0074; Fri, 31 Mar 2023 12:09:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB6626B0075; Fri, 31 Mar 2023 12:09:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7E2C6B0078; Fri, 31 Mar 2023 12:09:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BA6316B0074 for ; Fri, 31 Mar 2023 12:09:36 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 86090C107C for ; Fri, 31 Mar 2023 16:09:36 +0000 (UTC) X-FDA: 80629678752.18.DF14396 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id B896640029 for ; Fri, 31 Mar 2023 16:09:32 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ac3dk7Cy; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278972; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fsOLHSRSLMInNcD449MTPXsGwp1kWYvONeYqTpf/Ez4=; b=4UH/mMOGrepTpUU4uVihxDf8A1rJ3+w2QrrOwYBiH2mgWlC9/WRm97K3Jx1aDPEQloItw3 PpDf8CsnPQjBsYD/CAFiv+5frbkAvZfb7E8oajU8Gp0OHPcTHn9/fW0J/dyBX5hs2UtQZY gan969lOTlvVRlRaVLUy+wg34L3YRuA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ac3dk7Cy; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278972; a=rsa-sha256; cv=none; b=VurOxltdVZR6E6JAd8C/Thb19BzbxSdxIgLGlsyujCRnabwvvjuu+kuU4ILnum2SUQkmHD Btx1HAIOFCYFa4zS2v58MFfA9pjGys9+RlJDrfO5rYbzltFi24UIs65pwrQbkw2f7fGnfZ Z+t/viVIY1uMYQk/4zb9FMsf8lmtrGo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278972; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fsOLHSRSLMInNcD449MTPXsGwp1kWYvONeYqTpf/Ez4=; b=ac3dk7CykEzP8MlLpKqYjILvXOQ871HViHO9l7bAuVx24JOcOy4pYPXsbclqlTc9L03c32 iodOndV/tsWuAXvzav6Ml1xOgNHxdloDQ4i1Tf/ApYmDwS6Dj9y/Xz+WhbmYjkM67Jr2cQ xAGFD+V2T0TuXYUnSVu8tXNTwStLmt4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-613-JBunTOtONMCZ2eDMDHE0OA-1; Fri, 31 Mar 2023 12:09:25 -0400 X-MC-Unique: JBunTOtONMCZ2eDMDHE0OA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 56800855425; Fri, 31 Mar 2023 16:09:24 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CF52D4042AC0; Fri, 31 Mar 2023 16:09:21 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steve French , Shyam Prasad N , Rohith Surabattula , linux-cachefs@redhat.com, linux-cifs@vger.kernel.org Subject: [PATCH v3 01/55] netfs: Fix netfs_extract_iter_to_sg() for ITER_UBUF/IOVEC Date: Fri, 31 Mar 2023 17:08:20 +0100 Message-Id: <20230331160914.1608208-2-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: B896640029 X-Stat-Signature: kh3x53hchwjajqo1pg69xutpj7r6r1d9 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680278972-417547 X-HE-Meta: U2FsdGVkX19wK1PR0vyjxWKOktHZZiaqDjrlxAP1eH1e1XFaHwiJWtbR/MWIwJf4jiUNVmIC9ZXIDDX0ucEvOnggSZrbQ98Kw1GHBNrzK5P6GMMqHWLf6bcD250KY2Dj4yiCkZVvY9X0/NwBnMWvZWJo8mhhueCDOLJucOPQPloP6tPrXlLLcVjCMSPpoijoBpvGJC6QZ5aSc8fYltJb0Su0AzROR9uc4FcnLBJ4pVYV4Ej3y+SRhqTNj+JL/l2tpfw8h4cktKhbWtaBqWQYKeukOSdhXTXbsJUgTWdjD7tFWtmpMYynf6SgtDJj6Aba7FdsGPYHEUN8qDejh8QQCln6aZ84/m71DdOYhqcsdfdW4aWpIpdjlnmK2TJWFGROq8kOEv3eJZpJ6t+Yo+zlY0HLXd5GoyJeMBNrYWNV0xRrxycggmkT9uTGIhASoP5U2/XQp6lMFJjSUI++bqycbzGKf7foCkwLquH4XbFYTBcugYHLTeOMYhJq0vXMVHvn/43cbxaoAAF9dur3cxeaJ66yWbbNSmP8NKN792lmK0fUW9bvy6LMviR0AJxcYwpzlNMUGupM2dhEe8XJfPaWvTlLtUiWHXFoMIsrJN+rQJAe4VJYe0QB6xNhzU+BD7mYecF9XfR8kfnM0lbEJmwho0HCmQseqJq6oVvb08lTSzBwRLaJ+KPc2SYV4/hS1OkQ+ipH8/v/ff1MCTHgjhwDjwGM9dV6FxmVCX/MJDJY5zfZnR5yXXnGesbNWvoWxcXHJAvYalZFqroa5cIWE1nk5u7zC1R0lEYt3p7izyYyBdMX43McF6d7/8fAFGcMuRo5QQ4fnRVM4eh/ectTkEjerYKnJIp9mjIHTOCcHdADx1SxQk1hspkJutX2xjOHXckQLhf1KSbr2qoekmkK/cJNZpPbO/DcyF/JZteAs/Y8kLoadByVW3mf1F6b2KjtMHPb0DKZznfayuyIWTxZxiF 68IB6jxQ fd272WijHaD3j+sw4bU9yiPoJm2QfXQV/NqexRAfyYnBsFcvreOTvtUbYGdx+Q36JivF/u28XriSjAEMR3G0pBT1FeW2hudCfT1QnVrjxkeE53jTvBdxhe2Swk09FBg9S6rf53F374TIVFuuvpIb6THMJKvhUbGHtaraArfTUUsdakJITwdgWs61xmHiIeDsVU7n3QSpgxvOwKU06Oad9UYuVxPc1aeTaIAfOli8D3pp08MD3YpK7chLSrTgGbM9X3bmQIbXImgS03XdLf5AI+YpBGSyhPgqNO5I4OSc9rRmlaaufqAEaKKl7k6GO4995F7Kq/suCQnYW/sEdg94kZMemSkN7cmZPvM1MfA7KA0JgWQU84OtQ7JUkvFSHoC93Cur8nwQ/R2fm26XV/DjHdUVn3OXJOnOSyHL20Ii6E1Mh4AcHXO/3t+RaLkSLYd5MT8a4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix netfs_extract_iter_to_sg() for ITER_UBUF and ITER_IOVEC to set the size of the page to the part of the page extracted, not the remaining amount of data in the extracted page array at that point. This doesn't yet affect anything as cifs, the only current user, only passes in non-user-backed iterators. Fixes: 018584697533 ("netfs: Add a function to extract an iterator into a scatterlist") Signed-off-by: David Howells cc: Jeff Layton cc: Steve French cc: Shyam Prasad N cc: Rohith Surabattula cc: linux-cachefs@redhat.com cc: linux-cifs@vger.kernel.org cc: linux-fsdevel@vger.kernel.org Reviewed-by: Jeff Layton --- fs/netfs/iterator.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c index e9a45dea748a..8a4c86687429 100644 --- a/fs/netfs/iterator.c +++ b/fs/netfs/iterator.c @@ -139,7 +139,7 @@ static ssize_t netfs_extract_user_to_sg(struct iov_iter *iter, size_t seg = min_t(size_t, PAGE_SIZE - off, len); *pages++ = NULL; - sg_set_page(sg, page, len, off); + sg_set_page(sg, page, seg, off); sgtable->nents++; sg++; len -= seg; From patchwork Fri Mar 31 16:08:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DE76C76196 for ; Fri, 31 Mar 2023 16:09:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E06CE6B0075; Fri, 31 Mar 2023 12:09:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB6476B0078; Fri, 31 Mar 2023 12:09:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0A266B007B; Fri, 31 Mar 2023 12:09:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AFD306B0075 for ; Fri, 31 Mar 2023 12:09:37 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5E7C38120D for ; Fri, 31 Mar 2023 16:09:37 +0000 (UTC) X-FDA: 80629678794.11.3CD97E3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 90F4220020 for ; Fri, 31 Mar 2023 16:09:34 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BNYsksqP; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278974; a=rsa-sha256; cv=none; b=2KoPCZDGbwPGrYb40/mPIEFvIWr9Is0qj2FN9XGED9pJXLMtOHDFTEATBXFFGPn6BHhI46 df++NJRLYD7JAawG5QFc4HTWHOAuiqTEufhEEoByNvn+wcFZCmmiu0vKA17+70GRwt6X92 yyr4ZL2pWQumUVq8cihJR+qwN0IiiqM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BNYsksqP; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s7FY6IkB7NRnaaunO8kQ3wM8dgoy+C5dVBJnz6ApgXk=; b=yN7oz5Gg+6I+Bzh2Z/qH/qciFMkiuSCZ6uS/oA3ocHIqmvDxTzUSpx0cQB59H3dyNta/RD 2g5Erozo+/vtrnqUffTaUUwEl2mdwuXUO+hlYBNGQLn6DYHMCGo/ZZXh5ry84gjKaFbgun 4qqsA2VliP0P05SgQsdhcH0rsKxa3mU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s7FY6IkB7NRnaaunO8kQ3wM8dgoy+C5dVBJnz6ApgXk=; b=BNYsksqPYHOKOBr9tACXj7sXpTRoW2E1wSnLIihEQj8k9YY3wDHp2D5ah0BYwmEF0QoBd/ u3TgzE8PXNnRjykIQNpXBCPaKCpPUbjPqVEl7GbTVcPMQe0YLX7V1ieFUqnHol2RMzY4mW S5DhV2Qu8CIv7NjRu0bQUF/33JReNb4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-364-gA1T6ksvPKuG1wGWcmZ53Q-1; Fri, 31 Mar 2023 12:09:28 -0400 X-MC-Unique: gA1T6ksvPKuG1wGWcmZ53Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 12E9D1C05159; Fri, 31 Mar 2023 16:09:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 15A39140E94F; Fri, 31 Mar 2023 16:09:24 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org Subject: [PATCH v3 02/55] iov_iter: Remove last_offset member Date: Fri, 31 Mar 2023 17:08:21 +0100 Message-Id: <20230331160914.1608208-3-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Queue-Id: 90F4220020 X-Rspamd-Server: rspam01 X-Stat-Signature: afnskztirgop97yxcpbxh7j67ksnchxm X-HE-Tag: 1680278974-559861 X-HE-Meta: U2FsdGVkX18Co9ZB8hSS8rE9+MCCOflwd7a5gXAdJPRHCv9ukolwjOsBkeIkjjore8AabYTgGeQhB2xXNEElLEwB72i0cQ5YmywQJWJd5wouq/f50TM5oqpCca3xlmf/hau5lIMwg+Hm5hvaAiYjKmQErnZcYtJEhqQ0xG2A0Sxj55GlVh0v+5MGMhX901xTamFU5SwDUBYYRMAkfatmrSLP+usN7HZWb2eR+AAxwzNQ5HNNNT6Q+O+0nawdOIbfpQNAfNFiuYL1CGmwhpnhbNQYPJQphJBk799ymG4/pNt4COzMS+wtiR0HO2JfFMtiJMGD7hfCojb4LnbJAkofhRXUBjbmgABUHzVQ0PNu4kT+cQm7M7X2Ykcz8/bOrUsQLHk8JI9/mGefcIpwvxldZtTmes3uNj5qEqwF5uGeEoDi9mo6hy84IojdvfFIwDV8hoPnSy+FlEwv4NstcAcMXCi4bRfqNtcQVfpl/noClJPtvgPj6EMZoLQ1VVdGetFJIgCkHUGKgmxFD5fdYF43TC+WQAmitKco6AUBPl+7TCBQNumIqByjmC2oHgDmQSWoZVOX5rJrc5d70T6P8RV/A186vIalgufokfXjezFiJlIWbLo5vZIxiKcRchHErAlFUaZDOnpc55Bm1TgpS4wHgwoUNzBLYyymwLG8xdUEiAwDvB2rjRi3WBh66hY6i3ckJAxqBJOexnjd1UxkZwCb6SBgqc9KLoQkZC8gmWXdebGFY1ZTFpxEs0fUNCzuqVIL1JpBCP/aQbYvvN6JAPib0ro880cu0EwfgEIAlFxRRI4KFsSOW0Uepw78xqy+rxUVE+o6KhR3KxGaCWOi1sLNbv9/hk+eES5t3uA8lBfgo/WXlbSUxMgcna7FIdk1XtJ0HqS3UdjQ/pBXMJG80vqRU+mKmrTexkQVLEHcfm9onoOsEaKurKL3N50zxxjbD2QzDunwZmpjjy48f6t1PaT uSAds0UT KyJfxQ98YXBqPpOo8ogh5aql2UY49c3EFecj/IWccbmCFshUzOVo4KnQDWc9v0dUfrfXAJu1H0HHDiZCVsFQPm5gfUnYNz6nl2RYd7APPnDQY7bB4ny85jM3I3I7T+w8W945qmk99FpAUWJX3Schazhel6NrxEuTsoiSaNBBadNXRgr0jPJvb3m8Uc7P2rQ6qpcxg/4lOLgJTTcgw+g9oj05ncsChLB+ziKh/m9xi6t/BZbxLqph/X2rclfjoUzVqLgF6u4C/HNw+iQFG76MrqotfFLXZr+Po0jq+0+mk0K/RTZqVMIBx/1ApWAGgqchKSFDRUocUg51qwXxp0xu9aEsMhL9MXJxew3X3RzRmtznVpd0Wea+1vhX5k5p4+rKV/J4r6bLyR+DHUkyP2fSzbfxZEqBqP9/nANG4cZNCGRPNjxnHgR3OSN/3u1y6C3i8yL/G9QySOqwNZEbGctVwvDiQ9zshtv5a7G/L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the removal of ITER_PIPE, the last_offset member of struct iov_iter is no longer used, so remove it and un-unionise the remaining member. Signed-off-by: David Howells cc: Jens Axboe cc: Matthew Wilcox cc: Alexander Viro cc: Jeff Layton cc: linux-nfs@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org Reviewed-by: Jeff Layton --- include/linux/uio.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 74598426edb4..2d8a70cb9b26 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -43,10 +43,7 @@ struct iov_iter { bool nofault; bool data_source; bool user_backed; - union { - size_t iov_offset; - int last_offset; - }; + size_t iov_offset; size_t count; union { const struct iovec *iov; From patchwork Fri Mar 31 16:08:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AEB4C761A6 for ; Fri, 31 Mar 2023 16:09:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D42E86B0078; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCBBB6B007B; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B93FC6B007D; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AB8106B0078 for ; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7600116100D for ; Fri, 31 Mar 2023 16:09:42 +0000 (UTC) X-FDA: 80629679004.02.F8B1D0D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id A42AA18000E for ; Fri, 31 Mar 2023 16:09:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hAA9wneD; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2Rqs8gKWnfX7IQPFFFiE2mTLNitfAQsbWxPFIoMuGvc=; b=Ey15jeWYcVZL4+6pP1KyGFaUDwBYTlzoO+PUBr5GZVOyFK1eCeG3lO1mfra6p6YFXC2/C7 y/w95ZpBKfoWzcNNxWSR8TGKFJd8veaGcMh9P+zgfriDe9m6rwCwKS4gheohytzsUwWE5F RX+/dyMd0lkmreQ5x3g7mowfaeLc2WE= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hAA9wneD; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278978; a=rsa-sha256; cv=none; b=v95Tu7r6gb+46wwxykYIFNldB6FIsIvQyqMI9kcSCeDA3Jbmq1+hGlHuEpHuc7eKu1P3nv sYdekD+YAML0fBYR/BzGgf+BH8vcb2Yqdb807vB9+EUG587SF5mAlGM/UdPRK3R28pQeD9 fSyFdVpjIYVaSFRJJval+fQYkQySPug= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2Rqs8gKWnfX7IQPFFFiE2mTLNitfAQsbWxPFIoMuGvc=; b=hAA9wneDvHVU1ZIDDvOVUg1NyfOkB2EusPX9UYEQiAvjxN0k+CECFkIhxU7SxQ1xsM5Y55 QMcTVwmh8ENl4fDH2afhgTuiwev2bELyOA+r+8+UsCbC5UbVV60pAdnjkLgRPgSExceSdW Ks1jboZahUW6nZlnhiWks+lrv1jFOCQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-543-0AgUVjoEOGGNrIUOLBBj-A-1; Fri, 31 Mar 2023 12:09:30 -0400 X-MC-Unique: 0AgUVjoEOGGNrIUOLBBj-A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A4A9029ABA19; Fri, 31 Mar 2023 16:09:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A92CCC15BB8; Fri, 31 Mar 2023 16:09:27 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH v3 03/55] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Fri, 31 Mar 2023 17:08:22 +0100 Message-Id: <20230331160914.1608208-4-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A42AA18000E X-Stat-Signature: 5ogwkzzhed75rmtsz54dzmejttzrse1u X-Rspam-User: X-HE-Tag: 1680278978-512485 X-HE-Meta: U2FsdGVkX18kcEoSlreKEvM+26fI0n7gwChoKX2MOh+z2C4tIkJtTsvAmGSzzMNOwCOb0L8zP9qTlIl0Mj9drQzatCatZC9TvNCrjvZVI8wN1/pOD8Dv0a4wxnOVMycrezrwTo/zQTmMoqIm1RpPr/YXsRXTYXoqce5h3fYXdba2Ak1vIFr+BlPvDu/CmPPLlXxo+usH4gpKd9zMt3TRbzvZnGNo+u/JnllO9raVMQiuC8IvqeYEY17SUnHbPgXHma2VWUWyG9d0HW0NiGx3Ru6gnOIVJTsx459v0Nkv2wmQSAuYEAqoG8Gp9yAw3m6i80zdA0oM5d1yR0Ku2eMUB/4t9X7+ovDWi8YM/6fwldXiPoZDcU+OnYFsoJcODAM9kNCPYmxKq1yHAcfq73OoR6/eyGiLQz0FB8TF32tfTxF4p8IKRc1PnG+g+gAFPKuTNopACNZwIPZYKzilpPTJpGhPU2YkBucAFFICFBI8CA8co9ad+91VYAftNH1dzZ/0jZKvNjI02HnlHy6RdQHcKp8Bb2P6MaeTrFWt7iNkMvhlb1hxLJc9SW5//xYNd82dAkgZAN5aXopuPaImXRbJQnv4oz/lkw0lBIgQhURhXCJcKu7feKwshzHb1jbwBdwj2ajJZ/ynnMRFemMBjsNbsQtBj2scOwSwwr8R3G/G4/i3G5tDIkr/OkSDzRtA4vJwoRUVD5FnGtijmQV7GXgY1TeWZ96jpw/88cg2m8Q7EsVsIe30A0g7+iDP4YRj1leyrXh8+LPEkjfgyr5JPQl6AbMKqtfWWbgj1IBqxXr8tFxa9ulccrZBslyxgx607hw/rUU0ylX73BKRsfjTUzaR6sw63LCmQAmz7oB+2LgZrbQQzB0rNiuPhIPRR+UpeFnuUFl2PCaeqJ0ef2qLwKEvMbkXRcy7JhsozHhkMl0XsUfHHxuf2lhyx82bOnonTOJc7aJC7vdIgqWZcBpkc6X rbmp4C33 J1IsiOSWj9yTyzBH4mNrSCPnXEaVo0HeZ21myXeIIb4ItcUdA0dM3dsKShrIqdkUEBNuG00Jwp0EBqcQHsB4oL+PueH1mYRpqiEmfOSyurzKB8BlrmJWcVhW6Fni4o2MTbPBzc03rtlWmyl3bhw0Eh+w65PfNdxnuhYKT+kCtIMhKEB0FYzOnxTSQmV+EYslEVxfZsx7PSbNl7YyK1A2PyLHc7G+2RaPkLci+iosvo+bLtkeRIaeVIh3CRwmqEPf9dDyCfD4v6oMqcFBAqchVY7vQWc3ggHsh0uyOMgHagew5Y9aj4xj+cMkRtnNd9uRcVhtceB91hTHAqI6yKhDN5p4wN9zLJoFDvswemGHA0UWVHFXuDuBxmqhX0IZG5R2JBLwVVeABwPLXfMSBYJPtEdZq0Jm3SXbssXHB4CPxNFZnrsayIiYo6uV3SYeH7Benkvmfxn9ih+YpZ7EmoZ5XzuWV6+QPgbdnhqyw2uCmBR9Kc8Q6jpZP/Kv5vo6aCHgKrI4b7S71jrAXrR1wR7ce1RqMUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg and recvmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org Reviewed-by: Willem de Bruijn --- include/linux/socket.h | 3 +++ net/socket.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/net/socket.c b/net/socket.c index 6bae8ce7059e..0c39ce57d603 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2139,6 +2139,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2486,6 +2487,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, } msg_sys->msg_flags = flags; + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) msg_sys->msg_flags |= MSG_DONTWAIT; /* From patchwork Fri Mar 31 16:08:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D0BAC761AF for ; Fri, 31 Mar 2023 16:09:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AE066B007B; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1374D6B007D; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF6CC6B0080; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D0BD16B007D for ; Fri, 31 Mar 2023 12:09:42 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A85061404A7 for ; Fri, 31 Mar 2023 16:09:42 +0000 (UTC) X-FDA: 80629679004.01.8A16F17 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id E76CF4000D for ; Fri, 31 Mar 2023 16:09:39 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FTRg5gxa; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zHarMfhYqdBKP5ReoH3//xqNxFuro7P2+Q66shvU368=; b=WH2eHRLao9/0n5kPQiPokfUELx6/GpLzwBJLhhZoXXBE8rDeOLsZcGcCykdqdP1LKrbPOS Ltw01zE7KqMM8iQYy2m1OMS2V92oymUXbBhDfeDu9dlG7PqVxl6dhGUy/QJNG8cDI1NjLe ukub2pk67Panit9Nv1LxSE8wMxRJ0bg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FTRg5gxa; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278980; a=rsa-sha256; cv=none; b=q8KQuhFLQfwECXEDpZ4s6zjLtTpcoJhk2uJuGcNYDwuKZJmFYfg0ukfttPRrJ1mrdOSqV4 gMyN2orQams+MPv0pSFRKpyb8w2BVqjinudPsLftPqkPcCithbYCjqlcnDib10+I1Y4yaO 7hisCwVJGpgQ1mhxc8p6gz5tv/v/FU0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zHarMfhYqdBKP5ReoH3//xqNxFuro7P2+Q66shvU368=; b=FTRg5gxaq803CNPVyd9I3fy5he4+PdF9ZiHB5WEdwqfWkoZMmE4Z/1RZWOAZnIr9XcUjJ8 AmYMCvG2ii1B12lLXUi4/5Fu9vfixjrJUtmTmuOAT25MhtVwf7vwruFykeuSipVl/aoOpE GJN7apZja1lUjqdbAEqNvSdlwNf622o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-577-hhxNPGCHPQCBERrmjvntkg-1; Fri, 31 Mar 2023 12:09:33 -0400 X-MC-Unique: hhxNPGCHPQCBERrmjvntkg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 98114185A7A2; Fri, 31 Mar 2023 16:09:32 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D7882166B33; Fri, 31 Mar 2023 16:09:30 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [PATCH v3 04/55] mm: Move the page fragment allocator from page_alloc.c into its own file Date: Fri, 31 Mar 2023 17:08:23 +0100 Message-Id: <20230331160914.1608208-5-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Queue-Id: E76CF4000D X-Stat-Signature: h7a6ouz8g19sesjkx5brky7h8jqxhjm6 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680278979-833394 X-HE-Meta: U2FsdGVkX1+6mkeTWDQL63wXXQnNJUOiF6YUKK53Ff1ifEHJse40Xpokk8n6Yd30wQCfFyHobl6K0yCuvE87oJfwgoru5syPRwuyxua/fedFEEk+sl3gOsOCF7RPGeUSZs8/bjm76L3Uq6ZQJWh19V8B3leukcLPcsYJ0hiNL28mLkDpZ9xDwGPaAUPn62u2CQOTHXn28OC13sw3r7Ty5EnHt/1aS2GJckDOu5RWOwjoRprty61vyqMQK15OO91kyQtKN/DIeCjWQB5oCq1Cllv9u4blVjg8xgGKwqPhokL13Rdkdwwg2o8UxZXGa8e4qsHHJh7eWdMObVVfdjgr9zymZzdIEI/OnWQ8AaqJ2s1dIsToCNQB2qIrvgLS4+CfEl4WHu/SwxRccxS5zptXTWzZqkQIscd6JoNvJsbBZkVPNpjwOwixb5hpfbVUjo3w7i9UinX1C9FbbveVsau0w2TxAzmed99MIgX+9F2pgpUP25vVRNJLpmyrFsTvvYgF0vjUGWsYGxl8DoNgF6IFR4F3Li9Ci23oFrP9MXcjjnCgkuVkWPjA+vxWnZ81bbvP/uSKqYbE7KxCGB3VgVUvYvcsq6BRwqznZUZ9FctrfzXzwjOAcb7waU0V5adM9ZliBVqbIAcQ7UdF41fgGpLrPGEqeOPql/Ls4cqxzTGCm2Gqwyu/2T4NGY2qAHXpzsnSMKLsma/RLFeeO2t+uDFTXK2j7mQGjdGEn72q6ahiu6A7z0MnI5hYraVF0BEpe1kiUY/vdeUHeIYVxW61SEWwbNGvnb4ERr80bQBSJIIfsp72ZAqXAmahP/mw+PzvBhcxa6pHG848xe4YU0kPIZdJSeJVvXqQOQafeoTBk75Th9XUt4bF/Bz+j9PFwj84MJtbovssal9IathEEB2xVoX9MirAXsmY7H3sRbShZsK52hNY9fhu05RlgNyFbYvXPSMJBfjQOPHE6yjhPRYU7ex A3koabH4 oRsYy+gSRrBfHpiVNQc76ogW2FegfePHkfVtnACjhGuatvATqwUC1oUMuMJ9DokOX8ZJAHKVvtvkQumj3k/G97KC7wJ4POzcvJey0AvrhnFzN86FNjlFyX1qhitbAex6Q8mK2kEgOsS/wHOS43FkRWxPWGt05WnLE3LLViv09QGfbh09VO2PYaxFWT87V204iYmZulN9nX4hPzEZOZOlRo7z0VamRbGAc5TQnugbQV8olKHbvIbg7lO3m1b2OWLVyzDOUB1d0tzs6j8YxzmQ9goSb4mhBg/MYZYW2UmJLrl0Hp/JAvBuU3tTq6TSSxDcv1zHh+Z3+WX7Xi22rvYsjogmLo6DEB6AsaS+MkfvglENndW/to3bAKOiOGvm1oKs1lGKZ4s9hbayIQ23pRrxvaBuTvwtlRy9h7NB//BrvlEZIr6IA0PvWrqf/3vgpw66aoXGW04YztNWSou6JA+zJ40GLKHz1CWgP3KlnB7krJUN8kseCXi11hebURAm4aDZvX2BQHY+WdWzjjwnrIhqFByfHmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the page fragment allocator from page_alloc.c into its own file preparatory to changing it. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- mm/Makefile | 2 +- mm/page_alloc.c | 126 ----------------------------------------- mm/page_frag_alloc.c | 131 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 127 deletions(-) create mode 100644 mm/page_frag_alloc.c diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..4e6dc12b4cbd 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -52,7 +52,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o page_frag_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ac1fc986af44..c08847308907 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5694,132 +5694,6 @@ void free_pages(unsigned long addr, unsigned int order) EXPORT_SYMBOL(free_pages); -/* - * Page Fragment: - * An arbitrary-length arbitrary-offset area of memory which resides - * within a 0 or higher order page. Multiple fragments within that page - * are individually refcounted, in the page's reference counter. - * - * The page_frag functions below provide a simple allocation framework for - * page fragments. This is used by the network stack and network device - * drivers to provide a backing region of memory for use as either an - * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. - */ -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) -{ - struct page *page = NULL; - gfp_t gfp = gfp_mask; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; - - return page; -} - -void __page_frag_cache_drain(struct page *page, unsigned int count) -{ - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(__page_frag_cache_drain); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) -{ - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; - - if (unlikely(!nc->va)) { -refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) - return NULL; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; - } - - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (!page_ref_sub_and_test(page, nc->pagecnt_bias)) - goto refill; - - if (unlikely(nc->pfmemalloc)) { - free_the_page(page, compound_order(page)); - goto refill; - } - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); - - /* reset page count bias and offset to start of new frag */ - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - return NULL; - } - } - - nc->pagecnt_bias--; - offset &= align_mask; - nc->offset = offset; - - return nc->va + offset; -} -EXPORT_SYMBOL(page_frag_alloc_align); - -/* - * Frees a page fragment allocated out of either a compound or order 0 page. - */ -void page_frag_free(void *addr) -{ - struct page *page = virt_to_head_page(addr); - - if (unlikely(put_page_testzero(page))) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(page_frag_free); - static void *make_alloc_exact(unsigned long addr, unsigned int order, size_t size) { diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c new file mode 100644 index 000000000000..bee95824ef8f --- /dev/null +++ b/mm/page_frag_alloc.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Page fragment allocator + * + * Page Fragment: + * An arbitrary-length arbitrary-offset area of memory which resides within a + * 0 or higher order page. Multiple fragments within that page are + * individually refcounted, in the page's reference counter. + * + * The page_frag functions provide a simple allocation framework for page + * fragments. This is used by the network stack and network device drivers to + * provide a backing region of memory for use as either an sk_buff->head, or to + * be used in the "frags" portion of skb_shared_info. + */ + +#include +#include +#include + +static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) +{ + struct page *page = NULL; + gfp_t gfp = gfp_mask; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; +#endif + if (unlikely(!page)) + page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + + nc->va = page ? page_address(page) : NULL; + + return page; +} + +void __page_frag_cache_drain(struct page *page, unsigned int count) +{ + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, count - 1)) + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(__page_frag_cache_drain); + +void *page_frag_alloc_align(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align_mask) +{ + unsigned int size = PAGE_SIZE; + struct page *page; + int offset; + + if (unlikely(!nc->va)) { +refill: + page = __page_frag_cache_refill(nc, gfp_mask); + if (!page) + return NULL; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* Even if we own the page, we do not use atomic_set(). + * This would break get_page_unless_zero() users. + */ + page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + + /* reset page count bias and offset to start of new frag */ + nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + nc->offset = size; + } + + offset = nc->offset - fragsz; + if (unlikely(offset < 0)) { + page = virt_to_page(nc->va); + + if (page_ref_count(page) != nc->pagecnt_bias) + goto refill; + if (unlikely(nc->pfmemalloc)) { + page_ref_sub(page, nc->pagecnt_bias - 1); + __free_pages(page, compound_order(page)); + goto refill; + } + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* OK, page count is 0, we can safely set it */ + set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = size - fragsz; + if (unlikely(offset < 0)) { + /* + * The caller is trying to allocate a fragment + * with fragsz > PAGE_SIZE but the cache isn't big + * enough to satisfy the request, this may + * happen in low memory conditions. + * We don't release the cache page because + * it could make memory pressure worse + * so we simply return NULL here. + */ + return NULL; + } + } + + nc->pagecnt_bias--; + offset &= align_mask; + nc->offset = offset; + + return nc->va + offset; +} +EXPORT_SYMBOL(page_frag_alloc_align); + +/* + * Frees a page fragment allocated out of either a compound or order 0 page. + */ +void page_frag_free(void *addr) +{ + struct page *page = virt_to_head_page(addr); + + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(page_frag_free); From patchwork Fri Mar 31 16:08:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A9B8C76196 for ; Fri, 31 Mar 2023 16:09:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A385C6B007D; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 887486B007E; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D87E6B0080; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 55C376B007D for ; Fri, 31 Mar 2023 12:09:43 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24970411C6 for ; Fri, 31 Mar 2023 16:09:43 +0000 (UTC) X-FDA: 80629679046.22.15EC070 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 61F478001B for ; Fri, 31 Mar 2023 16:09:41 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BCaqYbIH; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278981; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w9vYmidIh5WhninpbPh1DpHK4Q4l9z+4/ehUwmRVCvg=; b=Vwr/8eM9GSv0mNE2CswGlkPqeq85E40hYdEXqfkbEj7wasBdeAR2rjsYE7sywjTfRtnqbA SU18R/N/sED9vul3VjOY9ccSP9iBj+v73RUeidwczrD5fZ8Ewlk/znmVT0YyHwB6Hbhfa8 gu5eiAG15NuLHB0O9lpbWt7WPYwKmSU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BCaqYbIH; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278981; a=rsa-sha256; cv=none; b=z8BbOx16qv37TOwz6zRAT8yc5JjnTX0xPNAnLUjxbvtvzGESTkgM4y3aJrvXu8+3pFXXVG SH8iu0a58tYaTDRG3fb3nvOT1wZC3Ebofka098tMt1AAXWDN+WO3NvGydH8KbAq0DU4PTc X6j9Xxjwfr7ZOd+NKoZRB634G28qj6k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278980; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w9vYmidIh5WhninpbPh1DpHK4Q4l9z+4/ehUwmRVCvg=; b=BCaqYbIHFUMcKud3wSStlZv5hun5jVrLw5zjrdW30axWOtE13UPNvX6eQCvF3wMTDZcUJY SQroYYCYD98hISIUnY9XTRssawrMdxdnuI7RajZbvJK1But9qu9nxpV0CVpeY/h7+D0x71 Haf4J8XikfKmNb9TGsgA7LLDGXQjEI8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-645-elJaY9qoPaOnZnoS3texTg-1; Fri, 31 Mar 2023 12:09:36 -0400 X-MC-Unique: elJaY9qoPaOnZnoS3texTg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E2D6185A7A2; Fri, 31 Mar 2023 16:09:35 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 39DD614171B6; Fri, 31 Mar 2023 16:09:33 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 05/55] mm: Make the page_frag_cache allocator use multipage folios Date: Fri, 31 Mar 2023 17:08:24 +0100 Message-Id: <20230331160914.1608208-6-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 4f89x9kz5u1hbokd84sxnzmuk8r4r1fa X-Rspamd-Queue-Id: 61F478001B X-HE-Tag: 1680278981-756549 X-HE-Meta: U2FsdGVkX1/OQxkn8Yht1OPYAeb3nCyO+eMM8LblXNxaFQ1txCRonMWp8onmZ/O9Ed5rKd/d95dcLncpqmw+WfWKIhJ59nQTWT0OSHrc2FWkgKlixGTz/fD+4Ud7BoK6N+R1xTaD+9mgovcNAX/L1xanoRdc2Io34P2ZIJuz6POHMzN+WqyCkeva4tBgN2a2UOjyKiQ7gYefu+KKjNQCtkwrJKmjSBUXe4i7KhkvvPYFL9H+bdsMpIzkI9XejbG+/eEZixePRq8bDpPLjxG5h2VCFd5sBssQWbt4bRYaFeBFpvDsViZ8P6YUEH469/S8D9wEoyV5P3UBfDn+MBjONaKXVi2AQ6NFAUfIj07alVYSUpEWt2Q6OzAfHwVazNPjLEihfMd0D7slYpNNI8acY+C1i8zQrn4rP+L04NvN3FX3xAC1JdIB9+O8Ni6EXwaxqKGAUkZXnBZuX6o+lWpCtEI3AhSxRa4+pxyEAxl14j0NetEbyIvL0nZqmBuNKykrszw8GrVoIOhtW4+iOdguG9NA9iXiVdMid7VbonqBNZ4aNzAFmISxOeYTdgrpuSV6rx1dwtyICW/fm1zVsJ5OrzkqUkSq+hXEhB+q1+7W5fLJgQqXxVPLpwvSRTK7r8r3ZSGfbfex9agQFMqBQZ/c5PUN5xyjtjKxdQ3wsQmDAV13dNNwdV845WbAn/1NsuwIZAkz1tCppOVaR8iGzJLj50w710oxWAkJwcmT+938vDLqZomE4DGl1wofx5CCXb9RWUSeIXhilTVrFWhT6F0AHITLVArflwjAAndT8yGULE7iE6qUojfJkyMj3LQWkjXiqzdLIelfqxRrM0dpg7+qgPwzdWsao6EuQG6etG2H1YyJ401EKPcWrFvRlU7nzAj1t2sTpusYhlOTIxKGj7OwgoCW9bLwJBDeN3ZIIRuJGKc9R6r5DPHbPMfwuRFgGQC5XVgCepe7VCQG4rfLLn3 yrnybxiN eez6dy1yiy8WdnUgogzBEQRaH+IevKsb/aPiKhuGtt316EoDrro8GNEzqby8E+3v+rymxPPHo3iq6xpPnJeml3pnuQxVrV+m+iZWp7LytoPF3DVjErts37LZu7DpCpo4WaGY8O01/p9p65A3V8BTwp5zWduic01vq8tzHwj30m6lISHVUMaOI9VpAVSnK/Kt8E0nHCYeYWK7GnesamnymShWCs2FQFLy1setk53HxTRY0JS0Q+VL4yx55npvjV0OXoDMTEKzcDEE3syzpc9mb0YdR7GRJ4kouZaVKaLtqW4gLPc0lPS6hK8IuifO6H78g3lIO8IWxBOakwxleR3SMQboxWkZJsy4WnsNJfgb6KJJfYj9ZpO3lel8qjEJPQjkbwW5BJB3ClYVIfYc22bVJSvZZy/qSHrcxhzPiiyEP6ViH9x2wLtKrcgQj6mllc4iLYgYw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/mm_types.h | 13 ++---- mm/page_frag_alloc.c | 88 +++++++++++++++++++--------------------- 2 files changed, 45 insertions(+), 56 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0722859c3647..49a70b3f44a9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index bee95824ef8f..c3792b68ce32 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,33 +16,34 @@ #include #include -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -50,54 +51,47 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with it. */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -107,15 +101,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= align_mask; nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -124,8 +120,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); From patchwork Fri Mar 31 16:08:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8816DC76196 for ; Fri, 31 Mar 2023 16:09:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26DEB6B007E; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D1276B0080; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D5B6B0081; Fri, 31 Mar 2023 12:09:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E35CC6B007E for ; Fri, 31 Mar 2023 12:09:47 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9BC50161087 for ; Fri, 31 Mar 2023 16:09:47 +0000 (UTC) X-FDA: 80629679214.14.453A663 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id C95974000F for ; Fri, 31 Mar 2023 16:09:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UFIOp80y; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/nU0M+T1DCbw3wZvEUGAhwFOIfw7rmUPAJ5895bBp2k=; b=h4Xwvqa3rrnu8UKRBvOKqhbXTj1QC3z3zYV6kr+O1uz+fjgeeXnI93sWv5HnMA77jYebMO DOxTOBEKZRUoMjXkNb6R7MY5149e6U3DW2vcpqTigtHD44z+vsvB12j9qibQQ9dcUBqOUX tVH05IxQIwgknKW33mL1hn+dFpDGV1g= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UFIOp80y; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278985; a=rsa-sha256; cv=none; b=wRWfrhctJHgNc3LXRQwvVpua8ExPXnya2CN863rk9WcRd54d/BeQq/lnfR6VT6XDuazRjR y08O3iIbvO84tev36FrmS0kK5ZjEUu/mpFNtc6G+Qc/drkVyzt6/NvE2lZgIpk1jqblWDr G0fCz+Rjbu/xgHaOdkhtQgSCzeHAepI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nU0M+T1DCbw3wZvEUGAhwFOIfw7rmUPAJ5895bBp2k=; b=UFIOp80ySS6CV88hF2dfHXqvOuk7j1onldLHFCtJ7zpkxdT3YVEbSxCNRCSzJ6kFtt4U/R hroRUEt9D4QHweni1q/kZaLjlYq95u9tmWc6plkDK9jQY4GlULs7N/2Zk0Sj6XrrBYGFxo 0xpDs7hz487lCj5sAJO6Er4Az7v5AcU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-22-9O5OK0CuMdG6Z0qaBSmbcg-1; Fri, 31 Mar 2023 12:09:41 -0400 X-MC-Unique: 9O5OK0CuMdG6Z0qaBSmbcg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8590E29ABA18; Fri, 31 Mar 2023 16:09:39 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id D5EB1202701F; Fri, 31 Mar 2023 16:09:35 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lorenzo Bianconi , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org, linux-mediatek@lists.infradead.org Subject: [PATCH v3 06/55] mm: Make the page_frag_cache allocator use per-cpu Date: Fri, 31 Mar 2023 17:08:25 +0100 Message-Id: <20230331160914.1608208-7-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: C95974000F X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 9fafdonosrfjoai67y75jfkj1p31fg44 X-HE-Tag: 1680278985-834659 X-HE-Meta: U2FsdGVkX18cuA+dtqfkHj97/PVKgMcTSSNbebwgstjsTamO5vlsheo36QOjxkJHglThSr/iks138LYJApSKRcS9gM3inzEyYRBclyPIduN6AAFYqxObyuWJRuGEgkCoGZMak6Ykvnj+rNEcQt8JeRmMwxSMwa/EzYW7G4RfjeFOLqzwDFdCeRF/e8ed2U8Xkm6UW9nrC8o5nuGCoZmHxxjv7m+55ywctyAG1jZfxtJmM1QXiTRYzu5NY9jxf2pP+uUt3rofm6sM1mbmsA0sJ7OcV5Br75z7wtY4moZ7kzCazxtr/QybP710Z9MjG7QyYYVfQCR17MP1M04nYku6AkM3h4YPjEMyC62lbYLo1wWk2RCf6+irrCYe8c23tOnaBCMggjKlQe4edgHYsVnagiclInojJALktM4d4YhQ/uDdn9GPyn/7oB5tuxmKz7fUbXl7WJS7qZ9x2lBCLN2fuvez9McO34zf+QaEev5nzmS8rpfvF5cD0cE9UcQyvjG4h0b7LWYzBawOCiZE5XrXrWkqh8L/ttxpwLP67jB2vqzk0N46BdsIE0cnPfxF5be2l8xLiZBm0v+UpEf+5QyjNsyaC1DATjKiYKrt7HlZ0w/KGXnXvAtDmEjkrHa/lwoZOU0qq4cEhik88N/xpnu+/2tLqv9jKmAgLv6yoXr3mISmMVEK7yaCxtzkp/7WdN5Aulwb0603rkLO+kI3Bh4oq1CxqRf+KbbqDJusbvHzyDfeiAujIRcoxrJZvkVtwbuqOLtqVNuLTwI/NqIrvLTGb/XEjxkXjh94OCt+aUVFn/GO+NNSld07lx5O/JwxvG4DNpfL+lw1C9ygeA+y+GtXf2kx0jUXe0bhc73mgoXeWM6sR8LU5NhlCMIvR08P8Z7sYiMhxtoqoutYvMLgc8Enl1Yazg+7XVnwUDb9R4YfYb5H1tSQ7qnwquz+IPbMxcx6/Zh2bnrPNhMSZRlaSm9 JjeZr8MQ DdOjpxUB1ts4VNhiLBR36wQckBmbhpq6fNvoVCryCLOKaPFNTW7RFSHNYa2RCgFOiRTfAk0ic1ahmTIWnBPyrmK7dzdfJVjRNWFx6w9oDqs/jLTrKxnDIrBYsJNwvV5MwsnzcSWDKOkjwlZU81y15cgZopXcWT82XGpr/bikDW2KP+TBqvA2SOc0fYyLX0QZ09RZPlwh+2Sf1K8PB0TFD5uoPRBchvv2XFyqHujpfPKtCH/KdVQtuwx0s5/61JASgjH7b4XIL70ESctKMGlxIkbQG4xFfxXkmFg6YIFU+0qR6Rd0SIM5IixR+dwhzT5HpqokAigDqkw4EPW1iyjUYYwofpDVFHeYqB9UuYgQkjB0KmGT5AvI+LbvRqdxBh/ZNMzXSsWa0lvK5lgWnRakEGjgtlWCJbimXEMWXtogc6R1T3cF/LLOuCW8X8WR8PYjLg0lUcRrUHbFqZJZSVwxGlSzAE78N1qelNjvKdvCE/d57LU0PvSEp3AJpki+YqHhU3v9pZK4LzajK3+5Wt36Lb7LuTVeIwu1ur/qs5CxFDLMohCPHA4WWX6VJ8eE4nZTzDQEr1yQi6NR/HEA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator have a separate allocation bucket for each cpu to avoid racing. This means that no lock is required, other than preempt disablement, to allocate from it, though if a softirq wants to access it, then softirq disablement will need to be added. Make the NVMe and mediatek drivers pass in NULL to page_frag_cache() and use the default allocation buckets rather than defining their own. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Lorenzo Bianconi cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Keith Busch cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-nvme@lists.infradead.org cc: linux-mediatek@lists.infradead.org --- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 19 +-- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 19 +-- drivers/nvme/target/tcp.c | 22 +-- include/linux/gfp.h | 17 +- mm/page_frag_alloc.c | 182 +++++++++++++++------ net/core/skbuff.c | 32 ++-- 7 files changed, 164 insertions(+), 129 deletions(-) diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 69fba29055e9..859f34447f2f 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -143,7 +143,7 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q, dma_addr_t addr; void *buf; - buf = page_frag_alloc(&q->cache, q->buf_size, GFP_ATOMIC); + buf = page_frag_alloc(NULL, q->buf_size, GFP_ATOMIC); if (!buf) break; @@ -286,7 +286,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -297,20 +296,11 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(entry->buf); entry->buf = NULL; } - - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -319,13 +309,6 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); } static void diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.h b/drivers/net/ethernet/mediatek/mtk_wed_wo.h index dbcf42ce9173..6f940db67fb8 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.h +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.h @@ -210,8 +210,6 @@ struct mtk_wed_wo_queue_entry { struct mtk_wed_wo_queue { struct mtk_wed_wo_queue_regs regs; - struct page_frag_cache cache; - struct mtk_wed_wo_queue_desc *desc; dma_addr_t desc_dma; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 7723a4989524..fa32969b532f 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,8 +147,6 @@ struct nvme_tcp_queue { __le32 exp_ddgst; __le32 recv_ddgst; - struct page_frag_cache pf_cache; - void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); @@ -470,9 +468,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx]; u8 hdgst = nvme_tcp_hdgst_len(queue); - req->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + req->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!req->pdu) return -ENOMEM; @@ -1288,9 +1285,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) struct nvme_tcp_request *async = &ctrl->async_req; u8 hdgst = nvme_tcp_hdgst_len(queue); - async->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + async->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!async->pdu) return -ENOMEM; @@ -1300,7 +1296,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1311,12 +1306,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 66e8f9fd0ca7..d6cc557cc539 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -143,8 +143,6 @@ struct nvmet_tcp_queue { struct nvmet_tcp_cmd connect; - struct page_frag_cache pf_cache; - void (*data_ready)(struct sock *); void (*state_change)(struct sock *); void (*write_space)(struct sock *); @@ -1312,25 +1310,25 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue, c->queue = queue; c->req.port = queue->port->nport; - c->cmd_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->cmd_pdu = page_frag_alloc(NULL, sizeof(*c->cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->cmd_pdu) return -ENOMEM; c->req.cmd = &c->cmd_pdu->cmd; - c->rsp_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->rsp_pdu = page_frag_alloc(NULL, sizeof(*c->rsp_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->rsp_pdu) goto out_free_cmd; c->req.cqe = &c->rsp_pdu->cqe; - c->data_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->data_pdu = page_frag_alloc(NULL, sizeof(*c->data_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->data_pdu) goto out_free_rsp; - c->r2t_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->r2t_pdu = page_frag_alloc(NULL, sizeof(*c->r2t_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->r2t_pdu) goto out_free_data; @@ -1438,7 +1436,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1460,9 +1457,6 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..b208ca315882 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -304,14 +304,17 @@ extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); -extern void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask); - -static inline void *page_frag_alloc(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask) +extern void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, + unsigned long align_mask); +extern void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask); + +static inline void *page_frag_alloc(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); + return page_frag_alloc_align(frag_cache, fragsz, gfp, ULONG_MAX); } extern void page_frag_free(void *addr); diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index c3792b68ce32..7844398afe26 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,25 +16,23 @@ #include #include +static DEFINE_PER_CPU(struct page_frag_cache, page_frag_default_allocator); + /* * Allocate a new folio for the frag cache. */ -static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +static struct folio *page_frag_cache_refill(gfp_t gfp) { - struct folio *folio = NULL; - gfp_t gfp = gfp_mask; + struct folio *folio; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); + folio = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (folio) + return folio; #endif - if (unlikely(!folio)) - folio = folio_alloc(gfp, 0); - if (folio) - nc->folio = folio; - return folio; + return folio_alloc(gfp, 0); } void __page_frag_cache_drain(struct page *page, unsigned int count) @@ -47,41 +45,68 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) +/** + * page_frag_alloc_align - Allocate some memory for use in zerocopy + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The size of the fragment desired + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and fragments are broken off + * sequentially and returned to the caller with a ref until the folio no longer + * has enough spare space - at which point the allocator's ref is dropped and a + * new folio is allocated. The folio remains in existence until the last ref + * held by, say, an sk_buff is discarded and then the page is returned to the + * page allocator. + * + * Returns a pointer to the memory on success and -ENOMEM on allocation + * failure. + * + * The allocated memory should be disposed of with folio_put(). + */ +void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, unsigned long align_mask) { - struct folio *folio = nc->folio; + struct page_frag_cache *nc; + struct folio *folio, *spare = NULL; size_t offset; + void *p; - if (unlikely(!folio)) { -refill: - folio = page_frag_cache_refill(nc, gfp_mask); - if (!folio) - return NULL; - - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + if (!frag_cache) + frag_cache = &page_frag_default_allocator; + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; + align_mask &= ~3UL; - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = folio_is_pfmemalloc(folio); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = folio_size(folio); + nc = get_cpu_ptr(frag_cache); +reload: + folio = nc->folio; + offset = nc->offset; +try_again: + + /* Make the allocation if there's sufficient space. */ + if (fragsz <= offset) { + nc->pagecnt_bias--; + offset = (offset - fragsz) & align_mask; + nc->offset = offset; + p = folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return p; } - offset = nc->offset; - if (unlikely(fragsz > offset)) { - /* Reuse the folio if everyone we gave it to has finished with it. */ - if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { - nc->folio = NULL; + /* Insufficient space - see if we can refurbish the current folio. */ + if (folio) { + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) goto refill; - } if (unlikely(nc->pfmemalloc)) { __folio_put(folio); - nc->folio = NULL; goto refill; } @@ -91,27 +116,56 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; offset = folio_size(folio); - if (unlikely(fragsz > offset)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - nc->offset = offset; + if (unlikely(fragsz > offset)) + goto frag_too_big; + goto try_again; + } + +refill: + if (!spare) { + nc->folio = NULL; + put_cpu_ptr(frag_cache); + + spare = page_frag_cache_refill(gfp); + if (!spare) return NULL; - } + + nc = get_cpu_ptr(frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + nc->pfmemalloc = folio_is_pfmemalloc(spare); + if (nc->folio) + goto reload; } - nc->pagecnt_bias--; - offset -= fragsz; - offset &= align_mask; + nc->folio = spare; + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* + * The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ nc->offset = offset; - - return folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return NULL; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -123,3 +177,25 @@ void page_frag_free(void *addr) folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); + +/** + * page_frag_memdup - Allocate a page fragment and duplicate some data into it + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + */ +void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask) +{ + void *q; + + q = page_frag_alloc_align(frag_cache, fragsz, gfp, align_mask); + if (!q) + return q; + + return memcpy(q, p, fragsz); +} +EXPORT_SYMBOL(page_frag_memdup); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index eb7d33b41e71..0506e4cf1ed9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -222,13 +222,13 @@ static void *page_frag_alloc_1k(struct page_frag_1k *nc, gfp_t gfp_mask) #endif struct napi_alloc_cache { - struct page_frag_cache page; struct page_frag_1k page_small; unsigned int skb_count; void *skb_cache[NAPI_SKB_CACHE_SIZE]; }; static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache); +static DEFINE_PER_CPU(struct page_frag_cache, napi_frag_cache); static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache); /* Double check that napi_get_frags() allocates skbs with @@ -250,11 +250,9 @@ void napi_get_frags_check(struct napi_struct *napi) void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) { - struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); - fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align_mask); } EXPORT_SYMBOL(__napi_alloc_frag_align); @@ -264,15 +262,12 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) fragsz = SKB_DATA_ALIGN(fragsz); if (in_hardirq() || irqs_disabled()) { - struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&netdev_alloc_cache, + fragsz, GFP_ATOMIC, align_mask); } else { - struct napi_alloc_cache *nc; - local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&napi_frag_cache, + fragsz, GFP_ATOMIC, align_mask); local_bh_enable(); } return data; @@ -656,7 +651,6 @@ EXPORT_SYMBOL(__alloc_skb); struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_t gfp_mask) { - struct page_frag_cache *nc; struct sk_buff *skb; bool pfmemalloc; void *data; @@ -681,14 +675,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_mask |= __GFP_MEMALLOC; if (in_hardirq() || irqs_disabled()) { - nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } else { local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache.page); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); local_bh_enable(); } @@ -776,8 +768,8 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, } else { len = SKB_HEAD_ALIGN(len); - data = page_frag_alloc(&nc->page, len, gfp_mask); - pfmemalloc = nc->page.pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } if (unlikely(!data)) From patchwork Fri Mar 31 16:08:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E10AC761AF for ; Fri, 31 Mar 2023 16:09:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFCA06B0080; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A82136B0081; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FB786B0082; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7E7C36B0080 for ; Fri, 31 Mar 2023 12:09:48 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 119761A0DC6 for ; Fri, 31 Mar 2023 16:09:48 +0000 (UTC) X-FDA: 80629679256.10.5222332 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 3F13520029 for ; Fri, 31 Mar 2023 16:09:46 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=It4jZ7Yh; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278986; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HlTSQeuXI3aut1lZSX+tmY2WXPBg0R+eepIbeSsvlZU=; b=Wygiyj1vMn8SaCCtJRV6H36ozxEeyoPooSkLK0qR29Rwl2+vNPoPWXBs7YrMJsJIgalpMs 6oLDs6H3AMB8irlD329nYRBc/ivWO/3a28I8ywoJwdFoAjBAGl5xHlRGCkFb0BB74n0d1x ENPXEUIIFmsQ2gSPYh0AOrbynLUYOEc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=It4jZ7Yh; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278986; a=rsa-sha256; cv=none; b=8a6kCTGDrZYNDWVGJvqrudVXyPNnbK5ajcvZb/tA7MKITY1EDoxJWBk42UuL+LmYIck1Rg y2++QWJuhl6QcPzPQP8ps2iwI5vc+Hy2mZ3Ldrfk6kNkEV0ZxRSFMY1Qa8qBmidtPlhXXG NyOqRNl/fIHMilw0X23qWo2gCCUgE90= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278985; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HlTSQeuXI3aut1lZSX+tmY2WXPBg0R+eepIbeSsvlZU=; b=It4jZ7YhbyNeo3iy+vE6Bk+c+zwyh5ZuQRp4bbumptgQPjuoML8pArMLz88q/R0s2SHHm3 mxjGk6zCcqOok2B8ipSZQDMmvrE5K1sPmQBgYVHF6if/NDxtk/VnAMQHeM+QR1lIVxOqps to2kaEoomvEQoljk193jYnyG89i/mzo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-4An1B-SyOQeBgjPKwSEzAg-1; Fri, 31 Mar 2023 12:09:43 -0400 X-MC-Unique: 4An1B-SyOQeBgjPKwSEzAg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0B59D101A54F; Fri, 31 Mar 2023 16:09:42 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 27382C15BA0; Fri, 31 Mar 2023 16:09:40 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 07/55] tcp: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:26 +0100 Message-Id: <20230331160914.1608208-8-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: 3F13520029 X-Stat-Signature: fc8m9prxqgfepxjfnxw1db44h5rsd65k X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680278986-718454 X-HE-Meta: U2FsdGVkX18DPL/xwsp7r4R/AYqix4kDPlpRuQU6E6Mvr6KI8U0vNCU2hwye4Im23tIclKACejRhaiKf4N5NeB9D8Mnw32BOEYnuG9rvInBbC2wGgG+OqVGgO8Lf5VvkOcVzJheF5OxfuHh7m7T0d4CMEboowzzdVAIL5ELFPu1yv0FoD2vlNPOIXixu2dPHCLNnb41MsHLTul3i3p1VAEmRkWyjdZeooJnjzLwIWo9eWtrBdLTqAC2eu/1ZbpsDHfhWXTVaVRCVmf8/jyaOnIWA0ivLXP2uRRL/hqr22vz09Wj7EORt4ielZB4T4kGSL4ZEvOkTNB8wG/en6YC+poNmDe5VeYlwu+XF6tc+ITilayfZvpP+TiN96FPpsMKwo371JDmh424yDplkZiH/ElLB8RHddB9TsHQd3Hbr2g/QpC620epvIL7oN94msdwdl8UcexUqRtwaIrQlYKKhff8kzi0bV7tlg0PDNQRKA3ekGkGrXxW6qdIKODg80SU9hmN0KNFkSHkjHbD6GwNFXhQZCOQKLm2Xtqw6fUSXsESrUblue1LBpQ4RASt7XU/SAk0cJdJZfXqlfSVOvHYvDXVhA4dUdPA0RRH6mEr/5VgFU5zkEdpCJMV19HI6sO6/5Jtli738nAIhBdbuaUPxMd43tfx+jg20hNwkgDzC3dzyMTayjSnDHBX8I0YefuoYBZKNkfA5w0qylcDIiZKNQ2VfhwSOGoptGypOJm9PPSse/TUI8XD3dOuobJ6p+lckfUyNBYhXzYG7XfO4V7m7Qj1+LqZPdztnOo5fZ4jkBchSnwigS5f8BPZWWevrsjlGXgZwPsix7gmsfzMQmGXWCoRXVo8iTKwU1nc9ml7kT9Aehjo1xyTaJHi0Y3Tssihm0p3xzVJ84iRNE/vDoVNRkTyicLwmZY1rLGvAkvogCU9P+QXhrWY8kXp2C8oUHk6vtgGVjCkqQL+Wd1SHYgk LYVpXb2B vQMBYiSJ5wCYmdrAtdQevd9jRwKcCYG+bDI0Y0HD0oO7BVdTh7BaZXgIp/2gb82Nffh8rUtnwDRlTrf1gVkfVrSKJLkR1UZDMdI3+8oATXeEgxj8Eds9dGWM9EnQHghgpMw1CPca0IRi7Mp8GFXDP63eS1EeBIPhnt5R2nlpq7abU8OKllRPN38M68fkH7DnGd9ohkpl9wi4g7yfMHIoONzyKYTh1PXrjipMUXFr9ywg3/J9lD5T7ELi5yPVL+rm5DDpw9dbPs6sqULbk5nqR2p5jYeZJ4PU8WMRn/sPxd+VsWtiV6RkXyEcBVxg1dd0SDjCmWyZm8vt4DyQWQO+5AZARoTNMF2akN9JgOPnicPm0OvR5wXXwLEMd2VuAYfu39TjMcWOIXG/bURcXUgt6ct4M76Yi3sUOZ0I8GJUV4J8iryh6WHE9ZexgWjDWQ2kalCRq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 67 ++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 288693981b00..910b327c236e 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1220,7 +1220,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,17 +1231,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = 2; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1304,7 +1309,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1345,7 +1350,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1390,7 +1395,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == 1) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1411,6 +1416,54 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == 2) { + /* Splice in data. */ + struct page *page = NULL, **pages = &page; + size_t off = 0, part; + bool can_coalesce; + int i = skb_shinfo(skb)->nr_frags; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, + copy, 1, 0, &off); + if (copy <= 0) { + err = copy ?: -EIO; + goto do_error; + } + + can_coalesce = skb_can_coalesce(skb, i, page, off); + if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { + tcp_mark_push(tp, skb); + iov_iter_revert(&msg->msg_iter, copy); + goto new_segment; + } + if (tcp_downgrade_zcopy_pure(sk, skb)) { + iov_iter_revert(&msg->msg_iter, copy); + goto wait_for_space; + } + + part = tcp_wmem_schedule(sk, copy); + iov_iter_revert(&msg->msg_iter, copy - part); + if (!part) + goto wait_for_space; + copy = part; + + if (can_coalesce) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + } else { + get_page(page); + skb_fill_page_desc_noacc(skb, i, page, off, copy); + } + page = NULL; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); + } if (!copied) From patchwork Fri Mar 31 16:08:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79BADC761A6 for ; Fri, 31 Mar 2023 16:09:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 182D76B0081; Fri, 31 Mar 2023 12:09:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 133776B0082; Fri, 31 Mar 2023 12:09:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D736B0083; Fri, 31 Mar 2023 12:09:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E13126B0081 for ; Fri, 31 Mar 2023 12:09:52 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BAEE1120D40 for ; Fri, 31 Mar 2023 16:09:52 +0000 (UTC) X-FDA: 80629679424.26.2AB34A1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id D944780026 for ; Fri, 31 Mar 2023 16:09:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="LGx1/oyd"; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278990; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4TWX2aSvGOZ9Qof5Bdvg67w1lje6zmFD7Oc0vC/yB3w=; b=Ok50b5YRkMhZOTodGjCk236HW2/mTp201QMjpb00qI4SGU1monuUUFoDlb7UwTDnhTOkdZ cMqUrPNFeZulGuoUpuECf9fLvLghUPeee/t5a6M7sZFeAhW6RBJcAS57zF9dZx9/U7voTH xt4JMyFGY7e55JbRxuhDU2pzqkn8p20= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="LGx1/oyd"; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278990; a=rsa-sha256; cv=none; b=0jXBCbvp+oClTu399oJ7aYE0yePUDfeePIT3cY4jAuPp234Er5wvYW7i6kzkNVg5XtUdo9 STgBTJf3F4YM1Lg6OrFJlx9N+yqtYc9TKBkENjicQfVSthuoHog2mxLXvEIolXbuNgiPgy 5tIa5scQuGi7xY7P8vEN8JuhiciPwZI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4TWX2aSvGOZ9Qof5Bdvg67w1lje6zmFD7Oc0vC/yB3w=; b=LGx1/oydwBRFOMRKacTWCY/Jp6hyFPRNpEcwUHEwsB/CwiZZz4AGiXRQmI8qVt6qspuUTz pB10ldEqzoYartoOqbja+xWCQB2j4GRPIIwTh3AjqzDVl68jJyQ7fhqW6YJ9A2nOY6G/xY DUxciUifpbEFhpu9G/Ika9Fkrx5xodE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-496-oMyBzLXdOyeTBAtGXK8UWA-1; Fri, 31 Mar 2023 12:09:45 -0400 X-MC-Unique: oMyBzLXdOyeTBAtGXK8UWA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A431F85A5A3; Fri, 31 Mar 2023 16:09:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id C01FC492C3E; Fri, 31 Mar 2023 16:09:42 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 08/55] tcp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Fri, 31 Mar 2023 17:08:27 +0100 Message-Id: <20230331160914.1608208-9-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D944780026 X-Stat-Signature: pr8999gyu1bctmiengdq44ugp5nc47kh X-Rspam-User: X-HE-Tag: 1680278990-809573 X-HE-Meta: U2FsdGVkX18yJ5ta8KEv5wMr0lwwzyf//LQIfiKd0FnwmP0if74WDgeOAnT3gyGeXp9x0cleISlyTZvMWWF6AoK0tnt7YTebvRpT6m8wyH3Y09CrIdT+qQ2e6aHP8x22jPpsz13CA2kFUTzEAkC5iEjEjJrUu9rpxmTL+cdJ0r0IweQlnbDPgePTCRjvqYkU/M5/cmvHpGMPiD5ix8lWVBSdykDbF1LMJDMEZqvxaCQIzyMlCTQ1OdRUyD4Wal39bW3LNPC2gfPplNU45RaA2cn3VgWO7S3tCmspNFqZ1tfceFEH0UmAldgqmSPBTAG1jBx/pB/605uk+7QvXQcQtuztmaGbV1WRvqPwJUB+tz6P4mwejRI+bv7rEqEJ4JkUqnZHFV2nZ/k3X4/A27zD7pSQtW1/qa8KoscZfDGXL77c/wABNSRT67GXr1j93VPeSCxykPkjPR9XQpiQtg04y/SGtjnu23YfaQnf/weC5orQ11Gily+XDq9CO7kqhZYOwfRjsQ6Rgm+/JMo4TZWztwkk2V00LaF5xIxIlAm01e6AGCXVWLvKK9+R8IYgUUGLy432uJtF+AbsfzLsY3nhIeym33pSWM/iCaK5ewoodWvKOhUy6u5Zf+Wo+L/21TNXUWt8nLMBR53Vive654fn3G0+8+Dt+7Riss2Fh/LlgksyikYKz2znfWwGI6LX7KCiLbJRruUkQkz2iKVlN5XMqoc/i2MzM4KP9ZYwfDy15r7FHuf3UNMcGEWGjGdW6ItxtczU21Vl2iHnBFgZoPZgwqGAlZHndmhYQUEKrNEvcaeihNpfySKkWs94+QfYw4REznWWv339KP3+8Md6RRYYeSp2utkUMalTjbs4PGto7M0IxhmjGhabEsQoksp2HX/2FkWyrtVPufxVqf2SmfWRnI+/N0JIuxxyd1F+k5jPyZQ9tpYa0cfKkeFEfdBs0Aun1s/CwE9e+JfMmoLNgMj 3fKE27Nk We3aiGx692jaSWPt4LSN5hA1SWnFPKvdz5LCC2kLSGnoqJM7CMXNSLxFasx1Sgl4j7e8bJpWhQKYH+wS/bzJFyXLG/sWEeEU5jJEXGbEaK03sxxyx4AHl7w2slW63UqPv2XtRBqBPD9hFvFaBLeSYeDfIRVILyweWbG3q5etMPwODJXlORt8WGRJtb3e2C7qzxSmACjO1YwJNCikNzd2Y6hVs1UX0ps5E4u5qlKaOUvz8OJ22Zwy+ToJ9U6aMvHlQOWdOYBMRqKh21banarzlP7ponft9CO1gU6GFI1NmKusmJSgtGXjVF1/gragaKICrjsM1l9Ot0mQ2M3kdhT410jjTITTxT1hNV69GYwQ9DLS5zAPawbmO0uc2URo3yq0rCppySihOMRs+tUNPV6x6XKMmzwaxOIFqojUeud+yRzaStRe0U5LPadCBxhrrt5InQjFi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make tcp_sendmsg() copy it. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 910b327c236e..6ef0518eb706 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1417,10 +1417,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; copy = err; } else if (zc == 2) { - /* Splice in data. */ + /* Splice in data if we can; copy if we can't. */ struct page *page = NULL, **pages = &page; size_t off = 0, part; - bool can_coalesce; + bool can_coalesce, put = false; int i = skb_shinfo(skb)->nr_frags; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, @@ -1447,12 +1447,34 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto wait_for_space; copy = part; + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + err = copy ?: -ENOMEM; + goto do_error; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + can_coalesce = false; + } + if (can_coalesce) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); } else { - get_page(page); + if (!put) + get_page(page); + put = false; skb_fill_page_desc_noacc(skb, i, page, off, copy); } + if (put) + put_page(page); page = NULL; if (!(flags & MSG_NO_SHARED_FRAGS)) From patchwork Fri Mar 31 16:08:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3743BC761A6 for ; Fri, 31 Mar 2023 16:09:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7F2F6B0072; Fri, 31 Mar 2023 12:09:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2FE46B0082; Fri, 31 Mar 2023 12:09:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD17A6B0083; Fri, 31 Mar 2023 12:09:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 99EA76B0072 for ; Fri, 31 Mar 2023 12:09:55 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 75AEC1604B8 for ; Fri, 31 Mar 2023 16:09:55 +0000 (UTC) X-FDA: 80629679550.20.61B3288 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 7371920024 for ; Fri, 31 Mar 2023 16:09:53 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hCr37aVU; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nhfSw6vyJXoYjtAsHCy4fHcMYpuQf7adDQC2jq05dOE=; b=30PT1piMjHh687CgGv3dvZCcV1IMMcysjj4bQMU1FwVhCxZoW6YBMwIT7ZebHh1nmpWF1t RdnzgWaI/REpWBmOtc1wE+wHRlGhk4nCEBkWWOSVMTVIhqsHRt8SY0Hyf8wonypb8+914c 8dYdnJ+qT6VkdSIvz/QNIaPfBKV5GLQ= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hCr37aVU; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278993; a=rsa-sha256; cv=none; b=RA0ZoneeXiIbyEfSNEiupYpyBIN98IqcSHAo/Zye1VY1E24H/44nAXCpHYpVtwJmt4qeWv fBjJakKRE+adD4oW/f/GVTnDc+7dWrME4bDG9E0TDT/4E5COyTQnwy3e6dIKsvlUgQCASm KEWkbdAVUnHQrkRWIKDAVt8pQfgtONo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278992; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nhfSw6vyJXoYjtAsHCy4fHcMYpuQf7adDQC2jq05dOE=; b=hCr37aVUtPACBBFImvJBiWCgBBjbLF1k/TtE+YHUVzYG7iJCavuU1s7WJBMhP+P89eaq6g Z1lfj3Ab5exvn2wXPJafHKHscuV7+gz9KfxOA5SRre6pII9bNhNyMiF5KTCYXCXQGeoaIJ 8gmQZemSLpdbat2ucQuZ+uCUUnK+12Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-138-rSYPRKkjNEe0Qp7x83krYQ-1; Fri, 31 Mar 2023 12:09:48 -0400 X-MC-Unique: rSYPRKkjNEe0Qp7x83krYQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2AC8285A5A3; Fri, 31 Mar 2023 16:09:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 46B784020C82; Fri, 31 Mar 2023 16:09:45 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 09/55] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:28 +0100 Message-Id: <20230331160914.1608208-10-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: nkcxbnmtc8cu3q5mytcuo9ze8nqbhzfx X-Rspamd-Queue-Id: 7371920024 X-HE-Tag: 1680278993-971390 X-HE-Meta: U2FsdGVkX19tr9+dLe3iZGogtBn4VQIiZqe/ycKZlIlDSKlBz+Ajn5LA1+iVgWWOVWUnXE3/pOB1ZOqPmALXS7OzrCFGxZ/nnZ2LJSgGQQFAaj+JENzCfUjbNLl0gBhQrXyncqnsIyYpQfFx+TIhJ5kIsYHkmDDUErU6cGolRAdoldoF752ybgEtn6AobvPR5Ng/2xZKnjC29UvuzudrJ15/OaQe9q3nKTHQF/T1cf4pVP/qeTrA8QDg0BaySEiTjgkCvGD3bJnQB8xOdH8d3/WliXpYm3JPxfdkFXza0hOwXIAM3ar/4RxFaByyH1HYqQXGaWmS/RN9UovDF4jmOuO5FZ9/7fStURavHrfE+XL0nxOS3+T6+yJwhOwsadGk9OQp6iT1IrmnokIrw2ZIsnYSRFnX1xsOJWv4JT4D4dO4IT6YBHHfazwAV1qrmzvDD9FjiPTm9pDi2T/pqf7exAJcaP4Jk7aUI+ozLM1LAVMFeSfMPEosBkyObO3RqY8jKs9s2FS63rcgJ1GN7BVEy3EneYdVmc2lKcta3+pPmZQ/40YuUIK6WdFkBoqjqHqjT241hKzl11pjgxWxLm2ATNd5IkHNnsdsk2YE6GgqekQY9jrMURXsW7/0hqavS3tpeproqjYLgWm19jO4E0LzzRPtheWAg7ENEB7gPeChjJKxM+K/0v+bmnpSGrSsBXfbNGGLS59uu8Mk0rWZvxjw+7xXOThtEdH09es6wJwhO8G3dBQeHrrBhmbMNsiL7X0VoV4jXTytCSr1LLTj3Ud2BX6XbhM/gPQwaSpXlhgCblV4hZeNEzGQ2OLWHw3SlK2e+UHflyTbDWeo91fQ5nHugwOedd2dl2aw+h62V62nk2Fp1N15XZuZp5Vdd030jMnMUofdh9zUNTMdXfJgxkzeY5RZrdGEAU4wu/VdKiHUw4tL2c8EWP892ttMK2JiiOxSMFadMGxNC3/6WWEiK5I oEsbgoKY LvM0VdRSqRlnJXXjxeGLgdzBY9TEZwTF7/Jl+AB7funtyyy1C/qOazLwJrRUNxMESYtkgSi8B7gEzS3Z6vplZqUOW55ldxXYsf/ns2PepSd6wVQwwPZ5fA1p6CLbDejq8ymHz/BOQO5x9WEOVBstDomLF6zKpTt05gGbhGU6XOcTzY7LF5vTNxfkekYLDGbkQ4AQ/2vzAHql8LObvhPtD5fYo8sm/UJkFL924DwpGKEVtaz9vmZYM5BVCK+y58olNB4mC9IuR4cV9IdJ+kS/gurHcFumjJ2CbW2GZi7qcEsLDxthHNE71G1vbKul7n8dcBMMs75dMS85NVNo6/sD18ly+oJPnichj1YzKHA07VOi9/xgKqxZxzh+eoyQiuOpIpVB5eY2SimNpz5s4K9fxJ1dN4/p5vgIcG5wG5c0J63JuknTeu3v1ikhqse0ADILGjxpo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 6ef0518eb706..edcf3a60c1b0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,163 +971,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Fri Mar 31 16:08:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11976C761A6 for ; Fri, 31 Mar 2023 16:09:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A837D6B0082; Fri, 31 Mar 2023 12:09:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A33FD6B0083; Fri, 31 Mar 2023 12:09:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FBB56B0085; Fri, 31 Mar 2023 12:09:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7DB196B0082 for ; Fri, 31 Mar 2023 12:09:58 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3398D411A6 for ; Fri, 31 Mar 2023 16:09:58 +0000 (UTC) X-FDA: 80629679676.05.08D08F1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 2F3C2180008 for ; Fri, 31 Mar 2023 16:09:55 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=II2y+WEu; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680278996; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=3vBhfePFmWWwXZCEZFAvn7MEpFz9in3dQUYrtNXGnEfeBBO/kUDJR6hh1XaObvgXMi0Kyo eR6uxZqq++F540iIRhw9+rjX8kKBWrtGaXAhweJSA/spyVyMBOhHuMPhryyZW7Dg0whudQ z6hIfkaPvqpgv8azyJ1NpRLJFolPOsI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=II2y+WEu; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680278996; a=rsa-sha256; cv=none; b=CRuAet671UjzTNJzW5Q68ra+AlHQ3l4X6cgj4e6xWzgGqV5o/upB3R7cMWYhAW07+LWqci H9oYcjPfQNhdcKhMutwnuoLPEae+Kd39YKRvWiFiQpy/4MFYXd/WOM1tY96Y1UNN8yrA5u xm8fhsL5yDAHUYvaUfwn5RoOKa8O9fE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278995; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=II2y+WEuRpmyNF09XU0RvehiVZB4qlJYjAIux3AixgyEV4COdtMQ8gnLSp85dTgF+iQz6w tmGA+/b5gLVfl3cobtlaCMY2KEOSVI8GBqd3r8fG34vafdCXwlf9Kot0haOLzJ45jJuJs+ BPcAFCLs7VxII4oStIHQcMuYvNeCAic= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-316-ldaH4sxnN8OIiGvOqC_Jew-1; Fri, 31 Mar 2023 12:09:51 -0400 X-MC-Unique: ldaH4sxnN8OIiGvOqC_Jew-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1C5C6101A550; Fri, 31 Mar 2023 16:09:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DFCB151FF; Fri, 31 Mar 2023 16:09:47 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH v3 10/55] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Fri, 31 Mar 2023 17:08:29 +0100 Message-Id: <20230331160914.1608208-11-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Queue-Id: 2F3C2180008 X-Stat-Signature: 7wcrqiiagwy3cs7agb4cw5r6uosqunjg X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680278995-820448 X-HE-Meta: U2FsdGVkX18CBfiTMpZ9JLwX/Z0t6T+ff5uWA7DnLd8kc/V2G7gHM3lGjWK2Fx0gYmx9yCbJR/Wz0vDxEBRvNHvCRjAvfVmMi634xyTyN5VF9mAqKMor+qlL2v3O/6F3JuOXWeeW16erKCRmP1UTA271XgxmKtJFVCshL5RZlHdXtkauGUFo0IuCZwvbWSbT/gxCDqrrVcI+bD6T9l5ufOumfbHpBADWGjaEHCNCC78w9wb1gwNgl44W8e6IWNkgMsHpsI0mTZkgePetCWmQSGUgHw+CaIe+2YziuGo4kZOIKT/JRTcoDrQG21Q9dz5/PVFsZ4XLbfuxgHz4fnfFSjPHnsutXb1/4ozgixzLLrZrGcac9mEnwD3bdeduYyXRu7PSDdf4zdp0Qe3eiSHAk+e7OFx6kddZirdLfkBZQ5nmiXaQ5SH2pyNc2zfkc5PAk16J1rYiu9YeoSQMT5V0x0YOf5ye2RyIn+sjApQnuGLLnY/7HxbiUTfXJLfAgymWr+MFtFc+GNWDW/m3ZORBLuVyFS2OnwEaTqNBWODjs4LQ7xoEvcjXeww7gwTYS8dVZuoUbu2IqBtoWm79BQ40y3L0WNeSewK2adsCcKTC6uug8GoRbvgvOQJHLAlL/ruPSAOnTihLmRB/o3SOqTlzNTVkn/80EPj4XswITA6WrVIv51GTzyzEew3fsUDqDLP7wqlM68n0HlDd/P02fQql4fJUwRx43lfrGXOordzVITUJuKJLYm5nX7lC6Eyi5xUEW/Jy/Nn9OSDksf9pEqfyi+mCVL6+dn2U8t57aDy6saeDWFIYNivm0R+eGqlkxt2LGYyT608vo5Y/8c8RznrUWGKDdkFTqJM5IXn8KNYCUk1rGvIiufkG+xk/irDHkmintRl+fheAB/K4lBmfkA8Fc/cmMHEx6bWFw7bXrSr3FdbB9WaccFhi7WTkKjSdjQRDcithjMiiiTmXArLqaNp XCXfDYud SJrj2Qf+1PYAYfnFlDo6VBc2y2GeAGJiAaVehZeOIDK0oMpNQ33iJ5/USBTOr6M5bfa9JG9qryIE6FtJWV0w1V5wXyh85hhPueEgYGJTwd02AO4oTVkTFM6TzdK3KtVXVXRPJG3FwLyisfcO/8xmUSXOP71PdlJPF6tc4s84F+/uKu4m3ujDySRqdchOIZahXGWlUTe7IfKsa+Wr3+QU9ef1YX/VIRLhSSyoXJpKZs743E5Uy2/iNhAi7XgnaK31qmbWZ45aDthhMam2K/8Ypc1h1lFCk5SqDepXZbdc8ILgaRFGmDSyguAbaBAfA3Vmlw9f1Q4tM61fRuV9z2YD+n1sjsxkx/7Cum54O3TXfR22hYlM+E48Svkahb6wKwn+aOAD7N+XMjKOVVLLkSPe6qJV1Ozb8lLrSFdUz8qVHuQvVHNxn9HID4Awj6xiUA9FaJUfmfqdjBMwOXFQ/gPbLVydQdKeN3kHXQBbqG0YrVCfCRDQUkt7QtTKLKdZ4lk4cnjqcWrxCu3hJTsGvP0lKymdl6q00+AxcgMwLG2hJa5TXdBCDCGCg6vd6Pg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index cf26d65ca389..7f17134637eb 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -398,7 +402,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Fri Mar 31 16:08:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7223C761A6 for ; Fri, 31 Mar 2023 16:10:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD3C66B0085; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A849D6B0087; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8844B6B0089; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 66B966B0085 for ; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1F4C3AC8C6 for ; Fri, 31 Mar 2023 16:10:03 +0000 (UTC) X-FDA: 80629679886.08.D9D52B3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 53B451C0005 for ; Fri, 31 Mar 2023 16:10:00 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GxPDgYvP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279000; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=JS8sap7aSPgbWI2GWyGRtKrqIMm+anRmpqdji1TODY4P17WEQykP/ja8qi4Ovyl//491aK BWTsdSKcVucnQNw4NOMSzLTQide/m4UrzaHYuwG2fKduD6xjYxKelXzT13mdpUdsBqHE1z n6DGPcPd6j2ie6KWjfRYMtMnHRDxE6M= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GxPDgYvP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279000; a=rsa-sha256; cv=none; b=YPW0Tq1hEF/Vhpf8ogE2qVR/W4Q/1QOqgZOkzML8Im83d7o5gk73IrVVuLNfZQCCZZyN2D 9FU37XvyQabXJE4UPhXfeyPAuBdq7YNwjCty8lpc/UtpksOmSIWW56al/laZa/FVHdnD78 rTjzi5hyfqeMWWcFRYjzckoZAVi6kyc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680278999; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=GxPDgYvPRX/ariiNcKyqM9dHkEBU9W7pRsxWUUYgFgs7xyfIR5LQGps9QuEgLeZLPhPUGm u0cQe8KxCQVWS7a34Hhm2PHvYIUpOrpR932woIqrC+/YbO0e3Ghmg3nZaNXyZdpel3l7Ov UYXQI9amaZo8/Kgyw+Q5XqBlIixa0eE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-68-mHwARYpIOEOh1orxqULtjA-1; Fri, 31 Mar 2023 12:09:54 -0400 X-MC-Unique: mHwARYpIOEOh1orxqULtjA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3C9393823A13; Fri, 31 Mar 2023 16:09:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B30A12166B33; Fri, 31 Mar 2023 16:09:50 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH v3 11/55] espintcp: Inline do_tcp_sendpages() Date: Fri, 31 Mar 2023 17:08:30 +0100 Message-Id: <20230331160914.1608208-12-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 53B451C0005 X-Stat-Signature: sr66gjiub98awwp1q8hjrto4cqegskwg X-HE-Tag: 1680279000-188830 X-HE-Meta: U2FsdGVkX1/7E8AncBeCdPMG8WCbeXAeLAQWWpeoBT5nLhMnPW1azizfbLAC055AyghL9SZ2D5YoQTbybVCNCzMbZ58Afn57WatH0aI5nCATPFEbFRJbvsLw9dI3cW/g9QvkJgKIljT/bFK0CIbm86srL4Q9i0dDvAq1I8h2PJtwom6U86aHbXqVtws3CBQdw2IN4WbjSJJWQbfd8qgTxX1ro8/P05AZPaWfmUKeVzUlFsTi5L6HQbxppEfc6nHcWvqXgioNG5fQt20yLAjr5BMYj9AOh78/cLreBAYC8qSJc/fyL11LW4vjUN4qREHkIy7Lfci+h3SmvpxVI6s+SZJkNjKQBZSq0WLMtP+VzMBBcSraY5xqJApAxLZKrK0wJb1HlZRsXB1zg3SSC1Zc9BQVS3OlfBxi84Waw2cHuL7kcP+ESoH0eswOtNFEbORq0YlX7xS3CuBDhg19bMQ2/tj6aaeblsFPYthKeJng+6kxCmrZNp1egOP0mQBdQcu7Pc2N2Lpwu6O6rZUu4aIoo9xgFrmN/NjRVc5HHMzvoOAjyoQE4nsdxmiX8OLlQj2NoN6B9mZey1AiVnY/CYnpcwzzAdboNNp9EdPUSYvXcTm5WILOsm1cZQQfHR9gBMxPjWuj+l7bCr98aR+H/AyAX9OhIirYUgWSsur01L98q1dyApKNTqS0y48uhjhlfVoK4vc+3DV/3m8dHlAmg0fZkL8KgJRBo4vTZAyyoiageqOIg9QlsQgsisr2LbrheIdR+MoxNVj59hP6DP5oYE4ngG6C7+ggtjAdsM8ZmtiaZOBkHr5Q0g7f8Knt2DJ6MX4STDyBztPZXkzUviadzelF+39g8Urx/+Nd+K5Mb75uEOnrK2SkGqdDQ4TH7Zn2Z92yEsoTE9stY3oVWI+zugyB+5HRMQlfbmRwbxZh/D1zC+cmVDu+Da3cnuu1JODJec9MX1McPNN33+J32WM24I7 gx0g/zu1 QIbsSrx2Hl7kCiyrgz2XseJ0jUMucqnbnWPL5+VQ8vWTtsnlDkG89r1b5EKjDigQl5eq9W/pvTQMKchEHDr8dd8S8fFUnFhCe1YQmyW+i5l6PdFeT+42WU8IgYMiNWaiJvYJAhzchhpyixsP4yCEZanW/bhI1F5ZO2tX3ExL8OWFJCRGWcdWqUi3TDYZZDsTekSziykHBwxpK33JOW5qGs56QRO5qWbuorIk70X9CEUysA/FfzN5OS2wmd6+i1FzDqBME1Ib17T+UhlqJ/rL8UcTd+Lwv4LYZEhN/k+jZ0C9VVQtdVpYWQ9GKMXoRACNv1Emcx2Cq3F+TIH2tCg8Rajv6x396tSvOYQVmKcEtbvLBcgYntDkwPG7PX0Tj91t1SI6+/bEaUzTPWHAWSHzt0B/3RsJX3c22P76baHGwbJN4OCQ/vyygh4mt2+tcdcWv8Zkb8LVrvN+0dP2Er1tT6bqMV8jaDbgU/gxi6DyfVs5lHS4H9i6iVc49sRkzJuKFLB4CDSs5VpkQR+dwXMMa3uLB7dobhlPMEYSQhqZtbQspCzmXTjvoeBz7tA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Fri Mar 31 16:08:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE673C76196 for ; Fri, 31 Mar 2023 16:10:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 750116B0083; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7046B6B0088; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52B816B0087; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 43B586B0083 for ; Fri, 31 Mar 2023 12:10:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0A7704047B for ; Fri, 31 Mar 2023 16:10:03 +0000 (UTC) X-FDA: 80629679886.03.D50B747 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 31288100028 for ; Fri, 31 Mar 2023 16:10:01 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LClZ2Cll; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279001; a=rsa-sha256; cv=none; b=jAT4JSix1aQ+PdxzhFDkZfc/3rahkyBDiMjG8ucwQmPmq0PNMJ7iw95Sx0ftmypJvY6jvG OR+YcOpRUxI1xY+lezizSGFZ211glJvKT+FngV70lBf4EkozZ4zS08S902lPKzLZZp9PZg +/IFuOACCHi7lknEAsArDxSA0z75Eo8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LClZ2Cll; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l3swE5hQADS6B6hDKlip5qVV7zk3Jb5E005A1dmhwhE=; b=NUufsbzzVwK+pQ18WWDcPZpOylGU7I2Lk/yToYwubo71D/buPuOdJvvgyoEoODUNJoPJsn 7VY+cPGU01ouUGCiyWwEb2agN8Pw+6Fup9DzhZ1lATFePev2VwvJFQItq5PujDArIFVRGq YvcDyXEKhnYgTmEEM7Aj8IMLlvxgB4w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l3swE5hQADS6B6hDKlip5qVV7zk3Jb5E005A1dmhwhE=; b=LClZ2Clld//Hl7/ldgYdmlrQQ/V41MPfHdB6aUophOod3ufVMLLotcoYGyZBmF9KNhljFX 0CSXe438fIHpuB1t41xGOAbJ76Ee+pI6JDj4c6bOYbhg+u/qZCJNXKAy2EebHSbkwJ8c6B FdAA2HhB0YJThgRCOMjeLECRcOS9nsU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-606-0osVrac6O6KfpaQgEswi9w-1; Fri, 31 Mar 2023 12:09:57 -0400 X-MC-Unique: 0osVrac6O6KfpaQgEswi9w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1A0F71C0A58E; Fri, 31 Mar 2023 16:09:56 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 049E61121314; Fri, 31 Mar 2023 16:09:53 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 12/55] tls: Inline do_tcp_sendpages() Date: Fri, 31 Mar 2023 17:08:31 +0100 Message-Id: <20230331160914.1608208-13-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Queue-Id: 31288100028 X-Rspamd-Server: rspam01 X-Stat-Signature: qai1jypi8ea8y3g4dznhyazqp8yxgwjp X-HE-Tag: 1680279001-867723 X-HE-Meta: U2FsdGVkX19YNmT7EX97uFkSDw22Ljn64veNAhe/4Y2oY0yZ917vx72xlzTnfP557mBGg0rfHcxILk3EHkzGpJp7KaIy79EvrSMkmsEPBcxmNQc4Qts3MX2HRHpYW+rvX1dvuD+kYa8CBvnUaajBKNuZl+VnWI+x43F+OrKpT7p73SW9FrwJa7IXu+R3N1CI/nxpSS3mMvWFcUDjtgYLWduDqvDfvDMwCLZ+7ccmfep9Y+jl7gPkUEkebQCInksaV/ZqVoi9m4QQgT7VCnUhVRlqWK/dDyZ8W3uEvFb1Ovy+rcsdukV9x4N5eKz2SB8KC/n3HyGdWsgfH2+jQZwFvUvFlX7iamYWvJSCyG0mUp37plkru5hWHUHM2mGmfeGpYge7YTMD7eYsRgES3Wzm4UjqCUHfy+FFqSriNLt3XUFZpVy3Qra30H7p+Yuw47VXHUnJE2g9USXV8pyKqEZxKg/ztO4EC1K7xqP9a10pmX14f/B+QmMVCY400rfgYAya0lXnysZiqDY5jSirT4apDZcaI7qu0OqSJOWZ7wrSyO165Y1ORQS7sfc6egsqSG1pdXQQj80fQjSgsMT2hz+/WyMbI6EM5gOnH6cTCiPxEfvAVbxPODM7jPB9JPslaHcLdzpnV6KkIwX0DqBzY/43fq2MU8gCuTUiMUsfyOqzgpg+JtzElmcVP9tPUeV8Jp8G/+T1MewrBcjAIfoOYQlEW7kGZfpD1j4mQYTMM20iahMqycdYJrPyoe8tY+3rgEugqj9Uq23xnvVxzAGs5iA4GjFkMDnNyNSoIbYvJaCZzL3w+Np2G78DNXTyVf6Q6z9EbMFG+ekHZLz9iYKgGwYUmFnFoecr4tX4AcvG2DOTItefTqleOI+CC9rDxQRQUqGRNyavcAwRS0srvjpj1eorH1bnBkZx4iAJD/Bw+szXBsyfGzQSMLC3LFJ+v7DqimrLeMLmxEev4GrZLCkaMbh 5Vlx+2yI DobMmwWBPSbQ05QVing4YwpZfsEZHoXlY6sZgqQzZSUGtD6SJ6vRn7cdgLcBmLI1M1NbypgdcIiVC4yBbAOCX4FONy+wq5Jr8i/j/tL7H39U/h+mMW6WkWMtogXCw7V6XcK8GkDaoUYcj7kc6xvPYD2Xnpr+5KPNIsQNDsKJl1Jl5qNK1krQ3P1/NZet8TSPIUMwdIehWuA4fsDpehgzVv/vOtdUic5I7IG0+pF1qmze9rYBtd6zrK4I6ZzStI3mBWPUFGPMrww3XNx8zSSTHzbzW33z8hiXHPLHbL1SRi8TMY3f/1pDXStvitvhSwVQ8cyJUrNMTdrv7+A3q5MlCR8sElJ6+YgUmLLy8SIDeenS1yZZgc3bX+ZgQcmwF7ibDM7uBaGcIDyFbyKvNBsnqRuSb6WmDOyYGWUVNQfS/Nl+J6VetQ0Abx+BK2HCFlosjbdzDtdj//MP7mXHNSwrLFcZVtrqSnCJd8R2bKOKz0EMzvcsTy/2wwySW/dAcdtvt9tmwQLGQo1za1jm+fy3z8zjbg5f31TtVfbsyNaHUmMrPA0y+NfSJTEDs0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 154949c7b0c8..d31521c36a84 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -256,7 +256,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 3735cb00905d..35b2f7ee2fa3 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -124,7 +124,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -133,16 +136,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -154,7 +160,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -168,7 +174,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -246,11 +252,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Fri Mar 31 16:08:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D215EC761A6 for ; Fri, 31 Mar 2023 16:10:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 727ED6B0088; Fri, 31 Mar 2023 12:10:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D8396B0089; Fri, 31 Mar 2023 12:10:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52B056B008A; Fri, 31 Mar 2023 12:10:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3B11E6B0088 for ; Fri, 31 Mar 2023 12:10:07 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E66F81410D1 for ; Fri, 31 Mar 2023 16:10:06 +0000 (UTC) X-FDA: 80629680012.20.06F6500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id EE7C240011 for ; Fri, 31 Mar 2023 16:10:04 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ch76O0xF; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279005; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pP5V7G7GRqo/82ZWWApO+0CTJP1pdqvbg0H8r7V6l6c=; b=f17OEC0g2r5N7EvWMlRgym4yR+GjpxbiK3pWagtsTtsHQepgl+/t3gKvB4i+swyK+1Rmzj 2BfMMx5SZQp7QygnQk7g5XUx9KJWRBKMUcx2w1FLLBFb0WTF8xjfep1eJ+Q0OxbZN4Oo0W eIGEB0KtAhGuyzkR65X225B9dCPIfdw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ch76O0xF; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279005; a=rsa-sha256; cv=none; b=iHbFwfjprLFkDSPXYgj/TqOz+oapuWBp8LHgVh0o+f0FQqZ8ygQ4eqaP+c4/seJmmyHco9 zCbKixTb4XucpZFc3OpQ6rienOeX+BidGBPaxAa2Wzbgc7k+MKmcodPfkRSORH88h2q+67 MC59aWckejchJ4db8k1jIy4COzvpMHY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279004; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pP5V7G7GRqo/82ZWWApO+0CTJP1pdqvbg0H8r7V6l6c=; b=Ch76O0xFNTJ0c26mjTiKgJqTubWtPuC1ZLST0HRjXmoSKZjnw0A6KJJNtOqQ5jglnzO0iz ZuF9LReHPxxCdqQ9NiEjT3ZP6UIQAoxIQvZlnSeQYzSzaqbtBFKRYGZ1uRpnvVN5DniBkZ vfO2BJYpvcC0pvsSybnCR8VT4EzFkko= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-642-8gYrt4CYNxeFfvIQEREI6A-1; Fri, 31 Mar 2023 12:09:59 -0400 X-MC-Unique: 8gYrt4CYNxeFfvIQEREI6A-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D9D123823A0F; Fri, 31 Mar 2023 16:09:58 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id AFCB8492B00; Fri, 31 Mar 2023 16:09:56 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [PATCH v3 13/55] siw: Inline do_tcp_sendpages() Date: Fri, 31 Mar 2023 17:08:32 +0100 Message-Id: <20230331160914.1608208-14-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EE7C240011 X-Stat-Signature: f4tju3qxujuby9kurf1c4qmapw4p5ym8 X-HE-Tag: 1680279004-469503 X-HE-Meta: U2FsdGVkX1+yW4AKRk3OJHKhdyKHrLYm1jMTkUjwm0c24wEHtQUL2tUI9UIysmgB91U3X5CmpaZB98lBkRnGDIztWiczO/fD1tlGYlpV6Di8TVhl5FCeW+erieZyCRoOpaSiU/8PRr+mqtY2su5pQ3UOjheI+l768fnmnsmWAyniH0+5MmXzhGENpRljwbcjtYapKM3ZdSMSuDoxog1rU6AQlo99zUPIcBWepHM2ZNwx3GHb+sv/2lznL4BI3SQ+XMKvIbWeBGJI/bn6IHBoTot3MefaNOXVuR6aZUF0TEw9osfW1Ryn78CRELufPc5+/VSJkrJZp1K1IYmNU85cD3CfxgUs6Fe6i9IG7vhGpjbYw0zZJuHXfpn25S0jyUAPWCDSMGf6I97P2ONM1c7AB/72hzHE5fPO7FkfkNwCjFMgLOUZJ4zPXoc1yfa7hkKSZ/uP4uuMaUwH4z7Zhl24YYWIBQsTGw7yJw07pGztUuDfbpzn8hwrHj2njBsKmn0LY8w3Sr6Cd4ZKw0Gef30P0N5wXFAsnI+kMRagyteXjqjlRg4psAzFTDCR+1BozW/0PWPs6c1vBCBAkV6RelBSmxvPkm6tDhLnThAWnM/6QdRB/X5o61mozqadm+XjPH9OZ3WD8mAsMMqZpV5oelulOpvULVEsBxVKkZ4g03jkiEjxg/wvaFxHfcWV+5s2yxenCtT6V/3HPJ111LVWZ6yYQKsNSX46dXsBS2naxVMdnbNIFw12hlOzMR7WSrYJ4tquqi4SxZbWo9lMwgcpbqX2Jp+DGR4hqKk/EHYt0SQ4QpGhA5JDtOq2/sKvk2ZpV6ABjGR6TSAqbBOxtEuMKZcO09k7m/za/wyjD4aQqZbtlV+DKAe6IrkS0EJqkThSngOdVtCkR17qa+24B76sGk4duC7SpiWRsTped/p6oI2jKkwk9qx8JQmmDOhIydBizWmFOH5TmcsVIQrdt5oljUO bhfruFCd ZK4njVg0aVsa7fin16Vk83SnXR/Ngmu3qwVOTfE1jFNImjN+n7iC5vbNPatuuBnngcAStZ0DgDDfzPmTzBpcOogTfcCGdwi8HlTkdypMSl5m64VgT6DlwP9Ih0bym/W960FvQVJ3S6u0mYdfBZrRbynjW0/IA82TE7nBDWk/jY/FvAzYYT1tEFngBu69r2+7E/Fc2Ytb11XdYYPTFNvy7SJ4wp2cz0R90M52ZJWyiJwLfDrnR+a/cizGrmtaGQ4qmBZSiwlB7vvvj02lx79s1BstJPPYzWXIlKqlQtpe3sQO2nd7rJdQCUF5bjKz4SPzU9ftrhZjMneZCWz3tmrOHhxoXYgXsmP2KKMZGLOCRBzNM+3nsIvBoxzad9ayBlMZ1Su6hUYOuUc0RZC8ce3RKUwC9KAnZfrFTf89mrbIFNW+9dICT4CN0RIdpi4Vd6vudDZrQ2GKlG/97UZfEWOrmFW2vfcYberrb/uiepXcYAPFSc+xW3FTGVuZpIdzH/vdxCwApAipVcWtkaYg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 05052b49107f..fa5de40d85d5 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -313,7 +313,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -324,20 +324,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags = MSG_MORE | MSG_DONTWAIT; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Fri Mar 31 16:08:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27073C761AF for ; Fri, 31 Mar 2023 16:10:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79DF46B008A; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74D256B008C; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5ED9C6B0092; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4D6B06B008A for ; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 205EE160487 for ; Fri, 31 Mar 2023 16:10:11 +0000 (UTC) X-FDA: 80629680222.21.64B09AF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 4851D40010 for ; Fri, 31 Mar 2023 16:10:09 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RTQfrxu3; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279009; a=rsa-sha256; cv=none; b=5WHySariQbTAlc1ckEhig6L77WMLQW90PTv7cBQkCHfPJFkDC0XkUI7zQRExGMaUvz352c OgI4x5Q23H+boecdcb4fZ1XvQzvJy/Q+X3u220D88iZBbbafBxGUnXnJzh/wBqr1yasAq4 ev3NvKWyTRc+hjQDy4q6FoImn02jChU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RTQfrxu3; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s/WKKKfYkgZLf01K/tBKWgYukXTjbxBIh6Ytrwu4rIE=; b=aRUAeVNQraY17iqzq+Gk2vSXPhdEQihwQSN7w1wADQBlZAoAbcZOA4WGaO5VdQdT5pmMXB 068TjkPSVUroTvhVZJ1S5TywxSwaBTBvk/UuQtrHMSmfRajpp7Kqv02UNF5vVr57hH9NQm dWWCB00WaPCWD9Rb9NR9DiWviWTiilw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279008; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s/WKKKfYkgZLf01K/tBKWgYukXTjbxBIh6Ytrwu4rIE=; b=RTQfrxu35RSMrhPeBnpkjhDYQ1cvjQmnAWei58jxpxyqJfGLBib8r6WAw4t63MpOz4/Hvo +57yQQxGAFD/0IVvG+ySa1AJ21izpqmRFwSNg0UarJ8lCKjNqB5zQFvkbscGn/nJPowADS 2vmM4CYkkISOx7LUF8zd1JMhj7SINjk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-637-LFWHst1IPamgtZLqqI4Kfw-1; Fri, 31 Mar 2023 12:10:02 -0400 X-MC-Unique: LFWHst1IPamgtZLqqI4Kfw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7D7C2858297; Fri, 31 Mar 2023 16:10:01 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9A3E1C15BA0; Fri, 31 Mar 2023 16:09:59 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 14/55] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Fri, 31 Mar 2023 17:08:33 +0100 Message-Id: <20230331160914.1608208-15-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Queue-Id: 4851D40010 X-Rspamd-Server: rspam01 X-Stat-Signature: orcbkeyuhmtzqq3hsrdchmizduybh4an X-HE-Tag: 1680279009-392693 X-HE-Meta: U2FsdGVkX1+am0e5rStfth69W7/cGbz6wjtoSEdMyxPQsnt4EhUHVcobLPo25Q309DpdbOSrF60UMlT6gRoRNacmmy5vvuJb8yNA/jX+Nte8OhOKT7kXCov8CgF1tafsHszoqi3T+tG2okCCvSDrOnK0RUXo890Cijjs+9PQBt7GRnZj1RXgA4Y1xofK+uxjoWMPw2vmzJfYeVq2BhtpT4Lev5iEK98TxRoqzKH4lC4pShCwbF3kWjg6XAVj6NXkA08bRmADKi2/c1MxyawdnbryBZWN4RdXUbsNynTf5sYyAuWhexOQEnywr3epkzne4au9KgOJb3WGX/rljChiGlsYL60QoHzS1lNEkEseTyT9DbkB3tyTXc9M6BFiPpCg5bO1Wyb6oto8XHKku3UeNuvx2z8uMDCwqsQ/LOB6vCtaL/HrkqKQ2PgC7mmzIKWDecKFG5K3ReQeaOw6NIls4PSmYff5Xl4otJLFkBYcpNIxp1AdTDpRrZpY9CGdTjNGOx2/RXsOEOWaw7AT9RKu+DYLA5O1zRBgwPFKivmzksErLYo9CkuKcZvf92UhyPYxJU5x3fU4c7MOh72YmfxtBUA0roRKiHD5ha2AiUAQ5MTl34qEuj1kfb6siiPMhnJeO0Njp5KdCtBsEdczERKOvisU5UeOuLsg0p/yK3Jl+KCPp2tmCNhhjsd0zzDYA4VMkKpfM1ldtI7r5dnAExfTshVOVNsqVmXaNG/m4G8I6na0i06p3JMKoiAU1/0JGi8RvL2GvUgoxpaP6ZdP5cChh2zx6rKSevFDiEE3mmd7mgBhWzssT8ZNiS4629lw2iJx03OLKe3pgHUCPgtS5zzYk4x96vyde2Gl/afV3yWd1iywjfslDJ/BpP7b2K0L+qvKF4Hn+wN23eNgAvhSkaqlKuUzf2igNZQRpklH5xmqQWDK+1C3is7dgkgjH1BO6SaAu3OtURiXC8wMRj97vXF GcE8So+w kVmG3sUufax+IJq7rdYtJVBVrXAHJHpQWbVHXimJlFyp3y9TI8AYv4bzjjnpaeQaaq1OP3obgrtyqokaL2xPg9LAGfYKRwryvhom9lZWLiGAu/P9lnuguf1S5vue0LTIhFbPfMT9hu0VEEN7BKt/QykbQ6WuJZGKhv/Bgv+wbkmcnzFaN+NtbFaOv1cFsnjUg5qEkVGqNFJSXz5KAsjRGfqi0J2uDcznYiC5dnwjCdTOC+LHvec+EC1a0r5MOS8wk37WjTv00ilsS7aewUH1HZfVbI/TQemPziashU53fx/evfXyaLXuIhyFVrEEAsK50JXo3K5MfsVDlAsW4WMkDWNPR8exeDyRI3GDpspHwbSpVE/nXSZ1iS2xr/IfiV8jgwf836CQ29iRDbyNHITQxIQt7ZII9cz4KLqL2xorTx4qIz7RFO5P3LVWQsx6LUeeJu6VX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index db9f828e9d1e..844bc8e6a714 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index edcf3a60c1b0..a8f8ccaed10e 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,12 +971,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -985,18 +990,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Fri Mar 31 16:08:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E1AEC761A6 for ; Fri, 31 Mar 2023 16:10:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AA496B0089; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15C926B008A; Fri, 31 Mar 2023 12:10:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D2C6B008C; Fri, 31 Mar 2023 12:10:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E49366B0089 for ; Fri, 31 Mar 2023 12:10:10 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AD1C11A0DB1 for ; Fri, 31 Mar 2023 16:10:10 +0000 (UTC) X-FDA: 80629680180.27.8D411C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 8980340014 for ; Fri, 31 Mar 2023 16:10:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TPSky1Nd; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279008; a=rsa-sha256; cv=none; b=v1PSshajEMjRyuv3t9D/ZGw2jrPXBhg/wfvck3ngK5QB3etotvASgNFJmouhnpEPQIq6ow I5BrOZRnBstjwSsKHh0hr95eUc103DpeP9ezudsUsWYb3aNRXYVrtrUnMiNVinovsJM6IM XVjImvSIXDqSgx2MX/Z9yn7tZ/tYSJc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TPSky1Nd; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279008; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zS5R68pySMwmjEGC4becLd7BE+l3zHJhW5dAqXR21oE=; b=aVxxGebLMrduu76IS+rqBpjlJ9Vl0BLN8XIZl01l4/EyZjIkgeR4Gdxv+wG6CE3811+8Yu axPZB3PQBiGWQXnEx50I1ELJzpJl/lKC/qXX4kwk/rAUZ94NGiwfdLzfTkQHCBEJahDp0P HvQTxkYwQcMtVmbc0PSqAEZR3Ge3aZ8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zS5R68pySMwmjEGC4becLd7BE+l3zHJhW5dAqXR21oE=; b=TPSky1NdyAiq7lS+vmlZvEFBH285b4EWlMouoU9hA+x32lJH62QP7uptDHrK2zHTS3ObKc Z7DvnaQkZyNSIe0jVjLducHq5fofun7eaTbVIvv/ejHgc1XaBqLWQRgpEHTTKn4SllZWNU 6ZoC3dvgdS4mam1kjlQHEAs9GRzGF1E= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-45-_6_3ZMiSNHmxuLq7kZsEhw-1; Fri, 31 Mar 2023 12:10:05 -0400 X-MC-Unique: _6_3ZMiSNHmxuLq7kZsEhw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1890F3C0D1AE; Fri, 31 Mar 2023 16:10:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F5EC2166B33; Fri, 31 Mar 2023 16:10:02 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH v3 15/55] ip, udp: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:34 +0100 Message-Id: <20230331160914.1608208-16-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Queue-Id: 8980340014 X-Rspamd-Server: rspam01 X-Stat-Signature: cx8jcyt1trxwto86847rwud4x1q5oros X-HE-Tag: 1680279008-33567 X-HE-Meta: U2FsdGVkX1+o1PDFUzl5QfSRKETLIwSUp2YvujpI1UBjlO3oJYGV2fpn/NSZV9Bl/pdST02A3eF41lvqYvhBEcwh2I6zFDTjZt4s7dal7D7KUgFEzz/lkvqSG4fxA5Qt+ZpClMzGWaIOdvgqHbCraiPE7ZLQKaWWbtZNOagB1WDCt6uYFjVAXEquYmQC10rKRohKrniZ8KFY0OxZ83m5dcVBopH/V+08dyUYVPUUDlFGm8zPAXjvVw0qouBHaVSOeCStcrF5uTrLqxX+3rXkM+dgIFnJk9tenfNBHVqjTIse632Jj75C6ULysjCdb5mvCduU57qdsd0N7eT9cSOoxMZd2ZSYjzA0Rf4MmW/g6eIvqj8S2fbwLDSIqqRxb96uPYMk0PaLDWGNIfXCkk8WWfsdS3RGvxp08yKupGD6fWcIX8kx4pJrZ5I0CKy4VwrfnLXfst30/sR+1+0dKglbL3BwKgBEKK7YzUuCCs79qU2uuM74BBOvZW75BWIBaDmhgUp1wjFkX9NaO7WQndllHAtcBP9E8VtCr6EP2f2TcrQX06MInNBFYNTz6DOGcWTvxS02wvTgXEf1lrvK/ZTN5LH9HVhsR5+6lIbJB4IPhnRXhbNxA6Sdl17G3wxk+pb/GPRLT6ZCF45Rge7JTA5ULvG59JQzdJ9Ty+IkH8Cb2ev5OaNiG6W1OKCL0TAxVBBsdxid1LSm6UavKbrJNDcT8acmTfQyw5agRE2/lbS53XcDRU+VeROLePxYF20zkqbi8QaZdyeYjzgBulygBa6Ka77hwr9nh8TYMUUmw1rjJ9Sa6/YS1cQgBf0n0xzitrxZg5WhCea0eBGyMwAfE8J1z3lLBpwJKGY+yg9odxrnJwN/vRbzh4KctYt0mgmIwVvimBQWLqpGARnbGsdMQA0WSt91TR5n6fjtSbgJO6GIeLAGSJ6aAXsYStfcFb7ojzGlNhL7z+dITHb4IWZYJDe nHy29fi1 3d/OxhsZBh11gFmxfXNVikzcshkL7AL6YrTIaZ4gV3yyazh0dIjO/OzD58fCZf2h+JHrjxP84sOBY6aboM937uob9+8IGIL9Sr0J3PHoeNkpmWoWex1TYRbRcwOJGKb0DsF9pkf7tYT7tdes7kNy6UZc0hfZYwEsRo9L0ICeuBcVlQFkCD7RVudOVC+M1gpTFqb+XDwjvmMeoZ/K40fB3C67U2GhEKLOFslNayhj1MRko7I8kg9xgwKTsbBRquHavet4rzfUf7XEV1x0FfyoELye/RDX7tJXo3+LSMaXZ2daDULE5aINbWAyOD3ncW+eLD+ExZaYJWk5Bg1Zrqn5cYEwPw3jFVtubEWXVobyYeCuG8ZFDwiV13LgWDx+C9J+NXYL+fKmpqxJf/tfN8r+c3yAPDr5KnYF6Y3e5lNH0TlGXIVe2c23yzW7x+ht+lrLnclS//qloT6c6FXDaCfAB6Q8zqTT74LTckZEm2WplmPdjGW4tT2OHcm9KC5HY55pfxPD4ui//VddBqCeHoMBP/3viMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 102 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 99 insertions(+), 3 deletions(-) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 4e4e308c3230..e2eaba817c1f 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -956,6 +956,79 @@ csum_page(struct page *page, int offset, int copy) return csum; } +/* + * Allocate a packet for MSG_SPLICE_PAGES. + */ +static int __ip_splice_alloc(struct sock *sk, struct sk_buff **pskb, + unsigned int fragheaderlen, unsigned int maxfraglen, + unsigned int hh_len) +{ + struct sk_buff *skb_prev = *pskb, *skb; + unsigned int fraggap = skb_prev->len - maxfraglen; + unsigned int alloclen = fragheaderlen + hh_len + fraggap + 15; + + skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); + if (unlikely(!skb)) + return -ENOBUFS; + + /* Fill in the control structures */ + skb->ip_summed = CHECKSUM_NONE; + skb->csum = 0; + skb_reserve(skb, hh_len); + + /* Find where to start putting bytes. */ + skb_put(skb, fragheaderlen + fraggap); + skb_reset_network_header(skb); + skb->transport_header = skb->network_header + fragheaderlen; + if (fraggap) { + skb->csum = skb_copy_and_csum_bits(skb_prev, maxfraglen, + skb_transport_header(skb), + fraggap); + skb_prev->csum = csum_sub(skb_prev->csum, skb->csum); + pskb_trim_unique(skb_prev, maxfraglen); + } + + /* Put the packet on the pending queue. */ + __skb_queue_tail(&sk->sk_write_queue, skb); + *pskb = skb; + return 0; +} + +/* + * Add (or copy) data pages for MSG_SPLICE_PAGES. + */ +static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, + void *from, int *pcopy) +{ + struct msghdr *msg = from; + struct page *page = NULL, **pages = &page; + ssize_t copy = *pcopy; + size_t off; + int err; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, copy, 1, 0, &off); + if (copy <= 0) + return copy ?: -EIO; + + err = skb_append_pagefrags(skb, page, off, copy); + if (err < 0) { + iov_iter_revert(&msg->msg_iter, copy); + return err; + } + + if (skb->ip_summed == CHECKSUM_NONE) { + __wsum csum; + + csum = csum_page(page, off, copy); + skb->csum = csum_block_add(skb->csum, csum, skb->len); + } + + skb_len_add(skb, copy); + refcount_add(copy, &sk->sk_wmem_alloc); + *pcopy = copy; + return 0; +} + static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -977,7 +1050,7 @@ static int __ip_append_data(struct sock *sk, int err; int offset = 0; bool zc = false; - unsigned int maxfraglen, fragheaderlen, maxnonfragsize; + unsigned int maxfraglen, fragheaderlen, maxnonfragsize, initial_length; int csummode = CHECKSUM_NONE; struct rtable *rt = (struct rtable *)cork->dst; unsigned int wmem_alloc_delta = 0; @@ -1017,6 +1090,7 @@ static int __ip_append_data(struct sock *sk, (!exthdrlen || (rt->dst.dev->features & NETIF_F_HW_ESP_TX_CSUM))) csummode = CHECKSUM_PARTIAL; + initial_length = length; if ((flags & MSG_ZEROCOPY) && length) { struct msghdr *msg = from; @@ -1047,6 +1121,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + initial_length = transhdrlen; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1074,6 +1156,16 @@ static int __ip_append_data(struct sock *sk, unsigned int alloclen, alloc_extra; unsigned int pagedlen; struct sk_buff *skb_prev; + + if (unlikely(flags & MSG_SPLICE_PAGES)) { + err = __ip_splice_alloc(sk, &skb, fragheaderlen, + maxfraglen, hh_len); + if (err < 0) + goto error; + continue; + } + initial_length = length; + alloc_new_skb: skb_prev = skb; if (skb_prev) @@ -1085,7 +1177,7 @@ static int __ip_append_data(struct sock *sk, * If remaining data exceeds the mtu, * we know we need more fragment(s). */ - datalen = length + fraggap; + datalen = initial_length + fraggap; if (datalen > mtu - fragheaderlen) datalen = maxfraglen - fragheaderlen; fraglen = datalen + fragheaderlen; @@ -1099,7 +1191,7 @@ static int __ip_append_data(struct sock *sk, * because we have no idea what fragment will be * the last. */ - if (datalen == length + fraggap) + if (datalen == initial_length + fraggap) alloc_extra += rt->dst.trailer_len; if ((flags & MSG_MORE) && @@ -1206,6 +1298,10 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + err = __ip_splice_pages(sk, skb, from, ©); + if (err < 0) + goto error; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Fri Mar 31 16:08:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B849DC761AF for ; Fri, 31 Mar 2023 16:10:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 553DE6B008C; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 503726B0092; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CB916B0093; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2D27D6B008C for ; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0B86F411A6 for ; Fri, 31 Mar 2023 16:10:16 +0000 (UTC) X-FDA: 80629680432.13.8D14B2B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 5643E20025 for ; Fri, 31 Mar 2023 16:10:14 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="QB/XkKcw"; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1cUbUkat/Gf1dtGRXqhXt4i30LjR64ymk07qbS2yB8=; b=3w3wKTkkqCqXfBk3Qfymm9mLv4ZUosqnLNAnflwCkW91b24V++jLqLE9xDSdk5S1B6fyLH B3Qvu23K5Fobl3HbfJsVjWaqinsO9txTBs0i1oRd3BuekyjzH30z3Asjn4HbYcKOXAKirj rRs3CCQsIEsOWkypaN4KTCL+KENtrEI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="QB/XkKcw"; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279014; a=rsa-sha256; cv=none; b=gvApFvEdAdc5wcZ/zgEsXctnFtRN5MTzlAGED0b7RtfpEEwI/dkPN2N1CxlJfvi/Eqeooe 87Ylg54PbvBzPcmb6K5GEPgrdSLK8kQwzuJtm2FAVqdgZYpxbGW3qGRraBdVDo825Au4JJ 4/WgXbxdAr9jd2Bxxfhfpt5IWLxKnRc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279013; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t1cUbUkat/Gf1dtGRXqhXt4i30LjR64ymk07qbS2yB8=; b=QB/XkKcwSCmDMDZ89e3D7GLaoPdL6NsXyaot9zQpUUXnILp9IrLDjoxx2vMH++LSH1kn/Y RrHE9JEPBm8z9QZNWsWbFs5Lka9B1iiaPM7mxYDP9dZ+0k0dQyMyjxfgD3lx20dTKTswKc nLjQgimmigJPQtZLugaWiMjNQIPSCKk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-488-2g961z0wNy-7npZvx0P1sg-1; Fri, 31 Mar 2023 12:10:10 -0400 X-MC-Unique: 2g961z0wNy-7npZvx0P1sg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C8816185A78B; Fri, 31 Mar 2023 16:10:06 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CD6FD1121314; Fri, 31 Mar 2023 16:10:04 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH v3 16/55] ip, udp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Fri, 31 Mar 2023 17:08:35 +0100 Message-Id: <20230331160914.1608208-17-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 7trmiqt7y18b4865rsuyy6wpzd5s13qo X-Rspamd-Queue-Id: 5643E20025 X-HE-Tag: 1680279014-139117 X-HE-Meta: U2FsdGVkX194RleDz1fRa+w/tiBOTxlpvEFMGaW2Ov/OFKpq+rnKZyAst0Jf1L4fEcX6VtOmQ0eiUVTLY87hzHZiQElo/Rlb2DnfRmh9i87HQP4ZG3ZxicEKjHB/UyeRnRwLmB6wPsBC7W5yICia6AeOh0ote4W4eXFmYTsv97zlZGzzQgMicTpq1tMn9hkyoBxmCbnoFKJmpCvyV5LFjYz1hQppjsUOmcMhnawXzb56fO7FBxr9RoG3rwLHlnViHLUnp8Igqj2IEl4YwJ4JG/DjaosEaeB/69IQr4wrvd5udvVPt5AVKZhKBlEXsaqMZpov4Cu0K+ZJdRSa5MBE7VJ648xK/4/5QovN/2AHinbxHycZb/fMGsKGm4LxtPUFwlHA8Asr0WtGyTP9Izrn4KbWhZeiUhh1NbWmaENp16s9zg1JHntFfvn0yPnqEEUH28PQ/mRvuNhZwPE/MQBTDzBSbtcZ5NJ+a9/Pu8t9mWLYnGLCgAdWhGrzUVHCLPHSMUCua1UbDWGG2wQ7Z8XaZ3V26BMNdeMAlONOZkv4UwcNLfW2u3jx7jZlTWDCV01+E25Ly69VjrxY8uvFqIwhwg5P+MLqg8NgmXP6E9eCpHpjulfQ8XICql4rZO7Vc6DGE54+0jeDXQkKqKTKw5vgCrZnMfkFbgDcs7gfXLq76Cqu8KSvWg3kJtTEIJTqUf5/wio31aXRKRUP2TNte8VsWYurf/Lnj633kXjvNuxiw2jNjFyHoPouUrAXFMWufVgxh84YuahHUoJyXy+W/XEDmBEMnlabqXZXZPoP7nPSs9P30bGbTg8NF7xPPNXe3sERqLRW98TfxYw6F+N3d6g+P2eMCG9TDyKHAznjnpAMiaQ1fqPTE0YqUNUAhkUToHZ6uvxtnQDngirjxA8WsqWAxOmdu8M5xDq5lka8zSnOAedqLiZghtKJ2McCb1BcApmb1T0bbzsO/9KdCEECwBP uWMj//pQ isA1jnQrD6Jt0ym1kH0VdGsrlBhMJr48G5RNuMKtAyUpyfXzNbXVvAc2/IWXd+BUOZl3dXs7aiLmoWmWEf1sJqwt5GEiZdyedrsQ9wPeD+9PeOtMKRV9ZzzeDfGOq/OPXC5P6aLtA5WeDZtnix8FCIfUhgFlafKRt3uoneLn7AHlf0jydvUvqXsKPW2GuTEpIP0fl+mLIHQ8kiBGMbqLlil8oTQ4UVbZIaOyqbplJ5n6v1SxCDFIl6zywcxUoU89hq6BEh1+Eu+aE+3d/WcD3xkktIyfAvCWy2AOAT9bc+y1Sv9gOAv24SKLwKr1sBqrZ53Ytlj470PlR7q1NEH1QvwYZTX9B6FZ23vBrfVISABC4Qzwc1IaHRFmv8eudDuMo7WaPaK6Wh59XrH+ZsyLFDuwJe9btAFDS3iNP0+9vXhffXoeEKBMU5m/IrqBczlieDwEY9INHMtW9fRgV/2vxHRhvVjYbVUhsWlBVSOZ4QNyIZDdsPbOZvAGZVB/iMRpKHvxHKhtwWcqCbHHuJa6OF3CBng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make __ip_append_data() copy it. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index e2eaba817c1f..41a954ac9e1a 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1004,13 +1004,32 @@ static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, struct page *page = NULL, **pages = &page; ssize_t copy = *pcopy; size_t off; + bool put = false; int err; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, copy, 1, 0, &off); if (copy <= 0) return copy ?: -EIO; + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + return -ENOMEM; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + err = skb_append_pagefrags(skb, page, off, copy); + if (put) + put_page(page); if (err < 0) { iov_iter_revert(&msg->msg_iter, copy); return err; From patchwork Fri Mar 31 16:08:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FA04C77B62 for ; Fri, 31 Mar 2023 16:10:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5A616B0092; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBA5B6B0093; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A81C96B0095; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 95D456B0092 for ; Fri, 31 Mar 2023 12:10:16 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5E9FF80473 for ; Fri, 31 Mar 2023 16:10:16 +0000 (UTC) X-FDA: 80629680432.11.75BF17D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 9BA621C0010 for ; Fri, 31 Mar 2023 16:10:14 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RtCSvynC; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3uNZWqu+MQKIO3OTB8U4ZJVMqWS1JR9mPIUS8nXZLn4=; b=wOPOpljo5vRqfsY3A3MLgRXpulgJxY2Yc2v3g12OMRrYUPeTYhzmvKmdoGXVRnMQcvTFRy vrIAd/ZdiWKYHnF+zF7zyhBIgIGnuMbzbT/cxVV/n6maXGPHfEzn08k4MKNXcoSlTgCT/T MlXdFOMCgndAaoNhSA//LzCeLuKWefo= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RtCSvynC; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279014; a=rsa-sha256; cv=none; b=rduY7GAb5l+YCTF5jda4EUsTJEe6zhtVYX1l5gAZCeSkZ6OfeXMKsUny4SLcfIJFsHgL4R 1ilQZUu1/afpWiAyDoAnCYgyYIJsE60c+gIPdF9Bso9b1lIXUefNjWGx3G+/ATAV/JqKis iMIr8gmCn1hMtGMOePa63u1kf5V23U4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3uNZWqu+MQKIO3OTB8U4ZJVMqWS1JR9mPIUS8nXZLn4=; b=RtCSvynC3uKTYhNIDtyC0SGMwicXgrlO3b/z+o5L6b0hZ/r9yOX7anYNZDhcRC2O1VA9rN hIQQzFm+iQZDTXA/FA6lPPl0o6BRleyHAdmMGrnCdfcLwWFcQUSBHaL99LEAt02WjNBNpX TEVPF8pg8nBKNKTg5wC0FuX6Sg1dgw8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-635-jQiJhrbpN_2494oFduAemA-1; Fri, 31 Mar 2023 12:10:10 -0400 X-MC-Unique: jQiJhrbpN_2494oFduAemA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 66A7829ABA23; Fri, 31 Mar 2023 16:10:09 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6B0F74020C82; Fri, 31 Mar 2023 16:10:07 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH v3 17/55] ip6, udp6: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:36 +0100 Message-Id: <20230331160914.1608208-18-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 9BA621C0010 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: kckusejisijqafng51uxk4uy6xtz7ijf X-HE-Tag: 1680279014-823703 X-HE-Meta: U2FsdGVkX1/8WmJutqS+iU1S7OwFKwj4n2lTqlM0FDE7QrUByJayOBE1RN4xX9xkb+Q9C7tCFMIRSs2PWAp6kpx2dIrnQvggu6XKnxCThJQYQ3ynCDVl8bFE48rkauy7UDwrfEHB5NUAp84Fl/dW+mSwoL6+EB7jfIH+ouop5zF4s2+pyRdp1RGFDAUAofYwsgKUoQLZsg/Dgj11XzQo9s/JHoli1wGQarjOkG9BNj6Wkrs+vY/XJTL8orlJdAY0+FZmwZBPJZ1j23AIYbQ2cwAXGVKK9B04190ERKNiBFP4habK8yTUAL13/AlgHV7efBGPDin6E4c7BfflpeRhnUsqv7l3khNAooql/7d8cpidySLbbLM7L4Ggbg0DSU9xN21bzpJAnFJTJOn/Ks2qJ3D2wPM0FlO7HTDutly6oFPhXwVhL/uIfOLH0Kniqp1nOYf6d4uFdZ0PDQvHH7bfeR4bycj+8SiKvm/Lpo/jAJ6j4U7OsEAmss3zFL4l1RqBSo1z7U7l2SeopwfKXz1RFw4Yq3Tg6nSjl6G897zAvnQiB6fdz/Brj/pON10SM69AX8e2wUJeosksmi9qrcq9mjlGnDf/TITSSCyku3W/gkam9DCAZwvXrlMiY4NYvXqP95azwovxV0WIfliTz9yigeBZ8D2hSz8/SLxeB0TmoLKUZAjrhCM7RhERvAP3NVnyyVGJ0c8sDMFVCw8O0I1i91JdR8/UCMP7pGCXD6BlDTkTxouaSLugY65nuAdqr1k25YVlUlCu8NAVKUrU9NJ17XvHw9rUDNlfa8Tnc26x10yRuVElR/ZBsbKgUnt3lhpTPjKm4Z6D5RYpilrpZdKt+/59luoCddetfB1116ooViwpA91ruk6qcA7Mn6QIFexClVEkRXJ1ulTE3X+ccSnKZff0sllg6z7VzrHWDvW0qVtl9DK9IwLCG1UU6pBFX2Mi6MaSjErDfJsHqkCAHGX ehgeNKCA 41FIuYzGQfwNX1Ij5wAQMlwX6VNKzcqXBFmTD8XY8W2+Fc4IIqaPNny7OoFrhdNOuqvAJIbTJq2IyAUNc/rGMAHib6AQuceO97bVvNlp8mxY9CzhW/f45L/uEuC6a5mGj2i+gHChLXdHuj/ik4BHkuSJhHOhg8mzS5fAO4zEzZxIvMfLL5e0C6i8PK0Nvx+EC1t2KBowreC3v0qbIuzLtPXFJZbFpFWmBIA0nVzy/xHnbeDAL+kIkeFbaie010K4+Me6StmdrviUTFn5WdSedXM8s3sFIiCnw+FND/dBR/C8H5BwPngZsO+dkCYe2z9+sRcz7jPm94Ksk6Add6Y7TXnGFobb8ciTUvfgn/41c9IxFSa6uTdrq/jAuRxc208xsL79FwQBwHC54rq+auNXUQSKYOcjKa3w8CAoflY5hS1XpnqpXeDyLSx4NtW1uSeop3VkmPmwDaQvxIqojsrhv8kp0hVJ5L76dT0QO18nCattZq0WkdceZJECm7TMI7h4B6l90DQ8q/BTvO1msaz0t7E+6QA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/ip.h | 4 ++++ net/ipv4/ip_output.c | 11 ++++++----- net/ipv6/ip6_output.c | 28 +++++++++++++++++++++++++--- 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..e27d2ceffcfa 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -211,6 +211,10 @@ int ip_local_out(struct net *net, struct sock *sk, struct sk_buff *skb); int __ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl, __u8 tos); void ip_init(void); +int __ip_splice_alloc(struct sock *sk, struct sk_buff **pskb, + unsigned int fragheaderlen, unsigned int maxfraglen, + unsigned int hh_len); +int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, void *from, int *pcopy); int ip_append_data(struct sock *sk, struct flowi4 *fl4, int getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb), diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 41a954ac9e1a..fa2546d944bc 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -959,9 +959,9 @@ csum_page(struct page *page, int offset, int copy) /* * Allocate a packet for MSG_SPLICE_PAGES. */ -static int __ip_splice_alloc(struct sock *sk, struct sk_buff **pskb, - unsigned int fragheaderlen, unsigned int maxfraglen, - unsigned int hh_len) +int __ip_splice_alloc(struct sock *sk, struct sk_buff **pskb, + unsigned int fragheaderlen, unsigned int maxfraglen, + unsigned int hh_len) { struct sk_buff *skb_prev = *pskb, *skb; unsigned int fraggap = skb_prev->len - maxfraglen; @@ -993,12 +993,12 @@ static int __ip_splice_alloc(struct sock *sk, struct sk_buff **pskb, *pskb = skb; return 0; } +EXPORT_SYMBOL_GPL(__ip_splice_alloc); /* * Add (or copy) data pages for MSG_SPLICE_PAGES. */ -static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, - void *from, int *pcopy) +int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, void *from, int *pcopy) { struct msghdr *msg = from; struct page *page = NULL, **pages = &page; @@ -1047,6 +1047,7 @@ static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, *pcopy = copy; return 0; } +EXPORT_SYMBOL_GPL(__ip_splice_pages); static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index c314fdde0097..c95d034cb45a 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1486,7 +1486,7 @@ static int __ip6_append_data(struct sock *sk, struct rt6_info *rt = (struct rt6_info *)cork->dst; struct ipv6_txoptions *opt = v6_cork->opt; int csummode = CHECKSUM_NONE; - unsigned int maxnonfragsize, headersize; + unsigned int maxnonfragsize, headersize, initial_length; unsigned int wmem_alloc_delta = 0; bool paged, extra_uref = false; @@ -1559,6 +1559,7 @@ static int __ip6_append_data(struct sock *sk, rt->dst.dev->features & (NETIF_F_IPV6_CSUM | NETIF_F_HW_CSUM)) csummode = CHECKSUM_PARTIAL; + initial_length = length; if ((flags & MSG_ZEROCOPY) && length) { struct msghdr *msg = from; @@ -1589,6 +1590,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + initial_length = transhdrlen; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1624,6 +1633,15 @@ static int __ip6_append_data(struct sock *sk, unsigned int fraggap; unsigned int alloclen, alloc_extra; unsigned int pagedlen; + + if (unlikely(flags & MSG_SPLICE_PAGES)) { + err = __ip_splice_alloc(sk, &skb, fragheaderlen, + maxfraglen, hh_len); + if (err < 0) + goto error; + continue; + } + initial_length = length; alloc_new_skb: /* There's no room in the current skb */ if (skb) @@ -1642,7 +1660,7 @@ static int __ip6_append_data(struct sock *sk, * If remaining data exceeds the mtu, * we know we need more fragment(s). */ - datalen = length + fraggap; + datalen = initial_length + fraggap; if (datalen > (cork->length <= mtu && !(cork->flags & IPCORK_ALLFRAG) ? mtu : maxfraglen) - fragheaderlen) datalen = maxfraglen - fragheaderlen - rt->dst.trailer_len; @@ -1672,7 +1690,7 @@ static int __ip6_append_data(struct sock *sk, } alloclen += alloc_extra; - if (datalen != length + fraggap) { + if (datalen != initial_length + fraggap) { /* * this is not the last fragment, the trailer * space is regarded as data space. @@ -1778,6 +1796,10 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + err = __ip_splice_pages(sk, skb, from, ©); + if (err < 0) + goto error; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Fri Mar 31 16:08:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F8FDC761AF for ; Fri, 31 Mar 2023 16:10:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A371B6B009D; Fri, 31 Mar 2023 12:10:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 970F96B009E; Fri, 31 Mar 2023 12:10:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 838A86B009F; Fri, 31 Mar 2023 12:10:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 73FBF6B009D for ; Fri, 31 Mar 2023 12:10:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 330C21404A0 for ; Fri, 31 Mar 2023 16:10:28 +0000 (UTC) X-FDA: 80629680936.15.72BD0DB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 614B820003 for ; Fri, 31 Mar 2023 16:10:26 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K1cA3h3w; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279026; a=rsa-sha256; cv=none; b=A0yYdlySTe08pT0UzUKUvWzg1IrwVjfp3kVkS2y01VA5lGn8ybRreaB4ZoV+7k+HznZO7R U8QpMLOCLyNPHjJo4o/gduUI5/UoLORs5voknHTJIQPn2bExMEY4WKBUrXefSosXkLi8xl ZDvdg8Sc5Vxw2dOIotxigekG/8CDP3Q= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K1cA3h3w; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=lyush0ViuMN4qjWnnjaTjNQGyalIGfxBnP2JKfHTZO5KZG+ySzIT8CnSbYtWK8XCYLEWWi zjFfGkxcsYp/BAjKlowc1HBdXXitT4aLyOuTPjajFNuOUsjKo4i6iIRjBsSfv/ZyNRh2NG jXPXyal0tto6oAfkT7UJ4aSxgfhxkYI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=K1cA3h3wwHV95gaH9veWPOQc9gl3PayLJnV4p7n6zGKQrbu7Lxo4Ywu45Ju9WXJELMntBY xsZVFqXQGnFoZ9W9xFHWUY84b01Be5BLXK4yg+AbZh/wleb67tEQlu0LW4RZ5ZbKgwzuRo IhlGT/UhUxcjA6vFZX7aEAW95SNO6rc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-404-Y_P62SwVN4u6m6gIEWawdQ-1; Fri, 31 Mar 2023 12:10:15 -0400 X-MC-Unique: Y_P62SwVN4u6m6gIEWawdQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0ADB829ABA1E; Fri, 31 Mar 2023 16:10:12 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FB4014171B6; Fri, 31 Mar 2023 16:10:09 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH v3 18/55] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:37 +0100 Message-Id: <20230331160914.1608208-19-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Queue-Id: 614B820003 X-Rspamd-Server: rspam01 X-Stat-Signature: n6ye7mpmugm9bzga3xax67bdn3m5q137 X-HE-Tag: 1680279026-621916 X-HE-Meta: U2FsdGVkX19VVGHK9in6c4m+ZyuHg7D0YQDQa6rI9AJ/s+DqOp57pxW23o3hPXKZmRvqpIr5O4RCkxq2JBTtIS/7MEEx8dmjA7tXCH37uajj61ERKY0+QlDteiclbTWwMkeZ74hLkUJmq76fZDLa0348r0fms8DC8dXryf1xW3KsK0KRJUcMIYAyk5AoD8PS2UOEREX/kHMRoBNiMNVLHyjIKbzbbFq9H2krRgN9cnN5pVPvX+TDw7VLVzeo+0FCnm6CDLNlzwbEOGeBoIrvAvX/AcwR8n5xX6Rq9dyXiA9zDn/+z42U64hZo1OZuimLxpIDY7AMuKw2j/GW0zQM8dv/UjC0P3ubf0HWRjFu3Q1FY0c7PqyposvP3niAugT4td8qgNTBMMiyxdwTHTIZN7/UBEpCD/FJH5rXYbpFwBN5s3lFaSuJC8Q22nz5m2G9itqxZFFIWiA+JLuWe8RKHZx3lAtFB0NX7qikTLzNhYo62HX9/MhoRzU2m87cIogniPSfunasNoIO/7h9hbXl11Al5/2+qH4tSi90wvmLVT+C9bwENVhz0S+CDBPsZlJMkABlB7Y8PepYj2AhzIWBOL9PQ9KUh/iN3QoOGfzIty97SIEOs5Ya2UjWf0DYldxsxdEi2EK7ndYGXNTaYNB9QvRujJ3f0xthF7wF5w+/Owyq8veKkBEztMWbmB/AxLYIKeBDN2AFFkFXwylfQNQJpAOD+0SYxpevyMIWKjb0znS4hMkLXoNzvgsjd5FuhgeRFXgo6kXYF58wrtgGYFlvWz7hLj3hAaI1vhWJkuSl8QpXbEdWiSMsGvOIuByCI5jSJHE+0nK65wnIkKQGtuZceblvHs7IkvQqqqY7U7a54oMRGL8fq8Zx/szQBCfKLwmiuqKdsJgXNE9hisV7ZqCqOl/kPwm9S+QjwN4ktoGm3vKttl3/ZLq1VBBjGtjosK1wa2d3+oOfmyETUyi9lIn aEdijLRB so8I3z1XF5K73njqrrNAAXFnj8uHPTkDRLQWkeWCHEqUnVoetaLRsmrWdFGQWnupx4PHPkbkvNn3NhG6tyA1ArxiSwSX3ebiiGU6GCE6toru3zO/vOeFXfQTRyoLoXLKLVABTKtxieC4+kI9Dt7CXXESZkFykZ+/nGYjREhlBuIteWRAh7Jyg8iifB3DF/Njvo1MjXWy95peivOC7EPYTpqQHerJ23evNmKgAz9BBRqltjKok5Yibvma1jV2a44XI/n/4eg96RcLvrHUJIFUHuiN7IaVw6sLtDIW0jhdOdKJng3FE8PvCVH7eps0To0fR3jFprX9mb4XN2Kh4z5QF8wpGCMsPbsjYpMDho4pPEyON1juzT2v+kRczGT4cWE5z8qis5f/mKq7ogamIpAwNlx3FkQ2bvzi+y7f3a/GKLh87z4TAiSdIdpTTA7IkbdwtTEe6tiVOL8UM+tDlDltppvJO1UOyVSQCyxowwFeRi4rVHSqf0ODJqkmxh8cZ5KIJsFjGpg6CHi/VcTrRdLwjs5OR9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/udp.c | 50 +++++++++----------------------------------------- 1 file changed, 9 insertions(+), 41 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index c605d171eb2d..097feb92e215 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,52 +1332,20 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE + }; int ret; - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; lock_sock(sk); - - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: + ret = udp_sendmsg(sk, &msg, size); release_sock(sk); return ret; } From patchwork Fri Mar 31 16:08:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5787EC77B62 for ; Fri, 31 Mar 2023 16:10:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F17906B0099; Fri, 31 Mar 2023 12:10:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC74B6B009A; Fri, 31 Mar 2023 12:10:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8FE16B009B; Fri, 31 Mar 2023 12:10:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C964A6B0099 for ; Fri, 31 Mar 2023 12:10:24 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 92AE380450 for ; Fri, 31 Mar 2023 16:10:24 +0000 (UTC) X-FDA: 80629680768.28.F0FEA4B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id C977D4000B for ; Fri, 31 Mar 2023 16:10:22 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="TjcWr/Lc"; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4OGPYhZN0j7iZw/KVrz34Q1/SXztHasPFDnntHCFNPo=; b=rumoHh8wdKqI5juWdVZZiFwrpxRvujUMV2T8/ZNrkPtnazIU83OfCc/L1BENoqrAxeL32s sls3nVvq9FTajvA8WjZ2jFohNCvchQ1l+dOBQ2wMnAapzcfQ8xgd0GAcjLhpIJzIlYYD5a yQriB8qBQjx31wW5oIJFdcogALzARn0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="TjcWr/Lc"; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279022; a=rsa-sha256; cv=none; b=zBJlsgfN5f82Q7nbEp4KcpYyyuGJ/0/+aMjcRD9mCCIq0j6FJoO80YVGCzf39LY9z6h3wK s0vihOcx67JOtSeK2WRTXLxWfohJujjffFRpypZ7zZ7w15hHN2hRh9M0i0HS0PyHs8oXdT Tiwl6H0J4LWBRiqNTpGtlDeoJye5afU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4OGPYhZN0j7iZw/KVrz34Q1/SXztHasPFDnntHCFNPo=; b=TjcWr/Lca1hfdCCp7j4aci55px/2eu+3ETokUZ8D7T82aEhhb0+yE0c36Cfekxfbb8MWlN 12nPZFbZ7Fl93kPA90Ve9M5jxUqUJ/LxTxNRPsI2hFtSOxlx/QzVmcLXVctglsvUSvWPRr q49CsJpkN/LOWPdx8IUoTdu06u+TXzc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-79-meTwAxIdNDKMLFAudYCPiQ-1; Fri, 31 Mar 2023 12:10:18 -0400 X-MC-Unique: meTwAxIdNDKMLFAudYCPiQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 85825886067; Fri, 31 Mar 2023 16:10:14 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A0AA7202701F; Fri, 31 Mar 2023 16:10:12 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 19/55] af_unix: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:38 +0100 Message-Id: <20230331160914.1608208-20-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: yka1k5ee8c3fkeit7wrpekohc37tfpzz X-Rspam-User: X-Rspamd-Queue-Id: C977D4000B X-Rspamd-Server: rspam06 X-HE-Tag: 1680279022-630171 X-HE-Meta: U2FsdGVkX1/UFjmSBSG7ZpOh90bq2lQ+yepUF3ONWqNWqS4bX9eMTwwcOdv+nf7k73BxI3oF731mYyfCj2Op//pPe0oth5lTQfKEKQXO9uu6+l0GLSpgu7rmw/gsSvsKZpS6o5r5lEpENS54ZrMrHV/DspWcuO5XHU6Djz+axTO11EVWEPXe2QEYMyxgvwICxJuUIzNh3P8hZ6ZOeyvDEomlBH40p9sCgmIVt3QQGerGERg1hL7v7OqHb+SnTnk0Z1S4sNE/pZXkV0B77zGNvIVA6GIBiTZxXGcbSyqPhx43MfkOGt/fKdjLbwA8STiLqxd2UKcp6r/gP9JowufaLf4g9ezObmvJyfTwSNs/7ZY8Q/qxBzju6cWooitu5L6VlD7JDU7gZcLsGWeA21A3Gb/6x+6pSFQuzap2SAFGun6Mrq8ZEfsC59Q29O1a+R6Db15BMe8tHUpKkO9SUsfxwPBmVFrqvPgB/ewk8JHi+YiRUL1v+S4ldR+KhxpxmIGdnpx3YYWv+70Faw0dCDd2FIoh2BFryYzRrdsMu9PVT6tcShInKlzfPTu1XKbQ5m69L4xbBpgfc4ach/yQat5DdvpdZ+4B4GjbPm1MZNte2IAV4rn6iwOxal/nKJNuLipHhF1ZrCBvWevVIUsyNB5wv5337chLwySNzb47vsRkWu4WYbACU7Krr06wrhXUDrSQvfkyBiQxpvyxsAXnQ16tBK5XsXpC72r+us2ZM/ogD5Lq8cdzYJH2euI74Ho22D1438bzerDSrbbBhNbPzQvLi14gxUF0ZO6XBPeNK3TJ8YYEOgQZ/Uvepm94BvEEXSTzMKLNlCLf+xhpxARxj3jRlKnnF/mHu9V18Re1996paUH/sWWb42NJyDL6kVYnKiMP3VyWUMX8AWqAJna7MywtCyvvC0dvU9NNF8RT0f/OIhgzJ2PqF16ageJzMCXe7OaQ1u3lVFkSmOTMomBn3vx kuYnd9ZL CcEclotWi96/wYPcdnPOeuScVMVes8rQSCFiQ7qUnCAKUNkZm0DpAMRcuOJBS/d/TdAsS0gLjQElk0nnkjW6Sor+/28qCz+Way7jtnLHnYqJoRzZ9TH6HXG7MqHPecEEr9zVK3laCfJ111IQpU9h1ogqJV1pma6JVpFQEgQN+fpvmNDHeXt8xzd2EDgrc7esou28eTb0tHKL+bGua1UdykAaxSeTPMpCkx9dnBYKShfn/rrhtxnIl0EzVeowQ0HmOvPy2P9DgDiLgK4TBTgQGcu7Luxf2lxQcMKzung+GBM+fddtZQjr2ukVr7M02DTxY8WlaP53SbcQJGD6KDgIQtuyfrBtWvJj4hgeaZzYm3p1XBrM5/su6m0tA8Nb57RfZ/wBb7TKPEN3OD2ABk83swgbuGCwc/ktl7E/zlGlAH/ueJ3V74IBFgI0LzZ0YWWNzkqmx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 93 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 347122c3575e..a9ad97f3c57f 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2151,6 +2151,53 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other } #endif +/* + * Extract pages from an iterator and add them to the socket buffer. + */ +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, + struct iov_iter *iter, ssize_t maxsize) +{ + struct page *pages[8], **ppages = pages; + unsigned int i, nr; + ssize_t ret = 0; + + while (iter->count > 0) { + size_t off, len; + + nr = min_t(size_t, MAX_SKB_FRAGS - skb_shinfo(skb)->nr_frags, + ARRAY_SIZE(pages)); + if (nr == 0) + break; + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + if (!ret) + ret = len ?: -EIO; + break; + } + + i = 0; + do { + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { + if (!ret) + ret = -EMSGSIZE; + goto out; + } + off = 0; + ret += part; + maxsize -= part; + len -= part; + } while (len > 0); + if (maxsize <= 0) + break; + } + +out: + return ret; +} + static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { @@ -2194,19 +2241,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2218,13 +2271,21 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + skb->data_len += size; + skb->len += size; + skb->truesize += size; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Fri Mar 31 16:08:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A365FC761A6 for ; Fri, 31 Mar 2023 16:10:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 490006B009B; Fri, 31 Mar 2023 12:10:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 440666B009C; Fri, 31 Mar 2023 12:10:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E1416B009D; Fri, 31 Mar 2023 12:10:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 171C96B009B for ; Fri, 31 Mar 2023 12:10:27 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DAB14160487 for ; Fri, 31 Mar 2023 16:10:26 +0000 (UTC) X-FDA: 80629680852.02.05807A4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 1F8B14001C for ; Fri, 31 Mar 2023 16:10:24 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VxDxDqhh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b8cNPLuNlp8TF1aeLVa/nfCczmmxDIoFIua9rMiLDvM=; b=tucQCb68tAwIXusKy3ydG2WIcDLSmadAQ9mPA5s4K8ppqyK5ugQVswUyIfjo7Eq+n+YJPP bt98l1S1D9ZgeLog2u6SlfSKzmK9lH1W6jzptIUkT1OiHTb1MOfu2ARqFCUT+/rDIvo5T/ QIHHGYfBZYVMm4J/0mN9f8yFmwTqJMg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VxDxDqhh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279025; a=rsa-sha256; cv=none; b=o7IxCw4Rc4aB/EdrKcJ/51u5zoLGfPZ/61ns5EdmJdm+m2/6GvqtIV6nRQFiBzYA3Kwf+B 86tkp365PpvXG6KwkHkEAkvcDgGFWPEFPnKkTCzJC174S/BoqoxGyAzlnvsphpfxgixyUq 3zyQRGUDUgsYB08jaPhEoe7CNegObNA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b8cNPLuNlp8TF1aeLVa/nfCczmmxDIoFIua9rMiLDvM=; b=VxDxDqhhPlk5jcB0SSkoIGUsd0iMf0byZGR8NxzGxDhsP35ZiyNKtAQQjQlIZUwOpHacmz tvS5NYDtuDD4QpK4GEaWgCBJzQDoSzy2vWWANkBYTwxngd74onjq8yk7xqcpc4QdLRlHe1 vHEU8ip287ZUrMxSyWZLn1P0ul+AVTM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-203-cb5mzzjVOOCi4dFlfSrt2Q-1; Fri, 31 Mar 2023 12:10:18 -0400 X-MC-Unique: cb5mzzjVOOCi4dFlfSrt2Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 27F62185A78B; Fri, 31 Mar 2023 16:10:17 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 44F731121314; Fri, 31 Mar 2023 16:10:15 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 20/55] af_unix: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Fri, 31 Mar 2023 17:08:39 +0100 Message-Id: <20230331160914.1608208-21-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1F8B14001C X-Stat-Signature: 9e1efdje89jr1z4dq7a6yaxy97ip4k4z X-HE-Tag: 1680279024-94590 X-HE-Meta: U2FsdGVkX190vtuZjfEWxPC7xnNMa8co2PwkkTvJ4dKFSncwyPLkUEjDfJH2VlxzIQwnn/QIzAdNtQwGGzvAdy1crlo9/82GWKYnq5rV5mp1/oM5fxGhW75GimuBhivwoPWleatimHItJsSEkLKla5/pchG3yK65tLuIxt7Hdmw9bFfczYSKI3JzvHoTrVXbyqurCGrvWTgNbK8cS3B0eL4qQC6uGKab+uhowWSlTfdE9DeKghE7fkf7SeaK1MU6EVVagy9U1oe5R1/4iZ6YdqAOCvwuUD9FKXudw3cKXT1cPLZNCEAbV4e/dmNm7QF+POtOuRg+1+t/UV9c9sw3/ljnq/2IVrQaVOfjvmSnDLIRNfAQZM5ux8NF0KtmKZVwEDZ4O1hpzCB8HHXTMu6nKNWGSs90H5bw33ElXBWnOsbvm7O0Jb3ikbgBg4DqzTxoGntBSdUmD31OWRpc+VLIwcIy377PJa/z2ltEAWDfnwh+IOsWLiA1N23E7fJClniIrM1aLldKC8/iN9fEzVUekf3FeF9uzfMXRvbZhJtU9BTeKVeXbpFzAOM43jWP/WTWtyBe6G7H5hUCrHhSlgeDuEIdyN3ZU9qa9GZpr6ix36DOOpJYCo/6dVZqgBPQlbGCa98FLMZ69zwJKJ939mBklBnzTrRjjaOPJ5gVbLdRQQk45aEbMQ2DYtwUkLzGn9gQtCsfWi2Y81zaPYCZF6APrCqq8qd1UWV6W9FGyXxAMwnXTYKToEme8zIGUp0sgmNuCRz2xHW4guX+WIYnsDKW/TxrsnWXVgMP2lMRWcClUTkBYoncVROvk7QUvs8d+T050c4B51Rt5nYcAg7yama3n03c5IcTe8j7v+W/HUnPBEtq0Z+SLeBMhwPQPZQ7umPLiZwuPQRYHa1XRDDQ+QGtAnea84Caj06CkBQBdm0HfszkUdyivP7VT6y0z1fLvngIv4J0xp9QxzA9wUyUycv X9CT4+2E cB8ix3blol8SNbC4/m8Vof16LltAPoM/A99bGgWvDh5dBdiyL9Lfy7Iq1mzMj85tOp+oXsi+R/gP4ByUZ4s9iFK1LAVjS3LxfFFLYNOIIMkKYqsm4N8rHQQfSnWIL85Q118Kxt+nUCQjKqodRTIPgwALrDZ8hN769N+3kL8Nzm9OovFY8kV58F6Cx5EHCXglFosUhXTZR7CaJJ+BI6ce2RZbyPMNRwlxwLQhHFkbhnCmCGJyJ0CbzMbgwTIwXy7pHzKzZHMjDxmDoXKU1t7FZf3yewDXchBa7xLnpGMTNL88kw2vu5/Q0CrLDh6lU56iF2yvGO7VMyy5OyFzi2AlLoOSIsj6tAbumEh/0bgLkH+CFyopIziy3GR37t4x4GRZgnLzR1KCfOGTYTcsdsmkNuvtUGZC0/z1oSRYg2nGTw7XYlcJtevcMqj+r/IXL+XgigL5oSG6nVaui7zIwXB9JQodxzJhhuB+GJdB1fcjpXqj6/X4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make unix_extract_bvec_to_skb() copy it. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index a9ad97f3c57f..88b91005567e 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2154,12 +2154,12 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other /* * Extract pages from an iterator and add them to the socket buffer. */ -static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, - struct iov_iter *iter, ssize_t maxsize) +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) { struct page *pages[8], **ppages = pages; unsigned int i, nr; - ssize_t ret = 0; + ssize_t spliced = 0, ret = 0; while (iter->count > 0) { size_t off, len; @@ -2171,31 +2171,52 @@ static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); if (len <= 0) { - if (!ret) - ret = len ?: -EIO; + ret = len ?: -EIO; break; } i = 0; do { + struct page *page = pages[i++]; size_t part = min_t(size_t, PAGE_SIZE - off, len); + bool put = false; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, gfp, + ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } - if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { - if (!ret) - ret = -EMSGSIZE; + ret = skb_append_pagefrags(skb, page, off, part); + if (put) + put_page(page); + if (ret < 0) { + iov_iter_revert(iter, len); goto out; } off = 0; - ret += part; + spliced += part; maxsize -= part; len -= part; } while (len > 0); + if (maxsize <= 0) break; } out: - return ret; + return spliced ?: ret; } static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, @@ -2272,7 +2293,8 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, fds_sent = true; if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { - size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size, + sk->sk_allocation); skb->data_len += size; skb->len += size; skb->truesize += size; From patchwork Fri Mar 31 16:08:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0EE0C761A6 for ; Fri, 31 Mar 2023 16:10:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E6C86B009E; Fri, 31 Mar 2023 12:10:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 997506B009F; Fri, 31 Mar 2023 12:10:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 838166B00A0; Fri, 31 Mar 2023 12:10:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 708946B009E for ; Fri, 31 Mar 2023 12:10:29 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EEA60120DE7 for ; Fri, 31 Mar 2023 16:10:27 +0000 (UTC) X-FDA: 80629680894.14.DFF07AA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 0CCFEC001F for ; Fri, 31 Mar 2023 16:10:25 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=anD6L70s; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jPGukvQXP4Tai7PUqcbnIswr++Y11hB7rT1QDpn/afo=; b=4c10YoFbObNYUB6yameqQvQ4WDg0NPQ4gFlKpeJAjDnpxP7ifxDZEQGmAfZAVuw6ZWqDlT Id7DJGuAucuIQJM9CU3dPzpLk8l0KT2Iq/ib6THNsUeLO+d49qMKidVAGfjKXAihl+Fboq 4OeBS+6oGc6tRk55S5A+wVNqtczn6XI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=anD6L70s; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279026; a=rsa-sha256; cv=none; b=i1IQqtckz3ZIa6PdDpgdgr8WdyU+vHWayapD9zjiTV8NdcNz+ImXZFqzyc3lXPv4DLG5qg VG2Ef+XQqlbRwqV/f9RJd1n/0IidYpSUmZR1eVHOBSnTNfszyzsyCq5x8/VUToVRPbMpAT 83tp2lL9mPdpwkVsWJIW2PBjmfgb5vw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jPGukvQXP4Tai7PUqcbnIswr++Y11hB7rT1QDpn/afo=; b=anD6L70sne4EL7ZtHVNQ17VAFjA1ywyRnHFq487kMqxtCrl1QXX7yW1TsWV6q9bHlSTOrn wTUKBBGDlc5rcvR49aiEkIuA+R49XnCaFpCKSqr/e2DAK6xfbtaX0BiLnLnSDtN2/eo6ua +FM8mzQg6HTFBpfJQBXzJ0hti1qd+Wo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-570-w91nJgUEPQ-xufIcE0rgFg-1; Fri, 31 Mar 2023 12:10:23 -0400 X-MC-Unique: w91nJgUEPQ-xufIcE0rgFg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CC107886063; Fri, 31 Mar 2023 16:10:19 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B7DEB14171B6; Fri, 31 Mar 2023 16:10:17 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 21/55] crypto: af_alg: Pin pages rather than ref'ing if appropriate Date: Fri, 31 Mar 2023 17:08:40 +0100 Message-Id: <20230331160914.1608208-22-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Stat-Signature: axziuiybrbcjuunthugkccghmmswdbx4 X-Rspam-User: X-Rspamd-Queue-Id: 0CCFEC001F X-Rspamd-Server: rspam06 X-HE-Tag: 1680279025-47605 X-HE-Meta: U2FsdGVkX1/edr9uwgpI8QIywr/IwNQVe/UefQa6j5e6Yb0mUnV5Zu+aiI2jHRwUyLQw2C+S9iFPmkTu7MDHreXBm6F8u/pGFX9P/DJZl40CspnBOhlCPKvxS5xvJ2O5X2upVdeRUESBoPg5HRyDz1TKEw+bIFvrhcWuE7WzCpdJ23Jmjd6rGktK+e+P0/RCotk6Oe5rLAIZLtcNwEVSj/Xs9T1psFtrtDR6qTgONcK5AY2neUatjCwHof/qDg3rwnx5VBlPNAT+wTrI2lHYrv9HRpgPtH5spn5C8NMxKj9XtQ2Us/GS+ZLF6TsCmjD6G3zAguh15XDhzC2pzhJ/25H/eZnlQGWQtwL6WCBIRv3EXCmhut4sKTkB+DxxHIfCPoektGSWiErxq/2m6uCcxGD8bRkL2PvFshGh94oPZ7co+vb2GANdUGYmu8QRxGbSxJlQ37NqIkV36LrulDG/vxgf5k6MQnfyUBaDKL1XD0lhhaORthLiHPVtUFSnsQNmk/KcyBdrQvxBh5kPkud+rsfmY4QBxMxeDAAAqbwvJ08iI4rPXM8Oul2I093irb/+EFU+wcP1c/JiGAG35+N5Kb7GtFqiimz+7dz+IHpZSVVTsyHM/rcFDPO6erZI1duu/vnxgM088ExxoN3OPJQKOQmH8sTccEgFWBHTQ2+k7+5w0Y4ZEbekkfB1OMI9bzuOn10r9OzDsedChr7TiLcUZGnORAArubNn33yfNRs6JaVgQdR1rd224H49VgpNprGt15lRZJOWaspg3D/PYyDTtIXcg7xzJrk5nLBbrDiN0VjbhNWnAmauXPzWSA8ePpw/ApgyoSn/LZbEKbvYgTGf88iY4v4RDLfygw5wMGItqX3C6FR/kY9OAzcDizfUCm/LcQ/yzNmcE/1vf7lm8OScdFf5PT44YIb+j7qF+HH3M7kFDLN4RUuvVDZ54TbCqj9NvTS9MGRzLEtP1jYobTG lB0E3zdm AEfxfCFf8ull2QhpfCNLuZn6BHb0vh0Kd2C0DIn4p4YGvgX58PJU9ZYAYXA6oygL4RLqHymx/Om6zaLOFf307IRNBsQY7UEOFzC9BXBj0XB+PNRtyRfr+3QZKxXaVpEm/AKbn3CbqgtGzZ6VsjNZSWa+ATJik8q8FglHZ7nM1kb7ipbeZ6lJhEPMO8+wBZja0xFh4duHQYY+BKE3KVGvOIUfHZpLRxjNZaeRGYT9C3VDvzhoccn6UQSjf9SU1i9N7nBoDms2e/IlMf/q4OfM44IDEIU3ns97R0m5Kj7TIYZ257K2uXx3URv4j+yoXKog6cUFPbj+EuCpOwmb6SgFrtumCz/e1jYNHX3vqIBX/ZLz/z2C6halgaH+FzXDE0peVSfDnt5Xw9p0CTO/g8awEeeXmicJ+XFLUA7I90aUS9/IHOFxCOhAWNDOH9pORObm3gDqLyPmX28vNp7qqrbBvG8T2wAGCi4bG0jIDYeMv+SfEIy0y8de3nYueOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert AF_ALG to use iov_iter_extract_pages() instead of iov_iter_get_pages(). This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO-read rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O would otherwise end up only visible to the child process and not the parent). Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 10 +++++++--- include/crypto/if_alg.h | 1 + 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 5f7252a5b7b4..7caff10df643 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -533,14 +533,17 @@ static const struct net_proto_family alg_family = { int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len) { + struct page **pages = sgl->pages; size_t off; ssize_t n; int npages, i; - n = iov_iter_get_pages2(iter, sgl->pages, len, ALG_MAX_PAGES, &off); + n = iov_iter_extract_pages(iter, &pages, len, ALG_MAX_PAGES, 0, &off); if (n < 0) return n; + sgl->need_unpin = iov_iter_extract_will_pin(iter); + npages = DIV_ROUND_UP(off + n, PAGE_SIZE); if (WARN_ON(npages == 0)) return -EINVAL; @@ -573,8 +576,9 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) { int i; - for (i = 0; i < sgl->npages; i++) - put_page(sgl->pages[i]); + if (sgl->need_unpin) + for (i = 0; i < sgl->npages; i++) + unpin_user_page(sgl->pages[i]); } EXPORT_SYMBOL_GPL(af_alg_free_sg); diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h index 7e76623f9ec3..46494b33f5bc 100644 --- a/include/crypto/if_alg.h +++ b/include/crypto/if_alg.h @@ -59,6 +59,7 @@ struct af_alg_sgl { struct scatterlist sg[ALG_MAX_PAGES + 1]; struct page *pages[ALG_MAX_PAGES]; unsigned int npages; + bool need_unpin; }; /* TX SGL entry */ From patchwork Fri Mar 31 16:08:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF22C76196 for ; Fri, 31 Mar 2023 16:10:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A2786B009F; Fri, 31 Mar 2023 12:10:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 551A96B00A0; Fri, 31 Mar 2023 12:10:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F2426B00A1; Fri, 31 Mar 2023 12:10:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2F37E6B009F for ; Fri, 31 Mar 2023 12:10:30 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CD2611C6B89 for ; Fri, 31 Mar 2023 16:10:29 +0000 (UTC) X-FDA: 80629680978.24.742DDA5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id D009E180019 for ; Fri, 31 Mar 2023 16:10:27 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Kf7LLzDt; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xCO1FTHFb6ixqT+m8PCC52zI8ZJyvPRkEVPay5Q0l+o=; b=oC87IvYOT6OuHxfkoDzRUnNxS22BlrcUT08n7iF3N5BHBUp97zHYq17fAZJYJRKxTl0P2t ryS9Ku1vqOYIA3wwcW4dlaHHl8qDuH+3aw5rxd3X2AmGwBhuJJUV/5ObPZ/fFzXlcf+/nx lWhZHqZ5QjLgH/NmsKJY1lCFUumRwwA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Kf7LLzDt; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279027; a=rsa-sha256; cv=none; b=LRhwn3IVlvuZpxq1/FI7bxKTwuz8T93r+5Rz+diJc/YXNXQQkaGrMEZPzI0eIUR3vmVoOX qq3FrwqrzOjWRMmKX1dM/Z6Hkv2Lqytu1DV90VFVAtKldQKCc8qxc6B+uL/8rffavvFlHu sZlhlS193RQl3Nn1HHJoQpKy8RD7uGc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xCO1FTHFb6ixqT+m8PCC52zI8ZJyvPRkEVPay5Q0l+o=; b=Kf7LLzDtF4JSz6zyFMZ2RV+b/6mNIfJIoTC9aRtGldXbKv4PnXQyHNh0cVru8cakys46ZU pDa42/RU3r163iuLQTVHEqaMUOjMrQFIAaLBQp+Gm+BsfvxPWX5+Nd870njIcygOgJhIBW FK+ua1aitqJgmyZrqmUa+qifqlM6JbU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-601-E9J8PItkP7uGZG6CiLxyVg-1; Fri, 31 Mar 2023 12:10:25 -0400 X-MC-Unique: E9J8PItkP7uGZG6CiLxyVg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9CDB988606A; Fri, 31 Mar 2023 16:10:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BDFC492B00; Fri, 31 Mar 2023 16:10:20 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 22/55] crypto: af_alg: Use netfs_extract_iter_to_sg() to create scatterlists Date: Fri, 31 Mar 2023 17:08:41 +0100 Message-Id: <20230331160914.1608208-23-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D009E180019 X-Rspam-User: X-Stat-Signature: zqhgfrik3ozj94h3fxunouj36yd6qoch X-HE-Tag: 1680279027-6730 X-HE-Meta: U2FsdGVkX18r/NoA5CHkMfgRlJIHbTzaMmv4+SWKSZ2UFDQ3gWiO0vw5LWno5oB0oIqKO6zN5rXySHHCueIZYiWcoOxYPbKAkth3tOsSfpB3p27/D2ndZuAYWwoHtLkrV+a2757BBRfrLHwzPVRWAev8Jmyu2tokKUc0PE5KxxCylIvLkf9QDSLqYyMPhpOHaGD5x9cWEl1NxaqlVKXw1oNLUXit4vs35NMM4hDbJqMbCRdhfJrYtxfYevvbfUvcaLum6b/JcPEbGtFWbby4thpi6TfoXqyopA/MC/UtGO2/fv1OtR76xkilGlQ5q0R5B1BktIQnFaPMP4SZWtkyJPLBhNKtBI9SKXZCUwEigbFKp673dQaoRblzu2CoX+nQWrFW6w7r2v/rGQli0/CIrrA6KoMENPEm7Lguvec1c9u91aNjGuzTIWNjpOAi/prxCiMP5Qh6jOR7v+uM5M2AGqg+bpMPtizg2Zj4/7aUBH+GHm3rGBEv4iKATMZwQ7dIq9Txwj/+PWILRmlX7xXYY+mPnLgK87K0FQ2Atps9UhEp9dW8pG5BABbLQWbne45aHIY3i54k0V/fE1ADcTiNv1I8krb5lIaRPiibbxDOyeA3uMcy0wpc7U+sQRn4YI7bY1HlBJZnzIZP5IJPQExobPEagds70nflsi0Ao4UNfMM6hRZOZvNnI1ltS9ylZmMZiX6GMSWZyJ1gqzJ0M8AMIaEbjZQ7/VGU5WGJ6/gBFkDYjcwAVuk/sqnfAue6Ws9MXscyFdC8VZi/EMKHUIi0Cb6zs9pWk2JE1k2X60JUDs0ik8RNQ7Syh0kYEmvLQbwvkM/DyyNbiP0E72kwgzfJ0xlfbt9IhAYTGRPqJSAaqdIlYSHhbnTGU83hMnObHjwfrttQgutDetEhLTtE/IRjqcGFQJucxBnLRv/Xsj1Ce1j87ZCXQr8A6lnASzYV0fW3kXvW4Tyda0sWY5J1VPo BpvUlyRy 7ZkBhh6I+lFh1464oPhrOx5tq7pBzJjiyWPytwlJCv4QMRjEjbPHZq2r9NHjt5qtchn3HRujsmdujRejFk00yrFeBLfelvdBuOGwkKv4rlGmzgIvJZHW82SfTBITfNEkktOriYM7mYm3AoONWvdwM2Ne8ZCg5q8byrvsEh1U0DoavpcWQ9Adau7M941LAO6E23hb05cEkE/jdSRnz5A2f/yl0ZpVvjaIS8hmTTlo4nZLfPGWAFw+X+Aobg/jJOJJFTuH5WxdO1O6V8pefVGLaANU58b5kgvHsn8TuStzowcPNhP08HeKOsD44d/4nxvRrL8umlFDQ8uWjW0J6RUKcZkT6sf48qePYjv4KABkk7XRBxJKns5oi4MI9ulIX7XoJGavcyxJwCMxJX9aRXPmkdXDeSl5u7AlHtflh0a9+Ht9EoQBxJ649mNpUhEweuPnfbcFxqOm+Jqj8Qoi/uKPwPdWYKA1ZTN3BG6JAZGgpDDvtXdt/hy2BLsLeFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use netfs_extract_iter_to_sg() to decant the destination iterator into a scatterlist in af_alg_get_rsgl(). af_alg_make_sg() can then be removed. [!] Note that if this fits, netfs_extract_iter_to_sg() should move to core code. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 55 ++++++++++------------------------------- crypto/algif_aead.c | 12 ++++----- crypto/algif_hash.c | 18 ++++++++++---- crypto/algif_skcipher.c | 2 +- include/crypto/if_alg.h | 6 ++--- 5 files changed, 35 insertions(+), 58 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 7caff10df643..1dafd088ad45 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -531,45 +532,11 @@ static const struct net_proto_family alg_family = { .owner = THIS_MODULE, }; -int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len) -{ - struct page **pages = sgl->pages; - size_t off; - ssize_t n; - int npages, i; - - n = iov_iter_extract_pages(iter, &pages, len, ALG_MAX_PAGES, 0, &off); - if (n < 0) - return n; - - sgl->need_unpin = iov_iter_extract_will_pin(iter); - - npages = DIV_ROUND_UP(off + n, PAGE_SIZE); - if (WARN_ON(npages == 0)) - return -EINVAL; - /* Add one extra for linking */ - sg_init_table(sgl->sg, npages + 1); - - for (i = 0, len = n; i < npages; i++) { - int plen = min_t(int, len, PAGE_SIZE - off); - - sg_set_page(sgl->sg + i, sgl->pages[i], plen, off); - - off = 0; - len -= plen; - } - sg_mark_end(sgl->sg + npages - 1); - sgl->npages = npages; - - return n; -} -EXPORT_SYMBOL_GPL(af_alg_make_sg); - static void af_alg_link_sg(struct af_alg_sgl *sgl_prev, struct af_alg_sgl *sgl_new) { - sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1); - sg_chain(sgl_prev->sg, sgl_prev->npages + 1, sgl_new->sg); + sg_unmark_end(sgl_prev->sgt.sgl + sgl_prev->sgt.nents - 1); + sg_chain(sgl_prev->sgt.sgl, sgl_prev->sgt.nents + 1, sgl_new->sgt.sgl); } void af_alg_free_sg(struct af_alg_sgl *sgl) @@ -577,8 +544,8 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) int i; if (sgl->need_unpin) - for (i = 0; i < sgl->npages; i++) - unpin_user_page(sgl->pages[i]); + for (i = 0; i < sgl->sgt.nents; i++) + unpin_user_page(sg_page(&sgl->sgt.sgl[i])); } EXPORT_SYMBOL_GPL(af_alg_free_sg); @@ -1292,8 +1259,8 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, while (maxsize > len && msg_data_left(msg)) { struct af_alg_rsgl *rsgl; + ssize_t err; size_t seglen; - int err; /* limit the amount of readable buffers */ if (!af_alg_readable(sk)) @@ -1310,16 +1277,20 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, return -ENOMEM; } - rsgl->sgl.npages = 0; + rsgl->sgl.sgt.sgl = rsgl->sgl.sgl; + rsgl->sgl.sgt.nents = 0; + rsgl->sgl.sgt.orig_nents = 0; list_add_tail(&rsgl->list, &areq->rsgl_list); - /* make one iovec available as scatterlist */ - err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen); + err = netfs_extract_iter_to_sg(&msg->msg_iter, seglen, + &rsgl->sgl.sgt, ALG_MAX_PAGES, 0); if (err < 0) { rsgl->sg_num_bytes = 0; return err; } + rsgl->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); + /* chain the new scatterlist with previous one */ if (areq->last_rsgl) af_alg_link_sg(&areq->last_rsgl->sgl, &rsgl->sgl); diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index 42493b4d8ce4..f6aa3856d8d5 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -210,7 +210,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, */ /* Use the RX SGL as source (and destination) for crypto op. */ - rsgl_src = areq->first_rsgl.sgl.sg; + rsgl_src = areq->first_rsgl.sgl.sgt.sgl; if (ctx->enc) { /* @@ -224,7 +224,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, * RX SGL: AAD || PT || Tag */ err = crypto_aead_copy_sgl(null_tfm, tsgl_src, - areq->first_rsgl.sgl.sg, processed); + areq->first_rsgl.sgl.sgt.sgl, processed); if (err) goto free; af_alg_pull_tsgl(sk, processed, NULL, 0); @@ -242,7 +242,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* Copy AAD || CT to RX SGL buffer for in-place operation. */ err = crypto_aead_copy_sgl(null_tfm, tsgl_src, - areq->first_rsgl.sgl.sg, outlen); + areq->first_rsgl.sgl.sgt.sgl, outlen); if (err) goto free; @@ -268,8 +268,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* RX SGL present */ struct af_alg_sgl *sgl_prev = &areq->last_rsgl->sgl; - sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1); - sg_chain(sgl_prev->sg, sgl_prev->npages + 1, + sg_unmark_end(sgl_prev->sgt.sgl + sgl_prev->sgt.nents - 1); + sg_chain(sgl_prev->sgt.sgl, sgl_prev->sgt.nents + 1, areq->tsgl); } else /* no RX SGL present (e.g. authentication only) */ @@ -278,7 +278,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* Initialize the crypto operation */ aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src, - areq->first_rsgl.sgl.sg, used, ctx->iv); + areq->first_rsgl.sgl.sgt.sgl, used, ctx->iv); aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen); aead_request_set_tfm(&areq->cra_u.aead_req, tfm); diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index 1d017ec5c63c..f051fa624bd7 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -14,6 +14,7 @@ #include #include #include +#include #include struct hash_ctx { @@ -91,13 +92,20 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, if (len > limit) len = limit; - len = af_alg_make_sg(&ctx->sgl, &msg->msg_iter, len); + ctx->sgl.sgt.sgl = ctx->sgl.sgl; + ctx->sgl.sgt.nents = 0; + ctx->sgl.sgt.orig_nents = 0; + + len = netfs_extract_iter_to_sg(&msg->msg_iter, len, + &ctx->sgl.sgt, ALG_MAX_PAGES, 0); if (len < 0) { err = copied ? 0 : len; goto unlock; } - ahash_request_set_crypt(&ctx->req, ctx->sgl.sg, NULL, len); + ctx->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); + + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, NULL, len); err = crypto_wait_req(crypto_ahash_update(&ctx->req), &ctx->wait); @@ -141,8 +149,8 @@ static ssize_t hash_sendpage(struct socket *sock, struct page *page, flags |= MSG_MORE; lock_sock(sk); - sg_init_table(ctx->sgl.sg, 1); - sg_set_page(ctx->sgl.sg, page, size, offset); + sg_init_table(ctx->sgl.sgl, 1); + sg_set_page(ctx->sgl.sgl, page, size, offset); if (!(flags & MSG_MORE)) { err = hash_alloc_result(sk, ctx); @@ -151,7 +159,7 @@ static ssize_t hash_sendpage(struct socket *sock, struct page *page, } else if (!ctx->more) hash_free_result(sk, ctx); - ahash_request_set_crypt(&ctx->req, ctx->sgl.sg, ctx->result, size); + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgl, ctx->result, size); if (!(flags & MSG_MORE)) { if (ctx->more) diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index ee8890ee8f33..a251cd6bd5b9 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -105,7 +105,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, /* Initialize the crypto operation */ skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm); skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sgt.sgl, len, ctx->iv); if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) { /* AIO operation */ diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h index 46494b33f5bc..34224e77f5a2 100644 --- a/include/crypto/if_alg.h +++ b/include/crypto/if_alg.h @@ -56,9 +56,8 @@ struct af_alg_type { }; struct af_alg_sgl { - struct scatterlist sg[ALG_MAX_PAGES + 1]; - struct page *pages[ALG_MAX_PAGES]; - unsigned int npages; + struct sg_table sgt; + struct scatterlist sgl[ALG_MAX_PAGES + 1]; bool need_unpin; }; @@ -164,7 +163,6 @@ int af_alg_release(struct socket *sock); void af_alg_release_parent(struct sock *sk); int af_alg_accept(struct sock *sk, struct socket *newsock, bool kern); -int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len); void af_alg_free_sg(struct af_alg_sgl *sgl); static inline struct alg_sock *alg_sk(struct sock *sk) From patchwork Fri Mar 31 16:08:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE631C76196 for ; Fri, 31 Mar 2023 16:10:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2D8A6B00A1; Fri, 31 Mar 2023 12:10:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDD566B00A2; Fri, 31 Mar 2023 12:10:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCBA06B00A3; Fri, 31 Mar 2023 12:10:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BE2E66B00A1 for ; Fri, 31 Mar 2023 12:10:35 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 914211A0CFE for ; Fri, 31 Mar 2023 16:10:35 +0000 (UTC) X-FDA: 80629681230.07.0EC0F99 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id D8EE280028 for ; Fri, 31 Mar 2023 16:10:33 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XI6amrfh; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279033; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cnsl2PBKcn/WeJcDUlvQg+kDRV/xI85q/f39rakA9vk=; b=bv7h8vribHqIl6vZhOsmiC7A0dHGguGh6dqnqd3MztZcFI/K6yp1/zYcifNQgbAQyb56sG HbNvzoJuudd7PxzCqPLraQ3lgj0FUBNbXPAsxuMZRylogv15Ts9bCkqoQGZqYgS6uMHP+1 At35AUCY4U8YT041+Filt9i0bxJXpik= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XI6amrfh; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279033; a=rsa-sha256; cv=none; b=yRcSDoF5v9tvSLv2Z0hEiSXw7+cfKRbFokh1u8DyXCgTYi/GQ5Muf/Bk5326kCOfpV2Kau H3VGlirOnL5EcLSb5hrcpsYbHTfdScQZUtlswIUPWGW49lg64VCzv6vCUaJ8A8B2e9cRaA skg05JN0P5G94WDWsiYCvWjtIcWVug0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cnsl2PBKcn/WeJcDUlvQg+kDRV/xI85q/f39rakA9vk=; b=XI6amrfhRiWRFwg5EjOLOK2iogcx337hgNfwNFXjcryGyD3zSmQgwI1vb5IL7A80QwWmTe 3Z94TKdsDo+2E5epf2kFhV5BTx0Y86fdU9WzCcG2mmJxsFGJWzbqJxMxb9S9cPxH6lueh3 /+yaUc0S/jHFUbdefhu9ILQ0yGzZSBQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-670-w26OBUVGPfurU15AUpPLuQ-1; Fri, 31 Mar 2023 12:10:26 -0400 X-MC-Unique: w26OBUVGPfurU15AUpPLuQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4B9C3380452A; Fri, 31 Mar 2023 16:10:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A525492C3E; Fri, 31 Mar 2023 16:10:23 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 23/55] crypto: af_alg: Indent the loop in af_alg_sendmsg() Date: Fri, 31 Mar 2023 17:08:42 +0100 Message-Id: <20230331160914.1608208-24-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: waa9wmdo3qobbxxmhkdnd73kzt851u1f X-Rspam-User: X-Rspamd-Queue-Id: D8EE280028 X-Rspamd-Server: rspam06 X-HE-Tag: 1680279033-436587 X-HE-Meta: U2FsdGVkX1+p2yFPwY/DEkn3f+eFucXqyJ9y+A+glY6WSAYir1gjawbTvm+LWIeZ5d2R1ljqW+sVidHfx1JyetFlhFh5m4+pJeYE3rMuNh4LIDc5WpFJ1psQedjMGARF5GD0p30ehVXpQG/ziIxcNG2rNIXwG/lx4o6OMqUqGIgSe9MwoLEW7G2Top80+nlROboxdjuAoQtsVV5+1lvu31dCaKhhHUp617cP7yj9ExP3PPDAgNSZcVRJXz30xJcx02cHkgi55kwonm0W4lOUHWfXvS0qQs3FFcmAg4O+9gO4RorNcNyV1jCJafyraD9lr2Mibr2N+84xRkh9tNJFBviGc5bQvz1neRRfykCYGPtP+JHvC0O1aAwCybV3gz2rQr+hHt7X9PVSj0iZQbEN3BkXCmbhqUCw2vfqZHi5n/8gHNnsrH/hZdlAAtyWPZZSz/Xcx3YtCdumnqfqT9sOrqy0C1hi3RaMAXI9u3G9DSO+9F9YmAICRsnp8rBNpwh4x7vv6qK2oH2QUoN5q+uYsQzbrP6hb2vdaHga/E0Ia7J1Y5gDCE+yyGGGCJ8jhr7FH5gt2alwgIZ3U3s+ippgEflxfxqBCbhLEgCgNqvhCcH96TP2dWeaiivvwBkYgz9Ap5mCqnvoQEjbyUXhtuP7k4cUCh2cAeJiDeZgrB9lNIY8t1iPuhTCh2XzrHxqHNAZJ7OytOH6BdoDaLMjVj/4N2U8gGIwqMxJ6VnHSX3HwQEmUzu9wvRGtCaxccA78U33J+yfeMsyHOGIQUcsDxlksOSzSFXY4+PPh5+DQC6QT6xI8PSBsRx5CRLirW0u2JDj0ckpRp4nofQ3G1U34eYZ3RNjrGF8z0njLHjYbvLxEim2BNOQfVmkCfk6rQ1L4+Ct/Na4BzTouajfFIrGi7B8xO2yLnnJdtnfJHEFodtZBnPe1PWT6Lq3nRpwVk98UiEhHV+1eV2OcPhxsn5COcV mQH789Wo uTVxUBXwvNP5KpxE6uLlyt/iojIS8MYXVvq13fZptBfbnGVBdfUPc3NiopRHYngdgNCr7Q07dZuhhbfk8g2MNra9l4Aahk+I1iZY1dDKkzGJurFQREPIb2KHrJQJOmQlAHm6FdDFiK0CrSH1O4GVmybURKFOURynveE3PjkMgsic6jheJpDxmGheSYyTPzA9Al1Fka9xIUOHirTbmMbaLw1wJ1Se3in5cvoadM7NcoSFgNZeDa32pPB2Fpe1IuUAIQmF4zRwxOIc+pl2DjNVvyDOoOojVcoB3XCaAsqBXKIQBRjilEsFJdzEjGmheuSmQoLyO0XuwZXQFI6NTqhwhPTOWZFD4RekXUqn/5Suf8ye3wbbYbjF8ddVXd5kBvWYnp0MFsKbQpQZqsyAjd/1IWBm9fHNRq4Z2w46ZVnr75ezkHSzml+0M6Zcyb7EovWKXfKE/geEqjv0bjUdnAiIY+a3MuI7SLXqXqC7EJ4bFsGVkvFS8xD+vh3Loqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put the loop in af_alg_sendmsg() into an if-statement to indent it to make the next patch easier to review as that will add another branch to handle MSG_SPLICE_PAGES to the if-statement. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 50 +++++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 1dafd088ad45..483821e310e9 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1031,35 +1031,37 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - do { - struct page *pg; - unsigned int i = sgl->cur; + if (1 /* TODO check MSG_SPLICE_PAGES */) { + do { + struct page *pg; + unsigned int i = sgl->cur; - plen = min_t(size_t, len, PAGE_SIZE); + plen = min_t(size_t, len, PAGE_SIZE); - pg = alloc_page(GFP_KERNEL); - if (!pg) { - err = -ENOMEM; - goto unlock; - } + pg = alloc_page(GFP_KERNEL); + if (!pg) { + err = -ENOMEM; + goto unlock; + } - sg_assign_page(sg + i, pg); + sg_assign_page(sg + i, pg); - err = memcpy_from_msg(page_address(sg_page(sg + i)), - msg, plen); - if (err) { - __free_page(sg_page(sg + i)); - sg_assign_page(sg + i, NULL); - goto unlock; - } + err = memcpy_from_msg(page_address(sg_page(sg + i)), + msg, plen); + if (err) { + __free_page(sg_page(sg + i)); + sg_assign_page(sg + i, NULL); + goto unlock; + } - sg[i].length = plen; - len -= plen; - ctx->used += plen; - copied += plen; - size -= plen; - sgl->cur++; - } while (len && sgl->cur < MAX_SGL_ENTS); + sg[i].length = plen; + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + sgl->cur++; + } while (len && sgl->cur < MAX_SGL_ENTS); + } if (!size) sg_mark_end(sg + sgl->cur - 1); From patchwork Fri Mar 31 16:08:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29371C761AF for ; Fri, 31 Mar 2023 16:10:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 507B76B00A0; Fri, 31 Mar 2023 12:10:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B58F6B00A2; Fri, 31 Mar 2023 12:10:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3558E6B00A1; Fri, 31 Mar 2023 12:10:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 21B0A6B007B for ; Fri, 31 Mar 2023 12:10:34 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 06BD61C6B7B for ; Fri, 31 Mar 2023 16:10:34 +0000 (UTC) X-FDA: 80629681188.02.A1E9763 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 2C63C1C0005 for ; Fri, 31 Mar 2023 16:10:31 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BPZAW7zV; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279032; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bQ5QnEnteMQfZ4j7G0XrCpLzGl+D3sZ9ge7q0vobANw=; b=7Z/WhxuDj5X0EfXb//HFT7dANrbiEtF1YwTLX/1Gl5j18FAgAeWshzl8AMfvDnVAH5eO9I 3IB57SwfgQkzDXOl1VJ2lT1HCNQCrJsV5NyjLdAzqqldOC3FbwrUJbOFWkFyzGo0sCVu8Q 7itbvwD2TuiEfkroqIsAsc2KJbjx+kY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BPZAW7zV; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279032; a=rsa-sha256; cv=none; b=PAQKPlZRW/AGuXCJqgaQ+0lDce0UzzQPnRDQIDuuu41V4Ar4wVJVh848aStmqWRg+omO6H YZz28NZZokG7ooLCMlWz/9q/v2BYwNdqE4MhOrzMDPVvUAwooRqB4PAbTa6TU0l3J6Igxp c/dN5YUnr5WXsBBJznA1ut76ZH4Z8gQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bQ5QnEnteMQfZ4j7G0XrCpLzGl+D3sZ9ge7q0vobANw=; b=BPZAW7zVRpdCJa6vInm9nI+YfIwuKXGQjgB7mOVgFD0FzNWDpKL7Y7qi9RpNeGleNpDH0d P5NO9iv1uZMTrEcNoDdorJlAbvqZ2gNn9Ae+yscVf64qECpNHp8SZoH5t0BOlsP1TkX5ud zJy4fTqkiI/OToM19dyzZUL5sBLsu80= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-451-zyE999oJOe6JnagrFD2R3w-1; Fri, 31 Mar 2023 12:10:29 -0400 X-MC-Unique: zyE999oJOe6JnagrFD2R3w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1847B85A5B1; Fri, 31 Mar 2023 16:10:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0691D1121315; Fri, 31 Mar 2023 16:10:25 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 24/55] crypto: af_alg: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:43 +0100 Message-Id: <20230331160914.1608208-25-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: gac5tpfy9ze1sdo144wtkdtinimxntcz X-Rspamd-Queue-Id: 2C63C1C0005 X-HE-Tag: 1680279031-846848 X-HE-Meta: U2FsdGVkX1+a6kiFY4OMwMvwIqUI5BhZkxqaIiATaT2Nv+LciFkHT6zSQ6+KSzT81eAG1tqlIEjV8gBRlqKb1D7FgIm/4mH9SJKNlMmmfSHtuengG+9kkhQHBOYu5mgju9G4kj0E9XlFFn7BDwNrpy8YLcx19x3hjsQbn0r0Vud+YLzZDRR7MP5jv0jGkPnKwZ4GXGDUJM//ZWdDK9CnS3d9e/ECA5HPC7O+pNMELQ35DR1yYY41dHBnPlEA/2C2zGq1Z6DK855xg2nqUa0ECt/U24JUhiGxoGoxKalLuyShVpR5l6a6nwisNJTYfkMmfZt9mjilGECTY9KSFUudd8j86g0KqQiaSMWmJUKLs9zFobb2D4l/ymOaDW+1BnUgNPOsyn+FeiM/W8vBFofrkDr202Vf62UXajLeOnqbDxDYvWqt8Bo8C34hlalhBUjMv/rv0VqDlT0ALw+r5PWmzhrNHtglowG+9gLRZiDNbDPisxiFEZFeE744fGYiTAcOrbCjhmoFc8i4sJDeiWzdbNsABA9ATsZcfpZXrYYIgvVuECVZQP40K/+VtJsF7qLgXKRcBV4kNFITmLbwRgPEriZ4mzi8yJCCC47wOiXE5W7SEk9q3UfHDpCjeanUOgIquaw9QuXXpgpu4hGmdmn6DGaWDAPCR9gWdSmHe8fN8P8P48jX1BeYSJlmO6qltzsz6fwZ3yOs/W7YLEc0YivgqxHsqIsRegGgwy4jYKO5w4N4bwcDVttINCPeSzJVDwlbcjrZiTtdIodWX6ArII/Bsey3yj6qnOiqZCMj1odtAOFGh67Qpouk0oRQeBr+9t2ZzmGp4NwbonVbZdEsFrCcPIf+0D28HQ+qQIoK7uymzZpm9MujNJEWYWWOXgjmxB6kuK4FOlxEzaLUz3+lUhrcCiSCjde4OxvD/au6lsvvDF5c3jmamNeczzBOnLJ2Bcpl67L0tM49PkGyrE1rlYW Fhs5YrI6 3EP1Mn8ye6gfqtdjwM6ruTGWLqNUscdkAAkE6duZ4lPSMCbIPpIPYu6rgOQHtjsnWLwyeOq2eLHUx0ewcA7yMH7Fuo6LzEWYxEGGrPlG2q9PAMBsn+v3wxG/43fV19j84YzaoDKLciKXOtnJVptwliPAneGdwGCjpyGWXNVPjlHg+B+aRWGqo4OGv+y1e0SuaZeiSxfRhuHF/TJ/V8wxKQOoC80Dmn8VjXMJ2INF0QdHnk4klEx5iRMC2SsJVGfW2ozimqyGUv5YOro1RlOC/A+ExIeAhiee2NBPygXPhOoRjxJdJHHH2qDK+ZHIQuLwuX31cZflLzuh1ZslYNzFbLgeS77SoGRXa3TdvibojVE83Es/mZy3f47ysEE0/0LZ7M3hKL0yPjTlffzEQ4xLWtR1dGYbM9Hdi6h7iHIV0tGtPys/lrqSY7/wty3C2+vJzaYyLUAV2Fr2VXmSwlP0JD6vWQv4lLfP7ylEUqOObkRdxFdR6NenNZtSPoQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_ALG sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/Kconfig | 1 + crypto/af_alg.c | 28 ++++++++++++++++++++++++++-- crypto/algif_aead.c | 22 +++++++++++----------- crypto/algif_skcipher.c | 8 ++++---- 4 files changed, 42 insertions(+), 17 deletions(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 9c86f7045157..8c04ecbb4395 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1297,6 +1297,7 @@ menu "Userspace interface" config CRYPTO_USER_API tristate + select NETFS_SUPPORT # for netfs_extract_iter_to_sg() config CRYPTO_USER_API_HASH tristate "Hash algorithms" diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 483821e310e9..3088ab298632 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -941,6 +941,10 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, bool init = false; int err = 0; + if ((msg->msg_flags & MSG_SPLICE_PAGES) && + !iov_iter_is_bvec(&msg->msg_iter)) + return -EINVAL; + if (msg->msg_controllen) { err = af_alg_cmsg_send(msg, &con); if (err) @@ -986,7 +990,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, while (size) { struct scatterlist *sg; size_t len = size; - size_t plen; + ssize_t plen; /* use the existing memory in an allocated page */ if (ctx->merge) { @@ -1031,7 +1035,27 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - if (1 /* TODO check MSG_SPLICE_PAGES */) { + if (msg->msg_flags & MSG_SPLICE_PAGES) { + struct sg_table sgtable = { + .sgl = sg, + .nents = sgl->cur, + .orig_nents = sgl->cur, + }; + + plen = netfs_extract_iter_to_sg(&msg->msg_iter, len, + &sgtable, MAX_SGL_ENTS, 0); + if (plen < 0) { + err = plen; + goto unlock; + } + + for (; sgl->cur < sgtable.nents; sgl->cur++) + get_page(sg_page(&sg[sgl->cur])); + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + } else { do { struct page *pg; unsigned int i = sgl->cur; diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index f6aa3856d8d5..b16111a3025a 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -9,8 +9,8 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be + * filled by user space with the data submitted via sendpage. Filling up + * the TX SGL does not cause a crypto operation -- the data will only be * tracked by the kernel. Upon receipt of one recvmsg call, the caller must * provide a buffer which is tracked with the RX SGL. * @@ -113,19 +113,19 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, } /* - * Data length provided by caller via sendmsg/sendpage that has not - * yet been processed. + * Data length provided by caller via sendmsg that has not yet been + * processed. */ used = ctx->used; /* - * Make sure sufficient data is present -- note, the same check is - * also present in sendmsg/sendpage. The checks in sendpage/sendmsg - * shall provide an information to the data sender that something is - * wrong, but they are irrelevant to maintain the kernel integrity. - * We need this check here too in case user space decides to not honor - * the error message in sendmsg/sendpage and still call recvmsg. This - * check here protects the kernel integrity. + * Make sure sufficient data is present -- note, the same check is also + * present in sendmsg. The checks in sendmsg shall provide an + * information to the data sender that something is wrong, but they are + * irrelevant to maintain the kernel integrity. We need this check + * here too in case user space decides to not honor the error message + * in sendmsg and still call recvmsg. This check here protects the + * kernel integrity. */ if (!aead_sufficient_data(sk)) return -EINVAL; diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index a251cd6bd5b9..b1f321b9f846 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -9,10 +9,10 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be - * tracked by the kernel. Upon receipt of one recvmsg call, the caller must - * provide a buffer which is tracked with the RX SGL. + * filled by user space with the data submitted via sendmsg. Filling up the TX + * SGL does not cause a crypto operation -- the data will only be tracked by + * the kernel. Upon receipt of one recvmsg call, the caller must provide a + * buffer which is tracked with the RX SGL. * * During the processing of the recvmsg operation, the cipher request is * allocated and prepared. As part of the recvmsg operation, the processed From patchwork Fri Mar 31 16:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0938C761AF for ; Fri, 31 Mar 2023 16:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52CA26B00A2; Fri, 31 Mar 2023 12:10:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 469206B00A3; Fri, 31 Mar 2023 12:10:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BB186B00A4; Fri, 31 Mar 2023 12:10:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 11B256B00A2 for ; Fri, 31 Mar 2023 12:10:38 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E91681A0CCE for ; Fri, 31 Mar 2023 16:10:37 +0000 (UTC) X-FDA: 80629681314.06.5B628D6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id EB4AC8001D for ; Fri, 31 Mar 2023 16:10:35 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwG5FE3d; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mERLnHIZ7Ufhp5QAaSwmDcCxGjZ2CWUtcWDMeYUTupY=; b=shThSy1OmpcmN1FUbD9OxnSv5BRkN9M1+mNyEyhU5HYSMbvqw9hiIIzUrmNlGHwSrPxSs0 wxsZZfckvaXxM/gbR1dnfMNKROdM4VWQLF1za3FqbBZ49Z16beuSLSSDe7j67jM742nKzk uXNyAp5Hu8TVk7VzkXAlhg9nBGsaOxM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwG5FE3d; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279036; a=rsa-sha256; cv=none; b=yQBZvtGmEap9Znu7RCxjH7ROigpSArwHuXdCLNG4Bs5mRQ94Ow4WM4lvew3uHjp73JdpzK 5026E7+PgglXtqyHJfZ4FcTm/YoBiSOYRVpZs7yYoWTcp5ic+ajwBYT3DqloWCvRC5PLCc mi9cl5QuZ+6Ec/T1aMsawV9PF6ojT+s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mERLnHIZ7Ufhp5QAaSwmDcCxGjZ2CWUtcWDMeYUTupY=; b=gwG5FE3dD6oDSnG8oyYnunUWjfkBJ1xaS3YoQU3idWN+rmI453zIieNYCOjl6HRLSDIkhc xDmvDwdHos2bFrkSMMnnByyek530O2xJjdiXvm4V2ZMVNcGQbBmfEmj3FaIGcgoMqF5Bo8 KJwFZT0X3RbmS6RfP9rjAX9cl2fvVAw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-396-nbKIJurtO2appj6WS6snYg-1; Fri, 31 Mar 2023 12:10:32 -0400 X-MC-Unique: nbKIJurtO2appj6WS6snYg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C13C3886068; Fri, 31 Mar 2023 16:10:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id ADCA81121314; Fri, 31 Mar 2023 16:10:28 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 25/55] crypto: af_alg: Convert af_alg_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:44 +0100 Message-Id: <20230331160914.1608208-26-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: EB4AC8001D X-Stat-Signature: 4a5gt1myma3igwqnfz8c1yoajxctx5t3 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680279035-801714 X-HE-Meta: U2FsdGVkX1/57OSDQi+EiIQ3ocqzuyvOezXwiL/46Y+xkyhUXtB7RvMFc6KKA1mo0/QL04hUiXJ3VyeB+UuP1Oda55zbBVViL0mkYUY3nOqs6XQIuEvkPVUzKbaIrnjg7yF7DWj3q5QsVEnumKsCHrSbLrOrGKatuIEXZ6w9+l7RTy3dimUQATDrPZn+BoBOemDZ3m56i5Ge7RkECEx08DH71vVnDMKYkgw0ozjFOADf4zOfZUyuGPjr2MBWaNbvJJVhhQ2v0ROkuVt3osnO0EVoY9T0PFdL281y/Tgako87daatgXZnH1cwYs8B8ZB+feApgXbXavsy0rCiXabY9nu9aG4j1kPF/+bI73+v4sba2dbqV7xCQxmkyvu63Hic0C0ADAcJ2SimCUsFhYcP/f8cmzfWaQWNYrnw2X1X1ccrQCSEFmd3u+T/rHibqmgRyICe7nfgNosY9SlacOfnG03PKoriKemIqZSIUSB+8QTrD9GexmXmxAmIAfaLLZaJ+57hlxRim77KSZxma3e+hb7lt/oDnAMGJaY/yzTxWG3RGZzAf3TCQAJv4jPEzYoDowEd9RHaZxpj1KCe+HAz5htTpJRtMYL1ZmKnDtpdRMD57T5riwuBdSMaIa/ly9a+Nv55EWgtWqvLrMZ6R2XRuJ3iHvqHPKrBxq05Eby4jBy7R0NHG3H7OQw1pbXRGjOrxqhT6KpSZ2hbxillXJVZXQanXJiO/mlnJEK7VbPvorE1nsrN92AXcixzEpgSOQpkM44gC9JRgw/KSTl+5XiOIPI3qOuDd9UobD4bPKexcDojFkiSOnu4K3Ae6sTNT+jso1ka9dipZaxBZWVFkmb3rNE3xx/M+zPdlpRIzZoTtzX8UAMSG1jskGpNSlhPhxoHCoDzaOJ3qyOAkiTkP/0o34V00pg1blZ6JyZJsmgmkdjN+8kdFtFIR8adcw6OWfT30tgQlxifiPXhm6kqK8R J0YiHc41 rBP2UxqJ06cqPlWeRb8iZq1rIE2As6Zf+IIO4wDrAqbx6/s4W80EB5OZY2rkms/talN+uj/Ywy7bJL/INRdhxcLbSHNeA+A8dWMDPa3VyLiKyO8ABqP40sJ9hOKCXsOAgvnfT0fpn5lwT2Gl6TKE0Wvd/wp2DzvH1R6gjOIIs/Db3LsSTimVbKmHAC8So5t+zZE9SeLWMHhTuHqaL1FAXiEFSDy3+MDom7AmhVAxDCwlFtUbPAuU3UxxT1UDFw8ZZhazyAJi7CI1DhmF7jkT+NpJpIo4cJjuCbq/h2gcUU0GjWdQdytUoCefr6ERpb1IiNfryoqBPrJ/hT8kSOsZ5lvNHtnw5sSFog8AS5wJBxrx0nS54a5M4GLBTmTqf3gpdl8v7fGMxKqrUOrBL05u5FF91t02P/uWYSV3aAlTWb/aYOdW4WVjz+QjgL7Mai9od+5Ck4K9lzBDkb6XHIGglglmiKlAoowsUikZkBvpfv+hSDnt4nzUz1uVsTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert af_alg_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 53 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 44 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 3088ab298632..7fe8c8db6bb5 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1118,53 +1118,18 @@ EXPORT_SYMBOL_GPL(af_alg_sendmsg); ssize_t af_alg_sendpage(struct socket *sock, struct page *page, int offset, size_t size, int flags) { - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct af_alg_ctx *ctx = ask->private; - struct af_alg_tsgl *sgl; - int err = -EINVAL; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - if (!ctx->more && ctx->used) - goto unlock; - - if (!size) - goto done; - - if (!af_alg_writable(sk)) { - err = af_alg_wait_for_wmem(sk, flags); - if (err) - goto unlock; - } - - err = af_alg_alloc_tsgl(sk); - if (err) - goto unlock; - - ctx->merge = 0; - sgl = list_entry(ctx->tsgl_list.prev, struct af_alg_tsgl, list); - - if (sgl->cur) - sg_unmark_end(sgl->sg + sgl->cur - 1); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - sg_mark_end(sgl->sg + sgl->cur); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - get_page(page); - sg_set_page(sgl->sg + sgl->cur, page, size, offset); - sgl->cur++; - ctx->used += size; - -done: - ctx->more = flags & MSG_MORE; - -unlock: - af_alg_data_wakeup(sk); - release_sock(sk); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return err ?: size; + return sock_sendmsg(sock, &msg); } EXPORT_SYMBOL_GPL(af_alg_sendpage); From patchwork Fri Mar 31 16:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96B9FC761AF for ; Fri, 31 Mar 2023 16:10:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 309536B00A3; Fri, 31 Mar 2023 12:10:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 291EB6B0075; Fri, 31 Mar 2023 12:10:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 159D86B00A3; Fri, 31 Mar 2023 12:10:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0686E6B0074 for ; Fri, 31 Mar 2023 12:10:43 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BA021120DC7 for ; Fri, 31 Mar 2023 16:10:42 +0000 (UTC) X-FDA: 80629681524.28.5F3FC0B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id D9509C0016 for ; Fri, 31 Mar 2023 16:10:40 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hvuJVlhU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M5DaLi1tovZrH11oTI95mtlHRv/XZcPSFvwL/aUxUh4=; b=zJfZo4GcXhaHfXxcYdzuJuCHdFe6C9nu83UbfjTIE3WzJ3A7ZpO7JN7DvYs8zxgOt6C+vF 3B3Np/LFjmZeSdP5n9AJ0FwUPW88bV2eEyiOrJU4crmBZLMo9jCTlQIhSVFNTmIUfWw4Ka dsrmqcY+OeKlf10NhjlrkznU0WyJ0xs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hvuJVlhU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279040; a=rsa-sha256; cv=none; b=lEexSgYtUKVEYVEKtP0CUgB1mdO3eLbiFO2nAVHkXYMoB4XDUC0M6Jhm9Mf+y9TNFYoVjQ 5j5fQU5oQ6dCChr03JQ83PdGeypc77UnPAxgkqTs4ug/Gc+Aekfn4Qr8ja6dhRmIWFGf95 5PAy7TUBqjDgqzVJ6PM7a+BKO3LJ1oE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M5DaLi1tovZrH11oTI95mtlHRv/XZcPSFvwL/aUxUh4=; b=hvuJVlhUbWk56eDcODE5dwNiIYjJlheVSuisaPadN0mshUCuer2jQJLJatzv0cy+pDtDYM 2Vh2o5aVnqXZNZCp4s0y2MtNUA4o/HGR4uOjzikqgsqbQtISkJKDNIxX+Xgb4RRS2iuas9 Qm6FpzNUjMX3LjOgqWzvYmS67yzI0dE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-185-bYCsTSCqNby-9SBEzU5ngA-1; Fri, 31 Mar 2023 12:10:36 -0400 X-MC-Unique: bYCsTSCqNby-9SBEzU5ngA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 986783803502; Fri, 31 Mar 2023 16:10:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 816BF4020C82; Fri, 31 Mar 2023 16:10:31 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 26/55] crypto: af_alg/hash: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:45 +0100 Message-Id: <20230331160914.1608208-27-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: D9509C0016 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 9qozw5imb1xu65p6pfjyw985uursyjgi X-HE-Tag: 1680279040-989243 X-HE-Meta: U2FsdGVkX1+90t0Bo/+reGBVrJktKGSFSqAcEqN7SxbHxtTXmunS4ZSp9QPn5LQXfUhYzDnTLHDcAKBbrS2nP166he+5Dcm6wjJVd9bwgTRV4mQXS17lQOwT9ak5YgxRbRM3KX3xjHgrhMmHJ1Ldr6pXeyadfrHtoffy0KebCArFWv9p1fgHcLI8bzjl3pkClFcpjWtjNUBtgTvgA7j+pjomyQrZSMeBiALOughk0Ks3XrmgdMT/O+0NoRTqIpOZ3ea0LCCBM+g2mE+s/d7FTpUgZe6irzIygIrb2UsJfL3DMDB0/1C+sklLBF2tODeAk+/JZTCidXqW2mdWErI+61Rj+bi2v0Im/5C6syG15unN16ha3Q3tKn7t6VxSugoZkqKwPBkYqoNLJ/5sgsLwuPnzX812DSy4N+/XZrVd55E1hhv+dvA9GUWaXurA3yt4I3FAZP3ox/X5d5DvmPTIWqBtLpEwkHVZzFUSKu3zMAP5fRkeMls3mQymK0rUSbfF3GZuEuJfQdwmucTxCBxseI/Lk4+GPhYhxqHJXRjqNbvUjpPYKFMKvPLgz95wm1p+Ma4KVuGqJgeoRLwcJdsPKBXKnpUqejo5+KrJuehW0TEYyASOdurJC9pIX7h4l67VssvENKqASWSplONycUmk2agploYZN4vGG/eFwNg7Uco1CPGYGs/6ieWQNq4TukvFDiYluwd+ORbxDgOk3g4uMblK56rgaSSA2BD3x20Ok3jNXHFVt0ehW3gqlesTnvOBt13TtBPf9VcHOEGXekGNxkjuhL39fIK+AN1jarrlA/70MbfEIlFe2bdk3zt5jhJofQjOnIyqpK+T7TK26IGqUN5ahf2Pc2XAbsoPHNna8IkHXVtV6bOOFUy4hbC1q/krlnmIAkPEsQWYyemqLHzpknXeWR4OisX+IC5sd+CNDobpqUizYLmANqtZifBAbkQyQPrbZVykzhbe5X87evV XufLnj3Y Tk7Ezc7MDxwh3vGwioXErcsf3jdbVK/4m7HY500Qoqq1jbykqPbv2DZCH1L+gNHSr8FQVLkhJmnYGDZeAUMfKkDh1k+vjlZDLCDVkWqRUVPHFSGGT+8b/ht8rg8ZvKaCZnUi1wFNY8JxYuvyxwcVdFtxNiwdqvnecofoxYTXSBGxiIlQdoxh75sa67I0y0dxtNEIxAKE93B55fqmwZ84uJRogc6tP3JKX08UlHLTPToTOa+BFeyPs2R+5NXdfDz5VOBR33bYKdjCaZjOrTfN21obqWf6PQvvWP40oOtW1akTXcfHGStxLRTxKoSsPRfRu+i07n3QTvJ5oW22Yft5C8VyaUquMyMdFsinvzeJxuzKyAmpQqVVtj4gxkvR9rv+2ZSn3t+E6KWOTFTOCHzvyRcvNKiDuYH+0W6bJ/vFZTNySXeWunpqnd97mPS0jzIb+3JJGqDJvkpORbm69WBfVn5dlbSHUe2YRCIPsTUJ8krHdIVREIsWiackYnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_ALG sendmsg() support MSG_SPLICE_PAGES in the hashing code. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 11 +++-- crypto/algif_hash.c | 99 ++++++++++++++++++++++++++++----------------- 2 files changed, 70 insertions(+), 40 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 7fe8c8db6bb5..686610a4986f 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -543,9 +543,14 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) { int i; - if (sgl->need_unpin) - for (i = 0; i < sgl->sgt.nents; i++) - unpin_user_page(sg_page(&sgl->sgt.sgl[i])); + if (sgl->sgt.sgl) { + if (sgl->need_unpin) + for (i = 0; i < sgl->sgt.nents; i++) + unpin_user_page(sg_page(&sgl->sgt.sgl[i])); + if (sgl->sgt.sgl != sgl->sgl) + kvfree(sgl->sgt.sgl); + sgl->sgt.sgl = NULL; + } } EXPORT_SYMBOL_GPL(af_alg_free_sg); diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index f051fa624bd7..b89c2c50cecc 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -64,77 +64,102 @@ static void hash_free_result(struct sock *sk, struct hash_ctx *ctx) static int hash_sendmsg(struct socket *sock, struct msghdr *msg, size_t ignored) { - int limit = ALG_MAX_PAGES * PAGE_SIZE; struct sock *sk = sock->sk; struct alg_sock *ask = alg_sk(sk); struct hash_ctx *ctx = ask->private; - long copied = 0; + ssize_t copied = 0; + size_t len, max_pages = ALG_MAX_PAGES, npages; + bool continuing = ctx->more, need_init = false; int err; - if (limit > sk->sk_sndbuf) - limit = sk->sk_sndbuf; + /* Don't limit to ALG_MAX_PAGES if the pages are all already pinned. */ + if (!user_backed_iter(&msg->msg_iter)) + max_pages = INT_MAX; + else + max_pages = min_t(size_t, max_pages, + DIV_ROUND_UP(sk->sk_sndbuf, PAGE_SIZE)); lock_sock(sk); - if (!ctx->more) { + if (!continuing) { if ((msg->msg_flags & MSG_MORE)) hash_free_result(sk, ctx); - - err = crypto_wait_req(crypto_ahash_init(&ctx->req), &ctx->wait); - if (err) - goto unlock; + need_init = true; } ctx->more = false; while (msg_data_left(msg)) { - int len = msg_data_left(msg); - - if (len > limit) - len = limit; - ctx->sgl.sgt.sgl = ctx->sgl.sgl; ctx->sgl.sgt.nents = 0; ctx->sgl.sgt.orig_nents = 0; - len = netfs_extract_iter_to_sg(&msg->msg_iter, len, - &ctx->sgl.sgt, ALG_MAX_PAGES, 0); - if (len < 0) { - err = copied ? 0 : len; - goto unlock; + err = -EIO; + npages = iov_iter_npages(&msg->msg_iter, max_pages); + if (npages == 0) + goto unlock_free; + + if (npages > ARRAY_SIZE(ctx->sgl.sgl)) { + err = -ENOMEM; + ctx->sgl.sgt.sgl = + kvmalloc(array_size(npages, sizeof(*ctx->sgl.sgt.sgl)), + GFP_KERNEL); + if (!ctx->sgl.sgt.sgl) + goto unlock_free; } + sg_init_table(ctx->sgl.sgl, npages); ctx->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); - ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, NULL, len); + err = netfs_extract_iter_to_sg(&msg->msg_iter, LONG_MAX, + &ctx->sgl.sgt, npages, 0); + if (err < 0) + goto unlock_free; + len = err; + sg_mark_end(ctx->sgl.sgt.sgl + ctx->sgl.sgt.nents - 1); - err = crypto_wait_req(crypto_ahash_update(&ctx->req), - &ctx->wait); - af_alg_free_sg(&ctx->sgl); - if (err) { - iov_iter_revert(&msg->msg_iter, len); - goto unlock; + if (!msg_data_left(msg)) { + err = hash_alloc_result(sk, ctx); + if (err) + goto unlock_free; } - copied += len; - } + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, ctx->result, len); - err = 0; + if (!msg_data_left(msg) && !continuing && !(msg->msg_flags & MSG_MORE)) { + err = crypto_ahash_digest(&ctx->req); + } else { + if (need_init) { + err = crypto_wait_req(crypto_ahash_init(&ctx->req), + &ctx->wait); + if (err) + goto unlock_free; + need_init = false; + } + + if (msg_data_left(msg) || (msg->msg_flags & MSG_MORE)) + err = crypto_ahash_update(&ctx->req); + else + err = crypto_ahash_finup(&ctx->req); + continuing = true; + } - ctx->more = msg->msg_flags & MSG_MORE; - if (!ctx->more) { - err = hash_alloc_result(sk, ctx); + err = crypto_wait_req(err, &ctx->wait); if (err) - goto unlock; + goto unlock_free; - ahash_request_set_crypt(&ctx->req, NULL, ctx->result, 0); - err = crypto_wait_req(crypto_ahash_final(&ctx->req), - &ctx->wait); + copied += len; + af_alg_free_sg(&ctx->sgl); } + ctx->more = msg->msg_flags & MSG_MORE; + err = 0; unlock: release_sock(sk); + return copied ?: err; - return err ?: copied; +unlock_free: + af_alg_free_sg(&ctx->sgl); + goto unlock; } static ssize_t hash_sendpage(struct socket *sock, struct page *page, From patchwork Fri Mar 31 16:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA62EC761A6 for ; Fri, 31 Mar 2023 16:10:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68B366B0075; Fri, 31 Mar 2023 12:10:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63CDF6B0078; Fri, 31 Mar 2023 12:10:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 503286B009D; Fri, 31 Mar 2023 12:10:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3CF4C6B0075 for ; Fri, 31 Mar 2023 12:10:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 02B9E8046F for ; Fri, 31 Mar 2023 16:10:46 +0000 (UTC) X-FDA: 80629681692.06.9A0D761 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 33852C002C for ; Fri, 31 Mar 2023 16:10:44 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SI28YrOD; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zbMUDKrdVAVwS+N9r9mW+fACtMcx+N9uUpcmjLnA2dk=; b=LBDqFcaHQoJv2uWovU/tLU5P+HjvxlW6h/8kBxiIL7MIWEWdKzUBrNTZX0L2X0hgeJSL8o bfbfFgMFOyVGaYwX+9a/I0JVzWmP0aMfwvlT6Ej/q+f3PshPxM45UHAr5uhm3lzuBDeaVs cYJEql8tuNB/jOUZB2KrhHhnO5OHt2M= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SI28YrOD; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279044; a=rsa-sha256; cv=none; b=CORVQ0n/8443ik4EORcQWAoOmAG6arhPsJ+WsYv6XoJL3Vvtkf/pV8z+7qEJboddjwF23K CHwkP7LOIvFUlf/diW1FQbLMjDjUdx2n4ULZdKEmA/zRkrverDQLPJ7FLgl5m0QxHyQ6Yf RZ4BghhxuM2YC+N/cCqFya3kgAAsy8Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zbMUDKrdVAVwS+N9r9mW+fACtMcx+N9uUpcmjLnA2dk=; b=SI28YrODEiCWIcM4TaSEbIiCVwEvYeCLTDu2GVeVlGgAtSViPPl32b9jWD+1/p+TlggcfK hhMPyQXP2OvYUacuqeaW473crux4aftEUpWeYKUy92VQBhX/ZSk4cCYldHbqexBTX1Ztdt KbIvEvgW4zQRdIln6Clrn5FAGXRD8Eo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-398-2LwuMPeSO5ywsd8zwlaphQ-1; Fri, 31 Mar 2023 12:10:38 -0400 X-MC-Unique: 2LwuMPeSO5ywsd8zwlaphQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48CE73802B94; Fri, 31 Mar 2023 16:10:36 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 35BAC492C3E; Fri, 31 Mar 2023 16:10:34 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 27/55] tls/device: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:46 +0100 Message-Id: <20230331160914.1608208-28-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 33852C002C X-Rspam-User: X-Stat-Signature: tkxu76iarh71htrzn76ykk5a9g1r1mbn X-HE-Tag: 1680279044-450106 X-HE-Meta: U2FsdGVkX198tUdtEjQsDJQ0/6qhhKG0lIP/ud6N5rrRnRowJnzOraIMKnl4J9osGAFEJBAn9UqQpB4wjFCMikWTzM76pRMwmoJkj4qeqCzHNVd/jQXNqCvz7xOZf0kbraflGQW/UhrRdTlxq1do0dxbX4VMlilHW/zYrPrywvZ1rTkrIxdiFEyA3lpd4gyJIEXZFZ7vxKlSJjdYMjbqFU3LPQFvwzyxcGxByUEGjBuTXYk+2kTqciFxeFCdzogfDld5PGJ25rHD82N1wu0EcP597010VfbFfsCbbZsVWz1pN9CFoIkvZkXo9cG7WVs9TiY6GhjtZ+PmBpMn2IUQWzB5YED90CIw8KnD8eUR3qA6hztJMYY0HDRakThjr7NQmOnorLyLuBpm8AQC473Iqo2bOlw9zZ0sn5WDQg5IY52qt8uuR8bfqtocBLSQbOMVqe/Z4SV40mTeLJBywYmrE4hFPJOckWpmZ1jzm+Y5FLs1ktiB8T+DL48+IGrZ1mfbGNcnKlyo1Ttft62yDqwZGXb40MIDq6xRn/dJ3B+D3OvV6GM04ukehleEVVI44ceEhXGIS3So5tuWTKPwTPiYBRS85c2WB3vTE+XIZ2y5lmCcz9WJhNnzpNTQsn8AR52NZFm5fcHKZMS4U0a1ROCZhla0THGaSXEt2Wzjv/PVHoOWaSC7LGmoCvekiu5a2eR7aOkc618evN3P8tGpsrG447DM9nb8P1/du+7Tdp8NhFJxbSZ70EbVVYGeMDy8duhgE9HQ6HDbCg+fsWbAB5Js8oyeXVzXMExbXlTQrOpU4vtaqgQNFDy0HQKwRuGv5y5bBLS0eTnR22B0TmtAqG6CcSr6QlAEhtUPWydEGpIpEaycU/P8iObihIqbFRINnl4UD/TDHpc0dgrSHNqstMHE5uH7rgAruWZPGdIYFpqvija1PhsFUI+LdvSM7tCQT14RutPlUd3oPJtL1jN2ckL /Eu0SvcQ +I+mCPdDY6COP0NnQAK3bIFkqdFrlrnF1hKdws/AbUSUfr037Z3E4tNgXryF7zmGX8GSAbQBhtiGhW5fBUJXCZUoo7gq9EQ/aKZjjq7z9nom6y8nF+FXatrJm4zUAnWgwzghBklmrMWPfl3AjJHpNICqnDv1RaA+dNQIP3YgHuJUdTM+/UuxqQWVoycZKj8UYssituVqzeuXorPY+yslIAPK9SaWIbmO1120Ze+cIoX5pyUyvBnhsnTcAWsAPQmQYo21XeWbtGWjN9jhpEk4m3EshhyXS+L29/uT5u+7Bh9Kso1ZVV+2GJLjli9m4mcSvJVeCVQTiEZxN6NQPr99/RdnH9P5r4Jp8q2Lwgl9PVoSkl+arSFPsuMQ/IQvCchrLFsdnpdzPPwFYAW9+lzYNyDasElFIsz6mavvasyZws4ugzLqOfIiQOPvzB8pMpvkQibfJNz5Pju2/Dmm/bx+nCmWp8BZXC9374/3XcAy87bOFilWa5eo46JpW00I5gmrEy/c787cYrYJEs623BS0qlB6dQ8jhG1EL7vPdjfNfWiBbwfkG5Go6e3E5pUXzaYxu3VfAX/bXR3WhqSM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TLS's device sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible and copied the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 6c593788dc25..f5c3b56ac1ce 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -508,7 +508,30 @@ static int tls_push_data(struct sock *sk, zc_pfrag.offset = iter_offset.offset; zc_pfrag.size = copy; tls_append_frag(record, &zc_pfrag, copy); + } else if (copy && (flags & MSG_SPLICE_PAGES)) { + struct page_frag zc_pfrag; + struct page **pages = &zc_pfrag.page; + size_t off; + + rc = iov_iter_extract_pages(iter_offset.msg_iter, &pages, + copy, 1, 0, &off); + if (rc <= 0) { + if (rc == 0) + rc = -EIO; + goto handle_error; + } + copy = rc; + + if (!sendpage_ok(zc_pfrag.page)) { + iov_iter_revert(iter_offset.msg_iter, copy); + goto no_zcopy_this_page; + } + + zc_pfrag.offset = off; + zc_pfrag.size = copy; + tls_append_frag(record, &zc_pfrag, copy); } else if (copy) { +no_zcopy_this_page: copy = min_t(size_t, copy, pfrag->size - pfrag->offset); rc = tls_device_copy_data(page_address(pfrag->page) + @@ -571,6 +594,9 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) union tls_iter_offset iter; int rc; + if (!tls_ctx->zerocopy_sendfile) + msg->msg_flags &= ~MSG_SPLICE_PAGES; + mutex_lock(&tls_ctx->tx_lock); lock_sock(sk); From patchwork Fri Mar 31 16:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46003C76196 for ; Fri, 31 Mar 2023 16:10:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F3C86B0078; Fri, 31 Mar 2023 12:10:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A4026B009D; Fri, 31 Mar 2023 12:10:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F61B6B00A4; Fri, 31 Mar 2023 12:10:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 106726B0078 for ; Fri, 31 Mar 2023 12:10:47 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C88B3160487 for ; Fri, 31 Mar 2023 16:10:46 +0000 (UTC) X-FDA: 80629681692.26.A401449 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 0192A4000C for ; Fri, 31 Mar 2023 16:10:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fH7BfXpL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nNsV/6c3ho01ZHWXYC82bpiErYX0eKAApSJsSVNmIO8=; b=px1hhQoAJBEcL8zqgJpGLC/YRsqyj3Hgwg3DYTgwKj02zrAJNd41oCyOA/vmtpi5n4YOIC GRutWwc35TsUQP1FfoWjWw8O8BXn1YaoEtMW56gTwy0kXW8K7LAFkgLtvDCDwjA/YOBj2i JG3f85kyEUrMK6ZTOmGczCuiIA7YYHU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fH7BfXpL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279044; a=rsa-sha256; cv=none; b=J7lJbBtXILdTHtX4SJ+JeDq6S2mAp8IVK90Z6iwgfBmSlbgj95CCa/N8XWLP8fcsl7+mQT ydyT3uDfLuBzoJjzLwksoEc/Vm31vqcCVLfoTmPzrrz3IbIfx9z9LVVJt85n8J0nmVkhqw Gn0pafNSWOlNrYmXpcqgx69fVQ/BMys= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nNsV/6c3ho01ZHWXYC82bpiErYX0eKAApSJsSVNmIO8=; b=fH7BfXpL7FgjF6WuxfsOqJh28brPsEvbH3YQfovWT6RMA4fVU33KXXhpO4FBzu7Uj9dPnh IwIflFhhDKaZA3x3ltkfvpbI4hZDL9Mz5JK2Ln2/zycRS9+9CvYep10NWniieclolmDOSF KjDrmntJcuiNDsTCgNbG4/TXao7FyEE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-138-CQ22tLM2M2iPd2GX0NVahQ-1; Fri, 31 Mar 2023 12:10:40 -0400 X-MC-Unique: CQ22tLM2M2iPd2GX0NVahQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 19982101A54F; Fri, 31 Mar 2023 16:10:39 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 08B612166B33; Fri, 31 Mar 2023 16:10:36 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 28/55] tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:47 +0100 Message-Id: <20230331160914.1608208-29-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0192A4000C X-Stat-Signature: t5d3m1kfyejgb15zzzo4ntwgcu58esma X-HE-Tag: 1680279043-629575 X-HE-Meta: U2FsdGVkX1+F+AQkKG+8gyZuDS1oPH2Nvsdv3bgLWBwMipkTwAWwvN35/KcxIl6nF2DQc/7YVWU9okQiW06b9hv93S+JOJM85EPIOMGs201qsIzw62ExR/TwDxFPqUp5JNUi66Z2tjOU6uvyIQBjalIUDQjxK9v+95NyXzpHm7EyaAL6UvLHMe1xa413sQYCp04d4fKqsBLjGmIjXfjI3wKjzjRMjWALRAIR7g7s7OYp5DTbwFdtJLZuUHtQBtJs/TqGZNi8RNneqo0BbpwANjDbNd0ooSpb4t/igWo5RkPA6rt/QsmGX+GlU0ftSISWH/KTd+ElLTXJJyVH+rz+330jWDx3qWqsFIHxRpFYr1fNHmnPIo5lb2co8yrywHDsgU4IP/8gaOPx0jMTmeW6a335Sb2nQ/o6BtjTygX9jrwEhqgfvA8No3zs3/HZ6ACu4MncQV9Uplc/dnYQHLHxoezIxBzP9LwMyr8+SXMvGG3QpNVwIGg3t8GY4zVT0KeGwaM8jcfy6ERAA1aypdSMlCYJDlv4KGJkIroPITHzlod+IPHMW+rpF5H66jaekJyQdc8gaQBAQMkbKoqqRHuBWNLbANX8D4qnUf1MU/7fetrIOPfwr8Fg7Gs01XVICt1UJvR6MXMGgIjyi4W+FEvctGcowCnF5OabfQ6zu70W3tbdPxm4E3dG9Ie3jSl9dH776vJiVPrH9x8FD1N03J6tZWiUf4RnFTuHmQ5t6dHI6aVT8txGcouwC5rJZJPc1C8+Sv8p+bqZZWcUY0RfM01fhkQVD5W2YQeNW4hy9lVGdZLcEDv7wNBYDpejo91LbdZrKLrDqR+/OEZnDH+tGePfHvUPqfzflwfEp0IpYmBPJHlaQ6wZZatvFsSU/B0Zc/hDBbL7X4fyH+c/7vOIPvjCZVcV9VZqA4PwgpqViPxhjWnbErplwRQTOp5QVh56roXALC1jLp3yjvFZyje7wex ROrQe4WX RFcZgJdxKPIS8aonAzxO1TgBkMO8cTYhWqZQzOiizm2xbqik38C4ZgAP+ouZe5Iq2FzwyPdkMjD8ZTEkMbJLyIJ5vbS4OzJw4aJJLMqhHRYe7KBuorAYLhJUMGeYhyqWk/5Tdo1NosRWW8YLbhy39KGuoutpXFssvsbOSymPvp1/1Zz7AreR2ZbtDgAZuAD2ixD8qgqv+J21tbLm7IjeqT37DxXjGjxwkFkwEAllR2OippyioQBgD8bkJ0Vl8c7n4pJu//q6L+m5nAL+cpUpnCf7mfvPnpOQtQOJZiX3JJAVUy8vU8u/YyRykjjf4lphyVxpsvGd8ZQPt/8200tEQN/sVR+W3rDNmCsyRlxgM2UrrDrJDY6ZedDFrVEkfhq196RRiKjyon+lRMP+gBDVNF8HEwh+agAa8FYrnPeR9oYuHscqfGHu7W/L/wm/dDjhv8qnpc4wAu1ttFCB24mYQcPbtLepqE1At2LDVHNvu4LWt8XFFVAl7ypGy6JoLkXLBQ8fvRyKMsWjPZ0DVuo8/kPYQx4TL7omHx2c0xv6Iqtcz1AryOZBEiFkkxxlZP3zCjwygngkSnrTV4MI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_device_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. With that, the tls_iter_offset union is no longer necessary and can be replaced with an iov_iter pointer and the zc_page argument to tls_push_data() can also be removed. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 79 ++++++++++---------------------------------- 1 file changed, 18 insertions(+), 61 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index f5c3b56ac1ce..6cfd1577a212 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -424,16 +424,10 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i) return 0; } -union tls_iter_offset { - struct iov_iter *msg_iter; - int offset; -}; - static int tls_push_data(struct sock *sk, - union tls_iter_offset iter_offset, + struct iov_iter *iter, size_t size, int flags, - unsigned char record_type, - struct page *zc_page) + unsigned char record_type) { struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_prot_info *prot = &tls_ctx->prot_info; @@ -501,19 +495,12 @@ static int tls_push_data(struct sock *sk, record = ctx->open_record; copy = min_t(size_t, size, max_open_record_len - record->len); - if (copy && zc_page) { - struct page_frag zc_pfrag; - - zc_pfrag.page = zc_page; - zc_pfrag.offset = iter_offset.offset; - zc_pfrag.size = copy; - tls_append_frag(record, &zc_pfrag, copy); - } else if (copy && (flags & MSG_SPLICE_PAGES)) { + if (copy && (flags & MSG_SPLICE_PAGES)) { struct page_frag zc_pfrag; struct page **pages = &zc_pfrag.page; size_t off; - rc = iov_iter_extract_pages(iter_offset.msg_iter, &pages, + rc = iov_iter_extract_pages(iter, &pages, copy, 1, 0, &off); if (rc <= 0) { if (rc == 0) @@ -523,7 +510,7 @@ static int tls_push_data(struct sock *sk, copy = rc; if (!sendpage_ok(zc_pfrag.page)) { - iov_iter_revert(iter_offset.msg_iter, copy); + iov_iter_revert(iter, copy); goto no_zcopy_this_page; } @@ -536,7 +523,7 @@ static int tls_push_data(struct sock *sk, rc = tls_device_copy_data(page_address(pfrag->page) + pfrag->offset, copy, - iter_offset.msg_iter); + iter); if (rc) goto handle_error; tls_append_frag(record, pfrag, copy); @@ -591,7 +578,6 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { unsigned char record_type = TLS_RECORD_TYPE_DATA; struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter; int rc; if (!tls_ctx->zerocopy_sendfile) @@ -606,8 +592,7 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) goto out; } - iter.msg_iter = &msg->msg_iter; - rc = tls_push_data(sk, iter, size, msg->msg_flags, record_type, NULL); + rc = tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags, record_type); out: release_sock(sk); @@ -618,44 +603,18 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter_offset; - struct iov_iter msg_iter; - char *kaddr; - struct kvec iov; - int rc; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (flags & MSG_OOB) { - rc = -EOPNOTSUPP; - goto out; - } - - if (tls_ctx->zerocopy_sendfile) { - iter_offset.offset = offset; - rc = tls_push_data(sk, iter_offset, size, - flags, TLS_RECORD_TYPE_DATA, page); - goto out; - } - - kaddr = kmap(page); - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg_iter, ITER_SOURCE, &iov, 1, size); - iter_offset.msg_iter = &msg_iter; - rc = tls_push_data(sk, iter_offset, size, flags, TLS_RECORD_TYPE_DATA, - NULL); - kunmap(page); + if (flags & MSG_OOB) + return -EOPNOTSUPP; -out: - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return rc; + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_device_sendmsg(sk, &msg, size); } struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, @@ -720,12 +679,10 @@ EXPORT_SYMBOL(tls_get_record); static int tls_device_push_pending_record(struct sock *sk, int flags) { - union tls_iter_offset iter; - struct iov_iter msg_iter; + struct iov_iter iter; - iov_iter_kvec(&msg_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter = &msg_iter; - return tls_push_data(sk, iter, 0, flags, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0); + return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA); } void tls_device_write_space(struct sock *sk, struct tls_context *ctx) From patchwork Fri Mar 31 16:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ADE3C761A6 for ; Fri, 31 Mar 2023 16:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F4C46B009D; Fri, 31 Mar 2023 12:10:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CC096B00A4; Fri, 31 Mar 2023 12:10:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CF626B00A5; Fri, 31 Mar 2023 12:10:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 078066B009D for ; Fri, 31 Mar 2023 12:10:49 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9DD1340458 for ; Fri, 31 Mar 2023 16:10:48 +0000 (UTC) X-FDA: 80629681776.30.595F643 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id A9A7C140016 for ; Fri, 31 Mar 2023 16:10:46 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ioa0UtKv; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279046; a=rsa-sha256; cv=none; b=4SYaZWJRdMTkGpzB3La+dwWdzVjVkdDJBthb8Th2gEQR5EsPhpSNgFrZ06zVwS/rTXT0kb dytwsz6UyVrYIAhAkguEHYS2ZPrt4ZJZqGfgHoUfl7XtjHa943U+/CSnwYVbew+ds2/URF a+YbmRsDgocQyokL2jjA7hwhZ2IyNCs= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ioa0UtKv; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NZJQb7iAop4LtoT19/C7V+Vk0vTB0ScTwxSGzzmAlmQ=; b=jhr/GnQTytKxnLvDBbXEPmSLbugFeF8g7iHneuHLbaZyrczq/niN9PsUo0LH4ag8WA5LHF D1983muugoj+tR0nRtxIp0QLjrUlHfWq9HiH66f35R2UJa3pQGmTKP+G0NypOYBTuedoz0 3oIXd9mvzaA3gHNZvD2xtzCYBa0rcPg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NZJQb7iAop4LtoT19/C7V+Vk0vTB0ScTwxSGzzmAlmQ=; b=ioa0UtKvTmua75QyXN1X77zHc/tVQin/xomX51fJSHILmG/kp8L5gX19JPo9JSI5dYfA6A tdOgAxaIypn6WI3NGYuzfQf2ZQB2R+8MdF4KE674hUz/l0okbxzsEl7ppGd66woMvu4KbB 4dqUfwx5WakHvgGYSfKPASNkHorQHCI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-380-s0-9Da1IPNaVGNWY3Pu9-w-1; Fri, 31 Mar 2023 12:10:42 -0400 X-MC-Unique: s0-9Da1IPNaVGNWY3Pu9-w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B8BF029ABA1D; Fri, 31 Mar 2023 16:10:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7E464020C82; Fri, 31 Mar 2023 16:10:39 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 29/55] tls/sw: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:48 +0100 Message-Id: <20230331160914.1608208-30-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Queue-Id: A9A7C140016 X-Rspamd-Server: rspam01 X-Stat-Signature: 5k1i4af8hixzzbw4akp34p86k83tgbbn X-HE-Tag: 1680279046-667356 X-HE-Meta: U2FsdGVkX19pEUzih3UTlVmIPUtb3XtBuo/Jo45pmnybbBXB6sX94MUEl1FHhqP5VD4okBHhU/TnsQbnh3jV9V84ePh+VMFjWlFRilhVuX4jDdi7V1zfnUgFKpwywmFfHdrPwXhu4oq5eOvGwDWc4j7DM4lOFYh062F1UXeEYwGlvJw9VXjCJlfQCWZMdSby3I3VeK46uffHuWEOBxkuZiN8M5+6DHn5dhzwzV15Mn6vN6+8PXRVN0uTA4nGsxzyoiCm83rlEeF8XYM4Q+dL8faKnlcVHz0+aRLyipGSaM/vc9b4vEG1XrrmK/icBCSfMNi5sFtVocYqvbUEkl8EW2l4nyFIG+5T6b11GX9NhmuDdIO/S+B+OH8mSnYxS3SXcN7i6vQi8AhMksSpkTupR8HTm4Sz0zUKs7FgaK2yROEbBGlkcRcuBZqbV32MV8nHq2XT6bFRrEY8kRqZ89dNfrl3tbyUPvxd2g73AzAuUUkU7g0CYVY1kdnuGxY8k5EslntrUqiAB8+8dooqqRk3fimklZf28rYwJ+g8i/9cm1EyTkAAg+/9OoGdpgU1pg0gv1WSNSJTNMBeNrIzd3e3jDsmwjSdIi+h5CnbzANBiFCOfSJgzcwWkrjLVeu86WoAVOTdSIU62sZM9muIR8lGeNy6XbepNVx1rFa0lMi2wPZPDpFQq7QS70pZKdxNDRni3EXzQEgLt+kftNMIp3h/dGS+14V0zw+9N5w3u5GhxAvSiOplj2rGIhbbvuiLsD2g80qMJAm1D/qmwg/JKPEF+ZU6v30/3o/WKzyOy3nEOhtw0ZjEB3kvNNDqg11xLlsk6uA9c+M+5mUx51e27pAbshi+UAOLoN4/DQNY/2Zx7/vd0JJ7AEBgvKBo2YDV3JiaPpdoa3cPU/P5+Zw4XHTr36zywnEz1tiMsDM43nNCbKBxo0ATVlbrf8k4zgbZhJwp75geLF6bmc7ZbY7I+EP OfP6m21X dNXg/9QFcAnKixALUOcHpf9tArtQFfEh0+30Tw7pQFPog0kZDs3KWKRzCMmLn9zLeZ67s9HaOYf/ciX23Ub/Ap0aETI1pWENSv+kGeeDZTQBllidUyLqpnV6uhsqdYMxrwguAZyWmvowPyrIkuI/mERAN6DLjdqxRqByxnAA8+wXbxf3lE6n5P6PQ4TN3dY+y2OaLugUBbhr4Ky7TG1uyFH/qS8zLtFdXiIVGBhrCxHk9sVrOfLsBYcRv06JKd8Un/IdFMCMeZMmdF80Q9Ca0IFctMlDWDWO64nLhipr/CZDWnf54K1KGf5XImhgPXB42geTVgLabCFuYp7IbBWMvvNT7X598Szf0LbWGa05Yy+TgM/I41Mfa0Azqk5Clq2MIXKowj/Fnp3x7dgwjKWfwhCuZjyQfniLcZRLLuuHY02lAZXMQLR54YnMG1Z2kjn3PO8JrGwAwtCjw+9oi9gNP+ElMYVOukChEunxvl6VoMsvH1jg33J+Xsv830b/dwxiBhx1Q4ZZhwI7TDZBjWdXhTiCaju/RIF1W+RNYShCRxIn0G1gm445R7+Z2hAjHIQFW+6QBzYRPpvOxmrs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TLS's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible and copied the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_sw.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 782d3701b86f..ce0c289e68ca 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -929,6 +929,49 @@ static int tls_sw_push_pending_record(struct sock *sk, int flags) &copied, flags); } +static int rls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, + struct sk_msg *msg_pl, size_t try_to_copy, + ssize_t *copied) +{ + struct page *page, **pages = &page; + + do { + ssize_t part; + size_t off; + bool put = false; + + part = iov_iter_extract_pages(&msg->msg_iter, &pages, + try_to_copy, 1, 0, &off); + if (part <= 0) + return part ?: -EIO; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, part); + return -ENOMEM; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + + sk_msg_page_add(msg_pl, page, part, off); + sk_mem_charge(sk, part); + if (put) + put_page(page); + *copied += part; + try_to_copy -= part; + } while (try_to_copy && !sk_msg_full(msg_pl)); + + return 0; +} + int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); @@ -1016,6 +1059,17 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) full_record = true; } + if (try_to_copy && (msg->msg_flags & MSG_SPLICE_PAGES)) { + ret = rls_sw_sendmsg_splice(sk, msg, msg_pl, + try_to_copy, &copied); + if (ret < 0) + goto send_end; + tls_ctx->pending_open_record_frags = true; + if (full_record || eor || sk_msg_full(msg_pl)) + goto copied; + continue; + } + if (!is_kvec && (full_record || eor) && !async_capable) { u32 first = msg_pl->sg.end; @@ -1078,8 +1132,9 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) /* Open records defined only if successfully copied, otherwise * we would trim the sg but not reset the open record frags. */ - tls_ctx->pending_open_record_frags = true; copied += try_to_copy; +copied: + tls_ctx->pending_open_record_frags = true; if (full_record || eor) { ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, record_type, &copied, From patchwork Fri Mar 31 16:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC74FC76196 for ; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83A1B6B00A4; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E98D6B00A5; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68BC96B00A6; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 56EC66B00A4 for ; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3A8A01404A7 for ; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) X-FDA: 80629681944.11.5219F57 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 6D5FB4000C for ; Fri, 31 Mar 2023 16:10:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MhzwnM/S"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279050; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qgrsaUecWqqjS48xBO3DK0ifte08BTz2ZXAzew8KrV0=; b=zP6Mol8mOQPttkHhZ4du6VugwCWuIqqV6FIWnN0jE84phJCo9gJ0oUHRv1xl7u5YSUh57U DzvsXDOdd6Ev4w8LZHR4ro6OYiNRatuGuqoFyUrV9uE5aXHSDSTikZOV+W6u0g5sTZr5cW qGenxox9bUNqCOs43kSK8I50F+ULixQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MhzwnM/S"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279050; a=rsa-sha256; cv=none; b=0YDDSSHeROKmhJ85Vv6rrcgzVAG5KIRXXVOJpXISUmYuAsHRQSIfL2ci2N0SMyGrPoTe2X GxcP9enRiDngsyzi2ugd31vtpcLFYIyV6AUPRAbAqRGsPhSVeb1706gJibBhojK8qiwccP C9y3VPcIKSVyGaEZLUzcdUM4Zi7peWM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qgrsaUecWqqjS48xBO3DK0ifte08BTz2ZXAzew8KrV0=; b=MhzwnM/Sf8/ug2PrOb4iRm104bG3aBUo0vHl704bsw85/uI7/upQfwioLQDxEEC+wxclCy f3fGW0lQN8XKpD/wZAG1axFHK21/QO6p175qmYhQdviYZ25b7SrSSz/FCiNyXSbaebs313 ZdK275BwMMNs6KT6vwZjE5QRwykRA2E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-146-u0t01DzwNZa0rrcf5yQq9A-1; Fri, 31 Mar 2023 12:10:46 -0400 X-MC-Unique: u0t01DzwNZa0rrcf5yQq9A-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 875E28028AD; Fri, 31 Mar 2023 16:10:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 745581415139; Fri, 31 Mar 2023 16:10:42 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 30/55] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:49 +0100 Message-Id: <20230331160914.1608208-31-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6D5FB4000C X-Stat-Signature: q8ho8hdwfwsjnpsowchcx7r85xbw6m6w X-HE-Tag: 1680279050-169094 X-HE-Meta: U2FsdGVkX1+EHLkpLGkPARURcBemLxmBSa+Y9nN0XYMUHi4C0jLAVB7cM8blq/i5zvmidUR01Qz/dH4Yztg0GHN1Lg7kcIBemX2XOQEPqKSND1X+hvhTDu4nPyIBXsxxH4RPRbFTTbwVstIMwgTX3DK2lN7AiGXgDSHXExuu3esjyHzjqXjpizDwu6IUU0d1XnqN+Vz+Msv2qgwp4OcJaX2QO3716QUBq39WO/eIO6MiH48EVLH/FdYfHqSd9XQlcEUTQsaPmCdj3cel201Mu06RhrjxBcgXoW23OlCtMvJSB96w8e8Mr2ZvketmI/lpBxsq4dAIUgmy/0Uz3BvF9k0ivYoQ+7FdQm3a+8JKOdpf8Sau2EQd5bPmlknuEHBaQcVun1xoCekOx0VMEq0pYMn/k/WEDqSI5AN3eLlXnqaNAkpOBbieHQEoKo578wBPffK5xdVNKzDQFouiARXbgcD70HuW21STkhmT5Lc/HkWvvSjK5Bsu02fiTKF3o+/F9+782/rFCun8YJe3xLYhueOhdzjUJLmKR8AFNr2vJEBapbgT0UKkXcvjRZIdF81y7PgGNXuTSuKMTIDg6XINYL+4zqq317OQQPB5pN/35wx9m6TvFPKllVmZDXDuXqDWAiJYPaRlWz4mcgrorNeT+7hy1UJkcvI6rxuRAxMtNDhgRErFnDhPfbebm9V/3t9DULzh6QJSprBplr2+DVzzEmB69v5jzkdmUm2/j4iO3hWIsSHk32GIAXBwdDzT5gSV4LHSTs9czFHMFlaCtKXA16pXYolqP6/Uh3e0sxx3k0xAG2h6Ldoag9WVJPpCsG3CdRAHiFDqmGLqVQGVTzHIZPmsNRYPNLGt8gKiDAQiI7L/ApyAEHQixOS0XPhstfHy6g+efFTtBt1yb7H7/BvPNKr+bh7Xwm3yszILVM9P933yeKc6sf6DQ7zfcNe7dcOpVRn38n9ez08qN46rafB pfQ+aiIX y5UksR+9V34EBgD4kiI0Of3wr2Ns2FlkT/coX+k2lTNbQpej/0oeDb9aV4OzbjSnA3MgBQuvvuCGtbKg3t4One0tP2cLVlXO9ptwJ9/FuFfaGqCkUV0Ma/fE+ml93jqKF3T1gNkXYtWbhDY+HvF/Up3xBoyDtn0rdMzDrFMJhtsK+mUhE/dsUSF0/pR1n7VrGIyrsKKXc7laBTAI04w4bs1D/k0JJB9iBv2vkQs59uqsyLnJoO5uc2wjWZ7SLiruq8vYSnjNpBaBuaiY0ndxG2n4ymfAhEAzIU14uVDPxbWlG1puzsPoULaOrP0Y3w3Asf02ecQ1rkavtI/6csC4uFDm8p+IB8f7oN/WOJlj61YI8joliGZfFdqj5uqZI/kVEJfwR5N4nswZI/07Z9+dxWC6sjzVJiH4ZVeTBIwyyK8xh/U3MwnXWBocTsh8lqtsocHqyrJ8hb2IbkDd1+PcKBh/eYcmBjaE6VwPrCrrj90MTW3AEsj6SVNvcykq8lycZ0S4pHN+UaB/OSOvgfcppFVv89gnQMhUtfiGmt3tuIz5IeA1HILujIjGPwctz8LlhgEZL1RUxfXCL1T8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_sw.c | 158 +++++++++-------------------------------------- 1 file changed, 28 insertions(+), 130 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index ce0c289e68ca..256824fca651 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -972,7 +972,7 @@ static int rls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -995,13 +995,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1202,155 +1195,60 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + mutex_lock(&tls_ctx->tx_lock); + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int From patchwork Fri Mar 31 16:08:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A2DBC761A6 for ; Fri, 31 Mar 2023 16:10:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC86C6B0099; Fri, 31 Mar 2023 12:10:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E79026B00A2; Fri, 31 Mar 2023 12:10:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D40B36B00A5; Fri, 31 Mar 2023 12:10:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C55D16B0099 for ; Fri, 31 Mar 2023 12:10:56 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9A1741A0D0B for ; Fri, 31 Mar 2023 16:10:56 +0000 (UTC) X-FDA: 80629682112.18.EC8270F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 2E15540021 for ; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EsCK6a1W; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279053; a=rsa-sha256; cv=none; b=AN+nG/rx2WEj2qRn6o6cuf37iNl8Q6EigGlCa8aFFyXi1L8M8ORNq4jFRDJKnBmVZDednb yhQKXEMK2X/2mnFN5e0ppePlMagS6ENG4u5sQToxl/cJRmYOexfq/S0DhTF3cLJtLrrwv/ HzJ6Eyr6wuUYj23W8Nd6i/aousPAY8w= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EsCK6a1W; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wy1bl7/wKEZT2hBAqmE7Plq7HNXSqd6J/GoEc2iata8=; b=Iku/3A5NpfyH5d9L6v+QQRvy0KUoP7tzyuuumEIYGkNIgzH06AdC45BiF8ywIop5/oB/+p pMBwx8/VJDRPNcxSWD0zEGl9sK6Cw42kMmJCxUnmgpgW7sTrj0St2zAaLQfmoeOrxqnldD nkBkZcrdAWJc1s++ljJBVyWI9ZHPuus= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279052; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wy1bl7/wKEZT2hBAqmE7Plq7HNXSqd6J/GoEc2iata8=; b=EsCK6a1W+nKFEWwKCpVDu3Olg2rwtNfzxRWTBVUhccMbS461Z0aIdpDQxWHR27/5OGHwo8 hr/TznEJdbMc/DfqvJW6Ig4G+TC9/F506JNsmvgTAVSG3N074wqUbkYLKpa7VXlqlLcG+D XACZVCUIF/x9hTLw6NZiWfnyHho3b/4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-408-TWrdWqY9OgKCvKHcSyW2Nw-1; Fri, 31 Mar 2023 12:10:48 -0400 X-MC-Unique: TWrdWqY9OgKCvKHcSyW2Nw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E33329ABA17; Fri, 31 Mar 2023 16:10:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 223762166B33; Fri, 31 Mar 2023 16:10:45 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ayush Sawal Subject: [PATCH v3 31/55] chelsio: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:50 +0100 Message-Id: <20230331160914.1608208-32-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Queue-Id: 2E15540021 X-Rspamd-Server: rspam01 X-Stat-Signature: msyrxf9cjuyddoqqocuonao1n9ne7wid X-HE-Tag: 1680279052-319480 X-HE-Meta: U2FsdGVkX19GFIp9h8dTk4Onrud2lvQn/MYB8OdStZIvzvOVJzmXv4+VrGfanmY0GE1thaFU+4j5FmT5Ep8kUba0vmDIAFmxMyM04k/6yW9LEsv8iskO5wl7+WTpqQ15ssFhoZtm3rmMmfrO5s9l0ZfOWfuvg2G4V5diwCu/n6hZw2W+WGG0JidKAYFaKl7GiZe1U0B4O1xvzMLr3b431I1VW4P2MUaNtVt/7jAEHQvWq3TvHu7TphsftL9ZCOi+497YHk8nWQVkYiRUXQygAKcIxpeuR6eB6+EKOotOwrIhYeRR7XPvxLRL6I9V7VSHlJ93Ts0dbBmvAQu/L1gmaWgUxDPV2E6I3Gkqdf2HgQnfwMgyQYiMuhF/0b96YDg+YahF4S+fxajuyjxpg5XEZWBe/B4sOGK3UhtDqNSXYs7iQ2bnepSxImnRp4n+3a0hc9593DnRRHGJNJlISro0MEYM1OyqUE6QlscxGjRB4ulFk3Nj7sspN6D56EB9sRffPDtUv1ZQ6JefSJ83Ls+Q1ZraPCtxw+zYk20Ib2Driy/0yPHafKUX1gaeDrWI3kP9RKstXdcggRgGpeIz6WT7pCzT3jiFrIm4tC1LJopcbimh5bZniSBQbgekfILpvefmobRexudZfY4xUGOCgwelsNlaGpYu2/ukZxdB98n8B6kNpaXf3SyMmSOnEyOqsqQbeuJ5PQG5zEEA0LUiDchTce69C+g7JLIyNq7eSCofUKJtc4Y71OUEmDgydzk8n4kBWKE14B97kvIkyrSMBs/er4u694T2n5CSSm8NinYvhKUfD6Pz4N2Ksneme2spjSPvjEsqwws38D8Y1ONZyZRpRIxMQ10SZYHlzoAlNSBgL0oLKCd36/YHbLt/axDQvkd32MDd913t/6QuJubqqcu4phniOY+/GGt8+273MiH7a9k+3Xc0jmySPRhkMsCLweQ+BzaK4IjTnWmZ1VszaJC nktvmR87 Mq8NcUEjLHeqdBAvEzfRPu9Mx+UOLRRccjT/hVLuPtaXqO7a2GY92kz/jgF0wcHCQiCEzZuTweATscHDNinuvjQnkSEuZW3Ubcv+6skSuRELvuGpyK/48VxuwFzU4gZ5CknJ0ZtDTyJlJJN9rSOF+tmlB6IRLPOnqeS0kAXWuN5I6nysVLsaCd5x0xqkEH86hy8BGQgpCh/AaCe3s/wJ6eSUYrseDIn0X95x6jorswCMtKPyic7M8sXdLsW3AL6Zk4mdJGqjRQ2EPEetXp15sqXea61ICKdr8ccCiN+urkbQSDWpjohhzNixCcuoM9eIj08+7nPXVMoaVYGXHLXRCK+X80xFRA5C7XJ9RUSD6L94zBOZH80LNZQxR9tHZ98NdtVLTDDGT0+dVvKKG1nsvoL5JGdeD26ISaayu4/w5bp59Qw25q6xn8h1YUWdPBlI8gUsVxRwc6ws7x8rwFwxysdrYKvLsiNdlCyI3TIlKvoJL50WfjG3TQOfjog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make Chelsio's TLS offload sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Ayush Sawal cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- .../chelsio/inline_crypto/chtls/chtls_io.c | 60 ++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c index ae6b17b96bf1..ca3daf5df95c 100644 --- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c +++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c @@ -1092,7 +1092,65 @@ int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) if (copy > size) copy = size; - if (skb_tailroom(skb) > 0) { + if (msg->msg_flags & MSG_SPLICE_PAGES) { + struct page *page, **pages = &page; + ssize_t part; + size_t off, spliced = 0; + bool put = false; + int i; + + do { + i = skb_shinfo(skb)->nr_frags; + part = iov_iter_extract_pages(&msg->msg_iter, &pages, + copy - spliced, 1, 0, &off); + if (part <= 0) { + err = part ?: -EIO; + goto do_fault; + } + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, part); + return -ENOMEM; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + + if (skb_can_coalesce(skb, i, page, off)) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], part); + spliced += part; + if (put) + put_page(page); + } else if (i < MAX_SKB_FRAGS) { + if (!put) + get_page(page); + skb_fill_page_desc(skb, i, page, off, spliced); + spliced += part; + put = false; + } else { + if (put) + put_page(page); + if (!spliced) + goto new_buf; + break; + } + } while (spliced < copy); + + copy = spliced; + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk->sk_wmem_queued += copy; + + } else if (skb_tailroom(skb) > 0) { copy = min(copy, skb_tailroom(skb)); if (is_tls_tx(csk)) copy = min_t(int, copy, csk->tlshws.txleft); From patchwork Fri Mar 31 16:08:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47CADC761AF for ; Fri, 31 Mar 2023 16:10:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2CE56B00A2; Fri, 31 Mar 2023 12:10:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDCAA6B00A5; Fri, 31 Mar 2023 12:10:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA5CE6B00A6; Fri, 31 Mar 2023 12:10:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BA2FB6B00A2 for ; Fri, 31 Mar 2023 12:10:58 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6D8D01A0D9B for ; Fri, 31 Mar 2023 16:10:58 +0000 (UTC) X-FDA: 80629682196.22.182440E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 5D6CF40008 for ; Fri, 31 Mar 2023 16:10:56 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h07aeNnD; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4EhmophDUNRsJifv6aZ6zofwzEVulzOQZ9lN1G4MZvA=; b=VS1WTmCSSO/QBVthJLJtJj6tlXecpe8BfA0uzgEscvtCla9w46mvw3LjBfA3HRr+iJ9D64 nEm3StcXrgxtjlGsDx7Plru0pAI68Xc+khvXkNPV/JJfFoPn7xk3W9sZwvKkmG913bsi3I w1a7I9CVawv1rZfplgRRz4J7zBuAASQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h07aeNnD; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279056; a=rsa-sha256; cv=none; b=YianVyda0KTEyq9YUXwSHMAU8XpaOuOshtp8a0JIBO5goP3Da31H9qlGVDKeTJ3pvbMKZj /TuD7Zgg0LjukjtpJe/3Xi5w2S92i68wYWzEexQeiC/YXHhM8RHhdE4GCJ2hgsbIBi9znY cME+n5OG+CFYNwPC9r3zTwQjK2UQBm8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4EhmophDUNRsJifv6aZ6zofwzEVulzOQZ9lN1G4MZvA=; b=h07aeNnDhSGMegthEzl8xPTK6lZf9ePgMm+uxYZGcWp9MYGjv0Pnje4USsEXZe2Ri4e5h0 anZdjdpBtKcafNgtNdVfS8Dlu9YZiRYS1PosKDuATnNqdZLAcIrvSwYLb4V6dJx1N3dFsQ xeyJ8ApqkFYfkIPtNSJBAVp4hyRIGK0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-377-HVkLcFjMPH6uSNSWjZfLnw-1; Fri, 31 Mar 2023 12:10:51 -0400 X-MC-Unique: HVkLcFjMPH6uSNSWjZfLnw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C56DE185A791; Fri, 31 Mar 2023 16:10:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CBA5914171B6; Fri, 31 Mar 2023 16:10:47 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ayush Sawal Subject: [PATCH v3 32/55] chelsio: Convert chtls_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:51 +0100 Message-Id: <20230331160914.1608208-33-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 5D6CF40008 X-Stat-Signature: rh4zf565agqk65wze4rdtfgjx7gt45px X-HE-Tag: 1680279056-994359 X-HE-Meta: U2FsdGVkX18r88P9i1Ct3XvzrW9AMyywOZbsN0PGSh6HCyKr3PAEH5AvQizqS6RqmaXSMDHUwjIfBl+WoBetvQGjcZxkBuCaU7XEUo7clk3T7nb4wY2sQqyZIEmNEUibXxAxBohDbN+FS7g1edTruZehWGUQ2FQR8XUQTlE/EqTvkjrOsacZPch92wAmgIz/bST1/n++cU9dwZ+4pXAkbeEr0Zn+dedQANg7yPLd7m23vHlm8BamsGNPjtAdycRuQvZdFaSpqUPeCNh09G8vs7eCVDCkjYjwUHpJrdMg9dS5Pwojqw95JrVwiFuM/YlYge/9qCkKZvNUYBRyqa0BDJnVojSSZks5HSjNCCSunYKvCjR5zFskj64kp+KnvW+v0HOMTvcU4A3GTRsSTWwlVdklLotV7AdpqjvJvsqXngvggemfmBZCmADMR7Yqe8C4aMZ+Yl7IpGa+r+3V15zqy+6WG0N3tv45aYnixosbTcDnNV66UuW9hQH2gNNkrQL7jJNRAwUWmXmaGqQ6dyhJgTxvxMrb9Q49DIHZZmus+wlobYnCiXF9QQIsTDkfCVD//2zrLBDV3Dof4pYQ0PNK7+ei7aNvp9wfrOe06WrYoficpX7nhGMsJx7KsxKeaK9Gvl8IWdIAs+1e+MWad/vGLGGaozZA+e8mVv97ThHO+3mHIPsT+NbmsBmGe4js1NvClc7ZHdVfGEvzyNNnWTqq4kfVmhFVfNnc0sA/k8I7+wxdJAwVlxQpPZLwCf1gVuzg8MK15mTx/WN+596u0M/ltrh+PBGTG6v/lCT5LC9F1Hux2bfb5Bur6cfKHezGoEwA0RegXjq0oLMAnAISGJGHJ8lmPEW6+MV0SSFsirwCjqB+pz7V112dQXBgn6LrqAbGEhyftieLhHtrzil7X9qETxiMvfMourequnWGls4zoOMGIIXqNtChuN55ri3MY5GxwWmeFMbkYYv6rzwoLyU WTZHouVc rZZ73LfO9T4+o9LncsoKb8LMzyyW59EkCa1azfXq7hF5o0cw4vmCn0V5o8/RzWcPFVuNwZzB7mSdQTLfVHG+BdH002KS6UbPotSJDU108++veB2cNjmlGtipTxyY5AqVFGllhmqX3PKaIk7NSi3e+/ax9x/spp+G0kYg/sl3HxeZPOyXia9mlpkk1V8en15FbiLQLJxiwC2Zs0YkuYf5s89bucXXybbNEuC2Hh6h+gmKiaMyAJJ9WuKbb0aqv/ORVP6pSOXzjILNow904+1dekO4Te5+ZlOIlE4C5dT1lbiB2jZemq1Uvz1p5IdqYd7WW7EQf7lFQ+7zeB9mFUS+Tcr2A0H/VjxROcrvQstBz6t19Ik9OY9YpTkLx8Iuv2BrGvdUG9zhk95MmJ6Zi4sgeDo4orYMHWnHGUntThxep9Et7bb1koPTg1lxb3bvnsNzYjvr6ZCpY1W6bK8/5NIr8Q/30MNuuc/m7urJqYOMCYY1LSwSAAvaSVa0orw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert chtls_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Ayush Sawal cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- .../chelsio/inline_crypto/chtls/chtls_io.c | 109 ++---------------- 1 file changed, 7 insertions(+), 102 deletions(-) diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c index ca3daf5df95c..5c397cb57300 100644 --- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c +++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_io.c @@ -1288,110 +1288,15 @@ int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int chtls_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct chtls_sock *csk; - struct chtls_dev *cdev; - int mss, err, copied; - struct tcp_sock *tp; - long timeo; - - tp = tcp_sk(sk); - copied = 0; - csk = rcu_dereference_sk_user_data(sk); - cdev = csk->cdev; - lock_sock(sk); - timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - err = sk_stream_wait_connect(sk, &timeo); - if (!sk_in_state(sk, TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && - err != 0) - goto out_err; - - mss = csk->mss; - csk_set_flag(csk, CSK_TX_MORE_DATA); - - while (size > 0) { - struct sk_buff *skb = skb_peek_tail(&csk->txq); - int copy, i; - - if (!skb || (ULP_SKB_CB(skb)->flags & ULPCB_FLAG_NO_APPEND) || - (copy = mss - skb->len) <= 0) { -new_buf: - if (!csk_mem_free(cdev, sk)) - goto wait_for_sndbuf; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (is_tls_tx(csk)) { - skb = get_record_skb(sk, - select_size(sk, size, - flags, - TX_TLSHDR_LEN), - true); - } else { - skb = get_tx_skb(sk, 0); - } - if (!skb) - goto wait_for_memory; - copy = mss; - } - if (copy > size) - copy = size; - - i = skb_shinfo(skb)->nr_frags; - if (skb_can_coalesce(skb, i, page, offset)) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else if (i < MAX_SKB_FRAGS) { - get_page(page); - skb_fill_page_desc(skb, i, page, offset, copy); - } else { - tx_skb_finalize(skb); - push_frames_if_head(sk); - goto new_buf; - } - - skb->len += copy; - if (skb->len == mss) - tx_skb_finalize(skb); - skb->data_len += copy; - skb->truesize += copy; - sk->sk_wmem_queued += copy; - tp->write_seq += copy; - copied += copy; - offset += copy; - size -= copy; - - if (corked(tp, flags) && - (sk_stream_wspace(sk) < sk_stream_min_wspace(sk))) - ULP_SKB_CB(skb)->flags |= ULPCB_FLAG_NO_APPEND; - - if (!size) - break; - - if (unlikely(ULP_SKB_CB(skb)->flags & ULPCB_FLAG_NO_APPEND)) - push_frames_if_head(sk); - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - err = csk_wait_memory(cdev, sk, &timeo); - if (err) - goto do_error; - } -out: - csk_reset_flag(csk, CSK_TX_MORE_DATA); - if (copied) - chtls_tcp_push(sk, flags); -done: - release_sock(sk); - return copied; - -do_error: - if (copied) - goto out; - -out_err: - if (csk_conn_inline(csk)) - csk_reset_flag(csk, CSK_TX_MORE_DATA); - copied = sk_stream_error(sk, flags, err); - goto done; + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return chtls_sendmsg(sk, &msg, size); } static void chtls_select_window(struct sock *sk) From patchwork Fri Mar 31 16:08:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B585C761A6 for ; Fri, 31 Mar 2023 16:11:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 293376B008C; Fri, 31 Mar 2023 12:11:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 243746B00A3; Fri, 31 Mar 2023 12:11:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10BEA6B00A5; Fri, 31 Mar 2023 12:11:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 01BA86B008C for ; Fri, 31 Mar 2023 12:11:04 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C9BFC1A0D6F for ; Fri, 31 Mar 2023 16:11:04 +0000 (UTC) X-FDA: 80629682448.27.B8C01AF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 162D540017 for ; Fri, 31 Mar 2023 16:11:01 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aZBFHWHc; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279062; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KbTdePCQuY00Eh43/mGhCoyrcL0EDX5k9L0XaPYIsA0=; b=ozMvYYkEdAFonj/TkptQPERQ1okmdFVsiNjKLKNT0Bq7aIWUfXAm3fheZwpYydzVit5Y8W exzgFR5XtrgxvXsDDGb9WbcTpIHfOKG+abA7LreSm0BJB4cZYAttvXrvhgF+HGhrUD8IbL zIhm49GDBe8ilB/EXpblSvYsYTqOXuQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aZBFHWHc; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279062; a=rsa-sha256; cv=none; b=YMlnvmmsAdoxyTMQ2+ij235trE9KnJhby5fwyuFEkxdS1uLmV4u9y/yDpdsXjHl0J/9d86 IhIEe3LZ5jr8CKsuZD90u4x5uGvejv6TG1LHz6j0cTr2s0woOKXr+IDArfG6LZu77SNINe 16eINPqqEA81HIXK5floNPSG9AF/Rxs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279061; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KbTdePCQuY00Eh43/mGhCoyrcL0EDX5k9L0XaPYIsA0=; b=aZBFHWHcgm3f9EzBuWjVyEjEzYlLXSkJmSWI2kq0KOHcHlqeXRVnjHSPBrQmFiuB7Wo6Dk qM1kfV0tder+9HZvf7KHfDoE4qp2q5ufNpaKgmEXoK3bXF/zBpCCQK+yukp/Um8xF3Md3x 8RBXHePRmPjf3LalwHZlZMUIk+XNcBM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-615--qgn__EoP-eE5uusdvkQ2A-1; Fri, 31 Mar 2023 12:10:54 -0400 X-MC-Unique: -qgn__EoP-eE5uusdvkQ2A-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7FD543802AC1; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 63D4851FF; Fri, 31 Mar 2023 16:10:50 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Tom Herbert , Tom Herbert Subject: [PATCH v3 33/55] kcm: Support MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:52 +0100 Message-Id: <20230331160914.1608208-34-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 8pjycai81g4jyqd8juhcxj6sdgs5ksq3 X-Rspamd-Queue-Id: 162D540017 X-HE-Tag: 1680279061-406551 X-HE-Meta: U2FsdGVkX199xVXGO1VpiyVFSNpbnCTyaVML7hFYqsX8+p7+qWrXg1n+9BsNyeUpZy5ZswcHrtxUHpWLGOuUbU0L76d8LiRI9ABwys0FivQaOjVASiybC2Kz9N6c5DiyiKQoTXGQF3KhKqP1JGWLd2AENinjNLI87art8W9ynZJxXy+2wF7GzbM9tZLxMUka/jnaxm8Kyqqh6FfiyHUlqTkJNz6knglIYlSKSkr+NLviakwRTUWU6lkTrI3J3qqqViSPlsV8/BpMjgmwp+ExhFYIC87g2onfNEOx4NBDoyBBNy/7sThsE78fEZ9GHV2SYrNUD2WIx38Yo9e+fdHDhQMnhup2xMhHQE4D/XTkC1QIxZYtoTcPUEvUcZJ9ryuO926DDjXVfZaoazMzl64sYV2H6eBWbcYQAMkdWMwmzwgylOPn80C2sAfVNDh6OeXEr3S7bJTl6HR1B+2wTwVieiOmbz/JgyJXLx52jF8GRxFkBHu64tmqvK6wXU+7bYSK451d+kDiKtefRnCztN4pen8rCD2zKckt0Mm3Fig2d6FlI2QiHNoGG36fCwH2YecYr9iCglC4I/TDOxrz3Uizt57EhikOnDyWZDof40JQrsKRoXLCE1aDT4NJoTbP3YtaMiSoEv8QKAGfrq3Exka0oVmYrerPUyzc05ciqA/6ygg0KqIy4I6U0OnHbb1Auv8/dMVxFip5XbPQp77WPKGtp31wAtTIHMTfJNkY5TEqqGaCDew5gIHzG2wFQzwRRH5bYbo0VY6TGwrYJhnRqKw7LubidIYVwFipaEhc+ZyjM9Gmaw5xWIRZNq3snGxEcnGD9QACY4MfVecUQnI6U0n2Or2C8SHVw2Cp5Dk2HFrMzbF8GeyNbe5/cGkVgnItMvUEKe/fP6sph3acn6uWcKTdmiwzKoPu0qqt5oE8xewlOh7UL8NwDSMqI9xXuKdl5QX6eNYt51GjkjOWRk9Th5K 2+LYnZe1 ipdxC0oCaMQ70GEdnIrPIuQDqmCIvclPT3v1ZEaQB8ZKm1vdvnqECYu8/xP7vyvDE0xi8uX2t0CReuLHSQZaiYdftn9+eMxDfw7BugGuzcSj2spixFzTQb1QcGhI6dJSVET3I0lQZesZsyFRKfj6kbx4MpjZkrUgOWFgolGxrX/3S0dqTxbw5Y32S3rJkY6CmFLxwtDcMaBYqD/xlBT/k5PhY953pZ82suClZJmDJycoq3vy00mtM2aH8ZtI591gL1mVyvXDKq3kKfoj0TbGHr/rwUSrA67fOrN4bV7YzacijX4+r4sHMTCnllGlmMVEKZrhDxVBeXUiRTEWgw+0Wg/qZKO+2tQb4MXKFgxuQzlPKUhJ2YKmQfx+b6tgpSUFqW3lRFjF3NR47jlA0j6E0Tpa0XrqOGHpZjPAPTVdueo/dFWc2MjnfenarVK08oic575HN8PoHaaSkYrtZSvNMH74xDA3/rG5d6ShqRTkB9dZVWFG0ozb6qiZwtHcIOD6VAGYB5bm3L/eY0X/GpdiCDFihl/fRDLdRHOca0nUWfeAVQPiOsgNVzVbLUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_KCM sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible and copied otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Tom Herbert cc: Tom Herbert cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/kcm/kcmsock.c | 89 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 72 insertions(+), 17 deletions(-) diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index cfe828bd7fc6..0a3f79d81595 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -989,29 +989,84 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) merge = false; } - copy = min_t(int, msg_data_left(msg), - pfrag->size - pfrag->offset); + if (msg->msg_flags & MSG_SPLICE_PAGES) { + struct page *page = NULL, **pages = &page; + size_t off; + bool put = false; + + err = iov_iter_extract_pages(&msg->msg_iter, &pages, + INT_MAX, 1, 0, &off); + if (err <= 0) { + err = err ?: -EIO; + goto out_error; + } + copy = err; - if (!sk_wmem_schedule(sk, copy)) - goto wait_for_memory; + if (skb_can_coalesce(skb, i, page, off)) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + goto coalesced; + } - err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb, - pfrag->page, - pfrag->offset, - copy); - if (err) - goto out_error; + if (!sk_wmem_schedule(sk, copy)) { + iov_iter_revert(&msg->msg_iter, copy); + goto wait_for_memory; + } + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; - /* Update the skb. */ - if (merge) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + err = -ENOMEM; + goto out_error; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + + skb_fill_page_desc_noacc(skb, i, page, off, copy); + if (put) + put_page(page); +coalesced: + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk->sk_wmem_queued += copy; + sk_mem_charge(sk, copy); + + if (head != skb) + head->truesize += copy; } else { - skb_fill_page_desc(skb, i, pfrag->page, - pfrag->offset, copy); - get_page(pfrag->page); + copy = min_t(int, msg_data_left(msg), + pfrag->size - pfrag->offset); + if (!sk_wmem_schedule(sk, copy)) + goto wait_for_memory; + + err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb, + pfrag->page, + pfrag->offset, + copy); + if (err) + goto out_error; + + /* Update the skb. */ + if (merge) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + } else { + skb_fill_page_desc(skb, i, pfrag->page, + pfrag->offset, copy); + get_page(pfrag->page); + } + + pfrag->offset += copy; } - pfrag->offset += copy; copied += copy; if (head != skb) { head->len += copy; From patchwork Fri Mar 31 16:08:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E154CC76196 for ; Fri, 31 Mar 2023 16:11:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7285A6B00A3; Fri, 31 Mar 2023 12:11:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D77C6B00A5; Fri, 31 Mar 2023 12:11:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 552A36B00A6; Fri, 31 Mar 2023 12:11:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4260E6B00A3 for ; Fri, 31 Mar 2023 12:11:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 10273C107C for ; Fri, 31 Mar 2023 16:11:06 +0000 (UTC) X-FDA: 80629682532.21.3DD4119 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 04359120027 for ; Fri, 31 Mar 2023 16:11:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Zi/ZhrXP"; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jPEZCXpX5VIblKmRhP3z5vIQLhv4QpeLvADltfz6mNU=; b=J+AHjJBmPRp30JSw8U5Z5nCDG3H5KvuxY/gWB271D1Q1FAUJIm8nNwnUeLpdzhbWEf1Xcc t+h9giiwKsdCH/dZxIBuzwByjvxxsM+e9zql136VuDgLv980Psv89F3bNoAlfN+hejCcdY B9z2XOOJBJB8tdCMJ+rs2gIAzmLyBJA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Zi/ZhrXP"; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279063; a=rsa-sha256; cv=none; b=KarfhjqM0XoQWonq9OMDMIYx9Hz+xnjFYQySyb8pvvGfg2gD//k6AslL7ugfBx7z9855fr GoEJZoBKffbKhO3OP7zmgZVHD+1Zrf4tgpoqEpFNYNG2oUWiv8oWe1OeKPCB1iiQ47s1wU puQsD/s7RSi6bU1TGT95vU7w+YQ5srs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jPEZCXpX5VIblKmRhP3z5vIQLhv4QpeLvADltfz6mNU=; b=Zi/ZhrXP4LU5gQln5jrLIxrBir4FDptykfPL0BitacG7C5ACbwBT6DsVzj2EkIPccwqYIp xntMlW6aPfmW5odwIUFv3CQNt6EX7+hCIKIqWEAHtk1h5RoK4axpvRj2AUUYaFIvCgrH8C LY7Gdo0fJQk/2IN5sjLm67NUZdhPnCU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-650-1RKJZrZKPJSsBYeQeItUoQ-1; Fri, 31 Mar 2023 12:10:56 -0400 X-MC-Unique: 1RKJZrZKPJSsBYeQeItUoQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 504CB801206; Fri, 31 Mar 2023 16:10:55 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D9F71121314; Fri, 31 Mar 2023 16:10:53 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Tom Herbert , Tom Herbert Subject: [PATCH v3 34/55] kcm: Convert kcm_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:53 +0100 Message-Id: <20230331160914.1608208-35-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 04359120027 X-Stat-Signature: 8yqywojb4r38xyfw557txxqdiui4gxw6 X-HE-Tag: 1680279062-983595 X-HE-Meta: U2FsdGVkX1+gjH4Upc1gwky6HCyepyPU41LithasUaVN2SQzW3LMwGQFzKHSxuYZriALHvODNFBTJzoTUAUS1VAv9IIi1SmN3KXf+Zyn1fSRXYtjmK0ZtsBQQVtrE25Zs00TZxW/QaP3II7oETCLHP/QzgrMdr/UWjKkKdMIioccGpqbJzu/j767n2nEFt5R9fnkqR75B8YhqXHTn1UjYpaspLWa9gDFCLFbcKRMR0HjXVcBGk+pew3fI5SiR7cfpsAgLnK8EJZvYAtQAJPwo5AlnFC0uMg25WRW9l5o9b+VNt6/+Zk+SdFsTqZmwXtMDxfmEDuKPqdDtqbQd9FHHUDfVEnUNoj+J5qpF3A6bysFI6MeR3/4mILt0a1pKCuBcXcFBdEC2G2cyZ9AAjyfw1RPbfdGVXRxG4PT113YdAeTjcratPCUNg9rfgQgoAHl5oSfnOEf5LPyPdNdPV5hjmVCdUpq13qiRjakh3Xflz/4Iir7RPf7PceZ3OuK684zDGoZ6hvRunepLr3oAGQVp9W1xIa2wtOMhZS+qrdm/2qJfkd9yW/tYECVEO3CUJ57HyTKgd3TBKs3FJx1csFggg9gmfFBUOvFytl9yr3XAz2P45yqi873dnZDIXlo5izuz/Mba3b/s+rMzBv+uynpseb2nqf1lduKY+lm54vurmHNjgb/hMVzYexKXeeMY3eT3DdR/xMWN9aPg/sF6PtGssYqcO2GZ0idhKDHpJtfl+BhGI6JDGsTJWNe+giRh9avC6XrgLJeh6gZHvhDO0RwA9kjxDAMcF86DEen3JlKE1n8mgSeIFieeOgHF0/naFyjv3WjwBRV2CAe9brJH9wh3Zez/c5u9/QYIVNxiQy84cejQdQU9B7oXLOC43iyKkjSYWoyZVpugmzCrxTe0OTZEYrx9ydAxJzknrQ3u95jASZ8HQMvAKGW0wYLk6elpTvTXatBYiHvT21cFfbEi8Z qz8agYZx 7tF4MFm9PRQ6/z9bHybT9Sjwqx8Pqayk4Lz8y22Y/7z/ub+6RFVBYmz5QjERnlNsqxVJyB7XJzxC5ie9dl2qvr2GXvHFJbGgPUBfUbMhqq4oeW4/o/kgd0Y5o3P7RRtkLfJksyl7tchwXM8xW0C4SyZ57YKceX+QFxAX4vG3LMHy7cvUvDZTIyeshTkHhEV5KfBdlByZC3oaKdOILWTnqlzD7az4mfz2SF5F1tQMyk+PiDINBI/+N6I+IUzhi6pnQhzACqZKYelFY2CP0ACm2N8Zn5vWPDPTzWeQ4qgaGU0yOhI/0nydygW7K0Q+YFBLIl9cZZ1/D3qBHeqlamq22Yx61PxW7UI8ZK2CBqSWSOC6ViP0n6Bo3BNyTXlTz/hF0W7JE1Z8qfD8sbjz9dc1TyRN+uxUopARuvSEFoW3KAzLnebPASeHRpEAw+hz+PhSGA4psDZVXCG6AJ1gTPfCxzH9SfaGrlAHIn/PmtVH7zZ6n3vKCPjCltRlsOJw9o2HRluZLf4/gTWYqXXKjlxahn+hKfARUhSecIMNOBsFOVeBs8uqPtr8CFexwdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert kcm_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Tom Herbert cc: Tom Herbert cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/kcm/kcmsock.c | 161 ++++++---------------------------------------- 1 file changed, 18 insertions(+), 143 deletions(-) diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index 0a3f79d81595..d77d28fbf389 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -761,149 +761,6 @@ static void kcm_push(struct kcm_sock *kcm) kcm_write_msgs(kcm); } -static ssize_t kcm_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) - -{ - struct sock *sk = sock->sk; - struct kcm_sock *kcm = kcm_sk(sk); - struct sk_buff *skb = NULL, *head = NULL; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - bool eor; - int err = 0; - int i; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - /* No MSG_EOR from splice, only look at MSG_MORE */ - eor = !(flags & MSG_MORE); - - lock_sock(sk); - - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - err = -EPIPE; - if (sk->sk_err) - goto out_error; - - if (kcm->seq_skb) { - /* Previously opened message */ - head = kcm->seq_skb; - skb = kcm_tx_msg(head)->last_skb; - i = skb_shinfo(skb)->nr_frags; - - if (skb_can_coalesce(skb, i, page, offset)) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - goto coalesced; - } - - if (i >= MAX_SKB_FRAGS) { - struct sk_buff *tskb; - - tskb = alloc_skb(0, sk->sk_allocation); - while (!tskb) { - kcm_push(kcm); - err = sk_stream_wait_memory(sk, &timeo); - if (err) - goto out_error; - } - - if (head == skb) - skb_shinfo(head)->frag_list = tskb; - else - skb->next = tskb; - - skb = tskb; - skb->ip_summed = CHECKSUM_UNNECESSARY; - i = 0; - } - } else { - /* Call the sk_stream functions to manage the sndbuf mem. */ - if (!sk_stream_memory_free(sk)) { - kcm_push(kcm); - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - err = sk_stream_wait_memory(sk, &timeo); - if (err) - goto out_error; - } - - head = alloc_skb(0, sk->sk_allocation); - while (!head) { - kcm_push(kcm); - err = sk_stream_wait_memory(sk, &timeo); - if (err) - goto out_error; - } - - skb = head; - i = 0; - } - - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, size); - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - -coalesced: - skb->len += size; - skb->data_len += size; - skb->truesize += size; - sk->sk_wmem_queued += size; - sk_mem_charge(sk, size); - - if (head != skb) { - head->len += size; - head->data_len += size; - head->truesize += size; - } - - if (eor) { - bool not_busy = skb_queue_empty(&sk->sk_write_queue); - - /* Message complete, queue it on send buffer */ - __skb_queue_tail(&sk->sk_write_queue, head); - kcm->seq_skb = NULL; - KCM_STATS_INCR(kcm->stats.tx_msgs); - - if (flags & MSG_BATCH) { - kcm->tx_wait_more = true; - } else if (kcm->tx_wait_more || not_busy) { - err = kcm_write_msgs(kcm); - if (err < 0) { - /* We got a hard error in write_msgs but have - * already queued this message. Report an error - * in the socket, but don't affect return value - * from sendmsg - */ - pr_warn("KCM: Hard failure on kcm_write_msgs\n"); - report_csk_error(&kcm->sk, -err); - } - } - } else { - /* Message not complete, save state */ - kcm->seq_skb = head; - kcm_tx_msg(head)->last_skb = skb; - } - - KCM_STATS_ADD(kcm->stats.tx_bytes, size); - - release_sock(sk); - return size; - -out_error: - kcm_push(kcm); - - err = sk_stream_error(sk, flags, err); - - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 && err == -EAGAIN)) - sk->sk_write_space(sk); - - release_sock(sk); - return err; -} - static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { struct sock *sk = sock->sk; @@ -1143,6 +1000,24 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) return err; } +static ssize_t kcm_sendpage(struct socket *sock, struct page *page, + int offset, size_t size, int flags) + +{ + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; + + if (flags & MSG_OOB) + return -EOPNOTSUPP; + + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return kcm_sendmsg(sock, &msg, size); +} + static int kcm_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, int flags) { From patchwork Fri Mar 31 16:08:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09591C761A6 for ; Fri, 31 Mar 2023 16:11:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C0426B00A7; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96FFF6B00A9; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 811826B00A8; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6BF9B6B00A6 for ; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 438231C6BA7 for ; Fri, 31 Mar 2023 16:11:09 +0000 (UTC) X-FDA: 80629682658.05.0CE0468 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 38FA7C001D for ; Fri, 31 Mar 2023 16:11:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FebAJZDd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rdhhmJu37H1diE1YTKpbylxaJjF+u38cNy50QUFNMbM=; b=bIt4M02UVVNKqClIlcNswFmqwptctUhuioD7wV+yexU3XBVPKgNoiiUEk84RqFapnmheWD IX3qe0xzlKGzPO9ufCWLvtNqK2Mlg41kaNme24xtuq3jJe6G1uop8wKhY2LI4A8Ts+Zbhj aCO6AJL2Q8Cc2jIVQuCXltudiDmYv3o= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FebAJZDd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279067; a=rsa-sha256; cv=none; b=jnn/OhSsvdMrs/ftGNWKu4pSxdmbcrW5EwZAC0/ep2LOzt2alBxW+nl/X8LGFsX5WmJuoR vdD8zxbB0U7uR0BBb5faVFVTISMvbhciKCMVThvcCEfD0EVE537agElPHjfq8nhiCFbVA4 zQ9ZRH5LGTZcByfH3nn3gG+X+/tC+Ys= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rdhhmJu37H1diE1YTKpbylxaJjF+u38cNy50QUFNMbM=; b=FebAJZDduiCZsBE2c/DWUBiM+9NL6xR6OJxOpFTMvwQNtoeXph8e6zUBAP5hE5S+ynRzno 6Zj2+yVhYzGU9wow6nBiwHlVfyOfztkiRg85nkBR0dkBy4yBowhHsV51IYrvSU/jnFgUwO Tc0w1g7l1l6nI5BB1Qf+ocjkBY1FHUc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-373-HXcdn5EjOZiGCXUameYioQ-1; Fri, 31 Mar 2023 12:11:02 -0400 X-MC-Unique: HXcdn5EjOZiGCXUameYioQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C329F3802AC3; Fri, 31 Mar 2023 16:10:57 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DF6F714171B6; Fri, 31 Mar 2023 16:10:55 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 35/55] splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() Date: Fri, 31 Mar 2023 17:08:54 +0100 Message-Id: <20230331160914.1608208-36-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 38FA7C001D X-Stat-Signature: pwgd6zip8f5baqafbd9q9tw7ahkkh3b6 X-HE-Tag: 1680279067-35722 X-HE-Meta: U2FsdGVkX1/Pw0kboOivN7gFHan/UFm39K4yObkqkIizXOqZJY0hlYT7wFiV4vlKtq5VbVm7naIQYOMAY7HQH/G3FJpLNVfyGqeuAupK4lJnAShoYwn8bCeFSReZJAFixe8LtPMm6Iud7OZHOhk9wDudebkxy2kc8cOX6aLzLM9OdB3oRLjRtPBXK8fLmjSIT2p4XBDHPdFJTOGqRhcOnrTMuKcsLb/eADbNhfnYf05lL9tLn5yRGvN8UaI5qu4dQno9K3ePNEH0R++he3H/+gqMlyFPNZLebchYgLOxQ6lOiR4GlI/d2MJgoOXdb98erjcIxUQNWo8ApODIQjOUcR3PK5P5NK0zo5Zuivr8bC33j+fvUJgqBeIcir/AWVvcQMl7PkL1ZnBFAZwMZwHqxaEGHAz+B2rpvHO5mZDIaimM/vMYQiRrmxXUDxdfL2Cn+c6eYonfmTFI0C+/6x0ecMxWzxCVIX4dXrncO+KSE1wVo90hA/1HZKF0yrrWIEhVD4sHx7v6qzfM5BrdTqcqY/LhtKRKhnhzZeaBcwrXMPlLrfls5rw24wPbR5cMuO2Iai1Lt+gwllFv/hPJN1EVSNtx9Ih6J+E0bfLSOge1n7+6YRFhVFo9VZ/KgMXNYOGnm5xdM3RL5PnO4nE8EUIVZcNNyiUYxvWOpwAEVJW7zElt+fgfpQM9zcUxTtpALKdoWgl1sTw2v2rvN2XQOOXV2n05w16wx01JXmNmJyde+qJOA9fw0KH4THxpVRSEpO9FKU4bzNAxHth5W6EO1xfSIJrwjmi9c0jAoDxVOxgoKGueDBr5S6M6tJCJXgFSN7dwk1MbCtKuTmTmCAjabTNQed9ZCg7//lf3+j8AseImStLMnYJHc02iqUV+bELMaMOd1rZe/ncGqp1t10vF6NG5EPqEGQ9Lkfo1xePb0DTNDWFCPyhlc6N4U0iApoTl+LzrjRs168xDitPQ8QugR8Z eumDUYXH 0cym1Pf2j6btLaaSB+ZhF5m21K/NjYOGkz2unXHwt3hlYg0YwdNrKjEXUJX/rizRdZwOfN9wXX89xU6B/dc96d5nd+Xq2Ku/xMvad1x2V707Rg/AVXSmPgaub2oItFk5987Gh/0hbj4ZLsS+e4udXts00VwQdeLZ5o7OPC+lFylS9LFCTn/hSXNo0pOJLQtEWsgOR0Im6qBw5MUVeIpFXYLTzuDzWgNBLGSepOG5DVfMSaUMCazdSL9AG+rV+/HpTyZk9MnFudUvWKOmApuYyjZm/MMZzOdODSmr5Yxvy0BZ6iHWdvRmJXMJAVJl+QvBahDg3i+50dgUiFkyWONSY2aiXm8d7OvBd06i8uXP/mmffAjD3TCXy4o8hEK+BDThiwS6WUNSkYO9cHOUMxt2NaXle5oFHEdxyKCdYF0820yPO2RDqKX8vOqaCGYJ5uycisBxz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() to splice data from a pipe to a socket. This paves the way for passing in multiple pages at once from a pipe and the handling of multipage folios. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- fs/splice.c | 42 +++++++++++++++++++++++------------------- include/linux/fs.h | 2 -- include/linux/splice.h | 2 ++ net/socket.c | 26 ++------------------------ 4 files changed, 27 insertions(+), 45 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index f46dd1fb367b..23ead122d631 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -410,29 +411,32 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); +#ifdef CONFIG_NET /* * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' * using sendpage(). Return the number of bytes sent. */ -static int pipe_to_sendpage(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) +static int pipe_to_sendmsg(struct pipe_inode_info *pipe, + struct pipe_buffer *buf, struct splice_desc *sd) { - struct file *file = sd->u.file; - loff_t pos = sd->pos; - int more; - - if (!likely(file->f_op->sendpage)) - return -EINVAL; + struct socket *sock = sock_from_file(sd->u.file); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES, + }; - more = (sd->flags & SPLICE_F_MORE) ? MSG_MORE : 0; + if (sd->flags & SPLICE_F_MORE) + msg.msg_flags |= MSG_MORE; if (sd->len < sd->total_len && pipe_occupancy(pipe->head, pipe->tail) > 1) - more |= MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; - return file->f_op->sendpage(file, buf->page, buf->offset, - sd->len, &pos, more); + bvec_set_page(&bvec, buf->page, sd->len, buf->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, sd->len); + return sock_sendmsg(sock, &msg); } +#endif static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { @@ -614,7 +618,7 @@ static void splice_from_pipe_end(struct pipe_inode_info *pipe, struct splice_des * Description: * This function does little more than loop over the pipe and call * @actor to do the actual moving of a single struct pipe_buffer to - * the desired destination. See pipe_to_file, pipe_to_sendpage, or + * the desired destination. See pipe_to_file, pipe_to_sendmsg, or * pipe_to_user. * */ @@ -795,8 +799,9 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, EXPORT_SYMBOL(iter_file_splice_write); +#ifdef CONFIG_NET /** - * generic_splice_sendpage - splice data from a pipe to a socket + * splice_to_socket - splice data from a pipe to a socket * @pipe: pipe to splice from * @out: socket to write to * @ppos: position in @out @@ -808,13 +813,12 @@ EXPORT_SYMBOL(iter_file_splice_write); * is involved. * */ -ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, struct file *out, - loff_t *ppos, size_t len, unsigned int flags) +ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendpage); + return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendmsg); } - -EXPORT_SYMBOL(generic_splice_sendpage); +#endif static int warn_unsupported(struct file *file, const char *op) { diff --git a/include/linux/fs.h b/include/linux/fs.h index c85916e9f7db..f3ccc243851e 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2740,8 +2740,6 @@ extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); -extern ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, - struct file *out, loff_t *, size_t len, unsigned int flags); extern long do_splice_direct(struct file *in, loff_t *ppos, struct file *out, loff_t *opos, size_t len, unsigned int flags); diff --git a/include/linux/splice.h b/include/linux/splice.h index 8f052c3dae95..e6153feda86c 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -87,6 +87,8 @@ extern long do_splice(struct file *in, loff_t *off_in, extern long do_tee(struct file *in, struct file *out, size_t len, unsigned int flags); +extern ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags); /* * for dynamic pipe sizing diff --git a/net/socket.c b/net/socket.c index 0c39ce57d603..3e9bd8261357 100644 --- a/net/socket.c +++ b/net/socket.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -126,8 +127,6 @@ static long compat_sock_ioctl(struct file *file, unsigned int cmd, unsigned long arg); #endif static int sock_fasync(int fd, struct file *filp, int on); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more); static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); @@ -162,8 +161,7 @@ static const struct file_operations socket_file_ops = { .mmap = sock_mmap, .release = sock_close, .fasync = sock_fasync, - .sendpage = sock_sendpage, - .splice_write = generic_splice_sendpage, + .splice_write = splice_to_socket, .splice_read = sock_splice_read, .show_fdinfo = sock_show_fdinfo, }; @@ -1062,26 +1060,6 @@ int kernel_recvmsg(struct socket *sock, struct msghdr *msg, } EXPORT_SYMBOL(kernel_recvmsg); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more) -{ - struct socket *sock; - int flags; - int ret; - - sock = file->private_data; - - flags = (file->f_flags & O_NONBLOCK) ? MSG_DONTWAIT : 0; - /* more is a combination of MSG_MORE and MSG_SENDPAGE_NOTLAST */ - flags |= more; - - ret = kernel_sendpage(sock, page, offset, size, flags); - - if (trace_sock_send_length_enabled()) - call_trace_sock_send_length(sock->sk, ret, 0); - return ret; -} - static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) From patchwork Fri Mar 31 16:08:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA228C761A6 for ; Fri, 31 Mar 2023 16:11:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C3446B00AA; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 773556B00AB; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 577E36B00AC; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 42ADE6B00AA for ; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1D18CAC901 for ; Fri, 31 Mar 2023 16:11:10 +0000 (UTC) X-FDA: 80629682700.19.6640B3F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 26EB420019 for ; Fri, 31 Mar 2023 16:11:07 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ihF2u1cs; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279068; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ecYVxyEqGMFmF1eT8L3egIdesMHxN/xKxl+aET3XOq4=; b=HfkSCRrHqW/lnMS0TpoasrLMT3omddBoh0+f7hMVshKKFzRaKAjaQfILW8T3DRzUvVJfq2 F3o9ZwxEP6uDO3IU8aa7MgVvPl3IfxXw/B5RUDgcc3XalqgzhqDlUETLKQyoRBa/nO+ls1 ixPhZYdXw1WmgQ7lCSl2mbf3aazWdpw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ihF2u1cs; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279068; a=rsa-sha256; cv=none; b=HYauAuG6ioarji3chLJll3FulVoE2IV50zY7/ZBHof5gBjAdem+XFbveCKZ06veGGZuVXL YN7rLKfNI0592RiTzr6qD9PrOutkicqfiJQSagDDNWudlB34EuiIfCJzBBm69qrg1j5SRE NUefqko1xrTSTyrZf0d6NxXu430VA7E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ecYVxyEqGMFmF1eT8L3egIdesMHxN/xKxl+aET3XOq4=; b=ihF2u1csuu+8JBU6QTuT9vy3+j2D3RuPeXa6oYQ24WwTYJyALwZ+gWWXBT0M7cjhP96ePe +HJ2IF5SUEZ+QvQkvmcu9h+XcQWNdUjNzUdTnVyQ5ZiL/+1ujoKsyCI1zj3geZMzoBKKhg QsjIm0QmfM/UwUEmi2vuGDVpRp6ZmW4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-364-3bh1vnySPnOS90RyB8ZPRw-1; Fri, 31 Mar 2023 12:11:04 -0400 X-MC-Unique: 3bh1vnySPnOS90RyB8ZPRw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 61AEC88606A; Fri, 31 Mar 2023 16:11:00 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7FD721121314; Fri, 31 Mar 2023 16:10:58 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 36/55] splice, net: Reimplement splice_to_socket() to pass multiple bufs to sendmsg() Date: Fri, 31 Mar 2023 17:08:55 +0100 Message-Id: <20230331160914.1608208-37-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 26EB420019 X-Stat-Signature: 89gdqop7c8ym3qykursbg3tjot55xtn8 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680279067-463571 X-HE-Meta: U2FsdGVkX19dHSnSBenxie9AtskAaIv0R7c1olpfveMPvqfp85Sd87CbGkJpyc0mGnl00f7KmGmtlRhHdMHZzoR4GPShv108A3+8q8qePcSqO1S7Xo9ts5dqcEsDf28F0wMLP8r/MiDg5OdPACoIzbxnV5V2eSZclNawTiua5hYTjxyicb3xhluhW2T/LSIvZ5H7y+kddt1P5pY5TFj0CuMpJ6GrEiIqpzhkSg2o/NyaJ//uC+H6+JT/uvu/jU+j1+OVfSiPdQZEDuDdm4Bd5FB03DlQjbq8tU2CxN65ee/k88peQc5sIGxeIHznCEivaOXYyWGi/vUSNbzIQis+HhEVeJFW4t16VlLJwtBtBpkzi9hS5vnbW9JxRIsLmTN833DnbSzB3kG3xs3+VuhMlIOQi0NVoy/Nwju6HpnU9QOIN80IJ4yCvLqPttDMPX2jdH0uJgdLjijwhBT+WgP6jRWRBA++fwONJimGYZwUce0Rv6t2mFz/qeXhRGrA0SYmXE+KGjzRehb3I8zfL2tHyeJNj5uVGkSPqvdhe5dh/iw/VdwWiYNdHiIrc27lOgYavxSeAv5InnMLwsVjiI2PMH1WG8VutewyD4y1ydNYcSqVFA8t+7/9hfdn7dyC7eXN6f02HUdm1kKHOYrtzM3LbIFYm7afGg1hNrLle6ejjsmRL1D8hQk6JaxccR+iYQ/A2yOmRZPXiV/46jM6djKYEcSTSZSXFLRck1LJRf6SdJIo2kET9SZVvn7tIBl67iJQLA6jbdt5YLVtPYMeJOwluR02X/gqXuTxLnBLk1n3te5KeJdSvll7X36k3kuTvb+OuWedvnG+SU6Uayz/UDifcC1AWxy0UxjuGSwB8zzd8/CtRVMIf1q8HpANAMRq9eaJ1iULZIHo82gk1L6pbYDOqqVEHt9LnjKYrbCSEyd/aLPtN2QNYOr+su0cNxFZAWjtWwwgGfCDszBHUvbPgmD KGZsPnR0 PDdD9wN4PaYhlsep8+yStoYW6JXKCyMjWlZ5BTF3WvioT0W/3rkDQf4lZMkRebmMaCD2HWlOjuAlNqpRzUqYeF7N1OiEzp9fgGe3I9LgelqO8gdNqho8GEu8yavXbO9FMNHyDBEGzuMm0W1yqRXk0yot6qX5HVWcJEY3dLY8mXseB17Ra4qn8ycUFDCelQutCXKo1Zc4oRppkcAv8agcoLVMYlQ3No6hFXwrdmkRgR4jkfOUu392aHLTO8/XZ4zYY6k2CVLBpfl/LEETeza6Vd/48KqXTHUwnM1U9VkyuyGgcZpQFw6mMM3TASgwvY1Mc3VCr8Qis4iyZDR1XVS0mOlJGWUir8qU4FXY1xoza87Tu+LpUDd8HPZ7xmIl0rSUlSHlLUb1VoigH7S2SshnsRAPoPxizy2aevZq5przr4MJ1YC1t1uQZVxOnEkSuLONHuuhq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reimplement splice_to_socket() so that it can pass multiple pipe buffer pages to sendmsg() in a single go. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- fs/splice.c | 148 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 120 insertions(+), 28 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 23ead122d631..d5bc28b59720 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -411,33 +411,6 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); -#ifdef CONFIG_NET -/* - * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' - * using sendpage(). Return the number of bytes sent. - */ -static int pipe_to_sendmsg(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) -{ - struct socket *sock = sock_from_file(sd->u.file); - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = MSG_SPLICE_PAGES, - }; - - if (sd->flags & SPLICE_F_MORE) - msg.msg_flags |= MSG_MORE; - - if (sd->len < sd->total_len && - pipe_occupancy(pipe->head, pipe->tail) > 1) - msg.msg_flags |= MSG_MORE; - - bvec_set_page(&bvec, buf->page, sd->len, buf->offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, sd->len); - return sock_sendmsg(sock, &msg); -} -#endif - static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { smp_mb(); @@ -816,7 +789,126 @@ EXPORT_SYMBOL(iter_file_splice_write); ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendmsg); + struct socket *sock = sock_from_file(out); + struct bio_vec bvec[16]; + struct msghdr msg = {}; + ssize_t ret; + size_t spliced = 0; + bool need_wakeup = false; + + pipe_lock(pipe); + + while (len > 0) { + unsigned int head, tail, mask, bc = 0; + size_t remain = len; + + /* + * Check for signal early to make process killable when there + * are always buffers available + */ + ret = -ERESTARTSYS; + if (signal_pending(current)) + break; + + while (pipe_empty(pipe->head, pipe->tail)) { + ret = 0; + if (!pipe->writers) + goto out; + + if (spliced) + goto out; + + ret = -EAGAIN; + if (flags & SPLICE_F_NONBLOCK) + goto out; + + ret = -ERESTARTSYS; + if (signal_pending(current)) + goto out; + + if (need_wakeup) { + wakeup_pipe_writers(pipe); + need_wakeup = false; + } + + pipe_wait_readable(pipe); + } + + head = pipe->head; + tail = pipe->tail; + mask = pipe->ring_size - 1; + + while (!pipe_empty(head, tail)) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg; + + if (!buf->len) { + tail++; + continue; + } + + seg = min_t(size_t, remain, buf->len); + seg = min_t(size_t, seg, PAGE_SIZE); + + ret = pipe_buf_confirm(pipe, buf); + if (unlikely(ret)) { + if (ret == -ENODATA) + ret = 0; + break; + } + + bvec_set_page(&bvec[bc++], buf->page, seg, buf->offset); + remain -= seg; + if (seg >= buf->len) + tail++; + if (bc >= ARRAY_SIZE(bvec)) + break; + } + + if (!bc) + break; + + msg.msg_flags = 0; + if (flags & SPLICE_F_MORE) + msg.msg_flags = MSG_MORE; + if (remain && pipe_occupancy(pipe->head, tail) > 0) + msg.msg_flags = MSG_MORE; + msg.msg_flags |= MSG_SPLICE_PAGES; + + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, bc, len - remain); + ret = sock_sendmsg(sock, &msg); + if (ret <= 0) + break; + + spliced += ret; + len -= ret; + tail = pipe->tail; + while (ret > 0) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg = min_t(size_t, ret, buf->len); + + buf->offset += seg; + buf->len -= seg; + ret -= seg; + + if (!buf->len) { + pipe_buf_release(pipe, buf); + tail++; + } + } + + if (tail != pipe->tail) { + pipe->tail = tail; + if (pipe->files) + need_wakeup = true; + } + } + +out: + pipe_unlock(pipe); + if (need_wakeup) + wakeup_pipe_writers(pipe); + return spliced ?: ret; } #endif From patchwork Fri Mar 31 16:08:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD0BBC76196 for ; Fri, 31 Mar 2023 16:11:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E9766B00A9; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06F9C6B00AA; Fri, 31 Mar 2023 12:11:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2AA16B00AB; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D020D6B00A9 for ; Fri, 31 Mar 2023 12:11:09 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9F252160339 for ; Fri, 31 Mar 2023 16:11:09 +0000 (UTC) X-FDA: 80629682658.27.ABAC211 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id D1B101C0021 for ; Fri, 31 Mar 2023 16:11:07 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EBY30ZxV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=13GTOYj8nLHk/MeewarbEqRTm2eeRZxrsOj/rjjnFUM8/ezt0n8Ju62RFLB5D7ulLOwfVN How4QAunYIcxG6JnRYaZcRPwFupia8/lTv5XEQioUUbxuzMp0aYxaymLEV/0m+Eb45/aKj dqJsGg+ZuFCtYanUDR9+oTIK9FVWDtg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EBY30ZxV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279067; a=rsa-sha256; cv=none; b=xXujFGZe7HFKKQqkDObxFynPG0EfrroWxCzIeSTB9rbt2g/ioAB9QPQ+/f67kF/ugA6Om1 nLa1nZAv7/tAeS6ma97rfNwcNiyevXEgcyGiweUQ9DOpFWSOUB4K8XVASKEWxVHWhKWL6H XXA5ZaOi+ucI+731tj8r7dlVxCetW1E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=EBY30ZxVQtFztYYRDl/WIVmOvBfNj0+0sJUH+ynvEjhmt0OAybPWLihknUsfEYpY9MNxyw q1pjsCGQckCE4PgPJBJEBZp/x3+/bObdAch7CHxLyGqVf4qox/eJKt9eQJRB9tFHS0IVpK VuxCvK+YqFRs68wVjCbjo6TbxbHZVgA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-557-yXllut-UOFGxacMfCAiErA-1; Fri, 31 Mar 2023 12:11:05 -0400 X-MC-Unique: yXllut-UOFGxacMfCAiErA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DF65D85A5A3; Fri, 31 Mar 2023 16:11:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF7242166B33; Fri, 31 Mar 2023 16:11:00 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 37/55] Remove file->f_op->sendpage Date: Fri, 31 Mar 2023 17:08:56 +0100 Message-Id: <20230331160914.1608208-38-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D1B101C0021 X-Stat-Signature: 5gm37qrk1u9exw7mjfeuo8pc71wz34uo X-HE-Tag: 1680279067-711517 X-HE-Meta: U2FsdGVkX19vdcFidRFlog6rR0184rimYN7WCuTrN1ugIUOp3kJJMknSBXDrX9At3TJgwuJRYtbik+cTZDnGt1453sK3r6l5tAYxWKBh+gBqjxurtmj0YGalMYrpQsGCrNYNMr7cXaIM54zo/av5uOv6kbC2y+wtIyKeN7XOKv0n8rWov0MTKYowlG48HyEDdrNQKncS1H8a7NnxkuPjQVvdhw4sbQg8z9VqYj3iPdcKiYn/BfVcy1uuJSnAa0haPeRmHZyDCfJtguVwI9DeBC4mIORY0ivLyij8nVd7er7UeuAmTCddtjB5Vd0L6L44rr15hlKdtz/r4r6H6rFi5Erx1JcPsHNf1kcjqCx2AO6mh9Ut0Tjdi2Oya05OTM7TqshZWsaGWgOhzGSoJq5FMYxxdvRtnDPsiuIOSvO5kJlcSvAWhdDfX/zjLLKwStIm2w6gB9a/UVzqes421E/QMbgGmuTglJqfBg0JpE6dBXsltYWsrXISnSUA4l+wYfCbSiUeHtRKsz4L46COcqGiwKUBCMfoUM9jJflWLsQlVv4SHkTbi8+JLqKXQXVUYKk8CDSZhtu4glnFfngGpEcH7oi7VDo1v2gotUUAs753BZGYRhd1jsfTr4HY9hidXWeSaZtusS4wECFhnvK3Ui0CRR7uttPolPz8dcJY7bARs1v1t8H55k3wKKTHMCfiyEHkIw5ytW/I8X4cMfkR8YpdKj5di2GOPot/i+AjYy+gH8wEYh491CgYwpwvQBTz3leI+kFPxziIdrmSVEl5VrK2HWPw18PxzLCJTTOpovziP4UFR0HhOgYoT5X2IKN/0zkQzWTVgV5vZx5wT7wPMoTHnYea/ocjC7pgwA5Mke1rQy17xylD1WDR/ozkgnDIaNhPo0L0ZblntiEzEmfYXTlbZjxQ8Fungt1dH28BFTEBz5f/hYxShvmF1HlLiej8e9LGvo+QGCzLKAjoJcJf/vm 74Xkls2R Tk+LxFVs79w0dArk+ZYqjj4qkNg34RuRWKXX3nUK+JfkcxYlnr7DNBiS+iEBC2p02qUHhTZKiuqeuofkxLX6iT66xABze+zGH8sEF3AXwW0Ck/mAfTM4O4zxURkJycmQKIEFcH3wQQyUKPV6iW1HT+jqVN9kza15SKoFXR/GzaJJYQItPTCrb9QQzt1y0MEzsDW9yzjwDavnPzt5W3tXJ/Qt5AQ5EHKB0+hOH6a8svRmdcUbGTZREOrFByMDdgzCI7rUUN95xabbXj2XxSD+SUf3vaBi76CmrX9pTCg4u6pAJofdsSa0jRnemnW9FMFLij0b5z3g0Np21mq1yVmMJfZ9TZw+lloUFFrCdV6WpYIi9fCulxl5e3QW+7NUvgnrA8Fw9Grj7AOTZhxy6V8LxAqjeeWVUjICZliAQtY4PzY+p1MDf4PHXT2fyFq4CXrX49tZj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove file->f_op->sendpage as splicing to a socket now calls sendmsg rather than sendpage. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/fs.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index f3ccc243851e..a9f1b2543d2c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1773,7 +1773,6 @@ struct file_operations { int (*fsync) (struct file *, loff_t, loff_t, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); int (*check_flags)(int); int (*flock) (struct file *, int, struct file_lock *); From patchwork Fri Mar 31 16:08:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1750C761A6 for ; Fri, 31 Mar 2023 16:11:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DF066B0099; Fri, 31 Mar 2023 12:11:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4403E6B00A0; Fri, 31 Mar 2023 12:11:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26BFA6B00A1; Fri, 31 Mar 2023 12:11:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 173606B0099 for ; Fri, 31 Mar 2023 12:11:16 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E76FF411C1 for ; Fri, 31 Mar 2023 16:11:15 +0000 (UTC) X-FDA: 80629682910.26.985491C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 06B80100006 for ; Fri, 31 Mar 2023 16:11:13 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ErAH9Z7W; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279074; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uOfuvlKLCw3PwI6FlP4CG+iWtw5a+KyQbWE9E7aqyN8=; b=vulFY73O90y2x6E1HLup0KK62H2AFS+/6QqWswr/Mx+tuCLTWxquq+SDqJYkcOqwx9lTDE pNylK7niYS671LDMW05z1qefp7El3cHi22hjxDoZ7Ak2H4RtRgsUtQMTCJ1XWXHz7Lt1cN lc4mCrhZDt2P6OyIvVAdmyTFHph0wqg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ErAH9Z7W; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279074; a=rsa-sha256; cv=none; b=PEdOW0GlJQLlEZopU0RbJZheQdJNVbL2HYqygc/pWidRqe9iSO+TITnMEqh3G6ImohgGkZ LCyDCfBCQoh8RxKGRjt4WRBktzKHcxJhVzSetphy+Ho8tT2aSqZt7k0YifPB+vdRrNew4t U5JThH6dRWcI+c3rtDTifKc+cIK8EK0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uOfuvlKLCw3PwI6FlP4CG+iWtw5a+KyQbWE9E7aqyN8=; b=ErAH9Z7Wfb3uMO80ElctwRoTMeAHtOlgA5SmlctjSoQ2HcUmnyOwSzTrMSkEN3kbsM6k0G SDfslDosuIYj+allqaAoXhktyXyhOvyd1fB+ZoyI4+Tm7ATOWD1EZMDghViutxMpNAsWYY ZyG+rqhr7ttCDlFduOhA9dEwDZYH3v8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-122-i1BP4N28P-Gza7I8sd9Uyg-1; Fri, 31 Mar 2023 12:11:06 -0400 X-MC-Unique: i1BP4N28P-Gza7I8sd9Uyg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8F453C025B9; Fri, 31 Mar 2023 16:11:05 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7A3281415139; Fri, 31 Mar 2023 16:11:03 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [PATCH v3 38/55] siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit Date: Fri, 31 Mar 2023 17:08:57 +0100 Message-Id: <20230331160914.1608208-39-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: 06B80100006 X-Stat-Signature: 6hfwcppskucjj7reyfm95obbjownbojp X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680279073-925264 X-HE-Meta: U2FsdGVkX18c+4U1Tx/Bz5ji18foNyxQb9wifW/vbQYWRcmqRyhLCOh3ZGfFFob1a2PLyCt7PitRsOOa/MtXVSanb9yjHmZQQMMqsmhezBySSlT0t0K2y0bxj2W5Nnatf9XKgyd+zwkbOATf/u523VBrKiifl/sDvcUCOEAvUskp2Wjhski6EmvLiZNijCVHlYqpjMqHqY2dMfL8nGSPHidH7h2+rowO1ifDGtUrr5I4mv6VgIAGADWRbdyspsNIlv754Hxdh07xYrf5MqSpJMlrcLzkYX9Kh08YHr6pmeQNz5vW6PaEgekKb1kYn7tal0QKw+6YblOr3spQx/MLk7vn0CVctADW/0P2NvFyI2mLzg0XyvB3q4QKs72OPV+AX+QUjiJ84LQhDIYL0Nzebzd09vXBlfeFOwqnmFNWymlopKdo2m/tj8yNIYNS/G7O3Ae/aR9ibUsq7uFKYSizYy+aOzbKrrDLjdKgk7jRbeQJg5jaVbBgzmCiS2E12QCj0RuLLK1BH0tRv1QsI+7nC5gTq52316sthTpMsrD1Qrz2ZJ5H9ybgSSxhMKW789A6iz2x8pQWAYpJpDsT0poIwK4OzKsYI+jCtORTdQtOmjvggb1qWf18dJqMkOqjPzXTBm5umvgKy4A55WDxkV+s/NCPnbFb39Cw+vk00p2UdunXZx1FTObFV8IK6v+TFR2bL8IQm86QQqeTZq1Eye1CjPMD1sxfhjdQsd2RNgOUuiXKgErGPBYNJx+2pyk62U1fTORAEIkkNdoov1zcxV3BxkJ5yla0FX2YT41SJi8Npp3WGR0WH/etq67NzLnerTmolmyQaIKPFX6/MPS+zo8wZ6pVFcctHRC5TkKd93YrGkT3604BkKGK19eQv4DcAcW4XS42kBpehNKCjRJ+4PaTPUh05wQdyDaeZ6btuGy+EjqSGv15YWR1whO/2SlDeLxiOq+vpIVIyiMeD6uGqBD +BqP2m11 uCbQLa2Qfe+RmAVgbrLkhHqsq/3jEisdLQQVZ4Hs6b2DPZtepRb+2CxqGILOuhf25+iv6wsIFnFvT/8KvHBT/NIw2C6f1+cuFOIg64C35cHZJC7Pzirdus6e6iit3Eo5kTePTVKUlJqKrDiO74LW/9ytWXEBjdhOGzQFoqFOfGw3KKTOpJbsj4A/NtYn4MjG/4d7wOfuVZcCgdEypeIXHNTD1vttJS4te5t1fFhGSRjGCkG9JCrAI+jptcQxU2x6DDoYq3PZMauiFna+rwWBF5KGo2aeRRFD506QWIdr2JTWZ/ZWdG8S45CYvu6ZbjPW5n98B3OByCEMbg+2EY9BymoVWCZuVyfOYhd+TiinzBx7iAtcOgRMjVQJ9jIpGStDd4/MrQZ9pYOVbf/LjH33ZXlMAKYWjQpQwi6IY9VGV8OHIrAbnID3TtS+Qg0dAa8Q6LMgVdF+XvEH1Ybs+Mu4hNTRRoVglE9qiC8KbUzcc3NOAiGVrlANjKVtrWyjtVpDINLqGaMSOQQj2CDHnTy7WxQJAsA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer (if present) are copied into page fragments that can be freed with put_page(). Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 234 ++++++-------------------- 1 file changed, 48 insertions(+), 186 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index fa5de40d85d5..fbe80c06d0ca 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,114 +312,8 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, return rv; } -/* - * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. - * - * Using sendpage to push page by page appears to be less efficient - * than using sendmsg, even if data are copied. - * - * A general performance limitation might be the extra four bytes - * trailer checksum segment to be pushed after user data. - */ -static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, - size_t size) -{ - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | - MSG_SPLICE_PAGES), - }; - struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0; - - while (size) { - size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); - - if (size + offset <= PAGE_SIZE) - msg.msg_flags = MSG_MORE | MSG_DONTWAIT; - - tcp_rate_check_app_limited(sk); - bvec_set_page(&bvec, page[i], bytes, offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - -try_page_again: - lock_sock(sk); - rv = tcp_sendmsg_locked(sk, &msg, size); - release_sock(sk); - - if (rv > 0) { - size -= rv; - sent += rv; - if (rv != bytes) { - offset += rv; - bytes -= rv; - goto try_page_again; - } - offset = 0; - } else { - if (rv == -EAGAIN || rv == 0) - break; - return rv; - } - i++; - } - return sent; -} - -/* - * siw_0copy_tx() - * - * Pushes list of pages to TCP socket. If pages from multiple - * SGE's, all referenced pages of each SGE are pushed in one - * shot. - */ -static int siw_0copy_tx(struct socket *s, struct page **page, - struct siw_sge *sge, unsigned int offset, - unsigned int size) -{ - int i = 0, sent = 0, rv; - int sge_bytes = min(sge->length - offset, size); - - offset = (sge->laddr + offset) & ~PAGE_MASK; - - while (sent != size) { - rv = siw_tcp_sendpages(s, &page[i], offset, sge_bytes); - if (rv >= 0) { - sent += rv; - if (size == sent || sge_bytes > rv) - break; - - i += PAGE_ALIGN(sge_bytes + offset) >> PAGE_SHIFT; - sge++; - sge_bytes = min(sge->length, size - sent); - offset = sge->laddr & ~PAGE_MASK; - } else { - sent = rv; - break; - } - } - return sent; -} - #define MAX_TRAILER (MPA_CRC_SIZE + 4) -static void siw_unmap_pages(struct kvec *iov, unsigned long kmap_mask, int len) -{ - int i; - - /* - * Work backwards through the array to honor the kmap_local_page() - * ordering requirements. - */ - for (i = (len-1); i >= 0; i--) { - if (kmap_mask & BIT(i)) { - unsigned long addr = (unsigned long)iov[i].iov_base; - - kunmap_local((void *)(addr & PAGE_MASK)); - } - } -} - /* * siw_tx_hdt() tries to push a complete packet to TCP where all * packet fragments are referenced by the elements of one iovec. @@ -439,15 +333,14 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) { struct siw_wqe *wqe = &c_tx->wqe_active; struct siw_sge *sge = &wqe->sqe.sge[c_tx->sge_idx]; - struct kvec iov[MAX_ARRAY]; - struct page *page_array[MAX_ARRAY]; + struct bio_vec bvec[MAX_ARRAY]; struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_EOR }; + void *trl, *t; int seg = 0, do_crc = c_tx->do_crc, is_kva = 0, rv; unsigned int data_len = c_tx->bytes_unsent, hdr_len = 0, trl_len = 0, sge_off = c_tx->sge_off, sge_idx = c_tx->sge_idx, pbl_idx = c_tx->pbl_idx; - unsigned long kmap_mask = 0L; if (c_tx->state == SIW_SEND_HDR) { if (c_tx->use_sendpage) { @@ -457,10 +350,15 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) c_tx->state = SIW_SEND_DATA; } else { - iov[0].iov_base = - (char *)&c_tx->pkt.ctrl + c_tx->ctrl_sent; - iov[0].iov_len = hdr_len = - c_tx->ctrl_len - c_tx->ctrl_sent; + const void *hdr = &c_tx->pkt.ctrl + c_tx->ctrl_sent; + void *h; + + rv = -ENOMEM; + hdr_len = c_tx->ctrl_len - c_tx->ctrl_sent; + h = page_frag_memdup(NULL, hdr, hdr_len, GFP_NOFS, ULONG_MAX); + if (!h) + goto done; + bvec_set_virt(&bvec[0], h, hdr_len); seg = 1; } } @@ -478,28 +376,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } else { is_kva = 1; } - if (is_kva && !c_tx->use_sendpage) { - /* - * tx from kernel virtual address: either inline data - * or memory region with assigned kernel buffer - */ - iov[seg].iov_base = - (void *)(uintptr_t)(sge->laddr + sge_off); - iov[seg].iov_len = sge_len; - - if (do_crc) - crypto_shash_update(c_tx->mpa_crc_hd, - iov[seg].iov_base, - sge_len); - sge_off += sge_len; - data_len -= sge_len; - seg++; - goto sge_done; - } while (sge_len) { size_t plen = min((int)PAGE_SIZE - fp_off, sge_len); - void *kaddr; if (!is_kva) { struct page *p; @@ -512,33 +391,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) p = siw_get_upage(mem->umem, sge->laddr + sge_off); if (unlikely(!p)) { - siw_unmap_pages(iov, kmap_mask, seg); wqe->processed -= c_tx->bytes_unsent; rv = -EFAULT; goto done_crc; } - page_array[seg] = p; - - if (!c_tx->use_sendpage) { - void *kaddr = kmap_local_page(p); - - /* Remember for later kunmap() */ - kmap_mask |= BIT(seg); - iov[seg].iov_base = kaddr + fp_off; - iov[seg].iov_len = plen; - - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - iov[seg].iov_base, - plen); - } else if (do_crc) { - kaddr = kmap_local_page(p); - crypto_shash_update(c_tx->mpa_crc_hd, - kaddr + fp_off, - plen); - kunmap_local(kaddr); - } + + bvec_set_page(&bvec[seg], p, plen, fp_off); } else { /* * Cast to an uintptr_t to preserve all 64 bits @@ -552,12 +410,15 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) * bits on a 64 bit platform and 32 bits on a * 32 bit platform. */ - page_array[seg] = virt_to_page((void *)(va & PAGE_MASK)); - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - (void *)va, - plen); + bvec_set_virt(&bvec[seg], (void *)va, plen); + } + + if (do_crc) { + void *kaddr = kmap_local_page(bvec[seg].bv_page); + crypto_shash_update(c_tx->mpa_crc_hd, + kaddr + bvec[seg].bv_offset, + bvec[seg].bv_len); + kunmap_local(kaddr); } sge_len -= plen; @@ -567,13 +428,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) if (++seg > (int)MAX_ARRAY) { siw_dbg_qp(tx_qp(c_tx), "to many fragments\n"); - siw_unmap_pages(iov, kmap_mask, seg-1); wqe->processed -= c_tx->bytes_unsent; rv = -EMSGSIZE; goto done_crc; } } -sge_done: + /* Update SGE variables at end of SGE */ if (sge_off == sge->length && (data_len != 0 || wqe->processed < wqe->bytes)) { @@ -582,15 +442,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) sge_off = 0; } } - /* trailer */ - if (likely(c_tx->state != SIW_SEND_TRAILER)) { - iov[seg].iov_base = &c_tx->trailer.pad[4 - c_tx->pad]; - iov[seg].iov_len = trl_len = MAX_TRAILER - (4 - c_tx->pad); - } else { - iov[seg].iov_base = &c_tx->trailer.pad[c_tx->ctrl_sent]; - iov[seg].iov_len = trl_len = MAX_TRAILER - c_tx->ctrl_sent; - } + /* Set the CRC in the trailer */ if (c_tx->pad) { *(u32 *)c_tx->trailer.pad = 0; if (do_crc) @@ -603,23 +456,29 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) else if (do_crc) crypto_shash_final(c_tx->mpa_crc_hd, (u8 *)&c_tx->trailer.crc); - data_len = c_tx->bytes_unsent; - - if (c_tx->use_sendpage) { - rv = siw_0copy_tx(s, page_array, &wqe->sqe.sge[c_tx->sge_idx], - c_tx->sge_off, data_len); - if (rv == data_len) { - rv = kernel_sendmsg(s, &msg, &iov[seg], 1, trl_len); - if (rv > 0) - rv += data_len; - else - rv = data_len; - } + /* Copy the trailer and add it to the output list */ + if (likely(c_tx->state != SIW_SEND_TRAILER)) { + trl = &c_tx->trailer.pad[4 - c_tx->pad]; + trl_len = MAX_TRAILER - (4 - c_tx->pad); } else { - rv = kernel_sendmsg(s, &msg, iov, seg + 1, - hdr_len + data_len + trl_len); - siw_unmap_pages(iov, kmap_mask, seg); + trl = &c_tx->trailer.pad[c_tx->ctrl_sent]; + trl_len = MAX_TRAILER - c_tx->ctrl_sent; } + + rv = -ENOMEM; + t = page_frag_memdup(NULL, trl, trl_len, GFP_NOFS, ULONG_MAX); + if (!t) + goto done_crc; + bvec_set_virt(&bvec[seg], t, trl_len); + + data_len = c_tx->bytes_unsent; + + if (c_tx->use_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, seg + 1, + hdr_len + data_len + trl_len); + rv = sock_sendmsg(s, &msg); + if (rv < (int)hdr_len) { /* Not even complete hdr pushed or negative rv */ wqe->processed -= data_len; @@ -680,6 +539,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } done_crc: c_tx->do_crc = 0; + if (c_tx->state == SIW_SEND_HDR) + folio_put(page_folio(bvec[0].bv_page)); + folio_put(page_folio(bvec[seg].bv_page)); done: return rv; } From patchwork Fri Mar 31 16:08:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF917C76196 for ; Fri, 31 Mar 2023 16:11:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 887596B00A1; Fri, 31 Mar 2023 12:11:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 837806B00A2; Fri, 31 Mar 2023 12:11:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D8CE6B00AB; Fri, 31 Mar 2023 12:11:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4DC7C6B00A1 for ; Fri, 31 Mar 2023 12:11:18 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1645FA0DD1 for ; Fri, 31 Mar 2023 16:11:18 +0000 (UTC) X-FDA: 80629683036.30.AA270B4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 2D64DC0027 for ; Fri, 31 Mar 2023 16:11:16 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A+qn46w9; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=6HvC06CoOeCQO7i0E8fyDa1U2bDUwtC0T+WDnmuAYvrz5sCmfg9IlIXMuNI35t1h9MLg3c VgfS3597l4E4oitgE2vpDIv3fzYphdv1hJdIBDnyMshgVJ2TVgTpYPpeAun7BN2iYgfT5h hpUJxKLRt8QhGJK+9qPWd0iSQDpG00U= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A+qn46w9; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279076; a=rsa-sha256; cv=none; b=gRxftVlDPof4G4jywqA9dBSQF+GZrMZfl2+JLs2DfdU8xVKVzYz/N4ifG4cXmmb7lVZkA4 fJbJuT06RFC7ENJrYIkHZNmxp+1t3mb/Ib7Qvl5IXZ/sUKvdYTFLdBoB7/Mw11uQodngAw ZQMCJJb78nZS/b3NrKR0+tFirsBcO8E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=A+qn46w9RUjQcZ5LakqmwX06iTsyXJKNOdBleaizK6CyIbO6y/+gBh1ldHIyskYZ43kyty SzGo8fzoz3ZWwLGHT9B111dwmVcCPdisn0s3Yj2D9v6VLO+KkD44HqBg1snTX1uocByXFx LIbgcRjyYABLfpwPqKgCpCCufmHEzN8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-519-amCcfeoRO16U4csq6Trr-w-1; Fri, 31 Mar 2023 12:11:11 -0400 X-MC-Unique: amCcfeoRO16U4csq6Trr-w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BE84088904E; Fri, 31 Mar 2023 16:11:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97DD02166B33; Fri, 31 Mar 2023 16:11:06 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [PATCH v3 39/55] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Fri, 31 Mar 2023 17:08:58 +0100 Message-Id: <20230331160914.1608208-40-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2D64DC0027 X-Stat-Signature: ey3tc9akt9t5cdf8dh4hxubsowncs46k X-HE-Tag: 1680279076-613577 X-HE-Meta: U2FsdGVkX19/OGDmrWy7aThnEdZLQ6hEUOJqe17QanRukD6Ey1GGKkPvbZm3mCuHw/HbAvCwr2lCdKVNVzZKVkwufxuAyS0XE0A+uMvkniiEEWik33Es8BbxZ4m7sN5Qt2zp8Jpz5ph+4M06MhVjYmHUnHlsxJ8RlxnLwvwC30XNDJKWQibPwfXcGZwn2qkppqaarasJbWMP+qpcmpTa/m7BJBHKd56xaDZlqh2P9kys5qBL58xqjvNz2L6GooteUdy8B3XNcltlcUnwXjdD9IEhFEs2baOBMj0xYyrhzLO/uIt3G1jiRPpSDYhBMWI6d6Ar55m38FDXL2cyUYyRiZai1ltaE6QYV5efxQ9etyK81csMVS0z6XxTQeJiAU53INAmiIa6OZYh0iyYSAfKbj8iQ62wY8q7Eedwq7HmPrIpGIGrXn8ZiVNFtJwOrNUL6+B5mev/pNCYJYTj/OGLI/SaB5Tp6qftePJ2YkY7fxLHq2Pa2tuHGDtV1lpJOhNsZBJS5eG4FmQ1bq/byDo96XmXI5KLT9UAIxcx8NvxRo8oHzKvpHyTAPfXG1+PVv/oaFmOjuAAsdqKirtZdKGY/yvyAv9VHpWzDgHe5ofgfi2S9UnInHBi5T24dTTfUXqT2NwAjjulftJMusrrQyS4fDRrxfweBO0kfA8MkSq7QyuvJzae87bJYfognxqUsqtHcw8IkPTNcPkN/wklQ3YhgjlUbpiC7PyYpiXyoKp7dXAhgOhzGDWV/ys1K5GBCW3XFyK37/Xsio5oCLR4a0it7bGWLHoY2m1pk+VznpZ70NclE+FnxG0CtEPQV4QY6lSJNNWvWP1t3TCjSxUol6hQstD4z+UBsixPCQOVKSyxbGeY4WT3hec6Ez/SjpmkERKhIFTH9NjQwpfGJ1RGmRhqMgPBEhAEJf3I6Jt/Qv92xFP6QFsw7W2sw/llWWju63+eVaelHEk6Zhg7/X0zhYa Trbn8gg0 MZrzMReIw6F+mXUUCCvkyr0tG0//shSHdmetwTU0WRZ1yzYQaFW49ykIF/Trwa2jGYfEb4/X+hVf5VBBFEhTE3XejpWflEaBBdJBMJKGNkZQ1kFOUqmjZ1oG00Upkh52RDcx99eClTbT71HdxmgRIwjfwtZlXDcEN/MAdo5JeXILxKVhFDCmydFI49GEheKPZEcCffJ4412v8heWaWSJYLW4hjJUe3lNq62oHkmxJFGPXFVcJAsByNjCkb7tN9HR92JcJeNf0XWTXWXDLXTC0HDfST8PAJgkhZ7AP+wgZaE3TI6LOnLJhGRYRg0Wg0B8q+5VWd1YAyX/wWzBSG4kti3Paawn73TytNI9915xUe0st3MwL0iPw7MkG0RkbrzD2qrlJRWrOyh0l+kJiLzUJRzRSlim0109Fqn9eqdhqT433kaGfBLZX1j8L8N9H5dsEA9DB7AZwiiBt9puQuhue7CcwKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v1.c | 58 ++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 39 deletions(-) diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index d664cb1593a7..b2d801a49122 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -74,37 +74,6 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov, return r; } -/* - * @more: either or both of MSG_MORE and MSG_SENDPAGE_NOTLAST - */ -static int ceph_tcp_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int more) -{ - ssize_t (*sendpage)(struct socket *sock, struct page *page, - int offset, size_t size, int flags); - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - int ret; - - /* - * sendpage cannot properly handle pages with page_count == 0, - * we need to fall back to sendmsg if that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag which - * triggers one of hardened usercopy checks. - */ - if (sendpage_ok(page)) - sendpage = sock->ops->sendpage; - else - sendpage = sock_no_sendpage; - - ret = sendpage(sock, page, offset, size, flags); - if (ret == -EAGAIN) - ret = 0; - - return ret; -} - static void con_out_kvec_reset(struct ceph_connection *con) { BUG_ON(con->v1.out_skip); @@ -464,7 +433,6 @@ static int write_partial_message_data(struct ceph_connection *con) struct ceph_msg *msg = con->out_msg; struct ceph_msg_data_cursor *cursor = &msg->cursor; bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; u32 crc; dout("%s %p msg %p\n", __func__, con, msg); @@ -482,6 +450,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ crc = do_datacrc ? le32_to_cpu(msg->footer.data_crc) : 0; while (cursor->total_resid) { + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST, + }; struct page *page; size_t page_offset; size_t length; @@ -494,9 +466,12 @@ static int write_partial_message_data(struct ceph_connection *con) page = ceph_msg_data_next(cursor, &page_offset, &length); if (length == cursor->total_resid) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, page, page_offset, length, - more); + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, length, page_offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, length); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) { if (do_datacrc) msg->footer.data_crc = cpu_to_le32(crc); @@ -526,7 +501,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ static int write_partial_skip(struct ceph_connection *con) { - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST | MSG_MORE, + }; int ret; dout("%s %p %d left\n", __func__, con, con->v1.out_skip); @@ -534,9 +512,11 @@ static int write_partial_skip(struct ceph_connection *con) size_t size = min(con->v1.out_skip, (int)PAGE_SIZE); if (size == con->v1.out_skip) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, ceph_zero_page, 0, size, - more); + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; + bvec_set_page(&bvec, ZERO_PAGE(0), size, 0); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) goto out; con->v1.out_skip -= ret; From patchwork Fri Mar 31 16:08:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10131C77B62 for ; Fri, 31 Mar 2023 16:11:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D91146B00A2; Fri, 31 Mar 2023 12:11:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF3C86B00AC; Fri, 31 Mar 2023 12:11:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF7686B00AD; Fri, 31 Mar 2023 12:11:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 994B66B00A2 for ; Fri, 31 Mar 2023 12:11:19 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4378F16100D for ; Fri, 31 Mar 2023 16:11:19 +0000 (UTC) X-FDA: 80629683078.12.003D0DE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 94842100021 for ; Fri, 31 Mar 2023 16:11:17 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RBrFM4XY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279077; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gZm+OH3nuU2w5lgBIDJ5zMpRPGFGHnStb+oddo8E9cs=; b=bJbgtwINnd4gQxlmPOT/iT+2Z2N1e7Kej7Vsx7oI0fJX9cOjKvcT5QItNEVi39BjF8EhFI MHWc+3X52N51GgYF+iFwGEN90msT9W7jjwMi78Ifyvpr0r9pAMTvvVEoOuMYEgSUoWoOMo ObNlHykN/eagGn70M/VxwGbXeyzYLK0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RBrFM4XY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279077; a=rsa-sha256; cv=none; b=5bLyYTY44254YpaWf8cOBWWvF7lqwxW5h+Qbp+Mi3DtrSuYJQVDWIPh/VKiQ+vi2zaokFA JCgbUaqxjwqUkZuWKKWojMOF/UJgBzByvWwcJc6dx8+u3rymqi+sOXLfwvY/bIVp0yXvPb wiajKrYgEFZJK3jFosj6ojtga3HORxo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gZm+OH3nuU2w5lgBIDJ5zMpRPGFGHnStb+oddo8E9cs=; b=RBrFM4XYpM+ziuhf1hmIjqOm1JH2sgRenm2aeUvUQNqeGwURuTu+ZpQYnktwtpn23CfVWO rngOoa4wbVWsg07Nh50IzF8uT3kBHkIR5JuWJPgUPYm+LvoDF/9iuH5lMWHOyp9BaANXlp 18DQ1GXM2h2F+VkLGiAXzzgv7+sgs9Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-641-aQ6YNqtSNLCDJnr_-8NcqQ-1; Fri, 31 Mar 2023 12:11:12 -0400 X-MC-Unique: aQ6YNqtSNLCDJnr_-8NcqQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 83939802D1A; Fri, 31 Mar 2023 16:11:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5AF014020C82; Fri, 31 Mar 2023 16:11:09 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Martin K. Petersen" , linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [PATCH v3 40/55] iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Fri, 31 Mar 2023 17:08:59 +0100 Message-Id: <20230331160914.1608208-41-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 94842100021 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ezmxcb1nwaw3e7k9sg43jabub5rd1hq3 X-HE-Tag: 1680279077-237573 X-HE-Meta: U2FsdGVkX191iq5fsvFoksl/Z4H56FmcWcbVgXCnkMrAc1gT02h0jJxezYB1xi1kJUQFm6pgN4mZucxu+WVYUYe7icyceiOyqqaJBaLS7tvWx0BrjkX1oAmAX8/SUepHzDqZLYFRE7Cg/++eqgnY95hu6jqHxx6LjpmFF+Kvp2ysWdnnYV/bz4qfn9nfixddscAAFMLoO9BE3VLDLgkq1bX87cxunNRthO9sVUBG7H4/dkVgb8GGrdX64hp68/tumT6YHqaTfnzhWoDmJEFFoQ4ZRSI31vH4CKbVoHFVLeo60T7BTJVjPlD+zoRYRG2lxG6h7W/1lKZCCbDe5UW1RAIERP+wsajo1QSQD2Xfeda26zBqW5UfQYG4YhHBbesLCCk2nvpox/ZW8ksOVQu3YnsjS9TI4E2rXMF+jLS7RcH4i746pXhRj0uDhLDgdjiAn0tAVLmeKaXwJxt6PyZfyflDPZ9/fNtpjHnHnCf1OigB2oXsP6qDAgEiwGrJqHQdrvn9/VfHj7msiuH+m4xzVhYn+4Ee2D4JwF5lG0sJ9TokEgKMivs5dEW6Izy67C9mRRBemRLqndX+Ey/r0GGz5pshPbnwdxV//GMiGveoyhi7LeCK7HlqwtpBkOnhi8Wg9WwwY0rjNrYoIKXhnC/Nngm3ViXdnQOLdsGetwnFF6DtY1cfFlhGazwHRaruJR2pQ2G4+R6UPxBMTJNwBQdcBmSsqeUbgCNwPp56Wde2TSXU7RFgLmrgwiMPZHKSS1IuZv6jGL7KDyERoOj5xi4YI9u5KNkmS+icX0SOqpbiyGmwrpUbiZhkXgNTpEds7piGMKsJ4jbbSj1zz/EMbIJ4CL4Zo2OlD5JZVz5xkW4GUxdfcrvpTa0uvKCzK2TY15RbPXpxjAOvMkb3uABesux0eDAT/M8N3hOUd2ngFpz7ZFNByxdu2rFmPJihktKMHEy7dGhhp8nouRjh3zlf3a6 L/5Rl7Wy CdY/rVv1kWK4aOEsd+/IIA2mO11G6BKHsy+PpXviAJYoPEVch4Zo48pHJNMoR4JmZeFXkGwWZseeM9NdPXBynm5ONmpi/d4xIzJpW0BWiK7CfbIOc+JszOgFJPu7/AgUZeMrTNrl3GqaRgFA0pLAgLwLOwj634qEoKWhkzvjvUM0D/zFuYP+7z6E/oLl4QP+PzuE/wqGWuXzUcIwd1DPSd2el2NLtf5adw0yBYyEGqjKmEbCL6dmv4hdoxqjfU/9dAFKpSVEpZQRDupG2BN28CIvpmf56s3+LVLtATyP3S/qEV7uQXnf7rOKGlkR+iQb9xw22mblK7ukSlo3ays32SnN9TmRHhzEzmOKwYS11mVzMWBr5uCREzrtDxReb5hFupEZbXj+OH8geYg962t6LY7Px7B7M51SCAgOsBzx3hJBjYabcYxCfxQWJawg+4hBI+rz4wX/VGjodjYUNnvfAvz7RSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage. This allows multiple pages and multipage folios to be passed through. TODO: iscsit_fe_sendpage_sg() should perhaps set up a bio_vec array for the entire set of pages it's going to transfer plus two for the header and trailer and page fragments to hold the header and trailer - and then call sendmsg once for the entire message. Signed-off-by: David Howells cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/scsi/iscsi_tcp.c | 31 ++++++++++++------------ drivers/scsi/iscsi_tcp.h | 2 +- drivers/target/iscsi/iscsi_target_util.c | 14 ++++++----- 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index c76f82fb8b63..cf3eb55d2a76 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -301,35 +301,37 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, while (!iscsi_tcp_segment_done(tcp_conn, segment, 0, r)) { struct scatterlist *sg; + struct msghdr msg = {}; + union { + struct kvec kv; + struct bio_vec bv; + } vec; unsigned int offset, copy; - int flags = 0; r = 0; offset = segment->copied; copy = segment->size - offset; if (segment->total_copied + segment->size < segment->total_size) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; if (tcp_sw_conn->queue_recv) - flags |= MSG_DONTWAIT; + msg.msg_flags |= MSG_DONTWAIT; - /* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { + if (tcp_conn->iscsi_conn->datadgst_en) + msg.msg_flags |= MSG_SPLICE_PAGES; sg = segment->sg; offset += segment->sg_offset + sg->offset; - r = tcp_sw_conn->sendpage(sk, sg_page(sg), offset, - copy, flags); + bvec_set_page(&vec.bv, sg_page(sg), copy, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &vec.bv, 1, copy); } else { - struct msghdr msg = { .msg_flags = flags }; - struct kvec iov = { - .iov_base = segment->data + offset, - .iov_len = copy - }; - - r = kernel_sendmsg(sk, &msg, &iov, 1, copy); + vec.kv.iov_base = segment->data + offset; + vec.kv.iov_len = copy; + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &vec.kv, 1, copy); } + r = sock_sendmsg(sk, &msg); if (r < 0) { iscsi_tcp_segment_unmap(segment); return r; @@ -746,7 +748,6 @@ iscsi_sw_tcp_conn_bind(struct iscsi_cls_session *cls_session, sock_no_linger(sk); iscsi_sw_tcp_conn_set_callbacks(conn); - tcp_sw_conn->sendpage = tcp_sw_conn->sock->ops->sendpage; /* * set receive state machine into initial state */ @@ -778,8 +779,6 @@ static int iscsi_sw_tcp_conn_set_param(struct iscsi_cls_conn *cls_conn, mutex_unlock(&tcp_sw_conn->sock_lock); return -ENOTCONN; } - tcp_sw_conn->sendpage = conn->datadgst_en ? - sock_no_sendpage : tcp_sw_conn->sock->ops->sendpage; mutex_unlock(&tcp_sw_conn->sock_lock); break; case ISCSI_PARAM_MAX_R2T: diff --git a/drivers/scsi/iscsi_tcp.h b/drivers/scsi/iscsi_tcp.h index 68e14a344904..d6ec08d7eb63 100644 --- a/drivers/scsi/iscsi_tcp.h +++ b/drivers/scsi/iscsi_tcp.h @@ -48,7 +48,7 @@ struct iscsi_sw_tcp_conn { uint32_t sendpage_failures_cnt; uint32_t discontiguous_hdr_cnt; - ssize_t (*sendpage)(struct socket *, struct page *, int, size_t, int); + bool can_splice_to_tcp; }; struct iscsi_sw_tcp_host { diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c index 26dc8ed3045b..c7d58e41ac3b 100644 --- a/drivers/target/iscsi/iscsi_target_util.c +++ b/drivers/target/iscsi/iscsi_target_util.c @@ -1078,6 +1078,8 @@ int iscsit_fe_sendpage_sg( struct iscsit_conn *conn) { struct scatterlist *sg = cmd->first_data_sg; + struct bio_vec bvec; + struct msghdr msghdr = { .msg_flags = MSG_SPLICE_PAGES, }; struct kvec iov; u32 tx_hdr_size, data_len; u32 offset = cmd->first_data_sg_off; @@ -1121,17 +1123,17 @@ int iscsit_fe_sendpage_sg( u32 space = (sg->length - offset); u32 sub_len = min_t(u32, data_len, space); send_pg: - tx_sent = conn->sock->ops->sendpage(conn->sock, - sg_page(sg), sg->offset + offset, sub_len, 0); + bvec_set_page(&bvec, sg_page(sg), sub_len, sg->offset + offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, sub_len); + + tx_sent = conn->sock->ops->sendmsg(conn->sock, &msghdr, sub_len); if (tx_sent != sub_len) { if (tx_sent == -EAGAIN) { - pr_err("tcp_sendpage() returned" - " -EAGAIN\n"); + pr_err("sendmsg/splice returned -EAGAIN\n"); goto send_pg; } - pr_err("tcp_sendpage() failure: %d\n", - tx_sent); + pr_err("sendmsg/splice failure: %d\n", tx_sent); return -1; } From patchwork Fri Mar 31 16:09:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71D4CC77B62 for ; Fri, 31 Mar 2023 16:11:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF7FD6B00AC; Fri, 31 Mar 2023 12:11:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A81B86B00AD; Fri, 31 Mar 2023 12:11:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94AF46B00AE; Fri, 31 Mar 2023 12:11:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 84C176B00AC for ; Fri, 31 Mar 2023 12:11:21 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 62812ACA30 for ; Fri, 31 Mar 2023 16:11:21 +0000 (UTC) X-FDA: 80629683162.09.750F3BD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 9D2F016000A for ; Fri, 31 Mar 2023 16:11:19 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bxWfp94r; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279079; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0V0VYuNRGUsctVySLnEH8GinGQ4l5nQcc0Sx6Q5aq0A=; b=HkxWAtGit3Yb+cMeZdn+XvimKd5pvWuUeJGQqkWkb64c3EAD8FGlbUxrIv67q5bCAxliaf yrN+RahX9vQuhSQAeE9POn5cPbwGhnVUC4soffWXG/ikCVELFxi2zYc0P1HTMgR1SlwHoP lFCbjVMcBysIjsxATh3y6ajde5laTbA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bxWfp94r; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279079; a=rsa-sha256; cv=none; b=So9paXbESyOUK4H3GAUQRk3HDjFMrkiP8pdzCdaAJAVkqc847LxBmOzYt88LZHmHh64Yh9 ChLhwAa37mwGPV0ZEOcwvZkuUkDhbWtI4OvIIiF0/O520Ht/YPdhWk/xS3GpNBa+uJu23D CxY3uCpt/xDd6pLsCRhCiNykEU89kxE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0V0VYuNRGUsctVySLnEH8GinGQ4l5nQcc0Sx6Q5aq0A=; b=bxWfp94rno9l79qgbPG5Jh7EhXksktzGDsyayEQdljpGwvw1J5xK82PQFjHduWHA+It26Z j7vvc30966fhUd2QBFNpmdSIhewqp/9GVGgU4VAdN6WVrzXJOo0BjjMO5gSU1xRpDRrpz+ tLaggZRVQsoFQoH0509zGFR8HRy0afA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-335-JJSK7rSYNzmbFtG3J_VYoA-1; Fri, 31 Mar 2023 12:11:15 -0400 X-MC-Unique: JJSK7rSYNzmbFtG3J_VYoA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6C1CF29ABA1F; Fri, 31 Mar 2023 16:11:14 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 42ABB40B3EDB; Fri, 31 Mar 2023 16:11:12 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Martin K. Petersen" , linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [PATCH v3 41/55] iscsi: Assume "sendpage" is okay in iscsi_tcp_segment_map() Date: Fri, 31 Mar 2023 17:09:00 +0100 Message-Id: <20230331160914.1608208-42-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 5pxg3ch3fb6zpfox8gghn813n5ghidn3 X-Rspamd-Queue-Id: 9D2F016000A X-HE-Tag: 1680279079-780442 X-HE-Meta: U2FsdGVkX19FdLpBCaxAUkpgUYFzPs1VS9+WPYkjM2wGIeGZPk5+TTBvW4c6jzrkJHSoutdloo5jYaZ6Cv1VdCr8LDdl6O/2CQ6yYR0nC8saBlFh92GcPUsJherSPexgfKYG92p9TxFv8eY1CZyFW15+0OG4NhhJoS7DM3WEXCO58BhBFVYjagK6l0X4eXmxoxbAx/8pDKSzd5yi2DjBXzHj/pw2Jma4Ke0s0e7HP6fgtqfK5mJDRhsG5DZY9W+7ZpUYb+40AA+Zs1J5Y50p0rpjBiFxRBH0zPy7QHi9xMifgoinAgbaCrtH77mo+APCXPfnix9cCQTswgq0qz9TuB/FG+mTknEyvs8FE/uFHaGF9aHwUh8FWncCYJ6JBDpwOEPgXTL4kfksetp/M+M03h1Kz8RWrfDBgPQaUGPZWEP8fTZVB1TJ1mnVDMT2/FsBQdxzKhTEwQOjJbpEsWlgDq4m2i4WjYhy+EydTFdQgLwNEd1cmO9AaCbtMluAee4WeZH02OzrUVl3I3jHTwsUqVGJgazHNYAR23/aaa3zPIiFRjQekpLCpQj/uHT9WsmzGy4AGqqt65i2QOwhFrA2XKneSfBMFNz1KFLPyqT1SeNRRsTWFOm9gXgsStm7acqEVwRX20AjAP1wFN2s2brmu559RSr5KHbp9sImWNrHJrKGMLkJ4f9Laqd8+fmr2VZG5w2jV++Gx+A0bpQXD57QdjEturq7qukQKP4223Ja+aYpBS4evg4H1QCYWSEdRSBJ4UMgArJHsK3SgnTO+TSykb/q7DhxdFLUzLaFFoH2rjmncppic2BvWiukajn9jd4tqAaAtKpjR0QyF+na8jygkYT+SEwrp6hqz4lWrvAT1IMJkJl2q70WYH4rQ+QXX/pJLIScK4kxs3n357rRlGJO/YFJMDXq/H314384HEhHBk/kFVOIILLGGMV3oEN9jziqlXeTugM1ugH+lTEogRT sh86LW3X mHKOrE0E3d9EDrf+SLvwS4RtFr0e3CAbu6GDZesSA9C9KZO6H83dMQrGrHSQse3y/95L3d269gHGTPLIs3F0nvXdxRFQ3DOOs5yqp8/bjIYrVSQl4j6qv12D6RkvcGviKeTysT6mTl0Is8gOCXyr86YpW3uE7Vn7rGduzGG0+y/UJWSIO01dfSl6W9CzJNUcRGi7Q/FSowqLII9lzjy3dzB2Fe2zTqVlnbArKNLnANoTuIZMTKt9LdKpBVy/5DspNqODo994BRzn7g9isRMUnnA6BRG/Ef+JULwdinZgsl+GEfEZZlKWJVnfZM4vl+mVQPvgOoKmXmb1qfju9Kps4CzE7hgAs1Ypiw/3wz6LlIlRv+G9yKv1LNm2Z5nW4MsqgpRV/jVQohAbsl2GBy8c9TK9SIbu9mUey44Z2SBwyORE53NakF/FWKZF6F39nMGHBcjOtok9eJ7avPbp+bwLqAbPfjg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As iscsi is now using sendmsg() with MSG_SPLICE_PAGES rather than sendpage, assume that sendpage_ok() will return true in iscsi_tcp_segment_map() and leave it to TCP to copy the data if not. Signed-off-by: David Howells cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/scsi/libiscsi_tcp.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c index c182aa83f2c9..07ba0d864820 100644 --- a/drivers/scsi/libiscsi_tcp.c +++ b/drivers/scsi/libiscsi_tcp.c @@ -128,18 +128,11 @@ static void iscsi_tcp_segment_map(struct iscsi_segment *segment, int recv) * coalescing neighboring slab objects into a single frag which * triggers one of hardened usercopy checks. */ - if (!recv && sendpage_ok(sg_page(sg))) + if (!recv) return; - if (recv) { - segment->atomic_mapped = true; - segment->sg_mapped = kmap_atomic(sg_page(sg)); - } else { - segment->atomic_mapped = false; - /* the xmit path can sleep with the page mapped so use kmap */ - segment->sg_mapped = kmap(sg_page(sg)); - } - + segment->atomic_mapped = true; + segment->sg_mapped = kmap_atomic(sg_page(sg)); segment->data = segment->sg_mapped + sg->offset + segment->sg_offset; } From patchwork Fri Mar 31 16:09:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3155DC77B62 for ; Fri, 31 Mar 2023 16:11:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7E786B00AE; Fri, 31 Mar 2023 12:11:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2E476B00B0; Fri, 31 Mar 2023 12:11:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD1CE6B00B1; Fri, 31 Mar 2023 12:11:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97B446B00AE for ; Fri, 31 Mar 2023 12:11:25 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 66FDFA0DC8 for ; Fri, 31 Mar 2023 16:11:25 +0000 (UTC) X-FDA: 80629683330.19.0531B93 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 9D26FC0020 for ; Fri, 31 Mar 2023 16:11:23 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FQ66NZIl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=wfRDuSHYAeth4U+fpzvLCOHvXX4/0zLt2//pe9e6xHCaAcykGLZkjZlRrOxdPtVvLDnvEJ fOLpmt/BKPhDSFNd8xZouXNbtmH7bHiACOyDl+/wV13qdJMJPfOn+LVBlz7BKht+ZS0zcl bZ2/T7uXLHSGQCy37fTWk4l1YP+V2UI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FQ66NZIl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279083; a=rsa-sha256; cv=none; b=UwLngrx27JZEBn916WNLtdXmOOlw0Yi1p7vma92tUqP2nYuVbC6vj8FUQ8udj+kHDpFyew xsACm4th46ocQYhvX0azY59a3Mg469J9agI2LKH4vTa//LoOvBZ8s40bt8bG9DcMdVjiMG 3MFuC4RR7umevqn3/L6kOyu0rooKNc8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279083; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=FQ66NZIlZ+FQKnFoJgDvWkq6jR1O5hJz/24+LFPSZSQTtJTXkr2DUxN80j7jUd/EbXqaGf IKJpu9QVGLyq4//x/sKxsIYoyYWQD/KH+aybQVvZhlUNM+hAnFbjcd1GEfTTGgohytP08t LEHXK11bN2alnIhFUjW3s8cKPl4E02k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-340-Fj3IfQ9LOuGHr1f7veDhWA-1; Fri, 31 Mar 2023 12:11:18 -0400 X-MC-Unique: Fj3IfQ9LOuGHr1f7veDhWA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30B60886065; Fri, 31 Mar 2023 16:11:17 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 072FE202701E; Fri, 31 Mar 2023 16:11:14 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH v3 42/55] tcp_bpf: Make tcp_bpf_sendpage() go through tcp_bpf_sendmsg(MSG_SPLICE_PAGES) Date: Fri, 31 Mar 2023 17:09:01 +0100 Message-Id: <20230331160914.1608208-43-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9D26FC0020 X-Stat-Signature: pb4dsi1hko95xeswzjicsrs4pcyn59ia X-HE-Tag: 1680279083-554968 X-HE-Meta: U2FsdGVkX18I36AQk/Hxr4JQ81sY/Q9E8UWbo5Tqjb/ipwERp8IRWg3BDiwFgS7dMeb4vEpYNhzVu590XdTw5aw2wzBMiflY06z7XvCOnvKShEsVmwANIZWcfIgHgb6MxDsmgADNKYiyKNbiEihpbIs2Wl1YE1AK07VuOqzdwIq9LMck9zNP62jqMS+4farz+dPBkenT/0J6on0yE+0aGchT8iFn+XJPNdv0A105W38lszZX9gNL9a/SPO9dYyJ/HReuYfaJFZySQ5o3szNWdIfODN3V79x80D/2vc1V7fT1GVPC4ZnizVkNQD7nvjP3aUXaYyfcYSRn5vbd+miC3xOORudclXWMNJTs2keIAL3DAK0e59eDnF/B92H3PSkqFtMZR7tokUaR0gRUnYpSR/ZCeC5+wuFeCY6SiEE15FhC+mzOwHTJ7GA+lP0ZrFrPWuYCNU+XD3tVNn3BEdvaHYWfP3/NrDpahM00jsQLkDbD6oOKIDFgXp5GP9+dCDY0MNrdxi9Obo6jYqxcc6R2d7yZLseSYmiA/Sx4JWu8bSOdT0rW7TqdRS3YkvAubJ8fH5HFV1MfzgWZf/TfZGi0Jki/uRBBPwAJxabyvc52YupfYYS8H7IaQ5StmL/8Gh7KbVGvDsjewTN1bEAVfFNHc9GnO4NoamrCz4UmGPafch2RrsWWjuhFNql4lrCblki9sRQQVJ0gy1nL9vCixUR6KBE/rimSvLUVdiw6znypOAJ81t1NxyBS8pYFSArUAdhD9fVjrlFTfhfXjmldOmb+om4R1DgFOUG5+8LaBe0h4yQqJWevnd0/GZD4+5kuYUUVgDK7MZyWaLkdsUepqQuw3J8U+jv67M+f2FV02R3w4Y/ViZ9a6XrQ4YckOeEV/OJV0/1E0aZ8D6e2iQqxUJopsPvx6vbpjI5niaCGgEtOG2GVW7Cc5+ZEnQJAgfNJBktZP4jYXRqD/fEP8SHtSBu yAVCZ03E W9SC5Xsgug+NXqg38eSJM/+zxKYK9k1n9vIEsFfUiBkUn64kMtvakk6UIsoXxUy0umIb10Z/LRSByZ9pvvFrIvKSsHdnOlP254ldeEHUUB3x+sHnWcJHdTHAx+TBdPGhUdhVYUeFCEAIdtORu++5HCeeagHyhH2fWlo4MbKKtnupUK0gJRa3O27VWulitqhOUuuvPyNhjurJR4RXVk//KBzy3eJE8tCx6FD3pUKq61aYI15TEnPMSjpUN9Z2Pa0QSjyTCKHYpM7DECKgYjKcX8KQxE+HxP9mAvRzxBYMBiJRSeYs7rsuXKLNoctlxscuCBd5hvYHT0wagwdBbHecmGncuk0KUrppvAe8wROyNnUnpkfSp54g+H9iZSVaJr3hDp1umMAdxZjSvDGPrWkLhbHGDxdm7j8mXQvRb86ZnOftDKs4djQryWPAerYVR6RmFmLZzycYewsSQnD+6mqR5p0VktLgWTWlwwYExTOQ0vabXtbiBNdct4EVa/VLPFbEJIeCWhW0ZziQT4otUWk60Lg97uJBqBzXQEz7+hur3AmrKpAz3eX5i3x7yQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Translate tcp_bpf_sendpage() calls to tcp_bpf_sendmsg(MSG_SPLICE_PAGES). Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: bpf@vger.kernel.org cc: netdev@vger.kernel.org --- net/ipv4/tcp_bpf.c | 49 +++++++++------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 7f17134637eb..de37a4372437 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -485,49 +485,18 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) static int tcp_bpf_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct sk_msg tmp, *msg = NULL; - int err = 0, copied = 0; - struct sk_psock *psock; - bool enospc = false; - - psock = sk_psock_get(sk); - if (unlikely(!psock)) - return tcp_sendpage(sk, page, offset, size, flags); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - lock_sock(sk); - if (psock->cork) { - msg = psock->cork; - } else { - msg = &tmp; - sk_msg_init(msg); - } + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - /* Catch case where ring is full and sendpage is stalled. */ - if (unlikely(sk_msg_full(msg))) - goto out_err; - - sk_msg_page_add(msg, page, size, offset); - sk_mem_charge(sk, size); - copied = size; - if (sk_msg_full(msg)) - enospc = true; - if (psock->cork_bytes) { - if (size > psock->cork_bytes) - psock->cork_bytes = 0; - else - psock->cork_bytes -= size; - if (psock->cork_bytes && !enospc) - goto out_err; - /* All cork bytes are accounted, rerun the prog. */ - psock->eval = __SK_NONE; - psock->cork_bytes = 0; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = tcp_bpf_send_verdict(sk, psock, msg, &copied, flags); -out_err: - release_sock(sk); - sk_psock_put(sk, psock); - return copied ? copied : err; + return tcp_bpf_sendmsg(sk, &msg, size); } enum { From patchwork Fri Mar 31 16:09:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 285A6C76196 for ; Fri, 31 Mar 2023 16:11:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C091A6B00B0; Fri, 31 Mar 2023 12:11:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B92626B00B1; Fri, 31 Mar 2023 12:11:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5A2C6B00B2; Fri, 31 Mar 2023 12:11:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 95DD16B00B0 for ; Fri, 31 Mar 2023 12:11:29 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 77F52160476 for ; Fri, 31 Mar 2023 16:11:29 +0000 (UTC) X-FDA: 80629683498.28.163E77A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id C658D14001F for ; Fri, 31 Mar 2023 16:11:27 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=axBOxic8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lLisHPaWAEED9gZclxyhbGIkHvgJDoUiXuyeeqOKByg=; b=klNZJD2zOc1EFQeZj/mXAlPN2XA9k0bupT1q20x4Dv32qyRuiTZyYQKbT61+B/LRGwbl8b QUjQpYLicHjbgWy875CbyhhHpeYfecRYwecZHhFxsFzDPLbwzReQNDRjfI4h9vHBZEsQXz JG1Xodd7SQsJA1zURqQlULNNLcMAl9Y= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=axBOxic8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279087; a=rsa-sha256; cv=none; b=DZ2zYHT1nvYzRjyQXfiv/MJSr8DO/tJ4JDjSlmXekjbhEAsRJkjsJMVY8uCjItwQHFfu7Z m8r4xP+bT5WD4qV7mocBXCj7S7ZphtzfxxhhBwboR3Aowqr1WDIyUvrPLgaEmeuwy+BfWI 3IDAMxMxJVb00+Epbb9YctVh2lYunsA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279087; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lLisHPaWAEED9gZclxyhbGIkHvgJDoUiXuyeeqOKByg=; b=axBOxic8u49+A9338mQoTPS0ZIkLS2cLOA852jBuGLQ3oCaywDEK/OU31xgVsFjSIJj7Hg fnaAK0HnfVeu61kPMOoIIVX2F73m31+DyiYxjtAgK+SOLDys8DrgcHzR0KlMtUJ2OJDZ6w s9fbyldlMt3uXPRGjpWP2vPAy/oAjSg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-626-FA2oWpmKPhmcnajIovqg2A-1; Fri, 31 Mar 2023 12:11:20 -0400 X-MC-Unique: FA2oWpmKPhmcnajIovqg2A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C7A3129ABA1C; Fri, 31 Mar 2023 16:11:19 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id E4EDD2027040; Fri, 31 Mar 2023 16:11:17 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 43/55] net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock() Date: Fri, 31 Mar 2023 17:09:02 +0100 Message-Id: <20230331160914.1608208-44-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C658D14001F X-Stat-Signature: 1tjdhtb9erqkb4b47o8apgax39i18kr1 X-HE-Tag: 1680279087-487082 X-HE-Meta: U2FsdGVkX1+FrCBKbQfox+E4ltpk0uOcQAY0y0HzEbDEpOS9zVdkzhHKO8ioYdjL9AnEY7wFaZdAd0QVZNotvZkVDMp6PwPeg9XCTf4G0qDZKBQ/EjDbbeikNoC7YY6r2V/4zmgh+kWyJcuzl3Nr4n/gT0CLZlQcglR5CjSH9sgHFNOMcIUwb4FjbkmdAp29swcUGsWKhOFoVXb3mFrcw5Ff2h2T4bWJQ5VuZsipAAFwOfdk//3/hQbRk/KV8NnOMa+xFeWo6D+JGlVi7kGX8MPhoDIxdvuoDSpbdVV2djj/rxB03+xFYLAuDAvkdylxOcFeb/uBlDyZahIeSfqXBDAxTUOzKpq3C/NjXRRgBfG/PTHWX7/3SZUUSBn3X9e/piZhVPs1/CUaDxG1hVn50LkiTKQlmErcNIaIOTo9DcJf7s4d7z2OYXdenudmHyWYe0c932+luH3P47baZP1+WF7oaXkC7qJEoY7+1N46w9wSX+wJdtDZWg88D06NAqY0or3sEj+przt3yBWzxF+Nb8Mdar4OBnAD4N5K0SfyXm7vlt9456iPpquxmSezMDTuWHbG2Ur5WXGsPsqaOQ+DZiY6Kb2fvxICYafLOnYufZP9P3ROpMajexRYhqWrzIGTitqpbeUkdrWPgaAYMgExG9fc10eRnzzK2mBcjfASJTv8x0LY6OzUupowguw5EqVq4qVGLylsIThuikAOPuQrBaRH9sQblxeg6MXUU7MuomzF+GvumckAhsjKy4A8D3413RmGfSR9gFp4tSTGwEXYqFHtyV/jC6zodYtaMj4Ig+rWkGTp++XxSdnxfzN86gFV9rHATS/G7RYPmIKUXFq0gRTxcKT+e3jfz4TI9SBl8KQcgUlyKm4qSiyP13VhHjPTcE/dw9RD/RAJP5EHRmaz5Fc6S6HjlDAP8hLmKoHzXFBn7nhddAbik6N/zBOE+v7l8SF8VBRzIvLv2hIrBVS RC8L8T5c 2EFJTvWm0ub6KElmxYTbriOrh2x13x3yQZ6l9ijjkf/C/V73zSw1lNmH3r3s9DG014dVp7oAex8xYpWkpyRpOPDGlOcqgj6w60e5UXOzFhHCIUc5w76BLPOgbot0FioHJf9TcLh8ga0/grEDcffNVDPLW6/Qm0IlloSlSPe2bHDVzKRxoPZlYXi7pcGkT+YGXVwFaVfsd8jREGhknSCHVJ5Tt6fklZrU8+ju51FknqeKbN108anPYiyQninbsmp3BXTK2MaSGoU59/gm4OBTucO5GXsiQsInqUMa8qDyaUzS9HhzKCAqcQR5NGAbNz9ClGSFfbfbaGxXPBKBWTwuOkpIK6C+4AkB04VtT/581lZf7z9+zUOt7Ht8JhuqWRHbQhXY43dnujQ9r9a8u2CBd1Tv7LJ1to5De5MCYTdsZO2khaGWMX2iRX2CXucRJxV6zO+ee X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage in skb_send_sock(). This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Note that this could perhaps be improved to fill out a bvec array with all the frags and then make a single sendmsg call, possibly sticking the header on the front also. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/core/skbuff.c | 49 ++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 0506e4cf1ed9..3693b3526d33 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2919,32 +2919,32 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset, } EXPORT_SYMBOL_GPL(skb_splice_bits); -static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size) +static int sendmsg_locked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; + size_t size = msg_data_left(msg); if (!sock) return -EINVAL; - return kernel_sendmsg(sock, msg, vec, num, size); + + if (!sock->ops->sendmsg_locked) + return sock_no_sendmsg_locked(sk, msg, size); + + return sock->ops->sendmsg_locked(sk, msg, size); } -static int sendpage_unlocked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; if (!sock) return -EINVAL; - return kernel_sendpage(sock, page, offset, size, flags); + return sock_sendmsg(sock, msg); } -typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size); -typedef int (*sendpage_func)(struct sock *sk, struct page *page, int offset, - size_t size, int flags); +typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg); static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, - int len, sendmsg_func sendmsg, sendpage_func sendpage) + int len, sendmsg_func sendmsg) { unsigned int orig_len = len; struct sk_buff *head = skb; @@ -2964,8 +2964,9 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, memset(&msg, 0, sizeof(msg)); msg.msg_flags = MSG_DONTWAIT; - ret = INDIRECT_CALL_2(sendmsg, kernel_sendmsg_locked, - sendmsg_unlocked, sk, &msg, &kv, 1, slen); + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &kv, 1, slen); + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -2996,11 +2997,17 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, slen = min_t(size_t, len, skb_frag_size(frag) - offset); while (slen) { - ret = INDIRECT_CALL_2(sendpage, kernel_sendpage_locked, - sendpage_unlocked, sk, - skb_frag_page(frag), - skb_frag_off(frag) + offset, - slen, MSG_DONTWAIT); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT, + }; + + bvec_set_page(&bvec, skb_frag_page(frag), slen, + skb_frag_off(frag) + offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, slen); + + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3037,16 +3044,14 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int skb_send_sock_locked(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, kernel_sendmsg_locked, - kernel_sendpage_locked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_locked); } EXPORT_SYMBOL_GPL(skb_send_sock_locked); /* Send skb data on a socket. Socket must be unlocked. */ int skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked, - sendpage_unlocked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked); } /** From patchwork Fri Mar 31 16:09:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE2FBC77B6C for ; Fri, 31 Mar 2023 16:11:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6B576B00B1; Fri, 31 Mar 2023 12:11:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCBCB6B00B2; Fri, 31 Mar 2023 12:11:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1DC26B00B3; Fri, 31 Mar 2023 12:11:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A05F66B00B1 for ; Fri, 31 Mar 2023 12:11:30 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7D78A1A0D0B for ; Fri, 31 Mar 2023 16:11:30 +0000 (UTC) X-FDA: 80629683540.20.5606C91 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id B038040018 for ; Fri, 31 Mar 2023 16:11:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UNrdfrp6; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=taClZ7tV+6NGiC2MYzyqiyjdYhpNZJPUqY1OQ7XAIPM=; b=aOzgzNGRCVlxoE/ciPRjMmbuak+1Kijxj1QXDEYBWzsknpKH5yIHOxNbR//choT5C1XbXA 7ZHbEDcZbrM+iKZn1eXJ4rzEZrOWfZ+4yWtrlf6khHdGV9ioy91PQiW9MbBbYr41GvuK/+ CAy+H5fgIWBtrA/MvqDX54bQxyydLqs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UNrdfrp6; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279088; a=rsa-sha256; cv=none; b=PWHCyc2jEw2hQkxALG7+tJprXr0GsTMsFdlffoqFJz8HX6Wsyzt71VPqQakLJofAn5fXts XWJdOAQSRCO/iR65jdWPblgUawyQ38T0jWA4jaHw3MkgpGKjKMpgdXspIlJRAR0BZuH7Mi HTKrWPWOVacDKg7FQ4kpxObKLoO3cfg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=taClZ7tV+6NGiC2MYzyqiyjdYhpNZJPUqY1OQ7XAIPM=; b=UNrdfrp6Yj/iiWYW8FfJx2Ym+h0wYHDzZm4TZJcMduY/JBIR/Ut7y13M2jxm7MwtwmGP+h Qx0YaUzrzw/3FinQlk1+nR5oqb7/p+UHq+fjz3WA5djJFFv/kCMPEAoGjwlBT8oHaBDO9e Nll8nnxJOp8wqfBpkZrrkXukF6NHgBo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-321-c9EO5cpKMmGD23YMrtD9Yg-1; Fri, 31 Mar 2023 12:11:23 -0400 X-MC-Unique: c9EO5cpKMmGD23YMrtD9Yg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 73F4B811E7C; Fri, 31 Mar 2023 16:11:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 628494020C82; Fri, 31 Mar 2023 16:11:20 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [PATCH v3 44/55] algif: Remove hash_sendpage*() Date: Fri, 31 Mar 2023 17:09:03 +0100 Message-Id: <20230331160914.1608208-45-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: B038040018 X-Stat-Signature: uhn4knnock4unrxriufjb8neen7xypj1 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680279088-208624 X-HE-Meta: U2FsdGVkX18ZUxAhAKS7MEMqXyR1lK1DfyybtpixQNAxCAEBQUDINXjTPckXIbZI9a2tY6enujUaWZ3tYgiJli5BUXVctpxPAaLpps6YbRq0JMM4AIoseqPJnwoRROZW7Oo70QdNDLDtAH6E9iONWUiMWrwM9deKOXxOEXV8cBSLbrrfVIQrYXklKI7oqH8vYUdi+vh+0g57P/BLnz5OUJAM9OK0XZ9NTGP1y7pFwYRtBbiMsHBdJ/k90GEc/0eDJal4xzfl1iPQ3p6iqsbSGGCFWY9jXzyhmF5tz/5rXdwtUO+fdqEOB0FewLYbg3Ys2DdCIOZK/dwbEKGRGEpUu7W2ecvedUQo0tCkST07Cm6lEpbkn4KpH36GrKc04ZRlB3HYckIZUPlH6VhoLua3hSKAr9nF6OSEskXDwdh9Km0eHH3JDGLUk80DteFDUfl2QANrys09KsrOalQYuLCsJ+wFksb9ItP/PGcqy2ubRTnVwqU5wzptxOf7SLbUjZ1YgMPL9w8yKwpNKTUrrf5xeyQnrVLXXgs74cig9kyveLD6/pJZPavsPtoJ7fYn/LrlG1Q+vmkHvL57sYGFGLVrl8Tkp1KdR8A5coiWNkGACadMcMHg71vPmN7+LyfOuDApYyG9GW3mg1CAhU8wAKlNBTcbC2vCGmlyYVR7vbDFBC65pFQcNmYXsqvGkw3TqQ0n1mChh9mlLVe9NsQq/VNeZSAEXj9P495JwRW8o+fvhBP7vVuR5Gn4bBxfcMORUpG5dUnlccsRwAHbqe6q/9eBwbqCWLgmiBT6IFMMcJHTNyCAcbgy2xPc725GYHAHs5HtakS54uy5A5yt0lp0sOQM0cZByNYHVIfuBBYyFwyYirzbVBIwAqeW2JQmfVHjkFauaP8u3dsKR3Or2OhLLQghO6HNFNt4ARrBzUvQydRXsx4ZZ4kk4eNWeSvmGl6fGHennMh8LrvTWVhHbk4a6t3 ScM/8GP1 gJVp07QxFAPK1aMKCThqhKOEIvFHhw7cGTwzwYqYun3Nz8Z9m0FAT8BX68N0TiRrwXfybpQKdO8O6KiGMyBHhtvYxOn7HMP86tzR0vBKEm6p+Y9P4JVqg+KbXwihjPPSLt8E4Nly08CKkabxLlplx7MG7lu0xebL+Z8XfUN3bwbThfWbJUxSp79coMcsh/pz23LHMiplD9b0/LusOWhtMgq8QQwVBD9/saCAyJPLhrTWO4s73UyC09/qAEqqeYwzkCVFw+p56kLmetPo4jr5FkvQVoxn5ssC2kDdPB2Bn0MGvwKpOyXaGEoiGpC2/qJbSuz67tRTbmmpFt/GQzyP3bW7mXGpERLdEVJJI8n4U6uCdl8Oy0qCqS0m752NSciACdqxd2NtnfOsXrPAQWNP/Kx5BR5q+BkBNt0f8akYmPVCjBCPRNW8i33SBUSyVuMmGEy0E5fH7FONfxi4gDUl4X8YCzXc3ejv2qYA0XsiXrSwJDLLyBsTavql7LA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove hash_sendpage*().. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/algif_hash.c | 66 --------------------------------------------- 1 file changed, 66 deletions(-) diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index b89c2c50cecc..dc6c45637b2d 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -162,58 +162,6 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, goto unlock; } -static ssize_t hash_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct hash_ctx *ctx = ask->private; - int err; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - sg_init_table(ctx->sgl.sgl, 1); - sg_set_page(ctx->sgl.sgl, page, size, offset); - - if (!(flags & MSG_MORE)) { - err = hash_alloc_result(sk, ctx); - if (err) - goto unlock; - } else if (!ctx->more) - hash_free_result(sk, ctx); - - ahash_request_set_crypt(&ctx->req, ctx->sgl.sgl, ctx->result, size); - - if (!(flags & MSG_MORE)) { - if (ctx->more) - err = crypto_ahash_finup(&ctx->req); - else - err = crypto_ahash_digest(&ctx->req); - } else { - if (!ctx->more) { - err = crypto_ahash_init(&ctx->req); - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - } - - err = crypto_ahash_update(&ctx->req); - } - - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - - ctx->more = flags & MSG_MORE; - -unlock: - release_sock(sk); - - return err ?: size; -} - static int hash_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, int flags) { @@ -318,7 +266,6 @@ static struct proto_ops algif_hash_ops = { .release = af_alg_release, .sendmsg = hash_sendmsg, - .sendpage = hash_sendpage, .recvmsg = hash_recvmsg, .accept = hash_accept, }; @@ -370,18 +317,6 @@ static int hash_sendmsg_nokey(struct socket *sock, struct msghdr *msg, return hash_sendmsg(sock, msg, size); } -static ssize_t hash_sendpage_nokey(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - int err; - - err = hash_check_key(sock); - if (err) - return err; - - return hash_sendpage(sock, page, offset, size, flags); -} - static int hash_recvmsg_nokey(struct socket *sock, struct msghdr *msg, size_t ignored, int flags) { @@ -420,7 +355,6 @@ static struct proto_ops algif_hash_ops_nokey = { .release = af_alg_release, .sendmsg = hash_sendmsg_nokey, - .sendpage = hash_sendpage_nokey, .recvmsg = hash_recvmsg_nokey, .accept = hash_accept_nokey, }; From patchwork Fri Mar 31 16:09:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8804C761A6 for ; Fri, 31 Mar 2023 16:11:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C09A6B00B3; Fri, 31 Mar 2023 12:11:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FACE6B00B4; Fri, 31 Mar 2023 12:11:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C23E6B00B5; Fri, 31 Mar 2023 12:11:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 69CD06B00B3 for ; Fri, 31 Mar 2023 12:11:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 245EBAC25E for ; Fri, 31 Mar 2023 16:11:32 +0000 (UTC) X-FDA: 80629683624.02.A671044 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 76BAB8002D for ; Fri, 31 Mar 2023 16:11:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jPjcc3i1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=CRURXkmllA2xL9ieN6VZi0pqT7oVjlTYx8+C8SGfzXvC/tvEPkwAE89x/IJHEhtuZ+9v/A vE57bPuGNg/RTnwSiU6lIqiB4MiXf59CI/U99qIwEECGtaOuZHxlEILZO+X4bdp9NVj57e 2HWUK7JGS9U2GWcePpeLbkfvdufriFA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jPjcc3i1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279090; a=rsa-sha256; cv=none; b=NkzDMQf0i8kbquewqvKyS5i8vMamqRmaUL1Pg4pIVok+qYI3bLecGJ9qv0Azml/PG08EQt wNUbJZjAWqtlK0ysT8tWWcXDJkQ1Jr5i68i9R6y0k2MgCVwmIiBRynAmvblukEKqHSMDqF bQ1RwqeprJgeWrM5hBWTZv8xTDHQ4ew= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=jPjcc3i1evQtTAfzlejbvELJyXK2x1P/+PmqqsAPppTlzdbex5aGg3RgdBvgEy73fqfNXE glt1gDYzcUPgvkYTnF2g60vEQ8LGL5woeNDlp0gW/HOCbCdMtYBf1PWA+AA5IKcYKodUZT rsWtxjSxHCey4Pgk+AkE2s0Z73mcn0U= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-186-Zs9E43zDOwakJiLhI1Mt6A-1; Fri, 31 Mar 2023 12:11:26 -0400 X-MC-Unique: Zs9E43zDOwakJiLhI1Mt6A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A289F3C0F383; Fri, 31 Mar 2023 16:11:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D14E2166B33; Fri, 31 Mar 2023 16:11:23 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [PATCH v3 45/55] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Fri, 31 Mar 2023 17:09:04 +0100 Message-Id: <20230331160914.1608208-46-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 76BAB8002D X-Stat-Signature: zm1ertntkbki98ek4tsz6tt4ng9g3gmm X-HE-Tag: 1680279090-498620 X-HE-Meta: U2FsdGVkX1+8DcCKZm8h4pwZ1MWlS641aPk0u3NSkcwIIFz1q7bNqT/VCy/JPEnrvFGMejIETwQPOkdWuSRw5MXo7ZtkdwBBYni3TRbrroTLATrlzoXdFdyHO6D19G9V7uJ91oVNK3CKkwoGFUJ51Q/vKse7M+Gl3FkdhOYW7+Sp/6YV2b9K7yvbAiZ+hHdTDyX/Z9Cqdp284RcGwPsffV6gO97/Hb7fsi2mJd88S//ZIqug7GNnBL1OBT5ZZ65bc326kZB+93Cx6pTg6uhgLr74VSqOtNqtkeOawGxpQnLhOiyo3CJHFwW17HfdmIzPsETLgCX6rO8C8rlIwOoFwjJjwGia7zd8GbF38lxfqj0ifiB/x163amFWGO8c2RaTnmDcGkjCwLLMHFZTuJKShnHV3Bmqb0b73m6xncd9xtWxj79+wk+f8Y7F5YwzD8z1XN17oyerErc56LaymUX8yo4gos2d9XZp0afjz7eM5Wmbmab5KzOM5mrHfhAHJCqrf+4pQsY0H1rp6PnEJKndY8qWc5Swlthk2aRvI1x+SEBhHyZZMh2nqpruK4MekNF0ipHDKYVhj1FF/J23kp0HlfLBJNaizPRbUGII3DRvnOa6HsNg49RbG/HjAzFYPtEnoJ0vfHJU4luIDea2xdb7EQ23U+B26n0dlRCdFEi/dpN95DsyWevEUjh1KH8ZfN/4MORqnXHTzkeZd4XNQQu77ALvBUAbYG+0j5G2fLboPg43taPa419l56W4xywN38dwrJca8qUTgHN2E3JgUbn+pbKK1I87LlaVV8hmgDa07FuEpd20fLaHxlQbdBCN/2laOjjaxVtEQ7KNm7WapX9HU5lQDkB/BTaSZcH5pbmgzVLtn4kF6Wb43FQAv/S9NdmqcZr4iqZx0oHF9IkNDuRJ5w9NZlQBq5zHjoS6pLFD/U9hL4YUfiwYPn3cBcOo7eLufldrPLLj0fytqBFkPwY CF0y+oUQ tR6mxRl7jF72A+ff2lV8OreheFbBF986tP69hwvL4G+5FVH6bwr5AmyAXrCjs8pWzLLJIGy9REKi0+xF3wEnOqiE8Hi1WgB27vwriXLfcVUWPA+nxh+09YbXn5U7agq6nGUDbWz/AgNxW4Uy+YxA0GgvykDv/tGDFR5quzw4GU/qPSdFpRjNHGOIyqgyMc/K/h0Q0H5zAH8ih1SR7ZVhw6rLgyQi1Enh+qoJmnISBgeVkJZb7n1ErI9LXlb7VhqhBOwJdAQgottmZLEMq2E6eFxubM3YkSYLRvSQo2h0hRWQn6qBykE2imoTGmFmLPc1YSR9hXa9lIpjK/l41S4hNaY6czKiy93i8/0QPHLGFhUjYnTqGsPv15Rl8ashwTs0tstTP+x4eJJszVObgDXUxgyFLEoua3aoRI9F2/NF6psGi81trdAUxtCBXyZyKv9uMJgPsO1XZSmQOP6cPs8KaI+JK/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v2.c | 89 +++++++++-------------------------------- 1 file changed, 18 insertions(+), 71 deletions(-) diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 301a991dc6a6..1637a0c21126 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) return ret; } -static int do_sendmsg(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - int ret; - - msg.msg_iter = *it; - while (iov_iter_count(it)) { - ret = sock_sendmsg(sock, &msg); - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - WARN_ON(msg_data_left(&msg)); - return 1; -} - -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - struct bio_vec bv; - int ret; - - if (WARN_ON(!iov_iter_is_bvec(it))) - return -EINVAL; - - while (iov_iter_count(it)) { - /* iov_iter_iovec() for ITER_BVEC */ - bvec_set_page(&bv, it->bvec->bv_page, - min(iov_iter_count(it), - it->bvec->bv_len - it->iov_offset), - it->bvec->bv_offset + it->iov_offset); - - /* - * sendpage cannot properly handle pages with - * page_count == 0, we need to fall back to sendmsg if - * that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag - * which triggers one of hardened usercopy checks. - */ - if (sendpage_ok(bv.bv_page)) { - ret = sock->ops->sendpage(sock, bv.bv_page, - bv.bv_offset, bv.bv_len, - CEPH_MSG_FLAGS); - } else { - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); - ret = sock_sendmsg(sock, &msg); - } - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - return 1; -} - /* * Write as much as possible. The socket is expected to be corked, * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. * * Return: - * 1 - done, nothing (else) to write + * >0 - done, nothing (else) to write * 0 - socket is full, need to wait * <0 - error */ static int ceph_tcp_send(struct ceph_connection *con) { + struct msghdr msg = { + .msg_iter = con->v2.out_iter, + .msg_flags = CEPH_MSG_FLAGS, + }; int ret; + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) + return -EINVAL; + + if (con->v2.out_iter_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + dout("%s con %p have %zu try_sendpage %d\n", __func__, con, iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); - if (con->v2.out_iter_sendpage) - ret = do_try_sendpage(con->sock, &con->v2.out_iter); - else - ret = do_sendmsg(con->sock, &con->v2.out_iter); + + ret = sock_sendmsg(con->sock, &msg); + if (ret > 0) + iov_iter_advance(&con->v2.out_iter, ret); + else if (ret == -EAGAIN) + ret = 0; + dout("%s con %p ret %d left %zu\n", __func__, con, ret, iov_iter_count(&con->v2.out_iter)); return ret; From patchwork Fri Mar 31 16:09:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6D44C761A6 for ; Fri, 31 Mar 2023 16:11:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 468756B00AA; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F09C6B00B4; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B8BC6B00B5; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1CF3E6B00AA for ; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A6DBDA0DD8 for ; Fri, 31 Mar 2023 16:11:36 +0000 (UTC) X-FDA: 80629683792.07.D2F6749 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id A63B2180006 for ; Fri, 31 Mar 2023 16:11:34 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8bwMiLm; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=HrfWnZkuf2JeSItqSqsRwr4MlCt5DEqQHMEqjShm+3nqvX3HfX0fm7EOI9Ofm63+Wx15md wDbk1WcKOKJmqasFxHsc9Nk4V3U54a3q7MTtZhv/kyyPSM40Ybhq4Yvz7x4zW4A8RZFszP ud87oITSHI+sewB7kLWDjcmgSyqj0vI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8bwMiLm; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279094; a=rsa-sha256; cv=none; b=8eKGeO0ZO34eWIjZjtfNkpFbzEKOWvGheNcwnjl5pmeKZeLO0/e4UL+72N1b1WOyDAVZur vngoL6o2rQCn17u12JsNzMJmnFYYIGHQEzY/yzs3ryIeILawWxPAQ8iisiT5uGHr8UcmbS DViimQYYbiV+yHQzumizOGs/ehwvcrM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=h8bwMiLmhys0FZ010tAMaYe063cgK0+OrbCwYI3qQGgOMC9ErPZhyMxhf33w/ygMoc9jm5 I9RlSLMx+PlZMMpKSCOLG3Y8WpXgezvW6XqZF7qJKcV+lejIK6BcDyzgoBJI9pIk5nSito xCAWdrQtb5vpNF7DlxY7u6MVGPvj2Sk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-7dchcmswOjSwjHDFElj0Zg-1; Fri, 31 Mar 2023 12:11:30 -0400 X-MC-Unique: 7dchcmswOjSwjHDFElj0Zg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9D36988904B; Fri, 31 Mar 2023 16:11:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 724AC4020C82; Fri, 31 Mar 2023 16:11:26 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH v3 46/55] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Fri, 31 Mar 2023 17:09:05 +0100 Message-Id: <20230331160914.1608208-47-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A63B2180006 X-Rspam-User: X-Stat-Signature: bxunzxy9idnsw1uhfhfqxg1e579keu6e X-HE-Tag: 1680279094-195512 X-HE-Meta: U2FsdGVkX19RuYxy/ZoF2tUMv8SZ4HgB0XCU/8eFVx0MLKCJolPl6SqXiBBp3MoTFeQ/Px6pg2/U0EwH4lnWQ1PZfR3NTZbHt7VMxVEbr3UvLWNck5D9464OsSDbihbXIYz5uFDNNKAfgMKoPK4tK3AbkAwDS71tguD1ZaOeQZ4ejfU9f1cDYynVTtpMKCIH/IL94JAtG819Z3U2hdytk4u2nmIWcPVf2zQKfEvM9hho3Mn67pLjP9l+Pc9aWUwYR3z369VYM8qaSO3h5n4gYPfOkNd9b0K+9sgrc+mxSVTZ92Wgu4ocmDOWUdzS1afuayD2ABKW4pNgv4sKq4yHziWU2OQSywSMpcSsqndJcO+r0Lta08ZsIpqs/Fj6CYp2U3qVWOm66HtePbiCiwqhKlRdPJZRcZJrV3LbANSznp798cojilo9ZvW/p3fmNyfJbVYi0PY4i1mFGBQBJH00WWqVtsGBF8NLngT9qJEBDwXFzzMMpuNb6lJ1WCsF4nCVqWiwMp4FliI7h+Peuqx/9yIciarCnTaKvLqsTSABdBHbCZZ7NYBZvdzmC4xTM1ZIhzryw6a56pECJxq8mRbaNaSu8KZMLXMCsdJrCmxS/vn9QBntmO5qU7NXUpxANs5sj382hg9pYaOM3D57pQjg6EONG2rHBa0BFG8apbOsVxnmmyiGl5HCoKjr0r+BTxMRX4BHMuSZyA/ezrlxG59iElqLsujWG1Rx3995S+m2f/yiwHMXJytv8KWnwnN/iHZzfkB5K1R+7z6euEf470Hp/6Al0QfWsTLe/DVxumFZ3eW/nh+5TlhHtecEJ7AndQOsYrZVO6a+N6vd3A/4ilutckNHvuq3frzyBnUw17iGKqDHfv+caYK9MGs9Z4by/runmNXNn4PjVt4YUJOI5uQX/HzIJL6RAvCO5alsCMQmkNaaftTh28Gj/vWH+q62l/4PDG0WUfo4kYSyVQF5fv7 ID5q2yRx 7yOaSDGFiUhbNUbZJ8AQoFVGdirq+2SuDX4R2frBsbxr7yjI6Ugkpg1i1hiW0yUvl9NllhX2cVaWtcMb8ie2tkJvawOVHR3mAWJEw7B0s24IoLA76Z9/sGeMoIfb/5siSs3oG4a+RPEu+yex+eoO7ARd/aRB/0KuWa4nckVzo3XxGvW2MeF6u9JiozF5I6xKKGZVA3XctptR3RIZVdTFH927ApwoDIDpXfx9bH2TbduGljOTz+IPQkCxTkv+9OySufI7foyKkbixTXmpEBMnMJ2uXypCxVrlxJmPjwTO0pF2V3qziAiINe0eZHY0fuA6ca0pBBOKnITaRqf8Ub7X9t5rOywKlC4g3zXpM113vX+6Sl1Dk84dkDd7TSMMxvqNB+0BoiBDtYMWPxp2UY2PKBBgO/xkkUMyMciCr5kF5QxZOdeW/45UR9aPF/d5AStZYQY0LezZ8Sd/kll1+RQVztAs7CA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 86 +++++++++++++++++++++------------------------- 1 file changed, 40 insertions(+), 46 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..660d9f203d99 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,24 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; + bool free_hdr = false; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -99,43 +94,37 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, if (hdr_off < sizeof(struct rds_header)) { /* see rds_tcp_write_space() */ + void *p; + set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) + ret = -ENOMEM; + p = page_frag_memdup(NULL, + (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off, + GFP_KERNEL, ULONG_MAX); + if (!p) goto out; + bvec_set_virt(&bvec[ix], p, sizeof(rm->m_inc.i_hdr) - hdr_off); + free_hdr = true; + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +147,11 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + if (bvec) { + if (free_hdr) + put_page(bvec[0].bv_page); + kfree(bvec); + } return done; } From patchwork Fri Mar 31 16:09:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CBB5C761AF for ; Fri, 31 Mar 2023 16:11:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BBEE6B009E; Fri, 31 Mar 2023 12:11:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 243736B00B4; Fri, 31 Mar 2023 12:11:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E4F76B00B5; Fri, 31 Mar 2023 12:11:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E6F546B009E for ; Fri, 31 Mar 2023 12:11:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2E263120D9E for ; Fri, 31 Mar 2023 16:11:38 +0000 (UTC) X-FDA: 80629683876.11.8266B94 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 657DB4001C for ; Fri, 31 Mar 2023 16:11:36 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XQ7Xl11J; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=LkhnTzXU1X4znEA7DVY9E02LvPRWVvY4w7DVJkAd+bnkN6wx3578hixSeg4qN4wvF7BlZM a7sPX9osyjvFtIc1/G0MBKIgyYa3tnI2ymi2+X0WiSmwMHtew8maTw7fBpYA5dEjGQOm5N F7HDhlonZG0d0i3sRbqTulnVGRjJhoY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XQ7Xl11J; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279096; a=rsa-sha256; cv=none; b=wcaMXqL2JVgtcmF1vSf4ISifPkYYHXixtxvauttYvT06sczF3F8mZ7Q7RV8Hosc9rFzkTf VXPQx/Dcn0qDVTb8B9iSiRuX56sBqOj+wJYi5dtbn1FtBpO9THNCcyJVyYKn9WTRguJuHf WbqXDfi/AlJrT61NBH4jPtkwKiAGkxY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=XQ7Xl11JcU0g0QoyHLqRjt1H/ub1Cf5WUiTkse8UlTEcpT1CQIgceHxljMidQ2N6VzMet0 NqAgMlT8Lb0P43qvPb/4zgn/K1g6nfNrvvCax/d86CHZwFB85Y3/RDp6YrhpgB4Vf3nDVV ZoG4xjpV4r0QwovLZ5WaLe8TzkVSGDg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-611-KDykX2VqNS617UfimOZiaQ-1; Fri, 31 Mar 2023 12:11:32 -0400 X-MC-Unique: KDykX2VqNS617UfimOZiaQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6DE3E29ABA19; Fri, 31 Mar 2023 16:11:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3AF201121314; Fri, 31 Mar 2023 16:11:29 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christine Caulfield , David Teigland , cluster-devel@redhat.com Subject: [PATCH v3 47/55] dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Fri, 31 Mar 2023 17:09:06 +0100 Message-Id: <20230331160914.1608208-48-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 657DB4001C X-Stat-Signature: 8aky1qmnknykpyojbwd5g7nyhs8w8qec X-HE-Tag: 1680279096-957652 X-HE-Meta: U2FsdGVkX18rHlZznHNzwq0X+NpROUW4ERJWCSO9oR4NRUkbCPuDT5ZsSZEvFSVmMyGhw6tbR5dG6IjysnPl439gUVo3VKG0nSvW5II7JuGJwLkZrEF/MRt5Eq1n9eBBNviHQ5Xx9o//9qMDF3wxqJlU8cfGTM1Xwl6KLurKoHrfW3o7+KoiWjlH8iFRGWj9tKZQrQMIFoj7R+WLtmfprvgGyXhU/iW4EBttzNGWLDXVOvEhQMzfr0vkFv1Rzf1FJWbKZj0TfCowkccJf3iZv1KKHf7Z+VU9LLGYyvo7/Gyo5ohQDY8vk47x8VFoqfIljTnsWYz4nFcG4yqZe/m71rnL7JRhevSBSbDivGRcUSKCPtO+oZ3CwiNv/phCLq+ADZdAE/Z0+CpyJ8Zksd10VUS71qgV1lwTovlyZsLmkE/NUC6MP48DTTgksz+gWqLcsnfYw4okOvblL5Ze2Og5LntB3vEcXJvBWCf322RdoZZxgKByYnhfzTmXTT9gaMMuaekog4fwkaW/yx81k5HbYV8Anpwz8aFdil2ItZN17H9iblUSUeDIwNrwosqawCS4AMPztZYCTbF3aBG6+n1wcAzaVBO61QyBdSIxl3/I3VwSLWFNoxGbKXAQ3xrz9wEChm3H+UQ1xel+UXttex80SN2yFjImNmYWeHFn45NqbDgIzfAxWWLIaU4RUcmTaROjor5wXv0FrE/4NQPerc6Tgo+fXw3jgbDIb9St08s05j/sJDtr0Ky3qrKc2+KpT3wA3nxB8HlL7j4dcMB8y98n3ALGagXPtxgZIRNtJBPeTRQ+Fd8tZzPzuL8B2cEOyZJpx9IspdC4CUPmdYKDMv4Vd6zI9psTkXSdCigu4n3KpQ2TwFf+8ZGiwe+q7NPCEsPZkrB6Swa/CI+wVOnxwbj+ROLu5nK11PK+erCtPuIb5JZ8fViG91T9BXq/aYXOQq4VFguuwTyAdixyA482T0x DMZ+lRXb /xRrylc6j7kPJtTHNl3OD7Svnodhb1axpoEv2Rn3QvpEDefY+4feqlxTaGc3EZ07A9ka0aWWlT09Esx82yGzXCIKfB/adb6M9TsjZdt/blu1FHiZg7c0Njxc9geKMnlBqaSTZzZpCfZIF4j21gsTRqrm76iSe/nWQhwy4TMKTMTWcIoZGE8ZQl5z1kaY+iMKPdjo3GsTmCUpLo3Kr1zUi3Z5oGrF1Rm4SSphsw6f91Aw+Ec1cnafT5VpWh80rngd5XLALoIyOxFx2yNig7NoVjrKJeHYK4wBL830lp8Sr/HMUxntUa57f1D5dRsBJJ3w48miJVxLMbAMvck5d+kyhyh49tbsGKgARmUMXcx8gMva/fW9oq9hPf6iVDh4rcfQU4XC2yP845rW88p6RHUkQbsZTINzgxDEYlmngQs7mk7hdN3ezl/o9mck3WQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down a layer using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather using sendpage. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Christine Caulfield cc: David Teigland cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: cluster-devel@redhat.com cc: netdev@vger.kernel.org --- fs/dlm/lowcomms.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index a9b14f81d655..9c0c691b6106 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1394,8 +1394,11 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg) /* Send a message */ static int send_to_sock(struct connection *con) { - const int msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL; struct writequeue_entry *e; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; int len, offset, ret; spin_lock_bh(&con->writequeue_lock); @@ -1411,8 +1414,9 @@ static int send_to_sock(struct connection *con) WARN_ON_ONCE(len == 0 && e->users == 0); spin_unlock_bh(&con->writequeue_lock); - ret = kernel_sendpage(con->sock, e->page, offset, len, - msg_flags); + bvec_set_page(&bvec, e->page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(con->sock, &msg); trace_dlm_send(con->nodeid, ret); if (ret == -EAGAIN || ret == 0) { lock_sock(con->sock->sk); From patchwork Fri Mar 31 16:09:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24AF4C76196 for ; Fri, 31 Mar 2023 16:11:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B52226B00B7; Fri, 31 Mar 2023 12:11:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF5EA6B00B8; Fri, 31 Mar 2023 12:11:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BDB06B00B9; Fri, 31 Mar 2023 12:11:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8CE3C6B00B7 for ; Fri, 31 Mar 2023 12:11:44 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 687881205B1 for ; Fri, 31 Mar 2023 16:11:44 +0000 (UTC) X-FDA: 80629684128.26.CCBA8F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 787B04001C for ; Fri, 31 Mar 2023 16:11:40 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Fcolera/"; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v5XUK+PQOR7d9klL6/6O6KO8pAUYr8F8Qal1kb1WUAE=; b=VTzm9ylZQu9Q8PSNhhHVbLyJW6Xejdbr6ONuMXculiWljmlvAJVOtV9Yg83PlUXTyX+0Bd qArTVcvmBrbz5OIYYSz+S84jFBO/RLIYBoSI4hsbuwi7w+gdDxYdIN+YC0c8ngjzGmOcLW yjTMNHfmJ4NgIJ28HxgTi2gWhO/DkuY= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Fcolera/"; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279100; a=rsa-sha256; cv=none; b=NenF82n+Ze3gjLHvJtFYCalI4U9O4dOmgdDJUujORMdD0q8oXar+Iy+BjeCxoOYCdeYTqI b0VRFO8qnEeEqweHMXtFUsLuDuCMqBvf4qzWMZYj2Jf2PHLny622Tw7maNO7GAD9WBiwj6 EtdMEGeBk/Ne3gt3icX8n8oYoMPLJQo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v5XUK+PQOR7d9klL6/6O6KO8pAUYr8F8Qal1kb1WUAE=; b=Fcolera/QKYuP1h0wLKreJG02FpTRtgI31yAn52toW35QLFmUOMX6qUXE/KRbb7c8V9ZUt QT9thTUHUJRf0x3Y2k6jgb+2KUUu066NxsrSbOQREyWzKJymbGmq340tBo1lBT3moSlLgk sJUSqbRAtWKrP2iwk3DWoz2sXztvP4k= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-256-fHdJvov2Og26gxE1ES-_Qg-1; Fri, 31 Mar 2023 12:11:35 -0400 X-MC-Unique: fHdJvov2Og26gxE1ES-_Qg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4E74229ABA17; Fri, 31 Mar 2023 16:11:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 22BEA492C3E; Fri, 31 Mar 2023 16:11:32 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Subject: [PATCH v3 48/55] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Fri, 31 Mar 2023 17:09:07 +0100 Message-Id: <20230331160914.1608208-49-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: sjuy8mwsr74kymx9k1cmf4unkgyetqcq X-Rspam-User: X-Rspamd-Queue-Id: 787B04001C X-Rspamd-Server: rspam06 X-HE-Tag: 1680279100-149352 X-HE-Meta: U2FsdGVkX18tN003biLCE/+xk4OfAHWccxheriyeTrm97r7RYGYPxR804E1s1rE+0Ge+JHFMq/LcH8MIWiBukmHtoq1Q04TQWaI3c7Rs0Xt7c+gHGGd5YJbQatXmfAdaSObCPHooQAe4+g1D57YNNqMAqTH5d/3R3YWmPCnOlsO/5TIFhKVtuvL89CJ9jO+jV8Bq2L1qgxt504H4FmHpWXwalDt4RmOcTGq6bcTmK+Ridu+mc3AhF+35VjLJOiDC08YVak7ka0OWKhNTnnnPXPTvpekgviY9bdBS60ku0ajqYAl2NYFQyElVUmKQjppOF6Rf8IhwM8KJGLmV9tLIi5RSy1Ob6ty8heiSB4oBUSF5B6+CRx4msZvZhm0OLnK2mKcSukU876FVg4Oom6roG5uRe5I0SjBF2XRvJhYscEob/MnQNl2Gu5JmUQbyseGsGLeL9N9vLm1c/KnDCFrbFVpIaFvZrp/mWzlryzjrr9VPLmY2pSmOjasYUsEzqMIAWr/zKsKKklq6hLGZWO7zIxzIs399DDB61BfxwS4cpiwQawu05rurL7K2QCI/BCbCDtxCT50Octjk2lrE2nQHEr1/7W2Tsih5df4dfM9zXiygLwQyLgScKMRnC+i6Ffxn0xVEX8g3I46pTJZpDMGJaL604XAfTdg/HzfkgzzgupByExI3ASrU3AEI6iU5pOD3Sznr9dYMOuLh0iWOJyU8nQL8zfZEv0VScPS2Umss7+0Z0jlwWeE3R7TaQ1LMrex5gxdOyoqZ4h8OpW1CKqtdHBgzvbnT8w6iZJGAQ1gop7xHGY1/homjgzneSazVlfQKnRG7vVBaB21i0OjvanG0fDU3o4uVL+gciidoyzRf9CNH7fgM4a2Odo9TJawXpIVot7LzuQCJGbmWHzMsqGs6VNJ7EbdVl8ZBSUWwJXrEpLcH80B89q748iFsF/xpw1UqXoy46aD35kHdPw1CZkF n6dFa9AE Dr/UfaIXU9XSqhAaGNwZfCz16XZGH2kYO6gwHaxJEczmpOzLq8PVjqG7VE3vp6WXgAFFsXQcjtS8Q+WV2KoZ6DgHf4wMVUulhfmPlVZYm5Zv1QRqRk3jW0PssYN6TM4Q/yQ5IRg2sVEoWBeevb0IUpVrk0QmC4+EcW/ZtrqSjidGVRWs6qOgXUpdJKb9VA30sgXZsim5erFTC7VjT9vufiUtGcAh5PEzYfJpTHs7nqq6qL8+duh37vlyDvxm5q54cTIOXjd09P1/hdwlNu9Hvj+AU1+BahrXfZ9tN+qckNe5QyGIR693ZIKfKRR85IXewpt4u3HiTavpLMGjlLWBQ8wSFUMGYnySM7YxfN5PTMer/vqWfOLNaeiUQp0xQzT9yvdH4tvwUoxwq5OKFWu8texGxXDNVl5B2rh4TQw4ozghIOpwVMTUZ1p/M8UkYmoJrwD11dQW//lFE1cKMnnGdE0mbFUBLCmoJUCkwIeOsgQym9NfKOkDRUvljbCMPnIFOnkGbShLvvvt/3LDQQKfJt2lzDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells Acked-by: Chuck Lever cc: Trond Myklebust cc: Anna Schumaker cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org --- include/linux/sunrpc/svc.h | 11 +++++------ net/sunrpc/svcsock.c | 38 ++++++++++++-------------------------- 2 files changed, 17 insertions(+), 32 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 877891536c2f..456ae554aa11 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -161,16 +161,15 @@ static inline bool svc_put_not_last(struct svc_serv *serv) extern u32 svc_max_payload(const struct svc_rqst *rqstp); /* - * RPC Requsts and replies are stored in one or more pages. + * RPC Requests and replies are stored in one or more pages. * We maintain an array of pages for each server thread. * Requests are copied into these pages as they arrive. Remaining * pages are available to write the reply into. * - * Pages are sent using ->sendpage so each server thread needs to - * allocate more to replace those used in sending. To help keep track - * of these pages we have a receive list where all pages initialy live, - * and a send list where pages are moved to when there are to be part - * of a reply. + * Pages are sent using ->sendmsg with MSG_SPLICE_PAGES so each server thread + * needs to allocate more to replace those used in sending. To help keep track + * of these pages we have a receive list where all pages initialy live, and a + * send list where pages are moved to when there are to be part of a reply. * * We use xdr_buf for holding responses as it fits well with NFS * read responses (that have a header, and some data pages, and possibly diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 03a4f5615086..3a015abac5bd 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1063,13 +1063,14 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec, int flags) { - return kernel_sendpage(sock, virt_to_page(vec->iov_base), - offset_in_page(vec->iov_base), - vec->iov_len, flags); + struct msghdr msg = { .msg_flags = MSG_SPLICE_PAGES | flags, }; + + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, vec, 1, vec->iov_len); + return sock_sendmsg(sock, &msg); } /* - * kernel_sendpage() is used exclusively to reduce the number of + * MSG_SPLICE_PAGES is used exclusively to reduce the number of * copy operations in this path. Therefore the caller must ensure * that the pages backing @xdr are unchanging. * @@ -1109,28 +1110,13 @@ static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, if (ret != head->iov_len) goto out; - if (xdr->page_len) { - unsigned int offset, len, remaining; - struct bio_vec *bvec; - - bvec = xdr->bvec + (xdr->page_base >> PAGE_SHIFT); - offset = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining > 0) { - len = min(remaining, bvec->bv_len - offset); - ret = kernel_sendpage(sock, bvec->bv_page, - bvec->bv_offset + offset, - len, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != len) - goto out; - remaining -= len; - offset = 0; - bvec++; - } - } + msg.msg_flags = MSG_SPLICE_PAGES; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, xdr->bvec, + xdr_buf_pagecount(xdr), xdr->page_len); + ret = sock_sendmsg(sock, &msg); + if (ret < 0) + return ret; + *sentp += ret; if (tail->iov_len) { ret = svc_tcp_send_kvec(sock, tail, 0); From patchwork Fri Mar 31 16:09:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00005C761AF for ; Fri, 31 Mar 2023 16:11:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B97BB6B007D; Fri, 31 Mar 2023 12:11:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B445D6B00B9; Fri, 31 Mar 2023 12:11:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E46C6B00BA; Fri, 31 Mar 2023 12:11:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 915D76B007D for ; Fri, 31 Mar 2023 12:11:46 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 71C4514045A for ; Fri, 31 Mar 2023 16:11:46 +0000 (UTC) X-FDA: 80629684212.08.08EBDCD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 907B51C0028 for ; Fri, 31 Mar 2023 16:11:44 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="BczYX/f1"; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279104; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XvqldHWl7xh/L1V+KGWEm9YizW/K8SDxYbPNnLnjdW8=; b=n61lPV0sucNwoLRJbxEMqRu7w3agF/HiPn26hFoTMy1BMZsFmD1jxdZeJ3yy/mNim2wUR/ sy+yZrQA97UKsfG7vqxI4XBVCqQ5tBOYNc1l2Ic1GI/I54spJ6bASFX0vpKtWKnMMmCfh3 muMYXwOUvPVhGQKfld88PLYeYLchMaU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="BczYX/f1"; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279104; a=rsa-sha256; cv=none; b=4YOBneT1UZ0dzlirtyfmqgn6XxLYl86nFnIqhD9fBrX95m+/0NeSXR7w8aV/RdvVTA7a7j oDbTdqeSJlPjpLM6nZFtB3XrJezojG9QVFF3oV5T+0up3rOW8fiZXSGPBSbY8EMAeol2nL tyipUbPiRjF3tvJb/fdG1QLX5zhZqSE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XvqldHWl7xh/L1V+KGWEm9YizW/K8SDxYbPNnLnjdW8=; b=BczYX/f1+Zpwra3Yzmu5h18GDiA1d5hxxYkDzbw5gNkQ5lglCmuH/kbY//PbuV91hsaXdN GQME5koznzL52e6DN2I7R0Q7vxBo1A0tHFk3BUZnPdyRAPtwZPHDpLD1pW1C94/g6ww1Ls gEVn+ZEKc+z2C8dl8lDZgtEQpqlVkPE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-173-1ZXUWRtzOLGY7fvr_wG5SQ-1; Fri, 31 Mar 2023 12:11:39 -0400 X-MC-Unique: 1ZXUWRtzOLGY7fvr_wG5SQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5FC60889047; Fri, 31 Mar 2023 16:11:37 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD7E5202701E; Fri, 31 Mar 2023 16:11:34 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org Subject: [PATCH v3 49/55] nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Fri, 31 Mar 2023 17:09:08 +0100 Message-Id: <20230331160914.1608208-50-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 907B51C0028 X-Stat-Signature: hwhq9srmcukmpcyrtak1m4c4s3dy17r1 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680279104-999460 X-HE-Meta: U2FsdGVkX18ISqrWQ4Kv5g4hAC8qr5jgbXTIDFKT7xpem3CcqO9xvR9SMDBa2RKyu1CRLasMzuvEC4ZNO4CI4IP4rBgGBCz3vD1SW6PfrfY99StlrT6Bpyicpo5D3pQRQBUvaQFwf6VdjPNZXhgPdSNyYg2/zXKVQHwrG4DUms842GqGuoE1xVte0V9RZnDkPCoLn8gsdRbfoJPC91xTGimLIYRi4flUCF3F+6Ud1FeiLKaRIB6yj1ckeqNUzaL5ycjzEnS0t6XExUBdDxKIIbWXYcIqgK1lbJGLqN43ZR2nHwIcXEhES0vB+ZbYW5/Q+IuplBHt8O5iXBnlaYetlayomQ6EZRxkWzn0fowDqO8IymwNp5sDwV9G/tP61D8LzvqxBr6KeocZMtMhGXWWk71G9kcAnuktjlBO+wwAZuGHW2gP0wkqQElqOvaMWZlze6qoRCBhr1Ru0CsgrdHRjnCRwDHbnKA7SBVWrqA+kUXsdsRxf1QXfrp9AFRTGztJ/gillXzqzsSANzaXwk/e9phxNHk5E9vEB9sHAWhMGR++NU4LnaM57NJW5ctcCKAclraK8bGE7odtG3zsbM2bDRXRuxFLVVJ5piEPNCXSkLNZu3Dei/iCt3nYmenGq1GNLUf9dOHbupYizUmK18JgCkYsyxixGZ1hCwgZNzyMcvmlyRnzQJYgR7rs0oqwle5pWyFaPN/ixzeOzeoFY83Wd+AtiSv+0WmGEaHRJYSqoBEgl7i3gnDgc4hGnH8S8I8T5kOFuUcyiK1GqtMDAaLl4HBR2q5u4yy8yp81vjCuvDlQJ8SmD7M4xYRMIordZGJGFGE6oZYWlv8vNoH0kF0hPJpgAoqKN/Kck4LbreTQyYTq3SHuedxKbAlObyMMBMCg5R2DvPioHkks+ss5pSQyWc0JuRaDY9Zq8uWqqgAFBIQTdmZOMqORqYhV/xrudzOgJTCUekNVK17R0NAMPYS Lrj02mXF I0Q2jF17V97SE0EFwDJf03uQy+Xi7kopbHbrijBtrwOzlUMnRnljVDfm/Ptcoy25W4p93RU5ywLsRoeCgRGDJ/l82U/0XpJtT7+VdhX484fhtysjxEt1qeWI9ZzuS0OdpaYDNbgQ77UqVaW/PnRiNKpuwZRSVw3J9tL8bPvHEsbwCBaa4GW1OUaXcnhk7c73wC6h9HOpCyJWmvBhono5eFLWhFN1qBWHmOkDRCY+sHyasgJRNmT3zqhHHw+syduQdL4EULm3G1wnjeLzKOnz/gUNMReuw5vYOgFcupjeLrf+OJgGWRUUbYNc559WMrxka6C9eIfzePU1emec97jd52VwvctrE/r9GkSTFw90bmfb6zF07vwM2ctVyz8zX10yDKOE0bT3JmtxTUpfzZojpIhTbWJUn0P8vmdn5HeP0cTk+OTFLqvy+rAIAUZDvM2f7xKFLa1AKvsA6+IcrqtblVrJl4Wp4lI+63c4tvoi8hNM1ycwWyaOldyCkjeiypwihq+5w36jbBE9cpOk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nvme@lists.infradead.org cc: netdev@vger.kernel.org --- drivers/nvme/host/tcp.c | 44 ++++++++++++++++++------------------ drivers/nvme/target/tcp.c | 47 +++++++++++++++++++++++++-------------- 2 files changed, 52 insertions(+), 39 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index fa32969b532f..cc617692702d 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -979,25 +979,23 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) u32 h2cdata_left = req->h2cdata_left; while (true) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; struct page *page = nvme_tcp_req_cur_page(req); size_t offset = nvme_tcp_req_cur_offset(req); size_t len = nvme_tcp_req_cur_length(req); bool last = nvme_tcp_pdu_last_send(req, len); int req_data_sent = req->data_sent; - int ret, flags = MSG_DONTWAIT; + int ret; if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; else - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; - if (sendpage_ok(page)) { - ret = kernel_sendpage(queue->sock, page, offset, len, - flags); - } else { - ret = sock_no_sendpage(queue->sock, page, offset, len, - flags); - } + bvec_set_page(&bvec, page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (ret <= 0) return ret; @@ -1036,22 +1034,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_cmd_pdu *pdu = req->pdu; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; bool inline_data = nvme_tcp_has_inline_data(req); u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) + hdgst - req->offset; - int flags = MSG_DONTWAIT; int ret; if (inline_data || nvme_tcp_queue_more(queue)) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; if (queue->hdr_digest && !req->offset) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, flags); + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; @@ -1075,6 +1075,8 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_data_pdu *pdu = req->pdu; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, }; u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) - req->offset + hdgst; int ret; @@ -1083,13 +1085,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); if (!req->h2cdata_left) - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); - else - ret = sock_no_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE); + msg.msg_flags |= MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST; + + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index d6cc557cc539..00b491abf50f 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -548,13 +548,18 @@ static void nvmet_tcp_execute_request(struct nvmet_tcp_cmd *cmd) static int nvmet_try_send_data_pdu(struct nvmet_tcp_cmd *cmd) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->data_pdu) - cmd->offset + hdgst; int ret; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->data_pdu), - offset_in_page(cmd->data_pdu) + cmd->offset, - left, MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); + bvec_set_virt(&bvec, (void *)cmd->data_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -575,17 +580,21 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) int ret; while (cmd->cur_sg) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; struct page *page = sg_page(cmd->cur_sg); u32 left = cmd->cur_sg->length - cmd->offset; - int flags = MSG_DONTWAIT; if ((!last_in_batch && cmd->queue->send_list_len) || cmd->wbytes_done + left < cmd->req.transfer_len || queue->data_digest || !queue->nvme_sq.sqhd_disabled) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; - ret = kernel_sendpage(cmd->queue->sock, page, cmd->offset, - left, flags); + bvec_set_page(&bvec, page, left, cmd->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -621,18 +630,20 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->rsp_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu), - offset_in_page(cmd->rsp_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->rsp_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; @@ -649,18 +660,20 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->r2t_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu), - offset_in_page(cmd->r2t_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->r2t_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; From patchwork Fri Mar 31 16:09:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21163C761A6 for ; Fri, 31 Mar 2023 16:11:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B7966B00B8; Fri, 31 Mar 2023 12:11:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 466136B00B9; Fri, 31 Mar 2023 12:11:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 307A46B00BA; Fri, 31 Mar 2023 12:11:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 239546B00B8 for ; Fri, 31 Mar 2023 12:11:45 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DCE1E1C6BA7 for ; Fri, 31 Mar 2023 16:11:44 +0000 (UTC) X-FDA: 80629684128.21.FBC826E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 3AC754000E for ; Fri, 31 Mar 2023 16:11:42 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HR+JpLyf; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V7IwXZpp4caT5icSmRzfTKcX2kIg430bJYtEqUuaqXc=; b=L6MBcyvNR/3i1X7EV/5adnGtejHcWd2lmmNKIHwDh+Byfl0w2hQpTuTV8BgbJBqu4YbzmO MKzZBeoE4gpj+QAZhX6gBIUDdR1eAAcBhpt0SbqpQMdG48/plHO9O8ipA4Pxj8PtUum6en oWgOZZiKo08txeyTxNz8EEp8qIiRuyM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HR+JpLyf; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279103; a=rsa-sha256; cv=none; b=Q80k7ys9pn+9laRbcE7WFkuaumQqOSpgo6lbObYFUb5h874XD+HrBXMTbs5C01vNAm1HLO x5CNKJ2Bmsez4W43/5L1CTHMg69ukNNp0ABnx0wbZmDAHGmV2fWYQZE+RukLwcAKE2TR+p obaUyh98nphkXZu9XjXX+SOpHLb//0o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V7IwXZpp4caT5icSmRzfTKcX2kIg430bJYtEqUuaqXc=; b=HR+JpLyfsmERPk+W195XvE1hJaYn/vsz22nFHZL+g/tFAKvRKevTgQlEO2KDhCQ2aUSSbr p58oASrjvRSRiuFDbDGa39GI9DJxueh/1IQF96lEvAZIVXseaV9pxZXU+MrrOYoh9eHRqp aqh7WrnXw7U1dK+whR5AWmXuK11tCoU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-499-KUirUmF0P5GTXUc33yxBYQ-1; Fri, 31 Mar 2023 12:11:41 -0400 X-MC-Unique: KUirUmF0P5GTXUc33yxBYQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2135D10119EB; Fri, 31 Mar 2023 16:11:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 120582166B33; Fri, 31 Mar 2023 16:11:37 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Tom Herbert , Tom Herbert Subject: [PATCH v3 50/55] kcm: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Fri, 31 Mar 2023 17:09:09 +0100 Message-Id: <20230331160914.1608208-51-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3AC754000E X-Rspam-User: X-Stat-Signature: str1euck67cir4bxqkdtyn4htmjq18oz X-HE-Tag: 1680279102-151839 X-HE-Meta: U2FsdGVkX1+x+OK9YsKrxbsFoMMRB7WBIFn1txdghZjm7/FuneFnbb8M0yKMvNwO5ZH3mfLdwRatPKpJfRf1JKLccpiPnKuGSSsmR6PHX2G0J+DckV+5om8CRatoui07lAO68zgpbzUQmPjLImvFmz6TCMxKd0V2cX4q93xdGeqfAsDLTDpd4i6atLiKE0NtQjjRER4I6+Zr5FWqaYQLjX7Gl5LC5tjnFwkywtZIbAGFxZuLcUnJlD2lDjYzXsQsf/DaG1gL1lmxaDDuLNi2qsCgB5rka/sl4heDo11zKmTaHxX0quHc8rkazsQfOhuF0H71FRPpS7EVpyzD/unejgzRwDbkMk32KfDBSXKSCBv6KF15NQy9lPScMqANSkTQpOFALRMMGtRZtsa1CbESebPS/mok6Y2GSGZPNeU50Oi7CR1AuJFhtT/gD6ZxbiYSjlmEWObOHbjNSaDzjx3OFgk2S56xH+dK4OkVlpPIB3f9pmxUyaba5s/ht+71CepTfKSak/JETX3opXJN2z9CaT+5oazw2Sk9TXXaXN0oqje772mJg7Y0tYs2wHtnDGL1++pX4jIEJu7V/nRrvF0/XhJXXjHZsuLV2g2PIY/yoqMSL5+jaKDvN9YyCctjX+PD2OHEiLuMHR5HGCS7wHvVdftH/ktnwD1xSz6SUWjhIZ6I0byXsC63XIy7CgzHPuA/ppuT+48Z+owUJLmixnV0ZkjZmxX0yY8UqhQFVRBGeYchNRlZt0sbkkJW0peg2euLUsvVNAxbUDjD8cLEWDLBncn39+49LrYgCLyGku0+FCnfzSVfn72xo5zDevsCgmzwMZHtAWLTb/Gz/nmG6iB2rec5BPFiGWBygIQq8k4TXc9ocs4SFVARc3giwHKeCxHyMzPbBUi1tfcqBFNBjp8ruXk+qfGyadGW0ptoefrNoyyAEOo9/ayckQBQJR3uDkd/ctjHcijvdj7GB/ws2ML agvTfTRo kz0icKCTNEHFM075Wg6xM3cwPsEeWncd9WK5IaSJiAjGz63t1emiyg1r50F4sxaMEVT3yifaeHJzxyPMBPr4pemTlgTq633Z1K2dXyVBde2A2y2qmm8bVBnwtVDHisoINetq+aS5yrGIOU54DJbG+qodoEvYvPZj457lwefgv3dDL/ncY7DBDJRecqFKr9ogijF/U/3k+q6/miQhlnOvkXoXnaeQLxxuqfWBSw8JroXCTQsHcvIhdBW7nVLx8e8x6fpga0brQT6j/PT9Q4lVoOO6VIZOGvQ+QEvBFJU1nXwv+2CkRdqscCKkq7jz8BiXbSJHsgYCQbGFPdDi+8j0aluhG+NRR4leIciRJrbDgDrlW7Ac1zgaJOGYD/3jrxffuttPIjciuc6LN13TJwNyrzBRzlt39DpdSjPTTDhxIrB7K5eCX574guqh0PN2524f7PpPrwMhCq7apzkOvFyRy7t5ILvkWSMjy3Vrn+mtYjt8A5TJrSQi32Dgg2AyjTIRHr/11AJ7sUfK7kCrgiUocGHOcTwUPoBGFQZwJbhPp1d05IfYxK3LjfIdJtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells cc: Tom Herbert cc: Tom Herbert cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/kcm/kcmsock.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index d77d28fbf389..9c9d379aafb1 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -641,6 +641,10 @@ static int kcm_write_msgs(struct kcm_sock *kcm) for (fragidx = 0; fragidx < skb_shinfo(skb)->nr_frags; fragidx++) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; skb_frag_t *frag; frag_offset = 0; @@ -651,11 +655,12 @@ static int kcm_write_msgs(struct kcm_sock *kcm) goto out; } - ret = kernel_sendpage(psock->sk->sk_socket, - skb_frag_page(frag), - skb_frag_off(frag) + frag_offset, - skb_frag_size(frag) - frag_offset, - MSG_DONTWAIT); + bvec_set_page(&bvec, + skb_frag_page(frag), + skb_frag_size(frag) - frag_offset, + skb_frag_off(frag) + frag_offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, bvec.bv_len); + ret = sock_sendmsg(psock->sk->sk_socket, &msg); if (ret <= 0) { if (ret == -EAGAIN) { /* Save state to try again when there's From patchwork Fri Mar 31 16:09:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF45CC76196 for ; Fri, 31 Mar 2023 16:11:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79C8B6B00BA; Fri, 31 Mar 2023 12:11:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74C7A6B00BB; Fri, 31 Mar 2023 12:11:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63B176B00BC; Fri, 31 Mar 2023 12:11:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 554176B00BA for ; Fri, 31 Mar 2023 12:11:51 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 32F3E1A0CA4 for ; Fri, 31 Mar 2023 16:11:51 +0000 (UTC) X-FDA: 80629684422.02.7F727D4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 68B6B4001D for ; Fri, 31 Mar 2023 16:11:49 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=adyVK3VS; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7fCBjYGLxHqikc/tNao5Lfq+HewWC5ZgTjyFTAyeW0Q=; b=sKvzLo4EJ3p6LnW6nH/z86Eu6QaptgM4vfQ7bCZi5jwupfS1MaCc5Q49OfKOBkKL2ZZGGG o1g7hMnSQtfREJWidK8A5U+NZTHa0QdpN+ejCwaQedZv8Z3dlaD9TLRz55Zw4+nIxuuJNi A3saPbC5+Oduuza0tGoDxEAkSYOYKoI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=adyVK3VS; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279109; a=rsa-sha256; cv=none; b=1NGfOIZzNECSWUudgeqULdeeKxSw0Wjhbm17e+NrjoJSTSMuOj2LUydJ0D0drplym496id izSOipli90LiWIBzz2+4LWoPqrz2Z0fg+PXIwLCUWTf7CPMejgPqw6fdGIaIMecqM2EuhE 04FTJ5IxHQbvE5fh+hw6FcVP4ERIHvQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279108; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7fCBjYGLxHqikc/tNao5Lfq+HewWC5ZgTjyFTAyeW0Q=; b=adyVK3VSaAJDnMu2axLqJz+WRzW1Vlcst0r6OfD2Oqi0mJemIc9Wd3QRfedNtei9n/UFVc dUBrlyQ6f2B/d4xMmcES08338NH94iOBDgcJITjBANjDm6caoB+fDQULgqRtKm2wps4oPo 1bTIv2mD3X0gkw0lfJHRq4V3q3OIWL0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-47-U_ILvhbNNGasHHvG4A8zOg-1; Fri, 31 Mar 2023 12:11:44 -0400 X-MC-Unique: U_ILvhbNNGasHHvG4A8zOg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3770D3C0F38A; Fri, 31 Mar 2023 16:11:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id AF40D140E94F; Fri, 31 Mar 2023 16:11:40 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Karsten Graul , Wenjia Zhang , Jan Karcher , linux-s390@vger.kernel.org Subject: [PATCH v3 51/55] smc: Drop smc_sendpage() in favour of smc_sendmsg() + MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:09:10 +0100 Message-Id: <20230331160914.1608208-52-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: er4kkoasne7tnhinkq6eszy7x18s6ziu X-Rspamd-Queue-Id: 68B6B4001D X-HE-Tag: 1680279109-877622 X-HE-Meta: U2FsdGVkX1/38FdMDI+i1NS23a4SGp7hxJMgdBBssauWaWVgznOvLpQh2XmzeKTTPNgMDsb0NXlWEDRtceUjk1vRt6AiTnAfD+Y9Zhn3DgHPAjKPA9c8ZVS04RoReoSwp8Xj2G/i3z/j+e7l4Op72+B9jxeL0xun7+DdIPfNfuYPNSUXX21WNFZBCynnQ7FOa1mgRk04vjxTR2xYJMgPI4q9+IBKDvP3YoZ1o9g1dYz80eo2np7IFWxitoZUMnW+alsvNK91zDGIWNLvoQoAdA0XX7sh1thd/+JCeS0ylb9pYq24ntMpOkCHHaesCep4SxZEVsrQA6bG1ZE+jCSHyByQtssY9Wf4QLqUlsBpIBQ6yGqlrMtcHcfq5VyEqHMLNr99Qot/Q8x6Z2cCWOQtLj5IqmQQMxPm6EgcoejNcTZmCyWU+Xn3UWt8Wz1yfW3XFY3rrfObWDofCmtwUD3ifgKPeYqfRXaaO5Rk9GW/OXfCnnLe02yVwU8WJJW0OIcYaF4+HG6azNLfwi/ndP/9tswNGXj+6prDKTRwZPoBmhqT1+5xFbnsHJYJw+7pbRhkCWVw7A1mNC1wK/+qT6Hn0Ns2qjVVC0DoDZW69LIss68W5ajs0kjVc8gMpx1gblY/3AiyqAEagDTSX0IQN/iLpUGkZBl4jZfKr2oJ/Z3CsiT3RoSVPGNxqfJjELotsYgcJeA5JAgWXgBgczrxhNFG2F/rioh7xeNjVkMOH63zDEV6GKe3lNUWgn0i8zspUdumcGKjGNr4ZSvycn6Noj8+/ZrlP2eetRhbThN3MPolxR+xkHg/UrEm81F1Guh0jEjNJ/FziEOADx+RYzIQKyHE/L3Y9e7vzzoxamVNKAGW0JxR/8XCvsD2yBSRepFom7m+DZkqvw6c88h2DuR1ypheMLnyBlPz5MfrqD8bEuDe17IkgG891jm5jM0Z/fDAb5L1tVNazEzJ/ibEJ+GF1sr H9QtGYbE FqP480GrEqzLA1C80Tq24DUxnOEmZJLuSxhQ2gk0ThgF7KNA24IL31nvoV08QEYXzISPxVAF3yoIm+N/CaIgwpw9tvoVvzCgzV+QMuNn3cugqoTxGxGhZJZueT7xTBaNdWpKhZpGAJ2No7jFBr3rK/smdL39GVdXWH1wVVwJbgYTgbTbN1RKDXFqRmb9poBpH63jRl0kmsVCErdCfUIwVSgHP3pnbGJpJlacaD/YgJ0kwml3tYxj/VO99MSqB0aUkkHOp4avFOCnzvYF42TUed3ufCLAhDyhnQhiWSFtEJxDeNt4Evz4sT1pkNwmsKXuXa+IEwW38sUkVzY9id+fhHbrvSWRh2lp3fiMyi1Wf2rFTKBqoMkqQy4jzVV2LvU3kiJsEE4JUYlBex4W1Tk54WSR1hacbEF86C4NNUkzrwnVTbUfw3j6w2yrc/ksJJzO0zL2psaYACHg8E8oJXwYmwRaoSg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop the smc_sendpage() code as smc_sendmsg() just passes the call down to the underlying TCP socket and smc_tx_sendpage() is just a wrapper around its sendmsg implementation. Signed-off-by: David Howells cc: Karsten Graul cc: Wenjia Zhang cc: Jan Karcher cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-s390@vger.kernel.org cc: netdev@vger.kernel.org --- net/smc/af_smc.c | 29 ----------------------------- net/smc/smc_stats.c | 2 +- net/smc/smc_stats.h | 1 - net/smc/smc_tx.c | 16 ---------------- net/smc/smc_tx.h | 2 -- 5 files changed, 1 insertion(+), 49 deletions(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index a4cccdfdc00a..d4113c8a7cda 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -3125,34 +3125,6 @@ static int smc_ioctl(struct socket *sock, unsigned int cmd, return put_user(answ, (int __user *)arg); } -static ssize_t smc_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct smc_sock *smc; - int rc = -EPIPE; - - smc = smc_sk(sk); - lock_sock(sk); - if (sk->sk_state != SMC_ACTIVE) { - release_sock(sk); - goto out; - } - release_sock(sk); - if (smc->use_fallback) { - rc = kernel_sendpage(smc->clcsock, page, offset, - size, flags); - } else { - lock_sock(sk); - rc = smc_tx_sendpage(smc, page, offset, size, flags); - release_sock(sk); - SMC_STAT_INC(smc, sendpage_cnt); - } - -out: - return rc; -} - /* Map the affected portions of the rmbe into an spd, note the number of bytes * to splice in conn->splice_pending, and press 'go'. Delays consumer cursor * updates till whenever a respective page has been fully processed. @@ -3224,7 +3196,6 @@ static const struct proto_ops smc_sock_ops = { .sendmsg = smc_sendmsg, .recvmsg = smc_recvmsg, .mmap = sock_no_mmap, - .sendpage = smc_sendpage, .splice_read = smc_splice_read, }; diff --git a/net/smc/smc_stats.c b/net/smc/smc_stats.c index e80e34f7ac15..ca14c0f3a07d 100644 --- a/net/smc/smc_stats.c +++ b/net/smc/smc_stats.c @@ -227,7 +227,7 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SENDPAGE_CNT, - smc_tech->sendpage_cnt, + 0, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CORK_CNT, diff --git a/net/smc/smc_stats.h b/net/smc/smc_stats.h index 84b7ecd8c05c..b60fe1eb37ab 100644 --- a/net/smc/smc_stats.h +++ b/net/smc/smc_stats.h @@ -71,7 +71,6 @@ struct smc_stats_tech { u64 clnt_v2_succ_cnt; u64 srv_v1_succ_cnt; u64 srv_v2_succ_cnt; - u64 sendpage_cnt; u64 urg_data_cnt; u64 splice_cnt; u64 cork_cnt; diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c index f4b6a71ac488..d31ce8209fa2 100644 --- a/net/smc/smc_tx.c +++ b/net/smc/smc_tx.c @@ -298,22 +298,6 @@ int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len) return rc; } -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags) -{ - struct msghdr msg = {.msg_flags = flags}; - char *kaddr = kmap(page); - struct kvec iov; - int rc; - - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, size); - rc = smc_tx_sendmsg(smc, &msg, size); - kunmap(page); - return rc; -} - /***************************** sndbuf consumer *******************************/ /* sndbuf consumer: actual data transfer of one target chunk with ISM write */ diff --git a/net/smc/smc_tx.h b/net/smc/smc_tx.h index 34b578498b1f..a59f370b8b43 100644 --- a/net/smc/smc_tx.h +++ b/net/smc/smc_tx.h @@ -31,8 +31,6 @@ void smc_tx_pending(struct smc_connection *conn); void smc_tx_work(struct work_struct *work); void smc_tx_init(struct smc_sock *smc); int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len); -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags); int smc_tx_sndbuf_nonempty(struct smc_connection *conn); void smc_tx_sndbuf_nonfull(struct smc_sock *smc); void smc_tx_consumer_update(struct smc_connection *conn, bool force); From patchwork Fri Mar 31 16:09:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 387F1C76196 for ; Fri, 31 Mar 2023 16:11:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D68226B00BD; Fri, 31 Mar 2023 12:11:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D18A76B00BE; Fri, 31 Mar 2023 12:11:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB9566B00BF; Fri, 31 Mar 2023 12:11:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AD9926B00BD for ; Fri, 31 Mar 2023 12:11:55 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 72C0DABA23 for ; Fri, 31 Mar 2023 16:11:55 +0000 (UTC) X-FDA: 80629684590.07.32DDC21 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 3ADE640027 for ; Fri, 31 Mar 2023 16:11:52 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Vp5jsb9P; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UieBw+fd7YOFzWwOOxTgfiaOtpF+6tMbkd08W9f/ez8=; b=1yo5MXXhuCZMVssBKXIjbDP8US+PwwnCcG4eC+qsYl7sGFpuL7S6acj3tTivAeinYsWCht WEmsSn6IH/QIc484V2u0rdqpp5eqBRRBNQJgERJ/VUDG3kxjbdfTmBu44ASKFf5HZU3WK6 PJPmUhUr8Ex7+ZqRuxVd8RpV0WnNowg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Vp5jsb9P; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279112; a=rsa-sha256; cv=none; b=McV+Y7FQxEgrJvHyl84VQyLfdAp0flcZthq8ny38qj0l9PjKxM1Sjop9tB8QMzQt8fKQIZ xGvyrc6OGscqfawzmdeZ+Ad8CGQN5TDp+l5iSD/YdrU8S+H/0CzfE/3tL2AcjB0ov1eNJ6 2/e3WLT3j2BxpGt9amaBJwNK1MhNRwI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279111; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UieBw+fd7YOFzWwOOxTgfiaOtpF+6tMbkd08W9f/ez8=; b=Vp5jsb9PSp0yzHzxUfv2Mop7Nq5uXqFqNnTu6bDdY3GPJxl8bpen1iJgeNtRbEa2GvXmKL +vuw3hrcoUCsAN0t7fGeH/z7OTjkFiokOaYxnBta4pnjWQw2HaHyBsXck/gP0PXe2cHDXk GCkh0vavqfYTjQFYWm1bIspD5kN5ieY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-383-U63PXVGmN_KbEEPQKdH0Bg-1; Fri, 31 Mar 2023 12:11:48 -0400 X-MC-Unique: U63PXVGmN_KbEEPQKdH0Bg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3E02A185A7A4; Fri, 31 Mar 2023 16:11:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD6B914171B6; Fri, 31 Mar 2023 16:11:43 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mark Fasheh , Joel Becker , Joseph Qi , ocfs2-devel@oss.oracle.com Subject: [PATCH v3 52/55] ocfs2: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Fri, 31 Mar 2023 17:09:11 +0100 Message-Id: <20230331160914.1608208-53-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3ADE640027 X-Stat-Signature: mu577cef3au9tcypefyt86b6g3mh35gy X-HE-Tag: 1680279112-546082 X-HE-Meta: U2FsdGVkX1+Z9Sx+/gYOMXJwcdqXicxzQT4pi8ihFFcXIOnJrRpilVpHh5MBPlnuBJTrwDu98HXwt/Cvmu9Px36wxqlxfzMfWGH3tqee/gnRND5w1NhPwFYoM8DmiDI4nnj2eQB5Da97nn9QufIOYWWRZok4lTs35kuZQO0Az4B0GLyMAQYvpoBS5qNL/CIfpn3VbVeUSeUULl9xPnU3te0FurKJRNrNRKzWLvdqdxFyghdIo1CgmQQN24aLvHSKKYULig5/pCzhmOAzcnipQlRc5k5rjE6hQKnHZSdikQQrLYjaGSaYPtNSIyp9yxV2yL7Q9Pe2clJuVTkvI9yRL3WuH+NlWjomFhNLgpwvvbL8qjKvYAqppqujZ3bkW7A/aw1IEIZ0ewM1QdfndditW3WXrn4L+28SmfQwSvTfGA43+PzWpEyGCCgI99zC9W+qWFnePosB1ekE7w4XfgL+jCsOP/Q0486mLNqREeoLattPBIUvPSCzV1PYY07QLH/rRMOPgAXY7DREBLlfiOQ9GBQwa1TqcwlyQWzTGqsyHQBM7PNpRW2xe8Cb1r6MMr6mksHu0E5mTQZvs1Y+cS85D4BQ8Dr7kSmHfHqyG0rYPRvM5Rh/lRf+/trGJmm960Hd5nXUfzvuzLmIgq6PKJesYD37O3oY4UxBbGJ+FeJBXxQrzEzDzIxJ+lAS5uS4bBvZ8xfu+I0yQmZsoowPHTsBNiEIwE/xyLMet7gxGs/kdBZ98cRH5EiVOL1M/elUxBmgx4qlcM4dDCzVQS4EkvPBvSdcNJ3Uztbug7xIOqdC/Y3rsX8qkrSgKoxCFUSc6N+5PwHAsYWufxQ4fljhJ+DA2gG3oZdwkD0OFLwY9O8N1VK9ZCiWohcqawCX8SnYhICrrhzYPR09Nr4HLVZii9vjfkltbkbFNMLLIcveocOPLSchuhgyQjNpDlVNEzQfB/+1/klc7VdltWfogscqzgI emUbdUjj z/WiK0Sl+Sn4sZ0DgVl89ge570zzwTf6mxaf6boj/O08zbKxxECgo2MweFw7PG760QVwrZCgc/H7DdrFamb3EvmNtg7wbrvHIcmIxAtZSRQZ8SEkXolojscjWutJaow5W+aEKoFbSlAJTuwnpg/jsaTA/h6Gh+id6u5p+4dZC1kvkhfSALUwSxInDoNBvOTaQ2KJgWfmEtzC3o+PNEsGxFFrV+ovJewe7uN9o0t1k7piEhk196+hyTuFFZlNmRiFGkFX8d4qB2AKwasZLL7BmIoiv2FYlEGlOlemgN7kuelLx7i6bBl5uEh9KhNlZuB4Ki7cf3y2Gi6QAyR6FibuGjJJglBm0sqlukARVCPGR8xI1TPwFhLumqRYOam0cinzFbxOS5peW0q8PQs6vZ4a71WX3cdnsjcbSDCZaJkFd5Fgj7bTXPfpjEGruhyrkykD+AfLFfwMMZ7bFEAFYSAAABbQWIrBftt4G8Pe6w9j46pM6EiiiufaAfzLMGj2yRpB678NUDEmhmCUQwmA/qQ0/74nIF63rCo2rHzzrolkvLFIXoTo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix ocfs2 to use the page fragment allocator rather than kzalloc in order to allocate the buffers for the handshake message and keepalive request and reply messages. Slab pages should not be given to sendpage, but fragments can be. Switch from using sendpage() to using sendmsg() + MSG_SPLICE_PAGES so that sendpage can be phased out. Signed-off-by: David Howells cc: Mark Fasheh cc: Joel Becker cc: Joseph Qi cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: ocfs2-devel@oss.oracle.com cc: netdev@vger.kernel.org --- fs/ocfs2/cluster/tcp.c | 107 ++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 49 deletions(-) diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index aecbd712a00c..e568ad2f34bf 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -110,9 +110,6 @@ static struct work_struct o2net_listen_work; static struct o2hb_callback_func o2net_hb_up, o2net_hb_down; #define O2NET_HB_PRI 0x1 -static struct o2net_handshake *o2net_hand; -static struct o2net_msg *o2net_keep_req, *o2net_keep_resp; - static int o2net_sys_err_translations[O2NET_ERR_MAX] = {[O2NET_ERR_NONE] = 0, [O2NET_ERR_NO_HNDLR] = -ENOPROTOOPT, @@ -930,19 +927,22 @@ static int o2net_send_tcp_msg(struct socket *sock, struct kvec *vec, } static void o2net_sendpage(struct o2net_sock_container *sc, - void *kmalloced_virt, - size_t size) + void *virt, size_t size) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct msghdr msg = {}; + struct bio_vec bv; ssize_t ret; + bvec_set_virt(&bv, virt, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, size); + while (1) { + msg.msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES; mutex_lock(&sc->sc_send_lock); - ret = sc->sc_sock->ops->sendpage(sc->sc_sock, - virt_to_page(kmalloced_virt), - offset_in_page(kmalloced_virt), - size, MSG_DONTWAIT); + ret = sock_sendmsg(sc->sc_sock, &msg); mutex_unlock(&sc->sc_send_lock); + if (ret == size) break; if (ret == (ssize_t)-EAGAIN) { @@ -1168,6 +1168,7 @@ static int o2net_process_message(struct o2net_sock_container *sc, struct o2net_msg *hdr) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct o2net_msg *keep_resp; int ret = 0, handler_status; enum o2net_system_error syserr; struct o2net_msg_handler *nmh = NULL; @@ -1186,8 +1187,16 @@ static int o2net_process_message(struct o2net_sock_container *sc, be32_to_cpu(hdr->status)); goto out; case O2NET_MSG_KEEP_REQ_MAGIC: - o2net_sendpage(sc, o2net_keep_resp, - sizeof(*o2net_keep_resp)); + keep_resp = page_frag_alloc(NULL, sizeof(*keep_resp), + GFP_KERNEL); + if (!keep_resp) { + ret = -ENOMEM; + goto out; + } + memset(keep_resp, 0, sizeof(*keep_resp)); + keep_resp->magic = cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); + o2net_sendpage(sc, keep_resp, sizeof(*keep_resp)); + folio_put(virt_to_folio(keep_resp)); goto out; case O2NET_MSG_KEEP_RESP_MAGIC: goto out; @@ -1439,15 +1448,22 @@ static void o2net_rx_until_empty(struct work_struct *work) sc_put(sc); } -static void o2net_initialize_handshake(void) +static struct o2net_handshake *o2net_initialize_handshake(void) { - o2net_hand->o2hb_heartbeat_timeout_ms = cpu_to_be32( - O2HB_MAX_WRITE_TIMEOUT_MS); - o2net_hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); - o2net_hand->o2net_keepalive_delay_ms = cpu_to_be32( - o2net_keepalive_delay()); - o2net_hand->o2net_reconnect_delay_ms = cpu_to_be32( - o2net_reconnect_delay()); + struct o2net_handshake *hand; + + hand = page_frag_alloc(NULL, sizeof(*hand), GFP_KERNEL); + if (!hand) + return NULL; + + memset(hand, 0, sizeof(*hand)); + hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); + hand->connector_id = cpu_to_be64(1); + hand->o2hb_heartbeat_timeout_ms = cpu_to_be32(O2HB_MAX_WRITE_TIMEOUT_MS); + hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); + hand->o2net_keepalive_delay_ms = cpu_to_be32(o2net_keepalive_delay()); + hand->o2net_reconnect_delay_ms = cpu_to_be32(o2net_reconnect_delay()); + return hand; } /* ------------------------------------------------------------ */ @@ -1456,16 +1472,22 @@ static void o2net_initialize_handshake(void) * rx path will see the response and mark the sc valid */ static void o2net_sc_connect_completed(struct work_struct *work) { + struct o2net_handshake *hand; struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_connect_work); + hand = o2net_initialize_handshake(); + if (!hand) + goto out; + mlog(ML_MSG, "sc sending handshake with ver %llu id %llx\n", (unsigned long long)O2NET_PROTOCOL_VERSION, - (unsigned long long)be64_to_cpu(o2net_hand->connector_id)); + (unsigned long long)be64_to_cpu(hand->connector_id)); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); +out: sc_put(sc); } @@ -1475,8 +1497,15 @@ static void o2net_sc_send_keep_req(struct work_struct *work) struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_keepalive_work.work); + struct o2net_msg *keep_req; - o2net_sendpage(sc, o2net_keep_req, sizeof(*o2net_keep_req)); + keep_req = page_frag_alloc(NULL, sizeof(*keep_req), GFP_KERNEL); + if (keep_req) { + memset(keep_req, 0, sizeof(*keep_req)); + keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); + o2net_sendpage(sc, keep_req, sizeof(*keep_req)); + folio_put(virt_to_folio(keep_req)); + } sc_put(sc); } @@ -1780,6 +1809,7 @@ static int o2net_accept_one(struct socket *sock, int *more) struct socket *new_sock = NULL; struct o2nm_node *node = NULL; struct o2nm_node *local_node = NULL; + struct o2net_handshake *hand; struct o2net_sock_container *sc = NULL; struct o2net_node *nn; unsigned int nofs_flag; @@ -1882,8 +1912,11 @@ static int o2net_accept_one(struct socket *sock, int *more) o2net_register_callbacks(sc->sc_sock->sk, sc); o2net_sc_queue_work(sc, &sc->sc_rx_work); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + hand = o2net_initialize_handshake(); + if (hand) { + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); + } out: if (new_sock) @@ -2090,21 +2123,8 @@ int o2net_init(void) unsigned long i; o2quo_init(); - o2net_debugfs_init(); - o2net_hand = kzalloc(sizeof(struct o2net_handshake), GFP_KERNEL); - o2net_keep_req = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - o2net_keep_resp = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - if (!o2net_hand || !o2net_keep_req || !o2net_keep_resp) - goto out; - - o2net_hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); - o2net_hand->connector_id = cpu_to_be64(1); - - o2net_keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); - o2net_keep_resp->magic = cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); - for (i = 0; i < ARRAY_SIZE(o2net_nodes); i++) { struct o2net_node *nn = o2net_nn_from_num(i); @@ -2122,21 +2142,10 @@ int o2net_init(void) } return 0; - -out: - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); - o2net_debugfs_exit(); - o2quo_exit(); - return -ENOMEM; } void o2net_exit(void) { o2quo_exit(); - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); o2net_debugfs_exit(); } From patchwork Fri Mar 31 16:09:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9693FC761A6 for ; Fri, 31 Mar 2023 16:12:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 355F06B00AA; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 306076B00BF; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CE8A6B00C1; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 066276B00AA for ; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9DF801C6BA8 for ; Fri, 31 Mar 2023 16:12:04 +0000 (UTC) X-FDA: 80629684968.12.35E3D3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id C772110001F for ; Fri, 31 Mar 2023 16:12:02 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=embEthkB; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279122; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZJvfcCj7beM2l3aL2q8ZqMjrzC72DifYH9bqCAjpsUU=; b=384mFAJvm9tLFc5kgTQbgTvKJW5kUG5W5ILu5nDMkhYWiN5zBiRk1xzrX8z67E6uLkgFyY Srhd+RGtEBSKcJBzWi0V94bskYkGwzUxAwQd2ZpwH9LcuVBsYXb2/3NiGzEEMHLD921Lon z3Ra9HxEv+rM8amLqMm81iT9X1OQvjc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=embEthkB; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279122; a=rsa-sha256; cv=none; b=rZoJ3yQKkhzyul6U177m1nvhMe1mcyUxfhshAJ1sg+/+GxsPW5C9zZ30B49nSsStHZih3U aqKfqvMCeej/JiEkMAR/YC5bZPSQFXim8dMCxz5qXUWQmBKliPXLnF/mMMuF7RZoSARJDf cXLJmNGLjNY0nFp5b+sH1SxavguSmeA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZJvfcCj7beM2l3aL2q8ZqMjrzC72DifYH9bqCAjpsUU=; b=embEthkBFt7qj12O8BEqCwkRtC1P4Dtpjnshepl6p8KR90nmwO5yN3Og151lB63h+OEc7I el11xgH3pmNIeCEzsCQuntOrx//9ep8xaEjRrJKzd08ZcOUu+wu50efD7PLECZf8aH/5B2 AHjnp+wb5w+QEt57PY/29TeoNqJLesk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-207-st4ET5OBOuuljcmLdN9DFg-1; Fri, 31 Mar 2023 12:11:56 -0400 X-MC-Unique: st4ET5OBOuuljcmLdN9DFg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 39B1529AA2DF; Fri, 31 Mar 2023 16:11:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CE1C52027040; Fri, 31 Mar 2023 16:11:46 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [PATCH v3 53/55] drbd: Use sendmsg(MSG_SPLICE_PAGES) rather than sendmsg() Date: Fri, 31 Mar 2023 17:09:12 +0100 Message-Id: <20230331160914.1608208-54-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: qxa8ndwzjodupmd7fzfzpj8mztor1at4 X-Rspamd-Queue-Id: C772110001F X-HE-Tag: 1680279122-162117 X-HE-Meta: U2FsdGVkX188ViwA3gwyXTP5SW4n18GX2MhxjiSgfeND2RL5jdOnZt9ZpwwcU8vjUicrQhEgt1w7X7bmadknCxLmXkvqyV8NmxWUNDpPxTQH6s+OtcvnLsT4EzZV9F0b2zLdrsmEyfIvG5dSgAQTKnOf1fPGGOKtEFlZIuBSfgfpakxd/QjFVfCnEzBH6etxJNPNSL/S8XGMymIuwzkA0LKojLHFPIoorS/QMDIzrjlFzG9D76/orBDshNoq2VWc9+0xC+oQIjScvRNEGds5NJ1AV/6gqMyPjPyaB9Mw92g+5YAd5HG8+JOMst1cqsb4KJc0qhukBBEelZsWfGpfpx8HimX7AA+3LEn/9UffTeDfIHyjxhCczXgsi6F4Bhe7Xi+xe+4RyXglUXGoiddJMcmP5eNx1+vDsafHYDxBSvNtlI5WzUM2PE1erLpnvW2qhpXGOd6hTOrkwcSk0IBXjUlGNCSUXGpRdeXZmi24XnPAiI6rA/DWkKvcOfqiKIOipCbS0u4A5xGK9872q2cRvg44admET8lx9M2WyyBilRxLLq5T9+hfJuoqHdDm3FnrTCLa+Hfer1DFl0uyFrtsMGFFMOYbOle0GwxIsON32EvkAf645z4hs9ksLvmvpsQbSwyC88Zq2cp8WmlOv3wdbodQd9SyuJb4cRooBTvjOdv28+qtIzsjHJ5mbDF9aRmvtwb7PCJP7tCaOn3ZXQvH+U75RF3PESQ7ND4HSKsPQD68Qa0GVE950mGuENlv2R3MTwbyYALhiN3HWKDqFe9y59JnKnqD6UmwZPk+maO/JnGur4To6mXyIbDTTJSI8KGF9CiobjwOhVi9MRxfPp5EeT1Lxlzg60W5D7kKB2fHJCpSa6aTOa21/W08LeK+ZYWB4COocjsShkjJLx9uukp2LmSQQKVr5SV7AE+9Whf3xtAQK1My2Vo9wYrGpyoc97zZ9UxRX+IbOViP/Pa7zlH n7iMWRHY R7LpRYeaX4CuQB0PRpcoWa3nSFIrM4dup1Xql5S1jqzw32ZJmXWgoZll4SjtsVi8OkN1lSgwQ/1Cn26bECWBpr8ydR2j37Fd7U7/f5O5CnfqW8cagcPymmhCIrbtcBn3YSeM3j69Es+0zMAHXYnjFEArdO2Mev0TWd47hOY1sMnvugOS1SMyXYNKGvyU7iJ/ZV/jkcpQrTXqyyrdX9P4bjbuCODQd40ar5Ww3FOTOVZYul6on9Ege1wgR02t0ulzFBXVrUzCTXX45BlLO7xRGeQQVheXsUe4lLFrFalP7Jo/tOiiuTz9J2gDzDRSCX/rdh6KuyY1o4aV37DaAvsrZaXf32oxwDR123HlYFheEPC0v1Id2C9cNP0TwuI7iuHLmFEslN8Sz7kALFZq07/9lQtdKsUs8iqCYKDT84MmQmxRd/c0gSIOiFbsPCFMoHI92FPMB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() conditionally with MSG_SPLICE_PAGES in _drbd_send_page() rather than calling sendpage() or _drbd_no_send_page(). Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/block/drbd/drbd_main.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 2c764f7ee4a7..e5f90abd29b6 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1532,7 +1532,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa int offset, size_t size, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; - int len = size; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = msg_flags, }; int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1541,33 +1542,33 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (drbd_disable_sendpage || !sendpage_ok(page)) - return _drbd_no_send_page(peer_device, page, offset, size, msg_flags); + if (!drbd_disable_sendpage && sendpage_ok(page)) + msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; + + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - msg_flags |= MSG_NOSIGNAL; drbd_update_congested(peer_device->connection); do { int sent; - sent = socket->ops->sendpage(socket, page, offset, len, msg_flags); + sent = sock_sendmsg(socket, &msg); if (sent <= 0) { if (sent == -EAGAIN) { if (we_should_drop_the_connection(peer_device->connection, socket)) break; continue; } - drbd_warn(peer_device->device, "%s: size=%d len=%d sent=%d\n", - __func__, (int)size, len, sent); + drbd_warn(peer_device->device, "%s: size=%d len=%zu sent=%d\n", + __func__, (int)size, msg_data_left(&msg), sent); if (sent < 0) err = sent; break; } - len -= sent; - offset += sent; - } while (len > 0 /* THINK && device->cstate >= C_CONNECTED*/); + } while (msg_data_left(&msg) /* THINK && device->cstate >= C_CONNECTED*/); clear_bit(NET_CONGESTED, &peer_device->connection->flags); - if (len == 0) { + if (!msg_data_left(&msg)) { err = 0; peer_device->device->send_cnt += size >> 9; } From patchwork Fri Mar 31 16:09:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D89EC761AF for ; Fri, 31 Mar 2023 16:12:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53D7E6B00BF; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DF8B6B00C2; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 269CE6B00C0; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0F8D06B00BF for ; Fri, 31 Mar 2023 12:12:05 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D9706AC8F8 for ; Fri, 31 Mar 2023 16:12:04 +0000 (UTC) X-FDA: 80629684968.08.FDB7F7F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 91758100019 for ; Fri, 31 Mar 2023 16:12:01 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JjyINlia; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/kAjj+4zLWTXkqKqP2B/MWL9umh9MFmha+7vx/dBng=; b=Ic5CtP1FXyVzwFeA9k25ZpvA5xKFBnvMxWuNdTIsQw6N/E/2H8Le8JCKLje3Hhro7n0t0H HoJbsWD7DJg/DLJ3PMK91YzuPaIArAuRn/GAo6odNCjQ0Zu0efCgoBoVY3pKkq204TQd3V opK7ucmQQ1C/eWUTQYV6iNvETjXBsEw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JjyINlia; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279121; a=rsa-sha256; cv=none; b=x5QHJJk69SwiIZ8YX1ST/vtjqTzdoDcGagLTPjF/5aXADm/dSppd4bDJnrP6Rsy0intPUU XItgb4MiG8SmVHSZc2TpXYqKwN7Y+/ODZGtCcUWDWF6HiiRZGIGQIdb4lFBIYZxCxGG/vV tnbEOXi+HIjRTwsvwBW+nnXpzSsfX+c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q/kAjj+4zLWTXkqKqP2B/MWL9umh9MFmha+7vx/dBng=; b=JjyINliaaP+WnkIBXKWQlZTtxJKgC2AOz3JLneeMsTPQIo9vCluBKu+G55OrVg75WQRF/6 PCy8UycEEasWbnzKVS2W+eRTZlcTVqLcoSO0e51W2Od9IDlqA+ketbvoQAvEz+peNcdiWk t/UjSP/NhPgxgKMfluB6hutivCQ3lZo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-54-NrSaPjGbPqK-bssP12GZRQ-1; Fri, 31 Mar 2023 12:11:53 -0400 X-MC-Unique: NrSaPjGbPqK-bssP12GZRQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48C5E29AA2CF; Fri, 31 Mar 2023 16:11:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id E198914171B8; Fri, 31 Mar 2023 16:11:49 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [PATCH v3 54/55] drdb: Send an entire bio in a single sendmsg Date: Fri, 31 Mar 2023 17:09:13 +0100 Message-Id: <20230331160914.1608208-55-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Stat-Signature: 5kx1bsd9op5kqcg945uojkbug6s3ujj6 X-Rspam-User: X-Rspamd-Queue-Id: 91758100019 X-Rspamd-Server: rspam06 X-HE-Tag: 1680279121-460054 X-HE-Meta: U2FsdGVkX1+NPIHNv73lWu72kKId6UtB15F9GN9rUlDxOPUrV02rATzAxQVBxEoQSDHczQ34D79N5cBazHb/qDdQISXFe+7Nsj37wx3XelhdFVK1ueGmG2RggZBiaL8C7k7JqPsTG87OBYrBZssG4lw0h+Q/6d1ntb5NVouK3HJbbLXoW87awcuFZNOn1s0RRP19FnhHrgEuHLqcs8rQK6eDxBpRqYovcw6b4bUePvfgdBFHpf7sXbbvwQUBWpcftuXVX953uvHp2j2ZilnAs7ZXnplFHqJ8MN0sdEP1CyPTLJTGUYQxhQFgtEDz3fBIgwvKVGgoNp2z4xspJFFoXoxfppsynQ/81UPNBHjZd0UFQjiCypXGJq0Z62588QUT3hTURgn11x+bZFZO9vLfUl5c0y5Thiu2z4zIXK4rGxyxsZweb4TcyalUx4qbL3qutoSzDLbYIrNCN+rXMZYQaqv9mv+pO3P7fkEcbk42iDdy3xYFKmr4xL1bItF3YeSOsDJWshuTWAc0AHMCUCUD3kut5LEdLSBoW7Jq6ZjkalhPqlu0QtYdq34imEZc+myYPv1jUxntGBOFSdO/n0bBBOBUTP+0BlBZJ4Il2FRGOB0g8Z0fFz+A1VBFxYtwi7vsvX0bbISD51GQ9KS44laOK6IRDg8b1+QANXsISGBUQpb6fSaIDgt5GM0Apun4r5w9ZtQ3M5f1wp0lZ+4PPn+kci5KGXDameD59sUC0NfIRxHi613QaMgMxl9yzLOPbq++sE5A/SpvEQtc8BpoiZCciNFaWx7NnO13bCeXEz2kh+DmM2I3XlNmAcX1/PQ+AhvQXz1vcnRRJVIKi/W+hX1JIp28zeY6W2Ce5DGTtDw13IztmfBJf9mOrGLD2AV2iVR5pqei6F/XNHkCcs4G5vqSn/pttU/f08o81zvIuVcrjlJ5DX4peSP/bnpqqJ4XlkEmwwl2nrYRwPztzwg2sly DXG4A8CN aP8e5tjWLlWfbicXfYQbOzPj9Ny7EFAXNpjtOgEBpOs+sfXqGqHQIxtQ7a6doFlf9j6AFOZQdcaO2KdRgYfC2D/vslNuVHF+SuhidEURMCx6iOxHs5LBmZcjXQyJtquIAu7Iy6RFnCjtfhxXs/s23DwxZfID86oExoFFGIYrtX4k9rvMPWIEy9uldFk8zYYHlFACICdSM4A0ys/BJo0JPYhEWaXf7vpH3zhuf/NQtGG6SGU35+we+rum0fscWSYQ7YRsO+5NtQisvkRvx7endS7WW7h3nE3s0/Xa+ij7s6joyWPLii4SBrx2Y9B9DEQKwiaLtEM+NPKTljewLxB0+Oj03taeCyhIdJvSSb+e/mEOFD6ZikSDWGGKE5H8Loja9OwNNn4PxuWr6QckolDz8TV1Fmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since _drdb_sendpage() is now using sendmsg to send the pages rather sendpage, pass the entire bio in one go using a bvec iterator instead of doing it piecemeal. Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/block/drbd/drbd_main.c | 77 +++++++++++----------------------- 1 file changed, 25 insertions(+), 52 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index e5f90abd29b6..ab63d6138407 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1512,28 +1512,15 @@ static void drbd_update_congested(struct drbd_connection *connection) * As a workaround, we disable sendpage on pages * with page_count == 0 or PageSlab. */ -static int _drbd_no_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) -{ - struct socket *socket; - void *addr; - int err; - - socket = peer_device->connection->data.socket; - addr = kmap(page) + offset; - err = drbd_send_all(peer_device->connection, socket, addr, size, msg_flags); - kunmap(page); - if (!err) - peer_device->device->send_cnt += size >> 9; - return err; -} - -static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) +static int _drbd_send_pages(struct drbd_peer_device *peer_device, + struct iov_iter *iter, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; - struct bio_vec bvec; - struct msghdr msg = { .msg_flags = msg_flags, }; + struct msghdr msg = { + .msg_flags = msg_flags | MSG_NOSIGNAL, + .msg_iter = *iter, + }; + size_t size = iov_iter_count(iter); int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1542,11 +1529,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (!drbd_disable_sendpage && sendpage_ok(page)) - msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; - - bvec_set_page(&bvec, page, offset, size); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + if (drbd_disable_sendpage) + msg.msg_flags &= ~(MSG_NOSIGNAL | MSG_SPLICE_PAGES); drbd_update_congested(peer_device->connection); do { @@ -1577,39 +1561,22 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa static int _drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_no_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) - ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, 0); } static int _drbd_send_zc_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, MSG_SPLICE_PAGES); } static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, @@ -1621,10 +1588,16 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, /* hint all but last page with MSG_MORE */ page_chain_for_each(page) { + struct iov_iter iter; + struct bio_vec bvec; unsigned l = min_t(unsigned, len, PAGE_SIZE); - err = _drbd_send_page(peer_device, page, 0, l, - page_chain_next(page) ? MSG_MORE : 0); + bvec_set_page(&bvec, page, 0, l); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, l); + + err = _drbd_send_pages(peer_device, &iter, + MSG_SPLICE_PAGES | + (page_chain_next(page) ? MSG_MORE : 0)); if (err) return err; len -= l;