From patchwork Sat Jun 17 12:11:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36379EB64D9 for ; Sat, 17 Jun 2023 12:12:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B4B26B0072; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 664E56B0075; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52C848E0001; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 41B776B0072 for ; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 043B01C8629 for ; Sat, 17 Jun 2023 12:12:24 +0000 (UTC) X-FDA: 80912127450.14.9A280DA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 3BFA24001C for ; Sat, 17 Jun 2023 12:12:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BzkIuIlI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003942; a=rsa-sha256; cv=none; b=GEoaYnkAcOD+u7aZDGDQBN9bYyoTXedk0QiYd7MpAkpENQqu62eqfI+faeAXYvZTCJhCgZ fgVu114zwU9PDwuM/6OoOmFhevlppoJNOZ60JuZLRTME8luYM781ZaWGdN740CTikkATKO mxiyFOyFEvBwH+oTUV0StIDrFdjMExw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BzkIuIlI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003942; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ejBheymwORocF7PA51bmsdG6ZmxRm3UYoYQNGQf509Y=; b=e8q6Xkaxe1ZAxER4HxOJFl5tfFtCA/9ZmCvtil4QqiGwGPl2buWyNQ9OVAw7eskXXQYRG+ 0Q9DEQlHee+4pqoqrofuP7i5XNiDsk21RGlOC9a7LU+lT7jjdy1tG0DdaKsKyVRQS9YkOQ YC04wA6j18PaQWNdP/iQoHQVo6gWDag= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003941; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejBheymwORocF7PA51bmsdG6ZmxRm3UYoYQNGQf509Y=; b=BzkIuIlIHwzYdxlPjqbA7VmNml5qsUKBZx9jaOkDkiDEvNeN8rLpf41nt2S4TMXfzLUUPR t56I/ahkBZ4+9yEeX0I8s+q1XkmVHMsrhfO+WBVIeztxRa3RZybp4EwKrxDPhLF0UVOyrE 3bty1MbCwSTQrXJLHdUsF0JcHfZ5lcI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-372-w6pbkuscPMuYgv1oelN8Vg-1; Sat, 17 Jun 2023 08:12:17 -0400 X-MC-Unique: w6pbkuscPMuYgv1oelN8Vg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1221E8352D1; Sat, 17 Jun 2023 12:12:13 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 40202422E2; Sat, 17 Jun 2023 12:12:11 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH net-next v2 01/17] net: Copy slab data for sendmsg(MSG_SPLICE_PAGES) Date: Sat, 17 Jun 2023 13:11:30 +0100 Message-ID: <20230617121146.716077-2-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3BFA24001C X-Stat-Signature: u3gcoo8579mqre3auds8uaaqi1dn69d4 X-HE-Tag: 1687003942-688314 X-HE-Meta: U2FsdGVkX193THU+eArkKZx7Z3WLJmpVGIDbY5jN6+I4YQMzNXcwZG9JFBZSaKBrk2M8tMqjEGFOzrdkJUGoD/TmQAqiMTyknA6Flu7jy1/5pNO79BTD4UG6DQXD4whdR4y+dWvH+tRQHzRJIe6gj6D0rbUey0oWwOpwFPYv3SrvLLQkNibEa0eacQrdSW1RWc/ooWkflbmX3fgitX1D5K6JKSMQDrHNQWYGQYnc9aotFATLuZGMamHtXcmUXpiSBJzUNW1AUv8SYTDADF5DSHGxJwtadLUvgKeFkyRRb6SQZO3dwqd3HLL+hEVsiYQVQWERk+3EC5YEm0R/Xr0ntwOTMj+sOroXpqEwvVZ4wGFCR3zuV36DFGr0rO8qqxrzMz1LC/tijeY+euSVvCCaYyQErrekHFIeP2cgHEcWeEPlcqWxybL8eEN5GtUPhMupRvh0Td+6up4PlC9/jRmvhS/9SHpY0VUn/Sij4+eWtR0WdwoEmpxRd0uwPWtBvZlhOR+A9/eF+nBifilriTq5YZDwraBFT/thVpVkjoXsdNfg0rW1JxYFAAEzR3w6i9dMlJiYJj0dy1r8y9gts4AQz5EA0WPqjocconJfQO8I2cFMA5vVzixVA1xU5gfFz0uIn4rHQgLH13S1+z1tjH8acnEcTD19DN+l0M0Fh2hOskpy8kM1lTeOERIkop+0KAALh0JlDyBCCVNba6dEPpS8GeNti01Xm/NqYPYYIF1nIs8fzGv0nUsCVW+Imo2Oo/OrcIEWdAwWGdeY3qQuPmy4xgKUUev9qQgBkn5FOVQFGz3j9PoDeeo4wQtH0hkxXAbZ3o8ctK9XIDpYTkwbSX0cbhMZHXZpsiwkPQYFl8czBxyj9YT99b/HaZoT5Y6MEYhei/sc1tF7X0q+mZqeljQZxq6krZewcpH2mGPx3oKc/90zoK7I03E84SWFAzMwVHL9lKPAUHzeRkA1zWP3u7T Ll+Clbi8 PfxZkxcu7nDfAjtAIhkmYZBaj4iuHtL+a2LsAL9+wev40rkoTpNDvzp0rbSiCDYhka2gn2FohDPaC6NqjsPsO0dcZ/DtnGBKXOiGReX/GNmccnz7M4aHH4yNodDmz43EZuzV6RAwlYy4Zse2pWyDQlzhWxU8Ll3eUOoHo4yI75NvQYAb2gAgiHpOwoSPGWLl6xwtHABa4tiUrtDf5PFGXfViTuDYjcxn15J00pL0fm8CfShTMQaA9dIIhgHyUtcLNPxJggvcBHWKRzNAZaTBuM1K+V9sJ6HB1x9aFPMk/p1OhLsJBHrygitp4lcZLhhmC3i/Y1l1eTNNRfDICoyDFcwvknVTscFea1Antr9280lgbfni6tZLmPhCF4J+wlVXDV01yZgBE+xRP5J6mF+SQAB4dYIjTQLtj6ZYmasDDcTmwYQvSrSTkcRBuMJRWQP5k2IsOsL8HE/MSnjwwP3IxE7+tZjL6h7xN8dZNlK2i5A/YnQxCMa8sKQqVYNL8m8bp1abpbxIaFVabMh+qFP87fqwd2MKdjd97BXtBcMllzS/W74ezjwrcNBG3yTfElmYjvE7GLphUFtB3QOA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() is passed MSG_SPLICE_PAGES and is given a buffer that contains some data that's resident in the slab, copy it rather than returning EIO. This can be made use of by a number of drivers in the kernel, including: iwarp, ceph/rds, dlm, nvme, ocfs2, drdb. It could also be used by iscsi, rxrpc, sunrpc, cifs and probably others. skb_splice_from_iter() is given it's own fragment allocator as page_frag_alloc_align() can't be used because it does no locking to prevent parallel callers from racing. alloc_skb_frag() uses a separate folio for each cpu and locks to the cpu whilst allocating, reenabling cpu migration around folio allocation. This could allocate a whole page instead for each fragment to be copied, as alloc_skb_with_frags() would do instead, but that would waste a lot of space (most of the fragments look like they're going to be small). This allows an entire message that consists of, say, a protocol header or two, a number of pages of data and a protocol footer to be sent using a single call to sock_sendmsg(). The callers could be made to copy the data into fragments before calling sendmsg(), but that then penalises them if MSG_SPLICE_PAGES gets ignored. Signed-off-by: David Howells cc: Alexander Duyck cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: Menglong Dong cc: netdev@vger.kernel.org --- Notes: ver #2) - Fix parameter to put_cpu_ptr() to have an '&'. include/linux/skbuff.h | 5 ++ net/core/skbuff.c | 171 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 173 insertions(+), 3 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 91ed66952580..0ba776cd9be8 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5037,6 +5037,11 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +void *alloc_skb_frag(size_t fragsz, gfp_t gfp); +void *copy_skb_frag(const void *s, size_t len, gfp_t gfp); +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, ssize_t maxsize, gfp_t gfp); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index fee2b1c105fe..d962c93a429d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6755,6 +6755,145 @@ nodefer: __kfree_skb(skb); smp_call_function_single_async(cpu, &sd->defer_csd); } +struct skb_splice_frag_cache { + struct folio *folio; + void *virt; + unsigned int offset; + /* we maintain a pagecount bias, so that we dont dirty cache line + * containing page->_refcount every time we allocate a fragment. + */ + unsigned int pagecnt_bias; + bool pfmemalloc; +}; + +static DEFINE_PER_CPU(struct skb_splice_frag_cache, skb_splice_frag_cache); + +/** + * alloc_skb_frag - Allocate a page fragment for using in a socket + * @fragsz: The size of fragment required + * @gfp: Allocation flags + */ +void *alloc_skb_frag(size_t fragsz, gfp_t gfp) +{ + struct skb_splice_frag_cache *cache; + struct folio *folio, *spare = NULL; + size_t offset, fsize; + void *p; + + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; + + cache = get_cpu_ptr(&skb_splice_frag_cache); +reload: + folio = cache->folio; + offset = cache->offset; +try_again: + if (fragsz > offset) + goto insufficient_space; + + /* Make the allocation. */ + cache->pagecnt_bias--; + offset = ALIGN_DOWN(offset - fragsz, SMP_CACHE_BYTES); + cache->offset = offset; + p = cache->virt + offset; + put_cpu_ptr(&skb_splice_frag_cache); + if (spare) + folio_put(spare); + return p; + +insufficient_space: + /* See if we can refurbish the current folio. */ + if (!folio || !folio_ref_sub_and_test(folio, cache->pagecnt_bias)) + goto get_new_folio; + if (unlikely(cache->pfmemalloc)) { + __folio_put(folio); + goto get_new_folio; + } + + fsize = folio_size(folio); + if (unlikely(fragsz > fsize)) + goto frag_too_big; + + /* OK, page count is 0, we can safely set it */ + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* Reset page count bias and offset to start of new frag */ + cache->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = fsize; + goto try_again; + +get_new_folio: + if (!spare) { + cache->folio = NULL; + put_cpu_ptr(&skb_splice_frag_cache); + +#if PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE + spare = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (!spare) +#endif + spare = folio_alloc(gfp, 0); + if (!spare) + return NULL; + + cache = get_cpu_ptr(&skb_splice_frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + cache->pfmemalloc = folio_is_pfmemalloc(spare); + if (cache->folio) + goto reload; + } + + cache->folio = spare; + cache->virt = folio_address(spare); + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + cache->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ + cache->offset = offset; + put_cpu_ptr(&skb_splice_frag_cache); + if (spare) + folio_put(spare); + return NULL; +} +EXPORT_SYMBOL(alloc_skb_frag); + +/** + * copy_skb_frag - Copy data into a page fragment. + * @s: The data to copy + * @len: The size of the data + * @gfp: Allocation flags + */ +void *copy_skb_frag(const void *s, size_t len, gfp_t gfp) +{ + void *p; + + p = alloc_skb_frag(len, gfp); + if (!p) + return NULL; + + return memcpy(p, s, len); +} +EXPORT_SYMBOL(copy_skb_frag); + static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, size_t offset, size_t len) { @@ -6808,17 +6947,43 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, break; } + if (space == 0 && + !skb_can_coalesce(skb, skb_shinfo(skb)->nr_frags, + pages[0], off)) { + iov_iter_revert(iter, len); + break; + } + i = 0; do { struct page *page = pages[i++]; size_t part = min_t(size_t, PAGE_SIZE - off, len); - - ret = -EIO; - if (WARN_ON_ONCE(!sendpage_ok(page))) + bool put = false; + + if (PageSlab(page)) { + const void *p; + void *q; + + p = kmap_local_page(page); + q = copy_skb_frag(p + off, part, gfp); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } else if (WARN_ON_ONCE(!sendpage_ok(page))) { + ret = -EIO; goto out; + } ret = skb_append_pagefrags(skb, page, off, part, frag_limit); + if (put) + put_page(page); if (ret < 0) { iov_iter_revert(iter, len); goto out; From patchwork Sat Jun 17 12:11:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72D6EEB64DA for ; Sat, 17 Jun 2023 12:12:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA4B46B0078; Sat, 17 Jun 2023 08:12:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E54746B007B; Sat, 17 Jun 2023 08:12:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF5DC8E0001; Sat, 17 Jun 2023 08:12:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BA3556B0078 for ; Sat, 17 Jun 2023 08:12:26 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 90D8F401D9 for ; Sat, 17 Jun 2023 12:12:26 +0000 (UTC) X-FDA: 80912127492.27.E22D2A2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id CBC3A100007 for ; Sat, 17 Jun 2023 12:12:24 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eykWtD6E; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+u30ziD9KgNs/SO68HcOpUwCa+4S9m3KYto4qjbvl9U=; b=jalkrpoH62jJNCr4jjRPathet8q4qswmo0g35AFMF5Vpgl6L8syvWveK4i5oE8Dsjiff9n T4Fchx+cBpJ6RKkN/KqSMqW92f3jP80cbKvzGvCfglYLlQRZ1HYvUo2ABrcCkUYAAPdZbI iPo4rw1M8tGB2VTqtNpUAVRY9XDpNEk= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eykWtD6E; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003944; a=rsa-sha256; cv=none; b=R+r443a5aD/z1wMO2PLkh7O2SKrtV6wci1aHrjDv6XsMmQfPbmIPioCHVoK8AQ9gM/7Lb7 nvdkJKborNIfz7dR9s5T8Yi5Np0NXISkhVrau9W8RIM9gzJmXCt5v71dZRfgXOKdyAnS1B rpwwOCKS21KP5WqhZSVo+6nN6OrEUuM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003944; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+u30ziD9KgNs/SO68HcOpUwCa+4S9m3KYto4qjbvl9U=; b=eykWtD6E6OIU1C1iJ0sVwxX6DHj1MUh7/7qErr5nj4fFiLX2j254q16QOpiJ6qF4+C/dXV dd4NjagI6pw2iKWQX9Khoi7l0q4aYCM90r9pcXZxbllAoqGnQSB0CI7JwByf1NYxigqiM4 zDtcnc3z7zVp0M5Fz5V0tuJGQgZPx3A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-OoK_dJyAPWWi4zKuB6nYXA-1; Sat, 17 Jun 2023 08:12:18 -0400 X-MC-Unique: OoK_dJyAPWWi4zKuB6nYXA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3AC0A8030B4; Sat, 17 Jun 2023 12:12:15 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id A932940C2077; Sat, 17 Jun 2023 12:12:13 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH net-next v2 02/17] net: Display info about MSG_SPLICE_PAGES memory handling in proc Date: Sat, 17 Jun 2023 13:11:31 +0100 Message-ID: <20230617121146.716077-3-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: CBC3A100007 X-Stat-Signature: t44q653fdxz4uwju63zfqnpm98ot1m3h X-HE-Tag: 1687003944-943804 X-HE-Meta: U2FsdGVkX18xCf8jVdRiJmQ2EW6ZmKyGKC9y5Hx30buD/rPQF+BsvkAwFncbQANP7J4T2HnB+0bybHekvqZQZqQin9pT/I+p07F4c4g7VpqA2xuz1efXz7lbK3boBnzXMuYhZ4YV29fvjnA3eS3HTken4cbYhIi5CmJsQakegRObS84Y4kO7fb46Vgd64U+CzFjxqhy3ss6vjtWgLYyKKTr60Z07cKKmZFUGLuhI3V1j0/NBeeVN0KFY2ag/nmsu9ERpKh7BgWhe7uZVqAUTYDkKuW8d/GGbnxP6Bk9LEWYT12cc6mUSGjbdk04ATCYL21ZfdHeNhakFNId90JlK5JfLLwhm2Ig5XNC13sqoHAdzHgRiapA5QADhRlMJgn7QznAVYHzGO4l4Ny4kuJPT5HEBXEDlB713c400A6SG7DRKP1gl9eEkNuM+eNmx66qbpgfUk6KqWdIWfSbUEzCvoPSVllBfzh5BXU8rGw3bFGgbOLeHM2h+xZGjs90RDVPIbOukxbed9HihEv9VCI4biSUvvUAJCU3a08+y8ca9Sav7DdxfpjruG1MnwZP/1xmzCgDN3iMH7OYJPyJBCfbgzn+NSRYHVEZZW1vQcxDNClbBzry0HAZwAWnV/kqFJnoJ1hdvw34Xpb2kHLlc0UjcCDqKkiKjX9K1AufQwGeczhNo/Nq5KKWH6PZbUZlrQubKW6oAnjf7aZzt8A8qnLhm2Rsr1R1/c7zw+4ya+ClvFP6jUHHN/a9c1V/qt9t1BnLMcB2n5DcPxN2A3W8KN6OamyF6FEb8xX/1tYqeoh97mi8Fj4d3A7yTV0M0DKRzYtos+5Z9n0Fu/J+Scx3Z9sRDfHOE4sTYED55rec0Yeu1HlekHCIl9sPGT/2DRRseDZG2Bu5OhIsuc8qQSlzw892f4Q6jDo9zu+cyKNxjdetXKDn9XIASl6OPGxO3pVFk9MEAlxTaifcqfLEpfhF2CRo WvDd6UAI 6K2WT6yuW7aBGpq/cX8M6oaYkpng56f35EZcKoMM5RWD94lf5Aa3rEJ3K3BA9H1//DUfEBi7iuNOV/zv8mWdQzXt+giL65F5jyX9uXW1pijL88HIahVhouVrN7XWrRLEAQxSMO2fD+AcqcPRGDbcO9sj7JyvrrbGsL/kdl5FOxXUGv549MtdIuZX9QzD8u+peEwaucUDsKj/91aU9HZjO+WoEqw8hjqVtojWYKqc7YxiCxemxjC1Xl0PmzeGDDXpJyjsDK7T8WES3M7PUbIJiAmUBpIUyXhV0ZdEylshHacBPx5PHRPbH5BUEk+PpjXR99G5Pq7aZ3kuYandMetbo1O6yQ+LMhnxIKZGouMEfjK9iYaa0HQJotQZpc0yABY0H9DyU3ffr2HqROJjdCIFsHgjQv9XR6CzoqQjf2oJN5wx7KCVxFSBiqFyqJnHchbmXiqM2ujM3UmOe0OeswpCFuI5gVN2laGYka1EPhKAskPwnJZOenfoQwzwW99swd/hnk60YozyfyAB3pRyxkAagytjFlgjFV1ljS54qqiJWERKuxUQF0vc/sXad15jZhjCTZrTSILd4y8OIK/E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Display information about the memory handling MSG_SPLICE_PAGES does to copy slabbed data into page fragments. For each CPU that has a cached folio, it displays the folio pfn, the offset pointer within the folio and the size of the folio. It also displays the number of pages refurbished and the number of pages replaced. Signed-off-by: David Howells cc: Alexander Duyck cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: Menglong Dong cc: netdev@vger.kernel.org --- net/core/skbuff.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index d962c93a429d..36605510a76d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -83,6 +83,7 @@ #include #include #include +#include #include "dev.h" #include "sock_destructor.h" @@ -6758,6 +6759,7 @@ nodefer: __kfree_skb(skb); struct skb_splice_frag_cache { struct folio *folio; void *virt; + unsigned int fsize; unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. @@ -6767,6 +6769,26 @@ struct skb_splice_frag_cache { }; static DEFINE_PER_CPU(struct skb_splice_frag_cache, skb_splice_frag_cache); +static atomic_t skb_splice_frag_replaced, skb_splice_frag_refurbished; + +static int skb_splice_show(struct seq_file *m, void *data) +{ + int cpu; + + seq_printf(m, "refurb=%u repl=%u\n", + atomic_read(&skb_splice_frag_refurbished), + atomic_read(&skb_splice_frag_replaced)); + + for_each_possible_cpu(cpu) { + const struct skb_splice_frag_cache *cache = + per_cpu_ptr(&skb_splice_frag_cache, cpu); + + seq_printf(m, "[%u] %lx %u/%u\n", + cpu, folio_pfn(cache->folio), + cache->offset, cache->fsize); + } + return 0; +} /** * alloc_skb_frag - Allocate a page fragment for using in a socket @@ -6803,17 +6825,21 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) insufficient_space: /* See if we can refurbish the current folio. */ - if (!folio || !folio_ref_sub_and_test(folio, cache->pagecnt_bias)) + if (!folio) goto get_new_folio; + if (!folio_ref_sub_and_test(folio, cache->pagecnt_bias)) + goto replace_folio; if (unlikely(cache->pfmemalloc)) { __folio_put(folio); - goto get_new_folio; + goto replace_folio; } fsize = folio_size(folio); if (unlikely(fragsz > fsize)) goto frag_too_big; + atomic_inc(&skb_splice_frag_refurbished); + /* OK, page count is 0, we can safely set it */ folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); @@ -6822,6 +6848,8 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) offset = fsize; goto try_again; +replace_folio: + atomic_inc(&skb_splice_frag_replaced); get_new_folio: if (!spare) { cache->folio = NULL; @@ -6848,6 +6876,7 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) cache->folio = spare; cache->virt = folio_address(spare); + cache->fsize = folio_size(spare); folio = spare; spare = NULL; @@ -6858,7 +6887,7 @@ void *alloc_skb_frag(size_t fragsz, gfp_t gfp) /* Reset page count bias and offset to start of new frag */ cache->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = folio_size(folio); + offset = cache->fsize; goto try_again; frag_too_big: @@ -7007,3 +7036,10 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, return spliced ?: ret; } EXPORT_SYMBOL(skb_splice_from_iter); + +static int skb_splice_init(void) +{ + proc_create_single("pagefrags", S_IFREG | 0444, NULL, &skb_splice_show); + return 0; +} +late_initcall(skb_splice_init); From patchwork Sat Jun 17 12:11:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 881D7EB64DB for ; Sat, 17 Jun 2023 12:12:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A84ED6B0075; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A35566B0078; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 923B48E0001; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7BA7A6B0078 for ; Sat, 17 Jun 2023 08:12:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4DABE1201D4 for ; Sat, 17 Jun 2023 12:12:25 +0000 (UTC) X-FDA: 80912127450.11.165E34C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 74B0240009 for ; Sat, 17 Jun 2023 12:12:23 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Z6Frxe6x; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003943; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/uSTCkptTsC4xYTqgPPYvFLo6KQm6ldgKXcPsN3l6RA=; b=pGq+uR2J+3KAycFTirZN6gYNVDNBfnDgKQbK0aMcvESWCphZfByjWxjlw8hKtTu0NPNOyE yGWoNgn/DLZVnS2ZwiXEMwiCBQozHgao0zGCGVMT49ETkRoPHDt3hXPCmvR3pnLeB4KrSz tsPbFdMAD8jdGb+6lrE3X8WC1JTgN8M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003943; a=rsa-sha256; cv=none; b=dVbGgOugDjBZmoV5EZCCdf+rJ26LZpGGFzANvd20+NCpnyzUPhvgsTu71ad6+u8eDla2d+ PJHrro5BqtOx24DXlj/9/z21PTsnoeg8FHxybyd2zz5HV3Nzc/2EUXQH7cFJB8rM2raDmO HNQpcxwONR0m7rztw2BZS7XG6ag7sZ8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Z6Frxe6x; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003942; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/uSTCkptTsC4xYTqgPPYvFLo6KQm6ldgKXcPsN3l6RA=; b=Z6Frxe6ximAio09RDRSSYVO4d9uNqsNKg+3CajR+wJqkkHsKyrxyf+A79+v+vtCIol0Dvp tmsqYGlHQ3Yp3fOGw8NePAVsjybMO7/75+sLAbgNggvFa6oJW9R+4OJ3eD51klCFoZDZLO Bd8JHt2T6WJ4v2Kb2L5l97aH2kV5/QE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558-nNaYjr9QPY2y9kjWTK_iSQ-1; Sat, 17 Jun 2023 08:12:21 -0400 X-MC-Unique: nNaYjr9QPY2y9kjWTK_iSQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8037D800962; Sat, 17 Jun 2023 12:12:19 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id E9EAE48FB01; Sat, 17 Jun 2023 12:12:15 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, John Fastabend , Jakub Sitnicki , Karsten Graul , Wenjia Zhang , Jan Karcher , "D. Wythe" , Tony Lu , Wen Gu , Boris Pismenny , Steffen Klassert , Herbert Xu , bpf@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH net-next v2 03/17] tcp_bpf, smc, tls, espintcp: Reduce MSG_SENDPAGE_NOTLAST usage Date: Sat, 17 Jun 2023 13:11:32 +0100 Message-ID: <20230617121146.716077-4-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Stat-Signature: gxykytcchbg8jfkjogbw34iftduxkuas X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 74B0240009 X-Rspam-User: X-HE-Tag: 1687003943-604104 X-HE-Meta: U2FsdGVkX1+OrC0zoeitkosTdsdkiXdw+XYKhi5punBwWLWygqkySWgQ29ZfxMWKiA1k1JJLF4AN4rWK+7fP83YdQPM0oQBL6jg8E0/MaVSeJ1LCHL3HwrmGkY0ppzQ3bDX3Q+wLag9sgG6k+Lnjpc97v/mL//hBueCpnJ0QlNWmoxy/lrzfl/ym0aELanvdp7liUe/HqqcRQxH50WffjHVO63HNKAqn3Pz1YysDJW01mjI8g6dJ1zmllMkgyAe9ZTLm/bcVP7PjkHJpuGKLFA3l3a/i78iP5ki77zHX9puTD7nAn3b3q68PnajGEnNUSBtjTSCjazvuJBKGDvJKkhwcUNSh7f3dH9a5y2A/rGPBE/+iy+btSJDzXE8wV3eU6BlLiQbEGZh+LppcgT+2eIhtATW3nuOaKpubPt0KcgYGUNu+TWeTo2QKygoC923A0RDQgGmCvvPBcAkNIl1UIx2jzBjgAMjZytFbTRj7HsEXr+mH6qoLEedruxOumWAR+gAY63+lVK9LrJSaiGQuVdkqBd7vys+eQy3PL5a7anxKCLTME4hzyoD/fWr6KmsVkWcYKlzGz042EJg/EKanzchWvKX1hggj5vHY5fhDsXEcYlMyaUPR6hA3/zXRKih5dcLi8KDIyHoLCXQFIHXe52rh1MyronpDxbLJx/fadWV2ySKmVuPJ2TPG71/Pso+BrqmKwavTlWF7I/u5FwBzgge2sB5czmhnQ6sXGWOvlpmupkITv+fN1w5CgaRi0Qfhwo4mKXdWdflef0qhJLEfXQXKkBWZNXM+29tzirtYXyvxqlk7+CygU025TQeZE8Vw1MYnyu8cVRwKKscb0E3N6bbPNO8AeOiracWQZUdfQxf643tpM7h3KsX0SbABmMdOd/NYQxi+Or/GENeXvVqSoBEw4j9+5UNqoHlhnVk17cVEkzCuYRpK4CVm8iycKPJib7ycfy15UUuAHaDjlsy Lz7UPWqJ pQNcUFAPP+YLbgvkQTer5qIqSjJ/T55ry6kPMto35OFScH4UVTbuaemzgigIFvjJ/7PKXDEsQ7feO41xD8ATx0PNVFK+bPi/4STk0LhzRQcqal0BAUi4RBOAGIR0JdGOcPZiNN38azoq7gRiLBL3VkrLzz1b8pOIxCEGscVNpYYE8OqE84Q1ePNP0BIQH+IlGIdLmvfgHTH/K/I5jsLezAovzdoGAY3eLnoYOWtxjkIrt51QhKTkZD8bMKajWhR3VvYaNqwT7Vp9g0LThlWJXa2r6dQxjXPrk+k/val7x8V5qazGiFGtHkUy8w9G5/MWw8WwL+78Om6em/WqSfX9PyjagTMRB9lk/z7N2pfmq6mZDepig2GxYI4/2ymksu+cJWNClxNbKLpk/9uqOjUeZfw0mYYxxlpUdQYxU6+27LbZNzqTyIe8f2d5bMMvO1IUxHEei2ctLM3LvqpVcwbiXt6Okxq5VZKiWPWmT+RXAoZDDi1dkOT0yL9HZtY3ZRIlKdPSf7PYXoBNGEUbKSAul+VEurhKMgo1MAaJEf8Bwvb3293kEL/fDk3i0al0q/O+N4EdmvfwQlH1wbT7aATt0NMtxlfrUFltvJIW5XJ/UEqJT9Chz1V5dTAQ9LcOZSaFz1c7kxZSihYLOemoruJaMy6u9MpFGzA+4Znhp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As MSG_SENDPAGE_NOTLAST is being phased out along with sendpage(), don't use it further in than the sendpage methods, but rather translate it to MSG_MORE and use that instead. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Karsten Graul cc: Wenjia Zhang cc: Jan Karcher cc: "D. Wythe" cc: Tony Lu cc: Wen Gu cc: Boris Pismenny cc: Steffen Klassert cc: Herbert Xu cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org cc: linux-s390@vger.kernel.org --- net/ipv4/tcp_bpf.c | 3 --- net/smc/smc_tx.c | 6 ++++-- net/tls/tls_device.c | 4 ++-- net/xfrm/espintcp.c | 10 ++++++---- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 5a84053ac62b..adcba77b0c50 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -111,9 +111,6 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, if (has_tx_ulp) msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; - if (flags & MSG_SENDPAGE_NOTLAST) - msghdr.msg_flags |= MSG_MORE; - bvec_set_page(&bvec, page, size, off); iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); ret = tcp_sendmsg_locked(sk, &msghdr, size); diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c index 45128443f1f1..9b9e0a190734 100644 --- a/net/smc/smc_tx.c +++ b/net/smc/smc_tx.c @@ -168,8 +168,7 @@ static bool smc_tx_should_cork(struct smc_sock *smc, struct msghdr *msg) * should known how/when to uncork it. */ if ((msg->msg_flags & MSG_MORE || - smc_tx_is_corked(smc) || - msg->msg_flags & MSG_SENDPAGE_NOTLAST) && + smc_tx_is_corked(smc)) && atomic_read(&conn->sndbuf_space)) return true; @@ -306,6 +305,9 @@ int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, struct kvec iov; int rc; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; + iov.iov_base = kaddr + offset; iov.iov_len = size; iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, size); diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index b82770f68807..975299d7213b 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -449,7 +449,7 @@ static int tls_push_data(struct sock *sk, return -sk->sk_err; flags |= MSG_SENDPAGE_DECRYPTED; - tls_push_record_flags = flags | MSG_SENDPAGE_NOTLAST; + tls_push_record_flags = flags | MSG_MORE; timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); if (tls_is_partially_sent_record(tls_ctx)) { @@ -532,7 +532,7 @@ static int tls_push_data(struct sock *sk, if (!size) { last_record: tls_push_record_flags = flags; - if (flags & (MSG_SENDPAGE_NOTLAST | MSG_MORE)) { + if (flags & MSG_MORE) { more = true; break; } diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 3504925babdb..d3b3f9e720b3 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,13 +205,15 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { - struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + struct msghdr msghdr = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE, + }; struct sk_msg *skmsg = &emsg->skmsg; + bool more = flags & MSG_MORE; struct scatterlist *sg; int done = 0; int ret; - msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { struct bio_vec bvec; @@ -221,8 +223,8 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; - if (sg_is_last(sg)) - msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; + if (sg_is_last(sg) && !more) + msghdr.msg_flags &= ~MSG_MORE; p = sg_page(sg); retry: From patchwork Sat Jun 17 12:11:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EF2BEB64D9 for ; Sat, 17 Jun 2023 12:12:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D7B16B007B; Sat, 17 Jun 2023 08:12:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 889AD8E0001; Sat, 17 Jun 2023 08:12:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7014E6B0080; Sat, 17 Jun 2023 08:12:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5A6836B007B for ; Sat, 17 Jun 2023 08:12:28 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2D19D120DE2 for ; Sat, 17 Jun 2023 12:12:28 +0000 (UTC) X-FDA: 80912127576.11.50BC76A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 6B8438001A for ; Sat, 17 Jun 2023 12:12:26 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DlmIJNJa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003946; a=rsa-sha256; cv=none; b=aMvF2iymHzyQ0uUcp7p7N0sxvauVw7DwsYcNghQbeuYl1CKlwJq5ZdfxW7oadwC/JXS/j0 SZKzNpB2ClUw6CKSwceEzSe1Qy5E5QSyd/f/+l2QMeWR2m4KVmq497VvCaV8bhI0tgo8zY KTyv1XtwoUG6ZeyQeRmWHrEbtOmEKRQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DlmIJNJa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VjthvFXeQGZeDxQ7unAKOEBDdmvRqvLtJxvDci7YFS0=; b=LWO4Wm1dINmb8Dq7WHxe6harrSqzPLtnybEYmeKVWmFwklO9DNPBsk8POhYNsecgP51cBf kdT7ASOBT19OLDMBquHlTOwicgQOj/R4XZD3X/resSc8UAuPzGpUhNGhqXsploRt43spbG mRdpuf5zkWBZsnBBkKMtRM2n2Npz+w0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VjthvFXeQGZeDxQ7unAKOEBDdmvRqvLtJxvDci7YFS0=; b=DlmIJNJalFi8KYR0UZWGl2MHtsZOdPq//nm8+uAxDmh8RVvQUbZgK2pXUlFmjojBxIqS0O BplW7xsNw4YUphkfeWXrFhbN43DVxdroUH8aw/YBERLbqUg6UWJk1b+f0lDQwSUFQmCA86 7ynRcDDVUzsgt9vX8e6gnOCw1xU6KvQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-658-0gggrxlKN_S56nOJ9-go3w-1; Sat, 17 Jun 2023 08:12:23 -0400 X-MC-Unique: 0gggrxlKN_S56nOJ9-go3w-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 28B3785A58A; Sat, 17 Jun 2023 12:12:22 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1E03D1415102; Sat, 17 Jun 2023 12:12:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v2 04/17] siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit Date: Sat, 17 Jun 2023 13:11:33 +0100 Message-ID: <20230617121146.716077-5-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6B8438001A X-Stat-Signature: mw7po5d1sz9jc6fa9i5q7zyq9tybhifm X-HE-Tag: 1687003946-917176 X-HE-Meta: U2FsdGVkX18Kqq+6HAM2H415BIR85BSr8bz9p/+3JjyKDV18SzGzNYUzPe1f1gQ8QG4S0IqeN99WVkXdTtYN7UKphUyG5pCsbKXU6iyPm+OpP7JQKL/CWsMVb49om7aHO9milFbNfb7igZY6hAV8/Rz8WDU9FK3DQWfuQig78H8XZn+0//qmn8Ss1M23LDFR9jgugxBeV++xjIGwMmNmJDmC+8V0SFkTND9Keu9fcgmhLJ5pCW+M9qgm/HJgLFxUg4nsE4dCk46DqE7EcOComUaNwQaR0lwNR0M6APEb3TyG3tfHDoTtNcXiKGjzPzHOZdwHkMp1Q1UtpOkA+0xZF/PZQjuuatH3RLi589Rxee9swFbLctWDHSU2/1UrU+mVd8D3kjLItJ6x3I8bWhD8MafR7ZYlwoUOH34GswnZgE7Japa4sEy7vSQ5NHPOlZkF0zi7VB/FdkKv5NOi4GqeZMV01eaJZrMrbVTeifi0ZHOZvCwjgHc77/gJKfkPOstu+XjNrhyW1PSRrOIfQ2D/U2g80BuAe1tO0/IBw0VOrb40GNVvtDEJRxbgfcSsV+I+UMKvPp/lsdTu19sYRhUB9pJCNKdyKDsGiKFwtIKaUFm4kuuhw/+p+S//UXcIEizY0duM9dHwF4f0VV0vBz5cP11gWLfEzdVJGzjsK2ZwQz93h4dZ8xJzmtxozJkgqqEBECpZ7EIZrGPEbl2DcugLi/4c6BqGaUZN9pYoRYFfYFYRiuzaxJ+vst+n/UoRNcUp0Q9MJUEYq30zLABvwsEfJ1Dg1I7Tot4ElIgvpzPUWVPfhEjE1OMqaph4WFS0uGZF3bYIFwl9k7rHHUSa2UGoyScbFfKILxx3rFLXbPMuCZDXiaY8hUdhAXXIvihpXUlbY2IRRurrzbHCpZ2sDImAWGGKsnrhe2bAnfcls4ZTDhOqHNbcr1aRRj7ruiOED3eoM+/nsuYfmw11x6trWGg YIEHHPqf 1Lq8PAdMzYOa+b1EgdwKBjD9ehmy/hdUvRClHw5MuBa4udpiATUwLTYMTRf6efefBUhc4VP9m9S0iCIXtgRHbqp3lUKhQOdha2iiaoqRG5tC7UNG9zW6VfoTmXzbSuaMLVJBlDc8y3HQqMB7vlb78pgxu90zVd5vyl7tIGoD2FhMa3Loc8VlkFcn08hc9nEhbTVMfuHtRzAmosl1w/PASKOpV0Hlyq80jwaaV5s/+7ab6UVcQGdN6979j+Men+aR+NkZCr1PDdwbNfCU38kjb+YsvNvgNZu2W5ITdIW8VEx3t2JDT2YkOIWiHcidSuj47zgKx6VXlanGR+46NkU0HXudOy49009EFJRRqwYLer+RfYN39AbRyP+oYNlAOQFrMyDYfz4qxDqvnVC9Q0mqrbZy+h5e9z89clGedZx2OkDLHbE+QM560dfrGfzeO3O2TkSoJ0coazZKBFonCTPHubmuw6+sCKjIuww7NN8+MUS6OVv1bfh1U//6XYOH20WCrtHvbP7AigKb5FZbpzIAoIo40zFvPbK/6kUhUb47iEo0n7MwsPdC1qFNt4hhxsFPAMZDeMnRpPM+xfCCA9irwWHLR5LEvY10FwKum3s5ROfHWd2KJ3D0evCRntRmttZ8dUk2C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer (if present) are copied into page fragments that can be freed with put_page(). Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #2) - Wrap lines at 80. drivers/infiniband/sw/siw/siw_qp_tx.c | 231 ++++---------------------- 1 file changed, 36 insertions(+), 195 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index ffb16beb6c30..2584f9da0dd8 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -311,114 +311,8 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, return rv; } -/* - * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. - * - * Using sendpage to push page by page appears to be less efficient - * than using sendmsg, even if data are copied. - * - * A general performance limitation might be the extra four bytes - * trailer checksum segment to be pushed after user data. - */ -static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, - size_t size) -{ - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | - MSG_SPLICE_PAGES), - }; - struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0; - - while (size) { - size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); - - if (size + offset <= PAGE_SIZE) - msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; - - tcp_rate_check_app_limited(sk); - bvec_set_page(&bvec, page[i], bytes, offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - -try_page_again: - lock_sock(sk); - rv = tcp_sendmsg_locked(sk, &msg, size); - release_sock(sk); - - if (rv > 0) { - size -= rv; - sent += rv; - if (rv != bytes) { - offset += rv; - bytes -= rv; - goto try_page_again; - } - offset = 0; - } else { - if (rv == -EAGAIN || rv == 0) - break; - return rv; - } - i++; - } - return sent; -} - -/* - * siw_0copy_tx() - * - * Pushes list of pages to TCP socket. If pages from multiple - * SGE's, all referenced pages of each SGE are pushed in one - * shot. - */ -static int siw_0copy_tx(struct socket *s, struct page **page, - struct siw_sge *sge, unsigned int offset, - unsigned int size) -{ - int i = 0, sent = 0, rv; - int sge_bytes = min(sge->length - offset, size); - - offset = (sge->laddr + offset) & ~PAGE_MASK; - - while (sent != size) { - rv = siw_tcp_sendpages(s, &page[i], offset, sge_bytes); - if (rv >= 0) { - sent += rv; - if (size == sent || sge_bytes > rv) - break; - - i += PAGE_ALIGN(sge_bytes + offset) >> PAGE_SHIFT; - sge++; - sge_bytes = min(sge->length, size - sent); - offset = sge->laddr & ~PAGE_MASK; - } else { - sent = rv; - break; - } - } - return sent; -} - #define MAX_TRAILER (MPA_CRC_SIZE + 4) -static void siw_unmap_pages(struct kvec *iov, unsigned long kmap_mask, int len) -{ - int i; - - /* - * Work backwards through the array to honor the kmap_local_page() - * ordering requirements. - */ - for (i = (len-1); i >= 0; i--) { - if (kmap_mask & BIT(i)) { - unsigned long addr = (unsigned long)iov[i].iov_base; - - kunmap_local((void *)(addr & PAGE_MASK)); - } - } -} - /* * siw_tx_hdt() tries to push a complete packet to TCP where all * packet fragments are referenced by the elements of one iovec. @@ -438,30 +332,21 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) { struct siw_wqe *wqe = &c_tx->wqe_active; struct siw_sge *sge = &wqe->sqe.sge[c_tx->sge_idx]; - struct kvec iov[MAX_ARRAY]; - struct page *page_array[MAX_ARRAY]; + struct bio_vec bvec[MAX_ARRAY]; struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_EOR }; + void *trl; int seg = 0, do_crc = c_tx->do_crc, is_kva = 0, rv; unsigned int data_len = c_tx->bytes_unsent, hdr_len = 0, trl_len = 0, sge_off = c_tx->sge_off, sge_idx = c_tx->sge_idx, pbl_idx = c_tx->pbl_idx; - unsigned long kmap_mask = 0L; if (c_tx->state == SIW_SEND_HDR) { - if (c_tx->use_sendpage) { - rv = siw_tx_ctrl(c_tx, s, MSG_DONTWAIT | MSG_MORE); - if (rv) - goto done; + void *hdr = &c_tx->pkt.ctrl + c_tx->ctrl_sent; - c_tx->state = SIW_SEND_DATA; - } else { - iov[0].iov_base = - (char *)&c_tx->pkt.ctrl + c_tx->ctrl_sent; - iov[0].iov_len = hdr_len = - c_tx->ctrl_len - c_tx->ctrl_sent; - seg = 1; - } + hdr_len = c_tx->ctrl_len - c_tx->ctrl_sent; + bvec_set_virt(&bvec[0], hdr, hdr_len); + seg = 1; } wqe->processed += data_len; @@ -477,28 +362,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } else { is_kva = 1; } - if (is_kva && !c_tx->use_sendpage) { - /* - * tx from kernel virtual address: either inline data - * or memory region with assigned kernel buffer - */ - iov[seg].iov_base = - ib_virt_dma_to_ptr(sge->laddr + sge_off); - iov[seg].iov_len = sge_len; - - if (do_crc) - crypto_shash_update(c_tx->mpa_crc_hd, - iov[seg].iov_base, - sge_len); - sge_off += sge_len; - data_len -= sge_len; - seg++; - goto sge_done; - } while (sge_len) { size_t plen = min((int)PAGE_SIZE - fp_off, sge_len); - void *kaddr; if (!is_kva) { struct page *p; @@ -511,33 +377,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) p = siw_get_upage(mem->umem, sge->laddr + sge_off); if (unlikely(!p)) { - siw_unmap_pages(iov, kmap_mask, seg); wqe->processed -= c_tx->bytes_unsent; rv = -EFAULT; goto done_crc; } - page_array[seg] = p; - - if (!c_tx->use_sendpage) { - void *kaddr = kmap_local_page(p); - - /* Remember for later kunmap() */ - kmap_mask |= BIT(seg); - iov[seg].iov_base = kaddr + fp_off; - iov[seg].iov_len = plen; - - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - iov[seg].iov_base, - plen); - } else if (do_crc) { - kaddr = kmap_local_page(p); - crypto_shash_update(c_tx->mpa_crc_hd, - kaddr + fp_off, - plen); - kunmap_local(kaddr); - } + + bvec_set_page(&bvec[seg], p, plen, fp_off); } else { /* * Cast to an uintptr_t to preserve all 64 bits @@ -545,12 +390,17 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) */ u64 va = sge->laddr + sge_off; - page_array[seg] = ib_virt_dma_to_page(va); - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - ib_virt_dma_to_ptr(va), - plen); + bvec_set_virt(&bvec[seg], + ib_virt_dma_to_ptr(va), plen); + } + + if (do_crc) { + void *kaddr = + kmap_local_page(bvec[seg].bv_page); + crypto_shash_update(c_tx->mpa_crc_hd, + kaddr + bvec[seg].bv_offset, + bvec[seg].bv_len); + kunmap_local(kaddr); } sge_len -= plen; @@ -560,13 +410,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) if (++seg >= (int)MAX_ARRAY) { siw_dbg_qp(tx_qp(c_tx), "to many fragments\n"); - siw_unmap_pages(iov, kmap_mask, seg-1); wqe->processed -= c_tx->bytes_unsent; rv = -EMSGSIZE; goto done_crc; } } -sge_done: + /* Update SGE variables at end of SGE */ if (sge_off == sge->length && (data_len != 0 || wqe->processed < wqe->bytes)) { @@ -575,15 +424,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) sge_off = 0; } } - /* trailer */ - if (likely(c_tx->state != SIW_SEND_TRAILER)) { - iov[seg].iov_base = &c_tx->trailer.pad[4 - c_tx->pad]; - iov[seg].iov_len = trl_len = MAX_TRAILER - (4 - c_tx->pad); - } else { - iov[seg].iov_base = &c_tx->trailer.pad[c_tx->ctrl_sent]; - iov[seg].iov_len = trl_len = MAX_TRAILER - c_tx->ctrl_sent; - } + /* Set the CRC in the trailer */ if (c_tx->pad) { *(u32 *)c_tx->trailer.pad = 0; if (do_crc) @@ -596,23 +438,23 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) else if (do_crc) crypto_shash_final(c_tx->mpa_crc_hd, (u8 *)&c_tx->trailer.crc); - data_len = c_tx->bytes_unsent; - - if (c_tx->use_sendpage) { - rv = siw_0copy_tx(s, page_array, &wqe->sqe.sge[c_tx->sge_idx], - c_tx->sge_off, data_len); - if (rv == data_len) { - rv = kernel_sendmsg(s, &msg, &iov[seg], 1, trl_len); - if (rv > 0) - rv += data_len; - else - rv = data_len; - } + /* Copy the trailer and add it to the output list */ + if (likely(c_tx->state != SIW_SEND_TRAILER)) { + trl = &c_tx->trailer.pad[4 - c_tx->pad]; + trl_len = MAX_TRAILER - (4 - c_tx->pad); } else { - rv = kernel_sendmsg(s, &msg, iov, seg + 1, - hdr_len + data_len + trl_len); - siw_unmap_pages(iov, kmap_mask, seg); + trl = &c_tx->trailer.pad[c_tx->ctrl_sent]; + trl_len = MAX_TRAILER - c_tx->ctrl_sent; } + bvec_set_virt(&bvec[seg], trl, trl_len); + + data_len = c_tx->bytes_unsent; + + if (c_tx->use_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, seg + 1, + hdr_len + data_len + trl_len); + rv = sock_sendmsg(s, &msg); if (rv < (int)hdr_len) { /* Not even complete hdr pushed or negative rv */ wqe->processed -= data_len; @@ -673,7 +515,6 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } done_crc: c_tx->do_crc = 0; -done: return rv; } From patchwork Sat Jun 17 12:11:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00D0FEB64DA for ; Sat, 17 Jun 2023 12:12:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBBCF6B007E; Sat, 17 Jun 2023 08:12:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF8186B0080; Sat, 17 Jun 2023 08:12:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFA306B0081; Sat, 17 Jun 2023 08:12:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9A7C16B007E for ; Sat, 17 Jun 2023 08:12:31 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7B0621C8A6C for ; Sat, 17 Jun 2023 12:12:31 +0000 (UTC) X-FDA: 80912127702.11.746FD0D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id D42E6180011 for ; Sat, 17 Jun 2023 12:12:29 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WK6Rhk0h; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003949; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1DmsLXNf6UEopicQGVGHzvB3R+Io/k4xNBk1zidfJPQ=; b=MFDj7bQjVacS/8Oxv1OkAZecSWvsAscAY47Wyh337eh8L22XLh3aDdwC+l2ofbpAbhXgUW VIRHhc3cQujWUeQBkv4vwOQ6d06PXXQEmTxEOWsVt505dGmfzsT+xwmN/WYKTvQfJPMofK 1n/qvOumwSO7pLmnl1W7WWVhIDZALRA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003949; a=rsa-sha256; cv=none; b=co1eL3qpz59KDrtf9/Huw/lEvS0YkKlFUlwGtCzocNgrlqrHc62Uigq+6Zdiy6tGM4xU+b hwjkDrMKxtck5jpAHyh84NNZtH/IqF8cXAqxGl8ZD8CFXTJ0RmK8Jd7XsLwx03fKjj86Z6 abY4aXOZLM/23ETYoY79Hbp48v1XmF4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WK6Rhk0h; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1DmsLXNf6UEopicQGVGHzvB3R+Io/k4xNBk1zidfJPQ=; b=WK6Rhk0hUsNh7r/nEn5rqq1NVKay1422026u2ydmNJei3hXqcts5AstoCKRCYpXN1w6P0/ 0Ys99LXqORD/4UjKdXrJzARLE3Ok+rFsf4Dql7haXvREbTpLYI2pv4Kdvet6imXc+LmuNe 7lO9uWWUVH2iR340w44GU8r9xPECDH8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-373-vtLjC08lOWSLLrXv76NNIw-1; Sat, 17 Jun 2023 08:12:25 -0400 X-MC-Unique: vtLjC08lOWSLLrXv76NNIw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B9013185A78E; Sat, 17 Jun 2023 12:12:24 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id D391840C5C1; Sat, 17 Jun 2023 12:12:22 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ilya Dryomov , Xiubo Li , Jeff Layton , ceph-devel@vger.kernel.org Subject: [PATCH net-next v2 05/17] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Sat, 17 Jun 2023 13:11:34 +0100 Message-ID: <20230617121146.716077-6-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: D42E6180011 X-Rspam-User: X-Stat-Signature: 7drnhp9ify4551ffqgow4nfgbcans33r X-Rspamd-Server: rspam03 X-HE-Tag: 1687003949-491184 X-HE-Meta: U2FsdGVkX199beBx7vENJdbpQapbXfI6lapfpPi1WdPTkWNINbVSyktWyXa6d0yWzEVTyJ3wSgwrzvwG2LtHA95ysHMoxOKLWoGqGMIHKQmyPwnQy0eUQ+lUMZGpBCvdIfBp05XF39JwACnmySIrN2UqGTVZr63s0aeqigFEs1WyRem4OA3rnhpEgiO8XSaBaZeVp6O5oBoBYJZFgyRyjyFqBgNhsQUsl09R/l5pOBj+7lgys9kK+WGgaRSYlPDO9TrozZgR0st6uyZNneXQ0fyjcZv8Csk3JZvRSpYbK8FA34PwDzx8Q92weQjLGqdwUNV18Q6u8zdsk4obD3+xgx93CsHxRRfJY2oeiVTVXiAopA7OhToWCG2PFuHxhFmRmRqDzIsJVsBnlf+p3X0Db9mDgSS44MF/ufhlOoMEIR7LWIt3UQKSxUDL+ryQe/S5rugbLPxwxFTrhPmECb3flv2eA1nxu76x9wSb/4IoYVXnEb/latRfedWJqcR4CdQhEIW2JjYsW3tE6hwhZNptAYTqOmHwnSARR1yWVolV/h7ky3DRPneoqFKs7QfxzoIdBkBdrrzHWdMmz2576IXqSbPWO33UUfYP+EChiFzdrj8vZYKtMSETf5RJ4l9GTi9wHo3wf0YWzeDA82+lf/5dqhJI2DKgTpNAOWWX5dehcEAb+EO5zOMZMmc6GJKnF67peyEWGdvbyaHe0v2HyKoxkKGzAJoaREWW3BJnxIgH84wHPs4/224S4ZXoaR8JJxJ2z1Y8wzFgyKWFQhT2H+0cn4Ky90luf31AmAiX8TwWRKuBWCGXy/AuEPB7sNZ8f2LTsy6mKsspkp9RtrmdbwUd9Os816X5DulLWvzIFNp0bOTSHWwVyMENFvM27Bxm15tul6gUB19d3fl/j8RZ1FM82dfdNUEz5vo5d/uFP5JAMl1u6FIet6lI8wYAA5C24257NEXVeeX+0MBPCHjRDyA iE3Mumqo GhG9ye33EcS0hT6yc9B7cw9FtE2yqmlhuvD6w62LJ0/ZHYlxMfGifYuimJNfit8tMLohZBVkoQPDCIg80oBUybhpzM0JUGtqbbM9FWWgU8j0frTLGNA8ti0FmmjpiHBR97wyVNhc7KUvuyKJOv2F6haUOqIJYXPFnsjmMUzq2cJKedJfwaab8x7z7cZETmAm6v0STIeP9UsmD/wVGJOWMjVQl0kw+8ZjN8Ysje673yc0neJqYGmE2KAvux/3qJmE/1wM/uEMuGIolTO+Xyn9yZMOEQYkRjlu5Y6bR3sEYo1ExTegJ/mZ7y6SkjDRsxrvNsUPi9ccChh9uzdq5lNtRcCF/W7Uqyd3CfdCy/GjqG3fjRDtWBp3bnQhWCjO34GG29022hgKnvtysl7lB2ttj5Xq77DF0qKniybixORg7I511AcTDn+Ss7c4PmEYyf6kYEOg4Bjbn1FfgZ8yRLtJmMQbw7eIj1pjbsIeQMrVpjJ5e48YjmPJT/8qayXOVCDF6CAmr/rW13Xu20Wk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v1.c | 58 ++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 39 deletions(-) diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index d664cb1593a7..f082e5c780a3 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -74,37 +74,6 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov, return r; } -/* - * @more: either or both of MSG_MORE and MSG_SENDPAGE_NOTLAST - */ -static int ceph_tcp_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int more) -{ - ssize_t (*sendpage)(struct socket *sock, struct page *page, - int offset, size_t size, int flags); - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - int ret; - - /* - * sendpage cannot properly handle pages with page_count == 0, - * we need to fall back to sendmsg if that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag which - * triggers one of hardened usercopy checks. - */ - if (sendpage_ok(page)) - sendpage = sock->ops->sendpage; - else - sendpage = sock_no_sendpage; - - ret = sendpage(sock, page, offset, size, flags); - if (ret == -EAGAIN) - ret = 0; - - return ret; -} - static void con_out_kvec_reset(struct ceph_connection *con) { BUG_ON(con->v1.out_skip); @@ -464,7 +433,6 @@ static int write_partial_message_data(struct ceph_connection *con) struct ceph_msg *msg = con->out_msg; struct ceph_msg_data_cursor *cursor = &msg->cursor; bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; u32 crc; dout("%s %p msg %p\n", __func__, con, msg); @@ -482,6 +450,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ crc = do_datacrc ? le32_to_cpu(msg->footer.data_crc) : 0; while (cursor->total_resid) { + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES, + }; struct page *page; size_t page_offset; size_t length; @@ -494,9 +466,12 @@ static int write_partial_message_data(struct ceph_connection *con) page = ceph_msg_data_next(cursor, &page_offset, &length); if (length == cursor->total_resid) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, page, page_offset, length, - more); + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, length, page_offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, length); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) { if (do_datacrc) msg->footer.data_crc = cpu_to_le32(crc); @@ -526,7 +501,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ static int write_partial_skip(struct ceph_connection *con) { - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_MORE, + }; int ret; dout("%s %p %d left\n", __func__, con, con->v1.out_skip); @@ -534,9 +512,11 @@ static int write_partial_skip(struct ceph_connection *con) size_t size = min(con->v1.out_skip, (int)PAGE_SIZE); if (size == con->v1.out_skip) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, ceph_zero_page, 0, size, - more); + msghdr.msg_flags &= ~MSG_MORE; + bvec_set_page(&bvec, ZERO_PAGE(0), size, 0); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) goto out; con->v1.out_skip -= ret; From patchwork Sat Jun 17 12:11:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C177EB64D9 for ; Sat, 17 Jun 2023 12:12:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 111BC6B0080; Sat, 17 Jun 2023 08:12:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 071D26B0081; Sat, 17 Jun 2023 08:12:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E55A56B0082; Sat, 17 Jun 2023 08:12:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D20C56B0080 for ; Sat, 17 Jun 2023 08:12:36 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A686CC0188 for ; Sat, 17 Jun 2023 12:12:36 +0000 (UTC) X-FDA: 80912127912.29.760AFA4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id D8BA12000C for ; Sat, 17 Jun 2023 12:12:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZY2kqKAr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003954; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XZXyJFmNcOo4OqBa8WBusNB3ZLLcguzSLjkBKTvuk5o=; b=KcBDvgQ+Dm1StGKtvq1MfaCcbT3QY73YElMk6+bOKjwO+YuOdjlzIveDp3rpGjvyqKe9Xa S0SnnEhGDyjq62QcfxpQpzIHYsc1nk4c4s66NkxiljTdoNeocXTHwoF/EM6BdjHOEKnDtK f0pIwTsz9q8qMIHbmwBYNsfeLbG2Spk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZY2kqKAr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003954; a=rsa-sha256; cv=none; b=Z8IvvT9RHWc0PwtWZuCdQGqKR8LhpBQ4Yovgs9bZeL9oN5d8pDr7DVGr1om3nrEuOrgBkI Dv6GZyBCyeqUg0Uh3tQK0aK/y/tN5c3Yt0Aikigf9sQ1kuiKMbr8U525fIP+ZcKh4SqBEp LPb+InRP0WybBkwMNrlCd1J0j5Oh+G4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XZXyJFmNcOo4OqBa8WBusNB3ZLLcguzSLjkBKTvuk5o=; b=ZY2kqKArIX8r+Yec4rAeIF+/uH5ZEVTc55B76twvokdXN3f6VgKGWDGxJNMwDR8EDh76++ SpM7kNgyUOFeUhOJhCN0Xx+CEm6yzxrHUIsdLzCu8OeCwpJqIy4UpKVofgXrLjrJPLuXo8 ILSkETeHlIOaGxH/qu6fiOcYLFsZbV4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-343-yGKuKhXgMiGBNdv2zBCseg-1; Sat, 17 Jun 2023 08:12:27 -0400 X-MC-Unique: yGKuKhXgMiGBNdv2zBCseg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BD337811E9E; Sat, 17 Jun 2023 12:12:26 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 579DA40C2077; Sat, 17 Jun 2023 12:12:25 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 06/17] net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock() Date: Sat, 17 Jun 2023 13:11:35 +0100 Message-ID: <20230617121146.716077-7-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: D8BA12000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: emsfqy81cqo16qdmczo8b3x68mbubcd4 X-HE-Tag: 1687003954-782552 X-HE-Meta: U2FsdGVkX19/rYgyBHqUzeVPqDQ63UFJs4tHvfsgn8atAhpx5YuhEyEps4wNTBoCforMqtIBXDZtCnWK+Ps5ihd8LteP/bWNGOT32m6sNatgP38hPSkgbA2T58qeVbSYbPjcIIU0K61/czobbX6dSsOp5j/6McWkL6dwfQ2hrz2gycnvWcgRjGxJgI0obrA9MhVlQJmywp7bfmZqfWS6xpc3SJQd+RexIgPeBMw/yAk/T7ZQCTaaiec4I94VH5JHJjPy/oRjC1k2OagJ16X8nP01JhLg5AroAcUpkWPdGOav8U+Tfa7Z190sh8rEL7cBimStYFQ9BufnDrszEzNtpydo7RC9flAh/5SaAL1RHmxTgg+BwxninofwvCZ3Ymgb/YY8AP/o06CX82HYVTMT0aRPZDethDtgozZsp8fUURpNAsd5Evg0pQNK6ECMNL82J+KTT4b7xlJxW+387iC3wDbUyfn2lPvB3Rxs2YFZbbqRadcSqy0tABpGcB7lIW/8rtOiH4B6Rn+Kwzy41OudgDs0JtkfElvLROR5T91gN91/NpJWlYBEpprr5Qe63d1QmlscGl7Qj9+ibpDHkhNdoFtQ7s8NuxQDZD4/S9r1aeZNMuX0lqXiw6t9Vs2PJX8ihzL6xyr9AH7Ablf/cvjCzffkfrdiB9EtWVE9KRFuRvMU7J+ZE1rvOnjIKG9zNGX9SScST6CwTSz5KytajC59TnfEQ0QLk5EMfjUsAycBameCVnq6PrNXB1CMHKCNo0cvSDMaAfy76MRIw/Cz+oIbvavBHxGWK3phCWZBOnkiqA8/ROvkNIpAwFVfGpzeRstjRf9Oa1Pv3J4k+ll6doYTv0ZgX43QTUcNKwWiINx+8mST0WJnBp87e9q+lsT6pmMr6aziSVTZSN3+7QIWiTRdKSr049BRQRtGN8YtLP555jRzpN5rXml47RghBbMpsDQknAyvDSCcrKhoOxeshKS 4uUQznzC GBeiZMaCtfowqPoXaJAi4t4C3UxCID3ReFoYWMgE0rWVlNt5sKxucstsHJSifJev8Q6XoMLFKYXxSI125gjLSS1s0gPBx3lQ5ZSdGUvX3iFxb5IiNj1tY0YKtboQ8jrwbYeX/AYQ+jJjPqxiNaMlnuWTtaUzW0EtwBQq1WGNln83yr1G+u/RqVO2pIqvFC54V2p/GWuQTa1hW/VMgY5gB79+ZEgEVZkNu7B4COSKoOPelGShJIDctnOD0dVaK3vD+MWjB7dD3E01mt6ax19VMEGa9EcOr4j4hmv8KpplZ2bgADUv6J2XsskDXykZFJu6c5qvknl4uB6jZwt3a3eIoPoR/ykKFFcaoHaR1vQvL5wLLzlJR9WxFk+w4/Jz94RngG0IN2bvy983iIzfMESuADsfMCkXX0aKmaMNP8e1d8D0Dl0oVBKYU6Ww0QIaC3xQfwPf9+ALnQJascyeGDpEKd2YM72TInzUKAx8HhJeOMuehlOQe00g1ul+ZBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage in skb_send_sock(). This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Note that this could perhaps be improved to fill out a bvec array with all the frags and then make a single sendmsg call, possibly sticking the header on the front also. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #2) - Wrap lines at 80. net/core/skbuff.c | 50 ++++++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 36605510a76d..ee2fc8fe31cb 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2990,32 +2990,32 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset, } EXPORT_SYMBOL_GPL(skb_splice_bits); -static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size) +static int sendmsg_locked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; + size_t size = msg_data_left(msg); if (!sock) return -EINVAL; - return kernel_sendmsg(sock, msg, vec, num, size); + + if (!sock->ops->sendmsg_locked) + return sock_no_sendmsg_locked(sk, msg, size); + + return sock->ops->sendmsg_locked(sk, msg, size); } -static int sendpage_unlocked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; if (!sock) return -EINVAL; - return kernel_sendpage(sock, page, offset, size, flags); + return sock_sendmsg(sock, msg); } -typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size); -typedef int (*sendpage_func)(struct sock *sk, struct page *page, int offset, - size_t size, int flags); +typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg); static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, - int len, sendmsg_func sendmsg, sendpage_func sendpage) + int len, sendmsg_func sendmsg) { unsigned int orig_len = len; struct sk_buff *head = skb; @@ -3035,8 +3035,9 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, memset(&msg, 0, sizeof(msg)); msg.msg_flags = MSG_DONTWAIT; - ret = INDIRECT_CALL_2(sendmsg, kernel_sendmsg_locked, - sendmsg_unlocked, sk, &msg, &kv, 1, slen); + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &kv, 1, slen); + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3067,11 +3068,18 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, slen = min_t(size_t, len, skb_frag_size(frag) - offset); while (slen) { - ret = INDIRECT_CALL_2(sendpage, kernel_sendpage_locked, - sendpage_unlocked, sk, - skb_frag_page(frag), - skb_frag_off(frag) + offset, - slen, MSG_DONTWAIT); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT, + }; + + bvec_set_page(&bvec, skb_frag_page(frag), slen, + skb_frag_off(frag) + offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, + slen); + + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3108,16 +3116,14 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int skb_send_sock_locked(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, kernel_sendmsg_locked, - kernel_sendpage_locked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_locked); } EXPORT_SYMBOL_GPL(skb_send_sock_locked); /* Send skb data on a socket. Socket must be unlocked. */ int skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked, - sendpage_unlocked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked); } /** From patchwork Sat Jun 17 12:11:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CA4FEB64DB for ; Sat, 17 Jun 2023 12:12:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBF146B0081; Sat, 17 Jun 2023 08:12:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1FCF6B0082; Sat, 17 Jun 2023 08:12:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C0858E0001; Sat, 17 Jun 2023 08:12:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 899A06B0081 for ; Sat, 17 Jun 2023 08:12:38 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 608321A014C for ; Sat, 17 Jun 2023 12:12:38 +0000 (UTC) X-FDA: 80912127996.17.926E0EA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 682981C0018 for ; Sat, 17 Jun 2023 12:12:36 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RiTObZ4f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003956; a=rsa-sha256; cv=none; b=GLdRYxMBsHJhdoEVSObx11uuElHxqJPj51V2craTPQp/4lSG+LPeRyk3PHHueiayIjg3JP 1l7caIv+71hKQJ2994mfwUx5jAppFr8lvzZiBmGQBheXfTPmGjfsZlwFSGHvNZ0Pa+cNza piOBmgEbh+AhDY2+Rb56B7EGxfAkETs= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RiTObZ4f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003956; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fSQZQ6YboxRaXsiohOL5R9kf0uRRv2lhrE9A+pyCn4U=; b=Ui5DV+FhcwthTIJPNYgKr8pUFsEBnpTzDCdxs+FE9wTeBx/Pau1EtgKnCa8cB/5xC5fcG8 /MNq7grDd4d4WFC1Zric/juLfnUTdIqNSwGw8yKN0Jj8rXMUnAAjRKI7Z9ZJrxOaupMFSf S246S4Y0XThpEv6fYiQWrkkTWjetlhs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003955; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fSQZQ6YboxRaXsiohOL5R9kf0uRRv2lhrE9A+pyCn4U=; b=RiTObZ4fo5MrGbfgl4IjzAZh+muFN9A/dGeMQR7Trp48+3ybHH/14LIwS2N5GVlrI/INVL a4bnS4Jvf6THFDY3zjf2HHAxHanEZuvrzlBeHP2YWCruDqyj8cVnRa7O7zO6W31/Krq12J sRQrEjZbcz3xEB9UrHADEovs7rGa3l8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-443-e8hi11tzOTydYlQ2AkFMIA-1; Sat, 17 Jun 2023 08:12:30 -0400 X-MC-Unique: e8hi11tzOTydYlQ2AkFMIA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F45D2A59540; Sat, 17 Jun 2023 12:12:29 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5FE79492C1B; Sat, 17 Jun 2023 12:12:27 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ilya Dryomov , Xiubo Li , Jeff Layton , ceph-devel@vger.kernel.org Subject: [PATCH net-next v2 07/17] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Sat, 17 Jun 2023 13:11:36 +0100 Message-ID: <20230617121146.716077-8-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 682981C0018 X-Stat-Signature: x86xba137add48itpz6gxopt48uabk4u X-HE-Tag: 1687003956-931389 X-HE-Meta: U2FsdGVkX1+q16OXRYTOI3hAvBpFUynvIERlsNmsGMfy4RjbB9tsIL4BAgAtpQ0GGVfaluShj+J5ICx68LAROmT0IBLvUQR/fJJHIUy4/0Fu6MaaFDbNr0DYjyhNvqwpY1kk2A2zxn6P0S1KAkG+iFUK/6+f6bzOxoeIMBCoPQ2cuct7I7Vj7i+T0Lku3rf5mosaaz99HNz/8xAWqokmmRRsmRymiVvzGRItGycBl0PLbsNeCananED0NxtKmkWiLBFaaKDmfftQ6pWM9EZd8HHNOrPg3l6vvmMbAUaBK7RkOJE/gy8Qymdv0VMfSzzxCi+aBWnscQFNDvjEJcP8OlIjSIxkOx5fUKFTIhMnWt85Y/tHe6Z8kWXex8zrdFGW5OmqLA28Uhw24tjIQ5jjLUOmEuAkbu3ssSuQ73mYrdulbGzT6ZLX7RDDFzMok9Ep60DcgHmPZ6HYk1gFhpKFQg6mWXvsstCk+6H73wY733hjDtWGcIJptFav8FRL/e65WiBlIiwpxNNN6BCpjTaxjFxmsfgTgtHO/gRssbX99mV34g59zZqNANbOXcZKvOyUPP5XdcuqFQNldT7QhUHMH63iE76KG5rGN7app9JY6KxZ3XAKOIP3yRXFr1kssMVoUeDlrr/hrko129hf3DN6A+Ev/bdAxjCw2M8CjsWlbyxyhwKnnPcIv4ZOaULQcaOH5Y5FklPmh+nkfV4lJrqXTI1j8UB8UnfZNqcNwMMoBSR74u1qmFDcc2whUMKOMjOB3hwOlxx/ep9Ljusvp+FQnKH/aKmUUogmebtTYYXj8f5sH5vdRofaMLA4jUzfn/dUzUfvYU+pRo9bCgVFsvBeYeps/+swFI7yYmJV3wsBtrzqsZ5bjpNB0q6UP0gkrN3ORF895cP5KyNB4NhSDlbZN0Q+5JuX3RvGojZUyEPAWMgnR9ehGPVbcoO1AFdqXFnqf5GYXjFEQezZKxTqDi+ hhgY09mt VkUdptnYD7LqSfb4qf+RmYWj5YEWMR4dmkFd7I/KPQ1/2+1f/FiX7CEX1+9YTfL0JL1PsTNzzzADc5/6v7ByREEf0jbz/yfuq7lLqJMJwAUBmxe+cAFEWM3egIByfCJLzD1o2mxqj8TTm+VBboLyfeMeZChiuAN2OhN/A5wWnqBEm4EwsE75pD0rTBfzQClcHkgYzreZSUWDhXst1bt8mp1VvBx37p85rba07Tseww8VlGACsuFVfxQ6Tep2kYdXEsvbF2K9EGnTC7PqNtL28MXZ+BAVZXFZQ3IUZuMRZlSiSvVDRTto8o32R7GSPQowFU5yDp1ytfXhE0DR3BGe0cPwwjg849vgoL27pWfQ6m5mxYQy8Fl5/yY+5d4xBd/K6olHwpXjYgVB9MCJlieZBr/lHdTtEsPaZG6QrJzD+LqM3FRUKfEX7TVnr/Tge9N6I15axlxAIYjm0VOCegg4UqyZCcJj2QKDu5PBeh2CaQehHuSCsae9wiu+d/+HsgfRAdynukUOPUDs1v+A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v2.c | 91 +++++++++-------------------------------- 1 file changed, 19 insertions(+), 72 deletions(-) diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 301a991dc6a6..87ac97073e75 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) return ret; } -static int do_sendmsg(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - int ret; - - msg.msg_iter = *it; - while (iov_iter_count(it)) { - ret = sock_sendmsg(sock, &msg); - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - WARN_ON(msg_data_left(&msg)); - return 1; -} - -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - struct bio_vec bv; - int ret; - - if (WARN_ON(!iov_iter_is_bvec(it))) - return -EINVAL; - - while (iov_iter_count(it)) { - /* iov_iter_iovec() for ITER_BVEC */ - bvec_set_page(&bv, it->bvec->bv_page, - min(iov_iter_count(it), - it->bvec->bv_len - it->iov_offset), - it->bvec->bv_offset + it->iov_offset); - - /* - * sendpage cannot properly handle pages with - * page_count == 0, we need to fall back to sendmsg if - * that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag - * which triggers one of hardened usercopy checks. - */ - if (sendpage_ok(bv.bv_page)) { - ret = sock->ops->sendpage(sock, bv.bv_page, - bv.bv_offset, bv.bv_len, - CEPH_MSG_FLAGS); - } else { - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); - ret = sock_sendmsg(sock, &msg); - } - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - return 1; -} - /* * Write as much as possible. The socket is expected to be corked, - * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. + * so we don't bother with MSG_MORE here. * * Return: - * 1 - done, nothing (else) to write + * >0 - done, nothing (else) to write * 0 - socket is full, need to wait * <0 - error */ static int ceph_tcp_send(struct ceph_connection *con) { + struct msghdr msg = { + .msg_iter = con->v2.out_iter, + .msg_flags = CEPH_MSG_FLAGS, + }; int ret; + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) + return -EINVAL; + + if (con->v2.out_iter_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + dout("%s con %p have %zu try_sendpage %d\n", __func__, con, iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); - if (con->v2.out_iter_sendpage) - ret = do_try_sendpage(con->sock, &con->v2.out_iter); - else - ret = do_sendmsg(con->sock, &con->v2.out_iter); + + ret = sock_sendmsg(con->sock, &msg); + if (ret > 0) + iov_iter_advance(&con->v2.out_iter, ret); + else if (ret == -EAGAIN) + ret = 0; + dout("%s con %p ret %d left %zu\n", __func__, con, ret, iov_iter_count(&con->v2.out_iter)); return ret; From patchwork Sat Jun 17 12:11:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CF9BEB64D9 for ; Sat, 17 Jun 2023 12:12:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A135C6B0082; Sat, 17 Jun 2023 08:12:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94D7D6B0083; Sat, 17 Jun 2023 08:12:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C4B38E0001; Sat, 17 Jun 2023 08:12:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 699936B0082 for ; Sat, 17 Jun 2023 08:12:39 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3590BA01BB for ; Sat, 17 Jun 2023 12:12:39 +0000 (UTC) X-FDA: 80912128038.06.7E523A2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 7122C1A0007 for ; Sat, 17 Jun 2023 12:12:37 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YzPKAeOv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fr8R9cUCocDj4LYXtGmeXkCgA5OgO4hqxYPqdNjfTX8=; b=tVfBBx+VL7CU+QTotOD33VIyqQTSeYQXYIcyQh1q6vRsMvbzfu68MJqUCEV3dtjII2uOy/ KEUZUPUVRq6XwZXCTcIkPDPhrrINUfMtgQi/w3j1Kz0CeeApnAd+w/i2qP6FRnN0/VZDq3 qITlK1/Pi/+nZKDf2QiMQpiGjVNuUM8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YzPKAeOv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003957; a=rsa-sha256; cv=none; b=AmTqp91X/IUXbTqIRQncWAlJi+lkORsqM38AcgcgQu13lWkl+dotrsdQI+qO5d27FWqBVn +KQR6zJC6mjW1cAwjreN+AOam2ePChcgt27CQX4OrDD4dfsLggOxAZY0p6STK1Y/2SW2Rk kUt0slkg8RP4PoDK8L9DSAYkFpF1jD0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003956; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fr8R9cUCocDj4LYXtGmeXkCgA5OgO4hqxYPqdNjfTX8=; b=YzPKAeOv9WsCtWwkqOyhSW8z/PLDBbZuMSNWbHK9M36Gbr44EJLk1YrIsXK44XPaFUiHuD 56K7UonSH4XnHiicuZncAlS8eXCasNKVUtPkv1GyFi/jHpkUXjhGutTkRvra0hRFSoPLrr Lew3RhQQP0F9UeukbCgQWBikweLIycg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-22-5_avo79NPta2RKQidrmpCQ-1; Sat, 17 Jun 2023 08:12:32 -0400 X-MC-Unique: 5_avo79NPta2RKQidrmpCQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB1253803506; Sat, 17 Jun 2023 12:12:31 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A20F1415102; Sat, 17 Jun 2023 12:12:29 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH net-next v2 08/17] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Sat, 17 Jun 2023 13:11:37 +0100 Message-ID: <20230617121146.716077-9-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7122C1A0007 X-Stat-Signature: uuodqt47mrabzc3opt9ekna774j43ja3 X-Rspam-User: X-HE-Tag: 1687003957-722103 X-HE-Meta: U2FsdGVkX1/ze/LgdL3WscMsvF2nQ3Zsweuuc6VzEwB28nKXtfztGn9W1271MDUoR3eUJkV3tY7OuRlPrbAnzCfBl4ROhSPxh3dMef6ibnJCUqClVGBQ1LJqH4ilp+AhnPLXnCZiE9stqvMj1Lh61DB5lPNN1bTY5+2pwjJDgSrpNZGogIoTGjPFryyH3Np1yQ2+XMdd4/odAxHUmVYQQxE8wrm7JIiSTTvJGmKyo4CpTTeBhcO0rEKEqVxaztXKfSapKqy/17UHzkFa2ivB09nY8iRZiwnYA9Kq6qg2qu4r91lUq+ECY4x6mWQVf5Q+08DF5nCA0AxgBEmBPMhURcUXgikIApSm7YL4f5YJdhd6FCmqFbhlWXkYzuclGSl52lzY5nQ2IrAy/GlVumRDSe5vYO1COeusos5GZM1WLoZAhE3W6YqV71YNvaZConzeHB/Qva2VazdjLyQevGnEWN0Lsa5zh+4fzQTQqwKw8T6BDrK1mUrFvu64JyhbFsHAwQYk1I6Q80S/ZaM0L2aPRK0OqdwKkerLXG7J4fcOPmlID7Bp2tire6cird+VJ/5se3GJW6zCj6KGEN4+u80KBo55mfR1F2VobLR01HdgsZH59HTHSlaumCHIz4AakdjlSKCcpvwfmzQq6a/mQNjs5us+lT7a19Mm7rC2/TKZNgZOyfP7CtIGZ1ShDYXxIU1rXdLG37WSxB28kA22vvZe0CI2PRRIfGCMb0YK1QR9KbjMn/v9bx1+/M+9nlisGvNPH7fURKAcOplas99YipYiUGQ8GQs7MJU/pIxqT7AzsZ7wVqVHEF8T9pTN8Ey8JnQsQMwEzckhunaXzyJ8rHGswOG6DELCWMi5kyvrGnJQpsgnTVcs+xa92tSNtG7MphybYHmZH0Rg9dd+gskIoAkQkOvBO68fT31KmuznaCatUhsczxKelOtAcLHqMzg6U402ghWne1x/sPkxNREYZC/ MB1nmAaY /Kh2bZqWJKrAxWFI4hNlYqyIU4opKSPB6CjvKaq0Z6L5U1qE1TEqaKPT/WiiXN3YmdHTrj3YlmwpUQp6TnVTwTnubM+dnusi2Li4F4I+VJI+GwPdRf2ephmXd47bm2GQabLKsGNpoPlc+PuCa2CTVaxbJYnVxoFHKAzr92ff4hNgXLYtInnfY37BfO49qGGY5v4zLRoFOfDo6h5deJf3+vGhqpcIGyUiBTm4ruqyRXC0Bmk7RLWgV+ifjRJd7aPKvmwyHNcgMIYYf33JTejPIBPHkxOIK4PZ6whuyLCRtJwjRvPosMcOlvxd+IKdrcB2cwrH03rxM6V1NkxxAjo4meas5SJWK6gnHgojAzge+oeovlYQIvg2fFGmRURwHINVtb6u+Z4TVzy9BdPfquyoSayguHITW/pDh1wFBhJsJBtbmOE8r9VoRL56DZeomLPHI0ZYq6UOpS+uI7/M2ORptQsUt1RtKo98vOgiY9EbySe3SOXfKGUbmpvVdN2Ayj/o+6dotXtZGTRH5InE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 74 +++++++++++++++++----------------------------- 1 file changed, 27 insertions(+), 47 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..550390d5ff2b 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,23 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -101,41 +95,26 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, /* see rds_tcp_write_space() */ set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) - goto out; + bvec_set_virt(&bvec[ix], (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off); + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +137,7 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + kfree(bvec); return done; } From patchwork Sat Jun 17 12:11:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42B33EB64DA for ; Sat, 17 Jun 2023 12:12:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5C096B0083; Sat, 17 Jun 2023 08:12:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0ADA6B0085; Sat, 17 Jun 2023 08:12:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A85888E0001; Sat, 17 Jun 2023 08:12:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 910666B0083 for ; Sat, 17 Jun 2023 08:12:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6448A1A01B8 for ; Sat, 17 Jun 2023 12:12:42 +0000 (UTC) X-FDA: 80912128164.10.9452039 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 09AC71C0009 for ; Sat, 17 Jun 2023 12:12:39 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G4Lymllm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CX0szcw738ZS8htZ0RxEmjBFgUMhbKSUPMuc1XxmplE=; b=igqFSkRz2ykr8RgMR41qfFgwu8Mex44RzFnPwPhnB25sZZXm2dRI5BNLAmWHJfYsDOAIFv eF8Eura5OPMmotOlv22mOdT/dI5MC194QR8FGYath1htuw75QaXdaDx63wnJ2vsuNG2xS3 m7KtraStlUagRL9IQwj534szn2LwfVk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G4Lymllm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003960; a=rsa-sha256; cv=none; b=WNy0qKCecZ3XZe7Tz020sKNfWRs4r5jZ3sQSS04Tybn2c9slBnoQ3XqZspuk3J2+9RaVoQ YKTfly/HWLb5i1H4c2HQWp8m8D/b7lBGoUTBiniaZfC5hRIPnbwCR+UmEOq4yA7xSmSTwb 3iT0HONRB4M1Svga4UpkgqSPBjaIMu4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CX0szcw738ZS8htZ0RxEmjBFgUMhbKSUPMuc1XxmplE=; b=G4LymllmGQix3iv8pz6y3kQnYz3GgTBzpjYcta8M4NhdtI4CgF5uZ/zX49VycDl0gHWbEU hbBdqifXojtT1Oqg+6YDuUomKF0JnauhYHQrCr+YYhlQTz3OP9k2Z40Y8UIWz/wzEez0Ks IggnyYrPJljBabvya2GUUZjkBgVR19o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-198-ozIpHJTiMYSJ2JP6C3x6Cg-1; Sat, 17 Jun 2023 08:12:35 -0400 X-MC-Unique: ozIpHJTiMYSJ2JP6C3x6Cg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3F71285A58C; Sat, 17 Jun 2023 12:12:35 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A69640C2077; Sat, 17 Jun 2023 12:12:32 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christine Caulfield , David Teigland , cluster-devel@redhat.com Subject: [PATCH net-next v2 09/17] dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Sat, 17 Jun 2023 13:11:38 +0100 Message-ID: <20230617121146.716077-10-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 09AC71C0009 X-Stat-Signature: ez3pdzbssohwa8fajjiqt36w1b13derz X-Rspam-User: X-HE-Tag: 1687003959-121337 X-HE-Meta: U2FsdGVkX1/R97RdCh3rcUIHwxdbuTQaQqLsvpGd7oSTH4bQh4BYWpPqXR6VVzyOMsv7G9O1b2sW7Tn+G4jWVCFUWQ/KS8p/hLDpOB3QXbPiBgMlqnhtwBKb/q+Xin0+il2lf2WxAsgCff5vm9TdDxYAn0Q7xX4MRhzhu+qzh73dHZvGiGyZJZrhNUTZrJbSlCv9KehpGZF+D4hX1mTPWwC0aRaJMxAEPnL1zEJOXCSbn2mydaZGqcctuDTq2LkHZNV9k5SWpcdcI/7bx4yZjybUMnH95m4GbsFSEtMhUqZsFWSWKXBnxoYXZpMlZkvFTFzyLXGuHZ0gOlIh4uoTSdtRCIIX7nmio/meNBXSPrmdmAWa1S8lU7Uu/uTr5iZ4I8v1wUTARpIMw4CfT2zt87K7/HWgtOjYh4DFt4jmnJtC3rBwXhu2tYwBa9c0aQaPbuziawVmb3VCmGC6TvIteAU2gq8JervGJBXZOZ5OL4aOmJ0DVXL/8xqViHLkzVfVNzgsDHP5PGUfMfjDcVNH2L8zv8RyZS2mRzA8f/r+HkQlR308WUKDriPNpHl+FpTK4bqsYDsKBQ57/I7bu+zounPGZ0uwIX4QQjItWIImWinx/Ke4XcnLgYxvnDNJtpr1LBOmxusaH2rszvjmqqLqy0lgrWZ9vaqi2t5meYU4DfiqUla22Y3UelDqz9W9V6bvCMGnU3oWi13/u0JfN+shdU4j0s05ctO0TKx3CnOGNbubT5ZkBxr8cyP3Mrv5zM3ZSpyhwgwkXqX2215ZTBbym4vKb1ZwbyWRHUn3mLlohhbO+NLm1ggoyJEOmDNYPgHEKqpvjZlYi3WcDDoHQLSgQ/qtS+ba7k4b0wcHSgKv8sWtIckWRak5ccRS6ahIhpRXi4mEBIkWgolvW6ovPDTagchAHrNajxtlV34p8r/k7MaSXf1i0Jsh87LHmVd2042YCQaBNt7EdwfPnPaUefq QVpIz3Np bgQlOprLOJHn/jyXWB6FUcN6bOlGxSH/q5ZpckGo3zYTdX1DX9KgTlIyiOj2YvcgyaD0VhkorS1jptP/3mciYGxdexFiSgZrY/xQAkA99G1UDG8bAH4+YEV2prXGCrVDuYfLnqitsHpWZr0koP8Wamzwo61bAR2sjUojq7z2tU+/NnCvSG6tmgCm+Xa/CoF1NFdHbEgkbKmHKe9B4Yvpivme6bIsnx4Ei/6rjGBIoC1VJ5oWCmGrfNuY4zkHpmGt/+Pz2QAwnrBjA1FZsqMMbRrNCWQBdLnyGJ/SksdlukzUBfflrmjjDhAclLwlfXOEFLow52Nd2kQjQELcgoFS5Hn9GKhi2rifFRihmB0a7n8+h8ssa7UDYwik2/JXDdjMmjd1kNBhMZHD+/1EfrUjoInnaHIEfycLzQshk6iHJj5wHFK91Vh5n0p2AJZmVRU4gqCmQGNVDl4h7mVe/hlArIzD8ere/c5poKRpft5TfGo4XFE6T3I12EEelnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down a layer using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather using sendpage. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Christine Caulfield cc: David Teigland cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: cluster-devel@redhat.com cc: netdev@vger.kernel.org --- fs/dlm/lowcomms.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 3d3802c47b8b..5c12d8cdfc16 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1395,8 +1395,11 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg) /* Send a message */ static int send_to_sock(struct connection *con) { - const int msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL; struct writequeue_entry *e; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; int len, offset, ret; spin_lock_bh(&con->writequeue_lock); @@ -1412,8 +1415,9 @@ static int send_to_sock(struct connection *con) WARN_ON_ONCE(len == 0 && e->users == 0); spin_unlock_bh(&con->writequeue_lock); - ret = kernel_sendpage(con->sock, e->page, offset, len, - msg_flags); + bvec_set_page(&bvec, e->page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(con->sock, &msg); trace_dlm_send(con->nodeid, ret); if (ret == -EAGAIN || ret == 0) { lock_sock(con->sock->sk); From patchwork Sat Jun 17 12:11:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC701EB64DA for ; Sat, 17 Jun 2023 12:12:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DFF36B0085; Sat, 17 Jun 2023 08:12:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 494B68E0001; Sat, 17 Jun 2023 08:12:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 330DD6B0088; Sat, 17 Jun 2023 08:12:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1CF866B0085 for ; Sat, 17 Jun 2023 08:12:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EACC1A01BB for ; Sat, 17 Jun 2023 12:12:45 +0000 (UTC) X-FDA: 80912128290.06.B2C40AE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 3976820017 for ; Sat, 17 Jun 2023 12:12:44 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F4XU7THc; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003964; a=rsa-sha256; cv=none; b=fl/vk8dA8j5KAtz+LgRC0PwKEJ348t0DMaJGI9O3hxDFqGBecvlgAm+L2ClQmNmU7Pl20B 2zwxEPP4XvNLmqoeWMPp55+m4yPuwXHdHzAeEGJJLDTlKzVmQVhLp4g0HpO9aGOqKpaQqa 5Uz85azVrKMH22crTLoVja3mZNbLONs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F4XU7THc; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rM5xhwav6LJluWMEyP50kZ7CJi2Lzlu13nu/U4g2kVY=; b=N03gnmc0hgzBDJq7fPU6Yq432iz3WabdatqWNkXtWkocm6m0aL/NNO4Oc44d8Xasf35mFd oP8AmzKpUkWBa5Hd2dTdxAQZKFDC9u+DrFyosRRHqe0Feop9KIf8MYcrEW8S49uLkeMk4h 4T88tjBKbf2Ol3+lnXytfAP5wOiIxjE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003963; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM5xhwav6LJluWMEyP50kZ7CJi2Lzlu13nu/U4g2kVY=; b=F4XU7THcayfQJvTF+ExMfjIsKI8WLM7OWw7T0FN5LQ+11D+CCn4G8qY3xAgKqNqMTCGLUu jFGnaJsxGYWjFIkfIdMvbYeZaBeNN8nGfj9qHlY+ZfMSjtBvlUPy9YM77W9lbzpxAjjMy3 dRU3gxXfen1gredlsHNVk1iwC+TvrMQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-507-mFr7GYahNEaJRqUoBKlnNw-1; Sat, 17 Jun 2023 08:12:39 -0400 X-MC-Unique: mFr7GYahNEaJRqUoBKlnNw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1C978805C3F; Sat, 17 Jun 2023 12:12:38 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id EBDA640C2077; Sat, 17 Jun 2023 12:12:35 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org Subject: [PATCH net-next v2 10/17] nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Sat, 17 Jun 2023 13:11:39 +0100 Message-ID: <20230617121146.716077-11-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3976820017 X-Stat-Signature: upcuer7wnnj4uoghetfwfj7f3az1dfx3 X-Rspam-User: X-HE-Tag: 1687003964-922769 X-HE-Meta: U2FsdGVkX1/mDKJVd23/sLfa488VPlRtP2dlHQLcQx8lcwShGJNdEoNqatwR9Os2Gud47/45X0xMRBK0iv9fNvECdMf4e5JY+ag5ONRJUL1uKYjhdMgJspCLblrSSHR7ZOTEuCsM2VuN3/h7yY1H4dzDp8ugBaFfFT007mkEgj0XCWGxgyr2luMPe70CIOTEAQwzopQJzEHlEeh7Fyvgy5cye8TxJQ8HwyzKqzVlWHuZQeGONKP4/TnGy7s5ZrmOf2TYm9i18TQr3idypTjgOfzGF7vCJE3J29qq/EYGFD3Zsq1ElA1NarifZH3nXwpG3UWjIfzMmJJRRTY9WWqgX4Q3VjUGF4h0Aj02ifzlkJTv31xO4b0WYmAiED67t6MQT1QoIGWaXdE4+x+x4uidgbD5WTQsXNEvjYSYnqPPcpdVB2P+QT+TEbkzjLBkI0WnJW8a5rdr7sjgB932orYA/aouHC1Y9bCYD7axapqKxPUMRF6wRVLdv2kNwA8VuglaFDkX+MstFMAzaGL4fy4kRMCoi9F6pAxQ9cdzeKb6t+b4OZvTxeS8fru+OYlla3aW3UreUY+nIkifbTCxDnrQ0prt8yGDO3m1Uf1hFrvFlJK9smdPa1IJhZGCxKIQMcYWUsfMHxTm6LlfsCTiuMgjvAmVxnidq9zgu8jwKexqLb7Oz/h4Wbiz5D9nWbXYwzlReX+/2hruQvz6tatPb6sYhtSHIG/TNfD1uQIEUJ7nx71XMrTuFEtERXnP7n5v12MvUZBZkWUJFzk/Cnfth0wt96gxVVinBW/zgwJ/9ov6GWU+M0oQwNYKKDKlwoksEfC+EXlh5Guuqazvk9DVwXYG7dUcrxziphaLyx+fgs+zmJpMaPboLM6Q4hdAop3J05Y4CnjJ32mlx2YM72x9rVopeZiqJ2qaTNsOIxFdxM8qGLCWDtF0snlQu511hQZkuiOel84Xf6s+cUtD1I0CIxL NmhsfDu1 qR5UQdKAAIV+I/wruh+cmiqlZ+ndscDf2pLScVX6NvbR934i5r7iCUltJC6crQAS18TTp5sucGE5AOCRw4QnhF6kskf8trFE6rJ9xito8QiuPOQ1imMaO2XwsMxlxmMFG7Lvub9DGo6xWDLn4qq4zUqiSNdLdJDvHjWVvkD/A20GEzIZMoiT6OHrVDjm2JePAe7KPo1DoEiGxaa8KmYe4lRmUwuCw5kxCEE9byVm9FRbYFQBL2YmpehIsX5qGO/Gluz/EvXE8dD22puRjfUM4BUPkQGGCNih+eLD16wLDutkIGbbZOR5/NgJ4jm4Kf07KT2oynKssro9/4TZWqH+jprC/jVbbAt4fqXq+HKL/CWD7IbsJRx1hA6XA5yPnILLkfWaHfagonxmzEn/XIeo2ykCvydKu387Fr4Pj3I74FgvlVwj32tThibitm/HrhBnCyTO7jFQhJQ2DFnPxkm6LTD1uDSX97sm8pBI+d5YedneW+NfmgyOg6nOW+QzH9HHHTUNsp0mRTiJwlrCvOWBwjbqWvTxYuej09gngthPbdwywV73UHTSLEzO9ydNpuNDlgWJsWl3LL/3nx6o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nvme@lists.infradead.org cc: netdev@vger.kernel.org Reviewed-by: Sagi Grimberg Acked-by: Willem de Bruijn Tested-by: Sagi Grimberg --- Notes: ver #2) - Wrap lines at 80. drivers/nvme/host/tcp.c | 46 ++++++++++++++++++++------------------- drivers/nvme/target/tcp.c | 46 ++++++++++++++++++++++++--------------- 2 files changed, 53 insertions(+), 39 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..6f31cdbb696a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -997,25 +997,25 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) u32 h2cdata_left = req->h2cdata_left; while (true) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; struct page *page = nvme_tcp_req_cur_page(req); size_t offset = nvme_tcp_req_cur_offset(req); size_t len = nvme_tcp_req_cur_length(req); bool last = nvme_tcp_pdu_last_send(req, len); int req_data_sent = req->data_sent; - int ret, flags = MSG_DONTWAIT; + int ret; if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; else - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; - if (sendpage_ok(page)) { - ret = kernel_sendpage(queue->sock, page, offset, len, - flags); - } else { - ret = sock_no_sendpage(queue->sock, page, offset, len, - flags); - } + bvec_set_page(&bvec, page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (ret <= 0) return ret; @@ -1054,22 +1054,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_cmd_pdu *pdu = nvme_tcp_req_cmd_pdu(req); + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; bool inline_data = nvme_tcp_has_inline_data(req); u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) + hdgst - req->offset; - int flags = MSG_DONTWAIT; int ret; if (inline_data || nvme_tcp_queue_more(queue)) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; if (queue->hdr_digest && !req->offset) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, flags); + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; @@ -1093,6 +1095,8 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_data_pdu *pdu = nvme_tcp_req_data_pdu(req); + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, }; u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) - req->offset + hdgst; int ret; @@ -1101,13 +1105,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); if (!req->h2cdata_left) - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); - else - ret = sock_no_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE); + msg.msg_flags |= MSG_SPLICE_PAGES; + + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index ed98df72c76b..868aa4de2e4c 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -576,13 +576,17 @@ static void nvmet_tcp_execute_request(struct nvmet_tcp_cmd *cmd) static int nvmet_try_send_data_pdu(struct nvmet_tcp_cmd *cmd) { + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_MORE | MSG_SPLICE_PAGES, + }; + struct bio_vec bvec; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->data_pdu) - cmd->offset + hdgst; int ret; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->data_pdu), - offset_in_page(cmd->data_pdu) + cmd->offset, - left, MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); + bvec_set_virt(&bvec, (void *)cmd->data_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -603,17 +607,21 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) int ret; while (cmd->cur_sg) { + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; struct page *page = sg_page(cmd->cur_sg); + struct bio_vec bvec; u32 left = cmd->cur_sg->length - cmd->offset; - int flags = MSG_DONTWAIT; if ((!last_in_batch && cmd->queue->send_list_len) || cmd->wbytes_done + left < cmd->req.transfer_len || queue->data_digest || !queue->nvme_sq.sqhd_disabled) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; - ret = kernel_sendpage(cmd->queue->sock, page, cmd->offset, - left, flags); + bvec_set_page(&bvec, page, left, cmd->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -649,18 +657,20 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; + struct bio_vec bvec; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->rsp_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu), - offset_in_page(cmd->rsp_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->rsp_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; @@ -677,18 +687,20 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; + struct bio_vec bvec; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->r2t_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu), - offset_in_page(cmd->r2t_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->r2t_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; From patchwork Sat Jun 17 12:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9572EEB64DB for ; Sat, 17 Jun 2023 12:12:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6422E8E0001; Sat, 17 Jun 2023 08:12:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57B666B0088; Sat, 17 Jun 2023 08:12:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CE9B8E0001; Sat, 17 Jun 2023 08:12:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 29CEE6B0087 for ; Sat, 17 Jun 2023 08:12:47 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ED3ECC012E for ; Sat, 17 Jun 2023 12:12:46 +0000 (UTC) X-FDA: 80912128332.23.DF71B3B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 12F53120010 for ; Sat, 17 Jun 2023 12:12:44 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DqkxZ5d+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003965; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iAUCq6AQpwiSNlqp6e2886GYsbEpCz4tvlCIXkUtiq4=; b=0LIwZoRPZQUvhehxn3XFzAMeA4SrnPFXNYDni++6LQFDG1lcbMt3QyJpf93lxgxYa++FJe FIHwzRWY7jV6dGrgHEBYgPQgvMBd8eSl5bnx25wJ0ydSO3/BjJq/JwdFkGX+ZXuPkJePUQ WZNAoGK1uRK4gxtY2awyasViRJfaUOM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DqkxZ5d+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003965; a=rsa-sha256; cv=none; b=FdyLC/6U2Q8XocIxoaPNGbTyeBr748iUgcs4KZkj+2yiciTnHwoLUu4W/4f66FY4QLmVcX qbJBt25VLI4t+uUlDlDvafYTAx7fSqT9JZPKgbsqyXjId4cuhe9s1Kn+yfCQorZiAUOk4W yclkmn5rh2lz2PbQT8ytCc+6WrfHFZw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003964; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iAUCq6AQpwiSNlqp6e2886GYsbEpCz4tvlCIXkUtiq4=; b=DqkxZ5d+8DFXE9IJWTDt9DNeCmJKjwClq2XYLlh4CrmnhQRo4Z7q2vOl7Qwcbhhmqo0G3D 2Fu7j6zgWeuMThpiVPRl+RVQ7oFPalRaBbnLalhhzhQjjp11ok7Ej6fhbobca8ebeWvHE1 bckQKhiy82Av+ThfDsaLPX6ttn9XJSs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-473-tTtJZPH1Nh6NXzEdHxJjJw-1; Sat, 17 Jun 2023 08:12:42 -0400 X-MC-Unique: tTtJZPH1Nh6NXzEdHxJjJw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E86F780123E; Sat, 17 Jun 2023 12:12:40 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id B3492C1603B; Sat, 17 Jun 2023 12:12:38 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Karsten Graul , Wenjia Zhang , Jan Karcher , "D. Wythe" , Tony Lu , Wen Gu , linux-s390@vger.kernel.org Subject: [PATCH net-next v2 11/17] smc: Drop smc_sendpage() in favour of smc_sendmsg() + MSG_SPLICE_PAGES Date: Sat, 17 Jun 2023 13:11:40 +0100 Message-ID: <20230617121146.716077-12-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Stat-Signature: njc39cmj5ftbhxe8zqjooqj7i8abjmz7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 12F53120010 X-HE-Tag: 1687003964-216794 X-HE-Meta: U2FsdGVkX18x7ciXGQgraYGT4gxUs92OlHFFQaA/Iy8tO1S4DQvUI3Xpi4/n5Un0FJtnRjbO3Z9NP9AtjdDEOQb1TSN37P9na35O3xyViWgNOC/HpAYZEGxjHPaYiZTjhYc+PCjNdI1aQm/8ZKeJrWJchTrerkSanDswv2sfP3bkhh3/K0Dre0lbOvDkT0/wZbhVMhJp2I4YfgYKCH9IVEcVpMlMNsTSOaBxFkVgif/vLrDjif6bXt2cZa4R1hBnAjUeHwPDe8i/Z5DwfZy17r8hwkb8znblP0GHomowD9NGtTUPIJNwX6GMQPCfbKZfTo2h+IVPl4ISglnaRizpJeXflQWXWnBdkDEf/BLleAVDhCvb9bzSC/XG9gYGt5ch6a83718JJ8yF1nBiG6iLPOQHpnL9jIIKxN/QL59Rko6/iBrcouO33wS7KUfNgxU9yJS2G9lFeh9DtXJgpG0G/spRMpF9PpEbb75tM4WXBdLoYOZgth3tlitfqYuGbq38aIRODiKPeJqCaT01vhbcTOWXTX23PfMoucVCngww5VYF2vADqbFoV0x0k98GlKC66Vjn08cBBFKY/+Sk/jyyctlrOzYsqgwKeLjbnhb2wjRVPTlML6tc9u7g6DkJIWyaHAXgSvMaV5P7GA7J9Mv599IDdyzT5QkbmDRmMbqPaCt++gUkOLJGcV+KoGVlKlwKGgrnBAKykd5PSw2dO1D9we4hZEv0vDkM7E/by3DjZ2OOpqxfgH8/S+4tHyXmdUNYiTjQw/HgGo4PSh63Kp0vsmn+xb0T2M00ct1rEmiqb42qaflbSrTvJiUmkEXihlRznoo8+J2mLIQNna26j/9YDFEuBoKApwd3O117mYjCL4XLM4USRRZdgX+c6iHHi1h4IA1H9M9x2PTHR2DxprixA0MOmvip+hcyosiWMiotwF70kCh/963tyB6RQlCzcTloC3efYI5lot1U/cQn06L LgwXPUqE qFeSvYv+j41+0qGWhR3b3PIeqPGgAoWEkgRhD0ABAEUYRbUrURexYPjYhG/R+44Zg8xRxYuvqbqiQrOryCweRMvEuhW5ss9uNr0yH17rJRbz32DyGz2jXYAK1aj8XsgmHpg9ronRSf4KrCHSjOxOrjxTj5ZNjINYBF6ZhD5o4wv31GyD1UmfylUKsrZKzzr96v0BvMuVApKw78LC+XjX4y3205dz3KwAQ9mHiX/t3Au+Zb5yl5Ba72+fjqWbdXgGwPVyxwIvb765LlTq0FYBIma4UOEt9xs6uizfi6aOpaBlbnbYZzCxeJFCfcFTsBhjMl030NzDoipt4rbZJJSdki0N43Jfr8jESwPb8qAPNaUgKGtueaD+LLqRKjTJo24KVQvs9i4jRvgIy/5RjuiIcduBIzS74eduznGPoS24TZhAmV/IT9Krbg6hxplvI9+LF+pUfWt6KzOQsBCpIfiehOfdE5ESPJLR3qjOVN7LTf01CaBSb0j0ElhMeaGlhEY9NpCGHa4TDexRo42rBvjkZBTLkESA8drW7Yw1Y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop the smc_sendpage() code as smc_sendmsg() just passes the call down to the underlying TCP socket and smc_tx_sendpage() is just a wrapper around its sendmsg implementation. Signed-off-by: David Howells cc: Karsten Graul cc: Wenjia Zhang cc: Jan Karcher cc: "D. Wythe" cc: Tony Lu cc: Wen Gu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-s390@vger.kernel.org cc: netdev@vger.kernel.org --- net/smc/af_smc.c | 29 ----------------------------- net/smc/smc_stats.c | 2 +- net/smc/smc_stats.h | 1 - net/smc/smc_tx.c | 22 +--------------------- net/smc/smc_tx.h | 2 -- 5 files changed, 2 insertions(+), 54 deletions(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 538e9c6ec8c9..a7f887d91d89 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -3133,34 +3133,6 @@ static int smc_ioctl(struct socket *sock, unsigned int cmd, return put_user(answ, (int __user *)arg); } -static ssize_t smc_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct smc_sock *smc; - int rc = -EPIPE; - - smc = smc_sk(sk); - lock_sock(sk); - if (sk->sk_state != SMC_ACTIVE) { - release_sock(sk); - goto out; - } - release_sock(sk); - if (smc->use_fallback) { - rc = kernel_sendpage(smc->clcsock, page, offset, - size, flags); - } else { - lock_sock(sk); - rc = smc_tx_sendpage(smc, page, offset, size, flags); - release_sock(sk); - SMC_STAT_INC(smc, sendpage_cnt); - } - -out: - return rc; -} - /* Map the affected portions of the rmbe into an spd, note the number of bytes * to splice in conn->splice_pending, and press 'go'. Delays consumer cursor * updates till whenever a respective page has been fully processed. @@ -3232,7 +3204,6 @@ static const struct proto_ops smc_sock_ops = { .sendmsg = smc_sendmsg, .recvmsg = smc_recvmsg, .mmap = sock_no_mmap, - .sendpage = smc_sendpage, .splice_read = smc_splice_read, }; diff --git a/net/smc/smc_stats.c b/net/smc/smc_stats.c index e80e34f7ac15..ca14c0f3a07d 100644 --- a/net/smc/smc_stats.c +++ b/net/smc/smc_stats.c @@ -227,7 +227,7 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SENDPAGE_CNT, - smc_tech->sendpage_cnt, + 0, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CORK_CNT, diff --git a/net/smc/smc_stats.h b/net/smc/smc_stats.h index 84b7ecd8c05c..b60fe1eb37ab 100644 --- a/net/smc/smc_stats.h +++ b/net/smc/smc_stats.h @@ -71,7 +71,6 @@ struct smc_stats_tech { u64 clnt_v2_succ_cnt; u64 srv_v1_succ_cnt; u64 srv_v2_succ_cnt; - u64 sendpage_cnt; u64 urg_data_cnt; u64 splice_cnt; u64 cork_cnt; diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c index 9b9e0a190734..5147207808e5 100644 --- a/net/smc/smc_tx.c +++ b/net/smc/smc_tx.c @@ -167,8 +167,7 @@ static bool smc_tx_should_cork(struct smc_sock *smc, struct msghdr *msg) * sndbuf_space is still available. The applications * should known how/when to uncork it. */ - if ((msg->msg_flags & MSG_MORE || - smc_tx_is_corked(smc)) && + if ((msg->msg_flags & MSG_MORE || smc_tx_is_corked(smc)) && atomic_read(&conn->sndbuf_space)) return true; @@ -297,25 +296,6 @@ int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len) return rc; } -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags) -{ - struct msghdr msg = {.msg_flags = flags}; - char *kaddr = kmap(page); - struct kvec iov; - int rc; - - if (flags & MSG_SENDPAGE_NOTLAST) - msg.msg_flags |= MSG_MORE; - - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, size); - rc = smc_tx_sendmsg(smc, &msg, size); - kunmap(page); - return rc; -} - /***************************** sndbuf consumer *******************************/ /* sndbuf consumer: actual data transfer of one target chunk with ISM write */ diff --git a/net/smc/smc_tx.h b/net/smc/smc_tx.h index 34b578498b1f..a59f370b8b43 100644 --- a/net/smc/smc_tx.h +++ b/net/smc/smc_tx.h @@ -31,8 +31,6 @@ void smc_tx_pending(struct smc_connection *conn); void smc_tx_work(struct work_struct *work); void smc_tx_init(struct smc_sock *smc); int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len); -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags); int smc_tx_sndbuf_nonempty(struct smc_connection *conn); void smc_tx_sndbuf_nonfull(struct smc_sock *smc); void smc_tx_consumer_update(struct smc_connection *conn, bool force); From patchwork Sat Jun 17 12:11:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5FCFEB64D9 for ; Sat, 17 Jun 2023 12:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7ED348E0002; Sat, 17 Jun 2023 08:12:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79CF66B0088; Sat, 17 Jun 2023 08:12:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63E118E0002; Sat, 17 Jun 2023 08:12:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4ECA56B0087 for ; Sat, 17 Jun 2023 08:12:52 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2E2A81A014C for ; Sat, 17 Jun 2023 12:12:52 +0000 (UTC) X-FDA: 80912128584.07.56705A0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 7C80F180003 for ; Sat, 17 Jun 2023 12:12:49 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hrb8WRoI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LPVmbLfi9bgga9t/1pR/YmjTCDUThBHs3vchY9icGtA=; b=N2pkX1j+7CH0nBeAoX5YNT0j792h/zJDCjheAld2D/9/QQFUob2t0DSuScMGzaCKpo0UHE IJXGdCyZCnBHAgukcBYtPHe0IbzxpTiQD/nfk2aMSpLwEI0LH/v1OSuMdYXWFekojpH+kM EomX724eCNp0pg/rpMocKA02jxr7vS8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hrb8WRoI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003969; a=rsa-sha256; cv=none; b=z/zcUgysmwh0rcKec5xg93qZhVFVqeH39/i+7PULrL1lSewazkU9IaVHNCK48IN/Gh4Pah cBPSHLKd03fjtff0G5Di3jc6pzGyaoKhpzHd59KhGo/qiMRFVYFRQOuTSF5HF5NoulNCiF lJ/wVu6DfMn4bUjNmrO6cnFwIHf7CSw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003968; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LPVmbLfi9bgga9t/1pR/YmjTCDUThBHs3vchY9icGtA=; b=Hrb8WRoIkOCBj6pgI6SJMZKxFWFlKQjctRarkr2Cb5vW31H4AwaXcbLChVkCp0nauS+8LP sXODrk4VnsrsyRmK1S4Yf+d5/uRFWhajos8hKcQqNZC/YzpWE2QNeaHKzqw232UoiAH0UV P7GcCcZp1Vn9R5rc9iMUIjIQsKiII0Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-73-i7jsZDRYNiqwgOuR6uvruA-1; Sat, 17 Jun 2023 08:12:45 -0400 X-MC-Unique: i7jsZDRYNiqwgOuR6uvruA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B54E58007CE; Sat, 17 Jun 2023 12:12:43 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id A9D0C40C2077; Sat, 17 Jun 2023 12:12:41 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mark Fasheh , Joel Becker , Joseph Qi , ocfs2-devel@oss.oracle.com Subject: [PATCH net-next v2 12/17] ocfs2: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Sat, 17 Jun 2023 13:11:41 +0100 Message-ID: <20230617121146.716077-13-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 7C80F180003 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 3osoyw7eiazbu7k4aqd5yknmdpfmcbgz X-HE-Tag: 1687003969-366756 X-HE-Meta: U2FsdGVkX1/pezBVZXytyYEcds1Z/r3SB3I/RMBThcYqzCkx5rM2oE/XiiqbNPMA0IsrU3AzYeQCnrTvdF+A7tz/JMh8/UHT73gqyYzP8wPBRplo5Q/Dhy0LBcn7okX4zSor8U+usiEak1Sh9VbXy6GfvS7ioGXqGExiowVH7fGFwyHMjKBbRLrcBP4WrHCdw76NgjSZgrK+dlJgA1yv29H2c0ivFmG1wwFfRLUCixcFeeAni64XVlisilY7P4b8MfCPmvrOcU3OmrBMv2jN2JQiAntbAHT7yk7KlFSViFV8M8AO/43cc9sJ31SpI0I+iVc8x1o/Mi9W1pfGIefCZm33gjErQ72c/Jmq7DvhxmeaM+4gVdR2rbz6xPWfQL0sBbUM/bHep+AgHxcu7BmtfyoQOG43iFQf04dPBT2DhFoJ1Ylx2Zxog7sLquig5OheCsY6oO0EhE3moQNftNAzzWVTUm5k52lwCbOHmGz3Zu05a8jC2D+juDWAHyZ6aHi0WEG6My1g6cl8Ys8nN39f7pvjziNlCsgulUkM+N2Og/SZd4IsKJ1wiQcF2JLqIxJQrZqloRpmeJrpKblWgrWB0NvrVbmIwwDECke6HUbTwnBlQzqrdURMTEFfTQWn5UXWeEIQkiQlSeMs9QMKmBmmQz7VJhVSbD55ftxH7YRtS8q4jwCLeikLedd1cKEh87neKZozqSVJyuTsx5nL4xJbpCbPb0Oii17ZEH7AYNGAl6rWbQSbV9ZknapQMpsY9RinqfhBdNMCO47t4ThffvE7HJPNcZiSdVNrnMUwkPurcsa61/OPPBOMJD56SqIeikSBbY/3aAsD4vtOLxHrL1Jy93iNnC9kOR/resYwBj17319xGTAQdcsrdEZ0PIpRXZWaQtZjHcNobUvL7AfKtf9fia7FkBJHDKnzgLCPWrHvdEqta0YvFL4WaU197JFLQJR327D31mwireKd5wKb88A edvGE30M hBWKGDhFX6SoPOnr5D1lmM5WTFZNefsBh4WtSk1r9Qnk1Ly54OLuhNvfmx5L1eZmV1l17RFjw7IaFbfk27xjrGDGZkzzhFviBZhVTzJH4+ktQq5SFCkm/FNsmIace19ar4FoM3uVIAIs1CXeYFQD3Mfsnwz/5VJVMtCXdfbUctJ4UPo5Iebr3YbrSTqg5qpYlbvIdJ+cmzxXtVKLCT2tZZOSGHq8mHsittWq5VpudA38l+fj3mYlbMDesEFujAfTgWsRzdD72n+sbNK85SyzrqPuRQ1e0eZnLPDSjEqQShEDxKFVqvZhLyfrJI4jKPne3w92sBEkJ86FVz2n9abrlthuZysRWYzo1IEqiBszZsUKLNb2v0ghk6K/Fdh2uy1pvtObBk7l7ULkG7HEr4q4A0gAsOnM/9E2KUYuFuCwpUrlueHdQJK/ut2uXvBGZVFM8/55cVpG9GsNGOH0+22rI1y6cQNPNXCY/U7raARIE0rf3zthr46XMm3ZzrYG/bQWIz8WKQEdbivxqRA+yyH6X4lU/Q/ueyUTq3i5OsiRM07a5t4bff7TwDsJfzeov2wtz7MFVrp/U4kqgqQ4yQ5aeOPtpp9Eb6LomAr0o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix ocfs2 to use the page fragment allocator rather than kzalloc in order to allocate the buffers for the handshake message and keepalive request and reply messages. Switch from using sendpage() to using sendmsg() + MSG_SPLICE_PAGES so that sendpage can be phased out. Signed-off-by: David Howells cc: Mark Fasheh cc: Joel Becker cc: Joseph Qi cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: ocfs2-devel@oss.oracle.com cc: netdev@vger.kernel.org --- Notes: ver #2) - Wrap lines at 80. fs/ocfs2/cluster/tcp.c | 109 +++++++++++++++++++++++------------------ 1 file changed, 60 insertions(+), 49 deletions(-) diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index aecbd712a00c..969d347ed9fa 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -110,9 +110,6 @@ static struct work_struct o2net_listen_work; static struct o2hb_callback_func o2net_hb_up, o2net_hb_down; #define O2NET_HB_PRI 0x1 -static struct o2net_handshake *o2net_hand; -static struct o2net_msg *o2net_keep_req, *o2net_keep_resp; - static int o2net_sys_err_translations[O2NET_ERR_MAX] = {[O2NET_ERR_NONE] = 0, [O2NET_ERR_NO_HNDLR] = -ENOPROTOOPT, @@ -930,19 +927,22 @@ static int o2net_send_tcp_msg(struct socket *sock, struct kvec *vec, } static void o2net_sendpage(struct o2net_sock_container *sc, - void *kmalloced_virt, - size_t size) + void *virt, size_t size) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct msghdr msg = {}; + struct bio_vec bv; ssize_t ret; + bvec_set_virt(&bv, virt, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, size); + while (1) { + msg.msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES; mutex_lock(&sc->sc_send_lock); - ret = sc->sc_sock->ops->sendpage(sc->sc_sock, - virt_to_page(kmalloced_virt), - offset_in_page(kmalloced_virt), - size, MSG_DONTWAIT); + ret = sock_sendmsg(sc->sc_sock, &msg); mutex_unlock(&sc->sc_send_lock); + if (ret == size) break; if (ret == (ssize_t)-EAGAIN) { @@ -1168,6 +1168,7 @@ static int o2net_process_message(struct o2net_sock_container *sc, struct o2net_msg *hdr) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct o2net_msg *keep_resp; int ret = 0, handler_status; enum o2net_system_error syserr; struct o2net_msg_handler *nmh = NULL; @@ -1186,8 +1187,17 @@ static int o2net_process_message(struct o2net_sock_container *sc, be32_to_cpu(hdr->status)); goto out; case O2NET_MSG_KEEP_REQ_MAGIC: - o2net_sendpage(sc, o2net_keep_resp, - sizeof(*o2net_keep_resp)); + keep_resp = alloc_skb_frag(sizeof(*keep_resp), + GFP_KERNEL); + if (!keep_resp) { + ret = -ENOMEM; + goto out; + } + memset(keep_resp, 0, sizeof(*keep_resp)); + keep_resp->magic = + cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); + o2net_sendpage(sc, keep_resp, sizeof(*keep_resp)); + folio_put(virt_to_folio(keep_resp)); goto out; case O2NET_MSG_KEEP_RESP_MAGIC: goto out; @@ -1439,15 +1449,23 @@ static void o2net_rx_until_empty(struct work_struct *work) sc_put(sc); } -static void o2net_initialize_handshake(void) +static struct o2net_handshake *o2net_initialize_handshake(void) { - o2net_hand->o2hb_heartbeat_timeout_ms = cpu_to_be32( - O2HB_MAX_WRITE_TIMEOUT_MS); - o2net_hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); - o2net_hand->o2net_keepalive_delay_ms = cpu_to_be32( - o2net_keepalive_delay()); - o2net_hand->o2net_reconnect_delay_ms = cpu_to_be32( - o2net_reconnect_delay()); + struct o2net_handshake *hand; + + hand = alloc_skb_frag(sizeof(*hand), GFP_KERNEL); + if (!hand) + return NULL; + + memset(hand, 0, sizeof(*hand)); + hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); + hand->connector_id = cpu_to_be64(1); + hand->o2hb_heartbeat_timeout_ms = + cpu_to_be32(O2HB_MAX_WRITE_TIMEOUT_MS); + hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); + hand->o2net_keepalive_delay_ms = cpu_to_be32(o2net_keepalive_delay()); + hand->o2net_reconnect_delay_ms = cpu_to_be32(o2net_reconnect_delay()); + return hand; } /* ------------------------------------------------------------ */ @@ -1456,16 +1474,22 @@ static void o2net_initialize_handshake(void) * rx path will see the response and mark the sc valid */ static void o2net_sc_connect_completed(struct work_struct *work) { + struct o2net_handshake *hand; struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_connect_work); + hand = o2net_initialize_handshake(); + if (!hand) + goto out; + mlog(ML_MSG, "sc sending handshake with ver %llu id %llx\n", (unsigned long long)O2NET_PROTOCOL_VERSION, - (unsigned long long)be64_to_cpu(o2net_hand->connector_id)); + (unsigned long long)be64_to_cpu(hand->connector_id)); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); +out: sc_put(sc); } @@ -1475,8 +1499,15 @@ static void o2net_sc_send_keep_req(struct work_struct *work) struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_keepalive_work.work); + struct o2net_msg *keep_req; - o2net_sendpage(sc, o2net_keep_req, sizeof(*o2net_keep_req)); + keep_req = alloc_skb_frag(sizeof(*keep_req), GFP_KERNEL); + if (keep_req) { + memset(keep_req, 0, sizeof(*keep_req)); + keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); + o2net_sendpage(sc, keep_req, sizeof(*keep_req)); + folio_put(virt_to_folio(keep_req)); + } sc_put(sc); } @@ -1780,6 +1811,7 @@ static int o2net_accept_one(struct socket *sock, int *more) struct socket *new_sock = NULL; struct o2nm_node *node = NULL; struct o2nm_node *local_node = NULL; + struct o2net_handshake *hand; struct o2net_sock_container *sc = NULL; struct o2net_node *nn; unsigned int nofs_flag; @@ -1882,8 +1914,11 @@ static int o2net_accept_one(struct socket *sock, int *more) o2net_register_callbacks(sc->sc_sock->sk, sc); o2net_sc_queue_work(sc, &sc->sc_rx_work); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + hand = o2net_initialize_handshake(); + if (hand) { + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); + } out: if (new_sock) @@ -2090,21 +2125,8 @@ int o2net_init(void) unsigned long i; o2quo_init(); - o2net_debugfs_init(); - o2net_hand = kzalloc(sizeof(struct o2net_handshake), GFP_KERNEL); - o2net_keep_req = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - o2net_keep_resp = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - if (!o2net_hand || !o2net_keep_req || !o2net_keep_resp) - goto out; - - o2net_hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); - o2net_hand->connector_id = cpu_to_be64(1); - - o2net_keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); - o2net_keep_resp->magic = cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); - for (i = 0; i < ARRAY_SIZE(o2net_nodes); i++) { struct o2net_node *nn = o2net_nn_from_num(i); @@ -2122,21 +2144,10 @@ int o2net_init(void) } return 0; - -out: - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); - o2net_debugfs_exit(); - o2quo_exit(); - return -ENOMEM; } void o2net_exit(void) { o2quo_exit(); - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); o2net_debugfs_exit(); } From patchwork Sat Jun 17 12:11:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92F19EB64D9 for ; Sat, 17 Jun 2023 12:12:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E6256B0087; Sat, 17 Jun 2023 08:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 294BE6B0088; Sat, 17 Jun 2023 08:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15C6F6B0089; Sat, 17 Jun 2023 08:12:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 04E836B0087 for ; Sat, 17 Jun 2023 08:12:57 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D4E1F1C9101 for ; Sat, 17 Jun 2023 12:12:56 +0000 (UTC) X-FDA: 80912128752.28.9F72703 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 0640A4000F for ; Sat, 17 Jun 2023 12:12:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NRU2NMM4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003975; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2HlaXseyq5U5o1i8LIFlWuZ5rATyuHBFais1x2nt408=; b=5sdA8FUsM4yyBjlr2rXuw8PYswZSdfFrxQU9lpQgosxHwyR+ME5xBOXziHXx09PzYLA9Gw 4VZjaQNKXuEstDkNXHQv0Mcwh59h1DTaBMhC4yN7wxZKfnGqqGW0Kefmw25hKkQWPPiDnZ lcFov5mFKy2qGdj91CH7oqanRIiDBFU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NRU2NMM4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003975; a=rsa-sha256; cv=none; b=x+bKkoWN+8UzzeWl514kaC8yYPL5neD9eJbvRGKksvJd45yJyIqXp4H+adztUJ7Kh8OBa8 RzeOKHbhOpPz+TNIAFyd2+tsdtAJUDO8W4tvfhhr8pKzupzDtVizKXx5UZtX8uH4J1a6sW rAwwVpA66qMaWakC78IHo3dFpo2Zhv0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003974; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2HlaXseyq5U5o1i8LIFlWuZ5rATyuHBFais1x2nt408=; b=NRU2NMM4IIAb7e40wuFtw7w2v/4OfytCJht/vwUcJSf1OP/qkAYgo/8HF31l8/9J0O6/bH 2tuj6ltMncxDcgz8wFjV0gzNvcPVwgSf84q9xYEtNT9wofy67rpzypMlCtKg/fvUwuJT/x 9IHYjxAyBstcVhHsUM6rUn9EFK+r3Zw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-NmwNDhxxO3e69TbO3-hfEA-1; Sat, 17 Jun 2023 08:12:49 -0400 X-MC-Unique: NmwNDhxxO3e69TbO3-hfEA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 35E311C08782; Sat, 17 Jun 2023 12:12:48 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30CB0492C1B; Sat, 17 Jun 2023 12:12:46 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [PATCH net-next v2 13/17] drbd: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Sat, 17 Jun 2023 13:11:42 +0100 Message-ID: <20230617121146.716077-14-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Queue-Id: 0640A4000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: tcz1tnkk71yq56puegojc66unqjdjfma X-HE-Tag: 1687003974-812796 X-HE-Meta: U2FsdGVkX1+/uFL4scynmUUiAPFjh5fhdBZBimhnUf9IoyyqAzJ426W6T8dbPtEwY5/n8tfZeBjMnr6ZBJSIAn0fIo0oKR40dLxQqb7QqpqQGs/2ty2/oeWletkVKsOXsR2PsGlHGS5pjfKkInlt4A53nld55zCess2+f3yXZATRjKmCzoZqs7UdAcY2qYblHFtcLacOWqgeJO2ZUc5c09anUODTyF2vfwjOY1Jdj8JKJ+CbOQC+HKm720InY0Fq0hGCmEnRN4Od+aCuZDXzLK7NFFcG2vlgiCDaf0PWOVdv5UstwJSpo0Iov2AWFGfBWXw38FgwPpbhgG+coXUF/d9up2pAFg6yUkfCp52oZ5kmMs3FmkScKY1uZBcS9NNToa0UhZIyfolNVwlNRoOqa7oS3K4dj6c9IcTlKYyJDfoc11YHn6xa9aPjjt1HWCgINCXkUJOF8cgT2bVfEyYalTadJAi1C21wcSx85uT4m/TVR7nGl+p1YOvd4JEfpkhtRaoh5tzhyuyu2cwZElWW0Qt9kXens4C4SGwWpwmtA2PTM9OxZF50Z/eYl9Bl2WJxYFlPtLvN3aMLQGwio5RDEtLnkWJ4+6m1j66auZj4Lt8bf1XanORaRBXLu5QPYQ/wAJae8aa1Jm3FW1nJpqq30TH14Nr1ybuARHc3i9gIUUPhT98KtA3yUqnjm0J8IoC3l715GRBnE/Cj+2Gzkyb9HPSvdjLomu2LtcB4Gzl/cRj8Ol9mnEl3aMtkFS+crG/YRi8D1vLWe/4z9lx72lH4UQseYNaAPm3yW4GyPA/OtUNFhUREmSRUc/umDxQTfC2s5LZdYQlN0MVSYE+ImtI2Y7oy212achzuGxOJrgfn+U84FLfF92hqW3w2QNtzRXA7JrVOQ7BNUTla3TIFMafBCygeaTql9E2RyZ5mrHNEI9C9mJgAI8w4Lwwvip0GhMs3SQRdlY+tVf2HKyWh7l6 yZYehkKo B1BbsLNL1N2xR7C9GtVArr3uN9FvTuulQ2rRnKgB9PbIvq/Wc2oqyUP9qyV0YNyfJx2CimFd1PaHa8GxDCVwdtr44B9VmE8nYsTlhPAf4UJDzeedl+7j4lpBcvi9nVIT7fSTVn0kIpz/QEOfmnpyOYa7DIX+lRhGlYXdTgpBB2mgU4STYbjs42CbU50n7zmuKguf57Lo6RDnotGgRjnbkSAsSUxiOA58NxDNmaTu+CL7uEzy2PDCwSHwRGQ5+Kh+QtsHzDJeoFdw2Uw5+AaMFdj+2CZ1NI1PP6Ifa0SNlgxexkXDdEz+mKKLVVDZhqCztTHNaawtxjmnOxtQj2cNunTtqgPzFAXiRJp6mU8MIoRcth2MHeeUIE7GffzmvC7qzdaOu/7NNvt/RFLQr2pSKUHhLWEAN5eB+hBCO6qV1Zdjl35ypRtCOgKFWmTZo3/AFzw0hSYs1wzLTX9CDQBAcQfVL6pgSvaWWPwQBCV5jcOnqEQDycV/is43vaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() conditionally with MSG_SPLICE_PAGES in _drbd_send_page() rather than calling sendpage() or _drbd_no_send_page(). Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #2) - Wrap lines at 80. drivers/block/drbd/drbd_main.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 83987e7a5ef2..8a01a18a2550 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1540,7 +1540,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa int offset, size_t size, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; - int len = size; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = msg_flags, }; int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1549,33 +1550,35 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (drbd_disable_sendpage || !sendpage_ok(page)) - return _drbd_no_send_page(peer_device, page, offset, size, msg_flags); + if (!drbd_disable_sendpage && sendpage_ok(page)) + msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; + + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - msg_flags |= MSG_NOSIGNAL; drbd_update_congested(peer_device->connection); do { int sent; - sent = socket->ops->sendpage(socket, page, offset, len, msg_flags); + sent = sock_sendmsg(socket, &msg); if (sent <= 0) { if (sent == -EAGAIN) { if (we_should_drop_the_connection(peer_device->connection, socket)) break; continue; } - drbd_warn(peer_device->device, "%s: size=%d len=%d sent=%d\n", - __func__, (int)size, len, sent); + drbd_warn(peer_device->device, "%s: size=%d len=%zu sent=%d\n", + __func__, (int)size, msg_data_left(&msg), + sent); if (sent < 0) err = sent; break; } - len -= sent; - offset += sent; - } while (len > 0 /* THINK && device->cstate >= C_CONNECTED*/); + } while (msg_data_left(&msg) + /* THINK && device->cstate >= C_CONNECTED*/); clear_bit(NET_CONGESTED, &peer_device->connection->flags); - if (len == 0) { + if (!msg_data_left(&msg)) { err = 0; peer_device->device->send_cnt += size >> 9; } From patchwork Sat Jun 17 12:11:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A7BEEB64DB for ; Sat, 17 Jun 2023 12:13:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBE0C6B0088; Sat, 17 Jun 2023 08:12:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6E3B6B0089; Sat, 17 Jun 2023 08:12:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A35E18E0003; Sat, 17 Jun 2023 08:12:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 942496B0088 for ; Sat, 17 Jun 2023 08:12:59 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6B7601A014C for ; Sat, 17 Jun 2023 12:12:59 +0000 (UTC) X-FDA: 80912128878.09.63866F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.145.221.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 9BF0DA001C for ; Sat, 17 Jun 2023 12:12:57 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qr90W41S; spf=fail (imf15.hostedemail.com: domain of dhowells@redhat.com does not designate 216.145.221.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XKse4vF5Frqs4bdWsYMKttBmqWRsH1PFiAHwvjH79qA=; b=WlPzm4yejbFnLPpWiLHAaUJHolXdT0Zcq/8AseR6zKO4n1EPHDuhTerfHSQCSIMdI31iTz jfU2h/EzBFLvC0TNj4UEoHBa/JwPLHM1CEFjQplGUryPEx+6IkU5DO6BP7qV2djw7bApEp WiVxapVcrfh2AJSZXzLHEBT5Bo19krM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003977; a=rsa-sha256; cv=none; b=z9u/Gckd14d5v32NMArmpx72kVaL818XNCSdXwxu5ibNAKcD3WILRBUAdLMrhMNUgPLtJj aa5EcMxC1ygzIHBriGoXu0fcgEJ7Ko3mDFCNqdRanoNrHHF5fkRyVFZD8qsRXLuE02eG+B RUthEQh8/OiaKV1z1pa6OMuBu5ziNAc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qr90W41S; spf=fail (imf15.hostedemail.com: domain of dhowells@redhat.com does not designate 216.145.221.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003976; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XKse4vF5Frqs4bdWsYMKttBmqWRsH1PFiAHwvjH79qA=; b=Qr90W41SWkWowRpBsku2C5edWyMCV6sjaK8fVDdpAP68+qClpZT35HEZOPBpbLnDT8KnnX 95PJWY1huLSUdRAAF2E1+oiH14gZCkgmZJdVFbh6NjoW+m5PcbYtV2cJoo+N5ybf8DFmWq UpZiD0NWoApirucgobby1/YaGqEsxr8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-2-7XIMq-mIPQOIc8pmfQb5Cg-1; Sat, 17 Jun 2023 08:12:51 -0400 X-MC-Unique: 7XIMq-mIPQOIc8pmfQb5Cg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EC560101A56C; Sat, 17 Jun 2023 12:12:50 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id E891B48FB01; Sat, 17 Jun 2023 12:12:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [PATCH net-next v2 14/17] drdb: Send an entire bio in a single sendmsg Date: Sat, 17 Jun 2023 13:11:43 +0100 Message-ID: <20230617121146.716077-15-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Stat-Signature: zxoo1u1f6ze97yribpb8smruzhiiqree X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9BF0DA001C X-Rspam-User: X-HE-Tag: 1687003977-861786 X-HE-Meta: U2FsdGVkX1/6Z3k1xyYRS/MrtlhoWaC6Ak+wnVI5wWDroEjkKQKK6qHD7w0FZ3+PKnAiIsholr5Eh4jZDvOAwoa24Cp83XfXf6CKQMeiVpgI+J3Q5nTeRVxUBI4ymKeBrBBw7cuYeyuQbd4UAop+ZcBSXJrDSvqticwbhiWNuasi0tKHpVTA4OdgnbwW86drjARi4faSY3HOm4j0TxHiQfi5ZkRFSnupWzAOocG7OptDi66b+sFqp4zzJO/9zmPnSopbB1CpMkgrfvb5arImcR29NjXezJygTNX9WPY2WLVAHjQ+JcLQ8cUrPUoBD+WcnWKW/zPqkTgUdmKhQBoo7dYNrPgBih705w+XiTVPl7Mp0YoLtJYesvKkHi3ccGokijsoc6foz9eOsB3+N71Iqzuqk80n+1c0TuoFN9kPiraDk1MeB1fPOukEhwpJw2ZSQdzGyouik/qJ3CwfTXwizQWNK+0lTWAWNut1+noTlt3SOuowMW4vEywV5SmxjQz8OdtpPPDEejG5B3YIU+nujonN/m1WXHb/osANTapVnMk9rG1fW7poFrwAfhd08x0RRaHLp7V4ya7cve+gNYUm15yOhBLimmVOvVf2QTh8Yg7CQ6eTpxUHAzljOmuUlRA/A8pG8LNQYCbzGLjIoLZ2NH3KdWNlMo5ar0oNCGOn/zWdhTpXDySO/KliVFc5nkTV0VOIW6LJ/p1uiEYl6LPEjpUlQFSUcqRsv8AdIRuZMm2FVxi36BKSzbRj2Sd4Ci3Mtj4PUEnWYqLGwF5O8VFFvTWidGprtUy+vSZ0ENK+JacWqAT22+JdErDknaSFM/NAP4HB+5UPKuPRd/RzAa6EY8LFVGoATUYlrgZPFZIPgPQE7ursPvJtNv6cwT+V4oLZxV1Vcy/bLo7Bn/uDIxECUQsklFVftHwyCIl48YP+txKF+hOCW8GV/Qh4689mEuM8JT2XJXiO/aeGGD6Z5nO 135Mbx5S 9lRC+eyu/hd5RGFDlVAyzSdmuUnqE/To9ET7EVo3/4W0r0heW6zX1WZKhHLg3nB5qpdd9whrnIOezgSVlkCQS2QZ8aUFF+gWL37ed2aJK0lO/Yadh6TELzDwkga3SRDq8evMmSB6oN2QanGAxMwTwxDVpqxDLl5SwvCIZXgjQHBvnwDR+xW50F3vvrtHXBGrVDJylTm3eF/Yu2+ZxdyV1tvQ5AOScqQCZ+wQ6qvQcV46tzsZO0r7U/tlary7v+FVeCaaw/F8XZ1JEzKqGKsxS38LQCziB9EW3Z8o5EYRH4LigHBR1kGm9PBDRILer4Egxzk2/wqCL0Z8kjWAwwwIaayCqHjYWDgclxZyifJ6aga7tSjs+dpSgJoRsBG47ntPbPqyVc6q7OWiuzoZuE4N27PSNG939TRt+PMOJggLGhkajZbwKqUiZlWRGtcigaoV+30yGzB95widEbvK11QUbFaIXf3tO3+xwW81zgJCUs6x+vvEaWQVWm1UWZuGI/8R/5whAMQZ9xWnNeCN4Dolasva/UNWd6YbTsCCbTvDK9CUVszQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since _drdb_sendpage() is now using sendmsg to send the pages rather sendpage, pass the entire bio in one go using a bvec iterator instead of doing it piecemeal. Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #2) - Use "unsigned int" rather than "unsigned". drivers/block/drbd/drbd_main.c | 77 +++++++++++----------------------- 1 file changed, 25 insertions(+), 52 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 8a01a18a2550..beba74ae093b 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1520,28 +1520,15 @@ static void drbd_update_congested(struct drbd_connection *connection) * As a workaround, we disable sendpage on pages * with page_count == 0 or PageSlab. */ -static int _drbd_no_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) -{ - struct socket *socket; - void *addr; - int err; - - socket = peer_device->connection->data.socket; - addr = kmap(page) + offset; - err = drbd_send_all(peer_device->connection, socket, addr, size, msg_flags); - kunmap(page); - if (!err) - peer_device->device->send_cnt += size >> 9; - return err; -} - -static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) +static int _drbd_send_pages(struct drbd_peer_device *peer_device, + struct iov_iter *iter, unsigned int msg_flags) { struct socket *socket = peer_device->connection->data.socket; - struct bio_vec bvec; - struct msghdr msg = { .msg_flags = msg_flags, }; + struct msghdr msg = { + .msg_flags = msg_flags | MSG_NOSIGNAL, + .msg_iter = *iter, + }; + size_t size = iov_iter_count(iter); int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1550,11 +1537,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (!drbd_disable_sendpage && sendpage_ok(page)) - msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; - - bvec_set_page(&bvec, page, offset, size); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + if (drbd_disable_sendpage) + msg.msg_flags &= ~(MSG_NOSIGNAL | MSG_SPLICE_PAGES); drbd_update_congested(peer_device->connection); do { @@ -1587,39 +1571,22 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa static int _drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_no_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) - ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, 0); } static int _drbd_send_zc_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, MSG_SPLICE_PAGES); } static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, @@ -1631,10 +1598,16 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, /* hint all but last page with MSG_MORE */ page_chain_for_each(page) { + struct iov_iter iter; + struct bio_vec bvec; unsigned l = min_t(unsigned, len, PAGE_SIZE); - err = _drbd_send_page(peer_device, page, 0, l, - page_chain_next(page) ? MSG_MORE : 0); + bvec_set_page(&bvec, page, 0, l); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, l); + + err = _drbd_send_pages(peer_device, &iter, + MSG_SPLICE_PAGES | + (page_chain_next(page) ? MSG_MORE : 0)); if (err) return err; len -= l; From patchwork Sat Jun 17 12:11:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8A9BEB64D9 for ; Sat, 17 Jun 2023 12:13:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F15A6B0089; Sat, 17 Jun 2023 08:13:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A21B8E0003; Sat, 17 Jun 2023 08:13:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452496B008C; Sat, 17 Jun 2023 08:13:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 331886B0089 for ; Sat, 17 Jun 2023 08:13:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0E427140146 for ; Sat, 17 Jun 2023 12:13:01 +0000 (UTC) X-FDA: 80912128962.23.18F7BA7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 5885CC0002 for ; Sat, 17 Jun 2023 12:12:59 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZH3uQ90Q; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003979; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bszGYXAPCfyFrCrKXBgf4DyEzlv1vS1+csO6wWtbX4c=; b=OHKQtK2bBtYGXNNMUQEosjMY8OAQ8RY4spygVM+GbCFzePp6o/+9NWqr+50STvQ+aH8LJW zzqkmL3jVJwVGbXGpNNHAjun0890Iil6N2lsP/B/vJLfdAnm5g09gPYnmEe42EuJcmXWP5 wzhTrJg0DKMLQEorpWQ0FrUpikmw0TA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZH3uQ90Q; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003979; a=rsa-sha256; cv=none; b=EdvUtGctreKl4xvSZou0F/K7gbChNT7KsdvIESzuVxqyJauhqzY3ezS4LLhFDxx12hkfdr 0jQZIXMK6JgTv4tY/23mbuw2Bk4SiG3D2SE+DT6xUsZMRRK0JOjrOsqfCXREM28UYEhTYq QIsxpZrBAFaATx/Ln03AoqJ4oa4O7RU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bszGYXAPCfyFrCrKXBgf4DyEzlv1vS1+csO6wWtbX4c=; b=ZH3uQ90QaWYZ91CERbG+c63lDgnvZVv5e3+xwOWM3lR06+s5upuNOWRO+fpv9AndENqu93 fL/YV2dhBpzftUiBB4NDJh8OLH3ValjIivB4g1YQygx8PXktcCWRt/l0EPCQl8ktgcWLoj TYIF1qqXbbGfQQqIsZ+kwE1DeEmnr/M= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-436-185vvMv5PdOOlfFYb8ORMQ-1; Sat, 17 Jun 2023 08:12:55 -0400 X-MC-Unique: 185vvMv5PdOOlfFYb8ORMQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 192423C11C6A; Sat, 17 Jun 2023 12:12:54 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 903F92026D49; Sat, 17 Jun 2023 12:12:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lee Duncan , Chris Leech , Mike Christie , Maurizio Lombardi , "James E.J. Bottomley" , "Martin K. Petersen" , Al Viro , open-iscsi@googlegroups.com, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [PATCH net-next v2 15/17] iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Sat, 17 Jun 2023 13:11:44 +0100 Message-ID: <20230617121146.716077-16-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5885CC0002 X-Stat-Signature: 6ooqiioxrq1o8yrm59fqce5syhiesbyn X-Rspam-User: X-HE-Tag: 1687003979-797780 X-HE-Meta: U2FsdGVkX1/6Gol7QEvzbQ44h6S0Cqh4uFx0TZo1Q4uGDgUUElTxYpaSYoTPC23PBY8ZbeDXUpUvQJp/kFE22QD7wFtW5GoLBhHeT0crfQlj9h1t8boOqg9uY9x1aEMz3mUNeB/WJ+u0Pc1N0+WG+XX0LR7Z4eKLDck6U2nSoGOTFBOjwrtTnHpqiyOd6p0uVYcjikPaxV2tniTJi67QdoLmfrV7mfbLYtRc+xGovx7RaXd9ggZUzs6I11GSRZic/jVsI446ClB5cIkxsWFhABRliRtBgO5QMlWbYCPFnGC1r+dBwNobghheAViNHYegUx9RsSgL4bPwNRDJ/YwjWSmbZwczNcIr3+te1jjNSgUI9hGzsBjMZQBqaFNLYv0GS5Fq/porTGECuN9oN52WNyb7j40kVeBGpACkzDyY/09rPVldXA1aYLIWl9NfXXf6zaOtCd5qG7LZlZkDvAA4NZZG4XssyDyb1Fa6FFcjVIYhJRHZDexEfBJUBSZbZyK9sjmUlJ0WtQeaNUogqBPFfdzVc5iMB0gqzj1Bo1z9uml+IXydyhR3Rp3W27XxrhvFO0B2x8iTaI5esu5NiQJ+4gfpXH5AArDoAopNkDuwTo5bdz0Zjw18JEdmsbJTln97Vgk5s37EK02CirY0NsRMHQndrMwiwrXoz2Y4zQnYJjFiILTHqZUSA9TWCiO5rOM92S8KVfFoVo5zeY1xcH+xFGMSomRWCMvvBAR53hlOi8UGjIM5V4JL3+adaB5EEZnmNaGNROtSa7P7tTHTYYegXqVD/WO7WvDlQO3zU9QLMb31jto3Kb+E+BzZNKTr7j4ixVaNv/iWGSgq8lYcyoZNLFQh7ngbGG5mmcT5MGJHyvFPbeVF9YXtztyOEXqPxSQQs2jKuGmcL13S4jnGcDpC5fjaeKx2MQIX9dnreJCqWsvvhIIBL9xaK2xs0WEIdKLwf0ej+bX8uc4JKaq1Kgs osbhDeHY 0ZxvLj1obzMagwqPPgkdvYsdBiJ8Lyc/bx11H5j7usk00eLdW8m291drXlvaNXbGWW4kdBgH8o7y3m7S22TpDS76n9s/mZnpK6ELgn4y4G4vqVUGRnFbwHAtbmX+j6Vjzf2w9Gxjgi+S+lmgiSjRFnqEKnouoT32J5nMkrFJ8z0D8cx6Doil6SIRFyDnc6fWfVrntGm+nM4UAbUrTW2HUBw/+fGwA+faHk1ZGxu7CP45ZaHB+n3PmlXbtp6JHhVOZKu/5nlTRV9Wu2OFQjBWeN3Bpo0BMUyCDD60DckxHzFtdjLAnoDwGgYWy89wUwGcPbLjUZrQRvK4K2CvCyh0dAp46nyBty4i3xaz4ZNvQ2VJgYmQtQXddF4DtKr2EdOnlFjxhjiDugjhXgCxyqq3Y99qeWFC6mslY755hd6ipA8eFPZ1wQWo6HQewlZ3Ly05u9pnO969RAgpeJmMdbqXOOc9tLWNcIGX7J8iva0I/DFtqIufPpHsFHfOLaqh0Ovv8NIymFPIwWITjcXPJeVWK3rBOlbCNqmgzfYfIfG3RxMuOGmDLArg9UcuroYfOtNaisY7e6nfKTIiQCDcaReSIZhchN3T4SHqgj/zom3Hti90432P4+5bMPT5shX7WADt7WpQnM7X9MiuAPSPCOgsU506sx+UrvX4/vN/KX1gjqlKykF3fggN0+VhmPUfYd2LvnWYIKaLiRspQEqHRxxxoZDjbmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage. This allows multiple pages and multipage folios to be passed through. TODO: iscsit_fe_sendpage_sg() should perhaps set up a bio_vec array for the entire set of pages it's going to transfer plus two for the header and trailer and page fragments to hold the header and trailer - and then call sendmsg once for the entire message. Signed-off-by: David Howells cc: Lee Duncan cc: Chris Leech cc: Mike Christie cc: Maurizio Lombardi cc: "James E.J. Bottomley" cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: Al Viro cc: open-iscsi@googlegroups.com cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #2) - Wrap lines at 80. drivers/scsi/iscsi_tcp.c | 26 +++++++++--------------- drivers/scsi/iscsi_tcp.h | 2 +- drivers/target/iscsi/iscsi_target_util.c | 15 ++++++++------ 3 files changed, 20 insertions(+), 23 deletions(-) diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index 9637d4bc2bc9..9ab8555180a3 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -301,35 +301,32 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, while (!iscsi_tcp_segment_done(tcp_conn, segment, 0, r)) { struct scatterlist *sg; + struct msghdr msg = {}; + struct bio_vec bv; unsigned int offset, copy; - int flags = 0; r = 0; offset = segment->copied; copy = segment->size - offset; if (segment->total_copied + segment->size < segment->total_size) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; if (tcp_sw_conn->queue_recv) - flags |= MSG_DONTWAIT; + msg.msg_flags |= MSG_DONTWAIT; - /* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { + if (!tcp_conn->iscsi_conn->datadgst_en) + msg.msg_flags |= MSG_SPLICE_PAGES; sg = segment->sg; offset += segment->sg_offset + sg->offset; - r = tcp_sw_conn->sendpage(sk, sg_page(sg), offset, - copy, flags); + bvec_set_page(&bv, sg_page(sg), copy, offset); } else { - struct msghdr msg = { .msg_flags = flags }; - struct kvec iov = { - .iov_base = segment->data + offset, - .iov_len = copy - }; - - r = kernel_sendmsg(sk, &msg, &iov, 1, copy); + bvec_set_virt(&bv, segment->data + offset, copy); } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, copy); + r = sock_sendmsg(sk, &msg); if (r < 0) { iscsi_tcp_segment_unmap(segment); return r; @@ -746,7 +743,6 @@ iscsi_sw_tcp_conn_bind(struct iscsi_cls_session *cls_session, sock_no_linger(sk); iscsi_sw_tcp_conn_set_callbacks(conn); - tcp_sw_conn->sendpage = tcp_sw_conn->sock->ops->sendpage; /* * set receive state machine into initial state */ @@ -777,8 +773,6 @@ static int iscsi_sw_tcp_conn_set_param(struct iscsi_cls_conn *cls_conn, return -ENOTCONN; } iscsi_set_param(cls_conn, param, buf, buflen); - tcp_sw_conn->sendpage = conn->datadgst_en ? - sock_no_sendpage : tcp_sw_conn->sock->ops->sendpage; mutex_unlock(&tcp_sw_conn->sock_lock); break; case ISCSI_PARAM_MAX_R2T: diff --git a/drivers/scsi/iscsi_tcp.h b/drivers/scsi/iscsi_tcp.h index 68e14a344904..d6ec08d7eb63 100644 --- a/drivers/scsi/iscsi_tcp.h +++ b/drivers/scsi/iscsi_tcp.h @@ -48,7 +48,7 @@ struct iscsi_sw_tcp_conn { uint32_t sendpage_failures_cnt; uint32_t discontiguous_hdr_cnt; - ssize_t (*sendpage)(struct socket *, struct page *, int, size_t, int); + bool can_splice_to_tcp; }; struct iscsi_sw_tcp_host { diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c index b14835fcb033..6231fa4ef5c6 100644 --- a/drivers/target/iscsi/iscsi_target_util.c +++ b/drivers/target/iscsi/iscsi_target_util.c @@ -1129,6 +1129,8 @@ int iscsit_fe_sendpage_sg( struct iscsit_conn *conn) { struct scatterlist *sg = cmd->first_data_sg; + struct bio_vec bvec; + struct msghdr msghdr = { .msg_flags = MSG_SPLICE_PAGES, }; struct kvec iov; u32 tx_hdr_size, data_len; u32 offset = cmd->first_data_sg_off; @@ -1172,17 +1174,18 @@ int iscsit_fe_sendpage_sg( u32 space = (sg->length - offset); u32 sub_len = min_t(u32, data_len, space); send_pg: - tx_sent = conn->sock->ops->sendpage(conn->sock, - sg_page(sg), sg->offset + offset, sub_len, 0); + bvec_set_page(&bvec, sg_page(sg), sub_len, sg->offset + offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, sub_len); + + tx_sent = conn->sock->ops->sendmsg(conn->sock, &msghdr, + sub_len); if (tx_sent != sub_len) { if (tx_sent == -EAGAIN) { - pr_err("tcp_sendpage() returned" - " -EAGAIN\n"); + pr_err("sendmsg/splice returned -EAGAIN\n"); goto send_pg; } - pr_err("tcp_sendpage() failure: %d\n", - tx_sent); + pr_err("sendmsg/splice failure: %d\n", tx_sent); return -1; } From patchwork Sat Jun 17 12:11:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13283556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 374E5EB64DD for ; Sat, 17 Jun 2023 12:13:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA7596B0092; Sat, 17 Jun 2023 08:13:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C58FB8E0005; Sat, 17 Jun 2023 08:13:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B47288E0003; Sat, 17 Jun 2023 08:13:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A33C16B0092 for ; Sat, 17 Jun 2023 08:13:13 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7C2191601D0 for ; Sat, 17 Jun 2023 12:13:13 +0000 (UTC) X-FDA: 80912129466.18.618F445 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id CA115100015 for ; Sat, 17 Jun 2023 12:13:10 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K8nDBzg2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687003990; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MKaOCCzXJedIVdLsu0rWphyHTQRA+RfNYbfx/yO3kzQ=; b=0IuzMmZg4pOXEfA86rvVR/nPWofRipw72J0bCzkRkGZ1UyeZDgCKdR9CGjMILXhVpc1nuQ +UIbGGwmLUNCGWhu2YsDbbFEJnnU6Qh3+hqW2BLEQAIdOsFEAv2cEpJOdWi/jS9qkPD5LF Icki8pRiwgT4PVnJd8XhAr4DqfmH6hw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K8nDBzg2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687003990; a=rsa-sha256; cv=none; b=onlitGAy+qxFR4Aba4AX1w6nUSUaD8IQDk5x3pbwfqs55w21UQOU6Yy+w0rPAsEod11sd5 xy7DRi7567hkp3AwQM6ono+rYWbLgElzsMWoMGncyXp85IFdnIzaXvp5uGRC1Hdy7qHPM6 bmdgJ0Lop796WwxM0Fd78HfOVjvWTqE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687003990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MKaOCCzXJedIVdLsu0rWphyHTQRA+RfNYbfx/yO3kzQ=; b=K8nDBzg20KRBLH5UC6LSP2LZf5p/X2GPAyQXq/knxVSAVzRZqyHr9/q4O14X7KnHaOIdos BMQ57ChdUQwOqhTGSIzhITbw708eUxl1LpEvxs939ygGfgxtAYubDRHqRcF7l5/kbcG6S6 sYySFbUC2QZ8Svkq0l6r9lm2sK4bnaA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-557-0frgcycTO9ec3FXSZq0ViQ-1; Sat, 17 Jun 2023 08:13:03 -0400 X-MC-Unique: 0frgcycTO9ec3FXSZq0ViQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 687B780006E; Sat, 17 Jun 2023 12:13:02 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.51]) by smtp.corp.redhat.com (Postfix) with ESMTP id 422AD1415102; Sat, 17 Jun 2023 12:12:59 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, dccp@vger.kernel.org, linux-afs@lists.infradead.org, linux-arm-msm@vger.kernel.org, linux-can@vger.kernel.org, linux-crypto@vger.kernel.org, linux-doc@vger.kernel.org, linux-hams@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-rdma@vger.kernel.org, linux-sctp@vger.kernel.org, linux-wpan@vger.kernel.org, linux-x25@vger.kernel.org, mptcp@lists.linux.dev, rds-devel@oss.oracle.com, tipc-discussion@lists.sourceforge.net, virtualization@lists.linux-foundation.org Subject: [PATCH net-next v2 17/17] net: Kill MSG_SENDPAGE_NOTLAST Date: Sat, 17 Jun 2023 13:11:46 +0100 Message-ID: <20230617121146.716077-18-dhowells@redhat.com> In-Reply-To: <20230617121146.716077-1-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Stat-Signature: fbmekseo9yb5ixsshubpxipmhkep7w9t X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CA115100015 X-HE-Tag: 1687003990-881885 X-HE-Meta: U2FsdGVkX18enjhB4ykcP7wdmndu4DFE5AznL19Og/j/iiZRePhAGAsmbd2KMxpbKJ5ojItua0+68cbREecEWrCwV2D/zko50SCHOkE50gY0zZWThf1tmTj3pwf4GDe/f3G4mjUuMn+SiwdIEyHT3xju0opNq5hG3Jr272rX7Y1tJdAKTRh+xlz6sSP6NjgE5KcnK0/mzqGBk+o7rlxdalpT7xDn+k3NKR6Hy8RNRb+bsT4gn9eERyAk1YydSCCjlF8/96U3ZgbWL6bM5e1iKsUN9xMPCJni2uzQRlOMyCcAWFZNncAmZRGy3XUs8RevFDSMsNbTqhh7czGO4McplcJ3gSZmalNxH3imoJ7sYUgUHIcFg0MzuogO/BXHAJ4aufT+KaCa0xrqsrq2FytMg94K+LT3BfP3GUq+qTNIpnBd8u3Ww9u9yAKq+aS8L8VaVSylD4xSxLnJFIIUjK7Wm22pydOJtBi/eKbP/xHBOAAG+xNtHASY2SAtNvPmxHQP2Rs8/T8nBYVCJHO5sMCohTyY6L7YEKVApETJqJNqqqiOkUpojh/WJPG7AjWR/AKPvyjcE3wnBf5msnGzq7uybnKhNjwMatGrPTv+/QuY2TKNOi25n7roslAyXM8Z7i5D0UxXHp+gGIIBU+Iv/DK5af5Ep3eumIoYBPnGL/OUOjf13c4l3IQm7kBF59WKLoQOhAMH7uFsapH4T4I5Ao+W1x9vNNFWLRw58B3Qd+0N579XEgC8zdq5ah2QHC8jc97cUKYAkJvjVKgJdu5zNa4Iax9MGMA4UGsWlCsat2kplzCyHzovX0Stlj/wilnmLCoKMoCrABGumUj1CSBUiDqM3u3qdrt8twXGrONdPnMxzvIYlRzM4FTVMEhSR+4kXoMhV0uVxSatHmBwiSAhdDb0e4XCWH/Hfnwxb+g9IFGgCnxUk9LCZY82ZaWym0bAEkT/81e2oO3gCtB0LWhqRIp LX1Ei5nb /x1a2WunftsKAaYsbuEGQw6ywR6ZxUqJGGGcqCERoes2Xfu6YSG6f700jwxPIKTY6CUayLcHRtZTvqEdUDkQtmMOMRENql3TQNiUAUllpcRl3WNZRsbCCK8zluf0t1Au8WfSSG9TYkt2jDp+6XLbyttkrGrBJan576GtW0t/+00RYh2HOOKw7ZYcjjPoo612OBwfDG5Kr+WVC+vPXusA23L6YLG5Sv6OlCON9IGxtEYjSKHJmr+pmwCnsikEydsZ+Egz+sayGiyy1J9rCdvooe9g83dzQDyNTcXWG7NDCSl3k/K86g+dfpNpjPHaqucrfOQv+l+AopNfrVp1+579eNrdcsoiYZJSqQZAdVoe5PxuGsEIxzJZTDhvmMBWW74TP3WqMlHpXDarBZD4tmW2LM6A8EYmT/YHJac9oRjGqjcqwiPLXFFCrKNgPk6ytvzmUIg/WK6PaEu3ZAwdoyTc9p9jE2LZ/s0PwGXRC5p++DFoJfhsFZ67mRT6RkgQ3ucB7SwQ+SBjVBaQDdqsgXZWTM8+hlaabxBJnXqfQgD0jLRYPu7wJcfVLFRFceHwXXY0vQScKeRKxcgV+zX3McJyIQTQJaotEm6A6N8lV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that ->sendpage() has been removed, MSG_SENDPAGE_NOTLAST can be cleaned up. Things were converted to use MSG_MORE instead, but the protocol sendpage stubs still convert MSG_SENDPAGE_NOTLAST to MSG_MORE, which is now unnecessary. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: bpf@vger.kernel.org cc: dccp@vger.kernel.org cc: linux-afs@lists.infradead.org cc: linux-arm-msm@vger.kernel.org cc: linux-can@vger.kernel.org cc: linux-crypto@vger.kernel.org cc: linux-doc@vger.kernel.org cc: linux-hams@vger.kernel.org cc: linux-perf-users@vger.kernel.org cc: linux-rdma@vger.kernel.org cc: linux-sctp@vger.kernel.org cc: linux-wpan@vger.kernel.org cc: linux-x25@vger.kernel.org cc: mptcp@lists.linux.dev cc: netdev@vger.kernel.org cc: rds-devel@oss.oracle.com cc: tipc-discussion@lists.sourceforge.net cc: virtualization@lists.linux-foundation.org --- include/linux/socket.h | 4 +--- net/ipv4/tcp_bpf.c | 4 +++- net/tls/tls_device.c | 3 +-- net/tls/tls_main.c | 2 +- net/tls/tls_sw.c | 2 +- tools/perf/trace/beauty/include/linux/socket.h | 1 - tools/perf/trace/beauty/msg_flags.c | 3 --- 7 files changed, 7 insertions(+), 12 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index 58204700018a..39b74d83c7c4 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -319,7 +319,6 @@ struct ucred { #define MSG_MORE 0x8000 /* Sender will send more */ #define MSG_WAITFORONE 0x10000 /* recvmmsg(): block until 1+ packets avail */ #define MSG_SENDPAGE_NOPOLICY 0x10000 /* sendpage() internal : do no apply policy */ -#define MSG_SENDPAGE_NOTLAST 0x20000 /* sendpage() internal : not the last page */ #define MSG_BATCH 0x40000 /* sendmmsg(): more messages coming */ #define MSG_EOF MSG_FIN #define MSG_NO_SHARED_FRAGS 0x80000 /* sendpage() internal : page frags are not shared */ @@ -341,8 +340,7 @@ struct ucred { /* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ #define MSG_INTERNAL_SENDMSG_FLAGS \ - (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_NOTLAST | \ - MSG_SENDPAGE_DECRYPTED) + (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_DECRYPTED) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 870c1cde4010..8f535e436ea3 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -90,7 +90,9 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; - struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + struct msghdr msghdr = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE, + }; struct page *page; int size, ret = 0; u32 off; diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 840ee06f1708..2021fe557e50 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -441,8 +441,7 @@ static int tls_push_data(struct sock *sk, long timeo; if (flags & - ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | - MSG_SPLICE_PAGES)) + ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SPLICE_PAGES)) return -EOPNOTSUPP; if (unlikely(sk->sk_err)) diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index d5ed4d47b16e..b6896126bb92 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -127,7 +127,7 @@ int tls_push_sg(struct sock *sk, { struct bio_vec bvec; struct msghdr msg = { - .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + .msg_flags = MSG_SPLICE_PAGES | flags, }; int ret = 0; struct page *p; diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 9b3aa89a4292..53f944e6d8ef 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -1194,7 +1194,7 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | - MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; ret = mutex_lock_interruptible(&tls_ctx->tx_lock); diff --git a/tools/perf/trace/beauty/include/linux/socket.h b/tools/perf/trace/beauty/include/linux/socket.h index 13c3a237b9c9..3bef212a24d7 100644 --- a/tools/perf/trace/beauty/include/linux/socket.h +++ b/tools/perf/trace/beauty/include/linux/socket.h @@ -318,7 +318,6 @@ struct ucred { #define MSG_MORE 0x8000 /* Sender will send more */ #define MSG_WAITFORONE 0x10000 /* recvmmsg(): block until 1+ packets avail */ #define MSG_SENDPAGE_NOPOLICY 0x10000 /* sendpage() internal : do no apply policy */ -#define MSG_SENDPAGE_NOTLAST 0x20000 /* sendpage() internal : not the last page */ #define MSG_BATCH 0x40000 /* sendmmsg(): more messages coming */ #define MSG_EOF MSG_FIN #define MSG_NO_SHARED_FRAGS 0x80000 /* sendpage() internal : page frags are not shared */ diff --git a/tools/perf/trace/beauty/msg_flags.c b/tools/perf/trace/beauty/msg_flags.c index ea68db08b8e7..b5b580e5a77e 100644 --- a/tools/perf/trace/beauty/msg_flags.c +++ b/tools/perf/trace/beauty/msg_flags.c @@ -8,9 +8,6 @@ #ifndef MSG_WAITFORONE #define MSG_WAITFORONE 0x10000 #endif -#ifndef MSG_SENDPAGE_NOTLAST -#define MSG_SENDPAGE_NOTLAST 0x20000 -#endif #ifndef MSG_FASTOPEN #define MSG_FASTOPEN 0x20000000 #endif