From patchwork Wed May 24 15:32:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53883C77B7C for ; Wed, 24 May 2023 15:33:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9DF390000D; Wed, 24 May 2023 11:33:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4D48900007; Wed, 24 May 2023 11:33:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D152A90000D; Wed, 24 May 2023 11:33:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C178C900007 for ; Wed, 24 May 2023 11:33:21 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 89D0D40A28 for ; Wed, 24 May 2023 15:33:21 +0000 (UTC) X-FDA: 80825542602.19.EE909B4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 98D5F14001D for ; Wed, 24 May 2023 15:33:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aFfZhxi8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942400; a=rsa-sha256; cv=none; b=coGSiR7GF9GQVrQ+zMBNpzy/KnivoeA7OZpCQERl3mCVwpvNHCnnuIuOmws92lYZPMhIS7 HL8BWB6R31VfV9mjCPH53rhuaxOzB09z7A43z7DhNCItmlySje5fpa2qbvr7vlNfAGtUV5 cqVlPW4sD+7UqJNU7pvJciB00ZDmHxs= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aFfZhxi8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942400; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=HDfdVGS42cRDlfVmkzM60v0gqW3TrtFRZVqqSMTDfdY=; b=r5Yi3XIjVr1AGwAQHRRNL4HQ6JHCxPd1QCZkrO+i62mSV7KpIIygcOsIqjky7uLZ7/a3Mv 4CZi7oqSDcGLlWq4OOuGxfgb396tTxSh54wgOSShkRXMIkLDa+KMa1nkEkQrGcqdBRnF5v o8ssvIm0CqscbYQhLVfFRI71Zl22qcM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=HDfdVGS42cRDlfVmkzM60v0gqW3TrtFRZVqqSMTDfdY=; b=aFfZhxi8oehXzWhEbQVlRTvVbMFGPma8OEbTx4wrmZUhG2YbGXpPUCQg6j2VMM7zUmd3zR HLgK2I/h4UgS1l6KsikywT1RfV7BBkxoo6e7cI/ca2jRUjOfr/erMDpYmgp6gkg8rr8tw2 QWROQ1NW17Bu/oCiK2LERYwgnauwtPI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-235-iEbf6p2kN9S3oSPRTvetUQ-1; Wed, 24 May 2023 11:33:15 -0400 X-MC-Unique: iEbf6p2kN9S3oSPRTvetUQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AA3AE280BC9A; Wed, 24 May 2023 15:33:14 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 03A338162; Wed, 24 May 2023 15:33:12 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 00/12] splice, net: Replace sendpage with sendmsg(MSG_SPLICE_PAGES), part 3 Date: Wed, 24 May 2023 16:32:59 +0100 Message-Id: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 98D5F14001D X-Stat-Signature: i6pi78zn6mxu9qcw5g4n95kxmwegqgph X-HE-Tag: 1684942399-535971 X-HE-Meta: U2FsdGVkX19xYrkjDBLKPbWw8UnTUTAeQv01sMjpVtPleYAbr9TFYuJD0WE9CkilevqUNMQKFxmy2mcX/ttDusG1DpkV7aCU95veMZwhimT/4urRUH3+PfVuIhWCHnFOfuBILjSJJDV3hEWbqbdPZa1XGMmxVl4E6xwRuzgZ+zESMsIBsm2kZPTDRE1svX98E67Wytzh9qUdY4tzuAHomb3K903KlpZ6kQzfUkMPIl/SVmaXucjWDyIHE226/TOKZbQW2894vCvRBoGGjD7+4UHozLbnWXnuBdp4MlI25qP0XPt8W/LHCRMgel9vPJYbU0qj65zBARIMFgFrA5LaXLWqjgy+DL2MoGNCUVCgluKS1ggBubD7bViaXG5z52l2gWVq1HtABrYBEFi3tpGdOH+kiAdFXXYomzGuEynvrynzZJuof1XW6Eag3A/pUV+Edfxu10bV4LNPVg36+Mc7bYyLFqKX784JvFIgfwuDeLVnkMuL6+CdNL++F7mKkwqca/l5oePtkPgvGajCv2T0JkUvSTcLXCCNrZpZ7mqFQpDaL9HEKt9hnUqR2BstRqyxzfWDiwUwN244a1vbgnIPrXA/k+TF+UOTiIsTw02+tnn5kGyIHQsD3b93OFd1ov+yRvSADhZSgfVMUv7SIiSjVF0eeSs4SiUF98HBreizCACtIO9n//By6WqYUKdS3QT9v8SsijnbOFYjBlegkJoATRtuCGiuDYwliADhWzX/pB6zK/9g3DVh9o+yawnuVJBsJIbcFfCqOTHrlQbblZCg1V5CYsaiWjFJwSOr3ugeiC3yVA3Xa8kllABr0a2ib5UL5JblTIyTKf/JENrdrEB2ZF870MvfF9eBV8jd7hTilV7sULLfYesbAnphS8uRZcDNnbKrKkyHBKXalYmdfO8R+DKYgqnkL2GQsvRID3K6sKsN4M/rrfsYhTL1qG0Z+CjLWG6XZ+Y2Qa7+zX/CO4N 4xmQyWnn FH4fOooBtXrvD4Z5hPlyIYJCia/5wScFjjTqEXAKQ1ooEppzux+V9iUOI8dbPxV0Hy9qNCgyIr49ATj+Nfc+1fq6LXA7fHqV6kjjrwblS13Y2xDE0I1TC/K453ZNd4++BE4hiD1PD8bIC0A3FCdPT3wvCxR+WLlRVShxf4ah7ADt1SAqBxphDTObjnHhR61B/Eo01Yr78cDh/jxG5t+4OS8fJhulVNKfKiJacOnuOzJll28P0BvuWRzwA+Oh757anHPR0QP0GumXbjBlvgPk8fag2ph68IE+TRbrj6uFPb4CSJ42JpaqkBVgSxPdlZqQLVwjJQpwjc4Rbqa4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here's the third tranche of patches towards providing a MSG_SPLICE_PAGES internal sendmsg flag that is intended to replace the ->sendpage() op with calls to sendmsg(). MSG_SPLICE_PAGES is a hint that tells the protocol that it should splice the pages supplied if it can and copy them if not. The primary focus of this tranche is to allow data passed in the slab to be copied into page fragments (appending it to existing free space within an sk_buff could also be possible), thereby allowing a single sendmsg() to mix data held in the slab (such as higher-level protocol pieces) and data held in pages (such as content for a network filesystem). This puts the copying in (mostly) one place: skb_splice_from_iter(). To make this work, some sort of locking is needed with the allocator. I've chosen to make the allocator internally have a separate bucket per cpu, as the netdev and napi allocators already do - and then share the allocated pages amongst those services that were using their own allocators. I'm not sure that the existing usage of the allocator is completely thread safe. TLS is also converted here because that does things differently and uses sk_msg rather than sk_buff - and so can't use skb_splice_from_iter(). So, firstly the page_frag_alloc_align() allocator is overhauled: (1) Split it out from mm/page_alloc.c into its own file, mm/page_frag_alloc.c. (2) Add a common function to clear an allocator. (3) Make the alignment specification consistent with some of the wrapper functions. (4) Make it use multipage folios rather than compound pages. (5) Make it handle __GFP_ZERO, rather than devolving this to the page allocator. Note that the current behaviour is potentially broken as the page may get reused if all refs have been dropped, but it doesn't then get cleared. This might mean that the NVMe over TCP driver, for example, will malfunction under some circumstances. (6) Give it per-cpu buckets to allocate from to avoid the need for locking against users on other cpus. (7) The netdev_alloc_cache and the napi fragment cache are then recast in terms of this and some private allocators are removed. We can then make use of the page fragment allocator to copy data that is resident in the slab rather than returning EIO: (8) Make skb_splice_from_iter() copy data provided in the slab to page fragments. (9) Implement MSG_SPLICE_PAGES support in the AF_TLS-sw sendmsg and make tls_sw_sendpage() just a wrapper around sendmsg(). (10) Implement MSG_SPLICE_PAGES support in AF_TLS-device and make tls_device_sendpage() just a wrapper around sendmsg(). I've pushed the patches here also: https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=sendpage-3 David Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=51c78a4d532efe9543a4df019ff405f05c6157f6 # part 1 David Howells (12): mm: Move the page fragment allocator from page_alloc.c into its own file mm: Provide a page_frag_cache allocator cleanup function mm: Make the page_frag_cache allocator alignment param a pow-of-2 mm: Make the page_frag_cache allocator use multipage folios mm: Make the page_frag_cache allocator handle __GFP_ZERO itself mm: Make the page_frag_cache allocator use per-cpu net: Clean up users of netdev_alloc_cache and napi_frag_cache net: Copy slab data for sendmsg(MSG_SPLICE_PAGES) tls/sw: Support MSG_SPLICE_PAGES tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES tls/device: Support MSG_SPLICE_PAGES tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES drivers/net/ethernet/google/gve/gve.h | 1 - drivers/net/ethernet/google/gve/gve_main.c | 16 -- drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 19 +- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 19 +- drivers/nvme/target/tcp.c | 22 +- include/linux/gfp.h | 17 +- include/linux/mm_types.h | 13 +- include/linux/skbuff.h | 28 +-- mm/Makefile | 2 +- mm/page_alloc.c | 126 ------------ mm/page_frag_alloc.c | 206 +++++++++++++++++++ net/core/skbuff.c | 94 +++++---- net/tls/tls_device.c | 93 ++++----- net/tls/tls_sw.c | 221 ++++++++------------- 16 files changed, 418 insertions(+), 463 deletions(-) create mode 100644 mm/page_frag_alloc.c