From patchwork Mon Aug 21 20:20:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45D9AEE49A5 for ; Mon, 21 Aug 2023 20:20:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10DA494000E; Mon, 21 Aug 2023 16:20:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BD4494000D; Mon, 21 Aug 2023 16:20:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA06E94000E; Mon, 21 Aug 2023 16:20:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D028B94000C for ; Mon, 21 Aug 2023 16:20:26 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A4152140CD4 for ; Mon, 21 Aug 2023 20:20:26 +0000 (UTC) X-FDA: 81149229252.27.B63D9D7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id EE3E9C000A for ; Mon, 21 Aug 2023 20:20:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UeWDfgiV; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ftMoTpPKDvJHF3KGIoW6evnR7Kcy68rm/mSseTTxyDE=; b=J5JeO0HaqxfaU6xS++7kIDIJIclFo/jzxvZbLRXWVAtrOSSRoWyI9iDXghY65DX6CUuAF/ pFvTCmyCxcgZh3jblzKabWLSsg5EWyMxLTYchc25qdx0T48GVSFRtchO9cYQ3e3vrlMwJj Tptc2+jarkKXfgVdHsL5f5Q+YoPs8nA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649225; a=rsa-sha256; cv=none; b=Zl/IxljmIF2khb3pJz4U/Pl26p7kilOAD2xpG++o/dBkDytvoyBGI/tSFdmCsA40kvwDC6 iv6nDoUnoCqWH9sq95QwJOwb2jXFdruGpaba7hx8ESBj8WYOn3S3FOtA10MNuOAP6Zv+bL 6B+qnEXDWfMed97snn2GctmRR+7dNto= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UeWDfgiV; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ftMoTpPKDvJHF3KGIoW6evnR7Kcy68rm/mSseTTxyDE=; b=UeWDfgiVvfANwzOmWZAyMTV5dO PtODFzfq5fwt4KNX841Xw565F2ny38hCVRGqSFXTetiJvlP4NauQmuhso+HMwmUJ/5l0ACV4cP40O vbfJ6MWPWeP5oQ004FdUXRizGoZ8RNh8d1Uk+y+/qEJ4JTqmDRRqz/+nrlGttzhIAV0Sf/mjtGMVn +LXUpHEU+FeFspMvpWqu9dpAtPqli3bfXKZCVMi5MXgoh9DQ8FHwWGNlcaX3bNtpzDJW9cCTdpphr mmKeRz3hSoROk6kcxQM+bMmhsT/ke8RGpBF4DgJKJuuMn/kWpCvaSK6c0jbIC025wcVOXzXwf8NmC HPEzDciw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBNq-00CD6u-DU; Mon, 21 Aug 2023 20:20:18 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-perf-users@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, urezki@gmail.com, hch@infradead.org, lstoakes@gmail.com Subject: [RFC PATCH 1/4] perf: Convert perf_mmap_(alloc,free)_page to folios Date: Mon, 21 Aug 2023 21:20:13 +0100 Message-Id: <20230821202016.2910321-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821202016.2910321-1-willy@infradead.org> References: <20230821202016.2910321-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EE3E9C000A X-Rspam-User: X-Stat-Signature: 8r4pfddguob8okr5rk4crhzczcwyia7t X-Rspamd-Server: rspam03 X-HE-Tag: 1692649224-223076 X-HE-Meta: U2FsdGVkX19/pybOjwkJd65jWu8Tm6a09wLDkZfKjmuEcva3NjTeKUBc1l0A2reI908KbirvJrzX+jLNvrV9+r0cAk/omsvFg2lmM0Y/m6eELxWaT8usHSqcY7EMsjh8XucOt5c3kwK7LOnmL+F7IeldEJEenPg4Mholr6SyhQY8EkJP54NAUqbWSk6+2G/uhVgSO+efsgqEnU4V1s8dho2Utz5eMgBDu4XWiWdPHOpRpmMLR9Dk5JKaUZHlqZykg+Acma06UjwbC69qn/WhYsJUdnTSiF0YK2s8u0B6JUons12RzaZWqmvLCatWJ/+si6ID/ZTb4cpjYi7N9Ifh9VftuKFbg/4lZvbO00aPM5E4DP6Uxf+WYhzjiyAIPSyr54J7pBD5l3XY7Rw8GPeJoCps83VPNS0z2vfLA1SaZqRWFj9gTZIGYqC6RgD+2yjntIEZ6dv0foAvWSRyKqs1/4zElbXpyOnd5jg4AIFukrZn2g4ltDj/EECSXaANye57C5YHv5gONU2CwDLiaAy0XOzyPSVta9FrEpNYROITANz77nQNr3IPLTp5OjYpbjl2/opnpMx750cSSPa6kjEhwW3yfa4Bb7pH79BkqKiCo85lpNQHY5ee70uqXnpdDeyHPiXN3bJu/sz1girjBBqx6sbuZh8z+iDAj7cITOp+fZEmwgt9yOqIHSxljWif6kgclnAavaFg5qQwT9eVBGIoSMhZk6pDTks0M05ZXPcrnKQhaerH6W5HoKToFjAgHUtC9xQEweWSZzId5BCwQIn+jnroIAPXEX9EwWxNu+951E3EI010OxWLSeGjVLkkitMTBzBI//+qswh9IJra8Kqs98u/We4UHmcFbppBW3fEO4T1ovD8SEuKumaOJ6ppu50zkSavWNDe+Eb6LjycHMabWKXvMELwPtrMClrIwoiezcsufyKb/apmpvM9wBzQoiogTXNAnpqO+n9GQO205zU A9p4BLLw CHX+w1dVmnwooWcH3T4ye9RGkyE8IfhjdyAR7hI9eqcqDX+ptMimspaoRy1SiZeEgqlEcz/fbZZjYFHpwHf2Bb8jYc15wNK3Ycv09kKYP0ys+OAseZq2UqhKWkxpyYlCUI29e0kY9UtYDOR6QrxiFYi1XTHIQYOBytGzMg+9thF0BjrBnYr05j5cHTkvOjKeuqSiPB4mZ6jMLrO+F/8HE/hDZr0GNpDjK4/lH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the folio API to alloc & free memory. Also pass in the node ID instead of the CPU ID since we already did that calculation in the caller. And call numa_mem_id() instead of leaving the node ID as -1 and having the MM call numa_mem_id() each time. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Yin Fengwei Reviewed-by: Lorenzo Stoakes --- kernel/events/ring_buffer.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index fb1e180b5f0a..cc90d5299005 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -770,7 +770,7 @@ void rb_free_aux(struct perf_buffer *rb) #ifndef CONFIG_PERF_USE_VMALLOC /* - * Back perf_mmap() with regular GFP_KERNEL-0 pages. + * Back perf_mmap() with regular GFP_KERNEL pages. */ static struct page * @@ -785,25 +785,23 @@ __perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) return virt_to_page(rb->data_pages[pgoff - 1]); } -static void *perf_mmap_alloc_page(int cpu) +static void *perf_mmap_alloc_page(int node) { - struct page *page; - int node; + struct folio *folio; - node = (cpu == -1) ? cpu : cpu_to_node(cpu); - page = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!page) + folio = __folio_alloc_node(GFP_KERNEL | __GFP_ZERO, 0, node); + if (!folio) return NULL; - return page_address(page); + return folio_address(folio); } static void perf_mmap_free_page(void *addr) { - struct page *page = virt_to_page(addr); + struct folio *folio = virt_to_folio(addr); - page->mapping = NULL; - __free_page(page); + folio->mapping = NULL; + folio_put(folio); } struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) @@ -818,17 +816,17 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) if (order_base_2(size) > PAGE_SHIFT+MAX_ORDER) goto fail; - node = (cpu == -1) ? cpu : cpu_to_node(cpu); + node = (cpu == -1) ? numa_mem_id() : cpu_to_node(cpu); rb = kzalloc_node(size, GFP_KERNEL, node); if (!rb) goto fail; - rb->user_page = perf_mmap_alloc_page(cpu); + rb->user_page = perf_mmap_alloc_page(node); if (!rb->user_page) goto fail_user_page; for (i = 0; i < nr_pages; i++) { - rb->data_pages[i] = perf_mmap_alloc_page(cpu); + rb->data_pages[i] = perf_mmap_alloc_page(node); if (!rb->data_pages[i]) goto fail_data_pages; } From patchwork Mon Aug 21 20:20:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B4A5EE49AA for ; Mon, 21 Aug 2023 20:20:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB1E28E0015; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6C8C8E0012; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C01D94000C; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7FF958E0012 for ; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4FC0D120C88 for ; Mon, 21 Aug 2023 20:20:25 +0000 (UTC) X-FDA: 81149229210.11.3B3F283 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 323E4C0013 for ; Mon, 21 Aug 2023 20:20:22 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Z2QgGOEX; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649223; a=rsa-sha256; cv=none; b=dBVp0bgawNHphTmO412uNLYMKKNRzRkCSXXle7T4Yxyt8b3M9LB3bxa6wP9+yieEcr97fq 5EJH1hWKsfDxuSoP3gDy5WlNx+x5Y6B/+AKytq30gLBePJUUjURQn6tEwDYTEO+ghNg0We PL/n7o3Jpl1xRg1hnPFSFxjVkkoF5sw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Z2QgGOEX; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tYGfFyUq7t7eG2DfnJgfGPfrNOUqXETtillKZ2Ic8a4=; b=pz6Z//vn0K9/x7P0pyuzKunj+q+8GNnMkdh5cmxq6cx6eHSNdTiYW2LT+kMIEyEOt3SLT+ 1P7Xq2RIyDEHg6XR2AZ//ky8KgoyfymlQycj7tkXI1IGLTcGrZ9f1H/ALHdz/Thi/faniu +v2Pv6nNHD6HQ8pv6UAdxbLPlig3nJ0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=tYGfFyUq7t7eG2DfnJgfGPfrNOUqXETtillKZ2Ic8a4=; b=Z2QgGOEX+YU88go4CxiBk1uRs2 aBGSbahWb4XB33exhSqsocsfpuLMOSih/d1J+oYJPR4M7YnasVgFl4BNQxJZ+JP3WXqkD4nezXWyn ceiFlMzm9x7eRYETe01vkUIGJHoop4hUOAM7xrLUa34dPoUSs6anrg1lmyuqJAmwfFJhTWpUB1zSJ AuQjc5pru+oOvKhkaJ2jNHETW1I9Z0T1/cSuPQAhLxEyAxax34fGoaiqIzkJM5WSqTF+DfqxYX7Yk l/xIoGs8v1hwFhRuO7znM7MScBvTbAeK3zOOnEdEWye3an37pIQ/tU4wx5t0S8iesyMcE9fpzm289 j+GPvXYA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBNq-00CD6w-Sa; Mon, 21 Aug 2023 20:20:19 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-perf-users@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, urezki@gmail.com, hch@infradead.org, lstoakes@gmail.com Subject: [RFC PATCH 2/4] mm: Add vmalloc_user_node() Date: Mon, 21 Aug 2023 21:20:14 +0100 Message-Id: <20230821202016.2910321-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821202016.2910321-1-willy@infradead.org> References: <20230821202016.2910321-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 323E4C0013 X-Stat-Signature: 76jn6g6s7ao36ijpzjbkt9hjzt7wi4n1 X-Rspam-User: X-HE-Tag: 1692649222-455053 X-HE-Meta: U2FsdGVkX1+ypu4MyjLG4TAW/9mG2tEgT2jKdG2UPB9fOahRTDRyCL3Cdpqc9sbMTofszWcgsrgQAArpPtrrRO9hmsR3d1rOEJ+f3I3WnIx6Yli0YQ0XUEgV/Mb7eth2++7t0LKfjJbRzKJYjW/vPQXJnK91ookVhl8gPOu4NwZ40xJ6B+NYrHFp4K4c4ulJnbOFOAJqts+awkxsc/Y04rsyddvS/xZhTRERhKCEOlVu4y5jiXlp0DqCRjc4Gb2bnWdM7zXvH3cV3aZzZrxOX80HmYsXK2xb9wnEikQxRdKz4qxYNf8W6RW4VkoIDRDXro1URS6cMk2upXkh2VWlKglXONnl5Q9JERChtGmaq8HNThzkOngvyLko3mIQD8FjrZRBLSgCCbV73OZld57SSX2fxb46h8iUiAIhknyYzPjsB/SbeAnj2jLkIEdW2EYC/vD/KmauLXNRPllysQIOgNC+1CqxWfOOGYLp/gDrBTwb3/faM6FZRcbpb+iOLIAEjn1Qpvq68EfIhX64uqIv1acQ4qNG2fX3LHD+XT1Hq72AKOb90fRfb4KUlcwoQR3dgA2V1CoVorG1qdFNR7y/rsg5o2PCKw1O/y4pWnthc42NkqIZ0SqPpr3DzEeDzFKvFjSH2c6D5T0Ot+hUT4Irz1BzTx0bbbG79uQJwo/oRd/nSFz6c5uyOL/I4bEVaqznGLR0BAmEhyYDcg6HCD7aXLYK5jsTkyaEhMPeCt+0X4ut3rqT+7736z45V/mKQAWUdwEFqNbHwTEBelQTKuCOqw1Ln7SKLII8YEgqMuoMEGFeDvw46s2SNH3Yb5Cj/ZbndIhyU2CCCO8CZkOs9WWiTbTCYscm8uId721/TPiDN3JFRn47kxxnLxM/M2jaHsvBvOeOXFDHMS1+VrS2L18Ds4pwdkMHUsJHKCuEXjFxxGD+Rf95jWOSSxp1ZkhvgnV8Gxvq6zyklz9ggk+pHmE Ro4Dy+p9 FUtWzjbF+RUAZ/mUk5mJFCNRnFb1b31bjtmHjxDlibyuK0kwUusYMZfLltGAPoa/8jGhov1zkZFpLnVew3+LxiA6ODRKb0kqxsL91bLcXmh8C5sIA3vpR/28SALo2gkSj2+QXjN+4OMneHGeCKGmntadTIbWfMsBtUTp4VAcuTKfMly0woRw+4AT3liQImRd05FSphLpuKRZ7mz7b5vC3vvDaqglNnY05tUtm+wPmrJJEd6rBY8jU9Opm1lxC1EtSMz71CRvTech3eNs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow memory to be allocated on a specified node. Use it in the perf ring-buffer code. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Peter Zijlstra (Intel) --- include/linux/vmalloc.h | 17 ++++++++++++++++- kernel/events/ring_buffer.c | 2 +- mm/vmalloc.c | 9 +++++---- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..030bfe1a60ab 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -6,6 +6,7 @@ #include #include #include +#include #include /* pgprot_t */ #include #include @@ -139,7 +140,7 @@ static inline unsigned long vmalloc_nr_pages(void) { return 0; } extern void *vmalloc(unsigned long size) __alloc_size(1); extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); +extern void *vmalloc_user_node(unsigned long size, int node) __alloc_size(1); extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); extern void *vmalloc_32(unsigned long size) __alloc_size(1); @@ -158,6 +159,20 @@ extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +/** + * vmalloc_user - allocate zeroed virtually contiguous memory for userspace + * @size: allocation size + * + * The resulting memory area is zeroed so it can be mapped to userspace + * without leaking data. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline void *vmalloc_user(size_t size) +{ + return vmalloc_user_node(size, NUMA_NO_NODE); +} + extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index cc90d5299005..c73add132618 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -918,7 +918,7 @@ struct perf_buffer *rb_alloc(int nr_pages, long watermark, int cpu, int flags) INIT_WORK(&rb->work, rb_free_work); - all_buf = vmalloc_user((nr_pages + 1) * PAGE_SIZE); + all_buf = vmalloc_user_node((nr_pages + 1) * PAGE_SIZE, node); if (!all_buf) goto fail_all_buf; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 228a4a5312f2..3616bfe4348f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3461,22 +3461,23 @@ void *vzalloc(unsigned long size) EXPORT_SYMBOL(vzalloc); /** - * vmalloc_user - allocate zeroed virtually contiguous memory for userspace + * vmalloc_user_node - allocate zeroed virtually contiguous memory for userspace * @size: allocation size + * @node: NUMA node * * The resulting memory area is zeroed so it can be mapped to userspace * without leaking data. * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_node(unsigned long size, int node) { return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, - VM_USERMAP, NUMA_NO_NODE, + VM_USERMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_node); /** * vmalloc_node - allocate memory on a specific node From patchwork Mon Aug 21 20:20:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF395EE49A6 for ; Mon, 21 Aug 2023 20:20:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 385F494000C; Mon, 21 Aug 2023 16:20:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 30C0E94000F; Mon, 21 Aug 2023 16:20:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 070BE94000C; Mon, 21 Aug 2023 16:20:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E485494000D for ; Mon, 21 Aug 2023 16:20:26 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BFA331A0C22 for ; Mon, 21 Aug 2023 20:20:26 +0000 (UTC) X-FDA: 81149229252.26.ECFD3EB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 2C69F1C0006 for ; Mon, 21 Aug 2023 20:20:24 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RhSHMGYj; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZjTkkOXXXSSKh7AkJZazpX1s5qsUP26GHpbaGZUnTo8=; b=Q84iu4X7OskoyEkEDf8LmCgSuKO+MaewNycmfPBYVZvnpYHkjfMWLzEQJx9d3oqhcKtb0e rteeVlK/HhHfnHAvANyj8PJHSMRfvlqscb+35lEogCZzvQt1GlgXce5ELK9uf0Oxb/ED/h 914YNcgM104TdBNBmlyDcPt/S6irkM0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RhSHMGYj; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649225; a=rsa-sha256; cv=none; b=C2CkmCZioMTkGwFMjnAz0tCK7GbJcfBNk+JQTY56ScYpO27eSftblPD8UnuntlSyvyAgt4 EXz6BCLL9U6ejGgJnwfqilaeR5oqTUdk3RJ5mj/L0agadJQlJZHuEvyWkqWOn3Q7CD+6RF /Yw0sfDxYeY4POemueefiwb78p7Q84Q= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ZjTkkOXXXSSKh7AkJZazpX1s5qsUP26GHpbaGZUnTo8=; b=RhSHMGYjzeabO+TPC6Vk0fsGAP ESwQD+kA9n05mFH4V7x5E1E/PK7+hX6AVEwb0xfW5mvuGVGJgTKBnfxSxTrNj9bcH6p//Sh8Ak0uq rPvFNvCQYo+40HrjKlG+G2jBE3tczpJuiIIO9bhjugOITLwOZalFgpX/o5/DZfdiMbMgRwny1WOEN TVHwPs+hl9Ud0NYAy6FhPagROuHL0gCeWZ/nij+EVwy9Nu6WS1aGddneN0uBfZEpYcAEwIxdTi+Sc 3SSVYmvOzhYL04ZJaaKKd5yd719t3TJ4WW2WFT7uVRi/f6Ycmzg3N8KIwx/VWQCKMFfHizqP5zsRC kUFKWb8A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBNr-00CD6y-Av; Mon, 21 Aug 2023 20:20:19 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-perf-users@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, urezki@gmail.com, hch@infradead.org, lstoakes@gmail.com Subject: [RFC PATCH 3/4] perf: Use vmalloc_to_folio() Date: Mon, 21 Aug 2023 21:20:15 +0100 Message-Id: <20230821202016.2910321-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821202016.2910321-1-willy@infradead.org> References: <20230821202016.2910321-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2C69F1C0006 X-Stat-Signature: sohszagjrxbq8qkr68k7iqijjm1qjhgi X-HE-Tag: 1692649224-674452 X-HE-Meta: U2FsdGVkX18XCyvxZfgLaKFoQAezc6e4aXdvb68qdd9Wq07VwEJiTQeE36xvDiL/FDJzrnPyTJ0TdX5y58Va7XQF/G3ejSvApgelgn9nndkJlv0MEQeCZXgh8jPRAvVnNFtM5D4XGDcJc6hhNSV35orLQoTsDiLfQvcGGyigDAgsO7wuXfz12iskDLjTnq/PBymh4SX5vIq2+rbwAEuuuFjKVWEyhKrHX0S4CLQX+LmZcCvEcm1MXF1TVwVSP+2Xd4APO7EAqOy0RIb5pwRDIBSGE6Ilbp0FguuXOk1ddMLStNdH6eqEM0eAW4koZ9pX6yaUBpNm/TXClNX+jprIe3lYgzrv9eXN09PZuwb9iR9NVGAGO0qIdBC06uNeta1i74y7f4KDmbcxCg13Y/hyChpW7BKw8qnz29A//KbKHQWWtKNWW+p+Lvi+9Sus2U11WR8Z8mqxglQ9iieVukrofITw2j4XxTW9ath6znCRM1We9mZPafandlCwEizUKWKvusa0boXijd0/QMC72a12BmZSTaqwIzExmuEKv4wRLa0K9Z2NNgcpjNQjnQEk9t+w5zG45LW4aJWwrmEDsQ3VliSew8wApn31gZUWEWcnz7Ja7arMQuRjfz9w7+bMuiPVu0A7tnK1E2SVGiltmyGNr0S0s1T4QQ3VVTKQW8QVNPufM6SVFgQ6TxYwskkTSaKkMaJu0xY5yQIDkL2zC03vWZ7xHBecXuF7sJktVR9HspzE/wPIwp8whEGaOKjPiY1802CCBkiB8JHuePQrMZNDGgWGAXUnb7NcrdwT97RjQpi59kv2HLvHy9HRQOxSlUcrW2zvcCLR+2fnsZYscHx0Xn8jDGuIx6t5CMUraiSHL8+0exfAVtwgTvHGNFvs1W/NFZNSTJ98h0qqJfNwO2ez6H//llQY7bBu46d4j89SB/xjtD7QfKEUthHoKcimNnU3bEdopGxoN4cs8mJ33qN 9bWT4KFc tBzvscn0FtZ1klhQH16b/VDdRQ96dTS20+q79QLHtqdoJ9A0C5kSQeWI+VwvAZEgE59Zyn/0IbXjJ6EDe4QQA6/icxLNS3V/2HezeUs9Q9lSATOYF6BH5lBXfijhhjkF9a/jcLpc4vGUvIGW5aFcJytc8+ktoFZOwoVUEdHpVgXm0hEIT41GDAQPHOxG+yx0ZnW8lgDIzrYmHWQSrMbPCB8HfUOI640qFoNpP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Eliminate a use of page->mapping by using vmalloc_to_folio() instead. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Lorenzo Stoakes --- include/linux/mm.h | 5 +++++ kernel/events/ring_buffer.c | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 840bae5f23b6..7d84a2843193 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1134,6 +1134,11 @@ int region_intersects(resource_size_t offset, size_t size, unsigned long flags, struct page *vmalloc_to_page(const void *addr); unsigned long vmalloc_to_pfn(const void *addr); +static inline struct folio *vmalloc_to_folio(const void *addr) +{ + return page_folio(vmalloc_to_page(addr)); +} + /* * Determine if an address is within the vmalloc range * diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index c73add132618..56939dc3bf33 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -873,9 +873,9 @@ __perf_mmap_to_page(struct perf_buffer *rb, unsigned long pgoff) static void perf_mmap_unmark_page(void *addr) { - struct page *page = vmalloc_to_page(addr); + struct folio *folio = vmalloc_to_folio(addr); - page->mapping = NULL; + folio->mapping = NULL; } static void rb_free_work(struct work_struct *work) From patchwork Mon Aug 21 20:20:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BCCCEE49A6 for ; Mon, 21 Aug 2023 20:20:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF8F594000B; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA8928E0012; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 948F394000B; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 82C348E0015 for ; Mon, 21 Aug 2023 16:20:25 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 568801404E0 for ; Mon, 21 Aug 2023 20:20:25 +0000 (UTC) X-FDA: 81149229210.10.BE254D0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 9CF0A140015 for ; Mon, 21 Aug 2023 20:20:23 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FvNZJNN8; dmarc=none; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692649223; a=rsa-sha256; cv=none; b=zORnsBYw97PU/Wu4Fyptse4XTmKDL0uNsHjLun0BmvcLEBSl53PCdTdtv1eT1Jepkg/AAC w2GnPqF6fWVyJoEZXhCnbLmNryjSmz+PWYomIMc0h6YCUfEKIYebaaNo4WduS/NdwSJiu5 QXnhvUj4gYmnaoPbCafk3CxJdSL4jdc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FvNZJNN8; dmarc=none; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692649223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bqsHWtg0sTY6IqpOvfeSnvnFR6wJrIohLDKSxFgMNTg=; b=wQLdZU1um8pe4ggJw67ZFihLy1m1g759bF2DfJyLeS7kmF/HvN6JklCH3UlPPUxgwhBfeh 9RE3i54kHMZWdlc9Y1Qj/hvlEitP4Z2uKFHOZ+S/2vpp0Nv10k/WNzVLkmDmx+yY9onMoO iO0DfRy2qHN8bXccFVwcxZtBhAeisIY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bqsHWtg0sTY6IqpOvfeSnvnFR6wJrIohLDKSxFgMNTg=; b=FvNZJNN8SUL6GfqaGvvyHGSebj B04lwo0uLo0HB4Ovf2BY9W7vSiLLv57FTFUd+gAEMLFpaVoYhXnv7Hu1CMd3sLB8RPorQidikfaaK 8AnrKAuFUCPgeUksT8EE8mOYLTghJrEl/lttVNSi5egYqWX6HD8zN+d+zaAEaHJUSZ81Xqo4V52Dx CJrv0mZXC8cbG+o0QMH4iDlmGhLhmk/U3V1VlVgBlJIdZ/Wyza2LRheOeA4zDDnZ/RdQbQKSBs/vo hacE20AMuNgi0XQhOO2aLi0N9IJXUjbKX8QsFSzQnLJng3iIdgpjCI8hYK2Au07dIkdvcKYe3fDwn za+Hsikg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBNr-00CD73-Q2; Mon, 21 Aug 2023 20:20:20 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-perf-users@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, urezki@gmail.com, hch@infradead.org, lstoakes@gmail.com Subject: [RFC PATCH 4/4] perf: Use folios for the aux ringbuffer & pagefault path Date: Mon, 21 Aug 2023 21:20:16 +0100 Message-Id: <20230821202016.2910321-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821202016.2910321-1-willy@infradead.org> References: <20230821202016.2910321-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9CF0A140015 X-Stat-Signature: b977n9y8kpk1ixrroyf7x8f8fwrykm3q X-HE-Tag: 1692649223-301562 X-HE-Meta: U2FsdGVkX18jZMSvPJLKaVLPOwDDTFEqZkDj1MJ9x8zURpE1C8ZIVCSb+FET79459ZFv35z1zgsiJRFxMfHQiYrOdSV53TaUhBntbU0B553Rp+1ugU/MBL42TbZA3huUzc6evtxE142oBtDnMvGPDNKhjxJPfUsuLEh92uhuLsCXSiuCE6nqsEXVIAGKV7bFD12fePku2GjJe+WgDp6QyQUqQybPAE7QJ9lFGnMylvI4GSAciILMZSTT4Tai3WWs4yoZi9kUskIMy8hI6ZtKlhEE6bucRaUSMawaXpVgJsBQJNpxzMzL0O5ow32iIHJEbeiDVzpOuGShpzHHjhEsFxIQ20hBut6g7b3+SUHWl/XPMCKN6cOVqvD9wv+fitj8XoplSb3pL+ML6mQs2PgWB6loHIb6M7rOyyptChEPe0atfO5U5zsoRNvfWuaAmIoZDxPjDjJ1b8x8ibQMwOs7pT0yBXmf+VN3Y6XBc2f+JwaEFv2YpyJ8zV2U3yg5pPjHFhdSQo0dUovrQXPDYSvw92LcJ5hn7hTRAg/HgqyJuPeehp3ttPUWASrxsVSoeXee+2dYNHBVU1W72et25h+uGfUy2mcb7cLJ7JVIgxLdWGPcWA3bIpSb61vAxvjndzPQKxPFTDZYyC+W2ABXmLwqbH7WxK3FFLmAifii+OZslME9EcC+734hLTHgOzS5TterOWM3wVhR2+xCj+Wmr6evIrZBYa4eUnuUKgb7N5Ir1iSq8GZwEbnNasX6eGxu77gUtGkuPfHUcYiF5iqrk6Qy+gI+WNu2U4Xs09n4VynmE71PLO4cegwx+Hzw0mGZ8qCIWeq7tmBUQ3ReRplbzPfmgz+IuPmvKh/SkTjpkEy6xXhgrV/bL90cuS2YgBm0GHCSYIx041LKkv2fiYiF7/coQgWVKhljEg7UOBbdgkbknwcdChqab8yZBXLoJBDxyXUONkqAYY6Qx0W4HSvFnvm tkPmUdMa +zZFzkO11Gz3csxjgz90P6hHx4IWATfPY8kpLAkPDP4571SDcS7PVMRIfTkB8JC46uNYk8Lw8n7+rWnmYjftnsyNaPtBe+K/Rw8a0ZYV6vpeqjwCBZ7r89riOfsH60l6ks9JCrNxRj9tjjncjywku/F+l+cH0quD6yh8Axu/whcdAbn6RYNt3GnhC0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of allocating a non-compound page and splitting it, allocate a folio and make its refcount the count of the number of pages in it. That way, when we free each page in the folio, we'll only actually free it when the last page in the folio is freed. Keeping the memory intact is better for the MM system than allocating it and splitting it. Now, instead of setting each page->mapping, we only set folio->mapping which is better for our cacheline usage, as well as helping towards the goal of eliminating page->mapping. We remove the setting of page->index; I do not believe this is needed. And we return with the folio locked, which the fault handler should have been doing all along. Signed-off-by: Matthew Wilcox (Oracle) --- kernel/events/core.c | 13 +++++++--- kernel/events/ring_buffer.c | 51 ++++++++++++++++--------------------- 2 files changed, 31 insertions(+), 33 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 4c72a41f11af..59d4f7c48c8c 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -6083,6 +6084,7 @@ static vm_fault_t perf_mmap_fault(struct vm_fault *vmf) { struct perf_event *event = vmf->vma->vm_file->private_data; struct perf_buffer *rb; + struct folio *folio; vm_fault_t ret = VM_FAULT_SIGBUS; if (vmf->flags & FAULT_FLAG_MKWRITE) { @@ -6102,12 +6104,15 @@ static vm_fault_t perf_mmap_fault(struct vm_fault *vmf) vmf->page = perf_mmap_to_page(rb, vmf->pgoff); if (!vmf->page) goto unlock; + folio = page_folio(vmf->page); - get_page(vmf->page); - vmf->page->mapping = vmf->vma->vm_file->f_mapping; - vmf->page->index = vmf->pgoff; + folio_get(folio); + rcu_read_unlock(); + folio_lock(folio); + if (!folio->mapping) + folio->mapping = vmf->vma->vm_file->f_mapping; - ret = 0; + return VM_FAULT_LOCKED; unlock: rcu_read_unlock(); diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index 56939dc3bf33..0a026e5ff4f5 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -606,39 +606,28 @@ long perf_output_copy_aux(struct perf_output_handle *aux_handle, #define PERF_AUX_GFP (GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY) -static struct page *rb_alloc_aux_page(int node, int order) +static struct folio *rb_alloc_aux_folio(int node, int order) { - struct page *page; + struct folio *folio; if (order > MAX_ORDER) order = MAX_ORDER; do { - page = alloc_pages_node(node, PERF_AUX_GFP, order); - } while (!page && order--); - - if (page && order) { - /* - * Communicate the allocation size to the driver: - * if we managed to secure a high-order allocation, - * set its first page's private to this order; - * !PagePrivate(page) means it's just a normal page. - */ - split_page(page, order); - SetPagePrivate(page); - set_page_private(page, order); - } + folio = __folio_alloc_node(PERF_AUX_GFP, order, node); + } while (!folio && order--); - return page; + if (order) + folio_ref_add(folio, (1 << order) - 1); + return folio; } static void rb_free_aux_page(struct perf_buffer *rb, int idx) { - struct page *page = virt_to_page(rb->aux_pages[idx]); + struct folio *folio = virt_to_folio(rb->aux_pages[idx]); - ClearPagePrivate(page); - page->mapping = NULL; - __free_page(page); + folio->mapping = NULL; + folio_put(folio); } static void __rb_free_aux(struct perf_buffer *rb) @@ -672,7 +661,7 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event, pgoff_t pgoff, int nr_pages, long watermark, int flags) { bool overwrite = !(flags & RING_BUFFER_WRITABLE); - int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu); + int node = (event->cpu == -1) ? numa_mem_id() : cpu_to_node(event->cpu); int ret = -ENOMEM, max_order; if (!has_aux(event)) @@ -707,17 +696,21 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event, rb->free_aux = event->pmu->free_aux; for (rb->aux_nr_pages = 0; rb->aux_nr_pages < nr_pages;) { - struct page *page; - int last, order; + struct folio *folio; + unsigned int i, nr, order; + void *addr; order = min(max_order, ilog2(nr_pages - rb->aux_nr_pages)); - page = rb_alloc_aux_page(node, order); - if (!page) + folio = rb_alloc_aux_folio(node, order); + if (!folio) goto out; + addr = folio_address(folio); + nr = folio_nr_pages(folio); - for (last = rb->aux_nr_pages + (1 << page_private(page)); - last > rb->aux_nr_pages; rb->aux_nr_pages++) - rb->aux_pages[rb->aux_nr_pages] = page_address(page++); + for (i = 0; i < nr; i++) { + rb->aux_pages[rb->aux_nr_pages++] = addr; + addr += PAGE_SIZE; + } } /*