From patchwork Wed Aug 16 15:11:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13355387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E750C04A6A for ; Wed, 16 Aug 2023 15:13:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDB81280023; Wed, 16 Aug 2023 11:13:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D89B9280021; Wed, 16 Aug 2023 11:13:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C50B2280023; Wed, 16 Aug 2023 11:13:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B8431280021 for ; Wed, 16 Aug 2023 11:13:05 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 92CB7B296D for ; Wed, 16 Aug 2023 15:13:05 +0000 (UTC) X-FDA: 81130310730.11.D4B90EF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 408A040232 for ; Wed, 16 Aug 2023 15:12:16 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=THK3UzQT; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692198737; a=rsa-sha256; cv=none; b=FGfMwmHVXFglQa2LjIzxF1PGDbsFxDjRp2yaVmM0YHNKeQAuN2QFZlGitwURABeFctUTws WtBmrmCUVgfX49Yal4yCrQWGTm4yGpuBDRULsykyZw/jzCmhYE1WF4JgQzO8XGpv7sV8Zs gOeiZrBs4H5F6kVDYAj4epeeS5K07XI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=THK3UzQT; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692198737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7hrNLrvBiputkK0Ez5NemxjwUyUDBXLdxg6jqp/GSn4=; b=Nau2DxkHb5jOf8uVBsvEShQQfxnwH06Ys+J+/AzOvUg5ZTro7KOyamNVG8EkViwlI7286z GKSs4+umF5ebC4Shdu4OrLLkB3xOiTJchkyHL9RdlG7s9/Ap8nJCVhNEv8YN6k4xfzUjQz ARCxvuNk216ExU9cHS5FMRb9hCcKoQs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7hrNLrvBiputkK0Ez5NemxjwUyUDBXLdxg6jqp/GSn4=; b=THK3UzQTMK0YVS/5sddRPJdEQz yzCRsqAftx1rcqtnfHPKVKCb1IJCJpqBMVJHvItV3dqPitxX1z+gd850NiqENXc15t6SrL9STeJyS 7qaBCI3YB4eRNvWq56ASqo6kwwLjxAW9mBqFHN1H7dDhzNBWWr5haIdfIpftOZ2kejf0J32b24CDs TrXd0Jf5/37br0+ab4qgGHO87F/Pi3HqL8HzX35v9piXwQlTUJGdsmDyNXuaM+yogo6vFd0LHS2Pn f/LBd40mpafoHB5cBIy4ag6iyrH6s8te6miyvm+SFaVIeJICVRp1iJsEuYR4obHQNsAfRwhpYg6J8 73YEidLw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qWIBu-00FL8R-17; Wed, 16 Aug 2023 15:12:10 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Jens Axboe , io-uring@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 02/13] mm: Call free_huge_page() directly Date: Wed, 16 Aug 2023 16:11:50 +0100 Message-Id: <20230816151201.3655946-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230816151201.3655946-1-willy@infradead.org> References: <20230816151201.3655946-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 408A040232 X-Stat-Signature: p5q67thhkgqrzf9xkskrc3gwdcrxrcws X-HE-Tag: 1692198736-335945 X-HE-Meta: U2FsdGVkX18+xWHOIhkxyWN5jr+hxUhVCttluYw40wbXoCKJV2Zy2jsN+ej1fWbjsiRegwbhhxUp6JE3ySE//KXnFBk1KStd5A+0Mor3a74rdysrk26JStz8HH5iuTZRF1woZUaraPgtfsPBzRTu9SY1PKNEK0e5BHoOpShaLqn7rhZZAlXFNJ39bo9bEENUdpG2XGOdLvBMmbqfu2IyydeX0GNJmGYc8XOFzSXBt7fwmqafTq3zm0V2KUA7puUXZwwbW4cs0K1AEyU0pCBGsGWe2lYw/zyabolfhdZM4PGg4m1AbmzQA3AnrfxwoQ2Z5b/YGU1f/3GY9Do3h2uNveHzMZVqFuCdDkZDicq6LH7NB7wbWdNlsMIIsaGXmY+OLBkxaJIbLNn40cEI+SUcDtq9zD8ETyQk4QHulI9zeIHs0GTG/AyzeTpByaxU0D2MG+6i2K8nV+9LwJF6DvfMa08BBc532kmGqBAKZX7w58oxrcTIV9MI7wcOBiuZj9VZZgz6F1rFaQmzwaca3e5C+ADNy3K3eiiaYTcHT4DGuSYYnjSwCYO/nlFSEpTjvRuaoyaORWeN7hD4Te3tiwKJACBdRIbmb20V3TAHmlGgUjLsRD/pXpxM8b/wyeLDy71vj4fqGBnTPTtxkMDhQIyeMehoEfa29n7yDp+nJgSSOjC5+jVX7qXsaJD49ne+3ZGdLcCuyNU0NlJMDrCWxGfzPePmQw3hdMitDs5GpBDNKm2h684M2whJirYrZxF5555j9s2GtNl42ZHT6d1K8e8FJMFi5N1qPGesZoHDc75n6uKgHmswHEVYTF4fG7ZPVcMMJ/INcmQ/mX4kPk7B6p4NQ4wRDNzMLMRVKpkVIF6+EP56V7nLk+Ghf8nrLccTV7/+vPw44iUTYrIG+w+d1GzMIhR3uCjzfSKb+xxn77qLPbmxB89xl1ZyLGDhWVzkLkGY/M5W4+d3jNhogyDV85y WgPWBWFi ouVaatFbDJaJou2JMQWnA3nuYAMKxKTsEi6K8DFu2Bck8Igexy8pZ1YKz//aepBgTTWly+IVzEy6OsWy8EJ+myhYipQshAofEj2LcoSFs1oMdYimiqWCV+geK15tnLBocdZa0U2oHIO80PwSAqx8F/Sqe8M8xHRii57KUdKZIwdLcpDr7LBhQxYZBSb/ROZK5F3ud+p3iBrT5lPcmwT5+XOZMz2bqXt65GQag X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Indirect calls are expensive, thanks to Spectre. Call free_huge_page() directly if the folio belongs to hugetlb. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/hugetlb.h | 3 ++- mm/page_alloc.c | 8 +++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 0a393bc02f25..5a1dfaffbd80 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -26,6 +26,8 @@ typedef struct { unsigned long pd; } hugepd_t; #define __hugepd(x) ((hugepd_t) { (x) }) #endif +void free_huge_page(struct page *page); + #ifdef CONFIG_HUGETLB_PAGE #include @@ -165,7 +167,6 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared); void folio_putback_active_hugetlb(struct folio *folio); void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int reason); -void free_huge_page(struct page *page); void hugetlb_fix_reserve_counts(struct inode *inode); extern struct mutex *hugetlb_fault_mutex_table; u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8fe9ff917850..548c8016190b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,9 +287,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { static compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { [NULL_COMPOUND_DTOR] = NULL, [COMPOUND_PAGE_DTOR] = free_compound_page, -#ifdef CONFIG_HUGETLB_PAGE - [HUGETLB_PAGE_DTOR] = free_huge_page, -#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE [TRANSHUGE_PAGE_DTOR] = free_transhuge_page, #endif @@ -622,6 +619,11 @@ void destroy_large_folio(struct folio *folio) { enum compound_dtor_id dtor = folio->_folio_dtor; + if (folio_test_hugetlb(folio)) { + free_huge_page(&folio->page); + return; + } + VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); }