From patchwork Mon Nov 19 13:48:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10688679 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 949D31709 for ; Mon, 19 Nov 2018 13:48:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 839C829627 for ; Mon, 19 Nov 2018 13:48:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76CB92964D; Mon, 19 Nov 2018 13:48:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B16429627 for ; Mon, 19 Nov 2018 13:48:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D8C76B1A39; Mon, 19 Nov 2018 08:48:43 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 688FF6B1A3A; Mon, 19 Nov 2018 08:48:43 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A0956B1A3B; Mon, 19 Nov 2018 08:48:43 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 18C3A6B1A39 for ; Mon, 19 Nov 2018 08:48:43 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id h86-v6so21473843pfd.2 for ; Mon, 19 Nov 2018 05:48:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=nf42ix51hpPXW0zyS9CAdJ7pbNPaGsC2WNOSHLy9hWo=; b=jbcxicTc9LFDTOGkeAmSwLoIKvWPtI36DvATW3sMX4TFt9R+J8pApwbA+dmyRL/VQR U8iCXXc9aGRY6s6vMcn5ENH9OJmV2EGZHymBq6oI8B062S7wsdXR0FlQQ6MxB6iRL1Oq ZOOB6SlOoqbx+Im0clcwIr0Zi7X5UB86SLOAdBaNXVMixtXYYVKaW643thfkIKOzq9un ldpu2vXFWjiStgMzBv2u3ABndq/il1SQFY24NytvNct5pic5LAEYcM9AegAv89Yu4RXy G7N5PXi6KIXM7wkoRqO/SECZVj8SCe+8n0f0hk1u3Qd1q+f698KHgK/VyNpIlbVzsxef d/eg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gI1/jicjv8z/90OvUK4NJYaEyby4R44cGdVLoLV4IiQD4lrkB/j 5+E1jYl+mznYQkvrV2EDFdbD401eHf2qMxZgf1O6nMPx1ZnU4Ve7KiR4ivcMNfON1dVTNUSfdyz 86tX+zkZbcJYEiki9a8Gp2+rSLsBNuwC11WAB5dVCYYgqcNTs41icnZQ2oby6ItU1pA== X-Received: by 2002:a63:86c2:: with SMTP id x185mr18581702pgd.230.1542635322759; Mon, 19 Nov 2018 05:48:42 -0800 (PST) X-Google-Smtp-Source: AJdET5dPvZXIiciCTrIwFlAhrGHrm3AX9lizsDjbBDelbQfCPczB86Cdj6acHIUUIqDjTfJySo4o X-Received: by 2002:a63:86c2:: with SMTP id x185mr18581671pgd.230.1542635321673; Mon, 19 Nov 2018 05:48:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542635321; cv=none; d=google.com; s=arc-20160816; b=jwb1Jep9b7xfN+zs687p6UkSUHsk+puJpvHrVyuai4uJXBbNfpB++udv6ay8z0WAk0 rcoZ2XMYt8X1bXv+zNHSVqJsgSKuFCoPsjg6WjpBN5mvvNHOSEE7i1WmSAxEtePjtpOr +FUzRVi1VJUSFcK3FDcewZ3+sBQoWOHniT8gA7u7yWsVmBEw0Db/HUOQ8NFF1qPYWLH5 eesKnt/d9GkFnOW738oj4M0ImHTTr1scesfDZnh4C3SWXjZtyt+RizYEIUfL4uieWuzq ZjdCTINDDi/HqvC+95LXg3/Hxb0vRaWmNjaexGgAFX4zAyWcZKKe5VddwqaCW0GVyL5y AhAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=nf42ix51hpPXW0zyS9CAdJ7pbNPaGsC2WNOSHLy9hWo=; b=hbY7/oRBx3A39MuuZMQTGL5e0R1F65OFTAkKCM6UpfMXWF/dyx7FjSLcwTD48kYth5 XJVAWaQzmKKo3Ch+OvCsvnZEh+xY5rxSC6bhoOu76yl0GMJMiOm8Q4IpUqvDzG3B19Iq k5m3f2LH3HGHTypHqo8ukSx66n31MGSbA2rV8VpFE4vM0yAtFfscNUf/+Dhetl+BOk1V v5As88HhX74mPI9upMIZ8jw4wmYUyO1g7Xel7+QiWqVbjjYnpCMpaohJiU5+wLVzomLT XWfuA09TRe7MutmAbV7IJu7s8KFXSb1WaTRs4UoT7x7xP3Eo+wwV5fJ4F3EZi3mOEWjC OHUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id v189si38253742pgb.398.2018.11.19.05.48.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 05:48:41 -0800 (PST) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 05:48:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,252,1539673200"; d="scan'208";a="87075895" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga007.fm.intel.com with ESMTP; 19 Nov 2018 05:48:37 -0800 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Andrew Morton , =?utf-8?q?Pawe=C5=82_Staszews?= =?utf-8?q?ki?= , Jesper Dangaard Brouer , Eric Dumazet , Tariq Toukan , Ilias Apalodimas , Yoel Caspersen , Mel Gorman , Saeed Mahameed , Michal Hocko , Vlastimil Babka , Dave Hansen , Alexander Duyck , Ian Kumlien Subject: [PATCH v2 RESEND 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free() Date: Mon, 19 Nov 2018 21:48:33 +0800 Message-Id: <20181119134834.17765-2-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181119134834.17765-1-aaron.lu@intel.com> References: <20181119134834.17765-1-aaron.lu@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP page_frag_free() calls __free_pages_ok() to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently. Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator: mlx5e_poll_tx_cq | --16.34%--napi_consume_skb | |--12.65%--__free_pages_ok | | | --11.86%--free_one_page | | | |--10.10%--queued_spin_lock_slowpath | | | --0.65%--_raw_spin_lock | |--1.55%--page_frag_free | --1.44%--skb_release_data Jesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2] I thought that __free_pages_ok() are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that __free_pages_ok() here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3] The free path as pointed out by Jesper is: skb_free_head() -> skb_free_frag() -> page_frag_free() And the pages being freed on this path are order-0 pages. Fix this by doing similar things as in __page_frag_cache_drain() - send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page. With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. Ilias' test on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11% performance boost testing with 64byte packets and __free_pages_ok() disappeared from perf top. [1]: https://www.spinics.net/lists/netdev/msg531362.html [2]: https://www.spinics.net/lists/netdev/msg531421.html [3]: https://www.spinics.net/lists/netdev/msg531556.html Reported-by: Paweł Staszewski Analysed-by: Jesper Dangaard Brouer Acked-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Jesper Dangaard Brouer Acked-by: Ilias Apalodimas Tested-by: Ilias Apalodimas Acked-by: Alexander Duyck Acked-by: Tariq Toukan --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 421c5b652708..8f8c6b33b637 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4677,8 +4677,14 @@ void page_frag_free(void *addr) { struct page *page = virt_to_head_page(addr); - if (unlikely(put_page_testzero(page))) - __free_pages_ok(page, compound_order(page)); + if (unlikely(put_page_testzero(page))) { + unsigned int order = compound_order(page); + + if (order == 0) + free_unref_page(page); + else + __free_pages_ok(page, order); + } } EXPORT_SYMBOL(page_frag_free); From patchwork Mon Nov 19 13:48:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10688681 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8CFFE13AD for ; Mon, 19 Nov 2018 13:48:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DC0F29627 for ; Mon, 19 Nov 2018 13:48:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 721572964D; Mon, 19 Nov 2018 13:48:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDF5529627 for ; Mon, 19 Nov 2018 13:48:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1FE66B1A3D; Mon, 19 Nov 2018 08:48:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ACF286B1A3E; Mon, 19 Nov 2018 08:48:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 971806B1A3F; Mon, 19 Nov 2018 08:48:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 499B16B1A3D for ; Mon, 19 Nov 2018 08:48:45 -0500 (EST) Received: by mail-pg1-f200.google.com with SMTP id 143so17846355pgc.3 for ; Mon, 19 Nov 2018 05:48:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Z+HWNC0yQApwBs3IwxspLcXteZviV+rk4Bcuf0ByWrU=; b=SWxwUtcIhIqR+gfqEDb+PN4l2+pWdYnArkB0jiIb7LnSHwtw4vfHgKj8OmklIbo2iQ 6B0G9w+FHjnbBXe5YAgPozYVUZAhqDHzffakUH8GGZWuh/fCdEY4JeTpvPJJe3b8EvNz 8/bGfFESNsEMK3WfSjToKFaF/V8Be8MbamvSrZDuA70bBYIlAC2L+3cD3ZiHPOYe+s+Z UQ9JB/1YOBw4GWuPBYO/G1AxuxjsXrkd1Gn9JQOlSOAgt6ox0ycloTy+/8fFtHp+yvUK AWQx6YitM76mmi/rAliLm/GRWy3h/+ByYLO/f/iaFQNePif7j7oUWbrEkz83cis2Wo8O LvLw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gIXFYHtGb+2vsWZ13rjtyU29p5TQxzEhYov7oQGG4STWWcEuuWF xTmU0xFmBUcpwyxj8TYAuP3Kb89x/JOZpv30Q4JSWlHhZglD5ijTjaJ6BNU5CP2m4NVo/e3DZwI 2GUXT7OapC3wibXV7V6ndxPfeJT1Fc9EvEGc0iFuwP0vvhQiI9b9pV888MaM9LJjJPg== X-Received: by 2002:a63:1d59:: with SMTP id d25mr18106897pgm.180.1542635324982; Mon, 19 Nov 2018 05:48:44 -0800 (PST) X-Google-Smtp-Source: AJdET5dLE/5pxtUvTGU147gXaOX0w/4hJgsusH1aREM8AppRpqEbLaRo5NePZEyQtHusTNZMH5e+ X-Received: by 2002:a63:1d59:: with SMTP id d25mr18106858pgm.180.1542635324316; Mon, 19 Nov 2018 05:48:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542635324; cv=none; d=google.com; s=arc-20160816; b=LF/J/WcWtnxdD7S124gB4zu1H3E/PICjjnFujgaf2g11gWuIe/5HnOi/w8VEkID1pJ aAbABOdvCMhCXSjPAXyZg+bDovkcoHLLnQHa8Qs7yeMsqjfDqpI151xUU4AfDt2tNnpE v9U5XoDCQlahPSULUQ2CAxSSWqcm/RjBxxuIP4RjvAD3w8mfuGcxBOOwLSZ0w8AhA2Qw 7UtEQP2ZAPcJsnfCDXdSdLpBXoWMY4aSSzFosVxt+IfrhK+expCdDDgIA/D474DcWmMn EvRQYWD4xjZX5p8EtITgKvS3frb73jWENRhO0we+4znl5IiRl8ZUe15RQfa0ViAuiyFj yv1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Z+HWNC0yQApwBs3IwxspLcXteZviV+rk4Bcuf0ByWrU=; b=Yh14tHzh3cigtpjcFVPHV+rFB8gBUuzPbBNXVV3wMSsVKZXkQ0BY8kMxWF7MFIIXXC AjND32W8tSVBAAkhxYiNVwlPwZxrd749GfxiyjnfRGLJo5h6DEq7aEI7wqPwscIcukaU czHDCjAb9bC7T+U0ZrMVRW/sOjNxSec7ltb9PT2j4WunGRM53meU46F9l9Dfd3bZ+PMB 1lK3DzIwsEKZKnRQO8jNJuX+vztNcLkiRQbpxxvJVEod+8GSwJSUvodUNMjSqABfUcVL rQ7IxOQBwmejcyLvKj/011wiZEuaAl2NvWsh70UuhVuM9K7MThYt0Z+did4dZph8qm3p b5wA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id v189si38253742pgb.398.2018.11.19.05.48.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 05:48:44 -0800 (PST) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 05:48:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,252,1539673200"; d="scan'208";a="87075901" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga007.fm.intel.com with ESMTP; 19 Nov 2018 05:48:40 -0800 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Andrew Morton , =?utf-8?q?Pawe=C5=82_Staszews?= =?utf-8?q?ki?= , Jesper Dangaard Brouer , Eric Dumazet , Tariq Toukan , Ilias Apalodimas , Yoel Caspersen , Mel Gorman , Saeed Mahameed , Michal Hocko , Vlastimil Babka , Dave Hansen , Alexander Duyck , Ian Kumlien Subject: [PATCH v3 RESEND 2/2] mm/page_alloc: use a single function to free page Date: Mon, 19 Nov 2018 21:48:34 +0800 Message-Id: <20181119134834.17765-3-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181119134834.17765-1-aaron.lu@intel.com> References: <20181119134834.17765-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are multiple places of freeing a page, they all do the same things so a common function can be used to reduce code duplicate. It also avoids bug fixed in one function but left in another. Acked-by: Vlastimil Babka Signed-off-by: Aaron Lu --- mm/page_alloc.c | 37 ++++++++++++++----------------------- 1 file changed, 14 insertions(+), 23 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8f8c6b33b637..93cc8e686eca 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4547,16 +4547,19 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) } EXPORT_SYMBOL(get_zeroed_page); -void __free_pages(struct page *page, unsigned int order) +static inline void free_the_page(struct page *page, unsigned int order) { - if (put_page_testzero(page)) { - if (order == 0) - free_unref_page(page); - else - __free_pages_ok(page, order); - } + if (order == 0) + free_unref_page(page); + else + __free_pages_ok(page, order); } +void __free_pages(struct page *page, unsigned int order) +{ + if (put_page_testzero(page)) + free_the_page(page, order); +} EXPORT_SYMBOL(__free_pages); void free_pages(unsigned long addr, unsigned int order) @@ -4605,14 +4608,8 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) { VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - if (page_ref_sub_and_test(page, count)) { - unsigned int order = compound_order(page); - - if (order == 0) - free_unref_page(page); - else - __free_pages_ok(page, order); - } + if (page_ref_sub_and_test(page, count)) + free_the_page(page, compound_order(page)); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -4677,14 +4674,8 @@ void page_frag_free(void *addr) { struct page *page = virt_to_head_page(addr); - if (unlikely(put_page_testzero(page))) { - unsigned int order = compound_order(page); - - if (order == 0) - free_unref_page(page); - else - __free_pages_ok(page, order); - } + if (unlikely(put_page_testzero(page))) + free_the_page(page, compound_order(page)); } EXPORT_SYMBOL(page_frag_free);