From patchwork Thu May 24 00:58:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10422549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E912E6032C for ; Thu, 24 May 2018 01:00:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D89AD292EA for ; Thu, 24 May 2018 01:00:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CD33C292EC; Thu, 24 May 2018 01:00:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02C14292EA for ; Thu, 24 May 2018 01:00:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DB646B000D; Wed, 23 May 2018 21:00:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 78F026B000E; Wed, 23 May 2018 21:00:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67C916B0010; Wed, 23 May 2018 21:00:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id 27EC96B000D for ; Wed, 23 May 2018 21:00:32 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id d4-v6so15058393plr.17 for ; Wed, 23 May 2018 18:00:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=iwN8nISYsFeJQvNhik795GdHaspYG22RCiPRU1540zI=; b=UThu99LjMNuO9kapOADVjWZoAtjKFCr29X9+d4NB1p2hSqcDYAEplI7jnfDDaI8juO pDdAfLRuIUAwsPpvH+RxRbaIDihfSk8D95uAQt7WwqBmERWCIFohPoVsiTCVSEcJxsSP nUWEXFC75y6S3NK6fAOBN+34QyEcWH1Z779Tl5kFE8RmRkhSjqcL+0k2KCoX9euGIa5F HaAocnejhKmO9z+rFAJAbpBDe/kxbYiA9nLaZXGlkmK16mw2IWUx+HH9x9wWTFukwZbY iuLyP8eyCmHacn6MZvL/vuqISDUIRklEdu9hlwuXB9bIphklhdO7owi9no6m3aRKKSas Hy2g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ALKqPwebiYhvZlORZz6yz9Y4BkAkfd4lnc+/PmorsTSRGPH8Eotqpio1 761C0uS6ZqeRiTFUVXjMJFZO6BDINMivLn+7j7zNhUTFEHRLVBJ4H7qYHQ6MTQgMqaPnFIw0OFZ KCISgkGpVwyaFNWeA+nrM5VkxgspoGvzxWoBAUQooAfAoVd+ize1JgcqM6zfbZFqEYQ== X-Received: by 2002:a62:4f0c:: with SMTP id d12-v6mr4981200pfb.220.1527123631804; Wed, 23 May 2018 18:00:31 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpwoDDItLYY9UUW/qCanOJAJUiLVXaIwQY/LWR4+nBgiHxN4KWseTA67WmqPQRW0XVQYjYD X-Received: by 2002:a62:4f0c:: with SMTP id d12-v6mr4981126pfb.220.1527123630647; Wed, 23 May 2018 18:00:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527123630; cv=none; d=google.com; s=arc-20160816; b=Lmh/FwQfoeZ2ZvFLHBua60gNSlZ9v5Z7BLQge9jB1EMJW4SbFKijJUHe3Bc5YLHGVT iKZyGwch4XRAiRienvGgzDXRzPHxTogzZ5HkZ8KsvYAAbjc6UVc6W3yDa2zGbxsY5h9B LrEmNyaxfsQwXuclWzF8tZUUqqIK11X6SYLgYh9kndJDQ6KcxYQQN4SkxmUHKdPo00Ln 4OiqXFXBY/1PI/qoaUZ3VsdY3esvmMuyl/I3NFtOiSGhztgiid9lbtcces9C912ix2qh UgCIXLDKc/dXJOG/JEkvW6PuWu7bLbUSpr44xkXb4kj6PhiGT+igpCaGB934cCvleVEF cmhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=iwN8nISYsFeJQvNhik795GdHaspYG22RCiPRU1540zI=; b=OPG85DW1SLLPRL63BJGsQ63/dnMOgmp9zTKQ+zgyr/ZzVHOqLmVeShKAfJrWr5rir8 4QNHG3gvWZ+BEZGjtCinZSzvd+CXn3rE6oLmYEu+WNHJ++ZNu+Up6RSqgVqFoxD7nQzC 5wos1XFRq6GRlG7YnC3UB4hUkRo3uhQGjQIKm9tzX0Mz1GxeH8KEEVYujzvvisNHG1SX sdxN2QG0l1AR3b3d7VjbUSrVbmZydStrF//kVOarpZMBTXsuxPNg4EL1IvoPDXkGQC7f pwluNIv8EswhKtF+SiwPNXBq2a0IImD8WaVQwiMggKUgGRrLthZKCfmnF2wrpTmHOYee KJaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id 203-v6si19885295pfz.160.2018.05.23.18.00.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 May 2018 18:00:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 May 2018 18:00:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,435,1520924400"; d="scan'208";a="42362529" Received: from yhuang6-ux31a.sh.intel.com ([10.239.197.97]) by fmsmga008.fm.intel.com with ESMTP; 23 May 2018 18:00:27 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Andi Kleen , Jan Kara , Michal Hocko , Andrea Arcangeli , "Kirill A. Shutemov" , Matthew Wilcox , Hugh Dickins , Minchan Kim , Shaohua Li , Christopher Lameter , Mike Kravetz Subject: [PATCH -V2 -mm 1/4] mm, clear_huge_page: Move order algorithm into a separate function Date: Thu, 24 May 2018 08:58:48 +0800 Message-Id: <20180524005851.4079-2-ying.huang@intel.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180524005851.4079-1-ying.huang@intel.com> References: <20180524005851.4079-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying In commit c79b57e462b5d ("mm: hugetlb: clear target sub-page last when clearing huge page"), to keep the cache lines of the target subpage hot, the order to clear the subpages in the huge page in clear_huge_page() is changed to clearing the subpage which is furthest from the target subpage firstly, and the target subpage last. This optimization could be applied to copying huge page too with the same order algorithm. To avoid code duplication and reduce maintenance overhead, in this patch, the order algorithm is moved out of clear_huge_page() into a separate function: process_huge_page(). So that we can use it for copying huge page too. This will change the direct calls to clear_user_highpage() into the indirect calls. But with the proper inline support of the compilers, the indirect call will be optimized to be the direct call. Our tests show no performance change with the patch. This patch is a code cleanup without functionality change. Signed-off-by: "Huang, Ying" Suggested-by: Mike Kravetz Cc: Andi Kleen Cc: Jan Kara Cc: Michal Hocko Cc: Andrea Arcangeli Cc: "Kirill A. Shutemov" Cc: Matthew Wilcox Cc: Hugh Dickins Cc: Minchan Kim Cc: Shaohua Li Cc: Christopher Lameter Reviewed-by: Mike Kravetz --- mm/memory.c | 90 ++++++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 56 insertions(+), 34 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 14578158ed20..b9f573a81bbd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4569,71 +4569,93 @@ EXPORT_SYMBOL(__might_fault); #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) -static void clear_gigantic_page(struct page *page, - unsigned long addr, - unsigned int pages_per_huge_page) -{ - int i; - struct page *p = page; - - might_sleep(); - for (i = 0; i < pages_per_huge_page; - i++, p = mem_map_next(p, page, i)) { - cond_resched(); - clear_user_highpage(p, addr + i * PAGE_SIZE); - } -} -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) +/* + * Process all subpages of the specified huge page with the specified + * operation. The target subpage will be processed last to keep its + * cache lines hot. + */ +static inline void process_huge_page( + unsigned long addr_hint, unsigned int pages_per_huge_page, + void (*process_subpage)(unsigned long addr, int idx, void *arg), + void *arg) { int i, n, base, l; unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { - clear_gigantic_page(page, addr, pages_per_huge_page); - return; - } - - /* Clear sub-page to access last to keep its cache lines hot */ + /* Process target subpage last to keep its cache lines hot */ might_sleep(); n = (addr_hint - addr) / PAGE_SIZE; if (2 * n <= pages_per_huge_page) { - /* If sub-page to access in first half of huge page */ + /* If target subpage in first half of huge page */ base = 0; l = n; - /* Clear sub-pages at the end of huge page */ + /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - clear_user_highpage(page + i, addr + i * PAGE_SIZE); + process_subpage(addr + i * PAGE_SIZE, i, arg); } } else { - /* If sub-page to access in second half of huge page */ + /* If target subpage in second half of huge page */ base = pages_per_huge_page - 2 * (pages_per_huge_page - n); l = pages_per_huge_page - n; - /* Clear sub-pages at the begin of huge page */ + /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - clear_user_highpage(page + i, addr + i * PAGE_SIZE); + process_subpage(addr + i * PAGE_SIZE, i, arg); } } /* - * Clear remaining sub-pages in left-right-left-right pattern - * towards the sub-page to access + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage */ for (i = 0; i < l; i++) { int left_idx = base + i; int right_idx = base + 2 * l - 1 - i; cond_resched(); - clear_user_highpage(page + left_idx, - addr + left_idx * PAGE_SIZE); + process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); cond_resched(); - clear_user_highpage(page + right_idx, - addr + right_idx * PAGE_SIZE); + process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); } } +static void clear_gigantic_page(struct page *page, + unsigned long addr, + unsigned int pages_per_huge_page) +{ + int i; + struct page *p = page; + + might_sleep(); + for (i = 0; i < pages_per_huge_page; + i++, p = mem_map_next(p, page, i)) { + cond_resched(); + clear_user_highpage(p, addr + i * PAGE_SIZE); + } +} + +static void clear_subpage(unsigned long addr, int idx, void *arg) +{ + struct page *page = arg; + + clear_user_highpage(page + idx, addr); +} + +void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) +{ + unsigned long addr = addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + + if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { + clear_gigantic_page(page, addr, pages_per_huge_page); + return; + } + + process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); +} + static void copy_user_gigantic_page(struct page *dst, struct page *src, unsigned long addr, struct vm_area_struct *vma,