From patchwork Thu Jul 19 08:48:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10533901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3D2DB60547 for ; Thu, 19 Jul 2018 08:49:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D632295D9 for ; Thu, 19 Jul 2018 08:49:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21B85295DD; Thu, 19 Jul 2018 08:49:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 834D3295DB for ; Thu, 19 Jul 2018 08:49:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 627516B0278; Thu, 19 Jul 2018 04:49:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5D7466B027A; Thu, 19 Jul 2018 04:49:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4ED0D6B027B; Thu, 19 Jul 2018 04:49:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 08EE76B0278 for ; Thu, 19 Jul 2018 04:49:04 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id m25-v6so3297679pgv.22 for ; Thu, 19 Jul 2018 01:49:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GHqJopNG80NmfeM+sLgUkIbKOOIjoyAn4GNjoxJyTLQ=; b=Ims1LJmT7Umy5zZIgkICvFeefd17mxQXWj8mxgWqJ26lmVb6PEilipq8xLqiUyIDmX IA7vIP5T4Gw52+5+4L0oXoXBEyFEipZZ4wFXei8iWaNdUIQ+X6c/Zt9/y8iyGPkPgfg0 WG59iS4h/7I3IQSR+0h1joUCGj7tTOkpAAVD8Xxfy3ISOufd2GzLDoN2e/ymIIacFada CsQ7JCNX2Ms0OGNomJh3RSZT4/6Us0Id1YgLd+eqW0D6x4ATzI5MDdU+mXZ4oQFYL0Wh wN+dVTY1moaKvhW82ghodPSZhot2sGKZRyAh/7ji/XBurcT7IJA4hYnDl8chfCljfIdr 05Xw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AOUpUlGAOY5OxZmtFfpJPdcrPXq15z/2D/QJAOjlzrNtd27B/6twKqlU 7+PJSAN/S5a5ukmZ5mMX9/yyip2o848m24WFnH87cL6FIwW8lVB9MNIWgf9fxutFvdgGJyQmVLF APrEwLVfYog4W4OD1i2VZnU0DPf2d4vv7uk+Uteb+QaKdDGMdRZgI1D83NKQ24Ukokw== X-Received: by 2002:a65:5bc4:: with SMTP id o4-v6mr9024331pgr.448.1531990143709; Thu, 19 Jul 2018 01:49:03 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfCjQTR4DKyi+OgJe4uKdM2m6QB1MXsMUwom61UI56HpvTAHmfTewjeKKEJwJ379hdvHTe5 X-Received: by 2002:a65:5bc4:: with SMTP id o4-v6mr9024277pgr.448.1531990142751; Thu, 19 Jul 2018 01:49:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531990142; cv=none; d=google.com; s=arc-20160816; b=MJ922XaZ5S2KrzVmY/FdOhz9I825+VfY5BcHiB3QazOkeVyOoXS9zChOI682s95KmJ YRd+Vm1WMfUv69G4+SREPk1a2uRMjXYzSZ6/IgHT/wT2F5Qq1UlC8l4eZwPauKqEMxUV 7X+GK/S6Dv+bQLX3dR534JODCKV1dqw3NsgTz+OU77jxSQKkSf2LMrrxeX5tGLn0PFnl 2xRj166bSQVLz9B6PHcM+RX614Kr9/zQDKs2BkY/iQHFembZ7TJzWdc3V6DJuavggKmn CUfUnMl31XuHYOoYc0GpxFhFlz2q8aZFCQDlCfa2qfVsGPYM5E+ztg+M34Do+k+NFGNn 2XIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=GHqJopNG80NmfeM+sLgUkIbKOOIjoyAn4GNjoxJyTLQ=; b=M6heMgeI9hkGFy5JlrN4eli+ySaqYZ30a3ohyNwuXgFAoD5WUSgKyiLjBxLsBP8uZA C/HgP53ih8OnJCWTziyRVnHG+hRDJ/nB5EC7O5W0gcupAf6/mteCwx3MbQFcJW7ZkJwn 4h8A2V7eX+5ynGIiDecnM/rDw/5EFlxqG/s3tPyHyKzJGEjYT5+wfM71F323BraBhZSy T1g+yndbG2reuNTRjJVAvB3mTwvj4aTAKcZ3piFprTQY54jKvWp7u5d2r6yojIT7Ys9m 7AU9bCK3GAmWVmhTDXxCWVrHrngQr+MpdDt8bkksDVhh2BYM0/wO6CHD2xJjmrjrhZDn e4eA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id x18-v6si4948122pll.193.2018.07.19.01.49.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Jul 2018 01:49:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Jul 2018 01:49:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,374,1526367600"; d="scan'208";a="76038305" Received: from yhuang6-ux31a.sh.intel.com ([10.239.195.246]) by orsmga002.jf.intel.com with ESMTP; 19 Jul 2018 01:49:00 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dan Williams , Dave Hansen , Daniel Jordan Subject: [PATCH v3 5/8] swap: Unify normal/huge code path in put_swap_page() Date: Thu, 19 Jul 2018 16:48:39 +0800 Message-Id: <20180719084842.11385-6-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180719084842.11385-1-ying.huang@intel.com> References: <20180719084842.11385-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In this patch, the normal/huge code path in put_swap_page() and several helper functions are unified to avoid duplicated code, bugs, etc. and make it easier to review the code. The removed lines are more than added lines. And the binary size is kept exactly same when CONFIG_TRANSPARENT_HUGEPAGE=n. Signed-off-by: "Huang, Ying" Suggested-by: Dave Hansen Reviewed-by: Daniel Jordan Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dan Williams --- mm/swapfile.c | 83 ++++++++++++++++++++++++++--------------------------------- 1 file changed, 37 insertions(+), 46 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 96018207b582..407f97287ab3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -205,8 +205,16 @@ static void discard_swap_cluster(struct swap_info_struct *si, #ifdef CONFIG_THP_SWAP #define SWAPFILE_CLUSTER HPAGE_PMD_NR + +#define swap_entry_size(size) (size) #else #define SWAPFILE_CLUSTER 256 + +/* + * Define swap_entry_size() as constant to let compiler to optimize + * out some code if !CONFIG_THP_SWAP + */ +#define swap_entry_size(size) 1 #endif #define LATENCY_LIMIT 256 @@ -1253,18 +1261,7 @@ void swap_free(swp_entry_t entry) /* * Called after dropping swapcache to decrease refcnt to swap entries. */ -static void swapcache_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p = _swap_info_get(entry); - if (p) { - if (!__swap_entry_free(p, entry, SWAP_HAS_CACHE)) - free_swap_slot(entry); - } -} - -static void swapcache_free_cluster(swp_entry_t entry) +void put_swap_page(struct page *page, swp_entry_t entry) { unsigned long offset = swp_offset(entry); unsigned long idx = offset / SWAPFILE_CLUSTER; @@ -1273,39 +1270,41 @@ static void swapcache_free_cluster(swp_entry_t entry) unsigned char *map; unsigned int i, free_entries = 0; unsigned char val; - - if (!IS_ENABLED(CONFIG_THP_SWAP)) - return; + int size = swap_entry_size(hpage_nr_pages(page)); si = _swap_info_get(entry); if (!si) return; - ci = lock_cluster(si, offset); - VM_BUG_ON(!cluster_is_huge(ci)); - map = si->swap_map + offset; - for (i = 0; i < SWAPFILE_CLUSTER; i++) { - val = map[i]; - VM_BUG_ON(!(val & SWAP_HAS_CACHE)); - if (val == SWAP_HAS_CACHE) - free_entries++; - } - if (!free_entries) { - for (i = 0; i < SWAPFILE_CLUSTER; i++) - map[i] &= ~SWAP_HAS_CACHE; - } - cluster_clear_huge(ci); - unlock_cluster(ci); - if (free_entries == SWAPFILE_CLUSTER) { - spin_lock(&si->lock); + if (size == SWAPFILE_CLUSTER) { ci = lock_cluster(si, offset); - memset(map, 0, SWAPFILE_CLUSTER); + VM_BUG_ON(!cluster_is_huge(ci)); + map = si->swap_map + offset; + for (i = 0; i < SWAPFILE_CLUSTER; i++) { + val = map[i]; + VM_BUG_ON(!(val & SWAP_HAS_CACHE)); + if (val == SWAP_HAS_CACHE) + free_entries++; + } + if (!free_entries) { + for (i = 0; i < SWAPFILE_CLUSTER; i++) + map[i] &= ~SWAP_HAS_CACHE; + } + cluster_clear_huge(ci); unlock_cluster(ci); - mem_cgroup_uncharge_swap(entry, SWAPFILE_CLUSTER); - swap_free_cluster(si, idx); - spin_unlock(&si->lock); - } else if (free_entries) { - for (i = 0; i < SWAPFILE_CLUSTER; i++, entry.val++) { + if (free_entries == SWAPFILE_CLUSTER) { + spin_lock(&si->lock); + ci = lock_cluster(si, offset); + memset(map, 0, SWAPFILE_CLUSTER); + unlock_cluster(ci); + mem_cgroup_uncharge_swap(entry, SWAPFILE_CLUSTER); + swap_free_cluster(si, idx); + spin_unlock(&si->lock); + return; + } + } + if (size == 1 || free_entries) { + for (i = 0; i < size; i++, entry.val++) { if (!__swap_entry_free(si, entry, SWAP_HAS_CACHE)) free_swap_slot(entry); } @@ -1329,14 +1328,6 @@ int split_swap_cluster(swp_entry_t entry) } #endif -void put_swap_page(struct page *page, swp_entry_t entry) -{ - if (!PageTransHuge(page)) - swapcache_free(entry); - else - swapcache_free_cluster(entry); -} - static int swp_entry_cmp(const void *ent1, const void *ent2) { const swp_entry_t *e1 = ent1, *e2 = ent2;