From patchwork Tue Apr 25 08:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13223082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C04EC77B71 for ; Tue, 25 Apr 2023 08:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0896A6B0074; Tue, 25 Apr 2023 04:46:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03A606B0075; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E44436B0078; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D58AF6B0074 for ; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A13D3C02A7 for ; Tue, 25 Apr 2023 08:46:43 +0000 (UTC) X-FDA: 80719282686.10.B4740D3 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf11.hostedemail.com (Postfix) with ESMTP id 973EE40015 for ; Tue, 25 Apr 2023 08:46:38 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gSi0OB4K; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682412401; a=rsa-sha256; cv=none; b=oJ1tnlmNla6XQsURcoBGfeZql9qoRU+G7iLpLMfLYn3zpki6lwl+brM/M543bUmi/pJg2+ yq/9RaNHmlrapc3RL66mABjx4L967HQ9ytCZDhRsfGxnQ7Z02yqNQV90qxbfeIi/0gTCI5 BroRLWMKFBcioAzrRE92wYcLLEHikec= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gSi0OB4K; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682412401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NtLUJYvqGFEpl+U/E0TiUclpFyCf3kGdSKtkoK2Z5Zw=; b=MjNmm5YvZzr6CIoWPypMmGej8yGEF6jgI90uwBZG230vsFS4Pw3rHZrO2YQ8AkrLQ9ueR1 CzKk6X80rIjZOJY9bRL8FdqojgNZ4aj1fDomHujYsIiHKzHI7oqwLa3EZZeQmMXVrQIPUp S48ubQDbx78XwVrHXivGJ3ized6+Au8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682412401; x=1713948401; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pk9XCUgoTnR9WpxWD2YhzR7meCtyBUVgtKd3rO4gYZ4=; b=gSi0OB4KIWLgLK+MZE3JL/+332NPyppQMf/R4isk10AOkmaLXg+8Su1m WG/iTI/EUpro+LtAzTW25zacF5cUvVcImKgx3hbCMmOXadhTdpDA49hv/ kPRIKv6Pjp75/V0xS3eyLZeQfzleZg27ld5YI7GqyVKJhOIgoSIuOPQJV N7lLSlr8cVUZQp3NqsP7TwVIU8IT2VF10vFq92NhRLR4wz61dTO0kmF9X n3u5+sU7sWHZUm2tKPOsb89lggCVjDZN6K76VrywcY4/kCse3mZs3jnoU MRTyYkrJPeUopul7wSMU8zsS8jksFppGADsRACLMRPcovZ9FpQN0hMxyo Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="345454721" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="345454721" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Apr 2023 01:46:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="696098037" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="696098037" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga007.fm.intel.com with ESMTP; 25 Apr 2023 01:46:35 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com, ying.huang@intel.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 1/2] THP: avoid lock when check whether THP is in deferred list Date: Tue, 25 Apr 2023 16:46:26 +0800 Message-Id: <20230425084627.3573866-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230425084627.3573866-1-fengwei.yin@intel.com> References: <20230425084627.3573866-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 973EE40015 X-Rspamd-Server: rspam01 X-Stat-Signature: fpsjwo6srk64g87x9wizwxehahud1981 X-HE-Tag: 1682412398-916771 X-HE-Meta: U2FsdGVkX1/rxoPWGVdB8Y8Ad6G20zbmlPHybtPL9DlbYcv8oTdM8aYy5tzpZ+WzO9ebZfXIT5yXjBUuMnRwWguZ7HFWSUPF/CrVgXcCwJRBtSC+qTOUVj8qSP1dPjJbXQLDRCM07fPJVctArT/AK1jFxZK0tugRZK/K7CHRmZ3aczw1u3Y4V1nKGL37EnPblgbQ/Zjhrou4DX4M6arFiIbhmv/roxUhqz7b9UTfykgsv/K8AEGN1QKCn0Ojp+P63bdvcv8sht2h/7utlr640MCXPF/FGzmRO2LzpLneMa9UM3+ZXBSmZtKr/VUKxoocVp92rTSz57BxvcIbxMXEOaxk3JbQivR071Mz41ZFSaS3UFLcddVKzU8fMWE44e9iXzwbHX9vQzJoKyA4DnXiwVgiKq2F+G0Wack9iT+PFVj3a9LQFZ9pEWVCqLqRHtJTInu2QcpnrKkvV1ArNbNwlYmEkQU2DxtcD0aga4DaGMapdxd7YWUF5dhAgZoFkHF5BC9qZsS+fx3ZKy2Puoxj9dhcSZTMN395yNg3t5hGZbSYGvCl1reX0Gb6qr9VVh7NbVGpg5Iwkq3jNnj997xD5SSQJ4V9zqXQlcnd6S1SnwEomFmCjP2b+XDZVL0QI3SmSUh/IAWol4U+nzSBaC3Zi4/7VLAw3J9Fr9NNnm+Ekb6UXjy6BLPiM8YnYEneBrg1yZxDHjMcYvb43NovQCOKovdGNe45sfgnO8kzW5HVUjTBvCmES7lFmooBKYVJX2Cfnv6Hviw2pNclLMrNYb+tlwUw7Bu1Q/WmsskRdcuocSCW0eCUJrSlBCEwWnIuwxfsITu6aiVexsX2Lrqi8MokkNsJEQlCq5e9tyxaPxIXvFGIuQ+WIrW3+vaLi9fYqB5Dw5dBQCPjSoJC4xbsA8XbGNmROGXQhL+K9gM+qrfShh2IjRw+8JPLnCCvjWUlQuo4UgUtEKrSIP+B4f/3r8o VptlO+KY UupqtP0HOp9CLQrmTGjFrmcJHQWHuCybEbcfDtjG1dwHTs9jj1BpgL9CTjRehdQmz+C6T3tlSukMnUSo9nmMAzmrN1ZCprRW/lYvocDxL0p58MFVvlKyNDEDC9WUL6wOOVYme6eA4ZYZ2WKsmYgdFkU2yNGIks1uIZiTrg7tGXJzuJryMVsJNRGg6f7ckpsxMU4FsSjlEK4STs/kdpPP8UtxvkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_transhuge_page() acquires split queue lock then check whether the THP was added to deferred list or not. It's safe to check whether the THP is in deferred list or not. When code hit free_transhuge_page(), there is no one tries to update the folio's _deferred_list. If folio is not in deferred_list, it's safe to check without acquiring lock. If folio is in deferred_list, the other node in deferred_list adding/deleteing doesn't impact the return value of list_epmty(@folio->_deferred_list). Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 71.28% 0.35% page_fault1_pro [kernel.kallsyms] [k] release_pages - 70.93% release_pages - 61.42% free_transhuge_page + 60.77% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Signed-off-by: Yin Fengwei Tested-by: Ryan Roberts --- mm/huge_memory.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 032fb0ef9cd1..c620f1f12247 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2799,12 +2799,25 @@ void free_transhuge_page(struct page *page) struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { + /* + * At this point, there is no one trying to queue the folio + * to deferred_list. folio->_deferred_list is not possible + * being updated. + * + * If folio is already added to deferred_list, add/delete to/from + * deferred_list will not impact list_empty(&folio->_deferred_list). + * It's safe to check list_empty(&folio->_deferred_list) without + * acquiring the lock. + * + * If folio is not in deferred_list, it's safe to check without + * acquiring the lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); ds_queue->split_queue_len--; list_del(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); }