From patchwork Mon Apr 17 07:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13213418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBB90C77B70 for ; Mon, 17 Apr 2023 07:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71BFF8E0003; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CB258E0001; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5939B8E0003; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 48D738E0001 for ; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 187881A0491 for ; Mon, 17 Apr 2023 07:55:25 +0000 (UTC) X-FDA: 80690123010.03.4E528E8 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf29.hostedemail.com (Postfix) with ESMTP id B932512000E for ; Mon, 17 Apr 2023 07:55:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KvH8DLjs; spf=pass (imf29.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681718123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n+Co3Djx/FKBpBr6IJkJ4VOe8yza5ZJvAJ+PK4QALyo=; b=Mm8s3+q2qcOuMnCruo7Is6hM4PaRr2a4NBDoN0xfCge2tlJTj1/TufbZK8k2wd1+9nyuiW XC2t6Z8ZSVHa657L28K1QM5CWkopTmloMmXrKrND6cPcUvnDFSF91PoaLZin03FUYaV0pY NhJI7APTFMxRD8Lo8QWLafmqFe4szDU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KvH8DLjs; spf=pass (imf29.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681718123; a=rsa-sha256; cv=none; b=U+3UOjhGF8w+nqApepMl4U2kJnNT5WRF8Hv4WXb3wk7Zs6uhW7faQfuYy+eLHOoYS/ghlA n2w58DNdbzSoATwvYvLibaCeEqAUYCFPXkcVQZoSN96IHsQWxMlQGehR6e8zuwM852n5V2 4bOr0mZFVBtGOeCDhx4STKIsL9fVLfQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681718122; x=1713254122; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Lrd7bJnxVLV7szkBP4122haJtiLJgqqBdVKx0Hm+jA8=; b=KvH8DLjs+NPj5j/NOyeNe5qmo4okrnel5XkM+oQ6RRk2YBChc2JYCjKm 8cdJcOvBDDgNPOoqmtFvdF7jgOvsNrINRpTiF5ymmFJHuCxC7UoOepH3y 5X/zsK3kWhzONOJsZ8m9AHA55e4BP8rmrqjMu2P8j6B7aNRoThnpWHWl5 APgySE/eZmLk5KRzUXEqs4LoaKLEC8fUns0zyiHZRNkQAgdGYr3uN+vim 5HaAnHzNHvQqNlyIy2eP4dL/vGkdOdeVEyGEq/Y9KXwSa1HTF8rfg0bKU WRFHBUNCue8qFEnRYjr0RjFd/yz+AGGBTF/J23GRD/UzDpVHqQPTjMd8L A==; X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="343586973" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="343586973" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Apr 2023 00:55:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="755226640" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="755226640" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga008.fm.intel.com with ESMTP; 17 Apr 2023 00:55:19 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com Cc: fengwei.yin@intel.com Subject: [PATCH 1/2] THP: avoid lock when check whether THP is in deferred list Date: Mon, 17 Apr 2023 15:56:42 +0800 Message-Id: <20230417075643.3287513-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230417075643.3287513-1-fengwei.yin@intel.com> References: <20230417075643.3287513-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B932512000E X-Stat-Signature: qdfhzhb6gb3zbrbkf6144socw89m9xyj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681718122-583375 X-HE-Meta: U2FsdGVkX1+aUDiJpXGAmjUyLHIU6xKaf6cIhpDOK/MHQelbtKbQv/OEkCwDhZ7JnV9a3M+749rWyB0bVH8DuslBfQZshVrzik2im8nHPl9ZKX2DgWPpnU4ooGLXSRhKrUZJ8G2H1yATFRxhXTCX7RLX41b8je5jSS1dj2AD+L3oDFg09Ee1AZYnX9tdjFoK6SU6C5Nc13FFdedcqn4uQMOkG91R7AkFYTr0H7fNa4YDrmtdCmJEyRh3ykOleOSl9U6ByHgD/28jlWBUvTTaDTZSJVQ1vzBPRXoEV05W9VNYZX7goqYap/kER5BwZQxGCejjYO19qw4gylnP5/YMjZltnNAv+I83EckG8cpIZYRW9KxX7YAMqB3qx8GEM7duJ7SvLkCmQUM5cLrtPBQ8VU0sBBZdaEnX3yZYKCqRC9hIKwvmV/08Un/E9x2lppQVof60xRWoCKwQf/SX9SiGW+Ik2YV5ALRpaxlKeKU4vMcNGng4Kn5qcuIEv1AC40UgO8OVWY1uLTpVCMSQAXhtWzZCRFW94Qq4ZZnlp1AHRDJb61q3w62siqWhDD7NlMMzpA5ychCmHpSNg5wZ7olzr63+Iv9fyyzipUcuSsbXf5ezPfJk1yXPiPa4xriI5NSR/j6IWtjnp5l5fT7I4XFZELuq+yoWMtu5+fUIsthHkGF12NQ1+NNmrK6cejtdeiomWefZgdQl27Lm3L9JtAkxlfNVySnKLBRbrZctI2D7X3YkUC4PBrTy4WwodqSQkZIVFcxCiAXeV7PLWuKaguNnIldrQAHa6Isl+e6MBDrPPX0ovHVBOnHt0dWOiQGHjMMBstNXq/B0l9hnn9kfIr9coUy+P141+snOOZHmhnAaziKUgtajWq5nquuIR0NVpO7B68C7m0ZgJq7zHGR5RAdb5AMnnuSG6pTschbtaxUJyAR0W/yNj65Pav7jaExaoIgOKs05eWnVhE4PeDaq6mp 8rUDNiXe NG5P2wnRD/oXuYmCQ32hKN9DO0/hdiT1r+EoKxeW4UEVJkK+v8cUh8gNy+AiR4sjiARrnQ+c6DKVNe8jFYcUyBh/TwA417M4OaSIaLTwFTLvEzRp0+zDsfDCW4CZj0JyOoFyGKhzYw07wqvUvTrleJetaKqYGA1VM88YBQYPsl1FRlog= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_transhuge_page() acquires split queue lock then check wether the THP was added to deferred list or not. It's safe to check whether the THP is in deferred list or not. When code hit free_transhuge_page(), there is no one tries to update the folio's _deferred_list. If folio is not in deferred_list, it's safe to check without acquiring lock. If folio is in deferred_list, the other node in deferred_list adding/deleteing doesn't impact the return value of list_epmty(@folio->_deferred_list). Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 71.28% 0.35% page_fault1_pro [kernel.kallsyms] [k] release_pages - 70.93% release_pages - 61.42% free_transhuge_page + 60.77% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Signed-off-by: Yin Fengwei --- mm/huge_memory.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b0252b418ef01..802082531e606 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2842,12 +2842,25 @@ void free_transhuge_page(struct page *page) struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { + /* + * At this point, there is no one trying to queue the folio + * to deferred_list. folio->_deferred_list is not possible + * being updated. + * + * If folio is already added to deferred_list, add/delete to/from + * deferred_list will not impact list_empty(&folio->_deferred_list). + * It's safe to check list_empty(&folio->_deferred_list) without + * acquiring the lock. + * + * If folio is not in deferred_list, it's safe to check without + * acquiring the lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); ds_queue->split_queue_len--; list_del(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); }