From patchwork Tue Apr 16 07:17:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13631448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1725BC05023 for ; Tue, 16 Apr 2024 07:17:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0D4A6B008C; Tue, 16 Apr 2024 03:17:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BE3F6B0092; Tue, 16 Apr 2024 03:17:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 837706B0093; Tue, 16 Apr 2024 03:17:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5B4776B008C for ; Tue, 16 Apr 2024 03:17:52 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E1B361C0CF4 for ; Tue, 16 Apr 2024 07:17:51 +0000 (UTC) X-FDA: 82014540342.07.00BEEE9 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf02.hostedemail.com (Postfix) with ESMTP id 4774480010 for ; Tue, 16 Apr 2024 07:17:50 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=cX0yp+jQ; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713251870; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0LQlo9tV6jA7edPLPREtk8jPXGN28KNPzfvTbxW9Ygw=; b=zRhyUq8sl02fEcVfJpul+AMikHAwQDNmCAyvDzuVPIDKehSPjtcDjl0A8z9T8SZgt7GUh0 NYSz6WEmcuI5LTo3l7vTkoY5Ttus8eP6TajgWX7EVOSRdH6VAyZfiXnFv/FGsZ1vjrS/qo Mvhhm6C7bUFN6M6WMT4/wEqNnwZXZRY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713251870; a=rsa-sha256; cv=none; b=T34aZeUYZVrdObHR7QuOn1PcHezXYxNt3PtEP/LKHjMLdNNgpSNF6TmLuHtzHoWawi9NLV IV67bHivl2ghtS1xlfxrrlmKmlhKRNaClqznlwLYH9lRpjGF3djfwv6diIn40BgdfbO4KJ lZBRRw4/3pzp5iRegty9dnG/anJGjhg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=cX0yp+jQ; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6edc61d0ff6so3747727b3a.2 for ; Tue, 16 Apr 2024 00:17:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251868; x=1713856668; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=0LQlo9tV6jA7edPLPREtk8jPXGN28KNPzfvTbxW9Ygw=; b=cX0yp+jQYvu2pB0hj3xSECgBzZ5ot9E4yVA8YHY61/kZiD07nT+9wrs1sPaG10IeY5 WR3N97LJ5oVLHOmH0tPWiX0jjL9i9/xeLgJu8bNqv3peEgk/HdJzofaR8un9c0X1x7pl ACIew8zpPjJJCQnuJt6+Q8jShCHxJZKiUpM86pHMIAJ+fs3cox0gp/6wSkUkfFcvBK6b 6SxskMZo30ST5ERkg5sW27X0mDyPngIO1sfb04wW3O3WWELUJBaZEr4qZ8PVoHLgzTbj jdIwTmOhQKzsJuFrEJXl3PQhNKl4ofZPF1GHRtLoJSAx7DTVl+tDb1EfmdYHY1yvcISj wIag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251868; x=1713856668; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=0LQlo9tV6jA7edPLPREtk8jPXGN28KNPzfvTbxW9Ygw=; b=qfe5aXVlNykpbqH7557GVq7+85QO5YEmUZW9eyzioI2Bh+Mjnrr5lOg4O7rWNqGYrW kloGwaA8nOyQHtKjoSRmMmCqiiCl74I+AemOkG7uUKp5v52UA7YAjOJbr1ht/3+t2kE8 JTYEOr7P4pFS5zxFObvaGWvXj5DGtq3P4wC0midsWyejt0GU0HjX/ObVPi7RrZVO4pOQ o3S+QTgLuEzGqidaSG7OnQUaECPj3Aa2C1khntt//UC+BFg+WccNBcLSDfLd+rE6v9vT 5jXpP83+bqgG/ZI0f30l0JGeZkBXYwfqlj/zs53H/+2FZrJxI15sZ910qv/w94C6M86J UE3g== X-Gm-Message-State: AOJu0Yx24kY1kghZICgcFirh70WbC/7kwKEjbWTXC7/4cgKirmoKuTJy qgPhaTTkC23+gz8UJqPWtG5jnQwG5Kf1W0jTnDUowOLm5xFgX6qjk6MAdQ13 X-Google-Smtp-Source: AGHT+IFA2zmy9+KvgrKxSYzQek06HbyAOJH+OqT9zi1O7kh6mqKJgpjWl+AmhEUdNRhj7ZdCRoeZAQ== X-Received: by 2002:a05:6a00:399d:b0:6ea:c2c7:5d6a with SMTP id fi29-20020a056a00399d00b006eac2c75d6amr15984911pfb.2.1713251868205; Tue, 16 Apr 2024 00:17:48 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:47 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 16 Apr 2024 15:17:19 +0800 Message-ID: <20240416071722.45997-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 4774480010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: iyjx1rc7cezrg5fqonh9mzeoyqdynikd X-HE-Tag: 1713251870-962001 X-HE-Meta: U2FsdGVkX1+E7WkkC4BE4us5I3/mbYFfLWa1pgWf2T8sIpGJuTUJlP8fdWwuGKEqW0y4W4myegvu00oGGbBH1+feDHu0JEyoi7qqIHMY2vs05WTSuTBf+y6G738A0O84fmIcXkY94r+B6uHvZGelKDc4JTZBEJneuUx8T6bYyBh6Lg9gpqHfx/0zDv+KcKx5eEDYZl490btnovGY5t+UG5kB476+zYY1Mqxvf6F35DN36VF08MfRJ9IcSbZTVVKqVBsJS9BWAoUmv9uH6j5Zbx+owj/yO8UOnQHIK76C/ZWhHS+B9ppqvGDaTufeKyq0IdzEIKHRHuxpVSfIwnMZdICGrYFPZPkWu7tEdpEgjrZbHqN9azFdcPy2AxwJtMMaNhnQiS/zfddUgLCqpYBh0f4lMA/28CZZJrbqAf2Gp/sTrOunVLcUyAS6LVinrfwCwnZol+lHQX0r50xpoRtVd+SBJH9YhSX0sR9wLI7xNhrJlpgwM618dX2NK6T5Pr3hPAFzd7b8OHlFhwMgt2FZ27BNBMsoCa9XvtIIXd1z/8f3xh8uf/Ubs1OpsaqF8QnwW0Jdl38rD4m1vggcj1x3T1l0AwD6WPbCHAltySwe7e6xLkzGYFPuqzrLaStbMIO+a5I8lInoidsHLZWZ2GB7eWmWRRG9vhIAaWAfrYITnAkZ8KP3C1sEmD0AJLWPUdF4Cc39Sl/FaDAnkz6rb7up4hra+lWPNyLv0B/q3CS38gWroIPT7qDmJiPFMkGYEQTudS1nXXC57/YsX41RcUPg3umaM/glYvVf+FGwLy0TRFvwxldS0oi5NcPgTDW7dgwcgmk1tT8UKdNjeiszw4lK3zripNSBgrWHMZpNwOMcJsrHNC4D9LOtBUZQHVjn93U2i29rLGKzn6RJSi32QrgXSvgw++LKCNVDDiKrx4Cjdz3qzks6ZYT2AsgLZ7BKtCjDL/reSL/D+lAGoDNgDIP 5vBE5UAv U4hZKlvkJt3Mk9KtmuE6R7vv1CazBCHvoqfQG0PSY/twVMgKLcgCxxLlcmovXXNv2ek4FrS5YspJE50i2ZBclLwIeHgg0OIsFJz4ekXbC8pmY4QAos4vfVQJV55d7KrKnS66EB4Z8sk8n83un9Q7n5alIYZI8BvOXdWnpO5M8urMs1hcOU/pFcQl6JTXs5qixQXm/BBkK8soDb5vmTsN2T9UdvZGEjeQKCa26LqVoi1ZcaQiCSZTYtys1cZIFatrruuov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 30de18c4fd28..d447b36ffd41 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, unsigned int order = xa_get_order(xas.xa, xas.xa_index); void *entry, *old = NULL; - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; From patchwork Tue Apr 16 07:17:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13631449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 283B0C4345F for ; Tue, 16 Apr 2024 07:17:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0F086B0092; Tue, 16 Apr 2024 03:17:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 996736B0095; Tue, 16 Apr 2024 03:17:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E9276B0096; Tue, 16 Apr 2024 03:17:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 562E46B0092 for ; Tue, 16 Apr 2024 03:17:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F16E812091E for ; Tue, 16 Apr 2024 07:17:53 +0000 (UTC) X-FDA: 82014540426.09.64FE00B Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) by imf17.hostedemail.com (Postfix) with ESMTP id 2EBCA4000C for ; Tue, 16 Apr 2024 07:17:51 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LPFlcR1D; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.160.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713251872; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F+Z2hNJP+z3T6EfHmF6/rzbCctrFBIWGBJWrf7iP/Ig=; b=X3GrJQIdHWd8yRBhdu6cmwSHXI78GOO3bkW5R3E5Rr3cJJBjYwXl0yrQv1MwMDIF4CHwgw Bi28bh8NgQABL3cTTTRl/aN/HWmoiITQVBMSxZnpyWE9qSSx3hAQfb/jEH+qdXYcDzoRpz b8DNJoFJWQuGOz5jyvz0cA9GZz7cswo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713251872; a=rsa-sha256; cv=none; b=aRHuBl0kq4ZywkdkZYGwZeslyHFTyp7vTm4W0+nCPe/X4Uu9Nya6RedZgWW/x1VQYWPNOm 11SeJ159yuo+JEO58sprvbHFRtafYEfRXDGO1M9AxGIxnnBnHkcjSs/Amw7KnSogGiIngP JD5S1w+gga2nUmRONPuKisFuggAVwAU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LPFlcR1D; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.160.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-22ed72b0082so2724218fac.3 for ; Tue, 16 Apr 2024 00:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251870; x=1713856670; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=F+Z2hNJP+z3T6EfHmF6/rzbCctrFBIWGBJWrf7iP/Ig=; b=LPFlcR1DkKNwgFcVGT+mcoTArg6m4O1dLTN3JRoq87l9GhWWb7thRiatGjToH8aWTX scBtsVJ6DbYjME6DJod/clHRzDszNRbw3TuetD5vkVXyhTaDZJO8bjyL/dzcVmr7XBBY RiYdoGuWjRU0jRcWaIeQAW9ceJu49zL3H5pUM4OljYP4m4F8jw9z/IObDmXBup+uEiEg WZvsZrDy7mpMJ8tkMSLpHK4FCq/De1xa3POhX1xl420wCk84fHe1FIweXK5QCxZ67KFN u9WCDCfwIYyyFk7noxm1HroJxdIrKQgFkEwcYRgF1X+QJLujCSL98iuCCH9sup+syukm J8gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251870; x=1713856670; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=F+Z2hNJP+z3T6EfHmF6/rzbCctrFBIWGBJWrf7iP/Ig=; b=IezYgETq/CCKqOjfbBQkcBRDgfwhEM0oUrHzLs/NVhB8y2I/27OtUh57J5jib4KWs9 0HvMmAtWWvDD1M5mRk1lS/ult5ZQQWJCDQnPDOzvFTfgQOodSwwiLfDAZjk1NnElvV1o qC3/lFpzMgkcaH8MxQo5QkJwxoYBJutySFh38vfg6Sf35UtcTpZAvDAmNJ8/MTxO3fkp FA7zD56NwttLPcGpluqA1uiorbE4Ry9KA6XA0bhBWqIUAeiccZA4L+4P/Uft0hSQJ725 Hk7JfGQgUeHulFNJZNnUTesqpu6/mToupG2NrqkklbhKD3uEWbmK+Em4NMlsp1YlcXlD 03pA== X-Gm-Message-State: AOJu0YxQXeJ2iGoZhUK0I44KEmDU+MM2GBltDLnUDillempf7HqQCo1j 3IJrbODT5Y4Qb1MIsG6FYY0SGB6re0onxoxOWQXPCYFLRIfPMrE71j0m8YIJ X-Google-Smtp-Source: AGHT+IGTlCysBm/YOmaGrEGYXumxXfgrNAPvhr2uKFZUXitXWelGd6XrYs4GaN3Yt1Z5t4vLHkZtFQ== X-Received: by 2002:a05:6870:7312:b0:22e:e0c5:c729 with SMTP id q18-20020a056870731200b0022ee0c5c729mr15807231oal.3.1713251870398; Tue, 16 Apr 2024 00:17:50 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:50 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 16 Apr 2024 15:17:20 +0800 Message-ID: <20240416071722.45997-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: qqks789hiegqaf4k8m9joafodihzmuyx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2EBCA4000C X-Rspam-User: X-HE-Tag: 1713251871-655627 X-HE-Meta: U2FsdGVkX19/PJ/gOJud9bTiLYoSBXq9S5ZU/qGeyBiE4e/or9GlU/v66XiQkR6ksHvZSETxq6OIikcavujUD8yB6dc1TSxIe6auzfkTRDn1I0xcdKimpEIrvBALcf8qXWrUWwsMNKUFprlKj0jlaQJ0kQlyOafO72O6NzbJplJpuG0qHDAlvy2RRsaYGq4NYcdz03miCaw711b8X6UxK7VyrlpqMuqFvL52FJnpGlui8PLlzUCiXAUOKwSaDEgJHwC4WaklTuWw/AkXQEPqTCjgBaPf+UPb+9EDULRb+ebEyBVOwLRG5jFKAFZnhysl6+xBjbVto9CHploBPENMEoxmme+rnn3C0Mm/T/OXxZ1dKUdv5SCBGkCa94umAffVrQMy2+809PRnAJxY/Bj2H0ifdVcLiYCrYL5e8ICZDitT9MWgc3x0aYhf7UxMKvuwVGlSSGIy+q5rmJEQdh34HOhnzkuacjuFIX7TXiEyzvxX2R1CkQPmoHsiqw3tDlsUTagRvm+Wn91ZVCZrUdia/e6195rp+ZZQ3wyrNujCYBsUXVfE/AlVDtnyC24eGs9UWFR1cyiEB2XneE3MiWMtRLgsV4AnBSM96mp7ER4/N9jfAEWCCcsa0OIFZvMXqXnWlJyRGPc8isGNR0WhsKhMWWxLEpSUv3csXAudIx6fEF+LotF2dmrm6WOSEEX8/qGgSh5S650JAKFNPcry/rdt6Nu7LgwO9Btw8AFXputHGuNRFHJxzODEEGuc0I89HSFETgb0QmJtVjt75t1fSidQBcx7+v+FxAVCQ1ilU5qh72Igsm6e0mqR4AwoUduGRKS6GhsilrD9FjH0TMKDmeBPsK+sMjXQwPBn4q5vwWfBsVuqatMxe/0tYfIcMMngjnqx+/DC3pUnntbjJia+dgU6bwSmsPBFrSpsxliilpMZCGEXSyQngN9dSIIhHCzeaYhBiuYT+lBvYTZCdkXO3Z4 D1+7Q81R 5J6bvx+V77tbLkjWNvvCj5fK5OVdEHDqHHOErWVnbSm3GK0jh2d/T8Ejjt72sm2pPbPcTZZc3cPViVaBmWdJLxCK4l/EtjhHcjNIe8w+3B5Mg+f7NA127aeAXWpOCrW/zvsNAWi9eNHkuLd3g1twvADWrs+Gp4JDwQSylHYA6mlhXFTlSDLL5aPnhpNaKXdgOzYIzmfdMjhb7ODE4rcTntRuKfQ2p9yj4asfQdgOjp1Fk0Wgl7RfxwQYOD0cFOUgMi+HrhQgotZayHXGNBAvGnNXERyMbXDieFfNC42HpMA4ONn2yhsc+lxOFZi2DJH/lPNZCOaMsdI5eApX1KsJxfOMqq5f6TnQBsteplvAE7l+HnmcDJuj52oXmcSqF90clzQRs0U5PWcNkWG7qUlE6LR/G/IYMVzOLcI+jkhAB+h19+xRl5mrEFxmeDXFLxztCMcXdR61UkI5CrCMv6zw8uDPAW+SCI2v2c6rufYopC/LdzY0TfDZyN2JHrJwC1Xiq6AsQbu4/HiODXr/HtMhfx5jR+/9yhZbbmUCLqL70YGXZZYqdW8zb7MXWPTWqLOP3uDMe48Ep0KhXrFj7myfv6vHtgioZ5xzIL8vKkVets6KLxa4c9a+ZihKYovGukiFIJT5BMrj8IF3BBcNh6g8/9j4EWB5LMOzQlrw+bNmAunKRxQYe7qdEkFeu/w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d447b36ffd41..7f734b7ad273 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge = folio_test_hugetlb(folio); - bool charged = false; - long nr = 1; + long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); - if (!huge) { - int error = mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged = true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr = folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping = NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, void *shadow = NULL; int ret; + ret = mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret = __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like From patchwork Tue Apr 16 07:17:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13631450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57A7FC04FF6 for ; Tue, 16 Apr 2024 07:17:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 663EC6B0096; Tue, 16 Apr 2024 03:17:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6143D6B0098; Tue, 16 Apr 2024 03:17:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 441126B009B; Tue, 16 Apr 2024 03:17:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1BDA46B0096 for ; Tue, 16 Apr 2024 03:17:56 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D401D140912 for ; Tue, 16 Apr 2024 07:17:55 +0000 (UTC) X-FDA: 82014540510.06.9883773 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf20.hostedemail.com (Postfix) with ESMTP id 36A151C0003 for ; Tue, 16 Apr 2024 07:17:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GRyxKPZ7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713251874; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MHV2+XE+cugEfsKZfQ4KNnLWAEqkFV7dIjX6To9XLTs=; b=JMyZv0clRv0KQ8XK1PPfqhE6FJSghHxwmIFAiw3kt6rjz64Ssxjuet2ok3xpNv8rOYV7D1 dPjQ4Za3sQwekbuPXgULroc71fFPrhmld9XwPthw1CgXSw2uutNvObwlKDOkNy/KIvi6SL 6HniYioouo5fpZlQeOft9UYxPPN/Hqo= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GRyxKPZ7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713251874; a=rsa-sha256; cv=none; b=fRjW3zuKym2InMA9PYLsEG7S7lUDS6+EN3Y2eFhVKcPph7Kw8402SM4JlBbFr5iqIEXvuB 3898Sto7spnJm0ikwSJvJmGdpaxxYKl85Qxlb0zTDe1pyqbsaCPPmDQeNtTGKGSQS+glDJ fEg4PFZyYizbmW4SA8Zb4pc9/0jDaiI= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6ecf8ebff50so2692782b3a.1 for ; Tue, 16 Apr 2024 00:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251872; x=1713856672; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=MHV2+XE+cugEfsKZfQ4KNnLWAEqkFV7dIjX6To9XLTs=; b=GRyxKPZ7oRpt/1LHGMT8pPgYPLzYxsYppYL04HoagE97dPThtQEH1m16ghwgEhHIqo bryteYWNyGOTK2uLytf85UWwFeISDXhusDl1KIyV2UZRE1Dj3XhNNMwkXGHvElFYbzJP Ff6PCBs5Stvm/wnfoc3+d3UWxRp7hYMaufWxQRbZYughlwfzStLR2FTTkBEwRkZ24Xi6 1XcoOSih8mwEJi5+03NaKrhgGESRw677G9toCLubMvokjisWWeWYTunljiAG6+kfj/NL xD1QeWjvFlOWol7H1/52Jd6obP/DjxlwFniWkT0LDcGVz8SaUBvj6vPxcY9mkSt6bBMg A1AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251872; x=1713856672; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=MHV2+XE+cugEfsKZfQ4KNnLWAEqkFV7dIjX6To9XLTs=; b=mXNoJGP8HPdITKkd11mby49aYmTz3Nop2fiVNWgSOvbPrTGJcl/qbjOJHAIanFcGko sUsNRciV69ED11+2T5IYVcb1SXZcZojeaALjHoeUqQG/WRj7+noOgCoZR9pTbQ6BrtXu y3N6lc9GgaDNatoWTGeqfOj48fZHARkabqReSR91lg4H/GZaywuUSHlGa/Ox0Hs3bkZS vALlpUqsQ7auoooFneDFbJRgcfJN4erdR/9JCnHzo8Di3qvFhgmCxoK3qsU/T2noeECD uBkVpaAt19nrp2SJIvlg3BPi+Glo7+Jj6xqHaktcfLufVFP6+D1JBuy++97oPkbW3AG3 VEzA== X-Gm-Message-State: AOJu0YzHDOWLcF3PHnHDmifgB5v7aKR0H/xjQSnwkM8nquQ+7Hsla6gc uySyLR/iYTIYwyun39dWz9w21Lw241IvgCHEsuMbRagOybXpx8aQmODf1daO X-Google-Smtp-Source: AGHT+IESJwVhtcMf3m99N3mRkzrlYgFLHCki4Z3/LDKtQDjTD4+lFVo8Wfkqwv+ogYZO1JOFNTHWzw== X-Received: by 2002:a05:6a00:39a6:b0:6ea:9b37:c288 with SMTP id fi38-20020a056a0039a600b006ea9b37c288mr1644456pfb.15.1713251872585; Tue, 16 Apr 2024 00:17:52 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:52 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 16 Apr 2024 15:17:21 +0800 Message-ID: <20240416071722.45997-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 36A151C0003 X-Stat-Signature: hju37yqnohfri7r3d9njy377kp89c5po X-Rspam-User: X-HE-Tag: 1713251874-484457 X-HE-Meta: U2FsdGVkX18gJ3/oTU2Av/k5ZeGBXsztUEg7hRE/XjTzM2AERQlJDWmjRCW71aluWgXDguj4+vHj1ALvEHrf1f6eihuQiZ0DGfxlbxtkEZA7gghJOkX64dBdUOt9SrDkOej4XNgrAS+FbDs+BZicwxv8XDCqQy7FTo78xD3gEmsqqTT25y3MjBLxdrP7FSooWPBpmjsTP//ij1yqXRoDEcuajdbIe9TR7njpbU5luxkwXdJAkJl6OSX3ES9fAhdlJdd0lfJD1o9lCCIzszu4tLq+OqYCwJdg2VNQkseeFCyn2nKdVDPfklXmc8ejtz+5bGbhqMpcHlXTVDZLSgaccLiPrZInS+JuvuoZBgYUbWJX4heP7xvMV/EPHwK7XKeDLHkU+yH+yEFGXPGS+0PcQNyn4B3Fpe02KOLj7aDwWlkNrGP0+T7K6QXGJG6Uklk2U8Y30/oZwJHbE/1Ggg1+v8RuO5A+zexEsTyWTXwHQE7+80IAhj3MTXGPe7XIIAUTZlK7Q2fdFr/fv/dYcJwwb4lhk540OcHcRDS5uZ+LFB0vjtkgc3h2cBO0dXFB3R0B+vGSsVuYbLW6Sxwfil78/egx6D9GWEgHDqHm6cakU+inFplikhN6bxm/x11jXCmZWLxeHTxNh5YLjaf0+AGYAWO4/giZKlclRJ9UKZ5zdxzSdCRRZJj0nfahRwD8SFZY0+b2K40gFSyoe/9/2rQwTAN8UOlLtaALHbcBaGH5J+ZYsRVG/kXvXQLjqXE2S3bC+4Eq7m8I0D2zZZRmgSN0nHd7YXl0bfSwpTj6jV67RNvwuUmjAULEZ4yoSfARtxhiHvJQ70AN3lYLW8OBhUfkr0h5VbFtul2fcT8Ssyaos4HifM8fGCf2MUpYhgvPOoi6hzkFY7Ynke2BAaqNjz0YUokcDubISyJOiErYBtHaclaZhXIha0ghc7dTkevzPo7C+d5885Jxx9wSada7PDo OpP8gNVg RdO6efxVsg07SWpmqr5EAc0stTq4IKLFqMjkOVv+5HlwFpKHtNDBClbCpmPFt9/SeMJIZ6ZVOAgDgzx0r3nIVONx9qkn9RJsApeEnNlJv7LnHlQYxHcGiSWHguZIzpKv7hX3OOJgiJxRPxvQ/3LBRcACwccvNz9/NP6AyCpgw9YXTBL6ccIC0a4OGbUh3eZqc9wncEg0cRNaQtisYuqqpLSxSoQV7PlxTKBKQo5LKptopimxh1kkULAwrnHT7AQerN6FWbLLT9SsC5nDa71NA0gkSgGViBJDHHGkg0gELc1dSvjUYmuFnAYFwkw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Added new test for xas_get_order, to make the test work, we have to export xas_get_order with EXPORT_SYMBOL_GPL. Also fix a sparse warning by checking the slot value with xa_entry instead of accessing it directly, as suggested by Matthew Wilcox. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 34 +++++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, gfp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, unsigned long index) return 0; } +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..0efde8f93490 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,39 @@ static noinline void check_get_order(struct xarray *xa) } } +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + for (j = i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + rcu_read_lock(); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + rcu_read_unlock(); + } + + xas_lock(&xas); + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2068,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..da79128ad754 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned long first, EXPORT_SYMBOL(xa_store_range); /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an entry. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order = 0; - rcu_read_lock(); - entry = xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; for (;;) { - unsigned int slot = xas.xa_offset + (1 << order); + unsigned int slot = xas->xa_offset + (1 << order); if (slot >= XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xa_entry(xas->xa, xas->xa_node, slot))) break; order++; } - order += xas.xa_node->shift; -unlock: + order += xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order = 0; + void *entry; + + rcu_read_lock(); + entry = xas_load(&xas); + if (entry) + order = xas_get_order(&xas); rcu_read_unlock(); return order; From patchwork Tue Apr 16 07:17:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13631451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45D3BC04FF8 for ; Tue, 16 Apr 2024 07:18:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68D416B009C; Tue, 16 Apr 2024 03:17:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EF536B009D; Tue, 16 Apr 2024 03:17:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 440BB6B009E; Tue, 16 Apr 2024 03:17:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1851F6B009C for ; Tue, 16 Apr 2024 03:17:59 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9A1FF1C1312 for ; Tue, 16 Apr 2024 07:17:58 +0000 (UTC) X-FDA: 82014540636.12.63C8FA5 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf02.hostedemail.com (Postfix) with ESMTP id BA6CB8000C for ; Tue, 16 Apr 2024 07:17:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bKlkqf8V; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713251876; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2PyPfFBA1Tc3D6QFaHrtVUXv3LRzBa54pk/bPz9W1MA=; b=FJWSM5k/cc2lMaTN/VEqEbkrrS2TI9zlRVAkXRyBDHsH0xLa+bDCuL8kdBw8XCnRtojLJN +pfQcYn5m95ibES9kyJ8nSdwK8cWB99h+/6LCZY71sFu2z8HmWELgDOijvDLL70aQS3Jmt lMLpK52MN1rpwJuyf7mtH8D2GVCl8GI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713251876; a=rsa-sha256; cv=none; b=8MP7IT50az48t+UWR6FrVbWOUgoVf8d78I6aLtpHI6+8IyZRhQN08l78DaEUU1E2O2nwbV CX0PqaNVBj4ylxY6nIgGjVPh5JbFyQlwD5eGM4WknNgGhakLI7O/Oj1aFDIjcPqb035WBn lIxzBV319a6G0u7aOQnty7+qCHo8+jE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bKlkqf8V; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6eaf9565e6bso2501096b3a.2 for ; Tue, 16 Apr 2024 00:17:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251875; x=1713856675; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=2PyPfFBA1Tc3D6QFaHrtVUXv3LRzBa54pk/bPz9W1MA=; b=bKlkqf8VCTNgPfi4nTb1mBLvbf7WD+E7o22nnqhjyxwI/QpoGPtu+WihMs1Zv/oP+N ViE5hEixlkJXIj62FgLni8KrMw1ciJI7ncxo/TO+mL9ctgiWjZ8mPKjcHt9DPQWJhWpO zw+QzYJFM74XWIbi9qMd+OzInTfSqM+6dwAT0W/hP0N5DQn7hU7nq3WxQOVEc61cuL5J 2v1QYZ+/Q3HGYbcBpkTRLcKL3V+LMwQIEeKUAKpgj/hh9XstQ+R0rnVXbEunJfjW3uaW aXs+8UJbQMPNJUXkBL0kHVYj6C4pWivx0aqWjdPUQVtVTkwoNAccTfTAWAZ70+2/uRDF DTug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251875; x=1713856675; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2PyPfFBA1Tc3D6QFaHrtVUXv3LRzBa54pk/bPz9W1MA=; b=jbUBOdSRR82zk+dTXlYIw7TRDkt0ii+zV3NKeUVJ0gl+A+SUXYlHTiICXKh4XPo2o0 zbCL8x9j5Tj6ALmqPZnF1tteYwcf1dAVu6cd86GizdPT/W6Gk/6mqYtumnRGTwuaYR7p TwZp//tgA8xfAShp1ifWIZYRXG39NbBjGhFB+fwX/gL0IAuaJd7oasJFj8vqjLE6afrU XEMhSTfsMUmEnIU/O7U6wqxwXsfYAyW/TBG2iW42Uk1aqKxOoZvOaPcqWJ6mYFPOydzp rXuGhdw/ePfXARY4DYv5/yZZzAWJWcUt4ga/1X5Ay6XIdW6hkW0hDJI+QNXPwdTd+ewA r0iw== X-Gm-Message-State: AOJu0YzpjOPwzPs1+Tpdm/E88sE0kX/vbI2e94t6FSYajmUeHVtAyHIf gccfsFSUjXAqlhmoiAcBfDayV0qVp5g12I2DgzCRYo0aI6s5JkgdgsDQiA9t X-Google-Smtp-Source: AGHT+IHr7oN8taByNZNOFuS818KVFfhROjhg9COx9F5A8nzf64zhs0XWSwf3xe5Glor8jesGCLKr/A== X-Received: by 2002:a05:6a00:2353:b0:6eb:4:3f26 with SMTP id j19-20020a056a00235300b006eb00043f26mr13968297pfj.18.1713251874742; Tue, 16 Apr 2024 00:17:54 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:54 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 16 Apr 2024 15:17:22 +0800 Message-ID: <20240416071722.45997-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: BA6CB8000C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: n9kd3mzda67z5myygks77aior96ncmhd X-HE-Tag: 1713251876-870930 X-HE-Meta: U2FsdGVkX1/af9M8kGKBW1NBLrHR922ZJDigkstQKBoBDuzOV3mkTQq9hcC9+b1NEWaERj3eelP0RMIm1XjKFVYIRoE97Q3ihOddxAgAaHhYMAp5TFtj/Sa7mlh9FP9cvzpYHLPaiuG/z6Ql1vvGcTsFwyOrsSGF0RxNwOp770MfpYfCk75xkYMEb9ehodehCnMd5d0KGS+mvYIh4f7ZeSgt5MpCTf5XLNRr71bsm+CVxxkBkN8+AmUQ7U8BWXoIHiIChpCRKg1fI2mfItK9BtpB2PuAoI6KxoiIaH/sF5rxLhMs4pJNqSROh3yIAOYZ2jg84PLKhkUMhCWhUDNMxl7BVKlvQT0MfwlcZyFwtJc1x0oamoj2lmhR6czdPGAQyn/fAMM0wuvij0T4RrdIHZG4CwaJRcP97AGnGh1gNXH3QhJ4stKqT6R37AD7t4tq1VIPsMpDzE/lHOK/S4VIDCAy55gPJu2R/0W1McC2aR4CUXLx4XLpIFku1TCubARvtCJSVIPIe7G5qg+SmOcTsmLuvb4/O3x+BnVu7a6hfFmmx+Yo6sYUnv722r2noz4ngyN+YRio9VoW0r5nNGClCuIKHT/Xv53jF/jfoVrLkufrkf+e58SVD/ZED/+sr50rUHosh44H9jmEN/Q+b2CTOEtL3qXrk6x3OKON/Ry/xbRVoqJPOYxWt+mTIfF8LkhsqaHz+QgsCt7/XU5fTrOyixPlMJdpyJiR4lpe79Hjj8mlw9v/uGAQ3fdk77qCWFU1WBmAlg3gCikVC58PrvJF4tTp5UGA94gcv1LUvURaVI78xoS9fNqcibXmNtu1GslDzBOcDmkQILsu9/iGiSKBlv2O0cQJpXl5sIo8axHLwXJ0sX/Pzl4QCvn2XC1vw3EoDecQZNpQ7cZz9IoabfpkUIQmKnIJY/4cQzheRNT/lNIlqElqzNfU1/uwFP927XCClSt41LjAFuPkYK7ul6X CGgxv2Ss 6X9epPwti1U+FJj95Stl/ZXIL6lImEHW1ub8ap38C595n4l9xo597mtF50BCugheruKDm32ZgfztguBpKEreJttPr1ZbkF+1hynmXrRgXFYQS+pPEJgyNIPRHEqNUjlTriXJ2eMjpe6HvOHAnTMM8mAQQuQqOZLJho/7Qxvkfyf6u0V6m9Ej4QsYrXBRsfIlX0HqmKHOEy0GMXa6XwvoKKFeIDgs7zjBjDQLOXBQ+SfxqM6dNnwI2oueoEEG51mo0Wgc+uo5UtWjhAiZD36Er5DiOjk2+wdTQSNJvXdrCLEBng9wBxv7WojNet+rP2EkqYvGaqJQBNJZGiAVv8houP5q+62Q1t7CoPl/+goTFR4TZJbf4EhWn3UKbfndk9Qhw1cL5KC0ermsz/QW1uun1j29qwB3Mgm9grI9jrygFngteMMIM6Esm3OyQcXdnoMQuBu7hwIDM4D1zL9wIAAiSmGENEZ/ySrMd/FaQaSWGuVU3udGvVmAfFerRkiK2KvrhlT+2o4wTYNm0zecohW9+BSRTROPU3gNB62IWLB/GQjoOJ28fUpRDOeKNJh5Q8IO6cGm6GBUEkXPxJU6m7cqiNamNUySZIqi8/Z9nR4Vp5ulxirTXs+lZ8zrZNBUMaXtRJlf5XdJxiwHfDwGvNIUdp/cfa9I1XbU5AKS+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691 iops : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691 After (+7.3%): bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976 iops : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976ยท READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec After (+12.5%): bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 ++++++++++++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 0efde8f93490..8732a311f613 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2017,6 +2017,64 @@ static noinline void check_xas_get_order(struct xarray *xa) } } +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j = i << order; + for (k = 0; k < order; k++) { + only_once = 0; + xas_set_order(&xas, j + (1 << k), k); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once = 0; + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_lock(&xas); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2069,6 +2127,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 7f734b7ad273..00177ecdc4a4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge = folio_test_hugetlb(folio); + void *alloced_shadow = NULL; + int alloced_order = 0; + bool huge; long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *mapping, VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge = folio_test_hugetlb(folio); nr = folio_nr_pages(folio); gfp &= GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space *mapping, folio->mapping = mapping; folio->index = xas.xa_index; - do { - unsigned int order = xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order = -1, split_order = 0; void *entry, *old = NULL; - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order == -1) + order = xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old != alloced_shadow || order != alloced_order)) { + xas_destroy(&xas); + alloced_order = 0; } if (old) { - if (shadowp) - *shadowp = old; - /* entry may have been split before we acquired lock */ - order = xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order = order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp = old; } xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow = old; + alloced_order = split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } if (xas_error(&xas)) goto error;