From patchwork Mon Apr 15 17:18:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13630432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56C01C04FF9 for ; Mon, 15 Apr 2024 17:19:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D570F6B009D; Mon, 15 Apr 2024 13:19:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D07486B009F; Mon, 15 Apr 2024 13:19:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCEE06B00A1; Mon, 15 Apr 2024 13:19:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A1A666B009D for ; Mon, 15 Apr 2024 13:19:17 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6D49DA05B9 for ; Mon, 15 Apr 2024 17:19:17 +0000 (UTC) X-FDA: 82012427154.11.793FDD7 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf21.hostedemail.com (Postfix) with ESMTP id C67EA1C0018 for ; Mon, 15 Apr 2024 17:19:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=juV33nph; spf=pass (imf21.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713201555; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B+L8KNGbsq36FmjcGvQ6kYVRELiG3WmiEI53/0190Sw=; b=Xjn7jQvnMM2PajgOCVVaVdnnEl7MlMnSjfGJpcgMeoRfbQuFXsZ8dfdHgJ9nmMowQir0WI Xbl/7IoW1G979p6yowxzg3eJ+3G9iuyP1uztVAXamQfTT1XUqVGgIJXAp7dlt74vjNKkdg Z/8kpiPbnjL5njaacJyml7bFlAyDflQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713201555; a=rsa-sha256; cv=none; b=EFV6ilVfNTBMmcly291wpJ4BIupRO0uynrAOPEckCQVuNLhrWCrLBp6s+kJOqG+OzcWvHM SgWeUwmIEpSVFUo7ytKnrMZBBNZMsUUj+tbUemBfXY/ZFC09nnSRVRdiV3Ml7+7hFj6xNP afnVIQ1T+IDMdyYJ0XZTkjsR/GH9Ep4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=juV33nph; spf=pass (imf21.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-5e152c757a5so1822087a12.2 for ; Mon, 15 Apr 2024 10:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201554; x=1713806354; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=B+L8KNGbsq36FmjcGvQ6kYVRELiG3WmiEI53/0190Sw=; b=juV33nphBL9IDeBbhyN/yVF0c/j3+yKPZ+ASegxU4b0YHUhmKJwJuocQRtnYGodle0 ubA4kvVrOuiVorO9WuzLkCqVVpj+NdyKLwKTGQ6PscKg32SWgpsEdh7GpwwWcl/EWWqi wWozadi0ugd3RufvO6PqxaOqfmFlJjd2Yusfxam7M4O224QD5tYKby1DBSl6UbnjnsCU XxojdSWa/AaGSZMobVjXnZnu6NrQjWA6IqRbizVYch0PzsRkPQoSPaJPUhvM0Q+6UlYg N+0bpU3Xli8TwkCS6owbJZjqnLkzpGiNAzRULElC2XnvP6bK+P2qySPzEQQxRX0KLYAD lwWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201554; x=1713806354; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=B+L8KNGbsq36FmjcGvQ6kYVRELiG3WmiEI53/0190Sw=; b=LLIcaI4SBo4oxlbKYMIrNfEJZXivWxEsWRQqaruk2f8yVwv2cyffFrkoc+WUPzp0ft M5Y7Q9huqBWHzHDpeWvN4i3N0RIwOAOwDnDyjwuTq+vP27zwyybJu6A9meBaMvuCKd1V wgZwnvET6Zpp4qCi/52LwuLNRjl2CfXSdB3oNE65yfSVG89CwGB2Vu+whGuOOnRsAWNU NjqNdIhrbM2qW0loBM06YsJ2L/HfdMqkjqeiQChNsDHcvkq7taucBk9qJAzcFfWID8Pu Iubi2tVurbsbKq76XyqngEwDoaUCWzuTWR4IgfaWRBxglLRVVcXqldMl8QDAc6YSB2cw dU7A== X-Gm-Message-State: AOJu0Yz0RD8TEJAEZNvEgYsuChNT/bUhOGXJ48rpTgSg8vX2cT4+6Jia TDygtocibLQ8ukFhctQX1BywrqvksfKlf+FkBwZYN9g0An0iJ7VkyCzkmhWY X-Google-Smtp-Source: AGHT+IGZsDnsN/0G69F9r9pl4nCedE8as+ni4AOnw741cJ5yApDTX6VIYEFjdIjzcaV4wIkNMeB2jQ== X-Received: by 2002:a17:90a:d998:b0:2a2:b096:f1b2 with SMTP id d24-20020a17090ad99800b002a2b096f1b2mr7627869pjv.32.1713201553844; Mon, 15 Apr 2024 10:19:13 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:13 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 16 Apr 2024 01:18:53 +0800 Message-ID: <20240415171857.19244-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: C67EA1C0018 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: gcmawsyfy4eo5kgj6qnkiuzhgfuhaiye X-HE-Tag: 1713201555-830924 X-HE-Meta: U2FsdGVkX1+WAbSe6dSDJ2xvGEK3riG37OojQcDDvqeGPSwfYZ4t0SD0OK6AM18RLrZPDjhsnSzm+Fyv0YB/KHftiAPyzcAx8zywZtD9vsaIYNuGXfCt2bbLKVcO6dAf5e4O5w/B5oEDBF5CG1J763l7seMTKlJfqFGEWRBwv70oPdW5ai/1wxQgiZvucppZQP12OEx2V4Fn5hhAqAf+rznc5HSesL5olTlV7BjedjXxuwEaQaCpEfeJ/AiIepmSbEACCi4jx/3xblzV3E3TzFL7GewGdG3U6T2/oe7jUUyZOQJMgOftdWtE0K5eHl3JaUlCEMGe4DpQ3E4Hfb79Av1rQ8hTB6vNd6JY97NND2c72rUU967W1NIwfoHeUxg2KMxBmGan4iBkA0cstX5tijPgzN8YF8KrkNsT3GDce96BVqMKtw4d5tEWUqxJMBcKQpHrY4Cpa6yNeUH79s7Fj5l0/LYJaTOQCggZw66kYM2NpLzfxvWtVGGv9VSKBX4lNeefFflcEeQJCWJdZecpJy72f/HKXt84Er2/Dv1H2lFHep18L+lBMnBdRPnNJI8G57lC9bkS0TJhLaRPwRmb08B22QxinyIQrJ8xNEw/hiUYTnVNWDZMbRTuW/EdNAaFCtU8bq52GqhSI3iy9TZtuV1fdTw62NTU2spHWBFPw/SvrmUjWuJ49xvBsajkxXBaAdOZbQiKWyIyNVZBSwUQncAkruFvWZAK9VVZkaI86Gau8v+LRIFisnkK47SdHRClUI+mQlrkN6YRu2YmjBcJgTveszPnAcB6Iqtv0ZUc/+CEamFFotpJwCSYUA6PL3iTFhscZUOjcxCIETMsnGPiL7EQvwJjEImX5rzkTf5W1djd+jxVOv4UhHBPoo3kcCRQacVrD0JD/t+f4I4TQYHBs0CtHny7T5DxW/l1YUTJtrvM2eUPbAbUCx/45VGvTj4nIcT7FRPTa+uQVoAIpdE XUdOHrRt WeE+iLnsrf41BNRzn13Lv8svgnxEgOvlU/i0UDv6CyL1+XO2oudL6wkph4g72qVk51jsMv7PhQcVLAXsw8Oel+IXP48JEmHKq8sAPC2Ad8BNPEcz5x5c/Jan4pRg3No1ttUik+MXuo6qaS+NBKihLWjx4q1BIHVic3gqD3VVv5qHGRbz/6L7swap2+97gv0DxrWFehEp0WP4ylJsPnz+Z2Oe3r0L/k9nf4PxLFMkffAXi1f/V4j1b6TGtXmV4ItyZIXnprJYQUw4q6WVTWRSzPd9B42593+/mpUI66XJeP0qAWh34rVTkZJ8XVFMWb4tqM3cFLRo6DLpi3tE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 480ae6589803..12089c24abfb 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, unsigned int order = xa_get_order(xas.xa, xas.xa_index); void *entry, *old = NULL; - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; From patchwork Mon Apr 15 17:18:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13630433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0B98C4345F for ; Mon, 15 Apr 2024 17:19:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D76F6B00A1; Mon, 15 Apr 2024 13:19:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 938416B00A3; Mon, 15 Apr 2024 13:19:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B1DE6B00A5; Mon, 15 Apr 2024 13:19:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 522856B00A1 for ; Mon, 15 Apr 2024 13:19:20 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E08EC8071F for ; Mon, 15 Apr 2024 17:19:19 +0000 (UTC) X-FDA: 82012427238.30.E2D69F4 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf06.hostedemail.com (Postfix) with ESMTP id 137A618000D for ; Mon, 15 Apr 2024 17:19:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VARa2z3w; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713201558; a=rsa-sha256; cv=none; b=47HN757w9xT2mXsFJCKCeYwPecCG6Z866uvpkwoM/Xf/acnYDR+ex3BCAO5yVEqNMFAzIB vsuq/VCPA5mf1vilBbKu+w4NdYyXj047IliCDJkBwlh9MWWbHFP2L1FNvhLlaP+hlZ6QPa 0bvMzsTc7dVmDa/edQXE7Sdt3o+2JTc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VARa2z3w; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713201558; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x5lu4xKhCQ/0haQPCj7eyNWwUk1BrbeMvC9dF21zPVk=; b=uBavVAPblHDzynKbwRUfa1NlqTvjlnkXn9BCWy7jeP2z8z0BEg4rtoVHRqdaAuJZK11gmB AuEGB+hVZYivzJjksHD1JZ0/1T8FBPrMheEMVnVPuNFIhil9ZN4+RXLR4b6GuCPI3NXmIT lylM95rxBlhie1CxTwJQbF7kTP5egRs= Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2a536b08d63so2015330a91.1 for ; Mon, 15 Apr 2024 10:19:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201556; x=1713806356; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=x5lu4xKhCQ/0haQPCj7eyNWwUk1BrbeMvC9dF21zPVk=; b=VARa2z3wMK0bHW4vgwWuOydk7ikolRu6OprrcY4R6kUYzjaBPNIXIOt/E5gkdVyklA zQCPFO9fVFxjNSYm3PZJZ9nTHNa4+crzICcmSYVaW1HTd7MWShTI1qa/F24z8cPn/A1i M9TscpyQ5awqxMgQ+TXE9bbVORGGSporAEtIbkl1pWqWCC3KYo2y1O8u1x10M1mZb2dO Kk+Fsj0m4mQNroOJXuzfuU4c3psTJSAOvg4ogmPYseYm3vPbkvgvqhTkLM0x+4tGFGjH hoWfNcT44CQv6HPUsUi50AI/4BZ6deywlYBuvE4xKylKg+XG0FTyj+ozmir13upK3cuF iQAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201556; x=1713806356; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=x5lu4xKhCQ/0haQPCj7eyNWwUk1BrbeMvC9dF21zPVk=; b=LZo3phV3Qwmj7jROiGxkArZfzn8HD1/J5hss2H8u+3yEJzqLofMrRc1uIsiWGxJHJC ZuCzV4xGbdfVWd4YB3/H98ZDJO//xDOPSrRDYtzUQxuTfkL0FUpojYH14fE2w2B4r6RZ 0pwOMAgJx3CGGsucAjJpSZhwSAw21MVIXa05+rRPlUR46zHMxBWFi9QGGyPNjG8CtlGI kTIJqDIvOYyxAmCdK3OarZ+NErrTFJg5WfvAtnwX7PZztQMSgksHrR6REe6pudu4e80V ka3NPw48MSmluzbdyR/2zPzidoA5oTWnQ4u+bj2RGciqB4TOYmTfgme40ATMKeSqd8H5 MgoA== X-Gm-Message-State: AOJu0Yzm8rRhM9+fAD8OqvxXoin7xwGP/7nnfmrg96XOz8PFiFk8dSoV 8pkuZ+4dI1scUpwTjsfgYs8A5vHKhCnatmtguYaD+elCCDPyLXHHP3TNIF9C X-Google-Smtp-Source: AGHT+IHCSW+MgtEmdHVUti9+WxD2hWCD0NmfCyU2sjka1LkmR7bv5TKrJwBuID2fpVaAEzlYPQwGNg== X-Received: by 2002:a17:90a:e50c:b0:2a2:7817:f591 with SMTP id t12-20020a17090ae50c00b002a27817f591mr7838981pjy.48.1713201555986; Mon, 15 Apr 2024 10:19:15 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:15 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 16 Apr 2024 01:18:54 +0800 Message-ID: <20240415171857.19244-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 137A618000D X-Stat-Signature: uggzecepwr99fnngt5qq1zbxknzepnku X-HE-Tag: 1713201557-505332 X-HE-Meta: U2FsdGVkX1+ls+qmBqc5B4F2hDEQmXdg1GEOV38aGx7H1G5KU4GJQ66PpnhbHF+91GMLvtyVA7YXFFwCNjHsCQKphhUxPmDQy3YybOGfRm1+mfOUZYc716t0qcHKH7jitq3ATdpKZ4I79NTLqV3djnHFuudqUHIJxeu0m/H9xdRnCu5zQ78MwomZhojdJSBnNdB7odSV1x0NCU2bUL0v3DKb9D96NvHEiMi4mBCCeWQDaH3Kg/3bDDcAGYzHPJrnd/sETcN+qhiN6yUW9gAmFzdsVAx+1Rw4VBYuYI0qKkAhDkj62yl7bWBG9qpyOlGDA/Nq0nnLc6ULnitRbMJLNiR+jZk0NII8lrQ7nm9aX5VHprMa586qtfCmEBx1Y55NjQ4KCJDvbqyhk7JemAqzUrvOa7/SgWOnMVxkWlWeYhkbDRD6Js7VGwr9mqYXzyH4/CCidO6UD64rfp4iWamlbyJOiIcHHP0py3xlENmBdmDVd929ESDtO2ujXPyNh92B8fFWpi2Tl+4m9vzWWLf1t7eQVLj9wcDKkFFQgz3yCKuvwc079s9Vjj60Gs2XU79tarI1xhuanRQl9hYMaUnPIoLRpJ51xXqZoVlQ/IJPUE3oFQ2nWqadZimNUf71HyHvlx9eeOjNoZWNLDBVmmeco0hs9VAxcBuEEmYT0xl2RRCUHLOfBmtLyOXSg34/ctan3oB2aAIKV+4TiNdiH9Dq/l48shyq0muYX8vS1PxYOzhcd+UrRwsCmlzr2im6I/fYmyBxO8OtIpXtwKoo2ztVoiHnl1L/UmH8lqSug7qgQfmT260IPih/0JOhn+FDBK7a5tIt2kcYVSGyXx5uhijm3PFT/FlQMC5s3arV0sG7KPypJguA/URgnqydymVWLNeZhdHDyvS0X7VlaH5V3hcqKsvYjTxldyYlv4hmwzNHvqUtj0Q+jzUjLZ1nsJUTVLZ/I0205xGQn3vL6C1CrM1 sQjK5F56 IcIhs6A7mXlQ7sfJ1KE8uO32mQ2umsHgj92+RqfQcd9EIGbUznEWxND5jtX9LpH9Ojt9slLfaY5sGGZtMEIrZJXA/Ff4d2O/TsXJK4sYWXwLdnfh/lJ7gtRCOhXrlWlvCIHxe2E/U6fZyutIkuQgD8ULkSCoCxcZeysydriRRoRaNPyZmyWhuahu5WwWmube6kaAGycN+yc+ZzVECu+fwaPfdgkECWeXXp5blU1runxDRc0OQzi+JZ53qUYLdKsXk8vbKX3EFmt4TS/fL5eq4aSAgAQWJn6yU1F7rjk4QALmEIkIPoA6OwcC07TaQJqLi4NW4YNtZahQX/Z8Hzex/wGr9Y24+DljLwHMJLj7JjTFr+YjENrXxEoWjrIaFlB5sWAUjD0oao1q06pSD5YNdYW//kUKdw+ArrvnqugaRsiX04oDQ7/h+HTZTRfrrJDNvWSsf1vJd7HCBvUmtP6qzKeHI2xtimXs0KhcAt5tyVATxqOoZtFlpWxqFw2R4B0QEX5qmsYNT5VcoSfdBnj0XAAHiz8tjL/1woH02IWVnKt6gg1zs3tXRpLa421i6FEpTKRsS8EveGndM9W6nnWDkK4Vs3syHoNwpAuxO6M6A3abYjDHCZg3YniGkk9fKBU9F31xmQcBrKAy9W6aTnYr+F4lu2MPLcPkj9VM6PbWBZifvmBM5sX62MLr++a7VirwAu7Pj1sQX+8stKW/xP3M++ASWnsBiYndEDsL3rdvSwEk/Y9wz3vhQouTfJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 12089c24abfb..17a66ea544e7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge = folio_test_hugetlb(folio); - bool charged = false; - long nr = 1; + long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); - if (!huge) { - int error = mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged = true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr = folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping = NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, void *shadow = NULL; int ret; + ret = mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret = __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like From patchwork Mon Apr 15 17:18:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13630434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DBAEC4345F for ; Mon, 15 Apr 2024 17:19:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51B936B00A3; Mon, 15 Apr 2024 13:19:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A3D46B00A5; Mon, 15 Apr 2024 13:19:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D1BF6B00A6; Mon, 15 Apr 2024 13:19:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F0C3E6B00A3 for ; Mon, 15 Apr 2024 13:19:21 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B8670A1777 for ; Mon, 15 Apr 2024 17:19:21 +0000 (UTC) X-FDA: 82012427322.13.FC3797B Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf11.hostedemail.com (Postfix) with ESMTP id D565C40019 for ; Mon, 15 Apr 2024 17:19:19 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WKL+jyOg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713201559; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cEjwwoQAn8m3grSTrvNR4sEkOR0nGA3RJ0TZjOfUHm4=; b=AOY0e9hf9Quo4Wm5fKMbV19NdY/EeikBedk4O9A/NqhhJNuvHseGC7In8G9s4pDlgbMtBP IRJ5s+YcVCb0CJWFFssXdtQryq3OayMMLvim4dTrZzadimvXGCwpq1nYhdrtg7mDeRbBuB nwKL8/w+C0bNirjzm8PBxtgUPS+OERs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WKL+jyOg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713201559; a=rsa-sha256; cv=none; b=Zb4DSlanpN3Xr1AJV4qslJlTtDz8V6hZCHRYfUMgqOjc8ElSMY5rW/ZMcZ4EFpve4yu8v6 SGHYoOnBCGiyb9+pg1o6Ew/o8hTPptrrQdgdgP9HS55LEODs1WS8SwDK8tYY1l1lupLZ5h CKXCiDlWJaVmQT/+N/X8vp38hjPvtpE= Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2a475bdd4a6so1933846a91.2 for ; Mon, 15 Apr 2024 10:19:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201558; x=1713806358; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=cEjwwoQAn8m3grSTrvNR4sEkOR0nGA3RJ0TZjOfUHm4=; b=WKL+jyOgVyNgz4Sp6bAk2juwaTtu8Wu8ppKjKuvondPAgGrBV15tXFwMFD2eZPffhA snVJlUB6w9ymf1lQmrEox9vXhB4g8seJF2sQTw83aNjPRIIUwooEj9Ji9fcH+9zysRZH bcPFCX4D0CIvKNehYUGGUbDnrXJnpltGnGRxOyWLcbYHPV/5Fx6m05s91ZXYPUM3UOzU sKpZ7pT54CBi8DF2zMWBax9ro8vQoL3rqP+axQX4VEOxQ1LlHGDKMGmC4XAs9aDJEXCs lOMBzcxfhH9udGTRXlYQslTrIMUlzvX4X5O/vL6jmYl811/LpJUnvXk3kaW1UAFdN2AI ryfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201558; x=1713806358; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=cEjwwoQAn8m3grSTrvNR4sEkOR0nGA3RJ0TZjOfUHm4=; b=KjwVu+rGns/fVgshMtApb1HaKEvkSUwotgLv88lFk4sQD6Mo57QD2bVUuV4XTSxxAB P2DCfyJ0I8iZQ1Micakzow3LK0Bua05RWpfGZ1jybH84rRLEw47ES4bEoqSf89kz4mWR XP5aOYCsrOZYgTBNzxNGm5jsqBR6qunLnyPa9aox6ymzZZm0U/LYQWXOND0MihzwzOt5 xSZZCJZLwVSuYYc548H+83gQne13pQlq1cWeH76U9gyyWrM2w33ojzGsd7orVb9DdGkP bLbI5RUIYLv9nGrh0ljmZ52PrJuVYDjsssQ/n9C4U6WKD64/05/0EdkmUVsg7p98hj7c GoVA== X-Gm-Message-State: AOJu0Yzp27E5s3lXFL9X/sylHdg/6OqNICHnsD+4lRmt/nSIh9SZcrfk e46+MrySegc89KSxAUkmAYsgLYrHP4yiLwGI6gFze/ha24EeRjtv75tFS2x4 X-Google-Smtp-Source: AGHT+IFva3FT+6Qo1iTE2UIbKwGFak0JgfA5Q5ANtjD68Wt1gDflM7XbEgZEqtWRZbI/FtzxK97aFQ== X-Received: by 2002:a17:90b:3d3:b0:2a5:ba9d:a06b with SMTP id go19-20020a17090b03d300b002a5ba9da06bmr10332693pjb.5.1713201558299; Mon, 15 Apr 2024 10:19:18 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:17 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 16 Apr 2024 01:18:55 +0800 Message-ID: <20240415171857.19244-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: D565C40019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: hum15b3qwmy1jud3bdgm9j3sckq13f7f X-HE-Tag: 1713201559-194207 X-HE-Meta: U2FsdGVkX1+2krcRx7wr0jnM/Dp0okAMboZ2LC/RbM5cekjlNyYh3TaHPPbZ8/Jgi5VehD/+9a9mdpYpY+HGlgtiRzcujVyEJYTlYvm74mHdRRffVlIJnXo+AObZSYjyxU4bRGkfclmmiGBqprHIdeGZTS7EFtd6sibXMCpA91/+wwLcgBoUxxTnLzX8AgMePgO3V+JBpbPuMuPWua+fzjNcQ0kTRk4dHSh2A12aXMwAc5DvixgxxzQkU9wdNeo3yRTN0Ph/zv72PaFa0Dgv4M5STJ9FDl5JQ/DlKC8I7SNX5dneBSpQPtGe+AfOl+ibhhTHPaosLFL2f+CJ/d8JuT0eYzV1BMhWRYGa8BXfgbXroS1O4nhib1/uqjmjhWZFbttmeQ03DSfLMZ0fTBsMUDwvZsPsAdYYpzh5WV6Wydhcd2/G/zAZRoNMKULRzBqYcGaawvzO9pwWkMgV8v/LldwcLOlevc7XyWdE2BcC4PNAmKROcSQlsHkgw4mjgtllcuUoxOZ2ilPJgTMeMjZWKuVzNRVlMi42sEgrMqnbWcfRkztALiBqJHkOJeGHZPA/sMFTpgAYkBxzYgl/fSWTHq7wmn1jvTGDB+H0Vp9mR4Snm5ES5d0cQGGdWvlg2Ut1flffTagXzH76Y98h/Ybzt5cYzxKNH8HKSHpfwt/wkjWIT8OevDgafdvWFaJviE2Mwb5pSdtu9dcl/5ClL15OV5RY0nX1nYwM5Kj1QX1QtC0fkvVsbJWITyyoAp5nCDBaTgECMZuSPLBj1fM4GTnL9qBYw49i+GgKwZk1meGUSpfN6AURBDQrxEouhmLYH9tGH2QH5jbKqEJRM1D/VoqDWrJP9WYdV77VcyKKSJrXp+CdpoRUwePEFt3n3vHVNinf3fa+e//cwDsScMy9Lv2z+dwKtWJC/d9V7Lr1WlwC8sTIIR12FYdcZYxAXKsl8H2XAXaawHhaojC6lejvR26 TgAXtYgw A1b9OEedcIapTnhaG3svD0Ve3W94NTBoFdJeatZcEnAdeIRHS/EVgDVNW2Oe4wZyk3aQoYHb4hU/eN4D4OOS36ko2yf7Bb6PXNeYc0VWO8bzR+7cI1M1q/CRwFm69CAutqOxh/8TN2M60AgeEb1amQ+7yVaY5mVmhxqArUokROKXmOlWupH8IQC8dd/ovF2Otiv/PDJLg7tifvV6STmiT/rfZW5rYwYeUYEgHzAZoNf5KdlRDYs610RntZTcPj3FItzpxAsL+bBFlvT7zJvV7mTuvqCmGSIRdmUbAlEnzaicwSSRShkUCEgFJ71N0grJIGjgnB0h+7UnlqtoHfec5+kyL8YbdH3bGMRl3zMBPw5+LvJTj4lHZKibgNIbqnEFOTLcxkT8FxzEE3a/4lMyiX5BshjTxmDBtE0oikUq4ul8qWBDV+tY5o45itAFCFnjcvpBMK1dG9QmsbQ3T2d9L0h6YJ+6G0Xbl6AhoG/mKjxGA8wu6CBsjvA/ejOYJT6kgiClVhpEJpPSnmYDO1HaY8wXcb9bgmExBJl08tpVNT7hRJI6qinBLqRKug8U5uEi3NArU6W8hpm0eR0883p6tN8xMyeBlfADtoCoGE/skGf3v5v9XjcIy0wXaalYSknTdgp9zxAETVu4y2HKV4tkAEa/bu230kWY4x9JS6xioTSnFKN8DTfGY81r7uEka/CeAUBF0rxv5gzUgicvJVr+6aDyn2PGyq5GxzBRS+3yYsogolc8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Also add new test for xas_get_order, to make the test work we have to export xas_get_order with EXPORT_SYMBOL_GPL. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 34 +++++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, gfp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, unsigned long index) return 0; } +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..0efde8f93490 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,39 @@ static noinline void check_get_order(struct xarray *xa) } } +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + for (j = i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + rcu_read_lock(); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + rcu_read_unlock(); + } + + xas_lock(&xas); + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2068,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..fbf1d1dd83bc 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned long first, EXPORT_SYMBOL(xa_store_range); /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order = 0; - rcu_read_lock(); - entry = xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; for (;;) { - unsigned int slot = xas.xa_offset + (1 << order); + unsigned int slot = xas->xa_offset + (1 << order); if (slot >= XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } - order += xas.xa_node->shift; -unlock: + order += xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order = 0; + void *entry; + + rcu_read_lock(); + entry = xas_load(&xas); + if (entry) + order = xas_get_order(&xas); rcu_read_unlock(); return order; From patchwork Mon Apr 15 17:18:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13630435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 005CDC4345F for ; Mon, 15 Apr 2024 17:19:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B40C6B00A6; Mon, 15 Apr 2024 13:19:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 762866B00A7; Mon, 15 Apr 2024 13:19:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 568BD6B00A9; Mon, 15 Apr 2024 13:19:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2C0136B00A6 for ; Mon, 15 Apr 2024 13:19:25 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B4BF8406D2 for ; Mon, 15 Apr 2024 17:19:24 +0000 (UTC) X-FDA: 82012427448.24.A8AC605 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf16.hostedemail.com (Postfix) with ESMTP id C8FE4180018 for ; Mon, 15 Apr 2024 17:19:22 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TkwTkdDM; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713201562; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7ftt1wlC5vn+OAyWyZNr8D22lSmnA0oAAZ/IwpKgu3E=; b=ix6Ookarob+QdhJ5nfPU65HdAdCsF4doXes+ECGnrDOW3jIXuhBxb6I4eT16wp9uWlBBds Yvm1/gwGOYOf/pP85Fta25JzHAYLZCYa47qoG8lBEoHovlQiXuqhJpTFdIAapTSG8O7xAE kHD8NjfOd7jVO4hQy09Kb6RtkP7av5U= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TkwTkdDM; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713201562; a=rsa-sha256; cv=none; b=N0DmSBBcnQolwTMjQr/NosCg0rTZSU+bG1wk2XKXwNTN3cBpoL3M5kUSLSt9i8J+66x50v w8jqomBe2HXZsFQ2HyRQEXeEzBxh2PP2j6re5nKeqC8LqfzMC9V4g4xsSzW+7IPTXE3k6t k6wcNq808crcGl3NgnrOFFjx4KRIhJ0= Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-5cfd95130c6so2336852a12.1 for ; Mon, 15 Apr 2024 10:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201561; x=1713806361; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=7ftt1wlC5vn+OAyWyZNr8D22lSmnA0oAAZ/IwpKgu3E=; b=TkwTkdDM1yzdct/xCHTWl3PCgR5KKqq74QNS267kQdpj+oUhZxAnrgrgbogCXO25oy QZX3AD2FAs+wVk1Ldt0lTsWQeyzLHg3gj1aVw7eg83mqMt7CFnaId+pFZtXzqvYLbmW9 2nnPJTjDs5FY2eRTCwyMh3G3G8QARYM8Yn2KdKPcu89Y156Mne+GHC1CYNZ4+XByw8p1 CbW3PR7IsIZYi2zsIluPEOa+0Jfl129nj9Tqpm3QnWUeZ4XbHsFy0uVFbEOvIOrEEgSg amKDQFsRsyFeI6u0mihLIXGlCXoCSvkQNiwAoPvrdY7W+Lu37RPoTlRIB/ih5nv5Ek0o jwDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201561; x=1713806361; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7ftt1wlC5vn+OAyWyZNr8D22lSmnA0oAAZ/IwpKgu3E=; b=vKvKFluVKcWded6HsrlMwdyJIqzGLO0N5XLRkaVstelQ+vB9DbWHJXH/X0jFibbaco FkwV+WwCJ6j7+9SplxjGdxEPMIEofYor1fDsdosmFEAFOFcJhylOZN4FL1mO3oazMh6I htqFFwf4jtb6wOkHokdZRTOmp79P1NX/lp9Gp9fiDD1aFetn2KeKF57Ijcsr5Huvwi0y vrgKPLlnUJHQaRwuecoyq2q4iUWCMRZURaGvCV5ijWtmXiEYQLrPlDzBi9ld3GV1jRt7 v34Gr0URI+m5gDAxb3eOrwzB8X+/EhY+bcyZSrmMYw1Gk8W4jIFhQZkH6Il6kL/OZDVS T4/Q== X-Gm-Message-State: AOJu0YyEr6mOmYeLmOyf8OLkD1o6HxZoOvcvBMRnUuedSaEle9t/LCIM gExFHAuhnnjaYvpVD8mUbZL2g1jDTmAwSdaPTkbK3ODpR6oZmS7lVMXZyxMW X-Google-Smtp-Source: AGHT+IF5I7CUmmaoFvwK3SVG9GhKVEy4AarbCQA8oQBcJAZLx+3bOZ1JGWosjLM5SoLJEKPUKgit5A== X-Received: by 2002:a17:90a:f2d7:b0:2a2:b9f8:e0b0 with SMTP id gt23-20020a17090af2d700b002a2b9f8e0b0mr6933990pjb.19.1713201560649; Mon, 15 Apr 2024 10:19:20 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:20 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 16 Apr 2024 01:18:56 +0800 Message-ID: <20240415171857.19244-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: C8FE4180018 X-Rspam-User: X-Stat-Signature: fwyaw4tqhg4ze7j416b1ynzyher6ytg7 X-Rspamd-Server: rspam01 X-HE-Tag: 1713201562-960948 X-HE-Meta: U2FsdGVkX1/4vUMIAFHmLBottvmalo790nAs32QQpyQwlSoSwmY1ZTWCQY7HtE+3KCDdzGcXAndGi5OhjtDM9OGOhw97dwiOeS2pBsqRa5ozCCTeO1nsm6P1z4HqAOWixt68GBL2UvHFyGfS4XMWWVRNlnb/ObnylPP1PjzF5BMtQRw3tGYyYr5Jcwe5RPyfbkl0F6VBFN6xCnoyOVxxKSVfsi1KxShvcU3b6fzzg399S5QA4mfMYISqihXaFbyqAEMngraRb0rszH/oZ9BVk6Y9eNMwhTG0I46uQzgUW2A+MnZO+LvqWTzACvjLCiy/8G4SPcRBYT+G/At/EKRLI9evSV5J4nIKZe6yJ6seBz7SbYzZL31EV2UcY0tPCqnt3fvERMGn64ORp7ZQTbZUxFmkxtU0INMgQAF0ASBAqDNbWr3PSLK7bw0q7WSrAjHNsHLt2Y8yT7Tu8tzH1FfW0C7suNo5S+M7wbUsy9ABwAGbtmrSasVqhZAwGTHqr7MNI5HuqtZMVnStL/1jozlaaV4UJpkKn5TPsoUG4tKvhDb1SSwKw6t25qCWculj14B5aWL8b9dYdZjCGWiBROSULIb7tMs7wCvNvCZSt44qs4jWWyga+NqzkLk/6tAos2dOS8UXE1wW6keJhpoyRtsUa2RIhcYpMvoUUb+gSpZPvCd3MGPOJbxXcC4VzHNzINwxJhPbSGp8EG+d7cqqh69XAHzhXRuVWgs+rm3SiTw+Qhw7DUfRtnisnMShBvA18A/QUXZSItXnom9rFRV/jFAMzMSq9+pRkHUkwYuXAvubrLwyyOt6xJ7SMxayR9oqJTq/KCI2jaEGHKaZ0I+QLlxS7/SOnFLIL0i+61y+gk6tTVKboQEbYtpFZDksFtEjsRVcZnWWIEACHapXHqiwrLGmzRycquR9sHBrmXf2Jx7Oxeg/WVscpQhEBd+VZksoprgCQcJECHOdgYrel/9Rih1 5MLgFr+j JtoMskVchECW4oqK0LivzUm53im3/vfxxuRvn534B73vDp2voIDir76yPORJciLpJAX8rmrYT4de6KsFmJEwxME280UvC1JeCGZsaStjaPWQt12Ptf6te6zSp3eatCb3YG3Z4u3HWB3iyKqnNvwiTU2V6BTT4Bz49J4SNbuN08y9ObokE0hS3IWozL0o691YMu+xChv9TLGWWPVJ7Z43nE6dGdEerVVtJSN6FYXcszFWOAmlv6yU8vKH+6ArsWTJdWcauegypU3hhBSV49k7soqGSK/8mQvNFqB/zq+UQu48T10ZYd4SpALTEFBCGxi7xVCRbO2EhUBGsQKZwQq05zJoyUJuM07bWoILU+dXQDagyOgkPQ7kA0QEu/1jErPNs2T2CaQm55FTo+Z7Gti7N7TWNH+8glaMVP9Xbh4dRkjRSGp/1LCAyjBCY0hglE3HYeugU5QG7WHGtDAsClX+uUBSDHG8ngUzU2m0c6uInFJXy5CKxW+oWVT00r6OGgtNbErevk1E0Jp88eZ1/HpbRw1EFfZ6o357OGgc/+vOm6bnlwnZuNd0+axHh2Ivoey60PIPVwcauG5vEN274KlsnWM+4xSK3XvupIIjw2KAepZyZl0ytHoKR38NRQMGpvVTURugh76J7ajJYij+uQCig74PmE+K0Ydxmp3jhj01WqHkvWBmV3FizhwquuoM0H/8xL041mNqkcDM8iNbh6yKSOJKajHEcaITyp3Jx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 1027, max= 3520, per=100.00%, avg=2445.02, stdev=18.90, samples=8691 iops : min=263001, max=901288, avg=625924.36, stdev=4837.28, samples=8691 After (+7.3%): bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4141, max=14202, per=100.00%, avg=7935.51, stdev=96.85, samples=18976 iops : min= 1029, max= 3548, avg=1979.52, stdev=24.23, samples=18976ยท READ: bw=4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=64.0KiB (65.5kB), run=14419-14419msec After (+12.5%): bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 ++++++++++++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 0efde8f93490..8732a311f613 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2017,6 +2017,64 @@ static noinline void check_xas_get_order(struct xarray *xa) } } +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order = 0; order < max_order; order++) { + for (i = 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j = i << order; + for (k = 0; k < order; k++) { + only_once = 0; + xas_set_order(&xas, j + (1 << k), k); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once = 0; + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry != xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) != order); + only_once++; + } + XA_BUG_ON(xa, only_once != 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_lock(&xas); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2069,6 +2127,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 17a66ea544e7..7b0b2229d4ed 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge = folio_test_hugetlb(folio); + void *alloced_shadow = NULL; + int alloced_order = 0; + bool huge; long nr; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *mapping, VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge = folio_test_hugetlb(folio); nr = folio_nr_pages(folio); gfp &= GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space *mapping, folio->mapping = mapping; folio->index = xas.xa_index; - do { - unsigned int order = xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order = -1, split_order = 0; void *entry, *old = NULL; - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old = entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order == -1) + order = xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old != alloced_shadow || order != alloced_order)) { + xas_destroy(&xas); + alloced_order = 0; } if (old) { - if (shadowp) - *shadowp = old; - /* entry may have been split before we acquired lock */ - order = xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order = order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp = old; } xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow = old; + alloced_order = split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } if (xas_error(&xas)) goto error;