From patchwork Tue Apr 18 19:12:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02107C77B78 for ; Tue, 18 Apr 2023 19:13:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 864198E0001; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B0EB900003; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A11B8E0001; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 33E9D8E0003 for ; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AD41F14041E for ; Tue, 18 Apr 2023 19:13:24 +0000 (UTC) X-FDA: 80695460328.22.3DEB14B Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by imf12.hostedemail.com (Postfix) with ESMTP id A277740014 for ; Tue, 18 Apr 2023 19:13:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=x7kihz2X; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.52 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845201; a=rsa-sha256; cv=none; b=WFpdmLU/rt/AMtQfXeOKEv5hVxsq+mKfU+n+xTcDwCvE3rt5AsZTQzLhv9a8X82g/0nnkJ GjbdMW4s6ECIEBcSopTD0M28fcYmR+m5E8tmj1shzcLADzzjceYly+vvYma/5wCKC3qgBq cBbiG36BHF+rm7WOUHyBNYIwwvRHL+A= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=x7kihz2X; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.52 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k7sno8AaRf5ATInZUw+i9Aey3v69SOK+YNtRO/2P9OU=; b=AkNH1kXkoCuCETjG3WnhQJBN6pzl5VfvntgLDQesfIsWUpPYK+vfWNeQimfDbG4Rm12EBn YA5TSwMz5Jo9IX6wdVI0Zr0ruPMrYFErw/6cDNyp5Mplxjt6reZCstsIwaKt0ZUOiie0hh 32l3idfRFZviES433SuOhOCZM3ADzxE= Received: by mail-qv1-f52.google.com with SMTP id m16so18343800qvx.9 for ; Tue, 18 Apr 2023 12:13:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845200; x=1684437200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k7sno8AaRf5ATInZUw+i9Aey3v69SOK+YNtRO/2P9OU=; b=x7kihz2Xek9Oawcxi+XxZhyApuNkNkSuXAlGa0k24xE/zxYmZiL9Tjnypqyo29xZrW Nsmvw/Opca5Epg2iXzQaqdiJrL5rqFjWGypHqp5urO1vUn8xS+vqGK55FJzonBr1mCwf wGaiiiFg2tItrhXI7KwlkaJG/RDfu09WqpY33e+D80EVxDPradvBJJ12Je5vjNrE+TPi ts+YdchVOie8BtYVf5YyCDVqV+r9hhn3Nl1kzyCzI5gGydNCC6lEPeXQFfpGWMSh2+pi CGhIO6Il7AoD+ydlRVCr5v40EZuhLfXbDgvfOUIzqKX+MJNjjoo0qr3XT7MqqIkU6bO/ IqJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845200; x=1684437200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k7sno8AaRf5ATInZUw+i9Aey3v69SOK+YNtRO/2P9OU=; b=AEyEf5m4jpXOJtE6F9u/zPXJNDFe0iSfXQ53MqqXVvuvGnAxSnsBnpM86qXEujO2/E g+qD1gJXFw+pBe5EwR6aPh/0UpNuTpNEiiAGgXy09EGtiMq4IN1iTNA2ZhD1JNgqAIbO bpMO6ckOIrYMQxdlsjNN5oqK4SUnDP/isl+Xm3TjTDmDIdf1tM+fElRree5/KoA4eIaw C1X89UGYlJWbKoK80Cdz3/QjI0ueC7yZclbv10fLgvq7TT005TOlLADTmDZW308NzL79 1CFTITjVe3+MEo6BjPXm7WsnZE8iJXiDuCuA9f7NO2BZmkStPVcXrfy5YVu/J9inMbKR zuyw== X-Gm-Message-State: AAQBX9dmZak7coi+4Tl96W0cm7YdVnSbmbFYRW3t9egY8blhnIYDjYVO oowGutI8+s73ZTrb/0am9aQoQSDen/Rwg8xwrVY= X-Google-Smtp-Source: AKy350Y2WvA1kj6Bsm7oquBxOdrZDbffru8om2OFKpHsG6Pe1CEinFIFTzd6IhJxiRR2FIVaaYlT9A== X-Received: by 2002:a05:6214:c4b:b0:5ef:807b:2a96 with SMTP id r11-20020a0562140c4b00b005ef807b2a96mr11066482qvj.21.1681845200482; Tue, 18 Apr 2023 12:13:20 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id k21-20020a05620a0b9500b0074e04516389sm659536qkh.35.2023.04.18.12.13.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:20 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 01/26] block: bdev: blockdev page cache is movable Date: Tue, 18 Apr 2023 15:12:48 -0400 Message-Id: <20230418191313.268131-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A277740014 X-Rspamd-Server: rspam01 X-Stat-Signature: 3gpps8hzyx3cidhoqsdkshkyarh7qchn X-HE-Tag: 1681845201-911749 X-HE-Meta: U2FsdGVkX19Fewrw/MUc1LI2anyjO6lPK4uGPwymSQeV+4obVpRvxSRYOiLN+bTyVcrKeJ1M65vk53rz9DnLxWVer+YNNpA2kQJm7Ow0glHsZsuvx4cSms/eqxp42tf8qdPmjrsbU/SYFSkbrPQFOUDJbR7iPZCwh0eUBg5QK/Jb5BEr00o1nQpEs7/YNmv9DFmzuTrk/yQYvaj+7WCBVqrLxPRzUfzQKGR1XlA36RXbTyrU3rtIlMClQC9kcUvGpYxlrLtltoarqLbySK9hocfD7ed4F1Li7W40NBvPPakahnBUnuY1Nnol92+malc2sfdGbzdjagapFKYXdXB3IXU0MZSFo1TnG0NZQ9cctvYgK7eOp9MdGJCEpISip3BorcNYwNAJiC3+pmh4Ma8k24uV6L3lmp25Dtx+oIiKkigTvVaV6VG50UXiEDf7YR0OBFpq+dtjAGbNr7XnBtJiCHTkNwXMCLCjj3Ki0CBF4GQt8m0X0uJ/IgXC0Blw8gcfp+vdQFay/myleJILbNhhiv9eJlBtpVJ4OKwUj+NnySTskfOoIbFkzcTDci1WJkJ3E6ZXmG6m0U8Ovp20t7TuEAAUiT+lR92tmm4U1+tp73mgVu7TJaPn+JusLR8rdoUORuJ2X/M7j/6k9qgDPvNnFF0VDClsJ6WS2ePtpY/mLKAXspRSabw6mB+iMRhwoF/ljOPny5oP9uBq9+PJ8TpAQojRdDbrwcQ/pvSwZsYgDINM1pgfAdE3nPAVWYstg76Yq70L35ctIC6YXfvHrQcI+TW7VIv3WC+DrvH9p+NxrkaRnETx5TUFK4ic988yEohb0uXsjbEWDfZsUlq1VJEAOtO7XNo3MWbAKvdOcv/9bs3DBAGovjaodvt59/xeROkQBwN8iyiwVKk3rp24PHXr/z0BbQgIGoGpsTQiQQMKFL442R5YNAQ77/X+NnkbJWgZfIQn/3Feo+FfZRcPcE7 GvARNTdF NyCSWkFexiwxjYaZDD9hw2rQrS8aWemePkRy4eJ3EM4X9Xa39URSAA0lKIO9BA75kxc1LtLJfikgM7fLBgpPqhRRC8JEfyN8S3M2swhwnXnz4F08DMeC1uQgiP+S8XFwEDChWVlEXNcBfrcAei7Kg4NSd4FbLhumsU8iby54heSWJyVb2rEukT/g/1zmIEzPWlgJ1VRHOs9WAbd3K0ffx0u5EQJln30uQrrWvsoD5RSdxX/CL9M2nFpTxY6xaIQNXz2DcXCGaOslFaSo2JdnoAzrHycdVZbZwbhcjz0NClh6dp5Ev4w8UZn8Rqsdy0KkDt+UZ3UEwJ7oo1zp7w8lxGjVih85U7WTFB+9ARgW2FqVrH977KITLd9YTaiVYCvLAniZWlJeU3GUtkKlU3lPY5FXDPqFqm8SAjtTT9pd6QSQ/2pZKOkzDSyfkkme1uKXMwXwWF1Pp1fLbqRh73tuEcmnQweeMQz1V8Xpi+98vOdIycto= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While inspecting page blocks for the type of pages in them, I noticed a large number of blockdev cache in unmovable blocks. However, these pages are actually on the LRU, and their mapping has a .migrate_folio callback; they can be reclaimed and compacted as necessary. Put them into movable blocks, so they don't cause pollution, and subsequent proliferation, of unmovable blocks. Signed-off-by: Johannes Weiner --- block/bdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/bdev.c b/block/bdev.c index edc110d90df4..6abe4766d073 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -488,7 +488,7 @@ struct block_device *bdev_alloc(struct gendisk *disk, u8 partno) inode->i_mode = S_IFBLK; inode->i_rdev = 0; inode->i_data.a_ops = &def_blk_aops; - mapping_set_gfp_mask(&inode->i_data, GFP_USER); + mapping_set_gfp_mask(&inode->i_data, GFP_USER|__GFP_MOVABLE); bdev = I_BDEV(inode); mutex_init(&bdev->bd_fsfreeze_mutex); From patchwork Tue Apr 18 19:12:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB137C6FD18 for ; Tue, 18 Apr 2023 19:13:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6044B8E0003; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B0A3900002; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 403458E0005; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 32E558E0001 for ; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AB8191A0443 for ; Tue, 18 Apr 2023 19:13:24 +0000 (UTC) X-FDA: 80695460328.20.05B357A Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf03.hostedemail.com (Postfix) with ESMTP id D01D02000C for ; Tue, 18 Apr 2023 19:13:22 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SR96bpw9; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf03.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.42 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845202; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+u1UuZT1pIMEXypNrA+9XQUlZTAHYDviJjF8gqonVQU=; b=XIQ1CK0Igrvs2XNcQWdaYaj0YTvGpBdV8ZddTooC8qjkv04ocQFhAQnEQv4a/tejB5KLc6 7t08eAi/Pg4VdDTjUaXFe0iM1byUwooLdPVfwVwLqS4N6zd3ipovvjI2dEizvytqNu9TAQ tvO9yBe1BUqOkAraJdeQ6ynsJzLoqe0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SR96bpw9; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf03.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.42 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845202; a=rsa-sha256; cv=none; b=ThaRrZoAPrIwOaKTX6dvFZp3VHeDRT0VnB3BuI31KckQJdviUf8RQWcA3q0tkn1k+68etG rsWP73ehDIu3r9gasifZT/c5LzPicvtk7lZnp7wtBKYuz6w3eyK4+1rzaSvPqe79l8Yn8f vxhvGXi3SSwDU6o+H35KJZSySdA6lNk= Received: by mail-qv1-f42.google.com with SMTP id m16so18343847qvx.9 for ; Tue, 18 Apr 2023 12:13:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845201; x=1684437201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+u1UuZT1pIMEXypNrA+9XQUlZTAHYDviJjF8gqonVQU=; b=SR96bpw9MNAx3LCxCgWDO8w9zD5dt4ByUGuuyAnVydDGhAmlwQbHzS187awsb5xwdO 3E5aqY4YcBiZQTeoz3fQUkUokUCWEDRMjGydvXec0gcHjnJOsbk/0vKlUc/tnBFoT/0C kWH+7WMwtws4YbPqzcCRt8bBH1bl+ICBQsLfkT1ezlUyIH/5+oOmw29dhYFofT1hkTdY fyMPwBPciCacz/1C0AgpYcdlzBi7yEJearS5hkjaNIgnLfuukw0DCDMopCTZGHeXjebj KTN/KuXdWL+pLGpBy6kKsnYoyQAxLtS4wrxhGbhjaIIdY/MpFdAuz6/NMu5kBVygLQNV /nvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845201; x=1684437201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+u1UuZT1pIMEXypNrA+9XQUlZTAHYDviJjF8gqonVQU=; b=TXjQca2kX+uCDV3NRinDbAtT9z5k33BHH/ELkjaMMKPl0QRYFeU17tl5Q/df58A/PJ aJRYOBOyPugYkhCklt3NLKESgr4MhpWs30bZOon+pkUkyY6cE6dWOhCj6kQn2IH2BbuP lm3vUkn+I9I277gDRBZ3PaWjgcHDJn72LTHN28UUes15hBMx6+bOZkKhayUd1Kf+ST5m vNhXVo+3103bkMtASsEdguAki9c0B/u0qydzbKJ5/xynGHwcd8wNXuLAaZQQ2LCV6ezv c/+tP22cWfJYyJNRv/RsYzd9/Zsq/eOKT4RbHU1AFRFLn1kdx1+go1OqEQCPWbCsOOc0 0kcA== X-Gm-Message-State: AAQBX9e8ACyB2LQf3lWTDdh738wDZ+qHm3g1EdI3j69VAGg4pwa5oj3X q0lizw0IcM+iTzw4ohvZwlysPJKMjo1cb+U26MY= X-Google-Smtp-Source: AKy350Y9G6t8iPg8KGaVLYACgNehTbb26PoR9hC3G0xtoef9vY2sm05dJCXh7ylUzUWaDJ07gr2qMA== X-Received: by 2002:ad4:5b82:0:b0:5e9:9eb:e026 with SMTP id 2-20020ad45b82000000b005e909ebe026mr25993947qvp.29.1681845201676; Tue, 18 Apr 2023 12:13:21 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id ay35-20020a05620a17a300b0074e0951c7e7sm428997qkb.28.2023.04.18.12.13.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:21 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 02/26] mm: compaction: avoid GFP_NOFS deadlocks Date: Tue, 18 Apr 2023 15:12:49 -0400 Message-Id: <20230418191313.268131-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D01D02000C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 8b7nc8x8cbqd1iiz9peofpm8e7j7crnh X-HE-Tag: 1681845202-425906 X-HE-Meta: U2FsdGVkX1972dphIHfoO3NucXXACkluWLvLC4/HkBbTBM5m6T86lowGZY9aDKHYIT9TWqLPYD+VbXqNVS+EX/KTB4yO+zXLYT+oTpoEIrLJ1zhc6w78v16D0z6Ki7GRWNFpg8ozn2pGnw4sj1oRoChcl00LgOfkF20M6ocq5FIm1CD7x4YmO+JKoQ0835lMAlVa+iyZzpfCSIeeI+w+vbbNCdg0hNkxqYNsrS9MhgBKViAQrufE0ZbeF4d2ZRa6QZGsP6/R4dnO5Gjuc8iDI7gdwHJTKqW0xSAi0ly29AfS729lSoutFqLtHCKaP3PKofW5HDDUBq3KLASzKsdVZzECWSsOnGhcZ8t5MMv/K6GJb1HVLdrHa0RYSe2tC1uXAPTK95XLpeOVM5F8c1wnVSi6FSoqzcFRIiYI/eix8/J2kVhm29lSxdXVVLFv5ZLw/M3QVrB+rfuIwyABDEe1lU/jz6eMiIQD+UBnxRYMGoL8rLQajl3VzDKUrmxtpfyoYf8Q+Ctb0LE/hvZlMncPNovE2kVIwf0EVpffs4E4APNGBeWLG1YtHfQf18kcnRrahTOvYNh0/PhGDrJxv+q1poUYlKrm0J992u6pgQ2ZMzbuG1eFiFZYpWretRVMtWph+L9CU4gqQ5H1jpUHu2B8sSyBedDSEV9i/rdQV1IjuPGVA2vpyz2FWdvf9M+RSct/zX5RcrBfVb82IysXwfLEWA++TsQOPjqfUxhwBpKhJS7LwIGy7byb8xDq5wf6vVaxX/eRHMRsaFnH9yHpZDseLt8ec9itDIK9P5+Icv3xYFwwFPNibOllbM0rlSZE2gL8pk4qEkYgToSfY6d41nywq2O0ESGhtC3z2k6jPvQ16Kv4VSIq0apBtEZ+GPOXtl1CuYhK6eLJuOe6nc/UGEV85pvIRcVII9YXPNwoQZ1SgvwsRwlCri+sXA2xPQfSJjGBwVxiFY7HOEfqJSK2wRI B1B0A0QF Y6YlFKHeIVREm5TrQK9krpBsRC9SSidtcorgm45bvdq7u/f4nN0LHB39NgNQoyUsPv0v4ArrB6rodBmvSJAAAQvqZIUKiAoDjZB3mOc/nHssS8CX/ZIo+4D9vSB1FhQhypEYGY9s93sRdvy6PVQEncw3BtcFCZTklbUTaH4NjFOOzwhGgw063lTgTk1CHYjhMYQA29HEnOgFuZ3SwnARQVFqfD/0nXIrCuTilcscPY0YbBfotLBFs03O45mXZso3CF2zzp1ZTG84hwS4GfRNXD4LTTu7vfl6UupSIKJ6GNUNTP2KW54Mj0BmI+8UkCB3V5UYqERqEE94B/n5HrrL17zggLc+FsPCnOmqblFFIOhn2TlU2SuzxqQtMS2EoJkSlEGe3Rjro+FcCEx98zCp7ZXFpfG+5cVB6GO8/v297MnBdLEutM5F/JjROE6VXwo7HlTq7m+7kgwBpYIkGlSi+J6no00UjGGy9W3zXuBeQ7Y2L89K37g+lqxWfKYwYk0zjIZv2NfPJ8gn0mfEpKEkHOxE95w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During stress testing, two deadlock scenarios were observed: 1. One GFP_NOFS allocation was sleeping on too_many_isolated(), and all CPUs were busy with compactors that appeared to be spinning on buffer locks. Give GFP_NOFS compactors additional isolation headroom, the same way we do during reclaim, to eliminate this deadlock scenario. 2. In a more pernicious scenario, the GFP_NOFS allocation was busy-spinning in compaction, but seemingly never making progress. Upon closer inspection, memory was dominated by file pages, which the fs compactor isn't allowed to touch. The remaining anon pages didn't have the contiguity to satisfy the request. Allow GFP_NOFS allocations to bypass watermarks when compaction failed at the highest priority. While these deadlocks were encountered only in tests with the subsequent patches (which put a lot more demand on compaction), in theory these problems already exist in the code today. Fix them now. Signed-off-by: Johannes Weiner --- mm/compaction.c | 15 +++++++++++++-- mm/page_alloc.c | 10 +++++++++- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 8238e83385a7..84db84e8fd3a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -745,8 +745,9 @@ isolate_freepages_range(struct compact_control *cc, } /* Similar to reclaim, but different enough that they don't share logic */ -static bool too_many_isolated(pg_data_t *pgdat) +static bool too_many_isolated(struct compact_control *cc) { + pg_data_t *pgdat = cc->zone->zone_pgdat; bool too_many; unsigned long active, inactive, isolated; @@ -758,6 +759,16 @@ static bool too_many_isolated(pg_data_t *pgdat) isolated = node_page_state(pgdat, NR_ISOLATED_FILE) + node_page_state(pgdat, NR_ISOLATED_ANON); + /* + * GFP_NOFS callers are allowed to isolate more pages, so they + * won't get blocked by normal direct-reclaimers, forming a + * circular deadlock. GFP_NOIO won't get here. + */ + if (cc->gfp_mask & __GFP_FS) { + inactive >>= 3; + active >>= 3; + } + too_many = isolated > (inactive + active) / 2; if (!too_many) wake_throttle_isolated(pgdat); @@ -806,7 +817,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * list by either parallel reclaimers or compaction. If there are, * delay for some time until fewer pages are isolated */ - while (unlikely(too_many_isolated(pgdat))) { + while (unlikely(too_many_isolated(cc))) { /* stop isolation if there are still pages not migrated */ if (cc->nr_migratepages) return -EAGAIN; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3bb3484563ed..ac03571e0532 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4508,8 +4508,16 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, prep_new_page(page, order, gfp_mask, alloc_flags); /* Try get a page from the freelist if available */ - if (!page) + if (!page) { + /* + * It's possible that the only migration sources are + * file pages, and the GFP_NOFS stack is holding up + * other compactors. Use reserves to avoid deadlock. + */ + if (prio == MIN_COMPACT_PRIORITY && !(gfp_mask & __GFP_FS)) + alloc_flags |= ALLOC_NO_WATERMARKS; page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); + } if (page) { struct zone *zone = page_zone(page); From patchwork Tue Apr 18 19:12:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8126C77B75 for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 390DB900002; Tue, 18 Apr 2023 15:13:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3409F8E0005; Tue, 18 Apr 2023 15:13:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19413900002; Tue, 18 Apr 2023 15:13:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id ECF408E0005 for ; Tue, 18 Apr 2023 15:13:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AE165C044C for ; Tue, 18 Apr 2023 19:13:25 +0000 (UTC) X-FDA: 80695460370.11.F7BE550 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf12.hostedemail.com (Postfix) with ESMTP id E0D3540009 for ; Tue, 18 Apr 2023 19:13:23 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=phgXjhYw; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R5SnIlaO0rXlDNrbIipd1QFsl3Kwll1/DWWk9w0ZS8w=; b=lY2ACfj4UWzeg42nJ4dd145OgfiNvsJ8LpMyVKeNkO2MqQdfYgPe9Uv2US78PGrPK6TVu7 3MVlLEa0/KRnqOpQk2wTLyGx5KxH5/jZ3VSNij/MgJhaK5f0gUbswN4g6Cn8soas4qx7D5 1qt0mqR1dp+TJWk29UFaJzL76Q+PB3c= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=phgXjhYw; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845203; a=rsa-sha256; cv=none; b=bmsh3SCQg4mpYxViSYmQFFPt7zP3bpzGumMoz5KY+IUnW6M8rOTYDmiElSKh7EYtdMjwhy A95igYn5oOzTbxkevZvJ0pvlI35FW97aar3KbBUsKxXFbybVENoO9ADjaqHrCEemF+O/36 l6GF+vzYFv+aYkUxscl0btC9gfvwGgc= Received: by mail-qv1-f48.google.com with SMTP id oj8so1342652qvb.11 for ; Tue, 18 Apr 2023 12:13:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845202; x=1684437202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R5SnIlaO0rXlDNrbIipd1QFsl3Kwll1/DWWk9w0ZS8w=; b=phgXjhYwXg3OCGPRKhBvTgFcaFQYHSKs8TQycpKy6Vj7NYOeHax3ZHG4ro7qbt6UuA JF51v3B18zEUVvZDZnNNDBXHiS2b6Tq1HXFiJxKw9p+6Mx89Rgd2RJU/zsjn7vLY241i MKeUzmB5AHKtmXkf4hDMRr20OIpBEF+bV8qpHw5olo3beiyALYxqKqMbkp/HbNpGk7NJ ukwixb6eSH0cNmhdMS8bZQt3NGWA7C/bL+/YY6wI83+dKvJxLNBr78X4ennNHHmOj0Yt 5r4Yf3GUa7UROctvzCXAsXuKLK1G6qtC30yjMZWjnRlXK/bEWWb7onJzOj7tlzRdljt6 Babg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845202; x=1684437202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R5SnIlaO0rXlDNrbIipd1QFsl3Kwll1/DWWk9w0ZS8w=; b=gGmZn1MFTOiCuAkjSJwW/lYFcXNrqAGwuv8JzBob3OMnNeKENA2fO9BvZBCHf6jSSM k7v3BD9T7RDWCH9gbljdO3lx0LNvAlDuLKvLle0Vmy0ZE+JR8g4XMpQ2ox17M1mgHk9e K/SrV3qDs4cqGVGDcEGV3je3m6bXUChdDluiOUDI+P+xdFn6GEJF1kEvw84y+62VOYQz CA8FHIKQblOm6tivq6EGTpRas/rnLYXEf0Hd6F4nleiuBpWo3H0ndAludRKiTorDpZlk qcNi2SzQ53VFcT+4dUTknwMnIILtDaX2rv3UcFuEBgwkQbLH9cxZZEY0pGwWI4jGm8j3 t5Gw== X-Gm-Message-State: AAQBX9ePcdphvams8Rmn/rNHNKOGt7GfvIJdFg54njoZ5g28XXiClnKy OsQU6NMePkVppVaYeedN4FG7ao8t43zXidFfTBg= X-Google-Smtp-Source: AKy350b9afMxBv6edVSpodj5/I8/s1MMj3kTb89vTQ4djsDWn/SVzn9x9uBL9GyIXdZ9FhPbmcE+vA== X-Received: by 2002:a05:6214:20ed:b0:5c5:c835:c8f1 with SMTP id 13-20020a05621420ed00b005c5c835c8f1mr25959976qvk.22.1681845202760; Tue, 18 Apr 2023 12:13:22 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id o5-20020a0cc385000000b005ef4ee791casm3855524qvi.120.2023.04.18.12.13.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:22 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 03/26] mm: make pageblock_order 2M per default Date: Tue, 18 Apr 2023 15:12:50 -0400 Message-Id: <20230418191313.268131-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E0D3540009 X-Rspam-User: X-Stat-Signature: 8r41fa487jfwoqw4gfofszqb91fk3hxp X-HE-Tag: 1681845203-469295 X-HE-Meta: U2FsdGVkX19af50ycMdTzht76vCQGcf1xUFLByMfL/BcYUPhQodI0FkV/0IUdHe2ihcGAL8FLt7gxFXUt+uB73eZc+Zbvtv6WUu8pbaOycJIvVdoceSfRpGWwLrYzI4Tr0+ceucgIDLyTamdJx8KmNmAjXWSU7fbYcetq16d4enSXKeq9/VtuTThsbmoF3qhuVMEwfX+qFJmB50JO/n/y5u/Cc7lQo7yo9O1Abi3PiFuLBi7GClNLHiu887bHWeSG+6J7CtD56LG4bhdOYPb+RpevQf8/vjwFMP4i12antcSgCL89t79O/p1FDidwA8ZDmduzaoHAZ55zdCd/ySvsir8NXK7ifjSBJvXGGCGE3gO8mQM4UC0MNeiAZGg5fx++SESpb+nCrXMOUAycUt8UzuKcn3avwhRrVIvWNXJyEaLC/V+PigfOrncVxbeUwomiXjedhSDI6qFmKevJghry4tdWIP36UJB6ClUsv4grB09weKBfTD/9RbMgZhLzfJcjfJe2lOsgxL/nRuj7a4kf9Iazj2rndfc4gjXTaedWnIhepC1hN0Czj6g+Gvyt+nHI0tSrC+ufzMDXtoPi+UM58JNo4FNyOLkSHgHPZyjIcfIVFMOWs9zQI9LaClFxoxN0bXolIBPGP/wTChaUL23EDtOvGlejeqna+hJiktSWexuYThvNPzKvdCb2nUBLQKm3QAUWv3dyZCe+xcabhPBw+rZit1nn2Hn5hEV34A8BFJ2tdcTj6kkNdOwH0enkJiM9rI5X3SpawCpcOWlrk/uqPakoqEiVoOm1fziFN1P8romWIuE7fOWMjzVBP76pgFnvftGY8Do58nFA03rtdZyiQeHlZJCvAhetF9kJ6Tzip2PvG0+JNNOuZv6pFNJe77nCKtBzm7vF3eDu5h0j35aQEQdrX8E0c/ayIfsBJ0HIi2YYNzCSmI+rJY6xAFEyrPR7oYmuvhngNRSKyfgj6l 7CdcvmhX JqqXXUgcNKh4YDVzoY7KJM5bOF5SH2KiMk4lJIPNRNsj8UxCbJBJvd2PXTqLBhiiC7vsm4pU8yjzP4phK6v2DtbDU5PAi7C/cxguT9XlZMvwzsFWB3LXnKGSEdXPA9nYLdbDxFvmzYmZ0IyiL4+G3yYnH3U1P2KjcjYGeTf2vlSIhv9Ze1Bo5A0jFi3bdHi9P2ZUDFzQ9pjoNlrIPqIKbpiFNjKeV4BOohBpLggmhM+Zr0ffZieIWz1WKVCBDgr6qM6TYH+qYFikP7r9kSjvjkB4s1ZPXc6fmwF+O9gg6/NnQ1iNhuBe9BgiZuLW+YCs7YC/vbV3DGrmadx/8DAa/aosHQgJ6ExZ9gJhk8WlernHUTW0hIezGS/pa/2Bc+6WdHY4Y0vXMgryheG4SixHdDTKu/i+uCCaRtiPQdqYjQh1WiiLDqrOYhWvdOOLeA+Qc6JCqF51pwEvk6u5e5c9UI2riz4sDiFRI4E274AX6VC0Ns2knbfPvnt4wzbUw9mo9U8rP+W1ft/UbMat9VpZVOUorBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pageblock_order can be of various sizes, depending on configuration, but the default is MAX_ORDER-1. Given 4k pages, that comes out to 4M. This is a large chunk for the allocator/reclaim/compaction to try to keep grouped per migratetype. It's also unnecessary as the majority of higher order allocations - THP and slab - are smaller than that. Before subsequent patches increase the effort that goes into maintaining migratetype isolation, it's important to first set the defrag block size to what's likely to have common consumers. Signed-off-by: Johannes Weiner --- include/linux/pageblock-flags.h | 4 ++-- mm/page_alloc.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index 5f1ae07d724b..05b6811f8cee 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -47,8 +47,8 @@ extern unsigned int pageblock_order; #else /* CONFIG_HUGETLB_PAGE */ -/* If huge pages are not used, group by MAX_ORDER_NR_PAGES */ -#define pageblock_order (MAX_ORDER-1) +/* Manage fragmentation at the 2M level */ +#define pageblock_order ilog2(2U << (20 - PAGE_SHIFT)) #endif /* CONFIG_HUGETLB_PAGE */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ac03571e0532..5e04a69f6a26 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7634,7 +7634,7 @@ static inline void setup_usemap(struct zone *zone) {} /* Initialise the number of pages represented by NR_PAGEBLOCK_BITS */ void __init set_pageblock_order(void) { - unsigned int order = MAX_ORDER - 1; + unsigned int order = ilog2(2U << (20 - PAGE_SHIFT)); /* Check that pageblock_nr_pages has not already been setup */ if (pageblock_order) From patchwork Tue Apr 18 19:12:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B89BC77B7D for ; Tue, 18 Apr 2023 19:13:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F4668E0005; Tue, 18 Apr 2023 15:13:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A560900003; Tue, 18 Apr 2023 15:13:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11D4E8E0006; Tue, 18 Apr 2023 15:13:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F0B7C8E0005 for ; Tue, 18 Apr 2023 15:13:26 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B36B3AC2A5 for ; Tue, 18 Apr 2023 19:13:26 +0000 (UTC) X-FDA: 80695460412.06.7D61D62 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf20.hostedemail.com (Postfix) with ESMTP id F038B1C0012 for ; Tue, 18 Apr 2023 19:13:24 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=kkfQYtWH; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TaODw93P+SR5wwuqKRfmGnBXSp8Lv6zFL4m1Q/yirc8=; b=aLyzoBurPCrxS6elij+x2beDSk9EHh6TDVJxQlSuksKqHFz0dn+iNTaNHLP5n07HII31RO RLaQK7Rb0HN7VRZ69L3AH7w9Gev5V36/Xhs6d2itC4o4jADokqgyElOaA0P/kf9djZEW// lM/IrwxWxafXR8gXxDi9aPT78zFIKFs= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=kkfQYtWH; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845205; a=rsa-sha256; cv=none; b=NChviwZBQS5xEPVpsuHv8bzwAb+HQhtE9LPtBXPexC81Vt9jHgFoHFK6yNXRoocp+5rkGV CmYff4tdOh+ir4bBdcp5UR8oqPAXFBaDBIIBaprzUSXCOIbqZrYAjKAIvqsI6mP5vrWX2G KP2xSDoSNGdEwDGwTbCBIs6Kv68kZ3Y= Received: by mail-qt1-f174.google.com with SMTP id fg9so9511064qtb.1 for ; Tue, 18 Apr 2023 12:13:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845204; x=1684437204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TaODw93P+SR5wwuqKRfmGnBXSp8Lv6zFL4m1Q/yirc8=; b=kkfQYtWHK0cjsqR3Tr2Evmha89C/CGQuIEEPuLJT4pN5xWQi7d8Vq8uFMbzxn0pTuC vDsdfEFFx8thdvtcSffx5yY2knbkiKEIrb36FC9V3aQefF2s2mRjSVPD8070oJiQg8ZU r5RGXeP3tLr0RU10A3wP/Cz9FMVpBN6V/JJLvDmUodacBd0JSArMMCR8nLaXExEBOlpe yNw3G0y7GS5KEOFNn1FFACN24lKo6bTWTBgwy+WWKudhSd5CtHRcuY63yhaEwdjZrzkl 7WYaotdrp8RPDc3Q3kZ6QS6x9WTWeqdC+Y/EyJLjEHZmM+bvNm4BrANxjwU1IMwgDCiF l0rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845204; x=1684437204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TaODw93P+SR5wwuqKRfmGnBXSp8Lv6zFL4m1Q/yirc8=; b=GS70F7xiW0Z9Ex2zIwArYzKSZVw7G220Htlq0WirVVmjUdHND/LeQeeL4uU87ovHO/ OtYLDQIcbOW6qbxIi/LAlEbS901J/PtS8OvSpl6qX2g6sY3b3QnLCOwjkTK6pFRqjHdF zymgvlJPlKLLGSu/wYNuOrThHxHpIn1W/nhLzueqnEh56Wl4CtGGmkcQHclIKzOa7TQs iEmY6ex2rLeoowyRQ+S1XNwqnM5VdWmFxm6pmUAcvE3SaCXfWRoFlipUHK/xno04rUUT fLVk2831sZvn8gH2m57No0NOVF0j/HMLitC8grY+E0Nbg6uhjgZUxQl8KPN/Y/hD2Ahv 64iQ== X-Gm-Message-State: AAQBX9d4j3kLXfxB2FZjzxp+OA2naxUHBd+k5h7zMgrrfKi8wWiV+rjR pKgsrlLChFxRaljoBLDuUKZvHQfG6QuQVUbhgAc= X-Google-Smtp-Source: AKy350aw5jRncIivAr25Em9/snSJvkqiUm+fvvm6Oxsi6j5yZZnDL1qzCh3Cxx7kfD1wTwtewWkLkA== X-Received: by 2002:a05:622a:1042:b0:3ef:2db1:6e75 with SMTP id f2-20020a05622a104200b003ef2db16e75mr1455616qte.24.1681845203869; Tue, 18 Apr 2023 12:13:23 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id p23-20020a05620a22f700b0074ad04609f8sm1217413qki.117.2023.04.18.12.13.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:23 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 04/26] mm: page_isolation: write proper kerneldoc Date: Tue, 18 Apr 2023 15:12:51 -0400 Message-Id: <20230418191313.268131-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: F038B1C0012 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 16samhfhpnkxh4zadc6hwz4g3a7z1qwb X-HE-Tag: 1681845204-273817 X-HE-Meta: U2FsdGVkX19Oef4/PPFZD7aNg3BRMdn2vfYkhvUuinPydN9HPKPiCviSRPNRHSpyGdrBjA0rDRGuJYto1XP6MQgm7VmHp9ljzOqpPxFwfrD1vGVcEDVvDGOpesKknNj8mnQdBE4UG3Bx2PUFeh16UrMyVewDNFq4KSCrqnbeVyn6IYRKwBluTUUQ1Q8AgRVRa2raOF382kT4aC+hiLxEB3zPI6TZ1ALHLuvTAc3wa07DkR8EbOcvQdavTzkiCQABaYEVXa3AVQ229v7/C/XRVvQze0KiCUgBDSovueUJFU3LxX5ZfCEkUlp29qRHRUmNcLf3r3HYxqyNAe+Y8exylY+NcbqhjvsG88p8q/7oxFYp+4c0CdBKSWWhAbVQ/PGABvhiBvO7P8OCZyALuDM8CWPR/qiok6Fc/EPE6iDuvB8QFCbbyrFT9yjvCHbC3nVEbk1zc4OYlf8cPVvdEo5PxFt6zXxV4lHwQl27wUVBYNDOkRJ+CIRWpGog43wVMsjhjwJaSXtRJgBMIgAjhkn2Y4ZTXze18DgkFrmSdvPFZM2AujkB+8gQEMMzALw2c7Dy4EDh8sSli+5rWKsfNG+bK47KVE7rlaeoWcyCTdWRjdDhX2ToL+KxSTXDrbSB0HToduQUHKvgzgCWWPiDZxPX/2R3prUv6Xec7pwEACjpVDm43H2o8ZJQaK9ysnFEDwv05PmegErbk/pEmvIxqtMYtlQiHRmABuAY8k28QRFD9it5Ckgh/Fna+xtlzG74QRT+QdLLljSQZtnH8dpjyuxmjD+ziWhM6vjchpzK1xMfq7km2pdAq5+oPJnWSj5jHSOwu8obXYNl5B3gROk0Bhwcu/YsqzybIxSkPMF8biTWi7WOvxUvu4ZICyrFWYaMdPxRuwlrJeo1/w4g5QH2W+1gO/ARn328SFnkhQDf2T1rn54Yy+i5IpJFB492HCxEtxbVfEYhixcvQQbnFkG75GK MktfYvEp 0Uq4iR3LpfGTAEWR9lmzNpH3+om52o9UgY4FDf/fqQ9LdLtK2YXz0KFIR3jC2CqCLEB5EgeMeiDvJL2nMcDquv42FvK5cK2KtsWexptYLmze0B8B9cJsuGTigKqEMYvBZvTO/58lcWyHRRO9xqInSXRyyAaUcOgLBzFO9Ic4uF90DN/L8iYBpEjwA4ycsJ0Azp6fCEm80J98gnSzEMn3MGHn9KSaR/c/zKsFt7jExpGAXfKNwDfBUAhsXU5kXA5cxzHQ+ocWmymOl8SNXoZZX7KYfqD+GX49QhDeFbug7iiodsVix4rMJHz3Tp0HADzcYau8BUysZqi8h06zt5sN7RbYnIKz/r/x6dhPayAO87wj2STpyXFqpaAls90hEUMHbjzOthsgeCcocS72VGgtOH61VGQc1nFYQIH7ufpxbk/ALo73mkLhrvtqWL2uJh+pCIo6LgQe3tZ6BjDlxhk2DLcjALccMLr+cNgcLf5QeZnQ+okcMbLlK/bnNjJn3cFuOvYLFtWDvhgnEW+Uvj8gPYNAGfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: And remove the incorrect header comments. Signed-off-by: Johannes Weiner --- include/linux/page-isolation.h | 24 ++++++------------------ mm/page_isolation.c | 29 ++++++++++++++++++++++++----- 2 files changed, 30 insertions(+), 23 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 5456b7be38ae..0ab089e89db4 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -37,24 +37,12 @@ void set_pageblock_migratetype(struct page *page, int migratetype); int move_freepages_block(struct zone *zone, struct page *page, int migratetype, int *num_movable); -/* - * Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE. - */ -int -start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - int migratetype, int flags, gfp_t gfp_flags); - -/* - * Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE. - * target range is [start_pfn, end_pfn) - */ -void -undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, - int migratetype); - -/* - * Test all pages in [start_pfn, end_pfn) are isolated or not. - */ +int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, + int migratetype, int flags, gfp_t gfp_flags); + +void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, + int migratetype); + int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn, int isol_flags); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 47fbc1696466..b67800f7f6b1 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -481,8 +481,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, } /** - * start_isolate_page_range() - make page-allocation-type of range of pages to - * be MIGRATE_ISOLATE. + * start_isolate_page_range() - mark page range MIGRATE_ISOLATE * @start_pfn: The lower PFN of the range to be isolated. * @end_pfn: The upper PFN of the range to be isolated. * @migratetype: Migrate type to set in error recovery. @@ -571,8 +570,14 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, return 0; } -/* - * Make isolated pages available again. +/** + * undo_isolate_page_range - undo effects of start_isolate_page_range() + * @start_pfn: The lower PFN of the isolated range + * @end_pfn: The upper PFN of the isolated range + * @migratetype: New migrate type to set on the range + * + * This finds every MIGRATE_ISOLATE page block in the given range + * and switches it to @migratetype. */ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype) @@ -631,7 +636,21 @@ __test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn, return pfn; } -/* Caller should ensure that requested range is in a single zone */ +/** + * test_pages_isolated - check if pageblocks in range are isolated + * @start_pfn: The first PFN of the isolated range + * @end_pfn: The first PFN *after* the isolated range + * @isol_flags: Testing mode flags + * + * This tests if all in the specified range are free. + * + * If %MEMORY_OFFLINE is specified in @flags, it will consider + * poisoned and offlined pages free as well. + * + * Caller must ensure the requested range doesn't span zones. + * + * Returns 0 if true, -EBUSY if one or more pages are in use. + */ int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn, int isol_flags) { From patchwork Tue Apr 18 19:12:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F802C77B78 for ; Tue, 18 Apr 2023 19:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69EF6900004; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64F4B900003; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 516AB900004; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3B01D900003 for ; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0A89D1C653C for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) X-FDA: 80695460496.15.00B48E6 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf30.hostedemail.com (Postfix) with ESMTP id 350EE8000E for ; Tue, 18 Apr 2023 19:13:26 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=HuPalACv; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845206; a=rsa-sha256; cv=none; b=zyxhIje8h/BBjRrM9hNxXEOqPqbvybZu+UgJt1c6qjsG5YWOE4mPCf7yOeeTyjdT58kJ37 728G5MjRZqJ3E3UZZXVFlTmgFLhFb1Ow1X4ooVLoeczbiKr8Jw65/T4U/gwBD3AxPorcf1 n86PFI2WyT6RKhwTzkVxFaC4FC/AcA8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=HuPalACv; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=ipdVFgZ/6USzCEncAgfVGLcsJlHLzLLVki88Mx8mqivGZTii1u6h2kaxdldZ/5gYIakbWW Yq45Ngu4OJ0bjxTFfSNl9w0NHZhEEX4oRSh/TZDKw6PSACIinSNG4lNXdNbh8W2Z5DMatm GVfNI8s7xI7BiuGOVml7FeV0nr7zFw0= Received: by mail-qv1-f50.google.com with SMTP id qh25so19655656qvb.1 for ; Tue, 18 Apr 2023 12:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845205; x=1684437205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=HuPalACvDhuLF7KvDz6g49PPKO7XyqovkGZVXCYy6pLcxgFDYRU6w4grAPkt0YvGk6 pHJ521bPBzsCouaZvOOPJXwsvNRmORjMrJzURqRnWoJKVVIQ0LYCfgJo8dAWEnQZSnm/ +qvQ+RSmv9XdJ2WBvUINtpcBP4b/WzY099qV6MGA2jIRIxot+itSnsyKvOmEvArOiu1Q S+M3Q1BZHNM97/Br+cYmnpOE4izzfu1L2C7HfcIzhT5hZSS40LopQDZ3JnFsmhJL3dMJ Cq3x+nKbD8VyFtiLxqGEhPZyLptBiJ+Tm3yhwtrd5LnIqpPyDSHTFaSAvqO642ZLRgyW OGog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845205; x=1684437205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=V8QxiDS2rPqnCpm0g3vd02i1cKAVbB8lpGq4S+3xe8dgrv0uwdLr3Q4v23pVtA294P ldIctdacqNpqTJu0arCiMhXVGYiX63IyIHWBJ6xDGMyDcbi0L4K5bkWDYhShS7LWgAg+ gVzoxIwQ4LaMKk08VupyNzhagxxqLXegBzjL18ivoOOqGVlJmWBS+3pFlAyOYDWWa1NZ a/eNrFDuc5lVxsmwAYSsrtX8GC8n5PgZrA5FDz0dUSnCTDVAc+lXUK2ZO/UoQ4Q6YoSm TNAEGd/LadedLnyjj/qB4JSF0Rm2KiyXHH9k/FmB8Zl17BSp1aWUwa40sXXXTfVcvpPv nlyw== X-Gm-Message-State: AAQBX9eycYvcElGm9JUILsU4DJWXvgPA8DWOKwe82CIYyV3rH6UzJwuR qOR0IRxCNvP3YZSFRK7yZKgb8JmWefKzU1EpC7g= X-Google-Smtp-Source: AKy350aDUeUhXJBSWIivTN/XnDLT5DJ+ST1mc086iGBK1cA1BfRASnqlSz8S0NO45keDkBhRWTpTfw== X-Received: by 2002:a05:6214:2305:b0:5f1:6a35:60be with SMTP id gc5-20020a056214230500b005f16a3560bemr288353qvb.23.1681845204992; Tue, 18 Apr 2023 12:13:24 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id m8-20020a05620a220800b007468b183a65sm4166837qkh.30.2023.04.18.12.13.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:24 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 05/26] mm: page_alloc: per-migratetype pcplist for THPs Date: Tue, 18 Apr 2023 15:12:52 -0400 Message-Id: <20230418191313.268131-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 350EE8000E X-Rspamd-Server: rspam01 X-Stat-Signature: 5jstomscikubkhiz7ctyshax5r354sns X-HE-Tag: 1681845206-182975 X-HE-Meta: U2FsdGVkX19rHKBjYKdNfubXTpKie/zluJhSEEDj8lCsP4hiPWBNgpGYqE3L6uXvKSOr9BzejeqMkEBJ+oYYpHV4dgdCYeEaaSJebHXjsaRe6e4Uv8Ng1Z7jqSM30PyAdk5pvQDzhnxPJDTh2yI3wWQ+zF6Im2Hi+iUQ8lFXQhVZlkQN1j7X6YSAFsi9JewbnyqJ2HU/PBOvFGUXoZDHO+OuceE4qrbP+npP59kSXUzc/PGKkEZ6VSNqZWL/LV3c7VmqigA8Ks2K5oAApTlACTKHS6c12o9yeNA5QRBLsdzbZ9aUKf3RBTO+3zhaBeQvgZMl/g9nyL8b3VohHh8P4kgGsm2lQEebc4tI5CwU9pVGcK7iyapk4ua87Td5lHKFUpEw3k2JAgygyaduK8VvUVkmXdu3hhszQachQGcrnvNhfATLa4/wRDg42GLHDbrQwQOD8AGj9VhyPTKXp8OnUSjEa6WJX/7vs8f5FpMwmLdiLhurYbI6ngwYxd7rZUSsFKeSKXu6HIhMEkdk7f9CHpb5WSgqSn3Yhq88/f9uRKXDXumNvhDNkIv+tHnpdpiNXUIcJcu7TNGhNqwo0TmH0f2XZytyAUgcWdErkTfCLKlIQRFjqekgmBkUQQmflpAvLSFREqoRonei2kvkjXVs+yNN287z2WnJgK+k8nPXTGLp3s0f/W/Noorm4254MRQJ3Z4SaYdOIj0qY+ZARPeDXMbh3dKJqf2jHKF7bDD8WTStdUjWJbRUHX+3DTzdpWaWLx+RxEeVbdQho7Cd37jIhQF8aad0gNQGtTkNj63L8k9P0KqD+4qPdyGR3tY8xUoR3yPGrvK8OT3tvjBVQz2DaV2y0FtLC+xe/vqEHg8hWSXOeTSli2B50Vqb8BWykVC/f3Wkv+w7chCSsxnr4K/MJio/X8NxogFbPTOQx99bMWalkhhcBcRvoT3gNWBY0Isu9bXkTmzaTpqai06fVDm YqH6MDzf ASltprdK+/EyfuaGwQYCWbtbkH+s9xvREKGrwyqYY0CWfEXMljqI7VYRgc/5GKYoMreG6/EMyJIir5yyhAHy7JPE97w5HegdBQaZw6y0EIF6+eJZYpaDm/YOe10bPMw9cJcz3GCzBpGo2H3cEp8kYag8U35CIrK/M4bf3E13CpGqIL0yStSMybNnQJXjsvvsq4sOeS7XMQj78u30t5J2IR6mhjt5/6HeRyrtcsl/QBkimf7XDnCQBGfEpud4sgDrpIgjbd9akGbU3/ArWhhhjrXSpQBpkRuyAxRtXlCCOrbQOsFjwq3kJUpBP2Mg7XhEvCRC4dZZAQx/wk3mii7gyyvgop5D1l0eFv2KNRjLunM2UmTaZTi9m45TezbGxmzuUX8/OS8G4TdxxWLewnT5+gapkRQVb699oVgUgH1zOwYrYXn/au3Jm7PsYgPpWwZz89KcnEqzRkLTmPGMa8Q/255aiJBtyCuLet4DgcWXRAdta58kggSZkscAW9VL8Me3DzLDwvvedQEzSnS/5eKGAt9oGnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right now, there is only one pcplist for THP allocations. However, while most THPs are movable, the huge zero page is not. This means a movable THP allocation can grab an unmovable block from the pcplist, and a subsequent THP split, partial free, and reallocation of the remainder will mix movable and unmovable pages in the block. While this isn't a huge source of block pollution in practice, it happens often enough to trigger debug warnings fairly quickly under load. In the interest of tightening up pageblock hygiene, make the THP pcplists fully migratetype-aware, just like the lower order ones. Signed-off-by: Johannes Weiner --- include/linux/mmzone.h | 8 +++----- mm/page_alloc.c | 4 ++-- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cd28a100d9e4..53e55882a4e7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -552,13 +552,11 @@ enum zone_watermarks { }; /* - * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list - * for THP which will usually be GFP_MOVABLE. Even if it is another type, - * it should not contribute to serious fragmentation causing THP allocation - * failures. + * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional set + * for THP (usually GFP_MOVABLE, but with exception of the huge zero page.) */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP MIGRATE_PCPTYPES #else #define NR_PCP_THP 0 #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5e04a69f6a26..d3d01019ce77 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -710,7 +710,7 @@ static inline unsigned int order_to_pindex(int migratetype, int order) #ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order > PAGE_ALLOC_COSTLY_ORDER) { VM_BUG_ON(order != pageblock_order); - return NR_LOWORDER_PCP_LISTS; + return NR_LOWORDER_PCP_LISTS + migratetype; } #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); @@ -724,7 +724,7 @@ static inline int pindex_to_order(unsigned int pindex) int order = pindex / MIGRATE_PCPTYPES; #ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (pindex == NR_LOWORDER_PCP_LISTS) + if (pindex >= NR_LOWORDER_PCP_LISTS) order = pageblock_order; #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); From patchwork Tue Apr 18 19:12:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CE92C6FD18 for ; Tue, 18 Apr 2023 19:13:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C48A900005; Tue, 18 Apr 2023 15:13:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6511E900003; Tue, 18 Apr 2023 15:13:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A2C2900005; Tue, 18 Apr 2023 15:13:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 22E81900003 for ; Tue, 18 Apr 2023 15:13:29 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CD431160460 for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) X-FDA: 80695460496.11.0797D22 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf20.hostedemail.com (Postfix) with ESMTP id 064C51C000F for ; Tue, 18 Apr 2023 19:13:26 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=POy4P8vq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845207; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Yvli2ztOIxf68cKgZ+2R5Rhio4udTEYI5UCWBnx3AD8=; b=BFZFAri66xSlHv94V8SJxmzFMqdvEkOq5YyhOn2/s1XlTfttQZJDTtEaLVmEc+zPOcy1uP l57S5D+CkX6/znBcGAjMI2Nplt+lZLrjfs7L3K9F2ZzR5do1Myo0TAHZ0hXxn5Sa6hGAUz hjv9CusW2IZuEWKi288ki6SpaqFvan0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=POy4P8vq; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845207; a=rsa-sha256; cv=none; b=UCk4uxN/GLdy4iFjA+6GWjwc0ry/RDfCiSJCcSK8mdsyxuazAH6UDJaNj+5h0Q2EBoZKea TSfOuB6xrWDuGSkKE6Ek9eEX8+LWntxPyso45p2fbgxqfc2MYhPzJrr5e1/INQqVN9rR9d a+tPQ82zbGk6V5A7pnNeknR6oAu5O60= Received: by mail-qt1-f174.google.com with SMTP id fg9so9511178qtb.1 for ; Tue, 18 Apr 2023 12:13:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845206; x=1684437206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Yvli2ztOIxf68cKgZ+2R5Rhio4udTEYI5UCWBnx3AD8=; b=POy4P8vqdPiLeVSxt76e8jJPxRl/eozN6LB9E3d68P5jpjweAEUzrlWviGMg34u66M DX44ULcevGJYU1hFf1mgK6kd8RFLuTvhIJeohRXfelo0YUOwbJ2hiSsbpjgzctQtJMmk aB8zg6UGDs1xRfRvOwzO+9s5eIWHOX8MZqZ9UQkd5da4H1q7eayqraQwt5iMYeDJ3wrF 2BEBL1DvnoTbl0fLZxWCzY8F3sCBapTDuPbo3yEPRPlXgpKT8Aqj+HuVZ1Ex7zqT3+Rx 7Z4lWFsd7xtsn4IW/qycOjrNsmfsYyltylCVUG3/5TIBnIDCJksCo9fcXBl148+k7yVp 25HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845206; x=1684437206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Yvli2ztOIxf68cKgZ+2R5Rhio4udTEYI5UCWBnx3AD8=; b=kiotncAJzQSq3j8JvMGIxRMxTD8fq1mmooRMQgIMVJmYf+IFKxhgCbd3e4HfjNM997 NmHCDna+bE4jX3Q3RFZ9orrbBBuClMcHvIYKpJ1/dyiN6QVrUi01lCa1YDNZoNX9u7j2 f4Gd2sSFh7rYELz6wx/PsJ8u0hEBbc0yOmhMXmUAgLI5/GiCBmP5WmKak+3J0gzIcqX9 03nyCABH+23xF2+mnXFbKC12R7ppSSPJ6MAhdXcP6VuwkWIYFOXG1L+0lbUFhNswL73N maB3ocMSGwzyKuozFSrWhh1KE55a1D2KPf8Nl3l4yPo6RnpXmhBP79ya7FlVKHnkjdaX raAQ== X-Gm-Message-State: AAQBX9dlFGCbPEleO7ZoLyStuLZ+ukFOB1Xne4eMnalhmzk2nuWLi+8R +7cnXJTIFGhtFOhyHCv1Aux2p/xkU1Dgu/M2cWA= X-Google-Smtp-Source: AKy350ZdHSWRh7iDOoY7e2deWBswJWxcHm+J7bJRVccNErJ7bE6cTwx7vNM6q+rfQXee9INBqqd6Cw== X-Received: by 2002:ac8:578d:0:b0:3bf:c3be:758e with SMTP id v13-20020ac8578d000000b003bfc3be758emr1593777qta.16.1681845206131; Tue, 18 Apr 2023 12:13:26 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id p23-20020a05620a22f700b0074ad04609f8sm1217427qki.117.2023.04.18.12.13.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:25 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 06/26] mm: page_alloc: consolidate free page accounting Date: Tue, 18 Apr 2023 15:12:53 -0400 Message-Id: <20230418191313.268131-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 064C51C000F X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: egckzfw37x9terpiocpmtrue8wwffxfo X-HE-Tag: 1681845206-594581 X-HE-Meta: U2FsdGVkX1+ABqaJJFpswRWBYhFHnYPlcBacy8VmI8adI3rQ87nN3FqaFKCMRgBcEj1NEGGamWlK+78aUu+7mbKe7Zv4Ze49GklA73sR+19CAYZ8YPnhPLxG7m/vVmDHeohSCYheIgyOTpLlko3KHuuaZseEcSqjkFHvYGI4j1HXYkvje9Ys+Zt38nt4mXM7X3zEK9aYTzV/nHcAfv20yliBVQWetookegKBp8ooXTU3v32r+t4JtdytdbSnuUeIuc9JXFBA0LrtT3vmpK7u7D9BoUVgNXjvB587azGHbsQDg9QNA/5jd0t6u7r06SBj+2dNwNY5ywrUplYR3JD87ctdjTfJAf/QhBYV11AjDg2NBhG7oZlrUJeASjldMT5bCmGjlIifPZjTrwfI5rHcibddLL7Yhxk9XVnLo/JCtIn2k4K0ot2+U8/bWw23fFhYkhXjSIZ0KaAWWmnzj/qnbVZh8tQiGTBK27ckr6iboUXNsbV6bwo0fqlITWeAL9Yv9EHcw7MaF5dp3PM8kIjBi9blv6CaqzSCYwwhdMQq+S9ULSrti5SbuFJANVj223ngggUreVqlC48NKPRlTe8RgKhAW7k4HJsPLC9rHwXh5nmha1k1z09AJL5wWfK9cubc/wM45Fcd6cM6uXdDzRqhkiy5IUru6b/XH01Y3JkmCxjyUPqGjJpzNE83OwlAzFRcphPrPJhSzk1NQ1MIiBMeReGU77auPJ77maf/EHlZVrinwvyOkOrDJewXMWs4Xc6shtnJOupZxKeYMK10PWVs9PgN4FdOBKQPcVju3dT6mpNkzbttzbLaa2IQs6R777M0ERT8xbzyZLhkDA1G4JiJ2c2c6QDdkJXrHWK28CEnqNp9Pdtv2N7NU72lWPthT8/lYIkTG8TFBx1l6GnS7pEuF+TxgiRnRuMXof8fR34kqxVhcUNUb/w399uLS/MjcEHRL5SGGWdetQLcNeE4FN1 XwhWEa2q qnRuNuvaneMZdpQWZ2vUyfx+cys5Sm0yGM7YJL/FzkgM3fMNNeM+DDAdCjQbTbHiCc02Szsh8U7yYrm9oEXPMhtztDuXBLx30koU3sWNayubvyyPqM4ZDmfF36bJVMMGOn49zjJA81PlHVHPjaH8CKjX0uxUqB5hqyEExtrbU2U8/awLZsLBv2jvMA8YoflY/bMFqa9Y2CAI6dd2UTQaSs6Nl+aLgd8OLZCYRM67BGkjjXmBzc/X3LC+A7ewVEgVVF0V1E20ImeFQHNrEbhP2j7/OXbi6iaFcmkT/2UzOl8vNqMJP/q5reYHydGNg3N5ceZIASbeNiD0uLee9qOzVQSE43X7YvQmH5+eyQWBv15Ifh7Obz8hfnlkuQ4KHI6FFZZb02AoI1/WPnr9Z4rWH56LSzEROU1MCswjmMZinbAW3DH6baEcISepTvLiYGiPy/irUEW2UxXwrdvnR2MgrjNlkkrI1BxiGE235+vomYE0+ULnpbC77wf6nvEOZblDhiw+aQuZbkQWsWMSnZEKvCiXNhoZb5kpk3E91 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Free page accounting currently happens a bit too high up the call stack, where it has to deal with guard pages, compaction capturing, block stealing and even page isolation. This is subtle and fragile, and makes it difficult to hack on the code. Push the accounting down to where pages enter and leave the physical freelists, where all these higher-level exceptions are of no concern. Signed-off-by: Johannes Weiner --- include/linux/page-isolation.h | 2 +- include/linux/vmstat.h | 8 -- mm/internal.h | 5 -- mm/page_alloc.c | 153 +++++++++++++++++---------------- mm/page_isolation.c | 13 ++- 5 files changed, 83 insertions(+), 98 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 0ab089e89db4..b519fffb3dee 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -35,7 +35,7 @@ static inline bool is_migrate_isolate(int migratetype) void set_pageblock_migratetype(struct page *page, int migratetype); int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable); + int old_mt, int new_mt, int *num_movable); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..219ccf3f91cd 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -481,14 +481,6 @@ static inline void node_stat_sub_folio(struct folio *folio, mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); } -static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, - int migratetype) -{ - __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); - if (is_migrate_cma(migratetype)) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); -} - extern const char * const vmstat_text[]; static inline const char *zone_stat_name(enum zone_stat_item item) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..024affd4e4b5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -781,11 +781,6 @@ static inline bool is_migrate_highatomic(enum migratetype migratetype) return migratetype == MIGRATE_HIGHATOMIC; } -static inline bool is_migrate_highatomic_page(struct page *page) -{ - return get_pageblock_migratetype(page) == MIGRATE_HIGHATOMIC; -} - void setup_zone_pageset(struct zone *zone); struct migration_target_control { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d3d01019ce77..b9366c002334 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -843,7 +843,7 @@ static int __init debug_guardpage_minorder_setup(char *buf) early_param("debug_guardpage_minorder", debug_guardpage_minorder_setup); static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order { if (!debug_guardpage_enabled()) return false; @@ -854,15 +854,12 @@ static inline bool set_page_guard(struct zone *zone, struct page *page, __SetPageGuard(page); INIT_LIST_HEAD(&page->buddy_list); set_page_private(page, order); - /* Guard pages are not available for any usage */ - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -(1 << order), migratetype); return true; } static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order) { if (!debug_guardpage_enabled()) return; @@ -870,14 +867,12 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, __ClearPageGuard(page); set_page_private(page, 0); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, (1 << order), migratetype); } #else static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) { return false; } + unsigned int order) { return false; } static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) {} + unsigned int order) {} #endif /* @@ -994,24 +989,36 @@ compaction_capture(struct capture_control *capc, struct page *page, } #endif /* CONFIG_COMPACTION */ -/* Used for pages not on another list */ -static inline void add_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void account_freepages(struct page *page, struct zone *zone, + int nr_pages, int migratetype) { - struct free_area *area = &zone->free_area[order]; + if (is_migrate_isolate(migratetype)) + return; - list_add(&page->buddy_list, &area->free_list[migratetype]); - area->nr_free++; + __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); + + if (is_migrate_cma(migratetype)) + __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); } /* Used for pages not on another list */ -static inline void add_to_free_list_tail(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void add_to_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype, + bool tail) { struct free_area *area = &zone->free_area[order]; - list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + + if (tail) + list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + else + list_add(&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + + account_freepages(page, zone, 1 << order, migratetype); } /* @@ -1020,16 +1027,23 @@ static inline void add_to_free_list_tail(struct page *page, struct zone *zone, * allocation again (e.g., optimization for memory onlining). */ static inline void move_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) + unsigned int order, int old_mt, int new_mt) { struct free_area *area = &zone->free_area[order]; - list_move_tail(&page->buddy_list, &area->free_list[migratetype]); + list_move_tail(&page->buddy_list, &area->free_list[new_mt]); + + account_freepages(page, zone, -(1 << order), old_mt); + account_freepages(page, zone, 1 << order, new_mt); } static inline void del_page_from_free_list(struct page *page, struct zone *zone, - unsigned int order) + unsigned int order, int migratetype) { + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + /* clear reported state and update reported page count */ if (page_reported(page)) __ClearPageReported(page); @@ -1038,6 +1052,8 @@ static inline void del_page_from_free_list(struct page *page, struct zone *zone, __ClearPageBuddy(page); set_page_private(page, 0); zone->free_area[order].nr_free--; + + account_freepages(page, zone, -(1 << order), migratetype); } /* @@ -1104,23 +1120,21 @@ static inline void __free_one_page(struct page *page, VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); VM_BUG_ON(migratetype == -1); - if (likely(!is_migrate_isolate(migratetype))) - __mod_zone_freepage_state(zone, 1 << order, migratetype); - VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); VM_BUG_ON_PAGE(bad_range(zone, page), page); while (order < MAX_ORDER - 1) { - if (compaction_capture(capc, page, order, migratetype)) { - __mod_zone_freepage_state(zone, -(1 << order), - migratetype); + int buddy_mt; + + if (compaction_capture(capc, page, order, migratetype)) return; - } buddy = find_buddy_page_pfn(page, pfn, order, &buddy_pfn); if (!buddy) goto done_merging; + buddy_mt = get_pageblock_migratetype(buddy); + if (unlikely(order >= pageblock_order)) { /* * We want to prevent merge between freepages on pageblock @@ -1128,8 +1142,6 @@ static inline void __free_one_page(struct page *page, * pageblock isolation could cause incorrect freepage or CMA * accounting or HIGHATOMIC accounting. */ - int buddy_mt = get_pageblock_migratetype(buddy); - if (migratetype != buddy_mt && (!migratetype_is_mergeable(migratetype) || !migratetype_is_mergeable(buddy_mt))) @@ -1141,9 +1153,9 @@ static inline void __free_one_page(struct page *page, * merge with it and move up one order. */ if (page_is_guard(buddy)) - clear_page_guard(zone, buddy, order, migratetype); + clear_page_guard(zone, buddy, order); else - del_page_from_free_list(buddy, zone, order); + del_page_from_free_list(buddy, zone, order, buddy_mt); combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn; @@ -1160,10 +1172,7 @@ static inline void __free_one_page(struct page *page, else to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); - if (to_tail) - add_to_free_list_tail(page, zone, order, migratetype); - else - add_to_free_list(page, zone, order, migratetype); + add_to_free_list(page, zone, order, migratetype, to_tail); /* Notify page reporting subsystem of freed page */ if (!(fpi_flags & FPI_SKIP_REPORT_NOTIFY)) @@ -1205,10 +1214,8 @@ int split_free_page(struct page *free_page, } mt = get_pageblock_migratetype(free_page); - if (likely(!is_migrate_isolate(mt))) - __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(free_page, zone, order, mt); - del_page_from_free_list(free_page, zone, order); for (pfn = free_page_pfn; pfn < free_page_pfn + (1UL << order);) { int mt = get_pfnblock_migratetype(pfn_to_page(pfn), pfn); @@ -2352,10 +2359,10 @@ static inline void expand(struct zone *zone, struct page *page, * Corresponding page table entries will not be touched, * pages will stay not present in virtual address space */ - if (set_page_guard(zone, &page[size], high, migratetype)) + if (set_page_guard(zone, &page[size], high)) continue; - add_to_free_list(&page[size], zone, high, migratetype); + add_to_free_list(&page[size], zone, high, migratetype, false); set_buddy_order(&page[size], high); } } @@ -2559,11 +2566,15 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, /* Find a page of the appropriate size in the preferred list */ for (current_order = order; current_order < MAX_ORDER; ++current_order) { + int actual_mt; + area = &(zone->free_area[current_order]); page = get_page_from_free_area(area, migratetype); if (!page) continue; - del_page_from_free_list(page, zone, current_order); + /* move_freepages_block() may strand types on wrong list */ + actual_mt = get_pageblock_migratetype(page); + del_page_from_free_list(page, zone, current_order, actual_mt); expand(zone, page, order, current_order, migratetype); set_pcppage_migratetype(page, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, @@ -2606,7 +2617,7 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, */ static int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned long end_pfn, - int migratetype, int *num_movable) + int old_mt, int new_mt, int *num_movable) { struct page *page; unsigned long pfn; @@ -2633,7 +2644,7 @@ static int move_freepages(struct zone *zone, VM_BUG_ON_PAGE(page_zone(page) != zone, page); order = buddy_order(page); - move_to_free_list(page, zone, order, migratetype); + move_to_free_list(page, zone, order, old_mt, new_mt); pfn += 1 << order; pages_moved += 1 << order; } @@ -2642,7 +2653,7 @@ static int move_freepages(struct zone *zone, } int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable) + int old_mt, int new_mt, int *num_movable) { unsigned long start_pfn, end_pfn, pfn; @@ -2659,8 +2670,8 @@ int move_freepages_block(struct zone *zone, struct page *page, if (!zone_spans_pfn(zone, end_pfn)) return 0; - return move_freepages(zone, start_pfn, end_pfn, migratetype, - num_movable); + return move_freepages(zone, start_pfn, end_pfn, + old_mt, new_mt, num_movable); } static void change_pageblock_range(struct page *pageblock_page, @@ -2786,8 +2797,9 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, if (!whole_block) goto single_page; - free_pages = move_freepages_block(zone, page, start_type, - &movable_pages); + free_pages = move_freepages_block(zone, page, old_block_type, + start_type, &movable_pages); + /* * Determine how many pages are compatible with our allocation. * For movable allocation, it's the number of movable pages which @@ -2825,7 +2837,8 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, return; single_page: - move_to_free_list(page, zone, current_order, start_type); + move_to_free_list(page, zone, current_order, + old_block_type, start_type); } /* @@ -2895,7 +2908,7 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone, if (migratetype_is_mergeable(mt)) { zone->nr_reserved_highatomic += pageblock_nr_pages; set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); - move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL); + move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC, NULL); } out_unlock: @@ -2935,11 +2948,13 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, spin_lock_irqsave(&zone->lock, flags); for (order = 0; order < MAX_ORDER; order++) { struct free_area *area = &(zone->free_area[order]); + int mt; page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); if (!page) continue; + mt = get_pageblock_migratetype(page); /* * In page freeing path, migratetype change is racy so * we can counter several free pages in a pageblock @@ -2947,7 +2962,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * from highatomic to ac->migratetype. So we should * adjust the count once. */ - if (is_migrate_highatomic_page(page)) { + if (is_migrate_highatomic(mt)) { /* * It should never happen but changes to * locking could inadvertently allow a per-cpu @@ -2970,8 +2985,8 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * may increase. */ set_pageblock_migratetype(page, ac->migratetype); - ret = move_freepages_block(zone, page, ac->migratetype, - NULL); + ret = move_freepages_block(zone, page, mt, + ac->migratetype, NULL); if (ret) { spin_unlock_irqrestore(&zone->lock, flags); return ret; @@ -3142,18 +3157,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ list_add_tail(&page->pcp_list, list); allocated++; - if (is_migrate_cma(get_pcppage_migratetype(page))) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, - -(1 << order)); } - - /* - * i pages were removed from the buddy list even if some leak due - * to check_pcp_refill failing so adjust NR_FREE_PAGES based - * on i. Do not confuse with 'allocated' which is the number of - * pages added to the pcp list. - */ - __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock_irqrestore(&zone->lock, flags); return allocated; } @@ -3615,11 +3619,9 @@ int __isolate_free_page(struct page *page, unsigned int order) watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; - - __mod_zone_freepage_state(zone, -(1UL << order), mt); } - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, mt); /* * Set the pageblock if the isolated page is at least half of a @@ -3715,8 +3717,6 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, return NULL; } } - __mod_zone_freepage_state(zone, -(1 << order), - get_pcppage_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order)); @@ -9578,8 +9578,9 @@ void __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn) BUG_ON(page_count(page)); BUG_ON(!PageBuddy(page)); + VM_WARN_ON(get_pageblock_migratetype(page) != MIGRATE_ISOLATE); order = buddy_order(page); - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE); pfn += (1 << order); } spin_unlock_irqrestore(&zone->lock, flags); @@ -9630,11 +9631,12 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, current_buddy = page + size; } - if (set_page_guard(zone, current_buddy, high, migratetype)) + if (set_page_guard(zone, current_buddy, high)) continue; if (current_buddy != target) { - add_to_free_list(current_buddy, zone, high, migratetype); + add_to_free_list(current_buddy, zone, high, + migratetype, false); set_buddy_order(current_buddy, high); page = next_page; } @@ -9662,12 +9664,11 @@ bool take_page_off_buddy(struct page *page) int migratetype = get_pfnblock_migratetype(page_head, pfn_head); - del_page_from_free_list(page_head, zone, page_order); + del_page_from_free_list(page_head, zone, page_order, + migratetype); break_down_buddy_pages(zone, page_head, page, 0, page_order, migratetype); SetPageHWPoisonTakenOff(page); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -1, migratetype); ret = true; break; } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index b67800f7f6b1..e119a37ac661 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -183,10 +183,8 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, - NULL); - - __mod_zone_freepage_state(zone, -nr_pages, mt); + nr_pages = move_freepages_block(zone, page, mt, + MIGRATE_ISOLATE, NULL); spin_unlock_irqrestore(&zone->lock, flags); return 0; } @@ -251,10 +249,9 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * onlining - just onlined memory won't immediately be considered for * allocation. */ - if (!isolated_page) { - nr_pages = move_freepages_block(zone, page, migratetype, NULL); - __mod_zone_freepage_state(zone, nr_pages, migratetype); - } + if (!isolated_page) + nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, + migratetype, NULL); set_pageblock_migratetype(page, migratetype); if (isolated_page) __putback_isolated_page(page, order, migratetype); From patchwork Tue Apr 18 19:12:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB2B3C77B75 for ; Tue, 18 Apr 2023 19:13:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FC3F900006; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95C8C900003; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D534900006; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6D1A7900003 for ; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 31B4E160435 for ; Tue, 18 Apr 2023 19:13:30 +0000 (UTC) X-FDA: 80695460580.03.80E1D7F Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf01.hostedemail.com (Postfix) with ESMTP id 63F6D40018 for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KYPCQZ61; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845208; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=YiLzWQg1iwcE9twyW60ETWur33o3e8V1P81VJZlUVpgER9l5MondtRkniF/Y4FxnwUbdyE OyWg0a10V8xUiy2MBuGdBAbt5cP8BGo8vJJTLsXSzBm8qIOXsbbEORxVI4+r/ojuKfaQPt 3QybBOFyMhYBX5oFaIzpbmidQbxHp4Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KYPCQZ61; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845208; a=rsa-sha256; cv=none; b=GXcTKb26osYiFQMA1HNiEgJu2iKB9H/YxLxFw6oHw1646W5XMmg04zXMtj5P8Nn8QdFNmI yMz3HVQYeF2IyhmjkoZbuMmCisknZ/EHsdA83CZCvf90LoZzRWtAaaBIyOg6XMxUoZGXzU L+85ec/SMANPcc56JSvOoFKAMZ38CNo= Received: by mail-qv1-f49.google.com with SMTP id js7so10389667qvb.5 for ; Tue, 18 Apr 2023 12:13:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845207; x=1684437207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=KYPCQZ61APdNYs9D70nl3yomvGcC38mIMM4ac/0Jxqh0MQrsikJDyqz8kNAKqw9iC1 1bmnq4F1MqaGWpFLA8ADTd1KOLEiSxDVYMfswSH15y4crGrkoG6QG/Dr8TRu4HNJqULs 3ondml+qKSu04IkF3zQ8fUI8BpomkXLoFtEZTlNb5W8qR5dw6PS9O8KDT/9Kxlz6inFB Zoapu1IejZKECDPcQcl9q/KpgYWJmZLUxb2uAQzBruh9kz4IWlq+6bWZCm5vzL8tOPu8 21csNAuzKAdA/rLnXVEEXuAuDwsTm2WP87lOxN6ifFsa6PFpUcCMCZg4Et1TQULauXd7 sXdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845207; x=1684437207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=Kg+pxnLLw9ab6nKXassXWPagtQQ/eJ1vQ1sS6amYxDxLtI5Cj1kcc7jivDiBwr0Ngg AwQ1M57uHbENj5q3pPpxmA0tv9R7DCxivizABh8lIdSkHL+h3jyJTD477qWaVK/dm9Qi UHv5LKypKO/XmcIJ1Z29lcoPw+DrGUHkkQAwVMJgv9g6woRxxltSPK8ipdbsGcSt/rl5 oDihcg2XRzEMwGSLvi0p4DYVNix/HagtZTg8TzdANaV7hakdlYvgVyfdg+V95wXyaK7V rEpd3uPG7/6Ofuinxjx6HCgAMIXvL0bDHy3okg9XWWPZMY1Pan7AEztrKBDK8YacEwQF uLGA== X-Gm-Message-State: AAQBX9eRoBUd/CvzJWvftCneMnGKHKkT8kNfZ81wPOJCl20lTIxbXsvq tZCAbejy1d1eUvI4XSYYkzFJOZ73r8Kr3JZgN0c= X-Google-Smtp-Source: AKy350bRhkIUr8N7okrlgh3Hwo07F/vQhkNz4v1tFl3I8eV2V83V9lNwfkKYujFHpq4QwdXTdItkPw== X-Received: by 2002:ad4:5946:0:b0:5ef:9b22:dc88 with SMTP id eo6-20020ad45946000000b005ef9b22dc88mr5188244qvb.0.1681845207300; Tue, 18 Apr 2023 12:13:27 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id db3-20020a056214170300b005e5b30eef24sm3907199qvb.56.2023.04.18.12.13.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:26 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 07/26] mm: page_alloc: move capture_control to the page allocator Date: Tue, 18 Apr 2023 15:12:54 -0400 Message-Id: <20230418191313.268131-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 63F6D40018 X-Stat-Signature: sxmaspmbrqjcsfnce18u3ja5rbzkpcum X-HE-Tag: 1681845208-319466 X-HE-Meta: U2FsdGVkX18/FzbSX5mEyoaRTWYRVJN3zof7EYJsY7wTwK+FkSw0Ef+ZAiA8CBNNGsI+hDYhduS0fCFLX0hc/wNHHU3CY6TjUdOUhvN9R3JOrWAUywUs6+92RO2x9rii7pRCocEbWKL5pPhKNwohD9f35vYjfNew62LsE1/aIkeFMOSOWvR1eObtolIOZ9vD8JHc7cHpCaA48XTvmMSmtiRlTMEdTH/onkZO5AVL/F5NIaLBf2W8dKGU5z03e+gQ4vy9sPAVz9BHeC4DqTrnNF/GuuLqLiAnLuGjt8r+TbZsByJR4YJn5HGZMSC0zGa/w7eyMec9y5/JP36msHY2gDvXjRP0seOUyjJl6zjPvMRUGI+eZe7V6eASQ1Dvd1zwsWHg/kaSvsvMFz7Ar3zKLpgd6PjsZ4KG8+VCCZxr1Wpia10ROG7uGQ58zyYjVKCvQDk01NXlqpcrHNQLcZ8lz/WpGql2V9Tyg5RFOjfp9ml2m3Bxp0BTHeQpOULNn+oXHnwfuTn7juqrehOhJppzSyHFehj6DCJ4XcaZfLXWUXvlOV+5dk/ZDenUPj3j5qT6ys+sdZVgE4PMyRVFfiTxBVTBdBNX0ni+22pPQPuw6CS4jkqm9qeq/6muO4rT+tyHLoqe+Yc68OLCF2/doEI25gGMdlIQITI227OT9A9uxIrUuTDsfMjJOuwUIOPsz9+72A62l7ZHmgJAyyObQ1fM2cl4QzwFBOTnm1kmbdUbUHgnK8sB+/+mqqGafUPR6kdGhKbuyxOsKo/hz6ESMsJnS1tTW7nz0Ua2uzlND+nKi7lJKVyVkgz7AT2XSm/9zxgtIdd1e80JRGSpEQ+a+tA3DuyOju0vV4IlCXtbqIEHeMe53Wlt5Nvzz9UPpA/OzJesYoreuSDut8sbJqSDjxoT3d1SrP1qS7/+W8mcDXh0hrYpe4g+5geXazNdAe+VPWgv6g2ejMt+L1ARmx5qkSW 9HkDdbWg wut+4Fopk2jvZ6pXUgzTEoPHSBLFD9RR+8cCZfeD8po/Ocx/hXfCjl0Cw3DnIg9jsnjaQSHgYuuT8mMmiFQgSJjKqPX0Kzta4+o6HJ+N6GNyYjCe5t2D7J2JZYsObqzGtEEi3dOAi9RqdcBANe1KzMVA6eT9LfhNWJusI8xEMOuHkzeqQBfqErhD0NjESQdhtreqSOb2h9IoTVl6YCMuu2GKuLOQcbTUXiaRyueFotzs0+LAI4Yo2YxJg+sCwhN0jyvJ3BQL5sv7IB31WJppX+OOtt8f6DQKBpm30DIUPikBrBtq9n35sq4yFv8Y1BOlQBLKvIw1ivYHIW9pLd4NWl5EXNWugIf5iKqrnlAk14ODQ9SOAsS995mlrrRwQDPE3qUIWGZFuhSO4f7hHKmMziSj93oZGWtbDWf/9ABZGdqG9p/Z3iDMIymt8ThTnb/kXCRov5HbUVK8836XGOcy3WK5GpiOsaVUbuIGHr0135KFFUDVuLE3sBUwN9njq7OT+X74fg+Wjy+W7xRVhrdckaogGEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Compaction capturing is really a component of the page allocator. Later patches will also disconnect allocation request size from the compaction size, so move the setup of the capturing parameters to the "request domain", i.e. the page allocator. No functional change. Signed-off-by: Johannes Weiner Acked-by: Mel Gorman --- include/linux/compaction.h | 3 ++- mm/compaction.c | 33 ++++++++++----------------------- mm/page_alloc.c | 25 ++++++++++++++++++++++--- 3 files changed, 34 insertions(+), 27 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 52a9ff65faee..06eeb2e25833 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -56,6 +56,7 @@ enum compact_result { }; struct alloc_context; /* in mm/internal.h */ +struct capture_control; /* in mm/internal.h */ /* * Number of free order-0 pages that should be available above given watermark @@ -94,7 +95,7 @@ extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, - struct page **page); + struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int highest_zoneidx); diff --git a/mm/compaction.c b/mm/compaction.c index 84db84e8fd3a..a2280001eea3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2510,7 +2510,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, unsigned int alloc_flags, int highest_zoneidx, - struct page **capture) + struct capture_control *capc) { enum compact_result ret; struct compact_control cc = { @@ -2527,38 +2527,25 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; - struct capture_control capc = { - .cc = &cc, - .page = NULL, - }; - /* - * Make sure the structs are really initialized before we expose the - * capture control, in case we are interrupted and the interrupt handler - * frees a page. - */ + /* See the comment in __alloc_pages_direct_compact() */ barrier(); - WRITE_ONCE(current->capture_control, &capc); + WRITE_ONCE(capc->cc, &cc); - ret = compact_zone(&cc, &capc); + ret = compact_zone(&cc, capc); + + WRITE_ONCE(capc->cc, NULL); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); - /* - * Make sure we hide capture control first before we read the captured - * page pointer, otherwise an interrupt could free and capture a page - * and we would leak it. - */ - WRITE_ONCE(current->capture_control, NULL); - *capture = READ_ONCE(capc.page); /* * Technically, it is also possible that compaction is skipped but * the page is still captured out of luck(IRQ came and freed the page). * Returning COMPACT_SUCCESS in such cases helps in properly accounting * the COMPACT[STALL|FAIL] when compaction is skipped. */ - if (*capture) + if (capc->page) ret = COMPACT_SUCCESS; return ret; @@ -2573,13 +2560,13 @@ int sysctl_extfrag_threshold = 500; * @alloc_flags: The allocation flags of the current allocation * @ac: The context of current allocation * @prio: Determines how hard direct compaction should try to succeed - * @capture: Pointer to free page created by compaction will be stored here + * @capc: The context for capturing pages during freeing * * This is the main entry point for direct page compaction. */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio, struct page **capture) + enum compact_priority prio, struct capture_control *capc) { int may_perform_io = (__force int)(gfp_mask & __GFP_IO); struct zoneref *z; @@ -2607,7 +2594,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac->highest_zoneidx, capture); + alloc_flags, ac->highest_zoneidx, capc); rc = max(status, rc); /* The allocation should succeed, stop compacting */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b9366c002334..4d20513c83be 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -944,7 +944,7 @@ static inline struct capture_control *task_capc(struct zone *zone) { struct capture_control *capc = current->capture_control; - return unlikely(capc) && + return unlikely(capc && capc->cc) && !(current->flags & PF_KTHREAD) && !capc->page && capc->cc->zone == zone ? capc : NULL; @@ -4480,22 +4480,41 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; + struct capture_control capc = { + .page = NULL, + }; if (!order) return NULL; + /* + * Make sure the structs are really initialized before we expose the + * capture control, in case we are interrupted and the interrupt handler + * frees a page. + */ + barrier(); + WRITE_ONCE(current->capture_control, &capc); + psi_memstall_enter(&pflags); delayacct_compact_start(); noreclaim_flag = memalloc_noreclaim_save(); *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &page); + prio, &capc); memalloc_noreclaim_restore(noreclaim_flag); psi_memstall_leave(&pflags); delayacct_compact_end(); - if (*compact_result == COMPACT_SKIPPED) + /* + * Make sure we hide capture control first before we read the captured + * page pointer, otherwise an interrupt could free and capture a page + * and we would leak it. + */ + WRITE_ONCE(current->capture_control, NULL); + page = READ_ONCE(capc.page); + + if (!page && *compact_result == COMPACT_SKIPPED) return NULL; /* * At least in one zone compaction wasn't deferred or skipped, so let's From patchwork Tue Apr 18 19:12:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7C9FC77B78 for ; Tue, 18 Apr 2023 19:13:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BC5C900007; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16C1B900003; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F281B900007; Tue, 18 Apr 2023 15:13:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D90F0900003 for ; Tue, 18 Apr 2023 15:13:31 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A994F1A0443 for ; Tue, 18 Apr 2023 19:13:31 +0000 (UTC) X-FDA: 80695460622.16.F29141B Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf10.hostedemail.com (Postfix) with ESMTP id AA70FC0028 for ; Tue, 18 Apr 2023 19:13:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=duezlBdf; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Fg7akH6DcyqQI9O6hhnfSSKo6TLLXotHGx/ELhTsVs=; b=3JJ8VKfOAabhfioHB6GoSODWHIo/WdFZLqVWXxoisat1oWAXxcUQLCCrl7cUgaPuKB1cDb zmoOaqZXeTmteY/bOK5TtTDCYBNB6TPNPPltzL1NiIryH45+5nwhbtRnPbVW36JoWwgQ9p D1N/hF40YTav+5TC9Ygzm2GL9tPGjX0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=duezlBdf; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845209; a=rsa-sha256; cv=none; b=LfI6Xl6Qe4dDpZaQVeNoJEbVgO3Rh2c2K4YjxdaYxzcUAXlJTw5yo/eAJ09NA7iaggPKpv pDCToDOmtsqX+0hVP2GDb2fEVqppqisgCqS+CKohdsFXvdB2EZmgQ3bbeiDi7YxDZdpLDA 2Pp35lTYyRt9WTa0k1iue+jYBakBJPI= Received: by mail-qv1-f46.google.com with SMTP id h14so18464044qvr.7 for ; Tue, 18 Apr 2023 12:13:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845208; x=1684437208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Fg7akH6DcyqQI9O6hhnfSSKo6TLLXotHGx/ELhTsVs=; b=duezlBdfFiEByRb2UhLuZ0Hbbz17B9Qr/emODLH8mZ2CGMjptqHGa11W+QzdSjQ/7F NrpNC/GaqkJEOfmZT8IQhhmClT+39ZedrNCcu8ABafH+q5j+YOBKEgdjdxPGtGKwAbo+ P2TEzDioWeN451luG7HEJwPMt/rOzqsoER6GOlwhz2qDDjhKRGqWoYpeyDSp91a0bKPf 25r1qZ+NsVNcaUBJQLFKjSOaPj15VqfhoKvKXLa7AyFqKWsmjO1Nkx11EK8FkuJho1GV PlZj02YYGkSrKmCNJW7AgCuXuYglqYWYf5+G3gp8PSNESKrtKbSuyF4B4khcnFe0QCvs SdMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845208; x=1684437208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Fg7akH6DcyqQI9O6hhnfSSKo6TLLXotHGx/ELhTsVs=; b=EftrhfBEmnUb3+DfInkgFj8096rTWzVttHonaA6BIjNb7ywLfMa9OY82j7INp78LdI nse33bTzF5EaLeY54SL8UxMbm20XosM6iAHPfISbSPXaHWgZp5zjf/R2VjeLmxePHQtj QRnhg87p0cChWNhxOo7+oaBVIHjpyQpnh7YZelYi2/ArkOZuKM0GrtwREmv1olbvge6p WyRGEjGJzRDizwO1KTaNK63H6cQwJ7QsxqXNFg0FlkdTgE6Q+fboRqKv/tkV3NbvkcI0 u+eQ7DHM+Jx81saYSz8zTVqSFcPfqaMSA/dMnSUC7rVqLf/VbVEmBiH141R7jcbkBKlZ JXYg== X-Gm-Message-State: AAQBX9cEGN2vbUrnGJmYy/+iu2hk7/htSfpSxgtv3UhBtU+H1nLIOsle aW7uGOPuHuh9uxPMi1HANTyXX5nn0o56RXi59wc= X-Google-Smtp-Source: AKy350aBP+4ihxJVgdmom0s8xvFh5tp4l/UPrJZnbm1VerN0bSQuYJNYtHePgve1c8xb8nyNL5IrOA== X-Received: by 2002:a05:6214:252a:b0:5ef:8004:e0b4 with SMTP id gg10-20020a056214252a00b005ef8004e0b4mr11443725qvb.48.1681845208456; Tue, 18 Apr 2023 12:13:28 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id c2-20020a0ceb42000000b005dd8b9345e3sm3924742qvq.123.2023.04.18.12.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:28 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 08/26] mm: page_alloc: claim blocks during compaction capturing Date: Tue, 18 Apr 2023 15:12:55 -0400 Message-Id: <20230418191313.268131-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AA70FC0028 X-Stat-Signature: mqmjgfa87crcbibxcgmx61azoww44gup X-Rspam-User: X-HE-Tag: 1681845209-747621 X-HE-Meta: U2FsdGVkX1+qQSVByXHwnGYQG9wCMqw7anTz14Bg8s+ekKzj077WLK6yAJgfwX3npwCWQnEeSrbAqO8QheZeA2tLFxVuTXmch0t4nqE9K3ev6gCJiLvI7NHYQZvolQFKbJYi4NSA2ADl1OAZU9/+uE1ndn3/UE2gusqronD3j+G3OaEl1H6MJ1yBOhkA63h8OUFClKVyPiYGT96ZUky4tYahB8LTjCq+mBPQ+2gvFc7YL5OxG5Or88CDZZ9OsGW90Ga59j/iayItqFfgEnfbqUHGTYyyEQs6XHX4AHjqU/zAlSPBEo6odRDFzULlguJJFfgJf88SBD/X5ScXE8JnGZpWS0AcIbFGde4ud1204Ls4tgVSW8wKfc9n+1bos9ktwyCelUggEF87NL9XKMpK2sLkzaqNKYGxwMQ4CbrQBhrUTq6W/K2cxbYAfhD7N0+SA9J0ftd3Vmt/gveYIEyZmlLgGe4tBS8zgaexocAde1QD6JoIF9VyPNU7PurOnLMUpa1sUtkrXtI0Y/46wg3qi4640LlIBAse7lVNbfgdeq+zObnKUkMONC053zCg3yW958Mw4mVI4w8Y0lwCr0sDo6b7SK5wiYvlzGgtw6QVz3ZxC5GqnUr6S5zih35/4NEvoyhaUzEZHOtKMuAPPYdVEunUQsLCjJDlJf5JurJpT7JAI+aU63FIo6KrVZz1jn2sVv9SDsf4xxQ7NxaBFCV7Pd3bhhGu0D0eITqUXxwV5Jhp4JTIT3CcE5JqoITsY62XHsn97d1DhPcMnIlVqz1HJ8DCNgH2pEOnK4ukRn6RBNJSH7ZWD2RPJOz1haVoQnCk4hinR5y6JJf0iqrJc3b6T70yL0Ah8Ar5pODwd6mlvkQ3l8orKAOGtxKwXXrHtccJJYKtS8Jogx5RUvSwL1I1ZbkfGqHIuwfoQdQwMSzTgfZCosZLyWwGdrvlAt4ciWc4745F4CAn31STcyxNu4k rVem4Doy Hmdtp/HRJaf8akmP4Xuhespg9oMjKVlfveUUXjxyTt5wJh8MlCFXjf9prCMSJa7X+/5HD1WqHvHG9Yzk2HuNDea1TfAGC1Mjp9ewbw38PnUQTqE0SY0FJfcjrCqB0jjgaMqi6XWJnVoUbHoJhfM8hnBox5xUH9G20MoYhd8ljB58hWJ3dr9U5r+ZgIavsDiXuARnWYNAvcIbAjJEiGHvZbhhrF7AhtlUZeglUL7K18qCgGvCkvtyvi4+rQlf28X5fzRY1KJt/YwlVCt9Q6j7mUL/KxVEBOkJMK89UtTpBYHpBHo24eG5+qJKe2pT29A+lmuTvpfPouET/+TXBv6oTKkt3/+VFnAoWgLDuhNGxUJJ4BN0qGDMgnpFncvwvVV4Ba/bv2Ewdevd7BzLFXMpjip6kxlVT+Ssq/ZqtGiePFT7xduzQfGBXbepO80hMef650/VmIpfeLm0H+B/RYCqJr1A2tf1Qt1W8MnxFLU+hQXNO9YzPAe9Nxx7cpqrXHUkZmPgUECmnmajzWdmuSDzmJF+vOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When capturing a whole block, update the migratetype accordingly. For example, a THP allocation might capture an unmovable block. If the THP gets split and partially freed later, the remainder should group up with movable allocations. Signed-off-by: Johannes Weiner --- mm/internal.h | 1 + mm/page_alloc.c | 42 ++++++++++++++++++++++++------------------ 2 files changed, 25 insertions(+), 18 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 024affd4e4b5..39f65a463631 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -432,6 +432,7 @@ struct compact_control { */ struct capture_control { struct compact_control *cc; + int migratetype; struct page *page; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4d20513c83be..8e5996f8b4b4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -615,6 +615,17 @@ void set_pageblock_migratetype(struct page *page, int migratetype) page_to_pfn(page), MIGRATETYPE_MASK); } +static void change_pageblock_range(struct page *pageblock_page, + int start_order, int migratetype) +{ + int nr_pageblocks = 1 << (start_order - pageblock_order); + + while (nr_pageblocks--) { + set_pageblock_migratetype(pageblock_page, migratetype); + pageblock_page += pageblock_nr_pages; + } +} + #ifdef CONFIG_DEBUG_VM static int page_outside_zone_boundaries(struct zone *zone, struct page *page) { @@ -962,14 +973,19 @@ compaction_capture(struct capture_control *capc, struct page *page, is_migrate_isolate(migratetype)) return false; - /* - * Do not let lower order allocations pollute a movable pageblock. - * This might let an unmovable request use a reclaimable pageblock - * and vice-versa but no more than normal fallback logic which can - * have trouble finding a high-order free page. - */ - if (order < pageblock_order && migratetype == MIGRATE_MOVABLE) + if (order >= pageblock_order) { + migratetype = capc->migratetype; + change_pageblock_range(page, order, migratetype); + } else if (migratetype == MIGRATE_MOVABLE) { + /* + * Do not let lower order allocations pollute a + * movable pageblock. This might let an unmovable + * request use a reclaimable pageblock and vice-versa + * but no more than normal fallback logic which can + * have trouble finding a high-order free page. + */ return false; + } capc->page = page; return true; @@ -2674,17 +2690,6 @@ int move_freepages_block(struct zone *zone, struct page *page, old_mt, new_mt, num_movable); } -static void change_pageblock_range(struct page *pageblock_page, - int start_order, int migratetype) -{ - int nr_pageblocks = 1 << (start_order - pageblock_order); - - while (nr_pageblocks--) { - set_pageblock_migratetype(pageblock_page, migratetype); - pageblock_page += pageblock_nr_pages; - } -} - /* * When we are falling back to another migratetype during allocation, try to * steal extra free pages from the same pageblocks to satisfy further @@ -4481,6 +4486,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, unsigned long pflags; unsigned int noreclaim_flag; struct capture_control capc = { + .migratetype = ac->migratetype, .page = NULL, }; From patchwork Tue Apr 18 19:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79B38C6FD18 for ; Tue, 18 Apr 2023 19:13:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E926900008; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 870EE900003; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69BDE900008; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 54A6E900003 for ; Tue, 18 Apr 2023 15:13:32 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 35226803FB for ; Tue, 18 Apr 2023 19:13:32 +0000 (UTC) X-FDA: 80695460664.10.C6CEA9D Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf12.hostedemail.com (Postfix) with ESMTP id 6149840014 for ; Tue, 18 Apr 2023 19:13:30 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=tASioUHS; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3ziCYFLP0gIvrTMKVGAOENHj/570OA5tLu/LhfprcHs=; b=2t9ZcNNUaw1Ek6urcUuzCAH1i9GP1S71PZba24uDz90CNjON2pJJo1za8zGhhIehoiR0Ni NZbAPjY8QOIkYa4t2tbOWpn5b50ziLKf8OaBU51EXGa3hkcSMSGraOKa0q43qZT/gNTT2K z3vVJoid0XbBiuD9yXPkofWZEUZ0yPI= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=tASioUHS; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845210; a=rsa-sha256; cv=none; b=O2X31uZvhdUI7e0pxe3KAIJPa00moLQvQihTG8Lt8CQ5TyTOa0K5KWGTXSrjkPDmHowOr1 Gv2kjIl6dZF44Fghh3ufJliZJk1pDl13di6d3WR/fS/i3KCHrH3CZ//BJmTWAu6zFZsDJJ fBrJFHfr0lP4HumZQQA/b8fJvv1FKcI= Received: by mail-qv1-f48.google.com with SMTP id oj8so1343027qvb.11 for ; Tue, 18 Apr 2023 12:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845209; x=1684437209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3ziCYFLP0gIvrTMKVGAOENHj/570OA5tLu/LhfprcHs=; b=tASioUHSg0O8JLU9lmpT2o9qieQ5UPbSCjZ2FFf4DPmc+sm1lG5E2sefokRjX82PWv /naa0xXtkW6heFV0tkRw9s2v57ltr8jqTBis+P8WeOGGhuTUTbP/NaZa2KY0naFlxm+B W0CWp0uB41dkHuQ1/kpuzy+c28wZgenF79DWz7OZV3qE+8W75G/CgO9SPnK2fL9a320t yXKyc58RVcAaAikakCIJxYNwp+SzurktP4fqmo2U9wFIB3ZelkMdhT468zPi5iN0Uonq qlshkMrXESVN82SilYYT0M1b22r6Yho1HwmEKjRb0CLDZMxgSBPEuAYI8Xtu54okMtlA dd3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845209; x=1684437209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3ziCYFLP0gIvrTMKVGAOENHj/570OA5tLu/LhfprcHs=; b=TAo/9NglGmwenbQ9zTcUsAS+Wb6OjFYSDjU3b+gdtukiO9ODhbQtWB+OerJlptxpZQ 5VEkLrZfGL2in192A8A823hBNLVhezC0soTq2JXiX8RyEbOsSwO9CKiFu24havDFvujj oe8o/FC4EOl4rSqn68okLxGexio+GR35jYBm2aFLZJD2jKwAyx9Nu43UM0vP+B5PbIr5 Sfqvv1/NUmMen8eUZymE6eAypOd7Jr4wFa0RkaaZzWCDi1/ObEbI0NAcWvX9ZynBVbSH 7XvX7qI/LNJqk/naWVHku61ptYi1k46y4zS08kibVbIufggsHHs1HcLVH9rS6mja2Eg+ ckAA== X-Gm-Message-State: AAQBX9dLu3vruqskKC3/+bEHrOBI2aeDiy4H/U7PQURKQjshh7+KBJE+ pxsD3s7/tFd3NEl7tg0Fgh8LOxvwDQNGkKEkcUA= X-Google-Smtp-Source: AKy350ZOi+88hEvm9GEFxPHg078nMJgdO2hJcrbQjNkq+nM1p74G7tq0XzGJcBNJQ1SbVH8WMthykg== X-Received: by 2002:a05:6214:f23:b0:5ef:77c4:4540 with SMTP id iw3-20020a0562140f2300b005ef77c44540mr16271521qvb.27.1681845209552; Tue, 18 Apr 2023 12:13:29 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id dr3-20020a05621408e300b005dd8b9345dbsm3884921qvb.115.2023.04.18.12.13.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:29 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 09/26] mm: page_alloc: move expand() above compaction_capture() Date: Tue, 18 Apr 2023 15:12:56 -0400 Message-Id: <20230418191313.268131-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6149840014 X-Rspam-User: X-Stat-Signature: hwk4ppedyn6hthi5yizjeoqp44uygg84 X-HE-Tag: 1681845210-434455 X-HE-Meta: U2FsdGVkX19zVpH4iJ7rPU0mduUv7RHMbaYjppB+Sk6h3Fo5cWmfpVLTJR8PMcvBuFOT3l5DD4QgQmK2V4F4SdK95DT5PrjmXJHhsKKLPGqwx3w4jOn5j03aCfePjILA6zdkE7zqsP05RfS28hoFKaCgPZBqHSA9QPf830Dh+ihNXX7uvZqlK4/AvMv8HNqV0gUTUTs3i/w1cv3sqiXRqb4JgjFqU4veFmIHpwFTsWrqvCwYSyKuRIfICDXmm20I5khrOYnwDrf6JiU8oLiFpvJUcm9Tk5hHxgvl0MeyNRgJaPmjcp7eJ4iqt5LHLYUOuS7VTdlZRIeaSq9lxnV49AZlw1TrL4fzsh+8rtqAID9byhNG9cTaVZ0JHfaGmuAJGJGvnN4ro0WLXooug2OZPwqNR60f+G/snuRFWVSGUuweBbWK8y9zEb5MjD+P9TlprHy5fMAlRarCk3uerH1m9l6VNxVL4YviG7jjHfgPijmeg16/pZhDjxmarkwyl3499gOhx48+5lNlt138mzfRReUXt7ygHd7gQYexhyE4AWzs/MrGElkW2UnoYz3gStDh+iWuBC5sHDJ64E7m1jTdPXddue4cGZhulIYeRAXlwAaGjeKUCMpTDqeB0g2cp717V9C7mnGC2DU+aDMb9VjiLyj0mzuQ6hkznoP+kcAsiTrUE7cAtar3fzpXiczdQ3e11vJ7ZkrE/NLwxfeGWJOp/Fhiq92gG0bpJFZAx516RHL0JE3j6aeqk//huc2MVU63IfTaDTVRdmB+foBYPwDymQUwglu39lpt4kndeM8US1Kv//9GgasWzyjz0eDs9B9kz4bdhrc0N/7R6qfe1/3lSPlZunzhLnzYTrin5XPKk/yJlrNEC5ykaYjfpeVIsOTihzPz3+yGOgQ3lhz2r2Tq3Nz1kBVKUjjjUsVF72MBZIHssW+XnBTXLqovHi0Y0RTHvOaojBL9weOz3G9gcP2 EYZXBnkp LWll41tDoEF/xeqPULezpNdlsUSgLD474pfrp6vcQORG5Rjx03xHyZbwhdHB6tW0Zsel0B2kqyU6wbjRrxrTbMD4p0WAvuMCKr9dNUA1o0/OCxodpVbicfbyOqY6S6tkC6keBU0t0YeaLKi4R+eGuKyY620ugLgZ8UftIAyuoAYHG0qNOqrBmL8Afbv1Qt2n9Hsh3Go6lhuppN88flYud2fS21/wNzlKyWl7AI/6yav678EpHrpIMFvtZ3nB0ID9GAJByMGvPstLFcy5NarqYT6A8v19ros8pxRnJyWUHaC5RnRMICDCJd72D8Y0emObC1hxwrCZ2dAcJf0EyTOHEvG1/IF8qexTX/cR+dfGwi7/Fj+2cmkZC6LWhy5y/PoYKZW/jAxRBCg9T1YxgogIPJht26zQqf8fKLe/qPxo6Bc/Eh1+ey5ACekR9BIudefTqLmtNdsNTzziy5dbuwYZFL0Y9THBZxvSOt14aZdxTxwIvS8KrVbdnP12y1JKOnvf/e9pWaEEKiVFTY0S2wgGS0M/NIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The next patch will allow compaction to capture from larger-than-requested page blocks and free the remainder. Rearrange the code in advance to make the diff more readable. No functional change. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 186 ++++++++++++++++++++++++------------------------ 1 file changed, 93 insertions(+), 93 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8e5996f8b4b4..cd86f80d7bbe 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -950,61 +950,6 @@ static inline void set_buddy_order(struct page *page, unsigned int order) __SetPageBuddy(page); } -#ifdef CONFIG_COMPACTION -static inline struct capture_control *task_capc(struct zone *zone) -{ - struct capture_control *capc = current->capture_control; - - return unlikely(capc && capc->cc) && - !(current->flags & PF_KTHREAD) && - !capc->page && - capc->cc->zone == zone ? capc : NULL; -} - -static inline bool -compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) -{ - if (!capc || order != capc->cc->order) - return false; - - /* Do not accidentally pollute CMA or isolated regions*/ - if (is_migrate_cma(migratetype) || - is_migrate_isolate(migratetype)) - return false; - - if (order >= pageblock_order) { - migratetype = capc->migratetype; - change_pageblock_range(page, order, migratetype); - } else if (migratetype == MIGRATE_MOVABLE) { - /* - * Do not let lower order allocations pollute a - * movable pageblock. This might let an unmovable - * request use a reclaimable pageblock and vice-versa - * but no more than normal fallback logic which can - * have trouble finding a high-order free page. - */ - return false; - } - - capc->page = page; - return true; -} - -#else -static inline struct capture_control *task_capc(struct zone *zone) -{ - return NULL; -} - -static inline bool -compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) -{ - return false; -} -#endif /* CONFIG_COMPACTION */ - static inline void account_freepages(struct page *page, struct zone *zone, int nr_pages, int migratetype) { @@ -1072,6 +1017,99 @@ static inline void del_page_from_free_list(struct page *page, struct zone *zone, account_freepages(page, zone, -(1 << order), migratetype); } +/* + * The order of subdivision here is critical for the IO subsystem. + * Please do not alter this order without good reasons and regression + * testing. Specifically, as large blocks of memory are subdivided, + * the order in which smaller blocks are delivered depends on the order + * they're subdivided in this function. This is the primary factor + * influencing the order in which pages are delivered to the IO + * subsystem according to empirical testing, and this is also justified + * by considering the behavior of a buddy system containing a single + * large block of memory acted on by a series of small allocations. + * This behavior is a critical factor in sglist merging's success. + * + * -- nyc + */ +static inline void expand(struct zone *zone, struct page *page, + int low, int high, int migratetype) +{ + unsigned long size = 1 << high; + + while (high > low) { + high--; + size >>= 1; + VM_BUG_ON_PAGE(bad_range(zone, &page[size]), &page[size]); + + /* + * Mark as guard pages (or page), that will allow to + * merge back to allocator when buddy will be freed. + * Corresponding page table entries will not be touched, + * pages will stay not present in virtual address space + */ + if (set_page_guard(zone, &page[size], high)) + continue; + + add_to_free_list(&page[size], zone, high, migratetype, false); + set_buddy_order(&page[size], high); + } +} + +#ifdef CONFIG_COMPACTION +static inline struct capture_control *task_capc(struct zone *zone) +{ + struct capture_control *capc = current->capture_control; + + return unlikely(capc && capc->cc) && + !(current->flags & PF_KTHREAD) && + !capc->page && + capc->cc->zone == zone ? capc : NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, + int order, int migratetype) +{ + if (!capc || order != capc->cc->order) + return false; + + /* Do not accidentally pollute CMA or isolated regions*/ + if (is_migrate_cma(migratetype) || + is_migrate_isolate(migratetype)) + return false; + + if (order >= pageblock_order) { + migratetype = capc->migratetype; + change_pageblock_range(page, order, migratetype); + } else if (migratetype == MIGRATE_MOVABLE) { + /* + * Do not let lower order allocations pollute a + * movable pageblock. This might let an unmovable + * request use a reclaimable pageblock and vice-versa + * but no more than normal fallback logic which can + * have trouble finding a high-order free page. + */ + return false; + } + + capc->page = page; + return true; +} + +#else +static inline struct capture_control *task_capc(struct zone *zone) +{ + return NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, + int order, int migratetype) +{ + return false; +} +#endif /* CONFIG_COMPACTION */ + /* * If this is not the largest possible page, check if the buddy * of the next-highest order is free. If it is, it's possible @@ -2345,44 +2383,6 @@ void __init init_cma_reserved_pageblock(struct page *page) } #endif -/* - * The order of subdivision here is critical for the IO subsystem. - * Please do not alter this order without good reasons and regression - * testing. Specifically, as large blocks of memory are subdivided, - * the order in which smaller blocks are delivered depends on the order - * they're subdivided in this function. This is the primary factor - * influencing the order in which pages are delivered to the IO - * subsystem according to empirical testing, and this is also justified - * by considering the behavior of a buddy system containing a single - * large block of memory acted on by a series of small allocations. - * This behavior is a critical factor in sglist merging's success. - * - * -- nyc - */ -static inline void expand(struct zone *zone, struct page *page, - int low, int high, int migratetype) -{ - unsigned long size = 1 << high; - - while (high > low) { - high--; - size >>= 1; - VM_BUG_ON_PAGE(bad_range(zone, &page[size]), &page[size]); - - /* - * Mark as guard pages (or page), that will allow to - * merge back to allocator when buddy will be freed. - * Corresponding page table entries will not be touched, - * pages will stay not present in virtual address space - */ - if (set_page_guard(zone, &page[size], high)) - continue; - - add_to_free_list(&page[size], zone, high, migratetype, false); - set_buddy_order(&page[size], high); - } -} - static void check_new_page_bad(struct page *page) { if (unlikely(page->flags & __PG_HWPOISON)) { From patchwork Tue Apr 18 19:12:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 474ABC77B78 for ; Tue, 18 Apr 2023 19:13:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE08F900003; Tue, 18 Apr 2023 15:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB908280001; Tue, 18 Apr 2023 15:13:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3158900009; Tue, 18 Apr 2023 15:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9AD15900003 for ; Tue, 18 Apr 2023 15:13:33 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6B845AB76E for ; Tue, 18 Apr 2023 19:13:33 +0000 (UTC) X-FDA: 80695460706.27.73BC47A Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf30.hostedemail.com (Postfix) with ESMTP id 9F5738000E for ; Tue, 18 Apr 2023 19:13:31 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=wn0fopvG; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845211; a=rsa-sha256; cv=none; b=bw5mY9VnVmzTBWWO2ZzoYtk1rpP8crhWr4t4LwM/W0d7uFpx4F8p6rHSwdiD8nNKjPbFWX gkFZd91s97z7poJsozWMRwBeNfNCLZoHIJVwYkRKyDbF0KB8gp7Z9Z7poXVQCGaD33nXIq h1JTpK431fECFHkZpq5AhoR2Mcxxi50= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=wn0fopvG; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W40OmA9FJpZ9KM8b83NoKn4oLbbGXwdSocdEiJSdSOk=; b=RcCCF77TIflkhch4JxvqFMvE5XkD5iVauySX6kSmxFjrLl0JeMudMQv6njFkbKESXOJ8oV IBEImHhvPlhwtUcK8h8NUunK5jUuE7A531aDpdfJruW/Nchwe01sFJypXALMBm5inGJKyb p+VgzGDlF4jnL46EvjfI4lyZYlNqUOs= Received: by mail-qv1-f50.google.com with SMTP id qh25so19655948qvb.1 for ; Tue, 18 Apr 2023 12:13:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845210; x=1684437210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W40OmA9FJpZ9KM8b83NoKn4oLbbGXwdSocdEiJSdSOk=; b=wn0fopvGsbR3i6PvjhCvPLQ1N5JWQrSjYBsgbrLuF6738z7mGxzLK6h6jbm6MmLsmJ pF8n/O/stCPiQEof58dQGBFCvd4zD48BHZayas7tdYm+iNpqPr5K40ekJd/L+fqSbnWx dF76tgKVdT587CaFRK8QZjlJ0JQitDXtdsiSxO7LH78byqCngXvJYUfGQXVnbLa3i4e/ Kwi9t5fN+cwPv3JGfkRVdEAeRey7ve3x28vGryxxL8HGqMYxVgzfDZIdBf6OLivjGG2u 1lZUigMFLGf6JwZyC57zeYywVprLLuC3qi6qYNpVO4TuaZDQqSJWnTgv3OsLRhreN2Jg PPfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845210; x=1684437210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W40OmA9FJpZ9KM8b83NoKn4oLbbGXwdSocdEiJSdSOk=; b=AoAx6+Vfo5khwdXhZbLsEuZ08F933iOG4o8y+GSfKux52bN9FAZhZHwjcgpMqgxUHs SLVbnsjfrX0/Cq8R0OEN1GYcJIqBEyv3400fAJyvcnI3Qc/l7Oem68rz/UVrRzou33zm mL8lH3ZKJA/kdZWXYs99KMBuSjWg/3PUfyHwsSse3CsN0R0lXTzNs4oZ1ZUY9ryKvq8G wB+P1bPZIVbvqIrK2fthvxUMm2NNFVdKIi4YPZ9Z0975UD7Ig3UcK5jNh13KYWwNkhfk ndPPUmJczuhF3wRnxGl6mTB+5vh1mMD+wtryl7rknw4MYztG07fnp66zQKLsFwAy7dQK HzPw== X-Gm-Message-State: AAQBX9f1e2qIDWOb1Qv44685xkB5JqZdKGNSdavchsR+8Q4jrFO5ts0k RGUuvXvqEB4gixhPHCJz/j2A810LqanQpZBo4uA= X-Google-Smtp-Source: AKy350ZdXxpY1FsQYWCci5zoHDWqXfr/R4sK4OR7K8k8+oSMxZVw0C1MOwCqOi9gSXpDKZHblUbJiQ== X-Received: by 2002:ad4:4ea4:0:b0:5ea:6a2a:140a with SMTP id ed4-20020ad44ea4000000b005ea6a2a140amr26931362qvb.16.1681845210696; Tue, 18 Apr 2023 12:13:30 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id r1-20020ac87941000000b003df7d7bbc8csm4249673qtt.75.2023.04.18.12.13.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:30 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 10/26] mm: page_alloc: allow compaction capturing from larger blocks Date: Tue, 18 Apr 2023 15:12:57 -0400 Message-Id: <20230418191313.268131-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9F5738000E X-Rspamd-Server: rspam01 X-Stat-Signature: xzez1169ryk1cnh6rdqftecz14zxockq X-HE-Tag: 1681845211-684320 X-HE-Meta: U2FsdGVkX18p+t68xTs64ZL+rs2sj3uKsKCj7Uw8zJzxorDBkJ4VmRVjxVUMpQ810Rft2SwKH3138J140B0HcpQ3fBjWISZrmdDFDZl+7qLPMawuZyPIaKvdVo9df8U9IESXg3hm3qLiQJKNZF8J5PLXeqY3vokicI9vht2QebCKfFuDyRBgakGCLr42b3CFDzywg7w0Fee1l1gmKz/MYblR4WLiBS4sOJNuP5xQJ3Htk9u2nro4TlE149fg3de2nX7fh2uSRE3/bicbgKWQAlvTHyus5PMzC+/xfWClpXHVDdettpvulybSwAms+svgTmN9Tl0DXJSSJG/w1w9Qfyain4s/hILlLCjSu5Wh9wputAsPMJZ4qCGxpUYRbZ746ftG8k2NGYKmHJuLgE06nSX+CaXXilF7xgAysvgbJSbdvMZUybEosTcRuumSyreQYkNLauKdbR2EJYq1O2DLtKL8r/NhEqdBj3noZmmW32wrHtiEEAzgRoq+eZcN5VTpp9tju/aFq7gtJUmfBODbaCK9fgYTu22SxywHg5FLAYqgQSCDPh5WVcV2Kk+QZXa+rUzwI/DgcPI1p5vR7jVngzh3HCZfggRenhCkegoGSB5WZnfu3f1i5Gi6FCiQCDUKonAr8JJy2/61Fiiu2OO0PUq/UPVkFEotgmqWAgJJaYdoPk3nYB8cAbKNbhp6sICJGvwHcciARSsCKrVPr09Q4IfUN616FS0dT8hrNNd2ZYh25eq1dBxVYvUbnIdz0U3xapdVkAkFdYopNU6dS/7pXGEDGcFA3kCxm9U5ZB7Hf2MhkJgWOuzU9oDluhEX5sCG6mE982L8gsIrMPblQ4BvicEjnAxE+9J6EP5bQzTpWDhaf/MfR3MVTC2s/OlO7WLK4scpWHZHuq+EZUR53WrX8in1rVPfnudrd6Ie5cqnGRdH3qvOq60bBVhtPJ75MfJoDMjDfBiYN8iFG3Th+1B SkCAegTH CEQXHIVSKGqD+jommSxrz1190Dt2slSg/TxqA9DE24h5nnkOD0WgabqxbzPfGeV9PRsfqCcD8c3MF40hKpXiqKSbPLqLgTEZRko6cERpaCvovQ72khlSjKpxye2X5tzUI6ORfCDOzGoymWv0aUAWYc0qN4ByyGCnRzWgXkzCGy/uL+xz7INlj2Wi6+NGdUetpEJAQljwgxrVd8qePbW//jUGoeH8bmWg0JKJjpjQvy7ug4koBYtdQB6otCmNbCxK0CfhNEOnpIuFSRh84yd10fgRLnPRcDLOWKcPmsCBqBVQIsOk6ArLmiFF3jpQFedTnad9jRy9gmAHCjCVrNRYRCOLbDIpk/HaoHpflzGrevmDNO0E9ZbBAYjgSrs4SFes/hFtfyfv6CZiDS91v1WkZKMwR12aT1haI+XcHR9kDESxw9h6C32UDc/61pbRER7MJl5Dfs/2AUd54W8oneqirFx2IXQaSCQt+GQaZlL1g7TW8MWDnU6m9mBWZiGbNsvFObZKpLGuN+dtkKcR18D0RnSU06A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, capturing only works on matching orders and matching migratetypes. However, if capturing is initially skipped on the migratetype, it's possible that merging continues up to a full pageblock, in which case the migratetype is up for grabs again. Allow capturing to grab smaller chunks from claimed pageblocks, and expand the remainder of the block back onto the freelists. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cd86f80d7bbe..5ebfcf18537b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1067,10 +1067,10 @@ static inline struct capture_control *task_capc(struct zone *zone) } static inline bool -compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) +compaction_capture(struct zone *zone, struct page *page, int order, + int migratetype, struct capture_control *capc) { - if (!capc || order != capc->cc->order) + if (!capc || order < capc->cc->order) return false; /* Do not accidentally pollute CMA or isolated regions*/ @@ -1092,6 +1092,9 @@ compaction_capture(struct capture_control *capc, struct page *page, return false; } + if (order > capc->cc->order) + expand(zone, page, capc->cc->order, order, migratetype); + capc->page = page; return true; } @@ -1103,8 +1106,8 @@ static inline struct capture_control *task_capc(struct zone *zone) } static inline bool -compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) +compaction_capture(struct zone *zone, struct page *page, int order, + int migratetype, struct capture_control *capc) { return false; } @@ -1180,7 +1183,7 @@ static inline void __free_one_page(struct page *page, while (order < MAX_ORDER - 1) { int buddy_mt; - if (compaction_capture(capc, page, order, migratetype)) + if (compaction_capture(zone, page, order, migratetype, capc)) return; buddy = find_buddy_page_pfn(page, pfn, order, &buddy_pfn); From patchwork Tue Apr 18 19:12:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1359DC77B75 for ; Tue, 18 Apr 2023 19:13:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16D86280002; Tue, 18 Apr 2023 15:13:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F9F4280001; Tue, 18 Apr 2023 15:13:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8C63280002; Tue, 18 Apr 2023 15:13:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D3184280001 for ; Tue, 18 Apr 2023 15:13:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9E13112041A for ; Tue, 18 Apr 2023 19:13:34 +0000 (UTC) X-FDA: 80695460748.14.4235FC6 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf12.hostedemail.com (Postfix) with ESMTP id D225C4000C for ; Tue, 18 Apr 2023 19:13:32 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=CMPWhKkM; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5cDejj0JwuwHnUJP/oEtDQ+Dx04Y5ly69JDrx/A+L3g=; b=wsbWwJmuDyWGetiERl3SvSpuATsUIm75RgT5bUJKCTVQIlJhZohstmJaujCIJZcJ/QkddP HepqDzeu9ysF4LC0SGFvrZgImi59rgBfIaQyqXTOVenQE7csbBy44XFIBbS3aaol5H3QOU /KIIJ8LHzpquKyYCxTqM/NITrLXPO+c= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=CMPWhKkM; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845212; a=rsa-sha256; cv=none; b=hORaiXvzOCgZVyNBG4Y5nKVcLRsYm6CMyHFy6AGaYx4FR5ZOyjbcQgzMERnvDLZlZD3tnq +n8h9pH2mHh1T3sAOQr5UF5fIvqEvRQAHj0y9E9BC8nrds4W63FaQE2+xWRpyzUQ/zAoH/ uchysgqgDQYQRxKW+RG2/ERkyniJu+8= Received: by mail-qv1-f48.google.com with SMTP id oj8so1343151qvb.11 for ; Tue, 18 Apr 2023 12:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845212; x=1684437212; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5cDejj0JwuwHnUJP/oEtDQ+Dx04Y5ly69JDrx/A+L3g=; b=CMPWhKkMWeLbM1Rq+FTNlaR0dSWXr97zWbAFsGnhIYGtP+CzQS0HVHu9IoQvtoACE8 mhYIBqbKGjtBs+LtkNOHMbMHPLnpfnolTnLtMYSphbh3QM/YHVg9xGMX5JifyMhJRDZN RvXl9lXXUKC1e1qgby2qHSWtJc2hSDPS51QzKdNqBwfuWtGPigSB5OT3E52N3lqpNtBI r4M586z/x6k8gKidXuSXIy+fp2eMvjQvShdWfKbhEZC/kyJxnvrBCpaNZNxoThlYVR5D o6a1X3kolTsoZyqQip7R4cFUcH4uGzvAAHQyusAJPnGmI3LEgBvst3Dp6eXqmeXOlajY VOcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845212; x=1684437212; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5cDejj0JwuwHnUJP/oEtDQ+Dx04Y5ly69JDrx/A+L3g=; b=I+XHzKoNta6WeLcktFfCLzE2x/OuCDCAmzeTVrtvhmuUQ+mJqAshH0Kel2D8deE2BW nfh3gvwSLx+nIFfkYakRHkmaRitgQxBI5oJgTWhhul6q079pl69PWh3hg3+bGNk2T5Qz bPtti18u2gqbQTLuyms1naiv/nhDHKt4gSlLG6HTL0gwpk9MT/jzcRNpsiBcaSbdUjg3 waNCeQcLaw6H/Q3r9c/dW93oo12WMj+4qQAEQ/Alqk9A2IQKz83744OHug7QvhzN0tRE qFU+ouVAw36cs0o8BkfwN2Im8YG7/xD2KoEbdaQObFYBe/Zkm5ltk25SgSU8Nak7H9qe beIA== X-Gm-Message-State: AAQBX9fKbYg2R2GFKDFha56iQJ0itITKnu6UXL6xEM2U2UfjIfihOt2l Rgjk2NReHLHGdzzIrAtmSDMhG4VZTPnQGp3wCpc= X-Google-Smtp-Source: AKy350aMgOd3m7oKqMvaJHVobJC6wjWKHn3Vg6kPYlU6Q91voRGhTKGXzatfaK4cnRY78f9Dp31ahQ== X-Received: by 2002:a05:6214:e81:b0:5ee:b788:2f48 with SMTP id hf1-20020a0562140e8100b005eeb7882f48mr26312276qvb.31.1681845211935; Tue, 18 Apr 2023 12:13:31 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id i2-20020a0cf942000000b005ef529dc39esm3619309qvo.108.2023.04.18.12.13.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:31 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 11/26] mm: page_alloc: introduce MIGRATE_FREE Date: Tue, 18 Apr 2023 15:12:58 -0400 Message-Id: <20230418191313.268131-12-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D225C4000C X-Rspam-User: X-Stat-Signature: ak9ao4iaq3iucgp8t85eyjmhr4nhhw54 X-HE-Tag: 1681845212-441986 X-HE-Meta: U2FsdGVkX1+sYmaY3aelloku+ViW7kkm05ylcYorFRNwO+nuPvGOIF1C78PpsK/QvkKZrfAlqW9MTW+juiKZY8crWGW+AZH9bR4YbwzN22tPHSEDHoG2wMIN/uul0e8x5BqgM63rMYzjJlz1h7jV6h7VdVBEUABeSPGYdSqToRwsf3SeyKGGvJNXwjs47vDUPNgfL/wppWB7QxbfsfLIF1fOwQd/ctJ/MxNBWAJGjnEh9aSgVQ1Pccpp3JLAiXX4vT6VSGcedDwETUnzSpkVnLl7p5E87qDMsOznQb5GHgqhoZUq4mHyOy4byB3iohPqxAOUKiNvW1lYGbiPiUu3gLo+1QkqD6LRit58cEVp0Do5pi4hVc2dGNAQT89SUoz9XoL53d3abO5EgGFJMN/rZEQ9c9ChSaSMWhdV2T4v7fSn+5U3ktVBQneKact/EBO3zKu1y9SUJcyUI9exOy2nNlPfpE/PEQZKRGK7ZNpBAO+zisDuf50LKh6l2zmLV1yQ9KJHPEdpssSXnbOa6lpK39QMxuNySxlmDSy5FU4IDqY+6YGOnn8sQYa1QnIHBmIx/jSSFH1xIlni1THGIIzKw4eUxZ86ctl9IPr78uJiI1t72oB2xbRKgDrhmRCdRTNrAnjOrhhwx3+7gJmkGzg/e2Ovroa8+3sVKDXgmaG13M/GSRt3itR1MRmY14Nn/HvHTY39DQbjEaCo/nah8MSZthbSWeOV1DTTIm644Ci6gNd6f4jTWeXtkTm+6dNjpB1lCeR5s6FqcEctbueXvHniwEuYxJZ97UG/kgXWZ8fJI2gqIjO6KOXsmsdniNmFjBqSsFF35QkQhclkqP5O0UhPGdyolYJEBgO8ztvmLqP3mBTxnPIzD+ZinMn7fA+5X5KI89fo303l80ZRHzE1KhvZdkZsgPmry5wkAuDySCQ+9k6x6jVWiykKFfVlXbF+QIQuFFMa88h951tyXTs+eSk uQ08LS9n MO8q0hI3qq7f+uNRh5OjRKs2wgJlG0S+o4PbSxT9WR6VCHZ8o/74wgEmX/03oJ/DlbXcbFsO6yj3CnnH+v4Mdg7wHp5WrWBdhmAPiJKTbpx0hgm4gVUJZAoXEEVvEheBgnr/aRhao+IHaJ89Vi4PYZtX6GYB9E8+gUMnM3NDEkbtqadzqohsjWDTsKCKsc/iA5Zk2VPsfRcbomigPiYvUZZFipgh7rVap3zcSQmnkdG50oGBHVNfUwiX5LxsRymVgLQPOQmCHoSy5k2saA5bbYbj/Gcs1NjlJfT7YsVpX+2yTyTqXHO+2NgbAuXG2I5bYXbu6gw09b+FHeJ7a03NMkLoglGXKYG/Y7H5ALICBonw38WiNLW/te1qv/KwUGBX8OKoAGACsuWYZnZAPu742JnU5OH8VhRv++Hyq2D1hmvU/8MRwPYQuOzJay1K8giWcj+r3E2PXTBI3Exx73G835auU1JyJ+nzeD3PwxBd51ckLhUDXyqjLYws+EbDfKPSyHvzw286zqZ42VzrFBCvIzV+2Vgz+BE2JzOkk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To cut down on type mixing, put empty pageblocks on separate freelists and make them the first fallback preference before stealing space from incompatible blocks. The neutral block designation will also be handy in subsequent patches that: simplify compaction; add per-mt freelist counts and make compaction_suitable() more precise; and ultimately make pageblocks the basis of free memory management. Signed-off-by: Johannes Weiner --- include/linux/mmzone.h | 3 +- mm/memory_hotplug.c | 4 +-- mm/page_alloc.c | 63 +++++++++++++++++++++++++++++++----------- 3 files changed, 51 insertions(+), 19 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 53e55882a4e7..20542e5a0a43 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -45,6 +45,7 @@ enum migratetype { MIGRATE_RECLAIMABLE, MIGRATE_PCPTYPES, /* the number of types on the pcp lists */ MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES, + MIGRATE_FREE, #ifdef CONFIG_CMA /* * MIGRATE_CMA migration type is designed to mimic the way @@ -88,7 +89,7 @@ static inline bool is_migrate_movable(int mt) */ static inline bool migratetype_is_mergeable(int mt) { - return mt < MIGRATE_PCPTYPES; + return mt < MIGRATE_PCPTYPES || mt == MIGRATE_FREE; } #define for_each_migratetype_order(order, type) \ diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index fd40f7e9f176..d7b9f0e70b58 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1129,7 +1129,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, build_all_zonelists(NULL); /* Basic onlining is complete, allow allocation of onlined pages. */ - undo_isolate_page_range(pfn, pfn + nr_pages, MIGRATE_MOVABLE); + undo_isolate_page_range(pfn, pfn + nr_pages, MIGRATE_FREE); /* * Freshly onlined pages aren't shuffled (e.g., all pages are placed to @@ -1951,7 +1951,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, failed_removal_isolated: /* pushback to free area */ - undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE); + undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_FREE); memory_notify(MEM_CANCEL_OFFLINE, &arg); failed_removal_pcplists_disabled: lru_cache_enable(); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5ebfcf18537b..44da23625f51 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -380,6 +380,7 @@ const char * const migratetype_names[MIGRATE_TYPES] = { "Movable", "Reclaimable", "HighAtomic", + "Free", #ifdef CONFIG_CMA "CMA", #endif @@ -1222,6 +1223,13 @@ static inline void __free_one_page(struct page *page, done_merging: set_buddy_order(page, order); + /* If we freed one or normal page blocks, mark them free. */ + if (unlikely(order >= pageblock_order && + migratetype_is_mergeable(migratetype))) { + change_pageblock_range(page, order, MIGRATE_FREE); + migratetype = MIGRATE_FREE; + } + if (fpi_flags & FPI_TO_TAIL) to_tail = true; else if (is_shuffle_order(order)) @@ -1961,14 +1969,14 @@ static void __init deferred_free_range(unsigned long pfn, /* Free a large naturally-aligned chunk if possible */ if (nr_pages == pageblock_nr_pages && pageblock_aligned(pfn)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); + set_pageblock_migratetype(page, MIGRATE_FREE); __free_pages_core(page, pageblock_order); return; } for (i = 0; i < nr_pages; i++, page++, pfn++) { if (pageblock_aligned(pfn)) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); + set_pageblock_migratetype(page, MIGRATE_FREE); __free_pages_core(page, 0); } } @@ -2612,10 +2620,10 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * * The other migratetypes do not have fallbacks. */ -static int fallbacks[MIGRATE_TYPES][3] = { - [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, - [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, - [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, +static int fallbacks[MIGRATE_TYPES][4] = { + [MIGRATE_UNMOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, + [MIGRATE_MOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, + [MIGRATE_RECLAIMABLE] = { MIGRATE_FREE, MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, }; #ifdef CONFIG_CMA @@ -2705,8 +2713,13 @@ int move_freepages_block(struct zone *zone, struct page *page, * is worse than movable allocations stealing from unmovable and reclaimable * pageblocks. */ -static bool can_steal_fallback(unsigned int order, int start_mt) +static bool can_steal_fallback(unsigned int order, int start_mt, + int fallback_mt) { + /* The first allocation in a free block *must* claim it. */ + if (fallback_mt == MIGRATE_FREE) + return true; + /* * Leaving this order check is intended, although there is * relaxed order check in next check. The reason is that @@ -2808,6 +2821,21 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, free_pages = move_freepages_block(zone, page, old_block_type, start_type, &movable_pages); + /* + * If we fell back into a free block, claim the whole thing + */ + if (old_block_type == MIGRATE_FREE) { + set_pageblock_migratetype(page, start_type); + if (!free_pages) { + /* + * This can leave some non-FREE pages on the + * FREE list. Future fallbacks will get them. + */ + goto single_page; + } + return; + } + /* * Determine how many pages are compatible with our allocation. * For movable allocation, it's the number of movable pages which @@ -2873,7 +2901,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order, if (free_area_empty(area, fallback_mt)) continue; - if (can_steal_fallback(order, migratetype)) + if (can_steal_fallback(order, migratetype, fallback_mt)) *can_steal = true; if (!only_stealable) @@ -3485,7 +3513,7 @@ void free_unref_page(struct page *page, unsigned int order) */ migratetype = get_pcppage_migratetype(page); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { - if (unlikely(is_migrate_isolate(migratetype))) { + if (unlikely(is_migrate_isolate(migratetype) || migratetype == MIGRATE_FREE)) { free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); return; } @@ -3529,7 +3557,7 @@ void free_unref_page_list(struct list_head *list) * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (unlikely(is_migrate_isolate(migratetype) || migratetype == MIGRATE_FREE)) { list_del(&page->lru); free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); continue; @@ -3632,10 +3660,10 @@ int __isolate_free_page(struct page *page, unsigned int order) del_page_from_free_list(page, zone, order, mt); /* - * Set the pageblock if the isolated page is at least half of a - * pageblock + * Set the pageblock if the isolated page is from a free block + * or at least half of a pageblock */ - if (order >= pageblock_order - 1) { + if (mt == MIGRATE_FREE || order >= pageblock_order - 1) { struct page *endpage = page + (1 << order) - 1; for (; page < endpage; page += pageblock_nr_pages) { int mt = get_pageblock_migratetype(page); @@ -4020,6 +4048,9 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, if (!area->nr_free) continue; + if (!free_area_empty(area, MIGRATE_FREE)) + return true; + for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) { if (!free_area_empty(area, mt)) return true; @@ -6081,6 +6112,7 @@ static void show_migration_types(unsigned char type) [MIGRATE_MOVABLE] = 'M', [MIGRATE_RECLAIMABLE] = 'E', [MIGRATE_HIGHATOMIC] = 'H', + [MIGRATE_FREE] = 'F', #ifdef CONFIG_CMA [MIGRATE_CMA] = 'C', #endif @@ -7025,7 +7057,7 @@ static void __init memmap_init_zone_range(struct zone *zone, return; memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, - zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE); + zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_FREE); if (*hole_pfn < start_pfn) init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); @@ -9422,8 +9454,7 @@ static int __alloc_contig_pages(unsigned long start_pfn, { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range(start_pfn, end_pfn, MIGRATE_FREE, gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, From patchwork Tue Apr 18 19:12:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F4DC6FD18 for ; Tue, 18 Apr 2023 19:13:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B734280003; Tue, 18 Apr 2023 15:13:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3677B280001; Tue, 18 Apr 2023 15:13:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22F59280003; Tue, 18 Apr 2023 15:13:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0EE0A280001 for ; Tue, 18 Apr 2023 15:13:36 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DED46160435 for ; Tue, 18 Apr 2023 19:13:35 +0000 (UTC) X-FDA: 80695460790.01.4010E82 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf20.hostedemail.com (Postfix) with ESMTP id 20D0E1C0012 for ; Tue, 18 Apr 2023 19:13:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=fnKaIHJG; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845214; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VKzxWkw9Wp7aX3AQonPLuV4v1F6kfO1MbtWYkZecR34=; b=K4RFTaX5dDkLWJoPV+xs2Q7HJJFM41dHausOGfSqYlP/GyY87jotVscr2hQ5vYDvJWkPfG wiFF9DMVFE+IhadFHPy/4L9h4QA7JfVoaY0VGgAgCWcZDQcKw9vU8JVP8ix1TPxSADN5PA MW82qj++QB7ZNGo7AzZdLe6c1217uyA= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=fnKaIHJG; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845214; a=rsa-sha256; cv=none; b=Epcw1GwaWOIKrrPVycJHNK973TQlKePOOsepUANrU0eaiiV5B8Pqo287fHGjZayWyDxXi0 S4pWvi3qHLeSPEN8mPl51baT9cTAohn7zYLj730dHvqyz9Zx/isqAL/S1pFIt55yel9EmF Wd8GdzBlDF3jvR5F4vy7Eiv06MPuEI0= Received: by mail-qt1-f181.google.com with SMTP id u37so5543791qtc.10 for ; Tue, 18 Apr 2023 12:13:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845213; x=1684437213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VKzxWkw9Wp7aX3AQonPLuV4v1F6kfO1MbtWYkZecR34=; b=fnKaIHJG7NChil0OqR30RHO2yD9ilNfbfzz+l5UrDoOhQMcIjn9Z7u0gwD85iPVAp0 tffEBcyMY2FouSlHJNy6srGCvt5E9IDj4dJBVKRE6XA/ar/BC0EWMcwavJ9D8HNRuvlG Qdrqep2d7P+SpC95Rfj9aYd53ubAphhsP8E2gw2I9iDczVYbpjshQr8hBgBoVIb0a4N3 k6mQGeB1gv6RgDAMcI2tUgO6k3f+/MPwwLRAHp5lTK9eH/oa31Gg7tq6UC02MZCEJxJ0 CZVghku4DrrkzUssr6FthK28bwb1ynE8SrIKpU9sP12KWUg87rBmxhFx+GjHDJFIysNI WwlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845213; x=1684437213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VKzxWkw9Wp7aX3AQonPLuV4v1F6kfO1MbtWYkZecR34=; b=XnLU/hz4iVqbAKqEOFG9lQm9B3GXl49DF37Mv0zuI0TWVXdh+9DQlMOs9WSYz74Myg 8wyGsfYqGQGZNpIxc6+siz7aX//pfsHiBUGndQz9uCWmNZXY/Op3R9rf0g4LHYz/SxoU d6RLk2rThqwekhBqGYosrUQdRvKKHVfmjk+pt3E3DFbw2oTaeNPSTjltqX1YNkhTsusg CpKtPNUtr7/FwedJnMaBzojEXSkdzCfywcmmdPj1SSVJg4JeQ1fX5rSWfjyjAYhm6XCR l01cajRG2OQxgufR5mBiQALTdIW/PpMM+7FYntuTOaB2LSNjeCNRGT8bsG0bj8kGBbLK WtHA== X-Gm-Message-State: AAQBX9dtrZfXbD91wZ2JGRmzZmaAxF2hOQUUmsI3A0LuuXleBnRa8YQG GAxLLJhPELhcvR0sg6XFHpIfGTE84e7MCG948rA= X-Google-Smtp-Source: AKy350YRTIp6emLy7bR9ZUZa/RhO/Obyaw6AxSNUEm8mzOLj17NAjGHuS3AbcjQstxdD3Htg+F7UQg== X-Received: by 2002:a05:622a:4d1:b0:3ef:3d3f:17a2 with SMTP id q17-20020a05622a04d100b003ef3d3f17a2mr1327574qtx.68.1681845213072; Tue, 18 Apr 2023 12:13:33 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id y30-20020a05620a09de00b0074683c45f6csm4141283qky.1.2023.04.18.12.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:32 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 12/26] mm: page_alloc: per-migratetype free counts Date: Tue, 18 Apr 2023 15:12:59 -0400 Message-Id: <20230418191313.268131-13-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 20D0E1C0012 X-Stat-Signature: swe4336seo8hsdxo8db85373ssemrgjb X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681845213-885557 X-HE-Meta: U2FsdGVkX184fgyErEjat8AW1sacYxj+ZYFQpkliRuhz6YfxFjxXg0znsGAN8hqDxnEm9ZF1RdeYUSoRPbdPpfWQ0FV4SlDW/U/Kaexa+l0SgESMeufEQCE3oWOpmik42TyWlqXmeXfN2ALX1ffv0XW0WD2DCAcFpENMb8YSo9m/KW+VHViIIgwI9g9y7DQgkKF7RE/Ig8cRthZHK2C69Tm99Qax4CoTdifJw2DMoJ3pDH2Uum6kVgjP8MDrnla8w4Hq6g/Jw+00dlsUL9iCzxmxrhaU3Tg/uZF2DGCHplQRMx+FziWbZ6JECvTjqTxUx0dekichroo8iK+3UmALwUWXf13qWDj7vM/8VktCDLfbUoSr1jkcFcG58Dqt3hBb121jBv888dgfAqdTM/B12eVxbaDrf69z1mUw6Z1AM9wUwqRAW6O59Qnlc734f0aFTwlewsLzF2+/iFqlXHi/d3FhSgHvmnWa7ULe3LAgwR5jHV1qHbk9hhCxacqCUsyCbNBThaHVYnucdQ6IOyMKSX7XBemFVKaG5O5dkcqzCNC8GHGgS5oVymFYP/k0r/jdDuvVV5ROjrYLFAzfBGEZQ7zjYgtu6mRsofvOp8tnQBLChNA94iBgpu60m9+VujG48Kp7uFe/H/R6RjaCrOumqttZxGfeSq22WDYY2ZFdlzZUCEV31TvOpCsmGLOxBvmk4Q8zLP7XVESOCHDPJ6vuXZlbRtK6FhivjUULaloNxIdnB2UA0GH8L+s2Qjzw+YJltijY+g6jynz/4bmnLqj/02r8G/igruE9dM1nYInrMgrGpct+WryL1+YV6UPao1qazjrgovCj8lHNSinBBM8EuqmadVm/bKmATPVMxDKzYMPg+cgKJg+YIuaidshSWInhEMhSYLBCQafRSNcV7k2FUIPCADOci2E5pcYLrZVKT/ARzyHIg7b6cNOCh1jJYCdvnE/vGCUHNdSLPlR8d8Q hMAYegsP pYtzg2I49s+mhvik5FBZq0wjt9csXnH64XYCP4ASDyOoV00bHOeEHUq58fWIWyVmzTq0fKybVtqJXKbjL5vazzqZw05QVdb3qywr1H8LjhH9TkDvCdoM4BG5+mqSJFQlB1HNQDJ9XcgvqjjV3C6YOSWojwEPg43HKmdF9FpAsRLqwWZcho1S9m/59KB6xJGNsgadjScckV7z8U41muAwgg9563xLIMSx+5RVa5qoguVV1RKGng5L9JKZ8vwgixckN4MFf0CqA1DBUKpa+SU7dgd8gFl3udATUIITsrBVswGBTm6C5bV137iO3qONeB7qspeWENdj0FbgymkvEyzUVhTWi0Dy1KQvxzXkKdZ6z8y9ShTjM6bFvKrZdIHOLUKbZA0QUFOSX/FXPjOr6htVuMh/R0TkWzmhnjt2o7x8cc0C/T7lR0ACI1ctQWRVavBvQVEPt6xUHt2UeeWdJV3aV9kU2U/EGfxn7rfDBw+ijvJ/DovO8mq3xiQ/wtBlIrP8FESE3g7tZEWdhJekcvHYXGdgtug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Increase visibility into the defragmentation behavior by tracking and reporting per-migratetype free counters. Subsequent patches will also use those counters to make more targeted reclaim/compaction decisions. Signed-off-by: Johannes Weiner --- include/linux/mmzone.h | 5 +++++ mm/page_alloc.c | 29 +++++++++++++++++++++++++---- mm/vmstat.c | 5 +++++ 3 files changed, 35 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 20542e5a0a43..d1083ab81998 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -139,6 +139,11 @@ enum numa_stat_item { enum zone_stat_item { /* First 128 byte cacheline (assuming 64 bit words) */ NR_FREE_PAGES, + NR_FREE_UNMOVABLE, + NR_FREE_MOVABLE, + NR_FREE_RECLAIMABLE, + NR_FREE_HIGHATOMIC, + NR_FREE_FREE, NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */ NR_ZONE_INACTIVE_ANON = NR_ZONE_LRU_BASE, NR_ZONE_ACTIVE_ANON, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 44da23625f51..5f2a0037bed1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -959,8 +959,12 @@ static inline void account_freepages(struct page *page, struct zone *zone, __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); - if (is_migrate_cma(migratetype)) + if (migratetype <= MIGRATE_FREE) + __mod_zone_page_state(zone, NR_FREE_UNMOVABLE + migratetype, nr_pages); + else if (is_migrate_cma(migratetype)) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); + else + VM_WARN_ONCE(1, "unexpected migratetype %d\n", migratetype); } /* Used for pages not on another list */ @@ -6175,7 +6179,9 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i " mapped:%lu shmem:%lu pagetables:%lu\n" " sec_pagetables:%lu bounce:%lu\n" " kernel_misc_reclaimable:%lu\n" - " free:%lu free_pcp:%lu free_cma:%lu\n", + " free:%lu free_unmovable:%lu free_movable:%lu\n" + " free_reclaimable:%lu free_highatomic:%lu free_free:%lu\n" + " free_cma:%lu free_pcp:%lu\n", global_node_page_state(NR_ACTIVE_ANON), global_node_page_state(NR_INACTIVE_ANON), global_node_page_state(NR_ISOLATED_ANON), @@ -6194,8 +6200,13 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i global_zone_page_state(NR_BOUNCE), global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE), global_zone_page_state(NR_FREE_PAGES), - free_pcp, - global_zone_page_state(NR_FREE_CMA_PAGES)); + global_zone_page_state(NR_FREE_UNMOVABLE), + global_zone_page_state(NR_FREE_MOVABLE), + global_zone_page_state(NR_FREE_RECLAIMABLE), + global_zone_page_state(NR_FREE_HIGHATOMIC), + global_zone_page_state(NR_FREE_FREE), + global_zone_page_state(NR_FREE_CMA_PAGES), + free_pcp); for_each_online_pgdat(pgdat) { if (show_mem_node_skip(filter, pgdat->node_id, nodemask)) @@ -6273,6 +6284,11 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i printk(KERN_CONT "%s" " free:%lukB" + " free_unmovable:%lukB" + " free_movable:%lukB" + " free_reclaimable:%lukB" + " free_highatomic:%lukB" + " free_free:%lukB" " boost:%lukB" " min:%lukB" " low:%lukB" @@ -6294,6 +6310,11 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i "\n", zone->name, K(zone_page_state(zone, NR_FREE_PAGES)), + K(zone_page_state(zone, NR_FREE_UNMOVABLE)), + K(zone_page_state(zone, NR_FREE_MOVABLE)), + K(zone_page_state(zone, NR_FREE_RECLAIMABLE)), + K(zone_page_state(zone, NR_FREE_HIGHATOMIC)), + K(zone_page_state(zone, NR_FREE_FREE)), K(zone->watermark_boost), K(min_wmark_pages(zone)), K(low_wmark_pages(zone)), diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..c8b8e6e259da 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1168,6 +1168,11 @@ int fragmentation_index(struct zone *zone, unsigned int order) const char * const vmstat_text[] = { /* enum zone_stat_item counters */ "nr_free_pages", + "nr_free_unmovable", + "nr_free_movable", + "nr_free_reclaimable", + "nr_free_highatomic", + "nr_free_free", "nr_zone_inactive_anon", "nr_zone_active_anon", "nr_zone_inactive_file", From patchwork Tue Apr 18 19:13:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA21C77B78 for ; Tue, 18 Apr 2023 19:13:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C127280004; Tue, 18 Apr 2023 15:13:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FBF1280001; Tue, 18 Apr 2023 15:13:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64D3E280004; Tue, 18 Apr 2023 15:13:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 52E07280001 for ; Tue, 18 Apr 2023 15:13:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 362438042A for ; Tue, 18 Apr 2023 19:13:37 +0000 (UTC) X-FDA: 80695460874.20.3FFF794 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf28.hostedemail.com (Postfix) with ESMTP id 5DF18C0015 for ; Tue, 18 Apr 2023 19:13:35 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=YCxaymFj; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fvrO/1+4yglcd2Jvs04tq1hDUyEyh2m1Sh3LN7l5OG8=; b=O+DITbFtruUSwcUQXDjrC5+pOPTyS7Rx8QEtfjQKuDJw0fg4A/JKPr0uZx0Bt7CRHK6Vqj Tg5XXdJmslFdFgG9z5nL9vUndHpBaEfD0Lt54SbW4ZJ21p3qHbE3+btmrt1AlCRDRtZXDU gyKO1BGr+nRROqZSryOvSJAPyEGZzXU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=YCxaymFj; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845215; a=rsa-sha256; cv=none; b=t+0gOIAOI+q/Fy78g0c5awd4m9inquWmsDIu+TbEh0BP2n0bAXkF19DI8a1NTkPzwv44ID ZVENugsP6Z9WOVUru+5Afe8utqKSyA8GgVwUQFvLg0zAFhtE4wAapTaJTBsdLO8P9nT3e0 rGCjeGsENV2pDnYt9yo0/OZSRTew36c= Received: by mail-qt1-f176.google.com with SMTP id l11so30662106qtj.4 for ; Tue, 18 Apr 2023 12:13:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845214; x=1684437214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fvrO/1+4yglcd2Jvs04tq1hDUyEyh2m1Sh3LN7l5OG8=; b=YCxaymFjTOB/xEVxOKn/iRYET7rdfjWPu3AZDYJ5KtaGsEjeOO2z8QOjdpBzKiI+fh zq/KjeYnjCaxkKVGYNcD9mghxgZwPN5syoeQUj2HsktOmS7F69bhZq+UdMOlukwdy6mr zhoMS/Tx7lhyB4yQTexZo6hq18nMIH/j4LR8HxWWTAFstqWcBJC/+aaBnM/4R2GA77/e eaZxJoD9eYoQfHGUVsharCxWu8P9awFx8F/6/oFYKGnqiIiwRsnchL6XPeWp9ufx15vN it9IVwB25jyUUFrrcgBYhumIpZbTgM/FEDmcZyOcdKC+L06MuZYxfBEyLL+xGpVRPQMZ aekw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845214; x=1684437214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fvrO/1+4yglcd2Jvs04tq1hDUyEyh2m1Sh3LN7l5OG8=; b=VVYij0k3XvPpTa+XsWzzlJkXJ8WN+3NqxT2RK191I2ZbjgkfhJKooQSfCmzQdC4mw1 05C2wverSQl5G+Bx51v8GVd75rdxx3tinGsqXu0Wm/ckKP/zbuIJclP5o5xA0srp8kq2 13mAIJFu3+S2OlFYFjMpAHVMaoLYmMPY54JJrLkD41mJYQLc6Ru1IvEPKQrayZYJ027H OpQALpyUt66Uf0Oa5VNW8efABZK/VUjI9UHi6voowO3eDGXN2DJSGKLPqar2LZLm0u2T QNX5TQBLR4jxfhQVxhqEVC3IvKjnSZk8HKRZgrpOHfpqF1HeOZOFvB0d+XdxnTfd7TSe 4QZg== X-Gm-Message-State: AAQBX9eKGMn8SlVJd9/rvHnZPXIXSpwzEoL/MjSK2s6Xkwlea2keE5ca +qPdHQTvvEVEVa1h3vJ3iHc8XPDHLc/rloZiJo0= X-Google-Smtp-Source: AKy350b8I9apZCXgd3Fc463SUtxO97BghAjCQl+N88Qns+QuGdCqUDlXOuuktIWM1UP9lMqTVC8ihw== X-Received: by 2002:ac8:4e93:0:b0:3ef:3880:9db6 with SMTP id 19-20020ac84e93000000b003ef38809db6mr1674175qtp.6.1681845214206; Tue, 18 Apr 2023 12:13:34 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id f17-20020a05622a1a1100b003ef415f0184sm39541qtb.69.2023.04.18.12.13.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:33 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 13/26] mm: compaction: remove compaction result helpers Date: Tue, 18 Apr 2023 15:13:00 -0400 Message-Id: <20230418191313.268131-14-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5DF18C0015 X-Stat-Signature: 5fjiqh4adrmwaypyayhfc3tfpzowqzn7 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681845215-866691 X-HE-Meta: U2FsdGVkX1+WDvtveXlruWPSHE/0XLcHvxCT4uil0qKc/bMsOfx5OVIR1sQp/b/puMZN2l1kCYXabdec5p2BPS0hPhm2aEU5sGLNsG9GjsUDyKFse2nKqbyKBeIFD/u7GUbQlD/ncViOkElgA9O19xjE65A8Ncmxx2JCxtcYbbrwWqUarbPuP68x6kJBLfsrNeyjRMrnIcd+Lt/2g2dJhLsfCElBFnyx8sXUY96OJOHEdzfbx+ZikzJ/Vc9Z8aommGuCmXtrsZCC6O3pYFE7p6KMq0lbD1si/qeJlkQQkSaIDvraUcJCASzSmdEyZAXDTE//ZFgIYLNA90ZYyKKrqKwHLC96eDHDfqCo1UmnlzGYpIqsLrXok18TyFa3INo9wmbkEIXpSVMXtGmxlGeCD7Qst2UynOtaU9OUeTkwL4aJyhMojuUO1uKdMbHMUAC1Ahw8fMmMhol443dg17ZkTNdLf+4zfWOvtagxca7tSz3hSSADubIJZLiGaeVh4cHSdWVn7yWCVeLX2bo7A+o6QRBU/N5rk5R/M3jv6Nin83pA0Af95ePNXxEGWL/YBDR30aQGLDKKzvtQkBwIRx2Wcal3ktQ/OXhRFT3poQeQTx+FIpcN8iVEQZByPVpnN4aqTfPYVKWzEOX7iFqgSivZaq6p3n8S7r1teGMzMhvjFCOlW4g31vWdim4n+igqyIwsFgdOEoQtSd4JZv4EpgHy11KeGZEhSsrK8r076spqmEZQ704LeTvGoS+S/XvA+DPZeRdLzA6z4315d+r8IYoP9XojXr19BYaFn7XKpzosoKcZbo2/p2U5sodOGUuIV9bp0MzJibANO1ItK3WANOzobFPNgjpl4x/z6r2Sx4dXaNsBY02lrPjxvtx9vMNvhQZYH8WaDBNhLvwLKPgfH7JhxmWBv0PsB2OmGVsrWzZi6+g8L5DWpKjaQ1w+IhB1IuuQNJtjYS1xzXOSU0Uc0Us A3GSN7ob iVTaieZCCkPY6Ho3CGCE48sWOwUT6S4JHSkok7jmJ9JCzLAPKEXDFObGNVL3sK3Oif61t5oNDMzIFfucTZ0bOk8Zs8viB8pD+GMqXC/xdUAQ6ZCKHsy8SdgVendTjMKPglMOUmk8W4Ce/Mk6h0NK5EqCCg3kB8sjEOhSwkq/p0o1VROMjmcQNZhA8XwfmNG0vltp/OoFWxH20OSWHmLaiIcgCfOCw3flcbkeOvE5hb/Kq5N0oDN/4J8enQcBj4vaS7fST7qgULAUUPazqzDb+rhVZDXeX48rYG+JEJhEMUD1t3zt5V3+OtjAD471LFlnwqdNZWcIM6u44vniXgsta9oZWN3lo/KmCDJBTUiNewfa5coXRCGaP3UC6UsaGcY8XX2OgR/x5/cAcgOOgv6eOEPIzjpGNo6uWHvu08yuUDwHLm7ca44asv4YAHV/yf4FY9uN+fG0H6ftB7oHWSxg/mYUgSYA1RMZIT8/VlLiTaR49Mqc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I found myself repeatedly looking up the implementation of these helpers while working on the code, which suggests they are not a helpful abstraction. Inline them. Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 92 ---------------------------------- include/trace/events/mmflags.h | 4 +- mm/page_alloc.c | 30 ++++++----- 3 files changed, 19 insertions(+), 107 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 06eeb2e25833..7635e220215a 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -103,78 +103,6 @@ extern enum compact_result compaction_suitable(struct zone *zone, int order, extern void compaction_defer_reset(struct zone *zone, int order, bool alloc_success); -/* Compaction has made some progress and retrying makes sense */ -static inline bool compaction_made_progress(enum compact_result result) -{ - /* - * Even though this might sound confusing this in fact tells us - * that the compaction successfully isolated and migrated some - * pageblocks. - */ - if (result == COMPACT_SUCCESS) - return true; - - return false; -} - -/* Compaction has failed and it doesn't make much sense to keep retrying. */ -static inline bool compaction_failed(enum compact_result result) -{ - /* All zones were scanned completely and still not result. */ - if (result == COMPACT_COMPLETE) - return true; - - return false; -} - -/* Compaction needs reclaim to be performed first, so it can continue. */ -static inline bool compaction_needs_reclaim(enum compact_result result) -{ - /* - * Compaction backed off due to watermark checks for order-0 - * so the regular reclaim has to try harder and reclaim something. - */ - if (result == COMPACT_SKIPPED) - return true; - - return false; -} - -/* - * Compaction has backed off for some reason after doing some work or none - * at all. It might be throttling or lock contention. Retrying might be still - * worthwhile, but with a higher priority if allowed. - */ -static inline bool compaction_withdrawn(enum compact_result result) -{ - /* - * If compaction is deferred for high-order allocations, it is - * because sync compaction recently failed. If this is the case - * and the caller requested a THP allocation, we do not want - * to heavily disrupt the system, so we fail the allocation - * instead of entering direct reclaim. - */ - if (result == COMPACT_DEFERRED) - return true; - - /* - * If compaction in async mode encounters contention or blocks higher - * priority task we back off early rather than cause stalls. - */ - if (result == COMPACT_CONTENDED) - return true; - - /* - * Page scanners have met but we haven't scanned full zones so this - * is a back off in fact. - */ - if (result == COMPACT_PARTIAL_SKIPPED) - return true; - - return false; -} - - bool compaction_zonelist_suitable(struct alloc_context *ac, int order, int alloc_flags); @@ -193,26 +121,6 @@ static inline enum compact_result compaction_suitable(struct zone *zone, int ord return COMPACT_SKIPPED; } -static inline bool compaction_made_progress(enum compact_result result) -{ - return false; -} - -static inline bool compaction_failed(enum compact_result result) -{ - return false; -} - -static inline bool compaction_needs_reclaim(enum compact_result result) -{ - return false; -} - -static inline bool compaction_withdrawn(enum compact_result result) -{ - return true; -} - static inline void kcompactd_run(int nid) { } diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 412b5a46374c..47bfeca4cf02 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -222,8 +222,8 @@ IF_HAVE_VM_SOFTDIRTY(VM_SOFTDIRTY, "softdirty" ) \ #define compact_result_to_feedback(result) \ ({ \ enum compact_result __result = result; \ - (compaction_failed(__result)) ? COMPACTION_FAILED : \ - (compaction_withdrawn(__result)) ? COMPACTION_WITHDRAWN : COMPACTION_PROGRESS; \ + (__result == COMPACT_COMPLETE) ? COMPACTION_FAILED : \ + (__result == COMPACT_SUCCESS) ? COMPACTION_PROGRESS : COMPACTION_WITHDRAWN; \ }) #define COMPACTION_FEEDBACK \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5f2a0037bed1..c3b7dc479936 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4620,35 +4620,39 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, if (fatal_signal_pending(current)) return false; - if (compaction_made_progress(compact_result)) + /* + * Compaction managed to coalesce some page blocks, but the + * allocation failed presumably due to a race. Retry some. + */ + if (compact_result == COMPACT_SUCCESS) (*compaction_retries)++; /* - * compaction considers all the zone as desperately out of memory - * so it doesn't really make much sense to retry except when the + * All zones were scanned completely and still no result. It + * doesn't really make much sense to retry except when the * failure could be caused by insufficient priority */ - if (compaction_failed(compact_result)) + if (compact_result == COMPACT_COMPLETE) goto check_priority; /* - * compaction was skipped because there are not enough order-0 pages - * to work with, so we retry only if it looks like reclaim can help. + * Compaction was skipped due to a lack of free order-0 + * migration targets. Continue if reclaim can help. */ - if (compaction_needs_reclaim(compact_result)) { + if (compact_result == COMPACT_SKIPPED) { ret = compaction_zonelist_suitable(ac, order, alloc_flags); goto out; } /* - * make sure the compaction wasn't deferred or didn't bail out early - * due to locks contention before we declare that we should give up. - * But the next retry should use a higher priority if allowed, so - * we don't just keep bailing out endlessly. + * If compaction backed due to being deferred, due to + * contended locks in async mode, or due to scanners meeting + * after a partial scan, retry with increased priority. */ - if (compaction_withdrawn(compact_result)) { + if (compact_result == COMPACT_DEFERRED || + compact_result == COMPACT_CONTENDED || + compact_result == COMPACT_PARTIAL_SKIPPED) goto check_priority; - } /* * !costly requests are much more important than __GFP_RETRY_MAYFAIL From patchwork Tue Apr 18 19:13:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82659C77B7D for ; Tue, 18 Apr 2023 19:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2917280005; Tue, 18 Apr 2023 15:13:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB1BF280001; Tue, 18 Apr 2023 15:13:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE11F280005; Tue, 18 Apr 2023 15:13:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9C31F280001 for ; Tue, 18 Apr 2023 15:13:38 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6B0BC1202A3 for ; Tue, 18 Apr 2023 19:13:38 +0000 (UTC) X-FDA: 80695460916.12.7098199 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf09.hostedemail.com (Postfix) with ESMTP id A1A8B140017 for ; Tue, 18 Apr 2023 19:13:36 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=fHLEDZEs; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845216; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l59ZyyVgyCUBMvUoPArfOg/Tu2nHrtxfgxoKhkDEsRQ=; b=zsCK8+tJTG7LExbEqIShJuy6/jN227W5z4+wDVR5kfcYkMqjNWIn2ir8UgQ0Qo/Bo+mkCX m9bRSYT08NRTbIcfGHTcEc6n5waZXLK4hY0muTT7USZnuCyCQyQsjDxf1t09gFIdqSh5qo 5Nvnqevw+LvekEHuE3NJuqfZVaSRyJY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=fHLEDZEs; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845216; a=rsa-sha256; cv=none; b=hIuU7Qvd+HUPeLXRypZG19OQbP4uU7jKPxCb6zD31a8j4AKqG1H8hs+x+l5VJJkTITKnCP 5Y7ZOoAcJiYmq+3f1bn3OelmaFjm9TGZxMtR7jvjQS+mDoLG7B0c/iwRPWSB5DRxRFn/pm sZPYWu0XG/Y3PteIlUwan0Uafj9kFdA= Received: by mail-qv1-f53.google.com with SMTP id a15so7395531qvn.2 for ; Tue, 18 Apr 2023 12:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845215; x=1684437215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l59ZyyVgyCUBMvUoPArfOg/Tu2nHrtxfgxoKhkDEsRQ=; b=fHLEDZEsV9N0D0X0rv9luI4kf2Ny4hu81LBmLjiQQISEpJJ+RqngAVpAPnFoR0YJg5 +rnu1yHL/o1ivzsyaceccwRhzNhhqiyoYCKe93qsF3ayw2qwII6swNfekI6XGKLg0WKN ufiP8FtZHi0gw8HoJMh10Ahc4nfREGT88vUp2AeAYI8oivqhu8YfXmH0mXPBZwkUumtM cdyBUXt7wmT0HtYc+n8SW7cn8uD60GO8c+eVN2+jqkKgf8Z6KnRU8bZkIL2jEokwqAx5 sPCj0+lVYfaoZN3UpGcQ5ZNRelLySP7T0KS9IIKSrs2ehaDuMkRxPwMV16n9lIi+UQCl Tevw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845215; x=1684437215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l59ZyyVgyCUBMvUoPArfOg/Tu2nHrtxfgxoKhkDEsRQ=; b=CJTBJmp2yaGG6gy+kGLjdGGEjASoZ9GL3uXMqEKopsc1mIJ1acxVob+r5+mESe/5XS Fk6Qx7E6T6/tXik6mLUzOuSUlk2vFLNKkBPu+uqUSKIm7oT7iF1t2O1Net7DZDKXGjD9 MrbXaAhUG55YJjVAGhGj+808BgGHq+3B/Yu5ObE2zS3qihcxoBwdyAKF9Ssd/518Fbhd dljTYPu+AnFbwHHh7G1dh2UjQU+vl0W4WgohDfG/x6CZai/vMYMgIwcs5rJQiHQ1ZWkH 9GA7eTnBFb6JQ7im4CE8dLjZvWx24xXwea7nt0YRvG0EDxGQ3LhR3Z31AsMxB+hhuZnx uiWw== X-Gm-Message-State: AAQBX9fvIPD1A9XLYQIszxeeLdDJs8tXeQ82mM9BkMpZ8yd/L2GjkWld tFXKb+5+dvnNbnmSjt9+CL4NFPI01DoY35pnR+U= X-Google-Smtp-Source: AKy350Y+h1X4MZEW+S29kRDU7ycQ7OqCtKgHBzDa7byusJ+iYdu4NQRhrwkjbrpSqc3/yAnVuA2tCQ== X-Received: by 2002:a05:6214:1942:b0:5f1:6904:a2d6 with SMTP id q2-20020a056214194200b005f16904a2d6mr1035437qvk.51.1681845215347; Tue, 18 Apr 2023 12:13:35 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id k15-20020a05620a414f00b007463509f94asm4089576qko.55.2023.04.18.12.13.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:35 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 14/26] mm: compaction: simplify should_compact_retry() Date: Tue, 18 Apr 2023 15:13:01 -0400 Message-Id: <20230418191313.268131-15-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A1A8B140017 X-Rspam-User: X-Stat-Signature: jqnjkmb96yweeg7yka4tnofehgyk436s X-HE-Tag: 1681845216-796075 X-HE-Meta: U2FsdGVkX1+ZiyqQZchjglJhlw1xn1dM2tDZvh1ZNPzTrkr5+mBZ5/ulTlfmy9WF9KpD0D9sci+rHz+MEp0kCzra6IM8LeExFjX7yTrq8mVPVnSLy+B2G5fNOlihoE66HYni/Xgh7LOSe/4Dk23fZJ6B0tZf6D/d/MVCSnNRw2rmrUi/IPlKmn4YgpZHhn+DYTKG42wykAJ5qFgCqNHIlI/W9zkmcj3A/0qDBwAAG7BCa5SKW5rPdb35qtrc945kxolI/s/oJvc/IUc48g/RwIQj9R2jSsggjxqNfw2wuWKet5YZOdVFoWxCaRVn2WyhiPZjZpLF43jSZrCOSiHQmZUHv8YlHUlg85AhmLyImhhOffEV93/Ojp76yvKP5PIQQmnWgiyO+nHU/JbT6fdcbds5ckQw+x1kw12czjvEHWr5FmD7FBDNDkr7yOVUC5ddMHWfXWvm63TipKKootQ81pn4RYdUMSY0YVMZmmsobsxh43R+GtvHzFnr2/qxJMaHhmNLRjFfApga47t0rJg9vQD52PhNUW2SYdMm6Shhj4asEXCSmmjdN31JX+ug/41PtLpZE8ZYpsy6CGhUssYvAx/3V6HS7Lo1GU4kdmjwfEyG4+359mYSimJ5GWqiQy7mXvnRxIw6Vtm0UEhudSDJj4hea8yi9HWzpEV86XXDG1ePJASgpwA8+onrLKdrtijQ4p4nevT0X7lo7GjIuTmjK2XZLXVGyUz3OblxTFVS/E3+yjy1prFgVwUVFZav4rxpavSCgO7h67OqRUDiut1e53SSKfTMeaq0Ey83aHoUXy8uUz//12ntmgzjn11wY4su109HrtagX1m0zsuuFF3wdzgrToCqK7Ruux2jFwtOhgH7ZCc3J140xdQA6YJDZ0Zka49anRzjzGuwe+b14kT8QlkgP1vcyx4m5wtpPQuDKUfHLyD12ql02yRK2NSLTq+x99K8UPvMIKJbFMkvWum T46InlXL +IcWGFlqKRqTy/ceFEDOh8QyByufD2Sy8GoPPVQmPZKirsoqCbnKLccLV57vF1gTF9akKH1BCNFSHvlSYDzohWQj9IgRQQ0mlzQ/Xv8QpN7J8FT1H56rvsNFFVjZgPBJZ5txAfrEMi/XAGbYGK1XqkDRYyTzdUSj6XOz4FCOlmEHZ8VvAbqwDYOFUMFQ1JyFj18WMAX0eC2e/wznwTQHqW0ZWgeU6SinfeD2v4Zuz5SyWuJVPpQ772BZxYvsKfJXnKRfPpoDiR40jpCMpXGWEwxrR3r5LCgnR3Vp7Ra+4F3LlbFVLub9Bx7AQ9ODt4fsDzi5O619DVVSBr1Ltw891A5vgRLQCcrAOGm4Ft7qBEdjlD836brQV8OinA1iwshK4gxjUKz5rqU71xXHcMbrPFNK/PWeUvGFKYZ25NGEspTOEbdd42/Wsq5h5fQM34biPZpPLU1aERXzASOILThpEb15UgfugXlOXsho+B2gUfp0kIWmVQueyEpEx4mfbnPhYHp/ILfKeLn+/p09pO+zUaTCMEg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The different branches for retry are unnecessarily complicated. There is really only three outcomes: progress, skipped, failed. Also, the retry counter only applies to loops that made progress, move it there. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 60 +++++++++++++++++-------------------------------- 1 file changed, 20 insertions(+), 40 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c3b7dc479936..18fa2bbba44b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4608,7 +4608,6 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, enum compact_priority *compact_priority, int *compaction_retries) { - int max_retries = MAX_COMPACT_RETRIES; int min_priority; bool ret = false; int retries = *compaction_retries; @@ -4621,19 +4620,27 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, return false; /* - * Compaction managed to coalesce some page blocks, but the - * allocation failed presumably due to a race. Retry some. + * Compaction coalesced some page blocks, but the allocation + * failed, presumably due to a race. Retry a few times. */ - if (compact_result == COMPACT_SUCCESS) - (*compaction_retries)++; + if (compact_result == COMPACT_SUCCESS) { + int max_retries = MAX_COMPACT_RETRIES; - /* - * All zones were scanned completely and still no result. It - * doesn't really make much sense to retry except when the - * failure could be caused by insufficient priority - */ - if (compact_result == COMPACT_COMPLETE) - goto check_priority; + /* + * !costly requests are much more important than + * __GFP_RETRY_MAYFAIL costly ones because they are de + * facto nofail and invoke OOM killer to move on while + * costly can fail and users are ready to cope with + * that. 1/4 retries is rather arbitrary but we would + * need much more detailed feedback from compaction to + * make a better decision. + */ + if (order > PAGE_ALLOC_COSTLY_ORDER) + max_retries /= 4; + + ret = ++(*compaction_retries) <= MAX_COMPACT_RETRIES; + goto out; + } /* * Compaction was skipped due to a lack of free order-0 @@ -4645,35 +4652,8 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, } /* - * If compaction backed due to being deferred, due to - * contended locks in async mode, or due to scanners meeting - * after a partial scan, retry with increased priority. - */ - if (compact_result == COMPACT_DEFERRED || - compact_result == COMPACT_CONTENDED || - compact_result == COMPACT_PARTIAL_SKIPPED) - goto check_priority; - - /* - * !costly requests are much more important than __GFP_RETRY_MAYFAIL - * costly ones because they are de facto nofail and invoke OOM - * killer to move on while costly can fail and users are ready - * to cope with that. 1/4 retries is rather arbitrary but we - * would need much more detailed feedback from compaction to - * make a better decision. - */ - if (order > PAGE_ALLOC_COSTLY_ORDER) - max_retries /= 4; - if (*compaction_retries <= max_retries) { - ret = true; - goto out; - } - - /* - * Make sure there are attempts at the highest priority if we exhausted - * all retries or failed at the lower priorities. + * Compaction failed. Retry with increasing priority. */ -check_priority: min_priority = (order > PAGE_ALLOC_COSTLY_ORDER) ? MIN_COMPACT_COSTLY_PRIORITY : MIN_COMPACT_PRIORITY; From patchwork Tue Apr 18 19:13:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66BEAC77B75 for ; Tue, 18 Apr 2023 19:13:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE3D1280006; Tue, 18 Apr 2023 15:13:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A92C1280001; Tue, 18 Apr 2023 15:13:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 933E4280006; Tue, 18 Apr 2023 15:13:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7F673280001 for ; Tue, 18 Apr 2023 15:13:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 597D616045A for ; Tue, 18 Apr 2023 19:13:39 +0000 (UTC) X-FDA: 80695460958.30.6CDEEEC Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) by imf26.hostedemail.com (Postfix) with ESMTP id 9FD39140013 for ; Tue, 18 Apr 2023 19:13:37 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=aeYONsgP; spf=pass (imf26.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.41 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845217; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oyoKJQkIW9YTYt09kObRpzKT5MnW6BvwQNuIxs7T4Tc=; b=CgJpFYCjz8q5Hb3iuzli7W2LZDuJZpkW6L4Y87mxl58X/y57+JpwdutDK6Klg9svX7d1Tb zAdZIcFCXg7MG2+OPk0z0n0Vl1LtdLcjSH0NV+T1tJgo8eJyWnHqtJGxDXUB4cC2Wt1pqD eYe53NuR4zCZqMiyYeNvkYQatuWg2NY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=aeYONsgP; spf=pass (imf26.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.41 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845217; a=rsa-sha256; cv=none; b=hIVrxDWu4riDlIO0DvmxwXBYrGdks3tVC/KUoaqcvXiZ6N8m6oui6ujpB+C7G+WX9n6xaC m/rbxxEBQrN5zJQWaBsKMo/nfOUvSuM2/Li7Bv8rm/JMsOMPOEHUbQ2vJ/JcVAsE85Sc4N /aVYTX20zCuSANGcgw7NUY6lS9aO0do= Received: by mail-qv1-f41.google.com with SMTP id op30so18259250qvb.3 for ; Tue, 18 Apr 2023 12:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845216; x=1684437216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oyoKJQkIW9YTYt09kObRpzKT5MnW6BvwQNuIxs7T4Tc=; b=aeYONsgPSzQQgWyb5FHu2pLc4enHXJiXdaQ/URnHTI4N+Nq62uPdJYSr8YJxsZo7DW gB6lifiemln0yXBoUpetZdAa+m3U7Murk1q9y6eEqAxekRgWZCiGw2kWjYInRjtT3eFT DkRD7r1ECPLst/vkbKvyzSRNiHJuj9OxC47RNcVmUl1WWTCQ+sn8Jx5NKLfVkdObtBW0 S8SXnhzF5QsXyMf5O/UpkE9MPOzstc3gDLc3FhYxPAPxbgDfog2ZjfNGJa31r+oOo2GA REeMAyuiQFEJZ21mYaqXnYQUFwEa8R+Jwrb338SotrhcoLFx+tzMMjRnzYViLbYZJo1o t/mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845216; x=1684437216; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oyoKJQkIW9YTYt09kObRpzKT5MnW6BvwQNuIxs7T4Tc=; b=Xlc08MX0KZyXeTKoXsJBXk3TLvUB8TE5VS/UCMgavrEF5SXPNCBkpHuhfyeo0h222S IKDmbqJirEp8nhAFCopTeEaIQqdxsyv6LXAqhS/tVibtPj8KZkP83tlrcArMVGBvE2Lq wxkwucwXEJ/Os7/+LrQnnOrJHZfZs9thEu0hHSN2P0TTK1NWxdSA9Sz2I/FejawptlSx uuLkqq3LmxRE/mynjcfiV2OG9u+OKzEci+53wPOqYDIPrDBadUXP39B9Tuo41DdfHs73 KY/HuMOEFE/hRI9ltSBarKTqV4ZcgU/ojN2RJBjl2puW4nWSu8YeLzcUNbFluDBy1ZO/ FAhA== X-Gm-Message-State: AAQBX9da6cmR9thv+p7h90NINRqSXjEY75s7/RowHNLWRDAGtblQgIOH V978wzP6nfRgmzZH6pLMvxR3ZWS67nwhRtZwRP0= X-Google-Smtp-Source: AKy350awQ/PRfC/BhFkVKh85YOny0ld5DKS6bQHcNwzqu22Ols+1ZFQkDT9bbDkAlKwae5T6XGpjKg== X-Received: by 2002:ad4:5762:0:b0:5ef:739a:1c46 with SMTP id r2-20020ad45762000000b005ef739a1c46mr16977002qvx.1.1681845216529; Tue, 18 Apr 2023 12:13:36 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id kr24-20020a0562142b9800b005eee320b5d7sm3844283qvb.63.2023.04.18.12.13.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:36 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 15/26] mm: compaction: simplify free block check in suitable_migration_target() Date: Tue, 18 Apr 2023 15:13:02 -0400 Message-Id: <20230418191313.268131-16-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sjgg4rme1ge9n5bj7bohysx1ki4wjipp X-Rspamd-Queue-Id: 9FD39140013 X-HE-Tag: 1681845217-452554 X-HE-Meta: U2FsdGVkX1+jQMLy/E8ZPLVi2dbx0jG95EtVCk4qNMv6F1CXDn2EDwtTC5JhxZg4hRAf7O4EhMg01oX4pgskfITr5DM4/riGoLYniboLvosRn7uuFsJ6Ug2bH1N1YRE0SSrOrlO0eWYUfKyjBA/g8SWNq3D9E9s6maUI6DixxGt/SPnDb4/p+fgN4degWoqG9oICDkJtAM94LXn03rLTN6jBtciqkSLDpl73E0nYYo6pXEBEkeuLM+uyRCJ9CKFcL6XPFULVf5fzIefDDROXRJdpyye1noSWgg5kUjNbCU9s4Lw1I21ho46FrMex02lOVFO5sUERNy1e/UEy3hUd8XHUMutYmQANkILUzmaPuFzZ/Z4kNOVsFpSt6kI23yxoek033oz8QxG4sx5U92sa68ZjoQbRpKqSWnl85GGupxlj8Wo7cnRm1z3qs3XZOTFvOC7kA5B8hcD/vSuLWVKadBkgeBCnaCODyTYClN9OU3Q4c+J+dLRDsime2CQQdwVrD9rlx42DoxEriukt1xmlDnECDy82txiKqjON2cD2UqEc8l4xPcmi7JG52V49r9693IXbWH090h0LQZw2X1Bh+Qh8Dd5z61o1+qYITWmlQfiIl5UAllNxs5CZGE0ZHDLl5suUih33Uw5ssDnE1cNwygEJPCUx6a8T7uNXunXWlgC3HYkRfroBMNg2K/VoPbnjMNM8RxWiwOADDkp+s35Ri31H+sDq3L6h1EVMbplOvnCXl5nVMnHlkAT4YG6xIW6/CdFf+Povgd3Q5YawMMZ9CU9kbRt3UkLQRUjr9zqaY9Vq6ceWCY624kO6APjKLacf3G5X7n2hr6bIBpGytlcamkIuI5e4zX0uW/j5c1J6b0z5bYglhgM5nXOYl4hCVUfGTUHmUobIXGpSJdIS2yQaPRJOEwVv9StqNZslw12PDCaXzEaiwO7tEJWuAtPOHV+fDFGpBT9TO/Lbrvyz2pt rem8UkfV P7HQK3f62eGsMkcI0SaYS/ZkOBQ2+IpPkjTk6BvPwqZ5+yobYfsp/Jaes43sKXS5BlmHWJT1t7m+mBFrlVxHSHDHwJ9UtGo3vYfuseg2WCZ4TVudJZ5ZMQxqTfNXb36szToQsM7QDL0tb71V/psFF13sLSszov2ALDP+iPVsqFS5GxEXTDL/ydh9ensEkS705R03P0ECqt6AjCHvlVIKe8owrL5SB9IupuejVkoZZsytNXHd5/pZcAIahvfmu2iUG1GcrJ6eu2jIf4eWc2PdOs5YRbfOjBl3FS0wyxJLIe42UkCq08xn0OjtXcaq5Nm/clxhsFP95ER8I/cAAourO96ZRBI+lOM+RjG14Kvuhcp/S64MAF86OPzq6rVYZXVme6HhZSbdEQ20scloIDZbTCoTBNRquAeCf6BJeV4il7dxqiE2bz3xWkURowZxBJDXe+R0Vkg98m7A27ba+UY4lRg/eClQxAgcpuVMHZ6yAmjT9cYoG9syn3C3vypGvf2dS7HHsJL8680KB/gdo0G66P8S+KQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Free page blocks are now marked MIGRATE_FREE. Consult that directly. Signed-off-by: Johannes Weiner --- mm/compaction.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index a2280001eea3..b9eed0d43403 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1281,22 +1281,17 @@ static bool suitable_migration_source(struct compact_control *cc, static bool suitable_migration_target(struct compact_control *cc, struct page *page) { + int mt = get_pageblock_migratetype(page); + /* If the page is a large free page, then disallow migration */ - if (PageBuddy(page)) { - /* - * We are checking page_order without zone->lock taken. But - * the only small danger is that we skip a potentially suitable - * pageblock, so it's not worth to check order for valid range. - */ - if (buddy_order_unsafe(page) >= pageblock_order) - return false; - } + if (mt == MIGRATE_FREE) + return false; if (cc->ignore_block_suitable) return true; /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */ - if (is_migrate_movable(get_pageblock_migratetype(page))) + if (is_migrate_movable(mt)) return true; /* Otherwise skip the block */ From patchwork Tue Apr 18 19:13:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52F4CC77B78 for ; Tue, 18 Apr 2023 19:13:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDC2D8E0001; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8BA1900009; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDDBE8E0003; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B87AC8E0001 for ; Tue, 18 Apr 2023 15:13:40 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8DC7A40450 for ; Tue, 18 Apr 2023 19:13:40 +0000 (UTC) X-FDA: 80695461000.20.F52B6BB Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf06.hostedemail.com (Postfix) with ESMTP id B74DC180010 for ; Tue, 18 Apr 2023 19:13:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ROm0ArlP; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=N6bi+bnm3ooprNj/Bp6FmNZbaBjgsN47cWMV7RvFp5C63Mfj1LA00S/tbCJEIXJ1dhG5RX a8502Qqej5QwXjCibiW3OOtgrjAV0RSCEgq4b/CtMJ1kLLpQYHqNMoUTkaLvknf7feHZku sv0wueYpWcK6uELI9iLTJ1gegdKvhIM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ROm0ArlP; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.182 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845218; a=rsa-sha256; cv=none; b=BDMTrWWz/GexrKA95ge1TgfE1ctdRlcRQ34/DC4GFyFrkpGbDy/JmqB/eGY25zwh/ommnt LHMEHr35S8a1jq/phz5L1dQwNPVVCs1yxFul/VxSdCOO3+XYU5nHD4E/KJQJm8M+hXSXVZ GUYimjBPYTUQma3AVMiorWuSph02ixE= Received: by mail-qt1-f182.google.com with SMTP id br19so1860695qtb.7 for ; Tue, 18 Apr 2023 12:13:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845217; x=1684437217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=ROm0ArlPuXBY1EzWc/8PRgqaTVBI+lb7icLJ0Q57BP9ugdMTZmrUmryBS96SSVH9Nh eeLrq3q4kINpOOeYwSiqOxAJrdmOgMg80zVTThvjx9VkzE5k0WCPil9tESj3Hku5cLuy SPJk1gyPDWl9cU4d7hzMlhCvpJn6xdGkNXplbkrtzq0OiV6LSdAYuNrP+KJa/f3MP5dx 4wKlDRUsNPqCre2MCxjIRsO9xKT8/coTjpyJXWIkKBFyHmv7zdnJQKHUNs+zIEe6gm+0 cTwYRc8KLsJOFaKN9aFOQI6B99BvcPA5pw+NCRvbGraHEF3iyw5uVssKuGhFRoPeJkSZ mtEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845217; x=1684437217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G1QAyddYnWJUKuKAIak/itfaS7Ph3sBa4ofaAGmayWk=; b=Nqfond4DSoj/7OE9RXcS7HkPaimoYf+3u9bEBV7M6ngwob3ZpVlhOZtcpoTgZPw2jI O/pBaOYMRqWNf9COHHwhaJpuSmyYiYsx9vITRViFse+XWl63HB4qy54Se3mU/08C2VEQ DlgWdwAzRXrnL5H/ko3nG47tiADS7Mvc6AHu6/FJY76GBocNmUWgkhS3T8u/nxuMLBT8 WLKVYM9a1mBF7NM8UCvMyy2r23Fw9d4gTS+2mcjdizHHtyZJrCEi+pNHUvKtqnXvTXjT p41MM6HdaII4szatCazFsnSn9zls4E7MZbB9JfxIwUFE/tgfvEQhoCIhpDnB2wjoiO0S E/Ww== X-Gm-Message-State: AAQBX9cni7NtchRd56BT2lH8eMioE+AnBLbiQitJtCxg3zmyXmiD8lNP NFZzDYIg8ev1BxxBwtjRe0vructnZUorXwShuJQ= X-Google-Smtp-Source: AKy350ZUW/3fdavMXIT1x5ul2i8OAycLqVKqq8/q+G7rUbGTG7TjkJBhgSj/nJexX1DU7fRe/5Ryig== X-Received: by 2002:a05:622a:1703:b0:3e4:deff:4634 with SMTP id h3-20020a05622a170300b003e4deff4634mr1499645qtk.24.1681845217590; Tue, 18 Apr 2023 12:13:37 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id w4-20020ac86b04000000b003e64d076256sm4281417qts.51.2023.04.18.12.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:37 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 16/26] mm: compaction: improve compaction_suitable() accuracy Date: Tue, 18 Apr 2023 15:13:03 -0400 Message-Id: <20230418191313.268131-17-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sugw8bc6eghqiqhkxsxemwwufswujdbc X-Rspamd-Queue-Id: B74DC180010 X-HE-Tag: 1681845218-273923 X-HE-Meta: U2FsdGVkX19tlf2gx/mVZRd1IcWYGO0m4Vv732YV7OL32ezsj/KAWymrWSlRKlz+yVbBvCeV93zQbM0b+O1brEFzjeuhmmlNkwZONsYLkQ7kDugf+pYFhT1RYQEFr13ooDIbRfJ8uxWrCgVQB5tbUDvWqXfEKk5zNWaaMlkkwPZgQwx3jHhPLoTHb7DYvawU/E+1LlKtIOeyVoaAYxNr/5DQj3Y6CFCBkniOHr/6OJ3vlrWsIcdlgttPMLUHnZSr4RK5i8Sm7r+t+9J1NHqXeNp27uvTfrTIEjUHsUkSY7396S7pjBpb/5ViN4NZ2OsOQ9px7aGlwePd20xYBqtCRifM2NUPAd27XNj67X+bMWhbfTjslYPs7N5aGkImMqz1rfYZw9KN+27YkyxHxZ4XuvwpC8ua5CNe/O4uxYogCYyBWRwJo8wOx3Kdl5k90X0MB3VvEgws66jW0R72N66fuPhWhNGukBOrvTvqQ054JIi8ui4xK6owPvL86vvp6PDFEqVO+cMT6zeqKWW3kI5RxL7Up57EbsnY2jHP2YKCKHT/qKgv3GVvevFE4Fk6axdmLIx7uSRYe3+eDVAqouO0pwTos4p/9zAdqhJrxlZlxF4E8bbmlzt8pxtqbWYDXyJH4a0slofm/YYdiMOviJ5uyHZ0OnrLuiCSaZzVnadtNWS1e7lSLeETbRVy+9weU75iJz6/hmFYQVuOkiBfdStxG7JOBsb2nivrWTtAyrS5AFfKLsuVb52Gkrwdkp5+7xnwI0fTeri6Km5EnZpQczpTU4pv3tGfKiXe2hmq36+ziVe4wulRI6JpqHjrKe6k7F+D83i3vhXEu7HgvZ80QLWtlBg7axahmLhXPusReTMb580HOOTHvZZEVXYAPImq9sdReJRYzawLoFcGywwzA2EpJoc7w1sOUxfZvrJoBBrFbRKG0+XR/oWg6hm09X98ef1SdMlD2LqfbisQhAxX+Ex WjFub14F IHLW+mHVXh5cozXKsz1VAnTib4A5pgG0mq53UpT+wLe7TO+5+vioPhc2VOBTxwDwJkJZflAwnTbKKCef5GHNFS5/qBO/3Gzmrs6gxJHxB5nLTb+CaNhkqa8+QWsyG8yIS2w5UJcHlyTCEYoCOlepo/BR5NOfHL+j8RTgsJhnTqOs/K5WHT80NWbAUusxVt4XDMo+ODMf1LIIeq+rwzec1Mm3WlgvIvpKo3WzEyRJDiDYNr8jBJ3ZfT+4w6LoPDfONbeb+XSF4l8qAYpwDShEPtkaxOg7q4Z4POeLuy5B4s++VLTmGpnlDsCStXlAgCiwfJd36UERVF3hpZ4v1P8aQAt+4uO6N2Ph6IwbyyTwhMiFfiJK8DXieEb/ncfNoeIMGpf70tBXvlFvOFKbJ1xmxqaI7VuEDAXG4/5V5TiRIj6SMfbtSuwjx6eztQlVdFT392DDCF3royzdNH1e9BK2NBn+Ql9zE1uni2J+WcJE+iiirLgfIb3NS2JeWINn3Z1Imx6lH0Nd3aXXuYoGamv96YyJsV6rRO/ZR5yeJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the new per-mt free counts, compaction can check the watermarks specifically against suitable migration targets. This ensures reclaim keeps going when the free pages are in blocks that aren't actually suitable migration targets: MIGRATE_FREE, UNMOVABLE, RECLAIMABLE. Signed-off-by: Johannes Weiner --- mm/compaction.c | 23 +++++++++++++++-------- mm/vmscan.c | 7 +++++-- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b9eed0d43403..f637b4ed7f3c 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2226,11 +2226,17 @@ enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int highest_zoneidx) { + unsigned long free_pages; enum compact_result ret; int fragindex; - ret = __compaction_suitable(zone, order, alloc_flags, highest_zoneidx, - zone_page_state(zone, NR_FREE_PAGES)); + /* Suitable migration targets */ + free_pages = zone_page_state(zone, NR_FREE_MOVABLE); + free_pages += zone_page_state(zone, NR_FREE_CMA_PAGES); + + ret = __compaction_suitable(zone, order, alloc_flags, + highest_zoneidx, free_pages); + /* * fragmentation index determines if allocation failures are due to * low memory or external fragmentation @@ -2273,19 +2279,20 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; - enum compact_result compact_result; + available = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); + available += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); /* * Do not consider all the reclaimable memory because we do not * want to trash just for a single high order allocation which * is even not guaranteed to appear even if __compaction_suitable * is happy about the watermark check. */ - available = zone_reclaimable_pages(zone) / order; - available += zone_page_state_snapshot(zone, NR_FREE_PAGES); - compact_result = __compaction_suitable(zone, order, alloc_flags, - ac->highest_zoneidx, available); - if (compact_result == COMPACT_CONTINUE) + available += zone_reclaimable_pages(zone) / order; + + if (__compaction_suitable(zone, order, alloc_flags, + ac->highest_zoneidx, + available) == COMPACT_CONTINUE) return true; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 5b7b8d4f5297..9ecf29f4dab8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6270,6 +6270,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) { unsigned long watermark; + unsigned long free_pages; enum compact_result suitable; suitable = compaction_suitable(zone, sc->order, 0, sc->reclaim_idx); @@ -6290,8 +6291,10 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) * we are already above the high+gap watermark, don't reclaim at all. */ watermark = high_wmark_pages(zone) + compact_gap(sc->order); - - return zone_watermark_ok_safe(zone, 0, watermark, sc->reclaim_idx); + free_pages = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); + free_pages += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); + return __zone_watermark_ok(zone, 0, watermark, sc->reclaim_idx, + ALLOC_CMA, free_pages); } static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc) From patchwork Tue Apr 18 19:13:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59513C6FD18 for ; Tue, 18 Apr 2023 19:13:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2C0190000A; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED961900009; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0F790000A; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C5DC1900009 for ; Tue, 18 Apr 2023 15:13:41 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8FB861C653C for ; Tue, 18 Apr 2023 19:13:41 +0000 (UTC) X-FDA: 80695461042.12.8917F48 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf10.hostedemail.com (Postfix) with ESMTP id ABF3DC0021 for ; Tue, 18 Apr 2023 19:13:39 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=PrIghyUC; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=YP/shsz0ozbiKVD0iH8zpNxUUANmDsMzMT7dJZK0oa6diQCEVmFKWE+sudBWj2Uz+rTu4o /H94kJ17TnjyFAHxTjYfJ9/lkaTKToOIElgb1Ub2JdVrZFju0LWVotMEauq7E9MqFyjPlg Ta7EOQwEg5ErxcW0VQcwpzp9DJ7C4Gw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=PrIghyUC; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845219; a=rsa-sha256; cv=none; b=i79zzm30A4FDZI4ccy6b60+BgkaoUv4mOneVTty7IVAdHzDfk3RMmBt7LCNkiYs4Zd7CW1 rTRNs9yvgmU1cPD+UPgbwGAKGB7GDlKNiNgW9nrfnAX95SBtcon9IvKGR3OKALcS+ZE2Wa tW7hl6ah1uYzZIJiyO53p4wfkQPcxu0= Received: by mail-qv1-f46.google.com with SMTP id h14so18464600qvr.7 for ; Tue, 18 Apr 2023 12:13:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845219; x=1684437219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=PrIghyUCPpn833RcxLlSVfU5Tk3SpnQwff6jbo/QPuj/XkyPt6wBzL/Dp9O4CsPEIx vlbtgOz19a8Bj5Q9Tvz3XxufWwoP1pyylG1i+Eubg/l0s5lNHABn8jx+RIo3qNp8yvJd xxmWan9BQx+LKGsE5QvB32mS3Fey4Ysn5JLrhFNxDsTGokonj/n7lqrpIjGCmJsUuM9+ Ab+1ebZi1/6VKaIDC4ffD88G61bRq54CAS9nhM+89W7ACAiv40xaJvzX03Xeni/fkhUD S0GccLFy6IRYr+8AC7mboIhQozDrEx6WmyjvaCp6CXY+ejn1Dw19uqM1MKn105sfCPkW reiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845219; x=1684437219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EFgZ3D8VKBLVvaJCzeMlC8+6IsY+n+UHK2kcdHMAd6Y=; b=UqlOyX7HCvDsRRSc6xoJcCijS5CAMuPo6TgqZZujDgGhZdOR8kfUS6PW/O2xSq8Bv9 3jE6VEqmMOt9gv4I7dbNMVRGNIMRB/bUuUolcfHpLli+ZXSpQvMiSPhEUOHLdBBz6BOP bQVYy7YXmWrLpYiqcRf0sbTFX9JGgb9YrM4djzaU+87uMTKfOF4mJCfGFpzSZozr0kne lsHKge6ugQH8R1qSoIM8AGfRIrH2WP6v/7e5MIg0iaYOP80Ng1fZ4JZ/ZlWFzztlyCRK l9RagVvWY3WmtlkhQN6YyVxeexgO6qI9MleZyD95nJR/IjlToZQPB5x0Ra5MKcm/hfEo Z4Yg== X-Gm-Message-State: AAQBX9fybOCKxS3N0OFz3bRBRhm/vrScik+m/3jLP2uNfmbnDX8+pSaz l1OyfgE6JLncQ7ONjTtkkjhee1LkMytiB3WmWF4= X-Google-Smtp-Source: AKy350YOTexSjADk98sBY4JcGk1SQDwC9c/+fGeG6k+kXfYRIwBupTZdOSOM2/pP7fHju0r9Wjpp2g== X-Received: by 2002:a05:6214:20e2:b0:56a:b623:9b09 with SMTP id 2-20020a05621420e200b0056ab6239b09mr22229605qvk.14.1681845218901; Tue, 18 Apr 2023 12:13:38 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id z9-20020a0ce609000000b005dd8b9345c7sm3914040qvm.95.2023.04.18.12.13.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:38 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 17/26] mm: compaction: refactor __compaction_suitable() Date: Tue, 18 Apr 2023 15:13:04 -0400 Message-Id: <20230418191313.268131-18-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ABF3DC0021 X-Stat-Signature: 5u3wcgupoe8fhbzmdpcf6f9cdshgyt7x X-Rspam-User: X-HE-Tag: 1681845219-18852 X-HE-Meta: U2FsdGVkX18o9d0RBi0prcn8IXk4VrHPU+imS1nxO664yvbGW52KWz9yXLtuOV7fVonDTD6/0IRzHqVnrD2VkF8VNCiuDf0l43gz7R3yJPeMSd2G678X61QcRxOE+cB1NiMHMLdYbZbI25TyY5qvmdoHO04kcWL/dGimg1cgFguSYIrkuqt2WZOR+Re5BR8mRylePWj9o3fvCurPlJ/T80GUcSRIwvHvlX4YR0Y6DMQ7xSHJZXrqiPNCst7WuEEEKvR35zTmK3UqI9DwhcNGN0x4Qanq6AU6ofDZYCgpJeNOiio6qZ+3Z+OqLg0xxG8hXNew18CIoH2AcM/Qwp/vEM1mcGJXpdm5mrytlqa5crjofRm5+Bk/99qbysg2jLMaKOConUUqP3/wMpJc4KHDfwmgOE+QpTHJ88yBeVE9qqMB5yvXPCTBMSMQbae7L76SvLA4VmceEZsMCIEhjlEkMr6Y5hUM1h3Gt/gDeXKhaXggJLAG9pgyvvjkHHVDUK+HlcYbPJcACV02kXTrfvH/lrlmcjSA0ugfwNi50CXcCqFtIMwIu7/Ybkd8LQw08HKTUtYLrmATyO0FFNHwAGl7MUobeb6TYRLLq+3IuZgEJ5IE5Cb+uhZ95MntEeFYDIe1k9Cs0H+MojzIf5IIwFv9gutuoNq9PQeeEC+buULXttgLHdmYqPhvKS4lD4GoKX4aBvl6kwgn4zZS2sOKEFsUL/O7V3dyAh6B9uOfVEu0S/52/6qPzzM8gGk5rC9fSsHBYoFUuR8VPRg2yqeeVbwdcyxt45YDnAc6fsxt+vwhCB9k7Uo9NhSrF9c0IALlxvjlXfMrBTfYp5Mo/rQSKyboE7cutBFhldwcVGXSeLWFIj5Cw9J/n6/oOmvqxlMXg977avOlF4xw/1LX7cnPsH/cTiguw7MSlr+urB4Bof02LtCcLfcFy9pzbXHKwYwjCkT5DKYqR56RK30cUlA6xaB z1ePmkm4 7Ve5nJslQqNGRY9vO+fnDvml+uDH4/qHGWj5y1ncfgj4p36UdnPxa1ZUgqUHsTBPHpuHcpc80uwnBOplam1Xx3KKT3RZmbq/eOMP5kUP1wvH4OC2jZbTUzLH4JQFIt6ot/O+TElFV235MGabtwkpzFu4jnKDL7ojnc80KyHXLtrWwnBneymJ20QhmmvEM8gHHmGlmzSaLxNJrKw2/36jKE/cY6pNrjo57G6JejZO2jKZmWp0Rlth6v0UrYduSP/66H1lHV63uFwjNWLqFkkLYGnFV/owKAlbJmu7pWIYTNrzrifDcmlwQAZSZ99mpxwCRJNTQKrMjLP/CxgleIdQHizp553tcf7zhTwkW6n793nmLJsJGX5NoU9JhBbDr+uK1jAcl/n1IEvjxJscjzb8CJH7TH9vibVtq63jEdebSszNQzaRkwgIr2M6vZnK+678KdEcv4b2YTHJP1rSVPbpj1KkDji7NVRt2arPVbAwijwsbRr1dN2Z57PqnoiAVMxd11uD+GRyZMj9E9U24naE+6Lb3qw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __compaction_suitable() is supposed to check for available migration targets. However, it also checks whether the operation was requested via /proc/sys/vm/compact_memory, and whether the original allocation request can already succeed. These don't apply to all callsites. Move the checks out to the callers, so that later patches can deal with them one by one. No functional change intended. Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 4 +- mm/compaction.c | 80 ++++++++++++++++++++++++-------------- mm/vmscan.c | 35 ++++++++++------- 3 files changed, 74 insertions(+), 45 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 7635e220215a..9e1b2c56df62 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -98,7 +98,7 @@ extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx); + int highest_zoneidx); extern void compaction_defer_reset(struct zone *zone, int order, bool alloc_success); @@ -116,7 +116,7 @@ static inline void reset_isolation_suitable(pg_data_t *pgdat) } static inline enum compact_result compaction_suitable(struct zone *zone, int order, - int alloc_flags, int highest_zoneidx) + int highest_zoneidx) { return COMPACT_SKIPPED; } diff --git a/mm/compaction.c b/mm/compaction.c index f637b4ed7f3c..d4b7d5b36600 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2173,24 +2173,10 @@ static enum compact_result compact_finished(struct compact_control *cc) } static enum compact_result __compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx, unsigned long wmark_target) { unsigned long watermark; - - if (is_via_compact_memory(order)) - return COMPACT_CONTINUE; - - watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); - /* - * If watermarks for high-order allocation are already met, there - * should be no need for compaction at all. - */ - if (zone_watermark_ok(zone, order, watermark, highest_zoneidx, - alloc_flags)) - return COMPACT_SUCCESS; - /* * Watermarks for order-0 must be met for compaction to be able to * isolate free pages for migration targets. This means that the @@ -2223,7 +2209,6 @@ static enum compact_result __compaction_suitable(struct zone *zone, int order, * COMPACT_CONTINUE - If compaction should run now */ enum compact_result compaction_suitable(struct zone *zone, int order, - unsigned int alloc_flags, int highest_zoneidx) { unsigned long free_pages; @@ -2234,8 +2219,7 @@ enum compact_result compaction_suitable(struct zone *zone, int order, free_pages = zone_page_state(zone, NR_FREE_MOVABLE); free_pages += zone_page_state(zone, NR_FREE_CMA_PAGES); - ret = __compaction_suitable(zone, order, alloc_flags, - highest_zoneidx, free_pages); + ret = __compaction_suitable(zone, order, highest_zoneidx, free_pages); /* * fragmentation index determines if allocation failures are due to @@ -2279,6 +2263,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; + unsigned long watermark; + + if (is_via_compact_memory(order)) + return true; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(zone, order, watermark, + ac->highest_zoneidx, alloc_flags)) + continue; available = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); available += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); @@ -2290,8 +2284,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, */ available += zone_reclaimable_pages(zone) / order; - if (__compaction_suitable(zone, order, alloc_flags, - ac->highest_zoneidx, + if (__compaction_suitable(zone, order, ac->highest_zoneidx, available) == COMPACT_CONTINUE) return true; } @@ -2322,14 +2315,26 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) INIT_LIST_HEAD(&cc->migratepages); cc->migratetype = gfp_migratetype(cc->gfp_mask); - ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, - cc->highest_zoneidx); - /* Compaction is likely to fail */ - if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) - return ret; - /* huh, compaction_suitable is returning something unexpected */ - VM_BUG_ON(ret != COMPACT_CONTINUE); + if (!is_via_compact_memory(cc->order)) { + unsigned long watermark; + + /* Allocation can already succeed, nothing to do */ + watermark = wmark_pages(cc->zone, + cc->alloc_flags & ALLOC_WMARK_MASK); + if (zone_watermark_ok(cc->zone, cc->order, watermark, + cc->highest_zoneidx, cc->alloc_flags)) + return COMPACT_SUCCESS; + + ret = compaction_suitable(cc->zone, cc->order, + cc->highest_zoneidx); + /* Compaction is likely to fail */ + if (ret == COMPACT_SKIPPED) + return ret; + + /* huh, compaction_suitable is returning something unexpected */ + VM_BUG_ON(ret != COMPACT_CONTINUE); + } /* * Clear pageblock skip if there were failures recently and compaction @@ -2803,7 +2808,16 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - if (compaction_suitable(zone, pgdat->kcompactd_max_order, 0, + if (is_via_compact_memory(pgdat->kcompactd_max_order)) + return true; + + /* Allocation can already succeed, check other zones */ + if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, + min_wmark_pages(zone), + highest_zoneidx, 0)) + continue; + + if (compaction_suitable(zone, pgdat->kcompactd_max_order, highest_zoneidx) == COMPACT_CONTINUE) return true; } @@ -2841,10 +2855,18 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (compaction_deferred(zone, cc.order)) continue; - if (compaction_suitable(zone, cc.order, 0, zoneid) != - COMPACT_CONTINUE) + if (is_via_compact_memory(cc.order)) + goto compact; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, cc.order, + min_wmark_pages(zone), zoneid, 0)) continue; + if (compaction_suitable(zone, cc.order, + zoneid) != COMPACT_CONTINUE) + continue; +compact: if (kthread_should_stop()) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index 9ecf29f4dab8..a0ebdbf3efcf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6076,14 +6076,17 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, if (!managed_zone(zone)) continue; - switch (compaction_suitable(zone, sc->order, 0, sc->reclaim_idx)) { - case COMPACT_SUCCESS: - case COMPACT_CONTINUE: + if (sc->order == -1) /* is_via_compact_memory() */ + return false; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) + return false; + + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_CONTINUE) return false; - default: - /* check next zone */ - ; - } } /* @@ -6271,16 +6274,20 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) { unsigned long watermark; unsigned long free_pages; - enum compact_result suitable; - suitable = compaction_suitable(zone, sc->order, 0, sc->reclaim_idx); - if (suitable == COMPACT_SUCCESS) - /* Allocation should succeed already. Don't reclaim. */ + if (sc->order == -1) /* is_via_compact_memory() */ + goto suitable; + + /* Allocation can already succeed, nothing to do */ + if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), + sc->reclaim_idx, 0)) return true; - if (suitable == COMPACT_SKIPPED) - /* Compaction cannot yet proceed. Do reclaim. */ - return false; + /* Compaction cannot yet proceed. Do reclaim. */ + if (compaction_suitable(zone, sc->order, + sc->reclaim_idx) == COMPACT_SKIPPED) + return false; +suitable: /* * Compaction is already possible, but it takes time to run and there * are potentially other callers using the pages just freed. So proceed From patchwork Tue Apr 18 19:13:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3FB5C77B78 for ; Tue, 18 Apr 2023 19:13:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AEA590000B; Tue, 18 Apr 2023 15:13:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7377F900009; Tue, 18 Apr 2023 15:13:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D7DA90000B; Tue, 18 Apr 2023 15:13:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 435E3900009 for ; Tue, 18 Apr 2023 15:13:43 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1D35216045E for ; Tue, 18 Apr 2023 19:13:43 +0000 (UTC) X-FDA: 80695461126.19.0D358D5 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf20.hostedemail.com (Postfix) with ESMTP id 550A21C0012 for ; Tue, 18 Apr 2023 19:13:41 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=NAETWyAR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845221; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=35b4U2+QOBa6ocJ8iDrcqYLn7rCBrGVWRfOUb6rO6cw=; b=ltbHf9F1Ogr+N1s7xxqm0Yv/TKlfnXzT8mghg7v087YfwJM5CjdI2Cfh75tYVE9zMuLI4H gILJqdJHu2o1DYBT4flJD7MQxD8oDyKPzGGn3yctnG77MIABiKCtTfhIZujFFsVbwkRfYn D4uqkOB8gNWfC2MvqLOkdb8qTMK03zM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=NAETWyAR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845221; a=rsa-sha256; cv=none; b=bc0I8Ec3Bkq/LK/l3TXIZiQekpTqc+1IOAYknHpuHlzIzmxG+xg2kokSPJEjfdOJqXaI/B Z0QyUggkEZ0z4WvaxxCPoFgLy9KQz5wMSYytLpzRrqrtvhB0aZtEe+RbccysG9wGEqE3Ig X9ojXI0kCdTg455vQ0tmgzHtBNEnDX0= Received: by mail-qt1-f175.google.com with SMTP id fv6so16946543qtb.9 for ; Tue, 18 Apr 2023 12:13:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845220; x=1684437220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=35b4U2+QOBa6ocJ8iDrcqYLn7rCBrGVWRfOUb6rO6cw=; b=NAETWyARHURXPOi9rz+E7BtKwnN/WCnP/+JO7LpCEDD8Yt6Nwt9wWNaKjVbRvN3xz2 CI34kcfshgq+HARendk3lJbfoBE5MkvdQWSrlrPBJRjmqsIYk3kDyWnUU7Frq+txzTlS H4HITpALdNoddu4n6BLDIclxULAYQC3G62W5TzlCuce+rhSeoOwKcEgfrWQxqneeYLbo 7wKg/+icg/zDzq1yq49DPPPiRs66pUC5NSRrssu+vBTZ4mZZ9Ce9QCpZuahqpc5imLz3 XtNWiOATBaotDlzMFP51F9IOzwQmK3bigZcA6HkmSjex3STf6qkddtjV7o1GG3sIRsXg ppGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845220; x=1684437220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=35b4U2+QOBa6ocJ8iDrcqYLn7rCBrGVWRfOUb6rO6cw=; b=RmjdwUHTYXdBbAdOqrzFXqssKieUrB9blRCXwgKvi6zqwy62HC2vv5Ypb6bw2tVjto QN6sOZwmOVFOXTqnagE6ME5NvpjICjyCX/77quKSOKin+Pti18MofKmmnaP++hY4pAA1 mrBx9TeTgy2717qEOLyUuDC6HDCJA29tnGlh6DAZh/9cFV1A/vqN88U5dt/nSPyIPeFn WW/6SlTwUpUbYIfCkG0C591mzAdCjxjHq/l9zPsgsYSWhkSkJnbvh4L+VzfSp7V40v+Y TlMfcDCcSOeFmTwhE/2sPjJImPJsztOVMXQYtLOlMsoX73Q6tvMDrIiv8u/wo7+RWrL6 p0WQ== X-Gm-Message-State: AAQBX9cvOD/37g+v1hs8tSGOef3QXfJj3oJ6irUhavnkGXHYObkOJbdL 3pKVbTPHXQoD7Z42jcwabRBuW/ngQmwWm767x4A= X-Google-Smtp-Source: AKy350Y9D95GmbNdWa9qjbsQ3a5RYAukR0KDWI7MiGw1X4tAzUYurG+5JZjzgpO5fAjmLVD2oLXHwg== X-Received: by 2002:ac8:7fd6:0:b0:3bd:1081:b939 with SMTP id b22-20020ac87fd6000000b003bd1081b939mr2038244qtk.0.1681845220210; Tue, 18 Apr 2023 12:13:40 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id d14-20020a37680e000000b0074d1b6a8187sm2482147qkc.130.2023.04.18.12.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:39 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 18/26] mm: compaction: remove unnecessary is_via_compact_memory() checks Date: Tue, 18 Apr 2023 15:13:05 -0400 Message-Id: <20230418191313.268131-19-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 550A21C0012 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: emmca1dfqkrwa87tcqqn7o7ss5jwxyum X-HE-Tag: 1681845221-321991 X-HE-Meta: U2FsdGVkX197YaftonD0D6WUH+xcuIjyMl27YjYCdd36ZW7R8uZs5oCuVRbwKz0q3L9I9lOmVGsZWzwv4BTdOuDx0Vq6NLEVhY+2BK5u+njTe8KD7M6B71OxA35baMKyAxTLWpuK0p7TM2YaMgmLxQJ12CUSTlb5L91UICVsNSiuoTpxpDG1VTxZmv4dlaFEeRK/mgh/NV4Qd+xa7spc9uW64isnBcCUdrl9+zbVARaxlxgEr/8LhkRNa2WgdGWPK3aVlOL+V6t/cLhSqx/6lRaLRfzndZFuc2hLkl3MF+73T3tHZurVyRJSgfDSKLaloG1shOkpbzgzu6XWbDOEDtOdorFF2zUEB8bI/s7p4N97jVTFuojPx1RyNxchv96l57m5nmMZ5OdeBaMTO6YPvZFNIjxrAv2HT4FnsSCXdpJ0r5DqkNA0ArBsivVTpQgwg3YE59mS+LtI7IudeWfVLL39u6lbUF99Ym+IwuCHJK7ZMHrre1OdCblqE+LlVm5UtYk7b9cG6NvrIbWC41q1+nTj/osjoiwSgYWH4sRrejxzMnrweoCjTz1Klsq9VGm7h+HgwqOcqKHLZgUsBWbExo7JXEoZxuEhziJS9K0glgupmoVNQRuc/kkoT5HAGd3tAfuqZCgWJeLThs1SKDHoNRi4oHNXP3RS7XBFHReA75abAylC7OQVUpHNvyWKiEtZ5Q1wddg+G/43laYaAIRIyWlEU0ErsHzLdeq8UMhg5WRtvND/Xl7jD1SonVVJcv6iinnDJ/Psk+6wEW4Nn9CbsjWShVLxds4yFQ4aupxE7hymCf8sguJIwDvUOcCKCFqZpbOIMl0R8s5JArdqg4DKHrMRGeMTboCdfM0KojxLEYBBEdhLXTLZ6e6KOLz1AdHJQ7JTOUKmIwZnPRHOguay29qeYmVNgwEl6eucZnh8go0WWQP4W+z3iPaaG1rW617pHkWAGDJ0hwEFc7u7Kvi HpNsARqy h3L5OZ7Dg1dFZckr4Ao+lksZ9m2gY4gLTSIbokVCqF6ecLMn8hCuRd59TgqtCX1Gv8qd8BwJO4xPTraydPcimUlXJMm85gQASCe2ZzTNKPPOuk/TU9atuKks3d/splBB4baVayztAoM8rNqxdi/vNGjE9NLgKOIHFapzXq4oTPrPmRedj/9G/hfyhsosXvtQIhs8eErJbimUR+W1Z3xLJKDvYu4y1i9Qk0Hy5Vdah6tkLhaWa+SUJGjnYiLclAA9Deof7k4tkEIypQxV3/PLnaguHYXl3CA54355B94+QgSjksgpHX4CpHg2fDcdyH/Lv9UMTB3OtHxfBEusVtarrd2mBORtxDu5nGdbB8QXQ8USsFLhfcXqZBfc3NECiuYSyhKz/hDLTsl5HFfAbFeipbthoDFb+T5UNwBKJ3yJh6rn2lhF/uVSsIWgLsh5pGlXEZlTyNhFLgAA+OpDerXL3pKDlWGnyEA3XFCAFqFJA1jL5auiLo+xvYsCZHF1it8+iyyORiAySXXape6zJisnBhQKBiHasBjqYsokL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove from all paths not reachable via /proc/sys/vm/compact_memory. Signed-off-by: Johannes Weiner --- mm/compaction.c | 11 +---------- mm/vmscan.c | 8 +------- 2 files changed, 2 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index d4b7d5b36600..0aa2a0a192dc 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2265,9 +2265,6 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, unsigned long available; unsigned long watermark; - if (is_via_compact_memory(order)) - return true; - /* Allocation can already succeed, nothing to do */ watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); if (zone_watermark_ok(zone, order, watermark, @@ -2808,9 +2805,6 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - if (is_via_compact_memory(pgdat->kcompactd_max_order)) - return true; - /* Allocation can already succeed, check other zones */ if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, min_wmark_pages(zone), @@ -2855,9 +2849,6 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (compaction_deferred(zone, cc.order)) continue; - if (is_via_compact_memory(cc.order)) - goto compact; - /* Allocation can already succeed, nothing to do */ if (zone_watermark_ok(zone, cc.order, min_wmark_pages(zone), zoneid, 0)) @@ -2866,7 +2857,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (compaction_suitable(zone, cc.order, zoneid) != COMPACT_CONTINUE) continue; -compact: + if (kthread_should_stop()) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index a0ebdbf3efcf..ee8c8ca2e7b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6076,9 +6076,6 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, if (!managed_zone(zone)) continue; - if (sc->order == -1) /* is_via_compact_memory() */ - return false; - /* Allocation can already succeed, nothing to do */ if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), sc->reclaim_idx, 0)) @@ -6275,9 +6272,6 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) unsigned long watermark; unsigned long free_pages; - if (sc->order == -1) /* is_via_compact_memory() */ - goto suitable; - /* Allocation can already succeed, nothing to do */ if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone), sc->reclaim_idx, 0)) @@ -6287,7 +6281,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) if (compaction_suitable(zone, sc->order, sc->reclaim_idx) == COMPACT_SKIPPED) return false; -suitable: + /* * Compaction is already possible, but it takes time to run and there * are potentially other callers using the pages just freed. So proceed From patchwork Tue Apr 18 19:13:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1E95C6FD18 for ; Tue, 18 Apr 2023 19:13:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C443900009; Tue, 18 Apr 2023 15:13:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8725F280001; Tue, 18 Apr 2023 15:13:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6754E90000C; Tue, 18 Apr 2023 15:13:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4E0A7900009 for ; Tue, 18 Apr 2023 15:13:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2D7A2120362 for ; Tue, 18 Apr 2023 19:13:44 +0000 (UTC) X-FDA: 80695461168.12.8602701 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf23.hostedemail.com (Postfix) with ESMTP id 6974614000D for ; Tue, 18 Apr 2023 19:13:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=mwxAjt5F; spf=pass (imf23.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bns8+mp3X6nXdCFxV2tvIrIcPUTCwJdDXntfPdL7pp0=; b=BZN0NGxa5tzS6UdoszkDBnjCzU9bepQeLQWa/0cp81fkHN3K4WtCComwMTavOhB3xJcQLV /XquBy6ZUgp84rp59L6GPOxJ1Ij9wdd6RJ9iNRRp9sKxxSRgqKbCfAAu6oq1elTKx7/Q+z Dm6iuXEWDX51rg19FHyDAnMWdDPUhco= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=mwxAjt5F; spf=pass (imf23.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845222; a=rsa-sha256; cv=none; b=DgCpXspTXfPsxJtvw5DtHdTNUCjX1s+UysImPCgjN7la5K/Vh+4YpDa8DtdWe51Iruq51E xAfjG6OLQHkaSzR73k6yIxRMTAHdgxdQh1KjrFcNQ3M0LXvgsWJcHIeCCyNawwpSpmUB4z Pk+1SFwyL60FhcgjCYUaa8EFjAs+jpk= Received: by mail-qt1-f170.google.com with SMTP id a23so25544622qtj.8 for ; Tue, 18 Apr 2023 12:13:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845221; x=1684437221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bns8+mp3X6nXdCFxV2tvIrIcPUTCwJdDXntfPdL7pp0=; b=mwxAjt5FHbXosh8eO+VsA0QaIzkPHSj509IMResMBLrFCDtFqQuAb39AWA+k+QpfGQ 2sTi7ak9g0/7anwaLuFZa8TnEH9/W0IsasBzTf35DDXZIjNZ/+iQTItjl6xvXY6x+qbV svWyu0WEhJTQsOP5r4KsHn0kvCVHMCqKPohNvbGE7v7GTuISWCmFDVvEjh+wYDMd6gPV QPpD51iUk1E7ChSCX2CJ4//Jg45qmYKpV2D2NrCvvOkGI9qVSedCxygcX/hpgxgpYiRw KQnkSn73bb0v0CpIU4XcXH/w9PR1l/pMtfWeS5WYklu48+aB4n8+2PHRP8wlzmglM4v3 6yMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845221; x=1684437221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bns8+mp3X6nXdCFxV2tvIrIcPUTCwJdDXntfPdL7pp0=; b=jpkwVuPN9NdgULoAXnhU9Mvw744U3QzC+jTS6OZCcgWMd+ac0OPudCxRujv8e4AGnd CiCq0nwjDoaL58z0ukT+9PgzBL020jufoQTZaCpltDkTYDzE9ZebLPIznY5csOPknOWq Jia5utiq5ekgE1bzDd9w2v98qv/n0D+fzfn59WJ9VzU8QWngNCh2dUw35cJSZxfhL9CB Px/j+zyGHyQzZAef8X7sIbTrSJsVg79mHhpeAtEyBr58tK6uYXFtwWYMDnVxRqI6ASam eXCk5bltN7S0FP8OBEM5mwPEXs2XSiru47gzFCmLqKvCf+UJyTndLRYIbtGE4ym72txQ x1sQ== X-Gm-Message-State: AAQBX9cj9vKYKpKVl2tVgX3aK5DZ71HaFC/ZdG9yaDVvWhS/QE/7c1Lh FEQSIkOlyNQA78SLKv1kgTht0WmnqEfKwUO04KU= X-Google-Smtp-Source: AKy350YxL97fUsu7ft+nTwgxWroY8EhxqBqMYvhNTWaLrHbHqzHqIxvgWZLrQ3ewFfKTOzgKcfLxmQ== X-Received: by 2002:a05:622a:20a:b0:3e4:e94a:5082 with SMTP id b10-20020a05622a020a00b003e4e94a5082mr1611125qtx.5.1681845221305; Tue, 18 Apr 2023 12:13:41 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id r15-20020a05622a034f00b003e4d43038e2sm1829712qtw.5.2023.04.18.12.13.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:41 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 19/26] mm: compaction: drop redundant watermark check in compaction_zonelist_suitable() Date: Tue, 18 Apr 2023 15:13:06 -0400 Message-Id: <20230418191313.268131-20-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6974614000D X-Rspam-User: X-Stat-Signature: wfdmqh9fg1qbknsymrtx6mk8crb46d43 X-HE-Tag: 1681845222-436338 X-HE-Meta: U2FsdGVkX19P+pWzIEzLxXTizO9x66JEM2I8g+fi+ucFsmYHewEtY+Q5GkDnR+qnGVEzpJ8ZSkLuHNn2SfyXZX8+ZB2uF811AmCQRj1YLJtk44Y58BrTu2veva+BU/qiDoMoZK7+PDezaHdCe9ZlHJbvNqfyrolwzzS5FQF6fmkAPv7g9ziZujnK2pOWrGaZ9jJxPXkvyGY2ondhfQzYAP48J6wXbEEGSnU37opemd8z8pk//7hChrN96k2T3/WUJiCSV13+k+As0B+AoHOiDx/8xBBTL/TF78QGbTt7uKdBGwaBUIsSgpJyvPY6FVFZ+QuF2stimMUNZESKj9ic4G3x9wILKyDfpaEoVHJIaVGvUvoYdhwZl3F+PCDGZAgJ53KZMOlLYFTlIeWUzJrWGbryxBudvQ5SjB8zJgGR000e7LjPJwgFmDztCdO7v6B4A0b00M8GasvpQelYQynAed4cvsixbKcxrlnGhcJiDiin7ZKRxO0Jd4fBLoG8Wqx0J+Wc+Dm0OgnDbVwWMK/vozgep/QhjQyd3atdmslx4nnHjRKpgFLv5WNXsdhfz8rtWDFp72JGiaAzA+oIkl780nRIRTxPRtIwyovuXyTyDHuDTVLK8mFaoLQJW0pqc5l3sI2IBZK92FibH3xZFXICEjzfpsSoMF1Cwht+3wunARyoLhen49oOf4bBDMq5DV8GyvXpA0xog15asvwtRV1ggZF053LdH5KpPj4f5fZ+Sx2iXewHfhyOe1q5lIrx0r4QkcSvF4p23Ns7hEs3U/4OpoSjz36wflk8OWL+YJPuAn688qad3LNU3vgtjvp2nr8uBrdjvBH7Q8pqMVIYW4aXC0OAknH5aFnJ36vMArp/6IkvqW+5qdFFSoSy7J9NqNdbcVyrFksGZPZBMHuYJJTphsiaXLm4bFMu82DQ+/bz76Wlhso8g3pM8H501GNzQtSdTBAnEdrgjmDS2ZyONPf ZttOzZ7+ wl/XMc1oCiaj1SfvTgKJFGrxECW8fIvJLuytNZMNaty4drTq5KZER21k0DDhfkjxNla4VxuNYdoDgjTHl2cW8GF9Stwym92Tq4OOia3Cg8uqgIxN8MXgf+qF1lwdm/wPA51xuLyrxvEWkslW7mNQZuZaEvEbTBOjq4cSYJjnYOlIpjStxXGpT3pq4ZaSYGCFI8PYKEUDpmtZkAqg28dctoZHzYHnaWsNcn6X4N8zoi5hZ1o5KRh/jc8GVcHGagP4LS2rn1BEosWm1E2HXCFGJn7EP51OUvtCeJMXtgyqrH1/j9ZbkOM5eexduvJRT9lOooCy9jnYrNhNPeNrAF7NNCaiKJJ8hsuzjzlucH9UloBncplpI22ZuiIig94sDkQBD8avm8ZIoMnjSGaloBkYwnb/AAPKiRAVSuxBnPrZNdBkITxCkgAfU1uJWXNxn/5fkPejl0AwIS0WWdljRtg4ZcqC+2xBhrzVJfcb7b/QRzoFVZt6SlvBZuaRdUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: should_compact_retry() is called after direct reclaim and compaction have failed to produce a page, to check if there is still enough compactable memory to warrant continuing. If there aren't, one last attempt at the freelists happens in __alloc_pages_may_oom(). The watermark check in should_compact_retry() is not necessary. Kill it. Signed-off-by: Johannes Weiner --- mm/compaction.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 0aa2a0a192dc..52103545d58c 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2263,13 +2263,6 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->highest_zoneidx, ac->nodemask) { unsigned long available; - unsigned long watermark; - - /* Allocation can already succeed, nothing to do */ - watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); - if (zone_watermark_ok(zone, order, watermark, - ac->highest_zoneidx, alloc_flags)) - continue; available = zone_page_state_snapshot(zone, NR_FREE_MOVABLE); available += zone_page_state_snapshot(zone, NR_FREE_CMA_PAGES); From patchwork Tue Apr 18 19:13:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E162DC77B78 for ; Tue, 18 Apr 2023 19:13:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 197B9280007; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AAD6280001; Tue, 18 Apr 2023 15:13:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC756280007; Tue, 18 Apr 2023 15:13:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CAB8D280001 for ; Tue, 18 Apr 2023 15:13:45 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 91D00120262 for ; Tue, 18 Apr 2023 19:13:45 +0000 (UTC) X-FDA: 80695461210.30.67B14F8 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf19.hostedemail.com (Postfix) with ESMTP id AD6581A0017 for ; Tue, 18 Apr 2023 19:13:43 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="ABr/1kuC"; spf=pass (imf19.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bD8cI1T5Y1yip08nzZ5ycgmWClJik+UbLC7ebo3KxqY=; b=B5Rc3eUTa95g/RyUDt2qhHMqQd7PZtASoT2ZDtjzcOYcROFcRumTCLbe4sxksg1uIHHpU3 rJCHk4a8l+uV4MYVixr/REsMFv2O0NKgYUNhimmENEJXUkHn13SVJGfe0tF3oPaQYt5WzZ m5PcokVWd5r/bMzcbXsRDIgdIy79W4U= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="ABr/1kuC"; spf=pass (imf19.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845223; a=rsa-sha256; cv=none; b=pdo+POdERiibLD6jOTQT/WVboVmWFLk8lAayJQ6VQjFYuAC3Dc3d8pDvHtoTgMvVgcz0qP NZHAsqeZC/h9oxgB6GJpWfSRQYKXNCFS7piHZ7nXotoF8aVQ1/OXHG7XDUAmHJEQG8yTY5 Or/5jtpo9cBa3FGpcwteocBgxM3N6hY= Received: by mail-qv1-f53.google.com with SMTP id me15so9556515qvb.4 for ; Tue, 18 Apr 2023 12:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845222; x=1684437222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bD8cI1T5Y1yip08nzZ5ycgmWClJik+UbLC7ebo3KxqY=; b=ABr/1kuC79+cxAX8L6WJR2/bWTqWb8HcQEjFbpPZF0HuB9YW1YJMuRW7gyOVzYVPRw 03H6aLq7CX52ef2Om4hh96VGqT7TjGkiKZRVbtTbvnP86DodSH+fBT3Bg8K3RBiskmEn PBPAPvjvrcmBXvcXT5D1ENEFAgwZsOibeMRYfxOUHZ/XIjDWRlAOEno0v6h4ZjF413dj hOYeaOUBMh9OFP6N6OvsQe4bSQ4Xtq8Jibe8ALXvvyUnSx8zCdaZb/tqXiSYgET+QTx0 uVr7LhBCB60jsNBnVVB4qFMsgKQHi2Oe9RlYbzhzQb172EdH3Z+Txzl+m/HRkKeizyY4 nKRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845222; x=1684437222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bD8cI1T5Y1yip08nzZ5ycgmWClJik+UbLC7ebo3KxqY=; b=a0pK9TiLAND6QUkAivdBw58NeqY2lgGma5gJXb+wqtwwYENgaPpKDftQBpSTk6g0Zf 2nzxpz66x39u7RYO4PLpqX0ZjrjZQERLbv9n9DhV4LomtkdF8ILTzuJ3z2fjdDQNtQ87 X1dujrCU+rJ04UHJt6D1+MC3t7KLMCA0GX381LsdlnB98G5sVdBexw6I7OfFrQYifAHY uyHx+BxOk2vASIx/67N6T9OZC1qnncZyzId3Z7PCovA0Y60Lpr58qV2DgJpKfSmImSF8 0iGcR3AAOeABI98Y1jkP/oy63ot2kLzHFzr8GigllfB8m3NdX3vdgciLpnX6GuBa/Fz5 dMcQ== X-Gm-Message-State: AAQBX9dw4rTeLTq54PyJShA9yUAWsHGJhBf+PmuRk/REj390Ex+w/ydL IgTTIr1q1+K6VR0ZTtdYDQD6u21TfTijwNpYftI= X-Google-Smtp-Source: AKy350ZedblEphVsszu1HNXxFw88IPawTMC/tacVCADIVXo/ITvdIoDkqo0MjIIYPO3ZAuvFaGP6+g== X-Received: by 2002:a05:6214:e49:b0:5ef:564a:3296 with SMTP id o9-20020a0562140e4900b005ef564a3296mr17532356qvc.44.1681845222547; Tue, 18 Apr 2023 12:13:42 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id ep19-20020a05621418f300b005dd8b934594sm3931380qvb.44.2023.04.18.12.13.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:42 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 20/26] mm: vmscan: use compaction_suitable() check in kswapd Date: Tue, 18 Apr 2023 15:13:07 -0400 Message-Id: <20230418191313.268131-21-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AD6581A0017 X-Stat-Signature: dmg5d5e5dof863opsb7hsqnq4h98s6im X-Rspam-User: X-HE-Tag: 1681845223-843021 X-HE-Meta: U2FsdGVkX19+kv2LG9oSYQ3j7/hbykLQXgWzE2AD+qdHjnEBtmOUyapHFTzn5/19uxh+NEejeib9YZuoMUGC7+TfeAE/rOgXqkOjrh8kmklESdyBe22YF/FyTx4KB3jyoQQ0ZQWiohdbTfFL5SccvVe/iFhJkzoGlvhJK1CT2xg98ThHj2/5cBvAtuNlxhSUiObaFqORgmX9IiHwQOoAa50PN+ZlpUNAYd0R+gmAXAIk9OBTLFUr9iL6l5whPJoSBqyeIYEnDWjuEkLR69fKcDN3Td2u9B93RZUkgHzX6M+aH3uxvO7nuIinENypZueSU4r4JIs1gxmMLM8x1SeHtQuw8Lc3kGRcP06VgkOF9jIgc17OnniFFR0tJXI37nDpfZgKOeeisPd7hRZ6lQBb410a7FbATbfU+QR9m1GfmaBNex1ifd5C0o96iAqcppcK9Gf3tzOlH6u8u7fbZ5wu1R/z8eTfJx7iNcb0ZKmoaxmcTW4TCYQRMVgg5lS9LZSdAyaH06ZKnHdbZzxpEvre6eugoe9fb/mlTJgScNy6ZWW4069LXC2/4EJ4MDKhSH9jEwlgfp0NUZLviKWJjQwZr0OwFUqroUdr26M5A/LZFggty8oZnSct8IWj0G+bVtiIfap2PJaAneB5N0jb3FyXQL/tsD2VpzIA43cqpOswTAyuiFWNPbg014LxV9y8PLcTdY3oPjbMmU173j25naiAKPcJSlJr1I9TvlPZu6f+cBqaZG3nH5Gzk6a4ooP4QjrnvTkZ0n4NGNqO85onojKyYGR/gDC6cIjPyJCwRUSxCClYr1ocexMWZiQCp4jiVq5/2DOG9OF761HdakgdjzfUegu+Iylz1A1MAs265KAjLNtKWLJe/bdDSVVip58qvdZIz2AQkUEeb8NkY2MIbhcOnek/g2Osy8umisIZafYJ/RaGMBHP1f/2yvrL+XpgXaiVjPxzfjmwDFNAdou83jJ Mfj43DYZ A1PzMDQDBSk5ZUDeW++qhWokPvs7SXQXb4H7np840fyfvDMCPjr67ME1zn0yunxXaVEPEhy5Wz/jFeSTdNEk9buals3XZiMiYz56mZ0T1V9NHp8OpWld2XJjMe5nSFZIhjscHkJTfWM8DOSf7LdzE+AX9KKyW318+GofC3shCATKujZJesw0IkTMiiUlNCosxdmzRvT7V1vQ/BtmSBo++lH6Z/Y28NX1rPEXLz3lcxk23wj+BwLAC9/+6dZmsitDo9DIyEQP65OBUV5GOKrYCm2gPAgR6O0o96rADVuqF2Nlr/XOrod+SX9Ns6RhQKSkBaSMFpOzhkMhUlwiF5GWw3RWyqYHEaD3FD9bZ5pGFV1n/zo32e+q8c6qD4XFCB889Q5bQNZ6tE9t72eR6b9EY88h/1wHHitNzeHMa8fM+yubKCeF68JqwsUF1SHEywvpcEvtEnHR8PHISwTRx1/yIMeE8gA8ORNimfzrLXFYpBmnksLCemrmJStJUryV3Td72kUb8gQ8aps1PI25LH9lYXk3ziw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kswapd currently bails on higher-order allocations with an open-coded check for whether it's reclaimed the compaction gap. compaction_suitable() is the customary interface to coordinate reclaim with compaction. Signed-off-by: Johannes Weiner --- mm/vmscan.c | 67 ++++++++++++++++++----------------------------------- 1 file changed, 23 insertions(+), 44 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ee8c8ca2e7b5..723705b9e4d9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6872,12 +6872,18 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) if (!managed_zone(zone)) continue; + /* Allocation can succeed in any zone, done */ if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) mark = wmark_pages(zone, WMARK_PROMO); else mark = high_wmark_pages(zone); if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) return true; + + /* Allocation can't succeed, but enough order-0 to compact */ + if (compaction_suitable(zone, order, + highest_zoneidx) == COMPACT_CONTINUE) + return true; } /* @@ -6968,16 +6974,6 @@ static bool kswapd_shrink_node(pg_data_t *pgdat, */ shrink_node(pgdat, sc); - /* - * Fragmentation may mean that the system cannot be rebalanced for - * high-order allocations. If twice the allocation size has been - * reclaimed then recheck watermarks only at order-0 to prevent - * excessive reclaim. Assume that a process requested a high-order - * can direct reclaim/compact. - */ - if (sc->order && sc->nr_reclaimed >= compact_gap(sc->order)) - sc->order = 0; - return sc->nr_scanned >= sc->nr_to_reclaim; } @@ -7018,15 +7014,13 @@ clear_reclaim_active(pg_data_t *pgdat, int highest_zoneidx) * that are eligible for use by the caller until at least one zone is * balanced. * - * Returns the order kswapd finished reclaiming at. - * * kswapd scans the zones in the highmem->normal->dma direction. It skips * zones which have free_pages > high_wmark_pages(zone), but once a zone is * found to have free_pages <= high_wmark_pages(zone), any page in that zone * or lower is eligible for reclaim until at least one usable zone is * balanced. */ -static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) +static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) { int i; unsigned long nr_soft_reclaimed; @@ -7226,14 +7220,6 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) __fs_reclaim_release(_THIS_IP_); psi_memstall_leave(&pflags); set_task_reclaim_state(current, NULL); - - /* - * Return the order kswapd stopped reclaiming at as - * prepare_kswapd_sleep() takes it into account. If another caller - * entered the allocator slow path while kswapd was awake, order will - * remain at the higher level. - */ - return sc.order; } /* @@ -7251,7 +7237,7 @@ static enum zone_type kswapd_highest_zoneidx(pg_data_t *pgdat, return curr_idx == MAX_NR_ZONES ? prev_highest_zoneidx : curr_idx; } -static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_order, +static void kswapd_try_to_sleep(pg_data_t *pgdat, int order, unsigned int highest_zoneidx) { long remaining = 0; @@ -7269,7 +7255,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o * eligible zone balanced that it's also unlikely that compaction will * succeed. */ - if (prepare_kswapd_sleep(pgdat, reclaim_order, highest_zoneidx)) { + if (prepare_kswapd_sleep(pgdat, order, highest_zoneidx)) { /* * Compaction records what page blocks it recently failed to * isolate pages from and skips them in the future scanning. @@ -7282,7 +7268,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o * We have freed the memory, now we should compact it to make * allocation of the requested order possible. */ - wakeup_kcompactd(pgdat, alloc_order, highest_zoneidx); + wakeup_kcompactd(pgdat, order, highest_zoneidx); remaining = schedule_timeout(HZ/10); @@ -7296,8 +7282,8 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o kswapd_highest_zoneidx(pgdat, highest_zoneidx)); - if (READ_ONCE(pgdat->kswapd_order) < reclaim_order) - WRITE_ONCE(pgdat->kswapd_order, reclaim_order); + if (READ_ONCE(pgdat->kswapd_order) < order) + WRITE_ONCE(pgdat->kswapd_order, order); } finish_wait(&pgdat->kswapd_wait, &wait); @@ -7308,8 +7294,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o * After a short sleep, check if it was a premature sleep. If not, then * go fully to sleep until explicitly woken up. */ - if (!remaining && - prepare_kswapd_sleep(pgdat, reclaim_order, highest_zoneidx)) { + if (!remaining && prepare_kswapd_sleep(pgdat, order, highest_zoneidx)) { trace_mm_vmscan_kswapd_sleep(pgdat->node_id); /* @@ -7350,8 +7335,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o */ static int kswapd(void *p) { - unsigned int alloc_order, reclaim_order; - unsigned int highest_zoneidx = MAX_NR_ZONES - 1; + unsigned int order, highest_zoneidx; pg_data_t *pgdat = (pg_data_t *)p; struct task_struct *tsk = current; const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id); @@ -7374,22 +7358,20 @@ static int kswapd(void *p) tsk->flags |= PF_MEMALLOC | PF_KSWAPD; set_freezable(); - WRITE_ONCE(pgdat->kswapd_order, 0); + order = 0; + highest_zoneidx = MAX_NR_ZONES - 1; + WRITE_ONCE(pgdat->kswapd_order, order); WRITE_ONCE(pgdat->kswapd_highest_zoneidx, MAX_NR_ZONES); + atomic_set(&pgdat->nr_writeback_throttled, 0); + for ( ; ; ) { bool ret; - alloc_order = reclaim_order = READ_ONCE(pgdat->kswapd_order); - highest_zoneidx = kswapd_highest_zoneidx(pgdat, - highest_zoneidx); - -kswapd_try_sleep: - kswapd_try_to_sleep(pgdat, alloc_order, reclaim_order, - highest_zoneidx); + kswapd_try_to_sleep(pgdat, order, highest_zoneidx); /* Read the new order and highest_zoneidx */ - alloc_order = READ_ONCE(pgdat->kswapd_order); + order = READ_ONCE(pgdat->kswapd_order); highest_zoneidx = kswapd_highest_zoneidx(pgdat, highest_zoneidx); WRITE_ONCE(pgdat->kswapd_order, 0); @@ -7415,11 +7397,8 @@ static int kswapd(void *p) * request (alloc_order). */ trace_mm_vmscan_kswapd_wake(pgdat->node_id, highest_zoneidx, - alloc_order); - reclaim_order = balance_pgdat(pgdat, alloc_order, - highest_zoneidx); - if (reclaim_order < alloc_order) - goto kswapd_try_sleep; + order); + balance_pgdat(pgdat, order, highest_zoneidx); } tsk->flags &= ~(PF_MEMALLOC | PF_KSWAPD); From patchwork Tue Apr 18 19:13:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6D21C77B75 for ; Tue, 18 Apr 2023 19:14:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8987D280008; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84891280001; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7104C280008; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 60E42280001 for ; Tue, 18 Apr 2023 15:13:46 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 41026AAE45 for ; Tue, 18 Apr 2023 19:13:46 +0000 (UTC) X-FDA: 80695461252.24.84B265F Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf10.hostedemail.com (Postfix) with ESMTP id 6C672C001F for ; Tue, 18 Apr 2023 19:13:44 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SD0CfPNo; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845224; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=1AR7P9x05XVNEZazx/71+1W44dg7SzJKFkskkKjpv8Vj072IkvxJA+aSyW+LvriI2oAzqy 3BHikv6AmW9BH+QEcKErOf1pP8OlM7/CCcdbfSlQN+R4hZqlOPiTBPT0jxrZWJaqfzSbdU 64pUTJMQetyPA02cLLeiqpZKoEIXVcs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=SD0CfPNo; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845224; a=rsa-sha256; cv=none; b=bbI75V7v+s8MQEkk6Aoo1f4Dgr4TumSiVoCgBGtIWNqT6AjTzqhWwNkNmcGzXmZzDlXFAr AcCp/iATIGSY8z8lDu5wRgQJg8iNaMP+Tk1Ef7b2Yh7/EEGdWe//NO40/hnveoQKsrCeq8 aa45uF6ZWg1LXZ2m3A96dKH6KRgG0Wg= Received: by mail-qv1-f46.google.com with SMTP id h14so18464855qvr.7 for ; Tue, 18 Apr 2023 12:13:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845223; x=1684437223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=SD0CfPNo/FtA52mn8PTuMlWwsWPdBrYNWFQm6HroBkjTNWL2qXQf7wrFmhK/WWm7IP xcIXlFfLFGEf/BrGyHz+UE3kHnP2QzzQB7WXPEZFLWPxISNTxWhAaa0h1tT0olaHgEUs 0hpjB9LzjiA5OHWB18U6c4fyqiiCG4R128OFbaQXw6cA+VNmAHuxaR45uultoYesSJx0 CVUbWebDBfJA3QQt/c1BCijGivD+dCjM9agqh7+5bqifw5Letf8PMz5QSYbJeZ5UTCUP tSHckFJNQ1uiBdu7JKbqXKX13Y5T5FnoLNhX7VZobyOD8tLkxHSMFOnr57kBpsZRcWht QozQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845223; x=1684437223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hsTBnxogc0dXbXldiQnTcfaSMnaw37pvCiHuIsUwhKs=; b=fAiwfPo9mZlpSOkeg/DxHBycx9njbDe/uQjrszkaxeVo/3EuGLd6BQlmlxDNCjtTFD /603WTsGVrQ4xPHqLxNaWpP/03JzIJdyHZehyhGSbMXxaVAPIsdQhSyJ5F2t11mEp08p Rqokm1kM+dl0r77OoKqMN82D0BthnHOVrnsnEfZzYJ7qFoFGNi+A47zbA9Qrmmvy7SIm W/AK4l0P3kLZhniIMMPLHR6KZQiJK1Xau2mdxej4gUlcnTrIgtN8Y3vpjoh5GpL4NOIg am/af356EIUCB3TsPHjvh4G/xY3C98TIK1SNAgqTj9DGDgGMGe8SGT5AZaD/KkCSB8DQ Cn6g== X-Gm-Message-State: AAQBX9eKs/QKWoOohwkpdviu7TkSNsyiL7VLfzcx8gadc95AKBc0CXUd eVJsNs03OiEJJTAUYsj5SqUrCFITI0uGWukr1as= X-Google-Smtp-Source: AKy350afvFhbEDO4u0w1iKrPXbytvWWwl+mYLWsq6qZEuaKTZTEbfiazBxZv7n8V0c8QP05wvdn3Kw== X-Received: by 2002:a05:6214:2607:b0:5c2:a8b0:d71a with SMTP id gu7-20020a056214260700b005c2a8b0d71amr27349805qvb.43.1681845223606; Tue, 18 Apr 2023 12:13:43 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id x18-20020a0ce252000000b005dd8b934598sm3884188qvl.48.2023.04.18.12.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:43 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 21/26] mm: compaction: align compaction goals with reclaim goals Date: Tue, 18 Apr 2023 15:13:08 -0400 Message-Id: <20230418191313.268131-22-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6C672C001F X-Stat-Signature: mit5erwgrnpeupdbgcu5iqg7xbtmzsn3 X-Rspam-User: X-HE-Tag: 1681845224-321041 X-HE-Meta: U2FsdGVkX18Jp4kw5uU29UcuyU9LTSi6jfgbuO0+8qtQLTUXl2SaDNo+YEqcGWfEHEiV1cncF1WebVK3TJ4s2XHEtCRVOLuXFCtdADG4C/WJAWrHutE6tp+rRLtt6ODfTLe5ZNoksSiCmyLtJBCmOnRyGZZBbvjS6JWxZfsT0LojSED438S8oMsxH10L7jIvaL/libiRFFdn0H0At1y1sn5x8dOl+7qE4EgCtPTAE2EFoQ6JAcxLPE6oLqW3TZVfsZcLQNqPPuegArUF3lzp4knG3GCFuBqnr04W72BjAFbQ6ulOs9PDTh8qseVZtEV2bdl5vFahK37I2zj8o1Chjky3l6O8fpTZqE553ZX80gB8d+XWvE0YyRvolQxTte1mBov9AWnAZZ+KfHMElZ2lBuCraEVdesHUnbPEEPfGvOMNUsBjtQlMm4fgaZCttwdlHpBp7QvmLQ191Rldz7Vf+xEqhIYLgiSs3BSJ8sQwBLxkX5MhvlRDodp6uClSPEYDpEL3mstWKkvXrTwWmKGSzbFA7/1YEZyi+Mjj/Ls/1Z39nv2YZn5TX82F+Fboajmzm0iSWagfSHX0KSmOizfy5GxKjRpoYQ2twZhP/mTQ7mOzmcBU8bHcY2SGzD5t1B8xcP2Ez0/WHLjepOhA9/z86hSI3AwcOJfwKgczXkvQgWJibKZNHmBZmVDsPl4Gpt9z4CBvpfcuPOClVdJnrjauRrapEQWuQFS1HKJQwIpHGRvK8LyfissJDNwEnDUqC1ll/HX4DMK3oNG/J1VNs5i9Ba7QVCS6fFbfP8nHN5n5+B6b6349Am0+DSeSHWet0k09Rm5LGFbaaPvVW75v6F6FPDlv6v4bEwH9bJdrtgBs0jd4c3n99YrMxv/0mQ+0ofp9q+X3AnYnxa8qkZdsCVNKYXpI+eVFfsOdgUK9DkXXKqbR8JEUloznkSC18zO/kKNWTCMrkW84ZWGqjMtAQjG nEcxCs60 rcTI6CRs2ewKoLngGklrk+RXlifkb1EPQeh2/LSYYhR23gQVCFJMIeM4LNfixRlzOYrnkiY770+NlgIazAjDTyHZ5tj7MVJ/uCHfRlp5To+II9dx37CjL70hMaPy25W+fKh5CbZufoYPSEjT2EmVhOQK2YtovuidtcpQiPWrNAxTtfACiuvWdXg4eHdoc6om/jaLxA0Cgw1WkXLWsOT8zWMfTSY4gAmz2gSC0Wr0HpjtFFbG3DEu3AUvEUR48proV/81shcvRhanSqdgvIFCD2PJgLc5kITU1f7cva2QBHyfNWMJ06UY7or2iRwD+IMhEyeirAkvARQiLzAOi3ibrmy5bffNHDZGfDHr5ZjrCEpAXcgmMuyxz4klOw/dlEMKejjqErZhTeCl0ti6yADidtqbJcC21lJJ6MHI1MOgMRqGSfQf9MQVPIZGN8qnsV+6OT84WzBXCMXQ+Dn78SmJC8JezVE94EUnNNkCujF87Hv/7C98= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kswapd's goal is to balance at least one zone in the node for the requested zoneidx, but no more than that. Kcompactd on the other hand compacts all the zones in the node, even if one of them is already compacted for the given request. This can hog kcompactd unnecessarily long on a requested zoneidx. It also has kcompactd working on zones without the cooperation of kswapd. There is a compaction_suitable() check of course, but whether that is true or not depends on luck, risking erratic behavior. Make kcompactd follow the same criteria as kswapd when deciding to work on a node, to keep them working in unison as much as possible. Likewise, direct reclaim can bail as soon as one zone in the zonelist is compaction_ready(), so check up front before hammering lower zones while higher zones might already be suitable. This matches compaction_zonelist_suitable() on the compaction side. Signed-off-by: Johannes Weiner --- mm/compaction.c | 5 +++-- mm/vmscan.c | 35 +++++++++++++++++------------------ 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 52103545d58c..8080c04e644a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2798,12 +2798,13 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) if (!populated_zone(zone)) continue; - /* Allocation can already succeed, check other zones */ + /* Allocation can succeed in any zone, done */ if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, min_wmark_pages(zone), highest_zoneidx, 0)) - continue; + return true; + /* Allocation can't succed, but enough order-0 to compact */ if (compaction_suitable(zone, pgdat->kcompactd_max_order, highest_zoneidx) == COMPACT_CONTINUE) return true; diff --git a/mm/vmscan.c b/mm/vmscan.c index 723705b9e4d9..14d6116384cc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6277,7 +6277,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc) sc->reclaim_idx, 0)) return true; - /* Compaction cannot yet proceed. Do reclaim. */ + /* Compaction cannot yet proceed, might need reclaim */ if (compaction_suitable(zone, sc->order, sc->reclaim_idx) == COMPACT_SKIPPED) return false; @@ -6357,6 +6357,21 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) sc->reclaim_idx = gfp_zone(sc->gfp_mask); } + /* Bail if any of the zones are already compactable */ + if (IS_ENABLED(CONFIG_COMPACTION) && + sc->order > PAGE_ALLOC_COSTLY_ORDER) { + for_each_zone_zonelist_nodemask(zone, z, zonelist, + sc->reclaim_idx, sc->nodemask) { + if (!cpuset_zone_allowed(zone, + GFP_KERNEL | __GFP_HARDWALL)) + continue; + if (compaction_ready(zone, sc)) { + sc->compaction_ready = true; + goto out; + } + } + } + for_each_zone_zonelist_nodemask(zone, z, zonelist, sc->reclaim_idx, sc->nodemask) { /* @@ -6368,22 +6383,6 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) GFP_KERNEL | __GFP_HARDWALL)) continue; - /* - * If we already have plenty of memory free for - * compaction in this zone, don't free any more. - * Even though compaction is invoked for any - * non-zero order, only frequent costly order - * reclamation is disruptive enough to become a - * noticeable problem, like transparent huge - * page allocations. - */ - if (IS_ENABLED(CONFIG_COMPACTION) && - sc->order > PAGE_ALLOC_COSTLY_ORDER && - compaction_ready(zone, sc)) { - sc->compaction_ready = true; - continue; - } - /* * Shrink each node in the zonelist once. If the * zonelist is ordered by zone (not the default) then a @@ -6420,7 +6419,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) if (first_pgdat) consider_reclaim_throttle(first_pgdat, sc); - +out: /* * Restore to original mask to avoid the impact on the caller if we * promoted it to __GFP_HIGHMEM. From patchwork Tue Apr 18 19:13:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A505C6FD18 for ; Tue, 18 Apr 2023 19:14:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5807280009; Tue, 18 Apr 2023 15:13:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A934A280001; Tue, 18 Apr 2023 15:13:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D4C2280009; Tue, 18 Apr 2023 15:13:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6BC0A280001 for ; Tue, 18 Apr 2023 15:13:48 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 242A540459 for ; Tue, 18 Apr 2023 19:13:48 +0000 (UTC) X-FDA: 80695461336.04.ADB285F Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf17.hostedemail.com (Postfix) with ESMTP id 34A3F40021 for ; Tue, 18 Apr 2023 19:13:46 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=tVrrto59; spf=pass (imf17.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xFydGPrrlcR5ue8H0k4/5BgZKiD2zceF6yJIdiDg/e0=; b=2+X+tvDRNUWKdB1Qx5c9bbJIoFVkfgwtO1FDdzGBbevwTBHe0OUI0NWS673dkv71c2Epfl mltno/+QfV7Md6pjT5MZ+MCyy7JZUHwmicNi3Prmk+VO2K9YxNnhdR+WlRAoxkyt62nqgR o4/3s+FHdW1oabwNX9bPc+6PrD9k7Bs= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=tVrrto59; spf=pass (imf17.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845226; a=rsa-sha256; cv=none; b=NP3j+ljVkfv8bSJ/YDxOhXsp9/4UkPc015G0jfgZ4MmhtJApceUcjDgHWO4t17n9eOAgAb Lyz/hZtQubxdUuAMG9/uavSjcIosdz2CFmAh1NDySfUkWo+KukahBmLI7c6fGXA0flpqiK PXBl05LHOJaJndtu/Q7iGtpEuzrd+w4= Received: by mail-qv1-f49.google.com with SMTP id qh25so19656694qvb.1 for ; Tue, 18 Apr 2023 12:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845225; x=1684437225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xFydGPrrlcR5ue8H0k4/5BgZKiD2zceF6yJIdiDg/e0=; b=tVrrto59kTHt+5P8OO0NTl96kO6fYFXftlF9A3CAuw7mkP6vDPrG6Y7lZfcyBzps7q /gMDUjmhruC6JAgv2lNt70nZGvgEKZPuKRjT85XlnbmpS0gHScOiwvV0CeKGtkebfUO5 GuRu2B/dbtgriKFhkU+bqrvVfqiYW8ac4FZqca9W6ySikMkgizqrdHAjbR5vdbcQrVed qXeZWHFuDKk0K/VtTx0r2tcS0PQ0RyfZX+iVchXZvObxU+M15TeBNYCSjPyOC7INUTz4 z7Q+PUSdqHdSQo5iQxaxwIudluosvB0gAAvUePxiBSpUWodkFxlz3QjRjOpnT/xEQ3i/ lkaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845225; x=1684437225; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xFydGPrrlcR5ue8H0k4/5BgZKiD2zceF6yJIdiDg/e0=; b=fmYGY05Z2ALF3tOynyn3VneYee/IhnfbU2GUMD4VF0gHNeic3VaZHRJZ5pigq+vkLn fEFkeLHfsND4UqvwDynotQsoxygkC2pO/OtGDfsQtt/ywbJI2NTziarKPI822iwpukaA G2QG4sJbTD2ng1Yygk5qocMNCsHoxBh4vr298Wtdc1/MVieCg8JixN7xxVyQezLQvnd7 79lPan8zmSNvfw+dxM1+I0OE07euEHKkdd6t1HCX3kBp/xnOdUE3JM6wAKh0VbuWiBlm z5NanpeTMgDo+HMlKaSERxP4g9k7D6Jh54+cEclpmxVaOtRygJlmcOle53ywH1x08Xji yh8Q== X-Gm-Message-State: AAQBX9c6rBaWExRei/abgOFA5AvgyVZtvGAUt3lpJz97DGy6Rs5EMu/J FJbPxEsOSmbQ7qLcVBYl5Oy0d1WA/j1o7cqAS4g= X-Google-Smtp-Source: AKy350aY/eDWP5VAtiKzy0kcSE6ktwznpTC01RhU2t2EYa7ev5AWfpSsbruwvnwNAwe0Wpm5J/BisA== X-Received: by 2002:ad4:5b8b:0:b0:570:ccb9:a4d0 with SMTP id 11-20020ad45b8b000000b00570ccb9a4d0mr20299103qvp.16.1681845224818; Tue, 18 Apr 2023 12:13:44 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id pp26-20020a056214139a00b005dd8b934571sm3935790qvb.9.2023.04.18.12.13.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:44 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 22/26] mm: page_alloc: manage free memory in whole pageblocks Date: Tue, 18 Apr 2023 15:13:09 -0400 Message-Id: <20230418191313.268131-23-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: fhr5n9nug6wa1oa91epymhnr9yexy73t X-Rspamd-Queue-Id: 34A3F40021 X-HE-Tag: 1681845226-349836 X-HE-Meta: U2FsdGVkX18WoQUbVGH4+Y8p1q6U7Os7K4616CupOqMAsc6NH18cA7W+US3phcwVk0Sojhr8/zJg+A7KN/uBbD7pbR7WwzN3E97fD0VGlUJYZaynXQ9VP71IvwnwXIGJkDTxpFZeT+RzeDN+WoYdTQ5AtRpwqpiVQJ5dI17PeicCg1FqiABqIHb6UGVvkt7HGloC+k1Bv231VOyeYtGx1llgRNe668bn/HCliQxBkhrfkBq3Eg192ntnwdiwLcHbmUMqINFPuNgSyFik7iZbI2kwkfBmzQGyw42qCxcWYejOjJFN6ZY3oXPwBjqamdlRNj1vbm5whBNYbm/U6MtexxHHc7UR66q1q/8W3fWLO/Mo93l34pNVcMp98uZ7Frz7uk39bLBHAV45xgE6lRwwUY2KWO9xguLn00VKLBsCoU4Fe3lGtoZmc9GMld9p8NVec4jlQaxM+ZRnklGrZlUEl0rALb8juHAjnDBvh3LYWv8CO0dHfUVOzkPxNISbIuzaO8t23TyQdRTrbf1taE/2+NsVmgxVV68g+v5FjUoL4NCIzo/JKsT0AAcNyC3xPWMuSLqUwgKsgh56TD8LDB69hryndMjUaDg8dhv4yyjbodrjU3/JkRp+8hVEcKqnMTfl5F1qDQuJKVtsb48rKvgwBwLzTSuvcqdWPkUzQB531KX7R9PwE0Ks5jvUm+r//8/Vevc1A2BJz/Rbd4NYMp62v86DngIzMEj+B4MBmqYfwBySqbx+Y3E4pYgtc9R+fOo+7vAFw7UKYgiA1H2muYVK2ZsL2G1pn5eHzy9SFNj4KKSp6yT4Q/go64Y16HFLizz4rURwmOpW9YfL2Ly2CEfHpdbUstT3rSTkIOrMQvjK+pUG4LTCzAfh9m0hVuzeyWE52n78Lpri9bMv/DkX8yS3SvNiDMJxExDqBodou2LMTDFsjrZlRtNl6eXLzH8Jh94kwIkjr4rOh9ZB+b+q156 7YbLn2G/ SgEH/YnfNokPNy0WnO8mTWeKRL94DydECsZAJw/LV1uxJ45DfFdsG8OUJc2L/QaayzcF6Gng0sXAAzwgN93MC4agxDCJKOFAHU17kvYxpJ05l/YXp/JtrWex+YVOOaesN3abK5TmHDxI4LcG+5UhVwSTDfxtLhI+OHuVH3fufBZjzK0cms3qtV35uctQ7HeuR9G+YeIG8zdcYcLItzg23VjOGmsWMJxUKrB4AQYyKXApqsFNjKy2NBnFvSYDhIDipt8SPxmIy/+HZM1uNw+haHdD+akNjOlFsamH0whPlE7y+p6IXjyJzNOWLG736lJEvKRi2zH/0brybGA3T0+bsjpKoOIe9d2umwydPgFVQa7g5pEwsY1SLcTTzhCt8xfYn4U2YLHlD8qtLs/CQ1gytjJzRlt8W6egQlLzeYreKB2yydTDYfQ3vzQuYazYE81YmQ4FjZbn+DZ0hXBfhbcuJVoQTSwV5wJDPuvDozLCeMT6aqnFAbS+S7oAitsHAkysQ+hgy4xFXmYDhWmx5ngT7bXvV1VgHP9pJLe1O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right now, allocation requests only reclaim (and compact) for their exact order. Since the majority of allocation requests are smaller than a pageblock, this is likely to result in partial blocks being freed, which are subsequently fragmented by fallbacks. This defeats the allocator's efforts to group pageblocks by mobility. Fix this mismatch between the allocator and reclaim/compaction: make the pageblock the default unit for free memory by enforcing watermarks against MIGRATE_FREE blocks, and have reclaim/compaction produce them. Signed-off-by: Johannes Weiner --- include/linux/compaction.h | 1 - mm/compaction.c | 65 ++++--------- mm/internal.h | 1 + mm/page_alloc.c | 183 ++++++++++++++++++++++--------------- mm/vmscan.c | 6 +- 5 files changed, 131 insertions(+), 125 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 9e1b2c56df62..52b2487ef901 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -10,7 +10,6 @@ enum compact_priority { COMPACT_PRIO_SYNC_FULL, MIN_COMPACT_PRIORITY = COMPACT_PRIO_SYNC_FULL, COMPACT_PRIO_SYNC_LIGHT, - MIN_COMPACT_COSTLY_PRIORITY = COMPACT_PRIO_SYNC_LIGHT, DEF_COMPACT_PRIORITY = COMPACT_PRIO_SYNC_LIGHT, COMPACT_PRIO_ASYNC, INIT_COMPACT_PRIORITY = COMPACT_PRIO_ASYNC diff --git a/mm/compaction.c b/mm/compaction.c index 8080c04e644a..e33c99eb34a8 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1784,15 +1784,6 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) if (cc->order <= PAGE_ALLOC_COSTLY_ORDER) return pfn; - /* - * Only allow kcompactd and direct requests for movable pages to - * quickly clear out a MOVABLE pageblock for allocation. This - * reduces the risk that a large movable pageblock is freed for - * an unmovable/reclaimable small allocation. - */ - if (cc->direct_compaction && cc->migratetype != MIGRATE_MOVABLE) - return pfn; - /* * When starting the migration scanner, pick any pageblock within the * first half of the search space. Otherwise try and pick a pageblock @@ -2065,8 +2056,7 @@ static bool should_proactive_compact_node(pg_data_t *pgdat) static enum compact_result __compact_finished(struct compact_control *cc) { - unsigned int order; - const int migratetype = cc->migratetype; + unsigned long mark; int ret; /* Compaction run completes if the migrate and free scanner meet */ @@ -2120,39 +2110,16 @@ static enum compact_result __compact_finished(struct compact_control *cc) if (!pageblock_aligned(cc->migrate_pfn)) return COMPACT_CONTINUE; - /* Direct compactor: Is a suitable page free? */ + /* Done when watermarks are restored */ ret = COMPACT_NO_SUITABLE_PAGE; - for (order = cc->order; order < MAX_ORDER; order++) { - struct free_area *area = &cc->zone->free_area[order]; - bool can_steal; - - /* Job done if page is free of the right migratetype */ - if (!free_area_empty(area, migratetype)) - return COMPACT_SUCCESS; - -#ifdef CONFIG_CMA - /* MIGRATE_MOVABLE can fallback on MIGRATE_CMA */ - if (migratetype == MIGRATE_MOVABLE && - !free_area_empty(area, MIGRATE_CMA)) - return COMPACT_SUCCESS; -#endif - /* - * Job done if allocation would steal freepages from - * other migratetype buddy lists. - */ - if (find_suitable_fallback(area, order, migratetype, - true, &can_steal) != -1) - /* - * Movable pages are OK in any pageblock. If we are - * stealing for a non-movable allocation, make sure - * we finish compacting the current pageblock first - * (which is assured by the above migrate_pfn align - * check) so it is as free as possible and we won't - * have to steal another one soon. - */ - return COMPACT_SUCCESS; - } - + if (cc->direct_compaction) + mark = wmark_pages(cc->zone, + cc->alloc_flags & ALLOC_WMARK_MASK); + else + mark = high_wmark_pages(cc->zone); + if (zone_watermark_ok(cc->zone, cc->order, mark, + cc->highest_zoneidx, cc->alloc_flags)) + return COMPACT_SUCCESS; out: if (cc->contended || fatal_signal_pending(current)) ret = COMPACT_CONTENDED; @@ -2310,8 +2277,12 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) unsigned long watermark; /* Allocation can already succeed, nothing to do */ - watermark = wmark_pages(cc->zone, - cc->alloc_flags & ALLOC_WMARK_MASK); + if (cc->direct_compaction) + watermark = wmark_pages(cc->zone, + cc->alloc_flags & + ALLOC_WMARK_MASK); + else + watermark = high_wmark_pages(cc->zone); if (zone_watermark_ok(cc->zone, cc->order, watermark, cc->highest_zoneidx, cc->alloc_flags)) return COMPACT_SUCCESS; @@ -2800,7 +2771,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) /* Allocation can succeed in any zone, done */ if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, - min_wmark_pages(zone), + high_wmark_pages(zone), highest_zoneidx, 0)) return true; @@ -2845,7 +2816,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) /* Allocation can already succeed, nothing to do */ if (zone_watermark_ok(zone, cc.order, - min_wmark_pages(zone), zoneid, 0)) + high_wmark_pages(zone), zoneid, 0)) continue; if (compaction_suitable(zone, cc.order, diff --git a/mm/internal.h b/mm/internal.h index 39f65a463631..5c76455f8042 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -432,6 +432,7 @@ struct compact_control { */ struct capture_control { struct compact_control *cc; + int order; int migratetype; struct page *page; }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 18fa2bbba44b..6f0bfc226c36 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1075,7 +1075,7 @@ static inline bool compaction_capture(struct zone *zone, struct page *page, int order, int migratetype, struct capture_control *capc) { - if (!capc || order < capc->cc->order) + if (!capc || order < capc->order) return false; /* Do not accidentally pollute CMA or isolated regions*/ @@ -1097,8 +1097,8 @@ compaction_capture(struct zone *zone, struct page *page, int order, return false; } - if (order > capc->cc->order) - expand(zone, page, capc->cc->order, order, migratetype); + if (order > capc->order) + expand(zone, page, capc->order, order, migratetype); capc->page = page; return true; @@ -3649,15 +3649,15 @@ int __isolate_free_page(struct page *page, unsigned int order) int mt = get_pageblock_migratetype(page); if (!is_migrate_isolate(mt)) { + long free_pages = zone_page_state(zone, NR_FREE_PAGES); unsigned long watermark; /* - * Obey watermarks as if the page was being allocated. We can - * emulate a high-order watermark check with a raised order-0 - * watermark, because we already know our high-order page - * exists. + * Keep a lid on concurrent compaction. MIGRATE_FREE + * watermarks alone cannot be checked here, because + * that's what the caller is trying to produce. */ watermark = zone->_watermark[WMARK_MIN] + (1UL << order); - if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) + if (!__zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA, free_pages)) return 0; } @@ -3976,27 +3976,59 @@ noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) } ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); -static inline long __zone_watermark_unusable_free(struct zone *z, - unsigned int order, unsigned int alloc_flags) +static long page_state(struct zone *zone, enum zone_stat_item item, bool safe) { - const bool alloc_harder = (alloc_flags & (ALLOC_HARDER|ALLOC_OOM)); - long unusable_free = (1 << order) - 1; + if (safe) + return zone_page_state_snapshot(zone, item); + else + return zone_page_state(zone, item); +} + +static long __zone_free_pages(struct zone *zone, int alloc_flags, bool safe) +{ + long free_pages; /* - * If the caller does not have rights to ALLOC_HARDER then subtract - * the high-atomic reserves. This will over-estimate the size of the - * atomic reserve but it avoids a search. + * Enforce watermarks against MIGRATE_FREE pages. This ensures + * that there is always a reserve of higher-order pages + * maintained for all migratetypes and allocation contexts. + * + * Allocations will still use up any compatible free pages + * that may exist inside claimed blocks first. But the reserve + * prevents smaller allocations from starving out higher-order + * requests (which may not be able to sleep, e.g. highatomic). + * + * The additional memory requirements of this are mininmal. If + * internal free pages already exceed the compact_gap(), only + * compaction is necessary to restore the watermarks. */ - if (likely(!alloc_harder)) - unusable_free += z->nr_reserved_highatomic; + free_pages = page_state(zone, NR_FREE_FREE, safe); + if (alloc_flags & (ALLOC_HARDER | ALLOC_OOM)) + free_pages += page_state(zone, NR_FREE_HIGHATOMIC, safe); + if (IS_ENABLED(CONFIG_CMA) && (alloc_flags & ALLOC_CMA)) + free_pages += page_state(zone, NR_FREE_CMA_PAGES, safe); -#ifdef CONFIG_CMA - /* If allocation can't use CMA areas don't use free CMA pages */ - if (!(alloc_flags & ALLOC_CMA)) - unusable_free += zone_page_state(z, NR_FREE_CMA_PAGES); -#endif + if (!IS_ENABLED(CONFIG_COMPACTION)) { + /* + * We can't reasonably defragment without compaction. + * Consider everything and do best-effort grouping. + */ + free_pages += page_state(zone, NR_FREE_UNMOVABLE, safe); + free_pages += page_state(zone, NR_FREE_MOVABLE, safe); + free_pages += page_state(zone, NR_FREE_RECLAIMABLE, safe); + } - return unusable_free; + return free_pages; +} + +static long zone_free_pages(struct zone *zone, int alloc_flags) +{ + return __zone_free_pages(zone, alloc_flags, false); +} + +static long zone_free_pages_safe(struct zone *zone, int alloc_flags) +{ + return __zone_free_pages(zone, alloc_flags, true); } /* @@ -4014,7 +4046,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, const bool alloc_harder = (alloc_flags & (ALLOC_HARDER|ALLOC_OOM)); /* free_pages may go negative - that's OK */ - free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); + free_pages -= (1 << order) - 1; if (alloc_flags & ALLOC_HIGH) min -= min / 2; @@ -4076,33 +4108,22 @@ bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, int highest_zoneidx, unsigned int alloc_flags) { return __zone_watermark_ok(z, order, mark, highest_zoneidx, alloc_flags, - zone_page_state(z, NR_FREE_PAGES)); + zone_free_pages(z, alloc_flags)); } static inline bool zone_watermark_fast(struct zone *z, unsigned int order, unsigned long mark, int highest_zoneidx, unsigned int alloc_flags, gfp_t gfp_mask) { - long free_pages; - - free_pages = zone_page_state(z, NR_FREE_PAGES); + long free_pages = zone_free_pages(z, alloc_flags); /* * Fast check for order-0 only. If this fails then the reserves * need to be calculated. */ - if (!order) { - long usable_free; - long reserved; - - usable_free = free_pages; - reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); - - /* reserved may over estimate high-atomic reserves. */ - usable_free -= min(usable_free, reserved); - if (usable_free > mark + z->lowmem_reserve[highest_zoneidx]) - return true; - } + if (!order && (free_pages - ((1 << order) - 1) > + mark + z->lowmem_reserve[highest_zoneidx])) + return true; if (__zone_watermark_ok(z, order, mark, highest_zoneidx, alloc_flags, free_pages)) @@ -4126,13 +4147,8 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, bool zone_watermark_ok_safe(struct zone *z, unsigned int order, unsigned long mark, int highest_zoneidx) { - long free_pages = zone_page_state(z, NR_FREE_PAGES); - - if (z->percpu_drift_mark && free_pages < z->percpu_drift_mark) - free_pages = zone_page_state_snapshot(z, NR_FREE_PAGES); - return __zone_watermark_ok(z, order, mark, highest_zoneidx, 0, - free_pages); + zone_free_pages_safe(z, 0)); } #ifdef CONFIG_NUMA @@ -4524,12 +4540,14 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, unsigned long pflags; unsigned int noreclaim_flag; struct capture_control capc = { + .order = order, .migratetype = ac->migratetype, .page = NULL, }; + int compact_order; - if (!order) - return NULL; + /* Use reclaim/compaction to produce neutral blocks */ + compact_order = max_t(int, order, pageblock_order); /* * Make sure the structs are really initialized before we expose the @@ -4543,8 +4561,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, delayacct_compact_start(); noreclaim_flag = memalloc_noreclaim_save(); - *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &capc); + *compact_result = try_to_compact_pages(gfp_mask, compact_order, + alloc_flags, ac, prio, &capc); memalloc_noreclaim_restore(noreclaim_flag); psi_memstall_leave(&pflags); @@ -4608,13 +4626,12 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, enum compact_priority *compact_priority, int *compaction_retries) { - int min_priority; bool ret = false; int retries = *compaction_retries; enum compact_priority priority = *compact_priority; - if (!order) - return false; + /* Use reclaim/compaction to produce neutral blocks */ + order = max_t(int, order, pageblock_order); if (fatal_signal_pending(current)) return false; @@ -4624,20 +4641,6 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, * failed, presumably due to a race. Retry a few times. */ if (compact_result == COMPACT_SUCCESS) { - int max_retries = MAX_COMPACT_RETRIES; - - /* - * !costly requests are much more important than - * __GFP_RETRY_MAYFAIL costly ones because they are de - * facto nofail and invoke OOM killer to move on while - * costly can fail and users are ready to cope with - * that. 1/4 retries is rather arbitrary but we would - * need much more detailed feedback from compaction to - * make a better decision. - */ - if (order > PAGE_ALLOC_COSTLY_ORDER) - max_retries /= 4; - ret = ++(*compaction_retries) <= MAX_COMPACT_RETRIES; goto out; } @@ -4654,16 +4657,13 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, /* * Compaction failed. Retry with increasing priority. */ - min_priority = (order > PAGE_ALLOC_COSTLY_ORDER) ? - MIN_COMPACT_COSTLY_PRIORITY : MIN_COMPACT_PRIORITY; - - if (*compact_priority > min_priority) { + if (*compact_priority > MIN_COMPACT_PRIORITY) { (*compact_priority)--; *compaction_retries = 0; ret = true; } out: - trace_compact_retry(order, priority, compact_result, retries, max_retries, ret); + trace_compact_retry(order, priority, compact_result, retries, MAX_COMPACT_RETRIES, ret); return ret; } #else @@ -4822,9 +4822,16 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; bool drained = false; + int reclaim_order; + + /* Use reclaim/compaction to produce neutral blocks */ + if (IS_ENABLED(CONFIG_COMPACTION)) + reclaim_order = max_t(int, order, pageblock_order); + else + reclaim_order = order; psi_memstall_enter(&pflags); - *did_some_progress = __perform_reclaim(gfp_mask, order, ac); + *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac); if (unlikely(!(*did_some_progress))) goto out; @@ -4856,6 +4863,10 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, pg_data_t *last_pgdat = NULL; enum zone_type highest_zoneidx = ac->highest_zoneidx; + /* Use reclaim/compaction to produce neutral blocks */ + if (IS_ENABLED(CONFIG_COMPACTION)) + order = max_t(unsigned int, order, pageblock_order); + for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, highest_zoneidx, ac->nodemask) { if (!managed_zone(zone)) @@ -4970,6 +4981,24 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, struct zoneref *z; bool ret = false; + /* + * In the old world, order-0 pages only need reclaim, and + * higher orders might be present but the order-0 watermarks + * aren't met yet. These things can be fixed by reclaim alone. + * + * In the new world, though, watermark checks are against + * MIGRATE_FREE blocks. That means if the watermarks aren't + * met, reclaim isn't going to be the solution. Neither for + * order-0 nor for anything else. Whether it makes sense to + * retry depends fully on whether compaction should retry. + * + * should_compact_retry() already checks for COMPACT_SKIPPED + * and compaction_zonelist_suitable() to test whether reclaim + * is needed. + */ + if (IS_ENABLED(CONFIG_COMPACTION)) + goto schedule; + /* * Costly allocations might have made a progress but this doesn't mean * their order will become available due to high fragmentation so @@ -5019,6 +5048,7 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, } } +schedule: /* * Memory allocation/reclaim might be called from a WQ context and the * current implementation of the WQ concurrency control doesn't @@ -8833,6 +8863,13 @@ static void __setup_per_zone_wmarks(void) mult_frac(zone_managed_pages(zone), watermark_scale_factor, 10000)); + /* + * Ensure the watermark delta is a multiple of the + * neutral block that reclaim/compaction produces. + */ + if (IS_ENABLED(CONFIG_COMPACTION)) + tmp = ALIGN(tmp, 1 << pageblock_order); + zone->watermark_boost = 0; zone->_watermark[WMARK_LOW] = min_wmark_pages(zone) + tmp; zone->_watermark[WMARK_HIGH] = low_wmark_pages(zone) + tmp; diff --git a/mm/vmscan.c b/mm/vmscan.c index 14d6116384cc..a7374cd6fe91 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -7438,8 +7438,7 @@ void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order, /* Hopeless node, leave it to direct reclaim if possible */ if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES || - (pgdat_balanced(pgdat, order, highest_zoneidx) && - !pgdat_watermark_boosted(pgdat, highest_zoneidx))) { + pgdat_balanced(pgdat, order, highest_zoneidx)) { /* * There may be plenty of free memory available, but it's too * fragmented for high-order allocations. Wake up kcompactd @@ -7447,8 +7446,7 @@ void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order, * needed. If it fails, it will defer subsequent attempts to * ratelimit its work. */ - if (!(gfp_flags & __GFP_DIRECT_RECLAIM)) - wakeup_kcompactd(pgdat, order, highest_zoneidx); + wakeup_kcompactd(pgdat, order, highest_zoneidx); return; } From patchwork Tue Apr 18 19:13:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6897C77B75 for ; Tue, 18 Apr 2023 19:14:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D9F790000C; Tue, 18 Apr 2023 15:13:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58998900006; Tue, 18 Apr 2023 15:13:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452A690000C; Tue, 18 Apr 2023 15:13:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 33547900006 for ; Tue, 18 Apr 2023 15:13:49 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 13BDC80430 for ; Tue, 18 Apr 2023 19:13:49 +0000 (UTC) X-FDA: 80695461378.23.9038B94 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf21.hostedemail.com (Postfix) with ESMTP id 399641C0003 for ; Tue, 18 Apr 2023 19:13:47 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="wmmelD/e"; spf=pass (imf21.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.54 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+cRr9qK9Ai5PrhDW9GBX61DMQfs8tenll5tn8q6xhSs=; b=Lf6w4oW2Xw9TDdOvwUVPjtcYImkTzQo4fUhd6LS6h0WBYXsgaUjdVvEewJMeXvXY9eLt4j np5lomnXz5YuadZEs69e9JCg5le5zt0DqgaD5LCYvP3C9UTKa4zafT8ttBBp8xWvbYiGlb qTAXp8JHBNBaj7QhmS+SDKMHbsi5K6o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="wmmelD/e"; spf=pass (imf21.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.54 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845227; a=rsa-sha256; cv=none; b=2aU85RpgJPPpoEjapdFqm+q13V2Lui6eX96S32FPzzxvuHWrX5XmquYNNMXruGhJb26MWo tVsFhfkhKsil3rAE0RwwSdwszdW5xtW6EnNFmCVtd8Ikm+Y7TkJaTohKxzBpE5eyrjO3hf 3Y1GjTPyiNAHATQtINLPHONGokMic8o= Received: by mail-qv1-f54.google.com with SMTP id oo30so15745189qvb.12 for ; Tue, 18 Apr 2023 12:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845226; x=1684437226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+cRr9qK9Ai5PrhDW9GBX61DMQfs8tenll5tn8q6xhSs=; b=wmmelD/et9ZFdJKWhHdBu3DZw0uP+uxtuqLIKxl7QWQ1gLIf0x4Dl4qBY5A5PexoJr OhMqGiuUdTE3V0nKLvSyHZAoIjUGav09Njn4JLlk8e/qdWSaEO8AbhxXtKfhN9YvxCXg l0z11YLxPSDIF42CiCn8RarJvNYgRQLIQ2zqvd+pz+X9iu9cWQQc/1+jMyxJFIG9sCWT OLKvnFIUidWTWQ+1Jb0/Pu2fD7pUcAOpyiK/lAooYzOXHFLzWNIhXh6zDBh53/UdDun2 oq1Tk9nVl3BXreuO2iAOhtOPWohJqF8SUU3eIRYB0cpXOnSCLJoCqIqVgvyWWmUEwzmF gGiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845226; x=1684437226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+cRr9qK9Ai5PrhDW9GBX61DMQfs8tenll5tn8q6xhSs=; b=Pn8YlhtZ389RUZ3hKpwzgjd13unaplrPhJ4jv1A/yZRcGT/EOxW9AqsfHLQNFmCqFP B/c+8bUx0mF8CA1f8mKbasE32RIdlEHJcOMq9opJ40ndUW9tsRWNZrTsWRzXQ4mKYhzr W24zruh6U5fD7M1j5CRP9+nl6JsvMHV8xNsdr1+bOLh0Uir8jpBUgtM0/EharGgt5kTW KQO981LrXYeO+APgit2wDm+kvLDhWpYflNO9m2papP9MbRi7adj/t+fvvEhCeb7N81Hu U3IEeFcyX70Bq8R5vtvtzGl2iK3oF2jJ9qmGUsoCoflL4mdnPjVYC/4jQofGW+hL7vF2 uSKQ== X-Gm-Message-State: AAQBX9ekPlCp/WlJGOwcLmdAuP1yye4WKz8nAlcf2rbuTeNhKlrPWIES v9PdqQzRS1SLkf0kiE2wgN+dRB+PQGKgDEfPTyA= X-Google-Smtp-Source: AKy350bbq+tUiFNnGALyl/LWu3vvTMgXSZpDDXIn0FD09vHJTVesFePd2LgKVoVwrtg0UDf72P1fJA== X-Received: by 2002:a05:6214:21c8:b0:5f1:5f73:aed4 with SMTP id d8-20020a05621421c800b005f15f73aed4mr4057561qvh.15.1681845225932; Tue, 18 Apr 2023 12:13:45 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id lu8-20020a0562145a0800b005eab96abc9esm3844744qvb.140.2023.04.18.12.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:45 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 23/26] mm: page_alloc: kill highatomic Date: Tue, 18 Apr 2023 15:13:10 -0400 Message-Id: <20230418191313.268131-24-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Stat-Signature: 75ahw16u3np4ho71mt3sdggrt9segzoy X-Rspam-User: X-Rspamd-Queue-Id: 399641C0003 X-Rspamd-Server: rspam06 X-HE-Tag: 1681845227-775130 X-HE-Meta: U2FsdGVkX1+NMhm/HjZ+vT/8rE7vvz6lqRuWf+MTRrCtLg69nNsmF6IGEkPop60U2+KLdY7QSWbb6J5FahFGxzKR3dKDy9iEccOAcIkCXr48miIvnPkQgVBkfULD0mIi7ziMAZaM4j8QJzio9aOj0/zvswXnG5CQUuMG9At5nZquC1KAhiljirXWo1LC+VGOE/w6/qblTHDcn0aO+jQE+59x9i0+QRSghtN71OeL8reMOj47mwmqEiOigMRDlZag1qv1ch4/jeNOa6PuJhJQG5Z36ZiVvOzbaVn3F1AcJ+T20DIafCf1bAUavLSwbjID5pdzjWIJdp/4Crw4phzFhcoh5M4R3e/LVNsaTOhbe+cP8iRgY0bbAHQlG1MqV0vuu9GYqr7TyIv+5yRYbVK/AtzpfDWeknfM2zJtwG23XV6XoQGnBr75XuAlzEd0lB3bn3eiicJjGfqV60pfwXZL0fjXt4rAvpL3/8QYmEHaoWEYOVAIjOzmEOEZ311L4tvD2g6pkCufyXlbCw24Wq5t1p6oFz7v9aVtC9EuPEfDtY0L98JVtVwAN9i6j79jvpwlpjMPDqFTMS8paXnkCqAe6wgX5hxoNR1/uNMHaah+pyqlCVB244DKfCfpIqlVpF9AXoIMvLxdybtOEcRGYUiCn5vGk3GvTrvNomzT5+gnwVNx8UUwU8A/YI2kubrZDK5utopWfnb2/PLXTZXz72VsSd33eKKh3nrsG+biCXsEBvPhTjx2IF6/B9Gdq9kKfABcLacsXfB2kS/uaXNW2+XP8jFAjTJNS4kPV/sHDjLCkZUl9x6tq48ptFbeXh9sfjd0FqhCsuqXqiNwuV56UwRq1LwKV2+HGdofqNWunR6dx8ZPZAmK4HyEG9CPOWHS8iLrJvT4BZieY3pgOMWp2zInX/kbF8Ifj5rwDHXB1xU8tX1A3TBCLoxiAuHSKeImuAJSsO6bs/JvphgviDGnr4i ZbQqWVu0 MRzHuMIMPViFwsTZXzsdrUwtlODpCztIMIa71s54olyjdBmf4HxduCVXjVlHSqw9hCSTWjQGrd9J9n3qyQrDW8Z2UqpVZivec2MxOMLGvo/5nBZSzRqRHSP3AF6IymfttEUWn7CuOWiywYU2K0F1BWMQ9TPJNn0FebbeEybBw492+FEuAa7VRTCe8fsUs1B2RefGwCx8mkfHODMTtNj3FwqkZj6T8GogNKiYlGy1l8Bm8s/Bbzinqk0JDUUm2jn5AYnAMDP/fgaVDZCIb3qEcY4XRVJBXto3PWZ8+/D9s3/h6MeC1lpAlXGuAL5qFkpDQjjUbu0zZTuTyv9Qe3Eq+wIVp/temc2WyPoXhLQrWBjyoUNHZuIYuNAbQ1js56vutrvR+jjiRB9yHhZxKQZE+Z2jn1iQVzAzOWq61CuYhUbrCrjRSOaU0jk//RTP6tZYLbYUuucW7+zArvB3G1A9ZwIOcl3XjIkZSAsXusWFaUQubQ5avIfHMNF4YfyyVOACmzs9oc5nUhe38EbsVm7xjqNbRAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The highatomic reserves are blocks set aside specifically for higher-order atomic allocations. Since watermarks are now required to be met in pageblocks, this is no longer necessary. Signed-off-by: Johannes Weiner --- include/linux/gfp.h | 2 - include/linux/mmzone.h | 6 +- mm/internal.h | 5 -- mm/page_alloc.c | 187 ++--------------------------------------- mm/vmstat.c | 1 - 5 files changed, 10 insertions(+), 191 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..78b5176d354e 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -19,8 +19,6 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) != ___GFP_MOVABLE); BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_MOVABLE); BUILD_BUG_ON((___GFP_RECLAIMABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_RECLAIMABLE); - BUILD_BUG_ON(((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> - GFP_MOVABLE_SHIFT) != MIGRATE_HIGHATOMIC); if (unlikely(page_group_by_mobility_disabled)) return MIGRATE_UNMOVABLE; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d1083ab81998..c705f2f7c829 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -44,8 +44,7 @@ enum migratetype { MIGRATE_MOVABLE, MIGRATE_RECLAIMABLE, MIGRATE_PCPTYPES, /* the number of types on the pcp lists */ - MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES, - MIGRATE_FREE, + MIGRATE_FREE = MIGRATE_PCPTYPES, #ifdef CONFIG_CMA /* * MIGRATE_CMA migration type is designed to mimic the way @@ -142,7 +141,6 @@ enum zone_stat_item { NR_FREE_UNMOVABLE, NR_FREE_MOVABLE, NR_FREE_RECLAIMABLE, - NR_FREE_HIGHATOMIC, NR_FREE_FREE, NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */ NR_ZONE_INACTIVE_ANON = NR_ZONE_LRU_BASE, @@ -713,8 +711,6 @@ struct zone { unsigned long _watermark[NR_WMARK]; unsigned long watermark_boost; - unsigned long nr_reserved_highatomic; - /* * We don't know if the memory that we're going to allocate will be * freeable or/and it will be released eventually, so to avoid totally diff --git a/mm/internal.h b/mm/internal.h index 5c76455f8042..24f43f5db88b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -778,11 +778,6 @@ extern const struct trace_print_flags pageflag_names[]; extern const struct trace_print_flags vmaflag_names[]; extern const struct trace_print_flags gfpflag_names[]; -static inline bool is_migrate_highatomic(enum migratetype migratetype) -{ - return migratetype == MIGRATE_HIGHATOMIC; -} - void setup_zone_pageset(struct zone *zone); struct migration_target_control { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6f0bfc226c36..e8ae04feb1bd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -379,7 +379,6 @@ const char * const migratetype_names[MIGRATE_TYPES] = { "Unmovable", "Movable", "Reclaimable", - "HighAtomic", "Free", #ifdef CONFIG_CMA "CMA", @@ -1202,7 +1201,7 @@ static inline void __free_one_page(struct page *page, * We want to prevent merge between freepages on pageblock * without fallbacks and normal pageblock. Without this, * pageblock isolation could cause incorrect freepage or CMA - * accounting or HIGHATOMIC accounting. + * accounting. */ if (migratetype != buddy_mt && (!migratetype_is_mergeable(migratetype) || @@ -2797,13 +2796,6 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, old_block_type = get_pageblock_migratetype(page); - /* - * This can happen due to races and we want to prevent broken - * highatomic accounting. - */ - if (is_migrate_highatomic(old_block_type)) - goto single_page; - /* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) { change_pageblock_range(page, current_order, start_type); @@ -2918,126 +2910,6 @@ int find_suitable_fallback(struct free_area *area, unsigned int order, return -1; } -/* - * Reserve a pageblock for exclusive use of high-order atomic allocations if - * there are no empty page blocks that contain a page with a suitable order - */ -static void reserve_highatomic_pageblock(struct page *page, struct zone *zone, - unsigned int alloc_order) -{ - int mt; - unsigned long max_managed, flags; - - /* - * Limit the number reserved to 1 pageblock or roughly 1% of a zone. - * Check is race-prone but harmless. - */ - max_managed = (zone_managed_pages(zone) / 100) + pageblock_nr_pages; - if (zone->nr_reserved_highatomic >= max_managed) - return; - - spin_lock_irqsave(&zone->lock, flags); - - /* Recheck the nr_reserved_highatomic limit under the lock */ - if (zone->nr_reserved_highatomic >= max_managed) - goto out_unlock; - - /* Yoink! */ - mt = get_pageblock_migratetype(page); - /* Only reserve normal pageblocks (i.e., they can merge with others) */ - if (migratetype_is_mergeable(mt)) { - zone->nr_reserved_highatomic += pageblock_nr_pages; - set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); - move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC, NULL); - } - -out_unlock: - spin_unlock_irqrestore(&zone->lock, flags); -} - -/* - * Used when an allocation is about to fail under memory pressure. This - * potentially hurts the reliability of high-order allocations when under - * intense memory pressure but failed atomic allocations should be easier - * to recover from than an OOM. - * - * If @force is true, try to unreserve a pageblock even though highatomic - * pageblock is exhausted. - */ -static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, - bool force) -{ - struct zonelist *zonelist = ac->zonelist; - unsigned long flags; - struct zoneref *z; - struct zone *zone; - struct page *page; - int order; - bool ret; - - for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx, - ac->nodemask) { - /* - * Preserve at least one pageblock unless memory pressure - * is really high. - */ - if (!force && zone->nr_reserved_highatomic <= - pageblock_nr_pages) - continue; - - spin_lock_irqsave(&zone->lock, flags); - for (order = 0; order < MAX_ORDER; order++) { - struct free_area *area = &(zone->free_area[order]); - int mt; - - page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); - if (!page) - continue; - - mt = get_pageblock_migratetype(page); - /* - * In page freeing path, migratetype change is racy so - * we can counter several free pages in a pageblock - * in this loop although we changed the pageblock type - * from highatomic to ac->migratetype. So we should - * adjust the count once. - */ - if (is_migrate_highatomic(mt)) { - /* - * It should never happen but changes to - * locking could inadvertently allow a per-cpu - * drain to add pages to MIGRATE_HIGHATOMIC - * while unreserving so be safe and watch for - * underflows. - */ - zone->nr_reserved_highatomic -= min( - pageblock_nr_pages, - zone->nr_reserved_highatomic); - } - - /* - * Convert to ac->migratetype and avoid the normal - * pageblock stealing heuristics. Minimally, the caller - * is doing the work and needs the pages. More - * importantly, if the block was always converted to - * MIGRATE_UNMOVABLE or another type then the number - * of pageblocks that cannot be completely freed - * may increase. - */ - set_pageblock_migratetype(page, ac->migratetype); - ret = move_freepages_block(zone, page, mt, - ac->migratetype, NULL); - if (ret) { - spin_unlock_irqrestore(&zone->lock, flags); - return ret; - } - } - spin_unlock_irqrestore(&zone->lock, flags); - } - - return false; -} - /* * Try finding a free buddy page on the fallback list and put it on the free * list of requested migratetype, possibly along with other pages from the same @@ -3510,18 +3382,11 @@ void free_unref_page(struct page *page, unsigned int order) /* * We only track unmovable, reclaimable and movable on pcp lists. - * Place ISOLATE pages on the isolated list because they are being - * offlined but treat HIGHATOMIC as movable pages so we can get those - * areas back if necessary. Otherwise, we may have to free - * excessively into the page allocator */ migratetype = get_pcppage_migratetype(page); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { - if (unlikely(is_migrate_isolate(migratetype) || migratetype == MIGRATE_FREE)) { - free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); - return; - } - migratetype = MIGRATE_MOVABLE; + free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); + return; } zone = page_zone(page); @@ -3740,24 +3605,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned long flags; do { - page = NULL; spin_lock_irqsave(&zone->lock, flags); - /* - * order-0 request can reach here when the pcplist is skipped - * due to non-CMA allocation context. HIGHATOMIC area is - * reserved for high-order atomic allocation, so order-0 - * request should skip it. - */ - if (order > 0 && alloc_flags & ALLOC_HARDER) - page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); - if (!page) { - page = __rmqueue(zone, order, migratetype, alloc_flags); - if (!page) { - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; - } - } + page = __rmqueue(zone, order, migratetype, alloc_flags); spin_unlock_irqrestore(&zone->lock, flags); + if (!page) + return NULL; } while (check_new_pages(page, order)); __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -4003,8 +3855,6 @@ static long __zone_free_pages(struct zone *zone, int alloc_flags, bool safe) * compaction is necessary to restore the watermarks. */ free_pages = page_state(zone, NR_FREE_FREE, safe); - if (alloc_flags & (ALLOC_HARDER | ALLOC_OOM)) - free_pages += page_state(zone, NR_FREE_HIGHATOMIC, safe); if (IS_ENABLED(CONFIG_CMA) && (alloc_flags & ALLOC_CMA)) free_pages += page_state(zone, NR_FREE_CMA_PAGES, safe); @@ -4098,8 +3948,6 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, return true; } #endif - if (alloc_harder && !free_area_empty(area, MIGRATE_HIGHATOMIC)) - return true; } return false; } @@ -4340,14 +4188,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask, alloc_flags, ac->migratetype); if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); - - /* - * If this is a high-order atomic allocation then check - * if the pageblock should be reserved for the future - */ - if (unlikely(order && (alloc_flags & ALLOC_HARDER))) - reserve_highatomic_pageblock(page, zone, order); - return page; } else { #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -4844,7 +4684,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, * Shrink them and try again */ if (!page && !drained) { - unreserve_highatomic_pageblock(ac, false); drain_all_pages(NULL); drained = true; goto retry; @@ -5013,10 +4852,8 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, * Make sure we converge to OOM if we cannot make any progress * several times in the row. */ - if (*no_progress_loops > MAX_RECLAIM_RETRIES) { - /* Before OOM, exhaust highatomic_reserve */ - return unreserve_highatomic_pageblock(ac, true); - } + if (*no_progress_loops > MAX_RECLAIM_RETRIES) + return false; /* * Keep reclaiming pages while there is a chance this will lead @@ -6129,7 +5966,6 @@ static void show_migration_types(unsigned char type) [MIGRATE_UNMOVABLE] = 'U', [MIGRATE_MOVABLE] = 'M', [MIGRATE_RECLAIMABLE] = 'E', - [MIGRATE_HIGHATOMIC] = 'H', [MIGRATE_FREE] = 'F', #ifdef CONFIG_CMA [MIGRATE_CMA] = 'C', @@ -6194,7 +6030,7 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i " sec_pagetables:%lu bounce:%lu\n" " kernel_misc_reclaimable:%lu\n" " free:%lu free_unmovable:%lu free_movable:%lu\n" - " free_reclaimable:%lu free_highatomic:%lu free_free:%lu\n" + " free_reclaimable:%lu free_free:%lu\n" " free_cma:%lu free_pcp:%lu\n", global_node_page_state(NR_ACTIVE_ANON), global_node_page_state(NR_INACTIVE_ANON), @@ -6217,7 +6053,6 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i global_zone_page_state(NR_FREE_UNMOVABLE), global_zone_page_state(NR_FREE_MOVABLE), global_zone_page_state(NR_FREE_RECLAIMABLE), - global_zone_page_state(NR_FREE_HIGHATOMIC), global_zone_page_state(NR_FREE_FREE), global_zone_page_state(NR_FREE_CMA_PAGES), free_pcp); @@ -6301,13 +6136,11 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i " free_unmovable:%lukB" " free_movable:%lukB" " free_reclaimable:%lukB" - " free_highatomic:%lukB" " free_free:%lukB" " boost:%lukB" " min:%lukB" " low:%lukB" " high:%lukB" - " reserved_highatomic:%luKB" " active_anon:%lukB" " inactive_anon:%lukB" " active_file:%lukB" @@ -6327,13 +6160,11 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i K(zone_page_state(zone, NR_FREE_UNMOVABLE)), K(zone_page_state(zone, NR_FREE_MOVABLE)), K(zone_page_state(zone, NR_FREE_RECLAIMABLE)), - K(zone_page_state(zone, NR_FREE_HIGHATOMIC)), K(zone_page_state(zone, NR_FREE_FREE)), K(zone->watermark_boost), K(min_wmark_pages(zone)), K(low_wmark_pages(zone)), K(high_wmark_pages(zone)), - K(zone->nr_reserved_highatomic), K(zone_page_state(zone, NR_ZONE_ACTIVE_ANON)), K(zone_page_state(zone, NR_ZONE_INACTIVE_ANON)), K(zone_page_state(zone, NR_ZONE_ACTIVE_FILE)), diff --git a/mm/vmstat.c b/mm/vmstat.c index c8b8e6e259da..a2f7b41564df 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1171,7 +1171,6 @@ const char * const vmstat_text[] = { "nr_free_unmovable", "nr_free_movable", "nr_free_reclaimable", - "nr_free_highatomic", "nr_free_free", "nr_zone_inactive_anon", "nr_zone_active_anon", From patchwork Tue Apr 18 19:13:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3488EC6FD18 for ; Tue, 18 Apr 2023 19:14:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 985CD900006; Tue, 18 Apr 2023 15:13:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9332D280001; Tue, 18 Apr 2023 15:13:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ACB790000D; Tue, 18 Apr 2023 15:13:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 66D7E900006 for ; Tue, 18 Apr 2023 15:13:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1E51D8042A for ; Tue, 18 Apr 2023 19:13:50 +0000 (UTC) X-FDA: 80695461420.18.5D10719 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 4A7C1A000F for ; Tue, 18 Apr 2023 19:13:48 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="a/1OWmR/"; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RcOWVPTnlhnbqqS83iZRHWvw9Z/GKVYeqba03KhzlhU=; b=h4C5O5k/8ZWPIlIqOToBxQ1JUygxYgJvM9s5m6b0EGxwNcvHYOmaVFQfYmgLhKPwWrpDnL IoyaeabmdeY775My6Mps/tGb+GKNjGR8htvxvpeS23v7DjVhh57L9PM6i3xNWH31Zq0f52 Bqnynda8AcUjiL+OUSPj8EUS6iSe8bY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b="a/1OWmR/"; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845228; a=rsa-sha256; cv=none; b=CS1UU53j4oLPS0Hb3/CSupYQxCQ0bJ5DFwyrpvCXa1RZks6YbMb08mwt9ih2omUy8nbg1l GpuHP6etE7HwjQbxEwkHc/UlDcruRN0fBLxNLF19ninpe9NEFpcjTFs9u/xrVQi9qFUHIO 0E6oF57/HhZbfZH2wY7zJTz8ecJ3RiQ= Received: by mail-qt1-f173.google.com with SMTP id ey8so6190180qtb.3 for ; Tue, 18 Apr 2023 12:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845227; x=1684437227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RcOWVPTnlhnbqqS83iZRHWvw9Z/GKVYeqba03KhzlhU=; b=a/1OWmR/tUcWE2Qqj1o11ZW4UdowGBE3dveFmCmMXs2fzxld+GikzPbYsZnkzQUhO1 rszkmym7seH29zcw8ulr+qDK26L0PsIcj2vrordWVhv4mNV7cpWqWuY5SQHsZw8z7mao 4Z/ONsQuuhKfFYEQ6RM3cdDgMfZH06DbmKmGbAFpl8E2Vd0qVeQ9k1m0bs22PIP//Pjj y/MIXkHHsDsENio2rDHGldHoYX51Hv15DAGeCwg3O+QTHInV8nsPqRyC21NUdBPxr8kG g6o0GlmUWbBX6RAAFG5tpS7kEnZq+h52JP5HlzaOA46WT6u1sWU3NzgkrM0ulNQCm7Q9 Iynw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845227; x=1684437227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcOWVPTnlhnbqqS83iZRHWvw9Z/GKVYeqba03KhzlhU=; b=MIeVoVjDTd81x/dNponIvfuAei2w1Jhe4UvJEp86yAnfctQJOWlqtHerCUfCbJ/muv CnX+SP0TTNuWocgBs4S5H9xODJnAKhcosoCve6luGqoNVK9o5ppUTLR6Np8ubyg60lZr Gcl42DUB5lM5F/sAMJngTPM8TAzVxX5veYA0tJDPhcTvPLw/TKR7p8al7BTn36dv/wvs WQ8N+WvvrWcFRhtCqH1/7wLgKVcrDVxIhCcMJ4A+mqhnt2iqjmN4saL4HqgCh9Mp4bJP dLL3iAsSFT4NKndr9UuDzxJA1R1Q7W0VQbxtuz0kygsuSNQC3zbNqmnMmM8ZP3ry9uoz 1uJg== X-Gm-Message-State: AAQBX9eY6QtWyTpvh2u4IUeXam4j16paYFa9jWPEKP3xa6tOEy4idny0 Pgzb5VXKaxKfjow04+2G5U0/+jOSKO9PskPGijs= X-Google-Smtp-Source: AKy350aQbCigm5jEeHDFUL4OtIuD38+nxMH8T2pdMpNgffiiQpVWw2yOWVdgL1WnHxicR6IJsCf4vw== X-Received: by 2002:a05:622a:1996:b0:3e6:71d6:5d5d with SMTP id u22-20020a05622a199600b003e671d65d5dmr1822298qtc.1.1681845227071; Tue, 18 Apr 2023 12:13:47 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id g18-20020ac85812000000b003ef189ffa82sm2022876qtg.90.2023.04.18.12.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:46 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 24/26] mm: page_alloc: kill watermark boosting Date: Tue, 18 Apr 2023 15:13:11 -0400 Message-Id: <20230418191313.268131-25-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: df1objxypdtyaraom4e4dxwtrj9nqeja X-Rspamd-Queue-Id: 4A7C1A000F X-HE-Tag: 1681845228-680632 X-HE-Meta: U2FsdGVkX18GEUrH+gHTJbm6IuId78+SUkOOb6BFCHkE6eZdE//C2fjS3oeGeZCXKi15J/ztd1cucP6GloehRrdquqjL4r8x6EJWNrlFmQW0B1tn/VKlnd/ifs6+AW5dOlYRh9Xla2w1rTszjARYFxWZPFw3kvEz+VmPCEpl8N0PHOXHDVfYYaGlCf3MPtiWtBRCRZJpU+WVMoVUNTLxwzAdvlaGJUIm56fAs4CY4owq+zAshEax7j7hwenC//ugN1bTZhjddmGWgQaSEGHRDlkLu9GGCEGJbI7sY1xN0SMzu7SauzWclmWJj1ZFt3N8Ld+1XXEmB6iM1POiURYTjfu9bBZVHPZ9w/hTS0ViyGVyQTFrXw8yl/bb3O4rkoLAyCeIV4SIuC+KoEKQ63Ic1UalSaFfBChBP3BzMvtVuIsIQ+eTr5KmYp/7Sr6HCmJ+bDv46Hbd8epOCxlPocLo0iIe55b/LK/heSLx9qt5vrbRT9rvg5UCg3p6yOJPf8L0AzqxZyCBM0zqKaG/iVDbvUt975qX0eDK+nwxLA/8Hx5PBFzN59xhFQjtP1X9gE8voKFvVYjB0nDaEpl2xKBgSYFua4TpAp0lC2HDVgtIScxYjgYdfnxTK4UZkl4tYRK8bRdH9EJ0N2n/4mnt53DJ70qUPh83a8F+C53D1kkfvbCDkbVHs65Zg9hwAb0OVOiWrKjWENBzZncYy7Hf8wKO7BRa8CS7H9KSKwiYIXQBJfUmcAgyk5ZRNiubQWno6QyiT/DLkLwQuplPkQ7AvQz16zsl8y1xG2QpyLSy73QBuU1JY8P2Lp+XSgWAC/Exii+czMuofQDbYaAL7ZR4ohvIkgwPSRch/GrpM+wpEkgBQwhKq+uPkSHtX2fpc9KuwJgnWfN9yM5BJpWFF2zNXJMpcEuoc8zIr5nKIEvFvp24wb/of9ZJicY7NZfKNgwGlENNbCagDx3YN/0vQHKAAMW C/Bq8Nr4 kqvej1FsIy5cNB49q5gNqF0UVVQJnkvZE9dezkbgejWet3GxIZL3OfjW7Hf8fpA8HoM0028UgYoU8850o16/ruEzUZ0L3aMjaegA03wLpyzRXfjaB2Wovu+az9Hfs+TyyN95tGD+XdtjoEK3pxrEbsYJZeC6yj49Niut5w7kFnkuLneeqSjgpzFlvmnExrRZtkabV2exakvSVY6CT6N2wqdEE5t8ogmdbUtlUlGSy+pIGN9oaHZkDqz3e2Pem1izFqHNK/ZKo+NhLvJ1gSAPf/rhCGwzS9g471VBEzUnpvqAE9ZzMbIf+zVgmI7Tq7bqttZP5ZCC+b4SaPCFkwDNahHsK+HMqP7eApLJFBfzlo0KLLfaZTjM3Z2jiSiKVjgRduqI0glJQKoYg6NjNrS7TnLdTxigK3gtEe0cRmpxXFAGIGgW0ZstTZzV0fyTAQJB9rT+gQ+HS4JpqSrcnShX/YZs4crVM/AFsIJ74rxNp40qYg78+x6B6VdQncN9k8mQY0T7NPoem6dILMnpssNhXM8B98gqlnGg4cJo6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Watermark boosting is meant to increase the chances of pageblock production when fallbacks are observed. Since reclaim/compaction now produce neutral pageblocks per default, this is no longer needed. Signed-off-by: Johannes Weiner --- Documentation/admin-guide/sysctl/vm.rst | 21 ----- include/linux/mm.h | 1 - include/linux/mmzone.h | 12 +-- kernel/sysctl.c | 8 -- mm/page_alloc.c | 67 -------------- mm/vmscan.c | 111 +----------------------- mm/vmstat.c | 2 - 7 files changed, 7 insertions(+), 215 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 988f6a4c8084..498655c322bc 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -72,7 +72,6 @@ files can be found in mm/swap.c. - unprivileged_userfaultfd - user_reserve_kbytes - vfs_cache_pressure -- watermark_boost_factor - watermark_scale_factor - zone_reclaim_mode @@ -968,26 +967,6 @@ directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are. -watermark_boost_factor -====================== - -This factor controls the level of reclaim when memory is being fragmented. -It defines the percentage of the high watermark of a zone that will be -reclaimed if pages of different mobility are being mixed within pageblocks. -The intent is that compaction has less work to do in the future and to -increase the success rate of future high-order allocations such as SLUB -allocations, THP and hugetlbfs pages. - -To make it sensible with respect to the watermark_scale_factor -parameter, the unit is in fractions of 10,000. The default value of -15,000 means that up to 150% of the high watermark will be reclaimed in the -event of a pageblock being mixed due to fragmentation. The level of reclaim -is determined by the number of fragmentation events that occurred in the -recent past. If this value is smaller than a pageblock then a pageblocks -worth of pages will be reclaimed (e.g. 2MB on 64-bit x86). A boost factor -of 0 will disable the feature. - - watermark_scale_factor ====================== diff --git a/include/linux/mm.h b/include/linux/mm.h index f13f20258ce9..e7c2631848ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2746,7 +2746,6 @@ extern void setup_per_cpu_pageset(void); /* page_alloc.c */ extern int min_free_kbytes; -extern int watermark_boost_factor; extern int watermark_scale_factor; extern bool arch_has_descending_max_zone_pfns(void); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c705f2f7c829..1363ff6caff3 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -567,10 +567,10 @@ enum zone_watermarks { #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) #define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) -#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost) -#define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost) -#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) -#define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost) +#define min_wmark_pages(z) (z->_watermark[WMARK_MIN]) +#define low_wmark_pages(z) (z->_watermark[WMARK_LOW]) +#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH]) +#define wmark_pages(z, i) (z->_watermark[i]) /* Fields and list protected by pagesets local_lock in page_alloc.c */ struct per_cpu_pages { @@ -709,7 +709,6 @@ struct zone { /* zone watermarks, access with *_wmark_pages(zone) macros */ unsigned long _watermark[NR_WMARK]; - unsigned long watermark_boost; /* * We don't know if the memory that we're going to allocate will be @@ -884,9 +883,6 @@ enum pgdat_flags { }; enum zone_flags { - ZONE_BOOSTED_WATERMARK, /* zone recently boosted watermarks. - * Cleared when kswapd is woken. - */ ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */ }; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 137d4abe3eda..68bcd3a7c9c6 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -2229,14 +2229,6 @@ static struct ctl_table vm_table[] = { .proc_handler = min_free_kbytes_sysctl_handler, .extra1 = SYSCTL_ZERO, }, - { - .procname = "watermark_boost_factor", - .data = &watermark_boost_factor, - .maxlen = sizeof(watermark_boost_factor), - .mode = 0644, - .proc_handler = proc_dointvec_minmax, - .extra1 = SYSCTL_ZERO, - }, { .procname = "watermark_scale_factor", .data = &watermark_scale_factor, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e8ae04feb1bd..f835a5548164 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -401,7 +401,6 @@ compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS] = { int min_free_kbytes = 1024; int user_min_free_kbytes = -1; -int watermark_boost_factor __read_mostly = 15000; int watermark_scale_factor = 10; static unsigned long nr_kernel_pages __initdata; @@ -2742,43 +2741,6 @@ static bool can_steal_fallback(unsigned int order, int start_mt, return false; } -static inline bool boost_watermark(struct zone *zone) -{ - unsigned long max_boost; - - if (!watermark_boost_factor) - return false; - /* - * Don't bother in zones that are unlikely to produce results. - * On small machines, including kdump capture kernels running - * in a small area, boosting the watermark can cause an out of - * memory situation immediately. - */ - if ((pageblock_nr_pages * 4) > zone_managed_pages(zone)) - return false; - - max_boost = mult_frac(zone->_watermark[WMARK_HIGH], - watermark_boost_factor, 10000); - - /* - * high watermark may be uninitialised if fragmentation occurs - * very early in boot so do not boost. We do not fall - * through and boost by pageblock_nr_pages as failing - * allocations that early means that reclaim is not going - * to help and it may even be impossible to reclaim the - * boosted watermark resulting in a hang. - */ - if (!max_boost) - return false; - - max_boost = max(pageblock_nr_pages, max_boost); - - zone->watermark_boost = min(zone->watermark_boost + pageblock_nr_pages, - max_boost); - - return true; -} - /* * This function implements actual steal behaviour. If order is large enough, * we can steal whole pageblock. If not, we first move freepages in this @@ -2802,14 +2764,6 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, goto single_page; } - /* - * Boost watermarks to increase reclaim pressure to reduce the - * likelihood of future fallbacks. Wake kswapd now as the node - * may be balanced overall and kswapd will not wake naturally. - */ - if (boost_watermark(zone) && (alloc_flags & ALLOC_KSWAPD)) - set_bit(ZONE_BOOSTED_WATERMARK, &zone->flags); - /* We are not allowed to try stealing from the whole block */ if (!whole_block) goto single_page; @@ -3738,12 +3692,6 @@ struct page *rmqueue(struct zone *preferred_zone, migratetype); out: - /* Separate test+clear to avoid unnecessary atomics */ - if (unlikely(test_bit(ZONE_BOOSTED_WATERMARK, &zone->flags))) { - clear_bit(ZONE_BOOSTED_WATERMARK, &zone->flags); - wakeup_kswapd(zone, 0, 0, zone_idx(zone)); - } - VM_BUG_ON_PAGE(page && bad_range(zone, page), page); return page; } @@ -3976,18 +3924,6 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, if (__zone_watermark_ok(z, order, mark, highest_zoneidx, alloc_flags, free_pages)) return true; - /* - * Ignore watermark boosting for GFP_ATOMIC order-0 allocations - * when checking the min watermark. The min watermark is the - * point where boosting is ignored so that kswapd is woken up - * when below the low watermark. - */ - if (unlikely(!order && (gfp_mask & __GFP_ATOMIC) && z->watermark_boost - && ((alloc_flags & ALLOC_WMARK_MASK) == WMARK_MIN))) { - mark = z->_watermark[WMARK_MIN]; - return __zone_watermark_ok(z, order, mark, highest_zoneidx, - alloc_flags, free_pages); - } return false; } @@ -6137,7 +6073,6 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i " free_movable:%lukB" " free_reclaimable:%lukB" " free_free:%lukB" - " boost:%lukB" " min:%lukB" " low:%lukB" " high:%lukB" @@ -6161,7 +6096,6 @@ void __show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_i K(zone_page_state(zone, NR_FREE_MOVABLE)), K(zone_page_state(zone, NR_FREE_RECLAIMABLE)), K(zone_page_state(zone, NR_FREE_FREE)), - K(zone->watermark_boost), K(min_wmark_pages(zone)), K(low_wmark_pages(zone)), K(high_wmark_pages(zone)), @@ -8701,7 +8635,6 @@ static void __setup_per_zone_wmarks(void) if (IS_ENABLED(CONFIG_COMPACTION)) tmp = ALIGN(tmp, 1 << pageblock_order); - zone->watermark_boost = 0; zone->_watermark[WMARK_LOW] = min_wmark_pages(zone) + tmp; zone->_watermark[WMARK_HIGH] = low_wmark_pages(zone) + tmp; zone->_watermark[WMARK_PROMO] = high_wmark_pages(zone) + tmp; diff --git a/mm/vmscan.c b/mm/vmscan.c index a7374cd6fe91..5586be6997cd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6827,30 +6827,6 @@ static void kswapd_age_node(struct pglist_data *pgdat, struct scan_control *sc) } while (memcg); } -static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) -{ - int i; - struct zone *zone; - - /* - * Check for watermark boosts top-down as the higher zones - * are more likely to be boosted. Both watermarks and boosts - * should not be checked at the same time as reclaim would - * start prematurely when there is no boosting and a lower - * zone is balanced. - */ - for (i = highest_zoneidx; i >= 0; i--) { - zone = pgdat->node_zones + i; - if (!managed_zone(zone)) - continue; - - if (zone->watermark_boost) - return true; - } - - return false; -} - /* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx @@ -7025,14 +7001,13 @@ static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) unsigned long nr_soft_reclaimed; unsigned long nr_soft_scanned; unsigned long pflags; - unsigned long nr_boost_reclaim; - unsigned long zone_boosts[MAX_NR_ZONES] = { 0, }; - bool boosted; struct zone *zone; struct scan_control sc = { .gfp_mask = GFP_KERNEL, .order = order, .may_unmap = 1, + .may_swap = 1, + .may_writepage = !laptop_mode, }; set_task_reclaim_state(current, &sc.reclaim_state); @@ -7041,29 +7016,11 @@ static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) count_vm_event(PAGEOUTRUN); - /* - * Account for the reclaim boost. Note that the zone boost is left in - * place so that parallel allocations that are near the watermark will - * stall or direct reclaim until kswapd is finished. - */ - nr_boost_reclaim = 0; - for (i = 0; i <= highest_zoneidx; i++) { - zone = pgdat->node_zones + i; - if (!managed_zone(zone)) - continue; - - nr_boost_reclaim += zone->watermark_boost; - zone_boosts[i] = zone->watermark_boost; - } - boosted = nr_boost_reclaim; - -restart: set_reclaim_active(pgdat, highest_zoneidx); sc.priority = DEF_PRIORITY; do { unsigned long nr_reclaimed = sc.nr_reclaimed; bool raise_priority = true; - bool balanced; bool ret; sc.reclaim_idx = highest_zoneidx; @@ -7089,40 +7046,9 @@ static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) } } - /* - * If the pgdat is imbalanced then ignore boosting and preserve - * the watermarks for a later time and restart. Note that the - * zone watermarks will be still reset at the end of balancing - * on the grounds that the normal reclaim should be enough to - * re-evaluate if boosting is required when kswapd next wakes. - */ - balanced = pgdat_balanced(pgdat, sc.order, highest_zoneidx); - if (!balanced && nr_boost_reclaim) { - nr_boost_reclaim = 0; - goto restart; - } - - /* - * If boosting is not active then only reclaim if there are no - * eligible zones. Note that sc.reclaim_idx is not used as - * buffer_heads_over_limit may have adjusted it. - */ - if (!nr_boost_reclaim && balanced) + if (pgdat_balanced(pgdat, sc.order, highest_zoneidx)) goto out; - /* Limit the priority of boosting to avoid reclaim writeback */ - if (nr_boost_reclaim && sc.priority == DEF_PRIORITY - 2) - raise_priority = false; - - /* - * Do not writeback or swap pages for boosted reclaim. The - * intent is to relieve pressure not issue sub-optimal IO - * from reclaim context. If no pages are reclaimed, the - * reclaim will be aborted. - */ - sc.may_writepage = !laptop_mode && !nr_boost_reclaim; - sc.may_swap = !nr_boost_reclaim; - /* * Do some background aging, to give pages a chance to be * referenced before reclaiming. All pages are rotated @@ -7173,15 +7099,6 @@ static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) * progress in reclaiming pages */ nr_reclaimed = sc.nr_reclaimed - nr_reclaimed; - nr_boost_reclaim -= min(nr_boost_reclaim, nr_reclaimed); - - /* - * If reclaim made no progress for a boost, stop reclaim as - * IO cannot be queued and it could be an infinite loop in - * extreme circumstances. - */ - if (nr_boost_reclaim && !nr_reclaimed) - break; if (raise_priority || !nr_reclaimed) sc.priority--; @@ -7193,28 +7110,6 @@ static void balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) out: clear_reclaim_active(pgdat, highest_zoneidx); - /* If reclaim was boosted, account for the reclaim done in this pass */ - if (boosted) { - unsigned long flags; - - for (i = 0; i <= highest_zoneidx; i++) { - if (!zone_boosts[i]) - continue; - - /* Increments are under the zone lock */ - zone = pgdat->node_zones + i; - spin_lock_irqsave(&zone->lock, flags); - zone->watermark_boost -= min(zone->watermark_boost, zone_boosts[i]); - spin_unlock_irqrestore(&zone->lock, flags); - } - - /* - * As there is now likely space, wakeup kcompact to defragment - * pageblocks. - */ - wakeup_kcompactd(pgdat, pageblock_order, highest_zoneidx); - } - snapshot_refaults(NULL, pgdat); __fs_reclaim_release(_THIS_IP_); psi_memstall_leave(&pflags); diff --git a/mm/vmstat.c b/mm/vmstat.c index a2f7b41564df..80ee26588242 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1682,7 +1682,6 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, } seq_printf(m, "\n pages free %lu" - "\n boost %lu" "\n min %lu" "\n low %lu" "\n high %lu" @@ -1691,7 +1690,6 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, "\n managed %lu" "\n cma %lu", zone_page_state(zone, NR_FREE_PAGES), - zone->watermark_boost, min_wmark_pages(zone), low_wmark_pages(zone), high_wmark_pages(zone), From patchwork Tue Apr 18 19:13:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC60EC77B78 for ; Tue, 18 Apr 2023 19:14:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1D9C28000A; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCB85280001; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A449928000A; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 93278280001 for ; Tue, 18 Apr 2023 15:13:51 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 72FB28021F for ; Tue, 18 Apr 2023 19:13:51 +0000 (UTC) X-FDA: 80695461462.04.DACFD22 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf05.hostedemail.com (Postfix) with ESMTP id A9F8E10000B for ; Tue, 18 Apr 2023 19:13:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KFvVKf3Q; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=64q1eAGIMoGl5+j+i9CN/f+1A67QqCU1EzH3Lq2TpDsCZRXDtCsgY0wyG1oC6uvh3kd2DS 1QBvlen62375+QeKYvI5PUBOYtBIfI4idVOrvh3Z1NQhukBYeNeCmsxMsCzLQYv1La9Z66 Uy3tk7uQ++YORvtX2BNrjMH9tVst+m4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KFvVKf3Q; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.173 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845229; a=rsa-sha256; cv=none; b=06Jn0FcCZEMMmnheafsbOVKURgiiU1pvLpmNZ8qGeE4xAMUw9O9BnoIP2urG6KWRaPSj9L bSDd0LCnm6p+uTsAZ/oukNK04fdqhNYL4Qw6/4ayeYf+CdzQiba3q9pMRl5SNqHe4A+bI2 FReAwjZ3Sm+uIuKcQliJyqT3QgK+66E= Received: by mail-qt1-f173.google.com with SMTP id ey8so6190226qtb.3 for ; Tue, 18 Apr 2023 12:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845228; x=1684437228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=KFvVKf3QkfWQMBDUHAZr4VeBeoisE8O1GGsOodjwgMNuADxrQdnvk+SbprdpW2IP5r ag93cZ6VTR9I76TSMGwf5woII5HGfN9l2WN88UYR1tzcnTBufAlpM3lp/xMmNVG97M43 eNnVONsHRxzcPVv/iZgrUj1Uz8RGQ4JjzKnpXZZNAJj/FsvWfEp9h8m3xbiQ4JGMIzXh Wd0SEOETtzw6tJD+77T1b7ruvzZOu+Nz8JErUfQPxnwB2Zm75Cbp78ju67EmWgaNaQRd j09JTsw/mPs41GthMRykMfarC7ahjfxAE2/z8L+r1gArqpDDy8cskY5OjwSOMNK5kLZu 3VpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845228; x=1684437228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3kWGHSuKTgkf/KcYFfj9Qt8QOcgzsovuy8ycEW8Qbl8=; b=FYy+QOptDCUIYkdrlMtU5iMP8fs9qvMfMJ5GpKQi0n1tb9rtrj/ATN4vfnOqQbyfrG mGmv6L8bCBG+9j/4B8QKota7Cy/saYkDXF8GeI/moL0JxxnGLihSLlYkB8PEiGGzZAmn ZH91pdE7x1CAiq7cAg8hf4fliMSdjUGsKcB3EYikRrbsn6gE0WVz7JHPRfh3nU6MOvMp NvaCFQW0OrKfyF5/TP1g7/KMgVCnZ3ATAIREnSUfH6djEpqbrgZYlPdoWqovEyYpFa4m eb72+VBYQEDjeu6EyvPvgM4Fed9wA9alL4eOXa7qn2o01/+rMRZgtxDDxim9C4Z7zzQV Hkyw== X-Gm-Message-State: AAQBX9eBRWIo7Uz9sQyIcX2RJ4COQeGNV7++cxlF3bTQSNWr4gaEp5SY 82hzKJg+/7uKVFso8vZm6qfwPIfIxV1fCe9yfWI= X-Google-Smtp-Source: AKy350ZnZudpfigs0VHBHaysAID+48G8C5CQkmg6KTT32vxOaKM82OTa8s5aOXsXXAzcGRMb+jKq5A== X-Received: by 2002:ac8:5f0b:0:b0:3e6:4069:9136 with SMTP id x11-20020ac85f0b000000b003e640699136mr1124473qta.45.1681845228420; Tue, 18 Apr 2023 12:13:48 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id x6-20020ac84d46000000b003ef231cceeasm1122594qtv.23.2023.04.18.12.13.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:48 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 25/26] mm: page_alloc: disallow fallbacks when 2M defrag is enabled Date: Tue, 18 Apr 2023 15:13:12 -0400 Message-Id: <20230418191313.268131-26-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Stat-Signature: 6se4p1yeqgd5fc6xo8asqyf76cqzrsrd X-Rspam-User: X-Rspamd-Queue-Id: A9F8E10000B X-Rspamd-Server: rspam06 X-HE-Tag: 1681845229-387765 X-HE-Meta: U2FsdGVkX19jRxbaVibv87PlpG4pa/O4M98Y/OBoW3W4Ly9AmSiwuVodoIsDil9fpzqH/wqPYaUxiTvDgEVtA6Jk+P/tBfwagbBHj9jHwVjDlxRTlSo9ZI+J7V2R1lGV4XS73DoMQd0rFmouRhR7zqUom2AGTyPgufIn2kDqo71s6n5MJoXamamPGL+dNGFCjaze+vf/+ozzRQXYk2bZHSjcgjXoj+mXM0Bm8jcHLWC/KfBHaJAGLUNbEVVHjCt/o5E11lotN0gckACeqyZNcz6oK944hlDDtmWMbB8yuVs3fk6jiN7P/vy7bePD+6oqxpAdeyMB+gsE68/dvbUfh4swuBQMCUM/xbc4pO529Pbvf6wyMTv+UHOeQuIUGtS39mohFYeCoU6dO6RS0qlBbqAnbxRv6wet4qFax7CPrduhZuz5xAZcdwwl1WdVb5tYL36KE0iaYpqciO9u7KwS6xVsbsguJ81OJoQcxGbErb6+I9Sap4yUpvuaIQ4ZG0G5RN9pNF6JFDr0mkpRi6GDbVFOpd8BoazrT+m9dsleYNVkBigE421OVxtahOzZwRMfsbh6ohGu0MKMuXXxqGaFRGpH/6uxcSfNyIC3aydyG2DdskaW0I+/ZVMiLthA8gCYQv45fUIPUstMa+7RoeZqFDR8PXiJZWDXh7m8J2gCb/A16Q5wvIRyaGaEXUIm3Zn0C+E8NwsV+lkNoq/6W/EDpZBdFijSp3PbOi5IfAfJmhYvlOolFXTkRID8blCMkCI5U1lEEhm9Lti0MIDTYuKfpuqtjuJSbZAlp+ukenxgRBK47dSABqtSyIseKTceeyvB6j7G1UVK2b33WndcCDDvxXNarcT5Vz8u+yojlSWaBgPwklhHdN4cbuRMeOBf79LcTgslBBaq1p5YyBjX29aoLG7vSNJSCZFVKN+Uw92r/BuSoyIfUs+JmpHHPH0Fl6uyWWM3+UHfQK7hz2jIYDC YoNc1DoM Bosl1EHgmsIcbSThuZJjFeI+4iXXNCIgYXnI+CrSqOmN16/HQ6xOFhnau4jr9YMqHfLBOkjZAYRRxC1dNKP4TOznD9VSut8tzplvZKBXlzPFJW0imbMJjDglpS4Pueb4jm1TmrK5z46DhU0FoTTMoooZxucYeTVZcj0Th0EdOGvkcDrQkofQE7i6Nr2ANfojgYB/7hX5TdTHWT88gzsYnyXQkNdrOoD+jhwKvllcwR5+2GFEDSgZFBj2CXPQ1w8JmkAqE2IBnLGgPE4uT8KkfjA4LjMCSseOHTvuEduTrOHpKVgfd1Sf3eYy1zy/BP9R7b002yJOvLk9BvrVvHT8zB+gkjG4/z/k117a8SYwl3KPaxd7XR3I/j+CfbsDcQ58b0TNk+FPpp01NAr8CpiqRS/YzgnS/+Zvgd8p6xVbo4/ijOwQbZDAmtFTxaPEPcT+VJFSyNZ5SvrQlSixhheXLOMMVWuDS6z0F7U4cQRWgOko52eTfwcnWUaaL/rV2SynUV34aEG6A5myhrVKsV/eE6+joD1NT3EKCSs7I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fallbacks are already unlikely due to watermarks being enforced against MIGRATE_FREE blocks. Eliminate them altogether. This allows compaction to look exclusively at movable blocks, reducing the number of pageblocks it needs to scan on an ongoing basis. Signed-off-by: Johannes Weiner --- mm/compaction.c | 52 +++++-------------------------------------------- mm/internal.h | 2 +- mm/page_alloc.c | 8 ++++++++ 3 files changed, 14 insertions(+), 48 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index e33c99eb34a8..37dfd1878bef 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1258,46 +1258,6 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn, #endif /* CONFIG_COMPACTION || CONFIG_CMA */ #ifdef CONFIG_COMPACTION -static bool suitable_migration_source(struct compact_control *cc, - struct page *page) -{ - int block_mt; - - if (pageblock_skip_persistent(page)) - return false; - - if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction) - return true; - - block_mt = get_pageblock_migratetype(page); - - if (cc->migratetype == MIGRATE_MOVABLE) - return is_migrate_movable(block_mt); - else - return block_mt == cc->migratetype; -} - -/* Returns true if the page is within a block suitable for migration to */ -static bool suitable_migration_target(struct compact_control *cc, - struct page *page) -{ - int mt = get_pageblock_migratetype(page); - - /* If the page is a large free page, then disallow migration */ - if (mt == MIGRATE_FREE) - return false; - - if (cc->ignore_block_suitable) - return true; - - /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */ - if (is_migrate_movable(mt)) - return true; - - /* Otherwise skip the block */ - return false; -} - static inline unsigned int freelist_scan_limit(struct compact_control *cc) { @@ -1620,7 +1580,7 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Check the block is suitable for migration */ - if (!suitable_migration_target(cc, page)) + if (!is_migrate_movable(get_pageblock_migratetype(page))) continue; /* If isolation recently failed, do not retry */ @@ -1927,14 +1887,12 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) continue; /* - * For async direct compaction, only scan the pageblocks of the - * same migratetype without huge pages. Async direct compaction - * is optimistic to see if the minimum amount of work satisfies - * the allocation. The cached PFN is updated as it's possible - * that all remaining blocks between source and target are + * The cached PFN is updated as it's possible that all + * remaining blocks between source and target are * unsuitable and the compaction scanners fail to meet. */ - if (!suitable_migration_source(cc, page)) { + if (pageblock_skip_persistent(page) || + !is_migrate_movable(get_pageblock_migratetype(page))) { update_cached_migrate(cc, block_end_pfn); continue; } diff --git a/mm/internal.h b/mm/internal.h index 24f43f5db88b..1c0886c3ce0e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -741,7 +741,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_HIGH 0x20 /* __GFP_HIGH set */ #define ALLOC_CPUSET 0x40 /* check for correct cpuset */ #define ALLOC_CMA 0x80 /* allow allocations from CMA areas */ -#ifdef CONFIG_ZONE_DMA32 +#if defined(CONFIG_ZONE_DMA32) && !defined(CONFIG_COMPACTION) #define ALLOC_NOFRAGMENT 0x100 /* avoid mixing pageblock types */ #else #define ALLOC_NOFRAGMENT 0x0 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f835a5548164..9db588a1de3b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2622,11 +2622,19 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * * The other migratetypes do not have fallbacks. */ +#ifdef CONFIG_COMPACTION +static int fallbacks[MIGRATE_TYPES][2] = { + [MIGRATE_UNMOVABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, + [MIGRATE_MOVABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, + [MIGRATE_RECLAIMABLE] = { MIGRATE_FREE, MIGRATE_TYPES }, +}; +#else static int fallbacks[MIGRATE_TYPES][4] = { [MIGRATE_UNMOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, [MIGRATE_MOVABLE] = { MIGRATE_FREE, MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, [MIGRATE_RECLAIMABLE] = { MIGRATE_FREE, MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, }; +#endif #ifdef CONFIG_CMA static __always_inline struct page *__rmqueue_cma_fallback(struct zone *zone, From patchwork Tue Apr 18 19:13:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F244FC77B75 for ; Tue, 18 Apr 2023 19:14:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0833828000B; Tue, 18 Apr 2023 15:13:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED852280001; Tue, 18 Apr 2023 15:13:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2AB928000B; Tue, 18 Apr 2023 15:13:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C01A2280001 for ; Tue, 18 Apr 2023 15:13:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9FBE21A0402 for ; Tue, 18 Apr 2023 19:13:52 +0000 (UTC) X-FDA: 80695461504.23.F8AAA2C Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf12.hostedemail.com (Postfix) with ESMTP id C330F40013 for ; Tue, 18 Apr 2023 19:13:50 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=A6AezfgF; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845230; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qWzWNVwkSkX/nKeV3fr2KFPOgnXDvxLFllKcLvZxGIY=; b=mIBETa8L3SWB0z//37ZbPEMEE5O4ue5cObmBc9GBgectf9M5Fa5TZD8B/tm3DviHMNcopJ gkincfCab0LXMu8G/V8dbHiVYwfVHKxwBNxvOMdo2qo/qBZN18NALWJxmmlHO7Oavn+dbR KChBRS898RtZt7ucSpH7Asjz4wvmPS0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=A6AezfgF; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845230; a=rsa-sha256; cv=none; b=d64yGwch5BytfMpNbrSINt3twZNeg2tJdkFcG5k5VMe6Ish0MZznrmz/XN5c+9WNizgTuc eT3/87mx3hwRR4BYBWcEIywXbc0RaOjX1jxCBW6jXOQTez/7T2LZYy2fS74o5M9V3fuLP/ efAuNjmWDsuHougtCXMthbP2mRDmShg= Received: by mail-qt1-f170.google.com with SMTP id ei23so1769759qtb.2 for ; Tue, 18 Apr 2023 12:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845229; x=1684437229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qWzWNVwkSkX/nKeV3fr2KFPOgnXDvxLFllKcLvZxGIY=; b=A6AezfgFb558g3zl8KYkj4x8/L0DIOCUiDlAuiOpPB/uNdrligymciqY3pviiovSz7 vKSf382DQ8ssov3wdJ9qL3nD7oIEAgvToPUv9CYpTlO4HbTHGzp2IZTc2DXrdM3mXahz 0py57a20ojr2O+PepMwl9/QHioEm1FDw2MRiTZYyYWKB9AsL5d0proRbKpQwyYUfMNP1 //zhEeIOanhXcaehUnWf/FVKyKlOdRk3NVSIwARpSIl5m2koxzkBtGtcG0JMTTTieE93 AO69YNUjxBNNIS0LLC1PusXdBE3chI7eqcQajUcD2Uoo7NcibA/0DG0uhwzJHlI926dN p8uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845229; x=1684437229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qWzWNVwkSkX/nKeV3fr2KFPOgnXDvxLFllKcLvZxGIY=; b=aa4zlzJ5U2+EOZk//HZNiBSuSO7nAk3GFKHnut5+6wKiORbNWwFs0gMxxJRBW7yfmw PvFpyyDirN14oawnqbrixnETR1SQE8ky3VXYhZB/aEi4rHML9jqB2qMod7s1qYcjsaF8 p7LF+KwvPNUFKUwQgDv+BudHSwJ+FQ/CCvRXKiljNLL91ULcUBXKHS0NUYHXdWo1oGUs pYRI2+zANhq4YmWxuu/1cvDqIodciO6AokAngOlEnqWdgvbJ0wu3qhEcnvyDjSCOhrBz jV1VGn+7KfYVTWZMY9p+zGns15XBKf36FgBNRIhwftD2/i+VHdK1I8L3BX+BZPGd+qQh pVfg== X-Gm-Message-State: AAQBX9e36OIEQ9yoDoO6iLHCcuSNFaqhZEK0wHTy/E4CwaOzKdrM5Qm2 ODWNO/mi5sMTyga+o0LZ0ZIK8K5DIY/1GUB/fuw= X-Google-Smtp-Source: AKy350ZdZcMKMoHMrh8Lve5YppGzlgXCzZWIFyu+Z0bcXvlyOp64kIBGTX1A+KzTiCSPMF180KBHNA== X-Received: by 2002:a05:622a:4a:b0:3b8:6ca4:bb23 with SMTP id y10-20020a05622a004a00b003b86ca4bb23mr1454631qtw.15.1681845229705; Tue, 18 Apr 2023 12:13:49 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id d9-20020ac81189000000b003eb136bec50sm3482413qtj.66.2023.04.18.12.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:49 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 26/26] mm: page_alloc: add sanity checks for migratetypes Date: Tue, 18 Apr 2023 15:13:13 -0400 Message-Id: <20230418191313.268131-27-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C330F40013 X-Stat-Signature: y5z1y9ub37n5sozh8zoq9dyaguh85uwg X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681845230-978066 X-HE-Meta: U2FsdGVkX18dgc1TDIymCeqM04ikqc7vcmcEuX4cMjh9ICr3Y6UIjQdPAX8Jr1H7Bth6hfeTq847mjcEgdVp+gXeI7rXk2K6nzb3Un8GNe6lXZfdyFVjZBLxmtDUxSqZV/vfBJ4tcOf8MsVTS59J+dKIdSJL6ED4OCg9LFfXL6EdMTXNP7RlKn5/0gf2skcDiuNgDRG3mFzmKry5B2RPkkeZEF2Wor7mAbgVuII4eRbiNaS8CL+3jfZO6//WH56Fy5XC2q8AHamEm0AbJbFnUNinlvAoPxo7bWbD/z8feewU+hko4EEZl5WIG00U6Q4aehf794+COXXPifJCoARAn/+GS7qzusVlKc1StQ6C9czmuZo20N1LG22cvgIOkbxq4RUKbT0uKimxFRSXiA6BEztUXHEi8gOB3orVMY2thF8SiFVRJY4N05DSvHu2t3brmf5bX7hzsaIY2uwkT9CJDA3zCP9VU8OyHDlH2bYhDE0TurxdfTm7fuC3v02z7gOFMEESLlZI7FL1VuvoSg7doQRGi93+3b0JZhOtB7j6JUUhEzFYzne5llpYCzQBfgNrSRhXE3MqLsv+WoPqwIX7tqAPvR8QabO3KK9whFQlsUhXusFJQryjV/YWeU+GCSGfIUa0E9nWDZJScsRVcU57AQmSmFBaqcPI0mfkMS+lWRYTLJtg1G9tUmAmLyBnT0GPKZFS8j2V1ODT3RF4oK954X0QZ9ZIm0nMv/iFVEeWnZywAmGVxkHVo2xHbNmK87K5i6cwhgX/akaK0ETW+w/qcTO7rfDzJx9I8cJcwx8C0fmb2Pp1Dv+6uQzxDC+c4IL3WQ2HCSqEQLSuk36aEKposNzZIAiBslJKQJLTdisCaYXPRlN6rUXISRa7K3hm1LkcIPwjiM3zU+67l8+RVxCZx0y8/nCvFVlWRH54i3wupxBnx3ooImWaHZV7/XuFfe8yFABxNRzcTQOlyS1APay kQTdCAuh Ons/65yNcilwqJI6g2CFk9OuF3KFIDkqf933GZjAIYOUEYhfWKtMZbOl9KKUWxq7rXan0nhVphp1zBFlBphwN7FVYvdgBr7MjUPGKTdeILnX9FY/uYgYFKXiU03Whzu87vwCVKERBxf7elgJ0exuJUo40bnb5ETHAZ0xUu/XATixV+JFK/wR06G3Af+kbhKijmQZ6iaP6QTFmbZVxupOW1S0vRoajX3V23+iikPX2HQtkNYk1zyn+LgCmvIhrDlK+yWsYCPMt2etc3UIvrQUWwr4EjL9IxiKrgOeK7LkdZFf8XRck1YCzUfphIghcrHUXweieSMcpW+7Vvb3K/uyTRSLi6YbK/onSCJPox41Rx8ZzPa/htuvcGT79OVmL9ZnegAb60sjyWnRHr5Qn0eeZVAAQ1L+PNdFh4Wkb8h4SDgNZ8yIzxh59zUWctBw79xQRNRq+eQLUWDcVYtC0C541TAo2WuongOT0/298dXEz8gNTaodK942gJLAZAWV+O+jRSrnNA+tr3pw8LPFoZd0b8GeFmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that known block pollution from fallbacks, !movable compaction, highatomic reserves and single THP pcplists is gone, add high-level sanity checks that ensure that pages coming out of the allocator are of the requested migratetype. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9db588a1de3b..b8767a6075e8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3675,6 +3675,7 @@ struct page *rmqueue(struct zone *preferred_zone, int migratetype) { struct page *page; + int buddy = 0; /* * We most definitely don't want callers attempting to @@ -3698,9 +3699,14 @@ struct page *rmqueue(struct zone *preferred_zone, page = rmqueue_buddy(preferred_zone, zone, order, alloc_flags, migratetype); + buddy = 1; out: VM_BUG_ON_PAGE(page && bad_range(zone, page), page); + VM_WARN_ONCE(page && get_pageblock_migratetype(page) != migratetype, + "%d:%s order=%u gfp=%pGg mt=%s alloc_flags=%x buddy=%d\n", + zone_to_nid(zone), zone->name, order, &gfp_flags, + migratetype_names[migratetype], alloc_flags, buddy); return page; }