From patchwork Tue Apr 18 19:12:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F802C77B78 for ; Tue, 18 Apr 2023 19:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69EF6900004; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64F4B900003; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 516AB900004; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3B01D900003 for ; Tue, 18 Apr 2023 15:13:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0A89D1C653C for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) X-FDA: 80695460496.15.00B48E6 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf30.hostedemail.com (Postfix) with ESMTP id 350EE8000E for ; Tue, 18 Apr 2023 19:13:26 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=HuPalACv; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845206; a=rsa-sha256; cv=none; b=zyxhIje8h/BBjRrM9hNxXEOqPqbvybZu+UgJt1c6qjsG5YWOE4mPCf7yOeeTyjdT58kJ37 728G5MjRZqJ3E3UZZXVFlTmgFLhFb1Ow1X4ooVLoeczbiKr8Jw65/T4U/gwBD3AxPorcf1 n86PFI2WyT6RKhwTzkVxFaC4FC/AcA8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=HuPalACv; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=ipdVFgZ/6USzCEncAgfVGLcsJlHLzLLVki88Mx8mqivGZTii1u6h2kaxdldZ/5gYIakbWW Yq45Ngu4OJ0bjxTFfSNl9w0NHZhEEX4oRSh/TZDKw6PSACIinSNG4lNXdNbh8W2Z5DMatm GVfNI8s7xI7BiuGOVml7FeV0nr7zFw0= Received: by mail-qv1-f50.google.com with SMTP id qh25so19655656qvb.1 for ; Tue, 18 Apr 2023 12:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845205; x=1684437205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=HuPalACvDhuLF7KvDz6g49PPKO7XyqovkGZVXCYy6pLcxgFDYRU6w4grAPkt0YvGk6 pHJ521bPBzsCouaZvOOPJXwsvNRmORjMrJzURqRnWoJKVVIQ0LYCfgJo8dAWEnQZSnm/ +qvQ+RSmv9XdJ2WBvUINtpcBP4b/WzY099qV6MGA2jIRIxot+itSnsyKvOmEvArOiu1Q S+M3Q1BZHNM97/Br+cYmnpOE4izzfu1L2C7HfcIzhT5hZSS40LopQDZ3JnFsmhJL3dMJ Cq3x+nKbD8VyFtiLxqGEhPZyLptBiJ+Tm3yhwtrd5LnIqpPyDSHTFaSAvqO642ZLRgyW OGog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845205; x=1684437205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sTPdiQIdreogGm8cBWdKbYJibQNRp8zcbNt+du8HIBU=; b=V8QxiDS2rPqnCpm0g3vd02i1cKAVbB8lpGq4S+3xe8dgrv0uwdLr3Q4v23pVtA294P ldIctdacqNpqTJu0arCiMhXVGYiX63IyIHWBJ6xDGMyDcbi0L4K5bkWDYhShS7LWgAg+ gVzoxIwQ4LaMKk08VupyNzhagxxqLXegBzjL18ivoOOqGVlJmWBS+3pFlAyOYDWWa1NZ a/eNrFDuc5lVxsmwAYSsrtX8GC8n5PgZrA5FDz0dUSnCTDVAc+lXUK2ZO/UoQ4Q6YoSm TNAEGd/LadedLnyjj/qB4JSF0Rm2KiyXHH9k/FmB8Zl17BSp1aWUwa40sXXXTfVcvpPv nlyw== X-Gm-Message-State: AAQBX9eycYvcElGm9JUILsU4DJWXvgPA8DWOKwe82CIYyV3rH6UzJwuR qOR0IRxCNvP3YZSFRK7yZKgb8JmWefKzU1EpC7g= X-Google-Smtp-Source: AKy350aDUeUhXJBSWIivTN/XnDLT5DJ+ST1mc086iGBK1cA1BfRASnqlSz8S0NO45keDkBhRWTpTfw== X-Received: by 2002:a05:6214:2305:b0:5f1:6a35:60be with SMTP id gc5-20020a056214230500b005f16a3560bemr288353qvb.23.1681845204992; Tue, 18 Apr 2023 12:13:24 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id m8-20020a05620a220800b007468b183a65sm4166837qkh.30.2023.04.18.12.13.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:24 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 05/26] mm: page_alloc: per-migratetype pcplist for THPs Date: Tue, 18 Apr 2023 15:12:52 -0400 Message-Id: <20230418191313.268131-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 350EE8000E X-Rspamd-Server: rspam01 X-Stat-Signature: 5jstomscikubkhiz7ctyshax5r354sns X-HE-Tag: 1681845206-182975 X-HE-Meta: U2FsdGVkX19rHKBjYKdNfubXTpKie/zluJhSEEDj8lCsP4hiPWBNgpGYqE3L6uXvKSOr9BzejeqMkEBJ+oYYpHV4dgdCYeEaaSJebHXjsaRe6e4Uv8Ng1Z7jqSM30PyAdk5pvQDzhnxPJDTh2yI3wWQ+zF6Im2Hi+iUQ8lFXQhVZlkQN1j7X6YSAFsi9JewbnyqJ2HU/PBOvFGUXoZDHO+OuceE4qrbP+npP59kSXUzc/PGKkEZ6VSNqZWL/LV3c7VmqigA8Ks2K5oAApTlACTKHS6c12o9yeNA5QRBLsdzbZ9aUKf3RBTO+3zhaBeQvgZMl/g9nyL8b3VohHh8P4kgGsm2lQEebc4tI5CwU9pVGcK7iyapk4ua87Td5lHKFUpEw3k2JAgygyaduK8VvUVkmXdu3hhszQachQGcrnvNhfATLa4/wRDg42GLHDbrQwQOD8AGj9VhyPTKXp8OnUSjEa6WJX/7vs8f5FpMwmLdiLhurYbI6ngwYxd7rZUSsFKeSKXu6HIhMEkdk7f9CHpb5WSgqSn3Yhq88/f9uRKXDXumNvhDNkIv+tHnpdpiNXUIcJcu7TNGhNqwo0TmH0f2XZytyAUgcWdErkTfCLKlIQRFjqekgmBkUQQmflpAvLSFREqoRonei2kvkjXVs+yNN287z2WnJgK+k8nPXTGLp3s0f/W/Noorm4254MRQJ3Z4SaYdOIj0qY+ZARPeDXMbh3dKJqf2jHKF7bDD8WTStdUjWJbRUHX+3DTzdpWaWLx+RxEeVbdQho7Cd37jIhQF8aad0gNQGtTkNj63L8k9P0KqD+4qPdyGR3tY8xUoR3yPGrvK8OT3tvjBVQz2DaV2y0FtLC+xe/vqEHg8hWSXOeTSli2B50Vqb8BWykVC/f3Wkv+w7chCSsxnr4K/MJio/X8NxogFbPTOQx99bMWalkhhcBcRvoT3gNWBY0Isu9bXkTmzaTpqai06fVDm YqH6MDzf ASltprdK+/EyfuaGwQYCWbtbkH+s9xvREKGrwyqYY0CWfEXMljqI7VYRgc/5GKYoMreG6/EMyJIir5yyhAHy7JPE97w5HegdBQaZw6y0EIF6+eJZYpaDm/YOe10bPMw9cJcz3GCzBpGo2H3cEp8kYag8U35CIrK/M4bf3E13CpGqIL0yStSMybNnQJXjsvvsq4sOeS7XMQj78u30t5J2IR6mhjt5/6HeRyrtcsl/QBkimf7XDnCQBGfEpud4sgDrpIgjbd9akGbU3/ArWhhhjrXSpQBpkRuyAxRtXlCCOrbQOsFjwq3kJUpBP2Mg7XhEvCRC4dZZAQx/wk3mii7gyyvgop5D1l0eFv2KNRjLunM2UmTaZTi9m45TezbGxmzuUX8/OS8G4TdxxWLewnT5+gapkRQVb699oVgUgH1zOwYrYXn/au3Jm7PsYgPpWwZz89KcnEqzRkLTmPGMa8Q/255aiJBtyCuLet4DgcWXRAdta58kggSZkscAW9VL8Me3DzLDwvvedQEzSnS/5eKGAt9oGnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right now, there is only one pcplist for THP allocations. However, while most THPs are movable, the huge zero page is not. This means a movable THP allocation can grab an unmovable block from the pcplist, and a subsequent THP split, partial free, and reallocation of the remainder will mix movable and unmovable pages in the block. While this isn't a huge source of block pollution in practice, it happens often enough to trigger debug warnings fairly quickly under load. In the interest of tightening up pageblock hygiene, make the THP pcplists fully migratetype-aware, just like the lower order ones. Signed-off-by: Johannes Weiner --- include/linux/mmzone.h | 8 +++----- mm/page_alloc.c | 4 ++-- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cd28a100d9e4..53e55882a4e7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -552,13 +552,11 @@ enum zone_watermarks { }; /* - * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list - * for THP which will usually be GFP_MOVABLE. Even if it is another type, - * it should not contribute to serious fragmentation causing THP allocation - * failures. + * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional set + * for THP (usually GFP_MOVABLE, but with exception of the huge zero page.) */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP MIGRATE_PCPTYPES #else #define NR_PCP_THP 0 #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5e04a69f6a26..d3d01019ce77 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -710,7 +710,7 @@ static inline unsigned int order_to_pindex(int migratetype, int order) #ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order > PAGE_ALLOC_COSTLY_ORDER) { VM_BUG_ON(order != pageblock_order); - return NR_LOWORDER_PCP_LISTS; + return NR_LOWORDER_PCP_LISTS + migratetype; } #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); @@ -724,7 +724,7 @@ static inline int pindex_to_order(unsigned int pindex) int order = pindex / MIGRATE_PCPTYPES; #ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (pindex == NR_LOWORDER_PCP_LISTS) + if (pindex >= NR_LOWORDER_PCP_LISTS) order = pageblock_order; #else VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);