From patchwork Thu Apr 3 07:43:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Hocko X-Patchwork-Id: 14036757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19010C3600C for ; Thu, 3 Apr 2025 07:43:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB59B280003; Thu, 3 Apr 2025 03:43:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A64E2280001; Thu, 3 Apr 2025 03:43:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90480280003; Thu, 3 Apr 2025 03:43:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 704DC280001 for ; Thu, 3 Apr 2025 03:43:44 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BD03CC1A59 for ; Thu, 3 Apr 2025 07:43:44 +0000 (UTC) X-FDA: 83291943168.28.A0BF405 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf19.hostedemail.com (Postfix) with ESMTP id BDEA51A0004 for ; Thu, 3 Apr 2025 07:43:42 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=DcliG1XP; spf=pass (imf19.hostedemail.com: domain of mhocko@suse.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743666223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WZpyhtXuLo3xH2Sml68WjXK8wPrHfMMIBvsDm+q1RIU=; b=zic6q0I4+U03RzFIWtVUUH02U8CsK8V/e53EMesnwLxUK5DH+vOV2OGKy3KZLKlbr68e8n DRKju2RXL3BbLm3b0BVBayXguN4l/0Swf7ib65FsLyWisggGTZ8ykidaw8NNB/UtbTHzAG KaR23c32jlOXJkITGMcozdanHHqj4SQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=suse.com header.s=google header.b=DcliG1XP; spf=pass (imf19.hostedemail.com: domain of mhocko@suse.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743666223; a=rsa-sha256; cv=none; b=Eje6sSLPnzeLKtgMAZWj7xEd0vZ4XuQLHNgHdhXbCizOUF1Go7flU4sjhA0lUfqynlSZFO 3Ye9yHc+QwrRxfSgUfg1hY2G37ECHpnSRPTKlYix/hi//LRxGhnbJQXoUpK54iOAeDK4iK Q+xYMmPF3+H4EdGhYS8SAFEQ9EP+L80= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-43cf257158fso3237095e9.2 for ; Thu, 03 Apr 2025 00:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1743666221; x=1744271021; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=WZpyhtXuLo3xH2Sml68WjXK8wPrHfMMIBvsDm+q1RIU=; b=DcliG1XPcVcRu1M9oudLwJlBKoLYuGgmkSxz+JqCjYOwX/UAJPbaiOcPT014lXTsMN ZnzUsV2bVW2TbsNounzHUiafCxg8U4DwdsV/egUYQGROJ8egvcaKm2cU7BhAaWXG2hNa gUj0ZzltHVHzVIxQPXcV0ZszIqEsQK0FLPGAkyXBJ6wzhvAe2mPaIEZdpyLfICCEfekk NvdjIudLyTUD3MVi8AOaWKKgL7o7YUzeztQAvKBXvrl47ReMhrrN+dpdAPmUZ+kpQ0Ae nPHRDGBU48Tp+0It2Ka+hfWhHzQFo2h4bUMg+OLUOMPj+2spY1gNZDpQcrcK6A4Ba22W AzSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743666221; x=1744271021; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WZpyhtXuLo3xH2Sml68WjXK8wPrHfMMIBvsDm+q1RIU=; b=mtsuWLF2Dh4hhC4WXVBftMyTcgS2Dp+y5APOrOdgB9FrNz79qNwcA06DtVZnbP9kvH OTXMid6nj4WvXvCCqi7TrKsiV/9+8Stc5qjePZ2lYtb7pE1kqROtOO/BpbnmoKOtrJQM BxqGtHC+lIa/q/NHJjvbizsdjfG1pbexG77mc+K5iAMqhGR+j39LfJhQl6QoYXDDaXTt zXINzSZbZbVyx0gnnWEaQP3eVxi4E2xqU5cFayjbgTlV3BqF8WoqfrozOWA+CJGItXOz ZNXGnL+eLYWnMDYOAyKWvxniLD4QBvW4c+552f69sTy0J963TVWALnH8VPu3vJu57VKM Y+2w== X-Forwarded-Encrypted: i=1; AJvYcCW7C93+aQXw13rIzEEg5hzld2zPvVi9okbf9VuQQNY0b9VY93idTPZr1anmp1B3xZD4WpUtUTgz5w==@kvack.org X-Gm-Message-State: AOJu0YyAFe0tBdM7JadOGc8LI76xaY85BUc1YkoW3v4kpOxmUhWn9ocE wKncJLCweqEeeYwCJSWJRXeHHuLPk1A6UDqicTrbe4ReV8abcZsJHAP3OHtlMsg= X-Gm-Gg: ASbGncsY3pi7iKwYBNsobLP++yK7RWObnODRAfGdQfyQSjahTcDfCfilKS5WdeJqXVu L6Y/lurTXMOpxjTasSTWPWZnit/O0BrIBiSAU2Oy+HrVTsgTxkEmvLnOnFVUIXTr8J97AsaIOQT Q2LpL4En1lWIkCFisTC4D3mTO1uK+oae1xIHxVLWC6aTOyrjF4bVk+i/8ZkYNuwLBn5Boi+ivsQ 1FggLCo+kJOP2R6yKrGB7fZrL8Xx+5cgJndPH2MWKIPHqbaa2xab9B6yZDBmok7vTy+kS/roJPM LzPJRFMdUfqZIoVgYTik6fKNJlaoz4Su0FpSt53dFv8dvz9gVnJxNB4= X-Google-Smtp-Source: AGHT+IHfBIPOH1Mhz0IdEXlZe/sRjvpPG8DdHc1g9vaBkJEbUYMjBRvymcScZwZ/vEw6LkeTX74Fwg== X-Received: by 2002:a05:600c:c12:b0:43c:f332:703a with SMTP id 5b1f17b1804b1-43ec150b3b3mr16744965e9.31.1743666221292; Thu, 03 Apr 2025 00:43:41 -0700 (PDT) Received: from localhost (109-81-82-69.rct.o2.cz. [109.81.82.69]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-39c301a727bsm1021905f8f.27.2025.04.03.00.43.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 00:43:41 -0700 (PDT) Date: Thu, 3 Apr 2025 09:43:39 +0200 From: Michal Hocko To: Shakeel Butt Cc: Dave Chinner , Yafang Shao , Harry Yoo , Kees Cook , joel.granados@kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Josef Bacik , linux-mm@kvack.org, Vlastimil Babka Subject: [PATCH] mm: kvmalloc: make kmalloc fast path real fast path Message-ID: References: <20250401073046.51121-1-laoar.shao@gmail.com> <3315D21B-0772-4312-BCFB-402F408B0EF6@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: BDEA51A0004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: m8zcw7b64rzwzzmcyswni57qeukfkypi X-HE-Tag: 1743666222-840235 X-HE-Meta: U2FsdGVkX189bhA3uNiFiSauA5GnD4MhlCpaIw3260sbhnvYfW3HOxkRnqwH45GzbxOAX/1FnkqF6KOUMHj5JPdxhiyrm6jjh+2eYAXTV5HJmyV0Ux+la+j/nlbUqJtb8BJK2cJoL88o1L/VSDnQw5tj1VCFowFdnG0i5kjO6bb/ECHXge8OaETZmZmRrxmLlKmYxDN24CTpDBvF1d0HZMUJjd682gdv9CkxYOGzL7iEhFAqEmJ8lF2U4Dqlz23twIG3pnW7cth8ve2uW48YWpVaIKPKAjWBzzDezJgTL7zIOhWSBEUjSkJ1hr5fdrF8FivqM2Xay0V5QfPTLDcp6WmBjlqDghNWSZNH8RlhbHapv6nZncJtW6/bLy/9SjTk037i95G/sMpNdTUoQrHt+//MgcNojMsEFIKPK8Uf/wlJJr3gNa8oM2CfvQOq6q6Ftok+ORe7xZNDQaWZXABdjty71QVuLlf7lg87WJzfKDW4J0+bqqFe5aEodA9Se6WWaKOn5NIB/fPw3POMQlCVJp8M/j2KQFDCFJ1ljq4RVDYCyX89Pk/9PfunkAYkCt9TBIsaxwVyaR74ygtkt1Q3JRpDnC3zaH7PkDOUqM+AhvOQzipfHss01J/Q8DX/ckgizlJc2BgFFNsozCVy124CgkmQPMLlBKML5J38Sf2fWJYccKMPGVSpbCmW6D84myR3c39qaoHHd4PXqhqCbnwmvS15/Xx2vwpoR0/JoOJlJkdusnyt1fJEAQ9JyMAzU8aaUez+MaCPHI67fHvR1l12HPSrEWsOsXw/obY2+K9ksCA7x+QHZvQ4V24GzShMLCHNBzTqgvJKRhj4jKmAhmpFrjlaGxGVKrF1NWgRStrmddtNY/rdA+t9HFlf5TiRnZT4CfPNIDM99X4ZpgwCYAAlbmjyUTARjawENbnA1LuEL5P/gAJzYFoThuWsGpU1qEMO/PO3H/LhEOLXVGe1Og0 2cKiFqW3 UhAdoQXVx9xi9xAMNLlwBZCQ3GgajF3eqkMj+tGx6vaa49lalWfxEz+pfVnZza/YkeJY2nTcTSRZ8tkrf4MzCdp7zujQcMADlsG+RcSl0+LgqEXwlzU/i2WwT2IeiR/wxbElyR3azG1bFcfT8EldHeOul2MFmUfWQ7wKP0P4C1pKCq2XprL+94YTudOreKEw1xFtOS93dFPah2ksaI3YjO9WNsHSBxDFjKIUbu2kN1OdWvPtwoqigs1uNAo7OSFEwbIuslLpKViH+/ML0jOb6n/ctHdExs/r2LmEDoeZ2TrU03soimntRbf7BvpRjxUGbwxg/JKAOUOYVhQysvIsgqnYFyfWE2omtQMILmgPsjcRdMvPapgi5kzFCgPeC8jX3s2unsXpSbjyIxQbZjPfRwwFGtoN1FpKpG3YlJIjtGUz+HwecmiiIBb0sBj+crefCFk6RCgOXd0Jdk06stFioO2sjHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are users like xfs which need larger allocations with NOFAIL sementic. They are not using kvmalloc currently because the current implementation tries too hard to allocate through the kmalloc path which causes a lot of direct reclaim and compaction and that hurts performance a lot (see 8dc9384b7d75 ("xfs: reduce kvmalloc overhead for CIL shadow buffers") for more details). kvmalloc does support __GFP_RETRY_MAYFAIL semantic to express that kmalloc (physically contiguous) allocation is preferred and we should go more aggressive to make it happen. There is currently no way to express that kmalloc should be very lightweight and as it has been argued [1] this mode should be default to support kvmalloc(NOFAIL) with a lightweight kmalloc path which is currently impossible to express as __GFP_NOFAIL cannot be combined by any other reclaim modifiers. This patch makes all kmalloc allocations GFP_NOWAIT unless __GFP_RETRY_MAYFAIL is provided to kvmalloc. This allows to support both fail fast and retry hard on physically contiguous memory with vmalloc fallback. There is a potential downside that relatively small allocations (smaller than PAGE_ALLOC_COSTLY_ORDER) could fallback to vmalloc too easily and cause page block fragmentation. We cannot really rule that out but it seems that xlog_cil_kvmalloc use doesn't indicate this to be happening. [1] https://lore.kernel.org/all/Z-3i1wATGh6vI8x8@dread.disaster.area/T/#u Signed-off-by: Michal Hocko Acked-by: Shakeel Butt --- mm/slub.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index b46f87662e71..2da40c2f6478 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4972,14 +4972,16 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) * We want to attempt a large physically contiguous block first because * it is less likely to fragment multiple larger blocks and therefore * contribute to a long term fragmentation less than vmalloc fallback. - * However make sure that larger requests are not too disruptive - no - * OOM killer and no allocation failure warnings as we have a fallback. + * However make sure that larger requests are not too disruptive - i.e. + * do not direct reclaim unless physically continuous memory is preferred + * (__GFP_RETRY_MAYFAIL mode). We still kick in kswapd/kcompactd to start + * working in the background but the allocation itself. */ if (size > PAGE_SIZE) { flags |= __GFP_NOWARN; if (!(flags & __GFP_RETRY_MAYFAIL)) - flags |= __GFP_NORETRY; + flags &= ~__GFP_DIRECT_RECLAIM; /* nofail semantic is implemented by the vmalloc fallback */ flags &= ~__GFP_NOFAIL;