From patchwork Wed Dec 18 03:07:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4328AE77183 for ; Wed, 18 Dec 2024 03:07:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84E0A6B0082; Tue, 17 Dec 2024 22:07:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FE786B0083; Tue, 17 Dec 2024 22:07:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69E9D6B0085; Tue, 17 Dec 2024 22:07:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4DC5F6B0082 for ; Tue, 17 Dec 2024 22:07:28 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6C7DC0B12 for ; Wed, 18 Dec 2024 03:07:27 +0000 (UTC) X-FDA: 82906593462.18.4D0D86E Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) by imf27.hostedemail.com (Postfix) with ESMTP id 3C58B40002 for ; Wed, 18 Dec 2024 03:06:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OBgkJHqb; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=kLUzsaLjLU+Ex9GXU6YIdFJojy3CkfRVGp+Tnzm8cTo=; b=bzqRc2ghKZyE02NQsg1Wy2VKM3QENF3JzyhDfk7zauIpFbFZ37StyQsKEy/M2FkWebtqEU hdcHhkDQDtYSLAhXCdzFycnOLxfb8eK27tAQeY7pFPEOpqqXO5jXHWfzWV38FcdXOBmQCX LwEo/18D5NSWye03iyVTaZxl3z22Yrg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491211; a=rsa-sha256; cv=none; b=jf1sjDsDo+zMdHQ6rG+azERX61UtcIdgaNhPxe796W16Ko2rXAST6vdPKqDCcxm/znMaUM 9owJNBholTIKsC2Fc89JEaVfrR4TZN0xD8ZCoOElw7nfkJAsLHtWsVRfI6W6251ZA6ngDD b3s+LeRjZv06XSVw5EgGsc0APJAJmdM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OBgkJHqb; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.53 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-71e1e051e50so1378799a34.0 for ; Tue, 17 Dec 2024 19:07:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491245; x=1735096045; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=kLUzsaLjLU+Ex9GXU6YIdFJojy3CkfRVGp+Tnzm8cTo=; b=OBgkJHqb9Bt8Q3AKldMwMIy2LHdh5AcLHc9tueedqx1DrfESnp0kjPCOpLOEgRW5gY wU6Ha7wsEHZhdiC407R/ZseRMgCzHaGFwVOwdJRVp6mQ0zxtIxzz3PZ7MbhFUDLAIHdX 7GIf8dtju+6DKf6uk8CKhRaJbWh27VyeD+q/R6mQtp9GKYh8oPLgj2Ykhu4Z1Mprrf5v gjWtlFbrN0n8+ZQhrVSSxslSCNqDLDwzciXfXCjPwFZBNgLxmGz8JntBgFQySN1PgpB9 HFHPbCwhOKWwWn/IMnTrd/CrNf0oqK2ptabzHSj9Eku5zgaTZGzMs1WWzjAd9dkGBz2f OT1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491245; x=1735096045; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kLUzsaLjLU+Ex9GXU6YIdFJojy3CkfRVGp+Tnzm8cTo=; b=GvYXyUQH5mnD/NZ8kIsD+1JgrXiwAIyZeX63Taot+Dgh/6adCqre7GvO9HjSbliPNG oeqzW4cLOwGRiMBeun/XPQn/kjapsjSnSKmIsPAFVH07eT8IvUhbmcK55vwadvDmbKzA uzbwYMfRHm1aat4P0jyhgaxDYEqvzDHNfkSU5pkBwUOgIIuRSb2xviQPNdeutd2JDRQb W5Ppl7ClK88MUuKVH9x2b1fS26pM3mnZs6/5yOrdPqvBTHUQSPxaFHoY/AJw0Bo9ADDy pKy8IefFxeVqS72pY8x3t6PdhEfuaHeXfwB9KMR8sr37LaP19g8KzVj7onSCQESGkEop THUQ== X-Forwarded-Encrypted: i=1; AJvYcCWFofscPjeJ3OoLZpRfRgSvNSBLx9Tl/ZOw3f0blRP/CwyD9uFA0EJUD+D8JkO7KpMbUyNSsJ05Zw==@kvack.org X-Gm-Message-State: AOJu0YzMQMPjvinmobxw7Jq02y8DxYeG6EwEmL0Md4AJlP76vKoYxTGw 3TKQc2DNVHmZNUQAvr3dJddzdfwjAUPUOL8/AvsA/tcAHdQ4OFQY X-Gm-Gg: ASbGnctlVss+vLBSbuJqiSUJmRtbh6iXhT6O/UXflUm79PPCxd9UVKz+IUanbF0M2oc LjpgLWMzi5mySN9CfcWbb0FVNXkrGsO7akuGYLdYBX6nlomkmQI5/nTCu0oDdQZP2emsVNpgz9Q NQapQo6//XAxLKdkQbG/vg+bPJ6kZ3wZi2gLnFndgSzI5minvGXWez1cnCsTEqA8MBf7kr6qYox OJvwsjpMY1RdUDz8RAdTZ25hplpemws8RWHojwk7P+R+U3rcflTY1XuJzhQRd0= X-Google-Smtp-Source: AGHT+IELkbGioOIBruYnfnBhjvrdD/9nxp1XARpuv68LnPIsDO9cv4uN+8AnnqNP7RjvvRS0m3wP4w== X-Received: by 2002:a05:6830:2115:b0:71e:5a:f4e6 with SMTP id 46e09a7af769-71fb762c98dmr791362a34.20.1734491244697; Tue, 17 Dec 2024 19:07:24 -0800 (PST) Received: from localhost ([2a03:2880:12ff:70::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-71e484ae1d4sm2469497a34.44.2024.12.17.19.07.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:23 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 0/6] bpf, mm: Introduce try_alloc_pages() Date: Tue, 17 Dec 2024 19:07:13 -0800 Message-ID: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-Rspamd-Queue-Id: 3C58B40002 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: nq57tb4wmpktqbc351yys7kriekys8cc X-HE-Tag: 1734491211-770467 X-HE-Meta: U2FsdGVkX191QuB/WYsi3Z8c5hZFfil17+r/kFUfCEOD0qDIoAj/GH7MTZu+J1dYpQeC+vxKupFpGek7CPqplf3EcPQO2e0AlgvyHlIsPO5TVmfoZRrlufStG/TT2NvahMzA8dQBJHoWZ9hLt77VB/gx9X2sgtSKeESDos54Z7q/WfRrDj2HfpSORQBm0/2WInUFxInhM/OKHCQncAzGbpDQaqztiuE4PZ/DlTGdON+q34SevyLJKGq+YwmRj1EGx7KpPRwnz3BQ3pkRfJYMPXTrNiWzcTzFqaqTbBWz+H5Vu5DCpKDsuv97Ve9aYmf1EM8NyKco1wzTqcY9uTqT53v5TpG20cE03+5c+4q1OYB+ZoUM5ZJQuNvqEtpua88XFYOZLq/xYKQRVcOgpy+LYyqQOOiYXBvEuCK5ppHmvx2O0l0WVrwxylTz17opmx2L/BfxtZI/BMhQkdXtD2aVLnWD4z6VVX9SvJGcBaMMt7Lv7gM3Yorqs2OpAfphxYvSP6ZN23gRrU0zoBnmFV4+ag8xRM1RWxaTY1m61UxIs6DW8W8LTjE2xN/3JR5lirLYeytLl/YqPZ5bMdEvP6cT94UWSEOsoQS4L9PfW9v+z3rK1Wzn3f2shYHwhghJhfUtLaVLn6YnEQgu+7DhINjPJqbJjhIoUTEzPDjHDDjelVoLcYvS8rMEPnrf5ROjGb8xNHG9omdEuRbP3yWNJlLTVauFsO2cE/OKzxIT6cU2uNXwrzoqiwQj0NRwvZxVLq4fNuEiOarcZXe4ydCAecwjMA7dscEMVaHT+ZLLQmxBPV3IlH/1ddjQ6eoFtFvOa4fvpfNEMkwISRMS25USORUVeYzt9LCsTm7lGDHDh5XJa7LGMAb3zHwSzdB80/hIRFMVGefh7Nmen+NFQ8xIgx1ykiBWCca1CHXLypNy/D+kcFLanSgvnbddlsnr7Wt7B4n39vWe0zKkItnEcUKuCHo KSAKDdl2 pOm5zLtiT30YHeRZ0Mg0b9mpZiVpSkiQGJkuvhHzc4fDeRujvrRKPugAOE1XJWr4D9TsGcaMIm3ivWQ++Zhx65/HlCaYglYKS6nYd120xO+09WX0bPThYTfgqH1VdvYKVGMZzfJS/1s9CL6rxdIgI/WJ8vYc7Mky5gjj69Wkrk5rSdwSbfNrrhO0OpYkdV/xLTNA65fW46mLpSpmdik4KTAJPqRWS/JwGzOo8smN+GJ/xCTk6FInrLuZSAkOLIsJUE3hwVnsKbF+afAJgSvzfBNAes/8+zMw+HRr+SMwNCinf1sty3K87apEwFgGDdmQIQBWFCmiytmFKdAvZ9Iux5ijbnf1yGjFsRmaRZnWqsggcW5Xf++sAXSseXiYtSfqTUs5WRWyRXNww9yv2/eicDDrtmLrSRP2KlsU/XOQTnyu4URGf5nFwYbEQyygEZvhC1AdR X-Bogosity: Ham, tests=bogofilter, spamicity=0.024731, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Hi All, The main motivation is to make alloc page and slab reentrant and remove bpf_mem_alloc. v2->v3: To address the issues spotted by Sebastian, Vlastimil, Steven: - Made GFP_TRYLOCK internal to mm/internal.h try_alloc_pages() and free_pages_nolock() are the only interfaces. - Since spin_trylock() is not safe in RT from hard IRQ and NMI disable such usage in lock_trylock and in try_alloc_pages(). In such case free_pages_nolock() falls back to llist right away. - Process trylock_free_pages llist when preemptible. - Check for things like unaccepted memory and order <= 3 early. - Don't call into __alloc_pages_slowpath() at all. - Inspired by Vlastimil's struct local_tryirq_lock adopted it in local_lock_t. Extra 4 bytes in !RT in local_lock_t shouldn't affect any of the current local_lock_t users. This is patch 3. - Tested with bpf selftests in RT and !RT and realized how much more work is necessary on bpf side to play nice with RT. The urgency of this work got higher. The alternative is to convert bpf bits left and right to bpf_mem_alloc. v2: https://lore.kernel.org/bpf/20241210023936.46871-1-alexei.starovoitov@gmail.com/ v1->v2: - fixed buggy try_alloc_pages_noprof() in PREEMPT_RT. Thanks Peter. - optimize all paths by doing spin_trylock_irqsave() first and only then check for gfp_flags & __GFP_TRYLOCK. Then spin_lock_irqsave() if it's a regular mode. So new gfp flag will not add performance overhead. - patches 2-5 are new. They introduce lockless and/or trylock free_pages_nolock() and memcg support. So it's in usable shape for bpf in patch 6. v1: https://lore.kernel.org/bpf/20241116014854.55141-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (6): mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation mm, bpf: Introduce free_pages_nolock() locking/local_lock: Introduce local_trylock_irqsave() memcg: Use trylock to access memcg stock_lock. mm, bpf: Use memcg in try_alloc_pages(). bpf: Use try_alloc_pages() to allocate pages for bpf needs. include/linux/gfp.h | 4 + include/linux/gfp_types.h | 1 + include/linux/local_lock.h | 9 ++ include/linux/local_lock_internal.h | 76 ++++++++++++-- include/linux/mm_types.h | 4 + include/linux/mmzone.h | 3 + kernel/bpf/syscall.c | 4 +- mm/internal.h | 2 + mm/memcontrol.c | 20 +++- mm/page_alloc.c | 152 +++++++++++++++++++++++++--- 10 files changed, 250 insertions(+), 25 deletions(-)