From patchwork Fri Feb 3 07:18:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C755C61DA4 for ; Fri, 3 Feb 2023 07:18:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF3E26B0072; Fri, 3 Feb 2023 02:18:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA3C16B0073; Fri, 3 Feb 2023 02:18:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C45BC6B0074; Fri, 3 Feb 2023 02:18:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B2D456B0072 for ; Fri, 3 Feb 2023 02:18:57 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8133AC02BE for ; Fri, 3 Feb 2023 07:18:57 +0000 (UTC) X-FDA: 80425128714.24.72E2428 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf08.hostedemail.com (Postfix) with ESMTP id BE994160004 for ; Fri, 3 Feb 2023 07:18:55 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=fknWjk6d; spf=pass (imf08.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408735; a=rsa-sha256; cv=none; b=sNc+CG4dHGMcFMVxqRIGsfrPiX5cZ08QMSQ6EjZHYqn7t4rn/l0qx6VdhfmbArLBhvoAto ns52xRaOvwKsl5lTecA3V6zwA7Uem/YoVe7qpVQoZ6tBGl8LuzbkqqDoCWk1fsPMuu9sbi 6W9xuTgt8Vzf36Icofd4cfIM9Xo+BGo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=fknWjk6d; spf=pass (imf08.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=FLYNxN+89Z9I2MvycqY9goYhkfwarRJMT/94Xq9kdE4=; b=gtf7SaBMe+faz7H+lxiQ108/WCizkl+wroIWS/+gdujjjfyu4/DX+kiz05O0aDBFJDNRqS EHrG/KxfkhdQ5oAIN8ebGwcjHDO+27oZpXWMWFkWxlzzyGdg4jan6V0p4ianembI+A3oKa TeVSuvEfeaZz4PfrmKSLg9NecOmgT/4= Received: by mail-pl1-f169.google.com with SMTP id z1so4400551plg.6 for ; Thu, 02 Feb 2023 23:18:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=FLYNxN+89Z9I2MvycqY9goYhkfwarRJMT/94Xq9kdE4=; b=fknWjk6dL5edxfKxDS5Rs5o8++Qprn0+hg/MmDkKE/a3jok8QnDK4GXiRe2ULmBeBb r3AuZ/P9LO8LzKC9NfLKoesrb9aZHs64mLBnC24H1Czlpx7VypPdpfU0qNMI+lIwPDma 8leU9efusRQrRNFqgL0/9ES/Oktaei06ROJx9nOPXjjV8ygBOXdXf68QcQBQFGNZQoxx zL5/x4XAHO6NyoNRy7qX8L/1OtMC5tu/lQ3zY7P0RnL8Sa0/pIWrHTFCLfhYx+gp3SQC r4sCVpiPX5GoJ6flZz5XYEM23RAQ6PCuB/FhPZS8G0B/9Lv4MihQxroiH169o+4gRXxU Y1uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FLYNxN+89Z9I2MvycqY9goYhkfwarRJMT/94Xq9kdE4=; b=Sb8hrn4YNQ7gXReM6XJKlUJ7CfyZlYcPCL00FXwlyJnOrFtCwc7WKt51cfE1GrzUs9 kWFuerme3N23yY6mkiR2VX4XqFpzNStMlMsOXadHyT2at8sIeraXz7e/8vB4gEYak7qg dbq8a6soVcJrW+DWBTIWGLdxDO6fY3E2sofLNgNqyv0fTJQzcRf2MGdFLVUIX1AxZFvp w56hv/otv/cW+VXAM0Z9sAld1j3UK/sB2h+SngZBQJhHCNin3KGZJJlftm76HkGVTnkz 3Q6KmxZJCFuJKN0gKsGMLDmqzx7Cp7YTrw2ImOCW0FL6lcXGJktVhisdiPLnEKLd2Tp2 HYeQ== X-Gm-Message-State: AO0yUKUhU1fN6Xxvf3VPzNRB+zftlx6TSM1GAaORX99xF39VMsxp0I7d pGHiOqrf0ohCxR2G8MWL6v0= X-Google-Smtp-Source: AK7set8UmQdNB4VaSy6Fxb5wxUcSBgBrZDhP2Lw7G1Jnu8w+a6SsBl4amQbJ+dEOBbUxVNaibU8ezg== X-Received: by 2002:a05:6a20:548c:b0:be:a082:670f with SMTP id i12-20020a056a20548c00b000bea082670fmr4974581pzk.28.1675408734548; Thu, 02 Feb 2023 23:18:54 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.18.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:18:54 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 0/5] shoot lazy tlbs (lazy tlb refcount scalability improvement) Date: Fri, 3 Feb 2023 17:18:32 +1000 Message-Id: <20230203071837.1136453-1-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BE994160004 X-Rspamd-Server: rspam01 X-Stat-Signature: 9364subx6hiq5nz9jftk7z5mnebwf4qx X-HE-Tag: 1675408735-108457 X-HE-Meta: U2FsdGVkX1/OdHFcX5q9GYdOjXnJ56Ei1nJzLEBgk1hEqaSanN7tJyZjlWyday0WLWli6T3zgvbhDnZGARWWO3rPED99TfFJ+rHbFptb8VvcNN0TJZSK4Iga83aDnB1nrmtvTWGafulNS4igZBAbgI5XI+ZdkuggkOHNVcdHfZiOLy4ZYeI6EabvmjA8R6/7a/bd0bLseKKIcp2nT2kghX6F5uMdAzqMmjr/RxUgLj4dUvafMR3eqWnL6q5YimsdXF/HcpEBJNyDdRDEMWdD7HHY0BY7MB5s9x+BQrZ/L/A11N8tJRanezTnHnZujf/w9tpZx+rzWr5oS/2i47hHokIKQJg3f41nPkVmFcP3RWB0rNPno6QL4KLCtm2ywFHiTTNt8Yp4bbCfjMVBjncKOmJbHtHbUXJDWUOxzUIoY8+BOCeX53uOmo+dnpknFresEWuar7YQWEufQjcFyL22ym6PvT852AZEmDfdQgNOkODYi+ot0QhTkY0B1yAueUOGVaSm3fnrdoIU6QmdozQLWFimV2/lnq7ndzRRbi3RrhQ0uBVzJvSQuJJghwrLYKWszdI4FpncbJbo+5ikYG0IC70YNfIs8L/r/+5DHrp6YIA7zUx0+K3ZVQYN6P9vG5cYDcYJqkHg9mM4PmQa/CMq4isKerKT8i0UKB9P+UpXGIfgYKFOVX2fONxavHHGEjtRa0S7cvupsoCli6ibMLS8MGqTSNGx0VrftTjCGRDP15pC5mm05FoLR7q7cjNj5KXI9qeKM2C0wRsdThpVYk25lO32eDIFs14hhClStd05q2Ks1h4np0ehRqxq8OhirWWvZJjXWMb8dTPoZ9NX5z2oN0uKHn21uL6/J9Bi7H5VhZ7g2jm7nM3rhDKzHAeqK/7ag49ULBVAhTC/FXMgk9pTZRq8oGgwjuUKkOzUWwIMJnOi8qAJTAOdlzZ5wNnyiS6OVTnHEUq38yeaLT6DIEF 8W8D8tmU p34u86OQuEIG+RUhMw0toXOhLXfss4lp/soTcLUui9QrnknCx+U0Nx2lXi22O2XV4Fw1YJ98Qe63W+oAI9rFy4H7QtlowUs94BmtRlTsW9jQwB6m8haW/FaL6IKqLFUrL8EZ22YAr0f2cbBfo1ZA+WzXFoYI2SbfNWdjvhArftTARmQf0C1GJwFcEf8yU+pKf+Wn5t9+8srJvgPdYKC0K2m++GlIs9ST4sGrQhvP4L/R5I7AUcRaGnTg0dHvxBpUzCeNjJy1MRUxKdKByPzmh/uq3JsrPB5a3Cwu2mUZARjxiGVNQNbl/nZlbaslc1HOPolergdoJhOH6KAAQIChvJS271VcJQX7/OlkzK6k0JgtVXRHN76QVpX/QVpCCnOhvOoNbm9Ch14ZG1T/kk3EsrBDtYIZl6qKSkJd+0VyGulnxlzSSNLpw2Mkp98qHbKyCSUrUGxXLb7M67LeTKCc608LeFSu219rWv+uyRkbVOz29cZivpZ6FeWP1X11vNtwo3YFjwN7u2UGV9t1k99hJ4lFM0VOI8mRaDcpYoBBFwEfgjJC8O5Tjr6DxE+UUO8rttmdT71wr+2EMImWPaJ65+EiKhA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: (Sorry about the double send) Hi Andrew, This series improves scalability of context switching between user and kernel threads on large systems with a threaded process spread across a lot of CPUs. Please consider these patches for mm. Discussion of v6 here: https://lore.kernel.org/linux-mm/20230118080011.2258375-1-npiggin@gmail.com/ No objections so far, Linus think they look okay in principle but has not reviewed in detail. With the exception of patch 1, there should be no functional change on non-powerpc archs with this series. Changes since v6: - Dropped the final patch to optimise powerpc more, as mentioned this will be taken through the powerpc tree after the base series is upstream. - Split the first patch into patch 1 and 2 in this series so the functional change is isolated to minimal patch. - Removed ifdefs and churn from sched/core.c that were not required because ifdefs in .h refcount functions do the same job. - Split DEBUG_VM option out to its own sub-option because it IPIs all CPUs on on every process exit which is pretty heavy. - Changed comment style as noted by Nadav. - Added description about how to test it, requested by Linus. - Added link and credit to Rik's earlier work in the same vein. - Did a pass over comments and changelogs to improve readability. Nicholas Piggin (5): kthread: simplify kthread_use_mm refcounting lazy tlb: introduce lazy tlb mm refcount helper functions lazy tlb: allow lazy tlb mm refcounting to be configurable lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Documentation/mm/active_mm.rst | 6 +++ arch/Kconfig | 32 ++++++++++++++ arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 +- fs/exec.c | 2 +- include/linux/sched/mm.h | 28 ++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/fork.c | 65 ++++++++++++++++++++++++++++ kernel/kthread.c | 22 ++++++---- kernel/sched/core.c | 15 ++++--- lib/Kconfig.debug | 10 +++++ 14 files changed, 170 insertions(+), 23 deletions(-)