From patchwork Fri Feb 3 07:18:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAADEC636CC for ; Fri, 3 Feb 2023 07:19:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67A966B0075; Fri, 3 Feb 2023 02:19:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6294B6B0078; Fri, 3 Feb 2023 02:19:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F15B6B007B; Fri, 3 Feb 2023 02:19:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 411006B0075 for ; Fri, 3 Feb 2023 02:19:14 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1F4701A0372 for ; Fri, 3 Feb 2023 07:19:14 +0000 (UTC) X-FDA: 80425129428.21.526A0F9 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf06.hostedemail.com (Postfix) with ESMTP id 392E3180016 for ; Fri, 3 Feb 2023 07:19:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pLnnqDRI; spf=pass (imf06.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408752; a=rsa-sha256; cv=none; b=Ybe7hBsDY51GZZrTTmMU4ELW6I7Rq2ogwtcinzon9aNUW+ioy5MO+u267YYTHpFv6AR6h5 DfZtrjcsaV0Kr/nLjdSYYMvzyMqh5bSSay3EPhFuPSSLdVzl5uQfCF/ZBFW8czeUaaUGoJ dX7aS0HwNeunTUgaQgkyAZcYTiMQTjQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pLnnqDRI; spf=pass (imf06.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=GxLKLk3pAE3wpo/riCVx/7iRhKmSmx+tYP/wQaj+6Cmy8HikTdpIBbD6lU2wtnd9f431df xfz9yut6FSL87jYKUsQ23XndpXI+C6YMgy++R7VY8Pkiu0QTTbhxb+2OXRlu7HggabiX40 ap/dbLh14evKmdKB5Io7zHSllv3fc0c= Received: by mail-pj1-f53.google.com with SMTP id c10-20020a17090a1d0a00b0022e63a94799so7995625pjd.2 for ; Thu, 02 Feb 2023 23:19:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=pLnnqDRIDYF0X+7odeq2MLz8sT29lO0h9nXJRzhqSWb2cKDmw+AjnJVXwKX4LO0+YN sy6TWCVWrrhdAqdi0Athmqz7hnxbv9sXhy7zHgw5JMbTMALz+lrRMnA7mhKMW0UOaN3X QCHhXYJFOAJf6mMzTg6U9SYCB0rngQkrXqgCC9sJi9COvb9A1wQ5Pk8YAxIHPyyrhAK8 45tFQTFNKq2y3gE/ag6BjXudpVKSfK+TUTK3G6sll8axgwNiKP1UqpP4lrWqQnLzlWn0 p2jjdVOgV/IsArGVMlzfnf1uVgzpoRPyeV1+V87NiHGnhmhY+aWACx4xhLz/DXOY+iS+ dKVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=NdUtk6+LFCBZhYSuQYE4t5qQPG1Zmb7bUz1Wft1I1BSOgOEpPXpYx9SyFNRZZBsOR4 EPC5jHwwUWhsQOJtYCzAX/eLrgCzvsBI6i9s6duarBv5OrDHqgZfICtNzQ8xPBxrlU1M EpfiQjw53tP+uYp3Z9Lb5Yj7Br0hxKdBgNDINp22Nls3ftZ7sbO765D8DNrsbN+0O68N e0e2xltky1Wgh18+/06C15J0ZK9fg8FsrBbLXNu3Y25pQB8wEezO0O7CXX4kiC9kWGpf hThAgfYYpFEcvKn4aTj0JiGcfd7jJClCoQd6GEbvB9lT+wkVYOy7tqyOMpWDDUWvAqsh 85LA== X-Gm-Message-State: AO0yUKXzahn1R4fmaziPi8osR4iZ87omtMS4eM598gH0WY/H5v/bat2A ELuWBQysLI+ShTzcsEgj6ss= X-Google-Smtp-Source: AK7set9Pa1+UA0tNLvIosl6qOQXe2T4uQwVT+pZWT/dLmS0GvpN8PpOxJcaipNBn1b+VrjOFZQErtw== X-Received: by 2002:a05:6a21:78a9:b0:be:a944:b07f with SMTP id bf41-20020a056a2178a900b000bea944b07fmr12686834pzc.61.1675408751037; Thu, 02 Feb 2023 23:19:11 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:10 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 3/5] lazy tlb: allow lazy tlb mm refcounting to be configurable Date: Fri, 3 Feb 2023 17:18:35 +1000 Message-Id: <20230203071837.1136453-4-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 392E3180016 X-Rspamd-Server: rspam01 X-Stat-Signature: dnph1hz8yj3rrukokqkutajrafiq6wif X-HE-Tag: 1675408752-790495 X-HE-Meta: U2FsdGVkX18BRGFUKe+QuvlF6Nq7Yyu352SiLvK8AIr3oeEEoqD4M36z0mdk+hwuDzs3rC+kbmsrynK4PKf3s/+jWSRKSzH867Or8kVMudDipZn7BTdx2omiqwlyMGVqUJ7R2hMk38Bb//S4LPGTMb9erBWcgKszv1saLVTuWmxbmd+GLms4dUTyNPZIQJ9nSiJEaPeNIUZdpxmNWyeDEnmOCX6AC2HcvUtcEBOpn819nUHsag3yoL02QFVr3PkS0cqfkEKwWd5q0nD5PJLJcPy7edyMXz/RtJB/4InYZrvSE7mUae6R+cakdWbKjQZGflUmTAAbeJhM2YeI0jMs6BRsCdvRB0Z/vxWphAP7CVzi8Llkk3QTtAqTbROg6B97+AHoGtJVppX/UinZpNbYrqNuVVa26jFKkfKqEPuYSnHElwCTGoKX7+YJaVmsc9n1fo0jBX0MR5YD8KFZPsOin+A91mn0od+6ohbdmS4r2zbTv9A8LQUPaf5Jf70vZ3ksdA1aCzUokmDT0rnKjPLC5GEcDC5K/bxAHfb6sbEoWBprr8a/GzHBOC/NJpi5IBv2J0q02nsYAkwiPonfAvY4DoSDmkLiVw+krdcVILbsvFTMCfYCX9xvdf1hnrF9Dme+sh8rSqJeEzOopR8EI0tqGMc9hkR1k5PWHF/0uH3Td4FQ3KH52L15R8ijTZTppazbITy9nQd3ZyLtwD1QCLHFAqB/BwPPuPiSSd0046oFbyYvnUJ4UZW6r01AIPE6a/AMW594NMhALGAokQMH0niha00wbF52CcYUFuYmVPNd/3EZ/WwBnnjDfIOvzZChOt9xC3Lqpb/EGsN4xxViPhLqoapSMFvxIG7fTYxBGZXT4mAkJ+hsaRC0zWDVlDVWTEQFrSyfO3aKUz8bo11FORxadAvFIW5KCZVWKqU+S6xSSR5OTpY+0u5wtH6fEcJH1l6/qc38CAZNBgIWmq0iYWr 9Y/jUhIg VB6vMf7XSkBrItvhR0vn/JoorFQ3mrBi3Qx3iewQ9ERDBmPnaav7pC/Ern96NFuHNxLxZ/LFzCXQWXmZbICi2OL+abAsLIyy4Xr/+Tpu4tg2rHnviOqGuC05d/wiYsYfAiGFFa56e2pua+i/IZR4XwOaN2uA8aNT8NmbkUGaogCw0rO4y9ZLPTHOXbOXtvzLXIFqrZSWur+xWhTu/64EkkbogiS+J+dXRzhKnroN/q0CVXokUqDtyb7iD0JDy6ORNB/EPQ/tebokvsYwaD0q1zqJoUMkeQ0XPmfdjV0JAukffPYsmoFLgxoCzAPULpH7V/44C4+aZvHuf4Ue6Jue0GbRbG18jNpi1YeBjIVN7bOt+DqdCKJDKjSzb4z/PEC3RBE3ff5P3tSQncKMYYTQLviFOICbicmoSg+9Idc5U+jFHBgOMBVQVK91iFK221BH+xjkdOpigc5vNoyCBLymAlB8V0ZZeNTn8CBS+GmZjkScmrUprK5XJNmh3tc0DVvCgTA/fmO1FLkoSYFD5/LCoQbKr7J3YTJdC6znlbPNB/cjC4+3IdUP9JDKQjQKMbz3PrqJqDlXOwW+rbD5D6miya8a2Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, so the patch introduces no functional change. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- Documentation/mm/active_mm.rst | 6 ++++++ arch/Kconfig | 17 +++++++++++++++++ include/linux/sched/mm.h | 18 +++++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/active_mm.rst b/Documentation/mm/active_mm.rst index 6f8269c284ed..0114d80d406a 100644 --- a/Documentation/mm/active_mm.rst +++ b/Documentation/mm/active_mm.rst @@ -4,6 +4,12 @@ Active MM ========= +Note, the mm_count refcount may no longer include the "lazy" users +(running tasks with ->active_mm == mm && ->mm == NULL) on kernels +with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy +references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() +helpers, which abstract this config option. + :: List: linux-kernel diff --git a/arch/Kconfig b/arch/Kconfig index 12e3ddabac9d..11e8915c0652 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -465,6 +465,23 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching +# to/from kernel threads when the same mm is running on a lot of CPUs (a large +# multi-threaded application), by reducing contention on the mm refcount. +# +# This can be disabled if the architecture ensures no CPUs are using an mm as a +# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm +# or its kernel page tables). This could be arranged by arch_exit_mmap(), or +# final exit(2) TLB flush, for example. +# +# To implement this, an arch *must*: +# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when manipulating +# the lazy tlb reference of a kthread's ->active_mm (non-arch code has been +# converted already). +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 5376caf6fcf3..689dbe812563 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -82,17 +82,29 @@ static inline void mmdrop_sched(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) { - mmdrop_sched(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmdrop_sched(mm); + else + smp_mb(); /* see mmdrop_lazy_tlb() above */ } /**