From patchwork Wed May 24 06:04:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13253280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1291C77B7A for ; Wed, 24 May 2023 06:05:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 560C6280001; Wed, 24 May 2023 02:05:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E985900002; Wed, 24 May 2023 02:05:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33CDB280001; Wed, 24 May 2023 02:05:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1FA3E900002 for ; Wed, 24 May 2023 02:05:14 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E9BFD1C768F for ; Wed, 24 May 2023 06:05:13 +0000 (UTC) X-FDA: 80824110906.15.B3441FD Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf29.hostedemail.com (Postfix) with ESMTP id 1E398120013 for ; Wed, 24 May 2023 06:05:11 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=XZsStjLC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of npiggin@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=npiggin@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684908312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KNLXI/ZXaW5BbdyJvbyW+B6lruy4aiQp89HdExaBNqE=; b=nVZMqHbmUrnJnSmw7w1a3saCAQpn8R4vjCWxk4oGPK7U/dgi1uV4uS7EPiSe1WIUah4AaD 6UIGjsXLCVeflUnuP5KzpeZPmp42YodB2s7eCxMXglGrDl9HXSs6nW1tgWy1wc4LaZ13fu MA5bbM7j/+7v8l09yy4/XHrPZM6Pdhw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=XZsStjLC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of npiggin@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=npiggin@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684908312; a=rsa-sha256; cv=none; b=G4lndIkcvkyE7LGqf5UM8GGPNJdwILK+c9Z25vG3a/o6IPH8m6DgTRnsbV9RrYYvhqZ4hJ +IlIuz6f06uRKuaUMtaKxySl/0bDTKQzBAELPzTPi07DkFsNiL5+sRo1AdcgS9256R8nAq V5x3U3zMySNbaU0RfsNI1QB2ZWOa+Ck= Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-64d341bdedcso362911b3a.3 for ; Tue, 23 May 2023 23:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684908311; x=1687500311; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KNLXI/ZXaW5BbdyJvbyW+B6lruy4aiQp89HdExaBNqE=; b=XZsStjLCv18FM2g7O3gXy/8cWdYtOMZq5OMcEmXO8qgBTvKakpexXgr6gxA973G4a9 ULcQPjBmdUtrgKCKx8Nnm+RYUVrUyFoqIMf7qOjcC0OAc6puDsd6I5yosXHIpOhFPfNS GS8UlLWA5yPc2pAzipi8vy1aBvVq8raTYwBMy7y8BeCPni/J5UsCM2os9B0DyFh0hYts cssUlCXS9k8JiV0husJODnB0hbUS/mkq6MNQ8K5MvipAa7Pn1cXTURPGRUBbxUqLGojr AWfnlSvRXfQvNucyl3Oe66sXszCd3uqR8kRLBhTJsab/b2kMzLp3rtpFeb0QCwW0xQik q7sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684908311; x=1687500311; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KNLXI/ZXaW5BbdyJvbyW+B6lruy4aiQp89HdExaBNqE=; b=dIi8YRN/b6gAO2IeC4EUpO2OJyWVjxqZpkk654nEzsOySTb4KQc/vgUxvp5BUZ1B1u Df8Ei0HQpQaNci5kRPnIHyWGCsFL7XQPIlF1TB+9sCSH2TOLd0DW1uvrSn/WQqeOv0l2 evme7Pmz/R5elLcnIdYFZsY+m8rhivv0xwMG0a7qPHb9/tGf0+6091VpUmGDyAS0LZ7W +4iRG9Hb9CbmSFC4NEZXMesYj7vrSSHcCoX3yGjJIAYcGDROeV69+sceAU5m5rZ2FdrA 8mpEcJvo21g2BWPgyhmtqf/xUGY9Be+0RGmAQ/ebL9wkj6JbE9I3doLOX8tXdCe6vpo5 6Vhg== X-Gm-Message-State: AC+VfDysVSc1RiiUd6gNO5RKOVtLGVetsVjHlmLVLMqcrOo+8gANxWFO M14M/1UeyM5ccbW8XC3ibdo= X-Google-Smtp-Source: ACHHUZ4irL3h3o0BuihBytIxkgP7P3dYhPfYa99TuQfqazLCyVb5fcQLFaIev+2r8sVmMIute6392A== X-Received: by 2002:a05:6a00:2287:b0:63b:19e5:a9ec with SMTP id f7-20020a056a00228700b0063b19e5a9ecmr2053192pfe.33.1684908310708; Tue, 23 May 2023 23:05:10 -0700 (PDT) Received: from wheely.local0.net ([220.240.241.243]) by smtp.gmail.com with ESMTPSA id k25-20020aa792d9000000b00646e7d2b5a7sm6678426pfa.112.2023.05.23.23.05.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 23:05:10 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Peter Zijlstra , linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 2/2] lazy tlb: consolidate lazy tlb mm switching Date: Wed, 24 May 2023 16:04:55 +1000 Message-Id: <20230524060455.147699-2-npiggin@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230524060455.147699-1-npiggin@gmail.com> References: <20230524060455.147699-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E398120013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: a8zsrqfgg99owbnxgu97fgen3f3mb3q7 X-HE-Tag: 1684908311-778261 X-HE-Meta: U2FsdGVkX1+5tBZ432ZDGjBs36+FDGiJKiusdiX0Isu86NBD11uiu4zLFUhnBdg+OhCAj5aktNIgbNh9jddNhoq21PZxdTWxz6rNK5/sfk8xncYtVE+spoZ27FjbyplkW+x6nU4HNwXcfFZkb81/Av9iqcPpxzNAqYbXUPYOqhns7Hu8CLnWC/5fpQNpd9TjFhYjaemRknlfxgvYEuMLrBJYDhk/yBWBizgAGcT/TvRimvUO7kaVGDyTOn5d71faymk9MAatdPyikLfI7wE0fwiw0+rWpcBcq9viJcL7KqWbclUt9p0TYns17hJSlh086Ir9Rlekw7fPwmPt+BiHVSNcmCqNYhfO2isO51TI2RugtAzRQKoIxep85iCdVUpgdu2MiDTK6OCiK1VB/uGp2ciIZzry/AELN2VBbHRxcvv5bekEjrLkA7lC3LfCpL3/IQArKWJa9P/4YLkbQJxbtTocsxX9BAMaSvzfrTDsH3TQKPlN6D6KQDmKdRPVW4LPStmtQzxP9gIrpwjuGlA5bPQreBIjBxqGmiCKzT125mKvQced/exRGj/7iLERGa1IR3/9ZC70epbyeerqZcLNetEZlukCD5LFhUvke7g027ipFpSUl1x9Hy43Rgc6R4Yug+v3o/hiDgjwcJJFMKZTm1GQ57gsQz2+3VH0l2nwWwDP2V7qrFQjajV6/aVLmWhiBEaiNMnnc825SQskCIhFG6/VeKA3TOim3wiD+MhwLhn/bU5z/7O9H+tLY38q4rPtgoGP1r9CZCCSANBLqB209FpV0jH93TdJyTSrzcaBdA4IlUFp/8Pti/08kiB418jMJ2Wl+YOJ44M44moHlqp2JaZo+tjJ0Eag0ZYgrM05+M3wSVXDdEapKsBeKjGb+WT+675JZeL3u8ar6BCBqK5oUERG/o12MRd9+i3tDEp6Y5iZG0Pwcp2NX7EUveR2xwg9Pxo4VN4yT/BKb3CL6Te CSB8WPAb zRdO5mQLnMb//kkzZHovzD/lQipRziE1yCSvCBxTWkwS4NNLpAzfjpnKj3HQm7O/6wKyN7D8sj+ZNgLKdkDNjl7qwB7ezIvH+wUI4mt3Du9RPR1AnbRPlP/v6gwjq7WoBXAhCl5v9DGIEi07OFMrlJOm8EX9ZqbkP9iOJ7+QLicjxlybALmm8owpVXE5lQYmmigBOA9dYjkQmf3UkPoAbjbKHG/CuIAEjzvjdk9pF7aBzxPgkYGN0RtmRLy8xx6msn3vCs2QA5rH31nd8qHcH7CmYw1KuI7xi28q76xf9rWUK5OI5953vOzNox4RkZ1b5gMjhN3eeBYZyznpxLpHHSHjfZTAwYQMZZ8U2G8fQvCA+GfCTaGYj114OQxXqh4bX0xF3Qh1yIdRLi9YNd8HNXai8Gtfiav+vjzTLRFpdEokPNYx+pmItJ1bad+xGmZHGEbNz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Switching a kernel thread using a "lazy tlb mm" to init_mm is a relatively common sequence that is not quite trivial. Consolidate this into a function. This fixes a bug in do_shoot_lazy_tlb() for any arch that implements finish_arch_post_lock_switch(). None select MMU_LAZY_TLB_SHOOTDOWN at the moment. Fixes: 2655421ae69fa ("lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme") Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/book3s64/radix_tlb.c | 6 +---- include/linux/sched/task.h | 2 ++ kernel/fork.c | 7 ++---- kernel/sched/core.c | 34 ++++++++++++++++++++-------- 4 files changed, 29 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index ce804b7bf84e..90953cf9f648 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -795,12 +795,8 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) goto out; if (current->active_mm == mm) { - WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab_lazy_tlb(&init_mm); - current->active_mm = &init_mm; - switch_mm_irqs_off(mm, &init_mm, current); - mmdrop_lazy_tlb(mm); + kthread_end_lazy_tlb_mm(); } /* diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index 537cbf9a2ade..23693b94a09b 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -61,6 +61,8 @@ extern int lockdep_tasklist_lock_is_held(void); extern asmlinkage void schedule_tail(struct task_struct *prev); extern void init_idle(struct task_struct *idle, int cpu); +extern void kthread_end_lazy_tlb_mm(void); + extern int sched_fork(unsigned long clone_flags, struct task_struct *p); extern void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs); extern void sched_post_fork(struct task_struct *p); diff --git a/kernel/fork.c b/kernel/fork.c index ed4e01daccaa..8b005c2c7c3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -854,11 +854,8 @@ static void do_shoot_lazy_tlb(void *arg) { struct mm_struct *mm = arg; - if (current->active_mm == mm) { - WARN_ON_ONCE(current->mm); - current->active_mm = &init_mm; - switch_mm(mm, &init_mm, current); - } + if (current->active_mm == mm) + kthread_end_lazy_tlb_mm(); } static void cleanup_lazy_tlbs(struct mm_struct *mm) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index bc4ef1f3394b..71706df22b41 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5346,6 +5346,29 @@ context_switch(struct rq *rq, struct task_struct *prev, return finish_task_switch(prev); } +/* + * If this kthread has a user process's mm for its active_mm (aka lazy tlb mm) + * then switch away from it, to init_mm. Must not be called while using an + * mm with kthread_use_mm(). + */ +void kthread_end_lazy_tlb_mm(void) +{ + struct mm_struct *mm = current->active_mm; + + WARN_ON_ONCE(!irqs_disabled()); + + if (WARN_ON_ONCE(current->mm)) + return; /* Not a kthread or doing kthread_use_mm */ + + if (mm != &init_mm) { + mmgrab_lazy_tlb(&init_mm); + current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); + finish_arch_post_lock_switch(); + mmdrop_lazy_tlb(mm); + } +} + /* * nr_running and nr_context_switches: * @@ -9375,17 +9398,8 @@ void sched_setnuma(struct task_struct *p, int nid) */ void idle_task_prepare_exit(void) { - struct mm_struct *mm = current->active_mm; - WARN_ON(!irqs_disabled()); - - if (mm != &init_mm) { - mmgrab_lazy_tlb(&init_mm); - current->active_mm = &init_mm; - switch_mm_irqs_off(mm, &init_mm, current); - finish_arch_post_lock_switch(); - mmdrop_lazy_tlb(mm); - } + kthread_end_lazy_tlb_mm(); /* finish_cpu() will mmdrop the init_mm ref after this CPU stops */ }