From patchwork Mon Jan 22 01:00:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A9C2C47DD3 for ; Mon, 22 Jan 2024 02:47:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F2836B0071; Sun, 21 Jan 2024 21:47:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 153A06B0072; Sun, 21 Jan 2024 21:47:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0F536B0074; Sun, 21 Jan 2024 21:47:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DE80E6B0071 for ; Sun, 21 Jan 2024 21:47:41 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7E240C0176 for ; Mon, 22 Jan 2024 02:47:41 +0000 (UTC) X-FDA: 81705411522.26.7B226C4 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf04.hostedemail.com (Postfix) with ESMTP id B537240019 for ; Mon, 22 Jan 2024 02:47:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf04.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891660; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; b=gLbkFrve7zrP9YjpYZ+TinRYCJE0r1e1AdOBGmtj4Ytdelbz+vz7gLnO1vDeNGlHPOOH5e x+bYasdgvTHf7JscZaH0V2PpJGQcWpK3eBUFNsqAVaqyt3tyWctDdfLQnxhGQAFZa274GH lw+7onPzFdMRP+CEpSHRKKdAJku/n18= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf04.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891660; a=rsa-sha256; cv=none; b=Xjl+tmSj6RkEKZsJCTXW8bZyQV/jKxvraRE6kdP4YPgewdaKJSZItLadORfUbJ0MbOGjoo sTre6Ii67hDw3yxSKXp91VaMppU2v6YGoFrdA234OuqyWkFOjZL7CvG7ug7ubHmoqz37MJ af3gruJNQTjAWm0xIPeSN84Liz88WWM= X-AuditID: a67dfc5b-d85ff70000001748-6d-65adbe43e950 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 1/8] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 22 Jan 2024 10:00:33 +0900 Message-Id: <20240122010040.74346-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnka7zvrWpBgunSVnMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+P27ZdMBav4Kpb8aGdrYJzG08XIwSEhYCJxZXNgFyMnmPnnxXt2EJtNQF3i xo2fzCC2iICZxMHWP0BxLg5mgY9MEqu/d7CAJIQFfCQmHm5mApnDIqAq8XmTE0iYV8BU4uDG 14wQM+UlVm84ADaHE2jOlF9dTCC2EFDN3A87GUFmSgi0s0ts7p3PBtEgKXFwxQ2WCYy8CxgZ VjEKZeaV5SZm5pjoZVTmZVboJefnbmIERsOy2j/ROxg/XQg+xCjAwajEw+vAvjZViDWxrLgy 9xCjBAezkggvv+qqVCHelMTKqtSi/Pii0pzU4kOM0hwsSuK8Rt/KU4QE0hNLUrNTUwtSi2Cy TBycUg2Mq44Xf1h3iJtfq3Lpgi+TuS5bvTrVez/0xVxXZn9fs1Mzzv/aUrjt2yStn3X5orzP BL0lxf6aXrm8XVrZQ8pIRqBwm+OlZKtD53yk1Ix05re7vZn+I3FPXNqsac+Vnlgmphrk6C4L te098muTMteG06GvGC9cSrfviLp4uVT27qzDZktdhJ8cU2Ipzkg01GIuKk4EAPEP+0yCAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwckZ/BZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgybt9+yVSwiq9iyY92tgbGaTxdjJwcEgImEn9evGcHsdkE 1CVu3PjJDGKLCJhJHGz9AxTn4mAW+Mgksfp7BwtIQljAR2Li4WamLkYODhYBVYnPm5xAwrwC phIHN75mhJgpL7F6wwGwOZxAc6b86mICsYWAauZ+2Mk4gZFrASPDKkaRzLyy3MTMHFO94uyM yrzMCr3k/NxNjMDgXlb7Z+IOxi+X3Q8xCnAwKvHwOrCvTRViTSwrrsw9xCjBwawkwsuvuipV iDclsbIqtSg/vqg0J7X4EKM0B4uSOK9XeGqCkEB6YklqdmpqQWoRTJaJg1OqgTHftnX2xfx3 FenT6ut/s2VKcJ9/P0X5S+rcc488mncab3jINtmTL5bhx+flJ8PMprJJv/2+6e6f40XNjtwH TiRp7hY0PyrlH2Vb6lzZ8G+n87uG50WGi89xLkqT+PEk5VbsYR2ObMHTm12uP57CJz5R+Hp0 X98Bu3C/2eb+b7/P+vq2x3/ltMVKLMUZiYZazEXFiQAKYOYmagIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: B537240019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kd65e4tbgk886ssj5e456tdawwswwg7b X-HE-Tag: 1705891659-890021 X-HE-Meta: U2FsdGVkX1+PlD4PBvmvvcUweHeU8pdTqsYsXix9iWrNAO14QTJ16CFcrph5WpjWCI7ZQxQst+p+7WmosqYTBKj4HIoKxTSo67bxShmTEl83R6Fb63YvKL6SE/8a7vfHVgxHE3wm1KT8fC7OSKMiJRzcMhyZecPxKlNRoNUMe35BHmCbAVPIgCTwbW2tFbEqaNq9Qaht4v2H9uiNzL5fDkrk0zA34/efkQcBZVyvChsW/YNK1Rq12I3awYaD2ZbnS+cjne9WhtVQznJeyvwYrySgem0JEKLh6Gw9GdYFDJcrV7T+tbeterqI+Ip6kj+is+kGz9mYYJd92yoa1qzY8b9BMNMdsJ+zSrNu4pboNTfPuNa+1FW4+Fd7ogqRKdKqpIlU+bZRYmI7J9++zK51UOIrJLDo5EXrVL+UrCkWxBPAcEI4SPT9clZy7CQNcQQI1bioEemfb5Yr9DkKnqMjXdACy19lFlAE/NuQV2LdBCSAUP1mvIMtb8It38TlqOkS1d4OHGe3HPkq3xEaTAxRmK7zFAUMKGEZiaxBLm9Ob3E2dgbWxb8lyargEcwj6j7qmyxsMNNe8i9pfhyYiqZ7E81bRzGkxtP6SX5d/6FzmvEqJaRyfJK8GtnG5dR91MjKaf42qIy+dafmy8I0Kl7mAn9XGyQ6xm3IRxdwk3h/shNdsYj66nV+AzOQtTczpZK3YcNLvgsCTcZ1664L8dXAJQ7CNTiyRT8Y8BdHPWpEMoLTmQPq44mz0FwSKXsiQgMRSZ9HeKg/6IrXpvRthzWK808nkDa22FMCMe+bVYsvc2YhFxYbirGPhsSZuNLVuO2C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Mon Jan 22 01:00:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 945DFC47DD3 for ; Mon, 22 Jan 2024 02:47:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21A976B006E; Sun, 21 Jan 2024 21:47:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A1D46B0072; Sun, 21 Jan 2024 21:47:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 069876B0074; Sun, 21 Jan 2024 21:47:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EC3EB6B006E for ; Sun, 21 Jan 2024 21:47:55 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BDE531A0724 for ; Mon, 22 Jan 2024 02:47:55 +0000 (UTC) X-FDA: 81705412110.08.0084A8C Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id E5D35C000A for ; Mon, 22 Jan 2024 02:47:53 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=23+JjFD9uRPbyMWGqJTLiQYjg6Gkh8EQ/TB1jpahIpQ=; b=A1fr3++GDAFUIjb0F/GfxVp2CeKpbcu+1kprG+VbH69p+gFmpouxkY732Dzh2zfZZVwgOi L5WAMkYjLWay9IXwXcDlWJR6GGo6oHZJGuPq8v+56WGz0ToGPlUglJcw/oRB504X7X+TPl cJPQnEdu8gCBGpNIHYJErEMVOdKsxfU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891674; a=rsa-sha256; cv=none; b=J+yGaeMBt5CIqTmprv4eovZf4tPzT3qf6+My4WBLncLruPrt5dkbAlh/YNJhpP6y4GjAfW 3rM2Bxm6T/udaMe5cWKwey9Ly4twTbY2Am42NR277mBz3hmPT5qM8NSapt5pvQ8HcLOkfv rbdri7JFIn9NF/CoeMHWbCd7dxoKUNA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d85ff70000001748-73-65adbe43a43e From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 2/8] arm64: tlbflush: Add APIs manipulating tlb batch's arch data Date: Mon, 22 Jan 2024 10:00:34 +0900 Message-Id: <20240122010040.74346-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnoa7zvrWpBot+KFjMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK2NP30P2gtlcFZvet7M1MO7l6GLk5JAQMJH4/3w2UxcjB5h9oSkYJMwmoC5x 48ZPZhBbRMBM4mDrH/YuRi4OZoGPTBKrv3ewgNQLC4RIvPnoC1LDIqAqcW3zRlYQm1fAVKKp +wMLxHh5idUbDoDN4QSaM+VXFxOILQRUM/fDTkaQmRIC7ewSk8/PZ4RokJQ4uOIGywRG3gWM DKsYhTLzynITM3NM9DIq8zIr9JLzczcxAqNhWe2f6B2Mny4EH2IU4GBU4uF1YF+bKsSaWFZc mXuIUYKDWUmEl191VaoQb0piZVVqUX58UWlOavEhRmkOFiVxXqNv5SlCAumJJanZqakFqUUw WSYOTqkGRkP5TabCu9ym5C+P2iPAftegtfjbl8jiiJhDeZF60SxHv/c/DV7xW+BnZ0PtU/1o htLOvXyqUrl1d2w35GsINN601w7UmN8X5fyuXHLhB/t9S34XPqsoYf4+ccXZ5JMRL/TWv1/T /eDQ0xVib3/9dzTmUKjvDD+1/A9v44zeQNs5/6crVrRcUWIpzkg01GIuKk4EAN1FibeCAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwbvbIhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgy9vQ9ZC+YzVWx6X07WwPjXo4uRg4OCQETiQtNwV2MnBxs AuoSN278ZAaxRQTMJA62/mHvYuTiYBb4yCSx+nsHC0i9sECIxJuPviA1LAKqEtc2b2QFsXkF TCWauj+wgNgSAvISqzccAJvDCTRnyq8uJhBbCKhm7oedjBMYuRYwMqxiFMnMK8tNzMwx1SvO zqjMy6zQS87P3cQIDO1ltX8m7mD8ctn9EKMAB6MSD68D+9pUIdbEsuLK3EOMEhzMSiK8/Kqr UoV4UxIrq1KL8uOLSnNSiw8xSnOwKInzeoWnJggJpCeWpGanphakFsFkmTg4pRoYnW+/+v/l VELxA0WXSX94RM0VWR/O01p7U5BHK3Nv956JuY017wXyv57qFsq7JaDwR2BWZYSrVOXN7WyT EpZ+VXqQv1hvjezfJQkMMUrC/Tv07/GLTz34fdMe//VH5TjcXybdbQlOF3UyrNERs5O8Zmt2 82bz5IjK2Ya3VX/XbgiJ4r/XLJikxFKckWioxVxUnAgAreUf/GkCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E5D35C000A X-Rspam-User: X-Stat-Signature: mmgkoei5srtfobifiwfhkcuggb11jgf5 X-Rspamd-Server: rspam03 X-HE-Tag: 1705891673-322767 X-HE-Meta: U2FsdGVkX19w55EE+TqGjz6Rw0woYEqHl9/fcwuURB21uhpxYevbo5l2Ujgw0ihGNWQUqtqvHJZYMIzN0L30p7H6GuWJCpLAM9jGIrCa4ONbjUSmr7PfEC3BAsL8dNyfvOPkutxOSfxiFDKtl+4EZrqeAAzcbdTiHVIx++3Wg6VTjAXlA0e6ZlJLb9g4DO78UP9D4v95u+6wdH1S0ROXLtSKALj0OWGIVIqQz0owMjvn0R6IqeRjxeI1ZdNPZ8JDEvcj8o1ygUCeyuk8MaRSoYWEihhSysqvzjHRf9X6PiQDDwtIOroTmLl+IvxDYVLkEFyVgjHI75h+bnoyXuwLobE51HJOuUeiMfG2k6pO+N2HcTDWydnt4a0EB9PHgYr1OI3uE8pCt/9jgFY+6ZLhVlPHyuu45hWpwHgyyr0StyDFF+hsMVpkwylcQezMqL8h5gA9rodGcvPQTE8UW9UKbrWFvmgzdQ0eZMJxLy2Z+D2rvEISFVe73NtnXoYV5qNoqiWS+gFZk3PVT/YeGj6tJytoId93YIT10QUB6ovHf7rSmaDgnm4+XGccijwoPTwyllDxSDMgUaK9eIdsxBKcG2PXFLDV6gQZfU7QkU19z03d1RY0WdQwoutwEOmk/bIS4yR5vZMrG4B6fmNwxIf6hGnth5cpSFKSDhnbu52IJ4S0ia6KEDy7b9WsTIgzM9jM81Bxb4ijRC2cNNyUc9MNQLJCImMHmBkBPkgzn3rS794/WLoEHx/AQ3a5AeS1m+jQAfVF1Q52VrNpKSljfIx4WAEGXhdCyZ+Ggp0bgYiC5XWewPbnUmzfRbUl81ASGXGp5woNpu8P+VawVydNrIE4cnW6HJZEt6P2TY91IiP+8CLdfBo9kAysg2jHjiZxgrpr0pfjC6Z03cCetDqkohtaSZ0TTyRg8W11fk+NK8ok0MLdKOWf5SwaSqhmIDohhdiyA8k0uiVVm6uOqS1jMBI EQIP5Ri8 kTMaSM0IsZB1NUWfkPiSEe0KAVdwR9FdIm/nTidqQVUGqlAYJMyO0DUAsMjTBj41rAWxmF1v7pohAptb4sSH7rRCk31ELFRRBXTW3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing with it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..4f2094843e7a 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -328,6 +328,25 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) dsb(ish); } +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ + + return false; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. From patchwork Mon Jan 22 01:00:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1582C47422 for ; Mon, 22 Jan 2024 02:48:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B1226B0071; Sun, 21 Jan 2024 21:48:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 439F66B0072; Sun, 21 Jan 2024 21:48:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DAD66B0074; Sun, 21 Jan 2024 21:48:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1C36C6B0071 for ; Sun, 21 Jan 2024 21:48:11 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B5AB9A03F2 for ; Mon, 22 Jan 2024 02:48:10 +0000 (UTC) X-FDA: 81705412740.05.5BD3121 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id DBE6B16001D for ; Mon, 22 Jan 2024 02:48:08 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; b=76nrReElZE0tzwki3e+3Arz9LpTMrbYfoL9nIkGX83UQiCXRFjNPnUsuaMcLbsc1ACuyC2 eWLY5J7qxarnKvWtSI7zlMGdlra/+XG46vtqKSuukVSud7NsX8tRNWgq06tPub13OLiptW 2UYZDvLE23VNC4NgWbiH/h+NFjoNwAo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891689; a=rsa-sha256; cv=none; b=fpj7iVkmx8YAP0Jgbn6tELg6PNTNcVWT+ogbsC9O5ES6mLWdTswTYHUhUGzL00TJNdf6QU LkIMbkNxJZR4ZpVjy6FJPFcWyTVDV5enurnxE1eR08Kuv4MKfBMDbP9dtzcW/3VkZ/w6Pa pkIOu+8Q8De9/YThYoi7WFnKHwCPXXs= X-AuditID: a67dfc5b-d85ff70000001748-77-65adbe43af5d From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 3/8] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Mon, 22 Jan 2024 10:00:35 +0900 Message-Id: <20240122010040.74346-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRmVeSWpSXmKPExsXC9ZZnka7zvrWpBre+KlnMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PrPKeC49IVHz7tYGxgfCDWxcjJISFgItG+6AVjFyMHmN01JQEkzCagLnHj xk9mEFtEwEziYOsf9i5GLg5mgY9MEqu/d7CAJIQFIiWuvPkMVsQioCqxc9ljZpA5vAKmEvda uCDGy0us3nAArIQTaM6UX11MILYQUMncDzsZQWZKCDSzS2zuP8MO0SApcXDFDZYJjLwLGBlW MQpl5pXlJmbmmOhlVOZlVugl5+duYgTGwrLaP9E7GD9dCD7EKMDBqMTD68C+NlWINbGsuDL3 EKMEB7OSCC+/6qpUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJM HJxSDYyLj7dlnOONTfeyrlUNXfxgcpBxyxJBcdvn18oaUntkTq8vSIwIYv6V7yf7dVvzum8/ 2+PileyNgiZyqjdF77DjXrJ9362pT27oXN5Y1fgh/P5sp+3LFDaUn9nzrVVn2ccZi59ZTGs0 0AudNd9Jew9nWKKPxZQah6D1h5dGSswR4H/1y7b5x0olluKMREMt5qLiRAClHd/2gQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrKLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwf5v4hZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyvs5zKjguXfHh0w7GBsYHYl2MHBwSAiYSXVMSuhg5OdgE 1CVu3PjJDGKLCJhJHGz9w97FyMXBLPCRSWL19w4WkISwQKTElTefwYpYBFQldi57zAwyh1fA VOJeCxdIWEJAXmL1hgNgJZxAc6b86mICsYWASuZ+2Mk4gZFrASPDKkaRzLyy3MTMHFO94uyM yrzMCr3k/NxNjMDAXlb7Z+IOxi+X3Q8xCnAwKvHwOrCvTRViTSwrrsw9xCjBwawkwsuvuipV iDclsbIqtSg/vqg0J7X4EKM0B4uSOK9XeGqCkEB6YklqdmpqQWoRTJaJg1OqgXH1b4H7U2L1 xfrqGgT3SCwsabBvOXDW3zi6j/3ttJuH3uSufiqcwSStlBP25lvu5Ta9q+vPvLeLipD4fsnp 6nOuIyXL65cZX2ROL/G57i/ibluhZda/wb9jpd4KHocnz1JEqj8ZGbNdFW9O9cuurvk0Ia7E uW0q+1Y2iYLWfiP5JeLsbHamSizFGYmGWsxFxYkAE0ppnGgCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: DBE6B16001D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8p4u899y35rc7t3woswumtfm8epwqmhe X-HE-Tag: 1705891688-861057 X-HE-Meta: U2FsdGVkX19X7gYoCClV70zdWjvm85y63OwbaocVrV6qEeDs+GMWleayycvosQCdtRflrYjiTzZ3MKaJpxyjoKVn8MIghZZBrolPo/wXLm1yiWL0gMscsw1D4hMbtUxIW7QBL+ibveqej2SlIvcNRpOB4wVHIsi7CfBHlKTr7MpupYaQzfOcoYWy2yUqJl9l/rakYbQs0Qmo+aQ2GcNGRjMH2j70bV6wtZMoLYsmDOfsajRP5pT948CcNyf3ff2i7UOkY6FTRf0nzbMdHD6K+HVgXZmBk+JicwW/toog4NCwa+dptIpyK9tH8AaC+bHaSukrikRHvpc7ueDNSBU/KRH2GjB7m6rtpl6uyVVWj1FE7Xk0+UdA4+hYWZjFvudKnY9+p6ZUrNmMzJN/UyPKn8T0dvprRM5BBOBeGmIw+9slckZZyudMrE7VBY5XnnWToWyAAqDcqQJnFwp1tEoBIIuGb6Youv19dmkSEZjeJNTYYbdR+bS48yZ0Bq5vEsemuJZwxemQ7BBLidQAfkiP/8cXN5AiZyhXHTudqChCWmo32qiV1H/SYGF4fHvhgaWy148ouWMDKzjsoNxAWAB0GXChlZuv3+8CZUcw94hM4gj6gnUCmC11LdH6UGYBUveZ5N8i6wtWdRNDL9mFjllGguBYxURvDST7jIzy3H1O12U8ZY32LajE8WDOy/xvP/DDbzf/jI2Wdk9DX63ZzN3uFVsw00gjy8ur4Lxm+Zm8jRfDcEMTtXrRxfzsOXRO9KA179yjg0iECVLmxfyAp0fDnDRr4IYB4Te83SYEFAlx9uz9s9nmD2CwdqegFl9BsrvYUSbfLsIO+UtRDsJjokgRMJyBu8meDm0yPiyEN9KFsirv4s7yWvqPxmCK5vxJ7Bz+bpanWlgz15atNoCPoUAy26GIbHC853rU6k/ZV7EN+3p/8Z01xprILtAQgrnR4uffmDUHHQ9gZQCmfrHhXDn 2rg7eIB0 2kdlC4eah5dXgSdnGWje+JzmebI1hi2GT7LkFAroVlCp8Vx7ZfcTTvSFjyFNpt04bOF/75Yd75v5Tdx0r/QeALog5H3ZdDiTUC1mZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Mon Jan 22 01:00:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 399A1C47422 for ; Mon, 22 Jan 2024 02:48:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8F786B0072; Sun, 21 Jan 2024 21:48:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF1356B0074; Sun, 21 Jan 2024 21:48:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943F66B0078; Sun, 21 Jan 2024 21:48:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 867FA6B0072 for ; Sun, 21 Jan 2024 21:48:13 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 482311206FD for ; Mon, 22 Jan 2024 02:48:13 +0000 (UTC) X-FDA: 81705412866.05.15C2E1B Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id 4D2C816000D for ; Mon, 22 Jan 2024 02:48:11 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=JD5mtfEWkRjuaUV9XVtob8Xf345yQ1B8wbzU19SAuqU=; b=2dDc1iYgklGOiDw6+5kTDGY93/RHbk5iW4oagm7FuzEqP1Gky5OJCzj5Pt7Mrnwlg+nzxb U0jufltk4Cl1vVLsyuva6OnT3VkoZzpC3EVXN1IyKRQBoQpipm1sRqIEOHpjJ/ctn6apLP /3EoBHRHukaCkLdXH+dN7oEGJstmMGU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891691; a=rsa-sha256; cv=none; b=N9weTeFbCnQhPRrQN8jOA5Su29cmvglt56lhVQLdM43hkkPybcejs1vL+bSB/a7mcR36r6 iP5YjItqDxU+DzYuUppdBTUFjk9Ghk4eqQRS9OzkjAqZPu+lmdSpUAbWpjRGlbpD+K5qGK vj28zrzg9I1psJ3Z08Bg+bPc8EJYOWo= X-AuditID: a67dfc5b-d85ff70000001748-7f-65adbe43fac1 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 4/8] x86/tlb, mm/rmap: Separate arch_tlbbatch_clear() out of arch_tlbbatch_flush() Date: Mon, 22 Jan 2024 10:00:36 +0900 Message-Id: <20240122010040.74346-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnka7zvrWpBtsnaljMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PonXtsBXM4K87/msrUwHiJvYuRg0NCwERiws+yLkZOMPPa0+WMIDabgLrE jRs/mUFsEQEziYOtf4DKuTiYBT4ySaz+3sECkhAWSJU49eo7WAOLgKrE1nMPwBp4BUwlli1+ yggxVF5i9YYDYHFOoEFTfnUxgdhCQDVzP+xkBBkqIdDMLvHz7lJWiAZJiYMrbrBMYORdwMiw ilEoM68sNzEzx0QvozIvs0IvOT93EyMwHpbV/onewfjpQvAhRgEORiUeXgf2talCrIllxZW5 hxglOJiVRHj5VVelCvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl 4uCUamDsXX6r+OLRd4dfr/vpuvrC4/vaq8wdzyqrcN7883LRhBntXtPMMtsbtsjd2vb2qYBQ BOuh3yZcD5z6blQF3XNKlRA65Sp2OOPWwe+GV/1z/R65/yysrbw0gaV57XrGxlTfZ+rz86S/ Khur/TEIqL1bMTHm2IZz7Mdts+LY595N197J+T30hMdfJZbijERDLeai4kQAW6vnxIMCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwaaJshZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyjt65x1Ywh7Pi/K+pTA2Ml9i7GDk5JARMJK49Xc4IYrMJ qEvcuPGTGcQWETCTONj6B6iGi4NZ4COTxOrvHSwgCWGBVIlTr76DNbAIqEpsPfcArIFXwFRi 2eKnjBBD5SVWbzgAFucEGjTlVxcTiC0EVDP3w07GCYxcCxgZVjGKZOaV5SZm5pjqFWdnVOZl Vugl5+duYgSG97LaPxN3MH657H6IUYCDUYmH14F9baoQa2JZcWXuIUYJDmYlEV5+1VWpQrwp iZVVqUX58UWlOanFhxilOViUxHm9wlMThATSE0tSs1NTC1KLYLJMHJxSDYwlu0V7RCOeRefs dO2/Kfs9avae3+sdFe8I7H7rq3adq4XhBoO2X8KkCa5mT+eEHr19tSLW8eyJNRn9d+UnsWh5 /fE16lfYLNfHI/ZHi/+30CqhO1c3crmc3FS6PsSS32y6+7UFl7jTX7hvvjNP5Vq8WdGl0pWS oq4PAu3VGs2m/DBev+ggV5USS3FGoqEWc1FxIgBG/k1hawIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 4D2C816000D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8cnh39u9mnktsi58f37xr5ahhjjqcdrw X-HE-Tag: 1705891691-379921 X-HE-Meta: U2FsdGVkX18uXBE9yWVwBH2E42UgefsshhRq9WgogiJCgwlm6jZS5d5REhFBimUWckXDqDROkI3/BagwISlclHzCfe6g0hiILqGC0a7LK5ixmW0SrZ58WC8MEfyYifTbyWXEzTU1uggS0R0enUHW95B7HO/Zl9J+LaunYCa6eGbiPnl5a3jN/leebhgLjtVJ2ZjpBnjSiM/PDXhPFW6nUT5C2Yt/+Ut+gTQVeQtcqBrGHrdN2z4gKeNPrzCYfmgS6tZ0zT+cKtAZ1uCfIYTtWiq19VxSy6opHvaxgdh8TXCJDCZgfGbXaMOubebmz+OUGWTElRlQT+JttM5Wg3YC2j1i47OJgNg21/5+Z5/Kertaqfz/OHGwstEyUmi+aNFsmqfk0NMfrRX3gPRn/QKjwtvdsfZ1yxqVtUpAGxKViNbNpcRTtsmh0/qQ9jNjOch/Q1TbKAXyuh1U+rUac6OgGe6t1LpPZEhg3thL39ygsod2B87lW+0TKE/o8h4gRDLGFmbgT6g8i9Q6WW+4y+2y6CPlPc+skggxiY7dKOdLA9gnjOWx+3Nm7IGnUEqBq3zkZe9DX83Wzo1PhusLtxrnkLuYnoZGYgYwo1WHxiU0URhbmg8CK1x4LEtgapfuoz28n/GY+Eh6A//eDCKvtuXFmVX+x4WcL98OZ4CfR7Vx42yE3msgaJpw0Uqut2cQZux8AbEbfWgDh9h+6zEHn8IGcwT1/tZS+q04DnLH1eNFhgp0dH/rW47jX4RPildzwdvheuieVUs3/4ms2MGzHcBWLeiHBZQyM4ie4Tokm0pfLPR4x/OXlxwJXSPC+0poMtDlqZiLlKdFK6hj0OEi5QRLXHnNzsoq/wJGODpb0OU5DNqKYg9y4+U8FPfg5HJsA3X8Nef2DUJIVfLZoKq6vwpYNO5jIv4Ydp4io5ftxcxTzxvtidOQIf84gUYuT9b3566RdZjbzRHM99yyvfzn3OJ yps8uzuy 7vcqUPexzoh4U14N6NbIrorUsUalggFZ3/Fw2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to avoid redundant TLB flushes by manipulating tlb batch's arch data after arch_tlbbatch_flush(). However, we cannot because the data is getting cleared inside arch_tlbbatch_flush(). So separated the part clearing the tlb batch's arch data out of arch_tlbbatch_flush(). Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 453ea95b667d..941f41df02f3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1268,8 +1268,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_flush_tlb_info(); put_cpu(); } diff --git a/mm/rmap.c b/mm/rmap.c index da36f23ff7b0..b484d659d0c1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -643,6 +643,7 @@ void try_to_unmap_flush(void) return; arch_tlbbatch_flush(&tlb_ubc->arch); + arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; } From patchwork Mon Jan 22 01:00:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E7C2C47422 for ; Mon, 22 Jan 2024 02:48:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C55856B0074; Sun, 21 Jan 2024 21:48:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B693B6B0078; Sun, 21 Jan 2024 21:48:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96C9A6B007B; Sun, 21 Jan 2024 21:48:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8351C6B0074 for ; Sun, 21 Jan 2024 21:48:15 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5CAB780746 for ; Mon, 22 Jan 2024 02:48:15 +0000 (UTC) X-FDA: 81705412950.10.A525843 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id 797EF16000E for ; Mon, 22 Jan 2024 02:48:13 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=hj/OU3wk3OzsqULfJnD2mvtgwTJyJb/ndDOd9gzDe6E=; b=sHC+tB31LNQ79HX2rgU/phzo482dXqtb4/OUOdD5bJPBHmB4qc/LXcFlI1wX9spel4cLis MsveMw/pNdEWuVoX/TxXgk72TFv3hHXiIPLGC3KCmbe/5h5ck2r+ZGAtmocTj85Eh1OIfp /LyZ5l+nie3+pORHgAnZjDSKlCauKGk= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891693; a=rsa-sha256; cv=none; b=fWdW9hGAByuDNDERNe0UHdPZG4ADMkZUJGPXooRwBoA4rA84LblJzJeqzOrfnrU1uBOYLu dIkdC1hXar+0yNYHdROPJ26ifH2nLnrJ6QsItWvdjrTTbiexK2iNK7iMLMsOXFbvSqAOY3 +nGXy0athrOWBxBlDIjN7q5GcJf76k8= X-AuditID: a67dfc5b-d85ff70000001748-84-65adbe43617b From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 5/8] mm: Separate move/undo doing on folio list from migrate_pages_batch() Date: Mon, 22 Jan 2024 10:00:37 +0900 Message-Id: <20240122010040.74346-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnoa7zvrWpBvNv61vMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PG+7vsBf06FYvmPGRvYFyp3MXIySEhYCJx+88+dhj7wvn5zCA2m4C6xI0b P8FsEQEziYOtf4BquDiYBT4ySaz+3sECkhAWiJVYtbCLFcRmEVCVOLv/EZjNK2AqcWHaJhaI ofISqzccABvECTRoyq8uJhBbCKhm7oedjCBDJQSa2SX6Pq9ihGiQlDi44gbLBEbeBYwMqxiF MvPKchMzc0z0MirzMiv0kvNzNzECI2JZ7Z/oHYyfLgQfYhTgYFTi4XVgX5sqxJpYVlyZe4hR goNZSYSXX3VVqhBvSmJlVWpRfnxRaU5q8SFGaQ4WJXFeo2/lKUIC6YklqdmpqQWpRTBZJg5O qQbGQiO9n3/fssqtcjRRZ5bLFnoyNcc53bTYzXHfqo1T118vOyGpGs/E6aFZNWnb8qa0Yxt6 ljamP5p1MeoV79IHnRdjg2cYbWQ5euxlTPmr5Y4iedFL/qsdeO255Mp0XrWtMxR8lPpevFA4 e2xl/oRHnzUvFTNmHjn7ODOrVmXT50utF+8ErVB4qcRSnJFoqMVcVJwIAG/OZi+EAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwb0zyhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgybry/y17Qr1OxaM5D9gbGlcpdjJwcEgImEhfOz2cGsdkE 1CVu3PgJZosImEkcbP3D3sXIxcEs8JFJYvX3DhaQhLBArMSqhV2sIDaLgKrE2f2PwGxeAVOJ C9M2sUAMlZdYveEA2CBOoEFTfnUxgdhCQDVzP+xknMDItYCRYRWjSGZeWW5iZo6pXnF2RmVe ZoVecn7uJkZgeC+r/TNxB+OXy+6HGAU4GJV4eB3Y16YKsSaWFVfmHmKU4GBWEuHlV12VKsSb klhZlVqUH19UmpNafIhRmoNFSZzXKzw1QUggPbEkNTs1tSC1CCbLxMEp1cB47ckGe8Z5rqdO b6u78sza8ZXU6/fhrgwTfFvZbzQ2zj76ZG3tYqE1Yq8l1bRuLQ1JWcijw922cfFj/crVak9v f8+V5vc13FRVzFhjeqQyu1vSQ47rTb1S+wwLj6k8NizivzR2/lwse8THzIxHcm3JjqyC8qjF DSrHI8sPLVM1vmJwcmdmP78SS3FGoqEWc1FxIgDyA45IawIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 797EF16000E X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 46zaw93ymwh9ck9n178fyts6i8mifjrk X-HE-Tag: 1705891693-341465 X-HE-Meta: U2FsdGVkX1/82XhGfnUrXuNVI2Gvot+DqhVxQ3dA4cZ9BrKkOdIJ1MYxcxMIuHlXvtyW/XH0aDDkKA2wlEytLSRCtwFMiCdbgA9y5o29ef0WaU2/R5jrCkptU6VvxodRLGDVfDcIkS3LjepPuoFiKVZE3P6TzUYdzRIlnYAEUECmZ8dQapM5QzfLY7JiR3q2LcO9mvNcLtkel/YNry3Utggk1lUsGwSdAQ8r5Madk7WYF8OzxNIp+2NTbDkU7nZh4TJ8o/saTEayKCNZJFjy9OwhaAU9Hq2LERWP9lcYfJQaVIL8CiRAxg2VsD4FqCCjHdwG8oMmYSxdr6VkuHn0H3kp4kqK/RE56svWTPxjR3MWyTa6ySGOy2GluH7Wkj2rpXYsdpVvVg47af8Lx4YXJy8hJ7RpQRTS085kVtXbJ7VKcBWNuSgjadIX5cHE/n5d/W7T/QydrEx/ObTRYbUgG4NDKu7pDeTfUPlkROSLn8EePgLjc8YYLLETV5VwZcHu3DGt0WMr/AWXqyTORM8QRs/2JeOcet5EYu2Cl7+eOqObcCegVT9edXJPA/QUsiLmfahqux3kSpB4ULRGoQzbCCJPUZCUP3V+IGwaPy6XjMF+zfZ5TVVdGadnnv/PmrEu4DOlDt9Sodphqfhb6qouQ0nYQzG8LkHXahT3yhWu6QrluIGuBWoZNsYTE0cdTN9Wf+rh71IsgXDl1W1T7F1GtfMflgVMDpkMKuwhSVG4mMsH3DwQ2CADUEbzlYoPPlIIVrPP18UTa6Jc8FJQDyz1JGnRs1kcCiqiNWVfFOVUT0TKJEdkhycIe3QNDpbBKPta99X9A+8Aod/YOn+hORcn6CPh0BSIPwzAKR73FN0kCs083xdEzeIaT2iGfH1W4QmpOwMXvZRaGzyUgsNkZ1B3JOWMf+ebG6aH2FDkmoRV4J2YE5ucda46D7Vvuo/n1ABZ7tv9glDv5p6FFFXPncF Oy7v5HBz 2reajEgsKqp3uOVoAYE04iWmeKAdn/gJtUzpu0TVHOobID3b+7n0KT++DL5X5ON7emyDp4Ux4SPhTV+D3s/7iTBqJK/qyu3kl+Q8qdaAJlxRnbjxWrN0CjVK1Xpjf5rXUKcGH9sHpsjWfwesVoqZPXUZ4Rg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to use separate folio lists for its own handling at migration. Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 397f2a6e34cb..bbe1ecef4956 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1611,6 +1611,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1633,7 +1708,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1769,42 +1844,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1813,20 +1857,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; } From patchwork Mon Jan 22 01:00:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2902C47422 for ; Mon, 22 Jan 2024 02:48:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 610F86B0078; Sun, 21 Jan 2024 21:48:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 59A116B007B; Sun, 21 Jan 2024 21:48:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 413BA8D0001; Sun, 21 Jan 2024 21:48:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2C8B66B0078 for ; Sun, 21 Jan 2024 21:48:26 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BD8F31208C3 for ; Mon, 22 Jan 2024 02:48:25 +0000 (UTC) X-FDA: 81705413370.04.A26D788 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf29.hostedemail.com (Postfix) with ESMTP id D9DA7120025 for ; Mon, 22 Jan 2024 02:48:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891704; a=rsa-sha256; cv=none; b=P6anPqN4KI1y2xul1oZXurkkl9c7h7+LGTRtFVbegH7klIA4nhNjUL746Ff3O6H3pAL9/9 cnjahnqvD0FhkFLY4BSLKuuc3YLD2tBLFxsD46sw3xaQrZhlFBKYHNAdmAa/5tHY0KM22h SrJ3TjTJI6KFU9Jl7CQEy05BOtRor+8= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891704; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=xyww7pMUuLyPBfG9ZuK6U6g9ZtqJWjn+tCNK2s25zA8=; b=5CTGxUahgES6A/e8JvcMnC5J94yFIrxMbb+f1T0MvlSS4nEey8jdWnrncDR9y25IeIUlCf OHMmsI9gL6dVEdI3DMjv1Susbhufc9Dod+EbO+3onLAwXZ4O+ELBiF8ilLpip9ji5hUoah 3nMKhBlGNEeA6lxI1eLkTaXcxJ8VIpc= X-AuditID: a67dfc5b-d85ff70000001748-8a-65adbe437328 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 22 Jan 2024 10:00:38 +0900 Message-Id: <20240122010040.74346-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrLLMWRmVeSWpSXmKPExsXC9ZZnka7zvrWpBtc2G1rMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK6PjOlvBH6mKw8d3Mzcw/hDtYuTgkBAwkdhxjAPG3N4s2MXIycEmoC5x48ZP ZhBbRMBM4mDrH/YuRi4OZoGPTBKrv3ewgCSEBSIkpmxfxA5iswioSjRMfAkW5xUwlTi//QUj iC0hIC+xesMBsEGcQIOm/OpiArGFgGrmftgJVdPOLvFmvz+ELSlxcMUNlgmMvAsYGVYxCmXm leUmZuaY6GVU5mVW6CXn525iBEbCsto/0TsYP10IPsQowMGoxMPrwL42VYg1say4MvcQowQH s5IIL7/qqlQh3pTEyqrUovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwcnFIN jFPjFoYkO/Srujx9E/MstvSd0JHVn5Yxz61ROWWgqS9hsud8W57y6arTa96rLrC/+m37qjid 5NYEs7I9DSeKudKLs3cfd9V7wiH55kvB4s0Mx66JLL0iGGZpWRRra61/8cAdNakXp2/f9vCO ym3bIhqsc9HpwW251btaprjeO3/w4i+N8y/TFyuxFGckGmoxFxUnAgBo49/vgAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwd+FahZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyOq6zFfyRqjh8fDdzA+MP0S5GDg4JAROJ7c2CXYycHGwC 6hI3bvxkBrFFBMwkDrb+Ye9i5OJgFvjIJLH6ewcLSEJYIEJiyvZF7CA2i4CqRMPEl2BxXgFT ifPbXzCC2BIC8hKrNxwAG8QJNGjKry4mEFsIqGbuh52MExi5FjAyrGIUycwry03MzDHVK87O qMzLrNBLzs/dxAgM7WW1fybuYPxy2f0QowAHoxIPrwP72lQh1sSy4srcQ4wSHMxKIrz8qqtS hXhTEiurUovy44tKc1KLDzFKc7AoifN6hacmCAmkJ5akZqemFqQWwWSZODilGhjDKx6vLkh3 Mvtea9p77d7RzOPWKxN4W3a4T6++ddqBbWbq3J+TtSaXTH7AaCzym1le+vjqQvfUxcflTvyI fl6VsP3tvEnl089+/LIpjVvXSUo8616gj+cV5VszBHuZKh6JVD5bXW3Isb9ggnP7l9PuRfF7 dKdeNrhel3VsVtIy0awtlZ9maq5VYinOSDTUYi4qTgQAE1Vq9mkCAAA= X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D9DA7120025 X-Stat-Signature: xg1qu9k8ip86dkrwwc35hta6x7iyw53k X-Rspam-User: X-HE-Tag: 1705891703-4570 X-HE-Meta: U2FsdGVkX1/cI/Z+TywbsFnSFeU9G1akC9MvvCrOJqxGipnZPgMhjgHyE5DJZrlc1ZITYheuiKWHw0T8LqDNekjgQvjY6c/hAh5YhjSSV8Kpgsl7MHRBfZtDOLBzrOH04KlguMuYEdbKd7wGAubfglRDcNoK6IoS5S1+hEfH8cF7VyNP18qodpYcZ1sKbevSCjeykajcYESxsqiCa2M2mRKISDB2cJ/mK0TFmII4g2yZXjcukE7tnkczoRba6UinkjKTzPCIL5YQt0Tyq6xx27fkRAM70jEzgR5jokoSTwsgcWHMfO1ekj5h2DCbSlF9cYjF5InK4hITekUzELY+RpJ8rdmYfORqgUDY8zw9DgOtFik4h/e0aQSCf4fvj8dRAXCxPpiLnpVcxxGl/ZY334fBgmphK9khNPr/Wr0duhpO5f7+kkMhxCzfa7qVD2PqvvDdNB9dWism4EUAcEkdYPOzCdIpUxWT3wev92qAxsr1B42+RNA/DisgZVavvaycfpjR/u9ZT8Ldd4bcf1GX6oprIsbog3QXIdXTStkKVB5oz3srovIxsRXuqmqiCI5NlPtbTm78ABhlqFV68IEVRIACJea1nD2xz8wLodrsCzIhEEf43zW6C+jMgreUW5PN+LSCkS1Pqf7Ns4WLsHPkR4vKFntV7Td2dXATYDzbC2nLs1Y1kLMwHdhtmUg56zXITpdDLGJzaQw25AvaX8sZMsYEYlblS9BxZMQhOXcPMGcQo+m4lv5h4wyEha2PsxQEyGn3fAQ7fmBAROSust8Zj1TVmMOPDQ7I4YsZ5f4CdNzCSWTYL3eS2hfi3FRK91oFkiv0y1yeWPEIUOaFfBYbl7aEjVUN4/DeE46c+j3S30b18+MN1lt1Uve7mXR9IblGHbEoogH2HI3yNvMUU/mArOFQ/9WwC0ilDbfUVaE/rq6ZjAXZxS4VCAp+//M+33wXFNg91rzymq8Wl4rt4ss XOTIG2Cv ShT0XNEx0urOBBk8IuetPd4nqLOiRyLrYnynLhHxDh4wgIwaqRIm0ioqL3k8xUFxrWv0HqbsHRJcswq691fWQBUQ97VeZHLU7YDsfz6cLSouZhb98lrBoKl7isbP3Qul0xOQU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time later, rather than the moment migrating folios. The folios freed by migrc are too old to keep in pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /* From patchwork Mon Jan 22 01:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2250BC47422 for ; Mon, 22 Jan 2024 02:48:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1A216B006E; Sun, 21 Jan 2024 21:48:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AA3AB8D0001; Sun, 21 Jan 2024 21:48:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F5746B007D; Sun, 21 Jan 2024 21:48:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7D4D46B006E for ; Sun, 21 Jan 2024 21:48:28 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5C48E404F5 for ; Mon, 22 Jan 2024 02:48:28 +0000 (UTC) X-FDA: 81705413496.29.E41F84F Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf29.hostedemail.com (Postfix) with ESMTP id 111F9120025 for ; Mon, 22 Jan 2024 02:48:25 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891706; a=rsa-sha256; cv=none; b=yRp2J8Yt+SBg/+SpagzTNcBQE2bu4ljHjgbbfAG60EgZdVM7c4zeojo/Qzfo6qcOHeQ0Tl Oc1CbUO0SecHg30GTiU3q7ezCovWB6hNK4Eub11x+e3qRze3FVN2FcgvTXCT2CvD8iYtqP xwFtNz/bL+7vzgBv+Jvi0DKG5xqguWc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=FKGmg+DAz0FrX3KjM/m3iaGERs+2p+YP5q2ZhuB76+k=; b=jANKZbq7gwuBtCYn3ConQl4tEbxvnv8WyD+8EWnZ1DV88Mt9uPSscFgKpX0QGDt6l27vBK 8oh5AgCOGw4fCfiXX84Ef4PMd8EcP2pzXovlDuv4CncoSQFfSCyYwtxbCSINoHeWETsclc FlrZv3/kRX5gCjeO99BETf+lbueuwDw= X-AuditID: a67dfc5b-d85ff70000001748-90-65adbe43b085 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 7/8] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Mon, 22 Jan 2024 10:00:39 +0900 Message-Id: <20240122010040.74346-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrELMWRmVeSWpSXmKPExsXC9ZZnka7zvrWpBqvbLSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlfHgBHtBzyrGiv/H/jI3MF5oY+xi5OSQEDCReD1tOjOMvevLahYQm01AXeLG jZ9gcREBM4mDrX/Yuxi5OJgFPjJJrP7eAVYkLBAt8eTmY3YQm0VAVeLDsflsIDavgKnElY+v WCCGykus3nAAbBAn0KApv7qYQGwhoJq5H3YyggyVEGhnl5jecw/qIkmJgytusExg5F3AyLCK USgzryw3MTPHRC+jMi+zQi85P3cTIzAmltX+id7B+OlC8CFGAQ5GJR5eB/a1qUKsiWXFlbmH GCU4mJVEePlVV6UK8aYkVlalFuXHF5XmpBYfYpTmYFES5zX6Vp4iJJCeWJKanZpakFoEk2Xi 4JRqYFQ53XT6rWVStfnR0EaJ8l8mO5VKp/hYGsun/8sTkF5y4pLYR/msW2++qDPtd+9f9K9q RVjivaqFQk0ZApuMI78wfZ3SmLKv++I+Qw5Xz0AlJcmf8f+WL2f6dcEhQ9Bls9VhKQ7VIjNv h87Vyzt4buc80eq+erhC5PKBnpJvcc5l5Q897yUpKbEUZyQaajEXFScCAMxhA7qFAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrJLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwZ9VWhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyHpxgL+hZxVjx/9hf5gbGC22MXYycHBICJhK7vqxmAbHZ BNQlbtz4yQxiiwiYSRxs/cPexcjFwSzwkUli9fcOsCJhgWiJJzcfs4PYLAKqEh+OzWcDsXkF TCWufHzFAjFUXmL1hgNggziBBk351cUEYgsB1cz9sJNxAiPXAkaGVYwimXlluYmZOaZ6xdkZ lXmZFXrJ+bmbGIEBvqz2z8QdjF8uux9iFOBgVOLhdWBfmyrEmlhWXJl7iFGCg1lJhJdfdVWq EG9KYmVValF+fFFpTmrxIUZpDhYlcV6v8NQEIYH0xJLU7NTUgtQimCwTB6dUA6Mmw7VrHrdb A1hYFrbkPq9ubnd/ab9occmTTZ+erP9+oj5xVuBq4+qgmIyMA3wdXXx3AgI77Puju71v8xTP uHDk1cuCEM3HkRFJliKPdl7hqm12jZ3/se521NRfHcE/32x9wlB0g/mg8A/Vqr9rJn23LI2s mmSyT+TRJPG/04NMM5wrlqfeXqXEUpyRaKjFXFScCAASkn00bAIAAA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 111F9120025 X-Stat-Signature: fsubmh6hfjabuay7bbmpek8ffjgi633s X-Rspam-User: X-HE-Tag: 1705891705-908460 X-HE-Meta: U2FsdGVkX18H40lvBYSEs3+sKQXTBgHMQpBL5AyY6kKLtP+sjn41mkA2qbe8UKWpDUEzmbbf/RCuhSXKSHnvjPKnfGEilBCHbDPTS69eVC4W0eCwvwV8x9rWtDauNfJkSGPoGgWCXBJbL8R1LPx4JihRpFOxOZ9Vb8r/CfWZeeT7f6qFud/+pDUa6bS0/uKSolrjbNKM8I8T4z5ICssDlGUbZjD/nvc8vPEUXVMrZbrBOXYjyOv+ua+MK4XDLWgNfEZEoze/4FVUVMNpMEV1bI/V7Jv7mN5ipwfVIKJWFpsGfNCXUPCgFmJCEG6vkbAMN02oR6dRCl+cLRDi5YqmbWMRkOpwmvvXsLiuL182+cbrGVcv9YcPHDa9Xg8EyUyba2qXTGqGN+ghnvq3MeVX9VK86ZGbEVlBUtJfsRNkj0v8enw6e2zmf/VWlPtPyweGO/AZ/bs34+9kFKSarGR9rCmiZFK7AXN7oAfMFniJZOqc7DUzGfjX00CG2WXVyMyJSkFyFIVT7l/zE8cCtSA6NTd5xfQaEFIdNntwYsHuIk+ArTRuQwZhtPLcj2YwgRij5um+oIp5PbMz8SEU2x+wxXO7AxCvcDVk+vK0KZWG6LLekndZew1oNeNhulPf4vYCUqykGZT7shYbQ3cWwZhsnDZkWP2laCjCu+RkOV3nMiqPdwL6luzWhHq52SIuqaVvAVUx0cUozrk6av1TeD15aiv7yXFRY9enK+v4Iy+XcrbAoZWHP19cbXSOAKFM2SECGw1By3Yn/OQaUc1LTzMjdPUL9k0WQGhpFv4DvklYLFgTjWIo2BRp/2NTG4gs+3DpIkh43MuKf1XObm+MiMUlHJtT/KsHPWNzIPoW9FDdGRcOGsrzSMLXY+70Q8z+nfZpWAoU6v0eucDL+wR8oq/kME/JWZUb46+NaT6JQQiiLmy/YzAB8IVwDWmUIPsvmw5RmCDjuabdTjaOZfzWTF2 JSg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. --- The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 53 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 12 +- 7 files changed, 358 insertions(+), 12 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..ab02cb8306e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,57 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +void migrc_flush_start(void); +void migrc_flush_end(struct tlbflush_unmap_batch *batch); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct tlbflush_unmap_batch *batch) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..cbe5372f159e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct tlbflush_unmap_batch *batch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index b484d659d0c1..39ab0d64665a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; + migrc_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + migrc_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; @@ -677,9 +679,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); From patchwork Mon Jan 22 01:00:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13524737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5022EC47DD3 for ; Mon, 22 Jan 2024 02:48:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2B4C6B0072; Sun, 21 Jan 2024 21:48:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB4758D0001; Sun, 21 Jan 2024 21:48:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A55646B007D; Sun, 21 Jan 2024 21:48:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 912AB6B0072 for ; Sun, 21 Jan 2024 21:48:41 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 67439A11E0 for ; Mon, 22 Jan 2024 02:48:41 +0000 (UTC) X-FDA: 81705414042.27.709029D Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf14.hostedemail.com (Postfix) with ESMTP id 86BDA10000B for ; Mon, 22 Jan 2024 02:48:39 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705891719; a=rsa-sha256; cv=none; b=tKjco4UCtC/TQxWS6p0PHDy+oTPe+8GUVTon6hXahYyfmkrnZon2iELeAuPmS8erLh0MHz 7ZZi1ErDSIzdyukWaanv2hp7UZCKz/bI1u6RzmT31TUWPsd65s/+PjOfYL30zPu6zpniW2 9cWSBjrfTjKTTM4KkpJGQBz9V5p44lY= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705891719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=z8N3K7cEzP29wp8OmV1x4SKhsG69RUNKzzZ3XrSlbEU=; b=FEtVaomDKb601EiLwC1sxl3X8Mtl425FwYmiSX8PT9RtQP3giUTk6NN8UzW8PBnGpUMhlK Af8oAN84jZKjCNGIMU4hjLvMi4wBgPkT0JxlsVFdoLA9FFNWFwBPyJ7rvIMqqrWyTOdJ1o 6AIEIurC96ySzzpamfdkZD20L6Ezi+g= X-AuditID: a67dfc5b-d85ff70000001748-95-65adbe43199d From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v7 8/8] mm: Pause migrc mechanism at high memory pressure Date: Mon, 22 Jan 2024 10:00:40 +0900 Message-Id: <20240122010040.74346-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240122010040.74346-1-byungchul@sk.com> References: <20240122010040.74346-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnoa7zvrWpBv0tlhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHF910NGi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AdRxctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnls/WXn0Tj1GpvH501yHu/mv2UL4I/isklJ zcksSy3St0vgyuhq+89WMFWl4vcmqwbG/bJdjJwcEgImEn+XPWSEsc9O38wGYrMJqEvcuPGT GcQWETCTONj6h72LkYuDWeAjk8Tq7x0sIAlhAU+J+4tngzWwCKhKzN23CyzOK2Aq8XjGAyaI ofISqzccABvECTRoyq8usLgQUM3cDzsZQYZKCPxnk7g++wgrRIOkxMEVN1gmMPIuYGRYxSiU mVeWm5iZY6KXUZmXWaGXnJ+7iREYD8tq/0TvYPx0IfgQowAHoxIPrwP72lQh1sSy4srcQ4wS HMxKIrz8qqtShXhTEiurUovy44tKc1KLDzFKc7AoifMafStPERJITyxJzU5NLUgtgskycXBK NTAmNca9qon/vGAqq/PkesONL//mz38uozZjjZLeqskW3PLXMzsO3Fpy7ETnkjjlN4xZKUoX u3ks+FuYwhOzzWoXpUwL32z+43D1pyg9jUl+CVbtFc5V/u77FOqLMk+vss5f17LVTTj29uWX /vP/fUvn6ViXHlI8O7f9jdeJ1LeyIXZeoTIXriixFGckGmoxFxUnAgAdfhCsgwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrOu8b22qwfs2XYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZXW3/2QqmqlT83mTVwLhftouRk0NCwETi7PTNbCA2m4C6 xI0bP5lBbBEBM4mDrX/Yuxi5OJgFPjJJrP7ewQKSEBbwlLi/eDZYA4uAqsTcfbvA4rwCphKP ZzxgghgqL7F6wwGwQZxAg6b86gKLCwHVzP2wk3ECI9cCRoZVjCKZeWW5iZk5pnrF2RmVeZkV esn5uZsYgcG9rPbPxB2MXy67H2IU4GBU4uF1YF+bKsSaWFZcmXuIUYKDWUmEl191VaoQb0pi ZVVqUX58UWlOavEhRmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QDo93xug+yIhccdm+b cqr6p337zc+uiesLuSd1Xdi6fhVfdI/iFIFzXTtuLj7U25n/5MI3r0j5DbZFDpxLZb/fuq7Q 3sdXMjlrx9JVX1MPK+5qv+6nGPlb9d3eyc660Q+0CszSQh3tj7vyv3SSP/dyW0Wi13sxt7YN vzzFfbI/9F/YwJlg28joocRSnJFoqMVcVJwIAOxAnm9qAgAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 86BDA10000B X-Stat-Signature: r8nr7ow4g9zi5dhfd8bg69pbk9djwoqs X-Rspam-User: X-HE-Tag: 1705891719-622546 X-HE-Meta: U2FsdGVkX1/eyqP69SgSeWAkf19g2CGRC5HuCfkL0+/LX6xjYIihWXApfwhwptk0AMq4szk8Zp1nAf18VfVzQN2o0ERkpcMsEnGPvgBPMglSNb1U3mEL/nRiCqDqym/0NVm8BGJ/ys+bOQ0It7OHoPsPKBCfmFArYbcghtAoGQEkf/fYWZR0Y2KpHChXsQPYl8MVO62NhJ3xptrk/AKwOF+Giu5rSB5n6gvwvC/asnXzi0HRJ2CFF/gqlDWeMvu/yT2a/Vd7wGIfKX8M5yKs8iDzVZwFKBl3063s1aLmXtqlfaT0WL+qAbCygocXu0KxhB2rJ7JsV1dLA28cQloyMQM4+ow78sHw+r0BgOwxCRQGULjKtNAekbabHQrbga5MHLVL+jJBjZ/6OjH597dryjPGEl61Fyyb1OoJJROpq2ORF2qWPsd/40SJxMUPeqz5RScmfQBiQr2Zi5HUUPIayYuyrPabLOtQ5LGvMfRNWVaxcKxjFDYGOq4oK0OLu/nvLQ+COngd7C/FPLWwbolAyhNhl7yc+rmQI9AL/sOIrad2tKEMLqaJggZceQ3sLg4QrvTxUwY1GumlBI3P707L8/Cfg4XafitWV1Mb/iAg8snXOwH/RDhD7sutkryYXKb4fDvv3sArGQ1ttck6+cwuwZuLrlj1v4dsI2pwk9kLCx69XfcdNaDOsSNkMicTRXtmHNfnC8nSQUfIpnvkznGsLF0NQWzleUuKYQz5ivyB/aYNBOzyK0R88VPa2p30423OIWG669NakYxpRHJUeMKMDGWl3gN/n9/41CKRZQmeBwe28gxbF9DersP4w+0OKBYG4DlVtNU+17GYlEKnoC7cORS4AktDRxX9CwR/qe5NAkzpCdzgIXucN4jRRt89Q8tqO+NHlSQspcKEAc1sUsJrzSJAmcBvjZGtAzo4fffIIwwDkrwYqFxpJNwoYLdLpINjYjf7MEWoFxsehe/5Xpe AeQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Regression was observed when the system is in high memory pressure with swap on, where migrc might keep a number of folios in its pending queue, which possibly makes it worse. So temporarily prevented migrc from working on that condition. Signed-off-by: Byungchul Park --- mm/internal.h | 20 ++++++++++++++++++++ mm/migrate.c | 18 +++++++++++++++++- mm/page_alloc.c | 13 +++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index ab02cb8306e2..55781f879fb2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1285,6 +1285,8 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, #endif /* CONFIG_SHRINKER_DEBUG */ #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern atomic_t migrc_pause_cnt; + /* * Reset the indicator indicating there are no writable mappings at the * beginning of every rmap traverse for unmap. Migrc can work only when @@ -1313,6 +1315,21 @@ static inline bool can_migrc_test(void) return current->can_migrc && current->tlb_ubc_ro.flush_required; } +static inline void migrc_pause(void) +{ + atomic_inc(&migrc_pause_cnt); +} + +static inline void migrc_resume(void) +{ + atomic_dec(&migrc_pause_cnt); +} + +static inline bool migrc_paused(void) +{ + return !!atomic_read(&migrc_pause_cnt); +} + /* * Return the number of folios pending TLB flush that have yet to get * freed in the zone. @@ -1332,6 +1349,9 @@ void migrc_flush_end(struct tlbflush_unmap_batch *batch); static inline void can_migrc_init(void) {} static inline void can_migrc_fail(void) {} static inline bool can_migrc_test(void) { return false; } +static inline void migrc_pause(void) {} +static inline void migrc_resume(void) {} +static inline bool migrc_paused(void) { return false; } static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } static inline bool migrc_flush_free_folios(void) { return false; } static inline void migrc_flush_start(void) {} diff --git a/mm/migrate.c b/mm/migrate.c index cbe5372f159e..fbc8586ed735 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -62,6 +62,12 @@ static struct tlbflush_unmap_batch migrc_ubc; static LIST_HEAD(migrc_folios); static DEFINE_SPINLOCK(migrc_lock); +/* + * Increase on entry of handling high memory pressure e.g. direct + * reclaim, decrease on the exit. See __alloc_pages_slowpath(). + */ +atomic_t migrc_pause_cnt = ATOMIC_INIT(0); + static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) { arch_tlbbatch_clear(&ubc->arch); @@ -1922,7 +1928,8 @@ static int migrate_pages_batch(struct list_head *from, */ init_tlb_ubc(&pending_ubc); do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && - (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED) && + !migrc_paused(); for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; @@ -1961,6 +1968,15 @@ static int migrate_pages_batch(struct list_head *from, continue; } + /* + * In case that the system is in high memory + * pressure, give up migrc mechanism this turn. + */ + if (unlikely(do_migrc && migrc_paused())) { + fold_ubc(tlb_ubc, &pending_ubc); + do_migrc = false; + } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ef0c22b1109..366777afce7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4072,6 +4072,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + bool migrc_paused = false; restart: compaction_retries = 0; @@ -4203,6 +4204,16 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (page) goto got_pg; + /* + * The system is in very high memory pressure. Pause migrc from + * expanding its pending queue temporarily. + */ + if (!migrc_paused) { + migrc_pause(); + migrc_paused = true; + migrc_flush_free_folios(); + } + /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) goto nopage; @@ -4330,6 +4341,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, warn_alloc(gfp_mask, ac->nodemask, "page allocation failure: order:%u", order); got_pg: + if (migrc_paused) + migrc_resume(); return page; }