From patchwork Thu Aug 17 08:05:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13356108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E3DDC2FC0E for ; Thu, 17 Aug 2023 08:09:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20E2E280039; Thu, 17 Aug 2023 04:09:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12ABF28003D; Thu, 17 Aug 2023 04:09:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7D28280039; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C299D28003B for ; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9AACDC0F54 for ; Thu, 17 Aug 2023 08:09:02 +0000 (UTC) X-FDA: 81132870924.05.0302610 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf17.hostedemail.com (Postfix) with ESMTP id CC32B40014 for ; Thu, 17 Aug 2023 08:09:00 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692259741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=A6d2qKWgqYrrK3URsslFOkEWag9AT2ivuocXCoU+D7o=; b=v2avN78/Uz4GtgBslPkLDyIayW3nSzD5jbAB9hbUPquBmBXmi8m7wDz2C8LsZeMiiz7Psw 1ltYr2e5SQaO/TTaMEDxB9B/ErSePO5l+9IjvenAiAqqy4X1G3rOz19xguGS8oDMRzvO0q 0x6AS7mcE1Vp/vTw7uJ3srA4OvsjRFI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692259741; a=rsa-sha256; cv=none; b=aSb29Tg0A71xFu+dvDQgyrk3jbWV8ic7bUq8PoVrCjIRIrZPrOcOau7RSXrwFgWykYMf3b kWPsQOEVArOlFAhzKS4vdqQ+8NM++KfENX65/iXr0Vc0GbqdPTwHcNNZGkaYzkp6pv4due TWblSJpbFH1elzjYfqzrX8dxzgwmh/g= X-AuditID: a67dfc5b-d85ff70000001748-c3-64ddd598cf3a From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, xhao@linux.alibaba.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, dave.hansen@linux.intel.com Subject: [RFC v2 0/6] Reduce TLB flushes under some specific conditions Date: Thu, 17 Aug 2023 17:05:53 +0900 Message-Id: <20230817080559.43200-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMLMWRmVeSWpSXmKPExsXC9ZZnke6Mq3dTDC4cULWYs34Nm8WLDe2M Fl/X/2K2ePqpj8Xi8q45bBb31vxntTi/ay2rxY6l+5gsru96yGhxvPcAk8XvH0DZOVOsLE7O msziwOuxYFOpx+YVWh6L97xk8ti0qpPNY9OnSeweJ2b8ZvHY+dDSY97JQI/3+66yeWz9Zefx eZOcx7v5b9kCeKK4bFJSczLLUov07RK4Mjac+MpY8EesYtHO7awNjB8Fuhg5OSQETCQe7NnF DmPv3f6VGcRmE1CXuHHjJ5gtImAmcbD1D1ANFwezwDImibsHzrGCJIQF3CV2LpzK1MXIwcEi oCoxraMSJMwrYCrxbdZJqJnyEqs3HGAG6ZUQ2MEm0fnuPlRCUuLgihssExi5FzAyrGIUyswr y03MzDHRy6jMy6zQS87P3cQIDMpltX+idzB+uhB8iFGAg1GJh9dh150UIdbEsuLK3EOMEhzM SiK8Pby3UoR4UxIrq1KL8uOLSnNSiw8xSnOwKInzGn0rTxESSE8sSc1OTS1ILYLJMnFwSjUw Or7sXRjeliNVaW7BvcVq9kmJUuFc2bt5V+pK5WVNezsSZ95vWfCp0fnW1qd5MpIVugwnduTF vMsJTA+6tSJ70wWBI7+2JAp+nK6u6x2cOId381PPm/JqPDscp/9zmmHzds3DsxJcKudm1NqU JDUxCq6Yk3dgt9se6dsN7vuTf25lXC8nunOrEktxRqKhFnNRcSIA8i1n7UYCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprFLMWRmVeSWpSXmKPExsXC5WfdrDvj6t0Ug923JS3mrF/DZvFiQzuj xdf1v5gtnn7qY7E4PPckq8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlc3/WQ0eJ47wEmi98/gLJz plhZnJw1mcWBz2PBplKPzSu0PBbvecnksWlVJ5vHpk+T2D1OzPjN4rHzoaXHvJOBHu/3XWXz WPziA5PH1l92Hp83yXm8m/+WLYA3issmJTUnsyy1SN8ugStjw4mvjAV/xCoW7dzO2sD4UaCL kZNDQsBEYu/2r8wgNpuAusSNGz/BbBEBM4mDrX/Yuxi5OJgFljFJ3D1wjhUkISzgLrFz4VSm LkYODhYBVYlpHZUgYV4BU4lvs06yQ8yUl1i94QDzBEaOBYwMqxhFMvPKchMzc0z1irMzKvMy K/SS83M3MQJDbFntn4k7GL9cdj/EKMDBqMTD67DrTooQa2JZcWXuIUYJDmYlEd4e3lspQrwp iZVVqUX58UWlOanFhxilOViUxHm9wlMThATSE0tSs1NTC1KLYLJMHJxSDYyOUeK3Sx/q1IRY lWc4fz6z8MgE1s1XZ1U28fjO67ooyS0RlzfLagOXP0//w2ufpkfMr+4qf/Z3xt7O+YYzWO71 bj5YHHZCqfovX0eJWrnXtzJlZheR7E3hjfJL3O+tfphoIhfQ3bXFK+xD23/jz+ueHl2qcW6l pe4rlq86lhFbVnrXB07X8VRiKc5INNRiLipOBADFyewTLQIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Stat-Signature: eayybofr3m3qewau7axo431us9scszrb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CC32B40014 X-HE-Tag: 1692259740-684737 X-HE-Meta: U2FsdGVkX1/JQe0gANYWWnY5Kx0F10xinTqT0+azCrWwAHsk1vaxI2nV8xhzPeEsT+Z8SI0puAs0121/RCMvWO20dsPP0d1vTqKoYWb2OO9oJqGWDEZmh/FHXwMoqPDpbQA0rYrW5iWnSbg+vjF5XHWfriKx0NsTC/i/BcPA5dhU+dsPuUYgma7U9JTvKLO22hKSarfGAlznlfyiJ+FQkxx+63m6gWGjt0EHKMxNMZD8sXy31xfVjMlUEzTb2GQZUPH7AoZkbPnEs0NOoRxSXpuUkPr2p+rLw80g5oiyDoztYD8IvyjlflF9WWRhiPN5WUTzizG6FxhXixyH53FMXef1SeCqDB8H6oHUG+yu3sUw+1oEN5K9SoK6baSUikgVh0ydQSeQIsBILphB/xYwG7DwxUODD1yiHEnvRBrOlkjZCj1XdbwMpjTHqsM7funJy4k0+1IQNCWlRHXFgVbc3VYFqyZ8XQZacAdNZ3NuD2HaVQRTHvoRH7o4e4GjZhOH2Lp5SBgiLzuCquZZ5mkm3xyb56nOau8GECaxCVuEhPyngr13l50h9REoy5gWmz2coZWFDqp2/pKLqj/orjZclj4CzN/aUOlnZJNp4OKqQfYegYvY8FkcQ97GunI5Nn6E1n093d7L+PmBdAnMC8XLj3RKp3VACK7l/Y91A+EXfJdlvGbHyOJQLCzOsnS/xcuzapagIeWceu9VfBeHwD0DgJtVhS/W7eIpgPvMojzUgC2f7+ga78yrXG72DDwuMSTK/weYKNJZRTLQjrVn8pAdCGSPvB8/WuwKs1OzfpDLlwnmmim1ojW23X50Lgr+fRc2tHjhsPJtKxiFfRWsWMiqFYzV1AFLGCT00JpJuGQ6TfvogtJFzuUUKJ8y60MuMK7vJi3wrggxlfy+huH5oZ+/tq0QJQo+O8ck X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, While I'm working with CXL memory, I have been facing migraion overhead esp. TLB shootdown on promotion or demotion between different tiers. Yeah.. most TLB shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE is inaccessible"). However, it's only for ones using hinting fault. I thought it'd be much better if we have a general mechanism to reduce # of TLB flushes that we can apply to any type of migration. I tried it only for tiering migration for now tho. I'm suggesting a mechanism to reduce TLB flushes by keeping source and destination of folios participated in the migrations until all TLB flushes required are done, only if those folios are not mapped with write permission PTE entries at all. I saw the number of TLB full flush reduced over 50% and the performance a little bit improved but not that big with the workload I tested with, XSBench. However, I believe that it would help more with other ones or any real ones. It'd be appreciated to tell me if I'm missing something. Byungchul --- Changes from RFC: 1. Fix a bug triggered when a destination folio at the previous migration becomes a source folio at the next migration, before the folio gets handled properly so that the folio can play with another migration. There was inconsistency in the folio's state. Fixed it. 2. Split the patch set into more pieces so that the folks can review better. (Feedbacked by Nadav Amit) 3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). (Feedbacked by Nadav Amit) 4. Tried to add sufficient comments to explain the patch set better. (Feedbacked by Nadav Amit) Byungchul Park (6): mm/rmap: Recognize non-writable TLB entries during TLB batch flush mm: Defer TLB flush by keeping both src and dst folios at migration mm, migrc: Skip TLB flushes at the CPUs that already have been done mm, migrc: Ajust __zone_watermark_ok() with the amount of pending folios mm, migrc: Add a sysctl knob to enable/disable MIGRC mechanism mm, migrc: Implement internal allocator to minimize impact onto vm arch/x86/include/asm/tlbflush.h | 9 + arch/x86/mm/tlb.c | 67 ++++++ include/linux/mm.h | 30 +++ include/linux/mm_types.h | 47 ++++ include/linux/mm_types_task.h | 4 +- include/linux/mmzone.h | 6 + include/linux/sched.h | 5 + init/Kconfig | 13 ++ mm/internal.h | 14 ++ mm/memory.c | 17 +- mm/migrate.c | 381 +++++++++++++++++++++++++++++++- mm/mm_init.c | 1 + mm/page_alloc.c | 19 ++ mm/rmap.c | 133 ++++++++++- 14 files changed, 734 insertions(+), 12 deletions(-)