From patchwork Tue May 23 14:02:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A165FC77B75 for ; Tue, 23 May 2023 14:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43267900008; Tue, 23 May 2023 10:05:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E1C2900002; Tue, 23 May 2023 10:05:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D06A900008; Tue, 23 May 2023 10:05:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1DCC4900002 for ; Tue, 23 May 2023 10:05:22 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6B65C05A4 for ; Tue, 23 May 2023 14:05:21 +0000 (UTC) X-FDA: 80821692042.12.5F16E2B Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf09.hostedemail.com (Postfix) with ESMTP id 68472140361 for ; Tue, 23 May 2023 14:02:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="i7Zz/GSp"; dkim=pass header.d=linutronix.de header.s=2020e header.b=4VxStdT1; spf=pass (imf09.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=S3pfkW/p/Z2TzFTHhq7W2MNUzLZm2BRRVE9xk/9SCdjAYMHAcplfUFCgPjp6efRethxb2x LixqIBCfDy/F/g5/TlOPbW+xvbMO+Jmqw13N/aK+bSSs6+m3Ksuf3jlmFNFgWi6S1mbqG/ jp98jhHo/NgEWNcbeucNlY0H5QjNj6U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850534; a=rsa-sha256; cv=none; b=nu2x/kcevmirO5HIcdvAbX5hlNSlYqdpZx2ppX4nKBANpGr8OXdif/ZUiWUawWkdMM1/eQ nJo3aR3TbU3GBpFcGNq08BemcS3gL7FYUyewmmoLeYfPLLhZNXerKHdz1AqklMF+s9Iaws LAJDeVTAWZDSU+bvN+G7y1vzgvqEH0Y= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="i7Zz/GSp"; dkim=pass header.d=linutronix.de header.s=2020e header.b=4VxStdT1; spf=pass (imf09.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de Message-ID: <20230523140002.575854344@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=i7Zz/GSpTwRjIXmIsCfnuDTcETDAMAP9b3V0um0Ob7586q4/4c/lGBA1BZGHL/5XlId3Ry 0U9Ol/e1vSPFPRmcSmrqAwLqzRoxsrXvcCvlUUzm8qVXLp9uemRTNPUQoecuvCn97XdvGZ oaJzW1qv08hMYZ8Gq4SPbcW1eNKt6x8T8nLvUEsjB3A1z2TyFGwrOKEqsE0ULc2hkKv2Gu 2XGFlA+t38dWINMi/B+WhL6SN1jtcyOFPcgKrQC0vnbFGIY3WejXGaqZXIYVI79LigY0iK doyleoxA3chyZOsdPhV+U5iqmVLJ/cEJB9EnX6Pz9SC7w/s6owul7tJIajkDow== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=4VxStdT1SRyjYcH4mJQcTeZpKGWnPdHJ87rXAjM+VYWX+X349GCh0ZnFOJ97S24jyDsDk0 7sE4aKnXdirpsoCg== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:11 +0200 (CEST) X-Rspamd-Queue-Id: 68472140361 X-Rspam-User: X-Stat-Signature: b6334ofy5p9j4baig6nawwi759oauc4w X-Rspamd-Server: rspam03 X-HE-Tag: 1684850533-375414 X-HE-Meta: U2FsdGVkX1+BJW+HN+d8OxWS7kTq2jg20hHXL2VOjCI2GKPhP1IqdUiI0O+cathpkiQ970X5jS6xPk6sONDbnhYkfmYtpLAn5ZxQJMxahR2XHlvml1WEjYA967zIR6x7Rx2hCl+hiCKB4ZaLgAU3HNI/Sdo0tLohw6Ad7F1UMkwNdebnDTnKDJKgZuilIhvycQbysenq5rEfYLAlhG7USfVUNpDzeho4Rok+3iZEQxBqanzWDNFIJ4Ij2kUBgaugZLiZRa2uVjkfrpp2xMDvBXFs+n5OomHpUcK1lb4lhdaYs/SwBp4o4dcseDHE/LTd78ynCQ3oxih3utDQbLSFTTBtYVkbNSBcW8pS8MytDq2wrTUUJJHhPRvb50N5oZDyTAsSDrR5D/r/OfdYUIibDAdKwHm84UrAWkDqAF/PL2WrqJV4RxZdQYBK6LeMY7oequRYxysY/tacdE4Mc9WV2WxPRrmrq5drHfiuwtUoYP3tgPnjQ/A/mVPHFiPHohpnAWto1TfqdLGA3zrnA5J9Ojf8kDjM90UytVasW/0yoLIzGXt20bDmKbm0jv6yvJDQBqGc5+RX/Idrfot9JolSzxVEOnf6saPL5b8JOviTNfoZ7/BoootJviIspHKi961fo8agXQDwyvnWvOFzVvLHJCU3NfAW7qB95ojFdbpMwWHuksz59oEmG+E3XSzpaGFkYvwhKp+X6k+wOXZSXyxSlvhdaOfeVBYLUQFzlgEaP9JNQGN/luH1d+F51l83Dbf0vT+JfivtqTAfrgIDJ0vQei9PgFOu7kPvPfOvSvENl4SWlOEnpsF34HMSTW89DQAGOUbq4mEBgZUhGYTaQ8sKdCkvasCXawKD8xTTuCjTiEZ1YQIS4X0oBmT3acTliBiH7Uafvi1rnWh+o5F3gne3oxowjMkgyGVeZJmTnwltCJN6HmfnH/+X5bEnLrVpQqk/jf4YwBCavPkP+TsfMKo +eORwaop o6tG4nx1bTnOE3QoLA2nHRQzyNeybA6Ky2U+u+QKQiPneZdjqKqKRo9Ka3Ly11cgdpC/gQgfzUg4cJFN2SSDAmxErluhvuwS0DSjT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: _vm_unmap_aliases() is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc. This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero. So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of _vm_unmap_aliases() that there are no stale TLBs after returning is broken: x = vb_alloc() // Removes vmap_block from free list because vb->free became 0 vb_free(x) // Unmaps page and marks in dirty_min/max range // Page is reused vm_unmap_aliases() // Can't find vmap block with the dirty space -> FAIL So instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists. Signed-off-by: Thomas Gleixner Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes Reviewed-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned l for_each_possible_cpu(cpu) { struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); struct vmap_block *vb; + unsigned long idx; rcu_read_lock(); - list_for_each_entry_rcu(vb, &vbq->free, free_list) { + xa_for_each(&vbq->vmap_blocks, idx, vb) { spin_lock(&vb->lock); if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start;