From patchwork Tue May 23 14:02:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A165FC77B75 for ; Tue, 23 May 2023 14:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43267900008; Tue, 23 May 2023 10:05:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E1C2900002; Tue, 23 May 2023 10:05:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D06A900008; Tue, 23 May 2023 10:05:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1DCC4900002 for ; Tue, 23 May 2023 10:05:22 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6B65C05A4 for ; Tue, 23 May 2023 14:05:21 +0000 (UTC) X-FDA: 80821692042.12.5F16E2B Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf09.hostedemail.com (Postfix) with ESMTP id 68472140361 for ; Tue, 23 May 2023 14:02:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="i7Zz/GSp"; dkim=pass header.d=linutronix.de header.s=2020e header.b=4VxStdT1; spf=pass (imf09.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=S3pfkW/p/Z2TzFTHhq7W2MNUzLZm2BRRVE9xk/9SCdjAYMHAcplfUFCgPjp6efRethxb2x LixqIBCfDy/F/g5/TlOPbW+xvbMO+Jmqw13N/aK+bSSs6+m3Ksuf3jlmFNFgWi6S1mbqG/ jp98jhHo/NgEWNcbeucNlY0H5QjNj6U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850534; a=rsa-sha256; cv=none; b=nu2x/kcevmirO5HIcdvAbX5hlNSlYqdpZx2ppX4nKBANpGr8OXdif/ZUiWUawWkdMM1/eQ nJo3aR3TbU3GBpFcGNq08BemcS3gL7FYUyewmmoLeYfPLLhZNXerKHdz1AqklMF+s9Iaws LAJDeVTAWZDSU+bvN+G7y1vzgvqEH0Y= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b="i7Zz/GSp"; dkim=pass header.d=linutronix.de header.s=2020e header.b=4VxStdT1; spf=pass (imf09.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de Message-ID: <20230523140002.575854344@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=i7Zz/GSpTwRjIXmIsCfnuDTcETDAMAP9b3V0um0Ob7586q4/4c/lGBA1BZGHL/5XlId3Ry 0U9Ol/e1vSPFPRmcSmrqAwLqzRoxsrXvcCvlUUzm8qVXLp9uemRTNPUQoecuvCn97XdvGZ oaJzW1qv08hMYZ8Gq4SPbcW1eNKt6x8T8nLvUEsjB3A1z2TyFGwrOKEqsE0ULc2hkKv2Gu 2XGFlA+t38dWINMi/B+WhL6SN1jtcyOFPcgKrQC0vnbFGIY3WejXGaqZXIYVI79LigY0iK doyleoxA3chyZOsdPhV+U5iqmVLJ/cEJB9EnX6Pz9SC7w/s6owul7tJIajkDow== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0Pnj4m7X1tKIyZdbjRJgPx9X8dUQ3jmNG+/bZOWa4Pg=; b=4VxStdT1SRyjYcH4mJQcTeZpKGWnPdHJ87rXAjM+VYWX+X349GCh0ZnFOJ97S24jyDsDk0 7sE4aKnXdirpsoCg== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:11 +0200 (CEST) X-Rspamd-Queue-Id: 68472140361 X-Rspam-User: X-Stat-Signature: b6334ofy5p9j4baig6nawwi759oauc4w X-Rspamd-Server: rspam03 X-HE-Tag: 1684850533-375414 X-HE-Meta: U2FsdGVkX1+BJW+HN+d8OxWS7kTq2jg20hHXL2VOjCI2GKPhP1IqdUiI0O+cathpkiQ970X5jS6xPk6sONDbnhYkfmYtpLAn5ZxQJMxahR2XHlvml1WEjYA967zIR6x7Rx2hCl+hiCKB4ZaLgAU3HNI/Sdo0tLohw6Ad7F1UMkwNdebnDTnKDJKgZuilIhvycQbysenq5rEfYLAlhG7USfVUNpDzeho4Rok+3iZEQxBqanzWDNFIJ4Ij2kUBgaugZLiZRa2uVjkfrpp2xMDvBXFs+n5OomHpUcK1lb4lhdaYs/SwBp4o4dcseDHE/LTd78ynCQ3oxih3utDQbLSFTTBtYVkbNSBcW8pS8MytDq2wrTUUJJHhPRvb50N5oZDyTAsSDrR5D/r/OfdYUIibDAdKwHm84UrAWkDqAF/PL2WrqJV4RxZdQYBK6LeMY7oequRYxysY/tacdE4Mc9WV2WxPRrmrq5drHfiuwtUoYP3tgPnjQ/A/mVPHFiPHohpnAWto1TfqdLGA3zrnA5J9Ojf8kDjM90UytVasW/0yoLIzGXt20bDmKbm0jv6yvJDQBqGc5+RX/Idrfot9JolSzxVEOnf6saPL5b8JOviTNfoZ7/BoootJviIspHKi961fo8agXQDwyvnWvOFzVvLHJCU3NfAW7qB95ojFdbpMwWHuksz59oEmG+E3XSzpaGFkYvwhKp+X6k+wOXZSXyxSlvhdaOfeVBYLUQFzlgEaP9JNQGN/luH1d+F51l83Dbf0vT+JfivtqTAfrgIDJ0vQei9PgFOu7kPvPfOvSvENl4SWlOEnpsF34HMSTW89DQAGOUbq4mEBgZUhGYTaQ8sKdCkvasCXawKD8xTTuCjTiEZ1YQIS4X0oBmT3acTliBiH7Uafvi1rnWh+o5F3gne3oxowjMkgyGVeZJmTnwltCJN6HmfnH/+X5bEnLrVpQqk/jf4YwBCavPkP+TsfMKo +eORwaop o6tG4nx1bTnOE3QoLA2nHRQzyNeybA6Ky2U+u+QKQiPneZdjqKqKRo9Ka3Ly11cgdpC/gQgfzUg4cJFN2SSDAmxErluhvuwS0DSjT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: _vm_unmap_aliases() is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc. This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero. So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of _vm_unmap_aliases() that there are no stale TLBs after returning is broken: x = vb_alloc() // Removes vmap_block from free list because vb->free became 0 vb_free(x) // Unmaps page and marks in dirty_min/max range // Page is reused vm_unmap_aliases() // Can't find vmap block with the dirty space -> FAIL So instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists. Signed-off-by: Thomas Gleixner Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes Reviewed-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned l for_each_possible_cpu(cpu) { struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); struct vmap_block *vb; + unsigned long idx; rcu_read_lock(); - list_for_each_entry_rcu(vb, &vbq->free, free_list) { + xa_for_each(&vbq->vmap_blocks, idx, vb) { spin_lock(&vb->lock); if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start; From patchwork Tue May 23 14:02:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC0D5C7EE26 for ; Tue, 23 May 2023 14:08:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47DEB280002; Tue, 23 May 2023 10:08:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42E99900002; Tue, 23 May 2023 10:08:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F633280002; Tue, 23 May 2023 10:08:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1FD9F900002 for ; Tue, 23 May 2023 10:08:33 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D1879A060D for ; Tue, 23 May 2023 14:08:32 +0000 (UTC) X-FDA: 80821700064.24.6A76885 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf03.hostedemail.com (Postfix) with ESMTP id 85882203EA for ; Tue, 23 May 2023 14:02:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=YXf55I3i; dkim=pass header.d=linutronix.de header.s=2020e header.b=BDGQmGjS; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf03.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=/ljeNuPK0Y0r+8ryvB1gqemQMdtD7u29uImiDwFSDwU=; b=hSKKYQ3rI2S96wDmzf7+hxdo2J0XEOftQWwl+m0EHI177kyZr6A20H9x/g+tA6eDOuWoyc J9y5H9bN7tKd1wyWsjSnE7G5Eu8dFB29HgzEiYWUe4dzwZmxYUMQvchPrmHB1VAaOpLMRI 57rcxKBpQbJbzC82R6PS1f1mLE9lR6U= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=YXf55I3i; dkim=pass header.d=linutronix.de header.s=2020e header.b=BDGQmGjS; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf03.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850535; a=rsa-sha256; cv=none; b=NGbHaWC+RTowVqCVgiqZcVVva23Qycqpo545zu1eQq0bL8x4GBhvACNZuXeIgzfiB4Jldi qCXniu6EKKuLnK66Ae7b9IlbH0H3QsKZDQeeQy0idq8Do4MTS87Z1JnkV5res0PCPJhvOS dkTOsWyP55hK/n9uWwqxZiZorgFLT68= Message-ID: <20230523140002.634591885@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=/ljeNuPK0Y0r+8ryvB1gqemQMdtD7u29uImiDwFSDwU=; b=YXf55I3i8YVr3NmFlljXi7ZWnKakzMztc8pcYxWMEm6SLsrhw7Qi6QFQghQmJxj9GXyNC8 zCpKHKvAxN79grvp4RRofvDE4cmA20z2sDmmG2kwt+g9wIoC7Y5VaNgwy76YxjJyo8+R0/ jFk7M/cMTT7YIT/g8bVTDYGvyYG/CVubzbGsFnlnLTW/1XcS86RHf3Nkbe1kCXRjCjd4FU 2inTrfz3ZnqiZu0TgOvrW2b3b0iDZ2aQOGzu2s7jZUSah9XXo69KjBtdykHriy78LJyAlR 48sjbUhJxPKfpwfiLtWV1wiVbwPvD5xmPirSP/T3b8R8z5T5BxkEsVmn/jrYAQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=/ljeNuPK0Y0r+8ryvB1gqemQMdtD7u29uImiDwFSDwU=; b=BDGQmGjSlNblIsuUQapdcvgx1rfcrZtxwg/Ht8GVdvW7b8nkoFNkj7xg43H77NK19YsNRr 6vcAJ//zJi+6DSAA== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 2/6] mm/vmalloc: Avoid iterating over per CPU vmap blocks twice References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:12 +0200 (CEST) X-Rspamd-Queue-Id: 85882203EA X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 6q1af8gat69pindxhtasgf8j954839p9 X-HE-Tag: 1684850533-28292 X-HE-Meta: U2FsdGVkX18VdPLfX8HxERcl5pme9OEpr137nfGWbOrtAluR78+wufnfFeb6vaeQu7VMv+lnExF178wfPDULXqoEF9dFwzCE0Mq0uyHIIaltRGRaiIZqYQU2vcHBs35hIggUQ5It7opGbPsQ9hkI81fEp1VVVTOW3N51JhV1nDnG8gLAo0eOdinOetZ6u7TvbXXH1O6qYJL7l7eYh4bz7m4WCNEwfYQ4AJDxFQSh1jGhHDZQW73cqqG9EV/Db1ci2leDQ9ZiYreAuIJEHjoL2MuGs9qnW7wOHlMD23G76taUpHQEY1FAT9VzIofsL+fugboGkT+hMgqYCsfj9rQNcwYu5QCeVbcu1GeWjsrlV/tXDqV+lmh3c6pHFjm6+mZuonlLmX1Q6NxhM2bygShdB5k9eOQ7TQoanK6xT66XTRGDEbqglDJHIPfzvlcFJC8i7lAbPlNpQcl5oAsi9BkpWvvDoi+94IrX7Hp4lPW+aLUDk2mx0QzL9B8uIyU1R9DbKypNIG0aPi6CyeDVYKlJvbrlc1ABoSPJo74v/LrtPNQ7COUXzejicM6dXodV+477A1h6hyadzDEqYSIksVTHsicTHkWrbYOpZPhcv55DC9DmNe3iHlpqZj4b3QS5abDlWSQk6Gp2W19RuchLkXu1fkfAKnFnSrFuAOnaBWpbtgf9OhVsaS/4G5/RGdbzL9ijla3iVMH/JG7u4D0/U8UANwfawe/ZLdqLslZE7vA0amS0LnfTjRi+IZyGclHniZd5NLawfRh59ATWs+lPvE6I2M6iokwFgidTyfwVCRmOCAMRuOJ/1NzbbjywsYE0cNxRCd3A3aal98CDg/m3sWf/h07iY6jHTs1Gybg07rGbQbc8i0MjRfF8HEAi9EbmeqNfG0xJOsCCvGL6UnkBiBM71jJ7Huy7GtxzWnqDgOC/Z9ekl9QhQqAOEpMtlskV4fn6Tb5AX7U7UuHl+aOGynN aRN7aWZl d7uQylNcEuixEAOdkKDSpYhe4+qwP7ICN4nvdB1Z2nxMF2/aqeSOiXaMBN8DBi7wF5DEOoSO2jltTNLQs9PU0WM2Yq3kpODP0GF0M X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: _vunmap_aliases() walks the per CPU xarrays to find partially unmapped blocks and then walks the per cpu free lists to purge fragmented blocks. Arguably that's waste of CPU cycles and cache lines as the full xarray walk already touches every block. Avoid this double iteration: - Split out the code to purge one block and the code to free the local purge list into helper functions. - Try to purge the fragmented blocks in the xarray walk before looking at their dirty space. Signed-off-by: Thomas Gleixner --- mm/vmalloc.c | 66 ++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 43 insertions(+), 23 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2086,39 +2086,52 @@ static void free_vmap_block(struct vmap_ kfree_rcu(vb, rcu_head); } +static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_block_queue *vbq, + struct list_head *purge_list) +{ + if (!(vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS)) + return false; + + /* prevent further allocs after releasing lock */ + vb->free = 0; + /* prevent purging it again */ + vb->dirty = VMAP_BBMAP_BITS; + vb->dirty_min = 0; + vb->dirty_max = VMAP_BBMAP_BITS; + spin_lock(&vbq->lock); + list_del_rcu(&vb->free_list); + spin_unlock(&vbq->lock); + list_add_tail(&vb->purge, purge_list); + return true; +} + +static void free_purged_blocks(struct list_head *purge_list) +{ + struct vmap_block *vb, *n_vb; + + list_for_each_entry_safe(vb, n_vb, purge_list, purge) { + list_del(&vb->purge); + free_vmap_block(vb); + } +} + static void purge_fragmented_blocks(int cpu) { LIST_HEAD(purge); struct vmap_block *vb; - struct vmap_block *n_vb; struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); rcu_read_lock(); list_for_each_entry_rcu(vb, &vbq->free, free_list) { - if (!(vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS)) continue; spin_lock(&vb->lock); - if (vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS) { - vb->free = 0; /* prevent further allocs after releasing lock */ - vb->dirty = VMAP_BBMAP_BITS; /* prevent purging it again */ - vb->dirty_min = 0; - vb->dirty_max = VMAP_BBMAP_BITS; - spin_lock(&vbq->lock); - list_del_rcu(&vb->free_list); - spin_unlock(&vbq->lock); - spin_unlock(&vb->lock); - list_add_tail(&vb->purge, &purge); - } else - spin_unlock(&vb->lock); + purge_fragmented_block(vb, vbq, &purge); + spin_unlock(&vb->lock); } rcu_read_unlock(); - - list_for_each_entry_safe(vb, n_vb, &purge, purge) { - list_del(&vb->purge); - free_vmap_block(vb); - } + free_purged_blocks(&purge); } static void purge_fragmented_blocks_allcpus(void) @@ -2226,12 +2239,13 @@ static void vb_free(unsigned long addr, static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) { + LIST_HEAD(purge_list); int cpu; if (unlikely(!vmap_initialized)) return; - might_sleep(); + mutex_lock(&vmap_purge_lock); for_each_possible_cpu(cpu) { struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); @@ -2241,7 +2255,14 @@ static void _vm_unmap_aliases(unsigned l rcu_read_lock(); xa_for_each(&vbq->vmap_blocks, idx, vb) { spin_lock(&vb->lock); - if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { + + /* + * Try to purge a fragmented block first. If it's + * not purgeable, check whether there is dirty + * space to be flushed. + */ + if (!purge_fragmented_block(vb, vbq, &purge_list) && + vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start; unsigned long s, e; @@ -2257,9 +2278,8 @@ static void _vm_unmap_aliases(unsigned l } rcu_read_unlock(); } + free_purged_blocks(&purge_list); - mutex_lock(&vmap_purge_lock); - purge_fragmented_blocks_allcpus(); if (!__purge_vmap_area_lazy(start, end) && flush) flush_tlb_kernel_range(start, end); mutex_unlock(&vmap_purge_lock); From patchwork Tue May 23 14:02:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A476AC7EE23 for ; Tue, 23 May 2023 14:04:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40A7F900007; Tue, 23 May 2023 10:04:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BA8A900002; Tue, 23 May 2023 10:04:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AA57900007; Tue, 23 May 2023 10:04:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1B368900002 for ; Tue, 23 May 2023 10:04:49 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9508914057C for ; Tue, 23 May 2023 14:04:48 +0000 (UTC) X-FDA: 80821690656.28.A481624 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf08.hostedemail.com (Postfix) with ESMTP id 59CA4160157 for ; Tue, 23 May 2023 14:02:15 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=eSaM0SKE; dkim=pass header.d=linutronix.de header.s=2020e header.b=dmb7T4Qo; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf08.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850536; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=AhYn3hs5NO1EoRmwAp1yAaGKnTBrw9UeYsFRRTOeg+E=; b=mzFzR09L9T66Jm93xtaZvQVN8YFBOM/yRifg3dGFE/DziwETRPgGkwuipadhkGty7MZzvr 84YzPovRLME4vLyQSi+6ztXumIMtg/J6R6/f9OKhTgbBPolqzhGXicmwy1187pcr6sw+b1 gvDnM8x7DNIUcV+KSRE+e+bVVsAvd8I= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=eSaM0SKE; dkim=pass header.d=linutronix.de header.s=2020e header.b=dmb7T4Qo; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf08.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850536; a=rsa-sha256; cv=none; b=QuKK1AaVUNCCjblOuBp6QTlQxF8lB1ldjuI2BLprAAAPJn2bu7Jgttsol9wNUDseQ9fw9w 0TNXq6beyMak5RGZBlCpV19D+MD0kErxjH8F6iQbrLhHc2Nr92L6+vrxtnP8Yd9kCiBMjU nrfSroKb1E/NCEJlJCzg5R/UTz8QEdk= Message-ID: <20230523140002.690874212@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=AhYn3hs5NO1EoRmwAp1yAaGKnTBrw9UeYsFRRTOeg+E=; b=eSaM0SKEPbzU7WfQgZ1kFUheSi1XWpqxu3GnG2f7talcZ7lD2U1TLcLrKpIKUxMQgK0gGD Ras3fqy3JziFztZQAaxtJV5vQybLRsM7f1L83y9pNO1CxDRYnMh//cLyZGkt5CWuz6oDxL fJiTQQIdmZq2/lYkPBoeOw4w8wXbVT5ztoXB4FDx9B+eziVq8TA42fuCGEY4AdPFrtHDoZ Fqp4AvcpnX5QnfRblTedSlYJNeHIGCLgAU1ceZZ4mmR2lGyX6ZnfQ3/lHZgpvNLE0k5PWF S1CafBGrPrAymO1TpQ0bX3bX+aD5su95FMnzYRxIC7s+XR9jtOMmSmFHtbtD8w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=AhYn3hs5NO1EoRmwAp1yAaGKnTBrw9UeYsFRRTOeg+E=; b=dmb7T4QogLctYltN4z60XH4rQQmI71XS4LQWBnII8prowgFgpZb7Ev4l+kojwUETlpZuwF hD06DLfW+ZD7CXCQ== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 3/6] mm/vmalloc: Prevent flushing dirty space over and over References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:13 +0200 (CEST) X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 59CA4160157 X-Stat-Signature: kygr7zh1ptz7bzyg3gf6mrfhum1eqq9z X-Rspam-User: X-HE-Tag: 1684850535-488379 X-HE-Meta: U2FsdGVkX18ZtuBzMbV4NInyFr2DbhnHT6Zt4H3fRda3uOEGg+t38FNqSelZlEOd8BjgcJl695KL2L3F3egN6jb/8v1carhU/Y0Ek3K1Ym0hdVE15bLbNAxnQwkDexUuf4ZKXfk0cti1ZYFb306NLL82pIsTNgdt3aijQ+KHnkeUzp4dm+MnhHqyWIXnxHlsAiWfYxIHZv0PRZrjSuJCm3VWZLsYc63qU4Hmc8LYSwCYqd+JoUqZDDjkAo1RrfJdsiI3b/l76bCoCjaGjL2azLWShbrJLgchroh4zJL5r1BVC5v5Cl0B337C0eWwtHJGKpScTA/3hmSLSMm1u28HaeoSKPJBSlminHjPTjsns0lt2v6uQtzFDkK0J8lyjbrr2khlA9UCoDhMsxkWVqZurzRr27og1OeAdb1Vn3LLs2AL2ZwD9uq4IcxKJUV5a6cQlVBFEcFD/IJxTfK51E3/XA+3e9TzWo+jPMGOdjM9ZA1iL5kpDBfpmNP/jLdSTd3UUISzO8X2jWWi/UUZxE2kjtEsgPuBFGDFraKEbZgOko/8zBTo6b5lCFleJXa5lfj7YFtIxD4TSYeiw09V9HPWKEJmn/TFonn6RTcI5iD2vxzZMHRaxJp64XR9GARKC+S9p6fWA1ZKU3waEz9JP1q3L+YJlAjuz+3qtfgkFfiCY35IqwY191bO/MI1cz9YX28tIGcPvoFO26IVraMasiukEv+AU8JDxeLSbUy+ZkHAicChcWnKhqLis3MkO9PfdSvjxkCJ4UiDD5PXm7WwsNP3VAnJ2DdHriOLwVveoKtyV3+2rvu0UfssLkY6ZlbcskQilxt8hhe2go5NFoeMza/R9dzKz9tTJ3NkIhqVbZg6zOWSm7Mghj8xBfwWDHstbkqIdBn/11UcuzaHXB7EUDw2Y8wtRD68wu0alGKyGkT0waUHEADZvUAD4ZrbVFd9PTp59DeyVWTKnT5v9hmuMFM lOY7xCUB Yt8lF0NVeNZG1PpE1U/7Zu2Km1CwuPgmiM0KMlWHvbfGT50N/uzGG301O1PBPdMa/eGylhJBWWdoWBHhJg53fk7aMB12YPj8PJ7x7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmap blocks which have active mappings cannot be purged. Allocations which have been freed are accounted for in vmap_block::dirty_min/max, so that they can be detected in _vm_unmap_aliases() as potentially stale TLBs. If there are several invocations of _vm_unmap_aliases() then each of them will flush the dirty range. That's pointless and just increases the probability of full TLB flushes. Avoid that by resetting the flush range after accounting for it. That's safe versus other invocations of _vm_unmap_aliases() because this is all serialized with vmap_purge_lock. Signed-off-by: Thomas Gleixner Reviewed-by: Baoquan He --- mm/vmalloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2224,7 +2224,7 @@ static void vb_free(unsigned long addr, spin_lock(&vb->lock); - /* Expand dirty range */ + /* Expand the not yet TLB flushed dirty range */ vb->dirty_min = min(vb->dirty_min, offset); vb->dirty_max = max(vb->dirty_max, offset + (1UL << order)); @@ -2262,7 +2262,7 @@ static void _vm_unmap_aliases(unsigned l * space to be flushed. */ if (!purge_fragmented_block(vb, vbq, &purge_list) && - vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { + vb->dirty_max && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start; unsigned long s, e; @@ -2272,6 +2272,10 @@ static void _vm_unmap_aliases(unsigned l start = min(s, start); end = max(e, end); + /* Prevent that this is flushed more than once */ + vb->dirty_min = VMAP_BBMAP_BITS; + vb->dirty_max = 0; + flush = 1; } spin_unlock(&vb->lock); From patchwork Tue May 23 14:02:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EDCBC77B75 for ; Tue, 23 May 2023 14:05:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0790900009; Tue, 23 May 2023 10:05:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB79F900002; Tue, 23 May 2023 10:05:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7EE0900009; Tue, 23 May 2023 10:05:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C9CA8900002 for ; Tue, 23 May 2023 10:05:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 61473A03BA for ; Tue, 23 May 2023 14:05:31 +0000 (UTC) X-FDA: 80821692462.15.DF3974B Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf18.hostedemail.com (Postfix) with ESMTP id 340D71C01F9 for ; Tue, 23 May 2023 14:02:16 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=JKtVAo6p; dkim=pass header.d=linutronix.de header.s=2020e header.b=lzrJhX+3; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf18.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=4pGoRnJ+fwwVnfQLimOLlNVCz8HekZITmO9TMsFFvE7VLpi7WBKrc8GSUY1SRVLyVR1FLM Lx7OyQL9k4mwfmCu1h9pVSZgZx5PrBqleQRKOAHrdztsTlG/OJZchc/vis7onJ5/y74/52 hskwxBIL6Uynk0X/PnDCZkwV6V2uRX0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=JKtVAo6p; dkim=pass header.d=linutronix.de header.s=2020e header.b=lzrJhX+3; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf18.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850538; a=rsa-sha256; cv=none; b=mv9wSM5fQjfm0weCWHv6C0XP9fOhYbYmrHyRSQtM/XCkakLQvNGysAOuamOQ2apvdmbosW gO/UsiDCYk7O3oHEvOiXCq5hXJo1JuCCgjzDHyVfRGUKKAYC/q0hrguZNhuucSYv0JM5zJ EA2QghDrrD6v1rLStfJ+6vTddvzo+a8= Message-ID: <20230523140002.747157575@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=JKtVAo6p0Asbn8XRblzsQCULEGEjNcpx2/hHNDNJb/T3IGaGYt18w1p0hZxzdVp3Nnc7xR oRJhr/FxdQWpf/f5crY59ShV8OPkpWAPTXrOW/u8vkKBg7vFL2NpU2j0BckGRfSdZRc2aI s0IFBXEMBiznw6+55y8L2q4KaYz1NAkEIhN+6sT0rjTl247RTF3szxP5tuvtKKLX5Sd5Cq L6C0YYe4q2i3JywUEfiEe+r2TmjdiZ197yn333H5cf3Yr59+iPZ0eQV2Gj2T0yz3d4h5pg qOH4PlYytyBt/uiLpe3+IhtwsFy5DT8/WGLSqW7k+BLpUM8FS8nJ4T10qb6a7Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=lzrJhX+3bK9luFzF+rv2CVBgXzS2NiI1KQDxMKxCAtOIssNPBKkRPdhuXr7WeLH3pPhpJL pSztWK30io/KXADg== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 4/6] mm/vmalloc: Check free space in vmap_block lockless References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:14 +0200 (CEST) X-Rspamd-Queue-Id: 340D71C01F9 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uoqozuq479iec4twkmdpftwnmyi38b5i X-HE-Tag: 1684850536-550971 X-HE-Meta: U2FsdGVkX1+/craFmQynr0/VnvJs2soHyFsz1nLCv2XNdRsMyaod4uAG/zmx1ZvEhbkxDiS5+D4wJEZx0hNuw7QffT0kC09CBe/uPu2/AV6UWJIpj3dYeL6tBd5YFhh1F/z5OhOLfnmOwMeRqPD4vh3XK9g9oLd8IR+iMPUci6+gOuluHwHLT7tDHwp9Tp2DFZP+q+scTRn9QU5N0/SS8MFlybs8ZTvcxKGsrJRnBk1VviDWfjF79Vb9Enlo6nwFgiwYx6zUPUWwh+Bh5CsY6ikk3ihea7wCf34UGEwopzZtCKWZXVqHINqmHsF9B8tVQXD8+uRdA7G52RPHNPhU6hzpQzolHxRo11jaeWjdvd/K2iL0jFtNvCNH2ycuC1MeYyLWTWoGOmlgtVidHVCDaH8Rq2BW0xNO9iwGLU5oLVX2ot9CBOHLxIXloUZV6qwz4lAoOB3nfsmUSEWaInsjiS3DLRC/NbKEK1AaSgGef9vkcB4Ya2531KvPruyf6O4NvkVba0qz87oIqVcvIDVxOEkd8KDMMi46+0QvE6/RvsZNb7qITaUwrmTC8MFchwxb9WMP6R9U1tRvQx4bWMC+2AyDheumGImUsyL1cs0LARlDBZw4QmLdpgWrAe+Q8h8yz00IBY8/vyEJuNjC5YYYs08S/GRyQwGEVqK5jR7kD17iJbsfxyz+R3g9h482r1lph63U/6Fqr6E021OB/eah7Y0ZOS7eoWsSkPFVx8YgYArMGZKlgY1dkHvqc9aRg3YRN7fyjfwp9d04xMOr/xvRUneNSUWqcfjrLZ/+I7g4kjG3HpwVA2fBPbOVCfy1Orxg5kU3NOboT7ZKeuugvctNRUkSFIFZejH0C3k6U1VAWc3iAInETvEz+FD7ohtmTQo+ohRu/t1LxWNjFptwN5dbfFLK9P2fO5MwCuMX066QVUQL8vxTNP5UlVjSheDt15xTvsumyN9EXIhYeTevseH nNhjaeJW hh8XBN3D4T/9OlfP36nnwfEN3nCETMsCahY0XDDgt1kFB/Wc+fxDlcDnn3h7Aa7LBZAyXhKE1/WWgGP45Q8XZr8Kah/WT4gcXox3o3t58ElHlijSBlBCvdR/KRFzCD+vXtfkaEtzaiWbzWis= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vb_alloc() unconditionally locks a vmap_block on the free list to check the free space. This can be done locklessly because vmap_block::free never increases, it's only decreased on allocations. Check the free space lockless and only if that succeeds, recheck under the lock. Signed-off-by: Thomas Gleixner Reviewed-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2166,6 +2166,9 @@ static void *vb_alloc(unsigned long size list_for_each_entry_rcu(vb, &vbq->free, free_list) { unsigned long pages_off; + if (READ_ONCE(vb->free) < (1UL << order)) + continue; + spin_lock(&vb->lock); if (vb->free < (1UL << order)) { spin_unlock(&vb->lock); @@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long size pages_off = VMAP_BBMAP_BITS - vb->free; vaddr = vmap_block_vaddr(vb->va->va_start, pages_off); - vb->free -= 1UL << order; + WRITE_ONCE(vb->free, vb->free - (1UL << order)); bitmap_set(vb->used_map, pages_off, (1UL << order)); if (vb->free == 0) { spin_lock(&vbq->lock); From patchwork Tue May 23 14:02:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A4BFC77B7A for ; Tue, 23 May 2023 14:10:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5715280002; Tue, 23 May 2023 10:10:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D071B900002; Tue, 23 May 2023 10:10:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCFAF280002; Tue, 23 May 2023 10:10:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AF6B8900002 for ; Tue, 23 May 2023 10:10:14 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7A0171C73F3 for ; Tue, 23 May 2023 14:10:14 +0000 (UTC) X-FDA: 80821704348.26.F35AF1A Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf14.hostedemail.com (Postfix) with ESMTP id 278001003F2 for ; Tue, 23 May 2023 14:02:17 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=FNHzUr4n; dkim=pass header.d=linutronix.de header.s=2020e header.b=CM3GaqCW; spf=pass (imf14.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Ibnqpyl4Lb4JYp4nO/JxOZHxvcUh8VlC1ZH296mXXb8=; b=vPZ3TaZfJRTlCSkq8r16dmvrxanrAaDvlJBkfIRtFWntS+mNFTlBwdPfDLKI6mRtC/vlre yILp3JKuHco9rMxaMmbqxE0pXNFlhw4G+O7ysw+mBgs2NKkcteNkEbsLmNij1bmbEUeWCr jIP9YEOEepp429Fe0Tsa/lCSPdFLWRA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850538; a=rsa-sha256; cv=none; b=uEc1+7bshPAWSd0p6P1s2NcrTWwsYcOyDsyqDAuK689rRzj5gCziQLjnkAi3NQcJi8cCmM eMFzo7ATijY0kV/y2r/IwLkAmtzicqMz2he39/u0IV52nnfWQ6Y0fpK4nRPC+7NGt9qidS jAzp1dbqZNDg4AV0ncs6BIECzOL2xr0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=FNHzUr4n; dkim=pass header.d=linutronix.de header.s=2020e header.b=CM3GaqCW; spf=pass (imf14.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de Message-ID: <20230523140002.799808756@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Ibnqpyl4Lb4JYp4nO/JxOZHxvcUh8VlC1ZH296mXXb8=; b=FNHzUr4nuDzZ128tl2CzhpkmPH9AVm9WZ2Z/CcDIN9dhVaS2d7TIQ9H3qASuPvEWl1TJCH +uNPlqu6p2D9CExxrB2JvU1NXvOSgX90tLmAdCohqOVx3+KXbSsyN7KTwWJhNetXR7fBzB nxaw3KTjIkukTHO+71MbTk4YaN0uwIF+ZRBAwN+QsLWPttY9OAGQaQaeQ7Buw8S0biDOFr UYUAtslxjjoFmpgKFvZcqGk1wRkZu+Gmn1MO5ty57aLybljms6naFsIvDK6AjxTPLORnKK aJyKSqL1ltAqigkaOQx6LxiVPDITRpK1gZq6wNLv6TaJ/fOc8b0hJnmcDSFdew== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Ibnqpyl4Lb4JYp4nO/JxOZHxvcUh8VlC1ZH296mXXb8=; b=CM3GaqCWE3ECvjI41+TwoJcXImXmhvO/ypN4ZfIpuQRQwa4lsAQN6+RAODV3UFyUc8zd0o JGCTd1bKfvd0/kBA== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 5/6] mm/vmalloc: Add missing READ/WRITE_ONCE() annotations References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:15 +0200 (CEST) X-Rspamd-Queue-Id: 278001003F2 X-Rspam-User: X-Stat-Signature: m9oah5poh7nsddezoku3rnh1s9eizupu X-Rspamd-Server: rspam03 X-HE-Tag: 1684850537-469136 X-HE-Meta: U2FsdGVkX1/yeqkfUX5+0PnfxWIr+Zj8zFTqs6CDEBOT3KjYXzzbNFF9ZJG9Z23fmbp6b1t3f8ZctwcMKxsnA1/8VdsL0y7m1imt+fGAZrL/fVkUQ9nwuqXFbP89gWaXByCYSFnNOqS1eRlR3/HoKuTqU0NIQPwtucOPVXRVe6GlxYVBhykkBOlg918Go3VhtZZXUxG/wPx9XrnFFIV3ttqgH86ArPUIYmIvyW0mHOF1q5i8xZqcIaB8t6WSUdQoCbWPfXU768IScIFCWiMCx4EX0DK2Q4uS9E5ihZYx9qruFCS909vj5MJwmtrLMewvBXMKjv24to4g+jWdfEBnlAK9a1y/h5ZECntTLTRcSJvO1bF8Yc+5zGq9NRNapMbpHBiAmvW1XZO7sy5EPQ4lc4VcwKrRf6dgG4m1v11nvP6fzSye04D+Oti8j+HDUUC7uWf6uH2w0vJ64cb597UEyXjHNPA0iTTAjbsIAKIVGAoES4ORZ/q4iD7TV9+tkk5JSQ3bfsnqX82qTmP+cCogbuwBBrGz/PFMEvjzInfKJQ1ThmuYsKdrcUhbHI9Rplv6KsD+T+Nf0rhoHdcmAgkADxgJ0ImtPZzZHI02cMG4CkCavemHE5GLunNN6X9K5aAkWpCZpOwT9Q90tHNAasrwLsfjYmsG0CKx1VhfKHUW1eVz+QoB3RLmLQuLpYvj+KkyERjx0ZvJllBnsSpXe8dHKIU0SkpTm7rirvUdfp6xht4sesLw/zLRltr1KFxhq3YfChhbPm2T5Rxn3uXMPcEgl6PYbLToDMFDdCFjhGfGTikpzByxAGYGvxH4enoPMCdlanYXs5vrwv/rrnySYPYW1BpqmjjqXPfDtMbpiWIXeUcDQXaPV3mvxbPc3ANdid0rOzyWN7g9TW6XbXCJGBw8YeTNwO8X82XI/Mupd25FwEtVrb8+K9IL6YMaNieCFz/32XasEPut7kPDKrphPTn +Nvyd9by ioTJyJbDWeA2mHDIABkxiMOyIZU6TuDczMfbmHgDehvJYkx9rBdp4CEv6YQiZfGLrqEIO7JbRzkBjHnoE5lhKp5BbQXTC4m/++fOpePS9mF4KT/8l1vKgjOM9mRD/9JoXIjMPru7Kt8cSKlQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: purge_fragmented_blocks() accesses vmap_block::free and vmap_block::dirty lockless for a quick check. Add the missing READ/WRITE_ONCE() annotations. Signed-off-by: Thomas Gleixner Reviewed-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2093,9 +2093,9 @@ static bool purge_fragmented_block(struc return false; /* prevent further allocs after releasing lock */ - vb->free = 0; + WRITE_ONCE(vb->free, 0); /* prevent purging it again */ - vb->dirty = VMAP_BBMAP_BITS; + WRITE_ONCE(vb->dirty, VMAP_BBMAP_BITS); vb->dirty_min = 0; vb->dirty_max = VMAP_BBMAP_BITS; spin_lock(&vbq->lock); @@ -2123,7 +2123,10 @@ static void purge_fragmented_blocks(int rcu_read_lock(); list_for_each_entry_rcu(vb, &vbq->free, free_list) { - if (!(vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS)) + unsigned long free = READ_ONCE(vb->free); + unsigned long dirty = READ_ONCE(vb->dirty); + + if (!(free + dirty == VMAP_BBMAP_BITS && dirty != VMAP_BBMAP_BITS)) continue; spin_lock(&vb->lock); @@ -2231,7 +2234,7 @@ static void vb_free(unsigned long addr, vb->dirty_min = min(vb->dirty_min, offset); vb->dirty_max = max(vb->dirty_max, offset + (1UL << order)); - vb->dirty += 1UL << order; + WRITE_ONCE(vb->dirty, vb->dirty + (1UL << order)); if (vb->dirty == VMAP_BBMAP_BITS) { BUG_ON(vb->free); spin_unlock(&vb->lock); From patchwork Tue May 23 14:02:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13252349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74655C77B75 for ; Tue, 23 May 2023 14:03:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1996E900005; Tue, 23 May 2023 10:03:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12317900002; Tue, 23 May 2023 10:03:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0664900005; Tue, 23 May 2023 10:03:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DB551900002 for ; Tue, 23 May 2023 10:03:31 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E654F1404C3 for ; Tue, 23 May 2023 14:03:30 +0000 (UTC) X-FDA: 80821687380.09.3A32769 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf21.hostedemail.com (Postfix) with ESMTP id E83A11C02FC for ; Tue, 23 May 2023 14:02:18 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=E8msUzrt; dkim=pass header.d=linutronix.de header.s=2020e header.b=2ZkOvG8P; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf21.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850540; a=rsa-sha256; cv=none; b=gjjtsdA8zHLKpIxZ0Vu/IsBGNRYWJYZHF3sWIleyM3y0mazk8byczaII2NfiGQKmjSIMeL 1ok1pBX9fuyTGXf4uOo3bFoDqu6a5gl6PiJThaxoTeQOK4M3b43ryuyuEy1GBHn9/VzDe8 C2MtAaOmoc4M1vdTo5auqnskIhren4s= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=E8msUzrt; dkim=pass header.d=linutronix.de header.s=2020e header.b=2ZkOvG8P; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf21.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=W0htcsfLUpxLaMjJlRhsnhJ2QB5K+K/5FLFN+owMbd8=; b=U/C6nZeWMi7m0pzHPUznJLZ9GtJybkJCNh/X/TDsf5A4lfUPAe4KAu1sL665R1AYJgLSnQ 4hFa3pUn5kHS4B7bgaQqCyDqzISCCEo9hZA+033WgbAeVT3e+O7jIZgKPql0OCA53HjdOU mjZ0DU+ocd7vQC7jq9etYVJC4tGJ1Xs= Message-ID: <20230523140002.852175941@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=W0htcsfLUpxLaMjJlRhsnhJ2QB5K+K/5FLFN+owMbd8=; b=E8msUzrtsiWtA2B5JJElvZ9RMci1JDZlT3dpqp7cFBl++PlTBFaUK7oAi3CmEOhk0GJLia Qpktjc9Lq2CMpsxEjhl2zM1c5dsyGr31VoQqRSGzSoueYsDJYr7zMj9ho6ZtJSaTsI6+P/ bu4GuutGZC0QDWZDxITmNZieXOlSrEEO36AeKmFcqP10Ml21zXS20vH7I4vzIIoYG+aA1D QfbHVXMrxAAXvtjG3ZbTrdLepc5XGWSYMtwWtL8TyuwkB8LBTfhfKd5C2hEZfvPOuDWKWC Q7R9E8Z/P3/gI+NjUdoeSAa66DOb0rT4+GT5dvOtz4kMqhUbgVhHwt2vD5WZqA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=W0htcsfLUpxLaMjJlRhsnhJ2QB5K+K/5FLFN+owMbd8=; b=2ZkOvG8PhJJhu+JdOqY+10Ly9n1nLuRhAsQTa+gmieHFEKo+gcYkT3As2ODkg2iowsRABX GxdoteIlF/RD9CCw== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 6/6] mm/vmalloc: Dont purge usable blocks unnecessarily References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Date: Tue, 23 May 2023 16:02:17 +0200 (CEST) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E83A11C02FC X-Stat-Signature: acpdsw8wgcn1aofp6cissfza8hyfq395 X-HE-Tag: 1684850538-921657 X-HE-Meta: U2FsdGVkX19eBfs7DBfXWoNW5Qz7MsQpCOqT0GyRI+l29ZsTHzmImwrVnD+nkztpcrUzewJDUW2Z9mzvZ3XLjAmd1C5raGRcfY5QDxfFHHAD5kd/Hc8YlrthWBGFDyr+WX51g3+jgXWiLWc9RIJXAxpT22hF7I8JpFNMyTZFm1qoLMjk1bUVcGiMEGLlwWR4Vy9pTMs3sKmTDWSnEse0IrC73322y3w+bYQpyWWot4u6j/oOGpNluKill+IoVOwsnlTvhibIwDpH43p2RedesY0kJ2TsdfCC+0Mlb3hVWJfxD6wXvdLSV6cWdcqulSjDyiYFDhSTOl8f0Ho2TgfgXUGSiDYGXKKG7I2NJJYfZ9oDHn+vEEgtzFwqshUCa7s/Vdro+WlosAMIHww17KNwaxf+Jfn+GQ4+Q9+0qMLbxxO8Z1njAWC1lUQlS1aIVhedGNLOIL/WHE3Kb440PeUrI/SNLiIOQ7sj/Q2+un4suiorHu8CAxFQaqFbv/vrzFAqfdDAAdGLkfLrZ+KXJD6mBroKShGX1SogQRTtWeU4lDU7GvOivTlxl9xINMWAsaiuWidhmhRWJrJQjBID8gBT5PsaQyEowD8kgp74YUXasg3pICZ9fcjieYXtzpDfa4m1hke4Xdql8RF0Ptz28nNyMMhm3aLj8OZkMXYSB4Ynp01GmAwW3WhqT7ZYMo2C/+mxNtf2R+Q2u9h2vtyMBh/3jEHGL9A0RKCV31A6kjtnkkitgsw15YosyRF5JJ6ongoRTJbN42BKWJNcinwA52HVxtdFIiGVcDNhnOKjy4lNBD57ONVr/LkQ7Y57l6VhaiDZbm/AM/OuGRMu/iSTgUaTz1RpMB4nOwSR1OMpV0sYcCOEXlqqmBCsp0o1fFzNnvbm/HOgFwnV3tPJFlmYMZ7zg9hkgx/sc2rR2ibG/+/PsHo3NNKN2hhV1tJP5musGKMvrdPM8K60AVpx3nBUEdS yUl/VP+9 EXbu11VnPbaP9Alc+3lZ/5Cz6M4pkzC8LBDBeKD8g71qC/WBGtUiq+Kk4Pp0VxvkjQbe5LhLJoYIyckQcdIq1MciJsr6hZb26K3Yf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Purging fragmented blocks is done unconditionally in several contexts: 1) From drain_vmap_area_work(), when the number of lazy to be freed vmap_areas reached the threshold 2) Reclaiming vmalloc address space from pcpu_get_vm_areas() 3) _unmap_aliases() #1 There is no reason to zap fragmented vmap blocks unconditionally, simply because reclaiming all lazy areas drains at least 32MB * fls(num_online_cpus()) per invocation which is plenty. #2 Reclaiming when running out of space or due to memory pressure makes a lot of sense #3 _unmap_aliases() requires to touch everything because the caller has no clue which vmap_area used a particular page last and the vmap_area lost that information too. Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list. But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally. Add a @force_purge argument to the relevant functions and if not true only purge fragmented blocks which have less than 1/4 of their capacity left. Signed-off-by: Thomas Gleixner Reviewed-by: Christoph Hellwig --- mm/vmalloc.c | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nod RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) -static void purge_vmap_area_lazy(void); +static void purge_vmap_area_lazy(bool force_purge); static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); static void drain_vmap_area_work(struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); @@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area overflow: if (!purged) { - purge_vmap_area_lazy(); + purge_vmap_area_lazy(true); purged = 1; goto retry; } @@ -1717,7 +1717,7 @@ static atomic_long_t vmap_lazy_nr = ATOM static DEFINE_MUTEX(vmap_purge_lock); /* for per-CPU blocks */ -static void purge_fragmented_blocks_allcpus(void); +static void purge_fragmented_blocks_allcpus(bool force_purge); /* * Purges all lazily-freed vmap areas. @@ -1787,10 +1787,10 @@ static bool __purge_vmap_area_lazy(unsig /* * Kick off a purge of the outstanding lazy areas. */ -static void purge_vmap_area_lazy(void) +static void purge_vmap_area_lazy(bool force_purge) { mutex_lock(&vmap_purge_lock); - purge_fragmented_blocks_allcpus(); + purge_fragmented_blocks_allcpus(force_purge); __purge_vmap_area_lazy(ULONG_MAX, 0); mutex_unlock(&vmap_purge_lock); } @@ -1908,6 +1908,12 @@ static struct vmap_area *find_unlink_vma #define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) +/* + * Purge threshold to prevent overeager purging of fragmented blocks for + * regular operations: Purge if vb->free is less than 1/4 of the capacity. + */ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) + #define VMAP_RAM 0x1 /* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2 /* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3 @@ -2087,12 +2093,16 @@ static void free_vmap_block(struct vmap_ } static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_block_queue *vbq, - struct list_head *purge_list) + struct list_head *purge_list, bool force_purge) { if (!(vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS)) return false; - /* prevent further allocs after releasing lock */ + /* Don't overeagerly purge usable blocks unless requested */ + if (!force_purge && vb->free < VMAP_PURGE_THRESHOLD) + return false; + + /* prevent further allocs after releasing lock */ WRITE_ONCE(vb->free, 0); /* prevent purging it again */ WRITE_ONCE(vb->dirty, VMAP_BBMAP_BITS); @@ -2115,7 +2125,7 @@ static void free_purged_blocks(struct li } } -static void purge_fragmented_blocks(int cpu) +static void purge_fragmented_blocks(int cpu, bool force_purge) { LIST_HEAD(purge); struct vmap_block *vb; @@ -2130,19 +2140,19 @@ static void purge_fragmented_blocks(int continue; spin_lock(&vb->lock); - purge_fragmented_block(vb, vbq, &purge); + purge_fragmented_block(vb, vbq, &purge, force_purge); spin_unlock(&vb->lock); } rcu_read_unlock(); free_purged_blocks(&purge); } -static void purge_fragmented_blocks_allcpus(void) +static void purge_fragmented_blocks_allcpus(bool force_purge) { int cpu; for_each_possible_cpu(cpu) - purge_fragmented_blocks(cpu); + purge_fragmented_blocks(cpu, force_purge); } static void *vb_alloc(unsigned long size, gfp_t gfp_mask) @@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow: spin_unlock(&free_vmap_area_lock); if (!purged) { - purge_vmap_area_lazy(); + purge_vmap_area_lazy(true); purged = true; /* Before "retry", check if we recover. */