From patchwork Wed Jul 10 06:09:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13728889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7269DC38150 for ; Wed, 10 Jul 2024 06:09:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0D856B0083; Wed, 10 Jul 2024 02:09:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABDA66B0088; Wed, 10 Jul 2024 02:09:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 985266B008A; Wed, 10 Jul 2024 02:09:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7C33F6B0083 for ; Wed, 10 Jul 2024 02:09:40 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C95C541B7B for ; Wed, 10 Jul 2024 06:09:39 +0000 (UTC) X-FDA: 82322816478.27.E3F59AE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 26297A000A for ; Wed, 10 Jul 2024 06:09:36 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LvS9CQuL; spf=pass (imf25.hostedemail.com: domain of 3oCWOZgYKCF8VRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3oCWOZgYKCF8VRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720591761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=h5uZ7CARkJp0bCCnNjPnh/UCvn8scxgN13mnhGZm9ws=; b=l/QV1iWWlITNCPSMQv9W9NCND2BrhKShT2Zsy0iFDyEco3YHu3iw5n5ofU3NQ6DjygMWjq 7iBD43ZI2exCsD0zMd5LboK8hArfnZ2QR4ojoLmLrdcokGyR1MdzxHq6oG3l0DTsfUmo7W RJIET0Pxj+11kJMy2Dy7cRIBdjCdZAg= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LvS9CQuL; spf=pass (imf25.hostedemail.com: domain of 3oCWOZgYKCF8VRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3oCWOZgYKCF8VRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720591761; a=rsa-sha256; cv=none; b=rjW28AbLkFbsymrjDcOuBkMxR9e9Ysd4lPOMxPOX9SXEJHDvMAL8klVe8cJXhAjMr9WODr ZkcttDq6dq4mhtyTnmq12G8qWq0Y+DtL4+BtKctXEhK/u5bq6Sbx9EuRnvD4qK/GAe6uNg UYF07Vs4fROQTwrNb7x9d7K6Mi0i6Zs= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e0335450936so11480887276.1 for ; Tue, 09 Jul 2024 23:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720591776; x=1721196576; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=h5uZ7CARkJp0bCCnNjPnh/UCvn8scxgN13mnhGZm9ws=; b=LvS9CQuLeASg7zt4a4zzrQZ9bGT6A+P4likI1jcKFIYywdZXkWXmunBgedQ+cWYR4W T8g44eB19LbKChyrmGZjJ2lDaRmFck14FkTptDYL5uK8CrROCztyH9bVHaK5Zl2qjmfu xVWGif0Rk93zksv+GmjamWd3tJQx+1ba1PwYLnQ41CMUoH/3/gyoGF8zAI5+IFF9O3MD gg15s7EGRdIs9g16kER+y0GkFaMoS33APLxzfhOv1N2hhTxGgPTSSbjB3EKiEENSYCEP xclNChAIWpiYUvG3bY5cTeclIttg3e6Xqegjl6M80cRCGtMeIUJBzu6aZ5olykGfjC/v LufQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720591776; x=1721196576; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=h5uZ7CARkJp0bCCnNjPnh/UCvn8scxgN13mnhGZm9ws=; b=rfOMor3SQ9FAEe5vwz5d9naeUKrMTTleDd19rIlwTIGi9x0xw3nxr9PfDnQ3KyfPZ2 Ntcu0tU8bvC5l9mdyP9OR+Qu6L2UIpzIAF/hLWP66btN/q05367RL7bU9cZL2qdGhTir U7AuCmuVltQq2yCMPhdaW8yYfbzjVWqiqbq+oIIIm8lRjl/h1vPXGADa2TY//+TY04+g MApWTEh7fKH3UEuuzYji9/wOTZxJ7wyPaki7J+PzQ1XBJtHzNkejXApfyteSJo/jDHt9 I24k71caGEhxvSpoj/QMm+AXAQdWe4mfyYrZZwjYA8euZOngNFwypCnnDwpO0Fhhxjtf DNaQ== X-Forwarded-Encrypted: i=1; AJvYcCWrcZHcKCJXlg1/CAVBPUWXfiBjqlVrNqYU4DnBs6nGOmTKWl2KyiUNau10IYmmzrLO4oAhV6xir7lWoWRvayiZ6ik= X-Gm-Message-State: AOJu0Yy1vYN6ILs1GGeRRYhpdA6F4tOgLmvFjXLmTAa4FnexF43Hz7dz 5y5XRRmfqnOD2z5Sc+ZfiYPISJ/bpCCcw7onF6V228srOrj5Y3ZAHe5FQGPKv7WFmzkpWIvpGCf Lpw== X-Google-Smtp-Source: AGHT+IHC47f78BVmo+sdyjTSABTw0/Tqy6HN60awY2Lq4hntIiDkKOsp2uNyQ+wFfxFT0ijOssIk3hgSFsg= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:32c1:5ac8:f94b:a9a]) (user=yuzhao job=sendgmr) by 2002:a05:6902:124a:b0:e03:31ec:8a25 with SMTP id 3f1490d57ef6-e041b10fd77mr10931276.10.1720591776148; Tue, 09 Jul 2024 23:09:36 -0700 (PDT) Date: Wed, 10 Jul 2024 00:09:33 -0600 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240710060933.3979380-1-yuzhao@google.com> Subject: [PATCH mm-unstable v2] mm/truncate: batch-clear shadow entries From: Yu Zhao To: Andrew Morton Cc: Bharata B Rao , "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 26297A000A X-Stat-Signature: r7i75kgyxgfmawr75pgag3da6cbsu8an X-HE-Tag: 1720591776-57027 X-HE-Meta: U2FsdGVkX18bSoDhPxQHdBEHOsJ7jLfV1wg1zwiaiE9K/vCrWbUJUfRWBlAN5EO9i521iw3jimvlWIqx9Sqif/kltwsuiZgXzDKaTMp0xqZ+CsqqvSDcj64V0OlJDsU6h6IfAIi7bMYHB38UD7tDC5Yjxe5tYX0fvcN6e6vIm9UKAihQu61dfVcwZs3UOI0QREB2wMrYjD0IrNjIo1iw2XGzw7hRFpZwQ9WoawNWMRB+HrvEiIoLqQbiKxkoskDbIq8gI0S3pSKb86zQU6Kmre9PTkeVc3owUMtcR0tbEqCdC+o/pLH+3TPLxY1hssa0Greq/jvKplOCp5s1c5op5jlBS1TDpJ0RafWXM2aJZLO+U/Mtq/SEDTr2gFKwCJaHx7DIRKKD5z6LrWV3+UrX3xApV/Hifran6HmZN9P8k8FrZlQQuJjLGlPddVhySNfAmZ81YwsXmz1qkBea8+NbsFzEkWuynqeS0DqbHZBj5EePoZnDaCCsXNiTLCy4NW42iM4E7OPHGPYPoI2XkG+709Enh7rPKe/RayZX46aBKaDWeOY4MIXr9vYvC8tjob3TggwSfzFmx8mBxCBblShPI972kNeUX2xDenZJ89yoMJIrEAoCEvqg7mHZ4Q5Ox2sGhWadvFIYwxds6H4lycGWnozWJUZ49PNEcVtI/qQ2LNseR3QIhgoxTHjjL1lehEemGDtBWkAgX841j8dvjPDnzdxHbKPiE54SYNCWDA6D+erHS6qcW3AJ/ymYFFYGSP29r71yyzQPA0VDVAVqBfOtmrRuHTjuqE4dVHi2cssy1Q6Cz15ujA6mfAFmyehZKYms74QWULBml5XYVREdFpHlhu4+p+F2Lo1jP68x8XXLvT1AJvWSmVsqwAThSmpm5fZ3KR7iQdx0BaPv7lpWTrTq7mmsKRxFRyf28yJU5kZiWz7zCWXvM6ujWDIVpDapf4O4TJuBmAM1A1P8/ofBhUX YFDUNxjw +C13+3SI7lV9KDIb0CwXdJ9Ag5asRz3suCJeRkTkSIzqxoZOcFDM6xPGfP6+1E1kjL/6X3rqzB4S7sepA6QCja6E9qspyGya5Yy30/v40TBIKQ7h0hEWNmq7OVTMenQFfIIMbdG4kAvSDKIaB9P0Ixv8nhXhhlK8uWRs+LcRE3Z+3AI3bwAkyZ7D7xqBtWAqh5FYxVgScD3HS0TGr+00meLE4zsxuK6fboFEwUAp040tJX+eyMMR5HcIZuUUUHM/mkZxzZbjq17dg3aDDJJoL3T8sv0Op5qGZQScu82BWram0nfJU7SXpdG2OUeUt+8j1zQkxq73Hzw+3tDFz/n5FdxjU/OplyuA5RES7WEGyo6BZn3wuXVh7EKzMqZRcXRLtT00gLDC+UXeFetbVXP5e4Vm9EHdGiiU7udc4J7/UcPRpmCHTHgrFjDxwX+miz2cWvNpNpR57p4qnw7bpOOMZulujdvt0WH0CxVbai6JAwOXIUsFz7OYGB4cZAi6j8KI8TI23tPijl7qrs2vFDvFJHGMJIInkg+f2zHFNcNc54Gg7DF4yRU30mE2kAh84PhrXzVN3b6/V44to+OzGtlvaurXS5i8wD1oDEyCu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make clear_shadow_entry() clear shadow entries in `struct folio_batch` so that it can reduce contention on i_lock and i_pages locks, e.g., watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649] clear_shadow_entry+0x3d/0x100 mapping_try_invalidate+0x117/0x1d0 invalidate_mapping_pages+0x10/0x20 invalidate_bdev+0x3c/0x50 blkdev_common_ioctl+0x5f7/0xa90 blkdev_ioctl+0x109/0x270 Also, rename clear_shadow_entry() to clear_shadow_entries() accordingly. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/d2841226-e27b-4d3d-a578-63587a3aa4f3@amd.com/ Tested-by: Bharata B Rao Signed-off-by: Yu Zhao --- mm/truncate.c | 68 +++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 37 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 5ce62a939e55..dfb3d1f4d456 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -39,12 +39,25 @@ static inline void __clear_shadow_entry(struct address_space *mapping, xas_store(&xas, NULL); } -static void clear_shadow_entry(struct address_space *mapping, pgoff_t index, - void *entry) +static void clear_shadow_entries(struct address_space *mapping, + struct folio_batch *fbatch, pgoff_t *indices) { + int i; + + /* Handled by shmem itself, or for DAX we do nothing. */ + if (shmem_mapping(mapping) || dax_mapping(mapping)) + return; + spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); - __clear_shadow_entry(mapping, index, entry); + + for (i = 0; i < folio_batch_count(fbatch); i++) { + struct folio *folio = fbatch->folios[i]; + + if (xa_is_value(folio)) + __clear_shadow_entry(mapping, indices[i], folio); + } + xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); @@ -105,36 +118,6 @@ static void truncate_folio_batch_exceptionals(struct address_space *mapping, fbatch->nr = j; } -/* - * Invalidate exceptional entry if easily possible. This handles exceptional - * entries for invalidate_inode_pages(). - */ -static int invalidate_exceptional_entry(struct address_space *mapping, - pgoff_t index, void *entry) -{ - /* Handled by shmem itself, or for DAX we do nothing. */ - if (shmem_mapping(mapping) || dax_mapping(mapping)) - return 1; - clear_shadow_entry(mapping, index, entry); - return 1; -} - -/* - * Invalidate exceptional entry if clean. This handles exceptional entries for - * invalidate_inode_pages2() so for DAX it evicts only clean entries. - */ -static int invalidate_exceptional_entry2(struct address_space *mapping, - pgoff_t index, void *entry) -{ - /* Handled by shmem itself */ - if (shmem_mapping(mapping)) - return 1; - if (dax_mapping(mapping)) - return dax_invalidate_mapping_entry_sync(mapping, index); - clear_shadow_entry(mapping, index, entry); - return 1; -} - /** * folio_invalidate - Invalidate part or all of a folio. * @folio: The folio which is affected. @@ -494,6 +477,7 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, unsigned long ret; unsigned long count = 0; int i; + bool xa_has_values = false; folio_batch_init(&fbatch); while (find_lock_entries(mapping, &index, end, &fbatch, indices)) { @@ -503,8 +487,8 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, /* We rely upon deletion not changing folio->index */ if (xa_is_value(folio)) { - count += invalidate_exceptional_entry(mapping, - indices[i], folio); + xa_has_values = true; + count++; continue; } @@ -522,6 +506,10 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, } count += ret; } + + if (xa_has_values) + clear_shadow_entries(mapping, &fbatch, indices); + folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); @@ -616,6 +604,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, int ret = 0; int ret2 = 0; int did_range_unmap = 0; + bool xa_has_values = false; if (mapping_empty(mapping)) return 0; @@ -629,8 +618,9 @@ int invalidate_inode_pages2_range(struct address_space *mapping, /* We rely upon deletion not changing folio->index */ if (xa_is_value(folio)) { - if (!invalidate_exceptional_entry2(mapping, - indices[i], folio)) + xa_has_values = true; + if (dax_mapping(mapping) && + !dax_invalidate_mapping_entry_sync(mapping, indices[i])) ret = -EBUSY; continue; } @@ -666,6 +656,10 @@ int invalidate_inode_pages2_range(struct address_space *mapping, ret = ret2; folio_unlock(folio); } + + if (xa_has_values) + clear_shadow_entries(mapping, &fbatch, indices); + folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched();