From patchwork Tue Jan 31 15:36:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13123120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E389CC38142 for ; Tue, 31 Jan 2023 15:37:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37B296B0071; Tue, 31 Jan 2023 10:37:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 32B326B0072; Tue, 31 Jan 2023 10:37:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CD3A6B0074; Tue, 31 Jan 2023 10:37:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0F4056B0071 for ; Tue, 31 Jan 2023 10:37:03 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E007D160CB6 for ; Tue, 31 Jan 2023 15:37:02 +0000 (UTC) X-FDA: 80415497484.04.55395AD Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf05.hostedemail.com (Postfix) with ESMTP id 071C1100012 for ; Tue, 31 Jan 2023 15:36:59 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=fDdFeQR7; spf=pass (imf05.hostedemail.com: domain of axboe@kernel.dk designates 209.85.216.51 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675179420; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=EQY0o2qkAKYFH7LWXdvXFVGAqLi+sKaNC+LqwIIUvZs=; b=2vxb0jU29CEeXga9RoBoD/fYV4mf+qti310D+J2TEutcX5yrYhMSMlH5CPQFiMUN4pmalL 8vyVINbG3zJpCSZAmT7neupc0z25moa/Vnt7NRbQAgDMRsbP5Vhm6I6Ht4yPM9BD+O+1j4 bXDdKl4g0I1C//tQGggc6v+USnT/6z8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=fDdFeQR7; spf=pass (imf05.hostedemail.com: domain of axboe@kernel.dk designates 209.85.216.51 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675179420; a=rsa-sha256; cv=none; b=B9UbczByXDMQGWt/D4dvctPgK03jpovhUDAaQflRU75tH6kpRaZkUB3Zr6q9ED/OXCrtHJ gZU9oEN9KEYu9Ql4DHTPSi21tymbGeN/Qa7M/f8FKc4cC/OT/NY5eFee3FsaaGBy4ewfNd BbOFcHDzyIh0BXi+p8EfHMJ6IrumgyY= Received: by mail-pj1-f51.google.com with SMTP id mi9so5372160pjb.4 for ; Tue, 31 Jan 2023 07:36:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:cc:to:subject:from:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=EQY0o2qkAKYFH7LWXdvXFVGAqLi+sKaNC+LqwIIUvZs=; b=fDdFeQR7aga3Mvpbjdn5BMRfIUa9WGkRrnp/3BaPk5OCOCRntiI2RQBBmupmnHxtbV /9ZkeAKJuemHbqzNl0JOUVvItnex8KdwqlGZlvAGa6HWbTyOvi2XeCMSUVx9Xurfcwzt YGq3fMG3VWd8Rdrt7pqldTYVaaOiR7/WMGk8aN5eZ31oYNHbOJLyS6tQs4PwRX1DYYfs C1i0JwRtm9994RjRYeCXo09BPQ7440tV/jk3BKSsyPapoO1YKZlyIys1K/74gnCFObYJ jXdy3B/UubW0Xk735auyRVglaXajPusx5bOnxTe9u/22OGHLINRcWu2ph6bVsCoNcjDX IBCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:from:content-language :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=EQY0o2qkAKYFH7LWXdvXFVGAqLi+sKaNC+LqwIIUvZs=; b=2G0aAumQ85TvMpTah7qla2KSh+bo9RfFYJwUNhxFcm8s0pYd7MCZ1hgGdbJGUy11wZ /H+fllHEy9yoRAGiRxxa0kOgS13CxlKemBZyMN139Q0/pFc/GKOEIcKHhwyNF/RAG2ud 92q9xgsHWg8191pe+c/aSvgGDTAgcYaP/s/OwNSgyxzlvsHCJsPSECR7gQ5lAC5aIxKH 4jTQSVmiVJ6azPiWuJnHxRt/VtrCh2fhepKo6rqCSB+wjb59gGcXN9fEczod2wJtPauY FHWmnUNajWILvlEg76XAgbeyHPqkf3Y9qatc+rs0+/z2FKBhUXhnSvgZyTbO0NX3B6G9 WA8w== X-Gm-Message-State: AO0yUKVuoJPtleLW5I70ggzPYM2VsEBEWV6APckwn7xjeWb0iNmrA5FQ JJzJFEf8yQiC4xI5E793zIiOAA== X-Google-Smtp-Source: AK7set9xzGsvDHDfGnInyRCfENWzJmt90g2AlvtJtH94RCocYUsoAiR/Ep4M4tSg3ju6CCwnbPRd+w== X-Received: by 2002:a17:90a:7c47:b0:22c:2483:2401 with SMTP id e7-20020a17090a7c4700b0022c24832401mr6802571pjl.2.1675179418822; Tue, 31 Jan 2023 07:36:58 -0800 (PST) Received: from [192.168.1.136] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id gf17-20020a17090ac7d100b0021904307a53sm8917119pjb.19.2023.01.31.07.36.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 31 Jan 2023 07:36:58 -0800 (PST) Message-ID: <54b0b07a-c178-9ffe-b5af-088f3c21696c@kernel.dk> Date: Tue, 31 Jan 2023 08:36:57 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Content-Language: en-US From: Jens Axboe Subject: [PATCH] mm: move FOLL_PIN debug accounting under CONFIG_DEBUG_VM To: David Hildenbrand , David Howells Cc: Linux-MM , "linux-block@vger.kernel.org" , John Hubbard X-Rspamd-Queue-Id: 071C1100012 X-Stat-Signature: suiabhrcs8g14385wb41crsj5ctbcgep X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1675179419-380939 X-HE-Meta: U2FsdGVkX18+5IEhzCGL7xclLA4YLfaP75LrCkBR4y6BVTuhRe2WbuivacHv13fgRcXBRttgO2t9R5HMET0heGIUKseGAjAucyi3XLZdifbdeLgmiaWV8sLplVK+chFUhKqcNYWbYOkM593KSHoGNfq4VzZkrko4W8KGeURe6v5sHWqxfvzACR1RWGLKB87tyuuX0nfRsy735caSM9oHtVXaG/LB13uNQOGV6Z+4hbPZSIW4TZpzt2RLMD+fXOgImnWH92Ezc7kJG/uISYhBYaov/lQw620KDt7+x6zNxD4fQLh8sRKzuqNii/vXfJuxLJej7EenUHtAgS3j+fr6WRd1iWu8W2RK7tBmsRVsPzxChrEz7lJ402MC3szoo8pXk4HT/hwsYNRZJsNaXw5hBp2danGJxwmeBm87MSuNMNfL8KvyvsjOJOsPUPyyVJl+8UpvqxAsfdylLBLHIWwbS7E6FxL57gbMaQNOmGn+kSffeu3yaX863nbf8nIZePqKJVgoZxF+GQl1Ix9fPtYnivmZ1X1HZqpJM5uAUicl4TTP2JDxnMLqqA6Bjt0wNsU8WDdNR7G1G4fY6MpIsUlWnjieNI/hNsiwEomb8xp997Mpc4SIFbndx91Lo/7Cp/xhkut30k7yiOHa/HVmJhwxdWNu2sMlBZ20uhVuXjo34WafiYegkZz+v9454+LZJtFF0vJUvVWhmIzKP6Joj/qDzQmzE9u2avIomg0mE5R9xc8s1ZrFtxIIntQQoIFObsVHtMxe+IS9lkdGvVa8oT0FHzC7M8qWSM6L4COSB8JlHNlQ3lV8u0qDVDIqAhTmrk87S8IAom/mcCgrOJ1tGLsSVPv00hJKk+USW1VjVZkDJvfXpo4R8cqKWiYBRowPUME5yfsOAYwBdkej6uToE5BWhVP6bXUyW+8kXr4JFVpZ9wfYpZkDRIOpqCN/2A8KruHRPvIybDYfectqyfJGHyk kwfUHakR L0Y/zeFyF3gFjKQmvJuVOWYc64RLWiAzPbCAwg4vg+SD598oMIwJUeC+Uprskdl7wH8/O/bUuBfeFi4+AB/LyV7eX0QE1lvP+z/S7cZkO1vrzy1wh/d6btB15RJSexjGYzSYpFaIcFDMLE6zKLXbynOCqtU0eN0yr25Aqi0VlBe62KvG9daJmvao1HHCmTfeuR927dU0or1rtnZD461PaWWQv1MBrTuBjvrNYZZUPzvFrmRn5D0JedPMP7/4uizoEYfg67dQ+jhGi4dHAIYem2aYWsxkJxWe+PycvKqC8gupY7rDVbCTe8UYleNtjo4b3BBPigTKAqOdU2D34mj9DlK9ptnAXScIMkDu2NlctgVajsSAwk53bIgXAYMrKiTKANKwg6jwNJjgWACMC3nDZwTyC3EYRM/74swb4aRjegXIB3W5F7jhxdAfLKnvg2hPyw9+HykP90Aye+h1nCQQM6hOGjazP61chjToVOtSfNHDNZuviLm+fidJJno3TMcLrkHic9FoHALsYTofzCfMJT9SLNz4X1O2oA0Lo5fuMg/LTJ7E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using FOLL_PIN for mapping user pages caused a performance regression of about 2.7%. Looking at profiles, we see: +2.71% [kernel.vmlinux] [k] mod_node_page_state which wasn't there before. The node page state counters are percpu, but with a very low threshold. On my setup, every 108th update ends up needing to punt to two atomic_lond_add()'s, which is causing this above regression. As these counters are purely for debug purposes, move them under CONFIG_DEBUG_VM rather than do them unconditionally. Fixes: fd20d0c1852e ("block: convert bio_map_user_iov to use iov_iter_extract_pages") Fixes: 920756a3306a ("block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages") Link: https://lore.kernel.org/linux-block/f57ee72f-38e9-6afa-182f-2794638eadcb@kernel.dk/ Signed-off-by: Jens Axboe Acked-by: David Hildenbrand Reviewed-by: John Hubbard --- I added fixes tags, even though it's not a strict fix for this commits. But it does fix a performance regression introduced by those commits. It's a useful hint for backporting. I'd prefer sticking this at the end of the iov-extract series that is already pulled in, so it can go with those patches. diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cd28a100d9e4..0153ec8a54ae 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -195,8 +195,10 @@ enum node_stat_item { NR_WRITTEN, /* page writings since bootup */ NR_THROTTLED_WRITTEN, /* NR_WRITTEN while reclaim throttled */ NR_KERNEL_MISC_RECLAIMABLE, /* reclaimable non-slab kernel pages */ +#ifdef CONFIG_DEBUG_VM NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */ +#endif NR_KERNEL_STACK_KB, /* measured in KiB */ #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */ diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..41abb16286ec 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -168,7 +168,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) */ smp_mb__after_atomic(); +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); +#endif return folio; } @@ -180,7 +182,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) { if (flags & FOLL_PIN) { +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); +#endif if (folio_test_large(folio)) atomic_sub(refs, folio_pincount_ptr(folio)); else @@ -236,8 +240,9 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) } else { folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); } - +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, 1); +#endif } return 0; diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..5cbd9a1924bf 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1227,8 +1227,10 @@ const char * const vmstat_text[] = { "nr_written", "nr_throttled_written", "nr_kernel_misc_reclaimable", +#ifdef CONFIG_DEBUG_VM "nr_foll_pin_acquired", "nr_foll_pin_released", +#endif "nr_kernel_stack", #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack",