From patchwork Tue Aug 29 08:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13368664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A7FFC83F14 for ; Tue, 29 Aug 2023 08:12:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5880F28003E; Tue, 29 Aug 2023 04:11:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 510D7280037; Tue, 29 Aug 2023 04:11:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38AC028003E; Tue, 29 Aug 2023 04:11:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 24CAF280037 for ; Tue, 29 Aug 2023 04:11:53 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E6ED314066D for ; Tue, 29 Aug 2023 08:11:52 +0000 (UTC) X-FDA: 81176423664.25.20F7ADA Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by imf20.hostedemail.com (Postfix) with ESMTP id 1FD391C002D for ; Tue, 29 Aug 2023 08:11:50 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=C+0ZPJKe; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.181 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693296711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X9p+6vvdULZ9qjvQwfG8p+E9kLRCcHqiWfMDpSf+E74=; b=pI7prEhfjqaFtEL75SU8LOUcV8oItgpT03smBYOOv3rJM5LFucH+Ck+dmi++OwDkooFUxh V0Z4fWfMBfDWs1y9xav8B8wj1PPWVl8hKSiY+zP8VvFh/bPtarShfERlZ/MCg03k61EiBL C5WqyLum8RAKC2lXDzY3k9DzSBdD4NY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=C+0ZPJKe; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.181 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693296711; a=rsa-sha256; cv=none; b=TAuDbMkJSjn+EfjsOv5aiXb98CQ0CqN0KhjZCvLQtPvuQFpBWrxD44+7AUBNiQpSB8qK5i FerLNDSL4/IDxLelfDgTT0P+HRJAlQBSKvnnxSxfNeLefECgPANaw3WL3dhdBOrSEBJ8jt c12gzYSzS0EnWl6OjqcuCU7lOxMbO5g= Received: by mail-lj1-f181.google.com with SMTP id 38308e7fff4ca-2bcb50e194dso62070171fa.3 for ; Tue, 29 Aug 2023 01:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693296709; x=1693901509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X9p+6vvdULZ9qjvQwfG8p+E9kLRCcHqiWfMDpSf+E74=; b=C+0ZPJKeh2AIn7VDgao2/FXWVrLlcnX1KA3YLoJi7cwPkwXL9e5UDGxjNsRNkYgr/B QjzB0HDDO9cPrCbMbmzlbj58HYnOEwytSJhrd1l0DnARX/+hknfvFf2YJASBluevQcEF +yzAAGe/wZSpO9edu3hwtl/H38HwNAdqnmh/yA0Zbszg1MKGDQ56gb0c2lfUNJkKdMlB OFqll7OSQXP932CL5iFUyLqfIJQIr8tPj0euSy+qV8bKyVZaU1zZjBccPJ/+qXwW5q4Z 4WHcDl9RIYZZxsBZfJkiqYhk9NRoTEKZAywVQarXxpFzSE/gcYhUE4lKKGSIP24Hj6Xu 8IGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693296709; x=1693901509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X9p+6vvdULZ9qjvQwfG8p+E9kLRCcHqiWfMDpSf+E74=; b=FAweNELdwQ8jvz3MJoB20BJ+LCqLioxgAAX5clSjToAaUFRfFnMVX7n4cMk6ikM8pf 2RoV1ZiYC3q38xrwoguLnUhCt6F/MBh8liwJQW96mEY7muidRioS+cV5hKxmh9bnBGlk z61DYw0I52JEp9FKwwHeWp6sVgJN8HX5XF+XnPJKCMIyZuj/Qx9J/EpXvbJuJMwmAPX6 G4NcWWH7ndPE6BCzNwSYlW162YyRDYJC7o2r5A0py7OQ+F7yBs+znMFwKLAvKqWTo4oc m1ztnT5Q2tEcAsEpCsFDpV/3CZGRKOdFdg2IJGSOsCC3iDzM0qdzfA5V2JQGqXwfiwLG Hmsw== X-Gm-Message-State: AOJu0Yw6/VH/U8ztP2OVO2fJFOe+DPHzOUCInRGj7CZXj1JU+O8K3Sdx PL+DQE8xdbgHPEnD1XkQn7/q4SB8Adg5pA== X-Google-Smtp-Source: AGHT+IEC0NgW8tXM5qfPn6xExqsQCK2vQIifZS1j4TtRZBSspdGvCpkqpIJnDNq8vdHZtzaDCvPsdg== X-Received: by 2002:a05:6512:3115:b0:500:79a6:38d4 with SMTP id n21-20020a056512311500b0050079a638d4mr16193053lfb.40.1693296709391; Tue, 29 Aug 2023 01:11:49 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id f25-20020a19ae19000000b004fbad341442sm1868026lfc.97.2023.08.29.01.11.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Aug 2023 01:11:49 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter Date: Tue, 29 Aug 2023 10:11:40 +0200 Message-Id: <20230829081142.3619-8-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230829081142.3619-1-urezki@gmail.com> References: <20230829081142.3619-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1FD391C002D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1rn9xoxxtzzkj6fsogfkr1gguxj5byot X-HE-Tag: 1693296710-950747 X-HE-Meta: U2FsdGVkX18bBdWl9EloRPVwcMCnOmOprzR4pZgcrBnSAMJvVeAtiTt9OlCFkpmroarptdSMeMK7wqHBr169glzNK6sZkg5gTYMiM3GwgTJ/D79cK584baVBzBB7PPPrh+4ZoRMY4MC6kDZY0TpYHnYnlegzbS3ONIUTkZxgexnXgSp+13c7Wb6DvbDgy6YBJbn+l9TKHiiRUBtFvJyVMv0sz/GJCtLAFugyHaD0K04kZph9NC+V4sVtryaj8L6fyUtiwQ0nvMFDZmatrbL3K6SMO9JFYvKvCwZIPI3jA0DPxqHP9TsAIxF4+ppVXJeEKiI3/B51VKKhZqjXiJB3ymJHPnSKsQ5AKs6G4OyYNprAvyweCD+p7K1ReV9y9EO/yT1ZKvFh0zN3bJxtoEY2gJAb+Bzdtvpb/0admWzAD3oeZBWA6OHOvRb5gJGmgMZGCTpQJnY2PjyXVDDwVNz3vV3YHQB4DQjhrq+wHbfUBhKi+vdpr84fnTAME+sfB6Bh7WxJi5TP+FUDGF3gcrhfqAmiLTgvOSATZJh2hTNYvU9QaZ//WTf/0UX620PmQlRBzw8bV5EWbQdh0xGyYg03bcbfqCGhqO1x+71RdL8063q5wFRd8vUZD3EwIUufM1a+dQFTF/jyBGz7KwWGf8uekC+bmfXfU2i0yGBLsQsaS6EVASOQEAV9BFisDR8e+43JfJYwfLEANXv3dtxreDPW4W1PG1OfRSi7MaaIWOLFdsG3Vw2RgDX2+9ohl9dXRCQAa5BRAXR/wHswqJGEKrj1w9l4itq0QsJYxgp92SYg74huOU/i+QIrzQcsb6goc4TjZL9XRv2eO+qqYWSP5d/HeD0oFoJ+7OEsbyF2CKH+UF/V4CKfetjwcpaT4CiU4ZfJcFDTtBZxmPNdCTBcDGqGu4yEWR7fCk+pd+qpoGa2eGa7dUk70StTs0KZkmamCy9vSNNa3TQi82DhW4msUTA oSMOR99C /eqgTrfiQzYOl2tR+TsUPLA8kYpTh5sDz+s3gDXWOIKMNV+hHxCiF1EheQEjCJWtvSfQt5viN0wUkrCkHNLrEzhPBx76cENDvbnh4kPTyB01NG5IZ5EmC+mhOW6ewy1uy5Ky5rTO2ARN5YjQYAjakJxKtIO19OBjvIKh0fHWWP0nyq8Cpkv2/tIwiDmArGh6jIg9AycCHOSeSKiaQNJhL3MfqWUPZPW1PzgMCtT4Os0z90u0rlikzTHwkmbwlAsmNsndQCnzvUwNBZmtFB75m+ZUTK0HD2uHtKd93ZmPrf/37HYbP7T0jBhC5fAuWKNQhNsZcvXuWbys8vU7EKCQzw7L8VakbwbI1Ju/UULXmSTdF9GeKi3Z4CCjCdYIwNvjiUX2PCETALcFoRXXIsvag58EUO+dhFJzoSTdYiOkR0zdMHHhr3F6ogquMV3vKwyw5jXdXEebbkyI0vnZrPRJsgm6maZ5nw9qCDUatgb6ikR9MnVY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extend the vread_iter() to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the /dev/kmem correctly reflects a vmalloc memory layout. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Baoquan He --- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4fd4915c532d..968144c16237 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -870,7 +870,7 @@ unsigned long vmalloc_nr_pages(void) /* Look up the first VA which satisfies addr < va_end, NULL if none. */ static struct vmap_area * -find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) +__find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) { struct vmap_area *va = NULL; struct rb_node *n = root->rb_node; @@ -894,6 +894,41 @@ find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) return va; } +/* + * Returns a node where a first VA, that satisfies addr < va_end, resides. + * If success, a node is locked. A user is responsible to unlock it when a + * VA is no longer needed to be accessed. + * + * Returns NULL if nothing found. + */ +static struct vmap_node * +find_vmap_area_exceed_addr_lock(unsigned long addr, struct vmap_area **va) +{ + struct vmap_node *vn, *va_node = NULL; + struct vmap_area *va_lowest; + int i; + + for (i = 0; i < nr_nodes; i++) { + vn = &nodes[i]; + + spin_lock(&vn->busy.lock); + va_lowest = __find_vmap_area_exceed_addr(addr, &vn->busy.root); + if (va_lowest) { + if (!va_node || va_lowest->va_start < (*va)->va_start) { + if (va_node) + spin_unlock(&va_node->busy.lock); + + *va = va_lowest; + va_node = vn; + continue; + } + } + spin_unlock(&vn->busy.lock); + } + + return va_node; +} + static struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *root) { struct rb_node *n = root->rb_node; @@ -4048,6 +4083,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) struct vm_struct *vm; char *vaddr; size_t n, size, flags, remains; + unsigned long next; addr = kasan_reset_tag(addr); @@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count; - /* Hooked to node_0 so far. */ - vn = addr_to_node(0); - spin_lock(&vn->busy.lock); - - va = find_vmap_area_exceed_addr((unsigned long)addr, &vn->busy.root); - if (!va) + vn = find_vmap_area_exceed_addr_lock((unsigned long) addr, &va); + if (!vn) goto finished_zero; /* no intersects with alive vmap_area */ if ((unsigned long)addr + remains <= va->va_start) goto finished_zero; - list_for_each_entry_from(va, &vn->busy.head, list) { + do { size_t copied; if (remains == 0) @@ -4084,10 +4116,10 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) WARN_ON(flags == VMAP_BLOCK); if (!vm && !flags) - continue; + goto next_va; if (vm && (vm->flags & VM_UNINITIALIZED)) - continue; + goto next_va; /* Pair with smp_wmb() in clear_vm_uninitialized_flag() */ smp_rmb(); @@ -4096,7 +4128,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) size = vm ? get_vm_area_size(vm) : va_size(va); if (addr >= vaddr + size) - continue; + goto next_va; if (addr < vaddr) { size_t to_zero = min_t(size_t, vaddr - addr, remains); @@ -4125,15 +4157,22 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) if (copied != n) goto finished; - } + + next_va: + next = va->va_end; + spin_unlock(&vn->busy.lock); + } while ((vn = find_vmap_area_exceed_addr_lock(next, &va))); finished_zero: - spin_unlock(&vn->busy.lock); + if (vn) + spin_unlock(&vn->busy.lock); + /* zero-fill memory holes */ return count - remains + zero_iter(iter, remains); finished: /* Nothing remains, or We couldn't copy/zero everything. */ - spin_unlock(&vn->busy.lock); + if (vn) + spin_unlock(&vn->busy.lock); return count - remains; }