From patchwork Tue Jan 2 18:46:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13509279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9F50C47074 for ; Tue, 2 Jan 2024 18:47:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E41D6B02D4; Tue, 2 Jan 2024 13:47:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 493D26B02D5; Tue, 2 Jan 2024 13:47:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30F996B02D6; Tue, 2 Jan 2024 13:47:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1D9EB6B02D4 for ; Tue, 2 Jan 2024 13:47:18 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F2236A1BE1 for ; Tue, 2 Jan 2024 18:47:17 +0000 (UTC) X-FDA: 81635253714.04.7CE84AB Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf10.hostedemail.com (Postfix) with ESMTP id 495B2C0010 for ; Tue, 2 Jan 2024 18:47:16 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="cah3/AyH"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704221236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QZAvt7CkxqIe5cgK0fSSW+F0xApr58BOh4APGf1P6Lc=; b=t5u9t0qYHALI3fDVBLPgASWXcLmET2GVm18FK3Pp7/1dvNMff8wS49XwaYWhpYXHqbfsVx i0X5WsITix9e77sKWwJfu8gEyA0Q+YB36/B7vKicMzif0cRWJA5vaAVIW1zTBjKlU++aOk ucITBE361qjXCSSuMd/wopLLn0TEQG8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="cah3/AyH"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704221236; a=rsa-sha256; cv=none; b=rWwKrP4KL4SCjPSdcZyQAiiGKwcnHL2UzRC7yxTQedW541fYvjVE3mg5mdqaCm/UWaR2Dv lA9iBD3iQI0zCsIeOmodBTaFqWk6ts3FJcYUSIM8gYx8UsQkdP0CjWINyUBqt7idV7ZSjq ajwhW/u/5xVdv56qtpbOjtQ1mxQe/FE= Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-50e7e6283bdso6452576e87.1 for ; Tue, 02 Jan 2024 10:47:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704221235; x=1704826035; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QZAvt7CkxqIe5cgK0fSSW+F0xApr58BOh4APGf1P6Lc=; b=cah3/AyHIFnsoNTxXALgHeU0UIGBr9XYXlC8xyHkQ2eRrhC0yyqjplqFfTMvG2xD8d BggCQS3QVTE8wNRVS1BJ/ncS1hi6zRm/Jv4SWktJyx0fvgtqO+XuC6289LD75qj4E/QI fQrKFskDHYAMFyxA85a3eXHODJE3aFqfEfH9w3vxh1GewsAmDPaqmw9gySQTqxSRTp9i H/tltcyxBAuupglrn+KtW7I4NtUwysMc+WyIMzk2zW8Apnf9vJZdNfrjUgvv4z7XBaQw 465x3fwn3Ql531IkycMyrEp+fyCACQF+E/TVJA3wviY5S3awbGVWNt8xaZX2E984dHv9 6MIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704221235; x=1704826035; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QZAvt7CkxqIe5cgK0fSSW+F0xApr58BOh4APGf1P6Lc=; b=v6NDuzjgvSKT7usXy0iBpgX6SEhsXsz6EJhsx55V4lxLGEgxN5KKhWMdqdB59tNZ9l is+R+GdXzBUMR+tAoZlyGqJ4gsMmjvePnMhSpJrzBkvPA2ODsEPzM5a04Xp3FlVYkpv4 ULSaSnS9o0clCXX5jxjP1BxOZgGPjCqR3r2tCzw5XaSGGXS8H9OYDJSYkn9qVvnXuN9Q y365YWJzvt9hbK2Hj8tcTotO3wGcMRJgD4Hjo6GQfZcciyv8nU3oyahk6E+fsxCDBzHA UhsyOt64YmPr3NaweVA9Ook9sBX1bIYupJlmoKlYrgjuRSGW+92+TxARAUEyb07XGjJY gEZA== X-Gm-Message-State: AOJu0Yxa0/P3dYxK9mKqPKcNsfvUDBXfSUJBlnOVKPcLYC5TzyAjusWp RlrSjkLjwixYAXqk7iZgAnVtKo7kXWNAVA== X-Google-Smtp-Source: AGHT+IG2+hyU4AcsC7fH/iKMYGpwIo7u9xULgZSVpY4T2CGFn4/x55AJtPvIKYTpeIpe7us9O7g2tw== X-Received: by 2002:ac2:520b:0:b0:50e:9a53:c22c with SMTP id a11-20020ac2520b000000b0050e9a53c22cmr794413lfl.126.1704221234559; Tue, 02 Jan 2024 10:47:14 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id q1-20020ac246e1000000b0050e7be886d9sm2592656lfo.56.2024.01.02.10.47.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 10:47:14 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v3 08/11] mm: vmalloc: Support multiple nodes in vread_iter Date: Tue, 2 Jan 2024 19:46:30 +0100 Message-Id: <20240102184633.748113-9-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240102184633.748113-1-urezki@gmail.com> References: <20240102184633.748113-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 495B2C0010 X-Stat-Signature: gtrjgjsyjnouh3j31geh8mm9b6s96fp3 X-HE-Tag: 1704221236-825974 X-HE-Meta: U2FsdGVkX199CPxJDLYHuYhLY+7J/VTfbFwJGSPmhqCNZibyrLV5CBw5umZq8Kb4K8u6aVtqR6fMHibuyQ074ll2bEor2taKsopHBpDEgxnFoB9cunNjYX02/f1GNTcslOSQm7fQM9Qc7+UNpQzNgmXVl/YFCCQFyhxS4rcARusS/ejGCe9TdLzP+uYSX9FHRMy+2NRLpdRqzvMcXQ+KWsWPcvmDDyC2dZBEuCbxhClNJ9nUzFxl0uwwVnOlydRj/Q4Fn9Y6byjYk7MT0bXjclS0fdcmjU5HkMC/JhVFIXG/r/76NNM9RKccolZUv5F/j4GryFkYhOB2ANNBxN1Z1kLpOwIZ5G7r9ZqX8MUfFnlH6+LiFhZGR7gh/jwx7ezIJNC1jCJIV5+LeGCnwdjb/EF0ULgu8Uc4CACENu2wx3W90Zp+OpKT2LTWBbbI6ESd+Sbv0/OWLsD4A84pnGRMfXEnAEzvTQd5g5QTaUVYsLIn9tPmTwsiQHQzGcvMnDsPEezoLuv0g9ofVjELjv2jgtEmxIPp2oLCLjg1m37kKsWmf2Col7rVWjAX7h7e+ykzg44Klxw9aEsHpkrSOHzGf0/JiwJsN48nlUz2bhdimYSZ2JLEWmuWgyO8O24Gx237ZC7ah/cy5gP8UkvKT6ac5PB/Nbax0sA6qlyLtxVh7Wtcs4/5uq2yT+XKKy9N3PK3TzD65tRleDFoolfTHjVJvJgTKULubT4nsc+C7Mf4u/lRZw00A36HA7Wv22yPqfV8/XfmA+T1YEl3KR9UKWHPrBGRiC2Ag28Pzut6KWe0QWj9BMameif+sQwrPZ3fI1lfRogm6haOCjvPdilfM5P5AQvJ/E72Y/qGwt8IREQSXXxdefjNy3zpky1ODbHGjCRnfA5HuMjgXVC27+TUkRdrwEgGsx9SBTTA4KtmKunw1LrfnKkcOtIBdlOhSliSNjmm4E8Ht4cu5nNvhs37bo7 7knZQHMu XcmgskVMUpRJpUwVjoqe47UNjQ5lenX+R6VXRIdz+HKm+u5Dee2ikdni2TD/v0HFG2IJJE7QOlwblEPfiKTCbNpWZSQpG6ColG+LDt0AaShuxHUKTMmBAqm/HJ3/FawaZ8mcypm2/UL+Gq3L3tb7gkmnftIRGRSfd31/z X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Extend the vread_iter() to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the /dev/kmem correctly reflects a vmalloc memory layout. Reviewed-by: Baoquan He Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fa4ab2bbbc5b..594ed003d44d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -906,7 +906,7 @@ unsigned long vmalloc_nr_pages(void) /* Look up the first VA which satisfies addr < va_end, NULL if none. */ static struct vmap_area * -find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) +__find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) { struct vmap_area *va = NULL; struct rb_node *n = root->rb_node; @@ -930,6 +930,41 @@ find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) return va; } +/* + * Returns a node where a first VA, that satisfies addr < va_end, resides. + * If success, a node is locked. A user is responsible to unlock it when a + * VA is no longer needed to be accessed. + * + * Returns NULL if nothing found. + */ +static struct vmap_node * +find_vmap_area_exceed_addr_lock(unsigned long addr, struct vmap_area **va) +{ + struct vmap_node *vn, *va_node = NULL; + struct vmap_area *va_lowest; + int i; + + for (i = 0; i < nr_vmap_nodes; i++) { + vn = &vmap_nodes[i]; + + spin_lock(&vn->busy.lock); + va_lowest = __find_vmap_area_exceed_addr(addr, &vn->busy.root); + if (va_lowest) { + if (!va_node || va_lowest->va_start < (*va)->va_start) { + if (va_node) + spin_unlock(&va_node->busy.lock); + + *va = va_lowest; + va_node = vn; + continue; + } + } + spin_unlock(&vn->busy.lock); + } + + return va_node; +} + static struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *root) { struct rb_node *n = root->rb_node; @@ -4102,6 +4137,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) struct vm_struct *vm; char *vaddr; size_t n, size, flags, remains; + unsigned long next; addr = kasan_reset_tag(addr); @@ -4111,19 +4147,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count; - /* Hooked to node_0 so far. */ - vn = addr_to_node(0); - spin_lock(&vn->busy.lock); - - va = find_vmap_area_exceed_addr((unsigned long)addr, &vn->busy.root); - if (!va) + vn = find_vmap_area_exceed_addr_lock((unsigned long) addr, &va); + if (!vn) goto finished_zero; /* no intersects with alive vmap_area */ if ((unsigned long)addr + remains <= va->va_start) goto finished_zero; - list_for_each_entry_from(va, &vn->busy.head, list) { + do { size_t copied; if (remains == 0) @@ -4138,10 +4170,10 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) WARN_ON(flags == VMAP_BLOCK); if (!vm && !flags) - continue; + goto next_va; if (vm && (vm->flags & VM_UNINITIALIZED)) - continue; + goto next_va; /* Pair with smp_wmb() in clear_vm_uninitialized_flag() */ smp_rmb(); @@ -4150,7 +4182,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) size = vm ? get_vm_area_size(vm) : va_size(va); if (addr >= vaddr + size) - continue; + goto next_va; if (addr < vaddr) { size_t to_zero = min_t(size_t, vaddr - addr, remains); @@ -4179,15 +4211,22 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) if (copied != n) goto finished; - } + + next_va: + next = va->va_end; + spin_unlock(&vn->busy.lock); + } while ((vn = find_vmap_area_exceed_addr_lock(next, &va))); finished_zero: - spin_unlock(&vn->busy.lock); + if (vn) + spin_unlock(&vn->busy.lock); + /* zero-fill memory holes */ return count - remains + zero_iter(iter, remains); finished: /* Nothing remains, or We couldn't copy/zero everything. */ - spin_unlock(&vn->busy.lock); + if (vn) + spin_unlock(&vn->busy.lock); return count - remains; }