From patchwork Fri Feb 21 22:25:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D663C021B3 for ; Fri, 21 Feb 2025 22:31:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E151A28000C; Fri, 21 Feb 2025 17:31:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC479280004; Fri, 21 Feb 2025 17:31:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3EEC28000C; Fri, 21 Feb 2025 17:31:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A21D6280004 for ; Fri, 21 Feb 2025 17:31:18 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5295C120894 for ; Fri, 21 Feb 2025 22:31:18 +0000 (UTC) X-FDA: 83145399036.26.8CC4F47 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf29.hostedemail.com (Postfix) with ESMTP id 4D84912000D for ; Fri, 21 Feb 2025 22:31:16 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cc3F+I27; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177076; a=rsa-sha256; cv=none; b=yssrdK7zVZgRfbsgnvaF1R13T7MFRqBHwLCP1ss1ek7Zwb82Ej8XtXXpz1TyIzjBryAC++ eI/PYfnyoYdL9R5SfWj2JW6josinKHwr6aM5C/FhyTNDX9vQL1ObqXWEJrTa7JZI5PwG+q caC3c1ftUo00XC73sWKvXXF3xod0RPk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cc3F+I27; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=zmvzJF0VA9q1hcetO/BOlP33R5jUtB2JXKI229/MUoPo30gx9OVGmR7hm6HiOiExsSvcgf C9qXz7C8ZzrqcQElWOk3BwK38xOHOnnh3CHh92qoJmnc3rr+ihQEqlxOVCTbAl2vPbkwN7 eOfeKJHa8ohTpCIxUzclp5LOA2JzaDs= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22100006bc8so47876155ad.0 for ; Fri, 21 Feb 2025 14:31:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177075; x=1740781875; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=Cc3F+I27A7vjELCOvmienxdunY/kbqHuUAFyM2YlU9p+36xPdDMNgzQ7QRd9DE9AGp EdbXSKTqJce9u2Sr/YlJEKiBUlQNVtlUIfhE5MM+frQvgSFY3l4c4dmMSxW3JBa3/yfD LxYVW/fSkB3mhGfVx1mP6Mr7g5aIWkJP6q2gI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177075; x=1740781875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=i5UjPZwFlnuWpwc/hS6oD7gXnGgqPCkC6S6zltUqTNrX0TZSmJifhSLR2/+yizjkar xOQXMhIVeACjc/7vCG3rm/6i7dm7CmD47RP3nJFfAXBQCT/ERNgqLxcFy/gUtdpIyoFN +X5yTTCqgEZtAfEIcsHE/nPrtdD6GVWRy5xoNiFnqeUgQvvULceQadS7lBv/gsapGOq6 HZsUWK+JhplEbL6H8Lm6Pew9pLVIFrhPq0HK+AY7HkJLKoVN2pYPHDhW+R09ttGlwlFK fxOw4RVUwdHEuAzb+hkMaIDk9jtjlqzImX55I+CDRGhuTfIz9ezH2PuYFH81UqvAmqY/ M67A== X-Forwarded-Encrypted: i=1; AJvYcCUUEByZ0mst0BGrbJgKyYY30y7eLz/Nedn0d6QzgEAV7wv5h+G6aOAynw8yDKC20DljIoqmSgUyTQ==@kvack.org X-Gm-Message-State: AOJu0YzOL+iKHMiq6/7EMQXo6iC44d39785WIqycPY6DrnzPr7enGtdY OQkfouhChIq8leWyJa3M4cnXgw0748RsgpO/mGuVLb1XxnRnAybErtVa7XgStw== X-Gm-Gg: ASbGncuGGjfNwErKrFPG6VkvBYEd20ertAE3rNKBR9JVlL/8455Khjp98e/h9c+7r85 7xexJPJ/WFMSpBxcJ1GcLMZKm3EAd0PHqDmCwI8ngQgHZy4xphR+VwZtpP/aUc+9TVqMIHGAq+2 uAyLJMoUs+kAAoyeVyB+hXYfKRgFI/yKux7YMCVaYHEFtBCRBQp3Xe0LladDfGe4LZ+qKjE44Ur JYE97XZQrR4h4jg+Ge+nOfI6lYWgD8edC2Wguy1TuL54+SOag5AjPaIm7Wx+O+IuWAQuqKrqdow irxQzMX3SWMyAkdB9NMAIm2w8Ug= X-Google-Smtp-Source: AGHT+IHgZKuXmy0HUVJX3N4A8fmolG1cpyf+MQMTUT8UUMyAXq6Z3SZOJ2I2KS9bokpF5ag/pOAYiA== X-Received: by 2002:a17:903:8c5:b0:21f:4c8b:c514 with SMTP id d9443c01a7336-2219fffa9f3mr71842835ad.45.1740177075271; Fri, 21 Feb 2025 14:31:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-22120093e41sm95374735ad.93.2025.02.21.14.31.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 12/17] zsmalloc: introduce new object mapping API Date: Sat, 22 Feb 2025 07:25:43 +0900 Message-ID: <20250221222958.2225035-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4D84912000D X-Stat-Signature: cg4cxys5ufgqbat3m9ibym5uzr3kq1o3 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740177076-949882 X-HE-Meta: U2FsdGVkX1+KrW/wLII8BtvcIl6fR4LLnGCcKgO3NDZTTsXthLkO1ING5DYL4KNJloOrKT9DNHJMEGyWzgcYU//jfL61cB3m3vqklFOfMBPbuG5sVIVqsLx8N1FX8FSuDNQNrlrFMp9aIn9YtJiD5s17Ou9jiS7r8g3EdFCuZrwVY3aZ5zMwjVWaFuIhCkj8jVCQQq5sKn9rlh4b9CSqyMQlAZCeS21UGi+uG3nHIPElljWaaSYFGksM97wjkLEChkXYEYPLIuexj75dRuKEYHOItvDNH6rVHaRgpCafoyDgcyAnTHYlIsIJEAwv2EN80SugBB+MOkzuNev/sNTJ2GZJZqBUY5j3q17urbrK5GM6AP3nZViH2/zcqcItbhg+mz1+/+on2KyNm2A7V2QTkB5XEbmNPqKsq5zzc9MhYmSm+LRYos1fDeWDhKxbj1t+BdHRJuaqB1lRTY/4suKT5qThBK1MIznR0C+PAZpF/mKlMjz5E6vpPOm2c7e6ceEQ8VEiSdv8nUFbB1LLZGBOeZR7uU/8pxOvK/AYh+WwzpOAt1JQLoGN48SMFy0PxlbjulZi46x+6p0y15ic9e4/LzRf4t2OBJnudQtjZOWY66mWeJZSkbCfmpFsXqLcbVdFW0uwzuuxs8urdKLWUgGcFUR/IlWLfyU5que6v92dYmwT1FJ7qDJCHTBGe7BlEIOZlUqV+TKsCps40Nw0n3Bf2VuejlfM/005Z+IaRch+mBZo5jB1u1kFcIXuP7te43kcUueiDo0hrWCSbHCrT/Yn3JI+qBuQXYof9TIE1zOrSxhPD357CFbD5Gxgj+sfMrPKQkVpgCSRT5fS+ibRuSioWm5QDy9cqXLNHJaXhPWJ+QXjIJGxQOSA0k9RwOZtJtweYYc57TBIfTgMVV+RhfOaAPF/WYbzPKDA7ynjLSPpEtwcKc3NbheGIT0TYOaZLDm5+VURcR6GQ97J6AqHY5w XqqOyr4W k5GV7HBoZDJkom/yuWf269EYmQw89J4E0Brgy5BCXCOdQ3UUj8r3KP5euvfTSezM6KGDOzZ44tJIWfczCghzJz8zyPdfJZwFcO2E8G6poYsqSOJuXYf00w6HCai5zkMYAwVaeJvR7E8mTZAau+QZzll2CD1CuV3n5rKuo5JsDZMpeORVmha7rcmrVHPQs6qcMt++NKvEeIXg9bVoje/vlljekzzDFVNIZF8oh7Z2k6EAOjCoxXnpyVzdQhLhd8iIEvJxles1sDxmQ92A+pzD/Crkz+rizgYVpTYBFVgecmPgju6avTfLC3YhWTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API ======= 3 first results out of 10 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API ======= 3 first results out of 10 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] T-test on all 10 runs ===================== Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s = 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. The split of map(RO) and map(WO) into read_{begin/end}/write is suggested by Yosry Ahmed. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 03710d71d022..1288a4120855 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1377,6 +1377,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class.