From patchwork Fri Feb 21 22:25:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADFE2C021B5 for ; Fri, 21 Feb 2025 22:30:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 477B46B0089; Fri, 21 Feb 2025 17:30:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 427DE280002; Fri, 21 Feb 2025 17:30:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C81E280001; Fri, 21 Feb 2025 17:30:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 10C5F6B0089 for ; Fri, 21 Feb 2025 17:30:18 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8A07316083D for ; Fri, 21 Feb 2025 22:30:17 +0000 (UTC) X-FDA: 83145396474.28.D8492C5 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf12.hostedemail.com (Postfix) with ESMTP id 98DB14000D for ; Fri, 21 Feb 2025 22:30:15 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=S5I4PwwQ; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Vyf4VQS9dbJdyz0MwhcPGTpR7cg7WEhYb8zFP/ep/Cc=; b=zoDufXHh3nNCMVf63AsQxCcchUxC8XRi2E75c71EoKtWAiwvq1Fz5p5RuOwK7Q8rsWpciq Q9VpzDyRZHG81kLsI6s9AzXfTKbOmeG5FhMI8J8MjPu7pr3fts8s05sfaCv1sMsTb4wI9f GsZ40WxgXDomlH3eOXnQ05dhE2KgP+Q= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=S5I4PwwQ; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177015; a=rsa-sha256; cv=none; b=GTKDVvPc+lPg744LLdksxYf3AkYb63or4enZvAX1i4U7++ju1g00qNpakdKYiDr4Ivwaun zC8OjJvfYvfwZgAeHxbRVPFJX2j6C9dYPuby4QCNNadfkn1J59kVVTNiNkH+MSStvXYreV TjLvM+UpOtb+s4UP4XAJpVHvUUvM/RE= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21c2f1b610dso75298885ad.0 for ; Fri, 21 Feb 2025 14:30:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177014; x=1740781814; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vyf4VQS9dbJdyz0MwhcPGTpR7cg7WEhYb8zFP/ep/Cc=; b=S5I4PwwQ7SP3kxKNclIfQ7JiF90fs75gVEqYljCGXC5yplUCrJUl6tZo42kWb+gMVF Gs6nZmHyCP6z3RgeKmC/ZWFHcA0p0PBB0e/esfT9KntBDz84Wg3UC6fGaiInPx4A8FN8 7cLueqZpk83Xb39hKjzp0fszsQ8AWnNopbXzA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177014; x=1740781814; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vyf4VQS9dbJdyz0MwhcPGTpR7cg7WEhYb8zFP/ep/Cc=; b=YXdcwW/2A2+gWRQl+ThMIFZ4i/K4BM9BVjee3rNc38DV1DQIUSI2XvxNt9N7afSvyo nM9qumB9E7u6k0ZPwvIFT6W2ebVhx3rldzAGFQmzbYFwMd2BmTa7mUrnwaNgtDe2rHL2 UPY9O3G3fzxzAEBypkmthfubyyAYWPlXI1304IvM4N3s/GBOj+32nshlunlrmMnLS2R/ m1v9NjPdQD1CObiqBemIOUWsY0RnrizuSPzgBi20T39sgNH1JPTt3+IdVYJ6vKh2E+m3 PBNo9ISXYlpwszU25htJ0UNJGtZCC8Ve1EG4ekojn4wzJ6HT+ce8HssBJTMFHyy18PUI oxQQ== X-Forwarded-Encrypted: i=1; AJvYcCUaEmvWBonAlwoZj4fddcNMSRfPqHGE8TCMi0zJUkQB68143ohdQdlG89LAuw2h5QRmgb/vZ0Ir/g==@kvack.org X-Gm-Message-State: AOJu0YxCbE+ynt3iNDhzroRFAv6kvWoTT7iooFKBVa8aBPAqtC4F1eH5 pBls+llYg3jFyTvZRQzdJH120Bt66LwkVilX3LhhUnqcsYaITlrEYuKkxMrjxQ== X-Gm-Gg: ASbGncuoHMLoh15WQPT31Ia1abI4FO4/4+tLrCB+pI32aygNS65iVANQFHCXCNQ8J3D FkEBtfv0VDeEas1l5ZaImcBwFtwPlHCjBYWhSH2paeNULlVIXoQDDTDqOF5uTz7zH5bZihZ8R+/ ceepDZjHK3HdN5B+3iRbsaY3iOKAltnsknQ9J1OxjwYtqerzfxWFUxaQfS5gxxC5f9KIyTYi3Kn Yfj0cJWaDKBhiwvDSt23L6nsvlGQAe6W/vxMD8BapGlkYRUCz6HqUytTFQBAxgYRr6VCwPot7Nf Rhg1eDuay3YK/euZWtB5hOhhTyU= X-Google-Smtp-Source: AGHT+IHofchBgt10CdVSO9Xcl4yHCl6fKy3NCLrLd0JwdaM3wh17mZmotiuBqqh8g/QoieHEYG/z2A== X-Received: by 2002:a05:6a20:9149:b0:1ee:dd60:195b with SMTP id adf61e73a8af0-1eef3dea587mr9136695637.41.1740177014436; Fri, 21 Feb 2025 14:30:14 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-adc9ff10056sm13408975a12.72.2025.02.21.14.30.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 01/17] zram: sleepable entry locking Date: Sat, 22 Feb 2025 07:25:32 +0900 Message-ID: <20250221222958.2225035-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: omdhpgmhby1jdayieqt4k7bf7rgefncm X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 98DB14000D X-HE-Tag: 1740177015-284140 X-HE-Meta: U2FsdGVkX199QpG0b4Ja1QnMyxubf0ZKtq4Ogc8Q6zZlj5D4rOGgXHUK6/qJfnR08T2BDSC/rNRKjXcIzpx74Fv+0Qj9HONo73XWEvA1zH1+nJX27h2JG4aZMb6FrMtHjR1C1KddFR3tOnQCRfIn11QEotukEASrHdVQWMaKRwv0pS7oj19Rv70ihL5VUtikmrriH+WEsuZhQKwriEavbu1MUMfRh2yqD48foW+BscBTF2E2VAhP1KvCtF4SmZqOlcEQ3FoAoPtdutEo+cRmiS61au1qG4Pe2rJacj5SIIgYt7gR/R1u3simy9SsWouJsLxku19CuDdNeaGqRkzw3+CLG4eDXK0SyvhCWDl+2vfUnnHsCKphDNmU9kGtxUGFOAi86Z+wrM4zY6BYEteXuT+v5EXAP59g4bkvcnZCO7CdYc0W2POHTJuxrqW5Uv7om/4F4zCrvKmVuC7xPnAmOgp4kUxxEgDIvbvcp9Yb2SFKuYHjAlHnsfwLJuue7uFZUEhQOKsATWmE24FqyPpBgMwWh9IWUUvMJs4woApjdcddCCoGda8wkKYr34m03Jovcpp8uGLdfh6K+LQn5wvY+weuxCwcxv77hxC5Uq4ote+rnVMoJKjK4Pe1Py1jz50jGTnyMsSoR7ZjvoUnH8pEhTL2r2Wzpfs1eFvyzYj9aZqpgaIexZKm8z86HUkNF2tXy8uhbFTavkwV+HTZcOcRLhOhW36rRcGd0r9g1Rn4EO80oBQGaVlMgJnlko5qKmce/Ddk9KIyJDXl8Rc71ytWbTH4YWx9vDcgm1aAVTD+nwOcQN2G+pJZlIWVHCmOH1DgG0VROcy26DXXDeAsmza2SRX3+dESoJRsfvYlzZC1CJnP84iiepTWF3GCc7Gt/vNuQ6Jfss+Y6WwJ/YzEMlAh0hXrG4CyZ2UBtl15ScoNZjN0QoCk5WjNvY4otXFZZ7/6pyxU2xsKwbEafxmcBM9 IkjeXGTj AHBCBm6KByZoTk1+jvz8csXPT45bgyjkco6lM5hSH3cK63c1WAjNSmwR/1lZntLR3OHBuMzYw+a97EetSP9pKYAdwgxRRENTE56zg/rqVg0Mmg4WnyCzdJHfj6GaUuv4xSGKsbVsGCgHTrx2J0ple1pQOFgxyNhcEkXegZHhPuafYOnKkzjcxpYiO2ocwXSLlKwBgMfWWfr6KulJxuHayDlHtz+1/XM2GMupT/IxIYbrNRSZbhRvMYqBP7VwsUHsTaNsZqP7tn58GJJIu0qi/NdJyxhAIvxdoUH0/CwrAh6AknjqG9419NWgrqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 62 ++++++++++++++++++++++++++++------- drivers/block/zram/zram_drv.h | 20 +++++++---- 2 files changed, 65 insertions(+), 17 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..37c5651305c2 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,62 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); -static int zram_slot_trylock(struct zram *zram, u32 index) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map) +#define zram_lock_class(zram) (&(zram)->lock_class) +#else +#define slot_dep_map(zram, index) NULL +#define zram_lock_class(zram) NULL +#endif + +static void zram_slot_lock_init(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); + lockdep_init_map(slot_dep_map(zram, index), + "zram->table[index].lock", + zram_lock_class(zram), 0); +} + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) +{ + unsigned long *lock = &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { + mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); + return true; + } + + lock_contended(slot_dep_map(zram, index), _RET_IP_); + return false; } static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + + mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_); + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); } static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + + mutex_release(slot_dep_map(zram, index), _RET_IP_); + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); } static inline bool init_done(struct zram *zram) @@ -93,7 +136,6 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1515,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) huge_class_size = zs_huge_class_size(zram->mem_pool); for (index = 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -2625,6 +2663,7 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; + lockdep_register_key(zram_lock_class(zram)); zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -2653,6 +2692,7 @@ static int zram_remove(struct zram *zram) zram->claim = true; mutex_unlock(&zram->disk->open_mutex); + lockdep_unregister_key(zram_lock_class(zram)); zram_debugfs_unregister(zram); if (claimed) { diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..794c9234e627 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME = ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,13 +58,18 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif @@ -137,5 +142,8 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; #endif From patchwork Fri Feb 21 22:25:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2AECC021B5 for ; Fri, 21 Feb 2025 22:30:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53B2B6B008C; Fri, 21 Feb 2025 17:30:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EB24280002; Fri, 21 Feb 2025 17:30:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3650A280001; Fri, 21 Feb 2025 17:30:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1669F6B008C for ; Fri, 21 Feb 2025 17:30:23 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B3669120869 for ; Fri, 21 Feb 2025 22:30:22 +0000 (UTC) X-FDA: 83145396684.05.53192B7 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf15.hostedemail.com (Postfix) with ESMTP id B6208A000C for ; Fri, 21 Feb 2025 22:30:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=WOVISUqq; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177020; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bQZ6+kL2+QjAG1mCzi3fyv64KzDue4pptM4PWageRhE=; b=xiKvbRqxOcMocemrtGhfmTIME0ipYqMZVcQP0xk+kJF9Qf+2VekAZTiHgPeyKO6c78oQFW rEtOx8RLSsnCRs4G6Gl6FKLz4JMqUse26nHSGljWi0xrII2xukE9quosLj2GxZqpgsssln Z6p7LDOTx+kQH7gouOkfPMfxBOO3BYA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=WOVISUqq; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177020; a=rsa-sha256; cv=none; b=6kTCyBNr97QZfCnhF3VYQE59CTeRlq+TU3AnwEopCSjjAvjZ7hVvj6Ux2NjJqBj7aE/xYj +tvpgolu9k2tTa64hxTSz3tSeJMkvnkT+f0nfcw8KlpdI4I3zENLjU4L9KK8zF0cDx+t7s Goksvqwi8SdvrS65mCrNmpB5ar3BRLo= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-220d398bea9so41112715ad.3 for ; Fri, 21 Feb 2025 14:30:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177019; x=1740781819; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bQZ6+kL2+QjAG1mCzi3fyv64KzDue4pptM4PWageRhE=; b=WOVISUqq/8IH47oKJ7/bmIDugS8NmWhWJpJhX1oGvlB0kPqIPDZIHr8oobzG59w1WK e70IUzJuABoi9B9qCtAywqdI5zzcNrMH4txUyuOZBItafAQoOFxMerlWzLXcUxHEf8Fo CQRJqBJlUA697UEwv0KPbELnNGcs+vFC6ayeg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177019; x=1740781819; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bQZ6+kL2+QjAG1mCzi3fyv64KzDue4pptM4PWageRhE=; b=OVnRQiv0GhqSrWdh21exg/Wkh1qcEeyTZnSJ0XEffRA0cpoO4QHrS6Lj2dYHo2hqdn Ya9FTpAEEoEVu0M7zVTSLF3tWxdl9boaE0mYDZE/zTwXkequ95oVXuybpJCk/NeafQOb vtN2J1o2gI4BL8kEfec2DDIuAOe+U3+QQN1HRbsR8t/AV8Ng61NN1snO/86bQmToOfai MX4PIW0YPx9UJms9PJHst1wGA3Xm5vZyj7O89nKWME+s+OdUJ8ajKFPH4benCD++3BsK W4EeDvnIMgYxQULXApidJQq509ZRUKUAhxYLCz0VhtEWxFEIZOsEeXM1xTSOWui3Nxyt xTqA== X-Forwarded-Encrypted: i=1; AJvYcCWCINN40pGHfpFPb+fqGtNJrpK04H+1EHhlT4w1SXv1XL5rjOrgdvU7M7wQ1FU3gsLULqKKJqnOIw==@kvack.org X-Gm-Message-State: AOJu0YzNIJXSDzcxi9jSVDVIzOx0hg02LbkKOJGLFHrSXWCitEdj7VDh 1cEP1CQz4uZG4r/NVPMDWeXtvdPHQuI/LnbXxC6DlKG7r9dx/URkIoTlfKs4Nw== X-Gm-Gg: ASbGncsdTWYEs7gdHnLJn5y+YP1mZwMu+SQXNz5WoWA5H9922YoI4jZeqUeTlmTqIQc wgxNH1BB/DMdHFd9R4Yr9JkrOJSJKDpBQ0AB+ofHdOpN9578O7Wf533U5SJJ05thp4gI23OjacK afz1W5eg1iN0TJ6AbNOqUib5FMvVoNcahucpo1dimdLgqy6g31zEwU0kyYQ8Sf9Obb+r8MSvrzb PpqdeGW2axBV2WbFVL9SjKOQft1iVx70eOSt4ggEY44Hd652fAHKsDeIULFjHqh664BPtgP/FBt pS//R/ysQfKPnT6vvqaUHSChS88= X-Google-Smtp-Source: AGHT+IHFhs7hFYEDScLROxnrA8v6yUZmmj9MauSx4nCbk/5QH6SmEjHqOBlgJx/hpKYsI1k0DH+1eg== X-Received: by 2002:a17:902:da8f:b0:220:fe50:5b44 with SMTP id d9443c01a7336-221a1103431mr80781945ad.31.1740177019665; Fri, 21 Feb 2025 14:30:19 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d558fe3asm141026215ad.234.2025.02.21.14.30.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:19 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 02/17] zram: permit preemption with active compression stream Date: Sat, 22 Feb 2025 07:25:33 +0900 Message-ID: <20250221222958.2225035-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B6208A000C X-Stat-Signature: e9rtwxysuz38811y3fn3hj4jhcuo7r76 X-HE-Tag: 1740177020-788421 X-HE-Meta: U2FsdGVkX18FNeg0+Z1W6t9wmQzOqpeHxxNpqiHoUGRdvSxZWiweoJ0YbVCYn8uz3mhpN86J0ZJLNztweUlBlDJnyLDKaUuo4mTiXKHdC5PzJXqM5AuAZbTLHNYdFNDowvJ60AFnNkj7n/+qmytJ8RCW1MNUEXYfyIvwEM3Zh+dtvLyC8R6YlW3BB2mJqo5P7K9cODSAilcFPkl/nuT9jtvXkAHpahuOIroDSCpDeou64fdWp3UqijAFVX+WjWOli2/Y/F8I1WccwIiYXcWITVe7hj04n3R8IyFqURezuZYRAx0VLVw+zgBT1uArHFtM8XEpCRi69IjxfSq6B27hUlrnWLYlUYyJwi+Gxx2qFkXqFe6DJv+Y03rtc7yEYqOJ0o+XGNMcKc1sgTV/BWBiNBCBdAdI+evhd0lXZnWU6ASIeL7LKxRcQPyPqIcdGgGz4lnSBeKdCNvx0jRU6hnwutQIYXzVgP86Y2lJoU5KWK355OmUVdpcnQvqJfPVH+rxotJ0bK2eKVRDDMI657DiwQvtlBQLoTH58ahkxwMnJEhRVHNukN9u6A5g31nZC94rvUPMgAZ1+p69VneQeyFD49kONd1oA3R+O3Qq6sNbL9qXApcH4S7FSI1pOm3vbHA1tDkMekDqCIoVW1yxEbgyX52dN7KQNeQt70h6I8v3YRIX7APTyAxs7SSlzfQsIocHkSPkWRwdSuH5GhELtRpdS6x5U+3nAfDUtSE3kocTBYS5KabI13WaB88fPlEgNxHyM33P9m7KZY/Jv2rw0ps11/hbtMGHiD6ZWwzWQWfnuchb/dpKL/dRRPyV0ZGBvfD9TTwTNZ/4xJK3gSNdoqERMDdUJG3jVlRUlnx76FzUc51vAUw9PxnzwL0OG+00j34SH203TQEQzDjGlApsp+jBBRYdXMBxAmp7As5VFOLNyV2ZlZwFXXFqbeQ0oJ9q3qfLFmUHfm+UMCsfv3qyQd7 C8F744NF 8urvKstHDzV2zPJ1JLe2WZXpFI2I6mb+lc5nBMNPQMwD4S8FuGVVeuNguWdwUdr1BfjiWBGMIfKctQxY+Uc+YtxClu3c5wTh4JvtQ0x+vv85anS/x/1Cxvhm279ur1duBKCPUxObuXmDheqOOEjtiiHayNUU7f3S9BKDHvhECwx2mljdgx2M8gh15Bnip+vjGEewIa7+IONk8saTo0BG4EBFo1XDyezlhMzyVd9kcxikCfsyeECKp6n7PjQwJPdPNEo5Mb/iwG2opvVTP8rjZwIx75LriEEKSjI/3l+/FanDsALVTbERgaahIaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Reviewed-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 41 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 ++--- drivers/block/zram/zram_drv.c | 20 ++++++++--------- 3 files changed, 42 insertions(+), 25 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..53e4c37441be 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include @@ -109,13 +109,29 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * from a CPU that has already destroyed its stream. If + * so then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +167,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); int ret; - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret = zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,16 +179,17 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); - zstrm = per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) { - int ret; + int ret, cpu; comp->stream = alloc_percpu(struct zcomp_strm); if (!comp->stream) @@ -186,6 +200,9 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) if (ret) goto cleanup; + for_each_possible_cpu(cpu) + mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock); + ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); if (ret < 0) goto cleanup; diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include +#include #define ZCOMP_PARAM_NO_LEVEL INT_MIN @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 37c5651305c2..1b5bb206239f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1613,7 +1613,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1774,14 +1774,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } @@ -1805,7 +1805,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1817,7 +1817,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1825,7 +1825,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); @@ -1984,7 +1984,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1994,7 +1994,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } @@ -2052,13 +2052,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle_new); From patchwork Fri Feb 21 22:25:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF3F2C021B5 for ; Fri, 21 Feb 2025 22:30:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FE35280002; Fri, 21 Feb 2025 17:30:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AEC9280001; Fri, 21 Feb 2025 17:30:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45025280002; Fri, 21 Feb 2025 17:30:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 28E54280001 for ; Fri, 21 Feb 2025 17:30:28 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DAB5EA148B for ; Fri, 21 Feb 2025 22:30:27 +0000 (UTC) X-FDA: 83145396894.19.6F5F0A0 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf12.hostedemail.com (Postfix) with ESMTP id E1A244000C for ; Fri, 21 Feb 2025 22:30:25 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=TGxpaw2g; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wRucftKd651mJZM449koMgQbInBtJspHM/LJa8qQfME=; b=Onrkc1d+7bFjqR3RvFFUBcxHr7kOIZTm2Z7E7vRUe5rCSh9HVrK2Ds46YX2BYDWjRk2fqq zSMMRHz/qHgZkF8SysBAhFyzjN1kJNAcy6Hjf0SRvvTmXWi9PgekqN1S/FzAU9NDrmCxo6 y88KL5aldBFWhSx0ZMgIy78vNFoi9NU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177025; a=rsa-sha256; cv=none; b=xWyQpfV77EkfNJWpGg/Dm/N6xiMmL+IxE3cquVY/Ra/4Gef+TwXZOeHD8orWB6fz878lJd hcuxJuTAcoV9EPGcyDEV6qKFCnUinjTY4iHB5fciem8iOioM6cU57+iLVhgmINAyabbdow pgRCZ5pRQmc5KPHE7VRVLfL4VMI/JKk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=TGxpaw2g; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-220dc3831e3so59055285ad.0 for ; Fri, 21 Feb 2025 14:30:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177025; x=1740781825; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wRucftKd651mJZM449koMgQbInBtJspHM/LJa8qQfME=; b=TGxpaw2g40x2NCgAPzQEcEAzCUmxATpYuCU62iC8hQhAuvs6yzAf7VoabPZD4ajUCQ E10NRlEkmVJah5vu6gtsXTihNjQz65khpQ12RX9DDuJE1Z2YUrOpvJ5qWNtTXgdegI+3 AuU7BzCsXjDY69GvDxyDL/FnBaXJgt1T5aRWs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177025; x=1740781825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wRucftKd651mJZM449koMgQbInBtJspHM/LJa8qQfME=; b=mL8R4zUMCWRpG8ptvcpVx2XFKNkqW77yM17yQH9BN5ODcKLZvW8y7WNzdLUmbq9r6l Ltssofgnwl4cGKF/HypEHKk3wPzQlbL59IuVP6RknCP2bNBa/v0c+OdQOUCfNJVm4TPA Tf11/XDVDRrRyY+pL9SbkZsrpVh62tPfe/C3w0ttbH3wDDb23M7YuvHSuj9SDIeP+V6E OIy6zKrrqDMOISt2ZQ6dofGMfkClgoRPWrxckBuad6FyXr3gHNHoRVL/SeeoGZOPWlLX KMqpAtwl+hoosD/mY52Z+ru/vIu8oK+tJelyK3DZ/iRnSwn5zeur386CY5IjNkPFG1dQ g5pw== X-Forwarded-Encrypted: i=1; AJvYcCW+JsSHTA81xbgw+iAL22Bp4qYp6eQfJJhmB9eJdbXOl3yqSrBaAJLwBL8nBxzOn2PsXwbIOcCAWQ==@kvack.org X-Gm-Message-State: AOJu0Yy6pQxKpEyFmaIAq+HQrrZTxkAJkUTTsUrlSPaLRzX6r0/rEFi7 f0A6eGbcYjlWOabNL2ZyIJ1qSAH6vRlbt2Z3BOBc92wmdPql8DpEZv6ErgZd9A== X-Gm-Gg: ASbGncv3rx7QQJDNvi3e+J+9TSaajTZsuqfm1hasTlXW3SfmRoKlEDn/ZUupw200pr5 HQSy0ns7F/EB+tlanwaHdkn3Kum6iUBV2tbDSMSanVKcGh6tjNLWZaRbdx56IT1jldKHxOY4+Lg 3SDFRzOtIjBTdEt4HFVS47vAQ3mtP7VRbu/Z/c2PdpRaV4E5nqmM4bu+vvW5oLSNPGETierDR8/ UQn9dF57tqBskyalgXFFIFL2CDrf7WpG54Uhekmn6hwPFPTjafZAp6C213MAkKwnTDlNlNdmyP4 eUWPYpJeISczlpUJnxNLRr/5MhQ= X-Google-Smtp-Source: AGHT+IFFK4qHRvCohfvbZU6uo/+Gpmd5dbH/UdpJOgA2xR/BA7bO5JXyADGSM8tFOOuSctitamoWFQ== X-Received: by 2002:a05:6a20:a11b:b0:1ee:cfec:9e5e with SMTP id adf61e73a8af0-1eef3e1ec9fmr7915821637.21.1740177024780; Fri, 21 Feb 2025 14:30:24 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-addc7e25d20sm11579253a12.30.2025.02.21.14.30.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:24 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 03/17] zram: remove unused crypto include Date: Sat, 22 Feb 2025 07:25:34 +0900 Message-ID: <20250221222958.2225035-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: E1A244000C X-Rspamd-Server: rspam07 X-Stat-Signature: kditpdsujfzgactwi3d8f3rt9ybcx578 X-HE-Tag: 1740177025-609872 X-HE-Meta: U2FsdGVkX1/rVXMxook0R44F90Xe7vyOpPnvHfq+ezO7Z4UgUA26vlw5qtOSaFp5CI+8yRnSPUj0epcgSyc0dlKAK2otwxdLHV6NNWMbftBPoohceV8vlJYIBZPI1tbw2Z7935xGWltmK515UZcskGHaQOSBCDSf7q6ByODaEIz34IOpA2O85J7GS4CKdQxWriNp6ApFSWkEFkbdTsLrvxXbEbCkdeu33OrIGqcK1MT64L5mDGv2XZ2jiwwSlYwucZK3+zhIYsxfrI0ft1KDwpV8GflXVyK9jj/ScUH0c+7mYpqacETH5bR88KSqyV5Clf/MNufvUDO+kqj4DosbF94VM1TWFF1/agkzDhqq4/g9uzUAvIBi3eVrz0uvPvlx2nIsOapz+B7Fj+4lLg+EZLDbDpucHc1Zq55JylzZmqhvHq6hy5dnWQ7pBq6VrDaGLkETevuSXBjSg0T+NQV9LPjcp6l5Q0e+4xAMBnp0ckxlz0dizm5wQB7mhcbH3KITFGKHKpNMeNhcKIHEfN/vCeK8AP3zu8Vty5/lkpJiZQeGAViFgjprMSZC7wUxcVQ877kkOEV0TPARDWC9qkIzS6NSsbz/QYjBKYlVcU+4WPyih7vriBf34pzqYLCsAHJU1B9Qe9UZ9Q/5eVYokhXzpoDV9VGPKxGmWw6YIwLDgVRkxoGsxsaIImy/B8t+aAscFphCxKihUurAIJifBvbkGfJg+VLyVSzUslRZLTcQd3Jr5bobcjxYsKh9TKkUc0z0UiUFoTfHes62cxxxPwqlRZEJa52i2asLhfzjCOqzwkW3bC351PVmeE0kJgcQliI62mB4C7qRl77IYUR/Wd97GWuWyMhFygy/cHQpcP/el46qhj6nTXb2XnyWiRf9F3iQ4vE+4jbpi36KrCTxq8hlADM/p6i0EeTKLNK44InC8hjYA+WIhYH8WeY/5SdLOWsZkOa+1oZ8KYWGGoOqAAl +IR0Jqmf Ybusq8FqSkTFimq4YDjcA4XAh1AiSB3DJIbN1joayZLt09oj2VjACgOM/o2aCWMKw5daL09i1/9TjvaM2pExPjz/M/XvZy1wpGTsE+6cqJLsoKR5ZWtsNOxBFsVH+Y51EBUeugW0PBMSidrUgqQbUv1tqQiYRhhgP4YqaIZ4+5Sie+prOAw6h4C2KdwqZBr01RlBXOYHg75nqGIq6nX2R4ktbGhBfnenotUvPRZCNCBMqzMHooK4EOo1MWqwlsAZo7ZCq+7vTk4cmUj8yQJ3JuRkChT6Q45ey2R85/85bVrLdkSptDkVPp0EFjxWD1dgXYin97B6koHIli+nlR6xGaPgykDdF3T5yYhHV X-Bogosity: Ham, tests=bogofilter, spamicity=0.003454, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We stopped using crypto API (for the time being), so remove its include and replace CRYPTO_MAX_ALG_NAME with a local define. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 53e4c37441be..cfdde2e0748a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1b5bb206239f..c73d8024f48f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor = CONFIG_ZRAM_DEF_COMP; +#define ZRAM_MAX_ALGO_NAME_SZ 128 + /* Module params (documentation at end) */ static unsigned int num_devices = 1; /* @@ -1154,7 +1156,7 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) size_t sz; sz = strlen(buf); - if (sz >= CRYPTO_MAX_ALG_NAME) + if (sz >= ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; compressor = kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 794c9234e627..2c380ea9a816 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ #include #include -#include #include "zcomp.h" From patchwork Fri Feb 21 22:25:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4EF5C021B5 for ; Fri, 21 Feb 2025 22:30:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EDCA280003; Fri, 21 Feb 2025 17:30:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 49DD0280001; Fri, 21 Feb 2025 17:30:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3177B280003; Fri, 21 Feb 2025 17:30:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 12871280001 for ; Fri, 21 Feb 2025 17:30:33 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D28FE1A05DF for ; Fri, 21 Feb 2025 22:30:32 +0000 (UTC) X-FDA: 83145397104.23.72766FC Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf04.hostedemail.com (Postfix) with ESMTP id E3C864000B for ; Fri, 21 Feb 2025 22:30:30 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=isdTeKl5; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YU0o+JhKC+VY+7R/4gnQD8yLERLPS3tBVJXP0N4K/mY=; b=D9OncaGYOCcUVCGBZAl2ARNMJHiNB5IgPT5dE3UhXKLDC/BCQvaC6/LhYdJ10pgZ7AAbLb 09UqaxmtzBUwsMfm3nNnx4OqCMX4SFTXFesiR7fprm17OS8+lrioaWlSckE0q5VegeEFsA bsjAEsNGN/dIvtHLHOjJC+RGgx3FR5w= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=isdTeKl5; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177031; a=rsa-sha256; cv=none; b=3BgFRyCwsEx+nAwNQNUnHNfs1Lzqu8z+iqL8ZJbRYLcBQTt+WKcrkiUZOlwV5RwFtRPL9X iEZ6clD2PPwThflST6v3qkR0d7olpmqqrFYe+rs/vvaZMtCai38+D9qFC+oW7WvntEte+i ABVOlbjS/RJGyLGWhCAn8QgElKjMgLg= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-22185cddbffso60074495ad.1 for ; Fri, 21 Feb 2025 14:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177030; x=1740781830; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YU0o+JhKC+VY+7R/4gnQD8yLERLPS3tBVJXP0N4K/mY=; b=isdTeKl5a4b5szryACzg1Y7SoD8QLQrF6YPnkUcuN2OrJhWeUEiCWx62654L5d0f0k FfasIDnNS3daKfI3mDGTKB6iE+++/ysCXMlHmsmoCxxKKlKRCZfWAB5Bl/g9XZ+jFo9k JCIHttYGhQ+zZled6ub5wpisMEIpKtdGrYYVg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177030; x=1740781830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YU0o+JhKC+VY+7R/4gnQD8yLERLPS3tBVJXP0N4K/mY=; b=RhDIUj84lkVjozg/y5gm7jT2JQn3V2lcPEgFtvz8Mbz77G2JyABnX36f9ihYkVJiof haOEYHRfwrNezxoXgcyQDfEIzFrAEbcxXY1DKPZ4GtaIvh+u8gDrtEvMUMpxLnCVclFj iWKwFhu1AfjosVmZDcbRfBxzpjqH9LKUy9PgWqDa/Ta3A5fTSyx/Wh5H8yExK9XeJR1V K6ARlR3u0YgNrOgXQpWNoyP0V9ubMAiLRwlPtbjWYFnwEujZJ22SopX+pWauGhB91Ezo VGj0GqBW6/HtSKwFZe6BaF6oBcD6ehx3NApysXeNv0MDoSlQBjPYMwfvSy17zas+bSb1 hTPg== X-Forwarded-Encrypted: i=1; AJvYcCUfBOkr+Lcb5Q+RxVfQLN1cMY2G3e6pPxhpmxWEt3dFyCz6PZIxHMO+HFZ6r+NnetW2zOYqessMSA==@kvack.org X-Gm-Message-State: AOJu0Yxh3G1zp4DUFDSh2NJPYZtAuBeEVX0N9Ylr9UslUv3WcQk5i73J yMSD8bxCq3cHf10nYcY6rOxMp7Khq/uW8MQ4Lc29gJWYGWx36sGeN5Hi+qzsyg== X-Gm-Gg: ASbGnctfM9Oir6gE1p832Wg0Tv+iWrKWRksaysqX9DauuXIpMcWnhjDUwigDM6eXEaV y9Es1VaeHx/29VrgtdcInDf2hMwg/stj1u8kKq5izBG9+Sfn0R220hDTqHvdRKc77KdbTUdZJOQ 8OlnmWza/MIKd8VbvclmClBsCQavoa2peJXeoWV5+WTPtRkzQdy3RUCb8iVIA5PMlH0dtKED/Wd 9zKsknJfmb8NJqksmPggZDvLf+bMSN7XNtS3rwdNEVyElloEeRJIWlKaPbR4jrVr2PT2SDCnKes SYEAMacCpneCbFmxO9FQhtW+97M= X-Google-Smtp-Source: AGHT+IFsYHSiiFTmNB7pEYKwqN7LmyvESgmGisHi9uPcT6wi7vp4qO6vSYkBBqsRRkgNdkoxHrI4ww== X-Received: by 2002:a05:6a00:3d10:b0:734:a78:2f36 with SMTP id d2e1a72fcca58-73425cde0d6mr7726805b3a.12.1740177029789; Fri, 21 Feb 2025 14:30:29 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73262567338sm12684434b3a.49.2025.02.21.14.30.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:29 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 04/17] zram: remove max_comp_streams device attr Date: Sat, 22 Feb 2025 07:25:35 +0900 Message-ID: <20250221222958.2225035-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 9a65oy8gqy3aj7f1jh4nfkitq3ucso5d X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E3C864000B X-HE-Tag: 1740177030-517862 X-HE-Meta: U2FsdGVkX183qeniol6rgzEVPrBjSst8CDq4DQBop4gAaISdR+ALwtTf/DDJuIPPBBIhvBdu0bEwVVf7+oyeXJHUfA9e3GJI/HhR5hvgqmjVaYSV4ohywRdfi+uA9b8TPrm8qGsMPxoWQq8XqVjHDhfDh6pptj9UgfLIUGaeUwSHB5J27nJMsUj/raN4zzLSTQcsGMZFA+/yozPT77/xxXlEABNpyY8riKzFLmR0BOOcaHNRJsgchO4xVKCVvxwcGXHdYG7VmswBeFOhlIimxZjsS1fRNJOE5By1OfuX6U8SYzRO1h5HvUvlqI85114c/6LtFkiOOYBirveGlzhcKuHomkqu2hHCaWUPfJsrXj+BGEGwj7Z6FY9tZnkH3PNwJbLY00R1ShWcxYT5QcWJXY9PrqU+tgL2B6z6dGwLuWz++eU0ZPArBRSu3hXrtPViQ20txVmVf+bUenLbhq1WJ8c+1El/S5OxOGOFvP3W4dv0KMJwLkG7WIXA3rGUeIa8Sfx/Z47gT3MVZ3i7fEYcbzoJcr6LmjbhQw638UMVjofIswYyi1ZiVanB4qipa43LncZshW7L7uadBmvnyO+VOn8LuM//JXvAiS4WG8ujwMA1xx//9u6sdiohIP9syTSQq9nqxx0z4jHPnYIyZtQP+wHijmNW6VN7ora0huJSFUJ/V+SXmzc+ejClD9bjNJM0kX80ZmPpnQqFRo5F5OWVLMctq7ZsL83Ki9G3IjueGvy2YZWGRgc71K8xxPTwX+UniBHqBvATMATZRA4I/gTITFYp7Fpya4+osTM4HVaxRx1HFA4TR3rbFwG2s3Io3hEBI7Oz7V9EnMKdYU3m5FZCrGFDgH/kltUekdjJIUNRLV3XGLAiBLG90Mm4FmHFRiMYXR5aMwww6kRSSL2RsCB1Jex+luwKAoMCEnR/lrsKDInfFdho31l57I37Skd27w5aVZwZGCfA9KRaGzeMN51 eYOYIZOq cVSDnqNW/HWW0DZbTyo/kBjVCIXJt+A2za4HvuMBMCccl+KhC23pW79+LVy9V5j6gq0Rw8Z+I2T/PQO0XdfGKsdVu2IFW7+mgM17C7DHYifXv/anIKvU87h2km+rfmCO69tk1dM1XnNXrLkhW/cbsjDRwjjT5sID2BmLu6kX0H0kTDRBLCJNtRkXROiuxg1+2MpjVzu/Zg/MQvTlehRSjIZBAqEqONiAYQKDDwxc7EY1Jh3mzQICHEdfkrThO8zkDqh8XQ5GFOmjozj6igmcdeReuklZq3feOId2yVB9sGvKARM7LpRYrg/9VzGXK0zFCdHklzgU3tWcUbsFGcF9nHp09Js4dZ1EFrsX2/hPtjC4Lo+PUoVAhiD9H/sPXIMOEgCcq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index 714a5171bfc0..7ad4c86f8258 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. -2) Set max number of compression streams -======================================== - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =============================== Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional ================================================= Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negatives values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). -5) Set Disksize +4) Set Disksize =============== Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than twice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful. -6) Set memory limit: Optional +5) Set memory limit: Optional ============================= Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit -7) Activate +6) Activate =========== :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp -8) Add/remove zram devices +7) Add/remove zram devices ========================== zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: echo X > /sys/class/zram-control/hot_remove -9) Stats +8) Stats ======== Per-device statistics are exported as various nodes under /sys/block/zram/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of memory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats separated by whitespace: Unit: 4K bytes ============== ============================================================= -10) Deactivate +9) Deactivate ============== :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 -11) Reset +10) Reset ========= Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c73d8024f48f..c7bc0c9f3f2f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1109,27 +1109,6 @@ static void zram_debugfs_register(struct zram *zram) {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) { /* Do not free statically defined compression algorithms */ @@ -2546,7 +2525,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2568,7 +2546,6 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, From patchwork Fri Feb 21 22:25:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0978C021B6 for ; Fri, 21 Feb 2025 22:30:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 736F1280005; Fri, 21 Feb 2025 17:30:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E6E0280004; Fri, 21 Feb 2025 17:30:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5878A280005; Fri, 21 Feb 2025 17:30:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3A1C6280004 for ; Fri, 21 Feb 2025 17:30:38 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CFF0A1A08B7 for ; Fri, 21 Feb 2025 22:30:37 +0000 (UTC) X-FDA: 83145397314.17.003BDC1 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf12.hostedemail.com (Postfix) with ESMTP id D8CE44000D for ; Fri, 21 Feb 2025 22:30:35 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LRn75+y9; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177035; a=rsa-sha256; cv=none; b=U7Z/OYP6tqgGmopR623V1op0BtEqfRUpoRhHxr/V3js67D6sot7+ixjsSTiv2QdgsIN3mE dYExF2aGdm8mxNuRRDIfKTcxBahUpAFCy+vN3a88X1DBJ14dIVOZDBSe370tsHIDilvhbm G1cRc7z1gwmrltdXVc+IVX1JVBIVP24= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LRn75+y9; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177035; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZGMz8miVpZuxydZjLRlixJe565Cxqs6XaHLZ0PLT/Cw=; b=K33I6GJDSopO58f2gExe1Dm1XCgAFTNv6rRo71A9qN/Z8jlh8zeupb29eAmrjv1rBjusvf 50iOaNIYuwc32GwnhIldVC/oYlCi1eVshZ8W3ih/RF5U6JeWvgbPBFz60g8Ryhau3Iom0V e+yqw9vdQDhjKqd2DMNpcb06QJOArz4= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-220d39a5627so41848325ad.1 for ; Fri, 21 Feb 2025 14:30:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177035; x=1740781835; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZGMz8miVpZuxydZjLRlixJe565Cxqs6XaHLZ0PLT/Cw=; b=LRn75+y9Pr5O6fohUWxdU9aMibOl59jVyBofwmmoKRAvgf/5P5NnMCOfLWljC9P8iP VCiWxzFz0qxVoBudTigEcfcm4MjGVMe0fr1I+FIEZbAxj/cU2W1RlfKasn7NRPhvvKXF 7QLleqT1HIXiU6zAepohR/yh44bfpoJylWqfk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177035; x=1740781835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZGMz8miVpZuxydZjLRlixJe565Cxqs6XaHLZ0PLT/Cw=; b=emmfkaE1S1Zc0OyiWmGPxjP0Uj7P3l6+bVrvZP0vt1wDEwX3A4j5BUrtAhF7HpzekV G9yJ9ezyANcT0uiIl2PBasc27hAm1acEF4ZT+CmfX8fNmiW2qxzE0pcJBMI1y2Z4ceUI TzDlenWgksEpE5OJeWA8kj1VQTPX7XKP177IISG6Mza7vb8a9yrIiHy2AKg0wAYuXJkL 5OJuegDQ/I51e+5GkyJWb3OQQyAo0FyDb8DinDKDJ2aQHM+Kw2pb7LYDAfwcNxVE/Hr3 epS9fyzGTpgLHe7ggO/6rOuLsAQ1/yuMu5v2KzG3DtwD869nd8dg+roXz6YEByBbGNt0 A/Gg== X-Forwarded-Encrypted: i=1; AJvYcCW885lCCVsWleFOmtiWUVLlcChUXbU8ThlrOXtyawijJVgKF7q+NLlJm1DJStVp4jEX+BFl/6k4KA==@kvack.org X-Gm-Message-State: AOJu0YypbEBvy1ukaPBaSzgT2WnQtdlRGhfGwoi0nm10XD9MpRlJv97Q K8eVQmx37bGF5ETZcAnHSNXGg1cqKhABq5924aBmndMBm8/iBDy+WrK/yyZK+A== X-Gm-Gg: ASbGncuORyx86+5cBLqmqZYQhAWHfKWSW9nm2RHcONHWc+hX5JG58zCc0PYbfTc4Cds 1aVZcTsopLASzWb5KYaiCHb5VIf9rFyF1F+io6y/DOFjAaeyJug8rTXOois4OBeP5D4DNHaygbd wuz84Xe4nJHHhybaiMuf85y3FiI2TBOmaXFghK+s6dVMnCx+bjXZ5O6uPysihb8Hgj1Hi44bMCW 5OZGYrj0i4Bw3mqhTSX4RfJbjIrVmtOZAOKZRUgbnVQsLuNdi3wvfa1YBQJuszJnYyRwHgPzjq3 OMgqaJI0stlBzVRP/q/S9sg+hpc= X-Google-Smtp-Source: AGHT+IHL8w233VXHSZOeZO8XETvndpBr3OMmgSLQskKaw+oSbhIFchdvDqKDGu9+zeJWRzKLmmc+Yw== X-Received: by 2002:a17:903:2288:b0:221:331:1d46 with SMTP id d9443c01a7336-2219ff8278fmr73678355ad.2.1740177034842; Fri, 21 Feb 2025 14:30:34 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d556fc57sm140834015ad.194.2025.02.21.14.30.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:34 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 05/17] zram: remove second stage of handle allocation Date: Sat, 22 Feb 2025 07:25:36 +0900 Message-ID: <20250221222958.2225035-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Stat-Signature: 97xc5us7o4diaeh6f3fcdukferf1p9gr X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D8CE44000D X-Rspam-User: X-HE-Tag: 1740177035-776752 X-HE-Meta: U2FsdGVkX19ZzS+pCaI9I2OctWi9qWvysg0F2OIpXPRBXASxb2Im2IW8xIp/MJ2wq1ymeQ5Pvknl5ntrnbF3Rm6v97IMd2y0YQ0UwrvF9wKyrhrMW0H0+2dZ4q9IGO909QEKOYfrsE7w4z90QWDodHY77Oi1rtTFqzc6gvoCkiXHp2Rn/RNI2uStcRW/7cT7XM3FejqCqToiaQan2u0V0GYrFxWilKbKdOl1Ki/m5j36RXWJx+PsDSkcjWSCRmlGCK4dURGFfBzE2F8aoyXE4Twrq+3q+82UZAwCXwOXVfaHb46CM03ez5GBToKaHYJQZ4NQ95a4JlrGdF9+ABTw6Gs/JHnBSa0fsQQkbgguq0cZ5ZbbNkDIf2VjNL396VEuUISrGzvJmakkw3K1Mr390mo1YWw7eBUhCh6ApIZVlD4ilreq3XTAqCBDm5j0UnKiEkjTgJsfD4zn0Q8hxoT3f+8zbfPoBmXD6edVm13JO5ZXyim7QIbJdE1qnzwWUiMct4Z5vJ4uDrGpkPDBSXFqfc5+kiSlrhuzkpqCn3dvfEahPzyJi2+ZXEUIVE+dhUnX9ywyvKPpPOm0jyF9HXvPzKIWBMRqf06JL61geek+zFkaGqjv9gIUikfHxoUNG18CubbtN3acFrizG3jvsTUc6nDzIPztWpkcjmfQtg5cGlRjSoKD6nVVDOh4ERdzwrlTr9zB5uagAptvZWHnbw+ZCpg6wpyfq0f2yy/UXOyASZMAJhzKeqSXUcEW6Wk9S5EHiiPvWW4dumGc/4FNNfB/DTUT8RTFDGbSlrIQyZ0dwQospNSwDYKFevU99Bx3OD8oJfBKHc/W1zeh0iJofPujTysQwA2F6UH3Y53p28Ci6tofFqgmzrLn8kWZ8dEqmkOoKKwyccg6hd/hcYzYdsYNmrlMHTRNkrrkvBZHJMu8zHQFKNyW8mGJGOHO/0MUfTOg1sKjBdhWcrbuQkYvH6o FWWTaCRb ROcwmIHv7tY5J5qVu03iehrfmN+NkST0wtW1QO+YeCuaMYKD+FquSqPfF5X7pf9Cofvn92LUxO5G5iZIrPLvX7gDDfcBWmJFklJe+n/pEV0iEZ7KqXoMEE/6ellQygQ/97umNpCFpUu48j4/BOSnifwZeS6npyWSLg19aeJKWD6bPSCt2hgnCMRY9BN5ZEkAhVDRNzkOyE6fmLWrcSVjc3uyZrtZpAAt3ux6xLHdiHRJYIKXkIbpqdtqg17716HYiSx9WgnYW60qB7nNZpPEYAF7PMLCbxJeoBgq57lSk6V7t7qyuModKUTT2jA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) if the fast path failed. Since we are not in atomic context anymore we can permit direct reclaim during handle allocation, and hence can have a single allocation path. There is no slow path anymore so we don't unlock per-CPU stream (and don't lose compressed data) which means that there is no need to do recompression now (which should reduce CPU and battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 38 ++++++----------------------------- 1 file changed, 6 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c7bc0c9f3f2f..4ccc1a1a8f20 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1729,11 +1729,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle = -ENOMEM; - unsigned int comp_len = 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element = 0; + unsigned long element; bool same_filled; /* First, free memory allocated to this slot (if any) */ @@ -1747,7 +1747,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); -compress_again: zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1757,7 +1756,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } @@ -1766,35 +1764,11 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } if (!zram_can_store_page(zram)) { From patchwork Fri Feb 21 22:25:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6410BC021B3 for ; Fri, 21 Feb 2025 22:30:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3D2F280006; Fri, 21 Feb 2025 17:30:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC477280004; Fri, 21 Feb 2025 17:30:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8C99280006; Fri, 21 Feb 2025 17:30:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A8940280004 for ; Fri, 21 Feb 2025 17:30:43 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3202B120869 for ; Fri, 21 Feb 2025 22:30:43 +0000 (UTC) X-FDA: 83145397566.06.8E4A31C Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 34FC440010 for ; Fri, 21 Feb 2025 22:30:41 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="abGP/iPt"; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177041; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8Ipa2djuhtNWtVeFchIK72Sxo8mMAH/HdQlr5E6ZYMQ=; b=g5kTYwr0ZdmyAtie1udw36q9UhOl0m56KAdmgSfB/f3LvK1dhBT/R9P6KUg9kgPZleGP5x Vt3pmJJ1WgYAJFbMuM2vlr9YDBX96b+1GVITdLZpuUJVTqzw+wxUvK8/Xic6ByN1RJmSeB Cx3v74MgFZ4wL4ccRhYFjQFH/1MrQw0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="abGP/iPt"; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177041; a=rsa-sha256; cv=none; b=JIPk/c9GpRjkD2zsRG4lDaJtil3dyA+PV6RwtiF1LY490zP2eOqZu58oYEk2NTIl4LLz2m +MvZcUnoy7bu4n/Z4xjgl5pUfShDYXgt1KfWsq5CLEN/xR/sj8bsYWByl90UFsLSd/Yk+E oK/p3hhsvTkAnLZWhpSK68rCnJl/Wbc= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-221206dbd7eso55731075ad.2 for ; Fri, 21 Feb 2025 14:30:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177040; x=1740781840; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8Ipa2djuhtNWtVeFchIK72Sxo8mMAH/HdQlr5E6ZYMQ=; b=abGP/iPtgwx9R9QDRsJEEX/QN5a5anO0RxeoNYJ4h6m1LMq8ac4ga+LtYQJk4rVZtX nZjMfoZImFizn4HJSglAau+h/pc3Y5GV1N8IQfUFmyuEqo97n+mgZTtkc0BB4SKK0Sfa BUDqLReeIUFoaQJpGVX9lkpYV3Je0j7laRq9k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177040; x=1740781840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Ipa2djuhtNWtVeFchIK72Sxo8mMAH/HdQlr5E6ZYMQ=; b=MKK48dWb0We2cx6x50urK6xPnffVR0ql2ryPuiaRqBETQ+8W66V1kIROotkkwmEN9T niARKVItAebpZMQhI5ZGUhG12l/chzMQdXbouJ2Ehitt2HVzZAn1EZ5KY98STk6jsmhV ZoTADblgby9oUHFAtq2iQe2xKVevDT4ZdgtD5Iy3V0q3cFcKRjWArTlyuKmVOmAz5v3F K7RQa4i/oSwNZQGLI1WlqkIQcb9ed/b5x0JCGtTKyGtW+Hi4SjqhMrcgJ1oJx2y6mS+l yrvHxdJBU+I0ntFKOu2OGQbgViq9HHzXZX59KE2zoqL6N9KR4WRK+WHgOobWeRU/EwhV LnOQ== X-Forwarded-Encrypted: i=1; AJvYcCXAmgFOPOjjpaOs6pPLKluYF5DuUXh7ZO4qxFRJiLiiBfBpFhqPmqtpXggiWnpb/CPeqhDQIkMhXg==@kvack.org X-Gm-Message-State: AOJu0YyUBgqu02pScofS77EtdRM/9+hwVwsk7qDjr1tFFB5VNaxNvEey 6D6JbBx+FeEbmmI26XgfohOMu2SwpUJ8Q6BZDd+5zO1ajif7NCCMt8mhMAcw6A== X-Gm-Gg: ASbGncumegfTCuINf2iWfQXWuCShKTx5Mwz/V9gCaHblmNRabnLJsawC51UStKEyPS2 YblR0Qi0dyt+Wxmjv/lj44+5LwIKYmC7ati1ZRDYkwJD7YMfS1hlbsYYDjx2zQWAWg7jDY+cOuh qrVo2W5xhUvimJYYDs51qcTsFWhdYQa1z//Rs/nxixWI1ozEkZ/WHOW4DXEk2NuxsQwDaub/4zL LvoQKq1lggkwJmrxB9fAzm3hXiwVAqdnykeu85xOrAUzFLtDmYr2Ov9othwv68g5Lt6I8edjMhF DFFhXV99nkhxj63Ls5wonauDGcU= X-Google-Smtp-Source: AGHT+IHnpU8YQA5PKzQqh5jRMk7SblIEQDtKyygZfmyprCwBS4IqLIgpeynjiVrVJz+BOzuLSs556Q== X-Received: by 2002:a17:902:ce82:b0:220:ee0a:73e7 with SMTP id d9443c01a7336-2219ff5ef09mr79718385ad.27.1740177040064; Fri, 21 Feb 2025 14:30:40 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5364676sm142838985ad.82.2025.02.21.14.30.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:39 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 06/17] zram: remove writestall zram_stats member Date: Sat, 22 Feb 2025 07:25:37 +0900 Message-ID: <20250221222958.2225035-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 34FC440010 X-Stat-Signature: z5nx7i6dff5356ch31b1b1nt6gzjskua X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740177041-690258 X-HE-Meta: U2FsdGVkX19b3TNGDfx9ZYS+dMPa5VVRA05HblluCSwzYcn5Gks3FfZZUEHfQ+fUFt1jXxv2BlEjj0T4ek9lyLb7p2vEbRgAyomZTUZaQX4nbdIVicEJJESBQn+oLAvY7DyJjJhRL8Xw4zEwX7MKkaPit3LM3LNi7ONz8yfZbf0bjCKJqP74UepvaBPfTIPiqEUD3LtDG8sgLK6kJ7NHWBBsLC68dyHLpT882jcz47uWKS5K5P1tX25+P2AsRG4eZA6+M2cS57o1PFGuJ2oCCx3D0DogCzUFtwPdUp8bLhXzdvSH4++7L6iGjYvNNTDxRvlkU7ECahIsqEAzjRfaxymcbp/AbKa29qaqDzICZ0NC88B9uVh7RWCdlQwz8Az8ryLPyCmOHO7OxPKNpF6rGNJUZN8hmikMpRe0w8Yqqifea2A5I3THAYHLKNX/gdK/V9x0RFgaBcXZwGU4sra80pf8pUvqBP4unwU/p+U4LrXYTQGU0aotf/PZ/lfJu7vf2498OP4ef9AggRiu1BxXpKJ4A3Bg2JMtmd9o+jo59zvLs6h+rfkR6ho51t1KtA95xn6x/AJA2cAAtQCi5w0QfY9BMzFTa+lWm3GcXDJUKbCYMtJfYvhKqq/15BZPRIvjbtu8hvgPlHQmtFhEOjknuXAmyL6c6Elwg2Rb0WNF8XOWOJHqWsHvnKrXvCuCTfX7yF6J/3SH9CCEwyI3gonfwdxo15RoQMQ70Wsw8VA/Tq4rxvdB4OoaMRMo9Si000ZHFo3dEJcgUjrxwchirOTK3Q/zWystjvejK3VHzPyK9d6sVo0b3KsRsPhdgG8SWna7WNUJUoibnKptrowfeUolNpB8V0xKqlU+GRmB6sn+7gdgPR2wpvdbDN/6aW1FI4fHJbV1/FiqQS0dLaD+X1NXMmDZ+XA6o8SYDd8OKO+ntFCYZk271fFYD6Geh2mBzEuuPyOXXKKpKaqnujFVba/ iVhcGIkQ uE6QMvbrzU8dtWqpDaUExgWhq0VrXfCxwzKuMwexMWav+CH4z8jrbFkkegXOSmg9eoOaSw9bLMzgN63q3px1w2L4TTz7TocSVP5asizXYJcdXidR2/XyxDHNRAluULu9pjEpd84E4Zb6lzH17PClimhf5GdGSBgfaGNMKjnVWIF3Qs+EfEwvYywJPxpk5TyyvaOjb5sLFf97mpRR+Q7ggcu2Psv2f9tfcdSKNEEjJr9EJSk1goZSsluN9nSaY/M4tArVskIXuAFAiINqyDJWgieTd7OFWPkCxwTd8Tz5PtsuC40APLRU6hCLZHnFd674dkcVTn49WIDEnlnCI1rRJDotXBTlP5tTSaQHf+wNnM64UDCk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000045, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 4ccc1a1a8f20..710b10c6e336 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1443,9 +1443,8 @@ static ssize_t debug_stat_show(struct device *dev, down_read(&zram->init_lock); ret = scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 2c380ea9a816..59c75154524f 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -84,7 +84,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ From patchwork Fri Feb 21 22:25:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3921C021B3 for ; Fri, 21 Feb 2025 22:30:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87140280007; Fri, 21 Feb 2025 17:30:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81F60280004; Fri, 21 Feb 2025 17:30:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BFAF280007; Fri, 21 Feb 2025 17:30:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4CE09280004 for ; Fri, 21 Feb 2025 17:30:49 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D3252120891 for ; Fri, 21 Feb 2025 22:30:48 +0000 (UTC) X-FDA: 83145397776.05.EA9264A Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf27.hostedemail.com (Postfix) with ESMTP id D887A4000C for ; Fri, 21 Feb 2025 22:30:46 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=L8B0JbJN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177047; a=rsa-sha256; cv=none; b=GMZX/ufA/4SpgD3B3r+RdwbaO7eA3Rp3vooz/LBMUkfyZFhGQi10Cw8G4HNyPZOYgoo+bE EOQx5QcBzbRdTYH9R8rR9zmUGVXiKU19FO6t5lZtCGKzcV4yGjfYxn8kcRivbNWevXdX5D jUDnwDgMdgCMbuCHtZH4rR1gj4KWDN4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=L8B0JbJN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177047; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jkJ2PKd3dLF29fHM+CjpZmmQFILFLiIANo+kf0KnLBg=; b=EhtDM629bUp3DTThOwMrlj/1/NC23n3zgDX0gz8Rarsu1E7Yw/zzHrF1iggzjOMHLKiIak Amro/wnjdsygwFEqu2IwrvRSRAm+QTUn1Z18Pbbf/wj6b+ESngQL3rNDkCfugk8ya1p2wd n9CgV3Qgo/i6NOVh7Fqyc/aPGDhbZJo= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-22104c4de96so47045795ad.3 for ; Fri, 21 Feb 2025 14:30:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177046; x=1740781846; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jkJ2PKd3dLF29fHM+CjpZmmQFILFLiIANo+kf0KnLBg=; b=L8B0JbJN0AssstmZAa/DCYipg6ro0y1qVj8cVww5elZ6foeasE6dzH1v5sIV0xfLEo MZERy4lI9iN3tBS9iXNidsqFd9JkT/LU430NNNjQv4Fy0Hbag1hKObpQdx0mMxi2Ff6X qUzncqx0vX6H9LSCCHi2bEMfOXT4nH25gTiNA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177046; x=1740781846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jkJ2PKd3dLF29fHM+CjpZmmQFILFLiIANo+kf0KnLBg=; b=ey/BX8d97+4hEMnbKgm5S4sdnDvA9dbWvyynV+4zMpXbyLpAxqeY3yB3MUCUQe4dT9 JG3bX7frzHKCorKkyA9OFBXwyItgBgR/ee2ctgPVCW3vyZ7BEeR4RxCnR5BHHCGUdxvd QdQsCErLCfH79lIF/DxmItoUlsb6ErVnIsG1YVIEg3H3s4fYLb0zk95Jx8Uyb0S7kZf5 1P9hxZ2crmPHPrpexnMZ5JeUuvPat7WqRVzIKOQ2dX7AUB/tBcu30fJMkGe6EaDPKEbZ caxeK0oJ7llBRsi+CPjRGI7KFM6v9oqG4ji7OjjbCS5KOec2XbZqWiaAIJinYmIgRPOa QiCw== X-Forwarded-Encrypted: i=1; AJvYcCWP7k7ZsQb9LDMniiekYJgIAlnD6XUbFTPnImLG3+pxmYs6FCmbQZy/FrMd8Z6lLzjXF2H4e7Dc+A==@kvack.org X-Gm-Message-State: AOJu0Yx0v+YcG+gbtpRmv76nwPVX78kmRKv/79nCRbxVWnV6mtczBYvo GwzKk+bGP5ACZIdOi/M0x5vAbGlL4rQ/Qo93XC8tbUMwrDQjOOlIeHAdfuFTfg== X-Gm-Gg: ASbGnctdPnJH0HiBnG6kVWHZg2Umnor5/O5C3c0AaxtWcOE4bX6RNazQBy+OJtyVMgc 7Y6YyJflEfqynS0wtsj3/izjznSApGDGBsPaNPuvhnVGnA22WfDJksNiD4DVLR9nXpRsScut2gI FhKxpcFu5801L4ZLj19oqIbx0dZp6ZJflKgxW2VDmUQNLnlI9kGXofUzYTlfdOMifoccg7Jl4Tw B6Fr/1NMYnthIST1e23eeG+l4JRIbwtif1alURma/Vbh2PeZRsvJh8dZQj9W0S9vBDQFsMwvlcH EuQQmWuqebCC4gmMTdV4P0dSEJM= X-Google-Smtp-Source: AGHT+IGqcuXdJ7OX1zr/wKQrluVdfXffpe/y2wZBMPKg94p0xgvFz73ixWr3hQi6scgLb0vWrXTzXA== X-Received: by 2002:a17:902:f644:b0:216:6769:9eca with SMTP id d9443c01a7336-2219ffddf80mr74177035ad.37.1740177045827; Fri, 21 Feb 2025 14:30:45 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d55963c0sm140523665ad.251.2025.02.21.14.30.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:45 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 07/17] zram: limit max recompress prio to num_active_comps Date: Sat, 22 Feb 2025 07:25:38 +0900 Message-ID: <20250221222958.2225035-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D887A4000C X-Stat-Signature: qgrqhpn17z6xnsm1d3fq4ow45wkh4pra X-Rspam-User: X-HE-Tag: 1740177046-105059 X-HE-Meta: U2FsdGVkX1/x0HymHt8G5B3zATQNyJIGxribLI/H3MGwSo6G5mBHNC2D3UNgFszBcDx+m+z2isqHIJ1BhmdulM88Lw1injoiAuvXovM2+rcOOuzNXYuediDUA9zfUzqw7gZjBOrSyC0S8oUBFc0up6CZsmyG4FMZX0SLV9zKQbVRIeHhFZrZUgEE1Z9md9GdchhFH6+gpgFz0AYrRDWdeb51AsXo1KG7iX3pOHnS6qAQP6JpPC5ltqXOd+g/WSK8ZFSCd5PqIoTtLmEMGMSNlaXEVBADijlO+AiUCUP/3smnLIlQyPeDQuQdJlkbWdSPPzkJG1qljsXrOzSaAwpZsfFeunIUVz7OIb7whQBRxH1DRx2iza62cyNBF93Su0NLAmxvj0Uq9MHlGCYtVhGAePLeoTHLecOw3HNePveD0hycy4NQJHzxiifyfkf4trz0RFGiT0foRkXNLbaNxP3RcaiOblP45UBOOqiUbq3LL3Hw4DIxmCOyEyBllnL0GautF8IVYyRXKs9rr+Ri6V1aIFlpb9owjx7YexG/eKxYTWAhb3aILeZqrJAkrdxUHj9x8u7AvoFzWRoBmfi9ps1Hi4/YUczzUVf6cTxNiAMwlewxp+NtFv/+Gy+7fHlfSRoy08j3n0awLUPxzxcc5PFrl+4Eqjfx/E9uEAJy1HR7T8YhO8kSjN1LfThflM/1wKtRmx0otC619zS23R7y2I1M/YOmas42LTVi8qwhsoLE7NC/1cbZpacPl3g/5sTY5Li/3LCo24qW+E4zN2+2XEYftMNPITEvn/5IG9T+bN5yewtGEqK+w6dnn4mRiqw+5Bpp7K00YXbX5M8PJyKH5k/Ujm0imuf8o1i/sNFmifiX7E6fQVlZAXsRcDw4VXiiT3FXXT9MjgF0Es89pO1DZwcbWFH5ZsfiXPOHBsQuTgQvRHlVuY5Yb8K/e1d7qgf1Y+0jETMrzpbzcJfkWRkt6HP Zd8sVIBK Jjzs/+8R7Nuw4jQM1lJkE5vQQlgUsp7nI5+EOlrfNvpQbk08WhyV8IGzSNhV76kVd5iTqr13h0hklBoLJtkFm0lBk7aKztp5jLyjEpSxaK+shIbfwVdT0iEHVY+J66xy49eGFLaViW0SOcKUphjn+5j5FrOhbOr2RFb9BRUXoG2xJfh+2QnztiO8+C/WTSeGJ4xFsbAejI6XGBTWOiWD8f1CxPnJiyR3VWEL0lvShaTaZ4YMYKD5lxEUaOxnW20dYMoXz1I3O72rKOE8Xdx6hokzXKPVuPhSJN8Hodq77MkPz/0ueLsynWv5yrMEA/xXMaUxFwnPgDE+ATMipdf93olb14C0CZIiy9NnnmU0J9DTw1iI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 710b10c6e336..b32b959046af 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2031,16 +2031,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio = ZRAM_SECONDARY_COMP, prio_max = ZRAM_MAX_COMPS; struct zram *zram = dev_to_zram(dev); char *args, *param, *val, *algo = NULL; u64 num_recomp_pages = ULLONG_MAX; struct zram_pp_ctl *ctl = NULL; struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; + u32 prio, prio_max; struct page *page; ssize_t ret; + prio = ZRAM_SECONDARY_COMP; + prio_max = zram->num_active_comps; + args = skip_spaces(buf); while (*args) { args = next_arg(args, ¶m, &val); @@ -2093,7 +2096,7 @@ static ssize_t recompress_store(struct device *dev, if (prio == ZRAM_PRIMARY_COMP) prio = ZRAM_SECONDARY_COMP; - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; continue; } } @@ -2121,7 +2124,7 @@ static ssize_t recompress_store(struct device *dev, continue; if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; found = true; break; } @@ -2133,6 +2136,12 @@ static ssize_t recompress_store(struct device *dev, } } + prio_max = min(prio_max, (u32)zram->num_active_comps); + if (prio >= prio_max) { + ret = -EINVAL; + goto release_init_lock; + } + page = alloc_page(GFP_KERNEL); if (!page) { ret = -ENOMEM; From patchwork Fri Feb 21 22:25:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EE05C021B5 for ; Fri, 21 Feb 2025 22:30:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A2B4280008; Fri, 21 Feb 2025 17:30:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2535E280004; Fri, 21 Feb 2025 17:30:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CD10280008; Fri, 21 Feb 2025 17:30:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DCFDF280004 for ; Fri, 21 Feb 2025 17:30:55 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1D900808CA for ; Fri, 21 Feb 2025 22:30:55 +0000 (UTC) X-FDA: 83145398070.22.873F802 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf14.hostedemail.com (Postfix) with ESMTP id 12E1F100012 for ; Fri, 21 Feb 2025 22:30:52 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=I7MwnhZi; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf14.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.41 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177053; a=rsa-sha256; cv=none; b=r90Pa/tGV6Tw/n9RwYIrGxD/oUxoUI6msj6Lkq2bLTxvguc/YZGgFj6Kq2Jw34uQ7jzxC4 /TfBYwP4bOs5GeNdqMzR+H8j0/7HG4JbqGRG8mucboKOnn9aLOKfz249tocEotFXgUIcrk sKELYjZq99tlhbT1XusrtDuTdmA7m6c= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=I7MwnhZi; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf14.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.41 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V8ADKywr3WoOFpGXL6UuUKwoYnHNcDBvl5iV/2gcJVI=; b=uG0D77mX2TplBnUSSBLylWYwXeilwb/QcSfBzR2NtDYzTjxmHegRfLDOSYgmpda4qDbK+2 lqMblJ4EVXbluTG2rPxF+QNFTxMtJvBtZM/vsRcICwYBbuk0Rdak8PrVRk6d63bJhtajSt 9mfxtMCs9D/sjI8wZW3G9uSX0ulJkHs= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2fc0bd358ccso5383645a91.2 for ; Fri, 21 Feb 2025 14:30:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177052; x=1740781852; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V8ADKywr3WoOFpGXL6UuUKwoYnHNcDBvl5iV/2gcJVI=; b=I7MwnhZip+KC+AfPvrbBZEm45ADHNr6H5sIFsIp9iweEV5wUqR0bidK00/UiZo25pL Eeh2qcaMjITB5GCUzQgBg6sMdjlt8SqvPrcdqKJYfIuHvs6eMtEHyO0PO/7+2wqhwnif dCkJFUhhi469nhiafYlcFE1giZ1bQGImNsbOQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177052; x=1740781852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V8ADKywr3WoOFpGXL6UuUKwoYnHNcDBvl5iV/2gcJVI=; b=DyQFJIGMu7/K8Lq3c/TStMEIaIb56/Ti/xO2hDowzahhqJhf/wpbhPlWTWxCgZ4njt gtQ4pWhnFsI8vAz62KGluJekpr2CpygFdPR+HHUj1VwXjxD2DYyq/GZPztqAPFrKLwH8 67S0JOVgNyTTwBkUoE3o/cKUL53/g7ua0J1Mb5axM1eVz8Vy2LEvDtOA+a4lxrFB1h/h C5t1MtmJ6uVhHMFp8ynZ0wEPaOfXOsoA0L+gCtcuh+pqZW9J31huh8FY7Lp7dGb5ieYE zOeSgvhW2uzOPQVDiGoNXghsVglCuXO7nZdXdkExYdWvhlfRTWhT1Lw0gcJFh2fo/WlW x3jw== X-Forwarded-Encrypted: i=1; AJvYcCXJB1mtRUjhgu7BeXZdH6xdEO9zhrSkk3y7mr4w/O9oPNw9AAWuaZjY8wHfuL2i4RTlhJBZkx87nQ==@kvack.org X-Gm-Message-State: AOJu0Yw7UBm7499Va+ymSj2GUzACJf+v9jsz401mkdHJiKjs9Mfk0VfO h1UhD8KsLyESFNclWERlFnX3XqAvg3EO/pEIZ/z0te+u6jccsnY/WPh7i2OG2w== X-Gm-Gg: ASbGnctt0dJPoEsRL6TLf0ALlcImZXBK2k8KtnkQsQHqxwzyAJmxlSHRsxr93LSyT7c kBvofjZRWGfAjcoaI9IA9cxZMCAkPNUSjp8U9DjLyyyVGzseDA5Spn6pfm1pb1EQV9SUOIQlem0 v4ucpQ1wRvhEGGXtD2nYyfL3u2hyTqQAtRY+n5Z5QAfHPiq1G09z5ovtjih8/vHgJHbnlxd1qY5 LnrH9jHKSE/tkgWBg1byO8Q8n2p65Ugh7Z8eJw0MKE/P3aoQD+oAiKLZbEdwNxAH7PelXZLtz8n 6aVKM6sQjft20rNE1ILAbD1xEyY= X-Google-Smtp-Source: AGHT+IFoSRURobpKbzExYiuyv2t2GitVExUur5YjtZeJcx2+K102Tu8eedtNsSQzkCzjNm/j9ms6AA== X-Received: by 2002:a17:90a:da87:b0:2ea:b564:4b31 with SMTP id 98e67ed59e1d1-2fce78c7871mr7538085a91.19.1740177051891; Fri, 21 Feb 2025 14:30:51 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fceb05f8b8sm1961471a91.26.2025.02.21.14.30.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:51 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 08/17] zram: filter out recomp targets based on priority Date: Sat, 22 Feb 2025 07:25:39 +0900 Message-ID: <20250221222958.2225035-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 12E1F100012 X-Stat-Signature: 7fyhtc7dhpywwy8kb1mhaoenbgfdgrkm X-Rspam-User: X-HE-Tag: 1740177052-254706 X-HE-Meta: U2FsdGVkX1+Urqr90lHTZhGOHAgtqLkkb6IVDglU/cKkYg3g3LyV2UrmRYDOkUPeC8OyDbm6GtbekhXPDF5KiboZu0A2YsEeN6m+PwBL/aCBz2Nluf/vBWPqjHU1DtIOd61ir68yl+aXhXwX+/1Hf5NjgwvGJ/4nz5lmRwehKkheykCyCWsxZ3TdCSJwcnVh393A16igJF/j58r7WA2frhVoCvbih0hddW3XpQtaZsCOe6GjquFXCscJUTe593YhQPTGy3aqZHsDl4GYa4HZCh3lHNrb1iOb4lXworD8yqyvw5VGSA7eDupUI5I6vupGYzhIO3HXyh/5aYjYQXqY5IazU9Ky4B5z9Ks7qFOjTgIcfpO9YKpxAVMEQaP7h/Xo1VMUa/pj7K5lsP3QiPwf5XjqSyK7IzXUD8v5Q2eydjzbkFPGf5BANMAq8dACT98xP+DyBOwrK5eVvDiV5gqDSQ6uhdeh2l9iPSgSZIh/t6Q7a8pRU4egjxmhxCaR4WTk/8wH+KK95hPtH5HyxpdfvOTSc7IeOEN0GJj1R0SuadgwjHHVZnQ8TeGKv5S4g+OkK7ZYS3tFiPZjE0B0vE9ffR0n5W8P2HtSdReXgJvh8cb8jrCJc12EeoOuxWAVioShW66cZHFJysZOb1bU3YryLkQzPcGm2Ps57jk33kKT7/1l5WFl6PxYy9c80XcGuuii/CU7bvYhXesLWyyjUvknTDG3pO7J/jDppO/eki7S5gt/SQMIVnA+VrsCqJzF6ozYFV4wqOhb2y4oGVLAwc1hztFD1Ea/27BNsfvD8JWy6ymV0+zhGNd3/WmO2xanWr2LTLFF5cMCN2n7Uoylfr2vvumcwv1+1S6naxF2Ec5pQq1VGUPVhYkR8Pn9AwpzXmkdFA4AbQv1dBRJczumXi9SZFcXL4aLuB5symJLqLNIegWFxCrDnp1UR+B0wKmVfOYwKnUhTLodFXYMnxCg5OI DYw9wMhu TFIpwI/x4U+kbplzORPguFwSAfdEHbUiROOfQEMOdtksZYWrJVaXMPIvlV90tjUHdxAEvWJb5zhiN2xtbWmFqEWdObhgshiSGqchwhp11vI6i5XzMCO5zeV4fSK3xAPgsduCipe3XCuV4gscxYI7wNGVj5I+WOlrLfq3TiQWO2XL6u1wE5FwMnrjbYlWakSn8HRhJAm+ByalcqVHJl1o3hO4x5LeGWrntMBx10Fo7KFj0czoegGHXp+s3724Mac6xFtHryHjloDcdopw41YKOn0ZxhgFXaM5c8ceg+57HKIjDjQHa6nisNvzg6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b32b959046af..92908495c904 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1827,7 +1827,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, struct zram_pp_ctl *ctl) { unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; @@ -1859,6 +1859,10 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >= prio_max) + goto next; + pps->index = index; place_pp_slot(zram, ctl, pps); pps = NULL; @@ -1915,6 +1919,16 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio = max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >= prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1923,13 +1937,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <= zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); @@ -2154,7 +2161,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); ret = len; while ((pps = select_pp_slot(ctl))) { From patchwork Fri Feb 21 22:25:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E41CC021B5 for ; Fri, 21 Feb 2025 22:31:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F015C280009; Fri, 21 Feb 2025 17:31:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EAF8D280004; Fri, 21 Feb 2025 17:31:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D295A280009; Fri, 21 Feb 2025 17:31:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B5247280004 for ; Fri, 21 Feb 2025 17:31:00 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 746771408F9 for ; Fri, 21 Feb 2025 22:31:00 +0000 (UTC) X-FDA: 83145398280.21.919FDE4 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 5DF544000F for ; Fri, 21 Feb 2025 22:30:58 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RzR9ZIoL; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177058; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yMiFPpRmjEuGJFug3yngS8shb+rlTOrw+KkttJGol4I=; b=1SEMFEZFMol5eN4S5uxjcfrWMQSHwPVMjlC2CQ1+zqSxwu0KADaxbFe/e8EjuabPpL22zW pcsHgpTI5w5X0fbWXlJyHreMohRBAKBlkHuAz2tq5h+apFoe32cCDt/ycTxG/VguZldLrR AsiSP4yjoQdGcAhZCsYLP4B18XTl+M8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RzR9ZIoL; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177058; a=rsa-sha256; cv=none; b=UyZdnvgvMXPC+EgZ/5TzqA3dHMiKDrcfqbfiQ0ddYaAUQY3eIlsF/IYeB/X6hAfvRX00bq 1HTQRLZrDNrkRa19SqVC1zObEJBGibKgjV+7hl40mD7aWv3TGtgXMoYNCd9aYMY1yO14a9 OuIJecHxP2+4BStEwPGTMkxsQKzZeOg= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-220bfdfb3f4so59879605ad.2 for ; Fri, 21 Feb 2025 14:30:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177057; x=1740781857; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yMiFPpRmjEuGJFug3yngS8shb+rlTOrw+KkttJGol4I=; b=RzR9ZIoLOs/UPBR/jXNmPBH3NvaILnVCOrFzVjzUMK/NvoRVn3s3VKeP+vE0YMdl6X eMoIIQ2Yji+saX/YgJNHKY/UHDya2+vBMTSxphBnJWBFMJd+BSe6VJ6lwpFNVaRJ2QQg IUYfFejj16QLkVLLq8uxnzlYG87Q9BDDRXqlk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177057; x=1740781857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yMiFPpRmjEuGJFug3yngS8shb+rlTOrw+KkttJGol4I=; b=QYY3QISqPdpPRZ95Ymh0Wqs/ygiQUFtkqQKmynKLJ2yUTn2CFWQcEwj+x78RrZpHeQ l11aL3X35fLK5IpRbs7d//QaK+enB181eS7FoNRem2xRL7mgO7n1qZ+9nFzKMUvpXcIL 3zCyVGyISlU8guZ56Zq9wccqmuCnSPh3/4cJQ8DouTDYO00DcI/E1sJeDUbLpQVk/L5m rSnu3wkyicxSJgZ34YpCMdJ3r6FqMW76Fjteba3WO7SlXDfsAIgRiqcdlY8ucEmdbXJ8 dauz6qUbwn0aii6Et3TjUlTQ5ANFjOUI8KRouuKLt/J+7GYWRNTZm/kVjVdJVeFShYLv CTJg== X-Forwarded-Encrypted: i=1; AJvYcCWLYXNcX/lmnsdIWP/G7ggqLQUzG/to/EjGOtpdIonoWL2nF/FjbOUP+9gJWSOfoGKAEJOGMimudg==@kvack.org X-Gm-Message-State: AOJu0YwW4xRj65z/B0R8zraqVTGomWK7Vbuoh4EQmGd00rT3NEqjI4MO o6b6C2UUHJoZaQ6VCJVGLxjD/EJ9058OuX/Ujd+ROb7pCBxf2gGURwDwhpdEBw== X-Gm-Gg: ASbGncv1w+UqHChwsh2IiqGI9+pxEmBxMCC7vnopGOpYOAH3e/KyntLfrxiSFWRzhmt BkEcREvzLKODqnl9MOXOUyE4g25Zg//rjf4jpAub3aINwcynG87BBWKKaBiWAXQNW5IedwNIB20 4tEEgAZNzCMLQgdGk2xcO8U5IxtCoqEwUkNZARtsyBbugibTtDSkgy3gBQdG6FKZLOx1ZZx+nbh GHoDmbRC+s90lehU+7VfFzKMOoiaIjrTCaO+0tTWx+Y8o724hiaGvRDKvJHk63X+HRSJbyQmcjn h12qKE0GKDb2ycXRa0CLLFgmdr8= X-Google-Smtp-Source: AGHT+IEQrQ7GyEMKY0xC/QjOZMueeZal8jjrcZZyRbZ+PqQnVupxiIMR8/upUlsyn1jWc83rlKp/Qw== X-Received: by 2002:a17:903:32c5:b0:21f:1096:7cb with SMTP id d9443c01a7336-2219ff50eecmr69781335ad.20.1740177057264; Fri, 21 Feb 2025 14:30:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5349056sm142860655ad.22.2025.02.21.14.30.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:30:56 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 09/17] zram: rework recompression loop Date: Sat, 22 Feb 2025 07:25:40 +0900 Message-ID: <20250221222958.2225035-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5DF544000F X-Stat-Signature: djk3eny3kkz9b9xa9nxt7rgqruxadq7z X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740177058-719546 X-HE-Meta: U2FsdGVkX1+TkYfd2C3jq/f8FHdDVcF/d35DsTDJirN6tivCRzUBlpjxeh24udwJp8kSnH53QlJho177cWW61zaXp1zNgxx0ob/pptHCKJixeptZh/jA57c9oIEV83qpEtvMQQBtfHk5dN3UAppxjCId71Wg3Qd7QEnOkKhrk1EFOpx1gX8Xe16wy6dyNW6CMsDJAGjGGAjs857gqk2KIyZGKxKv1Zu87x6Acan341HDucJm5TpWLV2DloLx+hM5jF3mpWssRlu1vkbPWvdqxxX7+AV8TT2Tl5jOqjLgDW/mJ1IYaXxM68ysTS1tb11EJDOwVX6D3F3cgf1oIEC1hErGzSQVMxxaYHbJTjmuGkuEpdr+hSF58+V/kCjI9spcZCt/Bnh2Tdzwlp3cdTvlSEpMwAAFz5Ad3UxaZZi5mrm2WBbE83cIgZLOLa6/s2dMaj77NFmupYQZ8T+/PWqXdIfy5UIvXEvz1qg/19GNxvQKCK5rzyhkMlgvgtmQYNPIi9y1j6JHNiaSgQPSumnLHXGzaTrcDTPNpO+wM8pFOgytjNEX46Azaips7+n9h2KgHHcqv3z0YnjsCLc+x3lcnZDBT12cm8NZOr9sV42L9n7iy/CvK8jnL9oUEPh+8MqKdrPS8vANcx/FJXCo9kKhC6btPw+P30HINUdLLxULS7RdslRsY/1gxoXt1p7usn9lG1LK/OVVieU+9XPqCJMVB4mugN3uBFgFYiov+S7/tGiDppIaPWGDiqZUEqkcBV5nFXU7FzJrMzhhvFNJkbEARB9QeaN8roj/FOPoQIytG00KcVmwS7b7jiySoXoSWdtFL+vsTOAh64r+M/Ij81G0Qi7ERnuT6uhX0lzTbb79hYfdfUuULgEAiKV94Jhxd9J8/l2+q90AIIeqSbt7WwY8q0X1hcLFnR8/4+oa3so7K6JXzSaVF6M2NhSV7Qak3UJwlpvUHuSD1drnSnqTsOl mpzo/WBa VFUgfgmbJt3/ggv0F5nEzLD2wX358nvgyDooxZxlCqJhU3iGHODkJoXme4GCdJuJmQmN7uU4YV8XNJz+DKJVWiqnTX5+WZn5oWKXvTKWX1i253iaaCmgF126+W72Shumtg++jG6ZCloG/FXmGROsZf1YrYdFAV/EwpNIZoj/ho0YjQjniwnPzdTQVAflSQ9URabe2Rie62acHTMUBptB+VjXQbIVH6wraWCBQfofv43sjjI3E1GfyJlqeDfChrtqpwSxf7lvmSb9MkJatDGvgR9fEyyjCW+tW+R4j4vRZmIEaDWflzbF9fNx0sw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successfull recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++---------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 92908495c904..b96be8576cbc 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1892,9 +1892,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps = 0; void *src, *dst; - int ret; + int ret = 0; handle_old = zram_get_handle(zram, index); if (!handle_old) @@ -1937,7 +1936,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); ret = zcomp_compress(zram->comps[prio], zstrm, @@ -1946,7 +1944,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm = NULL; + break; } class_index_new = zs_lookup_class_index(zram->mem_pool, @@ -1956,6 +1955,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { zcomp_stream_put(zstrm); + zstrm = NULL; continue; } @@ -1963,14 +1963,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, break; } - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1980,38 +1972,31 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (*num_recomp_pages) *num_recomp_pages -= 1; - if (class_index_new >= class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps == zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >= threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new = zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOWAIT | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); From patchwork Fri Feb 21 22:25:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F1ADC021B3 for ; Fri, 21 Feb 2025 22:31:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8AB628000A; Fri, 21 Feb 2025 17:31:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E39FA280004; Fri, 21 Feb 2025 17:31:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8C9828000A; Fri, 21 Feb 2025 17:31:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ACA93280004 for ; Fri, 21 Feb 2025 17:31:06 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5F69CC089A for ; Fri, 21 Feb 2025 22:31:06 +0000 (UTC) X-FDA: 83145398532.05.45551FA Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf15.hostedemail.com (Postfix) with ESMTP id 6085EA000D for ; Fri, 21 Feb 2025 22:31:04 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=PyW519XT; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177064; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=J3ZNLw+bHzrQ2FiPKUCUzq3ZWmHLnoMypefDP5KBkrV+tz0MthpjCbW06frwaY27r7iFGh WVynt1a7w01z2OD3iOTdkxgE4dP7547roLz6OdlDGIZdYN1bTCcNLJZ1CvVhY/RqBkLiuQ KZ0XynD7hKZ6XGE33i/KliCUyDoBFRE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177064; a=rsa-sha256; cv=none; b=elflGJRWHrRNC7I5D1NcyRy1uNAI+ZaGyUoIFzqe3Fs5BBT/jgOeooM82v9Gy+hisuZ2bN 0np25H/6rs2IygHcWv1mET7RS66XTojb1h8B5gWjEvodteGn/WwO0eh6PGAuCFp4mdt6ep pWG7ShLW1dbL59AiH2sClX4JsgO07CQ= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=PyW519XT; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-220f048c038so49789815ad.2 for ; Fri, 21 Feb 2025 14:31:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177063; x=1740781863; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=PyW519XTdYDZwOoiWyYshlGjMHm/cW6s+n4VRUpgbSpR6Gf6tyMePOyxsUHKuC5wVo iFl2+rbldNxQpsbVxxe1wwxX4KQNEUBbH9H8p2L6+2bYX+p++s0uY++FiMTQKxbrdVVJ upcdU3awYciS0qbjPk0H/Iqx7snsrH+rOjZ3E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177063; x=1740781863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=FWJTxiGQkNdRV6lVUrxivj58quOmAjN7GIG7bAsW5s3tSsss2lc+qpH+wEy66znrRH f5XXZdiMezgSJK25w/IHO+YCGjrSn8bb1Jg43g3l2ozINZfI6KhV7tGxgMDIjbV2gpiP 4WfMFlGt2W6l66zJk5uEH+Jz4GzdxXbn18O7ygGptTJ2CAWl7ERAyfFx0FGTFuensFjd oo2B0qIiSqm+zsNRrUvWpHq9x0M/0JNg7N+6275TdE8WDK11mPqrr4MwrHGo1mGRAv7r m0NZhCNz56KbVgIPenF2riwoEBZcIi0B2kl49+ZfD0NePIk+OzHnoFUFWhobiy5eg+xe CVOA== X-Forwarded-Encrypted: i=1; AJvYcCUR+tsZLbmRg6Uh70YLeWO6uXA6N2BjHvwlDcbaNUEwJeFnxmqGkNBMBSUBKu653Sh1u5kvLrgMhQ==@kvack.org X-Gm-Message-State: AOJu0Yz38z/bnjRyuEQ88d+P+Koj6/2JDrU8BTHlRlOCOqcSIwnA6Qdh dGJv8l+NkzsFqO2GE2o+r8xMWK/+Ad9Fs78dyRuF4bwxOma6fTK+XQOrYytXyw== X-Gm-Gg: ASbGncvEdAq6OjCPskuNOo2dZlcyNd8hvIhxCn2CtWIMgWxqOCMGH4GS4vVx4UYTm71 vqO/5F0fozTPHE+6LweTySGOps//QQibukC9QD37y4TwSu8gB64w15V9wpusOlM86/A/ogWQZAi I3Yio0jLmFQgC/l6WiRR1mIsrV4Or8y+Rl1V497ZVT+1gqj8tDKcOV46zp1byaRFXAn7haDZzxW CzI9Vko9ssywsevrNJJGxs9OyCXuZ+KP2GFErRgWTSDI7s98r0Bs1KMfQMqX0P8Q0+z4y5UsCHm /BG9d2KtjkKcc/kqyzmcJqB0HPo= X-Google-Smtp-Source: AGHT+IGH+xDdgjkZXGk8hJrCHMpFTK6rCRlJkYunn5adYXvOt3xJPKeFvrD1nPrfp4BOcOqmXKFuTQ== X-Received: by 2002:a05:6a21:730a:b0:1ee:e20f:f161 with SMTP id adf61e73a8af0-1eef3dc6819mr10529598637.34.1740177063308; Fri, 21 Feb 2025 14:31:03 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-adecac2150csm10427909a12.67.2025.02.21.14.31.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:02 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 10/17] zsmalloc: rename pool lock Date: Sat, 22 Feb 2025 07:25:41 +0900 Message-ID: <20250221222958.2225035-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 6085EA000D X-Rspamd-Server: rspam07 X-Stat-Signature: c4idmutann57xjsrkmpgs389qequaw9q X-HE-Tag: 1740177064-695494 X-HE-Meta: U2FsdGVkX18hxqYeoH8eK+/m69FiFSf+lHrsdzNOKUWGaE6LKiBzt/kKRzRFgC+XZMmkInAttHNlqkwKz/QpgkLvumMtDiDJvZTQ4oZtXVrAowKTlHzuItXL6/8qftNAvZ4hqvJmBTHq01x9+wxujselB80mbPzd38WYUKfn8cXfEoiNZ6T65yXIJJxeiW6L1gPMC5YB4yM+DqYkbNOBogiMU8bzu4IVm8UlX5okg4OBfeu2TCS54FAs0RSyC5+jtJp2QiYVaOu4Swfzr8ZWEEqcf2AnTobz+gX3z4FcwMdYNvtrsguawoF6oTK247gM2TDzE9Fw8E7tWpxXEsF4uFvhKpI5xcY1D/VGMun368/QgJdVIsQ3tme+WlT6H0wU8W5kR/S8SBSBVwyLbxh26D45Y+5zZOllU19LRPjZE7/eZJigzKC7tcGu4jbFVrsN0t4wnMVzqltHLkEQJ91eqdBkgU/h5dbFcxa7gaXo1vTWGnJOkkqn37rVBow2ILVoCrbvh2RJJkQjnGGjnpNRrIYDoqPyuKleOe5cK796Bwg5lKuc1BTtJ0Q26Ak3PPrnzUPOZ4VXioiVGlrXciUhJVNlU5vVa0IvLp4E6sImo4TfO23Iu5rUzo93RlN1k60XN58aJedyAXEdHWVFGhipoRUutO8zJoOr3NvGs3SPu3QFzbhSbEPM64+PMDYxCGXuRpJpAD0Pc/P7g1ClcQQCrwDBlpsLAonZgiO1Ug7tIpy6M2FQ9JSP43pimDqB6RW1SUOSSIi+XJG+7w2MRLrgfQeGPB4P7H0e1UZHy0BknH3B2ywHUEnLpC7UmtTPV2JVwWz9atMcIQJpadpastS7t+aswFmYwnBl7gn+vYxiTG2Bu5YRKn0U/R9fzMUY9dzARdRmXNMXbl+Suo+HcZIJhWmukvhp5LVHkrHGERb4N/4CIrsi0kWPY+X5fIXLjaQGJ9cXMGjIm6MYbM+TO34 gS9jRlqc CVO3AWobYmgrmM4Fg5a+y1mwN0S3Fq4V8qSTGVnS+3o1SX66X4sU6OGNE/ta+C+rloMtw1mQA0hAvYHTttWZ5XVx5cGH5efh7nKc/BIzlPXb+zYJitynVAMKA1GIInfxvsmW0RVdirdQy7ZrTYNiMDcYNJmq6gvUNa+wqUaoYosRyUTYg1pGLib5Rhz7Z+6AYhXIXHKblfNEPoUwkScO5RyhXaJz4xYonfU0PMy5s+6ylY+JE5b2ZhDqoivMxWrQriTJXxttI2BM2HkSkvjBCfQsZaXr1tMJ0J6AcJxL4fKh4DO8i3YKIPQ0gWA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The old name comes from the times when the pool did not have compaction (defragmentation). Rename it to ->lock because these days it synchronizes not only migration. Reviewed-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..1424ee73cbb5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -223,8 +223,8 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + /* protect zspage migration/compaction */ + rwlock_t lock; atomic_t compaction_in_progress; }; @@ -1206,7 +1206,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1218,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1450,16 +1450,16 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1796,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); class = zspage_class(pool, zspage); /* @@ -1833,7 +1833,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1956,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +1983,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || rwlock_is_contended(&pool->lock)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); } } @@ -2002,7 +2002,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); return pages_freed; } @@ -2014,10 +2014,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed = 0; /* - * Pool compaction is performed under pool->migrate_lock so it is basically + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2139,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); From patchwork Fri Feb 21 22:25:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79B88C021B5 for ; Fri, 21 Feb 2025 22:31:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0CFAC28000B; Fri, 21 Feb 2025 17:31:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07F0F280004; Fri, 21 Feb 2025 17:31:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E144828000B; Fri, 21 Feb 2025 17:31:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C290D280004 for ; Fri, 21 Feb 2025 17:31:12 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 822A71A08CD for ; Fri, 21 Feb 2025 22:31:12 +0000 (UTC) X-FDA: 83145398784.18.CBD6719 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf01.hostedemail.com (Postfix) with ESMTP id 784DD40015 for ; Fri, 21 Feb 2025 22:31:10 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="aI/4GTy+"; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177070; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y4hieFtM2l2Z0Gze73AM0fWmJUttGTKTNzwCP28e8QQ=; b=0lBbfaDD4KQFAjYeCnoIaGw+4QVl5cSe3uDY2+rhench4Tpumj+EJXh5buOlIjWatuml9S P/N8AnWOYVNMoSj4AJ9ucRh3/MvVVP6jB5WxYKwDmXhoksFZRookwA9gryqx4ctwKE3Jpw Yx+5AWEohLAR7of6ewQF90jD2JY/97Q= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="aI/4GTy+"; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177070; a=rsa-sha256; cv=none; b=KUPgooyR7Z0vvyvDzCAhkW/rvScd6HlVSJFjeSyBPkHQ/CapCh6E6CWCro1qauk1nN8bOA GeQvGMF7zhVvit0nl5UBieRgb16sbcMRBraf9pMAJoRdZE0UY3ea/vjKMWPLret3J4xr1G 3jC/jTS276eZq1odLvwk/sT7dXhWd9A= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-220f048c038so49791145ad.2 for ; Fri, 21 Feb 2025 14:31:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177069; x=1740781869; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y4hieFtM2l2Z0Gze73AM0fWmJUttGTKTNzwCP28e8QQ=; b=aI/4GTy+euRefg580f3J7rHx2HMGKfFnq1oN+bIwkqivFDivWE+iWbMffi4DgVAw3a XgxcLnKo74OfJhybYCKsZ+DngnZXAQnM+v8ej7Z6vQuDHHR8+S5RmlwoKzoAleIC0aKl t2oNn2QGhEjaBLVx1zKbwPF05r3doeKxtbtOg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177069; x=1740781869; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y4hieFtM2l2Z0Gze73AM0fWmJUttGTKTNzwCP28e8QQ=; b=Qtpv7LfTjRFXYRXv3iLl0Qe4o7xhfWAXW5Uav6J0gHBuNnonVMiE9iBn+SFcG8k7SD UJCDOzY/LoDa2hwHq/5yjEhjALtSFpYUcd6sQ+UUtSKnmlL/GXSvIr2EL49eJyHlypif 1xz9NySQu56zw3eIAU6iBIYI9T2/ixcHdivYd0fGJoJ2ISvEDhYfI4vdcaBH1hBwGxTu t3nczkdmXmDVQVcMovSHCwnYwjfpv2x8vbDblaTZkHyWOSq1qYD5fvPTw6FxgEWtxuV6 ZwImfKuoYyqPfd88EAz5vh8dORiaQYF93qRlP0GzpeQhrKFMhtpjA8PsHHPM234SUlAR heVg== X-Forwarded-Encrypted: i=1; AJvYcCUg9bsD5JRRwg34d9CbvS1qKucHb/Xked7RFDrjYnUxicPfNxxIJQrbk2nCIAeAnL4nbz1O0NpVEw==@kvack.org X-Gm-Message-State: AOJu0Yy2Dhd7H3HGlzDIN8SwSrMK9rVtxQKjTZyYQQIlXDr4J1HtJkvq atsSmSz0e6J0Eg39QqBb4ntezF1Q9/yvHBjZ98QkgfrSlOwem1KQAEe8JuAlyg== X-Gm-Gg: ASbGnctoFtoWRBxFap+ghHNoYv1c9QRajQkK1pSo10P/7mSLN5VZM2omUGU0CleGVk8 +vwQSfcWZDKR2XuxFZNb9IAjgZ69KL9hIH6hEYXS4DxOZwNa/IHF5M4M0PFZH8n3zIChResyLkK 8gDxCNdcRrFYSEfWAkAQZohQ5qwQnnz39RsEB4VtxjX5ggn2NvMJ/6OzMhlYYGj6KR90aj3j3ow YJcL0lH5DlsMCXzoAbJ/bERVc33FXZQaOdEmNwj60DAdxvwbBc+k/lMwsvHJYyBFkKMOWU/W2v4 WeA5LeOFzDBwcgzBXYginvgvijQ= X-Google-Smtp-Source: AGHT+IFjDhSFSvfLZz11cldDF51zOwCTwD6xM3YT/eF2+EZ/8qQPdJ+e/mCN9snGHWZACIXCyeoKVQ== X-Received: by 2002:a17:902:ea11:b0:216:55a1:35a with SMTP id d9443c01a7336-2219ff61e5cmr65829295ad.30.1740177069320; Fri, 21 Feb 2025 14:31:09 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d545df28sm141974715ad.153.2025.02.21.14.31.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:09 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 11/17] zsmalloc: make zspage lock preemptible Date: Sat, 22 Feb 2025 07:25:42 +0900 Message-ID: <20250221222958.2225035-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: ohday39dcc3p95tw15ooiccdtj6uf6jz X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 784DD40015 X-HE-Tag: 1740177070-341049 X-HE-Meta: U2FsdGVkX183irLNlDbPBcRXUQNQYdRQqoLsWVqfugpulCSdosxkaQ0lMcRYAmMs1N+NqzSqqNpY3X+FAbXoDRd9IffsQ0ReMgTAbGmBuNG2o+flMSWf+kzgsRjTt+Z4tU1+MDunEUDLtkfKNPC9rI1fh6wlJlsS616Lekst9HfAixEZDHOzMcS/AAIFjDiYJu8rfsQOlxafM27w88EQBI1ovWU/nJ011SVbOxyev3J+RHIJMlkax+GH40EKMe/hUByW+nlyj4l+C8FXL6wUNwGi4sBVkmhh4AZo9kJ7ZkSwjKNGjtx5DypeB50KP97FHGdSXMB09TbR9hGVwvQeU+RGlEVbAl7/8meRGqVG/memDTEySOk0FpVhE4d/lzWAOvBQgUjGDZGV6J1ue4XsXIt0WR1J5OiC/J/20uoSMqjs2q+i8idZ8XzzD5AbYxPtpBU986TszDyLvzvmKONeM52srFUNWQCh+Roz8PN74/R9RYtlqiw1rj6uNAkYcMmDzMUynXf3BQAXdhnp0e74pW9e0HFoGYbKZaZ2TQdFLuOcYUim8dKTEnhYI2If9gADbb41NKiY1dufCXhYVff2ITqSqW8EyEejQh93LzN2gWSOFUtfYLqDtNV1+qGMfdOFqV1xCRRCFcOEmHWALdjm/p0RTyuyo83DwP4nKvVYhTlCZUp3eAL29UmsfaIUcc0fSbuxUJqxJouphaLAWG/LsypHBw9giWoI5b3pucbnW1uJhJSKcqy1KCjVEphq2DvoRCJ8OH1fmVGW6cLnJzEr8tVPadbH/5cKJ6xEp1lV5UlYS0k+MbniRYirJCRIvefVfCRXxIUF9iicJXvfN8l1BNObcu1+dJiP+v6+fGAimq76ewgYeB9FGeS65Z0e1WftO/22M9VBA9spRaZdww8rAIgwwTGoEOvphv0BO7YPRfCgF2ktMZ72TRQdzWgwLJcKEYw0f1y8Cz5ZMVZpg8/ ZAQBbpHc 6Gplu/46BJ/ewR8DJ4qZFVBwPg5tJ0c3CxSEESU+3jyOwx7PP54+Icar2DH0pQ2viVnjFeqNttqCebDHln8onMtNNawfxdVxyFOB+PyQQUMnGrIrFOb353l5QxiU54P+kVR2R1yjtnjDoMipESQisI8AF1F3v4czb6D/DWeoIo8X8eFVxo5oCsZ2Zm0+7pdPtuIJ/0HjCrdvy6Tnfc58FwwFJzeZ8AlcwaHJmr35PXLocr93bwUxLUgi/oqzccYEPZiSmBR3h+igmwvP8XYi5JjUv4bYuaCnKqrGt3A/YCYZobpRAZEi8IHPtElqSVP5KIdesCnDpAm+cno8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to implement preemptible object mapping we need a zspage lock that satisfies several preconditions: - it should be reader-write type of a lock - it should be possible to hold it from any context, but also being preemptible if the context allows it - we never sleep while acquiring but can sleep while holding in read mode An rwsemaphore doesn't suffice, due to atomicity requirements, rwlock doesn't satisfy due to reader-preemptability requirement. It's also worth to mention, that per-zspage rwsem is a little too memory heavy (we can easily have double digits megabytes used only on rwsemaphores). Switch over from rwlock_t to a atomic_t-based implementation of a reader-writer semaphore that satisfies all of the preconditions. The spin-lock based zspage_lock is suggested by Hillf Danton. Suggested-by: Hillf Danton Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 184 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 131 insertions(+), 53 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1424ee73cbb5..03710d71d022 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -226,6 +226,9 @@ struct zs_pool { /* protect zspage migration/compaction */ rwlock_t lock; atomic_t compaction_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; static inline void zpdesc_set_first(struct zpdesc *zpdesc) @@ -257,6 +260,18 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + +struct zspage_lock { + spinlock_t lock; + int cnt; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +284,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + struct zspage_lock zsl; }; struct mapping_area { @@ -279,6 +294,93 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; +#ifdef CONFIG_DEBUG_LOCK_ALLOC +#define zsl_dep_map(zsl) (&(zsl)->dep_map) +#define zspool_lock_class(pool) (&(pool)->lock_class) +#else +#define zsl_dep_map(zsl) NULL +#define zspool_lock_class(pool) NULL +#endif + +static void zspage_lock_init(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + lockdep_init_map(zsl_dep_map(zsl), "zspage->lock", + zspool_lock_class(zspage->pool), 0); + spin_lock_init(&zsl->lock); + zsl->cnt = ZS_PAGE_UNLOCKED; +} + +/* + * The zspage lock can be held from atomic contexts, but it needs to remain + * preemptible when held for reading because it remains held outside of those + * atomic contexts, otherwise we unnecessarily lose preemptibility. + * + * To achieve this, the following rules are enforced on readers and writers: + * + * - Writers are blocked by both writers and readers, while readers are only + * blocked by writers (i.e. normal rwlock semantics). + * + * - Writers are always atomic (to allow readers to spin waiting for them). + * + * - Writers always use trylock (as the lock may be held be sleeping readers). + * + * - Readers may spin on the lock (as they can only wait for atomic writers). + * + * - Readers may sleep while holding the lock (as writes only use trylock). + */ +static void zspage_read_lock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_acquire_read(zsl_dep_map(zsl), 0, 0, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt++; + spin_unlock(&zsl->lock); + + lock_acquired(zsl_dep_map(zsl), _RET_IP_); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_release(zsl_dep_map(zsl), _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt--; + spin_unlock(&zsl->lock); +} + +static __must_check bool zspage_write_trylock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + spin_lock(&zsl->lock); + if (zsl->cnt == ZS_PAGE_UNLOCKED) { + zsl->cnt = ZS_PAGE_WRLOCKED; + rwsem_acquire(zsl_dep_map(zsl), 0, 1, _RET_IP_); + lock_acquired(zsl_dep_map(zsl), _RET_IP_); + return true; + } + + lock_contended(zsl_dep_map(zsl), _RET_IP_); + spin_unlock(&zsl->lock); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_release(zsl_dep_map(zsl), _RET_IP_); + + zsl->cnt = ZS_PAGE_UNLOCKED; + spin_unlock(&zsl->lock); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -290,12 +392,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -992,7 +1088,9 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool = pool; + zspage->class = class->index; + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1015,8 +1113,6 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool = pool; - zspage->class = class->index; return zspage; } @@ -1217,7 +1313,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->lock); class = zspage_class(pool, zspage); @@ -1277,7 +1373,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1671,18 +1767,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1789,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1785,9 +1856,6 @@ static int zs_page_migrate(struct page *newpage, struct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage = get_zspage(zpdesc); pool = zspage->pool; @@ -1803,8 +1871,15 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_write_trylock(zspage)) { + spin_unlock(&class->lock); + write_unlock(&pool->lock); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1835,7 +1910,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ write_unlock(&pool->lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -1971,9 +2046,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + if (!zspage_write_trylock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { @@ -2141,6 +2218,7 @@ struct zs_pool *zs_create_pool(const char *name) init_deferred_free(pool); rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); + lockdep_register_key(zspool_lock_class(pool)); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name) @@ -2233,7 +2311,6 @@ struct zs_pool *zs_create_pool(const char *name) * trigger compaction manually. Thus, ignore return code. */ zs_register_shrinker(pool); - return pool; err: @@ -2270,6 +2347,7 @@ void zs_destroy_pool(struct zs_pool *pool) kfree(class); } + lockdep_unregister_key(zspool_lock_class(pool)); destroy_cache(pool); kfree(pool->name); kfree(pool); From patchwork Fri Feb 21 22:25:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D663C021B3 for ; Fri, 21 Feb 2025 22:31:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E151A28000C; Fri, 21 Feb 2025 17:31:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC479280004; Fri, 21 Feb 2025 17:31:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3EEC28000C; Fri, 21 Feb 2025 17:31:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A21D6280004 for ; Fri, 21 Feb 2025 17:31:18 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5295C120894 for ; Fri, 21 Feb 2025 22:31:18 +0000 (UTC) X-FDA: 83145399036.26.8CC4F47 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf29.hostedemail.com (Postfix) with ESMTP id 4D84912000D for ; Fri, 21 Feb 2025 22:31:16 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cc3F+I27; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177076; a=rsa-sha256; cv=none; b=yssrdK7zVZgRfbsgnvaF1R13T7MFRqBHwLCP1ss1ek7Zwb82Ej8XtXXpz1TyIzjBryAC++ eI/PYfnyoYdL9R5SfWj2JW6josinKHwr6aM5C/FhyTNDX9vQL1ObqXWEJrTa7JZI5PwG+q caC3c1ftUo00XC73sWKvXXF3xod0RPk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cc3F+I27; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=zmvzJF0VA9q1hcetO/BOlP33R5jUtB2JXKI229/MUoPo30gx9OVGmR7hm6HiOiExsSvcgf C9qXz7C8ZzrqcQElWOk3BwK38xOHOnnh3CHh92qoJmnc3rr+ihQEqlxOVCTbAl2vPbkwN7 eOfeKJHa8ohTpCIxUzclp5LOA2JzaDs= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22100006bc8so47876155ad.0 for ; Fri, 21 Feb 2025 14:31:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177075; x=1740781875; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=Cc3F+I27A7vjELCOvmienxdunY/kbqHuUAFyM2YlU9p+36xPdDMNgzQ7QRd9DE9AGp EdbXSKTqJce9u2Sr/YlJEKiBUlQNVtlUIfhE5MM+frQvgSFY3l4c4dmMSxW3JBa3/yfD LxYVW/fSkB3mhGfVx1mP6Mr7g5aIWkJP6q2gI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177075; x=1740781875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3bYokSfTdUfiI0Hj0uXPo6i1n2cRTGAJHVL2SxOfWWQ=; b=i5UjPZwFlnuWpwc/hS6oD7gXnGgqPCkC6S6zltUqTNrX0TZSmJifhSLR2/+yizjkar xOQXMhIVeACjc/7vCG3rm/6i7dm7CmD47RP3nJFfAXBQCT/ERNgqLxcFy/gUtdpIyoFN +X5yTTCqgEZtAfEIcsHE/nPrtdD6GVWRy5xoNiFnqeUgQvvULceQadS7lBv/gsapGOq6 HZsUWK+JhplEbL6H8Lm6Pew9pLVIFrhPq0HK+AY7HkJLKoVN2pYPHDhW+R09ttGlwlFK fxOw4RVUwdHEuAzb+hkMaIDk9jtjlqzImX55I+CDRGhuTfIz9ezH2PuYFH81UqvAmqY/ M67A== X-Forwarded-Encrypted: i=1; AJvYcCUUEByZ0mst0BGrbJgKyYY30y7eLz/Nedn0d6QzgEAV7wv5h+G6aOAynw8yDKC20DljIoqmSgUyTQ==@kvack.org X-Gm-Message-State: AOJu0YzOL+iKHMiq6/7EMQXo6iC44d39785WIqycPY6DrnzPr7enGtdY OQkfouhChIq8leWyJa3M4cnXgw0748RsgpO/mGuVLb1XxnRnAybErtVa7XgStw== X-Gm-Gg: ASbGncuGGjfNwErKrFPG6VkvBYEd20ertAE3rNKBR9JVlL/8455Khjp98e/h9c+7r85 7xexJPJ/WFMSpBxcJ1GcLMZKm3EAd0PHqDmCwI8ngQgHZy4xphR+VwZtpP/aUc+9TVqMIHGAq+2 uAyLJMoUs+kAAoyeVyB+hXYfKRgFI/yKux7YMCVaYHEFtBCRBQp3Xe0LladDfGe4LZ+qKjE44Ur JYE97XZQrR4h4jg+Ge+nOfI6lYWgD8edC2Wguy1TuL54+SOag5AjPaIm7Wx+O+IuWAQuqKrqdow irxQzMX3SWMyAkdB9NMAIm2w8Ug= X-Google-Smtp-Source: AGHT+IHgZKuXmy0HUVJX3N4A8fmolG1cpyf+MQMTUT8UUMyAXq6Z3SZOJ2I2KS9bokpF5ag/pOAYiA== X-Received: by 2002:a17:903:8c5:b0:21f:4c8b:c514 with SMTP id d9443c01a7336-2219fffa9f3mr71842835ad.45.1740177075271; Fri, 21 Feb 2025 14:31:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-22120093e41sm95374735ad.93.2025.02.21.14.31.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 12/17] zsmalloc: introduce new object mapping API Date: Sat, 22 Feb 2025 07:25:43 +0900 Message-ID: <20250221222958.2225035-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4D84912000D X-Stat-Signature: cg4cxys5ufgqbat3m9ibym5uzr3kq1o3 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740177076-949882 X-HE-Meta: U2FsdGVkX1+KrW/wLII8BtvcIl6fR4LLnGCcKgO3NDZTTsXthLkO1ING5DYL4KNJloOrKT9DNHJMEGyWzgcYU//jfL61cB3m3vqklFOfMBPbuG5sVIVqsLx8N1FX8FSuDNQNrlrFMp9aIn9YtJiD5s17Ou9jiS7r8g3EdFCuZrwVY3aZ5zMwjVWaFuIhCkj8jVCQQq5sKn9rlh4b9CSqyMQlAZCeS21UGi+uG3nHIPElljWaaSYFGksM97wjkLEChkXYEYPLIuexj75dRuKEYHOItvDNH6rVHaRgpCafoyDgcyAnTHYlIsIJEAwv2EN80SugBB+MOkzuNev/sNTJ2GZJZqBUY5j3q17urbrK5GM6AP3nZViH2/zcqcItbhg+mz1+/+on2KyNm2A7V2QTkB5XEbmNPqKsq5zzc9MhYmSm+LRYos1fDeWDhKxbj1t+BdHRJuaqB1lRTY/4suKT5qThBK1MIznR0C+PAZpF/mKlMjz5E6vpPOm2c7e6ceEQ8VEiSdv8nUFbB1LLZGBOeZR7uU/8pxOvK/AYh+WwzpOAt1JQLoGN48SMFy0PxlbjulZi46x+6p0y15ic9e4/LzRf4t2OBJnudQtjZOWY66mWeJZSkbCfmpFsXqLcbVdFW0uwzuuxs8urdKLWUgGcFUR/IlWLfyU5que6v92dYmwT1FJ7qDJCHTBGe7BlEIOZlUqV+TKsCps40Nw0n3Bf2VuejlfM/005Z+IaRch+mBZo5jB1u1kFcIXuP7te43kcUueiDo0hrWCSbHCrT/Yn3JI+qBuQXYof9TIE1zOrSxhPD357CFbD5Gxgj+sfMrPKQkVpgCSRT5fS+ibRuSioWm5QDy9cqXLNHJaXhPWJ+QXjIJGxQOSA0k9RwOZtJtweYYc57TBIfTgMVV+RhfOaAPF/WYbzPKDA7ynjLSPpEtwcKc3NbheGIT0TYOaZLDm5+VURcR6GQ97J6AqHY5w XqqOyr4W k5GV7HBoZDJkom/yuWf269EYmQw89J4E0Brgy5BCXCOdQ3UUj8r3KP5euvfTSezM6KGDOzZ44tJIWfczCghzJz8zyPdfJZwFcO2E8G6poYsqSOJuXYf00w6HCai5zkMYAwVaeJvR7E8mTZAau+QZzll2CD1CuV3n5rKuo5JsDZMpeORVmha7rcmrVHPQs6qcMt++NKvEeIXg9bVoje/vlljekzzDFVNIZF8oh7Z2k6EAOjCoxXnpyVzdQhLhd8iIEvJxles1sDxmQ92A+pzD/Crkz+rizgYVpTYBFVgecmPgju6avTfLC3YhWTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API ======= 3 first results out of 10 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API ======= 3 first results out of 10 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] T-test on all 10 runs ===================== Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s = 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. The split of map(RO) and map(WO) into read_{begin/end}/write is suggested by Yosry Ahmed. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 03710d71d022..1288a4120855 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1377,6 +1377,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Fri Feb 21 22:25:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA6E3C021B3 for ; Fri, 21 Feb 2025 22:31:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 822C428000D; Fri, 21 Feb 2025 17:31:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D16B280004; Fri, 21 Feb 2025 17:31:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67EA328000D; Fri, 21 Feb 2025 17:31:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 48AD2280004 for ; Fri, 21 Feb 2025 17:31:24 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F2CDDA156B for ; Fri, 21 Feb 2025 22:31:23 +0000 (UTC) X-FDA: 83145399246.12.E01E0A3 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 1015A100004 for ; Fri, 21 Feb 2025 22:31:21 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="DsMS/qY1"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177082; a=rsa-sha256; cv=none; b=OgnSZ5RVzqXOYIgjb33nPDoXUGlVaojEFrpq0VUFPksJttCG0Q9SCbK5A6t8cRixbG67OC muKuTvCaTXgYlxxzH1gqX2hveSw9gxEihk+a3TEfUrjxLTGEXDHE/ifOg8Z+iURnSX7FwF 7ZuPaG7RUAVyZZBMnIbq7A7Q9WGi/NY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="DsMS/qY1"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+ywFL0qx8JBQIO9VI8pgvX03IYGXOc8sRHANyYI8RpE=; b=tPKpFXG6XCH6ITkei2Sr7WJqJO5EGqOhZDKbYiChgNMBR20+DeNP0SSlx8cSFudrBuRz2d m7hGLdwq6lq7jFfYqbry87ka2p46QfK5stCZbi1XEvWewFGqc4EHKRG7LXPS2UkuOhjZSt F0xLO12CM6IX6UAiMRmwTMx+VUxvxtQ= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-22114b800f7so51900735ad.2 for ; Fri, 21 Feb 2025 14:31:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177081; x=1740781881; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+ywFL0qx8JBQIO9VI8pgvX03IYGXOc8sRHANyYI8RpE=; b=DsMS/qY1jI34ipVw8vqg/0qt30WJMKI4LR8Y1nB8j7tVUU9m0s6bCF4dxMTF4w6/Kb YHliYyFGLkmyD6gej+sdhB0myV7T1I8H/8DciwkL5mN1hbsAveoQE2sHvmpD71rVkjZX FCXEUo4F+1+EkOEgyJRzmbAaciof+G4+O6ONY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177081; x=1740781881; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+ywFL0qx8JBQIO9VI8pgvX03IYGXOc8sRHANyYI8RpE=; b=pUlo3GEKQa40TzjUb42QKZq1mEr6EoIkEQGXW8895buniSEX9r2LqW114maD3Q/pU8 2lmVehn0vRsaLXcQQ2CBIo59nPWrhglCIb/kS4Xchygssqi4mZME7lEvIqIQkT7MKw/T +wnrbW/HXYrJYLuFTUzUZQSqqKiw6CwFTkI/GGSY3gtuC5b2ZgfPaRYYdrX4VvXFixED h4C01+9RP+9bM1ERxUu1L019kdMyMGagdlLbr7ILeprhUllJpxa/d3Ovmpv0nrmR6SmM 9e36+7MiUYmHYZqWF4/rQ1uhF8kexJrYbmYhpONe7ewgpL3pnhACQQ2hphjs4GQHXcu/ 8HEw== X-Forwarded-Encrypted: i=1; AJvYcCVUU4MDORIk0obIzouDUk3V+i+/L0AwOHPWswROcWwZOCa5l5Kj0p4XZ5MSiD++iuj5i7VsXdNWTA==@kvack.org X-Gm-Message-State: AOJu0YxCCNT76oMrCdS0EwWbb8MVEoHIw5T90zVzLK5VySqAjUBZ2dcR fw1njKt4Qw3mqAiZYhUf5J6AqX6wDfUmj+stTtDhj4UF1cfxge+QhtMISlSxqA== X-Gm-Gg: ASbGncss/HOMTlyo3RgI3iRc86CHgB7OynuZqTiIG6hTD9gRfLYO+s1TU7DQ5lSZ8dF e+1I4Zf4+ItUYjE2hsA3xiUNBDniuXlWvteIOsfCWPak7kzp09Icu5XV7VcrJPrOsn+hW7SuD/R SqO8G60nbwfz3RLN8r1Ir95UbIZ0aQUtdINSx29QQfrtJZkJI0V0+Lrc85Id3pc+fYIDN1oeCnO J9Rycy5LDasq6eMXVr4qvnCQcDLC2BOHn/3bXRMuEzbQ/12VWxarv2VaMTMfUDfPz+Y39SCV++g iZMxFAYkaDKD07oAwM6MoMjl2bI= X-Google-Smtp-Source: AGHT+IHKsKKftwTwJ0CiusP95+2mkUFfAjHaiXoSZhqOL+gmM2fD2wdcoOjiL2xiqFP6/U7Mmpfp4g== X-Received: by 2002:a17:902:cf10:b0:21a:8300:b9d5 with SMTP id d9443c01a7336-2219ff5f65emr84680315ad.23.1740177080968; Fri, 21 Feb 2025 14:31:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d55963d7sm143272675ad.257.2025.02.21.14.31.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 13/17] zram: switch to new zsmalloc object mapping API Date: Sat, 22 Feb 2025 07:25:44 +0900 Message-ID: <20250221222958.2225035-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 1015A100004 X-Rspamd-Server: rspam12 X-Stat-Signature: 311kdtjsba41f3d9fbp64enhu1bmi56b X-HE-Tag: 1740177081-532422 X-HE-Meta: U2FsdGVkX19iGnQL9UD2kW7aVM2cIqAVdWpDBsFMIbWEmEWhy9R/7ot75sYSeTbMUrNANFRkjdi0ebKtqr/iPLVddP5cnCNZ2fWftL/vOUVtc+DBVpQoSq87rl+4pe3zMO7j4fg6LnG6+wUHoNaaHGUJPMkyzPgBOXYkODyMrwyh5Hkv2FKCCC1Ur4wv7/aIPkI9BagrwIvnNYEjiWadEftuPbhoxseB/zXhtycWRKIMmjgaqrI0TQW7JI0eKnsnjeyP29Ajrk1y+V2g7QNVfirIzIQ7kAmstv/5J0Ni//ZFx2MCiABCfs0YOR2nL7hA8MzDzgQE9plnjPQuJdQrJc3yPYnwBamtvvfHqw2GKBfPPCzJCkFGGFGSVkRCvazmQpH6N3/xl4SEb2GrE4KICK+wg6m564DV5o0wRSwtnq7k1f/pNL2vsTzGIPxCjUFcuo1qTDR1aF6WyfARdOLy5nmdj2ENqt74LFuPMz4Xy0gvb6cnIIPZyJb5scAVfsmlNth3gZTq2RYODaNApdYlTdhkETsh6Qiz4REJeglOeTlNelpGhun+es4hDlMTIHXjtEZH/EraR4/4Hx7sKCZ+z0MS94xjAIUr3T0/DtMti6jRC6B1uo/1XbkAlQKc62lG9ZYtsw9xUkopYohF7IOKs9xngvPTMrLrnwGVBn2tJOiP2S0wj8xc14gmYuGOuaIZzY1hcjwrBQDo8q9pNmkQJnHcesnbbY8oIfmxTGiwxp8W51pM+kEzpIamKwpOBI9xTH7rYe94+Tx68QvS7sgB7nAl5rXDbj7QKAoPy9Y3tp/t9b044z4hMeIqd9nv89D4Hh7+/C7lPaDjeygvTp+ghr1pOMNpKIY9o110AKQSsH8fSK8cHrW77eP5LlfxKZjcV/HKNnrXFbFbn9N+Fx+8+Kp5d9W0UkmSshuSaQKhMa0VDzQe6smOkb4dd5haH/r2iQFVkWv4S0amZnKq2eA zXSQU1h7 hR+mAMkdC7yoliP1mYMG7w2pWkE5A0yv+je9Y7OIWDcHF4Vr/DMqE6sR+mxtBTAlb6JyMjwTSm3ex/g91EC6cvDPwwzTIUoo36Y/b5nzke98Q72A68U+UbN0iBFqPWwdSRXoUVr6pSULVw0UEtZoQjgNBFNowL2Nh69SZtivE+dc5z+xeM81ioj71npf+ZZMCAZ4wNGvXFvNNBVXNFXT0VXEC20pCEng2at4yenPaXffTSSs0CdP7UcuUgvV6eq3l8v8yQj/L/UWbCW07b+Tw1RSO9+tDFPAHb0ODtwBVvP7RYxjscvNn5jTKnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index cfdde2e0748a..a1d627054bb1 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer = NULL; } @@ -57,12 +58,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) if (ret) return ret; + zstrm->local_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b96be8576cbc..1ce981ce6f48 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1566,11 +1566,11 @@ static int read_incompressible_page(struct zram *zram, struct page *page, void *src, *dst; handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, NULL); dst = kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); return 0; } @@ -1588,11 +1588,11 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst = kmap_local_page(page); ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); return ret; @@ -1688,7 +1688,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; /* * This function is called from preemptible context so we don't need @@ -1705,11 +1705,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1730,7 +1728,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) int ret = 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1776,11 +1774,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1892,7 +1887,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret = 0; handle_old = zram_get_handle(zram, index); @@ -2002,12 +1997,9 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return PTR_ERR((void *)handle_new); } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); From patchwork Fri Feb 21 22:25:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFA90C021B6 for ; Fri, 21 Feb 2025 22:31:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4794F28000E; Fri, 21 Feb 2025 17:31:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 42946280004; Fri, 21 Feb 2025 17:31:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CA7328000E; Fri, 21 Feb 2025 17:31:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0B5E6280004 for ; Fri, 21 Feb 2025 17:31:30 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C3A184C308 for ; Fri, 21 Feb 2025 22:31:29 +0000 (UTC) X-FDA: 83145399498.24.812E538 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf27.hostedemail.com (Postfix) with ESMTP id E858E40008 for ; Fri, 21 Feb 2025 22:31:27 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GvG82uFb; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.49 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=YWvaSuUqmW8gwuLrYBh8Dn4Mrl0H3pep+NMTskDq+O8/EnF8l9C3qOLGq8EZ8kpEhmD3MT T8CNx5mBAUMqqlsTr/qnJRwmm4ZaN+6dcxY9jbIPO4svpy9kBDZcTfS2TlqjTJEPvNQGnR sRfBeuigv062EGR17+CUupvioGCMKw4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GvG82uFb; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.49 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177088; a=rsa-sha256; cv=none; b=zO698YdtRKVnqNErkMRadXwSxvVluZmYiwsIoaHUKtthduZ7hzjybASqQAE0JFdEwqyDOZ uei3vXzo1yI0WOlbhZ0qRnh7bzKZ9Zqz9G1o9USgkEwqFQPSXP5rbCuXuRS4TVR+gulhKO 4oYGZLR5MLoEvNVDRZanLXYgd4h+ZP4= Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2fa48404207so5402752a91.1 for ; Fri, 21 Feb 2025 14:31:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177087; x=1740781887; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=GvG82uFbqo6xJJkieZq19K7v1f4lbXR6f2FQgm2ZEDZlzEQGxZRnZtqZgmIrCWrwJu iDbdfpquBaqEfE5D5DSGhbH+WMNlHoCNMaDwVk2WY9SOVIiR5QNJ/9IHwFfOIRoxpzAK mw5YbA2yzV0cAnLZYBB/GcD7vSzsMXB7udIXo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177087; x=1740781887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=RMKDe7HqqAqn/m8T8AWQ9nfjDG06WRtBiItZssnIFfM1s80m8llOHJARyiBzCZ9LBd AcadDA910EibBKYg9ioLyPWwR/UaBnPtCRFsIKN3qOGOi0FBUt8RzBDkIYtkNF9w0qqR FIpolAzN2FltGP8wAaLYVh3ZrtzRztPW9OhhPwm9UC60XQTpMUptxDY1+5yaBJ8O4X9F BIv5+Kr1EEFNpPwIJUgJ5ksZTbYUjykXo/DUaX7l4hcr5op+zUCnNu1wlY0AqgDizQ7e CVjR29gOWEN7hpg+YZdzkjLdI72TJlmAI9QntqvFjeKqS6lY3lVg18Pwgu1FN+r74l86 i+1g== X-Forwarded-Encrypted: i=1; AJvYcCWxMBaRhXZ4zzYQV+VZ+hyi5eGdW/HVsefQq0tTr/HCFtmeidnMjvj6jSyDSl2E24dz4k0VKzqUrA==@kvack.org X-Gm-Message-State: AOJu0YxqexvbzJJpozEtnOAI32xHVXInwqBoKUyKMBVYyMOrMYdM0ytn wi8AE7KfK4Zdjl8PX81vfwtURc0a+RjJaVfBUwVpThIldBTYVIJcn/VPISBW3Q== X-Gm-Gg: ASbGncs3iiy3jYHL7193+G/5N3gmN5zcjsE4f9Aif+tb/eb5q+tSCKSz01v1UDYWhl2 KASxFtZ1AF+9YDTlk0Mil5YHxo8uzaA2rOWuKH0RvEYFznI7uzbzvth0pFesGbDXtqJ5Jlj4O/X EVqOeQhuWgrwaDZLY2iGFdEXtzAWrxPI9iTZSFFDk8rXL/dEfbhEU6GNg6yJmxkMGfQqBWZ5dUo r6AME9ryTE6AkutgP2Q7y6G6zoutgo7I3UJGY1s0GzyHxQd+6SXVlBvBBH2L4F9FGyuGyinpBsI 5vdDwcIR5dlPGB8VRc48u8R48LA= X-Google-Smtp-Source: AGHT+IGkPWWH6TqJ9/sgUjEBD1TJB2UYf0Tm0PHjDPg6aqJ3PVT1Kgk4bf+kZD6+22IEnkZqjfOIWw== X-Received: by 2002:a17:90b:2792:b0:2f8:49ad:4079 with SMTP id 98e67ed59e1d1-2fce779bc14mr7402899a91.6.1740177086823; Fri, 21 Feb 2025 14:31:26 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-addee79a984sm11481158a12.32.2025.02.21.14.31.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 14/17] zram: permit reclaim in zstd custom allocator Date: Sat, 22 Feb 2025 07:25:45 +0900 Message-ID: <20250221222958.2225035-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E858E40008 X-Stat-Signature: fn17scnk8cyz19ykse1shcmzwkpkmk68 X-Rspam-User: X-HE-Tag: 1740177087-255269 X-HE-Meta: U2FsdGVkX18Xe3qEL8gVZkMGFpP70dUd2yCMAQvcXe2FAnF9kXojOrHEwQaeXlCybnsE3Uq2iE5Gacy5cgLpGENtp/hg8dUqK119hKjSAkCPecH6NCbCCq2qGypBx4fVjAm7lY0e8x+t7zavrbfxUjHm+8TYCoEa9JBKIDDMblz3HZxCCrk1YjtsFaJJ4yNKPO6pHL2J9krdOL6d7tY37udGaiwomtpvdVhn5tQsJDasJgKru/Gg5Q/UDd1IWrsGPr6Cn+set6jGHiAtWynQUb4/Qf0xpGgnF+Ak1nbwhQobiU+EtFpUQeHuU05pmx/IwBrtD4m/tc51ZY2Eb8eWs45c+QTxIWrUPQlidDB9r2oZZYJodUuRSNm+zVrhIkkHaUVbgypzwdvRVINYzcnVli6eL0TxotdfsP7DSfOGHubwgstZJCqzIi3f73H6/mH2mUdPOtnMyV3/u2vuRm9XHfwxAsD90Pw2WABR+aCXjXTz2xq6YaO6tUqK1O5qUmKFjOhdtQ8nIEuN1y3PYfKvYE1keIi7K6hnnVuC4svcOeI5PL+PrBDtO+ryqa04acvnEJYyRSAZRiLbm7+NONvuA0HLgWyxAAI75xRX72sN0OJm3IWBNoIep+o1F+7y6qQNuMlJZr+j5uOKCE9rZ8X9jphtgFQ+qkLf10Z5j+00xDu4lTJEceTgLxNVvNJTDUs2jU3DjUbHB7Byu1DwBrlnBAklEp7Hit9e15C8eFywX4LPYZiS68QQZVoXplSPvmwjOaFS7W/1sSYRXtV0cazzmXi5w66WD5BNo/M+tVNWwYi290Rf5xBY/V+BUB40vNcClR+WxDhA+SUz94Kj75YAxaDXo4JwVotapzv4j+Tqu2GPIkE0y2zmeGb8I1Dzi7G8JF3qoibk20xliKBM//dd7KhrZJRWm5XZqU0JZHT9nF6W+/spSnRHWN5fEIND1OU5lW402st9nkMKkbxaS8f PppksxK4 DD6Hkn+42m55e11MX72w/3Kk+dL3+3vdRI1UfK9lRBOwLTZrtFMyPmjh925TlUW1AljWccTPeGZLdNMgMhkh8wAKoym+Qesn6ys73fhyJtJlRDNRR+qvnwmejSEYh10eICzvKXMYGWCyZlmeqjD8CUgmxeTXS30HrJxfuzJ34pb8ee8hFh7jVl7O+RUlzYQ9ZBSqtkXQQfexHuOHO1PlBG/LCOh/IYE1pPoYeywsXwVesMqf1zwnx8uR8dCCdsHETD7o6NCA4ufvL9pDfF1POZmIpcMDpO8yAossiW1ys/KjKMxUYc9ksxCL52g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000021, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compress() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } static void zstd_custom_free(void *opaque, void *address) From patchwork Fri Feb 21 22:25:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B511C021B3 for ; Fri, 21 Feb 2025 22:31:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A938C280010; Fri, 21 Feb 2025 17:31:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A43C728000F; Fri, 21 Feb 2025 17:31:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E3CE280010; Fri, 21 Feb 2025 17:31:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6F43128000F for ; Fri, 21 Feb 2025 17:31:36 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 22781160857 for ; Fri, 21 Feb 2025 22:31:36 +0000 (UTC) X-FDA: 83145399792.29.242D28A Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf23.hostedemail.com (Postfix) with ESMTP id 3D1A1140019 for ; Fri, 21 Feb 2025 22:31:34 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZLpeO0mu; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bqs0vWPwH5ieqcucXqEdSYkl3M4q+06+7pDXyJI2dK4=; b=OU7VATt0YeMkOawnOaVxma49JPLz2oshQNvKub+SP8f7u5vJKs1xSdE1vDg9+x+jqaDQOA ytfmwfT2dpteLe5iMcKvevbV8iyvdcxNPEuOrdw4pUDFrw6mlxCBSktkBF0MPgn0dOtBfo C1jlUhS2mEfGGyKMvk3hyl+cJT8OE+Q= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZLpeO0mu; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177094; a=rsa-sha256; cv=none; b=5/LEwSgdlkq4Cq00k8KUuVyjTjnzHoXn/lTXyOrE30ujoUkFbmXIHtdtNVz8/WtZ43neju oYA4KTjw5xb0hT78TJkKAWM51O02JyhbhT650BDu3LdUwCdmzbHZJIMyUC3fhvMkoVZl6M NG+xgQ/M9yByo4BZjm+FfqRpjuH3/3M= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-22104c4de96so47052555ad.3 for ; Fri, 21 Feb 2025 14:31:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177093; x=1740781893; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bqs0vWPwH5ieqcucXqEdSYkl3M4q+06+7pDXyJI2dK4=; b=ZLpeO0mudrLqt4qXHZKA0eytvRSO6bZVLjw6BGhHg7kviePsqJElrgpuKX/1lWfXri naeOnTRrQrHvJ+RVWz1CRuq0dlOPJMi+f0qwIzW/UUkLuiNH0KNmamNxucx50XEqTNa+ CA1zZ2iTRL81rz+1p4Q1uhQRJOfHvpI6wfP4U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177093; x=1740781893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bqs0vWPwH5ieqcucXqEdSYkl3M4q+06+7pDXyJI2dK4=; b=F+jVh2pLEsC3aMCc4nvcTl6GzuvuROTy1+J/n5fvi4ObcNvWdtTkxa0u6j2rQiNwBZ YKR8oPHTrpWiSYlyeyLb33YDmyIGsF5bweJ2KDQcmHcaAj1Br9sgZGXM8sxs9MrCUhsG qKiTc0QuhbO2WMpBLnuwB9PXEVpcbDHyV1oG8cQ3yJFEyEvVRnWFHVskjb9rsDm4Xlym q4Yv223eQmT46ASeOLLaDB6ZGcP9OcgK0H3cG5NmAKqPOdVwqFXJENAjObmmnPFG8alO bW2B3RW7GNSzZhAaNEg3g5awA88yyS9NVOHuIsEDRPj86FbfCDaYzuv/n4lntB6AzPxV wMkg== X-Forwarded-Encrypted: i=1; AJvYcCUsyDNoZgQO82yJaD/MypV9Es6REriZHooUYoe2tS9nMFrgBYdtJjmj0ZlBdbSFXjKWFF6O5ftsSw==@kvack.org X-Gm-Message-State: AOJu0YwltpKwqAFrqo0X7dPTyBH1C2KmN2X2NQQvaaHPbcgNfdNLU8Bg zCj4lzeVoRZUqjQATpqUGgnkmY69UgoAnAAFn72yfbFa3dcowyCY9fuKCIfJkg== X-Gm-Gg: ASbGncvOjX++t5c3rTqPP51M3hg6LF4BPGmNkqKpp0RwgoOdKK0K6VFrssCC75poBmd nS7dD+hy9DGwVdoH3tB6l6jULm14dautg01zABx7nWRAnrzrU2NBu5daPWktiyh/9t+kJFMzIKC HU3ehX8GHf/Ef9kJuhgLZhxwfR6sdCOBwqHSy6irqzSAu/qClGT7kKeUzR5h7ObXrGMEcrI/8oD x/gJWJ/+y5FhKh66vBTcuWQrLkgXR08pPBpceglKOWEa7ZLPbhS3EnmTqyrlj2MmPAKbB2i6m4G BWW2mnqKfBTqbZI+3XNOxhBvff0= X-Google-Smtp-Source: AGHT+IEILYc33c4jVA0dGK0BjQpLec5ckC66yqnrCIt209YHXfUD6zSxKU4JtkKRoduvqU5N8dK06Q== X-Received: by 2002:a05:6a20:12c6:b0:1ee:e7d0:5c54 with SMTP id adf61e73a8af0-1eef3dd815dmr9388338637.37.1740177093080; Fri, 21 Feb 2025 14:31:33 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7325f063782sm13714610b3a.148.2025.02.21.14.31.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 15/17] zram: do not leak page on recompress_store error path Date: Sat, 22 Feb 2025 07:25:46 +0900 Message-ID: <20250221222958.2225035-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3D1A1140019 X-Stat-Signature: 8opjt8x868rihsama6nhyp7c5apbishg X-HE-Tag: 1740177094-27697 X-HE-Meta: U2FsdGVkX18MCMcd1mdugAD4cvhK0sFQLRD4bUfgDJnMCxRKY7YcpmlU997RRV1Py2nZ11x4UMqeMQ0w/KlOZZeGfH+fI7p/CLs+BlUpEhFq+vcTeOfYCGYbgHTM5s+7qAmYym207804F9lGUkU1L31H/yGnBRvDiy1pesUAxTVtzGIcNdqe4ekFCkko2t4rZUEz3kTIi8EA2VohhYZ5R8PjTTqIAYdeuiMTkRwR2BwTRasdT9/o7Ql4fBmeWSupaF8IK7gOQUjysdt4jDTWUTvH1hdKM90UHcz4DOZZwj06Pv/lQoY0gSH16Ub0v6TyeS3yE+sxoRIReemhylSKTFLt6631pI8HDD+HiX1EOPgN3sQB14PGtJitARuUsJPWGQlbSTD7VW4zr67iuq7DXBWZx4ERttSNc3ni3LRprrqkA1W9SJAKLBm9tbRIMOau+Eb/MA6fsX9BGzjtFOSr3vdF1yimRmx8ushNmwDdmDxy86e1VAKCV8cwWqRRamceVlo878sBD9vwmSSgW6G3BUi/npINLgKeRO+iJUrePTR4/LtW1EW4LRqwtsf7xA+S31FoxSBEBtx5jqFwLARh+QemDKKtfpwkHDaZNq2qEgr0fXt4wCzm48aaZjYi/2y/Ldc/a1K+9jB5ki1hRe1TLvoJMBLT7mxEsaxLYz74i/19BeMt99RF9EIwXwQ6pZyGB9vM9yTFcOKQtZkjzml9Sn6EzVZOgRZQpJYoDqwstBtkMlVvzQ905miYmUL8sM2O/0qhBRPbrsTztJLdY2y3PxptDDVt6c9SA4YtVb5q74QFhWvfjRoTwqeWRE1FqsXkeuq49kWAoxvBhF1cyuScuVDwhV9s6xCfxEgvLM2vcEQLuDLRoaODko4x2/WxA46WCQNcow9BHhkAASFm55qUS5dhgke/de2MuLA6FsE0ysPE4BO5NS0qVHNjd53LQ62VPtnvs83UX0n/BtS2lHm dL3xFQ+Q rbDLMGiGi0sUn/93cHZIP6tWQw50I7RUGSVbOVZ5ARpzNIs9mz9SF0BfxKvvsSSnYs0vYR0RQDLOre6dCnM6yigpICfPjmjKY6R0MRmmfWxYnfLexM97+tsWh/UFs/8ubJV/2B3F1E5yWyFpb3/uZ1pne5hknRvEZ9TzeSC/T2s63n6s4i/DPUEhR1sazvjmkfEl3HGhKFAEFkVLW022evV9vlvT7PmWt3vtqlymLekgd0UHLj7rpHEwCMoYY2/CdYrQUkhaLCFNdg/w79Gr+1+UrP1uck5Fwj02mypJwVO321XVwj6wZlff0iiZs28OEh8KaVvSWnse4WDbTCnBPu05iHhMdjNkEprL1EwGgU1zmAVI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000146, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1ce981ce6f48..1da329cae8ce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2022,7 +2022,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; u32 prio, prio_max; - struct page *page; + struct page *page = NULL; ssize_t ret; prio = ZRAM_SECONDARY_COMP; @@ -2166,9 +2166,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Fri Feb 21 22:25:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AF01C021B5 for ; Fri, 21 Feb 2025 22:31:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00C13280011; Fri, 21 Feb 2025 17:31:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFEA928000F; Fri, 21 Feb 2025 17:31:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7777280011; Fri, 21 Feb 2025 17:31:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B7B2F28000F for ; Fri, 21 Feb 2025 17:31:41 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 749971A0873 for ; Fri, 21 Feb 2025 22:31:41 +0000 (UTC) X-FDA: 83145400002.14.047D59F Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf04.hostedemail.com (Postfix) with ESMTP id 8B24340004 for ; Fri, 21 Feb 2025 22:31:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZxLw7oJf; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177099; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JR4o+z78q2mq61XCgJA6k+y16NepOIdls2rvyffegGo=; b=mQQmwEJSUkQTorICtcx8g2WG7GJVYRY+wEveGmfMtMHXO9cZUxmfkO780IRJIB+xpoifNV pZ7a7L75fObT59arKC7G22Lo3GnxOk5jezlxOG+baLYQhl8qbxmksWgihGyw60Ll3fkKPr Kuud/LkvsuTAwwGrB29puCpce8RDChk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZxLw7oJf; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177099; a=rsa-sha256; cv=none; b=de2zNAIQ4+9S4HTUS2wrDwPjyXMlX+CVpxVcHk/Ra/CsUX1WXz+vDaKVbrS7GMXa9OOCOi Cll3B3bPsUo9xHMmJ/bwNMFkfh93BATyJMxGlw5E4ahQeYCjp5k2Iix3DWYqyM8gn5Y4eS 8O4hXawHkREBGh5bbrsm5MUVxa+Yprc= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-22104c4de96so47053475ad.3 for ; Fri, 21 Feb 2025 14:31:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177098; x=1740781898; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JR4o+z78q2mq61XCgJA6k+y16NepOIdls2rvyffegGo=; b=ZxLw7oJfArESg1mr6eD31DzRxvU4R7WlLVSlcgXtVj5AEzl4Vn9BUWVazEiUhFE792 9itjEBndW2G1QMhS0IV8JRzC7lBx7e03jwABIArkOGervIm/YQmdDkY4gErXT1URfRTa zvfENsQACXdTyBwcFbKyh5v0cdx+DyjlhTEtM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177098; x=1740781898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JR4o+z78q2mq61XCgJA6k+y16NepOIdls2rvyffegGo=; b=KMy1kuujQSMzPIQEVkkduZ/JoRFOxTtqgCXwj4zR6FKTcZmmXdTyh+FFhThtYESQcO Dk+TcDUi/+FkOQhUn/Juna6HaKz3TxnjW7rsdrOVjeOqnGSXwfR6Uf/3n/hiRriCxFo9 u0hQ1FRMRmSgjzCi/3gcTRbp+6A1g6JMJA/in5+iJTJSsTWzB0cLvq06Q/kmcuqdAV3E OnVgpd2EN6O0RaguQTxOx0YTv5qzlg6kwyEtxgyPK9ZjEWarWlKXhNlGdO3JEbfHtfP0 U3AmFk/BMNPaHalfeLeX4fIhjgfII4vqqZMRyaGe32lX88u8qtZbSie4QxirswMf+BFX 0BsA== X-Forwarded-Encrypted: i=1; AJvYcCV/De2L6t67TtTSoCPfLtNUfCllKvy+sTj7h/BcDXU16lOAJYO52N7bV885B2NfsT0UZgh9p7juhQ==@kvack.org X-Gm-Message-State: AOJu0YwBFmebl0/tacVndI03rpzr5xXygB8i9v+Ec77zywx9kWE8L+IA m0gBchPM+YcPo/qgRypiF7c4lnNGArqAPszS/XAUP8MBgk+udSgehJY/0U7zRQ== X-Gm-Gg: ASbGncvZaYyK3P+4LXNrRsgbv4FgtjclBbAaZ58jbqkaRBh621XJNKJ/hkLocoh1xwg 9XZ4FTC+ngBWbpO/voy4yRKKpnGl1SjQuU/RwuSkdxgmqjTP0tELFKbnhCWE8A5GVQO2XUr2669 ljkopf0yslm9tRo0naawG1VyYT2S2VxrLdenbuGRX7N+88vxl3H451ltaw+qIJFGqZmBq83p/bX tR7VkjgOqzQFfUWbqHQ8SG4ndMTi+D5oKGfiHU8QImzfFwk2NLi+uq04y5FLIANvCJA/noldf3m pC0g0ZZNJNNpJV2V4wGMrxv/bMA= X-Google-Smtp-Source: AGHT+IGz76l7D3zW41L+K8j03WiqF3YSL7loobTRLV3etxTMk2gUtAOA9cHRBz1EbaHcXZjvwLcRjA== X-Received: by 2002:a05:6a00:a01:b0:72f:d7ce:5003 with SMTP id d2e1a72fcca58-73426d9059bmr6835280b3a.22.1740177098531; Fri, 21 Feb 2025 14:31:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242546867sm16266956b3a.24.2025.02.21.14.31.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 16/17] zram: do not leak page on writeback_store error path Date: Sat, 22 Feb 2025 07:25:47 +0900 Message-ID: <20250221222958.2225035-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8B24340004 X-Stat-Signature: mzziuteeb5oio6zukhp9oqxo9pd455xe X-HE-Tag: 1740177099-458359 X-HE-Meta: U2FsdGVkX1/+ei8Obb53tbfbImWmrYNOeYHCE06SLamRRjDjFhOc8KAxKKXLvWGSYQE5TVS0UQ1bSQkcyb1HQ4kKqI97SUeBm14nTf2EMWNxnbhXVQdm5wfe/b0wW5nqgHsc3tAGryLk35K+I+i/spB4EyBB7oJHJy5YHOS1P+yAFwBTzMpGRSVl8EfnnY9LaXeALYl7VRyC/GD0r4OFEnzPcru4d6Ji4BiIat2RUuaogP3sQlBjoUq6JItoAmCTa2/OYh/LFRrLKdYOsZnEIiLihpInGrwKtteyNfkw5Dr6rK3GeKlXPvUyaQcEahyPvyxSE5CITltlZL3m3mFFwJ0yHNt4rbpGvG1CDo30K8Enh/bGVVBTN6RvcvTpLLcUw6DrV3BpUx6io9LES+srVNS+H7yVKdRrtIen0S+aXDCdni0YAodPTRBdQQM0Mang0ot57VpOOqKdIjZIcYJclC+udp8Xf8tzzU+QxW2pipjZnU9ni6FFdzYJKV55Wx4x3QJRhyprlk7Z11Hy0Vz0mrmNTflxmAsd1qSoXHCGC3WVpq69bp3lDqerqmO1WxWLiZdzYTXcP2BLTLusIjxM/zvictz8xMidt36uXLXjArwjIYJouGul2CmpauVtAcQ4E65u6Mf/OVwL9ig/TMQNtXAd9dfzaw6QajduUPcSqSZ8ANQi30dHWemuO44zJnO3wPwZccKxUPr3oNEuofSsZqcmGLe1M9vYb75yYEU5PB6UZxVGgLqfJaVff9buOVSomPzX5TaKUj+R7iZCwp9CgDvaImEI7zQpEjevO2nu4Lo2WyfrrC603BmirbHxT0cM8/XjsK2wJZlCwIEQrEw7lGtQ9Wd7LMWQHnt5zPIg3ckqTD5grEiAuoFWOEtgNkvWFnbQPtTQ8pQxKOhNKHhwqc5xBXhg23/jKrEdYkDNTJFnOng4pJQyi709Xc9x8Cyuv9ixmz5YJtwq3H/aiSf 7GQxvRyi UkfKMaefJy8khx2mbGpLUV+GIFpx7bLraMIvnBwsh78I9E1ycsIpp7F3kj+0Ykm8m3ZdJVGb7rZzfndj2bkt/80+uS1tHztVJ1bYdhRCKrW5WQffqZJLVdmKyJCiVBkLvqAGAxtO2vV3jgi8qCzHRnZfcp26hp4ysF9suBQ2KK/miPptN1SMcZQd9JYfVx+Zepx7TJ9l+z5TjnLKnnwbTf3RMFridXJ7MD4pKSV/AyyfuYZDgE70qzcuwsrPg+yvOGstPIIQiKlibMUb38+GPUR8Xn3aT8J1wmSsG0M+WncMViRUqkSJ/uo8DBn1z1nB6XneLa9zFlhlY0bciqFSLNDXQb5RAB6HA9FbjqvGK3oTM+rM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000873, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1da329cae8ce..4e9381b153da 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -792,7 +792,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index = 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page = NULL; ssize_t ret = len; int mode, err; unsigned long blk_idx = 0; @@ -934,8 +934,10 @@ static ssize_t writeback_store(struct device *dev, if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Fri Feb 21 22:25:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13986394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B395CC021B5 for ; Fri, 21 Feb 2025 22:31:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4521B280012; Fri, 21 Feb 2025 17:31:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4018328000F; Fri, 21 Feb 2025 17:31:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27B41280012; Fri, 21 Feb 2025 17:31:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 088AC28000F for ; Fri, 21 Feb 2025 17:31:48 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B2577A160F for ; Fri, 21 Feb 2025 22:31:47 +0000 (UTC) X-FDA: 83145400254.08.3C7BA1A Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf15.hostedemail.com (Postfix) with ESMTP id D017FA0008 for ; Fri, 21 Feb 2025 22:31:45 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=KXnbaoLP; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740177105; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=OgR/hD5KC4NlPdY4ftZwceGiDEzOPcpMyujIWB4ldxJ0p40st+WfxCYKEl9pm3N6/ck+lN ZYMJGhbVJYMk8QxX+MZz6V6hX2t/jz3kkeQaLa1z7+rOW4/tV/tj5BKm+frYes2YDNhOXg 2kY17mykP4tJOdH/gliF690DK5sf2HE= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=KXnbaoLP; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740177105; a=rsa-sha256; cv=none; b=Aji5PjtBCIdt8PM6yffO0LSHMVdEO3nVsze+TutlCjVavWU6mJAHczoAx8Bf16PRg7jbJd H80Hm5fUEVrvYCTKzD3pcw8jsOufNmroy5cHCUt2Y9phXoKJqFd1PtHFzKcfhoWQ3+9db9 BaqoCXbtZCjF/oceODmUCFIMEiicWvg= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-220d132f16dso42847845ad.0 for ; Fri, 21 Feb 2025 14:31:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740177105; x=1740781905; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=KXnbaoLPC5OPc8zg9UAARrtcZzr2FSFE0Nnfhw6+OrR2X1jCVNqp96T+YLo6IpfCrA mnTzI4bGbLELKNK0bBinPhpwR/AqfSH9Hfqdr4awUGiMb7QVRu9Ff82jVJYEJ24Kthg2 yv7wPh7Ajh0jqvpgXW5tbuvT1CeLCgtj57GFc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740177105; x=1740781905; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=WO6DiCWjbKzTcp69uAAgq4IKmgimTpyo+MXL5qtgMiSP/L986+8jCBbtFkMxwynvNE Ofq6P6PuAYc1RPoeUPoBv/UpUJnVssY9IGU4dujNn87L89DRREQYzUte5A72oiOmWpNo reOVAdN7AhO/IKqHxUJ223ozGM9gJWwjqCrBF6h5pLaN0lE7EUf8DfUvO1588WYoBC0O AHMfuF2QZnOSr7jX9l9FZRxChTCVFljj8BEkfzeSVqJ2QmgntM0DystNbvm5WYZX2f38 xUhUPlGV1zGbt9DX8O3YmKWtyULYXt9klwCZovFMgH36RZ6jJb2xZvFhU3YuzDYydOA/ iSOA== X-Forwarded-Encrypted: i=1; AJvYcCVR5DZBx9+8h+FUAkNE2RfOxWx+i4SEU4CJqnX9Io0lUGLN21LVwcVoRq9b0+vDmAClGS7p3fxTAw==@kvack.org X-Gm-Message-State: AOJu0YzeXZIeCeX/fsd6dqYk4T+UHU23w7wc7GQp0vxZ+7fAFR38vxlD HviBsfrQQ+eLs4k3I7yzJ+ICDTU3sT2hSPZsNR1ZKyw/DAegmr2DEilgUJP++g== X-Gm-Gg: ASbGnctRGkqSms8x/X162FBGPasOxoYpR+Gp1rF2/e5oAlf3zigHOPxoASFm4UDkNUn E5NEE5S1EupQOl5M4dYAY6mEA14naDFstKh16zKCU33UuG0DPKOL58rjV+XIKSQR09mpYJKPNH2 bewiQOOAujdpx+MzpeDwoZITWPkuQWQgTDDnxhI+o0F0ekBm5HmCGy8keQJQOQMAn/1YXesLmsE eWr1Zt9eJoR1S3Oh5jpSL8VIn3ueS6gkehvc/36Ofiyj67+Aus/LZNCuv31/fVqdhoZedKuYh33 hBL3Nbcr29AVegAmy1krLzNqq70= X-Google-Smtp-Source: AGHT+IGOUS6jHqktZGGxZojaBlDO9HK4TxnHkyGnN8Mo5dF34H643X32/pKEEVPpk8gt15j52Rs5cA== X-Received: by 2002:a17:902:d507:b0:220:c2bf:e8c6 with SMTP id d9443c01a7336-221a000add9mr82113705ad.53.1740177104772; Fri, 21 Feb 2025 14:31:44 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7326abf096csm12468634b3a.170.2025.02.21.14.31.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 14:31:44 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v8 17/17] zram: add might_sleep to zcomp API Date: Sat, 22 Feb 2025 07:25:48 +0900 Message-ID: <20250221222958.2225035-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221222958.2225035-1-senozhatsky@chromium.org> References: <20250221222958.2225035-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: D017FA0008 X-Stat-Signature: 6u7ou891otrcqbsc9hkaf1szcfmh5xpy X-HE-Tag: 1740177105-665549 X-HE-Meta: U2FsdGVkX18LfDm/Hqfnm3zvScIa+/X3GTEzpVBLuHsvymjm9TXCA3nrEGBDltf3A4KLYIvyNtTC3DbnWxTSgyUkohYyIVP7jrJ3O9KKmhEYazpx63QlL1ufa/yhqu3ZcLfledjsaT8L1xY+ONt4TpedAzxuqGWUOo6ddXdPx8dxpDVyiHtf+mTR/jZp2eZH89qQqHugyOE/ixs4MJf0IoA2EEDIXgs535THUHFzHz2hJrLym9jdCRk02HSoASb1yyx45cjAPt0bYrNw5JwT0QLRxtV/ypwEieIEx/GmELH8aENM+VlW3mpWvg5wlaFknn3FC1FexVncXgq6f0gVMPFBNV4O1xKgWsB14ksmAb/n7BpAackueO49UFTM6i3+YJ4Be+JMWE3EE5Fa0zT5lxuCxukb7SLZWxpLU0/zliLLrwME3i5d6JsXw/09jOrE1gQVZHU7rPaJX/brWS7+tAXJsSRHRzSuMaT6reW9YEY/+e8GpAmUr8ziS170sJifK7FN75SWNMQ7UDEg2bwjcBgA1MhhjOHzoL1I4aZZ7A9ZIbDGfOoxG07944RuhdfORBVwAm4dcVmyHyfNXfoFlAakwBTU/w/ePe9Us7VKiXVQaVxSQMf0Vb+ciNgQDIt0VQsNvwfClPihBpeVGUurH9k4EMy2cSMpSijxmq8SS0Cv+hMzTqUS8TFlqBPiwpruFQXLtVPHwNXoemyCCpn7hD1fQBExb9kYmV7izvTsRTbn2yyyWWntiAOLDj+9vEtxIeSsrM0LYft6grZSvRjRf+kTIvbg0QAI/gxk0wdUdlqQNgO8OJDX/1aanDuVgB/tBhSWDQWz5EiK0tWFmWK9XfzLOXUNS8yNUmgACGCZLuPU/CCh4CqGn4da6gTtZAu9dzciL1x9BGaN/B6K/jjqisTir0qiXS2o8tOqZYof6wk+XkBD+TG9/zuOzy4M2kJnBdMBPhyPrBIgVLOZ0a0 HfvUZ2Vx sa4L5cLWcGwsko/NJCtJdA1FBCsuuhDFoENfIE5QoDlTcRhD7+bm3eEp6weFHeqJw1paV6ow25r1nlj/HlxJAPk+S3CDRaXpmPZ3n4a8rGxRYds7q3oUFH85tlTu7ByTlsEJvvPF81pz6MYT3OQzwuNpO4GouZJTFiK5uFOZXtWyTnETxYSKgroo1fTaQbWlrg3lL9U1rtEGy0DKcGVrp77NbKpHqOoJ4rm+lzmOa2D0IKUy7+UjsS7OA+aijXUNk5Fhy7ADKYgxr1fK+MFlVkCgtolhZl2k/BaaMKAvT0UGooOh/V05Vv1jVH8WzMRDurcWQZW1OTeicfPTs1Wgwt5pYXbtTdJC6Nvh8kfzsRK8Gbgw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000724, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index a1d627054bb1..d26a58c67e95 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -146,6 +146,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, }; int ret; + might_sleep(); ret = comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len = req.dst_len; @@ -162,6 +163,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); }