From patchwork Tue Jan 30 01:36:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13536683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2982C47DB3 for ; Tue, 30 Jan 2024 01:43:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B0366B00BC; Mon, 29 Jan 2024 20:42:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 738F76B00BD; Mon, 29 Jan 2024 20:42:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 589906B00BE; Mon, 29 Jan 2024 20:42:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 41BEF6B00BC for ; Mon, 29 Jan 2024 20:42:37 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1ECA41A0A02 for ; Tue, 30 Jan 2024 01:42:37 +0000 (UTC) X-FDA: 81734277954.29.D4D9181 Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf11.hostedemail.com (Postfix) with ESMTP id 57E674000E for ; Tue, 30 Jan 2024 01:42:35 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=sNnnWUVe; spf=pass (imf11.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706578955; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F1iB6scB+xehMKrtYira4F0YAgxikUXsHI43Fnlp9WY=; b=bDwiAIxlEueCjKb2tC6xBIZuwGk3VSRavEdtWsYmEd7FB0WwH6t1o/GfoJHYTAI/5p/EBo azU3m1beLKURV1Q3Yc5vFyWK2i+6ltLz6N2HtN6XWaJhnpe7+NEmUVCWN+RjWPr3Tw+kTh qrxlim4Fg/QFWxVHCLsPLY0SGApolFM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706578955; a=rsa-sha256; cv=none; b=KZ+gHTPZruOtot1hvmOKQnpx5pjfGrnTDOTRylTReFJ8boL6K9R8v2Fe5VE27QWZ+ISet6 dIynrE86Qa0eh5FdaQlKjGBEJY3wYNdEeCGrEZOrB/HqcSZXRHXCevgxX3Ge4WaTeC8GKv QG2607uy6VpVYOlvOYNZ67xCQOiO4Ek= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=sNnnWUVe; spf=pass (imf11.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-68c2f4c3282so26793026d6.3 for ; Mon, 29 Jan 2024 17:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1706578954; x=1707183754; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F1iB6scB+xehMKrtYira4F0YAgxikUXsHI43Fnlp9WY=; b=sNnnWUVesOLl0H8VFn6CKm9Ud3+iOKTF7XqGksktxJAt6F3VC8TCBmilMmoR5Jms5l zVgUVTBlD2RUFa28yfLIimfHUx7FSuwUIQo5blEYUg15gVfLCYgUgsDCowcNRFSSFzk4 JNzMs0XfyVd5wyKW2OpDjd0M+YMk0fe+DAkVVjK1VFeS3F0tmRrP+KUKnVTkS1+FQVgQ Y5BJ1kvlmgbrsnXDiZiRpBz4ZkRqy+FO7hgia71VKBqAAumstNu9Ot+eOCatERupZPOx mJhrqc5h+mZENoQ8gASWUGoiZOVCEx5yUIWfQifVyfvQ/KtoUVfFIhEqiPlr1SsPzVyf DqsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706578954; x=1707183754; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F1iB6scB+xehMKrtYira4F0YAgxikUXsHI43Fnlp9WY=; b=iwD9yPz/fcY1t/g37hw0l07H4LYN1wkcaEhYELvqzKBtdoo/mrlx0TKiJ06giiRcHb 2MoLdOddF2YNEmfKWTlz4SUlK8L9Em4QSk2TU3+tGTo+YMSpZfNmEBuImyJgThX7eb9n Vaw/Rj1onPcRDcuJCRpjF14DDAjnZrpT40NN9t65Ty6+pT3AEAp37Peu9OF8R3UVcdHV 6qAVYr4pNR7i7OMdu5H1QlHEVrbSRChxYLw3X3yMe+ETWSJnaYz14dJ8e323KHQhdpBN P8I9+lEcvqvziaWAcpBtmhGdBGUDkQ9gnlgbbjf6q6UnLOdo8KEp/zCdrRbJQSNeqx+h 9MLA== X-Gm-Message-State: AOJu0YxnYHfZLfcm3sUGzBmkw0lYYiUnwALcBZfsQkUCgr/syQGwE88s 7fsIGgBDacEfyNZfX1NTY5d9QPtlT10QceyF9v/oeBGmT3LHeqUvQVm5BLxg1ZQ= X-Google-Smtp-Source: AGHT+IGzesXTSc5km00kKTgaDrZtfj8iYTZdT/vrOqGJQTpSYvP8oFDnRJlKxNfj+JGUQtlQVzjEvg== X-Received: by 2002:a05:6214:628:b0:681:5800:9f23 with SMTP id a8-20020a056214062800b0068158009f23mr8224751qvx.115.1706578954521; Mon, 29 Jan 2024 17:42:34 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id mw2-20020a05621433c200b006891e1f098csm3996652qvb.78.2024.01.29.17.42.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 17:42:34 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Nhat Pham , Yosry Ahmed , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 17/20] mm: zswap: function ordering: compress & decompress functions Date: Mon, 29 Jan 2024 20:36:53 -0500 Message-ID: <20240130014208.565554-18-hannes@cmpxchg.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130014208.565554-1-hannes@cmpxchg.org> References: <20240130014208.565554-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Stat-Signature: i46pwy5xgc9jbu7izosx7qzfxmo6grew X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 57E674000E X-Rspam-User: X-HE-Tag: 1706578955-14923 X-HE-Meta: U2FsdGVkX1+f9chI60UkRTIr/Q0Tfb7rJR+wu3qAMcjU0BRFDAgrRJtUcRl3InJiTfqBL3yGf5rpNiuL9RAn02Rmhl7OMiKuginzN+4LRqOA70ePMpSZ3Nj22ugEtmnVI1CMe+lnxSMdjyIJcW+pQVu8kjoAwfvUL6oqqtqURTayYaQD99ezIRg3jUlr9kSy8Tgv1Bw3JyUWO2jEVFyKFE4Q8TxUyDNvDHyNMSGYYUCrTaBvBwsTZ0UTJWlkaJEVp3P24qdREw0dYAwhAq2EdQMOWeYcnH2TfaEAJxoiwjnWKhhga5l5Y1fQFbwn2r8giq4tI6ETf8jFha72TFENuLN2VQPpg6ghovCYXoQYHT8bccL2JcGbvufVENd3OF/gbyNxVCMbU+DDqzqrBPOuDl2hvJGjBV/4VMUtW4wXVBV892bFHv3FOs2KVa91aKluiILyJ105ExGls+YlYt1zOr5dPrXtFscPcERJNi//fniSV7Ry/hEZfxqX6Opsbd60VxhvAgjEamnZoDdxGdtCc+/P0sOqEz9cHAGKLS8HgkINkBJ07ApT/CMFhbrRdgIAlYovPf9+lqE6xL4mHp5XzWACptr7B+l8xRie450N+yMzgFOhp2++as+kBenY7EzarWEIniay7IKh1hr5le9DlNmDYcF3IbhOl7j0ehhu3ZvTwl8cbvIvkussg3bWu3OZK9TgrKz0mcqgHUjC978J6vEOVxHZ5SmjeQsM/PniuNDderP3WkolDeZD6oCFURzlXoMHMBVZB4p2drMcW+/k5z+4mWZr8IXaKah24qhBwrb9itEQz6bUrEOPRBxLZNq+Fduoh64ysbsgv83GvBQWKp9gi/ltLc9Vc+QLXdIhWGUNGazsEkN8DCyy5Si4Gn9WqNmgdR0WRYXYN/6e4aAJWha9lzIZp4NcLhfXt5pUcUp4uk73UzZC5p+fZ+L1Hf4+ePeC6YIeHjSq0sy/0+Q G6rYx5ot iBvim0DlRiKCOknKkJOn78wpZm9Xl8YL6e0uMr7t9Gz5sobY8ba6UBd03Rg4p4UR2bVTxvy6Gf4E6lG3plhfKfpptS/D9yLtvFiwPOZmiEXJz4D/Avh89zqs7/DwGlnAl5DlQmWLxsgli85ByWHUkv4XpJlz/6v0PFhdh12x+5p7wjferNa9w+wNwcG2pWADZ/Hdi9myfEpY9ExKOFzlyz0hdMP1UzEH3NJiTmyd/+OmauRAtAugIg0Ssu/0ncCnYAuTHQhPbaAT1qNL5BPn6rYXQ/QsDZQEEicnphOccyjdIGFRe61jyjRcqTCWWvDBIm5kUATXUxt2VYhP05gvrMG3jbSYtN1VdActDUK7RUjlq+ZGR2CSrUIaUbUcHS+RwxJcG+hxguB2CTr6Ca1bXLNitcz2rfP8Y85o1cLIxVp2whmvwCr1jTV5XGn2P8kiEIcnAHUa05i70hVetK2jXloiwaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Writeback needs to decompress. Move the (de)compression API above what will be the consolidated shrinking/writeback code. Signed-off-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 207 +++++++++++++++++++++++++++-------------------------- 1 file changed, 105 insertions(+), 102 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 80adc2f7d1a2..17356b2e35c2 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -992,6 +992,111 @@ static void zswap_invalidate_entry(struct zswap_tree *tree, zswap_entry_put(entry); } +/********************************* +* compressed storage functions +**********************************/ +static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) +{ + struct crypto_acomp_ctx *acomp_ctx; + struct scatterlist input, output; + unsigned int dlen = PAGE_SIZE; + unsigned long handle; + struct zpool *zpool; + char *buf; + gfp_t gfp; + int ret; + u8 *dst; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + + mutex_lock(&acomp_ctx->mutex); + + dst = acomp_ctx->buffer; + sg_init_table(&input, 1); + sg_set_page(&input, &folio->page, PAGE_SIZE, 0); + + /* + * We need PAGE_SIZE * 2 here since there maybe over-compression case, + * and hardware-accelerators may won't check the dst buffer size, so + * giving the dst buffer with enough length to avoid buffer overflow. + */ + sg_init_one(&output, dst, PAGE_SIZE * 2); + acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + + /* + * it maybe looks a little bit silly that we send an asynchronous request, + * then wait for its completion synchronously. This makes the process look + * synchronous in fact. + * Theoretically, acomp supports users send multiple acomp requests in one + * acomp instance, then get those requests done simultaneously. but in this + * case, zswap actually does store and load page by page, there is no + * existing method to send the second page before the first page is done + * in one thread doing zwap. + * but in different threads running on different cpu, we have different + * acomp instance, so multiple threads can do (de)compression in parallel. + */ + ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + if (ret) { + zswap_reject_compress_fail++; + goto unlock; + } + + zpool = zswap_find_zpool(entry); + gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; + if (zpool_malloc_support_movable(zpool)) + gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; + ret = zpool_malloc(zpool, dlen, gfp, &handle); + if (ret == -ENOSPC) { + zswap_reject_compress_poor++; + goto unlock; + } + if (ret) { + zswap_reject_alloc_fail++; + goto unlock; + } + + buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); + memcpy(buf, dst, dlen); + zpool_unmap_handle(zpool, handle); + + entry->handle = handle; + entry->length = dlen; + +unlock: + mutex_unlock(&acomp_ctx->mutex); + return ret == 0; +} + +static void zswap_decompress(struct zswap_entry *entry, struct page *page) +{ + struct zpool *zpool = zswap_find_zpool(entry); + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->buffer, src, entry->length); + src = acomp_ctx->buffer; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + mutex_unlock(&acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); +} + /********************************* * shrinker functions **********************************/ @@ -1317,108 +1422,6 @@ static void shrink_worker(struct work_struct *w) zswap_pool_put(pool); } -static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) -{ - struct crypto_acomp_ctx *acomp_ctx; - struct scatterlist input, output; - unsigned int dlen = PAGE_SIZE; - unsigned long handle; - struct zpool *zpool; - char *buf; - gfp_t gfp; - int ret; - u8 *dst; - - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - - mutex_lock(&acomp_ctx->mutex); - - dst = acomp_ctx->buffer; - sg_init_table(&input, 1); - sg_set_page(&input, &folio->page, PAGE_SIZE, 0); - - /* - * We need PAGE_SIZE * 2 here since there maybe over-compression case, - * and hardware-accelerators may won't check the dst buffer size, so - * giving the dst buffer with enough length to avoid buffer overflow. - */ - sg_init_one(&output, dst, PAGE_SIZE * 2); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); - - /* - * it maybe looks a little bit silly that we send an asynchronous request, - * then wait for its completion synchronously. This makes the process look - * synchronous in fact. - * Theoretically, acomp supports users send multiple acomp requests in one - * acomp instance, then get those requests done simultaneously. but in this - * case, zswap actually does store and load page by page, there is no - * existing method to send the second page before the first page is done - * in one thread doing zwap. - * but in different threads running on different cpu, we have different - * acomp instance, so multiple threads can do (de)compression in parallel. - */ - ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - if (ret) { - zswap_reject_compress_fail++; - goto unlock; - } - - zpool = zswap_find_zpool(entry); - gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; - if (zpool_malloc_support_movable(zpool)) - gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; - ret = zpool_malloc(zpool, dlen, gfp, &handle); - if (ret == -ENOSPC) { - zswap_reject_compress_poor++; - goto unlock; - } - if (ret) { - zswap_reject_alloc_fail++; - goto unlock; - } - - buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); - memcpy(buf, dst, dlen); - zpool_unmap_handle(zpool, handle); - - entry->handle = handle; - entry->length = dlen; - -unlock: - mutex_unlock(&acomp_ctx->mutex); - return ret == 0; -} - -static void zswap_decompress(struct zswap_entry *entry, struct page *page) -{ - struct zpool *zpool = zswap_find_zpool(entry); - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - u8 *src; - - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); - - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->buffer, src, entry->length); - src = acomp_ctx->buffer; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); - BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); - BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(&acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); -} - /********************************* * writeback code **********************************/