From patchwork Fri Oct 18 06:40:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchana P Sridhar X-Patchwork-Id: 13841238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EED21D3C550 for ; Fri, 18 Oct 2024 06:41:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F6CA6B009C; Fri, 18 Oct 2024 02:41:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6ADFA6B009D; Fri, 18 Oct 2024 02:41:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 501AB6B009E; Fri, 18 Oct 2024 02:41:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 27A566B009C for ; Fri, 18 Oct 2024 02:41:15 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 957CCAC799 for ; Fri, 18 Oct 2024 06:40:51 +0000 (UTC) X-FDA: 82685775900.04.998BD6A Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) by imf19.hostedemail.com (Postfix) with ESMTP id 6610D1A0017 for ; Fri, 18 Oct 2024 06:40:59 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GW+Sqmmg; spf=pass (imf19.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729233526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6oPOptHy1TMVMHa9IycIcMocFy/J8hGQChnYgpe95JU=; b=iH8OFAYu/pTSaA97A3o+Jmm8Sx08DtvbfWO7SsBNUODtVoVpeognuYmbJg+snOTwB576vs 9h5osZl1KA9vl4R1mpGjzFZD3BVkdqkqhOrw51VqQ9bhrcJdZyvn75TkEFerdPos+tc4Tm fDo3GJ7T3GS6pfdud4hfOnTgv1S0nfg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729233526; a=rsa-sha256; cv=none; b=mrjcO9u8xWiA5eJdTBqKWojS9rSU09np/sUJ8oQyc45zm+v411nyafRGejqZKjudgWzx16 NQnxh7lyZocSsd93+IUAKtRPk4ssmsw8hLaKg10463FEHYiSU2/8wpbU78OPe8ieH03d61 ezD4id3VynH+ewuHa1Itzk6Vu6P+0As= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GW+Sqmmg; spf=pass (imf19.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729233673; x=1760769673; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ehxQAVad+zYWKv7x3rvdIyaL5PCorY7OSY9LU2CsBiY=; b=GW+SqmmgFLFx/4YeKFMM3+tXB/2lkQqQBdZT10ZFlkYeegpzvG7Ei27D ycpChiXpZdim3Sekweg75+v9rEt0hQDI5rkQFUOzKWYzP8/jHM53K4QIE f3439H5uT1X1JVRwnGi9FiGgSROb1JyA/DaGbs3ZNSiUmUxcrwdOB3MzD I4DrAJzBq6mTaf6AnKSex8AlfnH49f0Vqs5WfGxvJ5BaQIEoExCJbqaJt PQoHBVkz2Lfz/sfM1wh2QkXznVxjTJ+V3xv4EOS9Ewjo6Z699g0CDIaMT dkGas68tJDW2KromvvTspJz8tozMEKx5YHR/mPLZrNaqSwUz5FyBLBnWh g==; X-CSE-ConnectionGUID: tcKG0LrkTnmmAYeN/6OLbg== X-CSE-MsgGUID: IDQq8eNRStClVi38JsohKw== X-IronPort-AV: E=McAfee;i="6700,10204,11228"; a="28884948" X-IronPort-AV: E=Sophos;i="6.11,212,1725346800"; d="scan'208";a="28884948" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2024 23:41:04 -0700 X-CSE-ConnectionGUID: PBvCfG+vRvKunH74mxz5gg== X-CSE-MsgGUID: nJbh3AsKSIKgmJHgOz/6cA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="83607534" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa003.jf.intel.com with ESMTP; 17 Oct 2024 23:41:04 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, mcgrof@kernel.org, kees@kernel.org, joel.granados@kernel.org, bfoster@redhat.com, willy@infradead.org, linux-fsdevel@vger.kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [RFC PATCH v1 11/13] mm: swap: Add IAA batch compression API swap_crypto_acomp_compress_batch(). Date: Thu, 17 Oct 2024 23:40:59 -0700 Message-Id: <20241018064101.336232-12-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241018064101.336232-1-kanchana.p.sridhar@intel.com> References: <20241018064101.336232-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: 65zoahs1fmzttkcjd5pwafuebd5m1gtg X-Rspamd-Queue-Id: 6610D1A0017 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1729233659-141190 X-HE-Meta: U2FsdGVkX192qVtzymKqQcR2mti3OhsAvec5wFGtda+r2qfX4tMGXNff4av9HP7PsoJpDH4jRFQq+E5RVmoioEZiL8cIcKK4cN282sgVk0KXZhk5ct4Cv40lz58T/v2YvtHJKflOuKA7j9vPESOtdBoxSs538TySdmYcOQ1hb847hSXZUkDhgFP+h0VDDKh74VeaaH6rTSg/Ipbnpl2kCfSlMe/0b8Gie3+bsohmbOxYRX32P4ngAxniSqcj3D7TtNwn860s58RtCz/+fCp0V1Hq5Kl7xS93VdBxsoQ93TB4qpXYTr/aEElgcu62RLSl8blNbJD0pJCnm5jrgMaffqypx34ZFCNCBDuzdgzwjU6I0aFwYRs0jdV7rp2WdjGHpTHXp7TDyHPIpOu3Z6PJaFTKgiRMggs2UdvHAui429HPVUeuvy4Q7O8ievJGNrNAUdbS5GDXFFmeFyp4otej/UHsvAEjDpVmxFvzydsx8NGsz2Sd1RkO73mUHQOEdoPhLIT2AlkmVrMKljKdzxmPd2+iGY15QTWgA1IWyWkflyfT25Y2ED9mQY/ehqQFlpJSOYzoeQcaAcIjEykujHBAU7Y2yBOZ8xRaW1jF62aAjweck/fE37e5GUo0WWU4zf9ikxpEdlhqs31vmZtgSfAqnf+Oq/0UGtxa3M66/5hw+kl0pQabU2wrJwIE6Oqs+EcwE3aE8hkWi3c77ARUolPz4Lj8lve+9yZLkWWYuDLCLiuquIFbwHlEE1GHINGMx/CcX4a3knKSISEIWenWdTE8CKssV9GdWbjyZn7qvcUBfNBzBmxub9DrESYHu0DHfrbTmhU0LxIUa5SGK3sHDOb8qKkD7z90FuOeOzSGkIRi1NgVR2LhethCXoUccDTrwD+jEyyj9uvTmdNjOtHlGy3nN/P/uHvTxfmytHwOhKKuwuGNch45Atj3wKrZQBxHjwBf8CylpyA39Z4REwYD5+/ IibUmPH3 kHBNGv1iZTBIQl5FdL1+hSucf0Il0BwwxQqHbhgg8EPCBN0qgrKctR89ohYC+02+Faheq5SYaYawCbpKbYf4gKEHlFlIaBd5mZXF81nyUSMKYSinsONdIycp6N0P17fpA4KopT95WY50ROUS4p4SZOWNeZuXdxaLmIQmua/jFP5eQh2I+O81rxNXJV9JqqOLJ0zyMlH8lXyfVvaxObDJjAdz8DyzSmd3y55jFf+UPts/ii3M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Added a new API swap_crypto_acomp_compress_batch() that does batch compression. A system that has Intel IAA can avail of this API to submit a batch of compress jobs for parallel compression in the hardware, to improve performance. On a system without IAA, this API will process each compress job sequentially. The purpose of this API is to be invocable from any swap module that needs to compress large folios, or a batch of pages in the general case. For instance, zswap would batch compress up to SWAP_CRYPTO_SUB_BATCH_SIZE (i.e. 8 if the system has IAA) pages in the large folio in parallel to improve zswap_store() performance. Towards this eventual goal: 1) The definition of "struct crypto_acomp_ctx" is moved to mm/swap.h so that mm modules like swap_state.c and zswap.c can reference it. 2) The swap_crypto_acomp_compress_batch() interface is implemented in swap_state.c. It would be preferable for "struct crypto_acomp_ctx" to be defined in, and for swap_crypto_acomp_compress_batch() to be exported via include/linux/swap.h so that modules outside mm (for e.g. zram) can potentially use the API for batch compressions with IAA. I would appreciate RFC comments on this. Signed-off-by: Kanchana P Sridhar --- mm/swap.h | 45 +++++++++++++++++++ mm/swap_state.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++ mm/zswap.c | 9 ---- 3 files changed, 160 insertions(+), 9 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 566616c971d4..4dcb67e2cc33 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -7,6 +7,7 @@ struct mempolicy; #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ +#include /* * For IAA compression batching: @@ -19,6 +20,39 @@ struct mempolicy; #define SWAP_CRYPTO_SUB_BATCH_SIZE 1UL #endif +/* linux/mm/swap_state.c, zswap.c */ +struct crypto_acomp_ctx { + struct crypto_acomp *acomp; + struct acomp_req *req[SWAP_CRYPTO_SUB_BATCH_SIZE]; + u8 *buffer[SWAP_CRYPTO_SUB_BATCH_SIZE]; + struct crypto_wait wait; + struct mutex mutex; + bool is_sleepable; +}; + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * The acomp_ctx mutex should be locked/unlocked before/after calling this + * procedure. + * + * @pages: Pages to be compressed. + * @dsts: Pre-allocated destination buffers to store results of IAA compression. + * @dlens: Will contain the compressed lengths. + * @errors: Will contain a 0 if the page was successfully compressed, or a + * non-0 error value to be processed by the calling function. + * @nr_pages: The number of pages, up to SWAP_CRYPTO_SUB_BATCH_SIZE, + * to be compressed. + * @acomp_ctx: The acomp context for iaa_crypto/other compressor. + */ +void swap_crypto_acomp_compress_batch( + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx); + /* linux/mm/page_io.c */ int sio_pool_init(void); struct swap_iocb; @@ -119,6 +153,17 @@ static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr, #else /* CONFIG_SWAP */ struct swap_iocb; +struct crypto_acomp_ctx {}; +static inline void swap_crypto_acomp_compress_batch( + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx) +{ +} + static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) { } diff --git a/mm/swap_state.c b/mm/swap_state.c index 4669f29cf555..117c3caa5679 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -23,6 +23,8 @@ #include #include #include +#include +#include #include "internal.h" #include "swap.h" @@ -742,6 +744,119 @@ void exit_swap_address_space(unsigned int type) swapper_spaces[type] = NULL; } +#ifdef CONFIG_SWAP + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * The acomp_ctx mutex should be locked/unlocked before/after calling this + * procedure. + * + * @pages: Pages to be compressed. + * @dsts: Pre-allocated destination buffers to store results of IAA compression. + * @dlens: Will contain the compressed lengths. + * @errors: Will contain a 0 if the page was successfully compressed, or a + * non-0 error value to be processed by the calling function. + * @nr_pages: The number of pages, up to SWAP_CRYPTO_SUB_BATCH_SIZE, + * to be compressed. + * @acomp_ctx: The acomp context for iaa_crypto/other compressor. + */ +void swap_crypto_acomp_compress_batch( + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_pages, + struct crypto_acomp_ctx *acomp_ctx) +{ + struct scatterlist inputs[SWAP_CRYPTO_SUB_BATCH_SIZE]; + struct scatterlist outputs[SWAP_CRYPTO_SUB_BATCH_SIZE]; + bool compressions_done = false; + int i, j; + + BUG_ON(nr_pages > SWAP_CRYPTO_SUB_BATCH_SIZE); + + /* + * Prepare and submit acomp_reqs to IAA. + * IAA will process these compress jobs in parallel in async mode. + * If the compressor does not support a poll() method, or if IAA is + * used in sync mode, the jobs will be processed sequentially using + * acomp_ctx->req[0] and acomp_ctx->wait. + */ + for (i = 0; i < nr_pages; ++i) { + j = acomp_ctx->acomp->poll ? i : 0; + sg_init_table(&inputs[i], 1); + sg_set_page(&inputs[i], pages[i], PAGE_SIZE, 0); + + /* + * Each acomp_ctx->buffer[] is of size (PAGE_SIZE * 2). + * Reflect same in sg_list. + */ + sg_init_one(&outputs[i], dsts[i], PAGE_SIZE * 2); + acomp_request_set_params(acomp_ctx->req[j], &inputs[i], + &outputs[i], PAGE_SIZE, dlens[i]); + + /* + * If the crypto_acomp provides an asynchronous poll() + * interface, submit the request to the driver now, and poll for + * a completion status later, after all descriptors have been + * submitted. If the crypto_acomp does not provide a poll() + * interface, submit the request and wait for it to complete, + * i.e., synchronously, before moving on to the next request. + */ + if (acomp_ctx->acomp->poll) { + errors[i] = crypto_acomp_compress(acomp_ctx->req[j]); + + if (errors[i] != -EINPROGRESS) + errors[i] = -EINVAL; + else + errors[i] = -EAGAIN; + } else { + errors[i] = crypto_wait_req( + crypto_acomp_compress(acomp_ctx->req[j]), + &acomp_ctx->wait); + if (!errors[i]) + dlens[i] = acomp_ctx->req[j]->dlen; + } + } + + /* + * If not doing async compressions, the batch has been processed at + * this point and we can return. + */ + if (!acomp_ctx->acomp->poll) + return; + + /* + * Poll for and process IAA compress job completions + * in out-of-order manner. + */ + while (!compressions_done) { + compressions_done = true; + + for (i = 0; i < nr_pages; ++i) { + /* + * Skip, if the compression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = crypto_acomp_poll(acomp_ctx->req[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + compressions_done = false; + } else { + dlens[i] = acomp_ctx->req[i]->dlen; + } + } + } +} +EXPORT_SYMBOL_GPL(swap_crypto_acomp_compress_batch); + +#endif /* CONFIG_SWAP */ + static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start, unsigned long *end) { diff --git a/mm/zswap.c b/mm/zswap.c index 579869d1bdf6..cab3114321f9 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -150,15 +150,6 @@ bool zswap_never_enabled(void) * data structures **********************************/ -struct crypto_acomp_ctx { - struct crypto_acomp *acomp; - struct acomp_req *req[SWAP_CRYPTO_SUB_BATCH_SIZE]; - u8 *buffer[SWAP_CRYPTO_SUB_BATCH_SIZE]; - struct crypto_wait wait; - struct mutex mutex; - bool is_sleepable; -}; - /* * The lock ordering is zswap_tree.lock -> zswap_pool.lru_lock. * The only case where lru_lock is not acquired while holding tree.lock is