From patchwork Mon Mar 3 08:47:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13998350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7DB6C282C6 for ; Mon, 3 Mar 2025 08:47:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE580280009; Mon, 3 Mar 2025 03:47:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A7489280008; Mon, 3 Mar 2025 03:47:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C434280009; Mon, 3 Mar 2025 03:47:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 58AD6280002 for ; Mon, 3 Mar 2025 03:47:38 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 181501604CC for ; Mon, 3 Mar 2025 08:47:38 +0000 (UTC) X-FDA: 83179611396.07.A79FA04 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf24.hostedemail.com (Postfix) with ESMTP id DFC4C180008 for ; Mon, 3 Mar 2025 08:47:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XiRSzR5C; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740991656; a=rsa-sha256; cv=none; b=gc/O3dGUAD5gXAfK8zx6S8LO+TNuMAKhvufoJ7F1ZG8kAHhTkdpbmMBhANRqDedT3aj99D /MTbmzf4JZ58Vfoap0yjHnnAQUeFls/LSCi1oPQpCGMWRzWPDIrIZT8VMCsW9qnsd3GNUF l6NsFZ6bMQeQIQDfhj9qNnyC2usWtT4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XiRSzR5C; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740991656; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CxDBO9hF3bAE8sokaQQ0dSqjUz0eILnCamYYQTChDNg=; b=XdtXHkBSLS/Afsatdb0VP8d3nFTLFAM7K9uUftfAmLlVsdpOagSzeToq7ME8XtZR4KhIa3 NsgA5w1WBIFyR7ljDmKk180zUNMMeB9hwhQtGlFwfrMSRZNohJxZSDpfuu26u+EYNP9Mfw 91y+2FeYMDYNuTADppvNRP3O2SINokQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1740991656; x=1772527656; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yAxiyAJkmOi+qCLJ6YIryDvvFkW0bqE3LR6f2KKFmuI=; b=XiRSzR5C6tDwGAKQ5NnHyHxbJYUvK6wt+xIiNE2xqD3NdpxI5irK6NMq 6QJslMBFQSr+cTUwifj+o4Hgqkp5RXmzKukzAmvgdrgwVh2/J19clqlrN tN32u5z3kb7eV7kiBD3RvCIeR8x0dYQVys/jd3FZ1Z/+9Nc8M+69I8XQI RNVO1G6KU26SEA5CYWJIJV3JXvo2X9Y3RE6Q8rhE+NBF0sQtVB+CmgdXD N+Bk65ra5VqhyjANZH20tAOmdiTSZVp6TOuX2lHRvUMTNbX4RsgkuDHOg BqJ6lppUHZDahr95WX/0lvZmCkXCDiSZGSETCPQ52X+xc7Zg4Msjf2ADA w==; X-CSE-ConnectionGUID: K8FJg+EASMGteoUEsrtEfQ== X-CSE-MsgGUID: 8lbT7k+zSFyY9mlotnoNRQ== X-IronPort-AV: E=McAfee;i="6700,10204,11361"; a="42111898" X-IronPort-AV: E=Sophos;i="6.13,329,1732608000"; d="scan'208";a="42111898" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Mar 2025 00:47:31 -0800 X-CSE-ConnectionGUID: fvBsJymEQP6lwEgEmf6goA== X-CSE-MsgGUID: RpkS3VTXRlGfrTvv0qcbqg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.13,329,1732608000"; d="scan'208";a="118426786" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 03 Mar 2025 00:47:28 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v8 04/14] crypto: iaa - Implement batch compression/decompression with request chaining. Date: Mon, 3 Mar 2025 00:47:14 -0800 Message-Id: <20250303084724.6490-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250303084724.6490-1-kanchana.p.sridhar@intel.com> References: <20250303084724.6490-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DFC4C180008 X-Stat-Signature: ps368ongugxwnz8dgshb7b8abjxr7ncy X-Rspam-User: X-HE-Tag: 1740991655-976853 X-HE-Meta: U2FsdGVkX18pfHYSWh88xO27Y/tK5nQSmVgf8vzFdGP+OalZgBI3tz1y5Qb6Bcq8Wp9i9lC5haks9HzkeeCwDCxo4K1DIy0ehBph5EhNFE5o7r2Xua4PfqGhfsY46hKeKPT0WiHsYJsUuzb0YLy2iYctd86W+CNxlqEDzdBjKmEAzMLLTow+o+t5TX1tr5ZbbHdUmCnFAEKWpf7fRD6oeQlxccnpXZ9aY6YjbOpoyEEDfy+tlk0TcOntvAX596jzEgCGvamTHgTDFFy32wmPjjUS8ai1/zjmQcVcXh4VsgmYudh1aU9KhyQkz9j+60IijGklUKrdSJphVk+VraxAWPrsBgacLvSBj1b+3OsfcPphYR4LfYx8YckfAR8B1IgEKJHcqZec7BLx9dvQpS9Yt0lkOuMjNl69lYU4e+Q6vgbkOwlz9+fELvfCbqAUr3REPV26mLOp4yMgVWXEE49h8O+sQ/Fb64qwb3wKuZi3FhLmsL1zbj3vOa74CShwdD0rMOaMLQENwEriIT+XHhT6ThHvx24D6NatUxPjDv7P+tmtcL+jK0p25MGhOOOCAeVfKwitt2uRKKCA/IVPfGuo72JLcowjPKGnViKq46p89DawylujxSRIi5SVFvDgX8rLx5P5A0qa2PCtuf9bnpwWn7WJv3f65ljuSF7nB1jBXaFuIaj6VlJU9fijhXSLnzlWk6oHqO50znXeMFnW32xhx2TExLt3QLU8L/aAc7QWC6EDd/s0o33B/2v/gDk1fj3dTuGrZGeyWll9RPV6OQfExEu9yXZrZXzAurQp3WZKOLDf2FoXuV4HpMtvAFQ7LA0+Oebal8NRNn/QFob/I7s+EWC1S/900XxjwxFn0J3jvrrVFk8P2crPAzkfaKYsHwxujQY5j5VQVBw5cvQMGm18bb+emzeo/DL4sebdY34Que+H1RpPq3AsWjgGWk/3tCQ9UCVrRtxJw7N8INsShEw 2yti4Tve LzaJ01HwCwm+AofljRK9qVjEjD1TVzP7SrpAu9CeXvBQc0wI67bTiF6+RXk88djDxS72BDSmvB8RHZMmuuc2cw/8YraC4h5T8IEI0lx/aLPEjnohxZPwcBD2QLz8GO0EslR91Ccsuzks2uPZ0ejXjfUDUisEQHNY5AWgOIgS/NXaPU8iKCC8oEYYiEcs6se+C+hFoJ35KPGkkqQSgvuWsT+TlC+SvmoWVUwLq6rTEqknFNzo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch provides the iaa_crypto driver implementation for the newly added crypto_acomp "get_batch_size()" interface which will be called when swap modules invoke crypto_acomp_batch_size() to query for the maximum batch size. This will return an iaa_driver specific constant, IAA_CRYPTO_MAX_BATCH_SIZE (set to 8U currently). This allows swap modules such as zswap/zram to allocate required batching resources and then invoke fully asynchronous batch parallel compression/decompression of pages on systems with Intel IAA, by setting up a request chain, and calling crypto_acomp_compress() or crypto_acomp_decompress() with the head request in the chain. This enables zswap compress batching code to be developed in a manner similar to the current single-page synchronous calls to crypto_acomp_compress() and crypto_acomp_decompress(), thereby, facilitating encapsulated and modular hand-off between the kernel zswap/zram code and the crypto_acomp layer. This patch also provides implementations of IAA batching with request chaining for both iaa_crypto sync modes: asynchronous/no-irq and fully synchronous. Since iaa_crypto supports the use of acomp request chaining, this patch also adds CRYPTO_ALG_REQ_CHAIN to the iaa_acomp_fixed_deflate algorithm's cra_flags. Suggested-by: Herbert Xu Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 9 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 186 ++++++++++++++++++++- 2 files changed, 192 insertions(+), 3 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 56985e395263..45d94a646636 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -39,6 +39,15 @@ IAA_DECOMP_CHECK_FOR_EOB | \ IAA_DECOMP_STOP_ON_EOB) +/* + * The maximum compress/decompress batch size for IAA's implementation of + * batched compressions/decompressions. + * The IAA compression algorithms should provide the crypto_acomp + * get_batch_size() interface through a function that returns this + * constant. + */ +#define IAA_CRYPTO_MAX_BATCH_SIZE 8U + /* Representation of IAA workqueue */ struct iaa_wq { struct list_head list; diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index d7983ab3c34a..a9800b8f3575 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1807,6 +1807,185 @@ static void compression_ctx_init(struct iaa_compression_ctx *ctx) ctx->use_irq = use_irq; } +static int iaa_comp_poll(struct acomp_req *req) +{ + struct idxd_desc *idxd_desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + struct idxd_wq *wq; + bool compress_op; + int ret; + + idxd_desc = req->base.data; + if (!idxd_desc) + return -EAGAIN; + + compress_op = (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS); + wq = idxd_desc->wq; + iaa_wq = idxd_wq_get_private(wq); + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + ret = check_completion(dev, idxd_desc->iax_completion, true, true); + if (ret == -EAGAIN) + return ret; + if (ret) + goto out; + + req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + if (compress_op) { + update_total_comp_bytes_out(req->dlen); + update_wq_comp_bytes(wq, req->dlen); + } else { + update_total_decomp_bytes_in(req->slen); + update_wq_decomp_bytes(wq, req->slen); + } + + if (iaa_verify_compress && (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS)) { + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + + src_addr = sg_dma_address(req->src); + dst_addr = sg_dma_address(req->dst); + + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } +out: + /* caller doesn't call crypto_wait_req, so no acomp_request_complete() */ + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + idxd_free_desc(idxd_desc->wq, idxd_desc); + + dev_dbg(dev, "%s: returning ret=%d\n", __func__, ret); + + return ret; +} + +static unsigned int iaa_comp_get_batch_size(void) +{ + return IAA_CRYPTO_MAX_BATCH_SIZE; +} + +static void iaa_set_reqchain_poll( + struct acomp_req *req0, + bool set_flag) +{ + struct acomp_req *req; + + set_flag ? (req0->flags |= CRYPTO_ACOMP_REQ_POLL) : + (req0->flags &= ~CRYPTO_ACOMP_REQ_POLL); + + list_for_each_entry(req, &req0->base.list, base.list) + set_flag ? (req->flags |= CRYPTO_ACOMP_REQ_POLL) : + (req->flags &= ~CRYPTO_ACOMP_REQ_POLL); +} + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. Batching is implemented using request chaining. + * + * @req: The head asynchronous compress request in the chain. + * + * Returns the compression error status (0 or -errno) of the last + * request that finishes. Caller should call acomp_request_err() + * for each request in the chain, to get its error status. + */ +static int iaa_comp_acompress_batch(struct acomp_req *req) +{ + bool async = (async_mode && !use_irq); + int err = 0; + + if (likely(async)) + iaa_set_reqchain_poll(req, true); + else + iaa_set_reqchain_poll(req, false); + + + if (likely(async)) + /* Process the request chain in parallel. */ + err = acomp_do_async_req_chain(req, iaa_comp_acompress, iaa_comp_poll); + else + /* Process the request chain in series. */ + err = acomp_do_req_chain(req, iaa_comp_acompress); + + /* + * For the same request chain to be usable by + * iaa_comp_acompress()/iaa_comp_adecompress() in synchronous mode, + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs. + */ + iaa_set_reqchain_poll(req, false); + + return err; +} + +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. Batching is implemented using request chaining. + * + * @req: The head asynchronous decompress request in the chain. + * + * Returns the decompression error status (0 or -errno) of the last + * request that finishes. Caller should call acomp_request_err() + * for each request in the chain, to get its error status. + */ +static int iaa_comp_adecompress_batch(struct acomp_req *req) +{ + bool async = (async_mode && !use_irq); + int err = 0; + + if (likely(async)) + iaa_set_reqchain_poll(req, true); + else + iaa_set_reqchain_poll(req, false); + + + if (likely(async)) + /* Process the request chain in parallel. */ + err = acomp_do_async_req_chain(req, iaa_comp_adecompress, iaa_comp_poll); + else + /* Process the request chain in series. */ + err = acomp_do_req_chain(req, iaa_comp_adecompress); + + /* + * For the same request chain to be usable by + * iaa_comp_acompress()/iaa_comp_adecompress() in synchronous mode, + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs. + */ + iaa_set_reqchain_poll(req, false); + + return err; +} + +static int iaa_compress_main(struct acomp_req *req) +{ + if (acomp_is_reqchain(req)) + return iaa_comp_acompress_batch(req); + + return iaa_comp_acompress(req); +} + +static int iaa_decompress_main(struct acomp_req *req) +{ + if (acomp_is_reqchain(req)) + return iaa_comp_adecompress_batch(req); + + return iaa_comp_adecompress(req); +} + static int iaa_comp_init_fixed(struct crypto_acomp *acomp_tfm) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp_tfm); @@ -1829,13 +2008,14 @@ static void dst_free(struct scatterlist *sgl) static struct acomp_alg iaa_acomp_fixed_deflate = { .init = iaa_comp_init_fixed, - .compress = iaa_comp_acompress, - .decompress = iaa_comp_adecompress, + .compress = iaa_compress_main, + .decompress = iaa_decompress_main, .dst_free = dst_free, + .get_batch_size = iaa_comp_get_batch_size, .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa", - .cra_flags = CRYPTO_ALG_ASYNC, + .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_REQ_CHAIN, .cra_ctxsize = sizeof(struct iaa_compression_ctx), .cra_module = THIS_MODULE, .cra_priority = IAA_ALG_PRIORITY,