From patchwork Thu Feb 27 10:15:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 13994053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA185C19F2E for ; Thu, 27 Feb 2025 10:15:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 486C9280006; Thu, 27 Feb 2025 05:15:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 411B5280004; Thu, 27 Feb 2025 05:15:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E9A7280006; Thu, 27 Feb 2025 05:15:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 01D09280004 for ; Thu, 27 Feb 2025 05:15:06 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B4CD4160E0C for ; Thu, 27 Feb 2025 10:15:06 +0000 (UTC) X-FDA: 83165316612.07.A5B176A Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) by imf08.hostedemail.com (Postfix) with ESMTP id 97DF8160062 for ; Thu, 27 Feb 2025 10:15:04 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=hmeau.com header.s=formenos header.b=b0jjzhxD; spf=pass (imf08.hostedemail.com: domain of herbert@gondor.apana.org.au designates 144.6.53.87 as permitted sender) smtp.mailfrom=herbert@gondor.apana.org.au; dmarc=pass (policy=quarantine) header.from=apana.org.au ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740651305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=5bLm0Q94gEJgQpTepUh8jBt4BX7iBQgckVIz1+HDSw8=; b=Cyjx7ZYi19qtltxH3P6QIprSY2BWO1Kv8MXTRpgmgFx48jaJNRO3PvRGeZL7tbjN5wXSut I+J+NlWFNSS9X8RmDizOFbDWGHPN0wS/7QaMoTnVSuHj2/H5S6GhtQF+L4Hs3GUKCNIJ+G YMia/58a1ByrXoxnJ7LnEh/I7XKQBYE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=hmeau.com header.s=formenos header.b=b0jjzhxD; spf=pass (imf08.hostedemail.com: domain of herbert@gondor.apana.org.au designates 144.6.53.87 as permitted sender) smtp.mailfrom=herbert@gondor.apana.org.au; dmarc=pass (policy=quarantine) header.from=apana.org.au ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740651305; a=rsa-sha256; cv=none; b=JPIyWpB22gXyl8djPDIC/ESEHglMQu1/MyGHf7EK/ibbXhDAaLenMfX+vOYF8ghFQ3D2OS ferj430qPFV+XO2qILn0PBtXwAMXPjAQLuV33HXNZP6OSy++qm62QeaOuZu2VNJUkNudCU y4TSmxdAWuC4IEE3cmzgTu7kARIjGD0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=Cc:To:Subject:From:References:In-Reply-To:Message-Id:Date: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=5bLm0Q94gEJgQpTepUh8jBt4BX7iBQgckVIz1+HDSw8=; b=b0jjzhxDEm4w22vBdiS01nujcH tOwm/kRJycBH+iY8c3ynLYLldXGucQyNXLUH1C5HN8sVPf89f22M0/gB0y6c0Ww4UJVJgVmgITy1l mE3ukpTSVLy9hs7EBC8WzHQuT8inkTgul5MSFDk+X1nImEQ8/geww9KXM+i3mjD8B3lGT20jwhYF2 3qqjBxTAbZ1Hv2hxsQO1OYQh7RwCF47ccJyRUmIdYB6n1i7HUfZ5q63fzpfUAZ8fg6guKiG6VBivL QXVCaFTGPzO+WvRWzpdVhFK/myDTZPnu10hxF7axIZiVVfryaOihfYxYxUaVj6YDOOglRRIyrmmve VWK91Fbg==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1tnauy-002Dqy-10; Thu, 27 Feb 2025 18:15:01 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 27 Feb 2025 18:15:00 +0800 Date: Thu, 27 Feb 2025 18:15:00 +0800 Message-Id: In-Reply-To: References: From: Herbert Xu Subject: [PATCH 3/7] crypto: acomp - Add request chaining and virtual addresses To: Linux Crypto Mailing List Cc: linux-mm@kvack.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 97DF8160062 X-Stat-Signature: 81e9xwta68mqyygbhemzzqh9qosobbix X-HE-Tag: 1740651304-634949 X-HE-Meta: U2FsdGVkX186km0eAgbfXwavDCLxxuZ/TeVXYmkvuKlAhI8xKZN4xpek7CIP2KtUu7UXMmroKG1xWU6C/99lTK1r/mwrvAyojCBj1MRNsPEfr87LYu1YvelG//AB1LUdMWbTvYMRlvm0ahM8+3RnTAGL89zLtrcpDGkAZTdw/TmWqJGExbZHIUPPXQ335OG0+737aGXzZx9nvyeZS7GZgwU02OcLJWdV1Ol8QJCVoqvCuBFtLF9ps5N2lXXxaB0Ve5KTFKOjlQQdds7RYsrEAcD06BaBufFuc2wLEZAqqbSmbYFg8UJMjw+1xVNGaWZMe0JCrL6uxLeZq985JUmGeUsdeBUo7WUmi/0dXQ2ixmEQNsQfeUAWIE/3cqetkgCdgnyU/Sd+QiJtNH+AWzETuUL7axvtFnqtE6aPdzp0vu5HF/T5pHjfJV0ZU2WvTvyG6cQoPQvo4w8nN4tA83vCxnJmObrxb6cIRaBVhdZFjFZMO7NTct6Qe2+Iqr9Hq74Oe+OICLQkhinjEFeTvMd2helFp5t6jNvdKDeZKAShPmfDRaOVfRF+vogsy5aFWszzE12pSpXhPTfBjiYBKORv10+9e9cq5UA0AxXv0wXMg8D4yhdfrpi5nKwnUqKRO4nSkIpqo3x5FO2p+GH4FVkVR96OX/DRX5teXX24WeiIJD/i1ZF+hIToUUeWkR/NtLgmLWV6lLSPZY5gEDAZ/NDHUe5JmbY2x2pc2IuX1Rf1cLAKM+kJa3cEdUvarVm+Ms5Z+iF1RM8b1rbwf3Kt4q9dyYaTyaQO8D4Yud51Sjy+f6GS62yuwzGi6DLBJlLWu8h/cy/tJrSfHH0SAWkLkX5dd5+ViHnwtAgT5vzXXIty8l4wcOa7CtIjXvCIgpQnUiB/ChqOkHVbe01/BsXZEP+Wk/mB2186o5dzL1yz3ybCYj1PvLxcO6qCmx2CzhVXT1nUQH4IRPjbcNnE5IB29UD 4aTDcyF9 2mm/OjL2adN/GWHYgm9SntJp9RhZSNVrzDuf0m8X92lznTFVeJEUQM9BzSzfMa+y9eG1RmR5w7qZobAHjmSv2WNRQccvdlnjqC63pMpZq8VZjpb26BUcTAmUhwsFzuKNTnkn/bY85j8jRTfAQ2tybzeVMNbKRgJNTdmfBXTnCcO8B/IAYeRWd/2sYzN38rZAfGZRkpQq8BlW5eDNw03yHq2Xr4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This adds request chaining and virtual address support to the acomp interface. It is identical to the ahash interface, except that a new flag CRYPTO_ACOMP_REQ_NONDMA has been added to indicate that the virtual addresses are not suitable for DMA. This is because all existing and potential acomp users can provide memory that is suitable for DMA so there is no need for a fall-back copy path. Signed-off-by: Herbert Xu --- crypto/acompress.c | 201 ++++++++++++++++++++++++++++ include/crypto/acompress.h | 89 ++++++++++-- include/crypto/internal/acompress.h | 22 +++ 3 files changed, 299 insertions(+), 13 deletions(-) diff --git a/crypto/acompress.c b/crypto/acompress.c index 30176316140a..d2103d4e42cc 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -23,6 +23,8 @@ struct crypto_scomp; static const struct crypto_type crypto_acomp_type; +static void acomp_reqchain_done(void *data, int err); + static inline struct acomp_alg *__crypto_acomp_alg(struct crypto_alg *alg) { return container_of(alg, struct acomp_alg, calg.base); @@ -153,6 +155,205 @@ void acomp_request_free(struct acomp_req *req) } EXPORT_SYMBOL_GPL(acomp_request_free); +static bool acomp_request_has_nondma(struct acomp_req *req) +{ + struct acomp_req *r2; + + if (acomp_request_isnondma(req)) + return true; + + list_for_each_entry(r2, &req->base.list, base.list) + if (acomp_request_isnondma(r2)) + return true; + + return false; +} + +static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct acomp_req_chain *state = &req->chain; + + if (!acomp_is_async(tfm)) + return; + + state->compl = req->base.complete; + state->data = req->base.data; + req->base.complete = cplt; + req->base.data = state; + state->req0 = req; +} + +static void acomp_restore_req(struct acomp_req_chain *state) +{ + struct acomp_req *req = state->req0; + struct crypto_acomp *tfm; + + tfm = crypto_acomp_reqtfm(req); + if (!acomp_is_async(tfm)) + return; + + req->base.complete = state->compl; + req->base.data = state->data; +} + +static void acomp_reqchain_virt(struct acomp_req_chain *state, int err) +{ + struct acomp_req *req = state->cur; + unsigned int slen = req->slen; + unsigned int dlen = req->dlen; + + req->base.err = err; + if (!state->src) + return; + + acomp_request_set_virt(req, state->src, state->dst, slen, dlen); + state->src = NULL; +} + +static int acomp_reqchain_finish(struct acomp_req_chain *state, + int err, u32 mask) +{ + struct acomp_req *req0 = state->req0; + struct acomp_req *req = state->cur; + struct acomp_req *n; + + acomp_reqchain_virt(state, err); + + if (req != req0) + list_add_tail(&req->base.list, &req0->base.list); + + list_for_each_entry_safe(req, n, &state->head, base.list) { + list_del_init(&req->base.list); + + req->base.flags &= mask; + req->base.complete = acomp_reqchain_done; + req->base.data = state; + state->cur = req; + + if (acomp_request_isvirt(req)) { + unsigned int slen = req->slen; + unsigned int dlen = req->dlen; + const u8 *svirt = req->svirt; + u8 *dvirt = req->dvirt; + + state->src = svirt; + state->dst = dvirt; + + sg_init_one(&state->ssg, svirt, slen); + sg_init_one(&state->dsg, dvirt, dlen); + + acomp_request_set_params(req, &state->ssg, &state->dsg, + slen, dlen); + } + + err = state->op(req); + + if (err == -EINPROGRESS) { + if (!list_empty(&state->head)) + err = -EBUSY; + goto out; + } + + if (err == -EBUSY) + goto out; + + acomp_reqchain_virt(state, err); + list_add_tail(&req->base.list, &req0->base.list); + } + + acomp_restore_req(state); + +out: + return err; +} + +static void acomp_reqchain_done(void *data, int err) +{ + struct acomp_req_chain *state = data; + crypto_completion_t compl = state->compl; + + data = state->data; + + if (err == -EINPROGRESS) { + if (!list_empty(&state->head)) + return; + goto notify; + } + + err = acomp_reqchain_finish(state, err, CRYPTO_TFM_REQ_MAY_BACKLOG); + if (err == -EBUSY) + return; + +notify: + compl(data, err); +} + +static int acomp_do_req_chain(struct acomp_req *req, + int (*op)(struct acomp_req *req)) +{ + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + struct acomp_req_chain *state = &req->chain; + int err; + + if (crypto_acomp_req_chain(tfm) || + (!acomp_request_chained(req) && !acomp_request_isvirt(req))) + return op(req); + + /* + * There are no in-kernel users that do this. If and ever + * such users come into being then we could add a fall-back + * path. + */ + if (acomp_request_has_nondma(req)) + return -EINVAL; + + if (acomp_is_async(tfm)) { + acomp_save_req(req, acomp_reqchain_done); + state = req->base.data; + } + + state->op = op; + state->cur = req; + state->src = NULL; + INIT_LIST_HEAD(&state->head); + list_splice_init(&req->base.list, &state->head); + + if (acomp_request_isvirt(req)) { + unsigned int slen = req->slen; + unsigned int dlen = req->dlen; + const u8 *svirt = req->svirt; + u8 *dvirt = req->dvirt; + + state->src = svirt; + state->dst = dvirt; + + sg_init_one(&state->ssg, svirt, slen); + sg_init_one(&state->dsg, dvirt, dlen); + + acomp_request_set_params(req, &state->ssg, &state->dsg, + slen, dlen); + } + + err = op(req); + if (err == -EBUSY || err == -EINPROGRESS) + return -EBUSY; + + return acomp_reqchain_finish(state, err, ~0); +} + +int crypto_acomp_compress(struct acomp_req *req) +{ + return acomp_do_req_chain(req, crypto_acomp_reqtfm(req)->compress); +} +EXPORT_SYMBOL_GPL(crypto_acomp_compress); + +int crypto_acomp_decompress(struct acomp_req *req) +{ + return acomp_do_req_chain(req, crypto_acomp_reqtfm(req)->decompress); +} +EXPORT_SYMBOL_GPL(crypto_acomp_decompress); + void comp_prepare_alg(struct comp_alg_common *alg) { struct crypto_alg *base = &alg->base; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index b6d5136e689d..15bb13e47f8b 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -12,10 +12,34 @@ #include #include #include +#include +#include #define CRYPTO_ACOMP_ALLOC_OUTPUT 0x00000001 + +/* Set this bit for virtual address instead of SG list. */ +#define CRYPTO_ACOMP_REQ_VIRT 0x00000002 + +/* Set this bit for if virtual address buffer cannot be used for DMA. */ +#define CRYPTO_ACOMP_REQ_NONDMA 0x00000004 + #define CRYPTO_ACOMP_DST_MAX 131072 +struct acomp_req; + +struct acomp_req_chain { + struct list_head head; + struct acomp_req *req0; + struct acomp_req *cur; + int (*op)(struct acomp_req *req); + crypto_completion_t compl; + void *data; + struct scatterlist ssg; + struct scatterlist dsg; + const u8 *src; + u8 *dst; +}; + /** * struct acomp_req - asynchronous (de)compression request * @@ -24,14 +48,24 @@ * @dst: Destination data * @slen: Size of the input buffer * @dlen: Size of the output buffer and number of bytes produced + * @chain: Private API code data, do not use * @__ctx: Start of private context data */ struct acomp_req { struct crypto_async_request base; - struct scatterlist *src; - struct scatterlist *dst; + union { + struct scatterlist *src; + const u8 *svirt; + }; + union { + struct scatterlist *dst; + u8 *dvirt; + }; unsigned int slen; unsigned int dlen; + + struct acomp_req_chain chain; + void *__ctx[] CRYPTO_MINALIGN_ATTR; }; @@ -200,10 +234,14 @@ static inline void acomp_request_set_callback(struct acomp_req *req, crypto_completion_t cmpl, void *data) { + u32 keep = CRYPTO_ACOMP_ALLOC_OUTPUT | CRYPTO_ACOMP_REQ_VIRT; + req->base.complete = cmpl; req->base.data = data; - req->base.flags &= CRYPTO_ACOMP_ALLOC_OUTPUT; - req->base.flags |= flgs & ~CRYPTO_ACOMP_ALLOC_OUTPUT; + req->base.flags &= keep; + req->base.flags |= flgs & ~keep; + + crypto_reqchain_init(&req->base); } /** @@ -230,11 +268,42 @@ static inline void acomp_request_set_params(struct acomp_req *req, req->slen = slen; req->dlen = dlen; - req->base.flags &= ~CRYPTO_ACOMP_ALLOC_OUTPUT; + req->base.flags &= ~(CRYPTO_ACOMP_ALLOC_OUTPUT | CRYPTO_ACOMP_REQ_VIRT); if (!req->dst) req->base.flags |= CRYPTO_ACOMP_ALLOC_OUTPUT; } +/** + * acomp_request_set_virt() -- Sets virtual address request parameters + * + * Sets virtual address parameters required by an acomp operation + * + * @req: asynchronous compress request + * @src: virtual address pointer to input buffer + * @dst: virtual address pointer to output buffer. + * @slen: size of the input buffer + * @dlen: size of the output buffer. + */ +static inline void acomp_request_set_virt(struct acomp_req *req, + const u8 *src, u8 *dst, + unsigned int slen, + unsigned int dlen) +{ + req->svirt = src; + req->dvirt = dst; + req->slen = slen; + req->dlen = dlen; + + req->base.flags &= ~CRYPTO_ACOMP_ALLOC_OUTPUT; + req->base.flags |= CRYPTO_ACOMP_REQ_VIRT; +} + +static inline void acomp_request_chain(struct acomp_req *req, + struct acomp_req *head) +{ + crypto_request_chain(&req->base, &head->base); +} + /** * crypto_acomp_compress() -- Invoke asynchronous compress operation * @@ -244,10 +313,7 @@ static inline void acomp_request_set_params(struct acomp_req *req, * * Return: zero on success; error code in case of error */ -static inline int crypto_acomp_compress(struct acomp_req *req) -{ - return crypto_acomp_reqtfm(req)->compress(req); -} +int crypto_acomp_compress(struct acomp_req *req); /** * crypto_acomp_decompress() -- Invoke asynchronous decompress operation @@ -258,9 +324,6 @@ static inline int crypto_acomp_compress(struct acomp_req *req) * * Return: zero on success; error code in case of error */ -static inline int crypto_acomp_decompress(struct acomp_req *req) -{ - return crypto_acomp_reqtfm(req)->decompress(req); -} +int crypto_acomp_decompress(struct acomp_req *req); #endif diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h index 8831edaafc05..b3b48dea7f2f 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -109,4 +109,26 @@ void crypto_unregister_acomp(struct acomp_alg *alg); int crypto_register_acomps(struct acomp_alg *algs, int count); void crypto_unregister_acomps(struct acomp_alg *algs, int count); +static inline bool acomp_request_chained(struct acomp_req *req) +{ + return crypto_request_chained(&req->base); +} + +static inline bool acomp_request_isvirt(struct acomp_req *req) +{ + return req->base.flags & CRYPTO_ACOMP_REQ_VIRT; +} + +static inline bool acomp_request_isnondma(struct acomp_req *req) +{ + return (req->base.flags & + (CRYPTO_ACOMP_REQ_NONDMA | CRYPTO_ACOMP_REQ_VIRT)) == + (CRYPTO_ACOMP_REQ_NONDMA | CRYPTO_ACOMP_REQ_VIRT); +} + +static inline bool crypto_acomp_req_chain(struct crypto_acomp *tfm) +{ + return crypto_tfm_req_chain(&tfm->base); +} + #endif