From patchwork Tue Feb 20 02:55:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13563285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B664C48BF8 for ; Tue, 20 Feb 2024 02:56:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FFF76B0083; Mon, 19 Feb 2024 21:56:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B0D26B0087; Mon, 19 Feb 2024 21:56:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E93906B0088; Mon, 19 Feb 2024 21:56:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DA1536B0083 for ; Mon, 19 Feb 2024 21:56:14 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A6184A0508 for ; Tue, 20 Feb 2024 02:56:14 +0000 (UTC) X-FDA: 81810668268.30.4239A1E Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) by imf10.hostedemail.com (Postfix) with ESMTP id C3C78C001C for ; Tue, 20 Feb 2024 02:56:12 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mDB3dWtB; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.167.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708397772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tiMCpE6cizjhU1RT67E8V4r+8Ln0KAuPCSDlqhlksl4=; b=M04oh9DIL4in2Xjm8AlYLSFIf4yaKqlpKRmNhaMEjE0eluyDHv8lTnAURfvcFnj9m025CS 61hfaqSBQB+EAysSKDRpi1P1vJqlsTzD06my0/Tk2EGsnFK1Vz9yVlg3o3nl1Q586kQPvY Q2uZeHCn8FWvk3/OUr/19e4Yn7oPavc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708397772; a=rsa-sha256; cv=none; b=ReJnE2ofLiZnXtPmcWCgk0s7ZMIE1NHM3/fQJAqQop3v1B1FoTOjjjYZTdpGtriRDmEM1d 0vGF1/04KMNSZyEsFp+VsBL64OHeuEbBGbS0n6YwIxOunVjMWm1uuWJFB2fCTKJKoW1QWU 1KZYGOmXmc+mcJCjHHuC5Gn5A+Cxszw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mDB3dWtB; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.167.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3c049ccb623so2848584b6e.1 for ; Mon, 19 Feb 2024 18:56:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708397772; x=1709002572; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tiMCpE6cizjhU1RT67E8V4r+8Ln0KAuPCSDlqhlksl4=; b=mDB3dWtB5iKtrJRF7jL5c6/GCFhhTjNsnKjW5d9m7YnUJ54FA+hEJag26EN7qOOM9W jtSaF5t9KMOIhEuqrWb30csRROj06VGgurvVR0Ig5WfmmWLX4uaJ9Dp/3ZwzAjk7jobF wM7T/9vZPzrjEN6q4RFaHpupJU2LO0HO0T8t/C//voJPnggZwILaG3Kw0CGcLC39Rm2E ldgQfpsfkLOJV5EUkkbIU7BZx0R2KWQ8uUSRh2IKQnXzOMqRhKGaXMbKkEr+zLAfxZi/ cwwzUbgsTDVMK2Gf8jYRCfS7qJaZAE+jV/2VafqpVXYuH3nllAwDX9dsfxRVMDNb1DMO tIlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708397772; x=1709002572; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tiMCpE6cizjhU1RT67E8V4r+8Ln0KAuPCSDlqhlksl4=; b=V4xpomtAvbhiBZSfr5fB6ei/5NTDjQ0tXT8aLzbjVz+9JHuz+nBkQJH8bm9r1ygonT w2meg3pIBEsDUAl7CEn4UK/lAb/xeAwT6il/rrdAzTGX7HSQX+jsC83we9+6Aanh8btj eqnYvyypagGSxDHbj/9mWQMbosOPCgcj7oZ/Uwj6E3CQ591mU+AOsVHz2me2yBi9zBwt Os6ZOvJEYyfKAGEtnvlqTUTHrzfJP80iFuuVWtU1lZ1xV4XmGhQLeaj3YusAZZHrOxy2 qyKwM9O9HkIvB/0L+sQMoXjvNi5IBfE4WyyT5b0y5XoZGYctzRQ2J6Qy+soWXdvFfGh7 7EGw== X-Forwarded-Encrypted: i=1; AJvYcCW3GrLk59aExAGhzlcghd3hb0qpch1KPllcFFzLNNFcLPZJCkPWI3MpR6pNGkYsgVJJlePKbIEwKH9GR+xgkevuR3w= X-Gm-Message-State: AOJu0YzjOKgaw5xuSg1JSS7wSz8tm6VKE81eIA0ANednVN4ZEdtxaijP ZE0E04qaxxjFbTadd4cj7EGzvKCqE6x9LG8aY4n+PishobD11lkk X-Google-Smtp-Source: AGHT+IGt+eY+tIvCG4LwXzbVuI0ZKfC6ws0uDOdftQOJVvSjsz5UFEJGJlJE5E0UOzF0YnLQ2I4PrA== X-Received: by 2002:a05:6358:7f0e:b0:176:92d1:568f with SMTP id p14-20020a0563587f0e00b0017692d1568fmr13688884rwn.18.1708397771772; Mon, 19 Feb 2024 18:56:11 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a5bd:9c11:af2b:aecf]) by smtp.gmail.com with ESMTPSA id j34-20020a63fc22000000b005dc1edf7371sm5500047pgi.9.2024.02.19.18.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Feb 2024 18:56:11 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, davem@davemloft.net, hannes@cmpxchg.org, herbert@gondor.apana.org.au, linux-crypto@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com Cc: chriscli@google.com, chrisl@kernel.org, ddstreet@ieee.org, linux-kernel@vger.kernel.org, sjenning@redhat.com, vitaly.wool@konsulko.com, Barry Song , Yang Shen , Zhou Wang , Tom Zanussi Subject: [PATCH v4 1/3] crypto: introduce crypto_acomp_get_alg_flags to expose algorithm flags Date: Tue, 20 Feb 2024 15:55:43 +1300 Message-Id: <20240220025545.194886-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240220025545.194886-1-21cnbao@gmail.com> References: <20240220025545.194886-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C3C78C001C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f9njxfpruest3irbo1wdc4omzz8jq98y X-HE-Tag: 1708397772-948378 X-HE-Meta: U2FsdGVkX1/voBrcWN72Y96BqG/SJBo+ELxa2ExDiy3bIYJnicxg/zcYtX0TVQ0QHEtQVYglMtvFkmkKjv4nUy1vNQ7kiyszPjl8GzLcBIg8UhBCjZUnKzhSKPOuNUB8BOWwzThKAQ22f80+lcbdVGpUSWMovPe+aBfpO22WRQColfMtILDnV5Fksr3zRWfgV3N3BaU9jEK7Oj8SLkRJhj9vur6YlX++3I0mDpz14Vs1sYV/bGaM7zNaO89KsctOWrkceKPdDCrt+Sn3fT/UzymzHfIUDitX3b3Wo+4R33BkzcZOTTZoffcBSpkvqbX+A9KDsR6QGmXuRraRF9WJ8mM2slUtB+ZF4f/Aw2c1945QNPJiwsoYvLJR6wYbp/mCgYbNba3rqOuUnt+uJGA3wplmmxdakDzLOtKy8HYS3P3Up9bPJYzMypRWfo1MR+3FUfGo39ldcJVnY2TB+SFM6CBSpD1YjNhNW7J/WWudFuApORIiPpbc17atAB4bFB7lEnksZ/ygAtI1ZuqQd4BTUaV2VkEnU1eMggvv4cAhGk2utjFFMH6DGldXEe5B/D0S3hZ6oWZPl+1Lcb4ezRKWn6LGBewIyoGO52z/+Ls9VKfCkowQ45vOHGXCWyS8yPF7/uUmQl+ekg458HiT1awir1yRB/v8L2wQjWx0Xiac8SDbKPX7sXv8NIBGHAt74GX/ojXyf5zVEGuA7/1BzYNOozehiMjED5NfWI7woG/CKYA+5gUWlxgGQr83mtSSlf/F16AlZFdOdnRORoHldfm/DKwFgeEQ1DcgSB+hy+8TaPqJiDfYgfJN/Izv9hAZQHwmCWbiJqM2Z9rjmmv09lxtweeL4YpdXJ6DyA0vsANtcG1OQN8dNG4Hx+zUHt7rvb18A9zB19KOWpnHCxAglYbWH5v0Gm+KJcMH4s6PlhGpWWjImYzGwIPHRjU1OrLM+X7pbT1hUr6bKYn3QWALLnE DCu2oFeB Ip8i8Wz0bNCDOC7Zo6Kgx7yqt1kiMEKPfVdY8KkNI68x38Jb+nPL5QuKNqLfhHSokF+QhQyatR97pH2xfNpPG7RIcafTdW/X5sJeN8yrcBw0LuGeujI6qK1O1s0wUacO3obNOA4jrQcXth2q5GoD+Tl0NbFQHzTZLfP7H1Wpyl2l6TAgUjhanpbTeng/YjbH23Z01BwuebdQlQBo4xmPt6C3V8WZcc6sDs1wb/BGe5QaP2pf1srCG0nDnBxXF7XO3hLHrieqSKtb+gbu2LxfHYqGPnI9bU2K+EqORaWTgp2n8NMS0+xfI69OEP9IpLYuGiwiP0qGd7EF8w6E7+SLJYY7TNcU+TsW6130PInlu2TqGpDomqXlxfsjhiJbeVCG4PKSU8hA8z9UrO2MH8g0UQglx8c7PAh70TEaFWiaa3BrFgMZCwvJK+6qbW5zAq6qojqcKTOXBuolb+iOdurDUWiNlRPkKJWEfo4wRlP3nLpGBZjSp8u7SfY0rKxFO4Hpx8CQMKTJWHxAnyh8MyDwdZMyXs4kzJQN1cBFri1enEDml5BRb4wvNupdUrVA64qBv8NpbexhEiowFZr/fOeJkkyXzzfcrc+UHNHoSAss680usJqznXnlxVGpOMJBJVjenMNBIwrDeYzYmeXw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song acomp's users might want to know algorithm flags to optimize themselves. One typical user which can benefit from exposed alg flags is zswap. In zswap, zsmalloc is the most commonly used allocator for (and perhaps the only one). For zsmalloc, we cannot sleep while we map the compressed memory, so we copy it to a temporary buffer. By knowing the alg won't sleep can help zswap to avoid the need for a buffer. This shows noticeable improvement in load/store latency of zswap. This patch also fixes the missing ASYNC cra_flags in Intel iaa and Hisilicon zip drivers. Cc: Yang Shen Cc: Zhou Wang Cc: Tom Zanussi Signed-off-by: Barry Song --- drivers/crypto/hisilicon/zip/zip_crypto.c | 1 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 1 + include/crypto/acompress.h | 5 +++++ include/linux/crypto.h | 5 +++++ 4 files changed, 12 insertions(+) diff --git a/drivers/crypto/hisilicon/zip/zip_crypto.c b/drivers/crypto/hisilicon/zip/zip_crypto.c index c650c741a18d..94e2d66b04b6 100644 --- a/drivers/crypto/hisilicon/zip/zip_crypto.c +++ b/drivers/crypto/hisilicon/zip/zip_crypto.c @@ -591,6 +591,7 @@ static struct acomp_alg hisi_zip_acomp_deflate = { .base = { .cra_name = "deflate", .cra_driver_name = "hisi-deflate-acomp", + .cra_flags = CRYPTO_ALG_ASYNC, .cra_module = THIS_MODULE, .cra_priority = HZIP_ALG_PRIORITY, .cra_ctxsize = sizeof(struct hisi_zip_ctx), diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index dfd3baf0a8d8..91adf9d76a2e 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1916,6 +1916,7 @@ static struct acomp_alg iaa_acomp_fixed_deflate = { .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa", + .cra_flags = CRYPTO_ALG_ASYNC, .cra_ctxsize = sizeof(struct iaa_compression_ctx), .cra_module = THIS_MODULE, .cra_priority = IAA_ALG_PRIORITY, diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 574cffc90730..07bd8f6bc79a 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -160,6 +160,11 @@ static inline void acomp_request_set_tfm(struct acomp_req *req, req->base.tfm = crypto_acomp_tfm(tfm); } +static inline u32 crypto_acomp_get_alg_flags(struct crypto_acomp *tfm) +{ + return crypto_tfm_alg_flags(crypto_acomp_tfm(tfm)); +} + static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req) { return __crypto_acomp_tfm(req->base.tfm); diff --git a/include/linux/crypto.h b/include/linux/crypto.h index b164da5e129e..811bfaf8b6f8 100644 --- a/include/linux/crypto.h +++ b/include/linux/crypto.h @@ -467,6 +467,11 @@ static inline unsigned int crypto_tfm_alg_blocksize(struct crypto_tfm *tfm) return tfm->__crt_alg->cra_blocksize; } +static inline unsigned int crypto_tfm_alg_flags(struct crypto_tfm *tfm) +{ + return tfm->__crt_alg->cra_flags; +} + static inline unsigned int crypto_tfm_alg_alignmask(struct crypto_tfm *tfm) { return tfm->__crt_alg->cra_alignmask; From patchwork Tue Feb 20 02:55:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13563286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52F6FC48BF8 for ; Tue, 20 Feb 2024 02:56:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1F386B0088; Mon, 19 Feb 2024 21:56:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCFEC6B0089; Mon, 19 Feb 2024 21:56:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98046B008A; Mon, 19 Feb 2024 21:56:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B936C6B0088 for ; Mon, 19 Feb 2024 21:56:21 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 916D81A0501 for ; Tue, 20 Feb 2024 02:56:21 +0000 (UTC) X-FDA: 81810668562.10.21202F3 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf16.hostedemail.com (Postfix) with ESMTP id B4131180013 for ; Tue, 20 Feb 2024 02:56:19 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L50POOx5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708397779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9x09aF372n2E4KjAq0WuthNT6iC+2DOlm894AWq4Fis=; b=TwDTyiZPjgF7pZ4Td9P2QahIUV8B0FuPALCUqVwl+2s6QaKWoBGAMfjE/9kfVl7TPUuxB2 Y0WNtE7p6bV2lfs2hxLQiyP1UQ1H5O4TQSir9VwLX0d+jzgGqtnkKAeOh6w2ANGUClcxfF h3Z6AkRNCFUqgu7cKWlgS9zGN96yyx8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L50POOx5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708397779; a=rsa-sha256; cv=none; b=AINqteQGGi2ZG7EaY1iTm24UoLyfQ5tAwXLXeQtHx9e/IV+tP5rmj9EcoKjx0TbFMTEwRN AQvQyBTesqBmBPp2eAry6YXEi8Ul51qgvTPtgwdnZV/HDjljfg0lG7t9qoFiKsvWhGmXU2 inm5vyff5RtKsRea88ZX57kAyvWYxhA= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2995185dbbbso2471976a91.3 for ; Mon, 19 Feb 2024 18:56:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708397778; x=1709002578; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9x09aF372n2E4KjAq0WuthNT6iC+2DOlm894AWq4Fis=; b=L50POOx5QNFKS8c5OgNirCX41JAp2CwfzHCF55cp5biIKr+Wh2vfVEIehVCSQKE2r9 MyFbgnJ3rPbhlAVp5YIvHxGq00BlSCTsLrrbFOLJkidSfjujYVf9m+X2mOxOKewmg4qk /6cULJ3VthIm4fAi8a2lWUJpCUImRIKhpGaCHvR+bRWG7XGdNcggO2sKQ7aeBHnvKa1z TH+AtIS23bMtRIeMsZQRE/+FWumMFfjsx1Rkk3jqcpgsx1c76EbcSIrPMrtnVNg0o6gi ClRGtylrykZ5VwNsfkan5xr75nHGIAxwPsSvm9ia4ZzznCL9mnOhHfHUX5h6QEiH3PtN s4xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708397778; x=1709002578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9x09aF372n2E4KjAq0WuthNT6iC+2DOlm894AWq4Fis=; b=A6RJ6CBvwGdYiWSAbXf9VjaBVgBnnM8Hg2mIdq+0jck8JhAjqxKwzBhtfi4KIFa3i6 c2pd+oAHWgLuZWdnLOCTSvYmFRMtG1qqNqBaKNS2XHxIpVSMRxebBSaY3GciliAwgFbr raMHHNEItJ9esQkehlGPL+J+7wnDPWLnYGMMBoSRoqmq0f7Qyj2UIh4f1xIsIUkLsSaw EyZiuUJlacZ1AhL53TYPgJ9scT/HKDdGgoA57ho4t82nAaU3K7NfLv5ZOVGdLhpqV/eb boHlQXexi+2jK6kuwwNcBvZLKzsbj40gXma+vhmhGMpJnXC7ParErqxhqYNO8jxtE2xx Ygfg== X-Forwarded-Encrypted: i=1; AJvYcCVks78Ym6LMEDaEAaQDa4atF2c4iHxEjupP2PP/3VSzdh8s4WZe9tt469Zogdr3oytJ7BvRLPpDDNOylR53SUodwzM= X-Gm-Message-State: AOJu0YyRFdwdAFeml17hGWw8kgxN4Me5LoBOVe4iNSeu0L4qaTxRGwGa W3fcYNgZUMLMWXNNm4EMu183E2y+Q+T8eyeGrHwzqd+knUjEFy6/ X-Google-Smtp-Source: AGHT+IHU7hs7P+vFcx0S81/nTm/28ngSUvZjSrzTKAbOsDkdi7c59QtXqE4hy9y4VXKC0lM9IVR2mg== X-Received: by 2002:a17:90a:f48a:b0:299:2b9a:88bc with SMTP id bx10-20020a17090af48a00b002992b9a88bcmr10421142pjb.37.1708397778390; Mon, 19 Feb 2024 18:56:18 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a5bd:9c11:af2b:aecf]) by smtp.gmail.com with ESMTPSA id j34-20020a63fc22000000b005dc1edf7371sm5500047pgi.9.2024.02.19.18.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Feb 2024 18:56:18 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, davem@davemloft.net, hannes@cmpxchg.org, herbert@gondor.apana.org.au, linux-crypto@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com Cc: chriscli@google.com, chrisl@kernel.org, ddstreet@ieee.org, linux-kernel@vger.kernel.org, sjenning@redhat.com, vitaly.wool@konsulko.com, Barry Song Subject: [PATCH v4 2/3] mm/zswap: remove the memcpy if acomp is not sleepable Date: Tue, 20 Feb 2024 15:55:44 +1300 Message-Id: <20240220025545.194886-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240220025545.194886-1-21cnbao@gmail.com> References: <20240220025545.194886-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B4131180013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hczef6siz7hhubkkme5sf4fuw7dj7xqo X-HE-Tag: 1708397779-514734 X-HE-Meta: U2FsdGVkX1+YSfbiva9YnIeE0/6r5X+w66JR0JGkuJIc+C3e3yW6dmnkz2f4yu5sQauKoCi3ZvlxeEieZGl5uLgAVbpEe2GDT/roXWoL4xpIzTu63S2n39+EZgAjQwam5wRdfVy/I7W/6IGRbWBILp9s0K/qs87j3QhlTRYngJfV7Wxf/IwedWdomLuxpWEFU4N/t/pacGx21PKooSPUe0D9RTttwvV78zCdZHl1jGcDyyPtt5NoP/3jpVcZ7aWIzGsMtA1Gk+BXP2vujNVUEtgpu7KZsL6j6FY5pd8zDy8R0ZvOMkWh272FwHH/1BEQlbVSTN7O7kQeP/fwRWa3NZM9OpnINYyUFOJRbfWBXqDKMjh2Ml6RYxj51FqdLqPkoaBfKTN13dp+QdixTeMin/Tk3yrP41OsGi4lFI+Y5m7HBzAiUcBS3c8LhH49F5eWGoRiX76M3sDdXNhj8JSwQ/E28+Sw6Zg7MElCZ4oAextkjRmBcGjQq/ewEnQqKzvzV+Mshmia0L+Esqke+aNKY55VsxjbTssMzbOD/CTs9T2WReDphj2uY5Wz7/XFz/f7PmIi76YtHbczWs7E0R9kZMLXWPMM6It3lb3YNu/a0z9pqqon84Ndf/ukksyxblMhFYSjXDdZhQiNIj1bhIdHQ9iroByJI9dtPDI8DvlMVSF0LNxKrxSjOkNTFkT5o5z0VIuiOvQ118Wdh50smk6a3uyEBEbrNrn3rPnqWsZXaaxnwgojDqwAMtSjSCEOKDQBD2bTAeRnqFydaWD8VrBmzq/lkb0qWLYgYsReAl4gNyCBr/LSbSRu25Gw5oPPzf/rq8tgCBgzVwjaUlzi6dobqVSQU1RTDVj5L5ntoIuSQAmB30N9mekke8eyq/rrSoMC5VoPRvQgTKWToo0QBJac1jXTNDrrdgn2dAhtR9O0ZTQvScanl8DO09i0MJ/R/8UI7LPzZ80LpjhDEUgx0Uj rd48gV5G pRi0Kn2Jbyb9XTelkWBvfHkVcS/OdtUv2O9En7xz2DnJT7WAx1VST0M60kEjzn1NimSEUMlvMxWINw5b2sNAhfIlNlSdbytFjirQ68DyMxTdHk70JuMwHW7sWW7QvdGPytgHJKTkOhIYYIm2WHbWcmxM20DaUMPzlP9tFERcwGeDAclys9yGGNtHtqByOM8AgO6RyixLrEAZsEgY7bXs/wCjMa4crHeVZQsdNtgdoUQB98cx2rvHSbazrRjHG08hAU3F8WPi/VwM/kBANFY9KmU1GLkeB39bCgO7H37NX9KzFuGg2HRvIeThTPlHLw+8/hdZNDRgV8R9h6cR26zaK/JHhCbTAq1zUEawsXXkkWFhx3TgrYDgXpMQSQbclJA+VoEIID3gJAiQIyVfa5Nrwd0ZGR1pt8BCevbUx5oMvMRh8p3aY0JX91fcIQPA3B3MorpWK0BxqLPEtE5XXWPDRtoDe6Y0sgN1IVhUfwt7HtF7y5rDMwU64RCWiUsxVbjv2/XrmkidzzCICSbIMIQrRCOb7wNLoFXKQ95KeSFORAa3ok+SUenoRJUkhlqBm0EijtmhDEhnhq7riHGm5LSPBidDfqA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Most compressors are actually CPU-based and won't sleep during compression and decompression. We should remove the redundant memcpy for them. This patch checks the algorithm flags CRYPTO_ALG_ASYNC to know if acomp is sleepable. Generally speaking, async and sleepable are semantically similar but not equal. But for compress drivers, they are actually equal at least due to the below facts. Firstly, scompress drivers - crypto/deflate.c, lz4.c, zstd.c, lzo.c etc have no sleep. Secondly, zRAM has been using these scompress drivers for years in atomic contexts, and never worried those drivers going to sleep. Signed-off-by: Barry Song Tested-by: Chengming Zhou Reviewed-by: Nhat Pham Acked-by: Yosry Ahmed Reviewed-by: Chengming Zhou --- mm/zswap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 51de79aa8659..115a780fa2c1 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -162,6 +162,7 @@ struct crypto_acomp_ctx { struct crypto_wait wait; u8 *buffer; struct mutex mutex; + bool is_sleepable; }; /* @@ -973,6 +974,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) goto acomp_fail; } acomp_ctx->acomp = acomp; + acomp_ctx->is_sleepable = crypto_acomp_get_alg_flags(acomp) & + CRYPTO_ALG_ASYNC; req = acomp_request_alloc(acomp_ctx->acomp); if (!req) { @@ -1100,7 +1103,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct page *page) mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { + if (acomp_ctx->is_sleepable && !zpool_can_sleep_mapped(zpool)) { memcpy(acomp_ctx->buffer, src, entry->length); src = acomp_ctx->buffer; zpool_unmap_handle(zpool, entry->handle); @@ -1114,7 +1117,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct page *page) BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); mutex_unlock(&acomp_ctx->mutex); - if (zpool_can_sleep_mapped(zpool)) + if (!acomp_ctx->is_sleepable || zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); } From patchwork Tue Feb 20 02:55:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13563287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21031C48BF8 for ; Tue, 20 Feb 2024 02:56:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEAF36B008A; Mon, 19 Feb 2024 21:56:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A9B876B008C; Mon, 19 Feb 2024 21:56:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93C9B6B0092; Mon, 19 Feb 2024 21:56:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 836246B008A for ; Mon, 19 Feb 2024 21:56:28 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8AC4E160537 for ; Tue, 20 Feb 2024 02:56:27 +0000 (UTC) X-FDA: 81810668814.11.B797BE6 Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) by imf10.hostedemail.com (Postfix) with ESMTP id A17E4C0019 for ; Tue, 20 Feb 2024 02:56:25 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OLHOxZ70; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708397785; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kW/qQndLj7MYeSMYgAPMliKXg25U647vW1M3U8+Bgew=; b=AMI3a9/cENujLJJ4gE0vKKiimDbTY+w9WwnmXqJqorPMs0Itcg97d/scaJ0V7kbtDjJJWt KueDOrbCfHpRoaGIrW2RtT0lqRtmtoU9LGvOWtR9CLeb0yMVbTSP9hED1rVT7IP3c5+Rq3 UYAFGJrBarcOWlN3S+27h1Q1kp39Rgs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OLHOxZ70; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708397785; a=rsa-sha256; cv=none; b=RvpTCyjjwiAoh02Mi9sEOAsy7RQgg1dbw6QjUZuDxVwSIFJnD04rTNr2WivHfXYzbsxdjs XgHEyygaPnhd5WCJBvFSMWSNhSjaQYl6St3Hp+F6sqEBH5GmFCDbVQdHtGcFOaCD3VDiRu T8DQH3hg0R+TAp80rwvo3GGQhZqxrXY= Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-59883168a83so2209047eaf.2 for ; Mon, 19 Feb 2024 18:56:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708397785; x=1709002585; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kW/qQndLj7MYeSMYgAPMliKXg25U647vW1M3U8+Bgew=; b=OLHOxZ70pjinIX/jXni28PBBXizvgm6SFw3lAa3gQBFRmcReQa/SJ4evF1euRrp3Pl aQNjR16roFmrouA1/HWB89233Pu80f4AZi2kbrIeBkfqCdSzYv8c/2KFaXZtmZywS0N4 GQQWzAh3S5nuwFEzIUqD5mwXGEsTnr/ysdhMogvhNIKHcflu/zab9BFzCWF9YGpAVSkM TfJIayeoPYn9bVXSPMxjd6lL8xc1PQE89AjWZJiKJdmo55ZE4unkmQDEqx8krMtbfZh6 cUDkc6V6aI1x7AJCy+76+BoyK7Xu+Hr36Naq+yX+gYx5CdNXxw6EM/q1SIEeAa7q0fV0 XMjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708397785; x=1709002585; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kW/qQndLj7MYeSMYgAPMliKXg25U647vW1M3U8+Bgew=; b=ocaDqJNkohS1jsxPVZVT8RyUcpvGeItMB8FWy+P6qg2L2srTCcWe/YWhzILlGoh9DX 7RvB1UUilPH5dNNp8nszDgzSbMpg38Hxnp38m6J5ZuA/AmywZPftQp72dV98KZIOUL/f zvL6ixMoXlAjSeSEW7zkBLJQKT1IDNPgmcLKOEBL8Qh4JD2d6Lg+Lr/sy8xs77EsYXWI JHcVsQLEY2DpNBg+MrNx73MlUlRVcoxar3ySGbVy1FShDLl65tYyR0Vry6ARGSZF1cps OWaj5tviKWm1PmIoMQ16Vxl6yyhzOY19EDuPMKd+zDRgLHF003p8IjMv6XQ2NJb2FTLe jk3Q== X-Forwarded-Encrypted: i=1; AJvYcCWowaYf/AziKpOqhhqVBL/5g2qUSfLOIpfmmU7l+Wc3CWS5EAAnTyB8guJdzrnpXXDEszEJjMa90atf0HqnXJmbJqc= X-Gm-Message-State: AOJu0YzVmKq93rbDs3SYVw9WvxSQvcIesj27dCYQFp/mTMzUrmS+ogeo B/f0KurE85Ufkzvg+CdNg+MfVSqCRbYuGoIrvozuV6RKfmAdWGEf X-Google-Smtp-Source: AGHT+IHm1nRrDZtEWW28L4x2rAgwEL50c0JazukXW93V03ZXEqdBNo20ZbIjvDpQgzMenUad5lxqAw== X-Received: by 2002:a05:6358:6584:b0:176:8472:800f with SMTP id x4-20020a056358658400b001768472800fmr9579507rwh.31.1708397784669; Mon, 19 Feb 2024 18:56:24 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a5bd:9c11:af2b:aecf]) by smtp.gmail.com with ESMTPSA id j34-20020a63fc22000000b005dc1edf7371sm5500047pgi.9.2024.02.19.18.56.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Feb 2024 18:56:24 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, davem@davemloft.net, hannes@cmpxchg.org, herbert@gondor.apana.org.au, linux-crypto@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com Cc: chriscli@google.com, chrisl@kernel.org, ddstreet@ieee.org, linux-kernel@vger.kernel.org, sjenning@redhat.com, vitaly.wool@konsulko.com, Barry Song Subject: [PATCH v4 3/3] crypto: scompress: remove memcpy if sg_nents is 1 Date: Tue, 20 Feb 2024 15:55:45 +1300 Message-Id: <20240220025545.194886-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240220025545.194886-1-21cnbao@gmail.com> References: <20240220025545.194886-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A17E4C0019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: df3qt5ah65si43hasry3us5t688brwf3 X-HE-Tag: 1708397785-439057 X-HE-Meta: U2FsdGVkX1/54RTZhpvpah7ycwuBbAPX1NRXT7CcIEXVbu+irPe7lDXOZKtfb2F+Mej8gmjG7lKEvqlNbNXUHmJlxNa3L9jQm5xLL5IkEu2ftzXSko8lQoHoUg9NxrVHzP2byhU259JNtYDq2W0hHLkUcKObyGxgKjwOUc0NksXOyUuxSbEnLlCICJNqXFicwFbPWud5QB8ZYUhhEsYuMId7eIgh9d7b+X/WlEQJH76ABV3gLCfeV8bJhSNBhUbm5EOwPXk7duyqcnlxBZShI3kfi0CZodZDdqQljT/DXDAwy8gAkCCLhZa0kE9qTMWc3ESyyK/FNWRDJTAL2Gk69nJx58DnLym5xGqlnEWgNnX5UMvalCSgKYYTtGswK3wNkaVXa3P3tWlzB8mIQBrEZYogC6avi7w6QCOZIv8fbplikEm1/IQoklxm/NXhi44ctmJO0/TPbTfBHqBtFeiUcb11lCm+vndXYVzrMDP5V2l8AQh4lBx+KLPdELNp/8UftF7r1R7RGT/eRBfO+rVLpT09k9GY7Bq1o8yIb1uErZRXJ0JR9zW6reKCll0VU7vDxy9Ubr3Ka1gojLMkEJkGAo3Kwti509Ij8Bov18IHej7nhjjPNQGNX3aRxHgBaX7H/4xV2WmiYmfilkypKh8lZuUDgjwXOWBX6RSMmOGQg4JAgZUWdeMCjxpjRKNto02d+D38e5BCn+KdFnSz5/UP+j/kMJrSNNN14oxrmBzfj7VBQYIzheO9tAHVI6gLwR9osu5v+JzMd8CvTS49AIqqfllpsMFoBWCnjONbop8jwv9a08kkiMlJZXchwJJYKBz7jkVqPYy9oq+9MIUVt7/nKBJtvsZBYK3MkOsZq3vYPqGTHMS8gNWwUjvzHrL4meEPFmWXMRC/37Q+d4v4n/9aQg/KCf9frLruamKSefKVKQvyKK+o/s8/cfwZBok+/EgoV+jrvaCEUdZmpkC5Cyu sp9nhbFw dPY4fA6OalaQr0eK1c7SrqyiChor6OnjDKUwWUM8sKfwUfNyrSkZe1bc7I98TnCuOxc5oHSB9YxhDIfhirn60NWoc3RnUcZ9OkuXriqBZCmWcDpGDbpxyPOiqyn6jS1WZm6EmevZO52gzF0Q4bMmybN00WjkYL6pOv6UZfIBQwElMteEv4nSYQ6UFGFbuQShmt8IGrJ6AkuN1sChUOKcA5Gz+zd6oEgM/jnUODhy7XLKgJuj/i44Pm5TsQFo/4srao6R2RE3iCcNwlDZHQem1opd+zRjvr9/iWlKN7SxC3VUhKfE6Q/HCskcsmIvGoAsvAlisRyMdc2+uSFcELtTqeR8VpLI3u1P/23S+7Czc39HUThbC7nRU63SLviESmdtctbbUi9pp6IqS1S/2GND9HFdiP4FYKwm6Cm40JlKZA2/ny5prHCeNkh52Ps+IhTE4Qy2hGCWNn0ld9JAs6SXbT6CIr7Up/A9g3dzLoo0n3coGh4GJ6pPGTRMDX+gGIrtXzTNnKJbYvkeveP3LxhH2Pw8hf7lSMQT07w6rKJOcHgrxV20ciEN0sbsfrG68HbP+dSejU7H0FHEk/guNNZFoATrdkOAcHN0+oITObaq5CKo72EHhiVThnCwHVwJypMWOgNPsKPgGJyTfsBMAYbYEHTdIw19wTNcfKosNQpzIabkS+r6R7vZvR6g6sTMyhg4u4Sm2CrhPJuX2VIE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song while sg_nents is 1 which is always true for the current kernel as the only user - zswap is the case, we should remove two big memcpy. Signed-off-by: Barry Song Tested-by: Chengming Zhou --- crypto/scompress.c | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/crypto/scompress.c b/crypto/scompress.c index b108a30a7600..50a487eac792 100644 --- a/crypto/scompress.c +++ b/crypto/scompress.c @@ -117,6 +117,7 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) struct crypto_scomp *scomp = *tfm_ctx; void **ctx = acomp_request_ctx(req); struct scomp_scratch *scratch; + void *src, *dst; unsigned int dlen; int ret; @@ -134,13 +135,25 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) scratch = raw_cpu_ptr(&scomp_scratch); spin_lock(&scratch->lock); - scatterwalk_map_and_copy(scratch->src, req->src, 0, req->slen, 0); + if (sg_nents(req->src) == 1) { + src = kmap_local_page(sg_page(req->src)) + req->src->offset; + } else { + scatterwalk_map_and_copy(scratch->src, req->src, 0, + req->slen, 0); + src = scratch->src; + } + + if (req->dst && sg_nents(req->dst) == 1) + dst = kmap_local_page(sg_page(req->dst)) + req->dst->offset; + else + dst = scratch->dst; + if (dir) - ret = crypto_scomp_compress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_compress(scomp, src, req->slen, + dst, &req->dlen, *ctx); else - ret = crypto_scomp_decompress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_decompress(scomp, src, req->slen, + dst, &req->dlen, *ctx); if (!ret) { if (!req->dst) { req->dst = sgl_alloc(req->dlen, GFP_ATOMIC, NULL); @@ -152,10 +165,19 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) ret = -ENOSPC; goto out; } - scatterwalk_map_and_copy(scratch->dst, req->dst, 0, req->dlen, - 1); + if (dst == scratch->dst) { + scatterwalk_map_and_copy(scratch->dst, req->dst, 0, + req->dlen, 1); + } else { + flush_dcache_page(sg_page(req->dst)); + } } out: + if (src != scratch->src) + kunmap_local(src); + if (dst != scratch->dst) + kunmap_local(dst); + spin_unlock(&scratch->lock); return ret; }