From patchwork Wed Jan 31 20:27:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junaid Shahid X-Patchwork-Id: 10194779 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AB41760595 for ; Wed, 31 Jan 2018 20:27:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 958932887F for ; Wed, 31 Jan 2018 20:27:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89F5E28898; Wed, 31 Jan 2018 20:27:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EA97328896 for ; Wed, 31 Jan 2018 20:27:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751786AbeAaU1b (ORCPT ); Wed, 31 Jan 2018 15:27:31 -0500 Received: from mail-pf0-f195.google.com ([209.85.192.195]:45869 "EHLO mail-pf0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751579AbeAaU1a (ORCPT ); Wed, 31 Jan 2018 15:27:30 -0500 Received: by mail-pf0-f195.google.com with SMTP id a88so13831496pfe.12 for ; Wed, 31 Jan 2018 12:27:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5TH/2mXeq+qTRIpRrgjPR1r+5f9zUz+P5XWwWBJTEXA=; b=aCl/Y5d2D4Z2ykQYMaiT7LIJ3cD6JhsuYLtSEFz5imZe2qJSiVqj0YiArW0FxStsAO uy2LgI9RwxmCTnGH/ZoK7E2cn05lN1KogR3u09iQig9wHh2eBT281HYYJCAmH2+rR9SP 9tnIYdBJMWiILt3NdG9qOr9xm7FCSke0aI7rseSTOmZbtoXkhExl3NpXu3d0QP2KgKb4 AmL+zoz0cNxjJ5G3whqj81t87slHO7YAQVuZU7IoY+ZR2wlXmt+yiyAIfGFftgMkgdJk 291fOc86Q2Sl6lqOax9KMCVNQdbyjqI34dTzeleFtScy2MtZyvkiMOI8/q3nIAijitZs sp+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5TH/2mXeq+qTRIpRrgjPR1r+5f9zUz+P5XWwWBJTEXA=; b=lnJhH4zkFkhH0hD0VHG/SJ8rgKcHaLO5m0STY6QgYn9DNO1KjrsvwUSuioS9b/Zr7f dZLxtMkGkTJKZOFuaqM73zxBtQuGvO2HMJJdrPuxfVcbEkeL+wyHfNmmP5DwW8Zjm2b5 EzjJxIg6cMF9mJ0bBDbLUIAlcS3IO3cujsK/Aw+636p/ST8JB1ZK7In8ZA9rWLbs0hKQ cw6nFr4nVu4nqcwKUm+0c+yg0NnZ5Z2YiwN2tfflLsXW5lsP8qbdxvLd/Ut+rQ8e0Wt7 kdi0EOr0KvQ97Bfr27PowAx1Rp7eB4zGewDyaLw5mGSrUe4XNrLP/U41c3wXJPwweknF LT4g== X-Gm-Message-State: AKwxyteF4qr3UEmksIsPq0tDSjm3TTjVUpNT/yxLM1E4kZwxfsQ1Kn7h qIRaMUT8tMpjwP2dSHCBviAY4w== X-Google-Smtp-Source: AH8x224f/A//jz/SnNExe3PZLX2aIYsVrduMLd+ID0Fo1I+qqPcS4AvC4o1otk5gcg9t5nm2gU/gNw== X-Received: by 2002:a17:902:e81:: with SMTP id 1-v6mr1044474plx.169.1517430450022; Wed, 31 Jan 2018 12:27:30 -0800 (PST) Received: from js-desktop.svl.corp.google.com ([2620:15c:2cb:1:cdad:b4d5:21d1:e91e]) by smtp.gmail.com with ESMTPSA id q16sm46081788pfg.124.2018.01.31.12.27.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Jan 2018 12:27:29 -0800 (PST) From: Junaid Shahid To: herbert@gondor.apana.org.au Cc: linux-crypto@vger.kernel.org, andreslc@google.com, davem@davemloft.net, gthelen@google.com, ebiggers3@gmail.com, smueller@chronox.de Subject: [PATCH v3 4/4] crypto: aesni - Use zero-copy for gcm(aes) even if the AAD/Data/AuthTag are separate Date: Wed, 31 Jan 2018 12:27:22 -0800 Message-Id: <20180131202722.212273-5-junaids@google.com> X-Mailer: git-send-email 2.16.0.rc1.238.g530d649a79-goog In-Reply-To: <20180131202722.212273-1-junaids@google.com> References: <20180131202722.212273-1-junaids@google.com> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Enable the use of zero-copy even if the AAD and/or Auth Tag are in different buffers than the actual data, as long as each of them individually satisfies the zero-copy conditions (i.e. the entire buffer is either in low-mem or within a single high-mem page). Signed-off-by: Junaid Shahid --- arch/x86/crypto/aesni-intel_glue.c | 121 +++++++++++++++++++++++++++---------- 1 file changed, 89 insertions(+), 32 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 9e69e02076d2..7cebc99a0405 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -756,42 +756,91 @@ static u8 *map_buffer(struct scatterlist *sgl) } /* - * Maps the sglist buffer and returns a pointer to the mapped buffer in - * data_buf. + * Maps the sglist buffer and returns pointers to the mapped buffers in assoc, + * data and (optionally) auth_tag. * * If direct mapping is not feasible, then allocates a bounce buffer if one - * isn't already available in bounce_buf, and returns a pointer to the bounce - * buffer in data_buf. + * isn't already available in bounce_buf, and returns pointers within the bounce + * buffer in assoc, data and auth_tag. * - * When the buffer is no longer needed, put_request_buffer() should be called on - * the data_buf and the bounce_buf should be freed using kfree(). + * When the buffers are no longer needed, put_request_buffers() should be called + * and the bounce_buf should be freed using kfree(). */ -static int get_request_buffer(struct scatterlist *sgl, - unsigned long bounce_buf_size, - u8 **data_buf, u8 **bounce_buf, bool *mapped) +static int get_request_buffers(struct scatterlist *sgl, + unsigned long assoc_len, unsigned long data_len, + unsigned long auth_tag_len, + u8 **assoc, u8 **data, u8 **auth_tag, + u8 **bounce_buf, bool *mapped) { - if (sg_is_last(sgl) && is_mappable(sgl, sgl->length)) { + struct scatterlist sgl_data_chain[2], sgl_auth_tag_chain[2]; + struct scatterlist *sgl_data, *sgl_auth_tag; + + sgl_data = scatterwalk_ffwd(sgl_data_chain, sgl, assoc_len); + sgl_auth_tag = scatterwalk_ffwd(sgl_auth_tag_chain, sgl, + assoc_len + data_len); + + if (is_mappable(sgl, assoc_len) && is_mappable(sgl_data, data_len) && + (auth_tag == NULL || is_mappable(sgl_auth_tag, auth_tag_len))) { *mapped = true; - *data_buf = map_buffer(sgl); + + *assoc = map_buffer(sgl); + + if (sgl->length >= assoc_len + data_len) + *data = *assoc + assoc_len; + else + *data = map_buffer(sgl_data); + + if (auth_tag != NULL) { + if (sgl_data->length >= data_len + auth_tag_len) + *auth_tag = *data + data_len; + else + *auth_tag = map_buffer(sgl_auth_tag); + } + return 0; } *mapped = false; if (*bounce_buf == NULL) { - *bounce_buf = kmalloc(bounce_buf_size, GFP_ATOMIC); + *bounce_buf = kmalloc(assoc_len + data_len + auth_tag_len, + GFP_ATOMIC); if (unlikely(*bounce_buf == NULL)) return -ENOMEM; } - *data_buf = *bounce_buf; + *assoc = *bounce_buf; + *data = *assoc + assoc_len; + + if (auth_tag != NULL) + *auth_tag = *data + data_len; + return 0; } -static void put_request_buffer(u8 *data_buf, bool mapped) +static void put_request_buffers(struct scatterlist *sgl, bool mapped, + u8 *assoc, u8 *data, u8 *auth_tag, + unsigned long assoc_len, + unsigned long data_len, + unsigned long auth_tag_len) { - if (mapped) - kunmap_atomic(data_buf); + struct scatterlist sgl_data_chain[2]; + struct scatterlist *sgl_data; + + if (!mapped) + return; + + sgl_data = scatterwalk_ffwd(sgl_data_chain, sgl, assoc_len); + + /* The unmaps need to be done in reverse order of the maps. */ + + if (auth_tag != NULL && sgl_data->length < data_len + auth_tag_len) + kunmap_atomic(auth_tag); + + if (sgl->length < assoc_len + data_len) + kunmap_atomic(data); + + kunmap_atomic(assoc); } /* @@ -803,34 +852,38 @@ static void put_request_buffer(u8 *data_buf, bool mapped) static int gcmaes_crypt(struct aead_request *req, unsigned int assoclen, u8 *hash_subkey, u8 *iv, void *aes_ctx, bool decrypt) { - u8 *src, *dst, *assoc, *bounce_buf = NULL; + u8 *src, *src_assoc; + u8 *dst, *dst_assoc; + u8 *auth_tag; + u8 *bounce_buf = NULL; bool src_mapped = false, dst_mapped = false; struct crypto_aead *tfm = crypto_aead_reqtfm(req); unsigned long auth_tag_len = crypto_aead_authsize(tfm); unsigned long data_len = req->cryptlen - (decrypt ? auth_tag_len : 0); int retval = 0; - unsigned long bounce_buf_size = data_len + auth_tag_len + req->assoclen; if (auth_tag_len > 16) return -EINVAL; - retval = get_request_buffer(req->src, bounce_buf_size, &assoc, - &bounce_buf, &src_mapped); + retval = get_request_buffers(req->src, req->assoclen, data_len, + auth_tag_len, &src_assoc, &src, + (decrypt || req->src == req->dst) + ? &auth_tag : NULL, + &bounce_buf, &src_mapped); if (retval) goto exit; - src = assoc + req->assoclen; - if (req->src == req->dst) { + dst_assoc = src_assoc; dst = src; dst_mapped = src_mapped; } else { - retval = get_request_buffer(req->dst, bounce_buf_size, &dst, - &bounce_buf, &dst_mapped); + retval = get_request_buffers(req->dst, req->assoclen, data_len, + auth_tag_len, &dst_assoc, &dst, + decrypt ? NULL : &auth_tag, + &bounce_buf, &dst_mapped); if (retval) goto exit; - - dst += req->assoclen; } if (!src_mapped) @@ -843,16 +896,16 @@ static int gcmaes_crypt(struct aead_request *req, unsigned int assoclen, u8 gen_auth_tag[16]; aesni_gcm_dec_tfm(aes_ctx, dst, src, data_len, iv, - hash_subkey, assoc, assoclen, + hash_subkey, src_assoc, assoclen, gen_auth_tag, auth_tag_len); /* Compare generated tag with passed in tag. */ - if (crypto_memneq(src + data_len, gen_auth_tag, auth_tag_len)) + if (crypto_memneq(auth_tag, gen_auth_tag, auth_tag_len)) retval = -EBADMSG; } else aesni_gcm_enc_tfm(aes_ctx, dst, src, data_len, iv, - hash_subkey, assoc, assoclen, - dst + data_len, auth_tag_len); + hash_subkey, src_assoc, assoclen, + auth_tag, auth_tag_len); kernel_fpu_end(); @@ -862,9 +915,13 @@ static int gcmaes_crypt(struct aead_request *req, unsigned int assoclen, 1); exit: if (req->dst != req->src) - put_request_buffer(dst - req->assoclen, dst_mapped); + put_request_buffers(req->dst, dst_mapped, dst_assoc, dst, + decrypt ? NULL : auth_tag, + req->assoclen, data_len, auth_tag_len); - put_request_buffer(assoc, src_mapped); + put_request_buffers(req->src, src_mapped, src_assoc, src, + (decrypt || req->src == req->dst) ? auth_tag : NULL, + req->assoclen, data_len, auth_tag_len); kfree(bounce_buf); return retval;