From patchwork Sun Oct 29 16:38:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Idan Burstein X-Patchwork-Id: 10031563 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 401DB602B5 for ; Sun, 29 Oct 2017 16:38:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D97C205FC for ; Sun, 29 Oct 2017 16:38:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3264328650; Sun, 29 Oct 2017 16:38:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9D25205FC for ; Sun, 29 Oct 2017 16:38:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750959AbdJ2Qif (ORCPT ); Sun, 29 Oct 2017 12:38:35 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37548 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751036AbdJ2Qie (ORCPT ); Sun, 29 Oct 2017 12:38:34 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from idanb@mellanox.com) with ESMTPS (AES256-SHA encrypted); 29 Oct 2017 18:38:29 +0200 Received: from vnc5.mtl.labs.mlnx (vnc5.mtl.labs.mlnx [10.7.2.5]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id v9TGcQvd022558; Sun, 29 Oct 2017 18:38:29 +0200 From: idanb@mellanox.com To: leon@kernel.org, linux-nvme@lists.infradead.org, hch@lst.de, sagi@grimberg.me, linux-rdma@vger.kernel.org Cc: maxg@mellanox.com, idanb@mellanox.com Subject: [PATCH 2/2] nvme-rdma: Add remote_invalidation module parameter Date: Sun, 29 Oct 2017 18:38:21 +0200 Message-Id: <1509295101-14081-3-git-send-email-idanb@mellanox.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1509295101-14081-1-git-send-email-idanb@mellanox.com> References: <1509295101-14081-1-git-send-email-idanb@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Idan Burstein NVMe over Fabrics in its secure "register_always" mode registers and invalidates the user buffer upon each IO. The protocol enables the host to request the susbsystem to use SEND WITH INVALIDATE operation while returning the response capsule and invalidate the local key (remote_invalidation). In some HW implementations, the local network adapter may perform better while using local invalidation operations. The results below show that running with local invalidation rather then remote invalidation improve the iops one could achieve by using the ConnectX-5Ex network adapter by x1.36 factor. Nevertheless, using local invalidation induce more CPU overhead than enabling the target to invalidate remotly, therefore, because there is a CPU% vs IOPs limit tradeoff we propose to have module parameter to define whether to request remote invalidation or not. The following results were taken against a single nvme over fabrics subsystem with a single namespace backed by null_blk: Block Size s/g reg_wr inline reg_wr inline reg_wr + local inv ++++++++++++ ++++++++++++++ ++++++++++++++++ +++++++++++++++++++++++++++ 512B 1446.6K/8.57% 5224.2K/76.21% 7143.3K/79.72% 1KB 1390.6K/8.5% 4656.7K/71.69% 5860.6K/55.45% 2KB 1343.8K/8.6% 3410.3K/38.96% 4106.7K/55.82% 4KB 1254.8K/8.39% 2033.6K/15.86% 2165.3K/17.48% 8KB 1079.5K/7.7% 1143.1K/7.27% 1158.2K/7.33% 16KB 603.4K/3.64% 593.8K/3.4% 588.9K/3.77% 32KB 294.8K/2.04% 293.7K/1.98% 294.4K/2.93% 64KB 138.2K/1.32% 141.6K/1.26% 135.6K/1.34% Signed-off-by: Max Gurtovoy Signed-off-by: Idan Burstein --- drivers/nvme/host/rdma.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 92a03ff..7f8225d 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -146,6 +146,11 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl) MODULE_PARM_DESC(register_always, "Use memory registration even for contiguous memory regions"); +static bool remote_invalidation = true; +module_param(remote_invalidation, bool, 0444); +MODULE_PARM_DESC(remote_invalidation, + "request remote invalidation from subsystem (default: true)"); + static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id, struct rdma_cm_event *event); static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc); @@ -1152,8 +1157,9 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue, sg->addr = cpu_to_le64(req->mr->iova); put_unaligned_le24(req->mr->length, sg->length); put_unaligned_le32(req->mr->rkey, sg->key); - sg->type = (NVME_KEY_SGL_FMT_DATA_DESC << 4) | - NVME_SGL_FMT_INVALIDATE; + sg->type = NVME_KEY_SGL_FMT_DATA_DESC << 4; + if (remote_invalidation) + sg->type |= NVME_SGL_FMT_INVALIDATE; return 0; }