From patchwork Wed Jun 8 19:48:12 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lin X-Patchwork-Id: 9165631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A33426088F for ; Wed, 8 Jun 2016 19:49:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 945E227248 for ; Wed, 8 Jun 2016 19:49:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 85DF428309; Wed, 8 Jun 2016 19:49:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F262E27248 for ; Wed, 8 Jun 2016 19:49:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755560AbcFHTtG (ORCPT ); Wed, 8 Jun 2016 15:49:06 -0400 Received: from mail.kernel.org ([198.145.29.136]:60164 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1425804AbcFHTtE (ORCPT ); Wed, 8 Jun 2016 15:49:04 -0400 Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id AAD13202FF; Wed, 8 Jun 2016 19:49:02 +0000 (UTC) Received: from localhost.localdomain (unknown [159.203.220.84]) (using TLSv1.2 with cipher AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 663442034E; Wed, 8 Jun 2016 19:49:01 +0000 (UTC) From: Ming Lin To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Christoph Hellwig , Keith Busch , Jens Axboe , James Smart Subject: [PATCH 2/2] nvme-rdma: check the number of hw queues mapped Date: Wed, 8 Jun 2016 15:48:12 -0400 Message-Id: <1465415292-9416-3-git-send-email-mlin@kernel.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1465415292-9416-1-git-send-email-mlin@kernel.org> References: <1465415292-9416-1-git-send-email-mlin@kernel.org> X-Virus-Scanned: ClamAV using ClamSMTP Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Ming Lin The connect_q requires all blk-mq hw queues being mapped to cpu sw queues. Otherwise, we got below crash. [42139.726531] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [42139.734962] IP: [] blk_mq_get_tag+0x65/0xb0 [42139.977715] Stack: [42139.980382] 0000000081306e9b ffff880035dbc380 ffff88006f71bbf8 ffffffff8130a016 [42139.988436] ffff880035dbc380 0000000000000000 0000000000000001 ffff88011887f000 [42139.996497] ffff88006f71bc50 ffffffff8130bc2a ffff880035dbc380 ffff880000000002 [42140.004560] Call Trace: [42140.007681] [] __blk_mq_alloc_request+0x16/0x200 [42140.014584] [] blk_mq_alloc_request_hctx+0x8a/0xd0 [42140.021662] [] nvme_alloc_request+0x2e/0xa0 [nvme_core] [42140.029171] [] __nvme_submit_sync_cmd+0x2c/0xc0 [nvme_core] [42140.037024] [] nvmf_connect_io_queue+0x10a/0x160 [nvme_fabrics] [42140.045228] [] nvme_rdma_connect_io_queues+0x35/0x50 [nvme_rdma] [42140.053517] [] nvme_rdma_create_ctrl+0x490/0x6f0 [nvme_rdma] [42140.061464] [] nvmf_dev_write+0x728/0x920 [nvme_fabrics] [42140.069072] [] __vfs_write+0x23/0x120 [42140.075049] [] ? apparmor_file_permission+0x13/0x20 [42140.082225] [] ? security_file_permission+0x38/0xc0 [42140.089391] [] ? rw_verify_area+0x44/0xb0 [42140.095706] [] vfs_write+0xad/0x1a0 [42140.101508] [] SyS_write+0x41/0xa0 [42140.107213] [] entry_SYSCALL_64_fastpath+0x1e/0xa8 Say, on a machine with 8 CPUs, we create 6 io queues, echo "transport=rdma,traddr=192.168.2.2,nqn=testiqn,nr_io_queues=6" \ > /dev/nvme-fabrics Then actually only 4 hw queues were mapped to CPU sw queues. HW Queue 1 <-> CPU 0,4 HW Queue 2 <-> CPU 1,5 HW Queue 3 <-> None HW Queue 4 <-> CPU 2,6 HW Queue 5 <-> CPU 3,7 HW Queue 6 <-> None So when connecting to IO queue 3, it will crash in blk_mq_get_tag() because hctx->tags is NULL. This patches doesn't really fix the hw/sw queues mapping, but it returns error if not all hw queues were mapped. "nvme nvme4: 6 hw queues created, but only 4 were mapped to sw queues" Reported-by: James Smart Signed-off-by: Ming Lin --- drivers/nvme/host/rdma.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 4edc912..2e8f556 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1771,6 +1771,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = { static int nvme_rdma_create_io_queues(struct nvme_rdma_ctrl *ctrl) { struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; + int hw_queue_mapped; int ret; ret = nvme_set_queue_count(&ctrl->ctrl, &opts->nr_io_queues); @@ -1819,6 +1820,16 @@ static int nvme_rdma_create_io_queues(struct nvme_rdma_ctrl *ctrl) goto out_free_tag_set; } + hw_queue_mapped = blk_mq_hctx_mapped(ctrl->ctrl.connect_q); + if (hw_queue_mapped < ctrl->ctrl.connect_q->nr_hw_queues) { + dev_err(ctrl->ctrl.device, + "%d hw queues created, but only %d were mapped to sw queues\n", + ctrl->ctrl.connect_q->nr_hw_queues, + hw_queue_mapped); + ret = -EINVAL; + goto out_cleanup_connect_q; + } + ret = nvme_rdma_connect_io_queues(ctrl); if (ret) goto out_cleanup_connect_q;