From patchwork Thu Jan 18 09:58:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nicholas A. Bellinger" X-Patchwork-Id: 10172717 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EFE9260230 for ; Thu, 18 Jan 2018 09:58:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB6AA2624A for ; Thu, 18 Jan 2018 09:58:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CFC2A2625B; Thu, 18 Jan 2018 09:58:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B9DF2624A for ; Thu, 18 Jan 2018 09:58:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754881AbeARJ6r (ORCPT ); Thu, 18 Jan 2018 04:58:47 -0500 Received: from mail.linux-iscsi.org ([67.23.28.174]:60234 "EHLO linux-iscsi.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754815AbeARJ6o (ORCPT ); Thu, 18 Jan 2018 04:58:44 -0500 Received: from [192.168.1.66] (75-37-194-224.lightspeed.lsatca.sbcglobal.net [75.37.194.224]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: nab) by linux-iscsi.org (Postfix) with ESMTPSA id 8D8A4174001; Thu, 18 Jan 2018 10:12:24 +0000 (UTC) Message-ID: <1516269522.24576.274.camel@haakon3.daterainc.com> Subject: Re: SQ overflow seen running isert traffic with high block sizes From: "Nicholas A. Bellinger" To: Shiraz Saleem Cc: "Kalderon, Michal" , "Amrani, Ram" , Sagi Grimberg , "linux-rdma@vger.kernel.org" , "Elior, Ariel" , target-devel , Potnuri Bharat Teja Date: Thu, 18 Jan 2018 01:58:42 -0800 In-Reply-To: <20180115152236.GA15484@ssaleem-MOBL4.amr.corp.intel.com> References: <210f538f-8c98-f480-64fc-f2124ed6d01b@grimberg.me> <1499970579.7987.8.camel@haakon3.daterainc.com> <20171006224025.GA23364@ssaleem-MOBL4.amr.corp.intel.com> <1515992195.24576.156.camel@haakon3.daterainc.com> <20180115152236.GA15484@ssaleem-MOBL4.amr.corp.intel.com> X-Mailer: Evolution 3.4.4-1 Mime-Version: 1.0 Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Shiraz, Michal & Co, Thanks for the feedback. Comments below. On Mon, 2018-01-15 at 09:22 -0600, Shiraz Saleem wrote: > On Mon, Jan 15, 2018 at 03:12:36AM -0700, Kalderon, Michal wrote: > > > From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma- > > > owner@vger.kernel.org] On Behalf Of Nicholas A. Bellinger > > > Sent: Monday, January 15, 2018 6:57 AM > > > To: Shiraz Saleem > > > Cc: Amrani, Ram ; Sagi Grimberg > > > ; linux-rdma@vger.kernel.org; Elior, Ariel > > > ; target-devel ; > > > Potnuri Bharat Teja > > > Subject: Re: SQ overflow seen running isert traffic with high block sizes > > > > > > Hi Shiraz, Ram, Ariel, & Potnuri, > > > > > > Following up on this old thread, as it relates to Potnuri's recent fix for a iser- > > > target queue-full memory leak: > > > > > > https://www.spinics.net/lists/target-devel/msg16282.html > > > > > > Just curious how frequent this happens in practice with sustained large block > > > workloads, as it appears to effect at least three different iwarp RNICS (i40iw, > > > qedr and iw_cxgb4)..? > > > > > > Is there anything else from an iser-target consumer level that should be > > > changed for iwarp to avoid repeated ib_post_send() failures..? > > > > > Would like to mention, that although we are an iWARP RNIC as well, we've hit this > > Issue when running RoCE. It's not iWARP related. > > This is easily reproduced within seconds with IO size of 5121K > > Using 5 Targets with 2 Ram Disk each and 5 targets with FileIO Disks each. > > > > IO Command used: > > maim -b512k -T32 -t2 -Q8 -M0 -o -u -n -m17 -ftargets.dat -d1 > > > > thanks, > > Michal > > Its seen with block size >= 2M on a single target 1 RAM disk config. And similar to Michals report; > rather quickly, in a matter of seconds. > > fio --rw=read --bs=2048k --numjobs=1 --iodepth=128 --runtime=30 --size=20g --loops=1 --ioengine=libaio > --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --exitall --filename=/dev/sdb --name=sdb > A couple of thoughts. First, would it be helpful to limit maximum payload size per I/O for consumers based on number of iser-target sq hw sges..? That is, if rdma_rw_ctx_post() -> ib_post_send() failures are related to maximum payload size per I/O being too large there is an existing target_core_fabric_ops mechanism for limiting using SCSI residuals, originally utilized by qla2xxx here: target/qla2xxx: Honor max_data_sg_nents I/O transfer limit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8f9b565482c537821588444e09ff732c7d65ed6e Note this patch also will return a smaller Block Limits VPD (0x86) MAXIMUM TRANSFER LENGTH based on max_data_sg_nents * PAGE_SIZE, which means for modern SCSI initiators honoring MAXIMUM TRANSFER LENGTH will automatically limit maximum outgoing payload transfer length, and avoid SCSI residual logic. As-is, iser-target doesn't a propagate max_data_sg_ents limit into iscsi-target, but you can try testing with a smaller value to see if it's useful. Eg: Second, if the failures are not SCSI transfer length specific, another option would be to limit the total command sequence number depth (CmdSN) per session. This is controlled at runtime by default_cmdsn_depth TPG attribute: /sys/kernel/config/target/iscsi/$TARGET_IQN/$TPG/attrib/default_cmdsn_depth and on per initiator context with cmdsn_depth NodeACL attribute: /sys/kernel/config/target/iscsi/$TARGET_IQN/$TPG/acls/$ACL_IQN/cmdsn_depth Note these default to 64, and can be changed at build time via include/target/iscsi/iscsi_target_core.h:TA_DEFAULT_CMDSN_DEPTH. That said, Sagi, any further comments as what else iser-target should be doing to avoid repeated queue-fulls with limited hw sges..? --- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configf index 0ebc481..d8a4cc5 100644 --- a/drivers/target/iscsi/iscsi_target_configfs.c +++ b/drivers/target/iscsi/iscsi_target_configfs.c @@ -1553,6 +1553,7 @@ static void lio_release_cmd(struct se_cmd *se_cmd) .module = THIS_MODULE, .name = "iscsi", .node_acl_size = sizeof(struct iscsi_node_acl), + .max_data_sg_nents = 32, /* 32 * PAGE_SIZE = MAXIMUM TRANSFER LENGTH */ .get_fabric_name = iscsi_get_fabric_name, .tpg_get_wwn = lio_tpg_get_endpoint_wwn, .tpg_get_tag = lio_tpg_get_tag,