From patchwork Fri Feb 15 07:40:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1855B139A for ; Fri, 15 Feb 2019 07:40:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E39302EB69 for ; Fri, 15 Feb 2019 07:40:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E1FEA2EB7B; Fri, 15 Feb 2019 07:40:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 789EF2EB75 for ; Fri, 15 Feb 2019 07:40:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390208AbfBOHky (ORCPT ); Fri, 15 Feb 2019 02:40:54 -0500 Received: from mail-yw1-f65.google.com ([209.85.161.65]:36843 "EHLO mail-yw1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHky (ORCPT ); Fri, 15 Feb 2019 02:40:54 -0500 Received: by mail-yw1-f65.google.com with SMTP id 189so3377950ywi.3 for ; Thu, 14 Feb 2019 23:40:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=H/l1PG98U0MbiOKPtTCGueW/3XAsuGEJqyt11DU17L8=; b=HdrEAFDHf1yGEuCUY0XyWNEVEjFRVAcem2Y/+6UfFqo6AFvZ57fBR9YDt933GNW+Ma YQS1N5/4sTm2aIakUfxqCGZKOzh235vdk1t6rYzE0Oz96NXJsTqFjmUlqtT4nZbeA8Ui E+hl0Wf5v1IRp67mtPTandyjcp4tEKvchQvzg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=H/l1PG98U0MbiOKPtTCGueW/3XAsuGEJqyt11DU17L8=; b=l3xNHUzhYuF9BS8iaT2KzgKutns8MZ7rhoJ6vHpuQFMr/YotZ4Dp2PCyHF14jAYnzP FiKgWVo90zsFxuLFc9doAZsoAorhvs901IQkfZCotEqFzPqwM2nIDPm2ZxKDqDbz19Cz Mf4p7ISMxbvDhAeNsvLg6ekuE3ZqKSE03lxE6HfK1ecEO3HjNFEOxOgsdu2LWtBTjS3b SENQ2eq6LKjRqhEAoUz0CYSjRlBJY8hq2CBJ470+0yZ5Q6nihB15nGonG6aNfImFeD6f XkxI6o4swrmSwlSKSq7bvhgaeAUJd8i2UYKVoJnnGd4fZOq/mw1JEHUf9kIuPgLoj6eA UpXg== X-Gm-Message-State: AHQUAuZvy5HUvU2FMYh+xdEOq26glHiAuuUdi2irQ1DpBFRuPzeZOC1O 0KWIKkVL1H6wTMnYl4gR7RtQRj9oa65qSbLOtmOJ8dKop/4/B7CrxcgT1lG9ZdBvZrRgIVQe9d8 KvVJrKCoQqrkrQmPzsRiFUPxy9dI2vLhePeMBVn7I2L7EgvahUETLmBTCPUjNaQVwSQ+VmdsJkM QJztbvBzouNtTN+mXPfXOE X-Google-Smtp-Source: AHgI3Ib+UC0JDtym+hDegLwqwTgj2hYkwTedjeAxycaxwaxb8fthh/sOZNmYqzh+u9Be/UExra+YKw== X-Received: by 2002:a0d:e045:: with SMTP id j66mr6724306ywe.33.1550216453048; Thu, 14 Feb 2019 23:40:53 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.40.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:40:52 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 1/7] mpt3sas: Fix typo in request_desript_type. Date: Fri, 15 Feb 2019 02:40:24 -0500 Message-Id: <1550216430-36612-2-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Fixed typo in request_desript_type. request_desript_type --> request_descript_type. Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 0a6cb8f..4184d96 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -1395,7 +1395,7 @@ _base_interrupt(int irq, void *bus_id) struct adapter_reply_queue *reply_q = bus_id; union reply_descriptor rd; u32 completed_cmds; - u8 request_desript_type; + u8 request_descript_type; u16 smid; u8 cb_idx; u32 reply; @@ -1411,9 +1411,9 @@ _base_interrupt(int irq, void *bus_id) return IRQ_NONE; rpf = &reply_q->reply_post_free[reply_q->reply_post_host_index]; - request_desript_type = rpf->Default.ReplyFlags + request_descript_type = rpf->Default.ReplyFlags & MPI2_RPY_DESCRIPT_FLAGS_TYPE_MASK; - if (request_desript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) { + if (request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) { atomic_dec(&reply_q->busy); return IRQ_NONE; } @@ -1426,11 +1426,11 @@ _base_interrupt(int irq, void *bus_id) goto out; reply = 0; smid = le16_to_cpu(rpf->Default.DescriptorTypeDependent1); - if (request_desript_type == + if (request_descript_type == MPI25_RPY_DESCRIPT_FLAGS_FAST_PATH_SCSI_IO_SUCCESS || - request_desript_type == + request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_SCSI_IO_SUCCESS || - request_desript_type == + request_descript_type == MPI26_RPY_DESCRIPT_FLAGS_PCIE_ENCAPSULATED_SUCCESS) { cb_idx = _base_get_cb_idx(ioc, smid); if ((likely(cb_idx < MPT_MAX_CALLBACKS)) && @@ -1440,7 +1440,7 @@ _base_interrupt(int irq, void *bus_id) if (rc) mpt3sas_base_free_smid(ioc, smid); } - } else if (request_desript_type == + } else if (request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_ADDRESS_REPLY) { reply = le32_to_cpu( rpf->AddressReply.ReplyFrameAddress); @@ -1486,7 +1486,7 @@ _base_interrupt(int irq, void *bus_id) (reply_q->reply_post_host_index == (ioc->reply_post_queue_depth - 1)) ? 0 : reply_q->reply_post_host_index + 1; - request_desript_type = + request_descript_type = reply_q->reply_post_free[reply_q->reply_post_host_index]. Default.ReplyFlags & MPI2_RPY_DESCRIPT_FLAGS_TYPE_MASK; completed_cmds++; @@ -1509,7 +1509,7 @@ _base_interrupt(int irq, void *bus_id) } completed_cmds = 1; } - if (request_desript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) + if (request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) goto out; if (!reply_q->reply_post_host_index) rpf = reply_q->reply_post_free; From patchwork Fri Feb 15 07:40:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B2E313BF for ; Fri, 15 Feb 2019 07:41:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43F0A2EB13 for ; Fri, 15 Feb 2019 07:40:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 384862EB7C; Fri, 15 Feb 2019 07:40:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C06B22EB29 for ; Fri, 15 Feb 2019 07:40:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390213AbfBOHk5 (ORCPT ); Fri, 15 Feb 2019 02:40:57 -0500 Received: from mail-yw1-f67.google.com ([209.85.161.67]:37441 "EHLO mail-yw1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHk5 (ORCPT ); Fri, 15 Feb 2019 02:40:57 -0500 Received: by mail-yw1-f67.google.com with SMTP id k14so3373401ywe.4 for ; Thu, 14 Feb 2019 23:40:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LEd7pzhb3Qyzh2tAUBqKJkeSUbMBgDoqQjg3OAcFZKU=; b=MhS8to/j10uATxwcbgA8hUYYDKUd4sqfn7D3JsqgtAVG7Bv95Pl3ib82issB2i+NkR O2ZRzR1I2gd6HDhrVjPF3wJVuVMPp38GiiJqqZYSYp3aqQOrm0FY/8YkHMXKuL8UZtNu 9MZeoVc3mvQOiJAXC1ZiiFNqE2sV26T7QfmIE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LEd7pzhb3Qyzh2tAUBqKJkeSUbMBgDoqQjg3OAcFZKU=; b=qLyJF/4p+Q2dA/aj/xZpeKyKgrZaWzczH0b3C8o6DS78AWRKOSMmGc6yGiPGR6Y05U wa74x/1oJYltiNaNzRaPXDOVd4qWMiJCncbBsDnSYZIth2bJaAs5G8bepNd5j7wXkW/J qcnoGQ/Xlc6z7YafE5nqEUeRTXiHAQBYumVroy6RmzS04oENzGD+3/eBAQJ+uJgIN/4A xnvcYS8qD4ZfBsRIxwsX3vkjg0ShCa5MKE6y8/jmHuw0WrFE8qNwshCbCI81txmUGwEJ N8/xWJXWp5skwu6jhxwlpfhbF2DOI6eiWehogys+wXuxYLmn5oJN6UqbWUppWxNrbyTR QCdA== X-Gm-Message-State: AHQUAuZIbLYWgHFuJrFncWls/UAOUIoWCDlXn/vYjtjiejceijGae9DB NH6QMAVASy6vM+01OEXyZrpuXvy+jLEY87zl5D9fnUVKhHV3dxxfpfVwJL+llLo5H93FWWCrrpk ZFqphQuXuDoPGrduQL87Pm4+Zx2aA/TwufBxrV8W8WC6O8bPfKpe5WliY8sb3m3Kxn+m1C4QGZb wWvZr8mOHd0XyrZnIaMYEs X-Google-Smtp-Source: AHgI3IYtsZ0Rmd12BSwO4+ne3AWdW4diIwHYPnAzYMxPK/QE/qX1aFAMunce1+eSZgcyx9VrNdloOg== X-Received: by 2002:a81:71c5:: with SMTP id m188mr6787191ywc.353.1550216455965; Thu, 14 Feb 2019 23:40:55 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.40.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:40:55 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 2/7] mpt3sas: simplify interrupt handler. Date: Fri, 15 Feb 2019 02:40:25 -0500 Message-Id: <1550216430-36612-3-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Separate out processing of reply descriptor post queue from _base_interrupt to _base_process_reply_queue. Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.c | 49 +++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 4184d96..076428d 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -1383,16 +1383,16 @@ union reply_descriptor { }; /** - * _base_interrupt - MPT adapter (IOC) specific interrupt handler. - * @irq: irq number (not used) - * @bus_id: bus identifier cookie == pointer to MPT_ADAPTER structure + * _base_process_reply_queue - Process reply descriptors from reply + * descriptor post queue. + * @reply_q: per IRQ's reply queue object. * - * Return: IRQ_HANDLED if processed, else IRQ_NONE. + * Return: number of reply descriptors processed from reply + * descriptor queue. */ -static irqreturn_t -_base_interrupt(int irq, void *bus_id) +static int +_base_process_reply_queue(struct adapter_reply_queue *reply_q) { - struct adapter_reply_queue *reply_q = bus_id; union reply_descriptor rd; u32 completed_cmds; u8 request_descript_type; @@ -1404,21 +1404,18 @@ _base_interrupt(int irq, void *bus_id) Mpi2ReplyDescriptorsUnion_t *rpf; u8 rc; - if (ioc->mask_interrupts) - return IRQ_NONE; - + completed_cmds = 0; if (!atomic_add_unless(&reply_q->busy, 1, 1)) - return IRQ_NONE; + return completed_cmds; rpf = &reply_q->reply_post_free[reply_q->reply_post_host_index]; request_descript_type = rpf->Default.ReplyFlags & MPI2_RPY_DESCRIPT_FLAGS_TYPE_MASK; if (request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) { atomic_dec(&reply_q->busy); - return IRQ_NONE; + return completed_cmds; } - completed_cmds = 0; cb_idx = 0xFF; do { rd.word = le64_to_cpu(rpf->Words); @@ -1521,14 +1518,14 @@ _base_interrupt(int irq, void *bus_id) if (!completed_cmds) { atomic_dec(&reply_q->busy); - return IRQ_NONE; + return completed_cmds; } if (ioc->is_warpdrive) { writel(reply_q->reply_post_host_index, ioc->reply_post_host_index[msix_index]); atomic_dec(&reply_q->busy); - return IRQ_HANDLED; + return completed_cmds; } /* Update Reply Post Host Index. @@ -1555,7 +1552,27 @@ _base_interrupt(int irq, void *bus_id) MPI2_RPHI_MSIX_INDEX_SHIFT), &ioc->chip->ReplyPostHostIndex); atomic_dec(&reply_q->busy); - return IRQ_HANDLED; + return completed_cmds; +} + +/** + * _base_interrupt - MPT adapter (IOC) specific interrupt handler. + * @irq: irq number (not used) + * @bus_id: bus identifier cookie == pointer to MPT_ADAPTER structure + * + * Return: IRQ_HANDLED if processed, else IRQ_NONE. + */ +static irqreturn_t +_base_interrupt(int irq, void *bus_id) +{ + struct adapter_reply_queue *reply_q = bus_id; + struct MPT3SAS_ADAPTER *ioc = reply_q->ioc; + + if (ioc->mask_interrupts) + return IRQ_NONE; + + return ((_base_process_reply_queue(reply_q) > 0) ? + IRQ_HANDLED : IRQ_NONE); } /** From patchwork Fri Feb 15 07:40:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9CB1317E0 for ; Fri, 15 Feb 2019 07:41:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 70F432EB1F for ; Fri, 15 Feb 2019 07:41:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65DEC2EB7D; Fri, 15 Feb 2019 07:41:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E66A2EB1F for ; Fri, 15 Feb 2019 07:41:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390220AbfBOHk7 (ORCPT ); Fri, 15 Feb 2019 02:40:59 -0500 Received: from mail-yb1-f193.google.com ([209.85.219.193]:45007 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHk7 (ORCPT ); Fri, 15 Feb 2019 02:40:59 -0500 Received: by mail-yb1-f193.google.com with SMTP id j85so101590ybg.11 for ; Thu, 14 Feb 2019 23:40:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SvCEC2r9BdvIGQh/QEqw0Msp8zUsROgom5jm618oKEw=; b=CqJyWLGX2k4+sJIeo2I6hjdI67/ny6hYzUAeKjts6LCbWZ84sHK3kpbiDmcC3FJxS5 14skQetE0AMthDlXxlyv0p12JWuqKqZZ4lRyljtFQctVn1AI+WFAsdbjf4LLCoz/gLgP drChll40SJl+tpGQJW47gNhzm+bMnoRj8DREA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SvCEC2r9BdvIGQh/QEqw0Msp8zUsROgom5jm618oKEw=; b=awRt3vXGG1TX3mI0UHciY62TnwzNygWit4VKmkSV64pk2OxMntphD0S7q3Wgp4S36l rbWGBVL+fPuHQnxpymc52+reKZYd5B7oZ7xqq3DlKqiuYK1NQJKrb3U4GwK44qgPhFkK eJnCYzyisqJWurYtB5+XRBdiSMJBO3EPoZJsomDWVu5vf2o19G+4ep+iBIe9EkKLhGD9 81s7YQNhhjH5jjQ4E+RJA/ia3eD6Ttv95UlEWrq/W6EYl2XX2xCIpkCfcxnNeGdq4wMh Ew8S6VGJ7maZ1dt+/b6Fy5moWoHaNEXt6mrdkh3Yfbj2HTEhmG44oLCiilp+7PndZ/ad P+Ew== X-Gm-Message-State: AHQUAuY12QlTHdXsus1lQJdu9h+SESbdedjGJIjKydxkrcx6uKn2dDbT IYli1/pemy2Fx2LlRFDrXQo15gWitbmzIdORl4YP7Ntx7Gvhhtl/amxwV7TCyZKJm00LdymuVIK nrzF4u0NTVDAzAIN5aUmCFU7qW4nTGJ57kTmabTersbfkM+aPb8Wd3uSmq4AQrQBoWGCMhbURcN QHt+g0l/XItCG6FEad806S X-Google-Smtp-Source: AHgI3IaYOhcell+Zz/2yIUQSuRgQQzqerfEYUzX86UmUtNiiJrTiVBiGdTGL0+AmIF3HHDDEUTgIWw== X-Received: by 2002:a25:ae43:: with SMTP id g3mr6824201ybe.384.1550216458490; Thu, 14 Feb 2019 23:40:58 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.40.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:40:58 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 3/7] mpt3sas: Select IRQ_POLL to avoid build error. Date: Fri, 15 Feb 2019 02:40:26 -0500 Message-Id: <1550216430-36612-4-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In patch 4, driver uses irq poll, select it to avoid below build error. ERROR: "irq_poll_init" [drivers/scsi/mpt3sas/mpt3sas.ko] undefined! ERROR: "irq_poll_enable" [drivers/scsi/mpt3sas/mpt3sas.ko] undefined! ERROR: "irq_poll_sched" [drivers/scsi/mpt3sas/mpt3sas.ko] undefined! ERROR: "irq_poll_disable" [drivers/scsi/mpt3sas/mpt3sas.ko] undefined! ERROR: "irq_poll_complete" [drivers/scsi/mpt3sas/mpt3sas.ko] undefined! Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/scsi/mpt3sas/Kconfig b/drivers/scsi/mpt3sas/Kconfig index b736dbc..a072187 100644 --- a/drivers/scsi/mpt3sas/Kconfig +++ b/drivers/scsi/mpt3sas/Kconfig @@ -45,6 +45,7 @@ config SCSI_MPT3SAS depends on PCI && SCSI select SCSI_SAS_ATTRS select RAID_ATTRS + select IRQ_POLL ---help--- This driver supports PCI-Express SAS 12Gb/s Host Adapters. From patchwork Fri Feb 15 07:40:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814273 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF519139A for ; Fri, 15 Feb 2019 07:41:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C73B52EB16 for ; Fri, 15 Feb 2019 07:41:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9C1552EB32; Fri, 15 Feb 2019 07:41:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 477EC2EB32 for ; Fri, 15 Feb 2019 07:41:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390228AbfBOHlD (ORCPT ); Fri, 15 Feb 2019 02:41:03 -0500 Received: from mail-yw1-f68.google.com ([209.85.161.68]:35194 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHlC (ORCPT ); Fri, 15 Feb 2019 02:41:02 -0500 Received: by mail-yw1-f68.google.com with SMTP id s204so3378895ywg.2 for ; Thu, 14 Feb 2019 23:41:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=T35PGQ+wRdgSJv96ODUtux7/oHYkThy3VTp+8fxkHhA=; b=gLS+g2JxD4Et/nxPgrPZnH0Hn1nP0M+Q0NhX50qxafdOGpqDaejhIa+lkcxlOMJ9m2 DXPT44jbkoviw7UBkjB0ROGAGYCjCG4uwtJipvh7MKzulZAaqydMjHIDF3Ux/vhskTaL bH/Ki7SD0y5s1ijpiuLROmaagEUf0BdTtRBoY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=T35PGQ+wRdgSJv96ODUtux7/oHYkThy3VTp+8fxkHhA=; b=NCyHknzOWfrOFvB/MB0dCBYdcXmya8UNe6zHa5Vx1VXDHjbO49JT3DqZMW13NecuOb fcyGR8zmZ/o4nDtB3QvdSbeSUnpHEQ4fPJ9mekCQPW1CssxrFYv6WYBoCMhIrqIPkXjx WviTIhpKTII+vPF9e42TauHAzEB46Sj0CCCDwcmfAmTp7eCQx6778pcVExII+tM7ekxi uCZBVVNMTHYLXsgNZL0YwUbmuje9SDgnlmQ/2KpkK6ONVwSmbVpWg48dj3uoidpnIZ0b Hf2CKcLw8G+riflmjQ4jqnZUYJ+ShQbAl6CzaLAyOgAK0v7opOlU1BNbHWe8ZJeQDU3+ 6c8Q== X-Gm-Message-State: AHQUAuZWf/fou8LgolsV0lkw8W4kA9MkjjlgIRl9elL2haJZxQujl1sE MCyfjC71Wam2zG7OugyvXIyV8+cMjhyP5bzOCRD7PW/r14Xoxyn8r6lUTOaotNpY2tE1xpW+26o VurHjP0ZVRktx4w0ry2hpcRnsXpAiiZscEP0fM+zbNO83SzfRCwQ1N1z+nvGFcqfzPDnT1g6PDB MDL7v6YaLQ0Swh1n1SulSD X-Google-Smtp-Source: AHgI3IZqQ2kAYKtrP6qGyVRehE9t8xGa+F7iScqjVQ+KpaGpo9k8BJQ5Bm3QcWv2FCoQPcsjMdFnew== X-Received: by 2002:a81:7a46:: with SMTP id v67mr6915476ywc.4.1550216461008; Thu, 14 Feb 2019 23:41:01 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.40.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:41:00 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 4/7] mpt3sas: Irq poll to avoid CPU hard lockups. Date: Fri, 15 Feb 2019 02:40:27 -0500 Message-Id: <1550216430-36612-5-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Issue Discription: We have seen cpu lock up issue from fields if system has greater (more than 96) logical cpu count. SAS3.0 controller (Invader series) supports at max 96 msix vector and SAS3.5 product (Ventura) supports at max 128 msix vectors. This may be a generic issue (if PCI device support completion on multiple reply queues). Let me explain it w.r.t to mpt3sas supported h/w just to simplify the problem and possible changes to handle such issues. IT HBA (mpt3sas) supports multiple reply queues in completion path. Driver creates MSI-x vectors for controller as "min of (FW supported Reply queue, Logical CPUs)". If submitter is not interrupted via completion on same CPU, there is a loop in the IO path. This behavior can cause hard/soft CPU lockups, IO timeout, system sluggish etc. Example - one CPU (e.g. CPU A) is busy submitting the IOs and another CPU (e.g. CPU B) is busy with processing the corresponding IO's reply descriptors from reply descriptor queue upon receiving the interrupts from HBA. If the CPU A is continuously pumping the IOs then always CPU B (which is executing the ISR) will see the valid reply descriptors in the reply descriptor queue and it will be continuously processing those reply descriptor in a loop without quitting the ISR handler. Mpt3sas driver will exit ISR handler if it finds unused reply descriptor in the reply descriptor queue. Since CPU A will be continuously sending the IOs, CPU B may always see a valid reply descriptor (posted by HBA Firmware after processing the IO) in the reply descriptor queue. In worst case, driver will not quit from this loop in the ISR handler. Eventually, CPU lockup will be detected by watchdog. Above mentioned behavior is not common if "rq_affinity" set to 2 or affinity_hint is honored by irqbalance as "exact". If rq_affinity is set to 2, submitter will be always interrupted via completion on same CPU. If irqbalance is using "exact" policy, interrupt will be delivered to submitter CPU. Problem statement - If CPU counts to MSI-X vectors (reply descriptor Queues) count ratio is not 1:1, we still have exposure of issue explained above and for that we don't have any solution. Exposure of soft/hard lockup if CPU count is more than MSI-x supported by device. If CPUs count to MSI-x vectors count ratio is not 1:1, (Other way, if CPU counts to MSI-x vector count ratio is something like X:1, where X > 1) then 'exact' irqbalance policy OR rq_affinity = 2 won't help to avoid CPU hard/soft lockups. There won't be any one to one mapping between CPU to MSI-x vector instead one MSI-x interrupt (or reply descriptor queue) is shared with group/set of CPUs and there is a possibility of having a loop in the IO path within that CPU group and may observe lockups. For example: Consider a system having two NUMA nodes and each node having four logical CPUs and also consider that number of MSI-x vectors enabled on the HBA is two, then CPUs count to MSI-x vector count ratio as 4:1. e.g. MSIx vector 0 is affinity to CPU 0, CPU 1, CPU 2 & CPU 3 of NUMA node 0 and MSI-x vector 1 is affinity to CPU 4, CPU 5, CPU 6 & CPU 7 of NUMA node 1. numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 --> MSI-x 0 node 0 size: 65536 MB node 0 free: 63176 MB node 1 cpus: 4 5 6 7 -->MSI-x 1 node 1 size: 65536 MB node 1 free: 63176 MB Assume that user started an application which uses all the CPUs of NUMA node 0 for issuing the IOs. Only one CPU from affinity list (it can be any cpu since this behavior depends upon irqbalance) CPU0 will receive the interrupts from MSIx vector 0 for all the IOs. Eventually, CPU 0 IO submission percentage will be decreasing and ISR processing percentage will be increasing as it is more busy with processing the interrupts. Gradually IO submission percentage on CPU 0 will be zero and it's ISR processing percentage will be 100 percentage as IO loop has already formed within the NUMA node 0, i.e. CPU 1, CPU 2 & CPU 3 will be continuously busy with submitting the heavy IOs and only CPU 0 is busy in the ISR path as it always find the valid reply descriptor in the reply descriptor queue. Eventually, we will observe the hard lockup here. Chances of occurring of hard/soft lockups are directly proportional to value of X. If value of X is high, then chances of observing CPU lockups is high. Solution - Fix - Use IRQ poll interface defined in " irq_poll.c". mpt3sas driver will execute ISR routine in Softirq context and it will always quit the loop based on budget provided in IRQ poll interface. In these scenarios( i.e. where CPUs count to MSI-X vectors count ratio is X:1 (where X > 1)), IRQ poll interface will avoid CPU hard lockups due to voluntary exit from the reply queue processing based on budget. Note - Only one MSI-x vector is busy doing processing. Irqstat output - IRQs / 1 second(s) IRQ# TOTAL NODE0 NODE1 NODE2 NODE3 NAME 44 122871 122871 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix0 45 0 0 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix1 We have used the fix only if cpu count is more than FW supported MSI-x vector Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.c | 78 ++++++++++++++++++++++++++++++++++++- drivers/scsi/mpt3sas/mpt3sas_base.h | 8 ++++ 2 files changed, 84 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 076428d..1f358bc 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -1504,7 +1504,12 @@ _base_process_reply_queue(struct adapter_reply_queue *reply_q) MPI2_RPHI_MSIX_INDEX_SHIFT), &ioc->chip->ReplyPostHostIndex); } - completed_cmds = 1; + if (!reply_q->irq_poll_scheduled) { + reply_q->irq_poll_scheduled = true; + irq_poll_sched(&reply_q->irqpoll); + } + atomic_dec(&reply_q->busy); + return completed_cmds; } if (request_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) goto out; @@ -1570,12 +1575,67 @@ _base_interrupt(int irq, void *bus_id) if (ioc->mask_interrupts) return IRQ_NONE; - + if (reply_q->irq_poll_scheduled) + return IRQ_HANDLED; return ((_base_process_reply_queue(reply_q) > 0) ? IRQ_HANDLED : IRQ_NONE); } /** + * _base_irqpoll - IRQ poll callback handler + * @irqpoll - irq_poll object + * @budget - irq poll weight + * + * returns number of reply descriptors processed + */ +static int +_base_irqpoll(struct irq_poll *irqpoll, int budget) +{ + struct adapter_reply_queue *reply_q; + int num_entries = 0; + + reply_q = container_of(irqpoll, struct adapter_reply_queue, + irqpoll); + if (reply_q->irq_line_enable) { + disable_irq(reply_q->os_irq); + reply_q->irq_line_enable = false; + } + num_entries = _base_process_reply_queue(reply_q); + if (num_entries < budget) { + irq_poll_complete(irqpoll); + reply_q->irq_poll_scheduled = false; + reply_q->irq_line_enable = true; + enable_irq(reply_q->os_irq); + } + + return num_entries; +} + +/** + * _base_init_irqpolls - initliaze IRQ polls + * @ioc: per adapter object + * + * returns nothing + */ +static void +_base_init_irqpolls(struct MPT3SAS_ADAPTER *ioc) +{ + struct adapter_reply_queue *reply_q, *next; + + if (list_empty(&ioc->reply_queue_list)) + return; + + list_for_each_entry_safe(reply_q, next, &ioc->reply_queue_list, list) { + irq_poll_init(&reply_q->irqpoll, + ioc->hba_queue_depth/4, _base_irqpoll); + reply_q->irq_poll_scheduled = false; + reply_q->irq_line_enable = true; + reply_q->os_irq = pci_irq_vector(ioc->pdev, + reply_q->msix_index); + } +} + +/** * _base_is_controller_msix_enabled - is controller support muli-reply queues * @ioc: per adapter object * @@ -1613,6 +1673,17 @@ mpt3sas_base_sync_reply_irqs(struct MPT3SAS_ADAPTER *ioc) /* TMs are on msix_index == 0 */ if (reply_q->msix_index == 0) continue; + if (reply_q->irq_poll_scheduled) { + /* Calling irq_poll_disable will wait for any pending + * callbacks to have completed. + */ + irq_poll_disable(&reply_q->irqpoll); + irq_poll_enable(&reply_q->irqpoll); + reply_q->irq_poll_scheduled = false; + reply_q->irq_line_enable = true; + enable_irq(reply_q->os_irq); + continue; + } synchronize_irq(pci_irq_vector(ioc->pdev, reply_q->msix_index)); } } @@ -3032,6 +3103,8 @@ mpt3sas_base_map_resources(struct MPT3SAS_ADAPTER *ioc) if (r) goto out_fail; + if (!ioc->is_driver_loading) + _base_init_irqpolls(ioc); /* Use the Combined reply queue feature only for SAS3 C0 & higher * revision HBAs and also only when reply queue count is greater than 8 */ @@ -6514,6 +6587,7 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) if (r) goto out_free_resources; + _base_init_irqpolls(ioc); init_waitqueue_head(&ioc->reset_wq); /* allocate memory pd handle bitmask list */ diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index 19158cb..fb572cd 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -67,6 +67,7 @@ #include #include #include +#include #include "mpt3sas_debug.h" #include "mpt3sas_trigger_diag.h" @@ -882,6 +883,9 @@ struct _event_ack_list { * @reply_post_free: reply post base virt address * @name: the name registered to request_irq() * @busy: isr is actively processing replies on another cpu + * @os_irq: irq number + * @irqpoll: irq_poll object + * @irq_poll_scheduled: Tells whether irq poll is scheduled or not * @list: this list */ struct adapter_reply_queue { @@ -891,6 +895,10 @@ struct adapter_reply_queue { Mpi2ReplyDescriptorsUnion_t *reply_post_free; char name[MPT_NAME_LENGTH]; atomic_t busy; + u32 os_irq; + struct irq_poll irqpoll; + bool irq_poll_scheduled; + bool irq_line_enable; struct list_head list; }; From patchwork Fri Feb 15 07:40:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814275 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2023717E0 for ; Fri, 15 Feb 2019 07:41:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBCED2EB32 for ; Fri, 15 Feb 2019 07:41:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA4202EB88; Fri, 15 Feb 2019 07:41:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5AD7A2EB8C for ; Fri, 15 Feb 2019 07:41:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390238AbfBOHlF (ORCPT ); Fri, 15 Feb 2019 02:41:05 -0500 Received: from mail-yb1-f193.google.com ([209.85.219.193]:32860 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHlE (ORCPT ); Fri, 15 Feb 2019 02:41:04 -0500 Received: by mail-yb1-f193.google.com with SMTP id w2so3466534ybi.0 for ; Thu, 14 Feb 2019 23:41:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dy+e6oQ7CSinT9QlwhS+fuQA9alIwcaS7YY0JzYje+Q=; b=R8YI4zoHBv/2nTe+2CmxMExoEhLx3owA2FSfVEGBZpIEKpNwQj1ydy6JfAzQ5I5FMP 5K6sCCv/3lotvak4IPMXeOci5qdmAhXImSataQjwyCBFYSLYO5FZnwWgLp8kUzCUlkRf M6YzuZdeiEjS9kaSS1s3VtMKCGqjyzSjjYxGg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dy+e6oQ7CSinT9QlwhS+fuQA9alIwcaS7YY0JzYje+Q=; b=IkXDFBIUIZkXavxw5xlpo8FseoJiuj9oUvF0v6OuuSHmYXgNtEU+Jru1tU8qkMkGKq 8xELAYly2MLbfBqWtQzgZ7pKaGKKyZG8RpQXF0OhIDuMuzUMC7hWFhplpAqImcY/iVBp vw4uEFqBn5VxR8+92z5ZkcUkTvgcKfV48fmwArRiDPVvqbhqOhFZoK8gccTN57IK65l4 JzNr2OxNNzRxsb10aO5ZApIKEqiPH3d2v9/x3jwyS1zKnKQSxLb+hmk/TV2bxIKAdWaU RTzUGYPMq7vgSqgzU2HyZObydyHXLvjQllrAfS21uJIk2PEIUE+kbtCTRKJdVvI6X70f SoLw== X-Gm-Message-State: AHQUAuY8SO0xmVepTgXdy4g52WU2RVjnMNKicgPydSC88OGoOn/dGop+ pJEccmjzmFdmhXPQXuzXFdx9mSYurqSuR2NxjrxfTly4UIua27VU7/MtFV6YGhtd0YonIw+zls2 T4lWA74Pw4A9q/zpT8m8shzdgpEgaZOh6oHLTE9M3crx0J+HSd9+xHwBA1ly/YwKR1t6TWyyNH3 7BLVzSGlp0wMS8lENwKkeI X-Google-Smtp-Source: AHgI3Ia+uCnVII8PJE4zKwk5dCSZgYe+CHixRtpRSN2adu4353UA/tpiqX84q4PkLDnNnT55/IqsJQ== X-Received: by 2002:a5b:501:: with SMTP id o1mr6684091ybp.85.1550216463833; Thu, 14 Feb 2019 23:41:03 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.41.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:41:03 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 5/7] mpt3sas: Load balance to improve performance and avoid soft lockups. Date: Fri, 15 Feb 2019 02:40:28 -0500 Message-Id: <1550216430-36612-6-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Driver uses "reply descriptor post queues" in round robin fashion so that IO's are distributed to all the available reply descriptor post queues equally. With this each reply descriptor post queue load is balanced. This is enabled only if CPUs count to MSI-X vector count ratio is X:1 (where X > 1) This improves performance and also fixes soft lockups. Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.c | 21 +++++++++++++++++++++ drivers/scsi/mpt3sas/mpt3sas_base.h | 5 +++++ 2 files changed, 26 insertions(+) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 1f358bc..aadd9e2 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -1382,6 +1382,16 @@ union reply_descriptor { } u; }; +static u32 base_mod64(u64 dividend, u32 divisor) +{ + u32 remainder; + + if (!divisor) + pr_err("mpt3sas: DIVISOR is zero, in div fn\n"); + remainder = do_div(dividend, divisor); + return remainder; +} + /** * _base_process_reply_queue - Process reply descriptors from reply * descriptor post queue. @@ -2845,6 +2855,11 @@ _base_assign_reply_queues(struct MPT3SAS_ADAPTER *ioc) if (!_base_is_controller_msix_enabled(ioc)) return; + ioc->msix_load_balance = false; + if (ioc->reply_queue_count < num_online_cpus()) { + ioc->msix_load_balance = true; + return; + } memset(ioc->cpu_msix_table, 0, ioc->cpu_msix_table_sz); @@ -3248,6 +3263,12 @@ mpt3sas_base_get_reply_virt_addr(struct MPT3SAS_ADAPTER *ioc, u32 phys_addr) static inline u8 _base_get_msix_index(struct MPT3SAS_ADAPTER *ioc) { + /* Enables reply_queue load balancing */ + if (ioc->msix_load_balance) + return ioc->reply_queue_count ? + base_mod64(atomic64_add_return(1, + &ioc->total_io_cnt), ioc->reply_queue_count) : 0; + return ioc->cpu_msix_table[raw_smp_processor_id()]; } diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index fb572cd..3895407 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -1024,6 +1024,9 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc); * @msix_vector_count: number msix vectors * @cpu_msix_table: table for mapping cpus to msix index * @cpu_msix_table_sz: table size + * @total_io_cnt: Gives total IO count, used to load balance the interrupts + * @msix_load_balance: Enables load balancing of interrupts across + * the multiple MSIXs * @schedule_dead_ioc_flush_running_cmds: callback to flush pending commands * @scsi_io_cb_idx: shost generated commands * @tm_cb_idx: task management commands @@ -1200,6 +1203,8 @@ struct MPT3SAS_ADAPTER { u32 ioc_reset_count; MPT3SAS_FLUSH_RUNNING_CMDS schedule_dead_ioc_flush_running_cmds; u32 non_operational_loop; + atomic64_t total_io_cnt; + bool msix_load_balance; /* internal commands, callback index */ u8 scsi_io_cb_idx; From patchwork Fri Feb 15 07:40:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814277 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A44CD17E9 for ; Fri, 15 Feb 2019 07:41:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72B7A2EB0C for ; Fri, 15 Feb 2019 07:41:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 713822EB93; Fri, 15 Feb 2019 07:41:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B1A92EB0C for ; Fri, 15 Feb 2019 07:41:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390244AbfBOHlH (ORCPT ); Fri, 15 Feb 2019 02:41:07 -0500 Received: from mail-yb1-f194.google.com ([209.85.219.194]:45012 "EHLO mail-yb1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHlH (ORCPT ); Fri, 15 Feb 2019 02:41:07 -0500 Received: by mail-yb1-f194.google.com with SMTP id j85so101676ybg.11 for ; Thu, 14 Feb 2019 23:41:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=T41ztuycnt/Lyvj02Hp0CQi6itCWV3yTIAhY51tzQNw=; b=iDb0FYrHUNYArwyXURiafZjgQgVl75RCwjRJ3/GzO9g8j5OGOIBIOyy02v3aUZ1zpf XzBDAgRSopaPV22f3niqCkowW0W87S2CPZ5yJ865SiYwsGz/MKunm8TwEEpFfkbHR7Tk /XrYMEnIIXQ0uYLuB5CMYzCINJJ1I2VNaIrMo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=T41ztuycnt/Lyvj02Hp0CQi6itCWV3yTIAhY51tzQNw=; b=VOrIaSrNe5NHLvJjyeU1UVzUoF5el44tWRjve4XuHWKDMFP5mp3BF9LaipSAXAQKXo eReqEu1zyY0KRgQ+wB2mxI4P7wbtb9R7v4Sb+MSYEt57zSdApJuSZAvXawDAfob+rd33 CvlhXGrLw0pg9gfUvmZsgbg6VPxUneMTFPaGtRZtigEL53lZbeox3Vb+0iY2K+ayFFEv Wr6OmPu9el4fDlRt+O/4Hd/4xwRvh/TVs11AcZpvnBp+QCOLRIQ7IPbUtmkldHT+vjAJ YII9rsOIHSilCEqJ6btWwosauQigcHOcyng80ZEequQNWPfQwrGW1ea68Kvy1zPT1Rp3 /RbQ== X-Gm-Message-State: AHQUAuarf3LxikXwgLiLX8/VGUDofZrAcmM0hchsevS3nzMwtYwPVtoB TjPyKYYDx7WaZUl82rRyQY7biTg0QEUQeAsVLsEmU+n/YdNUdMlT9+OsC1rtktcMR0IWTUfGlPp wo7g3IdUQ6SVD9RRfwrL90AkImWb5YwIr7bM1z33uRBD0n7NQRtqVBWY0yLI952Z1GOcCnRyUav KEnB3GJ9jXS6jRUcRL67M0 X-Google-Smtp-Source: AHgI3IaT4MPrD0l3fK+smDMZp6WD5ok/E1YMR4KOeD5jhhs/od4tRJT7nA2VnKmWsOyiyhLS0SJvIA== X-Received: by 2002:a25:4f8a:: with SMTP id d132mr6807801ybb.370.1550216466351; Thu, 14 Feb 2019 23:41:06 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.41.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:41:05 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 6/7] mpt3sas: Improve the threshold value and introduce module param. Date: Fri, 15 Feb 2019 02:40:29 -0500 Message-Id: <1550216430-36612-7-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP * Reduce the threshold value to 1/4 of the queue depth. * With this FW can find enough entries to post the Reply Descriptors in the reply descriptor post queue. * With module param, user can play with threshold value, the same irqpoll_weight is used as the budget in processing of reply descriptor post queues in _base_process_reply_queue. Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.c | 14 ++++++++++++-- drivers/scsi/mpt3sas/mpt3sas_base.h | 3 +++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index aadd9e2..880bd22 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -94,6 +94,11 @@ module_param(max_msix_vectors, int, 0); MODULE_PARM_DESC(max_msix_vectors, " max msix vectors"); +static int irqpoll_weight = -1; +module_param(irqpoll_weight, int, 0); +MODULE_PARM_DESC(irqpoll_weight, + "irq poll weight (default= one fourth of HBA queue depth)"); + static int mpt3sas_fwfault_debug; MODULE_PARM_DESC(mpt3sas_fwfault_debug, " enable detection of firmware fault and halt firmware - (default=0)"); @@ -1404,7 +1409,7 @@ static int _base_process_reply_queue(struct adapter_reply_queue *reply_q) { union reply_descriptor rd; - u32 completed_cmds; + u64 completed_cmds; u8 request_descript_type; u16 smid; u8 cb_idx; @@ -1502,7 +1507,7 @@ _base_process_reply_queue(struct adapter_reply_queue *reply_q) * So that FW can find enough entries to post the Reply * Descriptors in the reply descriptor post queue. */ - if (completed_cmds > ioc->hba_queue_depth/3) { + if (!base_mod64(completed_cmds, ioc->thresh_hold)) { if (ioc->combined_reply_queue) { writel(reply_q->reply_post_host_index | ((msix_index & 7) << @@ -6608,6 +6613,11 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) if (r) goto out_free_resources; + if (irqpoll_weight > 0) + ioc->thresh_hold = irqpoll_weight; + else + ioc->thresh_hold = ioc->hba_queue_depth/4; + _base_init_irqpolls(ioc); init_waitqueue_head(&ioc->reset_wq); diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index 3895407..dab64f3 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -1028,6 +1028,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc); * @msix_load_balance: Enables load balancing of interrupts across * the multiple MSIXs * @schedule_dead_ioc_flush_running_cmds: callback to flush pending commands + * @thresh_hold: Max number of reply descriptors processed + * before updating Host Index * @scsi_io_cb_idx: shost generated commands * @tm_cb_idx: task management commands * @scsih_cb_idx: scsih internal commands @@ -1205,6 +1207,7 @@ struct MPT3SAS_ADAPTER { u32 non_operational_loop; atomic64_t total_io_cnt; bool msix_load_balance; + u16 thresh_hold; /* internal commands, callback index */ u8 scsi_io_cb_idx; From patchwork Fri Feb 15 07:40:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8C04139A for ; Fri, 15 Feb 2019 07:41:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC8072EB5E for ; Fri, 15 Feb 2019 07:41:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA7752EAFE; Fri, 15 Feb 2019 07:41:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B3DC2EB95 for ; Fri, 15 Feb 2019 07:41:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390248AbfBOHlK (ORCPT ); Fri, 15 Feb 2019 02:41:10 -0500 Received: from mail-yw1-f43.google.com ([209.85.161.43]:42324 "EHLO mail-yw1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHlJ (ORCPT ); Fri, 15 Feb 2019 02:41:09 -0500 Received: by mail-yw1-f43.google.com with SMTP id v201so3361878ywa.9 for ; Thu, 14 Feb 2019 23:41:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=18Ct0AMMAlksY/9mfDQX7szYwJalKF83DPthr/i+A2s=; b=Ewb2ppDCfNSiAnyJNkesdxx1edTD9gyiEFCA0oe5f8WheDBY4HeqeEcJ2854LOeW2W 7rByOdPXwZ2HZe/zlNvWn/yy/y5DytRP6ZbkzI9XaHiK2BQUQPgMn/CeKH4WM9QkTGnL zefldMND4ozqNnGRnp5l6yV3LKTu3zJGFQhQo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=18Ct0AMMAlksY/9mfDQX7szYwJalKF83DPthr/i+A2s=; b=SVJnZ8M2lsEVLZr0c/iw+zLeoWA0D9d8FUDLp0uoijuvmojuwX/tN0IiNJV0q5ViDc vnMSB7gETOPju7jPqSnSs5ivf3+Ct7Se78xjStl4Zi1T/z7bvSKE31vyaDgaX5Amzy7R Ci+y8ujTo+owQCeUtpFnSQEgVArv2LtjYgPc5grZAuuzB2GmiP/BJSQmQS2d9KIzq4Nx tbMknj+1fsJm37AetjqIo8DcwLYkSnRbivWhBTLoH8VxJSxnL6GD1uCnrQnNBzIizWef NBiUyRRTye3Jn698geLEtuaV0TWAPSIPDlaP0Yatf64cvtLdll3K1oRaPTYkjKhRvi/+ 7Zkw== X-Gm-Message-State: AHQUAuZ31pf7UxdfzQ14tRrtPTL/OnhKyOwN2PG5SyF/bEinhca/wg3X Dgdc17nu/TE77Fdh5uAyRz0O+7Cq1BzjqMYchaBhD6mvQtR8z/kbOmfPCw//EF2aHhoNcaTHmwg Tudxtow81S5pJb6F3i3eGunTV+OZHv4gkxYMdc26IM1p5Da/L2ZlUnoI1BZUVnr4gw+Jrui+zi2 rp3u7XCJMG1sKW2pE1NVqN X-Google-Smtp-Source: AHgI3IYLoQ6BzYRRPkbLHlQSIqia9jn77fZpsyTPhsj91tlkJ/hr4pqhNp9GN8Ku7VSenQCMWcIl+w== X-Received: by 2002:a81:9958:: with SMTP id q85mr6775316ywg.390.1550216468817; Thu, 14 Feb 2019 23:41:08 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.41.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:41:08 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 7/7] mpt3sas: Update mpt3sas driver version to 28.100.00.00 Date: Fri, 15 Feb 2019 02:40:30 -0500 Message-Id: <1550216430-36612-8-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Updated driver version to 28.100.00.00, which is equivalent to OOB Ph9 Signed-off-by: Suganath Prabu --- drivers/scsi/mpt3sas/mpt3sas_base.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index dab64f3..480219f 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -76,9 +76,9 @@ #define MPT3SAS_DRIVER_NAME "mpt3sas" #define MPT3SAS_AUTHOR "Avago Technologies " #define MPT3SAS_DESCRIPTION "LSI MPT Fusion SAS 3.0 Device Driver" -#define MPT3SAS_DRIVER_VERSION "27.102.00.00" -#define MPT3SAS_MAJOR_VERSION 27 -#define MPT3SAS_MINOR_VERSION 102 +#define MPT3SAS_DRIVER_VERSION "28.100.00.00" +#define MPT3SAS_MAJOR_VERSION 28 +#define MPT3SAS_MINOR_VERSION 100 #define MPT3SAS_BUILD_VERSION 0 #define MPT3SAS_RELEASE_VERSION 00