From patchwork Fri Feb 15 07:40:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 10814265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0E4013BF for ; Fri, 15 Feb 2019 07:40:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9662D2EB3A for ; Fri, 15 Feb 2019 07:40:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 945C12EB67; Fri, 15 Feb 2019 07:40:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC43B2EB3A for ; Fri, 15 Feb 2019 07:40:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390199AbfBOHkw (ORCPT ); Fri, 15 Feb 2019 02:40:52 -0500 Received: from mail-yb1-f196.google.com ([209.85.219.196]:36417 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728823AbfBOHkw (ORCPT ); Fri, 15 Feb 2019 02:40:52 -0500 Received: by mail-yb1-f196.google.com with SMTP id o129so1688510yba.3 for ; Thu, 14 Feb 2019 23:40:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id; bh=ni+axZl1T4E1qBoBWjhgC/gwhS+cSZafWpTRP99i67o=; b=Toxc+REi24cheuL/q80d7OfK+unsoQ2jkPrRHWZcF9yaUQ/z+MFrnYtvYEfnZpj2sA fGhtrng6QVG80gPPL7EkK4rAENGQsya0XGy744M3PhgToKwztUKrjmmcDWA4e5Xwn8aT STBljqd1Y/n9zVmkTq3aCKccjhjNNcsGkWt1Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=ni+axZl1T4E1qBoBWjhgC/gwhS+cSZafWpTRP99i67o=; b=cBsZHtRTgEQAbj6OrMzBuppGzC3b3L49KA9uqMhpmvWBQI7lfqWplHqwMvXj22MyfU gxGSuybJKYeBh+wN65YOLEDvK+Z+e193I7Kh+WuvCdha0CMbHSDwcrEX9BGsg/OLTyDW uJgiM62/PFkXDisVMuocZQs5tLQqoIzVozRrQtMPOszx7BtpF01jmfyTeixzAMcpa1ai zo63HO1Snh/scs/fQ9c7+4jCVm57nL7eA93uGcnoWj6+mgNzxEcScGsjg2tLVzKNsOLx 9HP6iYWM8hZfa2XCsv8yaX68HmdU+jXXyStsUv/k/cKMnv4kCz3XczmNV3rMHSEa9gvC vClQ== X-Gm-Message-State: AHQUAuaScAgaAH/ZK9qS8tYtjlygOtFOKAYUMA0XecQnch/lhdI87PaE Ov58ErfqmAFP0SrYeeSWjrqgI+5pnQNr4YyKDpNRdb/Ck3eYM5WCvqwx0z/nBVza0Dlg4PkyVSv AzCGcTERWYuNejvESoZmecH8WQvoRkPImpp555OZjyG04Li4zP4vXXl77hR4bkso9j7uZHzB/oG IXZFmCtKczjBTV/wnJKzn6 X-Google-Smtp-Source: AHgI3Ia6FnuF8qISESb3ZIkOWS4IYfXXf8dESHXPl4e0aZdi4WkIuIhreppds2BvyWDOpNsp/jWDzw== X-Received: by 2002:a25:770f:: with SMTP id s15mr6728777ybc.47.1550216450526; Thu, 14 Feb 2019 23:40:50 -0800 (PST) Received: from dhcp-10-123-20-72.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id c124sm1696388ywe.12.2019.02.14.23.40.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:40:49 -0800 (PST) From: Suganath Prabu To: linux-scsi@vger.kernel.org Cc: Sathya.Prakash@broadcom.com, sreekanth.reddy@broadcom.com, Suganath Prabu Subject: [v1 0/7] Irq poll to address cpu lockup. Date: Fri, 15 Feb 2019 02:40:23 -0500 Message-Id: <1550216430-36612-1-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 1.8.3.1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We have seen cpu lock up issue from fields if system has greater (more than 96) logical cpu count. SAS3.0 controller (Invader series) supports at max 96 msix vector and SAS3.5 product (Ventura) supports at max 128 msix vectors. This may be a generic issue (if PCI device support completion on multiple reply queues). Let me explain it w.r.t to mpt3sas supported h/w just to simplify the problem and possible changes to handle such issues. IT HBA (mpt3sas) supports multiple reply queues in completion path. Driver creates MSI-x vectors for controller as "min of (FW supported Reply queue, Logical CPUs)". If submitter is not interrupted via completion on same CPU, there is a loop in the IO path. This behavior can cause hard/soft CPU lockups, IO timeout, system sluggish etc. Example - one CPU (e.g. CPU A) is busy submitting the IOs and another CPU (e.g. CPU B) is busy with processing the corresponding IO's reply descriptors from reply descriptor queue upon receiving the interrupts from HBA. If the CPU A is continuously pumping the IOs then always CPU B (which is executing the ISR) will see the valid reply descriptors in the reply descriptor queue and it will be continuously processing those reply descriptor in a loop without quitting the ISR handler. Mpt3sas driver will exit ISR handler if it finds unused reply descriptor in the reply descriptor queue. Since CPU A will be continuously sending the IOs, CPU B may always see a valid reply descriptor (posted by HBA Firmware after processing the IO) in the reply descriptor queue. In worst case, driver will not quit from this loop in the ISR handler. Eventually, CPU lockup will be detected by watchdog. Above mentioned behavior is not common if "rq_affinity" set to 2 or affinity_hint is honored by irqbalance as "exact". If rq_affinity is set to 2, submitter will be always interrupted via completion on same CPU. If irqbalance is using "exact" policy, interrupt will be delivered to submitter CPU. Problem statement - If CPU counts to MSI-X vectors (reply descriptor Queues) count ratio is not 1:1, we still have exposure of issue explained above and for that we don't have any solution. Exposure of soft/hard lockup if CPU count is more than MSI-x supported by device. If CPUs count to MSI-x vectors count ratio is not 1:1, (Other way, if CPU counts to MSI-x vector count ratio is something like X:1, where X > 1) then 'exact' irqbalance policy OR rq_affinity = 2 won't help to avoid CPU hard/soft lockups. There won't be any one to one mapping between CPU to MSI-x vector instead one MSI-x interrupt (or reply descriptor queue) is shared with group/set of CPUs and there is a possibility of having a loop in the IO path within that CPU group and may observe lockups. For example: Consider a system having two NUMA nodes and each node having four logical CPUs and also consider that number of MSI-x vectors enabled on the HBA is two, then CPUs count to MSI-x vector count ratio as 4:1. e.g. MSIx vector 0 is affinity to CPU 0, CPU 1, CPU 2 & CPU 3 of NUMA node 0 and MSI-x vector 1 is affinity to CPU 4, CPU 5, CPU 6 & CPU 7 of NUMA node 1. numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 --> MSI-x 0 node 0 size: 65536 MB node 0 free: 63176 MB node 1 cpus: 4 5 6 7 -->MSI-x 1 node 1 size: 65536 MB node 1 free: 63176 MB Assume that user started an application which uses all the CPUs of NUMA node 0 for issuing the IOs. Only one CPU from affinity list (it can be any cpu since this behavior depends upon irqbalance) CPU0 will receive the interrupts from MSIx vector 0 for all the IOs. Eventually, CPU 0 IO submission percentage will be decreasing and ISR processing percentage will be increasing as it is more busy with processing the interrupts. Gradually IO submission percentage on CPU 0 will be zero and it's ISR processing percentage will be 100 percentage as IO loop has already formed within the NUMA node 0, i.e. CPU 1, CPU 2 & CPU 3 will be continuously busy with submitting the heavy IOs and only CPU 0 is busy in the ISR path as it always find the valid reply descriptor in the reply descriptor queue. Eventually, we will observe the hard lockup here. Chances of occurring of hard/soft lockups are directly proportional to value of X. If value of X is high, then chances of observing CPU lockups is high. Solution - Fix-1 ===== Use IRQ poll interface defined in " irq_poll.c". mpt3sas driver will execute ISR routine in Softirq context and it will always quit the loop based on budget provided in IRQ poll interface. In these scenarios( i.e. where CPUs count to MSI-X vectors count ratio is X:1 (where X > 1)), IRQ poll interface will avoid CPU hard lockups due to voluntary exit from the reply queue processing based on budget. Note - Only one MSI-x vector is busy doing processing. Irqstat output - IRQs / 1 second(s) IRQ# TOTAL NODE0 NODE1 NODE2 NODE3 NAME 44 122871 122871 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix0 45 0 0 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix1 Fix-2 ===== Driver should round robin the reply queue, so that each reply queue is load balanced. so that IO's are distributed to all the available reply descriptor post queues equally. With this each reply descriptor post queue load is balanced. This improves performance and also fixes soft lockups. Irqstat output after driver does reply queue load balance- Irqstat output - IRQs / 1 second(s) IRQ# TOTAL NODE0 NODE1 NODE2 NODE3 NAME 44 62871 62871 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix0 45 62718 62718 0 0 0 IR-PCI-MSI-edge mpt3sas0-msix1 In Summary, CPU completing IO which is not contributing to IO submission, may cause cpu lockup. If CPUs count to MSI-X vector count ratio is X:1 (where X > 1) then using irq poll interface, we can avoid the CPU lockups and by equally distributing the interrupts among the enabled MSI-x interrupts we can avoid performance issues. Patch 3 & 4 addresses Fix 1 and Fix 2 explained above, only if cpu count is more than FW supported MSI-x vector. Patch V1 changeset. Added patch 3 to add select irqpoll (Kconfig). Suganath Prabu (7): mpt3sas: Fix typo in request_desript_type. mpt3sas: simplify interrupt handler. mpt3sas: Select IRQ_POLL to avoid build error. mpt3sas: Irq poll to avoid CPU hard lockups. mpt3sas: Load balance to improve performance and avoid soft lockups. mpt3sas: Improve the threshold value and introduce module param. mpt3sas: Update mpt3sas driver version to 28.100.00.00 drivers/scsi/mpt3sas/Kconfig | 1 + drivers/scsi/mpt3sas/mpt3sas_base.c | 178 ++++++++++++++++++++++++++++++------ drivers/scsi/mpt3sas/mpt3sas_base.h | 22 ++++- 3 files changed, 170 insertions(+), 31 deletions(-)