From patchwork Mon Jun 19 09:22:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lino Sanfilippo X-Patchwork-Id: 13284228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E01DEB64DA for ; Mon, 19 Jun 2023 09:23:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231362AbjFSJXb (ORCPT ); Mon, 19 Jun 2023 05:23:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231386AbjFSJXT (ORCPT ); Mon, 19 Jun 2023 05:23:19 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63242130; Mon, 19 Jun 2023 02:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1687166558; x=1687771358; i=linosanfilippo@gmx.de; bh=YZh32p7Z0wnP31W9pdXwjnNl16gtCtu5z5U3zFyYmD0=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date; b=A2Ey5YQF1enTJU7sN7EMKgATH91rxZfwyOzE7vWkks0lERnePVGACZCA738C0/CkBs5lbBU q3Q10wB5kOsYd/qrHS0dckVo1t7vqMzmkP/ufjMJU6n8QOtpc2dQtEi7ZPSQIGUmU1rH9p19q hLQKgocE7ACzaAh+W3b1x6oEHTzs7zPL0Le8SxuJ01RkKzqBB0G95pMmHgjHxPElWPWTyiqNp o1qVSrya6hbbelphu7s332DBTn4uesNselmUbG0ohSrBUEQBKeck2zERQ0RuVlKQUqCbH6TQj Js1Q5Vb/e6WwHO3PtIabupQZxpodz2WVzIKOLCvJzEi4O6UoI8Mg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from Venus.speedport.ip ([84.162.2.106]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MLR1f-1qTfY50WmZ-00IVzb; Mon, 19 Jun 2023 11:22:38 +0200 From: Lino Sanfilippo To: peterhuewe@gmx.de, jarkko@kernel.org, jgg@ziepe.ca Cc: jsnitsel@redhat.com, hdegoede@redhat.com, oe-lkp@lists.linux.dev, lkp@intel.com, peter.ujfalusi@linux.intel.com, peterz@infradead.org, linux@mniewoehner.de, linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org, l.sanfilippo@kunbus.com, LinoSanfilippo@gmx.de, lukas@wunner.de, p.rosenberger@kunbus.com, kernel test robot Subject: [PATCH 3] tpm,tpm_tis: Disable interrupts after 1000 unhandled IRQs Date: Mon, 19 Jun 2023 11:22:19 +0200 Message-Id: <20230619092219.2600-1-LinoSanfilippo@gmx.de> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Provags-ID: V03:K1:taDtb8rWEaK48/aZ8g6C1/tF9k7JOfj7aFkWy7a6ZgDeQU/b5YC kidobZabla0nZJUvzmho1oKNMX4TDfrywG+qHTTmpKG1ZDC2EMkJsgA8faQ/9FTZ1+/W1W1 NvFh2Ug5Rzxwy26Vt6HgD76Z4Yp1HQdTC5dmCYVTIMwLfHdQYutaVBbtAl0yhm+npE1/sYR jVHVRAgd4zksWWoM92ikw== UI-OutboundReport: notjunk:1;M01:P0:qnDdBP6gpZM=;8RKMauwupN5cVf5Utd1kkg1WIx9 M9kkr1INUUcgiKO06kMNoPksemB+i58S+8GfRG2D8dbIIIlfumcbABeIVFSCHYQ+4fy6q0c9B MmLBJtnrZXPvG02xXGhrI7D6rNyimqovXNVT0G2QfB1yevLEnDqtgCTE6/bkozktdwtSnBAVj osK1OBWDRlrt87E/YZuRlS5hi0tVfj9eIbkv9eeSp2QYrzpHM5EyZeAoHjZ8e0rD4aA9qy93+ w+nZqBTSWb7AX5J2jTGELZVfdtQSgMr5kcbIWGESlZjuMJKSn67D2ry9AppFPLxwXcdmasRRV GVH1bTfsyLBKHW+DXEE3RiuhmJjwsNpEub96uQyPxwEf2N4GBxQp138Tr8temUCSuecksQaPw iueLG8LfCOTh6AnMdO0w7X9z4RkQEo3sKQMjqs4kAYwDBkWrABKlwaZDUttkazmkfEaYW27Hx TjinTENX0Coo6nzZ3cTTY9TQsIPCR3iJ1k7C2P/AoRB2ENZU2Oz32TZHkIZ3AmkUKM4UbphWb EtrF/ngcpiR1+B+VAJXf0eXBcMZfBTTaY24x4xRqq0nCjDnvxJyA042xMqg5j0JvkjFqxEd9I MEo1G6YC6bVt5C4Holo9UWhY4gWsWemKQsV/Vx6CiRXB66KLrPuJJJ35W8wgbuY5izxtD5Rhe vvQC1WmFz7EBy9ik7EXnu0jOW5jQAf73dOzwbAtN9iDCxC4FC1quw1VN7UhQk7yLOWI0XvJ+m bk55t3fNBtqjE6ZCcibsdm7hV1wxJeNj4T1Qhr5w68VCyu8gILlAZnFll42H4QwEqXzgVfNC+ tmo0V1UM2h+s/iACjWgIN7xMtGpWu+fi9/ylqfv4xopQew92kaMhu2pyVMYPvDdSXa4Ap33Cs SU6g5NnciTlWxFsXide2YQ08H/bIwK6R+rMWbqPbd3c7a+gVD2Gjkec7e0fx1C0E0/8E9YWGA FAXKcemle6p6oa4nsRzCGyMYNbc= Precedence: bulk List-ID: X-Mailing-List: linux-integrity@vger.kernel.org From: Lino Sanfilippo After activation of interrupts for TPM TIS drivers 0-day reports an interrupt storm on an Inspur NF5180M6 server. Fix this by detecting the storm and falling back to polling: Count the number of unhandled interrupts within a 10 ms time interval. In case that more than 1000 were unhandled deactivate interrupts entirely, deregister the handler and use polling instead. Also print a note to point to the tpm_tis_dmi_table. Since the interrupt deregistration function devm_free_irq() waits for all interrupt handlers to finish, only trigger a worker in the interrupt handler and do the unregistration in the worker to avoid a deadlock. Note: the storm detection logic equals the implementation in note_interrupt() which uses timestamps and counters stored in struct irq_desc. Since this structure is private to the generic interrupt core the TPM TIS core uses its own timestamps and counters. Furthermore the TPM interrupt handler always returns IRQ_HANDLED to prevent the generic interrupt core from processing the interrupt storm. Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305041325.ae8b0c43-yujie.liu@intel.com/ Suggested-by: Lukas Wunner Signed-off-by: Lino Sanfilippo --- drivers/char/tpm/tpm_tis_core.c | 117 ++++++++++++++++++++++++++++---- drivers/char/tpm/tpm_tis_core.h | 4 ++ 2 files changed, 106 insertions(+), 15 deletions(-) Changes to v2: - use define for max number of unhandles irqs(requested by Jarko) - rename intmask to int_mask (requested by Jarko) - rephrased short summary (requested by Jarko) - rename disable_interrupts to tpm_tis_disable_interrupts (requested by Jarko) - print info message concerning adding an entry to tpm_tis_dmi_table (suggested by Jerry) - amended commit message - handle failure of locality request by returning IRQ_NONE - dont take and release locality in __tpm_tis_disable_interrupts but in its caller base-commit: 45a3e24f65e90a047bef86f927ebdc4c710edaa1 diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c index 558144fa707a..d42537b985c5 100644 --- a/drivers/char/tpm/tpm_tis_core.c +++ b/drivers/char/tpm/tpm_tis_core.c @@ -24,9 +24,12 @@ #include #include #include +#include #include "tpm.h" #include "tpm_tis_core.h" +#define TPM_TIS_MAX_UNHANDLED_IRQS 1000 + static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value); static bool wait_for_tpm_stat_cond(struct tpm_chip *chip, u8 mask, @@ -468,25 +471,29 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len) return rc; } -static void disable_interrupts(struct tpm_chip *chip) +static void __tpm_tis_disable_interrupts(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + u32 int_mask = 0; + + tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &int_mask); + int_mask &= ~TPM_GLOBAL_INT_ENABLE; + tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), int_mask); + + chip->flags &= ~TPM_CHIP_FLAG_IRQ; +} + +static void tpm_tis_disable_interrupts(struct tpm_chip *chip) { struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); - u32 intmask; - int rc; if (priv->irq == 0) return; - rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); - if (rc < 0) - intmask = 0; - - intmask &= ~TPM_GLOBAL_INT_ENABLE; - rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + __tpm_tis_disable_interrupts(chip); devm_free_irq(chip->dev.parent, priv->irq, chip); priv->irq = 0; - chip->flags &= ~TPM_CHIP_FLAG_IRQ; } /* @@ -552,7 +559,7 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) if (!test_bit(TPM_TIS_IRQ_TESTED, &priv->flags)) tpm_msleep(1); if (!test_bit(TPM_TIS_IRQ_TESTED, &priv->flags)) - disable_interrupts(chip); + tpm_tis_disable_interrupts(chip); set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); return rc; } @@ -752,6 +759,71 @@ static bool tpm_tis_req_canceled(struct tpm_chip *chip, u8 status) return status == TPM_STS_COMMAND_READY; } +static irqreturn_t tpm_tis_reenable_polling(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + const char *product; + const char *vendor; + + dev_warn(&chip->dev, FW_BUG + "TPM interrupt storm detected, polling instead\n"); + + vendor = dmi_get_system_info(DMI_SYS_VENDOR); + product = dmi_get_system_info(DMI_PRODUCT_VERSION); + + if (vendor && product) { + dev_info(&chip->dev, + "Consider adding the following entry to tpm_tis_dmi_table:\n"); + dev_info(&chip->dev, "\tDMI_SYS_VENDOR: %s\n", vendor); + dev_info(&chip->dev, "\tDMI_PRODUCT_VERSION: %s\n", product); + } + + if (tpm_tis_request_locality(chip, 0) != 0) + return IRQ_NONE; + + __tpm_tis_disable_interrupts(chip); + tpm_tis_relinquish_locality(chip, 0); + + /* + * devm_free_irq() must not be called from within the interrupt handler, + * since this function waits for running handlers to finish and thus it + * would deadlock. Instead trigger a worker that takes care of the + * unregistration. + */ + schedule_work(&priv->free_irq_work); + + return IRQ_HANDLED; +} + +static irqreturn_t tpm_tis_check_for_interrupt_storm(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + irqreturn_t irqret = IRQ_HANDLED; + + /* + * The worker to free the TPM interrupt (free_irq_work) may already + * be scheduled, so make sure it is not scheduled again. + */ + if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) + return IRQ_HANDLED; + + if (time_after(jiffies, priv->last_unhandled_irq + HZ/10)) + priv->unhandled_irqs = 1; + else + priv->unhandled_irqs++; + + priv->last_unhandled_irq = jiffies; + + if (priv->unhandled_irqs > TPM_TIS_MAX_UNHANDLED_IRQS) + irqret = tpm_tis_reenable_polling(chip); + + /* + * Prevent the genirq code from starting its own interrupt storm + * handling by always reporting that the interrupt was handled. + */ + return irqret; +} + static irqreturn_t tis_int_handler(int dummy, void *dev_id) { struct tpm_chip *chip = dev_id; @@ -761,10 +833,10 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); if (rc < 0) - return IRQ_NONE; + goto unhandled; if (interrupt == 0) - return IRQ_NONE; + goto unhandled; set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); if (interrupt & TPM_INTF_DATA_AVAIL_INT) @@ -780,10 +852,13 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_write32(priv, TPM_INT_STATUS(priv->locality), interrupt); tpm_tis_relinquish_locality(chip, 0); if (rc < 0) - return IRQ_NONE; + goto unhandled; tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); return IRQ_HANDLED; + +unhandled: + return tpm_tis_check_for_interrupt_storm(chip); } static void tpm_tis_gen_interrupt(struct tpm_chip *chip) @@ -804,6 +879,15 @@ static void tpm_tis_gen_interrupt(struct tpm_chip *chip) chip->flags &= ~TPM_CHIP_FLAG_IRQ; } +static void tpm_tis_free_irq_func(struct work_struct *work) +{ + struct tpm_tis_data *priv = container_of(work, typeof(*priv), free_irq_work); + struct tpm_chip *chip = priv->chip; + + devm_free_irq(chip->dev.parent, priv->irq, chip); + priv->irq = 0; +} + /* Register the IRQ and issue a command that will cause an interrupt. If an * irq is seen then leave the chip setup for IRQ operation, otherwise reverse * everything and leave in polling mode. Returns 0 on success. @@ -816,6 +900,7 @@ static int tpm_tis_probe_irq_single(struct tpm_chip *chip, u32 intmask, int rc; u32 int_status; + INIT_WORK(&priv->free_irq_work, tpm_tis_free_irq_func); rc = devm_request_threaded_irq(chip->dev.parent, irq, NULL, tis_int_handler, IRQF_ONESHOT | flags, @@ -918,6 +1003,7 @@ void tpm_tis_remove(struct tpm_chip *chip) interrupt = 0; tpm_tis_write32(priv, reg, ~TPM_GLOBAL_INT_ENABLE & interrupt); + flush_work(&priv->free_irq_work); tpm_tis_clkrun_enable(chip, false); @@ -1021,6 +1107,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, chip->timeout_b = msecs_to_jiffies(TIS_TIMEOUT_B_MAX); chip->timeout_c = msecs_to_jiffies(TIS_TIMEOUT_C_MAX); chip->timeout_d = msecs_to_jiffies(TIS_TIMEOUT_D_MAX); + priv->chip = chip; priv->timeout_min = TPM_TIMEOUT_USECS_MIN; priv->timeout_max = TPM_TIMEOUT_USECS_MAX; priv->phy_ops = phy_ops; @@ -1179,7 +1266,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, rc = tpm_tis_request_locality(chip, 0); if (rc < 0) goto out_err; - disable_interrupts(chip); + tpm_tis_disable_interrupts(chip); tpm_tis_relinquish_locality(chip, 0); } } diff --git a/drivers/char/tpm/tpm_tis_core.h b/drivers/char/tpm/tpm_tis_core.h index 610bfadb6acf..b1a169d7d1ca 100644 --- a/drivers/char/tpm/tpm_tis_core.h +++ b/drivers/char/tpm/tpm_tis_core.h @@ -91,11 +91,15 @@ enum tpm_tis_flags { }; struct tpm_tis_data { + struct tpm_chip *chip; u16 manufacturer_id; struct mutex locality_count_mutex; unsigned int locality_count; int locality; int irq; + struct work_struct free_irq_work; + unsigned long last_unhandled_irq; + unsigned int unhandled_irqs; unsigned int int_mask; unsigned long flags; void __iomem *ilb_base_addr;