From patchwork Tue May 30 17:47:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lino Sanfilippo X-Patchwork-Id: 13260825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA169C77B73 for ; Tue, 30 May 2023 17:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230207AbjE3RsQ (ORCPT ); Tue, 30 May 2023 13:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230096AbjE3RsP (ORCPT ); Tue, 30 May 2023 13:48:15 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D9FEB2; Tue, 30 May 2023 10:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1685468859; i=linosanfilippo@gmx.de; bh=++AWfGg+huXGq6oS7nTH1/C1B8ZsQI4CHiv2Tr/rOQg=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date; b=MGxErC8KrqcS+ussukmTT8GRSHHm8QIlE8GxT/J8r9/rKabfKPqivojQEbULZ4W7a NCcOQoRDEszWdQK8oD732maLD80BRFbzBm0Cx9ASqMZFV9ShV3KNFFoWNsgFHAY0Ll Q8ZNdnlIUd6u2z7Yub2nxZzvwZJOXnk6C3x/fOMXo/vRdsoqzrIjP2wZPhIPnDYT3B LIjSC8KDFWov2wHm+uT7SIDFtRgJx1akgokJw/j0ZEf053S//reF8AkI6L0FXAlkeR dv67AQ2PQUKgvBDiIOh62Qv3pE10FDIYLeoADl64J+J5y9wWbuWZqaU1MwtQ1lnKeo svULC9Zhi10rQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from Venus.speedport.ip ([84.162.2.106]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MTzf6-1pd8en0aeY-00QyBO; Tue, 30 May 2023 19:47:39 +0200 From: Lino Sanfilippo To: peterhuewe@gmx.de, jarkko@kernel.org, jgg@ziepe.ca Cc: jsnitsel@redhat.com, hdegoede@redhat.com, oe-lkp@lists.linux.dev, lkp@intel.com, peter.ujfalusi@linux.intel.com, peterz@infradead.org, linux@mniewoehner.de, linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org, l.sanfilippo@kunbus.com, LinoSanfilippo@gmx.de, lukas@wunner.de, p.rosenberger@kunbus.com, kernel test robot Subject: [PATCH v2] tpm,tpm_tis: Handle interrupt storm Date: Tue, 30 May 2023 19:47:12 +0200 Message-Id: <20230530174712.6989-1-LinoSanfilippo@gmx.de> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Provags-ID: V03:K1:6I6uOiEXoQ914p4DH3/mKhxPboZliuI1/kHOD8HviEPdYFnOAIF UImaGD1j11GryfRWdZ9+Cz5WKBy6G74kFm0cavF+jNcx/UKjELl+kJcgv03hbaNN3z/LiJu lQRUxHLN9htBKIoOfaMlWHsqV12ESz++5btTXWEad1aD1J7eiEBpIh2aOyrGtieOpRV3eig 8XfHAvahfI04oX6muDz6w== UI-OutboundReport: notjunk:1;M01:P0:5MoqWh4wBzk=;I1siaBsEOSZY6/rpiK4svW0k1sK VjKtY0yzx/hI22+Xjt26Wq1ggeYJr0rt0P+fU15X40YbBWydESaU8Y4Vp6F/20DlNR99VM0vO SOgu5opqh6gYBioTPCSU8ANYdveQHFhHWdsekq9ZhSuTu/EfD0N5sK/bfy1Enhqt22PhOteY0 bE/frup9hqkuHW17eSsF4xygm34phxf1AttLzjk9Sy1vAewOy7kkBqkoPY4hzeQ2LE1Tix5S+ itS0lhGk0fiMoZWt/JrsQnmcD+dItbcBUYj5P2yGsTii1Yo3xfERFpkrVQWKVKZVvlLMMVReX RsWVFdyiWl621A3ZqCp8jNb/eU69Ntg4drNUFK25kMzhkiVx+lnv+od+/3uGHUV4OzxXqOwEk xxbXI6M4pmCQo094CKSATqQ5Y7OD2lbxOB2xVPKWxp30M0O3SbckvvT2rDjnR4cUoOyHkOAwG sDgqLIPUxTrlciVZaBErPSpxYdGqxW4PRWsle3XE1UhYZ5MAm7Ujx5eEaZ8/QIUXoj0AGSIbO LVnyqWXW03cf+QMRSJInB/MuJ9JOtV57CpDo000IpbeLXH4k7unwY0Ik7YTLi7lf6H/eZxFTN jTk3DVviiVLHNykkPacQLPTm7iZGUUZ0ZBWAy+43dAwklxDGbGRrEG5U//mmkRswXv2ULkCUw t5mIieBwlw/+Xq435js0C6sU5kpQK42Y0P/AzVOHyZ+ZCBYARhdFZzlKOOJSfoxK7gAib/gQq ripoOLSTT0QhvnqTb3U8f+h8Y5KpsWODXi5yDcFWVFlSsh2jRQW8SL5JDO69WFDHc936exGGU aZMIi6xlA2blzEMDFEQnAMTy/icdD/RaMQYh1EY16KIIDFCi+tRDwfW1KsKfWg7dmmFXQwTBz VNNDrEFUA0RSCfpXFvWcNTsyRjXxCuu0AfYQ1aoUJ9+d6utj5xhojnOkwSsmuE20eSPBeR235 1S4nsX9F2zQU2TYcoFYKYhzBqbw= Precedence: bulk List-ID: X-Mailing-List: linux-integrity@vger.kernel.org From: Lino Sanfilippo After activation of interrupts for TPM TIS drivers 0-day reports an interrupt storm on an Inspur NF5180M6/NF5180M6 server. Fix this by detecting the storm and falling back to polling: Count the number of unhandled interrupts within a 10 ms time interval. In case that more than 1000 were unhandled deactivate interrupts entirely, deregister the handler and use polling instead. The storm detection logic equals the implementation in note_interrupt() which uses timestamps and counters stored in struct irq_desc. Since this structure is private to the generic interrupt core the TPM TIS core uses its own timestamps and counters. Furthermore the TPM interrupt handler always returns IRQ_HANDLED to prevent the generic interrupt core from processing the interrupt storm. Since the interrupt deregistration function devm_free_irq() waits for all interrupt handlers to finish, only trigger a worker in the interrupt handler and do the unregistration in the worker to avoid a deadlock. Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305041325.ae8b0c43-yujie.liu@intel.com/ Suggested-by: Lukas Wunner Signed-off-by: Lino Sanfilippo --- drivers/char/tpm/tpm_tis_core.c | 93 ++++++++++++++++++++++++++++----- drivers/char/tpm/tpm_tis_core.h | 4 ++ 2 files changed, 85 insertions(+), 12 deletions(-) base-commit: 7877cb91f1081754a1487c144d85dc0d2e2e7fc4 diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c index 558144fa707a..7ae8228e803f 100644 --- a/drivers/char/tpm/tpm_tis_core.c +++ b/drivers/char/tpm/tpm_tis_core.c @@ -468,25 +468,32 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len) return rc; } +static void __tpm_tis_disable_interrupts(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + u32 intmask = 0; + + tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); + intmask &= ~TPM_GLOBAL_INT_ENABLE; + + tpm_tis_request_locality(chip, 0); + tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + tpm_tis_relinquish_locality(chip, 0); + + chip->flags &= ~TPM_CHIP_FLAG_IRQ; +} + static void disable_interrupts(struct tpm_chip *chip) { struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); - u32 intmask; - int rc; if (priv->irq == 0) return; - rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); - if (rc < 0) - intmask = 0; - - intmask &= ~TPM_GLOBAL_INT_ENABLE; - rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + __tpm_tis_disable_interrupts(chip); devm_free_irq(chip->dev.parent, priv->irq, chip); priv->irq = 0; - chip->flags &= ~TPM_CHIP_FLAG_IRQ; } /* @@ -752,6 +759,53 @@ static bool tpm_tis_req_canceled(struct tpm_chip *chip, u8 status) return status == TPM_STS_COMMAND_READY; } +static void tpm_tis_reenable_polling(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + + dev_warn(&chip->dev, FW_BUG + "TPM interrupt storm detected, polling instead\n"); + + __tpm_tis_disable_interrupts(chip); + + /* + * devm_free_irq() must not be called from within the interrupt handler, + * since this function waits for running handlers to finish and thus it + * would deadlock. Instead trigger a worker that takes care of the + * unregistration. + */ + schedule_work(&priv->free_irq_work); +} + +static irqreturn_t tpm_tis_check_for_interrupt_storm(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + const unsigned int MAX_UNHANDLED_IRQS = 1000; + + /* + * The worker to free the TPM interrupt (free_irq_work) may already + * be scheduled, so make sure it is not scheduled again. + */ + if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) + return IRQ_HANDLED; + + if (time_after(jiffies, priv->last_unhandled_irq + HZ/10)) + priv->unhandled_irqs = 1; + else + priv->unhandled_irqs++; + + priv->last_unhandled_irq = jiffies; + + if (priv->unhandled_irqs > MAX_UNHANDLED_IRQS) + tpm_tis_reenable_polling(chip); + + /* + * Prevent the genirq code from starting its own interrupt storm + * handling by always reporting that the interrupt was handled. + */ + return IRQ_HANDLED; +} + static irqreturn_t tis_int_handler(int dummy, void *dev_id) { struct tpm_chip *chip = dev_id; @@ -761,10 +815,10 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); if (rc < 0) - return IRQ_NONE; + goto unhandled; if (interrupt == 0) - return IRQ_NONE; + goto unhandled; set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); if (interrupt & TPM_INTF_DATA_AVAIL_INT) @@ -780,10 +834,13 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_write32(priv, TPM_INT_STATUS(priv->locality), interrupt); tpm_tis_relinquish_locality(chip, 0); if (rc < 0) - return IRQ_NONE; + goto unhandled; tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); return IRQ_HANDLED; + +unhandled: + return tpm_tis_check_for_interrupt_storm(chip); } static void tpm_tis_gen_interrupt(struct tpm_chip *chip) @@ -804,6 +861,15 @@ static void tpm_tis_gen_interrupt(struct tpm_chip *chip) chip->flags &= ~TPM_CHIP_FLAG_IRQ; } +static void tpm_tis_free_irq_func(struct work_struct *work) +{ + struct tpm_tis_data *priv = container_of(work, typeof(*priv), free_irq_work); + struct tpm_chip *chip = priv->chip; + + devm_free_irq(chip->dev.parent, priv->irq, chip); + priv->irq = 0; +} + /* Register the IRQ and issue a command that will cause an interrupt. If an * irq is seen then leave the chip setup for IRQ operation, otherwise reverse * everything and leave in polling mode. Returns 0 on success. @@ -816,6 +882,7 @@ static int tpm_tis_probe_irq_single(struct tpm_chip *chip, u32 intmask, int rc; u32 int_status; + INIT_WORK(&priv->free_irq_work, tpm_tis_free_irq_func); rc = devm_request_threaded_irq(chip->dev.parent, irq, NULL, tis_int_handler, IRQF_ONESHOT | flags, @@ -918,6 +985,7 @@ void tpm_tis_remove(struct tpm_chip *chip) interrupt = 0; tpm_tis_write32(priv, reg, ~TPM_GLOBAL_INT_ENABLE & interrupt); + flush_work(&priv->free_irq_work); tpm_tis_clkrun_enable(chip, false); @@ -1021,6 +1089,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, chip->timeout_b = msecs_to_jiffies(TIS_TIMEOUT_B_MAX); chip->timeout_c = msecs_to_jiffies(TIS_TIMEOUT_C_MAX); chip->timeout_d = msecs_to_jiffies(TIS_TIMEOUT_D_MAX); + priv->chip = chip; priv->timeout_min = TPM_TIMEOUT_USECS_MIN; priv->timeout_max = TPM_TIMEOUT_USECS_MAX; priv->phy_ops = phy_ops; diff --git a/drivers/char/tpm/tpm_tis_core.h b/drivers/char/tpm/tpm_tis_core.h index e978f457fd4d..b1fa42367052 100644 --- a/drivers/char/tpm/tpm_tis_core.h +++ b/drivers/char/tpm/tpm_tis_core.h @@ -91,11 +91,15 @@ enum tpm_tis_flags { }; struct tpm_tis_data { + struct tpm_chip *chip; u16 manufacturer_id; struct mutex locality_count_mutex; unsigned int locality_count; int locality; int irq; + struct work_struct free_irq_work; + unsigned long last_unhandled_irq; + unsigned int unhandled_irqs; unsigned int int_mask; unsigned long flags; void __iomem *ilb_base_addr;