From patchwork Tue Nov 16 08:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12692710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACDA9C433F5 for ; Tue, 16 Nov 2021 08:27:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 78D8F63218 for ; Tue, 16 Nov 2021 08:27:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 78D8F63218 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7JRK6ApBFAjyPWR2N2QH56NW9dyQ373wvrXcwD92QWA=; b=W7GRB32uol196j D6NC3o/aPPGypfawRcB09SHIhDkxRdXBwm5arprN4czj7Lh0JFghTIcH1gwR16SLo6+UROzMhtdWQ 2aZmMx/VR6ghrKrceFN7qs7XB6aggShtYfNG3z90HU6w2sFxDlGvJtG4/VoJkFRoDbbfBn9Q5Mnjw rSqg9xvHNec7YyeeZ7vTtW2ztYf5BNMZwOvy0z2/0QXDDXShOP+7Vtq78B+IRf6L9S17tPRvII2Ck WAeXISDrG17m0Ua+wXDix4CfhZM3CgCd3Jga5DKA92C9fxy8USps31R2Sd7oPe9j1e+Gb+v8XVINz NEN4Lp/u/xDAP3mnBprA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmg-000hJ7-RP; Tue, 16 Nov 2021 08:25:43 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmM-000hDf-U2 for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 08:25:25 +0000 Received: by mail-pf1-x42c.google.com with SMTP id o4so17427636pfp.13 for ; Tue, 16 Nov 2021 00:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hqFMIdJyaJEShckULa2FgjRnQWIhP9aK8GPkMVQJLb4=; b=It4SAbMnf2QadM4I4MSYROpfIQvR7YP1qg2P/fZRxoCaG76NW7tSt+4uw9fOnTVNQr jnpz89RB4/RhBf6iPztS5ZRXfE5zIOoIm92XY8Vv6V+4cy3dpT+Kk8xIWpV2F2bZagVF b3oU2RMn+hs56siMzUspt4VnlJTAZX0XBSLrHpnMznTUDOUPia4Nsvu7cAFLexIemRW1 LRRMHAVMX5LCgODKMvZqGEqQCs1hazZouE/NYC9SaEdrJEcBPLKuhYQHZSIuk7Ai4Rut WSP7kcqP91pP4/pqFBnN7hsP0d6QhsJ3z/OqyXgjRwnxj9NQ2w/Uag/WSN8/kVa3iiFe EPDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hqFMIdJyaJEShckULa2FgjRnQWIhP9aK8GPkMVQJLb4=; b=FMWN/f9Fuq9TqZ/k4PA0TaBP1OHgviEkRWGwnrqaR02SGArg3iOFFZko3Os15JEmdE cRbuh+h66q+R0uS+GMLNHQd60/GgM9xSNFuEwp+w5SYUjsQU9a+okHBkWjzyMgfJadpc L+uwevjRAFBkuivNgO9S8Qui7UZlIDWV1nw08izr9LSG4qlWdw4L2wQ7FWAmo/cCwjZN XolgzpOipaLRAIwLCCOunnK40sEFvtui2j56YklbSHyGmi4jK+gWVUkKd+AFYUdiTtnY /oqwM2TiXwRn2ItQDMVbg4YBy1l9Fc5pL/rYA5tX4el+nF7j0nzLRaz0mlUYxguroc8y 9Kwg== X-Gm-Message-State: AOAM531k54iSk+vaugzsJnMSYxkd/3LV/GNJBcNJ0TcK6mQnZN3EGv+d ujtVKiLNlTYQdGmo5XBBCleTCprqUQ== X-Google-Smtp-Source: ABdhPJxv0DVZZb6FhubzM7YyR1ApCXK8IqOIdPyd5tZXRxwWzCc8NeH0UAwvpJZmnjBFXsDisloaFQ== X-Received: by 2002:aa7:9257:0:b0:4a2:b444:ac33 with SMTP id 23-20020aa79257000000b004a2b444ac33mr16391813pfp.20.1637051121427; Tue, 16 Nov 2021 00:25:21 -0800 (PST) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 11sm17104011pfl.41.2021.11.16.00.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 00:25:21 -0800 (PST) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , "Paul E . McKenney" , Catalin Marinas , Will Deacon , Marc Zyngier , Thomas Gleixner , Joey Gouly , Sami Tolvanen , Julien Thierry , Yuichi Ito , rcu@vger.kernel.org Subject: [PATCHv3 1/4] arm64: entry: judge nmi ealier to avoid deadlock in RCU Date: Tue, 16 Nov 2021 16:24:47 +0800 Message-Id: <20211116082450.10357-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211116082450.10357-1-kernelfans@gmail.com> References: <20211116082450.10357-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_002523_032602_236C1782 X-CRM114-Status: GOOD ( 19.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Linux kernel places strict semantics between NMI and maskable interrupt. So does the RCU component, else deadlock may happen. But the current arm64 entry code can partially breach this rule through calling rcu_nmi_enter(). *** how a deadlock can happen if NMI mistaken as IRQ *** rcu_nmi_enter() { if (rcu_dynticks_curr_cpu_in_eqs()) { if (!in_nmi()) rcu_dynticks_task_exit(); ... if (!in_nmi()) { instrumentation_begin(); rcu_cleanup_after_idle(); instrumentation_end(); } ... } else if (!in_nmi()) { instrumentation_begin(); rcu_irq_enter_check_tick(); } } If a NMI is mistaken as a maskable interrupt, rcu_irq_enter_check_tick() can hit a deadlock, which is demonstrated by the following scenario: note_gp_changes() runs in a task context { local_irq_save(flags); // this protects against irq, but not NMI rnp = rdp->mynode; ... raw_spin_trylock_rcu_node(rnp) -------> broken in by (p)NMI, without taking __nmi_enter() rcu_nmi_enter() ->__rcu_irq_enter_check_tick() ->raw_spin_lock_rcu_node(rdp->mynode); deadlock happens!!! } *** On arm64, how pNMI mistaken as IRQ *** On arm64, pNMI is an analogue to NMI. In essence, it is a higher priority interrupt but not disabled by local_irq_disable(). In current implementation 1) If a pNMI from a context where IRQs were masked, it can be recognized as nmi, and calls __nmi_enter() immediately. This is no problem. 2) But it causes trouble if a pNMI from a context where IRQs were unmasked, and temporarily regarded as maskable interrupt. It is not treated as NMI, i.e. calling nmi_enter() until reading from GIC. __el1_irq() { irq_enter_rcu() ----> hit the deadlock bug gic_handle_nmi() nmi_enter() nmi_exit() irq_exit_rcu() } *** Remedy *** If the irqchip level exposes an interface for detecting pNMI to arch level code, it can meet the requirement at this early stage. That is the interface (*interrupt_is_nmi)() in this patch. Signed-off-by: Pingfan Liu Cc: Mark Rutland Cc: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Thomas Gleixner Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Yuichi Ito Cc: rcu@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/irq.h | 1 + arch/arm64/kernel/entry-common.c | 10 +++++++++- arch/arm64/kernel/irq.c | 18 ++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index fac08e18bcd5..f3eb13bfa65e 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -11,6 +11,7 @@ struct pt_regs; int set_handle_irq(void (*handle_irq)(struct pt_regs *)); #define set_handle_irq set_handle_irq int set_handle_fiq(void (*handle_fiq)(struct pt_regs *)); +int set_nmi_discriminator(bool (*discriminator)(void)); static inline int nr_legacy_irqs(void) { diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index f7408edf8571..5a1a5dd66d04 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -260,6 +260,14 @@ static void do_interrupt_handler(struct pt_regs *regs, extern void (*handle_arch_irq)(struct pt_regs *); extern void (*handle_arch_fiq)(struct pt_regs *); +extern bool (*interrupt_is_nmi)(void); + +static inline bool is_in_pnmi(struct pt_regs *regs) +{ + if (!interrupts_enabled(regs) || (*interrupt_is_nmi)()) + return true; + return false; +} static void noinstr __panic_unhandled(struct pt_regs *regs, const char *vector, unsigned int esr) @@ -454,7 +462,7 @@ static void noinstr el1_interrupt(struct pt_regs *regs, { write_sysreg(DAIF_PROCCTX_NOIRQ, daif); - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) + if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && is_in_pnmi(regs)) __el1_pnmi(regs, handler); else __el1_irq(regs, handler); diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index bda49430c9ea..fabed09ed966 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -81,8 +81,14 @@ static void default_handle_fiq(struct pt_regs *regs) panic("FIQ taken without a root FIQ handler\n"); } +static bool default_nmi_discriminator(void) +{ + return false; +} + void (*handle_arch_irq)(struct pt_regs *) __ro_after_init = default_handle_irq; void (*handle_arch_fiq)(struct pt_regs *) __ro_after_init = default_handle_fiq; +bool (*interrupt_is_nmi)(void) __ro_after_init = default_nmi_discriminator; int __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) { @@ -104,6 +110,18 @@ int __init set_handle_fiq(void (*handle_fiq)(struct pt_regs *)) return 0; } +#if IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) + +int __init set_nmi_discriminator(bool (*discriminator)(void)) +{ + if (interrupt_is_nmi != default_nmi_discriminator) + return -EBUSY; + + interrupt_is_nmi = discriminator; + return 0; +} +#endif + void __init init_IRQ(void) { init_irq_stacks();