From patchwork Thu Jun 21 09:18:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 10479543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8D0C860383 for ; Thu, 21 Jun 2018 09:18:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B497292C1 for ; Thu, 21 Jun 2018 09:18:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7978B292A3; Thu, 21 Jun 2018 09:18:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 13E5F292C1 for ; Thu, 21 Jun 2018 09:18:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CKBiPwt72uAg+tbAczCgbHehEkbVUcNKq7BzLzYk7dE=; b=HXgPUbycaT9Qmb /0sQDyWXZB9HjssCoQrI51SiC9lvfUKdpQz6Y8a2M5PYnn8BPXu8aARtOgBzul6bzUpQvt09tGAQv tClaognNsW/KjLqYebY2TJAGEkR8PSDEsGdjdK0nKndE+YuM4UsvJS1mQf4lURg7cj9wyTVDn3j3h OboZY0HbX93ro8SdbT2AZ9IB4I1SA1v/bZiGZpLQPhsIF5KJa3MPWR//Sn8hpkOqyuBJSP2LHm2vy lo73a7u73ANxZrpSnDG9O8H4L1Mgj3n1U2pAZnl2dcN3POohm1WMJTxo27US6VOgfsXZ790c0jhno XKUroMRGQYAhB7dJWv+w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fVvjb-0006KK-I7; Thu, 21 Jun 2018 09:18:31 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fVvjW-0006IN-6i for linux-arm-kernel@lists.infradead.org; Thu, 21 Jun 2018 09:18:28 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 54C5180D; Thu, 21 Jun 2018 02:18:15 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 235CA3F246; Thu, 21 Jun 2018 02:18:15 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 3F25F1AE101A; Thu, 21 Jun 2018 10:18:51 +0100 (BST) Date: Thu, 21 Jun 2018 10:18:51 +0100 From: Will Deacon To: James Morse Subject: Re: KVM guest sometimes failed to boot because of kernel stack overflow if KPTI is enabled on a hisilicon ARM64 platform. Message-ID: <20180621091850.GA22505@arm.com> References: <5B2A6218.3030201@hisilicon.com> <20180620144257.GB27776@arm.com> <5B2A7832.4010502@hisilicon.com> <5B2A7FE1.5040607@hisilicon.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180621_021826_812203_E51502CF X-CRM114-Status: GOOD ( 17.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, catalin.marinas@arm.com, Linuxarm , Hanjun Guo , xiexiuqi@huawei.com, Wei Xu , "kongxinwei \(A\)" , huangdaode , "Chenxin \(Charles\)" , "Zhuangyuzeng \(Yisen\)" , Zhangyi ac , suzuki.poulose@arm.com, marc.zyngier@arm.com, John Garry , "Xiongfanggou \(James\)" , jonathan.cameron@huawei.com, linux-arm-kernel@lists.infradead.org, Salil Mehta , linux-kernel@vger.kernel.org, Shameerali Kolothum Thodi , dave.martin@arm.com, "Wangzhou \(B\)" , "Liyuan \(Larry, Turing Solution\)" , libeijian@hisilicon.com, "Liguozhu \(Kenneth\)" , Shiju Jose Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Jun 21, 2018 at 09:38:53AM +0100, James Morse wrote: > On 20/06/18 17:25, Wei Xu wrote: > >     [    0.042421] Insufficient stack space to handle exception! > >     [    0.042423] ESR: 0x96000046 -- DABT (current EL) > >     [    0.043730] FAR: 0xffff0000093a80e0 > >     [    0.044714] Task stack: [0xffff0000093a8000..0xffff0000093ac000] > > This was a level 2 translation fault on a write, to an address that is within > the stack.... > > > >     [    0.051113] IRQ stack: [0xffff000008000000..0xffff000008004000] > >     [    0.057610] Overflow stack: [0xffff80003efce2f0..0xffff80003efcf2f0] > >     [    0.064003] CPU: 0 PID: 12 Comm: migration/0 Not tainted > > 4.17.0-45865-g2b31fe7-dirty #10 > >     [    0.072201] Hardware name: linux,dummy-virt (DT) > > >     [    0.076797] pstate: 604003c5 (nZCv DAIF +PAN -UAO) > >     [    0.081727] pc : el1_sync+0x0/0xb0 > > ... from the vectors. > > > >     [    0.085217] lr : kpti_install_ng_mappings+0x120/0x214 > > What I think is happening is: we come out of the kpti idmap with the stack > unmapped. Shortly after we access the stack, which faults. el1_sync faults as > well when it tries to push the registers to the stack, and we keep going until > we overflow the stack. > > I can't reproduce this with kvmtool or qemu in the model. Hmm, one thing that occurs to me is that the kpti_install_ng_mappings() code leaves the nG bit set in table entries, which is actually IGNORED in the architecture. Wei -- does the diff below help at all? Make sure you disable CONFIG_KASAN, otherwise your kernel will take an age to boot. Will --->8 diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 5f9a73a4452c..70d9e98467ca 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -272,8 +272,8 @@ ENTRY(idmap_kpti_install_ng_mappings) add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) do_pgd: __idmap_kpti_get_pgtable_ent pgd tbnz pgd, #1, walk_puds -next_pgd: __idmap_kpti_put_pgtable_ent_ng pgd +next_pgd: skip_pgd: add cur_pgdp, cur_pgdp, #8 cmp cur_pgdp, end_pgdp @@ -302,8 +302,8 @@ walk_puds: add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) do_pud: __idmap_kpti_get_pgtable_ent pud tbnz pud, #1, walk_pmds -next_pud: __idmap_kpti_put_pgtable_ent_ng pud +next_pud: skip_pud: add cur_pudp, cur_pudp, 8 cmp cur_pudp, end_pudp @@ -323,8 +323,8 @@ walk_pmds: add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) do_pmd: __idmap_kpti_get_pgtable_ent pmd tbnz pmd, #1, walk_ptes -next_pmd: __idmap_kpti_put_pgtable_ent_ng pmd +next_pmd: skip_pmd: add cur_pmdp, cur_pmdp, #8 cmp cur_pmdp, end_pmdp