From patchwork Mon Jul 16 13:15:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Tatashin X-Patchwork-Id: 10526767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 898F36020A for ; Mon, 16 Jul 2018 13:16:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7888D27F94 for ; Mon, 16 Jul 2018 13:16:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6BFE0287A1; Mon, 16 Jul 2018 13:16:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2AC0D27F94 for ; Mon, 16 Jul 2018 13:16:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07D4E6B0005; Mon, 16 Jul 2018 09:16:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 003A86B0007; Mon, 16 Jul 2018 09:16:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC0FB6B0008; Mon, 16 Jul 2018 09:16:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f197.google.com (mail-qk0-f197.google.com [209.85.220.197]) by kanga.kvack.org (Postfix) with ESMTP id B16406B0005 for ; Mon, 16 Jul 2018 09:16:30 -0400 (EDT) Received: by mail-qk0-f197.google.com with SMTP id o18-v6so45220465qko.21 for ; Mon, 16 Jul 2018 06:16:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:mime-version:references :in-reply-to:from:date:message-id:subject:to:cc; bh=HwvmoxJc90L5ORuaIMFg3/pIzzrVSj3wwF3/WubEUuk=; b=Rv8t4YvkGPUpNOZxAKkpBUFBLOt3m0QA1Ok/6S2vzV6ZOhP0hqn0miqI5aW1fMVrRQ QfyYpyT2Mi9Sba2EzNwK4me0ecbmE2UvX6RmIUmdG14N0TQqEWugkZO6YuY7vWuCqO5O dSRj46gjLt8cjxTa0XHC6EhWJg0eNx1NlDuVdNPRPYKtDu8Hxh+M+bDPdydZs5xDLiW+ XLRMH3Tmhw3AYuHvKFgvNMop/LgteOEYABXORHc7DTTLigPNxrmnNvHGkjEUZJdld7WR mW17HvMUefcJUHF02Y/dnWj96K4H4nJot2c8i+R+oPeUXQ7wveBcucTNs39g/j+CjRvy hT4g== X-Gm-Message-State: AOUpUlGneYrwBQS6NMctPkUuTMMu/fkgBgcUZE2tYqfpwW1rwnH56ZjZ zslS4taL/IKWwPvw+GRT5GEivWMzhLFBD+/ttadHVcPxUmm/B9ReaOwLXs8jNpwKSYzZd/ZuGMB +n67aPJ787yhVlrOH095wTqvBJThI0hfLm95Lnx7SGQ7ZdGR9YNq1GPxnO9Wk5JjbdA== X-Received: by 2002:a0c:ab43:: with SMTP id i3-v6mr18323872qvb.87.1531746990495; Mon, 16 Jul 2018 06:16:30 -0700 (PDT) X-Google-Smtp-Source: AAOMgpee35momF2g9Du2RxIAC4ktqf9QGHRyjcUb4+/8h2Rq6lDjRkd7vN33azMZgLzSVQ0VpDlI X-Received: by 2002:a0c:ab43:: with SMTP id i3-v6mr18323785qvb.87.1531746989545; Mon, 16 Jul 2018 06:16:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531746989; cv=none; d=google.com; s=arc-20160816; b=DhS5+zdhBGpFdOf2Tz138nBnpLIbu+kANhUB4jpSxjqevLFgbI8IQJo2oOumGuLjse xkSwceCbEdaBX4YuZ+MDewvu2dDPLEuwEJdnK13ihlvxJIqw830V1XT7HLhZPpmcPDuz d4Jqt/MPaNZiYfyGQG/IuM3Ly460qPoztVGuBJ2nz0ndKG+k1UZBxs2UcLuvpVocZ+OU OzLMsY48ISkjBE+m2ypZigiaqeL3I3g1xP5HhKEjyykk2U0QdzNzJmhUbghfdWMnc30v LVdE1fBiEmNhJAbFh2/4nXQnYTz4eq6d4jh5KkrZMQze/vCl733/1j71JfkLCUx0vA+M syYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature:arc-authentication-results; bh=HwvmoxJc90L5ORuaIMFg3/pIzzrVSj3wwF3/WubEUuk=; b=rVlRQmyg0Zw3AQL+gmW/e89xjhXbak5QROs+tOOdmZfX1wqS0cc3ncbBLQTcuKV8lX vFibSuQygr3lVMfKTNTwMGHoLuWZR6754ir0CbusgBkgOcRkZZVcQnSCfL6PzsePQk2x o7i6eZvuLDKpAl/w9IK65qJOSSYa1zJ/+1yqo87gFoQwaCIG6ehRYap5p1DGZ7wot1M7 8NxfoeCP4brR0W3DMUSyj4FhqbExDZ3PhZipesRmy31EXVt9nB2f6DY23I8Ttq6eya4/ VxzhJzsM3ckUc1uT3RVfXGZlXqbF65K5sAKoMXumizWOBiWEkcQXwCgnVeUtQD1WXqk8 v6Ug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=XcphlLL7; spf=pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=pasha.tatashin@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id q11-v6si21443720qkl.387.2018.07.16.06.16.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Jul 2018 06:16:29 -0700 (PDT) Received-SPF: pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=XcphlLL7; spf=pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=pasha.tatashin@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w6GDDbWZ143564 for ; Mon, 16 Jul 2018 13:16:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : cc : content-type; s=corp-2018-07-02; bh=HwvmoxJc90L5ORuaIMFg3/pIzzrVSj3wwF3/WubEUuk=; b=XcphlLL7KsJdTYDO6G2FmZliVcdFUt/FGHov8R8Mk0ojPzufTbiu8vsltUEujajyRjCm Gu1GqVNsla5dvQS/zpZOVkMe6iBfpmZ5xrEjXdhfDZ9FsVuOzs017hW40wxEedyGSk23 01iXKTJAWDlwpBIRndEIJaDAa3s6zlbFQ5yVZ6cx5sSCRNCCc5zcZyt7cJf8gGuRrMdI i00BMr8RUQC22uFO+QQ3xq2hvzLaY+bKL/3FWrhlLPEUebrwW7rip7fD6vlvuJPfQmoq rfUWMnpdmm1m/1/nZLlpgpGKcta9j4pT3MuUJP9EFzLNLY9BB7brOWri5dbVAorzr5s+ Ew== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2k7a3jmc0c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 16 Jul 2018 13:16:28 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id w6GDGRfp027425 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 16 Jul 2018 13:16:27 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id w6GDGQDO006226 for ; Mon, 16 Jul 2018 13:16:27 GMT Received: from mail-oi0-f47.google.com (/209.85.218.47) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 16 Jul 2018 13:16:26 +0000 Received: by mail-oi0-f47.google.com with SMTP id l10-v6so27976204oii.0 for ; Mon, 16 Jul 2018 06:16:26 -0700 (PDT) X-Received: by 2002:aca:3bc2:: with SMTP id i185-v6mr19496973oia.156.1531746985986; Mon, 16 Jul 2018 06:16:25 -0700 (PDT) MIME-Version: 1.0 References: <5b4a9633.1c69fb81.17984.f7b3@mx.google.com> <0ab16066-5498-374b-5391-3dd7979044aa@collabora.com> <20180716105451.GA20734@sirena.org.uk> <20180716110208.GA25227@kroah.com> In-Reply-To: From: Pavel Tatashin Date: Mon, 16 Jul 2018 09:15:50 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: mainline/master boot: 177 boots: 2 failed, 174 passed with 1 conflict (v4.18-rc4-160-gf353078f028f) To: gregkh@linuxfoundation.org Cc: broonie@kernel.org, guillaume.tucker@collabora.com, Linux Memory Management List , kernel@collabora.com, LKML , stable@vger.kernel.org, kernel-build-reports@lists.linaro.org, Michal Hocko , Linus Torvalds X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8955 signatures=668706 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=10 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807160156 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP I have figured out what is going with x86-32. Since it has FLATMEM memory layout, the memmap is now allocated after zero_resv_unavail(): Now, we have something like this: zero_resv_unavail() free_area_init_node() #ifdef CONFIG_FLAT_NODE_MEM_MAP alloc_node_mem_map() #endif At the time when zero_resv_unavail() is called, memmap for FLAT_NODE_MEM_MAP is not yet allocated. On the other hand, alloc_node_mem_map() calls memblock_virt_alloc_node_nopanic() which calls memset(0), so zero_resv_unavail() is not needed for this layout. The fix is: free_area_init_node(nid, NULL, This is just a temporary fix, I will do a proper fix later, when I will get rid of zero_resv_unavail(), but that will require more thinking, on how to ensure that no section in memmap is skipped while we go through memmap_init_zone(). Should I submit an updated patch for "mm: zero unavailable pages before memmap init", or just this incremental fix? Thank you, Pavel On Mon, Jul 16, 2018 at 7:56 AM Pavel Tatashin wrote: > > I have reproduced the problem on mainline. Use x86_32 defcontig + > qemu, and problem is reproduced immediately. I will send an update > once I figure out what is going on. > > Pavel > On Mon, Jul 16, 2018 at 7:02 AM Greg Kroah-Hartman > wrote: > > > > On Mon, Jul 16, 2018 at 11:54:51AM +0100, Mark Brown wrote: > > > On Mon, Jul 16, 2018 at 11:40:06AM +0100, Guillaume Tucker wrote: > > > > On 15/07/18 01:32, kernelci.org bot wrote: > > > > > mainline/master boot: 177 boots: 2 failed, 174 passed with 1 conflict (v4.18-rc4-160-gf353078f028f) > > > > > > > > > > Full Boot Summary: https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.18-rc4-160-gf353078f028f/ > > > > > Full Build Summary: https://kernelci.org/build/mainline/branch/master/kernel/v4.18-rc4-160-gf353078f028f/ > > > > > > > > > > Tree: mainline > > > > > Branch: master > > > > > Git Describe: v4.18-rc4-160-gf353078f028f > > > > > Git Commit: f353078f028fbfe9acd4b747b4a19c69ef6846cd > > > > > Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > > > > Tested: 67 unique boards, 25 SoC families, 21 builds out of 199 > > > > > > > > > > Boot Regressions Detected: > > > > [...] > > > > > x86: > > > > > > > > > > i386_defconfig: > > > > > x86-celeron: > > > > > lab-mhart: new failure (last pass: v4.18-rc4-147-g2db39a2f491a) > > > > > x86-pentium4: > > > > > lab-mhart: new failure (last pass: v4.18-rc4-147-g2db39a2f491a) > > > > > > > > Please see below an automated bisection report for this > > > > regression. Several bisections were run on other x86 platforms > > > > with i386_defconfig on a few revisions up to v4.18-rc5, they all > > > > reached the same "bad" commit. > > > > > > > > > > > > Unfortunately there isn't much to learn from the kernelci.org > > > > boot logs as the kernel seems to crash very early on: > > > > > > > > https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.18-rc5/ > > > > https://storage.kernelci.org/mainline/master/v4.18-rc4-160-gf353078f028f/x86/i386_defconfig/lab-mhart/lava-x86-celeron.html > > > > > > > > > > > > It looks like stable-rc/linux-4.17.y is also broken with > > > > i386_defconfig, which tends to confirm the "bad" commit found by > > > > the automated bisection which was applied there as well: > > > > > > > > https://kernelci.org/boot/all/job/stable-rc/branch/linux-4.17.y/kernel/v4.17.6-68-gbc0bd9e05fa1/ > > > > > > Adding Greg directly to the CCs due to the stable impact, not deleting > > > context for his benefit. > > > > Hey, I read all stable emails, who else would? :) > > > > > > The automated bisection on kernelci.org is still quite new, so > > > > please take the results with a pinch of salt as the "bad" commit > > > > found may not be the actual root cause of the boot failure. > > > > > > > > Hope this helps! > > > > > > > > Best wishes, > > > > Guillaume > > > > > > > > > > > > --------------------------------------8<-------------------------------------- > > > > > > > > > > > > > > > > Bisection result for mainline/master (v4.18-rc4-160-gf353078f028f) on x86-celeron > > > > > > > > Good: 2db39a2f491a Merge branch 'i2c/for-current' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux > > > > Bad: f353078f028f Merge branch 'akpm' (patches from Andrew) > > > > Found: e181ae0c5db9 mm: zero unavailable pages before memmap init > > > > > > > > Checks: > > > > revert: PASS > > > > verify: PASS > > > > > > > > Parameters: > > > > Tree: mainline > > > > URL: http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > > > Branch: master > > > > Target: x86-celeron > > > > Lab: lab-mhart > > > > Config: i386_defconfig > > > > Plan: boot > > > > > > > > Breaking commit found: > > > > > > > > ------------------------------------------------------------------------------- > > > > commit e181ae0c5db9544de9c53239eb22bc012ce75033 > > > > Author: Pavel Tatashin > > > > Date: Sat Jul 14 09:15:07 2018 -0400 > > > > > > > > mm: zero unavailable pages before memmap init > > > > We must zero struct pages for memory that is not backed by physical > > > > memory, or kernel does not have access to. > > > > Recently, there was a change which zeroed all memmap for all holes in > > > > e820. Unfortunately, it introduced a bug that is discussed here: > > > > https://www.spinics.net/lists/linux-mm/msg156764.html > > > > Linus, also saw this bug on his machine, and confirmed that reverting > > > > commit 124049decbb1 ("x86/e820: put !E820_TYPE_RAM regions into > > > > memblock.reserved") fixes the issue. > > > > The problem is that we incorrectly zero some struct pages after they > > > > were setup. > > > > The fix is to zero unavailable struct pages prior to initializing of > > > > struct pages. > > > > A more detailed fix should come later that would avoid double zeroing > > > > cases: one in __init_single_page(), the other one in > > > > zero_resv_unavail(). > > > > Fixes: 124049decbb1 ("x86/e820: put !E820_TYPE_RAM regions into memblock.reserved") > > > > Signed-off-by: Pavel Tatashin > > > > Signed-off-by: Linus Torvalds > > > > > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > > index 1521100f1e63..5d800d61ddb7 100644 > > > > --- a/mm/page_alloc.c > > > > +++ b/mm/page_alloc.c > > > > @@ -6847,6 +6847,7 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn) > > > > /* Initialise every node */ > > > > mminit_verify_pageflags_layout(); > > > > setup_nr_node_ids(); > > > > + zero_resv_unavail(); > > > > for_each_online_node(nid) { > > > > pg_data_t *pgdat = NODE_DATA(nid); > > > > free_area_init_node(nid, NULL, > > > > @@ -6857,7 +6858,6 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn) > > > > node_set_state(nid, N_MEMORY); > > > > check_for_memory(pgdat, nid); > > > > } > > > > - zero_resv_unavail(); > > > > } > > > > static int __init cmdline_parse_core(char *p, unsigned long *core, > > > > @@ -7033,9 +7033,9 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) > > > > void __init free_area_init(unsigned long *zones_size) > > > > { > > > > + zero_resv_unavail(); > > > > free_area_init_node(0, zones_size, > > > > __pa(PAGE_OFFSET) >> PAGE_SHIFT, NULL); > > > > - zero_resv_unavail(); > > > > } > > > > static int page_alloc_cpu_dead(unsigned int cpu) > > > > ------------------------------------------------------------------------------- > > > > So this patch breaks i386, ick. I'll wait for the fix to hit Linus's > > tree as that's a bit more important to have the large majority of the > > x86-64 boxes fixed with this patch for now. > > > > thanks, > > > > greg k-h > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5d800d61ddb7..9ec34218713b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6847,7 +6847,9 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn) /* Initialise every node */ mminit_verify_pageflags_layout(); setup_nr_node_ids(); +#ifndef CONFIG_FLAT_NODE_MEM_MAP zero_resv_unavail(); +#endif for_each_online_node(nid) { pg_data_t *pgdat = NODE_DATA(nid);