From patchwork Tue Nov 13 05:50:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679583 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 299A113B5 for ; Tue, 13 Nov 2018 05:51:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18E8B2A2A0 for ; Tue, 13 Nov 2018 05:51:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1704E2A2D9; Tue, 13 Nov 2018 05:51:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8C2BC2A2B7 for ; Tue, 13 Nov 2018 05:51:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE0E06B0275; Tue, 13 Nov 2018 00:51:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D3F076B0276; Tue, 13 Nov 2018 00:51:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C311B6B0277; Tue, 13 Nov 2018 00:51:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 82E266B0275 for ; Tue, 13 Nov 2018 00:51:11 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id d11-v6so8721125plo.17 for ; Mon, 12 Nov 2018 21:51:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=HO8NXoH9HC+Id3/X9jGgx4dOol+odVvFSfDdh7H+ymY=; b=pFEUC0x0h+Dw6vlIK8beQ5RcQ/k1CB4WmjpPFdqNxJ3YNv6r6HmrfR4G4FfKb5emdw IlLqNpSwkgEt40Q4exxfICzlj4rasHGwUszPYs+cV9KfWmQ9r8/s30eoHQeMSNimGa5I MYQ2hZ/1TUj7zZGbhvqgcFfuioUjSow59q1RCaMjUxu06LyDL1N0au5VG73Zes8aGnj1 UPXUIP/jA7eUTx6W+o1RTyg1ZuIbXsVilL9PabWykkg54KImfEvOx+ZnFDWgpzNCYh6E X97yPZUEgqNy/+okvJAyKez/pu7A45XMnqf2ybSvGNOMM4lksuFxntYpRpASnzz3bhA9 KQzA== X-Gm-Message-State: AGRZ1gImJZjk+Y9Hu2D6dDBUjHHbbt7cZgw0cGrxbqNlUgwrdAW0y8id RrFquvJxnTJnnmcoGbqmClq9hzBxVf1p48Eyjuppy+lVe5Ld3zjrCyI1RFbtQxtb5lGoeeT9WnS cLgajCDErNsCb7Ty8tAT+qrBnUidb0k7+BwNVVcZq8bI9Xbk3X9OYty8c//uNkb+M1w== X-Received: by 2002:a63:2141:: with SMTP id s1mr1677740pgm.148.1542088271207; Mon, 12 Nov 2018 21:51:11 -0800 (PST) X-Google-Smtp-Source: AJdET5cd3TSs6OeX+6x2kTrTZb9T48tffFfjMSgQ+8QkQATBpL+GS/vMY3qwsm7RVj5QsjCLMqaO X-Received: by 2002:a63:2141:: with SMTP id s1mr1677713pgm.148.1542088270434; Mon, 12 Nov 2018 21:51:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088270; cv=none; d=google.com; s=arc-20160816; b=WmCpGaE32PGShT/oHB/8R8+uDjwELz01Nu1eGytoWYFqGZvvpPaHSwsCHNsTbWKFiK m3ayv7xbY8ZFLYO2lkmvvecDFegFKcbD+U2ETmZYPmsDRD85ZdCqcf9eyyiQZOEiEF77 MzlztEN8Yx4xIZ24ZIoYiqYYoiAV/Aa+qXs0a0i9BYiXsLLSakyayTlM4r77T4grb4bw Oxnh8cLxQ4aBgQH4fTGuuyLFLxiTfQLAu1BrYO+q3A6gHl8O7Ehf7XKlkzWPIl7JpnIo ZBmDA57+dwl2QuN0Jj0CUnjzXB/oNSQK3RA0c4De6LlPa1aXW1ddGIbFqOTKArVo3XUv Y5Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HO8NXoH9HC+Id3/X9jGgx4dOol+odVvFSfDdh7H+ymY=; b=n1otRO3NAKKSY6aJCfFZGVIb4c4dGqCDhYNzpQN/YEHRXHL8cSfaGq/rnxcK2VhPd1 o03khDxFPfd5LLMByWb4yGTjxV1C1TuJHXmf4HYglIhN379OCKdwLpRGKPPs5Pcn+H7K zdlhEWU3XYPT3VijgR0ppG914RdLLGNVIHzF9iXzcMccIHVArxqgrIa9KKzMYPBXi739 L7sLT8impPSdY63ki1YpVa5fySK8emgywdxbLZyxOJkfmuaUOerErc5t4KvLkoDmJRlp 53l3xSqCIWCT5AOJR32KROV2yER2T7xl0cQXbGjciWjkMtemqSASRkj4Ii+w8sqHKioc 0j9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=DcEYEchS; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id b3-v6si19523218pgn.100.2018.11.12.21.51.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:10 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=DcEYEchS; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EF8462250F; Tue, 13 Nov 2018 05:51:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088270; bh=HdQw+5WkU8qh4QVSfkqGiYGxAIypFX1jiXQeWfoUlIU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DcEYEchSRjY6V75cFYwh3VdfMMWuJynN7OqPFTZw7fF4ps5gwpCxoSD3xzvATZSPX RZUQ0wbrp0xNLJZHgdzQVqZf/awudf90EKa+AMBM6mgTG9L6VKBeDj2Nzhs7OaFeET vaKzB8PG92eEPOwg5IRiHYTBprJkiR5sSUeM5hRw= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jan Kara , Ross Zwisler , Dan Williams , Dave Jiang , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 11/39] mm: Fix warning in insert_pfn() Date: Tue, 13 Nov 2018 00:50:25 -0500 Message-Id: <20181113055053.78352-11-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jan Kara [ Upstream commit f2c57d91b0d96aa13ccff4e3b178038f17b00658 ] In DAX mode a write pagefault can race with write(2) in the following way: CPU0 CPU1 write fault for mapped zero page (hole) dax_iomap_rw() iomap_apply() xfs_file_iomap_begin() - allocates blocks dax_iomap_actor() invalidate_inode_pages2_range() - invalidates radix tree entries in given range dax_iomap_pte_fault() grab_mapping_entry() - no entry found, creates empty ... xfs_file_iomap_begin() - finds already allocated block ... vmf_insert_mixed_mkwrite() - WARNs and does nothing because there is still zero page mapped in PTE unmap_mapping_pages() This race results in WARN_ON from insert_pfn() and is occasionally triggered by fstest generic/344. Note that the race is otherwise harmless as before write(2) on CPU0 is finished, we will invalidate page tables properly and thus user of mmap will see modified data from write(2) from that point on. So just restrict the warning only to the case when the PFN in PTE is not zero page. Link: http://lkml.kernel.org/r/20180824154542.26872-1-jack@suse.cz Signed-off-by: Jan Kara Reviewed-by: Andrew Morton Cc: Ross Zwisler Cc: Dan Williams Cc: Dave Jiang Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f94feec6518d..1f5a59204101 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1803,10 +1803,15 @@ static int insert_pfn(struct vm_area_struct *vma, unsigned long addr, * in may not match the PFN we have mapped if the * mapped PFN is a writeable COW page. In the mkwrite * case we are creating a writable PTE for a shared - * mapping and we expect the PFNs to match. + * mapping and we expect the PFNs to match. If they + * don't match, we are likely racing with block + * allocation and mapping invalidation so just skip the + * update. */ - if (WARN_ON_ONCE(pte_pfn(*pte) != pfn_t_to_pfn(pfn))) + if (pte_pfn(*pte) != pfn_t_to_pfn(pfn)) { + WARN_ON_ONCE(!is_zero_pfn(pte_pfn(*pte))); goto out_unlock; + } entry = *pte; goto out_mkwrite; } else From patchwork Tue Nov 13 05:50:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679585 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0733113BB for ; Tue, 13 Nov 2018 05:51:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9CC32A2E8 for ; Tue, 13 Nov 2018 05:51:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DDB5B2A329; Tue, 13 Nov 2018 05:51:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73D342A2E8 for ; Tue, 13 Nov 2018 05:51:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DC946B0277; Tue, 13 Nov 2018 00:51:17 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 48D946B0278; Tue, 13 Nov 2018 00:51:17 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 330396B0279; Tue, 13 Nov 2018 00:51:17 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id E1EF06B0277 for ; Tue, 13 Nov 2018 00:51:16 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id d6-v6so9549911pfn.19 for ; Mon, 12 Nov 2018 21:51:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=FuQBYXufUEeJ4TF8GEYL6ZrXKpuhuxW60MKqDeJz8eg=; b=bCaqoYWLE8LBJD8udwwKqrCAW0u1VBddlM9B/AiIk0fYY24RNanrF0UyaxOYEdDuru V9tJtq2aYAqdnV+Kh29oUKPtQCPOcabsKRLKAQ/KNvIQrKYFdajAlRrn6tEAmih/+BEv 0tupet2kbigXFwHE6iNo3oDNtAMutsnlMdiM/hr0h4bbu+Y47MVioS+CmGZErf9oGbCg DP+ElFvisenxuvYY7tjzClyogvuO/aS+klTuZRVPsNbuXPpgcTXoNqjnLNJ8cNSidt/b 6sB1/rz4IYlkCjCJq/Wzf0KdT3e1xWQFR9GZNlyBtLzbnf00S7YGq7ncbtsS2MlPA7e/ 2J1Q== X-Gm-Message-State: AGRZ1gJsB5SAui1cvz2OLrXQzYSc2zyj7UjEj6cW55KfDajVHb95kbgz bPxZVVdbMNaEfVo1l9NzQhIWH9TK1oaTQe76fgiRiuCPJQ+bQWoy0kW5EnWsta4XjILiGnnF3G2 pKOr/TGrxfYrFBVIekqP2CO4RLRXp9w3ii4MVbgiHVa6DUoAFZpIvOEvBhk6HO+w1pw== X-Received: by 2002:a17:902:d206:: with SMTP id t6mr3790117ply.193.1542088276571; Mon, 12 Nov 2018 21:51:16 -0800 (PST) X-Google-Smtp-Source: AJdET5eeCqF4tnB1ApXzqQZdQ84luolWfi8twgLnNbaGWHUlaSNUdBrsiTJJCh9ZysYvsE8YGZei X-Received: by 2002:a17:902:d206:: with SMTP id t6mr3790072ply.193.1542088275588; Mon, 12 Nov 2018 21:51:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088275; cv=none; d=google.com; s=arc-20160816; b=gK+WAKIpMRTfRbCE9OUjE9V9oCjEmTMSk1f7sSo+gn+BFplOR63U2Qf+HLx+jQjg8t qLb4GMiALlI4RgYfFZ8DvRAboEpJDvEmzOC/cmWSdJ4o/fu3Tgz0MJQQponB7HD7Oyo7 nmv6tKqjVEzf4RLsAUEpTHrqa+EYPuB4HdY2oHCWPDXqJGErmpXr5nlwlLBoYJyVLs4i nTBK2Obz/VlF2PIZ8esVutBaye/AqG1fZIkOiTU4byD5v1zB6BpI+/NZRERSyCpAC3Hg bbDh1RJrnpvZZ4BC2shhuMcSrtgGBQeSB2pKlNOyFaDd8qLX8kvWoJEtNlBFHV4hCL5m +8cA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FuQBYXufUEeJ4TF8GEYL6ZrXKpuhuxW60MKqDeJz8eg=; b=gZHm5TUfM5y9tNBsVzC0ZIr2DAtpY7Lw+y1eB1VBcbYOw9s47zxIRDjPBJ4HF9SnJr gMFjpzD1BrDW4E3u+LfKqUd3mD/fYpho7C6lx5D7aszgCFRvvnT5aFsyCVXznWCyDkQW 1ytJDYNEh+dl15HCFB8QCFBa8nkRlUnkhX+skRaB2d0pyBMzk+WqOd42Lpi7h0ODXyzc Mve3b+WCkTl6zwrVT1dWm6HjMGE3W+UWJeA+nqgfLAOPteJGuL5CqBZq4pCSw/VmTLy+ i0df8b9NR4cQLjEYtbfRexnlkTWK2VumRw+Q2X3w8VYfD+2iteXya3u+UEJQdGvllpcA W+VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b="IX/o61lK"; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id b25-v6si2812906pfo.240.2018.11.12.21.51.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:15 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b="IX/o61lK"; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D514F2250F; Tue, 13 Nov 2018 05:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088275; bh=t4xw4zlEwLYczh+tpwXn3PSD2ZBiO8xxxSHzfaUXxr0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IX/o61lKrqvJ8V8Tk6JSOz8D43+xJ/W0f9sixpW2PuHmLPdJj66wmrMJ70xHj8tBK QKJyR01GRHXRo2jri4C5V3jv4RCjy0XwiYPhmNq4RWlRnVERb+rSKXVxA1KZdKrbaE xg63cHY4MmXFW4NfxI41Hwn55gOQOo1ANq0yHdao= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , "Rafael J. Wysocki" , Len Brown , Greg Kroah-Hartman , Boris Ostrovsky , Juergen Gross , Nathan Fontenot , John Allen , Michal Hocko , Dan Williams , Joonsoo Kim , Vlastimil Babka , Mathieu Malaterre , Pavel Tatashin , YASUAKI ISHIMATSU , Balbir Singh , Haiyang Zhang , Heiko Carstens , Jonathan Corbet , Kate Stewart , "K. Y. Srinivasan" , Martin Schwidefsky , Michael Neuling , Philippe Ombredanne , Stephen Hemminger , Thomas Gleixner , Andrew Morton , Linus Torvalds , Sasha Levin , linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 12/39] mm/memory_hotplug: make add_memory() take the device_hotplug_lock Date: Tue, 13 Nov 2018 00:50:26 -0500 Message-Id: <20181113055053.78352-12-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: David Hildenbrand [ Upstream commit 8df1d0e4a265f25dc1e7e7624ccdbcb4a6630c89 ] add_memory() currently does not take the device_hotplug_lock, however is aleady called under the lock from arch/powerpc/platforms/pseries/hotplug-memory.c drivers/acpi/acpi_memhotplug.c to synchronize against CPU hot-remove and similar. In general, we should hold the device_hotplug_lock when adding memory to synchronize against online/offline request (e.g. from user space) - which already resulted in lock inversions due to device_lock() and mem_hotplug_lock - see 30467e0b3be ("mm, hotplug: fix concurrent memory hot-add deadlock"). add_memory()/add_memory_resource() will create memory block devices, so this really feels like the right thing to do. Holding the device_hotplug_lock makes sure that a memory block device can really only be accessed (e.g. via .online/.state) from user space, once the memory has been fully added to the system. The lock is not held yet in drivers/xen/balloon.c arch/powerpc/platforms/powernv/memtrace.c drivers/s390/char/sclp_cmd.c drivers/hv/hv_balloon.c So, let's either use the locked variants or take the lock. Don't export add_memory_resource(), as it once was exported to be used by XEN, which is never built as a module. If somebody requires it, we also have to export a locked variant (as device_hotplug_lock is never exported). Link: http://lkml.kernel.org/r/20180925091457.28651-3-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Pavel Tatashin Reviewed-by: Rafael J. Wysocki Reviewed-by: Rashmica Gupta Reviewed-by: Oscar Salvador Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: "Rafael J. Wysocki" Cc: Len Brown Cc: Greg Kroah-Hartman Cc: Boris Ostrovsky Cc: Juergen Gross Cc: Nathan Fontenot Cc: John Allen Cc: Michal Hocko Cc: Dan Williams Cc: Joonsoo Kim Cc: Vlastimil Babka Cc: Mathieu Malaterre Cc: Pavel Tatashin Cc: YASUAKI ISHIMATSU Cc: Balbir Singh Cc: Haiyang Zhang Cc: Heiko Carstens Cc: Jonathan Corbet Cc: Kate Stewart Cc: "K. Y. Srinivasan" Cc: Martin Schwidefsky Cc: Michael Neuling Cc: Philippe Ombredanne Cc: Stephen Hemminger Cc: Thomas Gleixner Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- .../platforms/pseries/hotplug-memory.c | 2 +- drivers/acpi/acpi_memhotplug.c | 2 +- drivers/base/memory.c | 9 ++++++-- drivers/xen/balloon.c | 3 +++ include/linux/memory_hotplug.h | 1 + mm/memory_hotplug.c | 22 ++++++++++++++++--- 6 files changed, 32 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c index c1578f54c626..79e074eac486 100644 --- a/arch/powerpc/platforms/pseries/hotplug-memory.c +++ b/arch/powerpc/platforms/pseries/hotplug-memory.c @@ -702,7 +702,7 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb) nid = memory_add_physaddr_to_nid(lmb->base_addr); /* Add the memory */ - rc = add_memory(nid, lmb->base_addr, block_sz); + rc = __add_memory(nid, lmb->base_addr, block_sz); if (rc) { dlpar_remove_device_tree_lmb(lmb); return rc; diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c index 6b0d3ef7309c..2ccfbb61ca89 100644 --- a/drivers/acpi/acpi_memhotplug.c +++ b/drivers/acpi/acpi_memhotplug.c @@ -228,7 +228,7 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) if (node < 0) node = memory_add_physaddr_to_nid(info->start_addr); - result = add_memory(node, info->start_addr, info->length); + result = __add_memory(node, info->start_addr, info->length); /* * If the memory block has been used by the kernel, add_memory() diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 622ab8edc035..7307a173dca8 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -519,15 +519,20 @@ memory_probe_store(struct device *dev, struct device_attribute *attr, if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL; + ret = lock_device_hotplug_sysfs(); + if (ret) + goto out; + nid = memory_add_physaddr_to_nid(phys_addr); - ret = add_memory(nid, phys_addr, - MIN_MEMORY_BLOCK_SIZE * sections_per_block); + ret = __add_memory(nid, phys_addr, + MIN_MEMORY_BLOCK_SIZE * sections_per_block); if (ret) goto out; ret = count; out: + unlock_device_hotplug(); return ret; } diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index 065f0b607373..6e39a0c10f93 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -401,7 +401,10 @@ static enum bp_state reserve_additional_memory(void) * callers drop the mutex before trying again. */ mutex_unlock(&balloon_mutex); + /* add_memory_resource() requires the device_hotplug lock */ + lock_device_hotplug(); rc = add_memory_resource(nid, resource, memhp_auto_online); + unlock_device_hotplug(); mutex_lock(&balloon_mutex); if (rc) { diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 4e9828cda7a2..5310945b50bb 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -321,6 +321,7 @@ static inline void remove_memory(int nid, u64 start, u64 size) {} extern int walk_memory_range(unsigned long start_pfn, unsigned long end_pfn, void *arg, int (*func)(struct memory_block *, void *)); +extern int __add_memory(int nid, u64 start, u64 size); extern int add_memory(int nid, u64 start, u64 size); extern int add_memory_resource(int nid, struct resource *resource, bool online); extern int arch_add_memory(int nid, u64 start, u64 size, diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 785252397e35..bdaaa450d83f 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1095,7 +1095,12 @@ static int online_memory_block(struct memory_block *mem, void *arg) return device_online(&mem->dev); } -/* we are OK calling __meminit stuff here - we have CONFIG_MEMORY_HOTPLUG */ +/* + * NOTE: The caller must call lock_device_hotplug() to serialize hotplug + * and online/offline operations (triggered e.g. by sysfs). + * + * we are OK calling __meminit stuff here - we have CONFIG_MEMORY_HOTPLUG + */ int __ref add_memory_resource(int nid, struct resource *res, bool online) { u64 start, size; @@ -1188,9 +1193,9 @@ int __ref add_memory_resource(int nid, struct resource *res, bool online) mem_hotplug_done(); return ret; } -EXPORT_SYMBOL_GPL(add_memory_resource); -int __ref add_memory(int nid, u64 start, u64 size) +/* requires device_hotplug_lock, see add_memory_resource() */ +int __ref __add_memory(int nid, u64 start, u64 size) { struct resource *res; int ret; @@ -1204,6 +1209,17 @@ int __ref add_memory(int nid, u64 start, u64 size) release_memory_resource(res); return ret; } + +int add_memory(int nid, u64 start, u64 size) +{ + int rc; + + lock_device_hotplug(); + rc = __add_memory(nid, start, size); + unlock_device_hotplug(); + + return rc; +} EXPORT_SYMBOL_GPL(add_memory); #ifdef CONFIG_MEMORY_HOTREMOVE From patchwork Tue Nov 13 05:50:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB6BF13BB for ; Tue, 13 Nov 2018 05:51:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BDEFC2A278 for ; Tue, 13 Nov 2018 05:51:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC5862A329; Tue, 13 Nov 2018 05:51:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 480A32A2EA for ; Tue, 13 Nov 2018 05:51:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0D2A6B0279; Tue, 13 Nov 2018 00:51:24 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9BCA36B027A; Tue, 13 Nov 2018 00:51:24 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AD036B027B; Tue, 13 Nov 2018 00:51:24 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 4BBAF6B0279 for ; Tue, 13 Nov 2018 00:51:24 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id a72-v6so9564959pfj.14 for ; Mon, 12 Nov 2018 21:51:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=T/FexRWnLKADXhFxi2gzLlqktSrjQHqWaORBypfakLg=; b=aYL291bmmQQnjFB48/i/3xNJs5uJb0Et4nWFdynzGXwX1yxQ1iJ+TT7Y2r0hIV3qsh ewffPzA6sOlI+eqSwqJE5PjiT8ku1OfwANbIHTyF6VEyLtwcDNp52TPeaVARVgOK62Yd k0OqAWitD3mKFbWDKDI3X+mphXY16Q7F1aBAEKXa8YJ2/X737FHsOYzzMGQK8spVZbeE MQDLCeHGq6OAfTk4unWCa/zbcs1qGXTsLr8xADeyhaTJWPmF8sDzkgbZ0vVC6832YfGH TZ6aUnHHPh4FPXokwPxD9vMHX92ozoUkCc3VEOCq4vW8JNFlHmcvwwnWMV3dBw+rgVNQ AKaA== X-Gm-Message-State: AGRZ1gLEePdaoONOd+djGus39B8PAKUyVI/YgO00e68VoVrgneuluhRX B0T7dRmOF32piOUoRsj+mUkmFvIuuALyHrySJp3fKDi92g6WDTBE2hNeZuQ11CS3LzQgNJu6xPn le0prVHCaiV6Ep6cvWygGwR9TStJwda/L0E5jmj9duc4m7OQC1KSBwXrJWvRmQuv2OA== X-Received: by 2002:a63:2b01:: with SMTP id r1mr3490638pgr.432.1542088283992; Mon, 12 Nov 2018 21:51:23 -0800 (PST) X-Google-Smtp-Source: AJdET5db/++svXA6y3xRXcraytozxzIx9HdRnkoUxQV1h9fiRk4HmaGtciMfG4rGc+T16hDFPCcs X-Received: by 2002:a63:2b01:: with SMTP id r1mr3490592pgr.432.1542088283211; Mon, 12 Nov 2018 21:51:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088283; cv=none; d=google.com; s=arc-20160816; b=TUlhgAGIWtelk68dhOwGR95Ka/ARAXuibclo6PLQ1aZkvAyHqvphwkRjzIBUTQP6C4 lln9jAZBzrJou8SvuVcoRm/DbCACXkgaru7uQ8qPWHsCDsgddmfMCIuC40Gt8jvK2zdF AAeZbqYuCCSb+nP8GOoltDe2FMoDEG4y1rShXbiE4m40TYftTqPWQfwlcc3/e+i5vX/K G84Uc9ycY+BXrKvrnCPhrjbBWZhsVpJDicYZtrJALQsCPlIBF+9wmt7fi67Lk8EzehjA 6ILUldrFnRLAj2l0UguK+vSt9zaNiDwA3MWaNyCBT5Ju9mGR6cSe1pm/CiDpia8iOK5+ OGXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T/FexRWnLKADXhFxi2gzLlqktSrjQHqWaORBypfakLg=; b=CW/WwE1g2J+IJ/TYtrjDCzNVJvXV9w0FIHzNQ1o6y3Uux2eAZgB674JGuDpPgRqIrd g1e+6nuMunFB/+lDGmk4AYWaAbQXIeojlMTk/sioL0XCR9Q5utUsaxH3Mze6Zl+yEUyH BjlUs7ER1w42ilH3Q6JIAEL8uJ1GXZbNSp4eIjFDVC6lQbi1FaF+cPMvUE5y/9mTRMTa vtXuas1hgOevaFKdO8Kdrx9s4qKFYw+VcqlKggYxDX+NZBglvw5JpzLgRxCBy2JXOQtV royIYarKIgP4HxJV6ENrfUGh4WxOnwixwFv2OY2EGPK0LXeAysddWTcR+bCeTui12QNo BGag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=HvdJ+1aV; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id e7si2519519pgv.499.2018.11.12.21.51.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:23 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=HvdJ+1aV; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AD7CC2250E; Tue, 13 Nov 2018 05:51:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088282; bh=7mQEjIV2IFnljtef90NPlS5B4LarDbNuW7HZDt0nVlo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HvdJ+1aV8KYlkDGo8BjZILp8xpzc7dsasGNKaxZgxO5atzzcy0aHEvMJ9xINP0+uy 1xFS3GDEotbQpAGkpZdOy8MoVmYwPPIiQ1u+Nt2RbUG4n+domF6iy7tjQmijMV2p4Y GD0fZ5roKQSDa12pwgmoXXPrxQUCGLAaRL6gSWLo= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Dan Carpenter , Stephen Rothwell , Keith Busch , "Michael S. Tsirkin" , Kees Cook , YueHaibing , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 16/39] mm/gup_benchmark.c: prevent integer overflow in ioctl Date: Tue, 13 Nov 2018 00:50:30 -0500 Message-Id: <20181113055053.78352-16-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Dan Carpenter [ Upstream commit 4b408c74ee5a0b74fc9265c2fe39b0e7dec7c056 ] The concern here is that "gup->size" is a u64 and "nr_pages" is unsigned long. On 32 bit systems we could trick the kernel into allocating fewer pages than expected. Link: http://lkml.kernel.org/r/20181025061546.hnhkv33diogf2uis@kili.mountain Fixes: 64c349f4ae78 ("mm: add infrastructure for get_user_pages_fast() benchmarking") Signed-off-by: Dan Carpenter Acked-by: Kirill A. Shutemov Reviewed-by: Andrew Morton Cc: Stephen Rothwell Cc: Keith Busch Cc: "Michael S. Tsirkin" Cc: Kees Cook Cc: YueHaibing Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/gup_benchmark.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/gup_benchmark.c b/mm/gup_benchmark.c index 7405c9d89d65..7e6f2d2dafb5 100644 --- a/mm/gup_benchmark.c +++ b/mm/gup_benchmark.c @@ -23,6 +23,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, int nr; struct page **pages; + if (gup->size > ULONG_MAX) + return -EINVAL; + nr_pages = gup->size / PAGE_SIZE; pages = kvcalloc(nr_pages, sizeof(void *), GFP_KERNEL); if (!pages) From patchwork Tue Nov 13 05:50:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679593 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9950213BB for ; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 895922A264 for ; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7DD4D2A305; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF6FD2A372 for ; Tue, 13 Nov 2018 05:51:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32F746B027B; Tue, 13 Nov 2018 00:51:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2DF886B027C; Tue, 13 Nov 2018 00:51:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CFE36B027D; Tue, 13 Nov 2018 00:51:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id CE5EE6B027B for ; Tue, 13 Nov 2018 00:51:38 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id 94-v6so8773785pla.5 for ; Mon, 12 Nov 2018 21:51:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=E7aKex6/K1LOcebDiqEWvP243L5GuhYc4GuDE6fnM9Y=; b=C5k23FRKoB01//Kt/kQRujdJYzAEzxD21e+md8zbKdvl+TnOOjVFvYn/XXysbNm297 kX6U+axxhVZenfWliLkjd/gqwqHZV3YnvOz6VdUKtcRAXR2kkBqxjAwufsMeNi0JXzM+ LLkd7czrwJoDH2556OtEKf4SWRCVHRZWU/j95scOLdV71Py4VlPP8lqt7+uHkKYH8LkD dWRXz/fP11gqUJw4V1adWqdWXdoO3Hs1c5d8m7d7Lg4Ua12Gd00T+v1c6Iw+8P4VaGCW 3M3ck9VC7UF2q6srUhQzskbVSZ7qXBLcLfjxgO5tbhG9n6NRHcNaQ9N1x5KahT7uEotG Q4Hg== X-Gm-Message-State: AGRZ1gJaG7Np2nHvsNBVHbFiNpEEr+9BKFiBLBXH1EaDVMmWKzV6+xRZ c5F+8nLUYgkh2J+9XCXCb81IdbeKvM0ZBXBPcuDN5q6Zvvk7LWMvvfYFoYca8wOrZ2DYy/mBHH5 v+XTs5d4oHp7DIbVN++kghwSMK1VLi4oeEJU31grsf3Yh6rsg642iCLKcBcouJUzi7g== X-Received: by 2002:a63:1258:: with SMTP id 24mr3490259pgs.114.1542088298473; Mon, 12 Nov 2018 21:51:38 -0800 (PST) X-Google-Smtp-Source: AJdET5eBKn5zXACZFJTgSbY3V3WlXbV5uWkfkS6Qnxa3o9/qiYqB2UeRRmgfMaXlr2h49a0Gdgfu X-Received: by 2002:a63:1258:: with SMTP id 24mr3490228pgs.114.1542088297456; Mon, 12 Nov 2018 21:51:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088297; cv=none; d=google.com; s=arc-20160816; b=ctCj3MwxcVrQicRv2O1JACJME+OiQ0ztLZym2GvRv4H343vDr0qgDgHKwaS5Ffd8HN FmO5kYwLukJgAYjoQUrW8/PBQ9yM1aBdh/sWkFyuMhXzfehihxJjd9NoIzO0kZgS+HXW FcDQLgzmbDBMrWR255+a1JYqyWau0Nvr1J08x1UrttNdyu3Vgqkxigg6N2bCyXZHX1Cd mo7qSA5x2txYXfMzfgZGBeWytR3KwY3a7oEfpAFIinSqsTeVifgdxzzalPuCUfHHNmq4 eVO+nnh3rFF9F1fFfCx4yqDOQHuxVAdqUOpiJ3MKWAxz5UPAqG+4ngQff8MdGz4uB0ab r9TA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=E7aKex6/K1LOcebDiqEWvP243L5GuhYc4GuDE6fnM9Y=; b=Qdtt79clEdGgBRGBWj1xr+cPVNv/aVpXJHU00aVOgnOMA9TnNT8KCptkTklLMtykLX zulzJTbZwp/kvvqcWe1N1K+pCZWkImseoih20COl/cHtNHsuPcPfed/vtSywd9pX4lAw TQGtaaGxob1qMKc+mXdbDmiooDfAJDN1aJp4CabU0Gf3NWS76gFSIfHQr1pS1qZBxNA6 gsv9fjh2jlytNJa22qng7Ug82GDHRi9yVEoyx5kXtNtu3uHYwIHVRASDvDROp2NvDBSZ zFs2K+JUjX65ztjSl2iDDeXmuAPrgfByFrZxXmunWf4/sPPnD60N7sfnQEQgCiUM0dNG lwFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=skOW9FC4; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id m14si18356640pgd.326.2018.11.12.21.51.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:37 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=skOW9FC4; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0820522510; Tue, 13 Nov 2018 05:51:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088297; bh=N6As6j6xCoPMaQzs4R/SYfp9cNiF7iuDlMLJPddbhIE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=skOW9FC48TgHb3LX9LE2GNXi9bg+8QOZPZx3WWr6IThlhoDplE/3fTrOxztxe+zRd VcyHQL/q7oBVOZx/G8L5CqIwNNsrC+rFnHJPiRk/OIPWJPitraZfpcwB0QfZZqNroW 3kbdOQpIKyY4K4lIiGOJ/aVaomg8v4hN2WHggjVA= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Jerome Glisse , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 33/39] mm: thp: fix MADV_DONTNEED vs migrate_misplaced_transhuge_page race condition Date: Tue, 13 Nov 2018 00:50:47 -0500 Message-Id: <20181113055053.78352-33-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andrea Arcangeli [ Upstream commit d7c3393413fe7e7dc54498ea200ea94742d61e18 ] Patch series "migrate_misplaced_transhuge_page race conditions". Aaron found a new instance of the THP MADV_DONTNEED race against pmdp_clear_flush* variants, that was apparently left unfixed. While looking into the race found by Aaron, I may have found two more issues in migrate_misplaced_transhuge_page. These race conditions would not cause kernel instability, but they'd corrupt userland data or leave data non zero after MADV_DONTNEED. I did only minor testing, and I don't expect to be able to reproduce this (especially the lack of ->invalidate_range before migrate_page_copy, requires the latest iommu hardware or infiniband to reproduce). The last patch is noop for x86 and it needs further review from maintainers of archs that implement flush_cache_range() (not in CC yet). To avoid confusion, it's not the first patch that introduces the bug fixed in the second patch, even before removing the pmdp_huge_clear_flush_notify, that _notify suffix was called after migrate_page_copy already run. This patch (of 3): This is a corollary of ced108037c2aa ("thp: fix MADV_DONTNEED vs. numa balancing race"), 58ceeb6bec8 ("thp: fix MADV_DONTNEED vs. MADV_FREE race") and 5b7abeae3af8c ("thp: fix MADV_DONTNEED vs clear soft dirty race). When the above three fixes where posted Dave asked https://lkml.kernel.org/r/929b3844-aec2-0111-fef7-8002f9d4e2b9@intel.com but apparently this was missed. The pmdp_clear_flush* in migrate_misplaced_transhuge_page() was introduced in a54a407fbf7 ("mm: Close races between THP migration and PMD numa clearing"). The important part of such commit is only the part where the page lock is not released until the first do_huge_pmd_numa_page() finished disarming the pagenuma/protnone. The addition of pmdp_clear_flush() wasn't beneficial to such commit and there's no commentary about such an addition either. I guess the pmdp_clear_flush() in such commit was added just in case for safety, but it ended up introducing the MADV_DONTNEED race condition found by Aaron. At that point in time nobody thought of such kind of MADV_DONTNEED race conditions yet (they were fixed later) so the code may have looked more robust by adding the pmdp_clear_flush(). This specific race condition won't destabilize the kernel, but it can confuse userland because after MADV_DONTNEED the memory won't be zeroed out. This also optimizes the code and removes a superfluous TLB flush. [akpm@linux-foundation.org: reflow comment to 80 cols, fix grammar and typo (beacuse)] Link: http://lkml.kernel.org/r/20181013002430.698-2-aarcange@redhat.com Signed-off-by: Andrea Arcangeli Reported-by: Aaron Tomlin Acked-by: Mel Gorman Acked-by: Kirill A. Shutemov Cc: Jerome Glisse Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/migrate.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index f49eb9589d73..38ad6365ed10 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2086,15 +2086,26 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); /* - * Clear the old entry under pagetable lock and establish the new PTE. - * Any parallel GUP will either observe the old page blocking on the - * page lock, block on the page table lock or observe the new page. - * The SetPageUptodate on the new page and page_add_new_anon_rmap - * guarantee the copy is visible before the pagetable update. + * Overwrite the old entry under pagetable lock and establish + * the new PTE. Any parallel GUP will either observe the old + * page blocking on the page lock, block on the page table + * lock or observe the new page. The SetPageUptodate on the + * new page and page_add_new_anon_rmap guarantee the copy is + * visible before the pagetable update. */ flush_cache_range(vma, mmun_start, mmun_end); page_add_anon_rmap(new_page, vma, mmun_start, true); - pmdp_huge_clear_flush_notify(vma, mmun_start, pmd); + /* + * At this point the pmd is numa/protnone (i.e. non present) and the TLB + * has already been flushed globally. So no TLB can be currently + * caching this non present pmd mapping. There's no need to clear the + * pmd before doing set_pmd_at(), nor to flush the TLB after + * set_pmd_at(). Clearing the pmd here would introduce a race + * condition against MADV_DONTNEED, because MADV_DONTNEED only holds the + * mmap_sem for reading. If the pmd is set to NULL at any given time, + * MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this + * pmd. + */ set_pmd_at(mm, mmun_start, pmd, entry); update_mmu_cache_pmd(vma, address, &entry); @@ -2108,7 +2119,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, * No need to double call mmu_notifier->invalidate_range() callback as * the above pmdp_huge_clear_flush_notify() did already call it. */ - mmu_notifier_invalidate_range_only_end(mm, mmun_start, mmun_end); + mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end); /* Take an "isolate" reference and put new page on the LRU. */ get_page(new_page); From patchwork Tue Nov 13 05:50:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679595 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9576F13B5 for ; Tue, 13 Nov 2018 05:51:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 83ACE2A2B7 for ; Tue, 13 Nov 2018 05:51:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81F3F2A33F; Tue, 13 Nov 2018 05:51:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACB422A2B7 for ; Tue, 13 Nov 2018 05:51:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52F2A6B027D; Tue, 13 Nov 2018 00:51:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4B3846B027E; Tue, 13 Nov 2018 00:51:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E0906B027F; Tue, 13 Nov 2018 00:51:42 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id D56E06B027D for ; Tue, 13 Nov 2018 00:51:41 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id 34-v6so8775761plf.6 for ; Mon, 12 Nov 2018 21:51:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=9SVzJUwv2InvqBXn2gVLt+voypV4GC2eUO8m8FuGdA8=; b=OmqdxLwyehBhcK3kZgesXyKsHhjhWXmzGuQjSM+N0rMNWQWqgOtfAtPP7dTHrRLD2Q bh/uUYZrmKguSlAoigrvgc+lcqvLS9HS8EvQsdiR1SJpm0AxPissSeckWS7qVwiy3e8w UIpfQTkCXx94Tl1lQ/d/KgQH2CxZ6YK7bSuEgDyFF1VqReFkP79Aa2QXReg6Q7O2j2Tl 1G4ho2E4IWljJX164kq1fz4fReH6uosr9tNLvgUQQs66H3cVEFuryPWrWKBCmVtdZ/D4 93P80Z8fjWEA7bRSbJUoXBEf8R90i5PqB0KqDpZQxi/bvwQTzZE3cynDnDu/qgZRATK3 UzNA== X-Gm-Message-State: AGRZ1gIhL0QEUxMLR5Ss8j53qrV7AOZcKH1aH0X8+woQknXktZSOzuey /TaVki5z+ugvf/HI1qaT/4OaLDa54Dp8LzNiavnTEEjnpwMaBMIrH4dz3ObvmEDJa5aUXh40JX6 IIiwKpiDGbTlUV79NobR/GptSzWmxL0TJrgvaizitV5G9tnvmVWXv7qwmSlWAUaGvHA== X-Received: by 2002:a63:e545:: with SMTP id z5mr3548341pgj.195.1542088301507; Mon, 12 Nov 2018 21:51:41 -0800 (PST) X-Google-Smtp-Source: AJdET5da0EcvNGyraEyJlqr07ziVd8jm90zGoyYakaNVZCxAhkoQV95JxpF9H3VagDJSD3UGF5P5 X-Received: by 2002:a63:e545:: with SMTP id z5mr3548303pgj.195.1542088300369; Mon, 12 Nov 2018 21:51:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088300; cv=none; d=google.com; s=arc-20160816; b=hCIiFRc+g7Sz6LssIuNvLvWhVDO4aCW/70AFTSUOyk6g3vmdXoTwTqfkne6oG+vAM5 ctzZKG80j0Ys6qNRo7mO5RwVLNLhyKQWfTkJWtKDN72JoNW+45BLpkPlP+qQbMYAoUNf miLGKISZY/rKQvR2+7Kw/tDQehb/DmR6LbK7WHt1nWm7KFxmv7FZxfSq5mhMLIIAMtIS 5uY9psnB1aew1ovCVxNdEWYIVftIiYpQJUsKo60d91LzmW4qOZL0NW9f0UCGIXc4kY8q +xYq4IN/5ObDz6s5Kd9ulB+lOPpAJVjhPKV6nljyHYLTnf++4yxwbehJ1Ik8VqibsbkY BOkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9SVzJUwv2InvqBXn2gVLt+voypV4GC2eUO8m8FuGdA8=; b=gN5tgZWKRqSuj1bqwL6scmTkLX7FMrOKuQdOoQuI9t5PyaFPPQwQ4v6vZ3DH5Z7FWf u+g2s1iYGiSpK+XSMA2w40732Y7Kua4J9vWwkhJOJ6c2+S3+Y1b0K7pjU8cqQauugewu 7d8Bo6I982NUsKDwhkcJf/ASsY/ELVCN2LrOZbhGiSP+xAqaLp+Ify6UE60dcn6seg7L vEhihEGsPpRoLHIyDkn344VSa/ZjCJ3uqtwUrIt03ody5NmvE/dGZE1SrurIPyAieFCb U9IMJfXHKDvbJPjdcbNkI79c5J5Ssb3AeXhYZh/uGnRrmrVCOfG96mklYMMWP2gs2cmC OR4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=2gp4l0aD; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id h80-v6si17091133pfj.112.2018.11.12.21.51.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:40 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=2gp4l0aD; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EA8A42251C; Tue, 13 Nov 2018 05:51:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088300; bh=mKnrCYdD2BT619GhtKJguAHMUTkzdAjXuz7ndxIz5CQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=2gp4l0aD/oUvqhD6CxdhV33rWILMENHK5Hk85a291glk0AAAcHCK7DQGDxVkEaKC2 csOBsukQSrVXQ0NpBfOaKRtDkPRxLY9Lv+8j4hvOIBNvEinV1rnGh/HmOv8zazb34+ bHyp+1z9Z/yhM98PcTJr6Nj1E80MmG2lrIOicg58= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Jerome Glisse , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 34/39] mm: thp: fix mmu_notifier in migrate_misplaced_transhuge_page() Date: Tue, 13 Nov 2018 00:50:48 -0500 Message-Id: <20181113055053.78352-34-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andrea Arcangeli [ Upstream commit 7066f0f933a1fd707bb38781866657769cff7efc ] change_huge_pmd() after arming the numa/protnone pmd doesn't flush the TLB right away. do_huge_pmd_numa_page() flushes the TLB before calling migrate_misplaced_transhuge_page(). By the time do_huge_pmd_numa_page() runs some CPU could still access the page through the TLB. change_huge_pmd() before arming the numa/protnone transhuge pmd calls mmu_notifier_invalidate_range_start(). So there's no need of mmu_notifier_invalidate_range_start()/mmu_notifier_invalidate_range_only_end() sequence in migrate_misplaced_transhuge_page() too, because by the time migrate_misplaced_transhuge_page() runs, the pmd mapping has already been invalidated in the secondary MMUs. It has to or if a secondary MMU can still write to the page, the migrate_page_copy() would lose data. However an explicit mmu_notifier_invalidate_range() is needed before migrate_misplaced_transhuge_page() starts copying the data of the transhuge page or the below can happen for MMU notifier users sharing the primary MMU pagetables and only implementing ->invalidate_range: CPU0 CPU1 GPU sharing linux pagetables using only ->invalidate_range ----------- ------------ --------- GPU secondary MMU writes to the page mapped by the transhuge pmd change_pmd_range() mmu..._range_start() ->invalidate_range_start() noop change_huge_pmd() set_pmd_at(numa/protnone) pmd_unlock() do_huge_pmd_numa_page() CPU TLB flush globally (1) CPU cannot write to page migrate_misplaced_transhuge_page() GPU writes to the page... migrate_page_copy() ...GPU stops writing to the page CPU TLB flush (2) mmu..._range_end() (3) ->invalidate_range_stop() noop ->invalidate_range() GPU secondary MMU is invalidated and cannot write to the page anymore (too late) Just like we need a CPU TLB flush (1) because the TLB flush (2) arrives too late, we also need a mmu_notifier_invalidate_range() before calling migrate_misplaced_transhuge_page(), because the ->invalidate_range() in (3) also arrives too late. This requirement is the result of the lazy optimization in change_huge_pmd() that releases the pmd_lock without first flushing the TLB and without first calling mmu_notifier_invalidate_range(). Even converting the removed mmu_notifier_invalidate_range_only_end() into a mmu_notifier_invalidate_range_end() would not have been enough to fix this, because it run after migrate_page_copy(). After the hugepage data copy is done migrate_misplaced_transhuge_page() can proceed and call set_pmd_at without having to flush the TLB nor any secondary MMUs because the secondary MMU invalidate, just like the CPU TLB flush, has to happen before the migrate_page_copy() is called or it would be a bug in the first place (and it was for drivers using ->invalidate_range()). KVM is unaffected because it doesn't implement ->invalidate_range(). The standard PAGE_SIZEd migrate_misplaced_page is less accelerated and uses the generic migrate_pages which transitions the pte from numa/protnone to a migration entry in try_to_unmap_one() and flushes TLBs and all mmu notifiers there before copying the page. Link: http://lkml.kernel.org/r/20181013002430.698-3-aarcange@redhat.com Signed-off-by: Andrea Arcangeli Acked-by: Mel Gorman Acked-by: Kirill A. Shutemov Reviewed-by: Aaron Tomlin Cc: Jerome Glisse Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/huge_memory.c | 14 +++++++++++++- mm/migrate.c | 19 ++++++------------- 2 files changed, 19 insertions(+), 14 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3238bb2d0c93..d8d46936e39e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1560,8 +1560,20 @@ int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) * We are not sure a pending tlb flush here is for a huge page * mapping or not. Hence use the tlb range variant */ - if (mm_tlb_flush_pending(vma->vm_mm)) + if (mm_tlb_flush_pending(vma->vm_mm)) { flush_tlb_range(vma, haddr, haddr + HPAGE_PMD_SIZE); + /* + * change_huge_pmd() released the pmd lock before + * invalidating the secondary MMUs sharing the primary + * MMU pagetables (with ->invalidate_range()). The + * mmu_notifier_invalidate_range_end() (which + * internally calls ->invalidate_range()) in + * change_pmd_range() will run after us, so we can't + * rely on it here and we need an explicit invalidate. + */ + mmu_notifier_invalidate_range(vma->vm_mm, haddr, + haddr + HPAGE_PMD_SIZE); + } /* * Migrate the THP to the requested node, returns with page unlocked diff --git a/mm/migrate.c b/mm/migrate.c index 38ad6365ed10..1d189fc0d206 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2022,8 +2022,8 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, int isolated = 0; struct page *new_page = NULL; int page_lru = page_is_file_cache(page); - unsigned long mmun_start = address & HPAGE_PMD_MASK; - unsigned long mmun_end = mmun_start + HPAGE_PMD_SIZE; + unsigned long start = address & HPAGE_PMD_MASK; + unsigned long end = start + HPAGE_PMD_SIZE; /* * Rate-limit the amount of data that is being migrated to a node. @@ -2058,11 +2058,9 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, WARN_ON(PageLRU(new_page)); /* Recheck the target PMD */ - mmu_notifier_invalidate_range_start(mm, mmun_start, mmun_end); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_same(*pmd, entry) || !page_ref_freeze(page, 2))) { spin_unlock(ptl); - mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end); /* Reverse changes made by migrate_page_copy() */ if (TestClearPageActive(new_page)) @@ -2093,8 +2091,8 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, * new page and page_add_new_anon_rmap guarantee the copy is * visible before the pagetable update. */ - flush_cache_range(vma, mmun_start, mmun_end); - page_add_anon_rmap(new_page, vma, mmun_start, true); + flush_cache_range(vma, start, end); + page_add_anon_rmap(new_page, vma, start, true); /* * At this point the pmd is numa/protnone (i.e. non present) and the TLB * has already been flushed globally. So no TLB can be currently @@ -2106,7 +2104,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, * MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this * pmd. */ - set_pmd_at(mm, mmun_start, pmd, entry); + set_pmd_at(mm, start, pmd, entry); update_mmu_cache_pmd(vma, address, &entry); page_ref_unfreeze(page, 2); @@ -2115,11 +2113,6 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, set_page_owner_migrate_reason(new_page, MR_NUMA_MISPLACED); spin_unlock(ptl); - /* - * No need to double call mmu_notifier->invalidate_range() callback as - * the above pmdp_huge_clear_flush_notify() did already call it. - */ - mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end); /* Take an "isolate" reference and put new page on the LRU. */ get_page(new_page); @@ -2144,7 +2137,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, ptl = pmd_lock(mm, pmd); if (pmd_same(*pmd, entry)) { entry = pmd_modify(entry, vma->vm_page_prot); - set_pmd_at(mm, mmun_start, pmd, entry); + set_pmd_at(mm, start, pmd, entry); update_mmu_cache_pmd(vma, address, &entry); } spin_unlock(ptl); From patchwork Tue Nov 13 05:50:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679597 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 167B013B5 for ; Tue, 13 Nov 2018 05:51:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F3BF22A312 for ; Tue, 13 Nov 2018 05:51:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE6AE2A1E4; Tue, 13 Nov 2018 05:51:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B43682A312 for ; Tue, 13 Nov 2018 05:51:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24FF66B027E; Tue, 13 Nov 2018 00:51:44 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 15A906B027F; Tue, 13 Nov 2018 00:51:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1DB56B0280; Tue, 13 Nov 2018 00:51:43 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id A43906B027E for ; Tue, 13 Nov 2018 00:51:43 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id h9so2250192pgm.1 for ; Mon, 12 Nov 2018 21:51:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=BpkhSdlmcMuUzjBF/ksvn6od71MSvZqWbWiDbsznuaw=; b=pk/IY9thWOW5SXszmzMXcSDDwW/XU/34YkuyXs+OaSvdwpvLhoEJ+7ftbYvEU6k0aE Hz6FWztQysjQO5BmUs0JaxkuPYyaVADiKfK+wippm4lF83v5AcUiekcNPU9jCUxPOzND xOULjggSG0a5pywtVRckmZNgcbIjSdxysXmijtZ51fYuuoArqyavsFRwuGVKnljfhhPU J8ZO7neSECWch1uFhlD6HyJtHcmlpGm2TtDOUUSPuvtvoE5hovcaOY0nNxzw6SfOqy/D NkQjWgfOJ63BFX6YJRr0BptfgKe0I7ZvJgAl15jxPwtt1uuCKH8G1gj70TGVRFh9q0RD UcLg== X-Gm-Message-State: AGRZ1gJxpFgtsQjfN83P4rGkaKRRA+tBF7GZK9vDvgqOhXgjEdSARmLV rxawn5Y5Dr1NSQtTvPdqamiCxxtiNE9LzqLshOqF1n2wsTbx7zaFDrPx3BfVf8N2nhG4PXw6kvS gqm3loKFDtEtjBcevKqADjoQtzA0inw9zvstRzkfpPIuVZcIcuGg2x4qQrOaxifTJxw== X-Received: by 2002:a17:902:24c:: with SMTP id 70-v6mr3624493plc.120.1542088303339; Mon, 12 Nov 2018 21:51:43 -0800 (PST) X-Google-Smtp-Source: AJdET5dadTMVbj8W2CGFwkU4hQw4vvN2C0vC6xi8SDmUFhtYD47Ima7YDPuTbVCVEBjAshr0wFT0 X-Received: by 2002:a17:902:24c:: with SMTP id 70-v6mr3624468plc.120.1542088302465; Mon, 12 Nov 2018 21:51:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088302; cv=none; d=google.com; s=arc-20160816; b=exIXrshx7hGvBidtZxBju3NxmwU6CTpeb4fYzuQ0TA2awG0RkvYiqONRBHrja0EMwK PE57LPGp1b1zxoMHxHAHFE22UvsLzTJacFwV4INrxTq8Z2ntx4BnNRfcP3YiMw/AhkFX rPn086YxrVZBHevCe5xA1u+L/niP2J2aXB1M+9KAv3D3OQFtkfYQD472vpa6ba8JdKsX f6Y0efauMN27cQeJ9NLK4OdLf/Pu8FpG3L8QGc+8cM1UkUam2FRpIU7rOHY6Z4Qcvzt6 iE285jGvfP3kC4IEBebK0lGlY95bpKv5rYOrV428IJt3VKuuj0719pozmT5wZequuEAn xIZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=BpkhSdlmcMuUzjBF/ksvn6od71MSvZqWbWiDbsznuaw=; b=LgyJfwf9Pr7fKTanSf8QGILSLV25MppXuDawd5qzVPiTV8E/wC/9qWgddMEOJbiX40 0nPLyp4/Q3w5o+HkPJ/RDqQ3OutgqocKntzq4xRaJOcmb4E7F8oLcxMhtYxtNiIUeqwG L02rPJwfxBC2keTC+mOxPz6rfx/pftW2GqWzQ/xQv6qMmw5LVsYwXTjx50oO4D0nv8rj 5T4y7GXzkwcZxG4BvlgSyh3M0w8Is94nLh1mI0G5DpdLkPfeNf9SDy8f/yJDcIrNxCXl ke1aQ96i5BGh7C2VvGcGdmnFB4Neb3B1h2GABh8ibvAJkeGWODvzahqwuT5pOfM7tc3k N1RQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=HnO6lzGt; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id 91-v6si20304449plc.409.2018.11.12.21.51.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:42 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=HnO6lzGt; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C7F5822511; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088302; bh=01UZQO41vzNh4Savahf7vssInXlOxgP+KXnpuRHFvB4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HnO6lzGtfavJ4a0yEe3+D9TCHyjIPQ+IKnISXuc0Y5dm7cfd/MskXamxBABRli8Fp QeueDu1iDlDy2Y7q9zc1+QNDPI61b36SwRWiWSBvh+qt4nVDXHylWfiKPMJi4Xh6XK VxUkSQOZAaiyp/xsUEwHSJoSk+Go/qGNuoWfrBYI= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Pavel Tatashin , Abdul Haleem , Baoquan He , Daniel Jordan , Dan Williams , Dave Hansen , David Rientjes , Greg Kroah-Hartman , Ingo Molnar , Jan Kara , =?utf-8?b?SsOpcsO0?= =?utf-8?b?bWUgR2xpc3Nl?= , "Kirill A . Shutemov" , Michael Ellerman , Michal Hocko , Souptick Joarder , Steven Sistare , Vlastimil Babka , Wei Yang , Pasha Tatashin , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 35/39] mm: calculate deferred pages after skipping mirrored memory Date: Tue, 13 Nov 2018 00:50:49 -0500 Message-Id: <20181113055053.78352-35-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Pavel Tatashin [ Upstream commit d3035be4ce2345d98633a45f93a74e526e94b802 ] update_defer_init() should be called only when struct page is about to be initialized. Because it counts number of initialized struct pages, but there we may skip struct pages if there is some mirrored memory. So move, update_defer_init() after checking for mirrored memory. Also, rename update_defer_init() to defer_init() and reverse the return boolean to emphasize that this is a boolean function, that tells that the reset of memmap initialization should be deferred. Make this function self-contained: do not pass number of already initialized pages in this zone by using static counters. I found this bug by reading the code. The effect is that fewer than expected struct pages are initialized early in boot, and it is possible that in some corner cases we may fail to boot when mirrored pages are used. The deferred on demand code should somewhat mitigate this. But this still brings some inconsistencies compared to when booting without mirrored pages, so it is better to fix. [pasha.tatashin@oracle.com: add comment about defer_init's lack of locking] Link: http://lkml.kernel.org/r/20180726193509.3326-3-pasha.tatashin@oracle.com [akpm@linux-foundation.org: make defer_init non-inline, __meminit] Link: http://lkml.kernel.org/r/20180724235520.10200-3-pasha.tatashin@oracle.com Signed-off-by: Pavel Tatashin Reviewed-by: Oscar Salvador Cc: Abdul Haleem Cc: Baoquan He Cc: Daniel Jordan Cc: Dan Williams Cc: Dave Hansen Cc: David Rientjes Cc: Greg Kroah-Hartman Cc: Ingo Molnar Cc: Jan Kara Cc: Jérôme Glisse Cc: Kirill A. Shutemov Cc: Michael Ellerman Cc: Michal Hocko Cc: Souptick Joarder Cc: Steven Sistare Cc: Vlastimil Babka Cc: Wei Yang Cc: Pasha Tatashin Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/page_alloc.c | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 65f2e6481c99..eb3b250c7c9a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -306,24 +306,33 @@ static inline bool __meminit early_page_uninitialised(unsigned long pfn) } /* - * Returns false when the remaining initialisation should be deferred until + * Returns true when the remaining initialisation should be deferred until * later in the boot cycle when it can be parallelised. */ -static inline bool update_defer_init(pg_data_t *pgdat, - unsigned long pfn, unsigned long zone_end, - unsigned long *nr_initialised) +static bool __meminit +defer_init(int nid, unsigned long pfn, unsigned long end_pfn) { + static unsigned long prev_end_pfn, nr_initialised; + + /* + * prev_end_pfn static that contains the end of previous zone + * No need to protect because called very early in boot before smp_init. + */ + if (prev_end_pfn != end_pfn) { + prev_end_pfn = end_pfn; + nr_initialised = 0; + } + /* Always populate low zones for address-constrained allocations */ - if (zone_end < pgdat_end_pfn(pgdat)) - return true; - (*nr_initialised)++; - if ((*nr_initialised > pgdat->static_init_pgcnt) && - (pfn & (PAGES_PER_SECTION - 1)) == 0) { - pgdat->first_deferred_pfn = pfn; + if (end_pfn < pgdat_end_pfn(NODE_DATA(nid))) return false; + nr_initialised++; + if ((nr_initialised > NODE_DATA(nid)->static_init_pgcnt) && + (pfn & (PAGES_PER_SECTION - 1)) == 0) { + NODE_DATA(nid)->first_deferred_pfn = pfn; + return true; } - - return true; + return false; } #else static inline bool early_page_uninitialised(unsigned long pfn) @@ -331,11 +340,9 @@ static inline bool early_page_uninitialised(unsigned long pfn) return false; } -static inline bool update_defer_init(pg_data_t *pgdat, - unsigned long pfn, unsigned long zone_end, - unsigned long *nr_initialised) +static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) { - return true; + return false; } #endif @@ -5462,9 +5469,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, struct vmem_altmap *altmap) { unsigned long end_pfn = start_pfn + size; - pg_data_t *pgdat = NODE_DATA(nid); unsigned long pfn; - unsigned long nr_initialised = 0; struct page *page; #ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP struct memblock_region *r = NULL, *tmp; @@ -5492,8 +5497,6 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, continue; if (!early_pfn_in_nid(pfn, nid)) continue; - if (!update_defer_init(pgdat, pfn, end_pfn, &nr_initialised)) - break; #ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP /* @@ -5516,6 +5519,8 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, } } #endif + if (defer_init(nid, pfn, end_pfn)) + break; not_early: page = pfn_to_page(pfn); From patchwork Tue Nov 13 05:50:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679601 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F96013BB for ; Tue, 13 Nov 2018 05:51:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21BA32A34B for ; Tue, 13 Nov 2018 05:51:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2030B2A373; Tue, 13 Nov 2018 05:51:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D0A42A35C for ; Tue, 13 Nov 2018 05:51:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 309B66B027F; Tue, 13 Nov 2018 00:51:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E1EA6B0281; Tue, 13 Nov 2018 00:51:46 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1811A6B0283; Tue, 13 Nov 2018 00:51:46 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id C23066B027F for ; Tue, 13 Nov 2018 00:51:45 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id 33-v6so8755342pld.19 for ; Mon, 12 Nov 2018 21:51:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=9GzhlHUs84p2WH8pQpTUt9cdXDvDp1wmIhMi42ebRBk=; b=MmhmSBQEvWilzPcgZIo9cLV8MNAVjkhw/7Yda9DhC1oKcHnspXhauaoqKW7YsAMOOa AbzYLn+Nd7rHPpZU5IeT900w3L3e4AnSGnCNIlmielh4CPKeb5O/m6+omtiUw7Xa7MZK r4Xx8aL9kz64HlFv3p7UujYjsySmiKXiQyqC3UE+SeDrlIrCZeL3uWlTxD+VQ3XTIEHA h3Xw2ooaKR1rD2fWXmcUljBLvG9u2u5BpFXJF8MmBVdf6ewVWWPYt/VPFSEvdiOes5nX Hr68i5e2803NdvlxVwW36L17DsKHxQCuViKl/7UOv2K29Bt9irxKMB92bJ5TvXoZhwwx nutQ== X-Gm-Message-State: AGRZ1gI8o/J1pdMekaWZP/oqTAlpdzEfumDDiBBwQQ5k7nSiuM4YJQMC NfpqqZBXGOCrjaYk6MG/d5TUwew/upQWMkM0v1Kyj5TycApG8L16nGdb4fat4qoIkxzPpdkIOLw FoXJX3+neOaGdwwhZwEr4gHWmWClhAECx6hcvbANzTLLifbyPAlxMxNKGfXtnx5nPMA== X-Received: by 2002:a62:31c2:: with SMTP id x185-v6mr3832509pfx.39.1542088305487; Mon, 12 Nov 2018 21:51:45 -0800 (PST) X-Google-Smtp-Source: AJdET5fcQUWUpLelVL5laG94i2pIM3xAeSP8dSbvU9po8yIxlZmrFd+M5GS+Kgb4T4Kr21HIIP0V X-Received: by 2002:a62:31c2:: with SMTP id x185-v6mr3832475pfx.39.1542088304727; Mon, 12 Nov 2018 21:51:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088304; cv=none; d=google.com; s=arc-20160816; b=Rb6ErIljCO+XIC4Z6Clwsxd+xTTGVD6O4KsMlzPvJvBBtUXkts4+XYRrAJZk+6Bn/e fl1+9oiVmw+lLqYmpS6EsLrSIMVwcBXVfwsEV0iKJSCjchfu5Y4wT4OILni0VVH2togt JLVhKBg/cCASaqdtE1s+fttIaXWQqMIblE0LN8FVMlCLhSsSkvjZuTmYxiMuhOP6IbnT UMYMTI0xovLnTMcvvhXKEnZm6mgdlDaN614f/fbtI9aQoyqZ0LXUpk9NKYWfC+hz0W7C 3iixWvdpQ5rLtcwX/v1B5HLuyQrovdQBnq29ow1m4ZjUSz9BCwnjY4M7V2J/7pmHZSk/ QQIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9GzhlHUs84p2WH8pQpTUt9cdXDvDp1wmIhMi42ebRBk=; b=vzG63/hITPB3y1uNlQQejWNzt466EYkif/J1BD3e7ReLNm6/joy9kCGvYbI3vMKfvu NQ/s01FL5kUNEFH5DEfQ+FhaazWe3u18fSBRnLwLlNjCK6U37al8dwCfGrbWAuGyUE51 LcrImbk68vsJ9hZsfMb+2gA3PqhywJSkeYbMluQsmGSC1G4gqZ4mF6tf99FGkRX6MFlU fHVYim3dqxZChR0Larv0MMvhfD8QPD3BuQu9VMdhj+f2RJe1uOD59qyZbsn+7uMxWedC miEld+oDFMIobX0ACmcDeklxaHGCXLS7hGWpYg9FF5nO/7DvfnOv+fOW+fpsVQPcycbI UB/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Y5L3QAsN; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id c128-v6si7115296pfb.28.2018.11.12.21.51.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:44 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=Y5L3QAsN; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B3BC22251C; Tue, 13 Nov 2018 05:51:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088304; bh=EZSKRIU6MFcjpKmoMayFTIflHESg+IBP15cK+xtexns=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y5L3QAsNUTv9+gJNZPnM69wuESxvGuCF12Ejf2Z/zB0wKNaBLiHi9oVeTlrq2VKH7 wPPEWJuBKerKvkzmlGR8STsSyVJkR+3G9/x+NvxjSbTmZZy421OzqDu/OeW3mxSGuj s5KOOh2e67ECUP7zmwbpoNW8rRnPqCuiBrGHZAHQ= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jann Horn , Davidlohr Bueso , Oleg Nesterov , Christoph Lameter , Kemi Wang , Andy Lutomirski , Ingo Molnar , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 36/39] mm/vmstat.c: assert that vmstat_text is in sync with stat_items_size Date: Tue, 13 Nov 2018 00:50:50 -0500 Message-Id: <20181113055053.78352-36-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jann Horn [ Upstream commit f0ecf25a093fc0589f0a6bc4c1ea068bbb67d220 ] Having two gigantic arrays that must manually be kept in sync, including ifdefs, isn't exactly robust. To make it easier to catch such issues in the future, add a BUILD_BUG_ON(). Link: http://lkml.kernel.org/r/20181001143138.95119-3-jannh@google.com Signed-off-by: Jann Horn Reviewed-by: Kees Cook Reviewed-by: Andrew Morton Acked-by: Roman Gushchin Acked-by: Michal Hocko Cc: Davidlohr Bueso Cc: Oleg Nesterov Cc: Christoph Lameter Cc: Kemi Wang Cc: Andy Lutomirski Cc: Ingo Molnar Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/vmstat.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index 7878da76abf2..b678c607e490 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1663,6 +1663,8 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos) stat_items_size += sizeof(struct vm_event_state); #endif + BUILD_BUG_ON(stat_items_size != + ARRAY_SIZE(vmstat_text) * sizeof(unsigned long)); v = kmalloc(stat_items_size, GFP_KERNEL); m->private = v; if (!v) From patchwork Tue Nov 13 05:50:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679603 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C94413B5 for ; Tue, 13 Nov 2018 05:51:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3AAAC2A35C for ; Tue, 13 Nov 2018 05:51:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D8642A34B; Tue, 13 Nov 2018 05:51:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 727342A35C for ; Tue, 13 Nov 2018 05:51:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79CB16B0281; Tue, 13 Nov 2018 00:51:48 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 74C986B0283; Tue, 13 Nov 2018 00:51:48 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63D3D6B0284; Tue, 13 Nov 2018 00:51:48 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 1EF816B0281 for ; Tue, 13 Nov 2018 00:51:48 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id e89so1534034pfb.17 for ; Mon, 12 Nov 2018 21:51:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=o1YoOJ2KchQObcJY1zCIrIn36jOmER0O+z8BpMipZr0=; b=cRHYE7mNKroupOCbBGdqYtawYQ2uHxgx3n8LXfW2iaVzdbSVVfrb3TOEeIfB9EFOEI 6jcwW/ojATnWAAC+rYHjpIylPUvG4XTqXUipB/GLQovv127p1j6droMI79VFCbQQC4xa naJsMVdVGAYjLkmNiWKx3rce0DYjtf73VI8AGD0Xut0fR2D6O3+EpgdkDu7ZPcGnvGsa osAIvzaFl6urBQheMkDcX3jos9FcbDElcYxDe+IOLz03OmD2+/5VcND+f5iw6q7z2mKl Y1dWsa06bNhwLAqpEK23zUv5xKYtvtUmTnE7Ku2H6cM7Ynh6xHUfIV0SJ2WLynLwwVtO eRYA== X-Gm-Message-State: AGRZ1gINbemH11eRQ9ECRZtn7aBWfQhQ4ChcqZEa1RPnK2IvQAxLrJkC K76Rk1rjvmJNC3J26OZ1uAkfJY3EAj7wJs2P8ySckXWKlOQXFfnWTlDvK3zzaCVWT21C8tPUy8V xnjeI75Llz6/MU4aIBV6f6t09bOpPjJ8ZXq4WzAAZ7FUbVA/er32A2sFq/Sqz2sdMfQ== X-Received: by 2002:a17:902:142:: with SMTP id 60-v6mr3778934plb.108.1542088307786; Mon, 12 Nov 2018 21:51:47 -0800 (PST) X-Google-Smtp-Source: AJdET5cIcpaHAWDOW/SVt6KtsxY3KEOJmnnNnNzWWaZkgj1vhbm69E3fc6klMCmJRlESqRCVDJ8u X-Received: by 2002:a17:902:142:: with SMTP id 60-v6mr3778890plb.108.1542088306742; Mon, 12 Nov 2018 21:51:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088306; cv=none; d=google.com; s=arc-20160816; b=IjiRkuH0aV45yIeJt/hspLq7nBA2oQ3sMeWePld8GTzOxxeVR6IXJz54/X96Ek0wu1 no/76CVSbd4CslZOzJ9nQmYnqsJLIgYdZ1OW9YlEQt4wngS3IY+nA9AH+AiloG545TDq REYXSfzZjDivkM34ivJdLjlSFViCodpssOj795LW/jnqkp/BgMn7rmJExAkW8QFYP5ek wCUCSkX732kjYD0gyG3E1ZppmenHDfL8KNhAVINujbil+cR3yYrERT/zc+pUY8D/C45z IiW5j5AivqXV2M+jlYQJRBrg+TlSH6Fu9vLhqpSdID9r/4uE5fUnOfGENoYvs/lxfe6h /h4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=o1YoOJ2KchQObcJY1zCIrIn36jOmER0O+z8BpMipZr0=; b=S7YPbf8643VorqPyeiVVgpHh+Eyz+AjFDeU7GFZNIfPPjhAi5DJGQd+CXcXqIYsVlf cHgZ7OSIfXn+RJNPB4ei4dGmdiFsqX3H8OjFBNwiilucBZ0DEtdCfBO/GEw7oEF58FFq PXRRhGxaD+9h+EeoHdb2fFE2UHKu1eBgbI5vQ0nfgZjcFqvmsLUG88fOHt0hCamP02Dt C5d7IcKLRSmIEkgak83DFp2ZkoV41snIFplXH8YGf+bgifqSLc9LoT2/Iy0ytSU7WEP3 qEgD176Hhjwf7ZhyCQW1hlfJjGIDuXpzYLgkC8PeWmib74XPnEsAFRZSQ63d9qLysWj8 GfoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=g+DYxkAV; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id s4si18320619pga.377.2018.11.12.21.51.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:46 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=g+DYxkAV; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 426E022507; Tue, 13 Nov 2018 05:51:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088306; bh=zvT9WxixCZyRIox0HPbugmtAY4OJxLd1LR7y8Ru+BHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g+DYxkAVrkj4dorkLGecCqLXTDc61MvFY0HOrVWbbCXkoR3iQLZMHfl9sSNWbUnWW j6SO9g4GSpFZsDEqD3rHn/9YB2zsOLZt72uMQEWWCh/p5/0ChetAhvlBo/NPCVCycF AuJspLysaqxRlz4vAyrPYk/WjAE43y8CueQE3mPA= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 37/39] userfaultfd: allow get_mempolicy(MPOL_F_NODE|MPOL_F_ADDR) to trigger userfaults Date: Tue, 13 Nov 2018 00:50:51 -0500 Message-Id: <20181113055053.78352-37-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andrea Arcangeli [ Upstream commit 3b9aadf7278d16d7bed4d5d808501065f70898d8 ] get_mempolicy(MPOL_F_NODE|MPOL_F_ADDR) called a get_user_pages that would not be waiting for userfaults before failing and it would hit on a SIGBUS instead. Using get_user_pages_locked/unlocked instead will allow get_mempolicy to allow userfaults to resolve the fault and fill the hole, before grabbing the node id of the page. If the user calls get_mempolicy() with MPOL_F_ADDR | MPOL_F_NODE for an address inside an area managed by uffd and there is no page at that address, the page allocation from within get_mempolicy() will fail because get_user_pages() does not allow for page fault retry required for uffd; the user will get SIGBUS. With this patch, the page fault will be resolved by the uffd and the get_mempolicy() will continue normally. Background: Via code review, previously the syscall would have returned -EFAULT (vm_fault_to_errno), now it will block and wait for an userfault (if it's waken before the fault is resolved it'll still -EFAULT). This way get_mempolicy will give a chance to an "unaware" app to be compliant with userfaults. The reason this visible change is that becoming "userfault compliant" cannot regress anything: all other syscalls including read(2)/write(2) had to become "userfault compliant" long time ago (that's one of the things userfaultfd can do that PROT_NONE and trapping segfaults can't). So this is just one more syscall that become "userfault compliant" like all other major ones already were. This has been happening on virtio-bridge dpdk process which just called get_mempolicy on the guest space post live migration, but before the memory had a chance to be migrated to destination. I didn't run an strace to be able to show the -EFAULT going away, but I've the confirmation of the below debug aid information (only visible with CONFIG_DEBUG_VM=y) going away with the patch: [20116.371461] FAULT_FLAG_ALLOW_RETRY missing 0 [20116.371464] CPU: 1 PID: 13381 Comm: vhost-events Not tainted 4.17.12-200.fc28.x86_64 #1 [20116.371465] Hardware name: LENOVO 20FAS2BN0A/20FAS2BN0A, BIOS N1CET54W (1.22 ) 02/10/2017 [20116.371466] Call Trace: [20116.371473] dump_stack+0x5c/0x80 [20116.371476] handle_userfault.cold.37+0x1b/0x22 [20116.371479] ? remove_wait_queue+0x20/0x60 [20116.371481] ? poll_freewait+0x45/0xa0 [20116.371483] ? do_sys_poll+0x31c/0x520 [20116.371485] ? radix_tree_lookup_slot+0x1e/0x50 [20116.371488] shmem_getpage_gfp+0xce7/0xe50 [20116.371491] ? page_add_file_rmap+0x1a/0x2c0 [20116.371493] shmem_fault+0x78/0x1e0 [20116.371495] ? filemap_map_pages+0x3a1/0x450 [20116.371498] __do_fault+0x1f/0xc0 [20116.371500] __handle_mm_fault+0xe2e/0x12f0 [20116.371502] handle_mm_fault+0xda/0x200 [20116.371504] __get_user_pages+0x238/0x790 [20116.371506] get_user_pages+0x3e/0x50 [20116.371510] kernel_get_mempolicy+0x40b/0x700 [20116.371512] ? vfs_write+0x170/0x1a0 [20116.371515] __x64_sys_get_mempolicy+0x21/0x30 [20116.371517] do_syscall_64+0x5b/0x160 [20116.371520] entry_SYSCALL_64_after_hwframe+0x44/0xa9 The above harmless debug message (not a kernel crash, just a dump_stack()) is shown with CONFIG_DEBUG_VM=y to more quickly identify and improve kernel spots that may have to become "userfaultfd compliant" like this one (without having to run an strace and search for syscall misbehavior). Spots like the above are more closer to a kernel bug for the non-cooperative usages that Mike focuses on, than for for dpdk qemu-cooperative usages that reproduced it, but it's still nicer to get this fixed for dpdk too. The part of the patch that caused me to think is only the implementation issue of mpol_get, but it looks like it should work safe no matter the kind of mempolicy structure that is (the default static policy also starts at 1 so it'll go to 2 and back to 1 without crashing everything at 0). [rppt@linux.vnet.ibm.com: changelog addition] http://lkml.kernel.org/r/20180904073718.GA26916@rapoport-lnx Link: http://lkml.kernel.org/r/20180831214848.23676-1-aarcange@redhat.com Signed-off-by: Andrea Arcangeli Reported-by: Maxime Coquelin Tested-by: Dr. David Alan Gilbert Reviewed-by: Mike Rapoport Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/mempolicy.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 01f1a14facc4..a7f7f5415936 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -797,16 +797,19 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) } } -static int lookup_node(unsigned long addr) +static int lookup_node(struct mm_struct *mm, unsigned long addr) { struct page *p; int err; - err = get_user_pages(addr & PAGE_MASK, 1, 0, &p, NULL); + int locked = 1; + err = get_user_pages_locked(addr & PAGE_MASK, 1, 0, &p, &locked); if (err >= 0) { err = page_to_nid(p); put_page(p); } + if (locked) + up_read(&mm->mmap_sem); return err; } @@ -817,7 +820,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, int err; struct mm_struct *mm = current->mm; struct vm_area_struct *vma = NULL; - struct mempolicy *pol = current->mempolicy; + struct mempolicy *pol = current->mempolicy, *pol_refcount = NULL; if (flags & ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) @@ -857,7 +860,16 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, if (flags & MPOL_F_NODE) { if (flags & MPOL_F_ADDR) { - err = lookup_node(addr); + /* + * Take a refcount on the mpol, lookup_node() + * wil drop the mmap_sem, so after calling + * lookup_node() only "pol" remains valid, "vma" + * is stale. + */ + pol_refcount = pol; + vma = NULL; + mpol_get(pol); + err = lookup_node(mm, addr); if (err < 0) goto out; *policy = err; @@ -892,7 +904,9 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, out: mpol_cond_put(pol); if (vma) - up_read(¤t->mm->mmap_sem); + up_read(&mm->mmap_sem); + if (pol_refcount) + mpol_put(pol_refcount); return err; } From patchwork Tue Nov 13 05:50:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679605 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3901413BB for ; Tue, 13 Nov 2018 05:52:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2828629BCF for ; Tue, 13 Nov 2018 05:52:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 26A2F2A34E; Tue, 13 Nov 2018 05:52:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 667E829BCF for ; Tue, 13 Nov 2018 05:52:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 973646B0283; Tue, 13 Nov 2018 00:51:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 948B36B0284; Tue, 13 Nov 2018 00:51:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 810716B0285; Tue, 13 Nov 2018 00:51:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 3E4406B0283 for ; Tue, 13 Nov 2018 00:51:51 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id l131so7395439pga.2 for ; Mon, 12 Nov 2018 21:51:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Cal766x8NKGJDINwRcNerg4OvXJvoc0iljtnm8GlA5Y=; b=QUnINoxSauQNUe2pbzYkxmy9oXQjn7rHB/Mt/OsE3NXbS9/52i6jQzyd20ITSDD8Pb Wce4t2LLIt8KxLOBbJobN0CzZmOF9RkCJ/qIbJlD5OLvtOOYaLBU5R5JPl3GK9SDq6Mu bFGcXjiR88Dw5NCsiVPXmr07LX/NROl6GGzkRH0LeLRCag0Zm53pelSgfuC53YV1lKO8 2E8B94g5+g9rD62weFE+6AAma843pi/HWbdWYVFdRGxUR6y8eT+P8GbjSliwTv3ewiD5 146ORl25ZGp8dxPXRN22LY05EjVw2GEZEsEY37uhWgPhbn+4VIllxzHUdVeXL3ORYwIh 3hrg== X-Gm-Message-State: AGRZ1gLk4ONjdzoOZM33/takwODI6wKqBIGWWPOS8VSEPQ2wPXVDsWJN PJU/K63+WZ8ZGC/RDWwDODcB4s3ArcunNNJtDfFu8xtTcEIe+AxoCzNMkNt8Cuj3ynuisl8fHOP 8WVfT+3w6UmjCLBwvwn26oMAxMIA/rxfDQlVGIEmgIQbKZfX+aFFO3Vy1oOCcQ2l9lA== X-Received: by 2002:a62:b9b:: with SMTP id 27-v6mr3764275pfl.235.1542088310930; Mon, 12 Nov 2018 21:51:50 -0800 (PST) X-Google-Smtp-Source: AJdET5fE9NKQpTvsWEu28T9LdR/2OkYDS64kVoMLIFClwUaXTtyuBDa4ISZ8LfZsZ2I0sXesXb9R X-Received: by 2002:a62:b9b:: with SMTP id 27-v6mr3764246pfl.235.1542088310065; Mon, 12 Nov 2018 21:51:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088310; cv=none; d=google.com; s=arc-20160816; b=UnpafMYi8eiNnRcW07+GsKSquDpj1pFHxqSqRya062QjyABWnlQa/oHevtifXDrVma c1pL/oppIZavekJ1nkKaSJ4Fy7kixZ0bb23GJm9y2tL6IXXHsk1fJW5yYETbw54sH5Mf fmTaC+umMhI+65kFBaUDLs0GijWbSUG5yQ0HMguy5jrbOh9Uugy8MQV/nEHZiuwDWinY FH6WqhVHL7obLIP1la9Hi6ul7YM/aW0fO5HG/8ZapR9zfy9eieySA1/hYEmiv/BbjW5Q n67O6TtehCk7jM7Xn3H5pJrmeusKWXq/hBm8oaVvSe/j4wgGH8Dspvy/TLi5wsSlZKCO 7K6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Cal766x8NKGJDINwRcNerg4OvXJvoc0iljtnm8GlA5Y=; b=yeySnyxKEQu5k6llnLqQ+wJf6yrJ3fkhBnkxIR97hGHuDiw9QgrjIK6xvZY30gwdhR fUdgIg3HewlZb+bJtZVHDRNi/BJqZU7cgIh1PHFJGQc6dDLR+/qo8OTSBrgckfhHRRrU NlmVZcAzgyos7KGkBvYjYfTv6kRMbOUjLn8oQCox0sw8d879DK/IcaxEallR87qM5gpt nVjeXTTlsCbsPBlmBka92indyxvBQobWatVA1NIsCCs9PUy8o4/NI5/nn7Ac9Oi4nyaz O9k3OcJ7zqsg/lbTSNfISvEeLWShIChMh4IdAsUCXp3FqewDoOOjTcJ6PMEYFgnm2Bkl NDdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=C3dMtBpS; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id bc3-v6si20190128plb.52.2018.11.12.21.51.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:50 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=C3dMtBpS; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7DF0A22507; Tue, 13 Nov 2018 05:51:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088309; bh=rPql8XgGWs45tSjiEdAy+Tg+skKBsCFaPro/SvFYm54=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C3dMtBpSIPiaWB3Qnqbo7KbBztGUQP09I+OxlxH/DDonlUzWJxKMTMvNZAUUDbhIZ O1oeCsJgiYEfimHQPb1I1Oj0imIAZobu3n+hqLSBE+wpM2+gfF9DIN1voYBMcijq7s yWF1TaCFAx9Wb5L8/5r4COnnYIRSDOzJeTUZ87Lc= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Johannes Weiner , Michal Hocko , Tejun Heo , Rik van Riel , Konstantin Khlebnikov , Matthew Wilcox , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 38/39] mm: don't miss the last page because of round-off error Date: Tue, 13 Nov 2018 00:50:52 -0500 Message-Id: <20181113055053.78352-38-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Roman Gushchin [ Upstream commit 68600f623d69da428c6163275f97ca126e1a8ec5 ] I've noticed, that dying memory cgroups are often pinned in memory by a single pagecache page. Even under moderate memory pressure they sometimes stayed in such state for a long time. That looked strange. My investigation showed that the problem is caused by applying the LRU pressure balancing math: scan = div64_u64(scan * fraction[lru], denominator), where denominator = fraction[anon] + fraction[file] + 1. Because fraction[lru] is always less than denominator, if the initial scan size is 1, the result is always 0. This means the last page is not scanned and has no chances to be reclaimed. Fix this by rounding up the result of the division. In practice this change significantly improves the speed of dying cgroups reclaim. [guro@fb.com: prevent double calculation of DIV64_U64_ROUND_UP() arguments] Link: http://lkml.kernel.org/r/20180829213311.GA13501@castle Link: http://lkml.kernel.org/r/20180827162621.30187-3-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Cc: Rik van Riel Cc: Konstantin Khlebnikov Cc: Matthew Wilcox Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- include/linux/math64.h | 3 +++ mm/vmscan.c | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index 837f2f2d1d34..bb2c84afb80c 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -281,4 +281,7 @@ static inline u64 mul_u64_u32_div(u64 a, u32 mul, u32 divisor) } #endif /* mul_u64_u32_div */ +#define DIV64_U64_ROUND_UP(ll, d) \ + ({ u64 _tmp = (d); div64_u64((ll) + _tmp - 1, _tmp); }) + #endif /* _LINUX_MATH64_H */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 03822f86f288..7b94e33823b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2287,9 +2287,11 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, /* * Scan types proportional to swappiness and * their relative recent reclaim efficiency. + * Make sure we don't miss the last page + * because of a round-off error. */ - scan = div64_u64(scan * fraction[file], - denominator); + scan = DIV64_U64_ROUND_UP(scan * fraction[file], + denominator); break; case SCAN_FILE: case SCAN_ANON: From patchwork Tue Nov 13 05:50:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679655 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D54F2109C for ; Tue, 13 Nov 2018 05:55:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3D0929FAC for ; Tue, 13 Nov 2018 05:55:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B712F2A264; Tue, 13 Nov 2018 05:55:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA16129FAC for ; Tue, 13 Nov 2018 05:55:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF1C26B029C; Tue, 13 Nov 2018 00:55:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C78DA6B029E; Tue, 13 Nov 2018 00:55:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1B4B6B029F; Tue, 13 Nov 2018 00:55:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 6AF296B029C for ; Tue, 13 Nov 2018 00:55:51 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id 129-v6so9545305pfx.11 for ; Mon, 12 Nov 2018 21:55:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=awUPsRmGv7aD/Cg9oCyKAUsZoVxKMnOuwQoRAGhNkxk=; b=ZadbvwFnzzwNDUnrARmRcEbELqm75IOigy3eyXd0CoIRaq1aEmAHJ5cO4cm7aR7iia wwrDWwSJKIS7xUnTGEV8xYWodljUocCKE4G8oGQTiBZOTZnu/K8XeyVvAiFm6L3S8A6b BEEP1Sq9VD8RaNPIjT4J0T1OQCZGlsLyCzEn93cveupa1eKdY0+DY3OyM9GfRK6DxgCI frRj4pEBsnobay+JZFP/UUaIgJlpzaFagae1+JoAbL9GrZ6tQiXMyXL0q/flDLCzFcBC Q5gFoyEhr2dYvMguVyMPSkje29vuD1jXqdgBCtsgOD6eGYe7cr1qfC8DXibCfCipEXKm FXlg== X-Gm-Message-State: AGRZ1gIbxMEI2qrExaiWzHMERTxU5F5iGLexXNRqJMRSYYHglrxlLcuI IMUA140yfTcJhYzVpSXFsTvOaGUPL7pXIroWQhhVUboIyIG1A1gP9n8NSnjCpGl8rxvhLqUQ6N+ /wXLwxR0iQeSdOOLj6W+fDJInK/o62Z3114faLu47R0gNoM6XLYIbEBkQqZgK1RF2ZfWOy8S4fN 1in538Swq1nkzj5pCQ9Yfa3gCN3JzRevljN2DNVwGxmqPPYJDDtI9xjdEPqHevkZq17VI1iTmzE 1yHdLdwcNoNEb90mXiEn451EjKS4i15y5tnMMtSsDVMfmY65xnGEgorsPzhIA2aiVyuqCkqZXxu k0uri7Ru98abDBXGg4YX4HbfzTayhLnO7w0jJATAkOjfCL/Z436JlVvTPQqMqBdRLUkgiluMNTb Q X-Received: by 2002:a63:ae01:: with SMTP id q1mr3470259pgf.402.1542088311846; Mon, 12 Nov 2018 21:51:51 -0800 (PST) X-Received: by 2002:a63:ae01:: with SMTP id q1mr3470252pgf.402.1542088311723; Mon, 12 Nov 2018 21:51:51 -0800 (PST) X-Google-Smtp-Source: AJdET5eUlH5v3k3IYIUxGjBB+FDHKqdjVOImsHdnK48LU69WOU324GKBRHzuWTofmyIGBC08pO87 X-Received: by 2002:a63:ae01:: with SMTP id q1mr3470211pgf.402.1542088310561; Mon, 12 Nov 2018 21:51:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088310; cv=none; d=google.com; s=arc-20160816; b=E0nswGeZDpNqx8jApDGoLdEhSQYLmJEb6jX8+5UN+qDy9mOz1NJZqOwCAWhkHslagD Swy/cEREYbhI2ld82N/4uVM9NPkJ7czWj+wOv8jSQn+QLdnnxg0pXW2Xzt9iE/ukt5Yk /Tzg83s7Dm/7glS/JfRUMYkN3T6uVZc9CVMYqvr4egRAJV+pANJErgR0Npxzk+YxOmaZ 26ZblBdKBs9SRGdtPHcLERlXSkDRMjI1UG6AxD8d7FW3U50BYTnzRk0OXaDTGlSCt+10 PTEhDQGLjST09kavLxGd3RCzbhDziS27BdAJ41vUJdPf2RwgEeKj3lm0xuVOODoJREnS FjKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=awUPsRmGv7aD/Cg9oCyKAUsZoVxKMnOuwQoRAGhNkxk=; b=fKGm/jol2H4ggul2v5kMouuz3Jc50d9oe+N3E7IlCdAhyc3LRN0+Xf1zdDdcLLTqmQ L3fytOetX+l+fr619HadveEnfmidMDJY+FKrFIN7FwQ1ixsrE2vKubQoCXS2r3PX6rEt CXjDNJ+kHg9POluLPp70wmEROch4alDiqMf/SXYz8o6rHnlvOQ2jFuN95NiZ7h1MkgVq uE2/2zFGNuxQKYIXeOduFDjpwfdmz3fe/zFS2hFxdvjxANZn8gRDVoDulKgQmu5K1E/+ BRndndjgcBZmddGKd250lMZw1Axc8ebRkCGD570aHRze+W/AoDBUFcTamV7TSBLyYSMj BOrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=hagp5elB; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id j7-v6si18831404plt.349.2018.11.12.21.51.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:50 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=hagp5elB; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0AA4D22510; Tue, 13 Nov 2018 05:51:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088310; bh=Lpm7SHpu7WLq7xIwdN24+tKRdX4Rqfv+5SIwBwozd4g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hagp5elBRNonOsJVyZAlAzUnNHNGCgcvywZn4gEFF24NSfDgeKru3jbC8/la1b888 058SDH8NXHGDslcHDIoXHFQ/iHLh6ezctN6DNC3sT4DT3g0eM1Wars4V/94VZU+5F3 mfT4n54aQ2jm99bJoSsvfDj0ihnmftdsMg2/pvLI= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Dmitry Vyukov , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 39/39] mm: don't warn about large allocations for slab Date: Tue, 13 Nov 2018 00:50:53 -0500 Message-Id: <20181113055053.78352-39-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Dmitry Vyukov [ Upstream commit 61448479a9f2c954cde0cfe778cb6bec5d0a748d ] Slub does not call kmalloc_slab() for sizes > KMALLOC_MAX_CACHE_SIZE, instead it falls back to kmalloc_large(). For slab KMALLOC_MAX_CACHE_SIZE == KMALLOC_MAX_SIZE and it calls kmalloc_slab() for all allocations relying on NULL return value for over-sized allocations. This inconsistency leads to unwanted warnings from kmalloc_slab() for over-sized allocations for slab. Returning NULL for failed allocations is the expected behavior. Make slub and slab code consistent by checking size > KMALLOC_MAX_CACHE_SIZE in slab before calling kmalloc_slab(). While we are here also fix the check in kmalloc_slab(). We should check against KMALLOC_MAX_CACHE_SIZE rather than KMALLOC_MAX_SIZE. It all kinda worked because for slab the constants are the same, and slub always checks the size against KMALLOC_MAX_CACHE_SIZE before kmalloc_slab(). But if we get there with size > KMALLOC_MAX_CACHE_SIZE anyhow bad things will happen. For example, in case of a newly introduced bug in slub code. Also move the check in kmalloc_slab() from function entry to the size > 192 case. This partially compensates for the additional check in slab code and makes slub code a bit faster (at least theoretically). Also drop __GFP_NOWARN in the warning check. This warning means a bug in slab code itself, user-passed flags have nothing to do with it. Nothing of this affects slob. Link: http://lkml.kernel.org/r/20180927171502.226522-1-dvyukov@gmail.com Signed-off-by: Dmitry Vyukov Reported-by: syzbot+87829a10073277282ad1@syzkaller.appspotmail.com Reported-by: syzbot+ef4e8fc3a06e9019bb40@syzkaller.appspotmail.com Reported-by: syzbot+6e438f4036df52cbb863@syzkaller.appspotmail.com Reported-by: syzbot+8574471d8734457d98aa@syzkaller.appspotmail.com Reported-by: syzbot+af1504df0807a083dbd9@syzkaller.appspotmail.com Acked-by: Christoph Lameter Acked-by: Vlastimil Babka Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/slab.c | 4 ++++ mm/slab_common.c | 12 ++++++------ 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index aa76a70e087e..d73c7a4820a4 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3675,6 +3675,8 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) struct kmem_cache *cachep; void *ret; + if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) + return NULL; cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; @@ -3710,6 +3712,8 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, struct kmem_cache *cachep; void *ret; + if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) + return NULL; cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; diff --git a/mm/slab_common.c b/mm/slab_common.c index 2296caf87bfb..8a0fba22c910 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1027,18 +1027,18 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) { unsigned int index; - if (unlikely(size > KMALLOC_MAX_SIZE)) { - WARN_ON_ONCE(!(flags & __GFP_NOWARN)); - return NULL; - } - if (size <= 192) { if (!size) return ZERO_SIZE_PTR; index = size_index[size_index_elem(size)]; - } else + } else { + if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { + WARN_ON(1); + return NULL; + } index = fls(size - 1); + } #ifdef CONFIG_ZONE_DMA if (unlikely((flags & GFP_DMA)))