From patchwork Thu Jul 12 20:37:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Tatashin X-Patchwork-Id: 10522269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D24B4603D7 for ; Thu, 12 Jul 2018 20:37:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C10D9298EC for ; Thu, 12 Jul 2018 20:37:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B4D0F29964; Thu, 12 Jul 2018 20:37:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18CF9298EC for ; Thu, 12 Jul 2018 20:37:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A3856B0003; Thu, 12 Jul 2018 16:37:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 078936B0006; Thu, 12 Jul 2018 16:37:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA9AF6B0007; Thu, 12 Jul 2018 16:37:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-vk0-f69.google.com (mail-vk0-f69.google.com [209.85.213.69]) by kanga.kvack.org (Postfix) with ESMTP id B9C206B0003 for ; Thu, 12 Jul 2018 16:37:56 -0400 (EDT) Received: by mail-vk0-f69.google.com with SMTP id x81-v6so12696135vkx.14 for ; Thu, 12 Jul 2018 13:37:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=H/7LwRAV2LNMIjm+RPf7DjB15iadDPIrSQ0bX2AQchg=; b=NIu1DFxQ1zlyiSAqnpSe4A513LnNXIcr0g430W0EuUnZ4TJGBPJ4H2DF3oel7jzImG mKGq4rKDSKvsmBGqEYWFnOv95CyjOtk3J8uW+giSEgD2OcDY9YJ2Jinrlqm1KNIWTWuM G+IpqjKHYULmvXj4rcniHq9PESPACGCnz8YP4p2VKOV1Wo3U8SZ2zg/mx6x8DJ5KutvB 6284IWYfhhQp8XchBtHQJph9UnJOFUQwcwH41ljYuaNDuizEl1zCdN2w8wcJcyVLdiqg pAo5FnkR/D+TROM/mMYsaJxCNH5Gn4cLHH6Te6hKg0Ip7hUPMs3y8O90pTesZABpmILo cO7w== X-Gm-Message-State: AOUpUlGIK7UykfYvs7aYhh2Y8WxW9MGycV1CQdGZI3AKW5wFVI2r6oaJ +xXUSAVZpz8IYQbIXnAlvfxbLa53dKvtG+D9GHYQDcfJYXxu8usBDR8wj5Li+Ctg/nc2D+PWUPR 4YMev50aMvAncKIhIPBoz2SH/MHocsg94go/d9HHLQ0d1B4QBlenfalgtPrLJaVHHeA== X-Received: by 2002:a1f:5107:: with SMTP id f7-v6mr2156180vkb.30.1531427876358; Thu, 12 Jul 2018 13:37:56 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcQw2LY3CFuIf45r+ropCXbbNGkWTkUNB99GxfJEZ4sHsW8Grlzuj0hFFOkZMHXdVZCLrf0 X-Received: by 2002:a1f:5107:: with SMTP id f7-v6mr2156156vkb.30.1531427875480; Thu, 12 Jul 2018 13:37:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531427875; cv=none; d=google.com; s=arc-20160816; b=bMdjypupVdocKpuoG/aQt1/dBUJiNIuRDJXetVtknc0JohRrItUwJUN6D1xf/7Nfe5 /Y9zVNjupvKgrhhJdj7ifCMBpitj+H7NmYQLQXu88Wa0OATGhbwVlAQ+ndeDyf3slwyq wB8jAjBBTmJ2RfhAWlGp1sdS56jXCc62nNuJv0MILVmxnQNhQRDSRKxk9rgtyufSyIY0 olbbq6mCsfVS/VOtuY7EFMpkbh0Jl7hGXNAPri40owDWchdu7Q0PozX/cw6clw/LUPDA qp1BiUpLvYxnHLYtcnJqZn471ojbFxS28f2fcsshiD3sO8ZuQeTDXeEAzhsq1Zth7PUX Jqiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature:arc-authentication-results; bh=H/7LwRAV2LNMIjm+RPf7DjB15iadDPIrSQ0bX2AQchg=; b=RYN43iS0U9fQ3VBDnHjVFDbGJ+hbOoJAaTSoZPW/JB9mbyjlCSInkKBLugWi/s+igd hjn7odQm+8GHX8Yo5ixHTnObAzewOxDMiwTA4BuX/tXgRla9hL+42CURKaOmqoQAzTtx xCloz/WRxZZNjp7TGE588la+UQ6+eBQGTqGOkWq6iudOYPoJGy8mkH5uPq264qWIgtrc /aJjng5njCgc9ZDuF1S8lEBI0bnpESGdStp+464fuWFTW2WOO4rupSnvrh/lXCfXa5AB zkSl6CpaoCzGJIQkaVGde8hVfDu0z2gdylwyHKMfNfZJm93hPvbKHDf0L/srzlhKrOKd 60bw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=AktqpNHg; spf=pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=pasha.tatashin@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id a132-v6si4770874vke.319.2018.07.12.13.37.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Jul 2018 13:37:55 -0700 (PDT) Received-SPF: pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=AktqpNHg; spf=pass (google.com: domain of pasha.tatashin@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=pasha.tatashin@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w6CKSvZ4153232; Thu, 12 Jul 2018 20:37:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=H/7LwRAV2LNMIjm+RPf7DjB15iadDPIrSQ0bX2AQchg=; b=AktqpNHgJpXW7fTuyoMzAaolB29umsktT0nKSFozmziHcQQvHKkW9YKdUggqelyiC2L+ izEFdxE3YSWfVUxYY0vlUWKYpk96Essoln24mBEmSYDvu69aNp42gcJ3cGVqCQRNXKOy dNj/gmHMm2urnka3cTb6eyV+61zMRBcS8uXvwom53oIGAmOf+LP80X40W+N3RwBj1/ek eSzKu1pYVTc6A4YjjcWMUEWs/pQsCY8pVsCS0hZ0YhPP2GmY4Z+NWgc7Ofrvr7NW3jRU v5nshnZjaqIRdEfSenMlJf4wdxGOkbnAJiDQJxUvG0c//ozDBplYtVQuxYNE+iwuLc5Q 5g== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2k2p76dd6s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Jul 2018 20:37:45 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id w6CKbiQM010079 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Jul 2018 20:37:44 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id w6CKbhLO030758; Thu, 12 Jul 2018 20:37:43 GMT Received: from localhost.localdomain (/73.69.118.222) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 12 Jul 2018 13:37:42 -0700 From: Pavel Tatashin To: steven.sistare@oracle.com, daniel.m.jordan@oracle.com, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, kirill.shutemov@linux.intel.com, mhocko@suse.com, linux-mm@kvack.org, dan.j.williams@intel.com, jack@suse.cz, jglisse@redhat.com, jrdr.linux@gmail.com, bhe@redhat.com, gregkh@linuxfoundation.org, vbabka@suse.cz, richard.weiyang@gmail.com, dave.hansen@intel.com, rientjes@google.com, mingo@kernel.org, osalvador@techadventures.net, pasha.tatashin@oracle.com, abdhalee@linux.vnet.ibm.com, mpe@ellerman.id.au Subject: [PATCH v5 1/5] mm/sparse: abstract sparse buffer allocations Date: Thu, 12 Jul 2018 16:37:26 -0400 Message-Id: <20180712203730.8703-2-pasha.tatashin@oracle.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180712203730.8703-1-pasha.tatashin@oracle.com> References: <20180712203730.8703-1-pasha.tatashin@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8952 signatures=668706 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807120216 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When struct pages are allocated for sparse-vmemmap VA layout, we first try to allocate one large buffer, and than if that fails allocate struct pages for each section as we go. The code that allocates buffer is uses global variables and is spread across several call sites. Cleanup the code by introducing three functions to handle the global buffer: sparse_buffer_init() initialize the buffer sparse_buffer_fini() free the remaining part of the buffer sparse_buffer_alloc() alloc from the buffer, and if buffer is empty return NULL Define these functions in sparse.c instead of sparse-vmemmap.c because later we will use them for non-vmemmap sparse allocations as well. Signed-off-by: Pavel Tatashin --- include/linux/mm.h | 4 ++++ mm/sparse-vmemmap.c | 40 ++++++---------------------------------- mm/sparse.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 54 insertions(+), 35 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 577e578eb640..a83d3e0e66d4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2655,6 +2655,10 @@ void sparse_mem_maps_populate_node(struct page **map_map, unsigned long map_count, int nodeid); +unsigned long __init section_map_size(void); +void sparse_buffer_init(unsigned long size, int nid); +void sparse_buffer_fini(void); +void *sparse_buffer_alloc(unsigned long size); struct page *sparse_mem_map_populate(unsigned long pnum, int nid, struct vmem_altmap *altmap); pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 95e2c7638a5c..b05c7663c640 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -43,12 +43,9 @@ static void * __ref __earlyonly_bootmem_alloc(int node, unsigned long goal) { return memblock_virt_alloc_try_nid_raw(size, align, goal, - BOOTMEM_ALLOC_ACCESSIBLE, node); + BOOTMEM_ALLOC_ACCESSIBLE, node); } -static void *vmemmap_buf; -static void *vmemmap_buf_end; - void * __meminit vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ @@ -76,18 +73,10 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) /* need to make sure size is all the same during early stage */ void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node) { - void *ptr; - - if (!vmemmap_buf) - return vmemmap_alloc_block(size, node); - - /* take the from buf */ - ptr = (void *)ALIGN((unsigned long)vmemmap_buf, size); - if (ptr + size > vmemmap_buf_end) - return vmemmap_alloc_block(size, node); - - vmemmap_buf = ptr + size; + void *ptr = sparse_buffer_alloc(size); + if (!ptr) + ptr = vmemmap_alloc_block(size, node); return ptr; } @@ -279,19 +268,9 @@ void __init sparse_mem_maps_populate_node(struct page **map_map, unsigned long map_count, int nodeid) { unsigned long pnum; - unsigned long size = sizeof(struct page) * PAGES_PER_SECTION; - void *vmemmap_buf_start; int nr_consumed_maps = 0; - size = ALIGN(size, PMD_SIZE); - vmemmap_buf_start = __earlyonly_bootmem_alloc(nodeid, size * map_count, - PMD_SIZE, __pa(MAX_DMA_ADDRESS)); - - if (vmemmap_buf_start) { - vmemmap_buf = vmemmap_buf_start; - vmemmap_buf_end = vmemmap_buf_start + size * map_count; - } - + sparse_buffer_init(section_map_size() * map_count, nodeid); for (pnum = pnum_begin; pnum < pnum_end; pnum++) { if (!present_section_nr(pnum)) continue; @@ -303,12 +282,5 @@ void __init sparse_mem_maps_populate_node(struct page **map_map, pr_err("%s: sparsemem memory map backing failed some memory will not be available\n", __func__); } - - if (vmemmap_buf_start) { - /* need to free left buf */ - memblock_free_early(__pa(vmemmap_buf), - vmemmap_buf_end - vmemmap_buf); - vmemmap_buf = NULL; - vmemmap_buf_end = NULL; - } + sparse_buffer_fini(); } diff --git a/mm/sparse.c b/mm/sparse.c index 2ea8b3dbd0df..ac57bae476f4 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -400,7 +400,14 @@ static void __init sparse_early_usemaps_alloc_node(void *data, } } -#ifndef CONFIG_SPARSEMEM_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP +unsigned long __init section_map_size(void) + +{ + return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); +} + +#else struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid, struct vmem_altmap *altmap) { @@ -457,6 +464,42 @@ void __init sparse_mem_maps_populate_node(struct page **map_map, } #endif /* !CONFIG_SPARSEMEM_VMEMMAP */ +static void *sparsemap_buf __meminitdata; +static void *sparsemap_buf_end __meminitdata; + +void __init sparse_buffer_init(unsigned long size, int nid) +{ + BUG_ON(sparsemap_buf); + sparsemap_buf = + memblock_virt_alloc_try_nid_raw(size, PAGE_SIZE, + __pa(MAX_DMA_ADDRESS), + BOOTMEM_ALLOC_ACCESSIBLE, nid); + sparsemap_buf_end = sparsemap_buf + size; +} + +void __init sparse_buffer_fini(void) +{ + unsigned long size = sparsemap_buf_end - sparsemap_buf; + + if (sparsemap_buf && size > 0) + memblock_free_early(__pa(sparsemap_buf), size); + sparsemap_buf = NULL; +} + +void * __meminit sparse_buffer_alloc(unsigned long size) +{ + void *ptr = NULL; + + if (sparsemap_buf) { + ptr = (void *)ALIGN((unsigned long)sparsemap_buf, size); + if (ptr + size > sparsemap_buf_end) + ptr = NULL; + else + sparsemap_buf = ptr + size; + } + return ptr; +} + #ifdef CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER static void __init sparse_early_mem_maps_alloc_node(void *data, unsigned long pnum_begin,