From patchwork Wed Apr 3 17:34:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5514C17E0 for ; Wed, 3 Apr 2019 17:37:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BD4927FA8 for ; Wed, 3 Apr 2019 17:37:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D2F9288F4; Wed, 3 Apr 2019 17:37:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DE2A27FA8 for ; Wed, 3 Apr 2019 17:37:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F7196B0008; Wed, 3 Apr 2019 13:36:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0E8816B026A; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF0686B0010; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id A37DA6B000E for ; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id p143so14465638iod.19 for ; Wed, 03 Apr 2019 10:36:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=0+GGGOXBJNB8WE2w6yip3kAhBo1m6CU7TV5HSXQxFt4=; b=hTtgNJICRvmRJ3T1G/Q9UpwSCAq6zMpCANv3QXm1bAeWo/LV6e5BAfqX11tEw4epdt wjhQo6C7l2GNdy/BepADFM8dgODzpnj9KRimeK9Aj6FxfY61ChrVPb5gY5Ji6HtuFClj pcr6HM2ibrKj/wY4rcNXlv6edzApjTQe+z55CNpgQlUxdUAnTst5UZ/fefAxFfUXT2hn DiI+D0f+7GGG0CnHbl8aoZkjqpMJAPOb9MJLwIIyOXVHj8EuUm7bAb+4lpJ7yZyuF4QU rh0QLr6SinWk7Otvf1k8bYLyVMh9dx9EuyHtW5hG/+pqytFxjT/k1Agj+oi5io+QYEAH FVzg== X-Gm-Message-State: APjAAAU3ZGC6sO+2+cNQzawGU3+gEGb+QuqSeYdpweJ74v9e721tnd8A /bwDbDfDR3hv658u4JZYnzQeWjdG3U61ULERZTEsWkoQTR23tsHX1TwiTGY5ZTAVO+Be10vbCdI OLKkHX4U/fSXFpgH3TLaje5BD2zUj3DuBEknETj0clRuyn46CD3gFo0oKy4/+W+RKeA== X-Received: by 2002:a24:ac6e:: with SMTP id m46mr1284059iti.49.1554313018432; Wed, 03 Apr 2019 10:36:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqwpJr3V42SGq+MiQ+0ziX8i1CLFb0Kg8Iatd4KxiZKnWuKhMggdp5NxmCYkZpg9dv2Wx9A8 X-Received: by 2002:a24:ac6e:: with SMTP id m46mr1284001iti.49.1554313017680; Wed, 03 Apr 2019 10:36:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313017; cv=none; d=google.com; s=arc-20160816; b=t9OFCyhcrSMp5eQXuX0KgA+2SXoKCOkif+BiQd47NXX1OZissGEKIUpI48HYlJQADN OBDCw/26xmqxcvCfaDFygYnK+Kbi9q3n80psWmHjLsY20Bp23ivQ0tkx1qGvGpJUet1I /vfB0ZKcz53N6Gby4Vlnm6VKAm14Q7C56X3jCFKFZDmF2H0Y+IJEQVYUr5kwYMRrWnc1 Tz03cjwCaWjtyfLFunj5XZlgifj0BBvsXoKLHG3nbgfzGfRoRWDdbqhr0ElS9b7iX4U3 JIsdrbSqHJOj8g5UdOHhmsUHlSS3xqvocXthNI4DkKIGnb8B0Kt+lJyWFCCJbRs6wnJr qhgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=0+GGGOXBJNB8WE2w6yip3kAhBo1m6CU7TV5HSXQxFt4=; b=ivNnmKKfCGJa7K//ebbAlr6c1PZELDq4sHJ7bbP0ZgFrjJSX84e1d0yMNhHx0S/xY3 uFY1hr9cKJSXxGbgGTHZ3/egqPv6Ws1JmcL88j3tkz2/qw71Y1+veuXZ3z4ftNycBMYg 1weXTyrw/SLpadlqyyJbAWiwQJnIyI1vUEHBeDuKtdBk7w2gSmNCmqK7tjzTAAkAf7x6 yxrKk9YhUjGEr4VCZLP+JBZw8GPm0TMcMHlMvkNY/ml3XtbYem3ltfL5fZruM78kxsVa 5pEZUxYTRaTvenOYxiUNnWThr9QB5pjpBUufqEb8Ke2DljbbNtddiABpQQCECwhGdjjF kg8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=288vFbrP; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id 191si8491665itu.68.2019.04.03.10.36.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:36:57 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=288vFbrP; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNpk0175413; Wed, 3 Apr 2019 17:35:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=0+GGGOXBJNB8WE2w6yip3kAhBo1m6CU7TV5HSXQxFt4=; b=288vFbrPKJZeEvZKUpB1q4+qhc8PIJAuLr1GF6exjMxG6eeOlw8STiY21uxJQnYUiFfC OwYQR0h3/QkWxocZ/8H1GT4YFL6aOSOKM1o6NldmVWMUGTB3px3AaAbWaib2Is7LzpkZ fEdhRb2MXC2PL+tWDOgJ2HW5Z08Q8JPH2kgnsjnPfPFgi1PzypJ4FD4KQOVbJS1wMNW1 GEGTqgHmx/W3766VvWR+qoAxoK121Kq7kQUvA47y2mjISZW8QjPPcV+WYpD+bh9DhZab KJkq6r3l5zkILWnkKASronC7BCRtWM4HTV470c/1v4dRtPl1dtZSQxRr19DuxdTS9Qx+ bw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 2rj13qae6y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:43 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIQI110879; Wed, 3 Apr 2019 17:35:42 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserp3030.oracle.com with ESMTP id 2rm8f5fye7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:42 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZMXB001176; Wed, 3 Apr 2019 17:35:22 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:21 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 01/13] mm: add MAP_HUGETLB support to vm_mmap Date: Wed, 3 Apr 2019 11:34:02 -0600 Message-Id: <155f6a436aa38be6b31e37965187d161e1233a1c.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen vm_mmap is exported, which means kernel modules can use it. In particular, for testing XPFO support, we want to use it with the MAP_HUGETLB flag, so let's support it via vm_mmap. Signed-off-by: Tycho Andersen Tested-by: Marco Benatto Tested-by: Khalid Aziz Cc: Khalid Aziz --- include/linux/mm.h | 2 ++ mm/mmap.c | 19 +------------------ mm/util.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 80bb6408fe73..3e4f6525d06b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2412,6 +2412,8 @@ struct vm_unmapped_area_info { extern unsigned long unmapped_area(struct vm_unmapped_area_info *info); extern unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info); +struct file *map_hugetlb_setup(unsigned long *len, unsigned long flags); + /* * Search for an unmapped address range. * diff --git a/mm/mmap.c b/mm/mmap.c index fc1809b1bed6..65382d942598 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1582,24 +1582,7 @@ unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, if (unlikely(flags & MAP_HUGETLB && !is_file_hugepages(file))) goto out_fput; } else if (flags & MAP_HUGETLB) { - struct user_struct *user = NULL; - struct hstate *hs; - - hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); - if (!hs) - return -EINVAL; - - len = ALIGN(len, huge_page_size(hs)); - /* - * VM_NORESERVE is used because the reservations will be - * taken when vm_ops->mmap() is called - * A dummy user value is used because we are not locking - * memory so no accounting is necessary - */ - file = hugetlb_file_setup(HUGETLB_ANON_FILE, len, - VM_NORESERVE, - &user, HUGETLB_ANONHUGE_INODE, - (flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); + file = map_hugetlb_setup(&len, flags); if (IS_ERR(file)) return PTR_ERR(file); } diff --git a/mm/util.c b/mm/util.c index 379319b1bcfd..86b763861828 100644 --- a/mm/util.c +++ b/mm/util.c @@ -357,6 +357,29 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, return ret; } +struct file *map_hugetlb_setup(unsigned long *len, unsigned long flags) +{ + struct user_struct *user = NULL; + struct hstate *hs; + + hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); + if (!hs) + return ERR_PTR(-EINVAL); + + *len = ALIGN(*len, huge_page_size(hs)); + + /* + * VM_NORESERVE is used because the reservations will be + * taken when vm_ops->mmap() is called + * A dummy user value is used because we are not locking + * memory so no accounting is necessary + */ + return hugetlb_file_setup(HUGETLB_ANON_FILE, *len, + VM_NORESERVE, + &user, HUGETLB_ANONHUGE_INODE, + (flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); +} + unsigned long vm_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flag, unsigned long offset) @@ -366,6 +389,15 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, if (unlikely(offset_in_page(offset))) return -EINVAL; + if (flag & MAP_HUGETLB) { + if (file) + return -EINVAL; + + file = map_hugetlb_setup(&len, flag); + if (IS_ERR(file)) + return PTR_ERR(file); + } + return vm_mmap_pgoff(file, addr, len, prot, flag, offset >> PAGE_SHIFT); } EXPORT_SYMBOL(vm_mmap); From patchwork Wed Apr 3 17:34:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFC311575 for ; Wed, 3 Apr 2019 17:37:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D61BF27FA8 for ; Wed, 3 Apr 2019 17:37:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C7881288F4; Wed, 3 Apr 2019 17:37:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16D2E27FA8 for ; Wed, 3 Apr 2019 17:37:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDE0F6B000E; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DB4156B0008; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2E856B0269; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 9DFC46B0008 for ; Wed, 3 Apr 2019 13:36:58 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id w11so14294719iom.20 for ; Wed, 03 Apr 2019 10:36:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=557xk95OakwCk6FJ0C0EA1LbpPbU7nMPF7pIFxhjjUw=; b=Z+y3whT254nyj8QmZyfEvbK0INcZqExAAMj4U3ip6HFqUPgrj3pjib87B0g7YKvGEj nHxldnPm5UzpDvPyWLST80CX61/UaijNG/QknvO6OHUmaT2V/yiqbKXZqkVwR8LnFvCm LnMLdFm0p/DwhcrNdw2UTswXEp4PJTvAfXb7i7ObszRjyRKPF4aTEh9gziFfnQve2098 E4aHtIh7OrAoU2r+S7L7D7XN8R8fstgXmHEgROZb5YLVPa5F0lSZoSZ/FFRNTR4AqM21 a7zJBdSg4cGQTZ7fKJCgevcdP2IVuwngecv/eqJVfu0TeWWD6OBI75EnNGGWxbkRG/uE CUUw== X-Gm-Message-State: APjAAAUwjsWWI7SeTMFFPnirjVAUqp6UWmphGfntLX0N0xJDjygMo1uc BzeT6YRJZceoe2pdR3NYdv5dvzFhXvxCxMMwyNaXQE3zfKSP+/FktbCV2jV0O1wuZzwrATGbO+L uqpaWk+nNXGiT3IojhylS9RbHS0faJcl51SbFjCy6a6aLXerznHRHQ9V+GrgNb8IT9w== X-Received: by 2002:a24:7f90:: with SMTP id r138mr1207961itc.163.1554313018369; Wed, 03 Apr 2019 10:36:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqxbRtofDF3itg+/kv/3uWyEd6KAU3wPZkrIo6ebPn/MOS4959QcPFw8RVvX01CHj2nkrJ5L X-Received: by 2002:a24:7f90:: with SMTP id r138mr1207894itc.163.1554313017374; Wed, 03 Apr 2019 10:36:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313017; cv=none; d=google.com; s=arc-20160816; b=V3/gjA6gmh6a7dHOPtTg1rkb1gvGaguwodjpPkoByk78lU0fP5ffNmVb5ihsRF1HzG 1c6WWtW3hf3BBZuGe3oaiNdmh9N6d0ZgNt/QYdTgr1Ls0Hnia4Bhlx1v4ZuYt+2AIEzC dHMI87U2JiRcwrWPJDeE6Qv4doS++7+e5Nu9NGHpCYroBZ0ix22EXe0+VWvYVzgi2I1O qGass3uJKDVg0bVuok2E1MZsoy/fo1I2h6Nil3MZFNC1CC6qb9e3+PwKrVxq72/EPGRQ N3r/C19vFK8jgIMmODkdXVdvQzT+UKfqiRws+/1lQvHLNJU/fi0DYDK/9zk5GCBvsCgP Csew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=557xk95OakwCk6FJ0C0EA1LbpPbU7nMPF7pIFxhjjUw=; b=AIIDm0d+LRlYbZEndnV3K67DXaUtXrmt6bVU7yRyVjzV8bZn0tTAWZf1vEU9w5pb1g QrFtQSBxZnSHWzbKwDH5Fayd/cIsVDH+17RqNFHPf1ma3waTIY3/+F7EEePc0LGOftXa ArYdgP25wLlMMrEgHUrNfhMgdTG23XPrc/XU9lF5aOosOyqlo831b4xAP3kIoyFe7si7 xXIn3azECWqUj//Ma7qbIBeNM1NE3C8BF5QAYB2a/+vZOZ1J3yYoFevfeacWevGBfT77 tYS0pvUQxxyazgirPECq8A0AKeLC3kXLlQW7agBXfecMjkmA5AiMEWd/dxlAGlA77DfK D9UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SFJGOxM0; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id 75si8623242itw.88.2019.04.03.10.36.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:36:57 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SFJGOxM0; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNnQk166346; Wed, 3 Apr 2019 17:35:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=557xk95OakwCk6FJ0C0EA1LbpPbU7nMPF7pIFxhjjUw=; b=SFJGOxM0x82gjwj/Br/Tc+Lkjt5bXEinVRIyhHOAKlVVQmQupdkriXJTTzNhWxREe9w1 w0Nl7diAhIzsCrldHqcj9mabQt4qhr0L3DCybhSZqCAMMqowLm0MS+s8VBHfcFkM55m0 ESknyl1X4cl2UqZ3ZiqXBUq7BAV6kZrkkEgghxjkqceObZ80u3Zsu57YJP8p35kjvBku dXhPigJE9uMc+kr1u15D+RkpnPH5TgHGWrn/3BXNqKR/7jY+IEUAYBgkZdKXN0F5VmiI hfV2B5EUjpENEmr6Izkl3nuC75/VPM0WzGvWfq6UVNy9lKjkHaQipdkHyUNdGeN3AKeo aw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 2rhyvtahkt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:42 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HYCNe081741; Wed, 3 Apr 2019 17:35:42 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3030.oracle.com with ESMTP id 2rm8f57yp3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:42 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZQ8g001194; Wed, 3 Apr 2019 17:35:26 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:26 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 02/13] x86: always set IF before oopsing from page fault Date: Wed, 3 Apr 2019 11:34:03 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen Oopsing might kill the task, via rewind_stack_do_exit() at the bottom, and that might sleep: Aug 23 19:30:27 xpfo kernel: [ 38.302714] BUG: sleeping function called from invalid context at ./include/linux/percpu-rwsem.h:33 Aug 23 19:30:27 xpfo kernel: [ 38.303837] in_atomic(): 0, irqs_disabled(): 1, pid: 1970, name: lkdtm_xpfo_test Aug 23 19:30:27 xpfo kernel: [ 38.304758] CPU: 3 PID: 1970 Comm: lkdtm_xpfo_test Tainted: G D 4.13.0-rc5+ #228 Aug 23 19:30:27 xpfo kernel: [ 38.305813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014 Aug 23 19:30:27 xpfo kernel: [ 38.306926] Call Trace: Aug 23 19:30:27 xpfo kernel: [ 38.307243] dump_stack+0x63/0x8b Aug 23 19:30:27 xpfo kernel: [ 38.307665] ___might_sleep+0xec/0x110 Aug 23 19:30:27 xpfo kernel: [ 38.308139] __might_sleep+0x45/0x80 Aug 23 19:30:27 xpfo kernel: [ 38.308593] exit_signals+0x21/0x1c0 Aug 23 19:30:27 xpfo kernel: [ 38.309046] ? blocking_notifier_call_chain+0x11/0x20 Aug 23 19:30:27 xpfo kernel: [ 38.309677] do_exit+0x98/0xbf0 Aug 23 19:30:27 xpfo kernel: [ 38.310078] ? smp_reader+0x27/0x40 [lkdtm] Aug 23 19:30:27 xpfo kernel: [ 38.310604] ? kthread+0x10f/0x150 Aug 23 19:30:27 xpfo kernel: [ 38.311045] ? read_user_with_flags+0x60/0x60 [lkdtm] Aug 23 19:30:27 xpfo kernel: [ 38.311680] rewind_stack_do_exit+0x17/0x20 To be safe, let's just always enable irqs. The particular case I'm hitting is: Aug 23 19:30:27 xpfo kernel: [ 38.278615] __bad_area_nosemaphore+0x1a9/0x1d0 Aug 23 19:30:27 xpfo kernel: [ 38.278617] bad_area_nosemaphore+0xf/0x20 Aug 23 19:30:27 xpfo kernel: [ 38.278618] __do_page_fault+0xd1/0x540 Aug 23 19:30:27 xpfo kernel: [ 38.278620] ? irq_work_queue+0x9b/0xb0 Aug 23 19:30:27 xpfo kernel: [ 38.278623] ? wake_up_klogd+0x36/0x40 Aug 23 19:30:27 xpfo kernel: [ 38.278624] trace_do_page_fault+0x3c/0xf0 Aug 23 19:30:27 xpfo kernel: [ 38.278625] do_async_page_fault+0x14/0x60 Aug 23 19:30:27 xpfo kernel: [ 38.278627] async_page_fault+0x28/0x30 When a fault is in kernel space which has been triggered by XPFO. Signed-off-by: Tycho Andersen CC: x86@kernel.org Tested-by: Khalid Aziz Cc: Khalid Aziz Signed-off-by: Tycho Andersen --- arch/x86/mm/fault.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 9d5c75f02295..7891add0913f 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -858,6 +858,12 @@ no_context(struct pt_regs *regs, unsigned long error_code, /* Executive summary in case the body of the oops scrolled away */ printk(KERN_DEFAULT "CR2: %016lx\n", address); + /* + * We're about to oops, which might kill the task. Make sure we're + * allowed to sleep. + */ + flags |= X86_EFLAGS_IF; + oops_end(flags, regs, sig); } From patchwork Wed Apr 3 17:34:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884285 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E28F91575 for ; Wed, 3 Apr 2019 17:37:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C0F2A27FA8 for ; Wed, 3 Apr 2019 17:37:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AD60828918; Wed, 3 Apr 2019 17:37:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D869727FA8 for ; Wed, 3 Apr 2019 17:37:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4416F6B0269; Wed, 3 Apr 2019 13:37:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 41D216B026A; Wed, 3 Apr 2019 13:37:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FA126B026B; Wed, 3 Apr 2019 13:37:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by kanga.kvack.org (Postfix) with ESMTP id E697F6B0269 for ; Wed, 3 Apr 2019 13:37:00 -0400 (EDT) Received: by mail-io1-f71.google.com with SMTP id w11so14294818iom.20 for ; Wed, 03 Apr 2019 10:37:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=2nOjJFWIwkGcFHXwrwiTDHk2pp6mkuTCa7ocMjVImag=; b=pIbMaT3O7jBrBtCKScrxHohz65O8OZn39gqB5m4YiCwupopJRt0ytaQkru4P8GEgsK b6N2cVslm+tx6wJRxx3ikfjhyR1wZzVftbX8m6CzjsmfWcAPT6VbP6ZV5YvWRp79M4Hg HTfWcjlAFBAvRo65r5g2FpcTWoSWQax3ANuNlcXbP+vm5xH1nsjwPKz12qEouAN+8/pG I2OCyVhy+c2qOmbjKZ1k+9kGDAkTa0w6JDeXFz11qleMcy2RZzkuQOCTk0RhFl/MGOR3 IMluk1BF8+RyPWX1i2TeSSdDRt1vKP0uuG1oHu8oWR4CZ1qLccq+xQfXjqyfjm6Ke4cZ M2gQ== X-Gm-Message-State: APjAAAUufwsG8ZEcndX7Wfi6lL4OZKPl8KgQ/L47rKpXQy40DtnbI3g8 hT9QFX/CqG8UdoXfEdG9oiF+LtzkjwT/77AdWrCd4LrlNQtlri95MS6GFX76pCEpK1/hpmUoGuP zpv7gHU1Zg4ZWygFezx5DDhZX5ACr5L+zKrkpUJIkniX61EPg5uptLdU2icIXt9EM/g== X-Received: by 2002:a24:7f8f:: with SMTP id r137mr1123773itc.56.1554313020605; Wed, 03 Apr 2019 10:37:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqwG3zhwwY/Xa+Z527cVGaCX6Onx6RcVbghp+zaccxlnlUlpcMiPYKuHogd6HRKvzTQGbISx X-Received: by 2002:a24:7f8f:: with SMTP id r137mr1123673itc.56.1554313018910; Wed, 03 Apr 2019 10:36:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313018; cv=none; d=google.com; s=arc-20160816; b=kucHyNDwAIdV5OtUeWUh1TDOMDlLfj8wGvBXifBxNabiGf7EVCYaWbsRCIA1C5voLN ICFvCZwkP/nIa0uPpCpDkrL/0ZbumxXBfASnuIQQT7/SgNZJkXP+MT8IDvdifSf+nVdX EnGbWWU2m1bCfaAb20yiBfJjB+9JmY+0cNxgQxm3iwsU5lfGyNRW/G6J9IzzMxGQn5hG OEjihhPrzymrQtSEoH72tjAH+ZEqSEC8Z1lNBEKIlfhKBPmWQbB9t6Cj13H7obPWT6we 23hWFfWYvF2DYppFPNAy4CjHfy8d8zPlMvWNK4M+pFsAkGyQHpacjahz6jog4wZBhnMy 9s0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=2nOjJFWIwkGcFHXwrwiTDHk2pp6mkuTCa7ocMjVImag=; b=u7li2/xNVHc1X8RYy5e0F/d2Ei2vWxolvVtPSso6864wJU8FNM10Y6TIzrEfUG5P54 wsJiey+TkUaSN/t6RopFfJl+QmPmXgMQtiHRjbNm341wNQ0SH01HwSiTlS0hM8645ZzT P4m+s4C8gZNQJzUHYH3CcBavYWxsuAs+aoWEjSmTikyY2L99r806cpi5gwFmXdM7rkLA gyEgDPm4gFbjSwiGibgbdAtHDbIaTWTincaVQEyTZESS+SCB0aaC1eMX+RKRP1yopR4J ndiTc85Ili2DOphyhwXnYkcxhsR/IzlA92pV/pWbl8ueKi571wZ7qXBYBJs/lnXPlzK3 WBdA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=r+t4nMq5; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id q66si8747905itb.76.2019.04.03.10.36.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:36:58 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=r+t4nMq5; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HO2kH166464; Wed, 3 Apr 2019 17:35:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=2nOjJFWIwkGcFHXwrwiTDHk2pp6mkuTCa7ocMjVImag=; b=r+t4nMq5KbsfIuHwqBZBCktClDXcFzD57iXWgb71zr5ONuOUJXo46Pj315Z99T/K07by iswiYdI6AZjxBFII07JF2Cjiwwhkt9tOjKf6z2jKudrDtLFUgi4KI3UQnAgm/QM4HYMe 2JtWs2cEzgSKuNPnByTklqAaKpkJDKMx+VX6zOo2ygYx39gO2A0K09rP7ghv38FSgp/p j+GMWNqPEvtkJnARAk54wdWBV93ZEErHkmSJSJgvX3CLIkAWJEBMSLkIy1IlNFvbTxlr +vR8xgwiB2wFuNf78NNU1iEdTBIBpc7N1m/8UlRx7PIat+i2zzw3wT7UoHa0z2GMJW9R 7g== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 2rhyvtahkw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:43 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HYC0e081772; Wed, 3 Apr 2019 17:35:42 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userp3030.oracle.com with ESMTP id 2rm8f57yp2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:42 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZVKk028584; Wed, 3 Apr 2019 17:35:31 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:30 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 03/13] mm: Add support for eXclusive Page Frame Ownership (XPFO) Date: Wed, 3 Apr 2019 11:34:04 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger This patch adds basic support infrastructure for XPFO which protects against 'ret2dir' kernel attacks. The basic idea is to enforce exclusive ownership of page frames by either the kernel or userspace, unless explicitly requested by the kernel. Whenever a page destined for userspace is allocated, it is unmapped from physmap (the kernel's page table). When such a page is reclaimed from userspace, it is mapped back to physmap. Individual architectures can enable full XPFO support using this infrastructure by supplying architecture specific pieces. Additional fields in the page struct are used for XPFO housekeeping, specifically: - two flags to distinguish user vs. kernel pages and to tag unmapped pages. - a reference counter to balance kmap/kunmap operations. - a lock to serialize access to the XPFO fields. This patch is based on the work of Vasileios P. Kemerlis et al. who published their work in this paper: http://www.cs.columbia.edu/~vpk/papers/ret2dir.sec14.pdf CC: x86@kernel.org Suggested-by: Vasileios P. Kemerlis Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Marco Benatto [jsteckli@amazon.de: encode all XPFO info in struct page] Signed-off-by: Julian Stecklina Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- v9: * Do not use page extensions. Encode all xpfo information in struct page (Julian Stecklina). * Split architecture specific code into its own separate patch * Move kmap*() to include/linux/xpfo.h for cleaner code as suggested for an earlier version of this patch * Use irq versions of spin_lock to address possible deadlock around xpfo_lock caused by interrupts. * Incorporated various feedback provided on v6 patch way back. v6: * use flush_tlb_kernel_range() instead of __flush_tlb_one, so we flush the tlb entry on all CPUs when unmapping it in kunmap * handle lookup_page_ext()/lookup_xpfo() returning NULL * drop lots of BUG()s in favor of WARN() * don't disable irqs in xpfo_kmap/xpfo_kunmap, export __split_large_page so we can do our own alloc_pages(GFP_ATOMIC) to pass it .../admin-guide/kernel-parameters.txt | 6 + include/linux/highmem.h | 31 +--- include/linux/mm_types.h | 8 + include/linux/page-flags.h | 23 ++- include/linux/xpfo.h | 147 ++++++++++++++++++ include/trace/events/mmflags.h | 10 +- mm/Makefile | 1 + mm/compaction.c | 2 +- mm/internal.h | 2 +- mm/page_alloc.c | 10 +- mm/page_isolation.c | 2 +- mm/xpfo.c | 106 +++++++++++++ security/Kconfig | 27 ++++ 13 files changed, 337 insertions(+), 38 deletions(-) create mode 100644 include/linux/xpfo.h create mode 100644 mm/xpfo.c diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 858b6c0b9a15..9b36da94760e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2997,6 +2997,12 @@ nox2apic [X86-64,APIC] Do not enable x2APIC mode. + noxpfo [XPFO] Disable eXclusive Page Frame Ownership (XPFO) + when CONFIG_XPFO is on. Physical pages mapped into + user applications will also be mapped in the + kernel's address space as if CONFIG_XPFO was not + enabled. + cpu0_hotplug [X86] Turn on CPU0 hotplug feature when CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off. Some features depend on CPU0. Known dependencies are: diff --git a/include/linux/highmem.h b/include/linux/highmem.h index ea5cdbd8c2c3..59a1a5fa598d 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -8,6 +8,7 @@ #include #include #include +#include #include @@ -77,36 +78,6 @@ static inline struct page *kmap_to_page(void *addr) static inline unsigned long totalhigh_pages(void) { return 0UL; } -#ifndef ARCH_HAS_KMAP -static inline void *kmap(struct page *page) -{ - might_sleep(); - return page_address(page); -} - -static inline void kunmap(struct page *page) -{ -} - -static inline void *kmap_atomic(struct page *page) -{ - preempt_disable(); - pagefault_disable(); - return page_address(page); -} -#define kmap_atomic_prot(page, prot) kmap_atomic(page) - -static inline void __kunmap_atomic(void *addr) -{ - pagefault_enable(); - preempt_enable(); -} - -#define kmap_atomic_pfn(pfn) kmap_atomic(pfn_to_page(pfn)) - -#define kmap_flush_unused() do {} while(0) -#endif - #endif /* CONFIG_HIGHMEM */ #if defined(CONFIG_HIGHMEM) || defined(CONFIG_X86_32) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2c471a2c43fa..d17d33f36a01 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -204,6 +204,14 @@ struct page { #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif + +#ifdef CONFIG_XPFO + /* Counts the number of times this page has been kmapped. */ + atomic_t xpfo_mapcount; + + /* Serialize kmap/kunmap of this page */ + spinlock_t xpfo_lock; +#endif } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 39b4494e29f1..3622e8c33522 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -101,6 +101,10 @@ enum pageflags { #if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT) PG_young, PG_idle, +#endif +#ifdef CONFIG_XPFO + PG_xpfo_user, /* Page is allocated to user-space */ + PG_xpfo_unmapped, /* Page is unmapped from the linear map */ #endif __NR_PAGEFLAGS, @@ -398,6 +402,22 @@ TESTCLEARFLAG(Young, young, PF_ANY) PAGEFLAG(Idle, idle, PF_ANY) #endif +#ifdef CONFIG_XPFO +PAGEFLAG(XpfoUser, xpfo_user, PF_ANY) +TESTCLEARFLAG(XpfoUser, xpfo_user, PF_ANY) +TESTSETFLAG(XpfoUser, xpfo_user, PF_ANY) +#define __PG_XPFO_USER (1UL << PG_xpfo_user) +PAGEFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +TESTCLEARFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +TESTSETFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +#define __PG_XPFO_UNMAPPED (1UL << PG_xpfo_unmapped) +#else +#define __PG_XPFO_USER 0 +PAGEFLAG_FALSE(XpfoUser) +#define __PG_XPFO_UNMAPPED 0 +PAGEFLAG_FALSE(XpfoUnmapped) +#endif + /* * On an anonymous page mapped into a user virtual memory area, * page->mapping points to its anon_vma, not to a struct address_space; @@ -780,7 +800,8 @@ static inline void ClearPageSlabPfmemalloc(struct page *page) * alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON) + (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON & ~__PG_XPFO_USER & \ + ~__PG_XPFO_UNMAPPED) #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h new file mode 100644 index 000000000000..93a1b5aceca3 --- /dev/null +++ b/include/linux/xpfo.h @@ -0,0 +1,147 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2017 Docker, Inc. + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * Tycho Andersen + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#ifndef _LINUX_XPFO_H +#define _LINUX_XPFO_H + +#include +#include +#include + +struct page; + +#ifdef CONFIG_XPFO + +DECLARE_STATIC_KEY_TRUE(xpfo_inited); + +/* Architecture specific implementations */ +void set_kpte(void *kaddr, struct page *page, pgprot_t prot); +void xpfo_flush_kernel_tlb(struct page *page, int order); + +void xpfo_init_single_page(struct page *page); + +static inline void xpfo_kmap(void *kaddr, struct page *page) +{ + unsigned long flags; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + if (!PageXpfoUser(page)) + return; + + /* + * The page was previously allocated to user space, so + * map it back into the kernel if needed. No TLB flush required. + */ + spin_lock_irqsave(&page->xpfo_lock, flags); + + if ((atomic_inc_return(&page->xpfo_mapcount) == 1) && + TestClearPageXpfoUnmapped(page)) + set_kpte(kaddr, page, PAGE_KERNEL); + + spin_unlock_irqrestore(&page->xpfo_lock, flags); +} + +static inline void xpfo_kunmap(void *kaddr, struct page *page) +{ + unsigned long flags; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + if (!PageXpfoUser(page)) + return; + + /* + * The page is to be allocated back to user space, so unmap it from + * the kernel, flush the TLB and tag it as a user page. + */ + spin_lock_irqsave(&page->xpfo_lock, flags); + + if (atomic_dec_return(&page->xpfo_mapcount) == 0) { +#ifdef CONFIG_XPFO_DEBUG + WARN_ON(PageXpfoUnmapped(page)); +#endif + SetPageXpfoUnmapped(page); + set_kpte(kaddr, page, __pgprot(0)); + xpfo_flush_kernel_tlb(page, 0); + } + + spin_unlock_irqrestore(&page->xpfo_lock, flags); +} + +void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, bool will_map); +void xpfo_free_pages(struct page *page, int order); + +#else /* !CONFIG_XPFO */ + +static inline void xpfo_init_single_page(struct page *page) { } + +static inline void xpfo_kmap(void *kaddr, struct page *page) { } +static inline void xpfo_kunmap(void *kaddr, struct page *page) { } +static inline void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, + bool will_map) { } +static inline void xpfo_free_pages(struct page *page, int order) { } + +static inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) { } +static inline void xpfo_flush_kernel_tlb(struct page *page, int order) { } + +#endif /* CONFIG_XPFO */ + +#if (!defined(CONFIG_HIGHMEM)) && (!defined(ARCH_HAS_KMAP)) +static inline void *kmap(struct page *page) +{ + void *kaddr; + + might_sleep(); + kaddr = page_address(page); + xpfo_kmap(kaddr, page); + return kaddr; +} + +static inline void kunmap(struct page *page) +{ + xpfo_kunmap(page_address(page), page); +} + +static inline void *kmap_atomic(struct page *page) +{ + void *kaddr; + + preempt_disable(); + pagefault_disable(); + kaddr = page_address(page); + xpfo_kmap(kaddr, page); + return kaddr; +} + +#define kmap_atomic_prot(page, prot) kmap_atomic(page) + +static inline void __kunmap_atomic(void *addr) +{ + xpfo_kunmap(addr, virt_to_page(addr)); + pagefault_enable(); + preempt_enable(); +} + +#define kmap_atomic_pfn(pfn) kmap_atomic(pfn_to_page(pfn)) + +#define kmap_flush_unused() do {} while (0) + +#endif + +#endif /* _LINUX_XPFO_H */ diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index a1675d43777e..6bb000bb366f 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -79,6 +79,12 @@ #define IF_HAVE_PG_IDLE(flag,string) #endif +#ifdef CONFIG_XPFO +#define IF_HAVE_PG_XPFO(flag,string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_XPFO(flag,string) +#endif + #define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \ @@ -105,7 +111,9 @@ IF_HAVE_PG_MLOCK(PG_mlocked, "mlocked" ) \ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ -IF_HAVE_PG_IDLE(PG_idle, "idle" ) +IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ +IF_HAVE_PG_XPFO(PG_xpfo_user, "xpfo_user" ) \ +IF_HAVE_PG_XPFO(PG_xpfo_unmapped, "xpfo_unmapped" ) \ #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/Makefile b/mm/Makefile index d210cc9d6f80..e99e1e6ae5ae 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,3 +99,4 @@ obj-$(CONFIG_HARDENED_USERCOPY) += usercopy.o obj-$(CONFIG_PERCPU_STATS) += percpu-stats.o obj-$(CONFIG_HMM) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/mm/compaction.c b/mm/compaction.c index ef29490b0f46..fdd5d9783adb 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -78,7 +78,7 @@ static void map_pages(struct list_head *list) order = page_private(page); nr_pages = 1 << order; - post_alloc_hook(page, order, __GFP_MOVABLE); + post_alloc_hook(page, order, __GFP_MOVABLE, false); if (order) split_page(page, order); diff --git a/mm/internal.h b/mm/internal.h index f4a7bb02decf..e076e51376df 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -165,7 +165,7 @@ extern void memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order); extern void prep_compound_page(struct page *page, unsigned int order); extern void post_alloc_hook(struct page *page, unsigned int order, - gfp_t gfp_flags); + gfp_t gfp_flags, bool will_map); extern int user_min_free_kbytes; #if defined CONFIG_COMPACTION || defined CONFIG_CMA diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0b9f577b1a2a..2e0dda1322a2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1062,6 +1062,7 @@ static __always_inline bool free_pages_prepare(struct page *page, if (bad) return false; + xpfo_free_pages(page, order); page_cpupid_reset_last(page); page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); @@ -1229,6 +1230,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif + xpfo_init_single_page(page); } #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -1938,7 +1940,7 @@ static bool check_new_pages(struct page *page, unsigned int order) } inline void post_alloc_hook(struct page *page, unsigned int order, - gfp_t gfp_flags) + gfp_t gfp_flags, bool will_map) { set_page_private(page, 0); set_page_refcounted(page); @@ -1947,6 +1949,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, kernel_map_pages(page, 1 << order, 1); kernel_poison_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); + xpfo_alloc_pages(page, order, gfp_flags, will_map); set_page_owner(page, order, gfp_flags); } @@ -1954,10 +1957,11 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags unsigned int alloc_flags) { int i; + bool needs_zero = !free_pages_prezeroed() && (gfp_flags & __GFP_ZERO); - post_alloc_hook(page, order, gfp_flags); + post_alloc_hook(page, order, gfp_flags, needs_zero); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) + if (needs_zero) for (i = 0; i < (1 << order); i++) clear_highpage(page + i); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index ce323e56b34d..d8730dd134a9 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -137,7 +137,7 @@ static void unset_migratetype_isolate(struct page *page, unsigned migratetype) out: spin_unlock_irqrestore(&zone->lock, flags); if (isolated_page) { - post_alloc_hook(page, order, __GFP_MOVABLE); + post_alloc_hook(page, order, __GFP_MOVABLE, false); __free_pages(page, order); } } diff --git a/mm/xpfo.c b/mm/xpfo.c new file mode 100644 index 000000000000..b74fee0479e7 --- /dev/null +++ b/mm/xpfo.c @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Docker, Inc. + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * Tycho Andersen + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include +#include +#include + +#include + +DEFINE_STATIC_KEY_TRUE(xpfo_inited); +EXPORT_SYMBOL_GPL(xpfo_inited); + +static int __init noxpfo_param(char *str) +{ + static_branch_disable(&xpfo_inited); + + return 0; +} + +early_param("noxpfo", noxpfo_param); + +bool __init xpfo_enabled(void) +{ + if (!static_branch_unlikely(&xpfo_inited)) + return false; + else + return true; +} + +void __meminit xpfo_init_single_page(struct page *page) +{ + spin_lock_init(&page->xpfo_lock); +} + +void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, bool will_map) +{ + int i; + bool flush_tlb = false; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + for (i = 0; i < (1 << order); i++) { +#ifdef CONFIG_XPFO_DEBUG + WARN_ON(PageXpfoUser(page + i)); + WARN_ON(PageXpfoUnmapped(page + i)); + lockdep_assert_held(&(page + i)->xpfo_lock); + WARN_ON(atomic_read(&(page + i)->xpfo_mapcount)); +#endif + if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { + /* + * Tag the page as a user page and flush the TLB if it + * was previously allocated to the kernel. + */ + if ((!TestSetPageXpfoUser(page + i)) || !will_map) { + SetPageXpfoUnmapped(page + i); + flush_tlb = true; + } + } else { + /* Tag the page as a non-user (kernel) page */ + ClearPageXpfoUser(page + i); + } + } + + if (flush_tlb) { + set_kpte(page_address(page), page, __pgprot(0)); + xpfo_flush_kernel_tlb(page, order); + } +} + +void xpfo_free_pages(struct page *page, int order) +{ + int i; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + for (i = 0; i < (1 << order); i++) { +#ifdef CONFIG_XPFO_DEBUG + WARN_ON(atomic_read(&(page + i)->xpfo_mapcount)); +#endif + + /* + * Map the page back into the kernel if it was previously + * allocated to user space. + */ + if (TestClearPageXpfoUser(page + i)) { + ClearPageXpfoUnmapped(page + i); + set_kpte(page_address(page + i), page + i, + PAGE_KERNEL); + } + } +} diff --git a/security/Kconfig b/security/Kconfig index e4fe2f3c2c65..3636ba7e2615 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -6,6 +6,33 @@ menu "Security options" source "security/keys/Kconfig" +config ARCH_SUPPORTS_XPFO + bool + +config XPFO + bool "Enable eXclusive Page Frame Ownership (XPFO)" + depends on ARCH_SUPPORTS_XPFO + help + This option offers protection against 'ret2dir' kernel attacks. + When enabled, every time a page frame is allocated to user space, it + is unmapped from the direct mapped RAM region in kernel space + (physmap). Similarly, when a page frame is freed/reclaimed, it is + mapped back to physmap. + + There is a slight performance impact when this option is enabled. + + If in doubt, say "N". + +config XPFO_DEBUG + bool "Enable debugging of XPFO" + depends on XPFO + help + Enables additional checking of XPFO data structures that help find + bugs in the XPFO implementation. This option comes with a slight + performance cost. + + If in doubt, say "N". + config SECURITY_DMESG_RESTRICT bool "Restrict unprivileged access to the kernel syslog" default n From patchwork Wed Apr 3 17:34:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68C3817E0 for ; Wed, 3 Apr 2019 17:37:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FD35288F3 for ; Wed, 3 Apr 2019 17:37:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 413A1288F4; Wed, 3 Apr 2019 17:37:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDE4028906 for ; Wed, 3 Apr 2019 17:37:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A6026B026C; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 332956B026B; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F016B026C; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id CB67A6B026A for ; Wed, 3 Apr 2019 13:37:01 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id k65so13055731ybc.12 for ; Wed, 03 Apr 2019 10:37:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=+01mLH7UeRlyUX9m4clHTbWXXa1hEWoneLzdofN2ooQ=; b=WXJLdD1hAHrvK2RnsONH6+cQA8pn4neko6473BSh4bLpH9VJY24hqpFXamBvygCI3s jPPOkHYPgbcz5aN9l92Pa7Nf1RA+/QjabeV2rxq9kdGL2IEDsg32oXfd89C6i4jU+iv0 Z7M/zwqOSmfPlDkAVGC8KXC3omwdG8WkFGWgsOla01NK8Qtdq9jn1j8OYL5A7pRSJK/3 iIbM4b6z/t3eSlzJ78N1rfH8UgF07NdMq+mbbBGpt2Ge03aOsn+DEAc8W3pbXhqWpFNP 0nTSKz2nu9bfRAZeP3KYll8nD2x/qaDBD/lA0USy9KFbKEYvtSsUrsdA43vbp/sH1Aec BxOA== X-Gm-Message-State: APjAAAXigGPM3415xogqiP7CbCWpf8djkmIIHMJ1wIMmQEX+GVePxgUw 3vUCvqVxCtjggZwydx5rgcr0QH/pcPLaVyJ40xDucYCYWg1g+mTWNr5N2XqTxr7I+hbErgYrW/0 f1LtgohCFwjoPRAM/iiN4LFwU7ZUHOc/765+DVAUEsPuk/d2s42KnREwCGE1QMronlQ== X-Received: by 2002:a25:bbc2:: with SMTP id c2mr1196903ybk.356.1554313021533; Wed, 03 Apr 2019 10:37:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqznKmWJFUBPRFR4TC0P99xt6TZ4cyMGkjoWBHgBE1tWO1ZOm42hRNgF1VTZynmXDzr8LpoA X-Received: by 2002:a25:bbc2:: with SMTP id c2mr1196800ybk.356.1554313020434; Wed, 03 Apr 2019 10:37:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313020; cv=none; d=google.com; s=arc-20160816; b=N8rbAkUmK/8bBk4TShDOTf2szCIgT7AqpANmYYwuCdSWGXq3nlZWB6DcOKwMenOHSh deXUP/Mkerma/3lb9nh41hX917d0NKhG4wZ2R+ITTvpHs0EwCQkG5y7VF+3prtSsJijm ijcH6DbcfzIlqMJ/hn3wkUTxEyPT55evnJDqJC1oTaW1bzn9tvG/vTbdSyh5sw14lZAR 8TNqRM6UjL9/rgTIj+c8pBfXzi/ESsk75UM2/RRYwi4cnYhXCB70L43Cjc51Zjmdf9mi ns/7Gq9dL9XeNnxt+2JDmj87Ho9X8N6FKbZOfIjSVtt+vFQ8KEizf6SuGAuXWnolNUsR 0s8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=+01mLH7UeRlyUX9m4clHTbWXXa1hEWoneLzdofN2ooQ=; b=YIyK9MB0eNeZXNWFSpbdbM6ZwvxUlYOUNdvaKi6f2T8CWdG7dX2rSBqBckTCJ7TwyG fZ/ia9y4tetdcARbhJzDM8AAWG59GGiNCJFX5Mk6z9KAnFmR74L6TOD9VUTkjfRarUkH hlqGejDita3NPYQ9rx54pg0lvP2eXrQN3rR9f2yiPxub635wDnICRCW/5+Z5MaENdGwC /Wb8N2+QojSRcdWDKWyLcGZYapqnxtbeVaZ4rUuTNWR9R9EfbsJZwkaq35htyTcjWU+X YeRB+CdbcmXpuGv/+I0l4VVdpm9wmu39t9Lgqt4EUKd4IK5oa2dcSssnEpcR7D4H15Dl GxVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="uA/GfXn2"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id s187si3340975ybf.317.2019.04.03.10.37.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:00 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="uA/GfXn2"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNml0172016; Wed, 3 Apr 2019 17:35:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=+01mLH7UeRlyUX9m4clHTbWXXa1hEWoneLzdofN2ooQ=; b=uA/GfXn2tZZmE7mbGgEjJtH6lkX7duJMRcKDROSU1jH8FAus0LkOk3aZo51FvqFswZ/l uAJ1y7abrVRzgsC2B86iaQT35HVOnjmEXz3KKtMfg647l24qTnwuLISV3AM22cHm+G8r 7BQ+BB4Rmk0UCL9EOGF2+mLd/4ZcZ3EvofXyofppe5A3iPWuAiazort95ATH7khXJvlW VgR+opBTIGKzeN/v1eHhAbWrlDrqdR4tIA8lfX7qM/V+5CdnokgKycQo83eFdt7Nz2w5 sJ10GZj9aZx9sUyw2MsjHTbydAnB2K9QWB6v6aWJQ0p9ODNrYhlK4CgSamr9LWXaA1ow 3g== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 2rhwydapdw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:43 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZgwL087983; Wed, 3 Apr 2019 17:35:42 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 2rm8f67xu5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:42 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZZq8001238; Wed, 3 Apr 2019 17:35:35 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:35 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 04/13] xpfo, x86: Add support for XPFO for x86-64 Date: Wed, 3 Apr 2019 11:34:05 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger This patch adds support for XPFO for x86-64. It uses the generic infrastructure in place for XPFO and adds the architecture specific bits to enable XPFO on x86-64. CC: x86@kernel.org Suggested-by: Vasileios P. Kemerlis Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Marco Benatto Signed-off-by: Julian Stecklina Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- .../admin-guide/kernel-parameters.txt | 10 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 26 ++++ arch/x86/mm/Makefile | 2 + arch/x86/mm/pageattr.c | 23 +--- arch/x86/mm/xpfo.c | 123 ++++++++++++++++++ include/linux/xpfo.h | 2 + 7 files changed, 162 insertions(+), 25 deletions(-) create mode 100644 arch/x86/mm/xpfo.c diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9b36da94760e..e65e3bc1efe0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2997,11 +2997,11 @@ nox2apic [X86-64,APIC] Do not enable x2APIC mode. - noxpfo [XPFO] Disable eXclusive Page Frame Ownership (XPFO) - when CONFIG_XPFO is on. Physical pages mapped into - user applications will also be mapped in the - kernel's address space as if CONFIG_XPFO was not - enabled. + noxpfo [XPFO,X86-64] Disable eXclusive Page Frame + Ownership (XPFO) when CONFIG_XPFO is on. Physical + pages mapped into user applications will also be + mapped in the kernel's address space as if + CONFIG_XPFO was not enabled. cpu0_hotplug [X86] Turn on CPU0 hotplug feature when CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 68261430fe6e..122786604252 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -209,6 +209,7 @@ config X86 select USER_STACKTRACE_SUPPORT select VIRT_TO_BUS select X86_FEATURE_NAMES if PROC_FS + select ARCH_SUPPORTS_XPFO if X86_64 config INSTRUCTION_DECODER def_bool y diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 2779ace16d23..5c0e1581fa56 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1437,6 +1437,32 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } +/* + * The current flushing context - we pass it instead of 5 arguments: + */ +struct cpa_data { + unsigned long *vaddr; + pgd_t *pgd; + pgprot_t mask_set; + pgprot_t mask_clr; + unsigned long numpages; + unsigned long curpage; + unsigned long pfn; + unsigned int flags; + unsigned int force_split : 1, + force_static_prot : 1; + struct page **pages; +}; + + +int +should_split_large_page(pte_t *kpte, unsigned long address, + struct cpa_data *cpa); +extern spinlock_t cpa_lock; +int +__split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, + struct page *base); + #include #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 4b101dd6e52f..93b0fdaf4a99 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -53,3 +53,5 @@ obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 14e6119838a6..530b5df0617e 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -28,23 +28,6 @@ #include "mm_internal.h" -/* - * The current flushing context - we pass it instead of 5 arguments: - */ -struct cpa_data { - unsigned long *vaddr; - pgd_t *pgd; - pgprot_t mask_set; - pgprot_t mask_clr; - unsigned long numpages; - unsigned long curpage; - unsigned long pfn; - unsigned int flags; - unsigned int force_split : 1, - force_static_prot : 1; - struct page **pages; -}; - enum cpa_warn { CPA_CONFLICT, CPA_PROTECT, @@ -59,7 +42,7 @@ static const int cpa_warn_level = CPA_PROTECT; * entries change the page attribute in parallel to some other cpu * splitting a large page entry along with changing the attribute. */ -static DEFINE_SPINLOCK(cpa_lock); +DEFINE_SPINLOCK(cpa_lock); #define CPA_FLUSHTLB 1 #define CPA_ARRAY 2 @@ -876,7 +859,7 @@ static int __should_split_large_page(pte_t *kpte, unsigned long address, return 0; } -static int should_split_large_page(pte_t *kpte, unsigned long address, +int should_split_large_page(pte_t *kpte, unsigned long address, struct cpa_data *cpa) { int do_split; @@ -926,7 +909,7 @@ static void split_set_pte(struct cpa_data *cpa, pte_t *pte, unsigned long pfn, set_pte(pte, pfn_pte(pfn, ref_prot)); } -static int +int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, struct page *base) { diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c new file mode 100644 index 000000000000..3045bb7e4659 --- /dev/null +++ b/arch/x86/mm/xpfo.c @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include + +#include + +extern spinlock_t cpa_lock; + +/* Update a single kernel page table entry */ +inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) +{ + unsigned int level; + pgprot_t msk_clr; + pte_t *pte = lookup_address((unsigned long)kaddr, &level); + + if (unlikely(!pte)) { + WARN(1, "xpfo: invalid address %p\n", kaddr); + return; + } + + switch (level) { + case PG_LEVEL_4K: + set_pte_atomic(pte, pfn_pte(page_to_pfn(page), + canon_pgprot(prot))); + break; + case PG_LEVEL_2M: + case PG_LEVEL_1G: { + struct cpa_data cpa = { }; + int do_split; + + if (level == PG_LEVEL_2M) + msk_clr = pmd_pgprot(*(pmd_t *)pte); + else + msk_clr = pud_pgprot(*(pud_t *)pte); + + cpa.vaddr = kaddr; + cpa.pages = &page; + cpa.mask_set = prot; + cpa.mask_clr = msk_clr; + cpa.numpages = 1; + cpa.flags = 0; + cpa.curpage = 0; + cpa.force_split = 0; + + + do_split = should_split_large_page(pte, (unsigned long)kaddr, + &cpa); + if (do_split) { + struct page *base; + + base = alloc_pages(GFP_ATOMIC, 0); + if (!base) { + WARN(1, "xpfo: failed to split large page\n"); + break; + } + + if (!debug_pagealloc_enabled()) + spin_lock(&cpa_lock); + if (__split_large_page(&cpa, pte, (unsigned long)kaddr, + base) < 0) { + __free_page(base); + WARN(1, "xpfo: failed to split large page\n"); + } + if (!debug_pagealloc_enabled()) + spin_unlock(&cpa_lock); + } + + break; + } + case PG_LEVEL_512G: + /* fallthrough, splitting infrastructure doesn't + * support 512G pages. + */ + default: + WARN(1, "xpfo: unsupported page level %x\n", level); + } + +} +EXPORT_SYMBOL_GPL(set_kpte); + +inline void xpfo_flush_kernel_tlb(struct page *page, int order) +{ + int level; + unsigned long size, kaddr; + + kaddr = (unsigned long)page_address(page); + + if (unlikely(!lookup_address(kaddr, &level))) { + WARN(1, "xpfo: invalid address to flush %lx %d\n", kaddr, + level); + return; + } + + switch (level) { + case PG_LEVEL_4K: + size = PAGE_SIZE; + break; + case PG_LEVEL_2M: + size = PMD_SIZE; + break; + case PG_LEVEL_1G: + size = PUD_SIZE; + break; + default: + WARN(1, "xpfo: unsupported page level %x\n", level); + return; + } + + flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); +} +EXPORT_SYMBOL_GPL(xpfo_flush_kernel_tlb); diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 93a1b5aceca3..c1d232da7ee0 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -25,6 +25,8 @@ struct page; #ifdef CONFIG_XPFO +#include + DECLARE_STATIC_KEY_TRUE(xpfo_inited); /* Architecture specific implementations */ From patchwork Wed Apr 3 17:34:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884297 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 76CA817E0 for ; Wed, 3 Apr 2019 17:37:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BD2727FA8 for ; Wed, 3 Apr 2019 17:37:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D803288F4; Wed, 3 Apr 2019 17:37:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D42527FA8 for ; Wed, 3 Apr 2019 17:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C1F06B026F; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 94A686B0271; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 775F06B0272; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 5473C6B0271 for ; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id s11so12705049ywa.18 for ; Wed, 03 Apr 2019 10:37:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=b6onsz6VNX7DtbTr5TqzUNVhpOrN6pO3jfRzW4prgsk=; b=doYu/AyARmrG+2JeDmBhQ1AjYLEbzMqgN2sF9R00ue+G3Q6QZsDw+0HajFwK6o3Yd5 XwT0Swr+siSSKmAhqjPU2uGwijQyyQ6qXrqZ84+s3ha06glVley+mBStUaP/hPtsv6Wg PH05tUJx4hGgwra7zuB5Z65QDlvngaQxJDU3V6OlRj5VL+b+RhNhIFZLWr0i7btjH1nI jo1JTloiSWGdt8+E82QjyUHJyZJqnWyAMbbiTDTvW2g7LnNzruF7eOUk3P8PljKT3wIl 6tADM/QRjCr4C3z69k9OqNMXp8BFiUdiMNY7gGZQJpv3tgnTT/cO/M5/TMlo9FLtfE+d +hjA== X-Gm-Message-State: APjAAAUxx3qIY/rxQYMu8eO6+lE8Y7DrJmM5sBPJ8s3plcjshDaUO1ED FhbYRjvKvAVcBSVy1gqN4pfarG6wYY8TVxM8n5b+3oLfmZZqIj3H58jB5JhT4wNqDe1/qkjgOUA lFHa18Iwkufw2yVMuzVju4kwOCYq5CwzPFyrcX0uf8g0MVvV0PRRMqcOIIZcRbZw4YQ== X-Received: by 2002:a25:e4c6:: with SMTP id b189mr1168070ybh.454.1554313024108; Wed, 03 Apr 2019 10:37:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqxUVzI1Nig6Qgtr8WKBhtQ/tJ/530wEIgI/WFnZyZY5ezmEpqTF/ksnwCK/6j8NhXincczG X-Received: by 2002:a25:e4c6:: with SMTP id b189mr1167996ybh.454.1554313023167; Wed, 03 Apr 2019 10:37:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313023; cv=none; d=google.com; s=arc-20160816; b=SX6twYi3nkTY5hVZFbSOseXBeZYLx85OwWfNmIzp1Dts9GtH0oo706R2DjDGRR0jzi owKB78o4DEkITHUTtkkycE6JLSBfmNzrHlWnWyzI0DHAf7zSxnItKZZ+47oxS2Ids68d ewWrcY+/mD/nF/3zdawH53E58eKsTi/g9561uv3z4Y5zoB3KXFOuxgjvMcJ+ISu+hZLY dcN+lY5D5kkcMfJS+Y3lLKjN/sn6a3S2fDt85cy7cuLz7dA554Xs8dYAB8GqSYTre8fe +10LkpIy9XcY75M8r3kw+8e0ei8tOHCiyi2dB2q6tHKh3CPIxosdFrWAdMx8o7P+cCl+ nIfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=b6onsz6VNX7DtbTr5TqzUNVhpOrN6pO3jfRzW4prgsk=; b=cIuLdyLoE/dNFkTvU7PPMHmyIqOuCDCWJAinOIuuBxkk80gb5ut70UgnfK79WLaToy hP8vfhSIRaVWMQSdLUpRJe2an6ifZIMFY6BRcMQFtH8FmoAuGs0qcA/whuLheDHPt5wo dzRg/hr5L1J/YPhMXZOB7Kd1lkebmJV7GXwcHSLCvG9p2vbAXnQRDbgjQszx2/bA2A4v sp7qOiHYRYk1kOzBanYlTVm8EdATBm09nrFpWQlLJVqF7LJCi7PgBislWE7ni2psXZ37 AJ9j4onsUr1it0TMf9AfyGI/d9UN87DGg6qCWgD7uHlYXqdpDE1AQTFhbXMwwXbmgaPK hllw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=m6ea2Uwq; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id g5si10686231ybb.287.2019.04.03.10.37.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:03 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=m6ea2Uwq; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNeek171670; Wed, 3 Apr 2019 17:35:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=b6onsz6VNX7DtbTr5TqzUNVhpOrN6pO3jfRzW4prgsk=; b=m6ea2UwqzBDaAmNQRgMpNsMJ0c4I7NC/WYQD9k10Ju8tN9Bh1BDehTDeYnj8QLawIjuV 4hpLBnsVvZP3329bGZa+hG8vig6xgrCjVOyQ7jGSs90Kr+Dt/MxPIJ77p+hr2lUP1qb2 nSz2yaRDJTsnGM2NanyqxY8wWRS5WZ+4lyPovf64GyOVZwYnqqhKm+PU5vQtz7cPPqMU WKKPyUr4OYpkMAEKhMm6bxwS0Iax+rXMZizx/7oJC5aHCdADQpLu4AvZHxhxOe4s8s88 pihbXFhaMN67EolwQFVqgtHWnBMXaQJYo/bybuJH5xUH+Ejh17OR9jA2osF8nLmayDMk CQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 2rhwydape0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:45 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIRj110867; Wed, 3 Apr 2019 17:35:45 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3030.oracle.com with ESMTP id 2rm8f5fyev-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:45 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x33HZeqd001363; Wed, 3 Apr 2019 17:35:40 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:39 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 05/13] mm: add a user_virt_to_phys symbol Date: Wed, 3 Apr 2019 11:34:06 -0600 Message-Id: <107cc547b68f044651a87ed1aa583e3526b053a6.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen We need someting like this for testing XPFO. Since it's architecture specific, putting it in the test code is slightly awkward, so let's make it an arch-specific symbol and export it for use in LKDTM. CC: linux-arm-kernel@lists.infradead.org CC: x86@kernel.org Signed-off-by: Tycho Andersen Tested-by: Marco Benatto Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- v7: * make user_virt_to_phys a GPL symbol v6: * add a definition of user_virt_to_phys in the !CONFIG_XPFO case arch/x86/mm/xpfo.c | 57 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/xpfo.h | 4 ++++ 2 files changed, 61 insertions(+) diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index 3045bb7e4659..b42513347865 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -121,3 +121,60 @@ inline void xpfo_flush_kernel_tlb(struct page *page, int order) flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); } EXPORT_SYMBOL_GPL(xpfo_flush_kernel_tlb); + +/* Convert a user space virtual address to a physical address. + * Shamelessly copied from slow_virt_to_phys() and lookup_address() in + * arch/x86/mm/pageattr.c + */ +phys_addr_t user_virt_to_phys(unsigned long addr) +{ + phys_addr_t phys_addr; + unsigned long offset; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset(current->mm, addr); + if (pgd_none(*pgd)) + return 0; + + p4d = p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return 0; + + if (p4d_large(*p4d) || !p4d_present(*p4d)) { + phys_addr = (unsigned long)p4d_pfn(*p4d) << PAGE_SHIFT; + offset = addr & ~P4D_MASK; + goto out; + } + + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return 0; + + if (pud_large(*pud) || !pud_present(*pud)) { + phys_addr = (unsigned long)pud_pfn(*pud) << PAGE_SHIFT; + offset = addr & ~PUD_MASK; + goto out; + } + + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return 0; + + if (pmd_large(*pmd) || !pmd_present(*pmd)) { + phys_addr = (unsigned long)pmd_pfn(*pmd) << PAGE_SHIFT; + offset = addr & ~PMD_MASK; + goto out; + } + + pte = pte_offset_kernel(pmd, addr); + phys_addr = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT; + offset = addr & ~PAGE_MASK; + +out: + return (phys_addr_t)(phys_addr | offset); +} +EXPORT_SYMBOL_GPL(user_virt_to_phys); diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index c1d232da7ee0..5d8d06e4b796 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -89,6 +89,8 @@ static inline void xpfo_kunmap(void *kaddr, struct page *page) void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, bool will_map); void xpfo_free_pages(struct page *page, int order); +phys_addr_t user_virt_to_phys(unsigned long addr); + #else /* !CONFIG_XPFO */ static inline void xpfo_init_single_page(struct page *page) { } @@ -102,6 +104,8 @@ static inline void xpfo_free_pages(struct page *page, int order) { } static inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) { } static inline void xpfo_flush_kernel_tlb(struct page *page, int order) { } +static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } + #endif /* CONFIG_XPFO */ #if (!defined(CONFIG_HIGHMEM)) && (!defined(ARCH_HAS_KMAP)) From patchwork Wed Apr 3 17:34:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884299 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB47C1575 for ; Wed, 3 Apr 2019 17:37:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B77127FA8 for ; Wed, 3 Apr 2019 17:37:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78FD5288F3; Wed, 3 Apr 2019 17:37:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9AEBB288F4 for ; Wed, 3 Apr 2019 17:37:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AB456B0271; Wed, 3 Apr 2019 13:37:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1363A6B0272; Wed, 3 Apr 2019 13:37:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F151E6B0274; Wed, 3 Apr 2019 13:37:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id CED4C6B0271 for ; Wed, 3 Apr 2019 13:37:05 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id w7so13137452ybp.13 for ; Wed, 03 Apr 2019 10:37:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=LzrGW3pZOFIG85sCwfxmi85ZudWsgckwNMXygdjGg74=; b=lgrbR0vyk/wNWYEBTMLpJW/lrMBNJhrcCDQANcWkgFHf/csFnKeg0swvqcQBc2g1ho 4/Rq+TPxiAsxgiFqDEQd8b5SZ7T1sIRKAor/jeFVuysgEbq4YC2UfxGkpcXnbUtdq5xj /KG3spFtayxl5qo+hLLL8CwacF5ZIzHyvoDPSZZuJM9+6QN4zjg9p/Ce2QLgRhTnDALe 3BAZVQl0Hq7p+cX47d4jqNxYk7N/FCLUTpVhmYqR5qo2euaXZQ8mn1LqbEiHLFVhJNUR aqSfVcdpR0qMHDs2tpM2l6wZ8rjzo6I5VslJneBxSjsKjCU4dF16v8OENTHHaJM7kJu9 hITg== X-Gm-Message-State: APjAAAW5UfW87K2ubQ0W+SGvQZvCNqC4xOSnoR09zADU+KfXWSBPr8hd YKZitWj0OEbA+Uj45bMXxKkbwFcc3r2PRG8M4jX5r0Dvyvz0XSDeDdrPbVbIYcaKYxDwSlGAvFB EnkA0jsS2bADMbvkMu/jnZ0cHOx4T7cwlC83FkY6DCSXaUJCXIA2rz72OQPVsaYVezg== X-Received: by 2002:a25:2d4e:: with SMTP id s14mr1146730ybe.451.1554313025575; Wed, 03 Apr 2019 10:37:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqybNu4uPmC9krpWR/T2A//H6wjveaLc2tH+jbWDr9wlb5V1vIbdb0jzijXrheF5Wo8GoRPM X-Received: by 2002:a25:2d4e:: with SMTP id s14mr1146648ybe.451.1554313024666; Wed, 03 Apr 2019 10:37:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313024; cv=none; d=google.com; s=arc-20160816; b=i+b4zW53UZdePXDhRi3YOp1xjQZHngT0SRu0u+o/rbQnriQxV6IAqThCU/y0FzqfpL hBlVrFHyQvE4Em3dGDmcTuytzqHumqApdHavOZysqNuDhH491YEwjOwjBMaVc5Z8bqoi F3V2p2bk/RP8aRrsmO5qU4Jj0aWZhWD5uublMwXFYU227403dyxDsOshGaO2TCB0Aeru mSCvOB8+GP8J66jYN1TrKzgJeAga4RgJjgw9OiMbyrkPswbfIHvK27g6vvy1aTTqSJ8/ bPp+PV6evY9IVFMWWGxuzKdMXeCqTVScG5/Un2yQweYnYAlucO0WI9Tt6ur8vzVSwana X2SA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=LzrGW3pZOFIG85sCwfxmi85ZudWsgckwNMXygdjGg74=; b=Wnm7031+rbJhQV3WPQIoldxYshZo5XuTyoLHbSQ6EiWEzchWTj2Muse7xVFsr6YoRq uFG8Ecc+Cl1TGBz0M9Tig8RCuUPdOv16Qqjtz/VhZXsvXM72rsGSbavUqScI2Avph+Oq IWOOho/I5oiJU83mMvJpM4zGWzvjFoHGVcZUrJ9bryNNDR/+upKuFSbQCgtQlyHcZYZC a4ed8uy6oyF9Q/2Kikp90Yd2eXt0nB/XwvnplKy2oBLh5EHGdk2ejrrVnevrCT8a7lAN cC+g1MT5JISt4OMFLVBieJ99aAzfJepKfTlAvXzdX4o90lwuYr7pwdhNG+3M6mAU5bDH fJAQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=yb2s85wr; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id p67si9738832yba.289.2019.04.03.10.37.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:04 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=yb2s85wr; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HO2Zk172124; Wed, 3 Apr 2019 17:35:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=LzrGW3pZOFIG85sCwfxmi85ZudWsgckwNMXygdjGg74=; b=yb2s85wrMl2MIk7Bn095Fz1NdBkZqLuibeAWQyKsKpyvGrfKVWLorsRSq1Thu+qi73my d52f5Yber4UKWoAppEwqAc+DiC9qptpIVd///1NlPfNywtq61IOahpH0kNs9G187AbwV 15mFm/B/N41Zh/mW9fkLqquUQR5mKPr+pTKsbCw5OJAJQ3qtJWA0tPVkoOVUf0vnMf4q NppUVHudJlARSoSc9r5j+yJ2fEZRVlkNw/pI5DbvQpIss9Sjwz8F8ZBPkD3wdUDnbhpY ue6t7/aaEh7DXxhyjZc1/B8q/VhDjk+1fyRKdORpH/tzk1cV7G4AnakvA9fV6blsiqcJ hA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 2rhwydaped-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:50 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HYDoo081801; Wed, 3 Apr 2019 17:35:49 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3030.oracle.com with ESMTP id 2rm8f57yqf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:49 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x33HZiJm001392; Wed, 3 Apr 2019 17:35:44 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:44 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 06/13] lkdtm: Add test for XPFO Date: Wed, 3 Apr 2019 11:34:07 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger This test simply reads from userspace memory via the kernel's linear map. v6: * drop an #ifdef, just let the test fail if XPFO is not supported * add XPFO_SMP test to try and test the case when one CPU does an xpfo unmap of an address, that it can't be used accidentally by other CPUs. Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Tested-by: Marco Benatto [jsteckli@amazon.de: rebased from v4.13 to v4.19] Signed-off-by: Julian Stecklina Tested-by: Khalid Aziz Cc: Khalid Aziz --- drivers/misc/lkdtm/Makefile | 1 + drivers/misc/lkdtm/core.c | 3 + drivers/misc/lkdtm/lkdtm.h | 5 + drivers/misc/lkdtm/xpfo.c | 196 ++++++++++++++++++++++++++++++++++++ 4 files changed, 205 insertions(+) create mode 100644 drivers/misc/lkdtm/xpfo.c diff --git a/drivers/misc/lkdtm/Makefile b/drivers/misc/lkdtm/Makefile index 951c984de61a..97c6b7818cce 100644 --- a/drivers/misc/lkdtm/Makefile +++ b/drivers/misc/lkdtm/Makefile @@ -9,6 +9,7 @@ lkdtm-$(CONFIG_LKDTM) += refcount.o lkdtm-$(CONFIG_LKDTM) += rodata_objcopy.o lkdtm-$(CONFIG_LKDTM) += usercopy.o lkdtm-$(CONFIG_LKDTM) += stackleak.o +lkdtm-$(CONFIG_LKDTM) += xpfo.o KASAN_SANITIZE_stackleak.o := n KCOV_INSTRUMENT_rodata.o := n diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c index 2837dc77478e..25f4ab4ebf50 100644 --- a/drivers/misc/lkdtm/core.c +++ b/drivers/misc/lkdtm/core.c @@ -185,6 +185,9 @@ static const struct crashtype crashtypes[] = { CRASHTYPE(USERCOPY_KERNEL), CRASHTYPE(USERCOPY_KERNEL_DS), CRASHTYPE(STACKLEAK_ERASING), + CRASHTYPE(XPFO_READ_USER), + CRASHTYPE(XPFO_READ_USER_HUGE), + CRASHTYPE(XPFO_SMP), }; diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h index 3c6fd327e166..6b31ff0c7f8f 100644 --- a/drivers/misc/lkdtm/lkdtm.h +++ b/drivers/misc/lkdtm/lkdtm.h @@ -87,4 +87,9 @@ void lkdtm_USERCOPY_KERNEL_DS(void); /* lkdtm_stackleak.c */ void lkdtm_STACKLEAK_ERASING(void); +/* lkdtm_xpfo.c */ +void lkdtm_XPFO_READ_USER(void); +void lkdtm_XPFO_READ_USER_HUGE(void); +void lkdtm_XPFO_SMP(void); + #endif diff --git a/drivers/misc/lkdtm/xpfo.c b/drivers/misc/lkdtm/xpfo.c new file mode 100644 index 000000000000..8876128f0144 --- /dev/null +++ b/drivers/misc/lkdtm/xpfo.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * This is for all the tests related to XPFO (eXclusive Page Frame Ownership). + */ + +#include "lkdtm.h" + +#include +#include +#include +#include +#include + +#include +#include + +#define XPFO_DATA 0xdeadbeef + +static unsigned long do_map(unsigned long flags) +{ + unsigned long user_addr, user_data = XPFO_DATA; + + user_addr = vm_mmap(NULL, 0, PAGE_SIZE, + PROT_READ | PROT_WRITE | PROT_EXEC, + flags, 0); + if (user_addr >= TASK_SIZE) { + pr_warn("Failed to allocate user memory\n"); + return 0; + } + + if (copy_to_user((void __user *)user_addr, &user_data, + sizeof(user_data))) { + pr_warn("copy_to_user failed\n"); + goto free_user; + } + + return user_addr; + +free_user: + vm_munmap(user_addr, PAGE_SIZE); + return 0; +} + +static unsigned long *user_to_kernel(unsigned long user_addr) +{ + phys_addr_t phys_addr; + void *virt_addr; + + phys_addr = user_virt_to_phys(user_addr); + if (!phys_addr) { + pr_warn("Failed to get physical address of user memory\n"); + return NULL; + } + + virt_addr = phys_to_virt(phys_addr); + if (phys_addr != virt_to_phys(virt_addr)) { + pr_warn("Physical address of user memory seems incorrect\n"); + return NULL; + } + + return virt_addr; +} + +static void read_map(unsigned long *virt_addr) +{ + pr_info("Attempting bad read from kernel address %p\n", virt_addr); + if (*(unsigned long *)virt_addr == XPFO_DATA) + pr_err("FAIL: Bad read succeeded?!\n"); + else + pr_err("FAIL: Bad read didn't fail but data is incorrect?!\n"); +} + +static void read_user_with_flags(unsigned long flags) +{ + unsigned long user_addr, *kernel; + + user_addr = do_map(flags); + if (!user_addr) { + pr_err("FAIL: map failed\n"); + return; + } + + kernel = user_to_kernel(user_addr); + if (!kernel) { + pr_err("FAIL: user to kernel conversion failed\n"); + goto free_user; + } + + read_map(kernel); + +free_user: + vm_munmap(user_addr, PAGE_SIZE); +} + +/* Read from userspace via the kernel's linear map. */ +void lkdtm_XPFO_READ_USER(void) +{ + read_user_with_flags(MAP_PRIVATE | MAP_ANONYMOUS); +} + +void lkdtm_XPFO_READ_USER_HUGE(void) +{ + read_user_with_flags(MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB); +} + +struct smp_arg { + unsigned long *virt_addr; + unsigned int cpu; +}; + +static int smp_reader(void *parg) +{ + struct smp_arg *arg = parg; + unsigned long *virt_addr; + + if (arg->cpu != smp_processor_id()) { + pr_err("FAIL: scheduled on wrong CPU?\n"); + return 0; + } + + virt_addr = smp_cond_load_acquire(&arg->virt_addr, VAL != NULL); + read_map(virt_addr); + + return 0; +} + +#ifdef CONFIG_X86 +#define XPFO_SMP_KILLED SIGKILL +#elif CONFIG_ARM64 +#define XPFO_SMP_KILLED SIGSEGV +#else +#error unsupported arch +#endif + +/* The idea here is to read from the kernel's map on a different thread than + * did the mapping (and thus the TLB flushing), to make sure that the page + * faults on other cores too. + */ +void lkdtm_XPFO_SMP(void) +{ + unsigned long user_addr, *virt_addr; + struct task_struct *thread; + int ret; + struct smp_arg arg; + + if (num_online_cpus() < 2) { + pr_err("not enough to do a multi cpu test\n"); + return; + } + + arg.virt_addr = NULL; + arg.cpu = (smp_processor_id() + 1) % num_online_cpus(); + thread = kthread_create(smp_reader, &arg, "lkdtm_xpfo_test"); + if (IS_ERR(thread)) { + pr_err("couldn't create kthread? %ld\n", PTR_ERR(thread)); + return; + } + + kthread_bind(thread, arg.cpu); + get_task_struct(thread); + wake_up_process(thread); + + user_addr = do_map(MAP_PRIVATE | MAP_ANONYMOUS); + if (!user_addr) + goto kill_thread; + + virt_addr = user_to_kernel(user_addr); + if (!virt_addr) { + /* + * let's store something that will fail, so we can unblock the + * thread + */ + smp_store_release(&arg.virt_addr, &arg); + goto free_user; + } + + /* Store to test address to unblock the thread */ + smp_store_release(&arg.virt_addr, virt_addr); + + /* there must be a better way to do this. */ + while (1) { + if (thread->exit_state) + break; + msleep_interruptible(100); + } + +free_user: + if (user_addr) + vm_munmap(user_addr, PAGE_SIZE); + +kill_thread: + ret = kthread_stop(thread); + if (ret != XPFO_SMP_KILLED) + pr_err("FAIL: thread wasn't killed: %d\n", ret); + put_task_struct(thread); +} From patchwork Wed Apr 3 17:34:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C83591575 for ; Wed, 3 Apr 2019 17:37:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD9A927FA8 for ; Wed, 3 Apr 2019 17:37:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A02E2288F4; Wed, 3 Apr 2019 17:37:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D513A27FA8 for ; Wed, 3 Apr 2019 17:37:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D4EA6B026A; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8855B6B026D; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B0CE6B026F; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by kanga.kvack.org (Postfix) with ESMTP id 3A5EF6B026A for ; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) Received: by mail-io1-f70.google.com with SMTP id r21so14208933iod.12 for ; Wed, 03 Apr 2019 10:37:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=MmLdvd3g+VOR33kiTfxTpT9StVgXRV1EaevyJ8pqRKM=; b=Qq6x7JTtsAVp7Jf7y8aFvpCePzZid/y4awHW5KSMQ/WPqqU3o6GybNAAScLPMIKR7D XyGIy22rte3dVMFmJ33JyYFVPdHN+VWFBJoBvl6SYJ6f71H2jhkR+WTVSoWFuN8/Q2KQ D1fx896iYuBa+5NKsjhPcilE4esVrdKHn37oVr95AgEIF4vTljo9Ifjk8VJpE+UCtmVe Z760ULr8N0OvpqaFxgGi1AxTg+CfX+3dcMVj2MBR+w+kFGKZm5Q5hK6KAxO2CmVZszSo ecC4F770g/IlRprJj1gn0IBugNUiaZrIMfOJlbwOlZoz+IjeZSgHYd/prfNwMBWrGEMj O09g== X-Gm-Message-State: APjAAAVzWoxiLEQKrpZLfuCQ32YpodOFQsw1xGBrjkk8YA4n2YIWEeMa 9Y8Vj/wPLb2b67YmmmhX1lycQv+5w2hIpJG34YJr4P4+aqH6CThfe9m79GCMnpzo83EZJhJ0COx 2xDuvlXwZtB6XNmpD0xEFOxv3E56zU5cyglHAzOqgQQxDVgxpfcaA3Q0qs9rsjmM4rw== X-Received: by 2002:a6b:e418:: with SMTP id u24mr1149492iog.128.1554313021981; Wed, 03 Apr 2019 10:37:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqysZdWXn47WxCEOSz/Or8sypnIgq07gKUhpAZwnYtDQM83mFuhMgenv6+FcpzhGZ0AtcP/B X-Received: by 2002:a6b:e418:: with SMTP id u24mr1149419iog.128.1554313021060; Wed, 03 Apr 2019 10:37:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313021; cv=none; d=google.com; s=arc-20160816; b=Ec3K4PM5mQmYHvjLUQNKl3Enzai3yrHXEaZT/H03c+U+22w+nDNq2B//03PI1QLLle J7YKwD6/nt4I7UlSJC7Vk6AZNCS3mpojDMiiPENhMqLqd3CfsHOgiYcVLh5ABkmTnrEQ ayWVO9aiXoj1pPOgX/7xau/+b1fGzjx2KIV5EAtL4JaTugYPEx1OgVjrpzJ5vO4kmP/C 7FE4WGyi8YPjXbrpTKlHADjY2Le034oS6oLFIYEK4YX7bJSjyClPufmrGMwI5PDDbx6R xdxFLLEwDoxGpFKLgXAguZjSaEe3FgruxH3a7uxbN2d3HnhXVF3xTvi33mYxQWtdsKUE LXMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=MmLdvd3g+VOR33kiTfxTpT9StVgXRV1EaevyJ8pqRKM=; b=rVvlr5Is7HC8bNekCpq8ReLadi3xPqWEvT5nrO26b6OJkJPfXylUUFKvdLmzyUIKbX 6DnL8s4eD0kjIkQONaIKzSbFsJax9gkFiUfeepb1BVVR6KaAv2emgp7vT+5eHQduSMM0 xMvJe9eXh+y5Sl+OYNpG3x7t8Ymv/5cDHa6h638ip59Qmf9fitDXM9n7ZS5gP9lPDZLX nOX8nQgE0CJ2HaSZS7TusJnS6T7/tUPPA9CvjmD1jcdP9xuxFIgYxLXqduZjPG6W0zKR uBTp4Urz+DE53CbAQmWwaBJYpPGxH06VYVKcvaYnmyyTZNG1VCt4rVS/Kdr8MqXPlo1r +CtA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=A1IlnQsb; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id y4si9365927jan.121.2019.04.03.10.37.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:01 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=A1IlnQsb; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNd4U171655; Wed, 3 Apr 2019 17:35:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=MmLdvd3g+VOR33kiTfxTpT9StVgXRV1EaevyJ8pqRKM=; b=A1IlnQsbVmk3r7/UQc6MdFG/cRK9jKFVE4Bgxh4JAosAGn0Rq0w+/bzoMtqIp222k2xa 07XSD9hIlM0Pmg9rsVAJJq/JE0OWmaeYHCdw3XNVIFx2cceW+kZ8IikrRB1SHKW2yCAd RW7TTYytsps+JD6RjMLkebFQ2smqWV6WN4g49EkBz096AjpZj2YvRX6bSKF1UwNS5zF7 Q3MZJo5yW3pIoe7JJGihO4P67rz/XassZge0JfcPNF+SQLCIunOcUZ2pVOrdZ0uDG5cH 8QarNSLrk8AHBL2zD3np5bgQ18hftrVKMCzSWbdsPgVwCZnWSs3kVGoRG9L1KDDtlPim Jg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 2rhwydapeu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:54 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZLl2152602; Wed, 3 Apr 2019 17:35:54 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserp3020.oracle.com with ESMTP id 2rm9mj6gta-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:54 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZnSR001401; Wed, 3 Apr 2019 17:35:49 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:48 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 07/13] arm64/mm: Add support for XPFO Date: Wed, 3 Apr 2019 11:34:08 -0600 Message-Id: <50011e8d1ae252c6b70806f8d6a2d6dd79cb1a8c.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger Enable support for eXclusive Page Frame Ownership (XPFO) for arm64 and provide a hook for updating a single kernel page table entry (which is required by the generic XPFO code). XPFO doesn't support section/contiguous mappings yet, so let's disable it if XPFO is turned on. Thanks to Laura Abbot for the simplification from v5, and Mark Rutland for pointing out we need NO_CONT_MAPPINGS too. CC: linux-arm-kernel@lists.infradead.org Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- .../admin-guide/kernel-parameters.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/mm/Makefile | 2 + arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/xpfo.c | 66 +++++++++++++++++++ 5 files changed, 71 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/mm/xpfo.c diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index e65e3bc1efe0..9fcf8c83031a 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2997,7 +2997,7 @@ nox2apic [X86-64,APIC] Do not enable x2APIC mode. - noxpfo [XPFO,X86-64] Disable eXclusive Page Frame + noxpfo [XPFO,X86-64,ARM64] Disable eXclusive Page Frame Ownership (XPFO) when CONFIG_XPFO is on. Physical pages mapped into user applications will also be mapped in the kernel's address space as if diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a4168d366127..9a8d8e649cf8 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -174,6 +174,7 @@ config ARM64 select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK + select ARCH_SUPPORTS_XPFO help ARM 64-bit (AArch64) Linux support. diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 849c1df3d214..cca3808d9776 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -12,3 +12,5 @@ KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o KASAN_SANITIZE_kasan_init.o := n + +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa52ac67..1673f7443d62 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -453,7 +453,7 @@ static void __init map_mem(pgd_t *pgdp) struct memblock_region *reg; int flags = 0; - if (rodata_full || debug_pagealloc_enabled()) + if (rodata_full || debug_pagealloc_enabled() || xpfo_enabled()) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; /* diff --git a/arch/arm64/mm/xpfo.c b/arch/arm64/mm/xpfo.c new file mode 100644 index 000000000000..7866c5acfffb --- /dev/null +++ b/arch/arm64/mm/xpfo.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include +#include + +#include + +/* + * Lookup the page table entry for a virtual address and return a pointer to + * the entry. Based on x86 tree. + */ +static pte_t *lookup_address(unsigned long addr) +{ + pgd_t *pgd; + pud_t *pud; + pmd_t *pmd; + + pgd = pgd_offset_k(addr); + if (pgd_none(*pgd)) + return NULL; + + pud = pud_offset(pgd, addr); + if (pud_none(*pud)) + return NULL; + + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return NULL; + + return pte_offset_kernel(pmd, addr); +} + +/* Update a single kernel page table entry */ +inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) +{ + pte_t *pte = lookup_address((unsigned long)kaddr); + + if (unlikely(!pte)) { + WARN(1, "xpfo: invalid address %p\n", kaddr); + return; + } + + set_pte(pte, pfn_pte(page_to_pfn(page), prot)); +} +EXPORT_SYMBOL_GPL(set_kpte); + +inline void xpfo_flush_kernel_tlb(struct page *page, int order) +{ + unsigned long kaddr = (unsigned long)page_address(page); + unsigned long size = PAGE_SIZE; + + flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); +} +EXPORT_SYMBOL_GPL(xpfo_flush_kernel_tlb); From patchwork Wed Apr 3 17:34:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884301 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DFE71575 for ; Wed, 3 Apr 2019 17:37:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 629E927FA8 for ; Wed, 3 Apr 2019 17:37:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 54403288F4; Wed, 3 Apr 2019 17:37:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B75D227FA8 for ; Wed, 3 Apr 2019 17:37:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A11856B0272; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8D39C6B0274; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79AEE6B0275; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 549736B0272 for ; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id v123so12740086ywf.16 for ; Wed, 03 Apr 2019 10:37:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=yCxbmCAk84npxguylmi/HxVQRuHK/tYnw0TSzCvTIZY=; b=oTzkhXA4wIFoLOl0WAa854VZ2Ojl9JA/eXr4mH1CdmhmknE0TJWFBk36X4mHiaCywt w5Uxjrxuq2wYyfCgXMDxQS6lyGZbcsbzCyq4I4ogp89w3ytw5DoT+0Lah36hIYqkrWAt 7FK7pM/e+0AveYtM3TjsdD2tD/9zzg0wZMzftynfmSYQx0fJuxa0uzcFgfH/dph30dpV 4rimgKNFrCDqcrU2V9PdKwOWjXAnFb49yWtfzlSsscSyMh67gdfBh5d2fPWeBd57LA+3 ns1UQIe00uJj/AqGRLuKw4wgP3hiIiMB+uIlSyU/JZkbkcw37TDV3g+Kq8eBpFzslpZw wohg== X-Gm-Message-State: APjAAAUxthvfePhnplrYKc+un6Pu/1p+R6pbt6IzIYAJfA+L2kStJWt+ UmgfrUDNrSYLPGaxnVkaehc9ZqS6L97AAy4sVSKN/q3M5NtaltTszI2Q8QYl2+5lN0mUzUiK/Dw 5e/mE/3yBTygRx8HB8e9ell2pRyzL/ljMU+uJ7fVtEUamOUM07Svmh1rCEu1V8T9fkw== X-Received: by 2002:a5b:749:: with SMTP id s9mr1096043ybq.447.1554313028106; Wed, 03 Apr 2019 10:37:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqxNa/V9tIbvDxIHJJHY2r/ImVa526f11RJKEg9XJxHBp8tD42q9h2Y+Sm8SdTEVLy6fh0Z+ X-Received: by 2002:a5b:749:: with SMTP id s9mr1095986ybq.447.1554313027480; Wed, 03 Apr 2019 10:37:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313027; cv=none; d=google.com; s=arc-20160816; b=A5ducJaG33101aAVBOVLXx5BAEUzgjAgsBTRnjglw92Xl+e/JErKLbe67LWF8Tc4iW ssk0GavHEngt6d1lgkIBdyyAKT5wb1qWkwKlKl804yFe4ZckwmooePxVmZ/i42wGRhcI LEJJ7m6CEIbWS5z3sBK3zZYhW1ZWiDmyRESgW1UydzekDRwB040T+w19nmDUd7k8i+hf PBwNwEWwBbEBmUuA3WNwAhIkASBmAAgtT8s0WRbeIwy0Ppcntp5jqjma46o8kAg1o6ct RImW/19IwXFwatgztKu1CIgI9Ox1kNxxsaWbY+Frq1l/m8wcU5q7o4mgz5Lkw6Z73B7x 9H2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=yCxbmCAk84npxguylmi/HxVQRuHK/tYnw0TSzCvTIZY=; b=DG8DwhRhB4WEEjxPrxLRaKvGXO7l6grPjeyUFKBvvSz4dJ5B53hLfJqn8Chdx1l+bm /Hq6pqpGwjdySYsmms2JAYwKXF1h8Qla6/utYW9wKqQCj7+RnglWCQyRNYXp5xup3TC2 i4R4Gu7xsIPhZhzSm1aVKxhFy5N+6PSTYzWJghqqi199jHPTs2RjMFUlWd5LeVjLM9gL +yQQb2ogFSFcMrvEMdLTw68rzKWPWg/EiZJLZzdl4SV0400p8WTVvPRMwCnO7mgmrOG3 O5oNXxyQNvkheoZTKIsPavHcrH0GFHRfxCLgrJHFpoTIkIi9ReLYHsSvutZ0p+440WcH bXMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=wc9JzFG9; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id x6si10251855ybh.421.2019.04.03.10.37.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:07 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=wc9JzFG9; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNrjN175419; Wed, 3 Apr 2019 17:35:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=yCxbmCAk84npxguylmi/HxVQRuHK/tYnw0TSzCvTIZY=; b=wc9JzFG90PcUo+AelC4KcKG0r3iYRO2KFvdJ4lxbZLAoAhrjwfOVTW4YISk+OFJrMJSn xCnuRpQAaoF5KYp30BDf3TJ1n5/4tQimwJRke+2FDLR3tzhS4CjfvjCJLu7BPeOV/jhE gKNTeenIeXXzPUxNFzKXOgdEHykEmIxny4FS+gcQOJFAjpb1MTF2x4bFF1KrE/GygqTZ 4fgtCo38B5aZz2/GDLPcQJHkb3zcSavurvW/+llRfTh+QZcSVxzr8xTYj1kU5BwlfG4T UWQa1LTfAn7tqRYUaIijkpGNwDoo9gPVXXl8vdAt7TrPsE3s5+A9xrFm4V5/VPoVXVXo HA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2120.oracle.com with ESMTP id 2rj13qae82-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:58 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HYDKs081810; Wed, 3 Apr 2019 17:35:58 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3030.oracle.com with ESMTP id 2rm8f57ysj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:35:58 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x33HZru2001521; Wed, 3 Apr 2019 17:35:53 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:53 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 08/13] swiotlb: Map the buffer if it was unmapped by XPFO Date: Wed, 3 Apr 2019 11:34:09 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger XPFO can unmap a bounce buffer. Check for this and map it back in if needed. Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Khalid Aziz Cc: Khalid Aziz Reviewed-by: Konrad Rzeszutek Wilk --- v9: * Added a generic check for whether a page is mapped or not (suggested by Chris Hellwig) v6: * guard against lookup_xpfo() returning NULL include/linux/highmem.h | 7 +++++++ kernel/dma/swiotlb.c | 3 ++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 59a1a5fa598d..cf21f023dff4 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -77,6 +77,13 @@ static inline struct page *kmap_to_page(void *addr) } static inline unsigned long totalhigh_pages(void) { return 0UL; } +static inline bool page_is_unmapped(struct page *page) +{ + if (PageHighMem(page) || PageXpfoUnmapped(page)) + return true; + else + return false; +} #endif /* CONFIG_HIGHMEM */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1fb6fd68b9c7..90a1a3709b55 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -392,8 +392,9 @@ static void swiotlb_bounce(phys_addr_t orig_addr, phys_addr_t tlb_addr, { unsigned long pfn = PFN_DOWN(orig_addr); unsigned char *vaddr = phys_to_virt(tlb_addr); + struct page *page = pfn_to_page(pfn); - if (PageHighMem(pfn_to_page(pfn))) { + if (page_is_unmapped(page)) { /* The buffer does not have a mapping. Map it in and copy */ unsigned int offset = orig_addr & ~PAGE_MASK; char *buffer; From patchwork Wed Apr 3 17:34:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 59E1417E0 for ; Wed, 3 Apr 2019 17:37:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E83927FA8 for ; Wed, 3 Apr 2019 17:37:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2E432288F4; Wed, 3 Apr 2019 17:37:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 824CE27FA8 for ; Wed, 3 Apr 2019 17:37:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30A356B026D; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2BEC96B026F; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E7036B0271; Wed, 3 Apr 2019 13:37:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id D94C86B026D for ; Wed, 3 Apr 2019 13:37:03 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id h125so13149481ybh.4 for ; Wed, 03 Apr 2019 10:37:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=ynty9flFqcPPDSeTkJifnT4Yutdn8/TkRysaHE6m4cE=; b=aqfAAr6Nx2l82KLyDdsXZjRxhnxK1j0wnB/GcR4v1hnYLanKXnPNwMg2It2h0Ey7L8 LsPQgL31ivQeGpAUZeG2D6El9hnBsnBpaA+5a2RtxY7668K+SEzuvD/BkrTTAVTaJ5BZ EBMiURqpM03t0Trh2X/kVkR9SZZPVnNqJ1vivOUNJoEVuMAZanm4r0+Qh1XMXciWl24n ND+gVBREhtH/Lz/a9YJCCmvr3NlOqvo2erfzfKZm8t61msENT2C0cS7WbTMbs5OGCOwR IX0P1OWtw0zEyJVn4mDATS9uLILYeHxMrey9UUBh98sRhhOnGVEqK6sFturiVQa2+y/S coGg== X-Gm-Message-State: APjAAAVoj0PZEgtPYS90GyWKZCqN0prAk5rk153LQDIEWrmhECSr2FYj TaHPwwSCwhMWVSxJOWDyLhpm5ZIlrMX8vscen9VGaCIWLj8KEP0iChuGpEjKFTjxbRDQb0ruWl0 K6rZFRsjilIEkS6U3Kzndd9ZW4u0IbDZCuUnnjweEmfCmBHfjz/OkfDathA0feGchbQ== X-Received: by 2002:a25:e744:: with SMTP id e65mr1161876ybh.432.1554313023616; Wed, 03 Apr 2019 10:37:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqxlcHAHza+jE7BrK4+uol142k6ObzsQ/M+E0C5FO8+9RJbCd9G6hkhkBFZ0OQhQ/XERqSzq X-Received: by 2002:a25:e744:: with SMTP id e65mr1161808ybh.432.1554313022846; Wed, 03 Apr 2019 10:37:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313022; cv=none; d=google.com; s=arc-20160816; b=tV+kvJGLJ1m8mDV/iDuPvu5ugJBAjDDpfJO93Vb762buCMG5WixAzEA9jHFiNFoum2 J+r3gREsHLEtQXhDJm+vL8zB0oSzDDn7JfKeKaQuuWKRxOei9hOdj0h+kITlliph8DU4 k6+7QMFe4U3Dy6MJIuTY+eCsSdfZi/hzJuScHWNSOYCqZQus5gyUu8xOcqldogSdIlDV OlYQqzBJ0hLydGYCXZmo9xXUyJBZarqrQbneM4IBaV7ubSa28nlzOl7gT/OnRGbL0xX2 0mtpKwQbY7ilZe5q46kOz+EYdMP9pwfFPcO11A8w6lKMQu4kpHIvGcNWaFED/VuDNpLY rIBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=ynty9flFqcPPDSeTkJifnT4Yutdn8/TkRysaHE6m4cE=; b=VlUz40cv9NP1r2xs7InagJhLFperQM+e3QYaBMnSeZ+svS4f5Zof7bv3PKPvb/I6Om WD0uPWo5/AlRzbUXwBf+3kBim2Dw8G8ajhnRb2vnusgND8sDEoADh1PltF5CUdgetkDz C2kn0Z95r1e1Aot11MkU+JOCk4r9TfZz1+rbcCFErPaBBs2BPtE12DG8s4h275Peu25S 2PLmpBZySEjIQNtsewPpeyAoq5KRgD0LnFMTrOrvcEn/5wxK83/2qCvZ3KJr0Y/R0D01 +w0FcTBSsEBOI/+HfWtKh9491xdWWeH47co+4C9PmfuUhMbZtCJCUj8osVIdxFHSDxTO NH/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=BaxcCITX; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id b2si10346722ywh.384.2019.04.03.10.37.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:02 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=BaxcCITX; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNdqq165965; Wed, 3 Apr 2019 17:36:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=ynty9flFqcPPDSeTkJifnT4Yutdn8/TkRysaHE6m4cE=; b=BaxcCITX+BZ4+lS5aQJjRB3VCfd26NfZpmG7792Zbur5ODmmm50Sno8QeknDLKaCAir9 Da0W1vB05l+2ASj/y+rE4hgV3tyBaxqqb91JOILD/MwVMrBRtVmhbFSrA9iMgDXoNJWS dRYrw4FjheDfi1giGH7j+C9Mh6xLGdSnbjK1Qg2rrxQNlCBwjMZjabA2o0bhjwvFsgvm ZGyPPSObB3PcM3+v3y/fDDKNwO6++dz2FVngxt1vH1XDVUGHsq570xDSyzQsC9TWwkKO 1h+WkiWSGvDnqnffj7boaGX3CkOtpbIGYXFlvMpx+ygTyz3/w/5rP9ib47WXL8Nh+xu6 xA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 2rhyvtahn9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:03 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIEn110862; Wed, 3 Apr 2019 17:36:03 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserp3030.oracle.com with ESMTP id 2rm8f5fyjx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:02 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HZwKI001502; Wed, 3 Apr 2019 17:35:58 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:35:57 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 09/13] xpfo: add primitives for mapping underlying memory Date: Wed, 3 Apr 2019 11:34:10 -0600 Message-Id: <9f7930eca60750aaf5381efbbcb45f6da192874f.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen In some cases (on arm64 DMA and data cache flushes) we may have unmapped the underlying pages needed for something via XPFO. Here are some primitives useful for ensuring the underlying memory is mapped/unmapped in the face of xpfo. Signed-off-by: Tycho Andersen Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- include/linux/xpfo.h | 21 +++++++++++++++++++++ mm/xpfo.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 5d8d06e4b796..2318c7eb5fb7 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -91,6 +91,15 @@ void xpfo_free_pages(struct page *page, int order); phys_addr_t user_virt_to_phys(unsigned long addr); +#define XPFO_NUM_PAGES(addr, size) \ + (PFN_UP((unsigned long) (addr) + (size)) - \ + PFN_DOWN((unsigned long) (addr))) + +void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len); +void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, + size_t mapping_len); + #else /* !CONFIG_XPFO */ static inline void xpfo_init_single_page(struct page *page) { } @@ -106,6 +115,18 @@ static inline void xpfo_flush_kernel_tlb(struct page *page, int order) { } static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } +#define XPFO_NUM_PAGES(addr, size) 0 + +static inline void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ +} + +static inline void xpfo_temp_unmap(const void *addr, size_t size, + void **mapping, size_t mapping_len) +{ +} + #endif /* CONFIG_XPFO */ #if (!defined(CONFIG_HIGHMEM)) && (!defined(ARCH_HAS_KMAP)) diff --git a/mm/xpfo.c b/mm/xpfo.c index b74fee0479e7..974f1b70ccd9 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -14,6 +14,7 @@ * the Free Software Foundation. */ +#include #include #include #include @@ -104,3 +105,32 @@ void xpfo_free_pages(struct page *page, int order) } } } + +void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ + struct page *page = virt_to_page(addr); + int i, num_pages = mapping_len / sizeof(mapping[0]); + + memset(mapping, 0, mapping_len); + + for (i = 0; i < num_pages; i++) { + if (page_to_virt(page + i) >= addr + size) + break; + + if (PageXpfoUnmapped(page + i)) + mapping[i] = kmap_atomic(page + i); + } +} +EXPORT_SYMBOL(xpfo_temp_map); + +void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ + int i, num_pages = mapping_len / sizeof(mapping[0]); + + for (i = 0; i < num_pages; i++) + if (mapping[i]) + kunmap_atomic(mapping[i]); +} +EXPORT_SYMBOL(xpfo_temp_unmap); From patchwork Wed Apr 3 17:34:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884293 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C6D51575 for ; Wed, 3 Apr 2019 17:37:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6323A27FA8 for ; Wed, 3 Apr 2019 17:37:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55061288F4; Wed, 3 Apr 2019 17:37:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C98DB27FA8 for ; Wed, 3 Apr 2019 17:37:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1AEA6B026B; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A84546B0271; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 777346B026B; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 4F41E6B026D for ; Wed, 3 Apr 2019 13:37:02 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id 186so14321820iox.15 for ; Wed, 03 Apr 2019 10:37:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=rvo2MdeHWW3FffgXBOnUmvIQSFmvjV1SQw64yMoM6qw=; b=kV6O50HxrcfimKoveoNTv7be8f02pRdMQXFoL6YkRTu3+DdV/jETl03wq/yv1WNUTX d3f7QL8B560X+y6VykgowXSBna5UgbIGex8FLvVTU53o0G07z822v5zpLfsE2KQ39qXz wPcHlmuvjl78YvDEqTtC/RCbNeGy78IaZcRla9bN+b6xkSM3fHPTMek+1LI0J74vgDNw VsmaRZ59Pb8OqCPDK16wxOzJxXuX4aLJn6vLHJV5du/7UuFnincV22kLQg5ry3rI4esZ t4UMa074X8+/fU4wmvJxICqCkATv2TfVcMxCuPbadZVaNJk7BLKeYcvXUinQzy+Z8YVb vAfA== X-Gm-Message-State: APjAAAW0NxiZJBs68tLOorgB6fb1GsBysHuJwrZNshBSnKQH60uAZYPr 98IzNE6EJfwVtYrUoQieo89K67Sv0qdalFaV0r+07P+bDQFoeOvnPS550laOTHsv15QjJgVtJWw xW4d1/51S3oRTpGCZCTtrOBmvzSOweMQOVuo2XWV+ysjvc5or2c00iSXJ0io+FKEX4g== X-Received: by 2002:a24:57c1:: with SMTP id u184mr1258911ita.38.1554313022104; Wed, 03 Apr 2019 10:37:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqy28EjyvR4pH2LY5MnqF9K3cOMAs6rJxYuGBLWLefZ3FZDNPEfBjbs+WOoYaZ6dXyxrtAmf X-Received: by 2002:a24:57c1:: with SMTP id u184mr1258856ita.38.1554313021210; Wed, 03 Apr 2019 10:37:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313021; cv=none; d=google.com; s=arc-20160816; b=LI8Wra2Ph/hG+gKjXUHegLLVHJgmmJFwOUa+1EBB4q+Z4jIq1fQpkwZRezyoV3cCib geSNNSYk/3lexfeH3hqSWQxvUOyQrzkuL4IVz08tZGlPXhNIcoetT56QwUmAr3o88oH2 EX62+eI9LdHXypohZClGH3PzMqx+1BBZwUS9xhw8iczvbKgB3Wx/RavGvmBIfPTRpwEf kynfFJ3CbB/mW3k7pEiFuzA+0smHBZGfrF710tmSee+Xtq3rh9yqp+xQpQx1OetxFCyY d+kcRGqb9N+NKl0fFvvyHOnc09mtwXLcldU4aSaLpji+6T0EJfp3U+E8Thsuv9pP5Fq2 lCxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=rvo2MdeHWW3FffgXBOnUmvIQSFmvjV1SQw64yMoM6qw=; b=oqwMGiZB1fXLZ+NXntRJImpOeltcrYu1qoI+/tRChDniZOGPKpwT9orE7ufrR57h7x v17ZMOZ62nBDoU5q5Rxc6drJChfzR6jfQi/oUd9kOVpLJbWAHc1zP9pMDznGchyeTbcA 1tX0A6t6gn1V0j502Xzjaxye9EZbImcT7ApLNrnfWvcO8JeRPI0QF8MUcMUidLVv6mGK zB0zcpdtF/5oNUq0mY+weeIkzPYsQ8s+URXf1qkSfoJ5d1GzPJuLfUnOXYq/EanyeSiK 3w2mqWFV8DjEJajKVfXHafbA4sxV86RSVeZwaSwPkOj7iLA+gV0CmEd6OlYpIe4zE1+0 wPtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=npNl9oJA; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id a192si8597803ita.13.2019.04.03.10.37.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:01 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=npNl9oJA; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNhR0175372; Wed, 3 Apr 2019 17:36:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=rvo2MdeHWW3FffgXBOnUmvIQSFmvjV1SQw64yMoM6qw=; b=npNl9oJAhSwyoIUJBpUdDPF4JZ2CEYHDa3hUF0rsMUd/TEUZukdAVOkSbIAOjqZ6DnhF 4PeMvlNZBGu3MlDHO96FNz7ZYY4KGvv7n5GotA4/y9ON0ExvPl/M9xWTwTgrYJlmrjPt KrY2ATZ/2p1EUBzuS+cPJ0H0Go+Gw49Ol8sGEGecBg6+4vpnuJRpl2iI+pGvq2WSHF5f cNT8uKqrqWkIzeQB8soGylpULikvh8aYZHPSbDYfFRqgwsMoWBi2heFAa2HwOw4ulleL bH5dtKuvZHXYPcnXA26kaww/oD554BWvyKYQjs0uON2qSiyZ7kHrJavD0EnUKzK+eaOS +A== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 2rj13qae91-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:08 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIaq110901; Wed, 3 Apr 2019 17:36:07 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3030.oracle.com with ESMTP id 2rm8f5fym6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:07 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x33Ha2pB001570; Wed, 3 Apr 2019 17:36:02 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:36:02 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org Subject: [RFC PATCH v9 10/13] arm64/mm, xpfo: temporarily map dcache regions Date: Wed, 3 Apr 2019 11:34:11 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=677 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=687 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger If the page is unmapped by XPFO, a data cache flush results in a fatal page fault, so let's temporarily map the region, flush the cache, and then unmap it. CC: linux-arm-kernel@lists.infradead.org Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen --- v6: actually flush in the face of xpfo, and temporarily map the underlying memory so it can be flushed correctly arch/arm64/mm/flush.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index 5c9073bace83..114e8bc5a3dc 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -28,9 +29,15 @@ void sync_icache_aliases(void *kaddr, unsigned long len) { unsigned long addr = (unsigned long)kaddr; + unsigned long num_pages = XPFO_NUM_PAGES(addr, len); + void *mapping[num_pages]; if (icache_is_aliasing()) { + xpfo_temp_map(kaddr, len, mapping, + sizeof(mapping[0]) * num_pages); __clean_dcache_area_pou(kaddr, len); + xpfo_temp_unmap(kaddr, len, mapping, + sizeof(mapping[0]) * num_pages); __flush_icache_all(); } else { /* From patchwork Wed Apr 3 17:34:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53F5A17E0 for ; Wed, 3 Apr 2019 17:37:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41AA227FA8 for ; Wed, 3 Apr 2019 17:37:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 34566288F4; Wed, 3 Apr 2019 17:37:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D43F27FA8 for ; Wed, 3 Apr 2019 17:37:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A6176B0276; Wed, 3 Apr 2019 13:37:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 505FA6B0277; Wed, 3 Apr 2019 13:37:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 309F96B0278; Wed, 3 Apr 2019 13:37:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-it1-f198.google.com (mail-it1-f198.google.com [209.85.166.198]) by kanga.kvack.org (Postfix) with ESMTP id 0E9CA6B0276 for ; Wed, 3 Apr 2019 13:37:16 -0400 (EDT) Received: by mail-it1-f198.google.com with SMTP id l202so6883713ita.1 for ; Wed, 03 Apr 2019 10:37:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=RQ6cfPE1oUdZhXOBb8lEKxLwdCK2MCPnapeo9zEXamM=; b=aJQsEpOfkVY+A/hdppfB0DGL7nh4JKqXbWWaToknOoEB4vKXD2FIvGZU5+A2aF+WKI lEOkb2D8jXaaNqgYDzCTRBeG4MAGqcHt3OoDb3CkzzdzB1bFb4ouZjw7/s+yT8kx22u0 wZS7mXXOHj4ZoEnUo3S7/H5O3KkZuAFKbXXWlRMB5im1sFbhTQyIal3XZIEKJMo+dJew aI7XVVLmsUllrN042RziVNJaHzG3KEYQuwLFbubO13pI4k2jlgxTHLoiCtzx2PXyK+i7 S2iUTAcWn4s6S2ayXzAe0QcKkZwLXYZV22QF/7beNIcxgc/nxNV6AiC/W3ELVcVtpghv s3ZA== X-Gm-Message-State: APjAAAU0ysgVvD0C/3p0VLdyPH7eSv+PYrJi8/C9BN4UsiaZKIkMkXgH ycWY1QI+yt+I4ImvBwumLL8HXyvCwNEZChYNDbRjgCAjmp/mARMbGI6J9+4iMJNWTpbWuBgtZGb 7NSLDezBsdyaBvR6WlF3waJSBoYiO5zKLsgq4NzS3hCWZB/ZAd+97fTeD65B8Qdd0Qw== X-Received: by 2002:a24:1f44:: with SMTP id d65mr1192066itd.65.1554313035715; Wed, 03 Apr 2019 10:37:15 -0700 (PDT) X-Google-Smtp-Source: APXvYqw3whXah+n+2st+DgWOal1BDt2OlpI3DyaFa4P1Tz0pqfQZWivSTsrN4PwNb6su/Uvybvaf X-Received: by 2002:a24:1f44:: with SMTP id d65mr1191992itd.65.1554313034778; Wed, 03 Apr 2019 10:37:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313034; cv=none; d=google.com; s=arc-20160816; b=XT+IO/OCkZ6Y5YG5aqOsgaESNLtafQdahEDTLf99PHYGI6XCxI5qLKldFMulc50XQh Jow3NGKcnjhByCuc1IoAG1vRGVSnrthaBm7ToqGnou8ivjz16Iv0W/tkgbCeZhvCYPfh Hnwet+UQ5/GM6ZJWKTVh99idf+OL1EXTV/0FXDuaRl4nk4xhirp3ud2TuLNUFkeKTPKt eQHIygUPrynwgWj/NReLl6kQoxogUqqAWazRKWGKrPkgTWuXx5CO7w9eFe/6iFh/CkQw urOSNNXATAa+Qus5YQuHIrSstDF8saJrl1UWUUIxzcdZP3YrwRS9/P3k9U16JjAgibeX lJCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=RQ6cfPE1oUdZhXOBb8lEKxLwdCK2MCPnapeo9zEXamM=; b=WpzRp7q9F9wbdy5y0mum/sqsPx/rGk5abDl8gtdJzXlWAYgpLYaAxDHfNsuQZ2RW69 hWq8A550F3fQpsVEUgbzpIMmEfrpdb4U7Z9PnSp/N4k7PjmdTuIuLRFiY07GrN5CXRn/ 2GpS8NeOjfJVxAcz2CGd3px1wVBIPY1CC6sAhZlE1fsjV/8py5ANd8rOrWEMoeAQDfAG W4CYUFHB6gbGtiXgPjWXcd/trPcoMPB//jpizqmmjJNhDYiKQZYr1Bo0CkhW3TJsBPiJ o+GXz5cG4zhqRWyFZFS6pO7BL+Onsos5pby6VLABf376O1DtWh/5XcJk1qu2psU133eA U3VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=FCi26eH2; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id x30si3012408jap.57.2019.04.03.10.37.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:14 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=FCi26eH2; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HO4Be175776; Wed, 3 Apr 2019 17:36:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=RQ6cfPE1oUdZhXOBb8lEKxLwdCK2MCPnapeo9zEXamM=; b=FCi26eH2yuIj/duNDQmSbddnHqKIgNwk5FitB/zF8l+pOrUfoRX+prFRbpqHu9Gr/3ss /IeG8UwqR1hfVmbLtEDQOcWMydXrIwj2ToNVug5qC2TokZppKngyobnDY6VBcytWlju/ oghOni+CK13eoXN4LvOyT8p5rYzAopR3y2MzmG0ZokbUJRieoruWDXzmgm834RPzDZ0Z V6YURMu87c3OumnQbKDenDg00eEPn5Z2IV7lnjC3ZZb4z+7P5Ga4/1YizcEZKAsQL2Jq rlwWi8L/5Lw5w+WDr7wK1Pjj1zc6Lhid+jNfzqvpM10qAhdsz56CtubdeBhgq1tKGrJM Zg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 2rj13qae9s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:14 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIGU110875; Wed, 3 Apr 2019 17:36:13 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserp3030.oracle.com with ESMTP id 2rm8f5fynf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:13 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33Ha78j001568; Wed, 3 Apr 2019 17:36:07 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:36:06 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khalid.aziz@oracle.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz , kernel-hardening@lists.openwall.com, "Vasileios P . Kemerlis" , Juerg Haefliger , David Woodhouse Subject: [RFC PATCH v9 11/13] xpfo, mm: optimize spinlock usage in xpfo_kunmap Date: Wed, 3 Apr 2019 11:34:12 -0600 Message-Id: <5bab13e12d4215112ad2180106cc6bb9b513754a.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina Only the xpfo_kunmap call that needs to actually unmap the page needs to be serialized. We need to be careful to handle the case, where after the atomic decrement of the mapcount, a xpfo_kmap increased the mapcount again. In this case, we can safely skip modifying the page table. Model-checked with up to 4 concurrent callers with Spin. Signed-off-by: Julian Stecklina Signed-off-by: Khalid Aziz Cc: Khalid Aziz Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse --- include/linux/xpfo.h | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 2318c7eb5fb7..37e7f52fa6ce 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -61,6 +61,7 @@ static inline void xpfo_kmap(void *kaddr, struct page *page) static inline void xpfo_kunmap(void *kaddr, struct page *page) { unsigned long flags; + bool flush_tlb = false; if (!static_branch_unlikely(&xpfo_inited)) return; @@ -72,18 +73,23 @@ static inline void xpfo_kunmap(void *kaddr, struct page *page) * The page is to be allocated back to user space, so unmap it from * the kernel, flush the TLB and tag it as a user page. */ - spin_lock_irqsave(&page->xpfo_lock, flags); - if (atomic_dec_return(&page->xpfo_mapcount) == 0) { -#ifdef CONFIG_XPFO_DEBUG - WARN_ON(PageXpfoUnmapped(page)); -#endif - SetPageXpfoUnmapped(page); - set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + spin_lock_irqsave(&page->xpfo_lock, flags); + + /* + * In the case, where we raced with kmap after the + * atomic_dec_return, we must not nuke the mapping. + */ + if (atomic_read(&page->xpfo_mapcount) == 0) { + SetPageXpfoUnmapped(page); + set_kpte(kaddr, page, __pgprot(0)); + flush_tlb = true; + } + spin_unlock_irqrestore(&page->xpfo_lock, flags); } - spin_unlock_irqrestore(&page->xpfo_lock, flags); + if (flush_tlb) + xpfo_flush_kernel_tlb(page, 0); } void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, bool will_map); From patchwork Wed Apr 3 17:34:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 152551575 for ; Wed, 3 Apr 2019 17:37:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE0E227FA8 for ; Wed, 3 Apr 2019 17:37:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF753288F4; Wed, 3 Apr 2019 17:37:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E42427FA8 for ; Wed, 3 Apr 2019 17:37:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04A016B0274; Wed, 3 Apr 2019 13:37:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F14B76B0275; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8E766B0276; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f197.google.com (mail-yb1-f197.google.com [209.85.219.197]) by kanga.kvack.org (Postfix) with ESMTP id A32F66B0275 for ; Wed, 3 Apr 2019 13:37:08 -0400 (EDT) Received: by mail-yb1-f197.google.com with SMTP id k188so3832911yba.2 for ; Wed, 03 Apr 2019 10:37:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=JykUAtBTCqWOFSJWEypPAtam2nSMNUtg95DLyHs89DM=; b=sON+MWGtS3+MF9Wnym3qCw++0vGXWdqUHZunNlyZsHwVz5Frm/93kZfK3Fdmq8PXtp rgGABCZVY80FOrK6eo4s4zlocm+EXk99Ngp0QokoxIDB+vf3NqE+sR5EPK6f6/BYHBf6 UNfCjfWCaEac9krISx5Fc2Yi1ZcxvYoByvAMsDjoGLav7vandueIts7qkL7iW4246/IT HA13wSlHrT0Sn+XQFmN4LchglMVh5UP7K73wrNTcmA0YYyVHpz1m0ORnpk5QIByzidXg opdW5CN3++c2Nqobzwu7facp50iR/8rzm2UKVfSzCUpgHxZ4Hq63BRvIKnWxbBWE5NHA jxxA== X-Gm-Message-State: APjAAAXNZk8uhMeLmp/FVgH/yTQ0p3rwrfVlmoWsQIz3KVMK8qf2iYZK MhqcaG9KPriAa4+6GPRmoPQ1XNHdvl+iVBv9GhSlH5Ay8OYJH+fPOnih23qb8/bQYdW3v5Jah+U chR+Yf5OxlKL3UmiPEE26PS4feUS9aNo6D5f+aqH1tD2TEReT4L+VA1oXcDMVB5hv4w== X-Received: by 2002:a25:98c5:: with SMTP id m5mr1144430ybo.143.1554313028410; Wed, 03 Apr 2019 10:37:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqxn2rY1CnRrqXNH7+px0JnDH+3BG9rpOyMHETeuLlLL8bGu8msUxpsAsFLuEeIHLH3CATNg X-Received: by 2002:a25:98c5:: with SMTP id m5mr1144345ybo.143.1554313027446; Wed, 03 Apr 2019 10:37:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313027; cv=none; d=google.com; s=arc-20160816; b=t8aZnRRc/7GBVjhT1mSqbg4CIsa9Hjz6lSQ9HUsXpcO/11nv06wthbriCVhbZuLeLJ LGF/VyQDECoJYhTUrNPyoBOBFndAmcGnqBDb2IYZowD9qeR6QPTdWYN7yWSBV9u8TwLI BdWIDRoBElc6vPfgwUdASPRcFfBv5JJXNTobUQpYjKA0t3tSAlAfoLQlX3uTSOnrF35F ByABLgrVzdjZnuT6vyPySZ6NnxeDofI85LYP7CoRZ4lqe7RxMZ1o2VBcYfQTCVOJHjaO hlJqSfQY3XD2oVrl8QSt2he+XCm4AIrpxLcZiKJl9OsOhPrSkxQoOrQ+lPmF0tmI3/En ay1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=JykUAtBTCqWOFSJWEypPAtam2nSMNUtg95DLyHs89DM=; b=I/QPNVE3Jgc+oAoBptXajEVDkbIDQ2T3UO79eKdbEuD/qBLg4Sj3bcFT4qRPBkjkFW IRiGS2N6A9MvYNIX5XBZwsbSTOJlGDvD9VvgMgz9kYPdlKYApUBzdjlMxF8dcwkuG2/+ LrSc4lK36uCjQ0QRgsMIdW4J5L6+g4NvRS9eLz5Bu9eQJExKwlVqyzXehL+CE/Xs+jtz BnEBuOXz4PhFJz+KMyc23dyKINBkztGWSyChTBmJHznXnReDGHUWf4XzZG+1wuK6w7/v 6kaKvAn6X8RBWdXZJU/MJu48baG61OorgE8mDvLHuOZQ5g2HpbQ6GGqKi27PLjaA3pcu s2dg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Vj53rEyo; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id 189si10434595ybw.173.2019.04.03.10.37.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:07 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Vj53rEyo; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNnQl166346; Wed, 3 Apr 2019 17:36:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=JykUAtBTCqWOFSJWEypPAtam2nSMNUtg95DLyHs89DM=; b=Vj53rEyoOJe9xDAKjiGECNCgQg2SQKri1UPS9G1XShke/cdscozdvEvpv/G+VVLu4xxe 3SEbsqyIaW0o9SzqBtcLR0z3jd6uCgOAdHg8hHH3RSEs4k7HdloEX4pHYaiYW0QoLA+6 sGuCFspAZ48Kv3DH7G+yVs8xRa321vSYhjjav8gxNROgbj7DobNvcpuMRE+EXBKgkL86 iQ+gzC2L6MkXDpuv5qrhgjcwIZGWMOm6D7ENZBV3qENHikAMqMeGUy50jXwKJ0oTW/ta snZKmgrGzzf181FHwbn6/EV1yUlcAyeNPNqJPhsxLDSYR+PQ2Cf03aLdSjJUunjnHHWy QQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 2rhyvtahp1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:13 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZL7i152627; Wed, 3 Apr 2019 17:36:12 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserp3020.oracle.com with ESMTP id 2rm9mj6gx1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:12 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x33HaBO4028950; Wed, 3 Apr 2019 17:36:11 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:36:11 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Khalid Aziz , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 12/13] xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Date: Wed, 3 Apr 2019 11:34:13 -0600 Message-Id: <4495dda4bfc4a06b3312cc4063915b306ecfaecb.1554248002.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP XPFO flushes kernel space TLB entries for pages that are now mapped in userspace on not only the current CPU but also all other CPUs synchronously. Processes on each core allocating pages causes a flood of IPI messages to all other cores to flush TLB entries. Many of these messages are to flush the entire TLB on the core if the number of entries being flushed from local core exceeds tlb_single_page_flush_ceiling. The cost of TLB flush caused by unmapping pages from physmap goes up dramatically on machines with high core count. This patch flushes relevant TLB entries for current process or entire TLB depending upon number of entries for the current CPU and posts a pending TLB flush on all other CPUs when a page is unmapped from kernel space and mapped in userspace. Each core checks the pending TLB flush flag for itself on every context switch, flushes its TLB if the flag is set and clears it. This patch potentially aggregates multiple TLB flushes into one. This has very significant impact especially on machines with large core counts. To illustrate this, kernel was compiled with -j on two classes of machines - a server with high core count and large amount of memory, and a desktop class machine with more modest specs. System time from "make -j" from vanilla 4.20 kernel, 4.20 with XPFO patches before applying this patch and after applying this patch are below: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20 950.966s 4.20+XPFO 25073.169s 26.366x 4.20+XPFO+Deferred flush 1372.874s 1.44x Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20 607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush 803.989s 1.32x This same code should be implemented for other architectures as well once finalized. Signed-off-by: Khalid Aziz Cc: Khalid Aziz --- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 52 +++++++++++++++++++++++++++++++++ arch/x86/mm/xpfo.c | 2 +- 3 files changed, 54 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index f4204bf377fc..92d23629d01d 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -561,6 +561,7 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void xpfo_flush_tlb_kernel_range(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 999d6d8f0bef..cc806a01a0eb 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -37,6 +37,20 @@ */ #define LAST_USER_MM_IBPB 0x1UL +/* + * A TLB flush may be needed to flush stale TLB entries + * for pages that have been mapped into userspace and unmapped + * from kernel space. This TLB flush needs to be propagated to + * all CPUs. Asynchronous flush requests to all CPUs can cause + * significant performance imapct. Queue a pending flush for + * a CPU instead. Multiple of these requests can then be handled + * by a CPU at a less disruptive time, like context switch, in + * one go and reduce performance impact significantly. Following + * data structure is used to keep track of CPUs with pending full + * TLB flush forced by xpfo. + */ +static cpumask_t pending_xpfo_flush; + /* * We get here when we do something requiring a TLB invalidation * but could not go invalidate all of the contexts. We do the @@ -321,6 +335,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, __flush_tlb_all(); } #endif + + /* + * If there is a pending TLB flush for this CPU due to XPFO + * flush, do it now. + */ + if (cpumask_test_and_clear_cpu(cpu, &pending_xpfo_flush)) { + count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED); + __flush_tlb_all(); + } + this_cpu_write(cpu_tlbstate.is_lazy, false); /* @@ -803,6 +827,34 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) } } +void xpfo_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + struct cpumask tmp_mask; + + /* + * Balance as user space task's flush, a bit conservative. + * Do a local flush immediately and post a pending flush on all + * other CPUs. Local flush can be a range flush or full flush + * depending upon the number of entries to be flushed. Remote + * flushes will be done by individual processors at the time of + * context switch and this allows multiple flush requests from + * other CPUs to be batched together. + */ + if (end == TLB_FLUSH_ALL || + (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) { + do_flush_tlb_all(NULL); + } else { + struct flush_tlb_info info; + + info.start = start; + info.end = end; + do_kernel_range_flush(&info); + } + cpumask_setall(&tmp_mask); + __cpumask_clear_cpu(smp_processor_id(), &tmp_mask); + cpumask_or(&pending_xpfo_flush, &pending_xpfo_flush, &tmp_mask); +} + void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info info = { diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index b42513347865..638eee5b1f09 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -118,7 +118,7 @@ inline void xpfo_flush_kernel_tlb(struct page *page, int order) return; } - flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); + xpfo_flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); } EXPORT_SYMBOL_GPL(xpfo_flush_kernel_tlb); From patchwork Wed Apr 3 17:34:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10884305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 271DE17E0 for ; Wed, 3 Apr 2019 17:37:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F8CB27FA8 for ; Wed, 3 Apr 2019 17:37:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01C86288F4; Wed, 3 Apr 2019 17:37:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B86DC27FA8 for ; Wed, 3 Apr 2019 17:37:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAD546B0275; Wed, 3 Apr 2019 13:37:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C08376B0276; Wed, 3 Apr 2019 13:37:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A898C6B0277; Wed, 3 Apr 2019 13:37:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 795246B0275 for ; Wed, 3 Apr 2019 13:37:15 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id 204so10294724ybf.5 for ; Wed, 03 Apr 2019 10:37:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=BPCoKe4N/Rog3LaZqAmbRh3NwY6xgOGLh9z8YMMslMU=; b=UDFrMYq9ETzRhfMYnFoSBFjDu62DMsPMaYmZ1RQpUZhc1DoW6jX14Atll7nvpxYlwi E7yZY9RD/1LpO3hfzEvJ0Nh4zP9VvRHz6vPMbMklnFJGigaRkCy+wiDNFNP/yh39oMAV ylwamm3P1WxvyWrk6tY/si8FezaGKjMcvDbjdujuhne9JRMltmND+rizyORgeEai3m0l Sayfh3vZ5SMydliFJvK2L7fIenjleiQWcY2+e2XUHwJZhAkPbTLShZvIf0/p0vuCnaL9 Z7iaR6p12Y7L8TmlNhYKk8qPmhNzZdG2L+gfELt6WeVkWQOuZc3W6U/AWHTwsanO/ze4 0S6w== X-Gm-Message-State: APjAAAUyhy49Oy7rwSxKtR7DWS4l8LRC3lUKaZrhWE5VPVeu0K7M8EB3 UfMu+1XfQI0uNu6PMcjSuUUgiqv2ezkXwG/Fs9FLo80kjegAIL1pGwJh4TtsTKOK1Y1IhFjdrQz Pf11EkjeOOgBaZud4blmUbSfknXpKiCoZPAS24SHb51jwU574AzCJHw1ea2+v2CLm+A== X-Received: by 2002:a25:4c84:: with SMTP id z126mr1181815yba.270.1554313035187; Wed, 03 Apr 2019 10:37:15 -0700 (PDT) X-Google-Smtp-Source: APXvYqwtPljSinaU0UQ4GYz189UEm5na8ZVr38kT5Z2E0J+NL+q0caCbZGCcRwymdmz7vWLoR1Yt X-Received: by 2002:a25:4c84:: with SMTP id z126mr1181714yba.270.1554313033975; Wed, 03 Apr 2019 10:37:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554313033; cv=none; d=google.com; s=arc-20160816; b=fgbOLBetZSbhBCJsbc3g82TMciC1+A41X6AYYQYxUlVu47AOmD0x/2mixYYZOgon3I tOsmyt3w3NIqUsu0QrBMjv9CE9V3jrtoepjkQ43RLITM596ERFa93cydXqdwd0lQFIxS 998JUHES+VAsvpOS+lR5FChSGvD3o5PZk0F6t5WnqUOTYJEzrE8mOnuwCUI5p5b84o5h z4PjsBS9D6WWVUdT8aoJZjKuHO6zvVPMNM3U0NNhgHqD8PkEn0Mb/+OSwXuHGs6a33ys xKem7gFajkDK6g7brSzqcv8qW58GzXWIezCqJAS7GpWctN2bnMJhtFkwzR2OrDgYapRj f6Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=BPCoKe4N/Rog3LaZqAmbRh3NwY6xgOGLh9z8YMMslMU=; b=EPUe3VpFP9L7lgWdvcv7SciwmlKiD7Rdq4DLRdnVC4XOB/1Ftgbu4ezog3YR67Ullo 1E3y9YAnNdrq0UldBU1fMZWKPB7IL+emPUOa6T2cmcUuLZpe/tZmWNVRqDpnnhPCjQIK HxbnEoPVnDCt1sIWP54aoobUFubIiS3cACJbrU7I7OQjddvVfnMga34no54i5QbQRm00 IsvTW9XSRBafpEdxdJiwwM3r+NY9WELEcPiWFDRJujqg+Zq2Fu7lWNehuxHnBxKxOwNB 9xJymihymwWfRVVoPk3il5U7N00+D9RAfiQoqj8QFjFLwDJVXS5HO2lpe6ic2roDxdDQ wQbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SG5anKd8; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id l18si6492681ywl.434.2019.04.03.10.37.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 10:37:13 -0700 (PDT) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SG5anKd8; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HNuv5175541; Wed, 3 Apr 2019 17:36:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=BPCoKe4N/Rog3LaZqAmbRh3NwY6xgOGLh9z8YMMslMU=; b=SG5anKd8PlRvt+kNrFuRkC7L7Qh8X5Jdc62zx1QoGlwH6aQfOAKP6SxUjbMqQVs6iCVv +eXoXspZBjqyfQPDahh3j8wMnrTqXsIBu6zJojc0zdtGCjF9nCEhG1RDhoRlKcgioRJW SpaUeb83RbynUMLVwGE5n/px02BNncuKu1ZQ8yDDpUP9ydNpbrjMj4q8UMOUdmtW4rSo FGUeIkRdaLcqesQ3RxPcB8LFOJ0jAESJHRZvkGU4fWMMuj9sXGV0ZgoFUWHa13yxI1Qn tmg6i4Oawm6EPZN9NR6XQrL1A7BW6NYt+2vDCXkEjhu7VSOvBuLJDxNy1m8K008GOGz+ zQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 2rj13qaeae-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:22 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x33HZIC1110882; Wed, 3 Apr 2019 17:36:21 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3030.oracle.com with ESMTP id 2rm8f5fyqd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Apr 2019 17:36:21 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x33HaGCv001670; Wed, 3 Apr 2019 17:36:16 GMT Received: from concerto.internal (/10.65.181.37) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 03 Apr 2019 10:36:15 -0700 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: Khalid Aziz , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com, akpm@linux-foundation.org, alexander.h.duyck@linux.intel.com, amir73il@gmail.com, andreyknvl@google.com, aneesh.kumar@linux.ibm.com, anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org, arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk, bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl, catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org, daniel.vetter@ffwll.ch, dan.j.williams@intel.com, gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org, hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com, jannh@google.com, jgross@suse.com, jkosina@suse.cz, jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com, jroedel@suse.de, keith.busch@intel.com, khlebnikov@yandex-team.ru, logang@deltatee.com, marco.antonio.780@gmail.com, mark.rutland@arm.com, mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz, mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com, m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de, paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com, rdunlap@infradead.org, richard.weiyang@gmail.com, riel@surriel.com, rientjes@google.com, robin.murphy@arm.com, rostedt@goodmis.org, rppt@linux.vnet.ibm.com, sai.praneeth.prakhya@intel.com, serge@hallyn.com, steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz, will.deacon@arm.com, willy@infradead.org, yang.shi@linux.alibaba.com, yaojun8558363@gmail.com, ying.huang@intel.com, zhangshaokun@hisilicon.com, iommu@lists.linux-foundation.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v9 13/13] xpfo, mm: Optimize XPFO TLB flushes by batching them together Date: Wed, 3 Apr 2019 11:34:14 -0600 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9216 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904030118 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When XPFO forces a TLB flush on all cores, the performance impact is very significant. Batching as many of these TLB updates as possible can help lower this impact. When a userspace allocates a page, kernel tries to get that page from the per-cpu free list. This free list is replenished in bulk when it runs low. Free list is being replenished for future allocation to userspace is a good opportunity to update TLB entries in batch and reduce the impact of multiple TLB flushes later. This patch adds new tags for the page so a page can be marked as available for userspace allocation and unmapped from kernel address space. All such pages are removed from kernel address space in bulk at the time they are added to per-cpu free list. This patch when combined with deferred TLB flushes improves performance further. Using the same benchmark as before of building kernel in parallel, here are the system times on two differently sized systems: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 5.0 913.862s 5.0+XPFO+Deferred flush+Batch update 1165.259s 1.28x Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 5.0 610.642s 5.0+XPFO+Deferred flush+Batch update 773.075s 1.27x Signed-off-by: Khalid Aziz Cc: Khalid Aziz Signed-off-by: Tycho Andersen --- v9: - Do not map a page freed by userspace back into kernel. Mark it as unmapped instead and map it back in only when needed. This avoids the cost of unmap and TLBV flush if the page is allocated back to userspace. arch/x86/include/asm/pgtable.h | 2 +- arch/x86/mm/pageattr.c | 9 ++++-- arch/x86/mm/xpfo.c | 11 +++++-- include/linux/xpfo.h | 11 +++++++ mm/page_alloc.c | 9 ++++++ mm/xpfo.c | 54 +++++++++++++++++++++++++++------- 6 files changed, 79 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5c0e1581fa56..61f64c6c687c 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1461,7 +1461,7 @@ should_split_large_page(pte_t *kpte, unsigned long address, extern spinlock_t cpa_lock; int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, - struct page *base); + struct page *base, bool xpfo_split); #include #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 530b5df0617e..8fe86ac6bff0 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -911,7 +911,7 @@ static void split_set_pte(struct cpa_data *cpa, pte_t *pte, unsigned long pfn, int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, - struct page *base) + struct page *base, bool xpfo_split) { unsigned long lpaddr, lpinc, ref_pfn, pfn, pfninc = 1; pte_t *pbase = (pte_t *)page_address(base); @@ -1008,7 +1008,10 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, * page attribute in parallel, that also falls into the * just split large page entry. */ - flush_tlb_all(); + if (xpfo_split) + xpfo_flush_tlb_all(); + else + flush_tlb_all(); spin_unlock(&pgd_lock); return 0; @@ -1027,7 +1030,7 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, if (!base) return -ENOMEM; - if (__split_large_page(cpa, kpte, address, base)) + if (__split_large_page(cpa, kpte, address, base, false)) __free_page(base); return 0; diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index 638eee5b1f09..8c482c7b54f5 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -47,7 +47,7 @@ inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) cpa.vaddr = kaddr; cpa.pages = &page; - cpa.mask_set = prot; + cpa.mask_set = canon_pgprot(prot); cpa.mask_clr = msk_clr; cpa.numpages = 1; cpa.flags = 0; @@ -57,7 +57,7 @@ inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) do_split = should_split_large_page(pte, (unsigned long)kaddr, &cpa); - if (do_split) { + if (do_split > 0) { struct page *base; base = alloc_pages(GFP_ATOMIC, 0); @@ -69,7 +69,7 @@ inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) if (!debug_pagealloc_enabled()) spin_lock(&cpa_lock); if (__split_large_page(&cpa, pte, (unsigned long)kaddr, - base) < 0) { + base, true) < 0) { __free_page(base); WARN(1, "xpfo: failed to split large page\n"); } @@ -90,6 +90,11 @@ inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) } EXPORT_SYMBOL_GPL(set_kpte); +void xpfo_flush_tlb_all(void) +{ + xpfo_flush_tlb_kernel_range(0, TLB_FLUSH_ALL); +} + inline void xpfo_flush_kernel_tlb(struct page *page, int order) { int level; diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 37e7f52fa6ce..01da4bb31cd6 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -32,6 +32,7 @@ DECLARE_STATIC_KEY_TRUE(xpfo_inited); /* Architecture specific implementations */ void set_kpte(void *kaddr, struct page *page, pgprot_t prot); void xpfo_flush_kernel_tlb(struct page *page, int order); +void xpfo_flush_tlb_all(void); void xpfo_init_single_page(struct page *page); @@ -106,6 +107,9 @@ void xpfo_temp_map(const void *addr, size_t size, void **mapping, void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, size_t mapping_len); +bool xpfo_pcp_refill(struct page *page, enum migratetype migratetype, + int order); + #else /* !CONFIG_XPFO */ static inline void xpfo_init_single_page(struct page *page) { } @@ -118,6 +122,7 @@ static inline void xpfo_free_pages(struct page *page, int order) { } static inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) { } static inline void xpfo_flush_kernel_tlb(struct page *page, int order) { } +static inline void xpfo_flush_tlb_all(void) { } static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } @@ -133,6 +138,12 @@ static inline void xpfo_temp_unmap(const void *addr, size_t size, { } +static inline bool xpfo_pcp_refill(struct page *page, + enum migratetype migratetype, int order) +{ + return false; +} + #endif /* CONFIG_XPFO */ #if (!defined(CONFIG_HIGHMEM)) && (!defined(ARCH_HAS_KMAP)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e0dda1322a2..7846b2590ef0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3031,6 +3031,8 @@ static struct page *__rmqueue_pcplist(struct zone *zone, int migratetype, struct list_head *list) { struct page *page; + struct list_head *cur; + bool flush_tlb = false; do { if (list_empty(list)) { @@ -3039,6 +3041,13 @@ static struct page *__rmqueue_pcplist(struct zone *zone, int migratetype, migratetype, alloc_flags); if (unlikely(list_empty(list))) return NULL; + list_for_each(cur, list) { + page = list_entry(cur, struct page, lru); + flush_tlb |= xpfo_pcp_refill(page, + migratetype, 0); + } + if (flush_tlb) + xpfo_flush_tlb_all(); } page = list_first_entry(list, struct page, lru); diff --git a/mm/xpfo.c b/mm/xpfo.c index 974f1b70ccd9..47d400f1fc65 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -62,17 +62,22 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp, bool will_map) WARN_ON(atomic_read(&(page + i)->xpfo_mapcount)); #endif if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { + bool user_page = TestSetPageXpfoUser(page + i); + /* * Tag the page as a user page and flush the TLB if it * was previously allocated to the kernel. */ - if ((!TestSetPageXpfoUser(page + i)) || !will_map) { - SetPageXpfoUnmapped(page + i); - flush_tlb = true; + if (!user_page || !will_map) { + if (!TestSetPageXpfoUnmapped(page + i)) + flush_tlb = true; } } else { /* Tag the page as a non-user (kernel) page */ ClearPageXpfoUser(page + i); + if (TestClearPageXpfoUnmapped(page + i)) + set_kpte(page_address(page + i), page + i, + PAGE_KERNEL); } } @@ -95,14 +100,12 @@ void xpfo_free_pages(struct page *page, int order) #endif /* - * Map the page back into the kernel if it was previously - * allocated to user space. + * Leave the page as unmapped from kernel. If this page + * gets allocated to userspace soon again, it saves us + * the cost of TLB flush at that time. */ - if (TestClearPageXpfoUser(page + i)) { - ClearPageXpfoUnmapped(page + i); - set_kpte(page_address(page + i), page + i, - PAGE_KERNEL); - } + if (PageXpfoUser(page + i)) + SetPageXpfoUnmapped(page + i); } } @@ -134,3 +137,34 @@ void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, kunmap_atomic(mapping[i]); } EXPORT_SYMBOL(xpfo_temp_unmap); + +bool xpfo_pcp_refill(struct page *page, enum migratetype migratetype, + int order) +{ + int i; + bool flush_tlb = false; + + if (!static_branch_unlikely(&xpfo_inited)) + return false; + + for (i = 0; i < 1 << order; i++) { + if (migratetype == MIGRATE_MOVABLE) { + /* GPF_HIGHUSER ** + * Tag the page as a user page, mark it as unmapped + * in kernel space and flush the TLB if it was + * previously allocated to the kernel. + */ + SetPageXpfoUser(page + i); + if (!TestSetPageXpfoUnmapped(page + i)) + flush_tlb = true; + } else { + /* Tag the page as a non-user (kernel) page */ + ClearPageXpfoUser(page + i); + } + } + + if (flush_tlb) + set_kpte(page_address(page), page, __pgprot(0)); + + return flush_tlb; +}