From patchwork Sun Mar 12 11:26:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13171299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36BF6C6FA99 for ; Sun, 12 Mar 2023 11:26:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C24FF8E0007; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD5C78E0001; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4E358E0007; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 981438E0001 for ; Sun, 12 Mar 2023 07:26:38 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6AAF61C5FBC for ; Sun, 12 Mar 2023 11:26:38 +0000 (UTC) X-FDA: 80560018476.01.D669FAA Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id 32D9110000D for ; Sun, 12 Mar 2023 11:26:35 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NRF0ntB8; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678620396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+EDOkCcFvj2LTHdnZWHE+HYEtQCPudT/tOJKFx84T2I=; b=YlRrWVyF+fpPZ42il7yWeyTYsadGuJuZx2HJ4dnfT+LOW9xTKqqjJGEhZMZmQwk7mPCCCT tZ7/zfxhzI3EUtdPbw304uFVPmLLjUH2ayRc0nR5ILQKS31tkVpYo9ea1ckyh5LDH/l1Tk /nFmD0644i7kC30WGDkpH9jwTNr7Zns= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NRF0ntB8; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678620396; a=rsa-sha256; cv=none; b=buXuVPEm9eQMt9hCWdKfOd5r7lNNnmMteKjRtvJv1Cu3EiAWZ2RBRkyjTF7sCHA00zoIyF PuKQqBVxmN1EqTePBD9JfOwMgUB8FxqINd7n2p4GR65XFGKvaZeodSWMt+1uCe8ubHJaCD rLxIuYXdiTXbF1Xi5ESTxnrB9oiC3oM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678620396; x=1710156396; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jhNKWlLSFQdtM5nCCWCne2jNIxj0oFZDDW74FDfmA8g=; b=NRF0ntB8riqF1pN8aA7F6+85mTZyCoGV1WMKVLaWUQEbJZMlKEG3eMRM o24Ths9sYay86XVfpoK0dehDutGwZQ4ZVYlKR/UqJcET5BxpJDJeUpKFo 7vdSu5MlsC3gR8vD2VdvSJWhLDsuyJT8HCeC4xQ8cKLBJ7OGRRlN0OWSl TRKiUFhm0AYa4+7F3wW4Qp4kXejoPbFjzj++51Y6K+x6pDz7A7Gr0TEZC L73iaYFEWlClur0ETYlRHqe6BxASyiPrzsJ2WNqmvCR99ye0J9bhqxGFv LuzIslVCir8NwER8nriCHoRAtNYShCGCGMZgiKlQBhCgl582PhX4achoS A==; X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="339349954" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="339349954" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10646"; a="852438029" X-IronPort-AV: E=Sophos;i="5.98,254,1673942400"; d="scan'208";a="852438029" Received: from nmoazzen-mobl1.amr.corp.intel.com (HELO box.shutemov.name) ([10.251.219.215]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2023 04:26:30 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 84A2710D7B3; Sun, 12 Mar 2023 14:26:19 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv16 05/17] mm: Introduce untagged_addr_remote() Date: Sun, 12 Mar 2023 14:26:00 +0300 Message-Id: <20230312112612.31869-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> References: <20230312112612.31869-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 32D9110000D X-Rspam-User: X-Stat-Signature: 3d3yzcq8bs3ib6gkz8popifjtfqm3a3h X-HE-Tag: 1678620395-472602 X-HE-Meta: U2FsdGVkX18/4YQRJWoXWRaqX9O7hC5p6ZROD15RE/leYeaK7RN3dVNYGILbX7emS15462s+Yrcl6pvV5f/8U024zVqDGkpDNhCkgvctn1WvP/u1G/NTqqt22jMvIity8spzdBfuxi4C220ftSSBQgZEbv5zmNC+CbrCeE2nRTPAddW/gcwuuETs3xHYhQlaBQnX3z0l2RbseDOraqjfr5i6ziOHZkh8JhzPgxjZoN0ZZLSnMOkCh7wm42u/fkcb8o5zVSVAkjVUxZV8L7grhm/2dftFnYZo8pw0fJAw5PIrjKl3Okf6kX4Q6BoSHKIUetCTVuT7GQrc0r/3o6wdx5+/rHQGvfnP6aX/X5mN+mE9lRGFzipxkSADr3/VdPxtRVX78c3bSc9hkT+ZGVFKB5z+itnGZdK/wf5ZumNVKZP0eJn5WZ+S66SFrgrDF3cSl/aRQi0KJqXYNE8aKO+mFSXBvHhdCJWRT+1tEuktx2SO6HqUUr9gZ7XnV5NMuTsoEehSBIPe7utXjGrAav36At4KMZ5MeD5xmmS9DMxj6fGEcOjsRccaUhIGFLApPGKMl5OYFpcb4mI3K1dMmUmA41ZZqjhKTelYtiHMr86OFYMrOyhJPt+rxcKiPk3Xcfguo/N1UMSRyfe3NgfNSVRHdQC7HmKfUskv2s+j/W18NX7Ly+HW9FHjjrRDWK7gt2BX+lrUTqj+lvMdikcklqrfC+RqauLi8Lnv78kaGxXif2b2aHlgcQN35VN6sm5iLQc2MhV8VHB5+XYODYiGfl6Hhm2WxViyNLljQof/e79YCk/XeCiM5RXTiSEd5sS16OftoPLRnr8RewxBLaDQwwoOa3uNp+zfi7ZYPNpzVkvPyHG55v4q3oZQwtKBGN081w8xlFbGnGnB1lmj2S8b31RV3+ho0K6swM2bWV7bW1mIoqssgIG+jMWxSmX+3O46kcsEFYFm8W7+J6SJHL4ADt4 dz5lROy3 OO52DFl49UNUUEXqsE2iygZI6wNnEKSkLTZKPnX7u9BBFFxCRJov2PQXY2oy6+Er3INHPm5yKYHxPhdVmiQYzgVAvcGRJMLbjp36eLGIcYmp9w6l/Nqo4p8nBdd9rJa/oId7sdaKLUF93mbKCmuqGr5sDFNBBXTR9keQuzlvlCBy4qFOgBqSUnYvn5EgwdxgSSfQ7dRZJi6FtgAe79AQOqt6W3r01cOmsPsGZ1Ifh1WJUVBCw1abpGRgwk0rb+YwZjxIt3VagbmcTIpboTGOjewA34TvoAEkT0x3m2oRJXrMOsr1NocXAkh77fugbYofttofVRDt4Ya1IebxfEh49QBlIPNm2RJ2q8/46C24MZZNqr5M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() removes tags/metadata from the address and brings it to the canonical form. The helper is implemented on arm64 and sparc. Both of them do untagging based on global rules. However, Linear Address Masking (LAM) on x86 introduces per-process settings for untagging. As a result, untagged_addr() is now only suitable for untagging addresses for the current proccess. The new helper untagged_addr_remote() has to be used when the address targets remote process. It requires the mmap lock for target mm to be taken. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko Reviewed-by: Rick Edgecombe --- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 9 +++++++-- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 22 ++++++++++++++++++++++ mm/gup.c | 4 ++-- mm/madvise.c | 5 +++-- mm/migrate.c | 11 ++++++----- 8 files changed, 43 insertions(+), 23 deletions(-) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 493c31de0edb..3d4dd9420c30 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -580,7 +580,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr_remote(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 6a96e1713fd5..29fd6b1f4058 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1689,8 +1689,13 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; - if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) { + ret = mmap_read_lock_killable(mm); + if (ret) + goto out_free; + start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); + mmap_read_unlock(mm); + } /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1f79667824eb..289ae4caf878 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -96,17 +96,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index ab9728138ad6..3064314f4832 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,28 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(addr) (addr) +#endif + +#ifndef untagged_addr_remote +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr(addr); \ +}) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/mm/gup.c b/mm/gup.c index eab18ba045db..5ee8b682a0fe 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1085,7 +1085,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr_remote(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1259,7 +1259,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr_remote(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; diff --git a/mm/madvise.c b/mm/madvise.c index 340125d08c03..d4b67f36f70f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1402,8 +1402,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) return -EINVAL; @@ -1435,6 +1433,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh mmap_read_lock(mm); } + start = untagged_addr_remote(mm, start); + end = start + len; + blk_start_plug(&plug); error = madvise_walk_vmas(mm, start, end, behavior, madvise_vma_behavior); diff --git a/mm/migrate.c b/mm/migrate.c index 98f1c11197a8..8cd11bc9208f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2097,15 +2097,18 @@ static int do_move_pages_to_node(struct mm_struct *mm, * target node * 1 - when it has been queued */ -static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, +static int add_page_for_migration(struct mm_struct *mm, const void __user *p, int node, struct list_head *pagelist, bool migrate_all) { struct vm_area_struct *vma; + unsigned long addr; struct page *page; int err; bool isolated; mmap_read_lock(mm); + addr = (unsigned long)untagged_addr_remote(mm, p); + err = -EFAULT; vma = vma_lookup(mm, addr); if (!vma || !vma_migratable(vma)) @@ -2211,7 +2214,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, for (i = start = 0; i < nr_pages; i++) { const void __user *p; - unsigned long addr; int node; err = -EFAULT; @@ -2219,7 +2221,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) @@ -2247,8 +2248,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, * Errors in the page lookup or isolation are not fatal and we simply * report them via status */ - err = add_page_for_migration(mm, addr, current_node, - &pagelist, flags & MPOL_MF_MOVE_ALL); + err = add_page_for_migration(mm, p, current_node, &pagelist, + flags & MPOL_MF_MOVE_ALL); if (err > 0) { /* The page is successfully queued for migration */