From patchwork Fri Nov 27 16:41:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11936841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AED9C83016 for ; Fri, 27 Nov 2020 16:42:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 388E22224D for ; Fri, 27 Nov 2020 16:42:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="eM72GLPv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 388E22224D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 34BBF6B0074; Fri, 27 Nov 2020 11:42:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1CF6B0075; Fri, 27 Nov 2020 11:42:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03FF86B0078; Fri, 27 Nov 2020 11:42:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id DB9F26B0074 for ; Fri, 27 Nov 2020 11:42:08 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9630C181AEF31 for ; Fri, 27 Nov 2020 16:42:08 +0000 (UTC) X-FDA: 77530765536.07.twig87_4e0237827389 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 77EE51804808A for ; Fri, 27 Nov 2020 16:42:08 +0000 (UTC) X-HE-Tag: twig87_4e0237827389 X-Filterd-Recvd-Size: 8244 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 27 Nov 2020 16:42:07 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id k14so6233319wrn.1 for ; Fri, 27 Nov 2020 08:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5mwQ6o8DhdhcroL4LW9wEcv777ijL/zuzq6//0vwABs=; b=eM72GLPvRb0+NA69AcDpHC3F4eppz8RzSuVRvWvj6NYFddTDwM0wR2gA/ig5q0vSbf vXbbdD9UJ1Ta3z794xTEdxGiXL3gVBc0stoXbWnWWa173wMg5uqTTolryg9S7FbD60Up U4+K/3S25bDJqLD/pWRVNKRCeFhX+cA0aQDgA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5mwQ6o8DhdhcroL4LW9wEcv777ijL/zuzq6//0vwABs=; b=gWs8Lv5h4Sjp2E9miVNo6YtG2gk8ZwbPa4f0Cfp+U0FdvmmE6FyXfXfDE4Y8W/3Mbk DpaOb1ixtD9KxuL05PjgC7WbL9Aa+DRae4w8l8aA3Q+/hT5Lk/UpcTM5l/D4QELcP9Qm DHPIaLC1lfw/WDyJgUITmbnJl+qFvCt++OyD4rRsMccwvyPtQQ/kc8Inb1MgR0ePMubh bCBzOb0Z0niEkd4jF4uRY31kTjn0dB0L8XIfe3KoeWrjrZ8blpGSHa/3rs8OOBX6BXsL wNODTfT28c3RVV+OD9R4cB02Wa5eZtWpQ62DitZzv4271dw1MgkRKC5EDJ+KrIZEpWfA 4KLA== X-Gm-Message-State: AOAM530r+yRRs55OcNoTvrrDUf9W3vCY0+3OVBlOR5tME9lzN94UtOV9 8Ys5rF36At649GNShAfxiL5FwA== X-Google-Smtp-Source: ABdhPJzXeZy21QC9Hw6PEXLMq1p5UY22aoFAupO7CL+myn2UqrybcS7gEaEayX4JMT1GuwWyf+ZKyg== X-Received: by 2002:a5d:6310:: with SMTP id i16mr11734297wru.284.1606495327078; Fri, 27 Nov 2020 08:42:07 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id q12sm14859078wrx.86.2020.11.27.08.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Nov 2020 08:42:06 -0800 (PST) From: Daniel Vetter To: DRI Development , LKML Cc: kvm@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-media@vger.kernel.org, Daniel Vetter , Daniel Vetter , Jason Gunthorpe , Dan Williams , Kees Cook , Benjamin Herrensmidt , Dave Airlie , Andrew Morton , John Hubbard , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jan Kara , Chris Wilson Subject: [PATCH v7 07/17] mm: Close race in generic_access_phys Date: Fri, 27 Nov 2020 17:41:21 +0100 Message-Id: <20201127164131.2244124-8-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201127164131.2244124-1-daniel.vetter@ffwll.ch> References: <20201127164131.2244124-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Way back it was a reasonable assumptions that iomem mappings never change the pfn range they point at. But this has changed: - gpu drivers dynamically manage their memory nowadays, invalidating ptes with unmap_mapping_range when buffers get moved - contiguous dma allocations have moved from dedicated carvetouts to cma regions. This means if we miss the unmap the pfn might contain pagecache or anon memory (well anything allocated with GFP_MOVEABLE) - even /dev/mem now invalidates mappings when the kernel requests that iomem region when CONFIG_IO_STRICT_DEVMEM is set, see 3234ac664a87 ("/dev/mem: Revoke mappings when a driver claims the region") Accessing pfns obtained from ptes without holding all the locks is therefore no longer a good idea. Fix this. Since ioremap might need to manipulate pagetables too we need to drop the pt lock and have a retry loop if we raced. While at it, also add kerneldoc and improve the comment for the vma_ops->access function. It's for accessing, not for moving the memory from iomem to system memory, as the old comment seemed to suggest. References: 28b2ee20c7cb ("access_process_vm device memory infrastructure") Signed-off-by: Daniel Vetter Cc: Jason Gunthorpe Cc: Dan Williams Cc: Kees Cook Cc: Benjamin Herrensmidt Cc: Dave Airlie Cc: Andrew Morton Cc: John Hubbard Cc: Jérôme Glisse Cc: Jan Kara Cc: Dan Williams Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-samsung-soc@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: Chris Wilson Signed-off-by: Daniel Vetter --- v2: Fix inversion in the retry check (John). v4: While at it, use offset_in_page (Chris Wilson) --- include/linux/mm.h | 3 ++- mm/memory.c | 46 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c4cc8ea1402c..b50cbb33d0f3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -574,7 +574,8 @@ struct vm_operations_struct { vm_fault_t (*pfn_mkwrite)(struct vm_fault *vmf); /* called by access_process_vm when get_user_pages() fails, typically - * for use by special VMAs that can switch between memory and hardware + * for use by special VMAs. See also generic_access_phys() for a generic + * implementation useful for any iomem mapping. */ int (*access)(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index c48f8df6e502..ac32039ce941 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4847,28 +4847,68 @@ int follow_phys(struct vm_area_struct *vma, return ret; } +/** + * generic_access_phys - generic implementation for iomem mmap access + * @vma: the vma to access + * @addr: userspace addres, not relative offset within @vma + * @buf: buffer to read/write + * @len: length of transfer + * @write: set to FOLL_WRITE when writing, otherwise reading + * + * This is a generic implementation for &vm_operations_struct.access for an + * iomem mapping. This callback is used by access_process_vm() when the @vma is + * not page based. + */ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write) { resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - int offset = addr & (PAGE_SIZE-1); + pte_t *ptep, pte; + spinlock_t *ptl; + int offset = offset_in_page(addr); + int ret = -EINVAL; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + return -EINVAL; + +retry: + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + return -EINVAL; + pte = *ptep; + pte_unmap_unlock(ptep, ptl); - if (follow_phys(vma, addr, write, &prot, &phys_addr)) + prot = pgprot_val(pte_pgprot(pte)); + phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; + + if ((write & FOLL_WRITE) && !pte_write(pte)) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + goto out_unmap; + + if (!pte_same(pte, *ptep)) { + pte_unmap_unlock(ptep, ptl); + iounmap(maddr); + + goto retry; + } + if (write) memcpy_toio(maddr + offset, buf, len); else memcpy_fromio(buf, maddr + offset, len); + ret = len; + pte_unmap_unlock(ptep, ptl); +out_unmap: iounmap(maddr); - return len; + return ret; } EXPORT_SYMBOL_GPL(generic_access_phys); #endif