From patchwork Fri Oct 23 12:21:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11853373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CAB4C55179 for ; Fri, 23 Oct 2020 12:40:29 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE53A208C3 for ; Fri, 23 Oct 2020 12:40:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="j4NgvgGv"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ESRA9XXX"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="kxcHr9Xa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE53A208C3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DfpMIO1TXypXyFSJzBjAD31liM/M4Op8JBXp7SoVNVY=; b=j4NgvgGvwyKPo5XPXcaPC3OV4 U666FUMyOF56lgSA/ZImOCR7Dh362/mnbl0b4wCVX/nCkhzX4b1LRJDzEkSf9I1hK33BKH11/2tYx iLWapWO1aGOMMfJKl54tvqWiSresKaiTX9up969z571XOKjVumkJ4ySBOOm3qKisZtlQoL75ILlVc DSqyZpqxbyICMdZWil/Cw1Q3KQyoIARl7AarSL8ZusFpyadBsL9R+LLnCtDscW+S8apeTGVYjbq2t GKhvqmR39j9AvkDryP7leIc/3kiyoZPq+Hzzq90f+0vxYd3O575az9Ims2MeIbXykJdJY914gnqCk bwmSC9lwA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVwLT-0004j4-7K; Fri, 23 Oct 2020 12:38:59 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVwLM-0004fb-0e for linux-arm-kernel@merlin.infradead.org; Fri, 23 Oct 2020 12:38:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-ID:Content-Description; bh=hPo/Bw6tNC6m8oopvfckb4iPuUPePa6aSZ/PC8jTJPc=; b=ESRA9XXXwiTZVtyxhIktoUhTXg xxHghVIBvp9U/+t8hCi/daLpfJ6MRTNDVVj5Nkoc193JiphcxEMw8ze96qXdG9EDa74A8ZSOmfDZq i5HAWC3ALllsMvUy9SjZzGoZ2Vd1Y1ig57qlOIv+ws5BC3ZCZH8C3oQA1z6j5J0+Uu8K8s8oC3iVh ps/g1/iOYJzxQCv029R8SkwougdCWXd7sMCuEQ7N/v4xZLlRCriy5Yam6tZB3bz2I62/2xS5YeBkN eYXWWf7hjI1CWzNEXtxAHAOhFW26dZjofHUxys3DaNlw4yY+CUDjYUMvZ8sDhivWY95/Nv8qPy76r aDKuwLkg==; Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVw68-0001OZ-M7 for linux-arm-kernel@lists.infradead.org; Fri, 23 Oct 2020 12:23:13 +0000 Received: by mail-wr1-x444.google.com with SMTP id s9so1566853wro.8 for ; Fri, 23 Oct 2020 05:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hPo/Bw6tNC6m8oopvfckb4iPuUPePa6aSZ/PC8jTJPc=; b=kxcHr9XaGgdgUN/6f9tRbwBS2R7y8JekVykqE3uxv65hBKf0VpkNLgGxd8FIvRj750 nO7ickZUiLfE5Y0mI+ot8RAg1bvzM+pU+EnQr+/RtXjGmgjqqxLTca/kFkBVNTJKhQjr XwcE78fbKjpMmELfqkY2vXpXpf6p2WAexMqJ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hPo/Bw6tNC6m8oopvfckb4iPuUPePa6aSZ/PC8jTJPc=; b=g4s4e8xgmcU9EXH83QOtFYuvJmsobqwe6tEhljhhl2U8ObYW8QI2hEEYp97xa3raT3 k+LMulGCc2wPCbxM12oAjOB2OLQok470SmpPe5F+GZ6A5TnTsTfT31wigYl1LCyXre9Y AOULjPcYDx0z82/XIGWP29k/zfL9M2jIrv8hPt/sO0cWCjlqc2VqisHtISBA556THIy2 xD3qlRDVJk3PMUxnOrKfL4gEIB7RrRcH35VnMolY8xZfy3iTQ+zUPi0M0HhyyTGKIQdw rmMT4z+hoPyGavRe9Zo2JHXozzhanFGxYppe+iIu6awtjzDAlNKqd5kztxG+K7v44pUi oA+g== X-Gm-Message-State: AOAM533SGv47Cdf3HNbHvIe/j1tvYqdMcCCvirNlZH959qKlmBNz8odd W0/20vRPUtml6s+6YCZKiMNn5g== X-Google-Smtp-Source: ABdhPJwnPss/3+FXK73SGyn1ajFye9TH2uvE+XJgRwBdhY+CfV9NESKavPXc5txtoFVhV9XkRhrz0Q== X-Received: by 2002:adf:e8d0:: with SMTP id k16mr2306084wrn.362.1603455784579; Fri, 23 Oct 2020 05:23:04 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id y4sm3056484wrp.74.2020.10.23.05.23.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Oct 2020 05:23:03 -0700 (PDT) From: Daniel Vetter To: DRI Development Subject: [PATCH 36/65] mm: Close race in generic_access_phys Date: Fri, 23 Oct 2020 14:21:47 +0200 Message-Id: <20201023122216.2373294-36-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201023122216.2373294-1-daniel.vetter@ffwll.ch> References: <20201021163242.1458885-1-daniel.vetter@ffwll.ch> <20201023122216.2373294-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201023_132309_326574_69BCDA19 X-CRM114-Status: GOOD ( 23.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-samsung-soc@vger.kernel.org, Jan Kara , Kees Cook , John Hubbard , linux-mm@kvack.org, Dave Airlie , Daniel Vetter , Intel Graphics Development , Chris Wilson , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrensmidt , Daniel Vetter , Dan Williams , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Way back it was a reasonable assumptions that iomem mappings never change the pfn range they point at. But this has changed: - gpu drivers dynamically manage their memory nowadays, invalidating ptes with unmap_mapping_range when buffers get moved - contiguous dma allocations have moved from dedicated carvetouts to cma regions. This means if we miss the unmap the pfn might contain pagecache or anon memory (well anything allocated with GFP_MOVEABLE) - even /dev/mem now invalidates mappings when the kernel requests that iomem region when CONFIG_IO_STRICT_DEVMEM is set, see 3234ac664a87 ("/dev/mem: Revoke mappings when a driver claims the region") Accessing pfns obtained from ptes without holding all the locks is therefore no longer a good idea. Fix this. Since ioremap might need to manipulate pagetables too we need to drop the pt lock and have a retry loop if we raced. While at it, also add kerneldoc and improve the comment for the vma_ops->access function. It's for accessing, not for moving the memory from iomem to system memory, as the old comment seemed to suggest. References: 28b2ee20c7cb ("access_process_vm device memory infrastructure") Signed-off-by: Daniel Vetter Cc: Jason Gunthorpe Cc: Dan Williams Cc: Kees Cook Cc: Benjamin Herrensmidt Cc: Dave Airlie Cc: Andrew Morton Cc: John Hubbard Cc: Jérôme Glisse Cc: Jan Kara Cc: Dan Williams Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-samsung-soc@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: Chris Wilson Signed-off-by: Daniel Vetter --- v2: Fix inversion in the retry check (John). v4: While at it, use offset_in_page (Chris Wilson) --- include/linux/mm.h | 3 ++- mm/memory.c | 46 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index acd60fbf1a5a..2a16631c1fda 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -566,7 +566,8 @@ struct vm_operations_struct { vm_fault_t (*pfn_mkwrite)(struct vm_fault *vmf); /* called by access_process_vm when get_user_pages() fails, typically - * for use by special VMAs that can switch between memory and hardware + * for use by special VMAs. See also generic_access_phys() for a generic + * implementation useful for any iomem mapping. */ int (*access)(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index eeae590e526a..1b46eae3b703 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4840,28 +4840,68 @@ int follow_phys(struct vm_area_struct *vma, return ret; } +/** + * generic_access_phys - generic implementation for iomem mmap access + * @vma: the vma to access + * @addr: userspace addres, not relative offset within @vma + * @buf: buffer to read/write + * @len: length of transfer + * @write: set to FOLL_WRITE when writing, otherwise reading + * + * This is a generic implementation for &vm_operations_struct.access for an + * iomem mapping. This callback is used by access_process_vm() when the @vma is + * not page based. + */ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write) { resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - int offset = addr & (PAGE_SIZE-1); + pte_t *ptep, pte; + spinlock_t *ptl; + int offset = offset_in_page(addr); + int ret = -EINVAL; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + return -EINVAL; + +retry: + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + return -EINVAL; + pte = *ptep; + pte_unmap_unlock(ptep, ptl); - if (follow_phys(vma, addr, write, &prot, &phys_addr)) + prot = pgprot_val(pte_pgprot(pte)); + phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; + + if ((write & FOLL_WRITE) && !pte_write(pte)) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; + if (follow_pte(vma->vm_mm, addr, &ptep, &ptl)) + goto out_unmap; + + if (!pte_same(pte, *ptep)) { + pte_unmap_unlock(ptep, ptl); + iounmap(maddr); + + goto retry; + } + if (write) memcpy_toio(maddr + offset, buf, len); else memcpy_fromio(buf, maddr + offset, len); + ret = len; + pte_unmap_unlock(ptep, ptl); +out_unmap: iounmap(maddr); - return len; + return ret; } EXPORT_SYMBOL_GPL(generic_access_phys); #endif