From patchwork Fri Mar 20 16:49:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3474C159A for ; Sat, 21 Mar 2020 10:45:07 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 110A2206F9 for ; Sat, 21 Mar 2020 10:45:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="LNxEfQC6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 110A2206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 19B7B6E31A; Sat, 21 Mar 2020 10:45:03 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4E33C6E198 for ; Fri, 20 Mar 2020 16:58:07 +0000 (UTC) Received: by mail-pg1-x542.google.com with SMTP id k191so2226692pgc.13 for ; Fri, 20 Mar 2020 09:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QfKkZBnQUXMbgR4f8R4xfhu6bSLBDWJchBJ45k6w/Gs=; b=LNxEfQC64b3jC76fcXKgvtSCR4fzjMtcUpvtpsK8q4LD9AtnfXoN4QNK13HeIV+anL nNLLy7NJ09/dO6aq9zF0bBZSyQAM26cPstSzeQZ8svB6cQQk8FxLUXzq7fLPQf/xinKT uofO33zYheKdUEelBMm0FujVreucBBxbigjUrJzEq7rTXHOu7y68SatzTartv+AJ9CFW ym9tIBIE8o0CYEy3G/S9V67xIWKwJ+XBNi8/ZwdhAjMlEi6RVCsM7/MX4DDvS5Urgx64 +6fbcT5BqvrROCRZUQ6bX9jEpPpYxO+g0PriTru5vss2tC9oBmFHINe8plJ7hA79gDjS DSoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QfKkZBnQUXMbgR4f8R4xfhu6bSLBDWJchBJ45k6w/Gs=; b=nX7856k5gt3IIEGfp7ya92n+RK2vKED7z1gLOExZ4kRuVkDI86NsslFnyVIUbYnrgL yv/vmhopKGyP0ahkE9k2SwJLZvmsdhoSfityhkdXmJP1JCLvQ3MVOuNcwftRhkqeRRzf EMhhPSvPKDV32qXXE+9SC38aOboIwFcOhDWqHQx/kbGUYUpGIwu9O4iKsX96+dhMfRJN ZSeW9Rv+sPolcM6GGey3Tj7XxM98isq9R9nIzUS2Cle5nrZUE1KStl4u1b+Apdz5MHLh yImtEBeZRkPiXnlObEppjxFUtnELX5TEIOt6N15JC4k/yvqnhCc8VzmIXxtnyx1L8uBu Ok4w== X-Gm-Message-State: ANhLgQ27SlAtLM4kZmMMZk4qOD2euMXqDg44InSTwGumUvXu8F/zDmcj lzbfEdxBmmE5054A59aOhGNFnlB7GnuqdQ== X-Google-Smtp-Source: ADFU+vsHZRjag+P2fuJIiFKNb4fd+uVwmgwhrUo1SYflZzrZ61HO0TNyRHDWaxWmaqgUP+ahj6sgeA== X-Received: by 2002:a05:6214:16d1:: with SMTP id d17mr9051379qvz.56.1584723025712; Fri, 20 Mar 2020 09:50:25 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id k66sm4665995qke.10.2020.03.20.09.50.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005jn-73; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 1/6] mm/hmm: remove pgmap checking for devmap pages Date: Fri, 20 Mar 2020 13:49:00 -0300 Message-Id: <20200320164905.21722-2-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:29 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe The checking boils down to some racy check if the pagemap is still available or not. Instead of checking this, rely entirely on the notifiers, if a pagemap is destroyed then all pages that belong to it must be removed from the tables and the notifiers triggered. Signed-off-by: Jason Gunthorpe --- mm/hmm.c | 50 ++------------------------------------------------ 1 file changed, 2 insertions(+), 48 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index a491d9aaafe45d..3a2610e0713329 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -28,7 +28,6 @@ struct hmm_vma_walk { struct hmm_range *range; - struct dev_pagemap *pgmap; unsigned long last; unsigned int flags; }; @@ -196,19 +195,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, fault, write_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { - if (pmd_devmap(pmd)) { - hmm_vma_walk->pgmap = get_dev_pagemap(pfn, - hmm_vma_walk->pgmap); - if (unlikely(!hmm_vma_walk->pgmap)) - return -EBUSY; - } + for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) pfns[i] = hmm_device_entry_from_pfn(range, pfn) | cpu_flags; - } - if (hmm_vma_walk->pgmap) { - put_dev_pagemap(hmm_vma_walk->pgmap); - hmm_vma_walk->pgmap = NULL; - } hmm_vma_walk->last = end; return 0; } @@ -300,15 +288,6 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, if (fault || write_fault) goto fault; - if (pte_devmap(pte)) { - hmm_vma_walk->pgmap = get_dev_pagemap(pte_pfn(pte), - hmm_vma_walk->pgmap); - if (unlikely(!hmm_vma_walk->pgmap)) { - pte_unmap(ptep); - return -EBUSY; - } - } - /* * Since each architecture defines a struct page for the zero page, just * fall through and treat it like a normal page. @@ -328,10 +307,6 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, return 0; fault: - if (hmm_vma_walk->pgmap) { - put_dev_pagemap(hmm_vma_walk->pgmap); - hmm_vma_walk->pgmap = NULL; - } pte_unmap(ptep); /* Fault any virtual address we were asked to fault */ return hmm_vma_fault(addr, end, fault, write_fault, walk); @@ -418,16 +393,6 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, return r; } } - if (hmm_vma_walk->pgmap) { - /* - * We do put_dev_pagemap() here and not in hmm_vma_handle_pte() - * so that we can leverage get_dev_pagemap() optimization which - * will not re-take a reference on a pgmap if we already have - * one. - */ - put_dev_pagemap(hmm_vma_walk->pgmap); - hmm_vma_walk->pgmap = NULL; - } pte_unmap(ptep - 1); hmm_vma_walk->last = addr; @@ -491,20 +456,9 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i = 0; i < npages; ++i, ++pfn) { - hmm_vma_walk->pgmap = get_dev_pagemap(pfn, - hmm_vma_walk->pgmap); - if (unlikely(!hmm_vma_walk->pgmap)) { - ret = -EBUSY; - goto out_unlock; - } + for (i = 0; i < npages; ++i, ++pfn) pfns[i] = hmm_device_entry_from_pfn(range, pfn) | cpu_flags; - } - if (hmm_vma_walk->pgmap) { - put_dev_pagemap(hmm_vma_walk->pgmap); - hmm_vma_walk->pgmap = NULL; - } hmm_vma_walk->last = end; goto out_unlock; } From patchwork Fri Mar 20 16:49:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2F09159A for ; Sat, 21 Mar 2020 10:44:53 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFB6B206F9 for ; Sat, 21 Mar 2020 10:44:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="o7rS5S0A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFB6B206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7ACBA6E2FF; Sat, 21 Mar 2020 10:44:43 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2F7DA6E1A3 for ; Fri, 20 Mar 2020 16:59:22 +0000 (UTC) Received: by mail-io1-xd41.google.com with SMTP id q9so6687522iod.4 for ; Fri, 20 Mar 2020 09:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BDvip0TEDdRy6Qebq3Jjm9q8IZushhhH41Cv5CSPsYg=; b=o7rS5S0AsnboKmeY0i8V6l4XrqKbMaIDN+dRT8M7ae7SN5uO0Y9cf5ue5fxCfIsNeK 7iTs5hK3BaB1JxF52ajkDDYSXgSFTdwuf6/PJzrPkUaNZxdW7j8GIbNnfDfPpx1JJ8QR iY9Bk8KNLT8H/jvnvDRYe8hfY4b8PFCjJv7Vojrcmd7ScA1lwNOpAK6ov2UvK4BihL52 DimyOsT04Yb3PYKXwsmVS/zgNQ1PtTiqXJ5fUO09wpABkfc6R/kKXb5kLhMWR0NNU65q 5pNmHCVQek+18grxw250tHfu3iPlEwtGlaO69s+IBK7W2mA6w+hcVKc1j0uQBi9jwJ37 RInQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BDvip0TEDdRy6Qebq3Jjm9q8IZushhhH41Cv5CSPsYg=; b=E85etcDxuzYnldCyENFsrFx0EeGj5o1wi6qI0w/JLkuS6g1YmxH50gtgMIdBOeNclF pFQrSHE/dqBudWG+C4OadOCifSd1MRI2tRS16pRkqXX5tWAoFrwsBkBIxOFkO8zV2gpP S1pQLPghIL9iizSHNmM8LTpvLyiCmLin4uTcvqsA0UXRFVVtdDVP+eR8QMIWLX23j6+P o0g7I/q974Atx8cr2zitkhOdlOvaliwjaXYGqLy6XUS/e0WEqnRiQTs5ttPwvk3JJUGJ IHwmQroTnBh/uTV0UTEt+RztEAEQTOaxkRlKRBMgkzURcwrBm0Zcbo7+Sfy7O0eSC9PW m0fw== X-Gm-Message-State: ANhLgQ29LlcN00QXPBiHkLgzdtRODuNAvV7ao8jGMobT8lqA7Iy1+Qeb wXzqIYcCLdC3WZwZW/X9b6VZ3KuSCXpiNw== X-Google-Smtp-Source: ADFU+vs0SluEmhvLyLkrqPiv0NLdzSfR4fL3A9Pz7jSUSruFagO/rzdeFOruZ31ExN9mYVZmWbzX6Q== X-Received: by 2002:a37:9b51:: with SMTP id d78mr9261623qke.65.1584723024806; Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id g3sm4393051qke.89.2020.03.20.09.50.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:23 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005k6-8B; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 2/6] mm/hmm: return the fault type from hmm_pte_need_fault() Date: Fri, 20 Mar 2020 13:49:01 -0300 Message-Id: <20200320164905.21722-3-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:31 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe Using two bools instead of flags return is not necessary and leads to bugs. Returning a value is easier for the compiler to check and easier to pass around the code flow. Convert the two bools into flags and push the change to all callers. Signed-off-by: Jason Gunthorpe --- mm/hmm.c | 153 ++++++++++++++++++++++++------------------------------- 1 file changed, 67 insertions(+), 86 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 3a2610e0713329..b4f662eadb7a7c 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -32,6 +32,11 @@ struct hmm_vma_walk { unsigned int flags; }; +enum { + NEED_FAULT = 1 << 0, + NEED_WRITE_FAULT = 1 << 1, +}; + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_range *range, enum hmm_pfn_value_e value) { @@ -49,8 +54,7 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s) * @addr: range virtual start address (inclusive) * @end: range virtual end address (exclusive) - * @fault: should we fault or not ? - * @write_fault: write fault ? + * @required_fault: NEED_FAULT_* flags * @walk: mm_walk structure * Return: -EBUSY after page fault, or page fault error * @@ -58,8 +62,7 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, * or whenever there is no page directory covering the virtual address range. */ static int hmm_vma_fault(unsigned long addr, unsigned long end, - bool fault, bool write_fault, - struct mm_walk *walk) + unsigned int required_fault, struct mm_walk *walk) { struct hmm_vma_walk *hmm_vma_walk = walk->private; struct hmm_range *range = hmm_vma_walk->range; @@ -68,13 +71,13 @@ static int hmm_vma_fault(unsigned long addr, unsigned long end, unsigned long i = (addr - range->start) >> PAGE_SHIFT; unsigned int fault_flags = FAULT_FLAG_REMOTE; - WARN_ON_ONCE(!fault && !write_fault); + WARN_ON_ONCE(!required_fault); hmm_vma_walk->last = addr; if (!vma) goto out_error; - if (write_fault) { + if (required_fault & NEED_WRITE_FAULT) { if (!(vma->vm_flags & VM_WRITE)) return -EPERM; fault_flags |= FAULT_FLAG_WRITE; @@ -91,14 +94,13 @@ static int hmm_vma_fault(unsigned long addr, unsigned long end, return -EFAULT; } -static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, - uint64_t pfns, uint64_t cpu_flags, - bool *fault, bool *write_fault) +static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, + uint64_t pfns, uint64_t cpu_flags) { struct hmm_range *range = hmm_vma_walk->range; if (hmm_vma_walk->flags & HMM_FAULT_SNAPSHOT) - return; + return 0; /* * So we not only consider the individual per page request we also @@ -114,37 +116,37 @@ static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, /* We aren't ask to do anything ... */ if (!(pfns & range->flags[HMM_PFN_VALID])) - return; + return 0; - /* If CPU page table is not valid then we need to fault */ - *fault = !(cpu_flags & range->flags[HMM_PFN_VALID]); /* Need to write fault ? */ if ((pfns & range->flags[HMM_PFN_WRITE]) && - !(cpu_flags & range->flags[HMM_PFN_WRITE])) { - *write_fault = true; - *fault = true; - } + !(cpu_flags & range->flags[HMM_PFN_WRITE])) + return NEED_FAULT | NEED_WRITE_FAULT; + + /* If CPU page table is not valid then we need to fault */ + if (!(cpu_flags & range->flags[HMM_PFN_VALID])) + return NEED_FAULT; + return 0; } -static void hmm_range_need_fault(const struct hmm_vma_walk *hmm_vma_walk, - const uint64_t *pfns, unsigned long npages, - uint64_t cpu_flags, bool *fault, - bool *write_fault) +static unsigned int +hmm_range_need_fault(const struct hmm_vma_walk *hmm_vma_walk, + const uint64_t *pfns, unsigned long npages, + uint64_t cpu_flags) { + unsigned int required_fault = 0; unsigned long i; - if (hmm_vma_walk->flags & HMM_FAULT_SNAPSHOT) { - *fault = *write_fault = false; - return; - } + if (hmm_vma_walk->flags & HMM_FAULT_SNAPSHOT) + return 0; - *fault = *write_fault = false; for (i = 0; i < npages; ++i) { - hmm_pte_need_fault(hmm_vma_walk, pfns[i], cpu_flags, - fault, write_fault); - if ((*write_fault)) - return; + required_fault |= + hmm_pte_need_fault(hmm_vma_walk, pfns[i], cpu_flags); + if (required_fault == (NEED_FAULT | NEED_WRITE_FAULT)) + return required_fault; } + return required_fault; } static int hmm_vma_walk_hole(unsigned long addr, unsigned long end, @@ -152,17 +154,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end, { struct hmm_vma_walk *hmm_vma_walk = walk->private; struct hmm_range *range = hmm_vma_walk->range; - bool fault, write_fault; + unsigned int required_fault; unsigned long i, npages; uint64_t *pfns; i = (addr - range->start) >> PAGE_SHIFT; npages = (end - addr) >> PAGE_SHIFT; pfns = &range->pfns[i]; - hmm_range_need_fault(hmm_vma_walk, pfns, npages, - 0, &fault, &write_fault); - if (fault || write_fault) - return hmm_vma_fault(addr, end, fault, write_fault, walk); + required_fault = hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0); + if (required_fault) + return hmm_vma_fault(addr, end, required_fault, walk); hmm_vma_walk->last = addr; return hmm_pfns_fill(addr, end, range, HMM_PFN_NONE); } @@ -183,16 +184,15 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, struct hmm_vma_walk *hmm_vma_walk = walk->private; struct hmm_range *range = hmm_vma_walk->range; unsigned long pfn, npages, i; - bool fault, write_fault; + unsigned int required_fault; uint64_t cpu_flags; npages = (end - addr) >> PAGE_SHIFT; cpu_flags = pmd_to_hmm_pfn_flags(range, pmd); - hmm_range_need_fault(hmm_vma_walk, pfns, npages, cpu_flags, - &fault, &write_fault); - - if (fault || write_fault) - return hmm_vma_fault(addr, end, fault, write_fault, walk); + required_fault = + hmm_range_need_fault(hmm_vma_walk, pfns, npages, cpu_flags); + if (required_fault) + return hmm_vma_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) @@ -229,18 +229,15 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, { struct hmm_vma_walk *hmm_vma_walk = walk->private; struct hmm_range *range = hmm_vma_walk->range; - bool fault, write_fault; + unsigned int required_fault; uint64_t cpu_flags; pte_t pte = *ptep; uint64_t orig_pfn = *pfn; *pfn = range->values[HMM_PFN_NONE]; - fault = write_fault = false; - if (pte_none(pte)) { - hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0, - &fault, &write_fault); - if (fault || write_fault) + required_fault = hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0); + if (required_fault) goto fault; return 0; } @@ -261,9 +258,8 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, return 0; } - hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0, &fault, - &write_fault); - if (!fault && !write_fault) + required_fault = hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0); + if (!required_fault) return 0; if (!non_swap_entry(entry)) @@ -283,9 +279,8 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, } cpu_flags = pte_to_hmm_pfn_flags(range, pte); - hmm_pte_need_fault(hmm_vma_walk, orig_pfn, cpu_flags, &fault, - &write_fault); - if (fault || write_fault) + required_fault = hmm_pte_need_fault(hmm_vma_walk, orig_pfn, cpu_flags); + if (required_fault) goto fault; /* @@ -293,9 +288,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, * fall through and treat it like a normal page. */ if (pte_special(pte) && !is_zero_pfn(pte_pfn(pte))) { - hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0, &fault, - &write_fault); - if (fault || write_fault) { + if (hmm_pte_need_fault(hmm_vma_walk, orig_pfn, 0)) { pte_unmap(ptep); return -EFAULT; } @@ -309,7 +302,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, fault: pte_unmap(ptep); /* Fault any virtual address we were asked to fault */ - return hmm_vma_fault(addr, end, fault, write_fault, walk); + return hmm_vma_fault(addr, end, required_fault, walk); } static int hmm_vma_walk_pmd(pmd_t *pmdp, @@ -322,7 +315,6 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, uint64_t *pfns = &range->pfns[(start - range->start) >> PAGE_SHIFT]; unsigned long npages = (end - start) >> PAGE_SHIFT; unsigned long addr = start; - bool fault, write_fault; pte_t *ptep; pmd_t pmd; @@ -332,9 +324,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, return hmm_vma_walk_hole(start, end, -1, walk); if (thp_migration_supported() && is_pmd_migration_entry(pmd)) { - hmm_range_need_fault(hmm_vma_walk, pfns, npages, - 0, &fault, &write_fault); - if (fault || write_fault) { + if (hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0)) { hmm_vma_walk->last = addr; pmd_migration_entry_wait(walk->mm, pmdp); return -EBUSY; @@ -343,9 +333,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, } if (!pmd_present(pmd)) { - hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0, &fault, - &write_fault); - if (fault || write_fault) + if (hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0)) return -EFAULT; return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); } @@ -375,9 +363,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * recover. */ if (pmd_bad(pmd)) { - hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0, &fault, - &write_fault); - if (fault || write_fault) + if (hmm_range_need_fault(hmm_vma_walk, pfns, npages, 0)) return -EFAULT; return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); } @@ -434,8 +420,8 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, if (pud_huge(pud) && pud_devmap(pud)) { unsigned long i, npages, pfn; + unsigned int required_fault; uint64_t *pfns, cpu_flags; - bool fault, write_fault; if (!pud_present(pud)) { spin_unlock(ptl); @@ -447,12 +433,11 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, pfns = &range->pfns[i]; cpu_flags = pud_to_hmm_pfn_flags(range, pud); - hmm_range_need_fault(hmm_vma_walk, pfns, npages, - cpu_flags, &fault, &write_fault); - if (fault || write_fault) { + required_fault = hmm_range_need_fault(hmm_vma_walk, pfns, + npages, cpu_flags); + if (required_fault) { spin_unlock(ptl); - return hmm_vma_fault(addr, end, fault, write_fault, - walk); + return hmm_vma_fault(addr, end, required_fault, walk); } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); @@ -484,7 +469,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, struct hmm_range *range = hmm_vma_walk->range; struct vm_area_struct *vma = walk->vma; uint64_t orig_pfn, cpu_flags; - bool fault, write_fault; + unsigned int required_fault; spinlock_t *ptl; pte_t entry; @@ -495,12 +480,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, orig_pfn = range->pfns[i]; range->pfns[i] = range->values[HMM_PFN_NONE]; cpu_flags = pte_to_hmm_pfn_flags(range, entry); - fault = write_fault = false; - hmm_pte_need_fault(hmm_vma_walk, orig_pfn, cpu_flags, - &fault, &write_fault); - if (fault || write_fault) { + required_fault = hmm_pte_need_fault(hmm_vma_walk, orig_pfn, cpu_flags); + if (required_fault) { spin_unlock(ptl); - return hmm_vma_fault(addr, end, fault, write_fault, walk); + return hmm_vma_fault(addr, end, required_fault, walk); } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); @@ -532,17 +515,15 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end, */ if ((vma->vm_flags & (VM_IO | VM_PFNMAP | VM_MIXEDMAP)) || !(vma->vm_flags & VM_READ)) { - bool fault, write_fault; - /* * Check to see if a fault is requested for any page in the * range. */ - hmm_range_need_fault(hmm_vma_walk, range->pfns + - ((start - range->start) >> PAGE_SHIFT), - (end - start) >> PAGE_SHIFT, - 0, &fault, &write_fault); - if (fault || write_fault) + if (hmm_range_need_fault(hmm_vma_walk, + range->pfns + + ((start - range->start) >> + PAGE_SHIFT), + (end - start) >> PAGE_SHIFT, 0)) return -EFAULT; hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); From patchwork Fri Mar 20 16:49:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8751D159A for ; Sat, 21 Mar 2020 10:45:43 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62F70206F9 for ; Sat, 21 Mar 2020 10:45:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="UicuVQOU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62F70206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1D9EA6E328; Sat, 21 Mar 2020 10:45:41 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5BA086E1A3 for ; Fri, 20 Mar 2020 16:59:47 +0000 (UTC) Received: by mail-io1-xd41.google.com with SMTP id q9so6688929iod.4 for ; Fri, 20 Mar 2020 09:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J9JcxoC0xHItzd7Bm+/rd4h3D6gYeCH34NADuZJ9STU=; b=UicuVQOUEOFXjcAue0sr6wmiMWeQXUmp/+3fIH5Zf7ToFdz2MJUF/GwaQV0CfokeCb R2ZYWS8t6p3i85jB4vGXhW7eauCyoXDMnLR5bqGUsHf6p7MUJiRrBD7IoIVVEIB4rwuA y7Ou48uJJJSNcpH0D02oeN5/knSWGgZa6OE2WopCjZyKOqlXD1J4bWaHUAy88Hi+g7z4 OQy9KADO+sZIv3HZ5Z5kH4FLvXskp7bNMpRHHZ5+zSKF0JlEddd3Ns59vClO3DaXjZ4u SHP/vweGSCa3pfVODg70IS9UBVRTZnSNlDoLy3DvscRGxnThtEJy7i8cblj+pDVJdQlx F1fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J9JcxoC0xHItzd7Bm+/rd4h3D6gYeCH34NADuZJ9STU=; b=X9ut8kRxj3Tz3jbNMQrhhzVAFEdtBqX3VKoi4sjBrvO7y6xD+Cqr8z3FXsc9RN/Y9j 2OsCuI8IppuhoHxkr2/EbjOhfJ+x/BFYOK99Xmhx9t42656rdl+7HJ75qqn3LTRr8v5n AAq1l3XISeIg1KkHYrYnhy1gPHh+0FdlcFLr6LLbU2At3yfYXHKd5dyEabeSrzsXO+X0 hz8K9sSZL7ynRP0Fa7Kg6fnvNvTBU+OZyXhtM2bWiYofwhRIZvCwzuJw9FbuS1JdjDbh 26F3hbe7dKmaZLO8Pd0Qhv4wE41WLxJ/3+YcWqZU9GaJaGfMhFs7zN4nK4KEHOGR1TTa V0XQ== X-Gm-Message-State: ANhLgQ2AnGDwIc3+efZnRBLHDXR7z5OxUpTypPVxYFD5WLfwQi+4AxBY RD9VHJBf2vf/Zi2Sp5rIgC00Id99Cl9uEg== X-Google-Smtp-Source: ADFU+vtuzdu+L+cKseNEMu1uY1fgXL01MDSyACtZa/2w0baVgXDZzqzudj1Yb+oXFP4S8/aXJlsNnQ== X-Received: by 2002:a37:9544:: with SMTP id x65mr8264783qkd.48.1584723025227; Fri, 20 Mar 2020 09:50:25 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id z11sm4894505qti.23.2020.03.20.09.50.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005kC-9J; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 3/6] mm/hmm: remove unused code and tidy comments Date: Fri, 20 Mar 2020 13:49:02 -0300 Message-Id: <20200320164905.21722-4-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:30 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe Delete several functions that are never called, fix some desync between comments and structure content, remove an unused ret, and move one function only used by hmm.c into hmm.c Signed-off-by: Jason Gunthorpe --- include/linux/hmm.h | 50 --------------------------------------------- mm/hmm.c | 12 +++++++++++ 2 files changed, 12 insertions(+), 50 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index bb6be4428633a8..184a8633260f9d 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -120,9 +120,6 @@ enum hmm_pfn_value_e { * * @notifier: a mmu_interval_notifier that includes the start/end * @notifier_seq: result of mmu_interval_read_begin() - * @hmm: the core HMM structure this range is active against - * @vma: the vm area struct for the range - * @list: all range lock are on a list * @start: range virtual start address (inclusive) * @end: range virtual end address (exclusive) * @pfns: array of pfns (big enough for the range) @@ -131,7 +128,6 @@ enum hmm_pfn_value_e { * @default_flags: default flags for the range (write, read, ... see hmm doc) * @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter * @pfn_shifts: pfn shift value (should be <= PAGE_SHIFT) - * @valid: pfns array did not change since it has been fill by an HMM function * @dev_private_owner: owner of device private pages */ struct hmm_range { @@ -171,52 +167,6 @@ static inline struct page *hmm_device_entry_to_page(const struct hmm_range *rang return pfn_to_page(entry >> range->pfn_shift); } -/* - * hmm_device_entry_to_pfn() - return pfn value store in a device entry - * @range: range use to decode device entry value - * @entry: device entry to extract pfn from - * Return: pfn value if device entry is valid, -1UL otherwise - */ -static inline unsigned long -hmm_device_entry_to_pfn(const struct hmm_range *range, uint64_t pfn) -{ - if (pfn == range->values[HMM_PFN_NONE]) - return -1UL; - if (pfn == range->values[HMM_PFN_ERROR]) - return -1UL; - if (pfn == range->values[HMM_PFN_SPECIAL]) - return -1UL; - if (!(pfn & range->flags[HMM_PFN_VALID])) - return -1UL; - return (pfn >> range->pfn_shift); -} - -/* - * hmm_device_entry_from_page() - create a valid device entry for a page - * @range: range use to encode HMM pfn value - * @page: page for which to create the device entry - * Return: valid device entry for the page - */ -static inline uint64_t hmm_device_entry_from_page(const struct hmm_range *range, - struct page *page) -{ - return (page_to_pfn(page) << range->pfn_shift) | - range->flags[HMM_PFN_VALID]; -} - -/* - * hmm_device_entry_from_pfn() - create a valid device entry value from pfn - * @range: range use to encode HMM pfn value - * @pfn: pfn value for which to create the device entry - * Return: valid device entry for the pfn - */ -static inline uint64_t hmm_device_entry_from_pfn(const struct hmm_range *range, - unsigned long pfn) -{ - return (pfn << range->pfn_shift) | - range->flags[HMM_PFN_VALID]; -} - /* Don't fault in missing PTEs, just snapshot the current state. */ #define HMM_FAULT_SNAPSHOT (1 << 1) diff --git a/mm/hmm.c b/mm/hmm.c index b4f662eadb7a7c..687d21c675ee60 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -37,6 +37,18 @@ enum { NEED_WRITE_FAULT = 1 << 1, }; +/* + * hmm_device_entry_from_pfn() - create a valid device entry value from pfn + * @range: range use to encode HMM pfn value + * @pfn: pfn value for which to create the device entry + * Return: valid device entry for the pfn + */ +static uint64_t hmm_device_entry_from_pfn(const struct hmm_range *range, + unsigned long pfn) +{ + return (pfn << range->pfn_shift) | range->flags[HMM_PFN_VALID]; +} + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_range *range, enum hmm_pfn_value_e value) { From patchwork Fri Mar 20 16:49:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A780159A for ; Sat, 21 Mar 2020 10:44:58 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EAF0B206F9 for ; Sat, 21 Mar 2020 10:44:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="YntWPst0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EAF0B206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F2DE86E301; Sat, 21 Mar 2020 10:44:50 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) by gabe.freedesktop.org (Postfix) with ESMTPS id BE34A6E1B6 for ; Fri, 20 Mar 2020 16:58:29 +0000 (UTC) Received: by mail-io1-xd41.google.com with SMTP id m15so1547263iob.5 for ; Fri, 20 Mar 2020 09:58:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QUocKGblQb2o2HFawCMp9Ld437bYJFvJgxJJBk7FIc4=; b=YntWPst0Bm7GqdqydwDPVPh011j6Lp9KcXvBtgIr9yGF3uuqLumDqHdiKD5YwJlwfL lW2XHPMTnoxwS2MertMZQQFfC3UCl0LR+pvNIRNPdT8RvAzcraDuwRlIHm5NffT5dD4v 5oQ3tsMUsB0uRC3JL/mPG9xXs3zOAeyxfgvbrzFcXjjaL98sR+oUX3ef5BaN8+9pJdJm BKstP+/1WLzBvBn1pq9PZ2QcfPaMVrB0tO+aSG7pYUuyBxB+8XaufyLrcNrcoMqi7TPF MrzrW0GAFjobUQICorGpUtMk9tj58wZf+SBtb9JVTMrTgmBY5/P2aVNyoFemJt+xJePd us6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QUocKGblQb2o2HFawCMp9Ld437bYJFvJgxJJBk7FIc4=; b=kGcbsooDbLlg57eR1+Froh4l3TmAN+2p7Elb0c3IZY6bABmEjmf5TSVoJFvkYfNeL6 nuWnzlsAPpXazg5Ze6xMq7TbyLJ9vuts7gqkaHLuEEupnLEnuMbFFW/oAhSHb7aLNbCZ IlYpsgD+MNAH+40+vXWcfJWLkcDVkJG9WVekd0pBgP7jCehpnVHljPcdubYXFbYI7htG 60Wc3Z/X3GTcftXBzw5kCBFHA6GSTifVnJU90aKMRvxjNMqvhJBIgDzwrvmzwQ/vgwpn 17DktHGv/RdK0rSJObPytxlEFpAa+HhKdo5TpxxGJ10mfM6L7+28yy9xg+h6/qoYurm9 ikhQ== X-Gm-Message-State: ANhLgQ2pKjb7EiipWbGJvWtf2NXCFdBYMjwz1EPEb3xuoACNuDxICZQ3 zEVAOcKty7Vf6nR+dEzbRU29XgXeK3Uwng== X-Google-Smtp-Source: ADFU+vsA7VjEQHoh/wxAGR7V+aaNfS2RnIkoq04zcGb+pjCRPYSK7O2+s9r+eM4YcA2K+jR1obAW6g== X-Received: by 2002:aed:39c9:: with SMTP id m67mr9438609qte.107.1584723026546; Fri, 20 Mar 2020 09:50:26 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id h11sm2171811qtr.38.2020.03.20.09.50.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005kI-BT; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 4/6] mm/hmm: remove HMM_FAULT_SNAPSHOT Date: Fri, 20 Mar 2020 13:49:03 -0300 Message-Id: <20200320164905.21722-5-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:29 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe Now that flags are handled on a fine-grained per-page basis this global flag is redundant and has a confusing overlap with the pfn_flags_mask and default_flags. Normalize the HMM_FAULT_SNAPSHOT behavior into one place. Callers needing the SNAPSHOT behavior should set a pfn_flags_mask and default_flags that always results in a cleared HMM_PFN_REQ_FAULT. Then no pages will be faulted, and HMM_FAULT_SNAPSHOT is not a special flow that overrides the masking mechanism. As this is the last flag, also remove the flags argument. If future flags are needed they can be part of the struct hmm_range function arguments. Signed-off-by: Jason Gunthorpe --- Documentation/vm/hmm.rst | 12 +++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +- include/linux/hmm.h | 5 +---- mm/hmm.c | 17 +++++++++-------- 5 files changed, 17 insertions(+), 21 deletions(-) diff --git a/Documentation/vm/hmm.rst b/Documentation/vm/hmm.rst index 95fec596836262..4e3e9362afeb10 100644 --- a/Documentation/vm/hmm.rst +++ b/Documentation/vm/hmm.rst @@ -161,13 +161,11 @@ device must complete the update before the driver callback returns. When the device driver wants to populate a range of virtual addresses, it can use:: - long hmm_range_fault(struct hmm_range *range, unsigned int flags); + long hmm_range_fault(struct hmm_range *range); -With the HMM_RANGE_SNAPSHOT flag, it will only fetch present CPU page table -entries and will not trigger a page fault on missing or non-present entries. -Without that flag, it does trigger a page fault on missing or read-only entries -if write access is requested (see below). Page faults use the generic mm page -fault code path just like a CPU page fault. +It will trigger a page fault on missing or read-only entries if write access is +requested (see below). Page faults use the generic mm page fault code path just +like a CPU page fault. Both functions copy CPU page table entries into their pfns array argument. Each entry in that array corresponds to an address in the virtual range. HMM @@ -197,7 +195,7 @@ The usage pattern is:: again: range.notifier_seq = mmu_interval_read_begin(&interval_sub); down_read(&mm->mmap_sem); - ret = hmm_range_fault(&range, HMM_RANGE_SNAPSHOT); + ret = hmm_range_fault(&range); if (ret) { up_read(&mm->mmap_sem); if (ret == -EBUSY) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 90821ce5e6cad0..c520290709371b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -856,7 +856,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) range->notifier_seq = mmu_interval_read_begin(&bo->notifier); down_read(&mm->mmap_sem); - r = hmm_range_fault(range, 0); + r = hmm_range_fault(range); up_read(&mm->mmap_sem); if (unlikely(r <= 0)) { /* diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c index 39c731a99937c6..e3797b2d4d1759 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -540,7 +540,7 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm, range.default_flags = 0; range.pfn_flags_mask = -1UL; down_read(&mm->mmap_sem); - ret = hmm_range_fault(&range, 0); + ret = hmm_range_fault(&range); up_read(&mm->mmap_sem); if (ret <= 0) { if (ret == 0 || ret == -EBUSY) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 184a8633260f9d..6b4004905aac89 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -167,13 +167,10 @@ static inline struct page *hmm_device_entry_to_page(const struct hmm_range *rang return pfn_to_page(entry >> range->pfn_shift); } -/* Don't fault in missing PTEs, just snapshot the current state. */ -#define HMM_FAULT_SNAPSHOT (1 << 1) - /* * Please see Documentation/vm/hmm.rst for how to use the range API. */ -long hmm_range_fault(struct hmm_range *range, unsigned int flags); +long hmm_range_fault(struct hmm_range *range); /* * HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range diff --git a/mm/hmm.c b/mm/hmm.c index 687d21c675ee60..7f77fb6e35cf78 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -29,7 +29,6 @@ struct hmm_vma_walk { struct hmm_range *range; unsigned long last; - unsigned int flags; }; enum { @@ -111,9 +110,6 @@ static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, { struct hmm_range *range = hmm_vma_walk->range; - if (hmm_vma_walk->flags & HMM_FAULT_SNAPSHOT) - return 0; - /* * So we not only consider the individual per page request we also * consider the default flags requested for the range. The API can @@ -146,10 +142,17 @@ hmm_range_need_fault(const struct hmm_vma_walk *hmm_vma_walk, const uint64_t *pfns, unsigned long npages, uint64_t cpu_flags) { + struct hmm_range *range = hmm_vma_walk->range; unsigned int required_fault = 0; unsigned long i; - if (hmm_vma_walk->flags & HMM_FAULT_SNAPSHOT) + /* + * If there is no way for valid to be set in hmm_pte_need_fault() then + * don't bother to call it. + */ + if (!(((range->flags[HMM_PFN_VALID] & range->pfn_flags_mask) | + range->default_flags) & + range->flags[HMM_PFN_VALID])) return 0; for (i = 0; i < npages; ++i) { @@ -559,7 +562,6 @@ static const struct mm_walk_ops hmm_walk_ops = { /** * hmm_range_fault - try to fault some address in a virtual address range * @range: range being faulted - * @flags: HMM_FAULT_* flags * * Return: the number of valid pages in range->pfns[] (from range start * address), which may be zero. On error one of the following status codes @@ -583,12 +585,11 @@ static const struct mm_walk_ops hmm_walk_ops = { * On error, for one virtual address in the range, the function will mark the * corresponding HMM pfn entry with an error flag. */ -long hmm_range_fault(struct hmm_range *range, unsigned int flags) +long hmm_range_fault(struct hmm_range *range) { struct hmm_vma_walk hmm_vma_walk = { .range = range, .last = range->start, - .flags = flags, }; struct mm_struct *mm = range->notifier->mm; int ret; From patchwork Fri Mar 20 16:49:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450827 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D31A014B4 for ; Sat, 21 Mar 2020 10:44:50 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ABE9A206F9 for ; Sat, 21 Mar 2020 10:44:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="Lmt+5Qnl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABE9A206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 257816E2EF; Sat, 21 Mar 2020 10:44:41 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-qv1-xf42.google.com (mail-qv1-xf42.google.com [IPv6:2607:f8b0:4864:20::f42]) by gabe.freedesktop.org (Postfix) with ESMTPS id 903A36E1A3 for ; Fri, 20 Mar 2020 16:58:58 +0000 (UTC) Received: by mail-qv1-xf42.google.com with SMTP id p19so743928qve.0 for ; Fri, 20 Mar 2020 09:58:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=plnR+AZUQA4jCFYWVdv/wzRwNH8DzG0o2FQjh9+mrVM=; b=Lmt+5Qnlb7tZ275uEF6CrSndb1nH26tPFDzh2G1CUyCkxbm+gfTR/9UE3crQdDlJQh zmDaHW+IgXVtcdbAmv4hoLUSr5lYkwBNwHpy0XvkP82YNW5ma7lMC3aBxDeZeT2kmy1U 2nAs/Opt9M0EKy6+Kh7fhaf32m2Di4KqjejRG9g0z4FnY2Pat8Htb0koubS/nf9KAPFs PSI4nUpk4CKGkqKtCWfJHuhhF+pADywPoMWjBRV3PCt6hQfR0Dgo7gdQXneMMYC7477Y ca/pQG/eWdcoGxoPCQJSbqxDMQwRysQoyM3BtxSKQP6+VSgziwwAYZeerX5zHFkfh2Hp cLXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=plnR+AZUQA4jCFYWVdv/wzRwNH8DzG0o2FQjh9+mrVM=; b=AULQ8GdutXN1Ug1OSfIbkrMo7uNRCDkn+i/35RxdtAgzUdDLEGKdgPodw3Zzm/JoC/ TZ+C2CihiRwoO9SiY8RW3tB7NPvhV4brUwuPFvM/el/o/HmlhaQX9zA2ncpnPIcxN0RY 2iFJFjzpUzk6vAGMyQyqnOF5Jgltrp84rEbJb+SL/1XaXVBIrelUJ9pk8WXcw7RBipu+ uqC2rEsaty8KtavlHR6rq+aoOWpVPr8xBTDx+lqCmdaO7nAW3u+mV3mmpgp3MhDhd2wS Z84Spbq8fLXqz0nARarsp9LIc7jYXnIPj7kVnq1uVuRaHCfGTRgirtzzTHPF97c0ZuS6 jOnQ== X-Gm-Message-State: ANhLgQ13a4aJRBV9pHDSXoNYrNMck57JzunUYVJdK+6Vv2qBji4ar26+ mDG+u9ecDG6wpoV/v4uJDMFEEK0OytLb7w== X-Google-Smtp-Source: ADFU+vtu5OS3SdeDn9uwmE+9CRkN0K3PbZf2rBY2gGT27gwYaZlGzWtAypX6nlEbkEhbBrccm2MO7w== X-Received: by 2002:a05:6214:364:: with SMTP id t4mr4400809qvu.124.1584723027047; Fri, 20 Mar 2020 09:50:27 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id 69sm4435782qki.131.2020.03.20.09.50.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005kO-Dn; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 5/6] mm/hmm: remove the CONFIG_TRANSPARENT_HUGEPAGE #ifdef Date: Fri, 20 Mar 2020 13:49:04 -0300 Message-Id: <20200320164905.21722-6-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:30 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe This code can be compiled when CONFIG_TRANSPARENT_HUGEPAGE is off, so remove the ifdef. Signed-off-by: Jason Gunthorpe --- mm/hmm.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 7f77fb6e35cf78..a09b4908e9c81a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -192,7 +192,6 @@ static inline uint64_t pmd_to_hmm_pfn_flags(struct hmm_range *range, pmd_t pmd) range->flags[HMM_PFN_VALID]; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, unsigned long end, uint64_t *pfns, pmd_t pmd) { @@ -215,11 +214,6 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, hmm_vma_walk->last = end; return 0; } -#else /* CONFIG_TRANSPARENT_HUGEPAGE */ -/* stub to allow the code below to compile */ -int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, - unsigned long end, uint64_t *pfns, pmd_t pmd); -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline bool hmm_is_device_private_entry(struct hmm_range *range, swp_entry_t entry) From patchwork Fri Mar 20 16:49:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11450825 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96CE814B4 for ; Sat, 21 Mar 2020 10:44:43 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 73AE1206F9 for ; Sat, 21 Mar 2020 10:44:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="WmQkIo8w" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73AE1206F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6B6EA6E2EA; Sat, 21 Mar 2020 10:44:35 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by gabe.freedesktop.org (Postfix) with ESMTPS id 881A26E1A3 for ; Fri, 20 Mar 2020 16:59:19 +0000 (UTC) Received: by mail-pg1-x541.google.com with SMTP id 37so3363308pgm.11 for ; Fri, 20 Mar 2020 09:59:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xRjX7T7eft/qC+6bf7FfyU/mPCvn6DuztC5RzuHCyVc=; b=WmQkIo8wgTcT+49UEOC761MjUtOoLumkxcQrNLGTV5XIF/8lT1bx+3pCSL8T9xMdZY cWLs8vQSQKBV1VXMxQM5NR78QAl2OuRSqAVNbbPsuMvJzg/GjnWpmqSoglfFIAkV31+z MP+DUTjrCLJ6J7p5c6jvdGCaDAE5JuorkDPWMCl3cc8f1/br3nbePeINlV2no2Zpv0lA x996mvxPsYAJubp73Du4S2nS/wtnN/9QxMhJQCWR+gg4WbHEtK82IFnTkGk0BMjclTq/ UasFQVqNKpkfmX/un/MPsIzZfDYZXMiTwx9ovm8X8Wk5Xzm8VVcGiy4D9LEPu/vagLD/ G03A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xRjX7T7eft/qC+6bf7FfyU/mPCvn6DuztC5RzuHCyVc=; b=XGz9t+SU0wNSe6p6ZrqkyesXl2lnLqEr+btPyprBzxNHIP5V0hrEGFhZHgHCjsSK+l LfW/NTEHO6ZZo3xqBQHxm+XlI8kIu0EbcqYrE5efvqg4PjwBrjnKVsvv+TKrZ1jGC0V8 GvLAb7C5j+E+jOGCvs6KnHJ7UrvbacvzT3TgRd/u740oxsHj4eVE902VHZw+jFngDi36 c2gxAkWfiSmw7HD73g4CsVKb15EMXBrH0mSHRj4tApaf8Ap3WJ1wzEnb0HkDFjsreLu1 v4ZfuiSwAaREsSlIe+JQ/FC35LM2lyo7V6PPaSz0KxHL47ZSb3LW8XaleKrgPCGUIaXx j67w== X-Gm-Message-State: ANhLgQ0orgzMmQ/qhuElRLXXEUThzschUz8QPRXQKH/iXBGupsQ/KsIf QVXRxTW2PVGWGpC089xbiXk3CrQehTihww== X-Google-Smtp-Source: ADFU+vtbRR/MYq87zJZvXdzKsxYqiR4x6njtZJXIu9qf6nhgSPf391EA27m+Q0Bt/DeLj3EN/58FoA== X-Received: by 2002:a05:6214:12c7:: with SMTP id s7mr9363766qvv.218.1584723027380; Fri, 20 Mar 2020 09:50:27 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-57-212.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.57.212]) by smtp.gmail.com with ESMTPSA id e2sm4452824qkg.63.2020.03.20.09.50.24 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 20 Mar 2020 09:50:24 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jFKql-0005kU-FT; Fri, 20 Mar 2020 13:50:23 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , Felix.Kuehling@amd.com Subject: [PATCH hmm 6/6] mm/hmm: use device_private_entry_to_pfn() Date: Fri, 20 Mar 2020 13:49:05 -0300 Message-Id: <20200320164905.21722-7-jgg@ziepe.ca> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200320164905.21722-1-jgg@ziepe.ca> References: <20200320164905.21722-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 21 Mar 2020 10:44:29 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang , John Hubbard , amd-gfx@lists.freedesktop.org, linux-mm@kvack.org, Jason Gunthorpe , dri-devel@lists.freedesktop.org, Christoph Hellwig Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jason Gunthorpe swp_offset() should not be called directly, the wrappers are supposed to abstract away the encoding of the device_private specific information in the swap entry. Signed-off-by: Jason Gunthorpe --- mm/hmm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index a09b4908e9c81a..fd9ee2b5fd9989 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -259,8 +259,8 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, * the PFN even if not present. */ if (hmm_is_device_private_entry(range, entry)) { - *pfn = hmm_device_entry_from_pfn(range, - swp_offset(entry)); + *pfn = hmm_device_entry_from_pfn( + range, device_private_entry_to_pfn(entry)); *pfn |= range->flags[HMM_PFN_VALID]; if (is_write_device_private_entry(entry)) *pfn |= range->flags[HMM_PFN_WRITE];