From patchwork Fri Jul 5 14:29:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mika Kuoppala X-Patchwork-Id: 11032749 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C24D3138D for ; Fri, 5 Jul 2019 14:29:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B08B82896F for ; Fri, 5 Jul 2019 14:29:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A32EA28A81; Fri, 5 Jul 2019 14:29:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F2AE02896F for ; Fri, 5 Jul 2019 14:29:31 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5C2796E4DD; Fri, 5 Jul 2019 14:29:30 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by gabe.freedesktop.org (Postfix) with ESMTPS id C47FC6E4D2 for ; Fri, 5 Jul 2019 14:29:26 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2019 07:29:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,455,1557212400"; d="scan'208";a="363191497" Received: from rosetta.fi.intel.com ([10.237.72.194]) by fmsmga005.fm.intel.com with ESMTP; 05 Jul 2019 07:29:25 -0700 Received: by rosetta.fi.intel.com (Postfix, from userid 1000) id 71BA584028D; Fri, 5 Jul 2019 17:29:09 +0300 (EEST) From: Mika Kuoppala To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jul 2019 17:29:05 +0300 Message-Id: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> X-Mailer: git-send-email 2.17.1 Subject: [Intel-gfx] [PATCH 1/4] drm/i915/gtt: pde entry encoding is identical X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matthew Auld MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP For all page directory entries, the pde encoding is identical. Don't compilicate call sites with different versions of doing the same thing. We check the existence of physical page before writing the entry into it. This further generalizes the pd so that manipulation in callsites will be identical, removing the need to handle pdps differently for gen8. v2: squash v3: inc/dec with set/clear (Chris) v4: inlines, warn, stray set_pd (Chris) Cc: Chris Wilson Cc: Matthew Auld Signed-off-by: Mika Kuoppala Reviewed-by: Chris Wilson --- drivers/gpu/drm/i915/i915_gem_gtt.c | 152 ++++++++++++---------------- drivers/gpu/drm/i915/i915_gem_gtt.h | 3 - 2 files changed, 63 insertions(+), 92 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index 57db2d7270c5..adf6eadd5009 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -211,10 +211,10 @@ static u64 gen8_pte_encode(dma_addr_t addr, return pte; } -static gen8_pde_t gen8_pde_encode(const dma_addr_t addr, - const enum i915_cache_level level) +static u64 gen8_pde_encode(const dma_addr_t addr, + const enum i915_cache_level level) { - gen8_pde_t pde = _PAGE_PRESENT | _PAGE_RW; + u64 pde = _PAGE_PRESENT | _PAGE_RW; pde |= addr; if (level != I915_CACHE_NONE) pde |= PPAT_CACHED_PDE; @@ -223,9 +223,6 @@ static gen8_pde_t gen8_pde_encode(const dma_addr_t addr, return pde; } -#define gen8_pdpe_encode gen8_pde_encode -#define gen8_pml4e_encode gen8_pde_encode - static u64 snb_pte_encode(dma_addr_t addr, enum i915_cache_level level, u32 flags) @@ -777,24 +774,55 @@ static void free_pd(struct i915_address_space *vm, kfree(pd); } -static void init_pd_with_page(struct i915_address_space *vm, - struct i915_page_directory * const pd, - struct i915_page_table *pt) +#define init_pd(vm, pd, to) { \ + GEM_DEBUG_BUG_ON(!pd_has_phys_page(pd)); \ + fill_px((vm), (pd), gen8_pde_encode(px_dma(to), I915_CACHE_LLC)); \ + memset_p((pd)->entry, (to), 512); \ +} + +static inline void +write_dma_entry(struct i915_page_dma * const pdma, + const unsigned short pde, + const u64 encoded_entry) +{ + u64 * const vaddr = kmap_atomic(pdma->page); + + vaddr[pde] = encoded_entry; + kunmap_atomic(vaddr); +} + +static inline void +__set_pd_entry(struct i915_page_directory * const pd, + const unsigned short pde, + struct i915_page_dma * const to, + u64 (*encode)(const dma_addr_t, const enum i915_cache_level)) { - fill_px(vm, pd, gen8_pde_encode(px_dma(pt), I915_CACHE_LLC)); - memset_p(pd->entry, pt, 512); + GEM_BUG_ON(atomic_read(&pd->used) > 512); + + atomic_inc(&pd->used); + pd->entry[pde] = to; + write_dma_entry(px_base(pd), pde, encode(to->daddr, I915_CACHE_LLC)); } -static void init_pd(struct i915_address_space *vm, - struct i915_page_directory * const pd, - struct i915_page_directory * const to) +static inline void +__clear_pd_entry(struct i915_page_directory * const pd, + const unsigned short pde, + struct i915_page_dma * const to, + u64 (*encode)(const dma_addr_t, const enum i915_cache_level)) { - GEM_DEBUG_BUG_ON(!pd_has_phys_page(pd)); + GEM_BUG_ON(atomic_read(&pd->used) == 0); - fill_px(vm, pd, gen8_pdpe_encode(px_dma(to), I915_CACHE_LLC)); - memset_p(pd->entry, to, 512); + write_dma_entry(px_base(pd), pde, encode(to->daddr, I915_CACHE_LLC)); + pd->entry[pde] = to; + atomic_dec(&pd->used); } +#define set_pd_entry(pd, pde, to) \ + __set_pd_entry((pd), (pde), px_base(to), gen8_pde_encode) + +#define clear_pd_entry(pd, pde, to) \ + __clear_pd_entry((pd), (pde), px_base(to), gen8_pde_encode) + /* * PDE TLBs are a pain to invalidate on GEN8+. When we modify * the page table structures, we mark them dirty so that @@ -824,18 +852,6 @@ static bool gen8_ppgtt_clear_pt(const struct i915_address_space *vm, return !atomic_sub_return(num_entries, &pt->used); } -static void gen8_ppgtt_set_pde(struct i915_address_space *vm, - struct i915_page_directory *pd, - struct i915_page_table *pt, - unsigned int pde) -{ - gen8_pde_t *vaddr; - - vaddr = kmap_atomic_px(pd); - vaddr[pde] = gen8_pde_encode(px_dma(pt), I915_CACHE_LLC); - kunmap_atomic(vaddr); -} - static bool gen8_ppgtt_clear_pd(struct i915_address_space *vm, struct i915_page_directory *pd, u64 start, u64 length) @@ -853,11 +869,7 @@ static bool gen8_ppgtt_clear_pd(struct i915_address_space *vm, spin_lock(&pd->lock); if (!atomic_read(&pt->used)) { - gen8_ppgtt_set_pde(vm, pd, vm->scratch_pt, pde); - pd->entry[pde] = vm->scratch_pt; - - GEM_BUG_ON(!atomic_read(&pd->used)); - atomic_dec(&pd->used); + clear_pd_entry(pd, pde, vm->scratch_pt); free = true; } spin_unlock(&pd->lock); @@ -868,20 +880,6 @@ static bool gen8_ppgtt_clear_pd(struct i915_address_space *vm, return !atomic_read(&pd->used); } -static void gen8_ppgtt_set_pdpe(struct i915_page_directory *pdp, - struct i915_page_directory *pd, - unsigned int pdpe) -{ - gen8_ppgtt_pdpe_t *vaddr; - - if (!pd_has_phys_page(pdp)) - return; - - vaddr = kmap_atomic_px(pdp); - vaddr[pdpe] = gen8_pdpe_encode(px_dma(pd), I915_CACHE_LLC); - kunmap_atomic(vaddr); -} - /* Removes entries from a single page dir pointer, releasing it if it's empty. * Caller can use the return value to update higher-level entries */ @@ -902,11 +900,7 @@ static bool gen8_ppgtt_clear_pdp(struct i915_address_space *vm, spin_lock(&pdp->lock); if (!atomic_read(&pd->used)) { - gen8_ppgtt_set_pdpe(pdp, vm->scratch_pd, pdpe); - pdp->entry[pdpe] = vm->scratch_pd; - - GEM_BUG_ON(!atomic_read(&pdp->used)); - atomic_dec(&pdp->used); + clear_pd_entry(pdp, pdpe, vm->scratch_pd); free = true; } spin_unlock(&pdp->lock); @@ -923,17 +917,6 @@ static void gen8_ppgtt_clear_3lvl(struct i915_address_space *vm, gen8_ppgtt_clear_pdp(vm, i915_vm_to_ppgtt(vm)->pd, start, length); } -static void gen8_ppgtt_set_pml4e(struct i915_page_directory *pml4, - struct i915_page_directory *pdp, - unsigned int pml4e) -{ - gen8_ppgtt_pml4e_t *vaddr; - - vaddr = kmap_atomic_px(pml4); - vaddr[pml4e] = gen8_pml4e_encode(px_dma(pdp), I915_CACHE_LLC); - kunmap_atomic(vaddr); -} - /* Removes entries from a single pml4. * This is the top-level structure in 4-level page tables used on gen8+. * Empty entries are always scratch pml4e. @@ -957,8 +940,7 @@ static void gen8_ppgtt_clear_4lvl(struct i915_address_space *vm, spin_lock(&pml4->lock); if (!atomic_read(&pdp->used)) { - gen8_ppgtt_set_pml4e(pml4, vm->scratch_pdp, pml4e); - pml4->entry[pml4e] = vm->scratch_pdp; + clear_pd_entry(pml4, pml4e, vm->scratch_pdp); free = true; } spin_unlock(&pml4->lock); @@ -1275,7 +1257,7 @@ static int gen8_init_scratch(struct i915_address_space *vm) } gen8_initialize_pt(vm, vm->scratch_pt); - init_pd_with_page(vm, vm->scratch_pd, vm->scratch_pt); + init_pd(vm, vm->scratch_pd, vm->scratch_pt); if (i915_vm_is_4lvl(vm)) init_pd(vm, vm->scratch_pdp, vm->scratch_pd); @@ -1298,6 +1280,11 @@ static int gen8_ppgtt_notify_vgt(struct i915_ppgtt *ppgtt, bool create) enum vgt_g2v_type msg; int i; + if (create) + atomic_inc(&ppgtt->pd->used); /* never remove */ + else + atomic_dec(&ppgtt->pd->used); + if (i915_vm_is_4lvl(vm)) { const u64 daddr = px_dma(ppgtt->pd); @@ -1414,9 +1401,7 @@ static int gen8_ppgtt_alloc_pd(struct i915_address_space *vm, spin_lock(&pd->lock); if (pd->entry[pde] == vm->scratch_pt) { - gen8_ppgtt_set_pde(vm, pd, pt, pde); - pd->entry[pde] = pt; - atomic_inc(&pd->used); + set_pd_entry(pd, pde, pt); } else { alloc = pt; pt = pd->entry[pde]; @@ -1458,13 +1443,11 @@ static int gen8_ppgtt_alloc_pdp(struct i915_address_space *vm, goto unwind; } - init_pd_with_page(vm, pd, vm->scratch_pt); + init_pd(vm, pd, vm->scratch_pt); spin_lock(&pdp->lock); if (pdp->entry[pdpe] == vm->scratch_pd) { - gen8_ppgtt_set_pdpe(pdp, pd, pdpe); - pdp->entry[pdpe] = pd; - atomic_inc(&pdp->used); + set_pd_entry(pdp, pdpe, pd); } else { alloc = pd; pd = pdp->entry[pdpe]; @@ -1490,12 +1473,9 @@ static int gen8_ppgtt_alloc_pdp(struct i915_address_space *vm, } spin_lock(&pdp->lock); if (atomic_dec_and_test(&pd->used)) { - gen8_ppgtt_set_pdpe(pdp, vm->scratch_pd, pdpe); - pdp->entry[pdpe] = vm->scratch_pd; - GEM_BUG_ON(!atomic_read(&pdp->used)); - atomic_dec(&pdp->used); GEM_BUG_ON(alloc); alloc = pd; /* defer the free to after the lock */ + clear_pd_entry(pdp, pdpe, vm->scratch_pd); } spin_unlock(&pdp->lock); unwind: @@ -1540,8 +1520,7 @@ static int gen8_ppgtt_alloc_4lvl(struct i915_address_space *vm, spin_lock(&pml4->lock); if (pml4->entry[pml4e] == vm->scratch_pdp) { - gen8_ppgtt_set_pml4e(pml4, pdp, pml4e); - pml4->entry[pml4e] = pdp; + set_pd_entry(pml4, pml4e, pdp); } else { alloc = pdp; pdp = pml4->entry[pml4e]; @@ -1567,10 +1546,9 @@ static int gen8_ppgtt_alloc_4lvl(struct i915_address_space *vm, } spin_lock(&pml4->lock); if (atomic_dec_and_test(&pdp->used)) { - gen8_ppgtt_set_pml4e(pml4, vm->scratch_pdp, pml4e); - pml4->entry[pml4e] = vm->scratch_pdp; GEM_BUG_ON(alloc); alloc = pdp; /* defer the free until after the lock */ + clear_pd_entry(pml4, pml4e, vm->scratch_pdp); } spin_unlock(&pml4->lock); unwind: @@ -1595,20 +1573,16 @@ static int gen8_preallocate_top_level_pdp(struct i915_ppgtt *ppgtt) if (IS_ERR(pd)) goto unwind; - init_pd_with_page(vm, pd, vm->scratch_pt); - gen8_ppgtt_set_pdpe(pdp, pd, pdpe); - - atomic_inc(&pdp->used); + init_pd(vm, pd, vm->scratch_pt); + set_pd_entry(pdp, pdpe, pd); } - atomic_inc(&pdp->used); /* never remove */ - return 0; unwind: start -= from; gen8_for_each_pdpe(pd, pdp, from, start, pdpe) { - gen8_ppgtt_set_pdpe(pdp, vm->scratch_pd, pdpe); + clear_pd_entry(pdp, pdpe, vm->scratch_pd); free_pd(vm, pd); } atomic_set(&pdp->used, 0); diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h index d0e0905acbbb..57a68ef4eda7 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.h +++ b/drivers/gpu/drm/i915/i915_gem_gtt.h @@ -72,9 +72,6 @@ struct intel_gt; typedef u32 gen6_pte_t; typedef u64 gen8_pte_t; -typedef u64 gen8_pde_t; -typedef u64 gen8_ppgtt_pdpe_t; -typedef u64 gen8_ppgtt_pml4e_t; #define ggtt_total_entries(ggtt) ((ggtt)->vm.total >> PAGE_SHIFT) From patchwork Fri Jul 5 14:29:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mika Kuoppala X-Patchwork-Id: 11032745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 02E4F13B1 for ; Fri, 5 Jul 2019 14:29:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E67EC2896F for ; Fri, 5 Jul 2019 14:29:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA48128A81; Fri, 5 Jul 2019 14:29:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 83B642896F for ; Fri, 5 Jul 2019 14:29:29 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2B9476E4D2; Fri, 5 Jul 2019 14:29:28 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 854326E4D2 for ; Fri, 5 Jul 2019 14:29:26 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2019 07:29:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,455,1557212400"; d="scan'208";a="167066832" Received: from rosetta.fi.intel.com ([10.237.72.194]) by orsmga003.jf.intel.com with ESMTP; 05 Jul 2019 07:29:25 -0700 Received: by rosetta.fi.intel.com (Postfix, from userid 1000) id 747EB840181; Fri, 5 Jul 2019 17:29:09 +0300 (EEST) From: Mika Kuoppala To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jul 2019 17:29:06 +0300 Message-Id: <20190705142908.4921-2-mika.kuoppala@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> References: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> Subject: [Intel-gfx] [PATCH 2/4] drm/i915/gtt: Tear down setup and cleanup macros for page dma X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP We don't use common codepaths to setup and cleanup page directories vs page tables. So their setup and cleanup macros are of no use. Signed-off-by: Mika Kuoppala Reviewed-by: Chris Wilson --- drivers/gpu/drm/i915/i915_gem_gtt.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index adf6eadd5009..f011ce1ae03a 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -594,8 +594,6 @@ static void cleanup_page_dma(struct i915_address_space *vm, #define kmap_atomic_px(px) kmap_atomic(px_base(px)->page) -#define setup_px(vm, px) setup_page_dma((vm), px_base(px)) -#define cleanup_px(vm, px) cleanup_page_dma((vm), px_base(px)) #define fill_px(vm, px, v) fill_page_dma((vm), px_base(px), (v)) #define fill32_px(vm, px, v) fill_page_dma_32((vm), px_base(px), (v)) @@ -697,7 +695,7 @@ static struct i915_page_table *alloc_pt(struct i915_address_space *vm) if (unlikely(!pt)) return ERR_PTR(-ENOMEM); - if (unlikely(setup_px(vm, pt))) { + if (unlikely(setup_page_dma(vm, &pt->base))) { kfree(pt); return ERR_PTR(-ENOMEM); } @@ -709,7 +707,7 @@ static struct i915_page_table *alloc_pt(struct i915_address_space *vm) static void free_pt(struct i915_address_space *vm, struct i915_page_table *pt) { - cleanup_px(vm, pt); + cleanup_page_dma(vm, &pt->base); kfree(pt); } @@ -752,7 +750,7 @@ static struct i915_page_directory *alloc_pd(struct i915_address_space *vm) if (unlikely(!pd)) return ERR_PTR(-ENOMEM); - if (unlikely(setup_px(vm, pd))) { + if (unlikely(setup_page_dma(vm, &pd->base))) { kfree(pd); return ERR_PTR(-ENOMEM); } @@ -769,7 +767,7 @@ static void free_pd(struct i915_address_space *vm, struct i915_page_directory *pd) { if (likely(pd_has_phys_page(pd))) - cleanup_px(vm, pd); + cleanup_page_dma(vm, &pd->base); kfree(pd); } @@ -1649,7 +1647,7 @@ static struct i915_ppgtt *gen8_ppgtt_create(struct drm_i915_private *i915) } if (i915_vm_is_4lvl(&ppgtt->vm)) { - err = setup_px(&ppgtt->vm, ppgtt->pd); + err = setup_page_dma(&ppgtt->vm, &ppgtt->pd->base); if (err) goto err_free_pdp; From patchwork Fri Jul 5 14:29:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mika Kuoppala X-Patchwork-Id: 11032743 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93A8413B1 for ; Fri, 5 Jul 2019 14:29:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EB802896F for ; Fri, 5 Jul 2019 14:29:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 700B528A81; Fri, 5 Jul 2019 14:29:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0C9112896F for ; Fri, 5 Jul 2019 14:29:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2A20A6E4CE; Fri, 5 Jul 2019 14:29:27 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7E66C6E4CE for ; Fri, 5 Jul 2019 14:29:26 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2019 07:29:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,455,1557212400"; d="scan'208";a="166549184" Received: from rosetta.fi.intel.com ([10.237.72.194]) by fmsmga007.fm.intel.com with ESMTP; 05 Jul 2019 07:29:25 -0700 Received: by rosetta.fi.intel.com (Postfix, from userid 1000) id 772408402D2; Fri, 5 Jul 2019 17:29:09 +0300 (EEST) From: Mika Kuoppala To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jul 2019 17:29:07 +0300 Message-Id: <20190705142908.4921-3-mika.kuoppala@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> References: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> Subject: [Intel-gfx] [PATCH 3/4] drm/i915/gtt: Setup phys pages for 3lvl pdps X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP If we setup backing phys page for 3lvl pdps, even they are not used, we lose 5 pages per ppgtt. Trading this memory on bsw, we gain more common code paths for all gen8+ directory manipulation. And those paths are now void of checks for page directory type, making the hot paths faster. v2: don't shortcut vm (Chris) Signed-off-by: Mika Kuoppala --- drivers/gpu/drm/i915/i915_gem_gtt.c | 77 +++++++++++++++++++---------- 1 file changed, 50 insertions(+), 27 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index f011ce1ae03a..0a55b0932c86 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -758,22 +758,14 @@ static struct i915_page_directory *alloc_pd(struct i915_address_space *vm) return pd; } -static inline bool pd_has_phys_page(const struct i915_page_directory * const pd) -{ - return pd->base.page; -} - static void free_pd(struct i915_address_space *vm, struct i915_page_directory *pd) { - if (likely(pd_has_phys_page(pd))) - cleanup_page_dma(vm, &pd->base); - + cleanup_page_dma(vm, &pd->base); kfree(pd); } #define init_pd(vm, pd, to) { \ - GEM_DEBUG_BUG_ON(!pd_has_phys_page(pd)); \ fill_px((vm), (pd), gen8_pde_encode(px_dma(to), I915_CACHE_LLC)); \ memset_p((pd)->entry, (to), 512); \ } @@ -1604,6 +1596,50 @@ static void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt) ppgtt->vm.vma_ops.clear_pages = clear_pages; } +static void init_pd_n(struct i915_address_space *vm, + struct i915_page_directory *pd, + struct i915_page_directory *to, + const unsigned int entries) +{ + const u64 daddr = gen8_pde_encode(px_dma(to), I915_CACHE_LLC); + u64 * const vaddr = kmap_atomic(pd->base.page); + + memset64(vaddr, daddr, entries); + kunmap_atomic(vaddr); + + memset_p(pd->entry, to, entries); +} + +static struct i915_page_directory * +gen8_alloc_top_pd(struct i915_address_space *vm) +{ + struct i915_page_directory *pd; + + if (i915_vm_is_4lvl(vm)) { + pd = alloc_pd(vm); + if (!IS_ERR(pd)) + init_pd(vm, pd, vm->scratch_pdp); + + return pd; + } + + /* 3lvl */ + pd = __alloc_pd(); + if (!pd) + return ERR_PTR(-ENOMEM); + + pd->entry[GEN8_3LVL_PDPES] = NULL; + + if (unlikely(setup_page_dma(vm, &pd->base))) { + kfree(pd); + return ERR_PTR(-ENOMEM); + } + + init_pd_n(vm, pd, vm->scratch_pd, GEN8_3LVL_PDPES); + + return pd; +} + /* * GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers * with a net effect resembling a 2-level page table in normal x86 terms. Each @@ -1640,34 +1676,21 @@ static struct i915_ppgtt *gen8_ppgtt_create(struct drm_i915_private *i915) if (err) goto err_free; - ppgtt->pd = __alloc_pd(); - if (!ppgtt->pd) { - err = -ENOMEM; + ppgtt->pd = gen8_alloc_top_pd(&ppgtt->vm); + if (IS_ERR(ppgtt->pd)) { + err = PTR_ERR(ppgtt->pd); goto err_free_scratch; } if (i915_vm_is_4lvl(&ppgtt->vm)) { - err = setup_page_dma(&ppgtt->vm, &ppgtt->pd->base); - if (err) - goto err_free_pdp; - - init_pd(&ppgtt->vm, ppgtt->pd, ppgtt->vm.scratch_pdp); - ppgtt->vm.allocate_va_range = gen8_ppgtt_alloc_4lvl; ppgtt->vm.insert_entries = gen8_ppgtt_insert_4lvl; ppgtt->vm.clear_range = gen8_ppgtt_clear_4lvl; } else { - /* - * We don't need to setup dma for top level pdp, only - * for entries. So point entries to scratch. - */ - memset_p(ppgtt->pd->entry, ppgtt->vm.scratch_pd, - GEN8_3LVL_PDPES); - if (intel_vgpu_active(i915)) { err = gen8_preallocate_top_level_pdp(ppgtt); if (err) - goto err_free_pdp; + goto err_free_pd; } ppgtt->vm.allocate_va_range = gen8_ppgtt_alloc_3lvl; @@ -1682,7 +1705,7 @@ static struct i915_ppgtt *gen8_ppgtt_create(struct drm_i915_private *i915) return ppgtt; -err_free_pdp: +err_free_pd: free_pd(&ppgtt->vm, ppgtt->pd); err_free_scratch: gen8_free_scratch(&ppgtt->vm); From patchwork Fri Jul 5 14:29:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mika Kuoppala X-Patchwork-Id: 11032747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9B91138D for ; Fri, 5 Jul 2019 14:29:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 999452896F for ; Fri, 5 Jul 2019 14:29:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8DEBB28A81; Fri, 5 Jul 2019 14:29:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 16F2B2896F for ; Fri, 5 Jul 2019 14:29:30 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 146EA6E4D6; Fri, 5 Jul 2019 14:29:30 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by gabe.freedesktop.org (Postfix) with ESMTPS id ADCB66E4CE for ; Fri, 5 Jul 2019 14:29:26 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2019 07:29:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,455,1557212400"; d="scan'208";a="363191491" Received: from rosetta.fi.intel.com ([10.237.72.194]) by fmsmga005.fm.intel.com with ESMTP; 05 Jul 2019 07:29:25 -0700 Received: by rosetta.fi.intel.com (Postfix, from userid 1000) id 79F5B840501; Fri, 5 Jul 2019 17:29:09 +0300 (EEST) From: Mika Kuoppala To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jul 2019 17:29:08 +0300 Message-Id: <20190705142908.4921-4-mika.kuoppala@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> References: <20190705142908.4921-1-mika.kuoppala@linux.intel.com> Subject: [Intel-gfx] [PATCH 4/4] drm/i915/gtt: Introduce release_pd_entry X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP By encapsulating the locking upper level and used check for entry into a helper function, we can use it in all callsites. v2: get rid of atomic_reads on lower level clears (Chris) Cc: Chris Wilson Signed-off-by: Mika Kuoppala --- drivers/gpu/drm/i915/i915_gem_gtt.c | 125 +++++++++++++--------------- 1 file changed, 60 insertions(+), 65 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index 0a55b0932c86..3d9612c776dc 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -813,6 +813,42 @@ __clear_pd_entry(struct i915_page_directory * const pd, #define clear_pd_entry(pd, pde, to) \ __clear_pd_entry((pd), (pde), px_base(to), gen8_pde_encode) +static bool +release_pt_entry(struct i915_page_directory * const pd, + const unsigned short pde, + struct i915_page_table * const pt, + struct i915_page_table * const scratch) +{ + bool free = false; + + spin_lock(&pd->lock); + if (atomic_dec_and_test(&pt->used)) { + clear_pd_entry(pd, pde, scratch); + free = true; + } + spin_unlock(&pd->lock); + + return free; +} + +static bool +release_pd_entry(struct i915_page_directory * const pd, + const unsigned short pde, + struct i915_page_directory * const entry, + struct i915_page_directory * const scratch) +{ + bool free = false; + + spin_lock(&pd->lock); + if (atomic_dec_and_test(&entry->used)) { + clear_pd_entry(pd, pde, scratch); + free = true; + } + spin_unlock(&pd->lock); + + return free; +} + /* * PDE TLBs are a pain to invalidate on GEN8+. When we modify * the page table structures, we mark them dirty so that @@ -827,11 +863,11 @@ static void mark_tlbs_dirty(struct i915_ppgtt *ppgtt) /* Removes entries from a single page table, releasing it if it's empty. * Caller can use the return value to update higher-level entries. */ -static bool gen8_ppgtt_clear_pt(const struct i915_address_space *vm, +static void gen8_ppgtt_clear_pt(const struct i915_address_space *vm, struct i915_page_table *pt, u64 start, u64 length) { - unsigned int num_entries = gen8_pte_count(start, length); + const unsigned int num_entries = gen8_pte_count(start, length); gen8_pte_t *vaddr; vaddr = kmap_atomic_px(pt); @@ -839,10 +875,11 @@ static bool gen8_ppgtt_clear_pt(const struct i915_address_space *vm, kunmap_atomic(vaddr); GEM_BUG_ON(num_entries > atomic_read(&pt->used)); - return !atomic_sub_return(num_entries, &pt->used); + + atomic_sub(num_entries, &pt->used); } -static bool gen8_ppgtt_clear_pd(struct i915_address_space *vm, +static void gen8_ppgtt_clear_pd(struct i915_address_space *vm, struct i915_page_directory *pd, u64 start, u64 length) { @@ -850,30 +887,21 @@ static bool gen8_ppgtt_clear_pd(struct i915_address_space *vm, u32 pde; gen8_for_each_pde(pt, pd, start, length, pde) { - bool free = false; - GEM_BUG_ON(pt == vm->scratch_pt); - if (!gen8_ppgtt_clear_pt(vm, pt, start, length)) - continue; + atomic_inc(&pt->used); - spin_lock(&pd->lock); - if (!atomic_read(&pt->used)) { - clear_pd_entry(pd, pde, vm->scratch_pt); - free = true; - } - spin_unlock(&pd->lock); - if (free) + gen8_ppgtt_clear_pt(vm, pt, start, length); + + if (release_pt_entry(pd, pde, pt, vm->scratch_pt)) free_pt(vm, pt); } - - return !atomic_read(&pd->used); } /* Removes entries from a single page dir pointer, releasing it if it's empty. * Caller can use the return value to update higher-level entries */ -static bool gen8_ppgtt_clear_pdp(struct i915_address_space *vm, +static void gen8_ppgtt_clear_pdp(struct i915_address_space *vm, struct i915_page_directory * const pdp, u64 start, u64 length) { @@ -881,24 +909,15 @@ static bool gen8_ppgtt_clear_pdp(struct i915_address_space *vm, unsigned int pdpe; gen8_for_each_pdpe(pd, pdp, start, length, pdpe) { - bool free = false; - GEM_BUG_ON(pd == vm->scratch_pd); - if (!gen8_ppgtt_clear_pd(vm, pd, start, length)) - continue; + atomic_inc(&pd->used); - spin_lock(&pdp->lock); - if (!atomic_read(&pd->used)) { - clear_pd_entry(pdp, pdpe, vm->scratch_pd); - free = true; - } - spin_unlock(&pdp->lock); - if (free) + gen8_ppgtt_clear_pd(vm, pd, start, length); + + if (release_pd_entry(pdp, pdpe, pd, vm->scratch_pd)) free_pd(vm, pd); } - - return !atomic_read(&pdp->used); } static void gen8_ppgtt_clear_3lvl(struct i915_address_space *vm, @@ -922,19 +941,13 @@ static void gen8_ppgtt_clear_4lvl(struct i915_address_space *vm, GEM_BUG_ON(!i915_vm_is_4lvl(vm)); gen8_for_each_pml4e(pdp, pml4, start, length, pml4e) { - bool free = false; GEM_BUG_ON(pdp == vm->scratch_pdp); - if (!gen8_ppgtt_clear_pdp(vm, pdp, start, length)) - continue; + atomic_inc(&pdp->used); - spin_lock(&pml4->lock); - if (!atomic_read(&pdp->used)) { - clear_pd_entry(pml4, pml4e, vm->scratch_pdp); - free = true; - } - spin_unlock(&pml4->lock); - if (free) + gen8_ppgtt_clear_pdp(vm, pdp, start, length); + + if (release_pd_entry(pml4, pml4e, pdp, vm->scratch_pdp)) free_pd(vm, pdp); } } @@ -1457,17 +1470,8 @@ static int gen8_ppgtt_alloc_pdp(struct i915_address_space *vm, goto out; unwind_pd: - if (alloc) { - free_pd(vm, alloc); - alloc = NULL; - } - spin_lock(&pdp->lock); - if (atomic_dec_and_test(&pd->used)) { - GEM_BUG_ON(alloc); - alloc = pd; /* defer the free to after the lock */ - clear_pd_entry(pdp, pdpe, vm->scratch_pd); - } - spin_unlock(&pdp->lock); + if (release_pd_entry(pdp, pdpe, pd, vm->scratch_pd)) + free_pd(vm, pd); unwind: gen8_ppgtt_clear_pdp(vm, pdp, from, start - from); out: @@ -1530,17 +1534,8 @@ static int gen8_ppgtt_alloc_4lvl(struct i915_address_space *vm, goto out; unwind_pdp: - if (alloc) { - free_pd(vm, alloc); - alloc = NULL; - } - spin_lock(&pml4->lock); - if (atomic_dec_and_test(&pdp->used)) { - GEM_BUG_ON(alloc); - alloc = pdp; /* defer the free until after the lock */ - clear_pd_entry(pml4, pml4e, vm->scratch_pdp); - } - spin_unlock(&pml4->lock); + if (release_pd_entry(pml4, pml4e, pdp, vm->scratch_pdp)) + free_pd(vm, pdp); unwind: gen8_ppgtt_clear_4lvl(vm, from, start - from); out: @@ -1572,8 +1567,8 @@ static int gen8_preallocate_top_level_pdp(struct i915_ppgtt *ppgtt) unwind: start -= from; gen8_for_each_pdpe(pd, pdp, from, start, pdpe) { - clear_pd_entry(pdp, pdpe, vm->scratch_pd); - free_pd(vm, pd); + if (release_pd_entry(pdp, pdpe, pd, vm->scratch_pd)) + free_pd(vm, pd); } atomic_set(&pdp->used, 0); return -ENOMEM;