From patchwork Thu Dec 18 17:10:11 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Thierry X-Patchwork-Id: 5515351 Return-Path: X-Original-To: patchwork-intel-gfx@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 2C2F2BEEBA for ; Thu, 18 Dec 2014 17:18:00 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 8FCFE2092D for ; Thu, 18 Dec 2014 17:17:58 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.kernel.org (Postfix) with ESMTP id EE1F62020E for ; Thu, 18 Dec 2014 17:17:56 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6E92F6EA51; Thu, 18 Dec 2014 09:17:56 -0800 (PST) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTP id 3F7846EA53 for ; Thu, 18 Dec 2014 09:17:54 -0800 (PST) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP; 18 Dec 2014 09:08:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,691,1406617200"; d="scan'208";a="501038583" Received: from michelth-linux.isw.intel.com ([10.102.226.150]) by orsmga003.jf.intel.com with ESMTP; 18 Dec 2014 09:05:59 -0800 From: Michel Thierry To: intel-gfx@lists.freedesktop.org Date: Thu, 18 Dec 2014 17:10:11 +0000 Message-Id: <1418922621-25818-15-git-send-email-michel.thierry@intel.com> X-Mailer: git-send-email 2.1.1 In-Reply-To: <1418922621-25818-1-git-send-email-michel.thierry@intel.com> References: <1418922621-25818-1-git-send-email-michel.thierry@intel.com> Subject: [Intel-gfx] [PATCH 14/24] drm/i915: Finish gen6/7 dynamic page table allocation X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Ben Widawsky This patch continues on the idea from the previous patch. From here on, in the steady state, PDEs are all pointing to the scratch page table (as recommended in the spec). When an object is allocated in the VA range, the code will determine if we need to allocate a page for the page table. Similarly when the object is destroyed, we will remove, and free the page table pointing the PDE back to the scratch page. Following patches will work to unify the code a bit as we bring in GEN8 support. GEN6 and GEN8 are different enough that I had a hard time to get to this point with as much common code as I do. The aliasing PPGTT must pre-allocate all of the page tables. There are a few reasons for this. Two trivial ones: aliasing ppgtt goes through the ggtt paths, so it's hard to maintain, we currently do not restore the default context (assuming the previous force reload is indeed necessary). Most importantly though, the only way (it seems from empirical evidence) to invalidate the CS TLBs on non-render ring is to either use ring sync (which requires actually stopping the rings in order to synchronize when the sync completes vs. where you are in execution), or to reload DCLV. Since without full PPGTT we do not ever reload the DCLV register, there is no good way to achieve this. The simplest solution is just to not support dynamic page table creation/destruction in the aliasing PPGTT. We could always reload DCLV, but this seems like quite a bit of excess overhead only to save at most 2MB-4k of memory for the aliasing PPGTT page tables. v2: Make the page table bitmap declared inside the function (Chris) Simplify the way scratching address space works. Move the alloc/teardown tracepoints up a level in the call stack so that both all implementations get the trace. v3: Updated trace event to spit out a name v4: Aliasing ppgtt is now initialized differently (in setup global gtt) Signed-off-by: Ben Widawsky Signed-off-by: Michel Thierry (v4) --- drivers/gpu/drm/i915/i915_debugfs.c | 3 +- drivers/gpu/drm/i915/i915_drv.h | 7 ++ drivers/gpu/drm/i915/i915_gem_gtt.c | 125 +++++++++++++++++++++++++++++++++--- drivers/gpu/drm/i915/i915_trace.h | 116 +++++++++++++++++++++++++++++++++ 4 files changed, 240 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c index 60f91bc..0f63076 100644 --- a/drivers/gpu/drm/i915/i915_debugfs.c +++ b/drivers/gpu/drm/i915/i915_debugfs.c @@ -2149,6 +2149,8 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev) seq_printf(m, "PP_DIR_BASE_READ: 0x%08x\n", I915_READ(RING_PP_DIR_BASE_READ(ring))); seq_printf(m, "PP_DIR_DCLV: 0x%08x\n", I915_READ(RING_PP_DIR_DCLV(ring))); } + seq_printf(m, "ECOCHK: 0x%08x\n\n", I915_READ(GAM_ECOCHK)); + if (dev_priv->mm.aliasing_ppgtt) { struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt; @@ -2165,7 +2167,6 @@ static void gen6_ppgtt_info(struct seq_file *m, struct drm_device *dev) get_pid_task(file->pid, PIDTYPE_PID)->comm); idr_for_each(&file_priv->context_idr, per_file_ctx, m); } - seq_printf(m, "ECOCHK: 0x%08x\n", I915_READ(GAM_ECOCHK)); } static int i915_ppgtt_info(struct seq_file *m, void *data) diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index 3047291f..d74db21 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -2819,6 +2819,13 @@ static inline bool i915_is_ggtt(struct i915_address_space *vm) return vm == ggtt; } +static inline bool i915_is_aliasing_ppgtt(struct i915_address_space *vm) +{ + struct i915_address_space *appgtt = + &((struct drm_i915_private *)(vm)->dev->dev_private)->mm.aliasing_ppgtt->base; + return vm == appgtt; +} + static inline struct i915_hw_ppgtt * i915_vm_to_ppgtt(struct i915_address_space *vm) { diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index c917301..02ccb18 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -1129,10 +1129,47 @@ static void gen6_ppgtt_unmap_pages(struct i915_hw_ppgtt *ppgtt) static int gen6_alloc_va_range(struct i915_address_space *vm, uint64_t start, uint64_t length) { + DECLARE_BITMAP(new_page_tables, GEN6_PPGTT_PD_ENTRIES); + struct drm_device *dev = vm->dev; + struct drm_i915_private *dev_priv = dev->dev_private; struct i915_hw_ppgtt *ppgtt = container_of(vm, struct i915_hw_ppgtt, base); struct i915_pagetab *pt; + const uint32_t start_save = start, length_save = length; uint32_t pde, temp; + int ret; + + BUG_ON(upper_32_bits(start)); + + bitmap_zero(new_page_tables, GEN6_PPGTT_PD_ENTRIES); + + /* The allocation is done in two stages so that we can bail out with + * minimal amount of pain. The first stage finds new page tables that + * need allocation. The second stage marks use ptes within the page + * tables. + */ + gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) { + if (pt != ppgtt->scratch_pt) { + WARN_ON(bitmap_empty(pt->used_ptes, I915_PPGTT_PT_ENTRIES)); + continue; + } + + /* We've already allocated a page table */ + WARN_ON(!bitmap_empty(pt->used_ptes, I915_PPGTT_PT_ENTRIES)); + + pt = alloc_pt_single(dev); + if (IS_ERR(pt)) { + ret = PTR_ERR(pt); + goto unwind_out; + } + + ppgtt->pd.page_tables[pde] = pt; + set_bit(pde, new_page_tables); + trace_i915_pagetable_alloc(vm, pde, start, GEN6_PDE_SHIFT); + } + + start = start_save; + length = length_save; gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) { int j; @@ -1150,11 +1187,32 @@ static int gen6_alloc_va_range(struct i915_address_space *vm, } } - bitmap_or(pt->used_ptes, pt->used_ptes, tmp_bitmap, + if (test_and_clear_bit(pde, new_page_tables)) + gen6_write_pdes(&ppgtt->pd, pde, pt); + + trace_i915_pagetable_map(vm, pde, pt, + gen6_pte_index(start), + gen6_pte_count(start, length), + I915_PPGTT_PT_ENTRIES); + bitmap_or(pt->used_ptes, tmp_bitmap, pt->used_ptes, I915_PPGTT_PT_ENTRIES); } + WARN_ON(!bitmap_empty(new_page_tables, GEN6_PPGTT_PD_ENTRIES)); + + /* Make sure write is complete before other code can use this page + * table. Also require for WC mapped PTEs */ + readl(dev_priv->gtt.gsm); + return 0; + +unwind_out: + for_each_set_bit(pde, new_page_tables, GEN6_PPGTT_PD_ENTRIES) { + struct i915_pagetab *pt = ppgtt->pd.page_tables[pde]; + ppgtt->pd.page_tables[pde] = NULL; + free_pt_single(pt, vm->dev); + } + return ret; } static void gen6_teardown_va_range(struct i915_address_space *vm, @@ -1166,8 +1224,27 @@ static void gen6_teardown_va_range(struct i915_address_space *vm, uint32_t pde, temp; gen6_for_each_pde(pt, &ppgtt->pd, start, length, temp, pde) { + + if (WARN(pt == ppgtt->scratch_pt, + "Tried to teardown scratch page vm %p. pde %u: %llx-%llx\n", + vm, pde, start, start + length)) + continue; + + trace_i915_pagetable_unmap(vm, pde, pt, + gen6_pte_index(start), + gen6_pte_count(start, length), + I915_PPGTT_PT_ENTRIES); + bitmap_clear(pt->used_ptes, gen6_pte_index(start), gen6_pte_count(start, length)); + + if (bitmap_empty(pt->used_ptes, I915_PPGTT_PT_ENTRIES)) { + trace_i915_pagetable_destroy(vm, pde, + start & GENMASK_ULL(63, GEN6_PDE_SHIFT), + GEN6_PDE_SHIFT); + gen6_write_pdes(&ppgtt->pd, pde, ppgtt->scratch_pt); + ppgtt->pd.page_tables[pde] = ppgtt->scratch_pt; + } } } @@ -1175,9 +1252,13 @@ static void gen6_ppgtt_free(struct i915_hw_ppgtt *ppgtt) { int i; - for (i = 0; i < ppgtt->num_pd_entries; i++) - free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev); + for (i = 0; i < ppgtt->num_pd_entries; i++) { + struct i915_pagetab *pt = ppgtt->pd.page_tables[i]; + if (pt != ppgtt->scratch_pt) + free_pt_single(ppgtt->pd.page_tables[i], ppgtt->base.dev); + } + /* Consider putting this as part of pd free. */ free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev); free_pd_single(&ppgtt->pd); } @@ -1242,7 +1323,7 @@ err_out: return ret; } -static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt) +static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt, bool preallocate_pt) { int ret; @@ -1250,10 +1331,14 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt) if (ret) return ret; + if (!preallocate_pt) + return 0; + ret = alloc_pt_range(&ppgtt->pd, 0, ppgtt->num_pd_entries, ppgtt->base.dev); if (ret) { + free_pt_scratch(ppgtt->scratch_pt, ppgtt->base.dev); drm_mm_remove_node(&ppgtt->node); return ret; } @@ -1261,7 +1346,17 @@ static int gen6_ppgtt_alloc(struct i915_hw_ppgtt *ppgtt) return 0; } -static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) +static void gen6_scratch_va_range(struct i915_hw_ppgtt *ppgtt, + uint64_t start, uint64_t length) +{ + struct i915_pagetab *unused; + uint32_t pde, temp; + + gen6_for_each_pde(unused, &ppgtt->pd, start, length, temp, pde) + ppgtt->pd.page_tables[pde] = ppgtt->scratch_pt; +} + +static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt, bool aliasing) { struct drm_device *dev = ppgtt->base.dev; struct drm_i915_private *dev_priv = dev->dev_private; @@ -1277,7 +1372,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) } else BUG(); - ret = gen6_ppgtt_alloc(ppgtt); + ret = gen6_ppgtt_alloc(ppgtt, aliasing); if (ret) return ret; @@ -1296,6 +1391,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) ppgtt->pd_addr = (gen6_gtt_pte_t __iomem *)dev_priv->gtt.gsm + ppgtt->pd.pd_offset / sizeof(gen6_gtt_pte_t); + if (!aliasing) + gen6_scratch_va_range(ppgtt, 0, ppgtt->base.total); + gen6_write_page_range(dev_priv, &ppgtt->pd, 0, ppgtt->base.total); DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n", @@ -1308,7 +1406,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) return 0; } -static int __hw_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt) +static int __hw_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt, + bool aliasing) { struct drm_i915_private *dev_priv = dev->dev_private; @@ -1316,7 +1415,7 @@ static int __hw_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt) ppgtt->base.scratch = dev_priv->gtt.base.scratch; if (INTEL_INFO(dev)->gen < 8) - return gen6_ppgtt_init(ppgtt); + return gen6_ppgtt_init(ppgtt, aliasing); else if (IS_GEN8(dev) || IS_GEN9(dev)) return gen8_ppgtt_init(ppgtt, dev_priv->gtt.base.total); else @@ -1327,7 +1426,7 @@ int i915_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt) struct drm_i915_private *dev_priv = dev->dev_private; int ret = 0; - ret = __hw_ppgtt_init(dev, ppgtt); + ret = __hw_ppgtt_init(dev, ppgtt, false); if (ret == 0) { kref_init(&ppgtt->ref); drm_mm_init(&ppgtt->base.mm, ppgtt->base.start, @@ -1437,6 +1536,8 @@ ppgtt_bind_vma(struct i915_vma *vma, flags |= PTE_READ_ONLY; if (vma->vm->allocate_va_range) { + trace_i915_va_alloc(vma->vm, vma->node.start, vma->node.size, + VM_TO_TRACE_NAME(vma->vm)); ret = vma->vm->allocate_va_range(vma->vm, vma->node.start, vma->node.size); @@ -1458,6 +1559,10 @@ static void ppgtt_unbind_vma(struct i915_vma *vma) vma->obj->base.size, true); if (vma->vm->teardown_va_range) { + trace_i915_va_teardown(vma->vm, + vma->node.start, vma->node.size, + VM_TO_TRACE_NAME(vma->vm)); + vma->vm->teardown_va_range(vma->vm, vma->node.start, vma->node.size); ppgtt_invalidate_tlbs(vma->vm); @@ -1981,7 +2086,7 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev, if (!ppgtt) return -ENOMEM; - ret = __hw_ppgtt_init(dev, ppgtt); + ret = __hw_ppgtt_init(dev, ppgtt, true); if (ret != 0) return ret; diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h index f004d3d..8ba11d8 100644 --- a/drivers/gpu/drm/i915/i915_trace.h +++ b/drivers/gpu/drm/i915/i915_trace.h @@ -156,6 +156,122 @@ TRACE_EVENT(i915_vma_unbind, __entry->obj, __entry->offset, __entry->size, __entry->vm) ); +#define VM_TO_TRACE_NAME(vm) \ + (i915_is_ggtt(vm) ? "GGTT" : \ + i915_is_aliasing_ppgtt(vm) ? "Aliasing PPGTT" : \ + "Private VM") + +DECLARE_EVENT_CLASS(i915_va, + TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name), + TP_ARGS(vm, start, length, name), + + TP_STRUCT__entry( + __field(struct i915_address_space *, vm) + __field(u64, start) + __field(u64, end) + __string(name, name) + ), + + TP_fast_assign( + __entry->vm = vm; + __entry->start = start; + __entry->end = start + length; + __assign_str(name, name); + ), + + TP_printk("vm=%p (%s), 0x%llx-0x%llx", + __entry->vm, __get_str(name), __entry->start, __entry->end) +); + +DEFINE_EVENT(i915_va, i915_va_alloc, + TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name), + TP_ARGS(vm, start, length, name) +); + +DEFINE_EVENT(i915_va, i915_va_teardown, + TP_PROTO(struct i915_address_space *vm, u64 start, u64 length, const char *name), + TP_ARGS(vm, start, length, name) +); + +DECLARE_EVENT_CLASS(i915_pagetable, + TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift), + TP_ARGS(vm, pde, start, pde_shift), + + TP_STRUCT__entry( + __field(struct i915_address_space *, vm) + __field(u32, pde) + __field(u64, start) + __field(u64, end) + ), + + TP_fast_assign( + __entry->vm = vm; + __entry->pde = pde; + __entry->start = start; + __entry->end = (start + (1ULL << pde_shift)) & ~((1ULL << pde_shift)-1); + ), + + TP_printk("vm=%p, pde=%d (0x%llx-0x%llx)", + __entry->vm, __entry->pde, __entry->start, __entry->end) +); + +DEFINE_EVENT(i915_pagetable, i915_pagetable_alloc, + TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift), + TP_ARGS(vm, pde, start, pde_shift) +); + +DEFINE_EVENT(i915_pagetable, i915_pagetable_destroy, + TP_PROTO(struct i915_address_space *vm, u32 pde, u64 start, u64 pde_shift), + TP_ARGS(vm, pde, start, pde_shift) +); + +/* Avoid extra math because we only support two sizes. The format is defined by + * bitmap_scnprintf. Each 32 bits is 8 HEX digits followed by comma */ +#define TRACE_PT_SIZE(bits) \ + ((((bits) == 1024) ? 288 : 144) + 1) + +DECLARE_EVENT_CLASS(i915_pagetable_update, + TP_PROTO(struct i915_address_space *vm, u32 pde, + struct i915_pagetab *pt, u32 first, u32 len, size_t bits), + TP_ARGS(vm, pde, pt, first, len, bits), + + TP_STRUCT__entry( + __field(struct i915_address_space *, vm) + __field(u32, pde) + __field(u32, first) + __field(u32, last) + __dynamic_array(char, cur_ptes, TRACE_PT_SIZE(bits)) + ), + + TP_fast_assign( + __entry->vm = vm; + __entry->pde = pde; + __entry->first = first; + __entry->last = first + len; + + bitmap_scnprintf(__get_str(cur_ptes), + TRACE_PT_SIZE(bits), + pt->used_ptes, + bits); + ), + + TP_printk("vm=%p, pde=%d, updating %u:%u\t%s", + __entry->vm, __entry->pde, __entry->last, __entry->first, + __get_str(cur_ptes)) +); + +DEFINE_EVENT(i915_pagetable_update, i915_pagetable_map, + TP_PROTO(struct i915_address_space *vm, u32 pde, + struct i915_pagetab *pt, u32 first, u32 len, size_t bits), + TP_ARGS(vm, pde, pt, first, len, bits) +); + +DEFINE_EVENT(i915_pagetable_update, i915_pagetable_unmap, + TP_PROTO(struct i915_address_space *vm, u32 pde, + struct i915_pagetab *pt, u32 first, u32 len, size_t bits), + TP_ARGS(vm, pde, pt, first, len, bits) +); + TRACE_EVENT(i915_gem_object_change_domain, TP_PROTO(struct drm_i915_gem_object *obj, u32 old_read, u32 old_write), TP_ARGS(obj, old_read, old_write),