| Message ID | 1495641251-30022-1-git-send-email-jon.bloomfield@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 24/05/2017 16:54, Jon Bloomfield wrote: > BXT has a H/W issue with IOMMU which can lead to system hangs when > Aperture accesses are queued within the GAM behind GTT Accesses. > > This patch avoids the condition by wrapping all GTT updates in stop_machine > and using a flushing read prior to restarting the machine. > > The stop_machine guarantees no new Aperture accesses can begin while > the PTE writes are being emmitted. The flushing read ensures that > any following Aperture accesses cannot begin until the PTE writes > have been cleared out of the GAM's fifo. > > Only FOLLOWING Aperture accesses need to be separated from in flight > PTE updates. PTE Writes may follow tightly behind already in flight > Aperture accesses, so no flushing read is required at the start of > a PTE update sequence. > > This issue was reproduced by running > igt/gem_readwrite and > igt/gem_render_copy > simultaneously from different processes, each in a tight loop, > with INTEL_IOMMU enabled. > > This patch was originally published as: > drm/i915: Serialize GTT Updates on BXT > > v2: Move bxt/iommu detection into static function > Remove #ifdef CONFIG_INTEL_IOMMU protection > Make function names more reflective of purpose > Move flushing read into static function > > Signed-off-by: Jon Bloomfield <jon.bloomfield@intel.com> > Cc: John Harrison <john.C.Harrison@intel.com> > Cc: Chris Wilson <chris@chris-wilson.co.uk> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > --- > drivers/gpu/drm/i915/i915_drv.h | 9 +++ > drivers/gpu/drm/i915/i915_gem_gtt.c | 114 ++++++++++++++++++++++++++++++++++++ > 2 files changed, 123 insertions(+) > > diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h > index 17883a8..b2783b2 100644 > --- a/drivers/gpu/drm/i915/i915_drv.h > +++ b/drivers/gpu/drm/i915/i915_drv.h > @@ -3006,6 +3006,15 @@ static inline bool intel_scanout_needs_vtd_wa(struct drm_i915_private *dev_priv) > return false; > } > > +static inline bool intel_gtt_update_needs_vtd_wa(struct drm_i915_private *dev_priv) > +{ > +#ifdef CONFIG_INTEL_IOMMU > + if (IS_BROXTON(dev_priv) && intel_iommu_gfx_mapped) > + return true; > +#endif > + return false; > +} > + > int intel_sanitize_enable_ppgtt(struct drm_i915_private *dev_priv, > int enable_ppgtt); > > diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c > index 7c769d7..ab91178 100644 > --- a/drivers/gpu/drm/i915/i915_gem_gtt.c > +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c > @@ -2191,6 +2191,110 @@ static void gen8_ggtt_clear_range(struct i915_address_space *vm, > gen8_set_pte(>t_base[i], scratch_pte); > } > > +static void bxt_vtd_gtt_wa(struct drm_i915_private *dev_priv) > +{ > + /* > + * Make sure the internal GAM fifo has been cleared of all GTT > + * Writes before exiting stop_machine(). This guarantees that > + * any Aperture accesses waiting to start in another process > + * cannot backup behind the GTT Writes causing a hang. > + * The register can be any arbitrary GAM register. > + */ > + POSTING_READ(GFX_FLSH_CNTL_GEN6); > +} > + > +struct insert_page { > + struct i915_address_space *vm; > + dma_addr_t addr; > + u64 offset; > + enum i915_cache_level level; > +}; > + > +static int bxt_vtd_ggtt_insert_page__cb(void *_arg) > +{ > + struct insert_page *arg = _arg; > + I don't remember seeing blank lines in declaration blocks anywhere else so I'd remove this (and below). But FWIW, in essence the patch looks good to me. I was even surprised that GCC understands not to complain about defined but unused functions when IOMMU is compiled out. One learns something new every day! (Until one forgets it again!) Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Regards, Tvrtko > + struct drm_i915_private *dev_priv = arg->vm->i915; > + > + gen8_ggtt_insert_page(arg->vm, arg->addr, > + arg->offset, arg->level, 0); > + > + bxt_vtd_gtt_wa(dev_priv); > + > + return 0; > +} > + > +static void bxt_vtd_ggtt_insert_page__BKL(struct i915_address_space *vm, > + dma_addr_t addr, > + u64 offset, > + enum i915_cache_level level, > + u32 unused) > +{ > + struct insert_page arg = { vm, addr, offset, level }; > + > + stop_machine(bxt_vtd_ggtt_insert_page__cb, &arg, NULL); > +} > + > + > +struct insert_entries { > + struct i915_address_space *vm; > + struct sg_table *st; > + u64 start; > + enum i915_cache_level level; > +}; > + > +static int bxt_vtd_ggtt_insert_entries__cb(void *_arg) > +{ > + struct insert_entries *arg = _arg; > + > + struct drm_i915_private *dev_priv = arg->vm->i915; > + > + gen8_ggtt_insert_entries(arg->vm, arg->st, > + arg->start, arg->level, 0); > + > + bxt_vtd_gtt_wa(dev_priv); > + > + return 0; > +} > + > +static void bxt_vtd_ggtt_insert_entries__BKL(struct i915_address_space *vm, > + struct sg_table *st, > + u64 start, > + enum i915_cache_level level, > + u32 unused) > +{ > + struct insert_entries arg = { vm, st, start, level }; > + > + stop_machine(bxt_vtd_ggtt_insert_entries__cb, &arg, NULL); > +} > + > +struct clear_range { > + struct i915_address_space *vm; > + u64 start; > + u64 length; > +}; > + > +static int bxt_vtd_ggtt_clear_range__cb(void *_arg) > +{ > + struct clear_range *arg = _arg; > + > + struct drm_i915_private *dev_priv = arg->vm->i915; > + > + gen8_ggtt_clear_range(arg->vm, arg->start, arg->length); > + > + bxt_vtd_gtt_wa(dev_priv); > + > + return 0; > +} > + > +static void bxt_vtd_ggtt_clear_range__BKL(struct i915_address_space *vm, > + u64 start, > + u64 length) > +{ > + struct clear_range arg = { vm, start, length }; > + stop_machine(bxt_vtd_ggtt_clear_range__cb, &arg, NULL); > +} > + > static void gen6_ggtt_clear_range(struct i915_address_space *vm, > u64 start, u64 length) > { > @@ -2789,6 +2893,16 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt) > > ggtt->base.insert_entries = gen8_ggtt_insert_entries; > > + /* Serialize GTT updates with Aperture on BXT if VT-d is on. */ > + if (intel_gtt_update_needs_vtd_wa(dev_priv)) { > + ggtt->base.insert_entries = bxt_vtd_ggtt_insert_entries__BKL; > + ggtt->base.insert_page = bxt_vtd_ggtt_insert_page__BKL; > + if (!USES_FULL_PPGTT(dev_priv) || > + intel_scanout_needs_vtd_wa(dev_priv)) { > + ggtt->base.clear_range = bxt_vtd_ggtt_clear_range__BKL; > + } > + } > + > ggtt->invalidate = gen6_ggtt_invalidate; > > return ggtt_probe_common(ggtt, size); >
On Thu, May 25, 2017 at 11:54:03AM +0100, Tvrtko Ursulin wrote: > > On 24/05/2017 16:54, Jon Bloomfield wrote: > >BXT has a H/W issue with IOMMU which can lead to system hangs when > >Aperture accesses are queued within the GAM behind GTT Accesses. > > > >This patch avoids the condition by wrapping all GTT updates in stop_machine > >and using a flushing read prior to restarting the machine. > > > >The stop_machine guarantees no new Aperture accesses can begin while > >the PTE writes are being emmitted. The flushing read ensures that > >any following Aperture accesses cannot begin until the PTE writes > >have been cleared out of the GAM's fifo. > > > >Only FOLLOWING Aperture accesses need to be separated from in flight > >PTE updates. PTE Writes may follow tightly behind already in flight > >Aperture accesses, so no flushing read is required at the start of > >a PTE update sequence. > > > >This issue was reproduced by running > > igt/gem_readwrite and > > igt/gem_render_copy > >simultaneously from different processes, each in a tight loop, > >with INTEL_IOMMU enabled. > > > >This patch was originally published as: > > drm/i915: Serialize GTT Updates on BXT > > > >v2: Move bxt/iommu detection into static function > > Remove #ifdef CONFIG_INTEL_IOMMU protection > > Make function names more reflective of purpose > > Move flushing read into static function > > > >Signed-off-by: Jon Bloomfield <jon.bloomfield@intel.com> > >Cc: John Harrison <john.C.Harrison@intel.com> > >Cc: Chris Wilson <chris@chris-wilson.co.uk> > >Cc: Daniel Vetter <daniel.vetter@intel.com> > >Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > >+static int bxt_vtd_ggtt_insert_page__cb(void *_arg) > >+{ > >+ struct insert_page *arg = _arg; > >+ > > I don't remember seeing blank lines in declaration blocks anywhere > else so I'd remove this (and below). > > But FWIW, in essence the patch looks good to me. I was even > surprised that GCC understands not to complain about defined but > unused functions when IOMMU is compiled out. One learns something > new every day! (Until one forgets it again!) > > Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Tidied up the whitespace to appease checkpatch, and pushed. Thanks for the fix and review, -Chris
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index 17883a8..b2783b2 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -3006,6 +3006,15 @@ static inline bool intel_scanout_needs_vtd_wa(struct drm_i915_private *dev_priv) return false; } +static inline bool intel_gtt_update_needs_vtd_wa(struct drm_i915_private *dev_priv) +{ +#ifdef CONFIG_INTEL_IOMMU + if (IS_BROXTON(dev_priv) && intel_iommu_gfx_mapped) + return true; +#endif + return false; +} + int intel_sanitize_enable_ppgtt(struct drm_i915_private *dev_priv, int enable_ppgtt); diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c index 7c769d7..ab91178 100644 --- a/drivers/gpu/drm/i915/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c @@ -2191,6 +2191,110 @@ static void gen8_ggtt_clear_range(struct i915_address_space *vm, gen8_set_pte(>t_base[i], scratch_pte); } +static void bxt_vtd_gtt_wa(struct drm_i915_private *dev_priv) +{ + /* + * Make sure the internal GAM fifo has been cleared of all GTT + * Writes before exiting stop_machine(). This guarantees that + * any Aperture accesses waiting to start in another process + * cannot backup behind the GTT Writes causing a hang. + * The register can be any arbitrary GAM register. + */ + POSTING_READ(GFX_FLSH_CNTL_GEN6); +} + +struct insert_page { + struct i915_address_space *vm; + dma_addr_t addr; + u64 offset; + enum i915_cache_level level; +}; + +static int bxt_vtd_ggtt_insert_page__cb(void *_arg) +{ + struct insert_page *arg = _arg; + + struct drm_i915_private *dev_priv = arg->vm->i915; + + gen8_ggtt_insert_page(arg->vm, arg->addr, + arg->offset, arg->level, 0); + + bxt_vtd_gtt_wa(dev_priv); + + return 0; +} + +static void bxt_vtd_ggtt_insert_page__BKL(struct i915_address_space *vm, + dma_addr_t addr, + u64 offset, + enum i915_cache_level level, + u32 unused) +{ + struct insert_page arg = { vm, addr, offset, level }; + + stop_machine(bxt_vtd_ggtt_insert_page__cb, &arg, NULL); +} + + +struct insert_entries { + struct i915_address_space *vm; + struct sg_table *st; + u64 start; + enum i915_cache_level level; +}; + +static int bxt_vtd_ggtt_insert_entries__cb(void *_arg) +{ + struct insert_entries *arg = _arg; + + struct drm_i915_private *dev_priv = arg->vm->i915; + + gen8_ggtt_insert_entries(arg->vm, arg->st, + arg->start, arg->level, 0); + + bxt_vtd_gtt_wa(dev_priv); + + return 0; +} + +static void bxt_vtd_ggtt_insert_entries__BKL(struct i915_address_space *vm, + struct sg_table *st, + u64 start, + enum i915_cache_level level, + u32 unused) +{ + struct insert_entries arg = { vm, st, start, level }; + + stop_machine(bxt_vtd_ggtt_insert_entries__cb, &arg, NULL); +} + +struct clear_range { + struct i915_address_space *vm; + u64 start; + u64 length; +}; + +static int bxt_vtd_ggtt_clear_range__cb(void *_arg) +{ + struct clear_range *arg = _arg; + + struct drm_i915_private *dev_priv = arg->vm->i915; + + gen8_ggtt_clear_range(arg->vm, arg->start, arg->length); + + bxt_vtd_gtt_wa(dev_priv); + + return 0; +} + +static void bxt_vtd_ggtt_clear_range__BKL(struct i915_address_space *vm, + u64 start, + u64 length) +{ + struct clear_range arg = { vm, start, length }; + stop_machine(bxt_vtd_ggtt_clear_range__cb, &arg, NULL); +} + static void gen6_ggtt_clear_range(struct i915_address_space *vm, u64 start, u64 length) { @@ -2789,6 +2893,16 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt) ggtt->base.insert_entries = gen8_ggtt_insert_entries; + /* Serialize GTT updates with Aperture on BXT if VT-d is on. */ + if (intel_gtt_update_needs_vtd_wa(dev_priv)) { + ggtt->base.insert_entries = bxt_vtd_ggtt_insert_entries__BKL; + ggtt->base.insert_page = bxt_vtd_ggtt_insert_page__BKL; + if (!USES_FULL_PPGTT(dev_priv) || + intel_scanout_needs_vtd_wa(dev_priv)) { + ggtt->base.clear_range = bxt_vtd_ggtt_clear_range__BKL; + } + } + ggtt->invalidate = gen6_ggtt_invalidate; return ggtt_probe_common(ggtt, size);
BXT has a H/W issue with IOMMU which can lead to system hangs when Aperture accesses are queued within the GAM behind GTT Accesses. This patch avoids the condition by wrapping all GTT updates in stop_machine and using a flushing read prior to restarting the machine. The stop_machine guarantees no new Aperture accesses can begin while the PTE writes are being emmitted. The flushing read ensures that any following Aperture accesses cannot begin until the PTE writes have been cleared out of the GAM's fifo. Only FOLLOWING Aperture accesses need to be separated from in flight PTE updates. PTE Writes may follow tightly behind already in flight Aperture accesses, so no flushing read is required at the start of a PTE update sequence. This issue was reproduced by running igt/gem_readwrite and igt/gem_render_copy simultaneously from different processes, each in a tight loop, with INTEL_IOMMU enabled. This patch was originally published as: drm/i915: Serialize GTT Updates on BXT v2: Move bxt/iommu detection into static function Remove #ifdef CONFIG_INTEL_IOMMU protection Make function names more reflective of purpose Move flushing read into static function Signed-off-by: Jon Bloomfield <jon.bloomfield@intel.com> Cc: John Harrison <john.C.Harrison@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Daniel Vetter <daniel.vetter@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> --- drivers/gpu/drm/i915/i915_drv.h | 9 +++ drivers/gpu/drm/i915/i915_gem_gtt.c | 114 ++++++++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+)