| Message ID | 20210907103407.432646-1-tvrtko.ursulin@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v4] drm/i915: Use Transparent Hugepages when IOMMU is enabled | expand |
Hi, For completeness sake, it might be worth mentioning specifically what (synthetic) test-cases regress with THP patch. * Skylake GT4e: 20-25% SynMark TexMem* (whereas all MemBW GPU tests either improve or are not affected) * Broxton J4205: 7% MemBW GPU texture 2-3% SynMark TexMem* * Tigerlake-H: 7% MemBW GPU blend I have no idea why on GEN9 texture accesses regress, but on GEN12 TGL it's render buffer blend that regresses. Blend (read+write) regressing is especially odd, as neither render buffer read nor write regresses. Maybe that is a GEN12 specific driver bug similar to Mesa/i965 bug from few years back in how its shaders access render buffer, that had caused SIMD32 accesses to regress memory BW bound test-cases perf a bit compared to SIMD16? (Blend test is likely to run nowadays as SIMD32.) - Eero On 7.9.2021 13.34, Tvrtko Ursulin wrote: > From: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > Usage of Transparent Hugepages was disabled in 9987da4b5dcf > ("drm/i915: Disable THP until we have a GPU read BW W/A"), but since it > appears majority of performance regressions reported with an enabled IOMMU > can be almost eliminated by turning them on, lets just do that. > > To err on the side of safety we keep the current default in cases where > IOMMU is not active, and only when it is default to the "huge=within_size" > mode. Although there probably would be wins to enable them throughout, > more extensive testing across benchmarks and platforms would need to be > done. > > With the patch and IOMMU enabled my local testing on a small Skylake part > shows OglVSTangent regression being reduced from ~14% (IOMMU on versus > IOMMU off) to ~2% (same comparison but with THP on). > > More detailed testing done in the below referenced Gitlab issue by Eero: > > Skylake GT4e: > > Performance drops from enabling IOMMU: > > 30-35% SynMark CSDof > 20-25% Unigine Heaven, MemBW GPU write, SynMark VSTangent > ~20% GLB Egypt (1/2 screen window) > 10-15% GLB T-Rex (1/2 screen window) > 8-10% GfxBench T-Rex, MemBW GPU blit > 7-8% SynMark DeferredAA + TerrainFly* + ZBuffer > 6-7% GfxBench Manhattan 3.0 + 3.1, SynMark TexMem128 & CSCloth > 5-6% GfxBench CarChase, Unigine Valley > 3-5% GfxBench Vulkan & GL AztecRuins + ALU2, MemBW GPU texture, > SynMark Fill*, Deferred, TerrainPan* > 1-2% Most of the other tests > > With the patch drops become: > > 20-25% SynMark TexMem* > 15-20% GLB Egypt (1/2 screen window) > 10-15% GLB T-Rex (1/2 screen window) > 4-7% GfxBench T-Rex, GpuTest Triangle > 1-8% GfxBench ALU2 (offscreen 1%, onscreen 8%) > 3% GfxBench Manhattan 3.0, SynMark CSDof > 2-3% Unigine Heaven + Valley, MemBW GPU texture > 1-3 GfxBench Manhattan 3.1 + CarChase + Vulkan & GL AztecRuins > > Broxton: > > Performance drops from IOMMU, without patch: > > 30% MemBW GPU write > 25% SynMark ZBuffer + Fill* > 20% MemBW GPU blit > 15% MemBW GPU blend, GpuTest Triangle > 10-15% MemBW GPU texture > 10% GLB Egypt, Unigine Heaven (had hangs), SynMark TerrainFly* > 7-9% GLB T-Rex, GfxBench Manhattan 3.0 + T-Rex, > SynMark Deferred* + TexMem* > 6-8% GfxBench CarChase, Unigine Valley, > SynMark CSCloth + ShMapVsm + TerrainPan* > 5-6% GfxBench Manhattan 3.1 + GL AztecRuins, > SynMark CSDof + TexFilterTri > 2-4% GfxBench ALU2, SynMark DrvRes + GSCloth + ShMapPcf + Batch[0-5] + > TexFilterAniso, GpuTest GiMark + 32-bit Julia > > And with patch: > > 15-20% MemBW GPU texture > 10% SynMark TexMem* > 8-9% GLB Egypt (1/2 screen window) > 4-5% GLB T-Rex (1/2 screen window) > 3-6% GfxBench Manhattan 3.0, GpuTest FurMark, > SynMark Deferred + TexFilterTri > 3-4% GfxBench Manhattan 3.1 + T-Rex, SynMark VSInstancing > 2-4% GpuTest Triangle, SynMark DeferredAA > 2-3% Unigine Heaven + Valley > 1-3% SynMark Terrain* > 1-2% GfxBench CarChase, SynMark TexFilterAniso + ZBuffer > > Tigerlake-H: > > 20-25% MemBW GPU texture > 15-20% GpuTest Triangle > 13-15% SynMark TerrainFly* + DeferredAA + HdrBloom > 8-10% GfxBench Manhattan 3.1, SynMark TerrainPan* + DrvRes > 6-7% GfxBench Manhattan 3.0, SynMark TexMem* > 4-8% GLB onscreen Fill + T-Rex + Egypt (more in onscreen than > offscreen versions of T-Rex/Egypt) > 4-6% GfxBench CarChase + GLES AztecRuins + ALU2, GpuTest 32-bit Julia, > SynMark CSDof + DrvState > 3-5% GfxBench T-Rex + Egypt, Unigine Heaven + Valley, GpuTest Plot3D > 1-7% Media tests > 2-3% MemBW GPU blit > 1-3% Most of the rest of 3D tests > > With the patch: > > 6-8% MemBW GPU blend => the only regression in these tests (compared > to IOMMU without THP) > 4-6% SynMark DrvState (not impacted) + HdrBloom (improved) > 3-4% GLB T-Rex > ~3% GLB Egypt, SynMark DrvRes > 1-3% GfxBench T-Rex + Egypt, SynMark TexFilterTri > 1-2% GfxBench CarChase + GLES AztecRuins, Unigine Valley, > GpuTest Triangle > ~1% GfxBench Manhattan 3.0/3.1, Unigine Heaven > > Perf of several tests actually improved with IOMMU + THP, compared to no > IOMMU / no THP: > > 10-15% SynMark Batch[0-3] > 5-10% MemBW GPU texture, SynMark ShMapVsm > 3-4% SynMark Fill* + Geom* > 2-3% SynMark TexMem512 + CSCloth > 1-2% SynMark TexMem128 + DeferredAA > > v2: > * Add Kconfig dependency to transparent hugepages and some help text. > * Move to helper for easier handling of kernel build options. > > v3: > * Drop Kconfig. (Daniel) > > v4: > * Add some benchmark results to commit message. > > References: b901bb89324a ("drm/i915/gemfs: enable THP") > References: 9987da4b5dcf ("drm/i915: Disable THP until we have a GPU read BW W/A") > References: https://gitlab.freedesktop.org/drm/intel/-/issues/430 > Co-developed-by: Chris Wilson <chris@chris-wilson.co.uk> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> > Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> > Cc: Matthew Auld <matthew.auld@intel.com> > Cc: Eero Tamminen <eero.t.tamminen@intel.com> > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> > Cc: Daniel Vetter <daniel@ffwll.ch> > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com> # v1 > --- > drivers/gpu/drm/i915/gem/i915_gemfs.c | 22 +++++++++++++++++++--- > 1 file changed, 19 insertions(+), 3 deletions(-) > > diff --git a/drivers/gpu/drm/i915/gem/i915_gemfs.c b/drivers/gpu/drm/i915/gem/i915_gemfs.c > index 5e6e8c91ab38..dbdbdc344d87 100644 > --- a/drivers/gpu/drm/i915/gem/i915_gemfs.c > +++ b/drivers/gpu/drm/i915/gem/i915_gemfs.c > @@ -6,7 +6,6 @@ > > #include <linux/fs.h> > #include <linux/mount.h> > -#include <linux/pagemap.h> > > #include "i915_drv.h" > #include "i915_gemfs.h" > @@ -15,6 +14,7 @@ int i915_gemfs_init(struct drm_i915_private *i915) > { > struct file_system_type *type; > struct vfsmount *gemfs; > + char *opts; > > type = get_fs_type("tmpfs"); > if (!type) > @@ -26,10 +26,26 @@ int i915_gemfs_init(struct drm_i915_private *i915) > * > * One example, although it is probably better with a per-file > * control, is selecting huge page allocations ("huge=within_size"). > - * Currently unused due to bandwidth issues (slow reads) on Broadwell+. > + * However, we only do so to offset the overhead of iommu lookups > + * due to bandwidth issues (slow reads) on Broadwell+. > */ > > - gemfs = kern_mount(type); > + opts = NULL; > + if (intel_vtd_active()) { > + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { > + static char huge_opt[] = "huge=within_size"; /* r/w */ > + > + opts = huge_opt; > + drm_info(&i915->drm, > + "Transparent Hugepage mode '%s'\n", > + opts); > + } else { > + drm_notice(&i915->drm, > + "Transparent Hugepage support is recommended for optimal performance when IOMMU is enabled!\n"); > + } > + } > + > + gemfs = vfs_kern_mount(type, SB_KERNMOUNT, type->name, opts); > if (IS_ERR(gemfs)) > return PTR_ERR(gemfs); > >
On 07/09/2021 12:13, Eero Tamminen wrote: > Hi, > > For completeness sake, it might be worth mentioning specifically what > (synthetic) test-cases regress with THP patch. > > * Skylake GT4e: > 20-25% SynMark TexMem* > (whereas all MemBW GPU tests either improve or are not affected) > > * Broxton J4205: > 7% MemBW GPU texture > 2-3% SynMark TexMem* > > * Tigerlake-H: > 7% MemBW GPU blend Ah right that makes sense. All the entries marker with asterisk under the "with patch" list. Okay if I just add an explanation on what does the asterisk mean for them at a single place? And about the Broxton one. In the bug you put "15-20% MemBW GPU texture" and "10% SynMark TexMem*" so from where are these numbers now? > > I have no idea why on GEN9 texture accesses regress, but on GEN12 TGL > it's render buffer blend that regresses. > > Blend (read+write) regressing is especially odd, as neither render > buffer read nor write regresses. > > Maybe that is a GEN12 specific driver bug similar to Mesa/i965 bug from > few years back in how its shaders access render buffer, that had caused > SIMD32 accesses to regress memory BW bound test-cases perf a bit > compared to SIMD16? > > (Blend test is likely to run nowadays as SIMD32.) No idea on this one from me, leaving to more qualified people to comment. Regards, Tvrtko > > - Eero > > On 7.9.2021 13.34, Tvrtko Ursulin wrote: >> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >> >> Usage of Transparent Hugepages was disabled in 9987da4b5dcf >> ("drm/i915: Disable THP until we have a GPU read BW W/A"), but since it >> appears majority of performance regressions reported with an enabled >> IOMMU >> can be almost eliminated by turning them on, lets just do that. >> >> To err on the side of safety we keep the current default in cases where >> IOMMU is not active, and only when it is default to the >> "huge=within_size" >> mode. Although there probably would be wins to enable them throughout, >> more extensive testing across benchmarks and platforms would need to be >> done. >> >> With the patch and IOMMU enabled my local testing on a small Skylake part >> shows OglVSTangent regression being reduced from ~14% (IOMMU on versus >> IOMMU off) to ~2% (same comparison but with THP on). >> >> More detailed testing done in the below referenced Gitlab issue by Eero: >> >> Skylake GT4e: >> >> Performance drops from enabling IOMMU: >> >> 30-35% SynMark CSDof >> 20-25% Unigine Heaven, MemBW GPU write, SynMark VSTangent >> ~20% GLB Egypt (1/2 screen window) >> 10-15% GLB T-Rex (1/2 screen window) >> 8-10% GfxBench T-Rex, MemBW GPU blit >> 7-8% SynMark DeferredAA + TerrainFly* + ZBuffer >> 6-7% GfxBench Manhattan 3.0 + 3.1, SynMark TexMem128 & CSCloth >> 5-6% GfxBench CarChase, Unigine Valley >> 3-5% GfxBench Vulkan & GL AztecRuins + ALU2, MemBW GPU texture, >> SynMark Fill*, Deferred, TerrainPan* >> 1-2% Most of the other tests >> >> With the patch drops become: >> >> 20-25% SynMark TexMem* >> 15-20% GLB Egypt (1/2 screen window) >> 10-15% GLB T-Rex (1/2 screen window) >> 4-7% GfxBench T-Rex, GpuTest Triangle >> 1-8% GfxBench ALU2 (offscreen 1%, onscreen 8%) >> 3% GfxBench Manhattan 3.0, SynMark CSDof >> 2-3% Unigine Heaven + Valley, MemBW GPU texture >> 1-3 GfxBench Manhattan 3.1 + CarChase + Vulkan & GL AztecRuins >> >> Broxton: >> >> Performance drops from IOMMU, without patch: >> >> 30% MemBW GPU write >> 25% SynMark ZBuffer + Fill* >> 20% MemBW GPU blit >> 15% MemBW GPU blend, GpuTest Triangle >> 10-15% MemBW GPU texture >> 10% GLB Egypt, Unigine Heaven (had hangs), SynMark TerrainFly* >> 7-9% GLB T-Rex, GfxBench Manhattan 3.0 + T-Rex, >> SynMark Deferred* + TexMem* >> 6-8% GfxBench CarChase, Unigine Valley, >> SynMark CSCloth + ShMapVsm + TerrainPan* >> 5-6% GfxBench Manhattan 3.1 + GL AztecRuins, >> SynMark CSDof + TexFilterTri >> 2-4% GfxBench ALU2, SynMark DrvRes + GSCloth + ShMapPcf + >> Batch[0-5] + >> TexFilterAniso, GpuTest GiMark + 32-bit Julia >> >> And with patch: >> >> 15-20% MemBW GPU texture >> 10% SynMark TexMem* >> 8-9% GLB Egypt (1/2 screen window) >> 4-5% GLB T-Rex (1/2 screen window) >> 3-6% GfxBench Manhattan 3.0, GpuTest FurMark, >> SynMark Deferred + TexFilterTri >> 3-4% GfxBench Manhattan 3.1 + T-Rex, SynMark VSInstancing >> 2-4% GpuTest Triangle, SynMark DeferredAA >> 2-3% Unigine Heaven + Valley >> 1-3% SynMark Terrain* >> 1-2% GfxBench CarChase, SynMark TexFilterAniso + ZBuffer >> >> Tigerlake-H: >> >> 20-25% MemBW GPU texture >> 15-20% GpuTest Triangle >> 13-15% SynMark TerrainFly* + DeferredAA + HdrBloom >> 8-10% GfxBench Manhattan 3.1, SynMark TerrainPan* + DrvRes >> 6-7% GfxBench Manhattan 3.0, SynMark TexMem* >> 4-8% GLB onscreen Fill + T-Rex + Egypt (more in onscreen than >> offscreen versions of T-Rex/Egypt) >> 4-6% GfxBench CarChase + GLES AztecRuins + ALU2, GpuTest 32-bit >> Julia, >> SynMark CSDof + DrvState >> 3-5% GfxBench T-Rex + Egypt, Unigine Heaven + Valley, GpuTest Plot3D >> 1-7% Media tests >> 2-3% MemBW GPU blit >> 1-3% Most of the rest of 3D tests >> >> With the patch: >> >> 6-8% MemBW GPU blend => the only regression in these tests (compared >> to IOMMU without THP) >> 4-6% SynMark DrvState (not impacted) + HdrBloom (improved) >> 3-4% GLB T-Rex >> ~3% GLB Egypt, SynMark DrvRes >> 1-3% GfxBench T-Rex + Egypt, SynMark TexFilterTri >> 1-2% GfxBench CarChase + GLES AztecRuins, Unigine Valley, >> GpuTest Triangle >> ~1% GfxBench Manhattan 3.0/3.1, Unigine Heaven >> >> Perf of several tests actually improved with IOMMU + THP, compared to no >> IOMMU / no THP: >> >> 10-15% SynMark Batch[0-3] >> 5-10% MemBW GPU texture, SynMark ShMapVsm >> 3-4% SynMark Fill* + Geom* >> 2-3% SynMark TexMem512 + CSCloth >> 1-2% SynMark TexMem128 + DeferredAA >> >> v2: >> * Add Kconfig dependency to transparent hugepages and some help text. >> * Move to helper for easier handling of kernel build options. >> >> v3: >> * Drop Kconfig. (Daniel) >> >> v4: >> * Add some benchmark results to commit message. >> >> References: b901bb89324a ("drm/i915/gemfs: enable THP") >> References: 9987da4b5dcf ("drm/i915: Disable THP until we have a GPU >> read BW W/A") >> References: https://gitlab.freedesktop.org/drm/intel/-/issues/430 >> Co-developed-by: Chris Wilson <chris@chris-wilson.co.uk> >> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> >> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> >> Cc: Matthew Auld <matthew.auld@intel.com> >> Cc: Eero Tamminen <eero.t.tamminen@intel.com> >> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> >> Cc: Daniel Vetter <daniel@ffwll.ch> >> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com> # v1 >> --- >> drivers/gpu/drm/i915/gem/i915_gemfs.c | 22 +++++++++++++++++++--- >> 1 file changed, 19 insertions(+), 3 deletions(-) >> >> diff --git a/drivers/gpu/drm/i915/gem/i915_gemfs.c >> b/drivers/gpu/drm/i915/gem/i915_gemfs.c >> index 5e6e8c91ab38..dbdbdc344d87 100644 >> --- a/drivers/gpu/drm/i915/gem/i915_gemfs.c >> +++ b/drivers/gpu/drm/i915/gem/i915_gemfs.c >> @@ -6,7 +6,6 @@ >> #include <linux/fs.h> >> #include <linux/mount.h> >> -#include <linux/pagemap.h> >> #include "i915_drv.h" >> #include "i915_gemfs.h" >> @@ -15,6 +14,7 @@ int i915_gemfs_init(struct drm_i915_private *i915) >> { >> struct file_system_type *type; >> struct vfsmount *gemfs; >> + char *opts; >> type = get_fs_type("tmpfs"); >> if (!type) >> @@ -26,10 +26,26 @@ int i915_gemfs_init(struct drm_i915_private *i915) >> * >> * One example, although it is probably better with a per-file >> * control, is selecting huge page allocations >> ("huge=within_size"). >> - * Currently unused due to bandwidth issues (slow reads) on >> Broadwell+. >> + * However, we only do so to offset the overhead of iommu lookups >> + * due to bandwidth issues (slow reads) on Broadwell+. >> */ >> - gemfs = kern_mount(type); >> + opts = NULL; >> + if (intel_vtd_active()) { >> + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { >> + static char huge_opt[] = "huge=within_size"; /* r/w */ >> + >> + opts = huge_opt; >> + drm_info(&i915->drm, >> + "Transparent Hugepage mode '%s'\n", >> + opts); >> + } else { >> + drm_notice(&i915->drm, >> + "Transparent Hugepage support is recommended for >> optimal performance when IOMMU is enabled!\n"); >> + } >> + } >> + >> + gemfs = vfs_kern_mount(type, SB_KERNMOUNT, type->name, opts); >> if (IS_ERR(gemfs)) >> return PTR_ERR(gemfs); >>
diff --git a/drivers/gpu/drm/i915/gem/i915_gemfs.c b/drivers/gpu/drm/i915/gem/i915_gemfs.c index 5e6e8c91ab38..dbdbdc344d87 100644 --- a/drivers/gpu/drm/i915/gem/i915_gemfs.c +++ b/drivers/gpu/drm/i915/gem/i915_gemfs.c @@ -6,7 +6,6 @@ #include <linux/fs.h> #include <linux/mount.h> -#include <linux/pagemap.h> #include "i915_drv.h" #include "i915_gemfs.h" @@ -15,6 +14,7 @@ int i915_gemfs_init(struct drm_i915_private *i915) { struct file_system_type *type; struct vfsmount *gemfs; + char *opts; type = get_fs_type("tmpfs"); if (!type) @@ -26,10 +26,26 @@ int i915_gemfs_init(struct drm_i915_private *i915) * * One example, although it is probably better with a per-file * control, is selecting huge page allocations ("huge=within_size"). - * Currently unused due to bandwidth issues (slow reads) on Broadwell+. + * However, we only do so to offset the overhead of iommu lookups + * due to bandwidth issues (slow reads) on Broadwell+. */ - gemfs = kern_mount(type); + opts = NULL; + if (intel_vtd_active()) { + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + static char huge_opt[] = "huge=within_size"; /* r/w */ + + opts = huge_opt; + drm_info(&i915->drm, + "Transparent Hugepage mode '%s'\n", + opts); + } else { + drm_notice(&i915->drm, + "Transparent Hugepage support is recommended for optimal performance when IOMMU is enabled!\n"); + } + } + + gemfs = vfs_kern_mount(type, SB_KERNMOUNT, type->name, opts); if (IS_ERR(gemfs)) return PTR_ERR(gemfs);