From patchwork Tue May 30 12:55:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 9754589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DA654602F0 for ; Tue, 30 May 2017 12:55:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCAEE2684F for ; Tue, 30 May 2017 12:55:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1AAB26C9B; Tue, 30 May 2017 12:55:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6A5BA2684F for ; Tue, 30 May 2017 12:55:40 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D4B8289EBD; Tue, 30 May 2017 12:55:35 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6C53089EBD; Tue, 30 May 2017 12:55:34 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from haswell.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 7033775-1500050 for multiple; Tue, 30 May 2017 13:55:20 +0100 Received: by haswell.alporthouse.com (sSMTP sendmail emulation); Tue, 30 May 2017 13:55:20 +0100 From: Chris Wilson To: dri-devel@lists.freedesktop.org Subject: [RFC] drm: Optimise drm_ioctl() for small user args Date: Tue, 30 May 2017 13:55:20 +0100 Message-Id: <20170530125520.14030-1-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.11.0 X-Originating-IP: 78.156.65.138 X-Country: code=GB country="United Kingdom" ip=78.156.65.138 Cc: intel-gfx@lists.freedesktop.org, Joonas Lahtinen X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP When looking at simple ioctls coupled with conveniently small user parameters, the overhead of the syscall and drm_ioctl() present large low hanging fruit. Profiling trivial microbenchmarks around i915_gem_busy_ioctl, the low hanging fruit comprises of the call to copy_user(). Those calls are only inlined by the macro where the constant is known at compile-time, but the ioctl argument size depends on the ioctl number. To help the compiler, explicitly add switches for the small sizes that expand to simple moves to/from user. Doing the multiple inlines does add significant code bloat, so it is very debatable as to its value. Back to the trivial, but frequently used, example of i915_gem_busy_ioctl() on a Broadwell avoiding the call gives us a 15-25% improvement: before after single 100.173ns 84.496ns parallel (x4) 204.275ns 152.957ns On a baby Broxton nuc: before after single 245.355ns 199.477ns parallel (x2) 280.892ns 232.726ns Looking at the cost distribution by moving an equivalent switch into arch/x86/lib/usercopy, the overhead to the copy user is split almost equally between the function call and the actual copy itself. It seems copy_user_enhanced_fast_string simply is not that good at small (single register) copies. Something as simple as @@ -28,6 +28,9 @@ copy_user_generic(void *to, const void *from, unsigned len) { unsigned ret; + if (len <= 16) + return copy_user_generic_unrolled(to, from, len); is enough to speed up i915_gem_busy_ioctl() by 10% :| Note that this overhead may entirely be x86 specific. Signed-off-by: Chris Wilson Cc: Joonas Lahtinen Reviewed-by: Joonas Lahtinen --- drivers/gpu/drm/drm_ioctl.c | 111 ++++++++++++++++++++++++++++++++------------ 1 file changed, 82 insertions(+), 29 deletions(-) diff --git a/drivers/gpu/drm/drm_ioctl.c b/drivers/gpu/drm/drm_ioctl.c index 865e3ee4d743..93ba59a30a85 100644 --- a/drivers/gpu/drm/drm_ioctl.c +++ b/drivers/gpu/drm/drm_ioctl.c @@ -715,11 +715,11 @@ long drm_ioctl(struct file *filp, const struct drm_ioctl_desc *ioctl = NULL; drm_ioctl_t *func; unsigned int nr = DRM_IOCTL_NR(cmd); - int retcode = -EINVAL; char stack_kdata[128]; - char *kdata = NULL; + char *kdata = stack_kdata; unsigned int in_size, out_size, drv_size, ksize; bool is_driver_ioctl; + int retcode; dev = file_priv->minor->dev; @@ -731,12 +731,12 @@ long drm_ioctl(struct file *filp, if (is_driver_ioctl) { /* driver ioctl */ if (nr - DRM_COMMAND_BASE >= dev->driver->num_ioctls) - goto err_i1; + goto err_invalid_ioctl; ioctl = &dev->driver->ioctls[nr - DRM_COMMAND_BASE]; } else { /* core ioctl */ if (nr >= DRM_CORE_IOCTL_COUNT) - goto err_i1; + goto err_invalid_ioctl; ioctl = &drm_ioctls[nr]; } @@ -758,29 +758,50 @@ long drm_ioctl(struct file *filp, if (unlikely(!func)) { DRM_DEBUG("no function\n"); - retcode = -EINVAL; - goto err_i1; + goto err_invalid; } retcode = drm_ioctl_permit(ioctl->flags, file_priv); if (unlikely(retcode)) - goto err_i1; - - if (ksize <= sizeof(stack_kdata)) { - kdata = stack_kdata; - } else { - kdata = kmalloc(ksize, GFP_KERNEL); - if (!kdata) { - retcode = -ENOMEM; - goto err_i1; + goto out; + + if (in_size) { + if (unlikely(!access_ok(VERIFY_READ, arg, in_size))) + goto err_invalid_user; + + switch (in_size) { + case 4: + if (unlikely(__copy_from_user(kdata, (void __user *)arg, + 4))) + goto err_invalid_user; + break; + case 8: + if (unlikely(__copy_from_user(kdata, (void __user *)arg, + 8))) + goto err_invalid_user; + break; + case 16: + if (unlikely(__copy_from_user(kdata, (void __user *)arg, + 16))) + goto err_invalid_user; + break; + + default: + if (ksize > sizeof(stack_kdata)) { + kdata = kmalloc(ksize, GFP_KERNEL); + if (unlikely(!kdata)) { + retcode = -ENOMEM; + goto out; + } + } + + if (unlikely(__copy_from_user(kdata, (void __user *)arg, + in_size))) + goto err_invalid_user; + break; } } - if (copy_from_user(kdata, (void __user *)arg, in_size) != 0) { - retcode = -EFAULT; - goto err_i1; - } - if (ksize > in_size) memset(kdata + in_size, 0, ksize - in_size); @@ -794,21 +815,53 @@ long drm_ioctl(struct file *filp, mutex_unlock(&drm_global_mutex); } - if (copy_to_user((void __user *)arg, kdata, out_size) != 0) - retcode = -EFAULT; - - err_i1: - if (!ioctl) - DRM_DEBUG("invalid ioctl: pid=%d, dev=0x%lx, auth=%d, cmd=0x%02x, nr=0x%02x\n", - task_pid_nr(current), - (long)old_encode_dev(file_priv->minor->kdev->devt), - file_priv->authenticated, cmd, nr); + if (out_size) { + if (unlikely(!access_ok(VERIFY_WRITE, arg, out_size))) + goto err_invalid_user; + + switch (out_size) { + case 4: + if (unlikely(__copy_to_user((void __user *)arg, + kdata, 4))) + goto err_invalid_user; + break; + case 8: + if (unlikely(__copy_to_user((void __user *)arg, + kdata, 8))) + goto err_invalid_user; + break; + case 16: + if (unlikely(__copy_to_user((void __user *)arg, + kdata, 16))) + goto err_invalid_user; + break; + default: + if (unlikely(__copy_to_user((void __user *)arg, + kdata, out_size))) + goto err_invalid_user; + break; + } + } +out: if (kdata != stack_kdata) kfree(kdata); if (retcode) DRM_DEBUG("ret = %d\n", retcode); return retcode; + +err_invalid_ioctl: + DRM_DEBUG("invalid ioctl: pid=%d, dev=0x%lx, auth=%d, cmd=0x%02x, nr=0x%02x\n", + task_pid_nr(current), + (long)old_encode_dev(file_priv->minor->kdev->devt), + file_priv->authenticated, cmd, nr); +err_invalid: + retcode = -EINVAL; + goto out; + +err_invalid_user: + retcode = -EFAULT; + goto out; } EXPORT_SYMBOL(drm_ioctl);