From patchwork Wed May 1 14:04:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Welty, Brian" X-Patchwork-Id: 10925097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6E2F1390 for ; Wed, 1 May 2019 14:03:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A564428E52 for ; Wed, 1 May 2019 14:03:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9971928BAA; Wed, 1 May 2019 14:03:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4B09628D32 for ; Wed, 1 May 2019 14:03:06 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7BFEC892FD; Wed, 1 May 2019 14:03:02 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 04C57892A9; Wed, 1 May 2019 14:03:00 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 May 2019 07:03:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,417,1549958400"; d="scan'208";a="145141383" Received: from nperf12.hd.intel.com ([10.127.88.161]) by fmsmga008.fm.intel.com with ESMTP; 01 May 2019 07:02:58 -0700 From: Brian Welty To: cgroups@vger.kernel.org, Tejun Heo , Li Zefan , Johannes Weiner , linux-mm@kvack.org, Michal Hocko , Vladimir Davydov , dri-devel@lists.freedesktop.org, David Airlie , Daniel Vetter , intel-gfx@lists.freedesktop.org, Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , =?utf-8?q?Christian_K=C3=B6nig?= , Alex Deucher , ChunMing Zhou , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= Subject: [RFC PATCH 0/5] cgroup support for GPU devices Date: Wed, 1 May 2019 10:04:33 -0400 Message-Id: <20190501140438.9506-1-brian.welty@intel.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP In containerized or virtualized environments, there is desire to have controls in place for resources that can be consumed by users of a GPU device. This RFC patch series proposes a framework for integrating use of existing cgroup controllers into device drivers. The i915 driver is updated in this series as our primary use case to leverage this framework and to serve as an example for discussion. The patch series enables device drivers to use cgroups to control the following resources within a GPU (or other accelerator device): * control allocation of device memory (reuse of memcg) and with future work, we could extend to: * track and control share of GPU time (reuse of cpu/cpuacct) * apply mask of allowed execution engines (reuse of cpusets) Instead of introducing a new cgroup subsystem for GPU devices, a new framework is proposed to allow devices to register with existing cgroup controllers, which creates per-device cgroup_subsys_state within the cgroup. This gives device drivers their own private cgroup controls (such as memory limits or other parameters) to be applied to device resources instead of host system resources. Device drivers (GPU or other) are then able to reuse the existing cgroup controls, instead of inventing similar ones. Per-device controls would be exposed in cgroup filesystem as: mount//.devices// such as (for example): mount//memory.devices//memory.max mount//memory.devices//memory.current mount//cpu.devices//cpu.stat mount//cpu.devices//cpu.weight The drm/i915 patch in this series is based on top of other RFC work [1] for i915 device memory support. AMD [2] and Intel [3] have proposed related work in this area within the last few years, listed below as reference. This new RFC reuses existing cgroup controllers and takes a different approach than prior work. Finally, some potential discussion points for this series: * merge proposed .devices into a single devices directory? * allow devices to have multiple registrations for subsets of resources? * document a 'common charging policy' for device drivers to follow? [1] https://patchwork.freedesktop.org/series/56683/ [2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html [3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html Brian Welty (5): cgroup: Add cgroup_subsys per-device registration framework cgroup: Change kernfs_node for directories to store cgroup_subsys_state memcg: Add per-device support to memory cgroup subsystem drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit drm/i915: Use memory cgroup for enforcing device memory limit drivers/gpu/drm/drm_drv.c | 12 + drivers/gpu/drm/drm_gem.c | 7 + drivers/gpu/drm/i915/i915_drv.c | 2 +- drivers/gpu/drm/i915/intel_memory_region.c | 24 +- include/drm/drm_device.h | 3 + include/drm/drm_drv.h | 8 + include/drm/drm_gem.h | 11 + include/linux/cgroup-defs.h | 28 ++ include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 10 + kernel/cgroup/cgroup-v1.c | 10 +- kernel/cgroup/cgroup.c | 310 ++++++++++++++++++--- mm/memcontrol.c | 183 +++++++++++- 13 files changed, 552 insertions(+), 59 deletions(-)