From patchwork Wed Feb 26 19:01:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407061 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1B4914B4 for ; Wed, 26 Feb 2020 19:02:19 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8F4232072D for ; Wed, 26 Feb 2020 19:02:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="Akw57VC8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F4232072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 370766EB91; Wed, 26 Feb 2020 19:02:10 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-eopbgr770075.outbound.protection.outlook.com [40.107.77.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2F07A6EB8D; Wed, 26 Feb 2020 19:02:07 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bcTe3AeUT+R87cb6775rKmvDnrfdMukKmcZQh811PKy8YUuAx3dnF1Ukrssyfrbz2KyX7Kd2iTyQkrMvDQ5S6+95i8TU805kXGMx3A+Pqypek68bMldB9TGTQGsUn/oHQcq3a4aE5/E0IOPo8uc8tfxQR4RUypBTlwrql1HTP1q22dZpNUzJbRNzHBLH6FrpiCVkr9gNTp+THjeOEmJB5ZJvL0W4JileL7bUswy4b7NlE/UX2wJkn9tjZ41huYT21p4kQtry/NyKcKp+DBJflm1WXijB1oMJYi3nkCog/4mX6oc2pu6Tur2QR41Cyi3QjOCIdStqfXZmd0kvS3iJwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NZBN1682Pm8WEeV5XlYZHllyETV/AMrAB1zuVH+Z6xA=; b=BzaItQU8x1Cz0bbu8XKhZgV8m3Uqf1GhqUUhC+v8x7U3OCGUUofrckTto4vFCiK8trUzWLneCZ1tNM1pmJwD8sd5zzCSP4705n+OHRTjolVFeAeB7nIb9sjDiYFBW2jwoHPRJwO5fPzt/MNawfnarwoJjBzF+V7EWU6ksqxmHlH6K0QwclKN5Ef1WSTPglB2tN+N7Tz0i0ZCzO/+aXL8oV1rzkO/FxOHb/1HO+8YC11bUTyrIqdme/fsvLOXng6Og9Q39YJNRO+jrBco+7Meljef9bvqM3eWm9aV1Q4qxDZ2DH7If9k2buQqjjPH9HgFr3Q06tGrCUmdCRqQqb2UiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NZBN1682Pm8WEeV5XlYZHllyETV/AMrAB1zuVH+Z6xA=; b=Akw57VC8d3PeP1KRRkXGInw8XIE5sCzqaIrC/8ZqI72sEQU4OBIqJrqRnGI7kbjXs/IQRdHldCfyi6cdkP7stRrsmpILYRnw7BP1OUbJmXpm7bHQXd8e9oTL4AuNwuJw5NVib+3M4ItCG7ZOyW21qmJxpHSs4OvTQlNEhlZLgTc= Received: from DM5PR19CA0027.namprd19.prod.outlook.com (2603:10b6:3:9a::13) by MN2PR12MB3311.namprd12.prod.outlook.com (2603:10b6:208:100::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:04 +0000 Received: from DM6NAM11FT063.eop-nam11.prod.protection.outlook.com (2603:10b6:3:9a:cafe::a1) by DM5PR19CA0027.outlook.office365.com (2603:10b6:3:9a::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:04 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by DM6NAM11FT063.mail.protection.outlook.com (10.13.172.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:04 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:03 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:03 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:02 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 01/11] cgroup: Introduce cgroup for drm subsystem Date: Wed, 26 Feb 2020 14:01:42 -0500 Message-ID: <20200226190152.16131-2-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(376002)(396003)(136003)(346002)(39860400002)(428003)(199004)(189003)(86362001)(336012)(2906002)(26005)(70206006)(81156014)(186003)(426003)(81166006)(110136005)(70586007)(8936002)(4326008)(36756003)(478600001)(8676002)(2616005)(316002)(5660300002)(6666004)(1076003)(356004)(7696005)(921003)(1121003)(83996005)(2101003); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR12MB3311; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9ffe2dd4-b3bc-45bd-57e2-08d7baee5e7b X-MS-TrafficTypeDiagnostic: MN2PR12MB3311: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 62qpnapjYYAefi+/vAtsCj6rWUYg7FJNKlcr77ZBTnfNZKzDu6Hy3iXqykXlqPj+/KI9RH/1JNw4clDBAFa6bSDhZp9p32I5B94KTip1Jm7IJfitbiv5DgsZ871oC/f4zDdyZ8A3bkIE91DNHKkWLTJU/MJlZMi7LuCRzqeLQzorT5xF4a1SZ/s389l4+eSvPXu/zGTeXBKlJ9fuxENpn5IGJRAdMA33BuR2PuXAOa99TSFYgddZcAQFXZVksd6qoPOcs+U+Ycihq/U+mxbCyXO9idU2Blg5NlHYRu8S3ewdwj4KdFEx51UUBNnviPnPZ6dGKL5qVEu6reedTLT2Gdl43Dn+8JjkpAyQw5DaTaQ/zWeeGMYFP3xA7DRQ21Yy/ncWCEj+5dvKzdKRnc60GGG5cTEWFIcpHWTn9owxvSbWNIXjDisWpEzc4Efswf1b943ZQ2ZDvqQRNtyboYyXqTHc6RkBohPxUuaBcZFYtDzC2c0ckXA5KzUruD0daEdtXzkdJnL7s2zr6lA+bj015hr8G/UuJUH4ijmNn6qYgv0= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:04.5419 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9ffe2dd4-b3bc-45bd-57e2-08d7baee5e7b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3311 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" With the increased importance of machine learning, data science and other cloud-based applications, GPUs are already in production use in data centers today. Existing GPU resource management is very coarse grain, however, as sysadmins are only able to distribute workload on a per-GPU basis. An alternative is to use GPU virtualization (with or without SRIOV) but it generally acts on the entire GPU instead of the specific resources in a GPU. With a drm cgroup controller, we can enable alternate, fine-grain, sub-GPU resource management (in addition to what may be available via GPU virtualization.) Change-Id: Ia90aed8c4cb89ff20d8216a903a765655b44fc9a Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 18 ++++- Documentation/cgroup-v1/drm.rst | 1 + include/linux/cgroup_drm.h | 92 +++++++++++++++++++++++++ include/linux/cgroup_subsys.h | 4 ++ init/Kconfig | 5 ++ kernel/cgroup/Makefile | 1 + kernel/cgroup/drm.c | 42 +++++++++++ 7 files changed, 161 insertions(+), 2 deletions(-) create mode 100644 Documentation/cgroup-v1/drm.rst create mode 100644 include/linux/cgroup_drm.h create mode 100644 kernel/cgroup/drm.c diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 0636bcb60b5a..7deff912185e 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -61,8 +61,10 @@ v1 is available under Documentation/admin-guide/cgroup-v1/. 5-6. Device 5-7. RDMA 5-7-1. RDMA Interface Files - 5-8. Misc - 5-8-1. perf_event + 5-8. GPU + 5-8-1. GPU Interface Files + 5-9. Misc + 5-9-1. perf_event 5-N. Non-normative information 5-N-1. CPU controller root cgroup process behaviour 5-N-2. IO controller root cgroup process behaviour @@ -2057,6 +2059,18 @@ RDMA Interface Files ocrdma1 hca_handle=1 hca_object=23 +GPU +--- + +The "gpu" controller regulates the distribution and accounting of +of GPU-related resources. + +GPU Interface Files +~~~~~~~~~~~~~~~~~~~~ + +TODO + + Misc ---- diff --git a/Documentation/cgroup-v1/drm.rst b/Documentation/cgroup-v1/drm.rst new file mode 100644 index 000000000000..5f5658e1f5ed --- /dev/null +++ b/Documentation/cgroup-v1/drm.rst @@ -0,0 +1 @@ +Please see ../cgroup-v2.rst for details diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h new file mode 100644 index 000000000000..345af54a5d41 --- /dev/null +++ b/include/linux/cgroup_drm.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: MIT + * Copyright 2019 Advanced Micro Devices, Inc. + */ +#ifndef _CGROUP_DRM_H +#define _CGROUP_DRM_H + +#include + +#ifdef CONFIG_CGROUP_DRM + +/** + * The DRM cgroup controller data structure. + */ +struct drmcg { + struct cgroup_subsys_state css; +}; + +/** + * css_to_drmcg - get the corresponding drmcg ref from a cgroup_subsys_state + * @css: the target cgroup_subsys_state + * + * Return: DRM cgroup that contains the @css + */ +static inline struct drmcg *css_to_drmcg(struct cgroup_subsys_state *css) +{ + return css ? container_of(css, struct drmcg, css) : NULL; +} + +/** + * drmcg_get - get the drmcg reference that a task belongs to + * @task: the target task + * + * This increase the reference count of the css that the @task belongs to + * + * Return: reference to the DRM cgroup the task belongs to + */ +static inline struct drmcg *drmcg_get(struct task_struct *task) +{ + return css_to_drmcg(task_get_css(task, gpu_cgrp_id)); +} + +/** + * drmcg_put - put a drmcg reference + * @drmcg: the target drmcg + * + * Put a reference obtained via drmcg_get + */ +static inline void drmcg_put(struct drmcg *drmcg) +{ + if (drmcg) + css_put(&drmcg->css); +} + +/** + * drmcg_parent - find the parent of a drm cgroup + * @cg: the target drmcg + * + * This does not increase the reference count of the parent cgroup + * + * Return: parent DRM cgroup of @cg + */ +static inline struct drmcg *drmcg_parent(struct drmcg *cg) +{ + return css_to_drmcg(cg->css.parent); +} + +#else /* CONFIG_CGROUP_DRM */ + +struct drmcg { +}; + +static inline struct drmcg *css_to_drmcg(struct cgroup_subsys_state *css) +{ + return NULL; +} + +static inline struct drmcg *drmcg_get(struct task_struct *task) +{ + return NULL; +} + +static inline void drmcg_put(struct drmcg *drmcg) +{ +} + +static inline struct drmcg *drmcg_parent(struct drmcg *cg) +{ + return NULL; +} + +#endif /* CONFIG_CGROUP_DRM */ +#endif /* _CGROUP_DRM_H */ diff --git a/include/linux/cgroup_subsys.h b/include/linux/cgroup_subsys.h index acb77dcff3b4..f4e627942115 100644 --- a/include/linux/cgroup_subsys.h +++ b/include/linux/cgroup_subsys.h @@ -61,6 +61,10 @@ SUBSYS(pids) SUBSYS(rdma) #endif +#if IS_ENABLED(CONFIG_CGROUP_DRM) +SUBSYS(gpu) +#endif + /* * The following subsystems are not supported on the default hierarchy. */ diff --git a/init/Kconfig b/init/Kconfig index a34064a031a5..bb78dff44d9d 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -965,6 +965,11 @@ config CGROUP_RDMA Attaching processes with active RDMA resources to the cgroup hierarchy is allowed even if can cross the hierarchy's limit. +config CGROUP_DRM + bool "DRM controller (EXPERIMENTAL)" + help + Provides accounting and enforcement of resources in the DRM subsystem. + config CGROUP_FREEZER bool "Freezer controller" help diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile index 5d7a76bfbbb7..31f186f58121 100644 --- a/kernel/cgroup/Makefile +++ b/kernel/cgroup/Makefile @@ -4,5 +4,6 @@ obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) += pids.o obj-$(CONFIG_CGROUP_RDMA) += rdma.o +obj-$(CONFIG_CGROUP_DRM) += drm.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_CGROUP_DEBUG) += debug.o diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c new file mode 100644 index 000000000000..5e38a8230922 --- /dev/null +++ b/kernel/cgroup/drm.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: MIT +// Copyright 2019 Advanced Micro Devices, Inc. +#include +#include +#include + +static struct drmcg *root_drmcg __read_mostly; + +static void drmcg_css_free(struct cgroup_subsys_state *css) +{ + struct drmcg *drmcg = css_to_drmcg(css); + + kfree(drmcg); +} + +static struct cgroup_subsys_state * +drmcg_css_alloc(struct cgroup_subsys_state *parent_css) +{ + struct drmcg *parent = css_to_drmcg(parent_css); + struct drmcg *drmcg; + + drmcg = kzalloc(sizeof(struct drmcg), GFP_KERNEL); + if (!drmcg) + return ERR_PTR(-ENOMEM); + + if (!parent) + root_drmcg = drmcg; + + return &drmcg->css; +} + +struct cftype files[] = { + { } /* terminate */ +}; + +struct cgroup_subsys gpu_cgrp_subsys = { + .css_alloc = drmcg_css_alloc, + .css_free = drmcg_css_free, + .early_init = false, + .legacy_cftypes = files, + .dfl_cftypes = files, +}; From patchwork Wed Feb 26 19:01:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407065 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DAC97930 for ; Wed, 26 Feb 2020 19:02:23 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B63EC2072D for ; Wed, 26 Feb 2020 19:02:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="lGjAlDV/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B63EC2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6C7ED6EB9C; Wed, 26 Feb 2020 19:02:10 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2054.outbound.protection.outlook.com [40.107.244.54]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6D6A96E107; Wed, 26 Feb 2020 19:02:06 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cPonj55NW3W4swJP8O23xVx552XApuCRzxz0tTvq1v35QqfSS014mqLYWOSTKBrviP7sVeSXxqCK051fAZXekZ3MeddCFn2nEb7RQrAzQmSW5otJljL4FY2TKZEykxPQk2sARhc27ml1eXz+dW/KO2D1Br5FQKcRFi910+ZMjPENMqYJxvMT4vVAMlAsfAgSDVaUSjBMIv6MaY12YAsOkB8WaHPv7QdIR1J6mySc+VBRApo7qEv4lJCos30Dk2DwK8we4lkFR5REpTlEcCFPnXxUbirQwLsdm3+K1oN8yBug8Ix0UHsyzlKDDmrmby6QGPgXehZJ5O+iSrFljwc0ZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dC80z490x8sni2ZqGHuoJ/3EBLF3zugdZGEK5Ukyck0=; b=ExRAGX0qM0efyV7aaJV33GgMU4apKLCpwDdWoZvlFz4QWugKygJt5SKlGddwNniJ60SwZvpTbWP+BrAsqvQufw8rZZY4g4dZCwQ892fOAsZmHoMykF+Qvk3uUQukV/hYMSRInEEd1DEeBnz4XlZiMnNT/Atyz23mRu4zzTg7LajlnagauFdleiE8G4ELo9TF6LGXSZId71o6EudgELt92g15jwy0EsQbIuoyZhDDYkbM2cg3lMIyZTu5GRs1ifXdgFNgpJ9ZzOtz3OEueSmiqkuokYHzdovfMD8ZxX7tf69YpX9Aflr79PAltN/Xu5LRnHpA8hM74YQ9OprC0rJr/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dC80z490x8sni2ZqGHuoJ/3EBLF3zugdZGEK5Ukyck0=; b=lGjAlDV/6q/ERlytj0/HlD6csUGaIJWXe2RX2tWv37Zsuc8UQpUOf0wLkhKCjFy1xloNzlcrdy5YGMYFZk/XOTjTfChJJJQrAXsJ13qI0aHVLnC7q+8zuROyIgJA4HE+lh6CmBPww9jhOW/VrDYAay9pS4I3F0Mr6Kp9kKp8Hc0= Received: from MN2PR16CA0032.namprd16.prod.outlook.com (2603:10b6:208:134::45) by MN2PR12MB4190.namprd12.prod.outlook.com (2603:10b6:208:1dd::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:05 +0000 Received: from BN8NAM11FT039.eop-nam11.prod.protection.outlook.com (2603:10b6:208:134:cafe::91) by MN2PR16CA0032.outlook.office365.com (2603:10b6:208:134::45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.15 via Frontend Transport; Wed, 26 Feb 2020 19:02:05 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT039.mail.protection.outlook.com (10.13.177.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:05 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:04 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:03 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:03 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 02/11] drm, cgroup: Bind drm and cgroup subsystem Date: Wed, 26 Feb 2020 14:01:43 -0500 Message-ID: <20200226190152.16131-3-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(136003)(346002)(39860400002)(428003)(189003)(199004)(81156014)(110136005)(2616005)(8676002)(70206006)(426003)(6666004)(26005)(70586007)(81166006)(186003)(356004)(1076003)(8936002)(478600001)(7696005)(86362001)(316002)(5660300002)(2906002)(36756003)(4326008)(336012)(921003)(2101003)(83996005)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR12MB4190; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f62ef0fb-7496-4f8c-fea5-08d7baee5ecf X-MS-TrafficTypeDiagnostic: MN2PR12MB4190: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2449; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: S1tcfA2YlO45KEOJ1Gxbj3YVLuBNUglMe5U7+9t4beaQyFcNP+/qA1l21YqYq+UffjPLPDVIg3+fDq9rloE8T9FczIt1uUSJSe3Inm7qrCWeu4/P7JnelOlvnsyVXKYNKIp2gqXSGhroACI/j5S/h/lzPEsEdJkzziWmK4Adx7tSA4HiRd+u3v91M8AYHoScB6tGVUUnzzgmiWuMP0s/ZlqhFUqoS/ttiWzHVXTEUNByv74sHARU0Y6vmc8wXSqzm+56pVnIO8ecfIYIhvZmuCvascAFqkCpLmokQGWWi4RUTEo8b1SRP2oSRh5+xH6QffRbqudwU62+32swUBcn+qGWAR10Y2NukmSnQAKVortPklW/gqTfTF6Mxi/GWkIIbUdmJhJM44dFUGUNIOmLSgcnuTRGGjklbOOqG1APSC+LtAYi6BTXM+0rloSllmNf2QyUmej0XQKa3jtlPf9IGoB74INF59r+oRlzYTs5ptmlA1YiA2mhflPQ3grrQoBjDRlqZtAk84GmyAk5jnK7/TYWn0izjSes0At6o1ECMZM= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:05.1447 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f62ef0fb-7496-4f8c-fea5-08d7baee5ecf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4190 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since the drm subsystem can be compiled as a module and drm devices can be added and removed during run time, add several functions to bind the drm subsystem as well as drm devices with drmcg. Two pairs of functions: drmcg_bind/drmcg_unbind - used to bind/unbind the drm subsystem to the cgroup subsystem as the drm core initialize/exit. drmcg_register_dev/drmcg_unregister_dev - used to register/unregister drm devices to the cgroup subsystem as the devices are presented/removed from userspace. Change-Id: I1cb6b2080fc7d27979d886ef23e784341efafb41 --- drivers/gpu/drm/drm_drv.c | 8 +++ include/drm/drm_cgroup.h | 39 +++++++++++ include/linux/cgroup_drm.h | 4 ++ kernel/cgroup/drm.c | 131 +++++++++++++++++++++++++++++++++++++ 4 files changed, 182 insertions(+) create mode 100644 include/drm/drm_cgroup.h diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 7c18a980cd4b..e418a61f5c85 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "drm_crtc_internal.h" #include "drm_internal.h" @@ -973,6 +974,8 @@ int drm_dev_register(struct drm_device *dev, unsigned long flags) ret = 0; + drmcg_register_dev(dev); + DRM_INFO("Initialized %s %d.%d.%d %s for %s on minor %d\n", driver->name, driver->major, driver->minor, driver->patchlevel, driver->date, @@ -1007,6 +1010,8 @@ EXPORT_SYMBOL(drm_dev_register); */ void drm_dev_unregister(struct drm_device *dev) { + drmcg_unregister_dev(dev); + if (drm_core_check_feature(dev, DRIVER_LEGACY)) drm_lastclose(dev); @@ -1113,6 +1118,7 @@ static const struct file_operations drm_stub_fops = { static void drm_core_exit(void) { + drmcg_unbind(); unregister_chrdev(DRM_MAJOR, "drm"); debugfs_remove(drm_debugfs_root); drm_sysfs_destroy(); @@ -1139,6 +1145,8 @@ static int __init drm_core_init(void) if (ret < 0) goto error; + drmcg_bind(&drm_minor_acquire, &drm_dev_put); + drm_core_init_complete = true; DRM_DEBUG("Initialized\n"); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h new file mode 100644 index 000000000000..530c9a0b3238 --- /dev/null +++ b/include/drm/drm_cgroup.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: MIT + * Copyright 2019 Advanced Micro Devices, Inc. + */ +#ifndef __DRM_CGROUP_H__ +#define __DRM_CGROUP_H__ + +#ifdef CONFIG_CGROUP_DRM + +void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)); + +void drmcg_unbind(void); + +void drmcg_register_dev(struct drm_device *dev); + +void drmcg_unregister_dev(struct drm_device *dev); + +#else + +static inline void drmcg_bind( + struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)) +{ +} + +static inline void drmcg_unbind(void) +{ +} + +static inline void drmcg_register_dev(struct drm_device *dev) +{ +} + +static inline void drmcg_unregister_dev(struct drm_device *dev) +{ +} + +#endif /* CONFIG_CGROUP_DRM */ +#endif /* __DRM_CGROUP_H__ */ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 345af54a5d41..307bb75db248 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -5,6 +5,10 @@ #define _CGROUP_DRM_H #include +#include + +/* limit defined per the way drm_minor_alloc operates */ +#define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) #ifdef CONFIG_CGROUP_DRM diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 5e38a8230922..061bb9c458e4 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -1,11 +1,142 @@ // SPDX-License-Identifier: MIT // Copyright 2019 Advanced Micro Devices, Inc. +#include +#include #include #include #include +#include +#include +#include static struct drmcg *root_drmcg __read_mostly; +/* global mutex for drmcg across all devices */ +static DEFINE_MUTEX(drmcg_mutex); + +static DECLARE_BITMAP(known_devs, MAX_DRM_DEV); + +static struct drm_minor (*(*acquire_drm_minor)(unsigned int minor_id)); + +static void (*put_drm_dev)(struct drm_device *dev); + +/** + * drmcg_bind - Bind DRM subsystem to cgroup subsystem + * @acq_dm: function pointer to the drm_minor_acquire function + * @put_ddev: function pointer to the drm_dev_put function + * + * This function binds some functions from the DRM subsystem and make + * them available to the drmcg subsystem. + * + * drmcg_unbind does the opposite of this function + */ +void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)) +{ + mutex_lock(&drmcg_mutex); + acquire_drm_minor = acq_dm; + put_drm_dev = put_ddev; + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_bind); + +/** + * drmcg_unbind - Unbind DRM subsystem from cgroup subsystem + * + * drmcg_bind does the opposite of this function + */ +void drmcg_unbind(void) +{ + mutex_lock(&drmcg_mutex); + acquire_drm_minor = NULL; + put_drm_dev = NULL; + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unbind); + +/** + * drmcg_register_dev - register a DRM device for usage in drm cgroup + * @dev: DRM device + * + * This function make a DRM device visible to the cgroup subsystem. + * Once the drmcg is aware of the device, drmcg can start tracking and + * control resource usage for said device. + * + * drmcg_unregister_dev reverse the operation of this function + */ +void drmcg_register_dev(struct drm_device *dev) +{ + if (WARN_ON(dev->primary->index >= MAX_DRM_DEV)) + return; + + mutex_lock(&drmcg_mutex); + set_bit(dev->primary->index, known_devs); + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_register_dev); + +/** + * drmcg_unregister_dev - Iterate through all stored DRM minors + * @dev: DRM device + * + * Unregister @dev so that drmcg no longer control resource usage + * of @dev. The @dev was registered to drmcg using + * drmcg_register_dev function + */ +void drmcg_unregister_dev(struct drm_device *dev) +{ + if (WARN_ON(dev->primary->index >= MAX_DRM_DEV)) + return; + + mutex_lock(&drmcg_mutex); + clear_bit(dev->primary->index, known_devs); + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unregister_dev); + +/** + * drm_minor_for_each - Iterate through all stored DRM minors + * @fn: Function to be called for each pointer. + * @data: Data passed to callback function. + * + * The callback function will be called for each registered device, passing + * the minor, the @drm_minor entry and @data. + * + * If @fn returns anything other than %0, the iteration stops and that + * value is returned from this function. + */ +static int drm_minor_for_each(int (*fn)(int id, void *p, void *data), + void *data) +{ + int rc = 0; + + mutex_lock(&drmcg_mutex); + if (acquire_drm_minor) { + unsigned int minor; + struct drm_minor *dm; + + minor = find_next_bit(known_devs, MAX_DRM_DEV, 0); + while (minor < MAX_DRM_DEV) { + dm = acquire_drm_minor(minor); + + if (IS_ERR(dm)) + continue; + + rc = fn(minor, (void *)dm, data); + + put_drm_dev(dm->dev); /* release from acquire_drm_minor */ + + if (rc) + break; + + minor = find_next_bit(known_devs, MAX_DRM_DEV, minor+1); + } + } + mutex_unlock(&drmcg_mutex); + + return rc; +} + static void drmcg_css_free(struct cgroup_subsys_state *css) { struct drmcg *drmcg = css_to_drmcg(css); From patchwork Wed Feb 26 19:01:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407067 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7EB714B4 for ; Wed, 26 Feb 2020 19:02:25 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A55532072D for ; Wed, 26 Feb 2020 19:02:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="Pm6HAe4/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A55532072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1598D6EBA7; Wed, 26 Feb 2020 19:02:12 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2069.outbound.protection.outlook.com [40.107.223.69]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0A1156EB91; Wed, 26 Feb 2020 19:02:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dm3SMRDL41zFZu799kIMSlmSOyDuQH1AgMfwX/1RjEp4MvrrvQPqCAxmrbrhrJ24sO66qXvQLz/Nu9cS4dbsGsDQ7+Zv1cELsaGe2CsTcSMrxg4neIenyZPv/eNycvF+kmn2qVXipZOVeAalpZLSLv5vj6mDQIJ1U6vTlths77DbzNz5i0+xY4ItfijO4gLMa6LYWHDi1KlwdighroO+UOHt5ENtMxXHZ6ykP0VM0BitWWHMrXp15KILJ7sJo5Bd/lEJMFg0pTA4vMBYLj1oFZldmvRdDUSPXY10IcvCadEc3GRUium3yjnssyzbQmFETj8dSj+3jpd55i6LkHV+Yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gDjmHQ9BMdiPJKyRL+KkHYw+s6jN5irmJDPeYaRghG8=; b=cpZh1c2uYVit0ls1p5UE65eiD2JKuI4ImX74m6wZnVjoRTboPuuYccf5+sDwgqVseRJSCbS8xuqABurkZjkD83Oc3C47vQDG8EbEsFxC7PWt0zOZQhpAnkwuu2vFxgQpgEq9j/9R6HaGgR++tA0GFq1TONtk/hjmD4HVoD/ai21IDB3hLFHeHRWCrxy5NeE30GnG+gaVgt5cqaUb3V0iW3Cp11V/eLxe8pUgevm30CUz3rMhcDCSj/zSuu2Qmg+HhHfzyWFDj0KOhcgbJ0Ac6shu/9IIJtSpEsbaWD4qv/gauc0pt4yuAboJ8s8eEaSORLBrO/8BGPUnk3uewacdLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gDjmHQ9BMdiPJKyRL+KkHYw+s6jN5irmJDPeYaRghG8=; b=Pm6HAe4/1NhI5/lHxI02gsnZSteHEFIykxs3RYi78tCDMK0u80OwHk3Os85nQDi7b5I6QtZ1/9nhJ8crfE5v8tWQdeiG1tCdD74JZBAnMVlIgc+yzcwstkxFXgKRgw122eA9AidnG9jpcmajNd3m4l87cI7k/FRBc+CaNoh2D+E= Received: from DM5PR19CA0036.namprd19.prod.outlook.com (2603:10b6:3:9a::22) by MW2PR12MB2540.namprd12.prod.outlook.com (2603:10b6:907:7::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:05 +0000 Received: from DM6NAM11FT063.eop-nam11.prod.protection.outlook.com (2603:10b6:3:9a:cafe::7b) by DM5PR19CA0036.outlook.office365.com (2603:10b6:3:9a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:05 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by DM6NAM11FT063.mail.protection.outlook.com (10.13.172.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:05 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:04 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:04 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:03 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 03/11] drm, cgroup: Initialize drmcg properties Date: Wed, 26 Feb 2020 14:01:44 -0500 Message-ID: <20200226190152.16131-4-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(376002)(136003)(396003)(428003)(199004)(189003)(36756003)(70206006)(70586007)(6666004)(356004)(478600001)(2906002)(8936002)(86362001)(81166006)(8676002)(81156014)(316002)(110136005)(26005)(7696005)(426003)(2616005)(5660300002)(4326008)(1076003)(336012)(186003)(921003)(83996005)(1121003)(2101003); DIR:OUT; SFP:1101; SCL:1; SRVR:MW2PR12MB2540; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6206e1c6-f6fb-46e5-7814-08d7baee5f1b X-MS-TrafficTypeDiagnostic: MW2PR12MB2540: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VhYj0yEz9HKrcefFgRF87N9JNqNnAMGmZOG6y74u01Dr1mYIoPmWBo7mRtLV0thglK1beL2oL5b1Dh2xdmb+MPdwA6whnCkbhGMmQi/ajbCJ8uLcjoqFEdUjmg7cMdXi+R35FiHEiu2HHlYIpZhQXaxk3EH+wZJsDRTsL+uPgCLgjbldCvS4VDwuDjp4brYLkyp5lLyj4RwQHCpcxr+/qTauubjcUi2zfO01eSnM9wUW0WDsqPXY/pglJTLUMsGJsC+nY1573A6J76HMWhQWotHZYmJInp3peyWgbo5zLdWmdd2r5tlNt+w2OQKnym1LJtXtnHd0yxXK5CtTKqQn+drHEFYq/aD6yBDZrVfltgghhfHnhMssJIFmHQa0PYFKxqhibtmdXHNxhB5YyLfmfTmNhL73rEXPJy+98bFnyZJdzFbUDJeUGXtm1Tb95SVQWfNv2e73+P88yE670iW/u6rduOrmLXjwOtCbWWMdG+lKR2i0d6H+QtdAOeSnoxoOOBrHedJf/3qdqtrYohN/98Yc3u/WQIC/azRT5LLlLb8= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:05.5943 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6206e1c6-f6fb-46e5-7814-08d7baee5f1b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2540 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drmcg initialization involves allocating a per cgroup, per device data structure and setting the defaults. There are two entry points for drmcg init: 1) When struct drmcg is created via css_alloc, initialization is done for each device 2) When DRM devices are created after drmcgs are created a) Per device drmcg data structure is allocated at the beginning of DRM device creation such that drmcg can begin tracking usage statistics b) At the end of DRM device creation, drmcg_register_dev will update in case device specific defaults need to be applied. Entry point #2 usually applies to the root cgroup since it can be created before DRM devices are available. The drmcg controller will go through all existing drm cgroups and initialize them with the new device accordingly. Change-Id: I64e421d8dfcc22ee8282cc1305960e20c2704db7 Signed-off-by: Kenny Ho --- drivers/gpu/drm/drm_drv.c | 4 ++ include/drm/drm_cgroup.h | 18 +++++++ include/drm/drm_device.h | 7 +++ include/drm/drm_drv.h | 9 ++++ include/linux/cgroup_drm.h | 12 +++++ kernel/cgroup/drm.c | 105 +++++++++++++++++++++++++++++++++++++ 6 files changed, 155 insertions(+) diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index e418a61f5c85..e10bd42ebdba 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -644,6 +644,7 @@ int drm_dev_init(struct drm_device *dev, mutex_init(&dev->filelist_mutex); mutex_init(&dev->clientlist_mutex); mutex_init(&dev->master_mutex); + mutex_init(&dev->drmcg_mutex); dev->anon_inode = drm_fs_inode_new(); if (IS_ERR(dev->anon_inode)) { @@ -680,6 +681,7 @@ int drm_dev_init(struct drm_device *dev, if (ret) goto err_setunique; + drmcg_device_early_init(dev); return 0; err_setunique: @@ -694,6 +696,7 @@ int drm_dev_init(struct drm_device *dev, drm_fs_inode_free(dev->anon_inode); err_free: put_device(dev->dev); + mutex_destroy(&dev->drmcg_mutex); mutex_destroy(&dev->master_mutex); mutex_destroy(&dev->clientlist_mutex); mutex_destroy(&dev->filelist_mutex); @@ -770,6 +773,7 @@ void drm_dev_fini(struct drm_device *dev) put_device(dev->dev); + mutex_destroy(&dev->drmcg_mutex); mutex_destroy(&dev->master_mutex); mutex_destroy(&dev->clientlist_mutex); mutex_destroy(&dev->filelist_mutex); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 530c9a0b3238..fda426fba035 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -4,8 +4,17 @@ #ifndef __DRM_CGROUP_H__ #define __DRM_CGROUP_H__ +#include + #ifdef CONFIG_CGROUP_DRM +/** + * Per DRM device properties for DRM cgroup controller for the purpose + * of storing per device defaults + */ +struct drmcg_props { +}; + void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), void (*put_ddev)(struct drm_device *dev)); @@ -15,8 +24,13 @@ void drmcg_register_dev(struct drm_device *dev); void drmcg_unregister_dev(struct drm_device *dev); +void drmcg_device_early_init(struct drm_device *device); + #else +struct drmcg_props { +}; + static inline void drmcg_bind( struct drm_minor (*(*acq_dm)(unsigned int minor_id)), void (*put_ddev)(struct drm_device *dev)) @@ -35,5 +49,9 @@ static inline void drmcg_unregister_dev(struct drm_device *dev) { } +static inline void drmcg_device_early_init(struct drm_device *device) +{ +} + #endif /* CONFIG_CGROUP_DRM */ #endif /* __DRM_CGROUP_H__ */ diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h index 1acfc3bbd3fb..a94598b8f670 100644 --- a/include/drm/drm_device.h +++ b/include/drm/drm_device.h @@ -8,6 +8,7 @@ #include #include +#include struct drm_driver; struct drm_minor; @@ -308,6 +309,12 @@ struct drm_device { */ struct drm_fb_helper *fb_helper; + /** \name DRM Cgroup */ + /*@{ */ + struct mutex drmcg_mutex; + struct drmcg_props drmcg_props; + /*@} */ + /* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED(CONFIG_DRM_LEGACY) diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index cf13470810a5..1f65ac4d9bbf 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -715,6 +715,15 @@ struct drm_driver { struct drm_device *dev, uint32_t handle); + /** + * @drmcg_custom_init + * + * Optional callback used to initialize drm cgroup per device properties + * such as resource limit defaults. + */ + void (*drmcg_custom_init)(struct drm_device *dev, + struct drmcg_props *props); + /** * @gem_vm_ops: Driver private ops for this object * diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 307bb75db248..ff94b48aa2dc 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -4,6 +4,7 @@ #ifndef _CGROUP_DRM_H #define _CGROUP_DRM_H +#include #include #include @@ -12,11 +13,19 @@ #ifdef CONFIG_CGROUP_DRM +/** + * Per DRM cgroup, per device resources (such as statistics and limits) + */ +struct drmcg_device_resource { + /* for per device stats */ +}; + /** * The DRM cgroup controller data structure. */ struct drmcg { struct cgroup_subsys_state css; + struct drmcg_device_resource *dev_resources[MAX_DRM_DEV]; }; /** @@ -70,6 +79,9 @@ static inline struct drmcg *drmcg_parent(struct drmcg *cg) #else /* CONFIG_CGROUP_DRM */ +struct drmcg_device_resource { +}; + struct drmcg { }; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 061bb9c458e4..351df517d5a6 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -1,11 +1,17 @@ // SPDX-License-Identifier: MIT // Copyright 2019 Advanced Micro Devices, Inc. #include +#include #include #include #include +#include +#include +#include +#include #include #include +#include #include #include @@ -54,6 +60,47 @@ void drmcg_unbind(void) } EXPORT_SYMBOL(drmcg_unbind); +/* caller must hold dev->drmcg_mutex */ +static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) +{ + int minor = dev->primary->index; + struct drmcg_device_resource *ddr = drmcg->dev_resources[minor]; + + if (ddr == NULL) { + ddr = kzalloc(sizeof(struct drmcg_device_resource), + GFP_KERNEL); + + if (!ddr) + return -ENOMEM; + } + + drmcg->dev_resources[minor] = ddr; + + /* set defaults here */ + + return 0; +} + +static inline void drmcg_update_cg_tree(struct drm_device *dev) +{ + struct cgroup_subsys_state *pos; + struct drmcg *child; + + if (root_drmcg == NULL) + return; + + /* init cgroups created before registration (i.e. root cgroup) */ + + /* use cgroup_mutex instead of rcu_read_lock because + * init_drmcg_single has alloc which may sleep */ + mutex_lock(&cgroup_mutex); + css_for_each_descendant_pre(pos, &root_drmcg->css) { + child = css_to_drmcg(pos); + init_drmcg_single(child, dev); + } + mutex_unlock(&cgroup_mutex); +} + /** * drmcg_register_dev - register a DRM device for usage in drm cgroup * @dev: DRM device @@ -71,6 +118,13 @@ void drmcg_register_dev(struct drm_device *dev) mutex_lock(&drmcg_mutex); set_bit(dev->primary->index, known_devs); + + if (dev->driver->drmcg_custom_init) + { + dev->driver->drmcg_custom_init(dev, &dev->drmcg_props); + + drmcg_update_cg_tree(dev); + } mutex_unlock(&drmcg_mutex); } EXPORT_SYMBOL(drmcg_register_dev); @@ -137,23 +191,61 @@ static int drm_minor_for_each(int (*fn)(int id, void *p, void *data), return rc; } +static int drmcg_css_free_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + kfree(drmcg->dev_resources[minor->index]); + + return 0; +} + static void drmcg_css_free(struct cgroup_subsys_state *css) { struct drmcg *drmcg = css_to_drmcg(css); + drm_minor_for_each(&drmcg_css_free_fn, drmcg); + kfree(drmcg); } +static int init_drmcg_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + int rc; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + mutex_lock(&minor->dev->drmcg_mutex); + rc = init_drmcg_single(drmcg, minor->dev); + mutex_unlock(&minor->dev->drmcg_mutex); + + return rc; +} + static struct cgroup_subsys_state * drmcg_css_alloc(struct cgroup_subsys_state *parent_css) { struct drmcg *parent = css_to_drmcg(parent_css); struct drmcg *drmcg; + int rc; drmcg = kzalloc(sizeof(struct drmcg), GFP_KERNEL); if (!drmcg) return ERR_PTR(-ENOMEM); + rc = drm_minor_for_each(&init_drmcg_fn, drmcg); + if (rc) { + drmcg_css_free(&drmcg->css); + return ERR_PTR(rc); + } + if (!parent) root_drmcg = drmcg; @@ -171,3 +263,16 @@ struct cgroup_subsys gpu_cgrp_subsys = { .legacy_cftypes = files, .dfl_cftypes = files, }; + +/** + * drmcg_device_early_init - initialize device specific resources for DRM cgroups + * @dev: the target DRM device + * + * Allocate and initialize device specific resources for existing DRM cgroups. + * Typically only the root cgroup exists before the initialization of @dev. + */ +void drmcg_device_early_init(struct drm_device *dev) +{ + drmcg_update_cg_tree(dev); +} +EXPORT_SYMBOL(drmcg_device_early_init); From patchwork Wed Feb 26 19:01:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407063 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0613930 for ; Wed, 26 Feb 2020 19:02:21 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ADE562072D for ; Wed, 26 Feb 2020 19:02:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="HBGl4rve" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ADE562072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 815E36EB9F; Wed, 26 Feb 2020 19:02:10 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2075.outbound.protection.outlook.com [40.107.223.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id AC4096E182; Wed, 26 Feb 2020 19:02:07 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mSGBdhqMukkKBCQ/THzozcDgEKHvucL8nuYI4yqqaefAjtJX9cgH3d4PFBP7KayJ8xuXkwdquSeSBEyJ17p1AYhVsAgT+/aDrE0E/+tEtpUPbgNp2fgV4fLCqwsZWfImDcA5HTZ4k1gtDbeBckHYdw1Upp6XSgcveAZEmwvSSS2C7PNMGepdpdz7xWuX/kmSPzEp6cQVTSMMDe9yFqeWeZc+2t0wObpCTOYSwYqyXry+HqqEMx/fXNMTIuc7ses7qIfhoCBWHJKQslzrhk+EYfr9E+/Grglu71E53XGXyFzXNCrBUm38TAJzSBvKK8l5cPiDnt2FbLqwba6dFi2JGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L7hi9C3bgjndcmDcy7j5BoWa6flwa/zQrZ4rC0zMA2s=; b=ewP2AYxCYINtaZAcUCuh9OyFf7KnFhMsbb8ClCzmvEtcYkZK6tqUCadjE8QHwtJIGe2YsAEhKIb9MucGThb/t/CxRFAw4venrVmADW3XeayOR62ARpLTwEXP+bvkfpg6nRPgz6JyaIr79rhLONijJKHVxFYJTBrkqAqRPX5fEC8Yrl9PdssxmE4zkOmsKEfjD2mLn4H7wmLfHPn8dTUOYrvme0Q0LcVc70KpynqVxqVaEJxvFsVNhNNJfJuvGXur4RGcRmCF6Wxf5udqGw8uSIu/FaXXkNrF1NhS1vdXhP0YY+Avl/dRry/I1ApvFuOPsBXofNBPMvthjbFneToEmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L7hi9C3bgjndcmDcy7j5BoWa6flwa/zQrZ4rC0zMA2s=; b=HBGl4rveDIlxXDvGSwaNqf31SSqJDasjDT7pFModmxKu6jtcp3RY/eq4Wn9r4pu4GynN6bohuK7Ah/zoeZO1R3/ks0zHvlPzXH3hB3a6hikjU5cjWXOX9B3q7MabPiUCeZfTLAMnE2eZksUGimeCgY4cf5pZVAjlcFBwgazcNTc= Received: from MN2PR01CA0014.prod.exchangelabs.com (2603:10b6:208:10c::27) by MN2PR12MB4094.namprd12.prod.outlook.com (2603:10b6:208:15f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:06 +0000 Received: from BN8NAM11FT021.eop-nam11.prod.protection.outlook.com (2603:10b6:208:10c:cafe::2b) by MN2PR01CA0014.outlook.office365.com (2603:10b6:208:10c::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21 via Frontend Transport; Wed, 26 Feb 2020 19:02:06 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT021.mail.protection.outlook.com (10.13.177.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:06 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:05 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:05 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:04 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 04/11] drm, cgroup: Add total GEM buffer allocation stats Date: Wed, 26 Feb 2020 14:01:45 -0500 Message-ID: <20200226190152.16131-5-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(39860400002)(376002)(346002)(136003)(428003)(189003)(199004)(30864003)(7696005)(316002)(110136005)(26005)(1076003)(8936002)(81156014)(8676002)(81166006)(4326008)(336012)(2616005)(426003)(70586007)(356004)(86362001)(5660300002)(186003)(478600001)(2906002)(6666004)(70206006)(36756003)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR12MB4094; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 442efcc0-950c-4b11-3f30-08d7baee5f63 X-MS-TrafficTypeDiagnostic: MN2PR12MB4094: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6/9/IuuoLHg8jQSWV0qoZ15nFvKBz7s7nbbJmZFrctaZ7WNlA0BTkc479Q4Zk6kXTO8vbh9wMyj/AO6Iq3Rr1qB3QocKOp9wFxMTZkU8xHk3o7NhAOofA5qssX/S6ih0zJISB30O/j1xKy3EwC+kyoj02vi2UGJvjIBFOaev1FR1Bzvxg5l87NwtySwDt2IEC39rvAZOYBLtu7d6UdU3/hAWAAO/rGXst1cCQJnZf9Lgrx1UinpboqO+WDCcjT56NmPWttiotQpFp6zTh9scBfjwnMmWTObOF/B4FaweagCOMluuJAJYTZoXeB1uQ83qRumh9YMnXdAyDZJr6bh5Pjyfe6v4BEMmySMpEqFg312Hnl+jtEf2u+5Xuw1Fah4YsdwGExytEXEZ7hkoL1LH8Xg0Xs1JBLQZa294+0FEy9D1K+Ou0NdiyhMNREOb85u0tNiXS2drTsDGh7nMwe/GqFGmTYzwh+s2HBWwRIV/rwqbkjjto+HN73xq/r8B4p3nx7Yyw9FtUMx61gvBg2twHLqd3uXq1NWSvtHYlN+1uIM= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:06.1107 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 442efcc0-950c-4b11-3f30-08d7baee5f63 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4094 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The drm resource being measured here is the GEM buffer objects. User applications allocate and free these buffers. In addition, a process can allocate a buffer and share it with another process. The consumer of a shared buffer can also outlive the allocator of the buffer. For the purpose of cgroup accounting and limiting, ownership of the buffer is deemed to be the cgroup for which the allocating process belongs to. There is one cgroup stats per drm device. Each allocation is charged to the owning cgroup as well as all its ancestors. Similar to the memory cgroup, migrating a process to a different cgroup does not move the GEM buffer usages that the process started while in previous cgroup, to the new cgroup. The following is an example to illustrate some of the operations. Given the following cgroup hierarchy (The letters are cgroup names with R being the root cgroup. The numbers in brackets are processes. The processes are placed with cgroup's 'No Internal Process Constraint' in mind, so no process is placed in cgroup B.) R (4, 5) ------ A (6) \ B ---- C (7,8) \ D (9) Here is a list of operation and the associated effect on the size track by the cgroups (for simplicity, each buffer is 1 unit in size.) == == == == == =================================================== R A B C D Ops == == == == == =================================================== 1 0 0 0 0 4 allocated a buffer 1 0 0 0 0 4 shared a buffer with 5 1 0 0 0 0 4 shared a buffer with 9 2 0 1 0 1 9 allocated a buffer 3 0 2 1 1 7 allocated a buffer 3 0 2 1 1 7 shared a buffer with 8 3 0 2 1 1 7 sharing with 9 3 0 2 1 1 7 release a buffer 3 0 2 1 1 7 migrate to cgroup D 3 0 2 1 1 9 release a buffer from 7 2 0 1 0 1 8 release a buffer from 7 (last ref to shared buf) == == == == == =================================================== gpu.buffer.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Total GEM buffer allocation in bytes. Change-Id: Ibc1f646ca7dbc588e2d11802b156b524696a23e7 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 50 +++++++++- drivers/gpu/drm/drm_gem.c | 9 ++ include/drm/drm_cgroup.h | 16 +++ include/drm/drm_gem.h | 10 ++ include/linux/cgroup_drm.h | 6 ++ kernel/cgroup/drm.c | 126 ++++++++++++++++++++++++ 6 files changed, 216 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 7deff912185e..c041e672cc10 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -63,6 +63,7 @@ v1 is available under Documentation/admin-guide/cgroup-v1/. 5-7-1. RDMA Interface Files 5-8. GPU 5-8-1. GPU Interface Files + 5-8-2. GEM Buffer Ownership 5-9. Misc 5-9-1. perf_event 5-N. Non-normative information @@ -2068,7 +2069,54 @@ of GPU-related resources. GPU Interface Files ~~~~~~~~~~~~~~~~~~~~ -TODO + gpu.buffer.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Total GEM buffer allocation in bytes. + +GEM Buffer Ownership +~~~~~~~~~~~~~~~~~~~~ + +For the purpose of cgroup accounting and limiting, ownership of the +buffer is deemed to be the cgroup for which the allocating process +belongs to. There is one cgroup stats per drm device. Each allocation +is charged to the owning cgroup as well as all its ancestors. + +Similar to the memory cgroup, migrating a process to a different cgroup +does not move the GEM buffer usages that the process started while in +previous cgroup, to the new cgroup. + +The following is an example to illustrate some of the operations. Given +the following cgroup hierarchy (The letters are cgroup names with R +being the root cgroup. The numbers in brackets are processes. The +processes are placed with cgroup's 'No Internal Process Constraint' in +mind, so no process is placed in cgroup B.) + +R (4, 5) ------ A (6) + \ + B ---- C (7,8) + \ + D (9) + +Here is a list of operation and the associated effect on the size +track by the cgroups (for simplicity, each buffer is 1 unit in size.) + +== == == == == =================================================== +R A B C D Ops +== == == == == =================================================== +1 0 0 0 0 4 allocated a buffer +1 0 0 0 0 4 shared a buffer with 5 +1 0 0 0 0 4 shared a buffer with 9 +2 0 1 0 1 9 allocated a buffer +3 0 2 1 1 7 allocated a buffer +3 0 2 1 1 7 shared a buffer with 8 +3 0 2 1 1 7 sharing with 9 +3 0 2 1 1 7 release a buffer +3 0 2 1 1 7 migrate to cgroup D +3 0 2 1 1 9 release a buffer from 7 +2 0 1 0 1 8 release a buffer from 7 (last ref to shared buf) +== == == == == =================================================== Misc diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index a9e4a610445a..589f8f6bde2c 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -46,6 +47,7 @@ #include #include #include +#include #include "drm_internal.h" @@ -164,6 +166,9 @@ void drm_gem_private_object_init(struct drm_device *dev, obj->resv = &obj->_resv; drm_vma_node_reset(&obj->vma_node); + + obj->drmcg = drmcg_get(current); + drmcg_chg_bo_alloc(obj->drmcg, dev, size); } EXPORT_SYMBOL(drm_gem_private_object_init); @@ -957,6 +962,10 @@ drm_gem_object_release(struct drm_gem_object *obj) fput(obj->filp); dma_resv_fini(&obj->_resv); + + drmcg_unchg_bo_alloc(obj->drmcg, obj->dev, obj->size); + drmcg_put(obj->drmcg); + drm_gem_free_mmap_offset(obj); } EXPORT_SYMBOL(drm_gem_object_release); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index fda426fba035..1eb3012e16a1 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -26,6 +26,12 @@ void drmcg_unregister_dev(struct drm_device *dev); void drmcg_device_early_init(struct drm_device *device); +void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size); + +void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size); + #else struct drmcg_props { @@ -53,5 +59,15 @@ static inline void drmcg_device_early_init(struct drm_device *device) { } +static inline void drmcg_chg_bo_alloc(struct drmcg *drmcg, + struct drm_device *dev, size_t size) +{ +} + +static inline void drmcg_unchg_bo_alloc(struct drmcg *drmcg, + struct drm_device *dev, size_t size) +{ +} + #endif /* CONFIG_CGROUP_DRM */ #endif /* __DRM_CGROUP_H__ */ diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h index 0b375069cd48..9c588c329da0 100644 --- a/include/drm/drm_gem.h +++ b/include/drm/drm_gem.h @@ -310,6 +310,16 @@ struct drm_gem_object { * */ const struct drm_gem_object_funcs *funcs; + + /** + * @drmcg: + * + * DRM cgroup this GEM object belongs to. + * + * This is used to track and limit the amount of GEM objects a user + * can allocate. + */ + struct drmcg *drmcg; }; /** diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index ff94b48aa2dc..34b0aec7c964 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -11,6 +11,11 @@ /* limit defined per the way drm_minor_alloc operates */ #define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) +enum drmcg_res_type { + DRMCG_TYPE_BO_TOTAL, + __DRMCG_TYPE_LAST, +}; + #ifdef CONFIG_CGROUP_DRM /** @@ -18,6 +23,7 @@ */ struct drmcg_device_resource { /* for per device stats */ + s64 bo_stats_total_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 351df517d5a6..addb096edac5 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -13,6 +13,7 @@ #include #include #include +#include #include static struct drmcg *root_drmcg __read_mostly; @@ -26,6 +27,18 @@ static struct drm_minor (*(*acquire_drm_minor)(unsigned int minor_id)); static void (*put_drm_dev)(struct drm_device *dev); +#define DRMCG_CTF_PRIV_SIZE 3 +#define DRMCG_CTF_PRIV_MASK GENMASK((DRMCG_CTF_PRIV_SIZE - 1), 0) +#define DRMCG_CTF_PRIV(res_type, f_type) ((res_type) <<\ + DRMCG_CTF_PRIV_SIZE | (f_type)) +#define DRMCG_CTF_PRIV2RESTYPE(priv) ((priv) >> DRMCG_CTF_PRIV_SIZE) +#define DRMCG_CTF_PRIV2FTYPE(priv) ((priv) & DRMCG_CTF_PRIV_MASK) + + +enum drmcg_file_type { + DRMCG_FTYPE_STATS, +}; + /** * drmcg_bind - Bind DRM subsystem to cgroup subsystem * @acq_dm: function pointer to the drm_minor_acquire function @@ -252,7 +265,66 @@ drmcg_css_alloc(struct cgroup_subsys_state *parent_css) return &drmcg->css; } +static void drmcg_print_stats(struct drmcg_device_resource *ddr, + struct seq_file *sf, enum drmcg_res_type type) +{ + if (ddr == NULL) { + seq_puts(sf, "\n"); + return; + } + + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + +static int drmcg_seq_show_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct seq_file *sf = data; + struct drmcg *drmcg = css_to_drmcg(seq_css(sf)); + enum drmcg_file_type f_type = + DRMCG_CTF_PRIV2FTYPE(seq_cft(sf)->private); + enum drmcg_res_type type = + DRMCG_CTF_PRIV2RESTYPE(seq_cft(sf)->private); + struct drmcg_device_resource *ddr; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + ddr = drmcg->dev_resources[minor->index]; + + seq_printf(sf, "%d:%d ", DRM_MAJOR, minor->index); + + switch (f_type) { + case DRMCG_FTYPE_STATS: + drmcg_print_stats(ddr, sf, type); + break; + default: + seq_puts(sf, "\n"); + break; + } + + return 0; +} + +int drmcg_seq_show(struct seq_file *sf, void *v) +{ + return drm_minor_for_each(&drmcg_seq_show_fn, sf); +} + struct cftype files[] = { + { + .name = "buffer.total.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -276,3 +348,57 @@ void drmcg_device_early_init(struct drm_device *dev) drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); + +/** + * drmcg_chg_bo_alloc - charge GEM buffer usage for a device and cgroup + * @drmcg: the DRM cgroup to be charged to + * @dev: the device the usage should be charged to + * @size: size of the GEM buffer to be accounted for + * + * This function should be called when a new GEM buffer is allocated to account + * for the utilization. This should not be called when the buffer is shared ( + * the GEM buffer's reference count being incremented.) + */ +void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size) +{ + struct drmcg_device_resource *ddr; + int devIdx = dev->primary->index; + + if (drmcg == NULL) + return; + + mutex_lock(&dev->drmcg_mutex); + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + + ddr->bo_stats_total_allocated += (s64)size; + } + mutex_unlock(&dev->drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_chg_bo_alloc); + +/** + * drmcg_unchg_bo_alloc - + * @drmcg: the DRM cgroup to uncharge from + * @dev: the device the usage should be removed from + * @size: size of the GEM buffer to be accounted for + * + * This function should be called when the GEM buffer is about to be freed ( + * not simply when the GEM buffer's reference count is being decremented.) + */ +void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size) +{ + int devIdx = dev->primary->index; + + if (drmcg == NULL) + return; + + mutex_lock(&dev->drmcg_mutex); + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) + drmcg->dev_resources[devIdx]->bo_stats_total_allocated + -= (s64)size; + mutex_unlock(&dev->drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unchg_bo_alloc); From patchwork Wed Feb 26 19:01:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407069 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29E9814B4 for ; Wed, 26 Feb 2020 19:02:28 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07ABC2072D for ; Wed, 26 Feb 2020 19:02:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="PAsKcD2D" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07ABC2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 875796EBA8; Wed, 26 Feb 2020 19:02:12 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-eopbgr760042.outbound.protection.outlook.com [40.107.76.42]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8150B6EB9E; Wed, 26 Feb 2020 19:02:10 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MdM+8aNQkGeTvuJoUFyySNjDA1CXpiEbpcLCxAbLWPDZGsor4jRb/1aBOeEj28tmIuSLv+CHMigZtJiekz3D8sPmKBXF1bkhxCOPB5p461mbF8hLuzoxOEkj1vwbukOr/KykiNwhfAb5XkcvN4Jn/2CmYW8NW7Kd1oDingRrLAa359tII4nxc1RvzNFU0UfYUA3onQ3AS4g9GqCU2uBtIIbC3XthRfSxtJp8QYaa8Rqviu8cfUp69HmCVH+93YOLMIeyirMLLOqab2AucbmU4BNiKnRtW8chkk0CQCbyw+opUol/FS2M5crffP5Gq/HaS7pPANH26VkMFnWFliq11Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PiAbqtACZxUAOCGygYgff2fSwPAHDJcNbF8JiwTst00=; b=SLvA45s5MsPo7PmqUoLuw3ipxEbXaX+6DBdz/BYulZxcdjXqoW/AtbZgrB48VX/g1BIe3GRmXx/clpm9uXuY6dAaXRsJeuQ1abUk+aGiEac5IBIs+c97VjuPILLQcuRPIPAZRNcEpR+GX9xmcz+QQqJ4zGmJGpXBZjM609SjFuhrvt2gCLprl31ZXqt3G8PUyXG3PpyuJxLVCXlat6hbqG+8Z4UzDUREkileceZ/uLGR0KCBZ6raDx1zFXKizaspFKL0uuVsxMsBEzjhErngxA/iheXphUV/TyuFK+CZ4oSC6pQaYYkfqnOoF4r204uFstJXWZQhuSxZaTe/RilVIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PiAbqtACZxUAOCGygYgff2fSwPAHDJcNbF8JiwTst00=; b=PAsKcD2DeUi9TP+Z2suMFjCq5FjP81CeYmYxfCFEaFU29FyQD8ZJeW5r7aqb4CCxLFdgoxC3BeP4Yo3UlnQS+rlNKinympu98pDAlajBtJNDu/F2UuhDcml+7R0xpRY9rMnXpNpeaJdwJkk5D8MEkmd+GegVHm2tY+yhmaXB4oo= Received: from BN6PR1701CA0013.namprd17.prod.outlook.com (2603:10b6:405:15::23) by CY4PR12MB1286.namprd12.prod.outlook.com (2603:10b6:903:44::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.17; Wed, 26 Feb 2020 19:02:09 +0000 Received: from BN8NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:405:15:cafe::44) by BN6PR1701CA0013.outlook.office365.com (2603:10b6:405:15::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:08 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT012.mail.protection.outlook.com (10.13.177.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:08 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:06 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:05 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 05/11] drm, cgroup: Add peak GEM buffer allocation stats Date: Wed, 26 Feb 2020 14:01:46 -0500 Message-ID: <20200226190152.16131-6-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(376002)(396003)(39860400002)(346002)(428003)(189003)(199004)(36756003)(336012)(6666004)(478600001)(2616005)(356004)(86362001)(426003)(1076003)(110136005)(316002)(5660300002)(4326008)(7696005)(26005)(70206006)(8936002)(2906002)(186003)(70586007)(81156014)(8676002)(81166006)(921003)(1121003)(83996005)(2101003); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1286; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 377ebe97-3a36-4e19-4f6b-08d7baee6090 X-MS-TrafficTypeDiagnostic: CY4PR12MB1286: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NsIjiT4hRk37weOpuzgU7mf57V82CLQLEi5OsPHKVShAeGLxTELKxuiFslHWjnAOZWoiYMGYLq/9i1ukM0Jz76yznHuJunUwxgPgGxU/rdQZyBiD5KWzN23iyfB5UDSj4gT3WA4uvIK8SCYg4EG3r8eAQnXaF0n0m+ipb0i2U/o8pcjkJlDRdkyXjz3q0uSeRP2o5f5rTBu9m3Ey0XVgtKKe67Kbd96cKiHH7le1dMFn8bR3HsZF+oSZwO2tk8UGylXGYe5h3znceTKJsO3TRrVNYziIrP3QiRMQBTD/QZD2Ay+8kGTNTpzkYIFCl8Fjmy/37yLW/VuWYem9WWTkO0fcO77Ic9mp0OSf6rTmykVM6zsM+SzJ8MQvJzOQde+WJCKHftI3t11jpA44Q36097gy9DY4zST1MibfEQDjpU+YskqhJPGqtB+MImMi6zx9gAocCkP49I2f/FzjWrVRDdq0qP1JPuQq3y+rfh3CrmM9gjNY3tAal3nD1WcXqi+DZPW+VTECGvDP7TLaI43xABhb0gd+HgBWqrb9SJLzydQ= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:08.0827 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 377ebe97-3a36-4e19-4f6b-08d7baee6090 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1286 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" gpu.buffer.peak.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Largest (high water mark) GEM buffer allocated in bytes. Change-Id: I40fe4c13c1cea8613b3e04b802f3e1f19eaab4fc Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 6 ++++++ include/linux/cgroup_drm.h | 3 +++ kernel/cgroup/drm.c | 12 ++++++++++++ 3 files changed, 21 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index c041e672cc10..6199cc9a978f 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2075,6 +2075,12 @@ GPU Interface Files Total GEM buffer allocation in bytes. + gpu.buffer.peak.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Largest (high water mark) GEM buffer allocated in bytes. + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 34b0aec7c964..d90807627213 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -13,6 +13,7 @@ enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, + DRMCG_TYPE_BO_PEAK, __DRMCG_TYPE_LAST, }; @@ -24,6 +25,8 @@ enum drmcg_res_type { struct drmcg_device_resource { /* for per device stats */ s64 bo_stats_total_allocated; + + s64 bo_stats_peak_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index addb096edac5..68b23693418b 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -277,6 +277,9 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_TOTAL: seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", ddr->bo_stats_peak_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -325,6 +328,12 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.peak.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -373,6 +382,9 @@ void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, ddr = drmcg->dev_resources[devIdx]; ddr->bo_stats_total_allocated += (s64)size; + + if (ddr->bo_stats_peak_allocated < (s64)size) + ddr->bo_stats_peak_allocated = (s64)size; } mutex_unlock(&dev->drmcg_mutex); } From patchwork Wed Feb 26 19:01:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407075 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4091C14B4 for ; Wed, 26 Feb 2020 19:02:37 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1E9BB2072D for ; Wed, 26 Feb 2020 19:02:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="Nw7DIMO1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E9BB2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C04946EBC4; Wed, 26 Feb 2020 19:02:26 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2082.outbound.protection.outlook.com [40.107.244.82]) by gabe.freedesktop.org (Postfix) with ESMTPS id 97A306EBA8; Wed, 26 Feb 2020 19:02:11 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RslJ6LOtyccCN2g3kVTp3s5bUxiB1nGWaZu35D79cqZtxOthqZcOyJ6ME2sutoX3Vv6vb1cx5GIEdE7CQqZWNIEbixr8kxxg+976/AiDfwQj3pcSN3+9LRQ2Emb7Wuzle9dU7/x2bkNNHSFSlemvy50oRDMYplePX2kiF0Kvr2LFtiUbXwC44T7C/ApTlR0W1IzIeRxarN8goSrFhY5o6WvqRhyvjWmFT5ZXii5mzNJvb7LmmQhGriFFKAWm7sJvqmXNqA9r4tAKX2K09+T5YZSHwXPG3ISysSDrpJIlPw98an244OTwAg30p4GGghDSM0LQ0FURbcMpkTbVjanH1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O5tifYjhosGpQ3nDTsS5i6AA9WKlN8ce5piRJCNyMsY=; b=LIDSpaY1EJn8GjsTThPkGI2PjMSeE8RkbxDAIxH5Qj4R2+BRqkfjbM8G+5E8VVXwq7W4REq+K+Io2Lhd2Nv23/bAA4k4uLXZdWC1DsnknTrAEoeKJUcsAsXNY8mugiU4QnfZuO8PYLHz0MpSiYJEsPDL4O61euqg57svrITVPSw8tn1+au+X0urUm6PnVEjQFnzCLe0iq6lM//iSbr+NGaRNZ9j4oPtgGRddAwPFytRwba7iBae51p3MQmqGJuxBDv2NMtsapTkVJ3Bky1Qxf8ON4FDfK712D4deLXFUgXV+3ZixR3FUWFcgruf67080h73essWb9acvze+VYTkQ4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O5tifYjhosGpQ3nDTsS5i6AA9WKlN8ce5piRJCNyMsY=; b=Nw7DIMO17S8HToGO5q1GmJdfi2HjJFTJeuDybUVscxMDtP2l2jM9x8N7VBUeYX4Y5FgsBYtozPjXFPNQi157U04q42ZnADzjk8w/qB+w39vDQ/gSmsQrwEDkn7uTAWo1Ht7S2m801GeFLeAtfIHXGE9d2W+ymkqAys+pLfB4kx0= Received: from BN6PR1701CA0003.namprd17.prod.outlook.com (2603:10b6:405:15::13) by MN2PR12MB3502.namprd12.prod.outlook.com (2603:10b6:208:c9::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.22; Wed, 26 Feb 2020 19:02:09 +0000 Received: from BN8NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:405:15:cafe::61) by BN6PR1701CA0003.outlook.office365.com (2603:10b6:405:15::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:09 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT012.mail.protection.outlook.com (10.13.177.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:09 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:06 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:06 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 06/11] drm, cgroup: Add GEM buffer allocation count stats Date: Wed, 26 Feb 2020 14:01:47 -0500 Message-ID: <20200226190152.16131-7-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(136003)(376002)(396003)(428003)(189003)(199004)(426003)(8936002)(36756003)(81156014)(70586007)(356004)(478600001)(2616005)(8676002)(81166006)(316002)(5660300002)(6666004)(1076003)(2906002)(4326008)(110136005)(7696005)(336012)(70206006)(86362001)(26005)(186003)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR12MB3502; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7276df0e-aa1e-4d8b-1204-08d7baee6151 X-MS-TrafficTypeDiagnostic: MN2PR12MB3502: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5Xq9zBZRUJIJ5BpjkO1Q5MLIfONw92dFEq3xEBvzpScKh4ZPBWLnyPPFhMTy9nwgw25Tab+50DKIotQrQnt8p8n5uwTDT2U/jSsXdXlYF1CWO1WZ7+6kH+FvfZMLPp9Hjd8Mk9xJ8woDoGLAik7nXOLXXWyYN0ySWg6xiPZIAsrHnncXlayHF+uwKlbMOKy2l9VUaMgimUWFq5fz6G1Zcmhm2inniRA/e9kj6aIcS5FlG+/I+pSHVL58ott3xK3aHB6cf5tna5HC1dfoRFXc8ybY+0LVteqwW08xX0kLNXZrbW+SQ+Q7lKBKzIzi4ediOyW9VHdJSOwBaYS3IoW28kXUaLWp0fW6GV9H0mRzKysImixbFw/7UcVH0c4fDahFJqN7re8uCf7BArp1QV80EjwFA1gwnIZvYHAw05UZz/gBAyRUi00aMKRDYs2Gp2vMy1A36aO349kO25Imkebznv37fz5OdkNqILK/QFp522Y8EhhIXIHviq92bWimMiKmfadeu5IOSjYqXnZAL/diMMVM/YS6xfkrRgDD5lYYyB8= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:09.3450 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7276df0e-aa1e-4d8b-1204-08d7baee6151 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3502 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" gpu.buffer.count.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Total number of GEM buffer allocated. Change-Id: Iad29bdf44390dbcee07b1e72ea0ff811aa3b9dcd Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 6 ++++++ include/linux/cgroup_drm.h | 3 +++ kernel/cgroup/drm.c | 22 +++++++++++++++++++--- 3 files changed, 28 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 6199cc9a978f..065f2b52da57 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2081,6 +2081,12 @@ GPU Interface Files Largest (high water mark) GEM buffer allocated in bytes. + gpu.buffer.count.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Total number of GEM buffer allocated. + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index d90807627213..103868d972d0 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -14,6 +14,7 @@ enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, DRMCG_TYPE_BO_PEAK, + DRMCG_TYPE_BO_COUNT, __DRMCG_TYPE_LAST, }; @@ -27,6 +28,8 @@ struct drmcg_device_resource { s64 bo_stats_total_allocated; s64 bo_stats_peak_allocated; + + s64 bo_stats_count_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 68b23693418b..5a700833a304 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -280,6 +280,9 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_PEAK: seq_printf(sf, "%lld\n", ddr->bo_stats_peak_allocated); break; + case DRMCG_TYPE_BO_COUNT: + seq_printf(sf, "%lld\n", ddr->bo_stats_count_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -334,6 +337,12 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.count.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_COUNT, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -385,6 +394,8 @@ void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, if (ddr->bo_stats_peak_allocated < (s64)size) ddr->bo_stats_peak_allocated = (s64)size; + + ddr->bo_stats_count_allocated++; } mutex_unlock(&dev->drmcg_mutex); } @@ -402,15 +413,20 @@ EXPORT_SYMBOL(drmcg_chg_bo_alloc); void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { + struct drmcg_device_resource *ddr; int devIdx = dev->primary->index; if (drmcg == NULL) return; mutex_lock(&dev->drmcg_mutex); - for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) - drmcg->dev_resources[devIdx]->bo_stats_total_allocated - -= (s64)size; + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + + ddr->bo_stats_total_allocated -= (s64)size; + + ddr->bo_stats_count_allocated--; + } mutex_unlock(&dev->drmcg_mutex); } EXPORT_SYMBOL(drmcg_unchg_bo_alloc); From patchwork Wed Feb 26 19:01:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407083 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADBC914B4 for ; Wed, 26 Feb 2020 19:02:43 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 894A92072D for ; Wed, 26 Feb 2020 19:02:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="y7I620ll" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 894A92072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6CEAB6EBCB; Wed, 26 Feb 2020 19:02:29 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-eopbgr750071.outbound.protection.outlook.com [40.107.75.71]) by gabe.freedesktop.org (Postfix) with ESMTPS id DEB7B6EBAE; Wed, 26 Feb 2020 19:02:11 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BjHQGUY2EhWuiTjV+jcp+w38PRGKXBJ6CFPZKv/XdQowJew8fcInCqv9YShfYmXoIirDYfonnnHgEm3BqMAtDFSZl53VVg8lYVtgFAxhA+vq16RQ2GSs2xCVtXsKkNoOS3MFZ01h6YZxzWoFzVmTyKj+vtVxPE1KlXmGhYaatLbNiwzsm2UN3I/ILem55uUrAekPOyEdvUYt9/GZ4iybqHjM92x3C56jvdkmnndb98qOM+OG4cb8jD3sIchUmtEWqIQhyx74aO9AJXr1+eLVfB67qkwyCGZTU3mCQSsjVxSirGxIwm/TDoCDMkOhh47snwMppBDnwtrCOwxO5C7/ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZtbIUCcTEiB8r0n5RAMV9tMcF6GnAvzY3cASx+IKURY=; b=emLkzoYWlH+fLlWzFez2/pBG1Fv/xf3vPmTrlwBFWBYEp8jzoMTMKwAiSNx03ZL+peDg3T3Q+41Un0ay7cPCc+eeJkoFocqu7wpUTcRs3Qkk62c+q1hL0XA8qyxNaJBH7UIH0IwCqk8AsHjswNqQHygsVM+lotzR7/Ly5oMWp6H2MTun/tKsnkQ0qeME+IK4nvxtxwCgfBVXqGME06Qe8wyurHuSe5TUuhYWt/rF1pdZdx8DUiPbauJh3g0pWuiWH/ptXWRj5o1oaFLZXLLqo8A7KBesSIHvyyiCCJCxHsDSg5ao0NiH7p/uPZ5kZDZWdh9yNjfjwSt5u0EzWpW9ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZtbIUCcTEiB8r0n5RAMV9tMcF6GnAvzY3cASx+IKURY=; b=y7I620llbdPyqeDCZVflQsR5Az+01WIJSNOa2yO4Ma9KoomvHIwlUQ/p/An1Wf2zT1muXlsvvv8cHcxkN5qD/orj5l5cU2ozNPIH3P2pOBKdH0rUG0KaY0dXzJm5sReTXfGYaQnc/ra4xN7ozusuUovf8p2+PIr+zkcsDzVVnVU= Received: from DM5PR04CA0057.namprd04.prod.outlook.com (2603:10b6:3:ef::19) by BY5PR12MB3713.namprd12.prod.outlook.com (2603:10b6:a03:1a6::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.22; Wed, 26 Feb 2020 19:02:09 +0000 Received: from DM6NAM11FT017.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ef:cafe::9e) by DM5PR04CA0057.outlook.office365.com (2603:10b6:3:ef::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:09 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by DM6NAM11FT017.mail.protection.outlook.com (10.13.172.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:08 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:08 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:07 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:07 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 07/11] drm, cgroup: Add total GEM buffer allocation limit Date: Wed, 26 Feb 2020 14:01:48 -0500 Message-ID: <20200226190152.16131-8-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(39860400002)(376002)(346002)(396003)(428003)(189003)(199004)(2906002)(70206006)(7696005)(316002)(36756003)(356004)(8676002)(70586007)(110136005)(81166006)(81156014)(6666004)(8936002)(4326008)(478600001)(1076003)(86362001)(30864003)(336012)(2616005)(426003)(26005)(186003)(5660300002)(921003)(83996005)(2101003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:BY5PR12MB3713; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bb9f5da8-6077-4a61-8670-08d7baee610c X-MS-TrafficTypeDiagnostic: BY5PR12MB3713: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CGezJMHQr/QbUaHhW9VOOG9/Tyvg26/AZBo0LeWjk5HOTWnrKNhQrYcqH11v6qqFmTD/MxwAPu0vFQhXNDBvhwIdPsRq+UEPKtVaT3pSylMJh5T2q+TWu41rJn8fTWGeDrvG5ydrRLXQ5CAIt9fR8X4Do1CgcnNJHClEPT1xWrTRSy3K30OHBHwZdA1MaF0YO6EFG8Rjgi3F2d21h6EqqjSldsf50qkqgm1/hDTmxjGgFvkny8s1TjE9NOi/3tvznl0oQPsCXebUEsr36DvV4TzDd2OH/Ax4RuMyaIc7zc3XJ1u0Uj+NE05m0luMqw69pALiQQomNHEvm5rjq8vM4+PTkoOcvMJ63QH7EYj58j+kpd9bhQLvUyn/JUZSx/3Kwp23GdzkPqC/Gd/+tAs+npr1FC4fDVE2wpwTrDoHK92yIDL+BM4X+W6sFqNN9TYJ3RpaaNE3GQ/CcMdPrh3M67ejtmmbGNQFFQ7Gjy2OnfvmLJdUCYV2niFWfB143rmoZzEf1HWJ2mhA7+7AN8JCA5XBv2kk+oFmx0JYnG+hp5g= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:08.8438 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bb9f5da8-6077-4a61-8670-08d7baee610c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3713 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The drm resource being limited here is the GEM buffer objects. User applications allocate and free these buffers. In addition, a process can allocate a buffer and share it with another process. The consumer of a shared buffer can also outlive the allocator of the buffer. For the purpose of cgroup accounting and limiting, ownership of the buffer is deemed to be the cgroup for which the allocating process belongs to. There is one cgroup limit per drm device. The limiting functionality is added to the previous stats collection function. The drm_gem_private_object_init is modified to have a return value to allow failure due to cgroup limit. The try_chg function only fails if the DRM cgroup properties has limit_enforced set to true for the DRM device. This is to allow the DRM cgroup controller to collect usage stats without enforcing the limits. gpu.buffer.default A read-only flat-keyed file which exists on the root cgroup. Each entry is keyed by the drm device's major:minor. Default limits on the total GEM buffer allocation in bytes. gpu.buffer.max A read-write flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Per device limits on the total GEM buffer allocation in byte. This is a hard limit. Attempts in allocating beyond the cgroup limit will result in ENOMEM. Shorthand understood by memparse (such as k, m, g) can be used. Set allocation limit for /dev/dri/card1 to 1GB echo "226:1 1g" > gpu.buffer.total.max Set allocation limit for /dev/dri/card0 to 512MB echo "226:0 512m" > gpu.buffer.total.max Change-Id: Id3265bbd0fafe84a16b59617df79bd32196160be Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 21 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 19 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 6 +- drivers/gpu/drm/drm_gem.c | 11 +- include/drm/drm_cgroup.h | 8 +- include/drm/drm_gem.h | 2 +- include/linux/cgroup_drm.h | 1 + kernel/cgroup/drm.c | 227 ++++++++++++++++++++- 8 files changed, 278 insertions(+), 17 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 065f2b52da57..f2d7abf5c783 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2087,6 +2087,27 @@ GPU Interface Files Total number of GEM buffer allocated. + gpu.buffer.default + A read-only flat-keyed file which exists on the root cgroup. + Each entry is keyed by the drm device's major:minor. + + Default limits on the total GEM buffer allocation in bytes. + + gpu.buffer.max + A read-write flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Per device limits on the total GEM buffer allocation in byte. + This is a hard limit. Attempts in allocating beyond the cgroup + limit will result in ENOMEM. Shorthand understood by memparse + (such as k, m, g) can be used. + + Set allocation limit for /dev/dri/card1 to 1GB + echo "226:1 1g" > gpu.buffer.total.max + + Set allocation limit for /dev/dri/card0 to 512MB + echo "226:0 512m" > gpu.buffer.total.max + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 6e1faf8a2bca..171397708855 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -1413,6 +1413,23 @@ amdgpu_get_crtc_scanout_position(struct drm_device *dev, unsigned int pipe, stime, etime, mode); } +#ifdef CONFIG_CGROUP_DRM + +static void amdgpu_drmcg_custom_init(struct drm_device *dev, + struct drmcg_props *props) +{ + props->limit_enforced = true; +} + +#else + +static void amdgpu_drmcg_custom_init(struct drm_device *dev, + struct drmcg_props *props) +{ +} + +#endif /* CONFIG_CGROUP_DRM */ + static struct drm_driver kms_driver = { .driver_features = DRIVER_USE_AGP | DRIVER_ATOMIC | @@ -1444,6 +1461,8 @@ static struct drm_driver kms_driver = { .gem_prime_vunmap = amdgpu_gem_prime_vunmap, .gem_prime_mmap = amdgpu_gem_prime_mmap, + .drmcg_custom_init = amdgpu_drmcg_custom_init, + .name = DRIVER_NAME, .desc = DRIVER_DESC, .date = DRIVER_DATE, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c index 5766d20f29d8..4d08ccbc541a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c @@ -34,6 +34,7 @@ #include #include +#include #include "amdgpu.h" #include "amdgpu_trace.h" #include "amdgpu_amdkfd.h" @@ -551,7 +552,10 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, bo = kzalloc(sizeof(struct amdgpu_bo), GFP_KERNEL); if (bo == NULL) return -ENOMEM; - drm_gem_private_object_init(adev->ddev, &bo->tbo.base, size); + if (!drm_gem_private_object_init(adev->ddev, &bo->tbo.base, size)) { + kfree(bo); + return -ENOMEM; + } INIT_LIST_HEAD(&bo->shadow_list); bo->vm_bo = NULL; bo->preferred_domains = bp->preferred_domain ? bp->preferred_domain : diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index 589f8f6bde2c..30adf730da0f 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -150,11 +150,17 @@ EXPORT_SYMBOL(drm_gem_object_init); * no GEM provided backing store. Instead the caller is responsible for * backing the object and handling it. */ -void drm_gem_private_object_init(struct drm_device *dev, +bool drm_gem_private_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size) { BUG_ON((size & (PAGE_SIZE - 1)) != 0); + obj->drmcg = drmcg_get(current); + if (!drmcg_try_chg_bo_alloc(obj->drmcg, dev, size)) { + drmcg_put(obj->drmcg); + obj->drmcg = NULL; + return false; + } obj->dev = dev; obj->filp = NULL; @@ -167,8 +173,7 @@ void drm_gem_private_object_init(struct drm_device *dev, drm_vma_node_reset(&obj->vma_node); - obj->drmcg = drmcg_get(current); - drmcg_chg_bo_alloc(obj->drmcg, dev, size); + return true; } EXPORT_SYMBOL(drm_gem_private_object_init); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 1eb3012e16a1..2783e56690db 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -13,6 +13,9 @@ * of storing per device defaults */ struct drmcg_props { + bool limit_enforced; + + s64 bo_limits_total_allocated_default; }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), @@ -26,7 +29,7 @@ void drmcg_unregister_dev(struct drm_device *dev); void drmcg_device_early_init(struct drm_device *device); -void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, +bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size); void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, @@ -59,9 +62,10 @@ static inline void drmcg_device_early_init(struct drm_device *device) { } -static inline void drmcg_chg_bo_alloc(struct drmcg *drmcg, +static inline bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { + return true; } static inline void drmcg_unchg_bo_alloc(struct drmcg *drmcg, diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h index 9c588c329da0..ef073a5e7d67 100644 --- a/include/drm/drm_gem.h +++ b/include/drm/drm_gem.h @@ -352,7 +352,7 @@ void drm_gem_object_release(struct drm_gem_object *obj); void drm_gem_object_free(struct kref *kref); int drm_gem_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size); -void drm_gem_private_object_init(struct drm_device *dev, +bool drm_gem_private_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size); void drm_gem_vm_open(struct vm_area_struct *vma); void drm_gem_vm_close(struct vm_area_struct *vma); diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 103868d972d0..71023654fb77 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -26,6 +26,7 @@ enum drmcg_res_type { struct drmcg_device_resource { /* for per device stats */ s64 bo_stats_total_allocated; + s64 bo_limits_total_allocated; s64 bo_stats_peak_allocated; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 5a700833a304..4b19e533941d 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -37,6 +37,8 @@ static void (*put_drm_dev)(struct drm_device *dev); enum drmcg_file_type { DRMCG_FTYPE_STATS, + DRMCG_FTYPE_LIMIT, + DRMCG_FTYPE_DEFAULT, }; /** @@ -90,6 +92,8 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) drmcg->dev_resources[minor] = ddr; /* set defaults here */ + ddr->bo_limits_total_allocated = + dev->drmcg_props.bo_limits_total_allocated_default; return 0; } @@ -289,6 +293,38 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, } } +static void drmcg_print_limits(struct drmcg_device_resource *ddr, + struct seq_file *sf, enum drmcg_res_type type) +{ + if (ddr == NULL) { + seq_puts(sf, "\n"); + return; + } + + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + +static void drmcg_print_default(struct drmcg_props *props, + struct seq_file *sf, enum drmcg_res_type type) +{ + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", + props->bo_limits_total_allocated_default); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + static int drmcg_seq_show_fn(int id, void *ptr, void *data) { struct drm_minor *minor = ptr; @@ -311,6 +347,12 @@ static int drmcg_seq_show_fn(int id, void *ptr, void *data) case DRMCG_FTYPE_STATS: drmcg_print_stats(ddr, sf, type); break; + case DRMCG_FTYPE_LIMIT: + drmcg_print_limits(ddr, sf, type); + break; + case DRMCG_FTYPE_DEFAULT: + drmcg_print_default(&minor->dev->drmcg_props, sf, type); + break; default: seq_puts(sf, "\n"); break; @@ -324,6 +366,130 @@ int drmcg_seq_show(struct seq_file *sf, void *v) return drm_minor_for_each(&drmcg_seq_show_fn, sf); } +static void drmcg_pr_cft_err(const struct drmcg *drmcg, + int rc, const char *cft_name, int minor) +{ + pr_err("drmcg: error parsing %s, minor %d, rc %d ", + cft_name, minor, rc); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); +} + +static int drmcg_process_limit_s64_val(char *sval, bool is_mem, + s64 def_val, s64 max_val, s64 *ret_val) +{ + int rc = strcmp("max", sval); + + + if (!rc) + *ret_val = max_val; + else { + rc = strcmp("default", sval); + + if (!rc) + *ret_val = def_val; + } + + if (rc) { + if (is_mem) { + *ret_val = memparse(sval, NULL); + rc = 0; + } else { + rc = kstrtoll(sval, 0, ret_val); + } + } + + if (*ret_val > max_val) + rc = -EINVAL; + + return rc; +} + +/** + * drmcg_limit_write - parse cgroup interface files to obtain user config + * + * Minimal value check to keep track of user intent. For example, user + * can specify limits greater than the values allowed by the parents. + * This way, the user configuration is kept and comes into effect if and + * when parents' limits are relaxed. + */ +static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + struct drmcg *drmcg = css_to_drmcg(of_css(of)); + enum drmcg_res_type type = + DRMCG_CTF_PRIV2RESTYPE(of_cft(of)->private); + char *cft_name = of_cft(of)->name; + char *limits = strstrip(buf); + struct drmcg_device_resource *ddr; + struct drmcg_props *props; + struct drm_minor *dm; + char *line; + char sattr[256]; + s64 val; + int rc; + int minor; + + while (limits != NULL) { + line = strsep(&limits, "\n"); + + if (sscanf(line, + __stringify(DRM_MAJOR)":%u %255[^\t\n]", + &minor, sattr) != 2) { + pr_err("drmcg: error parsing %s ", cft_name); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); + + continue; + } + + mutex_lock(&drmcg_mutex); + if (acquire_drm_minor) + dm = acquire_drm_minor(minor); + else + dm = NULL; + mutex_unlock(&drmcg_mutex); + + if (IS_ERR_OR_NULL(dm)) { + pr_err("drmcg: invalid minor %d for %s ", + minor, cft_name); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); + + continue; + } + + mutex_lock(&dm->dev->drmcg_mutex); + ddr = drmcg->dev_resources[minor]; + props = &dm->dev->drmcg_props; + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + rc = drmcg_process_limit_s64_val(sattr, true, + props->bo_limits_total_allocated_default, + S64_MAX, + &val); + + if (rc || val < 0) { + drmcg_pr_cft_err(drmcg, rc, cft_name, minor); + break; + } + + ddr->bo_limits_total_allocated = val; + break; + default: + break; + } + mutex_unlock(&dm->dev->drmcg_mutex); + + mutex_lock(&drmcg_mutex); + if (put_drm_dev) + put_drm_dev(dm->dev); /* release from acquire */ + mutex_unlock(&drmcg_mutex); + } + + return nbytes; +} + struct cftype files[] = { { .name = "buffer.total.stats", @@ -331,6 +497,20 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.total.default", + .seq_show = drmcg_seq_show, + .flags = CFTYPE_ONLY_ON_ROOT, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_DEFAULT), + }, + { + .name = "buffer.total.max", + .write = drmcg_limit_write, + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_LIMIT), + }, { .name = "buffer.peak.stats", .seq_show = drmcg_seq_show, @@ -363,12 +543,16 @@ struct cgroup_subsys gpu_cgrp_subsys = { */ void drmcg_device_early_init(struct drm_device *dev) { + dev->drmcg_props.limit_enforced = false; + + dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; + drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); /** - * drmcg_chg_bo_alloc - charge GEM buffer usage for a device and cgroup + * drmcg_try_chg_bo_alloc - charge GEM buffer usage for a device and cgroup * @drmcg: the DRM cgroup to be charged to * @dev: the device the usage should be charged to * @size: size of the GEM buffer to be accounted for @@ -377,29 +561,52 @@ EXPORT_SYMBOL(drmcg_device_early_init); * for the utilization. This should not be called when the buffer is shared ( * the GEM buffer's reference count being incremented.) */ -void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, +bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { struct drmcg_device_resource *ddr; int devIdx = dev->primary->index; + struct drmcg_props *props = &dev->drmcg_props; + struct drmcg *drmcg_cur = drmcg; + bool result = true; + s64 delta = 0; if (drmcg == NULL) - return; + return true; mutex_lock(&dev->drmcg_mutex); - for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { - ddr = drmcg->dev_resources[devIdx]; + if (props->limit_enforced) { + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + delta = ddr->bo_limits_total_allocated - + ddr->bo_stats_total_allocated; + + if (delta <= 0 || size > delta) { + result = false; + break; + } + } + } + + drmcg = drmcg_cur; + + if (result || !props->limit_enforced) { + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; - ddr->bo_stats_total_allocated += (s64)size; + ddr->bo_stats_total_allocated += (s64)size; - if (ddr->bo_stats_peak_allocated < (s64)size) - ddr->bo_stats_peak_allocated = (s64)size; + if (ddr->bo_stats_peak_allocated < (s64)size) + ddr->bo_stats_peak_allocated = (s64)size; - ddr->bo_stats_count_allocated++; + ddr->bo_stats_count_allocated++; + } } mutex_unlock(&dev->drmcg_mutex); + + return result; } -EXPORT_SYMBOL(drmcg_chg_bo_alloc); +EXPORT_SYMBOL(drmcg_try_chg_bo_alloc); /** * drmcg_unchg_bo_alloc - From patchwork Wed Feb 26 19:01:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407081 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12C9514B4 for ; Wed, 26 Feb 2020 19:02:42 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E4D292072D for ; Wed, 26 Feb 2020 19:02:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="KgJaquNe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E4D292072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 994186EBC6; Wed, 26 Feb 2020 19:02:27 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2047.outbound.protection.outlook.com [40.107.94.47]) by gabe.freedesktop.org (Postfix) with ESMTPS id D6B1A6EBA9; Wed, 26 Feb 2020 19:02:11 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZL68/XCvoVpArNIHFZUTGrT1jigxcEBdiyn7A72/b30WDqQpwofNrxC5AHjq9AP5E3phqxoylueuorU16NpkPIeN23r9nQXdpJNTZUAuuTIJ+cv7/mdfuYGM53REqXF4y2RulaKZ0HUfZ9/qI/EC9lnRH3uYy2dhuZiIYDA3k7NTN5a9Z6NS3XgN3ZCNlDPeNINSwgpQ0hGCMXcjr+MZNLKq8awIMihNYp1dk7ddhRVBtxBq2fu4lPPDKFfLT3BPr8wBP29MhAjL9XO6n465c565bRw+UGK37+HPJdGAtSqo6X2CT6/T1YHDDYob6rDe7snKltSNe+JHQUlBjSikcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W0yw5UORnhD9jhJZKoq3U0mozi1+naoh8kxFcJWfwYQ=; b=gYBHz9aNVpOj+aNocqbh2kTFmQy7L6EpvjgtBySNiGnHD7tI/pQ+cnhAONUJZTQXsGZ0Vm/VREDBfAifIi2g3qASSbhgIbVWgRtE4r/x/ES2XwsjGc1SY5YBMWNXbpx71wQYRn3UX13ePYs4Q+LtiK+UPaf6chHQsULEj0hPUx61rPWHPT10WHwJkIMZnqOQJ3Zfkr8xcsSAQPU/XdFGJer0gMlBaW3oxVSqecwYK1rvwzT526YzqwLNSyxGeKN3IuzBKYiZCwRRNt4qH3tC6tmXuj+E4Ndw5bQOUKrH3rjQ/UWonFwb2iuPTqj1WJNpY7S/KvgngyuIOM6XQqya2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W0yw5UORnhD9jhJZKoq3U0mozi1+naoh8kxFcJWfwYQ=; b=KgJaquNe0Hwu5kpjK6KkUnJ0pP6tS1H8sRS2QGP9xFnngd7UY9bqVDejDQ0SS7tPcSvTN3kLPt7Gcocw5f5cPvMpGjveMoUOe0Opc5fRz3oWMEbnvba9tvz3qnKwdjhF32ioJrCbDgNsY3ZubGGd0SJCG9XfJixctaG0J18bTlo= Received: from BN6PR1701CA0011.namprd17.prod.outlook.com (2603:10b6:405:15::21) by DM5PR12MB1337.namprd12.prod.outlook.com (2603:10b6:3:6e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.22; Wed, 26 Feb 2020 19:02:10 +0000 Received: from BN8NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:405:15:cafe::13) by BN6PR1701CA0011.outlook.office365.com (2603:10b6:405:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:10 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT012.mail.protection.outlook.com (10.13.177.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:09 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:08 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:08 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:08 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 08/11] drm, cgroup: Add peak GEM buffer allocation limit Date: Wed, 26 Feb 2020 14:01:49 -0500 Message-ID: <20200226190152.16131-9-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(136003)(346002)(376002)(39860400002)(428003)(189003)(199004)(186003)(2616005)(26005)(426003)(336012)(4326008)(81156014)(81166006)(8676002)(6666004)(2906002)(1076003)(8936002)(316002)(356004)(70206006)(36756003)(478600001)(86362001)(7696005)(70586007)(5660300002)(110136005)(921003)(2101003)(83996005)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR12MB1337; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c46743ce-005a-444d-a350-08d7baee61ad X-MS-TrafficTypeDiagnostic: DM5PR12MB1337: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kx2r1P3QAN8LF4XsYHkJ8LbzczTJ8h9aBtACrEHymsrkZkpDTE9r4nP2ftBJMks3+O2m+Nai43BKdMu0ndUA9T+/5JvS9Q7tjuHUPY5qS4cCzkSxS9INPzNA1i6CB6hCdMZy3KsLgQKQWTgaQFgP/5tarorcdDE/EI/x3MzzsxrFfNvwm4dyjJhZ/tmHUX9XQsCdGj3+UPnJj3/UcyivhksiuUySXkwVEBrnH83B4VMShXz7k0ifC7aTtVqoyulFDN6VPbNwyGo3KYM6u37dliD5GncU2LAqE4W6Avpj9IKazmxMlONrvEiUYmSkanw17EF6fj/E5mBEmU0H1rZpCOjeE2tLZq7TE0eWD/N6UGDdEH6rO6ZHlz3s2cMSkE0GQHTKrNi24Wmqa1chvPiEY1Yi1f8xQlaiV1PvJKhMMVHRJ9qH95tTJoJ9i47Hb/HmB+Qw9ojrWGvLyDinSqz/IaCQEe0+Lyr8WXIGVHFO65HOAvmrebFVwfIL4KgWuqLWElTNu43LOlKT7MddCjw8l3Qj+DpYzM4C5VVfXLFrarQ= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:09.9537 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c46743ce-005a-444d-a350-08d7baee61ad X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1337 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" gpu.buffer.peak.default A read-only flat-keyed file which exists on the root cgroup. Each entry is keyed by the drm device's major:minor. Default limits on the largest GEM buffer allocation in bytes. gpu.buffer.peak.max A read-write flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Per device limits on the largest GEM buffer allocation in bytes. This is a hard limit. Attempts in allocating beyond the cgroup limit will result in ENOMEM. Shorthand understood by memparse (such as k, m, g) can be used. Set largest allocation for /dev/dri/card1 to 4MB echo "226:1 4m" > gpu.buffer.peak.max Change-Id: I5ab3fb4a442b6cbd5db346be595897c90217da69 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 18 +++++++++++ include/drm/drm_cgroup.h | 1 + include/linux/cgroup_drm.h | 1 + kernel/cgroup/drm.c | 43 +++++++++++++++++++++++++ 4 files changed, 63 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index f2d7abf5c783..581343472651 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2108,6 +2108,24 @@ GPU Interface Files Set allocation limit for /dev/dri/card0 to 512MB echo "226:0 512m" > gpu.buffer.total.max + gpu.buffer.peak.default + A read-only flat-keyed file which exists on the root cgroup. + Each entry is keyed by the drm device's major:minor. + + Default limits on the largest GEM buffer allocation in bytes. + + gpu.buffer.peak.max + A read-write flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Per device limits on the largest GEM buffer allocation in bytes. + This is a hard limit. Attempts in allocating beyond the cgroup + limit will result in ENOMEM. Shorthand understood by memparse + (such as k, m, g) can be used. + + Set largest allocation for /dev/dri/card1 to 4MB + echo "226:1 4m" > gpu.buffer.peak.max + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 2783e56690db..2b41d4d22e33 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -16,6 +16,7 @@ struct drmcg_props { bool limit_enforced; s64 bo_limits_total_allocated_default; + s64 bo_limits_peak_allocated_default; }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 71023654fb77..aba3b26718c0 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -29,6 +29,7 @@ struct drmcg_device_resource { s64 bo_limits_total_allocated; s64 bo_stats_peak_allocated; + s64 bo_limits_peak_allocated; s64 bo_stats_count_allocated; }; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 4b19e533941d..62d2a9d33d0c 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -95,6 +95,9 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) ddr->bo_limits_total_allocated = dev->drmcg_props.bo_limits_total_allocated_default; + ddr->bo_limits_peak_allocated = + dev->drmcg_props.bo_limits_peak_allocated_default; + return 0; } @@ -305,6 +308,9 @@ static void drmcg_print_limits(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_TOTAL: seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", ddr->bo_limits_peak_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -319,6 +325,10 @@ static void drmcg_print_default(struct drmcg_props *props, seq_printf(sf, "%lld\n", props->bo_limits_total_allocated_default); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", + props->bo_limits_peak_allocated_default); + break; default: seq_puts(sf, "\n"); break; @@ -476,6 +486,19 @@ static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, ddr->bo_limits_total_allocated = val; break; + case DRMCG_TYPE_BO_PEAK: + rc = drmcg_process_limit_s64_val(sattr, true, + props->bo_limits_peak_allocated_default, + S64_MAX, + &val); + + if (rc || val < 0) { + drmcg_pr_cft_err(drmcg, rc, cft_name, minor); + break; + } + + ddr->bo_limits_peak_allocated = val; + break; default: break; } @@ -517,6 +540,20 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.peak.default", + .seq_show = drmcg_seq_show, + .flags = CFTYPE_ONLY_ON_ROOT, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_DEFAULT), + }, + { + .name = "buffer.peak.max", + .write = drmcg_limit_write, + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_LIMIT), + }, { .name = "buffer.count.stats", .seq_show = drmcg_seq_show, @@ -546,6 +583,7 @@ void drmcg_device_early_init(struct drm_device *dev) dev->drmcg_props.limit_enforced = false; dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; + dev->drmcg_props.bo_limits_peak_allocated_default = S64_MAX; drmcg_update_cg_tree(dev); } @@ -585,6 +623,11 @@ bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, result = false; break; } + + if (ddr->bo_limits_peak_allocated < size) { + result = false; + break; + } } } From patchwork Wed Feb 26 19:01:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407079 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 82036930 for ; Wed, 26 Feb 2020 19:02:40 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F3242072D for ; Wed, 26 Feb 2020 19:02:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="PIMxun9I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F3242072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 07AD86EBC5; Wed, 26 Feb 2020 19:02:27 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2084.outbound.protection.outlook.com [40.107.236.84]) by gabe.freedesktop.org (Postfix) with ESMTPS id A44606EBA9; Wed, 26 Feb 2020 19:02:14 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bUMMbuZM8ZuxYLqIzil8RdXG6RMy0cPBljgeiJ7VG/IX3dPFycGIBLpMj1PQ5iiFKcX0nMxQHIMFBs7EuwJnxm+V8qvhsUyqWOB86yekCMgCRcIJfutof4GJ4B1/vhXhbxS1Owfj9jADb48xVKmZ33I34fC59s93QL/C+69N5jH7oc5ldCdJLKymCvh+qKEW3Ye3+thU6IrZhYDcSEzHMh8a2sDEU/4tt3jSWhJ9ZYectMwbYsVq2IiZH5LjeUHTDo61c590QItiO0EB3qDI0Gb6ElWn+tumBKvDNumg0OOIhJ9EkD541tbsTcPb3CHUz2j30Y2HEniROxMUJEiG4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fZS7Y5c64/ZEFBwFzkh5BDxJTRdiicRP1BQpcSK8Pf4=; b=mZ/Ywv72rCQsi7MZFcFFiXgoum2yOqu2Cj59FADByhoMhJ8kot4DMt3CR9mTI2AdGOdYJ1NkKOXCmIL8QOElgIXbhqNSKxl58VTvbLad90CzqQDq36fo65MaSBY13PQXCAfsEY2o1jlbYnmjZdZZ1c+E0z2FhqAQkOACD3zDLpxdI6zZbw40BVysACAdlCUU23O0KuP9A7oLZVy+48Mymsva2DXveRimpJ3M2y1a2rcEeC9wtpQAzjupVKKwV59C52DntOOZNkFt9CLO0PDwl0ZrVPEZa8yIDApZElOEkwDt0AqR+Wt8d5ti0VSrvBdeRMKBijpjdBr2ciRg3vLl0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fZS7Y5c64/ZEFBwFzkh5BDxJTRdiicRP1BQpcSK8Pf4=; b=PIMxun9I+kyJMFzZGLPIWePdBPuyMJDae3MkHwey25jHG2DWdowd1GYYUveoGs4+gHT80yORKBy+3zlv+p9CCKqs8m31uaGVMDku0dOCpfALlCyUg6V/EbivxK7sXR19Nx59/IIQgSjxgJk2Bq0lwHRWXamiQxBKMAypfFAAYjA= Received: from DM5PR04CA0066.namprd04.prod.outlook.com (2603:10b6:3:ef::28) by DM6PR12MB2969.namprd12.prod.outlook.com (2603:10b6:5:115::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.17; Wed, 26 Feb 2020 19:02:10 +0000 Received: from DM6NAM11FT017.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ef:cafe::b6) by DM5PR04CA0066.outlook.office365.com (2603:10b6:3:ef::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:10 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by DM6NAM11FT017.mail.protection.outlook.com (10.13.172.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:10 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:09 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:09 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:08 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 09/11] drm, cgroup: Add compute as gpu cgroup resource Date: Wed, 26 Feb 2020 14:01:50 -0500 Message-ID: <20200226190152.16131-10-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(396003)(376002)(136003)(346002)(428003)(199004)(189003)(70206006)(356004)(6666004)(5660300002)(81156014)(81166006)(70586007)(8676002)(1076003)(36756003)(4326008)(86362001)(7696005)(30864003)(316002)(2906002)(478600001)(8936002)(186003)(26005)(2616005)(336012)(426003)(110136005)(921003)(83996005)(2101003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB2969; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9f200430-4511-432a-43e3-08d7baee61e1 X-MS-TrafficTypeDiagnostic: DM6PR12MB2969: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2449; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hudmHUcIjsfGTeIrJyggKrIhUo0DyjInxYJHdzQvFbb49YlS0R9/VkqKEZxOb/tOC7NPeF4oMm/Q3xjWhaEaav+aPuRyIa4dkFu+3IZH7Gt6ZjF2fp8R5kj8B2LO/nA+3YDtXdspiIZlQPcaeLU+fs7eEMpNYcRUvMVH/iaav6owvUC4RwGZ76j2kx7j1s05p0LC8t4Ps5mFhNu914CXB117Xxe8tsY4KZrV5/jkECL0yaGSRsqGtycGXm6DxQcyJDCAmx56sc3GaaPLqoOlrkzbW+LnPCsHVdaMZRe9KVBaw79TTBK9rwqMwVe+H8/vZbdXhnSrWczIEBiHoY90aPuAY+PqTAMjsOHVLxGUkzguc9Si+2Z2OuhpFoBrE0enYZbryOSrRtcAWrKB8XjHgB4Kl26ns46AhITJTmeXN5sHTK2GTfTGj3ma/+/XheoItFwn3SPF7ZOqGwtGMT55B5rgo5OB1E9HpJQBnpoowaIdrO6L6Fn+8pjsRQkfEbPgqGxN4m/ocqWi66uXXSY2o/s8anYo5gj1gaEwDt9sPpo= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:10.2320 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9f200430-4511-432a-43e3-08d7baee61e1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2969 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" gpu.compute.weight A read-write flat-keyed file which exists on all cgroups. The default weight is 100. Each entry is keyed by the DRM device's major:minor (the primary minor). The weights are in the range [1, 10000] and specifies the relative amount of physical partitions the cgroup can use in relation to its siblings. The partition concept here is analogous to the subdevice of OpenCL. gpu.compute.effective A read-only nested-keyed file which exists on all cgroups. Each entry is keyed by the DRM device's major:minor. It lists the GPU subdevices that are actually granted to this cgroup by its parent. These subdevices are allowed to be used by tasks within the current cgroup. ===== ============================================== count The total number of granted subdevices list Enumeration of the subdevices ===== ============================================== Change-Id: Idde0ef9a331fd67bb9c7eb8ef9978439e6452488 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 21 +++ include/drm/drm_cgroup.h | 3 + include/linux/cgroup_drm.h | 16 +++ kernel/cgroup/drm.c | 177 +++++++++++++++++++++++- 4 files changed, 215 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 581343472651..f92f1f4a64d4 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2126,6 +2126,27 @@ GPU Interface Files Set largest allocation for /dev/dri/card1 to 4MB echo "226:1 4m" > gpu.buffer.peak.max + gpu.compute.weight + A read-write flat-keyed file which exists on all cgroups. The + default weight is 100. Each entry is keyed by the DRM device's + major:minor (the primary minor). The weights are in the range + [1, 10000] and specifies the relative amount of physical partitions + the cgroup can use in relation to its siblings. The partition + concept here is analogous to the subdevice concept of OpenCL. + + gpu.compute.effective + A read-only nested-keyed file which exists on all cgroups. + Each entry is keyed by the DRM device's major:minor. + + It lists the GPU subdevices that are actually granted to this + cgroup by its parent. These subdevices are allowed to be used + by tasks within the current cgroup. + + ===== ============================================== + count The total number of granted subdevices + list Enumeration of the subdevices + ===== ============================================== + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 2b41d4d22e33..5aac47ca536f 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -17,6 +17,9 @@ struct drmcg_props { s64 bo_limits_total_allocated_default; s64 bo_limits_peak_allocated_default; + + int compute_capacity; + DECLARE_BITMAP(compute_slots, MAX_DRMCG_COMPUTE_CAPACITY); }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index aba3b26718c0..fd02f59cabab 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -11,10 +11,14 @@ /* limit defined per the way drm_minor_alloc operates */ #define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) +#define MAX_DRMCG_COMPUTE_CAPACITY 256 + enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, DRMCG_TYPE_BO_PEAK, DRMCG_TYPE_BO_COUNT, + DRMCG_TYPE_COMPUTE, + DRMCG_TYPE_COMPUTE_EFF, __DRMCG_TYPE_LAST, }; @@ -32,6 +36,18 @@ struct drmcg_device_resource { s64 bo_limits_peak_allocated; s64 bo_stats_count_allocated; + + /* compute_stg is used to calculate _eff before applying to _eff + * after considering the entire hierarchy + */ + DECLARE_BITMAP(compute_stg, MAX_DRMCG_COMPUTE_CAPACITY); + /* user configurations */ + s64 compute_weight; + /* effective compute for the cgroup after considering + * relationship with other cgroup + */ + s64 compute_count_eff; + DECLARE_BITMAP(compute_eff, MAX_DRMCG_COMPUTE_CAPACITY); }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 62d2a9d33d0c..2eadabebdfea 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -98,6 +99,11 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) ddr->bo_limits_peak_allocated = dev->drmcg_props.bo_limits_peak_allocated_default; + bitmap_copy(ddr->compute_stg, dev->drmcg_props.compute_slots, + MAX_DRMCG_COMPUTE_CAPACITY); + + ddr->compute_weight = CGROUP_WEIGHT_DFL; + return 0; } @@ -121,6 +127,104 @@ static inline void drmcg_update_cg_tree(struct drm_device *dev) mutex_unlock(&cgroup_mutex); } +static void drmcg_calculate_effective_compute(struct drm_device *dev, + const unsigned long *free_weighted, + struct drmcg *parent_drmcg) +{ + int capacity = dev->drmcg_props.compute_capacity; + DECLARE_BITMAP(compute_unused, MAX_DRMCG_COMPUTE_CAPACITY); + DECLARE_BITMAP(compute_by_weight, MAX_DRMCG_COMPUTE_CAPACITY); + struct drmcg_device_resource *parent_ddr; + struct drmcg_device_resource *ddr; + int minor = dev->primary->index; + struct cgroup_subsys_state *pos; + struct drmcg *child; + s64 weight_sum = 0; + s64 unused; + + parent_ddr = parent_drmcg->dev_resources[minor]; + + /* no static cfg, use weight for calculating the effective */ + bitmap_copy(parent_ddr->compute_stg, free_weighted, capacity); + + /* calculate compute available for dist by weight for children */ + bitmap_copy(compute_unused, parent_ddr->compute_stg, capacity); + unused = bitmap_weight(compute_unused, capacity); + css_for_each_child(pos, &parent_drmcg->css) { + child = css_to_drmcg(pos); + ddr = child->dev_resources[minor]; + + /* no static allocation, participate in weight dist */ + weight_sum += ddr->compute_weight; + } + + css_for_each_child(pos, &parent_drmcg->css) { + int c; + int p = 0; + child = css_to_drmcg(pos); + ddr = child->dev_resources[minor]; + + bitmap_zero(compute_by_weight, capacity); + for (c = ddr->compute_weight * unused / weight_sum; + c > 0; c--) { + p = find_next_bit(compute_unused, capacity, p); + if (p < capacity) { + clear_bit(p, compute_unused); + set_bit(p, compute_by_weight); + } + } + + drmcg_calculate_effective_compute(dev, compute_by_weight, child); + } +} + +static void drmcg_apply_effective_compute(struct drm_device *dev) +{ + int capacity = dev->drmcg_props.compute_capacity; + int minor = dev->primary->index; + struct drmcg_device_resource *ddr; + struct cgroup_subsys_state *pos; + struct drmcg *drmcg; + + if (root_drmcg == NULL) { + WARN_ON(root_drmcg == NULL); + return; + } + + rcu_read_lock(); + + /* process the entire cgroup tree from root to simplify the algorithm */ + drmcg_calculate_effective_compute(dev, dev->drmcg_props.compute_slots, + root_drmcg); + + /* apply changes to effective only if there is a change */ + css_for_each_descendant_pre(pos, &root_drmcg->css) { + drmcg = css_to_drmcg(pos); + ddr = drmcg->dev_resources[minor]; + + if (!bitmap_equal(ddr->compute_stg, + ddr->compute_eff, capacity)) { + bitmap_copy(ddr->compute_eff, ddr->compute_stg, + capacity); + ddr->compute_count_eff = + bitmap_weight(ddr->compute_eff, capacity); + } + } + rcu_read_unlock(); +} + +static void drmcg_apply_effective(enum drmcg_res_type type, + struct drm_device *dev, struct drmcg *changed_drmcg) +{ + switch (type) { + case DRMCG_TYPE_COMPUTE: + drmcg_apply_effective_compute(dev); + break; + default: + break; + } +} + /** * drmcg_register_dev - register a DRM device for usage in drm cgroup * @dev: DRM device @@ -143,7 +247,13 @@ void drmcg_register_dev(struct drm_device *dev) { dev->driver->drmcg_custom_init(dev, &dev->drmcg_props); + WARN_ON(dev->drmcg_props.compute_capacity != + bitmap_weight(dev->drmcg_props.compute_slots, + MAX_DRMCG_COMPUTE_CAPACITY)); + drmcg_update_cg_tree(dev); + + drmcg_apply_effective(DRMCG_TYPE_COMPUTE, dev, root_drmcg); } mutex_unlock(&drmcg_mutex); } @@ -297,7 +407,8 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, } static void drmcg_print_limits(struct drmcg_device_resource *ddr, - struct seq_file *sf, enum drmcg_res_type type) + struct seq_file *sf, enum drmcg_res_type type, + struct drm_device *dev) { if (ddr == NULL) { seq_puts(sf, "\n"); @@ -311,6 +422,17 @@ static void drmcg_print_limits(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_PEAK: seq_printf(sf, "%lld\n", ddr->bo_limits_peak_allocated); break; + case DRMCG_TYPE_COMPUTE: + seq_printf(sf, "%lld\n", ddr->compute_weight); + break; + case DRMCG_TYPE_COMPUTE_EFF: + seq_printf(sf, "%s=%lld %s=%*pbl\n", + "count", + ddr->compute_count_eff, + "list", + dev->drmcg_props.compute_capacity, + ddr->compute_eff); + break; default: seq_puts(sf, "\n"); break; @@ -358,7 +480,7 @@ static int drmcg_seq_show_fn(int id, void *ptr, void *data) drmcg_print_stats(ddr, sf, type); break; case DRMCG_FTYPE_LIMIT: - drmcg_print_limits(ddr, sf, type); + drmcg_print_limits(ddr, sf, type, minor->dev); break; case DRMCG_FTYPE_DEFAULT: drmcg_print_default(&minor->dev->drmcg_props, sf, type); @@ -499,9 +621,25 @@ static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, ddr->bo_limits_peak_allocated = val; break; + case DRMCG_TYPE_COMPUTE: + rc = drmcg_process_limit_s64_val(sattr, true, + CGROUP_WEIGHT_DFL, CGROUP_WEIGHT_MAX, + &val); + + if (rc || val < CGROUP_WEIGHT_MIN || + val > CGROUP_WEIGHT_MAX) { + drmcg_pr_cft_err(drmcg, rc, cft_name, minor); + break; + } + + ddr->compute_weight = val; + break; default: break; } + + drmcg_apply_effective(type, dm->dev, drmcg); + mutex_unlock(&dm->dev->drmcg_mutex); mutex_lock(&drmcg_mutex); @@ -560,12 +698,44 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_COUNT, DRMCG_FTYPE_STATS), }, + { + .name = "compute.weight", + .seq_show = drmcg_seq_show, + .write = drmcg_limit_write, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_COMPUTE, + DRMCG_FTYPE_LIMIT), + }, + { + .name = "compute.effective", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_COMPUTE_EFF, + DRMCG_FTYPE_LIMIT), + }, { } /* terminate */ }; +static int drmcg_online_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + drmcg_apply_effective(DRMCG_TYPE_COMPUTE, minor->dev, drmcg); + + return 0; +} + +static int drmcg_css_online(struct cgroup_subsys_state *css) +{ + return drm_minor_for_each(&drmcg_online_fn, css_to_drmcg(css)); +} + struct cgroup_subsys gpu_cgrp_subsys = { .css_alloc = drmcg_css_alloc, .css_free = drmcg_css_free, + .css_online = drmcg_css_online, .early_init = false, .legacy_cftypes = files, .dfl_cftypes = files, @@ -585,6 +755,9 @@ void drmcg_device_early_init(struct drm_device *dev) dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; dev->drmcg_props.bo_limits_peak_allocated_default = S64_MAX; + dev->drmcg_props.compute_capacity = MAX_DRMCG_COMPUTE_CAPACITY; + bitmap_fill(dev->drmcg_props.compute_slots, MAX_DRMCG_COMPUTE_CAPACITY); + drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); From patchwork Wed Feb 26 19:01:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407077 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA4B7930 for ; Wed, 26 Feb 2020 19:02:38 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B805C2072D for ; Wed, 26 Feb 2020 19:02:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="TBwDjCja" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B805C2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 87D3C6EBC3; Wed, 26 Feb 2020 19:02:26 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2080.outbound.protection.outlook.com [40.107.220.80]) by gabe.freedesktop.org (Postfix) with ESMTPS id CD6CC6EBB1; Wed, 26 Feb 2020 19:02:12 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZmLyK4z1NlcwPr9WZ3B98vFZL6TbIKb82HmC0WFlcY76mDqBM4l+UH5kmmiIqZsow8PgGMGX+kUrB1ylLj22MpBq2lQwIF245lVCo8OFtz1iOKE7eUXJvg5kkrHv5iOB4u1k922j8s7RDXpEI74OpQZAyN53gSkikzwF9dlUGf2TRh2vA0yFUtbEolOag8fdeFCUow1mISJLtBYUZORyYX2ejKgUczCQwuDBtI6ZGg77RmqAvskCNqaohnK4cSlKMy85zFvX+sLv+KKwQmHtm34UAcVRDSq71yk6Fz2FHHisclueJj9+NyBBzoHKczUZBQgzMuZYdKL8yunf8rwV0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jpbplLFkHuHULlyRgLQceVq8WbBsfo1Ixm0Kgnaxgfw=; b=Q82QbTWW5f+nFfUqISbhrt5CwZYhAZhXo2PJypf0JqtKWtcLz3qB75dscsT7YEbV5B6QxEIe5eikdpax+hLZL98kVguoC/X5jilQ4TR3Dfo7quOdOYAZktaVnfrQC2TG6vsgZWSxQ3qjdmkV/6I6nN2wOhbg7926Ae68zBQMkcUWsvCKKyKvml9KhjkKhYYPKXPTyG0C3qwHv8K2aUFSi8ijYXRXATTJ4Sleu5kWOXtx9BH8iOfA7wcYyieg+kBYHarV7jyrGsnjbq7FrE4ShMq6uEQZ/bgsa4AMjOzJyvric7s1sh8BovCF7wBkRxE22rVIcHtFBaTMvmVBUqgolw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jpbplLFkHuHULlyRgLQceVq8WbBsfo1Ixm0Kgnaxgfw=; b=TBwDjCjaVT982ve+94KXrgKUqBUIQzOymIyCDzHnPK1DX0hGCuTzyWasF/UBhAb2mKX2kc/lLjWepdZqqotWjvpGdesf0hN3cx4q89dmVJRs6khcBBwo1HOlSIxjLd46eetq8rviU4L4dnZJswU7sGPD7iblZcimLP/4fIwYBXE= Received: from BN6PR1701CA0005.namprd17.prod.outlook.com (2603:10b6:405:15::15) by MN2PR12MB4111.namprd12.prod.outlook.com (2603:10b6:208:1de::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:11 +0000 Received: from BN8NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:405:15:cafe::4) by BN6PR1701CA0005.outlook.office365.com (2603:10b6:405:15::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:11 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT012.mail.protection.outlook.com (10.13.177.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:11 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:10 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:10 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:09 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 10/11] drm, cgroup: add update trigger after limit change Date: Wed, 26 Feb 2020 14:01:51 -0500 Message-ID: <20200226190152.16131-11-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(136003)(346002)(39860400002)(428003)(199004)(189003)(2906002)(316002)(478600001)(110136005)(1076003)(70206006)(336012)(426003)(70586007)(2616005)(15650500001)(6666004)(26005)(86362001)(81156014)(356004)(81166006)(4326008)(186003)(36756003)(5660300002)(8676002)(8936002)(7696005)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR12MB4111; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1a535615-d7c7-4425-802d-08d7baee626b X-MS-TrafficTypeDiagnostic: MN2PR12MB4111: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1443; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9DF4S9Sokm9GXdfAyEQ5Rv+Np8elsrpTXKHcc9fLPsEPlzd68XA7cvE92UoaMeYRtNzQrSmBvsqBAlHtHkSX8gyIUygaAtIrEHyrQUuP+9IM0f3K0u90ZNLSX8wuc2OGLOx8huU0xdQ6BnSK+ph3t0+gxwYAElRN/7pW5Aj2i7jQxZMmkgoHo/LyjkPwVn9H2pWkZ/of4gruxJy2H1cEHw5REuRjoGkhdCjUgfy4ovZVT2cQfSoQqq/YT0z85TIxF6kpEAQDdobyDbIp4N5Xb+Mx4mgzO+v4sQZfrv6ODwWl6lzpWycib5ODhzxjEb8670gpio+Gj+tbtosQnhU03hixKt8zAr6W6GGn3jdIKcGJbqFSmiILUMwH/31YIRGd8u83GgHIIt7SgzW+x4o/Md7w8cx2/UcFUR6BnBIAsuojdBe+8lobtLx5jzSXqPi1T15baMcQOjnsksO365iaaNDfmpZsfVhk+Skl6TQsINtuBJgI1GQZERyXsRUkNUhl14hntg/x5rTi0PpWr0Z5vEDbNK4t5UjkWvDcKwNrgnE= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:11.1979 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1a535615-d7c7-4425-802d-08d7baee626b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4111 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Before this commit, drmcg limits are updated but enforcement is delayed until the next time the driver check against the new limit. While this is sufficient for certain resources, a more proactive enforcement may be needed for other resources. Introducing an optional drmcg_limit_updated callback for the DRM drivers. When defined, it will be called in two scenarios: 1) When limits are updated for a particular cgroup, the callback will be triggered for each task in the updated cgroup. 2) When a task is migrated from one cgroup to another, the callback will be triggered for each resource type for the migrated task. Change-Id: I0ce7c4e5a04c31bd0f8d9853a383575d4bc9a3fa Signed-off-by: Kenny Ho --- include/drm/drm_drv.h | 10 ++++++++ kernel/cgroup/drm.c | 58 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index 1f65ac4d9bbf..e7333143e722 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -724,6 +724,16 @@ struct drm_driver { void (*drmcg_custom_init)(struct drm_device *dev, struct drmcg_props *props); + /** + * @drmcg_limit_updated + * + * Optional callback + */ + void (*drmcg_limit_updated)(struct drm_device *dev, + struct task_struct *task, + struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type); + /** * @gem_vm_ops: Driver private ops for this object * diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 2eadabebdfea..da439a351b07 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -127,6 +127,26 @@ static inline void drmcg_update_cg_tree(struct drm_device *dev) mutex_unlock(&cgroup_mutex); } +static void drmcg_limit_updated(struct drm_device *dev, struct drmcg *drmcg, + enum drmcg_res_type res_type) +{ + struct drmcg_device_resource *ddr = + drmcg->dev_resources[dev->primary->index]; + struct css_task_iter it; + struct task_struct *task; + + if (dev->driver->drmcg_limit_updated == NULL) + return; + + css_task_iter_start(&drmcg->css.cgroup->self, + CSS_TASK_ITER_PROCS, &it); + while ((task = css_task_iter_next(&it))) { + dev->driver->drmcg_limit_updated(dev, task, + ddr, res_type); + } + css_task_iter_end(&it); +} + static void drmcg_calculate_effective_compute(struct drm_device *dev, const unsigned long *free_weighted, struct drmcg *parent_drmcg) @@ -208,6 +228,8 @@ static void drmcg_apply_effective_compute(struct drm_device *dev) capacity); ddr->compute_count_eff = bitmap_weight(ddr->compute_eff, capacity); + + drmcg_limit_updated(dev, drmcg, DRMCG_TYPE_COMPUTE); } } rcu_read_unlock(); @@ -732,10 +754,46 @@ static int drmcg_css_online(struct cgroup_subsys_state *css) return drm_minor_for_each(&drmcg_online_fn, css_to_drmcg(css)); } +static int drmcg_attach_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct task_struct *task = data; + struct drm_device *dev; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + dev = minor->dev; + + if (dev->driver->drmcg_limit_updated) { + struct drmcg *drmcg = drmcg_get(task); + struct drmcg_device_resource *ddr = + drmcg->dev_resources[minor->index]; + enum drmcg_res_type type; + + for (type = 0; type < __DRMCG_TYPE_LAST; type++) + dev->driver->drmcg_limit_updated(dev, task, ddr, type); + + drmcg_put(drmcg); + } + + return 0; +} + +static void drmcg_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) + drm_minor_for_each(&drmcg_attach_fn, task); +} + struct cgroup_subsys gpu_cgrp_subsys = { .css_alloc = drmcg_css_alloc, .css_free = drmcg_css_free, .css_online = drmcg_css_online, + .attach = drmcg_attach, .early_init = false, .legacy_cftypes = files, .dfl_cftypes = files, From patchwork Wed Feb 26 19:01:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11407073 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C29714B4 for ; Wed, 26 Feb 2020 19:02:35 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5A2BE2072D for ; Wed, 26 Feb 2020 19:02:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="QWpKUaq0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A2BE2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 30DB96EBB0; Wed, 26 Feb 2020 19:02:26 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2054.outbound.protection.outlook.com [40.107.237.54]) by gabe.freedesktop.org (Postfix) with ESMTPS id BC1F46EBA9; Wed, 26 Feb 2020 19:02:16 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NI3U8w40SGD9RgDiESHGArmcdoSIOHDd6I3hyZZ6/fJWiAdqs1ymKlg/HDr20943phyv95gsJlvOvhUKwiVpcUkLZQ2xejCECilFY5YgvZ06N9Jv7bi968sMEl7yvgiMllFMZ8lzCyGw9NTZ8YUUWfo0y7vwhZOWwwoiUeODrGNoEHZvJvG7KjdtHjVYRRD/jeIUZ35BoyX0a6nIYiVPpCy1hsR3bUK0NFe6yN/kvnzoQck/7Yb0C7gD21+2k1pvhUZzkJ3eA3LBlR4q+Lj0dytBKqA7MuE8TBf7EAbVmhzSvkApas0qpFfhPtfQUbQEvToh10Ho/HyxdBtgMR/Uxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W9U7NEPPNIxK9vLDx9ztf8Nw8JMRkapZr2Z7bfi02Ns=; b=Mb6HtP9HJ6rsO22+xTCWzbRMwMNzCU1IDYzJBfzNdQPdcUDR1CxdBsgXZYjDl1iVcFcKlnF9wPpWzGJ1lAN3zDwvfK7jGfcdLgyngSN0e9Qtd7bgXGRdlzHGgPD2xggc0AIJHSTiePwbWRDvh9HqeYMgGwYWHreIpGxLLHk3ZEPSztyKApB+7gjY/q8wj8hOrv4GXzyOeic3NipVx/aLMIuAFP1KcLBESvl1lv5dusMM9L4KsBHftZGstlZaISFj7BRcm4qrNQ/HuJTbbdBJXGeiJqkfrBnjr2R3OJrQ57qluwYS03qas1PDgHGLwP9YuVJ2p3nfc8kfnCmIwHpiaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W9U7NEPPNIxK9vLDx9ztf8Nw8JMRkapZr2Z7bfi02Ns=; b=QWpKUaq0iv9VYjHDJgQjF5MMYNX88XViZp6gsTR7bikY+lpGf1oQHb4KvO5/6T3WkWcF++WjX+n7tPDGZ0gcgncQjYCh52vYlpCClIlmlNODmt+UuL2WvvRult93zN2nKtLT3HNEQyFS+GPPaVWECfih6SDoH7vjd2E9yDTRDaQ= Received: from DM5PR04CA0062.namprd04.prod.outlook.com (2603:10b6:3:ef::24) by DM6PR12MB2988.namprd12.prod.outlook.com (2603:10b6:5:3d::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.21; Wed, 26 Feb 2020 19:02:12 +0000 Received: from DM6NAM11FT059.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ef:cafe::d) by DM5PR04CA0062.outlook.office365.com (2603:10b6:3:ef::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2750.18 via Frontend Transport; Wed, 26 Feb 2020 19:02:12 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by DM6NAM11FT059.mail.protection.outlook.com (10.13.172.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2772.14 via Frontend Transport; Wed, 26 Feb 2020 19:02:12 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:11 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 26 Feb 2020 13:02:11 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Wed, 26 Feb 2020 13:02:10 -0600 From: Kenny Ho To: , , , , , , , , , Subject: [PATCH v2 11/11] drm/amdgpu: Integrate with DRM cgroup Date: Wed, 26 Feb 2020 14:01:52 -0500 Message-ID: <20200226190152.16131-12-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200226190152.16131-1-Kenny.Ho@amd.com> References: <20200226190152.16131-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(136003)(396003)(376002)(428003)(189003)(199004)(4326008)(478600001)(26005)(86362001)(2906002)(316002)(5660300002)(110136005)(7696005)(336012)(1076003)(8676002)(186003)(36756003)(426003)(2616005)(356004)(81166006)(70586007)(6666004)(81156014)(70206006)(8936002)(921003)(1121003)(83996005)(2101003); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB2988; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 68aeacbe-ae5b-4aa9-82c1-08d7baee62fd X-MS-TrafficTypeDiagnostic: DM6PR12MB2988: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-Forefront-PRVS: 0325F6C77B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rDYwxrqMvkijOuqsl88Ag0EWbKY0O+6qsHJ/RKjoXSmF+q5sOvPwyI4tCdfllUZ1ObXCQYPBKGiDtHfd/u9R3yWPURrEHENgxef25zlk25Sfw/5FdTNHpLIlAwL7WJQtupb4BXzmVQtaQaEuvoPLMhjOOTu6J6ecHT+udVLv8lnboeXDS9KQcjEwccdQKsvok2uXWzaR11G9TpE0/rXRlVDzHLhU/GapS4I7CvXCoNrtGvk0/+pVoA31RfeNKhf5Nb/dECW5ZOnbGehUbcrgfZwcagCAOFAvq1Lwpi/886JAnOI8f7VMyTpVK6iunbI4YPUBsEGDb63BR4oHr/Zl8Y128zW7SbMO7StsiSWN6tzQr+WvvatFvp8LRMKHCXvKeEhf0+m1tRegH4E13M5BN+jBj/txZqkecQ4tTvYl8EUuzQDmaI+qIeO9CVY1UWxi7kaZiT/vH1DlLzHYHpbf/uYG8jWGn506lC5FD+tq1w3EcxyLFf+ns/r2jRbaNFg9jDu13TcPIUypI6JGccQfjf50Ko9CUVAxpnshyy5AkR4= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2020 19:02:12.0759 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 68aeacbe-ae5b-4aa9-82c1-08d7baee62fd X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2988 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The number of compute unit (CU) for a device is used for the gpu cgroup compute capacity. The gpu cgroup compute allocation limit only applies to compute workload for the moment (enforced via kfd queue creation.) Any cu_mask update is validated against the availability of the compute unit as defined by the drmcg the kfd process belongs to. Change-Id: I2930e76ef9ac6d36d0feb81f604c89a4208e6614 Signed-off-by: Kenny Ho --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 29 ++++ drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 7 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 + .../amd/amdkfd/kfd_process_queue_manager.c | 153 ++++++++++++++++++ 5 files changed, 196 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h index 0ee8aae6c519..1efbc0d3c03e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -199,6 +199,10 @@ uint8_t amdgpu_amdkfd_get_xgmi_hops_count(struct kgd_dev *dst, struct kgd_dev *s valid; \ }) +int amdgpu_amdkfd_update_cu_mask_for_process(struct task_struct *task, + struct amdgpu_device *adev, unsigned long *compute_bm, + unsigned int compute_bm_size); + /* GPUVM API */ int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, unsigned int pasid, void **vm, void **process_info, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 171397708855..595ad852080b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -1418,9 +1418,31 @@ amdgpu_get_crtc_scanout_position(struct drm_device *dev, unsigned int pipe, static void amdgpu_drmcg_custom_init(struct drm_device *dev, struct drmcg_props *props) { + struct amdgpu_device *adev = dev->dev_private; + + props->compute_capacity = adev->gfx.cu_info.number; + bitmap_zero(props->compute_slots, MAX_DRMCG_COMPUTE_CAPACITY); + bitmap_fill(props->compute_slots, props->compute_capacity); + props->limit_enforced = true; } +static void amdgpu_drmcg_limit_updated(struct drm_device *dev, + struct task_struct *task, struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type) +{ + struct amdgpu_device *adev = dev->dev_private; + + switch (res_type) { + case DRMCG_TYPE_COMPUTE: + amdgpu_amdkfd_update_cu_mask_for_process(task, adev, + ddr->compute_eff, dev->drmcg_props.compute_capacity); + break; + default: + break; + } +} + #else static void amdgpu_drmcg_custom_init(struct drm_device *dev, @@ -1428,6 +1450,12 @@ static void amdgpu_drmcg_custom_init(struct drm_device *dev, { } +static void amdgpu_drmcg_limit_updated(struct drm_device *dev, + struct task_struct *task, struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type) +{ +} + #endif /* CONFIG_CGROUP_DRM */ static struct drm_driver kms_driver = { @@ -1462,6 +1490,7 @@ static struct drm_driver kms_driver = { .gem_prime_mmap = amdgpu_gem_prime_mmap, .drmcg_custom_init = amdgpu_drmcg_custom_init, + .drmcg_limit_updated = amdgpu_drmcg_limit_updated, .name = DRIVER_NAME, .desc = DRIVER_DESC, diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 675735b8243a..a35596f2dc4e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -451,6 +451,13 @@ static int kfd_ioctl_set_cu_mask(struct file *filp, struct kfd_process *p, return -EFAULT; } + if (!pqm_drmcg_compute_validate(p, args->queue_id, + properties.cu_mask, cu_mask_size)) { + pr_debug("CU mask not permitted by DRM Cgroup"); + kfree(properties.cu_mask); + return -EACCES; + } + mutex_lock(&p->mutex); retval = pqm_set_cu_mask(&p->pqm, args->queue_id, &properties); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 063096ec832d..0fb619586e24 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -929,6 +929,9 @@ int pqm_get_wave_state(struct process_queue_manager *pqm, u32 *ctl_stack_used_size, u32 *save_area_used_size); +bool pqm_drmcg_compute_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size); + int amdkfd_fence_wait_timeout(unsigned int *fence_addr, unsigned int fence_value, unsigned int timeout_ms); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index cb1ca11b99c3..bd09403e07b5 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -23,9 +23,11 @@ #include #include +#include #include "kfd_device_queue_manager.h" #include "kfd_priv.h" #include "kfd_kernel_queue.h" +#include "amdgpu.h" #include "amdgpu_amdkfd.h" static inline struct process_queue_node *get_queue_by_qid( @@ -167,6 +169,7 @@ static int init_user_queue(struct process_queue_manager *pqm, struct queue_properties *q_properties, struct file *f, unsigned int qid) { + struct drmcg *drmcg; int retval; /* Doorbell initialized in user space*/ @@ -180,6 +183,37 @@ static int init_user_queue(struct process_queue_manager *pqm, if (retval != 0) return retval; +#ifdef CONFIG_CGROUP_DRM + drmcg = drmcg_get(pqm->process->lead_thread); + if (drmcg) { + struct amdgpu_device *adev; + struct drmcg_device_resource *ddr; + int mask_size; + u32 *mask; + + adev = (struct amdgpu_device *) dev->kgd; + + mask_size = adev->ddev->drmcg_props.compute_capacity; + mask = kzalloc(sizeof(u32) * round_up(mask_size, 32), + GFP_KERNEL); + + if (!mask) { + drmcg_put(drmcg); + uninit_queue(*q); + return -ENOMEM; + } + + ddr = drmcg->dev_resources[adev->ddev->primary->index]; + + bitmap_to_arr32(mask, ddr->compute_eff, mask_size); + + (*q)->properties.cu_mask_count = mask_size; + (*q)->properties.cu_mask = mask; + + drmcg_put(drmcg); + } +#endif /* CONFIG_CGROUP_DRM */ + (*q)->device = dev; (*q)->process = pqm->process; @@ -510,6 +544,125 @@ int pqm_get_wave_state(struct process_queue_manager *pqm, save_area_used_size); } +#ifdef CONFIG_CGROUP_DRM + +bool pqm_drmcg_compute_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size) +{ + DECLARE_BITMAP(curr_mask, MAX_DRMCG_COMPUTE_CAPACITY); + struct drmcg_device_resource *ddr; + struct process_queue_node *pqn; + struct amdgpu_device *adev; + struct drmcg *drmcg; + bool result; + + if (cu_mask_size > MAX_DRMCG_COMPUTE_CAPACITY) + return false; + + bitmap_from_arr32(curr_mask, cu_mask, cu_mask_size); + + pqn = get_queue_by_qid(&p->pqm, qid); + if (!pqn) + return false; + + adev = (struct amdgpu_device *)pqn->q->device->kgd; + + drmcg = drmcg_get(p->lead_thread); + ddr = drmcg->dev_resources[adev->ddev->primary->index]; + + if (bitmap_subset(curr_mask, ddr->compute_eff, + MAX_DRMCG_COMPUTE_CAPACITY)) + result = true; + else + result = false; + + drmcg_put(drmcg); + + return result; +} + +#else + +bool pqm_drmcg_compute_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size) +{ + return true; +} + +#endif /* CONFIG_CGROUP_DRM */ + +int amdgpu_amdkfd_update_cu_mask_for_process(struct task_struct *task, + struct amdgpu_device *adev, unsigned long *compute_bm, + unsigned int compute_bm_size) +{ + struct kfd_dev *kdev = adev->kfd.dev; + struct process_queue_node *pqn; + struct kfd_process *kfdproc; + size_t size_in_bytes; + u32 *cu_mask; + int rc = 0; + + if ((compute_bm_size % 32) != 0) { + pr_warn("compute_bm_size %d must be a multiple of 32", + compute_bm_size); + return -EINVAL; + } + + kfdproc = kfd_get_process(task); + + if (IS_ERR(kfdproc)) + return -ESRCH; + + size_in_bytes = sizeof(u32) * round_up(compute_bm_size, 32); + + mutex_lock(&kfdproc->mutex); + list_for_each_entry(pqn, &kfdproc->pqm.queues, process_queue_list) { + if (pqn->q && pqn->q->device == kdev) { + /* update cu_mask accordingly */ + cu_mask = kzalloc(size_in_bytes, GFP_KERNEL); + if (!cu_mask) { + rc = -ENOMEM; + break; + } + + if (pqn->q->properties.cu_mask) { + DECLARE_BITMAP(curr_mask, + MAX_DRMCG_COMPUTE_CAPACITY); + + if (pqn->q->properties.cu_mask_count > + compute_bm_size) { + rc = -EINVAL; + kfree(cu_mask); + break; + } + + bitmap_from_arr32(curr_mask, + pqn->q->properties.cu_mask, + pqn->q->properties.cu_mask_count); + + bitmap_and(curr_mask, curr_mask, compute_bm, + compute_bm_size); + + bitmap_to_arr32(cu_mask, curr_mask, + compute_bm_size); + + kfree(curr_mask); + } else + bitmap_to_arr32(cu_mask, compute_bm, + compute_bm_size); + + pqn->q->properties.cu_mask = cu_mask; + pqn->q->properties.cu_mask_count = compute_bm_size; + + rc = pqn->q->device->dqm->ops.update_queue( + pqn->q->device->dqm, pqn->q); + } + } + mutex_unlock(&kfdproc->mutex); + + return rc; +} + #if defined(CONFIG_DEBUG_FS) int pqm_debugfs_mqds(struct seq_file *m, void *data)