From patchwork Fri Feb 14 15:56:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382591 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12D8F1820 for ; Fri, 14 Feb 2020 15:57:16 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E51F12086A for ; Fri, 14 Feb 2020 15:57:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="EB1T3tPf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E51F12086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3A06F6F9E8; Fri, 14 Feb 2020 15:57:14 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-eopbgr680058.outbound.protection.outlook.com [40.107.68.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0F7176F9E6; Fri, 14 Feb 2020 15:57:13 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I6B86umGFIGimbvRam7c9HTA22+ad0bbwk7/UrQxe1E5sdaV5upRfrQWDKyq3a3ELqHHUGrA6vy6L1X0mphzG1X5AdDGBvIML8ZEjnEB26N6En9Fb2wOFdIJdyMgeQx9PiIuLJe7lZ54vXTNamIdaSdiG92uoaevPURI577OCWvbzz1VnmYk/ntZakTSHrkIECGmCs4juVtVTjFtAC7fsu/RfRJnrXTDFjMOj6zpa7N4BLAt3MgzuFqbXyF0CfnnWBtmrLHQJLc+RpdmdXNFylK7GLtnmCrxLYlADFUwmPKLNocmakIC2TaHZF/NVWJwWDtlwrzB9x7JmHgJC1hYcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IE8lLucDYfCrdS9utv/BkKHREyx2rBrFyW9gbmx+UyI=; b=PTW0h5KMffo8FiLxLhkiJkGPCjxdQVI1xylN3nkYfITf09JgmNpwhO6zSzhZpggYwx93ypEnIbDLdXiJcq5Jvqa8akL6/7LlQLb7Tl+JhS3OX/h8y7K6Asm/1soBD8ifOL+CR/L9KDz/wDqowuRXFb2q4/9uvuaOtNAMcpf1fJLJk0A+hUessLNmCXV1T36h8sLc4L9BcWmKMP7rIvKTDHLXlZnsUVsL9RlLBA/wUYUzqkRc91WiAyKaHfeO6SO1MBB/uLl6stu0+awPMujZdyw+f7HlHeKDPQJ3fcu8SfNWv7RX4LzjEeOJisT74ouPgfjWTDg7bNdrCOfkRlAJwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IE8lLucDYfCrdS9utv/BkKHREyx2rBrFyW9gbmx+UyI=; b=EB1T3tPfKYaLCKOYfeK0VTXNJx+lzUCXoJVwUOVPrpbG7K6oUN0eaXUYGqd642r0Eht8eof6ozsl+BuU8X8wbllSuc2CMWaSPVi0XUPfRqI03j0wC1Kx7yQa5mCzKT8JWuLU/psjojDoNivFBa7x+CQ3ZAF6JXzJOXzUjgei2DE= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by MW2PR12MB2428.namprd12.prod.outlook.com (2603:10b6:907:6::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.23; Fri, 14 Feb 2020 15:57:08 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:08 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:08 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:08 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:07 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 01/11] cgroup: Introduce cgroup for drm subsystem Date: Fri, 14 Feb 2020 10:56:40 -0500 Message-ID: <20200214155650.21203-2-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(136003)(376002)(396003)(346002)(428003)(189003)(199004)(186003)(86362001)(5660300002)(7696005)(26005)(8936002)(2616005)(81156014)(110136005)(426003)(356004)(6666004)(8676002)(81166006)(4326008)(36756003)(70586007)(316002)(70206006)(478600001)(336012)(6636002)(1076003)(2906002)(921003)(83996005)(2101003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:MW2PR12MB2428; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5c56c5ed-2b38-4082-1b3e-08d7b1668bca X-MS-TrafficTypeDiagnostic: MW2PR12MB2428: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 687Vh3NMemC7b1hvhqCp6VqKTORvVH9bwj2sIUvrHdZe8R52BwqJH7GfNrRQ/px6l5md7FK01HzRLpls81kKpImwNDRTEnYcloNbRrZD5XzXXBANEVTicsvc1jWx+sxinzo6L5Rlmhr29sDxEbYQqabr6c4pdAHalskMs0dr1ckBcJ4KreZw+1fhm/9tOPX2uVI0mfclHGhL3LtIRDwj+OCjsUFAihiOo9ZApHtc63PxrYXU44ZIefg1mRVUjS7DORNo6XM/z0ieEHk0gxHAs+iR26BXBS6CX4ebxT9XR7/re0H5hf+nACXmTzVxlPu7FvYuPveT4OI3uQ2rkcs+Xpn1Z2ZMz/O361Lr6/FNmN1krucB/+QJ1ddcxPWzVAH+hO9jyLN4EBITcPHmdUZ8bwZ3lHuZYx9jS05ZqKj5K0/PuXmceR4eEJEpo76KsMbkJ3u9YJ3JZ5dfAtJovdDgWqZq2UqIKWJ9pP6yQgSViPc7DdhKyo7FzkEldBB1DDTsZJAEliL2IBvjNrv+yugLKuniIa5911k//m+BuJ/gnFY= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:08.5843 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5c56c5ed-2b38-4082-1b3e-08d7b1668bca X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2428 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" With the increased importance of machine learning, data science and other cloud-based applications, GPUs are already in production use in data centers today. Existing GPU resource management is very coarse grain, however, as sysadmins are only able to distribute workload on a per-GPU basis. An alternative is to use GPU virtualization (with or without SRIOV) but it generally acts on the entire GPU instead of the specific resources in a GPU. With a drm cgroup controller, we can enable alternate, fine-grain, sub-GPU resource management (in addition to what may be available via GPU virtualization.) Change-Id: Ia90aed8c4cb89ff20d8216a903a765655b44fc9a Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 18 ++++- Documentation/cgroup-v1/drm.rst | 1 + include/linux/cgroup_drm.h | 92 +++++++++++++++++++++++++ include/linux/cgroup_subsys.h | 4 ++ init/Kconfig | 5 ++ kernel/cgroup/Makefile | 1 + kernel/cgroup/drm.c | 42 +++++++++++ 7 files changed, 161 insertions(+), 2 deletions(-) create mode 100644 Documentation/cgroup-v1/drm.rst create mode 100644 include/linux/cgroup_drm.h create mode 100644 kernel/cgroup/drm.c diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 5361ebec3361..384db8df0f30 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -61,8 +61,10 @@ v1 is available under Documentation/admin-guide/cgroup-v1/. 5-6. Device 5-7. RDMA 5-7-1. RDMA Interface Files - 5-8. Misc - 5-8-1. perf_event + 5-8. DRM + 5-8-1. DRM Interface Files + 5-9. Misc + 5-9-1. perf_event 5-N. Non-normative information 5-N-1. CPU controller root cgroup process behaviour 5-N-2. IO controller root cgroup process behaviour @@ -2051,6 +2053,18 @@ RDMA Interface Files ocrdma1 hca_handle=1 hca_object=23 +DRM +--- + +The "drm" controller regulates the distribution and accounting of +of DRM (Direct Rendering Manager) and GPU-related resources. + +DRM Interface Files +~~~~~~~~~~~~~~~~~~~~ + +TODO + + Misc ---- diff --git a/Documentation/cgroup-v1/drm.rst b/Documentation/cgroup-v1/drm.rst new file mode 100644 index 000000000000..5f5658e1f5ed --- /dev/null +++ b/Documentation/cgroup-v1/drm.rst @@ -0,0 +1 @@ +Please see ../cgroup-v2.rst for details diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h new file mode 100644 index 000000000000..ba7981ac3afc --- /dev/null +++ b/include/linux/cgroup_drm.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: MIT + * Copyright 2019 Advanced Micro Devices, Inc. + */ +#ifndef _CGROUP_DRM_H +#define _CGROUP_DRM_H + +#include + +#ifdef CONFIG_CGROUP_DRM + +/** + * The DRM cgroup controller data structure. + */ +struct drmcg { + struct cgroup_subsys_state css; +}; + +/** + * css_to_drmcg - get the corresponding drmcg ref from a cgroup_subsys_state + * @css: the target cgroup_subsys_state + * + * Return: DRM cgroup that contains the @css + */ +static inline struct drmcg *css_to_drmcg(struct cgroup_subsys_state *css) +{ + return css ? container_of(css, struct drmcg, css) : NULL; +} + +/** + * drmcg_get - get the drmcg reference that a task belongs to + * @task: the target task + * + * This increase the reference count of the css that the @task belongs to + * + * Return: reference to the DRM cgroup the task belongs to + */ +static inline struct drmcg *drmcg_get(struct task_struct *task) +{ + return css_to_drmcg(task_get_css(task, drm_cgrp_id)); +} + +/** + * drmcg_put - put a drmcg reference + * @drmcg: the target drmcg + * + * Put a reference obtained via drmcg_get + */ +static inline void drmcg_put(struct drmcg *drmcg) +{ + if (drmcg) + css_put(&drmcg->css); +} + +/** + * drmcg_parent - find the parent of a drm cgroup + * @cg: the target drmcg + * + * This does not increase the reference count of the parent cgroup + * + * Return: parent DRM cgroup of @cg + */ +static inline struct drmcg *drmcg_parent(struct drmcg *cg) +{ + return css_to_drmcg(cg->css.parent); +} + +#else /* CONFIG_CGROUP_DRM */ + +struct drmcg { +}; + +static inline struct drmcg *css_to_drmcg(struct cgroup_subsys_state *css) +{ + return NULL; +} + +static inline struct drmcg *drmcg_get(struct task_struct *task) +{ + return NULL; +} + +static inline void drmcg_put(struct drmcg *drmcg) +{ +} + +static inline struct drmcg *drmcg_parent(struct drmcg *cg) +{ + return NULL; +} + +#endif /* CONFIG_CGROUP_DRM */ +#endif /* _CGROUP_DRM_H */ diff --git a/include/linux/cgroup_subsys.h b/include/linux/cgroup_subsys.h index acb77dcff3b4..ddedad809e8b 100644 --- a/include/linux/cgroup_subsys.h +++ b/include/linux/cgroup_subsys.h @@ -61,6 +61,10 @@ SUBSYS(pids) SUBSYS(rdma) #endif +#if IS_ENABLED(CONFIG_CGROUP_DRM) +SUBSYS(drm) +#endif + /* * The following subsystems are not supported on the default hierarchy. */ diff --git a/init/Kconfig b/init/Kconfig index b4daad2bac23..8ea6bcfbe5e9 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -981,6 +981,11 @@ config CGROUP_RDMA Attaching processes with active RDMA resources to the cgroup hierarchy is allowed even if can cross the hierarchy's limit. +config CGROUP_DRM + bool "DRM controller (EXPERIMENTAL)" + help + Provides accounting and enforcement of resources in the DRM subsystem. + config CGROUP_FREEZER bool "Freezer controller" help diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile index 5d7a76bfbbb7..31f186f58121 100644 --- a/kernel/cgroup/Makefile +++ b/kernel/cgroup/Makefile @@ -4,5 +4,6 @@ obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) += pids.o obj-$(CONFIG_CGROUP_RDMA) += rdma.o +obj-$(CONFIG_CGROUP_DRM) += drm.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_CGROUP_DEBUG) += debug.o diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c new file mode 100644 index 000000000000..e97861b3cb30 --- /dev/null +++ b/kernel/cgroup/drm.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: MIT +// Copyright 2019 Advanced Micro Devices, Inc. +#include +#include +#include + +static struct drmcg *root_drmcg __read_mostly; + +static void drmcg_css_free(struct cgroup_subsys_state *css) +{ + struct drmcg *drmcg = css_to_drmcg(css); + + kfree(drmcg); +} + +static struct cgroup_subsys_state * +drmcg_css_alloc(struct cgroup_subsys_state *parent_css) +{ + struct drmcg *parent = css_to_drmcg(parent_css); + struct drmcg *drmcg; + + drmcg = kzalloc(sizeof(struct drmcg), GFP_KERNEL); + if (!drmcg) + return ERR_PTR(-ENOMEM); + + if (!parent) + root_drmcg = drmcg; + + return &drmcg->css; +} + +struct cftype files[] = { + { } /* terminate */ +}; + +struct cgroup_subsys drm_cgrp_subsys = { + .css_alloc = drmcg_css_alloc, + .css_free = drmcg_css_free, + .early_init = false, + .legacy_cftypes = files, + .dfl_cftypes = files, +}; From patchwork Fri Feb 14 15:56:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B45521820 for ; Fri, 14 Feb 2020 15:57:31 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 91D9F222C4 for ; Fri, 14 Feb 2020 15:57:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="HIbR4fNS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91D9F222C4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7D1966F9F5; Fri, 14 Feb 2020 15:57:20 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20620.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::620]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2378A6F9EC; Fri, 14 Feb 2020 15:57:15 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oH1rJhaEscMJGyCuHuyd4qwtgWi0MBXx+WpjZ2CI5Vw6XKMxFbJGqSJe4yrxTsGA++tgFf7whbQeHshLDXfncT9IeEobdvRAMigiEbBs23fSJgXIRSjYQOLV3dF4qiz1nioarF164KQLXTIdhzKOH5FtJ5fSqlNuIbH0muSLfUQyR5LK72RdVyfdi/fvTu8Ftt9RjO1bjUWU7zD/I+VMgfIwqKLtDt0JZkpyUBrRK1w5Jmiul1nkFrqHhxaRCDaJVU131DgNwdV9u4QigZAf/b4Tr/cQcmvRXggCWopl55+ToBt7TcHhhKHDP9fZURvyTT16TOoaS6w/uiH9gsD9Ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DltWDmUoat4C6/gr1wh/v/rSEdrfM7FwlqNY9LzYxuU=; b=awvJ5TGcrm+jc+bYOw9NYXt7ZrPtIjKzDSg5chs0pDLAHgUSsnzKCaDi0GJ+yCG50sUK2lXsifwi1nFNSSitEaH4oJ+55DZSfe7ihLubGY2AeplB6wy4Ru5T+zRj83rK5sduNJ71qbA+aKNzjwBURvMtMuQJI0MqeDsNoQGob/6+BakuOGMwNvMnVCfC2xv/kkc+C3ttqqeZTfToiAh1qCdGSGStfqEKuKV/hWDd7ax7qTgqa2jcYdg7dz/tdvuSVASi3rupPXRLkwd5AQ+aykFeLyedvm3cwJDCj5QK59AtaD6iDoHeMOHczqHYjfqlO9wFwP8utFh88+ti48IgEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DltWDmUoat4C6/gr1wh/v/rSEdrfM7FwlqNY9LzYxuU=; b=HIbR4fNS0VXt3PLHnFqh3JyfBN6gmCzzE9pnqfUuef1mEffYKkLvg9hajzpKdvjdplEd4kVXBndOharzZu1gBn2TeRNZqzpslj1HBQ0lq3e8Lowy6nIwmtv56vE0Bvx1i5gjV1kBhSADRYtFcoqws6gKcI3dTSWm2pELM2dXCbw= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by BY5PR12MB3699.namprd12.prod.outlook.com (2603:10b6:a03:195::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:11 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:11 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:10 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:09 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:08 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 02/11] drm, cgroup: Bind drm and cgroup subsystem Date: Fri, 14 Feb 2020 10:56:41 -0500 Message-ID: <20200214155650.21203-3-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(136003)(376002)(346002)(396003)(428003)(189003)(199004)(336012)(26005)(2616005)(186003)(426003)(86362001)(316002)(7696005)(110136005)(1076003)(356004)(6666004)(70206006)(70586007)(2906002)(8936002)(5660300002)(81156014)(8676002)(4326008)(478600001)(81166006)(36756003)(6636002)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:BY5PR12MB3699; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d500ff1c-ba2f-4568-7f55-08d7b1668d39 X-MS-TrafficTypeDiagnostic: BY5PR12MB3699: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2449; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oJpObbpCu8/Oo7EjQdw2TAH0q1Olek4LH8eXV/LZVppr3V6gB1tpesgj1kIeBZqpDq3DjJ80MXIMHmXcpELdyj+PkUQD6s8AJlxtJx7NBtnxpzK/naDZXpmcs0jFJ8a9Mj+npifRvTzOGwGhe4ypn/6IrAbMy3Kx1Lqj8NKpA3i1brJraWJFR/Dd3H+3ALN/v7Xo5dRaRSv5SljaSYa78Q4t91vz2AktOjnAYKoycrE4qgN+NQOaqzJnDyIkxtWsLlGPyA6i2USwFuhVXpfJm2pdXjHPM//CMygEqt6aBujGwe85OGk9aCpTOnydtp7gkFOCCPaPoyA91bCyt0rPbofbb+hOrsjvEQZX2VakKQFv+V00owaltAyJc7aF+yBORjSVQHXzbFEXt5qr/6mQ2ztuddJuMa+nA+JUVAF3ATI0i3V0J1usasvY202hawDNypVu+N8Qbv4PXh6oTxdajSYztnUERLHV2U+A5z83RaMbNeKMdboa3BKO9EX0pY/Tw0v9jN1T7Syy/DTAxDjEbLPa4TDW2ugfWl6j08B46BY= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:10.9920 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d500ff1c-ba2f-4568-7f55-08d7b1668d39 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3699 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since the drm subsystem can be compiled as a module and drm devices can be added and removed during run time, add several functions to bind the drm subsystem as well as drm devices with drmcg. Two pairs of functions: drmcg_bind/drmcg_unbind - used to bind/unbind the drm subsystem to the cgroup subsystem as the drm core initialize/exit. drmcg_register_dev/drmcg_unregister_dev - used to register/unregister drm devices to the cgroup subsystem as the devices are presented/removed from userspace. Change-Id: I1cb6b2080fc7d27979d886ef23e784341efafb41 --- drivers/gpu/drm/drm_drv.c | 8 +++ include/drm/drm_cgroup.h | 39 +++++++++++ include/linux/cgroup_drm.h | 4 ++ kernel/cgroup/drm.c | 131 +++++++++++++++++++++++++++++++++++++ 4 files changed, 182 insertions(+) create mode 100644 include/drm/drm_cgroup.h diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 1b9b40a1c7c9..8e59cc5a5bde 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "drm_crtc_internal.h" #include "drm_internal.h" @@ -972,6 +973,8 @@ int drm_dev_register(struct drm_device *dev, unsigned long flags) ret = 0; + drmcg_register_dev(dev); + DRM_INFO("Initialized %s %d.%d.%d %s for %s on minor %d\n", driver->name, driver->major, driver->minor, driver->patchlevel, driver->date, @@ -1006,6 +1009,8 @@ EXPORT_SYMBOL(drm_dev_register); */ void drm_dev_unregister(struct drm_device *dev) { + drmcg_unregister_dev(dev); + if (drm_core_check_feature(dev, DRIVER_LEGACY)) drm_lastclose(dev); @@ -1112,6 +1117,7 @@ static const struct file_operations drm_stub_fops = { static void drm_core_exit(void) { + drmcg_unbind(); unregister_chrdev(DRM_MAJOR, "drm"); debugfs_remove(drm_debugfs_root); drm_sysfs_destroy(); @@ -1138,6 +1144,8 @@ static int __init drm_core_init(void) if (ret < 0) goto error; + drmcg_bind(&drm_minor_acquire, &drm_dev_put); + drm_core_init_complete = true; DRM_DEBUG("Initialized\n"); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h new file mode 100644 index 000000000000..530c9a0b3238 --- /dev/null +++ b/include/drm/drm_cgroup.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: MIT + * Copyright 2019 Advanced Micro Devices, Inc. + */ +#ifndef __DRM_CGROUP_H__ +#define __DRM_CGROUP_H__ + +#ifdef CONFIG_CGROUP_DRM + +void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)); + +void drmcg_unbind(void); + +void drmcg_register_dev(struct drm_device *dev); + +void drmcg_unregister_dev(struct drm_device *dev); + +#else + +static inline void drmcg_bind( + struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)) +{ +} + +static inline void drmcg_unbind(void) +{ +} + +static inline void drmcg_register_dev(struct drm_device *dev) +{ +} + +static inline void drmcg_unregister_dev(struct drm_device *dev) +{ +} + +#endif /* CONFIG_CGROUP_DRM */ +#endif /* __DRM_CGROUP_H__ */ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index ba7981ac3afc..854591bbb430 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -5,6 +5,10 @@ #define _CGROUP_DRM_H #include +#include + +/* limit defined per the way drm_minor_alloc operates */ +#define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) #ifdef CONFIG_CGROUP_DRM diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index e97861b3cb30..37f98dc47268 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -1,11 +1,142 @@ // SPDX-License-Identifier: MIT // Copyright 2019 Advanced Micro Devices, Inc. +#include +#include #include #include #include +#include +#include +#include static struct drmcg *root_drmcg __read_mostly; +/* global mutex for drmcg across all devices */ +static DEFINE_MUTEX(drmcg_mutex); + +static DECLARE_BITMAP(known_devs, MAX_DRM_DEV); + +static struct drm_minor (*(*acquire_drm_minor)(unsigned int minor_id)); + +static void (*put_drm_dev)(struct drm_device *dev); + +/** + * drmcg_bind - Bind DRM subsystem to cgroup subsystem + * @acq_dm: function pointer to the drm_minor_acquire function + * @put_ddev: function pointer to the drm_dev_put function + * + * This function binds some functions from the DRM subsystem and make + * them available to the drmcg subsystem. + * + * drmcg_unbind does the opposite of this function + */ +void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), + void (*put_ddev)(struct drm_device *dev)) +{ + mutex_lock(&drmcg_mutex); + acquire_drm_minor = acq_dm; + put_drm_dev = put_ddev; + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_bind); + +/** + * drmcg_unbind - Unbind DRM subsystem from cgroup subsystem + * + * drmcg_bind does the opposite of this function + */ +void drmcg_unbind(void) +{ + mutex_lock(&drmcg_mutex); + acquire_drm_minor = NULL; + put_drm_dev = NULL; + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unbind); + +/** + * drmcg_register_dev - register a DRM device for usage in drm cgroup + * @dev: DRM device + * + * This function make a DRM device visible to the cgroup subsystem. + * Once the drmcg is aware of the device, drmcg can start tracking and + * control resource usage for said device. + * + * drmcg_unregister_dev reverse the operation of this function + */ +void drmcg_register_dev(struct drm_device *dev) +{ + if (WARN_ON(dev->primary->index >= MAX_DRM_DEV)) + return; + + mutex_lock(&drmcg_mutex); + set_bit(dev->primary->index, known_devs); + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_register_dev); + +/** + * drmcg_unregister_dev - Iterate through all stored DRM minors + * @dev: DRM device + * + * Unregister @dev so that drmcg no longer control resource usage + * of @dev. The @dev was registered to drmcg using + * drmcg_register_dev function + */ +void drmcg_unregister_dev(struct drm_device *dev) +{ + if (WARN_ON(dev->primary->index >= MAX_DRM_DEV)) + return; + + mutex_lock(&drmcg_mutex); + clear_bit(dev->primary->index, known_devs); + mutex_unlock(&drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unregister_dev); + +/** + * drm_minor_for_each - Iterate through all stored DRM minors + * @fn: Function to be called for each pointer. + * @data: Data passed to callback function. + * + * The callback function will be called for each registered device, passing + * the minor, the @drm_minor entry and @data. + * + * If @fn returns anything other than %0, the iteration stops and that + * value is returned from this function. + */ +static int drm_minor_for_each(int (*fn)(int id, void *p, void *data), + void *data) +{ + int rc = 0; + + mutex_lock(&drmcg_mutex); + if (acquire_drm_minor) { + unsigned int minor; + struct drm_minor *dm; + + minor = find_next_bit(known_devs, MAX_DRM_DEV, 0); + while (minor < MAX_DRM_DEV) { + dm = acquire_drm_minor(minor); + + if (IS_ERR(dm)) + continue; + + rc = fn(minor, (void *)dm, data); + + put_drm_dev(dm->dev); /* release from acquire_drm_minor */ + + if (rc) + break; + + minor = find_next_bit(known_devs, MAX_DRM_DEV, minor+1); + } + } + mutex_unlock(&drmcg_mutex); + + return rc; +} + static void drmcg_css_free(struct cgroup_subsys_state *css) { struct drmcg *drmcg = css_to_drmcg(css); From patchwork Fri Feb 14 15:56:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382595 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8E221820 for ; Fri, 14 Feb 2020 15:57:20 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B71BB2468A for ; Fri, 14 Feb 2020 15:57:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="uIWnIxTc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B71BB2468A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8453D6F9EF; Fri, 14 Feb 2020 15:57:15 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2062.outbound.protection.outlook.com [40.107.244.62]) by gabe.freedesktop.org (Postfix) with ESMTPS id D344B6F9E6; Fri, 14 Feb 2020 15:57:13 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lkXiGhSuaaXD7mv1ZJX/K66SsOzTF8BQTHrAJdW/D4m7UefI0nP7Z/6SsT2jCGVu2jQV2aDFjN5lS2O3Ys1UY/qlPiaKm8TA/UXLH89nfTcWCmxxa4HiZp79tJbFlv74nJwW02OI3vhfvZqpI57Y8Ne/zy+SbHJdBZrI9ISbJSBcoVoy82a/H9liXoriZZ/C6DmgKbLz9gni/onXMOROPx/hl0F5vzSB+DgVX1bRPQLOfbIIl1gfl4TE5DqLV6bE6OsbpZng8aF5c2fJVG/g81jHTZSPINVR/WsKmz8wXURDX6zsnu+h5ZPiKE1WQRfJo7T0uVLrTBObRiAkIk08ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CV7hGKAQeTmH37PsOyv+FEVbtbyZ1TJI7sXBsy6xaBQ=; b=U5d70NDL7K67BmzFZCeiH9WXlNWAj9Po4utrP53iv1Sbzze6ktc0gWRyUOu2TybvZAk08HpAEhJjSoO3b0VrhgpmXXQWqeWSPpAqNfiidLLaUH1rSjrAlxCApqSCyhlCThUDpn5zBhDuCAeK7Q/3DvG5M6Cz5GQ9Us3X5+EDS/Bk6ivK/30+zOV7oyonZEncf3OE6i9u+oqZRrpc8ZpCM1SH4UvuGT3PlHmxXn1VN+Hhbwh4HHp9M1lVbSDuuLs7teFp3AFiVOusreqTsCQuMSvuVAfiWc4DbDeFX0olXE+QG9gHdSBmA07lxKAlFLgeUCdAfzEaTE+YmlpddRkIXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CV7hGKAQeTmH37PsOyv+FEVbtbyZ1TJI7sXBsy6xaBQ=; b=uIWnIxTcOFpBfiPcqKacFI19Kf+Wa31RvTsYiNGnMeORaE/yoeLkidMNg95XyOPLXUPETTymHFanVCM8X81aYyHQ0Qr+pTQMQoGb9dcfnNP3GZ6iNxDybtlEROnwueuLyyBGUFB7+0L8WAf8ut7MmZ6AGE+tBkYI4k1jkEJbkVg= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by DM6PR12MB2761.namprd12.prod.outlook.com (2603:10b6:5:41::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.23; Fri, 14 Feb 2020 15:57:11 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:11 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:11 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:10 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:09 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 03/11] drm, cgroup: Initialize drmcg properties Date: Fri, 14 Feb 2020 10:56:42 -0500 Message-ID: <20200214155650.21203-4-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(346002)(39860400002)(396003)(376002)(428003)(199004)(189003)(6666004)(356004)(70206006)(86362001)(110136005)(316002)(81156014)(1076003)(8676002)(6636002)(8936002)(336012)(81166006)(70586007)(4326008)(26005)(5660300002)(478600001)(36756003)(2906002)(2616005)(186003)(426003)(7696005)(921003)(2101003)(83996005)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB2761; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 28801ced-5f46-4de6-7cf9-08d7b1668d99 X-MS-TrafficTypeDiagnostic: DM6PR12MB2761: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8a5IU/n/HswOpS3I2RDZLnARtdy/6Y0zR/MKTj5ikm565VAV1PuLwAC4GFliDyIfPT0k5n5TPGQObf2FeMjzUvrrlbDVsKBGZDTlHMTU6/oEQ5gSb8bc/p6pI1tJjgTX+o4QiV9ApgsCyq5Y3SGjJj4KBMFOgSMjF8pABzuL3RhRUsfu21sJJMb88pCI0b1YLZ5ODekOjs/sgHAMGPcJkcpsqxTq/OYPolZchqD9KzrgsaHLAcVVMCZsRXEjkaVWHW0LJMwNAUNP2fzw9oUjFMz3ktetEs2hhdfoqcuVGom33dsHFo+IIYKYt3+E6mqCcZvr0/13xInUGJl/zE8rLQ2ooFGWqEd4UoKw9SZNKDoMfi5QUwdspxfkBGLAqsZLOgacgJjE7WRbN31Q3p5GpGbfXNBiXTpQa8NODjylnRrdq7pu+n3rfOkmw6Q0BkOehui8F3Y28IyQUQuBB4Lc/V3bJuYiSDhobXWGfWkFHGoVftXIfEE1QHgDw2ANq7NobFHRAWsmO8PGHw7AY2Vi2biisbUCSA2BACWHvMIzYzM= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:11.6211 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 28801ced-5f46-4de6-7cf9-08d7b1668d99 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2761 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drmcg initialization involves allocating a per cgroup, per device data structure and setting the defaults. There are two entry points for drmcg init: 1) When struct drmcg is created via css_alloc, initialization is done for each device 2) When DRM devices are created after drmcgs are created a) Per device drmcg data structure is allocated at the beginning of DRM device creation such that drmcg can begin tracking usage statistics b) At the end of DRM device creation, drmcg_register_dev will update in case device specific defaults need to be applied. Entry point #2 usually applies to the root cgroup since it can be created before DRM devices are available. The drmcg controller will go through all existing drm cgroups and initialize them with the new device accordingly. Change-Id: I64e421d8dfcc22ee8282cc1305960e20c2704db7 Signed-off-by: Kenny Ho --- drivers/gpu/drm/drm_drv.c | 4 ++ include/drm/drm_cgroup.h | 18 +++++++ include/drm/drm_device.h | 7 +++ include/drm/drm_drv.h | 9 ++++ include/linux/cgroup_drm.h | 12 +++++ kernel/cgroup/drm.c | 105 +++++++++++++++++++++++++++++++++++++ 6 files changed, 155 insertions(+) diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 8e59cc5a5bde..44a66edc81c2 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -643,6 +643,7 @@ int drm_dev_init(struct drm_device *dev, mutex_init(&dev->filelist_mutex); mutex_init(&dev->clientlist_mutex); mutex_init(&dev->master_mutex); + mutex_init(&dev->drmcg_mutex); dev->anon_inode = drm_fs_inode_new(); if (IS_ERR(dev->anon_inode)) { @@ -679,6 +680,7 @@ int drm_dev_init(struct drm_device *dev, if (ret) goto err_setunique; + drmcg_device_early_init(dev); return 0; err_setunique: @@ -693,6 +695,7 @@ int drm_dev_init(struct drm_device *dev, drm_fs_inode_free(dev->anon_inode); err_free: put_device(dev->dev); + mutex_destroy(&dev->drmcg_mutex); mutex_destroy(&dev->master_mutex); mutex_destroy(&dev->clientlist_mutex); mutex_destroy(&dev->filelist_mutex); @@ -769,6 +772,7 @@ void drm_dev_fini(struct drm_device *dev) put_device(dev->dev); + mutex_destroy(&dev->drmcg_mutex); mutex_destroy(&dev->master_mutex); mutex_destroy(&dev->clientlist_mutex); mutex_destroy(&dev->filelist_mutex); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 530c9a0b3238..fda426fba035 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -4,8 +4,17 @@ #ifndef __DRM_CGROUP_H__ #define __DRM_CGROUP_H__ +#include + #ifdef CONFIG_CGROUP_DRM +/** + * Per DRM device properties for DRM cgroup controller for the purpose + * of storing per device defaults + */ +struct drmcg_props { +}; + void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), void (*put_ddev)(struct drm_device *dev)); @@ -15,8 +24,13 @@ void drmcg_register_dev(struct drm_device *dev); void drmcg_unregister_dev(struct drm_device *dev); +void drmcg_device_early_init(struct drm_device *device); + #else +struct drmcg_props { +}; + static inline void drmcg_bind( struct drm_minor (*(*acq_dm)(unsigned int minor_id)), void (*put_ddev)(struct drm_device *dev)) @@ -35,5 +49,9 @@ static inline void drmcg_unregister_dev(struct drm_device *dev) { } +static inline void drmcg_device_early_init(struct drm_device *device) +{ +} + #endif /* CONFIG_CGROUP_DRM */ #endif /* __DRM_CGROUP_H__ */ diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h index 1acfc3bbd3fb..a94598b8f670 100644 --- a/include/drm/drm_device.h +++ b/include/drm/drm_device.h @@ -8,6 +8,7 @@ #include #include +#include struct drm_driver; struct drm_minor; @@ -308,6 +309,12 @@ struct drm_device { */ struct drm_fb_helper *fb_helper; + /** \name DRM Cgroup */ + /*@{ */ + struct mutex drmcg_mutex; + struct drmcg_props drmcg_props; + /*@} */ + /* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED(CONFIG_DRM_LEGACY) diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index cf13470810a5..1f65ac4d9bbf 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -715,6 +715,15 @@ struct drm_driver { struct drm_device *dev, uint32_t handle); + /** + * @drmcg_custom_init + * + * Optional callback used to initialize drm cgroup per device properties + * such as resource limit defaults. + */ + void (*drmcg_custom_init)(struct drm_device *dev, + struct drmcg_props *props); + /** * @gem_vm_ops: Driver private ops for this object * diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 854591bbb430..2014097eb75c 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -4,6 +4,7 @@ #ifndef _CGROUP_DRM_H #define _CGROUP_DRM_H +#include #include #include @@ -12,11 +13,19 @@ #ifdef CONFIG_CGROUP_DRM +/** + * Per DRM cgroup, per device resources (such as statistics and limits) + */ +struct drmcg_device_resource { + /* for per device stats */ +}; + /** * The DRM cgroup controller data structure. */ struct drmcg { struct cgroup_subsys_state css; + struct drmcg_device_resource *dev_resources[MAX_DRM_DEV]; }; /** @@ -70,6 +79,9 @@ static inline struct drmcg *drmcg_parent(struct drmcg *cg) #else /* CONFIG_CGROUP_DRM */ +struct drmcg_device_resource { +}; + struct drmcg { }; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 37f98dc47268..30fd9aeffbe7 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -1,11 +1,17 @@ // SPDX-License-Identifier: MIT // Copyright 2019 Advanced Micro Devices, Inc. #include +#include #include #include #include +#include +#include +#include +#include #include #include +#include #include #include @@ -54,6 +60,47 @@ void drmcg_unbind(void) } EXPORT_SYMBOL(drmcg_unbind); +/* caller must hold dev->drmcg_mutex */ +static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) +{ + int minor = dev->primary->index; + struct drmcg_device_resource *ddr = drmcg->dev_resources[minor]; + + if (ddr == NULL) { + ddr = kzalloc(sizeof(struct drmcg_device_resource), + GFP_KERNEL); + + if (!ddr) + return -ENOMEM; + } + + drmcg->dev_resources[minor] = ddr; + + /* set defaults here */ + + return 0; +} + +static inline void drmcg_update_cg_tree(struct drm_device *dev) +{ + struct cgroup_subsys_state *pos; + struct drmcg *child; + + if (root_drmcg == NULL) + return; + + /* init cgroups created before registration (i.e. root cgroup) */ + + /* use cgroup_mutex instead of rcu_read_lock because + * init_drmcg_single has alloc which may sleep */ + mutex_lock(&cgroup_mutex); + css_for_each_descendant_pre(pos, &root_drmcg->css) { + child = css_to_drmcg(pos); + init_drmcg_single(child, dev); + } + mutex_unlock(&cgroup_mutex); +} + /** * drmcg_register_dev - register a DRM device for usage in drm cgroup * @dev: DRM device @@ -71,6 +118,13 @@ void drmcg_register_dev(struct drm_device *dev) mutex_lock(&drmcg_mutex); set_bit(dev->primary->index, known_devs); + + if (dev->driver->drmcg_custom_init) + { + dev->driver->drmcg_custom_init(dev, &dev->drmcg_props); + + drmcg_update_cg_tree(dev); + } mutex_unlock(&drmcg_mutex); } EXPORT_SYMBOL(drmcg_register_dev); @@ -137,23 +191,61 @@ static int drm_minor_for_each(int (*fn)(int id, void *p, void *data), return rc; } +static int drmcg_css_free_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + kfree(drmcg->dev_resources[minor->index]); + + return 0; +} + static void drmcg_css_free(struct cgroup_subsys_state *css) { struct drmcg *drmcg = css_to_drmcg(css); + drm_minor_for_each(&drmcg_css_free_fn, drmcg); + kfree(drmcg); } +static int init_drmcg_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + int rc; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + mutex_lock(&minor->dev->drmcg_mutex); + rc = init_drmcg_single(drmcg, minor->dev); + mutex_unlock(&minor->dev->drmcg_mutex); + + return rc; +} + static struct cgroup_subsys_state * drmcg_css_alloc(struct cgroup_subsys_state *parent_css) { struct drmcg *parent = css_to_drmcg(parent_css); struct drmcg *drmcg; + int rc; drmcg = kzalloc(sizeof(struct drmcg), GFP_KERNEL); if (!drmcg) return ERR_PTR(-ENOMEM); + rc = drm_minor_for_each(&init_drmcg_fn, drmcg); + if (rc) { + drmcg_css_free(&drmcg->css); + return ERR_PTR(rc); + } + if (!parent) root_drmcg = drmcg; @@ -171,3 +263,16 @@ struct cgroup_subsys drm_cgrp_subsys = { .legacy_cftypes = files, .dfl_cftypes = files, }; + +/** + * drmcg_device_early_init - initialize device specific resources for DRM cgroups + * @dev: the target DRM device + * + * Allocate and initialize device specific resources for existing DRM cgroups. + * Typically only the root cgroup exists before the initialization of @dev. + */ +void drmcg_device_early_init(struct drm_device *dev) +{ + drmcg_update_cg_tree(dev); +} +EXPORT_SYMBOL(drmcg_device_early_init); From patchwork Fri Feb 14 15:56:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382593 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4395A186E for ; Fri, 14 Feb 2020 15:57:19 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 21CC024676 for ; Fri, 14 Feb 2020 15:57:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="JllNquCx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 21CC024676 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 07BD36F9EB; Fri, 14 Feb 2020 15:57:15 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2042.outbound.protection.outlook.com [40.107.243.42]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1D2606F9E6; Fri, 14 Feb 2020 15:57:14 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UbDipHM4+zZL6clrE8DP+RxMtwqJddR4J84Kn7AO7JwcBRN0vcEj2kweWkmcpHQhuKcxn5Y2gs5Q7jcDdBVOSRKW/xx/vTLd3s/EP8X3+oz+FsCx/0D2rmlU73TCeCDYhcgDHSff2dPJSiBzzBIPcSRq7XV2QInJExAJeI66J5ofSH1xVtlMcRMbkAw6ZcGyvkhK7zD3aVlSN6pfYVvXiWjY20S3a0i9PutrDx5tWBxbdV2+Cy96QbsY/jHJqC1QXrDs7heotjZziSSOOBrKKxNBfryBBflYGJmp0aYTPSlcPJzyKdj2aqrP7+CFbfJq7DJKq6kZ2liVkN6y+vY0Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R9GfjKmC6WE9KgF6VLgLnREHgTv0N9zlsmLGWtvhZZk=; b=IotRfZPL57F7HMX+1O1mW0nemPPJ6LKECVEY7UcGyu9Hb7Gf39W0Nx51alAYTyhdbn//fKzau5lVlp2lbuoSjTOu2oi++6u/uWv5WvZQ3QtGbL4RfNGGyoUwDW7Z/GaGB/kqRyIK2hyQognM2hF2qtb4mSqh9kxL1POOCKRtuMYnAoJ1Yb5mG70rGIj4WuMln1W8P5kirQP0id5x+bCeKrllcYxNUQv1Mz0B0Klw080Ld8d6730hw9A1+B9wTUZ96VAbr/Xndss0d3m5phrnRpHcrWE899fJXodv0zwTZdj7/ULol5h+/oX+zTX780h7Ejeb8bg87Gtuas/1KN5KLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R9GfjKmC6WE9KgF6VLgLnREHgTv0N9zlsmLGWtvhZZk=; b=JllNquCxGdAyRzGcbJyQgkkiuFrHa/7lIhyBCSeyUqYXGGaCUtGxLZYJekFw8HceqmrcQuQLICMA2hAnkEsfNmSEbYLPnIpxXPc5ZqUz4YGPyfaMpvGlkY2u+knAWb1zfihsbAixou99R8gJMA6WvmefqsC9OmgUW1PIUOMRUEk= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by DM6PR12MB2873.namprd12.prod.outlook.com (2603:10b6:5:18a::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2707.29; Fri, 14 Feb 2020 15:57:12 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:12 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:12 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:11 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:10 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 04/11] drm, cgroup: Add total GEM buffer allocation stats Date: Fri, 14 Feb 2020 10:56:43 -0500 Message-ID: <20200214155650.21203-5-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(39860400002)(136003)(376002)(428003)(189003)(199004)(1076003)(7696005)(8676002)(478600001)(2906002)(30864003)(70586007)(81156014)(186003)(8936002)(316002)(70206006)(81166006)(356004)(110136005)(6666004)(26005)(86362001)(36756003)(336012)(4326008)(426003)(2616005)(6636002)(5660300002)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB2873; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9fee3391-cac0-4995-b70f-08d7b1668dee X-MS-TrafficTypeDiagnostic: DM6PR12MB2873: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D3HEl3tv9gHdkVFsUkpqW6XjlR/BflZuBTU+oxvGyuD3oio69wX2z+BkW8BMg6A8pDA5zjaEimsB4wW1ESea3z/7itcilmHLwrJ6ADYnJoVGLlbaTFxw7AiBKdQVPyg99j6/85l1qf+/1VpN6fNWMLixLekGXU8Hm/YrvxQ2GVQ6ivc/vFeuMx98zPwby8+NaYmMn8A9ntLfOZbaVe2+KuPXdvj0A2Q6/Bu8olJYjBsSZnKIfRs6PVit0lsIIYcqPEwJjzYaRY9IhjCei+Jr4la+6flbnrHO3FnrRPCsjydn7RdDA30uabjqueZqEn2us1wMbY5dmNt0fHkzXwZ4fXY8G84Oya0G6ZJ7qdDckRelFRGOHZnQ8RSWCqfF/GN2z1/HpOqkRdfCYvmH96Ao3yWJoXo/ypQqMi1ff99FhJIgOMUSiJ0/LvNiXkrPQMJa4kmpX9vEK0CGbfFzLlnQAADSHMGgk0fD6/iJBlFxtTDSKQcceOcEFHEOUVDhiZNh9BBQRvKD+iQ9J8SsRMjQuV0xLOe8DCpr930skYKnzs8= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:12.1806 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9fee3391-cac0-4995-b70f-08d7b1668dee X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2873 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The drm resource being measured here is the GEM buffer objects. User applications allocate and free these buffers. In addition, a process can allocate a buffer and share it with another process. The consumer of a shared buffer can also outlive the allocator of the buffer. For the purpose of cgroup accounting and limiting, ownership of the buffer is deemed to be the cgroup for which the allocating process belongs to. There is one cgroup stats per drm device. Each allocation is charged to the owning cgroup as well as all its ancestors. Similar to the memory cgroup, migrating a process to a different cgroup does not move the GEM buffer usages that the process started while in previous cgroup, to the new cgroup. The following is an example to illustrate some of the operations. Given the following cgroup hierarchy (The letters are cgroup names with R being the root cgroup. The numbers in brackets are processes. The processes are placed with cgroup's 'No Internal Process Constraint' in mind, so no process is placed in cgroup B.) R (4, 5) ------ A (6) \ B ---- C (7,8) \ D (9) Here is a list of operation and the associated effect on the size track by the cgroups (for simplicity, each buffer is 1 unit in size.) == == == == == =================================================== R A B C D Ops == == == == == =================================================== 1 0 0 0 0 4 allocated a buffer 1 0 0 0 0 4 shared a buffer with 5 1 0 0 0 0 4 shared a buffer with 9 2 0 1 0 1 9 allocated a buffer 3 0 2 1 1 7 allocated a buffer 3 0 2 1 1 7 shared a buffer with 8 3 0 2 1 1 7 sharing with 9 3 0 2 1 1 7 release a buffer 3 0 2 1 1 7 migrate to cgroup D 3 0 2 1 1 9 release a buffer from 7 2 0 1 0 1 8 release a buffer from 7 (last ref to shared buf) == == == == == =================================================== drm.buffer.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Total GEM buffer allocation in bytes. Change-Id: Ibc1f646ca7dbc588e2d11802b156b524696a23e7 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 50 +++++++++- drivers/gpu/drm/drm_gem.c | 9 ++ include/drm/drm_cgroup.h | 16 +++ include/drm/drm_gem.h | 10 ++ include/linux/cgroup_drm.h | 6 ++ kernel/cgroup/drm.c | 126 ++++++++++++++++++++++++ 6 files changed, 216 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 384db8df0f30..2d8162c109f3 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -63,6 +63,7 @@ v1 is available under Documentation/admin-guide/cgroup-v1/. 5-7-1. RDMA Interface Files 5-8. DRM 5-8-1. DRM Interface Files + 5-8-2. GEM Buffer Ownership 5-9. Misc 5-9-1. perf_event 5-N. Non-normative information @@ -2062,7 +2063,54 @@ of DRM (Direct Rendering Manager) and GPU-related resources. DRM Interface Files ~~~~~~~~~~~~~~~~~~~~ -TODO + drm.buffer.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Total GEM buffer allocation in bytes. + +GEM Buffer Ownership +~~~~~~~~~~~~~~~~~~~~ + +For the purpose of cgroup accounting and limiting, ownership of the +buffer is deemed to be the cgroup for which the allocating process +belongs to. There is one cgroup stats per drm device. Each allocation +is charged to the owning cgroup as well as all its ancestors. + +Similar to the memory cgroup, migrating a process to a different cgroup +does not move the GEM buffer usages that the process started while in +previous cgroup, to the new cgroup. + +The following is an example to illustrate some of the operations. Given +the following cgroup hierarchy (The letters are cgroup names with R +being the root cgroup. The numbers in brackets are processes. The +processes are placed with cgroup's 'No Internal Process Constraint' in +mind, so no process is placed in cgroup B.) + +R (4, 5) ------ A (6) + \ + B ---- C (7,8) + \ + D (9) + +Here is a list of operation and the associated effect on the size +track by the cgroups (for simplicity, each buffer is 1 unit in size.) + +== == == == == =================================================== +R A B C D Ops +== == == == == =================================================== +1 0 0 0 0 4 allocated a buffer +1 0 0 0 0 4 shared a buffer with 5 +1 0 0 0 0 4 shared a buffer with 9 +2 0 1 0 1 9 allocated a buffer +3 0 2 1 1 7 allocated a buffer +3 0 2 1 1 7 shared a buffer with 8 +3 0 2 1 1 7 sharing with 9 +3 0 2 1 1 7 release a buffer +3 0 2 1 1 7 migrate to cgroup D +3 0 2 1 1 9 release a buffer from 7 +2 0 1 0 1 8 release a buffer from 7 (last ref to shared buf) +== == == == == =================================================== Misc diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index 2f2b889096b0..d158470edd98 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -46,6 +47,7 @@ #include #include #include +#include #include "drm_internal.h" @@ -164,6 +166,9 @@ void drm_gem_private_object_init(struct drm_device *dev, obj->resv = &obj->_resv; drm_vma_node_reset(&obj->vma_node); + + obj->drmcg = drmcg_get(current); + drmcg_chg_bo_alloc(obj->drmcg, dev, size); } EXPORT_SYMBOL(drm_gem_private_object_init); @@ -957,6 +962,10 @@ drm_gem_object_release(struct drm_gem_object *obj) fput(obj->filp); dma_resv_fini(&obj->_resv); + + drmcg_unchg_bo_alloc(obj->drmcg, obj->dev, obj->size); + drmcg_put(obj->drmcg); + drm_gem_free_mmap_offset(obj); } EXPORT_SYMBOL(drm_gem_object_release); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index fda426fba035..1eb3012e16a1 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -26,6 +26,12 @@ void drmcg_unregister_dev(struct drm_device *dev); void drmcg_device_early_init(struct drm_device *device); +void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size); + +void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size); + #else struct drmcg_props { @@ -53,5 +59,15 @@ static inline void drmcg_device_early_init(struct drm_device *device) { } +static inline void drmcg_chg_bo_alloc(struct drmcg *drmcg, + struct drm_device *dev, size_t size) +{ +} + +static inline void drmcg_unchg_bo_alloc(struct drmcg *drmcg, + struct drm_device *dev, size_t size) +{ +} + #endif /* CONFIG_CGROUP_DRM */ #endif /* __DRM_CGROUP_H__ */ diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h index 97a48165642c..6ac7018923f7 100644 --- a/include/drm/drm_gem.h +++ b/include/drm/drm_gem.h @@ -312,6 +312,16 @@ struct drm_gem_object { * */ const struct drm_gem_object_funcs *funcs; + + /** + * @drmcg: + * + * DRM cgroup this GEM object belongs to. + * + * This is used to track and limit the amount of GEM objects a user + * can allocate. + */ + struct drmcg *drmcg; }; /** diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 2014097eb75c..174ab50701ef 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -11,6 +11,11 @@ /* limit defined per the way drm_minor_alloc operates */ #define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) +enum drmcg_res_type { + DRMCG_TYPE_BO_TOTAL, + __DRMCG_TYPE_LAST, +}; + #ifdef CONFIG_CGROUP_DRM /** @@ -18,6 +23,7 @@ */ struct drmcg_device_resource { /* for per device stats */ + s64 bo_stats_total_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 30fd9aeffbe7..425566753a5c 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -13,6 +13,7 @@ #include #include #include +#include #include static struct drmcg *root_drmcg __read_mostly; @@ -26,6 +27,18 @@ static struct drm_minor (*(*acquire_drm_minor)(unsigned int minor_id)); static void (*put_drm_dev)(struct drm_device *dev); +#define DRMCG_CTF_PRIV_SIZE 3 +#define DRMCG_CTF_PRIV_MASK GENMASK((DRMCG_CTF_PRIV_SIZE - 1), 0) +#define DRMCG_CTF_PRIV(res_type, f_type) ((res_type) <<\ + DRMCG_CTF_PRIV_SIZE | (f_type)) +#define DRMCG_CTF_PRIV2RESTYPE(priv) ((priv) >> DRMCG_CTF_PRIV_SIZE) +#define DRMCG_CTF_PRIV2FTYPE(priv) ((priv) & DRMCG_CTF_PRIV_MASK) + + +enum drmcg_file_type { + DRMCG_FTYPE_STATS, +}; + /** * drmcg_bind - Bind DRM subsystem to cgroup subsystem * @acq_dm: function pointer to the drm_minor_acquire function @@ -252,7 +265,66 @@ drmcg_css_alloc(struct cgroup_subsys_state *parent_css) return &drmcg->css; } +static void drmcg_print_stats(struct drmcg_device_resource *ddr, + struct seq_file *sf, enum drmcg_res_type type) +{ + if (ddr == NULL) { + seq_puts(sf, "\n"); + return; + } + + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + +static int drmcg_seq_show_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct seq_file *sf = data; + struct drmcg *drmcg = css_to_drmcg(seq_css(sf)); + enum drmcg_file_type f_type = + DRMCG_CTF_PRIV2FTYPE(seq_cft(sf)->private); + enum drmcg_res_type type = + DRMCG_CTF_PRIV2RESTYPE(seq_cft(sf)->private); + struct drmcg_device_resource *ddr; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + ddr = drmcg->dev_resources[minor->index]; + + seq_printf(sf, "%d:%d ", DRM_MAJOR, minor->index); + + switch (f_type) { + case DRMCG_FTYPE_STATS: + drmcg_print_stats(ddr, sf, type); + break; + default: + seq_puts(sf, "\n"); + break; + } + + return 0; +} + +int drmcg_seq_show(struct seq_file *sf, void *v) +{ + return drm_minor_for_each(&drmcg_seq_show_fn, sf); +} + struct cftype files[] = { + { + .name = "buffer.total.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -276,3 +348,57 @@ void drmcg_device_early_init(struct drm_device *dev) drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); + +/** + * drmcg_chg_bo_alloc - charge GEM buffer usage for a device and cgroup + * @drmcg: the DRM cgroup to be charged to + * @dev: the device the usage should be charged to + * @size: size of the GEM buffer to be accounted for + * + * This function should be called when a new GEM buffer is allocated to account + * for the utilization. This should not be called when the buffer is shared ( + * the GEM buffer's reference count being incremented.) + */ +void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size) +{ + struct drmcg_device_resource *ddr; + int devIdx = dev->primary->index; + + if (drmcg == NULL) + return; + + mutex_lock(&dev->drmcg_mutex); + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + + ddr->bo_stats_total_allocated += (s64)size; + } + mutex_unlock(&dev->drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_chg_bo_alloc); + +/** + * drmcg_unchg_bo_alloc - + * @drmcg: the DRM cgroup to uncharge from + * @dev: the device the usage should be removed from + * @size: size of the GEM buffer to be accounted for + * + * This function should be called when the GEM buffer is about to be freed ( + * not simply when the GEM buffer's reference count is being decremented.) + */ +void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, + size_t size) +{ + int devIdx = dev->primary->index; + + if (drmcg == NULL) + return; + + mutex_lock(&dev->drmcg_mutex); + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) + drmcg->dev_resources[devIdx]->bo_stats_total_allocated + -= (s64)size; + mutex_unlock(&dev->drmcg_mutex); +} +EXPORT_SYMBOL(drmcg_unchg_bo_alloc); From patchwork Fri Feb 14 15:56:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382597 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B344A159A for ; Fri, 14 Feb 2020 15:57:24 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9134D2086A for ; Fri, 14 Feb 2020 15:57:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="O/tD8aOn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9134D2086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7AEF66F9E5; Fri, 14 Feb 2020 15:57:18 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2085.outbound.protection.outlook.com [40.107.244.85]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3652D6F9EC; Fri, 14 Feb 2020 15:57:16 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D05vAlX4qGuwHh5eWOffGdlS6EGZbD6r1E4YRjfqSmHD/6bRmD0c6QUPg3zJOHR3Ql5FTPvgV1NEL1w5MlrlCstk1Rz0V8R6HtZ3ptqYLZM1UsbQTNJX6xfsxZDCmuHdpkh3dXzl0s93F/iSduUTL85k6g2t2hX4oNJhY01mSY3t9nwFQl8CqI9jMUYctifrRd+H8NyFjDlqHT/xHk4655Z4jFNDcqyZrh0t7uj6O5D9WYvNx3G/N4f24N/s9mgPhlo2IIsNJXvyUkb29NPrgIqQ6+j+7J7JKkCS9Bgwm+LocMRUYx/ILsA0MZ5pON23U7N7tEtzRiDa1HD85a1K1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Oy38XTm1AslX2SVWlbJy84/4buILNfghMRpt4e5sNT4=; b=V0t/Nu/aQiYuo6saC0F86HgwDa9ppNdXQy9GX6J94BMZ9kwGGGY2K2qxTlpCW0xEO/Awa/pUI6wN+o9eGs3MjGtlHw9nLKgZfjknszE1Hh261Gj0g5TggMikntkZh/cB8zRga0p5m0dX0W1QoOBTb7WMHArrb3fvSEhJK38GWRw6QCSDNFm2IM+obdZj2IO2e2WrtRT3SB80qBkbcMHAbkmDm4cRBNJdRP6n+xjQTR4XMDDMp0XZx462jwZ28uUHKtjYCTzY71gUdagfGySxw+IgGv5WutpFVIPiDd0DnOHgweBqB88ZQOdvWtnXjexP3CRKvnNzqO2eT8/tZ7gEfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Oy38XTm1AslX2SVWlbJy84/4buILNfghMRpt4e5sNT4=; b=O/tD8aOn9LPELaSLMQQEKZjmCEVywCBgQl6mzQbzYBHj9AZChiO5C59V0gxFj8yjMrVIxO9jZrDx7nsq6DFKD6U+7HOX7RQa/uo2V5vepfWLH0w0wrK+1p+1qk/COioxYPvKVn32iEQHUhTvsxc7L+2WCU6HJ2S3mczaIfi2AhY= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by BYAPR12MB3286.namprd12.prod.outlook.com (2603:10b6:a03:139::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:14 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:14 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:14 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:12 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:11 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 05/11] drm, cgroup: Add peak GEM buffer allocation stats Date: Fri, 14 Feb 2020 10:56:44 -0500 Message-ID: <20200214155650.21203-6-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(136003)(346002)(39860400002)(428003)(189003)(199004)(186003)(2906002)(26005)(86362001)(2616005)(1076003)(4326008)(6636002)(110136005)(316002)(36756003)(70206006)(70586007)(426003)(7696005)(336012)(356004)(6666004)(8676002)(8936002)(5660300002)(81166006)(81156014)(478600001)(921003)(83996005)(2101003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR12MB3286; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6b2d1b88-7b6c-4f28-ef7c-08d7b1668f2c X-MS-TrafficTypeDiagnostic: BYAPR12MB3286: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mZTP7oDK2XTQyX/6b38Q7fmmv5YXJpHsqv9bqaLN4JWoBiIe59dyR5iC9UOUDMW2rkqs1hd1sKo1ljGlBjczAt80o5C5fc9kmx/1eZMG9/mmzLP15V715ZAL56EuTn77fz3V+RZG3Gmbpx0vXeO/iE1OvkqKq7IzvOoi6gCVrgxH0AatjaAatL2sHs+Uy3SsuZgodpd+ZOKvdKvk3ofI2FmvNZkhHyPAF1CeQO7/jWIms8a1zh9Igq7uyk6z/C9J9vVDTTLxH7gxoN0r8VU+ca8YgB4qfuUedccjUJLErggXPMLdatsHOrVhXEU9gywZlZOlTjURhdPa62ruDyrsLU3Bll4wr2kcNuBWCFbbLY7roPw9oQwrhefBJ+nd3thD75m/O4HMATmqqFLWb2RE4kQISOwuP9picTQEGy0RQrTEarHiw5u/QVbVidT2y8NkBH0Sco1RrtrCkz5Vvqz1v8TbeeYLaz+0WH/dZUMCdowSAKtt3ER3cbs9F+x1EPTqlIcAuoijH7wpq//+A8HQiAWA5dFp5h2OAm+aKHBWMvY= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:14.2661 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6b2d1b88-7b6c-4f28-ef7c-08d7b1668f2c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3286 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drm.buffer.peak.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Largest (high water mark) GEM buffer allocated in bytes. Change-Id: I40fe4c13c1cea8613b3e04b802f3e1f19eaab4fc Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 6 ++++++ include/linux/cgroup_drm.h | 3 +++ kernel/cgroup/drm.c | 12 ++++++++++++ 3 files changed, 21 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 2d8162c109f3..75b97962b127 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2069,6 +2069,12 @@ DRM Interface Files Total GEM buffer allocation in bytes. + drm.buffer.peak.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Largest (high water mark) GEM buffer allocated in bytes. + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 174ab50701ef..593ad12602cd 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -13,6 +13,7 @@ enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, + DRMCG_TYPE_BO_PEAK, __DRMCG_TYPE_LAST, }; @@ -24,6 +25,8 @@ enum drmcg_res_type { struct drmcg_device_resource { /* for per device stats */ s64 bo_stats_total_allocated; + + s64 bo_stats_peak_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 425566753a5c..7a0da70c5a25 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -277,6 +277,9 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_TOTAL: seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", ddr->bo_stats_peak_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -325,6 +328,12 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.peak.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -373,6 +382,9 @@ void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, ddr = drmcg->dev_resources[devIdx]; ddr->bo_stats_total_allocated += (s64)size; + + if (ddr->bo_stats_peak_allocated < (s64)size) + ddr->bo_stats_peak_allocated = (s64)size; } mutex_unlock(&dev->drmcg_mutex); } From patchwork Fri Feb 14 15:56:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382601 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F93C159A for ; Fri, 14 Feb 2020 15:57:30 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D9DF2468C for ; Fri, 14 Feb 2020 15:57:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="ZXNvmeey" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D9DF2468C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 735F06F9FC; Fri, 14 Feb 2020 15:57:21 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2076.outbound.protection.outlook.com [40.107.244.76]) by gabe.freedesktop.org (Postfix) with ESMTPS id 41BBD6F9F3; Fri, 14 Feb 2020 15:57:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bYSJAfFMAYFE8gqu8576XVMEcL3dUiBsUOAdaHohAqoW0aadcOel1qJr0KSAHOBXJOYw0tgjZSzQ7+wRov+SyrOn0K02Tlk1YZgw0QK3R0FQBUEzk91N0zB3ZLYny9rVXCVPjZhcOQ/bheJO4o9rsmFdag0C2e9HI6SzRqsPKovmm1+Gb/vvqdQ4ujMtxJww6iQXWKsrxKNbS72kuCJQlSyStaMzzEZgdWOdRf5fMVBqyDHffdTffu/oX0BkRlAUHGe3+v0eHXXyW2eJ42zYq0tKRs8RppYyJqNiuHmsyhgI20c116C9pNdNCfl3A+s+A1MSTDrghV2eUL8RFTVMwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WDD+cg9HBA357VUjoIcAxcX6UVY3AQ/GVu0LDkHvt50=; b=fYSq2hIEq70calrJAHhH82wgCN9shLUqbYdj3wVF67vG3VQbTvj+LcHIGBS1C7IiTg8xWnU1gr7ohavKwwFpD92SfTYLpyoI9+uvMBLL4mmqGWaw9Udv7e+LWV3rP0w761Tebf07WEx5WgVYMRW0de9tZB0VEYu0g3yqdb/OnIhGxw81S4tBlpoGtLkrXTtCPbiKJFeX5zpIJKlmBNpCPzJBUVu0uWN78hqYXRn71QaodV9Kojdvkx27knjmyLAcdULOSN0gtvxib3DZcpiN24e6yzdHksja85uqyYyxgTsm5azSvEo+ZgKtM7vKtegNybG1bZIwlHSy2o6oC5e/bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WDD+cg9HBA357VUjoIcAxcX6UVY3AQ/GVu0LDkHvt50=; b=ZXNvmeeyMcR6EXG1zk4wJpm766woTM0kz3TirBQtRmx37K3t9N7cI7AiGX0jTjCFjp+EAfCcnDxZjxkV3BL5Kmf8uhWjksD/K9rjq2WajS6Ri8t+YhbtyW4zkckNw54UCfd+agEOhYfEsXUctJp3E7Hf7VXwrl7P0MRFaGf7Ues= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by DM6PR12MB3500.namprd12.prod.outlook.com (2603:10b6:5:11d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2707.28; Fri, 14 Feb 2020 15:57:15 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:15 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:15 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:13 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:12 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 06/11] drm, cgroup: Add GEM buffer allocation count stats Date: Fri, 14 Feb 2020 10:56:45 -0500 Message-ID: <20200214155650.21203-7-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(39860400002)(136003)(376002)(428003)(189003)(199004)(8936002)(186003)(110136005)(316002)(1076003)(8676002)(86362001)(4326008)(336012)(356004)(6666004)(426003)(7696005)(26005)(81156014)(81166006)(5660300002)(70586007)(2906002)(6636002)(2616005)(70206006)(478600001)(36756003)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB3500; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e8369b42-479c-4339-1fae-08d7b1669007 X-MS-TrafficTypeDiagnostic: DM6PR12MB3500: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kwqyLK1KmQC95I3c98hLGn9SDV7prZoUytAu4eFgTpU/zsmOledqFLxCe251O1iUx8DcoNJaTkGuSIcH+FrRVMFuzSDHa5KuAT0vIqlllg2uMCv43yZkJy0uUneeKYrmDMpo28ueJ5mrhpCicYyOOu9EImO8pUHEzoMIkmz7YEBsaMWy5E8smYJzHSdMNY9MV2FbwhGwWLIfuYzshtfvB+lHVN+4X8w/vgBIIspEMksPaS4N1xzEv65oDUzO0S+2nqNyO9vPTHp+IdTyuyOK1wSkGGJYOoLwAkinFAh5G/SLijKe/hhgm1W9bjMls22A1uloh7YaN6RrxEb25l8iJmmhWZo4JkeiDvYpxg1MgiVK9EP26CF1jYQZXE1Ez+mhkMiAK/h0fBwSX0cJ6sBTgmCivztVPwJ7Eeh4uRCLyNhQX11+a/fI+3vkR3QedONXOgvnw6f+SpfARyOyJbWkImHVs8FJTTcmz3tnkPQb8rYzuq2M5im7/CBS6dZWRSbo5QwFS7cpNbBB+zR+NRs8gFIyIiXYTaBNvfUpKmhoBbY= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:15.6940 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e8369b42-479c-4339-1fae-08d7b1669007 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3500 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drm.buffer.count.stats A read-only flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Total number of GEM buffer allocated. Change-Id: Iad29bdf44390dbcee07b1e72ea0ff811aa3b9dcd Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 6 ++++++ include/linux/cgroup_drm.h | 3 +++ kernel/cgroup/drm.c | 22 +++++++++++++++++++--- 3 files changed, 28 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 75b97962b127..19fcf54ace83 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2075,6 +2075,12 @@ DRM Interface Files Largest (high water mark) GEM buffer allocated in bytes. + drm.buffer.count.stats + A read-only flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Total number of GEM buffer allocated. + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 593ad12602cd..51a0cd37da92 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -14,6 +14,7 @@ enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, DRMCG_TYPE_BO_PEAK, + DRMCG_TYPE_BO_COUNT, __DRMCG_TYPE_LAST, }; @@ -27,6 +28,8 @@ struct drmcg_device_resource { s64 bo_stats_total_allocated; s64 bo_stats_peak_allocated; + + s64 bo_stats_count_allocated; }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 7a0da70c5a25..bc162aa9971d 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -280,6 +280,9 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_PEAK: seq_printf(sf, "%lld\n", ddr->bo_stats_peak_allocated); break; + case DRMCG_TYPE_BO_COUNT: + seq_printf(sf, "%lld\n", ddr->bo_stats_count_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -334,6 +337,12 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.count.stats", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_COUNT, + DRMCG_FTYPE_STATS), + }, { } /* terminate */ }; @@ -385,6 +394,8 @@ void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, if (ddr->bo_stats_peak_allocated < (s64)size) ddr->bo_stats_peak_allocated = (s64)size; + + ddr->bo_stats_count_allocated++; } mutex_unlock(&dev->drmcg_mutex); } @@ -402,15 +413,20 @@ EXPORT_SYMBOL(drmcg_chg_bo_alloc); void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { + struct drmcg_device_resource *ddr; int devIdx = dev->primary->index; if (drmcg == NULL) return; mutex_lock(&dev->drmcg_mutex); - for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) - drmcg->dev_resources[devIdx]->bo_stats_total_allocated - -= (s64)size; + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + + ddr->bo_stats_total_allocated -= (s64)size; + + ddr->bo_stats_count_allocated--; + } mutex_unlock(&dev->drmcg_mutex); } EXPORT_SYMBOL(drmcg_unchg_bo_alloc); From patchwork Fri Feb 14 15:56:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382599 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89D90159A for ; Fri, 14 Feb 2020 15:57:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 680142086A for ; Fri, 14 Feb 2020 15:57:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="1FBWZJEh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 680142086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7C36C6F9F2; Fri, 14 Feb 2020 15:57:20 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2087.outbound.protection.outlook.com [40.107.244.87]) by gabe.freedesktop.org (Postfix) with ESMTPS id B283A6F9F2; Fri, 14 Feb 2020 15:57:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HzqPuDhGE+lAyiiKEexfpIiqNI0u9Q1b2AMLw4+JtdT66raLbN2Bbg/ZIFLGKs/d8E6uezPyygBPNlS6fJSP367s4GsTnRLIK7UjbP5VcMf3VrW8jlYxASaaEZELcEcNd8S8HuNQJMuO6MjFo2RICbUOljsJs4TDeNaz3b6MsG3aARJ3L2tIVuBreMSuhRlnSPQuMwKRHf1Q+tfNQtrE6DQc39fHSCbyPs773naBIH+pqGe9DLp0LF1PXpjFoGuVGTTbOL/Mv9rlF6SlZSORDAfZb1JoRv17b4f0Y1NuYMC2NP5+vUf8xEqlQzIYStSrHLAhE/9ax9XyI+pvVoXDsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nE+NdonnuiVF4c6wxyhLbj83GLvODHYdGsIm5dEwkuI=; b=nGtseDWPIXQB8yBCivNbx8GY0RNneDKjyHIDNja7/ArfanHOsdPg4IFh+c4xwikJA0MOnOe/lCDJGkxmQoJG2axAaePWJR7qjmyEZKuoY38xZmZwZPFyV4/CAmLA+MrVbdHJ0k5Glx0NXuJaWIPLowssn0RE2KoBZkfF1PJcBZSe1WXAKpC609gNGREhxp6kHOQ+DOqbHFNn/HxbvUqkiKWJ84pwgJ+9m7XwjT9Xc+TMdScgvnh7n+GOKMcA22XbW/yXX62kxkdaxqJL2opqO1gjC8/8EbsylBvqnwfiWSV1TAOm60wQnL2O5Yk2y+amu+aYg9Bn1VNF40Eb99hkrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nE+NdonnuiVF4c6wxyhLbj83GLvODHYdGsIm5dEwkuI=; b=1FBWZJEhqFZCpmkPnCBXH2j2roBNkfrqzt0uZM3ousA7cuRla0gWQa7Qhjf2JAgyW1jGD1+78DAGALOpFg3LkLQBx/stRX+ZR6QAkQKpytRzS9/v/4dtZMN8FrlqBA35aH8LhmW1Jii9ve9BBLzhvraZkktdHi+xOT/DErfL8V4= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by DM6PR12MB2761.namprd12.prod.outlook.com (2603:10b6:5:41::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.23; Fri, 14 Feb 2020 15:57:16 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:16 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:16 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:14 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:14 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:13 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 07/11] drm, cgroup: Add total GEM buffer allocation limit Date: Fri, 14 Feb 2020 10:56:46 -0500 Message-ID: <20200214155650.21203-8-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(346002)(39860400002)(396003)(376002)(428003)(199004)(189003)(6666004)(356004)(70206006)(86362001)(110136005)(316002)(81156014)(1076003)(8676002)(6636002)(8936002)(336012)(30864003)(81166006)(70586007)(4326008)(26005)(5660300002)(478600001)(36756003)(2906002)(2616005)(186003)(426003)(7696005)(921003)(2101003)(83996005)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:DM6PR12MB2761; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 68551b83-9e56-416b-3433-08d7b1669058 X-MS-TrafficTypeDiagnostic: DM6PR12MB2761: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Blx27xS6Xjjg4hX0D9gQeHiwpfHKZM+zE2PgDWr22iaGr8cxnw8IE5KA5TMYaCgyiQv9+wQkecWLPxkiTTLF7tN3AZrTjqH2GM+UeSrOMsF9MV5N/kC0RIL2vhoQC0wjiBgPPbo1jtfu0XrscX4sWBdtZRrljAMrdrwyQs2dblXthLSiSBeC0myVCb6okv+JhVy1rgQ6PRkhM2bSP5mKr6qIGfqHjp1bL0NPL0hffEhATgsVzBUYbYvykMBE00xlKqP+F8mqm5DlwGU9xhmPL1UZdEWC7NxYJZcnsaKfPmGQKClh4r9+p99O5/5X3oUFtvk+ts+ji1+Wvb/36KOPJ903GHLqVjoyvyt6z54uBIo9OlOnRxw4kag/ag6Kwx48BFe6TEmAXRg4v9pDDaahUUrIC3+UASM4YUxi2991K+PgsZ7lK/b97ERA5xOla7tDP2XsjGaNambGqcFtH/XEu/9EjVH+wFDX1XHkvhnRRUjreC7q4Db6Eq2P7MIZTezD73b76/F3FbG9YwLTjlTs4OyRcIeYIIZ6RBpcw3IQitY= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:16.2264 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 68551b83-9e56-416b-3433-08d7b1669058 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2761 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The drm resource being limited here is the GEM buffer objects. User applications allocate and free these buffers. In addition, a process can allocate a buffer and share it with another process. The consumer of a shared buffer can also outlive the allocator of the buffer. For the purpose of cgroup accounting and limiting, ownership of the buffer is deemed to be the cgroup for which the allocating process belongs to. There is one cgroup limit per drm device. The limiting functionality is added to the previous stats collection function. The drm_gem_private_object_init is modified to have a return value to allow failure due to cgroup limit. The try_chg function only fails if the DRM cgroup properties has limit_enforced set to true for the DRM device. This is to allow the DRM cgroup controller to collect usage stats without enforcing the limits. drm.buffer.default A read-only flat-keyed file which exists on the root cgroup. Each entry is keyed by the drm device's major:minor. Default limits on the total GEM buffer allocation in bytes. drm.buffer.max A read-write flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Per device limits on the total GEM buffer allocation in byte. This is a hard limit. Attempts in allocating beyond the cgroup limit will result in ENOMEM. Shorthand understood by memparse (such as k, m, g) can be used. Set allocation limit for /dev/dri/card1 to 1GB echo "226:1 1g" > drm.buffer.total.max Set allocation limit for /dev/dri/card0 to 512MB echo "226:0 512m" > drm.buffer.total.max Change-Id: Id3265bbd0fafe84a16b59617df79bd32196160be Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 21 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 19 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 6 +- drivers/gpu/drm/drm_gem.c | 11 +- include/drm/drm_cgroup.h | 8 +- include/drm/drm_gem.h | 2 +- include/linux/cgroup_drm.h | 1 + kernel/cgroup/drm.c | 227 ++++++++++++++++++++- 8 files changed, 278 insertions(+), 17 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 19fcf54ace83..064172df63e2 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2081,6 +2081,27 @@ DRM Interface Files Total number of GEM buffer allocated. + drm.buffer.default + A read-only flat-keyed file which exists on the root cgroup. + Each entry is keyed by the drm device's major:minor. + + Default limits on the total GEM buffer allocation in bytes. + + drm.buffer.max + A read-write flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Per device limits on the total GEM buffer allocation in byte. + This is a hard limit. Attempts in allocating beyond the cgroup + limit will result in ENOMEM. Shorthand understood by memparse + (such as k, m, g) can be used. + + Set allocation limit for /dev/dri/card1 to 1GB + echo "226:1 1g" > drm.buffer.total.max + + Set allocation limit for /dev/dri/card0 to 512MB + echo "226:0 512m" > drm.buffer.total.max + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index f28d040de3ce..3ebef1d62346 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -1397,6 +1397,23 @@ amdgpu_get_crtc_scanout_position(struct drm_device *dev, unsigned int pipe, stime, etime, mode); } +#ifdef CONFIG_CGROUP_DRM + +static void amdgpu_drmcg_custom_init(struct drm_device *dev, + struct drmcg_props *props) +{ + props->limit_enforced = true; +} + +#else + +static void amdgpu_drmcg_custom_init(struct drm_device *dev, + struct drmcg_props *props) +{ +} + +#endif /* CONFIG_CGROUP_DRM */ + static struct drm_driver kms_driver = { .driver_features = DRIVER_USE_AGP | DRIVER_ATOMIC | @@ -1430,6 +1447,8 @@ static struct drm_driver kms_driver = { .gem_prime_vunmap = amdgpu_gem_prime_vunmap, .gem_prime_mmap = amdgpu_gem_prime_mmap, + .drmcg_custom_init = amdgpu_drmcg_custom_init, + .name = DRIVER_NAME, .desc = DRIVER_DESC, .date = DRIVER_DATE, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c index 46c76e2e1281..b81c608cb2cc 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c @@ -34,6 +34,7 @@ #include #include +#include #include "amdgpu.h" #include "amdgpu_trace.h" #include "amdgpu_amdkfd.h" @@ -551,7 +552,10 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, bo = kzalloc(sizeof(struct amdgpu_bo), GFP_KERNEL); if (bo == NULL) return -ENOMEM; - drm_gem_private_object_init(adev->ddev, &bo->tbo.base, size); + if (!drm_gem_private_object_init(adev->ddev, &bo->tbo.base, size)) { + kfree(bo); + return -ENOMEM; + } INIT_LIST_HEAD(&bo->shadow_list); bo->vm_bo = NULL; bo->preferred_domains = bp->preferred_domain ? bp->preferred_domain : diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index d158470edd98..06e7576f1758 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -150,11 +150,17 @@ EXPORT_SYMBOL(drm_gem_object_init); * no GEM provided backing store. Instead the caller is responsible for * backing the object and handling it. */ -void drm_gem_private_object_init(struct drm_device *dev, +bool drm_gem_private_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size) { BUG_ON((size & (PAGE_SIZE - 1)) != 0); + obj->drmcg = drmcg_get(current); + if (!drmcg_try_chg_bo_alloc(obj->drmcg, dev, size)) { + drmcg_put(obj->drmcg); + obj->drmcg = NULL; + return false; + } obj->dev = dev; obj->filp = NULL; @@ -167,8 +173,7 @@ void drm_gem_private_object_init(struct drm_device *dev, drm_vma_node_reset(&obj->vma_node); - obj->drmcg = drmcg_get(current); - drmcg_chg_bo_alloc(obj->drmcg, dev, size); + return true; } EXPORT_SYMBOL(drm_gem_private_object_init); diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 1eb3012e16a1..2783e56690db 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -13,6 +13,9 @@ * of storing per device defaults */ struct drmcg_props { + bool limit_enforced; + + s64 bo_limits_total_allocated_default; }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), @@ -26,7 +29,7 @@ void drmcg_unregister_dev(struct drm_device *dev); void drmcg_device_early_init(struct drm_device *device); -void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, +bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size); void drmcg_unchg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, @@ -59,9 +62,10 @@ static inline void drmcg_device_early_init(struct drm_device *device) { } -static inline void drmcg_chg_bo_alloc(struct drmcg *drmcg, +static inline bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { + return true; } static inline void drmcg_unchg_bo_alloc(struct drmcg *drmcg, diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h index 6ac7018923f7..5748845c45eb 100644 --- a/include/drm/drm_gem.h +++ b/include/drm/drm_gem.h @@ -354,7 +354,7 @@ void drm_gem_object_release(struct drm_gem_object *obj); void drm_gem_object_free(struct kref *kref); int drm_gem_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size); -void drm_gem_private_object_init(struct drm_device *dev, +bool drm_gem_private_object_init(struct drm_device *dev, struct drm_gem_object *obj, size_t size); void drm_gem_vm_open(struct vm_area_struct *vma); void drm_gem_vm_close(struct vm_area_struct *vma); diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index 51a0cd37da92..b03d90623763 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -26,6 +26,7 @@ enum drmcg_res_type { struct drmcg_device_resource { /* for per device stats */ s64 bo_stats_total_allocated; + s64 bo_limits_total_allocated; s64 bo_stats_peak_allocated; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index bc162aa9971d..ee85482edd90 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -37,6 +37,8 @@ static void (*put_drm_dev)(struct drm_device *dev); enum drmcg_file_type { DRMCG_FTYPE_STATS, + DRMCG_FTYPE_LIMIT, + DRMCG_FTYPE_DEFAULT, }; /** @@ -90,6 +92,8 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) drmcg->dev_resources[minor] = ddr; /* set defaults here */ + ddr->bo_limits_total_allocated = + dev->drmcg_props.bo_limits_total_allocated_default; return 0; } @@ -289,6 +293,38 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, } } +static void drmcg_print_limits(struct drmcg_device_resource *ddr, + struct seq_file *sf, enum drmcg_res_type type) +{ + if (ddr == NULL) { + seq_puts(sf, "\n"); + return; + } + + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + +static void drmcg_print_default(struct drmcg_props *props, + struct seq_file *sf, enum drmcg_res_type type) +{ + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + seq_printf(sf, "%lld\n", + props->bo_limits_total_allocated_default); + break; + default: + seq_puts(sf, "\n"); + break; + } +} + static int drmcg_seq_show_fn(int id, void *ptr, void *data) { struct drm_minor *minor = ptr; @@ -311,6 +347,12 @@ static int drmcg_seq_show_fn(int id, void *ptr, void *data) case DRMCG_FTYPE_STATS: drmcg_print_stats(ddr, sf, type); break; + case DRMCG_FTYPE_LIMIT: + drmcg_print_limits(ddr, sf, type); + break; + case DRMCG_FTYPE_DEFAULT: + drmcg_print_default(&minor->dev->drmcg_props, sf, type); + break; default: seq_puts(sf, "\n"); break; @@ -324,6 +366,130 @@ int drmcg_seq_show(struct seq_file *sf, void *v) return drm_minor_for_each(&drmcg_seq_show_fn, sf); } +static void drmcg_pr_cft_err(const struct drmcg *drmcg, + int rc, const char *cft_name, int minor) +{ + pr_err("drmcg: error parsing %s, minor %d, rc %d ", + cft_name, minor, rc); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); +} + +static int drmcg_process_limit_s64_val(char *sval, bool is_mem, + s64 def_val, s64 max_val, s64 *ret_val) +{ + int rc = strcmp("max", sval); + + + if (!rc) + *ret_val = max_val; + else { + rc = strcmp("default", sval); + + if (!rc) + *ret_val = def_val; + } + + if (rc) { + if (is_mem) { + *ret_val = memparse(sval, NULL); + rc = 0; + } else { + rc = kstrtoll(sval, 0, ret_val); + } + } + + if (*ret_val > max_val) + rc = -EINVAL; + + return rc; +} + +/** + * drmcg_limit_write - parse cgroup interface files to obtain user config + * + * Minimal value check to keep track of user intent. For example, user + * can specify limits greater than the values allowed by the parents. + * This way, the user configuration is kept and comes into effect if and + * when parents' limits are relaxed. + */ +static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + struct drmcg *drmcg = css_to_drmcg(of_css(of)); + enum drmcg_res_type type = + DRMCG_CTF_PRIV2RESTYPE(of_cft(of)->private); + char *cft_name = of_cft(of)->name; + char *limits = strstrip(buf); + struct drmcg_device_resource *ddr; + struct drmcg_props *props; + struct drm_minor *dm; + char *line; + char sattr[256]; + s64 val; + int rc; + int minor; + + while (limits != NULL) { + line = strsep(&limits, "\n"); + + if (sscanf(line, + __stringify(DRM_MAJOR)":%u %255[^\t\n]", + &minor, sattr) != 2) { + pr_err("drmcg: error parsing %s ", cft_name); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); + + continue; + } + + mutex_lock(&drmcg_mutex); + if (acquire_drm_minor) + dm = acquire_drm_minor(minor); + else + dm = NULL; + mutex_unlock(&drmcg_mutex); + + if (IS_ERR_OR_NULL(dm)) { + pr_err("drmcg: invalid minor %d for %s ", + minor, cft_name); + pr_cont_cgroup_name(drmcg->css.cgroup); + pr_cont("\n"); + + continue; + } + + mutex_lock(&dm->dev->drmcg_mutex); + ddr = drmcg->dev_resources[minor]; + props = &dm->dev->drmcg_props; + switch (type) { + case DRMCG_TYPE_BO_TOTAL: + rc = drmcg_process_limit_s64_val(sattr, true, + props->bo_limits_total_allocated_default, + S64_MAX, + &val); + + if (rc || val < 0) { + drmcg_pr_cft_err(drmcg, rc, cft_name, minor); + break; + } + + ddr->bo_limits_total_allocated = val; + break; + default: + break; + } + mutex_unlock(&dm->dev->drmcg_mutex); + + mutex_lock(&drmcg_mutex); + if (put_drm_dev) + put_drm_dev(dm->dev); /* release from acquire */ + mutex_unlock(&drmcg_mutex); + } + + return nbytes; +} + struct cftype files[] = { { .name = "buffer.total.stats", @@ -331,6 +497,20 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.total.default", + .seq_show = drmcg_seq_show, + .flags = CFTYPE_ONLY_ON_ROOT, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_DEFAULT), + }, + { + .name = "buffer.total.max", + .write = drmcg_limit_write, + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_TOTAL, + DRMCG_FTYPE_LIMIT), + }, { .name = "buffer.peak.stats", .seq_show = drmcg_seq_show, @@ -363,12 +543,16 @@ struct cgroup_subsys drm_cgrp_subsys = { */ void drmcg_device_early_init(struct drm_device *dev) { + dev->drmcg_props.limit_enforced = false; + + dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; + drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); /** - * drmcg_chg_bo_alloc - charge GEM buffer usage for a device and cgroup + * drmcg_try_chg_bo_alloc - charge GEM buffer usage for a device and cgroup * @drmcg: the DRM cgroup to be charged to * @dev: the device the usage should be charged to * @size: size of the GEM buffer to be accounted for @@ -377,29 +561,52 @@ EXPORT_SYMBOL(drmcg_device_early_init); * for the utilization. This should not be called when the buffer is shared ( * the GEM buffer's reference count being incremented.) */ -void drmcg_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, +bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, size_t size) { struct drmcg_device_resource *ddr; int devIdx = dev->primary->index; + struct drmcg_props *props = &dev->drmcg_props; + struct drmcg *drmcg_cur = drmcg; + bool result = true; + s64 delta = 0; if (drmcg == NULL) - return; + return true; mutex_lock(&dev->drmcg_mutex); - for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { - ddr = drmcg->dev_resources[devIdx]; + if (props->limit_enforced) { + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; + delta = ddr->bo_limits_total_allocated - + ddr->bo_stats_total_allocated; + + if (delta <= 0 || size > delta) { + result = false; + break; + } + } + } + + drmcg = drmcg_cur; + + if (result || !props->limit_enforced) { + for ( ; drmcg != NULL; drmcg = drmcg_parent(drmcg)) { + ddr = drmcg->dev_resources[devIdx]; - ddr->bo_stats_total_allocated += (s64)size; + ddr->bo_stats_total_allocated += (s64)size; - if (ddr->bo_stats_peak_allocated < (s64)size) - ddr->bo_stats_peak_allocated = (s64)size; + if (ddr->bo_stats_peak_allocated < (s64)size) + ddr->bo_stats_peak_allocated = (s64)size; - ddr->bo_stats_count_allocated++; + ddr->bo_stats_count_allocated++; + } } mutex_unlock(&dev->drmcg_mutex); + + return result; } -EXPORT_SYMBOL(drmcg_chg_bo_alloc); +EXPORT_SYMBOL(drmcg_try_chg_bo_alloc); /** * drmcg_unchg_bo_alloc - From patchwork Fri Feb 14 15:56:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382605 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 275FE1820 for ; Fri, 14 Feb 2020 15:57:33 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 058D32467B for ; Fri, 14 Feb 2020 15:57:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="So/sjdQG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 058D32467B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3A46F6F9F3; Fri, 14 Feb 2020 15:57:22 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2078.outbound.protection.outlook.com [40.107.244.78]) by gabe.freedesktop.org (Postfix) with ESMTPS id EC50F6F9F6; Fri, 14 Feb 2020 15:57:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f9Yp88VDC6oPRIUlltYLjqXabDu1ceOUuTE1ywHmbRYKkuUz/+ux/pChy5Gr54awfWg2XxOOHyUMpfRwoUujdJYU3dgWozZUdDcrII9BOXws2ZLeyz8MqEEABBJ0VTCXrJY0SOVA2+UUAjPOxC9A2gMCqBxenNFwVKGPH0vsUM/cTt1r6lKf96mWtPlNtINeRWcmJ18i2IQ4V7cSV9qesN0vXgm0JTtTfx0fJw1YEX0FV9p3RdOjUW8wCw8VSi7bEy3KYbMIH2air6k26uBhgOgDGcPfaBUuicZM6idUJYKtvZ5j4gtW9wKUAKjiL1oPJt+c9oYLktOAazdLwAQFKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fEzt0MlEkEuOUnc7BpB7xU8iofhUsMuYdD0QvPihCjM=; b=elktfAj+KjfW46tgiD8Z2QLYehbT/9rcDeC6kdG5tpUC2jH9TcAToGxa0ulmYG2x1Wq8o5e77EOe0VFYwSvRFFPUwx0u/bCy+axsGRgSDKipswiveJgthXRReDv01C5j5OMWDiG7mgDT+dRUJ7Lxx5Iqk+FzNBOnDpmTogWvzFxJtzmYRggcFjy0LRPw60l0FAZh9w/5C/Kmr2vdOAXof6PbhSk45aYMKOujRNeEvqgx4BVZI15LK2BMkOpItRmnAP2Gi16Tq//51PjwDRZ8nNLkCo40xdvd68YC98ovjeaqe6LIFMy4XREYX2LMOii7S+NpN59xier/hrYi8Z+5qA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fEzt0MlEkEuOUnc7BpB7xU8iofhUsMuYdD0QvPihCjM=; b=So/sjdQGnZckRJCyflSkqPJ8bF2RKHpWLvT0NBij1+VD4ELglC2H4E/gPvyxsXw1FHQ4hNWoTO7OkDJhIFX9Sg+L4mj7DKLErOgwrPvxVF0fpV9A4573cQDY8wvOBQkPlE3QeKiwucUJ5lxFA2HFAVe0gsDJ9PYJEGpDvkZ4oLM= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by BY5PR12MB3954.namprd12.prod.outlook.com (2603:10b6:a03:1af::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:17 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:16 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:16 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:15 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:14 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 08/11] drm, cgroup: Add peak GEM buffer allocation limit Date: Fri, 14 Feb 2020 10:56:47 -0500 Message-ID: <20200214155650.21203-9-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(39860400002)(136003)(346002)(428003)(199004)(189003)(26005)(186003)(81166006)(4326008)(7696005)(81156014)(5660300002)(316002)(86362001)(1076003)(426003)(70206006)(110136005)(36756003)(336012)(8936002)(70586007)(6666004)(2906002)(6636002)(478600001)(356004)(8676002)(2616005)(921003)(83996005)(1121003)(2101003); DIR:OUT; SFP:1101; SCL:1; SRVR:BY5PR12MB3954; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 17cb0a9f-4135-4bf7-97db-08d7b16690ac X-MS-TrafficTypeDiagnostic: BY5PR12MB3954: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TYIrwvkvoBPlK/3pZ8eL2qom+H94KgsFmo8nengnMeBjiqqnOW/bv7z6uk9hRA1BvUQMoAYMlztYkZJbwIwvNjpcyY0UkNRmo9kWkrgC6aT29V23S9l5D2kTuOTJoTLiNCtL8YUkodb7QCDsyKs5BnXoJa8SZ3o+2LOZzMyzl5yeHMTtS6qUWpGNGpkRPeha1/oUFyAmT0vSiPZKPtHeko1YI5UO9MtROAN5IVgLN1pbjGuTehUs2VkNYssuPNh0jSYKUYC7wSczKynUuismXHxIN/sIh4RoHUNePqz9tHtiOLuAbnCr+Hfvf/YfykEj2HCPjCQzIVKE39XGwqQL1LxwnAldGgSpJNsCHAsAlDvrU1dLRdaH466gGFHniDTQvdZl/DJawxcggYdt7c3RId0qIRbGhNnk8vv6YqhiiftjYaXZ5YYa7OuOWsynhQtNxPJC7/vmcocT7Kr1zIo1Owl84kQvVzQFifAiiWomUgjDfD++eLy0M4cLKuVF8T28yC7/eYLsNhRVTSVJmwjbEta5bAaJVQq1SkrNBZ1Iklg= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:16.7814 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 17cb0a9f-4135-4bf7-97db-08d7b16690ac X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3954 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drm.buffer.peak.default A read-only flat-keyed file which exists on the root cgroup. Each entry is keyed by the drm device's major:minor. Default limits on the largest GEM buffer allocation in bytes. drm.buffer.peak.max A read-write flat-keyed file which exists on all cgroups. Each entry is keyed by the drm device's major:minor. Per device limits on the largest GEM buffer allocation in bytes. This is a hard limit. Attempts in allocating beyond the cgroup limit will result in ENOMEM. Shorthand understood by memparse (such as k, m, g) can be used. Set largest allocation for /dev/dri/card1 to 4MB echo "226:1 4m" > drm.buffer.peak.max Change-Id: I5ab3fb4a442b6cbd5db346be595897c90217da69 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 18 +++++++++++ include/drm/drm_cgroup.h | 1 + include/linux/cgroup_drm.h | 1 + kernel/cgroup/drm.c | 43 +++++++++++++++++++++++++ 4 files changed, 63 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 064172df63e2..ce5dc027366a 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2102,6 +2102,24 @@ DRM Interface Files Set allocation limit for /dev/dri/card0 to 512MB echo "226:0 512m" > drm.buffer.total.max + drm.buffer.peak.default + A read-only flat-keyed file which exists on the root cgroup. + Each entry is keyed by the drm device's major:minor. + + Default limits on the largest GEM buffer allocation in bytes. + + drm.buffer.peak.max + A read-write flat-keyed file which exists on all cgroups. Each + entry is keyed by the drm device's major:minor. + + Per device limits on the largest GEM buffer allocation in bytes. + This is a hard limit. Attempts in allocating beyond the cgroup + limit will result in ENOMEM. Shorthand understood by memparse + (such as k, m, g) can be used. + + Set largest allocation for /dev/dri/card1 to 4MB + echo "226:1 4m" > drm.buffer.peak.max + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 2783e56690db..2b41d4d22e33 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -16,6 +16,7 @@ struct drmcg_props { bool limit_enforced; s64 bo_limits_total_allocated_default; + s64 bo_limits_peak_allocated_default; }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index b03d90623763..eae400f3d9b4 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -29,6 +29,7 @@ struct drmcg_device_resource { s64 bo_limits_total_allocated; s64 bo_stats_peak_allocated; + s64 bo_limits_peak_allocated; s64 bo_stats_count_allocated; }; diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index ee85482edd90..5fcbbc13fa1c 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -95,6 +95,9 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) ddr->bo_limits_total_allocated = dev->drmcg_props.bo_limits_total_allocated_default; + ddr->bo_limits_peak_allocated = + dev->drmcg_props.bo_limits_peak_allocated_default; + return 0; } @@ -305,6 +308,9 @@ static void drmcg_print_limits(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_TOTAL: seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", ddr->bo_limits_peak_allocated); + break; default: seq_puts(sf, "\n"); break; @@ -319,6 +325,10 @@ static void drmcg_print_default(struct drmcg_props *props, seq_printf(sf, "%lld\n", props->bo_limits_total_allocated_default); break; + case DRMCG_TYPE_BO_PEAK: + seq_printf(sf, "%lld\n", + props->bo_limits_peak_allocated_default); + break; default: seq_puts(sf, "\n"); break; @@ -476,6 +486,19 @@ static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, ddr->bo_limits_total_allocated = val; break; + case DRMCG_TYPE_BO_PEAK: + rc = drmcg_process_limit_s64_val(sattr, true, + props->bo_limits_peak_allocated_default, + S64_MAX, + &val); + + if (rc || val < 0) { + drmcg_pr_cft_err(drmcg, rc, cft_name, minor); + break; + } + + ddr->bo_limits_peak_allocated = val; + break; default: break; } @@ -517,6 +540,20 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, DRMCG_FTYPE_STATS), }, + { + .name = "buffer.peak.default", + .seq_show = drmcg_seq_show, + .flags = CFTYPE_ONLY_ON_ROOT, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_DEFAULT), + }, + { + .name = "buffer.peak.max", + .write = drmcg_limit_write, + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_PEAK, + DRMCG_FTYPE_LIMIT), + }, { .name = "buffer.count.stats", .seq_show = drmcg_seq_show, @@ -546,6 +583,7 @@ void drmcg_device_early_init(struct drm_device *dev) dev->drmcg_props.limit_enforced = false; dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; + dev->drmcg_props.bo_limits_peak_allocated_default = S64_MAX; drmcg_update_cg_tree(dev); } @@ -585,6 +623,11 @@ bool drmcg_try_chg_bo_alloc(struct drmcg *drmcg, struct drm_device *dev, result = false; break; } + + if (ddr->bo_limits_peak_allocated < size) { + result = false; + break; + } } } From patchwork Fri Feb 14 15:56:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382607 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA58C159A for ; Fri, 14 Feb 2020 15:57:35 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C7DDC24697 for ; Fri, 14 Feb 2020 15:57:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="GJbqEog+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7DDC24697 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EDE446F9F1; Fri, 14 Feb 2020 15:57:23 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2087.outbound.protection.outlook.com [40.107.93.87]) by gabe.freedesktop.org (Postfix) with ESMTPS id 80F1B6F9F6; Fri, 14 Feb 2020 15:57:20 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E1MK9CWLsXaKfKnklbWJY7f5SxyMoV9uThFds1vl7QF2hXHKumO+KiVVWDphFB6Sp15LECJbfueAGmVXTzmQ72bFpk3AhCwGtxDdDzCmNnhizhDz6jHTMGVGwMTFdyXoidcveeXdfZJF7+rooZKahB5kANp6UIlmN5y9fuxpZM0iqTjFB+ZjG6/2LUEVeBjMfoR9do4qBNAYYI/Twi6JroYlrsfbATbuovO4ds+q5diLclPCJrIH7xCCDclVMVAWfrjn8zH6I60Ddg4wCQgmi0Bfb9Nm/uNU/3F8zhoonzIGA41gbzZ/efrwXJmw4qUQnR4fdXACGE20jDc7sutMsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KLv/u15lZ9XpkHbBOGen0kbWuhUNiUYh6hxxozpkJkA=; b=NhCVgxIN2juxh0Ut0q/1xi2LgIgRB6RXpCcGCxz2SeV4nzZzlaaNubtIO5IBOlQGvXtUXOCKNrbWu9lkByGLeSqXNDZXTZkvCSIXmORVBaNWo1191Lfwm+ecOUWPBZjX1a0c8fGAFavmHeLzj4mUqNXI04SE7xB9nYKMxFhybr1WdSMlg0IWYDTeElim/NRmiSDp2TrLd6M5YBlXlqoEb0gfDbpCfYm4v0ifjAKhS9p0OiMX273mvSblZZ2yx3UlaNltklEQvUg6EsmC6VyM+y7tDu+9o5yFFZrP+EWI1CLVCWyJoc49+2YhWTQBVSrCXY6xd9epviu+r2XYLXVmYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KLv/u15lZ9XpkHbBOGen0kbWuhUNiUYh6hxxozpkJkA=; b=GJbqEog+gKWYrhcPDPaxc6xgwPaohE/K+eH+rdNLmRXaY2wxnbs8QN4t/oYQjg68LGn1kAEVMfpyAfmqyWQx/07YNx7M3e1TTMdZkor0vGrp26kJPLoUZEO0RBzBuYf9Rb/QJoqY/LqiVtxI3+luThQHxCDclHa7se7sXKrw0AU= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by BY5PR12MB3697.namprd12.prod.outlook.com (2603:10b6:a03:1a2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:17 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:17 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:17 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:16 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:15 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 09/11] drm, cgroup: Introduce lgpu as DRM cgroup resource Date: Fri, 14 Feb 2020 10:56:48 -0500 Message-ID: <20200214155650.21203-10-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(136003)(346002)(39860400002)(428003)(189003)(199004)(186003)(2906002)(26005)(86362001)(2616005)(1076003)(4326008)(6636002)(110136005)(316002)(36756003)(70206006)(70586007)(30864003)(426003)(7696005)(336012)(356004)(6666004)(8676002)(8936002)(5660300002)(81166006)(81156014)(478600001)(921003)(83996005)(2101003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:BY5PR12MB3697; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4fc46d2b-b6a3-4316-82c7-08d7b1669102 X-MS-TrafficTypeDiagnostic: BY5PR12MB3697: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3383; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5XnFPKQRDoyklldr+DAEP1uAhmiD+yJdWknvQmB6m9HcYUHV99HsLy28y7Xc96YkhGvIE26vTS/re6Fw+cHqbAU7gNeqkYp6y4HNXfRTtfVQTb3p3UNkATsjmgSR6sOsK9CIUn88qbBo0kh5wCl380DzFhtmKOp3gKM5+zyJ1eUCaGbRHIFhYHsoodSuoMvUuNJPbkm8PaJgGS75W+5xO1SRDWybyFjkB1guS/8/DT1IndJYb7pNgXnOUEwcU+dsu0wexl6aKVvXxanZb8zpS4qdDzkeDt1guij1eYdvsPUaZQKmtxRTHuLp8ExxQ2kfUUj9GXOR+mEEtE/jOXNKptJsT+IatPvqbk4SPCb6bFyhHDvrVnb58hDEHzedMPjDuoVkQakNaWgDde4HXi7DH5ARUqbTjfkUGNnhX7nzRAKAd/ax0YU+tmaebIDHrrDPEKBjaEZEKZhimxAi6f4BiKu7Irzy0C95S7ODVJJwTkze3IRQAjONhYS14sNAo7YnamuU9wLNMfekx3mvpjTD7OcOBsVdXgb5d+6rvLqBpv0= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:17.3438 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4fc46d2b-b6a3-4316-82c7-08d7b1669102 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3697 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" drm.lgpu A read-write nested-keyed file which exists on all cgroups. Each entry is keyed by the DRM device's major:minor. lgpu stands for logical GPU, it is an abstraction used to subdivide a physical DRM device for the purpose of resource management. This file stores user configuration while the drm.lgpu.effective reflects the actual allocation after considering the relationship between the cgroups and their configurations. The lgpu is a discrete quantity that is device specific (i.e. some DRM devices may have 64 lgpus while others may have 100 lgpus.) The lgpu is a single quantity that can be allocated in three different ways denoted by the following nested keys. ===== ============================================== weight Allocate by proportion in relationship with active sibling cgroups count Allocate by amount statically, treat lgpu as anonymous resources list Allocate statically, treat lgpu as named resource ===== ============================================== For example: 226:0 weight=100 count=256 list=0-255 226:1 weight=100 count=4 list=0,2,4,6 226:2 weight=100 count=32 list=32-63 226:3 weight=100 count=0 list= 226:4 weight=500 count=0 list= lgpu is represented by a bitmap and uses the bitmap_parselist kernel function so the list key input format is a comma-separated list of decimal numbers and ranges. Consecutively set bits are shown as two hyphen-separated decimal numbers, the smallest and largest bit numbers set in the range. Optionally each range can be postfixed to denote that only parts of it should be set. The range will divided to groups of specific size. Syntax: range:used_size/group_size Example: 0-1023:2/256 ==> 0,1,256,257,512,513,768,769 The count key is the hamming weight / hweight of the bitmap. Weight, count and list accept the max and default keywords. Some DRM devices may only support lgpu as anonymous resources. In such case, the significance of the position of the set bits in list will be ignored. The weight quantity is only in effect when static allocation is not used (by setting count=0) for this cgroup. The weight quantity distributes lgpus that are not statically allocated by the siblings. For example, given siblings cgroupA, cgroupB and cgroupC for a DRM device that has 64 lgpus, if cgroupA occupies 0-63, no lgpu is available to be distributed by weight. Similarly, if cgroupA has list=0-31 and cgroupB has list=16-63, cgroupC will be starved if it tries to allocate by weight. On the other hand, if cgroupA has weight=100 count=0, cgroupB has list=16-47, and cgroupC has weight=100 count=0, then 32 lgpus are available to be distributed evenly between cgroupA and cgroupC. In drm.lgpu.effective, cgroupA will have list=0-15 and cgroupC will have list=48-63. This lgpu resource supports the 'allocation' and 'weight' resource distribution model. drm.lgpu.effective A read-only nested-keyed file which exists on all cgroups. Each entry is keyed by the DRM device's major:minor. lgpu stands for logical GPU, it is an abstraction used to subdivide a physical DRM device for the purpose of resource management. This file reflects the actual allocation after considering the relationship between the cgroups and their configurations in drm.lgpu. Change-Id: Idde0ef9a331fd67bb9c7eb8ef9978439e6452488 Signed-off-by: Kenny Ho --- Documentation/admin-guide/cgroup-v2.rst | 80 ++++++ include/drm/drm_cgroup.h | 3 + include/linux/cgroup_drm.h | 22 ++ kernel/cgroup/drm.c | 324 +++++++++++++++++++++++- 4 files changed, 427 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index ce5dc027366a..d8a41956e5c7 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2120,6 +2120,86 @@ DRM Interface Files Set largest allocation for /dev/dri/card1 to 4MB echo "226:1 4m" > drm.buffer.peak.max + drm.lgpu + A read-write nested-keyed file which exists on all cgroups. + Each entry is keyed by the DRM device's major:minor. + + lgpu stands for logical GPU, it is an abstraction used to + subdivide a physical DRM device for the purpose of resource + management. This file stores user configuration while the + drm.lgpu.effective reflects the actual allocation after + considering the relationship between the cgroups and their + configurations. + + The lgpu is a discrete quantity that is device specific (i.e. + some DRM devices may have 64 lgpus while others may have 100 + lgpus.) The lgpu is a single quantity that can be allocated + in three different ways denoted by the following nested keys. + + ===== ============================================== + weight Allocate by proportion in relationship with + active sibling cgroups + count Allocate by amount statically, treat lgpu as + anonymous resources + list Allocate statically, treat lgpu as named + resource + ===== ============================================== + + For example: + 226:0 weight=100 count=256 list=0-255 + 226:1 weight=100 count=4 list=0,2,4,6 + 226:2 weight=100 count=32 list=32-63 + 226:3 weight=100 count=0 list= + 226:4 weight=500 count=0 list= + + lgpu is represented by a bitmap and uses the bitmap_parselist + kernel function so the list key input format is a + comma-separated list of decimal numbers and ranges. + + Consecutively set bits are shown as two hyphen-separated decimal + numbers, the smallest and largest bit numbers set in the range. + Optionally each range can be postfixed to denote that only parts + of it should be set. The range will divided to groups of + specific size. + Syntax: range:used_size/group_size + Example: 0-1023:2/256 ==> 0,1,256,257,512,513,768,769 + + The count key is the hamming weight / hweight of the bitmap. + + Weight, count and list accept the max and default keywords. + + Some DRM devices may only support lgpu as anonymous resources. + In such case, the significance of the position of the set bits + in list will be ignored. + + The weight quantity is only in effect when static allocation + is not used (by setting count=0) for this cgroup. The weight + quantity distributes lgpus that are not statically allocated by + the siblings. For example, given siblings cgroupA, cgroupB and + cgroupC for a DRM device that has 64 lgpus, if cgroupA occupies + 0-63, no lgpu is available to be distributed by weight. + Similarly, if cgroupA has list=0-31 and cgroupB has list=16-63, + cgroupC will be starved if it tries to allocate by weight. + + On the other hand, if cgroupA has weight=100 count=0, cgroupB + has list=16-47, and cgroupC has weight=100 count=0, then 32 + lgpus are available to be distributed evenly between cgroupA + and cgroupC. In drm.lgpu.effective, cgroupA will have + list=0-15 and cgroupC will have list=48-63. + + This lgpu resource supports the 'allocation' and 'weight' + resource distribution model. + + drm.lgpu.effective + A read-only nested-keyed file which exists on all cgroups. + Each entry is keyed by the DRM device's major:minor. + + lgpu stands for logical GPU, it is an abstraction used to + subdivide a physical DRM device for the purpose of resource + management. This file reflects the actual allocation after + considering the relationship between the cgroups and their + configurations in drm.lgpu. + GEM Buffer Ownership ~~~~~~~~~~~~~~~~~~~~ diff --git a/include/drm/drm_cgroup.h b/include/drm/drm_cgroup.h index 2b41d4d22e33..619a110cc748 100644 --- a/include/drm/drm_cgroup.h +++ b/include/drm/drm_cgroup.h @@ -17,6 +17,9 @@ struct drmcg_props { s64 bo_limits_total_allocated_default; s64 bo_limits_peak_allocated_default; + + int lgpu_capacity; + DECLARE_BITMAP(lgpu_slots, MAX_DRMCG_LGPU_CAPACITY); }; void drmcg_bind(struct drm_minor (*(*acq_dm)(unsigned int minor_id)), diff --git a/include/linux/cgroup_drm.h b/include/linux/cgroup_drm.h index eae400f3d9b4..bb09704e7f71 100644 --- a/include/linux/cgroup_drm.h +++ b/include/linux/cgroup_drm.h @@ -11,10 +11,14 @@ /* limit defined per the way drm_minor_alloc operates */ #define MAX_DRM_DEV (64 * DRM_MINOR_RENDER) +#define MAX_DRMCG_LGPU_CAPACITY 256 + enum drmcg_res_type { DRMCG_TYPE_BO_TOTAL, DRMCG_TYPE_BO_PEAK, DRMCG_TYPE_BO_COUNT, + DRMCG_TYPE_LGPU, + DRMCG_TYPE_LGPU_EFF, __DRMCG_TYPE_LAST, }; @@ -32,6 +36,24 @@ struct drmcg_device_resource { s64 bo_limits_peak_allocated; s64 bo_stats_count_allocated; + + /** + * Logical GPU + * + * *_cfg are properties configured by users + * *_eff are the effective properties being applied to the hardware + * *_stg is used to calculate _eff before applying to _eff + * after considering the entire hierarchy + */ + DECLARE_BITMAP(lgpu_stg, MAX_DRMCG_LGPU_CAPACITY); + /* user configurations */ + s64 lgpu_weight_cfg; + DECLARE_BITMAP(lgpu_cfg, MAX_DRMCG_LGPU_CAPACITY); + /* effective lgpu for the cgroup after considering + * relationship with other cgroup + */ + s64 lgpu_count_eff; + DECLARE_BITMAP(lgpu_eff, MAX_DRMCG_LGPU_CAPACITY); }; /** diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index 5fcbbc13fa1c..a4e88a3704bb 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,10 @@ enum drmcg_file_type { DRMCG_FTYPE_DEFAULT, }; +#define LGPU_LIMITS_NAME_LIST "list" +#define LGPU_LIMITS_NAME_COUNT "count" +#define LGPU_LIMITS_NAME_WEIGHT "weight" + /** * drmcg_bind - Bind DRM subsystem to cgroup subsystem * @acq_dm: function pointer to the drm_minor_acquire function @@ -98,6 +103,13 @@ static inline int init_drmcg_single(struct drmcg *drmcg, struct drm_device *dev) ddr->bo_limits_peak_allocated = dev->drmcg_props.bo_limits_peak_allocated_default; + bitmap_copy(ddr->lgpu_cfg, dev->drmcg_props.lgpu_slots, + MAX_DRMCG_LGPU_CAPACITY); + bitmap_copy(ddr->lgpu_stg, dev->drmcg_props.lgpu_slots, + MAX_DRMCG_LGPU_CAPACITY); + + ddr->lgpu_weight_cfg = CGROUP_WEIGHT_DFL; + return 0; } @@ -121,6 +133,120 @@ static inline void drmcg_update_cg_tree(struct drm_device *dev) mutex_unlock(&cgroup_mutex); } +static void drmcg_calculate_effective_lgpu(struct drm_device *dev, + const unsigned long *free_static, + const unsigned long *free_weighted, + struct drmcg *parent_drmcg) +{ + int capacity = dev->drmcg_props.lgpu_capacity; + DECLARE_BITMAP(lgpu_unused, MAX_DRMCG_LGPU_CAPACITY); + DECLARE_BITMAP(lgpu_by_weight, MAX_DRMCG_LGPU_CAPACITY); + struct drmcg_device_resource *parent_ddr; + struct drmcg_device_resource *ddr; + int minor = dev->primary->index; + struct cgroup_subsys_state *pos; + struct drmcg *child; + s64 weight_sum = 0; + s64 unused; + + parent_ddr = parent_drmcg->dev_resources[minor]; + + if (bitmap_empty(parent_ddr->lgpu_cfg, capacity)) + /* no static cfg, use weight for calculating the effective */ + bitmap_copy(parent_ddr->lgpu_stg, free_weighted, capacity); + else + /* lgpu statically configured, use the overlap as effective */ + bitmap_and(parent_ddr->lgpu_stg, free_static, + parent_ddr->lgpu_cfg, capacity); + + /* calculate lgpu available for distribution by weight for children */ + bitmap_copy(lgpu_unused, parent_ddr->lgpu_stg, capacity); + css_for_each_child(pos, &parent_drmcg->css) { + child = css_to_drmcg(pos); + ddr = child->dev_resources[minor]; + + if (bitmap_empty(ddr->lgpu_cfg, capacity)) + /* no static allocation, participate in weight dist */ + weight_sum += ddr->lgpu_weight_cfg; + else + /* take out statically allocated lgpu by siblings */ + bitmap_andnot(lgpu_unused, lgpu_unused, ddr->lgpu_cfg, + capacity); + } + + unused = bitmap_weight(lgpu_unused, capacity); + + css_for_each_child(pos, &parent_drmcg->css) { + child = css_to_drmcg(pos); + ddr = child->dev_resources[minor]; + + bitmap_zero(lgpu_by_weight, capacity); + /* no static allocation, participate in weight distribution */ + if (bitmap_empty(ddr->lgpu_cfg, capacity)) { + int c; + int p = 0; + + for (c = ddr->lgpu_weight_cfg * unused / weight_sum; + c > 0; c--) { + p = find_next_bit(lgpu_unused, capacity, p); + if (p < capacity) { + clear_bit(p, lgpu_unused); + set_bit(p, lgpu_by_weight); + } + } + + } + + drmcg_calculate_effective_lgpu(dev, parent_ddr->lgpu_stg, + lgpu_by_weight, child); + } +} + +static void drmcg_apply_effective_lgpu(struct drm_device *dev) +{ + int capacity = dev->drmcg_props.lgpu_capacity; + int minor = dev->primary->index; + struct drmcg_device_resource *ddr; + struct cgroup_subsys_state *pos; + struct drmcg *drmcg; + + if (root_drmcg == NULL) { + WARN_ON(root_drmcg == NULL); + return; + } + + rcu_read_lock(); + + /* process the entire cgroup tree from root to simplify the algorithm */ + drmcg_calculate_effective_lgpu(dev, dev->drmcg_props.lgpu_slots, + dev->drmcg_props.lgpu_slots, root_drmcg); + + /* apply changes to effective only if there is a change */ + css_for_each_descendant_pre(pos, &root_drmcg->css) { + drmcg = css_to_drmcg(pos); + ddr = drmcg->dev_resources[minor]; + + if (!bitmap_equal(ddr->lgpu_stg, ddr->lgpu_eff, capacity)) { + bitmap_copy(ddr->lgpu_eff, ddr->lgpu_stg, capacity); + ddr->lgpu_count_eff = + bitmap_weight(ddr->lgpu_eff, capacity); + } + } + rcu_read_unlock(); +} + +static void drmcg_apply_effective(enum drmcg_res_type type, + struct drm_device *dev, struct drmcg *changed_drmcg) +{ + switch (type) { + case DRMCG_TYPE_LGPU: + drmcg_apply_effective_lgpu(dev); + break; + default: + break; + } +} + /** * drmcg_register_dev - register a DRM device for usage in drm cgroup * @dev: DRM device @@ -143,7 +269,13 @@ void drmcg_register_dev(struct drm_device *dev) { dev->driver->drmcg_custom_init(dev, &dev->drmcg_props); + WARN_ON(dev->drmcg_props.lgpu_capacity != + bitmap_weight(dev->drmcg_props.lgpu_slots, + MAX_DRMCG_LGPU_CAPACITY)); + drmcg_update_cg_tree(dev); + + drmcg_apply_effective(DRMCG_TYPE_LGPU, dev, root_drmcg); } mutex_unlock(&drmcg_mutex); } @@ -297,7 +429,8 @@ static void drmcg_print_stats(struct drmcg_device_resource *ddr, } static void drmcg_print_limits(struct drmcg_device_resource *ddr, - struct seq_file *sf, enum drmcg_res_type type) + struct seq_file *sf, enum drmcg_res_type type, + struct drm_device *dev) { if (ddr == NULL) { seq_puts(sf, "\n"); @@ -311,6 +444,25 @@ static void drmcg_print_limits(struct drmcg_device_resource *ddr, case DRMCG_TYPE_BO_PEAK: seq_printf(sf, "%lld\n", ddr->bo_limits_peak_allocated); break; + case DRMCG_TYPE_LGPU: + seq_printf(sf, "%s=%lld %s=%d %s=%*pbl\n", + LGPU_LIMITS_NAME_WEIGHT, + ddr->lgpu_weight_cfg, + LGPU_LIMITS_NAME_COUNT, + bitmap_weight(ddr->lgpu_cfg, + dev->drmcg_props.lgpu_capacity), + LGPU_LIMITS_NAME_LIST, + dev->drmcg_props.lgpu_capacity, + ddr->lgpu_cfg); + break; + case DRMCG_TYPE_LGPU_EFF: + seq_printf(sf, "%s=%lld %s=%*pbl\n", + LGPU_LIMITS_NAME_COUNT, + ddr->lgpu_count_eff, + LGPU_LIMITS_NAME_LIST, + dev->drmcg_props.lgpu_capacity, + ddr->lgpu_eff); + break; default: seq_puts(sf, "\n"); break; @@ -329,6 +481,17 @@ static void drmcg_print_default(struct drmcg_props *props, seq_printf(sf, "%lld\n", props->bo_limits_peak_allocated_default); break; + case DRMCG_TYPE_LGPU: + seq_printf(sf, "%s=%d %s=%d %s=%*pbl\n", + LGPU_LIMITS_NAME_WEIGHT, + CGROUP_WEIGHT_DFL, + LGPU_LIMITS_NAME_COUNT, + bitmap_weight(props->lgpu_slots, + props->lgpu_capacity), + LGPU_LIMITS_NAME_LIST, + props->lgpu_capacity, + props->lgpu_slots); + break; default: seq_puts(sf, "\n"); break; @@ -358,7 +521,7 @@ static int drmcg_seq_show_fn(int id, void *ptr, void *data) drmcg_print_stats(ddr, sf, type); break; case DRMCG_FTYPE_LIMIT: - drmcg_print_limits(ddr, sf, type); + drmcg_print_limits(ddr, sf, type, minor->dev); break; case DRMCG_FTYPE_DEFAULT: drmcg_print_default(&minor->dev->drmcg_props, sf, type); @@ -415,6 +578,115 @@ static int drmcg_process_limit_s64_val(char *sval, bool is_mem, return rc; } +static void drmcg_nested_limit_parse(struct kernfs_open_file *of, + struct drm_device *dev, char *attrs) +{ + DECLARE_BITMAP(tmp_bitmap, MAX_DRMCG_LGPU_CAPACITY); + DECLARE_BITMAP(chk_bitmap, MAX_DRMCG_LGPU_CAPACITY); + enum drmcg_res_type type = + DRMCG_CTF_PRIV2RESTYPE(of_cft(of)->private); + struct drmcg *drmcg = css_to_drmcg(of_css(of)); + struct drmcg_props *props = &dev->drmcg_props; + char *cft_name = of_cft(of)->name; + int minor = dev->primary->index; + char *nested = strstrip(attrs); + struct drmcg_device_resource *ddr = + drmcg->dev_resources[minor]; + char *attr; + char sname[256]; + char sval[256]; + s64 val; + int rc; + + while (nested != NULL) { + attr = strsep(&nested, " "); + + if (sscanf(attr, "%255[^=]=%255[^=]", sname, sval) != 2) + continue; + + switch (type) { + case DRMCG_TYPE_LGPU: + if (strncmp(sname, LGPU_LIMITS_NAME_LIST, 256) && + strncmp(sname, LGPU_LIMITS_NAME_COUNT, 256) && + strncmp(sname, LGPU_LIMITS_NAME_WEIGHT, 256)) + continue; + + if (strncmp(sname, LGPU_LIMITS_NAME_WEIGHT, 256) && + (!strcmp("max", sval) || + !strcmp("default", sval))) { + bitmap_copy(ddr->lgpu_cfg, props->lgpu_slots, + props->lgpu_capacity); + + continue; + } + + if (strncmp(sname, LGPU_LIMITS_NAME_WEIGHT, 256) == 0) { + rc = drmcg_process_limit_s64_val(sval, + false, CGROUP_WEIGHT_DFL, + CGROUP_WEIGHT_MAX, &val); + + if (rc || val < CGROUP_WEIGHT_MIN || + val > CGROUP_WEIGHT_MAX) { + drmcg_pr_cft_err(drmcg, rc, cft_name, + minor); + continue; + } + + ddr->lgpu_weight_cfg = val; + continue; + } + + if (strncmp(sname, LGPU_LIMITS_NAME_COUNT, 256) == 0) { + rc = drmcg_process_limit_s64_val(sval, + false, props->lgpu_capacity, + props->lgpu_capacity, &val); + + if (rc || val < 0) { + drmcg_pr_cft_err(drmcg, rc, cft_name, + minor); + continue; + } + + bitmap_zero(tmp_bitmap, + MAX_DRMCG_LGPU_CAPACITY); + bitmap_set(tmp_bitmap, 0, val); + } + + if (strncmp(sname, LGPU_LIMITS_NAME_LIST, 256) == 0) { + rc = bitmap_parselist(sval, tmp_bitmap, + MAX_DRMCG_LGPU_CAPACITY); + + if (rc) { + drmcg_pr_cft_err(drmcg, rc, cft_name, + minor); + continue; + } + + bitmap_andnot(chk_bitmap, tmp_bitmap, + props->lgpu_slots, + MAX_DRMCG_LGPU_CAPACITY); + + /* user setting does not intersect with + * available lgpu */ + if (!bitmap_empty(chk_bitmap, + MAX_DRMCG_LGPU_CAPACITY)) { + drmcg_pr_cft_err(drmcg, 0, cft_name, + minor); + continue; + } + } + + bitmap_copy(ddr->lgpu_cfg, tmp_bitmap, + props->lgpu_capacity); + + break; /* DRMCG_TYPE_LGPU */ + default: + break; + } /* switch (type) */ + } +} + + /** * drmcg_limit_write - parse cgroup interface files to obtain user config * @@ -499,9 +771,15 @@ static ssize_t drmcg_limit_write(struct kernfs_open_file *of, char *buf, ddr->bo_limits_peak_allocated = val; break; + case DRMCG_TYPE_LGPU: + drmcg_nested_limit_parse(of, dm->dev, sattr); + break; default: break; } + + drmcg_apply_effective(type, dm->dev, drmcg); + mutex_unlock(&dm->dev->drmcg_mutex); mutex_lock(&drmcg_mutex); @@ -560,12 +838,51 @@ struct cftype files[] = { .private = DRMCG_CTF_PRIV(DRMCG_TYPE_BO_COUNT, DRMCG_FTYPE_STATS), }, + { + .name = "lgpu", + .seq_show = drmcg_seq_show, + .write = drmcg_limit_write, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_LGPU, + DRMCG_FTYPE_LIMIT), + }, + { + .name = "lgpu.default", + .seq_show = drmcg_seq_show, + .flags = CFTYPE_ONLY_ON_ROOT, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_LGPU, + DRMCG_FTYPE_DEFAULT), + }, + { + .name = "lgpu.effective", + .seq_show = drmcg_seq_show, + .private = DRMCG_CTF_PRIV(DRMCG_TYPE_LGPU_EFF, + DRMCG_FTYPE_LIMIT), + }, { } /* terminate */ }; +static int drmcg_online_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct drmcg *drmcg = data; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + drmcg_apply_effective(DRMCG_TYPE_LGPU, minor->dev, drmcg); + + return 0; +} + +static int drmcg_css_online(struct cgroup_subsys_state *css) +{ + return drm_minor_for_each(&drmcg_online_fn, css_to_drmcg(css)); +} + struct cgroup_subsys drm_cgrp_subsys = { .css_alloc = drmcg_css_alloc, .css_free = drmcg_css_free, + .css_online = drmcg_css_online, .early_init = false, .legacy_cftypes = files, .dfl_cftypes = files, @@ -585,6 +902,9 @@ void drmcg_device_early_init(struct drm_device *dev) dev->drmcg_props.bo_limits_total_allocated_default = S64_MAX; dev->drmcg_props.bo_limits_peak_allocated_default = S64_MAX; + dev->drmcg_props.lgpu_capacity = MAX_DRMCG_LGPU_CAPACITY; + bitmap_fill(dev->drmcg_props.lgpu_slots, MAX_DRMCG_LGPU_CAPACITY); + drmcg_update_cg_tree(dev); } EXPORT_SYMBOL(drmcg_device_early_init); From patchwork Fri Feb 14 15:56:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382609 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 666DC159A for ; Fri, 14 Feb 2020 15:57:37 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 444F824676 for ; Fri, 14 Feb 2020 15:57:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="dFb/Wyln" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 444F824676 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C2E456F9F6; Fri, 14 Feb 2020 15:57:25 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on20629.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eae::629]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4DEB86F9FB; Fri, 14 Feb 2020 15:57:21 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XpseHFhxO1itLqJSaF6NAytTgtp2WA+RDBx8SNwmWdNSsd/aqezvqLD8pbqOVXVA0tWrwGcdBAF/B7YVCvd5zruTjJ6eYSwW95RQKgD/te4ipgGc+nIQrC23oLu8ja18jk0PG7Av9fMB2SlEssqQuD1i+1F+u02lq3dss8nJZurisuz+4R3nRNYKP7v9sWX4K7vzj5HgMmm0eZbBSEdHomoYxIwb3NX0NOfjJ5iLIb0zNnddPLXqI61ObLCe4lAgeA4macvJ4tcvSE9Z03UDl5Ud4z/zuw2FyRw6orHn+iMslo6+7ErrXBQRsU1Qedp8ViBDG/q/7n0xiOh8+VC8Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCod5hSt1K1BbXJXcf/RSS+TJFsOmI14u7cdB2jdZBE=; b=VV3ccTX55zlnV+efqxeGuzJzbUCq8MtcfG1IukoDDZhBMlIr1uCTctpNBItiOZdL/OK/Zr3M4c6+4PJPHBgbulpGV1zjynRGBtLzffzq/1u/j8pKSfFFXb104feBsD1ywheXmxBSdsdz5nrfvNwlox787xwx6yGoo4H8TFbVfTAfi4S/ctZGupCFTuhTZr7s6lQok/QcTqsF76xBqaIYsub/ihEf1Xh2uAPlVpktM8ZSSxL9ZdIa4SB2WbhKfo6fbLBclTsBpPDYUDbsOTkkLmE0Z2ra/g1KGSd/JF8p6TO5vvKqJdM4fvQdKvIHcsf887DHFsaBmK1W/6VjTd31FQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCod5hSt1K1BbXJXcf/RSS+TJFsOmI14u7cdB2jdZBE=; b=dFb/Wyln4S2Lzy7ma7V6qQOMWEdmNWNOvtUdLJ8UxOTWqUL6rBYoGK2iy4v3swBGGGKedfu+dhSpvKRB4km+RbU2zqyaSIrKWnasLnZaw0rmz6OabnCNAAwnBrf5CVeJdSKsCaZSJ5y23uEYLcTp4Kwq8A7Tx8MGIwJLyH0zAIQ= Received: from SN1PR12CA0078.namprd12.prod.outlook.com (2603:10b6:802:20::49) by BN8PR12MB3426.namprd12.prod.outlook.com (2603:10b6:408:4a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:18 +0000 Received: from BN8NAM11FT029.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::208) by SN1PR12CA0078.outlook.office365.com (2603:10b6:802:20::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:18 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT029.mail.protection.outlook.com (10.13.177.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:18 +0000 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:17 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:16 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 10/11] drm, cgroup: add update trigger after limit change Date: Fri, 14 Feb 2020 10:56:49 -0500 Message-ID: <20200214155650.21203-11-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(396003)(376002)(136003)(39860400002)(428003)(199004)(189003)(5660300002)(2906002)(8936002)(81166006)(36756003)(6636002)(478600001)(81156014)(8676002)(4326008)(316002)(426003)(86362001)(110136005)(15650500001)(7696005)(336012)(26005)(2616005)(186003)(6666004)(70586007)(70206006)(356004)(1076003)(921003)(2101003)(83996005)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:BN8PR12MB3426; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; A:1; MX:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 817dc613-b77d-41e1-9e7a-08d7b1669190 X-MS-TrafficTypeDiagnostic: BN8PR12MB3426: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1824; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ftod30y8DGzek4AiNCqdJA1l/AWBKIb7BO9tZpWrJxKcu0CNi+/qI4z3c7Q7ZKIebN/Ayi/CstgxirJrHgQF3YlTaf6fpXiXOBXJz4jxXx8MytkqyiyUT2dt+pieMIQknrSTJTO14kyjJiTqjdh26gINDYby2D6/s9qRKZNJQgb3iBV1NJQbrGUAjYpI8UnMIYLlcjis1Dm3JBPJra8kwRjEgEUKDg3AGovHJX5y31VTh9nLDKRNGKXtT3922gthYLygR95jIzomhUWsYw42ASC/O4L/jfV8hYWIZnzZfl84QgmS4HSmBB8MIgaMQdi05hSA30/3zlkHEB61NJUZZ52FBjFVWnpiGutujAErSO/a/E3xgVEzbpbwy1NODsDC7SDvJSBDAZTLY/W/LwUQ/8AxJpk9JkaB27p1PbbfXxH0eLuGtyJa7wCChaY8xk53HP8furtsU/i8M20xG+Ao/8wNmhOj3JvvocjKvScppvgbzW2Q91nefubIo5YMJDLwEHOOkJx956y0kYYuoRpJ8FVqULu/oc7kZhLyVmRcRjs= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:18.2758 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 817dc613-b77d-41e1-9e7a-08d7b1669190 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3426 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Before this commit, drmcg limits are updated but enforcement is delayed until the next time the driver check against the new limit. While this is sufficient for certain resources, a more proactive enforcement may be needed for other resources. Introducing an optional drmcg_limit_updated callback for the DRM drivers. When defined, it will be called in two scenarios: 1) When limits are updated for a particular cgroup, the callback will be triggered for each task in the updated cgroup. 2) When a task is migrated from one cgroup to another, the callback will be triggered for each resource type for the migrated task. Change-Id: I0ce7c4e5a04c31bd0f8d9853a383575d4bc9a3fa Signed-off-by: Kenny Ho --- include/drm/drm_drv.h | 10 ++++++++ kernel/cgroup/drm.c | 59 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index 1f65ac4d9bbf..e7333143e722 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -724,6 +724,16 @@ struct drm_driver { void (*drmcg_custom_init)(struct drm_device *dev, struct drmcg_props *props); + /** + * @drmcg_limit_updated + * + * Optional callback + */ + void (*drmcg_limit_updated)(struct drm_device *dev, + struct task_struct *task, + struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type); + /** * @gem_vm_ops: Driver private ops for this object * diff --git a/kernel/cgroup/drm.c b/kernel/cgroup/drm.c index a4e88a3704bb..d3fa23b71f5f 100644 --- a/kernel/cgroup/drm.c +++ b/kernel/cgroup/drm.c @@ -133,6 +133,26 @@ static inline void drmcg_update_cg_tree(struct drm_device *dev) mutex_unlock(&cgroup_mutex); } +static void drmcg_limit_updated(struct drm_device *dev, struct drmcg *drmcg, + enum drmcg_res_type res_type) +{ + struct drmcg_device_resource *ddr = + drmcg->dev_resources[dev->primary->index]; + struct css_task_iter it; + struct task_struct *task; + + if (dev->driver->drmcg_limit_updated == NULL) + return; + + css_task_iter_start(&drmcg->css.cgroup->self, + CSS_TASK_ITER_PROCS, &it); + while ((task = css_task_iter_next(&it))) { + dev->driver->drmcg_limit_updated(dev, task, + ddr, res_type); + } + css_task_iter_end(&it); +} + static void drmcg_calculate_effective_lgpu(struct drm_device *dev, const unsigned long *free_static, const unsigned long *free_weighted, @@ -230,6 +250,8 @@ static void drmcg_apply_effective_lgpu(struct drm_device *dev) bitmap_copy(ddr->lgpu_eff, ddr->lgpu_stg, capacity); ddr->lgpu_count_eff = bitmap_weight(ddr->lgpu_eff, capacity); + + drmcg_limit_updated(dev, drmcg, DRMCG_TYPE_LGPU); } } rcu_read_unlock(); @@ -686,7 +708,6 @@ static void drmcg_nested_limit_parse(struct kernfs_open_file *of, } } - /** * drmcg_limit_write - parse cgroup interface files to obtain user config * @@ -879,10 +900,46 @@ static int drmcg_css_online(struct cgroup_subsys_state *css) return drm_minor_for_each(&drmcg_online_fn, css_to_drmcg(css)); } +static int drmcg_attach_fn(int id, void *ptr, void *data) +{ + struct drm_minor *minor = ptr; + struct task_struct *task = data; + struct drm_device *dev; + + if (minor->type != DRM_MINOR_PRIMARY) + return 0; + + dev = minor->dev; + + if (dev->driver->drmcg_limit_updated) { + struct drmcg *drmcg = drmcg_get(task); + struct drmcg_device_resource *ddr = + drmcg->dev_resources[minor->index]; + enum drmcg_res_type type; + + for (type = 0; type < __DRMCG_TYPE_LAST; type++) + dev->driver->drmcg_limit_updated(dev, task, ddr, type); + + drmcg_put(drmcg); + } + + return 0; +} + +static void drmcg_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) + drm_minor_for_each(&drmcg_attach_fn, task); +} + struct cgroup_subsys drm_cgrp_subsys = { .css_alloc = drmcg_css_alloc, .css_free = drmcg_css_free, .css_online = drmcg_css_online, + .attach = drmcg_attach, .early_init = false, .legacy_cftypes = files, .dfl_cftypes = files, From patchwork Fri Feb 14 15:56:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho, Kenny" X-Patchwork-Id: 11382611 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC38B159A for ; Fri, 14 Feb 2020 15:57:38 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9A39F24682 for ; Fri, 14 Feb 2020 15:57:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="XVkGieX5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9A39F24682 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B4C586F9FB; Fri, 14 Feb 2020 15:57:26 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-eopbgr760041.outbound.protection.outlook.com [40.107.76.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id BD6266FA01; Fri, 14 Feb 2020 15:57:22 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eIIvsVPPpVI6tWvUMrID6jEca77e1p3kTxs9egltLZFgcznx0nFZFz//OhdkUdqPzAQ30Br45ytpyDyN8LWkMQhvTGDF8xOse0Q863rdK2+lUIMJUo89fW1ALCpPhfJ6VFWtXyxiQobfwJZgLILQ9JFRnFOErYGuJ7hBl3PvWtKd1wNohJmRPceUHNg/EnHZ9fVC69MRwcxHv+e0mkHYlRYaMdHLUcSmulvsbp1bdQEc5OiAPFGAG+OEEhEelwmdy5oG4VSWEA65fd/H/td0HcRvnh/YpbgrzXitV5ZhUpAmms+zQtBB4muwS1yOIK0KMB08wBzxFZ8RYVhg5hoH3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3rRr9SDapif3gUzcHJzuaeIB3YgcZXIrBlrTW3u0h8A=; b=IN88N1EsUgcyKYZ92j4e+BbyjmKW2KmYdemrsLxjPEuqpByPvRITbVMrRDQTamlO+WKbtED8jCYkne2F2HASL5BSY2mVovKUVk0yxBlBaZTaHVYJUgPGw7zU4rdl78usgBcIC4dBn9rx1iWnT++9jx33/9LFo+hje1DZc6bX9DRfx+VKUAKZ3qXrnVTPbUBn6vNsqK9GKMl/sTjko+N+sOnHIZ1bHma3fmwUgb/dqYJ6BnMP3bSStY7LfPTYii68Ko9ckvpmRcnh7RkY8mmbbYUXwpnwnIRpblCCsl7ZwSAvNpbcs02D+56sZf/k+b6lMcawrjN6Gebf/iar5PVNdg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3rRr9SDapif3gUzcHJzuaeIB3YgcZXIrBlrTW3u0h8A=; b=XVkGieX5pf421kKdmQTXpyFO7Mrk446+MHJT+dr4T9nACLQ30IQjZcgz7+hjXUj1V16iRVtqfMvysYVAAOx5gu4lZT50NEw1B5jdfPrd+rzDG6TBx7jsIjEAP0p0/EFrF+5iKrWR9VaG+WWAzFRD47rvbTlPUTLOGsr/6LTBoXI= Received: from CY4PR1201CA0004.namprd12.prod.outlook.com (2603:10b6:910:16::14) by DM5PR1201MB0059.namprd12.prod.outlook.com (2603:10b6:4:54::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.25; Fri, 14 Feb 2020 15:57:19 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eae::201) by CY4PR1201CA0004.outlook.office365.com (2603:10b6:910:16::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.24 via Frontend Transport; Fri, 14 Feb 2020 15:57:19 +0000 Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB01.amd.com (165.204.84.17) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2729.22 via Frontend Transport; Fri, 14 Feb 2020 15:57:19 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:19 -0600 Received: from SATLEXMB02.amd.com (10.181.40.143) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 14 Feb 2020 09:57:18 -0600 Received: from yuho-zbook.amd.com (10.180.168.240) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Fri, 14 Feb 2020 09:57:17 -0600 From: Kenny Ho To: , , , , , , , , , , , , , , Subject: [PATCH 11/11] drm/amdgpu: Integrate with DRM cgroup Date: Fri, 14 Feb 2020 10:56:50 -0500 Message-ID: <20200214155650.21203-12-Kenny.Ho@amd.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200214155650.21203-1-Kenny.Ho@amd.com> References: <20200214155650.21203-1-Kenny.Ho@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(136003)(396003)(376002)(428003)(189003)(199004)(86362001)(81156014)(81166006)(4326008)(26005)(8676002)(5660300002)(186003)(478600001)(426003)(6636002)(8936002)(1076003)(316002)(2616005)(6666004)(110136005)(36756003)(70586007)(336012)(7696005)(70206006)(2906002)(356004)(921003)(1121003)(2101003)(83996005); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR1201MB0059; H:SATLEXMB01.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 892e099d-0f6f-4533-76ef-08d7b166924c X-MS-TrafficTypeDiagnostic: DM5PR1201MB0059: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-Forefront-PRVS: 03137AC81E X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kW4hNTcF366MKHSEB8dN+9P+qrp3ekradTTbRiMXmuo1hRldzMaiKfhaUGkddL3S9KvHTEan1vWfrgmp+Fu0n173dh7+R8+Faibm2zrZNXJpcTvMSIT9fHCDeyU7XHF8Zo+kobwuhgJLbqDF3JtL7/X2mSmeEGUv5W7VJgALf6udFPvHrUTT6aLCrVSDVulzYIc0BebLew2dEwy2xf6rsoF43lmWWY6+mx4uzIrRBJkOxRKwKXDMpDtEOowtfQtu8wJUoM0cgjo3B/43+UN65Sj5D0/ndIJgiwtAWPaYITOLxwsk/a6H6TN72CJ2uMamkTuRA1eqP02KPneeS325BIkbUueG/tsIGTNfUSpxqgQe82QI7HfUCFG8k92/xmF312xIfrk1F3NptV9yOS6ffUQiQ+3qi+Ds7prDbTcnO3gVOvnyRvzwYD/J9NOqCzktnSCSxPCVJwd3mC8Y406PocECAQGiFXlvu9K1alTg++LX/PJfmT1x7lulqSdptzrTl9G8/lIjiZwhm+2uF7mTTPAZfNx1J4/gCIj2uIjk3to= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2020 15:57:19.5049 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 892e099d-0f6f-4533-76ef-08d7b166924c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB01.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0059 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kenny Ho Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The number of logical gpu (lgpu) is defined to be the number of compute unit (CU) for a device. The lgpu allocation limit only applies to compute workload for the moment (enforced via kfd queue creation.) Any cu_mask update is validated against the availability of the compute unit as defined by the drmcg the kfd process belongs to. Change-Id: I2930e76ef9ac6d36d0feb81f604c89a4208e6614 Signed-off-by: Kenny Ho --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 29 ++++ drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 + .../amd/amdkfd/kfd_process_queue_manager.c | 153 ++++++++++++++++++ 5 files changed, 195 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h index 47b0f2957d1f..a45c7b5d23b1 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -198,6 +198,10 @@ uint8_t amdgpu_amdkfd_get_xgmi_hops_count(struct kgd_dev *dst, struct kgd_dev *s valid; \ }) +int amdgpu_amdkfd_update_cu_mask_for_process(struct task_struct *task, + struct amdgpu_device *adev, unsigned long *lgpu_bitmap, + unsigned int nbits); + /* GPUVM API */ int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, unsigned int pasid, void **vm, void **process_info, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 3ebef1d62346..dc31b9af2c72 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -1402,9 +1402,31 @@ amdgpu_get_crtc_scanout_position(struct drm_device *dev, unsigned int pipe, static void amdgpu_drmcg_custom_init(struct drm_device *dev, struct drmcg_props *props) { + struct amdgpu_device *adev = dev->dev_private; + + props->lgpu_capacity = adev->gfx.cu_info.number; + bitmap_zero(props->lgpu_slots, MAX_DRMCG_LGPU_CAPACITY); + bitmap_fill(props->lgpu_slots, props->lgpu_capacity); + props->limit_enforced = true; } +static void amdgpu_drmcg_limit_updated(struct drm_device *dev, + struct task_struct *task, struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type) +{ + struct amdgpu_device *adev = dev->dev_private; + + switch (res_type) { + case DRMCG_TYPE_LGPU: + amdgpu_amdkfd_update_cu_mask_for_process(task, adev, + ddr->lgpu_eff, dev->drmcg_props.lgpu_capacity); + break; + default: + break; + } +} + #else static void amdgpu_drmcg_custom_init(struct drm_device *dev, @@ -1412,6 +1434,12 @@ static void amdgpu_drmcg_custom_init(struct drm_device *dev, { } +static void amdgpu_drmcg_limit_updated(struct drm_device *dev, + struct task_struct *task, struct drmcg_device_resource *ddr, + enum drmcg_res_type res_type) +{ +} + #endif /* CONFIG_CGROUP_DRM */ static struct drm_driver kms_driver = { @@ -1448,6 +1476,7 @@ static struct drm_driver kms_driver = { .gem_prime_mmap = amdgpu_gem_prime_mmap, .drmcg_custom_init = amdgpu_drmcg_custom_init, + .drmcg_limit_updated = amdgpu_drmcg_limit_updated, .name = DRIVER_NAME, .desc = DRIVER_DESC, diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 275f79ab0900..f39555c0f1d8 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -449,6 +449,12 @@ static int kfd_ioctl_set_cu_mask(struct file *filp, struct kfd_process *p, return -EFAULT; } + if (!pqm_drmcg_lgpu_validate(p, args->queue_id, properties.cu_mask, cu_mask_size)) { + pr_debug("CU mask not permitted by DRM Cgroup"); + kfree(properties.cu_mask); + return -EACCES; + } + mutex_lock(&p->mutex); retval = pqm_set_cu_mask(&p->pqm, args->queue_id, &properties); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index c0b0defc8f7a..9053b1b7fb10 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -921,6 +921,9 @@ int pqm_get_wave_state(struct process_queue_manager *pqm, u32 *ctl_stack_used_size, u32 *save_area_used_size); +bool pqm_drmcg_lgpu_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size); + int amdkfd_fence_wait_timeout(unsigned int *fence_addr, unsigned int fence_value, unsigned int timeout_ms); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index 8fa856e6a03f..ff71b208d320 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -23,9 +23,11 @@ #include #include +#include #include "kfd_device_queue_manager.h" #include "kfd_priv.h" #include "kfd_kernel_queue.h" +#include "amdgpu.h" #include "amdgpu_amdkfd.h" static inline struct process_queue_node *get_queue_by_qid( @@ -167,6 +169,7 @@ static int init_user_queue(struct process_queue_manager *pqm, struct queue_properties *q_properties, struct file *f, unsigned int qid) { + struct drmcg *drmcg; int retval; /* Doorbell initialized in user space*/ @@ -180,6 +183,37 @@ static int init_user_queue(struct process_queue_manager *pqm, if (retval != 0) return retval; +#ifdef CONFIG_CGROUP_DRM + drmcg = drmcg_get(pqm->process->lead_thread); + if (drmcg) { + struct amdgpu_device *adev; + struct drmcg_device_resource *ddr; + int mask_size; + u32 *mask; + + adev = (struct amdgpu_device *) dev->kgd; + + mask_size = adev->ddev->drmcg_props.lgpu_capacity; + mask = kzalloc(sizeof(u32) * round_up(mask_size, 32), + GFP_KERNEL); + + if (!mask) { + drmcg_put(drmcg); + uninit_queue(*q); + return -ENOMEM; + } + + ddr = drmcg->dev_resources[adev->ddev->primary->index]; + + bitmap_to_arr32(mask, ddr->lgpu_eff, mask_size); + + (*q)->properties.cu_mask_count = mask_size; + (*q)->properties.cu_mask = mask; + + drmcg_put(drmcg); + } +#endif /* CONFIG_CGROUP_DRM */ + (*q)->device = dev; (*q)->process = pqm->process; @@ -508,6 +542,125 @@ int pqm_get_wave_state(struct process_queue_manager *pqm, save_area_used_size); } +#ifdef CONFIG_CGROUP_DRM + +bool pqm_drmcg_lgpu_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size) +{ + DECLARE_BITMAP(curr_mask, MAX_DRMCG_LGPU_CAPACITY); + struct drmcg_device_resource *ddr; + struct process_queue_node *pqn; + struct amdgpu_device *adev; + struct drmcg *drmcg; + bool result; + + if (cu_mask_size > MAX_DRMCG_LGPU_CAPACITY) + return false; + + bitmap_from_arr32(curr_mask, cu_mask, cu_mask_size); + + pqn = get_queue_by_qid(&p->pqm, qid); + if (!pqn) + return false; + + adev = (struct amdgpu_device *)pqn->q->device->kgd; + + drmcg = drmcg_get(p->lead_thread); + ddr = drmcg->dev_resources[adev->ddev->primary->index]; + + if (bitmap_subset(curr_mask, ddr->lgpu_eff, + MAX_DRMCG_LGPU_CAPACITY)) + result = true; + else + result = false; + + drmcg_put(drmcg); + + return result; +} + +#else + +bool pqm_drmcg_lgpu_validate(struct kfd_process *p, int qid, u32 *cu_mask, + unsigned int cu_mask_size) +{ + return true; +} + +#endif /* CONFIG_CGROUP_DRM */ + +int amdgpu_amdkfd_update_cu_mask_for_process(struct task_struct *task, + struct amdgpu_device *adev, unsigned long *lgpu_bm, + unsigned int lgpu_bm_size) +{ + struct kfd_dev *kdev = adev->kfd.dev; + struct process_queue_node *pqn; + struct kfd_process *kfdproc; + size_t size_in_bytes; + u32 *cu_mask; + int rc = 0; + + if ((lgpu_bm_size % 32) != 0) { + pr_warn("lgpu_bm_size %d must be a multiple of 32", + lgpu_bm_size); + return -EINVAL; + } + + kfdproc = kfd_get_process(task); + + if (IS_ERR(kfdproc)) + return -ESRCH; + + size_in_bytes = sizeof(u32) * round_up(lgpu_bm_size, 32); + + mutex_lock(&kfdproc->mutex); + list_for_each_entry(pqn, &kfdproc->pqm.queues, process_queue_list) { + if (pqn->q && pqn->q->device == kdev) { + /* update cu_mask accordingly */ + cu_mask = kzalloc(size_in_bytes, GFP_KERNEL); + if (!cu_mask) { + rc = -ENOMEM; + break; + } + + if (pqn->q->properties.cu_mask) { + DECLARE_BITMAP(curr_mask, + MAX_DRMCG_LGPU_CAPACITY); + + if (pqn->q->properties.cu_mask_count > + lgpu_bm_size) { + rc = -EINVAL; + kfree(cu_mask); + break; + } + + bitmap_from_arr32(curr_mask, + pqn->q->properties.cu_mask, + pqn->q->properties.cu_mask_count); + + bitmap_and(curr_mask, curr_mask, lgpu_bm, + lgpu_bm_size); + + bitmap_to_arr32(cu_mask, curr_mask, + lgpu_bm_size); + + kfree(curr_mask); + } else + bitmap_to_arr32(cu_mask, lgpu_bm, + lgpu_bm_size); + + pqn->q->properties.cu_mask = cu_mask; + pqn->q->properties.cu_mask_count = lgpu_bm_size; + + rc = pqn->q->device->dqm->ops.update_queue( + pqn->q->device->dqm, pqn->q); + } + } + mutex_unlock(&kfdproc->mutex); + + return rc; +} + #if defined(CONFIG_DEBUG_FS) int pqm_debugfs_mqds(struct seq_file *m, void *data)