From patchwork Thu May 27 23:08:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A326EC4708A for ; Thu, 27 May 2021 23:09:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 43AF161176 for ; Thu, 27 May 2021 23:09:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43AF161176 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DB5B36B006E; Thu, 27 May 2021 19:09:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8C786B0070; Thu, 27 May 2021 19:09:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2EC26B0071; Thu, 27 May 2021 19:09:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id 8F8036B006E for ; Thu, 27 May 2021 19:09:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2B61DB9E3 for ; Thu, 27 May 2021 23:09:42 +0000 (UTC) X-FDA: 78188555004.10.512F519 Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) by imf17.hostedemail.com (Postfix) with ESMTP id 5C6214142C6C for ; Thu, 27 May 2021 23:09:36 +0000 (UTC) Received: by mail-il1-f181.google.com with SMTP id o9so1762397ilh.6 for ; Thu, 27 May 2021 16:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1vTFumJbcjlC9H58za7civP9oS0BxFYnQlOIDDd8vSU=; b=HAIg1uSz4RBzsZCtp4d4QdB9b8SdCfGZGGkYyjQJ1gpqJOsXAvP98PnBrf2z4CMtNU QzSbafdmDgr4kszEaSkqeJcgdmzAvYEqqA5a70bmnambeDgQ0u4GM/6R/nmp3dJKVRo3 0bYjKH8I1+oipkfzOoA8chQFIQBF4MCZYCzKAekcY98QcSsEjbLAaoqZmOtHzPgcAvih KhwHdE9VI6bNbwHYvNtiWHV/Fk4zWs4Rv9xiflD+lQBLlEgjrCBraoaU//SEG3Z84rEv uUoYXnEnBbhbG9dgYvBhC8qb51RDUrreN+AAApT6iYh9ivrPx97e76JBAO9+apO1gsEa F0ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1vTFumJbcjlC9H58za7civP9oS0BxFYnQlOIDDd8vSU=; b=pvla0biSbqvHX0Gx6iOdwyxiQJBDIDXJfiSindc1sShTGwsemEHyO/otLQ9iF5qcMk e5fxuUXVvCtBqUTB4BISuyZ8IqViTCm/Ehw3Kaem9E4bdIe7A3FUXPkHc/4dhVxMS6pG WYH7CP3Bn4aTs5Oru/bQucYJsG/NaL181AeIWTC3vNqHIkKp7C6RrfXY1g/P5KR3Kf1P 8RwDB7DOrQTuK+sSx+FmwDvNgOLLHqr++KvsZUx3XBaAtxC83eMaRF5iQF1q7YiaFrds ibv6h/Tm2gxWf8Xg72SCutC9wKF+m6wo3yzBgyeyerlbRF0OuBfMuqMiHka2Dt/pkQ5B Q0JA== X-Gm-Message-State: AOAM530eVywCCEZ63kw17s8T6DnxA/v1Ez88FknMdzbXP1F4JP0dqn5W TAIe1bYDEQN6rxR0yU+PPzM= X-Google-Smtp-Source: ABdhPJxGyk+/fitlH3T/Ywo0Dq9sm0GcM01SIGfE1kwmfZ4Tql9lUnIKyyNl5J+eQtri+AKvK9GjAg== X-Received: by 2002:a92:3302:: with SMTP id a2mr4869468ilf.62.1622156981342; Thu, 27 May 2021 16:09:41 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id r5sm1860014ilb.1.2021.05.27.16.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 16:09:32 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: felix.kuehling@amd.com, akpm@linux-foundation.org, linux-mm@kvack.org Cc: hch@lst.de, jglisse@redhat.com, jgg@nvidia.com, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Alex Sierra Subject: [RFC PATCH 1/5] drm/amdkfd: add SPM support for SVM Date: Thu, 27 May 2021 19:08:05 -0400 Message-Id: <20210527230809.3701-2-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527230809.3701-1-Felix.Kuehling@amd.com> References: <20210527230809.3701-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5C6214142C6C Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=HAIg1uSz; spf=pass (imf17.hostedemail.com: domain of felixkuehling@gmail.com designates 209.85.166.181 as permitted sender) smtp.mailfrom=felixkuehling@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Stat-Signature: by8uj5i4mwehs7kkn5x6ym15chh3bm3t X-HE-Tag: 1622156976-264893 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alex Sierra When CPU is connected throug XGMI, it has coherent access to VRAM resource. In this case that resource is taken from a table in the device gmc aperture base. This resource is used along with the device type, which could be DEVICE_PRIVATE or DEVICE_GENERIC to create the device page map region. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 12 +++++++++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 - kernel/resource.c | 2 +- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index c8ca3252cbc2..f5939449a99f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -895,6 +895,7 @@ int svm_migrate_init(struct amdgpu_device *adev) struct resource *res; unsigned long size; void *r; + bool xgmi_connected_to_cpu = adev->gmc.xgmi.connected_to_cpu; /* Page migration works on Vega10 or newer */ if (kfddev->device_info->asic_family < CHIP_VEGA10) @@ -907,17 +908,22 @@ int svm_migrate_init(struct amdgpu_device *adev) * should remove reserved size */ size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20); - res = devm_request_free_mem_region(adev->dev, &iomem_resource, size); + if (xgmi_connected_to_cpu) + res = lookup_resource(&iomem_resource, adev->gmc.aper_base); + else + res = devm_request_free_mem_region(adev->dev, &iomem_resource, size); + if (IS_ERR(res)) return -ENOMEM; - pgmap->type = MEMORY_DEVICE_PRIVATE; pgmap->nr_range = 1; pgmap->range.start = res->start; pgmap->range.end = res->end; + pgmap->type = xgmi_connected_to_cpu ? + MEMORY_DEVICE_GENERIC : MEMORY_DEVICE_PRIVATE; pgmap->ops = &svm_migrate_pgmap_ops; pgmap->owner = SVM_ADEV_PGMAP_OWNER(adev); - pgmap->flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; + pgmap->flags = 0; r = devm_memremap_pages(adev->dev, pgmap); if (IS_ERR(r)) { pr_err("failed to register HMM device memory\n"); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 21f693767a0d..3881a93192ed 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -38,7 +38,6 @@ #define SVM_RANGE_VRAM_DOMAIN (1UL << 0) #define SVM_ADEV_PGMAP_OWNER(adev)\ ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) - struct svm_range_bo { struct amdgpu_bo *bo; struct kref kref; diff --git a/kernel/resource.c b/kernel/resource.c index 627e61b0c124..da137553b83e 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -783,7 +783,7 @@ struct resource *lookup_resource(struct resource *root, resource_size_t start) return res; } - +EXPORT_SYMBOL(lookup_resource); /* * Insert a resource into the resource tree. If successful, return NULL, * otherwise return the conflicting resource (compare to __request_resource())