From patchwork Thu May 27 20:55:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 493EAC47089 for ; Thu, 27 May 2021 20:56:16 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F3CB9611C2 for ; Thu, 27 May 2021 20:56:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3CB9611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 33AED6F4CA; Thu, 27 May 2021 20:56:13 +0000 (UTC) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6C5566E05A; Thu, 27 May 2021 20:56:11 +0000 (UTC) Received: by mail-io1-xd36.google.com with SMTP id n10so1935593ion.8; Thu, 27 May 2021 13:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=jZFOLIektZDd6zr4VRxO6/vlH0qTTmDpJ33fKVwfdZA=; b=EwMAVzP9yCXmu55r7KvHJm0HHMP1B33uPh6wi+GEsnT/BsMy0pmEnMyGyYWbYY0O7I 1gmYQUU1Ve7FeemQBuJqSUT7G5+pNO0YGoq+e4uV0YdceHmiTu1NFemCOxx5YKBAn06N uvLTz+NTiH7kX/9x5gg/u5EyIIVMufaII0GR1SA7AlR2shEnXN3oBm0iqXHpoSlk5MqZ LfZlQF/0sDkEkAT0iDXr/of3wzhG7LxRJXwh/A9tw3sq2uKbVbMWNThTkERcH9NpyINp KJN2vBGtMbd/GyL+yFgj68g+pzP/q1GXefO5uZVb/icygDRmZ6RvAdZLYvYrOTRcTy/N E+2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=jZFOLIektZDd6zr4VRxO6/vlH0qTTmDpJ33fKVwfdZA=; b=jISO5RtGdNN91f8ogkOclH6qSOAZDQAzp6d9SyVVP/C8kCMc63D14o4rFQ1oMZH6+T 9iXuICHoF9yJe3ecoitEk8y8cis+sxUrNvGNk2M02JLUtn9510XunCPiRUnw28b8A3jd 0xWqDi3XKaUmqtFlpJhFAiwpyukzlg7/oowk7LUKsGjV/wZ5i1qKOOw1zIMfT0WihxlS b6Sbf21t+DuEWH/xO79Xuht2oPnh6jPjN6lknaDybsSILGxrd9b/RgWXV++6D/+7yCze 2fn/LrfJvYTeTx+pP7Eo50dqEZpHK8oC2lrZhBeU1cQbNNzmSCrI6uEzD0Zvh9jOlFX0 qRFg== X-Gm-Message-State: AOAM532LHlHPP3YEQmS6yAdM992WM/o415ind5BophOQS3D5ds2JPwnv UKhCrZN+QUaG58fNkc+GUJkhge8uPd60iQ== X-Google-Smtp-Source: ABdhPJz54Cvm4flXQKGTpP+thZ4Cs/ZM2pmLCXmYO1iOFxtz/69rb+BH0CrOAljgKkgSCxLV36rPQg== X-Received: by 2002:a6b:1454:: with SMTP id 81mr4458339iou.7.1622148970002; Thu, 27 May 2021 13:56:10 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:09 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 01/10] drm/amdkfd: device pgmap owner at the svm migrate init Date: Thu, 27 May 2021 16:55:57 -0400 Message-Id: <20210527205606.2660-1-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra pgmap owner member at the svm migrate init could be referenced to either adev or hive, depending on device topology. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index fd8f544f0de2..11f7f590c6ec 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -426,7 +426,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, migrate.start = start; migrate.end = end; migrate.flags = MIGRATE_VMA_SELECT_SYSTEM; - migrate.pgmap_owner = adev; + migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev); size = 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t); size *= npages; @@ -641,7 +641,7 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange, migrate.start = start; migrate.end = end; migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; - migrate.pgmap_owner = adev; + migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev); size = 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t); size *= npages; @@ -907,7 +907,7 @@ int svm_migrate_init(struct amdgpu_device *adev) pgmap->range.start = res->start; pgmap->range.end = res->end; pgmap->ops = &svm_migrate_pgmap_ops; - pgmap->owner = adev; + pgmap->owner = SVM_ADEV_PGMAP_OWNER(adev); pgmap->flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; r = devm_memremap_pages(adev->dev, pgmap); if (IS_ERR(r)) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 573f984b81fe..4297250f259d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -35,6 +35,9 @@ #include "amdgpu.h" #include "kfd_priv.h" +#define SVM_ADEV_PGMAP_OWNER(adev)\ + ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) + struct svm_range_bo { struct amdgpu_bo *bo; struct kref kref; From patchwork Thu May 27 20:55:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 810C3C4708A for ; Thu, 27 May 2021 20:56:23 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 43E64611C2 for ; Thu, 27 May 2021 20:56:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43E64611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5C0D96F4CC; Thu, 27 May 2021 20:56:16 +0000 (UTC) Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) by gabe.freedesktop.org (Postfix) with ESMTPS id BD0EC6F4C7; Thu, 27 May 2021 20:56:12 +0000 (UTC) Received: by mail-io1-xd34.google.com with SMTP id b25so1958464iot.5; Thu, 27 May 2021 13:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D2lHOtb7OcvVwWqNFg2wQ8RkEDsPGivk/u/nV7fuLJE=; b=u3qvkfKEnPO9JLJp8Z4Tzxytg/h+2PhjaA/5XOnee8IiF6yfE0F20hm7K5ZSpysPla aKcKFt+bBUK+zCfT2a+YzPfTCxteHp1fJLKBm9q85rsOqod2fbLTUgg7x5Xd6nMME0fx ij1DeHzSvqSGsg5tSqEFL2HJbQU09SwmgQj8dMU4qqVFnxqdJc9IfpH5BdntX2+2WuZH X5wjOz1B0bSVHjXihCgP1VLgJqC9VOiF/MeP4fjGgbXK1r0lxhplhtahx5PYg3L7dTaQ qoALygn5P6Twk+g05NBeQdEc2kjskROpse7nQQjNt/XFAPA+TGXQMwMYXYpqNSq4huey 5EwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D2lHOtb7OcvVwWqNFg2wQ8RkEDsPGivk/u/nV7fuLJE=; b=ckPNs8fjB/BoQTrpnHAQeKIqAqn9hlABjuJT5M6d0IvkOnX3Jezu3bL3zeHKkJr+5S 9/K6Ll49rhC1FvFVXIC0eoCOTOr8OCsDaem+slJ9d8pixKMCFXkuJGEez+LhJquYkwrN gAXeKbU5diDMNS7mGQOoH9sR7BDwEX/++qcOBD7uEpDROVOy5u8sg/zRprfSMo+jmZHF ErSGKmGaUr9Km+4rEfk+xsBi+11QDSxPXi+uq4YTDODKDC0a5jYs1/AaGkqzP2OGOAh8 QThNaqHt/Sq788WhBE7g33TId0da7QCJ7Sg4/DxXRrz4re9iRNphHuQCLhNBK5p23DoY 0prw== X-Gm-Message-State: AOAM530tuy/I0nSVoPfgfVy/XsTf0vCs7ffLyGltgCEFPxrIreWhqK18 fq5rtGo89/XxYXK6Pb+Q6Vp6ZVQViv+qdw== X-Google-Smtp-Source: ABdhPJxYUUknRVx3MxA0BgEmC2o+6SOh44Rbmrkg/DTY/MzzlohivfYmGFsXfOBjaC0hdv/V80VC9Q== X-Received: by 2002:a05:6638:21a:: with SMTP id e26mr5281508jaq.58.1622148971284; Thu, 27 May 2021 13:56:11 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:10 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 02/10] drm/amdkfd: add owner ref param to get hmm pages Date: Thu, 27 May 2021 16:55:58 -0400 Message-Id: <20210527205606.2660-2-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra The parameter is used in the dev_private_owner to decide if device pages in the range require to be migrated back to system memory, based if they are or not in the same memory domain. In this case, this reference could come from the same memory domain with devices connected to the same hive. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 ++-- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c index 2741c28ff1b5..378c238c2099 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c @@ -160,7 +160,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, struct mm_struct *mm, struct page **pages, uint64_t start, uint64_t npages, struct hmm_range **phmm_range, bool readonly, - bool mmap_locked) + bool mmap_locked, void *owner) { struct hmm_range *hmm_range; unsigned long timeout; @@ -185,6 +185,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, hmm_range->hmm_pfns = pfns; hmm_range->start = start; hmm_range->end = start + npages * PAGE_SIZE; + hmm_range->dev_private_owner = owner; /* Assuming 512MB takes maxmium 1 second to fault page address */ timeout = max(npages >> 17, 1ULL) * HMM_RANGE_DEFAULT_TIMEOUT; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h index 7f7d37a457c3..14a3c1864085 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h @@ -34,7 +34,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, struct mm_struct *mm, struct page **pages, uint64_t start, uint64_t npages, struct hmm_range **phmm_range, bool readonly, - bool mmap_locked); + bool mmap_locked, void *owner); int amdgpu_hmm_range_get_pages_done(struct hmm_range *hmm_range); #if defined(CONFIG_HMM_MIRROR) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 7e7d8330d64b..c13f7fbfc070 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -709,7 +709,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) readonly = amdgpu_ttm_tt_is_readonly(ttm); r = amdgpu_hmm_range_get_pages(&bo->notifier, mm, pages, start, ttm->num_pages, >t->range, readonly, - false); + false, NULL); out_putmm: mmput(mm); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index b665e9ff77e3..b939f353ac8c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1392,7 +1392,7 @@ static int svm_range_validate_and_map(struct mm_struct *mm, r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, prange->start << PAGE_SHIFT, prange->npages, &hmm_range, - false, true); + false, true, NULL); if (r) { pr_debug("failed %d to get svm range pages\n", r); goto unreserve_out; @@ -2657,7 +2657,7 @@ void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm) r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, prange->start << PAGE_SHIFT, prange->npages, &hmm_range, - false, true); + false, true, NULL); if (!r) { amdgpu_hmm_range_get_pages_done(hmm_range); prange->validated_once = true; From patchwork Thu May 27 20:55:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4C9EC4708A for ; Thu, 27 May 2021 20:56:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6494B611C2 for ; Thu, 27 May 2021 20:56:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6494B611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3283D6F4D2; Thu, 27 May 2021 20:56:17 +0000 (UTC) Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1A8F76F4C7; Thu, 27 May 2021 20:56:13 +0000 (UTC) Received: by mail-il1-x12d.google.com with SMTP id h11so1472005ili.9; Thu, 27 May 2021 13:56:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pe/cFaapJbxX6Jz6Zk1ehmG+ymWmsWcMKoth41aEtds=; b=PwbAU4oxKHNGnMdcyu/Rknpkbe6RbooFWgmZvbQmi57QZ0W5VCynxl3/cfalHtt3yE K7V2FeaPUxCYMjSIFuF39JeEbW8lzPlfmh/+FOeIOf1iBzirVrwwBQ1b2iIaCo+0xSvp ARpVv/JCM7OJYigcAESHgg/+LyxS+YSpUY2KEIRCbw0qY2jwra/NVav6B3JoOfapafvd 7hlCruDn9sTAiT0N9th9Q5foTAo9vU+JQuLB4M5S+q2Q8S44zxyLZHp/7J8tM3tWxxOJ 28c2pIAl1JNTiWDcLOLdjHSGn0gA953V+eDGRRqE8t1m4d21QSGi0Tx26MyjvrJrWQ/f QGvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pe/cFaapJbxX6Jz6Zk1ehmG+ymWmsWcMKoth41aEtds=; b=kmF5e85ajl3N4pxqzcr3ZQ06c5sIMjbdi+9hjUOiAfESyxRkJ+obNG3SWO5lo8l3rT JioowKvX2PVXSW8G5IO0uO0CmROey+qtRqqiVWXmbKZLSagnSAVUZFn97/UJt7pcwgCf EKBYdVw5n25AtDChAk4FhXAW4tSvMXKRsAWUa03668T2BM8sqdLwW6JyCntJYD+F6Ycu 4AK+SH824gUT0ZfMEhha9EiroMVLhFqX4Rjj2EEm/PfjFwrTwMdoYqbAw3OCkD9uYt0Z v+yDbVXeRCV/7eabrnlEsNNDOFryzZjhMt2qR/gshq5zcr63EAWj0wwqEs0I0gC8E4jT OVRA== X-Gm-Message-State: AOAM533fmYAKU+aDRheHWJgCEFM5X1Btug1rHNV+9idcWs3gFrsiGJlh r4Gv1J/55cF2l5wLDckdMt2RYX08i0gLfg== X-Google-Smtp-Source: ABdhPJy0ZikwCFfPlut2KTJKCyWQZz/ccCwDsVpxc+y/xNY59QIE6vQzBdIyVVrHNRXuBIlr+98x7w== X-Received: by 2002:a92:cb42:: with SMTP id f2mr4335125ilq.71.1622148972183; Thu, 27 May 2021 13:56:12 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:11 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 03/10] drm/amdkfd: set owner ref to svm range prefault Date: Thu, 27 May 2021 16:55:59 -0400 Message-Id: <20210527205606.2660-3-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra svm_range_prefault is called right before migrations to VRAM, to make sure pages are resident in system memory before the migration. With partial migrations, this reference is used by hmm range get pages to avoid migrating pages that are already in the same VRAM domain. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 5 +++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 3 ++- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 11f7f590c6ec..b298aa8dea4d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -512,7 +512,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, prange->start, prange->last, best_loc); /* FIXME: workaround for page locking bug with invalid pages */ - svm_range_prefault(prange, mm); + svm_range_prefault(prange, mm, SVM_ADEV_PGMAP_OWNER(adev)); start = prange->start << PAGE_SHIFT; end = (prange->last + 1) << PAGE_SHIFT; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index b939f353ac8c..54f47b09b14a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -2646,7 +2646,8 @@ svm_range_best_prefetch_location(struct svm_range *prange) /* FIXME: This is a workaround for page locking bug when some pages are * invalid during migration to VRAM */ -void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm) +void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm, + void *owner) { struct hmm_range *hmm_range; int r; @@ -2657,7 +2658,7 @@ void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm) r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, prange->start << PAGE_SHIFT, prange->npages, &hmm_range, - false, true, NULL); + false, true, owner); if (!r) { amdgpu_hmm_range_get_pages_done(hmm_range); prange->validated_once = true; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 4297250f259d..08542fe39303 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -176,7 +176,8 @@ void schedule_deferred_list_work(struct svm_range_list *svms); void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, unsigned long offset, unsigned long npages); void svm_range_free_dma_mappings(struct svm_range *prange); -void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm); +void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm, + void *owner); #else From patchwork Thu May 27 20:56:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0147DC47089 for ; Thu, 27 May 2021 20:56:26 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9926A611C2 for ; Thu, 27 May 2021 20:56:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9926A611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C91C56F4D0; Thu, 27 May 2021 20:56:16 +0000 (UTC) Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) by gabe.freedesktop.org (Postfix) with ESMTPS id 41B1D6F4C7; Thu, 27 May 2021 20:56:14 +0000 (UTC) Received: by mail-io1-xd29.google.com with SMTP id b25so1958515iot.5; Thu, 27 May 2021 13:56:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G1tcho2vQuPwN9OsudHtISmc/DxgqXUVsaVYx3XpKI0=; b=KXXWj1sNI002LBIdmNhaICZ/YWCAqVnfw6WfaMeoBYYRjtg3qY4Yt1TstcTK64eHpG 9IfDDhadktV/3hmtI8WQjVafzuaLg3fImSBYJhqYMjGWBu9UopwTUR+TVn4PWvWQrp7S MWmjc+41Zya8J9r9MjF0uPVEXmvYYppN3zLz7o790YwzjtxOMMIln8Waq4jQVkJYUmIv AIuxl4+QwYuRz8Be2geNxlDHrmK22PvrSaSw+ijYZddXlZ2cvsJV2NbSUWBto0g9qPMH /so01GUi7PfThfg7lXqlaeJm1iGY/77GJ7WKH8meajebVAO6X9Du2MoSY1dFJ+Bqldip Ovzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G1tcho2vQuPwN9OsudHtISmc/DxgqXUVsaVYx3XpKI0=; b=gA9DMOyDRXq9Linowj43SsNDmLkulYnn/zuzLFXicM/pdy8sPhz4yEMlnCFf3pk7PK aCnr9aKuXPJ98v4RablTYrcNLm3HnWY7c2xXsHZLEwfBHUDnaTcg3XUOwCGeG42DdLL0 17bQmHgvDCzOyc7vJtWtfppe8ePzuTI8B0PCLylMpN3Dk397CjROANW37ICmZv1aLtAD mJ807Rr9HLxaqiBVu4p+ThDawt5asaHfM4htSyEv6N85DggM9VD5B8d73qdjgQ/Y2ZD4 Ksz2w8fqk3hcEGqR+kbZCp8gWGY96rUFrVwnLkXJ5rXKgtIcEm3F3phc4PsWv4+7mENR k3Ug== X-Gm-Message-State: AOAM5329cBbX2NO0xdq10n8xHpqNTT3I7W/qYelaoTI7daLiSigBgL8V O0Q8QDoRFTdYv637BgixijTFYmQXJ1IsFg== X-Google-Smtp-Source: ABdhPJzkzIlMs1CWQgZzYkdtpkN6yKDgFTQ/W4ltxmM435fml4U8mVCf5T3QDT7WQXEhaEb058d4tQ== X-Received: by 2002:a5d:9c58:: with SMTP id 24mr4476883iof.153.1622148973183; Thu, 27 May 2021 13:56:13 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:12 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 04/10] drm/amdgpu: get owner ref in validate and map Date: Thu, 27 May 2021 16:56:00 -0400 Message-Id: <20210527205606.2660-4-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Get the proper owner reference for amdgpu_hmm_range_get_pages function. This is useful for partial migrations. To avoid migrating back to system memory, VRAM pages, that are accessible by all devices in the same memory domain. Ex. multiple devices in the same hive. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 54f47b09b14a..2b4318646a75 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1313,6 +1313,17 @@ static void svm_range_unreserve_bos(struct svm_validate_context *ctx) ttm_eu_backoff_reservation(&ctx->ticket, &ctx->validate_list); } +static void *kfd_svm_page_owner(struct kfd_process *p, int32_t gpuidx) +{ + struct kfd_process_device *pdd; + struct amdgpu_device *adev; + + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + adev = (struct amdgpu_device *)pdd->dev->kgd; + + return SVM_ADEV_PGMAP_OWNER(adev); +} + /* * Validation+GPU mapping with concurrent invalidation (MMU notifiers) * @@ -1343,6 +1354,9 @@ static int svm_range_validate_and_map(struct mm_struct *mm, { struct svm_validate_context ctx; struct hmm_range *hmm_range; + struct kfd_process *p; + void *owner; + int32_t idx; int r = 0; ctx.process = container_of(prange->svms, struct kfd_process, svms); @@ -1389,10 +1403,19 @@ static int svm_range_validate_and_map(struct mm_struct *mm, svm_range_reserve_bos(&ctx); if (!prange->actual_loc) { + p = container_of(prange->svms, struct kfd_process, svms); + owner = kfd_svm_page_owner(p, find_first_bit(ctx.bitmap, + MAX_GPU_INSTANCE)); + for_each_set_bit(idx, ctx.bitmap, MAX_GPU_INSTANCE) { + if (kfd_svm_page_owner(p, idx) != owner) { + owner = NULL; + break; + } + } r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, prange->start << PAGE_SHIFT, prange->npages, &hmm_range, - false, true, NULL); + false, true, owner); if (r) { pr_debug("failed %d to get svm range pages\n", r); goto unreserve_out; From patchwork Thu May 27 20:56:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F041C4707F for ; Thu, 27 May 2021 20:56:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 075C2611C2 for ; Thu, 27 May 2021 20:56:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 075C2611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8AF056F4D4; Thu, 27 May 2021 20:56:17 +0000 (UTC) Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) by gabe.freedesktop.org (Postfix) with ESMTPS id 258A46F4CD; Thu, 27 May 2021 20:56:15 +0000 (UTC) Received: by mail-il1-x129.google.com with SMTP id k4so1505933ili.4; Thu, 27 May 2021 13:56:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cYgArYhWT+z/4+YUOepxn67gZTW6vYNK84K++vMOkHs=; b=WRxOTAVFWmz3qY9+B7G7QF3T0vUcUKj5BuW11w3t3c+YM7LvApuzHn5SpFL5cKknQK 50XjT3mrje7hY8gFU5USsRQH5OUQPjMcdNBycxXdNSKUQ2r7avtVUMHELPkhFHg0+qjK vS02LmYUQ0kZRU+6pWyi5Nd+r24IMUy6r0/FP+HDIMSTZPSnRaoLBlM6z9XCKQawhMca bYJSp3x4nnvWYp1EdM7DNHa+IgYfiysxJuk7aRjvf3LjC4n56HAklSK9qxhAHQuCEsI4 LBVV6RNEzVl6g0FA+bs9YznieTnVI3+pW3ho4olrCIHpaDOkbYqtuorxKkUMunsSG/YM SQyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cYgArYhWT+z/4+YUOepxn67gZTW6vYNK84K++vMOkHs=; b=HXiQPZM77tGba+KD70VNG+4SQa506HyWMr74jgwh2pIEz2dJLWln4apFHw6WOTsxrU XbdeCllacTeI8uURsXXmzpCeQkhvgbHb/UAQMjbyo/WwlN+Gyz7D8HOCszHA1EVXOfvQ yLFFQZJkOiiW1ofeBNV8ZnaI91LiBO9URTn/C/3W2c1sptU5Di9eMIQovfmMDDWcnxto 71oRsGrGNkKc/+C14rcfkUUznkKuLmcfqcO7fVFZRKq73uBLqClatTX1maWZsHJkOaJY fpS4GugWLp5wxGlNyXGgPIRhFTE7PiTVfnew0y8OeI06q66GtlfL/vbk2DzklMYGaQog 0FUw== X-Gm-Message-State: AOAM530eBBlnAhystLZL+eJJUjuUvGipuhuqKKugXh1kXLOij76YlHn0 t3vSvKUBCZs/maYpJctJg3wmMHITRnNCzg== X-Google-Smtp-Source: ABdhPJzuesr7wky56TrTaKRgKsSaC3GkVDswyJksMURbnn0KyXkg1sF3ojsumAw+QkoirYIsoKj5Fg== X-Received: by 2002:a92:cda7:: with SMTP id g7mr4210742ild.50.1622148974142; Thu, 27 May 2021 13:56:14 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:13 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 05/10] drm/amdkfd: classify and map mixed svm range pages in GPU Date: Thu, 27 May 2021 16:56:01 -0400 Message-Id: <20210527205606.2660-5-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra [Why] svm ranges can have mixed pages from device or system memory. A good example is, after a prange has been allocated in VRAM and a copy-on-write is triggered by a fork. This invalidates some pages inside the prange. Endding up in mixed pages. [How] By classifying each page inside a prange, based on its type. Device or system memory, during dma mapping call. If page corresponds to VRAM domain, a flag is set to its dma_addr entry for each GPU. Then, at the GPU page table mapping. All group of contiguous pages within the same type are mapped with their proper pte flags. v2: Instead of using ttm_res to calculate vram pfns in the svm_range. It is now done by setting the vram real physical address into drm_addr array. This makes more flexible VRAM management, plus removes the need to have a BO reference in the svm_range. v3: Remove mapping member from svm_range Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 72 +++++++++++++++++----------- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 2 +- 2 files changed, 46 insertions(+), 28 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 2b4318646a75..7b50395ec377 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -119,11 +119,12 @@ static void svm_range_remove_notifier(struct svm_range *prange) } static int -svm_range_dma_map_dev(struct device *dev, dma_addr_t **dma_addr, +svm_range_dma_map_dev(struct amdgpu_device *adev, dma_addr_t **dma_addr, unsigned long *hmm_pfns, uint64_t npages) { enum dma_data_direction dir = DMA_BIDIRECTIONAL; dma_addr_t *addr = *dma_addr; + struct device *dev = adev->dev; struct page *page; int i, r; @@ -141,6 +142,14 @@ svm_range_dma_map_dev(struct device *dev, dma_addr_t **dma_addr, dma_unmap_page(dev, addr[i], PAGE_SIZE, dir); page = hmm_pfn_to_page(hmm_pfns[i]); + if (is_zone_device_page(page)) { + addr[i] = (hmm_pfns[i] << PAGE_SHIFT) + + adev->vm_manager.vram_base_offset - + adev->kfd.dev->pgmap.range.start; + addr[i] |= SVM_RANGE_VRAM_DOMAIN; + pr_debug("vram address detected: 0x%llx\n", addr[i]); + continue; + } addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir); r = dma_mapping_error(dev, addr[i]); if (r) { @@ -175,7 +184,7 @@ svm_range_dma_map(struct svm_range *prange, unsigned long *bitmap, } adev = (struct amdgpu_device *)pdd->dev->kgd; - r = svm_range_dma_map_dev(adev->dev, &prange->dma_addr[gpuidx], + r = svm_range_dma_map_dev(adev, &prange->dma_addr[gpuidx], hmm_pfns, prange->npages); if (r) break; @@ -1003,21 +1012,22 @@ svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, } static uint64_t -svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) +svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange, + int domain) { struct amdgpu_device *bo_adev; uint32_t flags = prange->flags; uint32_t mapping_flags = 0; uint64_t pte_flags; - bool snoop = !prange->ttm_res; + bool snoop = (domain != SVM_RANGE_VRAM_DOMAIN); bool coherent = flags & KFD_IOCTL_SVM_FLAG_COHERENT; - if (prange->svm_bo && prange->ttm_res) + if (domain == SVM_RANGE_VRAM_DOMAIN) bo_adev = amdgpu_ttm_adev(prange->svm_bo->bo->tbo.bdev); switch (adev->asic_type) { case CHIP_ARCTURUS: - if (prange->svm_bo && prange->ttm_res) { + if (domain == SVM_RANGE_VRAM_DOMAIN) { if (bo_adev == adev) { mapping_flags |= coherent ? AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW; @@ -1032,7 +1042,7 @@ svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) } break; case CHIP_ALDEBARAN: - if (prange->svm_bo && prange->ttm_res) { + if (domain == SVM_RANGE_VRAM_DOMAIN) { if (bo_adev == adev) { mapping_flags |= coherent ? AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW; @@ -1061,14 +1071,14 @@ svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE; pte_flags = AMDGPU_PTE_VALID; - pte_flags |= prange->ttm_res ? 0 : AMDGPU_PTE_SYSTEM; + pte_flags |= (domain == SVM_RANGE_VRAM_DOMAIN) ? 0 : AMDGPU_PTE_SYSTEM; pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0; pte_flags |= amdgpu_gem_va_map_flags(adev, mapping_flags); pr_debug("svms 0x%p [0x%lx 0x%lx] vram %d PTE 0x%llx mapping 0x%x\n", prange->svms, prange->start, prange->last, - prange->ttm_res ? 1:0, pte_flags, mapping_flags); + (domain == SVM_RANGE_VRAM_DOMAIN) ? 1:0, pte_flags, mapping_flags); return pte_flags; } @@ -1138,31 +1148,40 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, { struct amdgpu_bo_va bo_va; uint64_t pte_flags; + unsigned long last_start; + int last_domain; int r = 0; + int64_t i; pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); - if (prange->svm_bo && prange->ttm_res) { + if (prange->svm_bo && prange->ttm_res) bo_va.is_xgmi = amdgpu_xgmi_same_hive(adev, bo_adev); - prange->mapping.bo_va = &bo_va; - } - prange->mapping.start = prange->start; - prange->mapping.last = prange->last; - prange->mapping.offset = prange->offset; - pte_flags = svm_range_get_pte_flags(adev, prange); + last_start = prange->start; + for (i = 0; i < prange->npages; i++) { + last_domain = dma_addr[i] & SVM_RANGE_VRAM_DOMAIN; + if ((prange->start + i) < prange->last && + last_domain == (dma_addr[i + 1] & SVM_RANGE_VRAM_DOMAIN)) + continue; - r = amdgpu_vm_bo_update_mapping(adev, bo_adev, vm, false, false, NULL, - prange->mapping.start, - prange->mapping.last, pte_flags, - prange->mapping.offset, - prange->ttm_res ? - prange->ttm_res->mm_node : NULL, - dma_addr, &vm->last_update); - if (r) { - pr_debug("failed %d to map to gpu 0x%lx\n", r, prange->start); - goto out; + pr_debug("Mapping range [0x%lx 0x%llx] on domain: %s\n", + last_start, prange->start + i, last_domain ? "GPU" : "CPU"); + pte_flags = svm_range_get_pte_flags(adev, prange, last_domain); + r = amdgpu_vm_bo_update_mapping(adev, bo_adev, vm, false, false, NULL, + last_start, + prange->start + i, pte_flags, + prange->offset + + ((last_start - prange->start) << PAGE_SHIFT), + NULL, + dma_addr, + &vm->last_update); + if (r) { + pr_debug("failed %d to map to gpu 0x%lx\n", r, prange->start); + goto out; + } + last_start = prange->start + i + 1; } r = amdgpu_vm_update_pdes(adev, vm, false); @@ -1176,7 +1195,6 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, *fence = dma_fence_get(vm->last_update); out: - prange->mapping.bo_va = NULL; return r; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 08542fe39303..27fbe1936493 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -35,6 +35,7 @@ #include "amdgpu.h" #include "kfd_priv.h" +#define SVM_RANGE_VRAM_DOMAIN (1UL << 0) #define SVM_ADEV_PGMAP_OWNER(adev)\ ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) @@ -113,7 +114,6 @@ struct svm_range { struct list_head update_list; struct list_head remove_list; struct list_head insert_list; - struct amdgpu_bo_va_mapping mapping; uint64_t npages; dma_addr_t *dma_addr[MAX_GPU_INSTANCE]; struct ttm_resource *ttm_res; From patchwork Thu May 27 20:56:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8403C4708A for ; Thu, 27 May 2021 20:56:30 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8F979611C2 for ; Thu, 27 May 2021 20:56:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F979611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CE0056F4D1; Thu, 27 May 2021 20:56:17 +0000 (UTC) Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by gabe.freedesktop.org (Postfix) with ESMTPS id 106336F4CE; Thu, 27 May 2021 20:56:16 +0000 (UTC) Received: by mail-io1-xd32.google.com with SMTP id z24so1962146ioi.3; Thu, 27 May 2021 13:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MPLNHWojaW2z7tHHYVN0iHifaKQ7ufRqQo9gvltiW3c=; b=Bk5UbzlrXxdNsuHq8F16cO8p5FRrv8/IL0vwclE542mpXtKmv3YBMuAlLIn6zy2QGm g43T20oSxWrLgisP6r7541Su3rzutPYvM6Af3O2XdsjwvK/BZdY/ZC9mtU1DlMU/JzeV G/isU0y28NX0f3d367u7QTusCz3EYuCN8SblmgwHHkLiUjxFeTN8GrF3tkBRC+MZ2JH6 +X+Mmi+NppE4nHfdoNZFEuixsh92GBubX9dOBaV/z3OKOCoTav5yU1JaK1bLFXFaHK3b 5I+ZmFbuNhZT4nC65I6Bwm9yAAyqK4r3HP82sHSVRAv6/ZttXeKxoMFw/lHAxg40Rs5d mrTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MPLNHWojaW2z7tHHYVN0iHifaKQ7ufRqQo9gvltiW3c=; b=GMRfj+xhkmG7HEzazmhBNObw58oWwjx3eXe8KmklTGMyVrGB9VyxRSFsrT4CaFt8Dp YBLZqaYaJmSsnSqfwGE4fCTyzURpgtTh8UliNztnK07xBax9UWkbXNfFRhFmdn4MkFNW capthi4PMowVpVIP+OWXXXN0ISfvomUrE3KP+LInsRhlsSJwaxescNz6NzriwPh7E40R vDxlpKVIyL/rfgXpXDBcbPdJ2fqK6AZ2k9SZgmjArnha5ziJI0qo9T38SFo6yE6mSeXH /I6w5zDzwSpk1NOdYZGEAH3pL+R0MuTp76IWlEgYg6Tf2XZbfj6yb91zrZ579qEFbTHR CWKw== X-Gm-Message-State: AOAM532l/bYRcO9KIn4xujWHJOfJSahLEd9buWoZiRUBwMGHu3yXsMCe 2u0LahYov/xfNgqLGORxaUWvu7rCrqARog== X-Google-Smtp-Source: ABdhPJzh6oOfELkaZzhVcv9tmyNGre4Cn0U3R/nlhSkOHYw1LymwnYNanGJ2wCmAbT0iwZ3wRuA5ew== X-Received: by 2002:a02:a10f:: with SMTP id f15mr5218331jag.124.1622148975147; Thu, 27 May 2021 13:56:15 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:14 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 06/10] drm/amdkfd: skip invalid pages during migrations Date: Thu, 27 May 2021 16:56:02 -0400 Message-Id: <20210527205606.2660-6-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Invalid pages can be the result of pages that have been migrated already due to copy-on-write procedure or pages that were never migrated to VRAM in first place. This is not an issue anymore, as pranges now support mixed memory domains (CPU/GPU). Signed-off-by: Alex Sierra Reviewed-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 38 +++++++++++------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index b298aa8dea4d..6fd68528c425 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -419,7 +419,6 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, size_t size; void *buf; int r = -ENOMEM; - int retry = 0; memset(&migrate, 0, sizeof(migrate)); migrate.vma = vma; @@ -438,7 +437,6 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, migrate.dst = migrate.src + npages; scratch = (dma_addr_t *)(migrate.dst + npages); -retry: r = migrate_vma_setup(&migrate); if (r) { pr_debug("failed %d prepare migrate svms 0x%p [0x%lx 0x%lx]\n", @@ -446,17 +444,9 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, goto out_free; } if (migrate.cpages != npages) { - pr_debug("collect 0x%lx/0x%llx pages, retry\n", migrate.cpages, + pr_debug("Partial migration. 0x%lx/0x%llx pages can be migrated\n", + migrate.cpages, npages); - migrate_vma_finalize(&migrate); - if (retry++ >= 3) { - r = -ENOMEM; - pr_debug("failed %d migrate svms 0x%p [0x%lx 0x%lx]\n", - r, prange->svms, prange->start, prange->last); - goto out_free; - } - - goto retry; } if (migrate.cpages) { @@ -547,9 +537,8 @@ static void svm_migrate_page_free(struct page *page) static int svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, - dma_addr_t *scratch) + dma_addr_t *scratch, uint64_t npages) { - uint64_t npages = migrate->cpages; struct device *dev = adev->dev; uint64_t *src; dma_addr_t *dst; @@ -566,15 +555,23 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, src = (uint64_t *)(scratch + npages); dst = scratch; - for (i = 0, j = 0; i < npages; i++, j++, addr += PAGE_SIZE) { + for (i = 0, j = 0; i < npages; i++, addr += PAGE_SIZE) { struct page *spage; spage = migrate_pfn_to_page(migrate->src[i]); - if (!spage) { - pr_debug("failed get spage svms 0x%p [0x%lx 0x%lx]\n", + if (!spage || !is_zone_device_page(spage)) { + pr_debug("invalid page. Could be in CPU already svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); - r = -ENOMEM; - goto out_oom; + if (j) { + r = svm_migrate_copy_memory_gart(adev, dst + i - j, + src + i - j, j, + FROM_VRAM_TO_RAM, + mfence); + if (r) + goto out_oom; + j = 0; + } + continue; } src[i] = svm_migrate_addr(adev, spage); if (i > 0 && src[i] != src[i - 1] + PAGE_SIZE) { @@ -607,6 +604,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); migrate->dst[i] |= MIGRATE_PFN_LOCKED; + j++; } r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j, @@ -664,7 +662,7 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange, if (migrate.cpages) { r = svm_migrate_copy_to_ram(adev, prange, &migrate, &mfence, - scratch); + scratch, npages); migrate_vma_pages(&migrate); svm_migrate_copy_done(adev, mfence); migrate_vma_finalize(&migrate); From patchwork Thu May 27 20:56:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D98DC4707F for ; Thu, 27 May 2021 20:56:32 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6C17E611C2 for ; Thu, 27 May 2021 20:56:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C17E611C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 97B096F4D7; Thu, 27 May 2021 20:56:18 +0000 (UTC) Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1B27B6F4D1; Thu, 27 May 2021 20:56:17 +0000 (UTC) Received: by mail-il1-x129.google.com with SMTP id m1so1470321ilg.10; Thu, 27 May 2021 13:56:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2FzI6x3fU0QUcf05c+RVHhBbWnKsL4r+PvrdiDWp0Ug=; b=TLwGz8LMDx2vbMXpzw+10gPyg2EpEOTpZtnbFC3FGLvl96Z7gVXEaf1ZN9fJk3n15k RGC11WKM5KoPHT0esRsXvVSdfl7otKBapsX2IxvkgIfhEfwlbuVWiXa3EgpDvNz4qzlW sroUyGg1PXooi6hMOssfwjYdL1xLVa20VbABkZEQjZDFWpXmjpRiQRRp/eP3TeFxPBUn 7ELXNpBt/SQ+hkmcx2dNGQZDN0RI7nhCnhNB8hgadfC1NsffNsHawfLkRNz+KTUlNHN4 K06DJAEg9yscow7F1n5pIj0eiiLOTJpzZTHuMF0FY2Ha7dW6mI8PrETFAJSdwNWFzfsZ LmlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2FzI6x3fU0QUcf05c+RVHhBbWnKsL4r+PvrdiDWp0Ug=; b=eW9e+D1CRNeL37uXyRJCtUlEapXK9QGscwNfZL3xiNcRMr0ce35kjbGeaKdTyoQ2TG Z/NKqWp1dUNzlDx2EEl1viYXJjjkXf2qrE133kg1Q/Pr7StrhFDgVyFELNYGb7YPnJ2V gXdox0WURzyDq5/SFwDMFT55ZnSq05vvSc2q5KtwuX4JqpYLgbj1GZhNjF0IyMFlw3Nz q7WWUCfzU5zlAKz9eORD4Z6w9zzS8iiXq/lCTd/FzzPVLp/R3Kc/V12Nww8RrSty+oe8 oyF6ir5wpJ6kqXhGiIZ+5W33m29vIBANt+Eef4ywCB0cpbMgVG/DuA1vHQQmRTkNBO9v OfXw== X-Gm-Message-State: AOAM531uh1yVBGcu5TS1jTt+XLV+N9QNsnJBsQGA6o67snHtQO0l6XeT 1tomcAmmvJsy4nnXv9dJZuQbRlOdaH4NSQ== X-Google-Smtp-Source: ABdhPJyJiLAdgpBimSCfjgkLAi2g2Ax2oqqgTx5gUini8s6TDoputCYi5yezgWbXkhnr9Xg33F19zw== X-Received: by 2002:a92:c104:: with SMTP id p4mr4392567ile.307.1622148976090; Thu, 27 May 2021 13:56:16 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:15 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 07/10] drm/amdkfd: skip migration for pages already in VRAM Date: Thu, 27 May 2021 16:56:03 -0400 Message-Id: <20210527205606.2660-7-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Migration skipped for pages that are already in VRAM domain. These could be the result of previous partial migrations to SYS RAM, and prefetch back to VRAM. Ex. Coherent pages in VRAM that were not written/invalidated after a copy-on-write. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 6fd68528c425..8a3f21d76915 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -329,14 +329,15 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, for (i = j = 0; i < npages; i++) { struct page *spage; - dst[i] = vram_addr + (j << PAGE_SHIFT); - migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); - svm_migrate_get_vram_page(prange, migrate->dst[i]); - - migrate->dst[i] = migrate_pfn(migrate->dst[i]); - migrate->dst[i] |= MIGRATE_PFN_LOCKED; - - if (migrate->src[i] & MIGRATE_PFN_VALID) { + spage = migrate_pfn_to_page(migrate->src[i]); + if (spage && !is_zone_device_page(spage)) { + dst[i] = vram_addr + (j << PAGE_SHIFT); + migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); + svm_migrate_get_vram_page(prange, migrate->dst[i]); + migrate->dst[i] = migrate_pfn(migrate->dst[i]); + migrate->dst[i] |= MIGRATE_PFN_LOCKED; + } + if (migrate->dst[i] & MIGRATE_PFN_VALID) { spage = migrate_pfn_to_page(migrate->src[i]); src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, DMA_TO_DEVICE); From patchwork Thu May 27 20:56:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF449C4707F for ; Thu, 27 May 2021 20:56:37 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9BE4260FEB for ; Thu, 27 May 2021 20:56:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BE4260FEB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AE8B86F4DC; Thu, 27 May 2021 20:56:35 +0000 (UTC) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by gabe.freedesktop.org (Postfix) with ESMTPS id 565DD6F4D5; Thu, 27 May 2021 20:56:18 +0000 (UTC) Received: by mail-io1-xd36.google.com with SMTP id n10so1935902ion.8; Thu, 27 May 2021 13:56:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=k1Ma8e4VgLih51j4GLcjEplAHTajoYpcn4LnehgSpAs=; b=CZdDPQGxmTFwM6p6C4CNikpcwJhyUrQ/DqfYSJYXz9igEti2BxRwt2wzpM9W0nEZ1A YT69TIB+NITXPm8fbOvVveR+VcmJrZs/V8OSHztdvJzviEehdqc3CEsCay+HCbRB7bv0 pZoVOuDz5iSHywRywPvdtSJehFXorFXn37M1xOnq+xsCvAoP5vbGRyuCH5ZLlHnCM4q5 QBeMQ3U1+cBcg8YZxmpdV+G0SbiaX7SCTHlLU0IggH2yOumBHNBZfwRM5XHj60h8sMfu fUhMF+5kR9Rvt7K9MYKzb+wjP5bTK9e7+mg5HHYI0ZqC5mUmTTpTKXffyjIexSeYo+b4 KAlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k1Ma8e4VgLih51j4GLcjEplAHTajoYpcn4LnehgSpAs=; b=Eiwr1+IO8s4rlQuIaVqY0MTcAAKFrhCUZcEP78jg0kzCEDsWnHDfXROHd+xnyPz2vR VifCKF2BAVYdO4PuvGpDUSTzHkLGPxxH1fhocGGH6cGuYn9NsyDVBs3O2Qpx3SHmcA3s 7zwMYQYo+bhRyxRmAPv5C2EtFVhdx/SGTPcKwoBDrZDLXyw3XcVF+q+aOuDhkzJUFE68 o8dVwVN3t17QCF0j9PVFESq5V7UML7hncHUBX34LiM3luwRj+Cdim5LME9YWpTfbFf0+ JCy0LPROKQGayjxL7u+2iB8stZmNyRLSVvEuP+ElgBw3dz/spvK2Fsi0j4e7YHhDwZKt z3lw== X-Gm-Message-State: AOAM533LZz6PP7NKUHm8chSGnZISq7vud0lxTECh7ll4JQGi8lqzGSdv vAOFcCY8FJeiZH6u7ZxM/RMinBb66UbJXg== X-Google-Smtp-Source: ABdhPJxwoq1pth4gPv4Rv4Zj7JgZlu02ITqu9/GOuwtpWVb9SnoVFOrJ2ciZayfOP8NjF41P8+aQjw== X-Received: by 2002:a05:6638:12c5:: with SMTP id v5mr5249627jas.11.1622148977494; Thu, 27 May 2021 13:56:17 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:16 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 08/10] drm/amdkfd: add invalid pages debug at vram migration Date: Thu, 27 May 2021 16:56:04 -0400 Message-Id: <20210527205606.2660-8-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra This is for debug purposes only. It conditionally generates partial migrations to test mixed CPU/GPU memory domain pages in a prange easily. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 8a3f21d76915..f71f8d7e2b72 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -404,6 +404,20 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, } } +#ifdef DEBUG_FORCE_MIXED_DOMAINS + for (i = 0, j = 0; i < npages; i += 4, j++) { + if (j & 1) + continue; + svm_migrate_put_vram_page(adev, dst[i]); + migrate->dst[i] = 0; + svm_migrate_put_vram_page(adev, dst[i + 1]); + migrate->dst[i + 1] = 0; + svm_migrate_put_vram_page(adev, dst[i + 2]); + migrate->dst[i + 2] = 0; + svm_migrate_put_vram_page(adev, dst[i + 3]); + migrate->dst[i + 3] = 0; + } +#endif out: return r; } From patchwork Thu May 27 20:56:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67A09C4708E for ; Thu, 27 May 2021 20:56:36 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2EF7260FEB for ; Thu, 27 May 2021 20:56:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2EF7260FEB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BE79E6F4DA; Thu, 27 May 2021 20:56:34 +0000 (UTC) Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 596936F4DB; Thu, 27 May 2021 20:56:19 +0000 (UTC) Received: by mail-io1-xd31.google.com with SMTP id k22so1936430ioa.9; Thu, 27 May 2021 13:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8NtsOMPx4MYkCLJ0TPZ28k3bl7JgFX4G9RGUV3lfIuo=; b=s6oFQ0ulr9y1gkGxTcI0G7Opo/22W70in/KBdRlCwSewauReb2dTeJZG8chtm9dy/O pYVLndfJ8PXPRxv6SLyd9eaK8Ndf/UVfJhwVWFDOuhCwwirO0izyLgcQYnrmAiFnZXR9 7NLvT/hj5N7mn9Q5CZtwY7nJNZlyyHNGKB1xFeeVr9MqNgmgveVry3FDhXvYMZIg8NFj u6vOZlqYe6xdWhnO8HleEAZYmTIIraCY/gvcmx/HjK6+e6Bs9pJzLDGHGbXJTFlxTqrR v4rOeJrlzI6jTH2Ri8CCo/NU2xJgTu5OOpaBLubqJtwiCmByDaJUbzxrGFft24IuhGpN g71g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8NtsOMPx4MYkCLJ0TPZ28k3bl7JgFX4G9RGUV3lfIuo=; b=M3K1GUYgapS9/PCsG1vLAzdZzLMmVxfkpiXukJyced0PY4aVCVYLl/DBE1GsisxDCd YiOfelZRp4I3ujj2ITCaEoH+LGBMy+dM1Cc9Ae++m0K6Zg61mgxfYhY/1XvHEhih+t2I THo1qLo9uis14RNk6gJo3i1CXUYvHKb1HP4rChp5JfcASfvBsmn59Omdi53Sv+FksELm DN3UgT4BTEfy/nQRh+EENaAxtH0qjhpbN3TZKQYeqqXkyFhsnMlB9Cq9idBBQ9uEEacq u4CKdB91W2KYBhOXt24MxV34X9/HBK9xbd39kCxhrlfgOUbg1GInRQsJHamZPGY3Xu/A 7ZOQ== X-Gm-Message-State: AOAM531XpCiXKRPyQ+5X6ZnsUnms26xF4DkNBoXAVlAUfGpH2DxHXRrP E59iYAPMy2G+b8PoAvTrHVYLNBhEJ28mAA== X-Google-Smtp-Source: ABdhPJzT6HgjvQzvbCBhkeTY5/c5SttWmlQTFK6CqeRkiK2Fkkdv+W6mILxxZg7EfoerYFFGF9HT7g== X-Received: by 2002:a6b:e817:: with SMTP id f23mr4234692ioh.183.1622148978438; Thu, 27 May 2021 13:56:18 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:18 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 09/10] drm/amdkfd: partially actual_loc removed Date: Thu, 27 May 2021 16:56:05 -0400 Message-Id: <20210527205606.2660-9-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra actual_loc should not be used anymore, as pranges could have mixed locations (VRAM & SYSRAM) at the same time. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 12 +--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 71 ++++++++++-------------- 2 files changed, 29 insertions(+), 54 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index f71f8d7e2b72..acb9f64577a0 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -501,12 +501,6 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, struct amdgpu_device *adev; int r = 0; - if (prange->actual_loc == best_loc) { - pr_debug("svms 0x%p [0x%lx 0x%lx] already on best_loc 0x%x\n", - prange->svms, prange->start, prange->last, best_loc); - return 0; - } - adev = svm_range_get_adev_by_id(prange, best_loc); if (!adev) { pr_debug("failed to get device by id 0x%x\n", best_loc); @@ -791,11 +785,7 @@ int svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc, struct mm_struct *mm) { - if (!prange->actual_loc) - return svm_migrate_ram_to_vram(prange, best_loc, mm); - else - return svm_migrate_vram_to_vram(prange, best_loc, mm); - + return svm_migrate_ram_to_vram(prange, best_loc, mm); } /** diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 7b50395ec377..1e15a6170635 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1420,42 +1420,38 @@ static int svm_range_validate_and_map(struct mm_struct *mm, svm_range_reserve_bos(&ctx); - if (!prange->actual_loc) { - p = container_of(prange->svms, struct kfd_process, svms); - owner = kfd_svm_page_owner(p, find_first_bit(ctx.bitmap, - MAX_GPU_INSTANCE)); - for_each_set_bit(idx, ctx.bitmap, MAX_GPU_INSTANCE) { - if (kfd_svm_page_owner(p, idx) != owner) { - owner = NULL; - break; - } - } - r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, - prange->start << PAGE_SHIFT, - prange->npages, &hmm_range, - false, true, owner); - if (r) { - pr_debug("failed %d to get svm range pages\n", r); - goto unreserve_out; - } - - r = svm_range_dma_map(prange, ctx.bitmap, - hmm_range->hmm_pfns); - if (r) { - pr_debug("failed %d to dma map range\n", r); - goto unreserve_out; + p = container_of(prange->svms, struct kfd_process, svms); + owner = kfd_svm_page_owner(p, find_first_bit(ctx.bitmap, + MAX_GPU_INSTANCE)); + for_each_set_bit(idx, ctx.bitmap, MAX_GPU_INSTANCE) { + if (kfd_svm_page_owner(p, idx) != owner) { + owner = NULL; + break; } + } + r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, + prange->start << PAGE_SHIFT, + prange->npages, &hmm_range, + false, true, owner); + if (r) { + pr_debug("failed %d to get svm range pages\n", r); + goto unreserve_out; + } - prange->validated_once = true; + r = svm_range_dma_map(prange, ctx.bitmap, + hmm_range->hmm_pfns); + if (r) { + pr_debug("failed %d to dma map range\n", r); + goto unreserve_out; } + prange->validated_once = true; + svm_range_lock(prange); - if (!prange->actual_loc) { - if (amdgpu_hmm_range_get_pages_done(hmm_range)) { - pr_debug("hmm update the range, need validate again\n"); - r = -EAGAIN; - goto unlock_out; - } + if (amdgpu_hmm_range_get_pages_done(hmm_range)) { + pr_debug("hmm update the range, need validate again\n"); + r = -EAGAIN; + goto unlock_out; } if (!list_empty(&prange->child_list)) { pr_debug("range split by unmap in parallel, validate again\n"); @@ -2740,20 +2736,9 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, *migrated = false; best_loc = svm_range_best_prefetch_location(prange); - if (best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED || - best_loc == prange->actual_loc) + if (best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED) return 0; - /* - * Prefetch to GPU without host access flag, set actual_loc to gpu, then - * validate on gpu and map to gpus will be handled afterwards. - */ - if (best_loc && !prange->actual_loc && - !(prange->flags & KFD_IOCTL_SVM_FLAG_HOST_ACCESS)) { - prange->actual_loc = best_loc; - return 0; - } - if (!best_loc) { r = svm_migrate_vram_to_ram(prange, mm); *migrated = !r; From patchwork Thu May 27 20:56:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5CE3C47089 for ; Thu, 27 May 2021 20:56:38 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B005360FEB for ; Thu, 27 May 2021 20:56:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B005360FEB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 707766F4DB; Thu, 27 May 2021 20:56:36 +0000 (UTC) Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 825F16F4DA; Thu, 27 May 2021 20:56:20 +0000 (UTC) Received: by mail-il1-x136.google.com with SMTP id o10so1456492ilm.13; Thu, 27 May 2021 13:56:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r/JjWzro4A2OCgWHcu1YXap6BV3Bmhd60DexHUrV220=; b=o07lZiXsxTVWUuuqUPVXT4q1SfjYXCOjeWjGTRsiI1vX7D2ekO2JFNe9Qcsk6I0wBV //HacdPj0E0Zkqf9GMcalvM7nThh9uUgVDFOXl2+e46PHtPQjugCz8/+Kt5d4JN+vO/H OvK3xTk0WCL/nQ9wbbNWzqTvMfcwsw/aYGxutPVEe4LcScGoygsMk5aClYMGwHr2677k TA89EdHN5qlLLKPAph0ycbaiOQn1pjHicbD7WS4WnrWkcTchMNbzRlJYhr64dfjPR3Am 0fUo3LHx4Q6zGL4bEwM+WQsFP4GRFF9nzSDjNDurjIhj06rjjrDhqP0/0nlwdkD5U6hn sawQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r/JjWzro4A2OCgWHcu1YXap6BV3Bmhd60DexHUrV220=; b=SwxxCyPbzi1rw4UEtDL+xuFILZNysLajOPuaJdlLdgaqwWVmn3TZDcHcn/Z95F9RcO ntTurMImVkA42dFSoxsnlpfgAVXkVNwOCjKRvqYYfVlXn3jdlYMZBlK88n4gtt0Zxk5z 2YaXXdRvGNdvKKSbmZw2RIdKCENgEadZhiDfIk5uZlWXOwwwJDFpUcCMVzs6bGGvkQ4m 4/9BAEIikH74WCWyBJpLrccd2spewmd8T9bDY1J1cAyfW06amaaLFFNSEYi2suRQr8bO mzQVin10I3fHgazW4DhGNdc+YJ9p1q/2PeXKw/K3gc/lBt9IcgOSdp20r6Z9rKbbIDwo OfSw== X-Gm-Message-State: AOAM532RHoNvugZLd1znkVXJTBChUc4vjsHCOxVqjgklBNcDNbQflvPu A6nOb2+P9DV58gsQS3nc2klSWVB1+ZV7Ug== X-Google-Smtp-Source: ABdhPJxl/iMYR8Z9Q3oCLtRIuSau+DDg/+nplNKRdqaAc8wE7+0J5MGQpLWf1HS0GKrZbtgyE05dbg== X-Received: by 2002:a05:6e02:b4b:: with SMTP id f11mr4551361ilu.253.1622148979406; Thu, 27 May 2021 13:56:19 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id b10sm1577265ioz.35.2021.05.27.13.56.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 13:56:19 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 10/10] drm/amdkfd: protect svm_bo ref in case prange has forked Date: Thu, 27 May 2021 16:56:06 -0400 Message-Id: <20210527205606.2660-10-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210527205606.2660-1-Felix.Kuehling@amd.com> References: <20210527205606.2660-1-Felix.Kuehling@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Keep track of all the pages inside of pranges referenced to the same svm_bo. This is done, by using the ref count inside this object. This makes sure the object has freed after the last prange is not longer at any GPU. Including references shared between a parent and child during a fork. Signed-off-by: Alex Sierra --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 10 ++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 10 +--------- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 10 +++++++++- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index acb9f64577a0..c8ca3252cbc2 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -245,7 +245,7 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) struct page *page; page = pfn_to_page(pfn); - page->zone_device_data = prange; + page->zone_device_data = prange->svm_bo; get_page(page); lock_page(page); } @@ -336,6 +336,7 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, svm_migrate_get_vram_page(prange, migrate->dst[i]); migrate->dst[i] = migrate_pfn(migrate->dst[i]); migrate->dst[i] |= MIGRATE_PFN_LOCKED; + svm_range_bo_ref(prange->svm_bo); } if (migrate->dst[i] & MIGRATE_PFN_VALID) { spage = migrate_pfn_to_page(migrate->src[i]); @@ -540,7 +541,12 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, static void svm_migrate_page_free(struct page *page) { - /* Keep this function to avoid warning */ + struct svm_range_bo *svm_bo = page->zone_device_data; + + if (svm_bo) { + pr_debug("svm_bo ref left: %d\n", kref_read(&svm_bo->kref)); + svm_range_bo_unref(svm_bo); + } } static int diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 1e15a6170635..2bc20752ee30 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -309,14 +309,6 @@ static bool svm_bo_ref_unless_zero(struct svm_range_bo *svm_bo) return true; } -static struct svm_range_bo *svm_range_bo_ref(struct svm_range_bo *svm_bo) -{ - if (svm_bo) - kref_get(&svm_bo->kref); - - return svm_bo; -} - static void svm_range_bo_release(struct kref *kref) { struct svm_range_bo *svm_bo; @@ -355,7 +347,7 @@ static void svm_range_bo_release(struct kref *kref) kfree(svm_bo); } -static void svm_range_bo_unref(struct svm_range_bo *svm_bo) +void svm_range_bo_unref(struct svm_range_bo *svm_bo) { if (!svm_bo) return; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 27fbe1936493..21f693767a0d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -150,6 +150,14 @@ static inline void svm_range_unlock(struct svm_range *prange) mutex_unlock(&prange->lock); } +static inline struct svm_range_bo *svm_range_bo_ref(struct svm_range_bo *svm_bo) +{ + if (svm_bo) + kref_get(&svm_bo->kref); + + return svm_bo; +} + int svm_range_list_init(struct kfd_process *p); void svm_range_list_fini(struct kfd_process *p); int svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, @@ -178,7 +186,7 @@ void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, void svm_range_free_dma_mappings(struct svm_range *prange); void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm, void *owner); - +void svm_range_bo_unref(struct svm_range_bo *svm_bo); #else struct kfd_process;