From patchwork Thu May 27 23:08:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12285587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA34DC4707F for ; Thu, 27 May 2021 23:09:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 466DF613E3 for ; Thu, 27 May 2021 23:09:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 466DF613E3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CA3D96B006C; Thu, 27 May 2021 19:09:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C54216B006E; Thu, 27 May 2021 19:09:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AC066B0070; Thu, 27 May 2021 19:09:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0150.hostedemail.com [216.40.44.150]) by kanga.kvack.org (Postfix) with ESMTP id 35F126B006C for ; Thu, 27 May 2021 19:09:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BFD2ABBDF for ; Thu, 27 May 2021 23:09:27 +0000 (UTC) X-FDA: 78188554374.20.EE0480B Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) by imf12.hostedemail.com (Postfix) with ESMTP id 2C2852BDD for ; Thu, 27 May 2021 23:09:16 +0000 (UTC) Received: by mail-il1-f173.google.com with SMTP id o9so1761991ilh.6 for ; Thu, 27 May 2021 16:09:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=4kLiEsgXJL4l0OE3QcUENS/6JLbXlz5zRsd7ODaSzsA=; b=AN9s3c3Nnqymav/1Na5LgRM+yBjehQGzQVHllAlGVj/anzx0uk05waX9p6Xtf+AB1L sMWyKD2IQV3IiuxHTkKcNwoXxLgx7iNowi6tWHHMy2FjfaXbbnAr8KECFOUxY2sfgw8a xCOFXm7f+fWr8Pi0DVhWkJuMKMg1d+BLm6/QcmgULUGLyiB6I9d1a3b/Xs7Cz7endodL sZ7Yn7Dvgw0ZidAvf+VzU3YQWI+cvURDkp/ocIflIPjdhuyA61n06Ec5h9MMkkE/XZnq z6K+JUN48Q+XvgRICM5Blx185WYOCsG84f7kPj97hhZihds/zRInHrm2Etmv3Er9TiFg QYQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=4kLiEsgXJL4l0OE3QcUENS/6JLbXlz5zRsd7ODaSzsA=; b=Zxz50KgCXarK8b3ptkuYYRf0jR+604u1AWsjSg3AqmZ3JxEWkTY2oaCjjOVbTSynSw B+UOT0/FC1ASaPvX/pMisZ6GDu2lQ8G2H8x4oSARW4Wq0aRIq1ls4/Yto3H7IR/wPV+M d4Ns3tkEcBtOYMtHulKazAvrdtcG1DYiKdj8fyu2rLewRv+//NFcWpNZBRcup1kqSqmZ NcANz0Yl+tqFUU0N5S731XtrN71vtldUS/PycbRH/kAIWSC5Ms8g9Aab+3nOAoZD34i7 6bPH6VJCblMpXJR/HDyNzCq4gGFM/nf1DjcrRIP+9ymFtGIFwBNbcvkuoxkUB33OWCOI Zg1g== X-Gm-Message-State: AOAM530OCKZnLq9exaGpm9oyY6YEhfp4TW8y8tpOKsS8qaQ/wZds/yHV yrqjIaGBNk416EdC2tw0S0Y= X-Google-Smtp-Source: ABdhPJwk5Q8Yq1NmCjqTz3x3UVzu8172YycfbZkjO+O53PFGq2hSwrHmFPyOgtebO7cEMlrgm4pSUA== X-Received: by 2002:a92:d9d2:: with SMTP id n18mr5096577ilq.10.1622156966745; Thu, 27 May 2021 16:09:26 -0700 (PDT) Received: from Harpoon.amd.com ([165.204.55.251]) by smtp.gmail.com with ESMTPSA id r5sm1860014ilb.1.2021.05.27.16.09.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 16:09:09 -0700 (PDT) From: Felix Kuehling X-Google-Original-From: Felix Kuehling To: felix.kuehling@amd.com, akpm@linux-foundation.org, linux-mm@kvack.org Cc: hch@lst.de, jglisse@redhat.com, jgg@nvidia.com, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_* Date: Thu, 27 May 2021 19:08:04 -0400 Message-Id: <20210527230809.3701-1-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=AN9s3c3N; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of felixkuehling@gmail.com designates 209.85.166.173 as permitted sender) smtp.mailfrom=felixkuehling@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2C2852BDD X-Stat-Signature: 4yn4hej49zfkuqtotpaamjfe1kzph7us X-HE-Tag: 1622156956-99735 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE. The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages. Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages. This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)? This work is based on HMM and our SVM memory manager that was recently upstreamed to Dave Airlie's drm-next branch [https://cgit.freedesktop.org/drm/drm/log/?h=drm-next]. On top of that we did some rework of our VRAM management for migrations to remove some incorrect assumptions, allow partially successful migrations and GPU memory mappings that mix pages in VRAM and system memory. [https://patchwork.kernel.org/project/dri-devel/list/?series=489811] In this RFC, patches 1 and 2 are for context to show how we are looking up the SPM memory and registering it with devmap. Patches 3-5 are the changes we are trying to upstream or rework to make them acceptable upstream. Alex Sierra (5): drm/amdkfd: add SPM support for SVM drm/amdkfd: generic type as sys mem on migration to ram include/linux/mm.h: helper to check zone device generic type mm: add generic type support for device zone page migration mm: changes to unref pages with Generic type drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 +++++++++++---- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 - include/linux/mm.h | 8 ++++++++ kernel/resource.c | 2 +- mm/memremap.c | 5 ++++- mm/migrate.c | 13 ++++++++----- 6 files changed, 32 insertions(+), 12 deletions(-)