From patchwork Thu Jul 23 22:30:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ralph Campbell X-Patchwork-Id: 11681703 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6C36618 for ; Thu, 23 Jul 2020 22:30:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 849A420737 for ; Thu, 23 Jul 2020 22:30:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="c5NVsizi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727897AbgGWWaw (ORCPT ); Thu, 23 Jul 2020 18:30:52 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:13862 "EHLO hqnvemgate24.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727861AbgGWWav (ORCPT ); Thu, 23 Jul 2020 18:30:51 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 23 Jul 2020 15:28:48 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 23 Jul 2020 15:30:50 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 23 Jul 2020 15:30:50 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 23 Jul 2020 22:30:42 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 23 Jul 2020 22:30:42 +0000 Received: from rcampbell-dev.nvidia.com (Not Verified[10.110.48.66]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Thu, 23 Jul 2020 15:30:41 -0700 From: Ralph Campbell To: , , , , , CC: Jerome Glisse , John Hubbard , Christoph Hellwig , Jason Gunthorpe , "Andrew Morton" , Shuah Khan , "Ben Skeggs" , Bharata B Rao , "Ralph Campbell" Subject: [PATCH v4 5/6] mm/hmm/test: use the new migration invalidation Date: Thu, 23 Jul 2020 15:30:03 -0700 Message-ID: <20200723223004.9586-6-rcampbell@nvidia.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200723223004.9586-1-rcampbell@nvidia.com> References: <20200723223004.9586-1-rcampbell@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1595543328; bh=BeVUOTJgQepW+LMWyDGWqpRY0jrU7O9ftBuhCh+HHBg=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=c5NVsiziP4cirzdcWC6K4LcVP4KSZN9WPHjiDB+wb369XeSgW+x0ffbg5/GsZEdHZ VNlqkYRVXriPkI7ArKZLO64O9s9bii6xIeiWVSn4lGPyTmw4PRgI+Gtqh+QLzLZuZj YWQRDbCiFOqf/ouBw74NnHOkciuxCGS3m1CrofSqd3zATJF3aqCFPcZZXNR6CqiIZz KekhEI0xT5i1MApDLFN41+rGbDo1hdbh0//Zubm+a3DuFfV5TO6Xv8RGAzhZEjAhEO j5W7XxfbfN+/isxrdbiUbqx1idn5RUo2XnM3MonM0s/2zPxlLTN2gpqZZau7Ejo0tO MGzZ+BIT5US8A== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use the new MMU_NOTIFY_MIGRATE event to skip MMU invalidations of device private memory and handle the invalidation in the driver as part of migrating device private memory. Signed-off-by: Ralph Campbell --- lib/test_hmm.c | 30 +++++++++++++++----------- tools/testing/selftests/vm/hmm-tests.c | 18 ++++++++++++---- 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index e78a1414f58e..e7dc3de355b7 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -214,6 +214,14 @@ static bool dmirror_interval_invalidate(struct mmu_interval_notifier *mni, { struct dmirror *dmirror = container_of(mni, struct dmirror, notifier); + /* + * Ignore invalidation callbacks for device private pages since + * the invalidation is handled as part of the migration process. + */ + if (range->event == MMU_NOTIFY_MIGRATE && + range->migrate_pgmap_owner == dmirror->mdevice) + return true; + if (mmu_notifier_range_blockable(range)) mutex_lock(&dmirror->mutex); else if (!mutex_trylock(&dmirror->mutex)) @@ -693,7 +701,7 @@ static int dmirror_migrate(struct dmirror *dmirror, args.dst = dst_pfns; args.start = addr; args.end = next; - args.pgmap_owner = NULL; + args.pgmap_owner = dmirror->mdevice; args.flags = MIGRATE_VMA_SELECT_SYSTEM; ret = migrate_vma_setup(&args); if (ret) @@ -983,7 +991,7 @@ static void dmirror_devmem_free(struct page *page) } static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, - struct dmirror_device *mdevice) + struct dmirror *dmirror) { const unsigned long *src = args->src; unsigned long *dst = args->dst; @@ -1005,6 +1013,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, continue; lock_page(dpage); + xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); copy_highpage(dpage, spage); *dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED; if (*src & MIGRATE_PFN_WRITE) @@ -1013,15 +1022,6 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, return 0; } -static void dmirror_devmem_fault_finalize_and_map(struct migrate_vma *args, - struct dmirror *dmirror) -{ - /* Invalidate the device's page table mapping. */ - mutex_lock(&dmirror->mutex); - dmirror_do_update(dmirror, args->start, args->end); - mutex_unlock(&dmirror->mutex); -} - static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) { struct migrate_vma args; @@ -1051,11 +1051,15 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) if (migrate_vma_setup(&args)) return VM_FAULT_SIGBUS; - ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror->mdevice); + ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror); if (ret) return ret; migrate_vma_pages(&args); - dmirror_devmem_fault_finalize_and_map(&args, dmirror); + /* + * No device finalize step is needed since + * dmirror_devmem_fault_alloc_and_copy() will have already + * invalidated the device page table. + */ migrate_vma_finalize(&args); return 0; } diff --git a/tools/testing/selftests/vm/hmm-tests.c b/tools/testing/selftests/vm/hmm-tests.c index b533dd08da1d..91d38a29956b 100644 --- a/tools/testing/selftests/vm/hmm-tests.c +++ b/tools/testing/selftests/vm/hmm-tests.c @@ -881,8 +881,9 @@ TEST_F(hmm, migrate) } /* - * Migrate anonymous memory to device private memory and fault it back to system - * memory. + * Migrate anonymous memory to device private memory and fault some of it back + * to system memory, then try migrating the resulting mix of system and device + * private memory to the device. */ TEST_F(hmm, migrate_fault) { @@ -924,8 +925,17 @@ TEST_F(hmm, migrate_fault) for (i = 0, ptr = buffer->mirror; i < size / sizeof(*ptr); ++i) ASSERT_EQ(ptr[i], i); - /* Fault pages back to system memory and check them. */ - for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i) + /* Fault half the pages back to system memory and check them. */ + for (i = 0, ptr = buffer->ptr; i < size / (2 * sizeof(*ptr)); ++i) + ASSERT_EQ(ptr[i], i); + + /* Migrate memory to the device again. */ + ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_MIGRATE, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i = 0, ptr = buffer->mirror; i < size / sizeof(*ptr); ++i) ASSERT_EQ(ptr[i], i); hmm_buffer_free(buffer);