From patchwork Wed Sep 28 12:01:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 12992186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52047C32771 for ; Wed, 28 Sep 2022 12:02:21 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0C19510E49C; Wed, 28 Sep 2022 12:02:12 +0000 (UTC) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2047.outbound.protection.outlook.com [40.107.96.47]) by gabe.freedesktop.org (Postfix) with ESMTPS id A148510E479; Wed, 28 Sep 2022 12:01:52 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UHEJEguBzni4ePR19CqTEzuBHAUbpRCCcwEWokqc0/W3ldxS4h25d2v5e+El0zNcwZIhgHRczaM04sZbEWUsG5NOrSsr1IhzqEL7fuK830apCsF035z3nOu3tVrgpFXti/N9KPXA6Ca+yBTuZA0HG8X3GaL1HlSk9Qu/sNFoVqcDh6iZXX/jeJy/I1YKdzAC+IrhTCTp8TuYwuitnknL48D/SSu+7Y3+hltDBDy86TpN0RH0yWOrfBfKPvYjJqK06YyvZz1JqmLj7ZREhcT8D2Zx43HgBYgq4qfoK9BkaoAfxtSad9OXY69uN3lr5d+tvDe5A03HWx0KM85AQoI7Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iXSzlgttX5rlmCnLMM95a+/kiNZ2711zyubrbhwBCVg=; b=QU1+wfxi7c4P4Yv59a41Jxt242PiFZxnamyP6CBAn4qciit1W5wrxM6bRPyB8Szprz+4ps0WTcziDHcXbkGKrsi+kltkQELaH5DVYPEBnIA5+5SRuIlF7SNcoF6cyIjy9j9hnIE9EX4NgQfX66pNdLsCRj/XSqWWtChbTS1xPzAdLlUG7iv0UHk+d6F8Itx8ZJ0s+o9nO8QaPlTkC6sVrwQ6kBasOK06CCBW8pIfyLuB8p09E3nojWEcwOPrANu3ww1EyyPRug9abc58qCACbwzTx9ZUL2IAjmLQzve8v5iPUczsW+DVjWB2gYe+ocXlpqIGZf3fOWnHhxORMq6Ikw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iXSzlgttX5rlmCnLMM95a+/kiNZ2711zyubrbhwBCVg=; b=KmHfUXuDo5U9QmYKzLJtrNeOLXZ5teMa8v8Cqx0faaRhoY0uIWaRLpto/KxsR72r6XcaRh0FnRq0RhYsMjBqEVLdiFbVUoJp1mqTu2BB0QZXgSZAnysnRoOv4XpX1UhA1u0LKYG2+bpstfrRCV/TAtf1PBNKd2uKNmQbKzb8dz/2e33laJlBAYK9DoizLr3UXqkvbi1YsyzTeBrbLwDQ3GCq4sRyVWp1p3tWTBLw5ofdEQC7YIikPdEvv4WriVcIeX+ueAOxfZ+z0CaNaJ04m2lWMB9KU2TbaEHhPpb1PftsAYh8qVuSKRvub1GXwH+l13tN9KuBaNgUaI7g2AsFSQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM6PR12MB4337.namprd12.prod.outlook.com (2603:10b6:5:2a9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.17; Wed, 28 Sep 2022 12:01:50 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4064:6c13:72e5:a936]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4064:6c13:72e5:a936%5]) with mapi id 15.20.5654.026; Wed, 28 Sep 2022 12:01:50 +0000 From: Alistair Popple To: Andrew Morton , linux-mm@kvack.org Subject: [PATCH v2 2/8] mm: Free device private pages have zero refcount Date: Wed, 28 Sep 2022 22:01:16 +1000 Message-Id: X-Mailer: git-send-email 2.35.1 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0085.ausprd01.prod.outlook.com (2603:10c6:10:110::18) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM6PR12MB4337:EE_ X-MS-Office365-Filtering-Correlation-Id: 45404f4b-7570-49e7-ff4e-08daa14939f9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2AQERJpNRRRHCGU79grVWkpGdxmjaYr/4QkPsUsOvZy3bjRvEXvEjpdlzfBGcfWzvpLKMIuAZO5JyfSxELZ9ttRi9YgrLpQA6jabh279eKAoInxoYRSrY6vtNjU/YLHCl89dTioEX19ZtSE+Bzviee382GfN7Abyqpxl2sL+dbXXziBJLmTT1nq1vr1tUG/J9pj19mNEWeALLLwoDOlPrFRus/kB1eXY3pzm4zN0qbsh2WVpoGC8GKBHyeAbE2zNt7M3lUkTmUmrNwSXCSthfzpK8TEqmesW/a5arpxtroIJofKkbBzmRp13bcb1pclf0fQnkbbeDPqkxozQm79rWm1buYW5pVXBy5LOfSup2tFDX+shGoLZtMSHnveaUnLMm58uNUOzSHEON36i49vSjbVm1dRKueoOVCWu9irlY0QEcuWdPApvo6pBRjQxa96t5FpOwml9crmd9of0scplGyGwDDR848szKdcmJ3lA89n8C3dMzJkZP3lPt91W/gQTYo4CFmrvzUH4cAiYLdGWvwPs+TNpHNgfv1kdrd7Mk76Uzs1AILfNeNj07DDYjShg+3/MXPes3DMAskJT6RXmXieAmOF0on7UDN71HkM4krOa96JjIXwjxcC/2wzpsO5l2FdmOQkJdtYS1r5PMoLupF2LNSYKb9VF14nSLeqINzZkfPiApfjOFyMENnWjBRs26Jm9d0ED1V/e9/QfizfuoDmnTQVH8+Zyny+SBXX6CVSDvmb2rv5Pv0jd7laydVVM5bAf5CgiYq1s917Kd7lWoQvvJBzJDpeZfK4YbP4r8C8= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR12MB3176.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(136003)(366004)(376002)(346002)(396003)(39860400002)(451199015)(86362001)(36756003)(2906002)(2616005)(6506007)(7416002)(6666004)(6512007)(26005)(38100700002)(83380400001)(66574015)(54906003)(316002)(186003)(966005)(8936002)(5660300002)(4326008)(66556008)(8676002)(66946007)(66476007)(6486002)(478600001)(41300700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?KEJIdETGu2OBGCExpYQFxQFbtTj/?= =?utf-8?q?5/BlEyzNGq5Ik7QQG4YhV4xYvlIONPqW6X4xPieslTMIeSKZ1BAtKCS6v0lh1VUHZ?= =?utf-8?q?Sh3EsAlEoTT3e2UV91qd0pF0am4L36CyUzG7foMDFT6IKc0xcbZefY3UIJ9fpSlGu?= =?utf-8?q?AGqDkGHgEttR+tZ5pW7DLeZ4bfzpaZFzXNwKDgBAKU9kJbDqS56zI4PArTedxLlEr?= =?utf-8?q?vdX3hxez0zFBIzopx47hbWDF3zYMMnaOKSqc+FwFacx/SCnGx2N3s4V6515loeQKW?= =?utf-8?q?6pvDY/DF44b9PUByC5q4JgZwDqTlb2KkJ4mrTC8NKfA+akVanliiJZIF4HhefUZE5?= =?utf-8?q?ZEoVPsTVCzM9Lvq/A3RBLIXevLNZpFI5Ey1SW43I14/xGr0ZGsMub8OSZrwgLh+5e?= =?utf-8?q?riGEf22AQZsn902gbSK1VtSkBADfP13J2noPHQrld5n9oItsnAUXj0XKqyCgSygwf?= =?utf-8?q?E3xJHP0z5hxdtq6OJpvEJJ+i+mXdVNPn9TZhA0BEUjO7PZPBkNq4UWXSfImbTH+I0?= =?utf-8?q?q382XYtBOcJP1hFM3CT0XxVPMrkdogieLcW668ZlmdtcshcbVi9Ff9twhfqDnebrq?= =?utf-8?q?13+nQilU9y0hFRwseXtWCQa5/5auRCtuc52lrE9b5twbUBsxJLxCOCbBsCIQOqBrs?= =?utf-8?q?31Sk6zxksCWUeiwUM9xivKl+gwcTae3xG2eMGAO5nalqVqjF5a700H+FpwBS9Zw6y?= =?utf-8?q?bEITlJjVXFSzyzh2QR8i7ukvMRfWp93yl3XWel3empuQrIYy90GArbgqlZveKtVwU?= =?utf-8?q?Bjt13RN3NbsN8QHcnKxaj542WoziJIGla1gG7hpW5gBcRpetkSEg95UjcwpaiTCKC?= =?utf-8?q?1u78n3SWGZmki78Duaz3fGZc2hgYQlGE91HLyDwm1NPV00kMhRVuAOsk2r3fPN2wO?= =?utf-8?q?DigoLVlhcgQw5x5aaoRCymc7EScy98Iw8z6w2qfupDmXUy/5100Mvlx8FmeBJiCdf?= =?utf-8?q?CndE6O8mZ6QlBHG6xF2oAgK+jeiNj2DYH7EBzGckV4DohvQkioDhbzXQG7RBMAGBE?= =?utf-8?q?OrGJEy8sQ2QECbegPg4bmLR7U9Ml+6SvYqleLpOkUwGi1ecGvteDkMqCjm5JygubN?= =?utf-8?q?A9E6qZdEY215VlLf66mdHs/7Ecy+5/YSIbGIrYY3Ca+2Q1vgmqdK3tXpPXvoQQ1CF?= =?utf-8?q?QOBPaYKfe3epsuxyNei/K13n4UK3Bf0YCbNq7TtkqQmXP5wSC54yB1suTpaqMToA1?= =?utf-8?q?ZReXzvMmzs8IBX/qZ07rwDD+HxAP5k/8NhII2AStvWzYA5DSwfx3QduMvcsJCYZ9M?= =?utf-8?q?aIVNcCzKwbpDyUZakXFXU+Lnqx6gzgWRYysjdhAACQaO8ciQfKqvfHMMGfJkdrTTc?= =?utf-8?q?jXOUOklYwTG5Ek7lEODFxPTDxOSMuLKd2BzrFIEKnMflhFDSsyNb34owoqhQp2Cjp?= =?utf-8?q?UvGfi/sS4jUqTc7dJsJiQ61q7yAWGrMH+ymlSAf2hB/VsUYXeZITTxhHdDUIRALkF?= =?utf-8?q?wq9VZ9IOabeac9AJo7/6pYDPQc/zTScQKYnt/3MTQzNiP+yUAcCKF/euchmJ4MYyr?= =?utf-8?q?D+aQU7fbccuR?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45404f4b-7570-49e7-ff4e-08daa14939f9 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2022 12:01:50.5122 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rOqWpQclmsW8U5eqg93LySU3wWOVAMBgMIMXWLyjkSVGLgCGd1D5QtSsjNI+0+chw+ImlZDaX1J94EI4+YS6gg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4337 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Ralph Campbell , John Hubbard , nouveau@lists.freedesktop.org, Felix Kuehling , Alistair Popple , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Jason Gunthorpe , Michael Ellerman , Alex Deucher , Dan Williams , =?utf-8?q?Christian_K=C3=B6nig?= , Ben Skeggs Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since 27674ef6c73f ("mm: remove the extra ZONE_DEVICE struct page refcount") device private pages have no longer had an extra reference count when the page is in use. However before handing them back to the owning device driver we add an extra reference count such that free pages have a reference count of one. This makes it difficult to tell if a page is free or not because both free and in use pages will have a non-zero refcount. Instead we should return pages to the drivers page allocator with a zero reference count. Kernel code can then safely use kernel functions such as get_page_unless_zero(). Signed-off-by: Alistair Popple Cc: Jason Gunthorpe Cc: Michael Ellerman Cc: Felix Kuehling Cc: Alex Deucher Cc: Christian König Cc: Ben Skeggs Cc: Lyude Paul Cc: Ralph Campbell Cc: Alex Sierra Cc: John Hubbard Cc: Dan Williams Acked-by: Felix Kuehling --- This will conflict with Dan's series to fix reference counts for DAX[1]. At the moment this only makes changes for device private and coherent pages, however if DAX is fixed to remove the extra refcount then we should just be able to drop the checks for private/coherent pages and treat them the same. [1] - https://lore.kernel.org/linux-mm/166329930818.2786261.6086109734008025807.stgit@dwillia2-xfh.jf.intel.com/ --- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- include/linux/memremap.h | 1 + lib/test_hmm.c | 2 +- mm/memremap.c | 9 +++++++++ mm/page_alloc.c | 8 ++++++++ 7 files changed, 22 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index d4eacf4..9d8de68 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -718,7 +718,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage = pfn_to_page(uvmem_pfn); dpage->zone_device_data = pvt; - lock_page(dpage); + zone_device_page_init(dpage); return dpage; out_clear: spin_lock(&kvmppc_uvmem_bitmap_lock); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 776448b..97a6845 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -223,7 +223,7 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) page = pfn_to_page(pfn); svm_range_bo_ref(prange->svm_bo); page->zone_device_data = prange->svm_bo; - lock_page(page); + zone_device_page_init(page); } static void diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 1635661..b092988 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -326,7 +326,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm) return NULL; } - lock_page(page); + zone_device_page_init(page); return page; } diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 1901049..f68bf6d 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -182,6 +182,7 @@ static inline bool folio_is_device_coherent(const struct folio *folio) } #ifdef CONFIG_ZONE_DEVICE +void zone_device_page_init(struct page *page); void *memremap_pages(struct dev_pagemap *pgmap, int nid); void memunmap_pages(struct dev_pagemap *pgmap); void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap); diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 89463ff..688c15d 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -627,8 +627,8 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) goto error; } + zone_device_page_init(dpage); dpage->zone_device_data = rpage; - lock_page(dpage); return dpage; error: diff --git a/mm/memremap.c b/mm/memremap.c index 25029a4..1c2c038 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -505,8 +505,17 @@ void free_zone_device_page(struct page *page) /* * Reset the page count to 1 to prepare for handing out the page again. */ + if (page->pgmap->type != MEMORY_DEVICE_PRIVATE && + page->pgmap->type != MEMORY_DEVICE_COHERENT) + set_page_count(page, 1); +} + +void zone_device_page_init(struct page *page) +{ set_page_count(page, 1); + lock_page(page); } +EXPORT_SYMBOL_GPL(zone_device_page_init); #ifdef CONFIG_FS_DAX bool __put_devmap_managed_page_refs(struct page *page, int refs) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9d49803..4df1e43 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6744,6 +6744,14 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, set_pageblock_migratetype(page, MIGRATE_MOVABLE); cond_resched(); } + + /* + * ZONE_DEVICE pages are released directly to the driver page allocator + * which will set the page count to 1 when allocating the page. + */ + if (pgmap->type == MEMORY_DEVICE_PRIVATE || + pgmap->type == MEMORY_DEVICE_COHERENT) + set_page_count(page, 0); } /*