From patchwork Thu Mar 20 17:39:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 14024211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FA0FC36002 for ; Thu, 20 Mar 2025 17:39:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDD75280007; Thu, 20 Mar 2025 13:39:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E89EA280006; Thu, 20 Mar 2025 13:39:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDF8A280007; Thu, 20 Mar 2025 13:39:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AD6FD280006 for ; Thu, 20 Mar 2025 13:39:40 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 52756161825 for ; Thu, 20 Mar 2025 17:39:41 +0000 (UTC) X-FDA: 83242641762.30.CC3706D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf16.hostedemail.com (Postfix) with ESMTP id 7AA53180016 for ; Thu, 20 Mar 2025 17:39:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mIlX91oQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 32lLcZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=32lLcZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742492379; a=rsa-sha256; cv=none; b=Dg46GsxhjjiUycbLYIEb/D55LQsiTExEPLBXgta8wD6pJ5JopXxDFZCJ6alhD4KEnXgVe4 go1hLT1rd/eTWmLtgPRgG0W5T1S4KLhbZ5FGcvG9dU60advbQXfD8v9fkJ3I4gvd/o6gMy 6W/wc/lXUwsO02rRSgiaeTZ2/NgCXNA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mIlX91oQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 32lLcZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=32lLcZwYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742492379; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=utO18jM3DqI43jAs46xftkc+iTkKZEsg2nm4Y+WIpWw=; b=BwnhLLKLkXER6cLpTHIxo3hc6WidjOo/7hJCULnBrBeht7ZZwjPuR+N0bNv2jHh7YkGsYm +mv/teirASA2j7BpSPrsgLuzGo9vEv5sQCfAzSzn2QfH/GYFNZHtWzMja17AqXfa7nNng+ 3QxWHZAzXGVd34zyvCA0vRao/cq4jGA= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ff4b130bb2so1508175a91.0 for ; Thu, 20 Mar 2025 10:39:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742492378; x=1743097178; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=utO18jM3DqI43jAs46xftkc+iTkKZEsg2nm4Y+WIpWw=; b=mIlX91oQ4FwxGcJ8LogEQnk2fP61iueFelOzaiZ7FQQnjX8KF1FKmul7cCYtZbsh5g 6YtzKMTtOahuQpdTC+llY1NwITccQKbSzQj3eywQmyw5fnxh6BUyysRZVgNuUoaCi/74 C5u/SOL2JgXRKeuE4QFexasZ++fk5wub2YyoPlLFDP8xg02wU4kEPR8kj7JAvVrgtc+S Fx//xOFC8MyRL8Y2HhpJS0Fg1CUZFxT5LVk9jWZqbhUbkmZ6j9yA4E7QcbnytdHLIkwk Tr1kOFjdAjyOvuIdGWckZIAdSsjWAN1gcV1BCwwG+5nDilR2d+XXBNR1m24KokvvE+aX m/5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742492378; x=1743097178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=utO18jM3DqI43jAs46xftkc+iTkKZEsg2nm4Y+WIpWw=; b=q83POjayRcwROEVSsgEq3o+DGa5f3efctPKaFes+Woh0WpysCwMmDjV9oji55shzjq ok6AM19IZcka1L8hjC6kdH8cRUgxasHh2pdHyTmN5QA088LDEL8sOcIwAvB9Zpj11/zv tTIkKxo6xEBKIYN/KBWnUy82YWu7q75hfwEgQqEJS5mbAeqLxgerMraMaXDOuYGu2dh6 46rTndw9HG7G39T25Y9BqwSjHjAePjZTjscgoKThTkn49Q+4g9VMaeJOIo4SRNXkJiH8 jHPHR2XsE9248xilrBhxEA09Cd39x1gZZAjaOeIe/NLlbS5wj4t0zF2O0d+05ROdrjRs s/Xg== X-Forwarded-Encrypted: i=1; AJvYcCWReBJL2KT5XEVoyzWqHsjQsaPeikscik4eEFsiQctlGsABVeQY+QQdu0AYXqW70d4Z24cOwEJ1KQ==@kvack.org X-Gm-Message-State: AOJu0YzBpWbmtjRVzoPepiPjKv0IcfNktRnkNLvb5S1Sgx+fTAwE+XNn EZcUQb9CKNb94j8595ktvEo5/0j8kGz3qPQ24iHjMN3IVQKmxu3o2Tm9tJ4cqIHkj1/X2NL8DRE ZYA== X-Google-Smtp-Source: AGHT+IGjhRps2uk6ZFQZIC7R3fpxDHdrRFbLqUZ0BLeU3xJE1wYlTDOJepff3lsj9OvAu2ohK9dAI+Xpphc= X-Received: from pjf11.prod.google.com ([2002:a17:90b:3f0b:b0:2fa:1803:2f9f]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2743:b0:2ff:5a9d:9390 with SMTP id 98e67ed59e1d1-3030fe779bbmr123144a91.8.1742492378297; Thu, 20 Mar 2025 10:39:38 -0700 (PDT) Date: Thu, 20 Mar 2025 10:39:30 -0700 In-Reply-To: <20250320173931.1583800-1-surenb@google.com> Mime-Version: 1.0 References: <20250320173931.1583800-1-surenb@google.com> X-Mailer: git-send-email 2.49.0.395.g12beb8f557-goog Message-ID: <20250320173931.1583800-3-surenb@google.com> Subject: [RFC 2/3] mm: introduce GCMA From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, david@redhat.com, vbabka@suse.cz, lorenzo.stoakes@oracle.com, liam.howlett@oracle.com, alexandru.elisei@arm.com, peterx@redhat.com, hannes@cmpxchg.org, mhocko@kernel.org, m.szyprowski@samsung.com, iamjoonsoo.kim@lge.com, mina86@mina86.com, axboe@kernel.dk, viro@zeniv.linux.org.uk, brauner@kernel.org, hch@infradead.org, jack@suse.cz, hbathini@linux.ibm.com, sourabhjain@linux.ibm.com, ritesh.list@gmail.com, aneesh.kumar@kernel.org, bhelgaas@google.com, sj@kernel.org, fvdl@google.com, ziy@nvidia.com, yuzhao@google.com, minchan@kernel.org, surenb@google.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Minchan Kim X-Rspamd-Server: rspam07 X-Rspam-User: X-Stat-Signature: stwfy1m679qpth9ncbf4w9x3o1p586ij X-Rspamd-Queue-Id: 7AA53180016 X-HE-Tag: 1742492379-344436 X-HE-Meta: U2FsdGVkX1+FS5Xfc2DESk0HOXwhdcBMXnEcS/T8jAXlQdKE00baZpv5BhAl3KP7PYOEN+2ZT7oARp3hfIx/JmNgPSCt2RRtRU66YQHqLpdkF+G4PukZKCZMUteYASuW71UMHMumF3ixRLgsDxaZyTw8Qdb4bzY+RZXQRoj4tYtwFvEhzoincd5IlPW4UFGTqNXQNataDtrppotHW291hiwaHxG70s4oFddUPVqjeXTkQ2n8LFJkEcvlGyRohznP+tFHzl7sUI2DY7oovW8rc9On+FOFohjEyjwLbBE1745MBSUdmIjTes5gOli4i3rKX7Y/oSFRAa0ZkqPnpJ11zwAOQdEKyD8ps7aUq6sJoYRhcN43VP9jrEUdxiYaOPJv4HE3BpEhJV9I6i+QJHCd9pr+tDv7dUY6irX8uImdahB2tYUGkBOxXEV9TWtyjTOPui95ZeMtlOConUpec6w/F6Rubo0j1XhMXqe5Kexf4X5L6k1FPL9Cde+pEsS8/BZb0cpGXqrIjDsh/BahtpqpYsgiUToZca37rLVHgqgi+wqaMq+KN1zmmAXZFJtAu9C1h0Q05GjkIFJ7FSB79N+KmHxKsGp4sKT6EmhnT7ku/r8kHMsYFdCdW6yTLffLu5AdJmS9qKadVqkLhfnGgCQpF3rfbXAJ87kVgr38QyBKx3wYWbWu9x8Y5kGgNTurKnWsBrmipb6Eafu7Xr2JrZbYzUe4CpjLlMXea9/DHOUDJquFt5Psh/M1hHtuf+i+41xVeOJLlmuUjls2WUnzQKn+xBPRt1S7xVmYjZegyhiY9eui8flnvzkgQh2PskTAdS4aH0tuNW3qmVB3VNEL8rP9TbXRk5KlgsACzgXZWNZcE381WSu96tGjfEUNemJPukxgXA6ZFM02Qs/JBw5ty1OiceAV7h/YI0fLegdHzvDiOPE9iQ3Xmp/ftb6uYZKb3vI6iqjXRKDXmzSsed2rV0V G2jawbdt hrku+hACrOjUD4n1xApp2lcF5aZts9xKkWtiA7tYmqPBvqaP01vCw6Aer93ALIrYX1HuM0GfFGPxA/36AkYSaayMUMlKn6MQdCHIMfP+hdjIZjNIQ9RFJacFih+WaLSXPHImR8wiL7Kbb9EggHkpuV8bO/SPOm8/50X7pUmgzu5+ghHC8zchlEZtiC3VZyp9uFhh9fmsKHdRgbJ36OYBXplBhzw9MdBkEIrI/2lC3AImJJ+m/+QFj9VtSfEe8WKP9zFuTj1PCSWNlqYVolOxWvpHxoRJzCHII6cn4pvBG0Xs+DxSauypYNlVu6MwmWPTFaUqFBv42IDFARUvWvziqpWAN2OUNiPbYW+pZFCzYaGTSLtTJoNGyUUJbqUNie3sWv8B2tfKtiAagUGWHlkMagMnL+K8xv/GqJme+fk807xa1oRFHM4cuYGKZ5dnhBogKDHxvPSwWkgpjIYqEaGgOEsJ8ZhZ4P4Mo9vd4ypsYH0isbQwrCdDtxXUhA1ws1LMu6/YYTFdGT+W2Ec/ceBblA4XRceHDpHQ2sNGVPki83bB7TRj4GZVprcvBNfllfTR4yxVj429GRRKhIjf80L7UHkqRzbIoKJfXCfUr7Rb3BPWrq96aHLnpXbIhQ3u6Bo4VPDdUloxpH9f4kDXfp//hNy31iggTLAYiFS3ISOy9h3Div2EvzgVVhXaqi6KeObmBNhLmLgPDdX/CmaOKuRdlprlDqk9FKsBwyT8J16+mSSrFiCw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Minchan Kim This patch introduces GCMA (Guaranteed Contiguous Memory Allocator) cleacache backend which reserves some amount of memory at the boot and then donates it to store clean file-backed pages in the cleancache. GCMA aims to guarantee contiguous memory allocation success as well as low and deterministic allocation latency. Notes: Originally, the idea was posted by SeongJae Park and Minchan Kim [1]. Later Minchan reworked it to be used in Android as a reference for Android vendors to use [2]. [1] https://lwn.net/Articles/619865/ [2] https://android-review.googlesource.com/q/topic:%22gcma_6.12%22 Signed-off-by: Minchan Kim Signed-off-by: Suren Baghdasaryan --- include/linux/gcma.h | 12 ++++ mm/Kconfig | 15 +++++ mm/Makefile | 1 + mm/gcma.c | 155 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+) create mode 100644 include/linux/gcma.h create mode 100644 mm/gcma.c diff --git a/include/linux/gcma.h b/include/linux/gcma.h new file mode 100644 index 000000000000..2ce40fcc74a5 --- /dev/null +++ b/include/linux/gcma.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __GCMA_H__ +#define __GCMA_H__ + +#include + +int gcma_register_area(const char *name, + unsigned long start_pfn, unsigned long count); +void gcma_alloc_range(unsigned long start_pfn, unsigned long count); +void gcma_free_range(unsigned long start_pfn, unsigned long count); + +#endif /* __GCMA_H__ */ diff --git a/mm/Kconfig b/mm/Kconfig index d6ebf0fb0432..85268ef901b6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1002,6 +1002,21 @@ config CMA_AREAS If unsure, leave the default value "8" in UMA and "20" in NUMA. +config GCMA + bool "GCMA (Guaranteed Contiguous Memory Allocator)" + depends on CLEANCACHE + help + This enables the Guaranteed Contiguous Memory Allocator to allow + low latency guaranteed contiguous memory allocations. Memory + reserved by GCMA is donated to cleancache to be used as pagecache + extension. Once GCMA allocation is requested, necessary pages are + taken back from the cleancache and used to satisfy the request. + Cleancache guarantees low latency successful allocation as long + as the total size of GCMA allocations does not exceed the size of + the memory donated to the cleancache. + + If unsure, say "N". + config MEM_SOFT_DIRTY bool "Track memory changes" depends on CHECKPOINT_RESTORE && HAVE_ARCH_SOFT_DIRTY && PROC_FS diff --git a/mm/Makefile b/mm/Makefile index 084dbe9edbc4..2173d395d371 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -149,3 +149,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o obj-$(CONFIG_CLEANCACHE) += cleancache.o +obj-$(CONFIG_GCMA) += gcma.o diff --git a/mm/gcma.c b/mm/gcma.c new file mode 100644 index 000000000000..263e63da0c89 --- /dev/null +++ b/mm/gcma.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * GCMA (Guaranteed Contiguous Memory Allocator) + * + */ + +#define pr_fmt(fmt) "gcma: " fmt + +#include +#include +#include +#include +#include +#include +#include + +#define MAX_GCMA_AREAS 64 +#define GCMA_AREA_NAME_MAX_LEN 32 + +struct gcma_area { + int area_id; + unsigned long start_pfn; + unsigned long end_pfn; + char name[GCMA_AREA_NAME_MAX_LEN]; +}; + +static struct gcma_area areas[MAX_GCMA_AREAS]; +static atomic_t nr_gcma_area = ATOMIC_INIT(0); +static DEFINE_SPINLOCK(gcma_area_lock); + +static void alloc_page_range(struct gcma_area *area, + unsigned long start_pfn, unsigned long end_pfn) +{ + unsigned long scanned = 0; + unsigned long pfn; + struct page *page; + int err; + + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + if (!(++scanned % XA_CHECK_SCHED)) + cond_resched(); + + page = pfn_to_page(pfn); + err = cleancache_backend_get_folio(area->area_id, page_folio(page)); + VM_BUG_ON(err); + } +} + +static void free_page_range(struct gcma_area *area, + unsigned long start_pfn, unsigned long end_pfn) +{ + unsigned long scanned = 0; + unsigned long pfn; + struct page *page; + int err; + + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + if (!(++scanned % XA_CHECK_SCHED)) + cond_resched(); + + page = pfn_to_page(pfn); + err = cleancache_backend_put_folio(area->area_id, + page_folio(page)); + VM_BUG_ON(err); + } +} + +int gcma_register_area(const char *name, + unsigned long start_pfn, unsigned long count) +{ + LIST_HEAD(folios); + int i, area_id; + int nr_area; + int ret = 0; + + for (i = 0; i < count; i++) { + struct folio *folio; + + folio = page_folio(pfn_to_page(start_pfn + i)); + list_add(&folio->lru, &folios); + } + + area_id = cleancache_register_backend(name, &folios); + if (area_id < 0) + return area_id; + + spin_lock(&gcma_area_lock); + + nr_area = atomic_read(&nr_gcma_area); + if (nr_area < MAX_GCMA_AREAS) { + struct gcma_area *area = &areas[nr_area]; + + area->area_id = area_id; + area->start_pfn = start_pfn; + area->end_pfn = start_pfn + count; + strscpy(area->name, name); + /* Ensure above stores complete before we increase the count */ + atomic_set_release(&nr_gcma_area, nr_area + 1); + } else { + ret = -ENOMEM; + } + + spin_unlock(&gcma_area_lock); + + return ret; +} +EXPORT_SYMBOL_GPL(gcma_register_area); + +void gcma_alloc_range(unsigned long start_pfn, unsigned long count) +{ + int nr_area = atomic_read_acquire(&nr_gcma_area); + unsigned long end_pfn = start_pfn + count; + struct gcma_area *area; + int i; + + for (i = 0; i < nr_area; i++) { + unsigned long s_pfn, e_pfn; + + area = &areas[i]; + if (area->end_pfn <= start_pfn) + continue; + + if (area->start_pfn > end_pfn) + continue; + + s_pfn = max(start_pfn, area->start_pfn); + e_pfn = min(end_pfn, area->end_pfn); + alloc_page_range(area, s_pfn, e_pfn); + } +} +EXPORT_SYMBOL_GPL(gcma_alloc_range); + +void gcma_free_range(unsigned long start_pfn, unsigned long count) +{ + int nr_area = atomic_read_acquire(&nr_gcma_area); + unsigned long end_pfn = start_pfn + count; + struct gcma_area *area; + int i; + + for (i = 0; i < nr_area; i++) { + unsigned long s_pfn, e_pfn; + + area = &areas[i]; + if (area->end_pfn <= start_pfn) + continue; + + if (area->start_pfn > end_pfn) + continue; + + s_pfn = max(start_pfn, area->start_pfn); + e_pfn = min(end_pfn, area->end_pfn); + free_page_range(area, s_pfn, e_pfn); + } +} +EXPORT_SYMBOL_GPL(gcma_free_range);