From patchwork Fri Jun 7 12:38:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13689835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7684C27C53 for ; Fri, 7 Jun 2024 12:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DE136B00B3; Fri, 7 Jun 2024 08:41:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58D456B00B4; Fri, 7 Jun 2024 08:41:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4098E6B00B5; Fri, 7 Jun 2024 08:41:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1D6D76B00B3 for ; Fri, 7 Jun 2024 08:41:45 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CCB3914047D for ; Fri, 7 Jun 2024 12:41:44 +0000 (UTC) X-FDA: 82204054128.26.009BE60 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf15.hostedemail.com (Postfix) with ESMTP id 6560BA0004 for ; Fri, 7 Jun 2024 12:41:42 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf15.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717764103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LAM7fkxS0eW5kauOyLML4k0MuzeYOhhEhnKziaG//wI=; b=nLlnRG9oBu0WMekKYKalPrdF6uw/jVpc7zgCQq/LSxEkbJ9m/IkD25/RAaeQLi8Lr8crSa 62JkrHxp1/+fk5uMwnu/70Fijo+kl25WEs/pNCSstyGWRKImnPnsDrrYN/klpIf0b5/3hH vPyXpyhIVhDMX7qMD7GPOdk1nCwGo4Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717764103; a=rsa-sha256; cv=none; b=2bQwRasgTzlPjqPlNN9Im0RPftwiylog8vslKELIZQLga6ACZJEPjdzFRv98eeKUYiZySy RK8wEdNnMLtKSBFs7wSjtoP73/J+P7+9nNiQY4mo+7SZLtx557UZ7ZPcQKKryIgtoYwNj4 r1tMcwwBzL8KNNpS671qZbH8Uir/ftM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf15.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4VwghN3SD9zwSJH; Fri, 7 Jun 2024 20:37:40 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id D544B18007A; Fri, 7 Jun 2024 20:41:39 +0800 (CST) Received: from localhost.localdomain (10.69.192.56) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Fri, 7 Jun 2024 20:41:39 +0800 From: Yunsheng Lin To: , , CC: , , Yunsheng Lin , Alexander Duyck , Andrew Morton , Subject: [PATCH net-next v7 12/15] mm: page_frag: introduce prepare/probe/commit API Date: Fri, 7 Jun 2024 20:38:15 +0800 Message-ID: <20240607123819.40694-13-linyunsheng@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20240607123819.40694-1-linyunsheng@huawei.com> References: <20240607123819.40694-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.56] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemf200006.china.huawei.com (7.185.36.61) X-Stat-Signature: oi6zk1cud448fspqqpucayyhiwxdo819 X-Rspamd-Queue-Id: 6560BA0004 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717764102-398554 X-HE-Meta: U2FsdGVkX19z7LVnjvdLDABrc7VT6JnK/fHJ5/sCJpdpYcOfjbzBPNTJ64DeVnB6Q2QVs9ydgQ5I4aWLLuadKcKT0WB434jv/3cirHpiaeQPqFbv5CyrrOxzc3zLlzb/GVtAjboIFip1RiwyfknSXkipvwpj/PFoPolREj2G/aGqNgIGzd/miam9/T3i2IRuRa3bB+5WaQSdD5NJKBur7gipyKYXujhfE8lm3HE+7cRK7v3WZF30ruw8diaMxNKtFmhES7NZG22q/+M+lJlCQ+Zm7lnIxuZoGiCuhG7z6t6g23hqv1L7clUu1iFYnL3mNJb21B72WMAokPlN919QmTzZXPbhCH5JJuRlEI93TXrx7Xpn8x5KejMlmuax8Q2xggbBZiPcKLCEp/ZL15nDrekmwThhMU+WNfsaigVzWRQrEi66Qeyg8y/DzGD09sRDdyiX2uz+fd+gjqMLsRzSRtNniK1WrePcQLUdwk3Enyw+BI1xrV9hHSme7uoVOW4TMxn/Gkc6bgaPvG0TiCA9AaTiuWDHOD5UqWj/g9GqvluEm2Ig/0djeFpKF7k4qGKduLccEDKlVT+dlJTR1TSB65ulOcobGhMRTMWTqM9F8F2b8bT3INGr2Zs9ybrrYqUdM8zr3U1hs3N/4LR4k1xg2o7QjWIXGo36Lr2co2A9mXezJkEmA2DaGjJM014LUaVE/6TEv6dZX+DuYOHv/xR8EphJculresLvRN0qk/tcT9O/66mCbabtvBEIIks51DEjkkjbWZ73ussGtwnDXFVeELSpKM5Qy4KZ/s0n521EwfDCdqK1LddWpQppq4m8GffD70xyQZ73ACb+p43lnsthjItOHJD94fxG/cPsYc8VQum50Fm6WtPRXKLz/LTrB7oNf0PlmdAVZZwq0UosZ12LU6hCNUjthTUDA5AIh10z5i+CeuceR+hrL8WwtyHhcDh15IzDOsaic4nQ4cozW+T EUJhnwjC b79oanLNDo34lOruOPw1DThAQg183VfstY6kb7YBA76gN8xQNqKz7sXgHxRXDT0/vSzoAjkA/oULS5jOecqa6hL3Q/MMKpdIfTFks5/xqzn8C8vpDkg8opx0yMp4WRacY7fHoSfLMXlyyFP4LDpCiCTXQaieYoPRuWC9rBZK4Y3DIGG0qQWnBp9TPN5r6n0K6Z6CR4bec6Y0Pc6Mu5urvVxVHXzcNI8akxRfCTW9N57nwcaZJgTe3CkfrAO3ae9uKf5D5s/IK492a2zKDvzvcxL+p0RN/HuUzXwZN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are many use cases that need minimum memory in order for forward progress, but more performant if more memory is available or need to probe the cache info to use any memory available for frag caoleasing reason. Currently skb_page_frag_refill() API is used to solve the above use cases, but caller needs to know about the internal detail and access the data field of 'struct page_frag' to meet the requirement of the above use cases and its implementation is similar to the one in mm subsystem. To unify those two page_frag implementations, introduce a prepare API to ensure minimum memory is satisfied and return how much the actual memory is available to the caller and a probe API to report the current available memory to caller without doing cache refilling. The caller needs to either call the commit API to report how much memory it actually uses, or not do so if deciding to not use any memory. As next patch is about to replace 'struct page_frag' with 'struct page_frag_cache' in linux/sched.h, which is included by the asm-offsets.s, using the virt_to_page() in the inline helper of page_frag_cache.h cause a "'vmemmap' undeclared" compiling error for asm-offsets.s, use a macro for probe API to avoid that compiling error. CC: Alexander Duyck Signed-off-by: Yunsheng Lin --- include/linux/page_frag_cache.h | 82 +++++++++++++++++++++++ mm/page_frag_cache.c | 114 ++++++++++++++++++++++++++++++++ 2 files changed, 196 insertions(+) diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h index b33904d4494f..e95d44a36ec9 100644 --- a/include/linux/page_frag_cache.h +++ b/include/linux/page_frag_cache.h @@ -4,6 +4,7 @@ #define _LINUX_PAGE_FRAG_CACHE_H #include +#include #define PAGE_FRAG_CACHE_MAX_SIZE __ALIGN_MASK(32768, ~PAGE_MASK) #define PAGE_FRAG_CACHE_MAX_ORDER get_order(PAGE_FRAG_CACHE_MAX_SIZE) @@ -87,6 +88,9 @@ static inline unsigned int page_frag_cache_page_size(struct encoded_va *encoded_ void page_frag_cache_drain(struct page_frag_cache *nc); void __page_frag_cache_drain(struct page *page, unsigned int count); +struct page *page_frag_alloc_pg(struct page_frag_cache *nc, + unsigned int *offset, unsigned int fragsz, + gfp_t gfp); void *__page_frag_alloc_va_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask); @@ -99,12 +103,90 @@ static inline void *page_frag_alloc_va_align(struct page_frag_cache *nc, return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, -align); } +static inline unsigned int page_frag_cache_page_offset(const struct page_frag_cache *nc) +{ + return page_frag_cache_page_size(nc->encoded_va) - nc->remaining; +} + static inline void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask) { return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, ~0u); } +void *page_frag_alloc_va_prepare(struct page_frag_cache *nc, unsigned int *fragsz, + gfp_t gfp); + +static inline void *page_frag_alloc_va_prepare_align(struct page_frag_cache *nc, + unsigned int *fragsz, + gfp_t gfp, + unsigned int align) +{ + WARN_ON_ONCE(!is_power_of_2(align) || align > PAGE_SIZE); + nc->remaining = nc->remaining & -align; + return page_frag_alloc_va_prepare(nc, fragsz, gfp); +} + +struct page *page_frag_alloc_pg_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, gfp_t gfp); + +struct page *page_frag_alloc_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va, gfp_t gfp); + +static inline struct encoded_va *__page_frag_alloc_probe(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va) +{ + struct encoded_va *encoded_va; + + *fragsz = nc->remaining; + encoded_va = nc->encoded_va; + *offset = page_frag_cache_page_size(encoded_va) - *fragsz; + *va = encoded_page_address(encoded_va) + *offset; + + return encoded_va; +} + +#define page_frag_alloc_probe(nc, offset, fragsz, va) \ +({ \ + struct page *__page = NULL; \ + \ + VM_BUG_ON(!*(fragsz)); \ + if (likely((nc)->remaining >= *(fragsz))) \ + __page = virt_to_page(__page_frag_alloc_probe(nc, \ + offset, \ + fragsz, \ + va)); \ + \ + __page; \ +}) + +static inline void page_frag_alloc_commit(struct page_frag_cache *nc, + unsigned int fragsz) +{ + VM_BUG_ON(fragsz > nc->remaining || !nc->pagecnt_bias); + nc->pagecnt_bias--; + nc->remaining -= fragsz; +} + +static inline void page_frag_alloc_commit_noref(struct page_frag_cache *nc, + unsigned int fragsz) +{ + VM_BUG_ON(fragsz > nc->remaining); + nc->remaining -= fragsz; +} + +static inline void page_frag_alloc_abort(struct page_frag_cache *nc, + unsigned int fragsz) +{ + nc->pagecnt_bias++; + nc->remaining += fragsz; +} + void page_frag_free_va(void *addr); #endif diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c index 525b577b03a9..9f86aa15bbeb 100644 --- a/mm/page_frag_cache.c +++ b/mm/page_frag_cache.c @@ -94,6 +94,120 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, return page; } +void *page_frag_alloc_va_prepare(struct page_frag_cache *nc, + unsigned int *fragsz, gfp_t gfp) +{ + struct encoded_va *encoded_va; + unsigned int remaining; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(!__page_frag_cache_refill(nc, gfp) || + *fragsz > PAGE_SIZE)) + return NULL; + + remaining = nc->remaining; + } + + encoded_va = nc->encoded_va; + *fragsz = remaining; + return encoded_page_address(encoded_va) + + page_frag_cache_page_size(encoded_va) - remaining; +} +EXPORT_SYMBOL(page_frag_alloc_va_prepare); + +struct page *page_frag_alloc_pg_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, gfp_t gfp) +{ + struct encoded_va *encoded_va; + unsigned int remaining; + struct page *page; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(*fragsz > PAGE_SIZE)) { + *fragsz = 0; + return NULL; + } + + page = __page_frag_cache_refill(nc, gfp); + remaining = nc->remaining; + encoded_va = nc->encoded_va; + } else { + encoded_va = nc->encoded_va; + page = virt_to_page(encoded_va); + } + + *offset = page_frag_cache_page_size(encoded_va) - remaining; + *fragsz = remaining; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_pg_prepare); + +struct page *page_frag_alloc_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va, gfp_t gfp) +{ + struct encoded_va *encoded_va; + unsigned int remaining; + struct page *page; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(*fragsz > PAGE_SIZE)) { + *fragsz = 0; + return NULL; + } + + page = __page_frag_cache_refill(nc, gfp); + remaining = nc->remaining; + encoded_va = nc->encoded_va; + } else { + encoded_va = nc->encoded_va; + page = virt_to_page(encoded_va); + } + + *offset = page_frag_cache_page_size(encoded_va) - remaining; + *fragsz = remaining; + *va = encoded_page_address(encoded_va) + *offset; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_prepare); + +struct page *page_frag_alloc_pg(struct page_frag_cache *nc, + unsigned int *offset, unsigned int fragsz, + gfp_t gfp) +{ + struct page *page; + + if (unlikely(fragsz > nc->remaining)) { + if (unlikely(fragsz > PAGE_SIZE)) + return NULL; + + page = __page_frag_cache_refill(nc, gfp); + if (unlikely(!page)) + return NULL; + + *offset = 0; + } else { + struct encoded_va *encoded_va = nc->encoded_va; + + page = virt_to_page(encoded_va); + *offset = page_frag_cache_page_size(encoded_va) - + nc->remaining; + } + + nc->remaining -= fragsz; + nc->pagecnt_bias--; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_pg); + void page_frag_cache_drain(struct page_frag_cache *nc) { if (!nc->encoded_va)