From patchwork Fri Jul 19 09:33:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13737065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02D54C3DA59 for ; Fri, 19 Jul 2024 09:37:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70BA56B00A5; Fri, 19 Jul 2024 05:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DFC96B00A6; Fri, 19 Jul 2024 05:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A81F6B00A7; Fri, 19 Jul 2024 05:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3A5E56B00A5 for ; Fri, 19 Jul 2024 05:37:26 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AD89F1A04EC for ; Fri, 19 Jul 2024 09:37:25 +0000 (UTC) X-FDA: 82355999250.09.A8EF603 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf06.hostedemail.com (Postfix) with ESMTP id 3E8B8180014 for ; Fri, 19 Jul 2024 09:37:22 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721381823; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P4RZEAqi34TO8H/JXzbCbGCoOSoytD2IxLH27RUFA2c=; b=y/ynKWiNqwYVh0cgoaoJ8+F6weAtldCOBhRN4Qp/tPtvp/69JUo/UrI6qEc/3PcipXbYJO fJ+1qsRb6LjaWmEgGyOFZlKPPprA425+D5hQn5xh8SATL+A5NaeHABD5HgIh4o5j+yJk26 lXTWc59tVCGoocCHi87QwJ1VaE6WREI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721381823; a=rsa-sha256; cv=none; b=dNAJIYhwwyP/3Z8l19LQQhHiXbrXyMgLDxSwrDcYjwVUtm+wbk6sNQfU/Ddmdxa80yxOVb 9QXpH+0v+FxJliudqNZ/GOQAQtkRIDu+vobrvUthWvKkxrV1da++u2HzSw1C0Um1cXWkAL 9atIspUAQQtDy1/5vO7oSfjS14jcHug= Received: from mail.maildlp.com (unknown [172.19.88.105]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4WQPfv6pFJzdj88; Fri, 19 Jul 2024 17:35:35 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id A2FB5140109; Fri, 19 Jul 2024 17:37:20 +0800 (CST) Received: from localhost.localdomain (10.69.192.56) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Fri, 19 Jul 2024 17:37:20 +0800 From: Yunsheng Lin To: , , CC: , , Yunsheng Lin , Alexander Duyck , Andrew Morton , Subject: [RFC v11 11/14] mm: page_frag: introduce prepare/probe/commit API Date: Fri, 19 Jul 2024 17:33:35 +0800 Message-ID: <20240719093338.55117-12-linyunsheng@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20240719093338.55117-1-linyunsheng@huawei.com> References: <20240719093338.55117-1-linyunsheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.56] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 3E8B8180014 X-Stat-Signature: 93615idsmj37imehjx48kr6oh8xsx3ad X-HE-Tag: 1721381842-29090 X-HE-Meta: U2FsdGVkX1+q4ntkUV5jpFnSNFC8J7D38uVsRiafRENnqJBOv7uqe0eqaZLgD0+gz+8zByKgXr5/1W79VphxIJffAP5ACtU3cYEyCihMffplyVsT6Shv33KuCR/rHfHVOt1Btsqm2r6Rra2/aIRmlsEjfW1KqnszTiv7JEG0jvKyS9LWaW2iS/u/AWM19X0I8Yieh+b6Xv28SZFEU1GKxAudvvCQ/HTQRTjCMqitbZe/4yzjtwrfdJGPLfKH6wQtXztfXkb5gTHEEB/5Y8+QL4Nh2xdAXkgWUbOr9NDzbbbNa863jwHYlsehMPHy7wbKB9JWK7G4tfNrP3EUq+acmEIhlmGfFCzRtuHxZWIqQpNO5PWp6L6nEskVEqI5azg2RUETC2TLbLZd7GzgUnv3Pc1pKUAUrDwfbj465UE8d3CFt1dFc1zitinTpNP3xDzSLqIDsC7R3T6Kb98XYXCUpXkxXuQD7h2hUNJzz6ZiKxYzp2dIcNfFBhZCOQQzmGvt+z5w5V1SIZXJ3mSJoqISgXPpHSskYu9+85/Ww9jHOF7SrIPWu1BnV8hbPizCZ6C0eBKO1BqnL2SDyszbYRQqzNkmll/gdqKP/n7SDb8Vso88Kfc5RuHI6cTEy2aNUwVfa1iE2PHLdjihomtXtnkmgtU4vctUJavpbJHQi1Sjl3WIyO1Rn8bKSwY4Cb8CVLVEvAjhRYzlC/BrNRxkJSV2fIJyyXl657RUxAXoEEo1+wySeXe+kSk18sjP81WxdhDkvAylIs9SX+tU/XQROUEMzABSreON5iaLczzi0DERQAyD9wrRJDb5ebRvlL06ZWvFvG4I/sTdtwUN28Vi+a1Uez8c/oA2s6mx6JPqJEySvJzZA2J2HMDARae73EfXOSiWbRDsEHaX1bvDVNMEo2YA6h+hmC3PH8niQTTlTnjlh+AL6bHuHGxfl87WY1kJKnyFijAz8+2qD6LlmxHcI55 3m136P3S r3168a5L+H3fnBtXX27KrdhtQl5XaeO4q0GRb/gE8GpS9fwRzCY/zTli2z/zrXxZcJp/6LCKI2uQ1D29st+S7RGw7gVMEaelBw8a9DslMkElQSoBfAStu/dwi260l9m5hTC/kvmpXR00QZJztT047yxkt/hzjg79g1Gy8hjbVHVn5SPlOlEiRRF0HjouOppUJ5kNPJtBlo1W3ttkMs/xEAffrqTik36Q6hcCHYKis5HEYzWVgbIRyCdt4mqqc5FA85ZchBND74t2K7vw1Yi7uJOQKRbHYdWEWWwsq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are many use cases that need minimum memory in order for forward progress, but more performant if more memory is available or need to probe the cache info to use any memory available for frag caoleasing reason. Currently skb_page_frag_refill() API is used to solve the above use cases, but caller needs to know about the internal detail and access the data field of 'struct page_frag' to meet the requirement of the above use cases and its implementation is similar to the one in mm subsystem. To unify those two page_frag implementations, introduce a prepare API to ensure minimum memory is satisfied and return how much the actual memory is available to the caller and a probe API to report the current available memory to caller without doing cache refilling. The caller needs to either call the commit API to report how much memory it actually uses, or not do so if deciding to not use any memory. CC: Alexander Duyck Signed-off-by: Yunsheng Lin --- include/linux/page_frag_cache.h | 76 +++++++++++++++++++++ mm/page_frag_cache.c | 114 ++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h index 5aa45de7a9a5..1242bfb53ccc 100644 --- a/include/linux/page_frag_cache.h +++ b/include/linux/page_frag_cache.h @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include #include @@ -65,6 +67,9 @@ static inline unsigned int page_frag_cache_page_size(unsigned long encoded_va) void page_frag_cache_drain(struct page_frag_cache *nc); void __page_frag_cache_drain(struct page *page, unsigned int count); +struct page *page_frag_alloc_pg(struct page_frag_cache *nc, + unsigned int *offset, unsigned int fragsz, + gfp_t gfp); void *__page_frag_alloc_va_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask); @@ -77,12 +82,83 @@ static inline void *page_frag_alloc_va_align(struct page_frag_cache *nc, return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, -align); } +static inline unsigned int page_frag_cache_page_offset(const struct page_frag_cache *nc) +{ + return page_frag_cache_page_size(nc->encoded_va) - nc->remaining; +} + static inline void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask) { return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, ~0u); } +void *page_frag_alloc_va_prepare(struct page_frag_cache *nc, unsigned int *fragsz, + gfp_t gfp); + +static inline void *page_frag_alloc_va_prepare_align(struct page_frag_cache *nc, + unsigned int *fragsz, + gfp_t gfp, + unsigned int align) +{ + WARN_ON_ONCE(!is_power_of_2(align)); + nc->remaining = nc->remaining & -align; + return page_frag_alloc_va_prepare(nc, fragsz, gfp); +} + +struct page *page_frag_alloc_pg_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, gfp_t gfp); + +struct page *page_frag_alloc_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va, gfp_t gfp); + +static inline struct page *page_frag_alloc_probe(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va) +{ + unsigned long encoded_va; + struct page *page; + + VM_BUG_ON(!*fragsz); + if (unlikely(nc->remaining < *fragsz)) + return NULL; + + *fragsz = nc->remaining; + encoded_va = nc->encoded_va; + *va = encoded_page_address(encoded_va); + page = virt_to_page(*va); + *offset = page_frag_cache_page_size(encoded_va) - *fragsz; + *va += *offset; + + return page; +} + +static inline void page_frag_alloc_commit(struct page_frag_cache *nc, + unsigned int fragsz) +{ + VM_BUG_ON(fragsz > nc->remaining || !nc->pagecnt_bias); + nc->pagecnt_bias--; + nc->remaining -= fragsz; +} + +static inline void page_frag_alloc_commit_noref(struct page_frag_cache *nc, + unsigned int fragsz) +{ + VM_BUG_ON(fragsz > nc->remaining); + nc->remaining -= fragsz; +} + +static inline void page_frag_alloc_abort(struct page_frag_cache *nc, + unsigned int fragsz) +{ + nc->pagecnt_bias++; + nc->remaining += fragsz; +} + void page_frag_free_va(void *addr); #endif diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c index 3f162e9d23ba..74d2a13d12a8 100644 --- a/mm/page_frag_cache.c +++ b/mm/page_frag_cache.c @@ -88,6 +88,120 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, return page; } +void *page_frag_alloc_va_prepare(struct page_frag_cache *nc, + unsigned int *fragsz, gfp_t gfp) +{ + unsigned long encoded_va; + unsigned int remaining; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(!__page_frag_cache_refill(nc, gfp) || + *fragsz > PAGE_SIZE)) + return NULL; + + remaining = nc->remaining; + } + + encoded_va = nc->encoded_va; + *fragsz = remaining; + return encoded_page_address(encoded_va) + + page_frag_cache_page_size(encoded_va) - remaining; +} +EXPORT_SYMBOL(page_frag_alloc_va_prepare); + +struct page *page_frag_alloc_pg_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, gfp_t gfp) +{ + unsigned long encoded_va; + unsigned int remaining; + struct page *page; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(*fragsz > PAGE_SIZE)) { + *fragsz = 0; + return NULL; + } + + page = __page_frag_cache_refill(nc, gfp); + remaining = nc->remaining; + encoded_va = nc->encoded_va; + } else { + encoded_va = nc->encoded_va; + page = virt_to_page((void *)encoded_va); + } + + *offset = page_frag_cache_page_size(encoded_va) - remaining; + *fragsz = remaining; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_pg_prepare); + +struct page *page_frag_alloc_prepare(struct page_frag_cache *nc, + unsigned int *offset, + unsigned int *fragsz, + void **va, gfp_t gfp) +{ + unsigned long encoded_va; + unsigned int remaining; + struct page *page; + + remaining = nc->remaining; + if (unlikely(*fragsz > remaining)) { + if (unlikely(*fragsz > PAGE_SIZE)) { + *fragsz = 0; + return NULL; + } + + page = __page_frag_cache_refill(nc, gfp); + remaining = nc->remaining; + encoded_va = nc->encoded_va; + } else { + encoded_va = nc->encoded_va; + page = virt_to_page((void *)encoded_va); + } + + *offset = page_frag_cache_page_size(encoded_va) - remaining; + *fragsz = remaining; + *va = encoded_page_address(encoded_va) + *offset; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_prepare); + +struct page *page_frag_alloc_pg(struct page_frag_cache *nc, + unsigned int *offset, unsigned int fragsz, + gfp_t gfp) +{ + struct page *page; + + if (unlikely(fragsz > nc->remaining)) { + if (unlikely(fragsz > PAGE_SIZE)) + return NULL; + + page = __page_frag_cache_refill(nc, gfp); + if (unlikely(!page)) + return NULL; + + *offset = 0; + } else { + unsigned long encoded_va = nc->encoded_va; + + page = virt_to_page((void *)encoded_va); + *offset = page_frag_cache_page_size(encoded_va) - + nc->remaining; + } + + nc->remaining -= fragsz; + nc->pagecnt_bias--; + + return page; +} +EXPORT_SYMBOL(page_frag_alloc_pg); + void page_frag_cache_drain(struct page_frag_cache *nc) { if (!nc->encoded_va)