From patchwork Wed May 24 15:33:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3780FC77B73 for ; Wed, 24 May 2023 15:33:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA17D90000E; Wed, 24 May 2023 11:33:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C51C6900007; Wed, 24 May 2023 11:33:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B194690000E; Wed, 24 May 2023 11:33:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A327C900007 for ; Wed, 24 May 2023 11:33:24 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5C3D6160A5C for ; Wed, 24 May 2023 15:33:24 +0000 (UTC) X-FDA: 80825542728.09.E72E14A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 3A0DE1C0023 for ; Wed, 24 May 2023 15:33:21 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jA8m9Dmk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VHLRJgykEA+vivKsr5PIe4LNQi6791a3nIaeRR1gBwE=; b=1/gBuvvxHVPXGC0x9eErs750BJYecfI7TkGPirfnUwk6Vh7/EsEkdQHbNdSqXGZjuHuYV7 mIJ7Ah8IaEjYJ4ZGllB/F5sBNMgDhtI8pavfYUOuQG0FZcWuz07yWZhzJNabbTBtQWLcCZ Rx3Mu5k4Fug86Ky/+Qau1eOFpGqMBEc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jA8m9Dmk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942402; a=rsa-sha256; cv=none; b=YVjG+7AdQYCU+r1TAgymddWWW0OWZ3yYkeVySm5z1Ztsi4Rceer/2/rbr7STMteFN3Y8f+ 0RkNXkgvQIWjvA9kG0+y99mBE3qk9KZLyRNqoNqPqOacdFXAHVuoNWOEA3Wt7ZVLcptoNL B1RNn8arJMy1sVpJCYkuICF+nbRWgeQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VHLRJgykEA+vivKsr5PIe4LNQi6791a3nIaeRR1gBwE=; b=jA8m9DmkiVhTGPcHbHkjoqi6MSuirpsxs4HopqNAgTutRQIFsqpYUbEah44wBXfdGjl/Wb ECBvopsYKuyIejix2E9AlvsDVxFUUEEzPiMN9Z25PikTLHJZzWgg7rOpNfG0adijINBqLV m8KBvrZikU84Ga6ncw1UaQvaSxzVvl0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-283-sw8TgtPTPZKZI2FxAmu1nw-1; Wed, 24 May 2023 11:33:18 -0400 X-MC-Unique: sw8TgtPTPZKZI2FxAmu1nw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 317F8185A793; Wed, 24 May 2023 15:33:17 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 60D09492B00; Wed, 24 May 2023 15:33:15 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Subject: [PATCH net-next 01/12] mm: Move the page fragment allocator from page_alloc.c into its own file Date: Wed, 24 May 2023 16:33:00 +0100 Message-Id: <20230524153311.3625329-2-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 3A0DE1C0023 X-Stat-Signature: rgcyqhb3o1ib9x68687m9ix4988e6i9a X-HE-Tag: 1684942401-997522 X-HE-Meta: U2FsdGVkX1+yNgs/1ZAWsAorSy8VN0NT1JI3ad1+68b3Edok72PDMI13g0hAyJDBIoWaJ3q6vnjUCKXjiGTfNEgdIyqm+NdyJ4rrozjQuFh4N12tD1HUkcC33SiRDexTy4ugZAspmh9UsoJjvGBoOcmCZwtCKdD3D1adx5wgOZ8bZn0qLivZguAirCxz2P9JexLXTSdRyF53KgU9NYwsYIfO05awPfppqbT4JhPx34j6mm3MxzYlaLepQGe1gjaqI2sUlA99TPAoch/rKlfEOBXbNPjsNr871xofrKvJzTuzMDSegzfZ7wTGJuP45CU5C4CUZpWkbr/y6a6PfDkM+UuTvhck0PGmFdb7JphvL49XBgS2HdooeNa1PKEni5yk2aonAap4SVAGdGtbsSDSLQaoqy7DduJrSVQhYn+FO/NFW8F4/oyrhlmel9PqoxDY0+4c3e47+tGvuI6wmWY7tuyedRtanAoJciaGLVFI51AjnDR8Kc+iH7iQa1GK6FtOdKJkhNfRhQu1MTSX1sE3iYePvlwE78F57XyOyEpYVrOhJa7KUccB0P+ysEnhVMjJVeUj0Ex1sKzcCmeRdaNEhTnHmPDzP4u/6gsHj8M64B4s5AvysRlPFGMif7JQXNtCSuydzNqJZfcbOYGu7KK8qYfzPwukVNRMH5Lid2NC59gGjHWJXxw8uI7oAdS5t5iBqzQXNbjXe5uHdC7K/vftPpnAN3KXcROse1/fJrHqbGZss/gU+eDw/pj2jqbwU89f6h7FQFmkBZHzzaK8GgZpngN9b3KvhiW5sJo+RD/OPpLAiGokRKb2MFT8eTrtZdGvHX7pMjgMqlBApqde9HntsKRaD/Oaqtt1jaaEV0i9f5mHW2gX7xW1foao8RIMkfSIG8M0epSZnQthKqnIXjqW9wDQAD3olH+Dw6cy45KwNUPBda1ybevmp429NMCONMeKRt3rvWD4L2AEEMLd+U6 vi5eihBl AHPMHpVOYARkQ6lbk4NmXJ91w7v3ZAwKS7Y6xxKO5NWHTPnIa1BKlvA6nXfF8rZCTbG1UN9RwapBkO2+GcD1Hfkf55n0/no2v6nErpYEBevUCPgD39lSQQLxYr0S4btxXrCZd1eaGYB93lNUw0u3opS9SntzG9tYlAieszZWnYsv548UlUxRSqd543zxGqS+4Ig2mQQY3MbNIJGI1HfD+fVGOQwfTxE16x4SSvyn5YV05CxXNBZ/YkgtwEacwhfXECbbYpvAZbXDWcwnfsxuc8cW1zGdV6g6GZCzJnj1x9/VK6s/Nq6wQawX5z6eMsG0SGZeGFIiQ2VO/799n/7bSqbaTxt2XSh+CEdpReh7gNs4r2Z3V+KwYsa68CGveILbbH7VpeD65OlurBa5/z1l7ZLgQMEdhvEsayuN3SAaRVTKR3aMo8LxWgSbxpLVi0FApEK76Fi67SQeO3I8gj9E60i5ILNQjb4WfQ5vSzIUBlKfHk0Lj29DqEHv8T2Sob49CEBljqyVEsIImtBgcgDLGtlt+LCZ7Bs2b+ytApZPyRioGUI+Ijx3hWuWJEojJF9cD0o/mxhHaYR+oSf0ZLPEVPbWkr6reGLeHztmWSnxqr0y3Kks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the page fragment allocator from page_alloc.c into its own file preparatory to changing it. Signed-off-by: David Howells cc: Andrew Morton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- mm/Makefile | 2 +- mm/page_alloc.c | 126 ----------------------------------------- mm/page_frag_alloc.c | 131 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 127 deletions(-) create mode 100644 mm/page_frag_alloc.c diff --git a/mm/Makefile b/mm/Makefile index e29afc890cde..0daa4c6f4552 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -51,7 +51,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o page_frag_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 47421bedc12b..29dc79dbeb22 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4871,132 +4871,6 @@ void free_pages(unsigned long addr, unsigned int order) EXPORT_SYMBOL(free_pages); -/* - * Page Fragment: - * An arbitrary-length arbitrary-offset area of memory which resides - * within a 0 or higher order page. Multiple fragments within that page - * are individually refcounted, in the page's reference counter. - * - * The page_frag functions below provide a simple allocation framework for - * page fragments. This is used by the network stack and network device - * drivers to provide a backing region of memory for use as either an - * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. - */ -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) -{ - struct page *page = NULL; - gfp_t gfp = gfp_mask; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; - - return page; -} - -void __page_frag_cache_drain(struct page *page, unsigned int count) -{ - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(__page_frag_cache_drain); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) -{ - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; - - if (unlikely(!nc->va)) { -refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) - return NULL; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; - } - - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (!page_ref_sub_and_test(page, nc->pagecnt_bias)) - goto refill; - - if (unlikely(nc->pfmemalloc)) { - free_the_page(page, compound_order(page)); - goto refill; - } - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); - - /* reset page count bias and offset to start of new frag */ - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - return NULL; - } - } - - nc->pagecnt_bias--; - offset &= align_mask; - nc->offset = offset; - - return nc->va + offset; -} -EXPORT_SYMBOL(page_frag_alloc_align); - -/* - * Frees a page fragment allocated out of either a compound or order 0 page. - */ -void page_frag_free(void *addr) -{ - struct page *page = virt_to_head_page(addr); - - if (unlikely(put_page_testzero(page))) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(page_frag_free); - static void *make_alloc_exact(unsigned long addr, unsigned int order, size_t size) { diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c new file mode 100644 index 000000000000..bee95824ef8f --- /dev/null +++ b/mm/page_frag_alloc.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Page fragment allocator + * + * Page Fragment: + * An arbitrary-length arbitrary-offset area of memory which resides within a + * 0 or higher order page. Multiple fragments within that page are + * individually refcounted, in the page's reference counter. + * + * The page_frag functions provide a simple allocation framework for page + * fragments. This is used by the network stack and network device drivers to + * provide a backing region of memory for use as either an sk_buff->head, or to + * be used in the "frags" portion of skb_shared_info. + */ + +#include +#include +#include + +static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) +{ + struct page *page = NULL; + gfp_t gfp = gfp_mask; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; +#endif + if (unlikely(!page)) + page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + + nc->va = page ? page_address(page) : NULL; + + return page; +} + +void __page_frag_cache_drain(struct page *page, unsigned int count) +{ + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, count - 1)) + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(__page_frag_cache_drain); + +void *page_frag_alloc_align(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align_mask) +{ + unsigned int size = PAGE_SIZE; + struct page *page; + int offset; + + if (unlikely(!nc->va)) { +refill: + page = __page_frag_cache_refill(nc, gfp_mask); + if (!page) + return NULL; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* Even if we own the page, we do not use atomic_set(). + * This would break get_page_unless_zero() users. + */ + page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + + /* reset page count bias and offset to start of new frag */ + nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + nc->offset = size; + } + + offset = nc->offset - fragsz; + if (unlikely(offset < 0)) { + page = virt_to_page(nc->va); + + if (page_ref_count(page) != nc->pagecnt_bias) + goto refill; + if (unlikely(nc->pfmemalloc)) { + page_ref_sub(page, nc->pagecnt_bias - 1); + __free_pages(page, compound_order(page)); + goto refill; + } + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* OK, page count is 0, we can safely set it */ + set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = size - fragsz; + if (unlikely(offset < 0)) { + /* + * The caller is trying to allocate a fragment + * with fragsz > PAGE_SIZE but the cache isn't big + * enough to satisfy the request, this may + * happen in low memory conditions. + * We don't release the cache page because + * it could make memory pressure worse + * so we simply return NULL here. + */ + return NULL; + } + } + + nc->pagecnt_bias--; + offset &= align_mask; + nc->offset = offset; + + return nc->va + offset; +} +EXPORT_SYMBOL(page_frag_alloc_align); + +/* + * Frees a page fragment allocated out of either a compound or order 0 page. + */ +void page_frag_free(void *addr) +{ + struct page *page = virt_to_head_page(addr); + + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(page_frag_free); From patchwork Wed May 24 15:33:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 254C2C77B7A for ; Wed, 24 May 2023 15:33:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA53D90000F; Wed, 24 May 2023 11:33:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A55BF900007; Wed, 24 May 2023 11:33:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CF6390000F; Wed, 24 May 2023 11:33:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7D9C5900007 for ; Wed, 24 May 2023 11:33:30 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5EF0C1A0236 for ; Wed, 24 May 2023 15:33:30 +0000 (UTC) X-FDA: 80825542980.28.FAD2E83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 8ECD4180015 for ; Wed, 24 May 2023 15:33:27 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ipcX4hbU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942407; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5Xt/kb1I9tBW/zHVjBQrREyGjijOb+tpZUMT6BFY+/c=; b=Tt3Sa6W2UMPdwXQ+5RDRGYbtIp24oaD7WS9xZpiwKaoJfW0Pit2571mbCNfPxbXIFj3TSK xuWS1rvTUdeVCrr3km5oCknjBbrGG0UxiYAU1NsQ+Zmu2xzcoH2DTA26nmVtvnZOHINTQM +H2mX+dpPkvpK2WOLfT5bR2+D1+0n8g= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ipcX4hbU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942407; a=rsa-sha256; cv=none; b=TvtEc/7HJvS/RriTklYPS0cDIit5K05O25Rv5voKHCbMBYkg4+LVTVwdOOb6r3wDnZLhNh lbwAxu9YjDI7hVOP0sHGpiC+FMOPt/r4srCaPED93yuxlaPlon2UP6wyxUupGK+l7JwGSv QPK4GwDrpTpTP1+UOhrLNFCbLkujM3g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5Xt/kb1I9tBW/zHVjBQrREyGjijOb+tpZUMT6BFY+/c=; b=ipcX4hbUtTGcvOpbBoKovNUg+AyRGe5NZ78quQ8zxsJNhOWBbWAy4Q0970Qckc6mMQ0v4f 8tCM/3+Tk7+GmnHVVGibHplldE14h5Dqi7T10mriarBOOxgSpVjrg8EC2XAYZ5WQ8TDM40 YcZwfzNpwVgzKTy9mS7BiJ6cUgTnWoA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-147-h5Ilq50YMBixAqm1S8Ipag-1; Wed, 24 May 2023 11:33:24 -0400 X-MC-Unique: h5Ilq50YMBixAqm1S8Ipag-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 52D53280BC8A; Wed, 24 May 2023 15:33:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24FB920296C8; Wed, 24 May 2023 15:33:18 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next 02/12] mm: Provide a page_frag_cache allocator cleanup function Date: Wed, 24 May 2023 16:33:01 +0100 Message-Id: <20230524153311.3625329-3-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8ECD4180015 X-Stat-Signature: pxwfwpukrp3dyynespbrx3xq6b4qr49e X-Rspam-User: X-HE-Tag: 1684942407-477521 X-HE-Meta: U2FsdGVkX19TCFYraRZtzxGxPQ6XJf07DkvcC8HVYOQq4LWN4Rh0LPfbGhM6E0Xr6IaXotWfu4y9MnrMOe9VgduexwaRzcNSYjx/+GWy5z9a8pwEhZvmYccJxTbJkVcbhOjJ4Wpetz0jCc3/cRgbPuudR6VuvVF+j0FkaIdQIp6a71Q82V5kfBeTJtuLDsfJEJcSOaC9Bc7cyjgVl4W68VGm1aLRSeNjvi11oUHJDd9CsSvD+DD3EFcZyhLwDjWKQyysZCStbO5NWPmXJcapafCA+rCowE629//iGiZEqCrG1Wc6OxjHSTJY/9vHOl0ljKidZJYCPS9dCFLkAKRWscocOryA5G6BXYJH7trr9BNZuC8tObDnnvpwVl82yMa+FrXLZEzmVz/m2B3ikl0EVJi09dcGZ6tPX9iCfRKSNbadW8rkbjYtZ/BF6UayGBT/6Am6se8yg6nglcIf6UF4g+hXvOqo/jtjljmArhT0X68pjJXuOeF7qlGxLUgbudgTtz74uV+svuLnLfE4uavhjwIbjircKjxbLZqDFPGv2wKAAKb/HT8dVhGR4Bv6BdAiA0uCqwNRNcMRqDcKWfsJsRKoGh4xZGCHr6cdY8eNcyUJTBOvGCzQ0ZQ15kMi3lCznzuilC8G5SjYM3ZdD44xJuiGvAHYCUrcDGe6nQDW2RTc2Rak2U44TUvCyBXbkmjzaXEaCauW80Nt0+eoZBMl/XazuPGiCo+58xJQCT1LWxLYtJnyj6qOSodQAfhSoV85kB9aE4xXXzZrHzaJi/5M2cNnQ+0S+37M0fS+o21Ci5+LK+40MFMephV87MM9R8oa+EuARZra8aEDlQJGpZPE0pkW1acWJxFVybLmDH1toRiNT5aEOWdCthXKtLWv1bSIfd6ZuAQ/yiNngsUHAr62iETky4IjdsISwXuCB865aZwqTb3MtdbYm3/pLavLrgAlMQFiI4QDqEhpdyv8SIv oZfAHTZt fBN1qlYNgkrXxub1pvcfXYgkfL7WhyMn/8a35jKpmkXkvt4Oi6/7y2idSbO+137CUfz9/8Cu0LkQjhMfV/SddbplnfqcV1DXshFCvSFs2UjFJPgpY2Yc+EmbiRWdRQTxSyksOWpdUU0N5Fz9yuCGkvxaIqXH5+55az7DeSLSzq0ezSdrQ0zmJDHnzkIsc0mJovYwO/AdqbX629oZQx+VFNpB6VW80mktZQzos06OC1aet6sWt+Tda3ri0JER1hbodAFOmsgsue94BRQWJXkG/DoZLxm23xy6GwzZ2ng04bRct+n8qqKqHwGw037DVO4nEHKDyzjAjR4gdFEdyyLocodNna7M1PwgtQ43aci2ZFLD2xGwN7aJPQ4MtFLe9TnMJ+EFutCFL3pv9T/ztaShFcHu0Bn2DpXrQy4wIhk5rckdMm4dG9e4cCFjFoIsou8bAqkYNHLWBEbDok8M7jrxU2DzAsd79XxpIKXOAN8saOV0dfzOP9JAQ6oY6eVv256lKcCDVDFB6LuLEu7pQXMhgwFusfIDvA0onJH6+sfhczecYLOrIQuuYOa62N8Y/y+fIGZUmqmi8fMGW8bYhcrUihQ6GzZL7KEylnRQnRbrovhonbtPliVu2G8OThMjQIDQ1YBBV1Eo9qbxYrNIfoxuItd4yJgyMXgVBVL6M88J015ZhQdrdsLdWXV4o1DdoUJCIk5o9kBZF3sTFswlwuo7cbGooR5W7DyhBzxjGuJ0tfLyGhQos/P+rPgLYbNpX7wcTNrC/F8XGwIFIRYWUGM9KVTRsDRIGvbFZ7Yd2jKD8Jrf10QH+E3VoCHoSSK49X9kG96iCtqQA1qTy3jmaXWGI7DMmjWgh3mOmwWBcvC7zpDLmNDumT5lpimGOMhk/NqCSoelX0qjc+FMMLCk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide a function to clean up a page_frag_cache allocator rather than doing it manually each time. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- drivers/net/ethernet/google/gve/gve_main.c | 11 ++--------- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 17 ++--------------- drivers/nvme/host/tcp.c | 8 +------- drivers/nvme/target/tcp.c | 5 +---- include/linux/gfp.h | 2 ++ mm/page_frag_alloc.c | 17 +++++++++++++++++ 6 files changed, 25 insertions(+), 35 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 8fb70db63b8b..55feab29bed9 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1251,17 +1251,10 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) static void gve_drain_page_cache(struct gve_priv *priv) { - struct page_frag_cache *nc; int i; - for (i = 0; i < priv->rx_cfg.num_queues; i++) { - nc = &priv->rx[i].page_cache; - if (nc->va) { - __page_frag_cache_drain(virt_to_page(nc->va), - nc->pagecnt_bias); - nc->va = NULL; - } - } + for (i = 0; i < priv->rx_cfg.num_queues; i++) + page_frag_cache_clear(&priv->rx[i].page_cache); } static int gve_open(struct net_device *dev) diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 69fba29055e9..d90fea2c7d04 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -286,7 +286,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -298,19 +297,12 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) entry->buf = NULL; } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -320,12 +312,7 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..dcc35f6bff8c 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1315,7 +1315,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1326,12 +1325,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - + page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index ed98df72c76b..984e6ce85dcd 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -1464,7 +1464,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1486,9 +1485,7 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); + page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index ed8cb537c6a7..03504beb51e4 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -314,6 +314,8 @@ static inline void *page_frag_alloc(struct page_frag_cache *nc, return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); } +void page_frag_cache_clear(struct page_frag_cache *nc); + extern void page_frag_free(void *addr); #define __free_page(page) __free_pages((page), 0) diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index bee95824ef8f..e02b81d68dc4 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -46,6 +46,23 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); +/** + * page_frag_cache_clear - Clear out a page fragment cache + * @nc: The cache to clear + * + * Discard any pages still cached in a page fragment cache. + */ +void page_frag_cache_clear(struct page_frag_cache *nc) +{ + if (nc->va) { + struct page *page = virt_to_head_page(nc->va); + + __page_frag_cache_drain(page, nc->pagecnt_bias); + nc->va = NULL; + } +} +EXPORT_SYMBOL(page_frag_cache_clear); + void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask) From patchwork Wed May 24 15:33:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCEADC77B7C for ; Wed, 24 May 2023 15:33:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 550B5900010; Wed, 24 May 2023 11:33:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B1D9900007; Wed, 24 May 2023 11:33:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3035D900010; Wed, 24 May 2023 11:33:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 20B98900007 for ; Wed, 24 May 2023 11:33:33 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CD49E40A34 for ; Wed, 24 May 2023 15:33:32 +0000 (UTC) X-FDA: 80825543064.30.F09BB5A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 0E5D020002 for ; Wed, 24 May 2023 15:33:30 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gFFk6/oh"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uN3mSK+PjIP6JAbZhfDAmtnx0lnfpAA82MAUedDHiDA=; b=xuRY+qKt3hnXQLl/wR42Sw+ZUGjt+CIQXsiC2Q7rlrH6h1IuQDBjJvHzD5edBwXyiXek47 YIUWlA6eNW8K/05Q724ViQPvwVWt9HT/W7JnETmViIqvY7mnMRmYaFQJ0Yas/xcq7mGMdh 8SVlQhVOBSfG3DSrtlVrRLDpWUaYvM4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gFFk6/oh"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942411; a=rsa-sha256; cv=none; b=tGCdGuAwbcTJyoYm1i6qpNZZsM7ryg6rkzWjSQLjieuUMbN/4nBzfmHEgF5HQa6gdLza5A qos52DKxD/VI4dZDG7lNC9+7mZ6d6dSkcjSYdCOd9xE8wk95gJcT2/RrTgRBZ0512MUtZ+ HRtbjy6pfJd/v+zq/XXmHjaW+tCUR/g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942410; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uN3mSK+PjIP6JAbZhfDAmtnx0lnfpAA82MAUedDHiDA=; b=gFFk6/ohsacLgGotYbnQBdO6LzZoL1AKEVRC0XAmzAU1oNrU4WFmeaVsDaFkoKjgT+6cfU H9AgQrIDR7dBfQrGhkWt3HCsXPN/GQnbZ6P4bmcju3B6Xd2I0xz39aj/pmcyOfzjbdah3J QcAg6moXIA/W+o1pDbClFzAP6sF7p/4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-Arb27wHhPVquoWD2zvAqmw-1; Wed, 24 May 2023 11:33:28 -0400 X-MC-Unique: Arb27wHhPVquoWD2zvAqmw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EB53C3C0D185; Wed, 24 May 2023 15:33:26 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 12905492B0B; Wed, 24 May 2023 15:33:22 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next 03/12] mm: Make the page_frag_cache allocator alignment param a pow-of-2 Date: Wed, 24 May 2023 16:33:02 +0100 Message-Id: <20230524153311.3625329-4-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0E5D020002 X-Stat-Signature: eyt8ua1fa54atw9cq7pcjbqc4tayi9wt X-Rspam-User: X-HE-Tag: 1684942410-509721 X-HE-Meta: U2FsdGVkX18Ad7xOCzsu7M+xLBi9kW8Q+OuDztQH8YYA7A/wKy0+Y5/2qJCsA2xm3BpzkDWwDODTYA4xEkGFqr3A7Y7PL/VUlXLMjYoG0U/kFH5nnS2x8JxETP3yE2dGzPSd4fZfkgsioXyJVFZPV4LQSRLX2I0ndocwXiyqM8OiLWO/pb0clV5JccBLFX0c/wCzsXZuRuk6Mkje0Nx8zA4Mhrw3cnLtgQkXE72rTblL8LYwjCEWpflNTyRmUDQ7cSf/uU0y61h7ripMwkfUu9vfBcEQ4eE9n+F0mGm9ada7hqOwU1ygh1VZIe4mpNIUROJ1y9iJFGQhNTnvqDWwoQHMN39yZa+aIYLK9jekUeyMIK4AzeIl4eB7dsUK7JeAesEjeswtvEsrE5ZDw3ZDbNk+NU1mbiHN2jVPNTWx6XuPZWeoOeCtOLkKZ5uAhY1icjkEVqrvbQREt44D/nMSNXGaXeP5KXg/+6Y63O0DD/ZoU5QOQ6/QTA6n9R8KvHalq+PkELxKQAmpv4QPqt4XiOIV9nhPSGbjCWPWAfLPVsg1Nist8atENpbmOUnnDElKKVbwQHHzZA9uJac7ko8zC/6w04+TN9OpWLAzUq8iqY0lFyIpeed8BFDbBHeyIrYIB1MJxQ6ROg7GdbsC97GbHhfX+f42vIFqwIb5ePQ8V5aIQT51Sa6pwA5+f7d/KR2p069hjSn17omcKkzgd4Ifyu93zM34sOTTE9Wl/jPVde9wucvKqjAvZLqsA5u4G3R+z8i2ytFd9IQiXjLTTHieMPWtJB2JfNp4SDPKhqzpKj4bq3sHtTlVL8vQO97jqEzQ6P/KNb+ZJYQ12m4vyAkWnnQIw2TARkRwMq68V9ChdpwpAVQI0jPQxB1bEECr81KpILbEWG3CIWxWgGh6nCcuxfy6ukI4avoNKwykfdVs3Wfxg3s2ae8sWiDqAtuOrO3ztyJIlmqEiHXVVGr3GSd cOVjLa6m DVuSzpm/Fp/b7IPOJU6A2RIcsvSdmw2QbK/0pyMB4mTG1aRNqEgee4/OeliMMO6TOcqLbouHuXlAsKJvPpz4meGMO7P8L/yuvSkCwgC0XBn86QF+kMEIYJkas+AIqxC26oCy4f6GsBREm+bWJ0qO1F/pd8spdqcJgoK9XtrhwWVfgfrwHt0fx2L+lGo8qvPTfMpUsUpATNmW4RKVhXACE1EEcWrr+px4y4KUx2T0MSOMI7ovTA06P8v2HuAa/YaNkp6jyuMYYXtZGJ2qbSyCWNp57l5GxIUe5radm5yt5BWqvlCfo9Gmbf27GzX38aicj2J97MMO3xM/L7MjauOzFLETTNiI2mLH33KKwykPYIqnwHMSPO1pjOeq5kjL2jFb1mAplsXqzs86T9JBUuzllWN4E3tofkHcN85WqtH84CspNE/WljMlBeKl8vvwpk3GcUF/o15n6cdeIaNhsSp8sLLqinZdwJpGHRsertRWGjUQgnwTovUByWzLz8lgD18ScVvdnZZLx0q/+anIaRZCmhfOoMhkf1YgeXtHibSZXAdcWbr0b7JgMJI1xZX+hV5SoBKO5RB4OmrlrlNcpcWbFbVOSxHVPyKxGHzS/zTwccH4flRKzMsS9KLVKruFx0vt9f6OaQHxQsVL/UfETrrWeICGQSLJqF5q9a76oY+nzm3ty/Noaq88HnEwtBQec37bLNSSonPqzNroXsefR4i553aqBVNssi5nVEzlNSl8stndkNv+QXgNkvQssHdmMmCxLtIw0+X9lasY5slgING8HHcraAtY33yau4uMQOR2aV0s2Y41WUR7MnZyFJ3w0Vg0T9uY12CbqzolG5FCIjAoIRMjzucBknXpX/U8Xnv2PnxIwjg2151zg7FyPhSbRvo21nKwb0NZ5Xbx9T5w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator's alignment parameter a power of 2 rather than a mask and give a warning if it isn't. This means that it's consistent with {napi,netdec}_alloc_frag_align() and allows __{napi,netdev}_alloc_frag_align() to be removed. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- include/linux/gfp.h | 4 ++-- include/linux/skbuff.h | 22 ++++------------------ mm/page_frag_alloc.c | 8 +++++--- net/core/skbuff.c | 14 +++++++------- 4 files changed, 18 insertions(+), 30 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 03504beb51e4..fa30100f46ad 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -306,12 +306,12 @@ struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); extern void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask); + unsigned int align); static inline void *page_frag_alloc(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); + return page_frag_alloc_align(nc, fragsz, gfp_mask, 1); } void page_frag_cache_clear(struct page_frag_cache *nc); diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 1b2ebf6113e0..41b63e72c6c3 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3158,7 +3158,7 @@ void skb_queue_purge(struct sk_buff_head *list); unsigned int skb_rbtree_purge(struct rb_root *root); -void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask); +void *netdev_alloc_frag_align(unsigned int fragsz, unsigned int align); /** * netdev_alloc_frag - allocate a page fragment @@ -3169,14 +3169,7 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask); */ static inline void *netdev_alloc_frag(unsigned int fragsz) { - return __netdev_alloc_frag_align(fragsz, ~0u); -} - -static inline void *netdev_alloc_frag_align(unsigned int fragsz, - unsigned int align) -{ - WARN_ON_ONCE(!is_power_of_2(align)); - return __netdev_alloc_frag_align(fragsz, -align); + return netdev_alloc_frag_align(fragsz, 1); } struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int length, @@ -3236,18 +3229,11 @@ static inline void skb_free_frag(void *addr) page_frag_free(addr); } -void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask); +void *napi_alloc_frag_align(unsigned int fragsz, unsigned int align); static inline void *napi_alloc_frag(unsigned int fragsz) { - return __napi_alloc_frag_align(fragsz, ~0u); -} - -static inline void *napi_alloc_frag_align(unsigned int fragsz, - unsigned int align) -{ - WARN_ON_ONCE(!is_power_of_2(align)); - return __napi_alloc_frag_align(fragsz, -align); + return napi_alloc_frag_align(fragsz, 1); } struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index e02b81d68dc4..9d3f6fbd9a07 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -64,13 +64,15 @@ void page_frag_cache_clear(struct page_frag_cache *nc) EXPORT_SYMBOL(page_frag_cache_clear); void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align) { unsigned int size = PAGE_SIZE; struct page *page; int offset; + WARN_ON_ONCE(!is_power_of_2(align)); + if (unlikely(!nc->va)) { refill: page = __page_frag_cache_refill(nc, gfp_mask); @@ -129,7 +131,7 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, } nc->pagecnt_bias--; - offset &= align_mask; + offset &= ~(align - 1); nc->offset = offset; return nc->va + offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index f4a5b51aed22..cc507433b357 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -289,17 +289,17 @@ void napi_get_frags_check(struct napi_struct *napi) local_bh_enable(); } -void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) +void *napi_alloc_frag_align(unsigned int fragsz, unsigned int align) { struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align); } -EXPORT_SYMBOL(__napi_alloc_frag_align); +EXPORT_SYMBOL(napi_alloc_frag_align); -void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) +void *netdev_alloc_frag_align(unsigned int fragsz, unsigned int align) { void *data; @@ -307,18 +307,18 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) if (in_hardirq() || irqs_disabled()) { struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align); } else { struct napi_alloc_cache *nc; local_bh_disable(); nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align); local_bh_enable(); } return data; } -EXPORT_SYMBOL(__netdev_alloc_frag_align); +EXPORT_SYMBOL(netdev_alloc_frag_align); static struct sk_buff *napi_skb_cache_get(void) { From patchwork Wed May 24 15:33:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BC48C77B73 for ; Wed, 24 May 2023 15:33:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDEC3900011; Wed, 24 May 2023 11:33:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8EC8900007; Wed, 24 May 2023 11:33:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5605900011; Wed, 24 May 2023 11:33:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 956D1900007 for ; Wed, 24 May 2023 11:33:39 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3C23F1409D6 for ; Wed, 24 May 2023 15:33:39 +0000 (UTC) X-FDA: 80825543358.26.B33C39E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 5ED8B12001A for ; Wed, 24 May 2023 15:33:37 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ad4CnB4d; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942417; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E/mkzJ+c6qr7yPOsPF9NXbENxWjJthPNWT3mBurv3yc=; b=wynLumbF3WxZmmQdE8Efbkue9KfQV2NydSl4Yw06h5UbVxXS5AF2EZAIMr8iLJUe4NQ0h7 /KEYcFKl4r5udhu8J7LzPDq7Qe9kaGPh1errAGOM+PqKQsUrye6IdUz8jqMJ9cL2EErcMI WY/bwFAXVjVD4Y/s6cqi/t5nr0P11xU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942417; a=rsa-sha256; cv=none; b=eVCHWfCz5wSc6BL7Wciza5SIPR3aETpkbOyCOQ/MMcuN9yyS20W2foE1DfSrW4PFK8cJF1 mk5lK+lIRfF7rmxfLw3X0fXCYfDXwkUgrBhoAiZm0BjMqSBoYrcoNBVOCyTrzk2EdNqq2r Osi8YuL0256T2mJANDfQT43vDPKS4CY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ad4CnB4d; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E/mkzJ+c6qr7yPOsPF9NXbENxWjJthPNWT3mBurv3yc=; b=ad4CnB4dMF57kwsg/xU7UcpF5Jr3YIO6MFWbAJFIHsMsLrY+dvXUEJu9368HSfKt7l63e/ mYZddq5p0LpSTsxBtsjCIkKSol+gXKTW+JwUMw6hcBhhGSlJXsHKMUMOv82RK5qm/u4zq2 cr5W1Q8QkTKnUe79DBM6Cnua5gKrLgg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-ozOdLk_aN02TMJY6p7EiVg-1; Wed, 24 May 2023 11:33:33 -0400 X-MC-Unique: ozOdLk_aN02TMJY6p7EiVg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B040C280BC80; Wed, 24 May 2023 15:33:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id C783840CFD45; Wed, 24 May 2023 15:33:27 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next 04/12] mm: Make the page_frag_cache allocator use multipage folios Date: Wed, 24 May 2023 16:33:03 +0100 Message-Id: <20230524153311.3625329-5-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 5ED8B12001A X-Rspam-User: X-Stat-Signature: tnuy1p3t4sitrbf7ryd9pt866co3wzjp X-Rspamd-Server: rspam03 X-HE-Tag: 1684942417-219341 X-HE-Meta: U2FsdGVkX194JJpiWutrO+A0Qye/TEQAJjiOWdx2OeCfUDkGyf5EdWuLsPOh6izPw+/ur1dTOLtbZAzIIEwEXj6zZy33n/l1LvwLum5KMW5YGu5usHVxYVuJBZqN45+yO1zZZZyzmp8n2zPtTd7bjG7zOegS8lMlhPvywF4aMbXNSlnyPn/o37bDIldqqMk2slNTsjMnbFuGBb1W3LmOQGa0PgqpTafUtP+Q3CXZw47Yi/ZqQuMKcLLTcQ6HdJSsFHpXRP92nQKW1+8bO5GMZGNfmSqkE4LeAg51QUfa4V5cxORaZg3NxD6wvlSJUt6MKkuz7Vo/KiIo2WCxgGXl30gWBB5lGxru51hMUNB79vQiWUKXtPP1wG5LFhlHO+8yacwX83gleQlZnD2quX7uovm/lOcsWvnc2orFKT9UUyfXU6jJr9AW8GDvAKfcrRdgma7osTG83JP+JUweOPuRb4JBfNLrJJfNiHQIWGwU42pj25mcsi3OoJ/FzCGbRe+WshPOYcrBGRq9mrDxrMVkDH0+S53KKXh+LRNYGPsO3QKHaFJaGTvZrt/uuIOaGcS+RzmKU+5Axja7ZCnTHvoHB5H8y3pMkSYbtYImutB7tXqMvExDy9RwVCDrrGJSqjR+zX2uaB4rajBO8nFfv8nLMdUetQ8DVh6fMkTugVCPfvyNy/PUQ6JMF/AFdIP/2dc+Tl6b8P2ZnbrmFxbP5uCIapNl5NAkE9Mn0pO9AFPKQgKF0M7Sz3hQxbtNY7Jffewoyg8Rz1wjOePxbGfJJa07dbiU4cEKMk1GfG6XJ+IseWo8DVTBpTij3xedm4eXLnIX33A+JQPDZNdo21PMXY7jiJBjY1+3q5p9L1ZBVC9MgE5ATpQ8sorxb2gJMTOl9JcnFPFEOygu79SnmEMcCJc8S6fHBjU4PgFIIwlnddLkXB1X5lzJv6F6Izf6DkCJMrHzHw8OH76IW+zREQcbcNx 4Wkv+uLc IOat/PdZQIEEJVdFcuD3xy38xi4cURyZMapzEsKLUaAIXYfYUJcm2PqDCtKuK1KnmmacOCUEZDu0qjr6gzU9FUCWMAMf8g4l8GRYH8dhZHWTJT+OmxcV9UD3R+W5gWetpoc1YHCK4LNfIFScV0ce26KuTuubfd3c/r1mcxdnRoX6BQYNuP9Vud/tiPIuwIOnggxoOgWIASx1UPP+D6iXMMrUQgAXhJEdqiYwWJifmixnsHcXrs8LqswVVUmkiLXbWDMC1OngfHrI4Tnp2emfeI6Je8V6Iwbm/wUKNKpiNhUqNwNiL3z3khjtYlMVLvHMQWZ6ugOZChou0/olhFkFaOLlQkDxKmZngpsEc37R60oshAmNfXI/e807ckEHsoupvpOX/mm1txqCM+S2Hb3KeCjwL1ppMpUG2I/0HzeVztr2ajvbZKsZvphW/P+FAKODdQORZB4eqTqXVr92r80D3COipwHjj7AZa2GDy4E5Jp18R3b6hmhEJTy/NbsgCkqV2j1HwEUN+RtqwOoN9lN7pUn1Z0ACaBbcScfjOQwSg6ogECOZTOxG/12PTDvXQ3xetd70iZss16v4wrmgA2lUXTPxfaz3gvFZZY2txC1MeuEkkUrgFMRapa5vXZMoBOasz7PtyhnZWpl+GvflOYCbwowJDBb+JCFMh7kcn110B56ZLAtz79jjORbdb1c3BfWKdi2BXtMovE35994nP77B11o+AjURg8Nbdbsdw7yIG46exL8G2Y2pGlRUZGSqEIR7P+hNYHC0R2z24ge9v527o/iLNJ0CsUm8IL6bSF/GtSoKUHr12BcOhfQleMHslTdaOdN+rY4CrFlartVKAh/UNMXIXzDkOK3TDZ33N9BcA8lELGLYowBkEv154C0TRGkYwBQ+r0dsV7gkCd7k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- include/linux/mm_types.h | 13 ++---- mm/page_frag_alloc.c | 99 +++++++++++++++++++--------------------- 2 files changed, 52 insertions(+), 60 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..d7c52a5979cc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index 9d3f6fbd9a07..ffd68bfb677d 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,33 +16,34 @@ #include #include -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - nc->va = page ? page_address(page) : NULL; - - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); + + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -54,11 +55,12 @@ EXPORT_SYMBOL(__page_frag_cache_drain); */ void page_frag_cache_clear(struct page_frag_cache *nc) { - if (nc->va) { - struct page *page = virt_to_head_page(nc->va); + struct folio *folio = nc->folio; - __page_frag_cache_drain(page, nc->pagecnt_bias); - nc->va = NULL; + if (folio) { + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + folio_put_refs(folio, nc->pagecnt_bias); + nc->folio = NULL; } } EXPORT_SYMBOL(page_frag_cache_clear); @@ -67,56 +69,51 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; WARN_ON_ONCE(!is_power_of_2(align)); - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with + * it. + */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -126,15 +123,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= ~(align - 1); nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -143,8 +142,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); From patchwork Wed May 24 15:33:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76736C77B73 for ; Wed, 24 May 2023 15:33:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A479900012; Wed, 24 May 2023 11:33:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15626900007; Wed, 24 May 2023 11:33:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01D31900012; Wed, 24 May 2023 11:33:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E64CC900007 for ; Wed, 24 May 2023 11:33:45 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BCDC31A0A2D for ; Wed, 24 May 2023 15:33:45 +0000 (UTC) X-FDA: 80825543610.24.9A7122B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id A7116C0023 for ; Wed, 24 May 2023 15:33:43 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=goBkd0Oo; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942423; a=rsa-sha256; cv=none; b=G2I8gi8Ujst7+vJvv8YZCOAasCbSx03bgqXyt+ugLRnEcP9Rtx4n+bj3p7dZQ1sr/letYt m1vMCOHP/Dw5S+QKF0kTgb2f3ovz/cql7PRpjvAXLbrkV6GeEwnG25h23XQ418mZsrCTuY 0vDqnHe2OvnFW+f3yE5vubp0vIEl/tY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=goBkd0Oo; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942423; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uekJL1E5Uz+9JLqowhoPzZVYALfVqMM/z7Y13jfCLzU=; b=UTi69orBQnJhbKbmg6i231yUQPY/rocqlXrQNsKoiDU12LlVB1dbHg9zsYzq3LDdGMYZS/ CBP22J9lgHNhwvtu/JbM87FclOotJ9OTv23tofumz7WRdg5v/B0C/TI4x9CCUEJKixCmI9 Cv4hvsEdh4jzQ8EaIYwGx9EcEvUadPI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uekJL1E5Uz+9JLqowhoPzZVYALfVqMM/z7Y13jfCLzU=; b=goBkd0OowYWyH3QUPeS3qWxBlGoHkHgSDnWtPPThmZjoQLf+Dx8VCmbgEpcr17+WibYYzg tnhem5GYfUXEOCXSMvqq7tI9kqo9RdMGEnkwxJAFzOP9WG9rWgquHsBKCDZtCLNII3j6F3 vcWQqo+xo3DMCX3bzqxinsHrBh9dC4U= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-168-UtOOYHJNMQqeOwkvKYu_rw-1; Wed, 24 May 2023 11:33:38 -0400 X-MC-Unique: UtOOYHJNMQqeOwkvKYu_rw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F027280BCA2; Wed, 24 May 2023 15:33:36 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A068492B0A; Wed, 24 May 2023 15:33:32 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next 05/12] mm: Make the page_frag_cache allocator handle __GFP_ZERO itself Date: Wed, 24 May 2023 16:33:04 +0100 Message-Id: <20230524153311.3625329-6-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A7116C0023 X-Stat-Signature: ymwdbaoqape9fjqgszkyi6i9mtwri41y X-Rspam-User: X-HE-Tag: 1684942423-294632 X-HE-Meta: U2FsdGVkX18CaKKB56Rh4SFXKI6TkbycwHvQShABpT0Xu5E32eQjpkMQK6AXq8d805D60Pj4b8JzItCfj2sfsztwPBT/uza9p/tG4gV/5fNS9/lIpoMRAAsYooYh5ou0sZsCnD4aDrm6aS0kBmLnkmCWKg0dQ+0ayJthppyMZfYrzQzZZoPm8vxIgSN92H0G7I7xYvuzJYOoGEdjsA12nwQwWABz38RiS+mdWesowACbyD3Q3BrTcb5QxXTVJotBCi8U/tUqIAkg355ij7mR8uWrFz89N6bOeAljEjMX9YBKbWTxbMoccLfunls07QHjMh6W98+/sfc5YAa0gxx5nf3d2em4KizADJdpcLqeT+tvOUNqqkOnDcnjdxU+ycoBk+iAoQOzoEflK7ngGBdvQS2jYFzEo5EI8vS/QnGXezdKO0KftrHSpj/9u931pigbXuPcmoXPTagQYBeqBCeaPWIKOUjYshgzGmJ8rgVZYpAQke3vBPepLguYaMw1qzA6RxBh1+Bi8Y6bZjXLltZWfxUmwDYXIL/VCMdoq2YyY72zfH5EVz1wnR2QIXaoyQhXFTXx/UEMt6T40arnn5lTLVC1jRcEqctVBADCdQAedsmxrHvxsZF/Na5E4jh47KGiYpNxidbbprMglhl2TZu3L6QI5B7QdUzV9Ufem+TcorZwiZa/aEL5UY9m/5OdCQNrqDT5WQ2q94CP5g0t876w8nIf3beofQhi57ZqNTInwfLBXCXrFh0SXer4f74ShTs9CgAcFY7CWPa17YR+0lcLf6gSG0ZF8OWm+gV/3zJORyRcxmCyWZuX2DawHHqaN+reJp+BmPDIONoo9POdsQuDRl512TbeAnJXDhBsZHnasCtCgNaFXLbV6ID2usRQFu9j6rBzQ6j9qurMplcxM8vGslXDHJub3gwet40ASyImj047cdiDCYJlN/gC2lOxoE5uhM6Tx83T4iuZpFa1rSv NJDJQe3K 3gRWQkpNgDTLIViUeRLndfld23Ybcp8btbctF43mncjrNprk5a1OUqt4g22tjQjMEj8kIszWpl0sSx5WgVPZsqbGCYQafG8Mqt9bXjE5WqV4muDmWfRhhAwN2GgkTKEDFaTptFBtjOAgfeXnbG89i3fCJakfVHNJAeQ/PMbjAey6wudO+Lkhdl2HftkH+JZ4QynrVdpTbUyt9d3J117WM+xaEQB75BNWFtql45hm5Tq6N+1A+Ge58ltl93HxWYuEgJ6n17qW7JNuQ3SgxUoQfyhx66ijhWw+2vfpkqWSaxa26Ckx9Ka2kBl5LJiYGk1QSkTjgaWMLIxGjUTXMRSHbhypxV+uA6mrqp4YCZZ06GxCJhMd909VqZXUiYVpghf9+xTMwqAV/wRtVwFPgxATDSJvh3bCe+JJGJgRV7nv8Vu1zW539EdFayHKiYkMzFyL769iTQzwrIGdD8KsE/NMLiyBt60uhEMnmORMZmD6KJ8UC204dRd7zLU6jV8k2mF3ik9iWf2qKRlzX9++IVg6ARj0mJh52DboDsIVOOQhraHL6MuJ512OfRmnSatYn6okL4HkRp4Kb9e9KC4wUNpZ1goVqulLoiB1jWCjidUOtToOfj1ML0e17sHa78jPTI2HqrkUy594ahgXRv72usAvdihdudnyrTOrWnzn/wmAtZpInVq4QnmD31Q7omT8tk3ynhNm2tlVuTEwZh7NP7dBZSf5QDLCF/idhPHmoIsmkaEDd+w6R2Acx1CMUQVwwejs2gBdMnvvQR6/LD1MXoErPX5w5GXtVXjZRARKaRtYKnvP0dzo7h+J97xju/kQ/JnEbMs2bqApk0R/lqkmbG2zyyqxjWBLgsvj1uxZ6ymagB+asp5NGCMchWNUmwgAqltcs962yNh+pUHXjGzc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator handle __GFP_ZERO itself rather than passing it off to the page allocator. There may be a mix of callers, some specifying __GFP_ZERO and some not - and even if all specify __GFP_ZERO, we might refurbish the page, in which case the returned memory doesn't get cleared. This is a potential bug in the nvme over TCP driver. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- mm/page_frag_alloc.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index ffd68bfb677d..2b73c7f5d9a9 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -23,7 +23,10 @@ static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, gfp_t gfp_mask) { struct folio *folio = NULL; - gfp_t gfp = gfp_mask; + gfp_t gfp; + + gfp_mask &= ~__GFP_ZERO; + gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; @@ -71,6 +74,7 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, { struct folio *folio = nc->folio; size_t offset; + void *p; WARN_ON_ONCE(!is_power_of_2(align)); @@ -133,7 +137,10 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, offset &= ~(align - 1); nc->offset = offset; - return folio_address(folio) + offset; + p = folio_address(folio) + offset; + if (gfp_mask & __GFP_ZERO) + return memset(p, 0, fragsz); + return p; } EXPORT_SYMBOL(page_frag_alloc_align); From patchwork Wed May 24 15:33:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B8EEC7EE2E for ; Wed, 24 May 2023 15:33:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8C54900015; Wed, 24 May 2023 11:33:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3CB0900007; Wed, 24 May 2023 11:33:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6710900015; Wed, 24 May 2023 11:33:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A60AF900007 for ; Wed, 24 May 2023 11:33:48 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 69C031A0A2F for ; Wed, 24 May 2023 15:33:48 +0000 (UTC) X-FDA: 80825543736.19.4C437CC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 8CB56A0018 for ; Wed, 24 May 2023 15:33:46 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IBgk8qpd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942426; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p71nalHI+W3yFqMm+4MkrrRPKSqs6pNOJ/pHSBSTc4w=; b=0fU8qWyUCIFrjdrGOkDsszNs+cZpV1veYdNCG5Hu9ntd4NIX+/huc0B261KxkhBP4UrQxy Ml4KCyTHlsqVQyaesVHtzh7xv67CkLg9jw4pN0iEbZEUWVhQkeke5SvfE1iG3sFlHMZ5PW Tm8EBNR9YPWo+3kVEG8ooOZ7PBkMlb4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IBgk8qpd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942426; a=rsa-sha256; cv=none; b=SvfQDxb3voMHs7D2g6wapWjHnjFeHi7plimdbVLoaKydhPrRZzlt1iupthom8/NgwkzW1B ZwYtOTw9Fq0smb75AhjerlUWKdA4kUCmZOQTlE36yxdyaPRMP2Nrw6IdMwHrxWndFU8Qp0 1vXb9KNgXz1AgdhVPwgcmOAIzQoKi1c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p71nalHI+W3yFqMm+4MkrrRPKSqs6pNOJ/pHSBSTc4w=; b=IBgk8qpdoYE7mccExShJQTtBgup5qhgUz1gm7wUXpvBFEzBaUkotkwz4SVp+inHP233+xM XT/taNcpKQqirJkjYgw2jEiUjumthe393TGWgzNpeUfHbqOAQ/sYcnNzhd2guXhK+h2DVR 4sV3hnqno1bh5NIU7C5vzirGwjGl6cM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-136-3nahLBV-MXSO5tBDJHqSWA-1; Wed, 24 May 2023 11:33:43 -0400 X-MC-Unique: 3nahLBV-MXSO5tBDJHqSWA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 19058800888; Wed, 24 May 2023 15:33:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 217A440C6CCC; Wed, 24 May 2023 15:33:37 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next 06/12] mm: Make the page_frag_cache allocator use per-cpu Date: Wed, 24 May 2023 16:33:05 +0100 Message-Id: <20230524153311.3625329-7-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 8CB56A0018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 8admdg1iqat4ytgciyadwi5wjoibsqej X-HE-Tag: 1684942426-528751 X-HE-Meta: U2FsdGVkX1/9mIS0Z89Nm2EaH68R0BjRx4JBYJeaV0P/+LJH9KPmHTmYOEZHrIHMxKJDiMwEWnDvYc5HxPR3Nc78+DMOeEHP8McuUhZf91Pb2eaRfEjeQEns78HUgGIH4/gT1QNxh7Rm85yrEklK6hry3Zzh8OVnHm5ShPWsdHB8IpenCJkwZn4CEKNA+x41mvTooN6tJssC8y64GhhfhMRhPCUmJf5OjjaiawtL8mu+3fi15MMS440EPZLA2IikeleSs1UPisiUqUUOzXLbU4V1Yn3BF4XAyUi8twyjpP6t6m6oe50gmDUgl1wnnR9I6/MvIl+Sk8MXpmFM4GmxbPWfky3EJv6vDnVT8pYFTJNMUMyfzlv8ukk/3Emvx+rjo5XR+VbE+1btCWMzbAsDJ0IxSNB6ZVFY4WVbjtnGofAr/h71p212n3XyMxo3NAX1YGkeHTBXL699SqlM4hZxLwyJk0LahraWt+texQaZovWe4tc2f39d1J8yvK2exslvkf8HNxY6suY4h6zjEY3bPvxcTh2K3sd1BqdnW7RIo76myCR8qGTl8f62/2+qgdRnFLCJvn3zioG7U+uQ3Ua+lHdGAzZlpQBhkw1WDeFXCuNzISu6sxKUxlp6UENDazgHBsAQjJ2MIpgvEvLjnh1HmPz37LLQWOnMci9chY91mWNgo0QN/Hu2AzOiv9qsXpS6Fsz9TN/bIvd3ynZDR43UARXM4uhr+P0pbK4bQl3LonSW7WFqu+RO4YHrGjXToems9w667JC17w9jQHE1BCFZsrOwZu59r1l/9hLb9jCspBBOdtayDbwpngVYyiy89x61W7d8i7IyLUwBs3PpTd1ZmOyAjZwFu9mxN6+og0DeOohvVXmWB4wd2BP7nAH+rc41UbLqLfiV15m5BilZadldqyGIkRSv91nFNbQx17t9LiHOc4O8EPrkx1bRQ/O0tkeEbcd32K0AbQAi3jZBySZ lFTTHyOc N9POTUEey0eqjN+ZwGdm9HgT7gS20YiEYIHpltlR5cFWo4dVY7w9XFBOwM15E1sUwqA5trxcws4RgIFT4483bEIaV/FdLZHpd/Apx5dqr4JHyxYNP9u10YyHJ6w2omsJxuqRWHvYQ1fAi73s2A9CUBvnSjmcdKVl+awbtiiqLEixKMMq9Ae11DnEkbzznVWaWOyhWZTvLl/IIQ6nuliG0lcOHWrEzkH2gXESAyE8kwY1eDIp7gllsRpoIHC98l+XMIYCmzoWRwddCGo+Owctfzj8dak+YLiYf0E8+5aVEu0R5zGkBi40ZXZMJc45sAa+/ck5p+9OjsYKICK63+qbmFnE+prg55jjDBxub+q4G1Gi7pgItWNWJaz7nBAOGkc+UDAAeqEBvMAYqly982LrT8XeHniGxuQ5pgmaM06zB9IvxEHaQZCFNgRCkBP/q1vYwppLFvzPtV63FXJvsFq4lzvoztFTlD1MoX6kdnYH/BfFebR5kOvfrpt+w3LQABQjqRsiA51F1NJaUQ8N4UNNve0D4bfGnKd78zbipogw0VlflPzJ7vrJUQJRdP0XtaNfSy/jw5gASiq+73LZs2JqZldQndomia5cVAZsTV1QqVKahHYE7dY0ZJTBK6WInB72w+eFACQ0pVyQef7f3FRIcVDtanFSGE7bi8BO2xzPXPTGjkRCcsZpxbRKyJDg178n+/emBvRzIOZVHrDsmQ0JIZul/Hlm2iplgS0wcdpbRCH4NXmvsISVFmfMQGdq4Oz9DhBgDeOojjad22be0S5zDETgARNRqSalConDRx+w0WSNvTC6umRlwSQyvRW6bfsiuMWckSRB+LmjaaIXdut0OzuUzHTN9YHLcw1rUowSGyzDrLx1BVt9CQhp6/R40YFoht/ABX5abVkmoY9E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator have a separate allocation bucket for each cpu to avoid racing. This means that no lock is required, other than preempt disablement, to allocate from it, though if a softirq wants to access it, then softirq disablement will need to be added. Make the NVMe, mediatek and GVE drivers pass in NULL to page_frag_cache() and use the default allocation buckets rather than defining their own. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- drivers/net/ethernet/google/gve/gve.h | 1 - drivers/net/ethernet/google/gve/gve_main.c | 9 - drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 6 +- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 13 +- drivers/nvme/target/tcp.c | 19 +- include/linux/gfp.h | 19 +- mm/page_frag_alloc.c | 202 +++++++++++++-------- net/core/skbuff.c | 32 ++-- 10 files changed, 163 insertions(+), 142 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index 98eb78d98e9f..87244ab911bd 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -250,7 +250,6 @@ struct gve_rx_ring { struct xdp_rxq_info xdp_rxq; struct xdp_rxq_info xsk_rxq; struct xsk_buff_pool *xsk_pool; - struct page_frag_cache page_cache; /* Page cache to allocate XDP frames */ }; /* A TX desc ring entry */ diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 55feab29bed9..9f0fb986d61e 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1249,14 +1249,6 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) } } -static void gve_drain_page_cache(struct gve_priv *priv) -{ - int i; - - for (i = 0; i < priv->rx_cfg.num_queues; i++) - page_frag_cache_clear(&priv->rx[i].page_cache); -} - static int gve_open(struct net_device *dev) { struct gve_priv *priv = netdev_priv(dev); @@ -1340,7 +1332,6 @@ static int gve_close(struct net_device *dev) netif_carrier_off(dev); if (gve_get_device_rings_ok(priv)) { gve_turndown(priv); - gve_drain_page_cache(priv); err = gve_destroy_rings(priv); if (err) goto err; diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c index d1da7413dc4d..7ae8377c394f 100644 --- a/drivers/net/ethernet/google/gve/gve_rx.c +++ b/drivers/net/ethernet/google/gve/gve_rx.c @@ -634,7 +634,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx, total_len = headroom + SKB_DATA_ALIGN(len) + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - frame = page_frag_alloc(&rx->page_cache, total_len, GFP_ATOMIC); + frame = page_frag_alloc(NULL, total_len, GFP_ATOMIC); if (!frame) { u64_stats_update_begin(&rx->statss); rx->xdp_alloc_fails++; diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index d90fea2c7d04..859f34447f2f 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -143,7 +143,7 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q, dma_addr_t addr; void *buf; - buf = page_frag_alloc(&q->cache, q->buf_size, GFP_ATOMIC); + buf = page_frag_alloc(NULL, q->buf_size, GFP_ATOMIC); if (!buf) break; @@ -296,8 +296,6 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(entry->buf); entry->buf = NULL; } - - page_frag_cache_clear(&q->cache); } static void @@ -311,8 +309,6 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - - page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.h b/drivers/net/ethernet/mediatek/mtk_wed_wo.h index 7a1a2a28f1ac..f69bd83dc486 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.h +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.h @@ -211,8 +211,6 @@ struct mtk_wed_wo_queue_entry { struct mtk_wed_wo_queue { struct mtk_wed_wo_queue_regs regs; - struct page_frag_cache cache; - struct mtk_wed_wo_queue_desc *desc; dma_addr_t desc_dma; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index dcc35f6bff8c..145cf6186509 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,8 +147,6 @@ struct nvme_tcp_queue { __le32 exp_ddgst; __le32 recv_ddgst; - struct page_frag_cache pf_cache; - void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); @@ -482,9 +480,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx]; u8 hdgst = nvme_tcp_hdgst_len(queue); - req->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + req->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!req->pdu) return -ENOMEM; @@ -1303,9 +1300,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) struct nvme_tcp_request *async = &ctrl->async_req; u8 hdgst = nvme_tcp_hdgst_len(queue); - async->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + async->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!async->pdu) return -ENOMEM; @@ -1325,7 +1321,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 984e6ce85dcd..cb352f5d2bbf 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -169,8 +169,6 @@ struct nvmet_tcp_queue { struct nvmet_tcp_cmd connect; - struct page_frag_cache pf_cache; - void (*data_ready)(struct sock *); void (*state_change)(struct sock *); void (*write_space)(struct sock *); @@ -1338,25 +1336,25 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue, c->queue = queue; c->req.port = queue->port->nport; - c->cmd_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->cmd_pdu = page_frag_alloc(NULL, sizeof(*c->cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->cmd_pdu) return -ENOMEM; c->req.cmd = &c->cmd_pdu->cmd; - c->rsp_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->rsp_pdu = page_frag_alloc(NULL, sizeof(*c->rsp_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->rsp_pdu) goto out_free_cmd; c->req.cqe = &c->rsp_pdu->cqe; - c->data_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->data_pdu = page_frag_alloc(NULL, sizeof(*c->data_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->data_pdu) goto out_free_rsp; - c->r2t_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->r2t_pdu = page_frag_alloc(NULL, sizeof(*c->r2t_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->r2t_pdu) goto out_free_data; @@ -1485,7 +1483,6 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fa30100f46ad..baa25a00d9e3 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -304,18 +304,19 @@ extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); -extern void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align); - -static inline void *page_frag_alloc(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask) +extern void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, + unsigned long align); +extern void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align); + +static inline void *page_frag_alloc(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, 1); + return page_frag_alloc_align(frag_cache, fragsz, gfp, 1); } -void page_frag_cache_clear(struct page_frag_cache *nc); - extern void page_frag_free(void *addr); #define __free_page(page) __free_pages((page), 0) diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index 2b73c7f5d9a9..b035bbb34fac 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,28 +16,25 @@ #include #include +static DEFINE_PER_CPU(struct page_frag_cache, page_frag_default_allocator); + /* * Allocate a new folio for the frag cache. */ -static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +static struct folio *page_frag_cache_refill(gfp_t gfp) { - struct folio *folio = NULL; - gfp_t gfp; + struct folio *folio; - gfp_mask &= ~__GFP_ZERO; - gfp = gfp_mask; + gfp &= ~__GFP_ZERO; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); + folio = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (folio) + return folio; #endif - if (unlikely(!folio)) - folio = folio_alloc(gfp, 0); - if (folio) - nc->folio = folio; - return folio; + return folio_alloc(gfp, 0); } void __page_frag_cache_drain(struct page *page, unsigned int count) @@ -51,63 +48,70 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) EXPORT_SYMBOL(__page_frag_cache_drain); /** - * page_frag_cache_clear - Clear out a page fragment cache - * @nc: The cache to clear + * page_frag_alloc_align - Allocate some memory for use in zerocopy + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The size of the fragment desired + * @gfp: Allocation flags under which to make an allocation + * @align: The required alignment + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and fragments are broken off + * sequentially and returned to the caller with a ref until the folio no longer + * has enough spare space - at which point the allocator's ref is dropped and a + * new folio is allocated. The folio remains in existence until the last ref + * held by, say, an sk_buff is discarded and then the page is returned to the + * page allocator. * - * Discard any pages still cached in a page fragment cache. + * Returns a pointer to the memory on success and -ENOMEM on allocation + * failure. + * + * The allocated memory should be disposed of with folio_put(). */ -void page_frag_cache_clear(struct page_frag_cache *nc) -{ - struct folio *folio = nc->folio; - - if (folio) { - VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - folio_put_refs(folio, nc->pagecnt_bias); - nc->folio = NULL; - } -} -EXPORT_SYMBOL(page_frag_cache_clear); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align) +void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, unsigned long align) { - struct folio *folio = nc->folio; + struct page_frag_cache *nc; + struct folio *folio, *spare = NULL; size_t offset; void *p; WARN_ON_ONCE(!is_power_of_2(align)); - if (unlikely(!folio)) { -refill: - folio = page_frag_cache_refill(nc, gfp_mask); - if (!folio) - return NULL; - - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + if (!frag_cache) + frag_cache = &page_frag_default_allocator; + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = folio_is_pfmemalloc(folio); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = folio_size(folio); + nc = get_cpu_ptr(frag_cache); +reload: + folio = nc->folio; + offset = nc->offset; +try_again: + + /* Make the allocation if there's sufficient space. */ + if (fragsz <= offset) { + nc->pagecnt_bias--; + offset = (offset - fragsz) & ~(align - 1); + nc->offset = offset; + p = folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + if (gfp & __GFP_ZERO) + return memset(p, 0, fragsz); + return p; } - offset = nc->offset; - if (unlikely(fragsz > offset)) { - /* Reuse the folio if everyone we gave it to has finished with - * it. - */ - if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { - nc->folio = NULL; + /* Insufficient space - see if we can refurbish the current folio. */ + if (folio) { + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) goto refill; - } if (unlikely(nc->pfmemalloc)) { __folio_put(folio); - nc->folio = NULL; goto refill; } @@ -117,30 +121,56 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; offset = folio_size(folio); - if (unlikely(fragsz > offset)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - nc->offset = offset; + if (unlikely(fragsz > offset)) + goto frag_too_big; + goto try_again; + } + +refill: + if (!spare) { + nc->folio = NULL; + put_cpu_ptr(frag_cache); + + spare = page_frag_cache_refill(gfp); + if (!spare) return NULL; - } + + nc = get_cpu_ptr(frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + nc->pfmemalloc = folio_is_pfmemalloc(spare); + if (nc->folio) + goto reload; } - nc->pagecnt_bias--; - offset -= fragsz; - offset &= ~(align - 1); + nc->folio = spare; + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* + * The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ nc->offset = offset; - - p = folio_address(folio) + offset; - if (gfp_mask & __GFP_ZERO) - return memset(p, 0, fragsz); - return p; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return NULL; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -152,3 +182,25 @@ void page_frag_free(void *addr) folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); + +/** + * page_frag_memdup - Allocate a page fragment and duplicate some data into it + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + */ +void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask) +{ + void *q; + + q = page_frag_alloc_align(frag_cache, fragsz, gfp, align_mask); + if (!q) + return q; + + return memcpy(q, p, fragsz); +} +EXPORT_SYMBOL(page_frag_memdup); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index cc507433b357..225a16f3713f 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -263,13 +263,13 @@ static void *page_frag_alloc_1k(struct page_frag_1k *nc, gfp_t gfp_mask) #endif struct napi_alloc_cache { - struct page_frag_cache page; struct page_frag_1k page_small; unsigned int skb_count; void *skb_cache[NAPI_SKB_CACHE_SIZE]; }; static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache); +static DEFINE_PER_CPU(struct page_frag_cache, napi_frag_cache); static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache); /* Double check that napi_get_frags() allocates skbs with @@ -291,11 +291,9 @@ void napi_get_frags_check(struct napi_struct *napi) void *napi_alloc_frag_align(unsigned int fragsz, unsigned int align) { - struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); - fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align); + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align); } EXPORT_SYMBOL(napi_alloc_frag_align); @@ -305,15 +303,12 @@ void *netdev_alloc_frag_align(unsigned int fragsz, unsigned int align) fragsz = SKB_DATA_ALIGN(fragsz); if (in_hardirq() || irqs_disabled()) { - struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align); + data = page_frag_alloc_align(&netdev_alloc_cache, + fragsz, GFP_ATOMIC, align); } else { - struct napi_alloc_cache *nc; - local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align); + data = page_frag_alloc_align(&napi_frag_cache, + fragsz, GFP_ATOMIC, align); local_bh_enable(); } return data; @@ -691,7 +686,6 @@ EXPORT_SYMBOL(__alloc_skb); struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_t gfp_mask) { - struct page_frag_cache *nc; struct sk_buff *skb; bool pfmemalloc; void *data; @@ -716,14 +710,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_mask |= __GFP_MEMALLOC; if (in_hardirq() || irqs_disabled()) { - nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } else { local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache.page); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); local_bh_enable(); } @@ -811,8 +803,8 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, } else { len = SKB_HEAD_ALIGN(len); - data = page_frag_alloc(&nc->page, len, gfp_mask); - pfmemalloc = nc->page.pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } if (unlikely(!data)) From patchwork Wed May 24 15:33:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D5DAC7EE32 for ; Wed, 24 May 2023 15:33:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A60D900016; Wed, 24 May 2023 11:33:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1301D900007; Wed, 24 May 2023 11:33:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC411900016; Wed, 24 May 2023 11:33:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D9955900007 for ; Wed, 24 May 2023 11:33:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B7A1540A2C for ; Wed, 24 May 2023 15:33:49 +0000 (UTC) X-FDA: 80825543778.06.E1D4D46 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 1594E40019 for ; Wed, 24 May 2023 15:33:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bjhZvAul; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hgbi0fncQFlNqYJvJIncCzqzDTHwL/0fY7UO/lvli4s=; b=QUH1z0ORzgtOQ8OAKVfijzg81ynOm0zVWBEjCTMROkbdrlX/rGLolqHvtApkVFiKzZeqeT x7un2WlOxaCsIgnY3ChfA6g9M4RrAMrHCJxuJjiWvQsFNan70YZEzLrZtaCMkDxNw8ezAd lhx/S+GzAaBuCyt4THFDOnEGjaDo65U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942428; a=rsa-sha256; cv=none; b=PLFRcZtbFyu1Xhzdw7H2IF2OXi1JIAvIH6CK/CL2E//3GDT/v942PHlSESV1OfzfEOIZvg ZIzN0O9sFhGiZxV88EYxmD4wpcEK9Oed8n08tYhSMwBMiKLlE4TbeVC/t1B/q8uwizczMZ SiZ5OAT2mAqooT+T76cniVwDO2gyXNg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bjhZvAul; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942427; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hgbi0fncQFlNqYJvJIncCzqzDTHwL/0fY7UO/lvli4s=; b=bjhZvAulGll7IJ/qVh+Q5ipODUMBSh9DoFt3h0jhJoCZ06vUTtVrAjFZZOYo7uU+44CXJM jNn1SW/kHaDLu/FJKy4TA2i5M3u3KyaaSfcsp5ZOR3ZDZGYeYiBB6Y+g64os+huIkgCphL 5knqKvpUIGZz1hRlPN/rVPJ4iRGumUE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-170-IhsFsMjsP6yLutFsBhfOSA-1; Wed, 24 May 2023 11:33:44 -0400 X-MC-Unique: IhsFsMjsP6yLutFsBhfOSA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6EDD280027F; Wed, 24 May 2023 15:33:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id C2DC4C1ED99; Wed, 24 May 2023 15:33:41 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 07/12] net: Clean up users of netdev_alloc_cache and napi_frag_cache Date: Wed, 24 May 2023 16:33:06 +0100 Message-Id: <20230524153311.3625329-8-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Stat-Signature: z94h9siopin3g9k5xgsoozzep5cbatwf X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1594E40019 X-Rspam-User: X-HE-Tag: 1684942427-486341 X-HE-Meta: U2FsdGVkX19WL4SQkQPd6R7AZZv2pZD2Ilrp5BIgJeFkf8Yw2VvcqsPCDnleSv/27usjn73miQuzB0ygmdfq/Qweaz12WceULoNo5lm3F2C+wIPgCC1cKGyrtHHZHmIYHPzhTi9l1Ryk5l8bdGbqDEK8VMLa0wg+gilNRmjpc20szoUGKbliirSByYmBvLA4n3ehhGFTgJqJ5ydLEN2zhW/ku8UY9mvPshrI9HkgeQWYm0CBMT51BolitwGvML5/3f1t72NDbVLdB8t+xv6msGQ3ukYXOaubvbXSWN92jlEh/W1nTl9di06z469SAe7CSEGxMeBPnhFETTSCbbTfVcoerYpsm4Q8j5blT22rtPTOft00Eqgy2kuiElGzE/lgeiBUQQhI2iNZTSoSaHLkqwBOOU6aXeYrLU4ZdbXigxHuKLTPBDNK0tpEMm+sd5lBfRs5mfyqFQoEQhJD/FwhXXMXvKWyZeSqlA7to+HLXLbgGp3orhyRrb/SCZ6Vcp88MZ/lUORM5hZXT7jkkGAgESa+tUWphVVwaDPGfRBE+Xd4ObfPkeTIQRc7UCWVhaNmfPcJT0E4Bn+hni/mJvRkMm9wTlF10X8vdFbhbd5+W7H5gmbwBfAejZs4yhycPWf6ECBX9qmyJ4gJfothp+hPWtrSoC54d3VqCwu8wEWEH2s9PjmMOa9PKGVROkBlmXRI2ShjJUXthPOwvEK+bINmmFn2tnlljSobtFLR8Syz7CdulvTBIcOqrFhr8zLsNxI4FNjRp8rz/Po1ssgzW6xHtCyjnAcRtSiT2+KNUb+u5H0iF3007E0dUZGo0qmEJHco1LDG+gOYNPH8s6FSHr6Dgft1fvQvyxFPWF0zptcX7/wxDtozIRPO312JLdYdqfFvG9H7pXM+Wv2qsChgorDkRCszXMghk4fvuYqhgnvIa+OnDLQFqVUIgPFx1xRYc54kD/KVARt7SW/w4r1pKgT J4kRzTwE Ox+nQCXk0dQG45gfEQlE5JDLLkWKNlmrzM2vlJ7HOktcjq42geYYs6Gqn8BP1PlTx0p10q+T6ZcWRMNDYastrMmmFKdbc6aNyZ5w7nzdhSdbUX8tdF4jo8Bg8C3sPAhF8aTxTEyMeoaS7gDOaqI/1g0gafciIympKXNs+2BS46E4EREqZ6TElLswZMyZmMf0/kY91TLyTlhhysYTM6qnO9BWekxnNfKIiiKhWqvh1y3AZQ2S9A9Ly4aVkkeeT0DwjFzkQvm0fZaN3WZRDNl3CXZE2yQ9j7qDEdb1ZCLVPWL01Txn/04AAZCSpthwZ4dxxuzThdLuTn/gxH5sUoZFOLMtbIOl57AndPXT0DE37wmJUpLV7Ou5GepVhccs42VrCBvrTX72ziVQYm/wadKLrI9oUMqbR5uzhtr0iANoeZKpPzq3bcup5P0YjG4TihlAWxqwLyRHeFpIeoPJtqY5TAkHXU+YC2B1Cu4dSYttC2z/P1KJsxuuuFADaPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The users of netdev_alloc_cache and napi_frag_cache don't need to take the bh lock around access to these fragment caches any more as the percpu handling is now done in page_frag_alloc_align(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-mm@kvack.org --- include/linux/skbuff.h | 3 ++- net/core/skbuff.c | 29 +++++++++-------------------- 2 files changed, 11 insertions(+), 21 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 41b63e72c6c3..e11a765fe7fa 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -252,7 +252,8 @@ /* Maximum value in skb->csum_level */ #define SKB_MAX_CSUM_LEVEL 3 -#define SKB_DATA_ALIGN(X) ALIGN(X, SMP_CACHE_BYTES) +#define SKB_DATA_ALIGNMENT SMP_CACHE_BYTES +#define SKB_DATA_ALIGN(X) ALIGN(X, SKB_DATA_ALIGNMENT) #define SKB_WITH_OVERHEAD(X) \ ((X) - SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 225a16f3713f..c2840b0dcad9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -291,27 +291,20 @@ void napi_get_frags_check(struct napi_struct *napi) void *napi_alloc_frag_align(unsigned int fragsz, unsigned int align) { - fragsz = SKB_DATA_ALIGN(fragsz); - + align = min_t(unsigned int, align, SKB_DATA_ALIGNMENT); return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align); } EXPORT_SYMBOL(napi_alloc_frag_align); void *netdev_alloc_frag_align(unsigned int fragsz, unsigned int align) { - void *data; - - fragsz = SKB_DATA_ALIGN(fragsz); - if (in_hardirq() || irqs_disabled()) { - data = page_frag_alloc_align(&netdev_alloc_cache, + align = min_t(unsigned int, align, SKB_DATA_ALIGNMENT); + if (in_hardirq() || irqs_disabled()) + return page_frag_alloc_align(&netdev_alloc_cache, fragsz, GFP_ATOMIC, align); - } else { - local_bh_disable(); - data = page_frag_alloc_align(&napi_frag_cache, + else + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align); - local_bh_enable(); - } - return data; } EXPORT_SYMBOL(netdev_alloc_frag_align); @@ -709,15 +702,11 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, if (sk_memalloc_socks()) gfp_mask |= __GFP_MEMALLOC; - if (in_hardirq() || irqs_disabled()) { + if (in_hardirq() || irqs_disabled()) data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); - pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); - } else { - local_bh_disable(); + else data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); - pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); - local_bh_enable(); - } + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); if (unlikely(!data)) return NULL; From patchwork Wed May 24 15:33:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B6D0C77B7C for ; Wed, 24 May 2023 15:33:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEC0A900017; Wed, 24 May 2023 11:33:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E9C5A900007; Wed, 24 May 2023 11:33:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3D23900017; Wed, 24 May 2023 11:33:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C222F900007 for ; Wed, 24 May 2023 11:33:52 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6CCFC160A47 for ; Wed, 24 May 2023 15:33:52 +0000 (UTC) X-FDA: 80825543904.05.7D4C953 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id ACBCB40002 for ; Wed, 24 May 2023 15:33:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QP6BAEVx; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942430; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PZf0byHPp2E2QKGUTDTD+TlMjPhCnr7dpojTO1v28qY=; b=kbd6/L9mV4fOFkAS+N8UXX9IfmASRJ2mYiUKD8AnC5XO2s38frgjTyGc2kzlevLP2Y/vfM gcg/Bs4wvOTohX5sWamXFIUbceF9OctRxTw8nL+mgtu7jtqrlbXRt8+PR8PHd6BA1VjHgK N4wQGg7ntXl2YHYKIUPVqorkgVJvcjo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942430; a=rsa-sha256; cv=none; b=AzN1RJ4ZoWosRVyw+fwSAkeE8G2UODweo8P/fgRX/WJAnFkOi8BKKG38TxP4SWdTpP47mR VpBs/r39/SmVUZgIn/i4X6ZZt/5FmjgFaHkmzMG2WfWZ2NS+dgizOiwoaPrZb3waETSrGY KPvWHCQzcA5Pz0N1liD1fTh7BzwPaqA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QP6BAEVx; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PZf0byHPp2E2QKGUTDTD+TlMjPhCnr7dpojTO1v28qY=; b=QP6BAEVxhqStJDRIg9Zth49eUjEOhXJ9poYwY1oF0Jz1xXTMrLuHNgGdlH4ykykXCaBNZG IqTUC72W8bSihIpjAzgMScHkWCVAlPXKpUis7LEQ9XtJ8oBXe4iEg3aPfzdVlfWb5Ih0G2 F+OzZvMHtiAKlxi4HLV16IJWWtErmJs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-26-wF7-Cb2KPtKxfSJZIAw0Dg-1; Wed, 24 May 2023 11:33:46 -0400 X-MC-Unique: wF7-Cb2KPtKxfSJZIAw0Dg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E46EB811E85; Wed, 24 May 2023 15:33:45 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4289240C6EC4; Wed, 24 May 2023 15:33:44 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 08/12] net: Copy slab data for sendmsg(MSG_SPLICE_PAGES) Date: Wed, 24 May 2023 16:33:07 +0100 Message-Id: <20230524153311.3625329-9-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: 4ucprwpjso8ds5pbibmws67wbe4fmgmw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: ACBCB40002 X-Rspam-User: X-HE-Tag: 1684942430-555625 X-HE-Meta: U2FsdGVkX1+U3ZgJ5zIn6BmUBdwiOQzMckQJ6RM7sa9SbPJSWzkoMZOUrLIsPk4arpSQkuFy2Hqzln84N0EHeUK0V3zlc6bLtDDxWYGPZak8lmVtGGLTM3Ks07mVF0mWmcKgACI7d9Rhqk1aX7WSIAtdNOcz2fsK3ExD4aKJw9kwGbsPEyWYpf+2DBVWN2lz1zQNyKr0iByXhy7nX9c3Kn/DX+WZxZV+rghEyAhISf6fxSyjZa/6a0Axl48XeVJTFn0WHUeoW84x6OUKW4QO5R7PomjMVYJ9Z8OzZpn97G7CmFtNKaGbtRaepOuaZIdQ21kHuzVWtCG6qzJ1lJ2LbNGsNi5hexti2QGZPTViywiZPQmUGnkMSzsVbuFvgTdznPdeWk7CkOcJcnrOw5DqcfLwp24TxVaZP4nEHHuX8tNtXMv+r35b21IsQVa4G3bxW1VUkxeaeLg+Z3ky48X3zdH6rl2Hb0dvWWX28bxx4xhwgYlfejUu8jp2uH53n2ukjAF/YsrngCnkG4BJP/Ppb1pggcVwPdk2isp0iIcZfx3xj57skQahbDh38yYV1TXEuKYZHPQjoxYzYCFPJ/QTGuHARItF+z99gJriFrOy6vPPkQ0d4KLjT7QV2BHgvZWWprYi2IvGr8BQByAHJbB4y191PDWYvSR8Qhde/fw0gY7Tjxi24x7KNoS8YhL5xVpralthVTUYj7GhYmWcWUk3yzdBZHRJe6u7C66Y354p2bNClRlUM4XykQS0/Js92jC2GQhVP5biABkLVYkRrCl6wYaCR/DZqyVRLkrXlrXzyxqmIxkisvVv3gawRIGoLWbFqtYqSWTdI8MQuZED7tBT4ZqZzNHsOcWDqOUi9L4fCAj6YiBIXD/dxccIPXnJ8t3GGgqsQOCTruxka7j//XY0ojqOYlsvcD8oszRDtkAa2bqQj2myLLpAW+YK9GKCit2qjoz1vkx5ctIOkkYNQ5E tS8L1jbD fMeHt7QXA3c5ZJzeb67q2oIh+9hhIyDwQVr4CXhFo0Cig69YXGJKKuCWSX+JIgI3mQS1J1Xi16buQoDe2+DwlfIkWcgRuD143qRLk3g4HRJ/d2GwHORLLIFqFJ6/tRuseRt8ayf1KxuMNyz3vGPHVQxwj1bWbMhTg6u1iaaYTRRXvEz8vP8T41SbTjyDnDMajJqBUiKSsZK208C0/RW7S75sz44UBxW+sSyAL4CKW5ADbkD/o4S9RMjfnPqwDuibou24piOrd9L5s1LsxSmU8hwC3IYSwkchLt+yZCU6vyCN57Y1sbi2lxvSKNWW36b7dEHg7B1faxJ7xuejIGfHv4ifCNtZnfBMjNzVSvPcEby5rsQPO0W3hgrLBlNF2fzwYEFn14CpJARyO9pU5YTlZvRUxEVUptNUAdFGrBmaRiiE0Bku3O1c+geF0eyvnpzUN1HvkH0fGHkWZVvRoufWWuTvrRz7bPbfe8DfX66uJns2xcbR32BQk8zvS8Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() is passed MSG_SPLICE_PAGES and is given a buffer that contains some data that's resident in the slab, copy/coalesce it rather than returning EIO. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 3 +++ net/core/skbuff.c | 33 ++++++++++++++++++++++++++++++--- 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index e11a765fe7fa..11d98990f5f1 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5084,6 +5084,9 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, ssize_t maxsize, gfp_t gfp); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index c2840b0dcad9..a16499b9942b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6927,17 +6927,44 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, break; } + if (space == 0 && + !skb_can_coalesce(skb, skb_shinfo(skb)->nr_frags, + pages[0], off)) { + iov_iter_revert(iter, len); + break; + } + i = 0; do { struct page *page = pages[i++]; size_t part = min_t(size_t, PAGE_SIZE - off, len); - - ret = -EIO; - if (WARN_ON_ONCE(!sendpage_ok(page))) + bool put = false; + + if (PageSlab(page)) { + const void *p; + void *q; + + p = kmap_local_page(page); + q = page_frag_memdup(NULL, p + off, part, gfp, + ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } else if (WARN_ON_ONCE(!sendpage_ok(page))) { + ret = -EIO; goto out; + } ret = skb_append_pagefrags(skb, page, off, part, frag_limit); + if (put) + put_page(page); if (ret < 0) { iov_iter_revert(iter, len); goto out; From patchwork Wed May 24 15:33:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8175BC7EE2D for ; Wed, 24 May 2023 15:34:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE0F7900019; Wed, 24 May 2023 11:33:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B696D900007; Wed, 24 May 2023 11:33:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2F31900019; Wed, 24 May 2023 11:33:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 946E9900007 for ; Wed, 24 May 2023 11:33:58 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 17675160A45 for ; Wed, 24 May 2023 15:33:58 +0000 (UTC) X-FDA: 80825544156.28.5F64568 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 217298001D for ; Wed, 24 May 2023 15:33:55 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HjdFC6t2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942436; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hDpyQQ1JedFG6lTxY3chHyZphdbNgdYt0GxXfK9/WHk=; b=8laJjx5HVeCVmNqS2epjjgyqcgn69byQgHDC3Nrvb5rELnVgeXhjkORwDqq8A0AErJA9mV GhZeQYeMtwwtMkzy8v07BOUkXhYaBTva1/rJJhndzBiJFNX1ib3u3X3gqQ9Dn2WCXSWVGR caomsJx8YBuPtaxkpFiws+95vR3hdN4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HjdFC6t2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942436; a=rsa-sha256; cv=none; b=crFU6Z8UBxp9NGtMrcLDxKSA46m8LBUuCYajXK5ZeT/YvxGz52qysRUWBbaXLbthaM4pfO 5ptefto8dWojRv7u2YF27nvtDbhWkscZ6RwTF2fqjlXY3349TgHSoG5mkI/VYPjH/BVIsv 3INUTuGInkd0Q6eU0yQcZvBvCstulQ8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hDpyQQ1JedFG6lTxY3chHyZphdbNgdYt0GxXfK9/WHk=; b=HjdFC6t2VP1aiwLsb2DMW0wim/+RCY5hWVK/NLATyj8Xf2OiEcKJL6HXAgpk/27gYVdRgM 3C7WQsBfQHDuBBlcWmqb5680fhxtuHzJsV/fxybz894VZKqkDqeabcVlTV5i1sirS2sz2c A1fY1oybZtzfKjTU8DuqyOfyzGFZGZY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-164-4pleeuJrPNyai-iLRApvog-1; Wed, 24 May 2023 11:33:49 -0400 X-MC-Unique: 4pleeuJrPNyai-iLRApvog-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A7B73849528; Wed, 24 May 2023 15:33:48 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9B045C164ED; Wed, 24 May 2023 15:33:46 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chuck Lever , Boris Pismenny , John Fastabend Subject: [PATCH net-next 09/12] tls/sw: Support MSG_SPLICE_PAGES Date: Wed, 24 May 2023 16:33:08 +0100 Message-Id: <20230524153311.3625329-10-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: 217298001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1c7jqejjbibymwp3qktnmecuhkiekwn4 X-HE-Tag: 1684942435-699510 X-HE-Meta: U2FsdGVkX1/8nmj7w+4Qm62144FmrXTIuBFohtwr9PYdEoop/kIOKiHuOauShXIzqLGTa61Ar10UAYBlkqm+tLxNXVcnyrFO+WN0TKD84WgsjcAnVCw+LoqJCYTREan8fJPD/ZQsnIW/ihklrmQUiO4vkglBy5xDQkCV15jXq9RrBZJ5Q3himfenk+3e6Oufs+//APSfKPmMO5J8GCsyZxWpnzlJ6JVgRJiZikV6/G9kGMxbYHPe8gWkTMQlAK3zOkbm87ibEM14xHU5I/4cw4lMA4JFNrUt/L09nkiymOQlHFwErWN+kei50kkLMIrQ6n66UT2B89zIdLNEjWiS23Kpbe/CtggehqVF9dIeHYPU7svScG9usP2nNoR75hbVVUBMm7w6SOZ+2OFVGQ1qRTrfKzP9Nv4c8whhcfoiJYXp2knNP9UrabUzOJ6Tq6JfD5BU+eEZ3JbGN57bau/4YMWzrF2FS8eVrhpeJdydcBkASQA/x26/PaOFlY14vwmxt6EcvniBA1aPCGOmicJLY2yY1ACVvhaein3Pei4EyVs/tAoFkJKI8E2E0W10k9iC6E1wHnuNXI7GqacdhWNGVVk3PzoHlUnhbeuDryOL46fpRCAflnVPCniYiTzdnqT0zyNPP+kjhf70SMm3noTrqARcfPMfisl+SIxpQivm5HcD43TDVdkv34BxJj8a5aXSakshTC1uwnoEVpNafqIxFEr20DyWdVVdonHv9s+7FXrD+nD3VYs6UmYcCXZCngn2m9NB2k6aQ24zRHJbQiamK4vQr1l7+TM0YQK4Hm2rB+SmuuZpYX6j2pUF+V4AAMOGG/FZkJPvcoB+aDNycUWB8+kRZLIC7FjBujlss4vLCXsC2Vly92OBXIkuZdK6CTPjwBXw6GmTuwcnmuGn3P1782QqeZFb3HnDi1SjO0+9KqxHHJoFachLA1NCuK6lDhbEbdrC5q3kEMbf6WEkWW3 kQAak4Yt dA5XUHCeoi3/0143+2IVINSKbv6xy6NQOnDnyGC0YVEc/XPJSkq5AuM2iaMieQ2HnMcJ7Eq2b9B4kat+bhO5W1oPbHBWKfm71zog3hmxNOxFaKvZHXrvwIyWWQ8QnXTmU++4evDm2W8SPyfkse8NeYgMKvjqIdu00Qleis/MXu79iH8MTDi5HvhBWMso/mXRwQJ9SxGFCWgvuvgTCd+C7I5+PHzdXop0KjuSiM9m2GS3BoKKn18gvWe6s3+97zYRboVpzDuuxKMeol7sqBUuHlFlPwk+Fwn+Ef0jfZ2UbDmxE+gQnT1T0CD4DTwyjIJ4zPX7ciBCtJaecTc3BsEj0YGlawvOUQ8tiGc0nFJk0dwefSyLRGZU6mrwWAQh22H2bFq01lptP9vNulQdVPZgu2N1NK7+2TPsDTyr64367fr1nEfcRD74KCQGggqIwbrVxHr9xV334UG/xuoj7SfgxwVvNh4pJgXL1s35iCs4tfLGF+facAIJco7v6IQTMB4lEBJ6g7xzJ4q4bQoGm954BHnC5elNA3Q3lNkd0dA0lY3WE8gU3tG4GfCetNg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TLS's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible and copied the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_sw.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 635b8bf6b937..0ccef8aa9951 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -929,6 +929,49 @@ static int tls_sw_push_pending_record(struct sock *sk, int flags) &copied, flags); } +static int rls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, + struct sk_msg *msg_pl, size_t try_to_copy, + ssize_t *copied) +{ + struct page *page = NULL, **pages = &page; + + do { + ssize_t part; + size_t off; + bool put = false; + + part = iov_iter_extract_pages(&msg->msg_iter, &pages, + try_to_copy, 1, 0, &off); + if (part <= 0) + return part ?: -EIO; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, part); + return -ENOMEM; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + + sk_msg_page_add(msg_pl, page, part, off); + sk_mem_charge(sk, part); + if (put) + put_page(page); + *copied += part; + try_to_copy -= part; + } while (try_to_copy && !sk_msg_full(msg_pl)); + + return 0; +} + int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); @@ -1018,6 +1061,17 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) full_record = true; } + if (try_to_copy && (msg->msg_flags & MSG_SPLICE_PAGES)) { + ret = rls_sw_sendmsg_splice(sk, msg, msg_pl, + try_to_copy, &copied); + if (ret < 0) + goto send_end; + tls_ctx->pending_open_record_frags = true; + if (full_record || eor || sk_msg_full(msg_pl)) + goto copied; + continue; + } + if (!is_kvec && (full_record || eor) && !async_capable) { u32 first = msg_pl->sg.end; @@ -1080,8 +1134,9 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) /* Open records defined only if successfully copied, otherwise * we would trim the sg but not reset the open record frags. */ - tls_ctx->pending_open_record_frags = true; copied += try_to_copy; +copied: + tls_ctx->pending_open_record_frags = true; if (full_record || eor) { ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, record_type, &copied, From patchwork Wed May 24 15:33:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A427C77B7A for ; Wed, 24 May 2023 15:33:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91FCB900018; Wed, 24 May 2023 11:33:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CEC4900007; Wed, 24 May 2023 11:33:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7985A900018; Wed, 24 May 2023 11:33:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 620CF900007 for ; Wed, 24 May 2023 11:33:57 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A7C2A160A62 for ; Wed, 24 May 2023 15:33:56 +0000 (UTC) X-FDA: 80825544072.17.81692FA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id D8713C0018 for ; Wed, 24 May 2023 15:33:54 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=X2IFqekL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942434; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/e3etHO5quwZ5TcFINBOwsTOV20jtXx9aa4v5VXGvJw=; b=u1pjnRsPMav9wpYajhiGRo8zVXXFntYrlA8cFtWaBU5FqwN9nITrf75HgVluQDgCOm68qV Agri4BUPxXh681rbwCQjqvUUuSPAabJ7BDUKjhpNt1iTShiCc5rM6w4VinBg7e9fG4pCWX oxG0eBER9BLIeyrLgVpETVp3WBX/aG0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=X2IFqekL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942434; a=rsa-sha256; cv=none; b=Ad4KJmPqdwg/MbMH3wLoFcK+sbilM79JYARfdxjKw637GsoQxDNdxgeS6/tN8cA5ify2zG BMCQWOlCiIYr4bQ5NNav3G4SFUVZJQX4E96PW1wPQLYxmONhn7KMmyhbkvsTF1cgClXJp4 NddYEfn2HN1g4fALAMppv8+E2F//BWY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942434; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/e3etHO5quwZ5TcFINBOwsTOV20jtXx9aa4v5VXGvJw=; b=X2IFqekLT2xWo93SGVSeRMRfAIjfv5zoFP/epABXwzGBVjSH4+I7v+hl7Se2N9kpIj5bp3 OwEFp72THZvzbZv8IxaJwOgZc+Lo0ameC0V67pjlvX0PNtcJZ+ROmc3L1q7zH+hIA2Izp6 1Uiln2fwFSsq3tQMGXkYDUFOMtqrHD8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-519-Sq90ewIjPbyQ2MCsFbryuA-1; Wed, 24 May 2023 11:33:52 -0400 X-MC-Unique: Sq90ewIjPbyQ2MCsFbryuA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 74E1D8032F5; Wed, 24 May 2023 15:33:51 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 74180C164ED; Wed, 24 May 2023 15:33:49 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chuck Lever , Boris Pismenny , John Fastabend Subject: [PATCH net-next 10/12] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 24 May 2023 16:33:09 +0100 Message-Id: <20230524153311.3625329-11-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: D8713C0018 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ciyybu58mb5pyu55gb1faaozaesuybir X-HE-Tag: 1684942434-921960 X-HE-Meta: U2FsdGVkX1/+xT71jV73r2ZXqPmMd0V2BVih1JPnH204f94iPi2uW6EVRSl0CSMEiNuWTZtQcNMSZvMBBAiovVSlHJCXhmn4Iz0g7n5qsWeOJoYDw4PnzksXP5ayQMK1MqxPQSDs7auo3q/gATCMFji08a036i9/Nk5mEJYgFub6JXZUJHq36dds5kjUzrhhsF7BYOmjoszvnSOW6NaDyUw6QhgdyyPMy0Pdsq2JWeqq6VMTWdZnk8E1etWjXUBvjlIxTN3WpN+ZM48Rqp3wJ1SXaEXuxXsJ2qH/v6RKrd0RPCzmyuCfsTdi0Rr/Y8aj3drzuDpiPF7g6zZPRnow4zzqL7iFK8u4g0pDQifRSBJRcHtImEFqLReslZo8qgm2p+HUVDlwV5SG6RgHaLZHi7+Sbi8iizL98Gvsm/oe45iTsZ2QNjBnO4H6M8pTXCLLwlwRrGZGhuCx1ogevmkAGjyULNATCaK1599p0sGKTCtJURpJfdYypxsX0XH+gbCfisbiO7foEnG1iFyYsTdLSQ5vI32DrvJbTlwBXMFjGEI5EXbotIyscIVLEMc+5xloygA9YHuT/Y65QFpqrodoTAmex8El7HPpcPf8J/rS8dDhJXJZSjWiSv59vcm1WrSzbNlyQ79ops0HdV57c/jQ2klhDSH7VxvYaB6ULnjayJMQXYf/gvNhYq9Jv1pFytloj4P0ux9E0vtoLzfCxB6d70LtgYvFc5r7hbwR69dDjpz8N8cmZ+ZpMTp8AQzcRCo2GRYmldITGhg5MVRBwbXHLmwKTSYOQxy9XlA+wg9OT1CxHf8QZwrLobA8nsnU52SIH3t/Q3wCiT3dUaj3D/7u9wHmXqmfxh+oQ1l4dJ85s41UNt5qj+AmQvN3hNGm+PPFq68Y2xlI/M4h8RliAmP4pEUR3CO9abmhuYStrMj8300advHhCVaBPicwwLtURcIKHVLdzwOeb+xpg0LlbBY ymap0GAf qaREcHUxttI9X4NDX+hP9PIvlKJz1CLqvpLU/Ok1k/vBisFY7Ra0iAqxpInYGozrr4ermvbOxWSUTxZoaHXtAwoTlAqXZHhcl0FmeBwZU8VLSd7JByfFuP5FyMBjAouhF+QxE5J/4vx20LF/2s7ZQAheccgc+47l6g6FoXLn16oSaP133x/PnJoX65Bz26HyHvLgF3Hp7mCA7qhiA68VtIJUkH6KPyp3ti7aafwaGuYkmmMb2qCSoaSTqCP8PTEQm50QdZu4P0JRcd8PWR3kxL5TdbHq3PnIapC9vPevn1nNbWi/B70QcsXPwC82+oy8GdCGZTCTrW4tantDAu/fZa93KN3G6Gzk/9UaXnJ2GELi3ns+NiQ/oxHJCeYEmg2Q/JDeFJE1Cp31HPY2++2eSKr5NsACiZ7Wa/3wcfTFuvnzMbaBAkUApbbakIYcyhhV6NPcZWl/EJzcQy2GG9OV+9Zj/5CcTof0QuoBaR4WeMX5fc+jgCq1ImHYYJFxGjDkM0C9ttij1f6n2yuj7qFVJPB9mQnHVQ4g0Kr2a11YkklsuG2uts/QLzL4DKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_sw.c | 164 +++++++++-------------------------------------- 1 file changed, 30 insertions(+), 134 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 0ccef8aa9951..1a5926cc3e84 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -972,7 +972,7 @@ static int rls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -995,15 +995,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1204,157 +1195,62 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + ret = mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int From patchwork Wed May 24 15:33:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C870C77B73 for ; Wed, 24 May 2023 15:34:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F398190001B; Wed, 24 May 2023 11:34:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE8F6900007; Wed, 24 May 2023 11:34:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD93890001B; Wed, 24 May 2023 11:34:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CFAFD900007 for ; Wed, 24 May 2023 11:34:21 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6A2CDC0971 for ; Wed, 24 May 2023 15:34:21 +0000 (UTC) X-FDA: 80825545122.24.8A70830 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 885AD40016 for ; Wed, 24 May 2023 15:34:19 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VoNrEJou; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942459; a=rsa-sha256; cv=none; b=k60e7KBfiundhwt6ay1uPsvkdQcJR/Vi6JIwGoD8FkZGb5M4CCgc35MsK0P+sWFucdxQno trXTEE7U+LI6PD5BVmb6hmgSsbTnSNiqv1aPdtslHmsEW8LJV5CsfJ63pERZ/bsRBb9u5S PDLhUJHphcIKDAxVi1Hsi658mzVZgsk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VoNrEJou; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E1PF2dIIOP/VptjcBGEUgdHYPyQQ9EA3G7wUORmRH+w=; b=k3qEY8ylOxonNlQMOwTuuQ4wAh1I720Mx2PEca08+es8sbvtoTCQYBPRd1qt0V0o/VMhkB DiYrN8jU2p0pwwRZgeV625wWf4nEz/bcM35ByICDB4A6zLQ9nZMjvfyFGBNmUEisxpks8T uYMGfoyxqUdXYuKtHbQfGp8wXXUTTyY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942458; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E1PF2dIIOP/VptjcBGEUgdHYPyQQ9EA3G7wUORmRH+w=; b=VoNrEJouLUbgpU8tQUrwgIhCrVSTsUsVu3wKb/jVX/l3K1P9XDYIwFL/wW/d6IaGxmXJQl +tycxZO5r+OZ39qhn9BlAwFiTGddTr3MOk2klp1Ln3cnOAQHTK1UmYqeV8sHVaaR+6w+/Y TjauGB3VvdDrVGCrtcYybpe/vokg8/A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-qbT634KSMquY-3XAUHekDg-1; Wed, 24 May 2023 11:34:16 -0400 X-MC-Unique: qbT634KSMquY-3XAUHekDg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 18F5B3C0F24B; Wed, 24 May 2023 15:34:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D91C2166B25; Wed, 24 May 2023 15:33:52 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chuck Lever , Boris Pismenny , John Fastabend Subject: [PATCH net-next 11/12] tls/device: Support MSG_SPLICE_PAGES Date: Wed, 24 May 2023 16:33:10 +0100 Message-Id: <20230524153311.3625329-12-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 885AD40016 X-Stat-Signature: oddg6qbij3ygq9hq7xj5i39zexb8iogf X-HE-Tag: 1684942459-519360 X-HE-Meta: U2FsdGVkX1/vSSu6g6SlC/2cdeYL0zgb6dDV95RNhEiSbYYsV+SDGvLzTd6StwLrxcM/9tDiRHAj4Q0aVyl4+VXfXX6tQlxFRf5RWFcuWG3kAZ7+q4V6EfKJiPIbRzCQpxTaLFckVYkAGZytJgx/Ov18gpNXLDSTX2HlRnjOizLz3nmovPi5e+kL3oAHGOrw3n7aYrYiYTQtviZ6X11PX5lm3BmFHpk5N6EQs1QmexI8Oi1k7ZbHQ8I2pWM+dQfa0lHylRSJzGt//0/A+pXPmiFvG2s67Lw4kaPGy/5H7NUQE/Iq+qPJY4abEWcEd8iB521oLLH+VTzVxb6bwaHRKIu7jh/B1YlaQv9/2aH76RjoZQEi+XRUoa5TRDybiRn2yLC51hT4HnQ2iezRphGgJw280/SDhjpgMbWYkOVVTPPOC+qWAu9sY3SiClAM5sU93ibHJcDhdRbMU1M9pU9vDMyEtk/pkCNb+QtHqbD6t2h/rUri4qymWyWeZRUSP4i2S4GeU+nbfA5ypM2BCdrSaLYbIg12JfoPCrT8IXoVb1qwRRHCEHLjx2OGfmatmlbu1WuB2po+/vThRZYSva4PB9ZdYrnoDRdXhjOdcwGXj/HlMTnsvpPEKYZCNSg1PFpk7oTce+Lg6NjBpFK67Q1du3HjVETSMEUOdwno1j2PfVHzot0nP2QXGiA6GdFlb5wyKQYd+IMIY7mZN8Vd4LSPrbsN0J8UW+TUidjMr9QUGamwESOByT5o6qCbuNCJfpYInCnmBdomcFs//mQjw8AqhCHe8kKfTgBAS7GDJDHpgCsEpRTo2NrgGuFAlRuIWcTIFJ2Sj4SL0vA8JmVBkevMzAuso/3CNkNqWkOJ600be7+rYAqWxqXCznSCqkOaq39dX/zsUyMLI49WZMjZQJXB1DEw6cZa88gtMBkwZ320vDU1xzsSU0+PN5cEdnlARfAgXants0tJ9tp5kC9LlnO 4ot4ofeA JpNGq1VNCAKysa6j8lNlsc7VIQvPx6AY5HNfYAuuaDBAvCg+jwJF7klnQeLN8X9Qary34tEOwBX3Xn4bLAI24wyeQCfEasBmKaxlHiV6nDlcfsE2xCY7iVBiUQMzg+2cvCaM28YiD/VdTlNB9yxJlOBWRBMU9D1JbZO3SdusK8XqfqQba16o14fcGkpZMmcz6fUq4znDbVteMAa6A0dS5vAVc0V4bQRyJYv0kx4ibMWXGiat2eXcQLMuaGxKa0gxzErA6/QM92pzw3F82orPnTN2LFNmtIwb+BT3ala+pzb92fGIdughBKXM0qjyt/RLOrhWCOc0ZgYsZydUbKejro+np3zJ9Bv/jBnIX3iI3TOXgkXEJT/BRke9BXkoEMCmr71X60ClTZfT+eobeIj2WezF8dJ6b1R41mlxZQYneBzgKXAUu1bl6gtfr7Xrv3wNU3Bu3+BKAwmw94rQUN4TkhBXTOdqTrp7DRHPIjMMbfX/txz3jpdw6aYEOe6k9FfRKASNNGVDhWEVi75wVfdgtTaeo5E6WqhF84metRat2S3ZstLRDQ1bg310QbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TLS's device sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible and copied the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index daeff54bdbfa..ee07f6e67d52 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -508,7 +508,30 @@ static int tls_push_data(struct sock *sk, tls_append_frag(record, &zc_pfrag, copy); iter_offset.offset += copy; + } else if (copy && (flags & MSG_SPLICE_PAGES)) { + struct page_frag zc_pfrag; + struct page **pages = &zc_pfrag.page; + size_t off; + + rc = iov_iter_extract_pages(iter_offset.msg_iter, &pages, + copy, 1, 0, &off); + if (rc <= 0) { + if (rc == 0) + rc = -EIO; + goto handle_error; + } + copy = rc; + + if (!sendpage_ok(zc_pfrag.page)) { + iov_iter_revert(iter_offset.msg_iter, copy); + goto no_zcopy_this_page; + } + + zc_pfrag.offset = off; + zc_pfrag.size = copy; + tls_append_frag(record, &zc_pfrag, copy); } else if (copy) { +no_zcopy_this_page: copy = min_t(size_t, copy, pfrag->size - pfrag->offset); rc = tls_device_copy_data(page_address(pfrag->page) + @@ -571,6 +594,9 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) union tls_iter_offset iter; int rc; + if (!tls_ctx->zerocopy_sendfile) + msg->msg_flags &= ~MSG_SPLICE_PAGES; + mutex_lock(&tls_ctx->tx_lock); lock_sock(sk); From patchwork Wed May 24 15:33:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13254206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06799C77B7A for ; Wed, 24 May 2023 15:34:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BB7F90001C; Wed, 24 May 2023 11:34:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96A7D900007; Wed, 24 May 2023 11:34:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 859CA90001C; Wed, 24 May 2023 11:34:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 771E0900007 for ; Wed, 24 May 2023 11:34:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0C448C0971 for ; Wed, 24 May 2023 15:34:28 +0000 (UTC) X-FDA: 80825545416.15.3653D7D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 4131614001E for ; Wed, 24 May 2023 15:34:26 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LycWylS2; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684942466; a=rsa-sha256; cv=none; b=Stk4KnI/pIbaOfEHMNOrB7JeIbc/YxbsGmrnVDNS+bBtMkPAwx4uk272TPG9ylMuAofjL7 jDDF3aZ+tSXAN9sg2i+a19k2jPqqK/i3NDKRrKZG6AFCStMNfPu9Xr7q8PY1iQerWRjt58 lFUk+GAVqSQsquu9FOPfLXFPnRw8UUI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LycWylS2; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684942466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k9cQP4HmvFAjJY7J8On2WvrNLYKoX68FiRQGhNJuMdw=; b=aGtp9q4hpjnzvoPxvTFsQk0g7a5WTSiHdtCWshcqi1KVIlAtTDxz0gdkgOxPVt+a+n7+YN G9r6kM3+ETx7PfMESCb2dC9s3ZHhmG2nceqMI7ds3Gcuz8EAfdCkBaIUEwjXsv1e9WunPK RnkbpRSCe3iXIGft1MT4392fc+Z8WVc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684942465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k9cQP4HmvFAjJY7J8On2WvrNLYKoX68FiRQGhNJuMdw=; b=LycWylS2QOr2xZFvgZZLsHWuLXlvfYp6KvOog/lqRPBf75VT/kELZQIMy9woEGA/VczUPj LcKWGS8vIvZ77iv+eYjaeZG1/HBZz2eVIFGX7z4c5zQFn86krtNZEGmhzsigS4lTVf2X4B 2NL9S6MpG6UXiNKiBG4c1pBxBIeQtdw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-121-TJrS7NBWNQOPag9a_0ZUCA-1; Wed, 24 May 2023 11:34:18 -0400 X-MC-Unique: TJrS7NBWNQOPag9a_0ZUCA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 712A928078D3; Wed, 24 May 2023 15:34:17 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE8F140CFD45; Wed, 24 May 2023 15:34:14 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chuck Lever , Boris Pismenny , John Fastabend Subject: [PATCH net-next 12/12] tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 24 May 2023 16:33:11 +0100 Message-Id: <20230524153311.3625329-13-dhowells@redhat.com> In-Reply-To: <20230524153311.3625329-1-dhowells@redhat.com> References: <20230524153311.3625329-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4131614001E X-Stat-Signature: jrdmby5bgtekqq9x7xz1x9gjwoyrzyao X-Rspam-User: X-HE-Tag: 1684942466-571660 X-HE-Meta: U2FsdGVkX1/mKY/oXG0gQ0/ophSAQWGDipidsUlI8Suvseqz66EvJ4uF3HLzMd7QWCjNcRCocHO6WjU3LHr6hGVetah2fb+iYv86ZrFbaswnH8rJCOEk00mGSdsjLLAPTma1qSglIV/+QbvnUZL5x81tZdqQo6kDosB1sW6vr45FGh+SoPYcc0vuXEkznMkFmGw9Kh2DCo9ZjXucR3kKqMojf4yVged/zOaPrSaGvR/+TGOgG/cXfiagy0t5j8TNG5cv2vQymv/XSVkzS0LkgCr/Geww6IOWPbsaUzQIPEeVts9HrLKeQkr6QUsLUye4rFdV18trSiVd58+/efYsCM6T0p2fNyGg4OjhOdo8wu9mlNY208DersIv7MrGLYKERlv56Xy+Fi6Si1SZFhggyObQ/xj8m7ujgh4mWF06vSIM/SpcBPD6M1m9YiN/QJNWEngXSl6vKHPfVkod78HnclNMJ/Oqj/ZGiL2Oj3vAmS45j4GF7j+Uea2dFZJFDIBr9E6BTtgefkXZj2n4g5BfpkFEorqaV5vo1yZpNUuSiKUnQL5BT6IIe85l4emoiWGa52Vxx0plMhwiOGDrc8Vr04iL4nXWy/lWK53cj8mU/ptI1ldfF310LgMhVSWZ9O0RmnqdlwQ60PJWYk72bWGYagZ/3Lv3Q+bQb+9rHhruOhkOr4hYKWOp9WUnBonSfd5s8pFkB2Imr3xxqSxd1DjTT4HJDEqRXOYFopyFZ+H8f8eZwSHuY3l4bWSsei4icvgcGkyPkQxtRxI+poRz3Fi5TqxBagJ3HC0464pmdve/uf6Vy8RGyscY98Z7kezWI8wLYwb7Df+WtbhKkd9t2F4HWbuUY0Q4NS9FNRPEg8rp4bGH2U3SLUAMtI3+BdIkrSOEXgrOGxRGZAvVDQNxexIcMWmD+F2STQ56oZZdrpn+CuKqZ+aBATSJfFOkA7hKoh0p+TesKLPZ0x81vSTTCRr LGdugzh3 JnyZwa/cSdXgQ4HiChPYHiiz1znKDjxVCOkYafSWWUY2O23LiI+kai1rTMhJrUCzQ+UmUqnjqUQ+mXqXpi2Tzqg2v8us99FUKFLKzx51qdTJxpQTs+fPCVPiQYiUqRMjPkReRl7TEpUjCzCtIekXQpY2uX9h3L3+/UJB4r8Emr9SDONUhJVj2B5DVcc7oxGN0yP9h78mjivuMVepsKOPrvqSRYi0Fhfg6eF04a3E0JfKF/juJjvCGRQ11wm/y/a0Lc8dXGtQ3fhpXH+EwEg/YS9wMtlam/oFyO32RnJ0EuZlAIDVZTT80Cvw/zs4xLhkJl4AUIXqn22ABTDkehE5Wtga4TrtAoDrcWm8DCSNYgJvF8Sv0r4PNVlJGeqwke33A/PwMvLIVYU45cE/GRIJNseUbgzXFD6vSzKnXh7ICCcOzOPDR1+notOmjp7tIDnOQwHXN2uc8pkFsx4jVLWQMJqDJwAeSwIbgARbhpBmjeq3PuQs6T9JZf6sRPIt1O4B4zvqVKy14t0xpmV9I3Xh1yUmQeoa8LxfniJtvbf23NAEdn6Cnkt0ODr8L5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_device_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. With that, the tls_iter_offset union is no longer necessary and can be replaced with an iov_iter pointer and the zc_page argument to tls_push_data() can also be removed. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 81 ++++++++++---------------------------------- 1 file changed, 18 insertions(+), 63 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index ee07f6e67d52..f2c895009314 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -422,16 +422,10 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i) return 0; } -union tls_iter_offset { - struct iov_iter *msg_iter; - int offset; -}; - static int tls_push_data(struct sock *sk, - union tls_iter_offset iter_offset, + struct iov_iter *iter, size_t size, int flags, - unsigned char record_type, - struct page *zc_page) + unsigned char record_type) { struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_prot_info *prot = &tls_ctx->prot_info; @@ -499,21 +493,12 @@ static int tls_push_data(struct sock *sk, record = ctx->open_record; copy = min_t(size_t, size, max_open_record_len - record->len); - if (copy && zc_page) { - struct page_frag zc_pfrag; - - zc_pfrag.page = zc_page; - zc_pfrag.offset = iter_offset.offset; - zc_pfrag.size = copy; - tls_append_frag(record, &zc_pfrag, copy); - - iter_offset.offset += copy; - } else if (copy && (flags & MSG_SPLICE_PAGES)) { + if (copy && (flags & MSG_SPLICE_PAGES)) { struct page_frag zc_pfrag; struct page **pages = &zc_pfrag.page; size_t off; - rc = iov_iter_extract_pages(iter_offset.msg_iter, &pages, + rc = iov_iter_extract_pages(iter, &pages, copy, 1, 0, &off); if (rc <= 0) { if (rc == 0) @@ -523,7 +508,7 @@ static int tls_push_data(struct sock *sk, copy = rc; if (!sendpage_ok(zc_pfrag.page)) { - iov_iter_revert(iter_offset.msg_iter, copy); + iov_iter_revert(iter, copy); goto no_zcopy_this_page; } @@ -536,7 +521,7 @@ static int tls_push_data(struct sock *sk, rc = tls_device_copy_data(page_address(pfrag->page) + pfrag->offset, copy, - iter_offset.msg_iter); + iter); if (rc) goto handle_error; tls_append_frag(record, pfrag, copy); @@ -591,7 +576,6 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { unsigned char record_type = TLS_RECORD_TYPE_DATA; struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter; int rc; if (!tls_ctx->zerocopy_sendfile) @@ -606,8 +590,7 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) goto out; } - iter.msg_iter = &msg->msg_iter; - rc = tls_push_data(sk, iter, size, msg->msg_flags, record_type, NULL); + rc = tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags, record_type); out: release_sock(sk); @@ -618,44 +601,18 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter_offset; - struct iov_iter msg_iter; - char *kaddr; - struct kvec iov; - int rc; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (flags & MSG_OOB) { - rc = -EOPNOTSUPP; - goto out; - } - - if (tls_ctx->zerocopy_sendfile) { - iter_offset.offset = offset; - rc = tls_push_data(sk, iter_offset, size, - flags, TLS_RECORD_TYPE_DATA, page); - goto out; - } - - kaddr = kmap(page); - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg_iter, ITER_SOURCE, &iov, 1, size); - iter_offset.msg_iter = &msg_iter; - rc = tls_push_data(sk, iter_offset, size, flags, TLS_RECORD_TYPE_DATA, - NULL); - kunmap(page); + if (flags & MSG_OOB) + return -EOPNOTSUPP; -out: - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return rc; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_device_sendmsg(sk, &msg, size); } struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, @@ -720,12 +677,10 @@ EXPORT_SYMBOL(tls_get_record); static int tls_device_push_pending_record(struct sock *sk, int flags) { - union tls_iter_offset iter; - struct iov_iter msg_iter; + struct iov_iter iter; - iov_iter_kvec(&msg_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter = &msg_iter; - return tls_push_data(sk, iter, 0, flags, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0); + return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA); } void tls_device_write_space(struct sock *sk, struct tls_context *ctx)