From patchwork Wed Jan 11 04:21:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13095996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1CADC5479D for ; Wed, 11 Jan 2023 04:22:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 108CF90000C; Tue, 10 Jan 2023 23:22:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06C17900003; Tue, 10 Jan 2023 23:22:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB42F90000C; Tue, 10 Jan 2023 23:22:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C1971900003 for ; Tue, 10 Jan 2023 23:22:28 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9F027C03DB for ; Wed, 11 Jan 2023 04:22:28 +0000 (UTC) X-FDA: 80341221576.12.AC095F8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 28493100002 for ; Wed, 11 Jan 2023 04:22:26 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dXjKNEzz; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673410947; a=rsa-sha256; cv=none; b=iNLib3MBISLDNyJNSEh3TmaYSobUKiPe3zuPTZ3jr0rrHIg5C3PRBZnHIYq/hiW887Xy3t FcV0jLC2633jQujBn4dFlaFtziWI+7yBJ4qhhq91agje92I0fOchoW8r7PAa9wYbUfdTwY Q7tIDHily4QY96b/yhhigBiEx+mZmhk= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dXjKNEzz; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673410947; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TVi+2uh0XxrIfC6B52OUoVXQFDNYl2L/xl03Bg8sz9o=; b=kQBXoOy/VJce11ayCYynEXE0ms80f9ZX1t1IC9ryyzuoLcuw/XPqy1rvXoAJLF//8pxM8p fMWK+0/6fM64zkCNrrPnYp+at1MfiC9Za07V8PDbFlgIheA3zWMRZ4RwUJnYeWJJGnDvju LrsVkpkTHVkOWsfxNLFEpo7QxAtCf+4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=TVi+2uh0XxrIfC6B52OUoVXQFDNYl2L/xl03Bg8sz9o=; b=dXjKNEzzK6eRfENac3kSUQEKT+ WtZ+nHNgZEzK600Le9mPFPHfQl3YeaPgti4hWU8jjq5SIjI1TKQQJaxLXVtyGm2APLp0Xwf/PpA+F I6FE2/oaBUxjZ3hih/l881XyT15lbLFAQzu/EjZm4TBNYb3Ab03jrn9UsxmIvmVTewjMM+81Jl+wc T3q9pmK8rOCat0MvWWb3u09VNK1hQBYw15vv13HWIMQX4KQYO6i1P8Adx63Yi637EOdhVkEaVyxIK Lh4fVJW0FrjdmmMNHe6/Q2nCS7KcyH26jHESOYdSZUMmRAAnAQkTsjugzgsspQAIyF2PYSR6rh+tO wZ13V62w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFScy-003nxu-O5; Wed, 11 Jan 2023 04:22:16 +0000 From: "Matthew Wilcox (Oracle)" To: Jesper Dangaard Brouer , Ilias Apalodimas Cc: "Matthew Wilcox (Oracle)" , netdev@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Jesper Dangaard Brouer , Jesse Brandeburg Subject: [PATCH v3 07/26] page_pool: Convert __page_pool_put_page() to __page_pool_put_netmem() Date: Wed, 11 Jan 2023 04:21:55 +0000 Message-Id: <20230111042214.907030-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111042214.907030-1-willy@infradead.org> References: <20230111042214.907030-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 28493100002 X-Rspamd-Server: rspam01 X-Stat-Signature: pdez6rm88gjkthq8bj5uprkbikjuwqa8 X-HE-Tag: 1673410946-186163 X-HE-Meta: U2FsdGVkX1/pLesOg2Bt5X+pTMfWWkBW+j+rjkL/ybq1xj7ev+9l2wShh0BWoybRL2rMEBBryd1b1bc6vjC6HWJUIdZmDm4zY2xFviuw1IkHrHLCSKuwFAUHjam2LVnJjy6G4duA6jwT3nrl9/iZQcDQCR0rbKjcTBtLBsRYrrwO2MK2gX+RL1laxj+tpGLUFG6O7dWg05aRDUiHq4kWyZ+vJFVQA/e+foff0r0wgSm9LKSDLVvJZEiBNmWpR91M6rrMEQTV5mai8HRGR4gdSn/LerjfcTevWnicMl39HrgKZvtb1HKvkAcYGfYjY6xtr/aB9nLG/ex1BKuO5CFP2YqAeNDv6mfMfuxQKIZCCllg3gsmjNHxtcNrxIQ4J48hzmjPYHOw2y8Knu4MYAthP+l/KObNshQVfJevf6WLr0WIk7Vw84F4LxCnqwNyHTpC+3v+zu8awCqhM66BJg+C7iSkan298vRdNXF2bGMzKtgP7Li4ZbdFwhN+IOYwSVfTYyCPBLzFveOFqOkE0McbfNlhqCudlSEehsY9IQmg2g7gmlFoVm6RlZlJa8LV6qQ3QlTg5HW9K6hoviLPscIY7X8oU45IH3Sic2VGh8/51kJlr2UnclsTs2PDvJgnpRNEWsFV9jdrGn3qNSE8G2VDZB+hcurXGzh/969VNpWzRoYiyeJ3jHxZrWyclTjpvnzAkAWGySrj+UWZQrn6NDQbNmIBjxyrFq45HHM2zj/9B7FTgbgbpwUsPC96tdTI0G6B8OuMOrOkIM01FqEdRxRtM2Vrx2stmqNmGcYRCIsD/z9uQ+Q9+LbqUCYfkjDszHzzEsZSdlNKzGSq/c0d4sov3aSVbpkQV/GawnRoqD3jFImOvtBru31Tt6Mnv2HNr0DFOU0T/GrRdrOnLuPLPOPwZ1O8WWrSUjDl6yw7DwYUWWxqqLsrxsPw5Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes the call to compound_head() hidden in put_page() which saves 169 bytes of kernel text as __page_pool_put_page() is inlined twice. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas Reviewed-by: Jesse Brandeburg --- net/core/page_pool.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index b606952773a6..8f3f7cc5a2d5 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -558,8 +558,8 @@ static bool page_pool_recycle_in_cache(struct page *page, * If the page refcnt != 1, then the page will be returned to memory * subsystem. */ -static __always_inline struct page * -__page_pool_put_page(struct page_pool *pool, struct page *page, +static __always_inline struct netmem * +__page_pool_put_netmem(struct page_pool *pool, struct netmem *nmem, unsigned int dma_sync_size, bool allow_direct) { /* This allocator is optimized for the XDP mode that uses @@ -571,19 +571,20 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, * page is NOT reusable when allocated when system is under * some pressure. (page_is_pfmemalloc) */ - if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { - /* Read barrier done in page_ref_count / READ_ONCE */ + if (likely(netmem_ref_count(nmem) == 1 && + !netmem_is_pfmemalloc(nmem))) { + /* Read barrier done in netmem_ref_count / READ_ONCE */ if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) - page_pool_dma_sync_for_device(pool, page, + page_pool_dma_sync_for_device(pool, netmem_page(nmem), dma_sync_size); if (allow_direct && in_serving_softirq() && - page_pool_recycle_in_cache(page, pool)) + page_pool_recycle_in_cache(netmem_page(nmem), pool)) return NULL; /* Page found as candidate for recycling */ - return page; + return nmem; } /* Fallback/non-XDP mode: API user have elevated refcnt. * @@ -599,13 +600,21 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, * will be invoking put_page. */ recycle_stat_inc(pool, released_refcnt); - /* Do not replace this with page_pool_return_page() */ - page_pool_release_page(pool, page); - put_page(page); + /* Do not replace this with page_pool_return_netmem() */ + page_pool_release_netmem(pool, nmem); + netmem_put(nmem); return NULL; } +static __always_inline struct page * +__page_pool_put_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct) +{ + return netmem_page(__page_pool_put_netmem(pool, page_netmem(page), + dma_sync_size, allow_direct)); +} + void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, unsigned int dma_sync_size, bool allow_direct) {