From patchwork Mon Nov 5 08:58:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10667649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FB3D14E2 for ; Mon, 5 Nov 2018 08:58:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D52729292 for ; Mon, 5 Nov 2018 08:58:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EC0729299; Mon, 5 Nov 2018 08:58:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,SUBJ_OBFU_PUNCT_FEW,SUBJ_OBFU_PUNCT_MANY autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 557FA29292 for ; Mon, 5 Nov 2018 08:58:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51A786B0007; Mon, 5 Nov 2018 03:58:26 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4CA866B0008; Mon, 5 Nov 2018 03:58:26 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E0616B000A; Mon, 5 Nov 2018 03:58:26 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 018626B0007 for ; Mon, 5 Nov 2018 03:58:26 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id 129-v6so8825700pfx.11 for ; Mon, 05 Nov 2018 00:58:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:mime-version:content-transfer-encoding; bh=iXsn0IQmvUd+qr/13cqqT6fG7fWOtXYZpCujpYKzSjs=; b=pPPIG7DsuRZUvAMmOzRG0qdMjplNOk9nu2c80QvejH/nRYuReaq0izJkFW0hPx8ONC I1aFhTIfjg3H0IjAKCjpHossYIYJSr8puVWiQdwjkjZdUz4u/bTcR9wTB0Ezzlg/dZL9 /C9OLc7/vSIsLockJSVbLjbx3+t/bNmImx4tFbT3BhibOmJbcuqWAYVRvj4Uhk+CMycf G8nCjREf1UNHld/uAH/pzcrL3fiTU8II//cPkVs4jO1vdDp8HcBigCuj6AtTw0YSKNVR hLlSJwSTIHrOAiCtSTC5fu9E/vpLsmXznCyicAZBiqHHGGBjGeCgfIVYYISQVn8+M538 seHw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gK0Xr0h9NjUCGxDJEQDMKctgVYMl823rPbxWv/BA1CWPVM0lBvL 2cz9Iky9092H+38YN+6gS8DliAymwpY4awMDVVFI6v6Hen2O4Kam4/WWKJJVCIWec4Oony1A6ll /BG6tfCNqQK+i95U2vZAlzLtAlAWKQI8biMvFu+JYnUBftc8E5rjBRwIjv+Rodtc9PA== X-Received: by 2002:a17:902:b598:: with SMTP id a24-v6mr12609112pls.231.1541408305660; Mon, 05 Nov 2018 00:58:25 -0800 (PST) X-Google-Smtp-Source: AJdET5dZicfnCDfADbZIBPx6RXy/uWscNo5u2rlDhGbG6kkkht9fdSNywO9vJOqHD84CXapI/X2l X-Received: by 2002:a17:902:b598:: with SMTP id a24-v6mr12609083pls.231.1541408304884; Mon, 05 Nov 2018 00:58:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541408304; cv=none; d=google.com; s=arc-20160816; b=yWoCpyZ4J2C6lobjGWtojjbKRN52aB4cdwEYap/+mXITB6aAvnDF2Edzb2dEKLTCDH 4zecW5Du9v4eZuIR4mwTY7I+jG0+EKeSOW3phQlJ5zzWIoWzZsF1+kjK01Pe6d7FB5Po xHiE20vn635qHpyYdV2if/XITxEHe4dU0T4j2Ou1JyZqvl265TmWhi/rrqHfuOkPGPZW oRAzXhvTc7yGQzY1m+ELrKHdYVItPLf04MJFip4Oz5tEpEtQDXfJWzCjKNEjJJu5BrdK +ge+5ZYHUOuhVYDvgvaOse89GAvpbXUUMfXI8ddQEtZALKE2JTj8Hdgf9Ogjnyg06TZl dsxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=iXsn0IQmvUd+qr/13cqqT6fG7fWOtXYZpCujpYKzSjs=; b=qCtCLQPmuiG5dmKDYoq0X4vQjDmUdhbzeAFC6hsRq7QCtNOuMM6CuojS2ifJgeV07X 7OjOixdGzRRd442SvNP1R9gg1NGMyEacHVIWDBKiZefkkgyJq1vbrurFpc+OfXbhlspY bkZWgVcZcW1l+P+OQPN1hH3vaQ/Z5tFH/J8HWybWnxz/lMd25YaEN1Ps0TZEm5pnRm3j LOyYjcMqQO0l8xoumF2Zw3oxmUzz/Gbo2GSQhAsQGUihAbwE9u1ziXPlprwEk631LJW3 0OPAvjf3uVjEu/wahxt9FkKtj7f6UQzO05rYKp3RUlMceU5anWVsFn1vGwQDagIx/Q2b HtFg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id 16si17376912pgh.58.2018.11.05.00.58.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 00:58:24 -0800 (PST) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 00:58:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,467,1534834800"; d="scan'208";a="103471773" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga004.fm.intel.com with ESMTP; 05 Nov 2018 00:58:21 -0800 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Andrew Morton , =?utf-8?q?Pawe=C5=82_Staszews?= =?utf-8?q?ki?= , Jesper Dangaard Brouer , Eric Dumazet , Tariq Toukan , Ilias Apalodimas , Yoel Caspersen , Mel Gorman , Saeed Mahameed , Michal Hocko , Vlastimil Babka , Dave Hansen Subject: [PATCH 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free() Date: Mon, 5 Nov 2018 16:58:19 +0800 Message-Id: <20181105085820.6341-1-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP page_frag_free() calls __free_pages_ok() to free the page back to Buddy. This is OK for high order page, but for order-0 pages, it misses the optimization opportunity of using Per-Cpu-Pages and can cause zone lock contention when called frequently. Paweł Staszewski recently shared his result of 'how Linux kernel handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer found the lock contention comes from page allocator: mlx5e_poll_tx_cq | --16.34%--napi_consume_skb | |--12.65%--__free_pages_ok | | | --11.86%--free_one_page | | | |--10.10%--queued_spin_lock_slowpath | | | --0.65%--_raw_spin_lock | |--1.55%--page_frag_free | --1.44%--skb_release_data Jesper explained how it happened: mlx5 driver RX-page recycle mechanism is not effective in this workload and pages have to go through the page allocator. The lock contention happens during mlx5 DMA TX completion cycle. And the page allocator cannot keep up at these speeds.[2] I thought that __free_pages_ok() are mostly freeing high order pages and thought this is an lock contention for high order pages but Jesper explained in detail that __free_pages_ok() here are actually freeing order-0 pages because mlx5 is using order-0 pages to satisfy its page pool allocation request.[3] The free path as pointed out by Jesper is: skb_free_head() -> skb_free_frag() -> skb_free_frag() -> page_frag_free() And the pages being freed on this path are order-0 pages. Fix this by doing similar things as in __page_frag_cache_drain() - send the being freed page to PCP if it's an order-0 page, or directly to Buddy if it is a high order page. With this change, Paweł hasn't noticed lock contention yet in his workload and Jesper has noticed a 7% performance improvement using a micro benchmark and lock contention is gone. [1]: https://www.spinics.net/lists/netdev/msg531362.html [2]: https://www.spinics.net/lists/netdev/msg531421.html [3]: https://www.spinics.net/lists/netdev/msg531556.html Reported-by: Paweł Staszewski Analysed-by: Jesper Dangaard Brouer Signed-off-by: Aaron Lu Acked-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Jesper Dangaard Brouer Acked-by: Ilias Apalodimas Tested-by: Ilias Apalodimas Acked-by: Alexander Duyck --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae31839874b8..91a9a6af41a2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4555,8 +4555,14 @@ void page_frag_free(void *addr) { struct page *page = virt_to_head_page(addr); - if (unlikely(put_page_testzero(page))) - __free_pages_ok(page, compound_order(page)); + if (unlikely(put_page_testzero(page))) { + unsigned int order = compound_order(page); + + if (order == 0) + free_unref_page(page); + else + __free_pages_ok(page, order); + } } EXPORT_SYMBOL(page_frag_free); From patchwork Mon Nov 5 08:58:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10667651 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F00D514E2 for ; Mon, 5 Nov 2018 08:58:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E085229292 for ; Mon, 5 Nov 2018 08:58:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D48C229299; Mon, 5 Nov 2018 08:58:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C39429292 for ; Mon, 5 Nov 2018 08:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 658356B0008; Mon, 5 Nov 2018 03:58:28 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5B59A6B000C; Mon, 5 Nov 2018 03:58:28 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 454846B000D; Mon, 5 Nov 2018 03:58:28 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 059A46B000C for ; Mon, 5 Nov 2018 03:58:28 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id b3-v6so9413364plr.17 for ; Mon, 05 Nov 2018 00:58:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9fMGtFssUieAMeKtAo7kTPgzL4OfZ+vZIMWNcPl8iDk=; b=DYNmEBAFk9ZizrrnDlNd6fGDAeBYcA4Y3EqMEi1LXLxUCTvH4yh1ZAn21GFI4vskuB 8knqbHSTeFKq8i44Pl9k063Kzz068lDmwDpj+3N8hDBImmf9i4BZA8DlgCpVMqn4SKEx 2/vQfeHa5y9doE57Kt2APwTVwsdzV2ZEH+NTlVBO5mr1Quj0cyWjtSih/3Z60cHsWH/E yWaCGSCF1rqVBXInNvIv1RFWVFnFN1rYB8/MgmaeVZxtfMgWW/uhApGkKXEhPNgm272z iUxFYU9ymVrZe0tT7r2fsCYprGqX/OpmTTY2r8SWe3adf0KEoZfZtVcD7kcQiypQw2qs dCPA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gIx7Wt9fbfZhaZqKjUvLhn999qJkDa6YTFP8yBKP2fp/4jRp9i5 3bjccrK072dXueMaYzbcRBK84/xWGU18j2L59wXcpWvWZBwu1zQvMOElmqI30MSsFsV7GnaQJdd s+oENUV8aLtvVA2ijBhCyYHBnSdrJ4Oj5cs67JnzcB05XM75qQXOkHCGOQImarqgniw== X-Received: by 2002:a63:cc0b:: with SMTP id x11-v6mr19126526pgf.33.1541408307666; Mon, 05 Nov 2018 00:58:27 -0800 (PST) X-Google-Smtp-Source: AJdET5d+/I4H8P+UDjT93+tknVccG9JkF1Sctwu5mIDGEq4cXTUTL2xZC/6diAZWI2CjJ3/g1cKa X-Received: by 2002:a63:cc0b:: with SMTP id x11-v6mr19126500pgf.33.1541408306962; Mon, 05 Nov 2018 00:58:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541408306; cv=none; d=google.com; s=arc-20160816; b=OfJ0k5F8qqreFezuau23BQorbXeUAmsHP7joF1xkVlSpck+Ab4nx+szl4+EiBeI7EE YuuCwlps1cZXvgvOzJP8nsSg/o++fY7We7wfW0xj0pKGjpJfvwlhS+PjDVbyEkQxfaKB /0VrM+X0C5/TPE8E7KJWTAeqk4uAu5uQwZCysxURI2h1EZ8zVskG5/4LniG7I/hBQa4H IOSVCp5qwV7QsmR221exNqbPueYZpB0jNsqMKc4iSsCIxNALf+7jdGvqGJPwQYERC3PU Fj1g9AH8jcXpVHMHimgAomIIGugCHz7nU0jvfQsqblTOhk2Yn/JPa8mHLKLWqyhggfAC 0pxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=9fMGtFssUieAMeKtAo7kTPgzL4OfZ+vZIMWNcPl8iDk=; b=sh1XZCIJZeqeXkV7Mhg2CIrW6+0KL8ADZkB1pErrY/8bWcsKocDBELqTGE32T+TBNK 67i6DivrmTSXE17+jvC/VOFOdcN37Z4RXMnvvzlB7DxxDeh14+Cf/oNAyEBy2rjl+ekh tbxWS0ONbnDl8rMC70/j5kbe1HmLuzV6EzlX7dvVPxa734CXWVuuHzKmEZO9W0272nHF GPt9Jefipel4/izoKm2thIr5gWvP7If6zlE+DnNZvjNiw182AgGdG6Ge86LlW1+QDszY 5wBMU/CBWFdGZ6q2qoPDmHR6FES6T22suJTMYDkM7IuXEl8Y3K1WPBC8C65nQhn9nzjr RbvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id 16si17376912pgh.58.2018.11.05.00.58.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 00:58:26 -0800 (PST) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 00:58:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,467,1534834800"; d="scan'208";a="103471781" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga004.fm.intel.com with ESMTP; 05 Nov 2018 00:58:24 -0800 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Andrew Morton , =?utf-8?q?Pawe=C5=82_Staszews?= =?utf-8?q?ki?= , Jesper Dangaard Brouer , Eric Dumazet , Tariq Toukan , Ilias Apalodimas , Yoel Caspersen , Mel Gorman , Saeed Mahameed , Michal Hocko , Vlastimil Babka , Dave Hansen Subject: [PATCH 2/2] mm/page_alloc: use a single function to free page Date: Mon, 5 Nov 2018 16:58:20 +0800 Message-Id: <20181105085820.6341-2-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181105085820.6341-1-aaron.lu@intel.com> References: <20181105085820.6341-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have multiple places of freeing a page, most of them doing similar things and a common function can be used to reduce code duplicate. It also avoids bug fixed in one function and left in another. Signed-off-by: Aaron Lu --- mm/page_alloc.c | 37 +++++++++++++++++-------------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91a9a6af41a2..2b330296e92a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4425,9 +4425,17 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) } EXPORT_SYMBOL(get_zeroed_page); -void __free_pages(struct page *page, unsigned int order) +/* + * Free a page by reducing its ref count by @nr. + * If its refcount reaches 0, then according to its order: + * order0: send to PCP; + * high order: directly send to Buddy. + */ +static inline void free_the_page(struct page *page, unsigned int order, int nr) { - if (put_page_testzero(page)) { + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, nr)) { if (order == 0) free_unref_page(page); else @@ -4435,6 +4443,11 @@ void __free_pages(struct page *page, unsigned int order) } } +void __free_pages(struct page *page, unsigned int order) +{ + free_the_page(page, order, 1); +} + EXPORT_SYMBOL(__free_pages); void free_pages(unsigned long addr, unsigned int order) @@ -4481,16 +4494,7 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) { - unsigned int order = compound_order(page); - - if (order == 0) - free_unref_page(page); - else - __free_pages_ok(page, order); - } + free_the_page(page, compound_order(page), count); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -4555,14 +4559,7 @@ void page_frag_free(void *addr) { struct page *page = virt_to_head_page(addr); - if (unlikely(put_page_testzero(page))) { - unsigned int order = compound_order(page); - - if (order == 0) - free_unref_page(page); - else - __free_pages_ok(page, order); - } + free_the_page(page, compound_order(page), 1); } EXPORT_SYMBOL(page_frag_free);