From patchwork Thu Mar 25 11:42:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12163827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64D6FC433E0 for ; Thu, 25 Mar 2021 11:42:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B5696619D5 for ; Thu, 25 Mar 2021 11:42:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5696619D5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1DF346B0070; Thu, 25 Mar 2021 07:42:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 18F146B0071; Thu, 25 Mar 2021 07:42:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02ED36B0072; Thu, 25 Mar 2021 07:42:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id DB91E6B0070 for ; Thu, 25 Mar 2021 07:42:41 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A37618417320 for ; Thu, 25 Mar 2021 11:42:41 +0000 (UTC) X-FDA: 77958209322.05.B27B5A0 Received: from outbound-smtp07.blacknight.com (outbound-smtp07.blacknight.com [46.22.139.12]) by imf05.hostedemail.com (Postfix) with ESMTP id 89D74E000243 for ; Thu, 25 Mar 2021 11:42:40 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp07.blacknight.com (Postfix) with ESMTPS id 4EF4C1C35A8 for ; Thu, 25 Mar 2021 11:42:39 +0000 (GMT) Received: (qmail 14641 invoked from network); 25 Mar 2021 11:42:39 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 25 Mar 2021 11:42:39 -0000 From: Mel Gorman To: Andrew Morton Cc: Chuck Lever , Jesper Dangaard Brouer , Christoph Hellwig , Alexander Duyck , Vlastimil Babka , Matthew Wilcox , Ilias Apalodimas , LKML , Linux-Net , Linux-MM , Linux-NFS , Mel Gorman Subject: [PATCH 0/9 v6] Introduce a bulk order-0 page allocator with two in-tree users Date: Thu, 25 Mar 2021 11:42:19 +0000 Message-Id: <20210325114228.27719-1-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 89D74E000243 X-Stat-Signature: sssq5wbmcahjaji6eet9966coacwef1o Received-SPF: none (techsingularity.net>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=outbound-smtp07.blacknight.com; client-ip=46.22.139.12 X-HE-DKIM-Result: none/none X-HE-Tag: 1616672560-656446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series is based on top of Matthew Wilcox's series "Rationalise __alloc_pages wrapper" and does not apply to 5.14-rc4. If Andrew's tree is not the testing baseline then the following git tree will work. git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git mm-bulk-rebase-v6r7 Changelog since v5 o Add micro-optimisations from Jesper o Add array-based versions of the sunrpc and page_pool users o Allocate 1 page if local zone watermarks are not met o Fix statistics o prep_new_pages as they are allocated. Batching prep_new_pages with IRQs enabled limited how the API could be used (e.g. list must be empty) and added too much complexity. Changelog since v4 o Drop users of the API o Remove free_pages_bulk interface, no users o Add array interface o Allocate single page if watermark checks on local zones fail Changelog since v3 o Rebase on top of Matthew's series consolidating the alloc_pages API o Rename alloced to allocated o Split out preparation patch for prepare_alloc_pages o Defensive check for bulk allocation or <= 0 pages o Call single page allocation path only if no pages were allocated o Minor cosmetic cleanups o Reorder patch dependencies by subsystem. As this is a cross-subsystem series, the mm patches have to be merged before the sunrpc and net users. Changelog since v2 o Prep new pages with IRQs enabled o Minor documentation update Changelog since v1 o Parenthesise binary and boolean comparisons o Add reviewed-bys o Rebase to 5.12-rc2 This series introduces a bulk order-0 page allocator with sunrpc and the network page pool being the first users. The implementation is not efficient as semantics needed to be ironed out first. If no other semantic changes are needed, it can be made more efficient. Despite that, this is a performance-related for users that require multiple pages for an operation without multiple round-trips to the page allocator. Quoting the last patch for the high-speed networking use-case Kernel XDP stats CPU pps Delta Baseline XDP-RX CPU total 3,771,046 n/a List XDP-RX CPU total 3,940,242 +4.49% Array XDP-RX CPU total 4,249,224 +12.68% From the SUNRPC traces of svc_alloc_arg() Single page: 25.007 us per call over 532,571 calls Bulk list: 6.258 us per call over 517,034 calls Bulk array: 4.590 us per call over 517,442 calls Both potential users in this series are corner cases (NFS and high-speed networks) so it is unlikely that most users will see any benefit in the short term. Other potential other users are batch allocations for page cache readahead, fault around and SLUB allocations when high-order pages are unavailable. It's unknown how much benefit would be seen by converting multiple page allocation calls to a single batch or what difference it may make to headline performance. Light testing of my own running dbench over NFS passed. Chuck and Jesper conducted their own tests and details are included in the changelogs. Patch 1 renames a variable name that is particularly unpopular Patch 2 adds a bulk page allocator Patch 3 adds an array-based version of the bulk allocator Patches 4-5 adds micro-optimisations to the implementation Patches 6-7 SUNRPC user Patches 8-9 Network page_pool user include/linux/gfp.h | 18 +++++ include/net/page_pool.h | 2 +- mm/page_alloc.c | 157 ++++++++++++++++++++++++++++++++++++++-- net/core/page_pool.c | 111 ++++++++++++++++++---------- net/sunrpc/svc_xprt.c | 38 +++++----- 5 files changed, 263 insertions(+), 63 deletions(-)