From patchwork Sun May 9 19:38:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12246523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CF44C433ED for ; Sun, 9 May 2021 19:38:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4E0A613AB for ; Sun, 9 May 2021 19:38:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4E0A613AB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C19006B0070; Sun, 9 May 2021 15:38:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA1656B0071; Sun, 9 May 2021 15:38:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F4496B0072; Sun, 9 May 2021 15:38:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 7E9056B0070 for ; Sun, 9 May 2021 15:38:57 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 31F05181AF5C2 for ; Sun, 9 May 2021 19:38:57 +0000 (UTC) X-FDA: 78122705514.09.8A080AC Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by imf29.hostedemail.com (Postfix) with ESMTP id 96C3EF4 for ; Sun, 9 May 2021 19:38:50 +0000 (UTC) Received: by mail-lj1-f173.google.com with SMTP id w4so18153298ljw.9 for ; Sun, 09 May 2021 12:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=P9KHoUwqxbexOhqNv8F1Ywyb8vsO2FBF8NuW/VNOiPo=; b=VaZ5jstwwQCops8VcXABpwtZedX2N9xGSqxNt94hO02IGPASRBxnIMwnDKVET16xIi qrW9cC7TB3wiCLgTDnnj9R1JMDHfvUU/N7jVA/46hjEV8jZkc9PdX4CtqaPZejlKJVUp ETz4JyWiit7zgBhH8pkTWPAL3OW++bggSY7s8K7R3cKRt8KBX9HqpHsIyJAxbyptwYUp 7tHEmMuazGZlkq8s1AxphPfoqkf71QP0FICuzJbptiSV+NoAzL5W24z1O5RNM3BfOrRn HXLZI4BQBCnWlqdDzHonZEbrNU7Wg+GhQBCjZar4gDpxl+NrGpuGnfuR35q+q74fSAVk 6hEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=P9KHoUwqxbexOhqNv8F1Ywyb8vsO2FBF8NuW/VNOiPo=; b=eszFiVI8IldqpW3TjgoHGSa0abCrsqeVvfyh7aOcBMfcR4JqrpJ7Q2MaIFQvGpbTtd tQgTHNOIiddPokb14ZLdRzsSh0DQhHqC7q8pjBTuELv1Uq4dG2GlY5HUAf1HP+SRZFtr 1WBph0rxblx78P0EwhEhCOgrRzufkV7UqMUXle4XbarkK35StlgselLHih5yqIdsRbmg FFuwclNrJ18H4YnE+Q8KpFjzoU8Kqv4KhOKE5fypdet8W5ix+WBZAzc9ERE5OrRGBb76 mBEgyknUs2WgWMzqZ/X55RnFRkN+jvncs9BcRpN5DF+s+565M93UkDphESx3kJFF+Ipq Y0WQ== X-Gm-Message-State: AOAM530GyQtAA3PN34EvwAYkQC+lWA1iVNHp6gnTLiXl8TBsmFeHwd/B gsGKZ/Eo+n6YIbdxE4l5Efc= X-Google-Smtp-Source: ABdhPJwoKB66eyiScOwS2EQBs77oKT0vsxLrKNhx3uXTnls4aHYUOxRXMZAoxnrRucSwVYo2UlrlIw== X-Received: by 2002:a2e:b5cd:: with SMTP id g13mr17743675ljn.0.1620589135179; Sun, 09 May 2021 12:38:55 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id m17sm1999269lfh.146.2021.05.09.12.38.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 May 2021 12:38:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Matthew Wilcox , Nicholas Piggin , Mel Gorman , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v2 1/2] mm/vmalloc: Switch to bulk allocator in __vmalloc_area_node() Date: Sun, 9 May 2021 21:38:43 +0200 Message-Id: <20210509193844.2562-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=VaZ5jstw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.173 as permitted sender) smtp.mailfrom=urezki@gmail.com X-Stat-Signature: 3dz3877qoja11iu9bto5bihsbojz4y5a X-Rspamd-Queue-Id: 96C3EF4 X-Rspamd-Server: rspam05 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-lj1-f173.google.com; client-ip=209.85.208.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620589130-717755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recently there has been introduced a page bulk allocator for users which need to get number of pages per one call request. For order-0 pages switch to __alloc_pages_bulk() instead of alloc_pages_node(), the reason is the former is not capable of allocating set of pages, thus a one call is per one page. Second, according to my tests the bulk allocator uses less cycles even for scenarios when only one page is requested. Running the "perf" on same test case shows below difference: - 45.18% __vmalloc_node - __vmalloc_node_range - 35.60% __alloc_pages - get_page_from_freelist 3.36% __list_del_entry_valid 3.00% check_preemption_disabled 1.42% prep_new_page - 31.00% __vmalloc_node - __vmalloc_node_range - 14.48% __alloc_pages_bulk 3.22% __list_del_entry_valid - 0.83% __alloc_pages get_page_from_freelist The "test_vmalloc.sh" also shows performance improvements: fix_size_alloc_test_4MB loops: 1000000 avg: 89105095 usec fix_size_alloc_test loops: 1000000 avg: 513672 usec full_fit_alloc_test loops: 1000000 avg: 748900 usec long_busy_list_alloc_test loops: 1000000 avg: 8043038 usec random_size_alloc_test loops: 1000000 avg: 4028582 usec fix_align_alloc_test loops: 1000000 avg: 1457671 usec fix_size_alloc_test_4MB loops: 1000000 avg: 62083711 usec fix_size_alloc_test loops: 1000000 avg: 449207 usec full_fit_alloc_test loops: 1000000 avg: 735985 usec long_busy_list_alloc_test loops: 1000000 avg: 5176052 usec random_size_alloc_test loops: 1000000 avg: 2589252 usec fix_align_alloc_test loops: 1000000 avg: 1365009 usec For example 4MB allocations illustrates ~30% gain, all the rest is also better. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 74 ++++++++++++++++++++++++++++------------------------ 1 file changed, 40 insertions(+), 34 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5d96fee17226..dbc6744400d5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2766,8 +2766,6 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, unsigned long array_size; unsigned int nr_small_pages = size >> PAGE_SHIFT; unsigned int page_order; - struct page **pages; - unsigned int i; array_size = (unsigned long)nr_small_pages * sizeof(struct page *); gfp_mask |= __GFP_NOWARN; @@ -2776,13 +2774,13 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, area->caller); } else { - pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node(array_size, nested_gfp, node); } - if (!pages) { + if (!area->pages) { free_vm_area(area); warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " @@ -2791,43 +2789,51 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, return NULL; } - area->pages = pages; - area->nr_pages = nr_small_pages; set_vm_area_page_order(area, page_shift - PAGE_SHIFT); - page_order = vm_area_page_order(area); - /* - * Careful, we allocate and map page_order pages, but tracking is done - * per PAGE_SIZE page so as to keep the vm_struct APIs independent of - * the physical/mapped size. - */ - for (i = 0; i < area->nr_pages; i += 1U << page_order) { - struct page *page; - int p; - - /* Compound pages required for remap_vmalloc_page */ - page = alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); - if (unlikely(!page)) { - /* Successfully allocated i pages, free them in __vfree() */ - area->nr_pages = i; - atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - warn_alloc(gfp_mask, NULL, - "vmalloc size %lu allocation failure: " - "page order %u allocation failed", - area->nr_pages * PAGE_SIZE, page_order); - goto fail; - } + if (!page_order) { + area->nr_pages = __alloc_pages_bulk(gfp_mask, node, + NULL, nr_small_pages, NULL, area->pages); + } else { + /* + * Careful, we allocate and map page_order pages, but tracking is done + * per PAGE_SIZE page so as to keep the vm_struct APIs independent of + * the physical/mapped size. + */ + for (area->nr_pages = 0; area->nr_pages < nr_small_pages; + area->nr_pages += 1U << page_order) { + struct page *page; + int i; + + /* Compound pages required for remap_vmalloc_page */ + page = alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); + if (unlikely(!page)) + break; - for (p = 0; p < (1U << page_order); p++) - area->pages[i + p] = page + p; + for (i = 0; i < (1U << page_order); i++) + area->pages[area->nr_pages + i] = page + i; - if (gfpflags_allow_blocking(gfp_mask)) - cond_resched(); + if (gfpflags_allow_blocking(gfp_mask)) + cond_resched(); + } } + atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) { + /* + * If not enough pages were obtained to accomplish an + * allocation request, free them via __vfree() if any. + */ + if (area->nr_pages != nr_small_pages) { + warn_alloc(gfp_mask, NULL, + "vmalloc size %lu allocation failure: " + "page order %u allocation failed", + area->nr_pages * PAGE_SIZE, page_order); + goto fail; + } + + if (vmap_pages_range(addr, addr + size, prot, area->pages, page_shift) < 0) { warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " "failed to map pages",