From patchwork Fri Jul 12 12:02:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11042189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFCD4138B for ; Fri, 12 Jul 2019 12:02:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCBCE28BFD for ; Fri, 12 Jul 2019 12:02:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B120A28C0A; Fri, 12 Jul 2019 12:02:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E4CC28C09 for ; Fri, 12 Jul 2019 12:02:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AA4C8E0140; Fri, 12 Jul 2019 08:02:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 25A068E00DB; Fri, 12 Jul 2019 08:02:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 149E88E0140; Fri, 12 Jul 2019 08:02:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id D15B18E00DB for ; Fri, 12 Jul 2019 08:02:51 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id 145so5414833pfw.16 for ; Fri, 12 Jul 2019 05:02:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=1N5T/DvjIbzKu60HNBBXne5dRJa1DkC2KbW+cEgIYvs=; b=IoZ1NCxU4Nid3xUGrqRLalUI6rGuliXp6rxC3tnRjnVooMfxjkLs4OqMyn6OYWtFUf a/XzSfO0Yz+OiMtgAfbFGvXQJ3Pn+c6HM73G3z8xWV8gJMMrCBaB/XrwWQL/JVcVjdYV 4JuRR7znEMqUo2bJpz/UO2Oez2NOmj1pYyso4fKTJ1SohuDT121JGtr4DROJ3Ten4Mbq iPwaLNsQoG+dzNajWjHDBTBqMKRI3j+bn+Z8gbz/cxLkGTXthyfF8PFw1evGShL1PKBI 8kJQXbZ4ySIQ8Fap5A0jxvOihCBljxn1m6I1vHHd8lX86HkRN9zK0gf8mD/TRzyOGUzT Mnxw== X-Gm-Message-State: APjAAAXEwOSoveV6O5m5Q19orSIYImEhjicBLSFPdxjryiH21WtG/WZK mit3HykKDYpFE3beKJb2CNT+w1JDQ0JMw0KafsHoxz6nkulaIAT1nbSMeOfcIhyGkFa1QRvfxZ8 0Fw9tO52gq3hT9D10yzy+XYNYbVAm1zMu56ihujmdg3VDy++V2WaIMIv3AZs6SnuTlQ== X-Received: by 2002:a17:90a:376f:: with SMTP id u102mr11450485pjb.5.1562932971522; Fri, 12 Jul 2019 05:02:51 -0700 (PDT) X-Received: by 2002:a17:90a:376f:: with SMTP id u102mr11450336pjb.5.1562932970200; Fri, 12 Jul 2019 05:02:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562932970; cv=none; d=google.com; s=arc-20160816; b=a7j8Kk/G6ACAJA7xbbu0LmizL6sw89cB3HarGXXEw2tn1+Ia8eTn4qQs16siA86s1x zR8rXhftLvpPICRm/6R2cKicyF1sJBu1uHZOrqDEIKQZ8mHRYwDoLpMxejHGiNi4g1M1 uq8IsaS5GXGVGBIdxNHHncHk8RTQ1WaEGlHVJ9ddLzNKT2jQr18eHh3gdhTdVwSbQxbP GTNeIh98pIFx9kUW9WQbIcmVIGbc2CM8G1hO4lsqioYLVrkMoMu/suutbCIjKgdRbQDT hCtROMI88n454DgKVKSRLb8322CcEks3w+MTGblW5hLwiN4WBfOc1a1Lw1Z1g7l3eNY5 +H0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=1N5T/DvjIbzKu60HNBBXne5dRJa1DkC2KbW+cEgIYvs=; b=l3FIfO6EAv+IfEeTppuEIluKQxtVNmTS2/yKzgmdFfMAq86u1KtdVo9Xh/+AzYlL+A kfDbE71F9zJHoDRTEXYpKOBSbSh/ew8I5QBisIa+N9a+pt4+WB0QLQKKdT6FK92W49k4 bJRxesVApz4mY4twTEFiT2Ycg/az++ix87c+VodV1u9vRuZn5bhzL5kTHy9KgU4KAnCA nCkXcWlq+S0dPjrU9xnYF+iveWEv9e9DKYJpTPF54IqO53PcOD4NkYHzMOKiqaDycziX qUQzsRheXF3n82SW301HZnnwNWetuQBjJyJQ5bPf1DyNvNSnlp2R1YhC50i1lNqQ8i1d 3zHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=DPXSuuA3; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u64sor4566568pfb.68.2019.07.12.05.02.50 for (Google Transport Security); Fri, 12 Jul 2019 05:02:50 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=DPXSuuA3; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1N5T/DvjIbzKu60HNBBXne5dRJa1DkC2KbW+cEgIYvs=; b=DPXSuuA3BhnztMcN3LH2s8P/GhVUIZ+INcZ8vH0qgqibYFB2TDvNG5V4bjJ5hB+roD 9EpihKRQWeERkyyQXrdjT8wWp/BDewlmTJ3r2t4MmzQW6ocAVC4lZ4JjuB1l4H46FKsa JQ4Y1J8oB8D4IZGUTPwPRPc+2tqj4g/+Zl5Wn8c5DGgduv8L++LeAt7lAsekZRaV6gZ1 IzKQmq6Yl7levcLSBipOot+VObYWOHgI4XgPSfKffhfRYX5cfC07p1ODC/jfobIFbNvS oiQYYr5f7Fw66slqU8MghTJuAYpLuSC+ClKCiB1aAR0qazH0j+v+zarzHF7/FQE2yxzp z6hQ== X-Google-Smtp-Source: APXvYqxR3apRKE+bjL/RIafrN09zwtwDK7k4VgH+z+nofeQhrdMlDJRdJAJmgBykSLy7PDtfNCHdgg== X-Received: by 2002:a65:614a:: with SMTP id o10mr9920097pgv.407.1562932969685; Fri, 12 Jul 2019 05:02:49 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:478:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id a128sm4605496pfb.185.2019.07.12.05.02.43 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 12 Jul 2019 05:02:49 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 1/2] mm/vmalloc: do not keep unpurged areas in the busy tree Date: Fri, 12 Jul 2019 20:02:12 +0800 Message-Id: <20190712120213.2825-2-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190712120213.2825-1-lpf.vector@gmail.com> References: <20190712120213.2825-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "Uladzislau Rezki (Sony)" The busy tree can be quite big, even though the area is freed or unmapped it still stays there until "purge" logic removes it. 1) Optimize and reduce the size of "busy" tree by removing a node from it right away as soon as user triggers free paths. It is possible to do so, because the allocation is done using another augmented tree. The vmalloc test driver shows the difference, for example the "fix_size_alloc_test" is ~11% better comparing with default configuration: sudo ./test_vmalloc.sh performance Summary: fix_size_alloc_test loops: 1000000 avg: 993985 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973554 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12617652 usec Summary: fix_size_alloc_test loops: 1000000 avg: 882263 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973407 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12593929 usec 2) Since the busy tree now contains allocated areas only and does not interfere with lazily free nodes, introduce the new function show_purge_info() that dumps "unpurged" areas that is propagated through "/proc/vmallocinfo". 3) Eliminate VM_LAZY_FREE flag. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 51 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4fa8d84599b0..9eb700a2087b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_LAZY_FREE 0x02 #define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); @@ -541,7 +540,7 @@ link_va(struct vmap_area *va, struct rb_root *root, static __always_inline void unlink_va(struct vmap_area *va, struct rb_root *root) { - if (WARN_ON(RB_EMPTY_NODE(&va->rb_node))) + if (RB_EMPTY_NODE(&va->rb_node)) return; if (root == &free_vmap_area_root) @@ -1167,7 +1166,11 @@ EXPORT_SYMBOL_GPL(unregister_vmap_purge_notifier); static void __free_vmap_area(struct vmap_area *va) { /* - * Remove from the busy tree/list. + * In most cases VA is not attached to the tree, but there + * are a few exceptions: + * + * - is linked only in case of pcpu, recovery part; + * - if radix_tree_preload gets failed, see new_vmap_block(). */ unlink_va(va, &vmap_area_root); @@ -1318,6 +1321,10 @@ static void free_vmap_area_noflush(struct vmap_area *va) { unsigned long nr_lazy; + spin_lock(&vmap_area_lock); + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr); @@ -2137,14 +2144,13 @@ struct vm_struct *remove_vm_area(const void *addr) might_sleep(); - va = find_vmap_area((unsigned long)addr); + spin_lock(&vmap_area_lock); + va = __find_vmap_area((unsigned long)addr); if (va && va->flags & VM_VM_AREA) { struct vm_struct *vm = va->vm; - spin_lock(&vmap_area_lock); va->vm = NULL; va->flags &= ~VM_VM_AREA; - va->flags |= VM_LAZY_FREE; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2152,6 +2158,8 @@ struct vm_struct *remove_vm_area(const void *addr) return vm; } + + spin_unlock(&vmap_area_lock); return NULL; } @@ -3431,6 +3439,22 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) } } +static void show_purge_info(struct seq_file *m) +{ + struct llist_node *head; + struct vmap_area *va; + + head = READ_ONCE(vmap_purge_list.first); + if (head == NULL) + return; + + llist_for_each_entry(va, head, purge_list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +} + static int s_show(struct seq_file *m, void *p) { struct vmap_area *va; @@ -3443,10 +3467,9 @@ static int s_show(struct seq_file *m, void *p) * behalf of vmap area is being tear down or vm_map_ram allocation. */ if (!(va->flags & VM_VM_AREA)) { - seq_printf(m, "0x%pK-0x%pK %7ld %s\n", + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start, - va->flags & VM_LAZY_FREE ? "unpurged vm_area" : "vm_map_ram"); + va->va_end - va->va_start); return 0; } @@ -3482,6 +3505,16 @@ static int s_show(struct seq_file *m, void *p) show_numa_info(m, v); seq_putc(m, '\n'); + + /* + * As a final step, dump "unpurged" areas. Note, + * that entire "/proc/vmallocinfo" output will not + * be address sorted, because the purge list is not + * sorted. + */ + if (list_is_last(&va->list, &vmap_area_list)) + show_purge_info(m); + return 0; } From patchwork Fri Jul 12 12:02:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11042191 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34213138B for ; Fri, 12 Jul 2019 12:03:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E86028BDA for ; Fri, 12 Jul 2019 12:03:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E32728BFD; Fri, 12 Jul 2019 12:03:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B53528BDA for ; Fri, 12 Jul 2019 12:03:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C55A8E0141; Fri, 12 Jul 2019 08:03:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 89C358E00DB; Fri, 12 Jul 2019 08:03:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73CB78E0141; Fri, 12 Jul 2019 08:03:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 3EA6A8E00DB for ; Fri, 12 Jul 2019 08:03:13 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id 6so5434227pfi.6 for ; Fri, 12 Jul 2019 05:03:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=pG1dMHtZH73TRwIerhkeF5MzPMCfsVBoJ6eMm8ohCck=; b=fVRGVrmygI5+gXJANb0EPLxbeivzrqi8WKMFgI7ABOetkCIhqpVGARpu1F7kPG6jdp fyy7TGKufjgQ1Ba4gXeMCwj2/ZbwDxEXrgehk2mfQuvj67TMJE0gy+NdtYaAKjB6SKpy b5y6bwGcFH/E7cXeC9CCwOt6SM14RBtDaouKPUvsfYB8KlhQNh/4lX6bsJKFwwUdnwPu tpPFZMXs/h1q2EL9UnF+RCZuf7tTWkfMPNPCVxB520iiAdYgvq7lEr3LAEhbV9kpAd+1 Zv6piXlVgkFLElkxJDuXm+loFfAcBwuCavVECkjvHa10Q4360VFfwwKOhojFgiv/DWVN C9zA== X-Gm-Message-State: APjAAAVJXLS2gZHPHZZvQCw6j5J5Rv3yKW7c7DzAPnlgNuenvaJVizBf w+rlZIPktwkbI+FPGAyiHmKRqZ1Y9Utau2Wo3EayfkVlLF3ICyiTGMAyw8Xa+q8udil7hMe6hdQ xtq8hEqTur61A9QJB6Ut6t6ZTAsyWWlsACF0XZGnasGD14r+zG+klui/jooEgDYN2Vg== X-Received: by 2002:a17:902:6a85:: with SMTP id n5mr10338882plk.73.1562932992859; Fri, 12 Jul 2019 05:03:12 -0700 (PDT) X-Received: by 2002:a17:902:6a85:: with SMTP id n5mr10338747plk.73.1562932991481; Fri, 12 Jul 2019 05:03:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562932991; cv=none; d=google.com; s=arc-20160816; b=Xk1Kd/i0LHA2md7YEm2Ve/TKWi8f33OyFDH7CcOocXW3OHXN7dFg93tGf4+eB7gnHe 5107q8gATOeKGJFRyh8oO7048tSQZV/D3ogKeEWhjPwWLwY9jUiCv7Uhz0RNwq0ERFdO Q9yqJ3XptMNjuex22g8WuSTVmTq9CFyHpyofAtbxFCmA4gY73XyQXeIXLKEjwIJ9pBTP LADGIXuOuclr70kaPu/xp/5Wqygy0fCk/bZN1UUKM5+YgI63azOQqmNZpKydtOYqDZAM FJzIEwTdSWPEfZN8OKSpO8qB7BisSutUewgWWKFBtxghdfVhoybbC3erEYcs8Nn400eO 7BdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pG1dMHtZH73TRwIerhkeF5MzPMCfsVBoJ6eMm8ohCck=; b=BtKHnuivK9JTfEaM3OFyRmob1iwxi6kIEMyOSPg8dalKzPIgflVbF1J7lCTPv9IsM4 0bbpOZ+Vo/1O7yX1XXufVuYh3g1YDfqXqgvX97Tp4FRMJnk4VdsgsvZlkkFQYheazHQK t9jtz7dSFVP5PpuU8Z5yrh4F+3VKNoJ/rp/LFki1wm0Qq9rN4sb44mMI/efUmiUuNo6A 7KD2wCfXG1QzZu9QrvYlJs2SXtAnOWmubHSgT6DCLRHejg5gYm+QKucekCYUBvoxHZXn l3qKzVFU30we60B4ZaT6TqVKiVIn1BQ8N3tfWqucs0igakOvnMkaaQKe4aAYrIiBpJln /H1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Po5R+v4T; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id f10sor4543484pgg.6.2019.07.12.05.03.11 for (Google Transport Security); Fri, 12 Jul 2019 05:03:11 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Po5R+v4T; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pG1dMHtZH73TRwIerhkeF5MzPMCfsVBoJ6eMm8ohCck=; b=Po5R+v4Tq9SSrH6dw19MSifuek7bEwcNoexmrF6UIV5Tic0O540E39uOdiAc+TZekt Yg1XDZ8wGwfvTBgfg03ER1ZzIHw1fnAd17nF1igYdadTeCHdMGqxav3ZdjyJ5dcEu4Ll +6xvShvyiUWzwC8GpyXvk1jQWe34DEo2q7aA19TOGfWTliGIiTjCJ2/4Do+j3zVIMGSJ kdyIv9YZfhODXjrGut8IFHEjEpbWw2A2kFvsW3G+O6LoTv1FscJ4zqyzNuePsfo40fsY NPnMmfLu69M433nqOSkZoKaW4WMb0WZQL9Hqn/x4gLADGzjSggjSfef6UOAhbUWRi3Ya tIUw== X-Google-Smtp-Source: APXvYqwah3U58lG/iFfoMWrOCymtN4tLD33/PiCMqjtD3w+TClssgHN7tIvimgRpgIKmVXP41rg6DQ== X-Received: by 2002:a63:d756:: with SMTP id w22mr10239371pgi.156.1562932991099; Fri, 12 Jul 2019 05:03:11 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:478:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id a128sm4605496pfb.185.2019.07.12.05.03.04 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 12 Jul 2019 05:03:10 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Pengfei Li Subject: [PATCH v4 2/2] mm/vmalloc.c: Modify struct vmap_area to reduce its size Date: Fri, 12 Jul 2019 20:02:13 +0800 Message-Id: <20190712120213.2825-3-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190712120213.2825-1-lpf.vector@gmail.com> References: <20190712120213.2825-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Objective --------- The current implementation of struct vmap_area wasted space. After applying this commit, sizeof(struct vmap_area) has been reduced from 11 words to 8 words. Description ----------- 1) Pack "vm" and "subtree_max_size" This is no problem because A) "vm" is only used when vmap_area is in "busy" tree B) "subtree_max_size" is only used when vmap_area is in "free" tree 2) Pack "purge_list" The variable "purge_list" is only used when vmap_area is in "lazy purge" list. So it can be packed with other variables, which are only used in rbtree and list sorted by address. 3) Eliminate "flags". Since only one flag VM_VM_AREA is being used, and the same thing can be done by judging whether "vm" is NULL, then the "flags" can be eliminated. Signed-off-by: Pengfei Li Suggested-by: Uladzislau Rezki (Sony) --- include/linux/vmalloc.h | 40 +++++++++++++++++++++++++++++++--------- mm/vmalloc.c | 28 +++++++++++++--------------- 2 files changed, 44 insertions(+), 24 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 9b21d0047710..6fb377ca9e7a 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -51,15 +51,37 @@ struct vmap_area { unsigned long va_start; unsigned long va_end; - /* - * Largest available free size in subtree. - */ - unsigned long subtree_max_size; - unsigned long flags; - struct rb_node rb_node; /* address sorted rbtree */ - struct list_head list; /* address sorted list */ - struct llist_node purge_list; /* "lazy purge" list */ - struct vm_struct *vm; + union { + /* In rbtree and list sorted by address */ + struct { + union { + /* + * In "busy" rbtree and list. + * rbtree root: vmap_area_root + * list head: vmap_area_list + */ + struct vm_struct *vm; + + /* + * In "free" rbtree and list. + * rbtree root: free_vmap_area_root + * list head: free_vmap_area_list + */ + unsigned long subtree_max_size; + }; + + struct rb_node rb_node; + struct list_head list; + }; + + /* + * In "lazy purge" list. + * llist head: vmap_purge_list + */ + struct { + struct llist_node purge_list; + }; + }; }; /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9eb700a2087b..1245d3285a32 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); /* Export for kexec only */ @@ -1115,7 +1114,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; - va->flags = 0; + va->vm = NULL; insert_vmap_area(va, &vmap_area_root, &vmap_area_list); spin_unlock(&vmap_area_lock); @@ -1279,7 +1278,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) llist_for_each_entry_safe(va, n_va, valist, purge_list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; - __free_vmap_area(va); + merge_or_add_vmap_area(va, + &free_vmap_area_root, &free_vmap_area_list); + atomic_long_sub(nr, &vmap_lazy_nr); if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) @@ -1919,7 +1920,6 @@ void __init vmalloc_init(void) if (WARN_ON_ONCE(!va)) continue; - va->flags = VM_VM_AREA; va->va_start = (unsigned long)tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; @@ -2017,7 +2017,6 @@ static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, vm->size = va->va_end - va->va_start; vm->caller = caller; va->vm = vm; - va->flags |= VM_VM_AREA; spin_unlock(&vmap_area_lock); } @@ -2122,10 +2121,10 @@ struct vm_struct *find_vm_area(const void *addr) struct vmap_area *va; va = find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) - return va->vm; + if (!va) + return NULL; - return NULL; + return va->vm; } /** @@ -2146,11 +2145,10 @@ struct vm_struct *remove_vm_area(const void *addr) spin_lock(&vmap_area_lock); va = __find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) { + if (va && va->vm) { struct vm_struct *vm = va->vm; va->vm = NULL; - va->flags &= ~VM_VM_AREA; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2853,7 +2851,7 @@ long vread(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -2933,7 +2931,7 @@ long vwrite(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -3463,10 +3461,10 @@ static int s_show(struct seq_file *m, void *p) va = list_entry(p, struct vmap_area, list); /* - * s_show can encounter race with remove_vm_area, !VM_VM_AREA on - * behalf of vmap area is being tear down or vm_map_ram allocation. + * If !va->vm then this vmap_area object is allocated + * by vm_map_ram. */ - if (!(va->flags & VM_VM_AREA)) { + if (!va->vm) { seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, va->va_end - va->va_start);