From patchwork Tue Jul 16 13:26:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11046135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E536A746 for ; Tue, 16 Jul 2019 13:26:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D27D427FB3 for ; Tue, 16 Jul 2019 13:26:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C551828429; Tue, 16 Jul 2019 13:26:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87F5427FB3 for ; Tue, 16 Jul 2019 13:26:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1BDD6B0005; Tue, 16 Jul 2019 09:26:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9CB846B0006; Tue, 16 Jul 2019 09:26:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 894DC8E0001; Tue, 16 Jul 2019 09:26:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 519C16B0005 for ; Tue, 16 Jul 2019 09:26:50 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id o6so10175469plk.23 for ; Tue, 16 Jul 2019 06:26:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=Nrxnco5rX3SqQ3K2z5bojQrAKYFeXol4hZ52BazKTzpGijEbLtaON8EOtNL1zGC12T jKA8fuRo+/2PnyLH+JMCxqmGHy84jE5sC+lmIKZrCtxVacPem9fRiZ9W4xV1Hp15ePBG kJZDIq905Q+zthrFTrt6k6qEt87sJ+QEnJ2DxPwwIv9+dLUhmZwHAqFPgEWrI/qqPX9h HwB9+pcsTvH4nx07b5pqC6iAWh+5sAkU0J+Rp/r7Wmx8R3GjUpKrOwxb3ccQmGsALs2t 4DjTgYkGw6tJdolsmiCZXiTVAV0H3PEiMsiCPQmQBhTBwBIhALHm5Cod0nzGG/Out74Q ncqg== X-Gm-Message-State: APjAAAXbxOFfmPFXyvTAjvalhExyCVzVGiVagOv7GlLDkY/+UOmFiFRh 8BFuAmk9HnIyzB7qiwIl5WYe3G7Ip+15LWMr/WeENZsE5QYfGgKiY/0gUBkV+PWe7YRkxulx3Ov tpe9gf6+WxzDaELIp/IakR+/EYPHGVw9fEk/TEci+k1FFUupgV12+BaKhbdywj63FnQ== X-Received: by 2002:a63:1046:: with SMTP id 6mr34718280pgq.111.1563283609743; Tue, 16 Jul 2019 06:26:49 -0700 (PDT) X-Received: by 2002:a63:1046:: with SMTP id 6mr34718129pgq.111.1563283608395; Tue, 16 Jul 2019 06:26:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563283608; cv=none; d=google.com; s=arc-20160816; b=MYvRq3sx4izrp6Mft3ZEixMvB1Gw4/GQUs3HqPl86nEZGUTcOk9EYZihsrAUVWZckw NsQuzGagcBl5+L+yZ4BbMz5W0WzDByEm0s5GosBP3Tnisu8zhtkLYH+nw91HPkm3Dkh6 BtboETGPZGZHdzY5o35sVFuYGOkXYAXL1NsCx9ApAYB7D3gQvf6Zag0oytKsCZExLU+i wCAW8zk0pSA2UoRgg322IptY2T4iftMCAueHJvHGLo3FmvgNoi2ddDBBXsfjulyaL/6z Ri2LhvuUCKA6cN/Yk5XoeEK7jVOu32EA4ivufe0yVoQG2NwLlQbljjES97KVx72YuQRf I58A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=nsQCoWCe7CCutdRQwKnaYsZUioiYu4H8v/i3EpehlYhewE6vhJ9jWUcPPcVr4mulKk CdsiTjSXdf07/9rE9sS/I6ib2dOfV3zfTuGRpvwdYrHJnyprHFpnArt+bIucUzWmT5xh bSXvFfTz/P11bcJYr6TiVwtpgqS+UHOZ5KhL4fUbov28vo8GCG66bsF5LFaFYqJWvTF6 5CwQoFxOoIPFzG9Ws2R4cVhniRnl0VZyNJ0HLcD2TfXSYeUFTHgkxjkaUV4Jbn+nD42Z O67kJVqe/UZ1qb0PbpAVeR6db57fIw81YrSMGNuV2EEdW0x8U050Lu05MO1lrnFcSEV+ tIEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=SfxbVk9z; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z25sor10580294pgv.71.2019.07.16.06.26.48 for (Google Transport Security); Tue, 16 Jul 2019 06:26:48 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=SfxbVk9z; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=SfxbVk9zgVz2Ps0un9Tse4hUUdRf4Prepk2LjLe8DpwR4BEOKi0VTBn5xTg778ccqU rTdChq6dKK0X9Rw/Y98Md2IuIVKqlLE0eM/jnTER1i49F+02juTX2RTXM87i5Yu5lnER gfVozyJJ0n+ugVPE2FNlkjNrangn//4DjlnFNPraQt6QSB4PgzoFKj3ahJN+Al4Yos/3 fMsiZ96gCoc2hJ9TJ48WLN2ca5STZiijBBKl915p0l5tQvnYXfP+8CeK6M+sFH3Lr34B vXPkppZyoVe2jZMNa0MNHbDJEkFFP8vVdUNd5PB1znbrTihyFqmk6J9cY0wyu/sTaoKi AQ0w== X-Google-Smtp-Source: APXvYqz5OuQb6D2Fp8RCUxssijj+aLGD4V7/ZrKKVKZHGaNlTSTN/MhpzdVUfmVeJIZ7W1vTufjRog== X-Received: by 2002:a63:9a41:: with SMTP id e1mr34297879pgo.210.1563283608045; Tue, 16 Jul 2019 06:26:48 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:bf0:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id q1sm21472311pfg.84.2019.07.16.06.26.41 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 16 Jul 2019 06:26:47 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org, willy@infradead.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 1/2] mm/vmalloc: do not keep unpurged areas in the busy tree Date: Tue, 16 Jul 2019 21:26:03 +0800 Message-Id: <20190716132604.28289-2-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190716132604.28289-1-lpf.vector@gmail.com> References: <20190716132604.28289-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "Uladzislau Rezki (Sony)" The busy tree can be quite big, even though the area is freed or unmapped it still stays there until "purge" logic removes it. 1) Optimize and reduce the size of "busy" tree by removing a node from it right away as soon as user triggers free paths. It is possible to do so, because the allocation is done using another augmented tree. The vmalloc test driver shows the difference, for example the "fix_size_alloc_test" is ~11% better comparing with default configuration: sudo ./test_vmalloc.sh performance Summary: fix_size_alloc_test loops: 1000000 avg: 993985 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973554 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12617652 usec Summary: fix_size_alloc_test loops: 1000000 avg: 882263 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973407 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12593929 usec 2) Since the busy tree now contains allocated areas only and does not interfere with lazily free nodes, introduce the new function show_purge_info() that dumps "unpurged" areas that is propagated through "/proc/vmallocinfo". 3) Eliminate VM_LAZY_FREE flag. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4fa8d84599b0..71d8040a8a0b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_LAZY_FREE 0x02 #define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); @@ -1276,7 +1275,14 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) llist_for_each_entry_safe(va, n_va, valist, purge_list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; - __free_vmap_area(va); + /* + * Finally insert or merge lazily-freed area. It is + * detached and there is no need to "unlink" it from + * anything. + */ + merge_or_add_vmap_area(va, + &free_vmap_area_root, &free_vmap_area_list); + atomic_long_sub(nr, &vmap_lazy_nr); if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) @@ -1318,6 +1324,10 @@ static void free_vmap_area_noflush(struct vmap_area *va) { unsigned long nr_lazy; + spin_lock(&vmap_area_lock); + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr); @@ -2137,14 +2147,13 @@ struct vm_struct *remove_vm_area(const void *addr) might_sleep(); - va = find_vmap_area((unsigned long)addr); + spin_lock(&vmap_area_lock); + va = __find_vmap_area((unsigned long)addr); if (va && va->flags & VM_VM_AREA) { struct vm_struct *vm = va->vm; - spin_lock(&vmap_area_lock); va->vm = NULL; va->flags &= ~VM_VM_AREA; - va->flags |= VM_LAZY_FREE; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2152,6 +2161,8 @@ struct vm_struct *remove_vm_area(const void *addr) return vm; } + + spin_unlock(&vmap_area_lock); return NULL; } @@ -3431,6 +3442,22 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) } } +static void show_purge_info(struct seq_file *m) +{ + struct llist_node *head; + struct vmap_area *va; + + head = READ_ONCE(vmap_purge_list.first); + if (head == NULL) + return; + + llist_for_each_entry(va, head, purge_list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +} + static int s_show(struct seq_file *m, void *p) { struct vmap_area *va; @@ -3443,10 +3470,9 @@ static int s_show(struct seq_file *m, void *p) * behalf of vmap area is being tear down or vm_map_ram allocation. */ if (!(va->flags & VM_VM_AREA)) { - seq_printf(m, "0x%pK-0x%pK %7ld %s\n", + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start, - va->flags & VM_LAZY_FREE ? "unpurged vm_area" : "vm_map_ram"); + va->va_end - va->va_start); return 0; } @@ -3482,6 +3508,16 @@ static int s_show(struct seq_file *m, void *p) show_numa_info(m, v); seq_putc(m, '\n'); + + /* + * As a final step, dump "unpurged" areas. Note, + * that entire "/proc/vmallocinfo" output will not + * be address sorted, because the purge list is not + * sorted. + */ + if (list_is_last(&va->list, &vmap_area_list)) + show_purge_info(m); + return 0; } From patchwork Tue Jul 16 13:26:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11046137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D2DA746 for ; Tue, 16 Jul 2019 13:26:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E60C28429 for ; Tue, 16 Jul 2019 13:26:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3254028585; Tue, 16 Jul 2019 13:26:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9798128450 for ; Tue, 16 Jul 2019 13:26:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E30A6B0006; Tue, 16 Jul 2019 09:26:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 894778E0003; Tue, 16 Jul 2019 09:26:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7841E8E0001; Tue, 16 Jul 2019 09:26:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 444516B0006 for ; Tue, 16 Jul 2019 09:26:57 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id u1so12611050pgr.13 for ; Tue, 16 Jul 2019 06:26:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2FLMt1biejUADnaZS+D7n9SrnPK40F0jXQ5om7TEM0E=; b=CHZYnANPe+66BElbLaf10niGrYa+jqV7m86Tf1KiZVlJdXaFZ0NM8L0/LjkZ9nEWFw QK4u/NvTeZWSaxlJd7VDToXIo6xCu4vGX6yWuBD1CpkINxNOeJk8AhEU3DHW4u4vnRNJ RlYi6rbEkWoMzgq3pQXhOuOUmn++Vqz7kflxKREAOARIkyK4jfIiaNFJAt6HtwC30hdV OHYFWQv3op33jEnTrE+NGUDQJtZkpblJCrzupRXnG1zaC6XduQljUG3TVxkufaHjZhGd acs7nqzA+nj766IBishfQ+KS94IQjOILPxZb9bkKNegac7NjKiXwPe0dASOO8HVW/GAK gJyg== X-Gm-Message-State: APjAAAXCk3l+WwPXL8RAiUJ1wE3llBR628GMqVzxzPGK2uZ/x3iXpK7D aT0AoFAmDmBvD4w+C/HBHTDVrBiLHE2Gz97jWr/n0k4M/Vl6hlVr8KLBWaR6gEDv7dNEGVtV2CX 9Myq6hiGuYF+Qrh7puIvKxUIMyMvYxo7jvjqoOT2vNYOBcCz5lYBcNxEGGZlmz5qxNQ== X-Received: by 2002:a17:902:2be8:: with SMTP id l95mr32227775plb.231.1563283616872; Tue, 16 Jul 2019 06:26:56 -0700 (PDT) X-Received: by 2002:a17:902:2be8:: with SMTP id l95mr32227692plb.231.1563283616106; Tue, 16 Jul 2019 06:26:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563283616; cv=none; d=google.com; s=arc-20160816; b=grWVVXGVrzf+ghdpjHfeODFpnL+m38BFExqrdksR6bV72VmvPyXUXQVb3siPTJ8hoS UKYbparggwV9JCj17PdfoGrxTwuF70IlfAr/ADI+Xp+A9G3493iLmLBKJeGG39k0Pwg9 UBoiOuVzxMzBlNxxOSK/tLNRoVe+45P+tu5pZMCd2BBhgv/97XomSQu3NqjCXhdFR/YN 2+S0zhqudnhoGnGw9uo44iSFq+xi76rfVMlCzmRodQAEhLtY0amFVYZLeO8J1HjVD9Vh lz9GQHxhYahe8CZfEtNjCa4JG+BairsIrbEqG61EkG6CuLi9C0xp7OKz2nLjhcYXvKvZ H47w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2FLMt1biejUADnaZS+D7n9SrnPK40F0jXQ5om7TEM0E=; b=La9YnIKV7T406ECozfVE8YbGW4ggcDgN9NEItUqvz8j916Uz1COTLtQAblOdExl1xL AF29gjYxdHTHJe4f2ZSjxEEdwaL9wRA4kD8oEk6tUfl+6rFScmCFTi0RQ5gnfUScyuw8 6UlNZDVb9ox7SfdEHTo8v1TsDyTgtIYzFVTdEMaL22c5h0szQ7gdDILf9Wdyyw/x8I4y ibGmcE62ogXEjhT+cIfKqOI2maYayXV633o/SQqp8yACoUx/sd+dGtdgfd/8w2Djx3pH g8ZI8fRsmAxVLLqUCPql4aLpTLMrSxpbFOtn4TVGnHYiQKbwhJoAKO+fdgcW84xr0uK3 bNpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R46pC3b7; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id bc5sor25078703plb.36.2019.07.16.06.26.56 for (Google Transport Security); Tue, 16 Jul 2019 06:26:56 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R46pC3b7; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2FLMt1biejUADnaZS+D7n9SrnPK40F0jXQ5om7TEM0E=; b=R46pC3b7xctfYiDbpcjsza4HTmNx/760I2ZaF5EDledhg5vHet72yrOirw34e4/Hwj a9XoTMetn8gr04TAIhE2LlWxSqPKgOVdS86OM+fcojB7GI2ad77E+i1jpeqKV1+2cEXe 2cnBZvd8RrCgjlzvvKJMItIGEgUzHUMwqJ08iwCMu+urNd/gsaxKdt4ji6oD/yqJ9OIH iA7wIo+ojN1xpialB5aVNj6g/NfG5lUi4MGaFRthczcja3mE2vVQnE/lPC/wRJoliTKR cdRyNH5A1gPD9fXBd/AjwTF+98c6JPOq0uOF0Ht1KinYxfj2YU02TOo6Oqg1Dy5trgYY UGYg== X-Google-Smtp-Source: APXvYqwLdB8Cef6LQ3pqCCQ0P7hsVjjgWF54ZHRKloxBfozTuaD/IY5JZ7PR7zFLuVVRnsib5VBBgA== X-Received: by 2002:a17:902:7448:: with SMTP id e8mr35514887plt.85.1563283615828; Tue, 16 Jul 2019 06:26:55 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:bf0:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id q1sm21472311pfg.84.2019.07.16.06.26.48 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 16 Jul 2019 06:26:55 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org, willy@infradead.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Pengfei Li Subject: [PATCH v5 2/2] mm/vmalloc: modify struct vmap_area to reduce its size Date: Tue, 16 Jul 2019 21:26:04 +0800 Message-Id: <20190716132604.28289-3-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190716132604.28289-1-lpf.vector@gmail.com> References: <20190716132604.28289-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Objective --------- The current implementation of struct vmap_area wasted space. After applying this commit, sizeof(struct vmap_area) has been reduced from 11 words to 8 words. Description ----------- 1) Pack "subtree_max_size", "vm" and "purge_list". This is no problem because A) "subtree_max_size" is only used when vmap_area is in "free" tree B) "vm" is only used when vmap_area is in "busy" tree C) "purge_list" is only used when vmap_area is in vmap_purge_list 2) Eliminate "flags". Since only one flag VM_VM_AREA is being used, and the same thing can be done by judging whether "vm" is NULL, then the "flags" can be eliminated. Signed-off-by: Pengfei Li Suggested-by: Uladzislau Rezki (Sony) --- include/linux/vmalloc.h | 20 +++++++++++++------- mm/vmalloc.c | 24 ++++++++++-------------- 2 files changed, 23 insertions(+), 21 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 9b21d0047710..a1334bd18ef1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -51,15 +51,21 @@ struct vmap_area { unsigned long va_start; unsigned long va_end; - /* - * Largest available free size in subtree. - */ - unsigned long subtree_max_size; - unsigned long flags; struct rb_node rb_node; /* address sorted rbtree */ struct list_head list; /* address sorted list */ - struct llist_node purge_list; /* "lazy purge" list */ - struct vm_struct *vm; + + /* + * The following three variables can be packed, because + * a vmap_area object is always one of the three states: + * 1) in "free" tree (root is vmap_area_root) + * 2) in "busy" tree (root is free_vmap_area_root) + * 3) in purge list (head is vmap_purge_list) + */ + union { + unsigned long subtree_max_size; /* in "free" tree */ + struct vm_struct *vm; /* in "busy" tree */ + struct llist_node purge_list; /* in purge list */ + }; }; /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 71d8040a8a0b..39bf9cf4175a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); /* Export for kexec only */ @@ -1115,7 +1114,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; - va->flags = 0; + va->vm = NULL; insert_vmap_area(va, &vmap_area_root, &vmap_area_list); spin_unlock(&vmap_area_lock); @@ -1922,7 +1921,6 @@ void __init vmalloc_init(void) if (WARN_ON_ONCE(!va)) continue; - va->flags = VM_VM_AREA; va->va_start = (unsigned long)tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; @@ -2020,7 +2018,6 @@ static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, vm->size = va->va_end - va->va_start; vm->caller = caller; va->vm = vm; - va->flags |= VM_VM_AREA; spin_unlock(&vmap_area_lock); } @@ -2125,10 +2122,10 @@ struct vm_struct *find_vm_area(const void *addr) struct vmap_area *va; va = find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) - return va->vm; + if (!va) + return NULL; - return NULL; + return va->vm; } /** @@ -2149,11 +2146,10 @@ struct vm_struct *remove_vm_area(const void *addr) spin_lock(&vmap_area_lock); va = __find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) { + if (va && va->vm) { struct vm_struct *vm = va->vm; va->vm = NULL; - va->flags &= ~VM_VM_AREA; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2856,7 +2852,7 @@ long vread(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -2936,7 +2932,7 @@ long vwrite(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -3466,10 +3462,10 @@ static int s_show(struct seq_file *m, void *p) va = list_entry(p, struct vmap_area, list); /* - * s_show can encounter race with remove_vm_area, !VM_VM_AREA on - * behalf of vmap area is being tear down or vm_map_ram allocation. + * If !va->vm then this vmap_area object is allocated + * by vm_map_ram. */ - if (!(va->flags & VM_VM_AREA)) { + if (!va->vm) { seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, va->va_end - va->va_start);