From patchwork Tue Jul 16 15:26:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11046329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 848FB112C for ; Tue, 16 Jul 2019 15:27:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 711822856B for ; Tue, 16 Jul 2019 15:27:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 62FBB285F1; Tue, 16 Jul 2019 15:27:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3FED2856B for ; Tue, 16 Jul 2019 15:27:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7B408E000A; Tue, 16 Jul 2019 11:27:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E05948E0006; Tue, 16 Jul 2019 11:27:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C31EE8E000A; Tue, 16 Jul 2019 11:27:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 88D5C8E0006 for ; Tue, 16 Jul 2019 11:27:27 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id n23so986690pgf.18 for ; Tue, 16 Jul 2019 08:27:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=J0mSbebrDB8NU9UqfAlSyM8IRaHVhdxLIyvqH6OqxoDSGuQW197+dgaQAzbTFuZcjU 94Wb9rTpV6uvqfoLca85ipl9aQmxF9UfSQdAoG43WopJZMLtP3RVY0WsH5aCParh6YVq qYYR8aK810mx5NRg4aXEjvRJaKPp7+ffdRMjyPva3H0mdlLchCg7/E9lsA5orkS5CHnw hukrUOd4fF1GvrNJWarsfVtgU4rMG8ivz6SUuCZtLx+ykzXFAFPxnQFvWgDux2UL3vDI RVdzi/cPOTSkMDJXEfRpaPbKU9edpVzJ4E0rosmc5RTP5ZdSpFr8drZ/eUpUroVTxq6b Bixg== X-Gm-Message-State: APjAAAUh16AFN4BS33KkLZ/dgln15qQvMMeQXvp0ScwFKSPJajMnl8vS rrnvhs0t/fN6YJ2tEEq1L7ybJhB2OjzStmj/Ot65HfQhStDeNEnigoNImok3E+XTMTMS++pxwGT Jo0Wu5FINvggwpTQAmBt2amjLTKq5DrEcxSU+mx5iL6JreWHemh4E61sRgNzNOPQkUQ== X-Received: by 2002:a17:90a:3544:: with SMTP id q62mr37728488pjb.53.1563290847079; Tue, 16 Jul 2019 08:27:27 -0700 (PDT) X-Received: by 2002:a17:90a:3544:: with SMTP id q62mr37728296pjb.53.1563290845696; Tue, 16 Jul 2019 08:27:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563290845; cv=none; d=google.com; s=arc-20160816; b=dTK2UCvSd6Q/nk+pk2leRS60ejfzNcIu0Shn3oYrUXK9hQNRTBaLs55/E/2Wb/5YTM xDuh+rCDUXbB4bD2QO+9tobnh00ikpBF1XdWaJwD0ZegWbvHqTu4z6tuwiy+n/1eMFaj HoKIXAkQACCsXrmdFLI1MAXbdX/tTbbxmimezli10A51+DP2i4aLQc8lc8sZBhHHi0Xo 5PtIKO/x3qNuahW9UEyT4KBQamIzwHha7aTAPQWHoIclUCQultJ7171aD2ahd9IxVL3T mNS05JYaLAW9xTJg8+htvZqtal+v0PqOPuxjwxSp6Oq4McXEmRPImDUO+uxa0iXYYAVr 7ECg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=cwRGMAtyI2DjhfrmtcNnS0kbQnBTSOJkLoWTqM9fB0wFi1ywDm4iN1/l6L7GHFxNn7 KPkLf3gKZvRX97/rj4EL1Jmch0jxbNFXUDtWwaavKrYwnybxCrJqQeANw2/0wc2tHfw4 vWWLAZPj6G6CQWPb1oI9xTBw3Eol8PQbzF8FkMXELknHqDCrY+I6u+/Adbvh+hWIKvdv 8sfWBoHfYcmufQdj6pZtTc0qhnevmLGbQlgiiOQTlvy2Sd9yv0W1GcMGciDsKJqrB33d ckcLuSy2hKjHFKPBDf4NzMxhCH/noPymR+BBQxhTmDEuC2Enb/RelgBjSD6HwYG/703G sEuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=glco4Syo; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z68sor11118424pgz.39.2019.07.16.08.27.25 for (Google Transport Security); Tue, 16 Jul 2019 08:27:25 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=glco4Syo; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vzHJ6zZC4j+G0RNaMkru3klGJtcfV37NNKvE8Og+hcc=; b=glco4SyouK9YNe9AOX80l/zZ96PB6sLsVT7aTl00fxtU1t6+DtddCpju1qN1GnakWN NrYJNLnKtAlI9BEqqjGZVVE7cTPHBd5l0frW7/DahVt1aVd40V4I3WZiaJ2xkwskuItH EH7GRA7bCHZI87LNQUK3NJXg7QDcCqnXqdxUG1uTo2JG+e1J7U0wLOaqMLy1LmuJIZZp RbjfH+VpPQIyWfGoyJs4s7lOzIrPRqKFHqTXJeJE3VTajMtsKIdvdM8OqrecYzL9RzXn LZym62jEYpAdDFoTY+e75Ca5UoDvQWrQ37rkBpVMIiO5bOwM1GYgCAgXuneXVgxbFwAM GPyg== X-Google-Smtp-Source: APXvYqwg+FuF9Z+ryq1LlKWCFuSwx1x2eeoS4E8Y1ZmKIG10l37NCWviEOWMWa8L2sOVma1B98CQOQ== X-Received: by 2002:a65:5202:: with SMTP id o2mr13584554pgp.29.1563290845380; Tue, 16 Jul 2019 08:27:25 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:bf0:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id h9sm27453651pgk.10.2019.07.16.08.27.17 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 16 Jul 2019 08:27:25 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org, willy@infradead.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 1/2] mm/vmalloc: do not keep unpurged areas in the busy tree Date: Tue, 16 Jul 2019 23:26:55 +0800 Message-Id: <20190716152656.12255-2-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190716152656.12255-1-lpf.vector@gmail.com> References: <20190716152656.12255-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "Uladzislau Rezki (Sony)" The busy tree can be quite big, even though the area is freed or unmapped it still stays there until "purge" logic removes it. 1) Optimize and reduce the size of "busy" tree by removing a node from it right away as soon as user triggers free paths. It is possible to do so, because the allocation is done using another augmented tree. The vmalloc test driver shows the difference, for example the "fix_size_alloc_test" is ~11% better comparing with default configuration: sudo ./test_vmalloc.sh performance Summary: fix_size_alloc_test loops: 1000000 avg: 993985 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973554 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12617652 usec Summary: fix_size_alloc_test loops: 1000000 avg: 882263 usec Summary: full_fit_alloc_test loops: 1000000 avg: 973407 usec Summary: long_busy_list_alloc_test loops: 1000000 avg: 12593929 usec 2) Since the busy tree now contains allocated areas only and does not interfere with lazily free nodes, introduce the new function show_purge_info() that dumps "unpurged" areas that is propagated through "/proc/vmallocinfo". 3) Eliminate VM_LAZY_FREE flag. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Pengfei Li --- mm/vmalloc.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4fa8d84599b0..71d8040a8a0b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_LAZY_FREE 0x02 #define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); @@ -1276,7 +1275,14 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) llist_for_each_entry_safe(va, n_va, valist, purge_list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; - __free_vmap_area(va); + /* + * Finally insert or merge lazily-freed area. It is + * detached and there is no need to "unlink" it from + * anything. + */ + merge_or_add_vmap_area(va, + &free_vmap_area_root, &free_vmap_area_list); + atomic_long_sub(nr, &vmap_lazy_nr); if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) @@ -1318,6 +1324,10 @@ static void free_vmap_area_noflush(struct vmap_area *va) { unsigned long nr_lazy; + spin_lock(&vmap_area_lock); + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr); @@ -2137,14 +2147,13 @@ struct vm_struct *remove_vm_area(const void *addr) might_sleep(); - va = find_vmap_area((unsigned long)addr); + spin_lock(&vmap_area_lock); + va = __find_vmap_area((unsigned long)addr); if (va && va->flags & VM_VM_AREA) { struct vm_struct *vm = va->vm; - spin_lock(&vmap_area_lock); va->vm = NULL; va->flags &= ~VM_VM_AREA; - va->flags |= VM_LAZY_FREE; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2152,6 +2161,8 @@ struct vm_struct *remove_vm_area(const void *addr) return vm; } + + spin_unlock(&vmap_area_lock); return NULL; } @@ -3431,6 +3442,22 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) } } +static void show_purge_info(struct seq_file *m) +{ + struct llist_node *head; + struct vmap_area *va; + + head = READ_ONCE(vmap_purge_list.first); + if (head == NULL) + return; + + llist_for_each_entry(va, head, purge_list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +} + static int s_show(struct seq_file *m, void *p) { struct vmap_area *va; @@ -3443,10 +3470,9 @@ static int s_show(struct seq_file *m, void *p) * behalf of vmap area is being tear down or vm_map_ram allocation. */ if (!(va->flags & VM_VM_AREA)) { - seq_printf(m, "0x%pK-0x%pK %7ld %s\n", + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start, - va->flags & VM_LAZY_FREE ? "unpurged vm_area" : "vm_map_ram"); + va->va_end - va->va_start); return 0; } @@ -3482,6 +3508,16 @@ static int s_show(struct seq_file *m, void *p) show_numa_info(m, v); seq_putc(m, '\n'); + + /* + * As a final step, dump "unpurged" areas. Note, + * that entire "/proc/vmallocinfo" output will not + * be address sorted, because the purge list is not + * sorted. + */ + if (list_is_last(&va->list, &vmap_area_list)) + show_purge_info(m); + return 0; } From patchwork Tue Jul 16 15:26:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11046331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3627C1395 for ; Tue, 16 Jul 2019 15:27:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24E472856B for ; Tue, 16 Jul 2019 15:27:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 18FBF285F1; Tue, 16 Jul 2019 15:27:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8BB1E2856B for ; Tue, 16 Jul 2019 15:27:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94CF58E000B; Tue, 16 Jul 2019 11:27:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8D5738E0006; Tue, 16 Jul 2019 11:27:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 750A98E000B; Tue, 16 Jul 2019 11:27:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 3A4B58E0006 for ; Tue, 16 Jul 2019 11:27:36 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id j22so12538497pfe.11 for ; Tue, 16 Jul 2019 08:27:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bgQ4RhrQ8SIWyyg74Fp++hfL4YmAINorli25JMx+HLI=; b=gQ4/vJ3aGkVtkK4e9qYHuMTbt4zMnP+EU4uWH5LKi3g8yVR3PvcyY+Eeo2uOlO0bVb 7nqPXp0UseknkLeaMxRntad94V+RUXb/fSrmIk+FsYPJ+GYJNUsiDkzy+S/ey0TPsoM7 WLbA5Tex7ULnUeJdH0+2YgkrHAjvl+71YhVGGuXYliMe8Xz/B0jeensm8J6TOouI8qzE +BSbPpC1tebZ3OtIuPJS7d3GWf0z+rc3taUoTqCoXl+O36Qi37011c2C5QXm7dJDBQn9 hPcDu2jB6cxiWVDlyR3bRu4UFvZxNwRqr74RZ8rVYSYoO5yAyk060Y30xzH9aPRNaMve w90Q== X-Gm-Message-State: APjAAAUXgoHG7lBMzIixdTWlApi4ZsqCatIDiT+2Ofl2qQHM9neYfG07 6SciEkVIRFiT83+28UbrXYzOzwOooLjympl4Kq3Po8FZvT5ZArY7LlYxp0aBsSM1F5QHZWiNpBH 4Y9BKSL7uhVd6bqKKuihcMQD5V1VWMA12BEBiHIZ5+GPTeTkQeitx9fwF4iCRqHXuAQ== X-Received: by 2002:a17:902:1129:: with SMTP id d38mr36823030pla.220.1563290855769; Tue, 16 Jul 2019 08:27:35 -0700 (PDT) X-Received: by 2002:a17:902:1129:: with SMTP id d38mr36822918pla.220.1563290854943; Tue, 16 Jul 2019 08:27:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563290854; cv=none; d=google.com; s=arc-20160816; b=jIP9DzinNnXmencvxdiP1O0RvRisQf39/PNt5/SYlLURz6/t86ld00pxOHGtk0S3fr AL/BG/eujreClvudsBzlvhNnhQFEVwQfrPaXIaP9THkAoMGHg3NMSgaDfZvkCaXTXKgE WGCze9nLff2HU3orNoOrv8YfwpfTgbcAtw6RIVutYyMSke52MCII5tRUETft8IaP1l+W Rsa0TRcglA01G88DksqtGZtBC4j61/fNeqYP0+1SaeA2QnbeD8/6FT5/Yk1Wl5ajQA3Y mYADGeiGzdQqyIiaTm2G7ApQwgK51jMCCoWdggLkm3eT5lPptv3p53VPPo3FmPSB/Xi/ /hkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bgQ4RhrQ8SIWyyg74Fp++hfL4YmAINorli25JMx+HLI=; b=GQGomeTifCATWKQffeMxtpkoKiq8zS1DaIwWMvS4/+rsMD7xAwDN1fSDqMXRdYr3K9 UArZ0JUtqVSKSjrYSZrxworW5/foKUyFE+D+bwfGGHsvMuQeUO5OgrOZVS70ngvrowrW AmhyX7J7Qjry8ebu+df954d29/V8b024Jp22s3GhmS/9NJeXDl5HeB00u48lvNFtdS3k 2zWJuDkzY6Rh8GKd8n5fIXLCWtEBfSdYEpPKRFvGogvIg29eW/seqomUj0vv8mvh+Edy GJ2A9lPSgKgRmCKh5xO4QEbrfu49uRMV/ee3twfpHKaWP7zOKauQ19wU887Siao2UUBz lXEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="nUXSP/Uj"; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h29sor11023530pgb.21.2019.07.16.08.27.34 for (Google Transport Security); Tue, 16 Jul 2019 08:27:34 -0700 (PDT) Received-SPF: pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="nUXSP/Uj"; spf=pass (google.com: domain of lpf.vector@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=lpf.vector@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bgQ4RhrQ8SIWyyg74Fp++hfL4YmAINorli25JMx+HLI=; b=nUXSP/UjVBYnWfyc4jlTs17JCc4taGD4zuIrEeMGCF2WE+YIc+d9FCDUzwatPo11yk HqC2h0chM/jepzjVBnURfdXKh25VLjxtjN/1Tz5ItpxKTcc9RlV2PUirx3R4MZ9atRtb jyoXVRhJYBvPOEsGIB+nAUr7ZtpzCFFjGenYYTgHkBxW9uDiT3Dlx7b4aRQe0wp41XBY OuKAUTgog5do0B51QJf+rsa0Ov9x9yXWySU4JP0CceKvQ6e9ynLGij7c7rE5pru5YcFI pSbKRZWDZtwkjTvmMa0g1ddOEPzMaYf4yutzX5TuXocttkvGc1syuWblapRkwporyYFy 6FqA== X-Google-Smtp-Source: APXvYqxqz7gSTFnrIhbMahLYP5IFtDrisUyyl7qIFt4erygqkCZSe869LC+F7nOwyncv1f9NyEyjzg== X-Received: by 2002:a63:490a:: with SMTP id w10mr34021506pga.6.1563290854559; Tue, 16 Jul 2019 08:27:34 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:bf0:b8c3:8577:bf2f:2]) by smtp.gmail.com with ESMTPSA id h9sm27453651pgk.10.2019.07.16.08.27.25 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 16 Jul 2019 08:27:34 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org, willy@infradead.org Cc: urezki@gmail.com, rpenyaev@suse.de, peterz@infradead.org, guro@fb.com, rick.p.edgecombe@intel.com, rppt@linux.ibm.com, aryabinin@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Pengfei Li Subject: [PATCH v6 2/2] mm/vmalloc: modify struct vmap_area to reduce its size Date: Tue, 16 Jul 2019 23:26:56 +0800 Message-Id: <20190716152656.12255-3-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190716152656.12255-1-lpf.vector@gmail.com> References: <20190716152656.12255-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Objective --------- The current implementation of struct vmap_area wasted space. After applying this commit, sizeof(struct vmap_area) has been reduced from 11 words to 8 words. Description ----------- 1) Pack "subtree_max_size", "vm" and "purge_list". This is no problem because A) "subtree_max_size" is only used when vmap_area is in "free" tree B) "vm" is only used when vmap_area is in "busy" tree C) "purge_list" is only used when vmap_area is in vmap_purge_list 2) Eliminate "flags". Since only one flag VM_VM_AREA is being used, and the same thing can be done by judging whether "vm" is NULL, then the "flags" can be eliminated. Signed-off-by: Pengfei Li Suggested-by: Uladzislau Rezki (Sony) Reviewed-by: Uladzislau Rezki (Sony) --- include/linux/vmalloc.h | 20 +++++++++++++------- mm/vmalloc.c | 24 ++++++++++-------------- 2 files changed, 23 insertions(+), 21 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 9b21d0047710..a1334bd18ef1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -51,15 +51,21 @@ struct vmap_area { unsigned long va_start; unsigned long va_end; - /* - * Largest available free size in subtree. - */ - unsigned long subtree_max_size; - unsigned long flags; struct rb_node rb_node; /* address sorted rbtree */ struct list_head list; /* address sorted list */ - struct llist_node purge_list; /* "lazy purge" list */ - struct vm_struct *vm; + + /* + * The following three variables can be packed, because + * a vmap_area object is always one of the three states: + * 1) in "free" tree (root is vmap_area_root) + * 2) in "busy" tree (root is free_vmap_area_root) + * 3) in purge list (head is vmap_purge_list) + */ + union { + unsigned long subtree_max_size; /* in "free" tree */ + struct vm_struct *vm; /* in "busy" tree */ + struct llist_node purge_list; /* in purge list */ + }; }; /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 71d8040a8a0b..2f7edc0466e7 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -329,7 +329,6 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -#define VM_VM_AREA 0x04 static DEFINE_SPINLOCK(vmap_area_lock); /* Export for kexec only */ @@ -1115,7 +1114,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; - va->flags = 0; + va->vm = NULL; insert_vmap_area(va, &vmap_area_root, &vmap_area_list); spin_unlock(&vmap_area_lock); @@ -1922,7 +1921,6 @@ void __init vmalloc_init(void) if (WARN_ON_ONCE(!va)) continue; - va->flags = VM_VM_AREA; va->va_start = (unsigned long)tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; @@ -2020,7 +2018,6 @@ static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, vm->size = va->va_end - va->va_start; vm->caller = caller; va->vm = vm; - va->flags |= VM_VM_AREA; spin_unlock(&vmap_area_lock); } @@ -2125,10 +2122,10 @@ struct vm_struct *find_vm_area(const void *addr) struct vmap_area *va; va = find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) - return va->vm; + if (!va) + return NULL; - return NULL; + return va->vm; } /** @@ -2149,11 +2146,10 @@ struct vm_struct *remove_vm_area(const void *addr) spin_lock(&vmap_area_lock); va = __find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) { + if (va && va->vm) { struct vm_struct *vm = va->vm; va->vm = NULL; - va->flags &= ~VM_VM_AREA; spin_unlock(&vmap_area_lock); kasan_free_shadow(vm); @@ -2856,7 +2852,7 @@ long vread(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -2936,7 +2932,7 @@ long vwrite(char *buf, char *addr, unsigned long count) if (!count) break; - if (!(va->flags & VM_VM_AREA)) + if (!va->vm) continue; vm = va->vm; @@ -3466,10 +3462,10 @@ static int s_show(struct seq_file *m, void *p) va = list_entry(p, struct vmap_area, list); /* - * s_show can encounter race with remove_vm_area, !VM_VM_AREA on - * behalf of vmap area is being tear down or vm_map_ram allocation. + * s_show can encounter race with remove_vm_area, !vm on behalf + * of vmap area is being tear down or vm_map_ram allocation. */ - if (!(va->flags & VM_VM_AREA)) { + if (!va->vm) { seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", (void *)va->va_start, (void *)va->va_end, va->va_end - va->va_start);