From patchwork Wed May 22 15:09:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10956105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4DDC6C5 for ; Wed, 22 May 2019 15:09:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2B4328C08 for ; Wed, 22 May 2019 15:09:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A715428C0D; Wed, 22 May 2019 15:09:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3385A28C08 for ; Wed, 22 May 2019 15:09:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CF3E6B0005; Wed, 22 May 2019 11:09:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 180F06B0006; Wed, 22 May 2019 11:09:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06FBD6B0007; Wed, 22 May 2019 11:09:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by kanga.kvack.org (Postfix) with ESMTP id 900756B0005 for ; Wed, 22 May 2019 11:09:52 -0400 (EDT) Received: by mail-lj1-f200.google.com with SMTP id w18so457373ljw.8 for ; Wed, 22 May 2019 08:09:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=cGTV7bQqwG5x9oiYauaYTWncIqmQlZcM9EHf5YH/t2wXnPTLXIq0I0xSoI2XRGoAS/ z2XB61jNfHKBrTs2fUthH7eDi0PFS26webRU+qei+zbW9XHRhnRb3tc1ca/gL/ryb3Ku NpLCoN6DTc2GfEbtsY5zeSLnnlMxBjKBhrQpqVyXvp90HOvwxZhy4soEcyjbz1Uv06ul SSVlcztZvbhdAs3PT0btcqvCC1zUVZ+X84CWzX/nHCL+7wh3GVE2Bk7inX0QkdV9GZ0B 7sQozclzc2hIBuRkY1RDr8iiQhurWy0eWv7XnjInC/MptsfzA+nEB7f/aderJYx8Nxrj C35w== X-Gm-Message-State: APjAAAUdaEqeB1z9w9JgxoZfkeM8ASmdzr4ndiSfVQSkkPcWc75uFAA7 hU2wGd1kvhcyGT6oNbEt4iyNELOgzgSL8kBGuWT343erM1/kznp7Zbq5RLi3ROwIrTUnGxN7ntI bcI0iijBEKVRVzMxyUrpdnRMVYVaDA8HvEOOcIxeIgi714ooMgSDtvR/cYWOjY0YDSQ== X-Received: by 2002:a2e:9b0c:: with SMTP id u12mr15208028lji.189.1558537792015; Wed, 22 May 2019 08:09:52 -0700 (PDT) X-Received: by 2002:a2e:9b0c:: with SMTP id u12mr15207973lji.189.1558537790831; Wed, 22 May 2019 08:09:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558537790; cv=none; d=google.com; s=arc-20160816; b=D0j3G9N+SuWyixydrl6kfMQxQjR1hWP8kwgfb6BKmU++h8aI+4AJ05Uiv68eoteKmq Ic56XR25264W+7BsaSy88vHU3V/czz/PgjOKEn5cEwUE0fEEKRjTLng4HLttdZ6ojl+/ 5uh4oyfFBFn0uuZYDUFAALw2bECdQigfb53eNqK8BUYdikliiU/gdf2zZzx69MS7Ch1e CU8Bsba26jFADcNXN/locE7wTVcHCKfvjpKeQAxp2kKw/R/CtJfEVA5QFHBcsDcKzXt/ tEE0U4yyp3vRG36SS/ZlNDFZAhs27idvYsqsV2O50nrkzj1qCHTuJCq2Fo1SDlmFpjJc hrCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from:dkim-signature; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=RJl2BqcHugpY0FwAMdnEFFwUe1BTn5uOzp4LdnX52aM0VqaQZrOue1AIsl24zlkFxx i7HSQ2b0GFwwFoBqoRhQa20V6ZTB+ZcjG5H4cz80YUKQIWKe7HtEy9+FIKNQTDfVhGWR z4hOf3R5s3MU5M4hMOLI78KLMnPKBDtJ6/3E0cvdv5CHwv3l1/sLo2IeDCLwtQ01Sg8H ZHxT3D/4GGATUY1TLgxFhNOBfuMP+sYCOeiECHY1axUoNEyuUfAje1GbrNPz7viBoQqP NGO5r3ZdYcBIkqa6g4vHqZ2mLSffoTsdVKCoIS8LLwg+dPqyN40/bV8uJg0g8eqE24Oi o8LQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=uchLVR9y; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h4sor7152925lfp.2.2019.05.22.08.09.50 for (Google Transport Security); Wed, 22 May 2019 08:09:50 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=uchLVR9y; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=uchLVR9yDZbCoZFcLTDWBlnr+qGgHM4l53knuFUtS4cszF3qq07ggcXLTPp4f1w3EH j1IHk6vbdley5IzodGb/tQoWS8N0fMqt6Ua8kNPLmWewxcufW+1xVNb0e7fViD9CgcIp pJhh60QlEvljx7FQwox0+5OIj33+h7stGNT/5aqWO1weU6FRfCFZdHnUWEfTxAC5OMij t74E5/Or9Z4xug+txof7zAQd6NGIvQn4vfDdCBnBtueDJS3roYMhnTqSiVT58a1ZFUXz 1chdk61oyigEWPE2i+mrhDPq+UCbJ9K7zrek8mWwqK7o2ecTG/tzzZbzagqaoi1i97aJ BgJw== X-Google-Smtp-Source: APXvYqx5GGN3+wrZiJyukNG7wlTWckT2VbaD042N799w2UxWzlHPswK9PIYTsPn1Y77SbReZyTnRUQ== X-Received: by 2002:ac2:5bc1:: with SMTP id u1mr41492644lfn.111.1558537790380; Wed, 22 May 2019 08:09:50 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id t22sm5303615lje.58.2019.05.22.08.09.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 May 2019 08:09:48 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: Roman Gushchin , Uladzislau Rezki , Michal Hocko , Matthew Wilcox , linux-mm@kvack.org, LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH 1/4] mm/vmap: remove "node" argument Date: Wed, 22 May 2019 17:09:36 +0200 Message-Id: <20190522150939.24605-1-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Remove unused argument from the __alloc_vmap_area() function. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index c42872ed82ac..ea1b65fac599 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -985,7 +985,7 @@ adjust_va_to_fit_type(struct vmap_area *va, */ static __always_inline unsigned long __alloc_vmap_area(unsigned long size, unsigned long align, - unsigned long vstart, unsigned long vend, int node) + unsigned long vstart, unsigned long vend) { unsigned long nva_start_addr; struct vmap_area *va; @@ -1062,7 +1062,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, * If an allocation fails, the "vend" address is * returned. Therefore trigger the overflow path. */ - addr = __alloc_vmap_area(size, align, vstart, vend, node); + addr = __alloc_vmap_area(size, align, vstart, vend); if (unlikely(addr == vend)) goto overflow; From patchwork Wed May 22 15:09:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10956107 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B5BFC6C5 for ; Wed, 22 May 2019 15:09:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A68B5200DF for ; Wed, 22 May 2019 15:09:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 999B42894E; Wed, 22 May 2019 15:09:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DA9A28B82 for ; Wed, 22 May 2019 15:09:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE3CB6B0006; Wed, 22 May 2019 11:09:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B45A66B0007; Wed, 22 May 2019 11:09:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0C306B0008; Wed, 22 May 2019 11:09:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by kanga.kvack.org (Postfix) with ESMTP id 3552A6B0006 for ; Wed, 22 May 2019 11:09:54 -0400 (EDT) Received: by mail-lf1-f72.google.com with SMTP id l17so354280lfp.0 for ; Wed, 22 May 2019 08:09:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=XS/exrL2qtsb1wQlcnKifMlE835u1lj3Ys1udy3P5H4=; b=fWV84bh1jSDT569ShOH4+fBo2gLEdpRBBlpePov88NTSIWeFmTbPqlRlm9WT2Eb4g7 /+ZXOKXq50YyUPp9g7RdhgJgwDLjxbh+grQpxphYXVa6tU4gfLvEGv+Z+xR8p354OeVd +6x0O2UfMQsEKoQ53U9dNSf4g5Q2gkqq/dig354vZ2gdCplHpwD7XSMYXHfKOVL8dInC Tw2cdmyY8k+g1e9VMmi3uRRy5yPkjNZqnpTF4Ub0fNr8uOlBLnRpEQ2R/vs1PPA3wgQ/ X9LoMzmQCpujbL3+yipqyUOFZovcm8dKA7PaLhgBOIz56JAyEAnPON/RbU0u5CJGVkek WQcA== X-Gm-Message-State: APjAAAV2EzTceVWEEPX93YE4TB7T5qDid6xXFeXZS/JXJogML/MzcgRd U1orHpO6vSAudHACtgYIwfuTA65iVd990gm+aoqSYjv2/qtoPLda/aW9H8VClyUkPoeCUjiAWAv PnBOiH5zgS0LzHVecVyibn3lwiH8J2YWBUsZsPIiPIBxLyOeZHQgPyRtPsmEmBa4oWw== X-Received: by 2002:a19:3f4b:: with SMTP id m72mr43674975lfa.32.1558537793606; Wed, 22 May 2019 08:09:53 -0700 (PDT) X-Received: by 2002:a19:3f4b:: with SMTP id m72mr43674898lfa.32.1558537792100; Wed, 22 May 2019 08:09:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558537792; cv=none; d=google.com; s=arc-20160816; b=0kW9tcZbFgu4DB0ISO4ERzEimmk5pzK1GzyGFzjjlVvAYLTrHwsw9UmD55Qf8RDyb8 2EIZnSb+cnbPj9a/O/zmR3FfyrfHQ1Z3617yhZGl2bniDWM03L4FNZPUffgn5UJjVNH1 VgqJnCvkpr+cGKdyW6sPG1RCovF4MDD8EQGEAU8SvX0VGN43Jf1LHVn3TODgYlxScn/6 cXMgzyiEmV0RD95w/QeclyDcUzQG0jiQSmfc3Egy0WiS3fp5aEhzoee00vR9xxHui7Qn FC0ZLtqC42WE3WaVgX0LwYd4rRT7DUYrjv/XqZ+LQeRjMGxmcjqi8wx1jl+HpxryDv1O 5OLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XS/exrL2qtsb1wQlcnKifMlE835u1lj3Ys1udy3P5H4=; b=IrYpCbh+oGAfakBAPBaFCSj8ysBRiqeC8JGRKLETVOgzxIBx90Dtz/730d/W1bwxQo VAdFQFl5x0eT6vW7exQBbsiOgLlo7ddqPK1tNfu1O4JKNHMafT525sRiW/avrxZhlY38 hXG8nFYmGYF6OdPSvVrZ5X5XtyD1W8/iYqwUOlafeS0/QSPr/j5eNsdBv9zMWSgQLSyr ZlKA/75bgA2Vnrv4CuIMzbetqeBRJNShWAciw8K3O/xk4RgwlHLuYHogRDAwz6tPMzA1 jXwl52qu3AM9P2NycnKN2m0OjU8e5tbtihC1SBv6KyK1uXK8EYRvIEyfCTwfZmBhxU6W CfDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pTaXTTu2; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id w10sor6913729lfn.19.2019.05.22.08.09.51 for (Google Transport Security); Wed, 22 May 2019 08:09:52 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pTaXTTu2; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XS/exrL2qtsb1wQlcnKifMlE835u1lj3Ys1udy3P5H4=; b=pTaXTTu2glGWzCtLk76Cl7B6tZw4mic9YljjZTNWqawSe1IdcLFtJdXry3pTReF+XP jS2hGtc7XmDPY/jxLfqgB2oopz2zMYo1t5ATW0D/YM6xBHS3X1ptj9Atx/I5rdJ6rLAa 1LGecflzmrOTWL7czlq9qcASE9EbnmfDcTVqqQh2vv5J+rcWAkTmlX2bdylKxCRVPqk4 QzOm6Eg6M/ZYU8xeGmcD/JHLadKWQeIu6aySuhyGBXUXFwuGj/nH7Peu3UnfPgs80Xax rSDrs1IFJINGs0egfYiEhho7OP6+0AxPZdIY/2d/E69jBS0ptLrfxDtUWaDRPHfkW1Q2 OmYg== X-Google-Smtp-Source: APXvYqw+RiBtacL9aZIC5RAbtHUdu2UF9erfKWA6sEHVLiTrV/brXIweM3I0dA9/b8+ynfOjSkNLVg== X-Received: by 2002:ac2:4a6e:: with SMTP id q14mr7392786lfp.46.1558537791665; Wed, 22 May 2019 08:09:51 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id t22sm5303615lje.58.2019.05.22.08.09.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 May 2019 08:09:51 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: Roman Gushchin , Uladzislau Rezki , Michal Hocko , Matthew Wilcox , linux-mm@kvack.org, LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH 2/4] mm/vmap: preload a CPU with one object for split purpose Date: Wed, 22 May 2019 17:09:37 +0200 Message-Id: <20190522150939.24605-2-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190522150939.24605-1-urezki@gmail.com> References: <20190522150939.24605-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce ne_fit_preload()/ne_fit_preload_end() functions for preloading one extra vmap_area object to ensure that we have it available when fit type is NE_FIT_TYPE. The preload is done per CPU and with GFP_KERNEL permissive allocation masks, which allow to be more stable under low memory condition and high memory pressure. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 78 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ea1b65fac599..5302e1b79c7b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -364,6 +364,13 @@ static LIST_HEAD(free_vmap_area_list); */ static struct rb_root free_vmap_area_root = RB_ROOT; +/* + * Preload a CPU with one object for "no edge" split case. The + * aim is to get rid of allocations from the atomic context, thus + * to use more permissive allocation masks. + */ +static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); + static __always_inline unsigned long va_size(struct vmap_area *va) { @@ -950,9 +957,24 @@ adjust_va_to_fit_type(struct vmap_area *va, * L V NVA V R * |---|-------|---| */ - lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); - if (unlikely(!lva)) - return -1; + lva = __this_cpu_xchg(ne_fit_preload_node, NULL); + if (unlikely(!lva)) { + /* + * For percpu allocator we do not do any pre-allocation + * and leave it as it is. The reason is it most likely + * never ends up with NE_FIT_TYPE splitting. In case of + * percpu allocations offsets and sizes are aligned to + * fixed align request, i.e. RE_FIT_TYPE and FL_FIT_TYPE + * are its main fitting cases. + * + * There are few exceptions though, as en example it is + * a first allocation(early boot up) when we have "one" + * big free space that has to be split. + */ + lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); + if (!lva) + return -1; + } /* * Build the remainder. @@ -1023,6 +1045,50 @@ __alloc_vmap_area(unsigned long size, unsigned long align, } /* + * Preload this CPU with one extra vmap_area object to ensure + * that we have it available when fit type of free area is + * NE_FIT_TYPE. + * + * The preload is done in non-atomic context thus, it allows us + * to use more permissive allocation masks, therefore to be more + * stable under low memory condition and high memory pressure. + * + * If success, it returns zero with preemption disabled. In case + * of error, (-ENOMEM) is returned with preemption not disabled. + * Note it has to be paired with alloc_vmap_area_preload_end(). + */ +static void +ne_fit_preload(int *preloaded) +{ + preempt_disable(); + + if (!__this_cpu_read(ne_fit_preload_node)) { + struct vmap_area *node; + + preempt_enable(); + node = kmem_cache_alloc(vmap_area_cachep, GFP_KERNEL); + if (node == NULL) { + *preloaded = 0; + return; + } + + preempt_disable(); + + if (__this_cpu_cmpxchg(ne_fit_preload_node, NULL, node)) + kmem_cache_free(vmap_area_cachep, node); + } + + *preloaded = 1; +} + +static void +ne_fit_preload_end(int preloaded) +{ + if (preloaded) + preempt_enable(); +} + +/* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. */ @@ -1034,6 +1100,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, struct vmap_area *va; unsigned long addr; int purged = 0; + int preloaded; BUG_ON(!size); BUG_ON(offset_in_page(size)); @@ -1056,6 +1123,12 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask & GFP_RECLAIM_MASK); retry: + /* + * Even if it fails we do not really care about that. + * Just proceed as it is. "overflow" path will refill + * the cache we allocate from. + */ + ne_fit_preload(&preloaded); spin_lock(&vmap_area_lock); /* @@ -1063,6 +1136,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, * returned. Therefore trigger the overflow path. */ addr = __alloc_vmap_area(size, align, vstart, vend); + ne_fit_preload_end(preloaded); + if (unlikely(addr == vend)) goto overflow; From patchwork Wed May 22 15:09:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10956109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0BE56C5 for ; Wed, 22 May 2019 15:09:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B153D1FF0B for ; Wed, 22 May 2019 15:09:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A591728B82; Wed, 22 May 2019 15:09:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 439F527480 for ; Wed, 22 May 2019 15:09:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B48B26B0007; Wed, 22 May 2019 11:09:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AD2446B0008; Wed, 22 May 2019 11:09:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 973156B000A; Wed, 22 May 2019 11:09:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by kanga.kvack.org (Postfix) with ESMTP id 3078F6B0007 for ; Wed, 22 May 2019 11:09:55 -0400 (EDT) Received: by mail-lf1-f69.google.com with SMTP id t13so510641lfq.8 for ; Wed, 22 May 2019 08:09:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=XfLoVJNG3YH3XU12QY7rmjEpG+NwOFxkxi1NTXn+kzk=; b=ICVBNSgYEnZ0+afROBCRYeSUSmcfWWON1uLfMV4yULBUpvUH1Wd9lrFQvxEjOsdmT6 zvSt52BOC/RCmDhdEaEHPeyyPfF2Noo4rG8k/VsTSNiaePy+lY+ooF8SA+L89xivlTfk h+pcIMeXldsQTttNoyQ9QydujZQiqwS1rXn8bAmQioc3/liuMaxA8Oxm/enves44R7nL bCHPwKLYOgLvrZRZt9M55X8zrwZ9yIvtlKwXA8F0/cY12jlKe4jwDbYOs4+0Caz/Yefl dFuDgCussrM8nyHPKRLIEOOnXord3HBGkESga40GldLZSszZHub17MI4sx5Epzqqa5mK e8ng== X-Gm-Message-State: APjAAAXWy3JRfVbqS7RDHQZq//ZSoN7P9KeoudjDtH1Kkcd/OrwL7eHj trYpjZoHdvZHyLULhDY+30Mnyur0LeolaQlV6kEN+MLrI7o24p6tBOSUgRLxMA8p0q+jEeCpdXe AVC9dW81iZC1NwiIkKjXRkbHxYYgY92q9Oci5zSfNMHxFI08fiAcXp28l9glWHaAQlA== X-Received: by 2002:a2e:85d1:: with SMTP id h17mr29509610ljj.1.1558537794658; Wed, 22 May 2019 08:09:54 -0700 (PDT) X-Received: by 2002:a2e:85d1:: with SMTP id h17mr29509564ljj.1.1558537793640; Wed, 22 May 2019 08:09:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558537793; cv=none; d=google.com; s=arc-20160816; b=eLnWOot6rzX+RbJezEoBML0whcsajeefAHuqOXWU/lqpvAGzzCK48WlhRGEoVIzfEs 3wBEO9fCvsWJo52UJRI21e+r6JM5JydNk5rYuVWlFZQFQan1JtPzZWDQ8pIdyLNB4T4P M41Qsf7VCB2N/dm2TGJnkUVFE9CIzNlr7jt5QmNadUZ8p2uz3Shu29RCdvKylmcrxWKJ I+df3qcSJ+dQn9CSuN4MEVYAUCEYKfI/83pc7qHBmAu55yB5KcSosRxgbLfaeu5jSLQ1 6/TLeNRKH5gkJf3Iuum1/IxM5Zfc1lTf9B5wDn7wYsNsSejatjFt+/Ovo/uyHoMyBsEB j64g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XfLoVJNG3YH3XU12QY7rmjEpG+NwOFxkxi1NTXn+kzk=; b=GYfrRHkDQjDzqnMtQX6hn4S6j1J4p3kWopG4R9WZWmNy8BWrJjwjm9WpWiPdoqK+4p 4fnyEjCpHG6dJYIOGpeJN+xyg83k9AE4UxxgtSDi1wfV/RrP9n1wAYHzVZfIigqId50y ANT2wFekal6o/E7afkiJZf/gjeKBC3ig1KcdFXCZ1Y1sSjcfP/6ZE1KKGsPGxnXorB0c qLfxtfxHGXNWWmYkLmrDcM+6LC/7B19/JQy9lZAwo1Xmr7ibSjizzbK7hCya26RXcW+c LM/XzY1YMz0845D2dR/8wUKbmBtrDu4PUnyKuCDNbhrlS6RmhIKRPzvyzQASAjy0UkCJ t+TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=iLgVs9bc; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id w10sor6913755lfn.19.2019.05.22.08.09.53 for (Google Transport Security); Wed, 22 May 2019 08:09:53 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=iLgVs9bc; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XfLoVJNG3YH3XU12QY7rmjEpG+NwOFxkxi1NTXn+kzk=; b=iLgVs9bciGGSfNr7lnaGuHN02LQSefOJFB8jCzlazwdjUT9zwD+9AiHD7mJH+QgV98 cgQi9kr2nQ2QSQGNirYGrvhwR5h1gGIjtJR3DfVwa61umhpH3Ef94m4SUmibVYaMECiz /F0K3Nj4OGjsuwFwPuGP0eMwTTre0PJrjW0GQB/NCao8IOA7XNg1vXoyyGLOmkBe+aKs F1M/zzYr0TP7k7R33Weoc4rYMBBm+pD3GczPbmGPhLn+g4mBM+UAfGeYRGKAyQg8nXG8 gl25OHegvI+FFXUwNoQTVY9rlqo+XeEykTmOAu97rVBqmr76OxM1M8IzK2DDEn/ITVkC sQAQ== X-Google-Smtp-Source: APXvYqyBoI7CHTOcBFOL7JPrYQSLJpWeJrCJebZpJdPWf/kriTBhBeULhSVwcS66AnUqV8yAuvhRlQ== X-Received: by 2002:ac2:4471:: with SMTP id y17mr17527691lfl.23.1558537793265; Wed, 22 May 2019 08:09:53 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id t22sm5303615lje.58.2019.05.22.08.09.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 May 2019 08:09:52 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: Roman Gushchin , Uladzislau Rezki , Michal Hocko , Matthew Wilcox , linux-mm@kvack.org, LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH 3/4] mm/vmap: get rid of one single unlink_va() when merge Date: Wed, 22 May 2019 17:09:38 +0200 Message-Id: <20190522150939.24605-3-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190522150939.24605-1-urezki@gmail.com> References: <20190522150939.24605-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It does not make sense to try to "unlink" the node that is definitely not linked with a list nor tree. On the first merge step VA just points to the previously disconnected busy area. On the second step, check if the node has been merged and do "unlink" if so, because now it points to an object that must be linked. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5302e1b79c7b..89b8f44e8837 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -718,9 +718,6 @@ merge_or_add_vmap_area(struct vmap_area *va, /* Check and update the tree if needed. */ augment_tree_propagate_from(sibling); - /* Remove this VA, it has been merged. */ - unlink_va(va, root); - /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); @@ -745,12 +742,12 @@ merge_or_add_vmap_area(struct vmap_area *va, /* Check and update the tree if needed. */ augment_tree_propagate_from(sibling); - /* Remove this VA, it has been merged. */ - unlink_va(va, root); + /* Remove this VA, if it has been merged. */ + if (merged) + unlink_va(va, root); /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); - return; } } From patchwork Wed May 22 15:09:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10956111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DBACA13AD for ; Wed, 22 May 2019 15:10:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C95F01FF0B for ; Wed, 22 May 2019 15:10:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCFBB27480; Wed, 22 May 2019 15:10:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E6CF1FF0B for ; Wed, 22 May 2019 15:10:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C8C46B0008; Wed, 22 May 2019 11:09:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5520F6B000A; Wed, 22 May 2019 11:09:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30C5B6B000C; Wed, 22 May 2019 11:09:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by kanga.kvack.org (Postfix) with ESMTP id BB8476B0008 for ; Wed, 22 May 2019 11:09:56 -0400 (EDT) Received: by mail-lj1-f197.google.com with SMTP id g8so455047lja.12 for ; Wed, 22 May 2019 08:09:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=HFExWlr1uO9x+HQMZPCc0rJnteE0HEqVX28pJUGOwfI=; b=pj/brO6BWQu6spQ8MAWXjfiI8wdW3CmpQGKySIUzZHroAZtq9Zr0LORKzqqDecMYsF gn+AJmPmvBsKpehq+j1yYHaH3fOFp8Jov8RQGGcNrzHxgAKBPdynFmfMt81MoKZIcIbq QUD6he/bSY0FTm/v70lpaP9Upqmuw3DnYG9Igx2g1kU+TmGzGkJ+TyCRyUZMvMVgnJBl d1haGu9d8CgwVi8UdJF5fIst3c6KauNHYnZtxCagUj22wjEwsVmn/VqfevLLdL4673vA M/WRI2za4xOl6qBEZzkNJ4WoHCqNBWpCYs7xx5HNY6fDKR7Hd9ryNeQ1M5IiERS+3AE0 zg7Q== X-Gm-Message-State: APjAAAVj4lrH0idctBoJlV3cZadxiBiz1gQNm7uOxKxTKkB0WMWyrbP6 81RcQQhsD5GY7+z3GisPYC7P2tSIc9xUf8iAbswcyt+VeidD/ckuitrBKxkVQOweQR2pnqn3GD8 dLyQQSiMpR8t3GBBRyk62Hxigc6DfDqfwivF88dobTQobBnQPpQ3yeFzyI1F/VlhEow== X-Received: by 2002:a19:2b84:: with SMTP id r126mr43734764lfr.86.1558537796160; Wed, 22 May 2019 08:09:56 -0700 (PDT) X-Received: by 2002:a19:2b84:: with SMTP id r126mr43734721lfr.86.1558537795109; Wed, 22 May 2019 08:09:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558537795; cv=none; d=google.com; s=arc-20160816; b=NzcYdWGfwIKYNL+vyGAfl9B7rzhpM57qIDTUZAJczCcGMB8d0mA8LLWjWAv96igjpc KycRaVOrH/b7R6LUGLEFyVom6esrQz9h6MI37DHpI05jRKSvPZPGsDZY2J8rgm/YbBJo wdxgqJNLOBXG/zaIyHjFXtNtnQQW53K5Zy90+Pqm/6GH5tgnMd/1aA0NUHTUICtUcJP4 pr3VDBgPW/9zevXeVPfq0D8hgDBjV9MNJaQYCIfD7iHxQYL/zYI597ZeJxxSJZBi/07p XdCsCG9oH6GJYu+bTKM54eavBUD+/mR/gOr7yCI/L/UYHWZyE1+BedisgY676NBvb57i NxxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HFExWlr1uO9x+HQMZPCc0rJnteE0HEqVX28pJUGOwfI=; b=OntXidEq/4PKRDlJdKSsi+6R1JE4P38x/bz3B40cTX+yrEfDT9JYS81XYTEKILqByy ouuNkzWj4L3eKy6P87v7zd+2qQlAbiMXphYNB6oDn55g/czlCocPQEA4vL3jT5cYQqhX uqN+ESD74K0KDhALeSSsLU7MbMnGFRq0kHSqrWEEAeOfxcW6qqZ6mHvNGD5ZuYEuC+aj n/EtGALuFU0nSifnDM7VI0XOhQYqR4Mf3b0AyxzJODCknEmD5ODAjt03df77FRSVc34j CWEr2PGA0sCI38SKEJvkQ+cgJBeULJ4jqGU7NrwzfC7d695kzHTvOOV+wtjE+GtAfBQb LhQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ruSnW7Uo; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id f17sor628184ljg.16.2019.05.22.08.09.54 for (Google Transport Security); Wed, 22 May 2019 08:09:55 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ruSnW7Uo; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HFExWlr1uO9x+HQMZPCc0rJnteE0HEqVX28pJUGOwfI=; b=ruSnW7UoKugOt8Wk1U1vu2eypYhGi77d8sGeCoBp0oSFU+eDgdC9MwQtjOXVfQNXuV 6KJdjb33wUXIuVd6jL6/hrMIkIwfRg6u25wtb98ZvjDdRkC8gQ5YNKjlKIERjDCtgBpF YTMccV5Ssgz03q4pn1PtYWTiu9ac7xUwgjKnVrljLksEI+e5xpoedgjRRhXPKAg0jQd7 bphzSaFhRjZD4d1D9guADzeRVcCpyJyuiNn/S7k2cEay/j2r19+H6JHZF1iVQFe7KZYI wIiDn56+llj1MsRwEDWcT2FNWLpU0Kj75jessgu3OTtCThc/G7QGX3XDY0oUaBgCEhVl VxPw== X-Google-Smtp-Source: APXvYqxhdEDHSoE2NHR+WwYFSvC801p+AYV2zVp/M5sMm0EjyGz/P8bAl+KcSdmxXbpFBa/oszd+gQ== X-Received: by 2002:a2e:9d09:: with SMTP id t9mr12001686lji.151.1558537794729; Wed, 22 May 2019 08:09:54 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id t22sm5303615lje.58.2019.05.22.08.09.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 May 2019 08:09:53 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: Roman Gushchin , Uladzislau Rezki , Michal Hocko , Matthew Wilcox , linux-mm@kvack.org, LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH 4/4] mm/vmap: move BUG_ON() check to the unlink_va() Date: Wed, 22 May 2019 17:09:39 +0200 Message-Id: <20190522150939.24605-4-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190522150939.24605-1-urezki@gmail.com> References: <20190522150939.24605-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Move the BUG_ON()/RB_EMPTY_NODE() check under unlink_va() function, it means if an empty node gets freed it is a BUG thus is considered as faulty behaviour. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 89b8f44e8837..47f7e7e83e23 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -533,20 +533,16 @@ link_va(struct vmap_area *va, struct rb_root *root, static __always_inline void unlink_va(struct vmap_area *va, struct rb_root *root) { - /* - * During merging a VA node can be empty, therefore - * not linked with the tree nor list. Just check it. - */ - if (!RB_EMPTY_NODE(&va->rb_node)) { - if (root == &free_vmap_area_root) - rb_erase_augmented(&va->rb_node, - root, &free_vmap_area_rb_augment_cb); - else - rb_erase(&va->rb_node, root); + BUG_ON(RB_EMPTY_NODE(&va->rb_node)); - list_del(&va->list); - RB_CLEAR_NODE(&va->rb_node); - } + if (root == &free_vmap_area_root) + rb_erase_augmented(&va->rb_node, + root, &free_vmap_area_rb_augment_cb); + else + rb_erase(&va->rb_node, root); + + list_del(&va->list); + RB_CLEAR_NODE(&va->rb_node); } #if DEBUG_AUGMENT_PROPAGATE_CHECK @@ -1190,8 +1186,6 @@ EXPORT_SYMBOL_GPL(unregister_vmap_purge_notifier); static void __free_vmap_area(struct vmap_area *va) { - BUG_ON(RB_EMPTY_NODE(&va->rb_node)); - /* * Remove from the busy tree/list. */