From patchwork Sun May 26 21:22:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10961903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78762912 for ; Sun, 26 May 2019 21:22:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6840B28A17 for ; Sun, 26 May 2019 21:22:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B65F28A99; Sun, 26 May 2019 21:22:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ECB4828A17 for ; Sun, 26 May 2019 21:22:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 203686B000E; Sun, 26 May 2019 17:22:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 18BB16B0010; Sun, 26 May 2019 17:22:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFA546B0266; Sun, 26 May 2019 17:22:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by kanga.kvack.org (Postfix) with ESMTP id 8A8B06B000E for ; Sun, 26 May 2019 17:22:25 -0400 (EDT) Received: by mail-lj1-f197.google.com with SMTP id y11so2795314ljc.20 for ; Sun, 26 May 2019 14:22:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=uARKyR8Ci0MO+NMqzO80D4+9ueGXy0kR629YBnExQG586nbKCT1llvml/ICuravWsZ AcoXODg+BEJr2uhz7km89p4Gpd7Ibjj8c8ibTi2CAM7KkAwbVDCpOK49RZ+5FMNhYKKl yV7UFJQuBbjJAzc6nR4JRpV92jQykDm5MXzD4ChyUbUpcdKTkJjpYrLYAjKJVJEBxIuF bImEEn5V3pHABffdrHQ5WWmjBK7RdYjxvU4pYTp4HjZG7CXRinp7Cw/mkRXaO7VEzBGs aNMac0sz2CPl1SKw69ij+3FJYxfOTMmo1M0TCSRyd6/PynC3+sEdyPCxdf90QCZA4K7x oRBA== X-Gm-Message-State: APjAAAXms2GplKK/ikUkIPAjbhOHho8KgL/lL1FB3s5CKN/EaVy8Gnp7 xUcWmXwqz5iJV2fhndAmCaND7mYIzMpZcOjSXOQwhF7FIaOAXVohG1Tx+cQc80lFwghkfUJ4nuZ V4GdBSjo0BzSubk6yd/VYx4bg+F5hNJOpwk1AF/+Ebzmenx+gN3uVpXCNgHOTdOMoug== X-Received: by 2002:ac2:4312:: with SMTP id l18mr44295218lfh.139.1558905744877; Sun, 26 May 2019 14:22:24 -0700 (PDT) X-Received: by 2002:ac2:4312:: with SMTP id l18mr44295196lfh.139.1558905743892; Sun, 26 May 2019 14:22:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558905743; cv=none; d=google.com; s=arc-20160816; b=kK0YeGRv62JuX6en62LcT3TLfPIzzxJzsdrZDtSzb4D0V3CLKHDg9antwaOt2ZQzoD kN7ob6MWxNxEfRs2AZ8mDAHhH49NuPSK7B/afsHrqeILarJDC3D3pT/DsscUq3lA2m+i vTC/3BAarjkhh0wd5Ss6dlZkZzN56Ux0mmmpGgsW+cOb3LCgb7JWEP9qOqXH4znclbn9 5eYRIY8EHiT1u/pQDlOslBKY8UWJMsVh0gFxAK/i9nvaNsJC/dVbElBsM5O+8L0bDPvC /A49x+WsPzdjgltbjf6pWDgGpg42YQ1lw3RnE5jxCl+8WcDGjrobzj9jIFEdW5u9rQot a0Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=btPRty1J47eic7W/Rj/121XMX//zpTfzgU1H4eTJfzU3eWueMbAmMzZ9wfQiCTNMHW yuLwaiJZ2uaNgfIj4YMr7heAKzLhLplZpkLXKmJLx/iM0WzHeHqW06zDqJoGRJ7oaqsy hgmYWF3KTsY51H6MvF9FxRBcI2+2ojdUa5FEI9hCoXE8qXMoQXlVRRqJOQbSc9Hcgo0v hOInEdtNB33CcNi2o/ypnUiC/i+DE0EQt8z9R/7W/B7n44bHDLtLLSU7e2OsCfXaMpjU QX+KnVwzj8YwKHy1X2iqz6RvyjgWdFBqHGAmz4QN2fan7FOYpwRutF59/ONv2f43bYaj d9zw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=UBdTX9+e; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c15sor4168204lji.24.2019.05.26.14.22.23 for (Google Transport Security); Sun, 26 May 2019 14:22:23 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=UBdTX9+e; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fnFIhfm6w+x6vlcZglDCubHYA2sqJDEBTXH+zDoT54E=; b=UBdTX9+e4FvlHF4JFOYIDGquBW1PRTf6/hoP7Ew9uurKyudFzrNtQsPK+PpY2kJbcz RrbaPQyrg87qkICEVv2tBAVhjYD+SMpSdxNurR6cJfHLCn1BtGBQfSdPP9AkBjsdJAfg aLzuFtmSM/Yp6QTaZ94jvl3W56Eqf8+1cBNozcS581wGb4CnsbDB+ce8MuQkYEWOHghV 2WNVzQwZavCHCwYE9pLaKNRczb/qIwTAeNH1O3VcpMY3WFa+yJHRgnyaecSjfbSODT0E gmJH9eFilpIPj/xNffX9FYFqVn5/b4fsxxF6OCytbaResF1Q+VfdYOqoeb1a+V4haXP8 IGOw== X-Google-Smtp-Source: APXvYqxKcunfWrPxCigO2o6ZR9K3hxrEk96bNlekmsQEeZIMZuGoXyAQXyF5tttat3bj0KboCwbv8g== X-Received: by 2002:a2e:249:: with SMTP id 70mr56398543ljc.178.1558905743514; Sun, 26 May 2019 14:22:23 -0700 (PDT) Received: from pc636.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id y4sm1885105lje.24.2019.05.26.14.22.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 May 2019 14:22:22 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton , linux-mm@kvack.org Cc: Roman Gushchin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH v2 1/4] mm/vmap: remove "node" argument Date: Sun, 26 May 2019 23:22:10 +0200 Message-Id: <20190526212213.5944-2-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190526212213.5944-1-urezki@gmail.com> References: <20190526212213.5944-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Remove unused argument from the __alloc_vmap_area() function. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index c42872ed82ac..ea1b65fac599 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -985,7 +985,7 @@ adjust_va_to_fit_type(struct vmap_area *va, */ static __always_inline unsigned long __alloc_vmap_area(unsigned long size, unsigned long align, - unsigned long vstart, unsigned long vend, int node) + unsigned long vstart, unsigned long vend) { unsigned long nva_start_addr; struct vmap_area *va; @@ -1062,7 +1062,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, * If an allocation fails, the "vend" address is * returned. Therefore trigger the overflow path. */ - addr = __alloc_vmap_area(size, align, vstart, vend, node); + addr = __alloc_vmap_area(size, align, vstart, vend); if (unlikely(addr == vend)) goto overflow; From patchwork Sun May 26 21:22:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10961905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 383C814B6 for ; Sun, 26 May 2019 21:22:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2845D28A17 for ; Sun, 26 May 2019 21:22:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C96628A99; Sun, 26 May 2019 21:22:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8690328A17 for ; Sun, 26 May 2019 21:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F66B6B0010; Sun, 26 May 2019 17:22:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7AA336B0269; Sun, 26 May 2019 17:22:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6951B6B026A; Sun, 26 May 2019 17:22:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by kanga.kvack.org (Postfix) with ESMTP id F0C1F6B0010 for ; Sun, 26 May 2019 17:22:26 -0400 (EDT) Received: by mail-lj1-f197.google.com with SMTP id t77so2792226lje.17 for ; Sun, 26 May 2019 14:22:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=MSM6WbF67OdKB/w9seZTGzSuynUfhCF7zBIV2w+Az9Q=; b=bhpiyG+fQrAwqd9DDrtMHY65m09+ZJLdNam4pYTOPyjDVGPwPCMsNJ4uhqk/mdviAZ 1BzuDd4VEuxZtq9V5gp5GVWnISkuom5EzG3DER8JO3aK+Rpr1FQEFD7H3RyDf07iQ1mU drI1iX/pQ0tWTh4oX/NaEjUHzBUtcrVegjqRTY1o1NT2L7/UMpRO3qkpyOxjhNso5KKk A0LoYpz0Kosdk1CqnXjG1haBXcXzPG4kCp+xTfoURtaNtnql3ww5rQ/b3/PF/yDYviJw 6NfAGskjqH0/ih+qekpTETYdO3+Xfyp+V/nfPNtKEIAWPMiJu7lU4EkiP3/PkiqmXfC9 Nc3A== X-Gm-Message-State: APjAAAVOUzoYaWpj8/ZFueogOspdB9vkz0wvogRGVyy/P1GPHzd1ZdDA FwNvyuE7HKWt4OUx9iM0KIWSkuT4hRzPntXQ0LaimBCJjSx6yYKbsJD6cyBHdx5YiojNCOiQf9j QMDcLbjU8XBVaN+NzUw0vIF0bUzHitbVHudEr54hdHyM4woEhS2DRE/P+KVwcqT1RMQ== X-Received: by 2002:a2e:978f:: with SMTP id y15mr41438926lji.125.1558905746429; Sun, 26 May 2019 14:22:26 -0700 (PDT) X-Received: by 2002:a2e:978f:: with SMTP id y15mr41438889lji.125.1558905745079; Sun, 26 May 2019 14:22:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558905745; cv=none; d=google.com; s=arc-20160816; b=p2msDoyUa1DYNEQJqynZErrFb6XSEgGaVMA1gaKPigi4xw4SU+3e1BNZOZmUYNehn3 oYVXqP5pxdCJQuUboX2LSSYawpZ/hGJR2YUheFSs5Y+oCCsAha/l4HPp+s2lHd8Mx5op b7sPGge0gzre2Kc/pIb64qY1NcJiNwEIlOFt1XP8kOsLlHVFb81xGue27rBfXgnkl6pR MOA8s60HUvQ0kcj0V06uRyrE3J2Wp75nJPUQHPYhSavQjfglRqkedATIKsXQnsSt35kg 3mnCdR8nu95Tjd0tU7iGOqn7MRaigg35eeCXYu+27MNSpeTo9Wp6hp1Cg0iJDrhjzvcC rnRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MSM6WbF67OdKB/w9seZTGzSuynUfhCF7zBIV2w+Az9Q=; b=Mx6IqSmdVYMzot7fy7DvDip1EeIAaq0DoE4SR+KTqe0xxZSXUekw15mJKrMG97GdCi oAKapUNf+X3jlxwNJYp9v1FP0d0dPAIQIBeHqrDT6UtiR+jp51tUKclZSx8dcppBtBTd Munn92Bz7E35YXWNTNoSMij4RPJ6K1+Az6uWh1txhSbvE83i5q7oyS8XFmc9JpR+gRmh j4iWdaGKlQeQzZKWwYdlxmIzNuciHbNqq7gNg1jL7507S/tAZ0pElpQvD6GEK6Ti0sE8 IZjTv55M6bHslKIOGAb6Sc2GKB8DOVPEHEkkV/EsZ73o1ZDlxMapyMxCzQcXILPMLbsQ JpsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ppKXhdcY; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p6sor4133494ljh.38.2019.05.26.14.22.24 for (Google Transport Security); Sun, 26 May 2019 14:22:25 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ppKXhdcY; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MSM6WbF67OdKB/w9seZTGzSuynUfhCF7zBIV2w+Az9Q=; b=ppKXhdcYv/KS+lHxU0WQYWLLtvUUgJpmMrzEsrFlLI1NVg4OvTrmnbOsrKvDddyYyv DXoCrVZIRc3jEPmAgywzlACas20zyM4yq5e20hOeZBFnFgqw/Ti/QTrYF1suEu4F3ZOg hk0g8OSW1fcjurJLDHFj8tADADT71lHJd0xKkRyJtMKAk4JJNpXiBztGevsr/vXF7BfS ixnfZ0yczhG9qtuh1y3tr9SOCPIMZlY1VzIopaMZ2Vly3KrhzJsZ0ewrzGrBxPGFOn9P M0U+MIiW3FWQLBm70HawONYxYDIMTgQkgHwW3Dvc3Kqn6mgsqaK/Y/Mh/5HQqhcLL2v+ IzgQ== X-Google-Smtp-Source: APXvYqzjRREPhs/jfl9SG9/cgGQN1br+FVaYC31Ecl9Wnipwc23oiap1J5otLWtvsLFSBf1EkOKeHQ== X-Received: by 2002:a2e:7d02:: with SMTP id y2mr32698875ljc.62.1558905744691; Sun, 26 May 2019 14:22:24 -0700 (PDT) Received: from pc636.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id y4sm1885105lje.24.2019.05.26.14.22.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 May 2019 14:22:24 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton , linux-mm@kvack.org Cc: Roman Gushchin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH v2 2/4] mm/vmap: preload a CPU with one object for split purpose Date: Sun, 26 May 2019 23:22:11 +0200 Message-Id: <20190526212213.5944-3-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190526212213.5944-1-urezki@gmail.com> References: <20190526212213.5944-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Refactor the NE_FIT_TYPE split case when it comes to an allocation of one extra object. We need it in order to build a remaining space. Introduce ne_fit_preload()/ne_fit_preload_end() functions for preloading one extra vmap_area object to ensure that we have it available when fit type is NE_FIT_TYPE. The preload is done per CPU in non-atomic context thus with GFP_KERNEL allocation masks. More permissive parameters can be beneficial for systems which are suffer from high memory pressure or low memory condition. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 76 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ea1b65fac599..b553047aa05b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -364,6 +364,13 @@ static LIST_HEAD(free_vmap_area_list); */ static struct rb_root free_vmap_area_root = RB_ROOT; +/* + * Preload a CPU with one object for "no edge" split case. The + * aim is to get rid of allocations from the atomic context, thus + * to use more permissive allocation masks. + */ +static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); + static __always_inline unsigned long va_size(struct vmap_area *va) { @@ -950,9 +957,24 @@ adjust_va_to_fit_type(struct vmap_area *va, * L V NVA V R * |---|-------|---| */ - lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); - if (unlikely(!lva)) - return -1; + lva = __this_cpu_xchg(ne_fit_preload_node, NULL); + if (unlikely(!lva)) { + /* + * For percpu allocator we do not do any pre-allocation + * and leave it as it is. The reason is it most likely + * never ends up with NE_FIT_TYPE splitting. In case of + * percpu allocations offsets and sizes are aligned to + * fixed align request, i.e. RE_FIT_TYPE and FL_FIT_TYPE + * are its main fitting cases. + * + * There are a few exceptions though, as an example it is + * a first allocation (early boot up) when we have "one" + * big free space that has to be split. + */ + lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); + if (!lva) + return -1; + } /* * Build the remainder. @@ -1023,6 +1045,48 @@ __alloc_vmap_area(unsigned long size, unsigned long align, } /* + * Preload this CPU with one extra vmap_area object to ensure + * that we have it available when fit type of free area is + * NE_FIT_TYPE. + * + * The preload is done in non-atomic context, thus it allows us + * to use more permissive allocation masks to be more stable under + * low memory condition and high memory pressure. + * + * If success it returns 1 with preemption disabled. In case + * of error 0 is returned with preemption not disabled. Note it + * has to be paired with ne_fit_preload_end(). + */ +static int +ne_fit_preload(int nid) +{ + preempt_disable(); + + if (!__this_cpu_read(ne_fit_preload_node)) { + struct vmap_area *node; + + preempt_enable(); + node = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, nid); + if (node == NULL) + return 0; + + preempt_disable(); + + if (__this_cpu_cmpxchg(ne_fit_preload_node, NULL, node)) + kmem_cache_free(vmap_area_cachep, node); + } + + return 1; +} + +static void +ne_fit_preload_end(int preloaded) +{ + if (preloaded) + preempt_enable(); +} + +/* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. */ @@ -1034,6 +1098,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, struct vmap_area *va; unsigned long addr; int purged = 0; + int preloaded; BUG_ON(!size); BUG_ON(offset_in_page(size)); @@ -1056,6 +1121,12 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask & GFP_RECLAIM_MASK); retry: + /* + * Even if it fails we do not really care about that. + * Just proceed as it is. "overflow" path will refill + * the cache we allocate from. + */ + preloaded = ne_fit_preload(node); spin_lock(&vmap_area_lock); /* @@ -1063,6 +1134,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, * returned. Therefore trigger the overflow path. */ addr = __alloc_vmap_area(size, align, vstart, vend); + ne_fit_preload_end(preloaded); + if (unlikely(addr == vend)) goto overflow; From patchwork Sun May 26 21:22:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10961907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 082CE14B6 for ; Sun, 26 May 2019 21:22:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBD9A28A17 for ; Sun, 26 May 2019 21:22:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DFF0928A99; Sun, 26 May 2019 21:22:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7624E28A17 for ; Sun, 26 May 2019 21:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FA816B0269; Sun, 26 May 2019 17:22:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 55E186B026A; Sun, 26 May 2019 17:22:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 425346B026B; Sun, 26 May 2019 17:22:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by kanga.kvack.org (Postfix) with ESMTP id D36A96B0269 for ; Sun, 26 May 2019 17:22:27 -0400 (EDT) Received: by mail-lf1-f70.google.com with SMTP id 17so2464659lfr.14 for ; Sun, 26 May 2019 14:22:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=zcDlkJTeHrjHVEgzRPAEZQALvFAG5US4eI5mM5YgZuY=; b=fPM3UeapCZOQMxYiXnqn504djpmdAXYqSYKYKN7Q+mtRtYyiro5g67RANJl79MTfUw k3PyfeIum3n9GjBqojzBtMLT0CpFIHWCF/ar4kVSNTWo2mCp9WzhjVwbyLjSajyLvdZ/ +Ki3wEgf8VoD+ngpLZRxWD/5y3FKcYhZ7W23ixpBnqMexl0M+9XQ+OEViyE5maveHNnz lhvcfRoFjEQO5HSz6M3WO6LDMTNUqf1hKdxZ9IfeZBEz5rjsTi2/6MhFMHuNaM6v9YS9 WIflZn4IzoIXEsY47eVxKYODwOlVXbir5fOEgwGQfWK2eq4E+X1T5e5ZrCcym6rkygVo g6VA== X-Gm-Message-State: APjAAAWvaSFRPD236J9rlX2wzaFfYHvmAroOFq5quLAjdYQ2uIyxs0ac Fc483OI/TpVOfZUgC/yr6ZYQYWWLMyleeGF2SGjzjpdLhHF8TpQlalNH0XUJR9WNsACt/3zETN3 nOYVxWj55KxeexDve9rwe52NvUmF/MspgZ4FXxyqSDXwQ2D4np9NeB1NjCqktlFBBQQ== X-Received: by 2002:a19:3f16:: with SMTP id m22mr2130741lfa.104.1558905747315; Sun, 26 May 2019 14:22:27 -0700 (PDT) X-Received: by 2002:a19:3f16:: with SMTP id m22mr2130707lfa.104.1558905746206; Sun, 26 May 2019 14:22:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558905746; cv=none; d=google.com; s=arc-20160816; b=KA7lf/O1zrHtWLr+iwwwEFG60VrnJTemEW0kqOnkx6m8lDSFn46xbLWk3E11tOOYXb kZuQ7w73PmzsTkug+bPZbno7Xv8afx/3+HSBAlg8KZLwrjKF5sr5m9tcBNCVwTysPT7W whQCry3K1HDM1q12sm4uz2Ywyw3JD1VY3DMneIIBiK5X9DE1R2hidVPoWKH6tsKKljSZ ehnafEvVyuAXa1PI4qNI5EQKl/kVeRZj2yF8zTbstkoQ3LhqFL8sO1uPtFgKFp40mBCQ oMJ7U7C7QYXK6mUzXkRiOWdSZYQG4Z5sXKACQHojiv+zyjlEnKOcwXeBFdVMK1H0rP77 PiHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zcDlkJTeHrjHVEgzRPAEZQALvFAG5US4eI5mM5YgZuY=; b=x1idipNSZ7pxtnHwhm/te8spgvs45TaxKrPwi62F3AHmepi/rxNR/JYR5meq57OWcc nP31eBA5UZ4j6FlRr/pnM+3fx481VNp1DsKk5WUkO+5kOZ1UBLMthl6mP6/jMWysuv6T MSUJaJb1mZHKCu6IiqPT8FilvqjdnH7/fhrO8SrDaNuMp//mSfws5o0q6T1XoVPBDS+F VY645M6IKqjET6FXtChqAX53gVzlMlGx9OQlOMOa4mX8ERwcXXrLutQnmi9/dyLlzAnB UT7PQtH7yPQ2pWczsacCITVrn1/2wBtZW8+LigyYiHAe/ypVWaNvILvom825/uEz81m/ cOKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Ao5CkBZM; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y5sor2126295lfl.36.2019.05.26.14.22.26 for (Google Transport Security); Sun, 26 May 2019 14:22:26 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Ao5CkBZM; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zcDlkJTeHrjHVEgzRPAEZQALvFAG5US4eI5mM5YgZuY=; b=Ao5CkBZMIKG8lMZLADM2cFJSxDg0Ojot0t///bzeuKFY03bvOBAnzd/IYRNsnx1egd 3cx0VIwpT7ShMf+DETVPyhQbGBL4J7Dq99mDqyJg9c16+rDrd31BAiUqhE+YSuLBA84g xyPOC78X5H2fEytgSnUVAtMS0oN4D55Qh4L+biWyJ0niL4cwDMbctSZPD0sko/8Wt+NL LwcBZwe5kMfaF35crXxjCxAx0oMGBleEaBvzu7biLO379bYGhX9Rla8wNqgEaWl9NWDQ Nu5wVqUKH58QonUtZTVWyTbAR65RhJDpanmma1bnW5KnExBlTs7P+ZcleNLu4AQoOqJP Y4WA== X-Google-Smtp-Source: APXvYqxcGZ6X45TO6AjIW2X5VxteP29qd3Xp+Mlo5MYszszCZhtL1+swSlNelKa8z6Y5OwrNMEa4KA== X-Received: by 2002:ac2:5922:: with SMTP id v2mr164163lfi.180.1558905745839; Sun, 26 May 2019 14:22:25 -0700 (PDT) Received: from pc636.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id y4sm1885105lje.24.2019.05.26.14.22.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 May 2019 14:22:25 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton , linux-mm@kvack.org Cc: Roman Gushchin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH v2 3/4] mm/vmap: get rid of one single unlink_va() when merge Date: Sun, 26 May 2019 23:22:12 +0200 Message-Id: <20190526212213.5944-4-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190526212213.5944-1-urezki@gmail.com> References: <20190526212213.5944-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It does not make sense to try to "unlink" the node that is definitely not linked with a list nor tree. On the first merge step VA just points to the previously disconnected busy area. On the second step, check if the node has been merged and do "unlink" if so, because now it points to an object that must be linked. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b553047aa05b..6f91136f2cc8 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -718,9 +718,6 @@ merge_or_add_vmap_area(struct vmap_area *va, /* Check and update the tree if needed. */ augment_tree_propagate_from(sibling); - /* Remove this VA, it has been merged. */ - unlink_va(va, root); - /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); @@ -745,12 +742,12 @@ merge_or_add_vmap_area(struct vmap_area *va, /* Check and update the tree if needed. */ augment_tree_propagate_from(sibling); - /* Remove this VA, it has been merged. */ - unlink_va(va, root); + /* Remove this VA, if it has been merged. */ + if (merged) + unlink_va(va, root); /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); - return; } } From patchwork Sun May 26 21:22:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 10961909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73AAC912 for ; Sun, 26 May 2019 21:22:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 639B828A17 for ; Sun, 26 May 2019 21:22:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5776D28A99; Sun, 26 May 2019 21:22:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0FC928A17 for ; Sun, 26 May 2019 21:22:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8925F6B026A; Sun, 26 May 2019 17:22:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8436E6B026B; Sun, 26 May 2019 17:22:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70A346B026C; Sun, 26 May 2019 17:22:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by kanga.kvack.org (Postfix) with ESMTP id 07B0A6B026A for ; Sun, 26 May 2019 17:22:29 -0400 (EDT) Received: by mail-lf1-f71.google.com with SMTP id a25so473149lfl.0 for ; Sun, 26 May 2019 14:22:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=nMiFTZq9m8Tud2oYGi2GiJ1Z+X/2sbQtc11N2qZi8ps=; b=UW9tDapsb+sI+puvb6VV9coMbBh+ko5zL752/Hn8y+mn/w6h9pgRRurak27Pm6PTiL 1ALcsJB27Y3apwYz4r8cX+M3BYKJQjnb2cInzZcilzJNx+/33JuUrdjKILio9oJRdTOz 0o48SngZlacJEG/TIFFwld7PHq2FtRsIMJJKpreGUJBnjaC/g4rp2sFEcq4DyOG1U5tO mxiaHXhILu4RQ5DzDrxeu/4UnCfN/0uuhYexSs9Shtk6hsnN+XjxOOoJ2J+NQM9Za6en 2D8Y3lY2TOw8WT/c954qC2A7ufM5hj0iqgrchCxCaFVRffaUz+yOzREw/ngpmR3Fmmbr I4kA== X-Gm-Message-State: APjAAAUB9tOYKOM6Wn6/rzgm/fYxMt5luQZ4SOZm4sqz9zhwMa4bcC+e KwcNHSB3HYjSWiH1KREBFcUPi1gzvJggvdd6AcMhm4vakxZOejF1uIbgQhFtMvlQcoEW2mJmu04 Z2s6hIYRAXJvhmBU6z8aGhAtcIF1M9NfgRHKc+M5+prZXaHrgQU2NdtH4UmU5dHTbtA== X-Received: by 2002:a19:e05c:: with SMTP id g28mr5620085lfj.167.1558905748479; Sun, 26 May 2019 14:22:28 -0700 (PDT) X-Received: by 2002:a19:e05c:: with SMTP id g28mr5620064lfj.167.1558905747430; Sun, 26 May 2019 14:22:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558905747; cv=none; d=google.com; s=arc-20160816; b=act50UKuhNEfQmAJeXtZY4ApK18MsyCBQbjqJ8kZdyUQwp07Sarx535NGSkJ8QSVJ+ j+w0tDNXgtBWNdOhRz17DpXxJwZF5/Cswv3zu/3hvkT3iIK6gCipQdekzjX8kWa3NMFe oR+D6IjJba8TYHa2t9goqwdwwYqhbc9PMQLWgNUy0z/PMwkipHvYVm+HS5HoZN4F7Xac okvmJDcyNid2dzqE0M5kqJ9hRTh6TXooHMA7OWFTgfmbAqmilYzaAO01PF+1IMr8P83Q Nxu3iSpx7OmZVwwxiHHL3E9gvF3UE1hFyM+9bGY67ZDtamn8/FQDdrCjSUy5R5QngPl/ Rh7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nMiFTZq9m8Tud2oYGi2GiJ1Z+X/2sbQtc11N2qZi8ps=; b=KDLdZKwCdyvBXFBO0GVyXshR+1GRvQv2k/trft4ktO5OtwJO3m0pgyROW9Tddj+ETg vT2eNrZmofdr2qLMw2ITCA8THTr0+MN5JaJcSFxa6MjTwdAVpSSPoRjayc+BIbM465Fm Osy7wc/R8pH0xfJVi5rzAoaUuLePor1WZiQZBTVkAT/ukx7VQVd03bjzKjhCrAy1Of1D fmj433lve5MhnZ1/CYT1hOuoOLfN2PzsBniwnV0J9eeNetBxu/KlllxyYML5XT/kCoCU uNU2VosHd96VVfSLNkNACNI5YVXVrC2QPQgdsRkQ5wInH+Z9PbXZsUokNP/pKB6it5dj aALA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=kW8EQGCX; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p27sor4135813ljp.8.2019.05.26.14.22.27 for (Google Transport Security); Sun, 26 May 2019 14:22:27 -0700 (PDT) Received-SPF: pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=kW8EQGCX; spf=pass (google.com: domain of urezki@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nMiFTZq9m8Tud2oYGi2GiJ1Z+X/2sbQtc11N2qZi8ps=; b=kW8EQGCX9RPA7Y1Hbh9+m37Rg3nnmFcAhCEEN+Wn4d6HHxofUWi5NJZicvFduiIiWH Z4tWfptAT8kbByu96/Af1k9A65xG9kG1xwVMP+zdYzfi5mBI3MfRi++wLymSduYdjNPS el25ZYIltNE8kQrBHpSMo9w+RS1fCViSv708fFUtKfLosmGswxo1c4tg/wZs9epUFFel ipOv7KktBtXHfm5LZmxBImQl05kSuQGLu5g/C2/JEqvzEanQeYq1WFYBsfHbm4lXQFMl 0gdsAkkrqKf4WYwHkn8KFBNHuGoBxYk55CqocVHv6l8pqAYsNa0wErxcHIB5sV+ZYsyz bOwA== X-Google-Smtp-Source: APXvYqwLtfKo3g1XsLcA3UgctcSE5rh7e9LElVzrf5Zs8QIa2LJrFz7rsW567IrrpGxEhHMpbftXEw== X-Received: by 2002:a2e:8985:: with SMTP id c5mr14828724lji.84.1558905747025; Sun, 26 May 2019 14:22:27 -0700 (PDT) Received: from pc636.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id y4sm1885105lje.24.2019.05.26.14.22.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 May 2019 14:22:26 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton , linux-mm@kvack.org Cc: Roman Gushchin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , LKML , Thomas Garnier , Oleksiy Avramchenko , Steven Rostedt , Joel Fernandes , Thomas Gleixner , Ingo Molnar , Tejun Heo Subject: [PATCH v2 4/4] mm/vmap: move BUG_ON() check to the unlink_va() Date: Sun, 26 May 2019 23:22:13 +0200 Message-Id: <20190526212213.5944-5-urezki@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190526212213.5944-1-urezki@gmail.com> References: <20190526212213.5944-1-urezki@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Move the BUG_ON()/RB_EMPTY_NODE() check under unlink_va() function, it means if an empty node gets freed it is a BUG thus is considered as faulty behaviour. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 6f91136f2cc8..0cd2a152826e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -533,20 +533,16 @@ link_va(struct vmap_area *va, struct rb_root *root, static __always_inline void unlink_va(struct vmap_area *va, struct rb_root *root) { - /* - * During merging a VA node can be empty, therefore - * not linked with the tree nor list. Just check it. - */ - if (!RB_EMPTY_NODE(&va->rb_node)) { - if (root == &free_vmap_area_root) - rb_erase_augmented(&va->rb_node, - root, &free_vmap_area_rb_augment_cb); - else - rb_erase(&va->rb_node, root); + BUG_ON(RB_EMPTY_NODE(&va->rb_node)); - list_del(&va->list); - RB_CLEAR_NODE(&va->rb_node); - } + if (root == &free_vmap_area_root) + rb_erase_augmented(&va->rb_node, + root, &free_vmap_area_rb_augment_cb); + else + rb_erase(&va->rb_node, root); + + list_del(&va->list); + RB_CLEAR_NODE(&va->rb_node); } #if DEBUG_AUGMENT_PROPAGATE_CHECK @@ -1188,8 +1184,6 @@ EXPORT_SYMBOL_GPL(unregister_vmap_purge_notifier); static void __free_vmap_area(struct vmap_area *va) { - BUG_ON(RB_EMPTY_NODE(&va->rb_node)); - /* * Remove from the busy tree/list. */