From patchwork Fri Dec 14 18:07:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10731521 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED975924 for ; Fri, 14 Dec 2018 18:07:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EBD42D912 for ; Fri, 14 Dec 2018 18:07:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 22F422D930; Fri, 14 Dec 2018 18:07:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B52B2D912 for ; Fri, 14 Dec 2018 18:07:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB1D48E01FE; Fri, 14 Dec 2018 13:07:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A746E8E01DC; Fri, 14 Dec 2018 13:07:50 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 967DA8E01FE; Fri, 14 Dec 2018 13:07:50 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 4969A8E01DC for ; Fri, 14 Dec 2018 13:07:50 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id l9so4097299plt.7 for ; Fri, 14 Dec 2018 10:07:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=7HiZSQwm/UReDKaG3pBCGsAVn/dcBqgos7ZO2ZPxoNo=; b=mM+8pXQz+hKlYrCBhsDSWNffrdKoImKz6pFS+CJe5yTTkkMRXXdrqdurMVexIb7Zqm G+htR+Ndz0WlybIamQEuQN7QTSZiVT9mYO99F0BS3KmG084AQO13+SZ/VDkF74bE22NT KhHohTwotNekbrDbM3xHSBVuwQcmBtrt7xQmuaGwybByAq3reKapGuB7Iuob+tIM8EIg QYmoC1EjsqyYJ6b6IoTYebmxLo56/TmjRnRN6jfOCjjveEF3QiUEktvgyc/yD8YxPzUH rYbD+ICiXFrJtbIHBUBMopLxOUuqRIJemPoVicpb9G+W81FMijziKTlvENoaWY8Qz9KH vZuQ== X-Gm-Message-State: AA+aEWbvWSz4Q+FzwM57VdfERTfwwRvIKzeYRqXLxKLCeTQlkgEquda7 w0kwRGpxcWVQ+f02F4cqH4PtTyCD+uDwnQEpes/f4k6oBZFnT+/gpMqSOTL+Dz31v/j0QdiVMsT upib+7Y9srFmYQb4r7GrFJ44RqpAprdhHOyxx1HLf9aD/Aseav9MxcpOovt0fGqKAqnYRDy/yWH ZPj0oR1EaS22jkCxVqLtFWGmvyp45GBVz5yLlLUgax5hEdEhQTeS7Vvr+dayLas9JitFJmTFrU5 tiY+7tVRQdi3Ccykig6eobKfjKBKbawOJX1K36+jPMEwyRh+M1QLBEw2X1VFlvz3Z2pBnomx3NC lKe8RpF4coAp/yh5VTy7teD9RMETLmmiNjnv/yohkCysmRn7Y4OzUZihBrIJ/3azgT936uppI/b o X-Received: by 2002:a63:2849:: with SMTP id o70mr3501954pgo.155.1544810869956; Fri, 14 Dec 2018 10:07:49 -0800 (PST) X-Received: by 2002:a63:2849:: with SMTP id o70mr3501907pgo.155.1544810869142; Fri, 14 Dec 2018 10:07:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544810869; cv=none; d=google.com; s=arc-20160816; b=bvewdud5+qA1wJ9mddtA1y/6aDKuHraj4qRZsH0WsBdXwrce0be1jEJvobreE3rd/h Vg+K2LP4ihJ2vm/PuPUyK49khWvW2Kzpv1yDdToD5qXt7qUxZwu0k/vUpjM6kElcypJA YWgMg4OKjGfvqB3NLYM24/1lK2e9G/UQqSJyt9Wuykvy6Q12ceDumloI8pEZkdrfeWEY a2cX3gvMOFgJiLWpMm8LfrR5HWczMzKsQ6G/aqW8rHfsBJFbz4ZgnpxF0tJTtC5loB3V pC26os7edY56BVs3/XV5PNIwuAp2oz3FaVYOTfEE3P5yd7nXgL+D2SagFVqR1H//zCUc pM/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=7HiZSQwm/UReDKaG3pBCGsAVn/dcBqgos7ZO2ZPxoNo=; b=wV/EckH0+0pNEqW0AEJt8nIskDv4FtarZk8SpjEb0b/Lapg8HYP18vaaV8JhZBOQTz nc4eny6i3jRtV17KToQlWVC2EBEceix7hmxbkrBN2uGRGSeAG+hD8F9As8FBZlqqvV1P 3eUuRh0N5TbTRo8o83NeiCa+DF3ayV3VWgxaWR59apETFT/DORyxWX7r46hOSYDTbF/G qeqtVp0ox3MM4J8Zl1Fmu78p8yfSNbbONGVA1HNHz97ovD6EWr1bbNkDHjDbXPf57rt0 hknh4DWIF39uo6f14fijkLsG+QgOtY/CRIkVni0Ju3TTrf6XJbuGgGgPv22RDlT/Hhcm v71Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mhNOnZvH; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u6sor9136357pgu.43.2018.12.14.10.07.49 for (Google Transport Security); Fri, 14 Dec 2018 10:07:49 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mhNOnZvH; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7HiZSQwm/UReDKaG3pBCGsAVn/dcBqgos7ZO2ZPxoNo=; b=mhNOnZvHpkLtR1ci0159pT81uVhecrQoFQMn8Ux9pYNAemWwJHJFw0JPigeXGzdcO7 mZ93GIpSqrUBjr9dPGEn6IPmNaJKmZyeN5KDV25eWf4/hFXTCr4HBcZ/nzi5QUWm3MBq UbKPBPSOeX0m94pozmrw0JBrBuOT8+9EOzpHhHz4fB14muAc4Wm4/pZJvQQHAfvmcSiL aeuiMjExueGK9mdKUdJ0NnvupOIMNWS4XfQxvN8g2/J7Nd8N0eXuAioxmxtmLeHX91iE x3WZdAbV/6tJuFiKad+nlQpFgmSEJfarJ5GmDN1MWeRID70MBA5V+rk9orPNCi0fAsoh /CAA== X-Google-Smtp-Source: AFSGD/Xf9mAIsXiLzGt9UCYFSH7fUUAPt1KYigA25sn8KiXN4CJ9IyX/x0DdDXq9vjaRcAeCTkz+1g== X-Received: by 2002:a63:40c6:: with SMTP id n189mr3479960pga.355.1544810868651; Fri, 14 Dec 2018 10:07:48 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::5:3d97]) by smtp.gmail.com with ESMTPSA id r2sm7180856pgo.18.2018.12.14.10.07.47 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 14 Dec 2018 10:07:48 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org Cc: Alexey Dobriyan , Johannes Weiner , Michal Hocko , Vlastimil Babka , linux-kernel@vger.kernel.org, kernel-team@fb.com, Roman Gushchin Subject: [RFC 1/4] mm: refactor __vunmap() to avoid duplicated call to find_vm_area() Date: Fri, 14 Dec 2018 10:07:17 -0800 Message-Id: <20181214180720.32040-2-guro@fb.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181214180720.32040-1-guro@fb.com> References: <20181214180720.32040-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP __vunmap() calls find_vm_area() twice without an obvious reason: first directly to get the area pointer, second indirectly by calling remove_vm_area(), which is again searching for the area. To remove this redundancy, let's split remove_vm_area() into __remove_vm_area(struct vmap_area *), which performs the actual area removal, and remove_vm_area(const void *addr) wrapper, which can be used everywhere, where it has been used before. On my test setup, I've got up to 12% speed up on vfree()'ing 1000000 of 4-pages vmalloc blocks. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Matthew Wilcox --- mm/vmalloc.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 97d4b25d0373..24a84d584609 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1462,6 +1462,24 @@ struct vm_struct *find_vm_area(const void *addr) return NULL; } +static struct vm_struct *__remove_vm_area(struct vmap_area *va) +{ + struct vm_struct *vm = va->vm; + + might_sleep(); + + spin_lock(&vmap_area_lock); + va->vm = NULL; + va->flags &= ~VM_VM_AREA; + va->flags |= VM_LAZY_FREE; + spin_unlock(&vmap_area_lock); + + kasan_free_shadow(vm); + free_unmap_vmap_area(va); + + return vm; +} + /** * remove_vm_area - find and remove a continuous kernel virtual area * @addr: base address @@ -1472,31 +1490,20 @@ struct vm_struct *find_vm_area(const void *addr) */ struct vm_struct *remove_vm_area(const void *addr) { + struct vm_struct *vm = NULL; struct vmap_area *va; - might_sleep(); - va = find_vmap_area((unsigned long)addr); - if (va && va->flags & VM_VM_AREA) { - struct vm_struct *vm = va->vm; - - spin_lock(&vmap_area_lock); - va->vm = NULL; - va->flags &= ~VM_VM_AREA; - va->flags |= VM_LAZY_FREE; - spin_unlock(&vmap_area_lock); - - kasan_free_shadow(vm); - free_unmap_vmap_area(va); + if (va && va->flags & VM_VM_AREA) + vm = __remove_vm_area(va); - return vm; - } - return NULL; + return vm; } static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; + struct vmap_area *va; if (!addr) return; @@ -1505,17 +1512,18 @@ static void __vunmap(const void *addr, int deallocate_pages) addr)) return; - area = find_vmap_area((unsigned long)addr)->vm; - if (unlikely(!area)) { + va = find_vmap_area((unsigned long)addr); + if (unlikely(!va || !va->vm)) { WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n", addr); return; } + area = va->vm; debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + __remove_vm_area(va); if (deallocate_pages) { int i; @@ -1530,7 +1538,6 @@ static void __vunmap(const void *addr, int deallocate_pages) } kfree(area); - return; } static inline void __vfree_deferred(const void *addr) From patchwork Fri Dec 14 18:07:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10731523 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB7916C5 for ; Fri, 14 Dec 2018 18:07:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D1D32D912 for ; Fri, 14 Dec 2018 18:07:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1150C2D930; Fri, 14 Dec 2018 18:07:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F6FC2D912 for ; Fri, 14 Dec 2018 18:07:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 917BA8E01FF; Fri, 14 Dec 2018 13:07:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C73D8E01DC; Fri, 14 Dec 2018 13:07:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DE1E8E01FF; Fri, 14 Dec 2018 13:07:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 3F0878E01DC for ; Fri, 14 Dec 2018 13:07:51 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id q63so4966668pfi.19 for ; Fri, 14 Dec 2018 10:07:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=31kslWA2pzzsHiz8Y5tC9Fd+SHDoREs2MeMPfRmzSCw=; b=thxaYJF0KjzmJVuFCXXL4Vdi0Jw/73r1meqZbI9oXoXaCjv0zENTEKgUShuS5rhRJb dCG7RtLJZ5s1Ty5AF2c6hF+6JnEq4ZYVagw9O3p76eO2zOi8E0SU+08hy4OqY4oq+WAF 29h6hYVA5+wEpvr54x4HuP80wY3g1TH87VgrCRak0fsxBYlL7nRKL1zUnpKM1v9lGC+T Ea/6RfXDJaljVylqLYaSljLX2ufx41MHB6YtkRq5Q4HmDbBfcQIC4L67riRMcv6BOw3C YhyLHzBVK9Datzq88phqWz7UJI9ogSsvVhq/75vC9NvegHBUrBxn5tAeibGZz6sw/c9q KFlg== X-Gm-Message-State: AA+aEWYcYb+3THJifnX20FRv7RYduPZAKVscC6FYd9fMQ8cer26BPddp kxSU7WZcyb5haFWBjpGjPgqxe2JcyzvuJmod5/NX09Z5GiFJp+XNe/pVHFquxlT2svSkcY7M1ht D8jXu3i8hAmlcDYmaZLfEykrxJ6D3F/N/7P0x8c4x7/cbPU+gQJiA5O/H+EyG/es2NuFgXbEc/1 ZzW1uPWfnm89P28tLhXqfAUEgukHx/hbnUnuaQjWziFZnYFFqA6ouTQiC5hgkp6uUZf7JCaiZs8 rdndP5wvG+QPMhlcH4dN6XUzIblj65buD3gj2o7WIpwy5DR/bhTQL9F+YCs7WglEGkGbo/0AxYe LrwMYFv0v13L20hUq5AmIG7ki6w8WdrlntQHVw1k8WShemqarmE6uzQB7zMALfopJNwDTuoM+nF 4 X-Received: by 2002:a17:902:4225:: with SMTP id g34mr3888601pld.152.1544810870917; Fri, 14 Dec 2018 10:07:50 -0800 (PST) X-Received: by 2002:a17:902:4225:: with SMTP id g34mr3888556pld.152.1544810870238; Fri, 14 Dec 2018 10:07:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544810870; cv=none; d=google.com; s=arc-20160816; b=lzCiRdUaq/o/C0FKJbcP5qIIHPS7FxDQfQ72ipKltVh+W1xrqEfZdWy1ZOYu7JSUno dLBvk2pbeRfoQLcJclwbDlcivCqp3uOE2bzotMitt3aQVoaHmGXv3nWrw0x6TacBhb6I +7Lf5K/QALcm3eym15h8SmZP2aHTPx9Q5Zwp0reEOeNVBNsxKFCUNvfpVGmJibkh5x9Q a4m2oT3Nm2TH92hWp/njmUsxck1L0+hluBi/jH4HT/NUgN6XSQ3XRm2TMcqbTXztIXWy 6K7NrvqNCtYH+pBsCA1+7JwaDG1gOttYle38Vm0ckiWued+pHL5qmBBe6bSadsDlNeOk Lm4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=31kslWA2pzzsHiz8Y5tC9Fd+SHDoREs2MeMPfRmzSCw=; b=Payxqi0FFUT7EM95NsY5sOLOFwIv/2lfcNZPsr1twq8zNmgap8bUA7E6QXImPlwT3x TD3CcM+WOM0pEYj3oOwm2rN3OA/Ohk1y6NkPacwQgOCbBA5HIEWH+LsLMwqkdOmZgWWi ShPWDHxcwNM8v+0evtO7pnE3D7/srejzXklGmr5J4NF+dyD5o0n/VU6chP3pcQj7zS9Z 33orO7MBJcx6NVFz4PgDTiA+iyh8ihy0UEbq5boQoaIGte9hgeFDC3YWgE3pjrqQxud0 mbZMoJPof1b9ZL0n1roxgi4glAo3RpG1TTH1PLHKS7V9bP86ghTQOoQQfqG6PAQTVN8W 3ipA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ravbIwCC; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p2sor9002218pgn.83.2018.12.14.10.07.50 for (Google Transport Security); Fri, 14 Dec 2018 10:07:50 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ravbIwCC; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=31kslWA2pzzsHiz8Y5tC9Fd+SHDoREs2MeMPfRmzSCw=; b=ravbIwCCdmKf4HxtAd0LBlD5fBNIcfvIVHABcxIopOo4LAyjDT/1svBMs2w8eK1Nd0 SXJ2E39boVT6Ne+SuVpPWKiXXPYLmLdmgw5V2NUswhpHJ6rpxBbvCYs+c2PcvPDSQEoi NxQoGkpL7V4rdegdU3INlnD9GYPv09+WsJ3M6KzNlV0IPBZZUbJFyAHxDJzVg+v1TVxo vt3p/0jsTIvDih+ftuiWNI8to8QkMJtNjycUcSNJwi0ojQ3qGIO13wzGOLR2qx3G4Yzb 6rcv6ciVXvHiZ4b7tj7iwvQH/+x9OfWuKTHSPWGbz4FJBzJ20kGhlhkjOWfmK+fJFXkx 3w6g== X-Google-Smtp-Source: AFSGD/WptKbrEs3BxathmN/kR19WDH39xE5sXtHrpcKCELKN5T/tzPlLlpfYs4rFDstZclsGpMduKw== X-Received: by 2002:a65:47ca:: with SMTP id f10mr3686922pgs.166.1544810869674; Fri, 14 Dec 2018 10:07:49 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::5:3d97]) by smtp.gmail.com with ESMTPSA id r2sm7180856pgo.18.2018.12.14.10.07.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 14 Dec 2018 10:07:49 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org Cc: Alexey Dobriyan , Johannes Weiner , Michal Hocko , Vlastimil Babka , linux-kernel@vger.kernel.org, kernel-team@fb.com, Roman Gushchin Subject: [RFC 2/4] mm: separate memory allocation and actual work in alloc_vmap_area() Date: Fri, 14 Dec 2018 10:07:18 -0800 Message-Id: <20181214180720.32040-3-guro@fb.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181214180720.32040-1-guro@fb.com> References: <20181214180720.32040-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP alloc_vmap_area() is allocating memory for the vmap_area, and performing the actual lookup of the vm area and vmap_area initialization. This prevents us from using a pre-allocated memory for the map_area structure, which can be used in some cases to minimize the number of required memory allocations. Let's keep the memory allocation part in alloc_vmap_area() and separate everything else into init_vmap_area(). Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Matthew Wilcox --- mm/vmalloc.c | 52 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 24a84d584609..24c8ab28254d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -395,16 +395,11 @@ static void purge_vmap_area_lazy(void); static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); -/* - * Allocate a region of KVA of the specified size and alignment, within the - * vstart and vend. - */ -static struct vmap_area *alloc_vmap_area(unsigned long size, - unsigned long align, - unsigned long vstart, unsigned long vend, - int node, gfp_t gfp_mask) +static int init_vmap_area(struct vmap_area *va, unsigned long size, + unsigned long align, unsigned long vstart, + unsigned long vend, int node, gfp_t gfp_mask) { - struct vmap_area *va; + struct rb_node *n; unsigned long addr; int purged = 0; @@ -416,11 +411,6 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, might_sleep(); - va = kmalloc_node(sizeof(struct vmap_area), - gfp_mask & GFP_RECLAIM_MASK, node); - if (unlikely(!va)) - return ERR_PTR(-ENOMEM); - /* * Only scan the relevant parts containing pointers to other objects * to avoid false negatives. @@ -512,7 +502,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, BUG_ON(va->va_start < vstart); BUG_ON(va->va_end > vend); - return va; + return 0; overflow: spin_unlock(&vmap_area_lock); @@ -534,10 +524,38 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, if (!(gfp_mask & __GFP_NOWARN) && printk_ratelimit()) pr_warn("vmap allocation for size %lu failed: use vmalloc= to increase size\n", size); - kfree(va); - return ERR_PTR(-EBUSY); + + return -EBUSY; +} + +/* + * Allocate a region of KVA of the specified size and alignment, within the + * vstart and vend. + */ +static struct vmap_area *alloc_vmap_area(unsigned long size, + unsigned long align, + unsigned long vstart, + unsigned long vend, + int node, gfp_t gfp_mask) +{ + struct vmap_area *va; + int ret; + + va = kmalloc_node(sizeof(struct vmap_area), + gfp_mask & GFP_RECLAIM_MASK, node); + if (unlikely(!va)) + return ERR_PTR(-ENOMEM); + + ret = init_vmap_area(va, size, align, vstart, vend, node, gfp_mask); + if (ret) { + kfree(va); + return ERR_PTR(ret); + } + + return va; } + int register_vmap_purge_notifier(struct notifier_block *nb) { return blocking_notifier_chain_register(&vmap_notify_list, nb); From patchwork Fri Dec 14 18:07:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10731525 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D6EA86C5 for ; Fri, 14 Dec 2018 18:07:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 158552D912 for ; Fri, 14 Dec 2018 18:07:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0433C2D930; Fri, 14 Dec 2018 18:07:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E4FC2D912 for ; Fri, 14 Dec 2018 18:07:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34A428E0200; Fri, 14 Dec 2018 13:07:54 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2FD018E01DC; Fri, 14 Dec 2018 13:07:54 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F32B8E0200; Fri, 14 Dec 2018 13:07:54 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id C531E8E01DC for ; Fri, 14 Dec 2018 13:07:53 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id e68so4115072plb.3 for ; Fri, 14 Dec 2018 10:07:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Cu/OfW+cnoBt6sWD0vpNz1lk+tLpOk7IYLf+TPCUhnk=; b=I2fMPqi1VuSBHUQBp0GINSwg2cXEq6pjZRn8/iPUepnku3aARpCJAtFrpJtcLrVaxz qN/2rlLbFuIn3lpJUW8lW4SkK2mQnYh0TgAfs/ir8Rp8vGxmJmyqt1Nvn+9FOKQYP44g Ph5om+JSu/UT3LXIVoiW/TDqk1K9D/P4snfvnwBd1dIESlk6x/Ubns2j5b05niaHuVP4 enVwg0Frlvcdch2xNC6+jdDzbgU7BR5uJvsCGUnqcKfV5qPHXtTAiCUpwEOIA5O1TUKs GelfXDvyVQh+8gC+AYu/0PLqh9rT9KBZo2yECb4nKRp2idWtNDd9jcyLyIZmqdIgsvmL ZYgQ== X-Gm-Message-State: AA+aEWZMhRvEuS9ZQZeOXjYgkOro2t8qYBumQL+qTr8lTPnPwU2zib3V KHdNXjlr9K89IuJmypllEs2FvQWDTB9QAPMK78Gu/u40fs8wuTvss+VqK172k9BR+zU2HQObO7J GHpf2UkB0pSJzxQBmfqvO2B3bLC/n90YCxYWP7221Kaf1tq8kRGDeGARC/HlPChD+TyxWBCcrY2 FGep3FIjCs3we9uXpqj2h446n4U/N/yHj8V+bem7mhOX3roNJDY56nK9R1MhgovJYCy+0pmnFXN B8IqezR+b+jZSe632Ey/oChQHCwbvOCASaqXlsQtXutJ3QCbuMCT+bPspAsTYglL0OeKKT1z9G9 lQK98uQrQahahECBrvu/RWZxbCPbSvjM8lSyM3gyyUT7ZG3O9ZRAFVCP2qKY80C4Ru0FRFUJkoz F X-Received: by 2002:a63:235f:: with SMTP id u31mr3572979pgm.122.1544810873455; Fri, 14 Dec 2018 10:07:53 -0800 (PST) X-Received: by 2002:a63:235f:: with SMTP id u31mr3572927pgm.122.1544810872566; Fri, 14 Dec 2018 10:07:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544810872; cv=none; d=google.com; s=arc-20160816; b=GabTJ3TCfTk8UZznyc8fHbeaoZ4SaRfvh08AIqT46Z/aEPVhnQdkwEvG2WtWaHWP+r THGW3GNzAOaJhy0mOF3MTTZkqXUwMH96j3MEYM0DQaj/K+XYDRyd9s7i8f2N4qXw9TYa mwGk8nmau76dHKRgbeAexGNe4ZFKoFeB6tzO3zCUYcuSVXZxkCXhKIPD78xoI8spmJ6k reURs1PUYOMT0u0vBNFROPYwxyOiTfCcybkPd+5TP+h5yn8In91+1X3leLwYtnCxr4/w UWzpPo9X8pIqBifKBaym5kusrcqlMJ2L+/tT6BOOsVk/D+772etoLzU7DEFQvGuThxf6 ahRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Cu/OfW+cnoBt6sWD0vpNz1lk+tLpOk7IYLf+TPCUhnk=; b=ekeVQjPIBrz8o8OkSC5xHRXZJ6LJge+c6k3+IhUWprwSbn7eqWOzpjo5dD2yB6gbjx Ndu5SokoWI50G52PpP4TNXV2+A/lMIawDc0GmvX04FEEjNmWeZTlI5I0z4THa0tf/JWx NV6P6vrQ3c5DIehNDIet6xHpmyHDIjopto5HeP9AaLzH4DIrx/NvWq/94tgG5F5nvSAI xS7K4okia21m14UaJSop1/ZrhGBN9HFP8smUVuYIdFNQvdx656omHR0E2tLcHS7mRXpE lMAsPKG5LGWvLPLfRBTJVv4fezdFEP+WTIQrCsvV3q96m/zoF+x+qfwOSDJiKcpdouhH m9Wg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Pi+CVtWd; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v28sor9504703pfk.14.2018.12.14.10.07.52 for (Google Transport Security); Fri, 14 Dec 2018 10:07:52 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Pi+CVtWd; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Cu/OfW+cnoBt6sWD0vpNz1lk+tLpOk7IYLf+TPCUhnk=; b=Pi+CVtWdIbcksx6L8kxgcpi+M3rxFgNK/QeCUI/Lb6gnQuuh1APN+0x7uSAK/HSWTf EfgNsasJujMSFkW4NYTvYyJVID6Q3zI5FKXc74vMFifI7IKRZnfCVHw2HhknVy4+4Vnn dUF9Xpeqbco6SScK8JjkMvinBvR0sDZi2qYHLe7OWD+XzmTM2Hk45tdVQDniDNNpzA9R rW7LMwDL5WIgrmM1qao08ev0nIeO2sBNHos1gqRDYAuUJWYKDwcL5xYlIeX1SDaxXMGb zYjfrECVPywTZbkxwRnUZ0Tvbhx7dVPfuofMfTrvtN2n402iStGfD7z8ebpYIAhn/Ek7 vNuA== X-Google-Smtp-Source: AFSGD/VD0is3Q+pAGp89nOAYixxBISYVqLojxpQohIub8YkqkUnyrB8BTcraPNhySoe4+M8XsGdobg== X-Received: by 2002:a62:f5da:: with SMTP id b87mr3782352pfm.253.1544810872100; Fri, 14 Dec 2018 10:07:52 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::5:3d97]) by smtp.gmail.com with ESMTPSA id r2sm7180856pgo.18.2018.12.14.10.07.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 14 Dec 2018 10:07:51 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org Cc: Alexey Dobriyan , Johannes Weiner , Michal Hocko , Vlastimil Babka , linux-kernel@vger.kernel.org, kernel-team@fb.com, Roman Gushchin Subject: [RFC 4/4] mm: show number of vmalloc pages in /proc/meminfo Date: Fri, 14 Dec 2018 10:07:20 -0800 Message-Id: <20181214180720.32040-5-guro@fb.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181214180720.32040-1-guro@fb.com> References: <20181214180720.32040-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Vmalloc() is getting more and more used these days (kernel stacks, bpf and percpu allocator are new top users), and the total % of memory consumed by vmalloc() can be pretty significant and changes dynamically. /proc/meminfo is the best place to display this information: its top goal is to show top consumers of the memory. Since the VmallocUsed field in /proc/meminfo is not in use for quite a long time (it has been defined to 0 by the commit a5ad88ce8c7f ("mm: get rid of 'vmalloc_info' from /proc/meminfo")), let's reuse it for showing the actual physical memory consumption of vmalloc(). Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- fs/proc/meminfo.c | 2 +- include/linux/vmalloc.h | 3 +++ mm/vmalloc.c | 10 ++++++++++ 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 568d90e17c17..465ea0153b2a 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -120,7 +120,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v) show_val_kb(m, "Committed_AS: ", committed); seq_printf(m, "VmallocTotal: %8lu kB\n", (unsigned long)VMALLOC_TOTAL >> 10); - show_val_kb(m, "VmallocUsed: ", 0ul); + show_val_kb(m, "VmallocUsed: ", vmalloc_nr_pages()); show_val_kb(m, "VmallocChunk: ", 0ul); show_val_kb(m, "Percpu: ", pcpu_nr_pages()); diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 1205f7a03b48..ea3a0fb3a60f 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -64,10 +64,12 @@ extern void vm_unmap_aliases(void); #ifdef CONFIG_MMU extern void __init vmalloc_init(void); +extern unsigned long vmalloc_nr_pages(void); #else static inline void vmalloc_init(void) { } +static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif extern void *vmalloc(unsigned long size); @@ -83,6 +85,7 @@ extern void *__vmalloc_node_range(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller); + #ifndef CONFIG_MMU extern void *__vmalloc_node_flags(unsigned long size, int node, gfp_t flags); static inline void *__vmalloc_node_flags_caller(unsigned long size, int node, diff --git a/mm/vmalloc.c b/mm/vmalloc.c index edd76953c23c..39db12fe8931 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -340,6 +340,13 @@ static unsigned long cached_align; static unsigned long vmap_area_pcpu_hole; +atomic_long_t nr_vmalloc_pages; + +unsigned long vmalloc_nr_pages(void) +{ + return atomic_long_read(&nr_vmalloc_pages); +} + static struct vmap_area *__find_vmap_area(unsigned long addr) { struct rb_node *n = vmap_area_root.rb_node; @@ -1594,6 +1601,7 @@ static void __vunmap(const void *addr, int deallocate_pages) BUG_ON(!page); __free_pages(page, 0); } + atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); if (area->flags & VM_EXT_PAGES) kvfree(area->pages); @@ -1739,12 +1747,14 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ area->nr_pages = i; + atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } area->pages[i] = page; if (gfpflags_allow_blocking(gfp_mask|highmem_mask)) cond_resched(); } + atomic_long_add(area->nr_pages, &nr_vmalloc_pages); if (map_vm_area(area, prot, area->pages)) goto fail;