From patchwork Wed Jul 18 13:36:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10532475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D5494601D2 for ; Wed, 18 Jul 2018 13:36:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6221292E3 for ; Wed, 18 Jul 2018 13:36:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B951F293BB; Wed, 18 Jul 2018 13:36:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29404292E3 for ; Wed, 18 Jul 2018 13:36:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E406D6B0007; Wed, 18 Jul 2018 09:36:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DEF066B000C; Wed, 18 Jul 2018 09:36:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDD456B000D; Wed, 18 Jul 2018 09:36:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 6D9DC6B0007 for ; Wed, 18 Jul 2018 09:36:31 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id d5-v6so1949537edq.3 for ; Wed, 18 Jul 2018 06:36:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7HUQv07uBdk1KPumkfGsGUVHBi4ADcLsUAztLR2H/O8=; b=Rus5zz82ZCDBZHiELynuLVawhTdX5Ej/rwreUWuSLr4ICBtExB+sM0CKvst0hltvd8 vvno3M728UVnPsKa5ikKhdp7fd5n3VJLIWkXMsRH07unzEYiYkYgldOh5KxT1bfuI6Jz vgdVOk7AACPeqZcXtQ0ke2bjjD0HGp52PJ8MymoFWNtNwRNH377jD7YK5sJ8R1NlqRjf +CfuCN/+xIRef/ev7B+8JnUi2Zn7DCGGW4gAlwfBiyl1Ev4NG/lafJhu4OMkQGk3cMDh zqrz69ESZPGaULqdYP0bT/oDGEjV+r3nRCs5nMD/huXH7pe8Td8SOAaBnRmeA7Ea5Go/ eafA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlEb7YMcOK69JF/gGPOwlz30keMLgCVi5ttNJP46s2BQhDFOe5Ot h/SAICJXvQzbgSEqh9XxKIy7D4D5Fht7YofpnhlxGyXfr+LC68JJh/wNrVvO0H+fMtgG++47QTw bCLu6cEE9JCukMXilaix8y/zIRGx2/xc8y826fxXgfAsiKYqgOdC6EmZH+8/N0qbNAQ== X-Received: by 2002:a50:d989:: with SMTP id w9-v6mr7339701edj.63.1531920990912; Wed, 18 Jul 2018 06:36:30 -0700 (PDT) X-Google-Smtp-Source: AAOMgpefeTkiy1t/Yc6Z1sirXAkL6G7YoxyPufpwgt9Ral8SDZ7Kbncm/T0rBuSbT4vc4/QE6Wew X-Received: by 2002:a50:d989:: with SMTP id w9-v6mr7339626edj.63.1531920989816; Wed, 18 Jul 2018 06:36:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531920989; cv=none; d=google.com; s=arc-20160816; b=SEF0p+Oex5CtbKZRun6nUwDv8B+V7saqMCoPYPi1ng8AS6VRCn840IZujJ63RxxRzQ 7TrsJFobhx5dqM3pRZRIoov2fOr1drA3Jr2sp0USqlv2KTRf7IJX8Ll545Qs8Gt5baXj nhSkrAagIaPm/cDWpz0TC4XI8TDDug5GyFY+bduQw+fuf2NtrAoG2N74XW7e50pGoKPg f9XLrxFPT8QVVB+NbTALlfzm7b1aS0GeK0rFuiSqjL1eaSEzsw9K8EVopYAcNw9esLEW t8oofLsEQI0Fo/AJPGeJWzkI70bJR7oUK3KEkl2JEoOhwffvHgl3rv6IOgXCfD4W+4oQ taZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=7HUQv07uBdk1KPumkfGsGUVHBi4ADcLsUAztLR2H/O8=; b=DLrJymMK91wwal9/n9SpW22Pxt7JbKfKKa39rZiFFmOn6YLlfjzrgsUWZZSkPW3Qbv 88BMvMisN43gV9j9S1czwio0u2eOc5G29LqOHLvVkiuSR6gyb9G+jO8aSPsg5SWe+ZOh KWlTJ8OBFvBcrbKKz3EdANmsHsNx6l3MqjWC4FnUWEtQ/4gvh0wykdH5qBOCqUZ6Kqon Gh77rmzUcsstk8HWLxZvqavYn3OA5LYVq9hsl88TLTpP+dkvppcS+/1V1Z8GSKjtv/hc 7t6t1K+mxlIKiQznQbhfJ/KqrwmYOPvf0tuDBHVcawuHk8m9/C1eJRWj7L08H7DNl4lA H2LQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id b40-v6si1085362edf.140.2018.07.18.06.36.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Jul 2018 06:36:29 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id D1E5DAD33; Wed, 18 Jul 2018 13:36:28 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v3 2/7] mm, slab/slub: introduce kmalloc-reclaimable caches Date: Wed, 18 Jul 2018 15:36:15 +0200 Message-Id: <20180718133620.6205-3-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180718133620.6205-1-vbabka@suse.cz> References: <20180718133620.6205-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Kmem caches can be created with a SLAB_RECLAIM_ACCOUNT flag, which indicates they contain objects which can be reclaimed under memory pressure (typically through a shrinker). This makes the slab pages accounted as NR_SLAB_RECLAIMABLE in vmstat, which is reflected also the MemAvailable meminfo counter and in overcommit decisions. The slab pages are also allocated with __GFP_RECLAIMABLE, which is good for anti-fragmentation through grouping pages by mobility. The generic kmalloc-X caches are created without this flag, but sometimes are used also for objects that can be reclaimed, which due to varying size cannot have a dedicated kmem cache with SLAB_RECLAIM_ACCOUNT flag. A prominent example are dcache external names, which prompted the creation of a new, manually managed vmstat counter NR_INDIRECTLY_RECLAIMABLE_BYTES in commit f1782c9bc547 ("dcache: account external names as indirectly reclaimable memory"). To better handle this and any other similar cases, this patch introduces SLAB_RECLAIM_ACCOUNT variants of kmalloc caches, named kmalloc-rcl-X. They are used whenever the kmalloc() call passes __GFP_RECLAIMABLE among gfp flags. They are added to the kmalloc_caches array as a new type. Allocations with both __GFP_DMA and __GFP_RECLAIMABLE will use a dma type cache. This change only applies to SLAB and SLUB, not SLOB. This is fine, since SLOB's target are tiny system and this patch does add some overhead of kmem management objects. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter --- include/linux/slab.h | 16 +++++++++++---- mm/slab_common.c | 48 ++++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 21 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4299c59353a1..d89e934e0d8b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -296,11 +296,12 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, (KMALLOC_MIN_SIZE) : 16) #define KMALLOC_NORMAL 0 +#define KMALLOC_RECLAIM 1 #ifdef CONFIG_ZONE_DMA -#define KMALLOC_DMA 1 -#define KMALLOC_TYPES 2 +#define KMALLOC_DMA 2 +#define KMALLOC_TYPES 3 #else -#define KMALLOC_TYPES 1 +#define KMALLOC_TYPES 2 #endif #ifndef CONFIG_SLOB @@ -309,12 +310,19 @@ extern struct kmem_cache *kmalloc_caches[KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; static __always_inline unsigned int kmalloc_type(gfp_t flags) { int is_dma = 0; + int is_reclaimable; #ifdef CONFIG_ZONE_DMA is_dma = !!(flags & __GFP_DMA); #endif - return is_dma; + is_reclaimable = !!(flags & __GFP_RECLAIMABLE); + + /* + * If an allocation is botth __GFP_DMA and __GFP_RECLAIMABLE, return + * KMALLOC_DMA and effectively ignore __GFP_RECLAIMABLE + */ + return (is_dma * 2) + (is_reclaimable & !is_dma); } /* diff --git a/mm/slab_common.c b/mm/slab_common.c index 4614248ca381..614fb7ab8312 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1107,10 +1107,21 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init new_kmalloc_cache(int idx, slab_flags_t flags) +static void __init +new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( - kmalloc_info[idx].name, + const char *name; + + if (type == KMALLOC_RECLAIM) { + flags |= SLAB_RECLAIM_ACCOUNT; + name = kasprintf(GFP_NOWAIT, "kmalloc-rcl-%u", + kmalloc_info[idx].size); + BUG_ON(!name); + } else { + name = kmalloc_info[idx].name; + } + + kmalloc_caches[type][idx] = create_kmalloc_cache(name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1122,22 +1133,25 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i; - int type = KMALLOC_NORMAL; + int i, type; - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, flags); + for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { + for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + if (!kmalloc_caches[type][i]) + new_kmalloc_cache(i, type, flags); - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) - new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) - new_kmalloc_cache(2, flags); + /* + * Caches that are not of the two-to-the-power-of size. + * These have to be created immediately after the + * earlier power of two caches + */ + if (KMALLOC_MIN_SIZE <= 32 && i == 6 && + !kmalloc_caches[type][1]) + new_kmalloc_cache(1, type, flags); + if (KMALLOC_MIN_SIZE <= 64 && i == 7 && + !kmalloc_caches[type][2]) + new_kmalloc_cache(2, type, flags); + } } /* Kmalloc array is now usable */