From patchwork Wed May 9 01:33:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10387941 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2197B60236 for ; Wed, 9 May 2018 01:34:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E17B628FF5 for ; Wed, 9 May 2018 01:34:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D408D28FF7; Wed, 9 May 2018 01:34:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A10C828FF5 for ; Wed, 9 May 2018 01:34:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 550D66B030C; Tue, 8 May 2018 21:34:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 411096B030D; Tue, 8 May 2018 21:34:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B44B6B030E; Tue, 8 May 2018 21:34:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f199.google.com (mail-qk0-f199.google.com [209.85.220.199]) by kanga.kvack.org (Postfix) with ESMTP id F14956B030C for ; Tue, 8 May 2018 21:34:16 -0400 (EDT) Received: by mail-qk0-f199.google.com with SMTP id l65so5527327qke.21 for ; Tue, 08 May 2018 18:34:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=fwmmgNrFgs8nPPXKsKxZE84u81qY7Q6cdbb0N8Y8nTGm6YF7+D2sz4pWErO6yNKKm3 GcZBmsPJxmsIe62bfQr8eck0vpJ4l2jSLQ/esj/8RJX+DH54s2rQWL88c4QWde/1TFH6 7OBQJz45iSl3v6Zn0inQJzcKuFqzbBNyENhhSW+0RemPYk+cMmM9BmG6TGTz+zUz7kb9 /tfn+9sARA8kdksglCniTX3cGBaXk0HBEgMHfG0tadWdz55dReRRo0mom7U+3hZnY65c 4UjFjlhR/0M+qdJA1pTGw9coGmrhjL24QfsQ22jsrSufqnGEIt9UZow3aJ/mkFSWCUyH Gbsw== X-Gm-Message-State: ALQs6tCnhOUmCwYJ9Q02rcgCaHqaf+B0wDMc70OLKz5CCr5yAVbDmr9L o94E0NzngXaMZocglcql3mXn7AHH9RnI4EkddeSXXUGKj4c9y/8hZVYTQlgmXdjPyAvDqINbXm8 I+V4OlQAXibrJnDAaAR3gg+WagpNTn68xPvdwHInwtS3LBudKXz+9tbALyUlgAZbTYZfU+nR5pi zab+VZLrEqwYIOKPVuffysKKem542yeABEHTiA7rEf33SnyGw/uPuk1hOu51mxlt+GNYX5A+lGc fdES9499la3JTiepAP6b2W1j9LsADstECiqaH6nqBgHx3ho/2+MVC9GPLYmxDXWbF77qGYhDux5 IFv+cbf9x+cCR6JkR6U0woJ9jhN1TczW+9HR6ZNK4wH471BsH+k8FRBjwkaZl5D3oKZHFrN157S O X-Received: by 2002:ac8:30cd:: with SMTP id w13-v6mr39320772qta.289.1525829656714; Tue, 08 May 2018 18:34:16 -0700 (PDT) X-Received: by 2002:ac8:30cd:: with SMTP id w13-v6mr39320738qta.289.1525829655932; Tue, 08 May 2018 18:34:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525829655; cv=none; d=google.com; s=arc-20160816; b=NHGhvSDM9DQmT43nK/zxFkibA6yb++eQwYqQfyW5YWRmHque/8632HrTVMUYmsFg+c Kl9jkCe4c29TMqm0DDVIOOiXVmetnCFS2oKJmK1zlgx8KBYMGfNquLLTt5s79urn5YJ2 DSukIlmyiDazrxjWE4E8xOJCNBEMQROOnmqIjaqufNxOjvMPNZ3wx/qcgjT0hHLuiAOE 61ljEvgAYXUMz65eSYHEtr906Dsvp8Tqwb8eqklJkXmgVXRCVS4NzDHESan6lp2t7NX+ t9dJHqjq1bRgSDSBz7ULEH3ZYBEANE2ef0B7+p9lMFGZrf9OQEghZlsQ7fjF1Rvaq+Zl Gdxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=kWaV4TP4WFi/6OvgDly14vI+f1lOo3VNOhdYhwThhLwbPGGz6xuMS0MU96smdIfDse w5MPR5IJLRCAtPDyMit3ZWVnSbUQZySV324zkqkBtv08ANhI7Zw0aMt8po9I8BOMj1+w m2ufxfuQiUg6PBeV7B8WJWmPxx6tk0CR2LqIWptncuiW9JIw314aG9YaxuNsbIzDjCmC biyE1v7ilQfZrerbguRyIwW/44TNzL+himO3L6kPG45cM65ypvDRRdpHJOejtCDPbQoA Bd2EdFaacLEg9BOiZfXXR/I3Mlks/GXIYqkMi8hx+NMBdkH+9/rGyJPrjrdweA7JOsND +P6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=OgQQUPVX; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id f31-v6sor16506518qta.8.2018.05.08.18.34.15 for (Google Transport Security); Tue, 08 May 2018 18:34:15 -0700 (PDT) Received-SPF: pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=OgQQUPVX; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DukDmIDOCuZvOvYV7AxI2za4rxww2vVAjhaHcLiyQT0=; b=OgQQUPVXu1Wt/pYZ4W5jyv14YX7JAGgy5HLjbGjNtTts6rquJEQMUxvekITplCxZGh B5/ZpkwxSm4oubpOgjYDR6wth9+IXwNAQnvsqszh2O0Vw8oj0Hat3uzDiM9HqbovoXC6 pbLNb59dhaa2hlI6d1WrIQusXHrNcJjwV9ybqn/lg7GmLev9kTkMstBjVXfN2NdeorJR jNTz4emOlZvcFnVawsbAQ4H3Fa5FsYMcFfV74jRVpSaR3ksVxE9SOLKQvcHn5IidhN0T GW+b3W2ektVNCcNrRWFUt7qKrPGowLF3aB+rdgPt3RPP/0AwE6zI2T3XNpk8ex6srvdp SrPQ== X-Google-Smtp-Source: AB8JxZqxwOfBPbgc/Xd1VBkVo72H35ZEwJTEQUI1/fPTSd8YWm+1/uxvAkGA/vavIT582wu32kKM2A== X-Received: by 2002:aed:2ba3:: with SMTP id e32-v6mr39821933qtd.272.1525829655527; Tue, 08 May 2018 18:34:15 -0700 (PDT) Received: from localhost.localdomain (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id x28-v6sm23719003qtx.95.2018.05.08.18.34.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 18:34:14 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, Jens Axboe , Ingo Molnar Cc: Kent Overstreet Subject: [PATCH 01/10] mempool: Add mempool_init()/mempool_exit() Date: Tue, 8 May 2018 21:33:49 -0400 Message-Id: <20180509013358.16399-2-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180509013358.16399-1-kent.overstreet@gmail.com> References: <20180509013358.16399-1-kent.overstreet@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Allows mempools to be embedded in other structs, getting rid of a pointer indirection from allocation fastpaths. mempool_exit() is safe to call on an uninitialized but zeroed mempool. Signed-off-by: Kent Overstreet --- include/linux/mempool.h | 34 +++++++++++++ mm/mempool.c | 108 ++++++++++++++++++++++++++++++---------- 2 files changed, 115 insertions(+), 27 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index b51f5c430c..0c964ac107 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -25,6 +25,18 @@ typedef struct mempool_s { wait_queue_head_t wait; } mempool_t; +static inline bool mempool_initialized(mempool_t *pool) +{ + return pool->elements != NULL; +} + +void mempool_exit(mempool_t *pool); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id); +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data); + extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, @@ -43,6 +55,14 @@ extern void mempool_free(void *element, mempool_t *pool); */ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); + +static inline int +mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) +{ + return mempool_init(pool, min_nr, mempool_alloc_slab, + mempool_free_slab, (void *) kc); +} + static inline mempool_t * mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) { @@ -56,6 +76,13 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) */ void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); + +static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) +{ + return mempool_init(pool, min_nr, mempool_kmalloc, + mempool_kfree, (void *) size); +} + static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) { return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, @@ -68,6 +95,13 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) */ void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); + +static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) +{ + return mempool_init(pool, min_nr, mempool_alloc_pages, + mempool_free_pages, (void *)(long)order); +} + static inline mempool_t *mempool_create_page_pool(int min_nr, int order) { return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, diff --git a/mm/mempool.c b/mm/mempool.c index 5c9dce3471..df90ace400 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -137,6 +137,28 @@ static void *remove_element(mempool_t *pool, gfp_t flags) return element; } +/** + * mempool_destroy - exit a mempool initialized with mempool_init() + * @pool: pointer to the memory pool which was initialized with + * mempool_init(). + * + * Free all reserved elements in @pool and @pool itself. This function + * only sleeps if the free_fn() function sleeps. + * + * May be called on a zeroed but uninitialized mempool (i.e. allocated with + * kzalloc()). + */ +void mempool_exit(mempool_t *pool) +{ + while (pool->curr_nr) { + void *element = remove_element(pool, GFP_KERNEL); + pool->free(element, pool->pool_data); + } + kfree(pool->elements); + pool->elements = NULL; +} +EXPORT_SYMBOL(mempool_exit); + /** * mempool_destroy - deallocate a memory pool * @pool: pointer to the memory pool which was allocated via @@ -150,15 +172,65 @@ void mempool_destroy(mempool_t *pool) if (unlikely(!pool)) return; - while (pool->curr_nr) { - void *element = remove_element(pool, GFP_KERNEL); - pool->free(element, pool->pool_data); - } - kfree(pool->elements); + mempool_exit(pool); kfree(pool); } EXPORT_SYMBOL(mempool_destroy); +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + spin_lock_init(&pool->lock); + pool->min_nr = min_nr; + pool->pool_data = pool_data; + pool->alloc = alloc_fn; + pool->free = free_fn; + init_waitqueue_head(&pool->wait); + + pool->elements = kmalloc_array_node(min_nr, sizeof(void *), + gfp_mask, node_id); + if (!pool->elements) + return -ENOMEM; + + /* + * First pre-allocate the guaranteed number of buffers. + */ + while (pool->curr_nr < pool->min_nr) { + void *element; + + element = pool->alloc(gfp_mask, pool->pool_data); + if (unlikely(!element)) { + mempool_exit(pool); + return -ENOMEM; + } + add_element(pool, element); + } + + return 0; +} +EXPORT_SYMBOL(mempool_init_node); + +/** + * mempool_init - initialize a memory pool + * @min_nr: the minimum number of elements guaranteed to be + * allocated for this pool. + * @alloc_fn: user-defined element-allocation function. + * @free_fn: user-defined element-freeing function. + * @pool_data: optional private data available to the user-defined functions. + * + * Like mempool_create(), but initializes the pool in (i.e. embedded in another + * structure). + */ +int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return mempool_init_node(pool, min_nr, alloc_fn, free_fn, + pool_data, GFP_KERNEL, NUMA_NO_NODE); + +} +EXPORT_SYMBOL(mempool_init); + /** * mempool_create - create a memory pool * @min_nr: the minimum number of elements guaranteed to be @@ -186,35 +258,17 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, gfp_t gfp_mask, int node_id) { mempool_t *pool; + pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); if (!pool) return NULL; - pool->elements = kmalloc_array_node(min_nr, sizeof(void *), - gfp_mask, node_id); - if (!pool->elements) { + + if (mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id)) { kfree(pool); return NULL; } - spin_lock_init(&pool->lock); - pool->min_nr = min_nr; - pool->pool_data = pool_data; - init_waitqueue_head(&pool->wait); - pool->alloc = alloc_fn; - pool->free = free_fn; - /* - * First pre-allocate the guaranteed number of buffers. - */ - while (pool->curr_nr < pool->min_nr) { - void *element; - - element = pool->alloc(gfp_mask, pool->pool_data); - if (unlikely(!element)) { - mempool_destroy(pool); - return NULL; - } - add_element(pool, element); - } return pool; } EXPORT_SYMBOL(mempool_create_node);